BigBear.ai
  • Home
  • Industries
    • Academia
    • Government
    • Healthcare
    • Manufacturing
  • Solutions
    • Cyber
    • Data Analytics
    • Enterprise Planning and Logistics
    • Intelligent Automation
    • Modeling Solutions
    • Professional Services
  • Company
    • About
    • Investor Relations
    • Partners
    • Team
  • Careers
    • Benefits
    • Culture
    • Explore Jobs
    • Military and Veterans
    • Applicant Login
    • Employee Login
  • Resources
    • Blog
    • Events
    • Newsroom
    • Resource Library
    • Online Store
  • Contact
Search

Home ETL Custom Conversion from a Relational Database to NEO4J

Blog

Custom Conversion from a Relational Database to NEO4J

Jim McHugh
July 13, 2021
  • Share
  • Share

My colleague, Justin Rasband, and I have been working on an exciting new project. We are converting a legacy relational database to a NEO4J graph database. The challenge set before us by the customer was to create a cost-effective, flexible, and scalable ETL solution that will allow the user to move the table attribute data into customized node hierarchies and relationships with metadata at every level.

We started our research by looking at existing software tools. We assumed that many people had done this before, and we could just use an existing software tool to do the work for us. First, we looked at creating cypher scripts. This was quickly dismissed as there were over 660 tables and 7K attributes that need to be converted. Next, we looked at loading directly from Excel but the challenge here was all attributes became node properties and we could not customize the hierarchy. We then tested GraphQL but it too was limited to converting tables to nodes. Finally, we looked at other graph ETL solutions but the ones we tested were immature and costly.

We knew that we needed to think outside the box, or the node circle, to solve this challenge and came up with a novel solution using a customized CSV file and an open-source ETL tool called KNIME that allowed us to meet all our customer requirements.

We started by dumping the data model (table name, column name, data type) out into a CSV file. We then worked with our customer to define the data contained in each node. Each attribute was placed in a separate attribute state node so that we could capture history, a feature the customer has wanted to implement for several years. We then created a node hierarchy and decided where in this new hierarchy each attribute node would be created. Finally, we created a relationship section to address those nodes not in a hierarchy that need to be connected.

Once the CSV file was complete, we went looking for an ETL tool that would meet our needs. We looked at several tools, but the KNIME Analytics Platform (KNIME) stood out as the best solution to our challenge. KNIME is an open-source software platform that allows users to create visual workflows using thousands of intuitive, drag and drop nodes and components found in the KNIME Hub. This platform provided us with the flexibility to leverage the CSV file we just created and create the workflow we needed to meet the customer’s requirements. In fact, we were able to do this by using less than 50 KNIME nodes and components. Please watch our NODES 2021 presentation on Custom Conversion from a Relational Database to NEO4J for an overview of the project and a demo of this easy but powerful solution.

 

About the Author

Jim McHugh is the Vice President of National Intelligence Service – Emerging Markets Portfolio. Jim is responsible for the delivery of Analytics and Data Management to the Intelligence Community.

Posted in Data Architecture, Data Modeling, ETL.

Related Content

Streaming_Workflow_500x204
ETLKNIME
Processing Large Datasets With KNIME
KNIME
KNIME: Table to JSON Node – Decoded
Data ModelingKNIME
What the Farkle? Recursion, Modeling, and Genetic Algorithms with KNIME, Part 1
BigBear.ai
  • Home
  • Industries
  • Solutions
  • Company
  • Careers
  • Blog
  • Investor Relations
  • Contact
  • Twitter
  • Facebook
  • Linkedin
  • Google My business for BigBear.ai
1-410-312-0885
[email protected]
  • Privacy Policy
  • Terms of Use
  • Accessibility
  • Site Map
© BigBear.ai 2023
We value your privacy
We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies.
Privacy Policy | Do not sell my personal information
AcceptCookie Settings
Manage Consent

Cookies Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
JSESSIONIDsessionThe JSESSIONID cookie is used by New Relic to store a session identifier so that New Relic can monitor session counts for an application.
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
CookieDurationDescription
__atuvc1 year 1 monthAddThis sets this cookie to ensure that the updated count is seen when one shares a page and returns to it, before the share count cache is updated.
__atuvs30 minutesAddThis sets this cookie to ensure that the updated count is seen when one shares a page and returns to it, before the share count cache is updated.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
CookieDurationDescription
_ga2 yearsThe _ga cookie, installed by Google Analytics, calculates visitor, session and campaign data and also keeps track of site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognize unique visitors.
_ga_NK4L4Q320Q2 yearsThis cookie is installed by Google Analytics.
_gat_gtag_UA_163894009_21 minuteSet by Google to distinguish users.
_gid1 dayInstalled by Google Analytics, _gid cookie stores information on how visitors use a website, while also creating an analytics report of the website's performance. Some of the data that are collected include the number of visitors, their source, and the pages they visit anonymously.
at-randneverAddThis sets this cookie to track page visits, sources of traffic and share counts.
CONSENT2 yearsYouTube sets this cookie via embedded youtube-videos and registers anonymous statistical data.
uvc1 year 1 monthSet by addthis.com to determine the usage of addthis.com service.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
CookieDurationDescription
f5avraaaaaaaaaaaaaaaa_session_sessionbusinesswire.com cookie
loc1 year 1 monthAddThis sets this geolocation cookie to help understand the location of users who share the information.
VISITOR_INFO1_LIVE5 months 27 daysA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.
YSCsessionYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.
yt-remote-connected-devicesneverYouTube sets this cookie to store the video preferences of the user using embedded YouTube video.
yt-remote-device-idneverYouTube sets this cookie to store the video preferences of the user using embedded YouTube video.
Save & Accept