I need to replicate and continuously sync a MySQL database to a graph database, to further query and analyze data from an eCommerce application.
Came across DZone Article which talks about syncing data between oracle and Neo4J. They are using Oracle GoldenGate and Apache Kafka for this.
However, I have a few questions on how this would be implemented -
What about the graph data model? Can I generate graph data model from existing RDBMS data model? Are there any open source ETL solutions to map data models between source and target?
The RDBMS data model may change as part of new features. How to keep data models in sync between a source RDBMS and target graph database?
Are there any other approaches to make this work?
I'm fairly new to SQL to NoSQL replication. Any inputs?
Related
Current version of our application holds data in MySQL. We have a new version, which is developed in Flowable.
What is the best way to migrate data from MySQL to Flowable?
Is there any APIs available to do this task?
If not, is there any
documentation about how to design an ETL process to load data to
Flowable?
I can see Flowable Database schema in their documentations but I couldn't find any clue about migrating data from an external system to Flowable. Any help will be much appreciated.
Flowable is not a database therefore migrating data from MySQL to Flowable is not something that can be done.
Flowable supports storing its data in different Databases.
What you are most likely looking for is migrating your business logic to Flowable. There are ways to do this using the Flowable Java APIs in order to bring your old business data in the new state you will need using Flowable.
There is no specific documentation for this because this highly depends on your own business needs.
I am new to MS Azure just with some SQL Server Background and now we are facing some design / architecture questions and I am somehow lost.
One the one hand, there is a DataWarehouse and a small SQL-Database in Azure and they store all the structured or not structured incoming data. Works fine!
Now we think of moving the MySQL Database for the first version of the website (we need to stay at MySQL for the web-service) to Azure. In version two of the website, we like to integrate some of the Data from SQL-Database and DataWarehouse so it sounds good to have all the stuff at one place.
As much as possible from all the structured Data, we would like to store at the SQL-Database and not in MySQL. MySQL should stay lightweight. But what will be the beste way to create some interaction between Webservice, MySQL und SQL-Database?
Our Webdesigner asked for some APIs and as the users should be able to change some settings in their account we would need a lot of get and set APIs. And those APIs will just handle traffic within Azure, no external Access is needed. I just discovered the option of external Tables in Azure MySQL but cant find use-cases or best practice of that.
I am looking for a solution, in which I can deliver the necessary data for the Webdesigner / Frontend and they do not need to work with any databases (as they do not like to do that).
The traffic between MySQL and SQL-Database will be low (our stream goes into Datawarehouse, some analysis in there and we save the results as structured Data in the SQL-Database) and up to now we do not need a connection between Datawarehouse and MySQL/Webservice.
Any suggestions? How would you design such a connection?
Using Azure Data Factory you can maintain both databases and transfer (called as COPY in Data Factory) from SQL DB to Azure DB.
I'm in a traditional Back to Front ETL stack from a data source (Adobe Analytics) to MySQL data warehouse to a Tableau front end for visualization. My question revolves around best practices for cleaning data / mapping and at what step.
1) Cleaning: We have no automated (SSIS, etc.) connector from the source (Adobe) to the data warehouse so we're left with periodic uploading of CSV files. For various reasons these files become less than optimal (misspellings, nulls, etc.) Question: should the 'cleaning' be done to the CSV files, or once the data is uploaded into MySQL data warehouse (in tables/views)?
2) Mapping: a number of different end user use cases require us to map the data to tables (geographic regions, type of accounts, etc.)... should this be done in the data warehouse (MySQL joins) or is it just as good in the front end (Tableau)? The real question pertains to performance, I believe, as you could do it relatively easily in either step.
Thanks!
1) Cleaning: I'd advise you to load the data in the CSV files into a staging database and clean it from there, before it reaches the database to which you connect Tableau to. This way you can keep the original files, which you can eventually reload if necessary. I'm not sure what a "traditional Back to Front ETL stack" is, but an ETL tool like Microsoft SSIS or Pentaho Data Integrator (free) will be a valuable help with building these processes, and you could then run your ETL jobs periodically or every time a new file is uploaded to the directory. Here is a good example of such a process: https://learn.microsoft.com/en-us/sql/2014/integration-services/lesson-1-create-a-project-and-basic-package-with-ssis
2) "Mapping": You should have a data model, probably a dimensional model, built on the database that Tableau connects to. This data model should store clean and "business modelled" data. You should perform the lookups (joins/mappings) when you are Transforming your data, so you can Load it into the data model. Having Tableau explore a dimensional model of clean data will also be better for UX/performance.
The overall flow would look something like: CSV -> Staging database -> Clean/Transform/Map -> Business data model (database) -> Tableau
I am novice in mysql server.
I use latest version of worbench.
using designer I created model:
But in databases I don't see my model:
I bad in mysql termins But I want to get so thing where I can insert data to Student and Group tables.
Modeling is more an abstract work to develop your schema structure systematically. It helps you to define needed objects, their relationships, users etc. that are needed. A model however has no data or any real representation of the objects it contains.
In order to use the objects you have to bring them on a real server. There are different ways to accomplish that, for instance you can forward engineer your model to a server (see Database menu). You have to select (or create) a connection to a server of your choice and let MySQL Workbench generate and apply a script to generate all the objects. Once this is done you can open those objects in the SQL IDE to fill them with data or work on them.
MySQL Workbench also has a synchronization feature that you can use after forward engineering. It's a two-way sync that allows to apply changes in your model to the schema objects on your server and also take over any changes made there into your model. All that can be fine tuned in the sync wizard.
I am trying to migrate a website from one content management system to another. There is a lot of content related data stored in a MySQL database, which will be stored in a different schema in the new database. I would want to somehow map the old schema onto the new and copy the data accordingly. Is there a free tool for such a task - or is there a native MySQL way of doing this?
By free, I mean free as in free beer.
To map data and migrate between two different schema look at ETL like Talend or Pentaho which are "free".