I want to query data from two different MySQL databases to a new MySQL database.
I have two databases with a lot of irrelevant data and I want to create what can be seen as a data warehouse where only relevent data should be present coming from the two databases.
As of now all data gets sent to the two old databases, however I would like to have scheduled updating so the new database is up to speed. There is a key between the two databases so in best case I would like all data to be present in one table however this is not crucial.
I have done similar work with Logstash and ES, however I do not know how to do it when it comes to MySQL.
Best way to do that is create a ETL process with Pentaho Data Integrator or any ETL tool. Where your source will be two different databases, in the transformation part you can remove or add any business logic then load those data into new database.
If you create this ETL you can schedule it once a day so that your database will be up to date.
If you want to do this without an ETL than your database must be in same host. Than you can just add database name just before table name in query. like SELECT * FROM database.table_name
Related
I have two different databases. I have to access data from one database and insert them into another ( with some data processing included, it is not only to copy data ) Also, the schema is really complex and each table has many rows, so copying data into schema in the second database is not an option. I have to do that using MySQL Workbench, so I have to do it using SQL queries. Is there a way to create a connection from one database to another and access its data?
While MySQL Workbench can be used to transfer data between servers (e.g. as part of a migration process) it is not useful when you have to process the data first. Instead you have 2 other options:
Use a dedicated tool you write yourself to do that (as eddwinpaz mentioned).
Use the capabilities of your server. That is, copy the data to the target server, into a temporary table (using dump and restore). Then use queries to modify the data as you need it. Finally copy it to the target table.
I have to compare two database tables and fields ( Not their DATA) .
Say Live DB and Development DB. In live DB there are 200 tables and
in development DB there are 218 Tables With some new column fields
added in old tables too.
During the final stage i have to update the Live DB by adding the new
columns and tables taken from the development DB .
I should not loss and data in the old DB.
I tried many ways to do this but it is taking a very long time for me
to finish. One of the tool i used is mysql workbench.
Is there any queries to perform this using information schema ?
To make the things easier i imported the development DB from development server and uploaded in the live db with different name . Now both the DB are in the same server but with diff name. E.g. sitedb , sitedevdb
Whenever you update the development DB with changes that will eventually be put live, you should write a .sql script that will repeat the changes on the live database.
Otherwise this operation is super-tricky.. you may have to manually figure out the added columns yourself, or you may be able to use each INFORMATION_SCHEMA.COLUMNS to compare the old tables.
For the new tables SHOW CREATE **table_name** is really useful.
A MySQL comparison tool is probably what you need for this. The company I work for, Redgate, offers MySQL Compare (for the database schema) and MySQL Data Compare (for data).
These tools are free for non-commercial use.
I have a question which is what I believe to be quite complicated and would like to see what experts like yourselves think about my solution.
Problem:
I am collecting data from a number of third parties into a central mySQL database. The third parties can have any database product (mySQL, Oracle, postgreSQL, etc). I will need to perform two tasks, which are the following:
On first start, replicate the full database onto the central mySQL database
Incrementally update the central mySQL database as changes are done to the third party database
Proposed solution:
I intend on creating database crawlers in C# per third party database. So say for example one of the third parties has an oracle database. I intend on creating a class which will query an oracle database for its definition and data and programmatically create sql queries to be executed by the central mySQL database.
ex: create table XYZ (id1 int, id2 int), Insert into XYZ values(1,2), etc.
Here is the problem which is giving me the greatest grief. Id like to know if anyone has ever queried a database to effectively ask it "what are your latest changes", or what are your changes since dd-mm-yyyy hh:MM:ss. By changes I mean both data changes and data definition changes. ex: add new column, update a [row][column] value, insert new row, etc.
if you have access onto the database writer program, couldn't you just inject a code which generates / writes the data to the central database (aka a LOGGER)?
I mean that you could make your central database as a "LOGGING Database".
If for some reason the third part database needs database rollback, you could just take the command from current timestamp to timestamp-rollback_days and change from INSERT to DELETE, DELETE to INSERT, etc.
That way, your database won't be suffering much because you replicate whole DB :)
We are handling a data aggregation project by having several microsoft sql server databases combining to one mysql database. all mssql database have the same schema.
The requirements are :
each mssql database can be imported to mysql independently
before being able to import each record to mysql we need to validates each records with a specific createrias via php.
each imported mssql database can be rollbacked. It means even it already imported to mysql, all the mssql database can be removed from the mysql.
we would still like to know where does each record imported to the mysql come from what mssql database.
All import process will be done with PHP .
we have difficulty in many aspects. we don't know what is the best approach to solve our problem.
your help will be highly appreciated.
ps: each mssql database has around 60 tables and each table can have a few hundred thousands .
Don't use PHP as a database administration utility. Any time you build a quick PHP script to transfer records directly from one database to another, you're going to cause yourself a world of hurt when that script becomes required for production operation.
You have a number of problems that you need solved:
You have multiple MSSQL databases with similar if not identical tables.
You have a single MySQL database that you want to merge the data into.
The imported data must be altered in a specific way before being merged.
You want to prevent all duplicate records in your import.
You want to know what database each record originally came from.
The solution?
Analyze the source MSSQL databases and create a merge strategy for them.
Create a database structure on the MySQL database that fits the merge strategy in #1, including all the new key constraints (like unique and foreign keys) required for the consolidation.
At this point you have two options left:
Dump the data from each of the source databases into raw data using your RDBMS administration utility of choice. Alter that data to fit your merge strategy and constraints. Document this, and then merge all of the data into your new database structure.
Use a tool like opendbcopy to map columns from one database to another and run a mass import.
Hope this helps.
I got a case where I have several databases running on the same server. There's one database for each client (company1, company2 etc). The structure of each of these databases should be identical with the same tables etc, but the data contained in each db will be different.
What I want to do is keep a master db that will contain no data, but manage the structure of all the other databases, meaning if I add, remove or alter any tables in the master db the changes will also be mirrored out to the other databases.
Example: If a table named Table1 is created in the master DB, the other databases (company1, company2 etc) will also get a table1.
Currently it is done by a script that monitors the database logs for changes made to the master database and running the same queries on each of the other databases. Looked into database replication, but from what I understand this will also bring along the data from the master database, which is not an option in this case.
Can I use some kind of logic against database schemas to do it?
So basicly what I'm asking here is:
How do I make this sync happen in the best possible way? Should I use a script monitoring the logs for changes or some other method?
How do I avoid existing data getting corrupted if a table is altered? (data getting removed if a table is dropped is okay)
Is syncing from a master database considered a good way to do what I wish (having an easy maintainable structure across several datbases)?
How will making updates like this affect the performance of the databases?
Hope my question was clear and that this is not a duplicate of some other thread. If more information and/or a better explantion of my problem is needed, let me know:)
You can get the list of tables for a given schema using:
select TABLE_NAME from information_schema.tables where TABLE_SCHEMA='<master table name>';
Use this list for a script or stored procedure ala:
create database if not exists <name>;
use <name>;
for each ( table_name in list )
create table if not exists <name>.table_name like <master_table>.table_name;
Now that Im thinking about it you might be able to put a trigger on the 'information_schema.tables' db that would call the 'create/maintain' script. Look for inserts and react accordingly.