I'm trying to replicate data between 2 MySQL databases. The issue is only some rows needs to be transferred to the second MySQL server based on a specific criteria.
I have 2 MySQL servers. The first one is intranet only, there is an application that reads/writes to it. The second MySQL server is online and the application connecting to it is read only.
I need to find a way to get the data from the first server to the second based on specific criteria (some rows are labeled as private and should not synchronized). I tried to do it with a trigger on the first server (trigger on insert/update) but I have way too many tables, it's very time consuming to do it like that.
What approaches do I have? dumping the entire data is not an option as there will a lot of records and the online server cannot afford to go offline just to get the information. Add to that that not all the records are for public usage.
1 - disable replication
2 - on the intranet, create an empty database and a view based on a query that shows exactly the rows you want to replicate to your internet server
3 - replicate the new database (the one containing the view) to a new database on your internet server
4 - on your internet server, you can cron a script that inserts the new rows to your desired table table, think about using dumps and LOAD DATA IN FILE, it should go verry quickly.
Related
My requirement is to load data on daily basis from source table from one server to destination table in another server. Both servers are different i.e one is sql server and another one is Oracle server.
what i want is to make the source query fast. Suppose today I Execute the package I should get only the new records instead of all records from source. If I read the whole table it takes much time. Even I am using Lookup transforming to check the record exists or not it is taking much time.
Please look into this
I am trying to setup a MySQL database that takes data from 3 other MySQL databases. The data that would be copied would be a query that standardizes the data format. The method would need to either be run daily as a script or synced in real time, either method would be fine for this project.
For example:
The query from source DB:
SELECT order_id, rate, quantity
WHERE date_order_placed = CUR_DATE()
FROM orders
Then I want to take the results of that query to be inserted into a destination DB.
The databases are on separate hosts.
I have tried creating scripts that run CSV and SQL exports/imports without success. I have also tried using Python pymysql library but seemed overkill. I'm pretty lost haha.
Thanks :)
Plan A:
Connect to source. SELECT ... INTO OUTFILE.
Connect to destination. LOAD DATA INFILE from the output above.
Plan B (both MySQL):
Set up Replication from the source (as a Master) and the destination (as a Slave)
Plan C (3 MySQL servers):
Multi-source replication to allow gathering data from two sources into a single, combined, destination.
I think MariaDB 10.0 is when they introduced multi-source repl. Caution: MariaDB's GTIDs are different than MySQL's. But I think there is a way to make the replication you seek to work. (It may be as simple as turning off GTIDs??)
Plan D (as mentioned):
Some ETL software.
Please ponder which Plan you would like to pursue, then ask for help in focusing on one. Meanwhile, your question is too broad.
I recently switch to a new hosting provider for my application. My employees used the old site until the new site went live, however, the database backup from the old site was taken two days before the new site went live. So in the midst of transferring, records were being entered into the old site database while the new site had no existence of them (hence my two day time lag). How do I merge the two databases to reflect the changes?
A couple of things to note are the primary keys might be duplicated for some tables and there are only timestamps on a few tables as well. I would do a 'diff' or something of the sort, but the tables are dumped in different formats.
Any thoughts?
This is something where you'll need to actually understand your database schema. You'll need to create a program that can look at both versions of the database, identify which records are shared, which are not, and which have conflicting primary keys (vs ones which were updated with the same keys). It then needs to copy over changes, possibly replacing the value of primary keys (including the values in other rows that refer to the row being renumbered!) This isn't easy, and it's not an exact science - you'll be writing heuristics, and expect to do some manual repairs as well.
Next time shut that database down when you grab the final backup :)
You don't need to create any additional programs. All what you need, to setup replications from the old DB to the new one.
All your data from the old DB will automatically transfer to the new DB. At this period you should use you old DB as the main data source. And as soon as all data will be copied to the new location, you'll need just brake replica connection and change the DB address in your code (or DNS pointer) to the new one.
1. oldDB ===> replication ==> newDB
R/W operations
2. oldDB ==/= brake ==/= newDB
R/W operations
MySQL Doc: 15.1.1. How to Set Up Replication
I've got a large-ish MySQL database which contains weather information. The DB is about 3.5 million rows, and 486 MB in size.
I get a refresh of this data every 6 hours, in the form of a mysql dump file that I can import. It takes more than 2 minutes to import the data and probably a similar amount of time to create the index.
Any thoughts on how I can import this data while still keeping the DB available and not losing responsiveness? My first thought was two have two databases within the same MySQL instance. I'd be running off DB1 and would load data into DB2 and then switch. However, I'm concerned that the load process would make DB1 unresponsive (or significantly slow).
So, my next thought is two have two different MySQL instances running, on different ports. While DB instance 1 is serving queries, DB instance 2 can be loaded with the next dataset. Then on the next query, the code switches to DB2.
That seems like it would work to me, but I wanted to check with some who have tried similar things in the past to see if there were any "gotchas" I was missing.
Thoughts?
Have two databases and switch between them after the import finishes each time.
Load on one database shouldn't make the other database unresponsive. 486MB is not too big for it all to fit in memory a couple of times over - depending I guess on whether you're in a small virtual server.
But even so, two MySQL instances on one server shouldn't present any differences in performance than two databases on one instance, except that two instances may actually take more memory and be more complicated to set up.
I have an SSIS package that connects to a mysql server and attempts to pulls data from different tables and inserts the rows into a SQL Server 2005 database.
One issue i notice is that at any given time it runs, regardless of what step it is on, it almost always fails to bring in the total records from mysql into sql server.
there are no errors thrown.
One morning it will have all 11M records and on the next run anywhere between 3K and 17K records.
Anyone notice anything like this?
I import data from two separate MySQL databases -- one over the Internet and one in-house. I've never had this type of issue. Can you describe how you setup your connection to the MySQL database? I used the ODBC driver available on the MySQL website and connect using an ADO.NET data source in my data flow that references the ODBC connection.
One possible way you could at least prevent yourself from loading incomplete data is only load new records. If the source table has an ID and the records never change once they are inserted, then you could feed in the maximum ID by checking your database first.
Another possible way to prevent loading incomplete data is loading the MySQL database into a staging table on your destination server and then only load records you haven't already loaded.
Yet another way to do it is load the data into a staging table, verify the records are greater than some minimum threshold such as the row count of the target table or the expected minimum number of transactions per day and then only commit the changes after this validation. If the rows are insufficent, then raise an error on the package and send a notification email. The advantage of raising an error is you can set your SQL Server Agent job to retry the step for a defined number of attempts to see if this resolves the issue.
Hope these tips help even if they don't directly address the root cause of your problem.
I've only tried MySQL -> SQL Server via SSIS once, but the error I found related to MySQL datetimes not converting to SQL Server datetimes. I would have thought this would break the whole dataflow, but depending on your configuration you could have set it purely to ignore bad rows?