I am having my application deployed on two separate regions say US-WEST and EU, both application has its own DB. And now I want to move the EU region DB to US-WEST.
This will lead to primary key collision since both the db has the tables with same primary auto increment id, can anybody give me suggestion to solve this.
Scenario:
User Table from DB1(say from US-WEST) has the following entries
ID Name
1 Rob
2 San
3 Tulip
User Table from DB2(say from EU) has the following entries
ID Name
1 John
2 Michael
3 Natasha
For every one of the two original databases (say db0 and db1):
Back up the db.
Lock database for use by this script only.
For all the tables in the database that have foreign keys defined without ON UPDATE CASCADE, change all these foreign keys constraints with this option.
For every table with an auto_increment (or a simple integer) Primary Key, run this (the cascading updates will make the rest):
.
UPDATE TableX
SET Pk = 2 * Pk - 0 --- for db0
ORDER BY Pk DESC
UPDATE TableX
SET Pk = 2 * Pk - 1 --- for db1
ORDER BY Pk DESC
Export the tables from each database.
Now merge the two databases by simply merging the corresponding tables. All data from db0 will have even ids and all from db1 will have odd ids. No collisions.
For tables without auto-incrementing Primary Keys or for tables which may have common rows, the merging should be different, off course.
Unlock.
You can read about auto_increment_increment and related system variables that you can change so from this point, the two databases produce different auto incremented ids (one odd ids, the other even ones).
Turn off auto-increment in your destination DB. Then first import data from DB1 and the from DB2. In your importing from DB2 add a constant value that is higher than your hightest id in the first DB. Like this:
insert into destination_table
select id + 10000, othercolumns from source_table
After importing the data you can turn on auto-increment again.
EDIT :
If your id column references to other tables then this method will break the relation to these tables.
I think you have to extend your destination DB with a column for example regionID and edit the primary key settings for this table. Use a Primary key with the two columns ID and regionID. Then import the data from the two tables like this:
Insert into destination_table values(regionID, ID, Name)
select 1,ID, Name from DB1
Insert into destination_table values(regionID, ID, Name)
select 2,ID, Name from DB2
Now, the tricky part. You have to do this for all tables, where you use the ID as a relation. After transferring all data you only have to edit your SQL statements to use regionID and ID combined as key.
Remove primery key and Turn off auto-increment from id field your destination DB table.
Then first import data from from both DB.
Delete id column from destination table.
Create again id column make that column auto increament primary key.
Related
I'm working with phpmyadmin and I have to merge two db with same structure but different data.
The db have relation between tables (foreign key).
The data in two db may have same id, and so their foreign key.
I would like to know if it's possible merge the two db keeping all data, so, if a row already "exist", insert it with new id and update its foreign key.
thanks a lot
No easy way unfortunately. If you have TableA as a foreign key to TableB, you will need to
1) Insert data from source tableA to target tableA
2) create a (temp) table to store the mapping between source tableA ids and target tableA ids
3) Use this mapping table when inserting data from tableB to convert the tableA ids to the new ones in the target db
... and so on. It can get quite hairy if you have a deep hierarchy of tables, but hopefully you get the idea. Take backups before you start.
Another idea that you might want to consider is using a cursor:
Assume table A is the one that you want to keep and table B is the one you want to remove.
Declare a cursor for table B and select all the records.
Loop each record selected from the cursor and check.
Case 1: If the ID is exists on table A, insert the record to table A with same details.
Case 2: If the ID is exists on table B, insert the record and modify the ID and foreign key.
Once all the records have been checked, drop table B.
Sorry, I just can give an idea at the moment.
Two tables share a unique identifier 'id'. Both tables are meant to be joined by using 'id'.
Defining 'id' as an auto incrementing primary key in both tables may risk update inconsistencies.
Is there some general pattern to avoid such a situation or do I have to deal with updating table1 first and table2 by utilizing the last inserted id after (therefore not declaring id as auto inc in table2)?
First, if you use InnoDB table engine in MySQL you could use both transactions and foreign keys for data consistency.
Second, after the insert in the first table, you could get the last insert id (depending on the way you access the db) and use it as foreign key.
Eg
Table 1: Users: user_id, username
Table 2: User_Profiles: user_id, name, phone
In User_Profiles you don't need to define user_id as auto increment, but first insert a record in Users table and use the user_id for the User_Profiles record. If you do this in transaction, the Users record won't be seen outside of the transaction connection until it's completed, this way you guarantee that even if something bad happens after you insert the user, but before you have inserted the profile - there won't be messed up data.
You could also define that the user_id column in User_Profiles table is foreign key of Users table thus if someone deletes a record from the Users table, the database would automatically delete the one in User_Profiles. There are many other options - read more about that.
There is no problem with same column name 'id' in any number of tables.
Several persistence layer frameworks do it same way.
Just use aliases in your SQL to distinct your tables accordingly.
do I have to deal with updating table1 first and table2 by utilizing the last inserted id after (therefore not declaring id as auto inc in table2)?
Yes. And make id a foreign key so it can only exist in table2 if it already exists in table1.
Yes you do, and remember to wrap the operation in a transaction.
I need to create an append query, that appends many records to a table. this table has a primary key, that is a sequential number. How do I make my append query, append records to the table and automatically assign the next sequential number for the primary key? I woudl need to run this query on a live multi-user MYSQL server throughout the day
thanks!
If the PK is a true auto-incremental field, you should be able to leave the PK out of your 'append' query. The table will automatically assign the next value in sequence to your data row(s) that you are inserting.
ex: If you have this data in table names
id name
1 Ken
2 Jon
3 Steve
And you run this query
INSERT INTO names (name) VALUES ('Peter')
Your table should automatically assign id # 4 to Peter
If the sequential PK is maintained manually, I would suggest you alter that field to be a true auto-incremental field if at all possible, or create a new auto-increment field and drop the old one. Just make sure you update any other related tables before you drop the field.
If I have a User table (with a global id as primary key) that is sharded across 10 databases (DB1-DB10) based on the username, and another table tries to refer to the User table using the User table row's global id, there is no way for me to know which DB (1-10) that user is located in.
What is the solution to this problem?
Either:
Change your sharding scheme so that you shard based on the key you use to look up this data. If it's by ID, always do it by ID, and shard it by ID (DB = ID % 10)
Make the primary key of the User table the username. Enforce its uniqueness and use the username as the foreign key in the other tables, not a synthetic identifier.
Create a lookup table you can reference which maps ids to shards.
We have two instances of a rails application which each talk to their own database; we're in the process of converting them to a single application with a single database. We've already made the parts of it which need to be specific to a particular domain work correctly; now we just need to merge the databases. We're going to copy the data from one instance into the other's database, and fix up the IDs so they don't overlap. There are a lot of tables with a lot of foreign keys. What's a good way of doing this, such that the foreign keys still point to the correct row in the new database?
If this is unclear, I'm happy to complicate matters with bad ascii art.
Most relational databases let you annotate a foreign key to be constrained to watch for when the primary key in the pointed-to table changes. You can set the foreign key to be "auto-updated" when this happens using ON UPDATE CASCADE. Do this for all foreign keys in both databases, then update all primary keys in both databases, and all foreign keys will be automatically converted.
How about updating every id column (including the foreign keys) to be its original value times 10, then add 1 for the first database and 2 for the second database.
That way id 1 becomes 11 on db 1 and 12 on db 2. Since both the primary and foreign keys go through the same change, you don't have to worry about how the records relate, you just make the updates with the same formula.
So it would go something like
On db 1:
UPDATE user SET id = id * 10 + 1;
UPDATE privilege SET id = id * 10 + 1, user_id = user_id * 10 + 1;
On db 2:
UPDATE user SET id = id * 10 + 2;
UPDATE privilege SET id = id * 10 + 2, user_id = user_id * 10 + 2;