I'm trying to export a fairly complex mySQL database from the working local server to upload on to an online server for collaborators to READ ONLY the data within.
The database has a number of Foreign keys, and every table has a primary key. However, since NO DATA WILL BE ADDED to this dumb "shadow" copy, these are irrelevant, and frankly creating a headache trying to get them to import successfully.
So... IS there a way to export a MySQL databases' structure (and possibly data) withOUT any keys, keeping the autoincrement column, but just treating it like any other INT column, and removing the foreign key constraints?
You can export a MySQL database without AUTO_INCREMENT options this way:
mysqldump --compatible no_field_options ...
I don't know of any way to omit the foreign key constraints. But if you use mysqldump, the export does SET FOREIGN_KEY_CHECKS=0; before creating any tables. This allows tables to be created out of order.
Re your comment:
If you really can't allow foreign key declarations in your table definitions, you'll have to edit them out of the MySQL export file. You can do this manually with any text editor, or else come up with a filter using sed or perl or a variety of other tools.
There's an example of a sed command in one of the answers to How do I dump MySQL file without Foreign Keys via command line?
It might be easier to drop the constraints in your tables before you export the database.
Let's say I have an old .SQL dump and since it was created, I have changed the table schema.
I could be running:
INSERT INTO `ec_product_campaign_relations` (`campaign_id`, `product_id`, `product_qty`) VALUES (30,28,1),(30,27,0),(30,31,0),(30,30,0);
But if column product_qty does no longer exist, the line will not get inserted.
How can I force the line to get inserted anyways and ignore that the column does not exist?
EDIT: It should mention I'm working in PHP and it is script used to sync table shema... So no "manual" control over this.
Since editing all your SQL dump won't be trivial, I suggest you to add the column to your table, make the import, then delete the column.
You might want to create a new database for this import and restore the dump as-is. Then, once you've got a handle on what changes have been made by comparing the schema in one to the new one, create a series of ALTER TABLE statements that bring it in sync.
I tend to record these in a text file in case I need to replay them later, and also keep them as a list of what's changed. You may have to do this more than once, so notes help.
Then, once you've cleaned them up to be column-compatible, dump this database table-by-table, and restore into the other as required.
I was having a "coming soon" page with a sign up form. Since a couple of weeks I've switched to another web hosting and continue working on my new site at my new host, but continued to let the visitors sign up at my old host.
Now my have pointed my domain to my new hosting and want to export all new subscribers and import them into the new DB. Some rows might use same ID.
How can I easily export and merge a table from the old DB to the new one? Please note I don't wan't it to overwrite any row that has the same ID, if it does, I want it to add a new row instead.
I've tried exporting my table "wp_csp3_subscribers" from my old DB and imported it in my new one. But get an error saying ID already exists.
If you're using phpMyAdming (based on your tags), you can expand the export options by selecting "Custom - display all possible options" instead of the default "Quick - display only the minimal options".
Under the data dump options, choose "INSERT IGNORE statements", which will tell mySQL to ignore any errors with duplicate primary keys.
Taken from the mySQL documentation on the IGNORE option:
For example, without IGNORE, a row that duplicates an existing UNIQUE index or PRIMARY KEY value in the table causes a duplicate-key error and the statement is aborted. With IGNORE, the row still is not inserted, but no error is issued.
Hope this answers your question.
also, if you have two sql files from a dump, insert one normally, maybe with Enable foreign key checks unchecked, then your second file would like to concatenate the data, remove the DROP TABLE IF EXISTS and CREATE TABLE lines in that second file. Leave only the LOCK TABLES table WRITE; and inserts
I know I can dump a single table using the "where clause" but I was wondering if its possible to dump a table and have all the linking records be dumped along with them if they belong to a certain account id?
All my tables are innodb and have been set up using foreign key constraints with cascade delete. If I delete the main table "account" where account_id = 1 then all the records that link to account_id of "1" will also be deleted.
So what I want is something similar in concept. I want to dump all the data for "Account_id=1" in all the tables that link to the "account" table in one command. If I do the following command I believe it will only dump the one table:
mysqldump -t -u [username] -p test account --where="account_id = 1"
Is there another way to dump on table with a where clause and automatically dump the data in liking tables without having to write separate dump commands for each table? ultimately I want to end up with a .sql file for each account like "account_1.sql", account_2.sql, etc.
I had put this question in my favorite list to see if someone comes with an idea, and as I was expecting no one did.
One rather funny way is to clone the DB, delete all not-required account ids (delete will cascade to all tables) and then dump the remaining (which will be all the account ids you require).
I was running through the same issue with MySQL, and DBIx::Class (an ORM in Perl). What I wanted to do was to clone a thousand of accounts (with obfuscated names and emails). I ended up writing a script to traverse the whole database through the foreign keys of a given user id and generate all the required insert statements in proper order.
I have one table spread across two servers running MySql 4. I need to merge these into one server for our test environment.
These tables literally have millions of records each, and the reason they are on two servers is because of how huge they are. Any altering and paging of the tables will give us too huge of a performance hit.
Because they are on a production environment, it is impossible for me to alter them in any way on their existing servers.
The issue is the primary key is a unique auto incrementing field, so there are intersections.
I've been trying to figure out how to use the mysqldump command to ignore certain fields, but the --disable-keys merely alters the table, instead of getting rid of the keys completely.
At this point it's looking like I'm going to need to modify the database structure to utilize a checksum or hash for the primary key as a combination of the two unique fields that actually should be unique... I really don't want to do this.
Help!
To solve this problem, I looked up this question, found #pumpkinthehead's answer, and realized that all we need to do is find+replace the primary key in each row with the NULL so that mysql will use the default auto_increment value instead.
(your complete mysqldump command) | sed -e "s/([0-9]*,/(NULL,/gi" > my_dump_with_no_primary_keys.sql
Original output:
INSERT INTO `core_config_data` VALUES
(2735,'default',0,'productupdates/configuration/sender_email_identity','general'),
(2736,'default',0,'productupdates/configuration/unsubscribe','1'),
Transformed Output:
INSERT INTO `core_config_data` VALUES
(NULL,'default',0,'productupdates/configuration/sender_email_identity','general'),
(NULL,'default',0,'productupdates/configuration/unsubscribe','1'),
Note: This is still a hack; For example, it will fail if your auto-increment column is not the first column, but solves my problem 99% of the time.
if you don't care what the value of the auto_increment column will be, then just load the first file, rename the table, then recreate the table and load the second file. finally, use
INSERT newly_created_table_name (all, columns, except, the, auto_increment, column)
SELECT all, columns, except, the, auto_increment, column
FROM renamed_table_name
You can create a view of the table without the primary key column, then run mysqldump on that view.
So if your table "users" has the columns: id, name, email
> CREATE VIEW myView AS
SELECT name, email FROM users
Edit: ah I see, I'm not sure if there's any other way then.
Clone Your table
Drop the column in clone table
Dump the clone table without the structure (but with -c option to get complete inserts)
Import where You want
This is a total pain. I get around this issue by running something like
sed -e "s/([0-9]*,/(/gi" export.sql > expor2.sql
on the dump to get rid of the primary keys and then
sed -e "s/VALUES/(col1,col2,...etc.) VALUES/gi" LinxImport2.sql > LinxImport3.sql
for all of the columns except for the primary key. Of course, you'll have to be careful that ([0-9]*, doesn't replace anything that you actually want.
Hope that helps someone.
SELECT null as fake_pk, `col_2`, `col_3`, `col_4` INTO OUTFILE 'your_file'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM your_table;
LOAD DATA INFILE 'your_file' INTO TABLE your_table
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n';
For added fanciness, you can set a before insert trigger on your receiving table that sets the new primary key for reach row before the insertion occurs, thereby using regular dumps and still clearing your pk. Not tested, but feeling pretty confident about it.
Use a dummy temporary primary key:
Use mysqldump normally --opts -c. For example, your primary key is 'id'.
Edit the output files and add a row "dummy_id" to the structure of your table with the same type as 'id' (but not primary key of course). Then modify the INSERT statement and replace 'id' by 'dummy_id'. Once imported, drop the column 'dummy_id'.
jimyi was on the right track.
This is one of the reasons why autoincrement keys are a PITA. One solution is not to delete data but add to it.
CREATE VIEW myView AS
SELECT id*10+$x, name, email FROM users
(where $x is a single digit uniquely identifying the original database) either creating the view on the source database (which you hint may not be possible) or use an extract routine like that described by Autocracy or load the data into staging tables on the test box.
Alternatively, don't create the table on the test system - instead put in separate tables for the src data then create a view which fetches from them both:
CREATE VIEW users AS
(SELECT * FROM users_on_a) UNION (SELECT * FROM users_on_b)
C.
The solution I've been using is to just do a regular SQL export of the data I'm exporting, then removing the primary key from the insert statements using a RegEx find&replace editor. Personally I use Sublime Text, but I'm sure TextMate, Notepad++ etc. can do the same.
Then I just run the query in which ever database the data should be inserted to by copy pasting the query into HeidiSQL's query window or PHPMyAdmin. If there's a LOT of data I save the insert query to an SQL file and use file import instead. Copy & paste with huge amounts of text often makes Chrome freeze.
This might sound like a lot of work, but I rarely use more than a couple of minutes between the export and the import. Probably a lot less than I would use on the accepted solution. I've used this solution method on several hundred thousand rows without issue, but I think it would get problematic when you reach the millions.
I like the temporary table route.
create temporary table my_table_copy
select * from my_table;
alter table my_table_copy drop id;
// Use your favorite dumping method for the temporary table
Like the others, this isn't a one-size-fits-all solution (especially given OP's millions of rows) but even at 10^6 rows it takes several seconds to run but works.