I want to duplicate a very large table, but I do not want to copy it row by row. Is there a way to duplicate it?
For example, you can TRUNCATE w/o deleting row/row, so i was wondering if there is something similar for copying entire tables
UPDATE: row by row insert is very painful (because of 120M rows). Anyway to avoid that?
MySQL no longer has a reliable "copy table" functionality - many reasons for this related to how data is stored. However, the below does row-by-row insertion but is pretty simple:
CREATE TABLE `new_table` LIKE `old_table`;
INSERT INTO `new_table` (SELECT * FROM `old_table`);
You could use INSERT INTO ... SELECT.
If you're using MyISAM you can copy the physical files on disk. Restart the service and you'll have the new table with indexes and everything the same.
INSERT INTO TABLE2 SELECT * FROM TABLE1
It's nontrivial to copy a large table; ultimately the database is likely to need to rebuild it.
In InnoDB the only way is really to rebuild it, which means insert ... select or such like, however, with 120M rows as it's going to happen in a single transaction, you will probably exceed the size of the rollback area, which will cause the insert to fail.
mysqldump followed by renaming the original table then restoring the dump should work, as mysqldump may cause a commit every lots of rows. However it will be slow.
oracle:
Create table t as select * from original_table
Related
mysql fastest way to delete a database content on freebsd? please help
i've tried to delete from navicat (400,000+ lines),
but in a hour... just 100,000 was deleted
I haven't phpmyadmin
To delete everything in a table:
TRUNCATE TABLE table_you_want_to_nuke
To delete certain rows, you have two options:
Follow these steps:
Create a temporary table using CREATE TABLE the_temp_table LIKE current_table
Drop all of the indexes on the temp table.
Copy the records you want to keep with INSERT INTO the_temp_table SELECT * FROM current_table WHERE ...
TRUNCATE TABLE current_table
INSERT INTO current_table SELECT * FROM the_temp_table
To speed this option up, you may want to drop all indexes from current_table before the final INSERT INTO, then recreate them after the INSERT. MySQL is much faster at indexing existing data than it is at indexing on the fly.
The option you're currently trying: DELETE FROM your_table WHERE whatever_condition. You probably need to break this into chunks using the WHERE condition or LIMIT, so you can batch it and not bog down the server forever.
Which is better/faster depends on lots of things, mostly the ratio of deleted records to retained records and the number of indexes involved. As always, test this carefully before doing it on a live database, as both DELETE and TRUNCATE will permanently destroy data.
I have an InnoDB table in MySQL which used to contain about 600k rows. After deleting 400k+ rows, my guess is that I need to run an OPTIMIZE.
However, since the table will be locked during this operation, the site will not be usable at that time. So, my question is: should I run the optimize on the live database table (with a little under 200k rows)? Or is it possible to create a copy of that table, run the OPTIMIZE on that copy and after that rename both tables so the copy the data back to the live table?
If you create a copy, then it should be optimised already if you do CREATE TABLE..AS SELECT... No need to run it separately
However, I'd consider copy the 200k rows to keep into a new table, then renaming the tables.
This way is less steps and less work all round.
CREATE TABLE MyTableCopy AS
SELECT *
FROM myTable
WHERE (insert Keep condition here);
RENAME TABLE
myTable TO myTable_DeleteMelater,
MyTableCopy TO myTable;
I have a MyISAM table containing ~20 million rows, and need to increase the maximum size of a varchar. No matter how I phrase the ALTER TABLE query, it takes forever as MySQL copies all the data to a temporary table, reindexes it, etc - it seems to think that the data needs to be converted. Since it's a varchar already, I'd have thought it wouldn't need to do any of this.
Is there some way to force MySQL to increase the maximum length of a varchar without going through this horribly slow and painful process? I'm ideally looking for something which can be done entirely using SQL, so no trickery involving copying .frm files around etc.
One alternative would be to
create a copy of the table, but with the increased VARCHAR you need the table to be. BUT NO INDEXES.
Populate the new version of the table:
INSERT INTO new_table
SELECT * FROM old_table
Apply indexes
Update application references to use new_table
Step 4 can be minimized by updating the application to use a view that points to old_table until the new_table is ready. Then you refresh the view, setting it to use new_table...
I have a table with 10 million records, what is the fastest way to delete & retain last 30 days.
I know this can be done in event scheduler, but my worry is if takes too much time, it might lock the table for much time.
It will be great if you can suggest some optimum way.
Thanks.
Offhand, I would:
Rename the table
Create an empty table with the same name as your
original table
Grab the last 30 days from your "temp" table and insert
them back into the new table
Drop the temp table
This will enable you to keep the table live through (almost) the entire process and get the past 30 days worth of data at your leisure.
You could try partition tables.
PARTITION BY LIST (TO_DAYS( date_field ))
This would give you 1 partition per day, and when you need to prune data you just:
ALTER TABLE tbl_name DROP PARTITION p#
http://dev.mysql.com/doc/refman/5.1/en/partitioning.html
Not that it helps you with your current problem, but if this is a regular occurance, you might want to look into a merge table: just add tables for different periods in time, and remove them from the merge table definition when no longer needed. Another option is partitioning, in which it is equally trivial to drop a (oldest) partition.
To expand on Michael Todd's answer.
If you have the space,
Create a blank staging table similar to the table you want to reduce in size
Fill the staging table with only the records you want to have in your destination table
Do a double rename like the following
Assuming:
table is the table name of the table you want to purge a large amount of data from
newtable is the staging table name
no other tables are called temptable
rename table table to temptable, newtable to table;
drop temptable;
This will be done in a single transaction, which will require an instantaneous schema lock. Most high concurrency applications won't notice the change.
Alternatively, if you don't have the space, and you have a long window to purge this data, you can use dynamic sql to insert the primary keys into a temp table, and join the temp table in a delete statement. When you insert into the temp table, be aware of what max_packet_size is. Most installations of MySQL use 16MB (16777216 bytes). Your insert command for the temp table should be under max_packet_size. This will not lock the table. You'll want to run optimize table to reclaim space for the rest of the engine to use. You probably won't be able to reclaim disk space, unless you were to shutdown the engine and move the data files.
Shutdown your resource,
SELECT .. INTO OUTFILE, parse output, delete table, LOAD DATA LOCAL INFILE optimized_db.txt - more cheaper to re-create, than to UPDATE.
I have a database with static tables which require to be updated from CSV weekly.
Tables are Mysql MyISAM and by static i mean they are used for read only (except when updated from CVS, obviously).
There're about 50 tables and in total about 200mb of data to be reloaded weekly.
I can think about 3 ways:
Truncate table
Load data from files
Or
For each table create a temporary table
Load data there
Truncate (or delete rows?) original table
Insert into original table select * from temporary table.
Or
Create table_new and load data there
Rename original table to table_old (or drop table altogether)
Rename table_new into original table
What do you reckon is the most efficient way?
Have you considered using mysqlimport? You can read about it here:
http://dev.mysql.com/doc/refman/5.1/en/mysqlimport.html
I probably wouldn't do anything with deleting the original tables, because then you have to re-create all your foreign keys, indexes, constraints, etc. which is a mess and a maintenance nightmare. Renaming tables can also cause problems (like if you have synonyms for the tables, I'm not sure if mysql has synonyms though).
What I would do, however, is disable the keys before loading the data.
ALTER TABLE tbl_name DISABLE KEYS
In other words, when loading the data you don't want it to be trying to update indexes because that will slow down the load. You want the indexes updated once the load is completed.
So I think by combining mysqlimport with the tip above, you should be able to get a really efficient load.
You could always do INSERT INTO ... ON DUPLICATE KEY UPDATE ... or REPLACE INTO .... You shouldn't get any down time (between a TRUNCATE and INSERT), and there's very little chance of corruption.
Be careful with REPLACE, since it will actually delete each record and re-insert it, firing any triggers you may have (unlikely in this case), but also giving you a new ID if you have an auto-increment field.
Your third option is the best, you can LOCK and DISABLE KEYS on the _new table while importing, and it'll be extra quick. You can even do a "batch atomic rename" of all your new tables to the "current ones", with zero downtime if they have relations between them.
I'm assuming the whole tables are contained in the weekly cvs updates (i.e. they're not incremental).
I would prefer the 3rd method and also keep the old table.
create table_new
drop table_old if exists
rename table to table_old
rename table_new to table
The advantage of this method is that it fast and safe with less effect on the readers. The creation of new table does not affect reads on existing table. The rename operation is faster (just a file rename in case of myisam) so the downtime is not that much. So the clients will not be affected by this that much. You also got to keep the old data in case something is wrong with the new data.
As you are not going to update it online I think it will be good if you do myisampack.