I'm doing an update to MySQL Database which includes MySQL scripts that make ALTER TABLE sentences, as well as DIU sentences (delete, insert, update).
The idea is to make a transactional update, so if a sentence fails, a rollback is made, but if I put ALTER TABLE sentences or others specified in http://dev.mysql.com/doc/refman/5.0/en/implicit-commit.html an implicit commit is made, so I can't make a complete rollback, because the indicated operations remains commited.
I tried to use mysqldump to make a backup which is used in case of error (mysql returns distinct to zero), but it is too slow and can fail too.
What can I do? I need this to ensure that future updates are safe and not too slow, because databases contains between 30-100 GB of data.
dump and reload might be your best options instead of alter table.
From mysql prompt or from the database script:
select * from mydb.myt INTO OUTFILE '/var/lib/mysql/mydb.myt.out';
drop table mydb.myt;
create tablemyt(your table ddl here)
load data infile '/var/lib/mysql/mydb.myt.out' INTO TABLE mydb.myt;
Check this out:
http://everythingmysql.ning.com/profiles/blogs/whats-faster-than-alter
I think it offers good guidance on "alternatives to alter".
Look at pt-online-schema change.
You can configure it to leave the 'old' table around after the online ALTER is completed. The old table will have an underscore prefix. If bad things happen, drop the tables you altered and renamed the OLD tables to the original tables. If everything is OK, then just drop the OLD tables.
http://www.percona.com/doc/percona-toolkit/2.1/pt-online-schema-change.html
Related
I'm using a transaction to make alterations to my database, this way if anything fails during the alterations i can roll back without any harm having been done.
However, since i run my queries based on a list of queries that puts my database in it's final state (any time i need to make changes, i simply add a new rule to the list), it fails when i try to drop a column that was added during the same transaction.
Example:
START TRANSACTION;
ALTER TABLE "servers" ADD COLUMN "test" INTEGER NOT NULL;
ALTER TABLE "servers" DROP COLUMN "test";
COMMIT;
When I run this, i get something along the lines of
column "test" in the middle of being added, try again later
I understand why this is happening, since the transaction hasn't been committed yet the column doesn't exist to be dropped. However is there a way around this specifically so that I can drop the column in the same transaction? Or at least queue it to be deleted once the transaction is committed.
I feel it worth noting that the queries being run within the transaction are generated using Eloquent ORM Blueprints.
MySQL doesn't support table changes in transactions. From the documentation:
Some statements cannot be rolled back. In general, these include data definition language (DDL) statements, such as those that create or drop databases, those that create, drop, or alter tables or stored routines.
I'm trying to drop a table containing several hundred thousand column-based records. Normally when creating the database I use a column-based engine (infinidb) but in this case I forgot to include the ENGINE statement. So the database is pretty much unusable for my needs. Now I have a database full of tables that are taking forever to drop (it's been two hours and nothing has happened). I tried the ALTER TABLE table ENGINE=INFINIDB command but again, it's taking forever (see above re: two hours). EDIT: The first command I tried was DROP TABLE. It hung with every single table. Then I tried the ALTER command in case that was faster for some reason, but it wasn't.
Is there another way to get rid of this database? E.g. manually going into the /mysql/ directory and deleting the database? I guess I could just rename it and leave it, but I'd rather get rid of it entirely so it's not taking up space.
First of all you said Can't drop table. But in post you mentioned ALTER TABLE table ENGINE=INFINIDB.
But DROP != ALTER it is two different things.
So you can do following:
CREATE new table with same structure but engine you need.
copy(UPDATE) data from old table to the one you just created.
DROP old table.
RENAMErename new one to old name
It turned out that another process (a website) was using this database and had a couple of queries that got 'stuck' in the SQL server and caused the table to hang due to the database using the wrong engine, which I'm assuming was InnoDB since I didn't specify an engine when I initially used the "CREATE TABLE table1 AS SELECT * FROM table2" command. We finally managed to wipe the database and start over. Thanks for your help.
I have a mysql table y in database xxx which I attempted to change compression type before using
alter table y row_format=compressed key_block_size=8
the process stopped half way. I removed temp file '#sql-ib265.frm and #sql-ib265' in mysql lib directory and restarted the server. However
Now when I attempt the alter table y (with the same command above) again I get error.
ERROR 1050 (42S01) at line 1: Table 'xxx/#sql-ib265' already exists
I can't drop table 'xxx/#sql-ib265' because it can't be found.
what should I do?
Edit
Solution:
I ended up dropping the old database and recreate the database.
Try to restart mysql client with the --skip-auto-rehash option and try DROP TABLE again.
If above does not work, try this from MySQL Manual:
You have a corrupt innodb data dictionary..
https://dev.mysql.com/doc/refman/5.0/en/innodb-troubleshooting-datadict.html
Problem with Temporary Table
If MySQL crashes in the middle of an ALTER TABLE operation, you may end up with an orphaned temporary table inside the InnoDB tablespace. Using the Table Monitor, you can see listed a table with a name that begins with #sql-. You can perform SQL statements on tables whose name contains the character “#” if you enclose the name within backticks. Thus, you can drop such an orphaned table like any other orphaned table using the method described earlier. To copy or rename a file in the Unix shell, you need to put the file name in double quotation marks if the file name contains “#”.
There are two ways to fix this problem.
As other answer suggests, official MySQL documentation suggests to drop a specially crafted table. But please note in versions >= 5.1 you need to prefix table name with #mysql50#.
Move (use RENAME TO) all good tables to a temporary database, drop&recreate the original one and then move the tables back. See a blog post for details.
in additional I'm loging in with root to do the recover job but failed. then i chown the .frm file to meet the owner of mysql service and succeed.
For anyone still facing this problem, I have just followed the following steps to solve it, which (to me at least) seem far less daunting than other solutions:
Use mysqldump to back up the database with all its data.
Drop and recreate the database.
Reload the database and all its schema from the file generated in (1).
Because the orphaned tables are hidden anyway, they don't get backed up, so you end up with a database without them. I had all my procedures/functions scripted out anyway, so was able to restore them easily - if you don't, make sure you use the --routines parameter to dump those too.
My dump file was around 1.5GB for the database in question (so it's not small), and the whole thing was completed in a few minutes.
I had the same error. I fixed it by switching the order in which I dropped the tables at the beginning of the file:
DROP TABLE IF EXISTS table_name;
This line is repeated for each table. Tables with foreign keys need to be deleted before the tables with the primary keys to which they point.
I have recently installed a new computer with Percona Server 5.6 instead of MySQL 5.6, and using InnoDB/XtraDB mostly, FWIW. The database I'm working on is merely a testing ground, but I have 1 issue: after I add a column to a table (or even remove one), I usually forget to INSERT or otherwise change another table's data, which keeps track of what column names are in which table; each table has ASCII name along with a number, and this number is the only difference between table names for simplicity. So, is there a way to auto-update the "relation" table so that the column name and table's number are added or changed, instead of using a cronjob ?
Now that I think, I could DROP that table and use information_schema instead ...
EDIT 0: Don't let the above realization stop you; it's just good to know if this is possible before going for a possible other way.
Yes, relying on the 'INFORMATION_SCHEMA.COLUMNS' may be best.
Unfortunately mysql does not support DDL TRIGGER events, as this would be what you are looking for.
triggers allow you to perform many SQL and procedural operations before insertion, update or deletion of rows in a specific table. However to the best of my knowledge - and I would be stoked if I were wrong - you cant set TRIGGER events on DDL statements like ALTER and DROP TABLE...
However still take the time to learn about triggers - they save a lot of time by eliminating the need for cronjobs and external updates for things like aggregate values.
https://dev.mysql.com/doc/refman/5.6/en/trigger-syntax.html
I have a MySQL database that is up to about 17 GB in size and has 38 million entries. At the moment, I need to both increase the size of one column (varchar 40 to varchar 80) and add more columns.
Many of the fields are indexed including the one that I need to change. It is part of a unique pair that is necessary for the applications to work. In attempting to just make the change yesterday, the query ran for almost four hours without finishing, when I decided to cut our outage and just bring the service back up.
What is the most efficient way to make changes to something of this size?
Many of these entries are also old and if there is a good way to sort of shard off entries but still have them available that might help with this problem by making the table a much more manageable size.
You have some choices.
In any case you should take a backup before you do this stuff.
One possibility is to take your service offline and do it in place, as you have tried. If you do that you should disable key checks and constraints.
ALTER TABLE bigtable DISABLE KEYS;
SET FOREIGN_KEY_CHECKS=0;
ALTER TABLE (whatever);
ALTER TABLE (whatever else);
...
SET FOREIGN_KEY_CHECKS=1;
ALTER TABLE bigtable ENABLE KEYS;
This will allow the ALTER TABLE operation to go faster. It will regenerate the indexes all at once when you do ENABLE KEYS.
Another possibility is to create a new table with the new schema you want, then disable the keys on the new table, then do as #Bader suggested and insert the contents of the old table.
After your new table is built you will re-enable the keys on it, then rename the old table to some name like "old_bigtable" then rename the new table to "bigtable".
It's possible that you can keep your service online while you're populating the new table. But that might work poorly.
A third possibility is to dump your giant table (to a flat file) and then load it to a new table with the new layout. That is pretty much like the second possibility except that you get a table backup for free. You can make this go pretty fast with SELECT DATA INTO OUTFILE and LOAD DATA INFILE. You'll need to have access to your server machine's file system to do this.
In all cases, disable, then re-enable, the constraints and keys to get things to go fast.
Create a new table with the new structure you want with a different name for example NewTable.
Then insert data into this new table from the old table using the following query:
INSERT INTO NewTable (field1, field2, etc...) SELECT field1, field2, ... FROM OldTable
After this is done, you can drop the old table and rename the new table to the original name
DROP TABLE `OldTable`;
RENAME TABLE `NewTable` TO `OldTable` ;
I have tried this approach on a very large table and it's much much faster than altering the table.
With MySQL 5.1 and again with 5.5 certain alter statements were enhanced to just modify the structure without rewriting the entire table ( http://dev.mysql.com/doc/refman/5.5/en/alter-table.html - search for in-place). The availability of this though varies by the type of change you are making and the engine in use, the most value comes from InnoDB Plugin. In the case of your specific changes though the entire table would be rewritten.
When we encounter these issues, we typically try to leverage replica databases. As long as you are adding and not removing you can run your DDL against the replica first and then schedule a brief outage for promoting the replica to the master role. If you happen to be on RDS this is even one of their suggested uses for their replica instances http://aws.amazon.com/about-aws/whats-new/2012/10/11/amazon-rds-mysql-rr-promotion/.
Some other alternatives include:
Selecting out a subset of records into a new table with the desired structure (use INTO OUTFILE to avoid a table lock). Once complete you can schedule a maintenance window and REPLACE INTO or UPDATE any records that have changed in the origin table since the initial data copy. Once the update is complete a RENAME TABLE... of both tables wraps the changes up.
Using a tool like Percona's pt-online-schema-change: http://www.percona.com/doc/percona-toolkit/2.1/pt-online-schema-change.html. This tool works with triggers so if you already have triggers on the tables you want to change this may not fit your needs.