I have a schema that is used to archive a data set on a daily basis. Some of the analysis needs to look back, so to optimise things I need to create a couple of indexes on each table. These would be seperate (I'm not trying to cross index or anything) just a simple non-unique index, but on each table in the schema.
The archive has already been building for over a year, so we have some 400 - 500 tables, making a manual ALTER query on each tablea bit too time consuming.
I could write a php script to do it, but wondered if there was a more elegant solution with a single query or transaction?
TIA
I have copied #Shadow's answer in the comments above here to show it as the answer:
Well, the alter table and add index sections will be string constants as you have to generate the alter table statements and then execute the alter table statements you generated in the first step. See an example here: stackoverflow.com/a/44527818/5389997
Related
Some databases, like MySQL [1] and PostgreSQL [2], support bundling of certain compatible ALTER TABLE statements (as non-standard SQL).
For example we can have:
ALTER TABLE `my_table`
DROP COLUMN `column_1`,
DROP COLUMN `column_2`,
...
or
ALTER TABLE
MODIFY `column_1` ... ,
MODIFY `column_2` ... ,
instead of having individual statements:
ALTER TABLE `my_table` DROP COLUMN `column_1`;
ALTER TABLE `my_table` DROP COLUMN `column_2`;
or
ALTER TABLE `my_table` MODIFY `column_1` ... ;
ALTER TABLE `my_table` MODIFY `column_2` ... ;
etc
For comparison of the same feature, PostgreSQL [2], which also implements this, will perform all operations in a single scan:
The main reason for providing the option to specify multiple changes in a single ALTER TABLE is that multiple table scans or rewrites can thereby be combined into a single pass over the table.
Although for DROP COLUMN specifically it will often not even need do that:
The DROP COLUMN form does not physically remove the column, but simply makes it invisible to SQL operations...
Questions:
Would the multi-column statement result in traversing all the rows just once and performing all changes needed?
How does MySQL actually perform DROP COLUMN? Does it also "hide" the columns first, or does it delete the data straight away?
Assumptions:
Using InnoDB
No indexes/complex defaults are involved in any of the columns we want to change/drop (so basically changes that would not require a temporary table when run as individual alter statements)
References:
[1] MySQL ALTER TABLE docs
[2] PostgreSQL ALTER TABLE docs
MySQL's InnoDB:
(This does not really answer the Questions, but provides a little more insight in the the bigger question of ALTER.)
If any of the alters needs to copy the table over, you are probably better off putting all alters into the same statement. Changing the PRIMARY KEY, for example, requires rebuilding the data that is clustered with the PK.
Some alters can be achieved by simply altering the schema; these are virtually instantaneous, and could be done via separate alter statements. Adding an option to ENUM was implemented long ago.
Some alters need some form of scan, but can do it "in the background". DROP INDEX can be done by quickly "hiding" it, then freeing up the BTree in the background.
I have left out a grey area in which you batch 'simple' alters. One would hope that ALTER is smart enough to simply go through them quickly, rather than deciding to copy the table over.
I got some useful feedback but decided to respond to my own question to provide a more concrete set of answers.
Would the multi-column statement result in traversing all the rows just once and performing all changes needed?
Yes, if the alter statement results in rebuilding the table then it only needs to do it once.*
* This answer comes from my own testing and other mostly anecdotal evidence (including #Uueerdo 's in this post). It would be useful to have some official docs for this...
How does MySQL actually perform DROP COLUMN? Does it also "hide" the columns first, or does it delete the data straight away?
MySQL will rebuild the table in place (rather than create a copy or just change metadata) for most column operations. Each specific case can be found in the Online DDL docs for InnoDB.
A few operations like renaming a column or setting a default value will just alter metadata, so they don't require a table rebuild.
However, dropping a column DOES require a full table rebuild.
I normally use phpMyAdmin for any database changes therefore rarely delve into making changes to schema via MySQL queries. I now however have multiple databases all exactly the same and it would be a huge time saver if I could alter all databases at the same time. I can use the following code for a single database
ALTER demo_database.demo_table ADD another_demo_column VARCHAR(255) AFTER demo_column_index
but I'm curious if I can use similar code like the following which obviously doesn't work or I wouldn't be asking the question.
ALTER demo_database.demo_table, demo_database_2.demo_table ADD another_demo_column VARCHAR(255) AFTER demo_column_index
I know I could obviously duplicate the same line of code over and over for each table but it would be good if there is a more elegant solution.
Any help is appreciated.
According to the MySQL documentation for ALTER you cannot combine the same alter statements across multiple tables.
You can however combine multiple alter statements to the same table.
If you had a lot to do then a stored procedure with prepared statements would be more elegant than copying and pasting.
Is there a way to do this?
In case the DBMS command history got cleaned or, in my case, when many ALTER TABLE were used in the course of time.
I'm using MySQL.
Yes, it is as simple as
SHOW CREATE TABLE yourtable;
This will include all the subsequent ALTER TABLE statements. You cannot retrieve the table's original state.
Here is the relevant documentation
I want to run this on my table:
ALTER TABLE table_name MODIFY col_name VARCHAR(255)
But my table is huge, it has more than 65M (65 million) rows. Now when I execute, it takes nearly 50mins to execute this command. Any better way to alter table?
Well, you need
ALTER TABLE table_name CHANGE col_name new_name VARCHAR(255)
But, you are right, it takes a while to make the change. There really isn't any faster way to change the table in MySQL.
Is your concern downtime during the change? If so, here's a possible approach: Copy the table to a new one, then change the column name on the copy, then rename the copy.
You probably have figured out that routinely changing column names in tables in a production system is not a good idea.
another variant to use percona toolkit
https://www.percona.com/doc/percona-toolkit/2.2/pt-online-schema-change.html
You can deal with schema change without downtime using Oak.
oak-online-alter-table copies schema of original table, applies your changes and then copies the data. The CRUD operations can still be invoked as oak puts some triggers on original table so no data is going to be lost during the operation.
Please refer to other question where author of oak gives detailed explanation about this mechanism and also suggests other tools.
Is there a way to rename a column on an InnoDB table without a major alter?
The table is pretty big and I want to avoid major downtime.
Renaming a column (with ALTER TABLE ... CHANGE COLUMN) unfortunately requires MySQL to run a full table copy.
Check out pt-online-schema-change. This helps you to make many types of ALTER changes to a table without locking the whole table for the duration of the ALTER. You can continue to read and write the original table while it's copying the data into the new table. Changes are captured and applied to the new table through triggers.
Example:
pt-online-schema-change h=localhost,D=databasename,t=tablename \
--alter 'CHANGE COLUMN oldname newname NUMERIC(9,2) NOT NULL'
Update: MySQL 5.6 can do some types of ALTER operations without rebuilding the table, and changing the name of a column is one of those supported as an online change. See http://dev.mysql.com/doc/refman/5.6/en/innodb-create-index-overview.html for an overview of which types of alterations do or don't support this.
If there aren't any constraints on it, you can alter it without a hassle as far as I know. If there are you'll have to remove the constraints first, alter and add the constraints back.
Altering a table with many rows can take a long time (though if the columns involved are not indexed, it may be trivial).
If you specifically want to avoid using the ALTER TABLE syntax created specifically for that purpose, you can always create a table with almost the exact same structure (but different name) and copy all the data into it, like so:
CREATE TABLE `your_table2` ...;
-- (using the query from SHOW CREATE TABLE `your_table`,
-- but modified with your new column changes)
LOCK TABLES `your_table` WRITE;
INSERT INTO `your_table2` SELECT * FROM `your_table`;
RENAME TABLE `your_table` TO `your_table_old`, `your_table2` TO `your_table`;
For some ALTER TABLE queries, the above can be quite a bit faster. However, for a simple column name change, it could be trivial. I might try creating an identical table and performing the change on it in order to see how much time you're actually looking at.