I've got a table with 10M rows, and I'm trying to ALTER TABLE to add another column (a VARCHAR(80)).
From a data-modelling perspective, that column should be NOT NULL - but the amount of time it takes to run the statement is a consideration, and the client code could be changed to deal with a NULL column if that's warranted.
Should the NULL-ability of the column I'm trying to add significantly impact the amount of time it takes to add the column either way?
More Information
The context in which I'm doing this is a Django app, with a migration generated by South - adding three separate columns, and adding an index on one of the newly-added columns. Looking at the South-generated SQL, it spreads this operation (adding three columns and an index) over 15 ALTER TABLE statements - which seems like it will make this operation take a whole lot longer than it should.
I've seen some references that suggest that InnoDB doesn't actually have to create a field in the on-disk file for nullable fields that are NULL, and just modifies a bitfield in the header. Would this impact the speed of the ALTER TABLE operation?
I don't think the nullability of the column has anything to do with the speed of ALTER TABLE. In most alter table operations, the whole table - with all the indexes - has to be copied (temporarily) and then the alteration is done on the copy. With 10M rows, it's kind of slow. From MySQL docs:
Storage, Performance, and Concurrency Considerations
In most cases, ALTER TABLE makes a temporary copy of the original table. MySQL waits for other operations that are modifying the table, then proceeds. It incorporates the alteration into the copy, deletes the original table, and renames the new one. While ALTER TABLE is executing, the original table is readable by other sessions. Updates and writes to the table that begin after the ALTER TABLE operation begins are stalled until the new table is ready, then are automatically redirected to the new table without any failed updates. The temporary table is created in the database directory of the new table. This can differ from the database directory of the original table for ALTER TABLE operations that rename the table to a different database.
If you want to make several changes in a table's structure, it's usually better to do them in one ALTER TABLE operation.
Allowing client code to make changes in tables is probably not the best idea - and you have hit on one good reason for not allowing that. Why do you need it? If you can't do otherwise, it would probably be better - for performance reasons - to allow your client code to be creating a table (with the new column and the PK of the existing table) instead of adding a column.
Related
Is there a way to perform ALTER TABLE in MySQL, telling the server to skip creating a backup of the table first? I have a backup of the table already and I'm doing some tests on it (adding indexes), so I don't care if the table gets corrupted in the process. I'll just restore it from the backup. But what I do care about is for the ALTER TABLE to finish quickly, so I can see the test results.
Given that I have a big MyISAM table (700 GB) it really isn't an option to wait for couple of hours so that MySQL can first finish creating a backup of the original table, before actually adding an index to it.
It's not doing a backup; it is building the new version. (The existing table serves as a backup in case of a crash.)
With InnoDB, there are many flavors of ALTER TABLE -- some of which take essentially zero time, regardless of the size of the table. MyISAM (mostly) does the brute force way: Create an empty table with the new schema; copy all the data and build all the indexes; swap tables. For some alters, InnoDB must also do the brute force way: Example changing the PRIMARY KEY.
I have a table with the primary key of max signed INT hit, 2147483647
Imagine I want to switch it to unsigned, and there are no negative values in the table since it is a primary key, because Im under the current belief that it is the fastest way to get the table going again.
Should the ALTER TABLE statement to switch it to an unsigned INT be a relatively quick process, since the values of the ids shouldn't change? What about locking?
Mysql documentation on ALTER TABLE command describes quite in a detailed manner under "Storage, Performance, and Concurrency Considerations" section which changes can be done quickly, without table copy and index rebuild, and what locks mysql will apply during the course of the command. Changing the column type is unfortunately not listed as something that can be done in place (of course, read the documentation corresponding to your mysql version, I just linked the newest one).
For some operations, an in-place ALTER TABLE is possible that does not
require a temporary table:
For ALTER TABLE tbl_name RENAME TO new_tbl_name without any other options, MySQL simply renames any files that correspond to the table
tbl_name without making a copy. (You can also use the RENAME TABLE
statement to rename tables. See Section 13.1.28, “RENAME TABLE
Syntax”.) Any privileges granted specifically for the renamed table
are not migrated to the new name. They must be changed manually.
Alterations that modify only table metadata and not table data are immediate because the server only needs to alter the table .frm file,
not touch table contents. The following changes are fast alterations
that can be made this way:
Renaming a column.
Changing the default value of a column.
Changing the definition of an ENUM or SET column by adding new enumeration or set members to the end of the list of valid member
values, as long as the storage size of the data type does not change.
For example, adding a member to a SET column that has 8 members
changes the required storage per value from 1 byte to 2 bytes; this
will require a table copy. Adding members in the middle of the list
causes renumbering of existing members, which requires a table copy.
ALTER TABLE with DISCARD ... PARTITION ... TABLESPACE or IMPORT ... PARTITION ... TABLESPACE do not create any temporary tables or
temporary partition files.
ALTER TABLE with ADD PARTITION, DROP PARTITION, COALESCE PARTITION, REBUILD PARTITION, or REORGANIZE PARTITION does not create
any temporary tables (except when used with NDB tables); however,
these operations can and do create temporary partition files.
ADD or DROP operations for RANGE or LIST partitions are immediate operations or nearly so. ADD or COALESCE operations for HASH or KEY
partitions copy data between all partitions, unless LINEAR HASH or
LINEAR KEY was used; this is effectively the same as creating a new
table, although the ADD or COALESCE operation is performed partition
by partition. REORGANIZE operations copy only changed partitions and
do not touch unchanged ones.
Renaming an index.
Adding or dropping an index, for InnoDB.
Locking:
While ALTER TABLE is executing, the original table is readable by
other sessions (with the exception noted shortly). Updates and writes
to the table that begin after the ALTER TABLE operation begins are
stalled until the new table is ready, then are automatically
redirected to the new table without any failed updates. The temporary
copy of the original table is created in the database directory of the
new table. This can differ from the database directory of the original
table for ALTER TABLE operations that rename the table to a different
database.
The exception referred to earlier is that ALTER TABLE blocks reads
(not just writes) at the point where it is ready to install a new
version of the table .frm file, discard the old file, and clear
outdated table structures from the table and table definition caches.
At this point, it must acquire an exclusive lock. To do so, it waits
for current readers to finish, and blocks new reads (and writes).
Logic says it should be really fast to add a NULL column even if adding to a very large (~800 GB) table.
Obviously if the ALTER TABLE has a NOT NULL for the new column it will take a long time, but how about for NULL columns? How does it compare to altering a small table?
I'm on MySql 5.5.
Thanks!
MySQL implements ALTER TABLE as a table recreation.
In most cases, ALTER TABLE makes a temporary copy of the original table (...)
incorporates the alteration into the copy, deletes the original table,
and renames the new one.
The manual goes on and explains what "most cases" means. Basically: adding, removing, and redefining columns.
Only the table size matters, the column type has little to no impact in terms of execution time.
I have a 12 GB table full of pictures, I'm trying to rename the blob column that holds the data, and it is taking forever. Can someone give me a blow by blow account of why it is taking so long to rename the column? I would have thought that this operation would be pretty quick, no matter the size of the table?
EDIT: The query I ran is as follows
alter table `rails_production`.`pictures` change `data` `image_file_data` mediumblob NULL
It appears that most of the time is spent waiting for mysql to make a temporary copy of the pictures table, which since it is very large is taking a while to do.
It is on the list of things to do, to change the picture storage from the database to the filesystem.
EDIT2: Mysql Server version: 5.0.51a-24+lenny2 (Debian)
I can't give you the blow-by-blow (feature request #34354 would help, except that it probably wouldn't be back-ported to MySQL 5.0), but the extra time is due to the fact that an ALTER ... CHANGE may change the type of the column (and column attributes, if any), which necessitates converting the values stored in the column and other checks. MySQL 5.0 doesn't include optimizations for when the new type and attributes are the same as the old. From the documentation for ALTER under MySQL 5.0:
In most cases, ALTER TABLE works by making a temporary copy of the original table. The alteration is performed on the copy, and then the original table is deleted and the new one is renamed. While ALTER TABLE is executing, the original table is readable by other sessions. Updates and writes to the table are stalled until the new table is ready, and then are automatically redirected to the new table without any failed updates.
[...]
If you use any option to ALTER TABLE other than RENAME, MySQL always creates a temporary table, even if the data wouldn't strictly need to be copied (such as when you change the name of a column).
Under 5.1, ALTER has some additional optimizations:
In some cases, no temporary table is necessary:
Alterations that modify only table metadata and not table data can be made immediately by altering the table's .frm file and not touching table contents. The following changes are fast alterations that can be made this way:
Renaming a column, except for the InnoDB storage engine.
[...]
Because MySQL will rebuild the entire table when you make schema changes.
This is done because it's the only way of doing it in some cases, and it makes it much easier for the server to rebuild it anyway.
Yes mysql does a temporary copy of the table. I don't think there's an easy way around that. You should really think about to store the pictures on the filesystem and only store paths in mysql. That's the only way to fasten it up, I guess.
I have a 12 GB table full of pictures, I'm trying to rename the blob column that holds the data, and it is taking forever. Can someone give me a blow by blow account of why it is taking so long to rename the column? I would have thought that this operation would be pretty quick, no matter the size of the table?
EDIT: The query I ran is as follows
alter table `rails_production`.`pictures` change `data` `image_file_data` mediumblob NULL
It appears that most of the time is spent waiting for mysql to make a temporary copy of the pictures table, which since it is very large is taking a while to do.
It is on the list of things to do, to change the picture storage from the database to the filesystem.
EDIT2: Mysql Server version: 5.0.51a-24+lenny2 (Debian)
I can't give you the blow-by-blow (feature request #34354 would help, except that it probably wouldn't be back-ported to MySQL 5.0), but the extra time is due to the fact that an ALTER ... CHANGE may change the type of the column (and column attributes, if any), which necessitates converting the values stored in the column and other checks. MySQL 5.0 doesn't include optimizations for when the new type and attributes are the same as the old. From the documentation for ALTER under MySQL 5.0:
In most cases, ALTER TABLE works by making a temporary copy of the original table. The alteration is performed on the copy, and then the original table is deleted and the new one is renamed. While ALTER TABLE is executing, the original table is readable by other sessions. Updates and writes to the table are stalled until the new table is ready, and then are automatically redirected to the new table without any failed updates.
[...]
If you use any option to ALTER TABLE other than RENAME, MySQL always creates a temporary table, even if the data wouldn't strictly need to be copied (such as when you change the name of a column).
Under 5.1, ALTER has some additional optimizations:
In some cases, no temporary table is necessary:
Alterations that modify only table metadata and not table data can be made immediately by altering the table's .frm file and not touching table contents. The following changes are fast alterations that can be made this way:
Renaming a column, except for the InnoDB storage engine.
[...]
Because MySQL will rebuild the entire table when you make schema changes.
This is done because it's the only way of doing it in some cases, and it makes it much easier for the server to rebuild it anyway.
Yes mysql does a temporary copy of the table. I don't think there's an easy way around that. You should really think about to store the pictures on the filesystem and only store paths in mysql. That's the only way to fasten it up, I guess.