We have a varchar column right now that is 255 chars in length. We're about to up it to 400 using this statement:
ALTER TABLE `resources` CHANGE `url` `url` varchar(400) NOT NULL;
I've read the docs about online ddl which states
Operation In-Place? Copies Table? Allows Concurrent DML? Allows Concurrent Query?
---------------------------|-----------|---------------|-----------------------|---------------------------------
Change data type of column No Yes No Yes
And I have these two questions:
does changing the col from varchar(255) to varchar(400) constitute a changing of data type?
will this lock the table for writes?
I guess on question two, it just seems unclear what concurrent DML really means. Does it mean I can't write to this table at all, or that the table goes through the copy/swap process?
We only have about 2.5 million rows in this table, so the migration only takes about 30 seconds, but I'd prefer the table not be locked out during the time period.
I had the same question and ran some tests based on advice from Percona. Here are my findings:
ALTER TABLE `resources` CHANGE `url` `url` varchar(400), ALGORITHM=INPLACE, LOCK=NONE;
Running this on 5.6 should produce something similar to:
[SQL]ALTER TABLE `resources` CHANGE `url` `url` varchar(400), ALGORITHM=INPLACE, LOCK=NONE;
[Err] 1846 - ALGORITHM=INPLACE is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY.
What this means is that you cannot perform this operation as MySQL is considering this to be a column type change, and therefore a full table copy must be performed.
So let's try to use the COPY algorithm as suggested in the output, but set LOCK=NONE:
ALTER TABLE `resources` CHANGE `url` `url` varchar(400), ALGORITHM=COPY, LOCK=NONE;
And we get:
[SQL]ALTER `resources` CHANGE `url` `url` varchar(400), ALGORITHM=COPY, LOCK=NONE;
[Err] 1846 - LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED.
Trying to set LOCK=SHARED and attempting an insert on the table results in the query waiting for a metadata lock.
I believe you are trying this in production and you wish not to hamper your system.
You can do this in another way. Lets say, you want to change a column url(varchar 255) to url(varchar 400).
create another column, url2(varchar 400)
copy all data from url-> url2
rename "url" column to "url3"
rename "url2" column to name
run query for 2rd and 4th step together and it will take milli seconds to execute.
So There will be no table lock and your application will run smoothly.
I'm 99% certain that an alter against any table regardless of engine type will result in the table being locked until the op is complete, even with InnoDB's 'row level' locking capabilities.
If you can stand a 30-45 second outage where some connections will may and ultimately lost, then the easiest choice is to just pull the trigger. Or you could implement on of the following:
Put your site into 'maintenance mode' a few minutes before op, execute the op, then take the site out of maintenance mode.
Or, if you have a master master replication setup with a floating ip and dns, you could stop do this:
Stop replication on standby master
Run alter
Switch floating ip to standby master
Stop replication on primary master
Run alter
Restart replication on both masters
Switch floating ip back to primary master
Related
I'm currently trying to add a column to a table of ~25m rows. I need to have near-0 down time, so was hoping to use online DDL. It runs for a while, but eventually runs into the issue:
"Duplicate entry '1234' for key 'PRIMARY'"
[SQL: u'ALTER TABLE my_table ADD COLUMN my_coumn BOOL NOT NULL DEFAULT false']
I think this is happening because I'm running INSERT ... ON DUPLICATE KEY UPDATE ... operations against the table while running the operation. This seems to be a known limitation.
After this didn't work, I tried using the Percona pt-online-schema-change tool, but unfortunately, because my table has generated columns, that didn't work either with error:
The value specified for generated column 'my_generated_column' in table '_my_table_new' is not allowed.
So, I'm now at a loss. What are my other options for adding a column without blocking DML operations?
Your Alter statement is creating a non nullable column with a default of false. I'd suspect this to place an exclusive lock on your table, attempt to create the column, then setting it to False across each row.
If you don't have any available downtime, I'd suggest you
Add the column as nullable and with no default
ALTER TABLE my_table ADD COLUMN my_coumn BOOL NULL;
Update the values for existing rows to false
update my_table set my_coumn=false;
Alter the table a second time to be not nullable and with a default.
ALTER TABLE my_table modify my_coumn BOOL NOT NULL DEFAULT false;
Alternatively you could use something like Percona which manages schema changes using triggers and is meant to offer the ability to update schemas without locking the table.
Either option I'd suggest you test in your development environment with some process writing to the table to simulate user activity.
I have a 8Go table in my CloudSQL database that (for now) doesn't have a primary key.
It is composed of 52 million rows of 20 columns each.
I would like to add one, since I will remove duplicates and doing so without primary key is too timeconsuming for MySQL
However, I am having issues to make this run, and I'm wondering if I'm actually doing it in the most efficient way.
I wanted to add this primary key using the following line :
ALTER TABLE mytable ADD COLUMN id INT NOT NULL AUTO_INCREMENT PRIMARY KEY FIRST
I expected this request to increase the size of my database by a few hundreds MB, and run rather quickly.
However, when I ran the request, I saw the database usage increase a lot ( + 14 GB), and after 2 hours of running time, I saw the following error message :
"Error 1114 : the table 'mytable' is full".
When this error occurs, the database storage usage goes back to normal, but no change has been made to mytable
My questions are :
- What is CloudSQL trying to do when I launch my request, and how come it takes him so long ?
- What could I do to make my request executed more quickly by CloudSQL ? What am I missing ?
From http://dev.mysql.com/doc/refman/5.7/en/alter-table.html
"If you use "ALTER TABLE mytable AUTO_INCREMENT = N" the MySQL server will create a new table, copy all the data from the old table to the new, delete the old and then rename the new one even though there is no structural change to the table. "
I still have no clue concerning what happened with this request :
ALTER TABLE mytable ADD COLUMN id INT NOT NULL AUTO_INCREMENT PRIMARY KEY FIRST
But this alternative worked :
Creating a copy table with the exact same fields than in mytable plus the auto_increment
insert every record of the first table to this new table
This alternative actually increased less the storage than the one I use in my question (which ends up failing)
I have a table tblsessions. At one time, only one session could be current as is session 2014-2015.
However, if i make 2015-2016 current, 2014-2015 should not be current anymore.
How could I implement this logic in table at design time?
Here is the table creation code waiting for your modification:
create table tblsessions(
sessionid int not null auto_increment,
sessionname varchar(9) not null,
current ????
primary key (sessionid)
);
You could perhaps use a trigger (depending on the version of MySQL you're running). I've assumed that current is a tinyint but you can adjust to whatever type you use:
CREATE TRIGGER curr_check BEFORE UPDATE ON tblsessions
FOR EACH ROW
BEGIN
IF NEW.current = 1 THEN
UPDATE tblsessions SET current = 0;
END IF;
END;
EDIT:
A.5.3: Does MySQL 5.6 have statement-level or row-level triggers?
In MySQL 5.6, all triggers are FOR EACH ROW—that is, the trigger is activated for each row that is inserted, updated, or deleted. MySQL 5.6 does not support triggers using FOR EACH STATEMENT.
ALTERNATIVE SOLUTION:
I have come up with another solution however I wonder if it really is a good solution.
I have created two tables:
TBLSESSIONS (session)
// session is primary key and stops duplicates
TBLCURRENTSESSION (csessionid, csession)
// csessionid is auto-int
// csession is foreign key to TBLSESSIONS.session
Each time user presses a button [Make This Session Default], I can insert that session into csession.
In code I can search for largest csessionid and find csession against it as the CURRENT SESSION.
This also allows user to switch sessions at time.
Being MySQL DBA, do you think it is a good approach to solving my basic problem? Do you see any dark sides of this solution?
Is there a method to track changes to a MySQL database? I develop offline and then commit all the changes to my server. For the app itself I use Git and it works nicely.
However, for the database, I'm changing everything manually because the live database contains customer data and I cannot just replace it with the development database.
Is there a way to only have the structural changes applied without completely replacing one db with another?
The term you're looking for is 'database migrations' (and no, it doesn't refer to moving from one RDBMS to another). Migrations are a way to programatically version control your database structure. Most languages have some kind of migrations toolkit, often as part of an ORM library/framework.
For PHP you can look at Doctrine
For Ruby it's Rails of course
The key to have keep track of your changes is Snapshots my friend.
Now, it's a wide field. The first thing you have to do is decide if you want to keep track of your database with some kind of data in it. If that's the case you have several options, from using LVM, copying InnoDB binary logs, and the simple mysqldump.
Now, if what you wanna do is have some smooth transition between your database changes (i mean, you added a column, for example), you have some other options.
The first one is replication. That's a great option, but is a little complex. With replication you may alter one slave and after it's done, with some locking, you can make it master, and replace master, and so on. It's really difficult, but is the better option.
If you cannot afford replication, what you must do is apply the changes to your single-master DB with the minimum downtime. Some good option is this:
Suppose you want to replace your Customer table to add a "facebook_account" field. First, you can use an alias table, like this:
The original table (it has data):
CREATE TABLE `customer` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB;
The new one:
CREATE TABLE `new_customer` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) NOT NULL,
`facebook_account` varchar(255) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB;
Or simply:
CREATE TABLE new_customer LIKE customer;
ALTER TABLE new_customer add column facebook_account VARCHAR(255);
Now we're gonna copy the data to the new table. We'll need to issue some other things first, i'll explain them each at a time.
First, you can allow other connections to modify the customer table while your making the change of table, so i'll issue a lock. If you want to learn more about this go here:
LOCK TABLES customer WRITE ,new_customer WRITE;
Now i flush the table to write any cache content to the filesystem:
FLUSH TABLES customer;
Now we can do the insert. First I disable the keys for performance issues. After the data is inserted i enable the keys again.
ALTER TABLE new_customer DISABLE KEYS;
INSERT INTO new_customer(id,name,facebook_account) SELECT customer.id,customer.name, Null FROM customer;
ALTER TABLE new_customer ENABLE KEYS;
Now we can switch the tables.
ALTER TABLE customer RENAME old_customer;
ALTER TABLE new_customer RENAME customer;
Finally we have to release the lock.
UNLOCK TABLES;
That's it. If you want to keep track of your modified tables you may want to rename your old_customer table, to something else or move it to other database.
The only issue i didn't cover here is about Triggers. You have to pay atention to any enabled trigger, but it will depend on your schema.
That's it, hope it helps.
I'm trying to make our mySQL database run faster and I've analyzed our slow query log and the most common slow query is this:
CREATE TABLE IF NOT EXISTS `wp_bad_behavior` (
`id` INT(11) NOT NULL auto_increment,
`ip` TEXT NOT NULL,
`date` DATETIME NOT NULL default '0000-00-00 00:00:00',
`request_method` TEXT NOT NULL,
`request_uri` TEXT NOT NULL,
`server_protocol` TEXT NOT NULL,
`http_headers` TEXT NOT NULL,
`user_agent` TEXT NOT NULL,
`request_entity` TEXT NOT NULL,
`key` TEXT NOT NULL,
INDEX (`ip`(15)),
INDEX (`user_agent`(10)),
PRIMARY KEY (`id`) );
I'm trying to understand why this query keeps getting called because after the table is setup it should not keep happening.
The EXPLAIN result for this is: Cannot convert to a SELECT statement.
Any ideas on this would be fantastic!
Paul
EXPLAIN will only work on SELECT queries, that's why it complains. As to why the query is in the slow query log: Either it is deleted and recreated - check the regular query log for DROP TABLEs - or it simply blocks because the table/database is busy - check the other slow queries first, especially the ones on the same table.
Of course you can't convert a CREATE statement to a SELECT statement...
The question therefore is, why a creation statement is called so frequently. It uses mysql's IF NOT EXISTS, so it might not even be a design issue, the table would only get created once despite the query is called bazillion times. Maybe your system that uses the database issues this statement from every single method, as a way to make sure actual database structure matches its expectations. Just a foolprof in case somebody deletes this super important table.
Its probably there as an easy way to recover from someone deleting the table. A good reason why someone would delete the table is that it's an easy way to get rid of old log files. However this obviously affects performance and so you should check that none of the code is deleting the table and then remove this check. Then you will be required to manually recreate the table when deleting old logs.