I'm trying to execute an ALTER TABLE in MySQL. MySQL only lets me execute it with the ALGORITHM=COPY (because I need to change the type of a column).
There aren't queries using that table (to write neither to read).
But, I don't know why, when I execute the ALTER there are queries (UPDATES) which are not using this table (they are in a transaction) locked. MySQL says "mysql waiting for metadata lock".
So the question is, why the query is waiting for metadata lock if the UPDATE is not using the table altered?
I read some doc:
https://dev.mysql.com/doc/refman/8.0/en/alter-table.html#alter-table-performance
https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-performance.html#innodb-online-ddl-locking-options
But I don't understand why the queries are locked for metadata.
Reproduction of the problem in dev environment:
First, do the alter:
ALTER TABLE API.SEARCHES_ELEMENTS
MODIFY COLUMN TYPE ENUM('A', 'B') NOT NULL,
ALGORITHM=COPY;
Second, change values in other tables (there isn't transaction):
UPDATE CLIENTS
SET NAME = CONCAT('test-', RAND())
WHERE ID_CLIENT = 1;
The locks:
SELECT *
FROM performance_schema.metadata_locks
INNER JOIN performance_schema.threads ON THREAD_ID = OWNER_THREAD_ID
WHERE
PROCESSLIST_ID <> CONNECTION_ID();
Maybe the problem is due the lock over the SCHEMA?
Seems like the idea is to avoid the ALGORITHM=COPY (rebuilds without in-place mode)
So instead of modify the column type
ALTER TABLE API.SEARCHES_ELEMENTS
MODIFY COLUMN TYPE ENUM('A', 'B') NOT NULL,
ALGORITHM=COPY;
is better to create a new column, copy the data and remove the old one:
ALTER TABLE API.SEARCHES_ELEMENTS
ADD COLUMN TYPE_NEW ENUM('A', 'B') NOT NULL AFTER TYPE,
ALGORITHM=INSTANT;
LOCK TABLES API.SEARCHES_ELEMENTS WRITE;
UPDATE API.SEARCHES_ELEMENTS SET TYPE_NEW = TYPE;
ALTER TABLE API.SEARCHES_ELEMENTS
RENAME COLUMN TYPE TO TYPE_OLD,
RENAME COLUMN TYPE_NEW TO TYPE,
ALGORITHM=INSTANT;
UNLOCK TABLES;
ALTER TABLE API.SEARCHES_ELEMENTS
DROP COLUMN TYPE_OLD,
ALGORITHM=INPLACE;
Note: adding a value in ENUM might be use the algorithm=instant
Modifying the definition of an ENUM or SET column by adding new
enumeration or set members to the end of the list of valid member
values may be performed instantly or in place, as long as the storage
size of the data type does not change. For example, adding a member to
a SET column that has 8 members changes the required storage per value
from 1 byte to 2 bytes; this requires a table copy. Adding members in
the middle of the list causes renumbering of existing members, which
requires a table copy.
A tables metadata is not only locked when a running query is using it, but also if it has previously been used in an active transaction until that transaction commits or rolls back, to prevent the table from changing while still being referenced by the transaction.
If you are running at least MySQL 5.7 and have the performance_schema enabled you can check for current metadata locks via the performance_schema.metadata_locks table.
See also:
https://dev.mysql.com/doc/refman/5.7/en/performance-schema-metadata-locks-table.html
https://dev.mysql.com/doc/refman/5.7/en/metadata-locking.html
Related
I need to rename MySQL table and create a new MySQL table at the same time.
There is critical live table with large number of records. master_table is always inserted records from scripts.
Need to backup the master table and create a another master table with same name at the same time.
General SQL is is like this.
RENAME TABLE master_table TO backup_table;
Create table master_table (id,value) values ('1','5000');
Is there a possibility to record missing data during the execution of above queries?
Any way to avoid missing record? Lock the master table, etc...
What I do is the following. It results in no downtime, no data loss, and nearly instantaneous execution.
CREATE TABLE mytable_new LIKE mytable;
...possibly update the AUTO_INCREMENT of the new table...
RENAME TABLE mytable TO mytable_old, mytable_new TO mytable;
By renaming both tables in one statement, they are swapped atomically. There is no chance for any data to be written "in between" while there is no table to receive the write. If you don't do this atomically, some writes may fail.
RENAME TABLE is virtually instantaneous, no matter how large the table. You don't have to wait for data to be copied.
If the table has an auto-increment primary key, I like to make sure the new table starts with an id value greater than the current id in the old table. Do this before swapping the table names.
SELECT AUTO_INCREMENT FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA='mydatabase' AND TABLE_NAME='mytable';
I like to add some comfortable margin to that value. You want to make sure that the id values inserted to the old table won't exceed the value you queried from INFORMATION_SCHEMA.
Change the new table to use this new value for its next auto-increment:
ALTER TABLE mytable_new AUTO_INCREMENT=<increased value>;
Then promptly execute the RENAME TABLE to swap them. As soon as new rows are inserted to the new, empty table, it will use id values starting with the increased auto-increment value, which should still be greater than the last id inserted into the old table, if you did these steps promptly.
Instead of renaming the master_backup table and recreating it, you could
just create a backup_table with the data from the master_table for the first backup run.
CREATE TABLE backup_table AS
SELECT * FROM master_table;
If you must add a primary key to the backup table then run this just once, that is for the first backup:
ALTER TABLE backup_table ADD CONSTRAINT pk_backup_table PRIMARY KEY(id);
For future backups do:
INSERT INTO backup_table
SELECT * FROM master_table;
Then you can delete all the data in the backup_table found in the master_table like:
DELETE FROM master_table A JOIN
backup_table B ON A.id=B.id;
Then you can add data to the master_table with this query:
INSERT INTO master_table (`value`) VALUES ('5000'); -- I assume the id field is auto_incrementable
I think this should work perfectly even without locking the master table, and with no missing executions.
I encountered the following problem:
I have temporary table
CREATE temporary TABLE IF NOT EXISTS tmp_GL_VAR
(
G_TABLE_NAME VARCHAR(100) DEFAULT '',
G_DATE DATETIME,
G_ERROR_CODE INT DEFAULT 0
);
And I need to use it several times in single queries,for instance,
update t1 set c1 = (select G_TABLE_NAME from tmp_GL_VAR), c2 = (select G_ERROR_CODE from tmp_GL_VAR);
in functions with cursors, etc. However, in all these cases MySQL throws error:
SQL Error(1137): Can't reopen table 'tmp_GL_VAR'.
Then I tried to create permanent table with STORAGE MEMORY clause (also tried ENGINE MEMORY clause), hoping that table will be cleaned when the session ends
CREATE TABLE GL_VAR
(
G_TABLE_NAME VARCHAR(100) DEFAULT '',
G_DATE DATETIME,
G_ERROR_CODE INT DEFAULT 0
) STORAGE MEMORY;
But unfortunately this option had no effect. The data was available across different sessions (connections).
Please advise how I can bypass 'Can't reopen table' without rewriting all queries, stored functions, etc. (there're far too many LOC).
The requirement is: table should be either dropped or at least truncated as session ends and the data from one session shouldn't be available in another session (each user can see only its own data in this table).
Any help is appreciated.
Yes. there are problems http://dev.mysql.com/doc/refman/5.1/en/temporary-table-problems.html
You cannot refer to a TEMPORARY table more than once in the same query. For example, the following does not work:
mysql> SELECT * FROM temp_table, temp_table AS t2;
ERROR 1137: Can't reopen table: 'temp_table'
This error also occurs if you refer to a temporary table multiple times in a stored function under different aliases, even if the references occur in different statements within the function.
I think u should use usual table for this purpose.
And add some session identifier to make it work in your multi-user system. (Yes, u will have to add this identifier to ALL other queries using this table)
Delete rows by this identifier (or old timestamps if you want) any time u wish
I ran into a problem and can't choose the right solution.
I have a SELECT query that selects records from table.
These records has an status column as seen below.
SELECT id, <...>, status FROM table WHERE something
Now, right after this SELECT I have to UPDATE the status column.
How can I do it to avoid a race condition?
What I want to achieve is once somebody (session) selected something, this something cannot be selected by anybody else until I do not release it manually (for example using a status column).
Thoughts?
There is some mysql documentation, thar may be interesting to solve your task, not sure if it fit you needs, but it describes right way to do select followed by update.
The technique described does not prevent other sessions reading, but prevent writing of selected record until the end of transaction.
It contains an example similar to your problem:
SELECT counter_field FROM child_codes FOR UPDATE;
UPDATE child_codes SET counter_field = counter_field + 1;
It is required that you tables use Innodb engine and your programs use transactions.
If you need locking only for short time, i.e. one session select row with lock, update it, and release lock in one session, then you do not need field status at all, just use select ... for update and select ... lock in share mode so if all sessions will use these two with conjunction with transactions select... for update then update to modify, and select ... with shared lock to just read - this will solve your requirements.
If you need to lock for long time, select and lock in one session and then update and release in another, then right you use some storage to keep lock statuses and all session should use as described below: select ... for update and set status and status owner in one session, then in another session select for update check status and owner, update and remove status - for updating scenario, and for read scenario: select ... with shared lock check status.
You can do it with some preparations. Add a column sessionId to your table. It has to be NULL-able and it will contain the unique ID of the session that acquires the row. Also add an index on this new column; we'll use the column to search for rows in the table.
ALTER TABLE `tbl`
ADD COLUMN `sessionId` CHAR(32) DEFAULT NULL,
ADD INDEX `sessionId`(`sessionId`)
When a session needs to acquire some rows (based on some criteria) run:
UPDATE `tbl`
SET `sessionId` = 'aaa'
WHERE `sessionId` IS NULL
AND ...
LIMIT bbb
Replace aaa with the current session ID and ... with the conditions you need to select the correct rows. Replace bbb with the number of rows you need to acquire. Add an ORDER BY clause if you need to process the rows in a certain order (if some of them have higher priority than others). You can also add status = ... in the UPDATE clause to change the status of the acquired rows (to pending f.e.) to let other instances of the code know those rows are processed right now.
The query above acquires some rows. Next, run:
SELECT *
FROM `tbl`
WHERE `sessionId` = 'aaa'
This query gets the acquired rows to be processed in the client code.
After each row is processed, you either DELETE the row or UPDATE it and set sessionId to NULL (release the row) and status to reflect its new status.
Also you should release the rows (using the same procedure as above) when the session is closed.
I was just trying to add a column called "location" to a table (main_table) in a database. The command I run was
ALTER TABLE main_table ADD COLUMN location varchar (256);
The main_table contains > 2,000,000 rows. It keeps running for more than 2 hours and still not completed.
I tried to use mytop
to monitor the activity of this database to make sure that the query is not locked by other querying process, but it seems not. Is it supposed to take that long time? Actually, I just rebooted the machine before running this command. Now this command is still running. I am not sure what to do.
Your ALTER TABLE statement implies mysql will have to re-write every single row of the table including the new column. Since you have more than 2 million rows, I would definitely expect it takes a significant amount of time, during which your server will likely be mostly IO-bound. You'd usually find it's more performant to do the following:
CREATE TABLE main_table_new LIKE main_table;
ALTER TABLE main_table_new ADD COLUMN location VARCHAR(256);
INSERT INTO main_table_new SELECT *, NULL FROM main_table;
RENAME TABLE main_table TO main_table_old, main_table_new TO main_table;
DROP TABLE main_table_old;
This way you add the column on the empty table, and basically write the data in that new table that you are sure no-one else will be looking at without locking as much resources.
I think the appropriate answer for this is using a feature like pt-online-schema-change or gh-ost.
We have done migration of over 4 billion rows with this, though it can take upto 10 days, with less than a minute of downtime.
Percona works in a very similar fashion as above
Create a temp table
Creates triggers on the first table (for inserts, updates, deletes) so that they are replicated to the temp table
In small batches, migrate data
When done, rename table to new table, and drop the other table
You can speed up the process by temporarily turning off unique checks and foreign key checks. You can also change the algorithm that gets used.
If you want the new column to be at the end of the table, use algorithm=instant:
SET unique_checks = 0;
SET foreign_key_checks = 0;
ALTER TABLE main_table ADD location varchar(256), algorithm=instant;
SET unique_checks = 1;
SET foreign_key_checks = 1;
Otherwise, if you need the column to be in a specific location, use algorithm=inplace:
SET unique_checks = 0;
SET foreign_key_checks = 0;
ALTER TABLE main_table ADD location varchar(256) AFTER othercolumn, algorithm=inplace;
SET unique_checks = 1;
SET foreign_key_checks = 1;
For reference, it took my PC about 2 minutes to alter a table with 20 million rows using the inplace algorithm. If you're using a program like Workbench, then you may want to increase the default timeout period in your settings before starting the operation.
If you find that the operation is hanging indefinitely, then you may need to look through the list of processes and kill whatever process has a lock on the table. You can do that using these commands:
SHOW FULL PROCESSLIST;
KILL PROCESS_NUMBER_GOES_HERE;
Alter table takes a long time with a big data like in your case, so avoid to use it in such situations, and use some code like this one:
select main_table.*,
cast(null as varchar(256)) as null_location, -- any column you want accepts null
cast('' as varchar(256)) as not_null_location, --any column doesn't accept null
cast(0 as int) as not_null_int, -- int column doesn't accept null
into new_table
from main_table;
drop table main_table;
rename table new_table TO main_table;
DB2 z/OS does a virtual add of the column instantly. And puts the table into Advisory-Reorg status. Anything that runs before the reorg gets the default value or null if no default. When updates are done, they expand the rows updated. Inserts are done expanded. The next reorg expands every unexpanded row and assigns the default value to anything it expands.
Only a real database handles this well. DB2 z/OS.
I have an ISAm table in mySql that was created similar to this:
create table mytable (
id int not null auto_increment primary key,
name varchar(64) not null );
create unique index nameIndex on mytable (name);
I have multiple processes inserting rows into this table. If two processes try to insert the same "name", I want to make sure that one of them either gets an error or finds the row with the matching "name".
Should I lock the table and in the lock make sure that the name doesn't exist, or should I rely on the server giving an error to one of the processes that try to insert a value that already exists in the unique indexed field?
I'm a bit hesitant to use a lock because I don't want to get into a deadlock situation.
Do not bother locking, your index will prevent duplicates. You should handle the error code from your application.
MySQL should return an error code of 1062 (or SQLSTATE 23000) when your unique key constraint is violated.
By the way you described the fear of encountering a DEADLOCK, the causation may not be clearly understood (unless there is more to your querying than described in the question).
A good summary someone else wrote:
Query 1 begins by locking resource A
Query 2 begins by locking resource B
Query 1, in order to continue, needs a lock on resource B, but Query 2 is locking that resource, so Query 1 starts waiting for it to release
In the meantime, Query 2 tries to finish, but it needs a lock on resource A in order to finish, but it can't get that because Query 1 has the lock on that.