This is a random behavior(didn't reoccur after restarting the mysql session) that when I wrapped the DDL query in a transaction T1 and at same time I initiate another transaction T2 with a select query on same table, DDL query gets stuck waiting for table metadata lock which is expected, and any other select queries waits for DDL query to finish. But after committing T2, DDL is supposed to get metadata lock and finish, but it remains waiting for table metadata lock state.
Connection 1 Query:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> ALTER TABLE merchants ADD COLUMN temp6 varchar(255);
Connection 2 Query:
mysql> begin;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from merchants where account_id=null;
Empty set (0.00 sec)
mysql> commit;
Query OK, 0 rows affected (0.00 sec)
Sequence of queries is :
Connection 1 : begin;
Connection 2 : begin;
Connection 2 : select * from merchants where account_id=null;
Connection 1 : ALTER TABLE merchants ADD COLUMN temp6 varchar(255);
Connection 2 : commit;
Server version: 8.0.13 MySQL Community Server - GPL
mysql> select ##autocommit;
+--------------+
| ##autocommit |
+--------------+
| 1 |
+--------------+
1 row in set (0.00 sec)
mysql> SELECT
OBJECT_TYPE,
OBJECT_SCHEMA,
OBJECT_NAME,
LOCK_TYPE,
LOCK_STATUS,
THREAD_ID,
PROCESSLIST_ID,
PROCESSLIST_INFO
FROM performance_schema.metadata_locks
INNER JOIN performance_schema.threads ON THREAD_ID = OWNER_THREAD_ID
WHERE PROCESSLIST_ID <> CONNECTION_ID();
+-------------+-------------------------+-----------------------------------+---------------------+-------------+-----------+----------------+-----------------------------------------------------+
| OBJECT_TYPE | OBJECT_SCHEMA | OBJECT_NAME | LOCK_TYPE | LOCK_STATUS | THREAD_ID | PROCESSLIST_ID | PROCESSLIST_INFO |
+-------------+-------------------------+-----------------------------------+---------------------+-------------+-----------+----------------+-----------------------------------------------------+
| GLOBAL | NULL | NULL | INTENTION_EXCLUSIVE | GRANTED | 126 | 87 | ALTER TABLE merchants ADD COLUMN temp6 varchar(255) |
| SCHEMA | merchant_onboarding_dev | NULL | INTENTION_EXCLUSIVE | GRANTED | 126 | 87 | ALTER TABLE merchants ADD COLUMN temp6 varchar(255) |
| TABLE | merchant_onboarding_dev | merchants | SHARED_UPGRADABLE | GRANTED | 126 | 87 | ALTER TABLE merchants ADD COLUMN temp6 varchar(255) |
| BACKUP LOCK | NULL | NULL | INTENTION_EXCLUSIVE | GRANTED | 126 | 87 | ALTER TABLE merchants ADD COLUMN temp6 varchar(255) |
| TABLESPACE | NULL | merchant_onboarding_dev/merchants | INTENTION_EXCLUSIVE | GRANTED | 126 | 87 | ALTER TABLE merchants ADD COLUMN temp6 varchar(255) |
| TABLE | merchant_onboarding_dev | #sql-63_57 | EXCLUSIVE | GRANTED | 126 | 87 | ALTER TABLE merchants ADD COLUMN temp6 varchar(255) |
| TABLE | merchant_onboarding_dev | merchants | EXCLUSIVE | PENDING | 126 | 87 | ALTER TABLE merchants ADD COLUMN temp6 varchar(255) |
+-------------+-------------------------+-----------------------------------+---------------------+-------------+-----------+----------------+-----------------------------------------------------+
7 rows in set (0.00 sec)
mysql> show full processlist;
+----+-----------------+-----------------+-------------------------+---------+--------+---------------------------------+-----------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+-----------------+-----------------+-------------------------+---------+--------+---------------------------------+-----------------------------------------------------+
| 4 | event_scheduler | localhost | NULL | Daemon | 115824 | Waiting on empty queue | NULL |
| 62 | root | localhost:59116 | merchant_onboarding_dev | Sleep | 107 | | NULL |
| 63 | root | localhost:59117 | merchant_onboarding_dev | Sleep | 107 | | NULL |
| 65 | root | localhost:59119 | NULL | Sleep | 1 | | NULL |
| 79 | root | localhost | merchant_onboarding_dev | Query | 0 | starting | show full processlist |
| 81 | root | localhost | merchant_onboarding_dev | Sleep | 838 | | NULL |
| 83 | root | localhost | merchant_onboarding_dev | Sleep | 821 | | NULL |
| 87 | root | localhost | merchant_onboarding_dev | Query | 842 | Waiting for table metadata lock | ALTER TABLE merchants ADD COLUMN temp6 varchar(255) |
+----+-----------------+-----------------+-------------------------+---------+--------+---------------------------------+-----------------------------------------------------+
8 rows in set (0.00 sec)
mysql> SHOW ENGINE INNODB STATUS;
=====================================
2019-04-08 02:30:45 0x700006ef3000 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 20 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 242 srv_active, 0 srv_shutdown, 73418 srv_idle
srv_master_thread log flush and writes: 0
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 87
OS WAIT ARRAY INFO: signal count 516
RW-shared spins 722, rounds 723, OS waits 1
RW-excl spins 1110, rounds 3537, OS waits 28
RW-sx spins 0, rounds 0, OS waits 0
Spin rounds per wait: 1.00 RW-shared, 3.19 RW-excl, 0.00 RW-sx
------------
TRANSACTIONS
------------
Trx id counter 15753
Purge done for trx's n:o < 15749 undo n:o < 0 state: running but idle
History list length 11
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 281479647891952, not started
mysql tables in use 1, locked 1
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 281479647891040, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 281479647893776, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 281479647892864, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 281479647890128, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 281479647889216, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: waiting for i/o request (insert buffer thread)
I/O thread 1 state: waiting for i/o request (log thread)
I/O thread 2 state: waiting for i/o request (read thread)
I/O thread 3 state: waiting for i/o request (read thread)
I/O thread 4 state: waiting for i/o request (read thread)
I/O thread 5 state: waiting for i/o request (read thread)
I/O thread 6 state: waiting for i/o request (write thread)
I/O thread 7 state: waiting for i/o request (write thread)
I/O thread 8 state: waiting for i/o request (write thread)
I/O thread 9 state: waiting for i/o request (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
950 OS file reads, 52924 OS file writes, 40911 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2, 0 merges
merged operations:
insert 0, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 34679, node heap has 3 buffer(s)
Hash table size 34679, node heap has 3 buffer(s)
Hash table size 34679, node heap has 7 buffer(s)
Hash table size 34679, node heap has 2 buffer(s)
Hash table size 34679, node heap has 2 buffer(s)
Hash table size 34679, node heap has 1 buffer(s)
Hash table size 34679, node heap has 2 buffer(s)
Hash table size 34679, node heap has 4 buffer(s)
0.00 hash searches/s, 0.00 non-hash searches/s
---
LOG
---
Log sequence number 89894761
Log buffer assigned up to 89894761
Log buffer completed up to 89894761
Log written up to 89894761
Log flushed up to 89894761
Added dirty pages up to 89894761
Pages flushed up to 89894761
Last checkpoint at 89894761
42886 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 137428992
Dictionary memory allocated 1037251
Buffer pool size 8191
Free buffers 3332
Database pages 4835
Old database pages 1764
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 634, not young 1
0.00 youngs/s, 0.00 non-youngs/s
Pages read 911, created 3927, written 8771
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4835, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Process ID=99, Main thread ID=0x700006689000 , state=sleeping
Number of rows inserted 176434, updated 4249, deleted 10814, read 2383183
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
What I am not able to understand is why DDL query kept on waiting for table metadata lock even though the transaction which held the lock commits.
In MySQL, most DDL statements actually generate an implicit commit. The ALTER TABLE statement does fall in this category. From the documentation:
The statements listed in this section (and any synonyms for them) implicitly end any transaction active in the current session, as if you had done a COMMIT before executing the statement.
Most of these statements also cause an implicit commit after executing.
This internal documentation goes into more details... and things tend to get messy:
In addition, some DDL statements issue interim transaction commits: for example, ALTER TABLE issues a commit after data is copied from the original table to the internal temporary table.
So while you execute the ALTER TABLE, MySQL implicitly tries to commit within the statement, but can't since another transaction is in progress. The documentation does not precisely covers this use case, but I would suspect that the server does not properly handle this situation, and generates the weird behavior that you are seeing. The documentation itself states that this type of behavior is badly defined.
Bottom line: you cannot rely on transactions to handle concurrency when running an ALTER TABLE statement. As a consequence, you don't want to ALTER a table while it is busy on some other transaction: this is not safe. Before running such statement, you want to make sure that no lock is set on the table. Yes, that surely incurs painful constraints when dealing with production-like databases, but it looks like it's just how MySQL does it...
I'm using Ubuntu 16.04.3 and mysql Ver 14.14 Distrib 5.7.22.
When I try to alter my db table column with the following command the database appears to freeze;
mysql> alter table records modify column name varchar(150);
Table size is 2.8MiB (6,000 records).
I'm simply trying to change the varchar(150) part.
I have tried to do the same thing in GUI programs such as HeidiSQL and Workbench, same result - program crashes and I have to restart the session.
I am logged into the db and server as root. I checked the /var/log/mysql/error.log but it only displays failed login attempts, nothing relevant to this error.
Any ideas what's going on or how I can troubleshoot this?
Do you have any apps running long-running transactions against this table? If there are any transactions holding a metadata lock, it won't be able to start the ALTER TABLE.
If you can run SHOW PROCESSLIST in another window while the ALTER TABLE appears to hang, it might show "waiting for metadata lock".
+----+------+-----------+------+---------+------+---------------------------------+------------------------+-----------+---------------+
| Id | User | Host | db | Command | Time | State | Info | Rows_sent | Rows_examined |
+----+------+-----------+------+---------+------+---------------------------------+------------------------+-----------+---------------+
| 4 | root | localhost | test | Query | 0 | init | show processlist | 0 | 0 |
| 5 | root | localhost | test | Query | 15 | Waiting for table metadata lock | alter table foo... | 0 | 0 |
+----+------+-----------+------+---------+------+---------------------------------+------------------------+-----------+---------------+
Then you need to figure out which app has an open transaction that is blocking the ALTER TABLE, and get that app to finish its transaction, or else kill the transaction.
My not-so-big table hangs on an ALTER command. What could it be?
Only 150k rows, 42 fields 142 MByte total. InnoDB storage engine and Server version: 5.5.44-MariaDB MariaDB Server. 1 field, 'slotindex', is primary key: bigint(20) and BTREE type.
The command:
MariaDB [mydb]> ALTER TABLE `runs` CHANGE `p_w_trans_x` `p_w_tran_x` FLOAT NOT NULL;
Stage: 1 of 2 'copy to tmp table' 65.7% of stage done
Stage: 2 of 2 'Enabling keys' 0% of stage done
Will completely hang forever in this stage 2.
The processlist is then as follows:
MariaDB [(none)]> show full processlist;
+--------+------+-----------------+-----------+---------+-------+---------------------------------+---------------------------------------------------------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+--------+------+-----------------+-----------+---------+-------+---------------------------------+---------------------------------------------------------------------+----------+
| 274226 | root | localhost:45423 | edc_proxy | Sleep | 16043 | | NULL | 0.000 |
| 274319 | root | localhost | myDB | Query | 99 | Waiting for table metadata lock | ALTER TABLE `runs` CHANGE `p_w_trans_x` `p_w_tran_x` FLOAT NOT NULL | 0.000 |
| 274416 | root | localhost | NULL | Query | 0 | NULL | show full processlist | 0.000 |
+--------+------+-----------------+-----------+---------+-------+---------------------------------+---------------------------------------------------------------------+----------+
This answer suggests checking the information_schema tables, not much there:
MariaDB [INFORMATION_SCHEMA]> SELECT * FROM INNODB_LOCK_WAITS;
Empty set (0.00 sec)
MariaDB [INFORMATION_SCHEMA]> SELECT * FROM INNODB_LOCKS ;
Empty set (0.00 sec)
MariaDB [INFORMATION_SCHEMA]> SELECT * FROM INNODB_TRX;
+----------+-----------+---------------------+-----------------------+------------------+------------+---------------------+-----------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+
| trx_id | trx_state | trx_started | trx_requested_lock_id | trx_wait_started | trx_weight | trx_mysql_thread_id | trx_query | trx_operation_state | trx_tables_in_use | trx_tables_locked | trx_lock_structs | trx_lock_memory_bytes | trx_rows_locked | trx_rows_modified | trx_concurrency_tickets | trx_isolation_level | trx_unique_checks | trx_foreign_key_checks | trx_last_foreign_key_error | trx_adaptive_hash_latched | trx_adaptive_hash_timeout |
+----------+-----------+---------------------+-----------------------+------------------+------------+---------------------+-----------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+
| 83A8B36E | RUNNING | 2016-12-08 11:13:02 | NULL | NULL | 0 | 274226 | NULL | NULL | 0 | 0 | 0 | 376 | 0 | 0 | 0 | REPEATABLE READ | 1 | 1 | NULL | 0 | 10000 |
+----------+-----------+---------------------+-----------------------+------------------+------------+---------------------+-----------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+
1 row in set (0.00 sec)
And the section on transactions from show engine innodb status;:
------------
TRANSACTIONS
------------
Trx id counter 83A8F071
Purge done for trx's n:o < 83A8CA86 undo n:o < 0
History list length 1490
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0, not started
MySQL thread id 274543, OS thread handle 0x7fbb863e6700, query id 85356480 localhost root
show engine innodb status
---TRANSACTION 83A8EB07, not started
mysql tables in use 1, locked 2
MySQL thread id 274542, OS thread handle 0x7fbb843f6700, query id 85354935 localhost root Waiting for table metadata lock
ALTER TABLE `runs` CHANGE `p_w_trans_x` `p_w_tran_x` FLOAT NOT NULL
---TRANSACTION 83A8B36E, ACTIVE 24627 sec
MySQL thread id 274226, OS thread handle 0x7fbb845f5700, query id 85337236 localhost 127.0.0.1 root
Trx read view will not see trx with id >= 83A8B36F, sees < 83A8B36D
----------------------------
END OF INNODB MONITOR OUTPUT
============================
Any pointers for further investigation, for circumventing the problem and for solving are appreciated!
A metadata lock is an implicit (from the user perspective) lock that prevents DDL against the table because something else needs the table to remain in its current form. In this case, it's a transaction that has been left running.
Task 1: Your alter will succeed if you kill the connection on thread 274226.
mysql> KILL 274226;
The problem here, as indicated by information_schema.innodb_trx, is that this thread has left a transaction running for several hours and we can infer that this table has been referenced by that transaction. A table can't be altered until no transactioms still have an MVCC view or any locks involving the table. This transaction holds a view, which we can again infer could impact this table, as shown in the last line:
--TRANSACTION 83A8B36E, ACTIVE 24627 sec
MySQL thread id 274226, OS thread handle 0x7fbb845f5700, query id 85337236 localhost 127.0.0.1 root
Trx read view will not see trx with id >= 83A8B36F, sees < 83A8B36D
Note that Sleep is not a real command, in this context, it's just the placeholder status for any idle connection. All connections are doing something, and in this case the "something" is sleeping -- in other words, idle and waiting for another query. But an idle connection is still a connection, and if your code (or query browser tool) leaves a transaction running, it just keeps running.
Task 2: find the bug or mistake that left that transaction running. In a live application, leaving transactions running potentially make a much bigger mess.
I was having a very similar issue with table lock, but it ended up being MySQL Workbench out of all things. Any RENAME TABLE or ALTER TABLE command in MySQL Workbench would just sit claiming a meta lock. I logged onto the server and was able to execute those types of queries no problem. Current version of Workbench is 8.0.22.
I had sifted through all these queries and was totally stumped when they showed no issues:
SHOW OPEN TABLES;
SHOW ENGINE inndodb STATUS;
SELECT * FROM INNODB_LOCK_WAITS;
SELECT * FROM INNODB_LOCKS;
SELECT * FROM INNODB_TRX;
SHOW FULL PROCESSLIST;
I have mysql MyISAM table on which I am doing a simple select id from mytable limit 1;. This just freezes the system.
I tried explain select id from mytable limit 1;. Again it freezes my system. Table demographics: 50k records, 10 mbs size, 2 indexes (primary key autoincrement), 8 columns.
I am clueless why the explain statement failed, as it is supposed to display the query plan, nothing else. Neither the table size is enormous nor the number of records, then why is mysql working so slow? Rather, what am I missing here?
It was due a waiting state on mytable. eggyal gave me the clue to use show processlist. It showed:
+-----+---------+-----------------+----------------+---------+------+---------------------------------+----------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+---------+-----------------+----------------+---------+------+---------------------------------+----------------------------------------------------+
| 349 | root | localhost:56612 | mydb | Query | 3582 | Waiting for table metadata lock | ALTER TABLE `mytable` ADD INDEX(`fk_to_02`) |
I planted a kill 349 to terminate that wait chain, and now the explain statement works as expected.
I have a MySQL query that is copying data from one table to another for processing. For some reason, this query that normally takes a few seconds locked up overnight and ran for several hours. When I logged in this morning, I tried to kill the query, but it is still listed in the process list.
| Id | User | Host | db | Command | Time | State | Info |
+---------+----------+-----------+------+---------+-------+--------------+--------------------------------------------------------------------------------------+
| 1061763 | tb_admin | localhost | dw | Killed | 45299 | Sending data | INSERT INTO email_data_inno_stage SELECT * FROM email_data_test LIMIT 4480000, 10000 |
| 1062614 | tb_admin | localhost | dw | Killed | 863 | Sending data | INSERT INTO email_data_inno_stage SELECT * FROM email_data_test LIMIT 4480000, 10000 |
What could have caused this, and how can I kill this process so I can get on with my work?
If the table email_data_test is MyISAM and it was locked, that would have held up the the INSERT.
If the table email_data_test is InnoDB, then a lot of MVCC data was being written in ib_logfiles, which may not have occurred yet.
In both cases, you had the LIMIT clause scroll through 4,480,000 rows just to get to 10,000 rows you actually needed to INSERT.
Killing the query only causes the InnoDB table email_data_inno_stage to execute a rollback.