TRUNCATE table on MariaDB just started hanging - mysql

Question

I am running 10.1.26-MariaDB-0+deb9u1 Debian 9.1 in multiple locations.
Just got a call today that some scripts are no longer running at one of the locations. I've diagnosed that whenever a script tries to execute TRUNCATE <table name> it just hangs.
I've tried it from the CLI and Workbench as well with the same results. I have also tried TRUNCATE TABLE <table name> with the same results.
I cannot figure out A) why this all of a sudden stopped working. and B) what's different between this location and other three, where it does work.

Answer 1

I expect you see something like this:
mysql> show processlist;
+----+-----------------+-----------+------+---------+------+---------------------------------+------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+-----------------+-----------+------+---------+------+---------------------------------+------------------------+
| 8 | msandbox | localhost | test | Query | 435 | Waiting for table metadata lock | truncate table mytable |
Try this experiment in a test instance of MySQL (like on your local development environment): open two shell windows and run the mysql client. Create a test table.
mysql> create table test.mytable ( answer int );
mysql> insert into test.mytable set answer = 42;
Now start a transaction and query the table, but do not commit the transaction yet.
mysql> begin;
mysql> select * from test.mytable;
+--------+
| answer |
+--------+
| 42 |
+--------+
In the second window, try to truncate that table.
mysql> truncate table mytable;
<hangs>
What it's waiting for is a metadata lock. It will wait for a number of seconds equal to the lock_wait_timeout configuration option.
Now go back to the first window, and commit.
mysql> commit;
Now see in your second window, the TRUNCATE TABLE stops waiting, and it finally does its work, truncating the table.
Any DDL statement like ALTER TABLE, TRUNCATE TABLE, DROP TABLE needs to acquire an exclusive metadata lock on the table. But any transaction that has been reading or writing that table holds a shared metadata lock. This means many concurrent sessions can do their work, like SELECT/UPDATE/INSERT/DELETE without blocking each other (because their locks are shared). But a DDL statement requires an exclusive metadata lock, meaning no other metadata lock, either shared or exclusive, can exist.
So I'd guess there's some transaction hanging around that has done some read or write against your table, without committing. Either the query itself is very long-running, or else the query has finished but the transaction hasn't.
You have to figure out where you have an outstanding transaction. If you are using MySQL 5.7 or later, you can read the sys.schema_lock_waits table while one of your truncate table statements is waiting.
select * from sys.schema_table_lock_waits\G
*************************** 1. row ***************************
object_schema: test
object_name: mytable
waiting_thread_id: 47
waiting_pid: 8
waiting_account: msandbox#localhost
waiting_lock_type: EXCLUSIVE
waiting_lock_duration: TRANSACTION
waiting_query: truncate table mytable
waiting_query_secs: 625
waiting_query_rows_affected: 0
waiting_query_rows_examined: 0
blocking_thread_id: 48
blocking_pid: 9
blocking_account: msandbox#localhost
blocking_lock_type: SHARED_READ
blocking_lock_duration: TRANSACTION
sql_kill_blocking_query: KILL QUERY 9
sql_kill_blocking_connection: KILL 9
This tells us which session is blocked, waiting for a metadata lock. The waiting_pid (8 in the above example) corresponds to the Id in the processlist of the blocked session.
The blocking_pid (9 in the above example) corresponds to the Id in the processlist of the session that currently holds the lock, and which is blocking the truncate table.
It even tells you exactly how to kill the session that's holding the lock:
mysql> KILL 9;
Once the session is killed, it must release its locks, and the truncate table finally finishes.
mysql> truncate table mytable;
Query OK, 0 rows affected (13 min 34.50 sec)
Unfortunately, you're using MariaDB 10.1. This doesn't support the sys schema or the performance_schema.metadata_locks table that it needs to report those locks. MariaDB is a fork from MySQL 5.5, which is nearly ten years old now, and they didn't have the metadata_locks table at that time.
I don't use MariaDB, but I googled and found that they have their own proprietary implementation for querying metadata locks: https://mariadb.com/kb/en/library/metadata_lock_info/ I haven't used it, so I'll leave it to you to read the docs about that.