I'm using Ubuntu 16.04.3 and mysql Ver 14.14 Distrib 5.7.22.
When I try to alter my db table column with the following command the database appears to freeze;
mysql> alter table records modify column name varchar(150);
Table size is 2.8MiB (6,000 records).
I'm simply trying to change the varchar(150) part.
I have tried to do the same thing in GUI programs such as HeidiSQL and Workbench, same result - program crashes and I have to restart the session.
I am logged into the db and server as root. I checked the /var/log/mysql/error.log but it only displays failed login attempts, nothing relevant to this error.
Any ideas what's going on or how I can troubleshoot this?
Do you have any apps running long-running transactions against this table? If there are any transactions holding a metadata lock, it won't be able to start the ALTER TABLE.
If you can run SHOW PROCESSLIST in another window while the ALTER TABLE appears to hang, it might show "waiting for metadata lock".
+----+------+-----------+------+---------+------+---------------------------------+------------------------+-----------+---------------+
| Id | User | Host | db | Command | Time | State | Info | Rows_sent | Rows_examined |
+----+------+-----------+------+---------+------+---------------------------------+------------------------+-----------+---------------+
| 4 | root | localhost | test | Query | 0 | init | show processlist | 0 | 0 |
| 5 | root | localhost | test | Query | 15 | Waiting for table metadata lock | alter table foo... | 0 | 0 |
+----+------+-----------+------+---------+------+---------------------------------+------------------------+-----------+---------------+
Then you need to figure out which app has an open transaction that is blocking the ALTER TABLE, and get that app to finish its transaction, or else kill the transaction.
Related
Is it possible to find the process ID or the command that is executed for a rollback ?
More specifically :
The Information_Schema of MySQL has a specific table named processlist. This contains the details about every ongoing process in MySQL. An example of the table looks like this:
mysql> select * from processlist;
+----+------+-----------+--------------------+---------+------+------------+---------------------------+
| ID | USER | HOST | DB | COMMAND | TIME | STATE | INFO |
+----+------+-----------+--------------------+---------+------+------------+---------------------------+
| 5 | root | localhost | information_schema | Query | 5 | User sleep | select sleep(20) |
| 4 | root | localhost | information_schema | Query | 0 | executing | select * from processlist |
+----+------+-----------+--------------------+---------+------+------------+---------------------------+
Is it possible for me to intercept a Rollback and grab its ID or INFO from this table ? The issue I have been facing is that any Rollback I execute gets completed before I am able to intercept its ID from the Table.
My not-so-big table hangs on an ALTER command. What could it be?
Only 150k rows, 42 fields 142 MByte total. InnoDB storage engine and Server version: 5.5.44-MariaDB MariaDB Server. 1 field, 'slotindex', is primary key: bigint(20) and BTREE type.
The command:
MariaDB [mydb]> ALTER TABLE `runs` CHANGE `p_w_trans_x` `p_w_tran_x` FLOAT NOT NULL;
Stage: 1 of 2 'copy to tmp table' 65.7% of stage done
Stage: 2 of 2 'Enabling keys' 0% of stage done
Will completely hang forever in this stage 2.
The processlist is then as follows:
MariaDB [(none)]> show full processlist;
+--------+------+-----------------+-----------+---------+-------+---------------------------------+---------------------------------------------------------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+--------+------+-----------------+-----------+---------+-------+---------------------------------+---------------------------------------------------------------------+----------+
| 274226 | root | localhost:45423 | edc_proxy | Sleep | 16043 | | NULL | 0.000 |
| 274319 | root | localhost | myDB | Query | 99 | Waiting for table metadata lock | ALTER TABLE `runs` CHANGE `p_w_trans_x` `p_w_tran_x` FLOAT NOT NULL | 0.000 |
| 274416 | root | localhost | NULL | Query | 0 | NULL | show full processlist | 0.000 |
+--------+------+-----------------+-----------+---------+-------+---------------------------------+---------------------------------------------------------------------+----------+
This answer suggests checking the information_schema tables, not much there:
MariaDB [INFORMATION_SCHEMA]> SELECT * FROM INNODB_LOCK_WAITS;
Empty set (0.00 sec)
MariaDB [INFORMATION_SCHEMA]> SELECT * FROM INNODB_LOCKS ;
Empty set (0.00 sec)
MariaDB [INFORMATION_SCHEMA]> SELECT * FROM INNODB_TRX;
+----------+-----------+---------------------+-----------------------+------------------+------------+---------------------+-----------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+
| trx_id | trx_state | trx_started | trx_requested_lock_id | trx_wait_started | trx_weight | trx_mysql_thread_id | trx_query | trx_operation_state | trx_tables_in_use | trx_tables_locked | trx_lock_structs | trx_lock_memory_bytes | trx_rows_locked | trx_rows_modified | trx_concurrency_tickets | trx_isolation_level | trx_unique_checks | trx_foreign_key_checks | trx_last_foreign_key_error | trx_adaptive_hash_latched | trx_adaptive_hash_timeout |
+----------+-----------+---------------------+-----------------------+------------------+------------+---------------------+-----------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+
| 83A8B36E | RUNNING | 2016-12-08 11:13:02 | NULL | NULL | 0 | 274226 | NULL | NULL | 0 | 0 | 0 | 376 | 0 | 0 | 0 | REPEATABLE READ | 1 | 1 | NULL | 0 | 10000 |
+----------+-----------+---------------------+-----------------------+------------------+------------+---------------------+-----------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+
1 row in set (0.00 sec)
And the section on transactions from show engine innodb status;:
------------
TRANSACTIONS
------------
Trx id counter 83A8F071
Purge done for trx's n:o < 83A8CA86 undo n:o < 0
History list length 1490
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0, not started
MySQL thread id 274543, OS thread handle 0x7fbb863e6700, query id 85356480 localhost root
show engine innodb status
---TRANSACTION 83A8EB07, not started
mysql tables in use 1, locked 2
MySQL thread id 274542, OS thread handle 0x7fbb843f6700, query id 85354935 localhost root Waiting for table metadata lock
ALTER TABLE `runs` CHANGE `p_w_trans_x` `p_w_tran_x` FLOAT NOT NULL
---TRANSACTION 83A8B36E, ACTIVE 24627 sec
MySQL thread id 274226, OS thread handle 0x7fbb845f5700, query id 85337236 localhost 127.0.0.1 root
Trx read view will not see trx with id >= 83A8B36F, sees < 83A8B36D
----------------------------
END OF INNODB MONITOR OUTPUT
============================
Any pointers for further investigation, for circumventing the problem and for solving are appreciated!
A metadata lock is an implicit (from the user perspective) lock that prevents DDL against the table because something else needs the table to remain in its current form. In this case, it's a transaction that has been left running.
Task 1: Your alter will succeed if you kill the connection on thread 274226.
mysql> KILL 274226;
The problem here, as indicated by information_schema.innodb_trx, is that this thread has left a transaction running for several hours and we can infer that this table has been referenced by that transaction. A table can't be altered until no transactioms still have an MVCC view or any locks involving the table. This transaction holds a view, which we can again infer could impact this table, as shown in the last line:
--TRANSACTION 83A8B36E, ACTIVE 24627 sec
MySQL thread id 274226, OS thread handle 0x7fbb845f5700, query id 85337236 localhost 127.0.0.1 root
Trx read view will not see trx with id >= 83A8B36F, sees < 83A8B36D
Note that Sleep is not a real command, in this context, it's just the placeholder status for any idle connection. All connections are doing something, and in this case the "something" is sleeping -- in other words, idle and waiting for another query. But an idle connection is still a connection, and if your code (or query browser tool) leaves a transaction running, it just keeps running.
Task 2: find the bug or mistake that left that transaction running. In a live application, leaving transactions running potentially make a much bigger mess.
I was having a very similar issue with table lock, but it ended up being MySQL Workbench out of all things. Any RENAME TABLE or ALTER TABLE command in MySQL Workbench would just sit claiming a meta lock. I logged onto the server and was able to execute those types of queries no problem. Current version of Workbench is 8.0.22.
I had sifted through all these queries and was totally stumped when they showed no issues:
SHOW OPEN TABLES;
SHOW ENGINE inndodb STATUS;
SELECT * FROM INNODB_LOCK_WAITS;
SELECT * FROM INNODB_LOCKS;
SELECT * FROM INNODB_TRX;
SHOW FULL PROCESSLIST;
I have a thread showing in PHPmyadmin under processes. However, when I click kill, I get the error:
phpMyAdmin was unable to kill thread 148. It probably has already been closed.
Why does this thread still then show as active? How can I remove it entirely?
Open mysql client and type
mysql> show processlist;
+-----+------+-----------+------+---------+------+-------+------------------+-----------+---------------+-----------+
| Id | User | Host | db | Command | Time | State | Info | Rows_sent | Rows_examined | Rows_read |
+-----+------+-----------+------+---------+------+-------+------------------+-----------+---------------+-----------+
| 106 | root | localhost | NULL | Query | 0 | NULL | show processlist | 0 | 0 | 0 |
+-----+------+-----------+------+---------+------+-------+------------------+-----------+---------------+-----------+
1 row in set (0.00 sec)
you'll see processes with ID, than you can do this:
mysql> kill 106;
and your process (id = 106) will be killed.
Between the time that phpMyAdmin received the list of processes and the time you clicked to kill one of them, this process had finished by itself.
See also https://sourceforge.net/p/phpmyadmin/feature-requests/1490/.
This phenomenon is caused by the connection used to access PHPmyadmin itself, hence it doesn't show on the direct MySQLQuery. It can't be killed, as it would close the PHPmyadmin connection.
My debian-based is booting so slow after I installed MySQL and imported some databases on it. Looking for some statement, I found this one during boot:
mysql> show full processlist;
+----+------------------+-----------+------+---------+------+----------------+----------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+------------------+-----------+------+---------+------+----------------+----------------------------------------------------------------------+
| 9 | debian-sys-maint | localhost | NULL | Query | 12 | Opening tables | select count(*) into #discard from `information_schema`.`PARTITIONS` |
| 10 | root | localhost | NULL | Query | 0 | NULL | show full processlist |
+----+------------------+-----------+------+---------+------+----------------+----------------------------------------------------------------------+
2 rows in set (0.00 sec)
Here the statement that causing trouble:
select count(*) into #discard from `information_schema`.`PARTITIONS`
I have +-10 databases totaling over 8gb of data.
Is there any configuration to disable this query on system booting ? If yes, why run it during boot ?
Information
I have a standard MySQL installation without custom configs.
Best regards.
It seems Debian, whose Linux Mint is based upon, have scripts that get executed when the mysql server is started or restarted, to check for corrupted tables and make an alert for that.
In my Debian server, the culprit seems to be /etc/mysql/debian-start bash script, which in turn calls /usr/share/mysql/debian-start.inc.sh , so check both scripts and comment out the function that is iterating all your tables, from a quick look it seems the following:
check_for_crashed_tables;
which is called from the debian-start script I mentioned above.
I have a MySQL query that is copying data from one table to another for processing. For some reason, this query that normally takes a few seconds locked up overnight and ran for several hours. When I logged in this morning, I tried to kill the query, but it is still listed in the process list.
| Id | User | Host | db | Command | Time | State | Info |
+---------+----------+-----------+------+---------+-------+--------------+--------------------------------------------------------------------------------------+
| 1061763 | tb_admin | localhost | dw | Killed | 45299 | Sending data | INSERT INTO email_data_inno_stage SELECT * FROM email_data_test LIMIT 4480000, 10000 |
| 1062614 | tb_admin | localhost | dw | Killed | 863 | Sending data | INSERT INTO email_data_inno_stage SELECT * FROM email_data_test LIMIT 4480000, 10000 |
What could have caused this, and how can I kill this process so I can get on with my work?
If the table email_data_test is MyISAM and it was locked, that would have held up the the INSERT.
If the table email_data_test is InnoDB, then a lot of MVCC data was being written in ib_logfiles, which may not have occurred yet.
In both cases, you had the LIMIT clause scroll through 4,480,000 rows just to get to 10,000 rows you actually needed to INSERT.
Killing the query only causes the InnoDB table email_data_inno_stage to execute a rollback.