We are running mariadb 10.3.25:
$ mysql --version
mysql Ver 15.1 Distrib 10.3.25-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
It seems that there is some sort of corruption in some of our databases’ tables.
Exhibit A:
MariaDB [etherpad]> select * from store;
ERROR 2013 (HY000): Lost connection to MySQL server during query
In the meantime, this happens in the log:
Jan 16 19:51:52 hostname mysqld[31236]: 2021-01-16 19:51:52 0x7f0c884b8700 InnoDB: Assertion failure in file /build/mariadb-10.3-RRxkin/mariadb-10.3-10.3.25/storage/innobase/row/row0sel.cc line 2972
Jan 16 19:51:52 hostname mysqld[31236]: InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
Jan 16 19:51:52 hostname mysqld[31236]: InnoDB: We intentionally generate a memory trap.
Jan 16 19:51:52 hostname mysqld[31236]: InnoDB: [...]
Jan 16 19:51:52 hostname mysqld[31236]: 210116 19:51:52 [ERROR] mysqld got signal 6 ;
Jan 16 19:51:52 hostname mysqld[31236]: This could be because you hit a bug. It is also possible that this binary
Jan 16 19:51:52 hostname mysqld[31236]: or one of the libraries it was linked against is corrupt, improperly built,
Jan 16 19:51:52 hostname mysqld[31236]: or misconfigured. This error can also be caused by malfunctioning hardware.
Jan 16 19:51:52 hostname mysqld[31236]: [...]
Jan 16 19:51:52 hostname mysqld[31236]: We will try our best to scrape up some info that will hopefully help
Jan 16 19:51:52 hostname mysqld[31236]: diagnose the problem, but since we have already crashed,
Jan 16 19:51:52 hostname mysqld[31236]: something is definitely wrong and this may fail.
Jan 16 19:51:52 hostname mysqld[31236]: Server version: 10.3.25-MariaDB-0+deb10u1-log
Jan 16 19:51:52 hostname mysqld[31236]: key_buffer_size=16777216
Jan 16 19:51:52 hostname mysqld[31236]: read_buffer_size=131072
Jan 16 19:51:52 hostname mysqld[31236]: key_buffer_size=16777216 [55/647]
Jan 16 19:51:52 hostname mysqld[31236]: read_buffer_size=131072
Jan 16 19:51:52 hostname mysqld[31236]: max_used_connections=16
Jan 16 19:51:52 hostname mysqld[31236]: max_threads=153
Jan 16 19:51:52 hostname mysqld[31236]: thread_count=22
Jan 16 19:51:52 hostname mysqld[31236]: It is possible that mysqld could use up to
Jan 16 19:51:52 hostname mysqld[31236]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 352736 K bytes of memory
Jan 16 19:51:52 hostname mysqld[31236]: Hope that's ok; if not, decrease some variables in the equation.
Jan 16 19:51:52 hostname mysqld[31236]: Thread pointer: 0x7f0c500093b8
Jan 16 19:51:52 hostname mysqld[31236]: Attempting backtrace. You can use the following information to find out
Jan 16 19:51:52 hostname mysqld[31236]: where mysqld died. If you see no messages after this, something went
Jan 16 19:51:52 hostname mysqld[31236]: terribly wrong...
Jan 16 19:51:52 hostname mysqld[31236]: stack_bottom = 0x7f0c884b7dd8 thread_stack 0x30000
Jan 16 19:51:52 hostname mysqld[31236]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x563337b2b05e]
Jan 16 19:51:52 hostname mysqld[31236]: /usr/sbin/mysqld(handle_fatal_signal+0x54d)[0x56333765e09d]
Jan 16 19:51:53 hostname mysqld[31236]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f0c91ef1730]
Jan 16 19:51:53 hostname mysqld[31236]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7f0c914ae7bb]
Jan 16 19:51:53 hostname mysqld[31236]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7f0c91499535]
Jan 16 19:51:53 hostname mysqld[31236]: /usr/sbin/mysqld(+0x4e3433)[0x5633373a2433]
Jan 16 19:51:53 hostname mysqld[31236]: /usr/sbin/mysqld(+0x4d5d6c)[0x563337394d6c]
Jan 16 19:51:53 hostname mysqld[31236]: /usr/sbin/mysqld(+0x9d8814)[0x563337897814]
Jan 16 19:51:53 hostname mysqld[31236]: /usr/sbin/mysqld(+0x9dcdcf)[0x56333789bdcf]
Jan 16 19:51:53 hostname mysqld[31236]: /usr/sbin/mysqld(+0x918681)[0x5633377d7681]
Jan 16 19:51:53 hostname mysqld[31236]: /usr/sbin/mysqld(_ZN7handler11ha_rnd_nextEPh+0x127)[0x563337662db7]
Jan 16 19:51:53 hostname mysqld[31236]: /usr/sbin/mysqld(_Z13rr_sequentialP11READ_RECORD+0x1c)[0x56333776a43c]
Jan 16 19:51:53 hostname mysqld[31236]: /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1e3)[0x5633374bdf03]
Jan 16 19:51:53 hostname mysqld[31236]: /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xaaa)[0x5633374e01ba]
Jan 16 19:51:53 hostname mysqld[31236]: /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x5633374e03d3]
Jan 16 19:51:53 hostname mysqld[31236]: /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xef)[0x5633374deaaf]
Jan 16 19:51:53 hostname mysqld[31236]: /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x14d)[0x5633374df38d]
Jan 16 19:51:53 hostname mysqld[31236]: /usr/sbin/mysqld(+0x5c1d8c)[0x563337480d8c]
Jan 16 19:51:53 hostname mysqld[31236]: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x5857)[0x56333748d087]
Jan 16 19:51:53 hostname mysqld[31236]: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x1c9)[0x56333748f879]
Jan 16 19:51:54 hostname mysqld[31236]: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x111d)[0x56333749172d]
Jan 16 19:51:54 hostname mysqld[31236]: /usr/sbin/mysqld(_Z10do_commandP3THD+0x122)[0x563337492e82]
Jan 16 19:51:54 hostname mysqld[31236]: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x23a)[0x5633375641ba]
Jan 16 19:51:54 hostname mysqld[31236]: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x56333756433d]
Jan 16 19:51:55 hostname mysqld[31236]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f0c91ee6fa3]
Jan 16 19:51:55 hostname mysqld[31236]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f0c915704cf]
Jan 16 19:51:55 hostname mysqld[31236]: Trying to get some variables.
Jan 16 19:51:55 hostname mysqld[31236]: Some pointers may be invalid and cause the dump to abort.
Jan 16 19:51:55 hostname mysqld[31236]: Query (0x7f0c50012e20): select * from store
Jan 16 19:51:55 hostname mysqld[31236]: Connection ID (thread ID): 733
Jan 16 19:51:55 hostname mysqld[31236]: Status: NOT_KILLED
Jan 16 19:51:55 hostname mysqld[31236]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,de
rived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_joi
n_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_d
erived=on,split_materialized=on
Jan 16 19:51:55 hostname mysqld[31236]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
Jan 16 19:51:55 hostname mysqld[31236]: information that should help you find out what is causing the crash.
Jan 16 19:51:55 hostname mysqld[31236]: Writing a core file...
Jan 16 19:51:55 hostname mysqld[31236]: Working directory at /var/lib/mysql
Jan 16 19:51:55 hostname mysqld[31236]: Resource Limits:
Jan 16 19:51:55 hostname mysqld[31236]: Limit Soft Limit Hard Limit Units
Jan 16 19:51:55 hostname mysqld[31236]: Max cpu time unlimited unlimited seconds
Jan 16 19:51:55 hostname mysqld[31236]: Max file size unlimited unlimited bytes
Jan 16 19:51:55 hostname mysqld[31236]: Max data size unlimited unlimited bytes
Jan 16 19:51:55 hostname mysqld[31236]: Max file size unlimited unlimited bytes [0/647]
Jan 16 19:51:55 hostname mysqld[31236]: Max data size unlimited unlimited bytes
Jan 16 19:51:55 hostname mysqld[31236]: Max stack size 8388608 unlimited bytes
Jan 16 19:51:55 hostname mysqld[31236]: Max core file size 0 unlimited bytes
Jan 16 19:51:55 hostname mysqld[31236]: Max resident set unlimited unlimited bytes
Jan 16 19:51:55 hostname mysqld[31236]: Max processes 15390 15390 processes
Jan 16 19:51:55 hostname mysqld[31236]: Max open files 65536 65536 files
Jan 16 19:51:55 hostname mysqld[31236]: Max locked memory 65536 65536 bytes
Jan 16 19:51:55 hostname mysqld[31236]: Max address space unlimited unlimited bytes
Jan 16 19:51:55 hostname mysqld[31236]: Max file locks unlimited unlimited locks
Jan 16 19:51:55 hostname mysqld[31236]: Max pending signals 15390 15390 signals
Jan 16 19:51:55 hostname mysqld[31236]: Max msgqueue size 819200 819200 bytes
Jan 16 19:51:55 hostname mysqld[31236]: Max nice priority 0 0
Jan 16 19:51:55 hostname mysqld[31236]: Max realtime priority 0 0
Jan 16 19:51:55 hostname mysqld[31236]: Max realtime timeout unlimited unlimited us
Jan 16 19:51:55 hostname mysqld[31236]: Core pattern: core
Jan 16 19:52:02 hostname mysqld[6672]: [... innodb crash recovery ...]
A very similar thing happens with some other tables as well.
What I tried:
I wanted to dump all data, purge the entire mariadb installation and restore. Unsurprisingly, mysqldump runs into the same corruption (?) and the database crashes during the dump.
I tried following a guide that advises to create a MyISAM table and fill that with data from the innodb table, but this fails for the same reason.
What can be done about this? Naturally, we need the data in these tables. It appears that once the query hits a certain record/block (I am oblivious to inner workings of mysql) it crashes the server. So how do we salvage the data?
UPDATE 2021-01-18 as requested, here are the variables and status queries:
MariaDB [(none)]> show global variables like '%thread%';
+-----------------------------------------+---------------------------+
| Variable_name | Value |
+-----------------------------------------+---------------------------+
| aria_repair_threads | 1 |
| binlog_optimize_thread_scheduling | ON |
| debug_no_thread_alarm | OFF |
| innodb_encryption_threads | 0 |
| innodb_purge_threads | 4 |
| innodb_read_io_threads | 4 |
| innodb_thread_concurrency | 0 |
| innodb_thread_sleep_delay | 10000 |
| innodb_write_io_threads | 4 |
| max_delayed_threads | 20 |
| max_insert_delayed_threads | 20 |
| myisam_repair_threads | 1 |
| performance_schema_max_thread_classes | 50 |
| performance_schema_max_thread_instances | -1 |
| slave_domain_parallel_threads | 0 |
| slave_parallel_threads | 0 |
| thread_cache_size | 8 |
| thread_concurrency | 10 |
| thread_handling | one-thread-per-connection |
| thread_pool_idle_timeout | 60 |
| thread_pool_max_threads | 65536 |
| thread_pool_oversubscribe | 3 |
| thread_pool_prio_kickup_timer | 1000 |
| thread_pool_priority | auto |
| thread_pool_size | 1 |
| thread_pool_stall_limit | 500 |
| thread_stack | 196608 |
| wsrep_slave_threads | 1 |
+-----------------------------------------+---------------------------+
28 rows in set (0.001 sec)
MariaDB [(none)]> show global status like '%thread%';
+------------------------------------------+-------+
| Variable_name | Value |
+------------------------------------------+-------+
| Delayed_insert_threads | 0 |
| Performance_schema_thread_classes_lost | 0 |
| Performance_schema_thread_instances_lost | 0 |
| Slow_launch_threads | 0 |
| Threadpool_idle_threads | 0 |
| Threadpool_threads | 0 |
| Threads_cached | 7 |
| Threads_connected | 12 |
| Threads_created | 98 |
| Threads_running | 6 |
| wsrep_applier_thread_count | 0 |
| wsrep_rollbacker_thread_count | 0 |
| wsrep_thread_count | 0 |
+------------------------------------------+-------+
13 rows in set (0.001 sec)
MariaDB [(none)]> show global variables like '%timeout%';
+---------------------------------------+----------+
| Variable_name | Value |
+---------------------------------------+----------+
| connect_timeout | 10 |
| deadlock_timeout_long | 50000000 |
| deadlock_timeout_short | 10000 |
| delayed_insert_timeout | 300 |
| idle_readonly_transaction_timeout | 0 |
| idle_transaction_timeout | 0 |
| idle_write_transaction_timeout | 0 |
| innodb_flush_log_at_timeout | 1 |
| innodb_lock_wait_timeout | 50 |
| innodb_rollback_on_timeout | OFF |
| interactive_timeout | 28800 |
| lock_wait_timeout | 86400 |
| net_read_timeout | 600 |
| net_write_timeout | 600 |
| rpl_semi_sync_master_timeout | 10000 |
| rpl_semi_sync_slave_kill_conn_timeout | 5 |
| slave_net_timeout | 60 |
| thread_pool_idle_timeout | 60 |
| wait_timeout | 28800 |
+---------------------------------------+----------+
19 rows in set (0.001 sec)
MariaDB [(none)]> show global status like '%timeout%';
+-------------------------------------+-------+
| Variable_name | Value |
+-------------------------------------+-------+
| Binlog_group_commit_trigger_timeout | 0 |
| Master_gtid_wait_timeouts | 0 |
| Ssl_default_timeout | 0 |
| Ssl_session_cache_timeouts | 0 |
+-------------------------------------+-------+
4 rows in set (0.001 sec)
MariaDB [(none)]> show global status like '%aborted%';
+------------------+-------+
| Variable_name | Value |
+------------------+-------+
| Aborted_clients | 3 |
| Aborted_connects | 0 |
+------------------+-------+
2 rows in set (0.001 sec)
The server has 5 GB of RAM.
About the store table:
MariaDB [etherpad]> show create table store;
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| store | CREATE TABLE `store` (
`key` varchar(100) COLLATE utf8_bin NOT NULL DEFAULT '',
`value` longtext COLLATE utf8_bin NOT NULL,
PRIMARY KEY (`key`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.000 sec)
MariaDB [etherpad]> select count(*) from store;
+----------+
| count(*) |
+----------+
| 779443 |
+----------+
1 row in set (1 min 19.378 sec)
Here is the iostat info:
$ iostat -xm 5 3
Linux 4.14.0-0.bpo.3-amd64 (hostname) 01/18/2021 _x86_64_ (1 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
8.63 2.39 16.53 22.68 0.23 49.54
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
xvdap2 7.67 37.91 0.07 0.67 0.13 37.91 1.67 50.00 16.35 2.54 0.05 9.40 18.01 4.35 19.82
xvdap1 0.51 1.25 0.00 0.01 0.02 0.07 3.58 5.64 7.52 27.01 0.03 4.15 4.24 1.63 0.29
avg-cpu: %user %nice %system %iowait %steal %idle
18.51 2.21 15.49 55.33 0.40 8.05
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
xvdap2 4.00 157.80 0.02 1.53 0.00 71.00 0.00 31.03 5.80 55.33 7.93 4.00 9.92 4.37 70.72
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
8.96 2.44 15.68 15.27 0.41 57.23
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
xvdap2 0.00 22.20 0.00 0.40 0.00 35.40 0.00 61.46 0.00 22.81 0.30 0.00 18.27 4.11 9.12
xvdap1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
UPDATE 2021-01-24: I tried to pinpoint the problem by quasy-bisecting the table with limit clauses and found that out of the ~800,000 records, every query that selects records after 663,187 crashes the DB. The few records preceding the 663,187. one contain seemingly mangled data, see below.
MariaDB [etherpad]> select * from store limit 663184, 1\G;
*************************** 1. row ***************************
key:
value:
f[Y
f[팩
Does this not hint at data corruption? What could I do about the problem? Get rid of these records?
From the information available at this time, consider in your my.cnf [mysqld] section
innodb_buffer_pool_size=2G # to use 40% of available RAM
REMOVE thread_cache_size to allow default sizing (or set it to 256)
REMOVE thread_stack to allow default calc of slightly larger thread_stack per ref man
When we know number of cores/cpus we may be able to provide additional suggestions.
When more than 1 cpu's are available, additional suggestions could be provided.
Your query: select count(*) from store;
may complete in less than a minute if your tried
SELECT COUNT(key) FROM store;
to only cause reading of the index, rather than every row.
Have a GREAT 2021.
Related
The I/O wait time is increasing for one of my servers. I haven't seen any long running queries in the process list. Can someone help me identify the cause?
The server is running Cent OS Linux 7.7, 4 core CPU , MySQL version 5.6.42
iotop output
Actual DISK READ: 57.85 M/s | Actual DISK WRITE: 74.90 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
31529 be/4 mysql 58.08 M/s 0.00 B/s 0.00 % 84.45 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib~p.geolearning.com.pid --socket=/var/lib/mysql/mysql.sock --port=3306
12331 be/4 mysql 7.68 K/s 61.46 K/s 0.00 % 0.66 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib~p.geolearning.com.pid --socket=/var/lib/mysql/mysql.sock --port=3306
6819 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.02 % [kworker/2:16]
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % systemd --switched-root --system --deserialize 22
2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
4 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H]
6 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0]
7 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0]
8 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_bh]
9 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_sched]
10 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [lru-add-drain]
Processlist:
+---------+----------------+------------------+---------+---------+------+--------------+------------------------------------------------------------------------------------------------------+-----------+---------------+
| Id | User | Host | db | Command | Time | State | Info | Rows_sent | Rows_examined |
+---------+----------------+------------------+---------+---------+------+--------------+------------------------------------------------------------------------------------------------------+-----------+---------------+
| 2128874 | lpaauser | 10.7.40.7:64197 | prodsup | Query | 0 | Sending data | REPLACE INTO course_auto_assignments_compute8 (domain_id,lms_user_id,filter_set_id)
S | 0 | 0 |
| 2129350 | mveeranki | localhost | | Query | 0 | init | show processlist | 0 | 0 |
+---------+----------------+------------------+---------+---------+------+--------------+------------------------------------------------------------------------------------------------------+-----
I have a question about optimize table of mysql,I have read many documents about this function,all of the document says：it can defragment an InnoDB table and reorganizes the physical storage of table data and associated index data, to reduce storage space,but no one can tell me the principle of optimize table and slove my questions,I have known those informations:
As of MySQL 5.6.17,OPTIMIZE TABLE uses online DDL for regular and partitioned InnoDB tables
,As of MySQL 5.6.17,The table rebuild triggered by OPTIMIZE TABLE and performed under the cover by ALTER TABLE ... FORCE is completed in place(ALGORITHM =inplace)
my questions:
does the optimize table of mysql really rebuild(recreate) table？ because it completed in place(not ALGORITHM=copy),is ALGORITHM=inplace real rebuild(recreate) table?
thank you for your response.
https://dev.mysql.com/doc/refman/5.6/en/innodb-online-ddl-operations.html
enter image description here
Demo:
I created a table on MySQL 5.6.37 and filled it with about 1GB of random data.
mysql> select version();
+-----------------+
| version() |
+-----------------+
| 5.6.37-82.2-log |
+-----------------+
mysql> show table status\G
*************************** 1. row ***************************
Name: mytable
Engine: InnoDB
Version: 10
Row_format: Compact
Rows: 61914
Avg_row_length: 20926
Data_length: 1295613952
Max_data_length: 0
Index_length: 1589248
Data_free: 4194304
Auto_increment: 131056
Create_time: 2019-02-20 15:54:23
Update_time: NULL
Check_time: NULL
Collation: utf8mb4_general_ci
Checksum: NULL
Create_options:
Comment:
I can observe the file in my datadir and note the inode number (the leftmost number).
/usr/local/var/mysql/test ls -li
total 2590752
8625784850 -rw-rw---- 1 bkarwin admin 67 Feb 20 07:54 db.opt
8625784853 -rw-rw---- 1 bkarwin admin 8580 Feb 20 07:54 mytable.frm
8625784854 -rw-rw---- 1 bkarwin admin 1321205760 Feb 20 07:56 mytable.ibd
I run optimize table.
mysql> optimize table mytable;
While it's running, I check progress by listing files in the datadir again. I see it has created a new temp file and it's filling it with a copy of the data.
Note the inode number of the temp file.
/usr/local/var/mysql/test ls -li
total 4395064
8625785082 -rw-rw---- 1 bkarwin admin 8580 Feb 20 07:57 #sql-337_16.frm
8625785083 -rw-rw---- 1 bkarwin admin 914358272 Feb 20 07:57 #sql-ib2021-3689447301.ibd
8625784850 -rw-rw---- 1 bkarwin admin 67 Feb 20 07:54 db.opt
8625784853 -rw-rw---- 1 bkarwin admin 8580 Feb 20 07:54 mytable.frm
8625784854 -rw-rw---- 1 bkarwin admin 1321205760 Feb 20 07:56 mytable.ibd
Finally my optimize table finishes. It took almost 2 minutes. If it wasn't doing a real table copy, why would it have taken so long?
+--------------+----------+----------+-------------------------------------------------------------------+
| Table | Op | Msg_type | Msg_text |
+--------------+----------+----------+-------------------------------------------------------------------+
| test.mytable | optimize | note | Table does not support optimize, doing recreate + analyze instead |
| test.mytable | optimize | status | OK |
+--------------+----------+----------+-------------------------------------------------------------------+
2 rows in set (1 min 5.52 sec)
Looking at the datadir again, I see the temp file has been renamed to be the new tablespace, and the old file has been removed. The new file has the inode number of the temp file, indicating it's the copy.
/usr/local/var/mysql/test ls -li
total 2557984
8625784850 -rw-rw---- 1 bkarwin admin 67 Feb 20 07:54 db.opt
8625785082 -rw-rw---- 1 bkarwin admin 8580 Feb 20 07:57 mytable.frm
8625785083 -rw-rw---- 1 bkarwin admin 1304428544 Feb 20 07:58 mytable.ibd
I have a lot of small binary log files , each of them just only 126 Bytes
-rw-rw----. 1 mysql mysql 126 Jan 6 16:13 mysql-bin.000001
-rw-rw----. 1 mysql mysql 126 Jan 6 16:16 mysql-bin.000002
-rw-rw----. 1 mysql mysql 150 Jan 7 10:34 mysql-bin.000003
-rw-rw----. 1 mysql mysql 126 Jan 8 09:57 mysql-bin.000004
-rw-rw----. 1 mysql mysql 126 Jan 8 11:08 mysql-bin.000005
-rw-rw----. 1 mysql mysql 107 Jan 8 11:09 mysql-bin.000006
-rw-rw----. 1 mysql mysql 126 Jan 8 13:21 mysql-bin.000007
-rw-rw----. 1 mysql mysql 342 Jan 8 16:25 mysql-bin.index
how to let them grow bigger? and my "binlog_cache_size" is more than 126 Byte
mysql> show variables like 'binlog_cache_size';
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id: 1
Current database: test
+-------------------+-------+
| Variable_name | Value |
+-------------------+-------+
| binlog_cache_size | 32768 |
+-------------------+-------+
1 row in set (0.00 sec)
can anybody tell me why(⊙ˍ⊙) ? thank you very much :)
My client has a server thats been running solidly for 2 years. He called yesterday b.c he noticed the server is down (it was down for a couple of days actually). I'm trying to figure out whats going on. I can't seem to start mysql.
When I run: /etc/init.d/mysqld start it fails. And running mysql results in
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
I tried tweaking the settings in /etc/my.cnf to change the socket to be in the tmp directory but that didn't work either. I think there's something else going on. Perhaps a memory issue? Or is something corrupt?
Below is my mysqld.log file.
140108 05:14:51 mysqld started
InnoDB: Log scan progressed past the checkpoint lsn 362 2789208032
140108 5:14:51 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 362 2789874367
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 1637 row operations to undo
InnoDB: Trx id counter is 0 138804224
140108 5:14:51 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Error: tried to read 344064 bytes at offset 6 41517056.
InnoDB: Was only able to read 339968.
140108 5:15:08 InnoDB: Operating system error number 0 in a file operation.
InnoDB: Error number 0 means 'Success'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/operating-system-error-codes.html
InnoDB: File operation call: 'read'.
InnoDB: Cannot continue operation.
140108 05:15:08 mysqld ended
My /var/lib/mysql directory looks like this
total 27654604
drwxr-xr-x 8 mysql mysql 4096 Jan 8 05:14 .
drwxr-xr-x 28 root root 4096 May 29 2012 ..
drwxr-xr-x 2 mysql mysql 24576 Apr 5 2013 db_stackme
drwxr-xr-x 2 mysql mysql 4096 May 30 2012 db_stackmeuser
-rwxr-xr-x 1 mysql mysql 28280094720 Dec 20 09:54 ibdata1
-rwxr-xr-x 1 mysql mysql 5242880 Jan 8 05:14 ib_logfile0
-rwxr-xr-x 1 mysql mysql 5242880 Dec 20 09:51 ib_logfile1
drwxr-xr-x 2 mysql mysql 4096 May 30 2012 mysql
drwxr-xr-x 2 mysql mysql 8192 May 30 2012 openx
drwxr-xr-x 2 mysql mysql 4096 Jun 1 2012 stackme_business
drwxr-xr-x 2 mysql mysql 4096 May 30 2012 test
I would GREATLY appreciate some input on how to solve this problem.
I'm using InnoDB in MySQL (0.5GB database) on Debian 6.0.
This morning (after 120 days of perfect uptime), suddenly the server hung (in fact the server hung a second time in the last two days) and I had to reboot it through hoster CP.
In kern.log, I've got this:
Apr 16 09:43:35 lhost kernel: [76680.208104] INFO: task mysqld:9755 blocked for more than 120 seconds.
Apr 16 09:43:35 lhost kernel: [76680.233913] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 16 09:43:35 lhost kernel: [76680.234249] mysqld D ffff88007dc67170 0 9755 1241 0x00000000
Apr 16 09:43:35 lhost kernel: [76680.234472] ffff88007dc67170 0000000000000086 0000000000000000 ffffffffa021c4c2
Apr 16 09:43:35 lhost kernel: [76680.234815] ffff880009ff1ad8 ffff880009ff1ad8 000000000000f9e0 ffff880009ff1fd8
Apr 16 09:43:35 lhost kernel: [76680.235181] 0000000000015780 0000000000015780 ffff88007d177170 ffff88007d177468
Apr 16 09:43:35 lhost kernel: [76680.235549] Call Trace:
Apr 16 09:43:35 lhost kernel: [76680.235732] [<ffffffffa021c4c2>] ? ext4_mark_iloc_dirty+0x46e/0x4f7 [ext4]
Apr 16 09:43:35 lhost kernel: [76680.235963] [<ffffffffa021cd61>] ? ext4_mark_inode_dirty+0x18f/0x1c4 [ext4]
Apr 16 09:43:35 lhost kernel: [76680.236212] [<ffffffff812fc2ab>] ? __mutex_lock_common+0x122/0x192
Apr 16 09:43:35 lhost kernel: [76680.236422] [<ffffffff812fc3d3>] ? mutex_lock+0x1a/0x31
Apr 16 09:43:35 lhost kernel: [76680.236626] [<ffffffff81113c8c>] ? __blockdev_direct_IO+0x1d3/0xa60
Apr 16 09:43:35 lhost kernel: [76680.236852] [<ffffffff810b51c2>] ? generic_file_buffered_write+0x1f5/0x278
Apr 16 09:43:35 lhost kernel: [76680.237097] [<ffffffffa021da26>] ? ext4_ind_direct_IO+0x10e/0x1be [ext4]
Apr 16 09:43:35 lhost kernel: [76680.237320] [<ffffffffa02205c9>] ? ext4_get_block+0x0/0xe2 [ext4]
Apr 16 09:43:35 lhost kernel: [76680.237528] [<ffffffff810b57bf>] ? generic_file_aio_read+0xf6/0x536
Apr 16 09:43:35 lhost kernel: [76680.237794] [<ffffffff81071dbf>] ? wake_futex+0x31/0x4e
Apr 16 09:43:35 lhost kernel: [76680.237984] [<ffffffff810ef061>] ? do_sync_read+0xce/0x113
Apr 16 09:43:35 lhost kernel: [76680.238278] [<ffffffff8100f79c>] ? __switch_to+0x285/0x297
Apr 16 09:43:35 lhost kernel: [76680.238471] [<ffffffff81065046>] ? autoremove_wake_function+0x0/0x2e
Apr 16 09:43:35 lhost kernel: [76680.238687] [<ffffffff81048282>] ? finish_task_switch+0x3a/0xaf
Apr 16 09:43:35 lhost kernel: [76680.238897] [<ffffffff810efa84>] ? vfs_read+0xa6/0xff
Apr 16 09:43:35 lhost kernel: [76680.239090] [<ffffffff810efb34>] ? sys_pread64+0x57/0x77
Apr 16 09:43:35 lhost kernel: [76680.239290] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
And just before complete blackout:
Apr 16 09:45:32 lhost kernel: [76800.253284] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
Apr 16 09:56:29 lhost kernel: [77453.643191] Clocksource tsc unstable (delta = -788398275 ns)
Apr 16 10:01:01 lhost kernel: imklog 4.6.4, log source = /proc/kmsg started.
So I decided to turn the slow queries log on, and this is what was logged:
# Time: 130416 13:38:41
# User#Host: lhost[lhost] # localhost []
# Query_time: 2.059521 Lock_time: 0.000026 Rows_sent: 0 Rows_examined: 1
use lhost;
SET timestamp=1366105121;
UPDATE t_fuser SET DATE_UPDATE=now() WHERE ID = 992277;
# Time: 130416 13:56:42
# User#Host: root[root] # localhost []
# Query_time: 3.663760 Lock_time: 0.000037 Rows_sent: 230 Rows_examined: 230
SET timestamp=1366106202;
SHOW TABLE STATUS FROM `lhost`;
# User#Host: root[root] # localhost []
# Query_time: 3.693291 Lock_time: 0.000031 Rows_sent: 230 Rows_examined: 230
SET timestamp=1366106202;
SHOW TABLE STATUS FROM `lhost`;
# Time: 130416 14:00:19
# User#Host: lhost[lhost] # localhost []
# Query_time: 7.139369 Lock_time: 0.000210 Rows_sent: 122 Rows_examined: 3027170
This looks really weird to me. I've tried to execute similar queries (there's more complicated ones on this log, but that's not the point) directly through the console and got results much faster, so it's not indexes.
What can I do to address this? Is it hardware?
# free -m
total used free shared buffers cached
Mem: 2012 1355 657 0 40 627
-/+ buffers/cache: 687 1325
Swap: 2047 129 1918
Thank in advance for your advice.
Additional data:
PROFILE of slow query:
mysql> show profile;
+--------------------------------+----------+
| Status | Duration |
+--------------------------------+----------+
| starting | 0.000020 |
| checking query cache for query | 0.000111 |
| checking permissions | 0.000003 |
| checking permissions | 0.000001 |
| checking permissions | 0.000002 |
| checking permissions | 0.000001 |
| checking permissions | 0.000003 |
| Opening tables | 0.000032 |
| System lock | 0.000004 |
| Table lock | 0.000006 |
| init | 0.000074 |
| optimizing | 0.000020 |
| statistics | 0.000171 |
| preparing | 0.000027 |
| Creating tmp table | 0.000160 |
| executing | 0.000003 |
| Copying to tmp table | 4.572503 |
| Creating sort index | 0.009622 |
| Copying to group table | 0.029744 |
| Removing duplicates | 0.010745 |
| Sorting result | 0.000086 |
| Sending data | 0.004350 |
| end | 0.000008 |
| removing tmp table | 0.001207 |
| end | 0.000005 |
| removing tmp table | 0.000040 |
| end | 0.000004 |
| query end | 0.000003 |
| freeing items | 0.000018 |
| removing tmp table | 0.000005 |
| freeing items | 0.002768 |
| logging slow query | 0.000004 |
| logging slow query | 0.000029 |
| cleaning up | 0.000006 |
+--------------------------------+----------+
34 rows in set (0.00 sec)
First try to execute fsck and then check IO performance: dd if=/dev/zero of=/var/lib/mysql/file1 bs=512 count=10000