On friday, I stupidly closed my lid when I was doing a large (3gb) database import.
This morning, I opened my mac and it never booted. I had to force a restart. Now, mysql no longer works. I am getting the following error: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)
Does anyone know a possible solution? I can't access mysql command prompt.
When I try to perform a restart/stop/start of the service, I get the following (thanks!):
**rmason202-> mysqld restart
2014-06-02 10:03:03 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2014-06-02 10:03:03 9772 [Warning] Setting lower_case_table_names=2 because file system for /usr/local/var/mysql/ is case insensitive
2014-06-02 10:03:03 9772 [Note] Plugin 'FEDERATED' is disabled.
2014-06-02 10:03:03 9772 [Note] InnoDB: The InnoDB memory heap is disabled
2014-06-02 10:03:03 9772 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2014-06-02 10:03:03 9772 [Note] InnoDB: Compressed tables use zlib 1.2.3
2014-06-02 10:03:03 9772 [Note] InnoDB: Using CPU crc32 instructions
2014-06-02 10:03:03 9772 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2014-06-02 10:03:03 9772 [Note] InnoDB: Completed initialization of buffer pool
2014-06-02 10:03:03 9772 [Note] InnoDB: Highest supported file format is Barracuda.
2014-06-02 10:03:03 9772 [Note] InnoDB: Log scan progressed past the checkpoint lsn 30416376018
2014-06-02 10:03:03 9772 [Note] InnoDB: Database was not shutdown normally!
2014-06-02 10:03:03 9772 [Note] InnoDB: Starting crash recovery.
2014-06-02 10:03:03 9772 [Note] InnoDB: Reading tablespace information from the .ibd files...
2014-06-02 10:03:03 9772 [Note] InnoDB: Restoring possible half-written data pages
2014-06-02 10:03:03 9772 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 30421618688
InnoDB: Doing recovery: scanned up to log sequence number 30426861568
InnoDB: Doing recovery: scanned up to log sequence number 30432104448
InnoDB: Doing recovery: scanned up to log sequence number 30437347328
InnoDB: Doing recovery: scanned up to log sequence number 30442590208
InnoDB: Doing recovery: scanned up to log sequence number 30447800979
2014-06-02 10:03:05 9772 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: InnoDB: Error: trying to access page number 480 in space 193870,
InnoDB: space name itrc_dev/interface_ip_addrs,
InnoDB: which is outside the tablespace bounds.
InnoDB: Byte offset 0, len 16384, i/o type 10.
InnoDB: If you get this error at mysqld startup, please check that
InnoDB: your my.cnf matches the ibdata files that you have in the
InnoDB: MySQL server.
2014-06-02 10:03:05 7fff7c945310 InnoDB: Assertion failure in thread 140735283483408 in file fil0fil.cc line 5423
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
14:03:05 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68235 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
0 mysqld 0x0000000103581769 my_print_stacktrace + 61
1 mysqld 0x00000001033df38e handle_fatal_signal + 696
2 libsystem_platform.dylib 0x00007fff998465aa _sigtramp + 26
3 ??? 0x747265737341203a 0x0 + 8390880602273947706
4 libsystem_c.dylib 0x00007fff99441bba abort + 125
5 mysqld 0x00000001035ff96f _Z6fil_iombmmmmmPvS_ + 1765
6 mysqld 0x00000001035d4bb3 _ZL17buf_read_page_lowP7dberr_tbmmmmxm + 427
7 mysqld 0x00000001035d59d7 _Z19buf_read_recv_pagesmmmPKmm + 403
8 mysqld 0x000000010366299f _Z26recv_apply_hashed_log_recsm + 919
9 mysqld** 0x0000000103664afc _Z36recv_recovery_from_checkpoint_finishv + 40
10 mysqld 0x00000001036c9ef4 _Z34innobase_start_or_create_for_mysqlv + 8969
11 mysqld 0x00000001036359ca _ZL13innobase_initPv + 1983
12 mysqld 0x0000000103326b32 _Z24ha_initialize_handlertonP13st_plugin_int + 75
13 mysqld 0x00000001034858a9 _ZL17plugin_initializeP13st_plugin_int + 92
14 mysqld 0x0000000103485397 _Z11plugin_initPiPPci + 3926
15 mysqld 0x00000001035175b1 _Z11mysqld_mainiPPc + 2655
16 libdyld.dylib 0x00007fff9257b5fd start + 1
17 ??? 0x0000000000000002 0x0 + 2
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Related
I have an Innodb database which have a big table with many rows. Once I'm trying to access the data or run any query(like "check table_name") against this specific table mysql crashes(restarts) and I'm getting the error:
ERROR 2013 (HY000): Lost connection to MySQL server during query
I have already tried to increase max_packetsize and the net_timeouts flags.
Any help will be appreciated, Thanks!
Edit:
Crash log:
2022-12-30T16:25:56.676880Z 208 [ERROR] InnoDB: In pages [page id: space=372, pa ge number=11135] and [page id: space=372, page number=11136] of index `GEN_CLUST _INDEX` of table `SHAS_2015_NEW`.`b_media_21_646_0000_11925`
InnoDB: broken FIL_PAGE_NEXT or FIL_PAGE_PREV links
2022-12-30T16:25:56.676925Z 208 [ERROR] InnoDB: In pages [page id: space=372, pa ge number=11135] and [page id: space=372, page number=11136] of index `GEN_CLUST _INDEX` of table `SHAS_2015_NEW`.`b_media_21_646_0000_11925`
InnoDB: 'compact' flag mismatch
2022-12-30T16:25:56.676936Z 208 [ERROR] InnoDB: Page index id 0 != data dictiona ry index id 577
2022-12-30 18:25:56 0x7fa1384e9700 InnoDB: Assertion failure in thread 14033041 1136768 in file btr0btr.cc line 4710
InnoDB: Failing assertion: !page_is_empty(page) || (level == 0 && page_get_page_ no(page) == dict_index_get_page(index))
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
16:25:56 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=1
max_threads=151
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 76388 K b ytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7fa0f81dd110
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fa1384e8e70 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xe8e68b]
/usr/sbin/mysqld(handle_fatal_signal+0x36f)[0x77524f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fa67f45d390]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7fa67e816438]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fa67e81803a]
/usr/sbin/mysqld[0x74a77c]
/usr/sbin/mysqld[0x10dd73d]
/usr/sbin/mysqld(_Z18btr_validate_indexP12dict_index_tPK5trx_tb+0x264)[0x10dec34 ]
/usr/sbin/mysqld(_ZN11ha_innobase5checkEP3THDP15st_ha_check_opt+0x278)[0xf2a378]
/usr/sbin/mysqld(_ZN7handler8ha_checkEP3THDP15st_ha_check_opt+0x7b)[0x7cd71b]
/usr/sbin/mysqld[0xdb48f4]
/usr/sbin/mysqld(_ZN19Sql_cmd_check_table7executeEP3THD+0x97)[0xdb5497]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THDb+0x2160)[0xc42570]
/usr/sbin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x3dd)[0xc473ad]
/usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x11a 0)[0xc48610]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x1c7)[0xc49ae7]
/usr/sbin/mysqld(handle_connection+0x290)[0xd0e140]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xeb03e4]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fa67f4536ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fa67e8e851d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fa0f8072240): check table b_media_21_646_0000_11925
Connection ID (thread ID): 208
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2022-12-30T16:25:56.970335Z 0 [Warning] Changed limits: max_open_files: 1024 (re quested 5000)
2022-12-30T16:25:56.970397Z 0 [Warning] Changed limits: table_open_cache: 431 (r equested 2000)
2022-12-30T16:25:57.145883Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see doc umentation for more details).
2022-12-30T16:25:57.148901Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.33-0ubuntu0.16 .04.1-log) starting as process 9454 ...
2022-12-30T16:25:57.153242Z 0 [Note] InnoDB: PUNCH HOLE support available
2022-12-30T16:25:57.153267Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-12-30T16:25:57.153272Z 0 [Note] InnoDB: Uses event mutexes
2022-12-30T16:25:57.153276Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2022-12-30T16:25:57.153283Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
2022-12-30T16:25:57.153286Z 0 [Note] InnoDB: Using Linux native AIO
2022-12-30T16:25:57.153526Z 0 [Note] InnoDB: Number of pools: 1
2022-12-30T16:25:57.153643Z 0 [Note] InnoDB: Using CPU crc32 instructions
2022-12-30T16:25:57.155093Z 0 [Note] InnoDB: Initializing buffer pool, total siz e = 20G, instances = 8, chunk size = 128M
2022-12-30T16:25:58.482523Z 0 [Note] InnoDB: Completed initialization of buffer pool
2022-12-30T16:25:58.719968Z 0 [Note] InnoDB: If the mysqld execution user is aut horized, page cleaner thread priority can be changed. See the man page of setpri ority().
2022-12-30T16:25:58.744995Z 0 [Note] InnoDB: Highest supported file format is Ba rracuda.
2022-12-30T16:25:58.915826Z 0 [Note] InnoDB: Log scan progressed past the checkp oint lsn 10408206134578
2022-12-30T16:25:58.915857Z 0 [Note] InnoDB: Doing recovery: scanned up to log s equence number 10408206134587
2022-12-30T16:25:58.915862Z 0 [Note] InnoDB: Database was not shutdown normally!
2022-12-30T16:25:58.915872Z 0 [Note] InnoDB: Starting crash recovery.
2022-12-30T16:25:59.270028Z 0 [Note] InnoDB: Last MySQL binlog file position 0 1 862900, file name mysql-bin.000001
2022-12-30T16:26:00.014203Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2022-12-30T16:26:00.014242Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-12-30T16:26:00.014281Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-12-30T16:26:00.075009Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2022-12-30T16:26:00.076863Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2022-12-30T16:26:00.076904Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2022-12-30T16:26:00.077640Z 0 [Note] InnoDB: Waiting for purge to start
2022-12-30T16:26:00.127880Z 0 [Note] InnoDB: 5.7.33 started; log sequence number 10408206134587
2022-12-30T16:26:00.128095Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2022-12-30T16:26:00.128605Z 0 [Note] Plugin 'FEDERATED' is disabled.
2022-12-30T16:26:00.138272Z 0 [Note] Recovering after a crash using /var/log/mysql/mysql-bin
2022-12-30T16:26:00.138325Z 0 [Note] Starting crash recovery...
2022-12-30T16:26:00.138384Z 0 [Note] Crash recovery finished.
If I'm Using select with Limit statement then I can select up to 681,000 rows
Error dump & conf below on a OSX MariaDb instance (installed via Brew). This db worked fine prior to my mac freezing up and restarting while the DB was running.
Now when I try to mysql.server start I get the following error:
mysqld_safe Starting mysqld daemon with databases from /usr/local/var/mysql
ERROR!
I have checked for surplus .pid files and deleted them (after trying other things such as starting in safe mode and doing a db check on table corruption, which passed it's check).
My .cnf file:
# This group is read both both by the client and the server
# use it for options that affect everything
#
[client-server]
#
# include all files from the config directory
#
!includedir /usr/local/etc/my.cnf.d
[mysqld]
innodb_buffer_pool_size=1676M
2019-01-28 12:39:28 4802311616 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-01-28 12:39:28 4802311616 [Note] InnoDB: Uses event mutexes
2019-01-28 12:39:28 4802311616 [Note] InnoDB: Compressed tables use zlib 1.2.8
2019-01-28 12:39:28 4802311616 [Note] InnoDB: Number of pools: 1
2019-01-28 12:39:28 4802311616 [Note] InnoDB: Using SSE2 crc32 instructions
2019-01-28 12:39:28 4802311616 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M
2019-01-28 12:39:28 4802311616 [Note] InnoDB: Completed initialization of buffer pool
2019-01-28 12:39:28 4802311616 [Note] InnoDB: Highest supported file format is Barracuda.
2019-01-28 12:39:29 4802311616 [Note] InnoDB: 3 transaction(s) which must be rolled back or cleaned up in total 2294467 row operations to undo
2019-01-28 12:39:29 4802311616 [Note] InnoDB: Trx id counter is 230784256
2019-01-28 12:39:29 4802311616 [Note] InnoDB: 128 out of 128 rollback segments are active.
2019-01-28 12:39:29 123145460723712 [Note] InnoDB: Starting in background the rollback of recovered transactions
2019-01-28 12:39:29 4802311616 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-01-28 12:39:29 4802311616 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2019-01-28 12:39:29 0x700009713000 InnoDB: Assertion failure in file /tmp/mariadb-20180328-24866-1e7eel7/mariadb-10.2.14/storage/innobase/include/fut0lst.ic line 85
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/
InnoDB: about forcing recovery.
190128 12:39:29 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.2.14-MariaDB
key_buffer_size=268435456
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 598321 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
2019-01-28 12:39:29 4802311616 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2019-01-28 12:39:29 4802311616 [Note] InnoDB: 5.7.21 started; log sequence number 732581563327
2019-01-28 12:39:29 4802311616 [Note] InnoDB: !!! innodb_force_recovery is set to 1 !!!
2019-01-28 12:39:29 123145466626048 [Note] InnoDB: Loading buffer pool(s) from /usr/local/var/mysql/ib_buffer_pool
0 mysqld 0x000000010edc812c my_print_stacktrace + 60
0 mysqld 0x000000010e814cdc handle_fatal_signal + 718
0 libsystem_platform.dylib 0x00007fff793fdb3d _sigtramp + 29
0 ??? 0x000000011e3a4e50 0x0 + 4802104912
2019-01-28 12:39:29 4802311616 [Note] Plugin 'FEEDBACK' is disabled.
0 libsystem_c.dylib 0x00007fff792bc1c9 abort + 127
2019-01-28 12:39:29 4802311616 [Note] Server socket created on IP: '::'.
0 mysqld 0x000000010eccadb0 _Z14ib_list_createv + 0
0 mysqld 0x000000010eba48ac _Z14flst_add_firstPhS_P5mtr_t + 1420
0 mysqld 0x000000010ecad0eb _Z36trx_purge_add_update_undo_to_historyP5trx_tPhP5mtr_t + 355
0 mysqld 0x000000010ecc84d3 _Z23trx_undo_update_cleanupP5trx_tPhP5mtr_t + 32
0 mysqld 0x000000010ecc0db3 _Z14trx_commit_lowP5trx_tP5mtr_t + 2750
0 mysqld 0x000000010ecc1103 _Z10trx_commitP5trx_t + 214
0 mysqld 0x000000010ecba6f2 _Z31trx_rollback_or_clean_recoveredm + 787
0 mysqld 0x000000010ecbacef trx_rollback_or_clean_all_recovered + 62
0 libsystem_pthread.dylib 0x00007fff79406339 _pthread_body + 126
0 libsystem_pthread.dylib 0x00007fff794092a7 _pthread_start + 70
0 libsystem_pthread.dylib 0x00007fff79405445 thread_start + 13
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2019-01-28 12:39:29 4350166464 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-01-28 12:39:29 4350166464 [Note] InnoDB: Uses event mutexes
2019-01-28 12:39:29 4350166464 [Note] InnoDB: Compressed tables use zlib 1.2.8
2019-01-28 12:39:29 4350166464 [Note] InnoDB: Number of pools: 1
2019-01-28 12:39:29 4350166464 [Note] InnoDB: Using SSE2 crc32 instructions
2019-01-28 12:39:29 4350166464 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M
2019-01-28 12:39:30 4350166464 [Note] InnoDB: Completed initialization of buffer pool
2019-01-28 12:39:30 4350166464 [Note] InnoDB: Highest supported file format is Barracuda.
2019-01-28 12:39:30 4350166464 [Note] InnoDB: 3 transaction(s) which must be rolled back or cleaned up in total 2294467 row operations to undo
2019-01-28 12:39:30 4350166464 [Note] InnoDB: Trx id counter is 230784256
2019-01-28 12:39:30 4350166464 [Note] InnoDB: 128 out of 128 rollback segments are active.
2019-01-28 12:39:30 123145347538944 [Note] InnoDB: Starting in background the rollback of recovered transactions
2019-01-28 12:39:30 0x700002b22000 InnoDB: Assertion failure in file /tmp/mariadb-20180328-24866-1e7eel7/mariadb-10.2.14/storage/innobase/include/fut0lst.ic line 85
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/
InnoDB: about forcing recovery.
190128 12:39:30 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.2.14-MariaDB
key_buffer_size=268435456
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 598321 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
2019-01-28 12:39:30 4350166464 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2019-01-28 12:39:30 4350166464 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-01-28 12:39:30 4350166464 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
0 mysqld 0x000000010128112c my_print_stacktrace + 60
0 mysqld 0x0000000100ccdcdc handle_fatal_signal + 718
0 libsystem_platform.dylib 0x00007fff793fdb3d _sigtramp + 29
0 ??? 0x0000000103471e50 0x0 + 4349959760
0 libsystem_c.dylib 0x00007fff792bc1c9 abort + 127
0 mysqld 0x0000000101183db0 _Z14ib_list_createv + 0
0 mysqld 0x000000010105d8ac _Z14flst_add_firstPhS_P5mtr_t + 1420
0 mysqld 0x00000001011660eb _Z36trx_purge_add_update_undo_to_historyP5trx_tPhP5mtr_t + 355
0 mysqld 0x00000001011814d3 _Z23trx_undo_update_cleanupP5trx_tPhP5mtr_t + 32
0 mysqld 0x0000000101179db3 _Z14trx_commit_lowP5trx_tP5mtr_t + 2750
0 mysqld 0x000000010117a103 _Z10trx_commitP5trx_t + 214
0 mysqld 0x00000001011736f2 _Z31trx_rollback_or_clean_recoveredm + 787
0 mysqld 0x0000000101173cef trx_rollback_or_clean_all_recovered + 62
0 libsystem_pthread.dylib 0x00007fff79406339 _pthread_body + 126
0 libsystem_pthread.dylib 0x00007fff794092a7 _pthread_start + 70
0 libsystem_pthread.dylib 0x00007fff79405445 thread_start + 13
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
My database crash with out of memory error ~1-2 times per day. Last crash was at 17:30 in logs and at 19:30 in metrics (different timezones).
Errors in mariadb.log:
181120 16:09:00 [Warning] IP address '185.156.177.144' could not be resolved: Name or service not known
181120 16:35:40 [Warning] IP address '58.221.58.248' could not be resolved: Name or service not known
181120 17:30:38 mysqld_safe Number of processes running now: 0
181120 17:30:38 mysqld_safe mysqld restarted
181120 17:30:41 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
181120 17:30:41 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 16636 ...
181120 17:30:41 [Warning] Changed limits: max_open_files: 1024 max_connections: 70 table_cache: 472
181120 17:30:41 InnoDB: The InnoDB memory heap is disabled
181120 17:30:41 InnoDB: Mutexes and rw_locks use GCC atomic builtins
181120 17:30:41 InnoDB: Compressed tables use zlib 1.2.7
181120 17:30:41 InnoDB: Using Linux native AIO
181120 17:30:41 InnoDB: Initializing buffer pool, size = 150.0M
181120 17:30:42 InnoDB: Completed initialization of buffer pool
181120 17:30:42 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
181120 17:30:42 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
181120 17:30:46 InnoDB: Waiting for the background threads to start
181120 17:30:47 Percona XtraDB (http://www.percona.com) 5.5.49-MariaDB-38.0 started; log sequence number 18741266161
181120 17:30:47 [Note] Plugin 'FEEDBACK' is disabled.
181120 17:30:48 [Note] Server socket created on IP: '0.0.0.0'.
181120 17:30:49 [Note] Event Scheduler: Loaded 0 events
181120 17:30:49 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.52-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
181120 17:30:53 [ERROR] mysqld: Table './admin_wikis/wp_posts' is marked as crashed and should be repaired
181120 17:30:53 [Warning] Checking table: './admin_wikis/wp_posts'
181120 17:31:58 mysqld_safe Number of processes running now: 0
181120 17:31:58 mysqld_safe mysqld restarted
181120 17:32:01 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
181120 17:32:01 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 16899 ...
181120 17:32:01 [Warning] Changed limits: max_open_files: 1024 max_connections: 70 table_cache: 472
181120 17:32:01 InnoDB: The InnoDB memory heap is disabled
181120 17:32:01 InnoDB: Mutexes and rw_locks use GCC atomic builtins
181120 17:32:01 InnoDB: Compressed tables use zlib 1.2.7
181120 17:32:01 InnoDB: Using Linux native AIO
181120 17:32:01 InnoDB: Initializing buffer pool, size = 150.0M
InnoDB: mmap(161447936 bytes) failed; errno 12
181120 17:32:01 InnoDB: Completed initialization of buffer pool
181120 17:32:01 InnoDB: Fatal error: cannot allocate memory for the buffer pool
181120 17:32:01 [ERROR] Plugin 'InnoDB' init function returned error.
181120 17:32:01 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
181120 17:32:01 [ERROR] mysqld: Out of memory (Needed 128917504 bytes)
181120 17:32:01 [ERROR] mysqld: Out of memory (Needed 96681984 bytes)
181120 17:32:01 [ERROR] mysqld: Out of memory (Needed 72499200 bytes)
181120 17:32:01 [Note] Plugin 'FEEDBACK' is disabled.
181120 17:32:02 [ERROR] Unknown/unsupported storage engine: InnoDB
181120 17:32:02 [ERROR] Aborting
181120 17:32:02 [Note] /usr/libexec/mysqld: Shutdown complete
There is nothing interesing in apache log in this time. Event requests count
$ sudo grep -r "GET \/" /etc/httpd/logs/domains/ | cut -d' ' -f4 | sed 's/\[//;s/\]//;s/\"//' | cut -d':' -f1-2 | sort | uniq -c
459 19/Nov/2018:03
426 19/Nov/2018:04
239 19/Nov/2018:05
350 19/Nov/2018:06
381 19/Nov/2018:07
415 19/Nov/2018:08
778 19/Nov/2018:09
500 19/Nov/2018:10
450 19/Nov/2018:11
633 19/Nov/2018:12
458 19/Nov/2018:13
527 19/Nov/2018:14
713 19/Nov/2018:15
654 19/Nov/2018:16
573 19/Nov/2018:17
413 19/Nov/2018:18
471 19/Nov/2018:19
499 19/Nov/2018:20
661 19/Nov/2018:21
452 19/Nov/2018:22
773 19/Nov/2018:23
934 20/Nov/2018:00
295 20/Nov/2018:01
369 20/Nov/2018:02
441 20/Nov/2018:03
384 20/Nov/2018:04
927 20/Nov/2018:05
524 20/Nov/2018:06
972 20/Nov/2018:07
612 20/Nov/2018:08
609 20/Nov/2018:09
677 20/Nov/2018:10
753 20/Nov/2018:11
615 20/Nov/2018:12
717 20/Nov/2018:13
474 20/Nov/2018:14
973 20/Nov/2018:15
510 20/Nov/2018:16
534 20/Nov/2018:17
415 20/Nov/2018:18
90 20/Nov/2018:19
I get some metrics:
Processes before crash:
Processes in the moment of crash:
Memory and CPU usage:
Total RAM 3.6G. Does someone have any ideas?
httpd is the largest memory user; decrease its settings, especially the number of 'children' it is willing to spawn. (Under 10 should suffice for most small installations.)
max_connections: 70 table_cache: 472 -- Cut each of those in half; this will help a little with memory allocation.
Next time the CPU hits 100%, do SHOW FULL PROCESSLIST; in MySQL. That may give you a clue of what runaway queries you have. Then we can focus on them. If that fails, turn on the slowlog; it will catch some of them.
Consider moving to an 8GB server. (Is this a cloud server?)
I've run into an issue that has me totally stumped. A little preface here - I'm not a MySQL guy. I know enough to get around in MAMP and a little through the terminal but that's the extent of my knowledge.
I recently upgraded to MAMP PRO 4.1 (I'm on Mac OS 10.11.6) and I've been pulling out my hair ever since. It won't run MySQL and the error log is returning this.
170123 18:52:07 mysqld_safe mysqld restarted
2017-01-23 18:52:07 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-01-23 18:52:07 0 [Note] --secure-file-priv is set to NULL. Operations related to importing and exporting data are disabled
2017-01-23 18:52:07 0 [Note] /Applications/MAMP/Library/bin/mysqld (mysqld 5.6.34) starting as process 13145 ...
2017-01-23 18:52:07 13145 [Warning] Setting lower_case_table_names=2 because file system for /Library/Application Support/appsolute/MAMP PRO/db/mysql56/ is case insensitive
2017-01-23 18:52:07 13145 [Note] Plugin 'FEDERATED' is disabled.
/Applications/MAMP/Library/bin/mysqld: Table 'mysql.plugin' doesn't exist
2017-01-23 18:52:07 13145 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
2017-01-23 18:52:07 13145 [Note] InnoDB: Using atomics to ref count buffer pool pages
2017-01-23 18:52:07 13145 [Note] InnoDB: The InnoDB memory heap is disabled
2017-01-23 18:52:07 13145 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-01-23 18:52:07 13145 [Note] InnoDB: Memory barrier is not used
2017-01-23 18:52:07 13145 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-01-23 18:52:07 13145 [Note] InnoDB: Not using CPU crc32 instructions
2017-01-23 18:52:07 13145 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2017-01-23 18:52:07 13145 [Note] InnoDB: Completed initialization of buffer pool
2017-01-23 18:52:07 13145 [Note] InnoDB: Highest supported file format is Barracuda.
2017-01-23 18:52:07 13145 [Note] InnoDB: Log scan progressed past the checkpoint lsn 57680420515
2017-01-23 18:52:07 13145 [Note] InnoDB: Database was not shutdown normally!
2017-01-23 18:52:07 13145 [Note] InnoDB: Starting crash recovery.
2017-01-23 18:52:07 13145 [Note] InnoDB: Reading tablespace information from the .ibd files...
2017-01-23 18:52:07 13145 [Note] InnoDB: Restoring possible half-written data pages
2017-01-23 18:52:07 13145 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 57680420525
2017-01-23 18:52:07 13145 [Note] InnoDB: 128 rollback segment(s) are active.
2017-01-23 18:52:07 13145 [Note] InnoDB: 5.6.34 started; log sequence number 57680420525
2017-01-23 18:52:08 13145 [Note] RSA private key file not found: /Library/Application Support/appsolute/MAMP PRO/db/mysql56//private_key.pem. Some authentication plugins will not work.
2017-01-23 18:52:08 13145 [Note] RSA public key file not found: /Library/Application Support/appsolute/MAMP PRO/db/mysql56//public_key.pem. Some authentication plugins will not work.
2017-01-23 18:52:08 13109 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 35
2017-01-23 18:52:08 13109 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2017-01-23 18:52:09 13109 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 35
2017-01-23 18:52:09 13109 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
23:52:09 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 134277 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7f91330cda00
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fff590196d0 thread_stack 0x40000
0 mysqld 0x0000000107131878 my_print_stacktrace + 72
1 mysqld 0x0000000106df6fe8 handle_fatal_signal + 952
2 libsystem_platform.dylib 0x00007fff9580b52a _sigtramp + 26
3 mysqld 0x0000000107b62c38 LOCK_plugin + 0
4 mysqld 0x000000010701c223 _Z9get_fieldP11st_mem_rootP5Field + 99
5 mysqld 0x0000000106e1d2d5 _ZL8acl_loadP3THDP10TABLE_LIST + 1941
6 mysqld 0x0000000106e1c5b0 _Z10acl_reloadP3THD + 1264
7 mysqld 0x0000000106e1c025 _Z8acl_initb + 405
8 mysqld 0x000000010705b029 _Z11mysqld_mainiPPc + 2281
9 mysqld 0x0000000106be7c22 main + 34
10 libdyld.dylib 0x00007fff985a45ad start + 1
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 0
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
I've tried to do my due diligence and killed all the mysqld processes and tried restarting again via:
ps aux | grep mysqld
sudo kill <process id of mysqld here>
However, the process id no longer exists by the time I run it. It's constantly starting, crashing, and trying to restart. Any help here would be hugely appreciated.
I had the same problem and fixed it as follows:
Exit MAMP
In terminal run killall -9 mysqld
Start MAMP
If this doesn;t work, try putting sudo at the start of the terminal command.
See http://twob.net/journal/fix-for-mamp-mysql/ for more detailed info
I tried running optimize on one of my tables, and when I did the server crashed. Now when I try to restart it, I get this error:
The server quit without updating PID file (/usr/local/mysql/data/Diskise.pid).
Then when I look at the error log to help me see what happened, I don't really understand it. What can I do to get my server back up?
131117 17:13:43 mysqld_safe Starting mysqld daemon with databases from /usr/local/mysql/data
2013-11-17 17:13:44 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2013-11-17 17:13:44 29352 [Note] Plugin 'FEDERATED' is disabled.
2013-11-17 17:13:44 29352 [Note] InnoDB: The InnoDB memory heap is disabled
2013-11-17 17:13:44 29352 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2013-11-17 17:13:44 29352 [Note] InnoDB: Compressed tables use zlib 1.2.3
2013-11-17 17:13:44 29352 [Note] InnoDB: Using Linux native AIO
2013-11-17 17:13:44 29352 [Note] InnoDB: Not using CPU crc32 instructions
2013-11-17 17:13:44 29352 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2013-11-17 17:13:44 29352 [Note] InnoDB: Completed initialization of buffer pool
2013-11-17 17:13:44 29352 [Note] InnoDB: Highest supported file format is Barracuda.
2013-11-17 17:13:44 29352 [Note] InnoDB: Log scan progressed past the checkpoint lsn 57196040504
2013-11-17 17:13:44 29352 [Note] InnoDB: Database was not shutdown normally!
2013-11-17 17:13:44 29352 [Note] InnoDB: Starting crash recovery.
2013-11-17 17:13:44 29352 [Note] InnoDB: Reading tablespace information from the .ibd files...
2013-11-17 17:13:44 29352 [Note] InnoDB: Restoring possible half-written data pages
2013-11-17 17:13:44 29352 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 57201283072
InnoDB: Doing recovery: scanned up to log sequence number 57204973005
InnoDB: Error: trying to access page number 377945542 in space 0,
InnoDB: space name ./ibdata1,
InnoDB: which is outside the tablespace bounds.
InnoDB: Byte offset 0, len 16384, i/o type 10.
InnoDB: If you get this error at mysqld startup, please check that
InnoDB: your my.cnf matches the ibdata files that you have in the
InnoDB: MySQL server.
2013-11-17 17:13:44 b73ae700 InnoDB: Assertion failure in thread 3074090752 in file fil0fil.cc line 5423
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
17:13:44 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 67592 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/local/mysql/bin/mysqld(my_print_stacktrace+0x33)[0x84e19b3]
/usr/local/mysql/bin/mysqld(handle_fatal_signal+0x3c8)[0x826c218]
[0xb76ef400]
[0xb76ef416]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb73de1df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb73e1825]
/usr/local/mysql/bin/mysqld[0x864f06f]
/usr/local/mysql/bin/mysqld[0x861d3f3]
/usr/local/mysql/bin/mysqld[0x861e03d]
/usr/local/mysql/bin/mysqld[0x8605e5c]
/usr/local/mysql/bin/mysqld[0x85d135f]
/usr/local/mysql/bin/mysqld[0x85c649d]
/usr/local/mysql/bin/mysqld[0x85c739e]
/usr/local/mysql/bin/mysqld[0x85c844f]
/usr/local/mysql/bin/mysqld[0x85b33d9]
/usr/local/mysql/bin/mysqld[0x84f593a]
/usr/local/mysql/bin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x43)[0x81c1c93]
/usr/local/mysql/bin/mysqld[0x82f32bd]
/usr/local/mysql/bin/mysqld(_Z11plugin_initPiPPci+0x7c8)[0x82f5058]
/usr/local/mysql/bin/mysqld(_Z11mysqld_mainiPPc+0x8c3)[0x81bae83]
/usr/local/mysql/bin/mysqld(main+0x1b)[0x819b30b]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb73c94d3]
/usr/local/mysql/bin/mysqld[0x81afa11]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
131117 17:13:44 mysqld_safe mysqld from pid file /usr/local/mysql/data/Diskise.pid ended