MySQL server has gone away, mysqld dead but subsys locked - mysql

I am running Centos 6.4 on i686 PC and here is a problem with mysqld. It restarts all the time, look example. Sorry bad english.
service mysqld status
mysqld (pid 21563) is running...
[root#ip mysql]# service mysqld status
mysqld (pid 21616) is running...
[root#ip mysql]# service mysqld status
mysqld (pid 21667) is running...
[root#ip mysql]# service mysqld status
mysqld (pid 21718) is running...
[root#ip mysql]# service mysqld status
mysqld dead but subsys locked
[root#ip mysql]# service mysqld status
mysqld dead but subsys locked
[root#ip mysql]# service mysqld status
mysqld (pid 21820) is running...
[root#ip mysql]# service mysqld status
mysqld (pid 21922) is running...
[root#ip mysql]# service mysqld status
mysqld (pid 21973) is running...
[root#ip mysql]# service mysqld status
mysqld (pid 22026) is running...
[root#ip mysql]# service mysqld status
mysqld (pid 22026) is running...
[root#ip mysql]# service mysqld status
mysqld (pid 22128) is running...
[root#ip mysql]# service mysqld status
mysqld (pid 22179) is running...
Here is /etc/my.cnf:
[mysqld]
bind-address = 127.0.0.1
general_log = 1
general_log_file = /var/log/mysql/mysql.log
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
wait_timeout = 28800
max_allowed_packet = 32M
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
[mysqld_safe]
wait_timeout = 28800
max_allowed_packet = 32M
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
Look at the /var/log/mysqld.log:
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/libexec/mysqld(my_print_stacktrace+0x2e) [0x84c6d3e]
/usr/libexec/mysqld(handle_fatal_signal+0x484) [0x82ec8b4]
[0xac3400]
[0xac3424]
/lib/libc.so.6(gsignal+0x51) [0x70f861]
/lib/libc.so.6(abort+0x17a) [0x71113a]
/usr/libexec/mysqld() [0x8396186]
/usr/libexec/mysqld(btr_cur_search_to_nth_level+0xabe) [0x839715e]
/usr/libexec/mysqld(row_search_index_entry+0x79) [0x8417949]
/usr/libexec/mysqld() [0x8416517]
/usr/libexec/mysqld(row_purge_step+0x5ab) [0x841779b]
/usr/libexec/mysqld(que_run_threads+0x535) [0x8404a45]
/usr/libexec/mysqld(trx_purge+0x375) [0x8432605]
/usr/libexec/mysqld(srv_master_thread+0x75b) [0x842a67b]
/lib/libpthread.so.0(+0x6b39) [0x6d0b39]
/lib/libc.so.6(clone+0x5e) [0x7c7ace]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
170621 20:14:38 mysqld_safe Number of processes running now: 0
170621 20:14:38 mysqld_safe mysqld restarted
170621 20:14:38 InnoDB: Initializing buffer pool, size = 8.0M
170621 20:14:38 InnoDB: Completed initialization of buffer pool
170621 20:14:38 InnoDB: Started; log sequence number 0 1965184248
170621 20:14:38 [Note] Event Scheduler: Loaded 0 events
170621 20:14:38 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
170621 20:14:39 InnoDB: Assertion failure in thread 2968296304 in file btr/btr0cur.c line 201
InnoDB: Failing assertion: btr_page_get_prev(get_page, mtr) == buf_frame_get_page_no(page)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
17:14:39 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=8384512
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 337742 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
And from mysql CLI sometimes I can connect and run queries:
mysql> show databases;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id: 1
Current database: *** NONE ***
+--------------------+
| Database |
+--------------------+
| information_schema |
| asterisk |
| asteriskcdrdb |
| drupal |
| drupal2 |
| drupal3 |
| drupal4test |
| mysql |
| statistics |
| test |
+--------------------+
10 rows in set (0.00 sec)
mysql> show databases;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
ERROR:
Can't connect to the server
mysql>
Any ideas?
Any information will be very good for me )

Related

homebrew installed mysql#5.7 does not start

I had to move from mysql (8.0.29) to mysql#5.7. MySQL in its newest formula does not compile any longer on my machine, because the machine is running Mojave in order to have native Aperture support.
I installed mysql#5.7 and changed path in my .zshrc to point to the new /usr/local/opt/mysql#5.7/bin.
After this I am still unable to start MySQL. When I run brew services list, I get:
mysql#5.7 stopped USER ~/Library/LaunchAgents/homebrew.mxcl.mysql#5.7.plist
When I try to run /usr/local/opt/mysql/bin/mysqld_safe --datadir=/usr/local/var/mysql I get:
√ ~ % /usr/local/opt/mysql/bin/mysqld_safe --datadir=/usr/local/var/mysql
2022-10-07T17:06:21.6NZ mysqld_safe Logging to '/usr/local/var/mysql/MACHINE.DOMAIN.TLD.err'.
2022-10-07T17:06:21.6NZ mysqld_safe Starting mysqld daemon with databases from /usr/local/var/mysql
/usr/local/opt/mysql/bin/mysqld_safe: line 199: 9704 Abort trap: 6 env MYSQLD_PARENT_PID=9530 nohup /usr/local/opt/mysql/bin/mysqld --basedir=/usr/local/opt/mysql --datadir=/usr/local/var/mysql --plugin-dir=/usr/local/opt/mysql/lib/plugin --log-error=MACHINE.DOMAIN.TLD.err --pid-file=mMACHINE.DOMAIN.TLD.pid < /dev/null >> /usr/local/var/mysql/MACHINE.DOMAIN.TLD.err 2>&1
2022-10-07T17:06:21.6NZ mysqld_safe mysqld from pid file /usr/local/var/mysql/MACHINE.DOMAIN.TLD.pid ended
When I check the error log file MACHINE.DOMAIN.TLD.err, I get the following output:
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68222 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
0 mysqld 0x0000000101a676a9 my_print_stacktrace + 58
1 mysqld 0x00000001019e5473 handle_fatal_signal + 698
2 libsystem_platform.dylib 0x00007fff6d0b6b5d _sigtramp + 29
3 ??? 0x0000000103ae5380 0x0 + 4356723584
4 libsystem_c.dylib 0x00007fff6cf706a6 abort + 127
5 mysqld 0x0000000101c568b9 _Z23ut_dbg_assertion_failedPKcS0_m + 161
6 mysqld 0x0000000101c5929b _ZN2ib5fatalD2Ev + 91
7 mysqld 0x0000000101c592d9 _ZN2ib5fatalD1Ev + 9
8 mysqld 0x0000000101affc6d _ZL18fil_node_open_fileP10fil_node_t + 2435
9 mysqld 0x0000000101b0946b _ZL23fil_node_prepare_for_ioP10fil_node_tP12fil_system_tP11fil_space_t + 191
10 mysqld 0x0000000101b09b71 _Z6fil_ioRK9IORequestbRK9page_id_tRK11page_size_tmmPvS8_ + 796
11 mysqld 0x0000000101ad0dfc _ZL17buf_read_page_lowP7dberr_tbmmRK9page_id_tRK11page_size_tb + 415
12 mysqld 0x0000000101ad0f53 _Z13buf_read_pageRK9page_id_tRK11page_size_t + 56
13 mysqld 0x0000000101abc5d4 _Z16buf_page_get_genRK9page_id_tRK11page_size_tmP11buf_block_tmPKcmP5mtr_tb + 971
14 mysqld 0x0000000101c464d4 _Z31trx_rseg_get_n_undo_tablespacesPm + 251
15 mysqld 0x0000000101c296ab _Z34innobase_start_or_create_for_mysqlv + 6608
16 mysqld 0x0000000101b589d3 _ZL13innobase_initPv + 3636
17 mysqld 0x00000001014cde2e _Z24ha_initialize_handlertonP13st_plugin_int + 78
18 mysqld 0x0000000101931b46 _ZL17plugin_initializeP13st_plugin_int + 81
19 mysqld 0x000000010193164a _Z40plugin_register_builtin_and_init_core_sePiPPc + 706
20 mysqld 0x00000001019da4e2 _Z11mysqld_mainiPPc + 2918
21 libdyld.dylib 0x00007fff6cecb3d5 start + 1
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2022-10-07T16:36:18.6NZ mysqld_safe mysqld from pid file /usr/local/var/mysql/MACHINE.DOMAIN.TLD.pid ended
2022-10-07T16:36:28.6NZ mysqld_safe Logging to '/usr/local/var/mysql/MACHINE.DOMAIN.TLD.err'.
2022-10-07T16:36:28.6NZ mysqld_safe Starting mysqld daemon with databases from /usr/local/var/mysql
2022-10-07T16:36:29.201378Z 0 [Note] --secure-file-priv is set to NULL. Operations related to importing and exporting data are disabled
2022-10-07T16:36:29.201633Z 0 [Note] /usr/local/opt/mysql#5.7/bin/mysqld (mysqld 5.7.39) starting as process 8643 ...
2022-10-07T16:36:29.205774Z 0 [Warning] Setting lower_case_table_names=2 because file system for /usr/local/var/mysql/ is case insensitive
2022-10-07T16:36:29.207723Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-10-07T16:36:29.207755Z 0 [Note] InnoDB: Uses event mutexes
2022-10-07T16:36:29.207775Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2022-10-07T16:36:29.207791Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.12
2022-10-07T16:36:29.210674Z 0 [Note] InnoDB: Number of pools: 1
2022-10-07T16:36:29.210835Z 0 [Note] InnoDB: Using CPU crc32 instructions
2022-10-07T16:36:29.212601Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2022-10-07T16:36:29.224493Z 0 [Note] InnoDB: Completed initialization of buffer pool
2022-10-07T16:36:29.273548Z 0 [ERROR] [FATAL] InnoDB: Table flags are 0 in the data dictionary but the flags in file ./ibdata1 are 0x4000!
2022-10-07 18:36:29 0x110a1e5c0 InnoDB: Assertion failure in thread 4574012864 in file ut0ut.cc line 921
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
16:36:29 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
I really hope someone can point me into the right direction. Maybe you cannot switch your databases back to 5.7 from 8?
Update
When I delete my v8 DB and try to run mysqld --initialize I get:
dyld: Library not loaded: /usr/local/opt/icu4c/lib/libicuuc.70.dylib
Referenced from: /usr/local/bin/mysqld
Reason: image not found
zsh: abort mysqld --initialize
After restoring the whole /usr/local directory from Timemachine to a state before updating and upgrading Homebrew and its formulae, everything appears to be working again.
I do not know what caused my dysfunctional system. I suspect that, when I upgraded single formulae or when I installed a new formulae last week, something must have gone wrong.
The symptoms were, that single apps refused to run by complaining about the missing library /usr/local/opt/icu4c/lib/libicuuc.70.dylib. In fact icu4c had been upgraded to version 71.1.
Restoring the whole directory /usr/local returned all the apps, which I discovered as non working to a running state. I haven't run a brew update or brew upgrade yet.

After including option --skip-grant-tables in configuration file, mysqld starts, runs, and then stops immediately

My situation is the following. I can't remember my root user passsword for accessing mysqld. I'm attempting to start mysqld with the option --skip-grant-tables option in the my.ini configuration file. However, I observe that when I enable the option and then try to restart mysqld (from Task Manager, for example), the service starts, runs, and then immediately stops. I've included a Screencastify video to show you the steps I take (1min50secs long). I'm also including the error file contents below for your review. I see a mention of an error but I don't understand its message.
2022-08-25T17:52:06.012094Z 0 [Warning] [MY-010915] [Server] 'NO_ZERO_DATE', 'NO_ZERO_IN_DATE' and 'ERROR_FOR_DIVISION_BY_ZERO' sql modes should be used with strict mode. They will be merged with strict mode in a future release.
2022-08-25T17:52:06.015237Z 0 [System] [MY-010116] [Server] C:\Program Files\MySQL\MySQL Server 8.0\bin\mysqld.exe (mysqld 8.0.28) starting as process 1048
2022-08-25T17:52:06.045494Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-25T17:52:06.398313Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-25T17:52:06.642873Z 0 [Warning] [MY-011311] [Server] Plugin mysqlx reported: 'All I/O interfaces are disabled, X Protocol won't be accessible'
2022-08-25T17:52:06.725383Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-25T17:52:06.726481Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-25T17:52:06.757168Z 0 [System] [MY-010931] [Server] C:\Program Files\MySQL\MySQL Server 8.0\bin\mysqld.exe: ready for connections. Version: '8.0.28' socket: '' port: 0 MySQL Community Server - GPL.
2022-08-25T17:52:06.759476Z 0 [ERROR] [MY-010131] [Server] TCP/IP, --shared-memory, or --named-pipe should be configured on NT OS
2022-08-25T17:52:06.760755Z 0 [ERROR] [MY-010119] [Server] Aborting
2022-08-25T17:52:07.903037Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: trx0sys.cc:644:UT_LIST_GET_LEN(trx_sys->mysql_trx_list) == 0 thread 26320
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
17:52:07 UTC - mysqld got exception 0x16 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
7ff78a48fd48 mysqld.exe!?my_print_stacktrace##YAXPEBEK#Z()
7ff7896c004b mysqld.exe!?print_fatal_signal##YAXH#Z()
7ff7896bfe13 mysqld.exe!?my_server_abort##YAXXZ()
7ff78a47643a mysqld.exe!?my_abort##YAXXZ()
7ff78a6ac579 mysqld.exe!??$endl#DU?$char_traits#D#std###std##YAAEAV?$basic_ostream#DU?$char_traits#D#std###0#AEAV10##Z()
7ff78a5e8167 mysqld.exe!?set_compression_level#Zstd_comp#compression#transaction#binary_log##UEAAXI#Z()
7ff78a67033b mysqld.exe!??$endl#DU?$char_traits#D#std###std##YAAEAV?$basic_ostream#DU?$char_traits#D#std###0#AEAV10##Z()
7ff78a568550 mysqld.exe!?set_compression_level#Zstd_comp#compression#transaction#binary_log##UEAAXI#Z()
7ff7894a452b mysqld.exe!?ha_finalize_handlerton##YAHPEAUst_plugin_int###Z()
7ff7894e2304 mysqld.exe!?memcached_shutdown##YAXXZ()
7ff7894e78d9 mysqld.exe!?plugin_unlock_list##YAXPEAVTHD##PEAPEAUst_plugin_int##_K#Z()
7ff7894e6630 mysqld.exe!?plugin_shutdown##YAXXZ()
7ff789482d49 mysqld.exe!?check_valid_arguments_processor#Item_func##UEAA_NPEAE#Z()
7ff78949679a mysqld.exe!?underflow#?$basic_stringbuf#DU?$char_traits#D#std##V?$allocator#D#2##std##MEAAHXZ()
7ff78949531f mysqld.exe!?setup_conn_event_handler_threads##YAXXZ()
7ff7894988be mysqld.exe!?win_main##YAHHPEAPEAD#Z()
7ff7894916f5 mysqld.exe!?mysql_service##YAHPEAX#Z()
7ff789491d54 mysqld.exe!?mysqld_main##YAHHPEAPEAD#Z()
7ff78ab0b298 mysqld.exe!?set_timespec_nsec##YAXPEAUtimespec##_K#Z()
7ffe8b437034 KERNEL32.DLL!BaseThreadInitThunk()
7ffe8b742651 ntdll.dll!RtlUserThreadStart()
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Any help will be appreciated. Thank you.
Enabling shared memory in server section of config file resolved the issue.
See this Stackoverflow post for context.
If you have an issue starting MySQL, try editing the my.ini configuration file, and set innodb_force_recovery to perhaps 3.
Here's how i've reset MySQL root password on Centos; note it's from the command line.
systemctl stop mysqld
systemctl set-environment MYSQLD_OPTS=--skip-grant-tables
systemctl start mysqld
mysql> UPDATE mysql.user SET authentication_string = PASSWORD ('ENTER_NEW_PASSWORD') WHERE User = 'root' AND Host = 'localhost';
mysql> FLUSH PRIVILEGES;
mysql> quit
systemctl stop mysqld
systemctl unset-environment MYSQLD_OPTS
systemctl start mysqld
# test #
mysql -u root#localhost -p

ERROR! The server quit without updating PID file - mysql56 via brew

160112 08:49:29 mysqld_safe Starting mysqld daemon with databases from /usr/local/var/mysql
2016-01-12 08:49:30 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-01-12 08:49:30 0 [Note] /usr/local/Cellar/mysql56/5.6.27/bin/mysqld (mysqld 5.6.27) starting as process 62810 ...
2016-01-12 08:49:30 62810 [Warning] Setting lower_case_table_names=2 because file system for /usr/local/var/mysql/ is case insensitive
2016-01-12 08:49:30 62810 [Note] Plugin 'FEDERATED' is disabled.
00:49:30 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68101 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7fff57e48f00
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fff57e489a0 thread_stack 0x40000
0 mysqld 0x0000000108066665 my_print_stacktrace + 61
1 mysqld 0x0000000107eb5db2 handle_fatal_signal + 696
2 libsystem_platform.dylib 0x00007fff848bb52a _sigtramp + 26
3 mysqld 0x00000001086464c8 PSI_server + 0
4 mysqld 0x0000000107fcfae9 _Z14open_table_defP3THDP11TABLE_SHAREj + 4816
5 mysqld 0x0000000107ee80ff _Z15get_table_shareP3THDP10TABLE_LISTPKcjjPij + 448
6 mysqld 0x0000000107eeb27b _Z10open_tableP3THDP10TABLE_LISTP18Open_table_context + 2019
7 mysqld 0x0000000107eed68b _Z11open_tablesP3THDPP10TABLE_LISTPjjP19Prelocking_strategy + 1252
8 mysqld 0x0000000107eedfcc _Z20open_and_lock_tablesP3THDP10TABLE_LISTbjP19Prelocking_strategy + 50
9 mysqld 0x0000000107f617a5 _Z11plugin_initPiPPci + 2678
10 mysqld 0x0000000107ff7fa6 _Z11mysqld_mainiPPc + 2674
11 libdyld.dylib 0x00007fff8896d5ad start + 1
12 ??? 0x0000000000000007 0x0 + 7
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 0
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
160112 08:49:30 mysqld_safe mysqld from pid file /usr/local/var/mysql/Services-iMac.local.pid ended
160112 08:49:43 mysqld_safe Starting mysqld daemon with databases from /usr/local/var/mysql
2016-01-12 08:49:43 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-01-12 08:49:43 0 [Note] /usr/local/Cellar/mysql56/5.6.27/bin/mysqld (mysqld 5.6.27) starting as process 62980 ...
2016-01-12 08:49:43 62980 [Warning] Setting lower_case_table_names=2 because file system for /usr/local/var/mysql/ is case insensitive
2016-01-12 08:49:43 62980 [Note] Plugin 'FEDERATED' is disabled.
00:49:43 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68101 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7fff573c1f00
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fff573c19a0 thread_stack 0x40000
0 mysqld 0x0000000108aed665 my_print_stacktrace + 61
1 mysqld 0x000000010893cdb2 handle_fatal_signal + 696
2 libsystem_platform.dylib 0x00007fff848bb52a _sigtramp + 26
3 mysqld 0x00000001090cd4c8 PSI_server + 0
4 mysqld 0x0000000108a56ae9 _Z14open_table_defP3THDP11TABLE_SHAREj + 4816
5 mysqld 0x000000010896f0ff _Z15get_table_shareP3THDP10TABLE_LISTPKcjjPij + 448
6 mysqld 0x000000010897227b _Z10open_tableP3THDP10TABLE_LISTP18Open_table_context + 2019
7 mysqld 0x000000010897468b _Z11open_tablesP3THDPP10TABLE_LISTPjjP19Prelocking_strategy + 1252
8 mysqld 0x0000000108974fcc _Z20open_and_lock_tablesP3THDP10TABLE_LISTbjP19Prelocking_strategy + 50
9 mysqld 0x00000001089e87a5 _Z11plugin_initPiPPci + 2678
10 mysqld 0x0000000108a7efa6 _Z11mysqld_mainiPPc + 2674
11 libdyld.dylib 0x00007fff8896d5ad start + 1
12 ??? 0x0000000000000007 0x0 + 7
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 0
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
160112 08:49:43 mysqld_safe mysqld from pid file /usr/local/var/mysql/Services-iMac.local.pid ended
➜ mysql git:(master) sudo kill -9 62810
kill: 62810: No such process
➜ mysql git:(master) sudo cat Services-iMac.local.err
160112 08:49:29 mysqld_safe Starting mysqld daemon with databases from /usr/local/var/mysql
2016-01-12 08:49:30 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-01-12 08:49:30 0 [Note] /usr/local/Cellar/mysql56/5.6.27/bin/mysqld (mysqld 5.6.27) starting as process 62810 ...
2016-01-12 08:49:30 62810 [Warning] Setting lower_case_table_names=2 because file system for /usr/local/var/mysql/ is case insensitive
2016-01-12 08:49:30 62810 [Note] Plugin 'FEDERATED' is disabled.
00:49:30 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68101 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7fff57e48f00
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fff57e489a0 thread_stack 0x40000
0 mysqld 0x0000000108066665 my_print_stacktrace + 61
1 mysqld 0x0000000107eb5db2 handle_fatal_signal + 696
2 libsystem_platform.dylib 0x00007fff848bb52a _sigtramp + 26
3 mysqld 0x00000001086464c8 PSI_server + 0
4 mysqld 0x0000000107fcfae9 _Z14open_table_defP3THDP11TABLE_SHAREj + 4816
5 mysqld 0x0000000107ee80ff _Z15get_table_shareP3THDP10TABLE_LISTPKcjjPij + 448
6 mysqld 0x0000000107eeb27b _Z10open_tableP3THDP10TABLE_LISTP18Open_table_context + 2019
7 mysqld 0x0000000107eed68b _Z11open_tablesP3THDPP10TABLE_LISTPjjP19Prelocking_strategy + 1252
8 mysqld 0x0000000107eedfcc _Z20open_and_lock_tablesP3THDP10TABLE_LISTbjP19Prelocking_strategy + 50
9 mysqld 0x0000000107f617a5 _Z11plugin_initPiPPci + 2678
10 mysqld 0x0000000107ff7fa6 _Z11mysqld_mainiPPc + 2674
11 libdyld.dylib 0x00007fff8896d5ad start + 1
12 ??? 0x0000000000000007 0x0 + 7
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 0
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
160112 08:49:30 mysqld_safe mysqld from pid file /usr/local/var/mysql/Services-iMac.local.pid ended
160112 08:49:43 mysqld_safe Starting mysqld daemon with databases from /usr/local/var/mysql
2016-01-12 08:49:43 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-01-12 08:49:43 0 [Note] /usr/local/Cellar/mysql56/5.6.27/bin/mysqld (mysqld 5.6.27) starting as process 62980 ...
2016-01-12 08:49:43 62980 [Warning] Setting lower_case_table_names=2 because file system for /usr/local/var/mysql/ is case insensitive
2016-01-12 08:49:43 62980 [Note] Plugin 'FEDERATED' is disabled.
00:49:43 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68101 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7fff573c1f00
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fff573c19a0 thread_stack 0x40000
0 mysqld 0x0000000108aed665 my_print_stacktrace + 61
1 mysqld 0x000000010893cdb2 handle_fatal_signal + 696
2 libsystem_platform.dylib 0x00007fff848bb52a _sigtramp + 26
3 mysqld 0x00000001090cd4c8 PSI_server + 0
4 mysqld 0x0000000108a56ae9 _Z14open_table_defP3THDP11TABLE_SHAREj + 4816
5 mysqld 0x000000010896f0ff _Z15get_table_shareP3THDP10TABLE_LISTPKcjjPij + 448
6 mysqld 0x000000010897227b _Z10open_tableP3THDP10TABLE_LISTP18Open_table_context + 2019
7 mysqld 0x000000010897468b _Z11open_tablesP3THDPP10TABLE_LISTPjjP19Prelocking_strategy + 1252
8 mysqld 0x0000000108974fcc _Z20open_and_lock_tablesP3THDP10TABLE_LISTbjP19Prelocking_strategy + 50
9 mysqld 0x00000001089e87a5 _Z11plugin_initPiPPci + 2678
10 mysqld 0x0000000108a7efa6 _Z11mysqld_mainiPPc + 2674
11 libdyld.dylib 0x00007fff8896d5ad start + 1
12 ??? 0x0000000000000007 0x0 + 7
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 0
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
160112 08:49:43 mysqld_safe mysqld from pid file /usr/local/var/mysql/Services-iMac.local.pid ended
Trying to start mysql doesn't load it.
➜ mysql git:(master) mysql.server start
Starting MySQL
. ERROR! The server quit without updating PID file (/usr/local/var/mysql/Services-iMac.local.pid).
This happened to me after re-installing mysql.
I originally installed the default version (mysql 5.7.x), but downgraded back to mysql56.
The issue may lie on the unclean shutdown of 5.7 as I issued a brew uninstall mysql while my launchd-managed mysqld instance was running. The symlinks, data, and old 5.7.x-related stuff were not cleaned thoroughly.
As to the error, the solution so far was to follow the recommendation here: http://bugs.mysql.com/bug.php?id=70431, and i.e. adding the skip-grant-tables in your my.cnf (create one under /etc if you don't have one)

mysqld_safe --skip-grant-tables not starting up in IBM i (AS400 / iseries)

I am trying to reset the grant privileges (I removed root privileges).
After I shut down the mysql daemon, I try to restart safe mode with: mysqld_safe --skip-grant-tables
I get this error & the default port is NOT listening after the attempted restart.:
150402 13:40:27 mysqld_safe Logging to '/usr/local/mysqldata/TDIDEV2.T***C.COM.err'.
150402 13:40:29 mysqld_safe Starting mysqld daemon with databases from /usr/local/mysqldata
150402 13:40:30 mysqld_safe mysqld from pid file /usr/local/mysqldata/TDIDEV2.T***C.COM.pid ended
Err file:
ñõðôðò#ñõzðñzñó#”¨¢˜“„m¢†…#⣙£‰•‡#”¨¢˜“„#„…”–•#¦‰£ˆ#„£‚¢…¢#†™–”#a¤¢™a“–ƒ“a”¨¢˜“„£%ñõðôðò#ñõzðñzñó#ºæ™•‰•‡»#â…££‰•‡#“–¦…™mƒ¢…m£‚“…m•”…¢~ò#‚…ƒ¤¢…#†‰“…#¢¨¢£…”#†–™#a¤¢™a“–ƒ“a”¨¢˜“„£a#‰¢#ƒ¢…#‰•¢…•¢‰£‰¥…%ñõðôðò#ñõzðñzñó#ºæ™•‰•‡»#Ö•…#ƒ•#–•“¨#¤¢…#£ˆ…#``¤¢…™#¢¦‰£ƒˆ#‰†#™¤••‰•‡#¢#™––£%%150402 15:01:13 [Note] Plugin 'FEDERATED' is disabled.
150402 15:01:13 InnoDB: Initializing buffer pool, size = 8.0M
150402 15:01:13 InnoDB: Completed initialization of buffer pool
InnoDB: Error: pthread_create returned 11
ñõðôðò#ñõzðñzñô#”¨¢˜“„m¢†…#”¨¢˜“„#†™–”#—‰„#†‰“…#a¤¢™a“–ƒ“a”¨¢˜“„£aãÄÉÄÅåòKãÉãÓÅÄÁãÁÉÕÃKÃÖÔK—‰„#…•„…„%
pthread_create 11 indicates an out of memory condition. Did you also change /usr/local/mysql/bin/my.cnf? 8.0M seems very low for innodb_buffer_pool_size, especially for IBM i. Mine is set to 100M.

MariaDB 10 CentOS 7 moving datadir woes

Brand new "minimal" install of CentOS 7 along with MariaDB 10. I have an additional mounted mirrored volume that I want to use for the datadir. Startup sequence is fine and completes normally when my.cnf [mysqld] is commented out. I've copied the data..
sudo cp -R -p /var/lib/mysql/* /mnt/mysql/
The permissions are identical to those of the original. The volume is in /etc/fstab and mounts fine
/dev/sdb1 /mnt/mysql xfs defaults 0 0
[root#femur mysql]# ls -la
total 110632
drwxr-xr-x. 5 mysql mysql 4096 Oct 20 15:27 .
drwxr-xr-x. 3 root root 18 Oct 16 16:46 ..
-rw-rw----. 1 mysql mysql 16384 Oct 20 15:27 aria_log.00000001
-rw-rw----. 1 mysql mysql 52 Oct 20 15:27 aria_log_control
-rw-r-----. 1 mysql root 7005 Oct 20 13:49 femur.err
-rw-rw----. 1 mysql mysql 12582912 Oct 20 15:27 ibdata1
-rw-rw----. 1 mysql mysql 50331648 Oct 20 15:27 ib_logfile0
-rw-rw----. 1 mysql mysql 50331648 Oct 20 12:21 ib_logfile1
-rw-rw----. 1 mysql mysql 0 Oct 20 12:22 multi-master.info
drwx--x--x. 2 mysql mysql 4096 Oct 20 12:21 mysql
drwx------. 2 mysql mysql 4096 Oct 20 13:37 performance_schema
drwxr-xr-x. 2 mysql mysql 6 Oct 20 12:21 test
this is in my.cnf
!includedir /etc/my.cnf.d
[mysqld]
log_error = /var/log/mysql-error.log
user = mysql
datadir = /mnt/mysql
socket = /mnt/mysql/mysql.sock
This is what I get when I try to start it...
'[root#femur mysql]# sudo systemctl start mysql.service
Job for mysql.service failed. See 'systemctl status mysql.service' and 'journalctl -xn' for details.'
Neither of those two files says much, but this is in /var/log/mysql-error.log
141020 16:07:09 mysqld_safe Starting mysqld daemon with databases from /mnt/mysql
141020 16:07:09 [Warning] Can't create test file /mnt/mysql/femur.lower-test
141020 16:07:09 [Note] InnoDB: Using mutexes to ref count buffer pool pages
141020 16:07:09 [Note] InnoDB: The InnoDB memory heap is disabled
141020 16:07:09 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
141020 16:07:09 [Note] InnoDB: Memory barrier is not used
141020 16:07:09 [Note] InnoDB: Compressed tables use zlib 1.2.7
141020 16:07:09 [Note] InnoDB: Using Linux native AIO
141020 16:07:09 [Note] InnoDB: Using CPU crc32 instructions
141020 16:07:09 [Note] InnoDB: Initializing buffer pool, size = 128.0M
141020 16:07:09 [Note] InnoDB: Completed initialization of buffer pool
2014-10-20 16:07:09 7f6cb59c9880 InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
141020 16:07:09 [ERROR] InnoDB: os_file_get_status() failed on './ibdata1'. Can't determine file permissions
141020 16:07:09 [ERROR] InnoDB: The system tablespace must be writable!
141020 16:07:09 [ERROR] Plugin 'InnoDB' init function returned error.
141020 16:07:09 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
141020 16:07:09 [ERROR] mysqld: File '/mnt/mysql/aria_log_control' not found (Errcode: 13 "Permission denied")
141020 16:07:09 [ERROR] mysqld: Got error 'Can't open file' when trying to use aria control file '/mnt/mysql/aria_log_control'
141020 16:07:09 [ERROR] Plugin 'Aria' init function returned error.
141020 16:07:09 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
141020 16:07:09 [Note] Plugin 'FEEDBACK' is disabled.
141020 16:07:09 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
141020 16:07:09 [ERROR] Unknown/unsupported storage engine: InnoDB
141020 16:07:09 [ERROR] Aborting
141020 16:07:09 [Note] /usr/sbin/mysqld: Shutdown complete
141020 16:07:09 mysqld_safe mysqld from pid file /mnt/mysql/femur.pid ended
http://www.reddit.com/r/linuxadmin/comments/2ebhpf/adventures_in_moving_mariadb_data_folder/ helped a bit, but I wasn't able to get it to work.
Any help would be greatly appreciated.
The issue is indeed SELinux; you need to do three things before MariaDB / MySQL will start on CentOS 7:
Ensure the user:group is mysql:mysql
Set the SELinux tag to mysqld_db_t
Set the SELinux user to system_u
This is as simple as:
chcon -Rt mysqld_db_t /database/db
chcon -Ru system_u /database/db
chown -R mysql:mysql /database/db
The whole thing I needed to do after plugging in a disk is below:
cfdisk /dev/sdb
pvcreate /dev/sdb1
vgcreate database /dev/sdb1
lvcreate -l 100%FREE -n db database
mkfs.ext4 /dev/database/db
mkdir /database
mount /database
mkdir /database/db
chcon -Rt mysqld_db_t /database/db
chcon -Ru system_u /database/db
chown -R mysql:mysql /database/db
systemctl start mariadb
Well that was interesting...
It turns out, that CentOS 7 "minimal" installs SELinux, which apparently was preventing mysql from writing to the mounted mirrored set. I was looking for security items that I might not have thought about and found it right there in the docs. I had previously thought (obviously erroneously) that SELinux was a distribution, not a module. Once I ran the test to see if it was there....
getenforce
I temporarily disabled it to test.
setenforce 0
Finally, I was able to start MariaDB with the directory in the mirrored set as the datadir and no errors. To make this change permanent (because this server is behind a firewall), in /etc/selinux/config, I made
- SELINUX=enforcing
+ SELINUX=disabled
I hope this helps someone else. Have a great day!
I found this step by step guide working for me: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/SELinux_Users_and_Administrators_Guide/sect-Managing_Confined_Services-MariaDB-Configuration_Examples.html
You must install:
yum install policycoreutils-python
Guide:
View the SELinux context of the default database location for mysql:
~]# ls -lZ /var/lib/mysql
drwx------. mysql mysql system_u:object_r:mysqld_db_t:s0 mysql
This shows mysqld_db_t which is the default context element for the location of database files. This context will have to be manually applied to the new database location that will be used in this example in order for it to function properly.
Stop the mysqld daemon:
~]# systemctl stop mariadb.service
Create a new directory for the new location of the database(s). In this example, /mysql/ is used:
~]# mkdir -p /mysql
Copy the database files from the old location to the new location:
~]# cp -R /var/lib/mysql/* /mysql/
Change the ownership of this location to allow access by the mysql user and group. This sets the traditional Unix permissions which SELinux will still observe:
~]# chown -R mysql:mysql /mysql
Run the following command to see the initial context of the new directory:
~]# ls -lZ /mysql
drwxr-xr-x. mysql mysql unconfined_u:object_r:usr_t:s0 mysql
The context usr_t of this newly created directory is not currently suitable to SELinux as a location for MariaDB database files. Once the context has been changed, MariaDB will be able to function properly in this area.
Open the main MariaDB configuration file /etc/my.cnf with a text editor and modify the datadir option so that it refers to the new location. In this example the value that should be entered is /mysql:
[mysqld]
datadir=/mysql
Save this file and exit.
Start mysqld. The service should fail to start, and a denial message will be logged to the /var/log/messages file:
~]# systemctl start mariadb.service
Job for mariadb.service failed. See 'systemctl status postgresql.service' and 'journalctl -xn' for details.
However, if the audit daemon is running and with him the setroubleshoot service, the denial will be logged to the /var/log/audit/audit.log file instead:
SELinux is preventing /usr/libexec/mysqld "write" access on /mysql. For complete SELinux messages. run sealert -l b3f01aff-7fa6-4ebe-ad46-abaef6f8ad71
The reason for this denial is that /mysql/ is not labeled correctly for MariaDB data files. SELinux is stopping MariaDB from having access to the content labeled as usr_t. Perform the following steps to resolve this problem:
Run the following command to add a context mapping for /mysql/. Note that the semanageutility is not installed by default. If it missing on your system, install the policycoreutils-pythonpackage.
**~]# semanage fcontext -a -t mysqld_db_t "/mysql(/.*)?"**
This mapping is written to the /etc/selinux/targeted/contexts/files/file_contexts.local file:
~]# grep -i mysql /etc/selinux/targeted/contexts/files/file_contexts.local
/mysql(/.*)? system_u:object_r:mysqld_db_t:s0
Now use the restorecon utility to apply this context mapping to the running system:
**~]# restorecon -R -v /mysql**
Now that the /mysql/ location has been labeled with the correct context for MariaDB, mysqldstarts:
~]# systemctl start mariadb.service
Confirm the context has changed for /mysql/:
~]$ ls -lZ /mysql
drwxr-xr-x. mysql mysql system_u:object_r:mysqld_db_t:s0 mysql
The location has been changed and labeled, and mysqld has started successfully. At this point all running services should be tested to confirm normal operation.