I recently rsync a bunch of MYI, MYD, and FRM files from one MySQL 5.0.95 server to another, both using MyISAM. Due to a forgotten hot-fix, it turns out that the original database was configured to default tables to utf8, and the new database was set to latin1. When we turned on the new database, it worked, but shortly thereafter, several queries caused it to segfault multiple times in a row. I suspect it has to do with the charset difference, but I don't understand the stack trace well enough to get to the bottom of the issue. Below is the decoded stack trace, and then the mysql.log.
Stack trace:
0x819b416 handle_segfault + 806
0xb76246a1 _end + -1358137335
0x8193936 Protocol::store_string_aux(char const*, unsigned int, charset_info_st*, charset_info_st*) + 166
0x8193a2b Protocol_simple::store(Field*) + 139
0x818f671 select_send::send_data(List<Item>&) + 161
0x81e5735 init_read_record_seq(st_join_table*) + 373
0x81e8629 join_read_always_key_or_null(st_join_table*) + 2153
0x81ecbc7 sub_select(JOIN*, st_join_table*, bool) + 135
0x81ef127 JOIN::rollup_write_data(unsigned int, st_table*) + 2487
0x81ff6ed JOIN::exec() + 1341
0x81fc72b _Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select + 219
0x82015a6 handle_select(THD*, st_lex*, select_result*, unsigned long) + 326
0x81b1240 mysql_execute_command(THD*) + 7728
0x81b6140 mysql_parse(THD*, char const*, unsigned int, char const**) + 640
0x81b88d0 dispatch_command(enum_server_command, THD*, char*, unsigned int) + 3408
0x81b9f30 handle_one_connection + 2624
0xb785f10f _end + -1355799945
0xb767c74e _end + -1357776714
mysql.log:
160623 17:39:52 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=209715200
read_buffer_size=262144
max_used_connections=4
max_connections=1500
threads_connected=4
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 3660800 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd=0x86799f8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xa648d168, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x819b416
0xb75d56a1
0x8193936
0x8193a2b
0x818f671
0x81e5735
0x81e8629
0x81ecbc7
0x81ef127
0x81ff6ed
0x81fc72b
0x82015a6
0x81b1240
0x81b6140
0x81b88d0
0x81b9f30
0xb781010f
0xb762d74e
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x875aaf0 = SELECT * FROM Data_197_21268_1 ORDER BY ID LIMIT 1
thd->thread_id=8
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Thank you in advance for any help you can offer. Also, I know I have to upgrade MySQL, this is in the works.
Related
I hope someone might have additional ideas on what is going wrong and how we could solve it.
Our MySQL Database keeps crashing due to Signal 11 (Segfault) from the OS.
Quick overview of the server:
Centos 7
MySQL Server Community Edition 8.0.23 via the official mysql repository
4 cores
8GB RAM
100GB HDD with 85GB free space
We started to get crashes with the following log:
15:09:23 UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x7f678c000d20
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f6800141c30 thread_stack 0x46000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x21e102d]
/usr/sbin/mysqld(handle_fatal_signal+0x313) [0x1009643]
/lib64/libpthread.so.0(+0xf5d0) [0x7f68181975d0]
/usr/sbin/mysqld() [0x24c7e0e]
/usr/sbin/mysqld(trx_undo_update_rec_get_update(unsigned char const*, dict_index_t const*, unsigned long, unsigned long, unsigned long, unsigned long, trx_t*, mem_block_info_t*, upd_t**, lob::undo_vers_t*, type_cmpl_t&)+0x7bb) [0x24c9d3b]
/usr/sbin/mysqld(trx_undo_prev_version_build(unsigned char const*, mtr_t*, unsigned char const*, dict_index_t const*, unsigned long*, mem_block_info_t*, unsigned char**, mem_block_info_t*, dtuple_t const**, unsigned long, lob::undo_vers_t*)+0x43d) [0x24cb7cd]
/usr/sbin/mysqld(row_vers_build_for_consistent_read(unsigned char const*, mtr_t*, dict_index_t*, unsigned long**, ReadView*, mem_block_info_t**, mem_block_info_t*, unsigned char**, dtuple_t const**, lob::undo_vers_t*)+0x279) [0x2476cb9]
/usr/sbin/mysqld(row_search_mvcc(unsigned char*, page_cur_mode_t, row_prebuilt_t*, unsigned long, unsigned long)+0x2b34) [0x2462c14]
/usr/sbin/mysqld(ha_innobase::index_read(unsigned char*, unsigned char const*, unsigned int, ha_rkey_function)+0x29a) [0x22c7e2a]
/usr/sbin/mysqld(handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function)+0x347) [0x11184e7]
/usr/sbin/mysqld() [0xe8c415]
/usr/sbin/mysqld(join_read_const_table(JOIN_TAB*, POSITION*)+0x87) [0xe8c4d7]
/usr/sbin/mysqld(JOIN::extract_func_dependent_tables()+0x4c3) [0xeaff03]
/usr/sbin/mysqld(JOIN::make_join_plan()+0x10e7) [0xec0fb7]
/usr/sbin/mysqld(JOIN::optimize()+0xbcb) [0xec247b]
/usr/sbin/mysqld(SELECT_LEX::optimize(THD*)+0xb6) [0xf1ebe6]
/usr/sbin/mysqld(SELECT_LEX_UNIT::optimize(THD*, TABLE*, bool)+0x7b) [0xf92dab]
/usr/sbin/mysqld(Sql_cmd_dml::execute_inner(THD*)+0x3d) [0xf1d63d]
/usr/sbin/mysqld(Sql_cmd_dml::execute(THD*)+0x525) [0xf272e5]
/usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0x9f0) [0xecb7a0]
/usr/sbin/mysqld(dispatch_sql_command(THD*, Parser_state*)+0x42b) [0xecff7b]
/usr/sbin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x1ed6) [0xed22f6]
/usr/sbin/mysqld(do_command(THD*)+0x19c) [0xed302c]
/usr/sbin/mysqld() [0xffa6b8]
/usr/sbin/mysqld() [0x2782d4e]
/lib64/libpthread.so.0(+0x7dd5) [0x7f681818fdd5]
/lib64/libc.so.6(clone+0x6d) [0x7f6816575ead]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f678c009a68): SELECT job_project_id AS projectid, user_id_fk AS userid, job_name AS jobname FROM job WHERE job_id=2463
Connection ID (thread ID): 9
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Reading through the logs it seems that the culprit is the following query:
SELECT job_project_id AS projectid, user_id_fk AS userid,
job_name AS jobname
FROM job
WHERE job_id=2463
The table job has the following columns:
Columns:
job_id int AI PK
job_name varchar(50)
job_status json
job_fileset_list json
user_id_fk int
job_project_id int
job_start_time datetime
job_end_time datetime
job_pipeline json
job_task_blobs json
job_exe_blobs json
The my.cnf is:
# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/8.0/en/server-configuration-defaults.html
[mysqld]
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
innodb_buffer_pool_size = 5500M
#
# Remove the leading "# " to disable binary logging
# Binary logging captures changes between backups and is enabled by
# default. It's default setting is log_bin=binlog
# disable_log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
#
# Remove leading # to revert to previous value for default_authentication_plugin,
# this will increase compatibility with older clients. For background, see:
# https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_default_authentication_plugin
# default-authentication-plugin=mysql_native_password
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
max_connections=500
We have only ever seen this error with above query. This segfault occurs semi-regularly, where we can run the query fine a couple of times and then it segfaults. We tried running the queries sequentially, no difference. We tried running them with breaks of 1 seconds, no difference.
The server definitely does not go out of memory, there is enough around when the segfault comes along.
I am completely out of ideas and I also cannot find people with the same problem.
The closest I found was a bug report
however this is for a much older version of mysql, and did not offer any resolution.
If anyone has any ideas, hunches or even had this problem themselves before, I would very much appreciate your input!
So this is not really a solution to the bug/problem itself.
But thanks to Don we have narrowed the cause down to most probably be MySQL itself. We have switched to MariaDB and the problem does not occur anymore. We will stick with MariaDB for the forseeable future.
I'm using MySQL 5.7.14 x64 on Windows Server 2008 R2
Sometimes (randomly times at day) mysql crashing with this stack trace
11:44:40 UTC - mysqld got exception 0x80000003 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
key_buffer_size=8388608
read_buffer_size=65536
max_used_connections=369
max_threads=2800
thread_count=263
connection_count=263
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3195125 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x2ee2b72b0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
13fe1bad2 mysqld.exe!my_sigabrt_handler()[my_thr_init.c:449]
1401c7979 mysqld.exe!raise()[winsig.c:587]
1401c6870 mysqld.exe!abort()[abort.c:82]
13ff1dd38 mysqld.exe!ut_dbg_assertion_failed()[ut0dbg.cc:67]
13ff1df51 mysqld.exe!ib::fatal::~fatal()[ut0ut.cc:916]
13ff0e008 mysqld.exe!buf_LRU_check_size_of_non_data_objects()[buf0lru.cc:1219]
13ff0f4ab mysqld.exe!buf_LRU_get_free_block()[buf0lru.cc:1303]
1400305cb mysqld.exe!buf_block_alloc()[buf0buf.cc:557]
13ff3767e mysqld.exe!mem_heap_create_block_func()[mem0mem.cc:319]
13ff37499 mysqld.exe!mem_heap_add_block()[mem0mem.cc:408]
13ffd87f4 mysqld.exe!RecLock::lock_alloc()[lock0lock.cc:1441]
13ffd795c mysqld.exe!RecLock::create()[lock0lock.cc:1534]
13ffd73a6 mysqld.exe!RecLock::add_to_waitq()[lock0lock.cc:1735]
13ffdcaaa mysqld.exe!lock_rec_lock_slow()[lock0lock.cc:2007]
13ffdc6ce mysqld.exe!lock_rec_lock()[lock0lock.cc:2081]
13ffd8cc7 mysqld.exe!lock_clust_rec_read_check_and_lock()[lock0lock.cc:6307]
140076fe3 mysqld.exe!row_ins_set_shared_rec_lock()[row0ins.cc:1502]
140072927 mysqld.exe!row_ins_check_foreign_constraint()[row0ins.cc:1739]
140072de8 mysqld.exe!row_ins_check_foreign_constraints()[row0ins.cc:1932]
140075d69 mysqld.exe!row_ins_sec_index_entry()[row0ins.cc:3356]
1400758a6 mysqld.exe!row_ins_index_entry_step()[row0ins.cc:3583]
140071b30 mysqld.exe!row_ins()[row0ins.cc:3721]
14007755a mysqld.exe!row_ins_step()[row0ins.cc:3907]
13ffaad50 mysqld.exe!row_insert_for_mysql_using_ins_graph()[row0mysql.cc:1735]
13fe7a7d3 mysqld.exe!ha_innobase::write_row()[ha_innodb.cc:7489]
13f6e5531 mysqld.exe!handler::ha_write_row()[handler.cc:7891]
13f8e54de mysqld.exe!write_record()[sql_insert.cc:1860]
13f8e916a mysqld.exe!read_sep_field()[sql_load.cc:1222]
13f8e7af4 mysqld.exe!mysql_load()[sql_load.cc:563]
13f716e86 mysqld.exe!mysql_execute_command()[sql_parse.cc:3649]
13f7194b3 mysqld.exe!mysql_parse()[sql_parse.cc:5565]
13f71267d mysqld.exe!dispatch_command()[sql_parse.cc:1430]
13f71368a mysqld.exe!do_command()[sql_parse.cc:997]
13f6d82bc mysqld.exe!handle_connection()[connection_handler_per_thread.cc:300]
140105122 mysqld.exe!pfs_spawn_thread()[pfs.cc:2191]
13fe1b93b mysqld.exe!win_thread_start()[my_thread.c:38]
1401c73ef mysqld.exe!_callthreadstartex()[threadex.c:376]
1401c763a mysqld.exe!_threadstartex()[threadex.c:354]
772859bd kernel32.dll!BaseThreadInitThunk()
773ba2e1 ntdll.dll!RtlUserThreadStart()
At this time active only 2 transactions
---TRANSACTION 1111758443, ACTIVE 565 sec
mysql tables in use 7, locked 7
7527 lock struct(s), heap size 876752, 721803 row lock(s), undo log entries 379321
MySQL thread id 166068, OS thread handle 1508, query id 112695582 localhost converter Waiting for table level lock
delete from pl
using
import_k2b_product_links ipl inner join k2b_products pSource on ipl.src_product = pSource.article and pSource.account_id = 22
inner join k2b_products pDest on ipl.dst_product = pDest.article and pDest.account_id = 22
inner join k2b_product_links pl on pl.src_product_id = pSource.id and pl.dst_product = pDest.id
where ipl.action = 1
---TRANSACTION 1111759716, ACTIVE 496 sec inserting, thread declared inside InnoDB 1
mysql tables in use 4, locked 4
7 lock struct(s), heap size 1304535248, 102060778 row lock(s), undo log entries 1
MySQL thread id 19436, OS thread handle 11664, query id 112301161 localhost exchange_central
LOAD DATA INFILE 'd:/kdm/temp/webCentral/ufrd1uwx.v2r'
INTO TABLE k2b_orders
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(id_status, dt, account_id, sms_sended, params, update_ts, exported, id_editor, dt_offset, device_id, gen, changer_device_id, total, creator_device_id, id, dt_server, device_category_id, original_params, order_num, sended, editor_comment, admin_comment)
I don't understand why transaction 1111758443 Waiting for table level lock?
And why transaction 1111759716 lock 102060778 rows while it load just only one from external file and it showed in undo log entries 1?
Which investigation I must done for known reason of this enormous locks and crash.
Thanks!
Two things make me think that the crash is not the 'real' problem.
Both queries in the log show 'huge' times, such as ACTIVE 565 sec.
And these are all quite large:
max_used_connections=369
max_threads=2800
thread_count=263
connection_count=263
When there are hundreds of threads simultaneously active, InnoDB stumbles over itself. Throughput stalls, and latency goes through the roof.
One cure is to avoid so many connections. This is sometimes best done at the client. What is the client? For example, Apache has MaxClients. A dozen Apaches, each with MaxClients = 50 would be trying to open 600 connections. Probably one Apache cannot effectively handle 50 threads at once. Lower that number.
Are there any VIEWs deceiving us?
Another thing to do is to pursue table level lock. Let's see SHOW CREATE TABLE for the tables involved. Check for appropriate indexes.
import_k2b_product_links: INDEX(action, ...)
k2b_products: INDEX(account_id, src_product) -- in either order
k2b_products: INDEX(account_id, dest_product) -- in either order
k2b_product_links: INDEX(src_product_id, dest_product_id) -- or PK, see below
Is k2b_product_links a many:many mapping table? If so, get rid of id auto_increment as discussed Here .
The index suggestions, if useful, could speed up the DELETE, thereby cutting down on possible contention.
I have an application that has been release in the App Store. About maybe 1% - 2% of my users are reporting that the App is crashing. This isn't exactly unexpected behavior, so I have asked for the crash logs. Here is the last exception backtrace (the part that actually shows the issue):
Last Exception Backtrace:
0 CoreFoundation 0x2ecc5e7e __exceptionPreprocess + 126
1 libobjc.A.dylib 0x390226c2 objc_exception_throw + 34
2 CoreFoundation 0x2ecc97b2 -[NSObject(NSObject) doesNotRecognizeSelector:] + 198
3 CoreFoundation 0x2ecc80aa ___forwarding___ + 702
4 CoreFoundation 0x2ec16dc4 __forwarding_prep_0___ + 20
5 MyAppName 0x000874d0 0x6c000 + 111824
6 UIKit 0x3157c310 -[UITableView _createPreparedCellForGlobalRow:withIndexPath:] + 404
7 UIKit 0x315246c8 -[UITableView _updateVisibleCellsNow:] + 1796
8 UIKit 0x31523eec -[UITableView layoutSubviews] + 180
As you can see the issue is on line 2, NSObject(NSObject) doesNotRecognizeSelector.
This tells me what is happening, but obviously not where or what. So go to line 5, and here is my App calling an unknown method. For some reason this is not symbolicated. It seems like this method is probably what is causing the issue.
Every single report (I have about 10 of them right now) is identical to this here. Any idea how I can see the proper symbolication? Thanks!
I have experienced these kinds of crash before.
It was due to data type issue in all my cases.
For example trying to call a method specific for one data type on another data type.
i.e
trying to call integerValue method on a dictionary or trying to call length method on a string but in essence it is a dictionary so it crashes with [NSObject doesNotRecognizeSelector:]:
It's better check data type and call the method on the object to avoid crashes like this.
When we're calling APIs , it can easily happen as sometimes APIs can return wrong datatypes or unexpected response.
I have been trying to understand why I get "SEGMENTATION FAULT" while running a program written in C languagge.
I tried gdb.
This is the message I got:
Program received signal SIGSEGV, Segmentation fault.
0x0016e5e0 in mysql_slave_send_query () from /usr/lib/libmysqlclient.so.16
(gdb) step
Single stepping until exit from function mysql_slave_send_query,
which has no line number information.
Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
The fact is that I get no compiler error/warning messages but the program doesn't work
properly.
Can anyone help me?
My query is:
char query[512];
sprintf(query, "SELECT t1.Art_Acquisto as 'Cod',t2.Des_Articolo as 'Descrizione',t4.Cod_Categoria as 'Cat.',t1.Data_Acquisto as" "'data',t1.Netto_Acquisto as'Importo',t3.Des_Fornitore as 'Fornitore' from "
"Aquisti as t1, Articoli as t2, Fornitori as t3, Categorie as t4 where t1.Art_Acquisto = t2.Cod_Articolo and "
"t1.fornitoreM = t3.codiceF and t4.codiceC = t2.categoriaA and Art_Acquisto ='%s'order by Data_Acquisto;",Cod_Articolo);
Check your arguments to mysql_slave_send_query, especially the length and the proper initialisation/allocation of the other arguments.
"Segmentation fault" means that your program is accessing an invalid memory location during the execution of the function named mysql_slave_send_query.
Therefore this means there is a bug in your C program.
How big is the contents of Cod_Articolo? Your query is nearly 400 characters, and the buffer you're storing it in is only 512 characters - if Cod_Articolo is over 110 characters, your code will break.
If this is the problem, you can prevent it by checking the size of Cod_Articolo before writing it into the query string. Even better, you could calculate the size the query string needs to be, and allocate exactly the right amount.
It's also good practice to use snprintf instead of sprintf, as then you can ensure you don't copy too many characters into the destination string.
I got an "unaligned memory accesses not supported error" and did a Google search for that
but there were no clear explanations.
The whole error message is:
/c:\cuda\include\math_functions_dbl_ptx1.h(1308): Error: Unaligned memory accesses not supported
The following code caused the error:
for (j = low; j <= high; j++)
The variables j and high are declared as int.
This kind of error was encountered before but resolved by itself (I did nothing).
Can anybody explain about this matter?
Theory
On many machines - but not Intel IA32 ones or their relatives - if you want to access a 2-byte quantity (integer), the address must be even, not odd; if you want to access a 4-byte quantity (integer or float) the address must be a multiple of 4 bytes; if you want to access an 8-byte quantity (integer or double), the address must be a multiple of 8 bytes; and so on.
Superficially, then, you have somehow got your code trying dereference a pointer which has bits set in the low-order parts when you should not. For example, one way of forcing the issue (in C) would be:
long l = 0x12345678;
void *v = (char *)&l + 1;
long *lp = v;
l = *lp;
By the time you've gone through the pointer arithmetic, the address in lp is not 4-byte (or 8-byte) aligned; it is off-by-one because of the +1. The last line would give an unaligned memory access pointer.
Practice
Since you have not shown the declarations for your code, we can't be sure what causes the trouble (though you do say j and high are int variables; no comment about low). Indeed, almost independent of the declarations, it seems unlikely that the quoted for loop is the source of the trouble. It may be code close to that, but it probably is not that line.
There is a chance that you have a buffer over-writing problem somewhere and are modifying a pointer by accident, and that modified pointer generates the error. But, since that line does not seem to include any pointers, it is unlikely to be that line that is actually triggering the trouble.