We migrated several development databases to mysql8 a couple months ago and it was onlyin running large imports/sql scripts that we realized how slow our inserts are.
As an example I created a table as follows and inserted 1000 records,
CREATE TABLE IF NOT EXISTS `ciiu_test2` (
`id` int(11) DEFAULT NULL,
`codbut` varchar(11) DEFAULT NULL,
`ciiu` varchar(8) DEFAULT NULL,
`dt` float(14,2) DEFAULT NULL
)
delete from ciiu_test2;
select UNIX_TIMESTAMP(CURTIME(3)) into #ctime;
INSERT INTO `ciiu_test2` (`id`, `codbut`, `ciiu`,dt) VALUES (1, '1', '1',UNIX_TIMESTAMP(CURTIME(3))-#ctime);
INSERT INTO `ciiu_test2` (`id`, `codbut`, `ciiu`,dt) VALUES (2, '2', '2',UNIX_TIMESTAMP(CURTIME(3))-#ctime);
etc...
With autocommit on, it took 45 seconds to insert 1000 records, with autocommit off, putting everything under one transaction, it took around .6 seconds
Now, I have read about logging and I've turned off logging that might cause a problem. WE don't have replication running. In fact, I have a mysql8 running off docker with the same configuration and no issue.
Here is the configuration for the mysql8 server and the one for docker
Variable_name
167value
51_8value
auto_generate_certs
ON
ON
auto_increment_increment
1
1
auto_increment_offset
1
1
autocommit
ON
ON
automatic_sp_privileges
ON
ON
avoid_temporal_upgrade
OFF
OFF
back_log
151
151
basedir
/usr/
/usr/
big_tables
OFF
OFF
bind_address
*
*
binlog_cache_size
32768
32768
binlog_checksum
CRC32
CRC32
binlog_direct_non_transactional_updates
OFF
OFF
binlog_error_action
ABORT_SERVER
ABORT_SERVER
binlog_format
ROW
ROW
binlog_group_commit_sync_delay
0
0
binlog_group_commit_sync_no_delay_count
0
0
binlog_gtid_simple_recovery
ON
ON
binlog_max_flush_queue_time
0
0
binlog_order_commits
ON
ON
binlog_row_image
FULL
FULL
binlog_rows_query_log_events
OFF
OFF
binlog_stmt_cache_size
32768
32768
binlog_transaction_dependency_history_size
25000
25000
binlog_transaction_dependency_tracking
COMMIT_ORDER
COMMIT_ORDER
block_encryption_mode
aes-128-ecb
aes-128-ecb
bulk_insert_buffer_size
8388608
8388608
character_sets_dir
/usr/share/mysql/charsets/
/usr/share/mysql-8.0/charsets/
check_proxy_users
OFF
OFF
concurrent_insert
AUTO
AUTO
connect_timeout
10
10
core_file
OFF
OFF
datadir
/var/lib/mysql/
/var/lib/mysql/
date_format
not_defined
not_defined
datetime_format
not_defined
not_defined
default_authentication_plugin
mysql_native_password
caching_sha2_password
default_password_lifetime
0
0
default_storage_engine
InnoDB
InnoDB
default_tmp_storage_engine
InnoDB
InnoDB
default_week_format
0
0
delay_key_write
ON
ON
delayed_insert_limit
100
100
delayed_insert_timeout
300
300
delayed_queue_size
1000
1000
disabled_storage_engines
disconnect_on_expired_password
ON
ON
div_precision_increment
4
4
end_markers_in_json
OFF
OFF
enforce_gtid_consistency
OFF
OFF
eq_range_index_dive_limit
200
200
error_count
0
0
event_scheduler
ON
ON
expire_logs_days
0
0
explicit_defaults_for_timestamp
ON
ON
external_user
flush
OFF
OFF
flush_time
0
0
foreign_key_checks
ON
ON
ft_boolean_syntax
+ -><()~*:""&|
+ -><()~*:""&|
ft_max_word_len
84
84
ft_min_word_len
4
4
ft_query_expansion_limit
20
20
ft_stopword_file
(built-in)
(built-in)
general_log
OFF
OFF
general_log_file
/var/lib/mysql/ns515408.log
/var/lib/mysql/093b8c5ba5be.log
group_concat_max_len
1024
1024
gtid_executed_compression_period
1000
1000
gtid_mode
OFF
OFF
gtid_next
AUTOMATIC
AUTOMATIC
gtid_owned
gtid_purged
have_compress
YES
YES
have_crypt
not_defined
not_defined
have_dynamic_loading
YES
YES
have_geometry
YES
YES
have_openssl
YES
YES
have_profiling
YES
YES
have_query_cache
NO
NO
have_rtree_keys
YES
YES
have_ssl
YES
YES
have_statement_timeout
YES
YES
have_symlink
DISABLED
DISABLED
host_cache_size
279
279
hostname
ns515408.ip-167-114-174.net
093b8c5ba5be
identity
0
0
ignore_builtin_innodb
not_defined
not_defined
ignore_db_dirs
not_defined
not_defined
init_connect
init_file
init_slave
innodb_adaptive_flushing
ON
ON
innodb_adaptive_flushing_lwm
10
10
innodb_adaptive_hash_index
ON
ON
innodb_adaptive_hash_index_parts
8
8
innodb_adaptive_max_sleep_delay
150000
150000
innodb_api_bk_commit_interval
5
5
innodb_api_disable_rowlock
OFF
OFF
innodb_api_enable_binlog
OFF
OFF
innodb_api_enable_mdl
OFF
OFF
innodb_api_trx_level
0
0
innodb_autoextend_increment
64
64
innodb_autoinc_lock_mode
2
2
innodb_buffer_pool_chunk_size
134217728
134217728
innodb_buffer_pool_dump_at_shutdown
ON
ON
innodb_buffer_pool_dump_now
OFF
OFF
innodb_buffer_pool_dump_pct
25
25
innodb_buffer_pool_filename
ib_buffer_pool
ib_buffer_pool
innodb_buffer_pool_instances
1
1
innodb_buffer_pool_load_abort
OFF
OFF
innodb_buffer_pool_load_at_startup
ON
ON
innodb_buffer_pool_load_now
OFF
OFF
innodb_buffer_pool_size
134217728
134217728
innodb_change_buffer_max_size
25
25
innodb_change_buffering
all
all
innodb_checksum_algorithm
crc32
crc32
innodb_checksums
not_defined
not_defined
innodb_cmp_per_index_enabled
OFF
OFF
innodb_commit_concurrency
0
0
innodb_compression_failure_threshold_pct
5
5
innodb_compression_level
6
6
innodb_compression_pad_pct_max
50
50
innodb_concurrency_tickets
5000
5000
innodb_data_file_path
ibdata1:12M:autoextend
ibdata1:12M:autoextend
innodb_data_home_dir
innodb_deadlock_detect
ON
ON
innodb_default_row_format
dynamic
dynamic
innodb_disable_sort_file_cache
OFF
OFF
innodb_doublewrite
ON
ON
innodb_fast_shutdown
1
1
innodb_file_format
not_defined
not_defined
innodb_file_format_check
not_defined
not_defined
innodb_file_format_max
not_defined
not_defined
innodb_file_per_table
ON
ON
innodb_fill_factor
100
100
innodb_flush_log_at_timeout
1
1
innodb_flush_log_at_trx_commit
1
1
innodb_flush_method
fsync
fsync
innodb_flush_neighbors
0
0
innodb_flush_sync
ON
ON
innodb_flushing_avg_loops
30
30
innodb_force_load_corrupted
OFF
OFF
innodb_force_recovery
0
0
innodb_ft_aux_table
innodb_ft_cache_size
8000000
8000000
innodb_ft_enable_diag_print
OFF
OFF
innodb_ft_enable_stopword
ON
ON
innodb_ft_max_token_size
84
84
innodb_ft_min_token_size
3
3
innodb_ft_num_word_optimize
2000
2000
innodb_ft_result_cache_limit
2000000000
2000000000
innodb_ft_server_stopword_table
innodb_ft_sort_pll_degree
2
2
innodb_ft_total_cache_size
640000000
640000000
innodb_ft_user_stopword_table
innodb_io_capacity
200
200
innodb_io_capacity_max
2000
2000
innodb_large_prefix
not_defined
not_defined
innodb_lock_wait_timeout
50
50
innodb_locks_unsafe_for_binlog
not_defined
not_defined
innodb_log_buffer_size
16777216
16777216
innodb_log_checksums
ON
ON
innodb_log_compressed_pages
ON
ON
innodb_log_file_size
50331648
50331648
innodb_log_files_in_group
2
2
innodb_log_group_home_dir
./
./
innodb_log_write_ahead_size
8192
8192
innodb_lru_scan_depth
1024
1024
innodb_max_dirty_pages_pct
90
90
innodb_max_dirty_pages_pct_lwm
10
10
innodb_max_purge_lag
0
0
innodb_max_purge_lag_delay
0
0
innodb_max_undo_log_size
1073741824
1073741824
innodb_monitor_disable
innodb_monitor_enable
innodb_monitor_reset
innodb_monitor_reset_all
innodb_numa_interleave
OFF
OFF
innodb_old_blocks_pct
37
37
innodb_old_blocks_time
1000
1000
innodb_online_alter_log_max_size
134217728
134217728
innodb_open_files
4000
4000
innodb_optimize_fulltext_only
OFF
OFF
innodb_page_cleaners
1
1
innodb_page_size
16384
16384
innodb_print_all_deadlocks
OFF
OFF
innodb_purge_batch_size
300
300
innodb_purge_rseg_truncate_frequency
128
128
innodb_purge_threads
4
4
innodb_random_read_ahead
OFF
OFF
innodb_read_ahead_threshold
56
56
innodb_read_io_threads
4
4
innodb_read_only
OFF
OFF
innodb_replication_delay
0
0
innodb_rollback_on_timeout
OFF
OFF
innodb_rollback_segments
128
128
innodb_sort_buffer_size
1048576
1048576
innodb_spin_wait_delay
6
6
innodb_stats_auto_recalc
ON
ON
innodb_stats_include_delete_marked
OFF
OFF
innodb_stats_method
nulls_equal
nulls_equal
innodb_stats_on_metadata
OFF
OFF
innodb_stats_persistent
ON
ON
innodb_stats_persistent_sample_pages
20
20
innodb_stats_sample_pages
not_defined
not_defined
innodb_stats_transient_sample_pages
8
8
innodb_status_output
OFF
OFF
innodb_status_output_locks
OFF
OFF
innodb_strict_mode
ON
ON
innodb_support_xa
not_defined
not_defined
innodb_sync_array_size
1
1
innodb_sync_spin_loops
30
30
innodb_table_locks
ON
ON
innodb_temp_data_file_path
ibtmp1:12M:autoextend
ibtmp1:12M:autoextend
innodb_thread_concurrency
0
0
innodb_thread_sleep_delay
10000
10000
innodb_tmpdir
innodb_undo_directory
./
./
innodb_undo_log_truncate
ON
ON
innodb_undo_logs
not_defined
not_defined
innodb_undo_tablespaces
2
2
innodb_use_native_aio
ON
ON
innodb_version
8.0.21
8.0.22
innodb_write_io_threads
4
4
insert_id
0
0
interactive_timeout
28800
28800
internal_tmp_disk_storage_engine
not_defined
not_defined
join_buffer_size
262144
262144
keep_files_on_create
OFF
OFF
key_buffer_size
8388608
8388608
key_cache_age_threshold
300
300
key_cache_block_size
1024
1024
key_cache_division_limit
100
100
keyring_operations
ON
ON
large_files_support
ON
ON
large_page_size
0
0
large_pages
OFF
OFF
last_insert_id
0
0
lc_messages
en_US
en_US
lc_messages_dir
/usr/share/mysql/
/usr/share/mysql-8.0/
lc_time_names
en_US
en_US
license
GPL
GPL
local_infile
OFF
OFF
lock_wait_timeout
31536000
31536000
locked_in_memory
OFF
OFF
log_bin
ON
ON
log_bin_basename
/var/lib/mysql/binlog
/var/lib/mysql/binlog
log_bin_index
/var/lib/mysql/binlog.index
/var/lib/mysql/binlog.index
log_bin_trust_function_creators
OFF
OFF
log_bin_use_v1_row_events
OFF
OFF
log_builtin_as_identified_by_password
not_defined
not_defined
log_error
/var/log/mysql/mysqld.log
stderr
log_error_verbosity
2
2
log_output
FILE
FILE
log_queries_not_using_indexes
OFF
OFF
log_slave_updates
ON
ON
log_slow_admin_statements
OFF
OFF
log_slow_slave_statements
OFF
OFF
log_statements_unsafe_for_binlog
ON
ON
log_syslog
not_defined
not_defined
log_syslog_facility
not_defined
not_defined
log_syslog_include_pid
not_defined
not_defined
log_syslog_tag
not_defined
not_defined
log_throttle_queries_not_using_indexes
0
0
log_timestamps
UTC
UTC
log_warnings
not_defined
not_defined
long_query_time
10
10
max_allowed_packet
67108864
67108864
max_binlog_cache_size
1.84467E+19
1.84467E+19
max_binlog_size
1073741824
1073741824
max_binlog_stmt_cache_size
1.84467E+19
1.84467E+19
max_connect_errors
100
100
max_connections
151
151
max_delayed_threads
20
20
max_digest_length
1024
1024
max_error_count
1024
1024
max_execution_time
0
0
max_heap_table_size
16777216
16777216
max_insert_delayed_threads
20
20
max_join_size
1.84467E+19
1.84467E+19
max_length_for_sort_data
4096
4096
max_points_in_geometry
65536
65536
max_prepared_stmt_count
16382
16382
max_relay_log_size
0
0
max_seeks_for_key
1.84467E+19
1.84467E+19
max_sort_length
1024
1024
max_sp_recursion_depth
0
0
max_tmp_tables
not_defined
not_defined
max_user_connections
0
0
max_write_lock_count
1.84467E+19
1.84467E+19
metadata_locks_cache_size
not_defined
not_defined
metadata_locks_hash_instances
not_defined
not_defined
min_examined_row_limit
0
0
multi_range_count
not_defined
not_defined
myisam_data_pointer_size
6
6
myisam_max_sort_file_size
9.22337E+18
9.22337E+18
myisam_mmap_size
1.84467E+19
1.84467E+19
myisam_recover_options
OFF
OFF
myisam_repair_threads
1
1
myisam_sort_buffer_size
8388608
8388608
myisam_stats_method
nulls_unequal
nulls_unequal
myisam_use_mmap
OFF
OFF
mysql_native_password_proxy_users
OFF
OFF
net_buffer_length
16384
16384
net_read_timeout
30
30
net_retry_count
10
10
net_write_timeout
60
60
new
OFF
OFF
open_files_limit
10000
1048576
optimizer_prune_level
1
1
optimizer_search_depth
62
62
optimizer_switch
index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on,index_condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_loop=on,batched_key_access=off,materialization=on,semijoin=on,loosescan=on,firstmatch=on,duplicateweedout=on,subquery_materialization_cost_based=on,use_index_extensions=on,condition_fanout_filter=on,derived_merge=on,use_invisible_indexes=off,skip_scan=on,hash_join=on,subquery_to_derived=off,prefer_ordering_index=on
index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on,index_condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_loop=on,batched_key_access=off,materialization=on,semijoin=on,loosescan=on,firstmatch=on,duplicateweedout=on,subquery_materialization_cost_based=on,use_index_extensions=on,condition_fanout_filter=on,derived_merge=on,use_invisible_indexes=off,skip_scan=on,hash_join=on,subquery_to_derived=off,prefer_ordering_index=on,hypergraph_optimizer=off,derived_condition_pushdown=on
optimizer_trace
enabled=off,one_line=off
enabled=off,one_line=off
optimizer_trace_features
greedy_search=on,range_optimizer=on,dynamic_range=on,repeated_subselect=on
greedy_search=on,range_optimizer=on,dynamic_range=on,repeated_subselect=on
optimizer_trace_limit
1
1
optimizer_trace_max_mem_size
1048576
1048576
optimizer_trace_offset
-1
-1
parser_max_mem_size
1.84467E+19
1.84467E+19
performance_schema
ON
ON
performance_schema_accounts_size
-1
-1
performance_schema_digests_size
10000
10000
performance_schema_events_stages_history_long_size
10000
10000
performance_schema_events_stages_history_size
10
10
performance_schema_events_statements_history_long_size
10000
10000
performance_schema_events_statements_history_size
10
10
performance_schema_events_transactions_history_long_size
10000
10000
performance_schema_events_transactions_history_size
10
10
performance_schema_events_waits_history_long_size
10000
10000
performance_schema_events_waits_history_size
10
10
performance_schema_hosts_size
-1
-1
performance_schema_max_cond_classes
100
100
performance_schema_max_cond_instances
-1
-1
performance_schema_max_digest_length
1024
1024
performance_schema_max_file_classes
80
80
performance_schema_max_file_handles
32768
32768
performance_schema_max_file_instances
-1
-1
performance_schema_max_index_stat
-1
-1
performance_schema_max_memory_classes
450
450
performance_schema_max_metadata_locks
-1
-1
performance_schema_max_mutex_classes
300
300
performance_schema_max_mutex_instances
-1
-1
performance_schema_max_prepared_statements_instances
-1
-1
performance_schema_max_program_instances
-1
-1
performance_schema_max_rwlock_classes
60
60
performance_schema_max_rwlock_instances
-1
-1
performance_schema_max_socket_classes
10
10
performance_schema_max_socket_instances
-1
-1
performance_schema_max_sql_text_length
1024
1024
performance_schema_max_stage_classes
175
175
performance_schema_max_statement_classes
218
218
performance_schema_max_statement_stack
10
10
performance_schema_max_table_handles
-1
-1
performance_schema_max_table_instances
-1
-1
performance_schema_max_table_lock_stat
-1
-1
performance_schema_max_thread_classes
100
100
performance_schema_max_thread_instances
-1
-1
performance_schema_session_connect_attrs_size
512
512
performance_schema_setup_actors_size
-1
-1
performance_schema_setup_objects_size
-1
-1
performance_schema_users_size
-1
-1
port
3306
3306
preload_buffer_size
32768
32768
profiling
ON
ON
profiling_history_size
15
15
protocol_version
10
10
proxy_user
pseudo_slave_mode
OFF
OFF
pseudo_thread_id
13855
13
query_alloc_block_size
8192
8192
query_cache_limit
not_defined
not_defined
query_cache_min_res_unit
not_defined
not_defined
query_cache_size
not_defined
not_defined
query_cache_type
not_defined
not_defined
query_cache_wlock_invalidate
not_defined
not_defined
query_prealloc_size
8192
8192
rand_seed1
0
0
rand_seed2
0
0
range_alloc_block_size
4096
4096
range_optimizer_max_mem_size
8388608
8388608
rbr_exec_mode
STRICT
STRICT
read_buffer_size
131072
131072
read_only
OFF
OFF
read_rnd_buffer_size
262144
262144
relay_log
ns515408-relay-bin
093b8c5ba5be-relay-bin
relay_log_basename
/var/lib/mysql/ns515408-relay-bin
/var/lib/mysql/093b8c5ba5be-relay-bin
relay_log_index
/var/lib/mysql/ns515408-relay-bin.index
/var/lib/mysql/093b8c5ba5be-relay-bin.index
relay_log_info_file
relay-log.info
relay-log.info
relay_log_info_repository
TABLE
TABLE
relay_log_purge
ON
ON
relay_log_recovery
OFF
OFF
relay_log_space_limit
0
0
replication_optimize_for_static_plugin_config
not_defined
not_defined
replication_sender_observe_commit_only
not_defined
not_defined
rpl_stop_slave_timeout
31536000
31536000
secure_auth
not_defined
not_defined
secure_file_priv
/var/lib/mysql-files/
NULL
server_id
1
1
server_id_bits
32
32
server_uuid
96c5e0e3-73bc-11eb-873b-a4bf011f053a
c16243c8-1f86-11eb-ab83-0242ac110002
session_track_gtids
OFF
OFF
session_track_schema
ON
ON
session_track_state_change
OFF
OFF
session_track_system_variables
time_zone,autocommit,character_set_client,character_set_results,character_set_connection
time_zone,autocommit,character_set_client,character_set_results,character_set_connection
session_track_transaction_info
OFF
OFF
sha256_password_auto_generate_rsa_keys
ON
ON
sha256_password_private_key_path
private_key.pem
private_key.pem
sha256_password_proxy_users
OFF
OFF
sha256_password_public_key_path
public_key.pem
public_key.pem
show_compatibility_56
not_defined
not_defined
show_create_table_verbosity
OFF
OFF
show_old_temporals
OFF
OFF
skip_external_locking
ON
ON
skip_name_resolve
OFF
ON
skip_networking
OFF
OFF
skip_show_database
OFF
OFF
slave_allow_batching
OFF
OFF
slave_checkpoint_group
512
512
slave_checkpoint_period
300
300
slave_compressed_protocol
OFF
OFF
slave_exec_mode
STRICT
STRICT
slave_load_tmpdir
/var/tmp
/tmp
slave_max_allowed_packet
1073741824
1073741824
slave_net_timeout
60
60
slave_parallel_type
DATABASE
DATABASE
slave_parallel_workers
0
0
slave_pending_jobs_size_max
134217728
134217728
slave_preserve_commit_order
OFF
OFF
slave_rows_search_algorithms
INDEX_SCAN,HASH_SCAN
INDEX_SCAN,HASH_SCAN
slave_skip_errors
OFF
OFF
slave_sql_verify_checksum
ON
ON
slave_transaction_retries
10
10
slave_type_conversions
slow_launch_time
2
2
slow_query_log
OFF
OFF
slow_query_log_file
/var/lib/mysql/ns515408-slow.log
/var/lib/mysql/093b8c5ba5be-slow.log
socket
/var/lib/mysql/mysql.sock
/var/run/mysqld/mysqld.sock
sort_buffer_size
262144
262144
sql_auto_is_null
OFF
OFF
sql_big_selects
ON
ON
sql_buffer_result
OFF
OFF
sql_log_bin
OFF
OFF
sql_log_off
OFF
OFF
sql_mode
ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
sql_notes
ON
ON
sql_quote_show_create
ON
ON
sql_safe_updates
OFF
OFF
sql_select_limit
1.84467E+19
1.84467E+19
sql_slave_skip_counter
0
0
stored_program_cache
256
256
super_read_only
OFF
OFF
sync_binlog
1
1
sync_frm
not_defined
not_defined
sync_master_info
10000
10000
sync_relay_log
10000
10000
sync_relay_log_info
10000
10000
system_time_zone
UTC
UTC
table_definition_cache
2000
2000
table_open_cache
4000
4000
table_open_cache_instances
16
16
thread_cache_size
9
9
thread_handling
one-thread-per-connection
one-thread-per-connection
thread_stack
286720
286720
time_format
not_defined
not_defined
time_zone
SYSTEM
SYSTEM
timestamp
1620141370
1620235604
tls_version
TLSv1,TLSv1.1,TLSv1.2,TLSv1.3
TLSv1,TLSv1.1,TLSv1.2,TLSv1.3
tmp_table_size
16777216
16777216
tmpdir
/var/tmp
/tmp
transaction_alloc_block_size
8192
8192
transaction_allow_batching
OFF
OFF
transaction_isolation
REPEATABLE-READ
REPEATABLE-READ
transaction_prealloc_size
4096
4096
transaction_read_only
OFF
OFF
transaction_write_set_extraction
XXHASH64
XXHASH64
tx_isolation
not_defined
not_defined
tx_read_only
not_defined
not_defined
unique_checks
ON
ON
updatable_views_with_limit
YES
YES
version
8.0.21
8.0.22
version_comment
Source distribution
MySQL Community Server - GPL
version_compile_machine
x86_64
x86_64
version_compile_os
Linux
Linux
wait_timeout
28800
28800
warning_count
0
0
activate_all_roles_on_login
OFF
OFF
admin_address
binlog_encryption
OFF
OFF
binlog_expire_logs_seconds
2592000
2592000
binlog_rotate_encryption_master_key_at_startup
OFF
OFF
binlog_row_event_max_size
8192
8192
binlog_row_metadata
MINIMAL
MINIMAL
binlog_row_value_options
binlog_transaction_compression
OFF
OFF
binlog_transaction_compression_level_zstd
3
3
caching_sha2_password_auto_generate_rsa_keys
ON
ON
caching_sha2_password_private_key_path
private_key.pem
private_key.pem
caching_sha2_password_public_key_path
public_key.pem
public_key.pem
create_admin_listener_thread
OFF
OFF
cte_max_recursion_depth
1000
1000
default_collation_for_utf8mb4
utf8mb4_0900_ai_ci
utf8mb4_0900_ai_ci
default_table_encryption
OFF
OFF
generated_random_password_length
20
20
group_replication_consistency
EVENTUAL
EVENTUAL
gtid_executed
histogram_generation_max_mem_size
20000000
20000000
immediate_server_version
999999
999999
information_schema_stats_expiry
86400
86400
innodb_buffer_pool_in_core_file
ON
ON
innodb_dedicated_server
OFF
OFF
innodb_directories
innodb_doublewrite_batch_size
0
0
innodb_doublewrite_dir
innodb_doublewrite_files
2
2
innodb_doublewrite_pages
4
4
innodb_fsync_threshold
0
0
innodb_idle_flush_pct
100
100
innodb_log_spin_cpu_abs_lwm
80
80
innodb_log_spin_cpu_pct_hwm
50
50
innodb_log_wait_for_flush_spin_hwm
400
400
innodb_parallel_read_threads
4
4
innodb_print_ddl_logs
OFF
OFF
innodb_redo_log_archive_dirs
innodb_redo_log_encrypt
OFF
OFF
innodb_spin_wait_pause_multiplier
50
50
innodb_temp_tablespaces_dir
./#innodb_temp/
./#innodb_temp/
innodb_undo_log_encrypt
OFF
OFF
innodb_validate_tablespace_paths
ON
ON
internal_tmp_mem_storage_engine
TempTable
TempTable
log_error_services
log_filter_internal; log_sink_internal
log_filter_internal; log_sink_internal
log_error_suppression_list
log_raw
OFF
OFF
log_slow_extra
OFF
OFF
mandatory_roles
mysqlx_bind_address
*
*
mysqlx_compression_algorithms
DEFLATE_STREAM,LZ4_MESSAGE,ZSTD_STREAM
DEFLATE_STREAM,LZ4_MESSAGE,ZSTD_STREAM
mysqlx_connect_timeout
30
30
mysqlx_deflate_default_compression_level
3
3
mysqlx_deflate_max_client_compression_level
5
5
mysqlx_document_id_unique_prefix
0
0
mysqlx_enable_hello_notice
ON
ON
mysqlx_idle_worker_thread_timeout
60
60
mysqlx_interactive_timeout
28800
28800
mysqlx_lz4_default_compression_level
2
2
mysqlx_lz4_max_client_compression_level
8
8
mysqlx_max_allowed_packet
67108864
67108864
mysqlx_max_connections
100
100
mysqlx_min_worker_threads
2
2
mysqlx_port
33060
33060
mysqlx_port_open_timeout
0
0
mysqlx_read_timeout
30
30
mysqlx_socket
/var/lib/mysql/mysqlx.sock
/var/run/mysqld/mysqlx.sock
mysqlx_ssl_ca
mysqlx_ssl_capath
mysqlx_ssl_cert
mysqlx_ssl_cipher
mysqlx_ssl_crl
mysqlx_ssl_crlpath
mysqlx_ssl_key
mysqlx_wait_timeout
28800
28800
mysqlx_write_timeout
60
60
mysqlx_zstd_default_compression_level
3
3
mysqlx_zstd_max_client_compression_level
11
11
original_commit_timestamp
3.60288E+16
3.60288E+16
original_server_version
999999
999999
partial_revokes
OFF
OFF
password_history
0
0
password_require_current
OFF
OFF
password_reuse_interval
0
0
performance_schema_error_size
4772
4808
performance_schema_max_digest_sample_age
60
60
persist_only_admin_x509_subject
persisted_globals_load
ON
ON
print_identified_with_as_hex
OFF
OFF
protocol_compression_algorithms
zlib,zstd,uncompressed
zlib,zstd,uncompressed
regexp_stack_limit
8000000
8000000
regexp_time_limit
32
32
require_row_format
OFF
OFF
resultset_metadata
FULL
FULL
rpl_read_size
8192
8192
schema_definition_cache
256
256
secondary_engine_cost_threshold
100000
100000
show_create_table_skip_secondary_engine
OFF
OFF
sql_require_primary_key
OFF
OFF
ssl_fips_mode
OFF
OFF
stored_program_definition_cache
256
256
table_encryption_privilege_check
OFF
OFF
tablespace_definition_cache
256
256
temptable_max_ram
1073741824
1073741824
temptable_use_mmap
ON
ON
tls_ciphersuites
use_secondary_engine
ON
ON
version_compile_zlib
1.2.11
1.2.11
windowing_use_high_precision
ON
ON
innodb_extend_and_initialize
only_in_51_8
ON
innodb_log_writer_threads
only_in_51_8
ON
performance_schema_show_processlist
only_in_51_8
OFF
select_into_buffer_size
only_in_51_8
131072
select_into_disk_sync
only_in_51_8
OFF
select_into_disk_sync_delay
only_in_51_8
0
We have InnoDB on dedicated server and from time to time a simple join query (normally very fast) causes an infinite wait and dies after 30 seconds.
The query looks like "SELECT * FROM orders LEFT JOIN orders_products USING(code)"
This is EXPLAIN:
I am convinced InnoDB is f this up and I don't know why, but it's when more people access the script at the same time.
I am NOT doing any updates or inserts in that script, so it makes no sense.
It could be that somewhere is a script that does a concurrent write and maybe then this main script fails? I have no clue. I never had issue on MyIsam and I don't use transactions.
Can we change settings somehow to fix this?
We have a big server, but neither the admin or I know how to set this properly.
Suggestions are welcome.
This is current settings:
Variable_name
Value
innodb_adaptive_flushing
ON
innodb_adaptive_flushing_lwm
10.000000
innodb_adaptive_hash_index
ON
innodb_adaptive_hash_index_partitions
1
innodb_adaptive_max_sleep_delay
150000
innodb_additional_mem_pool_size
8388608
innodb_api_bk_commit_interval
5
innodb_api_disable_rowlock
OFF
innodb_api_enable_binlog
OFF
innodb_api_enable_mdl
OFF
innodb_api_trx_level
0
innodb_autoextend_increment
64
innodb_autoinc_lock_mode
1
innodb_background_scrub_data_check_interval
3600
innodb_background_scrub_data_compressed
OFF
innodb_background_scrub_data_interval
604800
innodb_background_scrub_data_uncompressed
OFF
innodb_buf_dump_status_frequency
0
innodb_buffer_pool_dump_at_shutdown
OFF
innodb_buffer_pool_dump_now
OFF
innodb_buffer_pool_dump_pct
100
innodb_buffer_pool_filename
ib_buffer_pool
innodb_buffer_pool_instances
2
innodb_buffer_pool_load_abort
OFF
innodb_buffer_pool_load_at_startup
OFF
innodb_buffer_pool_load_now
OFF
innodb_buffer_pool_populate
OFF
innodb_buffer_pool_size
2147483648
innodb_change_buffer_max_size
25
innodb_change_buffering
all
innodb_checksum_algorithm
CRC32
innodb_checksums
ON
innodb_cleaner_lsn_age_factor
HIGH_CHECKPOINT
innodb_cmp_per_index_enabled
OFF
innodb_commit_concurrency
0
innodb_compression_algorithm
zlib
innodb_compression_failure_threshold_pct
5
innodb_compression_level
6
innodb_compression_pad_pct_max
50
innodb_concurrency_tickets
5000
innodb_corrupt_table_action
assert
innodb_data_file_path
ibdata1:12M:autoextend
innodb_data_home_dir
innodb_default_encryption_key_id
1
innodb_default_row_format
compact
innodb_defragment
OFF
innodb_defragment_fill_factor
0.900000
innodb_defragment_fill_factor_n_recs
20
innodb_defragment_frequency
40
innodb_defragment_n_pages
7
innodb_defragment_stats_accuracy
0
innodb_disable_sort_file_cache
OFF
innodb_disallow_writes
OFF
innodb_doublewrite
ON
innodb_empty_free_list_algorithm
BACKOFF
innodb_encrypt_log
OFF
innodb_encrypt_tables
OFF
innodb_encryption_rotate_key_age
1
innodb_encryption_rotation_iops
100
innodb_encryption_threads
0
innodb_fake_changes
OFF
innodb_fast_shutdown
1
innodb_fatal_semaphore_wait_threshold
600
innodb_file_format
Barracuda
innodb_file_format_check
ON
innodb_file_format_max
Antelope
innodb_file_per_table
ON
innodb_flush_log_at_timeout
1
innodb_flush_log_at_trx_commit
2
innodb_flush_method
O_DIRECT
innodb_flush_neighbors
1
innodb_flushing_avg_loops
30
innodb_force_load_corrupted
OFF
innodb_force_primary_key
OFF
innodb_force_recovery
0
innodb_foreground_preflush
EXPONENTIAL_BACKOFF
innodb_ft_aux_table
innodb_ft_cache_size
8000000
innodb_ft_enable_diag_print
OFF
innodb_ft_enable_stopword
ON
innodb_ft_max_token_size
84
innodb_ft_min_token_size
3
innodb_ft_num_word_optimize
2000
innodb_ft_result_cache_limit
2000000000
innodb_ft_server_stopword_table
innodb_ft_sort_pll_degree
2
innodb_ft_total_cache_size
640000000
innodb_ft_user_stopword_table
innodb_idle_flush_pct
100
innodb_immediate_scrub_data_uncompressed
OFF
innodb_instrument_semaphores
OFF
innodb_io_capacity
200
innodb_io_capacity_max
2000
innodb_kill_idle_transaction
0
innodb_large_prefix
ON
innodb_lock_schedule_algorithm
fcfs
innodb_lock_wait_timeout
50
innodb_locking_fake_changes
ON
innodb_locks_unsafe_for_binlog
OFF
innodb_log_arch_dir
./
Variable_name
Value
innodb_log_arch_expire_sec
0
innodb_log_archive
OFF
innodb_log_block_size
512
innodb_log_buffer_size
16777216
innodb_log_checksum_algorithm
INNODB
innodb_log_compressed_pages
ON
innodb_log_file_size
268435456
innodb_log_files_in_group
2
innodb_log_group_home_dir
./
innodb_lru_scan_depth
1024
innodb_max_bitmap_file_size
104857600
innodb_max_changed_pages
1000000
innodb_max_dirty_pages_pct
75.000000
innodb_max_dirty_pages_pct_lwm
0.001000
innodb_max_purge_lag
0
innodb_max_purge_lag_delay
0
innodb_mirrored_log_groups
1
innodb_monitor_disable
innodb_monitor_enable
innodb_monitor_reset
innodb_monitor_reset_all
innodb_mtflush_threads
8
innodb_old_blocks_pct
37
innodb_old_blocks_time
1000
innodb_online_alter_log_max_size
134217728
innodb_open_files
5000
innodb_optimize_fulltext_only
OFF
innodb_page_size
16384
innodb_prefix_index_cluster_optimization
OFF
innodb_print_all_deadlocks
OFF
innodb_print_lock_wait_timeout_info
OFF
innodb_purge_batch_size
300
innodb_purge_threads
1
innodb_random_read_ahead
OFF
innodb_read_ahead_threshold
56
innodb_read_io_threads
4
innodb_read_only
OFF
innodb_replication_delay
0
innodb_rollback_on_timeout
OFF
innodb_rollback_segments
128
innodb_sched_priority_cleaner
19
innodb_scrub_log
OFF
innodb_scrub_log_speed
256
innodb_show_locks_held
10
innodb_show_verbose_locks
0
innodb_simulate_comp_failures
0
innodb_sort_buffer_size
1048576
innodb_spin_wait_delay
6
innodb_stats_auto_recalc
ON
innodb_stats_include_delete_marked
OFF
innodb_stats_method
nulls_equal
innodb_stats_modified_counter
0
innodb_stats_on_metadata
OFF
innodb_stats_persistent
ON
innodb_stats_persistent_sample_pages
20
innodb_stats_sample_pages
8
innodb_stats_traditional
ON
innodb_stats_transient_sample_pages
8
innodb_status_output
OFF
innodb_status_output_locks
OFF
innodb_strict_mode
OFF
innodb_support_xa
ON
innodb_sync_array_size
1
innodb_sync_spin_loops
30
innodb_table_locks
ON
innodb_thread_concurrency
0
innodb_thread_sleep_delay
10000
innodb_tmpdir
innodb_track_changed_pages
OFF
innodb_undo_directory
.
innodb_undo_logs
128
innodb_undo_tablespaces
0
innodb_use_atomic_writes
OFF
innodb_use_fallocate
OFF
innodb_use_global_flush_log_at_trx_commit
ON
innodb_use_mtflush
OFF
innodb_use_native_aio
ON
innodb_use_stacktrace
OFF
innodb_use_sys_malloc
ON
innodb_use_trim
OFF
innodb_version
5.6.46-86.2
innodb_write_io_threads
4
So, I just split the query to 2 queries (many rows/conditions) and it works now.
Probably was difficult query (despite the fact it was quick) and when having more hits, it stalled. Idk.
We have a problem with peak loading while wring data to database
This is example of error
SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction
SQL: insert intoresponses(participant_id,survey_id,question_id,question_option_id) values (2204017, 188131, 5, 23)
The table is quite big, 4.5 millions rows, per day at peak we write about 150-200 thosands rows to it.
MySQL version is 5.6.39
Table uses InnoDb engine
MySQL instance configuration at RDS (Amazon service)
Instance class db.t2.xlarge
vCPU 4
RAM 16 GB
Table structure
id int(10) unsigned Auto Increment
participant_id int(10) unsigned
survey_id int(10) unsigned
question_id int(10) unsigned
question_option_id int(10) unsigned NULL
response_answer text NULL
created_at timestamp NULL
updated_at timestamp NULL
Indexes
PRIMARY id
INDEX participant_id
INDEX survey_id
INDEX question_id
INDEX question_option_id
Foreign keys
participant_id -> participants(id)
question_id -> questions(id)
question_option_id -> question_options(id)
survey_id -> surveys(id)
InnoDB Settings
innodb_adaptive_flushing ON
innodb_adaptive_flushing_lwm 10
innodb_adaptive_hash_index ON
innodb_adaptive_max_sleep_delay 150000
innodb_additional_mem_pool_size 8388608
innodb_api_bk_commit_interval 5
innodb_api_disable_rowlock OFF
innodb_api_enable_binlog OFF
innodb_api_enable_mdl OFF
innodb_api_trx_level 0
innodb_autoextend_increment 64
innodb_autoinc_lock_mode 1
innodb_buffer_pool_dump_at_shutdown OFF
innodb_buffer_pool_dump_now OFF
innodb_buffer_pool_filename ib_buffer_pool
innodb_buffer_pool_instances 8
innodb_buffer_pool_load_abort OFF
innodb_buffer_pool_load_at_startup OFF
innodb_buffer_pool_load_now OFF
innodb_buffer_pool_size 12465471488
innodb_change_buffer_max_size 25
innodb_change_buffering all
innodb_checksum_algorithm innodb
innodb_checksums ON
innodb_cmp_per_index_enabled OFF
innodb_commit_concurrency 0
innodb_compression_failure_threshold_pct 5
innodb_compression_level 6
innodb_compression_pad_pct_max 50
innodb_concurrency_tickets 5000
innodb_data_file_path ibdata1:12M:autoextend
innodb_data_home_dir /rdsdbdata/db/innodb
innodb_disable_sort_file_cache OFF
innodb_doublewrite ON
innodb_fast_shutdown 1
innodb_file_format Antelope
innodb_file_format_check ON
innodb_file_format_max Antelope
innodb_file_per_table ON
innodb_flush_log_at_timeout 1
innodb_flush_log_at_trx_commit 1
innodb_flush_method O_DIRECT
innodb_flush_neighbors 1
innodb_flushing_avg_loops 30
innodb_force_load_corrupted OFF
innodb_force_recovery 0
innodb_ft_aux_table
innodb_ft_cache_size 8000000
innodb_ft_enable_diag_print OFF
innodb_ft_enable_stopword ON
innodb_ft_max_token_size 84
innodb_ft_min_token_size 3
innodb_ft_num_word_optimize 2000
innodb_ft_result_cache_limit 2000000000
innodb_ft_server_stopword_table
innodb_ft_sort_pll_degree 2
innodb_ft_total_cache_size 640000000
innodb_ft_user_stopword_table
innodb_io_capacity 200
innodb_io_capacity_max 2000
innodb_large_prefix OFF
innodb_lock_wait_timeout 50
innodb_locks_unsafe_for_binlog OFF
innodb_log_buffer_size 8388608
innodb_log_compressed_pages ON
innodb_log_file_size 134217728
innodb_log_files_in_group 2
innodb_log_group_home_dir /rdsdbdata/log/innodb
innodb_lru_scan_depth 1024
innodb_max_dirty_pages_pct 75
innodb_max_dirty_pages_pct_lwm 0
innodb_max_purge_lag 0
innodb_max_purge_lag_delay 0
innodb_mirrored_log_groups 1
innodb_monitor_disable
innodb_monitor_enable
innodb_monitor_reset
innodb_monitor_reset_all
innodb_numa_interleave OFF
innodb_old_blocks_pct 37
innodb_old_blocks_time 1000
innodb_online_alter_log_max_size 134217728
innodb_open_files 2000
innodb_optimize_fulltext_only OFF
innodb_page_size 16384
innodb_print_all_deadlocks OFF
innodb_purge_batch_size 300
innodb_purge_threads 1
innodb_random_read_ahead OFF
innodb_read_ahead_threshold 56
innodb_read_io_threads 4
innodb_read_only OFF
innodb_replication_delay 0
innodb_rollback_on_timeout OFF
innodb_rollback_segments 128
innodb_sort_buffer_size 1048576
innodb_spin_wait_delay 6
innodb_stats_auto_recalc ON
innodb_stats_include_delete_marked OFF
innodb_stats_method nulls_equal
innodb_stats_on_metadata OFF
innodb_stats_persistent ON
innodb_stats_persistent_sample_pages 20
innodb_stats_sample_pages 8
innodb_stats_transient_sample_pages 8
innodb_status_output OFF
innodb_status_output_locks OFF
innodb_strict_mode OFF
innodb_support_xa ON
innodb_sync_array_size 1
innodb_sync_spin_loops 30
innodb_table_locks ON
innodb_thread_concurrency 0
innodb_thread_sleep_delay 10000
innodb_tmpdir
innodb_undo_directory .
innodb_undo_logs 128
innodb_undo_tablespaces 0
innodb_use_native_aio ON
innodb_use_sys_malloc ON
innodb_version 5.6.39
innodb_write_io_threads 4
I think increasing target & max IOPS above the defaults to avoid excessive SSD wear also disable proactively flushing pages could help
innodb_adaptive_flushing_lwm = 0
Innodb_flush_neighbors = off
innodb_io_capacity=1000
innodb_io_capacity_max=2500
Reference:
https://www.percona.com/blog/2019/12/18/give-love-to-your-ssds-reduce-innodb_io_capacity_max/
I am currently assisting with integration testing for migrating an application that currently runs against a mysql 5.5 DB to mysql 5.7. We have run into issues related to functionality that was optimised in mysql 5.5 to run about 30% faster by having certain parts be encapsulated in a read only transaction now throwing the below error:
Error Code: 1792 Cannot execute statement in a READ ONLY transaction.
I have scoured the internet and havent found any solutions. I know there is a related limitation added in mysql 5.6 documentation that mentions:
"In read-only mode, it remains possible to change tables created with
the TEMPORARY keyword using DML statements. Changes made with DDL
statements are not permitted, just as with permanent tables."
This presents a major limitation to us as some of these read only transactions call stored procedures that create temporary tables and it is not feasible for us to rewrite the application as this read only optimisation is widespread.
Essentially we are hoping to find a work around or a configuration we can tweak to allow the below example to not give a read only error.
START TRANSACTION READ only;
CREATE TEMPORARY TABLE IF NOT EXISTS table1 (account_id INT(10) UNSIGNED);
ROLLBACK;
Creating the table beforehand is not an option, mysql server session variables below if required. Any help would be much appreciated!
Session Variables:
Variable_name Value
auto_increment_increment 1
auto_increment_offset 1
autocommit ON
automatic_sp_privileges ON
avoid_temporal_upgrade OFF
back_log 80
basedir C:\\Program Files\\MySQL\\MySQL Server 5.7\\
big_tables OFF
bind_address *
binlog_cache_size 32768
binlog_checksum CRC32
binlog_direct_non_transactional_updates OFF
binlog_error_action ABORT_SERVER
binlog_format ROW
binlog_group_commit_sync_delay 0
binlog_group_commit_sync_no_delay_count 0
binlog_gtid_simple_recovery ON
binlog_max_flush_queue_time 0
binlog_order_commits ON
binlog_row_image FULL
binlog_rows_query_log_events OFF
binlog_stmt_cache_size 32768
binlog_transaction_dependency_history_size 25000
binlog_transaction_dependency_tracking COMMIT_ORDER
block_encryption_mode aes-128-ecb
bulk_insert_buffer_size 8388608
character_set_client utf8
character_set_connection utf8
character_set_database utf8
character_set_filesystem binary
character_set_results utf8
character_set_server utf8
character_set_system utf8
character_sets_dir C:\\Program Files\\MySQL\\MySQL Server 5.7\\share\\charsets\\
check_proxy_users OFF
collation_connection utf8_general_ci
collation_database utf8_general_ci
collation_server utf8_general_ci
completion_type NO_CHAIN
concurrent_insert AUTO
connect_timeout 10
core_file OFF
datadir C:\\ProgramData\\MySQL\\MySQL Server 5.7\\Data\\
date_format %Y-%m-%d
datetime_format %Y-%m-%d %H:%i:%s
default_authentication_plugin mysql_native_password
default_password_lifetime 0
default_storage_engine InnoDB
default_tmp_storage_engine InnoDB
default_week_format 0
delay_key_write ON
delayed_insert_limit 100
delayed_insert_timeout 300
delayed_queue_size 1000
disabled_storage_engines
disconnect_on_expired_password ON
div_precision_increment 4
end_markers_in_json OFF
enforce_gtid_consistency OFF
eq_range_index_dive_limit 200
error_count 0
event_scheduler OFF
expire_logs_days 0
explicit_defaults_for_timestamp OFF
external_user
flush OFF
flush_time 0
foreign_key_checks ON
ft_boolean_syntax "+ -><()~*:""""&|"
ft_max_word_len 84
ft_min_word_len 4
ft_query_expansion_limit 20
ft_stopword_file (built-in)
general_log OFF
general_log_file DESKTOP-11VS9FN.log
group_concat_max_len 1024
gtid_executed_compression_period 1000
gtid_mode OFF
gtid_next AUTOMATIC
gtid_owned
gtid_purged
have_compress YES
have_crypt NO
have_dynamic_loading YES
have_geometry YES
have_openssl DISABLED
have_profiling YES
have_query_cache YES
have_rtree_keys YES
have_ssl DISABLED
have_statement_timeout YES
have_symlink YES
host_cache_size 279
hostname BLUEOA-LP005
identity 0
ignore_builtin_innodb OFF
ignore_db_dirs
init_connect
init_file
init_slave
innodb_adaptive_flushing ON
innodb_adaptive_flushing_lwm 10
innodb_adaptive_hash_index ON
innodb_adaptive_hash_index_parts 8
innodb_adaptive_max_sleep_delay 150000
innodb_api_bk_commit_interval 5
innodb_api_disable_rowlock OFF
innodb_api_enable_binlog OFF
innodb_api_enable_mdl OFF
innodb_api_trx_level 0
innodb_autoextend_increment 64
innodb_autoinc_lock_mode 1
innodb_buffer_pool_chunk_size 8388608
innodb_buffer_pool_dump_at_shutdown ON
innodb_buffer_pool_dump_now OFF
innodb_buffer_pool_dump_pct 25
innodb_buffer_pool_filename ib_buffer_pool
innodb_buffer_pool_instances 1
innodb_buffer_pool_load_abort OFF
innodb_buffer_pool_load_at_startup ON
innodb_buffer_pool_load_now OFF
innodb_buffer_pool_size 8388608
innodb_change_buffer_max_size 25
innodb_change_buffering all
innodb_checksum_algorithm crc32
innodb_checksums ON
innodb_cmp_per_index_enabled OFF
innodb_commit_concurrency 0
innodb_compression_failure_threshold_pct 5
innodb_compression_level 6
innodb_compression_pad_pct_max 50
innodb_concurrency_tickets 5000
innodb_data_file_path ibdata1:12M:autoextend
innodb_data_home_dir
innodb_deadlock_detect ON
innodb_default_row_format dynamic
innodb_disable_sort_file_cache OFF
innodb_doublewrite ON
innodb_fast_shutdown 1
innodb_file_format Barracuda
innodb_file_format_check ON
innodb_file_format_max Barracuda
innodb_file_per_table ON
innodb_fill_factor 100
innodb_flush_log_at_timeout 1
innodb_flush_log_at_trx_commit 1
innodb_flush_method
innodb_flush_neighbors 1
innodb_flush_sync ON
innodb_flushing_avg_loops 30
innodb_force_load_corrupted OFF
innodb_force_recovery 0
innodb_ft_aux_table
innodb_ft_cache_size 8000000
innodb_ft_enable_diag_print OFF
innodb_ft_enable_stopword ON
innodb_ft_max_token_size 84
innodb_ft_min_token_size 3
innodb_ft_num_word_optimize 2000
innodb_ft_result_cache_limit 2000000000
innodb_ft_server_stopword_table
innodb_ft_sort_pll_degree 2
innodb_ft_total_cache_size 640000000
innodb_ft_user_stopword_table
innodb_io_capacity 200
innodb_io_capacity_max 2000
innodb_large_prefix ON
innodb_lock_wait_timeout 50
innodb_locks_unsafe_for_binlog OFF
innodb_log_buffer_size 1048576
innodb_log_checksums ON
innodb_log_compressed_pages ON
innodb_log_file_size 50331648
innodb_log_files_in_group 2
innodb_log_group_home_dir .\\
innodb_log_write_ahead_size 8192
innodb_lru_scan_depth 1024
innodb_max_dirty_pages_pct 75
innodb_max_dirty_pages_pct_lwm 0
innodb_max_purge_lag 0
innodb_max_purge_lag_delay 0
innodb_max_undo_log_size 1073741824
innodb_monitor_disable
innodb_monitor_enable
innodb_monitor_reset
innodb_monitor_reset_all
innodb_old_blocks_pct 37
innodb_old_blocks_time 1000
innodb_online_alter_log_max_size 134217728
innodb_open_files 300
innodb_optimize_fulltext_only OFF
innodb_page_cleaners 1
innodb_page_size 16384
innodb_print_all_deadlocks OFF
innodb_purge_batch_size 300
innodb_purge_rseg_truncate_frequency 128
innodb_purge_threads 4
innodb_random_read_ahead OFF
innodb_read_ahead_threshold 56
innodb_read_io_threads 4
innodb_read_only OFF
innodb_replication_delay 0
innodb_rollback_on_timeout OFF
innodb_rollback_segments 128
innodb_sort_buffer_size 1048576
innodb_spin_wait_delay 6
innodb_stats_auto_recalc ON
innodb_stats_include_delete_marked OFF
innodb_stats_method nulls_equal
innodb_stats_on_metadata OFF
innodb_stats_persistent ON
innodb_stats_persistent_sample_pages 20
innodb_stats_sample_pages 8
innodb_stats_transient_sample_pages 8
innodb_status_output OFF
innodb_status_output_locks OFF
innodb_strict_mode OFF
innodb_support_xa ON
innodb_sync_array_size 1
innodb_sync_spin_loops 30
innodb_table_locks ON
innodb_temp_data_file_path ibtmp1:12M:autoextend
innodb_thread_concurrency 9
innodb_thread_sleep_delay 0
innodb_tmpdir
innodb_undo_directory .\\
innodb_undo_log_truncate OFF
innodb_undo_logs 128
innodb_undo_tablespaces 0
innodb_use_native_aio ON
innodb_version 5.7.22
innodb_write_io_threads 4
insert_id 0
interactive_timeout 28800
internal_tmp_disk_storage_engine InnoDB
join_buffer_size 262144
keep_files_on_create OFF
key_buffer_size 8388608
key_cache_age_threshold 300
key_cache_block_size 1024
key_cache_division_limit 100
keyring_operations ON
large_files_support ON
large_page_size 0
large_pages OFF
last_insert_id 0
lc_messages en_US
lc_messages_dir C:\\Program Files\\MySQL\\MySQL Server 5.7\\share\\
lc_time_names en_US
license GPL
local_infile ON
lock_wait_timeout 31536000
log_bin OFF
log_bin_basename
log_bin_index
log_bin_trust_function_creators OFF
log_bin_use_v1_row_events OFF
log_builtin_as_identified_by_password OFF
log_error .\\DESKTOP-11VS9FN.err
log_error_verbosity 3
log_output FILE
log_queries_not_using_indexes OFF
log_slave_updates OFF
log_slow_admin_statements OFF
log_slow_slave_statements OFF
log_statements_unsafe_for_binlog ON
log_syslog ON
log_syslog_tag
log_throttle_queries_not_using_indexes 0
log_timestamps UTC
log_warnings 2
long_query_time 10
low_priority_updates OFF
lower_case_file_system ON
lower_case_table_names 1
master_info_repository FILE
master_verify_checksum OFF
max_allowed_packet 173741056
max_binlog_cache_size 1.84467E+19
max_binlog_size 1073741824
max_binlog_stmt_cache_size 1.84467E+19
max_connect_errors 100
max_connections 151
max_delayed_threads 20
max_digest_length 1024
max_error_count 64
max_execution_time 0
max_heap_table_size 70254592
max_insert_delayed_threads 20
max_join_size 1.84467E+19
max_length_for_sort_data 1024
max_points_in_geometry 65536
max_prepared_stmt_count 16382
max_relay_log_size 0
max_seeks_for_key 4294967295
max_sort_length 1024
max_sp_recursion_depth 0
max_tmp_tables 32
max_user_connections 0
max_write_lock_count 4294967295
metadata_locks_cache_size 1024
metadata_locks_hash_instances 8
min_examined_row_limit 0
multi_range_count 256
myisam_data_pointer_size 6
myisam_max_sort_file_size 1.07374E+11
myisam_mmap_size 1.84467E+19
myisam_recover_options OFF
myisam_repair_threads 1
myisam_sort_buffer_size 8388608
myisam_stats_method nulls_unequal
myisam_use_mmap OFF
mysql_native_password_proxy_users OFF
named_pipe OFF
net_buffer_length 16384
net_read_timeout 30
net_retry_count 10
net_write_timeout 60
new OFF
ngram_token_size 2
offline_mode OFF
old OFF
old_alter_table OFF
old_passwords 0
open_files_limit 6209
optimizer_prune_level 1
optimizer_search_depth 62
optimizer_switch "index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on,index_condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_loop=on,batched_key_access=off,materialization=on,semijoin=on,loosescan=on,firstmatch=on,duplicateweedout=on,subquery_materialization_cost_based=on,use_index_extensions=on,condition_fanout_filter=on,derived_merge=on"
optimizer_trace "enabled=off,one_line=off"
optimizer_trace_features "greedy_search=on,range_optimizer=on,dynamic_range=on,repeated_subselect=on"
optimizer_trace_limit 1
optimizer_trace_max_mem_size 16384
optimizer_trace_offset -1
parser_max_mem_size 1.84467E+19
performance_schema ON
performance_schema_accounts_size -1
performance_schema_digests_size 10000
performance_schema_events_stages_history_long_size 10000
performance_schema_events_stages_history_size 10
performance_schema_events_statements_history_long_size 10000
performance_schema_events_statements_history_size 10
performance_schema_events_transactions_history_long_size 10000
performance_schema_events_transactions_history_size 10
performance_schema_events_waits_history_long_size 10000
performance_schema_events_waits_history_size 10
performance_schema_hosts_size -1
performance_schema_max_cond_classes 80
performance_schema_max_cond_instances -1
performance_schema_max_digest_length 1024
performance_schema_max_file_classes 80
performance_schema_max_file_handles 32768
performance_schema_max_file_instances -1
performance_schema_max_index_stat -1
performance_schema_max_memory_classes 320
performance_schema_max_metadata_locks -1
performance_schema_max_mutex_classes 210
performance_schema_max_mutex_instances -1
performance_schema_max_prepared_statements_instances -1
performance_schema_max_program_instances -1
performance_schema_max_rwlock_classes 40
performance_schema_max_rwlock_instances -1
performance_schema_max_socket_classes 10
performance_schema_max_socket_instances -1
performance_schema_max_sql_text_length 1024
performance_schema_max_stage_classes 150
performance_schema_max_statement_classes 193
performance_schema_max_statement_stack 10
performance_schema_max_table_handles -1
performance_schema_max_table_instances -1
performance_schema_max_table_lock_stat -1
performance_schema_max_thread_classes 50
performance_schema_max_thread_instances -1
performance_schema_session_connect_attrs_size 512
performance_schema_setup_actors_size -1
performance_schema_setup_objects_size -1
performance_schema_users_size -1
pid_file C:\\ProgramData\\MySQL\\MySQL Server 5.7\\Data\\BLUEOA-LP005.pid
plugin_dir C:\\Program Files\\MySQL\\MySQL Server 5.7\\lib\\plugin\\
port 3306
preload_buffer_size 32768
profiling OFF
profiling_history_size 15
protocol_version 10
proxy_user
pseudo_slave_mode OFF
pseudo_thread_id 503
query_alloc_block_size 8192
query_cache_limit 1048576
query_cache_min_res_unit 4096
query_cache_size 1048576
query_cache_type OFF
query_cache_wlock_invalidate OFF
query_prealloc_size 8192
rand_seed1 0
rand_seed2 0
range_alloc_block_size 4096
range_optimizer_max_mem_size 8388608
rbr_exec_mode STRICT
read_buffer_size 8192
read_only OFF
read_rnd_buffer_size 1
relay_log
relay_log_basename C:\\ProgramData\\MySQL\\MySQL Server 5.7\\Data\\BLUEOA-LP005-relay-bin
relay_log_index C:\\ProgramData\\MySQL\\MySQL Server 5.7\\Data\\BLUEOA-LP005-relay-bin.index
relay_log_info_file relay-log.info
relay_log_info_repository FILE
relay_log_purge ON
relay_log_recovery OFF
relay_log_space_limit 0
report_host
report_password
report_port 3306
report_user
require_secure_transport OFF
rpl_stop_slave_timeout 31536000
secure_auth ON
secure_file_priv C:\\ProgramData\\MySQL\\MySQL Server 5.7\\Uploads\\
server_id 1
server_id_bits 32
server_uuid 215d32a3-5340-11e8-93c8-106530971942
session_track_gtids OFF
session_track_schema ON
session_track_state_change OFF
session_track_system_variables "time_zone,autocommit,character_set_client,character_set_results,character_set_connection"
session_track_transaction_info OFF
sha256_password_proxy_users OFF
shared_memory OFF
shared_memory_base_name MYSQL
show_compatibility_56 OFF
show_create_table_verbosity OFF
show_old_temporals OFF
skip_external_locking ON
skip_name_resolve OFF
skip_networking OFF
skip_show_database OFF
slave_allow_batching OFF
slave_checkpoint_group 512
slave_checkpoint_period 300
slave_compressed_protocol OFF
slave_exec_mode IDEMPOTENT
slave_load_tmpdir C:\\WINDOWS\\SERVIC~3\\NETWOR~1\\AppData\\Local\\Temp
slave_max_allowed_packet 1073741824
slave_net_timeout 60
slave_parallel_type DATABASE
slave_parallel_workers 0
slave_pending_jobs_size_max 16777216
slave_preserve_commit_order OFF
slave_rows_search_algorithms "TABLE_SCAN,INDEX_SCAN"
slave_skip_errors OFF
slave_sql_verify_checksum ON
slave_transaction_retries 10
slave_type_conversions
slow_launch_time 2
slow_query_log ON
slow_query_log_file DESKTOP-11VS9FN-slow.log
socket MySQL
sort_buffer_size 262144
sql_auto_is_null OFF
sql_big_selects ON
sql_buffer_result OFF
sql_log_bin ON
sql_log_off OFF
sql_mode
sql_notes ON
sql_quote_show_create ON
sql_safe_updates OFF
sql_select_limit 1.84467E+19
sql_slave_skip_counter 0
sql_warnings OFF
ssl_ca
ssl_capath
ssl_cert
ssl_cipher
ssl_crl
ssl_crlpath
ssl_key
stored_program_cache 256
super_read_only OFF
sync_binlog 1
sync_frm ON
sync_master_info 10000
sync_relay_log 10000
sync_relay_log_info 10000
system_time_zone AUS Eastern Standard Time
table_definition_cache 1400
table_open_cache 2000
table_open_cache_instances 16
thread_cache_size 10
thread_handling one-thread-per-connection
thread_stack 262144
time_format %H:%i:%s
time_zone SYSTEM
timestamp 1527639658
tls_version "TLSv1,TLSv1.1"
tmp_table_size 15728640
tmpdir C:\\WINDOWS\\SERVIC~3\\NETWOR~1\\AppData\\Local\\Temp
transaction_alloc_block_size 8192
transaction_allow_batching OFF
transaction_isolation REPEATABLE-READ
transaction_prealloc_size 4096
transaction_read_only OFF
transaction_write_set_extraction OFF
tx_isolation REPEATABLE-READ
tx_read_only OFF
unique_checks ON
updatable_views_with_limit YES
version 5.7.22-log
version_comment MySQL Community Server (GPL)
version_compile_machine x86_64
version_compile_os Win64
wait_timeout 28800
warning_count 0
*/ EDIT 1 */ We have decided to abandon the read only optimisations that were put in place for MySql 5.5 as we were unable to resolve this roadblock.
I know this is an old question but we have ended up finding a very easy solution to this problem.
The solution was simple as to not execute the stored procedure calls within a transaction. The only extra care we had to make was to ensure none of the affected stored procedures would make data change but apart from that the solutions was as simple as that.
I have a cs-cart installation based in Mysql (MyIsam) with about 3,000 items and a database size of 700MB and index size of 247MB.
In the last month for a reason i don't know the performances falled down and are very inconstant. I optimized the DB as I usually do and tried to change some Mysql variables according with some suggestion I found over the internet.
The dedicated server is quite old
Cpu dual core pentium E2180
6GB RAM
160GB SATA
and this is the screenshot of the top command
auto_increment_increment
1
auto_increment_offset
1
automatic_sp_privileges
ON
back_log
50
basedir
/usr/
bdb_cache_size
8384512
bdb_home
/var/lib/mysql/
bdb_log_buffer_size
262144
bdb_logdir
bdb_max_lock
10000
bdb_shared_data
OFF
bdb_tmpdir
/tmp/
binlog_cache_size
32768
bulk_insert_buffer_size
8388608
character_set_client
utf8
character_set_connection
utf8
character_set_database
latin1
character_set_filesystem
binary
character_set_results
utf8
character_set_server
latin1
character_set_system
utf8
character_sets_dir
/usr/share/mysql/charsets/
collation_connection
utf8_general_ci
collation_database
latin1_swedish_ci
collation_server
latin1_swedish_ci
completion_type
0
concurrent_insert
1
connect_timeout
10
datadir
/var/lib/mysql/
date_format
%Y-%m-%d
datetime_format
%Y-%m-%d %H:%i:%s
default_week_format
0
delay_key_write
ON
delayed_insert_limit
100
delayed_insert_timeout
300
delayed_queue_size
1000
div_precision_increment
4
keep_files_on_create
OFF
engine_condition_pushdown
OFF
expire_logs_days
0
flush
OFF
flush_time
0
ft_boolean_syntax
+ -><()~*:""&|
ft_max_word_len
84
ft_min_word_len
4
ft_query_expansion_limit
20
ft_stopword_file
(built-in)
group_concat_max_len
1024
have_archive
NO
have_bdb
YES
have_blackhole_engine
NO
have_compress
YES
have_community_features
YES
have_profiling
YES
have_crypt
YES
have_csv
NO
have_dynamic_loading
YES
have_example_engine
NO
have_federated_engine
NO
have_geometry
YES
have_innodb
YES
have_isam
NO
have_merge_engine
YES
have_ndbcluster
NO
have_openssl
DISABLED
have_ssl
DISABLED
have_query_cache
YES
have_raid
NO
have_rtree_keys
YES
have_symlink
YES
hostname
ltfs676.aus.us.siteprotect.com
init_connect
init_file
init_slave
innodb_additional_mem_pool_size
1048576
innodb_autoextend_increment
8
innodb_buffer_pool_awe_mem_mb
0
innodb_buffer_pool_size
8388608
innodb_checksums
ON
innodb_commit_concurrency
0
innodb_concurrency_tickets
500
innodb_data_file_path
ibdata1:10M:autoextend
innodb_data_home_dir
innodb_adaptive_hash_index
ON
innodb_doublewrite
ON
innodb_fast_shutdown
1
innodb_file_io_threads
4
innodb_file_per_table
OFF
innodb_flush_log_at_trx_commit
1
innodb_flush_method
innodb_force_recovery
0
innodb_lock_wait_timeout
50
innodb_locks_unsafe_for_binlog
OFF
innodb_log_arch_dir
innodb_log_archive
OFF
innodb_log_buffer_size
1048576
innodb_log_file_size
5242880
innodb_log_files_in_group
2
innodb_log_group_home_dir
./
innodb_max_dirty_pages_pct
90
Variable_name
Value
innodb_max_purge_lag
0
innodb_mirrored_log_groups
1
innodb_open_files
300
innodb_rollback_on_timeout
OFF
innodb_support_xa
ON
innodb_sync_spin_loops
20
innodb_table_locks
ON
innodb_thread_concurrency
8
innodb_thread_sleep_delay
10000
innodb_use_legacy_cardinality_algorithm
ON
interactive_timeout
28800
join_buffer_size
131072
key_buffer_size
367001600
key_cache_age_threshold
300
key_cache_block_size
1024
key_cache_division_limit
100
language
/usr/share/mysql/english/
large_files_support
ON
large_page_size
0
large_pages
OFF
lc_time_names
en_US
license
GPL
local_infile
OFF
locked_in_memory
OFF
log
OFF
log_bin
OFF
log_bin_trust_function_creators
OFF
log_error
/var/log/mysqld.log
log_queries_not_using_indexes
OFF
log_slave_updates
OFF
log_slow_queries
OFF
log_warnings
1
long_query_time
10
low_priority_updates
OFF
lower_case_file_system
OFF
lower_case_table_names
0
max_allowed_packet
1048576
max_binlog_cache_size
4294963200
max_binlog_size
1073741824
max_connect_errors
10
max_connections
100
max_delayed_threads
20
max_error_count
64
max_heap_table_size
16777216
max_insert_delayed_threads
20
max_join_size
4294967295
max_length_for_sort_data
1024
max_prepared_stmt_count
16382
max_relay_log_size
0
max_seeks_for_key
4294967295
max_sort_length
1024
max_sp_recursion_depth
0
max_tmp_tables
32
max_user_connections
50
max_write_lock_count
4294967295
multi_range_count
256
myisam_data_pointer_size
6
myisam_max_sort_file_size
2146435072
myisam_mmap_size
4294967295
myisam_recover_options
OFF
myisam_repair_threads
1
myisam_sort_buffer_size
8388608
myisam_stats_method
nulls_unequal
net_buffer_length
16384
net_read_timeout
30
net_retry_count
10
net_write_timeout
60
new
OFF
old_passwords
OFF
open_files_limit
20000
optimizer_prune_level
1
optimizer_search_depth
62
pid_file
/var/run/mysqld/mysqld.pid
plugin_dir
port
3306
preload_buffer_size
32768
profiling
OFF
profiling_history_size
15
protocol_version
10
query_alloc_block_size
8192
query_cache_limit
1048576
query_cache_min_res_unit
4096
query_cache_size
0
query_cache_type
ON
query_cache_wlock_invalidate
OFF
query_prealloc_size
8192
range_alloc_block_size
4096
read_buffer_size
131072
read_only
OFF
read_rnd_buffer_size
262144
relay_log
relay_log_index
relay_log_info_file
relay-log.info
relay_log_purge
ON
relay_log_space_limit
0
rpl_recovery_rank
0
secure_auth
OFF
secure_file_priv
server_id
0
skip_external_locking
ON
Variable_name
Value
skip_networking
OFF
skip_show_database
OFF
slave_compressed_protocol
OFF
slave_load_tmpdir
/tmp/
slave_net_timeout
3600
slave_skip_errors
OFF
slave_transaction_retries
10
slow_launch_time
2
socket
/var/lib/mysql/mysql.sock
sort_buffer_size
2097144
sql_big_selects
ON
sql_mode
sql_notes
ON
sql_warnings
OFF
ssl_ca
ssl_capath
ssl_cert
ssl_cipher
ssl_key
storage_engine
MyISAM
sync_binlog
0
sync_frm
ON
system_time_zone
CDT
table_cache
64
table_lock_wait_timeout
50
table_type
MyISAM
thread_cache_size
0
thread_stack
196608
time_format
%H:%i:%s
time_zone
SYSTEM
timed_mutexes
OFF
tmp_table_size
33554432
tmpdir
/tmp/
transaction_alloc_block_size
8192
transaction_prealloc_size
4096
tx_isolation
REPEATABLE-READ
updatable_views_with_limit
YES
version
5.0.95
version_bdb
Sleepycat Software: Berkeley DB 4.1.24: (December ...
version_comment
Source distribution
version_compile_machine
i386
version_compile_os
redhat-linux-gnu
wait_timeout
28800
Can anybody please help me tuning the system?
According to the status dump, your innodb_buffer_pool_size is the default 8mb (ie - tiny compared to the size of the database) and the MyIsam key_buffer_size is 350mb. Your machine also has 3GB of ram or so. A good chunk of it isn't being used - or is used by the OS for cache (which is where database may be - but it's not being actively controlled by Mysql, hence is slower).
I'd imagine that all the tables in the main database are set up as MyIsam tables. My first action if I were to go to optimise the server would be to increase the innodb_buffer_pool_size to around 1GB (a little more than the database size) and then convert all the databases to InnoDB. This brings the entire database into memory, under control of the database server.
Just changing those when you don't know what you are doing though, would not be be a good idea - but may give you enough thoughts to properly investigate Mysql server optimisations.
What CS-Cart version do you use? Try to upgrade up to 4.3.3 one. CS-Cart team made a good work to optimize the core and speed up page loading speed (especially pages with filters)