MySQL tuning - prestashop - mysql

Question

I use Prestashop+ NGINX + PHP-FPM, with more than 30.000 listings + variations, what are synchronized by an external API server.
My prestashop is running under:
AMD Ryzen™ 5 3600 (6 cores) + 64GB RAM + 2xNVME 500GB (RAID 1) + Gigabit ethernet/network
I am not an expert and i would love to get some recommendations
/etc/my.cnf
[mysql]
port = 3306
[mysqld]
# === Required Settings ===
basedir = /usr
bind_address = 127.0.0.1 # Change to 0.0.0.0 to allow remote connections
datadir = /var/lib/mysql
max_allowed_packet = 256M
max_connect_errors = 1000000
port = 3306
skip_external_locking
skip_name_resolve
tmpdir = /tmp
user = mysql
# === SQL Compatibility Mode ===
# Enable for b/c with databases created in older MySQL/MariaDB versions
# (e.g. when using null dates)
#sql_mode = ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION,ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES
# === InnoDB Settings ===
default_storage_engine = InnoDB
innodb_buffer_pool_instances = 42 # Use 1 instance per 1GB of InnoDB pool size
innodb_buffer_pool_size = 42G # Use up to 70-80% of RAM
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 0
innodb_flush_method = O_DIRECT
innodb_log_buffer_size = 16M
innodb_log_file_size = 2G
innodb_stats_on_metadata = 0
performance_schema = ON
#innodb_temp_data_file_path = ibtmp1:64M:autoextend:max:20G # Control the maximum size for the ibtmp1 file
#innodb_thread_concurrency = 4 # Optional: Set to the number of CPUs on your system (minus 1 or 2) to better
# contain CPU usage. E.g. if your system has 8 CPUs, try 6 or 7 and check
# the overall load produced by MySQL/MariaDB.
innodb_read_io_threads = 64
innodb_write_io_threads = 64
#innodb_io_capacity = 1000 # Max is 2000
# === MyISAM Settings ===
# The following 3 options are ONLY supported by MariaDB & up to MySQL 5.7
# Do NOT un-comment on MySQL 8.x+
query_cache_limit = 10M # UPD
query_cache_size = 50000M # UPD
query_cache_type = 1 # Enabled by default
key_buffer_size = 32M # UPD
low_priority_updates = 1
concurrent_insert = 2
# === Connection Settings ===
max_connections = 750 # UPD - Important: high no. of connections = high RAM consumption
back_log = 512
thread_cache_size = 100
thread_stack = 192K
interactive_timeout = 180
wait_timeout = 180
# For MySQL 5.7+ only (disabled by default)
#max_execution_time = 30000 # Set a timeout limit for SELECT statements (value in milliseconds).
# This option may be useful to address aggressive crawling on large sites,
# but it can also cause issues (e.g. with backups). So use with extreme caution and test!
# More info at: https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_max_execution_time
# For MariaDB 10.1.1+ only (disabled by default)
#max_statement_time = 30 # The equivalent of "max_execution_time" in MySQL 5.7+ (set above)
# The variable is of type double, thus you can use subsecond timeout.
# For example you can use value 0.01 for 10 milliseconds timeout.
# More info at: https://mariadb.com/kb/en/aborting-statements/
# === Buffer Settings ===
innodb_sort_buffer_size = 2M # UPD
join_buffer_size = 8M # UPD
read_buffer_size = 3M # UPD
read_rnd_buffer_size = 4M # UPD
sort_buffer_size = 4M # UPD
# === Table Settings ===
# In systemd managed systems like Ubuntu 16.04+ or CentOS 7+, you need to perform an extra action for table_open_cache & open_files_limit
# to be overriden (also see comment next to open_files_limit).
# E.g. for MySQL 5.7, please check: https://dev.mysql.com/doc/refman/5.7/en/using-systemd.html
# and for MariaDB check: https://mariadb.com/kb/en/library/systemd/
table_definition_cache = 40000 # UPD
table_open_cache = 40000 # UPD
open_files_limit = 80000 # UPD - This can be 2x to 3x the table_open_cache value or match the system's
# open files limit usually set in /etc/sysctl.conf or /etc/security/limits.conf
# In systemd managed systems this limit must also be set in:
# /etc/systemd/system/mysqld.service.d/override.conf (for MySQL 5.7+) and
# /etc/systemd/system/mariadb.service.d/override.conf (for MariaDB)
max_heap_table_size = 128M
tmp_table_size = 128M
# === Search Settings ===
ft_min_word_len = 3 # Minimum length of words to be indexed for search results
# === Logging ===
log_error = /var/lib/mysql/mysql_error.log
log_queries_not_using_indexes = 1
long_query_time = 5
slow_query_log = 1 # Disabled for production
slow_query_log_file = /var/lib/mysql/mysql_slow.log
[mysqldump]
# Variable reference
# For MySQL 5.7+: https://dev.mysql.com/doc/refman/5.7/en/mysqldump.html
# For MariaDB: https://mariadb.com/kb/en/library/mysqldump/
quick
quote_names
max_allowed_packet = 512M
tuning-primer.sh
[root#server2 /]# curl -L https://raw.githubusercontent.com/BMDan/tuning-primer.sh/master/tuning-primer.sh | bash
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 63986 100 63986 0 0 201k 0 --:--:-- --:--:-- --:--:-- 202k
-- MYSQL PERFORMANCE TUNING PRIMER --
- By: Matthew Montgomery -
MySQL Version 10.2.38-MariaDB-log x86_64
Uptime = 3 days 21 hrs 47 min 51 sec
Avg. qps = 18
Total Questions = 6119342
Threads Connected = 11
Server has been running for over 48hrs.
It should be safe to follow these recommendations
To find out more information on how each of these
runtime variables effects performance visit:
http://dev.mysql.com/doc/refman/10.2/en/server-system-variables.html
Visit http://www.mysql.com/products/enterprise/advisors.html
for info about MySQL's Enterprise Monitoring and Advisory Service
SLOW QUERIES
The slow query log is enabled.
Current long_query_time = 5.000000 sec.
Since startup, 102804 out of 6119356 queries have taken longer than <long_query_time-when-they-were-executed> to complete.
Your long_query_time may be too high, I typically set this under 5 sec.
BINARY UPDATE LOG
The binary update log is NOT enabled.
You will not be able to do point in time recovery
See http://dev.mysql.com/doc/refman/10.2/en/point-in-time-recovery.html
WORKER THREADS
Current thread_cache_size = 100
Current threads_cached = 42
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine
MAX CONNECTIONS
Current max_connections = 750
Current threads_connected = 11
Historic max_used_connections = 54
The number of used connections is 7% of the configured maximum.
You are using less than 10% of your configured max_connections.
Lowering max_connections could help to avoid an over-allocation of memory
See "MEMORY USAGE" section to make sure you are not over-allocating
INNODB STATUS
Current InnoDB index space = 137 M
Current InnoDB data space = 255 M
Current InnoDB buffer pool free = 99 %
Current innodb_buffer_pool_size = 42.00 G
Depending on how much space your innodb indexes take up it may be safe
to increase this value to up to 2 / 3 of total system memory
MEMORY USAGE
Max Memory Ever Allocated : 91.88 G
Configured Max Per-thread Buffers : 14.05 G
Configured Max Global Buffers : 90.87 G
Configured Max Memory Limit : 104.92 G
Physical Memory : 62.75 G
Max memory limit exceeds 90% of physical memory
KEY BUFFER
Current MyISAM index space = 185 K
Current key_buffer_size = 32 M
Key cache miss rate is 1 : 697
Key buffer free ratio = 81 %
Your key_buffer_size seems to be fine
QUERY CACHE
Query cache is enabled
Current query_cache_size = 48.82 G
Current query_cache_used = 1.17 G
Current query_cache_limit = 10 M
Current Query cache Memory fill ratio = 2.39 %
Current query_cache_min_res_unit = 4 K
Your query_cache_size seems to be too high.
Perhaps you can use these resources elsewhere
MySQL won't cache query results that are larger than query_cache_limit in size
SORT OPERATIONS
Current sort_buffer_size = 4 M
Current read_rnd_buffer_size = 4 M
Sort buffer seems to be fine
JOINS
Current join_buffer_size = 8.00 M
You have had 15352 queries where a join could not use an index properly
join_buffer_size >= 4 M
This is not advised
You should enable "log-queries-not-using-indexes"
Then look for non indexed joins in the slow query log.
OPEN FILES LIMIT
Current open_files_limit = 32768 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.
Your open_files_limit value seems to be fine
TABLE CACHE
Current table_open_cache = 15994 tables
Current table_definition_cache = 40000 tables
You have a total of 483 tables
You have 557 open tables.
The table_cache value seems to be fine
TEMP TABLES
Current max_heap_table_size = 128 M
Current tmp_table_size = 128 M
Of 146965 temp tables, 6% were created on disk
Created disk tmp tables ratio seems fine
TABLE SCANS
Current read_buffer_size = 3 M
Current table scan ratio = 88 : 1
read_buffer_size seems to be fine
TABLE LOCKING
Current Lock Wait ratio = 1 : 37
You may benefit from selective use of InnoDB.

Answer 1

Current query_cache_size = 48.82 G -- NO! Stop at about 50M; it gets terribly inefficient when it is that large.
Use pt-query-digest on the slowlog; then let's discuss the first few queries. That is likely to help performance a lot.
log_queries_not_using_indexes = 1 -- turn off; otherwise it clutters the slowlog.
We will tackle this in a second pass:
max_connections = 750 # UPD
table_definition_cache = 40000 # UPD
table_open_cache = 40000 # UPD
If you would like more analysis, see http://mysql.rjweb.org/doc.php/mysql_analysis#tuning
Digest
Query 1
UPDATE ps_stock_available sa
INNER JOIN ps_mmo_connector_product_map pm
ON sa.id_product = pm.id_product_shop
SET sa.quantity = '10'
WHERE sa.quantity > '10'\G
The first query in the digest is an Update that seems to cap quantity. Presumably, you can eliminate that by never changing the value to more than 10? That would eliminate half the burden on the server. If you don't want to do that, please provide these so I can check the indexes:
SHOW CREATE TABLE `presta`.`ps_stock_available`\G
SHOW CREATE TABLE `presta`.`ps_mmo_connector_product_map`\G
But... What is the problem? This implies that there is not much going on:
# Overall: 281.15k total, 799 unique, 0.54 QPS, 0.01x concurrency
And this says that the "worst" query (that Update) is not very slow or frequent:
# Exec time 48 1313s 18ms 1s 85ms 253ms 88ms 53ms
If you expect the server to become much, much, busier, then I will dig deeper. But for now, the only thing that server will do is snore.
While you are getting those Creates, grab these, too, so I can check the next few queries:
SHOW CREATE TABLE `presta`.`ps_category_product`\G
SHOW CREATE TABLE `presta`.`ps_category`\G
SHOW CREATE TABLE `presta`.`ps_category_shop`\G
SHOW CREATE TABLE `presta`.`ps_search_word`\G
SHOW CREATE TABLE `presta`.`ps_search_index`\G
SHOW TABLE STATUS FROM `presta` LIKE 'ps_tag_count'\G
I see a REPLACE that seems to be recomputing tag counts. At first glance, that seems a terribly inefficient way to do it. But looking at how complex the query is, it may not be practical to improve it. Do you the intent of Query 4?