I have a 2GB DB and it is very slow below my configuration file variable , what I shall change to make it faster
read_buffer_size=1M
read_rnd_buffer_size=256K
sort_buffer_size=256K
innodb_additional_mem_pool_size=2M
innodb_flush_log_at_trx_commit=1
innodb_log_buffer_size=1M
innodb_buffer_pool_size=16M
innodb_log_file_size=10M
innodb_thread_concurrency=8
max_connections=151
old_passwords=0
max_user_connections=30
thread_cache_size=4
thread_stack=196608
binlog_cache_size=32768
net_read_timeout=30
net_retry_count=10
net_write_timeout=30
thread_concurrency=10
open_files_limit=4161
table_definition_cache=400
Threads_cached 2
Threads_connected 2
Threads_created 12
Threads_running 1
Usually I'll just leave the rest alone and only adjust:
innodb_buffer_pool_size to around 70~80% or your 2GB, in this case 1.6G
innodb_log_file_size to half of above, in this case 800M
max_connections depends on your application, normally I'll set it to at least 200.
But "slow" is very general and it may be impacted by your logic/structure as well.
Related
I use Prestashop+ NGINX + PHP-FPM, with more than 30.000 listings + variations, what are synchronized by an external API server.
My prestashop is running under:
AMD Ryzen™ 5 3600 (6 cores) + 64GB RAM + 2xNVME 500GB (RAID 1) + Gigabit ethernet/network
I am not an expert and i would love to get some recommendations
/etc/my.cnf
[mysql]
port = 3306
[mysqld]
# === Required Settings ===
basedir = /usr
bind_address = 127.0.0.1 # Change to 0.0.0.0 to allow remote connections
datadir = /var/lib/mysql
max_allowed_packet = 256M
max_connect_errors = 1000000
port = 3306
skip_external_locking
skip_name_resolve
tmpdir = /tmp
user = mysql
# === SQL Compatibility Mode ===
# Enable for b/c with databases created in older MySQL/MariaDB versions
# (e.g. when using null dates)
#sql_mode = ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION,ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES
# === InnoDB Settings ===
default_storage_engine = InnoDB
innodb_buffer_pool_instances = 42 # Use 1 instance per 1GB of InnoDB pool size
innodb_buffer_pool_size = 42G # Use up to 70-80% of RAM
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 0
innodb_flush_method = O_DIRECT
innodb_log_buffer_size = 16M
innodb_log_file_size = 2G
innodb_stats_on_metadata = 0
performance_schema = ON
#innodb_temp_data_file_path = ibtmp1:64M:autoextend:max:20G # Control the maximum size for the ibtmp1 file
#innodb_thread_concurrency = 4 # Optional: Set to the number of CPUs on your system (minus 1 or 2) to better
# contain CPU usage. E.g. if your system has 8 CPUs, try 6 or 7 and check
# the overall load produced by MySQL/MariaDB.
innodb_read_io_threads = 64
innodb_write_io_threads = 64
#innodb_io_capacity = 1000 # Max is 2000
# === MyISAM Settings ===
# The following 3 options are ONLY supported by MariaDB & up to MySQL 5.7
# Do NOT un-comment on MySQL 8.x+
query_cache_limit = 10M # UPD
query_cache_size = 50000M # UPD
query_cache_type = 1 # Enabled by default
key_buffer_size = 32M # UPD
low_priority_updates = 1
concurrent_insert = 2
# === Connection Settings ===
max_connections = 750 # UPD - Important: high no. of connections = high RAM consumption
back_log = 512
thread_cache_size = 100
thread_stack = 192K
interactive_timeout = 180
wait_timeout = 180
# For MySQL 5.7+ only (disabled by default)
#max_execution_time = 30000 # Set a timeout limit for SELECT statements (value in milliseconds).
# This option may be useful to address aggressive crawling on large sites,
# but it can also cause issues (e.g. with backups). So use with extreme caution and test!
# More info at: https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_max_execution_time
# For MariaDB 10.1.1+ only (disabled by default)
#max_statement_time = 30 # The equivalent of "max_execution_time" in MySQL 5.7+ (set above)
# The variable is of type double, thus you can use subsecond timeout.
# For example you can use value 0.01 for 10 milliseconds timeout.
# More info at: https://mariadb.com/kb/en/aborting-statements/
# === Buffer Settings ===
innodb_sort_buffer_size = 2M # UPD
join_buffer_size = 8M # UPD
read_buffer_size = 3M # UPD
read_rnd_buffer_size = 4M # UPD
sort_buffer_size = 4M # UPD
# === Table Settings ===
# In systemd managed systems like Ubuntu 16.04+ or CentOS 7+, you need to perform an extra action for table_open_cache & open_files_limit
# to be overriden (also see comment next to open_files_limit).
# E.g. for MySQL 5.7, please check: https://dev.mysql.com/doc/refman/5.7/en/using-systemd.html
# and for MariaDB check: https://mariadb.com/kb/en/library/systemd/
table_definition_cache = 40000 # UPD
table_open_cache = 40000 # UPD
open_files_limit = 80000 # UPD - This can be 2x to 3x the table_open_cache value or match the system's
# open files limit usually set in /etc/sysctl.conf or /etc/security/limits.conf
# In systemd managed systems this limit must also be set in:
# /etc/systemd/system/mysqld.service.d/override.conf (for MySQL 5.7+) and
# /etc/systemd/system/mariadb.service.d/override.conf (for MariaDB)
max_heap_table_size = 128M
tmp_table_size = 128M
# === Search Settings ===
ft_min_word_len = 3 # Minimum length of words to be indexed for search results
# === Logging ===
log_error = /var/lib/mysql/mysql_error.log
log_queries_not_using_indexes = 1
long_query_time = 5
slow_query_log = 1 # Disabled for production
slow_query_log_file = /var/lib/mysql/mysql_slow.log
[mysqldump]
# Variable reference
# For MySQL 5.7+: https://dev.mysql.com/doc/refman/5.7/en/mysqldump.html
# For MariaDB: https://mariadb.com/kb/en/library/mysqldump/
quick
quote_names
max_allowed_packet = 512M
tuning-primer.sh
[root#server2 /]# curl -L https://raw.githubusercontent.com/BMDan/tuning-primer.sh/master/tuning-primer.sh | bash
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 63986 100 63986 0 0 201k 0 --:--:-- --:--:-- --:--:-- 202k
-- MYSQL PERFORMANCE TUNING PRIMER --
- By: Matthew Montgomery -
MySQL Version 10.2.38-MariaDB-log x86_64
Uptime = 3 days 21 hrs 47 min 51 sec
Avg. qps = 18
Total Questions = 6119342
Threads Connected = 11
Server has been running for over 48hrs.
It should be safe to follow these recommendations
To find out more information on how each of these
runtime variables effects performance visit:
http://dev.mysql.com/doc/refman/10.2/en/server-system-variables.html
Visit http://www.mysql.com/products/enterprise/advisors.html
for info about MySQL's Enterprise Monitoring and Advisory Service
SLOW QUERIES
The slow query log is enabled.
Current long_query_time = 5.000000 sec.
Since startup, 102804 out of 6119356 queries have taken longer than <long_query_time-when-they-were-executed> to complete.
Your long_query_time may be too high, I typically set this under 5 sec.
BINARY UPDATE LOG
The binary update log is NOT enabled.
You will not be able to do point in time recovery
See http://dev.mysql.com/doc/refman/10.2/en/point-in-time-recovery.html
WORKER THREADS
Current thread_cache_size = 100
Current threads_cached = 42
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine
MAX CONNECTIONS
Current max_connections = 750
Current threads_connected = 11
Historic max_used_connections = 54
The number of used connections is 7% of the configured maximum.
You are using less than 10% of your configured max_connections.
Lowering max_connections could help to avoid an over-allocation of memory
See "MEMORY USAGE" section to make sure you are not over-allocating
INNODB STATUS
Current InnoDB index space = 137 M
Current InnoDB data space = 255 M
Current InnoDB buffer pool free = 99 %
Current innodb_buffer_pool_size = 42.00 G
Depending on how much space your innodb indexes take up it may be safe
to increase this value to up to 2 / 3 of total system memory
MEMORY USAGE
Max Memory Ever Allocated : 91.88 G
Configured Max Per-thread Buffers : 14.05 G
Configured Max Global Buffers : 90.87 G
Configured Max Memory Limit : 104.92 G
Physical Memory : 62.75 G
Max memory limit exceeds 90% of physical memory
KEY BUFFER
Current MyISAM index space = 185 K
Current key_buffer_size = 32 M
Key cache miss rate is 1 : 697
Key buffer free ratio = 81 %
Your key_buffer_size seems to be fine
QUERY CACHE
Query cache is enabled
Current query_cache_size = 48.82 G
Current query_cache_used = 1.17 G
Current query_cache_limit = 10 M
Current Query cache Memory fill ratio = 2.39 %
Current query_cache_min_res_unit = 4 K
Your query_cache_size seems to be too high.
Perhaps you can use these resources elsewhere
MySQL won't cache query results that are larger than query_cache_limit in size
SORT OPERATIONS
Current sort_buffer_size = 4 M
Current read_rnd_buffer_size = 4 M
Sort buffer seems to be fine
JOINS
Current join_buffer_size = 8.00 M
You have had 15352 queries where a join could not use an index properly
join_buffer_size >= 4 M
This is not advised
You should enable "log-queries-not-using-indexes"
Then look for non indexed joins in the slow query log.
OPEN FILES LIMIT
Current open_files_limit = 32768 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.
Your open_files_limit value seems to be fine
TABLE CACHE
Current table_open_cache = 15994 tables
Current table_definition_cache = 40000 tables
You have a total of 483 tables
You have 557 open tables.
The table_cache value seems to be fine
TEMP TABLES
Current max_heap_table_size = 128 M
Current tmp_table_size = 128 M
Of 146965 temp tables, 6% were created on disk
Created disk tmp tables ratio seems fine
TABLE SCANS
Current read_buffer_size = 3 M
Current table scan ratio = 88 : 1
read_buffer_size seems to be fine
TABLE LOCKING
Current Lock Wait ratio = 1 : 37
You may benefit from selective use of InnoDB.
Current query_cache_size = 48.82 G -- NO! Stop at about 50M; it gets terribly inefficient when it is that large.
Use pt-query-digest on the slowlog; then let's discuss the first few queries. That is likely to help performance a lot.
log_queries_not_using_indexes = 1 -- turn off; otherwise it clutters the slowlog.
We will tackle this in a second pass:
max_connections = 750 # UPD
table_definition_cache = 40000 # UPD
table_open_cache = 40000 # UPD
If you would like more analysis, see http://mysql.rjweb.org/doc.php/mysql_analysis#tuning
Digest
Query 1
UPDATE ps_stock_available sa
INNER JOIN ps_mmo_connector_product_map pm
ON sa.id_product = pm.id_product_shop
SET sa.quantity = '10'
WHERE sa.quantity > '10'\G
The first query in the digest is an Update that seems to cap quantity. Presumably, you can eliminate that by never changing the value to more than 10? That would eliminate half the burden on the server. If you don't want to do that, please provide these so I can check the indexes:
SHOW CREATE TABLE `presta`.`ps_stock_available`\G
SHOW CREATE TABLE `presta`.`ps_mmo_connector_product_map`\G
But... What is the problem? This implies that there is not much going on:
# Overall: 281.15k total, 799 unique, 0.54 QPS, 0.01x concurrency
And this says that the "worst" query (that Update) is not very slow or frequent:
# Exec time 48 1313s 18ms 1s 85ms 253ms 88ms 53ms
If you expect the server to become much, much, busier, then I will dig deeper. But for now, the only thing that server will do is snore.
While you are getting those Creates, grab these, too, so I can check the next few queries:
SHOW CREATE TABLE `presta`.`ps_category_product`\G
SHOW CREATE TABLE `presta`.`ps_category`\G
SHOW CREATE TABLE `presta`.`ps_category_shop`\G
SHOW CREATE TABLE `presta`.`ps_search_word`\G
SHOW CREATE TABLE `presta`.`ps_search_index`\G
SHOW TABLE STATUS FROM `presta` LIKE 'ps_tag_count'\G
I see a REPLACE that seems to be recomputing tag counts. At first glance, that seems a terribly inefficient way to do it. But looking at how complex the query is, it may not be practical to improve it. Do you the intent of Query 4?
I',m having a huge issue with running MySQL#5.7 on my freshly installed 16" Macbook (with OSX 10.15.1 Catalina) During certain actions I get errors like
PDO::__construct(): MySQL server has gone away.
This is caused by the following error I found in the MySQL log.
2019-11-27T13:24:04.835245Z 0 [Warning] File Descriptor 3226 exceeded FD_SETSIZE=1024
After some research, I tried stuff like sudo launchctl limit maxfiles 65536 200000
When i run launchctl limit i get the follwing data
cpu unlimited unlimited
filesize unlimited unlimited
data unlimited unlimited
stack 8388608 67104768
core 0 unlimited
rss unlimited unlimited
memlock unlimited unlimited
maxproc 4096 4096
maxfiles 524288 524288
This looks fine to me. To get the max processes and max files correct I also tried
sudo sysctl -w kern.maxfilesperproc=524288
my.cnf looks like this
[mysqld]
open_files_limit=999999
local_infile=ON
secure_file_priv=""
max_allowed_packet=1073741824
max_connections=100000
key_buffer_size=2G
innodb_buffer_pool_size=12G
query_cache_size=67108864
query_cache_type=1
query_cache_limit=4194304
table_open_cache=4096
innodb_buffer_pool_instances=24
innodb_sort_buffer_size=2G
sort_buffer_size=1G
innodb_flush_log_at_trx_commit=0
innodb_log_file_size=3G
interactive_timeout=3600
max_connect_errors=1000000
thread_cache_size=4096
log_error=/var/log/mysql/error.log
[mysqld_safe]
open_files_limit=999999
There is of course a solution to change my table_open_cache to a lower value but that hurts performance and before i always had this on a higher number.
Anybody any clue where this FD_SETSIZE is coming from and how to change it so its used properly
rebooting has no effect by the way.
Resource explaining issue: https://expressionengine.com/blog/mysql-5.7-server-os-x-has-gone-away
try setting following environment variables in mysql configuration file (my.cnf)
interactive_timeout = 300
wait_timeout = 300
I was having this issue on Big Sur v 11.6. The solution for me was modifying the MySQL config with:
max_allowed_packet=256M
table_open_cache=250
3 weeks ago we migrated from Mysql 5.6.22 to 5.7.23 Using Logical Method. after that we started to get bug error which is already submitted, but now the DB started to slow down and we need to restart every 10 hours approximately.
This is reflected in a trigger on a table that receives about 700 Records per minute, when just started every record is processed in about 22ms. gradually start to take more and more time up to 300ms. when it is not enough and a bottleneck start to build, so we restart the DB and works fast again.
The innodb buffer pool does not even gets to 15% of usage.
The trigger performs various selects and gather information to fill up the rest of the record, nothing really fancy from tables with no more than 3k records.
Any suggestions of what may be causing the issue?
Thanks in Advance
Hardware Specs:
Nombre del sistema operativo: Microsoft Windows Server 2012 Standard
Versión del sistema operativo: 6.2.9200 N/D Compilación 9200
Fabricante del sistema operativo: Microsoft Corporation
Configuración del sistema operativo: Servidor independiente
Tipo de compilación del sistema operativo: Multiprocessor Free
Fabricante del sistema: VMware, Inc.
Modelo el sistema: VMware Virtual Platform
Tipo de sistema: x64-based PC
Procesador(es): 2 Procesadores instalados.
[01]: Intel64 Family 6 Model 45 Stepping 7 GenuineIntel ~2400 Mhz
[02]: Intel64 Family 6 Model 45 Stepping 7 GenuineIntel ~2400 Mhz
Versión del BIOS: Phoenix Technologies LTD 6.00, 22/06/2012
Cantidad total de memoria física: 49,151 MB
Memoria física disponible: 37,501 MB
Memoria virtual: tamaño máximo: 55,551 MB
Memoria virtual: disponible: 21,520 MB
Memoria virtual: en uso: 34,031 MB
Server Configuration:
[mysqld]
# Storage
basedir=e:/DB5723/
datadir=e:/DB5723/Data/
innodb_file_per_table=ON
#Memory Resources
innodb_buffer_pool_size = 24G
innodb_buffer_pool_instances=8
innodb_buffer_pool_dump_pct=75
innodb_lru_scan_depth=1024
innodb_log_buffer_size = 128M
innodb_log_file_size=512M
max_allowed_packet = 4M
max_heap_table_size = 64M
join_buffer_size = 1M
sort_buffer_size = 64M
read_rnd_buffer_size = 16M
query_cache_size=0
query_cache_type=OFF
# Connections
skip-name-resolve
max_connections = 151
max_connect_errors = 5
socket=e:/DB5723/tmp/mysql.sock
port=3307
wait_timeout=600
# Thread Setting
innodb_thread_concurrency=16
innodb_read_io_threads=8
innodb_write_io_threads=8
thread_cache_size = 9
#Error Control / Log
log_error_verbosity=3
general_log_file=e:/DB5723/Data/General.log
# Security ( Brute Force Attack )
plugin-load-add=connection_control.dll
connection-control=FORCE_PLUS_PERMANENT
connection-control-failed-login-attempts=FORCE_PLUS_PERMANENT
connection_control_failed_connections_threshold=3
connection_control_max_connection_delay=120000
connection_control_min_connection_delay=60000
# Replication
server_id=1
log_bin=g:/DB5723-bin
max_binlog_size=512M
binlog_format = ROW
sync_binlog=1
# Miscelaneous
character-set-server=utf8
event_scheduler = ON
innodb_lock_wait_timeout=50
explicit_defaults_for_timestamp=ON
innodb_checksum_algorithm=crc32
innodb_checksums=ON
sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
# I/O Hdd Performance Settings
innodb_io_capacity=800
innodb_io_capacity_max=1000
innodb_flush_log_at_trx_commit=2
A quick fix might be to change to
sync_binlog = 0
For more discussion, please provide the query that fires the trigger, the trigger definition, CREATE TABLE, and Max_used_connections.
Also, what type of disk is involved (HDD vs SSD)?
Suggestions to consider for your my.ini [mysqld] section
Rate Per Second=RPS
innodb_lru_scan_depth=100 # from 1024 see refman to reduce CPU load every SECOND
sort_buffer_size=2M # from 64M a per connection RAM request and 64M is TOO MUCH
read_rnd_buffer_size=256K # from 16M to reduce handler_read_rnd_next RPS
thread_cache_size=48 # from 9 to reduce threads_created count
innodb_io_capacity_max=2000 # from 1000 for default max
innodb_io_capacity=1600 # from 800 to allow more IOPS to your HDD
for additional suggestions, view my profile, Network profile for contact info, please.
Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 262145 bytes)
I try to Import a csv file 199MB into mysql database.
upload_max_filesize = 300M
post_max_size = 300M
memory_limit = 32M
It's keep pop the error, what should I change in php.ini
33554432 bytes is exactly 32M.
Try to boost up your memory_limit alot.
You shoul try to increase the memory limit. In PHP > 5.2.1 the default value is 128M.
; Maximum amount of memory a script may consume (128MB)
; http://www.php.net/manual/en/ini.core.php#ini.memory-limit
;memory_limit = 32M
memory_limit = 128M
I am getting this error from mysql
ERROR 20 (HY000) at line 1: Disk is full writing './homo_sapiens_core_60_37e/dna_align_feature.MYD' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)
I think this could be because i only have space on /local (df - h gives)
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/components-root
19G 18G 0 100% /
none 1.4G 240K 1.4G 1% /dev
none 1.4G 100K 1.4G 1% /dev/shm
none 1.4G 80K 1.4G 1% /var/run
none 1.4G 0 1.4G 0% /var/lock
none 1.4G 0 1.4G 0% /lib/init/rw
none 19G 18G 0 100% /var/lib/ureadahead/debugfs
/dev/sdb1 241G 43G 186G 19% /local
/dev/sda1 228M 51M 166M 24% /boot
How do you specify where mysql writes its databases to? Perhaps there is no space where it is writing the database?
How do you configure where mysql writes its databases? perhaps there is no space where it is trying to write teh data?
I am a unix novice and don't know much mysql either so please be gentle in your answers!
thanks
In your /etc/my.cnf, under the heading [mysqld], your the data directory is identified by datadir.
Also perhaos of interest, in case they were changed from their default, you should also check
the location of your temporary directory tmpdir
the location (log-bin) and history length (expire-log-days) of binary logs you're saving (if any)
the innodb directories (innodb_data_file_path and innodb_data_home_dir)
the location of your slow logs with slow-query-log-file (the size of these logs will depend on how low your slow threshold is).
You can check all of these by doing a SHOW VARIABLES from the MySQL command line. Also note that MySQL only reads the values in the /etc/my.cnf upon startup, so changing the file while MySQL is running won't do anything.
Lastly, you might get more help/ideas at serverfault.
If you have /etc/my.cnf then it may be helpful. If I understand correctly,, The data directory is specified either in startup options or it is defaulted from that file. You would want to edit my.cnf and choose a directory in /local and then manually transfer your existing data from wherever it was to your new place in /local. Make sure MySQL is not running during the transfer.