I'm moving from a shared web hosting company to a VPS. And I'm trying to find the optimum configuration for my system.
For 1 specific table with half a million record, 1 query that was taking few seconds on the previous server is now taking minutes.
I'm trying to improve the response time of my server, so I bought more memory (I have 2GB of RAM, and I can still buy more resources if needed and 2 cores). I have also "cheated" by copying my.cnf of the old company to my VPS:
[mysqld]
read_buffer_size=8M
read_rnd_buffer_size=8M
sort_buffer_size=32M
innodb_additional_mem_pool_size=503M
innodb_flush_log_at_trx_commit=1
innodb_log_buffer_size=16M
innodb_buffer_pool_size=1007M
innodb_log_file_size=256M
innodb_thread_concurrency=8
innodb_autoextend_increment=128
max_connections=8059
max_user_connections=50
thread_cache_size=128
thread_stack=196608
binlog_cache_size=2M
net_read_timeout=30
net_retry_count=10
net_write_timeout=30
thread_concurrency=10
open_files_limit=9940
max_heap_table_size=32M
tmp_table_size=64M
key_buffer_size=512M
key_buffer=128M
myisam_sort_buffer_size=64M
join_buffer=16M
record_buffer=8M
wait_timeout=300
connect_timeout=10
max_allowed_packet=16M
max_connect_errors=100
table_cache=1024
query_cache_size=32M
query_cache_type=1
ft_min_word_len=4
datadir=/var/lib/mysql
tmpdir=/tmp
socket=/var/lib/mysql/mysql.sock
old-passwords=0
[mysqldump]
quick
max_allowed_packet=16M
[myisamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M
The problem is that the query is still slow, and the server does not use the available memory!
total used free shared buffers cached
Mem: 2002 1986 15 0 6 1079
-/+ buffers/cache: 901 1101
Swap: 1747 2 1745
Any suggestions?
Regards
You can have a look at using
https://github.com/rackerhacker/MySQLTuner-perl
Which you can use to analyse MySQL and it will output recommended settings.
Your settings seem way high for 2gb memory.
IE max_connections=8059
That in its self would require way more memory than you have.
[mysqld]
max_connections=100
innodb_buffer_pool_size=256M
query_cache_size=256M
key_buffer_size=256M
innodb_flush_log_at_trx_commit=0
innodb_flush_method=O_DIRECT
query_cache_type=1
query_cache_limit=2M
table_cache=1024
join_buffer_size=4M
thread_cache_size=128
tmp_table_size=256M
max_heap_table_size=256MB
For 2gb RAM you could try using these settings in MySQL
These are the recommended settings for one of the 2gb servers I am currently using
Related
good morning This is my problem
I have two servers
first A: AMD Epyc 7371 - 16c / 32t - 3.1 GHz / 3.8 GHz 256 GB ECC 2400 MHz 2 × 960 GB NVMe SSD MySql 8.0.26
second B: Dual Intel Xeon Gold 6242R - 20c / 40t - 3.1 GHz / 4.1 GHz 384 GB ECC 2933 MHz 6 × 3.84 TB NVMe SSD 2 × 480 GB SATA SSD
On server A, despite being smaller, the database works well and transactions are very fast in terms of users on this server they reach over 200 in councurrency
On the much larger Server B this happens to me: up to 50 concurrent users is not a problem the database works well, after the 50 users to go up, the database starts to slow down.
I specify that both servers have the same database, same tables, same Stored Procedure, same applications, in practice they are a copy of each other
I lost sleep to figure out where the problem lies
This is the configuration file
[client]
pipe=
socket=MYSQL
port=3306
[mysql] no-beep
default-character-set=
server_type=1
[mysqld]
port=3306
datadir=E:/ProgramData/MySQL/MySQL Server 8.0\Data
character-set-server=
default-storage-engine=INNODB
sql-mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION"
log-output=FILE
general-log=0
general_log_file="NS31525947.log"
slow-query-log=1
slow_query_log_file="NS31525947-slow.log"
long_query_time=10
log-error="NS31525947.err"
log-bin="NS31525947-bin"
server-id=1
lower_case_table_names=1
secure-file-priv="C:/ProgramData/MySQL/MySQL Server 8.0/Uploads"
max_connections = 500
table_open_cache=3G
tmp_table_size=3G
thread_cache_size=100
myisam_max_sort_file_size=10G
myisam_sort_buffer_size=68G
key_buffer_size=61M
read_buffer_size=23M
read_rnd_buffer_size=256K
innodb_flush_log_at_trx_commit=0
innodb_log_buffer_size=512M
innodb_buffer_pool_size=10G
innodb_log_file_size=1G
innodb_thread_concurrency=0
innodb_autoextend_increment=64
innodb_buffer_pool_instances=8
innodb_concurrency_tickets=5000
innodb_old_blocks_time=1000
innodb_stats_on_metadata=0
innodb_file_per_table=1
innodb_checksum_algorithm=0
back_log=80
flush_time=0
join_buffer_size=256K
max_allowed_packet=4M
max_connect_errors=100
open_files_limit=10000
sort_buffer_size=256K
table_definition_cache=1400
binlog_row_event_max_size=8K
sync_relay_log=10000
sync_relay_log_info=10000
loose_mysqlx_port=33060
default_authentication_plugin = mysql_native_password
I state I also tried to put the same My.ini as server A on server B and nothing i put logs to track write times, and in this case past the 50 users on the B server the writes slow down The strange thing that i noticed that the value of the Key Efficient is always 0.0% unlike Server A the value is 100%
Please Help me Thanks
After upgraded the MySQL from version 5.7 to 8.0, I found out that the database performance is significant drop.
Before upgrade the MySQL the CPU usage is stable around 30%+-, but after upgraded the CPU usage is become unstable and frequently having large spike.
And recently I test out something very interesting, I'm keep run a same query for a few time, and found out that the duration taken becomes longer and longer. as per picture shown below.
I had read a lot of article and stack overflow post, but none of the solution is really get help.
So hope that someone can share some idea or experience on tuning the MySQL8.0 with me.
Will very appreciate it.
Please let me know if needed any info for further investigate.
Config my.ini:-
key_buffer_size = 2G
max_allowed_packet = 1M
;Added to reduce memory used (minimum is 400)
table_definition_cache = 600
sort_buffer_size = 4M
net_buffer_length = 8K
read_buffer_size = 2M
read_rnd_buffer_size = 2M
myisam_sort_buffer_size = 2G
;Path to mysql install directory
basedir="c:/wamp64/bin/mysql/mysql8.0.20"
log-error="c:/wamp64/logs/mysql.log"
;Verbosity Value 1 Errors only, 2 Errors and warnings , 3 Errors, warnings, and notes
log_error_verbosity=2
;Path to data directory
datadir="c:/wamp64/bin/mysql/mysql8.0.20/data"
;slow_query_log = ON
;slow_query_log_file = "c:/wamp64/logs/slow_query.log"
;Path to the language
;See Documentation:
; http://dev.mysql.com/doc/refman/5.7/en/error-message-language.html
lc-messages-dir="c:/wamp64/bin/mysql/mysql8.0.20/share"
lc-messages=en_US
; The default storage engine that will be used when create new tables
default-storage-engine=InnoDB
; New for MySQL 5.6 default_tmp_storage_engine if skip-innodb enable
; default_tmp_storage_engine=MYISAM
;To avoid warning messages
secure_file_priv="c:/wamp64/tmp"
skip-ssl
explicit_defaults_for_timestamp=true
; Set the SQL mode to strict
sql-mode=""
;sql-mode="STRICT_ALL_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_ZERO_DATE,NO_ZERO_IN_DATE,NO_AUTO_CREATE_USER"
;skip-networking
; Disable Federated by default
skip-federated
; Replication Master Server (default)
; binary logging is required for replication
;log-bin=mysql-bin
; binary logging format - mixed recommended
;binlog_format=mixed
; required unique id between 1 and 2^32 - 1
; defaults to 1 if master-host is not set
; but will not function as a master if omitted
server-id = 1
; Replication Slave (comment out master section to use this)
; New for MySQL 5.6 if no slave
skip-slave-start
; The InnoDB tablespace encryption feature relies on the keyring_file
; plugin for encryption key management, and the keyring_file plugin
; must be loaded prior to storage engine initialization to facilitate
; InnoDB recovery for encrypted tables. If you do not want to load the
; keyring_file plugin at server startup, specify an empty string.
early-plugin-load=""
;innodb_data_home_dir = C:/mysql/data/
innodb_data_file_path = ibdata1:12M:autoextend
;innodb_log_group_home_dir = C:/mysql/data/
;innodb_log_arch_dir = C:/mysql/data/
; You can set .._buffer_pool_size up to 50 - 80 %
; of RAM but beware of setting memory usage too high
innodb_buffer_pool_size = 4G
; Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 16M
innodb_log_buffer_size = 8M
innodb_thread_concurrency = 64
innodb_flush_log_at_trx_commit = 2
log_bin_trust_function_creators = 1;
innodb_lock_wait_timeout = 120
innodb_flush_method=normal
innodb_use_native_aio = true
innodb_flush_neighbors = 2
innodb_autoinc_lock_mode = 1
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
; Remove the next comment character if you are not familiar with SQL
;safe-updates
[isamchk]
key_buffer_size = 20M
sort_buffer_size = 20M
read_buffer_size = 2M
write_buffer_size = 2M
[myisamchk]
key_buffer_size = 256M ;20M hys
sort_buffer_size_size = 20M
read_buffer_size = 2M
write_buffer_size = 2M
[mysqlhotcopy]
interactive-timeout
[mysqld]
port = 3306
skip-log-bin
default_authentication_plugin= mysql_native_password
max_connections = 400
max_connect_errors = 100000
innodb_read_io_threads = 32
innodb_write_io_threads = 8
innodb_thread_concurrency = 64
Hardware:-
Ram: 16GB
CPU: 4 Cores 3.0 Ghz
SHOW GLOBAL STATUS:
https://pastebin.com/FVZrgnTw
SHOW ENGINE INNODB STATUS:
https://pastebin.com/Rewp84Gi
SHOW GLOBAL VARIABLES:
https://pastebin.com/3v6cM6KZ
Rate Per Second = RPS
Suggestions to consider for your my.ini [mysqld] section
It is unusual to have more than 1 [mysqld] section in the my.ini configuration
the section you have near the end of you my.ini could be moved to be just before
[mysqldump] to avoid confusion.
innodb_lru_scan_depth=100 # from 1024 to conserve 90% of CPU cycles used for function
key_buffer_size=16M # from 1G to conserve RAM - you are not using MyISAM data tables
read_rnd_buffer_size=64K # from 2M to reduce handler_read_rnd_next RPS of 1,872,921
innodb_io_capacity=900 # from 200 to more of your rotating drive IOPS capacity
You should find query completion time and CPU busy reduced with these changes.
select_scan averages 41 RPS and is caused by indexes not being available, causing delays.
For additional suggestions, view profile, Network profile for contact info, FAQ, additional tips and free downloadable Utility Scripts to assist with performance tuning.
I have found out the root cause, and post it in https://dba.stackexchange.com/questions/271785/query-performance-become-slower-after-upgrade-to-mysql-8-0-20 .
Thanks a lot for all the reply and suggestion. Appreciate it.
[Update: solved the problem at our site]
Actually I currently have had a very similar (maybe the same?) issue.
We have
Windows Server 2016, 4 CPUs, 32 GB RAM
MySQL 8 Community Edition
Java / Apache Tomcat based application on top
For 2 weeks we experienced severe application problems, with mysqld process taking 100% CPU as soon as application interaction happens -- rendering the server completely unresponsive.
The last change to the setup before this degradation was updating MySQL from 8.0.18 to 8.0.20 due to security fixes.
Query monitoring shows many occurrences of the same (simple) query
SELECT COUNT(1) FROM xxxxx;
which take 5-10 seconds (although the table only has about 3 rows, so it should rather take 5 milliseconds!).
One hypothesis was this MySQL issue: https://bugs.mysql.com/bug.php?id=99593
However the recommended workaround did not help me.
Solution for us:
Apparently there was an additional bug in MySQL Community Edition, introduced in 8.0.19 or 8.0.20.
After downgrading MySQL to 8.0.18 everything worked fine again!
Additional note:
Downgrading is not supported by MySQL!
Actually in order to provide a downgraded DB on the same machine, I...
did a backup of the application schema (with mysqldump command)
did a manual installation of MySQL 8.0.18 binaries (no installer)
created an additional MySQL instance (different data directory, different port)
imported the backup into the new instance (with mysql command)
created roles and permissions exactly like "before"
switched application config to new MySQL port
We are not able to create a stable configuration of a production server with MySQL and Tomcat Application Server. The MySQL throws very often an error:
MySQL: Out of memory (Needed 429496728 bytes)
It was a Windows Server 2012 with 64 GB of RAM. We see in the process tab, that the MySQLd commited 64 GB of RAM (that means all the available RAM in the server).
For your explaination, the Application uses completly inMemory based Tables in the MySQL Server.
See the my.ini Configuration
[client]
port=3306
[mysql]
default-character-set=utf8
[mysqld]
port=3306
basedir="F:/MySQL Server 5.6/"
datadir="F:/MySQL Server 5.6/data/"
character-set-server=utf8
default-storage-engine=MyISAM
sql-mode="STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
max_connections=200
query_cache_size=0
# table_cache=256
tmp_table_size=6G
max_heap_table_size=6G
max_tmp_tables=2048
open_files_limit=40000
thread_cache_size=8
myisam_max_sort_file_size=100G
myisam_sort_buffer_size=32M
myisam_use_mmap=0
concurrent_insert=2
key_buffer_size=2G
read_buffer_size=512M
read_rnd_buffer_size=512M
sort_buffer_size=128M
bulk_insert_buffer_size=32M
#skip-innodb
innodb_additional_mem_pool_size=15M
innodb_flush_log_at_trx_commit=1
innodb_log_buffer_size=7M
innodb_buffer_pool_size=686M
innodb_log_file_size=343M
innodb_thread_concurrency=34
slow_query_log=1
log-queries-not-using-indexes=1
long_query_time=1
log-output=FILE,TABLE
slow_query_log_file="F:/MySQL Server 5.6/localhost-slow-query.log"
event-scheduler=ON
Does anybody have a suitable solution to get this fixed?
We also use the MySQL Tuner to get informations, here the results:
What I cannot attach is the MySQL Tuner results as image (too few reputations). But look here.
Up for 39days
Data in Memory Tables 42G (Tables:281)
Data in MyISAM tables 31G (Tables 668)
total buffers: 2.3G global + 144.5M per Thread (200 max threads)
maximum possible memory usage: 30.5G (47% of installed RAM)
Key buffer size /total MyISAM indixes: 640M/7.0G
query cache disabled
joins performed without indixes: 337737
In an intranet Apache/PHP/MySQL web application that has run very well for several years now, I recently undertook an upgrade from 2.2/5.2.5/5.0.x to 2.4/5.5.11/5.6.17 respectively, all on Windows 8.
The most obvious consequence of this upgrade, sadly, has been to notice significant performance bottlenecks at most of my more intensive db queries.
In particular, I have a script that restores an entire db, consisting of over 250 tables, into an empty schema, using csv files and LOAD DATA INFILE.... Whereas this query used to get done in around one minute, it now takes over 5 minutes, and sometimes much longer.
I have also noticed slow performance during recreation of the tables (using mysqldump created script), even prior to loading data.
Apart from about 5 tables which are MyISAM, all others are InnoDB.
The hardware environment is a 'regular' HP laptop with 4GB of RAM and Intel 2.40GHz, dual core with 4 Logical Processors.
My my.ini is auto-created during installation (which I have always done using the windows executable option from MySQL website. The current configurations are:
[mysqld]
port=3306
datadir="C:/ProgramData/MySQL/MySQL Server 5.6/data/"
character-set-server=utf8
default-storage-engine=INNODB
sql-mode="STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
log-output=FILE
general-log=0
general_log_file="COMPUTERNAME.log"
slow-query-log=1
slow_query_log_file="COMPUTERNAME-slow.log"
long_query_time=10
log-error="COMPUTERNAME.err"
max_connections=100
query_cache_size=0
table_open_cache=2000
tmp_table_size=19M
thread_cache_size=9
myisam_max_sort_file_size=100G
myisam_sort_buffer_size=36M
key_buffer_size=8M
read_buffer_size=64K
read_rnd_buffer_size=256K
sort_buffer_size=256K
innodb_additional_mem_pool_size=2M
innodb_flush_log_at_trx_commit=1
innodb_log_buffer_size=4M
innodb_buffer_pool_size=512M
innodb_log_file_size=128M
innodb_thread_concurrency=9
innodb_autoextend_increment=64M
innodb_buffer_pool_instances=8
innodb_concurrency_tickets=5000
innodb_old_blocks_time=1000
innodb_open_files=300
innodb_stats_on_metadata=0
innodb_file_per_table=1
innodb_checksum_algorithm=0
back_log=70
flush_time=0
join_buffer_size=512K
max_allowed_packet=4M
max_connect_errors=100
open_files_limit=4110
query_cache_type=0
sort_buffer_size=256K
table_definition_cache=1400
binlog_row_event_max_size=8K
sync_master_info=10000
sync_relay_log=10000
sync_relay_log_info=10000
Please forgive the long question, as I reckon that these details might hold the key to an eventual solution.
Thank you.
I Have a dedicated server - Intel Xeon L5320 with 8GB of RAM and 2 x 500GB 7200RMP HDD
I need to optimize mysql to cope with a large 5Gb MyISAM table + around 25 - 30 smaller databases currently it looks like this:
key_buffer = 3G
thread_cache_size = 16
table_cache = 8192
query_cache_size = 512M
As it is the server really struggles and I get continues tmp disk full warnings could you please help me out / suggest the best my.cnf configuration for my server and or any other settings changes that would improve performance.
Thanks in advance
I recommend you use mytop and mysqltuner to analyze using mysql resources (RAM and CPU).
Too enable the option to log slow queries:
log_slow_queries = /var/log/mysql/mysql-slow.log
long_query_time = 3
And check out this post about ntpd service:
MySQL high CPU usage
Finally I leave you in a setting that I have a dedicated server for a high rate of transactions.
max_allowed_packet=16M
key_buffer_size=8M
innodb_additional_mem_pool_size=10M
innodb_buffer_pool_size=512M
join_buffer_size=40M
table_open_cache=1024
query_cache_size=40M
table_definition_cache=256
innodb_additional_mem_pool_size=10M
key_buffer_size=16M
max_allowed_packet=32M
max_connections = 300
query_cache_limit = 10M
log_slow_queries = /var/log/mysql/mysql-slow.log
long_query_time = 3
Greetings.
If /tmp is filling up, you are running some large, inefficient queries somewhere which are falling back to FILESORT. Well-written, efficient queries should typically not need this -- turn on slow query logging (if it isn't already) and check the log to see what needs optimizing.