I am using Mysql 5.2 to insert huge records ( almost a billion records)
I am using following settings .
Which parameters i can change to get BEST insert and Queries speed?
I will only create 5 tables and my queries are limitied to 10 containing multiple joins.
Following are the settings i am using:
back_log = 50
max_connections = 1024
max_connect_errors = 10
table_open_cache = 2048
max_allowed_packet = 16M
binlog_cache_size = 1M
max_heap_table_size = 64M
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
thread_cache_size = 64
thread_concurrency = 24
query_cache_size = 0
query_cache_limit = 0
ft_min_word_len = 4
default_storage_engine = MYISAM
thread_stack = 192K
transaction_isolation = READ-COMMITTED
tmp_table_size = 64M
log_bin = mysql-bin
expire_logs_days = 7
max_binlog_size = 100M
binlog_format = mixed
slow_query_log
long_query_time = 2
server_id = 11
skip_external_locking
key_buffer_size = 32M
delay_key_write = all
bulk_insert_buffer_size = 64M
myisam_sort_buffer_size = 128M
**myisam_max_sort_file_size = 10G**
myisam_repair_threads = 2
myisam_recover
innodb_additional_mem_pool_size = 16M
**innodb_buffer_pool_size = 16G**
innodb_data_file_path = ibdata1:32M;ibdata2:32M;ibdata3:32M;ibdata4:32M:autoextend
innodb_file_io_threads = 4
innodb_thread_concurrency = 24
innodb_flush_log_at_trx_commit = 0
innodb_log_buffer_size = 8M
innodb_log_file_size = 256M
innodb_log_files_in_group = 3
innodb_max_dirty_pages_pct = 90
innodb_doublewrite = 0
skip-innodb_doublewrite
innodb_lock_wait_timeout = 120
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
[myisamchk]
key_buffer_size = 512M
sort_buffer_size = 512M
read_buffer = 8M
write_buffer = 8M
[mysqlhotcopy]
interactive-timeout
Related
I have a Drupal 7 application running a large Mysql database. It's built on AWS instance (30GB RAM, 8 CPU). Below is the my.cnf copy.
Issue is that my application gets noticeably fast when I restart the mysql which makes me think that current configuration isn't optimal and once the caches get filled, they slow down the server speed.
Any experts who could point out the issue here?
[mysqld]
# moving datadir to /media/db ebs storage
datadir=/media/db/mysql
socket=/var/lib/mysql/mysql.sock
tmpdir=/dev/shm/
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
local-infile=0
log-warnings=2
skip-external-locking
bind-address = 127.0.0.1
expire_logs_days = 1
max_binlog_size = 100M
log-bin=bin.log
log-bin-index=bin-log.index
binlog_format=row
key_buffer_size = 512M
max_allowed_packet = 32M
thread_stack = 292K
myisam_sort_buffer_size = 64M
thread_cache_size = 50
myisam-recover = BACKUP
max_connections = 100
max_user_connections = 150
table_cache = 16384
table_open_cache = 10240
table_definition_cache = 4096
thread_concurrency = 8
join_buffer_size = 4M
query_cache_type = 1
query_cache_limit = 256K
query_cache_min_res_unit = 2k
query_cache_size = 200M
innodb_buffer_pool_size= 8G
tmp_table_size=64M
max_heap_table_size=64M
wait_timeout=50
interactive_timeout=50
connect_timeout=10
open-files-limit = 65535
innodb_flush_log_at_trx_commit = 0
innodb_large_prefix=true
innodb_file_format=barracuda
innodb_file_per_table=true
table_open_cache = 5000
innodb_log_buffer_size= 32M
innodb_log_file_size = 512m
read_rnd_buffer_size = 8M
transaction-isolation = READ-COMMITTED
innodb_lock_wait_timeout = 25
innodb_write_io_threads = 16
innodb_sync_spin_loops = 200
innodb_spin_wait_delay = 24
innodb_read_io_threads = 16
innodb_flush_method = O_DIRECT
innodb_buffer_pool_instances = 8
innodb_autoinc_lock_mode = 2
[mysqldump]
max_allowed_packet = 32M
[isamchk]
key_buffer = 32M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
set-variable=long_query_time=1
long_query_time=10
slow_query_log_file =/var/log/mysql/mysql-slow.log
Rate Per Second=RPS - Suggestions for your my.cnf [mysqld] section
read_rnd_buffer_size=256K # from 8M to reduce handler_read_rnd_next RPS
innodb_lru_scan_depth=100 # from 1024 to reduce CPU cycles used every SECOND
query_cache_min_res_unit=512 # from 2K to store more RESULTS in same query_cache_size
The BEST to you.
Our architecture is as follows:
Memory : 132GB
CPUs : 12
OS & Mysql : 64-bit
Disk Space : 2.0 TB
DB Engine : MyISAM
i only have 1 table innodb (around 400mb)
all the rest are myISAM (around 2dg)
my server load has started going through the roof and its down to API hits my sites is getting from its users (when i turn of api if goes right down)
my sql .conf is as follows
[mysqld]
innodb_file_per_table=1
bind-address=127.0.0.1
####### Performance Tweaks #######
##thread_cache_size = 64 # Default 0
##table_open_cache = 1024 # Default 64
##table_definition_cache = 512 # Default 256
##max_heap_table_size = 96M # Default 16M
##tmp_table_size = 96M # Default 16M (can't be bigger than max_heap_table_size)
##key_buffer_size = 96M # Default 8M
##sort_buffer_size = 256K # Default 2M
##join_buffer_size = 256K # Default 128K
##read_buffer_size = 256K # Default 128K.
### read_rnd_buffer_size = 512K # Default 256K.
##concurrent_insert = 2 # Deafult 1
##query_cache_limit = 512K # Default 1
##query_cache_size = 96M # Default 0
######### End Performance Tweaks ##########
#### ADDDED ALL BEWLO LINES ON JUNE,06,2014
# Memory usage
skip-external-locking
max_connections = 1000
max_user_connections = 3056
max_connect_errors = 999999
innodb_file_per_table = ON
innodb_buffer_pool_size = 8G
key_buffer = 1024M
#key_buffer_size = 512M
key_buffer_size = 8G
max_allowed_packet=268435456
group_concat_max_len = 12192
table_open_cache=2048
tmp_table_size = 512M
max_heap_table_size = 512M
sort_buffer_size = 32M
read_buffer_size = 32M
read_rnd_buffer_size = 20M
bulk_insert_buffer_size = 24M
myisam_sort_buffer_size = 200M
query_prealloc_size = 262144
query_alloc_block_size = 65535
transaction_alloc_block_size = 8192
transaction_prealloc_size = 4096
max_write_lock_count = 8
query_cache_limit = 5M
thread_cache_size = 382
query_cache_size= 256M
# Try number of CPU's*2 for thread_concurrency (eHound has 4 CPU's)
thread_concurrency = 24
# Disable Federated by default
skip-federated
open_files_limit=10000
default-storage-engine=MyISAM
[mysqld_safe]
how variablesog-error="/var/log/mysqld.log"
pid-file="/var/run/mysqld/mysqld.pid"
[mysqldump]
quick
max_allowed_packet = 64M
[mysql]
no-auto-rehash
[isamchk]
#key_buffer = 512M
key_buffer = 4G
sort_buffer_size = 512M
read_buffer = 256M
write_buffer = 256M
[myisamchk]
#key_buffer = 512M
key_buffer = 4G
sort_buffer = 512M
max_allowed_packet = 14M
read_buffer = 8256M
write_buffer = 256M
sort_buffer_size = 1M
join_buffer_size = 1M
thread_stack = 192K
[mysqlhotcopy]
interactive-timeout
Any suggestions? Thanks folks.
After comments ove made these changes
[mysqld]
innodb_file_per_table=1
bind-address=127.0.0.1
####### Performance Tweaks #######
##thread_cache_size = 64 # Default 0
##table_open_cache = 1024 # Default 64
##table_definition_cache = 512 # Default 256
##max_heap_table_size = 96M # Default 16M
##tmp_table_size = 96M # Default 16M (can't be bigger than max_heap_table_size)
##key_buffer_size = 96M # Default 8M
##sort_buffer_size = 256K # Default 2M
##join_buffer_size = 256K # Default 128K
##read_buffer_size = 256K # Default 128K.
### read_rnd_buffer_size = 512K # Default 256K.
##concurrent_insert = 2 # Deafult 1
##query_cache_limit = 512K # Default 1
##query_cache_size = 96M # Default 0
######### End Performance Tweaks ##########
#### ADDDED ALL BEWLO LINES ON JUNE,06,2014
# Memory usage
skip-external-locking
max_connections = 1000
max_user_connections = 3056
max_connect_errors = 999999
innodb_file_per_table = ON
innodb_buffer_pool_size = 2G
#key_buffer = 1024M
#key_buffer_size = 512M
key_buffer_size = 45G
max_allowed_packet=268435456
group_concat_max_len = 12192
table_open_cache=2048
tmp_table_size = 512M
max_heap_table_size = 512M
sort_buffer_size = 32M
read_buffer_size = 32M
read_rnd_buffer_size = 20M
bulk_insert_buffer_size = 24M
myisam_sort_buffer_size = 200M
query_prealloc_size = 262144
query_alloc_block_size = 65535
transaction_alloc_block_size = 8192
transaction_prealloc_size = 4096
max_write_lock_count = 8
query_cache_limit = 5M
thread_cache_size = 382
query_cache_size= 256M
# Try number of CPU's*2 for thread_concurrency (eHound has 4 CPU's)
thread_concurrency = 24
# Disable Federated by default
skip-federated
open_files_limit=10000
default-storage-engine=MyISAM
[mysqld_safe]
how variablesog-error="/var/log/mysqld.log"
pid-file="/var/run/mysqld/mysqld.pid"
[mysqldump]
quick
max_allowed_packet = 64M
[mysql]
no-auto-rehash
[isamchk]
#key_buffer = 512M
key_buffer = 4G
sort_buffer_size = 512M
read_buffer = 256M
write_buffer = 256M
[myisamchk]
#key_buffer = 512M
key_buffer = 4G
sort_buffer = 512M
max_allowed_packet = 14M
read_buffer = 8256M
write_buffer = 256M
sort_buffer_size = 4M
join_buffer_size = 4M
thread_stack = 192K
[mysqlhotcopy]
interactive-timeout
thanks again
i have enable my sql log and i get loads of these per second, which i can see if my users using my API
# Thread_id: 4273 Schema: DB_name QC_hit: No
# Query_time: 0.402709 Lock_time: 0.000058 Rows_sent: 1 Rows_examined: 141611
SET timestamp=1431343679;
select users.*, userroles.apirequests, userroles.downloadrequests, NOW() as now from users inner join userroles on userroles.ID = users.role where users.rsstoken = 'APIKEYHERE';
I just moved server and am working on the my.cnf to optimize it for my pc, if someone can give a review on it, as the site is loading very slow on initial connect. In Pingdom, it shows it as "WAIT", and it starts loading site after about 8 seconds.
Here's my My.cnf
[mysqld]
datadir="/var/lib/mysql"
socket="/var/lib/mysql/mysql.sock"
user=mysql
open_files_limit=26672
innodb_file_per_table=1
symbolic-links=0
default-storage-engine=MyISAM
tmp_table_size = 1G
key_buffer_size = 2G
max_heap_table_size = 2G
query_cache_size = 2G
max_connections = 500
thread_cache_size = 5G
table_definition_cache = 2G
max_allowed_packet=64M
table_cache=2G
max_connect_errors = 10
binlog_cache_size = 2G
read_buffer_size = 2G
read_rnd_buffer_size = 5G
sort_buffer_size = 2G
bulk_insert_buffer_size = 3G
join_buffer_size = 3G
myisam_sort_buffer_size = 4G
thread_cache_size = 2G
myisam_max_sort_file_size = 4G
thread_concurrency = 8
query_cache_size = 1G
query_cache_limit = 10G
innodb_additional_mem_pool_size = 1G
innodb_buffer_pool_size = 2G
innodb_data_file_path = ibdata1:10M:autoextend
innodb_file_io_threads = 4
innodb_thread_concurrency = 16
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 8M
innodb_log_file_size = 1G
innodb_log_files_in_group = 3
innodb_max_dirty_pages_pct = 90
innodb_lock_wait_timeout = 120
myisam_sort_buffer_size=2G
[mysqld_safe]
log="/var/log/mysqld.log"
log-error="/var/log/mysqld.error.log"
[myisamchk]
sort_buffer_size = 5M
read_buffer = 50M
write_buffer = 50M
All advice/ criticism are welcomed, i am an amatuer in this and have made this by reading up blogs and other resources.
I recommend you to use Percona's Configuration Wizard for MySQL:
https://tools.percona.com/wizard
And to check if you really need:
1) MyISAM engine;
2) query cache;
[mysqld]
open_files_limit=10000
skip-name-resolve
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
max_heap_table_size = 2GB
query_cache_limit = 256M
key_buffer_size = 6GB
innodb_table_locks = 0
innodb_buffer_pool_size = 100M
join_buffer_size = 32M
max_connections = 400
max_user_connections = 1400
#skip-bdb
key_buffer = 48M
# max_allowed_packet = 68M
max_allowed_packet = 256M
tmp_table_size = 3GB
table_cache = 800
sort_buffer_size = 1M
net_buffer_length = 8K
read_buffer_size = 1M
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
server-id = 1
wait_timeout = 28800
query_cache_size = 50M
thread_cache_size = 1024
long_query_time = 4
We have this config on 15GB RAM VPS server, which is used for MySQL only. The problem is that the server is using only 3GB of RAM during the heaviest traffic.
Can anyone suggest what should be adjusted in config to use all available RAM?
mysqltuner.pl says this:
[!!] Maximum possible memory usage: 15.0G (103% of installed RAM)
I am currently running on a cloud server, WITH ukfast
CPU GenuineIntel, Intel(R) Xeon(R) CPU E5-2620 0 # 2.00GHz
Version Parallels Plesk Panel v11.0.9_build110120608.16 os_CentOS 6
OS CentOS 6.3 (Final) 64-bit
RAM: 6GB
I have 12,000 SKU's but never get over 30 vistors on the site at one time and I am trying to optimise my.cnf file:
This is what I currently have:
log_slow_queries = /var/log/mysql-slow.log
long_query_time = 5
key_buffer = 32M
max_allowed_packet = 16M
table_cache = 1024
sort_buffer_size = 48M
read_buffer_size = 92M
read_rnd_buffer_size = 100M
myisam_sort_buffer_size = 100M
myisam_max_sort_file_size = 1G
myisam_max_extra_sort_file_size = 1G
myisam_repair_threads = 1
thread_cache_size = 32
query_cache_type = 1
query_cache_size = 200M
query_cache_limit = 2M
max_connections=50
wait_timeout=120
tmp_table_size = 256M
max_heap_table_size = 64M
innodb_buffer_pool_size = 650M
innodb_additional_mem_pool_size = 24M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid}
is there anything I am missing or anything I should increase or decrease?
any thoughts / advice would be great.
I have face this problem before.
Please try this
key_buffer = 32M
max_allowed_packet = 16M
table_cache = 1024
sort_buffer_size = 48M
read_buffer_size = 92M
read_rnd_buffer_size = 104M
myisam_sort_buffer_size = 100M
myisam_max_sort_file_size = 1024m
myisam_max_extra_sort_file_size = 1024m
myisam_repair_threads = 1
thread_cache_size = 32
query_cache_type = 1
query_cache_size = 256M
query_cache_limit = 2M
max_connections=50
wait_timeout=60
tmp_table_size = 256M
max_heap_table_size = 64M
innodb_buffer_pool_size = 650M
innodb_additional_mem_pool_size = 24M
or you have to change or upgrade your hosting to VPS server.
Houp this help.
^ ^