We have a MySQL Setup running Percona 5.7 (5.7.12-5-1.trusty) (36 cores, 768 GB Memory, 3.5 TB Raid-10 of 8 SSDs).
DB size is around 6 TB, some of the tables are larger then 1TB with billion of records. On average, we have around 12-18.0000 queries per second (about 70% of them are SELECTs).
Also we are using ZFS with lz4 compression and 16k recordsize.
Mysql Master sometimes stop to respond to queries in time and gathers alot of connections and doesnt respond to clients anymore for a minute or so.
In slow query logs we see that all slow queries group in the same 0.1 of second.
For example:
SELECT time, COUNT(*) FROM slow_query_log WHERE time BETWEEN '2016-08-26 13:16:00' AND '2016-08-26 13:19:00'
...
'2016-08-26 13:16:46', '1'
'2016-08-26 13:16:55', '1'
'2016-08-26 13:17:00', '1'
'2016-08-26 13:17:09', '1'
'2016-08-26 13:17:21', '1'
'2016-08-26 13:18:08', '340'
'2016-08-26 13:18:10', '1'
'2016-08-26 13:18:29', '1'
'2016-08-26 13:18:50', '1'
...
We see the same picture 2 times per hour. What is even more interesting, that all slow queries in that moment have pretty the same query time 15-25 sec,
even queries like select ##session.tx_read_only; #17 sec or SHOW GLOBAL STATUS; #21 sec. First, we thought that we might have some heave queries,
but then we've found select ##session.tx_read_only; slow queries, which should be executed instantly, and became totally confused.
We see no deadlocks, stuck queries, long running queries, system lock or anything when this happens, nothing in the logfile aswell.
Other configuration parts:
innodb_adaptive_hash_index = OFF
innodb_doublewrite = OFF
skip-innodb_doublewrite
innodb-log-files-in-group = 2
innodb-log-file-size = 512M
innodb-flush-log-at-trx-commit = 2
innodb-file-per-table = 1
innodb-buffer-pool-size = 400G
innodb_buffer_pool_instances = 32
innodb_write_io_threads = 64
innodb_read_io_threads = 64
innodb_flush_neighbors = 0
innodb_adaptive_hash_index = OFF
innodb_use_native_aio = 0
performance_schema=off
the buffer pool size is on 400gb for a reason. Anytime we raise this as example to 450, or even 650 we get "complete" db freezes for multiple minutes at randomintervals. Also if we accordingly set the correct pool instance values for the higher buffer pool. adaptive_hash_index is off because otherwise
truncate table will completely break the db with freezes aswell but that is a well known bug with adaptive hash index and large buffer pools. native_aio is off
because of ZFS. Performance Schema is off for performance reasons aswell.
iops are ok, no zfs arc issues or anything. cpu utilisation is almost never more then 40% of the available resources. Enough memory free etc.
Anyone got a idea on how to debug this problem? And yes we know its not ideal to store this much data in mysql ;)
Related
upon increasing total number of pool connection from application to MySQL 40000 getting this error ER_CANT_CREATE_THREAD.
MySQL conf file looks like this
innodb_buffer_pool_size=100G # from 128M to use half of your RAM for data/index storage
innodb_buffer_pool_instances=32 # from 1 to reduce mutex contention
innodb_lru_scan_depth=1024 # from 1024 to conserve 90% of CPU cycles used for function
max_connections=65536
max_prepared_stmt_count=204800
max_connect_errors=100000
thread_cache_size=8192
innodb_log_buffer_size = 256M
query_cache_size = 0
innodb_read_io_threads = 64
innodb_write_io_threads = 64
innodb_thread_concurrency = 0
innodb_io_capacity = 2000
sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
max_connections=65536 is unreasonably, perhaps dangerously, high. Lower it to 100.
If 90000 connections are active at once, MySQL will meltdown. It is better to avoid too many connections than to let them stumble over each other. (Consider what would happen if you let 10000 people into a grocery store at the same time. Traffic would be so clogged up that people might take an hour to buy just one item.)
If your IoT action is a quick connect-insert-disconnect, that should take very few milliseconds.
If the connections came in very smoothly and take 10ms elapsed each, they could handle 90K per minute with only max_connections=15.
Benchmarks have shown that MySQL gets bottlenecked at not much more than the number of CPU cores that you have.
So, by limiting the number of current active connections to max_connections=100 should be a safe compromise.
Set back_log to a higher number, maybe 1000. (I do not have a feel for what a good number is.) The idea is that delaying the connection is better than letting it in, only to be stalled.
I am confident that MySQL can handle 90K IoT inserts per minute, but only if you take the advice here.
You mentioned "connection pooling". The 100 and 1000 should be moved back into the pooling layer. Or get rid of the layer. (It is hard to say which would be better.)
Please lower innodb_read_io_threads and innodb_write_io_threads to 4. Those variables should be increased for docker environments.
If you want more than 4 threads each, make sure there is enough RAM allocated.
I am total newbie to database game.
& I am using mysql 8.0 RDS 20GB free tier.
I don't have much data on it.(just 300 Mb)
(wordpress running on ec2 is linked with this rds)
But I always get a mail that you have exceeded 85% of storage limit.(approx 17Gb)
Also Under monitoring tab in rds panel it shows:- Free Storage Space (MB) = 19.330Gb (Constantly without any drop)
I have turned off : general log, slow logs, binlog,
i have done these parameter changes:
binlog_expire_logs_seconds = 86400
general_log = 0
innodb_autoextend_increment = 1
innodb_buffer_pool_size = 5000000000
log_bin_trust_function_creators = 0
log_output = NONE
max_binlog_cache_size = 5368709120
max_connections = 1000
slow_query_log = 0
innodb_file_per_table = 1
SELECT name, count FROM information_schema.innodb_metrics ORDER BY count;
Highest counts seen on:
buffer_data_written = 6936939008
buffer_pool_size = 402653184
os_log_bytes_written = 313484800
buffer_pool_bytes_data = 204668928
Can someone please help me whats consuming these storage. Is this something to do with INNODB?
Ouch! innodb_buffer_pool_size = 5000000000 inside 1GB of RAM! This leads to swapping, which uses disk space and slows down MySQL terribly.
Don't set that about, say, 250M. Even that may be too high. Do you have other things running in the db.t2.micro?
max_connections = 1000 -- also too high. Set it to 20
If you want to change anything else, get some advice first. The general advice is: 1gb of RAM is very limited; raising tunables can lead to bad results.
Summary:
I haven't yet been able to get MySQL to use more than 1 core for a select statement and it doesn't get above 10 or 15 GB of RAM.
The machine:
I have a dedicated Database server running MariaDB using MySQL 5.6. The machine is strong with 48 cores and 192GB of RAM.
The data:
I have about 250 million rows in one large table (also several other tables ranging from 5-100 million rows). I have been doing a lot of reading from the tables, sometimes inserting into a new table to denormalize the data a bit. I am not setting this system up as a transactional system, rather, it will be used more similarly to a data warehouse with few connections.
The problem:
When I look at my server's stats, it looks like CPU is at around 70% for one core with a select query running, and memory is at about 5-8%. There is no IO waiting, so I am convinced that I have a problem with MySQL memory allocation. After searching on how to increase the usage of memory in MySQL I have noticed that the config file may be the way to increase memory usage.
The solution I have tried based on my online searching:
I have changed the tables to MyISAM engine and added many indexes. This has helped performance, but querying these tables is still incredibly slow. The write speed using load data infile is very fast, however, running a mildly complex select query takes hours or even days.
I have also tried adjusting the following configurations:
key-buffer-size = 64G
read_buffer_size = 1M
join_buffer_size = 4294967295
read_rnd_buffer_size = 2M
key_cache_age_threshold = 400
key_cache_block_size = 800
myisam_data_pointer_size = 7
preload_buffer_size = 2M
sort_buffer_size = 2M
myisam_sort_buffer_size = 10G
bulk_insert_buffer_size = 2M
myisam_repair_threads = 8
myisam_max_sort_file_size = 30G
max-allowed-packet = 256M
tmp-table-size = 32M
max-heap-table-size = 32M
query-cache-type = 0
query-cache-size = 0
max-connections = 500
thread-cache-size = 150
open-files-limit = 65535
table-definition-cache = 1024
table-open-cache = 2048
These config changes have slightly improved the amount of memory being used, but I would like to be able to use 80% of memory or so... or as much as possible to get maximum performance. Any ideas on how to increase the memory allocation to MySQL?
As you have already no IO waiting you are using a good amount of memory. Your buffers also seem quite big. So I would doubt that you can have significant CPU savings with using additional memory. You are limited by the CPU power of a single core.
Two strategies could help:
Use EXPLAIN or query analyzers to find out if you can optimize your queries to save CPU time. Adding missing indexes could help a lot. Sometimes you also might need combined indexes.
Evaluate an alternative storage engine (or even database) that is better suited for analytical queries and can use all of your cores. MariaDB supports InfiniDB but there are also other storage engines and databases available like Infobright, MonetDB.
Use show global variables like "%thread%" and you may get some clues on enabling thread concurrency options.
read_rnd_buffer_size at 2M tested at 16384 with your data may produce significant reduction in time required to complete your query.
I have spent several weeks crunching on this to no avail, so I'm hopeful you may be able to help. Generally, I have an update query that takes forever to run (I've given up after 12 hours). To knock the obvious out of the way, I have an index on the columns. Also, I am totally self-taught on MYSQL, so I may need additional clarification on data / processes etc. This DB is for my personal use, offline. Said another way... this is not my day job. While I enjoy MYSQL, I am not a super-user.
First, my system specs...
Laptop Samsung QX410
Windows 7, 64 bit
Intel i5, M 480 # 2.67 GHz
RAM: 8 GB (7.79 available)
WAMP 2.5 with MYSQL v5.6.17
Tables are INNODB
MYSQL set up:
' The MySQL server
[wampmysqld]
port = 3306
socket = /tmp/mysql.sock
key_buffer_size = 512M
max_allowed_packet = 32M
sort_buffer_size = 512K
net_buffer_length = 32K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
basedir=c:/wamp/bin/mysql/mysql5.6.17
log-error=c:/wamp/logs/mysql.log
datadir=c:/wamp/bin/mysql/mysql5.6.17/data
' Uncomment the following if you are using InnoDB tables
innodb_data_home_dir = C:\mysql\data/
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = C:\mysql\data/
innodb_log_arch_dir = C:\mysql\data/
' You can set .._buffer_pool_size up to 50 - 80 %
' of RAM but beware of setting memory usage too high
innodb_buffer_pool_size = 4000M
innodb_additional_mem_pool_size = 32M
' Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 512M
innodb_log_buffer_size = 256M
innodb_flush_log_at_trx_commit = 0
innodb_lock_wait_timeout = 50
Issue in more detail:
I have two tables Trade_List and Cusip_Table and am trying to populate one column in Trade_List (I need to pre-populate this value, since many queries will be run against it).
Trade_List has 11 columns, two of which are relevant.
CUSIP (varchar 45) - generally this is a 9 digit alpha-numeric number.
TICKER (varchar 45) - generally this is 10 letters or less. I want to populate this.
This table has roughly 10 million rows.
I have removed all indices from this table except one on CUSIP.
Cusip_Table has 5 columns, two of which are relevant.
CUSIP (varchar 45) - generally this is a 9 digit alpha-numeric number.
TICKER (varchar 45) - generally this is 10 letters or less. This is already populated.
This table has roughly 70,000 rows.
I have an index 'CTDuplicateCheck' on (Cusip, Ticker).
When I run...
Select A.cusip, B.ticker
From Trade_list A, Cusip_table B
Where A.cusip = B.cusip;
... MYSQL indicates that the query takes about 13 seconds, but in reality it seems to take about a minute, so I ran profiling on it...
starting 0.000093
checking permissions 0.000006
checking permissions 0.000005
Opening tables 0.000041
init 0.000037
System lock 0.000013
optimizing 0.000015
statistics 0.000041
preparing 0.000030
executing 0.000002
Sending data 10.982211
end 0.000014
query end 0.000010
closing tables 0.000018
freeing items 0.000070
logging slow query 0.000004
cleaning up 0.000019
I don't know what any of this means, but 10 seconds for sending data seems reasonable (the return set is ~9M rows.
Just for kicks, and to make sure the index is working, I ran an 'explain' (shown below). I think this says that my index is working correctly.
1 SIMPLE B index CTDuplicateCheck CTDuplicateCheck 96 53010 Using where; Using index
1 SIMPLE A ref TL1Cusip TL1Cusip 48 13f_master_data.B.CUSIP 154 Using index
**NOTE: 13f_Master_Data is the name of the database.
At any rate, when I run the same query, but change it to an update, everything falls apart and it will not complete. I would expect things to run a bit slower, but 12 hours +? I just can't imagine that this is normal for an update query that touches 9M rows. The original INSERT took less than an hour, and the select takes less than a minute. Code for the update is below...
Update Trade_list A, Cusip_table B
Set A.ticker = B.ticker
Where A.cusip = B.cusip;
Stuff I have tried:
Removed almost all index's from Trade_List. I left one in on CUSIP.
Upgraded RAM from 4 GB to 8 GB. This did nothing. Upon further investigation, my CPU and RAM are not limiting factors. CPU generally sits around 30%, RAM never gets above 5GB utilized. This leads me to believe that the issue is I/O. Is it possible MYSQL is doing a full table-scan? Why would it not utilize the index?
Changed all types of memory allocations per http://www.percona.com/blog/2013/09/20/innodb-performance-optimization-basics-updated/ and https://rtcamp.com/tutorials/mysql/mysqltuner/ and http://www.percona.com/blog/2006/09/29/what-to-tune-in-mysql-server-after-installation/. As far as I can tell, this did nothing. Again, I don't think the limiting factor is memory available. Also, I have no doubt that my memory allocations (shown above) are completely screwed up. I had no idea what I was doing and changed things all over the place. That said, I don't think the memory changes made anything any worse.
Upgraded MYSQL and Wamp versions (did nothing).
Read and learned a lot about index's. Candidly, I know very little about MYSQL, and am totally self-taught. I have learned a lot about memory on this foray, but need someone to step in and tell me where I have totally derailed. This database is for my own offline analysis. I am the only user.
I am happy to provide additional information that may help to analyze the issue. I'm at a total loss on this. The only thing I can come up with is that the system is doing full scans row by row... for every look-up in the update. Though, this could be completely false.
Your thoughts are much appreciated.
PM
We are constructing, for every day, mappings from tweet user id to the list of tweet ids of tweets made by that user. The storage engine we are using is Percona xtraDB "5.1.63-rel13.4 Percona Server (GPL), 13.4, Revision 443"
We are unsatisfied with the maximal throughput in terms of row inserts per second. Our maximal throughput to process tweets with xtraDB is around 6000 ~ 8000 tweets per second. (for example, if we had to rebuild data for some day from scratch, we'll have to wait for almost a day)
For the most part we are able to do this realtime enough with the full amount of twitter data (which is roughly 4000 ~ 5000 tweets per second).
We have narrowed down the bottleneck of our application to MySQL InnoDB insert. In our application, we read the feed from the disk and parse it with jackson (which happens at about 30,000 tweets per second). Our application then proceeds in batches of tweets. For the set of authors that generates these tweets, we partitioning them into 8 groups (simple partitioning with user id modulo 8). A table is allocated for each group and 1 thread is allocated to write the data to that table. Everyday there are roughly 26 million unique users that generates these tweets, and therefore each table have roughly 4 millions rows.
For a group of users, we only use one transaction for read and update. The group size is a runtime tunable. We have tried various sizes from 8 ~ 64000 , and we have determined 256 to be a good batch size.
the schema of our table is
CREATE TABLE `2012_07_12_g0` ( `userid` bigint(20) NOT NULL, `tweetId` longblob, PRIMARY KEY (`userid`)) ENGINE=InnoDB DEFAULT CHARSET=utf8
where tweetId is the compressed list of tweet ids long integers, compressed with Google snappy
Each thread uses
Select userid,tweetId from <tablename> where userid IN (....)
to resolve the userids to readback the data, and the threads use
INSERT INTO <tablename> (userid,tweetId) VALUES (...) ON DUPLICATE KEY UPDATE tweetId=VALUES(tweetId)
to update the rows with new tweetids.
We have tried setting various XtraDB parameters
innodb_log_buffer_size = 4M
innodb_flush_log_at_trx_commit = 2
innodb_max_dirty_pages_pct = 80
innodb_flush_method = O_DIRECT
innodb_doublewrite = 0
innodb_use_purge_thread = 1
innodb_thread_concurrency = 32
innodb_write_io_threads = 8
innodb_read_io_threads = 8
#innodb_io_capacity = 20000
#innodb_adaptive_flushing = 1
#innodb_flush_neighbor_pages= 0"
The table size for each day is roughly 8G for all tables, and InnoDB is given 24GB to work with.
We are using:
6-disk (crucial m4 SSD, 512 GB, 000F firmware) software RAID5.
Mysql innodb data, table space on the SSD partition
ext4 mount with noatime,nodiratime,commit=60
centos 6.2
sun jdk 1.6.30
Any tips for making our insert go faster would be greatly appreciated, thanks.
InnoDB is given 24GB
Do you mean this is the innodb_buffer_pool_size? You didn't say how much memory you have nor what CPUs you are using. If so then you should probably be using a larger innodb_log_buffer_size. What's your setting for innodb_log_file_size? It should probably be in the region of 96Mb.
innodb_write_io_threads = 8
ISTR that ext3 has some concurrency problems with multiple writers - but I don't know about ext4
Have you tried changing innodb_flush_method?
Which I/O scheduler are you using (in the absence of a smart disk controller, usually deadline is fastest, sometimes CFQ)?
Switching off the ext4 barriers will help with throughput - its a bit more risky - make sure you've got checksums enabled in JBD2. Similarly setting innodb_flush_log_at_trx_commit=0 should give a significant increase but more risky.
Since you're obviously not bothered about maintaining your data in a relational format, then you might consider using a noSQL database.
My initial suggestions would be:
As you don't have RAID card with memory you may want to comment out innodb_flush_method = O_DIRECT line to let system cache writes
as you disabled double write buffer you could also set innodb_flush_log_at_trx_commit to 0 which would be faster than 2
set innodb_log_buffer_size to cover at least one second of writes (approx 12Mb for 30K tweets)
in case you use binary logs - make sure you have sync_binlog = 0
On the hardware side I would strongly suggest to try RAID card with at least 256Mb RAM and battery unit (BBU) to improve write speed. There are RAID cards on the market that supports SSD.
Hope this helps. Please let me know how it goes.