I need to improve I/O performance for my database. I'm using the "2xlarge" HW described below & considering upgrading to the "4xlarge" HW (http://aws.amazon.com/ec2/instance-types/). Thanks for the help!
Details:
CPU usage is fine (usually under 30%), uptime load averages anywhere from 0.5 to 2.0 (but I believe I'm supposed to divide that by the number of CPU's) so that looks okay as well. However, the I/O is bad: iostat show favorable service times, but the time spent in queue (I suppose this means waiting to access the disk) is far too high. I've configured MySQL to flush to disk every 1sec instead of every write, which helps, but not enough. Profiling shows there are a handful of tables that are the culprits for most of the load (both read && write operations). Queries are already indexed & optimized, but not partitioned. Average MySQL states are: Sending Data # 45%, statistics # 20%, Updating # 15%, Sorting result # 8%.
Questions:
How much performance will I get by upgrading HW?
Same question, but if I partition the high-load tables?
Machines:
m2.2xlarge
64-bit
4 vCPU
13 ECU
34.2 Gb Mem
EBS-Optimized
Network Performance: "Moderate"
m2.4xlarge
64-bit
6 vCPU
26 ECU
68.4 Gb Mem
EBS-Optimized
Network Performance: "High"
In my experience, the biggest boost in MySQL performance comes from IO. You have alot of RAM. Try setting up a ram drive and point the tmpdir to it.
I have several MySQL servers that are very busy. My settings are below - maybe this can help you tweak your settings.
My Setup is:
-Dual 2.66 CPU 8 core box with a 6-drive Raid-1E array - 1.3TB.
-innodblogs on a separate SSD drives.
-tmpdir is on a 2GB tempfs partition.
-32GB of ram
InnoDB settings:
innodb_thread_concurrency=16
innodb_buffer_pool_size = 22G
innodb_additional_mem_pool_size = 20M
innodb_log_file_size = 400M
innodb_log_files_in_group=8
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 2 (This is a slave machine - 1 is not required fo my purposes)
innodb_flush_method=O_DIRECT
Current Queries per second avg: 5185.650
I am using Percona Server, which is quite a bit faster that other MySQLs from my testing.
Related
upon increasing total number of pool connection from application to MySQL 40000 getting this error ER_CANT_CREATE_THREAD.
MySQL conf file looks like this
innodb_buffer_pool_size=100G # from 128M to use half of your RAM for data/index storage
innodb_buffer_pool_instances=32 # from 1 to reduce mutex contention
innodb_lru_scan_depth=1024 # from 1024 to conserve 90% of CPU cycles used for function
max_connections=65536
max_prepared_stmt_count=204800
max_connect_errors=100000
thread_cache_size=8192
innodb_log_buffer_size = 256M
query_cache_size = 0
innodb_read_io_threads = 64
innodb_write_io_threads = 64
innodb_thread_concurrency = 0
innodb_io_capacity = 2000
sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
max_connections=65536 is unreasonably, perhaps dangerously, high. Lower it to 100.
If 90000 connections are active at once, MySQL will meltdown. It is better to avoid too many connections than to let them stumble over each other. (Consider what would happen if you let 10000 people into a grocery store at the same time. Traffic would be so clogged up that people might take an hour to buy just one item.)
If your IoT action is a quick connect-insert-disconnect, that should take very few milliseconds.
If the connections came in very smoothly and take 10ms elapsed each, they could handle 90K per minute with only max_connections=15.
Benchmarks have shown that MySQL gets bottlenecked at not much more than the number of CPU cores that you have.
So, by limiting the number of current active connections to max_connections=100 should be a safe compromise.
Set back_log to a higher number, maybe 1000. (I do not have a feel for what a good number is.) The idea is that delaying the connection is better than letting it in, only to be stalled.
I am confident that MySQL can handle 90K IoT inserts per minute, but only if you take the advice here.
You mentioned "connection pooling". The 100 and 1000 should be moved back into the pooling layer. Or get rid of the layer. (It is hard to say which would be better.)
Please lower innodb_read_io_threads and innodb_write_io_threads to 4. Those variables should be increased for docker environments.
If you want more than 4 threads each, make sure there is enough RAM allocated.
I am trying to benchmark MySQL with HammerDB using their TPROC-C benchmark. This is part of a research I'm doing where I am benchmarking MySQL and PostgreSQL. After running through all benchmarks with PostgreSQL, I started to benchmark MySQL by was surprised at the difference!
According to HammerDB's site, NOPM should be comparable between RDBMSs, but the numbers were so low that got me thinking about my methodology was wrong or my tuning parameters were hurting performance.
My intention was to run their benchmark with 100 warehouses and varying number of virtual users.
With PostgreSQL I go around 14000 NOPM with a single virtual user, but with MySQL, I get around 3800.
I AM NOT TRYING TO START A WAR
What would be really helpful is if someone can point me in the right direction.
I am running both MySQL and PostgreSQL on Docker, with the latest images.
Both images were limited to 12GB of RAM and HammerDB is running on the host machine.
MySQL is running on InnoDB.
These are the settings I changed from MySQL:
innodb_buffer_pool_size = 8G
innodb_buffer_chunk_size = 1G
innodb_buffer_pool_instances = 8
innodb_log_file_size = 2G
innodb_stats_on_metadata = OFF
innodb_file_per_table = ON
innodb_write_io_threads = 24
innodb_read_io_threads = 24
innodb_thread_concurrency = 0
innodb_io_capacity = 20000
key_buffer_size = 128M
thread_stack = 256K
thread_cache_size = 16
max_heap_table_size = 256M
I am a complete MySQL noob, but I want to get these tests right, that's why I've come here
I am running these on a laptop though, which is less than ideal. Here are the configs:
i7-1165G7 - 4 cores 8 threads HT 16GB RAM M.2 nvme SSD
As for HammerDB options:
Timed Driver Script (same as PG)
1000000 total txs per user (same as PG)
Prepared statements
1m ramp-up time (same as PG)
30m test time (same as PG)
Use all warehouses (same as PG)
Time profile (same as PG)
I'll be more than happy to answer any questions!
I'm not a native speaker, so forgive me for any mistakes.
The advice on the HammerDB website that the NOPM (New Orders per Minute) value is comparable between databases means that NOPM can be used to compare between different databases and systems rather than meaning all the databases should achieve the same throughput. For example, the commercial databases can achieve higher performance and scalability than the open source ones, however this is to be expected as the commercial databases have been tuned over many decades to perform well in the TPC-C tests which the HammerDB OLTP test is derived from. Additionally, databases have been designed to run multiple sessions on multiple CPUs concurrently, and therefore little can be determined about the capability of an individual database by a single user test. On an up-to-date 2 socket server in 2021 expect both MySQL and PostgreSQL to perform at peak in the range of 1,000,000 to 3,000,000 NOPM with multiple sessions depending on the hardware and software configuration. Versions MySQL 8.0.20 and PostgreSQL 13.0 upwards are recommended for higher performance.
The HammerDB discussions forum on github are the best place for specific HammerDB tuning advice.
Summary:
I haven't yet been able to get MySQL to use more than 1 core for a select statement and it doesn't get above 10 or 15 GB of RAM.
The machine:
I have a dedicated Database server running MariaDB using MySQL 5.6. The machine is strong with 48 cores and 192GB of RAM.
The data:
I have about 250 million rows in one large table (also several other tables ranging from 5-100 million rows). I have been doing a lot of reading from the tables, sometimes inserting into a new table to denormalize the data a bit. I am not setting this system up as a transactional system, rather, it will be used more similarly to a data warehouse with few connections.
The problem:
When I look at my server's stats, it looks like CPU is at around 70% for one core with a select query running, and memory is at about 5-8%. There is no IO waiting, so I am convinced that I have a problem with MySQL memory allocation. After searching on how to increase the usage of memory in MySQL I have noticed that the config file may be the way to increase memory usage.
The solution I have tried based on my online searching:
I have changed the tables to MyISAM engine and added many indexes. This has helped performance, but querying these tables is still incredibly slow. The write speed using load data infile is very fast, however, running a mildly complex select query takes hours or even days.
I have also tried adjusting the following configurations:
key-buffer-size = 64G
read_buffer_size = 1M
join_buffer_size = 4294967295
read_rnd_buffer_size = 2M
key_cache_age_threshold = 400
key_cache_block_size = 800
myisam_data_pointer_size = 7
preload_buffer_size = 2M
sort_buffer_size = 2M
myisam_sort_buffer_size = 10G
bulk_insert_buffer_size = 2M
myisam_repair_threads = 8
myisam_max_sort_file_size = 30G
max-allowed-packet = 256M
tmp-table-size = 32M
max-heap-table-size = 32M
query-cache-type = 0
query-cache-size = 0
max-connections = 500
thread-cache-size = 150
open-files-limit = 65535
table-definition-cache = 1024
table-open-cache = 2048
These config changes have slightly improved the amount of memory being used, but I would like to be able to use 80% of memory or so... or as much as possible to get maximum performance. Any ideas on how to increase the memory allocation to MySQL?
As you have already no IO waiting you are using a good amount of memory. Your buffers also seem quite big. So I would doubt that you can have significant CPU savings with using additional memory. You are limited by the CPU power of a single core.
Two strategies could help:
Use EXPLAIN or query analyzers to find out if you can optimize your queries to save CPU time. Adding missing indexes could help a lot. Sometimes you also might need combined indexes.
Evaluate an alternative storage engine (or even database) that is better suited for analytical queries and can use all of your cores. MariaDB supports InfiniDB but there are also other storage engines and databases available like Infobright, MonetDB.
Use show global variables like "%thread%" and you may get some clues on enabling thread concurrency options.
read_rnd_buffer_size at 2M tested at 16384 with your data may produce significant reduction in time required to complete your query.
We just switched over to Google Compute Engine and are having major issues with disk speed. It's been about 5% of Linode or worse. It's never exceeded 20M/s for writing and 10M/s for reading. Most of the time it's 15M/s for writing and 5M/s for reading.
We're currently running a n1-highmem-4 (4 vCPU, 26 GB memory) machine. CPU & memory aren't the bottleneck. Just running a script that reads rows from PostgreSQL database, processes them, then writes back to PostgreSQL. It's just for a common job to update database row in batch. Tried running 20 processes to take advantage of multi-core but the overall progress is still slow.
We're thinking disk may be bottleneck because traffic is abnormally low.
Finally we decided to do benchmarking. We found it's not only slow but seems to have a major bug which is reproducible:
create & connect to instance
run the benchmark at least three times:
dd if=/dev/zero bs=1024 count=5000000 of=~/5Gb.file
We found it becomes extremely slow and aren't able to finish the benchmarking at all.
Persistent Disk performance is proportional to the size of the disk itself and the VM that it is attached to. The larger the disk (or the VM), the higher the performance, so in essence, the price you are paying for the disk or the VM pays not only for the disk/CPU/RAM but also for the IOPS and throughput.
Quoting the Persistent Disk documentation:
Persistent disk performance depends on the size of the volume and the
type of disk you select. Larger volumes can achieve higher I/O levels
than smaller volumes. There are no separate I/O charges as the cost of
the I/O capability is included in the price of the persistent disk.
Persistent disk performance can be described as follows:
IOPS performance limits grow linearly with the size of the persistent disk volume.
Throughput limits also grow linearly, up to the maximum bandwidth for the virtual machine that the persistent disk is attached to.
Larger virtual machines have higher bandwidth limits than smaller virtual machines.
There's also a more detailed pricing chart on the page which shows what you get per GB of space that you buy (data below is current as of August 2014):
Standard disks SSD persistent disks
Price (USD/GB per month) $0.04 $0.025
Maximum Sustained IOPS
Read IOPS/GB 0.3 30
Write IOPS/GB 1.5 30
Read IOPS/volume per VM 3,000 10,000
Write IOPS/volume per VM 15,000 15,000
Maximum Sustained Throughput
Read throughput/GB (MB/s) 0.12 0.48
Write throughput/GB (MB/s) 0.09 0.48
Read throughput/volume per VM (MB/s) 180 240
Write throughput/volume per VM (MB/s) 120 240
and a concrete example on the page of what a particular size of a disk will give you:
As an example of how you can use the performance chart to determine
the disk volume you want, consider that a 500GB standard persistent
disk will give you:
(0.3 × 500) = 150 small random reads
(1.5 × 500) = 750 small random writes
(0.12 × 500) = 60 MB/s of large sequential reads
(0.09 × 500) = 45 MB/s of large sequential writes
I have published my website on Amazon EC2 (Singapore region) and I have used MySQL RDS medium instance for data storage in the same region.
In my case, most of the select queries have some COUNT functionality. These queries are showing very slow results. I have already created appropriate indexes on the table and I checked the EXPLAIN command to analyze these queries. It shows me that full table scans are necessary to get results.
On my RDS medium instance, I have configured the custom parameter group with the following settings.
log_queries_not_using_index = true,
slow_query_log = true,
long_query_time = 2 sec,
max_connections = 303,
innodb_buffer_pool_size = {DBInstanceClassMemory*3/4}
Yesterday my CPU utilization went above 95% and my site crashed due to this. There was no major increase in traffic.
Also, I dumped the data on my local system, and tested one of the COUNT queries. While it takes about 1.5 seconds for it to run on RDS, it takes only about 400 milliseconds for it to run on my local system. The configuration on my local system (4GB RAM, Intel core 2 duo 2.8GHz) is:
max_connections = 100,
slow_query_log = true,
long_query_time = 2 sec,
innodb_buffer_pool_size = 72351744
So, what could be the reason for the spike in CPU utilization as well as the difference in performance times between RDS and my local system?
Thanks,
Depending on the table size - the RDS instance uses EBS to store the data - if you're doing a table scan and its going to have to get the data from EBS instead of a locally cached in-memory key and then scan it. So - you're likely seeing the increased lag of the network between the RDS instance where the CPU resides and the EBS data in the SAN. When you do the same query on your local computer the only lag is the disk head seek time.
Then there is the difference between CPU time - an m1.medium has less CPU time (and therefore less opportunity to scan the results) than the core2 duo based on Amazon's definition of EC2 units.
HTH - in general, I'd try to avoid doing COUNT(s) in your queries as this is a terribly inefficient query (as you've seen) which can and will continue to cause nasty undesired results when the DB is under real-time varying levels of load.
R