I am trying to benchmark MySQL with HammerDB using their TPROC-C benchmark. This is part of a research I'm doing where I am benchmarking MySQL and PostgreSQL. After running through all benchmarks with PostgreSQL, I started to benchmark MySQL by was surprised at the difference!
According to HammerDB's site, NOPM should be comparable between RDBMSs, but the numbers were so low that got me thinking about my methodology was wrong or my tuning parameters were hurting performance.
My intention was to run their benchmark with 100 warehouses and varying number of virtual users.
With PostgreSQL I go around 14000 NOPM with a single virtual user, but with MySQL, I get around 3800.
I AM NOT TRYING TO START A WAR
What would be really helpful is if someone can point me in the right direction.
I am running both MySQL and PostgreSQL on Docker, with the latest images.
Both images were limited to 12GB of RAM and HammerDB is running on the host machine.
MySQL is running on InnoDB.
These are the settings I changed from MySQL:
innodb_buffer_pool_size = 8G
innodb_buffer_chunk_size = 1G
innodb_buffer_pool_instances = 8
innodb_log_file_size = 2G
innodb_stats_on_metadata = OFF
innodb_file_per_table = ON
innodb_write_io_threads = 24
innodb_read_io_threads = 24
innodb_thread_concurrency = 0
innodb_io_capacity = 20000
key_buffer_size = 128M
thread_stack = 256K
thread_cache_size = 16
max_heap_table_size = 256M
I am a complete MySQL noob, but I want to get these tests right, that's why I've come here
I am running these on a laptop though, which is less than ideal. Here are the configs:
i7-1165G7 - 4 cores 8 threads HT 16GB RAM M.2 nvme SSD
As for HammerDB options:
Timed Driver Script (same as PG)
1000000 total txs per user (same as PG)
Prepared statements
1m ramp-up time (same as PG)
30m test time (same as PG)
Use all warehouses (same as PG)
Time profile (same as PG)
I'll be more than happy to answer any questions!
I'm not a native speaker, so forgive me for any mistakes.
The advice on the HammerDB website that the NOPM (New Orders per Minute) value is comparable between databases means that NOPM can be used to compare between different databases and systems rather than meaning all the databases should achieve the same throughput. For example, the commercial databases can achieve higher performance and scalability than the open source ones, however this is to be expected as the commercial databases have been tuned over many decades to perform well in the TPC-C tests which the HammerDB OLTP test is derived from. Additionally, databases have been designed to run multiple sessions on multiple CPUs concurrently, and therefore little can be determined about the capability of an individual database by a single user test. On an up-to-date 2 socket server in 2021 expect both MySQL and PostgreSQL to perform at peak in the range of 1,000,000 to 3,000,000 NOPM with multiple sessions depending on the hardware and software configuration. Versions MySQL 8.0.20 and PostgreSQL 13.0 upwards are recommended for higher performance.
The HammerDB discussions forum on github are the best place for specific HammerDB tuning advice.
Related
I would like to optimize MySQL for a single user (me). Noone else is connected to the database. MySQL is installed locally on my PC. The data covers around 70 tables with 150 GB of data in total.
The data is static, so I won't do changes to it (no UPDATE or DELETE). I want to analyze the data, so I will run large and complex queries including SELECT and JOIN over large proportion of the data.
Machine:
Windows 7 64-bit
Intel Core i7-4800MQ # 2,70 GHz
32 GB RAM
2x 512 GB SSD
MySQL 5.7, INNODB
What I did so far:
Deactivated HyperThreading (MySQL uses only one virtual Core per
query - CPU usage 12.5% --> 25%)
Declare primary keys and
Indexed all foreign keys
innodb_buffer_pool_size = 25G
max_connections=10
Use of InnoDB
So what can I do to optimize the configuration (not the query itself) for single-user-queries?
database engine?
general configuration?
better cache all the information of the joins?
Note:
Under the current configuration, the CPU usage is the bottleneck because it is on 25% when I run a complex query. As test i tried some huge queries (fetching a lot of data). If I can believe the timing of MYSQL Workbench the duration was only 4 seconds, but after 10 hours of running it couldn't finish fetching the data...
Thanks a lot!
You could try MySiam. It is especially good for read-intensive (select) tables. And you should select a large key_buffer_size. But I am not sure about the JOIN performance... Give it a try
Summary:
I haven't yet been able to get MySQL to use more than 1 core for a select statement and it doesn't get above 10 or 15 GB of RAM.
The machine:
I have a dedicated Database server running MariaDB using MySQL 5.6. The machine is strong with 48 cores and 192GB of RAM.
The data:
I have about 250 million rows in one large table (also several other tables ranging from 5-100 million rows). I have been doing a lot of reading from the tables, sometimes inserting into a new table to denormalize the data a bit. I am not setting this system up as a transactional system, rather, it will be used more similarly to a data warehouse with few connections.
The problem:
When I look at my server's stats, it looks like CPU is at around 70% for one core with a select query running, and memory is at about 5-8%. There is no IO waiting, so I am convinced that I have a problem with MySQL memory allocation. After searching on how to increase the usage of memory in MySQL I have noticed that the config file may be the way to increase memory usage.
The solution I have tried based on my online searching:
I have changed the tables to MyISAM engine and added many indexes. This has helped performance, but querying these tables is still incredibly slow. The write speed using load data infile is very fast, however, running a mildly complex select query takes hours or even days.
I have also tried adjusting the following configurations:
key-buffer-size = 64G
read_buffer_size = 1M
join_buffer_size = 4294967295
read_rnd_buffer_size = 2M
key_cache_age_threshold = 400
key_cache_block_size = 800
myisam_data_pointer_size = 7
preload_buffer_size = 2M
sort_buffer_size = 2M
myisam_sort_buffer_size = 10G
bulk_insert_buffer_size = 2M
myisam_repair_threads = 8
myisam_max_sort_file_size = 30G
max-allowed-packet = 256M
tmp-table-size = 32M
max-heap-table-size = 32M
query-cache-type = 0
query-cache-size = 0
max-connections = 500
thread-cache-size = 150
open-files-limit = 65535
table-definition-cache = 1024
table-open-cache = 2048
These config changes have slightly improved the amount of memory being used, but I would like to be able to use 80% of memory or so... or as much as possible to get maximum performance. Any ideas on how to increase the memory allocation to MySQL?
As you have already no IO waiting you are using a good amount of memory. Your buffers also seem quite big. So I would doubt that you can have significant CPU savings with using additional memory. You are limited by the CPU power of a single core.
Two strategies could help:
Use EXPLAIN or query analyzers to find out if you can optimize your queries to save CPU time. Adding missing indexes could help a lot. Sometimes you also might need combined indexes.
Evaluate an alternative storage engine (or even database) that is better suited for analytical queries and can use all of your cores. MariaDB supports InfiniDB but there are also other storage engines and databases available like Infobright, MonetDB.
Use show global variables like "%thread%" and you may get some clues on enabling thread concurrency options.
read_rnd_buffer_size at 2M tested at 16384 with your data may produce significant reduction in time required to complete your query.
I need to improve I/O performance for my database. I'm using the "2xlarge" HW described below & considering upgrading to the "4xlarge" HW (http://aws.amazon.com/ec2/instance-types/). Thanks for the help!
Details:
CPU usage is fine (usually under 30%), uptime load averages anywhere from 0.5 to 2.0 (but I believe I'm supposed to divide that by the number of CPU's) so that looks okay as well. However, the I/O is bad: iostat show favorable service times, but the time spent in queue (I suppose this means waiting to access the disk) is far too high. I've configured MySQL to flush to disk every 1sec instead of every write, which helps, but not enough. Profiling shows there are a handful of tables that are the culprits for most of the load (both read && write operations). Queries are already indexed & optimized, but not partitioned. Average MySQL states are: Sending Data # 45%, statistics # 20%, Updating # 15%, Sorting result # 8%.
Questions:
How much performance will I get by upgrading HW?
Same question, but if I partition the high-load tables?
Machines:
m2.2xlarge
64-bit
4 vCPU
13 ECU
34.2 Gb Mem
EBS-Optimized
Network Performance: "Moderate"
m2.4xlarge
64-bit
6 vCPU
26 ECU
68.4 Gb Mem
EBS-Optimized
Network Performance: "High"
In my experience, the biggest boost in MySQL performance comes from IO. You have alot of RAM. Try setting up a ram drive and point the tmpdir to it.
I have several MySQL servers that are very busy. My settings are below - maybe this can help you tweak your settings.
My Setup is:
-Dual 2.66 CPU 8 core box with a 6-drive Raid-1E array - 1.3TB.
-innodblogs on a separate SSD drives.
-tmpdir is on a 2GB tempfs partition.
-32GB of ram
InnoDB settings:
innodb_thread_concurrency=16
innodb_buffer_pool_size = 22G
innodb_additional_mem_pool_size = 20M
innodb_log_file_size = 400M
innodb_log_files_in_group=8
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 2 (This is a slave machine - 1 is not required fo my purposes)
innodb_flush_method=O_DIRECT
Current Queries per second avg: 5185.650
I am using Percona Server, which is quite a bit faster that other MySQLs from my testing.
I have published my website on Amazon EC2 (Singapore region) and I have used MySQL RDS medium instance for data storage in the same region.
In my case, most of the select queries have some COUNT functionality. These queries are showing very slow results. I have already created appropriate indexes on the table and I checked the EXPLAIN command to analyze these queries. It shows me that full table scans are necessary to get results.
On my RDS medium instance, I have configured the custom parameter group with the following settings.
log_queries_not_using_index = true,
slow_query_log = true,
long_query_time = 2 sec,
max_connections = 303,
innodb_buffer_pool_size = {DBInstanceClassMemory*3/4}
Yesterday my CPU utilization went above 95% and my site crashed due to this. There was no major increase in traffic.
Also, I dumped the data on my local system, and tested one of the COUNT queries. While it takes about 1.5 seconds for it to run on RDS, it takes only about 400 milliseconds for it to run on my local system. The configuration on my local system (4GB RAM, Intel core 2 duo 2.8GHz) is:
max_connections = 100,
slow_query_log = true,
long_query_time = 2 sec,
innodb_buffer_pool_size = 72351744
So, what could be the reason for the spike in CPU utilization as well as the difference in performance times between RDS and my local system?
Thanks,
Depending on the table size - the RDS instance uses EBS to store the data - if you're doing a table scan and its going to have to get the data from EBS instead of a locally cached in-memory key and then scan it. So - you're likely seeing the increased lag of the network between the RDS instance where the CPU resides and the EBS data in the SAN. When you do the same query on your local computer the only lag is the disk head seek time.
Then there is the difference between CPU time - an m1.medium has less CPU time (and therefore less opportunity to scan the results) than the core2 duo based on Amazon's definition of EC2 units.
HTH - in general, I'd try to avoid doing COUNT(s) in your queries as this is a terribly inefficient query (as you've seen) which can and will continue to cause nasty undesired results when the DB is under real-time varying levels of load.
R
I would like to know how it is possible to set an upper limit on the amount of memory MySQL uses on a Linux server.
Right now, MySQL will keep taking up memory with every new query requested so that it eventually runs out of memory. Is there a way to place a limit so that no more than that amount is used by MySQL?
MySQL's maximum memory usage very much depends on hardware, your settings and the database itself.
Hardware
The hardware is the obvious part. The more RAM the merrier, faster disks ftw. Don't believe those monthly or weekly news letters though. MySQL doesn't scale linear - not even on Oracle hardware. It's a little trickier than that.
The bottom line is: there is no general rule of thumb for what is recommend for your MySQL setup. It all depends on the current usage or the projections.
Settings & database
MySQL offers countless variables and switches to optimize its behavior. If you run into issues, you really need to sit down and read the (f'ing) manual.
As for the database -- a few important constraints:
table engine (InnoDB, MyISAM, ...)
size
indices
usage
Most MySQL tips on stackoverflow will tell you about 5-8 so called important settings. First off, not all of them matter - e.g. allocating a lot of resources to InnoDB and not using InnoDB doesn't make a lot of sense because those resources are wasted.
Or - a lot of people suggest to up the max_connection variable -- well, little do they know it also implies that MySQL will allocate more resources to cater those max_connections -- if ever needed. The more obvious solution might be to close the database connection in your DBAL or to lower the wait_timeout to free those threads.
If you catch my drift -- there's really a lot, lot to read up on and learn.
Engines
Table engines are a pretty important decision, many people forget about those early on and then suddenly find themselves fighting with a 30 GB sized MyISAM table which locks up and blocks their entire application.
I don't mean to say MyISAM sucks, but InnoDB can be tweaked to respond almost or nearly as fast as MyISAM and offers such thing as row-locking on UPDATE whereas MyISAM locks the entire table when it is written to.
If you're at liberty to run MySQL on your own infrastructure, you might also want to check out the percona server because among including a lot of contributions from companies like Facebook and Google (they know fast), it also includes Percona's own drop-in replacement for InnoDB, called XtraDB.
See my gist for percona-server (and -client) setup (on Ubuntu): http://gist.github.com/637669
Size
Database size is very, very important -- believe it or not, most people on the Intarwebs have never handled a large and write intense MySQL setup but those do really exist. Some people will troll and say something like, "Use PostgreSQL!!!111", but let's ignore them for now.
The bottom line is: judging from the size, decision about the hardware are to be made. You can't really make a 80 GB database run fast on 1
GB of RAM.
Indices
It's not: the more, the merrier. Only indices needed are to be set and usage has to be checked with EXPLAIN. Add to that that MySQL's EXPLAIN is really limited, but it's a start.
Suggested configurations
About these my-large.cnf and my-medium.cnf files -- I don't even know who those were written for. Roll your own.
Tuning primer
A great start is the tuning primer. It's a bash script (hint: you'll need linux) which takes the output of SHOW VARIABLES and SHOW STATUS and wraps it into hopefully useful recommendation. If your server has ran some time, the recommendation will be better since there will be data to base them on.
The tuning primer is not a magic sauce though. You should still read up on all the variables it suggests to change.
Reading
I really like to recommend the mysqlperformanceblog. It's a great resource for all kinds of MySQL-related tips. And it's not just MySQL, they also know a lot about the right hardware or recommend setups for AWS, etc.. These guys have years and years of experience.
Another great resource is planet-mysql, of course.
We use these settings:
etc/my.cnf
innodb_buffer_pool_size = 384M
key_buffer = 256M
query_cache_size = 1M
query_cache_limit = 128M
thread_cache_size = 8
max_connections = 400
innodb_lock_wait_timeout = 100
for a server with the following specifications:
Dell Server
CPU cores: Two
Processor(s): 1x Dual Xeon
Clock Speed: >= 2.33GHz
RAM: 2 GBytes
Disks: 1×250 GB SATA
mysqld.exe was using 480 mb in RAM. I found that I added this parameter to my.ini
table_definition_cache = 400
that reduced memory usage from 400,000+ kb down to 105,000kb
Database memory usage is a complex topic. The MySQL Performance Blog does a good job of covering your question, and lists many reasons why it's hugely impractical to "reserve" memory.
If you really want to impose a hard limit, you could do so, but you'd have to do it at the OS level as there is no built-in setting. In linux, you could utilize ulimit, but you'd likely have to modify the way MySQL starts in order to impose this.
The best solution is to tune your server down, so that a combination of the usual MySQL memory settings will result in generally lower memory usage by your MySQL installation. This will of course have a negative impact on the performance of your database, but some of the settings you can tweak in my.ini are:
key_buffer_size
query_cache_size
query_cache_limit
table_cache
max_connections
tmp_table_size
innodb_buffer_pool_size
I'd start there and see if you can get the results you want. There are many articles out there about adjusting MySQL memory settings.
Edit:
Note that some variable names have changed in the newer 5.1.x releases of MySQL.
For example:
table_cache
Is now:
table_open_cache
About how MYSQL is eating memory: https://dev.mysql.com/doc/refman/8.0/en/memory-use.html
in /etc/my.cnf:
[mysqld]
...
performance_schema = 0
table_cache = 0
table_definition_cache = 0
max_connect_errors = 10
query_cache_size = 0
query_cache_limit = 0
...
Good work on server with 256MB Memory.
If you are looking for optimizing your docker mysql container then the below command may help. I was able to run mysql docker container from a default 480mb to mere 100 mbs
docker run -d -p 3306:3306 -e MYSQL_DATABASE=test -e MYSQL_ROOT_PASSWORD=tooor -e MYSQL_USER=test -e MYSQL_PASSWORD=test -v /mysql:/var/lib/mysql --name mysqldb mysql --table_definition_cache=100 --performance_schema=0 --default-authentication-plugin=mysql_native_password
Since I do not have enough reputation points to upvote a previous answer, I concur that the "table_definition_cache = 400" answer worked on my old Centos server.