I had a look at this:
http://www.mysqlperformanceblog.com/2009/01/12/should-you-move-from-myisam-to-innodb/
and:
http://www.mysqlperformanceblog.com/2007/11/01/innodb-performance-optimization-basics/
These answer a lot of my questions regarding INNODB vs MyISAM. There is no doubt in my mind that INNODB is the way I should go. However, I am working on my own and for development I have created a LAMP (ubuntu 10.10 x64) VM server. At present the server has 2 GB memory and a single SATA 20GB drive. I can increase both of these amounts without too much trouble to about 3-3.5 GB memory and a 200GB drive.
The reasons I hesitate to switch over to INNODB is:
A) The above articles mention that INNODB will vastly increase the size of the tables, and he recommends much larger amounts of RAM and drive space. While in a production environment I don't mind this increase, in a development environment, I fear I can not accommodate.
B) I don't really see any point in fine tuning the INNODB engine on my VM. This is likely something I will not even be allowed to do in my production environment. The articles make it sound like INNODB is doomed to fail without fine tuning.
My question is this. At what point is INNODB viable? How much RAM would I need to run INNODB on my server (with just my data for testing. This server is not open to anyone but me)? and also is it safe for me to assume that a production environment that will not allow me to fine tune the DB has likely already fine tuned it themselves?
Also, am I overthinking/overworrying about things?
IMHO, it becomes a requirement when you have tens of thousands of rows, or when you can forecast the rate of growth for data.
You need to focus on tuning the innodb buffer pool and the log file size. Also, make sure you have innodb_file_per_table enabled.
To get an idea of how big to make the innodb buffer pool in KB, run this query:
SELECT SUM(data_length+index_length)/power(1024,1) IBPSize_KB
FROM information_schema.tables WHERE engine='InnoDB';
Here it is in MB
SELECT SUM(data_length+index_length)/power(1024,2) IBPSize_MB
FROM information_schema.tables WHERE engine='InnoDB';
Here it is in GB
SELECT SUM(data_length+index_length)/power(1024,3) IBPSize_GB
FROM information_schema.tables WHERE engine='InnoDB';
I wrote articles about this kind of tuning
First Article
Second Article
Third Article
Fourth Article
IF you are limited by the amount of RAM on your server, do not surpass more than 25% of the installed for the sake of the OS.
I think you may be over thinking things. Its true that INNODB loves ram but if your database is small I don't think you'll have many problems. The only issue I have had with MYSQL or any other database is that as the data grows so do the requirements for accessing it quickly. You can also use compression on the tables to keep them smaller but INNODB is vastly better than MYISAM at data integrity.
I also wouldn't worry about tuning your application until you run into a bottleneck. Writing efficient queries and database design seems to be more important than memory unless you're working with very large data sets.
Related
For my development machine I need no data consistency in case of a crash. Is there a config for a Debian-like system, that optimizes MySQL for speed (even if it sacrifices reliability)?
So something like: Cache the last 1 GB in RAM. Don't touch the disk with data until the 1 GB is used.
What kind of queries are going on? One of my mantras: "You cannot configure your way out of a performance problem."
Here's one thing that speeds up InnoDB, wrt transactions:
innodb_flush_log_at_trx_commit = 2
There is a simple way to speed up single-row inserts by a factor of 10.
Some 'composite' indexes can speed up a SELECT by a factor of 100.
Reformulating a WHERE can sometimes speed up a query by a factor of 100.
You can disable many of the InnoDB configurations for durability, at the risk of increased risk of losing data. But sometimes you want to operate the database in Running with scissors mode because the original data is safely stored somewhere else, and the copy in your test database is easily recreated.
This blog describes Reducing MySQL durability for testing. You aren't going to see any official MySQL recommendation to do this for any purpose other than testing!
Here's a summary of changes you can make in your /etc/my.cnf:
[mysqld]
# log_bin (comment this out to disable the binary log)
# sync_binlog=0 (irrelevant if you don't use the binary log)
sync_frm=0
innodb_flush_log_at_trx_commit=0
innodb_doublewrite=0
innodb_checksums=0
innodb_support_xa=0
innodb_log_file_size=2048M # or more
He also recommends to increase innodb_buffer_pool_size, but the size depends on your available RAM.
For what it's worth, I recently tried to set innodb_flush_log_at_trx_commit=0 in the configuration in the default Vagrant box I built for developers on my team, but I had to back out that change because it was causing too much lost time for developers who were getting corrupted databases. Just food for thought. Sometimes it's not a good tradeoff.
This doesn't do exactly what you asked (keep the last 1GB of data in RAM), as it still operates InnoDB with transaction logging and the log flushes to disk once per second. There's no way to turn that off in MySQL.
You could try using MyISAM, which uses buffered writes for data and index, and relies on the filesystem buffer. Therefore it could cache some of your data (in practice I have found that the buffer flushes to disk pretty promptly, so you're unlikely to have a full 1GB in RAM at any time). MyISAM has other problems, like lack of support for transactions. Developing with MyISAM and then using InnoDB in production can set you up for some awkward surprises.
Here's a couple of other changes you could make in your MySQL sessions for the sake of performance, but I don't recommend these even for development, because it can change your application behavior.
set session unique_checks=0;
set session foreign_key_checks=0;
Some people recommend using the MEMORY storage engine. That has its own problems, like size limits, table-locking, and lack of support for transactions.
I've also experimented with trying to put tables or tmpdir onto a tmpfs, but I found that didn't give nearly the performance boost you might expect. There's overhead in an RDBMS that is not directly related to disk I/O.
You might also like to experiment with MyRocks, a version of MySQL including the RocksDB storage engine for MySQL. Facebook developed it and released it as open-source. See Facebook rocks an open source storage engine for MySQL (InfoWorld). They promise it reduces I/O, it compresses data, and does other neat things.
But again, it's a good rule of thumb to make your development environment as close as possible to your production environment. Using a different storage engine creates a risk of not discovering some bugs until your code reaches production.
Bottom line: Tuning MySQL isn't a magic bullet. Maybe you should consider designing your application to make more use of microservices, caches, and message queues, and less reliance on direct SQL queries.
Also, I'd recommend to always supply your developers the fastest SSD-based workstation you can afford. Go for the top of the line on CPU and RAM and disk speed.
#Bill Karwin's answer has useful mysql settings to improve performance. I have used them all and was able to achieve a roughly 2x performance improvement.
However, what gave me the biggest performance boost (nearly 15x faster) for my use case -- which was reloading a mysql dump -- was to mount the underlying filesystem (ext4) using the nobarriers option.
mount -o remount,nobarrier /
More info here
You should only consider this if you have a separate partition (or logical volume) mounted at /var/lib/mysql, so that you can make this tradeoff only for MySQL, not your entire system.
Although this answer may not hit exactly the questions you ask, consider creating your tables with MEMORY engine as documented here: http://dev.mysql.com/doc/refman/5.7/en/memory-storage-engine.html
A typical use case for the MEMORY engine involves these
characteristics:
Operations involving transient, non-critical data such as session
management or caching. When the MySQL server halts or restarts, the
data in MEMORY tables is lost.
In-memory storage for fast access and low latency. Data volume can fit
entirely in memory without causing the operating system to swap out
virtual memory pages.
A read-only or read-mostly data access pattern (limited updates).
Give that a shot.
My recommendation, even for a development machine, would be to use the default InnoDB. If you choose to do transactions, InnoDB will be helpful.
This blog can help you run MySQL off of tmpfs: http://jotschi.de/2014/02/03/high-performance-mysql-testdatabase/. User Jotschi also speaks about that in a SO answer #10692398
Our application was using MySql version 4.0.24 for a long time. We are trying to migrate it to version 5.6.27.
But, on testing the performance on 5.6.27, even the simple selects and updates are 30-40% slower when we are doing load testing. The CPU and IO speeds are much better than the older server. The storage engine of the tables is MyIsam in both versions. There's only one connection to the database. We tried the following options:
Changing storage engine to InnoDb - this reduce the performance drastically (70% slower)
Changing the innodb log size and buffer size - didn't help much
Increasing key buffer size with MyIsam storage engine for tables. - It made no difference
We tried modifying other parameters like query cache, tmp_table_size, heap_table_size. But, none of them made any difference.
Can you please let me know if there's any other option that we can try?
Here's a copy of my.cnf:
lower-case-table-names=1
myisam-recover=FORCE
key_buffer_size=2000M
Some things you can look at are whether the two servers have the same amount of RAM or not as it may be that the old server has more RAM and so can cache more things in memory.
You can also look at how are you connecting to the MySQL server - is it over a network? Is that network speed / quality different? Is one server accessed locally and the other over a network.
You tried tuning some good parameters, but in case there are ones you're missing, you can run mysql -e 'show variables' on both servers and then use a tool like Winmerge to compare the values of the two and see what might be different that you might not have thought of.
After trying multiple options, here are the configurations that worked for us. There may be other solutions as well but this worked for us.
Option 1:
Turned off the performance schema
Added a couple of jdbc connection parameters: useConfigs=maxPerformance&maintainTimeStats=false
Option 2:
Migrate to MariaDB. From a performance perspective, this worked out really well. It was giving a 5% better performance compared to mysql for our system. But, we couldn't pursue this option due to non-technical reasons.
Thank you for your inputs.
We have a windows 2003 server with MySQL 5.1 installed. Since a few days the mysqldump is almost 1 hour slower than normal. The total size of the databases was growing with approx. 1 MB per day. Not much differences.
Our technical support guy can't find strange things on our machine. He says that it would be a good idea to defrag our server with MySQL on it. I never thought about defragmentation but I have Defraggler installed. After anaylizing it says 80% fragmentation!
Could that be the issue / reason why the dump's are getting slower with almost 1 hour ?
My first idea was that some windows update may be the reason.
Please sent me your thoughts
And is it wise to do a defragmentation on a MySQL production server??
Defraggler says deframentation could take more than 1 day ?
Your database dump is probably disk-bound, meaning that the is limited by accessing data on the disk.
If the data on the disk is highly fragmented, it can significantly affect the performance. Depending on the layout of the data on the disk, MySQL may be filling in lots of small gaps in the disk with the new data, which will be much slower to work with than large contiguous blocks of data.
Regular defragmentation is a good idea. However, since MySQL likely has a lock on the data files for your databases, it may be necessary to shut down MySQL to defragment its files.
I suggest at least running defraggler without shutting down MySQL to defragment as much as possible. I'm not familiar with defraggler, but if it makes it possible, try to find out if MySQL's data is fragmented. It might have a table of fragmented files or a graphical view of the data on the disk. The MySQL data files will probably be some of the largest files so that might make it easier to find them. If you find that its files are heavily fragmented, you may then want to try to find some time to shut down MySQL and defragment its files.
I don't want to ask a subjective "which DBMS is best?" or "which DBMS of these two is better?". This doesn't have to be a fanboy debate.
Rather, I welcome any benchmark test results or specific experiences, when it comes to one specific criteria - performance - especially with respect to one particular application: WordPress.
I understand that WordPress doesn't use InnoDB, and so disabling InnoDB in MySQL can speed things up. On the other hand, Percona is a MySQL fork that replaces InnoDB with XtraDB and also claims to be highly efficient, high-performance.
How does each stack up on performance when it comes to running WordPress? (no need for competition...both might come out looking very well, for all I know)
I have tried searching generally on Google, but haven't come across so much as an intelligent discussion, let alone performance benchmark tests.
Would greatly appreciate if any of the experts here could share their experiences. Many thanks!
And please keep any smug, snide comments like "why don't YOU try" to yourself. If I could, I would. And the purpose of Stack Overflow is to share expertise and learn from each-other, not to do everything yourself.
This question is less "MySQL vs. Percona Server" than "MyISAM vs. InnoDB/XtraDB". They both have their own performance characteristics and which storage engine is right for you largely depends on your workload. Most Wordpress sites are low traffic and read-mostly, so as long as your data fits into your buffer pool (for InnoDB/XtraDB) or key cache (for MyISAM), I would expect not-too-dissimilar performance.
Having done a lot of work on Wordpress database optimization, I can tell you that the performance of your Wordpress site depends more upon the class of your hardware and your chosen plugins.
You should use a Caching plugin so that you can just avoid a ton of database read requests
You should avoid plugins that issue expensive queries (sadly, this covers most plugins)
You should prune your comments (usually comments are 99+% SPAM so the ones that are marked as spam are just sitting in your database taking up space)
Your host should have enough RAM for the hot dataset to fit in memory
If you really want to go into detail about MyISAM vs. InnoDB/XtraDB, you can check out the following links:
http://www.mysqlperformanceblog.com/2009/01/12/should-you-move-from-myisam-to-innodb/
http://www.rackspace.com/knowledge_center/article/mysql-engines-myisam-vs-innodb
So, to make a long answer even longer, you'll need to profile your MySQL instance after you can generate production traffic. I know you said you couldn't, but ... this question is kind of like me asking "What haircut would look best on me", without including a picture.
Wordpress can use InnoDB (or XtraDB) just fine. I have done consulting and training for sites that host WordPress at scale, using any of MyISAM, InnoDB, and XtraDB.
WordPress 3.5.1 creates tables without specifying the storage engine. So it honors the default storage engine on whatever instance of MySQL you're using. As of MySQL 5.5 (ca. December 2010), the default storage engine is InnoDB. I tested installing WordPress on a virtual host running MySQL 5.6.10, and it created tables using the InnoDB storage engine.
I don't have any benchmarks to share, but those would be of limited use anyway, because performance depends so much on the given hardware, the traffic load, and other factors.
A CMS like WordPress tends to be heavily weighted toward read-only queries. This is where InnoDB should give good benefit, because it caches data pages as well as indexes. MyISAM only caches indexes, and relies on the filesystem cache to hold data.
So the key to making WordPress perform well is to allocate enough innodb_buffer_pool_size to hold the data and indexes for all your tables. The data size of a WordPress site (even one with hundreds of articles) isn't typically very large, so you probably only need a few GB of buffer pool to hold all frequently-requested data in the buffer. Once the data and index pages have populated the InnoDB buffer pool, 99.9% of your queries will be served out of RAM, and the site will have great performance.
As with any caching system, the real killer to performance is when your "hot" data is larger than the cache, forcing queries to incur disk I/O. A single disk I/O is worth a few thousand RAM accesses, so you want to serve content completely out of RAM as much as possible.
The improvements in XtraDB are designed to help as the number of Threads_running gets higher, or the buffer pool gets larger (e.g. dozens of GB). It's unlikely that a single WP site will exercise either MySQL or Percona Server so heavily that these improvements will offer more than a slight advantage. Unless you're going to host hundreds of WP sites on a given server like a hosting company.
You may even find that the bottleneck ceases to be the database, and then you need to focus on front-end optimizations.
I currently have an application that is using 130 MySQL table all with MyISAM storage engine. Every table has multiple queries every second including select/insert/update/delete queries so the data and the indexes are constantly changing.
The problem I am facing is that the hard drive is unable to cope, with waiting times up to 6+ seconds for I/O access with so many read/writes being done by MySQL.
I was thinking of changing to just 1 table and making it memory based. I've never used a memory table for something with so many queries though, so I am wondering if anyone can give me any feedback on whether it would be the right thing to do?
One possibility is that there may be other issues causing performance problems - 6 seconds seems excessive for CRUD operations, even on a complex database. Bear in mind that (back in the day) ArsDigita could handle 30 hits per second on a two-way Sun Ultra 2 (IIRC) with fairly modest disk configuration. A modern low-mid range server with a sensible disk layout and appropriate tuning should be able to cope with quite a substantial workload.
Are you missing an index? - check the query plans of the slow queries for table scans where they shouldn't be.
What is the disk layout on the server? - do you need to upgrade your hardware or fix some disk configuration issues (e.g. not enough disks, logs on the same volume as data).
As the other poster suggests, you might want to use InnoDB on the heavily written tables.
Check the setup for memory usage on the database server. You may want to configure more cache.
Edit: Database logs should live on quiet disks of their own. They use a sequential access pattern with many small sequential writes. Where they share disks with a random access work load like data files the random disk access creates a big system performance bottleneck on the logs. Note that this is write traffic that needs to be completed (i.e. written to physical disk), so caching does not help with this.
I've now changed to a MEMORY table and everything is much better. In fact I now have extra spare resources on the server allowing for further expansion of operations.
Is there a specific reason you aren't using innodb? It may yield better performance due to caching and a different concurrency model. It likely will require more tuning, but may yield much better results.
should-you-move-from-myisam-to-innodb
I think that that your database structure is very wrong and needs to be optimised, has nothing to do with the storage