I have a pretty much default installation on mysql on Windows 2003. I am rebuilding some indexes and the process only seems to use 3-20% of the CPU.
Is there a way to allow it to use more and speed up the process?
This applies to every application/process, not only mysql. If your database is using 3-20% CPU and the final performance is still unacceptable it means that you don't lack processor power, since it is most of the time idle. What is most probable is your bottleneck is at your HDD or HDD-controller level. Have you tested the I/O bandwitch and access time of your HD?
Can you mount a ramdisk, and move your database tables to that instead? You'll need lots of RAM, but if your DB is only a few hundred MB, then you'd be skipping the heavy disk IO. Obviously, you'd want to be working from backups in case the power went out...
Also, along the lines of what Fernando mentioned, Try to figure out where you bottleneck is. It's probably the hard disk. Open up Perfmon, and add counters for PhysicalDisk to see if that's where your bottleneck is. From the activity you are doing, it's probably writing to the actual disk that is causing the slow down.
Related
As you can see from the screenshot attached, we have experienced sudden cpu spike and storage loss. We have nearly lost all storage and had to increase it manually.
When we check database size, it still has the size before this occured, so it seems its not database related. We have checked a lot of stuff (slow logs etc.) but couldn't find the problem.
Is it possible there has been an attack, or any other ideas why this happened and how to recover our free storage?
Thank you.
It's hard to say what the exact issue is but looking at the graph, it looks like you have some huge query running against your database that is filling any temp space up. When it runs itself out of room it is killing the query and that is then flushing a bunch of writes to the disk which could either be related to the query/statement or simply be unrelated queued inserts/updates.
You need to look at the slow query log if it's enabled to see if there's anything unusual there and also check your application(s) to see if they were trying to execute a ridiculous query/statement that was hammering the database.
My application's typical DB usage is to read/update on one large table. I wonder if MySQL scales read operations on a single multi-processor machine? How about write operations - are they able to utilize multi-processors?
By the way - unfortunately I am not able to optimize the table schema.
Thank you.
Setup details:
X64, quard core
Single hard disk (no RAID)
Plenty of memory (4GB+)
Linux 2.6
MySQL 5.5
If you're using conventional hard disks, you'll often find you run out of IO bandwidth before you run out of CPU cores. The only way to pin a four core machine is to have a very high performance SSD striped RAID array.
If you're not able to optimize the schema you have very limited options. This is like asking to tune a car without lifting the hood. Maybe you can change the tires or use better gasoline, but fundamental performance gains come from several factors, including, most notably, additional indexes and strategically de-normalizing data.
In database land, 4GB of memory is almost nothing, 8GB is the absolute minimum for a system with any loading, and a single disk is a very bad idea. At the very least you should have some form of mirroring for data integrity reasons.
Let's presume that I need to maximize my write performance and am willing to take a risk of a few minutes of lost data. My use case is a "burst" of activity for a few hours which will subside. The workload is append-heavy.
Let's presume, for the sake of argument, that the data is not so urgent that a few minutes of lost data will cause as many problems as a slow server. For reasons beyond my control, the master must run on EC2, so disk speed could be an issue.
My potentially crazy idea is to have an a master database that runs entirely in RAM (either as a MEMORY table or as InnoDB backed by a RAM disk) and then replicate to a slave for slightly delayed persistence. What will go wrong?
Just use InnoDB with a huge buffer pool and set innodb_flush_log_at_trx_commit=2 or 0. It's pretty much what you're describing.
I think with SQLite3, at least it doesn't keep any cached page because there is no server and each write will exit SQLite3, so it can't do any caching directly.
But when it is MySQL, Postgresql, or MongoDB, there will be a layer which, when the data is thought to be saved already, it is actually in the memory cache of the DBMS... to be written later to the disk.
And even when it is written to the disk, there is an OS layer that keeps sectors that are to be written to the disk.
And then there is the hard drive's cache. With it being 8MB, so maybe when the test is inserting data creating a 800MB database, then the error can be 1% or less.
But what about the other layers? There really needs to be flushing all the way to the OS layer. Otherwise, with computers having 4GB of RAM or 8GB of RAM, the whole database can easily reside in RAM when it is thought to be quite fast. How do we tell the test to flush the data all the way to the hard disk physical layer or at least out of the OS layer?
When you are doing benchmarking, you can never negate all the speed optimizations at every layer down to the OS or even CPU level, including caching. You don't need that. What you can do is to benchmark performance in the different lifecycle states of your system. Also, if you know what data is cached when (approximately) you can do benchmark before and after that. For example - clean start, first DB dataset access, subsequent DB access, etc. It is best to establish the bottlenecks first and then benchmark only there in greater details. Another good practise is to simulate a real life system load and benchmark that. Any syntetic benchmarks are practically pointless. Good luck ;)
I currently have an application that is using 130 MySQL table all with MyISAM storage engine. Every table has multiple queries every second including select/insert/update/delete queries so the data and the indexes are constantly changing.
The problem I am facing is that the hard drive is unable to cope, with waiting times up to 6+ seconds for I/O access with so many read/writes being done by MySQL.
I was thinking of changing to just 1 table and making it memory based. I've never used a memory table for something with so many queries though, so I am wondering if anyone can give me any feedback on whether it would be the right thing to do?
One possibility is that there may be other issues causing performance problems - 6 seconds seems excessive for CRUD operations, even on a complex database. Bear in mind that (back in the day) ArsDigita could handle 30 hits per second on a two-way Sun Ultra 2 (IIRC) with fairly modest disk configuration. A modern low-mid range server with a sensible disk layout and appropriate tuning should be able to cope with quite a substantial workload.
Are you missing an index? - check the query plans of the slow queries for table scans where they shouldn't be.
What is the disk layout on the server? - do you need to upgrade your hardware or fix some disk configuration issues (e.g. not enough disks, logs on the same volume as data).
As the other poster suggests, you might want to use InnoDB on the heavily written tables.
Check the setup for memory usage on the database server. You may want to configure more cache.
Edit: Database logs should live on quiet disks of their own. They use a sequential access pattern with many small sequential writes. Where they share disks with a random access work load like data files the random disk access creates a big system performance bottleneck on the logs. Note that this is write traffic that needs to be completed (i.e. written to physical disk), so caching does not help with this.
I've now changed to a MEMORY table and everything is much better. In fact I now have extra spare resources on the server allowing for further expansion of operations.
Is there a specific reason you aren't using innodb? It may yield better performance due to caching and a different concurrency model. It likely will require more tuning, but may yield much better results.
should-you-move-from-myisam-to-innodb
I think that that your database structure is very wrong and needs to be optimised, has nothing to do with the storage