I have an instance of MySQL Server 5.6.20 running on Windows Server 2012. One table in particular in my database is very large (23 GB on disk, 31 million rows). When I query this table, even for simple operations such as a count(*), the performance is terrible, frequently taking as long as 40 minutes to complete.
Checking the resource monitor, I see my Highest Active Time pinned at 100%, but only reading 1.5-2.0 MB per second from the disk (much below the peak performance of the drive). Internet research suggests this happens when reading highly fragmented files from disk, but the only file being read is the MySQL InnoDB database file. Am I interpreting this right that the data file itself is heavily fragmented? Is there an SQL specific solution or is windows defrag the correct approach to this problem?
EDIT
There are two Dell PERC H310 SCSI 1.8 TB Disks in the machine. Only one is formatted. RAID was never setup. No SSDs are installed.
Related
I'm doing a project that requires converting one SQL database (MySQL) to graph database (neo4j). Database has more than 20 tables, and the biggest one has about 13.000.000 entries. I've migrated only few tables, and I have more than 15.000.000 nodes and about the same number of relations.
Loading data is very slow (I've loaded nodes very fast, but I had problems with creating relations for the nodes from the biggest table). Also, MySQL queries are running faster than neo4j ones.
When I take a look at my task manager, I can see that neo4j is not using full computer power. How can I set maximum computer usage for neo4j?
I'm using a (Windows) laptop with
Intel Core i7 4702MQ, 2.2GHz
8 GB DDR3
SSD 850 EVO, 500GB
When a query is run, neo4j uses 2-3GB of ram (there are 2gb more) and maybe 20% of the CPU power. I've tried to change settings in Java VM Tunning options (neo4j-community.vmoptions) but that didn't gave me any result.
I'm using neo4j community edition, version 3.0.
I have a moderately large database (~75 million rows, ~300 GB), and a fairly powerful machine (amazon r3.8xlarge, 32 CPUs, 244 GB RAM). Unfortunately, MySQL doesn't seem to be using all this power. It barely touches a tenth of the RAM.
My application of this database is a python program which reads the tables, crunches numbers, and stores results back in the database. There's only one user, so no table locks are needed.
Ideally, nearly the entire database would be stored in RAM. I considered mounting a straight RAM filesystem, but this means having to offload everything back to a hard drive myself. I set the key_buffer_size and innodb_buffer_pool_size to 100G each, but it doesn't seem to have any effect.
Is there something obvious I'm missing?
We have a windows 2003 server with MySQL 5.1 installed. Since a few days the mysqldump is almost 1 hour slower than normal. The total size of the databases was growing with approx. 1 MB per day. Not much differences.
Our technical support guy can't find strange things on our machine. He says that it would be a good idea to defrag our server with MySQL on it. I never thought about defragmentation but I have Defraggler installed. After anaylizing it says 80% fragmentation!
Could that be the issue / reason why the dump's are getting slower with almost 1 hour ?
My first idea was that some windows update may be the reason.
Please sent me your thoughts
And is it wise to do a defragmentation on a MySQL production server??
Defraggler says deframentation could take more than 1 day ?
Your database dump is probably disk-bound, meaning that the is limited by accessing data on the disk.
If the data on the disk is highly fragmented, it can significantly affect the performance. Depending on the layout of the data on the disk, MySQL may be filling in lots of small gaps in the disk with the new data, which will be much slower to work with than large contiguous blocks of data.
Regular defragmentation is a good idea. However, since MySQL likely has a lock on the data files for your databases, it may be necessary to shut down MySQL to defragment its files.
I suggest at least running defraggler without shutting down MySQL to defragment as much as possible. I'm not familiar with defraggler, but if it makes it possible, try to find out if MySQL's data is fragmented. It might have a table of fragmented files or a graphical view of the data on the disk. The MySQL data files will probably be some of the largest files so that might make it easier to find them. If you find that its files are heavily fragmented, you may then want to try to find some time to shut down MySQL and defragment its files.
I've got a query that is running 5x slower on my staging server as opposed to my local dev machine.
Stackoverflow doesn't want to play nicely with the formatting; the query, describes, and explains are located here
Looking at the describe statements, I can't see any difference between the local and remote schemas.
The record counts for the 2 machines are in the same order of magnitude (500k vs 600k)
Edit In Response to Comments
It was my highly unscientific approach of throwing the queries into MySQL Workbench and looking at the query time. The local query time was on the order of 1.3 seconds and the remote query time was on the order of 5.2 seconds (so its 4x as slow). I'm sure there's a better way to test this query time.
The machines are different. My dev machine is a Mac Book Pro with 8 gigs of RAM. The staging server is a linode VPS with 512 megabytes of RAM. There shouldn't be much load on the staging server (I'm the only one that uses it). I've noticed most queries run in approximately the same time frame on the local machine and staging server, so I was confused as to why this one had such a drastically different time frame.
RAM Issue
Since a temporary table isn't being used (no mention in the EXPLAINS), is the amount of RAM still an issue?
Output from free
total used free shared buffers cached
Mem: 508576 453880 54696 0 4428 254200
-/+ buffers/cache: 195252 313324
Swap: 262140 19500 242640
Profiling Added to Gist
It looks like the remote is taking 2.5 seconds 'sending data' whereas the local is only taking 0.5 seconds. Is this an I/O issue? (Complete profiling info in gist)
Your staging server has one sixteenth of the RAM that you Mac Book Pro has.
Without knowing how much RAM is available to your two instances of MySQL, it's hard to be definitive, but that's the first place I'd look.
Also, if you run these queries from the MySQL command line, locally, how do the times compare?
It could be that the increase in time is in network transfer and not query processing.
Actually... network transfer time is the first place I'd look... then MySQL memory usage.
EDIT following question updates
The 'sending data' phase is the phase where the server is sending data to the client ref. I don't know exactly how large your dataset is, but 2.5s seems pretty high for what's probably 50kB of data or so.
Having looked at the profiling data, nearly all the time is spent sending data, so I'd strongly suspect the network here.
EDIT 2
Some research lead me to this page which indicates that the 'Sending data' is misleading and that this is actually the time spend executing your query.
Thus, I really think you need to be looking at CPU and memory usage on your server since it's specced at a level so much lower than your MacBook.
I'm trying to build a dev copy of a production MySQL database by loading one of the backups. How long should it take to do this if the uncompressed dump is ~20G?
This command has been running for something like 24h with 10% CPU load and I'm wondering if it's just slow or if it/I am doing something wrong.
mysql -u root -p < it_mysql_dump.sql
BTW it's on a beefy desktop dev machine with plenty of ram, but it might be reading and writing the same HDD. I think I'm using InnoDB.
Restoring MySQL dumps can take a long time. This is because it does really rebuild the entire tables.
Exactly what you need to do to fix it depends on the engine, but in general
I would say, do the following:
Zeroth rule: Only use a 64-bit OS.
Make sure that you have enough physical ram to fit the biggest single table into memory; include any overhead for the OS in this calculation (NB: On operating systems that use 4k pages i.e. all of them, the page tables take up a lot of memory themselves on large-memory systems - don't forget this)
Tune the innodb_buffer_pool such that it is bigger than the largest single table; or if using MyISAM, tune the key_buffer so that it is big enough to hold the indexes of the largest table.
Be patient.
Now, if you are still finding that it is slow having done the above, it may be that your particular database has a very tricky structure to restore.
Personally I've managed to rebuild a server with ~ 2Tb in < 48 hours, but that was a particular case.
Be sure that your development system has production-grade hardware if you intend to load production data into it.
In particular, if you think that you can bulk-load data into tables which don't fit into memory (or at least, mostly into memory), forget it.
If this all seems like too much, remember that you can just use a filesystem or LVM snapshot online with InnoDB, and then just copy the files. With MyISAM it's a bit trickier but can still be done.
Open another terminal, run mysql, and count the rows in some of the tables in your dump (SELECT COUNT(*) FROM table). Compare to the source database. That'll tell you the progress.
I INSERTed about 80GB of data into MySQL over a network in about 14 hours. They were one-insert-per-row dumps (slow) with a good bit of overhead, inserting on a server with fast disks.
24 hours is possible if the hardware is old enough, or your import is competing with something else for disk IO and memory.
I just went through the experience of restoring a 51.8 Gb database from a 36.8 Gb mysqldump file to create an imdb database. For me the restore which was not done over the network but was done from a file on the native machine took a little under 4 hours.
The machine is a Quad Core Server running Windows Server 2008. People have wondered if there is a way to monitor progress. There actually is. You can watch the restore create the database files by going to the Program Data directory and finding the MYSQL subdirectory and then finding the subdirectory with your database name.
The files are gradually built in the directory and you can watch them build up. No small comfort when you have a production issue and you are wondering if the restore job is hung up or just taking a long time.