I have a mysql server with production database and a staging database. I am using innodb for the production tables, but I don't need the features for staging which just moves data around. So I decided to switch those tables to myisam expecting an increase in insert performance.
The results were horrible. It looked more like inserting into an innodb table without autocommit disabled. If I keep the table innodb and turn off autocommit during the insert I can get several thousand inserts per second. As soon as I change the table to myisam I'm getting maybe a couple dozen inserts per second.
I thought maybe it was because I'm getting the data via our legacy backend using SSIS but that doesn't appear to be the issue. Using SSIS and going from our production db to the staging db (mysql to mysql) I still see the same results... innodb(no autocommit) far out performs myisam.
This makes no sense to me. If nothing else, in my experience, myisam should at least be comparable, hopefully even better.
Is there anything obvious I am overlooking? I haven't included specifics in hopes that it is something in general I am missing.
EDIT:
This appears to be related to SSIS and the ODBC Destination component. I'm using an ODBC Source that has my select statement and the output goes to the ODBC Destination which is table on the same server, but a different DB. Since the DBs are on the same server I ran, in SqlYog, an INSERT using the same SELECT query as the ODBC Source and it finished in a couple seconds. I will see if I can find a solution.
Related
I'm trying to understand an issue I am having with a MySQL 5.5 server.
This server hosts a number of databases. Each day at a certain time a process runs a series of inserts into TWO tables within this database. This process lasts from 5 to 15 minutes depending on the amount of rows being inserted.
This process runs perfectly. But it has a very unexpected side effect. All other inserts and update's running on tables unrelated to the two being inserted to just sit and wait until the process has stopped. Reads and writes outside of this database work just fine and SELECT statements too are fine.
So how is it possible for a single table to block the rest of a database but not the entire server (due to loading)?
A bit of background:-
Tables being inserted to are MyISAM with 10 - 20 million rows.
MySQL is Percona V5.5 and is serving one slave both running on
Debian.
No explicit locking is called for by the process inserting the
records.
None of the Insert statements do not select data from any other
table. They are also INSERT IGNORE statements.
ADDITIONAL INFO:
While this is happening there are no LOCK table entries in PROCESS LIST and the processor inserting the records causing this problem does NOT issue any table locks.
I've already investigated the usual causes of table locking and I think I've rules them out. This behaviour is either something to do with how MySQL works, a quirk of having large database files or possibly even something to do with the OS/File System.
After a few weeks of trying things I eventually found this: Yoshinori Matsunobu Blog - MyISAM and Disk IO Scheduler
Yoshinori demonstrates that changing the scheduler queue to 100000 (from the default 128) dramatically improves the throughput of MyISAM on most schedulers.
After making this change to my system there were no longer any dramatic instances of database hang on MyISAM tables while this process was running. There was slight slowdown as to be expected with the volume of data but the system remained stable.
Anyone experiencing performance issues with MyISAM should read Yoshinori's blog entry and consider this fix.
My database is periodicly being "deleted" by an automated command from the server (because the table is too big). What happens is that all data in a certain table becomes unaccessable with e.g. select. But if I do a "repair" on the table, all data comes back. I would like to stop this nonesense, but I can't find the command that does this. Any help?
Edit: I should note that the DB is on an external machine that I do not have access to.
I have now tried to do a "select" when the db was in this curious state. The table says it has 0 entries, but take 2.5 gb of storrage space. When I selected all I got one tuple, no errors.
Its likely your DB is becoming corrupt somehow. There's no command that does that (I hope).
Do yourself a favor and alter each and every one of your tables so they use the InnoDB engine instead of MyISAM. It's still be MySQL, but it'll be a lot less prone to data corruption.
And if changing DB altogether is an option, look into using PostgreSQL instead.
We have an update process which currently takes over an hour and means that our DB is unusable during this period.
If I setup up replication would this solve the problem or would the replicated DB suffer from exactly the same problem that the tables would be locked during the update?
Is it possible to have the replicated DB prioritize reading over updating?
Thanks,
D
I suspect that with replication you're just going to be dupolicating the issue (unless most of the time is spent in CPU and only results in a couple of records being updated).
Without knowing a lot more about the scema, distribution and size of data and the update process its impossible to say how best to resolve the problem - but you might get some mileage out of using innodb instead of C-ISAM and making sure that the update is implemented as a number of discrete steps (e.g. using stored procuedures) rather than a single DML statement.
MySQL gives you the ability to run queries delaye. Example: "INSERT DELAYED INTO...", this will cause the query to only be executed when MYSQL has time to take the query.
Based on your input, it sounds like you are using MyISAM tables, MyISAM only support table-wide locking. That means that a single update will lock the whole database table until the query is completed. InnoDB on the other hand uses row locking, which will not cause SELECT queries to wait(hang) for updates to complete.
So you have the best chances of a better sysadmin life if you change to InnoDB :)
When it comes to replication it is pretty normal to seperate updates and selects to two different MySQL servers, and that does tend to work very well. But if you are using MyISAM tables and does a lot of updates, the locking issue itself will still be there.
So my 2 cents: First get rid of MyISAM, then consider replication or a better scaled MySQL server if the problem still exists. (The key for good performance in MySQL is to have at least the size of all indexes across all databases as physical RAM)
I have three large MySQL tables. They are approaching 2 million records. Two of the tables are InnoDB and are currently around 500 MB in size. The other table is MyISAM and is about 2.5 GB.
We run an import script from FileMaker to insert and update records in these tables but lately it has become very slow - only inserting a few hundred records per hour.
What can I do to increase performance to make inserts and updates happen faster?
For INSERT it could have to do with the indexes you have defined on the tables (they have to be updated after each INSERT). Could you post more information about them? And are there triggers set on the tables?
For UPDATE it is a different story, it could be that not the record update is slow but finding the record is slow. Could you try to change the UPDATE into a SELECT and see if it is still slow? If yes, then you should investigate your indexes.
For the Innodb table, if it's an acceptable risk, I'd consider changing the innodb_flush_log_at_trx_commit level. Some more details in this blog post, along with some more Innodb tuning pointers.
For both engines, batching INSERTs together can speed things up to a point. See doc.
What version of MySQL are you running? There have been many improvements with the new InnoDB "Plugin" engine and concurrency of operations on servers with multiple processors.
Is the query slow when executed on MySQL from the command line?
If you're using the Execute SQL Script step from FileMaker, that connects and disconnects after every call, causing major slowdowns when executing large numbers of queries. We've had clients switch to our JDBC plugin (self-promotion disclaimer here) to avoid this, resulting in major speedups.
It turns out the reason for the slowness was from the FileMaker side of things. Exporting the FileMaker records to a CSV and running INSERT/UPDATE commands resulted in very fast execution.
sometimes i get an error like "table is marked as corrupt and shld be repaired". that DB (tables) is using MyISAM. recently that keeps happening. what could be the causes? most recently i am executing a batch insert
INSERT INTO table (..., ..., ...) VALUES (...), (...), (...) ...
and it just hung. or took very long to complete it seems hung to me. the next day, when i checked the table was marked as corrupt again. when i try to use mysqlcheck -r it said all tables OK when it reached that "corrupt" table it hung there again...
so, what can i do to prevent this. and what could be the causes. the DB is hosted 3rd party, how can i debug this?
is InnoDB a more reliable engine to use? i heard MyISAM is faster but others say InnoDB can be fast also but it takes abit more to optimize it. can i conclude that InnoDB is something more reliable but abit slower overall even with optimization?
If your tables get corrupt, you can use the repair table command to fix them:
REPAIR TABLE table;
If you run myisamchk while the server is still running (and inserts/selects are hitting the table), it could be what is corrupting your tables. Most of the corruption issues I run into are when trying to do things outside the server (copying the files, etc) while it is still running.
InnoDB is slower for read only databases, because it has features (ACID compliant, row level locking) that MyISAM leaves out. However, if you are doing a mixture of reads and writes, depending on the mixture, then InnoDB can offer serious performance improvements, because it doesn't have to lock the entire table to do a write. You also don't run into corruption issues.
Go with InnoDB.
ok, so the problem was the company's db exceeded the storage space allowed by the hosting company. so apparently noone told the company they exceeded the usage ... lousy host i guess.
btw, theres no way mysql could have known abt this?