I'm doing a search on a table (few inner joins) which takes max 5 seconds to run (7.5 million rows). It's a MyISAM table and I'm not using full-text indexing on it as I found there to be no difference in speed when using MATCH AGAINST and a normal "like" statement in this case from what I can see.
I'm now "suffering" from locked tables and queries running for several minutes before they complete because of it.
Would it benefit me at all to try and switch the engine to InnoDB? Or does that only help if I need to insert or update rows... not just select them? This whole table-locking thing is busy grinding my balls...
InnoDB supports row-level locking instead of table-level locking... so that should alleviate your problem (although I'm not sure it will remove it entirely).
Your best bet would be to use a dedicated search system (like Sphinx, Lucene, or Solr)
The difference between row-level and table-level locking is only important for insert and update queries. If you're mostly do selects (so the inserts/updates do not happen too often to lock the table) the difference will not be all that much (even though in recent benchmarks InnoDB seems to be outperforming MyISAM).
Other ways you could think about is to reorganise your data structure, perhaps including additional lookup table with 'tags' or 'keywords'. Implementing more efficient full text engine as suggested by webdestroya.
Last but not least, I'm also surprised that you got similar results with FULL TEXT vs LIKE. This could happen if the fields you're searching are not really wide, in which case maybe a stndard B-TREE index with = search would be enough?
Related
I have a Myisam table with composite unique key of 2 columns and 90 million data. Now we are facing memory and load issues and after going through the web I am planning to include partitioning and changing this table to Innodb for better performance. But I have following concerns:
Changing to innodb will have a huge downtime, Is it possible to minimize the downtime?
Most of the select query are on a particular column of the key on which I am planning to have the hash partitioning, how much it will effect the query on another key column?
Will these changes improve the performance to the extent mentioned theoretically? Is there any better solution for such cases. Any suggestion or experience can be helpful.
My queries are simple like
Select * from Table where Col1= "Value"
Select * from Table where Col1="Value" and Col2 IN (V1,V2,V3)
Inserts are very frequently.
InnoDB will probably help some. Conversion to InnoDB comes with some issues, as I state in My conversion blog.
Partitioning, per se, buys no performance gain. My partitioning blog lists 4 cases where you can, with design changes, gain performance.
Regardless of the Engine, your two queries will both benefit from
INDEX(col1, col2)
No form of partitioning will help. HASH partitioning is especially useless.
Conversion to InnoDB will take a lot of downtime, unless pt-online-schema-change will work for your case. Research it.
Also read my answers and comments on
Can i set up Mysql to auto-partition?
for more specifics.
It may be that adding that index is the main performance gain. But you have to do a lengthy ALTER to get it. MyISAM does not have ALGORITHM=INPLACE.
Innodb (about perfomance we are talking now) have sense only when there are alot of inserts and updates to your table, because of row-locking table.
If the most queries on your table are SELECTs then MyIsam will be faster.
Advice: put in my.cnf key_buffer_size equal to 25% of your free RAM.
If inserts on your database are very frequent, you will likely gain performance by switching to innodb, which won't lock down entire tables to insert, allowing other clients to select data concurrently.
Regarding question #1, if you are worried about downtime, I'd suggest you find a parallel dump/load solution for migrating your data to innodb. if you simply run an ALTER statement on your tables, this is a single threaded operation which will be much slower.
Regarding #2, you'd have to post a schema along with your partitioning strategy and the queries you are worried about.
A Database already has up to 25-30 tables and all are MyISAM. Most of these tables are related to each other meaning a lot of queries use joins on IDs and retrieve data.
One of the tables contain 7-10 Million records and it becomes slow if i want to perform a search or update or even retrieval of all data. Now i proposed a solution to my boss saying that converting tables into InnoDB might give better performance.
I also explained the benefits of InnoDB:
Since we anyways join multiple tables on keys and they are related, it will be better to use foreign keys and have relational database which will avoid Orphan Rows. I found around 10-15k orphan rows in one of the big tables and had to manually remove them.
Support for transactions, we perform big updates from time to time and if one of them fails on the way we have to replace the entire table with the backed-up one and run the update again to make sure that all queries were executed. With InnoDB we can revert back any changes from query 1 if query 2 fails.
Now the response i got from my boss is that I need to prove that InnoDB will run faster than MyISAM. My question is, wont above 2 things improve the speed of the application itself by eliminating orphan rows?
In general is MyISAM faster than InnoDB?
Note: using MySQL 5.5
You should also mention to your boss probably the biggest benefit you get from InnoDB for large tables with both read/write load - You get row-level locking rather than table-level locking. This can be a great performance benefit for the application in cases where you see a lot of waits for table locks to be released.
Of course the best way to convince your boss is to prove it. Make copies of your large table and place on a testing database. Make one version of data in MyISAM and one in InnoDB. Then run load testing against it with a load mix that approximates your current DB read/write activity. Find out for yourself if it is better.
Just updated for your comment that you are on 5.5. With 5.5 it is a no brainer to use InnoDB. MyISAM engine basically has seen no improvement over the last several years and development effort has been around InnoDB. InnoDB is THE MySQL engine of choice going forward.
I've read various threads about which is better between InnoDB and MyISAM. It seems that the debates are to use or the other. Is it not possible to use both, depending on the table?
What would be the disadvantages in doing this? As far as I can tell, the engine can be set during the CREATE TABLE command. Therefore, certain tables which are often read can be set to MyISAM, but tables that need transaction support can use InnoDB.
You can have both MyISAM and InnoDB tables in the same database. What you'll find though, when having very large tables is, MyISAM would cause table-lock issues. What this ultimately does is, it locks the whole table for a single transaction, which is very bad if you have many users using your website. e.g If you have a user searching for something on your site and the query takes minutes to complete, no other users could use your site during that period because the whole table is locked.
InnoDB on the other hand uses row-level locking, meaning, that row is the only row locked from the table during a transaction. InnoDB can be slower at searches because it doesn't offer full text search like MyISAM, but that isn't a big problem when you compare it to table-level locking of MyISAM. If you use InnoDB, like many large sites, then you could use a server side search engine like Sphinx for full text searches. Or you could even use a MyISAM table to do the searching like f00 suggested. I'd personally recommended InnoDB mainly because of the row-level locking feature, but also because you can implement full text searching in many other ways.
Basically, if you have a message board application with lots of selects, inserts as well as updates, InnoDB is probably the generally appropriate choice.
But if you're not building something like that (or any other thing with registered users) and your working mostly with static content (or more reads than writes), then you could use MyISAM.
Yes indeed you may use both in the same database, you may choose for each table separately.
In short, InnoDB is good if you are working on something that needs a reliable database that can handles a lot of INSERT and UPDATE instructions.
and, MyISAM is good if you needs a database that will mostly be taking a lot of read (SELECT) instructions rather than write (INSERT and UPDATES), considering its drawback on the table-lock thing.
you may want to check out;
Pros and Cons of InnoDB
Pros and Cons of MyISAM
You don't choose InnoDB or MyISAM on a database level, but instead on a table level. So within the one database you could have some tables running the InnoDB engine and some running MyISAM. As you pointed out, you could choose to use InnoDB on the tables that require transactions etc, and MyISAM where you need other features such as fulltext searching.
I have a huge file (~26 MB) with around 200 columns & 30000 records. I want to import it into a database (InnoDB Engine). I wont't be updating or deleting records ever. ALthough I will be querying a lot of records from the table with high complexity in where clause. Which table engine should i prefer for faster query response? Will it really make a lot of difference?
PS: All my other tables use InnoDB.
Also How can I avoid manually creating a table with 200 columns and specifying the datatype for each of them. Most of the columns are float and few are varchar and date.
Usually the answer to "which is faster, ISAM or innodb" would be ISAM
But for best performance with a table which has very few updates you might want to have a look at Infobright's columnar db (which is integrated into mysql).
However with only 30k rows you'll not see a significant difference between innodb, isam and infobright.
OTOH, you really should have a long hard look at whether you really need 200 columns in a single table. I suspect that's not the case - and the schema is far more important in determining performance than the storage engine.
when dealing with large amounts of data innodb fares better then myisam.,
http://www.mysqlperformanceblog.com/2007/01/08/innodb-vs-myisam-vs-falcon-benchmarks-part-1/
and
http://www.cftopper.com/index.cfm?blogpostid=84
James Day, a MySQL Support Engineer and Wikipedia engineer recommends that people use InnoDB all the time unless for some reason if becomes apparent that you need MyISAM:
"I'd go with InnoDB until it's been proved that it's unsuitable. The first reason is reliability. Get a crash with MyISAM and you have the unreliable and slow, related to table size, table repair process. Same thing with InnoDB and you instead get the fixed time, fast and reliable log apply/rollback process. As the data set gets bigger, this matters more and more, as it does if you want to do things like sleep instead of being woken up in the middle of the night to fix a crashed table.
For reliability and performance, we use InnoDB for almost everything at Wikipedia - we just can't afford the downtime implied by MyISAM use and check table for 400GB of data when we get a crash."
Consider this scenario, my database table has 300000 rows and has a fulltext index. Whenever a search is done it locks the database and doesn't allow anyone else to login to the portal.
Any advice on how to get things sorted out here will be really appreciable.
Does logging on perform a write to the table? eg. a 'last visit' time?
If so you may expect behaviour something like this because MyISAM writes do a lock over the entire table. Usually this is avoided by not using noddy MyISAM and going to InnoDB instead, which has row-level locking (amongst other desirable database features).
The problem, of course, is that you only get fulltext search with MyISAM.
So you'll need to split your tables up. If you can keep the read-heavy and fulltext stuff in a different table to the stuff that needs writing (but linked using the same primary key), you can probably make it so that the two operations don't affect each other.
Better, migrate the bulk of the table to InnoDB, leaving only a fulltext field in MyISAM. Everything except fulltext searches can then steer clear of the MyISAM table, and use only the InnoDB table which exhibits much better locking performance. Personally, I now tend to store everything in the InnoDB table, including the text, and store a second copy of the text in the MyISAM table purely for fulltext searchbait purposes; this simplifies queries and code and brings the advantages of InnoDB's consistency to the text content, and I also use it to process the searchbait to get stemming and other features MySQL's fulltext doesn't normally support. But it does mean you have to spend a lot more space on storage.
You can also improve matters by cutting down number of writes. For example if it is a 'last visit' timestamp you're writing, you can avoid writing that unless, say, a minute has passed between the previous time and now, on the basis that no-one needs to know the exact second someone last accessed the site.
If you use an external search engine or MySQL search plug-ins Lucene or Sphinx, they should be able to read and index without locking the table. They store a local version of the indexed records, so they don't have to read the table very often, and never need to write to it.