I'm running a django system over mysql in amazon's cloud, and the database default is innodb. But now I want to put a fulltext index on a couple of tables for searching, which evidently requires myisam.
The obvious solution is to just tell mysql to ALTER TABLE to myisam, but are there going to be any issues with that?
One that comes to mind is that I'll have to remember to do that any time I build a new version of the database, which should theoretically be rare, but there doesn't seem to be a way to tell django to please set the storage engine at the table level. I guess I could write a migration (we use south).
Any other things I might be missing? What could possibly go wrong?
Will the application notice? Probably not.
Will it cause problems? Only when things go wrong. MyISAM is not a transactional storage engine. If you change the data in a MyISAM table while inside of a transaction, then have to roll back changes, the changes in that table won't be rolled back. It's been a while since I tried to break it horribly, but I'm willing to wager that MySQL won't even issue a warning when this happens. This will lead to data consistency issues.
You should seriously consider external search software instead of a fulltext index, like ElasticSearch (integrates at the application level), or Sphinx (integrates at the MySQL level, though if you're using RDS instead of MySQL directly, I don't think you'll be able to use it).
the following may be of help:
use a myisam fulltext table to index back into your innodb tables for example:
Build your system using innodb:...
Any way to achieve fulltext-like search on InnoDB
Related
I read that MySQL fulltext search can cause table locking. It means people can't insert or update the table when it's being searched on.
I read that there are many search servers (Lucence and Sphinx) can do it without table locking and even faster. It requires many configuration and hard to implement.
Is there any other way to use fulltext or some searching like that without using search service? I don't want to configure one more server other than MySQL.
Create an extra table which will be used only to perform FULLTEXT searches. In your code you have to ensure that all data and actions (create, update, delete) are properly replicated to this table. This solution is also handy if your data tables are running e.g. InnoDB engine.
Apache Lucene doesn't need many configuration and isn't hard to implement. Moreover, it's one of the most popular fulltext search engine, and allows the users to do very precise queries, like "to be or not to be", j?hn d?e, func*, etc.
I already did some database indexing with Lucene, so if you could be a bit more precise about which fields of which tables you wanna index, I can give you pieces of code which should do the trick.
I vote for Sphinxsearch anyway. It has one of APIs close to Mysql, easy to install and configure. Not so universal as Apache Lucene, but jet quick and very helpful in my projects.
I noticed that my database server supports the Memory database engine. I want to make a database I have already made running InnoDB run completely in memory for performance.
How do I do that? I explored PHPMyAdmin, and I can't find a "change engine" functionality.
Assuming you understand the consequences of using the MEMORY engine as mentioned in comments, and here, as well as some others you'll find by searching about (no transaction safety, locking issues, etc) - you can proceed as follows:
MEMORY tables are stored differently than InnoDB, so you'll need to use an export/import strategy. First dump each table separately to a file using SELECT * FROM tablename INTO OUTFILE 'table_filename'. Create the MEMORY database and recreate the tables you'll be using with this syntax: CREATE TABLE tablename (...) ENGINE = MEMORY;. You can then import your data using LOAD DATA INFILE 'table_filename' INTO TABLE tablename for each table.
It is also possible to place the MySQL data directory in a tmpfs in thus speeding up the database write and read calls. It might not be the most efficient way to do this but sometimes you can't just change the storage engine.
Here is my fstab entry for my MySQL data directory
none /opt/mysql/server-5.6/data tmpfs defaults,size=1000M,uid=999,gid=1000,mode=0700 0 0
You may also want to take a look at the innodb_flush_log_at_trx_commit=2 setting. Maybe this will speedup your MySQL sufficently.
innodb_flush_log_at_trx_commit changes the mysql disk flush behaviour. When set to 2 it will only flush the buffer every second. By default each insert will cause a flush and thus cause more IO load.
Memory Engine is not the solution you're looking for. You lose everything that you went to a database for in the first place (i.e. ACID).
Here are some better alternatives:
Don't use joins - very few large apps do this (i.e Google, Flickr, NetFlix), because it sucks for large sets of joins.
A LEFT [OUTER] JOIN can be faster than an equivalent subquery because
the server might be able to optimize it better—a fact that is not
specific to MySQL Server alone.
-The MySQL Manual
Make sure the columns you're querying against have indexes. Use EXPLAIN to confirm they are being used.
Use and increase your Query_Cache and memory space for your indexes to get them in memory and store frequent lookups.
Denormalize your schema, especially for simple joins (i.e. get fooId from barMap).
The last point is key. I used to love joins, but then had to run joins on a few tables with 100M+ rows. No good. Better off insert the data you're joining against into that target table (if it's not too much) and query against indexed columns and you'll get your query in a few ms.
I hope those help.
If your database is small enough (or if you add enough memory) your database will effectively run in memory since it your data will be cached after the first request.
Changing the database table definitions to use the memory engine is probably more complicated than you need.
If you have enough memory to load the tables into memory with the MEMORY engine, you have enough to tune the innodb settings to cache everything anyway.
"How do I do that? I explored PHPMyAdmin, and I can't find a "change engine" functionality."
In direct response to this part of your question, you can issue an ALTER TABLE tbl engine=InnoDB; and it'll recreate the table in the proper engine.
In place of the Memory storage engine, one can consider MySQL Cluster. It is said to give similar performance but to support disk-backed operation for durability. I've not tried it, but it looks promising (and been in development for a number of years).
You can find the official MySQL Cluster documentation here.
Additional thoughts :
Ramdisk - setting the temp drive MySQL uses as a RAM disk, very easy to set up.
memcache - memcache server is easy to set up, use it to store the results of your queries for X amount of time.
I'm currently doing research on the best ways to provide an advanced search for my php project.
I have narrowed it down to using FULLTEXT search rather than using LIKE to find matching strings in a table row. However, to do this it appears I need to sacrifice using the InnoDB engine which will make me lose the ACIDity of transactions and table relationships.
Is it really worth using the MYISAM mysql engine or are there better ways of providing search functionality.
Any pointers would be appreciated!
It really depends on the application... Using MyISAM for anything that needs referential integrity is an instant fail. At the same time, it's text search isn't all that efficient.
Basically, there are two ways to go. If you find you don't need true referential integrity, consider a NoSQL datastore. MongoDB is a great document store database.
If, on the other hand, you really need referential integrity, but also need fast, indexed full-text searching, you might do better to use Sphinx or Apache Solr to create an indexed cache for full-text search.
Either way, I consider MyISAM to be a legacy datastore. I wouldn't use it on a new project. YMMV.
MyISAM has several drawbacks - lack of transaction support, table-level locks which makes it very slow in heavy read+write load type. Another inconvenience of MyISAM tables - they are not crash safe so you can lost some data in case of unexpected shutdown or power loss on server. However MyISAM is very fast on some queries.
Regarding the FullText search I would suggest to use InnoDB + external search engine like Lucene or Sphinx so you could benefit from both safe and reliable storage engine and fast Full-text queries.
For quick start with InnoDB and Sphinx you can refer to http://astellar.com/2011/12/replacing-mysql-full-text-search-with-sphinx/
MySQL 5.6 supports FULLTEXT indexes with InnoDB (released Feb 2013). See:
http://dev.mysql.com/doc/refman/5.6/en/fulltext-search.html
I hope that if my question is out of context you guys will let me pass by.I am not good at all when it comes to DBMS and all i can till in my 7+ years of IT experience is able to write some basic queries (with help from Google).
I am working on my own project and i am all done with the design phase and very will clear what all table structure i need to have for the project.I am going to use MySQL for this and since project is with the capabilities of small CMS in itself which means a lot of data to be handled at the database size.
I am using TOAD for MySQL to create schema and other things for MySQL but while doing this i have few queries which i want to address in the beginning.
What should be the storage engine type for MySQL (default is InnoDB)
Any other parameters needs to taken care of storage Engine
what Character set and Collation need to be set at table level and for column level if any (Application will have to take care of internationalization )
being not much knowledgeable about the database aspect these are the question keep popping in my mind and any help in this regard will be appreciated.
You should stick with InnoDB. It has transactions and a row-level locking. For high (writing) concurrency the row-level locking allows multiple INSERTs/UPDATEs/DELETEs at the same time. MyIsam has table-locking, meaning only one modifying query at a time. If you're read-heavy, MyISAM may perform better though. SELECT COUNT(*) FROM table; for example is a statistical thing MyISAM can answer instantly, while InnoDB actually does a COUNT(). InnoDB lets you define connections between your tables (Foreign Key Constraints) to help you with your data integrity.
If you're interested in things like the above, I recommend reading High Performance MySQL from O'Reilly Media.
Regarding the Character Set you should make sure that all tables / columns use the same character set. Preferably Unicode (UTF8). Make sure your mysql client is also set up to use UTF8, to avoid (transparent) transliteration of your data. (In fact, make sure the whole app does UTF8 from storage to display).
The Collation is something that lets you define your text being UTF8, but in a, say, German, context. With that knowledge, MySQL will properly sort "fööbar" and "foobar" according to the rules of the German language. (sorting amongst other things). I usually use "utf8_general_ci". Have a look at some examples over at mysql: http://dev.mysql.com/doc/refman/5.0/en/charset-collation-effect.html
Probably you will get better answer, but quickly from my experiences:
InnoDB is good for creating reliable data-driven web application and MyISAM is good for performance. But InnoDB supports Foreign Keys, transactions and row-level-locking which makes you honest while designing tables. It means if you don't want to create a simple and experimental database, the solution is using InnoDB.
You should look at Foreign Keys, transactions and Row-level locking (Only InnoDB supports these features)
utf8_general_ci is the most popular one I know if you want to storage different languages' data. It won't let you down.
I hope this help.
I read the following article yesterday (http://blogs.sitepoint.com/2010/11/19/mysql-mistakes-php-developers/) and it wrote the following:
MySQL has a number of database engines but you’re most likely to encounter MyISAM and InnoDB.
MyISAM is used by default. However, unless you’re creating a very simple or experimental database, it’s almost certainly the wrong choice! MyISAM doesn’t support foreign key constraints or transactions which are essential for data integrity. In addition, the whole table is locked whenever a record is inserted or updated: it causes a detrimental effect on performance as usage grows.
The solution is simple: use InnoDB.
I've always used MyISAM because it was the default. What do you think?
If I were to upgrade to InnoDB, in phpMyAdmin, can I just edit each table and change it to innoDB or is there a more complex process to perform?
Thanks!
Yes, you can swap in and out engines like used underwear, if you like, without much trouble. Just change it in phpmyadmin.
But I wouldn't change it for the sake of changing it. Do you need to use foreign keys? Change it. Do you need row-level-locking instead of table-locking? Change it.
It's worth noting that there are good reasons to use MyISAM, too. Take a look at FULLTEXT indexing. You can't do that with InnoDB.
UPDATE
As of MySQL 5.6 FULLTEXT has been implemented for InnoDB tables as well. Here is the manual.
Sorry for bumping an old question, but this also helped me a lot to choose which engine, especially since MyISAM is faster for reads and my database tables had more read the write:
http://www.rackspace.com/knowledge_center/article/mysql-engines-myisam-vs-innodb