mysql innodb inner join with longtext very slow - mysql

Question

I migrated all MySQL tables of one project from MyISAM to InnoDB last week, in order to support transaction. I used the command of alter table for this.
Most works fine, however one particular query runs very very slow, and it always gives the error Incorrect key file for table '/tmp/#sql_xxxx_x.MYI
Later I narrowed down the problem into the inner join of 2 tables, the user table and agreement table. And the inner join took place between the foreign key field of user (i.e. agreement_id) and primary key field of agreement (i.e. id).
The user table has only 50,000 rows of data, and the agreement table has, well, one single row. And we have set up the index for the agreement_id of user.
In any case, this seems to be a very light-weighted query, but it turns out to be the whole bottle neck.
Here is the full schema of agreement:
CREATE TABLE IF NOT EXISTS `agreement` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`remark` varchar(200) NOT NULL,
`content` longtext NOT NULL,
`is_active` tinyint(1) NOT NULL,
`date_stamp` datetime NOT NULL,
PRIMARY KEY (`id`)
) DEFAULT CHARSET=utf8 AUTO_INCREMENT=2 ;
One thing I doubt about is the longtext field of remark inside the agreement table, but we did NOT use the field for the inner join, in fact the query is slow even if we did NOT select remark in the query result.
finally, we converted the table of agreement from innoDB back to MyISAM, than everything becomes normal. And the query is finished in less than 1 second.
Now, my question is what actually is going on here? Does that mean once an innoDB table contains any text field, then the table could not be used for inner join?
I wish I could know the real reason so that I could avoid the same problems in the future.
thanks a lot.

Answer 1

This is a famous and tricky one. The most likely cause is that you're out of space in /tmp.
Here is a link I keep in my bookmarks that may help you: http://www.mysqlperformancetuning.com/a-fix-for-incorrect-key-file-for-table-mysql
In my experience, limited though it is, the primary reason for seeing
this error message is because your tmpdir has run out of space. Like
me you'll check how much free space you have: 1Gb, 2Gb, 4Gb. It may
not be enough. And here's why: MySQL can create temporary tables
bigger than that in a matter of seconds, quickly filling up any free
space. Depending on the nature of the query and the size of the
database naturally.
You may also try a REPAIR on your table but to me it is as usefull as breakdancing :/

Answer 2

InnoDB has it's own settings of buffer sizes etc. Check that out and if you can adjust it, go ahead. Just for test try to double them, if it helps, you may want to optimize it. It can make a big difference.
Some links that may help:
http://www.mysqlperformanceblog.com/2007/11/03/choosing-innodb_buffer_pool_size/
http://dev.mysql.com/doc/refman/5.5/en/innodb-buffer-pool.html

Answer 3

Maybe problem here is remark field defined as varchar(200)? Remember that temporary and memory tables stores varchar with fixed length. So 50k rows with varchar(200) can consume a lot of memory even if they are all empty.
If this is the problem then you can try one of several things:
If only few rows have value in column then create varchar(200) column with NULL allowed and always use NULL value instead of empty string
Change varchar(200) to text (there is of course drawback - it will always use temporary table on disk)
Maybe you don't need 200 characters? Try to use smaller VARCHAR size
Try to adjust tmp_table_size and max_heap_table_size, so you could handle larger temporary tables in memory http://dev.mysql.com/doc/refman/5.1/en/internal-temporary-tables.html
Use percona server as they support dynamic row format for memory tables http://www.mysqlperformanceblog.com/2011/09/06/dynamic-row-format-for-memory-tables/

Answer 4

What is the purpose of your query? I see from your comment that you only list the user information and nothing from agreement leading me to believe you are looking for users that has an agreement?
Since you are converting between engines it leads me to think you are doing cleanup before adding constraints. If so, consider a left join from the user table instead like:
select user.* from user left join agreement on user.agreement_id = agreement.id where user.agreement_id != 0;
If it's not cleanup, but simply looking for users with an agreement we make it simple
select user.* from user where user.agreement_id != 0;
If the purpose is something else, consider adding an index on user.agreement_id since an inner join may need it for speed. Let us know the real purpose and you may get better help.