mysql query taking too long - mysql

Question

I am new to advanced queries so I likely have something conceptually wrong because when the database has over 1 million records I get this response rom my query...
ERROR 2013: Lost connection to MySQL server during query
Yes! It actually takes so long that it pukes before it finishes.
My query is this...
SELECT users.username,
table_1.field_abc, table_1.field_def,
table_2.field_ghi, table_2.field_jkl
FROM users
LEFT JOIN table_1 ON table_1.username = users.username
LEFT JOIN table_2 ON table_2.username = users.username
WHERE
table_1.field_abc REGEXP "(spork|yellow)" OR
table_1.field_def REGEXP "(spork|yellow)" OR
table_2.field_ghi REGEXP "(spork|yellow)" OR
table_2.field_jkl REGEXP "(spork|yellow)"
GROUP BY users.username
ORDER BY
(
( CASE WHEN table_1.field_abc LIKE "%spork%" THEN 1 ELSE 0 END ) +
( CASE WHEN table_1.field_abc LIKE "%yellow%" THEN 1 ELSE 0 END ) +
( CASE WHEN table_1.field_def LIKE "%spork%" THEN 1 ELSE 0 END ) +
( CASE WHEN table_1.field_def LIKE "%yellow%" THEN 1 ELSE 0 END ) +
( CASE WHEN table_2.field_ghi LIKE "%spork%" THEN 1 ELSE 0 END ) +
( CASE WHEN table_2.field_ghi LIKE "%yellow%" THEN 1 ELSE 0 END ) +
( CASE WHEN table_2.field_jkl LIKE "%spork%" THEN 1 ELSE 0 END ) +
( CASE WHEN table_2.field_jkl LIKE "%yellow%" THEN 1 ELSE 0 END )
)DESC;
I posted a sample dataset (with only a few records) at http://sqlfiddle.com/#!2/cbbda/28
The sample at sqlfiddle runs quick because there are only a few records but I tried duplicating records on my own server and the query ran quick with only a few records and extremely slow after I added a million records.
Is there any possible way to get my results quick?

Answer 1

Well folks... With your help we have a solution... See... http://sqlfiddle.com/#!2/fcfbd/5
BUT I DO STILL HAVE A QUESTION...
I altered the table to add the indexes...
ALTER TABLE `users` ADD FULLTEXT ( `username` );
ALTER TABLE `table_1` ADD FULLTEXT ( `field_abc`,`field_def` );
ALTER TABLE `table_2` ADD FULLTEXT ( `field_ghi`,`field_jkl` );
I then took the advice of #Barmar and changed the code to this...
SELECT users.username,
table_1.field_abc, table_1.field_def,
table_2.field_ghi, table_2.field_jkl
FROM users
LEFT JOIN table_1 ON table_1.username = users.username
LEFT JOIN table_2 ON table_2.username = users.username
WHERE
MATCH(table_1.field_abc,table_1.field_def,table_2.field_ghi,table_2.field_jkl)
AGAINST ("spork yellow" IN BOOLEAN MODE)
GROUP BY users.username
ORDER BY
(
( CASE WHEN MATCH(table_1.field_abc) AGAINST ("spork" IN BOOLEAN MODE) THEN 1 ELSE 0 END ) +
( CASE WHEN MATCH(table_1.field_abc) AGAINST ("yellow" IN BOOLEAN MODE) THEN 1 ELSE 0 END ) +
( CASE WHEN MATCH(table_1.field_def) AGAINST ("spork" IN BOOLEAN MODE) THEN 1 ELSE 0 END ) +
( CASE WHEN MATCH(table_1.field_def) AGAINST ("yellow" IN BOOLEAN MODE) THEN 1 ELSE 0 END ) +
( CASE WHEN MATCH(table_2.field_ghi) AGAINST ("spork" IN BOOLEAN MODE) THEN 1 ELSE 0 END ) +
( CASE WHEN MATCH(table_2.field_ghi) AGAINST ("yellow" IN BOOLEAN MODE) THEN 1 ELSE 0 END ) +
( CASE WHEN MATCH(table_2.field_ghi) AGAINST ("spork" IN BOOLEAN MODE) THEN 1 ELSE 0 END ) +
( CASE WHEN MATCH(table_2.field_ghi) AGAINST ("yellow" IN BOOLEAN MODE) THEN 1 ELSE 0 END )
)DESC;
With over 1,000,000 records in my real database, I got my result in 6.5027 seconds. That is A LOT better than... well, taking so long that it puked!
My only question now is... Why does it only work with IN BOOLEAN MODE and not the other 2 options mentioned at http://dev.mysql.com/doc/refman/5.0/en/fulltext-search.html#function_match or http://dev.mysql.com/doc/refman/5.5/en/fulltext-search.html?

Answer 2

I don't think so - with this table as-is, I doubt you'll make that run fast with all of those LIKEs on them. Those have to run a ridiculous number of times.
If those values are fixed, then you can add new columns to the table called abc_like_yellow and abc_like_spork, etc., and populate those values one time, then you can easily query off of that column.
But if you're trying to do this dynamically, you might be out of luck.

Answer 3

Since we're joining on username it is likely that an index on this column will speed things up.
Also, are you able to use an inner join as opposed to a left join? This can also speed up the query to quite a large extent.
And finally, if necessary, the ordering can be done in memory as opposed to asking the database to do it (ie order the result set after it is returned).

Answer 4

I was using my first solution but found that it gave some false positives that I could not figure out so I came up with this...
(SELECT username, MATCH(field_abc,field_def) AGAINST ("spork yellow" IN BOOLEAN MODE) AS score FROM table_1 HAVING score>0)
UNION ALL
(SELECT username, MATCH(field_ghi,field_jkl) AGAINST ("spork yellow" IN BOOLEAN MODE) AS score FROM table_2 HAVING score >0)
Since each record was returned separately and I can't use GROUP BY I added this PHP code after my query finished:
while($row = mysql_fetch_array($result) )
{
if( in_array($row['username'],$usernames) )
{
$usernames_count[$row['username']] += $row['score'];
}
else
{
array_push($usernames,$row['username']);
$usernames_count[$row['username']]=$row['score'];
}
}
arsort($usernames_count); // Sort the results high->low
foreach($usernames_count as $key=>$value)
{
echo "Username: ".$key." had a score of ".$value." in the search results<br/>";
}
It now seems so simple compared to the other attempts I made.

Answer 5

When your server has to scan through millions of entries, it simply may not be powerful enough to process the query quickly.
In general, to improve the speed of your website, you could try CloudFlare
If you are specifically trying to speed up your SQL, Google Cloud SQL may be able to help. Google's powerful servers are designed to scan through billions of SQL entries, for example when a Google search is performed.
As long as there are no errors being returned, the above two services will help to dramatically speed up your query time.
I hope I could help!
VCNinc

Answer 6

If you have access to SQL Server, highlight your complete query in SQL server, and click + L
This will show the query execution plan. Optimize the query based on these results;
if for example you see table scans then an index may assist.
Write queries that do not use the term distinct.
Do not order results if the order is unimportant.
In your sample the complicated last set of order-by is very expensive.
Rather follow these steps:
Pull the core information into a temporary table, with 9 extra columns (type int, intially set to 0)
after populating the core data, update each of the 8 columns based on the 0 or 1 criteria
update the last column as the sum of the other 8 columns
retrieve info from the table , with only a single "order-by" based on column 9.
In my experience this approach only takes 20% of the time compared to doing the order-by in-house.