Trying to optimize a query and properly index tables - mysql

Question

I wanted to simulate large number of data in a database and test how my query would perform under such conditions. I was not surprised when query turned out to be slow. So here I am seeking advice on how I could better index my tables and improve my queries.
Before I post tables's sql and the query I use, Let me explain what is what. I have a user's table, which is populated by 100 000 records. Most of the columns in it are enum type, like hair color, looking_for, etc... The first query I have is generated when a search is done. The query would consist of a where statement where some or all column values are searched for and only ids are retrieved limited by 20.
Then I have 3 more tables that hold about 50 - 1000 records per each user, so numbers could really grow. these tables hold information on who visited who's profile, who marked who as a favorite, who blocked who, and messaging table. My goal is to retrieve 20 records that match the search criteria, but also determine if I (user who's browsing) have:
blocked them
favorited them
was favorited by them
have unread messages from them
have sent or received any messages from them
For this I tried using both joins and subqueries, but the problem is that second query that retrieves users and data listed above is still slow. I think I need a better index and possibly a better queries. here is what I have right now, tables definitions first and 2 queries in the end. First des sarch and determiens IDs, second uses ids from first query to retrieve data. I hope you guys can help me create better indexes and optimize the query.
CREATE TABLE user (id BIGINT AUTO_INCREMENT, dname VARCHAR(255) NOT NULL, email VARCHAR(255) NOT NULL UNIQUE, email_code VARCHAR(255), email_confirmed TINYINT(1) DEFAULT '0', password VARCHAR(255) NOT NULL, gender ENUM('male', 'female'), description TEXT, dob DATE, height MEDIUMINT, looks ENUM('thin', 'average', 'athletic', 'heavy'), looking_for ENUM('marriage', 'dating', 'friends'), looking_for_age1 BIGINT, looking_for_age2 BIGINT, color_hair ENUM('black', 'brown', 'blond', 'red'), color_eyes ENUM('black', 'brown', 'blue', 'green', 'grey'), marital_status ENUM('single', 'married', 'divorced', 'widowed'), smokes ENUM('no', 'yes', 'sometimes'), drinks ENUM('no', 'yes', 'sometimes'), has_children ENUM('no', 'yes'), wants_children ENUM('no', 'yes'), education ENUM('school', 'college', 'university', 'masters', 'phd'), occupation ENUM('no', 'yes'), country_id BIGINT, city_id BIGINT, lastlogin_at DATETIME, deleted_at DATETIME, created_at DATETIME NOT NULL, updated_at DATETIME NOT NULL, INDEX country_id_idx (country_id), INDEX city_id_idx (city_id), INDEX image_id_idx (image_id), PRIMARY KEY(id)) DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci ENGINE = INNODB;
CREATE TABLE block (id BIGINT AUTO_INCREMENT, blocker_id BIGINT, blocked_id BIGINT, created_at DATETIME NOT NULL, updated_at DATETIME NOT NULL, INDEX blocker_id_idx (blocker_id), INDEX blocked_id_idx (blocked_id), PRIMARY KEY(id)) DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci ENGINE = INNODB;
CREATE TABLE city (id BIGINT AUTO_INCREMENT, name_eng VARCHAR(30), name_geo VARCHAR(30), name_geo_shi VARCHAR(30), name_geo_is VARCHAR(30), country_id BIGINT NOT NULL, active TINYINT(1) DEFAULT '0', INDEX country_id_idx (country_id), PRIMARY KEY(id)) DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci ENGINE = INNODB;
CREATE TABLE country (id BIGINT AUTO_INCREMENT, code VARCHAR(2), name_eng VARCHAR(30), name_geo VARCHAR(30), name_geo_shi VARCHAR(30), name_geo_is VARCHAR(30), active TINYINT(1) DEFAULT '1', PRIMARY KEY(id)) DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci ENGINE = INNODB;
CREATE TABLE favorite (id BIGINT AUTO_INCREMENT, favoriter_id BIGINT, favorited_id BIGINT, created_at DATETIME NOT NULL, updated_at DATETIME NOT NULL, INDEX favoriter_id_idx (favoriter_id), INDEX favorited_id_idx (favorited_id), PRIMARY KEY(id)) DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci ENGINE = INNODB;
CREATE TABLE message (id BIGINT AUTO_INCREMENT, body TEXT, sender_id BIGINT, receiver_id BIGINT, read_at DATETIME, created_at DATETIME NOT NULL, updated_at DATETIME NOT NULL, INDEX sender_id_idx (sender_id), INDEX receiver_id_idx (receiver_id), PRIMARY KEY(id)) DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci ENGINE = INNODB;
CREATE TABLE visitor (id BIGINT AUTO_INCREMENT, visitor_id BIGINT, visited_id BIGINT, created_at DATETIME NOT NULL, updated_at DATETIME NOT NULL, INDEX visitor_id_idx (visitor_id), INDEX visited_id_idx (visited_id), PRIMARY KEY(id)) DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci ENGINE = INNODB;
SELECT s.id AS s__id FROM user s WHERE (s.gender = 'female' AND s.marital_status = 'single' AND s.smokes = 'no' AND s.deleted_at IS NULL) LIMIT 20
SELECT s.id AS s__id, s.dname AS s__dname, s.gender AS s__gender, s.height AS s__height, s.dob AS s__dob, s3.id AS s3__id, s3.code AS s3__code, s3.name_geo AS s3__name_geo, s4.id AS s4__id, s4.name_geo AS s4__name_geo, s5.id AS s5__id, s6.id AS s6__id, s7.id AS s7__id, s8.id AS s8__id, s9.id AS s9__id FROM user s LEFT JOIN country s3 ON s.country_id = s3.id LEFT JOIN city s4 ON s.city_id = s4.id LEFT JOIN block s5 ON ((s.id = s5.blocked_id AND s5.blocker_id = '1')) LEFT JOIN favorite s6 ON ((s.id = s6.favorited_id AND s6.favoriter_id = '1')) LEFT JOIN favorite s7 ON ((s.id = s7.favoriter_id AND s7.favorited_id = '1')) LEFT JOIN message s8 ON ((s.id = s8.sender_id AND s8.receiver_id = '1' AND s8.read_at IS NULL)) LEFT JOIN message s9 ON (((s.id = s9.sender_id AND s9.receiver_id = '1') OR (s.id = s9.receiver_id AND s9.sender_id = '1'))) WHERE (s.id IN ('22', '36', '53', '105', '152', '156', '169', '182', '186', '192', '201', '215', '252', '287', '288', '321', '330', '351', '366', '399')) GROUP BY s.id ORDER BY s.id
Here are the results of EXPLAIN of the 2 queries above:
First:
1 SIMPLE s ALL NULL NULL NULL NULL 100420 Using Where
Second:
1 SIMPLE s range PRIMARY PRIMARY 8 NULL 20 Using where; Using temporary; Using filesort
1 SIMPLE s2 eq_ref PRIMARY PRIMARY 8 sagule.s.image_id 1 Using index
1 SIMPLE s3 eq_ref PRIMARY PRIMARY 8 sagule.s.country_id 1
1 SIMPLE s4 eq_ref PRIMARY PRIMARY 8 sagule.s.city_id 1
1 SIMPLE s5 ref blocker_id_idx,blocked_id_idx blocked_id_idx 9 sagule.s.id 5
1 SIMPLE s6 ref favoriter_id_idx,favorited_id_idx favorited_id_idx 9 sagule.s.id 6
1 SIMPLE s7 ref favoriter_id_idx,favorited_id_idx favoriter_id_idx 9 sagule.s.id 6
1 SIMPLE s8 ref sender_id_idx,receiver_id_idx sender_id_idx 9 sagule.s.id 7
1 SIMPLE s9 index_merge sender_id_idx,receiver_id_idx receiver_id_idx,sender_id_idx 9,9 NULL 66 Using union(receiver_id_idx,sender_id_idx); Using where

Answer 1

I'm a MSSQL guy and havent used mysql but the concepts should be the same.
Firstly can you remove the group and order by and comment out all tables except for the first one "user". Also comment out any columns of the removed tables. As I have below.
SELECT s.id AS s__id,
s.dname AS s__dname,
s.gender AS s__gender,
s.height AS s__height,
s.dob AS s__dob
-- s3.id AS s3__id,
-- s3.code AS s3__code,
-- s3.name_geo AS s3__name_geo,
-- s4.id AS s4__id,
-- s4.name_geo AS s4__name_geo,
-- s5.id AS s5__id,
-- s6.id AS s6__id,
-- s7.id AS s7__id,
-- s8.id AS s8__id,
-- s9.id AS s9__id
FROM user s --LEFT JOIN
-- country s3 ON s.country_id = s3.id LEFT JOIN
-- city s4 ON s.city_id = s4.id LEFT JOIN
-- block s5 ON ((s.id = s5.blocked_id AND s5.blocker_id = '1')) LEFT JOIN
-- favorite s6 ON ((s.id = s6.favorited_id AND s6.favoriter_id = '1')) LEFT JOIN
-- favorite s7 ON ((s.id = s7.favoriter_id AND s7.favorited_id = '1')) LEFT JOIN
-- message s8 ON ((s.id = s8.sender_id AND s8.receiver_id = '1' AND s8.read_at IS NULL)) LEFT JOIN
-- message s9 ON (((s.id = s9.sender_id AND s9.receiver_id = '1') OR (s.id = s9.receiver_id AND s9.sender_id = '1')))
WHERE (s.id IN ('22', '36', '53', '105', '152', '156', '169', '182', '186', '192', '201', '215', '252', '287', '288', '321', '330', '351', '366', '399'))
Run the query and record the time. Then add one table and its columns back in at a time and run it until you find which one causes it to slow significantly.
SELECT s.id AS s__id,
s.dname AS s__dname,
s.gender AS s__gender,
s.height AS s__height,
s.dob AS s__dob,
s3.id AS s3__id,
s3.code AS s3__code,
s3.name_geo AS s3__name_geo
-- s4.id AS s4__id,
-- s4.name_geo AS s4__name_geo,
-- s5.id AS s5__id,
-- s6.id AS s6__id,
-- s7.id AS s7__id,
-- s8.id AS s8__id,
-- s9.id AS s9__id
FROM user s LEFT JOIN
country s3 ON s.country_id = s3.id --LEFT JOIN
-- city s4 ON s.city_id = s4.id LEFT JOIN
-- block s5 ON ((s.id = s5.blocked_id AND s5.blocker_id = '1')) LEFT JOIN
-- favorite s6 ON ((s.id = s6.favorited_id AND s6.favoriter_id = '1')) LEFT JOIN
-- favorite s7 ON ((s.id = s7.favoriter_id AND s7.favorited_id = '1')) LEFT JOIN
-- message s8 ON ((s.id = s8.sender_id AND s8.receiver_id = '1' AND s8.read_at IS NULL)) LEFT JOIN
-- message s9 ON (((s.id = s9.sender_id AND s9.receiver_id = '1') OR (s.id = s9.receiver_id AND s9.sender_id = '1')))
WHERE (s.id IN ('22', '36', '53', '105', '152', '156', '169', '182', '186', '192', '201', '215', '252', '287', '288', '321', '330', '351', '366', '399'))
My guess is that it would be the block and both favorites and message joins that is giving you the performance hit (the one with the most rows will be the biggest hit).
For the block table, Can you remove one of the indexes and change the other to be something along the lines of (I am not sure of the syntax but you'll get the point)
INDEX blocker_id_idx (blocker_id,blocked_id),
and try it with the columns order swapped around to find witch order is best for your query
INDEX blocker_id_idx (blocked_id,blocker_id),
For the favorite table, change the indexes to
INDEX favoriter_id_idx (favoriter_id,favorited_id),
INDEX favorited_id_idx (favorited_id,favoriter_id),
Again try it with the columns swapped around to find which give better performance.
Do the same for the message indexes.
Do that and let me know if things improved. There are a few other things that can be done to improve it further. - EDIT: It seams I lied about the few other things, what I had intended would not have made any difference. But I can speed up your first query which is below.
EDIT This is for your first select query.
This one is a bit long, but I wanted to show you how indexes work so you can make your own.
Lets say the table contains 100,000 rows.
When you select from it, this is the general process it will take.
Are there any indexes that cover or
mostly cover the columns that I need.
(I your case, no there isn't.)
So use Primary Index and scan though
every row in the table to check for a
match.
Every row in the table will need to
be read from disk to find which
columns match you criteria. So to
return the approx 10,000 rows (this
is a guess) that match you data the
database engine has read all 100,000
rows.
You do have a top 20 in you query, so it will limit the amount of rows the engine will read from disk.
Example
read row 1: is match so add to result
read row 2: no match - skip
read row 3: no match - skip
read row 4: is match so add to
result.
stop after 20 rows identified
You potentially read about 5000 rows from disk to return 20.
We need to create an index that will help us read as few records as possible from the table/disk, but still get the rows we are after. So here goes.
Your query uses 4 filters to get to the data.
s.gender = 'female' AND
s.marital_status = 'single' AND
s.smokes = 'no' AND
s.deleted_at IS NULL
What we need to do now is identify which filter by itself will return the least amount of rows. I cant tell as I don't have any data, but this is what I would guess to be in your table.
The gender column support 2 values and it would be fair to estimate that half of the records in your database are male and the other female, so that filter you need will return approx 50,000 rows.
Now for marital status, supports four values, so if we say the data has an equal spread, it would mean we would get roughly 25,000 rows back. Of course, it depends on th actual data and I would say, that there are not too many widowed in the data, so a better estimate may be 30% share between the other three. So lets say 30,000 records marked as single.
Now for the smokes column. I have read that here in Australia about 10% of people smoke which is a fairly low number compared to other countries. So lets say 25% either smoke or smoke sometimes. That leaves us with approx 75,000 non smokers.
Now for the last column, deleted. A fair guess on my part but lets say 5% are marked as deleted. That leaves us with approx 95,000 rows.
So in summary (remember, this is all pure guess work on my part, your data may be different)
Gender 50,000 rows or 50%
Marital status 30,000 rows or 30%
Smokes 75,000 rows or 75%
Deleted 95,000 rows or 95%
So if we create an index with the four columns using the one that returns the least amount of rows first, we would get the following
INDEX index01_idx (marital_status,gender,smokes,deleted_at),
Now this is what will happen when we run the select.
The server will find an index that
covers all the columns in the WHERE
clause
It will narrow down the result set to
30,000 "single" records.
Of those 30,000, 50% will be female
that leaves 15,000 records
Of those 15,000, 75% will not smoke
that leaves 11,250 records
Of those 11,250, 95% will not be
deleted,
That leaves us with just over 10,000 records out of 100,000 total that we have identified as the records we want but not yet read from disk. You also have a limit 20 in the query so the database engine just needs to read the first 20 of the 10,000 and return the result. Its super quick, the hard disk will love you and the scary DBA will even mumble and grunt with approval.

Answer 2

In your second SELECT query, you can remove the GROUP BY clause because you aren't using any Aggregate functions (count, min, max...) in your SELECT clause.
I doubt this will help much improving performance, though.
In any case, I recommend to watch the first half of this talk "A Look into a MySQL DBA's Toolchest".
(The first two thirds of the video are about free open-source admin-tools for mysql on Unix, the last third or so is about replication)
Video A Look into a MySQL DBA's Toolchest
From the same talk: The Guide To Understanding mysqlreport

Answer 3

Without some Data to Test, it is not so easy to make a good advice.
Generating an Index for fields that are searched frequently, can help make your query faster. But with an Index your Inserts and Updates can get slower. You have to think about the tradeoff. So index the Columns that get searched freqeuently, but test the new Index on the Data so you can see if it runs faster.
I don't know which Tools you are using, but with the MySQL Workbench there is a Command "Explain Current Statement" under the "Query"-Menu. There you can see which actions were done by MySQL and which keys were used. Your Query shows "null" which means no key was used and MySQL had to run through the whole data comparing with the search term.
Hope this helps a bit.