How to improve a MYSQL Query with multiple JOINS - mysql

Question

I have 2 search pages that are super slow in getting results.
I did not write the queries, but I know they are not written in an efficient way; I just do not have enough practice in MYSQL to figure out how to make them more efficient.
What should I do to improve the following query?
SELECT DISTINCT m.id AS memberID , m.login , m.age , p.gender
, p.name AS header , p.id AS profileID , p.city , p.state , p.lastlogin
, o.login AS online , c.name AS country , ph.filename_1 AS pic
FROM dt_members AS m
INNER JOIN dt_profile_approved AS p ON m.id=p.member_id
LEFT JOIN dt_privacy AS pv ON m.id=pv.member_id
INNER JOIN dt_countries AS c ON c.id=p.country
LEFT JOIN dt_photos AS ph ON m.id=ph.member_id
LEFT JOIN dt_usersonline AS o ON m.login=o.login
WHERE p.status=1 AND (pv.unsearchable IS NULL OR pv.unsearchable='')
AND p.gender='Female' AND m.age BETWEEN 25 AND 40
ORDER BY p.lastlogin DESC
LIMIT 0, 21;
it is terribly slow and showing 500 error often.
Output of explain:
Show Create Table:
CREATE TABLE `dt_members` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`login` varchar(25) DEFAULT NULL,
`pswd` varchar(20) DEFAULT NULL,
`email` varchar(255) DEFAULT NULL,
`name` varchar(40) DEFAULT NULL,
`gender` varchar(10) DEFAULT NULL,
`age` int(11) DEFAULT NULL,
`country` varchar(255) DEFAULT NULL,
`looking_for` varchar(255) DEFAULT NULL,
`ip_addr` varchar(15) DEFAULT NULL,
`reg_date` int(11) DEFAULT NULL,
`status` int(11) DEFAULT NULL,
`system_status` int(11) DEFAULT '0',
`system_status_end` int(11) DEFAULT NULL,
`unlimited` int(11) DEFAULT '0',
`unlimited_end` int(11) DEFAULT NULL,
`matchfinder` int(1) DEFAULT '0',
PRIMARY KEY (`id`),
KEY `login` (`login`),
KEY `pswd` (`pswd`),
KEY `email` (`email`),
KEY `name` (`name`),
KEY `gender` (`gender`),
KEY `age` (`age`),
KEY `country` (`country`),
KEY `looking_for` (`looking_for`),
KEY `ip_addr` (`ip_addr`),
KEY `reg_date` (`reg_date`),
KEY `status` (`status`),
KEY `system_status` (`system_status`),
KEY `system_status_end` (`system_status_end`),
KEY `unlimited` (`unlimited`),
KEY `unlimited_end` (`unlimited_end`),
KEY `matchfinder` (`matchfinder`)
) ENGINE=MyISAM AUTO_INCREMENT=29150 DEFAULT CHARSET=latin1
CREATE TABLE `dt_profile` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`member_id` int(11) DEFAULT NULL,
`country` int(11) DEFAULT NULL,
`state` varchar(255) DEFAULT NULL,
`city` varchar(255) DEFAULT NULL,
`email` varchar(255) DEFAULT NULL,
`name` varchar(255) DEFAULT NULL,
`gender` varchar(20) DEFAULT NULL,
`birth_day` int(11) DEFAULT NULL,
`birth_month` varchar(6) DEFAULT NULL,
`birth_year` int(11) DEFAULT NULL,
`marital_status` int(11) DEFAULT NULL,
`children` int(11) DEFAULT NULL,
`drinking` int(11) DEFAULT NULL,
`smoking` int(11) DEFAULT NULL,
`food` int(11) DEFAULT NULL,
`eye_color` int(11) DEFAULT NULL,
`hair_color` int(11) DEFAULT NULL,
`height` int(11) DEFAULT NULL,
`body_type` int(11) DEFAULT NULL,
`race` int(11) DEFAULT NULL,
`religion` int(11) DEFAULT NULL,
`occupation` int(11) DEFAULT NULL,
`education` int(11) DEFAULT NULL,
`lang_1` int(11) DEFAULT NULL,
`lang_1_rate` int(11) DEFAULT NULL,
`lang_2` int(11) DEFAULT NULL,
`lang_2_rate` int(11) DEFAULT NULL,
`lang_3` int(11) DEFAULT NULL,
`lang_3_rate` int(11) DEFAULT NULL,
`lang_4` int(11) DEFAULT NULL,
`lang_4_rate` int(11) DEFAULT NULL,
`looking_for` varchar(10) DEFAULT NULL,
`age_from` int(11) DEFAULT NULL,
`age_to` int(11) DEFAULT NULL,
`general_info` text,
`appearance_info` text,
`looking_for_info` text,
`status` int(11) DEFAULT NULL,
`finish_status` int(11) DEFAULT NULL,
`not_newbie` int(11) DEFAULT NULL,
`lastlogin` int(10) NOT NULL DEFAULT '0',
`zipcode` varchar(5) NOT NULL DEFAULT '',
`longitude` double DEFAULT NULL,
`latitude` double DEFAULT NULL,
`photo_pass` varchar(25) NOT NULL DEFAULT '',
`view_count` int(11) DEFAULT '0',
`wants_kids` int(11) DEFAULT NULL,
`kids_okay` int(11) DEFAULT NULL,
`relocate_domestic` int(11) DEFAULT NULL,
`relocate_international` int(11) DEFAULT NULL,
`pioneer` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `member_id` (`member_id`),
KEY `country` (`country`),
KEY `state` (`state`),
KEY `city` (`city`),
KEY `email` (`email`),
KEY `name` (`name`),
KEY `gender` (`gender`),
KEY `birth_day` (`birth_day`),
KEY `birth_month` (`birth_month`),
KEY `birth_year` (`birth_year`),
KEY `marital_status` (`marital_status`),
KEY `children` (`children`),
KEY `drinking` (`drinking`),
KEY `smoking` (`smoking`),
KEY `food` (`food`),
KEY `eye_color` (`eye_color`),
KEY `hair_color` (`hair_color`),
KEY `height` (`height`),
KEY `body_type` (`body_type`),
KEY `race` (`race`),
KEY `religion` (`religion`),
KEY `occupation` (`occupation`),
KEY `education` (`education`),
KEY `lang_1` (`lang_1`),
KEY `lang_1_rate` (`lang_1_rate`),
KEY `lang_2` (`lang_2`),
KEY `lang_2_rate` (`lang_2_rate`),
KEY `lang_3` (`lang_3`),
KEY `lang_3_rate` (`lang_3_rate`),
KEY `lang_4` (`lang_4`),
KEY `lang_4_rate` (`lang_4_rate`),
KEY `looking_for` (`looking_for`),
KEY `age_from` (`age_from`),
KEY `age_to` (`age_to`),
KEY `status` (`status`),
KEY `finish_status` (`finish_status`),
KEY `not_newbie` (`not_newbie`),
KEY `lastlogin` (`lastlogin`),
KEY `zipcode` (`zipcode`),
KEY `longitude` (`longitude`),
KEY `latitude` (`latitude`),
KEY `photo_pass` (`photo_pass`),
KEY `view_count` (`view_count`),
KEY `wants_kids` (`wants_kids`),
KEY `kids_okay` (`kids_okay`),
KEY `relocate_domestic` (`relocate_domestic`),
KEY `relocate_international` (`relocate_international`),
KEY `pioneer` (`pioneer`)
) ENGINE=MyISAM AUTO_INCREMENT=18389 DEFAULT CHARSET=latin1
CREATE TABLE `dt_privacy` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`member_id` int(11) DEFAULT NULL,
`online_yn` char(1) DEFAULT NULL,
`vkiss_yn` char(1) DEFAULT NULL,
`profiles_yn` char(1) DEFAULT NULL,
`IM_yn` char(1) DEFAULT NULL,
`featured_yn` char(1) DEFAULT NULL,
`HL_messaged_yn` char(1) DEFAULT NULL,
`HL_im_yn` char(1) DEFAULT NULL,
`HL_viewed_yn` char(1) DEFAULT NULL,
`HL_kissed_yn` char(1) DEFAULT NULL,
`HL_favorite_yn` char(1) DEFAULT NULL,
`unsearchable` char(1) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `member_id` (`member_id`),
KEY `online_yn` (`online_yn`),
KEY `vkiss_yn` (`vkiss_yn`),
KEY `profiles_yn` (`profiles_yn`),
KEY `IM_yn` (`IM_yn`),
KEY `featured_yn` (`featured_yn`),
KEY `HL_messaged_yn` (`HL_messaged_yn`),
KEY `HL_im_yn` (`HL_im_yn`),
KEY `HL_viewed_yn` (`HL_viewed_yn`),
KEY `HL_kissed_yn` (`HL_kissed_yn`),
KEY `HL_favorite_yn` (`HL_favorite_yn`),
KEY `unsearchable` (`unsearchable`)
) ENGINE=MyISAM AUTO_INCREMENT=26305 DEFAULT CHARSET=latin1
CREATE TABLE `dt_countries` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(100) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `name` (`name`)
) ENGINE=MyISAM AUTO_INCREMENT=226 DEFAULT CHARSET=latin1
CREATE TABLE `dt_photos` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`member_id` varchar(255) DEFAULT NULL,
`filename_1` varchar(255) DEFAULT NULL,
`filename_2` varchar(255) DEFAULT NULL,
`filename_3` varchar(255) NOT NULL DEFAULT '',
`filename_4` varchar(255) NOT NULL DEFAULT '',
`filename_5` varchar(255) NOT NULL DEFAULT '',
`filename_6` varchar(255) NOT NULL DEFAULT '',
`filename_7` varchar(255) NOT NULL DEFAULT '',
`filename_8` varchar(255) NOT NULL DEFAULT '',
`filename_9` varchar(255) NOT NULL DEFAULT '',
`filename_10` varchar(255) NOT NULL DEFAULT '',
`filename_11` varchar(255) NOT NULL DEFAULT '',
`filename_12` varchar(255) NOT NULL DEFAULT '',
`filename_13` varchar(255) NOT NULL DEFAULT '',
`filename_14` varchar(255) NOT NULL DEFAULT '',
`filename_15` varchar(255) NOT NULL DEFAULT '',
`filename_16` varchar(255) NOT NULL DEFAULT '',
`filename_17` varchar(255) NOT NULL DEFAULT '',
`filename_18` varchar(255) NOT NULL DEFAULT '',
`filename_19` varchar(255) NOT NULL DEFAULT '',
`filename_20` varchar(255) NOT NULL DEFAULT '',
`private_1` tinyint(1) NOT NULL DEFAULT '0',
`private_2` tinyint(1) NOT NULL DEFAULT '0',
`private_3` tinyint(1) NOT NULL DEFAULT '0',
`private_4` tinyint(1) NOT NULL DEFAULT '0',
`private_5` tinyint(1) NOT NULL DEFAULT '0',
`private_6` tinyint(1) NOT NULL DEFAULT '0',
`private_7` tinyint(1) NOT NULL DEFAULT '0',
`private_8` tinyint(1) NOT NULL DEFAULT '0',
`private_9` tinyint(1) NOT NULL DEFAULT '0',
`private_10` tinyint(1) NOT NULL DEFAULT '0',
`private_11` tinyint(1) NOT NULL DEFAULT '0',
`private_12` tinyint(1) NOT NULL DEFAULT '0',
`private_13` tinyint(1) NOT NULL DEFAULT '0',
`private_14` tinyint(1) NOT NULL DEFAULT '0',
`private_15` tinyint(1) NOT NULL DEFAULT '0',
`private_16` tinyint(1) NOT NULL DEFAULT '0',
`private_17` tinyint(1) NOT NULL DEFAULT '0',
`private_18` tinyint(1) NOT NULL DEFAULT '0',
`private_19` tinyint(1) NOT NULL DEFAULT '0',
`private_20` tinyint(1) NOT NULL DEFAULT '0',
`password` varchar(255) NOT NULL DEFAULT '',
`description_1` varchar(255) DEFAULT NULL,
`description_2` varchar(255) DEFAULT NULL,
`description_3` varchar(255) NOT NULL DEFAULT '',
`description_4` varchar(255) NOT NULL DEFAULT '',
`description_5` varchar(255) NOT NULL DEFAULT '',
`description_6` varchar(255) NOT NULL DEFAULT '',
`description_7` varchar(255) NOT NULL DEFAULT '',
`description_8` varchar(255) NOT NULL DEFAULT '',
`description_9` varchar(255) NOT NULL DEFAULT '',
`description_10` varchar(255) NOT NULL DEFAULT '',
`description_11` varchar(255) NOT NULL DEFAULT '',
`description_12` varchar(255) NOT NULL DEFAULT '',
`description_13` varchar(255) NOT NULL DEFAULT '',
`description_14` varchar(255) NOT NULL DEFAULT '',
`description_15` varchar(255) NOT NULL DEFAULT '',
`description_16` varchar(255) NOT NULL DEFAULT '',
`description_17` varchar(255) NOT NULL DEFAULT '',
`description_18` varchar(255) NOT NULL DEFAULT '',
`description_19` varchar(255) NOT NULL DEFAULT '',
`description_20` varchar(255) NOT NULL DEFAULT '',
PRIMARY KEY (`id`),
KEY `filename_12` (`filename_12`),
KEY `filename_13` (`filename_13`),
KEY `filename_14` (`filename_14`),
KEY `filename_15` (`filename_15`),
KEY `filename_16` (`filename_16`),
KEY `filename_17` (`filename_17`),
KEY `filename_18` (`filename_18`),
KEY `filename_19` (`filename_19`),
KEY `filename_20` (`filename_20`),
KEY `member_id` (`member_id`),
KEY `filename_1` (`filename_1`),
KEY `filename_2` (`filename_2`),
KEY `filename_3` (`filename_3`),
KEY `filename_4` (`filename_4`),
KEY `filename_5` (`filename_5`),
KEY `filename_6` (`filename_6`),
KEY `filename_7` (`filename_7`),
KEY `filename_8` (`filename_8`),
KEY `filename_9` (`filename_9`),
KEY `filename_10` (`filename_10`),
KEY `private_1` (`private_1`),
KEY `private_2` (`private_2`),
KEY `private_3` (`private_3`),
KEY `private_4` (`private_4`),
KEY `private_5` (`private_5`),
KEY `private_6` (`private_6`),
KEY `private_7` (`private_7`),
KEY `private_8` (`private_8`),
KEY `private_9` (`private_9`),
KEY `private_10` (`private_10`),
KEY `private_11` (`private_11`),
KEY `private_12` (`private_12`),
KEY `private_13` (`private_13`),
KEY `private_14` (`private_14`),
KEY `private_15` (`private_15`),
KEY `private_16` (`private_16`),
KEY `private_17` (`private_17`),
KEY `private_18` (`private_18`),
KEY `private_19` (`private_19`),
KEY `private_20` (`private_20`),
KEY `password` (`password`),
KEY `description_1` (`description_1`),
KEY `description_2` (`description_2`),
KEY `description_3` (`description_3`),
KEY `description_4` (`description_4`),
KEY `description_5` (`description_5`),
KEY `description_6` (`description_6`),
KEY `description_7` (`description_7`),
KEY `description_8` (`description_8`),
KEY `description_9` (`description_9`),
KEY `description_10` (`description_10`),
KEY `description_11` (`description_11`),
KEY `description_12` (`description_12`),
KEY `description_13` (`description_13`),
KEY `description_14` (`description_14`),
KEY `description_15` (`description_15`),
KEY `description_16` (`description_16`),
KEY `description_17` (`description_17`),
KEY `description_18` (`description_18`),
KEY `description_19` (`description_19`),
KEY `description_20` (`description_20`),
KEY `filename_10_2` (`filename_10`),
KEY `filename_10_3` (`filename_10`)
) ENGINE=MyISAM AUTO_INCREMENT=11174 DEFAULT CHARSET=latin1
CREATE TABLE `dt_usersonline` (
`id` int(8) NOT NULL AUTO_INCREMENT,
`timestamp` int(15) NOT NULL DEFAULT '0',
`ip` varchar(40) NOT NULL DEFAULT '',
`login` varchar(25) NOT NULL DEFAULT '',
`userid` int(10) NOT NULL DEFAULT '0',
`session_id` varchar(255) NOT NULL DEFAULT '',
PRIMARY KEY (`id`),
KEY `id_2` (`id`),
KEY `timestamp` (`timestamp`),
KEY `ip` (`ip`),
KEY `login` (`login`),
KEY `userid` (`userid`),
KEY `session_id` (`session_id`)
) ENGINE=MyISAM AUTO_INCREMENT=4424348 DEFAULT CHARSET=latin1

Answer 1

CREATE TABLE `dt_members` (
`id` int(11) NOT NULL AUTO_INCREMENT,...
CREATE TABLE `dt_photos` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`member_id` varchar(255) DEFAULT NULL,..
You are joining on your query as:
LEFT JOIN dt_photos AS ph ON m.id=ph.member_id
Notice how every other time you join on member_id to m.id the column looks like member_id int(11) DEFAULT NULL. Your join criteria should be on the same data type, otherwise it looks like it is doing a full table scan. It's not using the key because it believes that it has to scan the varchar field rather than utilizing indexes for on clause.
Try Altering column member_id to that of m.id from dt_members
It is interesting to note, had you actually tried to create a foreign key constraint, it would not have allowed you to because of the mismatched data types...
Warning: MYISAM table will be locked during the duration of the later table process.

Answer 2

Try moving where conditions into the corresponding join condition:
SELECT DISTINCT m.id AS memberID , m.login , m.age , p.gender
, p.name AS header , p.id AS profileID , p.city , p.state , p.lastlogin
, o.login AS online , c.name AS country , ph.filename_1 AS pic
FROM dt_members AS m
INNER JOIN dt_profile_approved AS p ON m.id=p.member_id
AND p.status=1 AND p.gender='Female' -- Moved from WHERE clause
LEFT JOIN dt_privacy AS pv ON m.id=pv.member_id
AND (pv.unsearchable IS NULL OR pv.unsearchable='') -- Moved from WHERE clause
INNER JOIN dt_countries AS c ON c.id=p.country
LEFT JOIN dt_photos AS ph ON m.id=ph.member_id
LEFT JOIN dt_usersonline AS o ON m.login=o.login
WHERE m.age BETWEEN 25 AND 40
ORDER BY p.lastlogin DESC
LIMIT 0, 21;
While where conditions are made after all joins, join conditions are evaluated during the join, so vast numbers of unnecessary joins can be avoided early.
Although theoretically the query optimizer should do this for you, I have found the mysql can be particularly dense in this regard.

Answer 3

It looks like most of the selectivity on your query is on your dt_profile_approved table. Try creating a compound index on (status, gender, lastlogin)
This should allow an index-only selection and ordering.
Your query does the notorious SELECT a lot ORDER BY something DESC LIMIT tinynumber pattern. This is expensive. Try a deferred join. Start by getting the interesting items from your dt_profile_approved table, like this:
SELECT member_id, lastlogin
FROM dt_profile_approved
WHERE status=1
AND gender='Female'
ORDER BY lastlogin DESC
This subquery can be optimized very cleanly with a compound index on (status, gender, lastlogin, member_id). That's called a covering index. It has a great benefit: no extra sorting is required, because the index is already sorted.
Based on the exact query you showed us, the order in the index of gender and status doesn't matter. But I guess you have another query that looks for males, and you may have one that omits that. So status is likely to be the more selective field across all your queries. (Guessing.)
Then, join that subquery into the rest of your query... that will look something like this.
SELECT DISTINCT
m.id AS memberID , m.login , m.age,
p.gender. p.name AS header , p.id AS profileID ,
p.city , p.state , p.lastlogin,
o.login AS online ,
c.name AS country , ph.filename_1 AS pic
FROM (
SELECT member_id, lastlogin
FROM dt_profile_approved
WHERE status=1
AND gender='Female'
ORDER BY lastlogin DESC
) AS sel
INNER JOIN dt_profile_approved AS p ON sel.member_id = p.member_id
INNER JOIN dt_members AS m ON m.id=sel.member_id
LEFT JOIN dt_privacy AS pv ON m.id=pv.member_id
INNER JOIN dt_countries AS c ON c.id=p.country
LEFT JOIN dt_photos AS ph ON m.id=ph.member_id
LEFT JOIN dt_usersonline AS o ON m.login=o.login
WHERE (pv.unsearchable IS NULL OR pv.unsearchable='')
AND m.age BETWEEN 25 AND 40
ORDER BY sel.lastlogin DESC
LIMIT 0, 21;
If this works, it will be because it can limit the labor of sorting using the index.