Perf of select mysql query is really bad - mysql

I'm not sure why this query is taking 4 minutes to complete:
SELECT
su.sid,u.uid,u.display_name,u.locale
FROM user u
LEFT JOIN subscription_user su ON su.uid = u.uid
ORDER BY u.display_name DESC
LIMIT 0,25;
Well, I know it's due to the order, remove it and it's very fast. If I change to using INNER JOIN instead it's fast but the issue is not all users may be in the subscription_user table.
CREATE TABLE `user` (
`uid` int(11) NOT NULL AUTO_INCREMENT,
`password` varchar(100) DEFAULT NULL,
`user_type` varchar(10) NOT NULL DEFAULT 'user',
`display_name` varchar(50) NOT NULL,
`email` varchar(100) NOT NULL,
`locale` varchar(8) DEFAULT 'en',
`last_login` datetime DEFAULT NULL,
`auth_type` varchar(10) DEFAULT NULL,
`auth_data` varchar(500) DEFAULT NULL,
`inactive` tinyint(4) NOT NULL DEFAULT '0',
`receive_email` tinyint(4) NOT NULL DEFAULT '1',
`stateid` int(10) DEFAULT NULL,
`owner_group_id` int(11) DEFAULT NULL,
`signature` varchar(500) DEFAULT NULL,
`raw_signature` varchar(500) DEFAULT NULL,
`round_robin` smallint(5) unsigned NOT NULL DEFAULT '0',
PRIMARY KEY (`uid`),
UNIQUE KEY `email` (`email`),
KEY `stateid` (`stateid`) USING BTREE,
KEY `user_type` (`user_type`) USING BTREE,
KEY `name` (`display_name`)
) ENGINE=InnoDB AUTO_INCREMENT=28343 DEFAULT CHARSET=latin1;
CREATE TABLE `subscription_user` (
`sid` varchar(50) NOT NULL,
`uid` int(11) NOT NULL,
`deleted` tinyint(4) NOT NULL DEFAULT '0',
`forum_user` varchar(50) NOT NULL,
PRIMARY KEY (`sid`,`uid`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

When you have an SQL query, the index can only really help you if the first column in the index is part of the query.
Your query joins su.uid = u.uid and the optimizer will not be able to use that to reference the first column in the subscription primary key index.
You should either reverse the order of the columns in the primary key, or alternatively, you should add a foreign key index, or an independent index on the uid

Related

How can i tune this mysql query to run successfully

This is my query on mysql version 8
select sender, fullName, phoneNumber, addressState, businessName, bvn, max(date)
from tranlog t INNER JOIN agent a on t.sender = a.realId
where captureDate < '2022-03-01' and active = 'Y' and thirdparty = 0
group by sender
CREATE TABLE `agent` (
`id` bigint NOT NULL AUTO_INCREMENT,
`realId` varchar(19) DEFAULT NULL,
`active` char(1) DEFAULT NULL,
`phoneNumber` varchar(15) DEFAULT NULL,
`address` varchar(255) DEFAULT NULL,
`addressState` varchar(50) DEFAULT NULL,
`addressCity` varchar(50) DEFAULT NULL,
`fullName` varchar(255) DEFAULT NULL,
`businessName` varchar(255) DEFAULT NULL,
`corporate` bit(1) DEFAULT b'0',
`thirdparty` bit(1) NOT NULL DEFAULT b'0',
PRIMARY KEY (`id`),
KEY `id` (`fee_group`),
KEY `realId` (`realId`),
KEY `agent_password` (`password`),
KEY `agent_idx` (`active`,`thirdparty`),
) ENGINE=InnoDB AUTO_INCREMENT=29784 DEFAULT CHARSET=latin1;
CREATE TABLE `tranlog` (
`id` bigint NOT NULL AUTO_INCREMENT
`date` datetime DEFAULT NULL
`captureDate` date DEFAULT NULL
`sender` varchar(50) DEFAULT NULL
PRIMARY KEY (`id`)
KEY `tranlog_date` (`date`)
KEY `sender` (`sender`)
KEY `tranlog_capturedate_idx` (`captureDate`)
) ENGINE=InnoDB AUTO_INCREMENT=49373312 DEFAULT CHARSET=latin1"
But i keep getting 'C:windows\TEMP#sql1234_2' is full which i believe is about temporary table
I have increased the size of tmp_table_size and max_heap_size to 3G yet no reprieve, the error keeps poping up.
Any ideas on how to tune the query
Add a 'LIMIT CLAUSE' to your query. It seems you have a very large result set.

MySQL use separate indices for JOIN and GROUP BY

I am trying to execute following query
SELECT
a.sessionID AS `sessionID`,
firstSeen, birthday, gender,
isAnonymous, LanguageCode
FROM transactions AS trx
INNER JOIN actions AS a ON a.sessionID = trx.SessionID
WHERE a.ActionType = 'PURCHASE'
GROUP BY trx.TransactionNumber
Explain provides the following output
1 SIMPLE trx ALL TransactionNumber,SessionID NULL NULL NULL 225036 Using temporary; Using filesort
1 SIMPLE a ref sessionID sessionID 98 infinitiExport.trx.SessionID 1 Using index
The problem is that I am trying to use one field for join and different field for GROUP BY.
How can I tell MySQL to use different indices for same table?
CREATE TABLE `transactions` (
`SessionID` varchar(32) NOT NULL DEFAULT '',
`date` datetime DEFAULT NULL,
`TransactionNumber` varchar(32) NOT NULL DEFAULT '',
`CustomerECommerceTrackID` int(11) DEFAULT NULL,
`SKU` varchar(45) DEFAULT NULL,
`AmountPaid` double DEFAULT NULL,
`Currency` varchar(10) DEFAULT NULL,
`Quantity` int(11) DEFAULT NULL,
`Name` tinytext NOT NULL,
`Category` varchar(45) NOT NULL DEFAULT '',
`customerInfoXML` text,
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`id`),
KEY `TransactionNumber` (`TransactionNumber`),
KEY `SessionID` (`SessionID`)
) ENGINE=InnoDB AUTO_INCREMENT=212007 DEFAULT CHARSET=utf8;
CREATE TABLE `actions` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`sessionActionDate` datetime DEFAULT NULL,
`actionURL` varchar(255) DEFAULT NULL,
`sessionID` varchar(32) NOT NULL DEFAULT '',
`ActionType` varchar(64) DEFAULT NULL,
`CustomerID` int(11) DEFAULT NULL,
`IPAddressID` int(11) DEFAULT NULL,
`CustomerDeviceID` int(11) DEFAULT NULL,
`customerInfoXML` text,
PRIMARY KEY (`id`),
KEY `ActionType` (`ActionType`),
KEY `CustomerDeviceID` (`CustomerDeviceID`),
KEY `sessionID` (`sessionID`)
) ENGINE=InnoDB AUTO_INCREMENT=15042833 DEFAULT CHARSET=utf8;
Thanks
EDIT 1: My indexes were broken, I had to add (SessionID, TransactionNumber) index to transactions table, however now, when I try to include trx.customerInfoXML table mysql stops using index
EDIT 2 Another answer does not really solved my problem because it's not standard sql syntax and generally not a good idea to force indices.
For ORM users such syntax is a unattainable luxury.
EDIT 3 I updated my indices and it solved the problem, see EDIT 1

How to improve a MYSQL Query with multiple JOINS

I have 2 search pages that are super slow in getting results.
I did not write the queries, but I know they are not written in an efficient way; I just do not have enough practice in MYSQL to figure out how to make them more efficient.
What should I do to improve the following query?
SELECT DISTINCT m.id AS memberID , m.login , m.age , p.gender
, p.name AS header , p.id AS profileID , p.city , p.state , p.lastlogin
, o.login AS online , c.name AS country , ph.filename_1 AS pic
FROM dt_members AS m
INNER JOIN dt_profile_approved AS p ON m.id=p.member_id
LEFT JOIN dt_privacy AS pv ON m.id=pv.member_id
INNER JOIN dt_countries AS c ON c.id=p.country
LEFT JOIN dt_photos AS ph ON m.id=ph.member_id
LEFT JOIN dt_usersonline AS o ON m.login=o.login
WHERE p.status=1 AND (pv.unsearchable IS NULL OR pv.unsearchable='')
AND p.gender='Female' AND m.age BETWEEN 25 AND 40
ORDER BY p.lastlogin DESC
LIMIT 0, 21;
it is terribly slow and showing 500 error often.
Output of explain:
Show Create Table:
CREATE TABLE `dt_members` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`login` varchar(25) DEFAULT NULL,
`pswd` varchar(20) DEFAULT NULL,
`email` varchar(255) DEFAULT NULL,
`name` varchar(40) DEFAULT NULL,
`gender` varchar(10) DEFAULT NULL,
`age` int(11) DEFAULT NULL,
`country` varchar(255) DEFAULT NULL,
`looking_for` varchar(255) DEFAULT NULL,
`ip_addr` varchar(15) DEFAULT NULL,
`reg_date` int(11) DEFAULT NULL,
`status` int(11) DEFAULT NULL,
`system_status` int(11) DEFAULT '0',
`system_status_end` int(11) DEFAULT NULL,
`unlimited` int(11) DEFAULT '0',
`unlimited_end` int(11) DEFAULT NULL,
`matchfinder` int(1) DEFAULT '0',
PRIMARY KEY (`id`),
KEY `login` (`login`),
KEY `pswd` (`pswd`),
KEY `email` (`email`),
KEY `name` (`name`),
KEY `gender` (`gender`),
KEY `age` (`age`),
KEY `country` (`country`),
KEY `looking_for` (`looking_for`),
KEY `ip_addr` (`ip_addr`),
KEY `reg_date` (`reg_date`),
KEY `status` (`status`),
KEY `system_status` (`system_status`),
KEY `system_status_end` (`system_status_end`),
KEY `unlimited` (`unlimited`),
KEY `unlimited_end` (`unlimited_end`),
KEY `matchfinder` (`matchfinder`)
) ENGINE=MyISAM AUTO_INCREMENT=29150 DEFAULT CHARSET=latin1
CREATE TABLE `dt_profile` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`member_id` int(11) DEFAULT NULL,
`country` int(11) DEFAULT NULL,
`state` varchar(255) DEFAULT NULL,
`city` varchar(255) DEFAULT NULL,
`email` varchar(255) DEFAULT NULL,
`name` varchar(255) DEFAULT NULL,
`gender` varchar(20) DEFAULT NULL,
`birth_day` int(11) DEFAULT NULL,
`birth_month` varchar(6) DEFAULT NULL,
`birth_year` int(11) DEFAULT NULL,
`marital_status` int(11) DEFAULT NULL,
`children` int(11) DEFAULT NULL,
`drinking` int(11) DEFAULT NULL,
`smoking` int(11) DEFAULT NULL,
`food` int(11) DEFAULT NULL,
`eye_color` int(11) DEFAULT NULL,
`hair_color` int(11) DEFAULT NULL,
`height` int(11) DEFAULT NULL,
`body_type` int(11) DEFAULT NULL,
`race` int(11) DEFAULT NULL,
`religion` int(11) DEFAULT NULL,
`occupation` int(11) DEFAULT NULL,
`education` int(11) DEFAULT NULL,
`lang_1` int(11) DEFAULT NULL,
`lang_1_rate` int(11) DEFAULT NULL,
`lang_2` int(11) DEFAULT NULL,
`lang_2_rate` int(11) DEFAULT NULL,
`lang_3` int(11) DEFAULT NULL,
`lang_3_rate` int(11) DEFAULT NULL,
`lang_4` int(11) DEFAULT NULL,
`lang_4_rate` int(11) DEFAULT NULL,
`looking_for` varchar(10) DEFAULT NULL,
`age_from` int(11) DEFAULT NULL,
`age_to` int(11) DEFAULT NULL,
`general_info` text,
`appearance_info` text,
`looking_for_info` text,
`status` int(11) DEFAULT NULL,
`finish_status` int(11) DEFAULT NULL,
`not_newbie` int(11) DEFAULT NULL,
`lastlogin` int(10) NOT NULL DEFAULT '0',
`zipcode` varchar(5) NOT NULL DEFAULT '',
`longitude` double DEFAULT NULL,
`latitude` double DEFAULT NULL,
`photo_pass` varchar(25) NOT NULL DEFAULT '',
`view_count` int(11) DEFAULT '0',
`wants_kids` int(11) DEFAULT NULL,
`kids_okay` int(11) DEFAULT NULL,
`relocate_domestic` int(11) DEFAULT NULL,
`relocate_international` int(11) DEFAULT NULL,
`pioneer` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `member_id` (`member_id`),
KEY `country` (`country`),
KEY `state` (`state`),
KEY `city` (`city`),
KEY `email` (`email`),
KEY `name` (`name`),
KEY `gender` (`gender`),
KEY `birth_day` (`birth_day`),
KEY `birth_month` (`birth_month`),
KEY `birth_year` (`birth_year`),
KEY `marital_status` (`marital_status`),
KEY `children` (`children`),
KEY `drinking` (`drinking`),
KEY `smoking` (`smoking`),
KEY `food` (`food`),
KEY `eye_color` (`eye_color`),
KEY `hair_color` (`hair_color`),
KEY `height` (`height`),
KEY `body_type` (`body_type`),
KEY `race` (`race`),
KEY `religion` (`religion`),
KEY `occupation` (`occupation`),
KEY `education` (`education`),
KEY `lang_1` (`lang_1`),
KEY `lang_1_rate` (`lang_1_rate`),
KEY `lang_2` (`lang_2`),
KEY `lang_2_rate` (`lang_2_rate`),
KEY `lang_3` (`lang_3`),
KEY `lang_3_rate` (`lang_3_rate`),
KEY `lang_4` (`lang_4`),
KEY `lang_4_rate` (`lang_4_rate`),
KEY `looking_for` (`looking_for`),
KEY `age_from` (`age_from`),
KEY `age_to` (`age_to`),
KEY `status` (`status`),
KEY `finish_status` (`finish_status`),
KEY `not_newbie` (`not_newbie`),
KEY `lastlogin` (`lastlogin`),
KEY `zipcode` (`zipcode`),
KEY `longitude` (`longitude`),
KEY `latitude` (`latitude`),
KEY `photo_pass` (`photo_pass`),
KEY `view_count` (`view_count`),
KEY `wants_kids` (`wants_kids`),
KEY `kids_okay` (`kids_okay`),
KEY `relocate_domestic` (`relocate_domestic`),
KEY `relocate_international` (`relocate_international`),
KEY `pioneer` (`pioneer`)
) ENGINE=MyISAM AUTO_INCREMENT=18389 DEFAULT CHARSET=latin1
CREATE TABLE `dt_privacy` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`member_id` int(11) DEFAULT NULL,
`online_yn` char(1) DEFAULT NULL,
`vkiss_yn` char(1) DEFAULT NULL,
`profiles_yn` char(1) DEFAULT NULL,
`IM_yn` char(1) DEFAULT NULL,
`featured_yn` char(1) DEFAULT NULL,
`HL_messaged_yn` char(1) DEFAULT NULL,
`HL_im_yn` char(1) DEFAULT NULL,
`HL_viewed_yn` char(1) DEFAULT NULL,
`HL_kissed_yn` char(1) DEFAULT NULL,
`HL_favorite_yn` char(1) DEFAULT NULL,
`unsearchable` char(1) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `member_id` (`member_id`),
KEY `online_yn` (`online_yn`),
KEY `vkiss_yn` (`vkiss_yn`),
KEY `profiles_yn` (`profiles_yn`),
KEY `IM_yn` (`IM_yn`),
KEY `featured_yn` (`featured_yn`),
KEY `HL_messaged_yn` (`HL_messaged_yn`),
KEY `HL_im_yn` (`HL_im_yn`),
KEY `HL_viewed_yn` (`HL_viewed_yn`),
KEY `HL_kissed_yn` (`HL_kissed_yn`),
KEY `HL_favorite_yn` (`HL_favorite_yn`),
KEY `unsearchable` (`unsearchable`)
) ENGINE=MyISAM AUTO_INCREMENT=26305 DEFAULT CHARSET=latin1
CREATE TABLE `dt_countries` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(100) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `name` (`name`)
) ENGINE=MyISAM AUTO_INCREMENT=226 DEFAULT CHARSET=latin1
CREATE TABLE `dt_photos` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`member_id` varchar(255) DEFAULT NULL,
`filename_1` varchar(255) DEFAULT NULL,
`filename_2` varchar(255) DEFAULT NULL,
`filename_3` varchar(255) NOT NULL DEFAULT '',
`filename_4` varchar(255) NOT NULL DEFAULT '',
`filename_5` varchar(255) NOT NULL DEFAULT '',
`filename_6` varchar(255) NOT NULL DEFAULT '',
`filename_7` varchar(255) NOT NULL DEFAULT '',
`filename_8` varchar(255) NOT NULL DEFAULT '',
`filename_9` varchar(255) NOT NULL DEFAULT '',
`filename_10` varchar(255) NOT NULL DEFAULT '',
`filename_11` varchar(255) NOT NULL DEFAULT '',
`filename_12` varchar(255) NOT NULL DEFAULT '',
`filename_13` varchar(255) NOT NULL DEFAULT '',
`filename_14` varchar(255) NOT NULL DEFAULT '',
`filename_15` varchar(255) NOT NULL DEFAULT '',
`filename_16` varchar(255) NOT NULL DEFAULT '',
`filename_17` varchar(255) NOT NULL DEFAULT '',
`filename_18` varchar(255) NOT NULL DEFAULT '',
`filename_19` varchar(255) NOT NULL DEFAULT '',
`filename_20` varchar(255) NOT NULL DEFAULT '',
`private_1` tinyint(1) NOT NULL DEFAULT '0',
`private_2` tinyint(1) NOT NULL DEFAULT '0',
`private_3` tinyint(1) NOT NULL DEFAULT '0',
`private_4` tinyint(1) NOT NULL DEFAULT '0',
`private_5` tinyint(1) NOT NULL DEFAULT '0',
`private_6` tinyint(1) NOT NULL DEFAULT '0',
`private_7` tinyint(1) NOT NULL DEFAULT '0',
`private_8` tinyint(1) NOT NULL DEFAULT '0',
`private_9` tinyint(1) NOT NULL DEFAULT '0',
`private_10` tinyint(1) NOT NULL DEFAULT '0',
`private_11` tinyint(1) NOT NULL DEFAULT '0',
`private_12` tinyint(1) NOT NULL DEFAULT '0',
`private_13` tinyint(1) NOT NULL DEFAULT '0',
`private_14` tinyint(1) NOT NULL DEFAULT '0',
`private_15` tinyint(1) NOT NULL DEFAULT '0',
`private_16` tinyint(1) NOT NULL DEFAULT '0',
`private_17` tinyint(1) NOT NULL DEFAULT '0',
`private_18` tinyint(1) NOT NULL DEFAULT '0',
`private_19` tinyint(1) NOT NULL DEFAULT '0',
`private_20` tinyint(1) NOT NULL DEFAULT '0',
`password` varchar(255) NOT NULL DEFAULT '',
`description_1` varchar(255) DEFAULT NULL,
`description_2` varchar(255) DEFAULT NULL,
`description_3` varchar(255) NOT NULL DEFAULT '',
`description_4` varchar(255) NOT NULL DEFAULT '',
`description_5` varchar(255) NOT NULL DEFAULT '',
`description_6` varchar(255) NOT NULL DEFAULT '',
`description_7` varchar(255) NOT NULL DEFAULT '',
`description_8` varchar(255) NOT NULL DEFAULT '',
`description_9` varchar(255) NOT NULL DEFAULT '',
`description_10` varchar(255) NOT NULL DEFAULT '',
`description_11` varchar(255) NOT NULL DEFAULT '',
`description_12` varchar(255) NOT NULL DEFAULT '',
`description_13` varchar(255) NOT NULL DEFAULT '',
`description_14` varchar(255) NOT NULL DEFAULT '',
`description_15` varchar(255) NOT NULL DEFAULT '',
`description_16` varchar(255) NOT NULL DEFAULT '',
`description_17` varchar(255) NOT NULL DEFAULT '',
`description_18` varchar(255) NOT NULL DEFAULT '',
`description_19` varchar(255) NOT NULL DEFAULT '',
`description_20` varchar(255) NOT NULL DEFAULT '',
PRIMARY KEY (`id`),
KEY `filename_12` (`filename_12`),
KEY `filename_13` (`filename_13`),
KEY `filename_14` (`filename_14`),
KEY `filename_15` (`filename_15`),
KEY `filename_16` (`filename_16`),
KEY `filename_17` (`filename_17`),
KEY `filename_18` (`filename_18`),
KEY `filename_19` (`filename_19`),
KEY `filename_20` (`filename_20`),
KEY `member_id` (`member_id`),
KEY `filename_1` (`filename_1`),
KEY `filename_2` (`filename_2`),
KEY `filename_3` (`filename_3`),
KEY `filename_4` (`filename_4`),
KEY `filename_5` (`filename_5`),
KEY `filename_6` (`filename_6`),
KEY `filename_7` (`filename_7`),
KEY `filename_8` (`filename_8`),
KEY `filename_9` (`filename_9`),
KEY `filename_10` (`filename_10`),
KEY `private_1` (`private_1`),
KEY `private_2` (`private_2`),
KEY `private_3` (`private_3`),
KEY `private_4` (`private_4`),
KEY `private_5` (`private_5`),
KEY `private_6` (`private_6`),
KEY `private_7` (`private_7`),
KEY `private_8` (`private_8`),
KEY `private_9` (`private_9`),
KEY `private_10` (`private_10`),
KEY `private_11` (`private_11`),
KEY `private_12` (`private_12`),
KEY `private_13` (`private_13`),
KEY `private_14` (`private_14`),
KEY `private_15` (`private_15`),
KEY `private_16` (`private_16`),
KEY `private_17` (`private_17`),
KEY `private_18` (`private_18`),
KEY `private_19` (`private_19`),
KEY `private_20` (`private_20`),
KEY `password` (`password`),
KEY `description_1` (`description_1`),
KEY `description_2` (`description_2`),
KEY `description_3` (`description_3`),
KEY `description_4` (`description_4`),
KEY `description_5` (`description_5`),
KEY `description_6` (`description_6`),
KEY `description_7` (`description_7`),
KEY `description_8` (`description_8`),
KEY `description_9` (`description_9`),
KEY `description_10` (`description_10`),
KEY `description_11` (`description_11`),
KEY `description_12` (`description_12`),
KEY `description_13` (`description_13`),
KEY `description_14` (`description_14`),
KEY `description_15` (`description_15`),
KEY `description_16` (`description_16`),
KEY `description_17` (`description_17`),
KEY `description_18` (`description_18`),
KEY `description_19` (`description_19`),
KEY `description_20` (`description_20`),
KEY `filename_10_2` (`filename_10`),
KEY `filename_10_3` (`filename_10`)
) ENGINE=MyISAM AUTO_INCREMENT=11174 DEFAULT CHARSET=latin1
CREATE TABLE `dt_usersonline` (
`id` int(8) NOT NULL AUTO_INCREMENT,
`timestamp` int(15) NOT NULL DEFAULT '0',
`ip` varchar(40) NOT NULL DEFAULT '',
`login` varchar(25) NOT NULL DEFAULT '',
`userid` int(10) NOT NULL DEFAULT '0',
`session_id` varchar(255) NOT NULL DEFAULT '',
PRIMARY KEY (`id`),
KEY `id_2` (`id`),
KEY `timestamp` (`timestamp`),
KEY `ip` (`ip`),
KEY `login` (`login`),
KEY `userid` (`userid`),
KEY `session_id` (`session_id`)
) ENGINE=MyISAM AUTO_INCREMENT=4424348 DEFAULT CHARSET=latin1
CREATE TABLE `dt_members` (
`id` int(11) NOT NULL AUTO_INCREMENT,...
CREATE TABLE `dt_photos` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`member_id` varchar(255) DEFAULT NULL,..
You are joining on your query as:
LEFT JOIN dt_photos AS ph ON m.id=ph.member_id
Notice how every other time you join on member_id to m.id the column looks like member_id int(11) DEFAULT NULL. Your join criteria should be on the same data type, otherwise it looks like it is doing a full table scan. It's not using the key because it believes that it has to scan the varchar field rather than utilizing indexes for on clause.
Try Altering column member_id to that of m.id from dt_members
It is interesting to note, had you actually tried to create a foreign key constraint, it would not have allowed you to because of the mismatched data types...
Warning: MYISAM table will be locked during the duration of the later table process.
Try moving where conditions into the corresponding join condition:
SELECT DISTINCT m.id AS memberID , m.login , m.age , p.gender
, p.name AS header , p.id AS profileID , p.city , p.state , p.lastlogin
, o.login AS online , c.name AS country , ph.filename_1 AS pic
FROM dt_members AS m
INNER JOIN dt_profile_approved AS p ON m.id=p.member_id
AND p.status=1 AND p.gender='Female' -- Moved from WHERE clause
LEFT JOIN dt_privacy AS pv ON m.id=pv.member_id
AND (pv.unsearchable IS NULL OR pv.unsearchable='') -- Moved from WHERE clause
INNER JOIN dt_countries AS c ON c.id=p.country
LEFT JOIN dt_photos AS ph ON m.id=ph.member_id
LEFT JOIN dt_usersonline AS o ON m.login=o.login
WHERE m.age BETWEEN 25 AND 40
ORDER BY p.lastlogin DESC
LIMIT 0, 21;
While where conditions are made after all joins, join conditions are evaluated during the join, so vast numbers of unnecessary joins can be avoided early.
Although theoretically the query optimizer should do this for you, I have found the mysql can be particularly dense in this regard.
It looks like most of the selectivity on your query is on your dt_profile_approved table. Try creating a compound index on (status, gender, lastlogin)
This should allow an index-only selection and ordering.
Your query does the notorious SELECT a lot ORDER BY something DESC LIMIT tinynumber pattern. This is expensive. Try a deferred join. Start by getting the interesting items from your dt_profile_approved table, like this:
SELECT member_id, lastlogin
FROM dt_profile_approved
WHERE status=1
AND gender='Female'
ORDER BY lastlogin DESC
This subquery can be optimized very cleanly with a compound index on (status, gender, lastlogin, member_id). That's called a covering index. It has a great benefit: no extra sorting is required, because the index is already sorted.
Based on the exact query you showed us, the order in the index of gender and status doesn't matter. But I guess you have another query that looks for males, and you may have one that omits that. So status is likely to be the more selective field across all your queries. (Guessing.)
Then, join that subquery into the rest of your query... that will look something like this.
SELECT DISTINCT
m.id AS memberID , m.login , m.age,
p.gender. p.name AS header , p.id AS profileID ,
p.city , p.state , p.lastlogin,
o.login AS online ,
c.name AS country , ph.filename_1 AS pic
FROM (
SELECT member_id, lastlogin
FROM dt_profile_approved
WHERE status=1
AND gender='Female'
ORDER BY lastlogin DESC
) AS sel
INNER JOIN dt_profile_approved AS p ON sel.member_id = p.member_id
INNER JOIN dt_members AS m ON m.id=sel.member_id
LEFT JOIN dt_privacy AS pv ON m.id=pv.member_id
INNER JOIN dt_countries AS c ON c.id=p.country
LEFT JOIN dt_photos AS ph ON m.id=ph.member_id
LEFT JOIN dt_usersonline AS o ON m.login=o.login
WHERE (pv.unsearchable IS NULL OR pv.unsearchable='')
AND m.age BETWEEN 25 AND 40
ORDER BY sel.lastlogin DESC
LIMIT 0, 21;
If this works, it will be because it can limit the labor of sorting using the index.

fix Using index, Using temporary, Using filesort

i have such 3 tables. Im trying to Left Join them
SELECT `t`.`title` AS `category_title`,`t`.`id` AS `category_id`, `st`.`title` AS
`subcategory_title`, `st`.`id` AS `subcategory_id`, `st`.`parent_id` AS
`subcategory_parent`, `n`.`title` AS `news_title`,`n`.`id` AS `news_id` FROM
`t_categories` `t` LEFT JOIN t_categories AS `st` ON `st`.`parent_id`=t.`id` LEFT JOIN
t_newsrelations AS `nr` ON `nr`.`category_id`=st.`id` LEFT JOIN t_news AS `n` ON
`n`.`id`=nr.`news_id` WHERE `t`.`enabled` = 1 AND `n`.`enabled` = 1 AND `n`.`type`!=1 AND
`n`.`type`!=5 ORDER BY `t`.`position`,`st`.`position`,`n`.`position` ASC
here is the structure of tables
CREATE TABLE IF NOT EXISTS `t_categories` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`parent_id` int(11) NOT NULL,
`title` tinytext NOT NULL,
`position` tinyint(4) unsigned NOT NULL DEFAULT '0',
`type` tinyint(1) unsigned NOT NULL,
`enabled` tinyint(1) unsigned NOT NULL DEFAULT '1',
UNIQUE KEY `id` (`id`),
KEY `type` (`type`),
KEY `parent_id` (`parent_id`),
KEY `enabled` (`enabled`),
KEY `id_parent_position_enabled` (`id`,`parent_id`,`position`,`enabled`),
KEY `position` (`position`),
KEY `parent_id_2` (`parent_id`,`enabled`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 ;
CREATE TABLE IF NOT EXISTS `t_news` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`title` tinytext NOT NULL,
`m_title` tinytext NOT NULL,
`url` varchar(2000) NOT NULL,
`keywords` text NOT NULL,
`description` text NOT NULL,
`body` longtext NOT NULL,
`position` tinyint(4) unsigned NOT NULL DEFAULT '0',
`type` tinyint(1) unsigned NOT NULL,
`city_id` int(4) NOT NULL,
`quickmenu_enabled` tinyint(1) unsigned NOT NULL DEFAULT '0',
`quickmenu` text NOT NULL,
`enabled` tinyint(1) unsigned NOT NULL DEFAULT '1',
PRIMARY KEY (`id`),
UNIQUE KEY `id` (`id`),
KEY `position` (`position`),
KEY `type` (`type`),
KEY `city_id` (`city_id`),
KEY `url` (`url`(333)),
KEY `quickmenu_enabled` (`quickmenu_enabled`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 ;
CREATE TABLE IF NOT EXISTS `t_newsrelations` (
`category_id` int(11) NOT NULL,
`news_id` int(11) unsigned NOT NULL,
KEY `category_id` (`category_id`),
KEY `news_id` (`news_id`),
KEY `category_id_2` (`category_id`,`news_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
And SELECT EXPLAIN shows me
t_newsrelations is intermediate table. Table t_categories contains categories and subcategories linked by parent_id column. Each item from t_news can be a member of more than one subcategories thats why they linked through t_newsrelations
how to optimise a query? Why it shows Using index, Using temporary, Using filesort?
ORDER BY `t`.`position`,`st`.`position`,`n`.`position` ASC
You can't eliminate the temp table and filesort in this query, given the tables you have, because you're sorting on columns from multiple tables. Optimizing sorting means using an index so that the query fetches rows in the order you want them. But there's no way in MySQL to create an index that spans multiple tables.
The only way to fix this is to denormalize until all three columns are in a single table, and create one index over the three columns. But denormalization comes with its own downsides.

Speeding up multi-table joins with MySQL

I have 3 tables in which I'm trying to preform joins on, and inserting the resulting data into another table. The query is taking anywhere between 15-30 mins depending on the dataset. The tables I'm selecting from and joining on are at least 25k records each but will quickly grow to be 500k+.
I tried adding indexes on the fields but still isn't helping that much. Are there any other things I can try or are joins on this scale just going to take this long?
Here is the query I'm trying to perform:
INSERT INTO audience.topitem
(runs_id, total_training_count, item, standard_index_value, significance, seed_count, nonseed_count, prod, model_type, level_1, level_2, level_3, level_4, level_5)
SELECT 5, seed_count + nonseed_count AS total_training_count,
ii.item, standard_index_value, NULL, seed_count, nonseed_count,
standard_index_value * seed_count AS prod, 'site', topic_L1, topic_L2, topic_L3, topic_L4, topic_L5
FROM audience.item_indexes ii
LEFT JOIN audience.usercounts uc ON ii.item = uc.item AND ii.runs_id = uc.runs_id
LEFT JOIN categorization.categorization at on ii.item = at.url
WHERE ii.runs_id = 5
Table: audience.item_indexes
CREATE TABLE `item_indexes` (
`item` varchar(1024) DEFAULT NULL,
`standard_index_value` float DEFAULT NULL,
`runs_id` int(11) DEFAULT NULL,
`model_type` enum('site','term','combo') DEFAULT NULL,
KEY `item_idx` (`item`(333))
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
Table: audience.usercounts
CREATE TABLE `usercounts` (
`item` varchar(1024) DEFAULT NULL,
`seed_count` int(11) DEFAULT NULL,
`nonseed_count` int(11) DEFAULT NULL,
`significance` float(19,6) DEFAULT NULL,
`runs_id` int(11) DEFAULT NULL,
`model_type` enum('site','term','combo') DEFAULT NULL,
KEY `item_idx` (`item`(333))
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
Table: audience.topitem
CREATE TABLE `topitem` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`total_training_count` int(11) DEFAULT NULL,
`item` varchar(1024) DEFAULT NULL,
`standard_index_value` float(19,6) DEFAULT NULL,
`significance` float(19,6) DEFAULT NULL,
`seed_count` int(11) DEFAULT NULL,
`nonseed_count` int(11) DEFAULT NULL,
`prod` float(19,6) DEFAULT NULL,
`cat_type` varchar(32) DEFAULT NULL,
`cat_level` int(11) DEFAULT NULL,
`conf` decimal(19,9) DEFAULT NULL,
`level_1` varchar(64) DEFAULT NULL,
`level_2` varchar(64) DEFAULT NULL,
`level_3` varchar(64) DEFAULT NULL,
`level_4` varchar(64) DEFAULT NULL,
`level_5` varchar(64) DEFAULT NULL,
`runs_id` int(11) DEFAULT NULL,
`model_type` enum('site','term','combo') DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=825 DEFAULT CHARSET=utf8;
Table: categorization.categorization
CREATE TABLE `AT_categorization` (
`url` varchar(760) NOT NULL ,
`language` varchar(10) DEFAULT NULL,
`category` text,
`entity` text,
`source` varchar(255) DEFAULT NULL,
`topic_L1` varchar(45) NOT NULL DEFAULT '',
`topic_L2` varchar(45) NOT NULL DEFAULT '',
`topic_L3` varchar(45) NOT NULL DEFAULT '',
`topic_L4` varchar(45) NOT NULL DEFAULT '',
`topic_L5` varchar(45) NOT NULL DEFAULT '',
`last_refreshed` datetime DEFAULT NULL,
PRIMARY KEY (`url`,`topic_L1`,`topic_L2`,`topic_L3`,`topic_L4`,`topic_L5`),
UNIQUE KEY `inx_url` (`url`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
If you add the following indexes, your query will run faster:
CREATE INDEX runs_idx ON audience.item_indexes (runs_id);
ALTER TABLE audience.usercounts
DROP INDEX item_idx,
ADD INDEX item_idx (runs_id, item(333));
Also, item_indexes is utf8, but AT_categorization is latin1, which keeps any indexes from being used. To address this issue, change AT_categorization to utf8:
ALTER TABLE AT_categorization CHARSET=utf8;
Lastly, for the AT_categorization table, the two indexes
PRIMARY KEY (`url`,`topic_L1`,`topic_L2`,`topic_L3`,`topic_L4`,`topic_L5`),
UNIQUE KEY `inx_url` (`url`)
are redundant. So you could DROP these, and simply have the url field be the primary key:
ALTER TABLE AT_categorization
DROP PRIMARY KEY,
DROP KEY `inx_url`,
ADD PRIMARY KEY (url);