optimizing mysql slow query - mysql

Question

The more records are added to database the query becomes slower. From 1 sec. to few seconds now, in result webpage load time is way to long
CREATE TABLE `ads` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) NOT NULL,
`user_status` enum('register','unregister') COLLATE latin1_general_ci NOT NULL DEFAULT 'register',
`title` varchar(255) COLLATE latin1_general_ci NOT NULL,
`tags` varchar(255) COLLATE latin1_general_ci NOT NULL,
`ad_type` enum('offer','want') COLLATE latin1_general_ci NOT NULL,
`price` float NOT NULL,
`image` varchar(255) COLLATE latin1_general_ci NOT NULL,
`address` varchar(255) COLLATE latin1_general_ci NOT NULL,
`google_address` varchar(255) COLLATE latin1_general_ci NOT NULL,
`country_id` int(11) NOT NULL,
`state_id` int(11) NOT NULL,
`address2` text COLLATE latin1_general_ci NOT NULL,
`city` varchar(255) COLLATE latin1_general_ci NOT NULL,
`location` int(11) NOT NULL,
`postal_code` varchar(255) COLLATE latin1_general_ci NOT NULL,
`Latitude` varchar(255) COLLATE latin1_general_ci NOT NULL,
`Longitude` varchar(255) COLLATE latin1_general_ci NOT NULL,
`working_remote` varchar(255) COLLATE latin1_general_ci NOT NULL,
`emergency_service` varchar(255) COLLATE latin1_general_ci NOT NULL,
`ad_description` text CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
`cat_id` int(11) NOT NULL,
`sub_cat_id` int(11) NOT NULL,
`sub_sub_cat_id` int(11) NOT NULL,
`status` enum('0','1') COLLATE latin1_general_ci NOT NULL,
`delete_status` enum('0','1') COLLATE latin1_general_ci NOT NULL DEFAULT '0',
`publication_days` varchar(255) COLLATE latin1_general_ci NOT NULL,
`publication_total` float(11,2) NOT NULL,
`added_date` datetime NOT NULL,
`expiry_date` datetime NOT NULL,
`payment_status` enum('pending','paid','cancel') COLLATE latin1_general_ci NOT NULL,
`closed_date` datetime NOT NULL,
`deleted_date` datetime NOT NULL,
`ad_status` enum('active','closed') COLLATE latin1_general_ci NOT NULL DEFAULT 'active',
`user_first_name` varchar(255) COLLATE latin1_general_ci NOT NULL,
`user_last_name` varchar(255) COLLATE latin1_general_ci NOT NULL,
`user_phone_number` varchar(255) COLLATE latin1_general_ci NOT NULL,
`user_email_id` varchar(255) COLLATE latin1_general_ci NOT NULL,
`ads_extend_date` datetime NOT NULL,
`ads_extend_expiry_date` datetime NOT NULL,
`ads_extend_status` enum('yes','no') COLLATE latin1_general_ci NOT NULL DEFAULT 'no',
`actvation_notification` enum('yes','no') COLLATE latin1_general_ci NOT NULL DEFAULT 'no',
`ads_view_count` int(11) NOT NULL,
`md5_key` varchar(100) COLLATE latin1_general_ci NOT NULL,
PRIMARY KEY (`id`),
KEY `user_id` (`user_id`,`user_status`,`title`,`ad_type`,`price`),
KEY `title` (`title`),
KEY `ad_type` (`ad_type`),
KEY `price` (`price`),
KEY `google_address` (`google_address`),
KEY `country_id` (`country_id`),
KEY `state_id` (`state_id`),
KEY `city` (`city`),
KEY `postal_code` (`postal_code`),
KEY `cat_id` (`cat_id`),
KEY `sub_cat_id` (`sub_cat_id`),
KEY `sub_sub_cat_id` (`sub_sub_cat_id`),
KEY `status` (`status`),
KEY `payment_status` (`payment_status`),
KEY `ad_status` (`ad_status`),
KEY `added_date` (`added_date`),
KEY `expiry_date` (`expiry_date`),
KEY `id_2` (`id`,`user_id`,`user_status`,`title`,`ad_type`,`country_id`,`state_id`,`city`,`postal_code`,`cat_id`,`sub_cat_id`,`sub_sub_cat_id`,`added_date`,`expiry_date`,`payment_status`,`ad_status`)
) ENGINE=MyISAM AUTO_INCREMENT=1208 DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci
Slow query log
# Query_time: 3.859838 Lock_time: 0.000368 Rows_sent: 340 Rows_examined: 1248768
SET timestamp=1448331158;
SELECT ads.id,ads.user_id, ads.user_status, ads.title, ads.ad_type,ads.price, ads.address, ads.google_address, ads.state_id, ads.address2, ads.city as city_id, ads. location as location_id, ads.postal_code, ads.Latitude, ads.Longitude, ads.working_remote, ads.emergency_service, ads.ad_description, ads.cat_id, ads. sub_cat_id, ads.sub_sub_cat_id, ads.status, ads.publication_total, ads.ads_view_count, ads.added_date,cat.category_name, sub_cat.category_name as sub_category_name, sub_sub_cat.category_name as sub_sub_category_name, usr.id as user_id, usr.username as user_name, usr.first_name as first_name, usr.rating as rating, adimg.thumbnail, state.state,state.state_abbr, city.city, location.location as locationname,
(SELECT added_date
FROM ads_publication as pub
WHERE pub.ad_id = ads.id
AND pub.publication_id != '0'
ORDER BY pub.sort_type ASC LIMIT 0,1) as publication_srt_id,
SQRT((((69.1*(ads.Latitude -(0)))*(69.1*(ads.Latitude -(0))))+((53*(ads.Longitude -(0)))*(53*(ads.Longitude -(0)))))) as dist_in_miles
FROM ads as ads
LEFT JOIN ads_images as adimg ON (ads.id = adimg.ad_id AND default_image = '1')
LEFT JOIN workrange as wr ON ads.user_id = wr.user_id
LEFT JOIN users as usr ON ads.user_id = usr.id
LEFT JOIN ads_service as price_list ON ads.id = price_list.ad_id
LEFT JOIN ads_publication as promot ON ads.id = promot.ad_id
LEFT JOIN user_languages as language ON ads.id = language.ad_id
LEFT JOIN categories as cat ON (ads.cat_id = cat.id AND cat.parent_category_id = 0)
LEFT JOIN categories as sub_cat ON ads.sub_cat_id = sub_cat.id
LEFT JOIN categories as sub_sub_cat ON ads.sub_sub_cat_id = sub_cat.id
LEFT JOIN location as state ON ads.state_id = state.locationId
LEFT JOIN location as city ON ads.city = city.locationId
LEFT JOIN location as location ON ads.location = location.locationId
WHERE ads.status = '1'
AND ads.payment_status = 'paid'
AND ads.delete_status = '0'
AND ads.expiry_date >= '2015-11-23 21:12:38'
AND ads.ad_status = 'active'
AND ads.ad_type = 'offer'
GROUP BY ads.id
ORDER BY ads.user_status ASC, publication_srt_id DESC, ads.added_date DESC;
Explain
id | select_type |table | type | possible_keys | key | key_len | ref | rows | Extra
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1 | PRIMARY | ads | index_merge | "ad_type,status,payment_status,ad_status,expiry_dat..." | "status,ad_status" | "1,1" | NULL | 173 | "Using intersect(status,ad_status); Using where; Us..."
1 | PRIMARY | adimg | index | NULL | id | 526 | NULL | 1398 | Using index
1 | PRIMARY | wr | ALL | NULL | NULL | NULL | NULL | 75 |
1 | PRIMARY | usr | eq_ref | PRIMARY | PRIMARY | 4 | serv_co_za.ads.user_id | 1 |
1 | PRIMARY | price_list | ALL | NULL | NULL | NULL | NULL | 57 |
1 | PRIMARY | promot | ref | "ad_id,ad_id_2,ad_id_3" | ad_id_3 | 4 | serv_co_za.ads.id | 11 | Using index
1 | PRIMARY | language | ALL | NULL | NULL | NULL | NULL | 393 |
1 | PRIMARY | cat | eq_ref | "PRIMARY,id" | PRIMARY | 4 | serv_co_za.ads.cat_id | 1 |
1 | PRIMARY | sub_cat | eq_ref | "PRIMARY,id" | PRIMARY | 4 | serv_co_za.ads.sub_cat_id | 1 |
1 | PRIMARY | state | eq_ref | PRIMARY | PRIMARY | 4 | serv_co_za.ads.state_id | 1 |
1 | PRIMARY | city | eq_ref | PRIMARY | PRIMARY | 4 | serv_co_za.ads.city | 1 |
1 | PRIMARY | location | eq_ref | PRIMARY | PRIMARY | 4 | serv_co_za.ads.location | 1 |
1 | PRIMARY | sub_sub_cat | index | NULL | id | 111 | NULL | 1193 | Using index
2 | DEPENDENT SUBQUERY | pub | ref | "ad_id,ad_id_2,ad_id_3" | ad_id | 4 | func | 115 | Using where; Using filesort
Config:
key_buffer_size 33554432
max_allowed_packet 268435456
query_cache_limit 1048576
query_cache_min_res_unit 4096
query_cache_size 33554432
myisam_sort_buffer_size 16777216
sort_buffer_size 524288
thread_cache_size 4
thread_concurrency 10
interactive_timeout 28800
wait_timeout 28800
What I noticed is when this part is removed from the very end of query
GROUP BY ads.id ORDER BY ads.user_status ASC, publication_srt_id DESC, ads.added_date DESC;
query time is about 0.06 sec.
Any help or starting point is highly appreciated.
Thank you in advance,
Derek

Answer 1

Using intersect(status,ad_status) -- A composite index will always beat that. So add INDEX(status, ad_status). Assuming those columns are simply flags, get rid of the individual indexes on them. (Get rid of other simple indexes on other status fields.)
WHERE ads.status = '1'
AND ads.payment_status = 'paid'
AND ads.delete_status = '0'
AND ads.expiry_date >= '2015-11-23 21:12:38'
AND ads.ad_status = 'active'
AND ads.ad_type = 'offer'
For that WHERE, this is better:
INDEX(status, payment_status, delete_status, ad_status, at_type, -- in any order
expiry_date) -- deliberately last
This will make the first step more efficient. Index Cookbook explains how I got that.
Remove LEFT unless the 'right' table is really optional. This could give the optimizer more choices on evaluating the query.
wr, price_list, and language need to scan ALL rows. Let's figure out why. They need indexes on user_id, ad_id, and ad_id respectively. And the datatypes must match what you are comparing to.
Don't use (M,N) (eg, float(11,2)) in FLOAT or DOUBLE, it leads to an extra rounding that could cause surprises. For currency, switch to DECIMAL(11,2) (or similar).
Don't use VARCHAR for continuous, numeric, values such as Latitude and Longitude. FLOAT or DOUBLE is good.
Consider moving to InnoDB. MyISAM is dying off.
DROP INDEX id_2 -- it is likely to serve no purpose.
Possible reasons for the query getting slower and slower:
wr, price_list, and language are getting larger. The indexes should cure that.
MyISAM involves table locks.
key_buffer_size should be set about 20% of available ram. As the tables grow, the key_buffer may be thrashing. (Note: a different setting is needed for InnoDB.)
((Edit))
Since Latitude needs to be converted for expression evaluation, it is even more important to use some numeric datatype.
pub needs INDEX(ads_id, sort_type)
publication_id may get in the way of the above INDEX; can you get rid of the test?