What is the best optimization for this table and its queries? - mysql

Question

I have this table:
CREATE TABLE IF NOT EXISTS `listings` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`type` tinyint(1) NOT NULL DEFAULT '1',
`hash` char(32) NOT NULL,
`source_id` int(10) unsigned NOT NULL,
`link` varchar(255) NOT NULL,
`short_link` varchar(255) NOT NULL,
`cat_id` mediumint(5) NOT NULL,
`title` mediumtext NOT NULL,
`description` mediumtext,
`content` mediumtext,
`images` mediumtext,
`videos` mediumtext,
`views` int(10) NOT NULL,
`comments` int(11) NOT NULL DEFAULT '0',
`comments_update` int(11) NOT NULL DEFAULT '0',
`editor_id` int(11) NOT NULL DEFAULT '0',
`auther_name` varchar(255) DEFAULT NULL,
`createdby_id` int(10) NOT NULL,
`createdon` int(20) NOT NULL,
`editedby_id` int(10) NOT NULL,
`editedon` int(20) NOT NULL,
`deleted` tinyint(1) NOT NULL,
`deletedon` int(20) NOT NULL,
`deletedby_id` int(10) NOT NULL,
`deletedfor` varchar(255) NOT NULL,
`published` tinyint(1) NOT NULL DEFAULT '1',
`publishedon` int(20) NOT NULL,
`publishedby_id` int(10) NOT NULL,
PRIMARY KEY (`id`),
KEY `hash` (`hash`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 ROW_FORMAT=COMPACT AUTO_INCREMENT=91628 ;
and some bad queries like this
SELECT id,type , source_id, link, short_link, cat_id, title,
description, images, views, comments, published, publishedon,
content, comments_update, editor_id, auther_name, createdby_id,
createdon, editedby_id, editedon, deleted, deletedon,
deletedby_id, deletedfor, publishedby_id
FROM listings
WHERE (cat_id IN ( 98 ))
AND (TYPE IN ('1'))
AND (source_id IN ('78'))AND (FROM_UNIXTIME( publishedon ) >= SUBDATE( NOW( ) ,
1 )
)
AND (deleted =0)
AND (published =1)
ORDER BY `publishedon` DESC
LIMIT 10 OFFSET 0
and
SELECT id,type,source_id,link,short_link,cat_id,title,description,
images,views,comments,published,publishedon
FROM listings
WHERE (title RLIKE 'اليمن|عدن')
AND (id != 89024)
AND (deleted = 0)
AND (published = 1)
ORDER BY publishedon DESC
LIMIT 6 OFFSET 0
and
SELECT MIN(id) FROM listings
WHERE (id > 91152) AND (cat_id = '134')
and
SELECT COUNT(id)
FROM listings
WHERE (publishedon >= '1442963362'
AND publishedon <= '1443568162'
)
AND (cat_id IN ('19', '20', '21', '22', '23', '24', '27',
'32', '35', '110', '54', '38', '39', '41', '42', '43',
'44', '45', '46', '47', '49', '56', '57', '51', '55',
'58', '59', '60', '61', '62', '102', '95', '96', '98',
'101', '103', '104', '105', '106', '124', '125', '130',
'131', '132', '133', '134', '135')
)
this query may takes 0.4 sec to be done. Any in one page it may contain 5 queries like this. This is very big problem; it causes server load and down time.
This query
SELECT *
FROM `listings`
WHERE id = 5455
takes 0.0004 sec to be done because its depend on the index of pk
How can I make indexes for columns in the first query?
And many times when I use "show processlist", I see this "Waiting for table level lock" too much and "sorting data".
The application insert/update many rows all the times; how can I solve this?

Answer 1

Your query is basically:
SELECT . . .
FROM listings l
WHERE cat_id = 98 AND
TYPE = 1 AND
source_id = 78 AND
deleted = 0 AND
published = 1 AND
FROM_UNIXTIME( publishedon ) >= SUBDATE( NOW( ) , 1 )
ORDER BY `publishedon` DESC
LIMIT 10
OFFSET 0
For performance, start with a composite index on (cat_id, type, source_id, deleted, published, publishedon). The syntax is:
create index idx_listings_6 on listings(cat_id, type, source_id, deleted, published, publishedon desc);
Next, I would suggest re-writing the where clause as:
SELECT . . .
FROM listings l
WHERE cat_id = 98 AND
TYPE = 1 AND
source_id = 78 AND
deleted = 0 AND
published = 1 AND
publishedon >= UNIX_TIMESTAMP(SUBDATE( NOW( ) , 1 ))
ORDER BY `publishedon` DESC
LIMIT 10
OFFSET 0
And use the same index above.

Answer 2

MySQL cannot declare an index to be DESC; it ignores that keyword and builds an ASC index. However, ORDER BY x DESC is still optimizable.
MySQL will use only one INDEX per SELECT. (Usually).
The optimal index starts with any WHERE clause fields with '='.
Query 1: Already discussed.
Query 2: INDEX(deleted, published, publishedon)
Query 3: INDEX(cat_id, id)
Query 4: Change COUNT(id) to COUNT(*) and add either INDEX(cat_id, publishedon) and/or INDEX(publishedon, cat_id). It is not obvious which index would be better. Add both and let the optimizer decide. As in your other question, it may be beneficial to PARTITION BY RANGE(publishedon) and use INDEX(cat_id, publishedon) (not the other one).
Consider switching to InnoDB.