MySQL Query is Extremely Slow - mysql

Hello I am looking for ways to optimize the mysql query, basically I am fetching the articles for the user which belong to category_id = 25 and source_id not in a table where I store source id's from which user has unsubscribed.
select
a.article_id,
a.article_title,
a.source_id,
a.article_publish_date,
a.article_details,
n.source_name
from sources n
INNER JOIN articles a
ON (a.source_id = n.source_id)
WHERE n.category_id = 25
AND n.source_id NOT IN(select
source_id
from news_sources_deselected
WHERE user_id = 5)
ORDER BY a.article_publish_date DESC
Schema for Articles Table
CREATE TABLE IF NOT EXISTS `articles` (<br>
`article_id` int(255) NOT NULL auto_increment,<br>
`article_title` varchar(255) NOT NULL,<br>
`source_id` int(255) NOT NULL,<br>
`article_publish_date` bigint(255) NOT NULL,<br>
`article_details` text NOT NULL,<br>
PRIMARY KEY (`article_id`),<br>
KEY `source_id` (`source_id`),<br>
KEY `article_publish_date` (`article_publish_date`)<br>
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Contains articles.';
Structure for Sources table
CREATE TABLE IF NOT EXISTS `sources` (<br>
`source_id` int(255) NOT NULL auto_increment,<br>
`category_id` int(255) NOT NULL,<br>
`source_name` varchar(255) character set latin1 NOT NULL,<br>
`user_id` int(255) NOT NULL,<br>
PRIMARY KEY (`source_id`),<br>
KEY `category_id` (`category_id`),<br>
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='News Sources.'
The articles table has around 0.3 Million records and sources table contains around 1000 records, the query takes around 180 seconds to execute.
Any help will be greatly appreciated.

Try using a derieved query with IS NULL condition. You explain says there is a dependent subquery. Ignore using it and use derieved query for your problem. This will increase the performance
select
a.article_id,
a.article_title,
a.source_id,
a.article_publish_date,
a.article_details,
n.source_name
from sources n
INNER JOIN articles a
ON (a.source_id = n.source_id)
LEFT JOIN (SELECT *
FROM news_sources_deselected
WHERE user_id = 5) AS nsd
ON nsd.source_id = n.source_id
WHERE n.category_id = 25
AND nsd.source_id IS NULL
ORDER BY a.article_publish_date DESC

Use EXPLAIN in front of your query and analyze results.
Here you can find how to start your optimization work.

I see few issues you could check.
You're not using relations despite using InnoDB engine.
You're selecting fields without index.
You're selecting all rows at once.
Do you need all those rows at once? Maybe consider splitting this query to multiple shards (paging)?

Try this query
select
a.article_id,
a.article_title,
a.source_id,
a.article_publish_date,
a.article_details,
n.source_name
from
sources n
INNER JOIN
articles a
ON
n.category_id = 25 AND
a.source_id = n.source_id
INNER JOIN
news_sources_deselected nsd
ON
nsd.user_id <> 5 AND n.source_id = nsd.source_id
ORDER BY
a.article_publish_date DESC
I have removed the extra query and added news_sources_deselected in join by accepting all source_id for user_id other than with id 5.
Or we can go for using only needed records for join as user raheelshan has mentioned
select
a.article_id,
a.article_title,
a.source_id,
a.article_publish_date,
a.article_details,
n.source_name
from
(select
*
from
sources
where
category_id = 25) n
INNER JOIN
articles a
ON
a.source_id = n.source_id
INNER JOIN
(select
*
from
news_sources_deselected
where
user_id <> 5) nsd
ON
n.source_id = nsd.source_id
ORDER BY
a.article_publish_date DESC
Hope this helps..

I fixed the issue by partitioning the table, but I am still open to suggestions.

Related

JOIN query taking long time and creating issue "converting HEAP to MyISAM

My query like below. here I used join query to take data. can u pls suggest how can I solve "converting HEAP to MyISAM" issue.
Can I use subquery here to update it? pls suggest how can I.
Here I have joined users table to check user is exist or not. can I refine it without join so that "converting HEAP to MyISAM" can solve.
Oh one more sometimes I will not check with specific user_id. like here I have added user_id = 16082
SELECT `user_point_logs`.`id`,
`user_point_logs`.`user_id`,
`user_point_logs`.`point_get_id`,
`user_point_logs`.`point`,
`user_point_logs`.`expire_date`,
`user_point_logs`.`point` AS `sum_point`,
IF(sum(`user_point_used_logs`.`point`) IS NULL, 0, sum(`user_point_used_logs`.`point`)) AS `minus`
FROM `user_point_logs`
JOIN `users` ON ( `users`.`id` = `user_point_logs`.`user_id` )
LEFT JOIN (SELECT *
FROM user_point_used_logs
WHERE user_point_log_id NOT IN (
SELECT DISTINCT return_id
FROM user_point_logs
WHERE return_id IS NOT NULL
AND user_id = 16082
)
)
AS user_point_used_logs
ON ( `user_point_logs`.`id` = `user_point_used_logs`.`user_point_log_used_id` )
WHERE expire_date >= 1563980400
AND `user_point_logs`.`point` >= 0
AND `users`.`id` IS NOT NULL
AND ( `user_point_logs`.`return_id` = 0
OR `user_point_logs`.`return_id` IS NULL )
AND `user_point_logs`.`user_id` = '16082'
GROUP BY `user_point_logs`.`id`
ORDER BY `user_point_logs`.`expire_date` ASC
DB FIDDLE HERE WITH STRUCTURE
Kindly try this, If it works... will optimize further by adding composite index.
SELECT
upl.id,
upl.user_id,
upl.point_get_id,
upl.point,
upl.expire_date,
upl.point AS sum_point,
coalesce(SUM(upl.point),0) AS minus -- changed from complex to readable
FROM user_point_logs upl
JOIN users u ON upl.user_id = u.id
LEFT JOIN (select supul.user_point_log_used_id from user_point_used_logs supul
left join user_point_logs supl on supul.user_point_log_id=supl.return_id and supl.return_id is null and supl.user_id = 16082) AS upul
ON upl.id=upul.user_point_log_used_id
WHERE
upl.user_id = 16082 and coalesce(upl.return_id,0)= 0
and upl.expire_date >= 1563980400 -- tip: if its unix timestamp change the datatype and if possible use range between dates
#AND upl.point >= 0 -- since its NN by default removing this condition
#AND u.id IS NOT NULL -- removed since the inner join matches not null
GROUP BY upl.id
ORDER BY upl.expire_date ASC;
Edit:
Try adding index in the column return_id on the table user_point_logs.
Since this column is used in join on derived query.
Or use composite index with user_id and return_id
Indexes:
user_point_logs: (user_id, expire_date)
user_point_logs: (user_id, return_id)
OR is hard to optimize. Decide on only one way to say whatever is being said here, then get rid of the OR:
AND ( `user_point_logs`.`return_id` = 0
OR `user_point_logs`.`return_id` IS NULL )
DISTINCT is redundant:
NOT IN ( SELECT DISTINCT ... )
Change
IF(sum(`user_point_used_logs`.`point`) IS NULL, 0,
sum(`user_point_used_logs`.`point`)) AS `minus`
to
COALESCE( ( SELECT SUM(point) FROM user_point_used_logs ... ), 0) AS minus
and toss LEFT JOIN (SELECT * FROM user_point_used_logs ... )
Since a PRIMARY KEY is a key, the second of these is redundant and can be DROPped:
ADD PRIMARY KEY (`id`),
ADD KEY `id` (`id`) USING BTREE;
After all that, we may need another pass to further simplify and optimize it.

Optimize table to avoid using temporary and using filesort

I have a messages table
CREATE TABLE `messages` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`author` int(11) DEFAULT NULL,
`time` int(10) unsigned DEFAULT NULL,
`text` text CHARACTER SET latin1,
`dest` int(11) unsigned DEFAULT NULL,
`type` tinyint(4) unsigned DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `author` (`author`),
KEY `dest` (`dest`)
) ENGINE=InnoDB AUTO_INCREMENT=2758 DEFAULT CHARSET=utf8;
I need to get messages between two users
SELECT
...
FROM
`messages` m
LEFT JOIN `people` p ON m.author = p.id
WHERE
(author = 1 AND dest = 2)
OR (author = 2 AND dest = 1)
ORDER BY
m.id DESC
LIMIT 0, 25
When I EXPLAIN this query I get
Please excuse any ignorance, but is there a way I could optimize this table to avoid using a temporary table and filesort for this query, for now it is not causing a problem but I'm pretty sure in future it is going to be troublesome?
First, I'm guessing the left join is not necessary. Second, consider using union all instead. Then one approach is:
(SELECT ...
FROM messages m JOIN
people p
ON m.author = p.id
WHERE author = 1 AND dest = 2
ORDER BY id DESC
LIMIT 25
)
UNION ALL
(SELECT ...
FROM messages m JOIN
people p
ON m.author = p.id
WHERE author = 2 AND dest = 1
ORDER BY id DESC
LIMIT 25
)
ORDER BY m.id DESC
LIMIT 0, 25
With this query, an index on messages(author, dest, id) should make it fast. (Note: you might need to include m.id in the SELECT list.)
To build on Gordon's answer:
SELECT m2..., p...
FROM
(
( SELECT id
FROM messages
WHERE author = 1
AND dest = 2
ORDER BY id DESC
LIMIT 75
)
UNION ALL
(
SELECT id
FROM messages
WHERE author = 2
AND dest = 1
ORDER BY id DESC
LIMIT 75
)
) ORDER BY id DESC
LIMIT 50, 25 ) AS m1
JOIN messages AS m2 ON m2.id = m1.id
JOIN people p ON p.id = m2.author
ORDER BY m1.id DESC
Notes:
Gordon's index is now "covering". (This adds efficiency, thereby masking some of the other stuff I added.)
Lazy evaluation means that it does not need to shovel all the bulky fields of more than 25 rows around. Instead, only 25 need to be handled. Also, I avoid touching people to start with.
The code shows what "page 3" should look like. Note LIMIT 75 versus LIMIT 50,25.
"Pagination via OFFSET" has several problems. See my blog.
This formulation still will not avoid "filesort" and "using temp". But speed is the real goal, correct? ("Filesort" is a misnomer -- if you don't include that TEXT column, the sort will be done in RAM.)
When you add INDEX(author, dest, id), INDEX(author) becomes redundant; drop it.
The ALL after UNION is not the default for UNION, but it avoids an extra pass (and temp table) to de-duplicate the data.
There will still be 2 or 3 temp tables involved. See EXPLAIN FORMAT=JSON SELECT ... for details.

MYSQL order large database by decimal 10,10

I have about 25 million rows containing 0.183463545, 0.183423545, 0.183443545, 0.183443445, 0.183447545 and so on.
I need to order these however it currently takes around 20 seconds. Any way to speed it up? AFAIK, I have my index put it place correctly.
Thank you!
SELECT `a`.`float_val`,
`a`.`num_id`,
`b`.`userID`,
`c`.`img`,
`c`.`username`,
`b`.`img`,
`d`.`exterior`
FROM `a`
INNER JOIN `b` ON `b`.`num_id` = `a`.`num_id`
INNER JOIN `d` ON `d`.`id` = `b`.`item`
INNER JOIN `c` ON `c`.`userID` = `b`.`userID`
WHERE `float_val` IS NOT NULL
AND `float_val` BETWEEN 0 AND 1
AND `username` = 'ABC'
ORDER BY `float_val` LIMIT 100
Index is on float_val, num_id, userID
CREATE TABLE `float` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`num_id` bigint(11) DEFAULT NULL,
`float_val` decimal(10,10) DEFAULT NULL,
`userID` char(17) DEFAULT NULL,
`last_checked` datetime DEFAULT NULL,
`index10` smallint(11) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `floatID` (`num_id`),
UNIQUE KEY `num_id` (`num_id`,`userID`),
KEY `userID` (`userID`),
KEY `float_val` (`float_val`),
KEY `last_checked` (`last_checked`),
KEY `index10` (`index10`)
) ENGINE=InnoDB AUTO_INCREMENT=25750916 DEFAULT CHARSET=latin1;
Edited to reflect table definitions shown.
Start by doing a query to obtain the lowest float_val rows from table a.
SELECT a.id
FROM a
INNER JOIN b ON b.num_id = a.num_id
INNER JOIN d ON d.id = b.item
INNER JOIN c ON c.userID = b.userID
WHERE c.username = 'ABC'
AND a.float_val BETWEEN 0.0 AND 1.0
ORDER BY a.float_val
LIMIT 100
If you have an index on a(float_val,num_id), and another on c.username this will be reasonably fast. It will spit out the id values for the rows of a that are candidates for your query. (If you're using MyISAM, you need an index on a(float_val, num_id, id). By the way, the BETWEEN clause also implies IS NOT NULL.
Then use that as a subquery to complete your query, as follows.
SELECT a.float_val,
a.num_id,
b.userID,
c.img,
c.username,
b.img,
d.exterior
FROM a
INNER JOIN (
SELECT a.id
FROM a
INNER JOIN b ON b.num_id = a.num_id
INNER JOIN d ON d.id = b.item
INNER JOIN c ON c.userID = b.userID
WHERE c.username = 'ABC'
AND a.float_val BETWEEN 0.0 AND 1.0
ORDER BY a.float_val
LIMIT 100
) q ON a.ID = q.id
INNER JOIN b ON b.num_id = a.num_id
INNER JOIN d ON d.id = b.item
INNER JOIN c ON c.userID = b.userID
WHERE c.username = 'ABC'
AND a.float_val BETWEEN 0.0 AND 1.0
ORDER BY a.float_val LIMIT 100
This kind of query contains a deferred join. That dramatically reduces the number of rows of the full query that need to be subjected to ORDER BY ... LIMIT. Without the deferred join, your original query sorts an enormous mess of quite long rows just to discard all but the first hundred of them. That's why it takes so long.
This should help. The next optimization step is to look at the EXPLAIN output from this query and the exact definitions of your tables.
Pro tip: In queries of this complexity, always qualify column names with table names or aliases. That is, use a.float_val throughout, rather than just float_val. This is a kindness to the next person to look at the query.

How to speed up left join queries by indexing?

At the moment I am experiencing some slower MySQL queries in my application which I want to speed up. Unfortunately I’m not quite sure which is the correct way to do it.
I have the following (fictitious) tables: Book, Page and Word.
Word is child of Page by word_page_id
Page is child of Book by page_book id
I already have individual indexes on page_book_id, word_page_id, book_user_id and book_flag_delete.
SELECT `book`.*, COUNT(word_id) AS `word_amount` FROM `book`
LEFT JOIN `page` ON page_book_id = book_id
LEFT JOIN `word` ON word_page_id = paragraph_id
WHERE (book_user_id = 1) AND (book_flag_delete IS NULL)
GROUP BY `book_id`
ORDER BY `book_id` ASC LIMIT 100
SELECT COUNT(DISTINCT `book_id`) AS `book_row_count` FROM `book`
LEFT JOIN `page` ON page_book_id = book_id
LEFT JOIN `word` ON word_page_id = page_id
WHERE (book_user_id = 59) AND (book_flag_delete IS NULL)
Any ideas how to speed up such queries?
Is there extra indexing involved?
Set indexes on the fields you use for joining.
Further make sure that these have both the same datatype, encoding, and collation, else the index will also not be used.
mysql> EXPLAIN <query> will show you the actually used fields (key column in output) and the available indexes (possible_keys output field).
For this query:
SELECT b.*, COUNT(w.word_id) AS `word_amount`
FROM `book` b LEFT JOIN
`page` p
ON p.page_book_id = b.book_id LEFT JOIN
`word` w
ON w.word_page_id = p.paragraph_id
WHERE (b.book_user_id = 1) AND (b.book_flag_delete IS NULL)
GROUP BY b.`book_id`
ORDER BY b.`book_id` ASC
LIMIT 100;
The best indexes are: book(user_id, book_flag_delete, book_id), page(page_book_id, paragraph_id), and word(word_page_id, word_id).
However, the overall group by might be expensive. You might try writing the query as:
SELECT b.*,
(SELECT COUNT(w.word_id)
FROM `page` p JOIN
`word` w
ON w.word_page_id = p.paragraph_id
WHERE p.page_book_id = b.book_id
) AS `word_amount`
FROM `book` b LEFT JOIN
WHERE (b.book_user_id = 1) AND (b.book_flag_delete IS NULL)
ORDER BY b.`book_id` ASC
LIMIT 100;
The same indexes indexes work here. But, this query should avoid a group by on all the data at once (instead, it uses the indexes for the aggregation).
The optimal schema for a many-to-many mapping table is
CREATE TABLE XtoY (
# No surrogate id for this table
x_id MEDIUMINT UNSIGNED NOT NULL, -- For JOINing to one table
y_id MEDIUMINT UNSIGNED NOT NULL, -- For JOINing to the other table
# Include other fields specific to the 'relation'
PRIMARY KEY(x_id, y_id), -- When starting with X
INDEX (y_id, x_id) -- When starting with Y
) ENGINE=InnoDB;
The details on 'why' are in my index cookbook
In your select you're gonna want to refrain from using the wildcard "*" to grab columns. Plus utilize aliases ALWAYS!! This will keep your db from having to create a "virtual" alias.
select book1.column1, book1.column2, page1.column1
from book book1
left join page page1
on page1.page_book_id = book1.book_id
..... blah

mysql having... > avg() doesn't work as expected

I've created two views to help calculate user_diary_number and then select users of whom diary numbers > average of total user's user_diary_number.
two views are like below:
create view user_diary_number as
(
select user_id,count( distinct diary_id ) as diary_num
from user_diary
group by user_id
);
and second using having and avg:
create view hw_diary as
(
select u.user_id, u.realname, ud.diary_num, school.school_name
from (user as u cross join user_diary_number as ud on u.user_id = ud.user_id )cross join school on u.school_id = school.school_id
having diary_num > avg(diary_num)
);
What the problem is now, the second view only have 1 row of result. and absolutely, we have more than 1 users whose diary number > average diary_num. Indeed, I have 251 diaries in total and 103 users. Some of users have 9, 4, 5 diaries.
But the result only comes in only 1 user who have 3 diaries.
my relative tables are:
CREATE TABLE IF NOT EXISTS `school` (
`school_id` int(11) NOT NULL,
`school_name` varchar(45) NOT NULL,
`location` varchar(45) NOT NULL,
`master` varchar(45) NOT NULL,
`numbers_of_student` int(11) NOT NULL,
PRIMARY KEY (`school_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE IF NOT EXISTS `user_diary` (
`diary_id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) NOT NULL,
`title` varchar(45) NOT NULL,
`content` varchar(255) NOT NULL,
`addtime` DATETIME NOT NULL,
PRIMARY KEY (`diary_id`,`user_id`),
KEY `fk_diary_user_id_idx` (`user_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 ;
Is it any problems with cross join? or something else?
thanks a lot!
You can't use avg that way. In my personal movie database,
select * from movie having year > avg(year);
produces nothing, and
select * from movie having year > (select avg (year) from movie);
produces the expected result.
You must calculate the average in a separate subquery.
Something like:
select ...
from ...
group by ...
having diary_num > (
select avg(diary_num)
from ...)
You can fill in the blanks with what makes sense
Something like this should return the resultset you are looking for:
SELECT u.user_id
, u.realname
, c.diary_num
, s.school_name
-- , a.diary_avg
FROM ( SELECT d.user_id
, COUNT(DISTINCT d.diary_id) AS diary_num
FROM user_diary d
) c
JOIN user u
ON u.user_id = c.user_id
JOIN school s
ON s.school_id = u.school_id
JOIN ( SELECT AVG(v.diary_num) AS diary_avg
FROM ( SELECT t.user_id
, COUNT(DISTINCT t.diary_id) AS diary_num
FROM user_diary t
) v
) a
ON a.diary_avg < c.diary_num
ORDER BY 1
The inline view aliased as c gets us the diary_num (count) for each user.
The inline view aliased as a gets us the average of all the diary_num for all users. That is getting us an "average" of the counts, which is what it looks like your original query was intending to do.
As an alternative, we could get the "average" number of diaries per user as ... the total count of all diaries divided by the total count of all users. To do that, replace that inline view aliased as a with something like this:
( SELECT COUNT(DISTINCT t.diary_id)
/ NULLIF(COUNT(DISTINCT v.user_id),0) AS diary_avg
FROM user v
LEFT
JOIN user_diary t
ON t.user_id = v.user_id
) a
This yields slightly different results, as its a calculation on total counts, rather than an average of a calculation.
NOTE
The CROSS keyword has no influence on the MySQL optimizer.
We do typically include the CROSS keyword as documentation for future reviewers. It indicates that we have purposefully omitted the usual ON clause. (As a reviwer, when we see a JOIN without an ON clause, our minds race to a "possible unintended Cartesian product"... the author's inclusion of the CROSS keyword alerts us (the reviewer) that the omission of the ON clause was purposeful.
But the MySQL optimizer doesn't care one whit whether the CROSS keyword is included or omitted.
One more question: Does MySQL support for "View's SELECT contains a subquery in the FROM clause"?
A: Really old versions (3.x ?) of MySQL did not support subqueries. But certainly, MySQL 5.1 and later do support subqueries.
To answer your question, yes, a SELECT statement can be used as an inline view as a rowsource for another query, e.g.
SELECT v.*
FROM (
SELECT 1 AS foo
) v