Given is mySQL table named "user_posts" with the following relevant fields:
user_id
user_status
influencer_status
indexed in all three fields
My running slow query is here and also i have created an dbFiddle . Output of Explain is in the dbfiddle:
SELECT
P.user_post_id,
P.user_id_fk,P.post_type,
P.who_can_see_post,
P.post_image_id,P.post_video_id,
U.user_name, U.user_fullname,U.influencer_status
FROM user_posts P FORCE INDEX (ix_user_posts_post_id_post_type)
INNER JOIN users U FORCE INDEX (ix_status_istatus)
ON P.user_id_fk = U.user_id
WHERE
U.user_status='1' AND
U.influencer_status = '1' AND
(P.who_can_see_post IN('everyone','influencer','friends')) AND
(P.post_type IN('image','video'))
AND p.user_post_id > 30
ORDER BY
P.user_post_id
DESC LIMIT 30
The query takes extremely long, around 6-15 seconds. The database is not very busy otherwise and performs well on other queries.
I am obviously wondering why the query is so slow.
Is there a way to tell exactly what is taking mySQL so long? Or is there any change I need to make to make the query run faster?
The definition of your ix_status_istatus key is preventing it being used to optimise the WHERE clause, as it includes user_id which is not used in the WHERE clause. Redefining the index as
ALTER TABLE `users`
ADD PRIMARY KEY (`user_id`),
ADD KEY ix_status_istatus (user_status, influencer_status);
allows it to be used and should speed up your query, changing the search on users to use index instead of temporary and filesort.
Demo on dbfiddle
Update
Further analysis on dbfiddle suggests that it is also better to remove the FORCE INDEX from the P table as it is not necessary (only the PRIMARY key is required) and changing the JOIN to a STRAIGHT_JOIN i.e. write the JOIN as:
FROM user_posts P
STRAIGHT_JOIN users U FORCE INDEX (ix_status_istatus)
ON P.user_id_fk = U.user_id
I think You should limit result set of joining part using conditions inside ON statement.
It’s like doing filtering during joining instead of joining then filtering.
I’ve checked query plan which shows me full utilization of indexes.
SELECT
P.user_post_id,
P.user_id_fk,P.post_type,
P.who_can_see_post,
P.post_image_id,
P.post_video_id,
U.user_name,
U.user_fullname,
U.influencer_status
FROM user_posts P
INNER JOIN
users U
FORCE INDEX (
users_user_status_index,
users_influencer_status_index
)
ON
U.user_id = P.user_id_fk AND
U.user_status='1' AND
U.influencer_status='1'
WHERE
P.who_can_see_post IN('everyone','influencer','friends') AND
P.post_type IN('image','video') AND
P.user_post_id > 30
ORDER BY
P.user_post_id DESC
LIMIT 30
Indexes that I created:
ALTER TABLE `users`
ADD PRIMARY KEY (`user_id`),
ADD KEY users_user_status_index(user_status),
ADD KEY users_influencer_status_index(influencer_status);
dbfiddle
Related
I'm a beginner in php and I want to ask you if the query and table schema I have set up is the right way for performance. Note: If you want me to follow a different way, please provide sample for me, thanks
$digerilanlar = DB::get('
SELECT Count(siparisler.hid) AS siparissayisi,
siparisler.hid, ilanlar.id, ilanlar.seflink, ilanlar.kategori, ilanlar.baslik,
ilanlar.yayin, ilanlar.tutar, ilanlar.sure, ilanlar.onecikan, ilanlar.guncellemetarihi,
uyeler.nick, uyeler.foto, uyeler.online, uyeler.ban FROM ilanlar
inner join uyeler ON uyeler.id=ilanlar.ilansahibi
LEFT JOIN siparisler ON ilanlar.id = siparisler.hid
WHERE ilanlar.kategori= '.$kat->id.' and ilanlar.yayin=1 and uyeler.ban=0
GROUP BY ilanlar.id
ORDER BY guncellemetarihi DESC
LIMIT 0,12');
DATABASE DESİGN
Table engine MyISAM MYSQL versiyon 5.7.14
TABLE:İLANLAR
ilansahibi (int)= index
kategori (int)= index
yayin (int)= index
TABLE:UYELER
ban (int)= index
TABLE:SİPARİSLER
hid (int)= index
This will probably require two temp tables and two sorts:
GROUP BY ilanlar.id
ORDER BY guncellemetarihi DESC
Assuming that guncellemetarihi is update_date, this is not identical, but probably gives you what you want, but with only one temp table and sort:
GROUP BY guncellemetarihi, id
ORDER BY guncellemetarihi DESC, id DESC
COUNT(x) checks x for being NOT NULL. If that is not necessary, simply do COUNT(*).
SELECT COUNT(hid), hid
does not make sense. The COUNT implies that there may be multiple "hids", but hid implies that there is only one. (Since I don't understand to objective, I cannot advise which direction to change things.)
This composite INDEX may help:
ilanlar: INDEX(kategori, yayin, ilansahibi, id)
You should switch from ENGINE=MyISAM to ENGINE=InnoDB.
More on making indexes: Index Cookbook
To discuss further, please provide SHOW CREATE TABLE and EXPLAIN SELECT ...
I have following tables on Production with the respective counts of records,
people count= '565367'
donors count= '556325'
telerec_recipients count= '115147'
person_addresses count= '563183'
person_emails count= '106958'
person_phones count= '676474'
person_sms count= '22275'
On the UI end I want to display some data by applying some joins or left joins, and at the end by grouping, ordering and paginating I'm generating a json response.
So for achieving this I tried 2 methods, one by creating the view file of the join queries and apply where, group by and order by clauses inside controller, and second by directly firing the laravel syntax of joins in my controller.
Because I have a large data in all the tables, The join query works good but when it comes to group by statement it takes much time to execute.
Help me out in order to optimize or faster this process.
My Example Query is:
SELECT people.id as person_id,donors.donor_id,telerec_recipients.id as telerec_recipient_id,people.first_name,people.last_name,people.birth_date,
person_addresses.address_id,person_emails.email_id,person_phones.phone_id,person_sms.sms_id
FROM `people`
inner join `donors`
on `people`.`id` = `donors`.`person_id`
left join `telerec_recipients`
on `people`.`id` = `telerec_recipients`.`person_id`
left join `person_addresses`
on `people`.`id` = `person_addresses`.`person_id`
left join `person_emails`
on `people`.`id` = `person_emails`.`person_id`
left join `person_phones`
on `people`.`id` = `person_phones`.`person_id`
left join `person_sms`
on `people`.`id` = `person_sms`.`person_id`
GROUP BY `people`.`id`, `telerec_recipients`.`id`
ORDER BY `last_name` ASC LIMIT 25 offset 0;
Result of the explain:
Donors Table Indexes
The problem is that MySQL uses the wrong index on the donors table, you need to instruct MySQL using force index index hint to use the primary key instead.
Explanation
From the output of the explain it is clear that sg really goes wrong with donors table, since you can see using temporary, using filesort in the extra column.
The key column tells you that MySQL uses a unique index on the donors.donor_id field. However, donor_id field is not used anywhere in the query for joining / grouping / filtering purposes. donors.person_id field is used in the join. Since the primary key of donors table is on the person_id field, myql should use that for the join. force index index hint tells MySQL that you firmly believe that a certain index should be used.
Note 1:
You have several duplicate indexes. Remove all duplicates, they are only slowing your system own.
Note 2:
I think you query is against the sql standards because you have fields in the select list that are neither in the group by list, nor are subject of an aggregate function such as sum(), nor are functionally dependent on the grouping fields. In MySQL under certain sql mode settings such queries are allowed to run, but they may not produce entirely the same output as you would like.
tag_relation table has tag_id and comment_id fields only and both of them are indexed. (there is no primary) it has InnoDB type.
Following query takes long time to execute. How can I make it faster?
All comment_id, tag_id, status, datetime fields are indexed. I really have no idea how to optimize it further.
SELECT
text
FROM comment
INNER JOIN tag_relation
ON tag_relation.comment_id=comment.comment_id
WHERE tag_id='1022278'
AND status=1
ORDER BY comment.datetime DESC LIMIT 0,20
Main cause of slowness is tag_relation table which has 1.5 million records. When it has less records execution time was faster.
Query plan:
This is your query:
SELECT c.text
FROM comment c INNER JOIN
tag_relation tr
ON tr.comment_id = c.comment_id
WHERE t.tag_id = 1022278 AND c.status = 1
ORDER BY c.datetime DESC
LIMIT 0, 20;
First, notice that I removed the single quotes from the value 1022278. If this is really a number, the single quotes can sometimes confuse SQL optimizers. There are two ways to go about optimizing this query, depending on the selectivity of the various conditions. The first is to have the indexes:
tag_relation(tag_id, comment_id)
comment(comment_id, status, datetime, text)
The second is a covering index for comments, and the most important part is the comment_id column.
The second is:
comment(status, comment_id, datetime)
tag_relation(comment_id, tag_id)
The basic issue is which table is scanned first for the join. Using this index, the query would be processed as:
SELECT c.text
FROM comment c INNER JOIN
tag_relation
WHERE c.status = 1 AND
EXISTS (SELECT 1
FROM tag_relation tr
WHERE tr.comment_id = c.comment_id AND tr.tag_id = 1022278
)
ORDER BY c.datetime DESC
LIMIT 0, 20;
I'm not 100% sure that this avoids the file sort on the result set, but it might work.
If I get it right you have one index for tag_id and another index for comment_id. Try creating an composite index like:
create index ... on tag_relation(tag_id, comment_id)
This will make the index with tag_id redundant so it can be dropped.
AFAIK MySQL cannot do index anding, but even if it could a composite index would be more efficient.
I think the problem is in the "status" field. Although it is indexed, the index is not being used. It says "using where" for that table. You can force the use of the index for status but I'm not sure it will be useful, depending on selectivity, i.e., how many different values can "status" take. Alternatively, the documentation says that if "status" allows for NULL then you'll see the "using where". Does it allow for NULLs? If so, consider restricting it.
I just noticed that I overlooked the "ORDER BY", comment.datetime will need an index.
If you already have an index, then try a subquery:
SELECT text
FROM tag_relation
INNER JOIN (SELECT c.comment_id, c.text, c.datetime
FROM comment c
WHERE c.status = 1) comment
ON tag_relation.comment_id = comment.comment_id
WHERE tag_id='1022278'
ORDER BY comment.datetime DESC LIMIT 0,20
I have this MySQL query which seems to be very very slow. It takes 3 seconds to run. This is trying to get all the posts from either who they're following or any interests they have. Also its trying to make sure it doesn'tshow any duplicate shares that match any post_id. What do you guys think I should do?
SELECT p.*,
IFNULL(post_data_share, UUID()) AS unq_share,
UNIX_TIMESTAMP(p.post_time) AS a
FROM posts p
LEFT JOIN users_interests i ON (i.user_id=1
AND p.post_interest = i.interest)
LEFT JOIN following f ON (f.user_id=1
AND p.post_user_id = f.follower_id)
WHERE (post_user_id=1
OR f.follower_id IS NOT NULL
OR i.interest IS NOT NULL)
AND (POST_DATA_SHARE NOT IN
(SELECT POST_ID
FROM posts p
LEFT JOIN following f ON f.user_id=1
AND p.post_user_id = f.follower_id
LEFT JOIN users_interests i ON (i.user_id=1
AND p.post_interest = i.interest)
WHERE (post_user_id=1
OR f.follower_id IS NOT NULL
OR i.interest IS NOT NULL))
OR POST_DATA_SHARE IS NULL)
GROUP BY unq_share
ORDER BY `post_id` DESC LIMIT 10;
Below are the Performance tips will definitely make difference.
Try Explain Statement.
Alter the Table by Normalize your tables by adding Primary Key and Foreign Key
Add Index for Repeated Values.
Avoid select * from table. Mention the specify column name.
Convert IS NULL to (='')
Convert IS NOT NULL to (!='')
Avoid More OR Condition.
MySQL Configurations to explore
key_buffer_size
innodb_buffer_pool_size
query_cache_size
thread_cache
Much more refer this SO Answer Best my.cnf configuration for a 8GB MySQL server with MyISAM use only
I would start by looking at the execution plan for the query. Here is a link to MySQL documentation on the EXPLAIN keyword to show you how the optimizer is structuring your query: http://dev.mysql.com/doc/refman/5.5/en/using-explain.html
If CPU usage is low, likely the bottleneck is disk access for large table scans.
The way the query is executed is often different from how it was written. Once you see how the execution plan is structured, you are probably going to create indexes on the largest joins. Every table should have one clustered index (often it is created by default), but other fields can often benefit from unclustered indexes.
If the problem is extremely bad and this is vital to your application, you may want to consider reorganizing the database.
I have two inner joins in my SQL query:
SELECT `M`.`msg_id`,
`U`.`username`,
`U`.`seo_username`
FROM `newdb2`.`users` AS `U`
INNER JOIN (SELECT subscriber_to_id
FROM subscriptions
WHERE subscriber_id = 434) AS subscriber
ON id = subscriber_to_id
INNER JOIN `newdb2`.`messages` AS `M`
ON (`M`.`uid_fk` = `U`.`id`)
ORDER BY id DESC LIMIT 10
When I execute this query I see that is really slow.
How can I modify thiş query to make it faster?
Quick fixes for things like this are adding indexes which allows your database server to quickly look up columns you are searching on. For more info on how to add indexes to columns, see the manual.
In this query, those columns are:
subscriptions.subscriber_id
subscriptions.subscriber_to_id
users.id
messages.uid_fk
The ORDER BY id should be OK as I assume your id column has a primary key index on it already, but ordering queries will slow it down too.
Subselect queries will also slow the query down. In this particular query, I can't see the alias subscriber (containing the results of your subquery, which is inner joined on) used anywhere, so remove that join completely.