I'm having problems with big table indexs.
I have the table (id,external_item_id,time_stamp,status_id);
What's the best index for this 3 queries:
SELECT *
FROM items pi
WHERE 1=1 AND external_item_id IN (1154,1155,1163,3660,6801,98)
ORDER BY pi.time_stamp DESC, pi.id DESC
LIMIT 12
SELECT *
FROM items pi
WHERE 1=1 AND external_item_id IN (1154,1155,1163,3660,6801,98) AND status_id < 20
ORDER BY pi.time_stamp DESC, pi.id DESC
LIMIT 12
SELECT *
FROM items pi
WHERE 1=1 AND external_item_id IN (1154,1155,1163,3660,6801,98) AND pi.time_stamp <= 13434534452 AND id < 1600
ORDER BY pi.time_stamp DESC, pi.id DESC
LIMIT 12
Because of the list of items for the in, it is hard to optimize this query.
There are basically two approaches the engine can take for these queries. Use the index for the where clause. Then either do the sort or use the index for the oder by. Because there are inequalities in the where (in is an "inequality"), the index cannot be directly used for the where.
The best indexes for the where are: items(external_item_id, status_id) and items(external_item_id, time_stamp).
An alternative execution plan is to use the index for the order by and then filter on the fly. This suggests trying: items(time_stamp, id, external_item_id, status_id). The last two columns are so the index can satisfy the where without going to the original data.
None of these are perfect solutions.
Related
I am developing a voting system.
I am facing issue in ordering.
Basically i want to get top ranking and maximum nominee first. I used "totalUserVoted" and "totalRating" in Descending order on both condition but my query ordering "totalUserVoted".
I am expecting the result in this order.
Here is my sql query.
SELECT
(SELECT (((SUM(`design`)*4)+(SUM(`usability`)*3)+(SUM(`creativity`)*2)+(SUM(`content`))*1))/ count(`nominee_id`) / 10
FROM `sk_award_nominee_rating`
WHERE `sk_award_nominee_rating`.`nominee_id`=`sk_award_nominee`.`nominee_id`) AS totalRating,
(SELECT count(`nominee_id`)
FROM `sk_award_nominee_rating`
WHERE `sk_award_nominee_rating`.`nominee_id`=`sk_award_nominee`.`nominee_id`) AS totalUserVoted,
`sk_award_nominee`.*,
`sk_user`.`username`,
`sk_user`.`email`,
`sk_user_profile`.`f_name`,
`sk_user_profile`.`m_name`,
`sk_user_profile`.`l_name`,
`sk_user_profile`.`address`
FROM `sk_award_nominee`
LEFT JOIN `sk_user` ON `sk_user`.`user_id`=`sk_award_nominee`.`user_id`
LEFT JOIN `sk_user_profile` ON `sk_award_nominee`.`user_id`=`sk_user_profile`.`user_id`
WHERE `sk_award_nominee`.`status` = 1
AND DATE(approval_date) = '2016-02-22'
ORDER BY `totalUserVoted` DESC,
`totalRating` DESC
Something like this ?
ORDER BY totalUserVoted DESC, totalRating DESC
PHP MySQL Order by Two Columns
I have two tables :
video (ID, TITLE, ..., UPLOADED_DATE)
join_video_category (ID (not used), ID_VIDEO_ ID_CATEGORY)
rows in video : 4 500 000 |
rows in join_video_category : 5 800 000
1 video can have many category.
I have a query works perfectly, 20 ms max to get result :
SELECT * FROM video WHERE ID IN
(SELECT ID_VIDEO FROM join_video_category WHERE ID_CATEGORY=11)
LIMIT 1000;
This query take 1000 video, the order is not important.
BUT, when i would like to get 10 latest video from a category, my query take arround 30-40 seconds :
SELECT * FROM video WHERE ID IN
(SELECT ID_VIDEO FROM join_video_category WHERE ID_CATEGORY=11)
ORDER BY UPLOADED_DATE DESC LIMIT 10;
I have index on ID_CATEGORY, ID_VIDEO, UPLOADED_DATE, PRIMARY ON ID video and join_video_category.
I have tested it with JOIN on my query, it's the same result.
First, the comparisons are to two very different queries. The first returns a bunch of videos whenever it encounters them. The second has to read all the videos and then sort them.
Try rewriting this as a JOIN:
SELECT v.*
FROM video v JOIN
join_video_category vc
ON v.id = bc.id_video
WHERE vc.ID_CATEGORY = 11
ORDER BY v.UPLOADED_DATE DESC
LIMIT 10;
That may or may not help. You have a lot of data and so you might have a lot of videos for a given category. If so, a where clause that gets more recent data might really help:
SELECT v.*
FROM video v JOIN
join_video_category vc
ON v.id = bc.id_video
WHERE vc.ID_CATEGORY = 11 AND v.UPLOADED_DATE >= '2015-01-01'
ORDER BY v.UPLOADED_DATE DESC
LIMIT 10;
Finally, if that doesn't work, consider adding something like UPLOADED_DATE into join_video_category. Then, this query should blaze:
select vc.video_id
from join_vdeo_category vc
where vc.ID_CATEGORY = 11
order by vc.UPLOADED_DATE desc
limit 10;
with an index on join_video_category(id_category, uploaded_date, video_id).
solution #1:
replacing "in" with "exists" would improve the performance, please try the below query.
SELECT * FROM video WHERE exists
(SELECT * FROM join_video_category WHERE ID_CATEGORY=11 AND join_video_category.ID_VIDEO = video.ID)
ORDER BY UPLOADED_DATE DESC LIMIT 10;
solution #2:
1) create tem_table
CREATE TABLE TEMP_TABLE AS SELECT * FROM join_video_category WHERE ID_CATEGORY=11;
2) use the temp table in solution #1
SELECT * FROM video WHERE exists
(SELECT * FROM temp_table WHERE temp_table.ID_VIDEO = video.ID)
ORDER BY UPLOADED_DATE DESC LIMIT 10;
Good Luck!!
If it is 1:Many, don't use an extra table between Video and Category. However, your row counts imply that it is Many:Many.
If it is 1:Many, simply have the category_id in the Video table, then simplify all the queries.
If it is Many:Many, then be sure to use this pattern for the junction table:
CREATE TABLE map_video_category (
video_id ...,
category_id ...,
PRIMARY KEY(video_id, category_id), -- both ids, one direction
INDEX (category_id, video_id) -- both ids, the other direction
) ENGINE=InnoDB; -- significantly better than MyISAM on INDEX handling here
The ID that you mentioned is a waste. The composite keys are optimal for all situations, and will improve performance in most situations.
Do not use IN ( SELECT ... ); the optimizer does a poor job of optimizing it. Change to a JOIN, LEFT JOIN, EXISTS, or some other construct.
I have the following table
As you can see It has 1868155 rows. I am attempting to make a realtime graph, but It is impossible since almost any query lasts 1 or 2 seconds.
For example, this query
SELECT sensor.nombre, temperatura.temperatura
FROM sensor, temperatura
WHERE sensor.id = temperatura.idsensor
ORDER BY temperatura.fecha DESC, idsensor ASC
LIMIT 4
Is supposed to show this
Ive tried everything, using indexes(perhaps not correctly), using only the fields i need instead of *, etc. but the results are the same!
These are the indexes of the table.
Explain of the query
EDITED
This is the explain of the query after implementing
ALTER TABLE temperatura
ADD INDEX `sensor_temp` (`idsensor`,`fecha`,`temperatura`)
And using inner join syntax for the query
SELECT s.nombre, t.temperatura
FROM sensor s
INNER JOIN temperatura t
ON s.id = t.idsensor
ORDER BY t.fecha DESC, t.idsensor ASC
LIMIT 4
This is my whole sensor table
Try the following:
ALTER TABLE temperatura
ADD INDEX `sensor_temp` (`idsensor`,`fecha`,`temperatura`)
I also recommend using modern join syntax:
SELECT s.nombre, t.temperatura
FROM sensor s
INNER JOIN temperatura t
ON s.id = t.idsensor
ORDER BY t.fecha DESC, t.idsensor ASC
LIMIT 4
Report the EXPLAIN again after making the above changes, if performance is still not good enough.
Attempt #2
After looking closely at what it appears you are trying to do, I believe this next query may be more effective:
SELECT
s.nombre, t.temperatura
FROM temperatura t
LEFT OUTER JOIN temperatura later_t
ON later_t.idsensor = t.idsensor
AND later_t.fecha > t.fecha
INNER JOIN sensor s
ON s.id = t.idsensor
WHERE later_t.idsensor IS NULL
ORDER BY t.idsensor ASC
You can also try:
SELECT
s.nombre, t.temperatura
FROM temperatura t
INNER JOIN (
SELECT
t.idsensor,
MAX(t.fecha) AS fecha
FROM temperatura t
GROUP BY t.idsensor
) max_fecha
ON max_fecha.idsensor = t.idsensor
AND max_fecha.fecha > t.fecha
INNER JOIN sensor s
ON s.id = t.idsensor
ORDER BY t.idsensor ASC
In my experience, if you are trying to find the most recent record, one of the two queries above will work. Which works best depends on various factors, so try them both.
Let me know how those perform, and if they still get you the data you want. Also, any query you run, run at least 3 times, and report all 3 times. That will help get an accurate measure of how fast a given query is, since various external factors can affect the speed of a query.
It is not possible to optimize a mixture of ASC and DESC, as in
ORDER BY t.fecha DESC, t.idsensor ASC
You tried a covering index:
INDEX `sensor_temp` (`idsensor`,`fecha`,`temperatura`)
However, this covering index may be better:
INDEX `sensor_temp` (`fecha`,`idsensor`,`temperatura`)
Then, if you are willing to get the sensors in a different order, use
ORDER BY t.fecha DESC, t.idsensor DESC
This will give you up to 4 sensors for the last fecha:
sensor: PRIMARY KEY(id)
tempuratura: INDEX(fecha, idsensor, tempuratura)
SELECT
( SELECT nombre FROM sensor WHERE id = t.idsensor ) AS nombre,
t.temperatura
FROM
( SELECT MAX(fecha) AS max_fecha FROM tempuratura ) AS z
JOIN temperatura AS t ON t.fecha = z.max_fecha
ORDER BY t.idsensor ASC
LIMIT 4;
I'm looking to speed this query up. I currently have an index on
users_score.appID
app_names.name
SELECT users_scores.username, users_scores.avatar, users_scores.score
FROM users_scores
RIGHT JOIN app_names ON app_names.id = users_scores.appID
WHERE app_names.name = "testapp1"
ORDER BY users_scores.score DESC
LIMIT 0 , 30
Do you have an index on your primary key? (users_score.id, or whatever you've named it). If not, keys should always be indexed... in fact, they ARE the index. app_names.id should also be primary key/index.
appID is a good index, however I see you searching for apps via name. It's faster if MySQL doesn't have to perform string comparisons on WHERE clauses. It would be much more efficient to search for an AppID. Given the app name is known ('testapp1'), you could do an inner query to determine the ID before searching, like this.
WHERE app_names.id = (SELECT id FROM app_names WHERE app_names.name = "testapp1")
You should never use RIGHT JOIN. Any RIGHT JOIN can and should be written as a LEFT JOIN.
Regardless, your WHERE clause automatically turns the query into an INNER JOIN:
SELECT users_scores.username, users_scores.avatar, users_scores.score
FROM app_names
INNER JOIN users_scores
ON users_scores.appID = app_names.id
WHERE app_names.name = 'testapp1'
ORDER BY users_scores.score DESC
LIMIT 30
Since you're not returning any data from app_names, you can get rid of the JOIN entirely by using a subquery:
SELECT username, avatar, score
FROM users_scores
WHERE appID = (SELECT id FROM app_names WHERE name = 'testapp1' LIMIT 1)
ORDER BY score DESC
LIMIT 30
MySQL executes non-correlated subqueries first, so MySQL can use an index on the app_names table for the search, then is able to utilize an index on users_scores for the search and sort.
For optimum read performance, add a multi-column index on user_scores(appID, score) to satisfy the search and the sort, or with an even larger covering index: user_scores(appID, score, username, avatar).
SELECT users_scores.username, users_scores.avatar, users_scores.score
FROM users_scores
RIGHT JOIN app_names -- Probably forces app_names to be first
ON app_names.id
= users_scores.appID -- users_scores second; Step 1
WHERE app_names.name = "testapp1" -- app_names first; Step 1
ORDER BY users_scores.score DESC
LIMIT 0 , 30
app_names needs INDEX(name)
users_scores needs INDEX(appID)
Even if you remove RIGHT (which might be noise), the Optimizer will pick app_names first because of the WHERE clause mentions only app_names.
All of this, plus more, is found in my blog on Creating an Index from a SELECT.
I was thinking a way to using one query with a subquery instead of using two seperate queries.
But turns out using a subquery is causing multiple requests for each row in result set. Is there a way to limit that count subquery result only one with in a combined query ?
SELECT `ad_general`.`id`,
( SELECT count(`ad_general`.`id`) AS count
FROM (`ad_general`)
WHERE `city` = 708 ) AS count,
FROM (`ad_general`)
WHERE `ad_general`.`city` = '708'
ORDER BY `ad_general`.`id` DESC
LIMIT 15
May be using a join can solve the problem but dunno how ?
SELECT ad_general.id, stats.cnt
FROM ad_general
JOIN (
SELECT count(*) as cnt
FROM ad_general
WHERE city = 708
) AS stats
WHERE ad_general.city = 708
ORDER BY ad_general.id DESC
LIMIT 15;
The explicit table names aren't required, but are used both for clarity and maintainability (the explicit table names will prevent any imbiguities should the schema for ad_general or the generated table ever change).
You can self-join (join the table to itself table) and apply aggregate function to the second.
SELECT `adgen`.`id`, COUNT(`adgen_count`.`id`) AS `count`
FROM `ad_general` AS `adgen`
JOIN `ad_general` AS `adgen_count` ON `adgen_count`.city = 708
WHERE `adgen`.`city` = 708
GROUP BY `adgen`.`id`
ORDER BY `adgen`.`id` DESC
LIMIT 15
However, it's impossible to say what the appropriate grouping is without knowing the structure of the table.