This query works and provides me with the information I need, but it is very slow: it takes 18 seconds to agregate a database of only 4,000 records.
I'm bringing it here to see if anyone has any advice on how to improve it.
SELECT COUNT( status ) AS quantity, status
FROM log_table
WHERE time_stamp
IN (SELECT MAX( time_stamp ) FROM log_table GROUP BY userid )
GROUP BY status
Here's what it does/what it needs to do in plain text:
I have a table full of logs, each log contains a "userid", "status" (integer between 1-12) and "time_stamp" (a time stamp of when the log was created). There may be many entries for a particular userid, but with a different time stamp and status. I'm trying to get the most recent status (based on time_stamp) for each userid, then count the occurrences of each most-recent status among all the users.
My initial idea was to use a sub query with GROUP BY userid, that worked fast - but that always returned the first entry for each userid, not the most recent. If I could do GROUP BY userid using time_stamp DESC to Identify which row should be the representative for the group, that would be great. But of course ORDER BY inside of group does not work.
Any suggestions?
The first thing to try is to make this an explicit join:
SELECT COUNT(status) AS quantity, status
FROM log_table join
(select lg.userid, MAX( time_stamp ) as maxts
from log_table lg
GROUP BY userid
) lgu
on lgu.userid = lg.userid and lgu.maxts = lg.time_stamp
GROUP BY status;
Another approach is to use a different where clause. This will work best if you have an index on log_table(userid, time_stamp). This approach is doing the filtering by saying "there is no timestamp bigger than this one for a given user":
SELECT COUNT(status) AS quantity, status
FROM log_table
WHERE not exists (select 1
from log_table lg2
where lgu.userid = lg.userid and lg2.time_stamp > lg.time_stamp
)
GROUP BY status;
Related
I have a subquery that aggregates some UNION ALL selects. Over that I prepare the SELECT to create cross-tab and limit it to let's say 20. I would like to be able to retrieve the total COUNT of sub query results before I am limiting them in main query. This is for the purpose of trying to build a pagination that receives the total number of records and then the specific page record grid.
Sample query:
SELECT
name,
sumIf(metric_value, metric_name = 'data') AS data,
sumif(....
FROM
(SELECT
name, metric_name, SUM(metric_value) as metric_value
FROM
(SELECT
name, 'data' AS metric_name, SUM(data) AS metric_value
FROM
table
WHERE
date > '2017-01-01 00:00:00'
GROUP BY
name
UNION ALL
SELECT
name, 'data' AS metric_name, SUM(data) AS metric_value
FROM
table2
WHERE
date > '2017-01-01 00:00:00'
GROUP BY
name
UNION ALL
SELECT
name, 'data' AS metric_name, SUM(data) AS metric_value
FROM
table3
WHERE
date > '2017-01-01 00:00:00'
GROUP BY
name
UNION ALL
.
.
.)
GROUP BY
name, metric_name)
GROUP BY
name
ORDER BY
name ASC
LIMIT 0,20;
The first subselect returns tons of data, so I thought I can count it and return as one column value, or row and it would propagate to main select that limits 20 results. Because I need to know the entire set of results but don;t want to call the same query twice without limit and with limit just to get COUNT. There are at least 12 UNION ALL third level sub selects, so why waste resources. I am looking to try generic SQL solutions not necessarily related to ClickHouse
I was thinking of using count(*) OVER (), however that is not supported, so if thats only option I know I need to run query twice.
The first thing that one should mention is that nobody is usually interested in the exact number of pages on a query. It can be easily estimated and almost no one will care how exact is the estimation. However, if you have a link to the last page in your GUI, people will often click to link just to see whether it works.
Nevertheless, there are cases when an analyst should visit all the pages, and then the GUI should display the exact amount of work. A good news is that in that latter case, a better strategy is to cache a snapshot of the whole results table and counting the rows in the table becomes not a problem anymore.
I mean, it makes sense to discuss with the customers whether they really need it, because unneeded full scans many times per day may have effect on the database load and billing sums.
Anyway, if you still need to estimate the number of rows, you can simplify the query just to count the number of rows. As I understand this is something like:
SELECT SUM(cnt) as row_count
FROM (
SELECT COUNT(DISTINCT name) as cnt FROM table1 WHERE date > ...
UNION ALL
SELECT COUNT(DISTINCT name) as cnt FROM table2 WHERE date > ...
...
) as counts;
or if data is a constant metric name
SELECT COUNT(DISTINCT name) as row_count
FROM (
SELECT DISTINCT name FROM table1 WHERE date > ...
UNION ALL
SELECT DISTINCT name FROM table2 WHERE date > ...
...
) as names;
Table temporary_search_table
post_id,property_status, property_address,....more 30 field
Table search_meta
meta_id,search_id,status,created_date
Ok I need Total data which created_date is yesterday. For each temporary_search_table data there may multiple entry within search_meta. So we need to pick last one field from search_meta and check created date is yesterday and property_status is pending. if yes then we can count the number. If there is no data available in search_meta for entry in temporary_search_table then we dont need to count that row within our results.
Here i am attaching my sql data. its work but for 30000 row it take lots of time.
SELECT COUNT(id) FROM temporary_search_table
WHERE property_status = 'pending' AND (1 = (SELECT DATEDIFF(NOW(), created_date)
FROM search_meta WHERE post_id = search_id ORDER BY created_date DESC LIMIT 0,1 ))
Thanks in advance.
Apart from checking the indexes on your table, it would probably be better to not use a correlated sub query and use a straight join instead.
SELECT COUNT(id)
FROM temporary_search_table
INNER JOIN search_meta ON post_id = search_id
WHERE property_status = 'pending' AND DATEDIFF(NOW(), created_date) = 1
ORDER BY created_date DESC
LIMIT 1
I have the following query:
SELECT t.ID, t.caseID, time
FROM tbl_test t
INNER JOIN (
SELECT ID, MAX( TIME )
FROM tbl_test
WHERE TIME <=1353143351
GROUP BY caseID
ORDER BY caseID DESC -- ERROR HERE!
) s
USING (ID)
It seems that I only get the correct result if I use the ORDER BY in the inner join. Why is that? I am using the ID for the join, so the order should take no effekt.
If I remove the order by, I get too old entries from the database.
ID is the primary key, the caseID is a kind of object with multiple entries with different timestamps.
This query is ambiguous:
SELECT ID, MAX( TIME )
FROM tbl_test
WHERE TIME <=1353143351
GROUP BY caseID
It's ambiguous because it does not guarantee that it returns the ID of the row where the MAX(TIME) occurs. It returns the MAX(TIME) for each distinct value of caseID, but the value of other columns (like ID) is chosen arbitrarily from members of the group.
In practice, MySQL chooses the row that it finds first in the group as it scans rows in storage order.
Example:
caseID ID time
1 10 15:00
1 12 18:00
1 14 13:00
The max time is 18:00, which is the row with ID 12. But the query will return ID 10, simply because it's the first one in the group. If you were to reverse the order with ORDER BY, it would return ID 14. Still not the row where the max time is found, but it's from the other end of the group of rows.
Your query works with ORDER BY caseID DESC because, by coincidence, your Time values increase with the increasing ID.
This sort of query is actually an error in standard SQL and most other brands of SQL database. MySQL permits it, trusting that you know how to form an unambiguous query.
The fix is to use columns in the select-list only if they are unambiguous, that is, if they are in the GROUP BY clause, then each group is guaranteed to have only one distinct value:
SELECT caseID, MAX( TIME )
FROM tbl_test
WHERE TIME <=1353143351
GROUP BY caseID
SELECT t.ID, t.caseID, time
FROM tbl_test t
INNER JOIN (
SELECT caseID, MAX( TIME ) maxtime
FROM tbl_test
WHERE TIME <=1353143351
GROUP BY caseID
) s
ON t.caseID = s.caseID and t.time = s.maxtime
You are seeing that issue because you are getting the MAX(TIME) per caseID, but since you are grouping by caseID and NOT ID, you are getting an arbitrary ID. That happens because when you use an aggregate function, like MAX, you must, for every non-grouped field in the select specify how you want to aggregate it. That means, if it's in the SELECT and NOT in the GROUP BY, you have to tell MySQL how to aggregate. If you don't then you get a RANDOM row (well, not random per se, but it's not going to be in an order that you necessarily expect).
The reason ORDER BY is working for you, is that it kind of tricks the query optimizer into sorting the results before grouping, which just so happens to produce the result you want, but be warned, that will not always be the case.
What you want is the ID that has the MAX(TIME) given a caseID. Which means your INNER join needs to connect by caseID (not ID) and time (which will give you 1 row per each 1 row in the outer table).
Barmar beat me to the actual query, but that's the way you want to go.
I have a social network I am coding but it's a bit different than others, in this case there is only ever one status show per user on a feed section.
So I need to sort the status by date with the latest ones on top but also group by the userID
unfortunately I can't seem to get this working....
This is my current query:
SELECT *
FROM status
WHERE userID != "83" #This is just so my statuses don't show
GROUP BY userID
ORDER BY addedDate DESC
LIMIT 10
I expect to see the latest status results and only one per user instead I see the first statuses so the group by is working but not the order by.
Thanks in advance.
As mentioned in the comments to Robin's answer, that approach is unreliable because MySQL does not guarantee that it will always return the most recent status from each group. You must instead join your table with a subquery that selects the most recent status (based on addedDate).
SELECT *
FROM status
NATURAL JOIN (
SELECT userID, MAX(addedDate) as addedDate
FROM status
GROUP BY userID
) AS mostRecent
ORDER BY addedDate DESC
LIMIT 10
Note that if a user has multiple status updates with the same addedDate, the server will return all of them (whereas Robin's query would return an indeterminate one); if you need control over such a situation, you will need to define how one determines which such status update should be selected.
SELECT userID, max(addedDate)
FROM status
WHERE userID != "83" #This is just so my statuses don't show
GROUP BY userID
SELECT *
FROM ( SELECT *
FROM status
WHERE userID != "83"
ORDER BY addedDate DESC) AS h
GROUP BY userID
ORDER BY addedDate DESC
LIMIT 10
You must ORDER BY before GROUP BY'ing.
Example 1
Example 2
I have the database table logs as the following:
alt text http://www.freeimagehosting.net/uploads/16e974703a.jpg
I would like to extract the last entry of device, pollDate, status. For eg.
deviceId, pollDate, status
1, 2010-95-06 10:53:28, 1
3, 2010-95-06 10:26:28, 1
I tried to run the following query but the distinct only selects the first records, not the latest
SELECT DISTINCT deviceId, pollDate, status
FROM logs
GROUP By deviceId
ORDER BY pollDate DESC
alt text http://www.freeimagehosting.net/uploads/5d181103f8.jpg
So, could you please help me to extract the latest entries from the table? Thanks.
If (deviceID, poll_date) is unique, you can do the following:
SELECT *
FROM (
SELECT deviceid, MAX(poll_date) AS md
FROM logs
GROUP BY
deviceid
) q
JOIN logs l
ON l.deviceid = q.deviceid
AND l.poll_date = q.md