I want to take the maximum value from a series of returned values but I can't figure out a simple way to do it. My query returns all rows so 1/2 way there. I can filter it down with PHP but I'd like to do it all in SQL. I tried with a max subquery but that returned all results still.
DDL:
create table matrix(
count int(4),
date date,
product int(4)
);
create table products(
id int(4),
section int(4)
);
DML:
select max(magic_count), section, id
from (
select sum(count) as magic_count, p.section, p.id
from matrix as m
join products as p on m.product = p.id
group by m.product
) as faketable
group by id, section
Demo with my current try.
Only ids 1 and 3 should be returned from the sample data because they have the highest cumulative count for each of the sections.
Here's a second SQL fiddle that demonstrates the same issue.
Here you go:
select a.id,
a.section,
a.magic_count
from (
select p.id,
p.section,
magic_count
from (
select m.product, sum(count) as magic_count
from matrix m
group by m.product
) sm
join products p on sm.product = p.id
) a
left join (
select p.id,
p.section,
magic_count
from (
select m.product, sum(count) as magic_count
from matrix m
group by m.product
) sm
join products p on sm.product = p.id
) b on a.section = b.section and a.magic_count < b.magic_count
where b.id is null
see a simplified example (and other methods) in the manual entry for The Rows Holding the Group-wise Maximum of a Certain Column
see it working live here
Here you have solution without using JOINs, it has better performance than the other answer, which uses lot of JOINs:
select #rn := 1, #sectionLag := 0;
select id, section, count from (
select id,
case when #sectionLag = section then #rn := #rn + 1 else #rn := 1 end rn,
#sectionLag := section,
section,
count
from (
select id, section, sum(count) count
from matrix m
join products p on m.product = p.id
group by id, section
) a order by section, count desc
) a where rn = 1
Variables at the beginning are used to imitate window functions (LAG and ROW_NUMBER), which are available in MySQL 8.0 or higher (if you are using such version, let me know, so I will give you solution also with window functions).
DEMO
Another demo, where you can compare performance of my and the other query. It contains ~20K rows and my query tends to be almost 2 times faster.
Related
Let's say we have a person table and survey table. survey is a set of attributes collected from a person at some point in time. Let's say survey has columns address and marriage_status
How do I select all persons whose address or marriage status has changed in the last survey?
Here's how I would write it if MySQL were able to magically interpret my intention:
SELECT *
FROM person
JOIN
(SELECT *
FROM survey
GROUP BY survey.person_id
ORDER BY survey.timestamp DESC
LIMIT 2 EACH) -- of course this part doesn't actually work. Trying to get last 2 records per person
surveys
ON surveys.person_id = person.id
WHERE surveys[0].address != surveys[1].address
OR surveys[0].marriage_status != surveys[1].marriage_status;
OR
SELECT *
FROM person
JOIN
(SELECT MOST RECENT survey FOR EACH person) latest_survey
ON latest_survey.person_id = person.id
JOIN
(SELECT SECOND MOST RECENT survey FOR EACH person) previous_survey
ON previous_survey.person_id = person.id
WHERE latest_survey.address != previous_survey.address
OR latest_survey.marriage_status != previous_survey.marriage_status;
This seems like a relatively straightforward query, but it's driving me crazy. I suspect I have tunnel vision and I'm not approaching this the right way.
EDIT: I am on MySQL v5. Based on the first couple answers, it seems like this might be the time to migrate to v8 (among other reasons)
So here's how I ended up doing it. It's a little long, but I think it's pretty straightforward? This felt amazing to get working.
(Note that underscores are used as prefixes in table aliases to help keep track of subquery depth)
SELECT person.*
FROM person
JOIN (
-- Join full survey data against each 'most recent' survey timestamp
SELECT s1.*
FROM survey s1
JOIN (
-- get most recent timestamp for each person
SELECT _s1.person_id, MAX(_s1.timestamp) timestamp
FROM survey _s1
GROUP BY person_id
) latest_surveys
ON latest_surveys.person_id = s1.person_id and latest_surveys.timestamp = s1.timestamp
) latest
ON latest.person_id = person.id
JOIN (
-- Join full survey data against each 'SECOND most recent' survey timestamp
select s2.*
from survey s2
JOIN (
-- to get SECOND most recent survey timestamp, do similar query, but exclude latest timestamp
SELECT _s2.person_id, MAX(_s2.timestamp) timestamp
FROM survey _s2
JOIN (
-- get most recent timestamp for each person (again)
SELECT __s2.person_id, MAX(__s2.timestamp) timestamp
FROM survey __s2
GROUP BY person_id
) _latest_surveys
-- Note the *NOT* equal here
ON _latest_surveys.person_id = _s2.person_id and _latest_surveys.timestamp != _s2.timestamp
GROUP BY _s2.person_id
) previous_surveys
ON previous_surveys.person_id = s2.person_id and previous_surveys.timestamp = s2.timestamp
) previous
ON previous.person_id = person.id
WHERE latest.address != previous.address
OR latest.marriage_status != previous.marriage_status;
Analytic functions make your question much more tractable. If you are not yet using MySQL 8+, then now would be a good time to upgrade. Assuming you are using MySQL 8+, we can try:
WITH cte AS (
SELECT *, ROW_NUMBER() OVER (PARTITION BY p.id ORDER BY s.timestamp DESC) rn
FROM person p
INNER JOIN survey s ON p.id = s.person_id
)
SELECT id
FROM cte
GROUP BY id
HAVING
MAX(CASE WHEN rn = 1 THEN address END) <> MAX(CASE WHEN rn = 2 THEN address END) OR
MAX(CASE WHEN rn = 1 THEN marriage_status END) <> MAX(CASE WHEN rn = 2 THEN marriage_status END);
The above query uses a pivot trick to isolate the latest, and second latest, addresses and marriage statuses for each person. It retains person id values for those whose latest and second latest addresses or marriage statuses are not identical.
This might be how you can achieve that:
SELECT *
FROM person
JOIN (
SELECT *,
MAX(survey_date) latest_survey,
SUBSTRING_INDEX(SUBSTRING_INDEX(GROUP_CONCAT(survey_date ORDER BY person_id, survey_date ASC),',',-2),',',1) previous_survey,
SUBSTRING_INDEX(GROUP_CONCAT(address ORDER BY person_id, survey_date ASC),',',-1) curadd,
SUBSTRING_INDEX(SUBSTRING_INDEX(GROUP_CONCAT(address ORDER BY person_id, survey_date ASC),',',-2),',',1) prevadd,
SUBSTRING_INDEX(GROUP_CONCAT(marriage_status ORDER BY person_id, survey_date ASC),',',-1) curms,
SUBSTRING_INDEX(SUBSTRING_INDEX(GROUP_CONCAT(marriage_status ORDER BY person_id, survey_date ASC),',',-2),',',1) prevms
FROM survey GROUP BY person_id
HAVING curadd != prevadd OR curms != prevms) A
ON person.id=A.person_id;
Using GROUP_CONCAT and SUBSTRING_INDEX to combine the data value then separate it again and using those to compare at the end. I know there are a bunch of ways to achieve without all these, like your second example is something that I think can be done but when I think about it, it's going to be a very long query. This query however, since you're not using MySQL 8+ is much shorter but the performance of this query is a concern especially on a large table.
It is not given, but I hope you have at least MySQL 8 or similar to have ability to use Common Table Expression. It can simplify the complex query.
The trick part is getting survey records #1 and #2 for each user. I will do it this way: see cte1 and cte2 definition
WITH
cte1 AS (
SELECT MAX(x1.id) AS id, x1.person_id
FROM survey x1
GROUP BY x1.person_id),
cte2 AS (
SELECT MAX(x2.id) AS id, x2.person_id
FROM survey x2
JOIN cte1 ON cte1.person_id = x2.person_id
AND cte1.id > x2.id
GROUP BY x2.person_id)
SELECT
p.*,
s1.address, s2.address address2,
s1.marriage_status, s2.marriage_status marriage_status2
FROM person AS p
JOIN (
cte1 JOIN survey s1 ON s1.id = cte1.id
) ON cte1.person_id = p.id
JOIN (
cte2 JOIN survey s2 ON s2.id = cte2.id
) ON cte2.person_id = p.id
WHERE
(s1.address <> s2.address)
OR (s1.marriage_status <> s2.marriage_status)
https://www.db-fiddle.com/f/hLwdHiZin4MkdUZ4aBz67H/2
Update: Thanks to Ian, I replaced MIN to MAX to get recent records
Let's assume I have 2 tables. One contains car manufacturer's names and their IDs, the second contains information about car models. I need to select few of them from the first table, but order them by quantity of linked from the second table data.
Currently, my query looks like this:
SELECT DISTINCT `manufacturers`.`name`,
`manufacturers`.`cars_link`,
`manufacturers`.`slug`
FROM `manufacturers`
JOIN `cars`
ON manufacturers.cars_link = cars.manufacturer
WHERE ( NOT ( `manufacturers`.`cars_link` IS NULL ) )
AND ( `cars`.`class` = 'sedan' )
ORDER BY (SELECT Count(*)
FROM `cars`
WHERE `manufacturers`.cars_link = `cars`.manufacturer) DESC
It was working ok for my table of scooters which size is few dozens of mb. But now i need to do the same thing for the cars table, which size is few hundreds megabytes. The problem is that the query takes very long time, sometimes it even causes nginx timeout. Also, i think, that i have all the necesary database indexes. Is there any alternative for the query above?
lets try to use subquery for your count instead.
select * from (
select distinct m.name, m.cars_link, m.slug
from manufacturers m
join cars c on m.cars_link=c.manufacturer
left join
(select count(1) ct, c1.manufacturer from manufacturers m1
inner join cars_link c2 on m1.cars_link=c2.manufacturer
where coalesce(m1.cars_link, '') != '' and c1.class='sedan'
group by c1.manufacturer) as t1
on t1.manufacturer = c.manufacturer
where coalesce(m.cars_link, '') != '' and c.class='sedan') t2
order by t1.ct
I have a MySQL table containing player points for serveral categories (p1, p2 etc) and player id (pid).
I have a query that counts SUM of points for each category, puts them as aliases and groups them by player id (pid).
SELECT *,
SUM(p1) as p1,
SUM(p2) as p2,
SUM(p3) as p3,
SUM(p4) as p4,
SUM(p6) as p6,
SUM(p13) as p13,
SUM(p14) as p14,
SUM(p15) as p15,
SUM(p16) as p16,
SUM(p17) as p17,
SUM(p18) as p18,
SUM(p19) as p19,
SUM(p20) as p20,
SUM(p21) as p21
FROM results GROUP BY pid
Futher I do a while loop and update other table with these alias values.
Now I have a need to count only top 5 or 12 (depending on a category) values for each group. I don't know where to start. I found similar questions, but none of them addresses putting value in an alias, so i don't have to change futher code.
Can someone help me, and write an example query for at least two categories, so i can understand a principle of doing this right?
Thank you in advance!
As we need to do sum of top n records, we need to use something like this:
SELECT pid, sum(p1)
FROM (SELECT p.*,
(#pn := if(#p = pid, #pn + 1,
if(#p := pid, 1, 1)
)
) as seqnum
FROM player p CROSS JOIN
(SELECT #p := 0, #pn := 0) as p1
ORDER BY pid, p1 DESC
) p
WHERE seqnum <= 1
GROUP BY pid;
Here, we can modify seqnum <= 1 condition as per the number of records needed. E.g. if we want 5 records then we need to write seqnum <= 5.
Please note that this will only calculate Top n sum for a particular field. If we want multiple fields then we may need to repeat the query.
Here is the SQL Fiddle example to play around with.
Building on the answer by #DarshanMehta , you can do repeated sub queries like that. Note that the variable names in each sub query need to be different.
Something like this, assuming you have a table of players:-
SELECT players.pid,
suba1.p1sum,
suba2.p2sum
FROM players
LEFT OUTER JOIN
(
SELECT pid, SUM(p1) AS p1sum
FROM (SELECT r.pid,
r.p1,
#p1n := if(#p1 = pid, #p1n + 1, 1) AS seqnum,
#p1 := pid
FROM results r
CROSS JOIN (SELECT #p1 := 0, #p1n := 0) as p1
ORDER BY r.pid, r.p1 DESC
) sub1
WHERE seqnum <= 5
GROUP BY pid
) suba1
ON players.pid = suba1.pid
LEFT OUTER JOIN
(
SELECT pid, SUM(p2) AS p1sum
FROM (SELECT r.pid,
r.p2,
#p2n := if(#p2 = pid, #p2n + 1, 1) AS seqnum,
#p2 := pid
FROM results r
CROSS JOIN (SELECT #p2 := 0, #p2n := 0) as p2
ORDER BY r.pid, r.p2 DESC
) sub1
WHERE seqnum <= 5
GROUP BY pid
) suba2
ON players.pid = suba1.pid
You can build a table with all that SUM information, and use this one:
SELECT * from newTable ORDER BY p1 DESC LIMIT 5;
and you can catch all info that you want, by changing the field p1 and LIMIT 5
This almost seems like a scope issue- the select statement in the subquery doesn't recognize table 'candidate':
SELECT
candidate.id AS id,
candidate.image AS image,
candidate.name AS name,
candidate.party AS party,
player.order AS player_order,
c_pcts.pct AS pct
FROM `candidate`
INNER JOIN players player ON player.candidate_id = candidate.id
INNER JOIN lineups lineup ON player.lineup_id = lineup.id
INNER JOIN (
SELECT
pct
FROM candidate_pcts p
INNER JOIN weekly_game game ON p.weekly_game_id = (
SELECT id FROM weekly_game ORDER BY date DESC LIMIT 1
) WHERE p.candidate_id = candidate.id
) c_pcts
WHERE lineup.id = '31'
ORDER BY player.order ASC
gives the error: "Unknown column 'candidate.id' in 'where clause'." If instead of "FROM candidate_pcts p" I put
FROM candidate_pcts p, candidate c
then it doesn't see 'p.weekly_game_id' ...huh?
Seems like I need to identify the 'candidate' table for the subquery somehow but everything I'm trying leads me only further astray. And I have tried a mess of things: order of the tables, explicitly identifying them everywhere i could think of, backticks. I should note that the nested subquery works like a charm. Here it is again:
SELECT
pct
FROM `candidate_pcts`
INNER JOIN weekly_game game ON candidate_pcts.weekly_game_id = (
SELECT id FROM weekly_game ORDER BY date DESC LIMIT 1
) WHERE candidate_pcts.candidate_id = '5'
with a hardcoded an id value there, of course. I can supply database structure if needed here, but this is long already. The 'weekly_game' table is simply a set of scores for each candidate each week and we only want the most recent week's score, thus the 'ORDER BY date DESC LIMIT 1' clause.
Thanks very much for your time.
Tables:
table candidate: {id, image, name, party}
table candidate_pcts: {id, candidate_id, pct, weekly_game_id}
table lineups: {id, date, user_id}
table players: {id,candidate_id,lineup_id,order}
table weekly_game: {id,date}
You are basically on the right track around the problem. In essence the nested sub-select does not know about candidate.id. It you break apart the query and just look at the sub-select in question:
SELECT
pct
FROM candidate_pcts p
INNER JOIN weekly_game game ON p.weekly_game_id = (
SELECT id FROM weekly_game ORDER BY date DESC LIMIT 1
) WHERE p.candidate_id = candidate.id
You can see there is NO reference whatsoever in that query to the candidate table other than in your where clause, thus this is an unknown column.
Since a subselect is, in essence, made before the outer select that references it, the subselect must be a standalone, executable query.
Thanks to all, especially Mike for that excellent explanation. What I did was restructured the query like so:
SELECT
candidate.id AS id,
candidate.image AS image,
candidate.name AS name,
candidate.party AS party,
player.order AS player_order,
pcts.pct AS pct
FROM `candidate`
INNER JOIN players player ON player.candidate_id = candidate.id
INNER JOIN lineups lineup ON player.lineup_id = lineup.id
LEFT JOIN (
SELECT
p.candidate_id AS pct_id, pct AS pct
FROM candidate_pcts p
INNER JOIN weekly_game game ON p.weekly_game_id = (
SELECT id FROM weekly_game ORDER BY date DESC LIMIT 1
)
) pcts
ON pct_id = candidate.id
WHERE lineup.id = '$lineup_id'
ORDER BY player.order ASC
I have something like this:
SELECT id, fruit, pip
FROM plant
WHERE COUNT(*) = 2;
This weird query is self explanatory I guess. COUNT(*) here means the number of rows in plant table. My requirement is that I need to retrieve values from specified fields only if total number of rows in table = 2. This doesn't work but: invalid use of aggregate function COUNT.
I cannot do this:
SELECT COUNT(*) as cnt, id, fruit, pip
FROM plant
WHERE cnt = 2;
for one, it limits the number of rows outputted to 1, and two, it gives the same error: invalid use of aggregate function.
What I can do is instead:
SELECT id, fruit, pip
FROM plant
WHERE (
SELECT COUNT(*)
FROM plant
) = 2;
But then that subquery is the main query re-run. I'm presenting here a small example of the larger part of the problem, though I know an additional COUNT(*) subquery in the given example isn't that big an overhead.
Edit: I do not know why the question is downvoted. The COUNT(*) I'm trying to get is from a view (a temporary table) in the query which is a large query with 5 to 6 joins and additional where clauses. To re-run the query as a subquery to get the count is inefficient, and I can see the bottleneck as well.
Here is the actual query:
SELECT U.UserName, E.Title, AE.Mode, AE.AttemptNo,
IF(AE.Completed = 1, 'Completed', 'Incomplete'),
(
SELECT COUNT(DISTINCT(FK_QId))
FROM attempt_question AS AQ
WHERE FK_ExcAttemptId = #excAttemptId
) AS Inst_Count,
(
SELECT COUNT(DISTINCT(AQ.FK_QId))
FROM attempt_question AS AQ
JOIN `question` AS Q
ON Q.PK_Id = AQ.FK_QId
LEFT JOIN actions AS A
ON A.FK_QId = AQ.FK_QId
WHERE AQ.FK_ExcAttemptId = #excAttemptId
AND (
Q.Type = #descQtn
OR Q.Type = #actQtn
AND A.type = 'CTVI.NotImplemented'
AND A.IsDelete = #status
AND (
SELECT COUNT(*)
FROM actions
WHERE FK_QId = A.FK_QId
AND type != 'CTVI.NotImplemented'
AND IsDelete = #status
) = 0
)
) AS NotEvalInst_Count,
(
SELECT COUNT(DISTINCT(FK_QId))
FROM attempt_question AS AQ
WHERE FK_ExcAttemptId = #excAttemptId
AND Mark = #mark
) AS CorrectAns_Count,
E.AllottedTime, AE.TimeTaken
FROM attempt_exercise AS AE
JOIN ctvi_exercise_tblexercise AS E
ON AE.FK_EId = E.PK_EId
JOIN ctvi_user_table AS U
ON AE.FK_UId = U.PK_Id
JOIN ctvi_grade AS G
ON AE.FK_GId = G.PK_GId
WHERE AE.PK_Id = #excAttemptId
-- AND COUNT(AE.*) = #number --the portion in contention.
Kindly ignore the above query and guide me to right direction from the small example query I posted, thanks.
In MySQL, you can only do what you tried:
SELECT id, fruit, pip
FROM plant
WHERE (
SELECT COUNT(*)
FROM plant
) = 2;
or this variation:
SELECT id, fruit, pip
FROM plant
JOIN
(
SELECT COUNT(*) AS cnt
FROM plant
) AS c
ON c.cnt = 2;
Whether the 1st or the 2nd is more efficient, depends on the version of MySQL (and the optimizer). I would bet on the 2nd one, on most versions.
In other DBMSs, that have window functions, you can also do the first query that #Andomar suggests.
Here is a suggestion to avoid the bottleneck of calculating the derived table twice, once to get the rows and once more to get the count. If the derived table is expensive to be calculated, and its rows are thousands or millions, calculating them twice only to throw them away, is a problem, indeed. This may improve efficiency as it will limit the intermediately (twice) calculated rows to 3:
SELECT p.*
FROM
( SELECT id, fruit, pip
FROM plant
LIMIT 3
) AS p
JOIN
( SELECT COUNT(*) AS cnt
FROM
( SELECT 1
FROM plant
LIMIT 3
) AS tmp
) AS c
ON c.cnt = 2 ;
After re-reading your question, you're trying to return rows only if there are 2 rows in the entire table. In that case I think your own example query is already the best.
On another DBMS, you could use a Windowing function:
select *
from (
select *
, count(*) over () as cnt
from plant
) as SubQueryAlias
where cnt = 2
But the over clause is not supported on MySQL.
old wrong anser below
The where clause works before grouping. It works on single rows, not groups of rows, so you can't use aggregates like count or max in the where clause.
To set filters that work on groups of rows, use the having clause. It works after grouping and can be used to filter with aggregates:
SELECT id, fruit, pip
FROM plant
GROUP BY
id, fruit, pip
HAVING COUNT(*) = 2;
The other answers do not fulfill the original question which was to filter the results "without using a subquery".
You can actually do this by using a variable in 2 consecutive MySQL statements:
SET #count=0;
SELECT * FROM
(
SELECT id, fruit, pip, #count:=#count+1 AS count
FROM plant
WHERE
) tmp
WHERE #count = 2;