Getting sum from a left table of leftjoined table - mysql

Below are the tables and the SQL query. I am doing a left join and trying to get SUM of a column that's in the left table and count from the right table.
Is it possible to get both in 1 query?
https://www.db-fiddle.com/f/3QuxG1DLgWJ8aGXNbnnwU1/1
select
s.test,
count(distinct s.name),
sum(s.score) score, -- need accurate score
count(a.id) attempts -- need accurate attempt count
from question s
left join attempts a on s.id = a.score_id
group by s.test
create table question (
id int auto_increment primary key,
test varchar(25),
name varchar(25),
score int
);
create table attempts (
id int auto_increment primary key,
score_id int,
attempt_no int
);
insert into question (test, name, score) values
('test1','name1', 10),
('test1','name2', 15),
('test1','name3', 20),
('test1','name4', 25),
('test2','name1', 15),
('test2','name2', 25),
('test2','name3', 30),
('test2','name4', 20);
insert into attempts (score_id, attempt_no) values
(1, 1),
(1, 2),
(1, 3),
(1, 4),
(2, 1),
(2, 2),
(2, 3),
(2, 4);

You need to pre-aggregate before the join:
select q.test, count(distinct q.name),
sum(q.score) score, -- need accurate score
sum(a.num_attempts) attempts -- need accurate attempt count
from question q left join
(select a.score_id, count(*) as num_attempts
from attempts a
group by a.score_id
) a
on q.id = a.score_id
group by q.test;
Here is a db-fiddle.

As Gordon said above, you can pre-aggregate, but his answer will get you the incorrect number of attempts, unfortunately. This is due to an issue with how you're structuring your DB schema. It looks like your question table really records scores of attempts at questions, and your attempts table is unnecessary. You should really have a question table that simply contains an ID and a name for the question, and a attempts table that contains an attempt ID, question ID, name, and score.
create table question (
id int auto_increment primary key,
test varchar(25)
);
create table attempts (
id int auto_increment primary key,
question_id int,
name varchar(25),
score int
);
Then your query becomes as simple as:
select
q.id as question_id,
count(distinct a.name) as attempters,
sum(a.score) as total_score,
count(a.id) as total_attempts
from question q join attempts a on q.id = a.question_id
group by q.id

Related

Delete all duplicate rows in mysql

i have MySQL data which is imported from csv file and have multiple duplicate files on it,
I picked all non duplicates using Distinct feature.
Now i need to delete all duplicates using SQL command.
Note i don't need any duplicates i just need to fetch only noon duplicates
thanks.
for example if number 0123332546666 is repeated 11 time i want to delete 12 of them.
Mysql table format
ID, PhoneNumber
Just COUNT the number of duplicates (with GROUP BY) and filter by HAVING. Then supply the query result to DELETE statement:
DELETE FROM Table1 WHERE PhoneNumber IN (SELECT a.PhoneNumber FROM (
SELECT COUNT(*) AS cnt, PhoneNumber FROM Table1 GROUP BY PhoneNumber HAVING cnt>1
) AS a);
http://sqlfiddle.com/#!9/a012d21/1
complete fiddle:
schema:
CREATE TABLE Table1
(`ID` int, `PhoneNumber` int)
;
INSERT INTO Table1
(`ID`, `PhoneNumber`)
VALUES
(1, 888),
(2, 888),
(3, 888),
(4, 889),
(5, 889),
(6, 111),
(7, 222),
(8, 333),
(9, 444)
;
delete query:
DELETE FROM Table1 WHERE PhoneNumber IN (SELECT a.PhoneNumber FROM (
SELECT COUNT(*) AS cnt, PhoneNumber FROM Table1 GROUP BY PhoneNumber HAVING cnt>1
) AS a);
you could try using a left join with the subquery for min id related to each phonenumber ad delete where not match
delete m
from m_table m
left join (
select min(id), PhoneNumber
from m_table
group by PhoneNumber
) t on t.id = m.id
where t.PhoneNumber is null
otherwise if you want delete all the duplicates without mantain at least a single row you could use
delete m
from m_table m
INNER join (
select PhoneNumber
from m_table
group by PhoneNumber
having count(*) > 1
) t on t.PhoneNumber= m.PhoneNumber
Instead of deleting from the table, I would suggest creating a new one:
create table table2 as
select min(id) as id, phonenumber
from table1
group by phonenumber
having count(*) = 1;
Why? Deleting rows has a lot of overhead. If you are bringing the data in from an external source, then treat the first landing table as a staging table and the second as the final table.

Sum of cost in MySQL

I have two tables, for example 1st has id, and name.
2nd has id, link to 1st table by id and COST.
CREATE TABLE FIRST_TABLE (id INT PRIMARY KEY AUTO_INCREMENT,
name VARCHAR (100));
CREATE TABLE SECOND_TABLE (id INT PRIMARY KEY, FIRST_TABLE_ID INT NOT NULL, cost DECIMAL(10,2),
FOREIGN KEY (FIRST_TABLE_ID) REFERENCES FIRST_TABLE (ID));
INSERT INTO FIRST_TABLE (NAME) VALUES
('ONE'),
('TWO'),
('THREE');
INSERT INTO SECOND_TABLE (ID, FIRST_TABLE_ID, COST) VALUES
(1, 1, 500),
(2, 2, 400),
(3, 3, 150),
(4, 1, 500),
(5, 2, 400),
(6, 3, 150);
How to get sum of elements COST (of 2nd table), which depends on NAME (of 1st table)?
What i tried to do:
select FIRST_TABLE.NAME, sum(SECOND_TABLE.COST) TOTAL_COST
from FIRST_TABLE
left join SECOND_TABLE on FIRST_TABLE_ID = SECOND_TABLE.ID
group by FIRST_TABLE.ID
The problem is:
I have only irregular sum of cost - 1050 for every NAME.
ONE - 1050
TWO - 1050
THREE - 1050
How to get genuine values for every NAME?
And how will it look like if i have three tables and for key in 1st i have to get sum of 2nd table and 3rd table?
Here:
from FIRST_TABLE
left join SECOND_TABLE on FIRST_TABLE_ID = SECOND_TABLE.ID
The join condition is actually equivalent to:
on SECOND_TABLE.FIRST_TABLE_ID = SECOND_TABLE.ID
Both operands of the equality relate to the same table. This is not what you want. Instead, use:
select FIRST_TABLE.NAME, sum(SECOND_TABLE.COST) TOTAL_COST
from FIRST_TABLE
left join SECOND_TABLE on SECOND_TABLE.FIRST_TABLE_ID = FIRST_TABLE.ID
group by FIRST_TABLE.ID
I would also recommend using table aliases to shorten the query and make it more readable:
select t1.NAME, sum(t2.COST) TOTAL_COST
from FIRST_TABLE t1
left join SECOND_TABLE t2 on t2.FIRST_TABLE_ID = t1.ID
group by t1.ID

Subselect in WHERE or FROM clause?

I have wondered about general performance of a query if specific subselect (subquery) is located in WHERE or FROM clause. I didn't find sufficient explanation which way is better. Are there some rules how we should apply subselect in this kind of queries?
I prepared following example
Query FROM
SELECT name
FROM users a
JOIN (SELECT user_id, AVG(score) as score
FROM scores GROUP BY user_id
) b ON a.id=b.user_id
WHERE b.score > 15;
Query WHERE
SELECT name
FROM users
WHERE
(SELECT AVG(score) as score
FROM scores WHERE scores.user_id=users.id GROUP BY user_id
) > 15;
Tables:
CREATE TABLE users (
id INT PRIMARY KEY AUTO_INCREMENT,
name VARCHAR(30));
CREATE TABLE scores (
id INT PRIMARY KEY AUTO_INCREMENT,
user_id INT,
score INT);
INSERT INTO users(name)
VALUES ('John'), ('Max'), ('Dan'), ('Alex');
INSERT INTO scores(user_id, score)
VALUES
(1, 20),
(1, 19),
(2, 15),
(2, 10),
(3, 20),
(3, 18),
(4, 13),
(4, 16),
(4, 15);
In both cases, scores needs INDEX(user_id, score) for performance.
It is hard to predict which will run faster.
There are times when a query similar to the first formulation is excellent. This is because it goes off an focuses on b and efficiently calculates all the AVGs all at once. Then it reaches over to the other table for the final info.
Let's tweak the second version slightly by adding some other test in the WHERE clause. Now the second one might be faster.
This may be even better:
SELECT name
FROM ( SELECT user_id -- Don't fetch AVG if not needed
FROM scores GROUP BY user_id
HAVING AVG(score) > 15; -- Note
) b
JOIN users a ON a.id = b.user_id
(The swapping of FROM and JOIN is not an optimization; it is just to show what order the Optimizer will perform the steps.)
In some other situations, EXISTS( SELECT ... ) is beneficial. (But I don't see such in your case.
Your question was about general optimization. I'm trying to emphasize that there is no general answer.
I think this request is faster than what you give above, as it has no subqueries.
SELECT u.name
FROM users u
JOIN scores s
ON (s.user_id = u.id)
GROUP BY s.user_id
HAVING AVG(s.score) > 15
You can see it on this link: http://sqlfiddle.com/#!9/b050f9/16
it shows the execution time of the next 3 Select queries:
SELECT name
FROM users a
JOIN (SELECT user_id, AVG(score) as score
FROM scores GROUP BY user_id
) b ON a.id=b.user_id
WHERE b.score > 15;
SELECT name
FROM users
WHERE
(SELECT AVG(score) as score
FROM scores WHERE scores.user_id=users.id GROUP BY user_id
) > 15;
SELECT u.name
FROM users u
JOIN scores s
ON (s.user_id = u.id)
GROUP BY s.user_id
HAVING AVG(s.score) > 15

Selecting parent records when child mathes criteria

I am trying to limit returned results of users to results that are "recent" but where users have a parent, I also need to return the parent.
CREATE TABLE `users` (
`id` int(0) NOT NULL,
`parent_id` int(0) NULL,
`name` varchar(255) NULL,
PRIMARY KEY (`id`)
);
CREATE TABLE `times` (
`id` int(11) NOT NULL,
`time` datetime DEFAULT NULL,
PRIMARY KEY (`id`)
);
INSERT INTO `users`(`id`, `parent_id`, `name`) VALUES (1, NULL, 'Alan');
INSERT INTO `users`(`id`, `parent_id`, `name`) VALUES (2, 1, 'John');
INSERT INTO `users`(`id`, `parent_id`, `name`) VALUES (3, NULL, 'Jerry');
INSERT INTO `users`(`id`, `parent_id`, `name`) VALUES (4, NULL, 'Bill');
INSERT INTO `users`(`id`, `parent_id`, `name`) VALUES (5, 1, 'Carl');
INSERT INTO `times`(`id`, `time`) VALUES (2, '2019-01-01 14:40:38');
INSERT INTO `times`(`id`, `time`) VALUES (4, '2019-01-01 14:40:38');
http://sqlfiddle.com/#!9/91db19
In this case I would want to return Alan, John and Bill, but not Jerry because Jerry doesn't have a record in the times table, nor is he a parent of someone with a record. I am on the fence about what to do with Carl, I don't mind getting the results for him, but I don't need them.
I am filtering tens of thousands of users with hundreds of thousands of times records, so performance is important. In general I have about 3000 unique id's coming from times that could be either an id, or a parent_id.
The above is a stripped down example of what I am trying to do, the full one includes more joins and case statements, but in general the above example should be what we work with, but here is a sample of the query I am using (full query is nearly 100 lines):
SELECT id AS reference_id,
CASE WHEN (id != parent_id)
THEN
parent_id
ELSE null END AS parent_id,
parent_id AS family_id,
Rtrim(last_name) AS last_name,
Rtrim(first_name) AS first_name,
Rtrim(email) AS email,
missedappt AS appointment_missed,
appttotal AS appointment_total,
To_char(birth_date, 'YYYY-MM-DD 00:00:00') AS birthday,
To_char(first_visit_date, 'YYYY-MM-DD 00:00:00') AS first_visit,
billing_0_30
FROM users AS p
RIGHT JOIN(
SELECT p.id,
s.parentid,
Count(p.id) AS appttotal,
missedappt,
billing0to30 AS billing_0_30
FROM times AS p
JOIN (SELECT missedappt, parent_id, id
FROM users) AS s
ON p.id = s.id
LEFT JOIN (SELECT parent_id, billing0to30
FROM aging) AS aging
ON aging.parent_id = p.id
WHERE p.apptdate > To_char(Timestampadd(sql_tsi_year, -1, Now()), 'YYYY-MM-DD')
GROUP BY p.id,
s.parent_id,
missedappt,
billing0to30
) AS recent ON recent.patid = p.patient_id
This example is for a Faircom C-Tree database, but I also need to implement a similar solution in Sybase, MySql, and Pervasive, so just trying to understand what I should do for best performance.
Essentially what I need to do is somehow get the RIGHT JOIN to also include the users parent.
NOTES:
based on your fiddle config I'm assuming you're using MySQL 5.6 and thus don't have support for Common Table Expressions (CTE)
I'm assuming each name (child or parent) is to be presented as separate records in the final result set
We want to limit the number of times we have to join the times and users tables (a CTE would make this a bit easier to code/read).
The main query (times -> users(u1) -> users(u2)) will give us child and parent names in separate columns so we'll use a 2-row dynamic table plus a case statement to to pivot the columns into their own rows (NOTE: I don't work with MySQL and didn't have time to research if there's a pivot capability in MySQL 5.6)
-- we'll let 'distinct' filter out any duplicates (eg, 2 'children' have same 'parent')
select distinct
final.name
from
-- cartesian product of 'allnames' and 'pass' will give us
-- duplicate lines of id/parent_id/child_name/parent_name so
-- we'll use a 'case' statement to determine which name to display
(select case when pass.pass_no = 1
then allnames.child_name
else allnames.parent_name
end as name
from
-- times join users left join users; gives us pairs of
-- child_name/parent_name or child_name/NULL
(select u1.id,u1.parent_id,u1.name as child_name,u2.name as parent_name
from times t
join users u1
on u1.id = t.id
left
join users u2
on u2.id = u1.parent_id) allnames
join
-- poor man's pivot code:
-- 2-row dynamic table; no join clause w/ allnames will give us a
-- cartesian product; the 'case' statement will determine which
-- name (child vs parent) to display
(select 1 as pass_no
union
select 2) pass
) final
-- eliminate 'NULL' as a name in our final result set
where final.name is not NULL
order by 1
Result set:
name
==============
Alan
Bill
John
MySQL fiddle

MySQL union within derived table (related_id=a AND related_id=b) OR (related_id=z)

I have the following tables: users, tags, tags_data.
tags_data contains tag_id and user_id columns to link the users with tags in a 1 user to many tags relationship.
What is the best way of listing all users that have either tag_id 1001 AND 1003, OR tag_id 1004?
EDIT: By this I mean there could be other related tags as well, or not, just so long as there is definitely either 1004 OR (1001 AND 1003).
At the moment I've got two methods of doing this, both using a UNION in a derived table, either in the FROM clause or in an INNER JOIN clause...
SELECT subsel.user_id, users.name
FROM ( SELECT user_id
FROM tags_data
WHERE tag_id IN (1001, 1003)
GROUP BY user_id
HAVING COUNT(tag_id)=2
UNION
SELECT user_id
FROM tags_data
WHERE tag_id=1004
) AS subsel
LEFT JOIN users ON subsel.user_id=users.user_id
Or
SELECT users.user_id, users.name
FROM users
INNER JOIN ( SELECT user_id
FROM tags_data
WHERE tag_id IN (1001, 1003)
GROUP BY user_id
HAVING COUNT(tag_id)=2
UNION
SELECT user_id
FROM tags_data
WHERE tag_id=1004
) AS subsel ON users.user_id=subsel.user_id
There are other tables which I'll be LEFT JOINing on to this. 50k+ rows in the users table and 150k+ rows in the tags_data table.
This is a batch job to export data to another system so not a real-time query run by an end user, so performance isn't massively critical. However I'd like to try and get the best result I can. The query for the derived table should actually be pretty fast and it makes sense to narrow the scope of the result set down before I then add further joins, functions and calculated fields to the results returned to the client. I will be running these on a larger dataset later to see if there is any performance difference but running EXPLAIN shows an almost identical execution plan.
Generally I try and avoid UNIONs unless absolutely necessary. But I think in this case I almost have to have a UNION somewhere by definition, because of the two effectively unrelated criteria.
Is there another method that I could be using here?
And is there some sort of specific database terminology for this sort of problem?
Full example schema:
CREATE TABLE IF NOT EXISTS `tags` (
`tag_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`tag_name` varchar(255) NOT NULL,
PRIMARY KEY (`tag_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1006 ;
INSERT INTO `tags` (`tag_id`, `tag_name`) VALUES
(1001, 'tag1001'),
(1002, 'tag1002'),
(1003, 'tag1003'),
(1004, 'tag1004'),
(1005, 'tag1005');
CREATE TABLE IF NOT EXISTS `tags_data` (
`tags_data_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`user_id` int(11) NOT NULL,
`tag_id` int(11) NOT NULL,
PRIMARY KEY (`tags_data_id`),
KEY `user_id` (`user_id`,`tag_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=11 ;
INSERT INTO `tags_data` (`tags_data_id`, `user_id`, `tag_id`) VALUES
(1, 1, 1001),
(2, 1, 1002),
(3, 1, 1003),
(4, 5, 1001),
(5, 5, 1003),
(6, 5, 1005),
(7, 8, 1004),
(8, 9, 1001),
(9, 9, 1002),
(10, 9, 1004);
CREATE TABLE IF NOT EXISTS `users` (
`user_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(255) NOT NULL,
PRIMARY KEY (`user_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=11 ;
INSERT INTO `users` (`user_id`, `name`) VALUES
(1, 'user1'),
(2, 'user2'),
(3, 'user3'),
(4, 'user4'),
(5, 'user5'),
(6, 'user6'),
(7, 'user7'),
(8, 'user8'),
(9, 'user9'),
(10, 'user10');
If you are looking for performance on MySQL you should definitely avoid using nested queries and unions — most of them result in a temporary table creation and scanning without indexes. There are rare examples that the derived temporary table still uses indexes and that only work on some specific circumstances and MySQL distributions.
My suggestion would be to rewrite the query to inner/outer joins only, like this:
select distinct u.* from users as u
left outer join tags_data as t on
t.user_id=u.user_id and t.tag_id=1003
inner join tags_data as t2 on
t2.user_id=u.user_id
and (t2.tag_id=1004 or (t2.tag_id=1001 and t.tag_id=1003));
If you can be sure that no user can have both 1004 and (1001 and 1003) tags, you may also remove the "distinct" from this query, which would avoid a temporary table creation.
You should also definitely use indexes, like these:
create index tags_data__user_id__idx on tags_data(user_id);
create index tags_data__tag_id__idx on tags_data(tag_id);
This would make a 150k+ result set very easy to query.
Use an inner query that groups up all tags for each user into one value, then use a simple filter in the where clause:
select u.*
from users u
join (
select user_id, group_concat(tag_id order by tag_id) tags
from tags_data
group by user_id
) t on t.user_id = u.user_id
where tags rlike '1001.*1003|1004'
See SQLFiddle of this query running against your sample data.
If there where many tags, you could add where tag_id in (1001, 1003, 1004) to the inner query to reduce the size of the tags list as a small optimization. Testing will show whether this makes much difference.
This should perform pretty well, because each table is scanned only once.
Efficient, but inelegant, and not flexible at all:
SELECT users.*
FROM users
LEFT JOIN tags_data AS tag1001
ON (tag1001.user_id = users.user_id AND tag1001.tag_id = 1001)
LEFT JOIN tags_data AS tag1003
ON (tag1003.user_id = users.user_id AND tag1003.tag_id = 1003)
LEFT JOIN tags_data AS tag1004
ON (tag1004.user_id = users.user_id AND tag1004.tag_id = 1004)
WHERE (tag1001.tag_id AND tag1003.tag_id) OR (tag1004.tag_id);