I hope someone can help me with my MySQL problem. I have a bug where if there is one left outer join on contribution table, result of amount is $100 (which is correct). If I include a second left outer join of another table (ikes). And I have 2 ikes, it doubles amount ($200), if I have 3 ikes, it triples ($300). For the life of me, I cannot figure this out. What do the ikes have any to do with the contribution amount? I've separated the queries and they work by themselves. But together they cause the problem.
Can anyone see the problem? I've included the query and the tables below.
SELECT COUNT(i.type) AS xlike,
SUM(c.amount) AS amount,
w.*
FROM wish w
LEFT OUTER JOIN contributions c ON w.ID=c.receiveid
LEFT OUTER JOIN ikes i ON w.ID=i.wishid
WHERE w.ID = 236
Tables:
CREATE TABLE IF NOT EXISTS `contributions` (
`ID` int(11) NOT NULL AUTO_INCREMENT,
`amount` decimal(19,2) NOT NULL,
PRIMARY KEY (`ID`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=3 ;
CREATE TABLE IF NOT EXISTS `ikes` (
`ID` int(11) NOT NULL AUTO_INCREMENT,
`type` enum('likes','dislikes') NOT NULL,
`wishid` int(11) NOT NULL,
PRIMARY KEY (`ID`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=1;
While most will tell you to use JOINs, you have to be aware that joins will duplicate parent records if more than one child record is associated to it. This is what can inflate values from aggregate functions.
I re-wrote your query as:
SELECT w.*,
COALESCE(x.amount, 0) AS amount,
COALESCE(y.type, 0) AS type
FROM WISH w
LEFT JOIN (SELECT c.receiveid,
SUM(c.amount) AS amount
FROM CONTRIBUTIONS c
GROUP BY c.receiveid) x ON x.receiveid = w.ID
LEFT JOIN (SELECT i.wishid,
COUNT(i.type) AS type
FROM IKES i
GROUP BY i.wishid) y ON y.wishid = w.ID
WHERE w.ID = 236
Related
I'm a student of Java and do SQL too. In a lesson we were presented with an example database sketch, and a query that a replicate in this question.
I have made an example with MySql and it has three tables,
CREATE TABLE `employed` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(45) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=utf8;
CREATE TABLE `department` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(45) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=utf8;
CREATE TABLE `employees_departments` (
`employed_id` int(11) NOT NULL,
`department_id` int(11) NOT NULL,
PRIMARY KEY (`employed_id`,`department_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
employed was filled with
(1 'Karl'), (2 'Bengt'), (3 'Adam'), (4 'Stefan')
department was filled with
(4, 'HR'), (5, 'Sälj'), (6, 'New departm')
employees_departments was filled with
1 4
2 5
3 4
So "Stefan" has no department, and "New departm" has no employed.
I wanted a query that would give the employees with all their departments, and employees without departments and departments with no employees. I found on solution like this:
select A.name, C.name from employed A
left join employees_departments B on (A.id=B.employed_id)
left join department C on (B.department_id = C.id)
union
select C.name, A.name from department A
left join employees_departments B on (A.id=B.department_id)
left join employed C on (B.employed_id = C.id)
Would be nice if there was a short query to make it...
Also, I made this without foreign key constraints, since I want to do it as simple as possible for this example.
Greetings
MySQL doesn't support a FULL OUTER join operation.
We can emulate that by combining two sets... the result of an OUTER JOIN and the result from an anti-JOIN.
(
SELECT ee.name AS employed_name
, dd.name AS department_name
FROM employed ee
LEFT
JOIN employees_departments ed
ON ed.employed_id = ee.id
LEFT
JOIN department dd
ON dd.id = ed.department_id
)
UNION ALL
(
SELECT nn.name AS employed_name
, nd.name AS department_name
FROM department nd
LEFT
JOIN employees_departments ne
ON ne.deparment_id = nd.id
LEFT
JOIN employeed nn
ON nn.id = nd.employee_id
WHERE nn.id IS NULL
)
The first SELECT returns all employed name, along with matching department name, including employed that have no department.
The second SELECT returns just department name that have no matching rows in employed.
The results from the two SELECT are combined/concatenated using a UNION ALL set operator. (The UNION ALL operation avoids a potentially expensive "Using filesort" operation that would be forced with the UNION set operator.
This is the shortest query pattern to return these rows.
We could make the SQL a little shorter. For example, if we have a foreign key relationships between employeed_department and employed (no indication in the original post that such a relationship is enforced, so we don't assume that there is one)... but if that is enforced, then we could omit the employed table from the second SELECT
UNION ALL
(
SELECT NULL AS employed_name
, nd.name AS department_name
FROM department nd
LEFT
JOIN employees_departments ne
ON ne.deparment_id = nd.id
WHERE ne.department_id IS NULL
)
With suitable indexes available, this is going to give us the most efficient access plan.
Is there shorter SQL that will return an equivalent result? If there is, it's likely not going to perform as efficiently as the above.
i have the current query:
select m.id, ms.severity, ms.risk_score, count(distinct si.id), boarding_date_tbl.boarding_date
from merchant m
join merchant_has_scan ms on m.last_scan_completed_id = ms.id
join scan_item si on si.merchant_has_scan_id = ms.id and si.is_registered = true
join (select m.id merchant_id, min(s_for_boarding.scan_date) boarding_date
from merchant m
left join merchant_has_scan ms on m.id = ms.merchant_id
left join scan s_for_boarding on s_for_boarding.id = ms.scan_id and s_for_boarding.scan_type = 1
group by m.id) boarding_date_tbl on boarding_date_tbl.merchant_id = m.id
group by m.id
limit 100;
when i run it on big scheme (about 2mil "merchant") it takes more then 20 sec.
but if i'll split it to:
select m.legal_name, m.unique_id, m.merchant_status, s_for_boarding.scan_date
from merchant m
join merchant_has_scan ms on m.id = ms.merchant_id
join scan s_for_boarding on s_for_boarding.id = ms.scan_id and s_for_boarding.scan_type = 1
group by m.id
limit 100;
and
select m.id, ms.severity, ms.risk_score, count(distinct si.id)
from merchant m
join merchant_has_scan ms on m.last_scan_completed_id = ms.id
join scan_item si on si.merchant_has_scan_id = ms.id and si.is_registered = true
group by m.id
limit 100;
both will take about 0.1 sec
the reason for that is clear, the low limit means it doesn't need to do much to get the first 100. it is also clear that the inner select cause the first query to run as much as it does.
my question is there a way to do the inner select only on the relevant merchants and not on the entire table?
Update
making a left join instead of a join before the inner query help reduce it to 6 sec, but it still a lot more then what i can get if i do 2 queries
UPDATE 2
create table for merchant:
CREATE TABLE `merchant` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`last_scan_completed_id` bigint(20) DEFAULT NULL,
`last_updated` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
CONSTRAINT `FK_9lhkm7tb4bt87qy4j3fjayec5` FOREIGN KEY (`last_scan_completed_id`) REFERENCES `merchant_has_scan` (`id`)
)
merchant_has_scan:
CREATE TABLE `merchant_has_scan` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`merchant_id` bigint(20) NOT NULL,
`risk_score` int(11) DEFAULT NULL,
`scan_id` bigint(20) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `unique_merchant_id` (`scan_id`,`merchant_id`),
CONSTRAINT `FK_3d8f81ts5wj2u99ddhinfc1jp` FOREIGN KEY (`scan_id`) REFERENCES `scan` (`id`),
CONSTRAINT `FK_e7fhioqt9b9rp9uhvcjnk31qe` FOREIGN KEY (`merchant_id`) REFERENCES `merchant` (`id`)
)
scan_item:
CREATE TABLE `scan_item` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`is_registered` bit(1) NOT NULL,
`merchant_has_scan_id` bigint(20) NOT NULL,
PRIMARY KEY (`id`),
CONSTRAINT `FK_avcc5q3hkehgreivwhoc5h7rb` FOREIGN KEY (`merchant_has_scan_id`) REFERENCES `merchant_has_scan` (`id`)
)
scan:
CREATE TABLE `scan` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`scan_date` datetime DEFAULT NULL,
`scan_type` int(11) NOT NULL,
PRIMARY KEY (`id`)
)
and the explain:
You don't have the latest version of MySQL, which would be able to create an index for the derived table. (What version are you running?)
The "derived table" (the subquery) will be the first table in the EXPLAIN because, well, it has to be.
merchant_has_scan is a many:many table, but without the optimization tips here -- fixing this may be the biggest factor in speeding it up. Caveat: The tips suggest getting rid of id, but you seem to have a use for id, so keep it.
The COUNT(DISTINCT si.id) and JOIN si... can be replaced by ( SELECT COUNT(*) FROM scan_item WHERE ...), thereby eliminating one of the JOINs and possibly diminishing the Explode-Implode .
LEFT JOIN -- are you sometimes expecting to get NULL for boarding_date? If not, please use JOIN, not LEFT JOIN. (It is better to state your intention than to leave the query open to multiple interpretations.)
If you can remove the LEFTs, then since m.id and merchant_id are specified to be equal, why list them both in the SELECT? (This is a confusion factor, not a speed question).
You say you split it into two -- but you did not. You added LIMIT 100 to the inner query when you pulled it out. If you need that, add it to the derived table, too. Then you may be able to remove GROUP BY m.id LIMIT 100 from the outer query.
I have three empty tables
--
-- Tabellenstruktur für Tabelle `projects`
--
CREATE TABLE IF NOT EXISTS `projects` (
`id_project` int(11) NOT NULL AUTO_INCREMENT,
`id_plan` int(11) DEFAULT NULL,
`name` varchar(255) NOT NULL,
`description` longtext NOT NULL,
PRIMARY KEY (`id_project`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=2 ;
-- --------------------------------------------------------
--
-- Tabellenstruktur für Tabelle `project_plans`
--
CREATE TABLE IF NOT EXISTS `project_plans` (
`id_plan` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) NOT NULL,
`description` longtext NOT NULL,
`max_projects` int(11) DEFAULT NULL,
`max_member` int(11) DEFAULT NULL,
`max_filestorage` bigint(20) NOT NULL DEFAULT '3221225472' COMMENT '3GB Speicherplatz',
PRIMARY KEY (`id_plan`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=2 ;
-- --------------------------------------------------------
--
-- Tabellenstruktur für Tabelle `project_users`
--
CREATE TABLE IF NOT EXISTS `project_users` (
`id_user` int(11) NOT NULL,
`id_project` int(11) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
All these tables are empty but i get a result with my query?
my query:
SELECT
A.id_plan,
A.name AS plan_name,
A.description AS plan_description,
A.max_projects,
A.max_member,
A.max_filestorage,
B.id_plan,
B.name AS project_name,
B.description AS project_description,
C.id_user,
C.id_project,
COUNT(*) AS max_project_member
FROM
".$this->config_vars["projects_plans_table"]." AS A
LEFT JOIN
".$this->config_vars["projects_table"]." AS B
ON
B.id_plan = A.id_plan
LEFT JOIN
".$this->config_vars["projects_user_table"]." AS C
ON
C.id_project = B.id_project
WHERE
C.id_project = '".$id."'
&& B.deleted = '0'
i think the problem is the COUNT (*) AS ...
how i can solve the problem?
For one, you are getting a record explicitly due to the COUNT(). Even though you have no records, you are asking the engine how many records which at worst case will return zero. Count(), like other aggregates are anticipated to have a group by, so even though you don't have one, you are still asking.
So the engine is basically stating hey... there are no records, but I have to send you a record so you can get the count() column to look at and do with what you will. So, it is doing what you asked.
Now, for the comment to the other question where you asked...
Yes but i want to count the project member from a project, how i can count the users from project_users where all users have the id_project 1.
Since you only care about a count, and not the specific WHO involved, you can get this result directly from the project_users table (which should have an index on both the ID_User and another on the ID_Project. Then
select count(*)
from project_users
where id_project = 1
To expand from basis of your original question to get the extra details, I would do...
select
p.id_project,
p.id_plan,
p.name as projectName,
p.description as projectDescription,
pp.name as planName,
pp.description as planDescription,
pp.max_projects,
pp.max_member,
pp.max_filestorage,
PJCnt.ProjectMemberCount
from
( select id_project,
count(*) as ProjectMemberCount
from
project_users
where
id_project = 1 ) PJCnt
JOIN Projects p
on PJCnt.id_project = p.id_project
JOIN Project_Plans PP
on p.id_plan = pp.id_plan
Now, based on this layout of tables, a plan can have a max member count, but there is nothing indicating max members for the plan based on all projects, or max per SINGLE project. So, if a plan allows for 20 people, can there be 20 people for 10 different projects under the same plan? That's something only you would know the impact of... just something to consider what you are asking for.
Your cleaned-up query should look like :
See sqlfidle demo as well : http://sqlfiddle.com/#!2/e693f5/9
SELECT
A.id_plan,
A.name AS plan_name,
A.description AS plan_description,
A.max_projects,
A.max_member,
A.max_filestorage,
B.id_plan,
B.name AS project_name,
B.description AS project_description,
C.id_user,
C.id_project,
COUNT(*) AS max_project_member
FROM
project_plans AS A
LEFT JOIN
projects AS B
ON
B.id_plan = A.id_plan
LEFT JOIN
project_users AS C
ON
C.id_project = B.id_project
WHERE
C.id_project = '".$id."';
This will return you null values for all the cols from the select because you have one legit return form the result set and that is the count(*) output 0.
To fix this just add a group by at the end (see group by example http://sqlfiddle.com/#!2/14d46/2) or
Remove the count(*) and the null values will be gone as well as the count(*) values 0
See simple sql example here : http://sqlfiddle.com/#!2/ab7dd/5
Just comment the count() and you fixed you null problem!
I've got the SQL query below:
SELECT message, sent_date, user_id
FROM messages
LEFT JOIN numbers ON messages.from_id = numbers.id
It returns all the rows (about 4000) in the messages table with additional columns coming from the numbers table. So far, this is what I would expect.
Now I left join this sub-query to another table, again using a left join:
SELECT message, sent_date
FROM (
SELECT message, sent_date, user_id
FROM messages
LEFT JOIN numbers ON messages.from_id = numbers.id
) AS table1
LEFT JOIN users ON table1.user_id = users.id
However, it only returns about 200 rows so many are missing. Since this is a left join I would expect all the rows from table1 to be in the result. Can anybody see what the issue is?
Edit:
So for information here are the 3 relevant tables (with irrelevant columns removed):
CREATE TABLE IF NOT EXISTS `messages` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`message` text CHARACTER SET utf8 NOT NULL,
`from_id` int(11) DEFAULT NULL,
`sent_date` datetime NOT NULL,
PRIMARY KEY (`id`),
KEY `from_id` (`from_id`),
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=101553 ;
CREATE TABLE IF NOT EXISTS `numbers` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) DEFAULT NULL,
`number` varchar(32) NOT NULL,
PRIMARY KEY (`id`),
KEY `user_id` (`user_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=6408 ;
CREATE TABLE IF NOT EXISTS `users` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(256) CHARACTER SET utf8 DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=2395 ;
You can try alternative method to debug the issue:
CREATE TEMPORARY table tmp1 AS SELECT message, sent_date, user_id
FROM messages
LEFT JOIN numbers
ON messages. from_id = numbers.id;
and then see whether this query works.
SELECT message, sent_date
FROM tmp1 table1
LEFT JOIN users
ON table1.user_id = users.id;
Also for your case make sure that there are no other insert or updates in between. otherwise use transactions.
table1 sometimes won't have a UserID - so that'll be null, so those results will be missing?
I don't have an exact answer to your question, but if I have to start thinking, I will first find out what 3800 rows are missing and try to see the pattern (is it because user_id are null or duplicate)
SELECT message, sent_date, user_id
FROM messages
LEFT JOIN numbers ON messages.from_id = numbers.id
MINUS
(SELECT table1.message, table1.sent_date, table1.user_id
FROM (
SELECT message, sent_date, user_id
FROM messages
LEFT JOIN numbers ON messages.from_id = numbers.id
) AS table1
LEFT JOIN users ON table1.user_id = users.id)
Try this, I think it's a scoping issue on user_id.
SELECT table1.message, table1.sent_date
FROM (
SELECT messages.message, messages.sent_date, numbers.user_id
FROM messages
LEFT JOIN numbers ON messages.from_id = numbers.id
) AS table1
LEFT JOIN users ON table1.user_id = users.id
I'm not sure if user_id is in messages or numbers.
There is no way this should happen.
Try this variation:
SELECT
m.message, m.sent_date, n.user_id
FROM
messages m
LEFT JOIN
numbers AS n ON m.from_id = n.id
LEFT JOIN
users AS u ON n.user_id = u.id ;
I am working on a property website and have record sets for property and for unit, unit has a one-to-many relationship with property. What I'm trying to figure out is how to best create a search function which will output results based on criteria from both. So if I search for a property with the location Manchester and a unit with a freehold tenure I'd like to eliminate all properties which don't have a unit with the tenure of freehold.
A potential solution I've considered is to create a record set for properties which match the property criteria and then create a unit record set for units which match the unit criteria and then finally loop through the property record set in server-side code and eliminate any properties which aren't related to any of the units in the unit record set. Really not sure if this is the best way to do things though so would be keen to hear any suggestions?
Thanks
EDIT (Added table structure and MySQL):
--
-- Table structure for table `property`
--
CREATE TABLE IF NOT EXISTS `property` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` text NOT NULL,
`street` text NOT NULL,
`town` text NOT NULL,
`postcode` text NOT NULL,
`description` longtext NOT NULL,
`team_member` varchar(255) NOT NULL DEFAULT '',
`pdf` text NOT NULL,
`default_image_id` int(11) DEFAULT NULL,
`virtual_tour_link` text NOT NULL,
`date` date NOT NULL DEFAULT '0000-00-00',
`archive` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COMMENT='' AUTO_INCREMENT=13 ;
--
-- Table structure for table `unit`
--
CREATE TABLE IF NOT EXISTS `unit` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` text NOT NULL,
`description` text NOT NULL,
`size_sq_ft` int(11) DEFAULT NULL,
`size_acres` float DEFAULT NULL,
`price` float DEFAULT NULL,
`rental_price` float DEFAULT NULL,
`on_application` tinyint(1) DEFAULT NULL,
UNIQUE KEY `id` (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COMMENT='Stores data for property units' AUTO_INCREMENT=5;
--
-- Table structure for table `property_to_unit`
--
CREATE TABLE IF NOT EXISTS `property_to_unit` (
`property_id` int(11) NOT NULL,
`unit_id` int(11) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
--
-- MySQL which produces list of properties
--
SELECT
P.id AS id,
P.name AS name,
P.street AS street,
P.town AS town,
P.postcode AS postcode,
P.description AS description,
P.team_member AS team_member,
P.pdf AS pdf,
P.virtual_tour_link AS virtual_tour_link,
P.date AS date,
P.archive AS archive,
PI.name as image,
P2.image_ids as image_ids,
L2.location_ids as location_ids,
U2.unit_ids as unit_ids
FROM property P
-- Get default image and join using property id
LEFT JOIN property_image PI ON PI.id = P.default_image_id
-- Create a list of image_ids from property_image and
-- property_to_property_image tables then join using property_id
LEFT JOIN (
SELECT
property_id,
GROUP_CONCAT(CAST(id AS CHAR)) as image_ids
FROM property_to_property_image PTPI
LEFT JOIN property_image PI ON PI.id = PTPI.property_image_id
GROUP BY property_id
) P2 ON P2.property_id = P.id
-- Create a list of locations from property_location table
-- and join using property_id
LEFT JOIN (
SELECT
property_id,
property_location_id,
GROUP_CONCAT(CAST(property_location.id AS CHAR)) AS location_ids
FROM property_to_property_location
INNER JOIN property_location ON property_location.id = property_to_property_location.property_location_id
GROUP BY property_id
) L2 ON L2.property_id = P.id
-- Create a list of units from unit table
-- and join using property_id
LEFT JOIN (
SELECT
property_id,
unit_id,
GROUP_CONCAT(CAST(unit_id AS CHAR)) AS unit_ids
FROM property_to_unit
INNER JOIN unit ON unit.id = property_to_unit.unit_id
GROUP BY property_id
) U2 ON U2.property_id = P.id
--
-- MySQL which produces list of units
--
SELECT
id,
name,
description,
size_sq_ft,
size_acres,
price,
rental_price,
on_application,
tenure_ids,
tenure_names,
type_ids,
type_names
FROM unit AS U
-- join tenure ids and names
LEFT JOIN (
SELECT
unit_id,
GROUP_CONCAT( CAST(UT.id AS CHAR) ) AS tenure_ids,
GROUP_CONCAT(UT.name) AS tenure_names
FROM unit_to_unit_tenure UTUT
INNER JOIN unit_tenure UT ON UT.id = UTUT.unit_tenure_id
GROUP BY unit_id
) UT ON UT.unit_id = U.id
-- join type ids and names
LEFT JOIN (
SELECT
unit_id,
GROUP_CONCAT( CAST(UTYPE.id AS CHAR) ) AS type_ids,
GROUP_CONCAT(UTYPE.name) AS type_names
FROM unit_to_unit_type UTUT
INNER JOIN unit_type UTYPE ON UTYPE.id = UTUT.unit_type_id
GROUP BY unit_id
) UTYPE ON UTYPE.unit_id = U.id
WHERE 0=0
I'm currently using a dynamically created WHERE statement appended to each MySQL query to filter the property and unit results.
You're making it a bit more complicated than it is. If I understand correctly, you can easily do this in a single query. This would search properties that have units with a particlar unit tenure id:
select *
from property p
where p.id in (
select pu.property_id
from property_to_unit pu
inner join unit u ON pu.unit_id = u.id
inner join unit_to_unit_tenure uut ON u.id = uut.unit_id
where uut.id = <cfqueryparam value="#uutid#">
)
Using two queries and then looping through to cross-check sounds like it could be dog slow.
Your situation requires a posted foreign key in the property table. Store the unit_id in the property table and use a join in your query such as:
select * from property p, unit u
where p.unit_id = u.id
and p.town = ....
EDIT: So I just noticed the rest of your SQL. If you require to keep the many-to-many relationship table for the unit -> property relationship then you will need to join unit and property off of that table.