Mysql select - where - order by clause (for Prestashop) - mysql

For those familiar with Prestashop, I am trying to add an extra sort order option in the category view. More specifically I want to add extra sort order for a selection of features.
This is the main part of the prestashop query to get the products: (the last column in the SELECT part as well as the last JOIN added by me)
$sql = 'SELECT p.*, product_shop.*, stock.out_of_stock, IFNULL(stock.quantity, 0) as quantity, MAX(product_attribute_shop.id_product_attribute) id_product_attribute, product_attribute_shop.minimal_quantity AS product_attribute_minimal_quantity, pl.`description`, pl.`description_short`, pl.`available_now`,
pl.`available_later`, pl.`link_rewrite`, pl.`meta_description`, pl.`meta_keywords`, pl.`meta_title`, pl.`name`, MAX(image_shop.`id_image`) id_image,
il.`legend`, m.`name` AS manufacturer_name, cl.`name` AS category_default,
DATEDIFF(product_shop.`date_add`, DATE_SUB(NOW(),
INTERVAL '.(Validate::isUnsignedInt(Configuration::get('PS_NB_DAYS_NEW_PRODUCT')) ? Configuration::get('PS_NB_DAYS_NEW_PRODUCT') : 20).'
DAY)) > 0 AS new, product_shop.price AS orderprice, fp.`id_feature_value`
FROM `'._DB_PREFIX_.'category_product` cp
LEFT JOIN `'._DB_PREFIX_.'product` p
ON p.`id_product` = cp.`id_product`
'.Shop::addSqlAssociation('product', 'p').'
LEFT JOIN `'._DB_PREFIX_.'product_attribute` pa
ON (p.`id_product` = pa.`id_product`)
'.Shop::addSqlAssociation('product_attribute', 'pa', false, 'product_attribute_shop.`default_on` = 1').'
'.Product::sqlStock('p', 'product_attribute_shop', false, $context->shop).'
LEFT JOIN `'._DB_PREFIX_.'category_lang` cl
ON (product_shop.`id_category_default` = cl.`id_category`
AND cl.`id_lang` = '.(int)$id_lang.Shop::addSqlRestrictionOnLang('cl').')
LEFT JOIN `'._DB_PREFIX_.'product_lang` pl
ON (p.`id_product` = pl.`id_product`
AND pl.`id_lang` = '.(int)$id_lang.Shop::addSqlRestrictionOnLang('pl').')
LEFT JOIN `'._DB_PREFIX_.'image` i
ON (i.`id_product` = p.`id_product`)'.
Shop::addSqlAssociation('image', 'i', false, 'image_shop.cover=1').'
LEFT JOIN `'._DB_PREFIX_.'image_lang` il
ON (image_shop.`id_image` = il.`id_image`
AND il.`id_lang` = '.(int)$id_lang.')
LEFT JOIN `'._DB_PREFIX_.'manufacturer` m
ON m.`id_manufacturer` = p.`id_manufacturer`
LEFT JOIN `'._DB_PREFIX_.'ps_feature_product` fp
ON p.`id_product` = fp.`id_product`
WHERE product_shop.`id_shop` = '.(int)$context->shop->id.'
AND cp.`id_category` = '.(int)$this->id
.($active ? ' AND product_shop.`active` = 1' : '')
.($front ? ' AND product_shop.`visibility` IN ("both", "catalog")' : '')
.($id_supplier ? ' AND p.id_supplier = '.(int)$id_supplier : '')
.' GROUP BY product_shop.id_product';
The table ps_feature_product looks like this:
CREATE TABLE IF NOT EXISTS `ps_feature_product` (
`id_feature` int(10) unsigned NOT NULL,
`id_product` int(10) unsigned NOT NULL,
`id_feature_value` int(10) unsigned NOT NULL,
PRIMARY KEY (`id_feature`,`id_product`,`id_feature_value`),
KEY `id_feature_value` (`id_feature_value`),
KEY `id_product` (`id_product`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
The table contains lots of different product features, but I need the features with id_feature_value 4 till 13 and that ID can be used as sort order as well.
So far no problem, a simple WHERE clause does the trick:
WHERE fp.`id_feature_value` BETWEEN 4 AND 13
And the ORDER clause is also straight forward:
ORDER BY fp.`id_feature_value` ASC
But now the tricky bit.
Products for which no id_feature_value in the range 4-13 is set, should be merged in as well, but they should be sorted to the end of the list.
And it's this last bit of the query that I cannot wrap my head around.
How do I select features within a range and at the same time select features NOT within that range and add a sort order.

Doesn't this include them?
WHERE fp.`id_feature_value` BETWEEN 4 AND 13
OR fp.`id_feature_value` IS NULL
EDIT: sorry, i misunderstood what you meant by "for which no ... is set".
I think you want to add something to your SELECT list upon which you can sort, like
IF(fp.`id_feature_value` BETWEEN 4 AND 13, 0, 1) AS my_sort
and then include my_sort ASC in your sort.
simplified example:
mysql> create table so1 (n integer);
Query OK, 0 rows affected (0.11 sec)
mysql> insert into so1 values (1),(2),(3),(4),(5),(NULL);
Query OK, 6 rows affected (0.02 sec)
Records: 6 Duplicates: 0 Warnings: 0
mysql> select n, if(n between 2 and 4, 0, 1) AS s from so1 order by s ASC, n;
+------+---+
| n | s |
+------+---+
| 2 | 0 |
| 3 | 0 |
| 4 | 0 |
| NULL | 1 |
| 1 | 1 |
| 5 | 1 |
+------+---+
6 rows in set (0.00 sec)

Here is what I was talking about in my comment:
It will automatically give you the results you seek: 4-13 first, then NON 4-13 glommed on at the end of the listing
SELECT
<<query details>>
WHERE fp.`id_feature_value` BETWEEN 4 AND 13
ORDER BY fp.`id_feature_value` ASC
UNION
SELECT
<<query details>>
WHERE fp.`id_feature_value` NOT BETWEEN 4 AND 13
ORDER BY fp.`id_feature_value` ASC

Related

getting data from multiple tables and applying arithmatic operation on the result

I want to fetch data from two table and apply arithmetic operation on the column.
This is wha I tried :
String sql = "SELECT SUM(S.san_recover-C.amount) as total
FROM sanction S
LEFT JOIN collection C ON S.client_id = C.client_id
WHERE S.client_id=?";
This code is working only when there is value in both tables, but if there is no value in one of two tables there is no result.
SELECT SUM(S.san_recover - C.amount) as total
FROM sanction S
LEFT JOIN collection C ON S.client_id = C.client_id
WHERE S.client_id = ?
The problem with your query lies in the SUM() function. When the left join does not bring back records, then c.amount is NULL. When substracting NULL from something, you get a NULL result, which then propagates across the computation, and you end up with a NULL result for the SUM().
You probably want COALESCE(), like so:
SELECT SUM(S.san_recover - COALESCE(C.amount, 0)) as total
FROM sanction S
LEFT JOIN collection C ON S.client_id = C.client_id
WHERE S.client_id = ?
Where there is a possibility that a client may exist in one table but no another a full join would be appropriate but since mysql does not have such a thing then a union in a sub query will do
drop table if exists sanctions,collections;
create table sanctions(client_id int, amount int);
create table collections(client_id int, amount int);
insert into sanctions values
(1,10),(1,10),(2,10);
insert into collections values
(1,5),(3,10);
Select sum(Samount - camount)
From
(Select sum(amount) Samount, 0 as camount from sanctions where client_id =3
Union all
Select 0,sum(amount) as camount from collections where client_id =3
) s
;
+------------------------+
| sum(Samount - camount) |
+------------------------+
| -10 |
+------------------------+
1 row in set (0.00 sec)
If you want to do this for all clients
Select client_id,sum(Samount - camount) net
From
(Select client_id,sum(amount) Samount, 0 as camount from sanctions group by client_id
Union all
Select client_id,0,sum(amount) as camount from collections group by client_id
) s
group by client_id
;
+-----------+------+
| client_id | net |
+-----------+------+
| 1 | 15 |
| 2 | 10 |
| 3 | -10 |
+-----------+------+
3 rows in set (0.00 sec)

Find two closest elements from one table to other element from another table

I have two tables:
DROP TABLE IF EXISTS `left_table`;
CREATE TABLE `left_table` (
`l_id` INT(11) NOT NULL AUTO_INCREMENT,
`l_curr_time` INT(11) NOT NULL,
PRIMARY KEY(l_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
DROP TABLE IF EXISTS `right_table`;
CREATE TABLE `right_table` (
`r_id` INT(11) NOT NULL AUTO_INCREMENT,
`r_curr_time` INT(11) NOT NULL,
PRIMARY KEY(r_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
INSERT INTO left_table(l_curr_time) VALUES
(3),(4),(6),(10),(13);
INSERT INTO right_table(r_curr_time) VALUES
(1),(5),(7),(8),(11),(12);
I want to map (if exists) two closest r_curr_time from right_table to each l_curr_time from left_table such that r_curr_time must be greater or equal to l_curr_time.
The expected result for given values should be:
+------+-------------+-------------+
| l_id | l_curr_time | r_curr_time |
+------+-------------+-------------+
| 1 | 3 | 5 |
| 1 | 3 | 7 |
| 2 | 4 | 5 |
| 2 | 4 | 7 |
| 3 | 6 | 7 |
| 3 | 6 | 8 |
| 4 | 10 | 11 |
| 4 | 10 | 12 |
+------+-------------+-------------+
I have following solution which works for one closest value. But I do not like it very much because it silently rely on fact that GROUP BY will remain the first occurrence from group:
SELECT l_id, l_curr_time, r_curr_time, time_diff FROM
(
SELECT *, ABS(r_curr_time - l_curr_time) AS time_diff
FROM left_table
JOIN right_table ON 1=1
WHERE r_curr_time >= l_curr_time
ORDER BY l_id ASC, time_diff ASC
) t
GROUP BY l_id;
The output is following:
+------+-------------+-------------+-----------+
| l_id | l_curr_time | r_curr_time | time_diff |
+------+-------------+-------------+-----------+
| 1 | 3 | 5 | 2 |
| 2 | 4 | 5 | 1 |
| 3 | 6 | 7 | 1 |
| 4 | 10 | 11 | 1 |
+------+-------------+-------------+-----------+
4 rows in set (0.00 sec)
As you can see I am doing JOIN ON 1=1 is this OK also for large data (e.g. if both left_table and right_table has 10000 rows then Cartesian product will be 10^8 long)? Despite this lack I thing JOIN ON 1=1 is the only possible solution because first I need to create all possible combinations from existing tables and then pick up the ones which satisfies the condition, but if I'm wrong please correct me. Thanks.
This question is not trivial. In SQL Server or postgrsql it would be very easy because of the row_number() over x statement. This is not present in mysql. In mysql you have to deal with variables and chained select statements.
To solve this problem you have to combine multiple concepts. I will try to explain them one after the other to came to a solution that fits your question.
Lets start easy: How to build a table that contains the information of left_table and right_table?
Use a join. In this particular problem a left join and as the join condition we set that l_curr_time has to be smaller than r_curr_time. To make the rest easier we order this table by l_curr_time and r_curr_time. The statement is like the following:
SELECT l_id, l_curr_time, r_curr_time
FROM left_table l
LEFT JOIN right_table r ON l.l_curr_time<r.r_curr_time
ORDER BY l.l_curr_time, r.r_curr_time;
Now we have a table that is ordered and contains the information we want... but too many of them ;) Because the table is ordered it would be amazing if mysql could select only the two first occurent rows for each value in l_curr_time. This is not possible. We have to do it by ourselfs
mid part: How to number rows?
Use a variable! If you want to number a table you can use a mysql variable. There are two things to do: First of all we have to declare and define the variable. Second we have to increment this variable. Let's say we have a table with names and we want to know the position of all names when we order them by name:
SELECT name, #num:=#num+1 /* increment */
FROM table t, (SELECT #num:=0) as c
ORDER BY name ASC;
Hard part: How to number subset of rows depending of the value of one field?
Use variables to count (take a look above) and a variable for state pattern. We use the same principe like above but now we take a variable and save the value of the field we want depend on. If the value changes we reset the counter variable to zero. Again: This second variable have to be declared and defined. New Part: resetting a different variable depending on the content of the state variable:
SELECT
l_id,
l_curr_time,
r_curr_time,
#num := IF( /* (re)set num (the counter)... */
#l_curr_time = l_curr_time,
#num:= #num + 1, /* increment if the variable equals the actual l_curr_time field value */
1 /* reset to 1 if the values are not equal */
) as row_num,
#l_curr_time:=l_curr_time as lct /* state variable that holds the l_curr_time value */
FROM ( /* table from Step 1 of the explanation */
SELECT l_id, l_curr_time, r_curr_time
FROM left_table l
LEFT JOIN right_table r ON l.l_curr_time<r.r_curr_time
ORDER BY l.l_curr_time, r.r_curr_time
) as joinedTable
Now we have a table that holds all combinations we want (but too many) and all rows are numbered depending on the value of the l_curr_time field. In other words: Each subset is numbered from 1 to the amount of matching r_curr_time values that are greather or equal than l_curr_time.
Again the easy part: select all the values we want and depending on the row number
This part is easy. because the table we created in 3. is ordered and numbered we can filter by the number (it has to be smaller or equal to 2). Furthermore we select only the columns we're interessted in:
SELECT l_id, l_curr_time, r_curr_time, row_num
FROM ( /* table from step 3. */
SELECT
l_id,
l_curr_time,
r_curr_time,
#num := IF(
#l_curr_time = l_curr_time,
#num:= #num + 1,
1
) as row_num,
#l_curr_time:=l_curr_time as lct
FROM (
SELECT l_id, l_curr_time, r_curr_time
FROM left_table l
LEFT JOIN right_table r ON l.l_curr_time<r.r_curr_time
ORDER BY l.l_curr_time, r.r_curr_time
) as joinedTable
) as numberedJoinedTable,(
SELECT #l_curr_time:='',#num:=0 /* define the state variable and the number variable */
) as counterTable
HAVING row_num<=2; /* the number has to be smaller or equal to 2 */
That's it. This statement returns exactly what you want. You can see this statement in action in this sqlfiddle.
JoshuaK has the right idea. I just think it could be expressed a little more succinctly...
How about:
SELECT n.l_id
, n.l_curr_time
, n.r_curr_time
FROM
( SELECT a.*
, CASE WHEN #prev = l_id THEN #i:=#i+1 ELSE #i:=1 END i
, #prev := l_id prev
FROM
( SELECT l.*
, r.r_curr_time
FROM left_table l
JOIN right_table r
ON r.r_curr_time >= l.l_curr_time
) a
JOIN
( SELECT #prev := null,#i:=0) vars
ORDER
BY l_id,r_curr_time
) n
WHERE i<=2;

Compare data using the year in a query

I have a data and they are recorded by each year, I am trying to compare two years( the past year and the current year) data within one mysql query
Below are my tables
Cost Items
| cid | items |
| 1 | A |
| 2 | B |
Cost
| cid | amount | year |
| 1 | 10 | 1 |
| 1 | 20 | 2 |
| 1 | 30 | 1 |
This is the result I am expecting when i want to compare the year 1 and year 2. Year 1 is the past year and year 2 is the current year
Results
items | pastCost | currentCost |
A | 10 | 20 |
A | 30 | 0 |
However the below query is what i used by gives a strange answer.
SELECT
IFNULL(ps.`amount`, '0') as pastCost
IFNULL(cs.`amount`, '0') as currentCost
FROM
`Cost Items` b
LEFT JOIN
`Cost` ps
ON
b.cID=ps.cID
AND
ps.Year = 1
LEFT JOIN
`Cost` cu
ON
b.cID=cu.cID
AND
cu.Year =2
This is the result i get from my query
items | pastCost | currentCost |
A | 10 | 20 |
A | 30 | 20 |
Please what am i doing wrong? Thanks for helping.
I'm missing something about your query; the SQL text shown can't produce that result.
There is no source for the items column in the SELECT list, and there is no table aliased as cs. (Looks like the expression in the SELECT list would need to be cu.amount
Aside from that, the results being returned look exactly like what we'd expect. Each row returned from year=2 is being matched with each row returned from year=1. If there were three rows for year=1 and two rows for year=2, we'd get six rows back... each row for year=1 "matched" with each row for year=2.
If (cid, year) tuple was UNIQUE in Cost, then this query would return a result similar to what you expect.
SELECT b.items
, IFNULL(ps.amount, '0') AS pastCost
, IFNULL(cu.amount, '0') AS currentCost
FROM `Cost Items` b
LEFT
JOIN `Cost` ps
ON ps.cid = b.cid
AND ps.Year = 1
LEFT
JOIN `Cost` cu
ON cu.cid = b.cid
AND cu.Year = 2
Since (cid, year) is not unique, you need some additional column to "match" a single row for year=1 with a single row for year=2.
Without some other column in the table, we could use an inline view to generate a value. I can illustrate how we can make MySQL return a resultset like the one you show, one way that could be done, but I don't think this is really the solution to whatever problem you are trying to solve:
SELECT b.items
, IFNULL(MAX(IF(a.year=1,a.amount,NULL)),0) AS pastCost
, IFNULL(MAX(IF(a.year=2,a.amount,NULL)),0) AS currentCost
FROM `Cost Items` b
LEFT
JOIN ( SELECT #rn := IF(c.cid=#p_cid AND c.year=#p_year,#rn+1,1) AS `rn`
, #p_cid := c.cid AS `cid`
, #p_year := c.year AS `year`
, c.amount
FROM (SELECT #p_cid := NULL, #p_year := NULL, #rn := 0) i
JOIN `Cost` c
ON c.year IN (1,2)
ORDER BY c.cid, c.year, c.amount
) a
ON a.cid = b.cid
GROUP
BY b.cid
, a.rn
A query something like that would return a resultset that looks like the one you are expecting. But again, I strongly suspect that this is not really the resultset you are really looking for.
EDIT
OP leaves comment with vaguely nebulous report of observed behavior: "the above solution doesnt work"
Well then, let's check it out... create a SQL Fiddle with some tables so we can test the query...
SQL Fiddle here http://sqlfiddle.com/#!9/e3d7e/1
create table `Cost Items` (cid int unsigned, items varchar(5));
insert into `Cost Items` (cid, items) values (1,'A'),(2,'B');
create table `Cost` (cid int unsigned, amount int, year int);
insert into `Cost` (cid, amount, year) VALUES (1,10,1),(1,20,2),(1,30,1);
And when we run the query, we get a syntax error. There's closing paren missing in the expressions in the SELECT list, easy enough to fix.
SELECT b.items
, IFNULL(MAX(IF(a.year=1,a.amount,NULL)),0) AS pastCost
, IFNULL(MAX(IF(a.year=2,a.amount,NULL)),0) AS currentCost
FROM `Cost Items` b
LEFT
JOIN ( SELECT #rn := IF(c.cid=#p_cid AND c.year=#p_year,#rn+1,1) AS `rn`
, #p_cid := c.cid AS `cid`
, #p_year := c.year AS `year`
, c.amount
FROM (SELECT #p_cid := NULL, #p_year := NULL, #rn := 0) i
JOIN `Cost` c
ON c.year IN (1,2)
ORDER BY c.cid, c.year, c.amount
) a
ON a.cid = b.cid
GROUP
BY b.cid
, a.rn
Returns:
items pastCost currentCost
------ -------- -----------
A 10 20
A 30 0
B 0 0

Query with multiple left joins - points column value is incorrect

I have the following database structure, and I am trying to run a single query that will show classrooms and how many students are part of the classroom, and how many rewards a classroom has allocated out, as well as how many points allocated to a single classroom (based on the classroom_id column).
Using the query at the very bottom I am trying to collect the 'totalPoints' that a classroom has assigned - based on counting the points column in the classroom_redeemed_codes table and return this as a single integer.
For some reason the values are incorrect for the totalPoints - I am doing something wrong but not sure what...
-- UPDATE --
Here is the sqlfiddle:-
http://sqlfiddle.com/#!2/a9f45
My Structure:
CREATE TABLE `organisation_classrooms` (
`classroom_id` int(11) NOT NULL AUTO_INCREMENT,
`title` varchar(255) NOT NULL,
`active` tinyint(1) NOT NULL,
`organisation_id` int(11) NOT NULL,
`period` int(1) DEFAULT '0',
`classroom_bg` int(2) DEFAULT '3',
`sortby` varchar(6) NOT NULL DEFAULT 'points',
`sound` int(1) DEFAULT '0',
PRIMARY KEY (`classroom_id`)
);
CREATE TABLE organisation_classrooms_myusers (
`classroom_id` int(11) NOT NULL,
`user_id` bigint(11) unsigned NOT NULL,
);
CREATE TABLE `classroom_redeemed_codes` (
`redeemed_code_id` int(11) NOT NULL AUTO_INCREMENT,
`myuser_id` bigint(11) unsigned NOT NULL DEFAULT '0',
`ssuser_id` bigint(11) NOT NULL DEFAULT '0',
`classroom_id` int(11) NOT NULL,
`order_product_id` int(11) NOT NULL DEFAULT '0',
`order_product_images_id` int(11) NOT NULL DEFAULT '0',
`date_redeemed` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`points` int(11) NOT NULL,
`type` int(1) NOT NULL DEFAULT '0',
`notified` int(1) NOT NULL DEFAULT '0',
`inactive` tinyint(3) NOT NULL,
PRIMARY KEY (`redeemed_code_id`),
);
SELECT
t.classroom_id,
title,
COALESCE (
COUNT(DISTINCT r.redeemed_code_id),
0
) AS totalRewards,
COALESCE (
COUNT(DISTINCT ocm.user_id),
0
) AS totalStudents,
COALESCE (sum(r.points), 0) AS totalPoints
FROM
`organisation_classrooms` `t`
LEFT OUTER JOIN classroom_redeemed_codes r ON (
r.classroom_id = t.classroom_id
AND r.inactive = 0
AND (
r.date_redeemed >= 1393286400
OR r.date_redeemed = 0
)
)
LEFT OUTER JOIN organisation_classrooms_myusers ocm ON (
ocm.classroom_id = t.classroom_id
)
WHERE
t.organisation_id =37383
GROUP BY title
ORDER BY t.classroom_id ASC
LIMIT 10
-- EDIT --
OOPS! I hate SQL sometimes... I have made a big mistake, I am trying to count the number of STUDENTS in the classroom_redeemed_codes rather than the organisation_classrooms_myuser table. I'm really sorry I should have picked that up sooner?!
classroom_id | totalUniqueStudents
16 1
17 2
46 1
51 1
52 1
There are 7 rows in the classroom_redeemed_codes table but as classroom_id 46 has two rows although with the same myuser_id (this is the student id) this should appear as one unique student.
Does this make sense? Essentially trying to grab the number of unique students in the classroom_redeemed_codes tables based on the myuser_id column.
e.g a classroom id 46 could have 100 rows in the classroom_redeemed_codes tables, but if it is the same myuser_id for each this should show the totalUniqueStudents count as 1 and not 100.
Let me know if this isn't clear....
-- update --
I have the following query which seems to work borrowed from a user below which seems to work... (my head hurts) i'll accept the answer again. Sorry for the confusion - I think I was just over thinking this somewhat
select crc.classroom_id,
COUNT(DISTINCT crc.myuser_id) AS users,
COUNT( DISTINCT crc.redeemed_code_id ) AS classRewards,
SUM( crc.points ) as classPoints, t.title
from classroom_redeemed_codes crc
JOIN organisation_classrooms t
ON crc.classroom_id = t.classroom_id
AND t.organisation_id = 37383
where crc.inactive = 0
AND ( crc.date_redeemed >= 1393286400
OR crc.date_redeemed = 0 )
group by crc.classroom_id
I ran by first doing a pre-query aggregate of your points per specific class, then used left-join to it. I am getting more rows in the result set than your sample expected, but don't have MySQL to test/confirm directly. Howeverhere is a SQLFiddle of your query By doing your query with sum of points, and having a Cartesian result when applying the users table, it is probably the basis of duplicating the points. By pre-querying on the redeem codes itself, you just grab that value, then join to users.
SELECT
t.classroom_id,
title,
COALESCE ( r.classRewards, 0 ) AS totalRewards,
COALESCE ( r.classPoints, 0) AS totalPoints,
COALESCE ( r.uniqStudents, 0 ) as totalUniqRedeemStudents,
COALESCE ( COUNT(DISTINCT ocm.user_id), 0 ) AS totalStudents
FROM
organisation_classrooms t
LEFT JOIN ( select crc.classroom_id,
COUNT( DISTINCT crc.redeemed_code_id ) AS classRewards,
COUNT( DISTINCT crc.myuser_id ) as uniqStudents,
SUM( crc.points ) as classPoints
from classroom_redeemed_codes crc
JOIN organisation_classrooms t
ON crc.classroom_id = t.classroom_id
AND t.organisation_id = 37383
where crc.inactive = 0
AND ( crc.date_redeemed >= 1393286400
OR crc.date_redeemed = 0 )
group by crc.classroom_id ) r
ON t.classroom_id = r.classroom_id
LEFT OUTER JOIN organisation_classrooms_myusers ocm
ON t.classroom_id = ocm.classroom_id
WHERE
t.organisation_id = 37383
GROUP BY
title
ORDER BY
t.classroom_id ASC
LIMIT 10
You need sum(r.points) and a subquery in the left outer join see below
SELECT
t.classroom_id,
title,
COALESCE (
COUNT(DISTINCT r.redeemed_code_id),
0
) AS totalRewards,
COALESCE(sum(r.points),0) AS totalPoints
,COALESCE(sum(T1.cnt),0) as totalStudents
FROM
`organisation_classrooms` `t`
left outer join (select classroom_id, count(user_id) cnt
from organisation_classrooms_myusers
group by classroom_id) T1 on (T1.classroom_id=t.classroom_id)
LEFT OUTER JOIN classroom_redeemed_codes r ON (
r.classroom_id = t.classroom_id
AND r.inactive = 0
AND (
r.date_redeemed >= 1393286400
OR r.date_redeemed = 0
)
)
WHERE
t.organisation_id =37383
GROUP BY title
ORDER BY t.classroom_id ASC
LIMIT 10
I simplified your query; there is no need to use COALLESCE together with COUNT() because COUNT() never returns NULL. For SUM() I prefer to use IFNULL() because it is shorter and more readable. The results displayed below contain only the data for classroom_id #16, #17 and #46 for easier comparison with the example provided in the question. The actual result sets are bigger and contain all the classroom_ids present in the tables. However, their presence is not needed to understand how and why it works.
SELECT
t.classroom_id,
t.title,
COUNT(DISTINCT r.redeemed_code_id) AS totalRewards,
COUNT(DISTINCT ocm.user_id) AS totalStudents,
IFNULL(SUM(r.points), 0) AS totalPoints
FROM `organisation_classrooms` t
LEFT JOIN `classroom_redeemed_codes` r
ON r.classroom_id = t.classroom_id
AND r.inactive = 0
AND (r.date_redeemed >= 1393286400 OR r.date_redeemed = 0)
LEFT JOIN `organisation_classrooms_myusers` ocm
ON ocm.classroom_id = t.classroom_id
WHERE t.organisation_id = 37383
GROUP BY t.classroom_id
ORDER BY t.classroom_id ASC
Let's try to split it in pieces and put them together after that. First, let's see what users are selected:
Query #1
SELECT
t.classroom_id,
t.title,
ocm.user_id
FROM `organisation_classrooms` t
LEFT JOIN `organisation_classrooms_myusers` ocm
ON ocm.classroom_id = t.classroom_id
WHERE t.organisation_id = 37383
ORDER BY t.classroom_id ASC
I removed the classroom_redeemed_codes table and it fields, removed GROUP BY and replaced the aggregate function COUNT(ocm.user_id) with ocm.user_id to see what users are selected.
The result show us this part of the query is correct:
classroom_id | title | user_id
-------------+-------+--------
16 | BLUE | 2
16 | BLUE | 1
17 | GREEN | 508835
17 | GREEN | 508826
46 | PINK | NULL
There are 2 users in classroom #16, another 2 in #7 and none in class #46.
Putting back the GROUP BY clause will make it return the correct values (2, 2, 0) in the totalStudents column.
Let's check now the relationship with table classroom_redeemed_codes:
Query #2
SELECT
t.classroom_id,
t.title,
r.redeemed_code_id, r.points
FROM `organisation_classrooms` t
LEFT JOIN `classroom_redeemed_codes` r
ON r.classroom_id = t.classroom_id
AND r.inactive = 0
AND (r.date_redeemed >= 1393286400 OR r.date_redeemed = 0)
WHERE t.organisation_id = 37383
ORDER BY t.classroom_id ASC
The result is:
classroom_id | title | redeemed_code_id | points
-------------+-------+------------------+-------
16 | BLUE | 7 | 50
17 | GREEN | 8 | 25
17 | GREEN | 9 | 75
46 | PINK | 5 | 250
46 | PINK | 6 | 100
Again, grouping by classroom_id will produce (1, 2, 2) in column totalRewards and (50, 100, 350) in column totalPoints which is correct.
The trouble starts when you want to combine these into a single query. No matter what kind of join you use, for the provided input you will get (2*1, 2*2, 1*2) rows for classroom_id having the values 16, 17 and 46 (in this order). The values I multiplied in parenthesis are the number of rows for each classroom_id in the first and in the query result set above.
Combined
Let' try the query that selects the rows before grouping them:
SELECT
t.classroom_id,
t.title,
r.redeemed_code_id, ocm.user_id, r.points
FROM `organisation_classrooms` t
LEFT JOIN `classroom_redeemed_codes` r
ON r.classroom_id = t.classroom_id
AND r.inactive = 0
AND (r.date_redeemed >= 1393286400 OR r.date_redeemed = 0)
LEFT JOIN `organisation_classrooms_myusers` ocm
ON ocm.classroom_id = t.classroom_id
WHERE t.organisation_id = 37383
ORDER BY t.classroom_id ASC
It returns this result set:
classroom_id | title | redeemed_code_id | user_id | points
-------------+-------+------------------+---------+-------
16 | BLUE | 7 | 2 | 50
16 | BLUE | 7 | 1 | 50 <- *
-------------+-------+------------------+---------+-------
17 | GREEN | 8 | 508835 | 25
17 | GREEN | 8 | 508826 | 25 <- *
17 | GREEN | 9 | 508835 | 75
17 | GREEN | 9 | 508826 | 75 <- *
-------------+-------+------------------+---------+-------
46 | PINK | 5 | NULL | 250
46 | PINK | 6 | NULL | 100
I added horizontal rules to separate the rows that belongs to the same group when we add the GROUP BY clause. This is basically the way a SQL query with GROUP BY is executed, no matter the name of the actual software that implements it.
As you can see, for each classroom, it combines all the redeemed codes associated with the classroom with all the users associated with the classroom. If you add more users and redeemed codes for classrooms #16, #17 and #46 in your tables you will get a much larger result set.
The next step on the execution of a GROUP BY query is to produce a single row from each group you see above. There is no problem with columns classroom_id and title, they contain a single value in each group. For the columns redeemed_code_id and user_id your query counts distinct values and that works fine too. The problem is with the addition of points.
If you just SUM() them, you get a redeemed code added for each user_id in the group. If you use SUM(DISTINCT points) it is also wrong because it will ignore the duplicates even when they are different entries in table classroom_redeemed_codes.
What you want is to add points for DISTINCT redeemed_code_id. I marked on the above result set the rows you don't want.
This is not possible using this query because on calculation of the aggregate values each column is independent of the other. We need a query that selects the desired rows before grouping them.
An Idea
We can try to add the missing columns (with NULL values) to the two simple queries above, UNION ALL them then select from this and GROUP BY.
First, let's be sure it selects what we need:
SELECT
t.classroom_id,
t.title,
NULL AS redeemed_code_id, ocm.user_id, NULL AS points
FROM `organisation_classrooms` t
LEFT JOIN `organisation_classrooms_myusers` ocm
ON ocm.classroom_id = t.classroom_id
WHERE t.organisation_id = 37383
UNION ALL
SELECT
t.classroom_id,
t.title,
r.redeemed_code_id, NULL AS user_id, r.points
FROM `organisation_classrooms` t
LEFT JOIN `classroom_redeemed_codes` r
ON r.classroom_id = t.classroom_id
AND r.inactive = 0
AND (r.date_redeemed >= 1393286400 OR r.date_redeemed = 0)
WHERE t.organisation_id = 37383
ORDER BY classroom_id
Attention! The ORDER BY clause applies to the UNIONed result set. If you want to order the rows of each SELECT (it doesn't help because UNION doesn't keep the order) you need to enclose that query in parenthesis and put the ORDER BY clauses there.
The result set looks great:
classroom_id | title | redeemed_code_id | user_id | points
-------------+-------+------------------+---------+-------
16 | BLUE | NULL | 1 | NULL
16 | BLUE | NULL | 2 | NULL
16 | BLUE | 7 | NULL | 50
-------------+-------+------------------+---------+-------
17 | GREEN | 8 | NULL | 25
17 | GREEN | 9 | NULL | 75
17 | GREEN | NULL | 508826 | NULL
17 | GREEN | NULL | 508835 | NULL
-------------+-------+------------------+---------+-------
46 | PINK | 5 | NULL | 250
46 | PINK | 6 | NULL | 100
46 | PINK | NULL | NULL | NULL
Now we could put some parenthesis around the query above (strip ORDER BY) and use it in another query, grouping the data by classroom_id, counting the users and the redeemed codes and summing their points.
You will get a query that looks awful and, on your current database schema, crawls when your tables have several hundred rows. This is why I will not write it here.
Attention!
Its performance can be improved by adding the missing indexes to your tables, on the fields that appear in the ON, WHERE, ORDER BY and GROUP BY clauses of the query.
It will bring a significant improvement but I won't rely very much on that. For really big tables (hundreds of thousands of rows) it will still crawl.
Another Idea
We can also add GROUP BY on both Query #1 and Query #2 first and UNION ALL them after that:
SELECT
t.classroom_id,
t.title,
NULL AS totalRewards,
COUNT(DISTINCT ocm.user_id) AS totalStudents,
NULL AS totalPoints
FROM `organisation_classrooms` t
LEFT JOIN `organisation_classrooms_myusers` ocm
ON ocm.classroom_id = t.classroom_id
WHERE t.organisation_id = 37383
GROUP BY t.classroom_id
UNION ALL
SELECT
t.classroom_id,
t.title,
COUNT(DISTINCT redeemed_code_id) AS totalRewards,
NULL AS totalStudents,
SUM(points) AS totalPoints
FROM `organisation_classrooms` t
LEFT JOIN `classroom_redeemed_codes` r
ON r.classroom_id = t.classroom_id
AND r.inactive = 0
AND (r.date_redeemed >= 1393286400 OR r.date_redeemed = 0)
WHERE t.organisation_id = 37383
GROUP BY t.classroom_id
ORDER BY classroom_id, totalRewards
This produces a nice result set:
classroom_id | title | totalRewards | totalStudents | totalPoints
-------------+-------+--------------+---------------+-------------
16 | BLUE | NULL | 2 | NULL
16 | BLUE | 1 | NULL | 50
17 | GREEN | NULL | 2 | NULL
17 | GREEN | 2 | NULL | 100
46 | PINK | NULL | 0 | NULL
46 | PINK | 2 | NULL | 350
This query can be embedded in another query that groups by classroom_id and SUM()s the total columns above to get the final result. But again, the final query is big and ugly and it
doesn't run very fast for large tables. And again, this is the reason I don't write it here.
Conclusion
It can be done in a single query but it doesn't look good and it doesn't work well on large tables.
Regarding the performance, put EXPLAIN in front of your query then check the values in columns type, key and Extra of the result. See the documentation for explanation of the possible values of these columns, what to try to achieve and what to avoid.
Both queries I created on both ideas produce joins of type range or ALL and having Using filesort in column Extra (all these are slow). Using them as sub-queries in bigger queries will not improve the way they are execution, on the contrary.
I recommend you to run the individual SELECT queries from the last code example as two separate queries; they will return the odd and the even rows from the above result set. Then combine their results into the client code. It will run faster this way.

Refactor of SQL Statement

The following is working as expected. It shows the default CPM and CPC for the country if it is missing from the original table.
I am using a temporary table and I will like to know if it can be done without using temp table.
mysql> create temporary table country_list (country_name varchar(100));
Query OK, 0 rows affected (0.00 sec)
mysql> insert into country_list values ('IN'), ('SA');
Query OK, 2 rows affected (0.00 sec)
Records: 2 Duplicates: 0 Warnings: 0
select dt.country_name as country, dt.operator,
if (dt.cpm is null, (SELECT cpm FROM ox_bidding_configuration where country = '' and operator = ''), dt.cpm) as updated_cpm,
if (dt.cpc is null, (SELECT cpc FROM ox_bidding_configuration where country = '' and operator = ''), dt.cpc) as updated_cpc,
if (dt.cpa is null, (SELECT cpa FROM ox_bidding_configuration where country = '' and operator = ''), dt.cpa) as updated_cpa
from (
select a.country_name, b.operator, cpm, cpc, cpa from country_list as a left join ox_bidding_configuration as b on a.country_name = b.country group by b.country, b.operator, cpm, cpc, cpa) as dt;
+---------+----------+-------------+-------------+-------------+
| country | operator | updated_cpm | updated_cpc | updated_cpa |
+---------+----------+-------------+-------------+-------------+
| SA | NULL | 1.0000 | 1.0000 | 1.0000 |
| IN | | 2.0000 | 2.0000 | 2.0000 |
| IN | abcd | 11.0000 | 23.0000 | 4.0000 |
+---------+----------+-------------+-------------+-------------+
The country SA is not in the primary table. So I can not use self left joins. I will like to know if there is a better way.
For starters, from where I'm standing you're using the (temporary) country_list table as the PRIMARY table and match that against the ox_bidding_configuration where possible (left outer join); hence your result-set only contains the countries SA & IN.
Although I'm not 100% sure this syntax works on MYSQL, being used to TSQL I would write your query as :
SELECT dt.country_name as country,
dt.operator,
COALESCE(dt.cpm, def.cpm) as updated_cpm,
COALESCE(dt.cpc, def.cpc) as updated_cpd,
COALESCE(dt.cpa, def.cpa) as updated_cpa
FROM (SELECT DISTINCT a.country_name,
b.operator,
b.cpm,
b.cpc,
b.cpa
FROM country_list a
LEFT OUTER JOIN ox_bidding_configuration as b
ON b.country = a.country_name ) dt
LEFT OUTER JOIN ox_bidding_configuration def
ON def.country = ''
AND def.operator = ''
IMHO this is more clear, and probably it's slightly faster too as the default config only needs to be fetched once.
To get rid of the country_list table, you could convert it into an in-line query, but frankly I don't see the benefit of it.
Something along the lines of below :
SELECT dt.country_name as country,
dt.operator,
COALESCE(dt.cpm, def.cpm) as updated_cpm,
COALESCE(dt.cpc, def.cpc) as updated_cpd,
COALESCE(dt.cpa, def.cpa) as updated_cpa
FROM (SELECT DISTINCT a.country_name,
b.operator,
b.cpm,
b.cpc,
b.cpa
FROM (SELECT 'SA' AS country_name
UNION ALL
SELECT 'IN') a
LEFT OUTER JOIN ox_bidding_configuration as b
ON b.country = a.country_name ) dt
LEFT OUTER JOIN ox_bidding_configuration def
ON def.country = ''
AND def.operator = ''