MySQL multidimensional select from views - mysql

I would like to display data re-arranged year by year and one of the possible solution is using views and select from them. The data matrix is something like (of course it's a ficticious demo dataset):
USA 2005 22 156
CAN 2005 14 101
MEX 2005 5 32
USA 2006 24 160
CAN 2006 16 103
USA 2007 26 163
MEX 2007 8 35
The SQL code to create and populate the table is:
DROP TABLE IF EXISTS `tab1`;<br>
CREATE TABLE `tab1` ( <br>
`id1` int(4) unsigned NOT NULL AUTO_INCREMENT,
`iso3` char(3) NOT NULL,
`year` int(4) unsigned NOT NULL,
`aaa` int(10) DEFAULT NULL,
`bbb` int(10) DEFAULT NULL,
PRIMARY KEY (`id1`)
)
INSERT INTO `tab1` VALUES
('1', 'USA', '2005', '22', '156'),
('2', 'CAN', '2005', '14', '101'),
('3', 'MEX', '2005', '5', '32'),
('4', 'USA', '2006', '24', '160'),
('5', 'CAN', '2006', '16', '103'),
('6', 'USA', '2007', '26', '163'),
('7', 'MEX', '2007', '8', '35');
COMMIT;
And now I would like to obtain for parameter 'aaa' a 2D table like this:
country 2005 2006 2007
USA 22 24 26
CAN 14 16
MEX 5 8
However the following SQL code is omitting all the lines with missing data, be it one single value and I only get one line
USA 22 24 26
The SQL code is:
SELECT view2005.Country, view2005.2005, view2006.2006, view2007.2007
FROM view2005, view2006, view2007
WHERE view2005.country = view2006.country
AND view2005.country = view2007.country
Any idea how to do it including lines with missing data? Thanks in advance.

Use left joins, and a view (or table, or inner select like below) which has all distinct countries:
SELECT c.country, view2005.2005, view2006.2006, view2007.2007
FROM (SELECT DISTINCT country FROM tab1) as c
LEFT JOIN view2005 ON view2005.country = c.country
LEFT JOIN view2006 ON view2006.country = c.country
LEFT JOIN view2007 ON view2007.country = c.country
GROUP BY c.country
EDIT:
In a more general context, what you are asking here is to create a pivot of this table, which is a common problem that has common solutions. Here is a nice "How To": http://www.artfulsoftware.com/infotree/queries.php?&bw=1339#78

It's better to use JOIN than implicit JOIN with WHERE. Additional advanatge is that you can convert it to a LEFT JOIN so data for 2005 that don't have a 2006 related row (and are not matched) will still be shown.
Use Galz's solution or search as correctly advised for how to create PIVOT queries.
One such logic to create a pivot query would be:
SELECT iso3 AS Country
, SUM(IF(year=2005, aaa, 0)) AS 2005
, SUM(IF(year=2006, aaa, 0)) AS 2006
, SUM(IF(year=2007, aaa, 0)) AS 2007
FROM tab1 AS t
GROUP BY iso3
If there are years without any data, you will get NULL in that column.
You can use COALESCE() function if you want 0 to be shown and not NULL:
SELECT iso3 AS Country
, COALESCE( SUM( IF(year=2004, aaa, 0) ) , 0) AS "2004"
, COALESCE( SUM( IF(year=2005, aaa, 0) ) , 0) AS "2005"
, COALESCE( SUM( IF(year=2006, aaa, 0) ) , 0) AS "2006"
, COALESCE( SUM( IF(year=2007, aaa, 0) ) , 0) AS "2007"
FROM tab1 AS t
GROUP BY iso3

Thank you Galz for the link to pivots and thank you ypercube for the SQL. It worked after enclosing the years into quotes to make them CHAR.
I was further intrigued by the question what happens if I add a row with no values at all or a row out of range of the years so I have added
INSERT INTO `tab1` VALUES
('7', 'ATA', '2004', '', '')
The result was that I got a mix of NULL and INT zero values. This is not good because zero is a valid number and legitimate data. So I have modified the query to get exactly the result I need:
SELECT iso3 AS countryб
SUM( IF(year=2004, aaa, NULL) ) AS "2004",
SUM( IF(year=2005, aaa, NULL) ) AS "2005",
SUM( IF(year=2006, aaa, NULL) ) AS "2006",
SUM( IF(year=2007, aaa, NULL) ) AS "2007"
FROM tab1
GROUP BY iso3

Related

MySQL Select Data From two tables and generate column name dynamically

1: Product info (corewp_product) 2: Product Metadata (corewp_productmeta)
I want to select (assume the user is searching) by price, color, size e.t.c metadata depending on the search parameter and metadata available.
Eg. search might be
where (color=red and price between 100 and 500)
Since metadata is dynamically added I don't want to create a new column for each metadata. Some of the products are in group (eg. Sneaker might be in red,blue with various prices)
My Tables are like this:
corewp_product
id
idname
title
category
price
type
meta
1
A1
Sneakers
2
0
grouped
0
2
A2
Branded Shirts for sale
1
0
grouped
0
3
A3
Long Sleeve Shirts
1
0
grouped
0
corewp_productmeta
id
postid_field
group_id
meta_name
meta_value
1
A1
G1
color
red
2
A1
G1
size
EU40
3
A1
G1
price
28
4
A1
G2
color
black
5
A1
G2
size
EU41
6
A1
G2
price
30
7
A1
G3
color
red
8
A1
G3
size
E40
9
A1
G3
price
50
10
A2
G1
color
any
11
A2
G1
size
L
12
A2
G1
price
60
13
A3
G1
color
red
14
A3
G1
price
30
Problem:
Selecting products with color = red and price between 0 and 50 or with other metadata.
Approach 1- using join:
I have tried to do it this way
SELECT corewp_product.id, corewp_product.idname, corewp_product.title, P.amount, C.color FROM corewp_product JOIN ( SELECT `postid_field` as priceId, `meta_value` as amount, `group_id` as ggroup FROM `corewp_productmeta` WHERE (`meta_name` = 'price' AND `meta_value` BETWEEN 10 AND 50)) AS P JOIN (SELECT `postid_field` as colorId, `meta_value` as color, `group_id` as ggroup FROM `corewp_productmeta` WHERE (`meta_name` = 'color' AND `meta_value` = 'red') GROUP BY `group_id`,`postid_field`) AS C ON p.ggroup = C.ggroup WHERE corewp_product.idname = P.priceId AND corewp_product.idname = C.colorId
But the problem with the code above is what happen when a new meta data is added e.g: brand name
id
postid_field
group_id
meta_name
meta_value
15
A1
G1
brand
nike
and the new search has to include brand name color = red and brand = nike and price between 0 and 50, I will have to alter the query above which is something am looking to avoid.
Approach 2- using view:
SET #sql = NULL;
SELECT
GROUP_CONCAT(DISTINCT
CONCAT(
'MAX(IF(pm.meta_name = ''',
meta_name,
''', pm.meta_value, NULL)) AS ',
meta_name
)
) INTO #sql
FROM corewp_productmeta;
SET #sql = CONCAT('SELECT p.idname , p.title, ', #sql, ' FROM corewp_product p LEFT JOIN corewp_productmeta AS pm ON p.idname = pm.postid_field GROUP BY pm.group_id,p.idname,p.title');
SET #qrys = CONCAT('CREATE OR REPLACE VIEW meta_view AS ',#sql);
PREPARE stmt FROM #qrys;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
The approach works as expected but now I have to get all the data from the view table which also I want to avoid now an issue comes when a new meta data is added e.g. brand same issue repeat.
It will be great if I could be able to query like select... where brand=xx and color=aa then results would come with column brand and name if brand doesn't exist then brand column returned as null or result not found same with other dynamic values passed in a query
Is there any way you can help me guys? I will appriciate.
N.B: this query will also include pagination limit (0,10) once full system is deployed.
SQL FORMAT
CREATE TABLE `corewp_product` (
`id` int(11) NOT NULL PRIMARY KEY AUTO_INCREMENT,
`idname` varchar(20) NOT NULL,
`title` varchar(60) NOT NULL,
`category` int(11) NOT NULL,
`price` double(20,2) NOT NULL,
`type` varchar(20) NOT NULL,
`meta` int(11) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
INSERT INTO `corewp_product` (`id`, `idname`, `title`, `category`, `price`, `type`, `meta`) VALUES
(1, 'A1', 'Sneakers', 2, 0.00, 'grouped', 0),
(2, 'A2', 'Branded Shirts for sale', 1, 0.00, 'grouped', 0),
(3, 'A3', 'Long Sleeve Shirts', 1, 0.00, 'grouped', 0);
CREATE TABLE `corewp_productmeta` (
`id` int(11) NOT NULL PRIMARY KEY AUTO_INCREMENT,
`postid_field` varchar(5) NOT NULL,
`group_id` varchar(5) NOT NULL,
`meta_name` varchar(50) NOT NULL,
`meta_value` varchar(100) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
INSERT INTO `corewp_productmeta` (`id`, `postid_field`, `group_id`, `meta_name`, `meta_value`) VALUES
(1, 'A1', 'G1', 'color', 'red'),
(2, 'A1', 'G1', 'size', 'EU40'),
(3, 'A1', 'G1', 'price', '28'),
(4, 'A1', 'G2', 'size', 'EU41'),
(5, 'A1', 'G2', 'color', 'black'),
(6, 'A1', 'G2', 'price', '30'),
(7, 'A1', 'G3', 'color', 'red'),
(8, 'A1', 'G3', 'size', 'E40'),
(9, 'A1', 'G3', 'price', '50'),
(10, 'A2', 'G1', 'color', 'any'),
(11, 'A2', 'G1', 'size', 'L'),
(12, 'A2', 'G1', 'price', '60'),
(13, 'A3', 'G1', 'color', 'red'),
(14, 'A3', 'G1', 'price', '30');
WITH vars AS (
SELECT postid_field,title,
GROUP_CONCAT(IF(meta_name='color', meta_value, NULL) SEPARATOR '') color,
GROUP_CONCAT(IF(meta_name='size', meta_value, NULL) SEPARATOR '') size,
GROUP_CONCAT(IF(meta_name='price', meta_value, NULL) SEPARATOR '') price
FROM corewp_productmeta,corewp_product
WHERE postid_field = idname
GROUP BY group_id,postid_field,title
)
select * from vars
where price > 29 AND price < 59
Query Demo
The query uses the WITH clause to create a sub-query that joins the two tables and assigns the resulting table to a variable eg: vars.
After that, you can query from the variable like any normal table and apply your filters in the where clause and you can filter using the extended columns. eg: where price > 29 AND price < 59
Check the Query Demo on the link above.
where (color=red and price between 100 and 500)
SELECT pr.*
FROM corewp_product pr
JOIN corewp_productmeta pm ON pr.idname = pm.postid_field
WHERE (pm.meta_name = 'color' AND meta_value = 'red')
OR (pm.meta_name = 'price' AND meta_value + 0 BETWEEN 100 AND 500)
GROUP BY pr.id
HAVING COUNT(*) = 2;
DEMO
Some explanations.
We select the metadata rows which matches any of the criteria. Then we group by the product and count the amount of matched meta values for it. Finally we return only those products which have the matched amount equal to total one.
This query does not need in dynamic SQL. You must only put correct values into the conditions.
Pay attention to this: meta_value + 0 BETWEEN 100 AND 500. The addition + 0 performs implicit data casting to numeric (of course we can use explicit CAST(meta_value AS UNSIGNED)). This allows make numeric comparing context. Without this addition the comparing context will be string, and we may obtain incorrect output (for example, for "price between 50 and 100").

How to get summary data for every months in mysql

I want to count the number of items sold(item_count) every month for every item,
--
-- Table structure for table `sales`
--
CREATE TABLE `sales` (
`id` int(11) NOT NULL,
`item_id` int(11) NOT NULL,
`date` date NOT NULL,
`item_count` int(11) NOT NULL,
`amount` float NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
--
-- Dumping data for table `sales`
--
INSERT INTO `sales` (`id`, `item_id`, `date`, `item_count`, `amount`) VALUES
(1, 1, '2018-01-15', 11, 110),
(2, 2, '2018-01-21', 5, 1000),
(3, 1, '2018-02-02', 7, 700),
(4, 2, '2018-02-11', 3, 3000);
I have tried this SQL, but it's not showing the data correctly.
SELECT `sales`.`item_id`,
(CASE WHEN MONTH(sales.date)=1 THEN sum(sales.item_count) ELSE NULL END) as JAN,
(case when MONTH(sales.date)=2 THEN sum(sales.item_count) ELSE NULL END) as FEB
FROM sales WHERE 1
GROUP BY sales.item_id
ORDER BY sales.item_id
This is my expected result,
item_id JAN FEB
1 11 7
2 5 3
I am getting this,
item_id JAN FEB
1 18 NULL
2 8 NULL
Here is an immediate fix to your query. You need to sum over a CASE expression, rather than the other way around.
SELECT
s.item_id,
SUM(CASE WHEN MONTH(s.date) = 1 THEN s.item_count END) AS JAN,
SUM(CASE WHEN MONTH(s.date) = 2 THEN s.item_count END) AS FEB
FROM sales s
GROUP BY
s.item_id
ORDER BY
s.item_id;
But the potential problem with this query is that in order to support more months, you need to add more columns. Also, if you want to cover mulitple years, then this approach also might not scale. Assuming you only have a few items, here is another way to do this:
SELECT
DATE_FORMAT(date, '%Y-%m') AS ym,
SUM(CASE WHEN item_id = 1 THEN item_count END) AS item1_total,
SUM(CASE WHEN item_id = 2 THEN item_count END) AS item2_total
FROM sales
GROUP BY
DATE_FORMAT(date, '%Y-%m');
This would generate output looking something like:
ym item1_total item2_total
2018-01 11 5
2018-02 7 3
Which version you use depends on how many months your report requires versus how many items might appear in your data.

Creating a weighted sum of values from different tables

I'm trying to create a list of students whose behaviour is statistically worst across each of our school's year groups.
We have a table named students.
We then have behavioural flags and alerts, plus sanctions.
However, different categories of flag/alert/sanction are deemed more serious than others. These are stored with labels in their respective _categories table, e.g. flag_categories and sanction_categories. The flag table will then have a column called Category_ID (alerts is a bit different as it's just a Type field with 'A', 'C', 'P' and 'S' values).
If I want to look at data which shows our highest-flagged students in a specific year group, I'd run this query:
SELECT
CONCAT(stu.Firstname, " ", stu.Surname) AS `Student`,
COUNT(f.ID) AS `Flags`
FROM `students` stu
LEFT JOIN `flags` f ON f.Student_ID = stu.id
WHERE stu.Year_Group = 9
GROUP BY stu.id
ORDER BY `Flags` DESC
LIMIT 0, 20
If I wanted to show our students with the most Crisis alerts in a specific year group, I'd run this query:
SELECT
CONCAT(stu.Firstname, " ", stu.Surname) AS `Student`,
COUNT(f.ID) AS `Flags`
FROM `students` stu
LEFT JOIN `flags` f ON f.Student_ID = stu.id
WHERE stu.Year_Group = 9
AND f.Category_ID = 10
GROUP BY stu.id
ORDER BY `Flags` DESC
LIMIT 0, 20
If I want to find how many Late or Mobile flags a student has, and perhaps add these together (with weightings), I can run the following query:
SELECT
CONCAT(stu.Firstname, " ", stu.Surname) AS `Student`,
SUM(CASE WHEN f.Category_ID = 10 THEN 1 ELSE 0 END) AS `Late Flags`,
SUM(CASE WHEN f.Category_ID = 12 THEN 2 ELSE 0 END) AS `Mobile Flags`,
## not sure about this line below... is there a nicer way of doing it? `Late Flags` isn't recognised as a field apparently
## so I can't just do ( `Late Flags` + `Mobile Flags` )
(SUM(CASE WHEN f.Category_ID = 10 THEN 1 ELSE 0 END) + SUM(CASE WHEN f.Category_ID = 12 THEN 2 ELSE 0 END)) AS `Points`
FROM `flags` f
LEFT JOIN `students` stu ON f.Student_ID = stu.id
WHERE stu.Year_Group = 9
GROUP BY stu.id
ORDER BY `Points` DESC
LIMIT 0, 20
What I don't understand is how I would do this across myriad tables. I need to be able to weight:
Late (flags, Category_ID = 10), Absconded (flags, Category_ID = 15) and Community flags (flags, Category_ID = 13) plus Safeguarding alerts (alerts, Type = 'S') are all worth 1 point
Behavioural flags (flags, Category_ID IN (1, 7, 8)) are worth 2 points
Process alerts (alerts, Type = 'P') and detention sanctions (sanctions, Category_ID = 1) are worth 3 points
So on and so forth. That's far from an exhaustive list but I've included enough variables to help me get my head round a multi-table weighted sum.
The outcome I'm looking for is just 2 columns - student's name and weighted points.
So, according to the bullet points above, if a student has received 2 Late flags (1 point each) and 1 Process alert (3 points), the output should just say Joe Bloggs and 5.
Can anyone help me to understand how I can get these weighted values from different tables into one SUM'd output for each student?
[edit] SQLFiddle here: http://sqlfiddle.com/#!9/449218/1/0
Note, I am not doing this for the bounty. Please give to someone else.
This could be done with a few LEFT JOINs of derived tables. Note you did not supply the sanctions table. But the below would appear to be well illustrative. So I created a temp table. It would seem to allow for maximum flexibility without overcomplicating a larger left join notion that might be hard to debug. Afterall, you said your real querying will be much more complicated than this. As such, build out the temp table structure more.
This loads a tmp table up with default 0's for the students in the "passed by parameter Student Year" to a stored procedure. Two updates are performed. Then selects for a result set.
Schema / Load:
create schema s38741386; -- create a test database
use s38741386;
CREATE TABLE `students` (
`id` int(11) PRIMARY KEY,
`Firstname` varchar(50) NOT NULL,
`Surname` varchar(50) NOT NULL,
`Year_Group` int(2) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
# STUDENT INSERTS
INSERT INTO `students`
(`id`, `Firstname`, `Surname`, `Year_Group`)
VALUES
(201, 'Student', 'A', 9),
(202, 'Student', 'B', 9),
(203, 'Student', 'C', 9),
(204, 'Student', 'D', 9),
(205, 'Student', 'E', 9);
CREATE TABLE `alert` (
`ID` int(11) PRIMARY KEY,
`Staff_ID` int(6) NOT NULL,
`Datetime_Raised` datetime NOT NULL,
`Room_Label` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`Type` enum('A','C','P','Q','S') COLLATE utf8_unicode_ci NOT NULL COMMENT 'A=Absconded, C=Crisis, P=Process, Q=Quiet, S=Safeguarding',
`Details` text COLLATE utf8_unicode_ci,
`Responder` int(8) DEFAULT NULL,
`Datetime_Responded` datetime DEFAULT NULL,
`Room_ID` int(11) NOT NULL COMMENT 'will be linked to internal room id.',
`Status` varchar(1) COLLATE utf8_unicode_ci DEFAULT NULL COMMENT 'O:ngoing, R:esolved'
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
# ALERT INSERTS
INSERT INTO `alert`
(`ID`, `Staff_ID`, `Datetime_Raised`, `Room_Label`, `Type`, `Details`, `Responder`, `Datetime_Responded`, `Room_ID`, `Status`)
VALUES
(1, '101', '2016-08-04 00:00:00', NULL, 'P', NULL, '103', '2016-08-04 00:00:01', '15', 'R'),
(2, '102', '2016-08-04 00:00:00', NULL, 'P', NULL, '103', '2016-08-04 00:00:01', '15', 'R'),
(3, '102', '2016-08-04 00:00:00', NULL, 'P', NULL, '103', '2016-08-04 00:00:01', '15', 'R'),
(4, '101', '2016-08-04 00:00:00', NULL, 'P', NULL, '103', '2016-08-04 00:00:01', '15', 'R');
CREATE TABLE `alert_students` (
`ID` int(11) PRIMARY KEY,
`Alert_ID` int(6) NOT NULL,
`Student_ID` int(6) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
# ALERT_STUDENT INSERTS
INSERT INTO `alert_students`
(`ID`, `Alert_ID`, `Student_ID`)
VALUES
(1, '1', '201'),
(2, '1', '202'),
(3, '2', '201'),
(4, '3', '202'),
(5, '4', '203'),
(6, '5', '204');
CREATE TABLE `flags` (
`ID` int(11) PRIMARY KEY,
`Staff_ID` int(11) NOT NULL,
`Student_ID` int(11) NOT NULL,
`Datetime` datetime NOT NULL,
`Category_ID` int(11) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
# ALERT INSERTS
-- TRUNCATE TABLE flags;
INSERT INTO `flags`
(`ID`, `Staff_ID`, `Student_ID`, `Datetime`, `Category_ID`)
VALUES
(1, '101', '201', '2016-08-04 00:00:01', 10),
(2, '102', '202', '2016-08-04 00:00:02', 12),
(3, '102', '203', '2016-08-04 00:00:03', 10),
(4, '101', '204', '2016-08-04 00:00:04', 13),
(5, '102', '202', '2016-08-04 00:00:02', 12),
(6, '102', '203', '2016-08-04 00:00:03', 10),
(7, '101', '204', '2016-08-04 00:00:04', 13),
(8, '102', '202', '2016-08-04 00:00:02', 10),
(9, '102', '203', '2016-08-04 00:00:03', 10),
(10, '101', '204', '2016-08-04 00:00:04', 7),
(11, '101', '204', '2016-08-04 00:00:07', 8),
(12, '101', '204', '2016-08-04 00:00:08', 1),
(13, '101', '204', '2016-08-04 00:00:09', 8);
Stored Procedure:
DROP PROCEDURE IF EXISTS rptSM_by_year;
DELIMITER $$
CREATE PROCEDURE rptSM_by_year
( pSY INT -- parameter student year
)
BEGIN
DROP TEMPORARY TABLE IF EXISTS tmpStudentMetrics;
CREATE TEMPORARY TABLE tmpStudentMetrics
( `StudentId` int(11) PRIMARY KEY,
LateFP INT NOT NULL,
MobiFP INT NOT NULL,
AbscFP INT NOT NULL,
CommFP INT NOT NULL,
SafeAP INT NOT NULL,
BehaFP INT NOT NULL,
ProcAP INT NOT NULL
)ENGINE=InnoDB;
INSERT tmpStudentMetrics (StudentId,LateFP,MobiFP,AbscFP,CommFP,SafeAP,BehaFP,ProcAP)
SELECT id,0,0,0,0,0,0,0
FROM students
WHERE Year_Group = pSY;
UPDATE tmpStudentMetrics tmp
JOIN
( SELECT
stu.id,
SUM(CASE WHEN f.Category_ID = 10 THEN 1 ELSE 0 END) AS `LateFP`,
SUM(CASE WHEN f.Category_ID = 15 THEN 1 ELSE 0 END) AS `AbscFP`,
SUM(CASE WHEN f.Category_ID = 13 THEN 1 ELSE 0 END) AS `CommFP`,
SUM(CASE WHEN f.Category_ID = 12 THEN 2 ELSE 0 END) AS `MobiFP`,
SUM(CASE WHEN f.Category_ID IN (1,7,8) THEN 2 ELSE 0 END) AS `BehaFP`
FROM `flags` f
LEFT JOIN `students` stu ON f.Student_ID = stu.id
WHERE stu.Year_Group = pSY
GROUP BY stu.id
) xDerived
ON xDerived.id=tmp.StudentId
SET tmp.LateFP=xDerived.LateFP,
tmp.AbscFP=xDerived.AbscFP,
tmp.CommFP=xDerived.CommFP,
tmp.MobiFP=xDerived.MobiFP,
tmp.BehaFP=xDerived.BehaFP;
UPDATE tmpStudentMetrics tmp
JOIN
( SELECT
stu.id,
SUM(CASE WHEN a.Type = 'S' THEN 1 ELSE 0 END) AS `SafeAP`,
SUM(CASE WHEN a.Type = 'P' THEN 3 ELSE 0 END) AS `ProcAP`
FROM `alert_students` als
JOIN `alert` a
ON a.ID=als.Alert_ID
JOIN `students` stu
ON stu.id=als.Student_ID and stu.Year_Group = pSY
GROUP BY stu.id
) xDerived
ON xDerived.id=tmp.StudentId
SET tmp.SafeAP=xDerived.SafeAP,
tmp.ProcAP=xDerived.ProcAP;
-- SELECT * FROM tmpStudentMetrics; -- check detail
SELECT stu.id,
CONCAT(stu.Firstname, " ", stu.Surname) AS `Student`,
tmp.LateFP+tmp.MobiFP+tmp.AbscFP+tmp.CommFP+tmp.SafeAP+tmp.BehaFP+tmp.ProcAP AS `Points`
FROM `students` stu
JOIN tmpStudentMetrics tmp
ON tmp.StudentId=stu.id
WHERE stu.`Year_Group` = pSY
ORDER BY stu.id;
-- SELECT * FROM tmpStudentMetrics; -- check detail
DROP TEMPORARY TABLE IF EXISTS tmpStudentMetrics;
-- TEMP TABLES are connection based. Explicityly dropped above for safety when done.
-- Depends on your connection type and life-span otherwise.
END$$
DELIMITER ;
Test:
call rptSM_by_year(9);
+-----+-----------+--------+
| id | Student | Points |
+-----+-----------+--------+
| 201 | Student A | 7 |
| 202 | Student B | 11 |
| 203 | Student C | 6 |
| 204 | Student D | 10 |
| 205 | Student E | 0 |
+-----+-----------+--------+
Cleanup:
drop schema s38741386; -- drop the test database
Think all you have asked can be done with a subquery and a couple of sub-SELECTs:
SELECT `Student`,
`Late Flags` * 1
+ `Absconded Flags` * 1
+ `Community Flags` * 1
+ `Safeguarding Alerts Flags` * 1
+ `Behavioural flags` * 2
+ `Process Alerts Flags` * 3 AS `Total Points`
FROM
(
SELECT
CONCAT(stu.Firstname, " ", stu.Surname) AS `Student`,
SUM(CASE WHEN f.Category_ID = 10 THEN 1 ELSE 0 END) AS `Late Flags`,
SUM(CASE WHEN f.Category_ID = 12 THEN 1 ELSE 0 END) AS `Mobile Flags`,
SUM(CASE WHEN f.Category_ID = 15 THEN 1 ELSE 0 END) AS `Absconded Flags`,
SUM(CASE WHEN f.Category_ID = 13 THEN 1 ELSE 0 END) AS `Community Flags`,
(SELECT COUNT(*) FROM `alert` a JOIN `alert_students` ast ON ast.`Alert_ID` = a.`ID`
WHERE ast.`Student_ID` = stu.`id` AND a.`Type` = 'S') AS `Safeguarding Alerts Flags`,
SUM(CASE WHEN f.Category_ID IN (1, 7, 8) THEN 1 ELSE 0 END) AS `Behavioural flags`,
(SELECT COUNT(*) FROM `alert` a JOIN `alert_students` ast ON ast.`Alert_ID` = a.`ID`
WHERE ast.`Student_ID` = stu.`id` AND a.`Type` = 'P') AS `Process Alerts Flags`
FROM `students` stu
LEFT JOIN `flags` f ON f.Student_ID = stu.id
WHERE stu.Year_Group = 9
GROUP BY stu.id
LIMIT 0, 20
) subq
ORDER BY `Total Points` DESC;
The above query includes everything you mentioned apart from sanctions (as your original SQL Fiddle demo didn't include this table).
Demo
An updated fiddle with the above query is here: http://sqlfiddle.com/#!9/449218/39.
You could use union all
Basically you create all your individual queries for each table and connect them all together using union all.
Here is an example, I used your student table twice but you would change the second one to what ever other table you want. SQLFiddle
You can do it with LEFT JOINS:
SELECT CONCAT(stu.firstname,' ', stu.surname) student,
COALESCE(f_group.weight_sum,0) + COALESCE(a_group.weight_sum,0) + COALESCE(s_group.weight_sum,0) points
FROM students stu
LEFT JOIN (
SELECT s_f.id, SUM(f.category_id IN (10,13,15) + 2 * f.category_id IN (1,7,8)) weight_sum
FROM students s_f
JOIN flags f
ON f.student_id = s_f.id
AND f.category_id IN (1,7,8,10,13,15)
WHERE s_f.year_group = :year_group
GROUP BY s_f.id
) f_group
LEFT JOIN (
SELECT s_a.id, 3 * COUNT(*) weight_sum
FROM students s_a
JOIN alerts a
ON a.student_id = s_a.id
AND a.type = 'P'
WHERE s_a.year_group = :year_group
GROUP BY s_a.id
) a_group
LEFT JOIN (
SELECT s_s.id, COUNT(*) weight_sum
FROM students s_s
JOIN sanctions s
ON s.student_id = s_s.id
AND s.category_id = 1
WHERE s_s.year_group = :year_group
GROUP BY s_s.id
) s_group
WHERE stu.year_group = :year_group
ORDER BY points DESC
LIMIT 0, 20
BUT if you have full access to the DB I'd be putting those weights in the respective categories and types, which will simplify the logic.

Get sub total of two column for particular criterion and another total as full group by X parameter

I want to group by keyword name, get sum of cf1+cf2 (where bug_status=CLOSED or RESOLVED) and get total sum (irrespective of bug status). Output will have 3 columns like mentioned.
Tried query but no luck:
SELECT keyworddefs.name as keyword, IFNULL(SUM(bugs.cf1 + bugs.cf2),0) as completed, (SELECT IFNULL(SUM(bugs.cf1 + bugs.cf2) ,0) FROM bugs, keywords, keyworddefs WHERE (keywords.bug_id = bugs .bug_id) AND (keyworddefs.id=keywords.keywordid) AND (keyworddefs.name LIKE 'K%')) as total FROM bugs, keywords, keyworddefs WHERE (keywords.bug_id = bugs .bug_id) AND (keyworddefs.id=keywords.keywordid) AND (bugs.bug_status = 'VERIFIED' OR bugs.bug_status = 'CLOSED') GROUP BY keyworddefs.name DESC;
Here's the query formatted.
SELECT keyworddefs.name as keyword,
IFNULL(SUM(bugs.cf1 + bugs.cf2),0) as completed,
(SELECT IFNULL(SUM(bugs.cf1 + bugs.cf2) ,0)
FROM bugs, keywords, keyworddefs
WHERE (keywords.bug_id = bugs .bug_id)
AND (keyworddefs.id=keywords.keywordid)
AND (keyworddefs.name LIKE 'K%')) as total
FROM bugs, keywords, keyworddefs
WHERE (keywords.bug_id = bugs .bug_id)
AND (keyworddefs.id=keywords.keywordid)
AND (bugs.bug_status = 'VERIFIED' OR bugs.bug_status = 'CLOSED')
GROUP BY keyworddefs.name DESC;
SQL Fiddle:
http://sqlfiddle.com/#!9/a11b4/7
Expected:
Matching records:
cf1+cf2 bugid, keyword bug_status
5 (102, 'K1') CLOSED
3 (565, 'K2') CLOSED
3 (1352, 'K1') VERIFIED
4 (13634, 'K1') NEW
# Query output should be:
keyword completed total
K1 8 12
K2 3 3
DDLs:
bugs table1 (master table) :
CREATE TABLE `bugs` (
`bug_id` int(11) NOT NULL,
`bug_date` date NOT NULL,
`cf1` int(11) NOT NULL,
`cf2` int(11) NOT NULL,
`bug_status` varchar(200) NOT NULL)
ENGINE=InnoDB DEFAULT CHARSET=latin1;
INSERT INTO `bugs` (`bug_id`, `bug_date`, `cf1`, `cf2`, `bug_status`) VALUES
(102, '2016-07-19', 2, 1, 'CLOSED'),
(72123, '2016-07-19', 2, 1, 'VERIFIED'),
(57234, '2016-07-19', 2, 1, 'VERIFIED'),
(1352, '2016-07-19', 2, 1, 'VERIFIED'),
(565, '2016-07-19', 2, 1, 'CLOSED'),
(13634, '2016-07-22', 2, 2, 'NEW');
keywords table2 (having keyword ids):
CREATE TABLE `keywords` (
`bug_id` int(11) NOT NULL,
`keywordid` varchar(11) NOT NULL)
ENGINE=InnoDB DEFAULT CHARSET=latin1;
INSERT INTO `keywords` (`bug_id`, `keywordid`) VALUES
(102, '3'),
(565, '4'),
(398, '1'),
(565, '2'),
(1352, '1'),
(57234, '2'),
(1363, '1'),
(72123, '2'),
(13634, '3');
keyworddefs table3 (having keyword names according to keywordid):
CREATE TABLE `keyworddefs` (
`id` int(11) NOT NULL,
`name` varchar(200) NOT NULL,
`description` varchar(200) NOT NULL)
ENGINE=InnoDB DEFAULT CHARSET=latin1;
INSERT INTO `keyworddefs` (`id`, `name`, `description`) VALUES
(1, 'J1', 'My J1 item'),
(2, 'J2', 'My J2 item'),
(3, 'K1', 'My K1 item'),
(4, 'K2', 'My K2 item');
How can I get output like expected?
It looks to me like you're making this too complicated.
For one thing, you should join your keywords and keyworddefs tables ON keywords.keywordid = keyworddefs.name. You're using keyworddefs.id. That's a number. So, your old-timey early 1990s comma join yields no results.
For another thing, you don't need to join the keyworddefs table to get your result.
SUM() operations rarely yield NULL results. So, you should put your conditionals inside the parentheses of SUM() rather than outside.
Finally, you need a GROUP BY operation with two SUM() aggregates in it. One should be conditioned on the bug_status and the other should not.
http://sqlfiddle.com/#!9/a11b4/11/0
Something like this should work.
SELECT keywords.keywordid,
SUM(CASE WHEN bugs.bug_status IN ('CLOSED', 'RESOLVED')
THEN bugs.cf1 + bugs.cf2
ELSE 0 END) completed,
SUM(bugs.cf1 + bugs.cf2) total
FROM bugs
JOIN keywords ON bugs.bug_id = keywords.bug_id
GROUP BY keywords.keywordid
ORDER BY keywords.keywordid
If you need to filter your results by keywords.keywordid LIKE 'K%', you can just add a where clause.
Extended query from Ollie's comment, it works fine with couple of changes.
Highly appreciated Ollie!
SELECT keyworddefs.name,
SUM(CASE WHEN bugs.bug_status IN ('CLOSED', 'VERIFIED') THEN bugs.cf1 + bugs.cf2 ELSE 0 END) completed,
SUM(bugs.cf1 + bugs.cf2) total
FROM bugs
JOIN keywords ON bugs.bug_id = keywords.bug_id
JOIN keyworddefs ON keyworddefs.id = keywords.keywordid
WHERE keyworddefs.name LIKE 'K%'
GROUP BY keywords.keywordid
ORDER BY keyworddefs.name DESC;

How to improve ColdFusion MySQL query time?

I have 15 queries that generate data for a table on a page dynamically for the purpose of reports. Each query takes between 250 and 900ms which means a page loading time of 4 to 13 seconds depending on server load. The loading time is causing some users to the think the page is not going to load at all.
I was wondering if there was some way I could streamline the queries to give a more acceptable loading time. Here is one of the queries:
<cfquery datasource="MeetingDB" name="One">
SELECT COUNT( meetingID ) AS countatron
FROM case_meeting
WHERE meetingID
IN (
SELECT DISTINCT a.meetingID
FROM case_meeting a
INNER JOIN meeting b ON a.meetingID = b.meetingID
WHERE b.categoryID = '1'
AND SUBSTRING( meetingCode, 5, 2 )
BETWEEN 12
AND 22
AND SUBSTRING( meetingCode, 7, 2 )
BETWEEN 01
AND 12
AND SUBSTRING( meetingCode, 9, 2 )
BETWEEN 01
AND 31
)
AND caseID
IN (
'1', '2', '3', '28', '29', '30', '39', '40', '45'
)
GROUP BY meetingID
HAVING COUNT( caseID ) > 0 AND COUNT( caseID ) < 2
</cfquery>
<td><cfoutput> #One.recordcount# </cfoutput></td>
Try this query
SELECT COUNT( a.meetingID ) AS countatron
FROM case_meeting a, case_meeting b
WHERE a.meetingID = b.meetingID
AND b.categoryID = '1'
AND SUBSTRING( b.meetingCode, 5, 2 )
BETWEEN 12
AND 22
AND SUBSTRING( b.meetingCode, 7, 2 )
BETWEEN 01
AND 12
AND SUBSTRING( b.meetingCode, 9, 2 )
BETWEEN 01
AND 31
AND b.caseID
IN (
'1', '2', '3', '28', '29', '30', '39', '40', '45'
)
GROUP BY a.meetingID
HAVING COUNT( a.caseID ) = 1
Might be worth trying to do joins on subselects rather than using IN.
Something like this:-
SELECT COUNT( case_meeting.meetingID ) AS countatron
FROM case_meeting
INNER JOIN (
SELECT DISTINCT a.meetingID
FROM case_meeting a
INNER JOIN meeting b ON a.meetingID = b.meetingID
WHERE b.categoryID = '1'
AND SUBSTRING( meetingCode, 5, 2 ) BETWEEN 12 AND 22
AND SUBSTRING( meetingCode, 7, 2 ) BETWEEN 01 AND 12
AND SUBSTRING( meetingCode, 9, 2 ) BETWEEN 01 AND 31
) Sub1
ON case_meeting.meetingID = Sub1.meetingID
INNER JOIN (
SELECT meetingID, COUNT( caseID ) AS MeetingCaseCount
FROM case_meeting
WHERE caseID IN ('1', '2', '3', '28', '29', '30', '39', '40', '45')
GROUP BY meetingID
) Sub2
ON case_meeting.meetingID = Sub2.meetingID
WHERE Sub2.MeetingCaseCount > 0 AND Sub2.MeetingCaseCount < 2
GROUP BY case_meeting.meetingID
If you have a lot of queries that don't depend on each other, then take a look at cfthread. This will allow you to run the queries concurrently.
Make sure you test it thoroughly. I've had one experience where the use of cfthread had adverse effects on a database server.
It's still worth a shot though.
i suppose creating a procedure in the MySql and invoking it with arguments from my sql is most appropriate. you could also create views with dynamic param