Group by Query - Select row with maximum date - mysql

Hi I have this scores table, And in my report on front end, I have to display the keyword and url and score for latest scan.
CREATE TABLE `scores` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`keyword` varchar(200) DEFAULT NULL,
`url` varchar(200) DEFAULT NULL,
`score` int(11) DEFAULT NULL,
`check_date` date DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=latin1;
Here is my Example Data:
Sample Data
| id | keyword | url | score | check_date |
|----|----------|----------------------|-------|------------|
| 1 | facebook | https://facebook.com | 10 | 2020-10-21 |
| 2 | facebook | https://facebook.com | 30 | 2020-10-25 |
| 3 | fb | https://facebook.com | 55 | 2020-10-23 |
| 4 | fb | https://facebook.com | 20 | 2020-10-24 |
My Query
SELECT s1.*
FROM scores s1
JOIN scores s2
ON s1.id = s2.id
WHERE s1.check_date = s2.check_date
GROUP BY keyword,url
It returns correct check_date for a specific keyword, url but score is not according to that date. Please help.

Do not use aggregation for this. A simple method is a correlated subquery:
select s.*
from scores s
where s.check_date = (select max(s2.check_date)
from scores s2
where s2.keyword = s.keyword and s2.url = s.url
);
If you are intent on using an explicit join you can use a left join, look for a larger date, and return the rows that have no larger date:
select s.*
from scores s left join
scores slater
on slater.keyword = s.keyword and
slater.url = s.url and
slater.check_date > s.check_date
where slater.check_date is null;

Related

Change json field in mysql to include data from other table

I have these two tables:
CREATE TABLE `config_support_departments` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`title` varchar(200) NOT NULL DEFAULT '',
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `support_tickets_filters` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`filter_departments` json DEFAULT NULL,
PRIMARY KEY (`id`),
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
config_support_departments:
+----+-------------------------+
| id | title |
+----+-------------------------+
| 1 | Projects / File Support |
| 2 | Sales Support |
| 3 | IT Support |
+----+-------------------------+
support_tickets_filters:
+----+--------------------+
| id | filter_departments |
+----+--------------------+
| 1 | ["2", "3"] |
+----+--------------------+
What I need is when query the support_tickets_filters table to also include the title from the config_support_departments table. Hence, the result should be something like this:
+----+-----------------------------------------------+
| id | filter_departments |
+----+-----------------------------------------------+
| 1 | {"2":"Sales Support","3":"IT Support"} |
+----+-----------------------------------------------+
You can use JSON_OBJECTAGG() function along with JSON_TABLE() if DB version is 8.0+
SELECT s.id, JSON_OBJECTAGG(c.id,c.title) AS filter_departments
FROM `support_tickets_filters` AS s
JOIN JSON_TABLE(
s.`filter_departments`,
'$[*]' COLUMNS (id INT PATH '$')
) j
JOIN `config_support_departments` AS c
ON j.id = c.id
GROUP BY s.id
For DB version 5.7, you can use one of the DB metadata tables such as information_schema.tables in order to generate index values as 0,1,... upto the length of the array (filter_departments) for extracting the related value from that iteratively
SELECT s.id, JSON_OBJECTAGG(c.id,c.title) AS filter_departments
FROM
(
SELECT #i := #i + 1 AS n, s.id,
JSON_UNQUOTE(JSON_EXTRACT(`filter_departments`,CONCAT('$[',#i,']'))) AS elm
FROM `support_tickets_filters` AS s
JOIN (SELECT #i := -1) AS iter
LEFT JOIN information_schema.tables AS t
ON #i < JSON_LENGTH(`filter_departments`) - 1 ) AS s
JOIN `config_support_departments` AS c
ON s.elm = c.id
GROUP BY s.id
Demo

How to divide the current price to the last price in the same column in sql

How can I divide ((89-95)/95)*100 or ((95-100)/100)*100
CREATE TABLE `priceindex` (
`priceIndexId` int(11) NOT NULL,
`Date` date DEFAULT NULL,
`Price` decimal(31,9) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
priceIndexId | Date | Price | Currentvalue/previous value
1 | 2017-11-30 | 100 |
2 | 2017-12-06 | 95 | answer should be(95-100)/100)*100 = -0.50
3 | 2017-12-13 | 89 | answer should be(89-95)/95)*100 = -0.63
Thanks in advance
You need to get the previous value. One method uses a correlated subquery. I would suggest using a subquery for the calculation:
select pi.priceIndexId, pi.Date, pi.Price,
(pi.Price - pi.prev_price) / pi.prev_price as change
from (select pi.*,
(select pi2.price
from priceindex pi2
where pi2.date < pi.date
order by pi2.date desc
limit 1
) as prev_price
from priceindex pi
) pi;

select from tables with different numbers of rows

I'm hoping there is a simple answer to this. Competitors race over a series of 3 races. Some competitors only show up for one race. How could I show a final result for ALL competitors?
race 1
+------+--------+
| name | result |
+------+--------+
| Ali | 30 |
| Bob | 28 |
| Cal | 26 |
+------+--------+
race 2
+------+--------+
| name | result |
+------+--------+
| Ali | 32 |
| Bob | 31 |
| Dan | 24 |
+------+--------+
race 3
+------+--------+
| name | result |
+------+--------+
| Eva | 23 |
| Dan | 25 |
+------+--------+
The final result should look like this:
+------+--------+--------+--------+
| name | result | result | result |
+------+--------+--------+--------+
| Ali | 30 | 32 | |
| Bob | 28 | 31 | |
| Cal | 26 | | |
| Dan | | 24 | 25 |
| Eva | | | 23 |
+------+--------+--------+--------+
The problem I have is with ordering by name from multiple tables.
Here is the example data:
CREATE TABLE race (name varchar(20), result int);
CREATE TABLE race1 LIKE race;
INSERT INTO race1 VALUES ('Ali', '30'), ('Bob', '28'), ('Cal', '26');
CREATE TABLE race2 like race;
insert INTO race2 VALUES ('Ali', '32'), ('Bob', '31'), ('Dan', '24');
CREATE TABLE race3 LIKE race;
INSERT INTO race3 VALUES ('Eva', '23'), ('Dan', '25');
Many thanks!
Here we go !!!
select race1.name as name, race1.result, race2.result, race3.result from race1
left join race2 on race2.name = race1.name
left join race3 on race3.name = race1.name
union
select race2.name as name, race1.result, race2.result, race3.result from race2
left join race1 on race1.name = race2.name
left join race3 on race3.name = race2.name
union
select race3.name as name, race1.result, race2.result, race3.result from race3
left join race1 on race1.name = race3.name
left join race2 on race2.name = race3.name;
It is working :)
select s.name,
max(case when s.R = 'Result1' then s.result else '' end) as result1,
max(case when s.R = 'Result2' then s.result else '' end) as result2,
max(case when s.R = 'Result3' then s.result else '' end) as result3
from
(
select 'Result1' as R,r1.* from race1 r1
union all
select 'Result2' as R,r2.* from race2 r2
union all
select 'Result3' as R,r3.* from race3 r3
) s
group by s.name
result
+------+---------+---------+---------+
| name | result1 | result2 | result3 |
+------+---------+---------+---------+
| Ali | 30 | 32 | |
| Bob | 28 | 31 | |
| Cal | 26 | | |
| Dan | | 24 | 25 |
| Eva | | | 23 |
+------+---------+---------+---------+
5 rows in set (0.00 sec)
I personally would create the schema in a different way.
One table for the users, one for the races and one that connects both:
-- Create syntax for TABLE 'races'
CREATE TABLE `races` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(255) DEFAULT NULL,
`created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
-- Create syntax for TABLE 'users'
CREATE TABLE `users` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(255) DEFAULT NULL,
`created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
-- Create syntax for TABLE 'race_results'
CREATE TABLE `race_results` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`race_id` int(11) NOT NULL,
`user_id` int(11) NOT NULL,
`result` int(11) NOT NULL,
`created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Let's insert some data (should be equal to your data set).
-- Insert data
INSERT INTO users (name)values('Ali'),('Bob'),('Cal'),('Dan'), ('Eva');
INSERT INTO races (name)values('Race1'),('Race2'),('Race3');
INSERT INTO race_results (user_id, race_id, result)values(1,1,30),(2,1,30),(1,2,28),(2,2,31),(3,1,26),(4,2,24),(4,3,25),(5,3,23);
Then you could write the query like this:
-- Static version
SELECT us.name, sum(if(ra.name='Race1', result, null)) as Race1, sum(if(ra.name='Race2', result, null)) as Race2, sum(if(ra.name='Race3', result, null)) as Race3
FROM race_results as rr
LEFT JOIN users as us on us.id = rr.user_id
LEFT JOIN races as ra on ra.id = rr.race_id
GROUP BY us.id;
Which gives you the result you're looking for. (I changed the column names to make it more obvious which result belongs to which race.)
But I've to admit that this works fine for 3 races but what if you have 30 or more?
Here is a more dynamic version of the above query, which kind of creates itself ;)
-- Dynamic version
SET #sql = '';
SELECT
#sql := CONCAT(#sql,if(#sql='','',', '),temp.output)
FROM
(SELECT
CONCAT("sum(if(ra.name='", race.name, "', result, null)) as ", race.name) as output
FROM races as race
) as temp;
SET #sql = CONCAT("SELECT us.name,", #sql, " FROM race_results as rr LEFT JOIN users as us on us.id = rr.user_id LEFT JOIN races as ra on ra.id = rr.race_id GROUP BY 1;");
SELECT #sql;
PREPARE stmt FROM #sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;

MySQL GROUP BY with sorting

I'm having some trouble writing succinct code to generate the desired result efficiently (on a multiple million records DB).
items will be grouped by time
items will be selected by provider being that B takes precedence over A (and C over B)
value must match value of selected provider
Table vs wanted result:
// given this table
id | provider | time | value
---+----------+------------+-----------
1 | A | 2013-07-01 | 0.1
2 | A | 2013-07-02 | 0.2
3 | B | 2013-07-02 | 0.3
4 | A | 2013-07-03 | 0.4
// extrapolate this result
---+----------+------------+-----------
1 | A | 2013-07-01 | 0.1
3 | B | 2013-07-02 | 0.3
4 | A | 2013-07-03 | 0.4
The queries to generate table and populate data:
data_teste CREATE TABLE `data_teste` (`id` int(11) unsigned NOT NULL AUTO_INCREMENT,`provider` varchar(12) NOT NULL,`time` date NOT NULL,`value` double NOT NULL,PRIMARY KEY (`id`),UNIQUE KEY `index` (`provider`,`time`),KEY `provider` (`provider`),KEY `time` (`time`)) ENGINE=InnoDB DEFAULT CHARSET=utf8;
INSERT INTO data_teste(`provider`, `time`, `value`) VALUES('A', '2013-07-01', 0.1),('A', '2013-07-02', 0.2),('B', '2013-07-02', 0.3),('A', '2013-07-03', 0.4);
This is the classic group_by/sort problem windowed.
Thank you very much.
select d.*
from data_teste d
inner join
(
select `time`, max(provider) mp
from data_teste
group by `time`
) x on x.mp = d.provider
and x.`time` = d.`time`
order by `time` asc,
provider desc
How well does this perform?
SELECT
*
FROM
`data_teste` dt1
LEFT JOIN `data_teste` dt2 ON ( dt2.time = dt1.time
AND dt2.provider > dt1.provider )
WHERE
dt2.ID IS NULL

MySQL Query Optimization with MAX()

I have 3 tables with the following schema:
CREATE TABLE `devices` (
`device_id` int(11) NOT NULL auto_increment,
`name` varchar(20) default NULL,
`appliance_id` int(11) default '0',
`sensor_type` int(11) default '0',
`display_name` VARCHAR(100),
PRIMARY KEY USING BTREE (`device_id`)
)
CREATE TABLE `channels` (
`channel_id` int(11) NOT NULL AUTO_INCREMENT,
`device_id` int(11) NOT NULL,
`channel` varchar(10) NOT NULL,
PRIMARY KEY (`channel_id`),
KEY `device_id_idx` (`device_id`)
)
CREATE TABLE `historical_data` (
`date_time` datetime NOT NULL,
`channel_id` int(11) NOT NULL,
`data` float DEFAULT NULL,
`unit` varchar(10) DEFAULT NULL,
KEY `devices_datetime_idx` (`date_time`) USING BTREE,
KEY `channel_id_idx` (`channel_id`)
)
The setup is that a device can have one or more channels and each channel has many (historical) data.
I use the following query to get the last historical data for one device and all it's related channels:
SELECT c.channel_id, c.channel, max(h.date_time), h.data
FROM devices d
INNER JOIN channels c ON c.device_id = d.device_id
INNER JOIN historical_data h ON h.channel_id = c.channel_id
WHERE d.name = 'livingroom' AND d.appliance_id = '0'
AND d.sensor_type = 1 AND ( c.channel = 'ch1')
GROUP BY c.channel
ORDER BY h.date_time, channel
The query plan looks as follows:
+----+-------------+-------+--------+-----------------------+----------------+---------+---------------------------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-----------------------+----------------+---------+---------------------------+--------+-------------+
| 1 | SIMPLE | c | ALL | PRIMARY,device_id_idx | NULL | NULL | NULL | 34 | Using where |
| 1 | SIMPLE | d | eq_ref | PRIMARY | PRIMARY | 4 | c.device_id | 1 | Using where |
| 1 | SIMPLE | h | ref | channel_id_idx | channel_id_idx | 4 | c.channel_id | 322019 | |
+----+-------------+-------+--------+-----------------------+----------------+---------+---------------------------+--------+-------------+
3 rows in set (0.00 sec)
The above query is currently taking approximately 15 secs and I wanted to know if there are any tips or way to improve the query?
Edit:
Example data from historical_data
+---------------------+------------+------+------+
| date_time | channel_id | data | unit |
+---------------------+------------+------+------+
| 2011-11-20 21:30:57 | 34 | 23.5 | C |
| 2011-11-20 21:30:57 | 9 | 68 | W |
| 2011-11-20 21:30:54 | 34 | 23.5 | C |
| 2011-11-20 21:30:54 | 5 | 316 | W |
| 2011-11-20 21:30:53 | 34 | 23.5 | C |
| 2011-11-20 21:30:53 | 2 | 34 | W |
| 2011-11-20 21:30:51 | 34 | 23.4 | C |
| 2011-11-20 21:30:51 | 9 | 68 | W |
| 2011-11-20 21:30:49 | 34 | 23.4 | C |
| 2011-11-20 21:30:49 | 4 | 193 | W |
+---------------------+------------+------+------+
10 rows in set (0.00 sec)
Edit 2:
Mutliple channel SELECT example:
SELECT c.channel_id, c.channel, max(h.date_time), h.data
FROM devices d
INNER JOIN channels c ON c.device_id = d.device_id
INNER JOIN historical_data h ON h.channel_id = c.channel_id
WHERE d.name = 'livingroom' AND d.appliance_id = '0'
AND d.sensor_type = 1 AND ( c.channel = 'ch1' OR c.channel = 'ch2' OR c.channel = 'ch2')
GROUP BY c.channel
ORDER BY h.date_time, channel
I've used OR in the c.channel where clause because it was easier to generated pro grammatically but it can be changed to use IN if necessary.
Edit 3:
Example result of what I'm trying to achieve:
+-----------+------------+---------+---------------------+-------+
| device_id | channel_id | channel | max(h.date_time) | data |
+-----------+------------+---------+---------------------+-------+
| 28 | 9 | ch1 | 2011-11-21 20:39:36 | 0 |
| 28 | 35 | ch2 | 2011-11-21 20:30:55 | 32767 |
+-----------+------------+---------+---------------------+-------+
I have added the device_id to the example but my select will only need to return channel_id, channel, last date_time i.e max and the data. The results should be the last record from the historical_data table for each channel for one device.
It seems that removing an re-creating the index on date_time by deleting and creating it again sped up my original SQL up to around 2secs
I haven't been able to test this, so I'd like to ask you to run it and let us know what happens.. if it gives you the desired result and if it runs faster than your current:
CREATE DEFINER=`root`#`localhost` PROCEDURE `GetLatestHistoricalData_EXAMPLE`
(
IN param_device_name VARCHAR(20)
, IN param_appliance_id INT
, IN param_sensor_type INT
, IN param_channel VARCHAR(10)
)
BEGIN
SELECT
h.date_time, h.data
FROM
historical_data h
INNER JOIN
(
SELECT c.channel_id
FROM devices d
INNER JOIN channels c ON c.device_id = d.device_id
WHERE
d.name = param_device_name
AND d.appliance_id = param_appliance_id
AND d.sensor_type = param_sensor_type
AND c.channel = param_channel
)
c ON h.channel_id = c.channel_id
ORDER BY h.date_time DESC
LIMIT 1;
END
Then to run a test:
CALL GetLatestHistoricalData_EXAMPLE ('livingroom', 0, 1, 'ch1');
I tried working it into a stored procedure so that even if you get the desired results using this for one device, you can try it with another device and see the results... Thanks!
[edit] : : In response to Danny's comment here's an updated test version:
CREATE DEFINER=`root`#`localhost` PROCEDURE `GetLatestHistoricalData_EXAMPLE_3Channel`
(
IN param_device_name VARCHAR(20)
, IN param_appliance_id INT
, IN param_sensor_type INT
, IN param_channel_1 VARCHAR(10)
, IN param_channel_2 VARCHAR(10)
, IN param_channel_3 VARCHAR(10)
)
BEGIN
SELECT
h.date_time, h.data
FROM
historical_data h
INNER JOIN
(
SELECT c.channel_id
FROM devices d
INNER JOIN channels c ON c.device_id = d.device_id
WHERE
d.name = param_device_name
AND d.appliance_id = param_appliance_id
AND d.sensor_type = param_sensor_type
AND (
c.channel IN (param_channel_1
,param_channel_2
,param_channel_3
)
c ON h.channel_id = c.channel_id
ORDER BY h.date_time DESC
LIMIT 1;
END
Then to run a test:
CALL GetLatestHistoricalData_EXAMPLE_3Channel ('livingroom', 0, 1, 'ch1', 'ch2' , 'ch3');
Again, this is just for testing, so you'll be able to see if it meets your needs..
I would first add an index on the devices table ( appliance_id, sensor_type, name ) to match your query. I don't know how many entries are in this table, but if large, and many elements per device, get right to it.
Second, on your channels table, index on ( device_id, channel )
Third, on your history data, index on ( channel_id, date_time )
then,
SELECT STRAIGHT_JOIN
PreQuery.MostRecent,
PreQuery.Channel_ID,
PreQuery.Channel,
H2.Data,
H2.Unit
from
( select
c.channel_id,
c.channel,
max( h.date_time ) as MostRecent
from
devices d
join channels c
on d.device_id = c.device_id
and c.channel in ( 'ch1', 'ch2', 'ch3' )
join historical_data h
on c.channel_id = c.Channel_id
where
d.appliance_id = 0
and d.sensor_type = 1
and d.name = 'livingroom'
group by
c.channel_id ) PreQuery
JOIN Historical_Data H2
on PreQuery.Channel_ID = H2.Channel_ID
AND PreQuery.MostRecent = H2.Date_Time
order by
PreQuery.MostRecent,
PreQuery.Channel