I have a table,
| PAGELETS | CREATE TABLE `PAGELETS` (
`page_key` int(32) unsigned NOT NULL,
`pagelet_serial` int(32) unsigned NOT NULL,
`pagelet_shingle` int(32) unsigned NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8
I would like to:
1) Find all the pagelet_shingles where quantity > 1 ( occurs more than once)
2) out of these only output those that have different page_key
This is the query that produces the a semi-correct answer:
SELECT * FROM PAGELETS WHERE pagelet_shingle IN( SELECT pagelet_shingle FROM PAGELETS GROUP BY pagelet_shingle HAVING COUNT(DISTINCT page_key) > 1) ORDER BY pagelet_shingle;
Unfortunately, on a small dataset it takes about 18 seconds;
I have another query,
SELECT dt1.* FROM
(SELECT * FROM PAGELETS
GROUP BY page_key, pagelet_shingle HAVING COUNT(*) = 1)
dt1 JOIN
(SELECT * FROM PAGELETS GROUP BY pagelet_shingle HAVING COUNT(*) > 1)
dt2 USING (pagelet_shingle) ORDER BY pagelet_shingle
given by an expert which is not technically correct (something to do with you can't SELECT * .. GROUP ) but produces results that are A LOT faster, with the case where
SELECT * FROM PAGELETS WHERE pagelet_shingle=57
+----------+----------------+-----------------+
| page_key | pagelet_serial | pagelet_shingle |
+----------+----------------+-----------------+
| 1 | 99 | 57 |
| 1 | 99 | 57 |
| 2 | 228 | 57 |
| 2 | 228 | 57 |
+----------+----------------+-----------------+
The semi-correct query produces
+----------+----------------+-----------------+
| page_key | pagelet_serial | pagelet_shingle |
+----------+----------------+-----------------+
| 1 | 99 | 57 |
| 1 | 99 | 57 |
| 2 | 228 | 57 |
| 2 | 228 | 57 |
+----------+----------------+-----------------+
While the incorrect query doesn't have pagelet_shingle =57 in its resultset
My desired result is to have
+----------+----------------+-----------------+
| page_key | pagelet_serial | pagelet_shingle |
+----------+----------------+-----------------+
| 1 | 99 | 57 |
| 2 | 228 | 57 |
+----------+----------------+-----------------+
Each occuring once only.
a pagelet_shingle occuring twice in the same pagelet_serial will be omitted.
So I would like to ask the following:
1) Is there a way to to speed up the csemi orrect query to reach the speed of the incorrect one
2) or is there a way to fix the incorrect one to produce the result of the correct one ( I don't care about strictness )
Sounds like SELECT DISTINCT p.* ... would be your choice.
P.S. And I would really recommend the second one! make everything slow (like you just noticed) and should only be used where necessary.
doesn't this query solve your issue?
SELECT dt1.* FROM
(SELECT DISTINCT * FROM PAGELETS
GROUP BY page_key, pagelet_shingle HAVING COUNT(*) = 1)
dt1 JOIN
(SELECT * FROM PAGELETS GROUP BY pagelet_shingle HAVING COUNT(*) > 1)
dt2 USING (pagelet_shingle) GROUP BY pagelet_shingle
use GROUP BY and HAVING, e.g.
SELECT *
FROM `pagelets`
GROUP BY `pagelet_shingle`
HAVING COUNT(*) > 1
additionally you can do a self join to output all columns, though in mysql it should work that way (different from SQL standard)
What is
SELECT * FROM PAGELETS GROUP BY pagelet_serial, pagelet_shingle HAVING COUNT(*) > 0
giving you?
Judging from what I read, what you are looking for is:
SELECT DISTINCT p1.page_key, p1.pagelet_serial, p1.pagelet_shingle
FROM PAGELETS p1
JOIN PAGELETS p2 ON p2.page_key = p1.page_key
AND p2.pagelet_serial = p1.pagelet_serial
AND p2.pagelet_shingle <> p1.pagelet_shingle
That query would make full use of an index on (page_key, pagelet_serial) and should complete in tenth of seconds, not seconds.
If this was not what you were looking for, please show us what result you would expect if the values in your table were those: (1,2,3),(1,2,3),(1,1,3),(1,1,3),(1,2,4),(1,2,4),(1,1,4),(1,1,4)
Have you tried using exists instead of in ?
Check this out:
http://decipherinfosys.wordpress.com/2007/01/30/in-vs-exists/
Hope this helps
Related
I'm trying remove the duplicate rows using this query but MySQL don't return nothing and crash:
DELETE FROM project_category WHERE prc_id IN (SELECT prc_id
FROM project_category
GROUP BY prc_proid, prc_catid
HAVING COUNT(*) > 1)
I want remove the duplication:
+--------+-----------+-----------+
| prc_id | prc_proid | prc_catid |
+--------+-----------+-----------+
| 1691 | 207 | 16 |
| 1692 | 207 | 16 |
+--------+-----------+-----------+
MySql does not allow direct reference to the table where the DELETE takes place in the WHERE clause. Do it like this:
DELETE FROM project_category
WHERE prc_id IN (
SELECT prc_id FROM (
SELECT prc_id
FROM project_category
GROUP BY prc_proid, prc_catid
HAVING COUNT(*) > 1
) t
)
I have the following SQL query
SELECT *
FROM `sensor_data` AS `sd1`
WHERE (sd1.timestamp BETWEEN '2017-05-13 00:00:00'
AND '2017-05-14 00:00:00')
AND (`id` =
(
SELECT `id`
FROM `sensor_data` AS `sd2`
WHERE sd1.mid = sd2.mid
AND sd1.sid = sd2.sid
ORDER BY `value` DESC, `id` DESC
LIMIT 1)
)
Background:
I've checked the validity of the query by changing LIMIT 1 to LIMIT 0, and the query works without any problem. However with LIMIT 1 the query doesn't complete, it just states loading until I shutdown and restart.
Breaking the Query down:
I have broken down the query with the date boundary as follows:
SELECT *
FROM `sensor_data` AS `sd1`
WHERE (sd1.timestamp BETWEEN '2017-05-13 00:00:00'
AND '2017-05-14 00:00:00')
This takes about 0.24 seconds to return the query with 8200 rows each having 5 columns.
Question:
I suspect the second half of my Query, is not correct or well optimized.
The tables are as follows:
Current Table:
+------+-------+-------+-----+-----------------------+
| id | mid | sid | v | timestamp |
+------+-------+-------+-----+-----------------------+
| 51 | 10 | 1 | 40 | 2015-05-13 11:56:01 |
| 52 | 10 | 2 | 39 | 2015-05-13 11:56:25 |
| 53 | 10 | 2 | 40 | 2015-05-13 11:56:42 |
| 54 | 10 | 2 | 40 | 2015-05-13 11:56:45 |
| 55 | 10 | 2 | 40 | 2015-05-13 11:57:01 |
| 56 | 11 | 1 | 50 | 2015-05-13 11:57:52 |
| 57 | 11 | 2 | 18 | 2015-05-13 11:58:41 |
| 58 | 11 | 2 | 19 | 2015-05-13 11:58:59 |
| 59 | 11 | 3 | 58 | 2015-05-13 11:59:01 |
| 60 | 11 | 3 | 65 | 2015-05-13 11:59:29 |
+------+-------+-------+-----+-----------------------+
Q: How would I get the MAX(v)for each sid for each mid?
NB#1: In the example above ROW 53, 54, 55 have all the same value (40), but I would like to retrieve the row with the most recent timestamp, which is ROW 55.
Expected Output:
+------+-------+-------+-----+-----------------------+
| id | mid | sid | v | timestamp |
+------+-------+-------+-----+-----------------------+
| 51 | 10 | 1 | 40 | 2015-05-13 11:56:01 |
| 55 | 10 | 2 | 40 | 2015-05-13 11:57:01 |
| 56 | 11 | 1 | 50 | 2015-05-13 11:57:52 |
| 58 | 11 | 2 | 19 | 2015-05-13 11:58:59 |
| 60 | 11 | 3 | 65 | 2015-05-13 11:59:29 |
+------+-------+-------+-----+-----------------------+
Structure of the table:
NB#2:
Since this table has over 110 million entries, it is critical to have have date boundaries, which limits to ~8000 entries over a 24 hour period.
The query can be written as follows:
SELECT t1.id, t1.mid, t1.sid, t1.v, t1.ts
FROM yourtable t1
INNER JOIN (
SELECT mid, sid, MAX(v) as v
FROM yourtable
WHERE ts BETWEEN '2015-05-13 00:00:00' AND '2015-05-14 00:00:00'
GROUP BY mid, sid
) t2
ON t1.mid = t2.mid
AND t1.sid = t2.sid
AND t1.v = t2.v
INNER JOIN (
SELECT mid, sid, v, MAX(ts) as ts
FROM yourtable
WHERE ts BETWEEN '2015-05-13 00:00:00' AND '2015-05-14 00:00:00'
GROUP BY mid, sid, v
) t3
ON t1.mid = t3.mid
AND t1.sid = t3.sid
AND t1.v = t3.v
AND t1.ts = t3.ts;
Edit and Explanation:
The first sub-query (first INNER JOIN) fetches MAX(v) per (mid, sid) combination. The second sub-query is to identify MAX(ts) for every (mid, sid, v). At this point, the two queries do not influence each others' results. It is also important to note that ts date range selection is done in the two sub-queries independently such that the final query has fewer rows to examine and no additional WHERE filters to apply.
Effectively, this translates into getting MAX(v) per (mid, sid) combination initially (first sub-query); and if there is more than one record with the same value MAX(v) for a given (mid, sid) combo, then the excess records get eliminated by the selection of MAX(ts) for every (mid, sid, v) combination obtained by the second sub-query. We then simply associate the output of the two queries by the two INNER JOIN conditions to get to the id of the desired records.
Demo
select * from sensor_data s1 where s1.v in (select max(v) from sensor_data s2 group by s2.mid)
union
select * from sensor_data s1 where s1.v in (select max(v) from sensor_data s2 group by s2.sid);
IN ( SELECT ... ) does not optimize well. It is even worse because of being correlated.
What you are looking for is a groupwise-max .
Please provide SHOW CREATE TABLE; we need to know at least what the PRIMARY KEY is.
Suggested code
You will need:
With the WHERE: INDEX(timestamp, mid, sid, v, id)
Without the WHERE: INDEX(mid, sid, v, timestamp, id)
Code:
SELECT id, mid, sid, v, timestamp
FROM ( SELECT #prev_mid := 99999, -- some value not in table
#prev_sid := 99999,
#n := 0 ) AS init
JOIN (
SELECT #n := if(mid != #prev_mid OR
sid != #prev_sid,
1, #n + 1) AS n,
#prev_mid := mid,
#prev_sid := sid,
id, mid, sid, v, timestamp
FROM sensor_data
WHERE timestamp >= '2017-05-13'
timestamp < '2017-05-13' + INTERVAL 1 DAY
ORDER BY mid DESC, sid DESC, v DESC, timestamp DESC
) AS x
WHERE n = 1
ORDER BY mid, sid; -- optional
Notes:
The index is 'composite' and 'covering'.
This should make one pass over the index, thereby providing 'good' performance.
The final ORDER BY is optional; the results may be in reverse order.
All the DESC in the inner ORDER BY must be in place to work correctly (unless you are using MySQL 8.0).
Note how the WHERE avoids including both midnights? And avoids manually computing leap-days, year-ends, etc?
With the WHERE (and associated INDEX), there will be filtering, but a 'sort'.
Without the WHERE (and the other INDEX), sort will not be needed.
You can test the performance of any competing formulations via this trick, even if you do not have enough rows (yet) to get reliable timings:
FLUSH STATUS;
SELECT ...
SHOW SESSION STATUS LIKE 'Handler%';
This can also be used to compare different versions of MySQL and MariaDB -- I have seen 3 significantly different performance characteristics in a related groupwise-max test.
I have two tables: contacts and client_profiles. A contact has many client_profiles, where client_profiles has foreign key contact_id:
contacts:
mysql> SELECT id,first_name, last_name FROM contacts;
+----+-------------+-----------+
| id | first_name | last_name |
+----+-------------+-----------+
| 10 | THERESA | CAMPBELL |
| 11 | donato | vig |
| 12 | fdgfdgf | gfdgfd |
| 13 | some random | contact |
+----+-------------+-----------+
4 rows in set (0.00 sec)
client_profiles:
mysql> SELECT id, contact_id, created_at FROM client_profiles;
+----+------------+---------------------+
| id | contact_id | created_at |
+----+------------+---------------------+
| 6 | 10 | 2014-10-09 17:17:43 |
| 7 | 10 | 2014-10-10 11:38:01 |
| 8 | 10 | 2014-10-10 12:20:41 |
| 9 | 10 | 2014-10-10 12:24:19 |
| 11 | 12 | 2014-10-10 12:35:32 |
+----+------------+---------------------+
I want to get the latest client_profiles for each contact. That means There should be two results. I want to use subqueries to achieve this. This is the subquery I came up with:
SELECT `client_profiles`.*
FROM `client_profiles`
INNER JOIN `contacts`
ON `contacts`.`id` = `client_profiles`.`contact_id`
WHERE (client_profiles.id =
(SELECT `client_profiles`.`id` FROM `client_profiles` ORDER BY created_at desc LIMIT 1))
However, this is only returning one result. It should return client_profiles with id 9 and 11.
What is wrong with my subquery?
It looks like you were trying to filter twice on the client_profile table, once in the JOIN/ON clause and another time in the WHERE clause.
Moving everything in the where clause looks like this:
SELECT `cp`.*
FROM `contacts`
JOIN (
SELECT
`client_profiles`.`id`,
`client_profiles`.`contact_id`,
`client_profiles`.`created_at`
FROM `client_profiles`
ORDER BY created_at DESC
LIMIT 1
) cp ON `contacts`.`id` = `cp`.`contact_id`
Tell me what you think.
Should be something like maybe:
SELECT *
FROM `client_profiles`
INNER JOIN `contacts`
ON `contacts`.`id` = `client_profiles`.`contact_id`
GROUP BY `client_profiles`.`contact_id`
ORDER BY created_at desc;
http://sqlfiddle.com/#!2/a3f21b/9
You need to prequery the client profiles table grouped by each contact.. From that, re-join to the client to get the person, then again to the client profiles table based on same contact ID, but also matching the max date from the internal prequery using max( created_at )
SELECT
c.id,
c.first_name,
c.last_name,
IDByMaxDate.maxCreate,
cp.id as clientProfileID
from
( select contact_id,
MAX( created_at ) maxCreate
from
client_profiles
group by
contact_id ) IDByMaxDate
JOIN contacts c
ON IDByMaxDate.contact_id = c.id
JOIN client_profiles cp
ON IDByMaxDate.contact_id = cp.contact_id
AND IDByMaxDate.maxCreate = cp.created_at
I am struggling in to get result from mysql in the following way. I have 10 records in mysql db table having date and unit fields. I need to get used units on every date.
Table structure as follows, adding today unit with past previous unit in every record:
Date Units
---------- ---------
10/10/2012 101
11/10/2012 111
12/10/2012 121
13/10/2012 140
14/10/2012 150
15/10/2012 155
16/10/2012 170
17/10/2012 180
18/10/2012 185
19/10/2012 200
Desired output will be :
Date Units
---------- ---------
10/10/2012 101
11/10/2012 10
12/10/2012 10
13/10/2012 19
14/10/2012 10
15/10/2012 5
16/10/2012 15
17/10/2012 10
18/10/2012 5
19/10/2012 15
Any help will be appreciated. Thanks
There's a couple of ways to get the resultset. If you can live with an extra column in the resultset, and the order of the columns, then something like this is a workable approach.
using user variables
SELECT d.Date
, IF(#prev_units IS NULL
,#diff := 0
,#diff := d.units - #prev_units
) AS `Units_used`
, #prev_units := d.units AS `Units`
FROM ( SELECT #prev_units := NULL ) i
JOIN (
SELECT t.Date, t.Units
FROM mytable t
ORDER BY t.Date, t.Units
) d
This returns the specified resultset, but it includes the Units column as well. It's possible to have that column filtered out, but it's more expensive, because of the way MySQL processes an inline view (MySQL calls it a "derived table")
To remove that extra column, you can wrap that in another query...
SELECT f.Date
, f.Units_used
FROM (
query from above goes here
) f
ORDER BY f.Date
but again, removing that column comes with the extra cost of materializing that result set a second time.
using a semi-join
If you are guaranteed to have a single row for each Date value, either stored as a DATE, or as a DATETIME with the timecomponent set to a constant, such as midnight, and no gaps in the Date value, and Date is defined as DATE or DATETIME datatype, then another query that will return the specifid result set:
SELECT t.Date
, t.Units - s.Units AS Units_Used
FROM mytable t
LEFT
JOIN mytable s
ON s.Date = t.Date + INTERVAL -1 DAY
ORDER BY t.Date
If there's a missing Date value (a gap) such that there is no matching previous row, then Units_used will have a NULL value.
using a correlated subquery
If you don't have a guarantee of no "missing dates", but you have a guarantee that there is no more than one row for a particular Date, then another approach (usually more expensive in terms of performance) is to use a correlated subquery:
SELECT t.Date
, ( t.Units - (SELECT s.Units
FROM mytable s
WHERE s.Date < t.Date
ORDER BY s.Date DESC
LIMIT 1)
) AS Units_used
FROM mytable t
ORDER BY t.Date, t.Units
spencer7593's solution will be faster, but you can also do something like this...
SELECT * FROM rolling;
+----+-------+
| id | units |
+----+-------+
| 1 | 101 |
| 2 | 111 |
| 3 | 121 |
| 4 | 140 |
| 5 | 150 |
| 6 | 155 |
| 7 | 170 |
| 8 | 180 |
| 9 | 185 |
| 10 | 200 |
+----+-------+
SELECT a.id,COALESCE(a.units - b.units,a.units) units
FROM
( SELECT x.*
, COUNT(*) rank
FROM rolling x
JOIN rolling y
ON y.id <= x.id
GROUP
BY x.id
) a
LEFT
JOIN
( SELECT x.*
, COUNT(*) rank
FROM rolling x
JOIN rolling y
ON y.id <= x.id
GROUP
BY x.id
) b
ON b.rank= a.rank -1;
+----+-------+
| id | units |
+----+-------+
| 1 | 101 |
| 2 | 10 |
| 3 | 10 |
| 4 | 19 |
| 5 | 10 |
| 6 | 5 |
| 7 | 15 |
| 8 | 10 |
| 9 | 5 |
| 10 | 15 |
+----+-------+
This should give the desired result. I don't know how your table is called so I named it "tbltest".
Naming a table date is generally a bad idea as it also refers to other things (functions, data types,...) so I renamed it "fdate". Using uppercase characters in field names or tablenames is also a bad idea as it makes your statements less database independent (some databases are case sensitive and some are not).
SELECT
A.fdate,
A.units - coalesce(B.units, 0) AS units
FROM
tbltest A left join tbltest B ON A.fdate = B.fdate + INTERVAL 1 DAY
Assuming that I have the below customer_offer table.
My question is:
How to select all the rows where the key(s) are duplicated in that table?
+---------+-------------+------------+----------+--------+---------------------+
| link_id | customer_id | partner_id | offer_id | key | date_updated |
+---------+-------------+------------+----------+--------+---------------------+
| 1 | 99 | 11 | 14 | mmmmmq | 2011-09-21 12:40:46 |
| 2 | 100 | 11 | 14 | qmmmmq | 2011-09-21 12:40:46 |
| 3 | 101 | 11 | 14 | 8mmmmq | 2011-09-21 12:40:46 |
| 4 | 99 | 11 | 14 | Dmmmmq | 2011-09-21 12:59:28 |
| 5 | 100 | 11 | 14 | Nmmmmq | 2011-09-21 12:59:28 |
+---------+-------------+------------+----------+--------+---------------------+
UPDATE:
Thanks so much for all your answer. There are many answers are good. Now I got the solution to do.
select *
from customer_offer
where key in
(select key from customer_offer group by key having count(*) > 1)
Update:
As mentioned from #Scorpi0, if with a big table, it is better to use join. And from mysql6.0 the new optimizer will convert this kind of subqueries into joins.
Self join
SELECT * FROM customer_offer c1 inner join customer_offer c2
on c1.key = c2.key
or group by the field then take when count > 1
SELECT COUNT(key),link_id FROM customer_offer c1
group by key, link_id
having COUNT(Key) > 1
SELECT DISTINCT c1.*
FROM customer_offer c1
INNER JOIN customer_offer c2
ON c1.key = c2.key
AND c1.link_id != c2.link_id
Assuming link_id is a primary key.
Use a sub-query to do the count check, and the main query to select the rows. The count check query is simply:
SELECT `link_id` FROM `customer_offer` GROUP BY `key` HAVING COUNT(`key`) > 1
Then the outer query will use this by joining into it:
SELECT customer_offer.* FROM customer_offer
INNER JOIN (SELECT `link_id` FROM `customer_offer` GROUP BY `key` HAVING COUNT(`key`) > 1) AS count_check
ON customer_offer.link_id = count_check.link_id
There are many threads on the mysql website which explains how to do this. This link will explain how to do this using mysql: http://forums.mysql.com/read.php?10,180556,180567#msg-180567
As a brief example the code below is from the link with a slight modification which better suits your example.
SELECT *
FROM tbl
GROUP BY key
HAVING COUNT(key)>1;
You can also use a joing which is my prefered method, as this removes the slower count method:
SELECT *
FROM this_table t
inner join this_table t1 on t.key = t1.key
SELECT link_id, key, count(key) as Occurrences
FROM table
GROUP BY key
HAVING COUNT(key)>1;