I have long list of rows with random values:
| id | value |
|----|-------|
| 1 | abcd |
| 2 | qwer |
| 3 | jklm |
| 4 | yxcv |
| 5 | tzui |
Then I have an array of few values:
array('qwer', 'jklm');
And I need to know, if this sequence of values from array already exists in table in given order. In this case the sequence of values exists.
I tried to concat all values from table and array and match two strings, which works great with few rows but there are actually hundred of thousand of rows in table. I believe there should be a better solution.
If your list is short, you could just do a self-join and spell out the conditions for each joined table reference:
select t1.id from MyTable as t1 join MyTable as t2
where t1.value='qwer' and t2.value='jklm' and t1.id=t2.id-1;
This returns an empty set if there's no such sequence. And of course it assumes that the id numbers are consecutive (they are in your example, but in general that's a risky assumption).
This doesn't work well if your list gets really long. There's a hard limit of 63 table references MySQL supports in a single query.
Here's another solution, which works for any size list, but only if your id values are known to be consecutive:
select t1.id from MyTable as t1 join MyTable as t2
on t2.id between t1.id and t1.id+1
where t1.value = 'qwer' and t2.value in ('qwer','jklm')
group by t1.id
having group_concat(t2.value order by t2.id) = 'qwer,jklm';
The t1 row is the beginning of the potential matching sequence of rows, so it must match the first value in your list.
Then join to the t2 rows, which are the complete set of potentially matching rows.
The set of t2 rows is also limited to a set no more than N rows, based on the size of your list of N values you're searching for. But SQL has no way of making a group based on the number of rows, we can only limit based on some value in the row. So that's why this works if your id values can be assumed to be consecutive.
This way you can do it for the whole set:
select value1, value2
from
(
select *
from (
SELECT [IMEPAC] value1 , ROW_NUMBER() over(order by [MATBR]) rn1
FROM [PACM]
) a1 join
(
SELECT [IMEPAC] value2 , ROW_NUMBER() over(order by [MATBR]) rn2
FROM [PACM]
) a2 on a1.rn1 = a2.rn2 + 1
) a
group by value1, value2
having count(*) > 1
It is written for MS SQL but you can easily rewrite it to fit mysql too.
I run this against table with > 400000 rows on IMEPAC which is not part of any index and it run (first and only once) for 6 sec.
Here is Mysql version:
select value1, value2, count(*) count
from
(
select *
from (
SELECT #row_number1:= #row_number1 + 1 AS rn1, content as value1
FROM docs,(SELECT #row_number1:=0) AS t
order by id
) a1 join
(
SELECT #row_number2:= #row_number2 + 1 AS rn2, content value2
FROM docs,(SELECT #row_number2:=0) AS t
order by id
) a2 on a1.rn1 = a2.rn2 + 1
) a
group by value1, value2
having count(*) > 1;
SQL Fiddle here
I have a MySQL table like this
id Name count
1 ABC 1
2 CDF 3
3 FGH 4
using simply select query I get the values as
1 ABC 1
2 CDF 3
3 FGH 4
How I can get the result like this
1 ABC 1
2 CDF 3
3 FGH 4
4 NULL 0
You can see Last row. When Records are finished an extra row in this format
last_id+1, Null ,0 should be added. You can see above. Even I have no such row in my original table. There may be N rows not fixed 3,4
The answer is very simple
select (select max(id) from mytable)+1 as id, NULL as Name, 0 as count union all select id,Name,count from mytable;
This looks a little messy but it should work.
SELECT a.id, b.name, coalesce(b.`count`) as `count`
FROM
(
SELECT 1 as ID
UNION
SELECT 2 as ID
UNION
SELECT 3 as ID
UNION
SELECT 4 as ID
) a LEFT JOIN table1 b
ON a.id = b.id
WHERE a.ID IN (1,2,3,4)
UPDATE 1
You could simply generate a table that have 1 column preferably with name (ID) that has records maybe up 10,000 or more. Then you could simply join it with your table that has the original record. For Example, assuming that you have a table named DummyRecord with 1 column and has 10,000 rows on it
SELECT a.id, b.name, coalesce(b.`count`) as `count`
FROM DummyRecord a LEFT JOIN table1 b
ON a.id = b.id
WHERE a.ID >= 1 AND
a.ID <= 4
that's it. Or if you want to have from 10 to 100, then you could use this condition
...
WHERE a.ID >= 10 AND
a.ID <= 100
To clarify this is how one can append an extra row to the result set
select * from table union select 123 as id,'abc' as name
results
id | name
------------
*** | ***
*** | ***
123 | abc
Simply use mysql ROLLUP.
SELECT * FROM your_table
GROUP BY Name WITH ROLLUP;
select
x.id,
t.name,
ifnull(t.count, 0) as count
from
(SELECT 1 AS id
-- Part of the query below, you will need to generate dynamically,
-- just as you would otherwise need to generate 'in (1,2,3,4)'
UNION ALL SELECT 2
UNION ALL SELECT 3
UNION ALL SELECT 4
UNION ALL SELECT 5
) x
LEFT JOIN YourTable t
ON t.id = x.id
If the id does not exist in the table you're selecting from, you'll need to LEFT JOIN against a list of every id you want returned - this way, it will return the null values for ones that don't exist and the true values for those that do.
I would suggest creating a numbers table that is a single-columned table filled with numbers:
CREATE TABLE `numbers` (
id int(11) unsigned NOT NULL
);
And then inserting a large amount of numbers, starting at 1 and going up to what you think the highest id you'll ever see plus a thousand or so. Maybe go from 1 to 1000000 to be on the safe side. Regardless, you just need to make sure it's more-than-high enough to cover any possible id you'll run into.
After that, your query can look like:
SELECT n.id, a.*
FROM
`numbers` n
LEFT JOIN table t
ON t.id = n.id
WHERE n.id IN (1,2,3,4);
This solution will allow for a dynamically growing list of ids without the need for a sub-query with a list of unions; though, the other solutions provided will equally work for a small known list too (and could also be dynamically generated).
I googled a bit and looked on SO but I didn't find anything that helped me.
I have a working MySQL query that selects some columns (accross three tables, with two JOIN statements) and I am looking to do something extra on the result set.
I would like to SELECT all rows from the 3 most recent groups. (I can only assume I have to use a GROUP BY on that column) I'm having a hard time explaining this clearly so I'll use an example:
id | group
--------------
1 | 1
2 | 2
3 | 2
4 | 2
5 | 3
6 | 3
7 | 4
8 | 4
Of course, I dumbed it down a lot for the sake of simplicity (and my current query doesn't include an id column).
Right now my ideal query would return, in order (that's the id field):
8, 7, 6, 5, 4, 3, 2
If I were to add the following 9th element:
id | group
--------------
9 | 5
My ideal query would then return, in order:
9, 8, 7, 6, 5
Because these are all the rows from the most 3 recent groups. Also, when two rows have the same group (and are still in the results set), I would like to ORDER them BY another field (which I have not included in my dumbed down example).
In my search I only found how to do actions on elements of GROUPS (MAX of each, AVG of group elements, etc.) and not GROUPS themselves (first 3 groups ordered by a field).
Thank you in advance for your help!
Edit: Here is what my real query looks like.
SELECT t1.f1, t1.f2, t2.f1, t2.f2, t2.f3, t3.f1, t3.f2, t3.f3, t3.f4
FROM t1
LEFT JOIN t2 ON t2.f1=t1.f3
LEFT JOIN t3 ON t2.f1=t3.f5
WHERE t1.f4='some_constant' AND t2.f4='some_other_constant'
ORDER BY t1.f2 DESC
SELECT `table`.* FROM
(SELECT DISTINCT `group`
FROM `table`
ORDER BY `group` DESC LIMIT 3) t1
INNER JOIN `table` ON `table`.`group` = t1.`group`
the subquery should return the three groups with the largest value, the INNER JOIN will ensure no rows are included which do not have these group values.
assuming t1.f2 is your group column:
SELECT a,b,c,d,e,f,g,h,i
FROM
(
SELECT t1.f1 as a, t1.f2 as b, t2.f1 as c, t2.f2 as d, t2.f3 as e, t3.f1 as f, t3.f2 as g, t3.f3 as h, t3.f4 as i
FROM t1
LEFT JOIN t2 ON t2.f1=t1.f3
LEFT JOIN t3 ON t2.f1=t3.f5
WHERE t1.f4='some_constant' AND t2.f4='some_other_constant'
ORDER BY t1.f2 DESC
) first_table
INNER JOIN
(
SELECT DISTINCT `f2`
FROM `t1`
ORDER BY `f2` DESC LIMIT 3
) second_table
ON first_table.b = second_table.f2
Note that this may be very inefficient depending on your table structure, but is the best I can do without more information.
how about this way... (i use groupId instead of 'group'
[QUERY] => something like (SELECT id, groupId from tables.....) (your query with 2 joins).
-- with this query you have the last thre groups.
[QUERY2] => SELECT distinct(groupId) as groupId FROM ([QUERY]) ORDER BY groupId DESC LIMIT 0,3
and finally you will have:
SELECT id, groupId from tables----...... WHERE groupId in ([QUERY2]) order by groupId DESC, id DESC
We have a database with a table whose values were imported from another system. There is an auto-increment column, and there aren’t any duplicate values, but there are missing values. For example, running this query:
select count(id) from arrc_vouchers where id between 1 and 100
should return 100, but it returns 87 instead. Is there a query I can run that will return the values of the missing numbers? For example, the records may exist for id 1-70 and 83-100, but there aren’t any records with id's of 71-82. I want to return 71, 72, 73, etc.
Is this possible?
A better answer
JustPlainMJS provided a much better answer in terms of performance.
The (not as fast as possible) answer
Here's a version that works on a table of any size (not just on 100 rows):
SELECT (t1.id + 1) as gap_starts_at,
(SELECT MIN(t3.id) -1 FROM arrc_vouchers t3 WHERE t3.id > t1.id) as gap_ends_at
FROM arrc_vouchers t1
WHERE NOT EXISTS (SELECT t2.id FROM arrc_vouchers t2 WHERE t2.id = t1.id + 1)
HAVING gap_ends_at IS NOT NULL
gap_starts_at - first id in current gap
gap_ends_at - last id in current gap
This just worked for me to find the gaps in a table with more than 80k rows:
SELECT
CONCAT(z.expected, IF(z.got-1>z.expected, CONCAT(' thru ',z.got-1), '')) AS missing
FROM (
SELECT
#rownum:=#rownum+1 AS expected,
IF(#rownum=YourCol, 0, #rownum:=YourCol) AS got
FROM
(SELECT #rownum:=0) AS a
JOIN YourTable
ORDER BY YourCol
) AS z
WHERE z.got!=0;
Result:
+------------------+
| missing |
+------------------+
| 1 thru 99 |
| 666 thru 667 |
| 50000 |
| 66419 thru 66456 |
+------------------+
4 rows in set (0.06 sec)
Note that the order of columns expected and got is critical.
If you know that YourCol doesn't start at 1 and that doesn't matter, you can replace
(SELECT #rownum:=0) AS a
with
(SELECT #rownum:=(SELECT MIN(YourCol)-1 FROM YourTable)) AS a
New result:
+------------------+
| missing |
+------------------+
| 666 thru 667 |
| 50000 |
| 66419 thru 66456 |
+------------------+
3 rows in set (0.06 sec)
If you need to perform some kind of shell script task on the missing IDs, you can also use this variant in order to directly produce an expression you can iterate over in Bash.
SELECT GROUP_CONCAT(IF(z.got-1>z.expected, CONCAT('$(',z.expected,' ',z.got-1,')'), z.expected) SEPARATOR " ") AS missing
FROM ( SELECT #rownum:=#rownum+1 AS expected, IF(#rownum=height, 0, #rownum:=height) AS got FROM (SELECT #rownum:=0) AS a JOIN block ORDER BY height ) AS z WHERE z.got!=0;
This produces an output like so
$(seq 1 99) $(seq 666 667) 50000 $(seq 66419 66456)
You can then copy and paste it into a for loop in a bash terminal to execute a command for every ID
for ID in $(seq 1 99) $(seq 666 667) 50000 $(seq 66419 66456); do
echo $ID
# Fill the gaps
done
It's the same thing as above, only that it's both readable and executable. By changing the "CONCAT" command above, syntax can be generated for other programming languages. Or maybe even SQL.
A quick-and-dirty query that should do the trick:
SELECT a AS id, b AS next_id, (b - a) -1 AS missing_inbetween
FROM
(
SELECT a1.id AS a , MIN(a2.id) AS b
FROM arrc_vouchers AS a1
LEFT JOIN arrc_vouchers AS a2 ON a2.id > a1.id
WHERE a1.id <= 100
GROUP BY a1.id
) AS tab
WHERE
b > a + 1
This will give you a table showing the id that has ids missing above it, and next_id that exists, and how many are missing between... E.g.,
id next_id missing_inbetween
1 4 2
68 70 1
75 87 11
If you are using a MariaDB database, you have a faster (800%) option using the sequence storage engine:
SELECT * FROM seq_1_to_50000 WHERE SEQ NOT IN (SELECT COL FROM TABLE);
If there is a sequence having gap of maximum one between two numbers (like
1,3,5,6) then the query that can be used is:
select s.id+1 from source1 s where s.id+1 not in(select id from source1) and s.id+1<(select max(id) from source1);
table_name - source1
column_name - id
An alternative solution that requires a query + some code doing some processing would be:
select l.id lValue, c.id cValue, r.id rValue
from
arrc_vouchers l
right join arrc_vouchers c on l.id=IF(c.id > 0, c.id-1, null)
left join arrc_vouchers r on r.id=c.id+1
where 1=1
and c.id > 0
and (l.id is null or r.id is null)
order by c.id asc;
Note that the query does not contain any subselect that we know it's not handled performantly by MySQL's planner.
That will return one entry per centralValue (cValue) that does not have a smaller value (lValue) or a greater value (rValue), i.e.:
lValue |cValue|rValue
-------+------+-------
{null} | 2 | 3
8 | 9 | {null}
{null} | 22 | 23
23 | 24 | {null}
{null} | 29 | {null}
{null} | 33 | {null}
Without going into further details (we'll see them in next paragraphs) this output means that:
No values between 0 and 2
No values between 9 and 22
No values between 24 and 29
No values between 29 and 33
No values between 33 and MAX VALUE
So the basic idea is to do a RIGHT and LEFT joins with the same table seeing if we have adjacents values per value (i.e., if central value is '3' then we check for 3-1=2 at left and 3+1 at right), and when a ROW has a NULL value at RIGHT or LEFT then we know there is no adjacent value.
The complete raw output of my table is:
select * from arrc_vouchers order by id asc;
0
2
3
4
5
6
7
8
9
22
23
24
29
33
Some notes:
The SQL IF statement in the join condition is needed if you define the 'id' field as UNSIGNED, therefore it will not allow you to decrease it under zero. This is not strictly necessary if you keep the c.value > 0 as it's stated in the next note, but I'm including it just as doc.
I'm filtering the zero central value as we are not interested in any previous value and we can derive the post value from the next row.
I tried it in a different manner, and the best performance that I found was this simple query:
select a.id+1 gapIni
,(select x.id-1 from arrc_vouchers x where x.id>a.id+1 limit 1) gapEnd
from arrc_vouchers a
left join arrc_vouchers b on b.id=a.id+1
where b.id is null
order by 1
;
... one left join to check if the next id exists, only if next if is not found, then the subquery finds the next id that exists to find the end of gap. I did it because the query with equal (=) is better performance than the greater than (>) operator.
Using the sqlfiddle it does not show so a different performance compared to the other queries, but in a real database this query above results in 3 times faster than the others.
The schema:
CREATE TABLE arrc_vouchers (id int primary key)
;
INSERT INTO `arrc_vouchers` (`id`) VALUES (1),(4),(5),(7),(8),(9),(10),(11),(15),(16),(17),(18),(19),(20),(21),(22),(23),(24),(25),(26),(27),(28),(29)
;
Follow below all the queries that I made to compare the performance:
select a.id+1 gapIni
,(select x.id-1 from arrc_vouchers x where x.id>a.id+1 limit 1) gapEnd
from arrc_vouchers a
left join arrc_vouchers b on b.id=a.id+1
where b.id is null
order by 1
;
select *, (gapEnd-gapIni) qt
from (
select id+1 gapIni
,(select x.id from arrc_vouchers x where x.id>a.id limit 1) gapEnd
from arrc_vouchers a
order by id
) a where gapEnd <> gapIni
;
select id+1 gapIni
,(select x.id from arrc_vouchers x where x.id>a.id limit 1) gapEnd
#,coalesce((select id from arrc_vouchers x where x.id=a.id+1),(select x.id from arrc_vouchers x where x.id>a.id limit 1)) gapEnd
from arrc_vouchers a
where id+1 <> (select x.id from arrc_vouchers x where x.id>a.id limit 1)
order by id
;
select id+1 gapIni
,coalesce((select id from arrc_vouchers x where x.id=a.id+1),(select x.id from arrc_vouchers x where x.id>a.id limit 1)) gapEnd
from arrc_vouchers a
order by id
;
select id+1 gapIni
,coalesce((select id from arrc_vouchers x where x.id=a.id+1),concat('*** GAT *** ',(select x.id from arrc_vouchers x where x.id>a.id limit 1))) gapEnd
from arrc_vouchers a
order by id
;
You can see and test my query using this SQL Fiddle:
http://sqlfiddle.com/#!9/6bdca7/1
It is probably not relevant, but I was looking for something like this to list the gaps in a sequence of numbers and found this post that has multiple different solutions depending upon exactly what you are looking for. I was looking for the first available gap in the sequence (i.e., next available number), and this seems to work fine.
SELECT MIN(l.number_sequence + 1) as nextavabile
from patients as l
LEFT OUTER JOIN patients as r on l.number_sequence + 1 = r.number_sequence
WHERE r.number_sequence is NULL
Several other scenarios and solutions discussed there, from 2005!
How to Find Missing Values in a Sequence With SQL
Create a temporary table with 100 rows and a single column containing the values 1-100.
Outer Join this table to your arrc_vouchers table and select the single column values where the arrc_vouchers id is null.
This should work:
select tempid from temptable
left join arrc_vouchers on temptable.tempid = arrc_vouchers.id
where arrc_vouchers.id is null
Although these all seem to work, the result set returns in a very lengthy time when there are 50,000 records.
I used this, and it find the gap or the next available (last used + 1) with a much faster return from the query.
SELECT a.id as beforegap, a.id+1 as avail
FROM table_name a
where (select b.id from table_name b where b.id=a.id+1) is null
limit 1;
Based on the answer given by matt, this stored procedure allows you to specify the table and column names that you wish to test to find non-contiguous records - thus answering the original question and also demonstrating how one could use #var to represent tables &/or columns in a stored procedure.
create definer=`root`#`localhost` procedure `spfindnoncontiguous`(in `param_tbl` varchar(64), in `param_col` varchar(64))
language sql
not deterministic
contains sql
sql security definer
comment ''
begin
declare strsql varchar(1000);
declare tbl varchar(64);
declare col varchar(64);
set #tbl=cast(param_tbl as char character set utf8);
set #col=cast(param_col as char character set utf8);
set #strsql=concat("select
( t1.",#col," + 1 ) as starts_at,
( select min(t3.",#col,") -1 from ",#tbl," t3 where t3.",#col," > t1.",#col," ) as ends_at
from ",#tbl," t1
where not exists ( select t2.",#col," from ",#tbl," t2 where t2.",#col," = t1.",#col," + 1 )
having ends_at is not null");
prepare stmt from #strsql;
execute stmt;
deallocate prepare stmt;
end
A simple, yet effective, solution to find the missing auto-increment values:
SELECT `id`+1
FROM `table_name`
WHERE `id`+1 NOT IN (SELECT id FROM table_name)
Another simple answer that identifies the gaps. We do a query selecting just the odd numbers and we right join it to a query with all the even numbers. As long as you're not missing id 1; this should give you a comprehensive list of where the gaps start.
You'll still have to take a look at that place in the database to figure out how many numbers the gap is. I found this way easier than the solution proposed and much easier to customize to unique situations.
SELECT *
FROM (SELECT * FROM MyTABLE WHERE MYFIELD % 2 > 0) AS A
RIGHT JOIN FROM (SELECT * FROM MyTABLE WHERE MYFIELD % 2 = 0) AS B
ON A.MYFIELD=(B.MYFIELD+1)
WHERE a.id IS NULL;
This works for me:
SELECT distinct(l.membership_no + 1) as nextavabile
from Tablename as l
LEFT OUTER JOIN Tablename as r on l.membership_no + 1 = r.membership_no
WHERE r.membership_no is NULL and l.membership_no is not null order by nextavabile asc;
Starting from the comment posted by user933161,
select l.id + 1 as start from sequence as l inner join sequence as r on l.id + 1 = r.id where r.id is null;
is better in that it will not produce a false positive for the end of the list of records. (I'm not sure why so many are using left outer joins.)
Also,
insert into sequence (id) values (#);
where # is the start value for a gap will fill that start value. (If there are fields that cannot be null, you will have to add those with dummy values.)
You could alternate between querying for start values and filling in each start value until the query for start values returns an empty set.
Of course, this approach would only be helpful if you're working with a small enough data set that manually iterating like that is reasonable. I don't know enough about things like phpMyAdmin to come up with ways to automate it for larger sets with more and larger gaps.
CREATE TABLE arrc_vouchers (id int primary key);
INSERT INTO `arrc_vouchers` (`id`) VALUES (1),(4),(5),(7),(8),(9),(10),(11),(15),(16);
WITH RECURSIVE odd_num_cte (id) AS
(
SELECT (select min(id) from arrc_vouchers)
union all
SELECT id+1 from odd_num_cte where id <(SELECT max(id) from arrc_vouchers)
)
SELECT cte.id
from arrc_vouchers ar right outer join odd_num_cte cte on ar.id=cte.id
where ar.id is null;
I have a table with rows like id, length, time and some of them are duplicates, where length and time is the same in some rows. I want to delete all copies of the first row submitted.
id | length | time
01 | 255232 | 1242
02 | 255232 | 1242 <- Delete that one
I have this to show all duplicates in table.
SELECT idgarmin_track, length , time
FROM `80dage_garmin_track`
WHERE length in
( SELECT length
FROM `80dage_garmin_track`
GROUP
BY length
HAVING count(*) > 1 )
ORDER BY idgarmin_track, length, time LIMIT 0,500
DELETE FROM `80dage_garmin_track` t1
WHERE EXISTS (SELECT 1 from `80dage_garmin_track` t2
WHERE t1.Length = t2.Length
AND t1.Time = t2.Time
AND t1.idgarmin_track > t2.idgarmin_track)
If you can take your table offline for a period, then the simplest way is to build a new table containing the data you want and then drop the original table:
create table `80dage_garmin_track_un` like `80dage_garmin_track`;
insert into `80dage_garmin_track_un`
select min(idgarmin_track), length, time
group by length, time;
rename table `80dage_garmin_track` to old, `80dage_garmin_track_un` to `80dage_garmin_track`;
drop table old;
i have the same problem Holsteinkaa, i just use it like this:
delete from table where id in ( select * from (
SELECT id FROM table t1
WHERE EXISTS (SELECT 1 from table t2
WHERE t1.field = t2.field
AND t1.id > t2.id
)
) as tmp )
i was trying to put this like a comment to Michael Pakhantsov answer but i cant :/ sorry