I need to delete duplicates from a table while keeping one item. Since I cannot delete from a table while accessing this very same table in a subquery inside the where-statement, i decided to store my affected IDs in a temporary table:
create temporary my_temp_table (
id int not null
) engine memory;
and then insert IDs using a select:
insert into my_temp_table
-- select query works
select
id
from
-- innodb
table_with_duplicates
where
hash_code in (
select
hash_code
from
table_with_duplicates
group by
hash_code
having
count(id) > 1
)
and date_created < '2015-01-01'
;
Later I want to use these IDs to delete them:
delete from table_with_duplicates
where id in (
select id from my_temp_table
)
;
Just executing the select part of the insert statement works fine. Adding the insert part, however, causes 1 CPU core to go up to 100% and the query seems to never end. Nothing is inserted.
On my dev environment, table_with_duplicates contains about 20000 rows with 1 duplicate.
Any ideas?
EDIT:
Thank you for your answers. I have tried a select distinct(... approach, which didn't help much. Maybe I've used in the wrong place/subselect. Played around with it quite a lot already. To clarify, I have something like this:
ID date_created hash_code
1 2013-06-06 ABCDEFGH <-- delete this one
2 2013-08-08 HGFEDCBA
3 2015-11-11 ABCDEFGH <-- keep this one
Well, I chose different approach to solve this problem. I've written a small PHP command-line script that fetches all my IDs into an array and then executes a DELETE, using all IDs as ....
"delete from table_with_duplicates where id in (".explode(',' $arrIDs).")";
Having about 9000 affected rows and this (hopefully) being a one-time action, this worked for me.
I also played around with
SET SESSION optimizer_search_depth = 1;
but had no luck either.
Related
I am a Linux admin with only basic knowledge in Mysql Queries
I want to delete many table entries which are ip address from my table using id,
currently i am using
DELETE from tablename where id=1;
DELETE from tablename where id=2;
but i have to delete 254 entries,so this method is going to take hours,how can i tell mysql to delete rows that i specify,coz i want to skip deleting some entries out of this 254.
Deleting whole table and importing needed entries is not an option.
The best way is to use IN statement :
DELETE from tablename WHERE id IN (1,2,3,...,254);
You can also use BETWEEN if you have consecutive IDs :
DELETE from tablename WHERE id BETWEEN 1 AND 254;
You can of course limit for some IDs using other WHERE clause :
DELETE from tablename WHERE id BETWEEN 1 AND 254 AND id<>10;
how about using IN
DELETE FROM tableName
WHERE ID IN (1,2) -- add as many ID as you want.
if you need to keep only a few rows, consider
DELETE FROM tablename WHERE id NOT IN (5,124,221);
This will keep only some records and discard others.
Something like this might make it a bit easier, you could obviously use a script to generate this, or even excel
DELETE FROM tablename WHERE id IN (
1,
2,
3,
4,
5,
6
);
If you have some 'condition' in your data to figure out the 254 ids, you could use:
delete from tablename
where id in
(select id from tablename where <your-condition>)
or simply:
delete from tablename where <your-condition>
Simply hard coding the 254 values of id column would be very tough in any case.
Others have suggested IN, this is fine. You can also use a range:
DELETE from tablename where id<254 and id>3;
If the ids to delete are contiguous.
Use IN Clause
DELETE from tablename where id IN (1,2);
OR you can merge the use of BETWEEN and NOT IN to decrease the numbers you have to mention.
DELETE from tablename
where (id BETWEEN 1 AND 255)
AND (id NOT IN (254));
DELETE FROM table_name WHERE id BETWEEN 1 AND 256;
Try This.
Hope it helps:
DELETE FROM tablename
WHERE tablename.id = ANY (SELECT id FROM tablename WHERE id = id);
DELETE FROM tablename WHERE id > 0;
OR
DELETE FROM tablename WHERE id <255;
It deletes id from 1 to 254
I'm pretty stuck on a Mysql query.
I have a table with three columns;
user_id | person_id | score.
The table is going to be used to store top 5 highscores for each person.
I need at query that checks if there is less than five rows for a specific person.
Is there is less, insert new row. But if there is five rows I have to replace the lowest score with the new one.
It is for a webservice written in PHP and the data about the new score is posted to the method as params.
Been stuck for some hours now — is it even possible to make this happen in one query ?
You can use stored procedure in mysql. I dont know the names of the tables but if you look closer you will understand how it works.
DELIMITER $$
DROP PROCEDURE IF EXISTS test $$
CREATE PROCEDURE test( IN testparam VARCHAR(22) )
BEGIN
DECLARE count INT(11);
SET count = (SELECT COUNT(*) FROM persons );
IF count < 5 THEN
insert into table_needed_for_insert values(testparam);
ELSE
update table_needed_for_insert where score=(select min(score) from table_needed_for_insert);
END IF;
select * from table_needed_for_insert
END $$
DELIMITER;
And how to execute this thing CALL test(1); 1 is the parameter, you can create as many as you need.
And from php you can call directly as like
$result = mysql_query("call test(".$param.")");
And here you can check a tutorial on mysql stored procedures:
http://www.mysqltutorial.org/mysql-stored-procedure-tutorial.aspx
It might be possible if you have a unique key which identifies the lowest score. Then you could use the
INSERT ... ON DUPLICATE KEY construct. But you would have to install a trigger which keeps explicit track of the lowest score.
I would propose this scenario (I have not tried it, it is just an idea):
as I understand, you only need 5 ids. you can run a subqueries like these
SELECT MAX(id) AS last_id FROM table
SELECT MIN(score), id AS lowest_id FROM table
then
insert or replace into table (id, ...) values ( MIN(last_id+1, lowest_id), ... )
there are possible mistakes and also only one subquery is possible, but I hope you get the main idea
The simplest way imo is to insert data,
INSERT INTO top_scores (user_id, person_id, score_id) VALUES (1,2,3)
then delete inappropriate rows
DELETE top_scores FROM top_scores
INNER JOIN
(SELECT * FROM top_scores WHERE person_id = 2 ORDER BY score ASC LIMIT 5, 1000000) AS inappropriate_rows
USING (user_id, person_id, score)
Hi i have a mysql table without primary key and i need to delete the duplicated rows. how can i do so?
user_id category_id
1 2
1 3
1 4
1 2
2 2
2 3
2 2
CREATE TABLE temp SELECT DISTINCT * FROM tablename;
ALTER TABLE tablename RENAME junk;
ALTER TABLE temp RENAME tablename;
Since you cannot differentiate 2 identical rows, you cannot delete just one of them. The way you need to think about it is like this:
insert into new_better_table
select user_id, category_id from old_table group by user_id, category_id
It's possible to use a dirty flag with default value 1 and copy only single records with the flag set to 0, then simply remove all dirty records. This way you don't need another table.
Assuming you already created a dirty flag with default value 1:
insert into mytable
select fld1,fld2,fldN,0 as dirty
from mytable
group by duplicate_field
Then you can just delete dirty records:
delete from mytable where dirty = 1
Don't forget to remove the dirty flag. You're done.
The select distinct * solution proposed in another answer will not work in cases where your rows contain null values. A better solution that also prevents future duplicates from appearing in my_table in the first place is as follows:
create table no_dupes like my_table;
alter table no_dupes add unique index(unique_column,or_columns);
insert ignore into no_dupes select * from my_table;
rename table my_table to junk, no_dupes to my_table;
First, here's the concise summary of the question:
Is it possible to run an INSERT statement conditionally?
Something akin to this:
IF(expression) INSERT...
Now, I know I can do this with a stored procedure.
My question is: can I do this in my query?
Now, why would I want to do that?
Let's assume we have the following 2 tables:
products: id, qty_on_hand
orders: id, product_id, qty
Now, let's say an order for 20 Voodoo Dolls (product id 2) comes in.
We first check if there's enough Quantity On Hand:
SELECT IF(
( SELECT SUM(qty) FROM orders WHERE product_id = 2 ) + 20
<=
( SELECT qty_on_hand FROM products WHERE id = 2)
, 'true', 'false');
Then, if it evaluates to true, we run an INSERT query.
So far so good.
However, there's a problem with concurrency.
If 2 orders come in at the exact same time, they might both read the quantity-on-hand before any one of them has entered the order.
They'll then both place the order, thus exceeding the qty_on_hand.
So, back to the root of the question:
Is it possible to run an INSERT statement conditionally, so that we can combine both these queries into one?
I searched around a lot, and the only type of conditional INSERT statement that I could find was ON DUPLICATE KEY, which obviously does not apply here.
INSERT INTO TABLE
SELECT value_for_column1, value_for_column2, ...
FROM wherever
WHERE your_special_condition
If no rows are returned from the select (because your special condition is false) no insert happens.
Using your schema from question (assuming your id column is auto_increment):
insert into orders (product_id, qty)
select 2, 20
where (SELECT qty_on_hand FROM products WHERE id = 2) > 20;
This will insert no rows if there's not enough stock on hand, otherwise it will create the order row.
Nice idea btw!
Try:
INSERT INTO orders(product_id, qty)
SELECT 2, 20 FROM products WHERE id = 2 AND qty_on_hand >= 20
If a product with id equal to 2 exists and the qty_on_hand is greater or equal to 20 for this product, then an insert will occur with the values product_id = 2, and qty = 20. Otherwise, no insert will occur.
Note: If your product ids are note unique, you might want to add a LIMIT clause at the end of the SELECT statement.
Not sure about concurrency, you'll need to read up on locking in mysql, but this will let you be sure that you only take 20 items if 20 items are available:
update products
set qty_on_hand = qty_on_hand - 20
where qty_on_hand >= 20
and id=2
You can then check how many rows were affected. If none were affected, you did not have enough stock. If 1 row was affected, you have effectively consumed the stock.
You're probably solving the problem the wrong way.
If you're afraid two read-operations will occur at the same time and thus one will work with stale data, the solution is to use locks or transactions.
Have the query do this:
lock table for read
read table
update table
release lock
I wanted to insert into a table using values so I found this solution to insert the values using the IF condition
DELIMITER $$
CREATE PROCEDURE insertIssue()
BEGIN
IF (1 NOT IN (select I.issue_number from issue as I where I.series_id = 1)) THEN
INSERT IGNORE INTO issue ( issue_number, month_published, year_published, series_id, mcs_issue_id) VALUES (1, 1, 1990, 1, 1);
END IF;
END$$
DELIMITER ;
If you later on want to call the procedure it's as simple as
CALL insertIssue()
You can find more information about PROCEDURES and if conditions in this site
I'm having trouble getting an INSERT query to execute properly, and I can't seem to find anything on Google or Stack Overflow that solves this particular issue.
I'm trying to create a simple table for featured entries, where the entry_id is saved to the table along with it's current order.
My desired output is this:
If the featured table currently has these three entries:
featured_id entry_id featured_order
1 27 0
2 54 1
4 23 2
I want the next entry to save with featured_order=3.
I'm trying to get the following query to work with no luck:
INSERT INTO `featured`
(
`entry_id`, `featured_order`
)
VALUES
(
200,
(SELECT COUNT(*) AS `the_count` FROM `featured`)
)
The error I'm getting is: You can't specify target table 'featured' for update in FROM clause.
Can anyone help with a solution that gets the count without causing an error?
Thanks in advance!
Here is a cool thing: MySQL's INSERT . . . SELECT:
INSERT INTO `featured`
(
`entry_id`, `featured_order`
)
SELECT 200, COUNT(*) + 1
FROM `featured`
No subquery required.
#Bohemian has a good point:
Better to use max(featured_order) + 1 if you use this approach
So a better query would probably be:
INSERT INTO `featured`
(
`entry_id`, `featured_order`
)
SELECT 200, MAX(`featured_order`) + 1
FROM `featured`
His trigger method describe in his answer is also a good way to accomplish what you want.
The potential problem with query 1 is if you ever delete a row the rank will be thrown off, and you'll have a duplicate in featured_order. With the second query this is not a problem, but you will have gaps, just as if you were using an auto-increment column.
If you absolutely must have an order with no gaps the best solution I know of is to run this series of queries:
SET #pos:=0;
DROP TABLE IF EXISTS temp1;
CREATE TEMPORARY TABLE temp1 LIKE featured;
ALTER TABLE featured ORDER BY featured_order ASC;
INSERT INTO temp1 (featured_id, entry_id, featured_order)
SELECT featured_id, entry_id, #pos:=#pos+1 FROM words;
UPDATE featured
JOIN temp1 ON featured.featured_id = temp1.featured_id
SET featured.rank = temp1.rank;
DROP TABLE temp1;
Whenever you delete a row
Use a trigger:
drop trigger if exists featured_insert_trigger;
delimiter //
create trigger featured_insert_trigger before insert on featured
for each row
begin
set new.featured_order = ifnull((select max(featured_order) from featured), -1) + 1;
end; //
delimiter ;
Now your inserts look like this:
insert into featured (entry_id) values (200);
featured_order will be set to the highest featured_order value plus one. This caters for rows being deleted/updated and always guarantee uniqueness.
The ifnull is there in case there are no rows in the table, in which case the first value will be zero.
This code has been tested as works correctly.
INSERT INTO `featured`
(
`entry_id`, `featured_order`
)
VALUES
(
200,
(SELECT COUNT(*) AS `the_count` FROM `featured` F1)
)
Correction is just adding "F1" table alias.
This standard sql solution works fine on various dbms (not only mysql)
I also suggest an improvement over:
SELECT COUNT(*) +1 (Problem: if some row gets deleted you may collide with existing index)
SELECT MAX(featured_order)+1 (Problem: the first insert with empty table gets error)
SELECT (COALESCE(MAX(featured_order), 0)+1) (no Problem)
You have to simpley use alias that will solve the problem :
INSERT INTO `featured`
(
`entry_id`, `featured_order`
)
VALUES
(
200,
(SELECT COUNT(*) AS `the_count` FROM `featured` as f1)
)
From the MySQL manual regarding subqueries:
Another restriction is that currently you cannot modify a table and select from the same table in a subquery.
Perhaps an alias or a join (otherwise useless) in the subquery would help here.
EDIT: It turns out that there's a work-around. The work-around is described http://www.xaprb.com/blog/2006/06/23/how-to-select-from-an-update-target-in-mysql/.