I have about 100K records that I have to run for the following query:
delete from users where name in #{String}
where the string could be 100K strings of this form: Joe,Kate etc.
For performance is it better to run the above statement or delete one record in a loop with one session.commit(); in the end?
EDITED
There could be only one record for each value
If you can create batches of queries to run then breaking it up into batches would most likely be the fastest:
delete from users where name in ('name1','name2','name3',.....'nameX');
delete from users where name in ('nameX+1','nameX+2','nameX+3',.....'nameX+X');
etc..
If you have the names in a table already you can just do this:
delete from users where name in (select name from table_with_names_to_be_deleted)
It would be better not to use such statements!
The query optimizer will have a field day parsing such a query. I suggest using some temporary table to join against, or some other WHERE-clause that the records to be deleted have in common?
Related
I have a huge table of mysqlwhich contains more than 33 million records .How I could compare my table to found non duplicate records , but unfortunately select statement doesn't work. Because it's huge table.
Please provide me a solution
First, Create a snapshot of your database or the tables you want to compare.
Optionally you can also limit the range of data you want to compare , for example only 3 years of data. This way your select query won't hog all the resources.
Snapshot will be bunch of files each representing a table containg your primary key or business key for each record ( I am assuming you can compare data based on aforementioned key . If thats not the case record all the field in your file)
Next, read each records from the file and do a select against the corresponding table. If there are more than 1 record you know it is a duplicate
Thanks
Look at the explain plan and see if what the DB is actually doing for the NOT IN.
You could try refactoring, with an index on subscriber as Roy suggested if necessary. I'm not familiar enough with MySQL to know whether the optimizer will execute these identically.
SELECT *
FROM contracts
WHERE NOT EXISTS
( SELECT 1
FROM edms
WHERE edms.subscriber=contracts.subscriber
);
-- or
SELECT C.*
FROM contracts AS C
LEFT
JOIN edms AS E
ON E.subscriber = C.subscriber
WHERE E.subscriber IS NULL;
Good morning.
I have a table on MySQL DataBase.
In this table there are 5 robots that can write like 10 record each per hour.
Every 3 month a script that I have created, make a copy of the table and then delete all the table entries (In this way I can keep the IDs in a certain order).
My question is.
That are two different statement:
CREATE TABLE omologationResult_{date} AS SELECT * FROM omologationResult
DELETE FROM omologationResult
if the script is going to copy the table at point 0, and a record will be added from the robots, there's no problem, because the SQL statement starts from the lowest ID 'till the end. But if the script is going to delete the table and the robot is writing in it. What will happen? I lose the last robot record?
And if it's true. What can I do to make a copy of the table and then remove only the data that I've copied?
Thank you
Yes, this is not a safe operation because it's not atomic. It's quite possible for another thread to insert values into that table in between your CREATE .. SELECT and the DELETE. One option you have is to use a multi table DELETE
CREATE TABLE omologationResult_{date} AS SELECT * FROM omologationResult;
DELETE omologationResult FROM omologationResult
INNER JOIN omologationResult_{date} ON omologationResult_{date}.id = omologationResult.id
Will ensure that only items that exist in both tables have been deleted from omologationResult
I have an sql query that finds and groups these duplicates using very complicated conditions:
SELECT right(post_url, LOCATE('-', REVERSE(post_url),LOCATE('-',REVERSE(post_url))+1) -1) as name,
left(post_name,LOCATE('-',post_url,LOCATE('-',post_url)+1) - 1) as city,
post_title as original,ID,post_name,count(*)
FROM table WHERE post_type='finder'
GROUP BY name,city having count(*) > 1
To explain the query, post_url is basically a url name, ending with the name of someone, e.g : new-jersey-something-something-donald-t
I go to the second dash from the right and get the name that way. Then I get the city/state which is in the second dash from the left. In this manner, I've successfully found the duplicates in this database-but I'm having trouble thinking of a way to isolate the duplicate and delete it. In addition, I only want to delete the copy that does not have %near% in post_url. my question is, using the query here, how would I change this to delete the duplicate?
You're not going to be able to do it in one query. That's because you need to write a query that looks something like this:
DELETE FROM table
WHERE id IN (SELECT ... FROM table WHERE ...)
MySQL specifically prohibits this. You can't delete based on a subquery that references the same table. You also can't rewrite this query using JOINs.
There is an easy solution, though: use a temporary table and two queries.
-- build the list of IDs to delete
CREATE TEMPORARY TABLE temp
SELECT ... FROM table WHERE ...
-- now delete those items
DELETE FROM table
WHERE id IN (SELECT id FROM temp);
You can improve performance with JOINs and indexes.
The key to "isolating" the duplicates is to ensure that every item you want to delete has a primary key - that way you can easily build a list of IDs to delete. If your table don't have primary keys, you are reduced to doing WHERE clauses and JOINs on multiple columns - that gets messy very quickly.
I'm trying update rows in a table (called "users") based on values found in another table (called "users_temp"). I have the query:
UPDATE users
INNER JOIN users_temp on users_temp.email = users.email
SET users.list_id = users_temp.list_id
Both tables have about 50k rows and this query basically just crashes MySQL.
Can someone suggest a better way of doing this or tell me why MySQL is crashing?
Thanks!
Your query is ok. To boost the performance, try indexing the "email" column in both tables before you run the query.
Without the indexes, it has to scan the entire 50k rows in both tables (so total of 100k rows) to find the matches.
I have this query that works fine. Its deletes records that are old based on current time.
$cleanacc_1 = "DELETE FROM $acc_1
WHERE `Scheduled` < DATE_SUB(UTC_TIMESTAMP(), INTERVAL 30 SECOND)";
$result = mysql_query($cleanacc_1);
However, there are over 100 tables (accounts) that need deleting and I was wondering if I can combine them into one query. If possible how?
This implies you create a new table for every account. Why are you not creating a record for each account within a single table?
For example...
create table account (id int unsigned primary key auto_increment, other fields...);
If you alter your table structure you will be able to delete individual account records with a single query...
delete from account where condition=true;
Individual transaction records for each account are then stored in another table and contain the account id they relate to...
create table transaction (id, account_id, other transaction fields);
If you don't change the database design you'll need to write PHP code that loops through each table and runs your delete query. This is very inefficient and I urge you to redesign the table as suggested.
If you don't understand why my table redsign suggestion is a better approach, post more information about your database and I'll explain in more detail with a working example.
No way to do that, AFAIK; anyways, I don't think it would be a big problem to run 100 queries, assuming you are not running that for each request or so..
Are you expecting performance issues? If that's the case, I'd probably use a cron job to run that query every X minutes..
You could setup a view of the tables and do then run the delete sql against the view. That should delete the underlying table data as well. Your table schema and permissions could have an affect whether this will work or not. Check out this answer, it might help as well.
Does deleting row from view delete row from base table - MYsql?
Please consider the following example.
I have three tables in following structure.
Table names : t1,t2,t3
Fields : Id, name
Im going to perform delete query with one condition which recode id must less than 10.
DELETE FROM t1, t2,t3 USING t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.id<10 and t2.id<10 and t3.id<10.
The query has been successfully executed ( MySql ). I got the expected output.
So please try the same way with your condition.