Good morning.
I have a table on MySQL DataBase.
In this table there are 5 robots that can write like 10 record each per hour.
Every 3 month a script that I have created, make a copy of the table and then delete all the table entries (In this way I can keep the IDs in a certain order).
My question is.
That are two different statement:
CREATE TABLE omologationResult_{date} AS SELECT * FROM omologationResult
DELETE FROM omologationResult
if the script is going to copy the table at point 0, and a record will be added from the robots, there's no problem, because the SQL statement starts from the lowest ID 'till the end. But if the script is going to delete the table and the robot is writing in it. What will happen? I lose the last robot record?
And if it's true. What can I do to make a copy of the table and then remove only the data that I've copied?
Thank you
Yes, this is not a safe operation because it's not atomic. It's quite possible for another thread to insert values into that table in between your CREATE .. SELECT and the DELETE. One option you have is to use a multi table DELETE
CREATE TABLE omologationResult_{date} AS SELECT * FROM omologationResult;
DELETE omologationResult FROM omologationResult
INNER JOIN omologationResult_{date} ON omologationResult_{date}.id = omologationResult.id
Will ensure that only items that exist in both tables have been deleted from omologationResult
Related
My Python application generates a CSV file containing a few hundred unique records, one unique row per line. It runs hourly and very often the data remains the same from one run to another. If there are changes, then they are small, e.g.
one record removed.
a few new records added.
occasional update to an existing record.
Each record is just four simple fields (name, date, id, description), and there will be no more than 10,000 records by the time the project is at maximum, so it can all be contained in a single table.
What the best way to merge changes into the table?
A few approaches I'm considering are:
1) empty the table and re-populate on each run.
2) write the latest data to a staging table and run a DB job to merge the changes into the main table.
3) read the existing table data into my python script, collect the new data, find the differences, run multiple 'CRUD' operations to apply the changes one by one
Can anyone suggest a better way?
thanks
I would do this in the following way:
Load the new CSV file into a second table.
DELETE rows in the main table that are missing from the second table:
DELETE m FROM main_table AS m
LEFT OUTER JOIN new_table AS t ON m.id = t.id
WHERE t.id IS NULL;
Use INSERT ON DUPLICATE KEY UPDATE to update rows that need to be updated. This becomes a no-op on each row that already contains the same values.
INSERT INTO main_table (id, name, date, description)
SELECT id, name, date, description FROM new_table
ON DUPLICATE KEY UPDATE
name = VALUES(name), date = VALUES(date), description = VALUES(description);
Drop the second table once you're done with it.
This is assuming id is the primary key and you have no other UNIQUE KEY in the table.
Given a data set size of 10,000 rows, this should be quick enough to do it in one batch. Once the data set gets 10x larger, you may have to reconsider the solution, for example do batches of 10,000 rows at a time.
Could someone please explain (or point in right direction) how I would move multiple rows from one table to another and remove the row from the original table based on a set criteria?
I understand
INSERT INTO table2 SELECT * FROM table1
to copy the data from one table to another but I need to then remove the original. The reason being it has been suggested to speed up the querying of the table I should move all redundant data (ended, expired, products older than 3 months) from the main table to another one.
A bit of background, I have a table that holds products, some products have expired but the products still need to be accessible. There are about 50,000 products that have expired and 2,000 which are active. There is a status column (int 1 = active, 2 = expired etc) to determine what to show on the front end.
I guess this post is 2 questions:
Is there a better way to speed up the querying of the product table without removing expired items?
If not, how to move rows from one table to another
Many many thanks!
INSERT INTO table2 (column_name1, column_name2) SELECT column_name1,
column_name2 FROM table 1 WHERE (where clause here)
DELETE FROM table1 WHERE (where clause here)
Source for above: mysql move row between tables
50,000 records in the table really isn't that many. If you're having performance issues, I'd look at your queries and your indexes to help speed up performance. And since those expired records still need to be accessed, then it could be more difficult having multiple tables to maintain.
However, to move data from one table to another as you've asked, you just need to run 2 different statements. Assuming you want to move inactive products:
INSERT INTO ProductsBackup SELECT * FROM Products WHERE Status <> 1
DELETE FROM Products WHERE WHERE Status <> 1
If you have Identities on your columns, you might be better off specifying the column names. But assuming the ProductId is the Identity, then be careful moving those to a different table as you probably don't want to lose that original id as it may point to other tables.
Good luck.
the scripts i've been working with in SQL work with close to 40,000 records and i've noticed a huge increase in execution time for when i use an UPDATE command
in 2 tables that have like 10 fields in them each, INSERT executes quicker for both combined than this UPDATE command
UPADTE table1
INNER JOIN table2 ON table1.primarykey = table2.primarykey
SET table1.code = table2.code
really what the UPDATE is doing is copying the code from one table to another where the identical records exists, this is because table1 is a staging table between 2 databases while table2 is a possessing table to insert staging table's data across multiple tables, both tables have the same number of record which is about 40,000
now to me UPDATE should be executing a lot quicker, considering it's only connecting 2 identical tables and inserting data for 1 field it should be running quicker than 2 INSERTS where 40,000 records are being created over 10 fields (so in other words, inserting 800,000 pieces of data) and i'm running the queries in a SQL console windows to avoid php timeouts
is UPDATE somehow more resource hungry than INSERT and is there any way to get it to go faster (apart from changing the fact that i use a separate table for processing, the staging table updates frequently so i copy the data like a snapshot and work with that, the code field is NULL to begin with so i only copy over records with a NULL code meaning records where code is not NULLhave already been worked with)
Is that UPDATE command the actual SQL? Because you need a WHERE clause to avoid updating every record in the table...
Also, INSERT doesn't first need to find the record to update from 2 joined tables.
I have this query that works fine. Its deletes records that are old based on current time.
$cleanacc_1 = "DELETE FROM $acc_1
WHERE `Scheduled` < DATE_SUB(UTC_TIMESTAMP(), INTERVAL 30 SECOND)";
$result = mysql_query($cleanacc_1);
However, there are over 100 tables (accounts) that need deleting and I was wondering if I can combine them into one query. If possible how?
This implies you create a new table for every account. Why are you not creating a record for each account within a single table?
For example...
create table account (id int unsigned primary key auto_increment, other fields...);
If you alter your table structure you will be able to delete individual account records with a single query...
delete from account where condition=true;
Individual transaction records for each account are then stored in another table and contain the account id they relate to...
create table transaction (id, account_id, other transaction fields);
If you don't change the database design you'll need to write PHP code that loops through each table and runs your delete query. This is very inefficient and I urge you to redesign the table as suggested.
If you don't understand why my table redsign suggestion is a better approach, post more information about your database and I'll explain in more detail with a working example.
No way to do that, AFAIK; anyways, I don't think it would be a big problem to run 100 queries, assuming you are not running that for each request or so..
Are you expecting performance issues? If that's the case, I'd probably use a cron job to run that query every X minutes..
You could setup a view of the tables and do then run the delete sql against the view. That should delete the underlying table data as well. Your table schema and permissions could have an affect whether this will work or not. Check out this answer, it might help as well.
Does deleting row from view delete row from base table - MYsql?
Please consider the following example.
I have three tables in following structure.
Table names : t1,t2,t3
Fields : Id, name
Im going to perform delete query with one condition which recode id must less than 10.
DELETE FROM t1, t2,t3 USING t1 INNER JOIN t2 INNER JOIN t3 WHERE t1.id<10 and t2.id<10 and t3.id<10.
The query has been successfully executed ( MySql ). I got the expected output.
So please try the same way with your condition.
I have a table containing about 500 000 rows. Once a day, I will try to synchronize this table with an external API. Most of the times, there are few- or no changes made since last update. My question is basically how should I construct my MySQL query for best performance? I have thought about using insert ignore, but it doesn't feel like the best way to go since only a few rows will be inserted and MySQL must loop through all rows in the table. I have also thought about using LOAD_DATA_INFILE to insert all rows in a temporary table and then select the rows not already in my original table, and then remove the temporary table. Maybe someone else has a better suggestion?
Thank you in advance!
I usually use a temporary table and the LOAD DATA INFILE bulk loader. The bulk loader is much more efficient that trying to insert records using a dynamically created query.
If you index your permanent tables with appropriate unique keys that relate to the keys in the API then you should find the the INSERT and UPDATE statements work pretty fast. An example of the type of INSERT query I use is as follows:
INSERT INTO keywords(api_adgroup_id, api_keyword_id, keyword_text, match_type, status)
SELECT a.api_id, a.keyword_text, a.match_type, a.status
FROM tmp_keywords a LEFT JOIN keywords b ON a.api_adgroup_id = b.api_adgroup_id AND a.api_keyword_id = b.api_keyword_id
WHERE b.api_keyword_id IS NULL
In this example, I perform an OUTER JOIN on the keywords table to check if it already exists. Only new rows in the temporary table where there isn't a match in the main table (the api_keyword_id in the keywords table is NULL) are inserted.
Also note that in this example I need to use both the ad group id AND the keyword id to uniquely identify the keyword because the AdWords API gives the same keyword/match type combination the same id when it exists in more than one ad group.