I've got a Wordpress site. And some of the database tables are getting large--from data collected by the plug-ins. I know of two ways to delete this data:
I can delete rows manually--1000 at a time--in phpmyadmin.
I can empty the whole table in phpmyadmin.
What I'm looking for is a third way, so that I can delete just the data collected before a certain time period.
As noted above, I know that I can sort the rows by date and delete 1000 at a time--to get rid of old ones. But, there are over a million rows. Is there a single procedure I can use to delete--for example--all of the rows that are more than 60 days old?
To delete records that are more than 60 days old you can do something like this
DELETE
FROM table1
WHERE datefield < DATE_SUB(CURDATE(), INTERVAL 60 DAY)
SQLFiddle
Related
So basically I have a table called acc_rec_pay_old which stores the deleted data, the table has its own date so how can I delete all records from acc_rec_pay_old after 2 years AUTOMATICALLY?
Setup a cronjob that runs every day that executes the following query:
DELETE FROM acc_rec_pay_old WHERE DATE_ADD(date_field, INTERVAL 2 YEAR) = CURRENT_DATE;
Write a script in whatever language your comfortable with, create a config file which reads your table name and duration of records to be kept.
Run this via crontab
I have 2 database servers with identical schema.
Basically, I have a crontab job (every minute) to fetch all new or updated records within last 2 minutes from database 1 to database 2.
Database 2 is a Mysql server.
(Database 1 is not mysql server.)
To make sure I have all the data, I use "replace-into" statement.
REPLACE INTO myTable (ID, Data, CreateTime, UpdateTime) VALUES ....
Normally, it works well.
However, there are too many connections to database 1. Occasionally, a fetching takes over a minute due to connection problem, which jammed with the fetching in the next minute. The data batch which is fetched in time T was put into database 2 later than the batch in time T+1. Hence, a record with older data overwrite the newer one.
I want these to happen:
If the record is not exist in database 2, insert the records.
If the record exists in database 2, update the records only if createTime >= existing createTime or updateTime >= existing updateTime. (*If both createTime and updateTime are same as the existing ones, it is ok to overwrite the current records.)
The above replace-into can be rewritten into INSERT...ON DUPLICATE UPDATE, but my main problem is I need a where-condition to control the updating.
Thanks in advance.
How can I create a lifetime of a row so after a specific time say 2 weeks the row will automatically erase? Any info would be great.
RDBMS don't generally allow rows to automatically self destruct. It's bad for business.
More seriously, some ideas, depending on your exact needs
run a scheduled job to run a DELETE to remove rows based on some date/time column
(more complex idea) use a partitioned table with a sliding window to move older rows to another partition
use a view to only show rows less than 2 weeks old
Add a timestamp column to the table that defaults to CURRENT_TIMESTAMP, and install a cron job on the server that frequently runs and prunes old records.
DELETE FROM MyTable WHERE datediff(now(), myTimestamp) >= 14;
Or you can add timestamp column and always select like this:
SELECT * FROM myTable WHERE timetampColumn>=date_sub(now(), interval 2 week);
It is better if you don't need to erase the data and you want to show only data from last 2 weeks.
I find myself wanting to delete rows older than (x)-days on a rolling basis in a lot of applications. What is the best way to do this most efficiently on a high-traffic table?
For instance, if I have a table that stores notifications and I only want to keep these for 7 days. Or high scores that I only want to keep for 31 days.
Right now I keep a row storing the epoch time posted and run a cron job that runs once per hour and deletes them in increments like this:
DELETE FROM my_table WHERE time_stored < 1234567890 LIMIT 100
I do that until mysql_affected_rows returns 0.
I used to do it all at once but that caused everything in the application to hang for 30 seconds or so while INSERTS piled up. Adding the LIMIT worked to alleviate this but I'm wondering if there is a better way to do this.
Try creating Event that will run on database automatically after the time interval you want.
Here is an Example:
If you want to delete entries that are more than 30 days old from some table 'tableName', having column entry 'datetime'. Then following query runs every day which will do required clean-up action.
CREATE EVENT AutoDeleteOldNotifications
ON SCHEDULE AT CURRENT_TIMESTAMP + INTERVAL 1 DAY
ON COMPLETION PRESERVE
DO
DELETE LOW_PRIORITY FROM databaseName.tableName WHERE datetime < DATE_SUB(NOW(), INTERVAL 30 DAY)
We need to add ON COMPLETION PRESERVE to keep the event after each run. You can find more info here: http://www.mysqltutorial.org/mysql-triggers/working-mysql-scheduled-event/
Check out MySQL Partitioning:
Data that loses its usefulness can often be easily removed from a partitioned table by dropping the partition (or partitions) containing only that data. Conversely, the process of adding new data can in some cases be greatly facilitated by adding one or more new partitions for storing specifically that data.
See e.g. this section to get some ideas on how to apply it:
MySQL Partition Pruning
And this one:
Partitioning by dates: the quick how-to
Instead of executing the delete against the table alone, try gathering the matching keys first and then do a DELETE JOIN
Given you sample query above
DELETE FROM my_table WHERE time_stored < 1234567890 LIMIT 100 ;
You can leave the LIMIT out of it.
Let say you want to delete data that over 31 days old.
Let's compute 31 days in seconds (86400 X 31 = 2678400)
Start with key gathering
Next, index the keys
Then, perform DELETE JOIN
Finally, drop the gathered keys
Here is the algorithm
CREATE TABLE delete_keys SELECT id FROM my_table WHERE 1=2;
INSERT INTO delete_keys
SELECT id FROM
(
SELECT id FROM my_table
WHERE time_stored < (UNIX_TIMESTAMP() - 2678400)
ORDER BY time_stored
) A LIMIT 100;
ALTER TABLE delete_keys ADD PRIMARY KEY (id);
DELETE B.* FROM delete_keys
INNER JOIN my_table B USING (id);
DROP TABLE delete_keys;
If the key gathering is less than 5 minutes, then run this query every 5 minutes.
Give it a Try !!!
UPDATE 2012-02-27 16:55 EDT
Here is something that should speed up key gathering a little more. Add the following index:
ALTER TABLE my_table ADD INDEX time_stored_id_ndx (time_stored,id);
This will better support the subquery that populates the delete_keys table because this provides a covering index so that the fields are retrieved frok the index only.
UPDATE 2012-02-27 16:59 EDT
Since you have to delete often, you may want to try this every two months
OPTIMIZE TABLE my_table;
This will defrag the table after all those annoying little deletes every 5 minutes for two months
At my company, we have a similar situation. We have a table that contains keys that have an expiration. We have a cron that runs to clean that out:
DELETE FROM t1 WHERE expiration < UNIXTIME(NOW());
This ran once an hour, but we were having similar issues to what you are experiencing. We increased it to once per minute. Then 6 times per minute. Setup a cron with a bash script that basically does the query, then sleeps for a few seconds and repeats until the minute is up.
The increased frequency significantly decreased the number of rows that we were deleting. Which relieved the contention. This is the route that I would go.
However, if you find that you still have too many rows to delete, use the limit and do a sleep between them. For example, if you have 50k rows to delete, do a 10k chunk with a 2 second sleep between them. This will help the queries from stacking up, and it will allow the server to perform some normal operations between these bulk deletes.
You may want to consider introducing a master/slave (replication) solution into your design. If you shift all the read traffic to the slave, you open up the master to handle 'on-the-fly' CRUD activities, which then replicate down to the slave (your read server).
And because you are deleting so many records you may want to consider running an optimize on the table(s) from where the rows are being deleted.
Ended up using this to leave only 100 last rows in place, so significant lag when executed frequently (every minute)
delete a from tbl a left join (
select ID
from tbl
order by id desc limit 100
) b on a.ID = b.ID
where b.ID is null;
I have a table with a timestamp column that records when the record is modified.
I would like to on a nightly basis move all records that are older than 6 days.
should I use
insert into archive_table select * from regualr_table where datediff( now(), audit_updated_date)>=6;
delete from regular_table where datediff( now(), audit_updated_date)>=6;
since there are 1 million rows in regular_table, is there anyway to optimize the query so they run faster? Also will the delete be locking the regular_table?
My main concern is the read query to the db won't be slowed down by this archiving process.
Two suggestions:
Compute the value of the cutoff date in a variable, and query compared to that, e.g.:
SET #archivalCutoff = DATE_SUB(NOW(), INTERVAL 6 DAY);
insert into archive_table select * from regular_table where audit_updated_date < #archivalCutoff;
delete from regular_table where audit_updated_date)< #archivalCutoff;
In fact what you have in your question runs into problems, especially with lots of records, because the cutoff moves, you may get records in your regular and archive tables, and you may get records that are deleted but not archived.
The second suggestion is to index the audit_updated field.