I have a database which has a timestamp column and I want outdated data to be deleted.
So my idea is to write a MySQL query to a .php file which deletes every row where timestamp < current_timestamp - const. As there will be a lot of rows where this has to be checked, I am going to set an index to the timestamp column.
So how can I run this script automatically every n seconds? I heard about Linux crontab - can I use this on my webserver (not the db server) to execute the .php file periodically and is this overall a good technique to delete outdated rows from a database?
The database is set on a RDS instance on Amazon Web Services. My webserver is a EC2 instance (also Amazon Web Services).
Doing such a thing requires setting up an event or job. Such efforts keep the database very busy.
I would strongly recommend a different approach. Use a view to access the data you want:
create view v_t as
select t.*
from t
where timestamp > CURRENT_TIMESTAMP - ??;
Then use this view to access the data.
Then, periodically, go in an clean the table to get rid of the rows that you don't don't want. You can do this once a day, once a week, once an hour -- the deletions can occur at times when the database load is lighter, so it doesn't affect users.
I think you should check out lambda service on AWS.
It allows you to run commands against AWS services without another instance running.
Here's an example on how to set it up.
http://docs.aws.amazon.com/lambda/latest/dg/vpc-rds-deployment-pkg.html
Good luck
Eugene
Gordon Linoff's approach is ideal, but if you want to go the route of scheduled jobs, MySQL Event Scheduler is something you can try. The following example, runs daily and delete records older than a week.
CREATE EVENT
clean_my_table
ON SCHEDULE EVERY 1 DAY
DO
DELETE FROM my_table
WHERE time_stamp < date_sub(now(), INTERVAL 1 WEEK);
MySQL Event Reference page
https://dev.mysql.com/doc/refman/5.7/en/create-event.html
Related
I am using a host gator shared hosting plan, and need to execute a simple command every minute:
UPDATE table_info SET expired = 1 WHERE TIMESTAMP(dateTime) <= NOW()
My problem is that I cant turn on the event scheduler because I don't have permissions for it. Can anyone think of a simple way to do this without using events?
You can put your update statement into simple PHP script, and run it via cron.
Using cron or any other OS based task scheduler is stricktly speaking an answer to this question.
However, it is not really a best practice to run any event every minute. Particularly not, if the operation done could be executed on the fly, like here. The expired flag's value depends on a simple calculation:
dateTime <= NOW()
Instead of running this calculation every minute on the entire table_info, you could
place the sql command into a regularly executed script
just embedd the logic into any reporting or other operation which depend on the expired field and get rid of the flag completely.
I want to create a monitoring script that will check my servers. Right now I'm stuck on a problem, I need to find out a way to get uptime percentage. Basically all data is stored in MySQL server, for me the easiest way to get uptime is to create a function that will add a new record to mysql server every minute with date, time, information is it online etc. but if I will use this method and I will have for example 1000 servers to monitor, I will end up with 518 400 000 records in MySQL server per year.
Another idea was to create one record per server with two rows online and offline, but without any date and time I'm not able to get uptime...
Any ideas how to design database for monitoring system ?
The MySQL information_schema contains uptime information (expressed in seconds) for each server. I am not sure how accurate your figure has to be, but you could get this value at a set interval and compare it to the previous value. Depending on the interval, this could give you a good approximation.
SELECT Variable_Value FROM SESSION_STATUS S WHERE Variable_Name = 'UPTIME';
Also, the MySQL error log contains a date and time stamp when the server starts. Scrape this info periodically and add to your server table.
Is there a way to insert a row into SQL with an expiration (c.f. you can insert a new key that expires in a minute with Memcached)?
The context is that I want an integration test to insert rows into a database, but I'd prefer not deleting them myself, as it's shared by many. Those delete queries must be manual, or they may not be run, or they may have disastrous typos, etc. I'd prefer the system to do it for me if it can (i.e. automatically and efficiently and well-tested).
(I assume this is not part of the SQL standard and the answer is no.)
related: SQL entries that expire after 24 hours
related: What is the best way to delete old rows from MySQL on a rolling basis?
CONTEXT: I can't make any changes to the database schema, or any of the associated infrastructure.
If you were doing unit testing, I would suggest wrapping each unit test in a BEGIN TRAN / ROLLBACK.
Since you are doing integrated testing, you probably need the data to live outside the scope of a single transaction. SQL Agent would work fine here, except that it would not distinguish between test data and real data. However, you could get around this by INSERTing some identifier to the specific records to be deleted upon expiration. That could be done in a single stored proc..
You might be able to accomplish this by using SQL Server Service Broker. I have not worked with the service broker, but maybe there is a way to delay message processing until a specific time has passed.
add an expiration date column to your table(s). create a job that will delete data that is past expiration on some schedule (say nightly).
I am trying to use mysql event schedule in my application, I have not use it before so i have some confusions.
I want to know if my computer is off on the schedule date, then schedule will continue on next day, after starting my computer?
Like:
my schduled is for beginning at every month (no predefined time set)
if in the above date my computer/Server is off,
will mysql continue scheduled event in next day after turning on my computer/server?
If no, then please suggest a solution.
Hmmmm, have you looked at something like this?
MySQL: Using the Event Scheduler
... or:
How to create MySQL Events
... or even: [MySQL :: MySQL 5.1 Reference Manual: 19.4.1. Event Scheduler Overview](19.4.1. Event Scheduler Overview)?
Also please keep in mind that SQL DBMS servers are written with the rather strong presumption that they will be kept up and operating 24 hours per day with only brief periods of downtime for maintenance or repairs. There is generally very little consideration for operation on machines which are shutdown at night and while not in use.
If you simply store a table of dates and events then your can simply query that table for events which have passed or are upcoming within any range you like ... and you can run the program(s) containing those queries (and performing any appropriate activities based on the results) whenever you start you computer and periodically while it's up and running.
These links refer to a feature of MySQL which is designed to have the server internally execute certain commands (MySQL internal commands, such as re-indexing, creating/updating views, cleaning tables of data which "expires" and so on. I don't know if a MySQL server would attempt to execute all events which have passed during downtime, though it should only be a little bit of work to follow the tutorial, schedule some event for some time (say 15 minutes after the time you expect to hit [Enter]) ... then shutdown your computer (or even just the MySQL server) and go off to lunch. Then come back, start it up and see what happens.
The scheduled event could be something absurdly simple, like inserting the "current" time into some table you set up.
I have a MySQL database with about 30,000 rows. I update this database to a remote server nightly, but never are more than 50 rows updated at a time. I am the only one who updates the database. I would like to develop a method in which only CHANGED rows are exported to the remote server.
To save space in the database and to save time when I export to the remote server, I have built "archive" tables (on the remote server) with records that will no longer be updated, and which do not reside on the local database. But I know splitting up this data into multiple tables is bad design that could lead to problems if the structure of the tables ever needs to be changed.
So I would like to rebuild the database so that ALL the records with similar table structures are in a single table (like they were when the database was much smaller). The size of the resulting table (with all archived records) would exceed 80,000 rows, much to large to export in a whole-database package.
To do this, I would like to
(1) Update a "last updated" timestamp in each row when the row is added or modified
(2) Select only rows in tables for export when their "last update" timestamp is greater than the timestamp of the last export operation
(3) Write a query that builds the export .sql file with only new and updated rows
(4) Update the timestamp for the export operation to be used for comparison during the next export
Has anyone ever done this? If so, I would be grateful for some guidance on how to accomplish this.
Steve
If you add a column with a timestamp datatype, for example last_updated timestamp, it will be automatically updated to now() every time a row changes.
Then early every day, simply ship yesterday's changes:
select * from mytable
where last_updated between subdate(CURDATE(), 1) and CURDATE()
Why not just setup the remote server as a replication slave? MySQL will only send updated rows in that situation, and very quickly / efficiently at that.
Using an official replication strategy is generally advisable rather than rolling your own. You'll have lots of examples to work from and lots of people who understand what's going on if you run into problems.