I want to remove a table row from my table new_data once the row is 45 mins old and then input it in another table called old_data.
The only way i can think for this to work, it to query the database lets say every min and remove any row thats (current_time - time inserted) > 45 mins.
Is there any other way of doing this? if not how could i set up a table to record inserted_time?
edit added
How could i write this statement to retrieve the correct data into the old_data table
SELECT * FROM new_spots WHERE (NOW()-created_at)>45mins
and then insert the above into the old_data table
you can specify value of time column upon insertion:
INSERT INTO x (created_at) VALUES (NOW());
additionally you can setup VIEW to show you only recent entries.
you are asking for some kind of auto expiration feature, it is not built into mysql. Memcached provides this feature. So it might be cleaner to achieve your goal as:
when you insert data into your system, you do:
insert your data into memcached with 45 minutes expiration time -- after 45 minutes, the data automatically disappear from memcached.
insert the data into the old_data table with a created_at column -- in case you need to rebuild your memcached when your memcached have to restart or other issue.
So everytime you just need to get the new data from the memcached -- as a side effect, it is faster than get the data from mysql :).
#keymone showed you how to capture the insert time. Then, periodically (every minute seems excessive - every 5 mins? 15 mins?) go through and build a list that meets the criteria, and for each entry, insert into your second table and delete from your first table.
I don't think there is an automatic way to do this. Here are some alternative ideas.
Use CRON
I have a similar scenario where we need to aggregate data from one table into another. A simple command line tool running via CRON suffices. We receive a few messages a second into our Web server and each results in a database insert. So volumes aren't huge but they are reasonably similar to your scenario
We use the NOW() function to record the insert time and after the records are 1hr old, we process them. It isn't exactly an hour but it is good enough. You can see the created_on field below.
CREATE TABLE glossaries (
id int(11) NOT NULL auto_increment,
# Our stuff ...
created_on datetime default NULL,
KEY owner_id (owner_id),
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Use CRON and a Trigger
Alternatively you could use a database trigger to kick off the processing. You would still need something scheduled to cause the trigger to fire but you would get max performance/
Chris
Related
I'm using Pentaho Data Integration to churn large JSON data into a MySQL database.
Anyway, of course there is the option of pure table insert, but also a table insert/ update (it will lookup a key to update, and if not found, insert the row).
This data is basically the current representation/ state of tickets -- so I need all data columns for a given row updated based on the unique Ticket ID.
With Pentaho, and likely most applications, updating is far slower than inserting. Especially since, at least with this application, you can have up to 25 connections writing (inserting) data at the same time, but only (1) process doing the lookup-then-write.
Therefore I'd rather do the following: Insert all the data (once a day or so) .. and then immediately after, run a query that deletes all duplicate Ticket IDs -- EXCEPT for the one with the greatest (aka latest) "update time."
I feel like that would be much faster -- and am wondering how to accomplish this in MySQL. I think it's something like
Delete from [table-name] where ticket_id = ticket_id and update_time < update_time, but I'm not certain if this is correct or variables are needed to compare data in an inter-row fashion.
I feel like I found this method on another page:
DELETE FROM ztable zt
WHERE EXISTS (
SELECT *
FROM ztable ex
WHERE ex.CaseKey = zt.CaseKey
AND ex.recordDate > zt.recordDate
);
I have a table named Warehouse for my database, it has Warehouse_idWarehouse and Warehouse_name as primary keys. What i want to do is to efficiently store a maximum of N recent changes that have been made to each warehouse that is stored in the table. I have considered creating a "helper" table (e.g. warehouse_changes) and taking care of the updates through my application, but honestly it feels like there is a smarter way around this.
Is there a way to store a specific amount of entries per warehouse and automatically manage updating the right element through mysql workbench? Thanks in advance and keep in mind that i'm not particularly advanced in this field.
There is a very detailed article on O'Reilly Answers that describes how to do exactly what you want using triggers.
When explained in two words, you need to create a helper table and a trigger per each operation type that you want to store. For example, here's how a trigger for updates looks like according to that article:
-- Creating a trigger that will run after each update
-- for each affected row
CREATE TRIGGER au_warehouse AFTER UPDATE ON Warehouse FOR EACH ROW
BEGIN
-- Insert new values into a log table, or you can insert old values
-- using the OLD row reference
INSERT INTO warehouse_log (action, id, ts, name)
VALUES('update', NEW.id, NOW(), NEW.name);
END;
After that you can get the latest 1000 changes using a simple SQL query:
SELECT * FROM warehouse_log ORDER BY ts DESC LIMIT 1000;
I have a MySQL table called: "regkeys". This table has two columns: "keyCat" and "keyNum". In my web-app I have a keygen module that adds a key(1134fb) and a category(mds or dts, etc.) into this table.
What I want to accomplish is this: after issuing (adding into the db) a key, I'd like to create a MySQL event to delete the keys that stay/stayed in my table longer than 2 days (or so), this way expiring the keys (for example: - from the time it was created, counting 2 days, each key has to be deleted, having certain keys getting deleted sooner or later than others, depending on when they were created). I looked at the MySQL's API, but I need some help with the logic. I do not know how to tell the event to delete only the keys that were stored in for 2 days (or so). I was hoping somebody could give me a quick example or direct me to a clear tutorial.
Thank you very much in advance.
Edited: I think I found this other question that helps a bit with my problem. I think I was going about it the wrong way (key based). Since every key and key category get inserted into a row, the scheduler should deal with rows instead of keys.
This is the solution:
Enabled the event_scheduler in the db like this: SET GLOBAL event_scheduler = ON;
Added a timestamp column in my table using MySQL default values
I created an event, CREATE EVENT <name>
Called the event to run on a schedule like so: ON SCHEDULE EVERY 20 SECOND
Add the SQL query to the event: DO DELETE FROM <table_name> WHERE <time_stamp_column> < NOW() - INTERVAL 5 MINUTE.
I have a database intensive application that needs to run every couple hours. Is there a way to detect whether a given table has changed since the last time this application ran?
The most efficient way to detect changes is this.
CHECKSUM TABLE tableName
A couple of questions:
Which OS are you working on?
Which storage engine are you using?
The command [http://dev.mysql.com/doc/refman/5.5/en/show-table-status.html](SHOW TABLE STATUS) can display some info depending on storage engine though.
It also depends on how large is the interval between runs of your intensive operation.
The most precise way I believe is with the use of triggers (AFTER INSERT/UPDATE) as #Neuticle mentioned, and just store the CURRENT_TIMESTAMP next to the table name.
CREATE TABLE table_versions(
table_name VARCHAR(50) NOT NULL PRIMARY KEY,
version TIMESTAMP NOT NULL
);
CREATE TRIGGER table_1_version_insert AFTER INSERT
ON table_1
FOR EACH ROW
BEGIN
REPLACE INTO table_versions VALUES('table_1', CURRENT_TIMESTAMP);
END
Could you set a trigger on the tables you want to track to add to a log table on insert? If that would work you only have to read the log tables on each run.
Use timestamp. Depending upon your needs you can set it to update on new rows, or just changes to existing rows. Go here to see a reference:
http://dev.mysql.com/doc/refman/5.0/en/timestamp-initialization.html
A common way to detect changes to a table between runs is with a query like this:
SELECT COUNT(*),MAX(t) FROM table;
But for this to work, a few assumptions must be true about your table:
The t column has a default value of NOW()
There is a trigger that runs on UPDATE and always sets the t column to NOW().
Any normal changes made to the table will then cause the output of the above query to change:
There are a few race conditions that can make this sort of check not work in some instances.
Have used CHECKSUM TABLE tablename and that works just splendid.
Am calling it from an AJAX request to check for table updates. If changes are found a screen refresh is performed.
For database "myMVC" and table "detail" it returns one row with fields "table" and "Checksum" set to "mymvc.detail" and "521719307" respectively.
Using a MySQL DB, I am having trouble with a stored procedure and event timer that I created.
I made an empty table that gets populated with data from another via SELECT INTO.
Prior to populating, I TRUNCATE the current data. It's used to track only log entries that occur within 2 months from the current date.
This turns a 350k+ log table into about 750 which really speeds up reporting queries.
The problem is that if a client sends a query precisely between the TRUNCATE statement and the SELECT INTO statement (which has a high probability considering the EVENT is set to run every 1 minute), the query returns no rows...
I have looked into locking a read on the table while this PROCEDURE is ran, but locks are not allowed in STORED PROCEDURES.
Can anyone come up with a workaround that (preferably) doesn't require a remodel?
I really need to be pointed in the right direction here.
Thanks,
Max
I'd suggest an alternate approach instead of truncating the table, and then selecting into it...
You can instead select your new data set into a new table. Next, using a single RENAME command, rename the new table to the existing table and the existing table to some backup name.
RENAME TABLE existing_table TO backup_table, new_table TO existing_table;
This is a single, atomic operation... so it wouldn't be possible for the client to read from the data after it is emptied but before it is re-populated.
Alternately, you could change your TRUNCATE to a DELETE FROM, and then wrap this in a transaction along with the SELECT INTO:
START TRANSACTION
DELETE FROM YourTable;
SELECT INTO YourTable...;
COMMIT