I have a database intensive application that needs to run every couple hours. Is there a way to detect whether a given table has changed since the last time this application ran?
The most efficient way to detect changes is this.
CHECKSUM TABLE tableName
A couple of questions:
Which OS are you working on?
Which storage engine are you using?
The command [http://dev.mysql.com/doc/refman/5.5/en/show-table-status.html](SHOW TABLE STATUS) can display some info depending on storage engine though.
It also depends on how large is the interval between runs of your intensive operation.
The most precise way I believe is with the use of triggers (AFTER INSERT/UPDATE) as #Neuticle mentioned, and just store the CURRENT_TIMESTAMP next to the table name.
CREATE TABLE table_versions(
table_name VARCHAR(50) NOT NULL PRIMARY KEY,
version TIMESTAMP NOT NULL
);
CREATE TRIGGER table_1_version_insert AFTER INSERT
ON table_1
FOR EACH ROW
BEGIN
REPLACE INTO table_versions VALUES('table_1', CURRENT_TIMESTAMP);
END
Could you set a trigger on the tables you want to track to add to a log table on insert? If that would work you only have to read the log tables on each run.
Use timestamp. Depending upon your needs you can set it to update on new rows, or just changes to existing rows. Go here to see a reference:
http://dev.mysql.com/doc/refman/5.0/en/timestamp-initialization.html
A common way to detect changes to a table between runs is with a query like this:
SELECT COUNT(*),MAX(t) FROM table;
But for this to work, a few assumptions must be true about your table:
The t column has a default value of NOW()
There is a trigger that runs on UPDATE and always sets the t column to NOW().
Any normal changes made to the table will then cause the output of the above query to change:
There are a few race conditions that can make this sort of check not work in some instances.
Have used CHECKSUM TABLE tablename and that works just splendid.
Am calling it from an AJAX request to check for table updates. If changes are found a screen refresh is performed.
For database "myMVC" and table "detail" it returns one row with fields "table" and "Checksum" set to "mymvc.detail" and "521719307" respectively.
Related
Is there any way to detect when an ALTER TABLE statement is executed in MySQL? For example, if the following statement were executed on some_table, is there any way to detect that the column name changed from column_name_a to column_name_b and log it in another table in the DB?
ALTER TABLE `some_table`
CHANGE COLUMN `column_name_a` `column_name_b` VARCHAR(255) NULL DEFAULT NULL;
Thanks.
To my knowledge it is unfortunately not possible to put triggers on the INFORMATION_SCHEMA tables, since they are strictly spoken views and triggers can't be made to work on views. If triggers would be possible on the INFORMATION SCHEMA, then you could have a trigger on updates of the INFORMATION_SCHEMA.COLUMNS table to identify name changes.
However, what you can do is one of the following things:
option 1) Maintain a real table with all column names. Then create a function that checks for a discrepancy between the INFORMATION_SCHEMA.COLUMNS table abd your table. If there is one, you know the name has changed. You need to copy over the new name to your column name table and do whatever else you wanted to do upon name change.
The function to check for discrepancies then must be run periodically via the mysql scheduler in order to detect name changes as quickly as possible. Note that this is not a real time solution. There will be a lag between the ÀLTER TABLE command and its detection. If this is unacceptable in your scenario you need to go with
option 2) Do not call ÀLTER TABLE directly, but wrap it in a function. Within this function you can also call other functions to achieve what you need to achieve. If may be worth while to formulate the needed steps in a higher programming language that you use to drive your application. If this is not possible, you will be limited to the possibilities that are offered in functions/procedures in the mysql environment.
Sorry to not have a simpler way of doing this for you.
I'm running MariaDB 5.5.56.
I'm looking to copy an entire row in a database, change one column, then insert the entire row back into the original database (I don't want to have to specify the individual fields because there's a lot of them). The problem I'm running into is how to deal with an auto-increment/primary key column.
example:
create temporary table t_ownership like ownership;
insert into t_ownership (select * from ownership where name='x' LIMIT 1);
update t_ownership set id='something else';
insert into ownership (select * from t_ownership);
I have a column "recno" that is an auto-increment that will create a collision in the database when I try to re-insert the slightly changed record back into the original table.
Something like this seems to work but doesn't result in an insert:
insert into ownership (select * from t_ownership) ON DUPLICATE KEY UPDATE recno=LAST_INSERT_ID(ownership.recno);
The above statement executes without error but does not add a row to table ownership.
So I think I'm close but not quite there...
What would be the best way to do this? I'd like to avoid doing an insert where I manually specify field/values. I just need to regenerate a new A.I. recno column on the insert.
NULL values inserted into auto-incremented fields end up just getting the next auto-increment value, behaving equivalent to INSERTing without specifying the field; so you should be able to update the source (temp copy) to have NULL for that field.
However, one potential issue that could present itself in scenarios like yours is that the CREATE TEMPORARY TABLE ... LIKE could result in a table that would not allow you to set such fields to NULL; this would require you to either ALTER the temporary table, or create it in a more explicit manner. Either way, it now makes code/queries that do not specify columns even more reliant on knowing columns.
Personally, I would take this route in the first place.
INSERT INTO theTable([list all but the auto-inc column])
SELECT [list all but the auto-inc column, with any replacements or modifications desired]
FROM ...[original query]...
It accomplishes the task in one query, makes the queries more self documenting, and only at the cost of a little typing (most of which a decent database browser, or query builder, will do for you).
The only argument really in favor of your current approach is that the table involved can be changed without necessarily breaking your queries; but that begs the question of whether it would be better for such table changes to break the queries, forcing them to be re-examined. If it is not an issue, it is a minor revision; but the alternative is queries that continue to be valid that have the potential to cause unexpected behavior due to copying information they were never intended to.
I have a table tblsessions. At one time, only one session could be current as is session 2014-2015.
However, if i make 2015-2016 current, 2014-2015 should not be current anymore.
How could I implement this logic in table at design time?
Here is the table creation code waiting for your modification:
create table tblsessions(
sessionid int not null auto_increment,
sessionname varchar(9) not null,
current ????
primary key (sessionid)
);
You could perhaps use a trigger (depending on the version of MySQL you're running). I've assumed that current is a tinyint but you can adjust to whatever type you use:
CREATE TRIGGER curr_check BEFORE UPDATE ON tblsessions
FOR EACH ROW
BEGIN
IF NEW.current = 1 THEN
UPDATE tblsessions SET current = 0;
END IF;
END;
EDIT:
A.5.3: Does MySQL 5.6 have statement-level or row-level triggers?
In MySQL 5.6, all triggers are FOR EACH ROW—that is, the trigger is activated for each row that is inserted, updated, or deleted. MySQL 5.6 does not support triggers using FOR EACH STATEMENT.
ALTERNATIVE SOLUTION:
I have come up with another solution however I wonder if it really is a good solution.
I have created two tables:
TBLSESSIONS (session)
// session is primary key and stops duplicates
TBLCURRENTSESSION (csessionid, csession)
// csessionid is auto-int
// csession is foreign key to TBLSESSIONS.session
Each time user presses a button [Make This Session Default], I can insert that session into csession.
In code I can search for largest csessionid and find csession against it as the CURRENT SESSION.
This also allows user to switch sessions at time.
Being MySQL DBA, do you think it is a good approach to solving my basic problem? Do you see any dark sides of this solution?
I'm extremely new to Views so please forgive me if this is a silly question, but I have a View that is really helpful in optimizing a pretty unwieldy query, and allows me to select against a small subset of columns in the View, however, I was hoping that the View would actually be stored somewhere so that selecting against it wouldn't take very long.
I may be mistaken, but I get the sense (from the speed with which create view executes and from the duration of my queries against my View) that the View is actually run as a query prior to the external query, every time I select against it.
I'm really hoping that I'm overlooking some mechanism whereby when I run CREATE VIEW it can do the hard work of querying the View query *then, so that my subsequent select against this static View would be really swift.
BTW, I totally understand that obviously this VIEW would be a snapshot of the data that existed at the time the VIEW was created and wouldn't reflect any new info that was inserted/updated subsequent to the VIEW's creation. That's actually EXACTLY what I need.
TIA
What you want to do is materialize your view. Have a look at http://www.fromdual.com/mysql-materialized-views.
What you're talking about are materialised views, a feature of (at least) DB2 but not MySQL as far as I know.
There are ways to emulate them by creating/populating a table periodically, or on demand, but a true materialised view knows when the underlying data has changed, and only recalculates if required.
If the data will never change once the view is created (as you seem to indicate in a comment), just create a brand new table to hold the subset of data and query that. People always complain about slow speed but rarely about data storage requirements :-)
You can do this with:
A MySQL Event
A separate table (for caching)
The REPLACE INTO ... SELECT statement.
Here's a working example.
-- create dummy data for testing
CREATE TABLE MyTable (
id INT NOT NULL,
groupvar INT NOT NULL,
myvar INT
);
INSERT INTO MyTable VALUES
(1,1,1),
(2,1,1),
(3,2,1);
-- create the view, making sure rows have a unique identifier (groupvar)
CREATE VIEW MyView AS
SELECT groupvar, SUM(myvar) as myvar_sum
FROM MyTable
GROUP BY groupvar;
-- create cache table, setting primary key to unique identifier (groupvar)
CREATE TABLE MyView_Cache (PRIMARY KEY (groupvar))
SELECT *
FROM MyView;
-- create a table to keep track of when the cache has been updated (optional)
CREATE TABLE MyView_Cache_updated (update_id INT NOT NULL AUTO_INCREMENT, PRIMARY KEY (update_id));
-- create event to update cache table (e.g., daily)
DELIMITER |
CREATE EVENT MyView_Cache_Event
ON SCHEDULE EVERY 1 DAY STARTS CURRENT_TIMESTAMP + INTERVAL 1 HOUR
DO
BEGIN
REPLACE INTO MyView_Cache
SELECT *
FROM MyView_Cache;
INSERT INTO MyView_Cache_updated
SELECT NULL, NOW() AS last_updated;
END |
DELIMITER ;
You can now query MyView_Cache for faster response times, and query MyView_Cache_updated to inform users of the last time the cache was updated (in this example, daily).
Since a view is basically a SELECT statement you can use query cache to improve performance.
But first you should check if :
you can add indexes in the tables involved to speed up the query (use EXPLAIN)
the data isn't changing very often you can materialize the view (make snapshots)
Use a materiallised view.. It can store data like count sum etc but yes after updating the table you need to refresh the view to get correct results as they are not auto updated.. Moreover after querying from view the results are stored in cache so the memory cycles reduces to 2 which are 4 in case of querying from the table itself. So it gets efficient from the second time.. When you query for 1st time from view the data is fetched from main memory and is stored in cache after it.
I want to remove a table row from my table new_data once the row is 45 mins old and then input it in another table called old_data.
The only way i can think for this to work, it to query the database lets say every min and remove any row thats (current_time - time inserted) > 45 mins.
Is there any other way of doing this? if not how could i set up a table to record inserted_time?
edit added
How could i write this statement to retrieve the correct data into the old_data table
SELECT * FROM new_spots WHERE (NOW()-created_at)>45mins
and then insert the above into the old_data table
you can specify value of time column upon insertion:
INSERT INTO x (created_at) VALUES (NOW());
additionally you can setup VIEW to show you only recent entries.
you are asking for some kind of auto expiration feature, it is not built into mysql. Memcached provides this feature. So it might be cleaner to achieve your goal as:
when you insert data into your system, you do:
insert your data into memcached with 45 minutes expiration time -- after 45 minutes, the data automatically disappear from memcached.
insert the data into the old_data table with a created_at column -- in case you need to rebuild your memcached when your memcached have to restart or other issue.
So everytime you just need to get the new data from the memcached -- as a side effect, it is faster than get the data from mysql :).
#keymone showed you how to capture the insert time. Then, periodically (every minute seems excessive - every 5 mins? 15 mins?) go through and build a list that meets the criteria, and for each entry, insert into your second table and delete from your first table.
I don't think there is an automatic way to do this. Here are some alternative ideas.
Use CRON
I have a similar scenario where we need to aggregate data from one table into another. A simple command line tool running via CRON suffices. We receive a few messages a second into our Web server and each results in a database insert. So volumes aren't huge but they are reasonably similar to your scenario
We use the NOW() function to record the insert time and after the records are 1hr old, we process them. It isn't exactly an hour but it is good enough. You can see the created_on field below.
CREATE TABLE glossaries (
id int(11) NOT NULL auto_increment,
# Our stuff ...
created_on datetime default NULL,
KEY owner_id (owner_id),
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Use CRON and a Trigger
Alternatively you could use a database trigger to kick off the processing. You would still need something scheduled to cause the trigger to fire but you would get max performance/
Chris