I have a MySQL database that is updated by a different application which I want to subscribe to for changes from my node.js server. Is it possible to monitor the database for any updates without long polling all the rows/columns for any changes to their value?
One potential solution I have seen is to use redis to subscribe to the database to listen for any changes and then it informs my client (which will be my server in this case). How do I subscribe redis to MySQL database, if this is possible?
Could you not just add an updated column to your tables?
You could add ON UPDATE CURRENT_TIMESTAMP on the column, which would automatically store the current time every time that row was updated. The rule is applied to the database itself, so you don't need to update any other clients which use the database, it will work automatically.
Any client could then make queries based on the last time it checked for updates. You just need to SELECT rows based on its updated field.
You're only checking one column that way, and its quite a fast query.
You could index the datetime field too, apperently. That would probably make the queries very fast indeed.
Related
I'm writing the back-end for a web app in Spring and it uses a MySQL database on an AWS RDS instance to keep track of user data. Right now the SQL tables are separated by user groups (just a value in a column), so different groups have different access to data. Whenever a person using the app does a certain operation, we want to back up their part of the database, which can be viewed later, or replace their data in the current branch if they want.
The only way I can figure out how to do this is to create separate copies of every table for each backup and keep another table to keep track of what all the names of the tables are. This feels very inelegant and labor intensive.
So far all operations I do on the database are SQL queries from the server, and I would like to stay consistent with that.
Is there a nice way to do what I need?
Why would you want a separate table for each backup? You could have a single table that mirrored the main table but had a few additional fields to record some metadata about the change, for example the person making it, a timestamp, and the type of change either update or delete. Whenever a change is made, simply copy the old value over to this table and you will then have a complete history of the state of the record over time. You can still enforce the group-based access by keeping that column.
As for doing all this with queries, you will need some for viewing or restoring these archived changes, but the simplest way for maintaining the archived records is surely to create TRIGGERS on the main tables. If you add BEFORE UPDATE and BEFORE DELETE TRIGGERS these can copy the old version of each record over to the archive (and also add the metadata at the same time) each time a record is updated or deleted.
Is there a way to insert a row into SQL with an expiration (c.f. you can insert a new key that expires in a minute with Memcached)?
The context is that I want an integration test to insert rows into a database, but I'd prefer not deleting them myself, as it's shared by many. Those delete queries must be manual, or they may not be run, or they may have disastrous typos, etc. I'd prefer the system to do it for me if it can (i.e. automatically and efficiently and well-tested).
(I assume this is not part of the SQL standard and the answer is no.)
related: SQL entries that expire after 24 hours
related: What is the best way to delete old rows from MySQL on a rolling basis?
CONTEXT: I can't make any changes to the database schema, or any of the associated infrastructure.
If you were doing unit testing, I would suggest wrapping each unit test in a BEGIN TRAN / ROLLBACK.
Since you are doing integrated testing, you probably need the data to live outside the scope of a single transaction. SQL Agent would work fine here, except that it would not distinguish between test data and real data. However, you could get around this by INSERTing some identifier to the specific records to be deleted upon expiration. That could be done in a single stored proc..
You might be able to accomplish this by using SQL Server Service Broker. I have not worked with the service broker, but maybe there is a way to delay message processing until a specific time has passed.
add an expiration date column to your table(s). create a job that will delete data that is past expiration on some schedule (say nightly).
I have a local mysql db and a production mysql db. I make changes locally and use a third party tool to sync the changes to the live server. The tool uses the checksum feature to identify the rows changed. My db structure is simple, one varchar200 field (acts as primary key), and a text field.
The problem is the sync is taking ages since there are thousands of rows. I believe adding a timestamp field will held in getting the checksum quickly for the tool to identify the rows to be synced. this created further more problems, as the timestamp field is different in local and prod servers due to the timezone differences.
I am looking for a useful idea or an alternative to timestamp that gets changed when a row is modified.
PS: I posted a similar question but didn't get any useful answers. I dont wany to rely on additional tables.
My tip: Don't use TIMESTAMP datatype, use DATETIME. They hold the same kind of data, but difference is TIMESTAMP is updated every time you touch the row, even if you don't set that column, it will be updated with "now", including insertions.
This means when you use TIMESTAMP, you can never truly synch the two databases - that column will always be different. If you use DATETIME, you can preserve that column's data.
If you can't code your applications to update the DATETIME column with "now", simply create a trigger that will do it for you.
You could do several things:
add a column for "dirty" to the source table. Make it a single BIT that you flip when the row gets changed and flip it back when it gets sync'd. If the row id is a primary key this is a simple insert ... on duplicate key update
store all your times as GMT. So no more fighting over timezone. This is standard practice anywhere time is being stored anyway.
setup replication between the two servers so MySQL will do the copying / updating for you. This is precisely what its designed for and it works well.
Timestamping the rows is in fact a bad idea, by the time the client updates its own latest syncing time, many rows may get updated on the server and you may miss these. Use a counter which increases by one every time you add or modify rows on the server, the client update itself and get the latest value of the counter. The client may not get the very last value for the counter (e.g. some row get updated while the client is requesting the update) but it's guarantee to catch up at the next update
Is there a way that if there's a change in records, that a query that changed the data (update, delete, insert) can be added to a "history" table transparently?
For example, if mySQL detects a change in a record or set of records, is there a way for mySQL to add that query statement into a separate table so that way, we can track the changes? That would make "rollback" possible since every query (other than SELECT) would be able to reconstruct database from its first row. Right?
I use PHP to interact with mySQL.
You need to enable the MySQL BinLog. This automatically logs all the alteration statements to a binary log which can be replied as needed.
The alternative is to use an auditing function through Triggers
Read about transaction logging in MySQL. This is built in to MySQL.
MySQL has logging functionality that can be used to log all queries. I usually leave this turned off since these logs can grow very rapidly, but it is useful to turn on when debugging.
If you are looking to track changes to records so that you can "roll back" a sequence of queries if some error condition presents itself, then you may want to look into MySQL's native support of transactions.
I have a test server that uses data from a test database. When I'm done testing, it gets moved to the live database.
The problem is, I have other projects that rely on the data now in production, so I have to run a script that grabs the data from the tables I need, deletes the data in the test DB and inserts the data from the live DB.
I have been trying to figure out a way to improve this model. The problem isn't so much in the migration, since the data only gets updated once or twice a week (without any action on my part). The problem is having the migration take place only when it needs to. I would like to have my migration script include a quick check against the live tables and the test tables and, if need be, make the move. If there haven't been updates, the script quits.
This way, I can include the update script in my other scripts and not have to worry if the data is in sync.
I can't use time stamps. For one, I have no control over the tables on the live side once it goes live, and also because it seems a bit silly to bulk up the tables more for conviencience.
I tried doing a "SHOW TABLE STATUS FROM livedb" but because the tables are all InnoDB, there is no "Update Time", plus, it appears that the "Create Time" was this morning, leading me to believe that the database is backed up and re-created daily.
Is there any other property in the table that would show which of the two is newer? A "Newest Row Date" perhaps?
In short: Make the development-live updating first-class in your application. Instead of depending on the database engine to supply you with the necessary information to enable you to make a decision (to update or not to update ... that is the question), just implement it as part of your application. Otherwise, you're trying to fit a round peg into a square hole.
Without knowing what your data model is, and without understanding at all what your synchronization model is, you have a few options:
Match primary keys against live database vs. the test database. When test > live IDs, do an update.
Use timestamps in a table to determine if it needs to be updated
Use the md5 hash of a database table and modification date (UTC) to determine if a table has changed.
Long story short: Database synchronization is very hard. Implement a solution which is specific to your application. There is no "generic" solution which will work ideally.
If you have an autoincrement in your tables, you could compare the maximum autoincrement values to see if they're different.
But which version of mysql are you using?
Rather than rolling your own, you could use a preexisting solution for keeping databases in sync. I've heard good things about SQLYog's SJA (see here). I've never used it myself, but I've been very impressed with their other programs.