I am try to find a scalable way to allow for my desktop application to run command when a change in the database is made.
The application is for running a remote command on your PC. The user logs into the website and can choose the run the command. Currently, users have to download a desktop application that checks the database every few seconds to see if a value has changed. The value can only be changed when they login to a website and press a button.
For now it seems to be working fine since there aren't many users. But when I hit 100+ users hitting the database 100+ times every few seconds is not good. What might be a better approach?
It's true that polling for changes is too expensive, especially if you have many clients. The queries are often very costly, and it's tempting to run the queries frequently to make sure the client gets notified promptly after a change. It's better to avoid polling the database.
One suggestion in the comments above is to use a UDF called from a trigger. But I don't recommend this, because a trigger runs when you do an INSERT/UPDATE/DELETE, not when you COMMIT the change. So a client could be notified of a change, and then when they check the database the change appears to not be there, because either the transaction was rolled back, or else the transaction simply hasn't been committed yet.
Another reason the trigger solution is not good is that MySQL triggers execute once for each row changed, not once for each INSERT/UPDATE/DELETE statement. So you could cause notification spam, if you do an UPDATE that affects thousands of rows.
A different solution is to use a message queue like RabbitMQ or ActiveMQ or Amazon SQS (there are many others). When a client commits their INSERT/UPDATE/DELETE, they confirm the commit succeeded, then post a message on a message queue topic. Many clients can be notified efficiently this way. But it requires that every client who commits changes to the database write code to post to the message queue.
Another solution is for clients to subscribe to MySQL's binary log and read it as a change data capture log. Every committed change to the database is logged in the binary log. You can make clients read this, and it has no more impact to the database server than a replication client (MySQL can easily support hundreds of replicas).
A hybrid solution is to consume the binary log, and turn those changes into events in a message queue. This is how a product like Debezium works. It reads the binary log, and posts events to an Apache Kafka message queue. Then other clients can wait for events on the Kafka queue and respond to them.
Im researching the best way of logging queries in MySQL database. The log should be used for two things:
Documentation of system activity
Used to recreate the database (in case the database is hacked or otherwise corrupted)
It's possible to log all queries in a MySQL database (like this example)
Question: It is possible to recreate a database on the basis of the log file, or should I use a different approach?
You can use replication logs for this - they store the complete set of operations. You should be able to create a new database from original sources and apply all changes upon it.
You can do complete dumps (i.e. once a week) and archive the replication logs on daily basis.
I want to write a listener which detects the DML changes on a table and perform some actions. This listener cannot be embedded in the application and it runs separately.
I thought let the application write to blackhole table and I will detect the changes from the binary log file.
But in the docs I found that enabling binary logging slows down the mysql performance slightly. Thats why i was wondering is there a way i can make the mysql master to log the changes related to a specific table.
Thanks!
SQL is the best way to track DML change and call function based on that. But, as you want to explore other options you may try
writing a cronjob with General Query Log which includes SELECT / SHOW statements as well which you don't need
mysqlbinlog : It slows down performance just a little, but it is necessary for point in time data recovery and replication.
Suggestions:
On a prod environment, MySQL binary log must be enabled. and general
query log must be disabled as general query logs almost everything
and gets filled very quickly and might run out of disk space if not
rotated properly.
On a dev/qa environment, general query log can be enabled with proper
rotation policy.
I have a table in my database which contains all of the users for my application. Unfortunately, when I launched my application, I didn't think to include a column which tracked the time at which a particular user signed up, and now I wish I had (bad idea, yes indeed).
Is there, by any shred of luck, a way that MySQL tracks when a particular record is inserted (such as in record metadata???), and would allow me to grab it and insert in into a new dedicated column for this purpose?
I am running on a shared cPanel host, so I doubt I have access to the MySQL logs.
Thank you for your time.
Only if you have binary logging enabled will you be able to trace exact times for the transaction.
http://dev.mysql.com/doc/refman/5.5/en/binary-log.html
Its not just for replication, but also a form of transactional recording in case of emergency.
Is it possible to restore table to last time with data if all data was deleted accidentally.
There is another solution, if you have binary logs active on your server you can use mysqlbinlog
generate a sql file with it
mysqlbinlog binary_log_file > query_log.sql
then search for your missing rows.
If you don't have it active, no other solution. Make backups next time.
Sort of. Using phpMyAdmin I just deleted one row too many. But I caught it before I proceeded and had most of the data from the delete confirmation message. I was able to rebuild the record. But the confirmation message truncated some of a text comment.
Someone more knowledgeable than I regarding phpMyAdmin may know of a setting so that you can get a more complete echo of the delete confirmation message. With a complete delete message available, if you slow down and catch your error, you can restore the whole record.
(PS This app also sends an email of the submission that creates the record. If the client has a copy, I will be able to restore the record completely)
As Mitch mentioned, backing data up is the best method.
However, it maybe possible to extract the lost data partially depending on the situation or DB server used. For most part, you are out of luck if you don't have any backup.
I'm sorry, bu it's not posible, unless you made a backup file earlier.
EDIT: Actually it is possible, but it gets very tricky and you shouldn't think about it if data wasn't really, really important. You see: when data get's deleted from a computer it still remains in the same place on the disk, only its sectors are marked as empty. So data remains intact, except if it gets overwritten by new data. There are several programs designed for this purpose and there are companies who specialize in data recovery, though they are rather expensive.
For InnoDB tables, Percona has a recovery tool which may help. It is far from fail-safe or perfect, and how fast you stopped your MySQL server after the accidental deletes has a major impact. If you're quick enough, changes are you can recover quite a bit of data, but recovering all data is nigh impossible.
Of cours, proper daily backups, binlogs, and possibly a replication slave (which won't help for accidental deletes but does help in case of hardware failure) are the way to go, but this tool could enable you to save as much data as possible when you did not have those yet.
No this is not possible. The only solution will be to have regular backups. This is very important.
Unfortunately, no. If you were running the server in default config, go get your backups (you have backups, right?) - generally, a database doesn't keep previous versions of your data, or a revision of changes: only the current state.
(Alternately, if you have deleted the data through a custom frontend, it is quite possible that the frontend doesn't actually issue a DELETE: many tables have a is_deleted field or similar, and this is simply toggled by the frontend. Note that this is a "soft delete" implemented in the frontend app - the data is not actually deleted in such cases; if you actually issued a DELETE, TRUNCATE or a similar SQL command, this is not applicable.)
If you use MyISAM tables, then you can recover any data you deleted, just
open file: mysql/data/[your_db]/[your_table].MYD
with any text editor