I have to update a row into a table(InnoDB) and then right after select the last registry that I updated and make an insert. If the connection is too slow(for the update statement), can the select statement get the wrong row? Assuming that I'm using two different queries.
Are you using SQL to run your script or are you running it somewhere else? (ex PHP, Python, C#)
A Script from SQL should* always complete one line before moving on to the next but if you're unsure you could call something like the sleep function or the wait delay function to pause before you run your second line.
*I say should as I've seen some extremely rare random cases, usually with longer running queries that don't. If your first job takes a long time to complete it may be worth the effort to schedule the first job in Job Agent, then later that day schedule the second job.
MySQL does not keep records of row insertion order. Any algorithm that's based on last registry that I updated must implement its own means to gather the required information. If it doesn't, it will get the wrong row sooner or later. (Network speed is probably not as relevant as concurrent access.)
Related
I have to run a command every day at a different time. The times are known in advance and saved in a MySQL database in the familiar YYYY-MM-DD HH:MM:SS format.
What I thought of:
cron schedule the job for the exact time the first day, then make the script itself modify the crontab entry with the correct time for the next day.
cron scheduling the job at approximately the right time, then make it read the exact time from the database and sleep until then.
cron schedule the job execution every minute, and leave it to the script to determine whether the current date/time corresponds to the right execution time; proceded if it is, exit if not.
at submit the job the first day with at, then make it read the next day's time from the database and resubmit itself for then with at.
Additional info:
The command is a PHP script that composes the message of the day and sends it to all users registered to the website. I can consider other technologies if they solve this problem better. I would like to retain the ability of rebooting the server (outside of the intended execution hour) without worrying too much about jobs getting lost, therefore solutions 1. and 3. look better under this aspect. I'm starting with two commands to be run at two different times of the day, but I could soon end up with dozens more of similar jobs to be scheduled at different times every day, so I would prefer to avoid clutter as much as possible. I'd probably go with option 3 at this point.
The question(s):
Is there a better / preferred / established way of accomplishing this task? Solutions other that those mentioned above are welcome. What are the main drawbacks (of your recommended solution) I should be aware of?
I do believe you need to build your custom application for implementing the logic you want to implement.
Eventually you can use the cron system to start the process or to make sure that the process is running (in case it died or it was killed).
In your place what I would do, is to write a custom PHP program (or python or you name it) that performs the following:
Opens a connection to DB
Checks when the next execution is scheduled
Calculates if it is time to run
if not, it sleeps for X seconds (this depends on your preference)
it it is time to run, it performs its duty
sleeps again, and the loop begins
An alternative would be to check the every time the execution schedule, to check for changes in the schedule.
Another one would be read once and sleep until the execution time, but in this other case you would not catch changes in the schedule
This all depends on you, all in the all the program is an extremely easy one
I ended up using solution 3. above and am quite satisfied with it so far.
All the logic is in the .php file, which is responsible to:
save the current date/time in a variable (e.g. $now)
perform any considerations on it
scan the database in search of a matching date/time
This actually allows for a reasonable degree of flexibility:
I can choose not to run any commands if a certain semaphore file exists:
if (file_exists($filename)) {exit;}
I can set parameters in an option file enabling e.g. debug or test modes:
include parameters.php
if ($debug === true) {error_reporting(E_ALL);}
I can avoid bothering users if it is, let's say, new year's day:
if (date('m-d') == '01-01') {exit;}
I can introduce delays based on custom logic:
if (date('w', strtotime($now)) === '0') {$now = date('Y-m-d H:i:s', strtotime($now . ' +15 minutes'));}
So one of the projects I'm working on requires us to take every query that is ran on the server and automatically paste that query into a table inside of the database. The reason for this is so that the DBA is able to view all prior SQL Queries that have been ran on the box. Unfortunately I don't have any leeway to do this differently as the client is requiring this implementation.
Has anybody done this before or has any code that I could use that will automatically do this? Thanks.
Be careful! If you do an INSERT for every action taken, you will need to do an INSERT for that INSERT, at which point, you will ...
That is, the first logged query will hang the server and fill up the disk!
Instead of doing the task the way it is asked, turn on the "general log" and periodically scrape what it in it into another machine, which does not have this logging turned on.
Other arguments against the task as stated...
If a table has TRIGGERs, you will not be able to add another TRIGGER.
If "every query" really means "every", it is impossible (with a TRIGGER) since you can't write a SELECT or SHOW trigger.
"as the client is requiring this implementation". I would approach this unreasonable constraint by politely finding out what the real goal is. He has only described is an implementation.
If his goal is some kind of audit log, then my suggestion about the general log should suffice.
I have a routine in MySQL that is very long and has multiple SELECT, INSERT, and UPDATE statements in it with some IFs and REPEATs. It's been running fine until lately, where it's hanging an taking over 20 seconds to complete (which is unacceptable considering it used to take 1 second or so).
What is the quickest and easiest way for me to find out where in the routine the bottleneck is coming from? Basically the routine is getting stopped up and some point... how can I find out where that is without breaking apart the routine and testing one-by-one each section?
If you use Percona Server (a free distribution of MySQL with many enhancements), you can make the slow-query log record times for individual queries, using the log_slow_sp_statements configuration variable. See http://www.percona.com/doc/percona-server/5.5/diagnostics/slow_extended_55.html
If you're using stock MySQL, you can add statements in the stored procedure to set a series of session variables to the value returned by the SYSDATE() function. Use a different session variable at different points in the SP. Then after you run the SP in a test execution, you can inspect the values of these session variables to see what section of the SP took the longest.
To analyze the query can see the execution plan of the same. It is not always an easy task but with a bit of reading will find the solution. I leave some useful links
http://dev.mysql.com/doc/refman/5.5/en/execution-plan-information.html
http://dev.mysql.com/doc/refman/5.0/en/explain.html
http://dev.mysql.com/doc/refman/5.0/en/using-explain.html
http://www.lornajane.net/posts/2011/explaining-mysqls-explain
I'm currently building a system that does running computations, and every 5 seconds inserts or updates information based on those computations to a few rows in MySQL. I'm working on running this system on a few different servers at once right now with a few agents that are each doing similar processing and then writing on the same set of rows. I already randomize the order in which each agent writes its set of rows, but there's still a lot of deadlock happening. What's the best/fastest way to get through those deadlocks? Should I just rerun the query each time one happens, or do row locks, or something else entirely?
I suggest you try something that won't require more than one client to update your 'few rows.'
For example, you could have each agent that produces results do an INSERT to a staging table with the MEMORY access method.
Then, every five seconds you can run a MySQL event (a stored procedure within the server) that loops through all the rows in that table, posting their results to your 'few rows' and then deleting them. If it's important for the rows in your staging table to be processed in order, then you can use an AUTO_INCREMENT id field. But it might not be important for them to be in order.
If you want to get fancier and more scalable than that, you'll need a queue management system like Apache ActiveMQ.
Like my title describes: how can I implement something like a watchdog service in SQL Server 2008 with following tasks: Alerting or making an action when too many inserts are committed on that table.
For instance: Error table gets in normal situation 10 error messages in one second. If more than 100 error messages (100 inserts) in one second then: ALERT!
Would appreciate it if you could help me.
P.S.: No. SQL Jobs are not an option because the watchdog should be live and woof on the fly :-)
Integration Services? Are there easier ways to implement such a service?
Kind regards,
Sani
I don't understand your problem exactly, so I'm not entirely sure whether my answer actually solves anything or just makes an underlying problem worse. Especially if you are facing performance or concurrency problems, this may not work.
If you can update the original table, just add a datetime2 field like
InsertDate datetime2 NOT NULL DEFAULT GETDATE()
Preferrably, make an index on the table and then with whatever interval that fits, poll the table by seeing how many rows have an InsertDate > GetDate - X.
For this particular case, you might benefit from making the polling process read uncommitted (or use WITH NOLOCK), although one has to be careful when doing so.
If you can't modify the table itself and you can't or won't make another process or job monitor the relevant variables, I'd suggest the following:
Make a 'counter' table that just has one Datetime2 column.
On the original table, create an AFTER INSERT trigger that:
Deletes all rows where the datetime-field is older than X seconds.
Inserts one row with current time.
Counts to see if too many rows are now present in the counter-table.
Acts if necessary - ie. by executing a procedure that will signal sender/throw exception/send mail/whatever.
If you can modify the original table, add the datetime column to that table instead and make the trigger count all rows that aren't yet X seconds old, and act if necessary.
I would also look into getting another process (ie. an SQL Jobs or a homemade service or similar) to do all the housekeeping, ie. deleting old rows, counting rows and acting on it. Keeping this as the work of the trigger is not a good design and will probably cause problems in the long run.
If possible, you should consider having some other process doing the housekeeping.
Update: A better solution will probably be to make the trigger insert notifications (ie. datetimes) into a queue - if you then have something listening against that queue, you can write logic to determine whether your threshold has been exceeded. However, that will require you to move some of your logic to another process, which I initially understood was not an option.