I have an NP-hard data transformation problem that I know can run for hours. So I'm fine tuning some heuristics to keep the complexity under control.
But the process may take 5 minutes to run with one setting and 10 hours with an adjustment of 20% on just one parameter. As this also depends on the actual data.
I would like to stop the process if it is running for too long. This is both in regular operation as well as when I trigger the event manually.
The idea was to track the show processlist, identify this event and kill it if it's been running over 15 minutes. Unfortunately I don't seem to see any running process from the event - even though I can see log entries written at the same time.
How can I kill a process fired by the event scheduler?
I'm using MySQL 5.7 on CentOS.
Related
So i have this database running on a Synology NAS, for a restaurant's app made with Laravel, and i have this event here that should start every day at 4am
The content of this event is nothing special:
UPDATE shipping_times SET shipping_times.available = shipping_times.max_quantity
Thing is, every night at midnight the event scheduler variable auto sets to OFF even if i do GLOBAL event_scheduler = ON.
This is quite a problem since the event is used to "replenish" available orders.
Since my event should occur at 4AM i don't think the problem is event-related.
What could it be?
Alternative: Laravel Task Scheduling
Laravel Tasks come to mind for solving this instead: https://laravel.com/docs/8.x/scheduling
(other Laravel version docs available there as well)
Adding recurring tasks in this way puts them all in one place instead of hidden somewhere in a menu. They are also independant of your elusive event_scheduler.
You can also use a command and call it from the task scheduling code. See BKF's comment.
Are your events firing?
I am not sure but could it be true the event is only firing if you always have traffic between 00:00 and 04:00? Laravel Tasks also fire after the fact, AFAIK.
Add this to my.cnf (or whaterver the config file is):
event_scheduler = ON
and restart mysqld.
Apparently, something is shutting down MySQL every night, perhaps a backup? The above setting will turn it on each time it restarts.
I would like to delay deletion of data from the database. I am using MySQL, nest.js. I heard that CRON is what I need. I want to delete the entry in a week. Can you help me with this? CRON is what I need, or i need to use something another?
A cron job (or at in Windows) or a MySQL EVENT can be created to periodically check for something and take action. The resolution is only 1 minute.
If you need a very precise resolution, another technique would be required. For example, if you don't want to show a user something that is more than 1 week old to the second, then simply exclude that from the SELECT. That is add something like this to the WHERE: AND created_date >= NOW() - INTERVAL 7 DAY.
Doing the above gives you the freedom to schedule the actual DELETE for only, say, once a day -- rather than pounding on the database only to usually find nothing to do.
If you do choose to "pound on the database", be aware of the following problem. If one instance of the deleter script is running for a long time (for any of a number of reasons), it might not be finished before the next copy comes along. In some situations these scripts can stumple over each other to the extent of effectively "crashing" the server.
That leads to another solution -- a single script that runs forever. It has a simple loop:
Do the actions needed (deleting old rows)
Sleep 1 -- or 10 or 60 or whatever -- this is to be a "nice guy" and not "pound on the system".
The only tricky part is making sure that starts up after any server restart or crash of the script.
You can configure a cronjob to periodically delete it.
There are several ways to configure a cron job.
You can write a shell script that periodically deletes entities in the db using linux crontab, or you can configure an application that provides cronjobs such as jenkins or airflow.
AWS lambda also provides cronjob.
Using crontab provided by nestjs seems to be the simplest to solve the problem.
See this link
https://docs.nestjs.com/techniques/task-scheduling
I have a series of complex python / bash scripts doing all sorts of stuff.
Some of them run mysql queries to create a database and tables.
Say i was too lazy to go through all of them to see when and who creates my tables, and i just wanted to pause the script execution at that point.
I expect 2 things at this point:
To know that mysql stopped because a table is being created (or whatever else)
To see in my console that my installation scripts stopped - and spot the script currently running.
So is there a way to pause such types of queries?
In my web app I use two recurring events that "clean up" one of the tables in the database, both executed every 15 minutes or so.
My question is, could this lead to problems in performance in the future? Because I've read somewhere -I don't recall where exactly- that MySQL events are supposed to be scheduled to run once a month or so. Thing is, this same events keep the table in a pretty reduced size (as they delete records older than 15~ minutes), maybe this compensates the frequency of their execution, right?
Also, is it better to have one big MySQL event or many small ones if they are be called in the same frequency?
I don't think there's a performance indication in the monthly base just more of a suggestion of what to do with it. So i think you're ok with doing your cleanup using the events.
In the end the documentation suggets that the events are
Conceptually, this is similar to the idea of the Unix crontab (also known as a “cron job”) or the Windows Task Scheduler.
And the concept for those is that you can run a task every minute if you wish to do so.
On the second part of that question:
Serialize or spread it up. If you split them up into many events that will run at the same time you will create spikes of possibly very high cpu usage that might slow down the application while processing the events.
So either pack everything into one event so it runs in succession or spread the single events up so they execute on different times during the 15 minutes timeframe. Personally i think the first one is to be preferred, pack them up into a single event as then they are guaranteed to run in succession, even if a single one of them keeps running longer than usual.
The same goes for cronjobs. If you shedule 30 long-running exports at a single time your application is going to fail miserably during that timeslot (learned that the hard way).
is there any software that does "lazy" deletion of the rows from the table. I would like to do maintenance of my tables when my server is idle, and ideally i should be able to define what "idle" is (num of database connections/system load/ requests per second). Is there anything remotely similar to this?
If you are on a linux server, you can make your table cleanup scripts only run based on the output of the command "w" which will show you a system load. If your system load is under say .25 you can run your script. Do this with shell scripting.
To some degree, from an internal perspective InnoDB already does this. Rows are initially marked as deleted, but only made free as part of a background operation.
My advice: You can get in to needlessly complicated problems if you try and first check if the server is idle. i.e.
What if it was idle, but the cleanup takes 2 minutes. During that 2 minutes the server load peaks?
What if the server never becomes idle enough? Now you just have an unlimited backlog.
If you just background the task you might improve performance enough, since now at least no users will be sitting in front of web pages waiting for it to complete. Look at activity graphs as to what is the best time to schedule it (3am, 5am etc).