I have created a simple workflow in Github Actions that runs daily via a cron trigger. On each run it executes some SQL code that uses CURRENT_DATE, which then is the date of the run. However, if the job fails and I have to rerun it on a later date, it will (logically) run with a different date than the first time. In Airflow it can be solved by using execution_date and in BigQuery's scheduled tasks there's the #run_time variable that is used on retries.
Is there any way I can persist settings used in failed jobs and use again?
Related
I have to develop/use a scheduler in my NodeJS project to check if any subscriptions need to be renewed. Job needs to run once every day and check my database if 'enddate' of a any user subscription record falls on that day. If it is, run a piece of logic to renew it (i have my own renew logic which doesnt depend on any external services like payment providers/gateways).
Application will be in cloud which means all my app instances have to be stateless. Database (MySQL) is the only stateful part of the system and Renew logic has to run only once.
How can i do this ?
My database table row for a subscription looks like this:
Id: 1, User: John Doe, Plan: Monthly Premium, Start Date: 1-10-2015, End Date: 30-10-2014, status : "ACTIVE"
Problems:
1.) Trigger job running logic everyday - Easy to solve in nodejs with setInterval or a scheduler library
2.) Running a worker or a piece of code that checks the db and renews it - This needs to happen only once per user subscription. Many instances can potentially run the renew logic at the same time when daily scheduler triggers the process. Not sure how to solve this part of the problem.
One possible solution consists of a shell script, put in your crontab, that would run each day.
This script would do :
Using a sql statement or stored procedure, find records where subscription
end date is today
For those records, apply your logic. (send email or anything else)
Flag the record to indicate that a something has been done
for this user (and put in another field the process date).
you can create a worker and use node-cron to trigger subscription expire everyday.
i'm not sure what renew process does. you can use lock something until the renew process complete and release it. I was thinking i would use redlock
You can use Agenda, but it requires a MongoDb to maintain its state. If you are okay with adding one more DB, then you can give this a shot.
I have a set of independent SSIS packages say A,B,C. I'm running them manually and in parallel using DTEXEC command. Finishing time of these jobs can be random i.e, there is no certainty that only a particular job finishes last every time.
Now I want to send a notification mail when all the packages are completed. How can I accomplish this without modifying the packages? Also I may not be able to use task scheduler or SQL agent.
Users of my application need to be able to schedule certain task to run at certain times (e.g. once only, every every minute, every hour, etc.). My plan is to have a cron run a script every minute to check the application to see if it has tasks to execute. If so, then execute the tasks.
Questions:
Is the running of cron every minute a good idea?
How do I model in the database intervals like cron does (e.g. every minute, ever 5th minute of every hour, etc.)?
I'm using LAMP.
Or, rather than doing any, you know, real work, simply create an interface for the users, and then publish entries in cron! Rather than having cron call you every minute, have it call scripts as directed by the users. When they add or change jobs, rewrite the crontab.
No big deal.
In unix, cron allows each user (unix login that is) to have their own crontab, so you can have one dedicated to your app, don't have to use the root crontab for this.
Do you mean that you have a series of user-defined jobs that need executed in user-defined intervals, and you'd like to have cron facilitate the processing of those jobs? If so, you'd want to have a database with at least 2 fields:
JOB,
OFTEN
where OFTEN is how often they'd like the job to run, using syntax similar to CRON.
you'd then need to write a script (in python, ruby, or some similar language) to parse that data. this script would be what runs every 1 minute via your actual cron.
take a look at this StackOverflow question, and this StackOverflow question, regarding how to parse crontab data via python.
I have a job NIGHTLY that runs one time each night by a periodical timer.
Now I want to change so that the NIGHTLY job is only run if the last execution of another job FOO in Hudson is successful.
Note:
Job FOO is run many times each day and is triggered by SCM.
NIGHTLY should only be run one time per night and at a specific time.
Currently I have another job NIGHTLY_TRIGGER that runs a bash script that access the remote API of job FOO to check if job FOO is successful and if so triggers the NIGHTLY job.
Are there a nicer/cleaner way to do this? (preferable by using some Hudson plugins)
You could check out the Hudson Join Plugin which is made for this kind of scenario (wait for the conclusion of a job before executing another one).
The end result wouldn't be much different from what you are already doing, but this would be neatly parameterized:
So you would still have to check the status of FOO job, but at least you would check it right after FOO job completion.
I have a job A that is run every hour. Also job B is run after each commit to github (integration tests job). How can i know before running job A if last build of job B was successful and discard build of A if last build of B was unstable?
Thanks.
As far as I know, this is not possible when you use hudson out of the box. Without any specifics about your job dependencies it is also not easy to design the right workaround.
Different Options:
If your job A runs fast, let it run anyway.
Since job A runs every hour, can you go away with job B running every hour. In this case Job B is successful it will trigger job A.
Have an external shell script that triggers job A every hour. Before triggering, check the status of your last build from job B (http:///job//api/xml?xpath=/mavenModuleSetBuild/result/text%28%29). For info on how to trigger a build have a look at the "Trigger builds remotely" option in your job.
This list is probably not exhaustive.