I have two Jenkins projects that share a database. They must not be run simultaneously. Strictly speaking, there is no particular dependency between them beyond non concurrency, but at the moment I partially manage this constraint by running one "downstream" of the other. This works most of the time, but not always. If a source control change happens while the second is running, the first will start up again, and they'll be running concurrently and probably both fail miserably.
This is similar, but not identical, to How to prevent certain Jenkins jobs from running simultaneously? The difference is that I don't have a "number of threads" problem -- I'm already only running at most one thread of any given project at any one time, even in the case where two (different-project) builds stomp each other. This seems to rule out all the several suggestions in that thread.
The Locks and Latches plugin should resolve your problem. Create a lock and have both jobs use the same lock. That will prevent the jobs from running concurrently.
Install the plugin in "Manage Jenkins: Manage Plugins."
Define (provide a name for) your lock(s) in "Manage Jenkins: Configure System."
For each job you want to participate in the exclusion,
in ": Configure: Build Environment," check "Locks",
and pick your lock name from the drop list.
The Lockable Resources Plugin. Simple and working well for me May 2016.
Install the plugin.
In Manage Jenkins > Configure System go to Lockable Resources Manager.
Select Add Lockable Resource.
Enter values for field: Name and hit Save.
Warning: Do not enter spaces in Name field.
In Jenkins > job_name > Configure > General,
Select checkbox: This build requires lockable resources.
Enter name or names in value for field: Resources.
Start a build.
Under build #number select Locked Resources.
You should see something like:This build has locked the following resources: resource_name - resource_description.
Start a different build which uses the same resource.
You will see Build Queue in Jenkins status/menu showing job name.
Hover text shows Started by, Waiting for resources resources_list, Waiting for time.
(also resource tags/labels can be used)
Adding screenshot of Job Configuration page as there seems to be a problem for some users where "This build requires lockable resources" is not visible: ** when the checkbox is not selected you should only see "[_] This build requires lockable resources"
EDIT: Below information is effective as of 04/10/2014
Exclusion plugin, https://wiki.jenkins-ci.org/display/JENKINS/Exclusion-Plugin Very useful if few build use the same resource - e.g. a test database. All you need to do is to update configuration of all jobs using this resource and as a result they will never run in parallel but wait for others to complete.
Taken from : http://www.kaczanowscy.pl/tomek/2012-07/jenkins-plugins-part-iii-towards-continuous-delivery
This plugin does block two or more jobs from running in parallel.
To test, do this for job1
Configure
Under Build Environment check "Add resource to manage exclusion."
Then Add -> New Resource -> Name -> lock
Under Build -> Add build step
Critical Block Start
Add build step -> Add whatever you want to add.(add sleep 15 to make sure it lasts longer to check concurrency.)
Add build step -> Critical block end
Repeat the above steps for job2, make sure you use the same lock name 'lock'.
manually build both jobs concurrently.
Monitor the run progress under jenkins -> Exclusion administration.
1 December 2021
Use Build Blocker plugin, Install from Manage Jenkins > Plugin Manager
For example, you have two pipelines React-build and React-tests:
Go to React-build -> Configure -> Block build
if I don't need React-tests to run concurrently with the current React-build job, add it in the blocking list,
Regex expressions can also be used, i.e. to avoid concurrent builds for all projects starting with React-, add React-.* to the list,
Replace React-tests with any pipeline-name you want not to run parallel, with global or node level options,
When tried to run any blocked jobs together with configured React-build job, it gets moved to pending state,
Related
I have two pipelines that I want to run with the same runner, is it possible?
my single runner installed on a Linux virtual machine and I want to use it to run all my pipelines.
If the pipelines are for different projects you will need to make sure the runner is accessible to each project.
Depending on the level of control you want, you can utilise gitlab cis keyword for tags, this will then enable you to determine which runner handles which pipeline.
If you want to run the jobs in parallel you will need to make sure the runner is enabled for concurrent running and also that the jobs are in the same stages within the pipelines.
The only way to do this at the moment is to define one tag for one runner only and user this tag for your project.
This way everything is run on this single runner.
This has of course the disadvantage that the load is not spread to different runners, so be careful.
You could improve this solution if you a child pipeline to get a new free runner tag and create a child pipeline using it.
There are active issues about this problem in gitlab, see this one and this.
There is a forum entry about it as well.
In a Github action workflow, is there a way to access the "Set up Job" -> "Virtual Environment" values? Ideally, I'd like to get them from variables already present, but getting them from the output of a command would be just fine too.
I can see the values in the Github actions UI by clicking on a specific job, then "Set up Job", then expending "Virtual Environment".
Virtual Environment
Environment: ubuntu-20.04
Version: 20220425.1
Included Software: https://github.com/actions/virtual-environments/blob/ubuntu20/20220425.1/images/linux/Ubuntu2004-Readme.md
Image Release: https://github.com/actions/virtual-environments/releases/tag/ubuntu20%2F20220425.1
I'd like to create a cache key based on that info so that, when it changes, the cache is made anew.
Background:
I have some tests that run on both ["ubuntu-latest", "macos-latest"]. I also have a setup action that in part builds some shared libraries (which is slow) and caches them. The shared libraries are external though and only need to be rebuilt for updates to the runner image (or for new versions of the libraries, which isn't a concern). The current cache key is ${{ runner.os }}-${{ needs.setup.outputs.cache-key-suffix }}, which only ever changes when we bump the cache-key-suffix hard-coded string. Using that key, the cache is shared among all the workflows and branches, saving lots of needless duplicated work.
Deep Background:
The specific problem I'm solving is that one of those shared libraries is Rocks DB. The tests were working fine for a while, but recently stopped; most tests using Rocks DB started failing with signal: illegal instruction (core dumped). The only thing that might have changed is an update to ubuntu-latest. So I figure that it'd be nice to automatically have the cache recreated when that happens.
I've tried digging through the Github Actions: Variables and Github Actions: Contexts documentation, and some general Google-based research, but haven't been able to find a way to get those values for use in a workflow.
Recently I created a framework with Cucumber-JUnit where I am able to execute Scenarios in parallel (For now keeping one scenario per feature) without any issue.
Now I have a situation where some of the features has to run in parallel and some in sequence.
Is there any way that we can control with tags or any other configuration to choose which has to run in parallel or sequence?
Let me put some overview how it is currently
Parallel and its thread size controlled as per cucumber official documentation - Maven surefire
pom.xml
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M5</version>
<configuration>
<parallel>methods</parallel>
<threadCount>${threadSize}</threadCount>
<perCoreThreadCount>false</perCoreThreadCount>
</configuration>
</plugin>
CucumberRunner:
#RunWith(Cucumber.class)
#CucumberOptions(
features = {"src/test/resources/features"},
glue = {"com.tests.binding.steps"},
tags = "#regression"
)
public class RunCucumberFeatures {
}
command using to run tests
mvn clean test -Dcucumber.filter.tags="${toExecute} and not (#smoke)" -DthreadCount=${ThreadSize} -Dcucumber.execution.dry-run="false"
For toExecute parameter - we pass multiple tags like #customerClaim or #employeeClaim
Now, In my case features with tags #employeeClaim should execute in parallel and tags with #customerClaim should execute in sequence.
Is it possible with current design or any other way?
Is there any way that we can control with tags or any other configuration to choose which has to run in parallel or sequence?
Not with cucumber-junit and JUnit 4. However with JUnit 5 you can use the cucumber-junit-platform-engine and use JUnit 5s support for exclusive resources.
https://github.com/cucumber/cucumber-jvm/tree/main/junit-platform-engine
To synchronize a scenario on a specific resource, the scenario must be tagged and this tag mapped to a lock for the specific resource. A resource is identified by an arbitrary string and can be either locked with a read-write-lock, or a read-lock.
For example, the following tags:
Feature: Exclusive resources
#reads-and-writes-system-properties
Scenario: first example
Given this reads and writes system properties
When it is executed
Then it will not be executed concurrently with the second example
#reads-system-properties
Scenario: second example
Given this reads system properties
When it is executed
Then it will not be executed concurrently with the first example
with this configuration:
cucumber.execution.exclusive-resources.reads-and-writes-system-properties.read-write=java.lang.System.properties
cucumber.execution.exclusive-resources.reads-system-properties.read=java.lang.System.properties
when executing the first scenario tagged with #reads-and-writes-system-properties will lock the java.lang.System.properties resource with a read-write lock and will not be concurrently executed with the second scenario that locks the same resource with a read lock.
Note: The # from the tag is not included in the property name. Note: For canonical resource names see junit5/Resources.java
So by making an exclusive resource for #customerClaim you can prevent these scenarios from running in parallel. However to the best of my knowledge JUnit 5 makes no guarantees made about the order of execution so they should still be independent from each other.
It would be nice if we could select which tests to run in parallel only by tags. I will be fighting with microservices very soon and I would like most of my tests to run in parallel to save time. So far I have been testing a monolith and my framework is already done with multiple .feature files and tags configured to run on different environments. So if there is an easy way to configure which tests to run in parallel and the rest in sequence using tags - it would be nice to know.
I have the following in my system:
4 File folders
5 Applications that do some processing on files in the folders and then move files to the next folder (processing: read files, update db..)
The process is defined by Stages: 1,2,3,4,5.
As the files are moved along, the Stage field within them is updated to the next Stage.
Sometimes there are exceptions in the system, not necessarily exception in code but exception in the process.
For instance, there is an error in transmitting the file to the next folder. In this case the stage is not updated and an record is written in the DB for this file.
What I want to do, what is the best approach?
I want to plug a utility of some sort or add code to the applications that will capture any exceptions in the process. Like if a file was not moved, I want to know what stage and why. This will help in figuring out the break down in the process.
I need something that will provide the overall health of the process.
Now sure how to go about doing this from an architectural point of view.
The scheduler? Well that might knock the idea out anyway.
Exit code is still up and running from dos days.
it's a property of the Application Class (0 the default) is success
So from your app you'd detect an error and set ApplicationExitCode to some meaning number like 1703 (boo hoo)
Application.ShutDown(1703);// is the .net4 way
However seeing as presumably the scheduler is just running the app, you'd have to script it all up. Might as well just write a common logging dll and add it to each app as mess about with that, especially if you want the same behaviour if it's run from outside the scheduler.
Another option would be delegating. ie you write an app that runs the app (passed in as a command line parameter) and logs the result (via exit code for instance) and then change scheduler items to call that with the requisite parameter.
I am new at this and I was wondering how I can setup that I save the artifacts, only if less than 90% of the tests have passed.
Any idea how I can do this?
thanks
This is not currently possible with Hudson. What is the motivation to avoid archiving artifacts on every build?
How about a rather simple workaround. You create a post build step (or additional build step) that calls your tests from the command line. Be sure to capture all errors so Hudson don't count it as a failure. Than you evaluate your condition and set the error level accordingly. In addition you need to save reports (probably outside hudson) before you set the error level, so they are available even or only when the build fails.
My assumption here is, that it is OK, not to run the tests when building the app fails. However, you can separate the building and testing in two jobs. See here.