CircleCI is unable to read the JUnit xml generated by Behave while splitting the test by timings - junit

Ok, so I am trying to split my Appium tests by their timing in CircleCI for running the test in parallel. My tests are written in behave(Python) and I am generating the JUnit XML file. Here is my config.yml file
version: 2.1
orbs:
macos: circleci/macos#2.2.0
jobs:
example-job:
macos:
xcode: 13.4.1
parallelism: 4
resource_class: large
steps:
- checkout
- run:
name: Install appium server
command: |
sudo npm update -g
sudo npm install -g appium
sudo npm install -g wd
- run:
name: Start appium server
command: appium --address localhost --port 4723
background: true
- run:
name: Installing Dependencies
command: pip3 install -r requirements.txt
- run:
name: Test application
command: |
TEST=$(circleci tests glob "features/featurefiles/*.feature" | circleci tests split --split-by=timings --timings-type=classname)
echo $TEST
behave $TEST --junit
- store_test_results:
path: reports
- store_artifacts:
path: ./Logs
destination: logs-file
- store_artifacts:
path: ./screenshots
workflows:
example-workflow:
jobs:
- example-job
When I am running the test, I am getting the error "No timing found for "features/featurefiles/XXX.feature" and it is splitting the test by the filename. It runs well but the split is not happening by the timing.
When the execution is done, I can see the data in the TESTS tab, also in the Timing Tab
I believe CircleCI is not able to read the JUnit file generated by Behave, it is searching for the different JUnit XML file. How can we make CircleCI read the JUnit file generated by Behave?

If anyone face such issue, please take a look at the classname in the JUnit report. There is a format in which CircleCI read the classname. In my case, the classname in the JUnit report was mentioned as
features.featurefiles.Login.feature
But the CircleCI was looking for the classname in below format
features/featurefiles/Login.feature
I had to write a utility to change the classname in the report once the execution was completed.Once it was done, CircleCI was able to read the timings.
Hope it help someone :)

Related

How to fail github action using the new dotnet code-coverage command

I am trying to create a build script to restore, build, test and code coverage a .net solution and then publish the results to sonarcloud. I have succeeded in most of this, but there is a new dotnet step for collecting code-coverage that has been suggested which can be used as follows:
- name: Install dotnet coverage
run: dotnet tool install --global dotnet-coverage
- name: Coverage
run: dotnet-coverage collect dotnet test --no-build -f xml -o 'coverage.xml'
The step runs absolutely fine, collects code coverage as expected, but it took me a while to realise that because of the nested call to the dotnet test command, this step does NOT fail when a test fails. To me I'm expecting that to be the default behaviour. Now, aside from splitting out the test command into a separate step to force this, is there a way of making the above line fail when a test fails. Right now I've been forced to do this:
- name: Test
run: dotnet test --no-build
- name: Coverage
run: dotnet-coverage collect dotnet test --no-build -f xml -o 'coverage.xml'
Which basically means running the tests twice which seems inefficient. Is there an easy way around this?
You can use collect option for dotnet test command to collect coverage data while test runs. You can also set data collector, result file format, name and directory. For example:
dotnet test --collect:"Code Coverage;Format=Xml;CoverageFileName=coverage.xml" --results-directory ./coverage
Pay attention that although you can specify result directory another one directory(with Guid as name) will automatically be created inside it. Look at this issue.

Unable to resolve jar dependency in Java project for

I have 2 java projects on GitHub, the first project produces a java library and the second project uses it.
The first project is build via first job in yaml file and it installs the library in /home/runner/.m2/repository/ successfully
Then when the second project is build via the second job in the same yaml file , it fails with error
Failed to execute goal on project <>: Could not resolve dependencies
for project Could not find artifact <>:jar:1.0-
SNAPSHOT -> [Help 1]
In my second project, I have included the dependency to load this library jar(produced by first project) from the repository.
MY understanding is that my second project should load the jar library from the /home/runner/.m2/repository/ but its not loading it.
Any help is appreciated.
Also can we access /home/runner/.m2/repository/ to check if it contains the library jar.
./project_account_library/account_library => Contains maven project
./project_account_service/account_service => Contains maven project
./.github/workflows/build.yml
Here is the yml file
name: Build account service
on:
push:
branches: [main]
env:
IMAGE_NAME: accountservice
jobs:
build_dependency:
runs-on: ubuntu-latest
permissions:
packages: write
contents: read
steps:
- uses: actions/checkout#v2
- uses: actions/setup-java#v2
with:
distribution: 'temurin' # See 'Supported distributions' for available options
java-version: '17'
- name: Build Dependency
run: |
cd ./project_account_library/account_library
mvn clean install
build_accountservice:
runs-on: ubuntu-latest
permissions:
packages: write
contents: read
steps:
- uses: actions/checkout#v2
- uses: actions/setup-java#v2
with:
distribution: 'temurin' # See 'Supported distributions' for available options
java-version: '17'
- name: Build Account Service
run: |
cd ./project_account_service/account_service
mvn clean install
First Step
[INFO] Installing /home/runner/work/<>/<>/target/account_
library_1.0-SNAPSHOT.jar to
/home/runner/.m2/repository/com/<>/<>/account_library/1.0-
SNAPSHOT/account_library-1.0-SNAPSHOT.jar
Second Step
Error: Failed to execute goal on project <>: Could not resolve
dependencies for project <>: Could not find artifact com.<>.
<>:account_library:jar:1.0-SNAPSHOT -> [Help 1]
Thanks Frennky
I now used only 1 job and below is my new .yml file
- name: Build Account Service
run: |
cd ./project_account_library/account_library
mvn clean install
cd ../../project_account_service/account_service
mvn clean install
First directory is successfully built but it fails to cd to ../../project_account_service/account_service and fails to build with below error
/home/runner/work/_temp/ea229141-b0c3-455d-82f0-12ff588d420a.sh: line 12: cd ../../project_account_service/account_service: No such file or directory
The reason you're missing dependencies on your second job is because those 2 jobs, by default, run in parallel and they do so on 2 different runners. From docs:
Each job will run inside its own virtual machine runner, or inside a
container...
There's few ways you can go around this. I can suggest one quick and easy, and another that would probably be a way to go, but require slightly more effort.
Easy way out of this is to put all steps in a single job.
Better way would be to make use of Github Package repository, where
you could push your lib/dependencies and then resolve them for your
service.
For more details on Github Package repository you can check docs.
Update:
Ok, I've reread your new error and I think I understand the issue.
If I got it right your dir structure is as follows:
repo
project_account_library
account_library
project_account_service
account_service
Each run step actually starts at root of you repo. You can use working-directory like so:
# ...
- name: Build Dependency
run: mvn clean install
working-directory: ./project_account_library/account_library
# ...
- name: Build Account Service
run: mvn clean install
working-directory: ./project_account_service/account_service

Snyk test returns Failed to test pip project error

I'm running security scan with Snyk CLI for python project. Unfortunately snyk test command returns Failed to test pip project error. What am I missing, snyk test works just fine while scanning npm project ?
$ snyk test --file=requirements.txt
Failed to test pip project
I found the cause basically you need to do two things:
Make sure that packages that your project uses are installed.
Make sure that you are using correct python.
Solution
$ pip3 install -r requirements.txt
$ snyk test --file=requirements.txt --command=python3
Info
You can bypass missing python packages by passing the --allow-missing pip parameter through snyk by using the additional -- argument.
$ snyk test --file=requirements.txt --command=python3 -- --allow-missing
Docs
-- [COMPILER_OPTIONS]
Pass extra arguments directly to Gradle or Maven. E.g. snyk test
-- --build-cache
Python options
--command=COMMAND
Indicate which specific Python commands to use based on Python
version. The default is python which executes your systems de-
fault python version. Run 'python -V' to find out what version
is it. If you are using multiple Python versions, use this pa-
rameter to specify the correct Python command for execution.
Default: python Example: --command=python3
snyk monitor command will also return undefined if it is not ran with
pip3 install -r requirements.txt
snyk test --file=requirements.txt --command=python3
snyk monitor --file=requirements.txt --command=python3
If you are using Snyk and VScode, and you open a repo that has a Python VirtualEnv, you can get this error in your VScode terminal window.
[Error] Open Source Security test failed for "/home/{user}/path/to/repo". Failed to test pip project
Fix for VScode:
Close that VScode window.
From a terminal, navigate to the top folder of that repo.
Run the command to activate the virtual env
Example: . .venv/bin/activate
Open VScode for that folder
Example: run code .
The Snyk Open Source Security test should run without that error now.
If you are using virtual environments, then make sure you have activated the venv with
. venv/Scripts/activate
Then try running Snyk Test again.
Snyk monitor and other cli commands should work from that! :)

Chocolatey fails to install visualstudio2017-workload-vctools in GitHub Actions

We have a GitHub Action that uses Chocolatey to install visualstudio2017-workload-vctools to do a Windows build. Here's what the relevant YAML looks like:
steps:
- uses: actions/checkout#v1
- name: Set up GraalVM (Java 11)
uses: DeLaGuardo/setup-graalvm#8bbfe44ef9c6f5c07e5af036a1bffd561c037d18
with:
graalvm-version: '20.3.0.java11'
- name: Install native-image
run: gu install native-image
shell: cmd
- name: Set up Visual C Build Tools Workload for Visual Studio 2017 Build Tools
run: |
choco install visualstudio2017-workload-vctools
- name: Configure Pagefile
uses: al-cheb/configure-pagefile-action#v1.2
- name:
run: |
(Get-CimInstance Win32_PageFileUsage).AllocatedBaseSize
A few days ago, choco install visualstudio2017-workload-vctools started resulting in an error and we've been unable to fix it.
visualstudio2017buildtools has been installed.
The install of visualstudio2017buildtools was successful.
Software install location not explicitly set, could be in package or
default install location if installer.
visualstudio2017-workload-vctools v1.3.2 [Approved]
visualstudio2017-workload-vctools package files install completed. Performing other installation steps.
ERROR: Unable to detect any supported Visual Studio product. You may try passing --installPath or both --productId and --channelId parameters.
The install of visualstudio2017-workload-vctools was NOT successful.
Error while running 'C:\ProgramData\chocolatey\lib\visualstudio2017-workload-vctools\tools\ChocolateyInstall.ps1'.
See log for details.
We tried pinning the version, but that doesn't seem to help.
Full build failure here.
Workaround: Use the windows-2016 image instead of windows-latest / windows-2019 image.
I've also filed a related bug report

Azure DevOps PublishTestResults not found junit cypress test result file

I'm trying to set up a pipeline on Azure DevOps with Cypress tests.
Locally test output file is created correctly.
I'm using npx cypress run command
I'm getting an error/warning for Publish Test Results:
##[warning]No test result files matching **/test-output-*.xml were found.
Here is my cypress.json file:
{
"reporter": "junit",
"reporterOptions": {
"mochaFile": "tests/test-output-[hash].xml",
"toConsole": true,
"attachments": true
},
"video": false,
"pluginsFile": "cypress/plugins/index.js",
"supportFile": "cypress/support/index.js"
}
Here is azure-pipelines.yml:
# Node.js
# Build a general Node.js project with npm.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/javascript
trigger:
- develop
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm install
displayName: 'npm install'
- script:
npx cypress run
displayName: 'Execute cypress tests'
- task: PublishTestResults#2
displayName: "Publish Test Results"
condition: always()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/test-output-*.xml'
searchFolder: '$(System.DefaultWorkingDirectory)'
- task: PublishBuildArtifacts#1
condition: always()
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'drop'
publishLocation: 'Container'
I tried to do all weird stuff, but nothing helps.
Checked all StackOverflow topics like those below:
Azure DevOps test -xml not found after running the Cypress tests
Is there any way to show Cypress Test Results in Azure DevOps Test Results Tab?
Azure DevOps test -xml not found after running the Cypress tests
No test result files were found using search pattern '...\**\TEST-*.xml
Cypress Integration with DevOps
All is looking to be set up correctly according to Cypress documentation and blogs etc.
Maybe test output file is not created on Azure?
Maybe someone has a clue?
EDIT:
I checked using ls -al command, that tests folder is not created.
But even if I created it using mkdir tests before starting cypress the folder is empty after the cypress job.
So Cypress is not creating test output report. Why locally the file is created but on Azure no?
Please check with the following steps:
Set the pipeline variable system.debug to be true, and run the pipeline again.
After the step "Execute cypress tests" is completed, check if you can get more details for troubleshooting from the debug logs on the console window.
You mentioned that the same npx cypress run command can work fine on your local machine, please try to install a self-hosted agent on your local machine to run the pipeline to see if the problem still exists.
If the problem still exists, for us to investigate this problem further, please share the complete logs of the test step with us.
Just had the same issue. Had my artifact download task set on a specific build, so I never get the new build with the correct cypress.config file. Updated the build target and now everything is working. So thanks #DuduA, I thought I'll just answer it so it's a bit easier to see if someone has the same issue.
Please add a command line task to check if cypress run command is running at which folder.
I think you might be running the cypress command in wrong folder. The issue will be resolved if you will provide the correct folder structure and run cypress run command where cypress.json file exists.
Also in cypress.json file check the path of plugins file and support file.