I am trying to create a build script to restore, build, test and code coverage a .net solution and then publish the results to sonarcloud. I have succeeded in most of this, but there is a new dotnet step for collecting code-coverage that has been suggested which can be used as follows:
- name: Install dotnet coverage
run: dotnet tool install --global dotnet-coverage
- name: Coverage
run: dotnet-coverage collect dotnet test --no-build -f xml -o 'coverage.xml'
The step runs absolutely fine, collects code coverage as expected, but it took me a while to realise that because of the nested call to the dotnet test command, this step does NOT fail when a test fails. To me I'm expecting that to be the default behaviour. Now, aside from splitting out the test command into a separate step to force this, is there a way of making the above line fail when a test fails. Right now I've been forced to do this:
- name: Test
run: dotnet test --no-build
- name: Coverage
run: dotnet-coverage collect dotnet test --no-build -f xml -o 'coverage.xml'
Which basically means running the tests twice which seems inefficient. Is there an easy way around this?
You can use collect option for dotnet test command to collect coverage data while test runs. You can also set data collector, result file format, name and directory. For example:
dotnet test --collect:"Code Coverage;Format=Xml;CoverageFileName=coverage.xml" --results-directory ./coverage
Pay attention that although you can specify result directory another one directory(with Guid as name) will automatically be created inside it. Look at this issue.
Related
Ok, so I am trying to split my Appium tests by their timing in CircleCI for running the test in parallel. My tests are written in behave(Python) and I am generating the JUnit XML file. Here is my config.yml file
version: 2.1
orbs:
macos: circleci/macos#2.2.0
jobs:
example-job:
macos:
xcode: 13.4.1
parallelism: 4
resource_class: large
steps:
- checkout
- run:
name: Install appium server
command: |
sudo npm update -g
sudo npm install -g appium
sudo npm install -g wd
- run:
name: Start appium server
command: appium --address localhost --port 4723
background: true
- run:
name: Installing Dependencies
command: pip3 install -r requirements.txt
- run:
name: Test application
command: |
TEST=$(circleci tests glob "features/featurefiles/*.feature" | circleci tests split --split-by=timings --timings-type=classname)
echo $TEST
behave $TEST --junit
- store_test_results:
path: reports
- store_artifacts:
path: ./Logs
destination: logs-file
- store_artifacts:
path: ./screenshots
workflows:
example-workflow:
jobs:
- example-job
When I am running the test, I am getting the error "No timing found for "features/featurefiles/XXX.feature" and it is splitting the test by the filename. It runs well but the split is not happening by the timing.
When the execution is done, I can see the data in the TESTS tab, also in the Timing Tab
I believe CircleCI is not able to read the JUnit file generated by Behave, it is searching for the different JUnit XML file. How can we make CircleCI read the JUnit file generated by Behave?
If anyone face such issue, please take a look at the classname in the JUnit report. There is a format in which CircleCI read the classname. In my case, the classname in the JUnit report was mentioned as
features.featurefiles.Login.feature
But the CircleCI was looking for the classname in below format
features/featurefiles/Login.feature
I had to write a utility to change the classname in the report once the execution was completed.Once it was done, CircleCI was able to read the timings.
Hope it help someone :)
I'm trying to set up a pipeline on Azure DevOps with Cypress tests.
Locally test output file is created correctly.
I'm using npx cypress run command
I'm getting an error/warning for Publish Test Results:
##[warning]No test result files matching **/test-output-*.xml were found.
Here is my cypress.json file:
{
"reporter": "junit",
"reporterOptions": {
"mochaFile": "tests/test-output-[hash].xml",
"toConsole": true,
"attachments": true
},
"video": false,
"pluginsFile": "cypress/plugins/index.js",
"supportFile": "cypress/support/index.js"
}
Here is azure-pipelines.yml:
# Node.js
# Build a general Node.js project with npm.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/javascript
trigger:
- develop
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm install
displayName: 'npm install'
- script:
npx cypress run
displayName: 'Execute cypress tests'
- task: PublishTestResults#2
displayName: "Publish Test Results"
condition: always()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/test-output-*.xml'
searchFolder: '$(System.DefaultWorkingDirectory)'
- task: PublishBuildArtifacts#1
condition: always()
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'drop'
publishLocation: 'Container'
I tried to do all weird stuff, but nothing helps.
Checked all StackOverflow topics like those below:
Azure DevOps test -xml not found after running the Cypress tests
Is there any way to show Cypress Test Results in Azure DevOps Test Results Tab?
Azure DevOps test -xml not found after running the Cypress tests
No test result files were found using search pattern '...\**\TEST-*.xml
Cypress Integration with DevOps
All is looking to be set up correctly according to Cypress documentation and blogs etc.
Maybe test output file is not created on Azure?
Maybe someone has a clue?
EDIT:
I checked using ls -al command, that tests folder is not created.
But even if I created it using mkdir tests before starting cypress the folder is empty after the cypress job.
So Cypress is not creating test output report. Why locally the file is created but on Azure no?
Please check with the following steps:
Set the pipeline variable system.debug to be true, and run the pipeline again.
After the step "Execute cypress tests" is completed, check if you can get more details for troubleshooting from the debug logs on the console window.
You mentioned that the same npx cypress run command can work fine on your local machine, please try to install a self-hosted agent on your local machine to run the pipeline to see if the problem still exists.
If the problem still exists, for us to investigate this problem further, please share the complete logs of the test step with us.
Just had the same issue. Had my artifact download task set on a specific build, so I never get the new build with the correct cypress.config file. Updated the build target and now everything is working. So thanks #DuduA, I thought I'll just answer it so it's a bit easier to see if someone has the same issue.
Please add a command line task to check if cypress run command is running at which folder.
I think you might be running the cypress command in wrong folder. The issue will be resolved if you will provide the correct folder structure and run cypress run command where cypress.json file exists.
Also in cypress.json file check the path of plugins file and support file.
I'm using the Hibernate framework along with Maven in IntelliJ. I'm creating a MySQL database, I also have some ORM classes that map the MySQL database, and then I'm running some JUNIT tests to make sure everything works.
Where I'm having trouble is in two places, which are related to each other:
When I run mvn test, sometimes my JUNIT tests work fine and are
able to query the simulated database, establish a connection (even
though it's just with the simulated database), execute a statement,
etc. However, sometimes, if I run mvn clean before running mvn test,
while the JUNIT tests still execute, the tests output with failures
(not errors, just failures, thought this is still bad, of course).
The problem outlined in #1 is essentially duplicated when I upload to GitHub and run CircleCI (which isn't surprising, since CircleCI runs mvn test when doing its integration testing). Most of my uploads failed, but one of them finally worked. However, I'm not exactly sure why the "final" upload was successful while the others weren't.
The error messages I'm getting either from mvn test or the CircleCI builds are typically as follows. These errors are from my pent-ultimate upload, the one I did just prior to the next upload which actually worked:
java.sql.SQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
com.mysql.cj.exceptions.CJException: Public Key Retrieval is not allowed
java.sql.SQLNonTransientConnectionException: Could not create connection to database server
I should also note that my intention is to run mvn clean first, then upload to CircleCI, however, running mvn clean seems to be somehow involved in perpetuating these errors.
As far as different resources I'm using go, here they are. If I'm forgetting something, please let me know and I should be able to include it.
In my hibernate.cfg.xml file, I have the following lines:
<property name="connection.url">jdbc:mysql://localhost:3306/stocks</property>
<property name="connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="hibernate.dialect">org.hibernate.dialect.MySQLDialect</property>
At the end of the word "stocks" on the first line, I have sometimes appended any of the following (sometimes I only appended one of the following, other times I combined them, depending on the error(s) from either Maven or CircleCI). Appending some combination of these lines seemed to help get things to work, but running mvn clean seemed to halt any effect these additions were having:
autoReconnect=true
useSSL=false
allowPublicKeyRetrieval=true
Running the JUNIT tests from within IntelliJ usually works, but if I run mvn clean first, then IntelliJ usually won't work, unless I then go back into this file and append ?autoReconnect=true&useSSL=false. If I do that, then IntelliJ will run the JUNIT tests fine.
In my config.yml file for CircleCI, I have the following code. Certain statements were added in MAVEN_OPTS based on other research I did to try to counteract the errors I was getting, but I don't know if these statements are having any impact one way or the other:
# Java Maven CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-java/ for more details
#
version: 2
jobs:
build:
docker:
# specify the version you desire here
- image: circleci/openjdk:8-jdk
# Specify service dependencies here if necessary
# CircleCI maintains a library of pre-built images
# documented at https://circleci.com/docs/2.0/circleci-images/
# - image: circleci/postgres:9.4
- image: circleci/mysql:latest-ram
environment:
- MYSQL_ROOT_PASSWORD: (my real password goes here)
- MYSQL_DATABASE: stocks
- MYSQL_USER: bob
- MYSQL_PASSWORD: (the real password goes here)
working_directory: ~/repo
environment:
# Customize the JVM maximum heap limit
MAVEN_OPTS: -Xmx3200m -Dmaven.wagon.http.ssl.insecure=true -Dmaven.wagon.http.ssl.allowall=true -Dmaven.wagon.http.ssl.ignore.validity.dates=true
steps:
- checkout
- run: sudo apt install -y mysql-client
# Download and cache dependencies
- restore_cache:
keys:
- v1-dependencies-{{ checksum "pom.xml" }}
# fallback to using the latest cache if no exact match is found
- v1-dependencies-
- run: mvn dependency:go-offline
- save_cache:
paths:
- ~/.m2
key: v1-dependencies-{{ checksum "pom.xml" }}
# run tests!
- run: mvn integration-test
If anyone has any idea what's going on, I appreciate the help. My goal is to be able to upload to CircleCI by first running mvn clean so only the src files, pom.xml file, and .circleci folder are included in the upload. Also, not to belabor the point, but my most recent upload to CircleCI did in fact work, but I'm not sure what made that build work while all the others ones did not.
I've got a github project which I want to integrate with travis. Currently I execute my tests with:
phpunit tests
command. I would like to make the tests execute with:
phpunit
since travis needs it. I guess it's a matter of phpunit.xml.dist file, but I can't work out how to set it. Any help would be appreciated
This should tell you the answer: https://github.com/travis-ci/travis-ci-php-example/blob/master/.travis.yml
In case that link ever ceases to exist, in your .travis.yml include the following lines:
# omitting "script:" will default to phpunit
# use the $DB env variable to determine the phpunit.xml to use
script: phpunit --configuration phpunit_$DB.xml --coverage-text
I have a bunch of Maven projects building in Hudson with Sonar sitting in the side-lines. Sonar gives me Sonar stats, FindBugs stats, and code-coverage.
I've noticed that regardless of if I use Sonar or if I use EMMA via Maven directly, the entire build cycle runs twice. This includes init (which in my case, reinitializes the database -- expensive) and unit tests (a few hundred -- also expensive).
How can I prevent this? I did a lot of reading, and it seems like this is due to the design of code-coverage plugins -- to keep uninstrumented classes separated from instrumented ones.
I've tried configurations like:
Maven runs: deploy, EMMA
Maven runs: deploy; deploy to Sonar on completion
The sonar documentation recommends running the sonar plugin in 2 stages:-
mvn clean install -Dtest=false -DfailIfNoTests=false
mvn sonar:sonar
The tests are bypassed in the first phase and run implicitly in the second stage.
A one line alternative is to run the following command:-
mvn clean install sonar:sonar -Dmaven.test.failure.ignore=true
but this will run the tests twice - as you have found.
To add to #Strawberry's answer, you could reuse the unit test reports instead of running them again. Refer to the section Reuse existing unit test reports in the sonar documentation
Once this is done, you can configure the following in Hudson
clean deploy sonar:sonar