I've got files stored in my workspace as:
selenium-tests/results/results-0.xml
selenium-tests/results/results-1.xml
selenium-tests/results/results-2.xml
selenium-tests/results/results-3.xml
selenium-tests/results/results-4.xml
I'm publishing JUnit test results with the fileset pattern: **/results/results-*.xml. All five files aren't present in the test results. It's also never the same result files included.
These are three runs back to back to back with the same exact configuration and tests. They're failing because our staging environment is down right now.
I've even gone as far as **/results/results-0.xml, **/results/results-1.xml, **/results/results-2.xml, **/results/results-3.xml, **/results/results-4.xml and am still having the same issues.
Related
We have .NET Core 3 codebase and the unit and integration tests are ran on the Gitlab CI.
Problem is, when one or more unit/int. tests fail, nothing specific is shown, you have to look at the entire pipeline dump and search to see for individual failed tests.
Looking at https://docs.gitlab.com/ee/ci/unit_test_reports.html, Junit report is exactly solving this issue.
Consulting with the How to capture structured xUnit test output in Gitlab CI?, I still wasn't able to find a proper solution.
Main problem is, there are multiple test projects that are executed with the dotnet test command:
current snapshot of gitlab.yaml file:
artifacts:
when: always
reports:
junit: ./Test.xml
script:
- for proj in $(dotnet sln MySolution.sln list | grep 'Test.csproj$'); do dotnet test --logger "junit;LogFilePath=Test.xml" $proj; done
Now the problematic is the script part, where we iterate through all the test assemblies and do the dotnet test for each project.
Is there a way to somehow produce a single junit xml log file out of each project and feed it to junit test report in the
reports:
junit: ./Test.xml
line?
You don't have to combine the reports. The artifacts:reports:junit key accepts multiple values, including glob patterns.
artifacts:
reports:
junit:
- "test.xml"
- "./tests/*.xml"
So, one solution would be to have all your XML output files in a particular directory and use a glob pattern in your .gitlab-ci.yml file that matches the many files.
If you really want to merge the xUnit XML files isntead, see this answer
I'm using Cypress to run a suite of automated tests.
The current version of cypress provides mocha-junit-reporter out of the box, and provides configuration options to pass to the reporter. One of the options is the 'mochaFile'.
I'm using the recommended [hash] tag to output reports across multiple spec files.
this results in a flat mess of files that look like 'results/test-output-abc12345.xml'.
What I want instead is for the test file's relative path and filename to be pass in as the reporter's output file path.
This would give me a structured, feature first view of the output, and in Azure Dev Ops, which aggregates the test output, it would give me correct filenames to detect intermittently failing tests.
Things I've tried that haven't worked:
I've tried to use hooks to modify Cypress's config or set environment variables to try to override the reporterOptions/mochaFile per test at (hopefully) the right time.
I've tried to grab the outputted defaultly-named xml file, and copy it to the correct path+filename given the Cypress.spec.name context, but I can't seem to find the right hook or time to do this.
after and afterEach don't work - I don't think the test report has saved the file yet.
Using a plugin, hooking to some event on test:before:run or test:after:run seem promising, but I'm flying blind since I can't debug into it, so I've been unsucessful in modifying the reporter's output path or copying the file.
I'd love it if someone could show a working example using mocha-junit-reporter, or even a different mocha compatible reporter, if the reporter would play well with Azure Dev Ops, and can help me discover intermittently failing tests.
Need to run a job in Jenkins after successfully running the tests it needs to perform as post-build action for generating test reports
For this i have configured
Publish JUnit test result report
In the field
Test Report XMLs: continuum/*/target/surefire-reports/*TestSuite.xml
'continuum/*/target/surefire-reports/TestSuite.xml' doesn't match anything: 'continuum' exists but not 'continuum//target/surefire-reports/*TestSuite.xml'
Can you please help me out in resolving the error....??
I assume you have an 'Execute JUnit tests' Build step. This will produce a JUnit XML file to a location you specify, say, TestOutput/junitresults.xml.
In the 'Publish JUnit test result report' Post Build step you just need to specify TestOutput/junitresults.xml.
As long as your tests executed and produced the output file the Post Build step won't fail to publish it, whether the tests failed or not.
You shouldn't be trying to publish files in the surefire-reports directory unless that it where you told JUnit to write its output file. Normally you wouldn't.
If you want to make it even simpler just tell JUnit to write its output file to the Jenkins WORKSPACE root by removing the TestOutput/ and just specify junitresults.xml.
I'm stuck configuring tests for Jenkins. In the Publish JUnit test result report I have test-reports/*.xml but I'm getting the error:
'test-reports/*.xml' doesn't match anything: 'test-reports' exists but not 'test-reports/*.xml'
When I try */.xml I get:
Did not manage to validate **/*.xml (may be too slow)
When I throw an .xml file into test-reports folder manually it is deleted after the build.
What do?
It might actually be an issue with XCode. We figured out that it was actually skipping the application tests and it just wasn't evident anywhere on Jenkins' console output.
I was trying to integrate my cucumber tests with maven-cucumber-reporting(from masterthought.net). I downloaded their Jenkins plugin and incorporated in my pom.xml file.
I have 2 tests in my eclipse project and when I ran these tests though maven it gets executed successfully. But, when I see my HTML report and also the cucumber.json , it displays details of only one test case.
I have been using Cucumber-JVM
Can you please let me know if I'm going wrong somewhere.
Thanks,
Prashanth
Do you have two RunCukesTest files for your two tests? If so make sure that you are creating unique json files for each tests (Note Test1.json below, make sure you have that distinction for each of your tests). Otherwise they will be overwritten by each other and you will only see the results from your last test.
#Cucumber.Options(format = { "pretty", "html:target/cucumber-html-report",
"json-pretty:target/Test1.json" },
features = { "src/test/resources/features" })