I have a test setup for a cloud system that uses a mixture of python for process level control and junit for internal state inspection. Essentially, I bring up several VMs to server as the cloud and then a junit VM which is a member of the cloud but drives tests and checks internal state. Our existing cloud management stuff is driven by python and I would like to maintain this.
I have a working setup that will run the JUnit command line via
java -ea -cp <classpath> org.junit.runner.JUnitCore <tests>
but this does not produce an report file. I know that ant is capable of producing an xml report, but I do not want to involve ant in this process (I have enough moving parts already).
Is there a way to launch junit from the command line such that it produces a report?
Ideally, I would have the junit tests produce xml reports, the python tests produce xml reports, and then merge them together for consumption by our CI system.
Update: The command line execution must support Windows, Linux, and Mac. We are not allowed to ship an external ant, although packaging an internal ant might be an option.
The JUnit library does not have any XML output options. To achieve such a thing, you'll need to write your own RunListener, which listens for the output and will in your case write the XML file.
However, to get the XML file in the correct format so that it can be read by CI system, I think it would be far easier to just use ant, either via the command line using a build.xml (JUnitReport), or using the java api: How can i use Apache ANT Programmatically.
EDIT: Initially, we had four options:
Use ant from the command line
Use ant programmatically (using the Java API)
Use the XMLJUnitResultFormatter directly with JUnitCore
Create a custom RunListener which produces the correct XML output.
Given the restrictions added by the OP, we can't use ant from the command line, which eliminates 1.
After looking more closely at the Ant JUnit task, it seems to be impossible to use this with JUnitCore (adding a TestListener), because ant uses the name of the test class directly, so you can't do a bridge class. From XMLJUnitResultFormatter.java
private void formatError(String type, Test test, Throwable t) {
...
nested.setAttribute(ATTR_TYPE, t.getClass().getName());
String strace = JUnitTestRunner.getFilteredTrace(t);
Text trace = doc.createTextNode(strace);
nested.appendChild(trace);
}
This eliminates 3.
Invoke Ant programmatically, via the Java API. I can't find any recent documentation on this. This seems to be hard.
So, finally, I would do 4, a custom RunListener, using the code from XMLJUnitResultFormatter as a base. And then, I'd publish it on github.com, so this question could be answered properly :-)
Related
I'm using Cypress to run a suite of automated tests.
The current version of cypress provides mocha-junit-reporter out of the box, and provides configuration options to pass to the reporter. One of the options is the 'mochaFile'.
I'm using the recommended [hash] tag to output reports across multiple spec files.
this results in a flat mess of files that look like 'results/test-output-abc12345.xml'.
What I want instead is for the test file's relative path and filename to be pass in as the reporter's output file path.
This would give me a structured, feature first view of the output, and in Azure Dev Ops, which aggregates the test output, it would give me correct filenames to detect intermittently failing tests.
Things I've tried that haven't worked:
I've tried to use hooks to modify Cypress's config or set environment variables to try to override the reporterOptions/mochaFile per test at (hopefully) the right time.
I've tried to grab the outputted defaultly-named xml file, and copy it to the correct path+filename given the Cypress.spec.name context, but I can't seem to find the right hook or time to do this.
after and afterEach don't work - I don't think the test report has saved the file yet.
Using a plugin, hooking to some event on test:before:run or test:after:run seem promising, but I'm flying blind since I can't debug into it, so I've been unsucessful in modifying the reporter's output path or copying the file.
I'd love it if someone could show a working example using mocha-junit-reporter, or even a different mocha compatible reporter, if the reporter would play well with Azure Dev Ops, and can help me discover intermittently failing tests.
I have a task to send reports of periodic execution of FitNesse tests to some specific endpoint in some specific JSON format.
I set periodic execution of tests in Jenkins properties and saving it in XML, but now I need to parse information about results of it.
It cannot be just step in "after build" property in Jenkins (or can, but I don't know a plugin for it), but what it would be and how I can do this?
Especially, I don't need information about the test, only general moments like date of the test, pass rate, status, name of the project, etc.
I think the best way to solve this is to make a script that parses the XML file, and creates the required JSON file. We normally use python scripts for this.
If you need certain generic information of the build in the script, like build number, you can pass this to your script using the Jenkins environments.
To call the script just add a batch or shell step, and place it below your fitnesse build step, to make sure the XML is generated before calling the script.
FitNesse comes with a jUnit runner which allows you to execute a test/suite. If you create a test class annotated with #RunWith(FitNesseRunner.class) and include its execution in a Jenkins Maven job (where the jUnit class is executed by either surefire or failsafe plugin), the outcome of the tests executed will be picked up automatically by Jenkins, just like it picks up other/regular jUnit tests (as surefire or failsafe will include them in their XML reports and Jenkins will pick these up).
You can find a sample Maven FitNesse project using (a slightly customised version of) this approach at https://github.com/fhoeben/sample-fitnesse-project. How to run the tests on Jenkins is described at https://github.com/fhoeben/hsac-fitnesse-fixtures#to-run-the-tests-on-a-build-server:
Have the build server checkout the project and execute mvn clean test-compile failsafe:integration-test. The result in JUnit XML results can be found in: target/failsafe-reports (Jenkins will pick these up automatically for a Maven job)
You indicate you don't need the HTML results, but they will be made available. They can be found in: target/fitnesse-results/index.html, and you could choose to use the 'HTML Publisher' Jenkins plugin to link to them from each build.
Need to run a job in Jenkins after successfully running the tests it needs to perform as post-build action for generating test reports
For this i have configured
Publish JUnit test result report
In the field
Test Report XMLs: continuum/*/target/surefire-reports/*TestSuite.xml
'continuum/*/target/surefire-reports/TestSuite.xml' doesn't match anything: 'continuum' exists but not 'continuum//target/surefire-reports/*TestSuite.xml'
Can you please help me out in resolving the error....??
I assume you have an 'Execute JUnit tests' Build step. This will produce a JUnit XML file to a location you specify, say, TestOutput/junitresults.xml.
In the 'Publish JUnit test result report' Post Build step you just need to specify TestOutput/junitresults.xml.
As long as your tests executed and produced the output file the Post Build step won't fail to publish it, whether the tests failed or not.
You shouldn't be trying to publish files in the surefire-reports directory unless that it where you told JUnit to write its output file. Normally you wouldn't.
If you want to make it even simpler just tell JUnit to write its output file to the Jenkins WORKSPACE root by removing the TestOutput/ and just specify junitresults.xml.
We have a requirement of creating several Windows 8 apps for tablets. There is a common solution and news apps are created by passing different resource to the same code.
For Mobile it was a cakewalk - Used the Microsoft.Build.Evaluation.Project class to get the xap file.
But for Surface, building from code does not give the direct appx output, while building using MSBuild gives appx as the output.
I tried several methods to avoid calling MSBuild from C#(by creating a Command Process) like creating a zip file - myapp.appx - and then signing it using this c++ code. It didn't workout because of an extern reference and I gave up.
Then I tried to use SignTool.exe by creating a Command Process in C#. That too failed.
So, I am wondering if there is any way to build an appx directly from C# without MSBuild.
The reason why I am trying to avoid MSBuild is to get a status from the build process, which Microsoft.Build.Evaluation.Project.Build() provides.
First try adding a pfx key to your project and then try the build via Microsoft.Build.Evaluation.Project.Build() again. The pfx is required from what I read:
See Candy's answer here:
MSBuild target to create the .appx package
So then the trick would be to use SignTool to get the pfx in the first place and update the project with that pfx, but first see if the above works.
I'm using Jasmine-reporters with Jasmine to output a bunch of JUnitXML format files that I need Hudson to parse for me and report on success/failure. Does anybody know how I would ask Hudson to go test a bunch of XML as part of the build process? Thanks!
In the job's "Post-build Actions", there should be a check box to "Publish JUnit test result report". You can put an ant-glob expression (as if you were writing an "includes" element) to match your xml files. I'm basing this on what I see in my Jenkins server; Hudson should be equivalent for your question.