Is there a way to (easily) generate a HTML report that contains the tests results ? I am currently using JUnit in addition to Selenium for testing web apps UI.
PS: Given the project structure I am not supposed to use Ant :(
I found the above answers quite useful but not really general purpose, they all need some other major build system like Ant or Maven.
I wanted to generate a report in a simple one-shot command that I could call from anything (from a build, test or just myself) so I have created junit2html which can be found here: https://github.com/inorton/junit2html
You can install it by doing:
pip install junit2html
Alternatively for those using Maven build tool, there is a plugin called Surefire Report.
The report looks like this : Sample
If you could use Ant then you would just use the JUnitReport task as detailed here: http://ant.apache.org/manual/Tasks/junitreport.html, but you mentioned in your question that you're not supposed to use Ant.
I believe that task merely transforms the XML report into HTML so it would be feasible to use any XSLT processor to generate a similar report.
Alternatively, you could switch to using TestNG ( http://testng.org/doc/index.html ) which is very similar to JUnit but has a default HTML report as well as several other cool features.
You can easily do this via ant. Here is a build.xml file for doing this
<project name="genTestReport" default="gen" basedir=".">
<description>
Generate the HTML report from JUnit XML files
</description>
<target name="gen">
<property name="genReportDir" location="${basedir}/unitTestReports"/>
<delete dir="${genReportDir}"/>
<mkdir dir="${genReportDir}"/>
<junitreport todir="${basedir}/unitTestReports">
<fileset dir="${basedir}">
<include name="**/TEST-*.xml"/>
</fileset>
<report format="frames" todir="${genReportDir}/html"/>
</junitreport>
</target>
</project>
This will find files with the format TEST-*.xml and generate reports into a folder named unitTestReports.
To run this (assuming the above file is called buildTestReports.xml) run the following command in the terminal:
ant -buildfile buildTestReports.xml
Junit xml format is used outside of Java/Maven/Ant word.
Jenkins with http://wiki.jenkins-ci.org/display/JENKINS/xUnit+Plugin is a solution.
For the one shot solution I have found this tool that does the job:
https://www.npmjs.com/package/junit-viewer
junit-viewer --results=surefire-reports --save=file_location.html
--results= is directory with xml files (test reports)
I found xunit-viewer, which has deprecated junit-viewer mentioned by #daniel-kristof-kiss.
It is very simple, automatically recursively collects all relevant files in ANT Junit XML format and creates a single html-file with filtering and other sweet features.
I use it to upload test results from Travis builds as Travis has no other support for collecting standard formatted test results output.
There are multiple options available for generating HTML reports for Selenium WebDriver scripts.
1. Use the JUNIT TestWatcher class for creating your own Selenium HTML reports
The TestWatcher JUNIT class allows overriding the failed() and succeeded() JUNIT methods that are called automatically when JUNIT tests fail or pass.
The TestWatcher JUNIT class allows overriding the following methods:
protected void failed(Throwable e, Description description)
failed() method is invoked when a test fails
protected void finished(Description description)
finished() method is invoked when a test method finishes (whether passing or failing)
protected void skipped(AssumptionViolatedException e, Description
description)
skipped() method is invoked when a test is skipped due to a failed assumption.
protected void starting(Description description)
starting() method is invoked when a test is about to start
protected void succeeded(Description description)
succeeded() method is invoked when a test succeeds
See below sample code for this case:
import static org.junit.Assert.assertTrue;
import org.junit.Test;
public class TestClass2 extends WatchManClassConsole {
#Test public void testScript1() {
assertTrue(1 < 2); >
}
#Test public void testScript2() {
assertTrue(1 > 2);
}
#Test public void testScript3() {
assertTrue(1 < 2);
}
#Test public void testScript4() {
assertTrue(1 > 2);
}
}
import org.junit.Rule;
import org.junit.rules.TestRule;
import org.junit.rules.TestWatcher;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;
public class WatchManClassConsole {
#Rule public TestRule watchman = new TestWatcher() {
#Override public Statement apply(Statement base, Description description) {
return super.apply(base, description);
}
#Override protected void succeeded(Description description) {
System.out.println(description.getDisplayName() + " " + "success!");
}
#Override protected void failed(Throwable e, Description description) {
System.out.println(description.getDisplayName() + " " + e.getClass().getSimpleName());
}
};
}
2. Use the Allure Reporting framework
Allure framework can help with generating HTML reports for your Selenium WebDriver projects.
The reporting framework is very flexible and it works with many programming languages and unit testing frameworks.
You can read everything about it at http://allure.qatools.ru/.
You will need the following dependencies and plugins to be added to your pom.xml file
maven surefire
aspectjweaver
allure adapter
See more details including code samples on this article:
http://test-able.blogspot.com/2015/10/create-selenium-html-reports-with-allure-framework.html
I have created a JUnit parser/viewer that runs directly in the browser. It supports conversion to JSON and the HTML report can be easily reused.
https://lotterfriends.github.io/online-junit-parser/
If you are still missing a feature feel free to create an issue on Github. :)
Related
I am fairly new to Cucumber. I was experimenting with it by just creating few test features when I noticed the difference when running a single feature vs running the whole suite (from the IntelliJ).
I noticed that when I run single feature it runs using the cucumber-jvm option and in this case, the CucumberConfig(the blank class to define the runner and cucumber options) and the Runner is not utilized. However, when I run the whole suite it runs as a JUnit test and obviously, in this case, the Config class and the runner comes into the picture.
I confirmed this with the following sample code:
#RunWith(CustomRunner.class)
#CucumberOptions()
public class CucumberConfig {
#BeforeClass
public static void beforeClass()
{
System.out.println("This is run before Once: ");
}
#AfterClass
public static void afterClass()
{
System.out.println("This is run after Once: ");
}
}
CustomRunner
public class CustomRunner extends Cucumber {
public CustomRunner(Class clazz) throws InitializationError, IOException {
super(clazz);
System.out.println("I am in the custom runner.");
}
}
Also, I understand that while running as cucumber-junit we can't pass specific feature to run as in cucumber-jvm. Correct me if I am wrong.
My doubt is, is this the default behavior or am I doing something wrong. And, if this is default how can I make cucumber to always use the Config file.
I'll appreciate if someone can provide some insight on this.
When you're using IntelliJ IDEA to run the tests, IDEA will use cucumber.api.Main to run the tests. As such it will ignore CucumberConfig neither will it run #BeforeClass nor #AfterClass, these are only used by the JUnit runner.
I have a Java class with a main method that I invoke to occasionally run some tests. Specifically, I'm trying to come up with a solution for quickly testing various code snippets that use the AWS SDK to create/read some S3 objects. I'm not really trying to build regular unit/integration tests, and I'm not interested in mocking the S3 code. I'm trying to quickly develop/debug some code using a test framework. I found the following SO question, and the answer about using JUnit5 Jupiter's Launcher and it intrigued me:
How do I run JUnit tests from inside my java application?
So I read the Junit5 chapter on the Launcher API and followed the example code. I came up with something like this:
class S3ManualTest {
public static void main(String[] args) {
LauncherDiscoveryRequest request =
LauncherDiscoveryRequestBuilder
.request()
.selectors(selectPackage("com.xyz.s3util"),
selectClass(S3ManualTest.class),
selectMethod(S3ManualTest.class, "happyPath")
)
.build();
Launcher launcher = LauncherFactory.create();
SummaryGeneratingListener listener = new SummaryGeneratingListener();
launcher.execute(request, listener);
TestExecutionSummary summary = listener.getSummary();
System.out.println("# of containers found: " + summary.getContainersFoundCount());
System.out.println("# of containers skipped: " + summary.getContainersSkippedCount());
System.out.println("# of tests found: " + summary.getTestsFoundCount());
System.out.println("# of tests skipped: " + summary.getTestsSkippedCount());
}
void happyPath() {
assertTrue(true); // Do useful stuff here
}
}
The launcher doesn't find any tests to run, even though I specifically selected the "happyPath" method. I have tried annotating the happyPath() method with #Test, and that seems to work, but it also has the undesired side effect that the method gets executed if I run all tests in that package, either from gradle, or from inside the IDE. Essentially, I want my test methods to be invoked with the JUnit5 framework, but only when I manually run the main method in the class. I was thinking about some custom annotations, or implementing some interface that would get picked up by the test engine, but haven't gone down that route yet. I'm guessing there's some easy way of accomplishing what I'm trying to do. Thanks.
I could only find a work around: disabling the happyPath() test method by default and override it in your program like explained here: https://junit.org/junit5/docs/current/user-guide/#extensions-conditions-deactivation
#Test
#Disabled
void happyPath() {
assertTrue(true); // Do useful stuff here
}
And in your launcher setup, deactivate the DisabledCondition:
LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder
.request()
.selectors(selectMethod(S3ManualTest.class, "happyPath"))
.configurationParameter(
"junit.jupiter.conditions.deactivate",
"org.junit.*DisabledCondition")
.build();
You may also specify a dedicated switch, if you don't want deactivate DisabledCondition for the entire run:
#Test
#EnabledIf("'true'.equals(junitConfigurationParameter.get('manual'))")
void happyPath() {
assertTrue(true); // Do useful stuff here
}
with
LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder
...
.configurationParameter("manual", "true")
.build();
The second work-around, if applied to more then a few methods, screams for a dedicated ExecutionCondition extension. See details at https://junit.org/junit5/docs/current/user-guide/#writing-tests-conditional-execution-scripts
I need to execute some code before the #Before method of each unit test is executed. The problem is that I also need to know which test (the name is sufficient) will be executed afterwards.
I can either use AspectJ or Java Agents with bytecode manipulation to achieve this. Also the solution should work for tests where there is no #Before annotation present.
Any ideas?
EDIT: I can't modify the unit tests themselves, as I'm developing a framework for executing tests of other projects
You might want to look into the TestName rule in JUnit:
http://junit.org/junit4/javadoc/4.12/org/junit/rules/TestName.html
About the ordering, a solution could be to define a super class for your tests and put a #Before in there, as #Before methods in super classes are run before those in sub classes.
If you want to write a Java agent and you are not bound to Javassist or AspectJ, have a look at Byte Buddy for doing so. You can add the code in the MyAdvice class to any method annotated with #Test given that the type name ends with Test (as an example) by:
public class MyAgent {
public static void premain(String arg, Instrumentation inst) {
new AgentBuilder.Default()
.type(nameEndsWith("Test"))
.transform((type, cl, builder) -> builder.visit(Advice
.to(MyAdvice.class)
.on(isAnnotatedWith(Test.class)))
.installOn(instrumentation);
}
}
class MyAdvice {
#Advice.OnMethodEnter
static void enter() {
System.out.println("foo");
}
}
Just bundle the above code to a Javaagent with the proper manifest code and register it before running. If you are running on a JDK, you can also attach the agent programmatically using the byte-buddy-agent project.
I have a database that stores my test results. I'm interested in writing a plugin for intellij13 that will let me rerun the test failures from the database using the JUnit run configuration. I can't find any documentation on this.
I'd like to see an example for some method like:
public void runTest(String testClass, String testName) {...}
I looked into IntelliJ 13.x and I was able to create JUnit runtime configuration. You need to do the following.
In your META-INF/plugin.xml add dependency on JUnit plugin, otherwise necessary JUnit plugin classes will not be available in your plugin class loader.
<depends optional="false">JUnit</depends>
Here's the sample code to create JUnit runtime configuration. Although it works, it is just a stub, you will have to populate all attributes.
import com.intellij.execution.RunManager;
import com.intellij.execution.impl.RunManagerImpl;
import com.intellij.execution.impl.RunnerAndConfigurationSettingsImpl;
import com.intellij.execution.junit.JUnitConfigurationType;
import com.intellij.openapi.project.Project;
...
RunManagerImpl runManager = (RunManagerImpl) RunManager.getInstance(project);
JUnitConfigurationType type = JUnitConfigurationType.getInstance();
RunnerAndConfigurationSettingsImpl runnerAndConfigurationSettings = (RunnerAndConfigurationSettingsImpl)runManager.createRunConfiguration("junit test run", type.getConfigurationFactories()[0]);
runManager.addConfiguration(runnerAndConfigurationSettings, false);
And here we go, JUnit run configuration.
I do the following:
From the Package Explorer I select "New, Other, JUnit Test Case"
I write this code:
package dk.sample;
import org.junit.*;
import static org.junit.Assert.*;
public class TestCase {
#Test
public void alwaysTrue(){
assertTrue( true );
}
}
I then select "Run As, JUnit test"
Get this error: "Class not found dk.sample.TestCase
java.lang.ClassNotFoundException: ...."
What do I miss? Have tried with different Run Configurations - but it seems like I miss a classpath somewhere? But to what and where?
To make JUnit work within Domino Designer you need to perform few additional steps:
set up source control for your application
adjust the on-disk project to be recognized as Java application
run JUnit tests within your on-disk project
Please note that java agents have to be tested in a different way..
You can find more detailed explanation about enabling JUnit for both XPages and Agents in the following blog post: Unit Tests for Lotus Domino Applications
Here's also a great how-to on this topic.
Coundn't get JUnit to work inside the Domino Designer. Instead of running the tests from DDE, I now run the tests from a XPages. This works like a dream. Made my own 'JUnit runner' class - that is, I just call the JUnit runners but handles the result my self in order to display it as html on the XPage.
Code can be found here: http://xpages.dk/wp-content/uploads/2013/10/junitrunner.txt
Danish blog post here: http://xpages.dk/?p=1162