I have implemented a very high number of test cases and I would like to avoid running them all at once.
I created my own TestSuite in order to select only a bunch of them:
package com.mytests;
import junit.framework.Test;
import junit.framework.TestSuite;
import com.mytests.MyTestClass1;
import com.mytests.MyTestClass2;
import com.mytests.MyTestClass3;
public class CustomSuite extends TestSuite {
public static Test suite() {
TestSuite suite = new TestSuite();
suite.addTest(createTest(MyTestClass1.class, "test3"));
suite.addTest(createTest(MyTestClass2.class, "test2"));
suite.addTestSuite(MyTestClass3.class);
return suite;
I am able to launch it through Eclipse (Run As >> Android JUnit Test) without any problem. But my goal is to be able to run it through the startcts script.
cts_host > start --plan MyPlan -p com.mytests.CustomSuite
The specific test package does not exist: com.mytests.CustomSuite
cts_host > start --plan MyPlan -t com.mytests.CustomSuite#Test
The specific test does not exist: com.mytests.CustomSuite#Test
I am running out of ideas. Does anyone know how to do that?
Thanks,
Vincent
I'm not expert in this area, but I have one thing to advice.
In CTS, there is plans directory which contains xml files. They contains list of CTS test which is used startcts. Is your test in any of them?
Related
I have a problem using Junit tests with Java and Eclipse. All of my tests run just fine when I invoke them standalone. By this I mean that HDLmTreeTest, HDLmTreeTest1, and HDLmTreeTest2 all run fine when I run them by themselves. However, I have a Java source module with all of the test files in it. See below.
package com.headlamp;
import org.junit.platform.runner.JUnitPlatform;
import org.junit.platform.suite.api.SelectClasses;
import org.junit.runner.RunWith;
#RunWith(JUnitPlatform.class)
#SelectClasses({HDLmConfigTest.class, HDLmDefinesTest.class,
HDLmErrorTest.class, HDLmStringTest.class, HDLmTreeTest.class,
HDLmTreeTest1.class, HDLmTreeTest2.class,
HDLmModTest.class, HDLmBuildJSTest.class, HDLmFindTest.class,
HDLmAssertTest.class, HDLmBuildLinesTest.class, HDLmUtilityTest.class,
HDLmSavedChangeTest.class, HDLmCurlApacheTest.class, HDLmMainTest.class,
HDLmJettyTest.class, HDLmCurlJettyTest.class, HDLmEditorServletTest.class,
HDLmApacheTest.class, HDLmProxyTest.class, HDLmSessionTest.class,
HDLmLogMsgTest.class, HDLmMatchTest.class, HDLmImageInformationTest.class,
HDLmClusteringTest.class, HDLmJsonTest.class})
class HDLmAllTests { }
When I run this file, all of my tests get invoked except for HDLmTreeTest1.class and HDLmTreeTest2.class. I should say that HDLmTreeTest1.class and HDLmTreeTest2.class were just recently created and added to the SelectClasses list. For some reason, they are not invoked, but all of the other classes are invoked. What am I doing wrong?
I figured this out (with lots of online help). The problem was the name(s) of my test classes. Test classes (the names of test classes) must end with 'Test' or 'Tests'. They can not end with 'Test1' or 'Test2'. That was my error. I changed the class names and the problem went away. See https://howtodoinjava.com/junit5/junit5-test-suites-examples/ for some details.
I am fairly new to Cucumber. I was experimenting with it by just creating few test features when I noticed the difference when running a single feature vs running the whole suite (from the IntelliJ).
I noticed that when I run single feature it runs using the cucumber-jvm option and in this case, the CucumberConfig(the blank class to define the runner and cucumber options) and the Runner is not utilized. However, when I run the whole suite it runs as a JUnit test and obviously, in this case, the Config class and the runner comes into the picture.
I confirmed this with the following sample code:
#RunWith(CustomRunner.class)
#CucumberOptions()
public class CucumberConfig {
#BeforeClass
public static void beforeClass()
{
System.out.println("This is run before Once: ");
}
#AfterClass
public static void afterClass()
{
System.out.println("This is run after Once: ");
}
}
CustomRunner
public class CustomRunner extends Cucumber {
public CustomRunner(Class clazz) throws InitializationError, IOException {
super(clazz);
System.out.println("I am in the custom runner.");
}
}
Also, I understand that while running as cucumber-junit we can't pass specific feature to run as in cucumber-jvm. Correct me if I am wrong.
My doubt is, is this the default behavior or am I doing something wrong. And, if this is default how can I make cucumber to always use the Config file.
I'll appreciate if someone can provide some insight on this.
When you're using IntelliJ IDEA to run the tests, IDEA will use cucumber.api.Main to run the tests. As such it will ignore CucumberConfig neither will it run #BeforeClass nor #AfterClass, these are only used by the JUnit runner.
Can anyone explain the difference to me between Cucumber and Junit
From my understanding they are both used to test Java code although I am not sure of the difference?
Are they simply difference implementations of the same test suite or aimed at testing different things?
Cucumber and JUnit are different and solve different things.
Cucumber is a Behavior Driven Design (BDD) framework that takes "stories" or scenarios written in human readable languages such as English and turns those human readable text into a software test.
here's an Example cucumber story:
cucumber will then knows how to turn this text into a software test to make sure the software works as described. The output will tell you if the story is actually what the software does and if not, what was different:
Here's where the code is fixed to make the cucumber test pass:
This makes what is called an "Executable Specification" which is a nice way of documenting all of the features your software supports. This is different than normal documentation because without the corresponding test, someone reading the document doesn't know if the documentation is up to date.
Other Benefits of Executable Specifications:
Non-programmers can read and understand the tests
Non-programmers can write the tests since they are in plain English.
BDD results and Executable Specifications are very high level. They cover the overall features and perhaps a few edge cases as examples but don't test every possible condition or every code path. Also BDD tests are "integration tests" in that they test how all your code modules work together, but they don't test everything thoroughly.
This is where JUnit comes in.
JUnit is a lower level "Unit test" tool that allows developers to test every possible code path in their code. Each module of your code (or classes, or even methods) is tested in isolation. It is much more low level than a BDD framework. Using the same calculator story as the Cucumber example, JUnit tests would test lots of different calculation examples and invalid inputs to make sure the program responds correctly and computes the values correctly.
Hope that helps
I think Cucumber is more used for integration tests, while JUnit is more used in behaviour tests instead. Besides, Cucumber syntax is more accurate than JUnit, but much more complex. Here you can see a Cucumber test example:
package com.c0deattack.cucumberjvmtutorial;
import cucumber.annotation.en.Given;
import cucumber.annotation.en.Then;
import cucumber.annotation.en.When;
import cucumber.runtime.PendingException;
public class DepositStepDefinitions {
#Given("^a User has no money in their account$")
public void a_User_has_no_money_in_their_current_account() {
User user = new User();
Account account = new Account();
user.setAccount(account);
}
#When("^£(\\d+) is deposited in to the account$")
public void £_is_deposited_in_to_the_account(int arg1) {
// Express the Regexp above with the code you wish you had
throw new PendingException();
}
#Then("^the balance should be £(\\d+)$")
public void the_balance_should_be_£(int arg1) {
// Express the Regexp above with the code you wish you had
throw new PendingException();
}
private class User {
private Account account;
public void setAccount(Account account) {
this.account = account;
}
}
private class Account {
}
}
You can see that JUnit is more simple, but not necessarily less powerful:
import static org.junit.Assert.assertEquals;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
public class MyClassTest {
#Test(expected = IllegalArgumentException.class)
public void testExceptionIsThrown() {
MyClass tester = new MyClass();
tester.multiply(1000, 5);
}
#Test
public void testMultiply() {
MyClass tester = new MyClass();
assertEquals("10 x 5 must be 50", 50, tester.multiply(10, 5));
}
}
Hope it helps,
Clemencio Morales Lucas.
Cucumber is in which you can do BDD Behavior Driven Development. Something like you can convert your functional use case into Cucumber story. In that sense you can also take Cucumber to be DSL for Functional Use Case Document.
JUnit on other side is for unit testing which would be a method in Java. So a use can be unit test (rarely though) to inyegration test or a full system test, that's your first case Cucumber. Unit Test will be Unit Test only.
I have a database that stores my test results. I'm interested in writing a plugin for intellij13 that will let me rerun the test failures from the database using the JUnit run configuration. I can't find any documentation on this.
I'd like to see an example for some method like:
public void runTest(String testClass, String testName) {...}
I looked into IntelliJ 13.x and I was able to create JUnit runtime configuration. You need to do the following.
In your META-INF/plugin.xml add dependency on JUnit plugin, otherwise necessary JUnit plugin classes will not be available in your plugin class loader.
<depends optional="false">JUnit</depends>
Here's the sample code to create JUnit runtime configuration. Although it works, it is just a stub, you will have to populate all attributes.
import com.intellij.execution.RunManager;
import com.intellij.execution.impl.RunManagerImpl;
import com.intellij.execution.impl.RunnerAndConfigurationSettingsImpl;
import com.intellij.execution.junit.JUnitConfigurationType;
import com.intellij.openapi.project.Project;
...
RunManagerImpl runManager = (RunManagerImpl) RunManager.getInstance(project);
JUnitConfigurationType type = JUnitConfigurationType.getInstance();
RunnerAndConfigurationSettingsImpl runnerAndConfigurationSettings = (RunnerAndConfigurationSettingsImpl)runManager.createRunConfiguration("junit test run", type.getConfigurationFactories()[0]);
runManager.addConfiguration(runnerAndConfigurationSettings, false);
And here we go, JUnit run configuration.
Is there a way to (easily) generate a HTML report that contains the tests results ? I am currently using JUnit in addition to Selenium for testing web apps UI.
PS: Given the project structure I am not supposed to use Ant :(
I found the above answers quite useful but not really general purpose, they all need some other major build system like Ant or Maven.
I wanted to generate a report in a simple one-shot command that I could call from anything (from a build, test or just myself) so I have created junit2html which can be found here: https://github.com/inorton/junit2html
You can install it by doing:
pip install junit2html
Alternatively for those using Maven build tool, there is a plugin called Surefire Report.
The report looks like this : Sample
If you could use Ant then you would just use the JUnitReport task as detailed here: http://ant.apache.org/manual/Tasks/junitreport.html, but you mentioned in your question that you're not supposed to use Ant.
I believe that task merely transforms the XML report into HTML so it would be feasible to use any XSLT processor to generate a similar report.
Alternatively, you could switch to using TestNG ( http://testng.org/doc/index.html ) which is very similar to JUnit but has a default HTML report as well as several other cool features.
You can easily do this via ant. Here is a build.xml file for doing this
<project name="genTestReport" default="gen" basedir=".">
<description>
Generate the HTML report from JUnit XML files
</description>
<target name="gen">
<property name="genReportDir" location="${basedir}/unitTestReports"/>
<delete dir="${genReportDir}"/>
<mkdir dir="${genReportDir}"/>
<junitreport todir="${basedir}/unitTestReports">
<fileset dir="${basedir}">
<include name="**/TEST-*.xml"/>
</fileset>
<report format="frames" todir="${genReportDir}/html"/>
</junitreport>
</target>
</project>
This will find files with the format TEST-*.xml and generate reports into a folder named unitTestReports.
To run this (assuming the above file is called buildTestReports.xml) run the following command in the terminal:
ant -buildfile buildTestReports.xml
Junit xml format is used outside of Java/Maven/Ant word.
Jenkins with http://wiki.jenkins-ci.org/display/JENKINS/xUnit+Plugin is a solution.
For the one shot solution I have found this tool that does the job:
https://www.npmjs.com/package/junit-viewer
junit-viewer --results=surefire-reports --save=file_location.html
--results= is directory with xml files (test reports)
I found xunit-viewer, which has deprecated junit-viewer mentioned by #daniel-kristof-kiss.
It is very simple, automatically recursively collects all relevant files in ANT Junit XML format and creates a single html-file with filtering and other sweet features.
I use it to upload test results from Travis builds as Travis has no other support for collecting standard formatted test results output.
There are multiple options available for generating HTML reports for Selenium WebDriver scripts.
1. Use the JUNIT TestWatcher class for creating your own Selenium HTML reports
The TestWatcher JUNIT class allows overriding the failed() and succeeded() JUNIT methods that are called automatically when JUNIT tests fail or pass.
The TestWatcher JUNIT class allows overriding the following methods:
protected void failed(Throwable e, Description description)
failed() method is invoked when a test fails
protected void finished(Description description)
finished() method is invoked when a test method finishes (whether passing or failing)
protected void skipped(AssumptionViolatedException e, Description
description)
skipped() method is invoked when a test is skipped due to a failed assumption.
protected void starting(Description description)
starting() method is invoked when a test is about to start
protected void succeeded(Description description)
succeeded() method is invoked when a test succeeds
See below sample code for this case:
import static org.junit.Assert.assertTrue;
import org.junit.Test;
public class TestClass2 extends WatchManClassConsole {
#Test public void testScript1() {
assertTrue(1 < 2); >
}
#Test public void testScript2() {
assertTrue(1 > 2);
}
#Test public void testScript3() {
assertTrue(1 < 2);
}
#Test public void testScript4() {
assertTrue(1 > 2);
}
}
import org.junit.Rule;
import org.junit.rules.TestRule;
import org.junit.rules.TestWatcher;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;
public class WatchManClassConsole {
#Rule public TestRule watchman = new TestWatcher() {
#Override public Statement apply(Statement base, Description description) {
return super.apply(base, description);
}
#Override protected void succeeded(Description description) {
System.out.println(description.getDisplayName() + " " + "success!");
}
#Override protected void failed(Throwable e, Description description) {
System.out.println(description.getDisplayName() + " " + e.getClass().getSimpleName());
}
};
}
2. Use the Allure Reporting framework
Allure framework can help with generating HTML reports for your Selenium WebDriver projects.
The reporting framework is very flexible and it works with many programming languages and unit testing frameworks.
You can read everything about it at http://allure.qatools.ru/.
You will need the following dependencies and plugins to be added to your pom.xml file
maven surefire
aspectjweaver
allure adapter
See more details including code samples on this article:
http://test-able.blogspot.com/2015/10/create-selenium-html-reports-with-allure-framework.html
I have created a JUnit parser/viewer that runs directly in the browser. It supports conversion to JSON and the HTML report can be easily reused.
https://lotterfriends.github.io/online-junit-parser/
If you are still missing a feature feel free to create an issue on Github. :)