What is the difference between test suite, test case and test category.
I found a partial answer here
But what about categories?
Test case is a set of test inputs, execution conditions, and expected results developed to test a particular execution path. Usually, the case is a single method.
Test suite is a list of related test cases. Suite may contain common initialization and cleanup routines specific to the cases included.
Test category/group is a way to tag individual test cases and assign them to categories. With categories, you don't need to maintain a list of test cases.
Testing framework usually provides a way to specify which categories to include or exclude from a given test run. This allows you to mark related test cases across different test suites. This is useful when you need to disable/enable cases that have common dependencies (API, library, system, etc.) or attributes (slow, fast cases).
As far as I can understand, test group and test category are different names for the same concept used in different frameworks:
#Category in JUnit
#group annotation in PHPUnit
TestCategory in MSTest
Test Categories is like a sub Test Suite. Take this documentation by example. One a class file, you will have multiple test cases. A Test Suite is a grouping of Test classes that you want to run. A Test Category is a sub grouping of Test Cases. You could put a annotation at some test cases in your class file, and create to test suite pointing to the same test class, but filtering one of the suites to test only some categories. Example from documentation:
public interface FastTests { /* category marker */ }
public interface SlowTests { /* category marker */ }
public class A {
#Test
public void a() {
fail();
}
#Category(SlowTests.class)
#Test
public void b() {
}
}
#Category({SlowTests.class, FastTests.class})
public class B {
#Test
public void c() {
}
}
#RunWith(Categories.class)
#IncludeCategory(SlowTests.class)
#SuiteClasses( { A.class, B.class }) // Note that Categories is a kind of Suite
public class SlowTestSuite {
// Will run A.b and B.c, but not A.a
}
#RunWith(Categories.class)
#IncludeCategory(SlowTests.class)
#ExcludeCategory(FastTests.class)
#SuiteClasses( { A.class, B.class }) // Note that Categories is a kind of Suite
public class SlowTestSuite {
// Will run A.b, but not A.a or B.c
}
Notice that both Test Suite points to the same test classes, but they will run different test cases.
Related
Student here. In JUnit 5, what is the best way to implement conditional test execution based on whether another test succeeded or failed? I presume it would involve ExecutionCondition, but I am unsure how to proceed. Is there a way to do this without having to add my own state to the test class?
To note, I am aware of dependent assertions, but I have multiple nested tests that represent distinct substates, and so I would like a way to do this at the test level itself.
Example:
#Test
void testFooBarPrecondition() { ... }
// only execute if testFooBarPrecondition succeeds
#Nested
class FooCase { ... }
// only execute if testFooBarPrecondition succeeds
#Nested
class BarCase { ... }
#Nested tests give the test writer more capabilities to express the
relationship among several groups of tests. Such nested tests make use
of Java’s nested classes and facilitate hierarchical thinking about
the test structure. Here’s an elaborate example, both as source code
and as a screenshot of the execution within an IDE.
As stated from JUnit 5 documentation, #Nested is use for convenient display in your IDE. I would rather use Assumptions for your preconditions inside a Depentent Assertions.
assertAll(
() -> assertAll(
() -> assumeTrue(Boolean.FALSE)
),
() -> assertAll(
() -> assertEquals(10, 4 + 6)
)
);
You can fix the problem by extracting common precondition logic in a #BeforeEach/#BeforeAll setup method, then use assumptions, which was developed exactly for the purpose of condition test execution. Some sample code:
class SomeTest {
#Nested
class NestedOne {
#BeforeEach
void setUp() {
boolean preconditionsMet = false;
//precondition code goes here
assumeTrue(preconditionsMet);
}
#Test // not executed when precodition is not met
void aTestMethod() {}
}
#Nested
class NestedTwo {
#Test // executed
void anotherTestMethod() { }
}
}
I need the following test
#runwith(cache, memory)
class CollectionA is -- this is a suite (aka folder)
class Cache { -- this is a sub-suite (aka folder)
#test testCache1() -- this is a method (aka file)
#test testCache2()
#test testCache3()
}
class RAM { -- this is a sub-suite (aka folder)
#test testRAM1()
#test testRAM2()
}
#test testIO()
#test testKeyboard()
#test testMouse()
#test testMonitor()
#test testPower()
#test testBoot()
Please note that only Cache and RAM need to be grouped. The hierarchy helps to fight the complexity and run related tests, e.g. Cache subsystem, alone, when necessary. The problem that is as soon I use #runwith to do that grouping, all the single test methods besides the RAM and Cache collections are ignored by JUnit. It seems that you cannot have sibling files and folders in JUnit design. The comments in the official example of grouping also hints that
#RunWith(Suite.class)
#Suite.SuiteClasses({
TestA.class,
TestA.class
})
public class FeatureTestSuite {
// the class remains empty,
// used only as a holder for the above annotations
// HEY!!! WHAT ABOUT MY #Tests HERE?
}
The answers say that whether I need to wrap every single test, e.g. testPower into their singletone suit or flatten the suite - get rid if the hierarchy completely.
So, is it right that JUnit is designed to disallow mixing single files (#test methods) with the folders (#runwith suites)? Why? How can this be worked around? Might be there is an alternative to #runwith.Suite?
What you like to create is a mixin type, which is not supported by the JUnit runner. So yes, you are right, it is not possible out of the box.
For this purpose I created an add-on that can be used to create hierarchical contexts for your test. In my point of view this is a missing feature in JUnit and I also stay in contact to get this included into the JUnit core.
The add-on provides an HierarchicalContextRunner which allows to use inner class to group your tests into contexts. Each context can contain tests or other contexts. It also allows to have #Before, #After, #Rule methods and fields, as well as other feature like #Ignore of the standard Runner. :-)
Example:
#RunWith(HierarchicalContextRunner.class)
public class CollectionA {
public class Cache {
#Test testCache1() {...}
#Test testCache2() {...}
#Test testCache3() {...}
}
public class RAM {
#Test testRAM1() {...}
#Test testRAM2() {...}
}
#Test testIO() {...}
#Test testKeyboard() {...}
#Test Mouse() {...}
#Test testMonitor() {...}
#Test testPower() {...}
#Test testBoot() {...}
}
Give it a try:
https://github.com/bechte/junit-hierarchicalcontextrunner/wiki
Votes and feedback are highly appreciated. :)
Your design should be something like this:
// folder com.myco.project
SuiteX.java
TestA.java
TestB.java
// contents of TestA.java
public class TestA{
#Test
public void someTestInA(){...}
}
// contents of TestB.java
public class TestB{
#Test
public void someTestInB(){...}
}
// contents of SuiteX.java
#RunWith(Suite.class)
#Suite.SuiteClasses({
TestA.class,
TestB.class
})
public class FeatureTestSuite {
// the class remains empty,
// used only as a holder for the above annotations
}
As I state in my comment, use separate java files for each test class. Do not use inner classes.
So I'm trying to run tests that will evaluate certain properties of different websites. The actual evaluation is being handled by a pay-per-call resource, so I want to minimize the number of times I generate the resource. Also, I need this to run in JUnit to fit into a larger automated test suite.
I've been doing this with parameterized tests so far, but I just learned that they instantiate a new instance for each test method.
Now I'm trying to figure out a way to have the resource created just once for each parameter that is being fed into the constructor of my testing class. #BeforeClass does it just once, and #Before does it once before each test.
All the help topics I've been able to find have dealt with creating expensive resources once for all tests, but in this case I need the resource to be recreated for each new set of parameters.
I've written some example code / output below to better show what I'm looking for:
#RunWith(Parameterized.class)
public class MyTestClass {
private static Resource expensiveToCreateResource;
public MyTestClass(String url) {
System.out.println("Constructing resource for " + url);
expensiveToCreateResource = new Resource(url); //This is getting created 4x, which is wrong
}
#Parameters
public static Collection<Object[]> data() {
return Arrays.asList(new Object[][] {{"url1"},{"url2"}});
}
#Test
public test1() {
expensiveToCreateResource.method1();
System.out.println("test1");
}
#Test
public test2() {
expensiveToCreateResource.method2();
System.out.println("test2");
}
}
would produce output:
Constructing resource for url1
test1
test2
Constructing resource for url2
test1
test2
Any ideas / solutions? Thanks.
If you want to have the class instantiated once per parameter, you'll have to write your own JUnit test runner. Instead I'd try to cache the information as needed, e.g. in a static map that maps URLs to resources.
I wanted to know if there's any way to add test suites dynamically in junit 4.
For example I have a TestClassA as mentioned below having test case "test1"
class TestClassA
{
#Test
public void test1()
{
createTestClassDynamically(); // this creates a test class having
// setUp(), tearDown() methods and one test case .
}
}
Test case test1 has a method createTestClassDynamically() that dynamically creates a new test class (lets say TestClassB) having setUp(), tearDown() methods and one test case (lets say test2()).
I want to run the test1 and then when TestClassB is dynamically generated I want test case "test2" also to be executed.
I know this is quite complicated and not the best thing to do but in my framework I need to do it to generate large number of test classes dynamically rather than having them physically in the package.
Can anyone please provide any help/suggestions?
I have solved this is my framework using the Parameterized feature of Junit 4 which helps to execute same test case with different parameters.
Below mentioned is the sample code on how I acheived it, thought to post it if it helps anyone.
Also, if someone has a better solution, feel free to post it.
class TestClassA
{
private TestClassB classBObj;
public TestClassA(TestClassB obj) {
classBObj= obj;
}
#Test
public void test1()
{
// createTestClassDynamically(); // remove this method as Parameterized
// feature will take care of dynamic test execution.
}
#Test
public void test2()
{
// Test case from Test class B using TestClassB object (classBObj)
}
public static Collection<Object[]> getParameters() {
Collection<Object[]> parameteres = new ArrayList<Object[]>();
Object[] obj1 = new Object[]{new TestClassB()};
Object[] obj2 = new Object[]{new TestClassB()};
parameteres.add(obj1);
parameteres.add(obj2);
// ....... add more test data this way or create a loop
return parameteres;
}
}
I have the classical structure for tests, I have a test suite of different suites like DatabaseTests, UnitTests etc. Sometimes those suites contains other suites like SlowDatabaseTests, FastDatabaseTests etc.
What I want is to randomize the running order of tests so I will make sure they are not dependent to each other. Randomization should be at every level, like suite should shuffle test class order, and test class should shuffle test method order.
If it is possible to do this in Eclipse that will be the best.
You do have a Sortable but I can't see how you would use it.
You could extend BlockJUnit4ClassRunner and have computeTestMethods() return a randomized copy of super.computeTestMethods(). Then use the #RunWith to set that as the runner to use.
e.g.
package com.stackoverflow.mlk;
import java.util.Collections;
import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.InitializationError;
public class RandomBlockJUnit4ClassRunner extends BlockJUnit4ClassRunner {
public RandomBlockJUnit4ClassRunner(Class<?> klass)
throws InitializationError {
super(klass);
}
protected java.util.List<org.junit.runners.model.FrameworkMethod> computeTestMethods() {
java.util.List<org.junit.runners.model.FrameworkMethod> methods = super.computeTestMethods();
Collections.shuffle(methods);
return methods;
}
}
Then
#RunWith(com.stackoverflow.mlk.RandomBlockJUnit4ClassRunner.class)
public class RandomOrder {
#Test
public void one() {
}
#Test
public void two() {
}
#Test
public void three() {
}
}
https://github.com/KentBeck/junit/pull/386 introduces some orders but not RANDOM. Probably you do not really want this; tests should run deterministically. If you need to verify that different permutations of tests still pass, either test all permutations; or, if this would be impractically slow, introduce a “random” seed for shuffling that is determined by an environment variable or the like, so that you can reproduce any failures. http://hg.netbeans.org/main/file/66d9fb12e98f/nbjunit/src/org/netbeans/junit/MethodOrder.java gives an example of doing this for JUnit 3.
In general what you need to do is to write your own test runner and in the test runner class aggregate the methods and randomly run each test (make sure you don't run a test twice).
Read more about the test framework and how to write your own test runner here:
http://www.ddj.com/architect/184415674
In JUnit 4.13, to run the tests within a test class in random order, write a small helper class:
import org.junit.runner.manipulation.Ordering;
import java.util.Random;
public class RandomOrder implements Ordering.Factory {
#Override
public Ordering create(Ordering.Context context) {
long seed = new Random().nextLong();
System.out.println("RandomOrder: seed = " + seed);
return Ordering.shuffledBy(new Random(seed));
}
}
Then, annotate your test class with:
#OrderWith(RandomOrder.class)
This way, the test methods of this one class are run in random order. Plus, if they unexpectedly fail, you know the random seed to repeat exactly this order.
I don't know though how to configure this for a whole project or a test suite.
I will make sure they are not dependent to
each other
You should make sure that this is the case without relying on random execution order. What makes you fear that dependencies may exist?
This issue is open on JUnit GitHub since 2 years, and point out 2 independent issues:
- Tests depending on the execution order;
- Non repeatable tests.
Consider adressing the issue at the root, rather than trying to use the framework to do the job afterwards. Use setUp and tearDown method to guarantee isolation, and test at the smallest level.
Here is a solution with Gradle and JUnit 5.8.0
Step 1 : Ensure that you have latest JUnit version dependency.
Step 2 : Define the required properties under build.gradle test section
test {
useJUnitPlatform()
systemProperties([
//Random in method level
'junit.jupiter.testmethod.order.default': 'org.junit.jupiter.api.MethodOrderer$Random',
// Random in class level
'junit.jupiter.testclass.order.default' : 'org.junit.jupiter.api.ClassOrderer$Random',
// Log configuration to see the seed
'java.util.logging.config.file' : file('src/test/resources/logging.properties')
])
//To print the JUnit logs in console
testLogging {
events "passed", "skipped", "failed", "standardOut", "standardError"
}
}
Step 3: Define logging.properties under src/test/resources
.level=CONFIG
java.util.logging.ConsoleHandler.level=CONFIG
org.junit.jupiter.api.ClassOrderer$Random.handlers=java.util.logging.ConsoleHandler
org.junit.jupiter.api.MethodOrderer$Random.handlers=java.util.logging.ConsoleHandler
Step 4 : Run test. gradlew clean test
You can see the seed used for the random test in the console
CONFIG: ClassOrderer.Random default seed: 65423695211256721
CONFIG: MethodOrderer.Random default seed: 6542369521653287
In case of flaky test, you can reproduce it by configuring the same seed where the JUnit tests were failing
systemProperties([
'junit.jupiter.execution.class.order.random.seed' : '65423695211256721'
'junit.jupiter.execution.order.random.seed' : '6542369521653287'
])
References : how-to-randomize-tests-in-junit , Random