I'm basically attempting to run my scenarios in sequence and trying to find if there is a way to get the handle to Results in #AfterClass method.
I know that it is possible when running in parallel, but our necessity is to get a break down of all the test executed in test report which is not possible when implemented like below.
public class AllTestRunner {
#Test
public void testParallel() {
Results results = Runner.parallel(1, "classpath:karate/feature");
}
}
The issue with this approach is that test reports appear as only 1 test executed which is not expected.
Is there a way to get Results when running like below ?
#RunWith(Karate.class)
#KarateOptions(features = "classpath:karate/feature")
public class AllTestRunner {
#AfterClass
public void testParallel() {
Results results = ...;
}
}
Basically the purpose of trying to get the Results is to perform an API call if a scenario has failed, by checking result.isFailed().
Again, running tests in parallel is generating in test report like below.
1 tests 0 failures 0 ignored 7.373s duration 100% successful
Tests Standard output
Test Duration Result
testParallel 7.373s passed
No there isn't. Maybe you can explore the hooks, but be warned that the API will change slightly in 1.0 - https://stackoverflow.com/a/59080128/143475
If you have unusual needs like this, you should consider contributing code to Karate. The so called "sequential" mode is a convenience to connect with JUnit. There should not be a problem with the parallel runner, I don't understand what you mean by "only 1 test executed".
We run a set of tests in a CI pipeline and call our test classes like this:
java -classpath junit-jupiter-api-5.0.1.jar:junit-platform-console-standalone-1.0.1.jar org.junit.platform.console.ConsoleLauncher --select-class xy.Test
If class xy.Test cannot be found on the classpath an error message appears but ConsoleLauncher's return value is 0! Since our CI system runs unattended the return value is the only important return value!
As I have seen this behaviour got updated in JUnit 5.0.0 M2 but I regard this as I mistake: If I define a class by --select-class and the class cannot be found then something has gone wrong!
As I countermeasure I hacked (by means of introspection) org.junit.platform.commons.util.BlacklistedExceptions by overwriting blacklist's field with OutOfMemoryError (=default) and PreconditionViolationException (=case where class could not be found).
(If the standard behaviour shall not be changed...) I think there should be a better way to get this behaviour!
Student here. In JUnit 5, what is the best way to invoke a #Nested test class multiple times, but with slightly different state each time?
I see that JUnit 5 has an (experimental) #ParameterizedTest feature that is based on the (non-experimental) #TestTemplate feature, but both of those apply to test methods only, rather than nested test classes.
I have a heavy #Nested test class that needs to be invoked once for each value of an enum (preferably with a distinct #Tag value for each invocation), and I would prefer to avoid the "copy-and-paste" method of parameterization.
It is not currently possible to execute a test class multiple times in JUnit Jupiter.
To participate in the discussion, see the following issue: https://github.com/junit-team/junit5/issues/878
Here is my specific scenario.
I have a class QueryQueue that wraps the QueryTask class within the ArcGIS API for Flex. This enables me to easily queue up multiple query tasks for execution. Calling QueryQueue.execute() iterate through all the tasks in my queue and call their execute method.
When all the results have been received and processed QueryQueue will dispatch the completed event. The interface to my class is very simple.
public interface IQueryQueue
{
function get inProgress():Boolean;
function get count():int;
function get completed():ISignal;
function get canceled():ISignal;
function add(query:Query, url:String, token:Object = null):void;
function cancel():void;
function execute():void;
}
For the QueryQueue.execute method to be considered successful several things must occur.
task.execute must be called on each query task once and only once
inProgress = true while the results are pending
inProgress = false when the results have been processed
completed is dispatched when the results have been processed
canceled is never called
The processing done within the queue correctly processes and packages the query results
What I am struggling with is breaking these tests into readable, logical, and maintainable tests.
Logically I am testing one state, that is the successful execution state. This would suggest one unit test that would assert #1 through #6 above are true.
[Test] public mustReturnQueryQueueEventArgsWithResultsAndNoErrorsWhenAllQueriesAreSuccessful:void
However, the name of the test is not informative as it does not describe all the things that must be true in order to be considered a passing test.
Reading up online (including here and at programmers.stackexchange.com) there is a sizable camp that asserts that unit tests should only have one assertion (as a guideline). As a result when a test fails you know exactly what failed (i.e. inProgress not set to true, completed displayed multiple times, etc.) You wind up with potentially a lot more (but in theory simpler and clearer) tests like so:
[Test] public mustInvokeExecuteForEachQueryTaskWhenQueueIsNotEmpty():void
[Test] public mustBeInProgressWhenResultsArePending():void
[Test] public mustNotInProgressWhenResultsAreProcessedAndSent:void
[Test] public mustDispatchTheCompletedEventWhenAllResultsProcessed():void
[Test] public mustNeverDispatchTheCanceledEventWhenNotCanceled():void
[Test] public mustReturnQueryQueueEventArgsWithResultsAndNoErrorsWhenAllQueriesAreSuccessful:void
// ... and so on
This could wind up with a lot of repeated code in the tests, but that could be minimized with appropriate setup and teardown methods.
While this question is similar to other questions I am looking for an answer for this specific scenario as I think it is a good representation of a complex unit testing scenario exhibiting multiple states and behaviors that need to be verified. Many of the other questions have, unfortunately, no examples or the examples do not demonstrate complex state and behavior.
In my opinion, and there will probably be many, there are a couple of things here:
If you must test so many things for one method, then it could mean your code might be doing too much in one single method (Single Responsibility Principle)
If you disagree with the above, then the next thing I would say is that what you are describing is more of an integration/acceptance test. Which allows for multiple asserts, and you have no problems there. But, keep in mind that this might need to be relegated to a separate section of tests if you are doing automated tests (safe versus unsafe tests)
And/Or, yes, the preferred method is to test each piece separately as that is what a unit test is. The closest thing I can suggest, and this is about your tolerance for writing code just to have perfect tests...Is to check an object against an object (so you would do one assert that essentially tests this all in one). However, the argument against this is that, yes it passes the one assert per test test, but you still lose expressiveness.
Ultimately, your goal should be to strive towards the ideal (one assert per unit test) by focusing on the SOLID principles, but ultimately you do need to get things done or else there is no real point in writing software (my opinion at least :)).
Let's focus on the tests you have identified first. All except the last one (mustReturnQueryQueueEventArgs...) are good ones and I could immediatelly tell what's being tested there (and that's very good sign, indicating they're descriptive and most likely simple).
The only problem is your last test. Note that extensive use of words "and", "with", "or" in test name usually rings problems bell. It's not very clear what it's supposed to do. Return correct results comes to mind first, but one might argue it's vague term? This holds true, it is vague. However you'll often find out that this is indeed pretty common requirement, described in details by method/operation contract.
In your particular case, I'd simplify last test to verify whether correct results are returned and that would be all. You tested states, events and stuff that lead to results building already, so there is no need to that again.
Now, advices in links you provided are quite good ones actually, and generally, I suggest sticking to them (single assertion for one test). The question is, what single assertion really stands for? 1 line of code at the end of test? Let's consider this simple example then:
// a method which updates two fields of our custom entity, MyEntity
public void Update(MyEntity entity)
{
entity.Name = "some name";
entity.Value = "some value";
}
This method contract is to perform those 2 operations. By success, we understand entity to be correctly updated. If one of them for some reasons fails, method as a unit is considered to fail. You can see where this is going; you'll either have two assertions or write custom comparer purely for testing purposes.
Don't be tricked by single assertion; it's not about lines of code or number of asserts (however, in majority of tests you'll write this will indeed map 1:1), but about asserting single unit (in the example above, update is considered to be an unit). And unit might be in reality multiple things that don't make any sense at all without eachother.
And this is exactly what one of questions you linked quotes (by Roy Osherove):
My guideline is usually that you test one logical CONCEPT per test. you can have multiple asserts on the same object. they will usually be the same concept being tested.
It's all about concept/responsibility; not the number of asserts.
I am not familiar with flex, but I think I have good experience in unit testing, so you have to know that unit test is a philosophy, so for the first answer, yes you can make a multiple assert but if you test the same behavior, the main point always in unit testing is to be very maintainable and simple code, otherwise the unit test will need unit test to test it! So my advice to you is, if you are new in unit testing, don't use multiple assert, but if you have good experience with unit testing, you will know when you will need to use them
Time elapsed (time taken?) to run(and pass) my Junit test case is shown as 0.00. I have tried failing the test case (by doing assertEquals for a wrong value) and its still showing time elapsed as 0.00.
The rest of test cases have non-zero time elapsed values.
EDIT: Assumption - I assumed that it would surely take non-zero time to execute and that if it didn't happen, nothing was happening within the test case - example, I have a if condition in there and no corresponding else. So, if the condition failed, it just came out of the method without doing anything and since an error didn't occur, it didn't fail the test either.
Is there any reason this is happening? Am I missing something?
Thanks,
Pratyusha.
It's not clear why you think it should be non-zero. Is it possible that your test case is just passing (or failing) really quickly?
Just for the sake of experimentation, put a Thread.sleep(1000) call in there... if that changes things, it suggests that everything's fine and your test is just fast.
Are all your test methods shown to be executing? It's not something simple like failing to annotate the methods or failing to keep to the naming convention of testXXX (depending on which version of JUnit you're using)?