I'm using webdriver and JUnit to write tests for a website. The website has 4 levels of users. The tests I'm writing are mainly to confirm they can or cannot use a particular feature on the site, and that the feature works.
Currently I have my test project setup so that it runs in Chrome, FF and IE. First logging in as a user and then running a set of test cases. Some of these test cases are repeated for multiple types of users so my wish is to be able to user the same test case twice over multiple suites but when I try this running all the suites together JUnit never gives me a pass, fail or error the second time it's suppose to run the shared test cases.
Is what I'm trying to do possible with JUnit? How should I go about doing this?
I have my test suites layed out as follows:
AllTests
User1
Test something shared
Test something unique
User2
Test something shared
Test something unique
User3
Test something unique
User4
Test something unique
This is what my results looks like when the tests complete. As you can see it should have failed the suite on the first user on the Edit test(I coded it to fail purposely) but doesn't show an icon at all, and the suite comes back saying it worked fine.
One approach is to use a class hierarchy rather than a test suite. You could have a class per user type, each extending from a base class that contains the shared tests.
Related
I'm new to Junit. I've created a test suite with 50 test cases. If I run it passes only 30 test cases and 20 test cases are failing. How I can achieve running only those 20 failed test cases again with the help of Junit? Is it possible? Can someone guide on this?
Deciding which tests to run, and in which order, is the job of the test runner. So, to rerun only failed tests you need to use a test runner that records which tests passed or failed, and which can be instructed to run only tests it knows failed.
The tested runner in the Eclipse IDE can do this. Typically, and IDE is the only place you need this functionality, because an IDE is interactive and when using it you want fast feedback on whether a fix works. This functionality is not so useful elsewhere because in other contexts we typically want to check that the program passes all its tests.
I am running UI testing with Jest and I am using a custom reporter to generate a JUNIT.xml file at the end of the run https://github.com/jest-community/jest-junit , so that my azure pipeline can read it and generate nice analytics. My test framework is organize around Test suite that represent a big functionality, then each aspect of that functionality is check within a test contain in the suite ( That check might require multiple steps ) and I would like show each of those steps. This way i think it would be more readable for anyone looking at the report and it would be very easy to get context on why a test failed.
I try to put assertion at each steps. But JUNIT only record the assertion that failed.
I also try to change the way my test are organize and make a step a test itself. But, in Jest, and it seem in a lot of other runner as well ( at least in Node ) it seem that it's not possible to guarantee easily that test are run in a specific sequence. Also, it's really verbose to code suite like this.
Does anybody have an idea on how I could achieve this granularity ?
Thank you.
I come from Java world and I mostly used JUnit, and now I have some problems expressing some aspects of tests with NUnit 3. In JUnit, each test creates its own instance of a test class, so it's perfectly valid to create some instance variables in a test class, set up them in #Before method, test method and helpers can access these variables freely without worrying they would be overwritten by other tests run in parallel, and #After tears down the test data nicely. With NUnit it does not work and SetUp and TearDown methods seem to be useless in this case, because test fixture instance is reused between invocations of test method(s), so fields of test fixture class can (and are) overwritten by every invocation of a test method (my class has a few test methods, and each of them generates several test cases, so there are some tens of invocations in one test run).
I do not know how to work around this problem. In my scenario, set up would create a temporary folder, which would be used as a work folder for following test case. Tear down would delete the temporary folder afterwards, cleaning up all intermediate files created by tested method. But now, when SetUp creates and stores a temporary folder path in instance field (so it can be read by test logic and somewhat complicated asserts and verifiers), the value of such field is overwritten by test cases run in parallel. I considered several approaches:
implement an IDisposable which would represent a context of each test, and enclose it with using in each test method - I do not like this idea, because I do not like the idea of IDisposable being used as anything else than resource management tool and combinig IDisposable with using to simulate set up/tear down smells to me like an abuse of this particular language feature,
create a method which accepts a delegate for actual test logic, and which invokes custom SetUpTestCase/TearDownTestCase methods. The method would invoke set up, then test delegate, and tear down afterwards. What I do not like about this approach is that it does not play well with test methods which accept parameters - each set of test methods parametrized in particular way would need a corresponding delegate type. Also it somewhat seems to be against spirit of NUnit and the way of describing test methods with attributes - after all, why should the main logic of my test be delegated to anything? Shouldn't the [Test] or [TestCase] method be actual test?
maybe there's some way to use more advanced aspects of NUnit, like actions or some callbacks/triggers/whatever, I am just too unexperienced to see these. What I particularly miss is the way to transfer data from set up method (for example, a path to a temporary folder created by it) to the test method that follows. I cannot use instance fields for this, and I do not know whether there exists any "tag" structure which would pass test-specific data between methods invoked on different stages of a test lifecycle?
Generally, SetUp and TearDown attributes seem pretty useless to me, if they cannot set up the test case without their result being overwritten immediately by another test case run in parallel. What am I missing here?
How can I implemented such per-test case, scoped setup/tear down behavior with NUnit? What do I do wrong, or what do I miss?
As you have established, the TestFixture class is instantiated once before the OneTimeSetUp is called; then for each test it runs a set of SetUp, Test and TearDown; and finally, the OneTimeTearDown.
If you want the tests to be run in parallel (which is not the default) then you must specify The Parallelizable Attribute. Whether you do that or not, it is a good idea for your tests to be written independently, so they do not conflict with each other - they need to be structured.
The AAA (Arrange, Act, Assert) pattern is a common way of structuring unit tests for a method under test. If your tests are to be run in parallel, then TestFixture fields are not suitable for holding information which may conflict across parallel tests, in the same way that it wouldn't be suitable in a multithreaded class.
I'd suggest using a private method in the TestFixture to set up the temporary folder - it will need to have some way of providing a unique folder name, so that the parallel tests do not interact - perhaps use a Guid or CallerMemberName as part of the folder name, and return the folder name.
This method should be called from the Arrange part of the test. And you'll need a try...finally wrapping the rest of the Test to ensure the folder gets torn down. Or you could go with your IDisposable idea - I don't think there's anything wrong with that: the whole point of that is to guarantee tidying up resources (both managed and unmanaged) when something goes out of scope.
Your second suggestion of a delegate would also be fine if you used lambda expressions rather than strictly-defined delegates - the lambda expression can capture variables from the containing scope.
Imagine you have a large project, with several thousands of JUnit tests.
Let's says that running all thoses tests takes 7 minutes.
This looks short when you build your project from an ant/maven script.
But when you are using Eclipse, you cannot run all your test very often, because 7 minutes is too long time.
So here is the question:
When you modify some classes, is there a way to let JUnit runs only tests that may have been impacted by thoses class changes ?
I mean, this sounds feasible using classloader feature : after running each test, it's possible to know which classes have been loaded for this test, and to store somewhere (even in memory) a signature of each class used for this test.
When Junit is launched again, it could, for each test, check if classes used by this test have been modified since the very last run, and then NOT launch the test if it was ok and if no class impacting the test has been changed. (If the test were OK for the last run, it should be OK)
Does someone know if this has been done/implemented already ?
You could try using Infinitest from either Eclipse or IntelliJ. (Edited spelling)
I want to add some hints to my build, to run certain tests "first" without re-running them later.
Simply add Class names to a "priority" string in an input parameter to my test task, or
Have JUnit's testers smart enough to remember/persist failing test class names, so that the next time around the builder runs those first.
What is the most idiomatic way of doing this in Ant?
The following tools might help you to achieve the desired JUnit test execution order, but they depend on Eclipse usage:
Continuous Testing for Eclipse (CT-Eclipse)
JUnit Max
infinitest
I have not used any of those tools, and I have no Ant-only solution.
You might consider these related posts:
Run JUnit automatically when building Eclipse project
Starting unit tests automatically after saving a file