We are having couple of test cases marked as inconclusive for maintenance, issue is with our Hudson build which is considering Inconclusive test cases as Error.
We have enabled failonerror = "true" in build xml. Guess MsTest is making decision on error status and not Hudson.
is there any command line argument to ignore Inconclusive test as error.
Thanks.
MSTest reports Inconclusive as separate from failure, but returns a execution result of 1 if any tests are inconclusive (unlike NUnit, which does not). The build will interpret the 1 result code as a failure.
There is no command line option to turn this off (see http://msdn.microsoft.com/en-us/library/ms182489.aspx )
It may be possible to turn off the failonerror flag, and add a build step to parse for errors, but if you wish to turn off a test for maintenance, it would be better to use an [Ignore] attribute, like this:
[TestMethod, Ignore]
public void my_test () { ... }
Unlike NUnit, you can't add a reason for the ignore, so best leave a comment.
Related
Say I have CI tests running via GitHub actions. The program I test has a module that checks whether its input parameters are valid. Hence, I run a test where I intentionally provide improper input parameters, so the program catches this and exits with an error (exit 1).
Problem: I want GitHub Actions to mark this test as success. I am aware of continue-on-error: true for a run. Still, this will mark any failed run as success, no matter whether my program exits intentionally with an error due to improper input parameters (as described above), or because there is a bug in my code which then should actually return a failed CI test. So far I am manually inspecting the Actions logs but there must be an automated way to catch this.
Any ideas?
I am running Junit tests using Eclipse Luna. I have implemented #Test method. I loop within the #Test method for multiple records and use Assert.assertEquals for Non-XML messages and XMLAssert.assertXMLEqual for XML messages.
The problem is, when I run the Junit with single or multiple test cases, I do not get the proper result in the Junit View. It always shows "Runs: 1/1" and does not show the correct count of runs. Even the failures and success are not shown correctly. Am I missing something here?
If you only have a single #Test method, there is only one thing running, so the Runs: 1/1 is correct. If you want to have more show up, make each assertion in its own test.
I'm looking for the best practice for following (simplified) scenario:
#Test
public void someTest() {
for(String someText : someTexts) {
Assert.true(checkForValidity(someText));
}
}
This test iterates through x-thousands of texts and in this case I don't want it to be stopped for each failure. I want the errors to be buffered and in case of error(s) to fail at the end. Has JUnit got something on board for for my aim?
First of all, it's not really the correct way to implement this. JUnit allows parametrizing tests by defining a collection of inputs/outputs with the Parametrized test runner. Doing it this way ensures that each test case becomes a unique instance, making test report clearly state which samples passed and which ones failed.
If you still insist on doing it your way you should have a look at AssertJ's Soft Assertions which allow "swallowing" individual assertion failures, accumulating them and only reporting after the test is finished. The linked documentation section uses a nice example and is definitely worth reading.
I have ported my code to the RTM version of both WinRT and Rx. I use ReactiveUI in my ViewModels. Before porting the code my unit tests were running without problem but now I got a strange behavior.
Here the test:
var sut = new MyViewModel();
myViewModel.MyCommand.Execute(null) //ReactiveAsyncCommand
Assert.AreEqaul(0, sut.Collection.Count)
If I debug the test step by step, the assertion is not failing, but using the test runner it's failing...
The Collection asserted is modified by a method subscribing to the command:
MyCommand.RegisterAsyncTask(_ => DoWork())
.ObserveOn(SynchronizationContext.Current)
.Subscribe(MethodModifyingCollection);
The code was working before moving it to the RTM. I tried also to remove the ObserveOn and add an await Task.Delay() before the Assert without success.
Steven's got the rightish answer, but there are a few RxUI specific things missing. This is definitely related to scheduling in a test runner, but the reason is that the WinRT version of ReactiveUI can't detect properly whether it's in a test runner at the moment.
The dumb workaround for now is to set this at the top of all your tests:
RxApp.DeferredScheduler = Scheduler.CurrentThread;
Do not use the TestScheduler for every test, it's overkill and actually isn't compatible with certain kinds of testing. TestScheduler is good for tests where you're simulating time passing.
Your problem is that MSTest unit tests have a default SynchronizationContext. So ObserveOn and ReactiveAsyncCommand will marshal to the thread pool instead of to the WPF context. This causes a race condition.
Your first and best option is the Rx TestScheduler.
Another option is to await some completion signal (and ensure your test method is async Task, not async void).
Otherwise, if you just need a SynchronizationContext, you can use AsyncContext from my AsyncEx library to execute the tests within your own SynchronizationContext.
Finally, if you have any code that directly uses Dispatcher instead of SynchronizationContext, you can use WpfContext from the Async CTP download.
As per the documents, “assert” will fail the test and abort the current running test case, whereas a “verify” will fail the test and continue to run the test case.
But verifyTrue(false) is not failing the case(rather continue with the next step and mark the case as passed).
Assuming that's a Selenium call, then according to this, "[verify methods] don't stop the test when they fail. Instead, verification errors are all thrown at once during tearDown."