We run a set of tests in a CI pipeline and call our test classes like this:
java -classpath junit-jupiter-api-5.0.1.jar:junit-platform-console-standalone-1.0.1.jar org.junit.platform.console.ConsoleLauncher --select-class xy.Test
If class xy.Test cannot be found on the classpath an error message appears but ConsoleLauncher's return value is 0! Since our CI system runs unattended the return value is the only important return value!
As I have seen this behaviour got updated in JUnit 5.0.0 M2 but I regard this as I mistake: If I define a class by --select-class and the class cannot be found then something has gone wrong!
As I countermeasure I hacked (by means of introspection) org.junit.platform.commons.util.BlacklistedExceptions by overwriting blacklist's field with OutOfMemoryError (=default) and PreconditionViolationException (=case where class could not be found).
(If the standard behaviour shall not be changed...) I think there should be a better way to get this behaviour!
Related
I'm using Chisel and blackbox to run my chisel logic against a verilog register file.
The registerfile does not have reset signal so I expect the register to be randomly initialized.
I passed the --x-initial unique to verilator,
Basically this is how I launch the test:
private val backendName = "verilator"
"NOCDMA" should s" do blkwrite and blkread correctly (with $backendName)" in {
Driver.execute(Array("--fint-write-vcd","--backend-name",s"$backendName",
"--more-vcs-flags","--trace-depth 1 --x-initial unique"),
()=>new DMANetworkWithMem(memAddrWidth,memDataWidth)(nocDataWidth)(nNodesX,nNodesY)){
c => new DMANetworkRWTest(c)
}
}
But The data I read from the register file is all zero before I wrote anything to it.
The read data is correct after I wrote to it.
So, is there anything inside chisel that I need to tune or I did not do everything properly ?
Any suggestions?
I'm not certain, but I found the following issue on Verilator with a similar issue: https://github.com/verilator/verilator/issues/1399.
From skimming the above issue, I think you also need to pass +verilator+seed+<value> and +verilator+rand+reset+<value> at runtime. I am not an expert in the iotesters, but I believe you can add these runtime values through the iotesters argument: --more-vcs-c-flags.
Side note, I would also set --x-assign unique in Verilator if there are cases in the Verilog where runtime would otherwise inject an X (eg. out-of-bounds index).
I hope this helps!
I have test cases defined in an Excel sheet. I am reading a string from this sheet (my expected result) and comparing it to a result I read from a database (my actual result). I then use AssertEquals(expectedResult, actualResult) which prints any errors to a log file (i'm using log4j), e.g. I get java.lang.AssertionError: Different output expected:<10> but was:<7> as a result.
I now need to write that result into the Excel sheet (the one that defines the test cases). If only AssertEquals returned String, with the AssertionError text that would be great, as I could just write that immediately to my Excel sheet. Since it returns void though I got stuck.
Is there a way I could read the AssertionError without parsing the log file?
Thanks.
I think you're using junit incorrectly here. THis is why
assertEquals not AssertEquals ( ;) )
you shouldnt need to log. You should just let the assertions do their job. If it's all green then you're good and you dont need to check a log. If you get blue or red (eclipse colours :)) then you have problems to look at. Blue is failure which means that your assertions are wrong. For example you get 7 but expect 10. Red means error. You have a null pointer or some other exception that is throwing while you are running
You should need to read from an excel file or databse for the unit tests. If you really need to coordinate with other systems then you should try and stub or mock them. With the unit test you should work on trying to testing the method in code
if you are bootstrapping on JUnit to try and compare an excel sheet and database then I would ust export the table in excel as well and then just do a comparison in excel between columns
Reading from/writing to files is not really what tests should be doing. The input for the tests should be defined in the test, not in the external file which can change - this can either introduce false negatives or even worse false positives (making your tests effectively useless while also giving false confidence that everything is ok because tests are green).
Given your comment (a loop with 10k different parameters coming from file), I would recommend converting this excel file into JUnit Parameterized test. You may want to put the array definition in another class, because 10k lines is quite a lot.
If it is some corporate bureaucracy, and you need to have this excel file, then it makes sense to not write a classic "test". I would recommend just a main method that does the job - reads the file, runs the code, checks the output using simple if (output.equals(expected)) and then writes back to file.
Wrap your AssertEquals(expectedResult, actualResult) with try catch
in catch
catch(AssertionError e){
//deal with e.getMessage or etc.
}
But it not good idea for some reasons, I guess.
And try google something like soft assert
Documentation on assertEquals is pretty clear on what the method does:
Asserts that two objects are equal. If they are not, an AssertionError
without a message is thrown.
You need to wrap the assertion with try-catch block and in the exception handling do Your logging. You will need to make Your own message using the information from the specific test case, but this what You asked for.
Note:
If expected and actual are null, they are considered equal.
I have the following piece of code:
PowerMockito.mockStatic(DateUtils.class);
//And this is the line which does the exception - notice it's a static function
PowerMockito.when(DateUtils.isEqualByDateTime (any(Date.class),any(Date.class)).thenReturn(false);
The class begins with:
#RunWith(PowerMockRunner.class)
#PrepareForTest({CM9DateUtils.class,DateUtils.class})
And I get org.Mockito.exceptions.InvalidUseOfMatchersException...... You cannot use argument matchers outside of verification or stubbing..... (The error appears twice in the Failure Trace - but both point to the same line)
In other places in my code the usage of when is done and it's working properly. Also, when debugging my code I found that any(Date.class) returns null.
I have tried the following solutions which I saw other people found useful, but for me it doesn't work:
Adding
#After
public void checkMockito() {
Mockito.validateMockitoUsage();
}
or
#RunWith(MockitoJUnitRunner.class)
or
#RunWith(PowerMockRunner.class)
Change to
PowerMockito.when(new Boolean(DateUtils.isEqualByDateTime(any(Date.class), any(Date.class)))).thenReturn(false);
Using anyObject() (it doesn't compile)
Using
notNull(Date.class) or (Date)notNull()
Replace
when(........).thenReturn(false);
with
Boolean falseBool=new Boolean(false);
and
when(.......).thenReturn(falseBool);
As detailed on the PowerMockito Getting Started Page, you'll need to use both the PowerMock runner as well as the #PrepareForTest annotation.
#RunWith(PowerMockRunner.class)
#PrepareForTest(DateUtils.class)
Ensure that you're using the #RunWith annotation that comes with JUnit 4, org.junit.runner.RunWith. Because it's always accepted a value attribute, it's pretty odd to me that you're receiving that error if you're using the correct RunWith type.
any(Date.class) is correct to return null: Rather than using a magic "matches any Date" instance, Mockito uses a hidden stack of matchers to track which matcher expressions correspond to matched arguments, and returns null for Objects (and 0 for integers, and false for booleans, and so forth).
So in the end,what worked for me was to export the line that does the exception to some other static function. I called it compareDates.
My implementation:
In the class that is tested (e.g - MyClass)
static boolean compareDates(Date date1, Date date2) {
return DateUtils.isEqualByDateTime (date1, date2);
}
and in the test class:
PowerMockito.mockStatic(MyClass.class);
PowerMockito.when(MyClass.compareDates(any(Date.class), any(Date.class))).thenReturn(false);
Unfortunately I can't say I fully understand why this solution works and the previous didn't.
maybe it has to do with the fact that DateUtils class is not my code, and I can't access it's source, only the generated .class file, but i'm really not sure about it.
EDIT
The above solution was just a workaround that did not solve the need to cover the DateUtils isEqualByDateTime call in the code.
What actually solved the real problem was to import the DateUtils class which has the actual implementation and not the one that just extends it, which was what I imported before.
After doing this I could use the original line
PowerMockito.mockStatic(DateUtils.class);
PowerMockito.when(DateUtils.isEqualByDateTime (any(Date.class),any(Date.class)).thenReturn(false);
without any exception.
I had a similar problem with TextUtils in my Kotlin Test Class
PowerMockito.`when`(TextUtils.isEmpty(Mockito.anyString())).thenReturn(true)
Resolved it by adding below code on top of my Test Class
#PrepareForTest(TextUtils::class)
and called mockStatic this before PowerMockito.`when`
PowerMockito.mockStatic(TextUtils::class.java)
FIREBREATH 1.6 -- VC2010 --
No logging with FBLOG_TRACE("StaticInitialize()", "INIT-trace");
settings
outMethods.push_back(std::make_pair(FB::Log::LogMethod_File, "U:/logs/PT.log"));
...
FB::Log::LogLevel getLogLevel(){
return FB::Log::LogLevel_Trace;
...
changing "FBLOG_TRACE" to "FBLOG_INFO" logging to Logfile works. I don´t understand the reason.
function not inserted in its respective area
FB::Log::LogLevel getLogLevel(){
return FB::Log::LogLevel_Trace; // Now Trace and above is logged.
}
Discription Logging here.
Enabling logging
...
regenerate your project using the prep* scripts
open up Factory.cpp in your project. You need to define the following function inside the class definition for PluginFactory:
...
About log levels
...
If you want to change the log level, you need to define the following in your Factory.cpp:
Referring to the above that means somewhere in "Factory.cpp". that´s incorrect. The description should say -->
If you want to change the log level, you need to define the following function inside the class definition for PluginFactory:
I drag it from bottom of "Factory.cpp" to inside Class PluginFactory.
Now it works as expected !!!
The entire purpose of having different log levels (FBLOG_FATAL, FBLOG_ERROR, FBLOG_WARN, FBLOG_INFO, FBLOG_DEBUG, FBLOG_TRACE) is so that you can configure which level to use and anything below that level is hidden. The default log level in FireBreath is FB::Log::LogLevel_Info, which means that nothing below INFO (such as DEBUG or TRACE) will be visible.
You can change this by overriding FB::FactoryBase::getLogLevel() in your Factory class to return FB::Log::LogLevel_Trace.
The method you'd be overriding is: https://github.com/firebreath/FireBreath/blob/master/src/PluginCore/FactoryBase.cpp#L78
The definition of the LogLevel enum:
https://github.com/firebreath/FireBreath/blob/master/src/ScriptingCore/logging.h#L69
There was a version of FireBreath in which this didn't work; I think it was fixed by 1.6.0, but I don't remember for certain. If that doesn't work try updating to the latest on the 1.6 branch (which is currently 1.6.1 as of the time of this writing but I haven't found time to release yet)
I am using Moq, NUnit, WPF, MVVM, Ninject.
I am writing a test for my LoginViewModel, and in the test when I use the constructor of the LoginViewModel to create a new instance, I am getting a NullReferenceException error. The code compiles and runs, (i.e. when I run the program the LoginView shows, and works with the LoginViewModel to create the correct behaviour etc) but for some reason the UnitTest is crashing.
this is the constructor:
public LoginViewModel(ILoginServices loginServices,IDialogService dialogServices)
{
InitializeFields();
_loginServices = loginServices;
_dialogService = dialogServices;
DomainList = _loginServices.GetDomainListing();
}
I have mocked the dependencies as follows:
Mock<ILoginServices> moq = new Mock<ILoginServices>();
moq.Setup(log =>
log.LoginUser(It.IsAny<string>(),
It.IsAny<string>(),
It.IsAny<string>()))
.Callback<string, string, string>((i, j, k) => CheckArgs(i, j, k));
moq.Setup(log2 =>
log2.GetDomainListing()).Returns(new List<string> { "Domain" });
Mock<IDialogService> moq2 = new Mock<IDialogService>();
I have also tried inserting real services as the parameters.
I have verified that the mocks do work, and the objects these mocks
return are not null.
I have commented out all the code in the constructor.
I have tried inserting the line
LoginViewModel test = new LoginViewModel(_fakeLoginService,_fakeDialogService);
in front of the call to the constructor (to see if it had to do with the original local variable being disposed or something before) and this line crashed instead.
From all I can see this must be the constructor,(but not the code I have written inside it) and that this is solely related to NUnit / Moq as my code still compiles and runs fine.
I have no idea on this one guys, can anyone point me in the right direction?
[Edit]
Ok so I have run through the code and the error comes from this line of code:
ImageSource = (ImageSource)Application.Current.FindResource(_imageName);
This code is going to a ImageDictionary and getting a reference to the image for an undo button in the WindowViewModel (which my LoginViewModel inherits).
My hypotheses as to why its working in the normal running of the application, but not in the testing are:
1) Because I am running the program code through NUnit, the Application.Current object isnt getting property assigned/there is no Application.Current object to get.
**or**
2) Something to do with the fact that because the program code is being run in NUnit, the code doesn't have access to/can't resolve the ImageDictionary to find the image.
I'm leaning more strongly to the first hypothesis, but I'm as of yet not 100% sure, and I am having trouble finding the values of the Application.Current at runtime, cause when I move my cursor over the code the tooltip that normally appears showing the detail of the object that is not appearing.
My new question is: Does any of this make sense? Do you guys know if the Application.Current object exists / can be accessed when running the testing project through NUnit?
Any help will be appreciated.
You are correct. Application.Current is null for Unit tests. You can work around this by injecting the Application object as referencing singletons in code can make life tricky.