I have 2 JSON requests in JMeter, both have similar structure, with some Response Assertions under them. These are the structures of those requests:
But when I run the requests, for the first one, the request is shown as passed, even when all the assertions are failing, while for the second, the entire request fails if even one assertion fails:
I expect the second behaviour, but don't understand why the same is not happening in the first. Is there any way to fix this and get the same behaviour? What could be going wrong here? Or is this some bug?
What do you mean by inconsistent? Any chance you have Ignore Status box checked? Because if you do, then according to JMeter Documentation
When the Ignore Status checkbox is selected, the Response status is forced to successful before evaluating the Assertion
it means that each Assertion artificially treats the SampleResult as successful before evaluating even if the previous assertions have failed.
That's why given the last assertion success the sampler will be marked as successful.
There are 2 possible options:
Untick "Ignore status" box everywhere, in this case any failed assertion will cause the sampler failure (remaining assertions won't be executed)
Or add another JSR223 Assertion to explicitly set the SampleResult to failed if any assertion fails. Example code:
prev.getAssertionResults().each { if (it.isFailure()) { prev.setSuccessful(false) } }
More information: Scripting JMeter Assertions in Groovy - A Tutorial
Related
According to https://labs.omniti.com/labs/jsend,
Fail: When an API call is rejected due to invalid data or call conditions
Error: When an API call fails due to an error on the server
Can this be interpreted as 4xx errors (such as a 404) should always return a Fail, but 5xx errors always correspond to a Error?
This is a great question and I'll answer it with a concrete example from my own experience. In an API for one of my projects, I allow the upload of an Excel spreadsheet, which is processed and the resulting JSON is stored on the server.
If there is an error saving the data to disk, then I would respond with a JSend "error" along with the appropriate error message since the data should have been able to be saved.
If, on the other hand, the data in one of the rows is not valid (maybe an incorrect data type or a range error), then I know the exact row (or rows) of the spreadsheet that were incorrect. In this case, a "fail" response is appropriate as the data property of the JSend response will contain a list of all rows (the row number and the error message) for each unprocessable row.
In the case of an "error" response, I wouldn't have that ability as I would be limited to a single message property. But with the "fail" response, I have the data property available where I can respond with a fine-grained list of issues.
So, while there was no error, the data itself was not correct and the user should go back and look at their spreadsheet and fix the issues identified by the data property in the "fail" response.
Thinking of this in terms of HTTP errors, although an interesting exercise, doesn't always result in an exact mapping (4xx = fail, 5xx = error). It's more about what you want to communicate to the client: something bad happened that shouldn't have happened ("error") or the server is working fine but your data isn't quite up to standards ("fail").
Finally, whether you want to also use HTTP errors is entirely up to you. You could always respond with a 200 and let JSend do the talking. But that's a slightly different (and somewhat religious) discussion. :-)
I hope that helps.
I am using Jmeter and I am testing if a JSON response from a GET request is correct by using JSR223 Assertion.
When the script is correct then the results are correct (unless of course there is something wrong with the response). However if the script is incorrect the test fails even though that the response is accurate, which is the expected behavior.
But then I have to check each line of the script so that I can find the differencies with the response in order to fix it. This wastes a lot of time.
I am not speaking of missing symbols but rather additional lines that are compared to the JSON response but are not actually in it. For example I am comparing country code in the assertion but there is no country code in the response.
Is there a way that JSR223 Assertion can return the differencies in the debugger for Jmeter?
Thank you in advance!
You have AssertionResult shorthand which is an instance of AssertionResult class therefore you can use the following methods:
AssertionResult.setFailure() - to indicate whether assertion successful or not
AssertionResult.setFailureMessage() - to set a custom message on assertion failure
Example code:
if (1 == 1) {
AssertionResult.setFailure(true);
AssertionResult.setFailureMessage("Expecteed something but it wasn't found");
}
Will produce output like:
More information on using JMeter Assertions: How to Use JMeter Assertions in Three Easy Steps
I've implemented logging of all requests/responses using (in my AppHost.cs):
this.RequestFilters.Add(serviceLogFilter.LogRequest);
this.ResponseFilters.Add(serviceLogFilter.LogResponse);
During the logging of the request, I get the new ID from the logging table and put it into IHttpRequest.Items.
My question is: when an error occurs, like if the validation of the incoming data fails, or my business logic throws an Exception, is there a way I can customize the returned error to contain my log reference? Preferably by adding a JSON body to the error response. Alternatively, setting a custom HTTP header?
The idea is that my customer could give me a log reference if something unexcpected happens, and I would find his request and could easily reproduce the problem.
See this question on different ways to Handle exceptions from the old and New API.
To use the IHttpRequest.Items dictionary to pass data between request filters, services and response filters in the same request.
Also the InMemoryRollingRequestLogger implementation for the Request Logger plugin shows another way to be able to log all requests.
I stumbled over a practice that I found to be quite widespread. I even found a web page that gave this a name, but I forgot the name and am not able to find that page on google anymore.
The practice is that every JSON response from a REST service should have the following structure:
{
"status": "ok",
"data": { ... }
}
or in an error case:
{
"status": "error",
"message": "Something went wrong"
}
My question: What is the point why such a "status" property should be required in the JSON? In my opinion that is what HTTP status codes were made for.
REST uses the HTTP means of communication between client and server, for example the "DELETE" verb should be used for deleting. In the same way, 404 should be used if a resource is not found, etc. So inline with that thinking, any error cases should be encoded properly in the HTTP status.
Are there specific reasons to return a HTTP 200 status code in an error case and have the error in the JSON instead? It just seems to make the javascript conditional branches more complex when processing the response.
I found some cases where status could be "redirect" to tell the application to redirect to a certain URL. But if the proper HTTP status code was used, the browser would perform the redirection "for free", maintaining the browsing history properly.
I picture mainly two possible answers from you:
Either there are two quarreling communities with their favorite approach each (use HTTP status always vs. use HTTP status never)
or I am missing an important point and you'll tell me that although the HTTP status should be used for some cases, there are specific cases where a HTTP status does not fit and the "status" JSON property comes into play.
You are right. I think what you are seeing is a side-effect of people not doing REST correctly. Or just not doing REST at all. Using REST is not a pre-requisite for a well-designed application; there is no rule that webapps have to be REST-ful.
On the other hand, for the error condition, sometimes apps want to return a 200 code but an error to represent a business logic failure. The HTTP error codes don't always match the semantics of application business errors.
You are mixing two different Layers here:
HTTP is for establishing (high-level) connections and transferring data. The HTTP status codes thus informs you if and how the connection was established or why it was not. On a successful connection the body of the HTTP request could then contain anything (e.g. XML, JSON, etc.), thus these status code have to define a general meaning. It does not inform you about the correctness or type (e.g. error message or data) of the response.
When using JSON for interchanging data you could certainly omit the status property, however it is easier for you to parse the JSON, if you know if it includes the object you were requesting or an error message by just reading one property.
So, yes, it is perfectly normal to return a 200 status code and have a "status": "error" property in your JSON.
HTTP status codes can be caused by a lot of things, including load balancers, proxies, caches, firewalls, etc. None of these are going to modify your JSON output, unless they completely break it, which can also be treated as an error.
Bottom line: it's more reliable to do it via JSON.
We are in the middle of a ongoing discussion about how to handle REST exceptions.
Response Content type : JSON
Two solutions we have:
Throw all the unchecked exceptions as a JSON response.
Send Request Invalid Response code.
Arguments:
When its a error, why return JSON? Just send a invalid response code.
Counter Argument:
Response code are too technical to handle for normal developers.
Whats your say??
For a JSON API I recently developed I do both. I always respond with valid JSON (well, assuming I respond at all). If I detect an invalid request, I use status 400. If I detect a server error (which I don't believe is caused by an invalid request), I use a 5xx status. The JSON object contains a special key that is only set for errors, with a string value.
I think this is a good solution that respects REST principles, and can be used in multiple ways. The same solution is used by some other JSON APIs, such as Yahoo Search. Try http://search.yahooapis.com/ImageSearchService/V1/imageSearch?appid=YahooDemo&output=json .
Use error codes like for HTTP. So 50* for any exception cause by some internal problem. And 40* for bad arguments. Avoid using your own defined codes as far as its possible. The idea is to have a "uniform" interface.
In general.
204 for success without sending any content
200 for success with a json representation of the resource
And if its not a successful operation return appropriate response code. You can choose to optionally return a json. To simplify things you can have a common format (json) for all error responses.
http://en.wikipedia.org/wiki/REST is a must read before you freeze on your api specs.