I am having problem of extracting the json value when the data has a leading [. Ex: [{"userID":"12"}]
I used "jp#gc - Dummy Sampler" to mock a test json data and when removed the leading [ and trailing ], the JSON Extractor seems to be able to read the json. Ex: {"userID":"12"}
A leading [ is valid JSON format; therefore, I am not sure if my assumption is correct. Is my finding sounds correct? If yes, what is the best way for me to remove the leading and trailing [].
thanks
You can use .. - a deep scan operator in order to get the values from JSON no matter how many nested levels deep they are:
You may also find JMeter's JSON Path Extractor Plugin - Advanced Usage Scenarios article useful as it contains several most frequently required examples of working with JSON Path Extractor
"[]" means it's an array.
So to extract 12 you would use:
[0].userID
An alternative is to use:
$..userID
Related
I'm using a Microsoft REST API to query a Azure application, oauth and request goes without problem.
The response from InvokeHTTP has this format
{"#odata.context":"https://****.dynamics.com/api/data/v9.1/$metadata#endpoint","value":[ here comes the actual JSON result in format {
"#odata_etag" : "W/\"555598\"", "field":"value...},...]
,"#odata.nextLink":"https://****.dynamics.com/api/data/v9.1/endpoint?$skiptoken.....}
I need to extract the nextLink for pagination and Value to continue the flow and store the result.
When I try to parse with inferAvroSchema so I can start working with it throws this error "Illegal initial character: #odata.etag"
My Idea was to inferAvroSchema, then EvaluateJsonPath to extract the odata tags and then extract the values.
I tried using EvaluateJsonPath on the result asking to create an attribute for $.#odata.context but it doesn't find the item either, I'm sure is something about the #.
I can also replace all the # of the incoming flow for another char, but don't know if that makes sense.
I'm feeling that i'm not using a correct approach, but NIFI + odata doesn't give me results on google or here.
I'm open to any suggestions!
thank you!
Schema fields cannot contain #. You could replace the #, however you must be sure not to replace it in actual content like email addresses. Another solution is to transform the API response using JoltTransformJSON processor, such that your flow can work with it:
GenerateFlowFile:
For the JoltTransformJSON processor provide following Jolt specification:
[
{
"operation": "shift",
"spec": {
"\\#odata.nextLink": "next"
}
}
]
Leave the default values for the other properties. You can play around with Jolt here: http://jolt-demo.appspot.com/
EvaluateJsonPath:
Result:
Notice that the url is now part of the flowfile attributes.
Your hunch is correct, you can only have valid characters for the field names on the schema type you are using, avro or JSON.
You could get NiFi to remove illegal characters with the replacetext proceasor, have a read here on what is valid: http://avro.apache.org/docs/current/spec.html#names
It appears that the standard Apache NiFi readers/writers can only parse JSON input based on Avro schema.
Avro schema is limiting for JSON, e.g. it does not allow valid JSON properties starting with digits.
JoltTransformJSON processor can help here (it doesn't impose Avro limitations to how the input JSON may look like), but it seems that this processor does not support batch FlowFiles. It is also not based on the readers and writers (maybe because of that).
Is there a way to read arbitrary valid batch JSON input, e.g. in multi-line form
{"myprop":"myval","12345":"12345",...}
{"myprop":"myval2","12345":"67890",...}
and transform it to other JSON structure, e.g. defined by JSON schema, and e.g. using JSON Patch transformation, without writing my own processor?
Update
I am using Apache NiFi 1.7.1
Update 2
Unfortunately, #Shu's suggestion did work. I am getting same error.
Reduced the case to a single UpdateRecord processor that reads JSON with numeric properties and writes to a JSON without such properties using
myprop : /data/5836c846e4b0f28d05b40202
mapping. Still same error :(
it does not allow valid JSON properties starting with digits?
This bug NiFi-4612 fixed in NiFi-1.5 version, We can use AvroSchemaRegistry to defined your schema and change the
Validate Field Names
false
Then we can have avro schema field names starting with digits.
For more details refer to this link.
Is there a way to read arbitrary valid batch JSON input, e.g. in multi-line form?
This bug NiFi-4456 fixed in NiFi-1.7, if you are not using this version of NiFi then we can do workaround to create an array of json messages with ,(comma delimiter) by using.
Flow:
1.SplitText //split the flowfile with 1 line count
2.MergeRecord //merge the flowfiles into one
3.ConvertRecord
For more details regards to this particular issues refer to this link(i have explained with the flow).
Currently testing system where the output is in the form of formatted json.
As part of my tests I need to extract and validate two values from the json record.
The values both have individual identifiers on them but don't appear in the same part of the record, so I can't just grab a single long string.
Loose format of the information in both cases:
"identifier1": [{"identifier2":"idname","values":["bit_I_want!]}]
In the case of the bit I want, this can either be a single quoted value (e.g. "12345") or multiple quoted values (e.g. "12345","23456","98765").
In both cases I'm only interested in validating the whole string of values, not individual values from the set.
Can anyone recommend which of the various extractors in Jmeter would be best to achieve this?
Many Thanks!
The most obvious choicse seems to be JSON Path Assertion (available via JMeter Plugins), it allows not only executing arbitrary JSON queries but conditionally failing the sampler basing on actual and expected result match.
The recommended way of installing JMeter Plugins and keeping them up-to-date is using JMeter Plugins Manager
JMeter 3.1 comes with JSON Extractor to parse JSON response. you could use this expression $.identifier1[0].values
as the JSON Path to extract the values.
If your JSON response is going to simple always as shown in your question, you could use Regular Expression Extractor as well. Advantage is it is faster than JSON extractor. The regular expression would be "values":\[(.*?)\]
Reference: http://www.testautomationguru.com/jmeter-response-data-extractors-comparison/
I'm not using JMeter too often, and I've run into very specific issue.
My REST response is always "the same", but nodes are not in the same order due to various reasons. As well, I can't put here whole response due to sensitive data, but let's use these dummy one:
First time response might be:
{
"properties":{
"prop1":false,
"prop2":false,
"prop3":165,
"prop4":"Audi",
"prop5":true,
"prop6":true,
"prop7":false,
"prop8":"1",
"prop9":"2.0",
"prop10":0
}
}
Then other time it might be like this:
{
"properties":{
"prop2":false,
"prop1":false,
"prop10":0,
"prop3":165,
"prop7":false,
"prop5":true,
"prop6":true,
"prop8":"1",
"prop9":"2.0",
"prop4":"Audi"
}
}
As you can see, the content it self is the same, but order of nodes it's not. I have 160+ nodes and thousand of possible response orders.
Is there an easy way to compare two JSON responses comparing matching key - values, or at least to sort the response, and then compare it with sorted one in assertion patterns?
I'm not using any plugins, just basic Apache JMeter.
Thanks
I've checked using Jython, you need to download the Jython Library and save to your jmeter lib directory.
I've checked 2 JSONs with Sampler1 and Sampler2, on Sampler1 I've add a BeanShell PostProcessor with this code:
vars.put("jsonSampler1",prev.getResponseDataAsString());
On Sampler2 I've add a BSF Assertion, specifying jython as the language and with the following code:
import json
jsonSampler1 = vars.get("jsonSampler1")
jsonSampler2 = prev.getResponseDataAsString()
objectSampler1 = json.loads(jsonSampler1)
objectSampler2 = json.loads(jsonSampler2)
if ( objectSampler1 != objectSampler2 ):
AssertionResult.setFailure(True)
AssertionResult.setFailureMessage("JSON data didn't match")
Yoy can find the whole jmx in this GistHub
You will most probably have to do this with a JSR223 Assertion and Groovy.
http://jmeter.apache.org/usermanual/component_reference.html#JSR223_Assertion
http://docs.groovy-lang.org/latest/html/api/groovy/json/JsonSlurper.html
Note that if you know Python, you might look at using Jython + JSR223.
I would just set up 10 jp#gc - JSON Path Assertions. Documentation for figuring out JSON Path format is here and you can test how it would work here.
For your example you would the assertion (Add > Assertion > jp#gc - JSON Path Assertions), then to test the prop 1 put:
$.properties.prop1
in the JSON Path field, click the Validate Against Expected Value checkbox, and put
false
in the expected value field. Repeat those steps for the other 9 changing the last part of the path to each key and the value you expected back in the expected value field.
This extractor is jmeter add on found here.
In JMeter, I need to extract some fields (City, Classification, and Chain) from a JSON response:
{
"StoreCode": "111243",
"StoreName": "Spencer - Sec 14 Gurgaon",
"Address1": "Gurgaon-Sector-14",
"Address2": "NCR",
"Pin": "110000",
"City": "NCR",
"Classification": "Vol 4",
"Chain": "Spencers",
"Version": "20281",
"VisitType": "Weekly"
}
Can it be done using the regular expression extractor? Is there another option?
If this piece of JSON is the all the response - it makes sense to use Regular Expression Extractor.
If you receive larger or more complicated structure - it's better to use special JSON Path Extractor available through plugin. JSON Path expressions for your JSON response would be something like $.City, $.Chain, etc.
See "Parsing JSON" chapter of Using the XPath Extractor in JMeter guide for more details on JSON Path language and on how to install the plugin.
Very easy with the plugin mentioned. See this for example. Here is link to plugin.
My biggest thing to understand was the flow. In your jmeter test you need to have an httprequest that returns data (in this case json data). So running your test you'd see json in the Response Data tab if you have a View Results Tree listener going. If you have this right click on the HttpRequest you want data from. ADD => Post Processors => jp#gc - JSON Path Extractor. Inside that extractor, you can name it anything you want.
The variable name should be one you already have defined in a User Defined Variables config element. The JSON Path would start with a dollar sign, then a dot, then the name of the key you want the value for from your json. So for me: $.logId the value from ... "logId": 4, ... in my json. It will store the number 4 in my userdefined variable. The default value can be set to something you'd never see like -1 or null or false etc...
BTW you reference your variable in your tests with ${variablename}. If putting into json and its a string do "${variablename}". Hope it helps.
Lots of the way to find out with the help of regular expression. Sharing one of them.
"City": "(.*)",
"Classification": "(.*)",
"Chain": "(.*)",