I'm trying to capture multiple values from a JSON Response I get but seem to be unable to get them from the same random node.
I've tried to place multiple variables in the same extractor using ";" and this works but it goes through the nodes randomly and doesn't extract the values I need from the same one
The source would be something like
[
{
"Disabled": false,
"Group": null,
"Selected": false,
"Text": "Text1",
"Value": "Value1"
},
{
"Disabled": false,
"Group": null,
"Selected": false,
"Text": "Text2",
"Value": "Value2"
}
]
and I would like to get from any of the 2 nodes (randomly matched) both the Text and Value in either a array that I can use or 2 variables.
So far it seems to take value from one node and text from another (in lengthier sources)
so my desired outcome would be either text1 and value1 or text2 and value2, not a mix of both..
Add JSR223 PostProcessor as a child of the request which returns the above JSON
Put the following code into "Script" area:
def json = new groovy.json.JsonSlurper().parse(prev.getResponseData())
def randomEntry = json.get(org.apache.commons.lang3.RandomUtils.nextInt(0, json.size()))
vars.put('strBrandID', randomEntry.Value)
vars.put('strBrandName', randomEntry.Text)
That's it, you should be able to refer the Text/Value pairs as ${strBrandID} and ${strBrandName} where required
More information:
Groovy: Parsing and producing JSON
Apache Groovy - Why and How You Should Use It
Related
Consider these two JSON files:
{
"tables": [
{
"columns": [
{
"name": "action",
"type": "string"
},
{
"name": "region",
"type": "string"
},
{
"name": "count_",
"type": "long"
},
{
"name": "min_timestamp",
"type": "datetime"
},
{
"name": "max_timestamp",
"type": "datetime"
}
],
"name": "PrimaryResult",
"rows": [
[
"ChangePassword",
"CN",
1,
"2022-07-19T11:51:52.8430729Z",
"2022-07-19T11:51:52.8430729Z"
]
]
}
]
}
{
"tables": [
{
"columns": [
{
"name": "action",
"type": "string"
},
{
"name": "region",
"type": "string"
},
{
"name": "count_",
"type": "long"
},
{
"name": "min_timestamp",
"type": "datetime"
},
{
"name": "max_timestamp",
"type": "datetime"
}
],
"name": "PrimaryResult",
"rows": [
[
"ChangePassword",
"CN",
1,
"2022-07-19T11:51:52.8430729Z",
"2022-07-19T11:51:52.8430729Z"
],
[
"Register",
"CN",
1,
"2022-07-19T11:51:52.8430729Z",
"2022-07-19T11:51:52.8430729Z"
]
]
}
]
}
They have the exact same schema and are almost the same. The difference is that first file has 1 row in the rows property, the second file has 2 rows in the rows property.
I use ConvertFrom-Json to load the files into PowerShell. When I start accessing .tables.rows[0], I expect that it will return the first row.
However, in the first file that has only 1 row, it actually returns the first column of the first row.
(gc file1.json | ConvertFrom-Json).tables.rows[0]
Outputs:
ChangePassword
If there are more than 1 row as in the second file, then the .tables.rows[0] behaves as expected.
(gc file2.json | ConvertFrom-Json).tables.rows[0]
Outputs:
ChangePassword
CN
1
2022-07-19T11:51:52.8430729Z
2022-07-19T11:51:52.8430729Z
How can I reliably process these JSON files in PowerShell regardless of if they have 1 row or multiple rows in them?
btw. These are actually JSON files produced by az cli as results from Azure Application Insights queries.
tl;dr
You need to avoid member-access enumeration if you want to access collection-valued properties as-is, which in your case means the following, as nimizen demonstrated in a comment:
# Note the [0] after .tables
# (-Raw was added for efficiency)
(gc -Raw file1.json | ConvertFrom-Json).tables[0].rows[0]
Because your tables property is itself an array, accessing just .tables results in member-access enumeration (even though the array happens to have just one element).
Using index [0] to target that one element explicitly allows access to its (one and only) .rows property without the latter's value being subject to implicit enumeration, as explained in the next section.
You're seeing a surprising aspect of PowerShell's member-access enumeration feature:
The values obtained from the members of a collection's elements are emitted as if they were sent to the pipeline, one by one, which has the following implications:
As happens by default in the pipeline, any such value is enumerated if it happens to be collection-valued, i.e. its elements are emitted, one by one.
On accessing the resulting output, if there's only one output object (whether or not that happens to be a collection itself), it is captured as-is; if there are two or more, they are implicitly collected in a regular PowerShell array (of type [object[]]).
The upshot for enumerated member values that happen to contain collections is:
The input collections themselves are always lost, because only their elements are output.
If there are multiple collection-valued member values, you'll end up with a single, flat array, invariably of type [object[]], that is the concatenation of the elements of all collections involved.
If there is only one collection-valued member value, and that value happens to be a single-element collection, you'll end up with that single element as-is (only with multiple elements would you end up with an [object[]] array).
This surprising behavior is the subject of GitHub issue #6802, though note that the behavior is unlikely to change, so as not to break backward compatibility.
A simplified example to demonstrate what happened in your case:
Note: JSON is incidental to the problem, so I'm using a [pscustomobject] instance below.
# Construct a single-element array that has a [pscustomobject] element.
# whose .prop property contains a single-element array whose only
# element is another array, typed [int[]].
$array = , [pscustomobject] #{ prop = , [int[]] (1, 2) }
# Use member-access enumeration to access the .prop member value
# of all elements in the array (here, there's only one), and get
# the first element.
$array.prop[0] # !! -> scalar 1, i.e. the first elem. of the *nested* array,
# !! because member-access enumeration *enumerated* the
# !! single-element outer array.
# Without member-access enumeration, the value of .prop is accessed as-is,
# as a nested array:
$array[0].prop[0] # -> [int[]] (1, 2), as expected.
don't know how to stop powershell behaving that way,
but got it "reliable" by
$test=Get-Item ".\file.json" | get-content | convertfrom-json
if($test.tables.rows[0].count -gt 1 ){
$result = $test.tables.rows[0][0]
}else{
$result = $test.tables.rows[0]
}
{
"ID":0,
"OrganizationId":"",
"OrganizationName":"",
"Name":"",
"IsActive":"True",
"Type":2,
"AppliesTo":1,
"TagHOD":"",
"DisplayAsPrimary":"false",
"Values":[
]
}
Above is my json request which I have stored in a data file
Below is my json request body which I am getting after sending a parameter into it. It is sorted into alphabetical order which I don't want. I want the same order as above eg ID Should be first then OrganizationId
{
"AppliesTo": 1,
"DisplayAsPrimary": "false",
"ID": 0,
"IsActive": "True",
"Name": "TAG1205510333275",
"OrganizationId": 2404,
"OrganizationName": "",
"TagHOD": "",
"Type": 2,
"Values": [
{
"HODEmail": "tagsapiautomationae#mailinator.com",
"Id": 1,
"IsDeleted": false,
"Text": "Level20"
}
]
}
The JSON specification states: "An object is an unordered set of name/value pairs."
When working with JSON (and objects in most languages), the properties in objects are inherently unordered. You can't rely on different systems giving you the properties in the same order you supply them. In fact, you can't even rely on a single system giving you the properties in the same order all the time within a given execution of the code, even though many systems do behave that way.
If you want to preserve ordering, you either need to use an array to store the data, or you can use an array of object property names that stores the keys in the order you want, so you can use that array to reference them in the desired order later.
EG:
keyorder = ["ID",
"OrganizationId",
"OrganizationName",
"Name",
"IsActive",
"Type",
"AppliesTo",
"TagHOD",
"DisplayAsPrimary",
"Values"
]
You can then loop over this array when accessing elements in your object, so you are always accessing them in your defined order.
In python, with an object named "data" this would look like:
for key in keyorder:
print data.get(key)
I want to split and transfer the json data in NiFi, Here is my json structure look like this;
I want to split json by id1,id2 array of json transfer to respective processor group say example processor_group a,b. I tried with evaluate json path $.id1,$.id2 but i didn't get exact solution. Can you please help me out from this issue;
{
"id1": [{
"u_name": "aa"
}, {
"addr": "bb"
}],
"id2": [{
"u_name": "aa"
}, {
"addr": "bb"
}]
}
The processor you're looking for is SplitJSON.
Configure it as follows:
Then, you'll receive two FlowFiles:
First one will contain the id1:
[{
"u_name": "aa"
}, {
"addr": "bb"
}]
second one will contain id2:
[{
"u_name": "aa"
}, {
"addr": "bb"
}]
Here is how to get to the values you want with EvaluateJsonPath:
#varun_rathinam Accessing json in an array object via EvaluateJsonPath can be quite confusing. I also notice the structure of your json is kind of confusing with same values in both. I have adjusted id2 for cc and dd for testing so that I can tell id1 and id2 values apart.
The solution you want is (see template for exact string values):
Notice we use the normal tree for each json object ( $.object ) then access the array ( 0, 1 ) then access the array's objects. Also notice it is possible to access the json object array with or without a . before the [.
Reference:
https://community.cloudera.com/t5/Support-Questions/how-to-extract-fields-in-flow-file-which-are-surrounded-by/m-p/208635
You can also find my template during testing of your issue on my GitHub:
https://github.com/steven-dfheinz/NiFi-Templates/blob/master/NiFI_EvaluateJsonPath_Demo.xml
I'm trying strong text to extract json with regular extraction then post it in the next request body data with formatted json. For the json that I extracted, they are not formatted and I'm just wondering if there is any function or way to format it?
Get request with regular expression extractor (extracted the bold section)
{
"groupedData": [{
"key": "FirstItem",
"count": 1,
"groupID": 1,
"items": [{
**"keyID": 97215,
"film": {
"name": xxxx,
"id": xxx,
"vendorID": 0,
"type": "PG",
"xxxx": xxx
},
"subGroups": null**
}],
"totalRows": 1
}]
}
Post in the next request with extracted data (JSON data extracted from above request with regular expression is appearing as whole string and just wondering how can i format in this body data?)
{
"keyID": 123,
"name": "SYSGEN",
"period": {
"keyID": 427,
},
"periodID": 427,
"items": [{
**${JSON}**
}],
"group": 0,
"selRow": false,
"rowId": 1,
"$rowState": {
"invalid": false,
},
"XXXX": XXXX,
}],
"ZZZZZZ": "ZZZZZ"
}
You can format the JSON using __groovy() function, i.e.
If you have a JMeter Variable foo where the extracted JSON data is stored and refer to it as ${foo} in the HTTP Request
Replace your ${foo} variable reference with the following function:
${__groovy(groovy.json.JsonOutput.prettyPrint(vars.get('foo')),)}
That's it, the above Groovy expression will format the JSON which lives in ${foo} JMeter Variable
If you want to get response text between given boundaries use Boundary Extractor:
Left Boundary:
"items": [{
Right Boundary:
}],
You can also test it using View Results Tree
The Boundary Extractor Tester only works for text responses. It shows the plain text in the upper panel. The "Test" button allows the user to apply the Boundary Extractor query to the upper panel and the results will be displayed in the lower panel.
I'm getting following data in response of a request:
{
"items": [
{
"id": 54925,
"currCode": "USD",
"lastUpdated": 1531233169000
},
{
"id": 54926,
"currCode": "USD",
"lastUpdated": 1531233169000
},
{
"id": 54927,
"currCode": "USD",
"lastUpdated": 1531233169000
}
],
"totalCount": 3
}
As we can see there are three different ids in the data(54925,54926,54927)
I want to perform iterate over all these ids and perform some operation( basically I want to use like foreach(String id: ids) { request(id);}
I added a JSON extractor as follows:
As per my research(research link) it's supposed to store all the ids in the id_list
After this added a foreach loop to iterate over these values:
But somehow the it's not going inside this for loop. What I'm doing wrong here?
is there any other way to fetch all these ids and loop through them?
You forgot to mention in JSON Extractor you expect it to return all values by setting Match No. as -1
-1 means extract all results, they will be named as _N