How to split multiple values of a parameter in JSON response? - json

My JSON response has multiple values for a single attribute, as copied below.
{
"version": "10.2.2.48",
"systemMessages": [ {
"code": -8010,
"error": "",
"type": "error",
"module": "BROKER"
}],
"locations": [ {
"id": "10101102",
"name": "Bus Stop",
"disassembledName": "Bus",
"coord": [
3755258,
4889121
],
"type": "stop",
"matchQuality": 1000,
"isBest": true,
"modes": [
1,
5,
11
],
"parent": {
"id": "95301001|1",
"name": "Sydney",
"type": "locality"
},
"assignedStops": [ {
"id": "10101102",
"name": "Bus Stop",
"type": "stop",
"coord": [
3755258,
4889121
],
"parent": {
"name": "Sydney",
"type": "locality"
},
"modes": [
1,
5,
11
],
"connectingMode": 100
}]
}]
}
Observe that "modes" has 3 values. If I try to extract the value of modes through JsonSlurper script assertion, it gives the value as [1,5,11] and count as just 1. I want to split them into 3 array elements or variables and need the count as 3. What's the possible script assertion code?
Assertion:
import groovy.json.JsonSlurper
def resp = messageExchange.response.responseContent;
def jsl = new JsonSlurper().parseText(resp);
def modes = jsl.locations.modes
log.info modes
log.info modes.size()
Result:
Wed Feb 13 10:50:49 AEDT 2019:INFO:[[1, 5, 11]]
Wed Feb 13 10:50:49 AEDT 2019:INFO:1

What you are dealing with in this example is a shorthand version of Groovy's spread operator and your code returns a valid result. When you call jsl.locations you actually access a list of all locations objects (a singleton list in your example). When you call
jsl.locations.modes
you use a shorthand version of
jsl.locations*.modes
which is an equivalent of the following code:
jsl.locations.collect { it.modes }
This code means: iterate locations and transform a list of locations into a list of lists of modes of these locations - [[1,5,11]].
Applying the correct solution depends on a few more factors. For instance, you need to consider locations list containing multiple locations - in this case, transformation jsl.locations.modes may produce a result like [[1,5,11],[1,5,11],[2,4,9]] - a list of 3 modes lists.
If you assume that there is always a single location returned then you can simply flatten the final list like:
def modes = jsl.locations.modes.flatten()
assert modes == [1,5,11]
assert modes.size() == 3
However, if locations contains another JSON object, let's say, with exactly the same modes, then it will produce a completely different result:
def modes = jsl.locations.modes.flatten()
assert modes == [1,5,11,1,5,11]
assert modes.size() == 6
In this case, it might be better to use assertions like:
def modes = jsl.locations.modes
assert modes == [[1,5,11],[1,5,11]]
assert modes*.size() == [3,3]
which means:
modes stores 2 lists [1,5,11] and [1,5,11],
and the size of the first list is 3, and the size of the second list is also 3.

Related

JSON query matching

For the given input JSON:
{
"person": {
"name": "John",
"age": 25
},
"status": {
"title": "assigned",
"type": 3
}
}
I need to build a string query that I could use to answer if the given JSON matches it or not. For example if the given person's name is "John" and his age is in the range of 20..30 and his status is not 4.
I need the query to be presented as a string and a commonly known library that can run it. I need it on multiple platforms (iOS, Android, Xamarin). I've tried JSON Path and JSON schemas, but didn't really figure out if it's able to achieve that with them. JSON Path seems to be specified on finding a single value in the JSON by a certain condition and JSON Schema mostly checks for the data structure and types.
Ok, found the solution. If I format the whole input object as a single-element array like that:
[
{
"person": {
"name": "John",
"age": 25
},
"status": {
"title": "assigned",
"type": 3
}
}
]
That will allow me to use JSON Path expressions:
$[?(#.person.name == 'John' && #.person.age >= 20 && #.status.type != 4)]
Basically if it doesn't match there won't be a result.

pandas json_normalize nested json where dictionary only exists on some records

I am trying to run pandas.json_normalize on a data file that has highly varied, nested json, where the content of the records can vary considerably.
I am processing a house listing file and trying to pull out prices. The prices data is stored as follows, and 'prices' is at the first nesting level within the json file:
"prices": [
{
"amountMax": 420000,
"amountMin": 420000,
"availability": "false",
"currency": "USD",
"dateSeen": [
"2020-12-21T11:57:17.190Z",
"2020-12-25T02:35:41.009Z"
],
"isSale": "false",
"isSold": "true",
"pricePerSquareFoot": 235,
"sourceURLs": [
"https://www.redfin.com/FL/Coconut-Creek/.../home/4146834"
]
}, # followed by additional entries
I am using the following line of code, which works if I edit the input file down to a single record that includes a 'prices' section:
df3 = pd.json_normalize(df['records'], record_path='prices',
meta=['id'],
errors='ignore'
)
However, the full file includes many records that do not include a prices section. If I run the code against a file with 2 records (one with, one without), it fails with KeyError: 'prices'
Clearly the 'errors='ignore'' in the json_normalize is not enough to handle the error.
What can I do? I would just like to skip the records without prices entirely.
A list comprehension on your JSON will do it. I've synthesized some JSON to match your description of input data.
js = {
"records": [
{
"prices": [
{
"amountMax": 420000,
"amountMin": 420000,
"availability": "false",
"currency": "USD",
"dateSeen": [
"2020-12-21T11:57:17.190Z",
"2020-12-25T02:35:41.009Z"
],
"isSale": "false",
"isSold": "true",
"pricePerSquareFoot": 235,
"sourceURLs": [
"https://www.redfin.com/FL/Coconut-Creek/.../home/4146834"
]
}
],
"id": 1
},{"id":2}
]
}
pd.json_normalize({"records":[r for r in js["records"] if "prices" in r.keys()]}["records"],record_path="prices",meta="id")

Using Power Query to extract data from nested arrays in JSON

I'm relatively new to Power Query, but I'm pulling in this basic structure of JSON from a web api
{
"report": "Cost History",
"dimensions": [
{
"time": [
{
"name": "2019-11",
"label": "2019-11",
…
},
{
"name": "2019-12",
"label": "2019-12",
…
},
{
"name": "2020-01",
"label": "2020-01",
…
},
…
]
},
{
"Category": [
{
"name": "category1",
"label": "Category 1",
…
},
{
"name": "category2",
"label": "Category 2",
…
},
…
]
}
],
"data": [
[
[
40419.6393798211
],
[
191.44
],
…
],
[
[
2299.652439184997
],
[
0.0
],
…
]
]
}
I actually have 112 categories and 13 "times". I figured out how to do multiple queries to turn the times into column headers and the categories into row labels (I think). But the data section is alluding me. Because each item is a list within a list I'm not sure how to expand it all out. Each object in the date array will have 112 numbers and there will be 13 objects. If that all makes sense.
So ultimately I want to make it look like
2019-11 2019-20 2020-01 ...
Category 1 40419 2299
Category 2 191 0
...
First time asking a question on here, so hopefully this all makes sense and is clear. Thanks in advance for any help!
i am also researching this exact thing and looking for a solution. In PQ, it displays nested arrays as a list and there is a function to extract values choosing a separating characterenter image description here
So this becomes, this
enter image description here
= Table.TransformColumns(#"Filtered Rows", {"aligned_to_ids", each Text.Combine(List.Transform(_, Text.From), ","), type text})
However the problem i'm trying to solve is when the nested json has multiple values like this: enter image description here
And when these LIST are extracted then an error message is caused, = Table.TransformColumns(#"Extracted Values1", {"collaborators", each Text.Combine(List.Transform(_, Text.From), ","), type text})
Expression.Error: We cannot convert a value of type Record to type Text.
Details:
Value=
id=15890
goal_id=323
role_id=15
Type=[Type]
It seems the multiple values are not handled and PQ does not recognise the underlying structure to enable the columns to be expanded.

Karate- Error when comparing json objects

I am trying to match some parameters with json reposne
my actual response is like
{
"timestamp": 1595994767386,
"country": "MH",
"accessible_device_types": [
{
"name": "ESS Client",
"raw_name": "ABC",
"permission": 7,
"permission_bits": {
"INSTALL_LIMITED_RELEASE_SOFTWARE": true,
"INSTALL_LATEST_SOFTWARE_ONLY": true,
"INSTALL_SOFTWARE": true
}
},
used below statment for comparing:
match response.accessible_device_types contains [{"raw_name": "ABC"}]
Reason for error from report: expected: {raw_name=ABC}, reason: actual value does not contain expected
Looks like comparing without quotes. Why is it taking out the quotes? Any recommendations
How to compare "INSTALL_SOFTWARE": true
2 options:
* def nameAbc = {"raw_name": "ABC"}
* match response.accessible_device_types contains '#(^nameAbc)'
This will work in 0.9.6.RC4 onwards:
* match response.accessible_device_types contains deep {"raw_name": "ABC"}

How to get the length of a JSON array using JSONPath?

If I have JSON document like this
[
{
"number" : "650-462-9154",
"type" : "main"
},
{
"number" : "650-462-1252",
"type" : "fax"
}
]
What JSONPath can I use to get the array length (which is 2), without hardcoding any property values?
Using the tool I have, here is some examples they gave, which doesn't help me figure out what value I need.
[
{
"type": "add",
"id": "tt0484562",
"version": 1,
"lang": "en",
"fields": {
"title": "The Seeker: The Dark Is Rising",
"director": "Cunningham, David L.",
"genre": ["Adventure","Drama","Fantasy","Thriller"],
"actor": ["McShane, Ian","Eccleston, Christopher","Conroy, Frances",
"Crewson, Wendy","Ludwig, Alexander","Cosmo, James",
"Warner, Amelia","Hickey, John Benjamin","Piddock, Jim",
"Lockhart, Emma"]
}
},
{
"type": "delete",
"id": "tt0484575",
"link_ref": null,
"version": 2
}
]
$.[0].genre ---> 0
$.[0].fields.genre ---> 1
$.[0].fields.genre[*] ---> 4
$.[*].type ---> 2
$.[1].link_ref ---> 1
You can get the length of the array from the first example by using the .length() function like this :
$.length
That should return something like this:
[
2
]
As for the second example, you can access any array keys using regular dot notation ($.node1.node2.node3) and call the length() function on the node key you wish to get the length of. For example, if you wanted to get the number of values in the actor array you could do something like:
$..actor.length
Which will return something like this:
[
10
]
Tested on https://codebeautify.org/jsonpath-tester and http://jsonpath.com/ .
You can find more functions in the jsonpath github repo.