JQ: How to create JSON object using data picked by jq? - json

I have a complex JSON file that contains hundreds of "attributes" with their types identified by "objectTypeAttributeId".
I know that objectTypeAttributeId=328 means tickedid, objectTypeAttributeId=329 contains array of hostnames etc..
There is simplified version of the file:
{
"objectEntries": [
{
"attributes": [
{
"id": 279792,
"objectTypeAttributeId": 328,
"objectAttributeValues": [
{
"displayValue": "ITSM-24210"
}
]
},
{
"id": 279795,
"objectTypeAttributeId": 329,
"objectAttributeValues": [
{
"displayValue": "testhost1"
},
{
"displayValue": "testhost2"
}
]
},
{
"id": 279793,
"objectTypeAttributeId": 330,
"objectAttributeValues": [
{
"displayValue": "28.02.2020 11:45"
}
]
}
]
}
]
}
I need to create output JSON using particular values picked out (according to the "objectTypeAttributeId" value) of input JSON in format like this:
{
"tickets": [
{
"ticketid": "ITSM-24210",
"hostnames": ["testhost1", "testhost2"],
"date": "28.02.2020 11:45"
}
]
}
I am new in jq, in the XSLT it is solvable using static template with placeholders for picked values.
I have tried this approach, there is my jq filter:
.objectEntries[].attributes[] |
{ticketid: select(.objectTypeAttributeId == 328) | .objectAttributeValues[0].displayValue},
{hostnames: select(.objectTypeAttributeId == 329) | [.objectAttributeValues[].displayValue]},
{date: select(.objectTypeAttributeId == 330) | .objectAttributeValues[0].displayValue}
but the result of this approach is:
{
"ticketid": "ITSM-24210"
}
{
"hostnames": [
"testhost1",
"testhost2"
]
}
{
"date": "28.02.2020 11:45"
}
And all my subsequent tries to format output better ends in broken jq filter or filter that does not return anything.
Please any ideas how to solve this problem?

Assuming a ticket is to be generated for each object entry:
{tickets: [
.objectEntries[]
| [.attributes[]
| [.objectTypeAttributeId,
(.objectAttributeValues | map(.displayValue))] as [$id, $val]
| if $id == 328 then {ticketId: $val[0]}
elif $id == 329 then {hostnames: $val}
elif $id == 330 then {date: $val[0]}
else empty end
] | add
]}
Online demo

Here we go, it's not pretty, there may be a better solution but it works: https://jqplay.org/s/sxussfa2Vj
.objectEntries | {tickets: map(.attributes |
{ticketID: (reduce .[] as $r (null; if $r.objectTypeAttributeId == 328
then $r.objectAttributeValues[0].value else . end)),
date: (reduce .[] as $r (null; if $r.objectTypeAttributeId == 330
then $r.objectAttributeValues[0].value else . end)),
hostnames: (reduce .[] as $r ([]; if $r.objectTypeAttributeId == 329
then $r.objectAttributeValues | map(.value) else . end))})}
There's a lot of unpacking and repacking going on here that sort of distracts from the core. You have an array of tickets (aka entries), and over those we map. The various properties we have to grab from different entries of an array, which is done using reduce. Reduce goes through the array of objects and picks out the right one and keeps track of the value.
Maybe there's a nice way, but this works already, so you can play with it further, trying to simplify.
Your original solution almost works, you did a good job there, just needed a map:
.objectEntries[].attributes |
{ticketid: . | map(select(.objectTypeAttributeId == 328))[0] |
.objectAttributeValues[0].displayValue,
date: . | map(select(.objectTypeAttributeId == 330))[0] |
.objectAttributeValues[0].displayValue,
hostnames: . | map(select(.objectTypeAttributeId == 329))[0] |
[.objectAttributeValues[].displayValue]}
Try it out, it even works with multiple tickets ;)
https://jqplay.org/s/ydoCgv9vsI

Related

Intersecting nested JSON documents with jq

I'm looking for a general approach to intersect two nested JSON-objects, in order to retrieve all common key-value pairs.
Given the following JSON-documents
{
"a": 0,
"b": [
"ba",
{
"bb": {
"bba": "a",
"bbc": [1, 2]
}
},
{
"bc": {
"bca": [2],
"bcb": 5
}
}
],
"c": 1
}
and
{
"a": 1,
"b": [
"ba",
{
"bb": {
"bba": "a",
"bbc": [1,3]
}
}
],
"c": 1
}
I want to find their intersection.
I expect to get the following JSON-document which only contains those key-value-pairs JSON-objects that are present in both inputs:
{
"b": [
"ba",
{
"bb": {
"bba": "a",
"bbc": [1]
}
}
],
"c": 1
}
I looked into jq's documentation, but could not find a hint on how to do this.
My tries to use the minus-operator were unsuccessful yielding object ... and object cannot be subtracted.
cat obj-list.txt | jq -c '.[1] - (.[0] - .[1])'
Can you give me some hints on how to accomplish a intersection of nested JSON-objects with jq?
Thank you already in advance for helping me out.
Update:
Given that the JSON-objects do not have the exact same structure, but differ like a = {"a": [{"aa":1]}, b = {"a": 0} for example. I'm interested in a solution on how to add error checking to catch Cannot index number with string-errors and the like?
Assuming $a holds the contents of one of your JSON entities and $b holds the other, then the following will perform the type of intersection you describe:
reduce ($a|paths) as $p (null;
($a|getpath($p)) as $va
| [try ($b|getpath($p)) // empty] as $vb
| if ($vb | length > 0) and ($va == $vb[0])
then setpath($p;$va) else . end)
You may however wish to explore variations of this, e.g. if there is nothing in common.
Footnote 1:
The above is easily adapted to similar problems. For example, if the common structure is all that matters:
def structural_intersection($a;$b):
reduce ($a|paths|select(.[-1]|type=="string")) as $p (null;
($a|getpath($p)) as $va
| [try ($b|getpath($p)) // empty] as $vb
| if ($vb | length > 0)
then setpath($p;null) else . end) ;
Footnote 2:
To handle arrays using array-intersection, you might wish to consider the following, but please be aware that in some cases, this hybrid approach will probably produce results that you might not expect:
def special_intersection($a;$b):
def i(x;y): x - (x-y);
reduce ($a|paths) as $p (null;
($a|getpath($p)) as $va
| [try ($b|getpath($p)) // empty] as $vb
| if ($vb | length > 0)
then if ($va == $vb[0])
then setpath($p;$va)
elif ($va|type == "array") and ($vb[0]|type) == "array"
then setpath($p; i($va; $vb[0]))
else . end
else . end) ;

How to filter out a JSON based on list of paths in JQ

Given an arbitrary JSON input:
{
"id":"038020",
"title":"Teenage Mutant Ninja Turtles: Out of the Shadows",
"turtles":[
{
"name":"Leonardo",
"mask":"blue"
},
{
"name":"Michelangelo",
"mask":"orange"
},
{
"name":"Donatello",
"mask":"purple"
},
{
"name":"Raphael",
"mask":"red"
}
],
"summary":"The Turtles continue to live in the shadows and no one knows they were the ones who took down Shredder",
"cast":"Megan Fox, Will Arnett, Tyler Perry",
"director":"Dave Green"
}
And an arbitrary list of JQ paths like [".turtles[].name", ".cast", ".does.not.exist"], or any similar format
How can I create new JSON with only the information contained in the paths of the list?
In this case the expected result would be:
{
"turtles":[
{
"name":"Leonardo"
},
{
"name":"Michelangelo"
},
{
"name":"Donatello"
},
{
"name":"Raphael"
}
],
"cast":"Megan Fox, Will Arnett, Tyler Perry"
}
I've seen similar solutions in problems like "removing null entries" from a JSON using the walk function present in jq1.5+, somewhat along the lines of:
def filter_list(input, list):
input
| walk(
if type == "object" then
with_entries( select(.key | IN( list )))
else
.
end);
filter_list([.], [.a, .b, .c[].d])
But it should take in account the full path in the JSON somehow.
What is the best approach to solve this problem?
If $paths contains an array of explicit jq paths (such as [ ["turtles", 0, "name"], ["cast"]]), the simplest approach would be to
use the following filter:
. as $in
| reduce $paths[] as $p (null; setpath($p; $in | getpath($p)))
Extended path expressions
In order to be able to handle extended path expressions such as ["turtles", [], "name"], where [] is intended to range over the indices of the turtles array, we shall define the following helper function:
def xpath($ary):
. as $in
| if ($ary|length) == 0 then null
else $ary[0] as $k
| if $k == []
then range(0;length) as $i | $in[$i] | xpath($ary[1:]) | [$i] + .
else .[$k] | xpath($ary[1:]) | [$k] + .
end
end ;
For the sake of exposition, let us also define:
def paths($ary): $ary[] as $path | xpath($path);
Then with the given input, the expression:
. as $in
| reduce paths([ ["turtles", [], "name"], ["cast"]]) as $p
(null; setpath($p; $in | getpath($p)) )
produces the output shown below.
Using path
It is worth point out that one way to handle expressions such as ".turtles[].name" would be to use the builtin filter path/1.
For example:
# Emit a stream of paths:
def paths: path(.turtles[].name), ["cast"];
. as $in
| reduce paths as $p (null; setpath($p; $in | getpath($p)))
Output:
{
"turtles": [
{
"name": "Leonardo"
},
{
"name": "Michelangelo"
},
{
"name": "Donatello"
},
{
"name": "Raphael"
}
],
"cast": "Megan Fox, Will Arnett, Tyler Perry"
}

jq json move object to nested object and iterating over unknown names/numbers of objects

I've been looking over several examples of 'jq' parsing of json strings, all very helpful, but not conclusive for my particular problem
Here's my input json :
{
"outcome" : "TrBean",
"result" : {"TrAct" : {
"executiontime" : 16938570,
"invocations" : 133863,
"waittime" : 4981
}}
}
{
"outcome" : "WwwBean",
"result" : {}
}
{
"outcome": "CRFeatureBean",
"result": {
"CRChannels": {
"executiontime": 78127,
"invocations": 9983,
"waittime": 213
},
"getCRChannels": {
"executiontime": 98704,
"invocations": 10113,
"waittime": 212
},
"getCRToMigrate": {
"executiontime": 32,
"invocations": 4,
"waittime": 0
},
"getCRId": {
"executiontime": 28198633,
"invocations": 747336,
"waittime": 19856
}
}
}
I'm trying to feed graphite via collectd exec plugin (PUTVAL), so I need info in one line. I tried with ./jq '.result|to_entries[]|{"method:" .key, "inv": .value.invocations}|"PUTVAL \(.method)/invoke:\(.invokes)"' ... but I need to have "outcome" in every line too.
Also I do not know the amount, nor the names of the result-objects
So, I'd like to end up with :
TrBean_TrAct
WwwBean
CRFeatureBean_CRChannels
CRFeatureBean_getCRChannels
CRFeatureBean_getCRToMigrate
CrFeatureBean_getCRId
The following jq filter produces the desired output when jq is invoked with the -r command-line option:
((.result | keys_unsorted[]) // null) as $key
| if $key == null then .outcome
else [.outcome, $key] | join("_")
end
There are of course many possible variations, e.g.
((.result | keys_unsorted[]) // null) as $key
| [.outcome, ($key // empty)]
| join("_")
or if you want a short one-liner:
.outcome + ("_" + (.result | keys_unsorted[]) // null)
In any case, the key to simplicity here is to generate the keys of .result as a stream. Handling the "edge case" makes the solution slightly more complicated than it would otherwise be, i.e. .outcome + "_" + (.result | keys_unsorted[])
Example invocation: jq -r -f program.jq input.json

Building objects with jq

Using jq I'd like to convert data of the format:
{
"key": "something-else",
"value": {
"value": "bloop",
"isEncrypted": false
}
}
{
"key": "something",
"value": {
"value": "blah",
"isEncrypted": false
}
}
To the format:
{
something: "blah",
something-else: "bloop"
}
Filtering out 'encrypted values' along the way. How can I achieve this? I've gotten as far as the following:
.parameters | to_entries[] | select (.value.isEncrypted == false) | .key + ": " + .value.value
Which produces:
"something-else: bloop"
"something: blah"
Close, but not there just yet. I suspect that there's some clever function for this.
Given the example input, here's a simple solution, assuming the stream of objects is available as an array. (This can be done using jq -s if the JSON objects are given as input to jq, or in your case, following your example, simply using .parameters | to_entries).
map( select(.value.isEncrypted == false) | {(.key): .value.value } )
| add
This produces the JSON object:
{
"something-else": "bloop",
"something": "blah"
}
The key ideas here are:
the syntax for object construction: {( KEYNAME ): VALUE}
add
One way to gain an understanding of how this works is to run the first part of the filter (map(...)) first.
Using keys_unsorted
If you want to avoid the overhead of to_entries, you might want to consider the following approach, which piggy-backs off your implicit description of .parameters:
.parameters
| [ keys_unsorted[] as $k
| if .[$k].isEncrypted == false
then { ($k) : .[$k].value } else empty end ]
| add

How do I count specific JSON items in Bash, or another shell?

I have JSON data in the following form:
{
"home": [{
"pageid": "about yft",
"pageData": "0908782"
},
{
"pageData": "09897"
}]}
How do I get the total number of home->pageid items?
Here is what I have tried so far:
$curFileName ={{Path_of_json_file.json}}
appName0=$(cat $curFileName | jq -c '.["home"][]["pageid"]' | sed 's/"//g');
${#appName0[#]} # to get the length of pageid but didn't success..
But it didn't return the desired results.
Collect the pageid items of "string" type into an array, then return the length of the array:
n=$(jq '[.home[].pageid | select(type == "string")] | length' < file.json)
Alternatively, check if the items in the home array have pageid. If the item has the property, put 1 into the result array, otherwise put zero. Finally sum the zeroes and ones with add function:
n=$(jq '[.home[] | if has("pageid") then 1 else 0 end] | add' < file.json)
Here is how you should print the number:
printf '%d\n' "$n"
Sample Output
3
The input as originally given by the OP was invalid, and also didn't include an object without .pageid. In this response, the following JSON will be used:
{
"home": [
{
"pageid": "about yft",
"pageData": "0908782"
},
{
"pageData": "09897"
}
]
}
Consider now these two filters, both of which yield 1 for the above JSON:
[.home[] | select(has("pageid")) ] | length
[.home[] | .pageid//empty ] | length
If .home was huge, then a more efficient approach would be as follows:
def count(s): reduce s as $i(0; .+1);
count(.home[] | select(has("pageid")))
And here's an efficient, one-line variant:
reduce (.home[].pageid?//empty) as $x (0; .+1)
Counting the number of distinct items
To obtain the number of distinct home->pageid items, the simplest would be to use unique|length rather than length, e.g. in either of the first two solutions above.
After doing lots of search get the best result which is given below
echo `cat $curFileName | jq '.["home"][]["pageid"]' | wc -l `
Where is:
$curFileName = {{Path_of_json_file.json}}
.["home"][]["pageid"] = target Json Object
json_file.json :
{
"home": [
{
"pageid": "about yft",
"pageData": "0908782"
},
{
"pageData": "09897"
}
]
}
for more
http://www.compciv.org/recipes/cli/jq-for-parsing-json/