Related
I have a json like this
[
{
"name": "hosts",
"ipaddress": "1.2.3.4",
"status": "UP",
"randomkey": "randomvalue"
},
{
"name": "hosts",
"ipaddress": "5.6.7.8",
"status": "DOWN",
"newkey": "newvalue"
},
{
"name": "hosts",
"ipaddress": "9.10.11.12",
"status": "RESTART",
"anotherkey": "anothervalue"
}
]
I want to merge the objects and looking for some output like this
[
{
"name": "hosts", //doesn't matter if it is ["hosts"]
"ipaddress": ["1.2.3.4", "5.6.7.8", "9.10.11.12"],
"status": ["UP", "DOWN", "RESTART"],
"randomkey": ["randomvalue"],
"newkey": ["newvalue"],
"anotherkey": ["anothervalue"]
}
]
I can hardcode each and every key and do something like this - { ipaddress: (map(.ipaddress) | unique ) } + { status: (map(.status) | unique ) } + { randomkey: (map(.randomkey) | unique ) }
The important ask here is the values are random and cannot be hardcoded.
Is there a way i can merge all the keys without hardcoding the key here?
Using reduce, then unique would be one way:
jq '[
reduce (.[] | to_entries[]) as {$key, $value} ({}; .[$key] += [$value])
| map_values(unique)
]'
[
{
"name": [
"hosts"
],
"ipaddress": [
"1.2.3.4",
"5.6.7.8",
"9.10.11.12"
],
"status": [
"DOWN",
"RESTART",
"UP"
],
"randomkey": [
"randomvalue"
],
"newkey": [
"newvalue"
],
"anotherkey": [
"anothervalue"
]
}
]
Demo
Using group_by and map, then unique again would be another:
jq '[
map(to_entries[]) | group_by(.key)
| map({key: first.key, value: map(.value) | unique})
| from_entries
]'
[
{
"anotherkey": [
"anothervalue"
],
"ipaddress": [
"1.2.3.4",
"5.6.7.8",
"9.10.11.12"
],
"name": [
"hosts"
],
"newkey": [
"newvalue"
],
"randomkey": [
"randomvalue"
],
"status": [
"DOWN",
"RESTART",
"UP"
]
}
]
Demo
I have a JSON file like this :
{
"filter": [
{
"id": "id_1",
"criteria": {
"from": "mail1#mail.com"
},
"action": {
"addLabelIds": [
"Label_1"
],
"removeLabelIds": [
"IMPORTANT",
"SPAM"
]
}
},
{
"id": "id_2",
"criteria": {
"from": "mail2#mail.com"
},
"action": {
"addLabelIds": [
"Label_2"
],
"removeLabelIds": [
"IMPORTANT",
"SPAM"
]
}
}
]
}
I would like to extract id value only where from = mail2#mail.com
I have tried this but does not work :
jq '.filter[] | select(.criteria.from | test("mail2#mail.com"; "i")) | .id'
Have you got any idea?
Thanks
L.
thanks to #Charles Duffy
solution:
select(.criteria.from == "mail2#mail.com")
Using Groovy, requirement is to collect a map's nested element values, along with its top-level element value.
Unsure if a recursive method is needed.
Sample JSON
{
"items": [
{
"attribute": "Type",
"options":
[
{
"label": "Type1",
"value": "1"
},
{
"label": "Type2",
"value": "2"
}
]
},
{
"attribute": "Size",
"options": [
{
"label": "SizeA",
"value": "A"
},
{
"label": "SizeB",
"value": "B"
}
]
}
]
}
Expected output after collect
[
{attribute=Type,label=Type1,value=1},
{attribute=Type,label=Type2,value=2},
{attribute=Size,label=SizeA,value=A},
{attribute=Size,label=SizeB,value=B}
]
You can solve this combining multiple lists of options obtained through the collect method using a collectMany.
See the following snippet of code:
def input = """
{
"items": [
{
"attribute": "Type",
"options": [
{
"label": "Type1",
"value": "1"
},
{
"label": "Type2",
"value": "2"
}
]
},
{
"attribute": "Size",
"options": [
{
"label": "SizeA",
"value": "A"
},
{
"label": "SizeB",
"value": "B"
}
]
} ]
}"""
def json = new groovy.json.JsonSlurper().parseText(input)
/* collectMany directly flatten the two sublists
* [ [ [ type1 ], [ type2 ] ], [ [ sizeA ], [ sizeB ] ] ]
* into
* [ [type1], [type2], [sizeA], [sizeB] ]
*/
def result = json.items.collectMany { item ->
// collect returns a list of N (in this example, N=2) elements
// [ [ attribute1: ..., label1: ..., value1: ... ],
// [ attribute2: ..., label2: ..., value2: ... ] ]
item.options.collect { option ->
// return in [ attribute: ..., label: ..., value: ... ]
[ attribute: item.attribute ] + option
}
}
assert result == [
[ attribute: "Type", label: "Type1", value: "1" ],
[ attribute: "Type", label: "Type2", value: "2" ],
[ attribute: "Size", label: "SizeA", value: "A" ],
[ attribute: "Size", label: "SizeB", value: "B" ],
]
Thanks to Giuseppe!
I had solved this using a less groovy method:
def result = [];//Create new list with top-level element and nested elements
json.items.each {item ->
result.addAll(item.options.collect{ option ->
[attribute: /"${item?.attribute?:''}"/, label: /"${option?.label?:''}"/, value: /"${option?.value?:''}"/ ]
})
};
I've been thinking and searching for a long time, but I didn't find out what I'm looking for.
I'm using JQ to parse tshark (-ek) json output, but I'm a jq newby
When a frame is multivalue I have a JSON similar to this:
{
"timestamp": "1525627021656",
"layers": {
"frame_time_epoch": [
"1525627021.656417000"
],
"ip_src": [
"10.10.10.10"
],
"ip_src_host": [
"test"
],
"ip_dst": [
"10.10.10.11"
],
"ip_dst_host": [
"dest_test"
],
"diameter_Event-Timestamp": [
"May 6, 2018 19:17:02.000000000 CEST",
"May 6, 2018 19:17:02.000000000 CEST"
],
"diameter_Origin-Host": [
"TESTHOST",
"TESTHOST"
],
"diameter_Destination-Host": [
"DESTHOST",
"DESTHOST"
],
"diameter_CC-Request-Type": [
"2",
"2"
],
"diameter_CC-Request-Number": [
"10",
"3"
],
"diameter_Rating-Group": [
"9004",
"9001"
],
"diameter_Called-Station-Id": [
"testing",
"testing"
],
"diameter_User-Name": [
"testuser",
"testuser"
],
"diameter_Subscription-Id-Data": [
"66666666666",
"77777777777"
],
"gtp_qos_version": [
"0x00000008",
"0x00000005"
],
"gtp_qos_max_dl": [
"8640",
"42"
],
"diameter_Session-Id": [
"test1;sessionID1;test1",
"test2;sessionID2;test2"
]
}
}
As you can see, many keys are array and I want to iterate them to create different json objects in a result like this:
{
"frame_time_epoch": [
"1525627021.656417000"
],
"ip_src": [
"10.10.10.10"
],
"ip_src_host": [
"test"
],
"ip_dst": [
"10.10.10.11"
],
"ip_dst_host": [
"dest_test"
],
"diameter_Event-Timestamp": [
"May 6, 2018 19:17:02.000000000 CEST"
],
"diameter_Origin-Host": [
"TESTHOST"
],
"diameter_Destination-Host": [
"DESTHOST"
],
"diameter_CC-Request-Type": [
"2"
],
"diameter_CC-Request-Number": [
"3"
],
"diameter_Rating-Group": [
"9001"
],
"diameter_Called-Station-Id": [
"testing"
],
"diameter_User-Name": [
"testuser"
],
"diameter_Subscription-Id-Data": [
"77777777777"
],
"gtp_qos_version": [
"0x00000005"
],
"gtp_qos_max_dl": [
"42"
],
"diameter_Session-Id": [
"test2;sessionID2;test2"
]
}
{
"frame_time_epoch": [
"1525627021.656417000"
],
"ip_src": [
"10.10.10.10"
],
"ip_src_host": [
"test"
],
"ip_dst": [
"10.10.10.11"
],
"ip_dst_host": [
"dest_test"
],
"diameter_Event-Timestamp": [
"May 6, 2018 19:17:02.000000000 CEST"
],
"diameter_Origin-Host": [
"TESTHOST"
],
"diameter_Destination-Host": [
"DESTHOST"
],
"diameter_CC-Request-Type": [
"2"
],
"diameter_CC-Request-Number": [
"10"
],
"diameter_Rating-Group": [
"9004"
],
"diameter_Called-Station-Id": [
"testing"
],
"diameter_User-Name": [
"testuser"
],
"diameter_Subscription-Id-Data": [
"66666666666"
],
"gtp_qos_version": [
"0x00000008"
],
"gtp_qos_max_dl": [
"8640"
],
"diameter_Session-Id": [
"test1;sessionID1;test1"
]
}
Another hand made example:
INPUT:
{
"key_single": ["single_value"],
"key2": ["single_value"],
"multiple_value_key": ["value1" , "value2"],
"any_key_name": ["value4" ,"value5"]
}
{
"key_single": ["single_value"],
"key2": ["single_value"],
"multiple_value_key": ["value6" , "value7", "value8"],
"any_key_name": ["value9" ,"value10" , "value11"]
}
Desired output:
{
"key_single": ["single_value"],
"key2": ["single_value"],
"multiple_value_key": ["value1"],
"any_key_name": ["value4"],
}
{
"key_single": ["single_value"],
"key2": ["single_value"],
"multiple_value_key": ["value2"],
"any_key_name": ["value5"],
}
{
"key_single": ["single_value"],
"key2": ["single_value"],
"multiple_value_key": ["value6"],
"any_key_name": ["value9"],
}
{
"key_single": ["single_value"],
"key2": ["single_value"],
"multiple_value_key": ["value7"],
"any_key_name": ["value10"],
}
{
"key_single": ["single_value"],
"key2": ["single_value"],
"multiple_value_key": ["value8"],
"any_key_name": ["value11"],
}
Could you help Me?
Thanks in advance.
It looks like you want to take, in turn, the i-th element of the selected arrays. Using your second example, this could be done like so:
range(0; .multiple_value_key|length) as $i
| . + { multiple_value_key: [.multiple_value_key[$i]],
any_key_name: [.any_key_name[$i]] }
The output in compact form:
{"key_single":["single_value"],"key2":["single_value"],"multiple_value_key":["value1"],"any_key_name":["value4"]}
{"key_single":["single_value"],"key2":["single_value"],"multiple_value_key":["value2"],"any_key_name":["value5"]}
{"key_single":["single_value"],"key2":["single_value"],"multiple_value_key":["value6"],"any_key_name":["value9"]}
{"key_single":["single_value"],"key2":["single_value"],"multiple_value_key":["value7"],"any_key_name":["value10"]}
{"key_single":["single_value"],"key2":["single_value"],"multiple_value_key":["value8"],"any_key_name":["value11"]}
Here is a simple solution to the problem as described in the "comments", though the
output differs slightly from that shown in the Q.
For clarity, a helper function is defined for producing the $i-th slice of an object,
that is, for all array-valued keys with array-length greater than 1,
the value is replaced by the $i-th item in the array.
def slice($i):
map_values(if (type == "array" and length>1)
then [.[$i]]
else . end);
The solution is then simply:
.layers
| range(0; [.[] | length] | max) as $i
| slice($i)
Output
{
"frame_time_epoch": [
"1525627021.656417000"
],
"ip_src": [
"10.10.10.10"
],
"ip_src_host": [
"test"
],
"ip_dst": [
"10.10.10.11"
],
"ip_dst_host": [
"dest_test"
],
"diameter_Event-Timestamp": [
"May 6, 2018 19:17:02.000000000 CEST"
],
"diameter_Origin-Host": [
"TESTHOST"
],
"diameter_Destination-Host": [
"DESTHOST"
],
"diameter_CC-Request-Type": [
"2"
],
"diameter_CC-Request-Number": [
"10"
],
"diameter_Rating-Group": [
"9004"
],
"diameter_Called-Station-Id": [
"testing"
],
"diameter_User-Name": [
"testuser"
],
"diameter_Subscription-Id-Data": [
"66666666666"
],
"gtp_qos_version": [
"0x00000008"
],
"gtp_qos_max_dl": [
"8640"
],
"diameter_Session-Id": [
"test1;sessionID1;test1"
]
}
{
"frame_time_epoch": [
"1525627021.656417000"
],
"ip_src": [
"10.10.10.10"
],
"ip_src_host": [
"test"
],
"ip_dst": [
"10.10.10.11"
],
"ip_dst_host": [
"dest_test"
],
"diameter_Event-Timestamp": [
"May 6, 2018 19:17:02.000000000 CEST"
],
"diameter_Origin-Host": [
"TESTHOST"
],
"diameter_Destination-Host": [
"DESTHOST"
],
"diameter_CC-Request-Type": [
"2"
],
"diameter_CC-Request-Number": [
"3"
],
"diameter_Rating-Group": [
"9001"
],
"diameter_Called-Station-Id": [
"testing"
],
"diameter_User-Name": [
"testuser"
],
"diameter_Subscription-Id-Data": [
"77777777777"
],
"gtp_qos_version": [
"0x00000005"
],
"gtp_qos_max_dl": [
"42"
],
"diameter_Session-Id": [
"test2;sessionID2;test2"
]
}
I'm trying to extract the sids, ll, state, name, smry values in my JSON file using jq and export to a csv.
JSON File (out.json):
{
"data": [
{
"meta": {
"uid": 74529,
"ll": [
-66.9333,
47.0667
],
"sids": [
"CA008102500 6"
],
"state": "NB",
"elev": 1250,
"name": "LONG LAKE"
},
"smry": [
[
"42",
"1955-02-23"
]
]
},
{
"meta": {
"uid": 74534,
"ll": [
-67.2333,
45.9667
],
"sids": [
"CA008103425 6"
],
"state": "NB",
"elev": 150.9,
"name": "NACKAWIC"
},
"smry": [
[
"40",
"1969-02-23"
]
]
},
{
"meta": {
"uid": 74549,
"ll": [
-67.4667,
47.4667
],
"sids": [
"CA008104933 6"
],
"state": "NB",
"elev": 794,
"name": "ST QUENTIN"
},
"smry": [
[
"M",
"M"
]
]
},
{
"meta": {
"uid": 74550,
"ll": [
-67.2667,
45.1833
],
"sids": [
"CA008104936 6"
],
"state": "NB",
"elev": 36.1,
"name": "ST STEPHEN"
},
"smry": [
[
"48",
"1900-02-23"
]
]
},
{
"meta": {
"uid": 74554,
"ll": [
-67.25,
47.2667
],
"sids": [
"CA008105000 6"
],
"state": "NB",
"elev": 915.4,
"name": "SISSON DAM"
},
"smry": [
[
"35",
"1955-02-23"
]
]
}
]
}
Terminal Code:
jq '.data | [ {sids, ll, state, name, smry} ]' out.json
I am getting the following errors:
assertion "cb == jq_util_input_next_input_cb" failed: file "/usr/src/ports/jq/jq-1.5-3.x86_64/src/jq-1.5/util.c", line 371, function: jq_util_input_get_position
Aborted (core dumped)
Example Expected Output:
sids, ll, state, name, smry
CA008102500, -66.9333, 47.0667, NB, LONG LAKE, 42,1955-02-23
CA008103425, -67.2333, 45.9667, NB, NACKAWIC, 35,1955-02-23
What am I doing wrong?
It's a bit more complex because you need to flatten sids, ll and smry before you can flatten the whole record. I recommend to create a jq file:
foo.jq:
.data[]|{
"sids":(.meta.sids[0]|split(" ")[0]),
"ll":(.meta.ll|map(tostring)|join(",")),
"state":.meta.state,
"name":.meta.name,
"smry":(.smry[]|join(","))
}|join(",")
# or, for robust csv output
# } | #csv
And then call:
jq -rf foo.jq file.json
Output:
CA008102500,-66.9333,47.0667,NB,LONG LAKE,42,1955-02-23
CA008103425,-67.2333,45.9667,NB,NACKAWIC,40,1969-02-23
CA008104933,-67.4667,47.4667,NB,ST QUENTIN,M,M
CA008104936,-67.2667,45.1833,NB,ST STEPHEN,48,1900-02-23
CA008105000,-67.25,47.2667,NB,SISSON DAM,35,1955-02-23