group json output by array input value - json

Input data: /tmp/h1.tabs, tab delimited:
INPUT A1
Flavor Controller
Comment Disabled
State DiskOne
INPUT B2
Flavor Controller
Comment Not Applicable
State Not Applicable
ConnectorCount 12
Alarm Alarm Not present
INPUT C3
Flavor Controller
Comment Not Applicable
Media Not Applicable
ConnectorCount 0
State Alarm Not present
Desired Output:
{
"A1": {
"Comment": "Disabled",
"Flavor": "Controller",
"State": "DiskOne"
},
"B2": {
"Alarm": "Alarm Not present",
"Comment": "Not Applicable",
"ConnectorCount": "12",
"Flavor": "Controller",
"State": "Not Applicable"
},
"C3": {
"Comment": "Not Applicable",
"ConnectorCount": "0",
"Flavor": "Controller",
"Media": "Not Applicable",
"State": "Alarm Not present"
}
}
Each INPUT dictionary key could also be an array instead of another dictionary.
{
"A1": [
{ "Comment": "Disabled" },
{ "Flavor": "Controller" },
{ "State": "DiskOne" }
],
About as close as I am able to get is something like this:
jq -Rsn '[inputs|. / "\n"|.[] / "\t"|select(length > 0)|. as $input|(if $input[0] == "INPUT" then $input[1] else { ($input[0]): $input[1] } end)]' /tmp/h1.tabs
[
"A1",
{
"Flavor": "Controller"
},
{
"Comment": "Disabled"
},
{
"State": "DiskOne"
},
"B2",
{
"Flavor": "Controller"
},
I've tried expressions like if $input[0] == "INPUT" then $block = $input[1], but I am not having any luck with an assignment, so I can't use the assignment in the output. Really what I think I need is a variable set to whatever INPUTs value is every time I pass it. Then I can format the output as needed. I am just missing some key magic. I've been banging on this for a while, here is more that doesn't work...
# vim:ft=ansible:tabstop=8 expandtab shiftwidth=2 softtabstop=2
"unknown" as $block
|[
inputs
|. / "\n"
|
(
.[]
| select(length > 0)
|.
)
]
|(.[] / "\t")
|select(length > 0)
|. as $input
|
(
if $input[0] == "INPUT" then $block = $input[1] else empty end
|({($block): [($input[0]):($input[1])]})
) | add
Still learning :-)

reduce is your friend..
reduce (inputs / "\t") as [$k, $v] ([];
if $k == "INPUT" then
.[0] = $v
else
.[1][.[0]] += {($k): $v}
end
) | .[1]
Note that you need to specify -n and -R options on the command line for this to work

Using GNU awk, gawkextlib and gawk-json:
$ gawk '
#load "json"
BEGIN{
FS="\t"
}
{
if($1=="INPUT")
tl=$2
else
data[tl][$1]=$2
}
END {
print json_toJSON(data)
}' file # | jq '.' # for eye-friendly formating
Output (jq assisted):
{
"C3": {
"Comment": "Not Applicable",
"State": "Alarm Not present",
"ConnectorCount": "0",
"Flavor": "Controller",
"Media": "Not Applicable"
},
"A1": {
"Comment": "Disabled",
"State": "DiskOne",
"Flavor": "Controller"
},
"B2": {
"Comment": "Not Applicable",
"State": "Not Applicable",
"Alarm": "Alarm Not present",
"ConnectorCount": "12",
"Flavor": "Controller"
}
}

Related

jq update json document to alter an array element

I know this has to be simple, but for some reason it's eluding me how to find an element given a condition and modify one of its fields. The doc should be fully output (sed style) with the edit made.
{
"state": "wait",
"steps": {
"step1": [
{ "name":"Foo", "state":"wait" },
{ "name":"Bar", "state":"wait" }
],
"step2": [
{ "name":"Foo", "state":"wait" },
{ "name":"Zoinks", "state":"ready" }
],
"step3": [
{ "name":"Foo", "state":"cancel" }
]
}
}
I'm expecting something like this should be workable.
jq '. | (select(.steps[][].name=="Foo" and .steps[][].state=="wait") |= . + {.state:"Ready"}'
or
jq '. | (select(.steps[][]) | if (.name=="Foo" and .state=="wait") then (.state="Ready") else . end)
The desired output, of course, would be
{
"state": "wait",
"steps": {
"step1": [
{ "name":"Foo", "state":"ready" },
{ "name":"Bar", "state":"wait" }
],
"step2": [
{ "name":"Foo", "state":"ready" },
{ "name":"Zoinks", "state":"ready" }
],
"step3": [
{ "name":"Foo", "state":"cancel" }
]
}
}
Instead, when I'm not getting cryptic errors, I'm either modifying a top-level field in the document or modifying the field for all the elements or repeated the entire doc multiple times.
Any insights greatly appreciated.
Thanks.
p.s. is there a better syntax than [] to wildcard the named-elements under steps? Or after the pipe to identify the indices discovered by the select?
Pipe the output of .steps[][] into a select call that chooses the objects with the desired name and state values, then set the state value on the result.
$ jq '(.steps[][] | select(.name == "Foo" and .state == "wait")).state = "ready"' tmp.json
{
"state": "wait",
"steps": {
"step1": [
{
"name": "Foo",
"state": "ready"
},
{
"name": "Bar",
"state": "wait"
}
],
"step2": [
{
"name": "Foo",
"state": "ready"
},
{
"name": "Zoinks",
"state": "ready"
}
],
"step3": [
{
"name": "Foo",
"state": "cancel"
}
]
}
}
You can help confirm this using diff (the first jq just normalizes the formatting so that only the changes made by the second one show up in the diff):
$ diff <(jq . tmp.json) <(jq '...' tmp.json)
7c7
< "state": "wait"
---
> "state": "ready"
17c17
< "state": "wait"
---
> "state": "ready"

How to check for null or empty in jq and substitute for empty string in jq transformation

How to check for null or empty in jq and substitute for empty string in jq transformation.
Example in below JSON, this is the JQ
JQ:
.amazon.items[] | select(.name | contains ("shoes")) as $item |
{
activeItem: .amazon.activeitem,
item : {
id : $item.id,
state : $item.state,
status : if [[ $item.status = "" or $item.status = null ]];
then 'IN PROCESS' ; else $item.status end
}
}
JSON:
{
"amazon": {
"activeitem": 2,
"items": [
{
"id": 1,
"name": "harry potter",
"state": "sold"
},
{
"id": 2,
"name": "adidas shoes",
"state": "in inventory"
},
{
"id": 3,
"name": "watch",
"state": "returned"
},{
"id": 4,
"name": "Nike shoes",
"state": "in inventory"
}
]
}
}
I want to add a default string "In Process" if the status is empty or Null.
Based on Item condition, using the query below and take the first object from the filtered results.
code
.amazon.items[] | select(.name | contains ("shoes"))
code
Expected Output:
{
"activeitem": 2,
"item": {
"id": 2,
"name": "adidas shoes",
"state": "in inventory",
"status": "IN PROCESS"
}
}
The key here is to use |=:
.amazon.item.status |=
if . == null or . == ""
then "IN PROCESS"
else .
end

Print result of timestamp function one line before string

At the bottom is some example json data which I'd like to add timestamps to. i.e "time": "1544785866.176123088" The timestamps need to be unique on each iteration and need to occur on the line above "bladesetName". Unfortunately, what I have so far is not working. After printing the time to the file in the needed places I was going to add 1ns to each entry. Is there a better way of doing this?
timestamp() {
date +%s.%N
}
awk "/bladesetName/{ print '"time" "$(timestamp)"' }1" volumes3.json
This throws an error
awk: /bladesetName/{ print 'time 1544786158.644385726' }1
awk: ^ invalid char ''' in expression
.
{
"pasxml": {
"#version": "6.0.0",
"system": {
"name": "example1",
"IPV4": "0.0.0.0",
"alertLevel": "critical",
"state": "online"
},
"volumes": {
"volume": [
{
"#id": "1",
"name": "/",
"bladesetName": {
"#id": "1",
"#text": "Set-1"
},
"state": "Online",
"raid": "Object RAID6+",
"director": "Shelf-001,1",
"volservice": "0x0400000000000004(FM)",
"objectId": "I-xD0200000000000004-xG7ee84b0c-xU00004a75726a656e",
"recoveryPriority": "1",
"efsaMode": "retry",
"spaceUsedGB": "0",
"spaceAvailableGB": "693903.38",
"hardQuotaGB": "0.52",
"softQuotaGB": "0.52",
"userQuotaPolicy": {
"#inherit": "0",
"#text": "disabled"
},
"stats": null
},
{
"#id": "8",
"name": "/datacentre/archvol/pan101",
"bladesetName": {
"#id": "1",
"#text": "Set-1"
},
"alertLevel": "critical",
"state": "Online",
"raid": "Object RAID6+",
"director": "Shelf-008,1",
"volservice": "0x04000000000000ec(FM)",
"objectId": "I-xD02000000000000ec-xG5c7aef6f-xU00004a75726a656e",
"recoveryPriority": "50",
"efsaMode": "retry",
"spaceUsedGB": "117000.09",
"spaceAvailableGB": "693903.38",
"hardQuotaGB": "117000.00",
"softQuotaGB": "90000.00",
"userQuotaPolicy": {
"#inherit": "1",
"#text": "disabled"
},
"stats": null
},
Here’s one approach, which uses jq:
def addnow:
. as $in
| to_entries
| (map(.key)|index("bladesetName")) as $i
| if $i
then .[0:$i]+[{key:"time",value:now}]+.[$i:] | from_entries
else $in
end ;
walk(if type=="object" then addnow else . end)
If you really want the time as a string, simply add tostring, as in (now|tostring)
If your jq does not have walk, then now would be a good time to upgrade; otherwise simply copy and paste its def, which can easily be found on the web, e.g. by googling: jq “def walk”

jq recursively update values for certain elements

The intent for the JSON data below is to update the value of the field dst with the value of src within all elements of type t, regardless of depth within the tree, while at the same time preserving the whole structure of the data.
Is this possible with jq? My several attempts have boiled down to the following command that is not working to achieve the intended purpose:
$ jq -r 'map_values(select(.. | .type? == "t" |= (.dst = .src)))'
{
"a": "b",
"c": [
{
"type": "t",
"src": "xx",
"dst": "zz"
},
{
"type": "t",
"src": "xx",
"dst": "zz"
}
],
"d": [
{
"e": [
{
"type": "t",
"src": "xx",
"dst": "zz"
}
]
},
{
"type": "t2",
"src": "xx",
"dst": "zz"
}
]
}
Is this possible with jq?
jq is Turing-complete :-)
Here's a simple solution:
walk( if type == "object" and .type == "t" then .dst = .src else . end)
If your jq does not have walk/1, then it might be a good time to upgrade (to jq 1.6); otherwise, you can snarf its def from the web, e.g. by googling: jq "def walk"
Alternatively ...
reduce paths as $x (.;
if (getpath($x)|.type? // false) == "t"
then setpath( $x + ["dst"]; getpath( $x + ["src"] ))
else . end)

How to search a json with jq for values?

I have a json of this structure:
{
"nodes": {
"60e327ee58a0": {
"nodeinfo": {
"network": {
"mesh": {
"bat0": {
"interfaces": {
"wireless": [
"<mac-address-removed>"
],
"tunnel": [
"<mac-address-removed>"
]
}
}
},
"mac": "<mac removed>",
"addresses": [
"<ipv6 removed>",
"<ipv6 removed>"
]
},
"hardware": {
"model": "TP-Link TL-WR841N/ND v10",
"nproc": 1
},
"software": {
"batman-adv": {
"compat": 15,
"version": "2015.1"
},
"autoupdater": {
"branch": "stable",
"enabled": true
},
"firmware": {
"release": "v2016.1+1.0.1",
"base": "gluon-v2016.1"
},
"status-page": {
"api": 1
},
"fastd": {
"enabled": true,
"version": "v17"
}
},
"hostname": "Antoniusweg12",
"system": {
"site_code": "ffmsd03"
},
"node_id": "60e327ee58a0"
},
"lastseen": "2016-04-14T12:39:04",
"flags": {
"gateway": false,
"online": true
},
"firstseen": "2016-03-16T15:14:04",
"statistics": {
"clients": 1,
"gateway": "de:ad:be:ef:43:02",
"rootfs_usage": 0.6041666666666667,
"loadavg": 0.09,
"uptime": 1822037.41,
"memory_usage": 0.8124737210932025,
"traffic": {
"rx": {
"packets": 50393821,
"bytes": 5061895206
},
"forward": {
"packets": 173,
"bytes": 17417
},
"mgmt_rx": {
"packets": 47453745,
"bytes": 6623785282
},
"tx": {
"packets": 1205695,
"bytes": 173509528,
"dropped": 5683
},
"mgmt_tx": {
"packets": 37906725,
"bytes": 11475209742
}
}
}
},
"30b5c2b042f4": {
<next block...>
And I want to query it with jq for the hostname, the mac or the IPv6.
cat nodes.json |jq -c '.nodes[] | select(.nodes[]| contains("Antoniusweg12"))'
Most examples do not fit this kind of json structure as the objects have an index
Thanks for help in advance.
If you're going to filter, you need to drill down to the property that you want to check for and see if it matches your criteria. You can't expect to just give a name and you'll magically be presented with the results you want.
Searching by hostname, it is found on the .nodeinfo.hostname property of each node:
$ jq -c --arg hostname "Antoniusweg12" \
'.nodes[] | select(.nodeinfo.hostname == $hostname)' nodes.json
Similarly for the mac address, it's found on the .nodeinfo.network.mac property:
$ jq -c --arg mac "aa:bb:cc:dd:ee:ff" \
'.nodes[] | select(.nodeinfo.network.mac == $mac)' nodes.json
For the ip addresses, there's an array of them but it's not that much different in the query. They're found on the .nodeinfo.network.addresses property:
$ jq -c --arg ip "aaaa:bbbb:cccc:dddd::1" \
'.nodes[] | select(.nodeinfo.network.addresses[] == $ip)' nodes.json
Here's another take on the question. Suppose you want to find all occurrences of the key "hostname" for which the value is "Antoniusweg12",
no matter where the key/value combination occurs.
The following will reveal the path to the key/value combination of interest:
paths as $p
| select ( $p[-1] == "hostname" and getpath($p) == "Antoniusweg12" )
| $p
The result for the given input JSON:
[
"nodes",
"60e327ee58a0",
"nodeinfo",
"hostname"
]
If you wanted the path to the containing object, then replace the final $p with $p[0:-1]; and if you want the containing object itself: getpath($p[0:-1])
Here is a solution which searches for nodes where the specified $needle is present in any of the addresses, mac or hostname fields.
"<ipv6 removed>" as $needle # set to whatever you like
| foreach (.nodes|keys[]) as $k (
.
; .
; ( .nodes[$k].nodeinfo.network.addresses?
+ [ .nodes[$k].nodeinfo.network.mac?
, .nodes[$k].nodeinfo.hostname?
]
) as $haystack
| if $haystack | index($needle)
then {($k): .nodes[$k]}
else empty
end
)
EDIT: I now realize a filter of the form foreach E as $X (.; .; R) can almost always be rewritten as E as $X | R so the above is really just
"<ipv6 removed>" as $needle
| (.nodes|keys[]) as $k
| ( .nodes[$k].nodeinfo.network.addresses?
+ [ .nodes[$k].nodeinfo.network.mac?
, .nodes[$k].nodeinfo.hostname?
]
) as $haystack
| if $haystack | index($needle)
then {($k): .nodes[$k]}
else empty
end