Loop through an array ignores \n - json

#edit
Below is the JSON getting saved as $content
[
{
"date_created": 1234,
"fingerprint_hash": null,
"address": "xx:xx:xx:xx:xx:xx",
"name": null,
"manufacturer": "xxx",
"date_updated": 1234,
"active": true,
"date_last_active": 1234,
"mac_group_info": {
"name": null,
"id": 0,
"remarks": null
},
"id": 1234,
"remarks": null,
"arp_mapping_info": [
{
"ip_info": {
"date_changed": 1234,
"date_last_kerberos_login": null,
"dns_name": "xxxx",
"id": 6,
"remarks": null,
"date_created": 1234,
"kerberos_user_name": null,
"ip_group_info": {
"remarks": null,
"id": 0,
"name": null
},
"address": "x.x.x.x"
},
"interface_info": [
{
"link": "on",
"name": "x",
"discovery_info": null,
"speed": 1234,
"host_info": {
"backup_sensor_id": 0,
"type": "xxxx",
"radius_coa_flags": null,
"moment": 1,
"name": null,
"host_group_info": {
"name": null,
"id": 0,
"remarks": null
},
"interface_mib": "D",
"uplink_count": 1,
"sensor_info": {
"id": 1,
"os_version": "xxxx",
"architecture": "xxxx",
"remarks": null,
"last_contact": "xxxx",
"queue": 0,
"address": "x.x.xx",
"status": "active",
"os_name": "linux",
"name": "xxxx",
"version": "x.x.x-x"
},
"mode": "xx",
"address": "x.x.x.x",
"radius_secret": null,
"arp_mib": "2",
"cam_count": 1,
"discovery2_mib": "C",
"discovery1_mib": "3",
"radius_coa_port": null,
"radius_requests_count": 0,
"vlan_count": 1,
"access_to_port": 1,
"vlan_mib": "C",
"interval": 1,
"snmp_traps_community": null,
"engine_mode": "dynamic",
"manufacturer": "ciscoSystems",
"engine_id": null,
"snmp_traps_version": "1",
"mib_options": null,
"remarks": null,
"snmp_write_version": "2c",
"snmp_write_community": "private",
"access_to_cam": 1,
"status": "ok",
"access_to_cto": 1,
"access_to_interface": 1,
"snmp_read_version": "1",
"snmp_read_community": "public",
"id": 1,
"cam_mib": "C",
"arp_count": 1,
"access_to_arp": 1,
"port_count": 1,
"main_sensor_id": 1
},
"status": "on",
"index": 1,
"protocols": [
"ARP",
"NDP"
],
"id": 1,
"remarks": null,
"sniffer_mode": null
}
],
"host_id": 1
}
],
"cam_mapping_info": []
}
]
Thats my code:
foreach ($content){
$content =~ s/\[/\[\n\t/g;
$content =~ s/{/{\n\t/g;
$content =~ s/\]/\n\]/g;
$content =~ s/}/}\n/g;
$content =~ s/,/\n\t/g;
print $content;
}
Todo: get the same output as the JSON above but with a perl skript runing it withouth any modules, normaly we use postman for it. As I asked my "couch" for help he only said: just learn to programm. Yeah...
The output I get with my regex is almost the solution. It just stops tabing after the first tab, doent repeat it on another opening bracket.

$line eq "{" || "[" is interpreted as
(($line eq '{') or '[')
(as Deparse would show you). Use
$line eq '{' || $line eq '['
Moreover, $data is a scalar variable, i.e. it contains a single value. Iterating over it makes just a single step.
Normally, you'd iterate over an array:
foreach my $line (#lines)
or maybe an array reference:
foreach my $line (#$data)
or whatever else
foreach my $line ($line1, $line2, #rest_of_lines)
# or
foreach my $line (split /\n/, $message)

Using the YAML module, you can get a quick human readable format. I'm sure there are other and/or better ways to format your text, but this is one example. The benefit here is that you can transform a JSON data structure (using JSON::PP, a core Perl module that comes pre-installed) into a Perl data structure and then into a YAML structure that you can print easily. I did this with a few lines of code and a few modules, using some sample JSON data.
use strict;
use warnings;
use JSON::PP;
use YAML;
my $text = do { local $/; <DATA> };
my $json = decode_json($text);
print Dump $json;
__DATA__
{
"ITEM":[
{
"-itemID": "1000000" ,
"-itemName": "DisneyJuniorLA" ,
"-thumbUrl": "" ,
"-packageID": "1" ,
"-itemPrice": "0" ,
"-isLock": "true"
},
{
"-itemID": "1000001" ,
"-itemName": "31 minutos" ,
"-thumbUrl": "" ,
"-packageID": "1" ,
"-itemPrice": "0" ,
"-isLock": "true"
},
{
"-itemID": "1000002" ,
"-itemName": "Plaza SÚsamo" ,
"-thumbUrl": "" ,
"-packageID": "1" ,
"-itemPrice": "0" ,
"-isLock": "true"
}
]
}
Output:
---
ITEM:
- -isLock: true
-itemID: 1000000
-itemName: DisneyJuniorLA
-itemPrice: 0
-packageID: 1
-thumbUrl: ''
- -isLock: true
-itemID: 1000001
-itemName: 31 minutos
-itemPrice: 0
-packageID: 1
-thumbUrl: ''
- -isLock: true
-itemID: 1000002
-itemName: Plaza S┌samo
-itemPrice: 0
-packageID: 1
-thumbUrl: ''
EDIT: With the new json data, I get this output:
- active: !!perl/scalar:JSON::PP::Boolean 1
address: xx:xx:xx:xx:xx:xx
arp_mapping_info:
- host_id: 1
interface_info:
- discovery_info: ~
host_info:
access_to_arp: 1
access_to_cam: 1
access_to_cto: 1
access_to_interface: 1
access_to_port: 1
address: x.x.x.x
arp_count: 1
arp_mib: 2
backup_sensor_id: 0
cam_count: 1
cam_mib: C
discovery1_mib: 3
discovery2_mib: C
engine_id: ~
engine_mode: dynamic
host_group_info:
id: 0
name: ~
remarks: ~
id: 1
interface_mib: D
interval: 1
main_sensor_id: 1
manufacturer: ciscoSystems
mib_options: ~
mode: xx
moment: 1
name: ~
port_count: 1
radius_coa_flags: ~
radius_coa_port: ~
radius_requests_count: 0
radius_secret: ~
remarks: ~
sensor_info:
address: x.x.xx
architecture: xxxx
id: 1
last_contact: xxxx
name: xxxx
os_name: linux
os_version: xxxx
queue: 0
remarks: ~
status: active
version: x.x.x-x
snmp_read_community: public
snmp_read_version: 1
snmp_traps_community: ~
snmp_traps_version: 1
snmp_write_community: private
snmp_write_version: 2c
status: ok
type: xxxx
uplink_count: 1
vlan_count: 1
vlan_mib: C
id: 1
index: 1
link: on
name: x
protocols:
- ARP
- NDP
remarks: ~
sniffer_mode: ~
speed: 1234
status: on
ip_info:
address: x.x.x.x
date_changed: 1234
date_created: 1234
date_last_kerberos_login: ~
dns_name: xxxx
id: 6
ip_group_info:
id: 0
name: ~
remarks: ~
kerberos_user_name: ~
remarks: ~
cam_mapping_info: []
date_created: 1234
date_last_active: 1234
date_updated: 1234
fingerprint_hash: ~
id: 1234
mac_group_info:
id: 0
name: ~
remarks: ~
manufacturer: xxx
name: ~
remarks: ~

Asked a coworker for help and he has just writen it in 5 minutes without big thinking...
use strict;
use warnings;
my $content = #JSON#
my #arr = split //, $content;
my $length = scalar #arr;
my $tabcounter = 0;
for (my $i=0; $i < $length; $i++) {
print $arr[$i];
if ( $arr[$i] eq '[' || $arr[$i] eq ']' || $arr[$i] eq '{' || $arr[$i] eq '}' || $arr[$i] eq ',') {
print "\n";
if ( $arr[$i] eq '[' || $arr[$i] eq '{' ) {
$tabcounter++;
}
if ( $arr[$i] eq '}' || $arr[$i] eq ']' ) {
$tabcounter--;
}
for (my $j=0; $j < $tabcounter; $j++) {
print "\t";
}
}
}
So if anyone has to read and print out of a JSON file without any modules, here is the solution.

Related

Adding a new JSON key from an existing JSON value

I have the following JSON;
[
{
"id": 1,
"start": "2022-06-20",
"result": 24
},
{
"id": 2,
"start": "2022-06-21",
"result": 56
},
{
"id": 3,
"start": "2022-06-21",
"result": 78
}
]
I'm wanting to add 2 new values to each array above using JQ, dimension and date, but date needs to be a copy of the existing key value start. The expected output is as below;
[
{
"id": 1,
"start": "2022-06-20",
"result": 24,
"date": "2022-06-20",
"dimension": "new"
},
{
"id": 2,
"start": "2022-06-21",
"result": 56,
"date": "2022-06-21",
"dimension": "new"
},
{
"id": 3,
"start": "2022-06-21",
"result": 78,
"date": "2022-06-21",
"dimension": "new"
}
]
The jq I have at present can add the new key dimension, but I can't figure out how to copy start -> date
jq '.[] += {"dimension": "new"}' input.json
Thanks for any help
Just create the new key / value pairs.
jq 'map(.date = .start | .dimension = "new")' input.json
You might have tried the following:
.[] += { date: .start, dimension: "new" } // WRONG
But it's not quite right since . is the array, not the element of the array. You can use |= as a topicalizer.
.[] |= ( . += { date: .start, dimension: "new" } )
But I'd use map instead.
map( . += { date: .start, dimension: "new" } )
Alternatively,
. += { date: .start, dimension: "new" }
can also also be achieved using
.date = .start | .dimension: "new"
So you could use
.[] |= ( .date = .start | .dimension: "new" )
or
map( .date = .start | .dimension = "new" )

Use JQ to create new object where the key comes from one object and the value comes from another

I have the following input:
{
"Columns": [
{
"email": 123,
"name": 456,
"firstName": 789,
"lastName": 450,
"admin": 900,
"licensedSheetCreator": 617,
"groupAdmin": 354,
"resourceViewer": 804,
"id": 730,
"status": 523,
"sheetCount": 298
}
]
}
{
"Users": [
{
"email": "abc#def.com",
"name": "Abc Def",
"firstName": "Abc",
"lastName": "Def",
"admin": false,
"licensedSheetCreator": true,
"groupAdmin": false,
"resourceViewer": true,
"id": 521,
"status": "ACTIVE",
"sheetCount": 0
},
{
"email": "aaa#bbb.com",
"name": "Aaa Bob",
"firstName": "Aaa",
"lastName": "Bob",
"admin": false,
"licensedSheetCreator": true,
"groupAdmin": false,
"resourceViewer": false,
"id": 352,
"status": "ACTIVE",
"sheetCount": 0
}
]
}
I need to change the key for all key value pairs in users to match the value in Columns, like so:
{
"Columns": [
{
"email": 123,
"name": 456,
"firstName": 789,
"lastName": 450,
"admin": 900,
"licensedSheetCreator": 617,
"groupAdmin": 354,
"resourceViewer": 804,
"id": 730,
"status": 523,
"sheetCount": 298
}
]
}
{
"Users": [
{
123: "abc#def.com",
456: "Abc Def",
789: "Abc",
450: "Def",
900: false,
617: true,
354: false,
804: true,
730: 521,
523: "ACTIVE",
298: 0
},
{
123: "aaa#bbb.com",
456: "Aaa Bob",
789: "Aaa",
450: "Bob",
900: false,
617: true,
354: false,
804: false,
730: 352,
523: "ACTIVE",
298: 0
}
]
}
I don't mind if I update the Users array or create a new array of objects.
I have tried several combinations of with entries, to entries, from entries, trying to search for keys using variables but the more I dive into it, the more confused I get.
Elements of a stream are processed independently. So we have to change the input.
We could group the stream elements into an array. For an input stream, this can be achieved using --slurp/-s.[1]
jq -s '
( .[0].Columns[0] | map_values( tostring ) ) as $map |
(
.[0],
(
.[1:][] |
.Users[] |= with_entries(
.key = $map[ .key ]
)
)
)
'
Demo on jqplay
Alternatively, we could use --null-input/-n in conjunction with input and/or inputs to read the input.
jq -n '
input |
( .Columns[0] | map_values( tostring ) ) as $map |
(
.,
(
inputs |
.Users[] |= with_entries(
.key = $map[ .key ]
)
)
)
'
Demo on jqplay
Note that your desired output isn't valid JSON. Object keys must be strings. So the above produces a slightly different document than requested.
Note that I assumed that .Columns is always an array of one exactly one element. This is a nonsense assumption, but it's the only way the question makes sense.
For a stream the code generates, you could place the stream generator in an array constructor ([]). reduce can also be used to collect from a stream. For example, map( ... ) can be written as [ .[] | ... ] and as reduce .[] as $_ ( []; . + [ $_ | ... ] ).
The following has the merit of simplicity, though it does not sort the keys.
It assumes jq is invoked with the -n option and of course produces a stream of valid JSON objects:
input
| . as $Columns
| .Columns[0] as $dict
| input # Users
| .Users[] |= with_entries(.key |= ($dict[.]|tostring))
| $Columns, .
If having the keys sorted is important, then you could easily add suitable code to do that; alternatively, if you don't mind having the keys of all objects sorted, you could use the -S command-line option.

group json output by array input value

Input data: /tmp/h1.tabs, tab delimited:
INPUT A1
Flavor Controller
Comment Disabled
State DiskOne
INPUT B2
Flavor Controller
Comment Not Applicable
State Not Applicable
ConnectorCount 12
Alarm Alarm Not present
INPUT C3
Flavor Controller
Comment Not Applicable
Media Not Applicable
ConnectorCount 0
State Alarm Not present
Desired Output:
{
"A1": {
"Comment": "Disabled",
"Flavor": "Controller",
"State": "DiskOne"
},
"B2": {
"Alarm": "Alarm Not present",
"Comment": "Not Applicable",
"ConnectorCount": "12",
"Flavor": "Controller",
"State": "Not Applicable"
},
"C3": {
"Comment": "Not Applicable",
"ConnectorCount": "0",
"Flavor": "Controller",
"Media": "Not Applicable",
"State": "Alarm Not present"
}
}
Each INPUT dictionary key could also be an array instead of another dictionary.
{
"A1": [
{ "Comment": "Disabled" },
{ "Flavor": "Controller" },
{ "State": "DiskOne" }
],
About as close as I am able to get is something like this:
jq -Rsn '[inputs|. / "\n"|.[] / "\t"|select(length > 0)|. as $input|(if $input[0] == "INPUT" then $input[1] else { ($input[0]): $input[1] } end)]' /tmp/h1.tabs
[
"A1",
{
"Flavor": "Controller"
},
{
"Comment": "Disabled"
},
{
"State": "DiskOne"
},
"B2",
{
"Flavor": "Controller"
},
I've tried expressions like if $input[0] == "INPUT" then $block = $input[1], but I am not having any luck with an assignment, so I can't use the assignment in the output. Really what I think I need is a variable set to whatever INPUTs value is every time I pass it. Then I can format the output as needed. I am just missing some key magic. I've been banging on this for a while, here is more that doesn't work...
# vim:ft=ansible:tabstop=8 expandtab shiftwidth=2 softtabstop=2
"unknown" as $block
|[
inputs
|. / "\n"
|
(
.[]
| select(length > 0)
|.
)
]
|(.[] / "\t")
|select(length > 0)
|. as $input
|
(
if $input[0] == "INPUT" then $block = $input[1] else empty end
|({($block): [($input[0]):($input[1])]})
) | add
Still learning :-)
reduce is your friend..
reduce (inputs / "\t") as [$k, $v] ([];
if $k == "INPUT" then
.[0] = $v
else
.[1][.[0]] += {($k): $v}
end
) | .[1]
Note that you need to specify -n and -R options on the command line for this to work
Using GNU awk, gawkextlib and gawk-json:
$ gawk '
#load "json"
BEGIN{
FS="\t"
}
{
if($1=="INPUT")
tl=$2
else
data[tl][$1]=$2
}
END {
print json_toJSON(data)
}' file # | jq '.' # for eye-friendly formating
Output (jq assisted):
{
"C3": {
"Comment": "Not Applicable",
"State": "Alarm Not present",
"ConnectorCount": "0",
"Flavor": "Controller",
"Media": "Not Applicable"
},
"A1": {
"Comment": "Disabled",
"State": "DiskOne",
"Flavor": "Controller"
},
"B2": {
"Comment": "Not Applicable",
"State": "Not Applicable",
"Alarm": "Alarm Not present",
"ConnectorCount": "12",
"Flavor": "Controller"
}
}

How to check for null or empty in jq and substitute for empty string in jq transformation

How to check for null or empty in jq and substitute for empty string in jq transformation.
Example in below JSON, this is the JQ
JQ:
.amazon.items[] | select(.name | contains ("shoes")) as $item |
{
activeItem: .amazon.activeitem,
item : {
id : $item.id,
state : $item.state,
status : if [[ $item.status = "" or $item.status = null ]];
then 'IN PROCESS' ; else $item.status end
}
}
JSON:
{
"amazon": {
"activeitem": 2,
"items": [
{
"id": 1,
"name": "harry potter",
"state": "sold"
},
{
"id": 2,
"name": "adidas shoes",
"state": "in inventory"
},
{
"id": 3,
"name": "watch",
"state": "returned"
},{
"id": 4,
"name": "Nike shoes",
"state": "in inventory"
}
]
}
}
I want to add a default string "In Process" if the status is empty or Null.
Based on Item condition, using the query below and take the first object from the filtered results.
code
.amazon.items[] | select(.name | contains ("shoes"))
code
Expected Output:
{
"activeitem": 2,
"item": {
"id": 2,
"name": "adidas shoes",
"state": "in inventory",
"status": "IN PROCESS"
}
}
The key here is to use |=:
.amazon.item.status |=
if . == null or . == ""
then "IN PROCESS"
else .
end

How to search a json with jq for values?

I have a json of this structure:
{
"nodes": {
"60e327ee58a0": {
"nodeinfo": {
"network": {
"mesh": {
"bat0": {
"interfaces": {
"wireless": [
"<mac-address-removed>"
],
"tunnel": [
"<mac-address-removed>"
]
}
}
},
"mac": "<mac removed>",
"addresses": [
"<ipv6 removed>",
"<ipv6 removed>"
]
},
"hardware": {
"model": "TP-Link TL-WR841N/ND v10",
"nproc": 1
},
"software": {
"batman-adv": {
"compat": 15,
"version": "2015.1"
},
"autoupdater": {
"branch": "stable",
"enabled": true
},
"firmware": {
"release": "v2016.1+1.0.1",
"base": "gluon-v2016.1"
},
"status-page": {
"api": 1
},
"fastd": {
"enabled": true,
"version": "v17"
}
},
"hostname": "Antoniusweg12",
"system": {
"site_code": "ffmsd03"
},
"node_id": "60e327ee58a0"
},
"lastseen": "2016-04-14T12:39:04",
"flags": {
"gateway": false,
"online": true
},
"firstseen": "2016-03-16T15:14:04",
"statistics": {
"clients": 1,
"gateway": "de:ad:be:ef:43:02",
"rootfs_usage": 0.6041666666666667,
"loadavg": 0.09,
"uptime": 1822037.41,
"memory_usage": 0.8124737210932025,
"traffic": {
"rx": {
"packets": 50393821,
"bytes": 5061895206
},
"forward": {
"packets": 173,
"bytes": 17417
},
"mgmt_rx": {
"packets": 47453745,
"bytes": 6623785282
},
"tx": {
"packets": 1205695,
"bytes": 173509528,
"dropped": 5683
},
"mgmt_tx": {
"packets": 37906725,
"bytes": 11475209742
}
}
}
},
"30b5c2b042f4": {
<next block...>
And I want to query it with jq for the hostname, the mac or the IPv6.
cat nodes.json |jq -c '.nodes[] | select(.nodes[]| contains("Antoniusweg12"))'
Most examples do not fit this kind of json structure as the objects have an index
Thanks for help in advance.
If you're going to filter, you need to drill down to the property that you want to check for and see if it matches your criteria. You can't expect to just give a name and you'll magically be presented with the results you want.
Searching by hostname, it is found on the .nodeinfo.hostname property of each node:
$ jq -c --arg hostname "Antoniusweg12" \
'.nodes[] | select(.nodeinfo.hostname == $hostname)' nodes.json
Similarly for the mac address, it's found on the .nodeinfo.network.mac property:
$ jq -c --arg mac "aa:bb:cc:dd:ee:ff" \
'.nodes[] | select(.nodeinfo.network.mac == $mac)' nodes.json
For the ip addresses, there's an array of them but it's not that much different in the query. They're found on the .nodeinfo.network.addresses property:
$ jq -c --arg ip "aaaa:bbbb:cccc:dddd::1" \
'.nodes[] | select(.nodeinfo.network.addresses[] == $ip)' nodes.json
Here's another take on the question. Suppose you want to find all occurrences of the key "hostname" for which the value is "Antoniusweg12",
no matter where the key/value combination occurs.
The following will reveal the path to the key/value combination of interest:
paths as $p
| select ( $p[-1] == "hostname" and getpath($p) == "Antoniusweg12" )
| $p
The result for the given input JSON:
[
"nodes",
"60e327ee58a0",
"nodeinfo",
"hostname"
]
If you wanted the path to the containing object, then replace the final $p with $p[0:-1]; and if you want the containing object itself: getpath($p[0:-1])
Here is a solution which searches for nodes where the specified $needle is present in any of the addresses, mac or hostname fields.
"<ipv6 removed>" as $needle # set to whatever you like
| foreach (.nodes|keys[]) as $k (
.
; .
; ( .nodes[$k].nodeinfo.network.addresses?
+ [ .nodes[$k].nodeinfo.network.mac?
, .nodes[$k].nodeinfo.hostname?
]
) as $haystack
| if $haystack | index($needle)
then {($k): .nodes[$k]}
else empty
end
)
EDIT: I now realize a filter of the form foreach E as $X (.; .; R) can almost always be rewritten as E as $X | R so the above is really just
"<ipv6 removed>" as $needle
| (.nodes|keys[]) as $k
| ( .nodes[$k].nodeinfo.network.addresses?
+ [ .nodes[$k].nodeinfo.network.mac?
, .nodes[$k].nodeinfo.hostname?
]
) as $haystack
| if $haystack | index($needle)
then {($k): .nodes[$k]}
else empty
end