Extracting values from nested arrays - json

I'm trying to extract values from nested arrays in JSON below and output as CSV.
Fields to extract:
templates.name
items.name
triggers.name
Output as:
templates.name; items.name; triggers.name
Anticipated output something like:
"Template App Agent"; "Host name of zabbix_agentd running"; "Host name of zabbix_agentd was changed on {HOST.NAME}"
"Template App Agent"; "Agent ping"; "Zabbix agent on {HOST.NAME} is unreachable for 5 minutes"
"Template App Agent"; "Version of zabbix_agent(d) running"; ""
Note:
Not every item has a trigger.
Several triggers may exist for an item.
I'm new to JQ. So far only success is extracting the template name.
jq '.[] | {templates: [.templates[].name]}'
Data:
{
"zabbix_export": {
"version": "5.4",
"date": "2022-05-17T06:25:59Z",
"groups": [
{
"uuid": "7df96b18c230490a9a0a9e2307226338",
"name": "Templates"
}
],
"templates": [
{
"uuid": "e60e6598cf19448089a5f5a6c5d796a2",
"template": "Template App Agent",
"name": "Template App Agent",
"groups": [
{
"name": "Templates"
}
],
"items": [
{
"uuid": "24c03ed734d54dc8868a282a83a02200",
"name": "Host name of zabbix_agentd running",
"key": "agent.hostname",
"delay": "1h",
"history": "1w",
"trends": "0",
"value_type": "CHAR",
"request_method": "POST",
"tags": [
{
"tag": "Application",
"value": "Zabbix agent"
}
],
"triggers": [
{
"uuid": "d2d12d9e7dfe4fedb252f19b85e5e6aa",
"expression": "(last(/Template App Agent/agent.hostname,#1)<>last(/Template App Agent/agent.hostname,#2))>0",
"name": "Host name of zabbix_agentd was changed on {HOST.NAME}",
"priority": "INFO"
}
]
},
{
"uuid": "abacad4ca5eb46d29864d8a4998f1cbb",
"name": "Agent ping",
"key": "agent.ping",
"history": "1w",
"description": "The agent always returns 1 for this item. It could be used in combination with nodata() for availability check.",
"valuemap": {
"name": "Zabbix agent ping status"
},
"request_method": "POST",
"tags": [
{
"tag": "Application",
"value": "Zabbix agent"
}
],
"triggers": [
{
"uuid": "6d2a73199f3b4288bf36331a142c1725",
"expression": "nodata(/Template App Agent/agent.ping,5m)=1",
"name": "Zabbix agent on {HOST.NAME} is unreachable for 5 minutes",
"priority": "AVERAGE"
}
]
},
{
"uuid": "2cc337555efd43d181c28c792f8cbbdb",
"name": "Version of zabbix_agent(d) running",
"key": "agent.version",
"delay": "1h",
"history": "1w",
"trends": "0",
"value_type": "CHAR",
"request_method": "POST",
"tags": [
{
"tag": "Application",
"value": "Zabbix agent"
}
]
}
],
"valuemaps": [
{
"uuid": "3d66c59a28c04b0ca8227c87902ddb4d",
"name": "Zabbix agent ping status",
"mappings": [
{
"value": "1",
"newvalue": "Up"
}
]
}
]
}
]
}
}

.zabbix_export.templates[] | .name as $tn | .items[] | [ $tn, .name, .triggers[]?.name? ] | join("; ")
Loop over the templates
.zabbix_export.templates[]
Save the template name in a var
.name as $tn
Loop over the items
.items[]
Create an array with fields you like (including the name from step 1
[ $tn, .name, .triggers[]?.name? ]
Join the array to a string
join("; ")
Will output:
"Template App Agent; Host name of zabbix_agentd running; Host name of zabbix_agentd was changed on {HOST.NAME}"
"Template App Agent; Agent ping; Zabbix agent on {HOST.NAME} is unreachable for 5 minutes"
"Template App Agent; Version of zabbix_agent(d) running"
Online demo

This is a nested structure, you need to iterate level by level and add up the items you want to be in one output line. Store values from previous levels in variables.
To account for an inexistent .triggers array, you may use the Error Suppression Operator ? in combination with Alternative Operator //.
Finally, wrap the items in quotes (here using map), join them using join, and output them as raw text using the -r option
jq -r '
.[].templates[] | .name as $t
| .items[] | .name as $i
| [$t, $i, (.triggers[].name)? // ""]
| map("\"\(.)\"") | join("; ")
'
"Template App Agent"; "Host name of zabbix_agentd running"; "Host name of zabbix_agentd was changed on {HOST.NAME}"
"Template App Agent"; "Agent ping"; "Zabbix agent on {HOST.NAME} is unreachable for 5 minutes"
"Template App Agent"; "Version of zabbix_agent(d) running"; ""
Demo
Also consider using the #csv builtin, which gives you valid CSV right away (properly encoded (not just quoted) items, but separated with commas, not semicolons):
jq -r '
.[].templates[] | .name as $t
| .items[] | .name as $i
| [$t, $i, (.triggers[].name)? // ""]
| #csv
'
"Template App Agent","Host name of zabbix_agentd running","Host name of zabbix_agentd was changed on {HOST.NAME}"
"Template App Agent","Agent ping","Zabbix agent on {HOST.NAME} is unreachable for 5 minutes"
"Template App Agent","Version of zabbix_agent(d) running",""
Demo

Related

Converting json into csv with jq while only capturing specific keys

This is my first post so apologies if I make mistakes. Consider I have the following json output
{
"records": [
{
"title": "root logon",
"login": "61819009",
"uid": "ajsd879asdjksasda123asd1asd1",
"password": "OizfD19jC$ySaV$MKpSF",
"login_url": "http://192.168.0.1/",
"notes": ""
},
{
"title": "important admin account",
"login": "admin",
"uid": "asdjhkasdh89eoajdiuas98ue9aoi",
"password": "0z5gDUC#Rb354TlLq$KJ",
"login_url": "",
"notes": "",
"folders": [
{
"shared_folder": "Department",
"folder": "Important",
"option1": false,
"option2": false
}
]
}
{
"title": "another important admin",
"login": "admin#domain.com",
"uid": "asjhe98asiajsijeouiaueiaiu",
"password": "3pUs#uXEqsxCv7PRkDlJ",
"login_url": "http://192.168.0.2/",
"notes": ""
},
{
"title": "switch admin",
"login": "admin",
"uid": "asjhe89ausiodjakljskea90ik",
"password": "hMB!eMsAE8q4aDQuM4LY",
"login_url": "",
"notes": "",
"folders": [
{
"shared_folder": "Department2",
"folder": "network\\switches",
"option1": false,
"option2": false
}
]
}
]
}
from the list above, I would like to only export title, login, uid, shared folder, and folder fields to csv file. When I run the following command
jq --raw-output '.records[] | [.title, .login, .uid, .shared_folder, .folder] | # tsv > file
what I am expecting as a CSV output is
title
login
uid
shared_folder
folder
root logon
61819009
ajsd879asdjksasda123asd1asd1
important admin account
admin
asdjhkasdh89eoajdiuas98ue9aoi
Department
Important
another important admin
admin#domain.com
asjhe98asiajsijeouiaueiaiu
switch admin
admin
asjhe89ausiodjakljskea90ik
Department2
network\switches
instead I'm getting the following CSV output:
title
login
uid
root logon
61819009
ajsd879asdjksasda123asd1asd1
important admin account
admin
asdjhkasdh89eoajdiuas98ue9aoi
another important admin
admin#domain.com
asjhe98asiajsijeouiaueiaiu
switch admin
admin
asjhe89ausiodjakljskea90ik
my goal is to eliminate capturing following fields
password
login_url
notes
option1
option2
You an error in your JSON, you must add a ,.
Once done, the command is this
jq --raw-output '.records[] | [.title, .login, .uid, .folders[0].shared_folder,.folders[0].folder] |#tsv'
Here's one way:
.records[]
| [.title, .login, .uid] + ((.folders[]? // null) | [.shared_folder, .folder])
| #tsv

How to select multiple parameters from JSON output, which meets a condition & further select individual value

I have a json output, from which I need to get id value and IPv4_address value where IPv4_address exists (this shouldn't be null). Have to use this ID value for another request along with random generated string.
Here is the breakdown of the requirement :
STEP 1 :
In the following example, for the ipv4_address:1.1.1.1 & ipv4_address:1.1.1.2, i need to get the id output which is "4e-0365-4e29-95ca-329165eecf8a" and "c9061b6674a8546cea" along with IP address.
Example of my output should look like (something similar):
1.1.1.1 4e-0365-4e29-95ca-329165eecf8a
1.1.1.2 c9061b6674a8546cea
I was trying to use jq but with this I'm not able to get the both values :
ID="$(echo "$test" \n | jq -r '.USER[] | select(.ipv4_address) | .ipv4_address')"
ID1="$(echo "$test" \n | jq -r '.USER[] | select(.ipv4_address) | .id')"
Sample output which is getting displayed with the above 2 commands :
ID value is : 1.1.1.1 1.1.1.2
ID1 value is : 4e-0365-4e29-95ca-329165eecf8a c9061b6674a8546cea
STEP 2: Profile creation: I need to use each $ID1 value in another request along with random generated string. Random string is generated as per the count of $ID1's (so here I will generate 2 random string)
And thus 2 profiles are created.
Ques: How can I get each ID from the $ID1 variable ? I tried something like ID1[0] but that seems to be wrong
STEP 3 :
Will use each ID and random string for another request, Once its done or if that step is failed, i need to provide the output to a file & output should look like :
My requirement for the final output is :
1.1.1.1 4e-0365-4e29-95ca-329165eecf8a <randomvalue-1> <profile-1> DONE
1.1.1.2 c9061b6674a8546cea <randomvalue-2> <profile-2> FAILED
where random value will be generated randomly and shall be used against the ID.
JSON output which needs to be parsed:
{
"errorcode": 0,
"message": "Done",
"operation": "get",
"resourceType": "USER",
"username": "root",
"tenant_name": "Owner",
"tenant_id": "05db6674ad458546cd2",
"resourceName": "",
"USER": [
{
"is_default": "false",
"session_timeout": "0",
"permission": "root",
"name": "ee",
"session_timeout_unit": "",
"tenant_id": "55bcb6674ad45854",
"id": "4e-0365-4e29-95ca-329165eecf8a",
"ipv4_address": "1.1.1.1",
"state": "Up",
"tenant_name": "Owner",
"encrypted": "false",
"groups": [
"owner"
],
"root_user": ""
},
{
"is_default": "false",
"session_timeout": "0",
"permission": "read",
"name": "test",
"session_timeout_unit": "",
"tenant_id": "bc906674ad458546cd2",
"id": "12cd0-fb7f-4abf-b060-48e98b794b06",
"tenant_name": "Owner",
"encrypted": "false",
"groups": [
"read"
],
"root_user": ""
},
{
"is_default": "true",
"session_timeout": "0",
"permission": "root",
"name": "root",
"session_timeout_unit": "",
"tenant_id": "c905db6d458546cd2",
"id": "c9061b6674a8546cea",
"ipv4_address": "1.1.1.2",
"state": "Not Reachable",
"tenant_name": "Owner",
"encrypted": "false",
"groups": [
"owner"
],
"root_user": ""
},
{
"is_default": "false",
"session_timeout": "0",
"permission": "readonly",
"name": "a",
"session_timeout_unit": "",
"tenant_id": "c905674ad458546cd2",
"id": "bc8a-4fd6-bc09-8c39c131b54e",
"tenant_name": "Owner",
"encrypted": "false",
"groups": [
"read"
],
"root_user": ""
}
]
}
Not quite clear with the logic of marking it DONE and FAILED. But to answer your first question where you want to select the multiple fields, you can do something like this:
$ cat input.js | jq -r '.USER[] | select(.ipv4_address) | "\(.ipv4_address) \(.id)"' > result.js
This will output the result in a file named result.js. You can apply your custom logic of marking DONE and Failed on this file.
In the above command when you do select(.ipv4_address) It basically drops all the records for which ipv4_address value is null or it is not present.
if you want to select the records which have ipv4_address as null, then your select statement would become something like this
select(.ipv4_address == null)

ConvertFrom-JSON won't accept convertto-json with children when working with WebServiceProxy

I am pulling data from an API using the New-WebServiceProxy in PowerShell 4.0 and then piping it out to a JSON file for review and import on another API service (same API version, etc, just a different host).
$tasklist.Taskconfig | ConvertTo-JSON-Depth 50 -As String | Out-File -FilePath $exportpath\$name.xml -Force
Gives me my XML containing the TaskConfig. In this case, TaskConfig is an object type automatically generated by the API I'm interfacing with. When I want to import the content I am using:
$taskconfig = (Get-Content "$taskjson") -join "`n" | ConvertFrom-Json
but when I run this it's unable to create the object. I assume this is because the JSON contains nested children, giving the error-
Cannot convert value "#{Name=plugindive; Value=;> Children=System.Object[]}" to type "Microsoft.PowerShell.Commands.NewWebserviceProxy.AutogeneratedTypes.WebServiceProxy1rcleWeb_WebClientAPI_asmx_wsdl.TaskConfig". Error: "Cannot convert the "#{Name=plugindive; Value=;Children=System.Object[]}" value of type "System.Management.Automation.PSCustomObject" to type "Microsoft.PowerShell.Commands.NewWebserviceProxy.AutogeneratedTypes.WebServiceProxy1rcleWeb_WebClientAPI_asmx_wsdl.TaskConfig"."
I've tried explictly stating the type of object:
$taskconfig = [Microsoft.PowerShell.Commands.NewWebserviceProxy.AutogeneratedTypes.WebServiceProxy1rcleWeb_WebClientAPI_asmx_wsdl.TaskConfig](Get-Content "$taskjson" | Out-string | ConvertFrom-Json)
as well as creating the object then trying to add the children from my JSON -
$taskconfig.children = $json.children
But these all fail in the same way.
I don't seem to get this same issue in PowerShell 5.0 interestingly enough, but I can't verify why - is there another way to approach this?
Added example JSON below
{"Name": "plugindive",
"Value": null,
"Children": [{
"Name": "auto",
"Value": "False",
"Children": [
]
},
{
"Name": "categories",
"Value": null,
"Children": [{
"Name": "Module Z",
"Value": "False",
"Children": [
]
},
{
"Name": "Module A",
"Value": "False",
"Children": [
]
},
{
"Name": "Module B",
"Value": "False",
"Children": [
]
},
{
"Name": "Module C",
"Value": "False",
"Children": [
]
}
]
}
]
}
It seems as if this doesn't work in PowerShell v3.0, so I simply ended up making posts with the explicit XML directly, rather than converting to JSON.

AWS Data Pipeline - Set Hive site values during EMR Creation

We are upgrading our Data pipeline version from 3.3.2 to 5.8, so those bootstrap actions on old AMI release have changed to be setup using configuration and specifying them under classification / property definition.
So my Json looks like below
{
"enableDebugging": "true",
"taskInstanceBidPrice": "1",
"terminateAfter": "2 Hours",
"name": "ExportCluster",
"taskInstanceType": "m1.xlarge",
"schedule": {
"ref": "Default"
},
"emrLogUri": "s3://emr-script-logs/",
"coreInstanceType": "m1.xlarge",
"coreInstanceCount": "1",
"taskInstanceCount": "4",
"masterInstanceType": "m3.xlarge",
"keyPair": "XXXX",
"applications": ["hadoop","hive", "tez"],
"subnetId": "XXXXX",
"logUri": "s3://pipelinedata/XXX",
"releaseLabel": "emr-5.8.0",
"type": "EmrCluster",
"id": "EmrClusterWithNewEMRVersion",
"configuration": [
{ "ref": "configureEmrHiveSite" }
]
},
{
"myComment": "This object configures hive-site xml.",
"name": "HiveSite Configuration",
"type": "HiveSiteConfiguration",
"id": "configureEmrHiveSite",
"classification": "hive-site",
"property": [
{"ref": "hive-exec-compress-output" }
]
},
{
"myComment": "This object sets a hive-site configuration
property value.",
"name":"hive-exec-compress-output",
"type": "Property",
"id": "hive-exec-compress-output",
"key": "hive.exec.compress.output",
"value": "true"
}
],
"parameters": []
With the above Json file it gets loaded into Data Pipeline but throws an error saying
Object:HiveSite Configuration
ERROR: 'HiveSiteConfiguration'
Object:ExportCluster
ERROR: 'configuration' values must be of type 'null'. Found values of type 'null'
I am not sure what this really means and could you please let me know if i am specifying this correctly which i think i am according to http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html
The below block should have the name as "EMR Configuration" only then its recognized correctly by the AWS Data pipeline and the Hive-site.xml is being set accordingly.
{
"myComment": "This object configures hive-site xml.",
"name": "EMR Configuration",
"type": "EmrConfiguration",
"id": "configureEmrHiveSite",
"classification": "hive-site",
"property": [
{"ref": "hive-exec-compress-output" }
]
},

Delete or parse data in JSON file with JQ or any Linux Tool

I have this JSON file:
{
"vulnerable_configuration_cpe_2_2": [
"cpe:/o:apple:apple_tv:9.1",
"cpe:/o:apple:watchos:2.1",
"cpe:/o:apple:iphone_os:9.2",
"cpe:/o:apple:mac_os_x:10.11.2"
],
"vulnerable_configuration": [
{
"title": "cpe:2.3:o:apple:apple_tv:9.1",
"id": "cpe:2.3:o:apple:apple_tv:9.1"
},
{
"title": "cpe:2.3:o:apple:watchos:2.1",
"id": "cpe:2.3:o:apple:watchos:2.1"
},
{
"title": "cpe:2.3:o:apple:iphone_os:9.2",
"id": "cpe:2.3:o:apple:iphone_os:9.2"
},
{
"title": "cpe:2.3:o:apple:mac_os_x:10.11.2",
"id": "cpe:2.3:o:apple:mac_os_x:10.11.2"
}
],
"summary": "The Disk Images component in Apple iOS before 9.2.1, OS X before 10.11.3, and tvOS before 9.1.1 allows local users to gain privileges or cause a denial of service (memory corruption) via unspecified vectors.",
"references": [
"https://support.apple.com/HT206168",
"https://support.apple.com/HT205732",
"https://support.apple.com/HT205731",
"https://support.apple.com/HT205729",
"http://lists.apple.com/archives/security-announce/2016/Mar/msg00001.html",
"http://lists.apple.com/archives/security-announce/2016/Jan/msg00005.html",
"http://lists.apple.com/archives/security-announce/2016/Jan/msg00003.html",
"http://lists.apple.com/archives/security-announce/2016/Jan/msg00002.html"
],
"Modified": "2016-03-29T19:01:26.810-04:00",
"Published": "2016-02-01T06:59:01.090-05:00",
"access": {
"vector": "LOCAL",
"complexity": "LOW",
"authentication": "NONE"
},
"cvss": 7.2,
"cvss-time": "2016-03-29T09:43:55.537-04:00",
"cwe": "CWE-119",
"id": "CVE-2016-1717",
"impact": {
"integrity": "COMPLETE",
"confidentiality": "COMPLETE",
"availability": "COMPLETE"
}
I want to delete the following data using JQ or any Linux Tool
access (including vector, complexity, authentication)
references: only first or 1 reference
cvss-time
vulnerable_configuration_cpe_2_2 (Incluide cpe:/o:apple:apple_tv:9.1,
etc)
The result, I want it to be this:
{
"vulnerable_configuration": [
{
"title": "cpe:2.3:o:apple:apple_tv:9.1",
"id": "cpe:2.3:o:apple:apple_tv:9.1"
},
{
"title": "cpe:2.3:o:apple:watchos:2.1",
"id": "cpe:2.3:o:apple:watchos:2.1"
},
{
"title": "cpe:2.3:o:apple:iphone_os:9.2",
"id": "cpe:2.3:o:apple:iphone_os:9.2"
},
{
"title": "cpe:2.3:o:apple:mac_os_x:10.11.2",
"id": "cpe:2.3:o:apple:mac_os_x:10.11.2"
}
],
"summary": "The Disk Images component in Apple iOS before 9.2.1, OS X before 10.11.3, and tvOS before 9.1.1 allows local users to gain privileges or cause a denial of service (memory corruption) via unspecified vectors.",
"references": [
"https://support.apple.com/HT206168",
],
"Modified": "2016-03-29T19:01:26.810-04:00",
"Published": "2016-02-01T06:59:01.090-05:00",
"cvss": 7.2,
"cwe": "CWE-119",
"id": "CVE-2016-1717",
"impact": {
"integrity": "COMPLETE",
"confidentiality": "COMPLETE",
"availability": "COMPLETE"
}
I try
cat file.json | jq ('del(.cvss-time)' and 'access')
but no work
What is the command JQ I have to use? Or what tool can I use?
Tanks!
The following filter for jq >= 1.5 meets your requirements:
del(.access)
| .references |= [.[0]]
| del(."cvss-time")
| del(.vulnerable_configuration_cpe_2_2)
If you are using jq 1.4, use del(.["cvss-time"]) instead of del(."cvss-time").