I'd like to use JQ to grab only the sub-records that match an if-then statement. When I use
jq 'if .services[].banner == "FQMDAAICCg==" then .services[].port else empty end
it grabs all of the ports for the record. (there are multiple services under each record and I want to restrict my then statement to only the services scope where I actually found the if condition).
How do I just get the port, banner, etc. for the specific service underneath the record which hit my condition?
example:
{
"services": [
{
"tls_detected": false,
"banner_is_raw": true,
"transport_protocol": "tcp",
"banner": "PCFET0NUWVBFIEhU",
"certificate": null,
"timestamp": "2020-03-22T00:38:01.074Z",
"protocol": null,
"port": 4444
},
{
"tls_detected": false,
"banner_is_raw": true,
"transport_protocol": "tcp",
"banner": "SFRUUC8xLjEgMzA",
"certificate": null,
"timestamp": "2020-03-19T01:39:45.288Z",
"protocol": null,
"port": 8080
},
{
"tls_detected": false,
"banner_is_raw": true,
"transport_protocol": "tcp",
"banner": "FQMDAAICCg==",
"certificate": null,
"timestamp": "2020-03-19T01:39:45.288Z",
"protocol": null,
"port": 8085
},
{
"tls_detected": false,
"banner_is_raw": false,
"transport_protocol": "tcp",
"banner": "Q2FjaGUtQ29ud",
"certificate": null,
"timestamp": "2020-03-20T04:25:24Z",
"protocol": "http",
"port": 8080
}
],
"ip": "103.238.62.68",
"autonomous_system": {
"description": "CHAPTECH-AS-AP Chaptech Pty Ltd",
"asn": 133493,
"routed_prefix": "103.238.62.0/24",
"country_code": "AU",
"name": "CHAPTECH-AS-AP Chaptech Pty Ltd",
"path": [
11164,
3491,
63956,
7594,
7594,
7594,
7594,
133493
]
},
"location": {
"country_code": "AU",
"registered_country": "Australia",
"registered_country_code": "AU",
"continent": "Oceania",
"timezone": "Australia/Sydney",
"latitude": -33.494,
"longitude": 143.2104,
"country": "Australia"
}
}
Update:
Thanks to peak but I couldn't get the additional goals bit working below. I ended up using
jq 'select(.services[].banner == "FQMDAAICCg==") | {port: .services[].port, banner: .services[].banner, ip: .ip}' censys.json | jq 'if .banner == "FQMDAAICCg==" then .ip,.port else empty end'
which is ugly but did the trick and still allowed me to stream the data to the first filter.
Original question
How do I just get the port, banner, etc. for the specific service underneath the record which hit my condition?
To get just the "port" for the service matching the condition, you could modify your query:
.services[]
| if .banner == "FQMDAAICCg==" then .port else empty end
Equivalently:
.services[]
| select(.banner == "FQMDAAICCg==")
| .port
Additional goal
I want to end up in this example with '8085' + '103.238.62.68'
If you really want the two values in that format, you could write something along the following lines, invoking jq with the -r option:
.ip as $ip
| (.services[] | select(.banner == "FQMDAAICCg==") | .port) as $port
| "'\($port)' + '\($ip)'"
or more briefly but less readably:
"'\(.services[] | select(.banner == "FQMDAAICCg==") | .port)' + '\(.ip)'"
Related
I'm trying to extract values from nested arrays in JSON below and output as CSV.
Fields to extract:
templates.name
items.name
triggers.name
Output as:
templates.name; items.name; triggers.name
Anticipated output something like:
"Template App Agent"; "Host name of zabbix_agentd running"; "Host name of zabbix_agentd was changed on {HOST.NAME}"
"Template App Agent"; "Agent ping"; "Zabbix agent on {HOST.NAME} is unreachable for 5 minutes"
"Template App Agent"; "Version of zabbix_agent(d) running"; ""
Note:
Not every item has a trigger.
Several triggers may exist for an item.
I'm new to JQ. So far only success is extracting the template name.
jq '.[] | {templates: [.templates[].name]}'
Data:
{
"zabbix_export": {
"version": "5.4",
"date": "2022-05-17T06:25:59Z",
"groups": [
{
"uuid": "7df96b18c230490a9a0a9e2307226338",
"name": "Templates"
}
],
"templates": [
{
"uuid": "e60e6598cf19448089a5f5a6c5d796a2",
"template": "Template App Agent",
"name": "Template App Agent",
"groups": [
{
"name": "Templates"
}
],
"items": [
{
"uuid": "24c03ed734d54dc8868a282a83a02200",
"name": "Host name of zabbix_agentd running",
"key": "agent.hostname",
"delay": "1h",
"history": "1w",
"trends": "0",
"value_type": "CHAR",
"request_method": "POST",
"tags": [
{
"tag": "Application",
"value": "Zabbix agent"
}
],
"triggers": [
{
"uuid": "d2d12d9e7dfe4fedb252f19b85e5e6aa",
"expression": "(last(/Template App Agent/agent.hostname,#1)<>last(/Template App Agent/agent.hostname,#2))>0",
"name": "Host name of zabbix_agentd was changed on {HOST.NAME}",
"priority": "INFO"
}
]
},
{
"uuid": "abacad4ca5eb46d29864d8a4998f1cbb",
"name": "Agent ping",
"key": "agent.ping",
"history": "1w",
"description": "The agent always returns 1 for this item. It could be used in combination with nodata() for availability check.",
"valuemap": {
"name": "Zabbix agent ping status"
},
"request_method": "POST",
"tags": [
{
"tag": "Application",
"value": "Zabbix agent"
}
],
"triggers": [
{
"uuid": "6d2a73199f3b4288bf36331a142c1725",
"expression": "nodata(/Template App Agent/agent.ping,5m)=1",
"name": "Zabbix agent on {HOST.NAME} is unreachable for 5 minutes",
"priority": "AVERAGE"
}
]
},
{
"uuid": "2cc337555efd43d181c28c792f8cbbdb",
"name": "Version of zabbix_agent(d) running",
"key": "agent.version",
"delay": "1h",
"history": "1w",
"trends": "0",
"value_type": "CHAR",
"request_method": "POST",
"tags": [
{
"tag": "Application",
"value": "Zabbix agent"
}
]
}
],
"valuemaps": [
{
"uuid": "3d66c59a28c04b0ca8227c87902ddb4d",
"name": "Zabbix agent ping status",
"mappings": [
{
"value": "1",
"newvalue": "Up"
}
]
}
]
}
]
}
}
.zabbix_export.templates[] | .name as $tn | .items[] | [ $tn, .name, .triggers[]?.name? ] | join("; ")
Loop over the templates
.zabbix_export.templates[]
Save the template name in a var
.name as $tn
Loop over the items
.items[]
Create an array with fields you like (including the name from step 1
[ $tn, .name, .triggers[]?.name? ]
Join the array to a string
join("; ")
Will output:
"Template App Agent; Host name of zabbix_agentd running; Host name of zabbix_agentd was changed on {HOST.NAME}"
"Template App Agent; Agent ping; Zabbix agent on {HOST.NAME} is unreachable for 5 minutes"
"Template App Agent; Version of zabbix_agent(d) running"
Online demo
This is a nested structure, you need to iterate level by level and add up the items you want to be in one output line. Store values from previous levels in variables.
To account for an inexistent .triggers array, you may use the Error Suppression Operator ? in combination with Alternative Operator //.
Finally, wrap the items in quotes (here using map), join them using join, and output them as raw text using the -r option
jq -r '
.[].templates[] | .name as $t
| .items[] | .name as $i
| [$t, $i, (.triggers[].name)? // ""]
| map("\"\(.)\"") | join("; ")
'
"Template App Agent"; "Host name of zabbix_agentd running"; "Host name of zabbix_agentd was changed on {HOST.NAME}"
"Template App Agent"; "Agent ping"; "Zabbix agent on {HOST.NAME} is unreachable for 5 minutes"
"Template App Agent"; "Version of zabbix_agent(d) running"; ""
Demo
Also consider using the #csv builtin, which gives you valid CSV right away (properly encoded (not just quoted) items, but separated with commas, not semicolons):
jq -r '
.[].templates[] | .name as $t
| .items[] | .name as $i
| [$t, $i, (.triggers[].name)? // ""]
| #csv
'
"Template App Agent","Host name of zabbix_agentd running","Host name of zabbix_agentd was changed on {HOST.NAME}"
"Template App Agent","Agent ping","Zabbix agent on {HOST.NAME} is unreachable for 5 minutes"
"Template App Agent","Version of zabbix_agent(d) running",""
Demo
Given the json, I need to group by key userName the object userClientDetailDTOList across all sites->buildings->floors and remove any duplicate mac addresses.
I have been able to do it using jq expression -
[.billingDetailPerSiteDTOList[].billingDetailPerBuildingDTOList[].billingDetailsPerFloorDTOList[].userClientDetailDTOList[] ] | group_by(.userName) | map((.[0]|del(.associatedMacs)) + { associatedMacs: (map(.associatedMacs[]) | unique) })
This groups by userName and also removes duplicate macs belonging to particular user. This results in a list as
[
{
"userName": "1",
"associatedMacs": [
"3:3:3:3:3:3",
"5:5:5:5:5:5"
]
},
{
"userName": "10",
"associatedMacs": [
"4:4:4:4:4:4",
"6:6:6:6:6:6"
]
},
{
"userName": "2",
"associatedMacs": [
"1:1:1:1:1:1",
"2:2:2:2:2:2"
]
},
{
"userName": "3",
"associatedMacs": [
"2:2:2:2:2:2"
]
}
]
Live example
Questions:
Can the expression be simplified?
How do I remove duplicate mac addresses across all users? The mac address 2:2:2:2:2:2 is repeated for users 2 and 3
The filter is practically as good as it can get. If you really wanted to, you could still change
del(.associatedMacs) to {userName} for a positive definition, and
(…) + {…} to {userName: …, associatedMacs: …} to avoid the addition,
resulting in
… | map({userName: (.[0].userName), associatedMacs: (map(.associatedMacs[]) | unique)})
Demo
As for the second question, if you treated the input as an INDEX on the IPs, you could mostly reuse the code from earlier (of course, the unique part wouldn't be necessary anymore)
[INDEX(…; .associatedMacs[])[]] | group_by(.userName) | map(…)
[
{
"userName": "1",
"associatedMacs": [
"3:3:3:3:3:3",
"5:5:5:5:5:5"
]
},
{
"userName": "10",
"associatedMacs": [
"4:4:4:4:4:4",
"6:6:6:6:6:6"
]
},
{
"userName": "2",
"associatedMacs": [
"1:1:1:1:1:1"
]
},
{
"userName": "3",
"associatedMacs": [
"2:2:2:2:2:2"
]
}
]
Demo
This is my first post so apologies if I make mistakes. Consider I have the following json output
{
"records": [
{
"title": "root logon",
"login": "61819009",
"uid": "ajsd879asdjksasda123asd1asd1",
"password": "OizfD19jC$ySaV$MKpSF",
"login_url": "http://192.168.0.1/",
"notes": ""
},
{
"title": "important admin account",
"login": "admin",
"uid": "asdjhkasdh89eoajdiuas98ue9aoi",
"password": "0z5gDUC#Rb354TlLq$KJ",
"login_url": "",
"notes": "",
"folders": [
{
"shared_folder": "Department",
"folder": "Important",
"option1": false,
"option2": false
}
]
}
{
"title": "another important admin",
"login": "admin#domain.com",
"uid": "asjhe98asiajsijeouiaueiaiu",
"password": "3pUs#uXEqsxCv7PRkDlJ",
"login_url": "http://192.168.0.2/",
"notes": ""
},
{
"title": "switch admin",
"login": "admin",
"uid": "asjhe89ausiodjakljskea90ik",
"password": "hMB!eMsAE8q4aDQuM4LY",
"login_url": "",
"notes": "",
"folders": [
{
"shared_folder": "Department2",
"folder": "network\\switches",
"option1": false,
"option2": false
}
]
}
]
}
from the list above, I would like to only export title, login, uid, shared folder, and folder fields to csv file. When I run the following command
jq --raw-output '.records[] | [.title, .login, .uid, .shared_folder, .folder] | # tsv > file
what I am expecting as a CSV output is
title
login
uid
shared_folder
folder
root logon
61819009
ajsd879asdjksasda123asd1asd1
important admin account
admin
asdjhkasdh89eoajdiuas98ue9aoi
Department
Important
another important admin
admin#domain.com
asjhe98asiajsijeouiaueiaiu
switch admin
admin
asjhe89ausiodjakljskea90ik
Department2
network\switches
instead I'm getting the following CSV output:
title
login
uid
root logon
61819009
ajsd879asdjksasda123asd1asd1
important admin account
admin
asdjhkasdh89eoajdiuas98ue9aoi
another important admin
admin#domain.com
asjhe98asiajsijeouiaueiaiu
switch admin
admin
asjhe89ausiodjakljskea90ik
my goal is to eliminate capturing following fields
password
login_url
notes
option1
option2
You an error in your JSON, you must add a ,.
Once done, the command is this
jq --raw-output '.records[] | [.title, .login, .uid, .folders[0].shared_folder,.folders[0].folder] |#tsv'
Here's one way:
.records[]
| [.title, .login, .uid] + ((.folders[]? // null) | [.shared_folder, .folder])
| #tsv
I'm trying to parse the results of a cURL command and the information I need is in a structure.
I tried getting to the data unsuccessfully and tried converting to PS Object but not sure how to access the structure as I'm new to PS.
Below is a sample of our cURL response.
I have a git commit hash ('c64a568399a572e82c223d55cb650b87ea1c22b8' matches latestCommit in fromRef for entry id 1101) and I need to find the corresponding displayId ('develop' in toRef)
I've done this in Linux using jq but need to replicate this in PS.
jq '.values | map(select(.fromRef.latestCommit=="'"$HASH"'")) | .[0].toRef.displayId'
I'm having 2 issues.
I can get to fromRef but it looks like #{id=refs/heads/feature/add-support; displayId=feature/add-support; latestCommit=c64a568399a572e82c223d55cb650b87ea1c22b8; repository=} and I cannot figure out how to parse
I'm not sure how to get the id so I can find the correct corresponding toRef
Any help would be greatly appreciated.
{
"size": 15,
"limit": 20,
"isLastPage": true,
"values": [
{
"id": 1101,
"version": 0,
"title": "Added header",
"description": "Added notes in header",
"state": "OPEN",
"open": true,
"closed": false,
"createdDate": 1595161367863,
"updatedDate": 1595161367863,
"fromRef": "#{id=refs/heads/feature/add-support; displayId=feature/add-support; latestCommit=c64a568399a572e82c223d55cb650b87ea1c22b8; repository=}",
"toRef": "#{id=refs/heads/develop; displayId=develop; latestCommit=58b3e3482bb35f3a735048849c2474cc676fbd9b; repository=}",
"locked": false,
"author": "#{user=; role=AUTHOR; approved=False; status=UNAPPROVED}",
"reviewers": " ",
"participants": "",
"properties": "#{mergeResult=; resolvedTaskCount=0; openTaskCount=0}",
"links": "#{self=System.Object[]}"
},
{
"id": 1053,
"version": 4,
"title": "Help with checking,",
"description": "fixed up code.",
"state": "OPEN",
"open": true,
"closed": false,
"createdDate": 1591826401310,
"updatedDate": 1595018917357,
"fromRef": "#{id=refs/heads/bugfix/checking-2.7; displayId=bugfix/checking-2.7; latestCommit=cf7d8860262c6a46b0b65ef5b6d66ae8cd698b75; repository=}",
"toRef": "#{id=refs/heads/hotfix/2.7_Improvements; displayId=hotfix/2.7_Improvements; latestCommit=01f1100c559ba41ec317421399c3bfb9a0aea91f; repository=}",
"locked": false,
"author": "#{user=; role=AUTHOR; approved=False; status=UNAPPROVED}",
"reviewers": " ",
"participants": "",
"properties": "#{mergeResult=; resolvedTaskCount=0; commentCount=4; openTaskCount=0}",
"links": "#{self=System.Object[]}"
}
],
"start": 0
}
Once you have converted the result with ConvertTo-Json and the correct -Depth parameter, you can get the values of the returned object quite easily in PowerShell.
Let's say you have used something like $json = $curlResult | ConvertTo-Json -Depth 100, then finding the displayId from the corresponding toRef can be done like this:
# this is the known hashvalue of the `fromRef` value to look for
$latestCommitHash = "c64a568399a572e82c223d55cb650b87ea1c22b8"
# get the value item. from here you can get all other properties belonging to that item
$valueItem = $json.values | Where-Object { $_.fromRef.latestCommit -eq $latestCommitHash }
# get the displayId value of the corresponding 'toRef' element:
$displayId = $valueItem.toRef.displayId
Returns
develop
I have a json output, from which I need to get id value and IPv4_address value where IPv4_address exists (this shouldn't be null). Have to use this ID value for another request along with random generated string.
Here is the breakdown of the requirement :
STEP 1 :
In the following example, for the ipv4_address:1.1.1.1 & ipv4_address:1.1.1.2, i need to get the id output which is "4e-0365-4e29-95ca-329165eecf8a" and "c9061b6674a8546cea" along with IP address.
Example of my output should look like (something similar):
1.1.1.1 4e-0365-4e29-95ca-329165eecf8a
1.1.1.2 c9061b6674a8546cea
I was trying to use jq but with this I'm not able to get the both values :
ID="$(echo "$test" \n | jq -r '.USER[] | select(.ipv4_address) | .ipv4_address')"
ID1="$(echo "$test" \n | jq -r '.USER[] | select(.ipv4_address) | .id')"
Sample output which is getting displayed with the above 2 commands :
ID value is : 1.1.1.1 1.1.1.2
ID1 value is : 4e-0365-4e29-95ca-329165eecf8a c9061b6674a8546cea
STEP 2: Profile creation: I need to use each $ID1 value in another request along with random generated string. Random string is generated as per the count of $ID1's (so here I will generate 2 random string)
And thus 2 profiles are created.
Ques: How can I get each ID from the $ID1 variable ? I tried something like ID1[0] but that seems to be wrong
STEP 3 :
Will use each ID and random string for another request, Once its done or if that step is failed, i need to provide the output to a file & output should look like :
My requirement for the final output is :
1.1.1.1 4e-0365-4e29-95ca-329165eecf8a <randomvalue-1> <profile-1> DONE
1.1.1.2 c9061b6674a8546cea <randomvalue-2> <profile-2> FAILED
where random value will be generated randomly and shall be used against the ID.
JSON output which needs to be parsed:
{
"errorcode": 0,
"message": "Done",
"operation": "get",
"resourceType": "USER",
"username": "root",
"tenant_name": "Owner",
"tenant_id": "05db6674ad458546cd2",
"resourceName": "",
"USER": [
{
"is_default": "false",
"session_timeout": "0",
"permission": "root",
"name": "ee",
"session_timeout_unit": "",
"tenant_id": "55bcb6674ad45854",
"id": "4e-0365-4e29-95ca-329165eecf8a",
"ipv4_address": "1.1.1.1",
"state": "Up",
"tenant_name": "Owner",
"encrypted": "false",
"groups": [
"owner"
],
"root_user": ""
},
{
"is_default": "false",
"session_timeout": "0",
"permission": "read",
"name": "test",
"session_timeout_unit": "",
"tenant_id": "bc906674ad458546cd2",
"id": "12cd0-fb7f-4abf-b060-48e98b794b06",
"tenant_name": "Owner",
"encrypted": "false",
"groups": [
"read"
],
"root_user": ""
},
{
"is_default": "true",
"session_timeout": "0",
"permission": "root",
"name": "root",
"session_timeout_unit": "",
"tenant_id": "c905db6d458546cd2",
"id": "c9061b6674a8546cea",
"ipv4_address": "1.1.1.2",
"state": "Not Reachable",
"tenant_name": "Owner",
"encrypted": "false",
"groups": [
"owner"
],
"root_user": ""
},
{
"is_default": "false",
"session_timeout": "0",
"permission": "readonly",
"name": "a",
"session_timeout_unit": "",
"tenant_id": "c905674ad458546cd2",
"id": "bc8a-4fd6-bc09-8c39c131b54e",
"tenant_name": "Owner",
"encrypted": "false",
"groups": [
"read"
],
"root_user": ""
}
]
}
Not quite clear with the logic of marking it DONE and FAILED. But to answer your first question where you want to select the multiple fields, you can do something like this:
$ cat input.js | jq -r '.USER[] | select(.ipv4_address) | "\(.ipv4_address) \(.id)"' > result.js
This will output the result in a file named result.js. You can apply your custom logic of marking DONE and Failed on this file.
In the above command when you do select(.ipv4_address) It basically drops all the records for which ipv4_address value is null or it is not present.
if you want to select the records which have ipv4_address as null, then your select statement would become something like this
select(.ipv4_address == null)