How can I parse nested JSON in PowerShell? - json

I'm trying to parse the results of a cURL command and the information I need is in a structure.
I tried getting to the data unsuccessfully and tried converting to PS Object but not sure how to access the structure as I'm new to PS.
Below is a sample of our cURL response.
I have a git commit hash ('c64a568399a572e82c223d55cb650b87ea1c22b8' matches latestCommit in fromRef for entry id 1101) and I need to find the corresponding displayId ('develop' in toRef)
I've done this in Linux using jq but need to replicate this in PS.
jq '.values | map(select(.fromRef.latestCommit=="'"$HASH"'")) | .[0].toRef.displayId'
I'm having 2 issues.
I can get to fromRef but it looks like #{id=refs/heads/feature/add-support; displayId=feature/add-support; latestCommit=c64a568399a572e82c223d55cb650b87ea1c22b8; repository=} and I cannot figure out how to parse
I'm not sure how to get the id so I can find the correct corresponding toRef
Any help would be greatly appreciated.
{
"size": 15,
"limit": 20,
"isLastPage": true,
"values": [
{
"id": 1101,
"version": 0,
"title": "Added header",
"description": "Added notes in header",
"state": "OPEN",
"open": true,
"closed": false,
"createdDate": 1595161367863,
"updatedDate": 1595161367863,
"fromRef": "#{id=refs/heads/feature/add-support; displayId=feature/add-support; latestCommit=c64a568399a572e82c223d55cb650b87ea1c22b8; repository=}",
"toRef": "#{id=refs/heads/develop; displayId=develop; latestCommit=58b3e3482bb35f3a735048849c2474cc676fbd9b; repository=}",
"locked": false,
"author": "#{user=; role=AUTHOR; approved=False; status=UNAPPROVED}",
"reviewers": " ",
"participants": "",
"properties": "#{mergeResult=; resolvedTaskCount=0; openTaskCount=0}",
"links": "#{self=System.Object[]}"
},
{
"id": 1053,
"version": 4,
"title": "Help with checking,",
"description": "fixed up code.",
"state": "OPEN",
"open": true,
"closed": false,
"createdDate": 1591826401310,
"updatedDate": 1595018917357,
"fromRef": "#{id=refs/heads/bugfix/checking-2.7; displayId=bugfix/checking-2.7; latestCommit=cf7d8860262c6a46b0b65ef5b6d66ae8cd698b75; repository=}",
"toRef": "#{id=refs/heads/hotfix/2.7_Improvements; displayId=hotfix/2.7_Improvements; latestCommit=01f1100c559ba41ec317421399c3bfb9a0aea91f; repository=}",
"locked": false,
"author": "#{user=; role=AUTHOR; approved=False; status=UNAPPROVED}",
"reviewers": " ",
"participants": "",
"properties": "#{mergeResult=; resolvedTaskCount=0; commentCount=4; openTaskCount=0}",
"links": "#{self=System.Object[]}"
}
],
"start": 0
}

Once you have converted the result with ConvertTo-Json and the correct -Depth parameter, you can get the values of the returned object quite easily in PowerShell.
Let's say you have used something like $json = $curlResult | ConvertTo-Json -Depth 100, then finding the displayId from the corresponding toRef can be done like this:
# this is the known hashvalue of the `fromRef` value to look for
$latestCommitHash = "c64a568399a572e82c223d55cb650b87ea1c22b8"
# get the value item. from here you can get all other properties belonging to that item
$valueItem = $json.values | Where-Object { $_.fromRef.latestCommit -eq $latestCommitHash }
# get the displayId value of the corresponding 'toRef' element:
$displayId = $valueItem.toRef.displayId
Returns
develop

Related

Splunk not recognizing regex

I'm struggling to make a regex work with splunk. It works with regex 101, but splunk doesn't seem to recognize it!
Regex: \"([\w]+)\":([^,}]+)
Log entry:
May 20 12:22:21 127.0.0.1 {"rootId": "AXIxikL8ao-yaSvA", "requestId": "f6a873jkjjkjk:-8000:5738",
"details": {"flag": false, "title": "task 1", "status": "Waiting", "group": "", "order": 0},
"operation": "Creation", "objectId": "AXIyCN5Oao-H5aYyaSvd", "startDate": 1589977341890,
"objectType": "case_task", "base": true, "object": {"_routing": "AXIxikL8ao-H5aYyaSvA", "flag":
false, "_type": "case_task", "title": "task 1", "createdAt": 1589977341516, "_parent": "AXIxikL8ao-
H5aYyaSvA", "createdBy": "user", "_id": "AXIyCN5Oao-H5aYyaSvd", "id": "AXIyCN5Oao-H5aYyaSvd",
"_version": 1, "order": 0, "status": "Waiting", "group": ""}}
Regex 101 link:
https://regex101.com/r/XBuz9Y/2/
I suspect splunk may have a different regex syntax, but i don't really know how to adapt it.
Any help?
Thanks!
You may use
... | rex max_match=0 "\"(?<key>\w+)\":(?<value>[^,}]+)"
Here, max_match=0 will enable multiple matching (by defauly, if you do not use max_match parameter, only the first match is returned) and the named capturing groups (here, see (?<key>...) and (?<value>...)) will ensure field creation.
See more about the Splunk rex command.
Grab the JSON fragment of your event using rex, and then use spath to do the extraction.
rex field=_raw "^[^{]+(?<json>.*)" | spath input=json
This should extract the JSON fields with the appropriate structure.

How to select multiple parameters from JSON output, which meets a condition & further select individual value

I have a json output, from which I need to get id value and IPv4_address value where IPv4_address exists (this shouldn't be null). Have to use this ID value for another request along with random generated string.
Here is the breakdown of the requirement :
STEP 1 :
In the following example, for the ipv4_address:1.1.1.1 & ipv4_address:1.1.1.2, i need to get the id output which is "4e-0365-4e29-95ca-329165eecf8a" and "c9061b6674a8546cea" along with IP address.
Example of my output should look like (something similar):
1.1.1.1 4e-0365-4e29-95ca-329165eecf8a
1.1.1.2 c9061b6674a8546cea
I was trying to use jq but with this I'm not able to get the both values :
ID="$(echo "$test" \n | jq -r '.USER[] | select(.ipv4_address) | .ipv4_address')"
ID1="$(echo "$test" \n | jq -r '.USER[] | select(.ipv4_address) | .id')"
Sample output which is getting displayed with the above 2 commands :
ID value is : 1.1.1.1 1.1.1.2
ID1 value is : 4e-0365-4e29-95ca-329165eecf8a c9061b6674a8546cea
STEP 2: Profile creation: I need to use each $ID1 value in another request along with random generated string. Random string is generated as per the count of $ID1's (so here I will generate 2 random string)
And thus 2 profiles are created.
Ques: How can I get each ID from the $ID1 variable ? I tried something like ID1[0] but that seems to be wrong
STEP 3 :
Will use each ID and random string for another request, Once its done or if that step is failed, i need to provide the output to a file & output should look like :
My requirement for the final output is :
1.1.1.1 4e-0365-4e29-95ca-329165eecf8a <randomvalue-1> <profile-1> DONE
1.1.1.2 c9061b6674a8546cea <randomvalue-2> <profile-2> FAILED
where random value will be generated randomly and shall be used against the ID.
JSON output which needs to be parsed:
{
"errorcode": 0,
"message": "Done",
"operation": "get",
"resourceType": "USER",
"username": "root",
"tenant_name": "Owner",
"tenant_id": "05db6674ad458546cd2",
"resourceName": "",
"USER": [
{
"is_default": "false",
"session_timeout": "0",
"permission": "root",
"name": "ee",
"session_timeout_unit": "",
"tenant_id": "55bcb6674ad45854",
"id": "4e-0365-4e29-95ca-329165eecf8a",
"ipv4_address": "1.1.1.1",
"state": "Up",
"tenant_name": "Owner",
"encrypted": "false",
"groups": [
"owner"
],
"root_user": ""
},
{
"is_default": "false",
"session_timeout": "0",
"permission": "read",
"name": "test",
"session_timeout_unit": "",
"tenant_id": "bc906674ad458546cd2",
"id": "12cd0-fb7f-4abf-b060-48e98b794b06",
"tenant_name": "Owner",
"encrypted": "false",
"groups": [
"read"
],
"root_user": ""
},
{
"is_default": "true",
"session_timeout": "0",
"permission": "root",
"name": "root",
"session_timeout_unit": "",
"tenant_id": "c905db6d458546cd2",
"id": "c9061b6674a8546cea",
"ipv4_address": "1.1.1.2",
"state": "Not Reachable",
"tenant_name": "Owner",
"encrypted": "false",
"groups": [
"owner"
],
"root_user": ""
},
{
"is_default": "false",
"session_timeout": "0",
"permission": "readonly",
"name": "a",
"session_timeout_unit": "",
"tenant_id": "c905674ad458546cd2",
"id": "bc8a-4fd6-bc09-8c39c131b54e",
"tenant_name": "Owner",
"encrypted": "false",
"groups": [
"read"
],
"root_user": ""
}
]
}
Not quite clear with the logic of marking it DONE and FAILED. But to answer your first question where you want to select the multiple fields, you can do something like this:
$ cat input.js | jq -r '.USER[] | select(.ipv4_address) | "\(.ipv4_address) \(.id)"' > result.js
This will output the result in a file named result.js. You can apply your custom logic of marking DONE and Failed on this file.
In the above command when you do select(.ipv4_address) It basically drops all the records for which ipv4_address value is null or it is not present.
if you want to select the records which have ipv4_address as null, then your select statement would become something like this
select(.ipv4_address == null)

Use jq to Convert json File to csv

I am using curl to pull Alien Vault OTX pulses from their API, the initial output I receive is in json format and I need to convert this json into csv as so it can be read by some other software. I aim to use jq as many others have recommended it.
{ "count": 1210, "next": "https://otx.alienvault.com/api/v1/pulses/subscribed?page=2", "results": [
{
"industries": [],
"tlp": "white",
"description": "Tropic Trooper (also known as KeyBoy) levels its campaigns against Taiwanese, Philippine, and Hong Kong targets, focusing on their government, healthcare, transportation, and high-tech industries. Its operators are believed to be very organized and develop their own cyberespionage tools that they fine-tuned in their recent campaigns. Many of the tools they use now feature new behaviors, including a change in the way they maintain a foothold in the targeted network.",
"created": "2018-03-14T17:24:48.014000",
"tags": [
"china",
"keyboy",
"tropic trooper"
],
"modified": "2018-03-14T17:24:48.014000",
"author_name": "AlienVault",
"public": 1,
"extract_source": [],
"references": [
"https://blog.trendmicro.com/trendlabs-security-intelligence/tropic-trooper-new-strategy/"
],
"targeted_countries": [],
"indicators": [
{
"indicator": "CVE-2018-0802",
"description": "",
"created": "2018-03-14T17:25:03",
"title": "",
"content": "",
"type": "CVE",
"id": 406248965
},
{
"indicator": "fb9c9cbf6925de8c7b6ce8e7a8d5290e628be0b82a58f3e968426c0f734f38f6",
"description": "",
"created": "2018-03-14T17:25:03",
"title": "",
"content": "",
"type": "FileHash-SHA256",
"id": 438581959
}
],
"more_indicators": false,
"revision": 1,
"adversary": "Tropic Trooper",
"id": "5aa95ae02781860367e354e4",
"name": "Tropic Troopers New Strategy"
}
I am looking to use jq to extract certain fields and convert to csv. My expected output would look something like:
"CVE-2018-0802","CVE"
"tibetnews.today","domain"
"02281e26e89b61d84e2df66a0eeb729c5babd94607b1422505cd388843dd5456","FileHash-SHA256"
So far I have tried:
<AV.json jq -r '.results.indicators[] | [.indicator, .type] | #csv' AV.csv
Any help is greatly appreciated.
Cheers,
George
.results is an array so you'll have to expand it too. This can be done either by:
.results[] | .indicators[] | [.indicator, .type] | #csv
or more compactly:
.results[].indicators[] | [.indicator, .type] | #csv
You'll also have to direct the output to the designated file, e.g.:
jq -r -f program.jq < AV.json > AV.csv
Output
"CVE-2018-0802","CVE"
"fb9c9cbf6925de8c7b6ce8e7a8d5290e628be0b82a58f3e968426c0f734f38f6","FileHash-SHA256"

jq only show when object doesnt match

I'm trying to set up an alert for when the following JSON object state says anything but started. I'm beginning to play around with conditional jq but I'm unsure how to implement regex into this.
{
"page": 0,
"page_size": 100,
"total_pages": 10,
"total_rows": 929,
"headers": [
"*"
],
"rows": [
{
"id": "168",
"state": "STARTED"
},
{
"id": "169",
"state": "FAILED"
},
{
"id": "170",
"state": "STARTED"
}
]
}
I only want to display the id and state of the failed object, this is what I tried
jq '.rows[] | .id, select(.state | contains("!STARTED"))' test.json
I'd like my output to be something like
{
"id": "169",
"state": "FAILED"
}
If you simply want to print out the objects for which .state is NOT "STARTED", just use negation:
.rows[] | select(.state != "STARTED")
If the "started" state is associated with multiple values, please give further details. There might not be any need to use regular expressions. If you really do need to use regular expressions, then you will probably want to use test.

Not able to Iterate through JSON response object with JQ in Unix shell

I am trying to iterate through a JSON object in UNIX, where the idea is to pickup different values and append it to a string and forward it as a syslog. Below is the code.
//picking up the length of Object
count=$(jq '.content | length' red)
#echo $count
enter code here
for((i=0;i<$count;i++))
do
echo "MY VALUE OF I"
echo $i
//THE BELOW LINE GIVES ERROR UPON USAGE of $i
id="$(cat red | jq '.content[$i].id')"
source=$(cat red | jq '.content[$i].entitySummary.source')
.
.
#syslogString="ID=$id SOURCE=$source SUMMARY=$summaryText TITLE=$title DESCRIPTION=$description SEVERITY=$severity MITIGATION=$mitigation IMPACT=$impactDescrip$
echo $id
echo "value of ID ($id)"
I am receiving compilation error with content[$i] and cant get a workaround the same.
The response class looks like this:
Page {
content ( array[ ClientIncident ] )
The list of results that make up the page. The number of elements should be less than or equal to the currentPage size.
currentPage ( Pagination )
Size and offset information about the current page.
total ( integer )
The total number of results found. If there are a large number of results, this may be an estimate. Accuracy should improve as the page approaches the end of the resultset.
}
under content the JSON response looks as below:
{
"content": [
{
"id": 951653,
"version": 12,
"score": 100,
"entitySummary": {
"source": "somewebsite",
"summaryText": "someTEXT here",
"domain": "www.domian.com",
"sourceDate": "2014-12-19T17:00:00.000Z",
"type": "WEB_PAGE"
},
"type": "SomeTYPE",
"title": "some Title",
"description": "some description ",
"occurred": "2014-12-19T17:00:00.000Z",
"verified": "2014-12-19T17:17:22.326Z",
"tags": [
{
"id": 424,
"name": "Data Breach or Compromise",
"type": "IMPACT_EFFECTS"
},
{
"id": 1064,
"name": "United States",
"type": "TARGET_GEOGRAPHY"
},
],
"severity": "MEDIUM",
"clientId": "NET",
"alerted": "2014-12-19T17:39:55.500Z",
"mitigation": "MititgationINFO",
"impactDescription": "IMpact description": 0
},
{
"id": 951174,
"version": 8,
"score": 100,
"entitySummary": {
Ok I got the answer for this.
We can use the below syntax to make is work in the for loop.
id=$(cat red | jq '.content['${i}'].id')