remember previous json value in array - json

I have a sorted array list and I wish to make a formatting decision depending on the value of a json bool in the previous iteration of the array. The code example uses the current value of igroup but I need to have a MARK on the first instance of igroup=true and then all subsequent igroup=true to be SPACE. To do this I need to know what the previous value of igroup was. It seems a variable to try and remember the prev value is not possible so I am at a loss as to how I can make this happen.
Code:
.result
| keys[] as $c
|
(
(.[$c].segments[0].lines | keys[]) as $l |
[
"line format",
( if .[$c].segments[0].lines[$l].igroup then "SPACE"
else "MARK" end ),
( [ .[$c].segments[0].lines[$l].products[].name ] | join(",")),
"end line format"
]
),
["END","0"]
| join("|")
example data:
{
"result": [
{
"cn": "abc",
"segments": [
{
"lines": [
{
"igroup": false,
"products": [
{
"name": "Should be MARK"
}
]
},
{
"igroup": false,
"products": [
{
"name": "Should be MARK"
},
{
"name": "Addon to MARK"
}
]
},
{
"igroup": true,
"products": [
{
"name": "Should be MARK First igroup=true BROKEN!!!"
}
]
},
{
"igroup": true,
"products": [
{
"name": "Should be SPACE !! After first igroup=true"
}
]
},
{
"igroup": true,
"products": [
{
"name": "Should be SPACE until next igroup=false"
}
]
},
{
"igroup": false,
"products": [
{
"name": "Should be MARK"
}
]
}
]
}
]
}
],
"id": 1
}
The output:
"line format|MARK|Should be MARK|end line format"
"line format|MARK|Should be MARK,Addon to MARK|end line format"
"line format|SPACE|Should be MARK First igroup=true BROKEN!!!|end line format"
"line format|SPACE|Should be SPACE !! After first igroup=true|end line format"
"line format|SPACE|Should be SPACE until next igroup=false|end line format"
"line format|MARK|Should be MARK|end line format"
"END|0"

There's a couple of approaches you could take, reduce over the indices so you could index into the array to check the values. But this requires keeping a separate reference to the original input.
Otherwise using foreach allows you to keep track of the previous igroup through the accumulator.
foreach .result[].segments[].lines[] as {$igroup,$products} (
[null,false]; # init: pair of previous and current value
[.[1],$igroup]; # update: update values
[
"line format",
if .[0] and .[0] == $igroup then "SPACE" else "MARK" end,
([$products[].name] | join(",")),
"end line format"
] # extract: the collection of values to output
), ["END",0] | join("|")
"line format|MARK|Should be MARK|end line format"
"line format|MARK|Should be MARK,Addon to MARK|end line format"
"line format|MARK|Should be MARK First igroup=true BROKEN!!!|end line format"
"line format|SPACE|Should be SPACE !! After first igroup=true|end line format"
"line format|SPACE|Should be SPACE until next igroup=false|end line format"
"line format|MARK|Should be MARK|end line format"
"END|0"
Tweak the output as needed.
jqplay

I was able to make some progress on my own but am still not happy with the result. I was able to use the index to check the prev arrays value. I dont like the 3 deep if-then-else-end blocks but at least it works. What it does do is allows me access to the multitude of other fields in the segments and lines arrays that the test data doesn't show.
.result
| keys[] as $c
|
(
(.[$c].segments[0].lines | keys[]) as $l |
[
( $l ),
( if $l > 0
then
( if .[$c].segments[0].lines[$l].igroup
then
( if .[$c].segments[0].lines[$l - 1].igroup
then "SPACE"
else "MARK" end
)
else "MARK" end
)
else "MARK" end
),
"line format",
( [ .[$c].segments[0].lines[$l].products[].name ] | join(",")),
"end line format"
]
),
["END","0"]
| join("|")
I will look into the solution above as an alternative.

Related

jq - return array value if its length is not null

I have a report.json generated by a gitlab pipeline.
It looks like:
{"version":"14.0.4","vulnerabilities":[{"id":"64e69d1185ecc48a1943141dcb6dbd628548e725f7cef70d57403c412321aaa0","category":"secret_detection"....and so on
If no vulnerabilities found, then "vulnerabilities":[]. I'm trying to come up with a bash script that would check if vulnerabilities length is null or not. If not, print the value of the vulnerabilities key. Sadly, I'm very far from scripting genius, so it's been a struggle.
While searching web for a solution to this, I've come across jq. It seems like select() should do the job.
I've tried:
jq "select(.vulnerabilities!= null)" report.json
but it returned {"version":"14.0.4","vulnerabilities":[{"id":"64e69d1185ecc48a194314... instead of expected "vulnerabilities":[{"id":"64e69d1185ecc48a194314...
and
map(select(.vulnerabilities != null)) report.json
returns "No matches found"
Would you mind pointing out what's wrong apart from my 0 experience with bash and JSON parsing? :)
Thanks in advance
Just use . filter to identify the object vulnerabilities.
these is some cases below
$ jq '.vulnerabilities' <<END
heredoc> {"version":"14.0.4","vulnerabilities":[{"id":"64e69d1185ecc48a1943141dcb6dbd628548e725f7cef70d57403c412321aaa0","category":"secret_detection"}]}
heredoc> END
[
{
"id": "64e69d1185ecc48a1943141dcb6dbd628548e725f7cef70d57403c412321aaa0",
"category": "secret_detection"
}
]
if vulnerabilities null, then jq will return null
$ jq '.vulnerabilities' <<END
{"version":"14.0.4","vulnerabilities":null}
END
null
then with pipe |, you can change it to any output you wanted.
change null to []: .vulnerabilities | if . == null then [] else . end
filter empty array: .vulnerabilities | select(length > 0)
For further information about jq filters, you can read the jq manual.
Assuming, by "print the value of the vulnerabilities key" you mean the value of an item's id field. You can retrieve it using .id and have it extracted to bash with the -r option.
If in case the array is not empty you want all of the "keys", iterate over the array using .[]. If you just wanted a specific key, let's say the first, address it using a 0-based index: .[0].
To check the length of an array there is a dedicated length builtin. However, as your final goal is to extract, you can also attempt to do so right anyway, suppress a potential unreachability error using the ? operator, and have your bash script read an appropriate exit status using the -e option.
Your bash script then could include the following snippet
if key=$(jq -re '.vulnerabilities[0].id?' report.json)
then
# If the array was not empty, $key contains the first key
echo "There is a vulnerability in key $key."
fi
# or
if keys=$(jq -re '.vulnerabilities[].id?' report.json)
then
# If the array was not empty, $keys contains all the keys
for k in $keys
do echo "There is a vulnerability in key $k."
done
fi
Firstly, please note that in the JSON world, it is important to distinguish
between [] (the empty array), the values 0 and null, and the absence of a value (e.g. as the result of the absence of a key in an object).
In the following, I'll assume that the output should be the value of .vulnerabilities
if it is not `[]', or nothing otherwise:
< sample.json jq '
select(.vulnerabilities != []).vulnerabilities
'
If the goal were to differentiate between two cases based on the return code from jq, you could use the -e command-line option.
You can use if-then-else.
Filter
if (.vulnerabilities | length) > 0 then {vulnerabilities} else empty end
Input
{
"version": "1.1.1",
"vulnerabilities": [
{
"id": "111",
"category": "secret_detection"
},
{
"id": "112",
"category": "secret_detection"
}
]
}
{
"version": "1.2.1",
"vulnerabilities": [
{
"id": "121",
"category": "secret_detection 2"
}
]
}
{
"version": "3.1.1",
"vulnerabilities": []
}
{
"version": "4.1.1",
"vulnerabilities": [
{
"id": "411",
"category": "secret_detection 4"
},
{
"id": "412",
"category": "secret_detection"
},
{
"id": "413",
"category": "secret_detection"
}
]
}
Output
{
"vulnerabilities": [
{
"id": "111",
"category": "secret_detection"
},
{
"id": "112",
"category": "secret_detection"
}
]
}
{
"vulnerabilities": [
{
"id": "121",
"category": "secret_detection 2"
}
]
}
{
"vulnerabilities": [
{
"id": "411",
"category": "secret_detection 4"
},
{
"id": "412",
"category": "secret_detection"
},
{
"id": "413",
"category": "secret_detection"
}
]
}
Demo
https://jqplay.org/s/wicmr4uVRm

Omitting null values for sub() in JQ

I'm trying to change # to %23 in every context value, but I'm having problem with null values.
The shortened JSON is:
{
"stats": {
"suites": 1
},
"results": [
{
"uuid": "676-a46b-47a1-a49f-4da4e46c1120",
"title": "",
"suites": [
{
"uuid": "gghjh-56a9-4713-b139-0d5b36bc7fbc",
"title": "Login process",
"tests": [
{
"pass": false,
"fail": true,
"pending": false,
"context": "\"screenshots/login.spec.js/Login process -- should login #11 (failed).png\""
},
{
"pass": false,
"fail": false,
"pending": true,
"context": null
}
]
}
]
}
]
}
And the JQ command I think it's closest to correct is:
jq '.results[].suites[].tests[].context | strings | sub("#";"%23")'
But the problem is that I need to get in return full edited file. How could I achieve that?
You were close. To retain the original structure, you need to use the update operator (|=) instead of pipe. Enclosing the entire expression to the left of it in parentheses is also necessary, otherwise the original input will be invisible to |=.
(.results[].suites[].tests[].context | strings) |= sub("#"; "%23")
Online demo
change # to %23 in every context value
You might wish to consider:
walk( if type=="object" and (.context|type)=="string"
then .context |= gsub("#"; "%23")
else . end )

jq - output array as csv followed by other fields in the object

I have the following json layout:
test.json
{
"end": 9,
"previous_page_uri": null,
"messages": [
{
"error_message": null,
"num_media": "1",
"status": "received"
},
{
"error_message": null,
"num_media": "2",
"status": "received"
}
],
"end1": "end page 1",
"end2": "end page 2"
}
I want to output the .messages object as csv, followed by the "end1" value.
Is there a way to do that in jq?
To produce the csv:
jq '.messages[] | [.error_message, .num_media, .status]|#csv' test.json
which produces this:
",\"1\",\"received\""
",\"2\",\"received\""
How can I add .end1?
You can use , Comma to concatenate the output of two filters and ( ) Parenthesis to specify them separately. For example with the sample input you provided the filter
( .messages[] | [.error_message, .num_media, .status] | #csv ), .end1
generates
",\"1\",\"received\""
",\"2\",\"received\""
"end page 1"
Try it online!

Handling multiple top level elements with jq

I'm trying to modify the deluge web.conf file with jq and I'm having some issues. The deluge web config seems to be invalid json
{
"file": 1,
"format": 1
}
{
"sidebar_show_zero": false,
"show_session_speed": false,
"pwd_sha1": "CHANGEME",
"show_sidebar": true,
"sessions": {},
"enabled_plugins": [],
"base": "/",
"first_login": true,
"theme": "gray",
"pkey": "ssl/daemon.pkey",
"default_daemon": "",
"cert": "test",
"session_timeout": 3600,
"https": false,
"interface": "0.0.0.0",
"sidebar_multiple_filters": true,
"pwd_salt": "salt",
"port": 8112
}
It has multiple top level elements which aren't separated by a comma so if I try to edit the file with jq using something like this jq '.pwd_sha1 = "NEW HASH"' web.conf I get the following
{
"file": 1,
"format": 1,
"pwd_sha1": "NEW HASH"
}
{
"sidebar_show_zero": false,
"show_session_speed": false,
"pwd_sha1": "NEW HASH",
"show_sidebar": true,
"sessions": {},
"enabled_plugins": [],
"base": "/",
"first_login": true,
"theme": "gray",
"pkey": "ssl/daemon.pkey",
"default_daemon": "",
"cert": "test",
"session_timeout": 3600,
"https": false,
"interface": "0.0.0.0",
"sidebar_multiple_filters": true,
"pwd_salt": "salt",
"port": 8112
}
jq is adding a new element to the first top level object and changing the second top level element's value. How can I get this to only change the existing item in the second top level element?
The web.conf you show is a stream of JSON entities. Fortunately for you, jq is stream-oriented, and it appears from your example that you could simply write:
jq 'if .pwd_sha1 then .pwd_sha1 = "NEW HASH" else . end' web.conf
In general, though, it might be more appropriate to write something with a more stringent test, e.g.
jq 'if type == "object" and has("pwd_sha1")
then .pwd_sha1 = "NEW HASH" else . end' web.conf
"changing the second top level element's value"
To edit the second top-level item only, you could use foreach inputs with the -n command-line option:
foreach inputs as $in (0; .+1;
if . == 2 then $in | .pwd_sha1 = "NEW_HASH"
else $in end)

Not able to Iterate through JSON response object with JQ in Unix shell

I am trying to iterate through a JSON object in UNIX, where the idea is to pickup different values and append it to a string and forward it as a syslog. Below is the code.
//picking up the length of Object
count=$(jq '.content | length' red)
#echo $count
enter code here
for((i=0;i<$count;i++))
do
echo "MY VALUE OF I"
echo $i
//THE BELOW LINE GIVES ERROR UPON USAGE of $i
id="$(cat red | jq '.content[$i].id')"
source=$(cat red | jq '.content[$i].entitySummary.source')
.
.
#syslogString="ID=$id SOURCE=$source SUMMARY=$summaryText TITLE=$title DESCRIPTION=$description SEVERITY=$severity MITIGATION=$mitigation IMPACT=$impactDescrip$
echo $id
echo "value of ID ($id)"
I am receiving compilation error with content[$i] and cant get a workaround the same.
The response class looks like this:
Page {
content ( array[ ClientIncident ] )
The list of results that make up the page. The number of elements should be less than or equal to the currentPage size.
currentPage ( Pagination )
Size and offset information about the current page.
total ( integer )
The total number of results found. If there are a large number of results, this may be an estimate. Accuracy should improve as the page approaches the end of the resultset.
}
under content the JSON response looks as below:
{
"content": [
{
"id": 951653,
"version": 12,
"score": 100,
"entitySummary": {
"source": "somewebsite",
"summaryText": "someTEXT here",
"domain": "www.domian.com",
"sourceDate": "2014-12-19T17:00:00.000Z",
"type": "WEB_PAGE"
},
"type": "SomeTYPE",
"title": "some Title",
"description": "some description ",
"occurred": "2014-12-19T17:00:00.000Z",
"verified": "2014-12-19T17:17:22.326Z",
"tags": [
{
"id": 424,
"name": "Data Breach or Compromise",
"type": "IMPACT_EFFECTS"
},
{
"id": 1064,
"name": "United States",
"type": "TARGET_GEOGRAPHY"
},
],
"severity": "MEDIUM",
"clientId": "NET",
"alerted": "2014-12-19T17:39:55.500Z",
"mitigation": "MititgationINFO",
"impactDescription": "IMpact description": 0
},
{
"id": 951174,
"version": 8,
"score": 100,
"entitySummary": {
Ok I got the answer for this.
We can use the below syntax to make is work in the for loop.
id=$(cat red | jq '.content['${i}'].id')