Bash Array from JSON Array [duplicate] - json

This question already has answers here:
convert JSON array to bash array preserving whitespaces
(2 answers)
Bash: JSON Array to Bash Array of Strings
(1 answer)
Closed last month.
I am working on a bash script which needs to ingest some data from a json (formatted with jq) and do some simple work with it, however in the process of doing so I noticed elements are getting broken down and even though I am ingesting elements containing a whitespace enclosed in double-quotes, elements are processed incorrectly.
Here's an exemplary json I will be processing
{
"start": 1689652086,
"finish": 1679652100,
"contestants": [
{
"name": "Joan-Juan Frank",
"comment": "I reek of havoc."
},
{
"name": "Kimi-Kinder Karten",
"output": "I love chocolate."
},
{
"name": "Peter-Parker Plays Piano",
"output": "And I am tired of it"
}
]
}
I composed this command in bash to get this formatted string out of the json - I am basically selecting each value from name property of each element in the json array
cat json | jq '.contestants[].name' | tr '\n' ' '
this results in this basic format which should have worked as an array declaration
"Joan-Juan Frank" "Kimi-Kinder Karten" "Peter-Parker Plays Piano"
I tried each of these two version of creating an array (none of them worked)
contestants=$(cat json | jq '.contestants[].name' | tr '\n' ' ')
array=($contestants)
or
array=($(cat json | jq '.contestants[].name' | tr '\n' ' '))
Lastly, I used this for loop to output each element
for i in "${array[#]}"; do
echo "Working on: $i"
done
output of the script is
Working on: "Joan-Juan
Working on: Frank"
Working on: "Kimi-Kinder
Working on: Karten"
Working on: "Peter-Parker
Working on: Plays
Working on: Piano"
I also went back and found out that the issue seems to be somewhere at the level of the array declaration. When I echoed the variable before declaring an array and then echoed elements of the array, here's what I got:
echo of $contestants - "Joan-Juan Frank" "Kimi-Kinder Karten" "Peter-Parker Plays Piano"
echo of $array
echo "${array[0]}"
"Joan-Juan
echo "${array[1]}"
Frank"
echo "${array[2]}"
"Kimi-Kinder
echo "${array[3]}"
Karten"
...
I have then tried to declare the array without any variable
array=("Joan-Juan Frank" "Kimi-Kinder Karten" "Peter-Parker Plays Piano" )
and this was processed correctly, as I would expect
Working on: Joan-Juan Frank
Working on: Kimi-Kinder Karten
Working on: Peter-Parker Plays Piano
I was trying suggestions from other threads like this one, this one or this one, but in my case, I wasn't able to make this work when I am ingesting content from a variable, instead of just declaring a static array. Although, it is possible I might have made a mistake along the way.
Do you have any idea how I could fix this?

You could declare the Bash array using the -a option, and have jq escape its output using the #sh builtin:
unset array
declare -a array="($(jq -r '.contestants[].name | #sh' json))"
This should have the same effect as
unset array
array=("Joan-Juan Frank" "Kimi-Kinder Karten" "Peter-Parker Plays Piano")

Related

Replace variable in JSON file in bash [duplicate]

This question already has answers here:
How to replace values in a JSON dictionary with their respective shell variables in jq?
(2 answers)
Closed 6 months ago.
EDIT: I've updated the question as the answers so far are correct but unfortunately I oversimplified the problem to the point that it will likely need a new question entirely. The issue I'm running into is that I need to replace only a part of value, and that having more than one variable causes sed to gobble multiple variables at once.
here's what I've been trying to do:
# test.json
{
"name": "Mr. ${FIRSTNAME} ${LASTNAME}"
}
I'd like to load this file, replace the variable, and store it in a variable:
NAME=Frodo JSON=$(eval "echo \"$(cat test.json)\"")
echo $JSON
I'd like it to print { "name": "Mr. Frodo Baggins" }, however it seems like eval is stripping out the double quotes, producing invalid JSON: { name: Frodo }. Any idea on how I can do this?
I should also mention that this approach seemed the cleanest, as the variables act as a key-value map. The other approach I tried was with an associative array and sed loop, which seems to work:
KEYS=( "FIRSTNAME" "LASTNAME" )
VALUES=( "Frodo" "Baggins" )
JSON=$(cat test.json)
for index in "${!KEYS[#]}"; do
JSON=$(echo "$JSON" | sed -E "s/\\$\{${KEYS[0]}}/${VALUES[0]}/g")
done
echo $JSON
NOTE: I'm aware that eval is a security risk, however this script is for testing purposes only
edit for altered OP
I'd write the script dynamically, rather than eval during runtime.
$: keys=( "FIRSTNAME" "LASTNAME" )
$: values=( "Frodo" "Baggins" )
$: sed "$(for i in "${!keys[#]}"; do echo "s/\${${keys[i]}}/${values[i]}/;"; done)" test.json
{
"name": "Mr. Frodo Baggins"
}
end edit
First, use jq if you can.
That being said...
If you are VERY CAREFUL with your quoting and ORDER of operations, eval isn't needed.
$: JSON="$(NAME=Frodo; sed s/\${NAME}/$NAME/ json)" && echo "--> $JSON <--"
--> {
"name": "Frodo"
} <--
(When testing, remember to unset JSON between runs.) ;)
This does NOT work without the internal semicolon!
$: JSON="$(NAME=Frodo sed s/\${NAME}/$NAME/ json)" && echo "--> $JSON <--"
--> {
"name": ""
} <--
Remember, assigning NAME=Frodo happens on the same parse of the command line, so you can't use $NAME in that pass. It has to either be a subsequent command or a subshell.
This works, because the parse runs in the subshell where the variable has been set, rather than is being set.
$: echo 'sed s/\${NAME}/$NAME/ json' >script
$: JSON="$(NAME=Frodo ./script)" && echo "--> $JSON <--"
--> {
"name": "Frodo"
} <--

Unable to parse JSON using jq tool due to node name containing unexpected character [duplicate]

This question already has answers here:
How to use jq when the variable has reserved characters?
(3 answers)
Closed 1 year ago.
I tried with jq to parse some JSON output inside my GitLab CI pipeline so I can extract needed information. I've tried many different ways, but I can't get the desired information out of the target node, because it has special characters and when I get to that node, the pipeline fails in each case. This is the current state of my pipeline.
This is the problematic job:
get results (dev branch):
stage: Results of scanning image
variables:
RESULTS: ""
STATUS: ""
SEVERITY: ""
image: alpine
only:
refs:
- dev
allow_failure: true
before_script:
- apk update && apk upgrade
- apk --no-cache add curl
- apk add jq
script:
- 'RESULTS=$(curl -H "Authorization: Basic `echo -n ${HARBOR_USER}:${HARBOR_PASSWORD} | base64`" -X GET "https://url.to.registry/api/v2.0/projects/project/repositories/repo-name/artifacts/latest?page=1&page_size=10&with_tag=true&with_label=true&with_scan_overview=true&with_signature=true&with_immutable_status=true")'
- echo $RESULTS
- RESULTS=$RESULTS | tr 'application/vnd.scanner.adapter.vuln.report.harbor+json; ' 'myobject'
- echo $RESULTS
- "STATUS=$RESULTS | jq '.scan_overview .myobjectversion=1.0 .scan_status'"
- "SEVERITY=$RESULTS | jq '.scan_overview .myobjectversion=1.0 .severity'"
- echo "Printing the results of the image scanning process on Harbor registry:"
- echo "status of scan:$STATUS"
- echo "severity of scan:$SEVERITY"
- echo "For more information of scan results please visit Harbor registry!"
tags:
- dev
- docker
This is the JSON output that I get from the curl command:
{
"addition_links":{
"build_history":{
"absolute":false,
"href":"..."
},
"vulnerabilities":{
"absolute":false,
"href":"...."
}
},
"digest":"sha256:bcd665be2b7c6725b410029db385d7c6c71a9ce557427cbd0f54d01a9",
"extra_attrs":{
"architecture":"amd64",
"author":null,
"created":"2021-10-22T10:28:46.058276455Z",
"os":"linux"
},
"icon":"sha256:0048162a053ee7518615bef084403614f8bca43b40ae2e762e11e06",
"id":362,
"labels":null,
"manifest_media_type":"application/vnd.docker.distribution.manifest.v2+json",
"media_type":"application/vnd.docker.container.image.v1+json",
"project_id":3,
"pull_time":"2021-10-22T10:28:55.305Z",
"push_time":"2021-10-22T10:28:49.341Z",
"references":null,
"repository_id":12,
"scan_overview":{
"application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0":{
"complete_percent":100,
"duration":8,
"end_time":"2021-10-22T10:28:57.356Z",
"report_id":"e83854eb-2304-4c58-85c9-a3e0fd9067a8",
"scan_status":"Success",
"severity":"Critical",
"start_time":"2021-10-22T10:28:49.827Z",
"summary":{
"summary":{
"Critical":7,
"High":47,
"Low":18,
"Medium":47
},
"total":119
}
}
}
}
My initial idea was to using jq to extract scan_status and severity with this command:
RESULTS=$RESULTS | jq '.scan_overview .application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0 .scan_status'
after running that command I got this error:
jq: error: syntax error, unexpected ';', expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.scan_overview .application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0 .scan_status
jq: 1 compile error
Now I am trying text replacement, but that doesn't work either.
How should I proceed in this case?
Use square brackets and double quotes around the problematic key:
jq '.scan_overview["application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0"].scan_status'
Also, this doesn't do what you think:
RESULTS=$RESULTS | tr 'application/vnd.scanner.adapter.vuln.report.harbor+json; ' 'myobject'
First of all, tr doesn't replace strings, it replaces characters. sed can replace strings.
Moreover, the pipe | is used when the left hand side produces output. Variable assignment doesn't produce any output.
Finally, to assign the output of a command to a variable, you need to use Command Substitution:
var=$(command)
~~ ~
Wrap the key in question inside barckets and quotes like so:
.scan_overview["application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0"].scan_status
Demo
From a shell syntax perspective, this looks wrong:
RESULTS=$RESULTS | jq '...'
Assigning a variable produces no output, so jq on the other side of the pipe has no input.
If it is okay for you to ignore the name of the problematic key, then you can substitute it with empty brackets:
RESULTS=$RESULTS | jq '.scan_overview [] .scan_status'
This is a bit sloppy and may match more than you would like, but in your narrow example, it will successfully pull out the value of .scan_status.
The reason this works, in this case, is because the application/vnd... property is the only property in the scan_overview object.
A more confident match would be achievable by using more quotes:
RESULTS=$RESULTS | jq '.scan_overview ."application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0" .scan_status'

Passing a path ("key1.key2") from a bash variable to jq

I am having trouble accessing bash variable inside 'jq'.
The snippet below shows my bash loop to check for missing keys in a Json file.
#!/bin/sh
for key in "key1" "key2.key3"; do
echo "$key"
if ! cat ${JSON_FILE} | jq --arg KEY "$key" -e '.[$KEY]'; then
missingKeys+=${key}
fi
done
JSON_FILE:
{
"key1": "val1",
"key2": {
"key3": "val3"
}
}
The script works correctly for top level keys such as "key1". But it does not work correctly (returns null) for "key2.key3".
'jq' on the command line does return the correct value
cat input.json | jq '.key2.key3'
"val3"
I followed answers from other posts to come to this solution. However can't seem to figure out why it does not work for nested json keys.
Using --arg prevents your data from being incorrectly parsed as syntax. Usually, a shell variable you're passing into jq contains literal data, so this is the correct thing.
In this case, your variable contains syntax, not literal data: The . isn't part of the string you want to do a lookup by, but is instead an instruction to jq to do two separate lookups one after the other.
So, in this case, you should do the more obvious thing, instead of using --arg:
jq -e ".$KEY"

Loop through JSON array shell script

I am trying to write a shell script that loops through a JSON file and does some logic based on every object's properties. The script was initially written for Windows but it does not work properly on a MacOS.
The initial code is as follows
documentsJson=""
jsonStrings=$(cat "$file" | jq -c '.[]')
while IFS= read -r document; do
# Get the properties from the docment (json string)
currentKey=$(echo "$document" | jq -r '.Key')
encrypted=$(echo "$document" | jq -r '.IsEncrypted')
# If not encrypted then don't do anything with it
if [[ $encrypted != true ]]; then
echoComment " Skipping '$currentKey' as it's not marked for encryption"
documentsJson+="$document,"
continue
fi
//some more code
done <<< $jsonStrings
When ran on a MacOs, the whole file is processed at once, so it does not loop through objects.
The closest I got to making it work - after trying a lot of suggestions - is as follows:
jq -r '.[]' "$file" | while read i; do
for config in $i ; do
currentKey=$(echo "$config" | jq -r '.Key')
echo "$currentKey"
done
done
The console result is parse error: Invalid numeric literal at line 1, column 6
I just cannot find a proper way of grabbing the JSON object and reading its properties.
JSON file example
[
{
"Key": "PdfMargins",
"Value": {
"Left":0,
"Right":0,
"Top":20,
"Bottom":15
}
},
{
"Key": "configUrl",
"Value": "someUrl",
"IsEncrypted": true
}
]
Thank you in advance!
Try putting the $jsonStrings in doublequotes: done <<< "$jsonStrings"
Otherwise the standard shell splitting applies on the variable expansion and you probably want to retain the line structure of the output of jq.
You could also use this in bash:
while IFS= read -r document; do
...
done < <(jq -c '.[]' < "$file")
That would save some resources. I am not sure about making this work on MacOS, though, so test this first.

Read JSON data in a shell script [duplicate]

This question already has answers here:
Parsing JSON with Unix tools
(45 answers)
Closed 6 years ago.
In shell I have a requirement wherein I have to read the JSON response which is in the following format:
{ "Messages": [ { "Body": "172.16.1.42|/home/480/1234/5-12-2013/1234.toSort", "ReceiptHandle": "uUk89DYFzt1VAHtMW2iz0VSiDcGHY+H6WtTgcTSgBiFbpFUg5lythf+wQdWluzCoBziie8BiS2GFQVoRjQQfOx3R5jUASxDz7SmoCI5bNPJkWqU8ola+OYBIYNuCP1fYweKl1BOFUF+o2g7xLSIEkrdvLDAhYvHzfPb4QNgOSuN1JGG1GcZehvW3Q/9jq3vjYVIFz3Ho7blCUuWYhGFrpsBn5HWoRYE5VF5Bxc/zO6dPT0n4wRAd3hUEqF3WWeTMlWyTJp1KoMyX7Z8IXH4hKURGjdBQ0PwlSDF2cBYkBUA=", "MD5OfBody": "53e90dc3fa8afa3452c671080569642e", "MessageId": "e93e9238-f9f8-4bf4-bf5b-9a0cae8a0ebc" } ] }
Here I am only concerned with the "Body" property value. I made some unsuccessful attempts like:
jsawk -a 'return this.Body'
or
awk -v k="Body" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}
But that did not suffice. Can anyone help me with this?
There is jq for parsing json on the command line:
jq '.Body'
Visit this for jq: https://stedolan.github.io/jq/
tl;dr
$ cat /tmp/so.json | underscore select '.Messages .Body'
["172.16.1.42|/home/480/1234/5-12-2013/1234.toSort"]
Javascript CLI tools
You can use Javascript CLI tools like
underscore-cli:
json:select(): CSS-like selectors for JSON.
Example
Select all name children of a addons:
underscore select ".addons > .name"
The underscore-cli provide others real world examples as well as the json:select() doc.
Similarly using Bash regexp. Shall be able to snatch any key/value pair.
key="Body"
re="\"($key)\": \"([^\"]*)\""
while read -r l; do
if [[ $l =~ $re ]]; then
name="${BASH_REMATCH[1]}"
value="${BASH_REMATCH[2]}"
echo "$name=$value"
else
echo "No match"
fi
done
Regular expression can be tuned to match multiple spaces/tabs or newline(s). Wouldn't work if value has embedded ". This is an illustration. Better to use some "industrial" parser :)
Here is a crude way to do it: Transform JSON into bash variables to eval them.
This only works for:
JSON which does not contain nested arrays, and
JSON from trustworthy sources (else it may confuse your shell script, perhaps it may even be able to harm your system, You have been warned)
Well, yes, it uses PERL to do this job, thanks to CPAN, but is small enough for inclusion directly into a script and hence is quick and easy to debug:
json2bash() {
perl -MJSON -0777 -n -E 'sub J {
my ($p,$v) = #_; my $r = ref $v;
if ($r eq "HASH") { J("${p}_$_", $v->{$_}) for keys %$v; }
elsif ($r eq "ARRAY") { $n = 0; J("$p"."[".$n++."]", $_) foreach #$v; }
else { $v =~ '"s/'/'\\\\''/g"'; $p =~ s/^([^[]*)\[([0-9]*)\](.+)$/$1$3\[$2\]/;
$p =~ tr/-/_/; $p =~ tr/A-Za-z0-9_[]//cd; say "$p='\''$v'\'';"; }
}; J("json", decode_json($_));'
}
use it like eval "$(json2bash <<<'{"a":["b","c"]}')"
Not heavily tested, though. Updates, warnings and more examples see my GIST.
Update
(Unfortunately, following is a link-only-solution, as the C code is far
too long to duplicate here.)
For all those, who do not like the above solution,
there now is a C program json2sh
which (hopefully safely) converts JSON into shell variables.
In contrast to the perl snippet, it is able to process any JSON,
as long as it is well formed.
Caveats:
json2sh was not tested much.
json2sh may create variables, which start with the shellshock pattern () {
I wrote json2sh to be able to post-process .bson with Shell:
bson2json()
{
printf '[';
{ bsondump "$1"; echo "\"END$?\""; } | sed '/^{/s/$/,/';
echo ']';
};
bsons2json()
{
printf '{';
c='';
for a;
do
printf '%s"%q":' "$c" "$a";
c=',';
bson2json "$a";
done;
echo '}';
};
bsons2json */*.bson | json2sh | ..
Explained:
bson2json dumps a .bson file such, that the records become a JSON array
If everything works OK, an END0-Marker is applied, else you will see something like END1.
The END-Marker is needed, else empty .bson files would not show up.
bsons2json dumps a bunch of .bson files as an object, where the output of bson2json is indexed by the filename.
This then is postprocessed by json2sh, such that you can use grep/source/eval/etc. what you need, to bring the values into the shell.
This way you can quickly process the contents of a MongoDB dump on shell level, without need to import it into MongoDB first.