Retrieving value from JSON object in bash using JQ - json

I have the following script:
#!/bin/bash
CONFIG_RECORDER=`aws configservice describe-configuration-recorders`
NAME=$(jq -r ‘.ConfigurationRecorders[].name’ <<<“$CONFIG_RECORDER”)
echo $NAME
I am trying to retrieve the value for name from the following JSON object:
{
"ConfigurationRecorders": [{
"name": "default",
"roleARN": "arn:aws:iam::xxxxxxxxxxxx:role/Config-Recorder",
"recordingGroup": {
"allSupported": true,
"includeGlobalResourceTypes": true,
"resourceTypes": []
}
}]
}
When running the script, I am getting an error stating that jq could not open the file. That is because I am trying to pass in the result stored in a variable. How can I move past this? Below is the error:
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end
(Unix shell quoting issues?) at <top-level>, line 1:
‘.ConfigurationRecorders[].name’
jq: 1 compile error

NAME=$(jq -r '.ConfigurationRecorders[].name' <<<"$CONFIG_RECORDER")
<<< is known as here-string.
-r removes the quotes in the result.
[] in the jq query is necessary as ConfigurationRecorders is a JSON array.
$( … ) is just an other form of command substitution than backticks. I prefer this one as it is more readable to me.

Related

How to access property via jq with special chars? [duplicate]

This question already has answers here:
Escape field name in jq that contains '#' and '-'? [duplicate]
(2 answers)
jq special characters in nested keys
(1 answer)
Closed 3 years ago.
Given this json file wtf.json:
{
"I-am-test-v2": {
"exist": true
},
"works": {
"exist": true
}
}
I can verify it has these keys via:
$ jq 'keys' wtf.json
[
"I-am-test-v2",
"works"
]
I can access works via:
$ jq .works wtf.json
{
"exist": true
}
yet I cannot select:
$ jq .I-am-test-v2 wtf.json
as that will yield in error:
jq: error: am/0 is not defined at <top-level>, line 1:
.I-am-test-v2
jq: error: test/0 is not defined at <top-level>, line 1:
.I-am-test-v2
jq: error: v2/0 is not defined at <top-level>, line 1:
.I-am-test-v2
jq: 3 compile errors
I assume it has to do with the special char -, yet I am unsure how to quote or escape it, as these attempts also fail with the same error:
jq ".I-am-test-v2" wtf.json
jq ."I-am-test-v2" wtf.json
or some different error via:
jq ."I\-am\-test\-v2" wtf.json
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.I\-am\-test\-v2
jq: 1 compile error
I also tried:
jq .["I-am-test-v2"] wtf.json
How am I supposed to access the key?
Your key contains the dash - that is interpreted as the substraction operator.
You need to tell jq that it's a string by double quoting your key.
If you do that in command line, enclose your jq "script" into single quote in order to avoid your shell interpreting any special characters.
<wtf.json jq '."I-am-test-v2"'
You can access it via proper enquotation via:
jq '.["I-am-test-v2"]' wtf.json
{
"exist": true
}
Then it also works with escaping:
jq ".[\"I-am-test-v2\"]" wtf.json
Note that you cannot reverse the quote style:
jq ".['I-am-test-v2']" wtf.json
jq: error: syntax error, unexpected INVALID_CHARACTER (Unix shell quoting issues?) at <top-level>, line 1:
.['I-am-test-v2']
jq: 1 compile error

How to dynamically parse a JSON object with shell jq

I got a question about shell's jq. So my JSON object is:
{"1543446000": {"name": "John", "company": "foo"}, "1543359600": {"name": "Kate", "company": "bar"}}
The numers 1543446000 and 1543359600 are UNIX timestamps. How can I parse one of the JSON objects by the timestamp with a shell variable?
My shell script so far:
#!/bin/sh
URL="https://pastebin.com/raw/w7awz7kZ"
export DATE=$(date -d "$today 0" +%s)
JSON=$(curl -H "Accept: application/json" $API_URL)
JSON=$(echo $JSON | jq --arg date $DATE '.$date')
echo $JSON
Doesn't seem to work. My intention is to select the inner JSON object described by one of the timestamps, which are basically midnight of today. So I want to select today's data set.
Any suggestions?
Greets,
Innoberger
You need to use the full syntax for key access, as the dollar sign preclude you using the shorter form. The error message should provide this suggestion.
$ jq --arg date 1543359600 '.$date' tmp.json
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.$date
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.$date
jq: 2 compile errors
Note the error message
'try .["field"] instead of .field'.
You won't need the quotes, though, as that would be how you specify a literal key $date.
$ jq --arg date 1543359600 '.[$date]' tmp.json
{
"name": "Kate",
"company": "bar"
}

"Invalid numeric literal" error from jq trying to modify JSON with variable

I am wanting to pipe values into a bash script that will change values in a json file using jq. I've been working on this for a while now and I can't get past the first set of errors.
Here is my simple json file:
{
"0000000": {
"pogo": "AJHVUYKJBOIHKNNLNM"
},
"7000000": {
"pogo": "PPPVUYKJBOIHKNNLNM"
}
}
Here is my script
#!/bin/bash
#-- pass into script
pgid="${1}"
tpogo="${2}"
file="file.json"
#-- tmp files
tmp_dir="$(mktemp -d -t 'text.XXXXX' || mktemp -d 2>/dev/null)"
tmp_input1="${tmp_dir}/temp_input1.txt"
if [ ! -n "$2" ]; then
{ echo "Check your command as you are missing a variable ..."; echo "Example: script.sh "00000001" "jvkkjbkjbd" ..."; }
exit 1;
fi
jq -r ".${pgid}.pogo = \"${tpogo}\"" "$file" > "$tmp_input1"
cat "$tmp_input1"
rm -rf "$tmp_dir"
Here are the errors:
jq: error: Invalid numeric literal at EOF at line 1, column 9 (while parsing '.0000000.') at <top-level>, line 1:
.0000000.pogo = "XXXXXXX"
jq: error: syntax error, unexpected IDENT, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.0000000.pogo = "XXXXXXX"
jq: 2 compile errors
I have tried so many variations from stack and most of them look similar to what I am doing now.
As for the immediate issue: .key works with { "foo": "value" }, but .100 doesn't work with { "100": "value" }; the syntax you're relying is sugar, only available for a limited subset of keys. .["100"] would work, but generating that by expanding shell variables into strings parsed as code is fragile (jq isn't a side-effecting language as of current releases, but in languages that do support I/O operations, such substitutions can be leveraged for injection attacks). To do things the Right Way, pass your variables out-of-band from your code, and use them for lookups in a manner that doesn't rely on what they contain.
The jq equivalent to awk's -v var="$value" is --arg var "$value", used thusly:
jq --arg pgid "$pgid" \
--arg tpogo "$tpogo" \
'.[$pgid].pogo = $tpogo'
Testing this with your data:
json='{"0000000":{"pogo":"AJHVUYKJBOIHKNNLNM"},"7000000":{"pogo":"PPPVUYKJBOIHKNNLNM"}}'
pgid="0000000"
tpogo="XXXXXXX"
jq --arg pgid "$pgid" --arg tpogo "$tpogo" \
'.[$pgid].pogo = $tpogo' <<<"$json"
...emits as output:
{
"0000000": {
"pogo": "XXXXXXX"
},
"7000000": {
"pogo": "PPPVUYKJBOIHKNNLNM"
}
}

Invalid numeric literal with jq

I have a large amount of JSON from a 3rd party system which I would like to pre-process with jq, but I am having difficulty composing the query, test case follows:
$ cat test.json
{
"a": "b",
"c": "d",
"e": {
"1": {
"f": "g",
"h": "i"
}
}
}
$ cat test.json|jq .e.1.f
jq: error: Invalid numeric literal at EOF at line 1, column 3 (while parsing '.1.') at <top-level>, line 1:
.e.1.f
How would I get "g" as my output here? Or how do I cast that 1 to a "1" so it is handled correctly?
From jq manual :
You can also look up fields of an object using syntax like .["foo"]
(.foo above is a shorthand version of this, but only for
identifier-like strings).
You also need quotes and use -r if you want raw output :
jq -r '.e["1"].f' test.json
I wrote a shell script function that calls the curl command, and pipes it into the jq command.
function getName {
curl http://localhost:123/getname/$1 | jq;
}
export -f getName
When I ran this from the CLI,
getName jarvis
I was getting this response:
parse error: Invalid numeric literal at line 1, column 72
I tried removing the | jq from the curl command, and I got back the result without jq parsing:
<Map><timestamp>1234567890</timestamp><status>404</status><error>Not Found</error><message>....
I first thought that I had a bad character in the curl command, or that I was using the function param $1 wrong.
Then I counted the number of chars in the result string, and I noticed that the 72nd char in that string was the empty space between "Not Found".
The underlying issue was that I didn't have a method named getname yet in my spring REST controller, so the response was coming back 404 Not Found. But in addition, jq wasn't handling the empty space in the response except by outputting the error message.
I'm new to jq so maybe there is a way to get around the empty space issue, but that's for another day.

jq read .txt file and write the values to json file

I want to use jq to parse a .txt file with a list of country codes and write them to the value in a JSON object.
Here is what I have so far:
cat myfile.json |
jq -R -f test_id.txt 'select(.country == []).country = "test_id.txt"' > newfile.json
Where .txt file looks like this:
"NSC"
"KZC"
"KCC"
"KZL"
"NZG"
"VRU"
"ESM"
"KZF"
"SFU"
"EWF"
"KQY"
"KQV"
and my JSON looks like this:
{
"scsRequestId": null,
"includeMetadata": true,
"includeHoldings": true,
"country": [],
"region": [],
"oclcSymbol": []
}
Here is the error I am getting:
jq: error: syntax error, unexpected QQSTRING_START, expecting $end (Unix shell quoting issues?) at <top-level>, line 2:
"KZC"
jq: 1 compile error
I want the list of country codes to go into the country array.
-f's argument is the file to read the filter to run from. If you want to read data from a file, that's a use for --slurpfile, not -f.
Thus:
jq --slurpfile countries test_id.txt '.country=$countries' <myfile.json >newfile.json
When run with your provided inputs, the resulting contents in newfile.json are:
{
"scsRequestId": null,
"includeMetadata": true,
"includeHoldings": true,
"country": [
"NSC",
"KZC",
"KCC",
"KZL",
"NZG",
"VRU",
"ESM",
"KZF",
"SFU",
"EWF",
"KQY",
"KQV"
],
"region": [],
"oclcSymbol": []
}