This question already has answers here:
Escape field name in jq that contains '#' and '-'? [duplicate]
(2 answers)
jq special characters in nested keys
(1 answer)
Closed 3 years ago.
Given this json file wtf.json:
{
"I-am-test-v2": {
"exist": true
},
"works": {
"exist": true
}
}
I can verify it has these keys via:
$ jq 'keys' wtf.json
[
"I-am-test-v2",
"works"
]
I can access works via:
$ jq .works wtf.json
{
"exist": true
}
yet I cannot select:
$ jq .I-am-test-v2 wtf.json
as that will yield in error:
jq: error: am/0 is not defined at <top-level>, line 1:
.I-am-test-v2
jq: error: test/0 is not defined at <top-level>, line 1:
.I-am-test-v2
jq: error: v2/0 is not defined at <top-level>, line 1:
.I-am-test-v2
jq: 3 compile errors
I assume it has to do with the special char -, yet I am unsure how to quote or escape it, as these attempts also fail with the same error:
jq ".I-am-test-v2" wtf.json
jq ."I-am-test-v2" wtf.json
or some different error via:
jq ."I\-am\-test\-v2" wtf.json
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.I\-am\-test\-v2
jq: 1 compile error
I also tried:
jq .["I-am-test-v2"] wtf.json
How am I supposed to access the key?
Your key contains the dash - that is interpreted as the substraction operator.
You need to tell jq that it's a string by double quoting your key.
If you do that in command line, enclose your jq "script" into single quote in order to avoid your shell interpreting any special characters.
<wtf.json jq '."I-am-test-v2"'
You can access it via proper enquotation via:
jq '.["I-am-test-v2"]' wtf.json
{
"exist": true
}
Then it also works with escaping:
jq ".[\"I-am-test-v2\"]" wtf.json
Note that you cannot reverse the quote style:
jq ".['I-am-test-v2']" wtf.json
jq: error: syntax error, unexpected INVALID_CHARACTER (Unix shell quoting issues?) at <top-level>, line 1:
.['I-am-test-v2']
jq: 1 compile error
Related
Trying to make jq work with ANSI color codes.
Test cases:
$ echo '{"a":"b","c":"d"}' | jq -r .c
d # Matches my expected output
$ echo '{"a":"b","c":"\033[31md\033[0m"}' | jq -r .c
parse error: Invalid escape at line 1, column 31 # returns err code 4
$ echo '{"a":"b","c":"d"}' | jq -r '"foo"+.c+"bar"'
foodbar # Correct
$ echo '{"a":"b","c":"d"}' | jq -r '"\033[31m"+.c+"\033[0m"'
jq: error: Invalid escape at line 1, column 4 (while parsing '"\0"') at <top-level>, line 1:
"\033[31m"+.c+"\033[0m"
jq: error: Invalid escape at line 1, column 4 (while parsing '"\0"') at <top-level>, line 1:
"\033[31m"+.c+"\033[0m"
jq: 2 compile errors # returns err code 3
$ jq -rn '"\033[31mbar\033[0m"'
jq: error: Invalid escape at line 1, column 4 (while parsing '"\0"') at <top-level>, line 1:
"\033[31mbar\033[0m"
jq: error: Invalid escape at line 1, column 4 (while parsing '"\0"') at <top-level>, line 1:
"\033[31mbar\033[0m"
jq: 2 compile errors # returns err code 4
P.S. in case it matters, I am using the bash shell with version 5.1.16(1)-release on Linux.
Conslusion: ANSI colors do not work with jq, whether in the JSON string or directly concatenating it through the + operator.
Question: how to make ANSI colors work in jq? Any help would be appreciated.
Octal escape sequences are not valid JSON syntax, so you need to encode the ASCII escape character as \u001b rather than \033. Also, to add to the confusion, some versions of echo will attempt to interpret backslash (escape) sequences itself before passing them to jq, so in cases like this it's much safer to use printf '%s\n':
$ printf '%s\n' '{"a":"b","c":"\u001b[31md\u001b[0m"}' | jq -r .c
d
(You can't see it, but that "d" is red in my terminal.)
BTW, an easy way to find things like this out is to get jq to encode them in JSON for you. Here, I'll set the shell variable to the actual string (using bash's $'...' string format, which interprets ANSI-C escape sequences like \033), then use --arg to pass that to jq:
$ seq=$'\033[31md\033[0m'
$ jq -nc --arg seq "$seq" '{"a":"b","c":$seq}'
{"a":"b","c":"\u001b[31md\u001b[0m"}
I have the following script:
#!/bin/bash
CONFIG_RECORDER=`aws configservice describe-configuration-recorders`
NAME=$(jq -r ‘.ConfigurationRecorders[].name’ <<<“$CONFIG_RECORDER”)
echo $NAME
I am trying to retrieve the value for name from the following JSON object:
{
"ConfigurationRecorders": [{
"name": "default",
"roleARN": "arn:aws:iam::xxxxxxxxxxxx:role/Config-Recorder",
"recordingGroup": {
"allSupported": true,
"includeGlobalResourceTypes": true,
"resourceTypes": []
}
}]
}
When running the script, I am getting an error stating that jq could not open the file. That is because I am trying to pass in the result stored in a variable. How can I move past this? Below is the error:
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end
(Unix shell quoting issues?) at <top-level>, line 1:
‘.ConfigurationRecorders[].name’
jq: 1 compile error
NAME=$(jq -r '.ConfigurationRecorders[].name' <<<"$CONFIG_RECORDER")
<<< is known as here-string.
-r removes the quotes in the result.
[] in the jq query is necessary as ConfigurationRecorders is a JSON array.
$( … ) is just an other form of command substitution than backticks. I prefer this one as it is more readable to me.
I have been trying to reduce the array to a string to be used in string interpolation.
For example.
input = ["123", "456"]
expected output = array=123,456
Here is my try
$ echo '["123", "456"]' | jq 'array=\(.|join(","))'
jq: error: syntax error, unexpected INVALID_CHARACTER (Unix shell quoting issues?) at <top-level>, line 1:
array=\(.|join(","))
jq: 1 compile error
Using string interpolation \(.), you can do something like below. Your idea is almost right, but interpolation using \(..) needs the filter to present be inside a string with the expression to be used defined inside parens after a backslash
jq --raw-output '"array=\(join(","))"'
echo '["123", "456"]' | jq -r '"array=" + join(",")'
I try to construct a json object with jq. I start with an empty object and want to add keys and values dynamically.
This works but the key is not variable. It's fixed to "foo":
echo '{"foo": ["baz"]}' | jq --arg value "bar" '.foo += [$value]'
output as expected:
{"foo": ["baz", "bar"]}
What I actually want do do is something like this:
echo '{"foo": ["baz"]}' | jq --arg key "foo" --arg value "bar" '.($key) += [$value]'
Unfortunately this does not work. Here is the output:
jq: error: syntax error, unexpected '(' (Unix shell quoting issues?) at <top-level>, line 1:
.($key) += [$value]
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.($key) += [$value]
jq: 2 compile errors
I couldn't find a solution or figure it out.
I know that this works: jq --null-input --arg key foo '{($key): "bar"}' but it doesn't solve my problem since I want to append values to existing lists as you can see in the examples.
You need to use square parens [..] instead of (..) as reported in the error message. Just do
jq --arg key "foo" --arg value "bar" '.[$key] += [$value]'
This error line is quite verbose to recommend you the right syntax to use. The emphasis with # is mine
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1
# ^^^^^^^^^^^^^
This question already has answers here:
Parse or view JSON data fields using JQ tool utility where field names have a "-" dash in the key name
(4 answers)
Closed 4 years ago.
I use jq 1.5 in a Windows Environment to modify an json object that i receiving from Amazon s3. I have there a funny Problem. I use jq to extract single keys of the object:
{
"s3_direct_url": "https://fanzo-photos.s3.amazonaws.com/photos/images/034/005/322/screen1.jpg",
"url": "https://fanzo-photos.s3.amazonaws.com",
"fields": {
"key": "photos/images/034/005/322/screen1.jpg",
"success_action_status": "200",
"Content-Type": "image/jpeg",
"acl": "public-read",
"policy": "eyJleHBpcmF0aW9uIjoiMjAxOC0xMS0wMlQxMzo0NzoxNVoiLCJjb25kaXRpb25zIjpbeyJidWNrZXQiOiJmYW56by1waG90b3MifSx7ImtleSI6InBob3Rvcy9pbWFnZXMvMDM0LzAwNS8zMjIvc2NyZWVuMS5qcGcifSx7InN1Y2Nlc3NfYWN0aW9uX3N0YXR1cyI6IjIwMCJ9LHsiQ29udGVudC1UeXBlIjoiaW1hZ2UvanBlZyJ9LHsiYWNsIjoicHVibGljLXJlYWQifSx7IngtYW16LWNyZWRlbnRpYWwiOiJBS0lBSlkzWVRCV1NMQzQ2SFdCQS8yMDE4MTEwMi91cy1lYXN0LTEvczMvYXdzNF9yZXF1ZXN0In0seyJ4LWFtei1hbGdvcml0aG0iOiJBV1M0LUhNQUMtU0hBMjU2In0seyJ4LWFtei1kYXRlIjoiMjAxODExMDJUMTI0NzE1WiJ9XX0=",
"x-amz-credential": "AKIAJY3YTBWSLC46HWBA/20181102/us-east-1/s3/aws4_request",
"x-amz-algorithm": "AWS4-HMAC-SHA256",
"x-amz-date": "20181102T124715Z",
"x-amz-signature": "52d8246536e8743fba8e7668cb65a08a1142221d54a58676b6ab14e3835482a3"
},
"id": 34005322,
"media_type": "InputMedia"
}
If i extract informations from the 'fields' object without a '-' in the key name that works fine. If i try to extract a key with '-' in the name i got following error:
jq: error: amz/0 is not defined at <top-level>, line 1:
.fields.x-amz-credential
jq: error: credential/0 is not defined at <top-level>, line 1:
.fields.x-amz-credential
jq: 2 compile errors
exit status 3
Update:
After the hint with the FAQ and the " i rebuild the jq command and tested it in jqplay: .fields."Content-Type" where it works as expected. Under powershell that variant didn't working since the powershell didn't excepting the quotating.
.\jq .fields."Content-Type" jq: error: Type/0 is not defined at <top-level>, line 1: .fields.Content-Type jq: 1 compile error
BR
Timo
Any tips?
Yes! If you cannot find the answer in the onine jq manual, check the jq FAQ:
𝑸: How can I access the value of a key with hyphens or $ or other
special characters in it? Why does .a.["$"] produce a syntax error?
A: The basic form for accessing the value of a key is .["KEYNAME"]
where "KEYNAME" is any valid JSON string, but recent versions of jq
also allow ."KEYNAME".
Using the basic form might require explicit use of the pipe symbol, as
in .["a-b"]|.["x-y"], but this can be abbreviated to .["a-b"]["x-y"].