This question already has answers here:
Parse or view JSON data fields using JQ tool utility where field names have a "-" dash in the key name
(4 answers)
Closed 4 years ago.
I use jq 1.5 in a Windows Environment to modify an json object that i receiving from Amazon s3. I have there a funny Problem. I use jq to extract single keys of the object:
{
"s3_direct_url": "https://fanzo-photos.s3.amazonaws.com/photos/images/034/005/322/screen1.jpg",
"url": "https://fanzo-photos.s3.amazonaws.com",
"fields": {
"key": "photos/images/034/005/322/screen1.jpg",
"success_action_status": "200",
"Content-Type": "image/jpeg",
"acl": "public-read",
"policy": "eyJleHBpcmF0aW9uIjoiMjAxOC0xMS0wMlQxMzo0NzoxNVoiLCJjb25kaXRpb25zIjpbeyJidWNrZXQiOiJmYW56by1waG90b3MifSx7ImtleSI6InBob3Rvcy9pbWFnZXMvMDM0LzAwNS8zMjIvc2NyZWVuMS5qcGcifSx7InN1Y2Nlc3NfYWN0aW9uX3N0YXR1cyI6IjIwMCJ9LHsiQ29udGVudC1UeXBlIjoiaW1hZ2UvanBlZyJ9LHsiYWNsIjoicHVibGljLXJlYWQifSx7IngtYW16LWNyZWRlbnRpYWwiOiJBS0lBSlkzWVRCV1NMQzQ2SFdCQS8yMDE4MTEwMi91cy1lYXN0LTEvczMvYXdzNF9yZXF1ZXN0In0seyJ4LWFtei1hbGdvcml0aG0iOiJBV1M0LUhNQUMtU0hBMjU2In0seyJ4LWFtei1kYXRlIjoiMjAxODExMDJUMTI0NzE1WiJ9XX0=",
"x-amz-credential": "AKIAJY3YTBWSLC46HWBA/20181102/us-east-1/s3/aws4_request",
"x-amz-algorithm": "AWS4-HMAC-SHA256",
"x-amz-date": "20181102T124715Z",
"x-amz-signature": "52d8246536e8743fba8e7668cb65a08a1142221d54a58676b6ab14e3835482a3"
},
"id": 34005322,
"media_type": "InputMedia"
}
If i extract informations from the 'fields' object without a '-' in the key name that works fine. If i try to extract a key with '-' in the name i got following error:
jq: error: amz/0 is not defined at <top-level>, line 1:
.fields.x-amz-credential
jq: error: credential/0 is not defined at <top-level>, line 1:
.fields.x-amz-credential
jq: 2 compile errors
exit status 3
Update:
After the hint with the FAQ and the " i rebuild the jq command and tested it in jqplay: .fields."Content-Type" where it works as expected. Under powershell that variant didn't working since the powershell didn't excepting the quotating.
.\jq .fields."Content-Type" jq: error: Type/0 is not defined at <top-level>, line 1: .fields.Content-Type jq: 1 compile error
BR
Timo
Any tips?
Yes! If you cannot find the answer in the onine jq manual, check the jq FAQ:
𝑸: How can I access the value of a key with hyphens or $ or other
special characters in it? Why does .a.["$"] produce a syntax error?
A: The basic form for accessing the value of a key is .["KEYNAME"]
where "KEYNAME" is any valid JSON string, but recent versions of jq
also allow ."KEYNAME".
Using the basic form might require explicit use of the pipe symbol, as
in .["a-b"]|.["x-y"], but this can be abbreviated to .["a-b"]["x-y"].
Related
This question already has answers here:
How to use jq when the variable has reserved characters?
(3 answers)
Closed 1 year ago.
I tried with jq to parse some JSON output inside my GitLab CI pipeline so I can extract needed information. I've tried many different ways, but I can't get the desired information out of the target node, because it has special characters and when I get to that node, the pipeline fails in each case. This is the current state of my pipeline.
This is the problematic job:
get results (dev branch):
stage: Results of scanning image
variables:
RESULTS: ""
STATUS: ""
SEVERITY: ""
image: alpine
only:
refs:
- dev
allow_failure: true
before_script:
- apk update && apk upgrade
- apk --no-cache add curl
- apk add jq
script:
- 'RESULTS=$(curl -H "Authorization: Basic `echo -n ${HARBOR_USER}:${HARBOR_PASSWORD} | base64`" -X GET "https://url.to.registry/api/v2.0/projects/project/repositories/repo-name/artifacts/latest?page=1&page_size=10&with_tag=true&with_label=true&with_scan_overview=true&with_signature=true&with_immutable_status=true")'
- echo $RESULTS
- RESULTS=$RESULTS | tr 'application/vnd.scanner.adapter.vuln.report.harbor+json; ' 'myobject'
- echo $RESULTS
- "STATUS=$RESULTS | jq '.scan_overview .myobjectversion=1.0 .scan_status'"
- "SEVERITY=$RESULTS | jq '.scan_overview .myobjectversion=1.0 .severity'"
- echo "Printing the results of the image scanning process on Harbor registry:"
- echo "status of scan:$STATUS"
- echo "severity of scan:$SEVERITY"
- echo "For more information of scan results please visit Harbor registry!"
tags:
- dev
- docker
This is the JSON output that I get from the curl command:
{
"addition_links":{
"build_history":{
"absolute":false,
"href":"..."
},
"vulnerabilities":{
"absolute":false,
"href":"...."
}
},
"digest":"sha256:bcd665be2b7c6725b410029db385d7c6c71a9ce557427cbd0f54d01a9",
"extra_attrs":{
"architecture":"amd64",
"author":null,
"created":"2021-10-22T10:28:46.058276455Z",
"os":"linux"
},
"icon":"sha256:0048162a053ee7518615bef084403614f8bca43b40ae2e762e11e06",
"id":362,
"labels":null,
"manifest_media_type":"application/vnd.docker.distribution.manifest.v2+json",
"media_type":"application/vnd.docker.container.image.v1+json",
"project_id":3,
"pull_time":"2021-10-22T10:28:55.305Z",
"push_time":"2021-10-22T10:28:49.341Z",
"references":null,
"repository_id":12,
"scan_overview":{
"application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0":{
"complete_percent":100,
"duration":8,
"end_time":"2021-10-22T10:28:57.356Z",
"report_id":"e83854eb-2304-4c58-85c9-a3e0fd9067a8",
"scan_status":"Success",
"severity":"Critical",
"start_time":"2021-10-22T10:28:49.827Z",
"summary":{
"summary":{
"Critical":7,
"High":47,
"Low":18,
"Medium":47
},
"total":119
}
}
}
}
My initial idea was to using jq to extract scan_status and severity with this command:
RESULTS=$RESULTS | jq '.scan_overview .application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0 .scan_status'
after running that command I got this error:
jq: error: syntax error, unexpected ';', expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.scan_overview .application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0 .scan_status
jq: 1 compile error
Now I am trying text replacement, but that doesn't work either.
How should I proceed in this case?
Use square brackets and double quotes around the problematic key:
jq '.scan_overview["application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0"].scan_status'
Also, this doesn't do what you think:
RESULTS=$RESULTS | tr 'application/vnd.scanner.adapter.vuln.report.harbor+json; ' 'myobject'
First of all, tr doesn't replace strings, it replaces characters. sed can replace strings.
Moreover, the pipe | is used when the left hand side produces output. Variable assignment doesn't produce any output.
Finally, to assign the output of a command to a variable, you need to use Command Substitution:
var=$(command)
~~ ~
Wrap the key in question inside barckets and quotes like so:
.scan_overview["application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0"].scan_status
Demo
From a shell syntax perspective, this looks wrong:
RESULTS=$RESULTS | jq '...'
Assigning a variable produces no output, so jq on the other side of the pipe has no input.
If it is okay for you to ignore the name of the problematic key, then you can substitute it with empty brackets:
RESULTS=$RESULTS | jq '.scan_overview [] .scan_status'
This is a bit sloppy and may match more than you would like, but in your narrow example, it will successfully pull out the value of .scan_status.
The reason this works, in this case, is because the application/vnd... property is the only property in the scan_overview object.
A more confident match would be achievable by using more quotes:
RESULTS=$RESULTS | jq '.scan_overview ."application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0" .scan_status'
This question already has answers here:
Escape field name in jq that contains '#' and '-'? [duplicate]
(2 answers)
jq special characters in nested keys
(1 answer)
Closed 3 years ago.
Given this json file wtf.json:
{
"I-am-test-v2": {
"exist": true
},
"works": {
"exist": true
}
}
I can verify it has these keys via:
$ jq 'keys' wtf.json
[
"I-am-test-v2",
"works"
]
I can access works via:
$ jq .works wtf.json
{
"exist": true
}
yet I cannot select:
$ jq .I-am-test-v2 wtf.json
as that will yield in error:
jq: error: am/0 is not defined at <top-level>, line 1:
.I-am-test-v2
jq: error: test/0 is not defined at <top-level>, line 1:
.I-am-test-v2
jq: error: v2/0 is not defined at <top-level>, line 1:
.I-am-test-v2
jq: 3 compile errors
I assume it has to do with the special char -, yet I am unsure how to quote or escape it, as these attempts also fail with the same error:
jq ".I-am-test-v2" wtf.json
jq ."I-am-test-v2" wtf.json
or some different error via:
jq ."I\-am\-test\-v2" wtf.json
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.I\-am\-test\-v2
jq: 1 compile error
I also tried:
jq .["I-am-test-v2"] wtf.json
How am I supposed to access the key?
Your key contains the dash - that is interpreted as the substraction operator.
You need to tell jq that it's a string by double quoting your key.
If you do that in command line, enclose your jq "script" into single quote in order to avoid your shell interpreting any special characters.
<wtf.json jq '."I-am-test-v2"'
You can access it via proper enquotation via:
jq '.["I-am-test-v2"]' wtf.json
{
"exist": true
}
Then it also works with escaping:
jq ".[\"I-am-test-v2\"]" wtf.json
Note that you cannot reverse the quote style:
jq ".['I-am-test-v2']" wtf.json
jq: error: syntax error, unexpected INVALID_CHARACTER (Unix shell quoting issues?) at <top-level>, line 1:
.['I-am-test-v2']
jq: 1 compile error
I use JQ 1.5 in a Windows10 enviroment (PowerShell).
I built a jq statement that works on the example data on jqplay but throws a error on my enviroment:
Sample: Code share
Code:
. | { last_update: .starbase_detailed_scan.last_update_time, user_name: .starbase_detailed_scan.owner_name, alliance_id: .starbase_detailed_scan.owner_alliance_id, drydocks: .starbase_detailed_scan.num_drydocks, tier: .starbase_detailed_scan.owner_level, defence_plattform: .starbase_detailed_scan.num_defence_platforms, shield_triggered: .starbase_detailed_scan.player_shield.triggered_on, shield_end: .starbase_detailed_scan.player_shield.expiry_time, parsteel: .resources."325683920".current_amount, tritanium: .resources."743985951".current_amount, dilithium: .resources."2614028847".current_amount, user_id: .starbase_detailed_scan.owner_user_id, defence_rating: .starbase_detailed_scan.defense_rating }
The problem are the JSON objects with a numeric identifier. On jqplay I got the correct values. On PowerShell jq I get an error. I expected that this is a PowerShell problem so I tried to move the filter into a filter file. The error is then gone but I get only NULL as value for the three objects.
Numbers in the json path need to be marked Oldschool like:
.starbase_detailed_scan.resources["2614028847"]
BR
Timo
I have a large amount of JSON from a 3rd party system which I would like to pre-process with jq, but I am having difficulty composing the query, test case follows:
$ cat test.json
{
"a": "b",
"c": "d",
"e": {
"1": {
"f": "g",
"h": "i"
}
}
}
$ cat test.json|jq .e.1.f
jq: error: Invalid numeric literal at EOF at line 1, column 3 (while parsing '.1.') at <top-level>, line 1:
.e.1.f
How would I get "g" as my output here? Or how do I cast that 1 to a "1" so it is handled correctly?
From jq manual :
You can also look up fields of an object using syntax like .["foo"]
(.foo above is a shorthand version of this, but only for
identifier-like strings).
You also need quotes and use -r if you want raw output :
jq -r '.e["1"].f' test.json
I wrote a shell script function that calls the curl command, and pipes it into the jq command.
function getName {
curl http://localhost:123/getname/$1 | jq;
}
export -f getName
When I ran this from the CLI,
getName jarvis
I was getting this response:
parse error: Invalid numeric literal at line 1, column 72
I tried removing the | jq from the curl command, and I got back the result without jq parsing:
<Map><timestamp>1234567890</timestamp><status>404</status><error>Not Found</error><message>....
I first thought that I had a bad character in the curl command, or that I was using the function param $1 wrong.
Then I counted the number of chars in the result string, and I noticed that the 72nd char in that string was the empty space between "Not Found".
The underlying issue was that I didn't have a method named getname yet in my spring REST controller, so the response was coming back 404 Not Found. But in addition, jq wasn't handling the empty space in the response except by outputting the error message.
I'm new to jq so maybe there is a way to get around the empty space issue, but that's for another day.
I understand that jq search needs to be blocked by {} and the key needs to be encased with ", for example:
{
"id": 36815684
}
But if I have something like this:
X-RateLimit-Reset: 1452786798
I get this error:
parse error: Invalid numeric literal at line 1, column 9
Do I need to fall back to sed/awk/perl .. or is there a more elegant way of using jq?
Apart from not using jq at all, you have two main options:
(1) pre-processing the non-JSON to make it JSON
(2) using the -R command-line option, e.g.
echo "X-RateLimit-Reset: 1452786798" | jq -R 'split(":")'
[
"X-RateLimit-Reset",
" 1452786798"
]
Thus, if you know the value is going to be numeric:
echo "X-RateLimit-Reset: 1452786798" |
jq -Rc 'split(":") | {(.[0]) : (.[1]|tonumber)}'
{"X-RateLimit-Reset":1452786798}
Note that although the "j" in jq is for JSON, jq (with the -R option) does just fine for text-processing.