I understand that jq search needs to be blocked by {} and the key needs to be encased with ", for example:
{
"id": 36815684
}
But if I have something like this:
X-RateLimit-Reset: 1452786798
I get this error:
parse error: Invalid numeric literal at line 1, column 9
Do I need to fall back to sed/awk/perl .. or is there a more elegant way of using jq?
Apart from not using jq at all, you have two main options:
(1) pre-processing the non-JSON to make it JSON
(2) using the -R command-line option, e.g.
echo "X-RateLimit-Reset: 1452786798" | jq -R 'split(":")'
[
"X-RateLimit-Reset",
" 1452786798"
]
Thus, if you know the value is going to be numeric:
echo "X-RateLimit-Reset: 1452786798" |
jq -Rc 'split(":") | {(.[0]) : (.[1]|tonumber)}'
{"X-RateLimit-Reset":1452786798}
Note that although the "j" in jq is for JSON, jq (with the -R option) does just fine for text-processing.
Related
I have a json file test.json with the content:
[
{
"name": "Akshay",
"id": "234"
},
{
"name": "Amit",
"id": "28"
}
]
I have a shell script with content:
#!/bin/bash
function display
{
echo "name is $1 and id is $2"
}
cat test.json | jq '.[].name,.[].id' | while read line; do display $line; done
I want name and id of a single item to be passed together as arguments to the function display but the output is something like this :
name is "Akshay" and id is
name is "Amit" and id is
name is "234" and id is
name is "28" and id is
What should be the correct way to implement the code?
PS: I specifically want to use jq so please base the answer in terms of jq
Two major issues, and some additional items that may not matter for your current example use case but can be important when you're dealing with real-world data from untrusted sources:
Your current code iterates over all names before writing any ids.
Your current code uses newline separators, but doesn't make any effort to read multiple lines into each while loop iteration.
Your code uses newline separators, but newlines can be present inside strings; consequently, this is constraining the input domain.
When you pipe into a while loop, that loop is run in a subshell; when the pipeline exits, the subshell does too, so any variables set by the loop are lost.
Starting up a copy of /bin/cat and making jq read a pipe from its output is silly and inefficient compared to letting jq read from test.json directly.
We can fix all of those:
To write names and ids in pairs, you'd want something more like jq '.[] | (.name, .id)'
To read both a name and an id for each element of the loop, you'd want while IFS= read -r name && IFS= read -r id; do ... to iterate over those pairs.
To switch from newlines to NULs (the NUL being the only character that can't exist in a C string, or thus a bash string), you'd want to use the -j argument to jq, and then add explicit "\u0000" elements to the content being written. To read this NUL-delimited content on the bash side, you'd need to add the -d '' argument to each read.
To move the while read loop out of the subshell, we can use process substitution, as described in BashFAQ #24.
To let jq read directly from test.json, use either <test.json to have the shell connect the file directly to jq's stdin, or pass the filename on jq's command line.
Doing everything described above in a manner robust against input data containing JSON-encoded NULs would look like the following:
#!/bin/bash
display() {
echo "name is $1 and id is $2"
}
cat >test.json <<'EOF'
[
{ "name": "Akshay", "id": "234" },
{ "name": "Amit", "id": "28" }
]
EOF
while IFS= read -r -d '' name && IFS= read -r -d '' id; do
display "$name" "$id"
done < <(jq -j '
def stripnuls: sub("\u0000"; "<NUL>");
.[] | ((.name | stripnuls), "\u0000", (.id | stripnuls), "\u0000")
' <test.json)
You can see the above running at https://replit.com/#CharlesDuffy2/BelovedForestgreenUnits#main.sh
You can use string interpolation.
jq '.[] | "The name is \(.name) and id \(.id)"'
Result:
"The name is Akshay and id 234"
"The name is Amit and id 28"
"The name is hi and id 28"
If you want to get rid of the double-quotes from each object, then:
jq --raw-output '.[] | "The name is \(.name) and is \(.id)"'
https://jqplay.org/s/-lkpHROTBk0
This question already has answers here:
How to use jq when the variable has reserved characters?
(3 answers)
Closed 1 year ago.
I tried with jq to parse some JSON output inside my GitLab CI pipeline so I can extract needed information. I've tried many different ways, but I can't get the desired information out of the target node, because it has special characters and when I get to that node, the pipeline fails in each case. This is the current state of my pipeline.
This is the problematic job:
get results (dev branch):
stage: Results of scanning image
variables:
RESULTS: ""
STATUS: ""
SEVERITY: ""
image: alpine
only:
refs:
- dev
allow_failure: true
before_script:
- apk update && apk upgrade
- apk --no-cache add curl
- apk add jq
script:
- 'RESULTS=$(curl -H "Authorization: Basic `echo -n ${HARBOR_USER}:${HARBOR_PASSWORD} | base64`" -X GET "https://url.to.registry/api/v2.0/projects/project/repositories/repo-name/artifacts/latest?page=1&page_size=10&with_tag=true&with_label=true&with_scan_overview=true&with_signature=true&with_immutable_status=true")'
- echo $RESULTS
- RESULTS=$RESULTS | tr 'application/vnd.scanner.adapter.vuln.report.harbor+json; ' 'myobject'
- echo $RESULTS
- "STATUS=$RESULTS | jq '.scan_overview .myobjectversion=1.0 .scan_status'"
- "SEVERITY=$RESULTS | jq '.scan_overview .myobjectversion=1.0 .severity'"
- echo "Printing the results of the image scanning process on Harbor registry:"
- echo "status of scan:$STATUS"
- echo "severity of scan:$SEVERITY"
- echo "For more information of scan results please visit Harbor registry!"
tags:
- dev
- docker
This is the JSON output that I get from the curl command:
{
"addition_links":{
"build_history":{
"absolute":false,
"href":"..."
},
"vulnerabilities":{
"absolute":false,
"href":"...."
}
},
"digest":"sha256:bcd665be2b7c6725b410029db385d7c6c71a9ce557427cbd0f54d01a9",
"extra_attrs":{
"architecture":"amd64",
"author":null,
"created":"2021-10-22T10:28:46.058276455Z",
"os":"linux"
},
"icon":"sha256:0048162a053ee7518615bef084403614f8bca43b40ae2e762e11e06",
"id":362,
"labels":null,
"manifest_media_type":"application/vnd.docker.distribution.manifest.v2+json",
"media_type":"application/vnd.docker.container.image.v1+json",
"project_id":3,
"pull_time":"2021-10-22T10:28:55.305Z",
"push_time":"2021-10-22T10:28:49.341Z",
"references":null,
"repository_id":12,
"scan_overview":{
"application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0":{
"complete_percent":100,
"duration":8,
"end_time":"2021-10-22T10:28:57.356Z",
"report_id":"e83854eb-2304-4c58-85c9-a3e0fd9067a8",
"scan_status":"Success",
"severity":"Critical",
"start_time":"2021-10-22T10:28:49.827Z",
"summary":{
"summary":{
"Critical":7,
"High":47,
"Low":18,
"Medium":47
},
"total":119
}
}
}
}
My initial idea was to using jq to extract scan_status and severity with this command:
RESULTS=$RESULTS | jq '.scan_overview .application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0 .scan_status'
after running that command I got this error:
jq: error: syntax error, unexpected ';', expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.scan_overview .application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0 .scan_status
jq: 1 compile error
Now I am trying text replacement, but that doesn't work either.
How should I proceed in this case?
Use square brackets and double quotes around the problematic key:
jq '.scan_overview["application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0"].scan_status'
Also, this doesn't do what you think:
RESULTS=$RESULTS | tr 'application/vnd.scanner.adapter.vuln.report.harbor+json; ' 'myobject'
First of all, tr doesn't replace strings, it replaces characters. sed can replace strings.
Moreover, the pipe | is used when the left hand side produces output. Variable assignment doesn't produce any output.
Finally, to assign the output of a command to a variable, you need to use Command Substitution:
var=$(command)
~~ ~
Wrap the key in question inside barckets and quotes like so:
.scan_overview["application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0"].scan_status
Demo
From a shell syntax perspective, this looks wrong:
RESULTS=$RESULTS | jq '...'
Assigning a variable produces no output, so jq on the other side of the pipe has no input.
If it is okay for you to ignore the name of the problematic key, then you can substitute it with empty brackets:
RESULTS=$RESULTS | jq '.scan_overview [] .scan_status'
This is a bit sloppy and may match more than you would like, but in your narrow example, it will successfully pull out the value of .scan_status.
The reason this works, in this case, is because the application/vnd... property is the only property in the scan_overview object.
A more confident match would be achievable by using more quotes:
RESULTS=$RESULTS | jq '.scan_overview ."application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0" .scan_status'
I have file that looks like this:
$ cat sample-test.json |jq .
{
"logRef": "c4fa4367-23f6-462f-b5fd-f972d0916a30",
"timestamp": 1563268297545,
"someOtherField": "nonImportantValue"
}
{
"logRef": "c4fa4367-23f6-462f-b5fd-f972d0916a31",
"timestamp": 1563268297595,
"someOtherField2": "nonImportantValue3"
}
And I would like to convert it to csv like this:
logRef;timestamp
c4fa4367-23f6-462f-b5fd-f972d0916a30;1563268297545
c4fa4367-23f6-462f-b5fd-f972d0916a31;1563268297595
I was trying
$ cat sample-test.json |jq '.logRef, .timestamp |#csv'
jq: error (at <stdin>:1): string ("c4fa4367-2...) cannot be csv-formatted, only array
jq: error (at <stdin>:2): string ("c4fa4367-2...) cannot be csv-formatted, only array
Your input is fine (it's a JSON stream).
The problem with your filter is that #csv expects an array. So this will work:
[.logRef,.timestamp] | #csv
However it quotes strings, so if you want your strings unquoted (which might mean the result won't be CSV), then you could use:
"\(.logRef),\(.timestamp)"
In all cases, you'll need to use jq's-r command-line option.
The problem in your json file. Looks like it has incorrect format (without root array element [] and commas between documents). If you fix it, jq will work as expected.
> cat sample-test.json
[{
"logRef": "c4fa4367-23f6-462f-b5fd-f972d0916a30",
"timestamp": 1563268297545,
"someOtherField": "nonImportantValue"
},
{
"logRef": "c4fa4367-23f6-462f-b5fd-f972d0916a31",
"timestamp": 1563268297595,
"someOtherField2": "nonImportantValue3"
}]
cat sample-test.json |jq -r 'map(.logRef), map(.timestamp) | #csv'
"c4fa4367-23f6-462f-b5fd-f972d0916a30","c4fa4367-23f6-462f-b5fd-f972d0916a31"
1563268297545,1563268297595
I've also fixed the command with map() function.
I'm trying to extract a series of properties (named in an input file) in jq and getting error when I feed those from bash via a loop:
while read line; do echo $line; cat big.json | jq ".$line"; sleep 1; done < big.properties.service
cfg.keyload.service.count
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
When i try to do it manually it works
$ line=cfg.keyload.service.count
$ echo $line
cfg.keyload.service.count
$ cat big.json | jq ".$line"
1
Is there any way to get it work in loop?
Here is example
cat >big.json <<EOF
{
"cfg": {
"keyload": {
"backend": {
"app": {
"shutdown": {
"timeout": "5s"
},
"jmx": {
"enable": true
}
}
}
}
}
}
EOF
cat >big.properties.service <<EOF
cfg.keyload.backend.app.shutdown.timeout
cfg.keyload.backend.app.jmx.enable
cfg.keyload.backend.app.jmx.nonexistent
cfg.nonexistent
EOF
...output should be:
cfg.keyload.backend.app.shutdown.timeout
"5s"
cfg.keyload.backend.app.jmx.enable
true
cfg.keyload.backend.app.jmx.nonexistent
null
cfg.nonexistent
null
Immediate Issue - Invalid Input
The "invalid character" at hand here is almost certainly a carriage return. Use dos2unix to convert your input file to a proper UNIX text file, and your original code will work (albeit very inefficiently, rereading your whole big.json every time it wants to extract a single property).
Performant Implementation - Loop In JQ, Not Bash
Don't use a bash loop for this at all -- it's much more efficient to have jq do the looping.
Note the sub("\r$"; "") used in this code to remove trailing carriage returns so it can accept input in DOS format.
jq -rR --argfile infile big.json '
sub("\r$"; "") as $keyname
| ($keyname | split(".")) as $pieces
| (reduce $pieces[] as $piece ($infile; .[$piece]?)) as $value
| ($keyname, ($value | tojson))
' <big.properties.service
properly emits as output, when given the inputs in the question:
cfg.keyload.backend.app.shutdown.timeout
"5s"
cfg.keyload.backend.app.jmx.enable
true
cfg.keyload.backend.app.jmx.nonexistent
null
cfg.nonexistent
null
Your properties file is effectively paths in the json that you want to retrieve values from. Convert them to paths that jq recognizes so you can get those values. Just make an array of keys that would need to be traversed. Be sure to read your properties file as raw input (-R) since it's not json, and use raw output (-r) to be able to output the paths as you want.
$ jq --argfile big big.json '
., (split(".") as $p | $big | getpath($p) | tojson)
' -Rr big.properties.service
cfg.keyload.backend.app.shutdown.timeout
"5s"
cfg.keyload.backend.app.jmx.enable
true
cfg.keyload.backend.app.jmx.nonexistent
null
cfg.nonexistent
null
I have a large amount of JSON from a 3rd party system which I would like to pre-process with jq, but I am having difficulty composing the query, test case follows:
$ cat test.json
{
"a": "b",
"c": "d",
"e": {
"1": {
"f": "g",
"h": "i"
}
}
}
$ cat test.json|jq .e.1.f
jq: error: Invalid numeric literal at EOF at line 1, column 3 (while parsing '.1.') at <top-level>, line 1:
.e.1.f
How would I get "g" as my output here? Or how do I cast that 1 to a "1" so it is handled correctly?
From jq manual :
You can also look up fields of an object using syntax like .["foo"]
(.foo above is a shorthand version of this, but only for
identifier-like strings).
You also need quotes and use -r if you want raw output :
jq -r '.e["1"].f' test.json
I wrote a shell script function that calls the curl command, and pipes it into the jq command.
function getName {
curl http://localhost:123/getname/$1 | jq;
}
export -f getName
When I ran this from the CLI,
getName jarvis
I was getting this response:
parse error: Invalid numeric literal at line 1, column 72
I tried removing the | jq from the curl command, and I got back the result without jq parsing:
<Map><timestamp>1234567890</timestamp><status>404</status><error>Not Found</error><message>....
I first thought that I had a bad character in the curl command, or that I was using the function param $1 wrong.
Then I counted the number of chars in the result string, and I noticed that the 72nd char in that string was the empty space between "Not Found".
The underlying issue was that I didn't have a method named getname yet in my spring REST controller, so the response was coming back 404 Not Found. But in addition, jq wasn't handling the empty space in the response except by outputting the error message.
I'm new to jq so maybe there is a way to get around the empty space issue, but that's for another day.