I have an empty output.json and I want to populate it with {key, value} pairs where key is a string and value is a Json array read from file. I need to go through this for multiple files to populate the output.json. So far, value is being populated successfuly.
$ jq --argjson cves "$(cat my-scan-result-N.json)" '.+={"TODO": $cves}' output.json
{
"TODO": [
{
"cvePK": "CVE-2020-11656",
"summary": "In SQLite through 3.31.1, the ALTER TABLE implementation has a use-after-free, as demonstrated by an ORDER BY clause that belongs to a compound SELECT statement.",
"cvss": 7.5,
"notes": ""
},
{
"cvePK": "CVE-2019-19646",
"summary": "pragma.c in SQLite through 3.30.1 mishandles NOT NULL in an integrity_check PRAGMA command in certain cases of generated columns.",
"cvss": 7.5,
"notes": ""
}
]
}
However, when I add another --argjson to populate the key ("TODO") with desired value $FQDN, it fails with an error.
$ FQIN="example.com/foo/bar:7.0.3" # Tried \""example.com/foo/bar:7.0.3"\" as well but doesn't work.
$ jq --argjson cves "$(cat my-scan-result.json)" --argjson fqin="FQIN" '.+={$fqin: $cves}' output.json
C:\ProgramData\chocolatey\lib\jq\tools\jq.exe: invalid JSON text passed to --argjson
Use C:\ProgramData\chocolatey\lib\jq\tools\jq.exe --help for help with command-line options,
or see the jq manpage, or online docs at https://stedolan.github.io/jq
So my goal is to have something like below, but above error message is not helpful enough. Any help would be appreciated.
{
"example.com/foo/bar:7.0.3": [
{
"cvePK": "CVE-2020-11656",
"summary": "In SQLite through 3.31.1, the ALTER TABLE implementation has a use-after-free, as demonstrated by an ORDER BY clause that belongs to a compound SELECT statement.",
"cvss": 7.5,
"notes": ""
},
{
"cvePK": "CVE-2019-19646",
"summary": "pragma.c in SQLite through 3.30.1 mishandles NOT NULL in an integrity_check PRAGMA command in certain cases of generated columns.",
"cvss": 7.5,
"notes": ""
}
]
}
The line:
jq --argjson cves "$(cat my-scan-result.json)" --argjson fqin="FQIN" '.+={$fqin: $cves}' output.json
has several errors:
The phrase --argjson fqin="FQIN" is incorrect. Please see the jq manual for details. Suffice it to say here that you could achieve the desired effect by writing --arg fqin "$FQIN".
The jq expression {$fqin: $cves} is incorrect. When a key name is specified using a variable, the variable must be enclosed in parentheses: {($fqin): $cves}. (Indeed, whenever the key name is specified indirectly, the specifying expression must be enclosed in parentheses.)
You can do
FQIN="example.com/foo/bar:7.0." jq -n --argjson cves "$(cat my-scan-result.json)" '.+={(env.FQIN): $cves}'
Related
I'm new to jq (around 24 hours). I'm getting the filtering/selection already, but I'm wondering about advanced I/O features. Let's say I have an existing jq query that works fine, producing a stream (not a list) of objects. That is, if I pipe them to a file, it produces:
{
"id": "foo"
"value": "123"
}
{
"id": "bar"
"value": "456"
}
Is there some fancy expression I can add to my jq query to output each object individually in a subdirectory, keyed by the id, in the form id/id.json? For example current-directory/foo/foo.json and current-directory/bar/bar.json?
As #pmf has pointed out, an "only-jq" solution is not possible. A solution using jq and awk is as follows, though it is far from robust:
<input.json jq -rc '.id, .' | awk '
id=="" {id=$0; next;}
{ path=id; gsub(/[/]/, "_", path);
system("mkdir -p " path);
print >> path "/" id ".json";
id="";
}
'
As you will need help from outside jq anyway (see #peak's answer using awk), you also might want to consider using another JSON processor instead which offers more I/O features. One that comes to my mind is mikefarah/yq, a jq-inspired processor for YAML, JSON, and other formats. It can split documents into multiple files, and since its v4.27.2 release it also supports reading multiple JSON documents from a single input source.
$ yq -p=json -o=json input.json -s '.id'
$ cat foo.json
{
"id": "foo",
"value": "123"
}
$ cat bar.json
{
"id": "bar",
"value": "456"
}
The argument following -s defines the evaluation filter for each output file's name, .id in this case (the .json suffix is added automatically), and can be manipulated to further needs, e.g. -s '"file_with_id_" + .id'. However, adding slashes will not result in subdirectories being created, so this (from here on comparatively easy) part will be left over for post-processing in the shell.
I need to grab variables from JSON properties.
The JSON array looks like this (GitHub API for repository tags), which I obtain from a curl request.
[
{
"name": "my-tag-name",
"zipball_url": "https://api.github.com/repos/path-to-my-tag-name",
"tarball_url": "https://api.github.com/repos/path-to-my-tag-name-tarball",
"commit": {
"sha": "commit-sha",
"url": "https://api.github.com/repos/path-to-my-commit-sha"
},
"node_id": "node-id"
},
{
"name": "another-tag-name",
"zipball_url": "https://api.github.com/repos/path-to-my-tag-name",
"tarball_url": "https://api.github.com/repos/path-to-my-tag-name-tarball",
"commit": {
"sha": "commit-sha",
"url": "https://api.github.com/repos/path-to-my-commit-sha"
},
"node_id": "node-id"
},
]
In my actual JSON there are 100s of objects like these.
While I loop each one of these I need to grab the name and the commit URL, then perform more operations with these two variables before I get to the next object and repeat.
I tried (with and without -r)
tags=$(curl -s -u "${GITHUB_USERNAME}:${GITHUB_TOKEN}" -H "Accept: application/vnd.github.v3+json" "https://api.github.com/repos/path-to-my-repository/tags?per_page=100&page=${page}")
for row in $(jq -r '.[]' <<< "$tags"); do
tag=$(jq -r '.name' <<< "$row")
# I have also tried with the syntax:
url=$(echo "${row}" | jq -r '.commit.url')
# do stuff with $tag and $url...
done
But I get errors like:
parse error: Unfinished JSON term at EOF at line 2, column 0 jq: error
(at :1): Cannot index string with string "name" } parse error:
Unmatched '}' at line 1, column 1
And from the terminal output it appears that it is trying to parse $row in a strange way, trying to grab .name from every substring? Not sure.
I am assuming the output from $(jq '.[]' <<< "$tags") could be valid JSON, from which I could again use jq to grab the object properties I need, but maybe that is not the case? If I output ${row} it does look like valid JSON to me, and I tried pasting the results in a JSON validator, everything seems to check out...
How do I grab the ".name" and ".commit.url" for each of these object before I move onto the next one?
Thanks
It would be better to avoid calling jq more than once. Consider, for example:
while read -r name ; do
read -r url
echo "$name" "$url"
done < <( curl .... | jq -r '.[] | .name, .commit.url' )
where curl .... signifies the relevant invocation of curl.
Using jq, is it possible to replace the value of each parameter in the sample JSON with the value of the variable that is the initial value?
In my scenario, Azure DevOps does not carryout any kind of variable substitution on the JSON file, so I need to do it manually. So for example, say $SUBSCRIPTION_ID is set to abc-123, I'd like to use jq to update the JSON file.
I can pull out the values using .parameters[].value, but I can't seem to find a way of setting each individual value.
The main challenge here is that the solution should be reusable, and different JSON files will have different parameters, so I don't think I can use --argjson.
Example
Original JSON
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/parametersTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"subscriptionId": {
"value": "$SUBSCRIPTION_ID"
},
"topicName": {
"value": "$TOPIC_NAME"
}
}
}
Variables
SUBSCRIPTION_ID="abc-123"
TOPIC_NAME="SomeTopic"
Desired JSON
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/parametersTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"subscriptionId": {
"value": "abc-123"
},
"topicName": {
"value": "SomeTopic"
}
}
}
Export those variables so that you can access them from within jq.
export SUBSCRIPTION_ID TOPIC_NAME
jq '.parameters[].value |= (env[.[1:]] // .)' file
//. part is for leaving variables absent in the environment as is, you can drop it if not necessary
Use --argjson; essentially, you are just going to ignore the attempt at parameterizing the JSON and simply replace the values unconditionally.
jq --argjson x "$SUBSCRIPTION_ID" \
--argjson y "$TOPIC_NAME" \
'.parameters.subscriptionId.value = $x; .parameters.topicName.value = $y' \
config.json
Here is a "data-driven" approach based on the contents of the schema and the available environment variables:
export SUBSCRIPTION_ID="abc-123"
export TOPIC_NAME="SomeTopic"
< schema.json jq '.parameters
|= map_values(if .value | (startswith("$") and env[.[1:]])
then .value |= env[.[1:]] else . end)'
Notice that none of the template names appear in the jq program.
If your shell supports it, you could avoid the "export" commands by prefacing the jq command with the variable assignments along the lines of:
SUBSCRIPTION_ID="abc-123" TOPIC_NAME="SomeTopic" jq -f program.jq schema.json
Caveat
Using environment variables to pass in the parameter values may not be such a great idea. Two alternatives would be to provide the name-value pairs in a text file or as a JSON object. See also Using jq as a template engine
I'm trying to retrieve the last value inserted into a table in influxdb. What I need to do is then post it to another system via HTTP.
I'd like to do all this in a bash script, but I'm open to Python also.
$ curl -sG 'https://influx.server:8086/query' --data-urlencode "db=iotaWatt" --data-urlencode "q=SELECT LAST(\"value\") FROM \"grid\" ORDER BY time DESC" | jq -r
{
"results": [
{
"statement_id": 0,
"series": [
{
"name": "grid",
"columns": [
"time",
"last"
],
"values": [
[
"2018-01-17T04:15:30Z",
690.1
]
]
}
]
}
]
}
What I'm struggling with is getting this value into a clean format I can use. I don't really want to use sed, and I've tried jq but it complains the data is a string and not an index:
jq: error (at <stdin>:1): Cannot index array with string "series"
Anyone have a good suggestion?
Pipe that curl to the jq below
$ your_curl_stuff_here | jq '.results[].series[]|.name,.values[0][]'
"grid"
"2018-01-17T04:15:30Z"
690.1
The results could be stored into a bash array and used later.
$ results=( $(your_curl_stuff_here | jq '.results[].series[]|.name,.values[0][]') )
$ echo "${results[#]}"
"grid" "2018-01-17T04:15:30Z" 690.1
# Individual values could be accessed using "${results[0]}" and so, mind quotes
All good :-)
Given the JSON shown, the jq query:
.results[].series[].values[]
produces:
[
"2018-01-17T04:15:30Z",
690.1
]
This seems to be the output you want, but from the point of view of someone who is not familiar with influxdb, the requirements seem very opaque, so you might want to consider a variant, such as:
.results[-1].series[-1].values[-1]
which in this case produces the same result, as it happens.
If you just want the atomic values, you could simply append [] to either of the queries above.
Suppose I have a file with this JSON:
[
{
"label" : "deploy",
"pk" : 2388175,
"key" : "gsfd45"
},
{
"label" : "jenkins",
"key" : "eQtIAwP",
"pk" : 2388165
}
]
I want to get the value for key "pk" if it is in the hash that has label = "deploy".
How can I do this? Do I need to write a script?
To parse JSON in Bash, use jq!
$ jq '.[] | select(.label=="deploy").pk' file
2388175
If you want to store deploy in a variable, use --arg. From jq manual → Invoking jq:
--arg name value
This option passes a value to the jq program as a predefined variable. If you run jq with --arg foo bar, then $foo is available in the program and has the value "bar". Note that value will be treated as a string, so --arg foo 123 will bind $foo to "123".
$ v="deploy"
$ jq --arg var "$v" '.[] | select(.label==$var).pk' file
2388175
$ v="blabla"
$ jq --arg var "$v" '.[] | select(.label==$var).pk' file
# empty!
$ v="jenkins"
$ jq --arg var "$v" '.[] | select(.label==$var).pk' file
2388165
By pieces:
Print everything:
$ jq '.[]' file
{
"key": "gsfd45",
"pk": 2388175,
"label": "deploy"
}
{
"pk": 2388165,
"key": "eQtIAwP",
"label": "jenkins"
}
Print those records where label equals "deploy":
$ jq '.[] | select(.label=="deploy")' file
{
"key": "gsfd45",
"pk": 2388175,
"label": "deploy"
}
Print just the field pk in such case:
$ jq '.[] | select(.label=="deploy").pk' file
2388175
If jq was not availale on your server, python should be there, right? ^_*
#!/bin/python
import json
with open('data.json') as data_file:
data = json.load(data_file)
for d in data:
if d['label'] == 'deploy':
print(d["pk"])
assume your file named as data.json save it as id.py, and run with:
python id.py
It needs python3 installed on your system.
change the line print (d["pk"]) into print d["pk"] if you only have python2 installed.
The output would be:
2388175
Edit
added the if check, didn't notice OP wanted to check the label.
In awk. It's a bit incomplete but as you didn't have anything to show, you can work on this one:
$ awk -F: '$1~/"label"/{l=$2} l~/deploy/ && $1 ~ /pk/ {sub(/,/,"",$2);print $2}' file
2388175
When awk meets a record with "label" on it, it stores the $2. Once the pk is found and flag l has deploy in it, remove comma and print.
If the elegant solution provided by James Brown does not work (e.g. different ordering of the key/value pairs) here is something that tries to get at least the string between the braces into one record (by setting RS), then the string is splittet at the key value pair with key "pk" (by setting FS).
After that setup the pattern looks for the label/deploy key/value pair in $0 and then, only if there are two fields (e.g. the pk was present and a field split took place) the string after the comma in $2 is deleted and the value of key pk remains and is printed:
script.awk
BEGIN {
RS="[{}[\\]]"
FS="\"pk\"[^:]*:"
}
/"label"[^:]*:[^\"]*"deploy"/ {
if( NF == 2 ) {
# "pk" is present in $0, remove everything after comma
sub(/,.*/, "", $2)
print $2
}
}
You use this script with awk like this: awk -f script.awk yourfile.
I have only tried it with GNU awk, but RS and FS should also work with awk, too.