I need a jq command to replace array values with non-json standard values i.e. in the following json input I need to replace the "webOrigins" array values with a non-json value, which is a Jinja2 template variable replacement as per the second json block below.
Input (example.json)
{
"clients": [
{
"clientId": "abc",
"webOrigins": [ "/", "/api" ]
},
{
"clientId": "xyz",
"webOrigins": [ ]
}
]
}
Desired Output
{
"clients": [
{
"clientId": "abc",
"webOrigins": {{clients.abc.webOrigins}}
},
{
"clientId": "xyz",
"webOrigins": {{clients.xyz.webOrigins}}
}
]
}
My current attempt of shell script calling jq to loop through the input json and replace with template variable is this
for clientId in $(jq -r '.clients[] | .clientId' example.json)
do
jq '(.clients[] | select(.clientId == "'${clientId}'") | .webOrigins) |= {{ clients['\'${clientId}\''].webOrigins | default([]) }}' example.json > tmp.j2; mv -f tmp.j2 example.json
done
But fails with the error:
jq: error: syntax error, unexpected '{' (Unix shell quoting issues?) at <top-level>, line 1:
(.clients[] | select(.clientId == "abc") | .webOrigins) |= {{ clients['abc'].webOrigins | default([]) }}
jq: 1 compile error
Of course if I add double quotes around the template variable to make the replacement variable valid json the script works but I need the value to not have quotes.
It would be impractical to implement a jq-only solution, but jq can certainly help, e.g. the following works in your case:
jq '.clients[] |= (.webOrigins = "{{clients.\(.clientId).webOrigins}}")' |
sed '/"webOrigins":/ { s/"[{][{]/{{/; s/[}][}]"$/}}/; }'
Related
I'm trying to parse some json with jq that looks like this:
{
"currentServerId": 1,
"isCurrentServerAvailable": true,
"isClusterOwner": false,
"clusterOwnerServerId": "2",
"numInvalidEventMessages": "0",
"numInvalidoperationalStateMessages": "0",
"servers": {
"1": {
"isAvailableViaServerObjective": true,
"eventMessagesPendingInDb": "0",
"neo4jClusterRole": "slave",
"monitoredServerHealth": null
},
"2": {
"isAvailableViaServerObjective": true,
"eventMessagesPendingInDb": "0",
"neo4jClusterRole": "master",
"monitoredServerHealth": {
"serverId": 2,
"healthState": "ALIVE",
"healthCount": 1,
"serverIdAsString": "2",
"serverUid": "*"
}
},
"3": {
"isAvailableViaServerObjective": true,
"eventMessagesPendingInDb": "0",
"neo4jClusterRole": "slave",
"monitoredServerHealth": null
}
}
}
I can obtain the value I need with a filter like this:
# cat test.json | jq .'servers."2".monitoredServerHealth.healthState'
"ALIVE"
If I pass in a single variable as a key it works:
# echo $var
servers
# cat json.tst | jq --arg keyvar "$var" '.[$keyvar]'
{
"1": {
"isAvailableViaServerObjective": true,
"eventMessagesPendingInDb": "0",
"neo4jClusterRole": "slave",
"monitoredServerHealth": null
},
<omitted the rest for brevity>
When I try to do the same for a nested block (where one level of the key is from a variable), it fails. I've tried a number of variations but they all fail.
What is the correct syntax?
# echo $var
"2"
# cat test.json | jq --arg keyvar "$var" '.servers.$[keyvar].monitoredServerHealth.healthState'
jq: error: syntax error, unexpected '$', expecting FORMAT or QQSTRING_START (Unix shell quoting issues?) at <top-level>, line 1:
.servers.$[keyvar].monitoredServerHealth.healthState
jq: 1 compile error
Split up the path into two parts:
$ jq --arg keyvar "$var" '.servers | .[$keyvar].monitoredServerHealth.healthState' input.json
"ALIVE"
From the documentation:
A filter of the form .foo.bar is equivalent to .foo|.bar.
I don't know why it doesn't work combined as .servers.[$keyvar]; might be a parser bug or something. At least the workaround is trivial.
Another way uses string interpolation:
jq --arg keyvar "$var" '.servers."\($keyvar)".monitoredServerHealth.healthState' input.json
With the following input file:
{
"events": [
{
"mydata": {
"id": "123456",
"account": "21234"
}
},
{
"mydata": {
"id": "123457",
"account": "21234"
}
}
]
}
When I run it through this JQ filter,
jq ".events[] | [.mydata.id, .mydata.account]" events.json
I get a set of arrays:
[
"123456",
"21234"
]
[
"123457",
"21234"
]
When I put this output through the #csv filter to create CSV output:
jq ".events[] | [.mydata.id, .mydata.account] | #csv" events.json
I get a CSV file with one record per row:
"\"123456\",\"21234\""
"\"123457\",\"21234\""
I would like CSV file with two records per row, like this:
"123456","21234"
"123457","21234"
What am I doing wrong?
Use the -r flag.
Here is the explanation in the manual:
--raw-output / -r: With this option, if the filter's result is a string then it will be written directly to standard output rather than
being formatted as a JSON string with quotes.
jq -r '.events[] | [.mydata.id, .mydata.account] | #csv'
Yields
"123456","21234"
"123457","21234"
I have the given JSON and want to change the id value of all elements, which starts with test in the name element:
{
"other-value": "some-id",
"values": [
{
"name": "test-2017-12-01",
"id": "1"
},
{
"name": "othert",
"id": "2"
}
]
}
The following jq commands works jqplay
jq (.values[] | select(.name == "test-afs").id) |= "NEWID"
But when I try it with startswith it stops working, what am I missing? jqplay
(.values[] | select(.name | startswith("test")).id) |= "NEWID"
jq: error (at :14): Invalid path expression near attempt to access element "id" of {"name":"test-afs","id":"id"}
exit status 5
You can also use map, like this:
jq '(.values)|=(map((if .name|startswith("test") then .id="NEWID" else . end)))' file
Output:
{
"other-value": "some-id",
"values": [
{
"name": "test-2017-12-01",
"id": "NEWID"
},
{
"name": "othert",
"id": "2"
}
]
}
Please note that since the release of jq 1.5, jq has been enhanced to support the query that previously failed. For example, using the current 'master' version:
jq -c '(.values[] | select(.name | startswith("test")).id) |= "NEWID"'
{"other-value":"some-id","values":[{"name":"test-2017-12-01","id":"NEWID"},{"name":"othert","id":"2"}]}
Using earlier versions of jq, if/then/else/end can be used in this type of situation as follows:
.values[] |= if .name | startswith("test") then .id = "NEWID" else . end
If using map, a minimalist expression would be:
.values |= map(if .name|startswith("test") then .id = "NEWID" else . end)
I have JSON like this that I'm parsing with jq:
{
"data": [
{
"item": {
"name": "string 1"
},
"item": {
"name": "string 2"
},
"item": {
"name": "string 3"
}
}
]
}
...and I'm trying to get "string 1" "string 2" and "string 3" into a Bash array, but I can't find a solution that ignores the whitespace in them. Is there a method in jq that I'm missing, or perhaps an elegant solution in Bash for it?
Current method:
json_names=$(cat file.json | jq ".data[] .item .name")
read -a name_array <<< $json_names
The below assume your JSON text is in a string named s. That is:
s='{
"data": [
{
"item1": {
"name": "string 1"
},
"item2": {
"name": "string 2"
},
"item3": {
"name": "string 3"
}
}
]
}'
Unfortunately, both of the below will misbehave with strings containing literal newlines; since jq doesn't have support for NUL-delimited output, this is difficult to work around.
On bash 4 (with slightly sloppy error handling, but tersely):
readarray -t name_array < <(jq -r '.data[] | .[] | .name' <<<"$s")
...or on bash 3.x or newer (with very comprehensive error handling, but verbosely):
# -d '' tells read to process up to a NUL, and will exit with a nonzero exit status if that
# NUL is not seen; thus, this causes the read to pass through any error which occurred in
# jq.
IFS=$'\n' read -r -d '' -a name_array \
< <(jq -r '.data[] | .[] | .name' <<<"$s" && printf '\0')
This populates a bash array, contents of which can be displayed with:
declare -p name_array
Arrays are assigned in the form:
NAME=(VALUE1 VALUE2 ... )
where NAME is the name of the variable, VALUE1, VALUE2, and the rest are fields separated with characters that are present in the $IFS (input field separator) variable.
Since jq outputs the string values as lines (sequences separated with the new line character), then you can temporarily override $IFS, e.g.:
# Disable globbing, remember current -f flag value
[[ "$-" == *f* ]] || globbing_disabled=1
set -f
IFS=$'\n' a=( $(jq --raw-output '.data[].item.name' file.json) )
# Restore globbing
test -n "$globbing_disabled" && set +f
The above will create an array of three items for the following file.json:
{
"data": [
{"item": {
"name": "string 1"
}},
{"item": {
"name": "string 2"
}},
{"item": {
"name": "string 3"
}}
]
}
The following shows how to create a bash array consisting of arbitrary JSON texts produced by a run of jq.
In the following, I'll assume input.json is a file with the following:
["string 1", "new\nline", {"x": 1}, ["one\ttab", 4]]
With this input, the jq filter .[] produces four JSON texts -- two JSON strings, a JSON object, and a JSON array.
The following bash script can then be used to set x to be a bash array of the JSON texts:
#!/bin/bash
x=()
while read -r value
do
x+=("$value")
done < <(jq -c '.[]' input.json)
For example, adding this bash expression to the script:
for a in "${x[#]}" ; do echo a="$a"; done
would yield:
a="string 1"
a="new\nline"
a={"x":1}
a=["one\ttab",4]
Notice how (encoded) newlines and (encoded) tabs are handled properly.
I have the following in a file (which I will call "myfile"):
[{
"id": 123,
"name": "John",
"aux": [{
"abc": "random",
"def": "I want this"
}],
"blah": 23.11
}]
I could parse it if the file did not have the first [ and last ] as follows:
$ cat myfile | jq -r '.aux[] | .def'
I want this
$
but with the [ and ] I get:
$ cat myfile | jq -r '.aux[] | .def'
jq: error: Cannot index array with string
How can I deal with the [ and ] using jq? (I'm sure I could parse them off with a different tool but I want to learn correct usage of jq.
It should be:
jq '.[].aux[].def' file.json
.[] iterates over the outer array, .aux[] then iterates over the the aux array of every node and .def prints their .def property.
This will output:
"I want this"
If you want to get rid of the double quotes pass -r (--raw) to jq:
jq -r '.[].aux[].def' file.json
Output:
I want this