Modifying JSON by using jq - json

I want to modify a JSON file by using the Linux command line.
I tried these steps:
[root#localhost]# INPUT="dsa"
[root#localhost]# echo $INPUT
dsa
[root#localhost]# CONF_FILE=test.json
[root#localhost]# echo $CONF_FILE
test.json
[root#localhost]# cat $CONF_FILE
{
"global" : {
"name" : "asd",
"id" : 1
}
}
[root#localhost]# jq -r '.global.name |= '""$INPUT"" $CONF_FILE > tmp.$$.json && mv tmp.$$.json $CONF_FILE
jq: error: dsa/0 is not defined at <top-level>, line 1:
.global.name |= dsa
jq: 1 compile error
Desired output:
[root#localhost]# cat $CONF_FILE
{ "global" : {
"name" : "dsa",
"id" : 1 } }

Your only problem was that the script passed to jq was quoted incorrectly.
In your particular case, using a single double-quoted string with embedded \-escaped " instances is probably simplest:
jq -r ".global.name = \"$INPUT\"" "$CONF_FILE" > tmp.$$.json && mv tmp.$$.json "$CONF_FILE"
Generally, however, chepner's helpful answer shows a more robust alternative to embedding the shell variable reference directly in the script: Using the --arg option to pass a value as a jq variable allows single-quoting the script, which is preferable, because it avoids confusion over what elements are expanded by the shell up front and obviates the need for escaping $ instances that should be passed through to jq.
Also:
Just = is sufficient to assign the value; while |=, the so-called update operator, works too, it behaves the same as = in this instance, because the RHS is a literal, not an expression referencing the LHS - see the manual.
You should routinely double-quote your shell-variable references and you should avoid use of all-uppercase variable names in order to avoid conflicts with environment variables and special shell variables.
As for why your quoting didn't work:
'.global.name |= '""$INPUT"" is composed of the following tokens:
String literal .global.name |= (due to single-quoting)
String literal "" - i.e., the empty string - the quotes will be removed by the shell before jq sees the script
An unquoted reference to variable $INPUT (which makes its value subject to word-splitting and globbing).
Another instance of literal "".
With your sample value, jq ended up seeing the following string as its script:
.global.name |= dsa
As you can see, the double quotes are missing, causing jq to interpret dsa as a function name rather than a string literal, and since no argument was passed to (non-existent) function dsa, jq's error message referenced it as dsa/0 - a function with no (0) arguments.

It's much simpler and safer to pass the value using the --arg option:
jq -r --arg newname "$INPUT" '.global.name |= $newname' "$CONF_FILE"
This ensures that the exact value of $INPUT is used and quoted as a JSON value.

Using jq with a straight forward filter, should do it for you.
.global.name = "dsa"
i.e.
jq '.global.name = "dsa"' json-file
{
"global": {
"name": "dsa",
"id": 1
}
}
You can play around with your json-filters, here.

Related

Passing a path ("key1.key2") from a bash variable to jq

I am having trouble accessing bash variable inside 'jq'.
The snippet below shows my bash loop to check for missing keys in a Json file.
#!/bin/sh
for key in "key1" "key2.key3"; do
echo "$key"
if ! cat ${JSON_FILE} | jq --arg KEY "$key" -e '.[$KEY]'; then
missingKeys+=${key}
fi
done
JSON_FILE:
{
"key1": "val1",
"key2": {
"key3": "val3"
}
}
The script works correctly for top level keys such as "key1". But it does not work correctly (returns null) for "key2.key3".
'jq' on the command line does return the correct value
cat input.json | jq '.key2.key3'
"val3"
I followed answers from other posts to come to this solution. However can't seem to figure out why it does not work for nested json keys.
Using --arg prevents your data from being incorrectly parsed as syntax. Usually, a shell variable you're passing into jq contains literal data, so this is the correct thing.
In this case, your variable contains syntax, not literal data: The . isn't part of the string you want to do a lookup by, but is instead an instruction to jq to do two separate lookups one after the other.
So, in this case, you should do the more obvious thing, instead of using --arg:
jq -e ".$KEY"

jq truncates ENV variable after whitespace

Trying to write a bash script that replaces values in a JSON file we are running into issues with Environment Variables that contain whitespaces.
Given an original JSON file.
{
"version": "base",
"myValue": "to be changed",
"channelId": 0
}
We want to run a command to update some variables in it, so that after we run:
CHANNEL_ID=1701 MY_VALUE="new value" ./test.sh
The JSON should look like this:
{
"version": "base",
"myValue": "new value",
"channelId": 1701
}
Our script is currently at something like this:
#!/bin/sh
echo $MY_VALUE
echo $CHANNEL_ID
function replaceValue {
if [ -z $2 ]; then echo "Skipping $1"; else jq --argjson newValue \"${2}\" '. | ."'${1}'" = $newValue' build/config.json > tmp.json && mv tmp.json build/config.json; fi
}
replaceValue channelId ${CHANNEL_ID}
replaceValue myValue ${MY_VALUE}
In the above all values are replaced by string and strings are getting truncated at whitespace. We keep alternating between this issue and a version of the code where substitutions just stop working entirely.
This is surely an issue with expansions but we would love to figure out, how we can:
- Replace values in the JSON with both strings and values.
- Use whitespaces in the strings we pass to our script.
You don't have to mess with --arg or --argjson to import the environment variables into jq's context. It can very well read the environment on its own. You don't need a script separately, just set the values along with the invocation of jq
CHANNEL_ID=1701 MY_VALUE="new value" \
jq '{"version": "base", myValue: env.MY_VALUE, channelId: env.CHANNEL_ID}' build/config.json
Note that in the case above, the variables need not be exported globally but just locally to the jq command. This allows you to not export multiple variables into the shell and pollute the environment, but just the ones needed for jq to construct the desired JSON.
To make the changes back to the original file, do > tmp.json && mv tmp.json build/config.json or more clearly download the sponge(1) utility from moreutils package. If present, you can pipe the output of jq as
| sponge build/config.json
Pass variables with --arg. Do:
jq --arg key "$1" --arg value "$2" '.[$key] = $value'
Notes:
#!/bin/sh indicates that this is posix shell script, not bash. Use #!/bin/bash in bash scripts.
function replaceValue { is something from ksh shell. Prefer replaceValue() { to declare functions. Bash obsolete and deprecated syntax.
Use newlines in your script to make it readable.
--argjson passes a json formatted argument, not a string. Use --arg for that.
\"${2}\" doesn't quote $2 expansion - it only appends and suffixes the string with ". Because the expansion is not qouted, word splitting is performed, which causes your input to be split on whitespaces when creating arguments for jq.
Remember to quote variable expansions.
Use http://shellcheck.net to check your scripts.
. | means nothing in jq, it's like echo $(echo $(echo))). You could jq '. | . | . | . | . | .' do it infinite number of times - it passes the same thing. Just write the thing you want to do.
Do:
#!/bin/bash
echo "$MY_VALUE"
echo "$CHANNEL_ID"
replaceValue() {
if [ -z "$2" ]; then
echo "Skipping $1"
else
jq --arg key "$1" --arg value "$2" '.[$key] = $value' build/config.json > tmp.json &&
mv tmp.json build/config.json
fi
}
replaceValue channelId "${CHANNEL_ID}"
replaceValue myValue "${MY_VALUE}"
#edit Replaced ."\($key)" with easier .[$key]
jq allows you to build new objects:
MY_VALUE=foo;
CHANNEL_ID=4
echo '{
"version": "base",
"myValue": "to be changed",
"channelId": 0
}' | jq ". | {\"version\": .version, \"myValue\": \"$MY_VALUE\", \"channelId\": $CHANNEL_ID}"
The . selects the whole input, and inputs that (|) to the construction of a new object (marked by {}). For version is selects .version from the input, but you can set your own values for the other two. We use double quotes to allow the Bash variable expansion, which means escaping the double quotes in the JSON.
You'll need to adapt my snippet above to scriptify it.

Select and replace field of key referenced by environment variable with special character using jq

Given the json below:
./versions.json
{
"my-app-1": "v1.0.0",
"my-app-2": "v0.9.1",
"my-app-3": "v2.1.7"
}
I want to replace the version of $APP_NAME to the new version $NEW_VERSION. Given APP_NAME=my-app-2 and NEW_VERSION=v1.0.0, I tried the following:
jq '(."$APP_NAME") = "$NEW_VERSION"' ./versions.json > updated_versions.json
which gives:
./updated_versions.json
{
"my-app-1": "v1.0.0",
"my-app-2": "v0.9.1",
"my-app-3": "v2.1.7",
"$APP_NAME": "$NEW_VERSION"
}
this:
jq "(.$APP_NAME) = \"$NEW_VERSION\"" versions.json > updated_versions.json
gives:
jq: error: app/0 is not defined at <top-level>, line 1:
(.my-app-1) = "v1.0.0"
jq: 1 compile error
How can I escape the special character in the environment variable? I have tried setting APP_NAME=my\\-app\\-1 with no luck.
Thanks
Pass the two variables to jq using its --arg option :
jq --arg appName my-app-2 --arg newVersion v1.0.0 '.[$appName]=$newVersion'
In your first try the jq command was enclosed in single-quotes, so it was left to jq to resolve the variables, but it doesn't look for them in the outer shell context.
Your second try was nearly good (but not very good practice) because the variables were expanded by the shell, but my-app-1 contains special characters (the dashes) and needs to be accessed with either ."my-app-1" or .["my-app-1"].

Dynamically add key value pair in jSON object using shell

I have a json object named version6json as follows
{
"20007.098": {
"os_version": "6.9",
"kernel": "2.6.32-696",
"sfdc-release": "2017.08"
},
"200907.09678”: {
"os_version": "6.9",
"kernel": "2.6.32-696",
"sfdc-release": "201.7909"
},
"206727.1078”: {
"os_version": "6.9",
"kernel": "2.6.32-696.10.2.el6.x86_64",
"sfdc-release": "20097.109”
}
}
I want to add one more key value pair. The key is also a variable and the value too. bundle_release="2019.78" and value= {"release":"2018.1006","kernel":"2.6.32-754.3.5.el6.x86_64","os":"6.10","current":true}
Now I want the bundle_release as key and value as its value, So the new entry would be "2018.1006": {"release":"2018.1006","kernel":"2.6.32-754.3.5.el6.x86_64","os":"6.10","current":true}
To achieve this, I am doing the folllowing
echo "$version6json" | jq --arg "$bundle_release" "$value" '. + {$bundle_release: "${value}"}'
Any help will be appriciated.
P.S- The question is edited as suggested by peak
First, when specifying a key name using a variable in the way you are doing, the variable must be parenthesized, so you would have:
{($bundle_release): ...}
Next, jq variables are not the same as shell variables and should be specified without quoting them, and without using bash-isms.
Third, when setting the value of the shell variable named value, you would have to quote the expression appropriately.
Fourth, to simplify things, use --argjson for $value.
Fifth, your sample JSON is not quite right. Once it's fixed, the following will work in a bash or bash-like environment (assuming you're using a version of jq that supports --argjson):
bundle_release="1034,567"
value='{"release":"2018.1006","kernel":"2.6.32-754.3.5.el6.x86_64","os":"6.10","current":true}'
jq --arg b "$bundle_release" --argjson v "$value" '
. + {($b): $v}' <<< "$version6json"
You're not giving the --arg option enough parameters: from the manual:
--arg name value:
This option passes a value to the jq program as a predefined variable. If you run jq with --arg foo bar, then
$foo is available in the program and has the value "bar". Note that value will be treated as a string, so
--arg foo 123 will bind $foo to "123".

Match a key-value pattern in JSON text and get the value for another key

Suppose I have a file with this JSON:
[
{
"label" : "deploy",
"pk" : 2388175,
"key" : "gsfd45"
},
{
"label" : "jenkins",
"key" : "eQtIAwP",
"pk" : 2388165
}
]
I want to get the value for key "pk" if it is in the hash that has label = "deploy".
How can I do this? Do I need to write a script?
To parse JSON in Bash, use jq!
$ jq '.[] | select(.label=="deploy").pk' file
2388175
If you want to store deploy in a variable, use --arg. From jq manual → Invoking jq:
--arg name value
This option passes a value to the jq program as a predefined variable. If you run jq with --arg foo bar, then $foo is available in the program and has the value "bar". Note that value will be treated as a string, so --arg foo 123 will bind $foo to "123".
$ v="deploy"
$ jq --arg var "$v" '.[] | select(.label==$var).pk' file
2388175
$ v="blabla"
$ jq --arg var "$v" '.[] | select(.label==$var).pk' file
# empty!
$ v="jenkins"
$ jq --arg var "$v" '.[] | select(.label==$var).pk' file
2388165
By pieces:
Print everything:
$ jq '.[]' file
{
"key": "gsfd45",
"pk": 2388175,
"label": "deploy"
}
{
"pk": 2388165,
"key": "eQtIAwP",
"label": "jenkins"
}
Print those records where label equals "deploy":
$ jq '.[] | select(.label=="deploy")' file
{
"key": "gsfd45",
"pk": 2388175,
"label": "deploy"
}
Print just the field pk in such case:
$ jq '.[] | select(.label=="deploy").pk' file
2388175
If jq was not availale on your server, python should be there, right? ^_*
#!/bin/python
import json
with open('data.json') as data_file:
data = json.load(data_file)
for d in data:
if d['label'] == 'deploy':
print(d["pk"])
assume your file named as data.json save it as id.py, and run with:
python id.py
It needs python3 installed on your system.
change the line print (d["pk"]) into print d["pk"] if you only have python2 installed.
The output would be:
2388175
Edit
added the if check, didn't notice OP wanted to check the label.
In awk. It's a bit incomplete but as you didn't have anything to show, you can work on this one:
$ awk -F: '$1~/"label"/{l=$2} l~/deploy/ && $1 ~ /pk/ {sub(/,/,"",$2);print $2}' file
2388175
When awk meets a record with "label" on it, it stores the $2. Once the pk is found and flag l has deploy in it, remove comma and print.
If the elegant solution provided by James Brown does not work (e.g. different ordering of the key/value pairs) here is something that tries to get at least the string between the braces into one record (by setting RS), then the string is splittet at the key value pair with key "pk" (by setting FS).
After that setup the pattern looks for the label/deploy key/value pair in $0 and then, only if there are two fields (e.g. the pk was present and a field split took place) the string after the comma in $2 is deleted and the value of key pk remains and is printed:
script.awk
BEGIN {
RS="[{}[\\]]"
FS="\"pk\"[^:]*:"
}
/"label"[^:]*:[^\"]*"deploy"/ {
if( NF == 2 ) {
# "pk" is present in $0, remove everything after comma
sub(/,.*/, "", $2)
print $2
}
}
You use this script with awk like this: awk -f script.awk yourfile.
I have only tried it with GNU awk, but RS and FS should also work with awk, too.