I have a json object I am reading with jq and trying to write some properties with local variables.
I am setting local variable in my shell script like so:
LOCATION_NAME="stag5"
DOMAIN_LOCATION="example.io"
I am then building out the following variable:
echo "Build New ID"
DOMAIN_NAME_BUILT="$LOCATION_NAME.$DOMAIN_LOCATION.s3-website.us-east-2.amazonaws.com"
I am trying to read my distconfig.json file and set the properties with the above variables.
tmp=$(mktemp)
jq '.Origins.Items[0].DomainName = "$DOMAIN_NAME_BUILT"' distconfig.json > "$tmp" && mv "$tmp" distconfig.json
The command is working, but it is passing in the variable as a string to my new json file.
So when I view the property in the new created json file it is being saved as "$DOMAIN_NAME_BUILT" instead of stag5.example.io.s3-website.us-east-2.amazonaws.com
How can I instead of passing the string pass the variable for $DOMAIN_NAME_BUILT and write it to the new json file
Use the --argjson option instead of parameter interpolation.
jq --argjson dnb "$DOMAIN_NAME_BUILT" \
'.Origins.Items[0].DomainName = $dnb' distconfig.json > "$tmp" &&
mv "$tmp" distconfig.json
(Your immediate issue is that parameter expansion doesn't occur inside single quotes, but building a static filter that takes an argument is safer than building a filter dynamically.)
Related
I am writing script (bash script for Azure pipeline) and I need to combine JSON from different variables. For example, I have:
TYPE='car'
COLOR='blue'
ADDITIONAL_PARAMS='{"something": "big", "etc":"small"}'
So, as you can see, I have several string variables and one which consist JSON.
I need to combine these variables with this format (and I cant :( ):
some_script --extr-vars --extra_vars '{"var_type": "'$TYPE'", "var_color": "'$COLOR'", "var_additional_data": "'$ADDITIONAL_PARAMS'"}'
But this combination is not working, I have a string something like:
some_script --extr-vars --extra_vars '{"var_type": "car", "var_color": "blue", "var_additional_data": " {"something": "big", "etc":"small"} "}'
which is not correct and valid JSON.
How I can combine existing JSON (already formatted with double quotes ") with other variables? I am using bash / console / yq utilite (to convert yaml to json)
Use jq to generate the JSON. (You can probably do this in one step with yq, but I'm not as familiar with that tool.)
ev=$(jq --arg t "$TYPE" \
--arg c "$COLOR" \
--argjson ap "$ADDITIONAL_PARAMS" \
-n '{var_type: $t, var_color: $c var_additional_data: $ap}')
some_script --extr-vars --extra_vars "$ev"
This question already has answers here:
Exporting JSON to environment variables
(7 answers)
Closed 1 year ago.
I am trying to read the json file and store the keys as bash variable and the values as variable value. For instance here is my test.json:
{
"Day":'0',
"num":'85',
"Qtr":"[20183, 20184, 20191, 20192]",
"pSize":"75"
}
I need to store the variables like this in a bash file:
$Day=0
$num=85
$Qtr=[20183, 20184, 20191, 20192]
$psize=75
I found a way to extract the value using jq, but I am unable to store the key as variable in bash
-bash-4.2$ Qtr=`jq .Qtr test.json`
-bash-4.2$ echo $Qtr
"[20183, 20184, 20191, 20192]"
Could someone please provide a snippet as to how to loop through a json file and store the key as variables and values as values?
Thank you in advance
Would you please try the following:
#!/bin/bash
while IFS== read key value; do
printf -v "$key" "$value"
done < <(jq -r 'to_entries|map("\(.key)=\(.value|tostring)")|.[]' test.json)
# test of the variables
echo "Day: $day"
echo "Qtr: $Qtr"
echo "num: $num"
echo "pSize: $pSize"
The jq command jq -r 'to_entries|map("\(.key)=\(.value|tostring)")|.[]' test.json converts the json object into key=value pairs.
The output of jq is fed to the while loop via the process substitution
<(command).
The -v varname option to the printf command assigns the variable
indirecly expressed with varname to the output.
I have a json in which I want to modify a particular value but the terminal always displays the json with the modified value but it does not actually change the value in the particular file. Sample json:
{
name: 'abcd',
age: 30,
address: 'abc'
}
I want to change the value of address in the file itself but so far I've been unable to do so. I tried using:
jq '.address = "abcde"' test.json
but it didn't work. Any suggestions?
Use a temporary file; it's what any program that claims to do in-place editing is doing.
tmp=$(mktemp)
jq '.address = "abcde"' test.json > "$tmp" && mv "$tmp" test.json
If the address isn't hard-coded, pass the correct address via a jq argument:
address=abcde
jq --arg a "$address" '.address = $a' test.json > "$tmp" && mv "$tmp" test.json
AFAIK jq does not support in-place editing, so you must redirect to a temporary file first and then replace your original file with it, or use sponge utility from the moreutils package, like that:
jq '.address = "abcde"' test.json|sponge test.json
There are other techniques to "redirect to the same file", like saving your output in a variable e.t.c. "Unix & Linux StackExchange" is a good place to start, if you want to learn more about this.
Temp files add more complexity when not needed (unless you are truly dealing with JSON files so large you cannot fit them in memory (GB to 100's of GB or TB, depending on how much RAM/parallelism you have)
The Pure bash way.
contents="$(jq '.address = "abcde"' test.json)" && \
echo -E "${contents}" > test.json
Pros
No temp file to juggle
Pure bash
Don't need an admin to install sponge, which is not installed by default
Simpler
Cons
This works perfectly fine for json because it cannot contain a literal null character. If you were to try this outside the json arena, it would fail when a null is encountered (and you would have to do some encoding/decoding workarounds). Bash variables cannot store literal nulls.
Note: this can not be combined as "one command" (like #codekandis
suggested), since redirection sometimes starts before the left hand side (LHS) of an expression is run, and starting redirection before running jq erroneously empties the file, hence two separate commands. It may "seem" to work when you try it, but this is misleading and has a very high probability of failing as soon as the circumstances change.
Update: Added -E option to disable escape characters just in case you are on systems where they are interpreted by default. (Which I've never actually seen)
Just to add to chepner answer and if you want it in a shell script.
test.json
{
"name": "abcd",
"age": 30,
"address": "abc"
}
script.sh
#!/bin/bash
address="abcde"
age=40
# Strings:
jq --arg a "${address}" '.address = $a' test.json > "tmp" && mv "tmp" test.json
# Integers:
jq --argjson a "${age}" '.age = $a' test.json > "tmp" && mv "tmp" test.json
Example for nested json with changing single and multiple values.
config.json
{
"Parameters": {
"Environment": "Prod",
"InstanceType": "t2.micro",
"AMIID": "ami-02d8e11",
"ConfigRegion": "eu-west-1"
}
}
with the below command, you can edit multiple values.
tmp=$(mktemp)
jq '.Parameters.AMIID = "ami-02d8sdfsdf" | .Parameters.Environment = "QA"' config.json > "$tmp" && mv "$tmp" config.json
with the below command, you can edit single value.
tmp=$(mktemp)
jq '.Parameters.AMIID = "ami-02d8sdfsdf"' config.json > "$tmp" && mv "$tmp" config.json
this should work
address = aaaaa
echo $(jq --arg a "$address" '.address = ($a)' test.json) > test.json
for whatever reason, without the echo, it makes a bin file and my python script was not able to parse it.
I took the best of a couple answers here and here.
This uses a parameter named actionname as an input to an assignment of the name property at the document level. ACTION_NAME is just an envvar I want to use as the replacement value.
contents="$(jq --arg actionname ${ACTION_NAME} '.name = $actionname' ./${ACTION_NAME}/package.json)" && \
echo -E "${contents}" > ${ACTION_NAME}/package.json;
I didn't like any of the solutions and created the sde utility.
pip install sde
Then, e.g. for the following JSON data:
{
"Parameters": {
"Environment": "Prod",
"InstanceType": "t2.micro",
"AMIID": "ami-02d8e11",
"ConfigRegion": "eu-west-1"
}
}
you can simply do:
sde Parameters.Environment Dev test.json
I want to modify a JSON file by using the Linux command line.
I tried these steps:
[root#localhost]# INPUT="dsa"
[root#localhost]# echo $INPUT
dsa
[root#localhost]# CONF_FILE=test.json
[root#localhost]# echo $CONF_FILE
test.json
[root#localhost]# cat $CONF_FILE
{
"global" : {
"name" : "asd",
"id" : 1
}
}
[root#localhost]# jq -r '.global.name |= '""$INPUT"" $CONF_FILE > tmp.$$.json && mv tmp.$$.json $CONF_FILE
jq: error: dsa/0 is not defined at <top-level>, line 1:
.global.name |= dsa
jq: 1 compile error
Desired output:
[root#localhost]# cat $CONF_FILE
{ "global" : {
"name" : "dsa",
"id" : 1 } }
Your only problem was that the script passed to jq was quoted incorrectly.
In your particular case, using a single double-quoted string with embedded \-escaped " instances is probably simplest:
jq -r ".global.name = \"$INPUT\"" "$CONF_FILE" > tmp.$$.json && mv tmp.$$.json "$CONF_FILE"
Generally, however, chepner's helpful answer shows a more robust alternative to embedding the shell variable reference directly in the script: Using the --arg option to pass a value as a jq variable allows single-quoting the script, which is preferable, because it avoids confusion over what elements are expanded by the shell up front and obviates the need for escaping $ instances that should be passed through to jq.
Also:
Just = is sufficient to assign the value; while |=, the so-called update operator, works too, it behaves the same as = in this instance, because the RHS is a literal, not an expression referencing the LHS - see the manual.
You should routinely double-quote your shell-variable references and you should avoid use of all-uppercase variable names in order to avoid conflicts with environment variables and special shell variables.
As for why your quoting didn't work:
'.global.name |= '""$INPUT"" is composed of the following tokens:
String literal .global.name |= (due to single-quoting)
String literal "" - i.e., the empty string - the quotes will be removed by the shell before jq sees the script
An unquoted reference to variable $INPUT (which makes its value subject to word-splitting and globbing).
Another instance of literal "".
With your sample value, jq ended up seeing the following string as its script:
.global.name |= dsa
As you can see, the double quotes are missing, causing jq to interpret dsa as a function name rather than a string literal, and since no argument was passed to (non-existent) function dsa, jq's error message referenced it as dsa/0 - a function with no (0) arguments.
It's much simpler and safer to pass the value using the --arg option:
jq -r --arg newname "$INPUT" '.global.name |= $newname' "$CONF_FILE"
This ensures that the exact value of $INPUT is used and quoted as a JSON value.
Using jq with a straight forward filter, should do it for you.
.global.name = "dsa"
i.e.
jq '.global.name = "dsa"' json-file
{
"global": {
"name": "dsa",
"id": 1
}
}
You can play around with your json-filters, here.
I'm working on parsing JSON data using JSON.sh. And I wanted to read data from json file (test.json) whose content will be something like,
{
"/home/ukrishnan/projects/test.yml": {
"LOG_DRIVER": "syslog",
"IMAGE": "mysql:5.6"
},
"/home/ukrishnan/projects/mysql/app.xml": {
"ENV_ACCOUNT_BRIDGE_ENDPOINT": "/u01/src/test/sample.txt"
}
}
And I try to parse this JSON using JSON.sh by using,
test_parser=`sh ./lib/JSON.sh < test/test.json`
echo $test_parser
It prints,
["/home/ukrishnan/projects/test.yml","LOG_DRIVER"] "syslog" ["/home/ukrishnan/projects/test.yml","IMAGE"] "mysql:5.6" ["/home/ukrishnan/projects/test.yml"] {"LOG_DRIVER":"syslog","IMAGE":"mysql:5.6"} ["/home/ukrishnan/projects/mysql/app.xml","ENV_ACCOUNT_BRIDGE_ENDPOINT"] "/u01/src/test/sample.txt" ["/home/ukrishnan/projects/mysql/app.xml"] {"ENV_ACCOUNT_BRIDGE_ENDPOINT":"/u01/src/test/sample.txt"} [] {"/home/ukrishnan/projects/test.yml":{"LOG_DRIVER":"syslog","IMAGE":"mysql:5.6"},"/home/ukrishnan/projects/mysql/app.xml":{"ENV_ACCOUNT_BRIDGE_ENDPOINT":"/u01/src/test/sample.txt"}}
Whereas, the same command (sh ./lib/JSON.sh < test/test.json), if I run through terminal, it is printing with line breaks,
["/home/ukrishnan/projects/test.yml","LOG_DRIVER"] "syslog"
["/home/ukrishnan/projects/test.yml","IMAGE"] "mysql:5.6"
["/home/ukrishnan/projects/test.yml"] {"LOG_DRIVER":"syslog","IMAGE":"mysql:5.6"}
["/home/ukrishnan/projects/mysql/app.xml","ENV_ACCOUNT_BRIDGE_ENDPOINT"] "/u01/src/test/sample.txt"
["/home/ukrishnan/projects/mysql/app.xml"] {"ENV_ACCOUNT_BRIDGE_ENDPOINT":"/u01/src/test/sample.txt"}
[] {"/home/ukrishnan/projects/test.yml":{"LOG_DRIVER":"syslog","IMAGE":"mysql:5.6"},"/home/ukrishnan/projects/mysql/app.xml":{"ENV_ACCOUNT_BRIDGE_ENDPOINT":"/u01/src/test/sample.txt"}}
I wanted to read this and assign to bash variables like,
file_name='/home/ukrishnan/projects/test.yml'
key='LOG_DRIVER'
value='syslog'
As I'm almost completely new to shell script and grep or awk, I don't have much idea of how to achieve this. Any help on this would be greatly appreciated.
I wrote a JSON serializer / deserializer for gawk, if you're interested. Save that script and modify it, replacing everything above # === FUNCTIONS === with the following:
#!/usr/bin/gawk -f
# capture JSON string from beginning to end into a scalar variable
{ json = json ORS $0 }
END {
# objectify JSON string to the multilevel array "obj"
deserialize(json, obj)
for (filename in obj) {
print "file_name=" quote(filename)
for (key in obj[filename]) {
# print key="value"
print key "=" quote(obj[filename][key])
}
}
}
Do chmod 755 json.awk and execute it. Output will resemble this:
$ ./json.awk test5.json
file_name="/home/ukrishnan/projects/mysql/app.xml"
ENV_ACCOUNT_BRIDGE_ENDPOINT="/u01/src/test/sample.txt"
file_name="/home/ukrishnan/projects/test.yml"
LOG_DRIVER="syslog"
IMAGE="mysql:5.6"
Hopefully the logic is reasonably easy to follow. If you prefer to output filename=, key=, and value= on every loop iteration, modify the nested for loops accordingly:
for (filename in obj) {
for (key in obj[filename]) {
print "file_name=" quote(filename)
print "key=" quote(key)
print "value=" quote(obj[filename][key])
}
}
That change will result in the following output:
$ ./json.awk test5.json
file_name="/home/ukrishnan/projects/mysql/app.xml"
key="ENV_ACCOUNT_BRIDGE_ENDPOINT"
value="/u01/src/test/sample.txt"
file_name="/home/ukrishnan/projects/test.yml"
key="LOG_DRIVER"
value="syslog"
file_name="/home/ukrishnan/projects/test.yml"
key="IMAGE"
value="mysql:5.6"
Anyway, with that output, you can do something silly in BASH like this to populate and act upon the variables:
#!/bin/bash
./test.awk test5.json | while read -r line; do {
eval $line
[ "${line/=*/}" = "value" ] && {
echo "bash: file_name=$file_name"
echo "bash: key=$key"
echo "bash: value=$value"
echo "------"
}
}; done
It'd probably be more graceful just to do all processing within gawk from start to finish and not mess with the polyglot handoff, though.
Getting back to json.awk, if you prefer to keep json.awk modular for easy reuse in future projects, you could remove everything above # === FUNCTIONS ===, create a separate main.awk containing the code block at the top of this answer, and #include "json.awk" as a helper library pretty much anywhere outside of END {...} (just below the shbang, for example).
JSON.sh (from http://json.org) offers a nice bash friendly means of flattening out a JSON file. Which you've already provided how it looks in your question. So, the flatten form is the format:
[node] tab value
You have to think in UNIX script in extracting the information you want, you'll note the lines you're interested in actually follow this pattern:
["filename","key"] tab ["value"]
In regex notation, we replace:
filename with (.*)
key with (.*)
tab with \t
value with (.*)
We can retrieve the first, second and third matching groups with \1, \2, \3 respectively.
When used in sed we also note that these symbols []() need to be escaped with a backslash \, resulting in the following script:
./lib/JSON.sh < test/test.json | sed 's/\["\(.*\)","\(.*\)\"]\t"\(.*\)"/\1,\2,\3/;t;d'
/home/ukrishnan/projects/test.yml,LOG_DRIVER,syslog
/home/ukrishnan/projects/test.yml,IMAGE,mysql:5.6
/home/ukrishnan/projects/mysql/app.xml,ENV_ACCOUNT_BRIDGE_ENDPOINT,/u01/src/test/sample.txt
Now we put the lines in a loop and for each line, we can extract out filename,key,value:
for line in $(./lib/JSON.sh < test/test.json | sed 's/\["\(.*\)","\(.*\)\"]\t"\(.*\)"/\1,\2,\3/;t;d')
do
IFS="," read -ra arr <<< $line
filename=${arr[0]}
key=${arr[1]}
value=${arr[2]}
cat <<EOF
filename : $filename
key : $key
value : $value
EOF
done
Which outputs:
filename : /home/ukrishnan/projects/test.yml
key : LOG_DRIVER
value : syslog
filename : /home/ukrishnan/projects/test.yml
key : IMAGE
value : mysql:5.6
filename : /home/ukrishnan/projects/mysql/app.xml
key : ENV_ACCOUNT_BRIDGE_ENDPOINT
value : /u01/src/test/sample.txt