Save tree view json to file using jq library - json

I have like this json data in my case:
sshpass -p $PASSWORD ssh -o StrictHostKeyChecking=no $LOGIN /bin/bash -s << EOT
echo "Saving to file.."
USER_DATA="{\"id\":"$USER_ID",\"sha\":"$USER_SHA"}"
echo "{\"id\":"$PRODUCT_ID",\"user\":"$USER_DATA"}" | jq -re > ~/user.json
EOT
When I tried save this json to file get error with message:
parse error: Unmatched '}' at line 1, column 21
How I can save correctly my json data to file?

You should use jq to create your json. Try something like this :
sshpass -p $PASSWORD ssh -o StrictHostKeyChecking=no $LOGIN /bin/bash -s << EOT
echo "Saving to file.."
jq --arg id "$PRODUCT_ID" -n '{$id}' | \
jq --arg id "$USER_ID" \
--arg sha "$USER_SHA" \
--arg user "user" \
'.[$user]={$id, $sha}' > ~/user.json
EOT
Explanations:
--arg id "$PRODUCT_ID" : create a variable for jq, called id which contains the value taken from $PRODUCT_ID
-n '{$id}' : create a new json template
This will take the variable name as well as its value
Here, this creates the json {id : $id}
jq ... | jq ... : we pass the output of the 1st jq command as the input of the 2nd command
'.[$user]={$id, $sha}' : we add new elements to the input
.[$user] : the element name from the $user variable
{$id, $sha} : will create an object from the variables, i.e. {id : $id, sha : $sha}
the whole command creates the element : user : {id : $id, sha : $sha}

Related

bash & jq: add attribute with object value

I'm looking for a solution to add a new attribute with a JSON object value into an existing JSON file.
My current script:
if [ ! -f "$src_file" ]; then
echo "Source file $src_file does not exists"
exit 1
fi
if [ ! -f "$dst_file" ]; then
echo "Destination file $dst_file does not exists"
exit 1
fi
if ! jq '.devDependencies' "$src_file" >/dev/null 2>&1; then
echo "The key "devDependencies" does not exists into source file $src_file"
exit 1
fi
dev_dependencies=$(jq '.devDependencies' "$src_file" | xargs )
# Extract data from source file
data=$(cat $src_file)
# Add new key-value
data=$(echo $data | jq --arg key "devDependencies" --arg value "$dev_dependencies" '. + {($key): ($value)}')
# Write data into destination file
echo $data > $dst_file
It's working but the devDependencies value from $dev_dependencies is wrote as string:
"devDependencies": "{ #nrwl/esbuild: 15.6.3, #nrwl/eslint-pl[...]".
How can I write it as raw JSON ?
I think you want the --argjson option instead of --arg. Compare
$ jq --arg k '{"foo": "bar"}' -n '{x: $k}'
{
"x": "{\"foo\": \"bar\"}"
}
with
$ jq --argjson k '{"foo": "bar"}' -n '{x: $k}'
{
"x": {
"foo": "bar"
}
}
--arg will create a string variable. Use --argjson to parse the value as JSON (can be object, array or number).
From the docs:
--arg name value:
This option passes a value to the jq program as a predefined variable.
If you run jq with --arg foo bar, then $foo is available in the
program and has the value "bar". Note that value will be treated as a
string, so --arg foo 123 will bind $foo to "123".
Named arguments are also available to the jq program as $ARGS.named.
--argjson name JSON-text:
This option passes a JSON-encoded value to the jq program as a
predefined variable. If you run jq with --argjson foo 123, then $foo
is available in the program and has the value 123.
Note that you don't need multiple invocations of jq, xargs, command substitution or variables (don't forget to quote all your variables when expanding).
To "merge" the contents of two files, read both files with jq and let jq do the work. This avoids all the complications that arise from jumping between jq and shell context. A single line is all that's needed:
jq --slurpfile deps "$dep_file" '. + { devDependencies: $deps[0].devDependencies }' "$source_file" > "$dest_file"
or
jq --slurpfile deps "$dep_file" '. + ($deps[0]|{devDependencies})' "$source_file" > "$dest_file"
alternatively (still a one-liner):
jq --slurpfile deps "$dev_file" '.devDependencies = $deps[0].devDependencies' "$source_file" > "$dest_file"
peak's answer here reminded me of the very useful input filter, which can make the program even shorter as it avoids the variable:
jq '. + (input|{devDependencies})' "$source_file" "$dep_file" > "$dest_file"

Create Bash loop from JSON string using jq

I am writing a bash script that reads a JSON string then loop based on the JSON values to execute a CLI command.
#!/usr/bin/env bash
jq --version > /dev/null 2>&1 || { echo >&2 "jq is required but it's not installed. Aborting."; exit 1; }
read -r -d '' USER_ACTIONS << EOM
{
"user1": [
"action1"
],
"user2": [
"action2",
"action3"
]
}
EOM
USERS= #TODO
for user in USERS; do
ACTIONS= #TODO
for action in ACTIONS; do
echo "Executing ${command} ${user}-${action}"
done
done
If jq is present in the server, how do I populate the USERS and ACTIONS variable?
Depending on what command you want to execute, if it can be performed from within jq, it's easier to also move the loop inside. There are several ways to accomplish that. Here are some examples, all yielding the same output:
jq -r 'to_entries[] | "Executing command \(.key)-\(.value[])"' <<< "$USER_ACTIONS"
jq -r 'keys[] as $user | "Executing command \($user)-\(.[$user][])"' <<< "$USER_ACTIONS"
jq -r --stream '"Executing command \(.[0][0])-\(.[1]? // empty)"' <<< "$USER_ACTIONS"
Output:
Executing command user1-action1
Executing command user2-action2
Executing command user2-action3
It seems better to play with vanilla js (nodejs):
const myvar={
"user1": [
"action1"
],
"user2": [
"action2",
"action3"
]
};
for (let user in myvar) {
myvar[user].forEach((action) => {
console.log("Executing command " + user + "-" + action);
});
}
Output
Executing command user1-action1
Executing command user2-action2
Executing command user2-action3
Usage
node script.js
Usage with bash
You can remove Executing string, then:
node script.js | bash
Would you please try the following:
#!/bin/bash
declare -a users # array of users
declare -A actions # array of actions indexed by user
read -r -d '' user_actions << EOM
{
"user1": [
"action1"
],
"user2": [
"action2",
"action3"
]
}
EOM
while IFS=$'\t' read -r key val; do
users+=( "$key" ) # add the user to the array "users"
actions[$key]="$val" # associate the actions with the user
done < <(jq -r 'to_entries[] | [.key, .value[]] | #tsv' <<< "$user_actions")
for user in "${users[#]}"; do # loop over "users"
IFS=$'\t' read -r -a vals <<< "${actions[$user]}"
for action in "${vals[#]}"; do
echo yourcommand "$user" "$action"
done
done
Output:
yourcommand user1 action1
yourcommand user2 action2
yourcommand user2 action3
[Explanations]
The jq command outputs TSV which looks like:
user1\taction1
user2\taction2\taction3
where \t represents a tab character used as a field delimiter.
The first read builtin command assigns key to the first field and val
to the remaining field(s). If val contains two or more fields,
it will be splitted with tne next read builtin in the next loop.
if the output looks good, drop echo and replace the string yourcommand with the command name.

shell script-Unable to get expected response by running through loop inside .sh file, but getting it by individually defining variables in file

Fileread.sh
#!/bin/bash
s=ch.qos.logback
e=logback-access
curl -s "https://search.maven.org/solrsearch/select?q=g:$s+AND+a:$e&core=gav&rows=1&wt=json" | jq ".response.docs[].v"`
output:"1.2.11"
This code is working perfectly fine But when I try storing the s and e values in a .txt file with : seperated and then try running, I get nothing in response
textFile.txt
ch.qos.logback:logback-access
fileread.sh
#!/bin/bash
read -p "Enter file name:" filename
while IFS=':' read -r s e
do
curl -s "https://search.maven.org/solrsearch/select?q=g:${s}+AND+a:${e}&core=gav&rows=1&wt=json" | jq ".response.docs[].v"
done < $filename
I have tried :
xy=$(curl -s "https://search.maven.org/solrsearch/select?q=g:${s}+AND+a:${e}&core=gav&rows=1&wt=json" | jq ".response.docs[].v")
echo "$xy"
xy=$(curl -s "'https://search.maven.org/solrsearch/select?q=g:'${s}'+AND+a:'${e}&core=gav&rows=1&wt=json" | jq ".response.docs[].v")
echo "$xy"
url=`https://search.maven.org/solrsearch/select?q=g:${s}+AND+a:${e}&core=gav&rows=1&wt=json`
echo url
xx=`curl -s "$url" | jq ".response.docs[].v"`
echo $xx
Try this:
#!/bin/bash
echo "Enter file name:"
read filename
IFS=':' read -r s e < $filename
echo $s $e
curl -s "https://search.maven.org/solrsearch/select?q=g:${s}+AND+a:${e}&core=gav&rows=1&wt=json" | jq ".response.docs[].v"
~

How to store json key value pair in and store it in one variable in linux

I am calling python command which will return data is JSON key-value pair.
I have put python command and other command in one shell script named as - a.sh
Code (a.sh):
cd /home/drg/Code/dth
a=$(python3 main.py -z shell -y droub -i 56)
echo "$a"
When I am calling this script I am getting output as:
{'password': 'XYZ', 'name': 'Stguy', 'port': '5412', 'host': 'igtet', 'db_name': 'test3'}
And after getting this output I want to pass the output value like password, name to psql command to run postgresql query.
So, what I want is that I should be able to store password value in one variable, name in one variable like:
a= xyz
b=Stguy
p= port
So, that I can use this variables to pass in psql query as:
psql -h $a -p $p -U $b -d $db -c "CREATE SCHEMA IF NOT EXISTS ${sname,,};"
Can someone please help me with this?
Note: Env is linux(Centos 8)
Thanks in advance!
One way of solving this could be a combination of jq for value extraction and shell-builtin read for multiple variable assignment:
JSON='{"name": "Stguy", "port": 5412, "host": "igtet", "db_name": "test3"}'
read -r a b c <<<$( echo $JSON | jq -r '"\(.host) \(.port) \(.name)"' )
echo "a: $a, b: $b, c: $c"
doing jq string interpolation "\( )" to print result in one line
You can aslo go with sed or awk:
PSQL="$( python3 main.py -z shell -y droub -i 56 | sed "s/^[^:]*: *'\([^']*\)'[^:]*: *'\([^']*\)'[^:]*: *'\([^']*\)'[^:]*: *'\([^']*\)'[^:]*: *'\([^']*\)'}/psql -h '\4' -p '\1' -U '\2' -d '\5'/")"
[ "${PSQL:0:5}" = "psql " ] && ${PSQL} -c "CREATE SCHEMA IF NOT EXISTS ${sname,,};"
For security consideration, i urge you anyway to avoid passing account data (user passwd) through environment variables.
It would be better if your python script had an option to directly launch psql with required parameters.

Read JSON file & parse to get element values in shell script

I have a json file names test.json with the below content.
{
"run_list": ["recipe[cookbook-ics-op::setup_server]"],
"props": {
"install_home": "/test/inst1",
"tmp_dir": "/test/inst1/tmp",
"user": "tuser
}
}
I want to read this file into a variable in shell script & then extract the values of install_home,user & tmp_dir using expr. Can someone help, please?
props=cat test.json
works to get the json file into a variable. Now how can I extract the values using expr. Any help would be greatly appreciated.
Install jq
yum -y install epel-release
yum -y install jq
Get the values in the following way
install_home=$(cat test.json | jq -r '.props.install_home')
tmp_dir=$(cat test.json | jq -r '.props.tmp_dir')
user=$(cat test.json | jq -r '.props.user')
For a pure bash solution I suggest this:
github.com/dominictarr/JSON.sh
It could be used like this:
./json.sh -l -p < example.json
print output like:
["name"] "JSON.sh"
["version"] "0.2.1"
["description"] "JSON parser written in bash"
["homepage"] "http://github.com/dominictarr/JSON.sh"
["repository","type"] "git"
["repository","url"] "https://github.com/dominictarr/JSON.sh.git"
["bin","JSON.sh"] "./JSON.sh"
["author"] "Dominic Tarr <dominic.tarr#gmail.com> (http://bit.ly/dominictarr)"
["scripts","test"] "./all-tests.sh"
From here is pretty trivial achive what you are looking for
jq is a dedicated parser for JSON files. Install jq.
values in the json can be retrieved as:
jq .<top-level-attr>.<next-level-attr> <json-file-path>
if JSON contains an array
jq .<top-level-attr>.<next-level-array>[].<elem-in-array> <json-file-path>
if you want a value in a shell variable
id = $(jq -r .<top-level-attr>.<next-level-array>[].<next-level-attr> <json-file-path>)
echo id
use -r if you need unquoted value
For simple JSON, it may be treated as a plain text file.
In that case, we can use simple text pattern matching to extract the information we need.
If you observe the following lines:
"install_home": "/test/inst1",
"tmp_dir": "/test/inst1/tmp",
"user": "user"
There exists a pattern on each line that can be described as key and value:
"key" : "value"
We can use perl with regular expressions to exact the value for any given key:
"key" hardcoded for each case "install_home", "tmp_dir" and "user"
"value" as (.*) regular expression
Then we use the $1 matching group to retrieve the value.
i=$(perl -ne 'if (/"install_home": "(.*)"/) { print $1 . "\n" }' test.json)
t=$(perl -ne 'if (/"tmp_dir": "(.*)"/) { print $1 . "\n" }' test.json)
u=$(perl -ne 'if (/"user": "(.*)"/) { print $1 . "\n" }' test.json)
cat <<EOF
install_home: $i
tmp_dir : $t
user : $u
EOF
Which outputs:
install_home: /test/inst1
tmp_dir : /test/inst1/tmp
user : tuser