Update values in json with jq (shell script) - json

I am trying to update a couple values in a json array (separate file) with a shell script. Basically the logic is, set an environment variable called URL, ping that URL and add it to the json-- if 0, update another json field to SUCCESS, else update to FAILED.
Here is are the files:
info.json:
{
"name": "PingTest",",
"metrics": [
{
"event_type": "PingResult",
"provider": "test",
"providerUrl": "URL",
"providerResult": "RESULT"
}
]
}
pinger.sh:
#!/bin/sh
JSON=`cat info.json` #read in JSON
#Assume URL variable is set to www.dksmfdkf.com
ping -q -c 1 "$URL" > /dev/null #ping url
if [ $? -eq 0 ]; then #if ping success, replace result in json template
JSON=`echo ${JSON} | jq --arg v "$URL" '.metrics[].providerUrl |= $v'
info.json`
JSON=`echo ${JSON} | jq '.metrics[].providerResult |= "SUCCESS"' info.json`
else
JSON=`echo ${JSON} | jq --arg v "$URL" '.metrics[].providerUrl |= $v'
info.json`
JSON=`echo ${JSON} | jq '.metrics[].providerResult |= "FAILED"' info.json`
fi
#Remove whitespace from json
JSON=`echo $JSON | tr -d ' \t\n\r\f'`
#Print the result
echo "$JSON"
The problem is my json file isn't getting updated properly, example result when running:
home:InfraPingExtension home$ ./pinger.sh
ping: cannot resolve : Unknown host
{
"name": "PingTest",
"metrics": [
{
"event_type": "PingResult",
"provider": "test",
"providerUrl": "",
"providerResult": "RESULT"
}
]
}
{
"name": "PingTest",
"metrics": [
{
"event_type": "PingResult",
"provider": "test",
"providerUrl": "URL",
"providerResult": "FAILED"
}
]
}
{"name":"PingTest","metrics":[{"event_type":"PingResult","provider":"test","providerUrl":"URL","providerResult":"RESULT"}]}

This would be greatly simplified by only calling jq once.
host=${URL#http://}; host=${host#https://}; host=${host%%/*}
if ping -q -c 1 "$host"; then
result=SUCCESS
else
result=FAILED
fi
JSON=$(
jq -c \
--arg url "$URL" \
--arg result "$result" \
'.metrics[].providerUrl |= $url
| .metrics[].providerResult |= $result
' info.json
)

Related

how to extract one filed from curl response

I wanna extract one file from my curl response and jq doesn't work for me.
here is my response:
response = {"response":[{"id":"0-0","enabled":true}]}
I wanna extract the response decision (for the above case the decision is true as the value of "enabled" is true). any suggestions? thanks.
I tried
echo $response | grep -o '"enabled":"[^"]*' | grep -o '[^"]*$'
jq -r .enabled <<< $respons
for jq: command not found
#!/bin/bash
response="{\"response\":[{\"id\":\"0-0\",\"enabled\":true}]}"
echo "$response" | jq '.response[0].enabled'
Will return true.
Read up on jq here: http://www.compciv.org/recipes/cli/jq-for-parsing-json/
How I built the jq string:
$ echo "$response" | jq '.'
{
"response": [
{
"id": "0-0",
"enabled": true
}
]
}
$ echo "$response" | jq '.response'
[
{
"id": "0-0",
"enabled": true
}
]
$ echo "$response" | jq '.response[0]'
{
"id": "0-0",
"enabled": true
}
$ echo "$response" | jq '.response[0].enabled'
true

Construct json through jq tool in shell script loop

json can be created using the following command.
jq -n \
--arg v1 "Value1" \
--arg v2 "Value2" \
'{k1: "$v1", k2:$v2'}
But when my key is mutable, how should I loop? For example, the script I execute is
test.sh k1=v1 k2=v2 k3=v3
test.sh is as follows
index=1
while ((index <= "$#")); do
data_i_key=$(echo ${!index} | awk -F "=" '{print $1}')
data_i_value=$(echo ${!index} | awk -F "=" '{print $2}')
let index++
JSON_STRING=$(jq -n \
--arg value "$data_i_value" \
'{'"$data_i_key"': $value'})
echo $JSON_STRING
The above print result is
{ "K3": "V3" }
if I replace it with
JSON_STRING+=$(jq -n \
--arg val_value "$dataValue" \
'{'"$data_i_key"': $val_value'})
The print result is
{ "k1": "v1" }{ "k2": "v2" }{ "K3": "V3" }
The above two methods have not achieved the goal, do you have a good way to deal with it? My desired output is
{ "k1": "v1" , "k2": "v2" ,"K3": "V3" }
hope you can help me.
I'd suggest a totally different, but simpler, approach:
for kv; do
echo "$kv" | jq -R './"=" | {key:first,value:last}'
done | jq -s 'from_entries'
It builds {key: …, value: …} objects from your positional parameters (splitting them by the equal sign) and then slurping all those objects in a second jq process, converting them into a single object via from_entries.
Alternatively, using -n (--null-input), --args and then accessing via $ARGS.positional. This avoids the loop and the second jq call altogether.
jq -n '$ARGS.positional | map(./"=" | {key:first,value:last}) | from_entries' --args "$#"
If your values can contain = themselves, you have to join all but the first value of the array:
jq -n '$ARGS.positional
| map(./"=" | {key:first,value:.[1:]|join("=")})
| from_entries' --args "$#"
Use parentheses around the key expression to have it evaluated:
jq -n --arg k1 "Key1" --arg v1 "Value1" '{($k1): $v1}'
{
"Key1": "Value1"
}
Perhaps this, which will handle an arbitrary number of key=value arguments:
#!/usr/bin/env bash
args=()
for arg; do
IFS="=" read -r k v <<<"$arg"
args+=("$k" "$v")
done
jq -n --args '
$ARGS.positional as $a
| reduce range(0; $a | length; 2) as $i ({}; .[$a[$i]] = $a[$i + 1])
' "${args[#]}"
Produces:
$ ./test.sh k1=v1 k2=v2 k3=v3
{
"k1": "v1",
"k2": "v2",
"k3": "v3"
}

How to iterate each object in json array jq [duplicate]

This question already has answers here:
Iterating through JSON array in Shell script
(10 answers)
Closed last year.
Intention: Need to iterate the each item in json array using jq and proccess seperately
api returns data in form
{
"name": [
{
"first": "first",
"class": false,
"type": "B"
},
{
"first": "second",
"class": false,
"type": "B"
},
{
"first": "third",
"class": false,
"type": "A"
}
]
}
And i am able to parse it as
data=`curl http://some.ur;l`
echo "$data" | jq -rc '.name[] | select(.type=="B") | .class=true'
it returns data as
{"first": "first","class": true, "type": "B"}
{"first": "second", "class": true, "type": "B"}
Now i want to proccess these two outputs in such a way so that i can make a PUT call for each of them . I tried to learn some concepts of xargs but could not make it done
I piped the output to | xargs -n1 but it removed all the quotes from the string
You could loop through using while read:
while IFS= read -r line
do
# your PUT goes here
# dummy
printf 'PUTting JSON: %s\n' "$line"
done < <(jq -c '.name[] | select(.type == "B") | .class = true')
If you absolutely need to use xargs then use a newline character as delimiter
jq -c '.name[] | select(.type == "B") | .class = true' \
| xargs -d $'\n' printf 'PUTting JSON: %s\n' # dummy
The only part you're missing is a while read loop.
data=$(curl ...)
while IFS= read -r line; do
curl -XPUT -d"$line" http://some_url
done < <(jq -c ... <<<"$data")
...or...
curl ... |
jq -c ... |
xargs -d $'\n' -n 1 curl -XPUT http://some_url -d
Note:
xargs -d $'\n' tells xargs to treat only newlines (and not other whitespace) as separators between items; it also turns off treatment of quotes and backslashes as syntactic rather than literal.
xargs -n 1 tells xargs only to pass one data item to each copy of curl.
Making the last argument to curl be -d means that xargs will place the data immediately after that position.

Replacing key value pair with spaces using jq

I have a properties file and a json file. The property file holds the key value pairs which needs to be replaced in the json.
When the value has no spaces it works as expected, but when the value has spaces, the value is not replaced.
Script that replaces the values
#!bin/bash
echo hello
while read line;
do
#echo $line
key=$(echo "$line" |cut -d':' -f1)
#echo $part1
value=$(echo "$line" |cut -d':' -f2)
if [[ ! -z $value ]];
then
key="\"$key\""
value="\"$value\""
echo $key : $value
jq '.parameters |= map( if .name == '$key' then .default = '$value' else . end )' cam.json > cam1.json
mv cam1.json cam.json
fi
done < prop.properties
property file
git_con_type:something
git_host_fqdn:something again
git_user:something again again
git_user_password:something
git_repo:something
git_repo_user:
git_branch:
JSON file
{
"name": "${p:Instance_Name}",
"parameters": [
{
"name": "git_con_type",
"default": "",
"immutable_after_create": false
},
{
"name": "git_host_fqdn",
"default": "hello",
"immutable_after_create": false
},
{
"name": "git_user",
"default": "",
"immutable_after_create": false
},
{
"name": "git_user_password",
"default": "Passw0rd",
"immutable_after_create": false
},
{
"name": "git_repo",
"default": "lm",
"immutable_after_create": false
},
{
"name": "git_repo_user",
"default": "-Life",
"immutable_after_create": false
},
{
"name": "git_branch",
"default": "master",
"immutable_after_create": false
},
{
"name": "git_clone_dir",
"default": "/opt/git",
"immutable_after_create": false
}
]
}
Error
jq: error: syntax error, unexpected $end, expecting QQSTRING_TEXT or QQSTRING_INTERP_START or QQSTRING_END (Unix shell quoting issues?) at , line 1:
.parameters |= map( if .name == "git_host_fqdn" then .default = "something
jq: error: Possibly unterminated 'if' statement at , line 1:
.parameters |= map( if .name == "git_host_fqdn" then .default = "something
jq: 2 compile errors
how can I make jq accept values with space? I have tried on jqplay with spaces and it works there, but not on the script.
It would be much better to pass $key and $value into jq as illustrated in the following (deliberately minimalist) tweak of your script:
#!/bin/bash
while read line
do
key=$(echo "$line" |cut -d':' -f1)
value=$(echo "$line" |cut -d':' -f2)
if [[ ! -z "$value" ]]
then
jq --arg key "$key" --arg value "$value" '.parameters |= map( if .name == $key then .default = $value else . end )' cam.json > cam1.json
mv cam1.json cam.json
fi
done < prop.properties
It would be still better to avoid the bash loop altogether and do everything with just one invocation of jq.
One-pass solution
The idea is to create a dictionary ($dict) of key-value pairs, which can easily be done using the builtin filter INDEX/1:
INDEX(inputs | split(":") | select(.[1] | length > 0); .[0])
| map_values(.[1]) as $dict
| $cam
| .parameters |= map( $dict[.name] as $value | if $value then .default = $value else . end )
Invocation
With the above jq program in program.jq:
jq -n -R -f program.jq --argfile cam cam.json prop.properties > cam1.json && mv cam1.json cam.json
Or using sponge:
jq -n -R -f program.jq --argfile cam cam.json prop.properties | sponge cam.json
Note in particular the -n option, which is needed because program.jq uses inputs.

How to parse JSON format output of "kubectl get pods" using jq and create an array

The JSON output returned to me after running this command
kubectl get pods -o json | jq '.items[].spec.containers[].env'
on my kuberntes cluster is this
[
{
"name": "USER_NAME",
"value": "USER_NAME_VALUE_A"
},
{
"name": "USER_ADDRESS",
"value": "USER_ADDRESS_VALUE_A"
}
]
[
{
"name": "USER_NAME",
"value": "USER_NAME_VALUE_B"
},
{
"name": "USER_ADDRESS",
"value": "USER_ADDRESS_VALUE_B"
}
]
I'd like to create a unified array/dictionary (Using Bash script) which looks like the example below and how can I get the value of each key?
[
{
"USER_NAME": "USER_NAME_VALUE_A",
"USER_ADDRESS": "USER_ADDRESS_VALUE_A"
},
{
"USER_NAME": "USER_NAME_VALUE_B",
"USER_ADDRESS": "USER_ADDRESS_VALUE_B"
}
]
use the jsonpath
C02W84XMHTD5:~ iahmad$ kubectl get pods --all-namespaces -o=jsonpath='{range .items[*]}{.metadata.name}{"\n"}'
coredns-c4cffd6dc-nsd2k
etcd-minikube
kube-addon-manager-minikube
kube-apiserver-minikube
kube-controller-manager-minikube
kube-dns-86f4d74b45-d5njm
kube-proxy-pg89s
kube-scheduler-minikube
kubernetes-dashboard-6f4cfc5d87-b7n7v
storage-provisioner
tiller-deploy-778f674bf5-vt4mj
https://kubernetes.io/docs/reference/kubectl/jsonpath/
it can output key values as well
C02W84XMHTD5:~ iahmad$ kubectl get pods --all-namespaces -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.startTime}{"\n"}{end}'
coredns-c4cffd6dc-nsd2k 2018-10-16T21:44:19Z
etcd-minikube 2018-10-29T17:30:56Z
kube-addon-manager-minikube 2018-10-29T17:30:56Z
kube-apiserver-minikube 2018-10-29T17:30:56Z
kube-controller-manager-minikube 2018-10-29T17:30:56Z
kube-dns-86f4d74b45-d5njm 2018-10-16T21:44:16Z
kube-proxy-pg89s 2018-10-29T17:32:05Z
kube-scheduler-minikube 2018-10-29T17:30:56Z
kubernetes-dashboard-6f4cfc5d87-b7n7v 2018-10-16T21:44:19Z
storage-provisioner 2018-10-16T21:44:19Z
tiller-deploy-778f674bf5-vt4mj 2018-11-01T13:45:23Z
then you can split those by space and form json or list
This will do it in bash. You'd be surprised how much you can do with bash:
#!/bin/bash
NAMES=`kubectl get pods -o=jsonpath='{range .items[*]}{.spec.containers[*].env[*].name}{"\n"}' | tr -d '\011\012\015'`
VALUES=`kubectl get pods -o=jsonpath='{range .items[*]}{.spec.containers[*].env[*].value}{"\n"}' | tr -d '\011\012\015'`
IFS=' ' read -ra NAMESA <<< "$NAMES"
IFS=' ' read -ra VALUESA <<< "$VALUES"
MAXINDEX=`expr ${#NAMESA[#]} - 1`
printf "[\n"
for i in "${!NAMESA[#]}"; do
printf " {\n"
printf " \"USER_NAME\": \"${NAMESA[$i]}\",\n"
printf " \"USER_ADDRESS\": \"${VALUESA[$i]}\"\n"
if [ "$i" == "${MAXINDEX}" ]; then
printf " }\n"
else
printf " },\n"
fi
done
printf "]\n"
While you are using jq as a filter, why not use it as a transformer, too?
kubectl get pods -o json | \
jq '.items|map(.spec.containers|map(.env|map({key: .name, value})|from_entries)|add)'
I know this is totally a necromancer badge, but still ;)