Calculate values from json file - json

I have a script
#!/bin/bash
#create path to redirect output.json to same directory as output.txt
path=$(dirname $1)
#create variables for name line, test lines and result line
firstline=$(cat $1 | head -n +1 | cut -d "[" -f2 | cut -d "]" -f1)
testname=$(echo $firstline)
tests=$(cat $1 | tail -n +3 | head -n -2)
results=$(cat $1 | tail -n1)
#create main JSON variable
json=$(jq -n --arg tn "$testname" '{testname:$tn,tests:[],summary:{}}')
#test's names, status, duration and updating JSON variable
IFS=$'\n'
for i in $tests
do
if [[ $i == not* ]]
then
stat=false
else
stat=true
fi
if [[ $i =~ expecting(.+?)[0-9] ]]
then
var=${BASH_REMATCH[0]}
name=${var%,*}
fi
if [[ $i =~ [0-9]*ms ]]
then
test_duration=${BASH_REMATCH[0]}
fi
json=$(echo $json | jq \
--arg na "$name" \
--arg st "$stat" \
--arg dur "$test_duration" \
'.tests += [{name:$na,status:$st|test("true"),duration:$dur}]')
done
#final success, failed, rating, duration and finishing JSON variable
IFS=$'\n'
for l in $results
do
if [[ $l =~ [0-9]+ ]]
then
success=${BASH_REMATCH[0]}
fi
if [[ $l =~ ,.[0-9]+ ]]
then
v=${BASH_REMATCH[0]}
failed=${v:2}
fi
if [[ $l =~ [0-9]+.[0-9]+% ]] || [[ $l =~ [0-9]+% ]]
then
va=${BASH_REMATCH[0]}
rating=${va%%%}
fi
if [[ $l =~ [0-9]*ms ]]
then
duration=${BASH_REMATCH[0]}
fi
json=$(echo $json | jq \
--arg suc "$success" \
--arg fa "$failed" \
--arg rat "$rating" \
--arg dur "$duration" \
'.summary += {success:$suc|tonumber,failed:$fa|tonumber,rating:$rat|tonumber,duration:$dur}')
done
#redirect variable's output to file
echo $json | jq "." > $path"/output.json"
Output.json looks like:
{
"testname": "Behave test",
"tests": [
{
"name": "some text",
"status": true,
"duration": "7ms"
},
{
"name": "some text",
"status": false,
"duration": "27ms"
}
],
"summary": {
"success": 1,
"failed": 1,
"rating": 50,
"duration": "34ms"
}
}
Output is much more than in my example above. My question is, how I can calculate "summary" values and put it to json?
"success" - tests with status true;
"failed" - tests with status false;
"rating" - success / total, and it can be float or int
"duration" - sum of "duration" field
I will be very grateful for help.

This adds (or replaces) a .summary field which is calculated by reduceing the .tests array to an object, initialized as {success: 0, failed: 0}. For each array item, either .success or .failed is incremented, depending on the current array item's boolean .state field. The .rating field is incremented unconditionally, thus counting the total number of items which is later used to calculate the true rating. As for the duration, the array item's .duration field is converted into a number by chopping off the ms suffix, and added to the summary's (numeric) .duration field. After the reduction, the rating is corrected by dividing the the success count by it (and multiplied by 100 for convenience). The ? // 0 part catches division by zero issues, in case the .tests array was empty. Finally, the .duration field is re-equipped with the ms suffix.
.summary = (reduce .tests[] as $t ({success: 0, failed: 0};
(if $t.status then .success else .failed end, .rating) += 1
| .duration += ($t.duration | rtrimstr("ms") | tonumber)
) | .rating = (100 * (.success / .rating)? // 0) | .duration |= "\(.//0)ms")
{
"testname": "Behave test",
"tests": [
{
"name": "some text",
"status": true,
"duration": "7ms"
},
{
"name": "some text",
"status": false,
"duration": "27ms"
}
],
"summary": {
"success": 1,
"failed": 1,
"rating": 50,
"duration": "34ms"
}
}
Demo

Related

Append multiple dummy objects to json array with jq

Lets say this is my array :
[
{
"name": "Matias",
"age": "33"
}
]
I can do this :
echo "$response" | jq '[ .[] | select(.name | test("M.*"))] | . += [.[]]'
And it will output :
[
{
"name": "Matias",
"age": "33"
},
{
"name": "Matias",
"age": "33"
}
]
But I cant do this :
echo "$response" | jq '[ .[] | select(.name | test("M.*"))] | . += [.[] * 3]'
jq: error (at <stdin>:7): object ({"name":"Ma...) and number (3) cannot be multiplied
I need to extend an array to create a dummy array with 100 values. And I cant do it. Also, I would like to have a random age on the objects. ( So later on I can filter the file to measure performance of an app .
Currently jq does not have a built-in randomization function, but it's easy enough to generate random numbers that jq can use. The following solution uses awk but in a way that some other PRNG can easily be used.
#!/bin/bash
function template {
cat<<EOF
[
{
"name": "Matias",
"age": "33"
}
]
EOF
}
function randoms {
awk -v n=$1 'BEGIN { for(i=0;i<n;i++) {print int(100*rand())} }'
}
randoms 100 | jq -n --argfile template <(template) '
first($template[] | select(.name | test("M.*"))) as $t
| [ $t | .age = inputs]
'
Note on performance
Even though the above uses awk and jq together, this combination is about 10 times faster than the posted jtc solution using -eu:
jq+awk: u+s = 0.012s
jtc with -eu: u+s = 0.192s
Using jtc in conjunction with awk as above, however, gives u+s == 0.008s on the same machine.

Replacing key value pair with spaces using jq

I have a properties file and a json file. The property file holds the key value pairs which needs to be replaced in the json.
When the value has no spaces it works as expected, but when the value has spaces, the value is not replaced.
Script that replaces the values
#!bin/bash
echo hello
while read line;
do
#echo $line
key=$(echo "$line" |cut -d':' -f1)
#echo $part1
value=$(echo "$line" |cut -d':' -f2)
if [[ ! -z $value ]];
then
key="\"$key\""
value="\"$value\""
echo $key : $value
jq '.parameters |= map( if .name == '$key' then .default = '$value' else . end )' cam.json > cam1.json
mv cam1.json cam.json
fi
done < prop.properties
property file
git_con_type:something
git_host_fqdn:something again
git_user:something again again
git_user_password:something
git_repo:something
git_repo_user:
git_branch:
JSON file
{
"name": "${p:Instance_Name}",
"parameters": [
{
"name": "git_con_type",
"default": "",
"immutable_after_create": false
},
{
"name": "git_host_fqdn",
"default": "hello",
"immutable_after_create": false
},
{
"name": "git_user",
"default": "",
"immutable_after_create": false
},
{
"name": "git_user_password",
"default": "Passw0rd",
"immutable_after_create": false
},
{
"name": "git_repo",
"default": "lm",
"immutable_after_create": false
},
{
"name": "git_repo_user",
"default": "-Life",
"immutable_after_create": false
},
{
"name": "git_branch",
"default": "master",
"immutable_after_create": false
},
{
"name": "git_clone_dir",
"default": "/opt/git",
"immutable_after_create": false
}
]
}
Error
jq: error: syntax error, unexpected $end, expecting QQSTRING_TEXT or QQSTRING_INTERP_START or QQSTRING_END (Unix shell quoting issues?) at , line 1:
.parameters |= map( if .name == "git_host_fqdn" then .default = "something
jq: error: Possibly unterminated 'if' statement at , line 1:
.parameters |= map( if .name == "git_host_fqdn" then .default = "something
jq: 2 compile errors
how can I make jq accept values with space? I have tried on jqplay with spaces and it works there, but not on the script.
It would be much better to pass $key and $value into jq as illustrated in the following (deliberately minimalist) tweak of your script:
#!/bin/bash
while read line
do
key=$(echo "$line" |cut -d':' -f1)
value=$(echo "$line" |cut -d':' -f2)
if [[ ! -z "$value" ]]
then
jq --arg key "$key" --arg value "$value" '.parameters |= map( if .name == $key then .default = $value else . end )' cam.json > cam1.json
mv cam1.json cam.json
fi
done < prop.properties
It would be still better to avoid the bash loop altogether and do everything with just one invocation of jq.
One-pass solution
The idea is to create a dictionary ($dict) of key-value pairs, which can easily be done using the builtin filter INDEX/1:
INDEX(inputs | split(":") | select(.[1] | length > 0); .[0])
| map_values(.[1]) as $dict
| $cam
| .parameters |= map( $dict[.name] as $value | if $value then .default = $value else . end )
Invocation
With the above jq program in program.jq:
jq -n -R -f program.jq --argfile cam cam.json prop.properties > cam1.json && mv cam1.json cam.json
Or using sponge:
jq -n -R -f program.jq --argfile cam cam.json prop.properties | sponge cam.json
Note in particular the -n option, which is needed because program.jq uses inputs.

Change entry in a JSON list that matches a condition without discarding rest of document

I am trying to open a file, look through the file and change a value based on the value and pass this either to a file or var.
Below is an example of the JSON
{
"Par": [
{
"Key": "12345L",
"Value": "https://100.100.100.100:100",
"UseLastValue": true
},
{
"Key": "12345S",
"Value": "VAL2CHANGE",
"UseLastValue": true
},
{
"Key": "12345T",
"Value": "HAPPY-HELLO",
"UseLastValue": true
}
],
"CANCOPY": false,
"LOGFILE": ["HELPLOG"]
}
i have been using jq and i have been successful in isolating the object group and change the value.
cat jsonfile,json | jq '.Par | map(select(.Value=="VAL2CHANGE")) | .[] | .Value="VALHASBEENCHANGED"'
This gives
{
"Key": "12345S",
"Value": "VALHASBEENCHANGED",
"UseLastValue": true
}
What id like to achieve is to retain the full JSON output with the changed value
{
"Par": [
{
"Key": "12345L",
"Value": "https://100.100.100.100:100",
"UseLastValue": true
},
{
"Key": "12345S",
"Value": "VALHASBEENCHANGED",
"UseLastValue": true
},
{
"Key": "12345T",
"Value": "HAPPY-HELLO",
"UseLastValue": true
}
],
"CANCOPY": false,
"LOGFILE": ["HELPLOG"]
}
I.E.
jq '.Par | map(select(.Value=="VAL2CHANGE")) | .[] | .Value="VALHASBEENCHANGED"' (NOW PUT IT BACK IN FILE)
OR
open file, look in file, file value to be changed and change this and output this to a file or to screen
To add, the json file will only contain the value im looking for once as im creating this. If any other values need changing i will name differently.
jq --arg match "VAL2CHANGE" \
--arg replace "VALHASBEENCHANGED" \
'.Par |= map(if .Value == $match then (.Value=$replace) else . end)' \
<in.json
To more comprehensively replace a string anywhere it may be in a nested data structure, you can use the walk function -- which will be in the standard library in jq 1.6, but can be manually pulled in in 1.5:
jq --arg match "VAL2CHANGE" \
--arg replace "VALHASBEENCHANGED" '
# taken from jq 1.6; will not be needed here after that version is released.
# Apply f to composite entities recursively, and to atoms
def walk(f):
. as $in
| if type == "object" then
reduce keys_unsorted[] as $key
( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f
elif type == "array" then map( walk(f) ) | f
else f
end;
walk(if . == $match then $replace else . end)' <in.json
If you're just replacing based on the values, you could stream the file and replace the values as you rebuild the result.
$ jq --arg change 'VAL2CHANGE' --arg value 'VALHASBEENCHANGED' -n --stream '
fromstream(inputs | if length == 2 and .[1] == $change then .[1] = $value else . end)
' input.json

Update values in json with jq (shell script)

I am trying to update a couple values in a json array (separate file) with a shell script. Basically the logic is, set an environment variable called URL, ping that URL and add it to the json-- if 0, update another json field to SUCCESS, else update to FAILED.
Here is are the files:
info.json:
{
"name": "PingTest",",
"metrics": [
{
"event_type": "PingResult",
"provider": "test",
"providerUrl": "URL",
"providerResult": "RESULT"
}
]
}
pinger.sh:
#!/bin/sh
JSON=`cat info.json` #read in JSON
#Assume URL variable is set to www.dksmfdkf.com
ping -q -c 1 "$URL" > /dev/null #ping url
if [ $? -eq 0 ]; then #if ping success, replace result in json template
JSON=`echo ${JSON} | jq --arg v "$URL" '.metrics[].providerUrl |= $v'
info.json`
JSON=`echo ${JSON} | jq '.metrics[].providerResult |= "SUCCESS"' info.json`
else
JSON=`echo ${JSON} | jq --arg v "$URL" '.metrics[].providerUrl |= $v'
info.json`
JSON=`echo ${JSON} | jq '.metrics[].providerResult |= "FAILED"' info.json`
fi
#Remove whitespace from json
JSON=`echo $JSON | tr -d ' \t\n\r\f'`
#Print the result
echo "$JSON"
The problem is my json file isn't getting updated properly, example result when running:
home:InfraPingExtension home$ ./pinger.sh
ping: cannot resolve : Unknown host
{
"name": "PingTest",
"metrics": [
{
"event_type": "PingResult",
"provider": "test",
"providerUrl": "",
"providerResult": "RESULT"
}
]
}
{
"name": "PingTest",
"metrics": [
{
"event_type": "PingResult",
"provider": "test",
"providerUrl": "URL",
"providerResult": "FAILED"
}
]
}
{"name":"PingTest","metrics":[{"event_type":"PingResult","provider":"test","providerUrl":"URL","providerResult":"RESULT"}]}
This would be greatly simplified by only calling jq once.
host=${URL#http://}; host=${host#https://}; host=${host%%/*}
if ping -q -c 1 "$host"; then
result=SUCCESS
else
result=FAILED
fi
JSON=$(
jq -c \
--arg url "$URL" \
--arg result "$result" \
'.metrics[].providerUrl |= $url
| .metrics[].providerResult |= $result
' info.json
)

update json with jq through shell script

I created a shell script to create and update json files using jq. To create json files is working well. I have a variable that is passed as argument to jq command
#!/bin/sh
OPER=$1
FILE_PATH=$2
DATE_TIME=`date +%Y-%m-%d:%H:%M:%S`
DATE=`date +%Y-%m-%d`
CSV=$3
STEP=$4
STATUS=$5
CODE=$6
MESSAGE=$7
if [ "$#" -eq 7 ]; then
if [ "$OPER" == "create" ]; then
# echo "FILE_PATH: $FILE_PATH - CSV: $CSV - STEP: $STEP - STATUS: $STATUS - CODE: $CODE - MESSAGE: $MESSAGE"
REPORT="{\"date\": \"$DATE\", \"csv\": \"$CSV\", \"messages\": [{ \"timestamp\": \"$DATE_TIME\", \"step\": \"$STEP\", \"status\": \"$STATUS\", \"code\": \"$CODE\", \"message\": \"$MESSAGE\" }] }"
echo ${REPORT} | jq . > $FILE_PATH
elif [ "$OPER" == "update" ]; then
echo "FILE_PATH: $FILE_PATH - CSV: $CSV - STEP: $STEP - STATUS: $STATUS - CODE: $CODE - MESSAGE: $MESSAGE"
REPORT="{\"timestamp\": \"$DATE_TIME\", \"step\": \"$STEP\", \"status\": \"$STATUS\", \"code\": \"$CODE\", \"message\": \"$MESSAGE\"}"
echo "REPORTTTTT: "$REPORT
REPORT="jq '.messages[.messages| length] |= . + $REPORT' $FILE_PATH"
#echo $REPORT
echo `jq '.messages[.messages| length] |= . + $REPORT' $FILE_PATH` > $FILE_PATH
else
echo "operation not recognized: $OPER"
fi
else
echo "wrong parameters."
fi
jq . $FILE_PATH
But to update the json file I am getting an error. My $REPORT variable is correct. The cotes are correct. I think I must use another jq argument instead of |= . +. I used that command with plain text and it worked. But when I create the REPORT variable dynamically it throws an error.
Any clue? Thanks
REPORTTTTT: {"timestamp": "2017-02-17:12:11:11", "step": "2", "status": "OK", "code": "34", "message": "message 34 file.xml"}
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
'.messages[.messages|
jq: 1 compile error
jq: error: REPORT/0 is not defined at <top-level>, line 1:
.messages[.messages| length] |= . + $REPORT
jq: 1 compile error
Here is the example at the command line>>
$ jq . file.json
{
"date": "2017-02-17",
"csv": "file.csv",
"messages": [
{
"timestamp": "2017-02-17:12:31:21",
"step": "1",
"status": "OK",
"code": "33",
"message": "message 33"
}
]
}
$ export REPORT="{\"timestamp\": \"2017-02-17:11:51:14\", \"step\": \"2\", \"status\": \"OK\", \"code\": \"34\", \"message\": \"message 34 file.xml\"}"
$ echo $REPORT
{"timestamp": "2017-02-17:11:51:14", "step": "2", "status": "OK", "code": "34", "message": "message 34 file.xml"}
$ jq '.messages[.messages| length] |= . + $REPORT' file.json
jq: error: REPORT/0 is not defined at <top-level>, line 1:
.messages[.messages| length] |= . + $REPORT
jq: 1 compile error
Pass the argument $REPORT as a json argument using the --argjson flag supported from jq-1.5 onwards,
--argjson name JSON-text:
This option passes a JSON-encoded value to the jq program as a predefined variable.
Change your line to,
jq --argjson args "$REPORT" '.data.messages += [$args]' file
Don't generate JSON by hand; let jq do it. This is important if the values you want to add to the JSON need to be quoted properly, which will not happen if you are just performing string interpolation in the shell:
$ foo='"hi" he said'
$ json="{ \"text\": \"$foo\" }"
$ echo "$json"
{ "text": ""hi" he said" } # Wrong: should be { "text": "\"hi\" he said" }
Further, jq can generate the date and time strings; there is no need to run date (twice) as well.
#!/bin/sh
if [ $# -eq 7 ]; then
printf "Wrong number of parameters: %d\n" "$#" >&2
exit 1
fi
oper=$1 file_path=$2
# Generate the new message using arguments 3-7
new_message=$(
jq -n --arg csv "$3" --arg step "$4" \
--arg status "$5" --arg code "$6" \
--arg message "$7" '{
timestamp: now|strftime("%F:%T"),
csv: $csv, step: $step, status: $status, code: $code, message: $message}'
)
case $oper in
create)
jq -n --argjson new_msg "$new_message" --arg csv "$3" '{
date: now|strftime("%F"),
csv: $csv,
messages: [ $new_msg ]
}' > "$file_path"
;;
update)
jq --argjson new_msg "$new_message" \
'.messages += [ $new_msg ]' \
"$file_path" > "$file_path.tmp" && mv "$file_path.tmp" "$file_path" ;;
*) printf 'operation not recognized: %s\n' "$oper" >&2
exit 1 ;;
esac
jq '.' "$file_path"
Thanks #Inian. I changed a little bit but worked.... here is the solution.
if [ "$#" -eq 7 ]; then
if [ "$OPER" == "update" ]; then
echo "update Json"
echo "FILE_PATH: $FILE_PATH - CSV: $CSV - STEP: $STEP - STATUS: $STATUS - CODE: $CODE - MESSAGE: $MESSAGE"
REPORT="{\"timestamp\": \"$DATE_TIME\", \"step\": \"$STEP\", \"status\": \"$STATUS\", \"code\": \"$CODE\", \"message\": \"$MESSAGE\"}"
echo "REPORTTTTT: "$REPORT
echo `jq --argjson args "$REPORT" '.data.messages += [$args]' $FILE_PATH` > $FILE_PATH
else
echo "operation not recognized: $OPER"
fi
else
echo "wrong parameters."
fi