jq command prints the json twice instead of only once - json

I need to set some new value using jq command in a JSON file.
Here is an example of file.json
{
"Balise1": true,
"Balise2": true,
"OtherThingEnabled": false,
"proxySettings": {
"port": 0
},
"mailSettings": {},
"maxRunningActivitiesPerJob": 5,
"maxRunningActivities": 5,
}
In order to set the proxySettings value I use the following command
jq --arg host "proxy.hub.gcp.url.com" --arg port "80" '.proxySettings = { host: $host, port: $port }' file.json
Instead of printing the modified version of file.json, it prints both original and modified version like that:
{
"Balise1": true,
"Balise2": true,
"OtherThingEnabled": false,
"proxySettings": {
"port": 0
},
"mailSettings": {},
"maxRunningActivitiesPerJob": 5,
"maxRunningActivities": 5,
}
{
"Balise1": true,
"Balise2": true,
"OtherThingEnabled": false,
"proxySettings": {
"host": "proxy.hub.gcp.url.com"
"port": "80"
},
"mailSettings": {},
"maxRunningActivitiesPerJob": 5,
"maxRunningActivities": 5,
}
I was expecting to print only the modified version.
How could I create a new JSON file only with the modified version?

Comma , feeds the same input to two filters, therefore programs such ., . will output their input twice.
If two filters are separated by a comma, then the same input will be fed into both and the two filters' output value streams will be concatenated in order: first, all of the outputs produced by the left expression, and then all of the outputs produced by the right. For instance, filter .foo, .bar, produces both the "foo" fields and "bar" fields as separate outputs.
Demo:
echo '"foobar"' | jq 'length, length'
6
6
Instead, you want to combine both filters sequentially with the pipe filter |, since the plain assignment operator = outputs its modified input.
jq --arg host "proxy.hub.gcp.url.com" \
--arg port "80" \
'.proxySettings = { host: $host, port: $port } | .mailSettings = {value1: "Value1"}'
Your initial question didn't include the full jq program (so it was missing a proper minimal reproducible example!), only your self-answer included the crucial details.

I found the issue.
I my shell I was modifying 2 thing on the same command like this :
jq --arg host "proxy.hub.gcp.url.com" --arg port "80" '.proxySettings = { host: $host, port: $port }, .mailSettings = '{value1: "Value1"}' file.json
The command was printing the first change, then the second one. I had to create two seperate jq command.

Related

Bash JSON compare two list and delete id

I have a JSON endpoint which I can fetch value with curl and yml local file. I want to get the difference and delete it with id of name present on JSON endpoint.
JSON's endpoint
[
{
"hosts": [
"server1"
],
"id": "qz9o847b-f07c-49d1-b1fa-e5ed0b2f0519",
"name": "V1_toto_a"
},
{
"hosts": [
"server2"
],
"id": "a6aa847b-f07c-49d1-b1fa-e5ed0b2f0519",
"name": "V1_tata_b"
},
{
"hosts": [
"server3"
],
"id": "a6d9ee7b-f07c-49d1-b1fa-e5ed0b2f0519",
"name": "V1_titi_c"
}
]
files.yml
---
instance:
toto:
name: "toto"
tata:
name: "tata"
Between JSON's endpoint and local file, I want to delete it with id of tata, because it is the difference between the sources.
declare -a arr=(_a _b _c)
ar=$(cat files.yml | grep name | cut -d '"' -f2 | tr "\n" " ")
fileItemArray=($ar)
ARR_PRE=("${fileItemArray[#]/#/V1_}")
for i in "${arr[#]}"; do local_var+=("${ARR_PRE[#]/%/$i}"); done
remote_var=$(curl -sX GET "XXXX" | jq -r '.[].name | #sh' | tr -d \'\")
diff_=$(echo ${local_var[#]} ${remote_var[#]} | tr ' ' '\n' | sort | uniq -u)
output = titi
the code works, but I want to delete the titi with id dynamically
curl -X DELETE "XXXX" $id_titi
I am trying to delete with bash script, but I have no idea to continue...
Your endpoint is not proper JSON as it has
commas after the .name field but no following field
no commas between the elements of the top-level array
If this is not just a typo from pasting your example into this question, then you'd need to address this first before proceeding. This is how it should look like:
[
{
"hosts": [
"server1"
],
"id": "qz9o847b-f07c-49d1-b1fa-e5ed0b2f0519",
"name": "toto"
},
{
"hosts": [
"server2"
],
"id": "a6aa847b-f07c-49d1-b1fa-e5ed0b2f0519",
"name": "tata"
},
{
"hosts": [
"server3"
],
"id": "a6d9ee7b-f07c-49d1-b1fa-e5ed0b2f0519",
"name": "titi"
}
]
If your endpoint is proper JSON, try the following. It extracts the names from your .yml file (just as you do - there are plenty of more efficient and less error-prone ways but I'm trying to adapt your approach as much as possible) but instead of a Bash array generates a JSON array using jq which for Bash is a simple string. For your curl output it's basically the same thing, extracting a (JSON) array of names into a Bash string. Note that in both cases I use quotes <var>="$(…)" to capture strings that may include spaces (although I also use the -c option for jq to compact it's output to a single line). For the difference between the two, everything is taken over by jq as it can easily be fed with the JSON arrays as variables, perform the subtraction and output in your preferred format:
fromyml="$(cat files.yml | grep name | cut -d '"' -f2 | jq -Rnc '[inputs]')"
fromcurl="$(curl -sX GET "XXXX" | jq -c 'map(.name)')"
diff="$(jq -nr --argjson fromyml "$fromyml" --argjson fromcurl "$fromcurl" '
$fromcurl - $fromyml | .[]
')"
The Bash variable diff now contains a list of names only present in the curl output ($fromcurl - $fromyml), one per line (if, other than in your example, there happens to be more than one). If the curl output had duplicates, they will still be included (use $fromcurl - $fromyml | unique | .[] to get rid of them):
titi
As you can see, this solution has three calls to jq. I'll leave it to you to further reduce that number as it fits your general workflow (basically, it can be put together into one).
Getting the output of a program into a variable can be done using read.
perl -M5.010 -MYAML -MJSON::PP -e'
sub get_next_file { local $/; "".<> }
my %filter = map { $_->{name} => 1 } values %{ Load(get_next_file)->{instance} };
say for grep !$filter{$_}, map $_->{name}, #{ decode_json(get_next_file) };
' b.yaml a.json |
while IFS= read -r id; do
curl -X DELETE ..."$id"...
done
I used Perl here because what you had was no way to parse a YAML file. The snippet requires having installed the YAML Perl module.

How can jq be used to insert dynamic field names recursively for all objects in an array?

'm new to jq, and hoping to convert JSON below so that, for each object in the records array , the "Account" object is deleted and replaced with an "AccountID" field which has a the value of Account.Id.
Assuming I don't know what the name of the field (eg. Account ) is prior to executing, so it Has to be dynamically included as an argument to --arg.
Contacts.json:
{
"records": [
{
"attributes": {
"type": "Contact",
"referenceId": "ContactRef1"
},
"Account": {
"attributes": {
"type": "Account",
"url": "/services/data/v51.0/sobjects/Account/asdf"
},
"Id": "asdf"
}
},
{
"attributes": {
"type": "Contact",
"referenceId": "ContactRef2"
},
"Account": {
"attributes": {
"type": "Account",
"url": "/services/data/v51.0/sobjects/Account/qwer"
},
"Id": "qwer"
}
}
]
}
to
{
"records": [
{
"attributes": {
"type": "Contact",
"referenceId": "ContactRef1"
},
"AccountID": "asdf"
}
},{
"attributes": {
"type": "Contact",
"referenceId": "ContactRef2"
},
"AccountID": "qwer"
}
}
]
}
This example above is a little contrived because in actuality, I need to be able to dynamically name the ID field to be able to port the new JSON structure into destination system. For my use case, it's not always valid to tack "ID" onto the field name ( eg. Account .. ID ), so I passed the field names to --arg .
This is as close as I got.. but it's not quite there. and I suspect there is better way.
jq -c --arg field "Account" --arg field_name_id "AccountID" '. |= . + if .records?[]?[$field] != null then { "\($field_name_id)" : .records[][$field].Id } else empty end | if .records?[]?[$field] != null then del(.records[][$field]) else empty end' Contacts.json
I've wrestled with this quite a while, but this is as far as I'm able to manage without running into tons of syntax errors. I really appreciate any help to add an AccountID field on each object in the records array.
Here's the actual bash script where jq is being run ( relevant parts are where FIELD(S) is being used )
#! /bin/bash
# This script takes a of soql file as first and only argument
# The main purpose is to tweak the json results from an sfdx:data:tree:export so the json is compatible with sfdx:data:tree:import
# This is needed because sfdx export & import are inadequate to use whne relationships more than 2 levels deep in the export query.
# grab all unique object names within the soql file for any objects where the ID field is being SELECTed ( eg. "Account Iteration__r Profile UserRole" )
FIELDS=`grep -oe '\([A-Za-z_]\+\)\.[iI][dD]' $1 | cut -f 1 -d . - | sort -u`
#find all json files in file and rewrite the relationship FIELDS blocks into someting sfdx can import
for FIELD in $FIELDS;
do
if [[ $FIELD =~ __r ]]
then
FIELD_NAME_ID=`sed 's/__r/__c/' <<< $FIELD`
else
FIELD_NAME_ID="${FIELD}ID"
fi
JSON_FILES=`ls *.json`
#Loop all json files in direcotry
for DATA_FILE in $JSON_FILES
do
#replace any email addresses left in custom data( just in case )
#using gsed becuse Mac lacks -i flag for in-place substitution
gsed -i 's/[^# "]*#[^#]*\.[^# ,"]*/fake#test.com/g' $DATA_FILE
# make temporary file to hold the rewritten json
TEMP_FILE="temp-${DATA_FILE}.bk"
echo $DATA_FILE $FIELD $FIELD_NAME_ID
#For custom relationship jttrs. change __r to __c to get the name of Id field, otherwise just add "ID".
jq -c --arg field $FIELD --arg field_name_id $FIELD_NAME_ID '. |= . + if .records?[]?[$field] != null then { "\($field_name_id)" : .records[][$field].Id } else empty end | if .records?[]?[$field] != null then del(.records[][$field]) else empty end' $DATA_FILE 1> ./$TEMP_FILE 2> modify-json.errors
# if TEMP_FILE is not empty, then jq revised it, so replace contents the original JSON DATA_FILE
if [[ -s ./$TEMP_FILE ]]
then
#JSON format spacing/line-breaks
jq '.' $TEMP_FILE > $DATA_FILE
fi
rm $TEMP_FILE
done
done
The key to a simple solution is |=. Here's one using map:
.records |= map( .Account.Id as $x
| del(.Account)
| . + {AccountID: $x} )
which can be simplified to:
.records |= map( . + {AccountID: .Account.Id}
| del(.Account) )
Either of these can easily be adapted to the case where the two field names are passed in as arguments, or if they must be inferred from the "owner" of "Id".
Adapting peak's answer to use the dynamic field name:
jq -c --arg field "Account" \
--arg field_name_id "AccountID" '
.records |= map(.[$field].Id as $x
| del(.[$field])
| . + {($field_name_id): $x})
'

How to Add a JSON Field to the Root JSON Object Using jq

I am trying to create the following JSON object structure:
{
"hard-coded-value": false,
"dynamic-value-1": true,
"dynamic-value-2": true,
"dynamic-value-3": true
}
My array of dynamic values is called DYNAMIC_VALUES.
I wrote the following bash code:
DYNAMIC_VALUES=("dynamic-value-1" "dynamic-value-2" "dynamic-value-3")
JSON_OBJECT=$( jq -n '{"hard-coded-value": false}' )
for i in "${DYNAMIC_VALUES[#]}"
do
JSON_OBJECT+=$( jq -n \
--arg key "$i" \
'{($key): true}' )
done
echo $JSON_OBJECT
The above code prints the following
{ "hard-coded-value": false }{ "dynamic-value-1": true }{ "dynamic-value-2": true }{ "dynamic-value-3": true }
What I want is this output to look like the output outlined at the top of this question, but I can't figure out how to tell jq to append to the root JSON object instead of creating a bunch of objects.
You don't need a loop there.
$ dynamic_values=('dynamic-value-1' 'dynamic-value-2' 'dynamic-value-3')
$ printf '%s\n' "${dynamic_values[#]}" | jq -nR '{hardcoded_value: false} | .[inputs] = true'
{
"hardcoded_value": false,
"dynamic-value-1": true,
"dynamic-value-2": true,
"dynamic-value-3": true
}
This will break if one of the array elements contains a line feed though. For that JQ 1.6 has --args, which can be used as shown below.
$ dynamic_values=('dynamic-value-1' $'dynamic-value-2\n' 'dynamic-value-3')
$ jq -n '{hardcoded_value: false} | .[$ARGS.positional[]] = true' --args "${dynamic_values[#]}"
{
"hardcoded_value": false,
"dynamic-value-1": true,
"dynamic-value-2\n": true,
"dynamic-value-3": true
}

Dynamically modify JSON file in bash script with jq - use heredocs or key assignments?

Below is the part of my JSON file, the JSON file itself is longer.
{
"anomalyDetection": {
"loadingTimeThresholds": {
"enabled": false,
"thresholds": []
},
"outageHandling": {
"globalOutage": true,
"localOutage": true,
"localOutagePolicy": {
"affectedLocations": 1,
"consecutiveRuns": 2
}
}
}
}
Here I need to modify affectedLocations and consecutiveRuns values with supplied parameters, assign the modified JSON to a var which then is used during the API call.
I created two different solutions with bash to accomplish the above.
Option 1 - use heredocs
# Assign the template file to a template var
get_template() {
read -r -d '' template <<JSONTEMPLATE
{
"anomalyDetection": {
"loadingTimeThresholds": {
"enabled": false,
"thresholds": []
},
"outageHandling": {
"globalOutage": true,
"localOutage": true,
"localOutagePolicy": {
"affectedLocations": 1,
"consecutiveRuns": 2
}
}
}
}
JSONTEMPLATE
}
modify_json_with_heredoc() {
# Call the template var assignment
get_template
read -r -d '' json_keys_updated <<JSONSTRING
{
"globalOutage": true,
"localOutage": true,
"localOutagePolicy": {
"affectedLocations": ${1:-1},
"consecutiveRuns": ${2:-2}
}
}
JSONSTRING
# Replace key values from the template with the updated parameters
updated_JSON=$(jq --argjson update_outageHandling "$json_keys_updated" \
'.anomalyDetection.outageHandling|=$update_outageHandling' \
<<<"$template")
}
Option 2 - modify keys in the JSON directly
modify_json_with_vars() {
# Call the template assignment
get_template
updated_JSON=$(jq --argjson update_affectedLocations "${1:-1}" --argjson update_consecutiveRuns "${2:-2}" \
' .anomalyDetection.outageHandling.globalOutage|=true
| .anomalyDetection.outageHandling.localOutagePolicy.affectedLocations|=$update_affectedLocations
| .anomalyDetection.outageHandling.localOutagePolicy.consecutiveRuns|=$update_consecutiveRuns' \
<<<"$template")
}
I need to be able to call functions with/without parameters. If functions are called without parameters, then need to keep the default values for keys, that's why ${1:-1} ${2:-2} are there.
Both options work fine - for example,modify_json_with_vars 2 3 or modify_json_with_heredoc. My question is which of the above is better/faster/safer, as well as more 'right' thing to do.
As I've mentioned, the template file is larger. At this moment I need to modify only the keys in the object .anomalyDetection.outageHandling.localOutagePolicy, however, in the future I might need to change other keys as well and need a solution which is scalable and maintainable.
I think that the first option is neater because jq is called only with one parameter. But if I will need to change keys all over the source template JSON file, probably the second option is more scalable.
Suggestions are welcome.
I too would suggest a different approach for the reasons outlined below.
#!/usr/bin/env bash
function template {
cat<<EOF
{
"anomalyDetection": {
"loadingTimeThresholds": {
"enabled": false,
"thresholds": []
},
"outageHandling": {
"globalOutage": true,
"localOutage": true,
"localOutagePolicy": {
"affectedLocations": 1,
"consecutiveRuns": 2
}
}
}
}
EOF
}
function modify_json() {
template |
jq --argjson update_affectedLocations "${1:-1}" \
--argjson update_consecutiveRuns "${2:-2}" '
.anomalyDetection.outageHandling.localOutagePolicy
|= (.affectedLocations |= $update_affectedLocations
| .consecutiveRuns |= $update_consecutiveRuns )
'
}
updated_JSON=$(modify_json 42 666)
echo "$updated_JSON"
Considerations
It is generally ill-advised to write functions which set "global" variables (such as $template) if possible. By defining a function (template), we also introduce only one name instead of one for the function and one for the variable.
It appears that the parameters update_affectedLocations and update_consecutiveRuns are supposed to be numbers, so using --argjson makes sense, especially if some resilience and simplicity are considerations.
The jq program achieves "DRY-ness" (i.e. it avoids unnecessary repetition) by using the update-assignment operator freely.
Using "|" at the beginning of lines in a jq program makes it easy to achieve a certain amount of visual elegance while highlighting the logic.
Notice also that the suggested script is quite succinct and uncomplicated, while being reasonably robust.
Some would deprecate using the keyword function to define shell functions, but it does make locating function definitions trivially easy, both for human eyes and human fingers at the keyboard.
Another alternative is handling all the arguments within jq:
#!/usr/bin/env sh
# Assign the template file to a template var
get_template() {
template='{
"anomalyDetection": {
"loadingTimeThresholds": {
"enabled": false,
"thresholds": []
},
"outageHandling": {
"globalOutage": true,
"localOutage": true,
"localOutagePolicy": {
"affectedLocations": 1,
"consecutiveRuns": 2
}
}
}
}'
}
modify_json() {
# Call the template assignment
get_template
updated_JSON=$(
jq \
--arg update_affectedLocations "$1" \
--arg update_consecutiveRuns "$2" \
--null-input \
"$template"'|
if $update_affectedLocations | try tonumber catch false
then
.anomalyDetection.outageHandling.localOutagePolicy.affectedLocations |= (
$update_affectedLocations | tonumber
)
else .
end |
if $update_consecutiveRuns | try tonumber catch false
then
.anomalyDetection.outageHandling.localOutagePolicy.consecutiveRuns |= (
$update_consecutiveRuns | tonumber
)
else .
end '
)
}
modify_json 42 666
echo "$updated_JSON"

Building new JSON with JQ and bash

I am trying to create JSON from scratch using bash.
The final structure needs to be like:
{
"hosts": {
"a_hostname" : {
"ips" : [
1,
2,
3
]
},
{...}
}
}
First I'm creating an input file with the format:
hostname ["1.1.1.1","2.2.2.2"]
host-name2 ["3.3.3.3","4.4.4.4"]
This is being created by:
for host in $( ansible -i hosts all --list-hosts ) ; \
do echo -n "${host} " ; \
ansible -i hosts $host -m setup | sed '1c {' | jq -r -c '.ansible_facts.ansible_all_ipv4_addresses' ; \
done > hosts.txt
The key point here is that the IP list/array, is coming from a JSON file and being extracted by jq. This extraction outputs an already valid / quoted JSON array, but as a string in a txt file.
Next I'm using jq to parse the whole text file into the desired JSON:
jq -Rn '
{ "hosts": [inputs |
split("\\s+"; "g") |
select(length > 0 and .[0] != "") |
{(.[0]):
{ips:.[1]}
}
] | add }
' < ~/hosts.txt
This is almost correct, everything except for the IPs value which is treated as a string and quoted leading to:
{
"hosts": {
"hostname1": {
"ips": "[\"1.1.1.1\",\"2.2.2.2\"]"
},
"host-name2": {
"ips": "[\"3.3.3.3\",\"4.4.4.4\"]"
}
}
}
I'm now stuck at this final hurdle - how to insert the IPs without causing them to be quoted again.
Edit - quoting solved by using {ips: .[1] | fromjson }} instead of {ips:.[1]}.
However this was completely negated by #CharlesDuffy's help suggesting converting to TSV.
Original Q body:
So far I've got to
jq -n {hosts:{}} | \
for host in $( ansible -i hosts all --list-hosts ) ; \
do jq ".hosts += {$host:{}}" | \
jq ".hosts.$host += {ips:[1,2,3]}" ; \
done ;
([1,2,3] is actually coming from a subshell but including it seemed unnecessary as that part works, and made it harder to read)
This sort of works, but there seems to be 2 problems.
1) Final output only has a single host in it containg data from the first host in the list (this persists even if the second problem is bypassed):
{
"hosts": {
"host_1": {
"ips": [
1,
2,
3
]
}
}
}
2) One of the hostnames has a - in it, which causes syntax and compiler errors from jq. I'm stuck going around quote hell trying to get it to be interpreted but also quoted. Help!
Thanks for any input.
Let's say your input format is:
host_1 1 2 3
host_2 2 3 4
host-with-dashes 3 4 5
host-with-no-addresses
...re: edit specifying a different format: Add #tsv onto the JQ command producing the existing format to generate this one instead.
If you want to transform that to the format in question, it might look like:
jq -Rn '
{ "hosts": [inputs |
split("\\s+"; "g") |
select(length > 0 and .[0] != "") |
{(.[0]): .[1:]}
] | add
}' <input.txt
Which yields as output:
{
"hosts": {
"host_1": [
"1",
"2",
"3"
],
"host_2": [
"2",
"3",
"4"
],
"host-with-dashes": [
"3",
"4",
"5"
],
"host-with-no-addresses": []
}
}