I have this command in a bash script
kubectl get svc --selector='app.kubernetes.io/component=sentinel' --all-namespaces -o json |
jq -r '
.items
| map(
{label:.metadata.name,
sentinels: [{host:(.metadata.name + "." + .metadata.namespace + "." + "svc" + "." + "cluster" + "." + "local"),port: .spec.ports[0].port}],
sentinelName:"mymaster",
sentinelPassword: ""
dbIndex: 0
})
| {connections: . }' /
>local.json
which produces something like this output
{
"connections": [
{
"label": "Redis1",
"sentinels": [
{
"host": "Redis1.default.svc.cluster.local",
"port": 26379
}
],
"sentinelName": "mymaster",
"sentinelPassword": "",
"dbIndex": 0
},
{
"label": "Redis2",
"sentinels": [
{
"host": "Redis2.development.svc.cluster.local",
"port": 26379
}
],
"sentinelName": "mymaster",
"sentinelPassword": "",
"dbIndex": 0
}
]
}
This config file is injected into container via an init-container so Redis-Commander fetches the redis instances without the user having to manually input any connection config data. This works fine however but one of the instances requires a sentinelPassword value.
I can fetch the password using kubectl get secret but I'm trying to figure out how to insert that password into the config file for the particular instance that requires it.
I've been trying something along the lines of this but getting my jq syntax wrong. Any help or alternative ways of going around would be appreciated.
#store output in var
JSON=$(kubectl get svc --selector='app.kubernetes.io/component=sentinel' --all-namespaces -o json |
jq -r '
.items
| map(
{label:.metadata.name,
sentinels: [{host:(.metadata.name + "." + .metadata.namespace + "." + "svc" + "." + "cluster" + "." + "local"),port: .spec.ports[0].port}],
sentinelName:"mymaster",
sentinelPassword: "",
dbIndex: 0
})
| {connections: . }')
# Find instance by their host value which is unique. (Can't figure out how to do this bit)
if $JSON host name contains "Redis2.development.svc.cluster.local"
#then do something like this
"$JSON" | jq '.[]'| .sentinelPassword = "$password" #var stored from kubectl get secret cmd
#save output to file
"$JSON">/local.json
Assuming for the moment that you want to invoke jq on the sample JSON (local.json) as shown, you could run:
password=mypassword
< local.json jq --arg password "$password" '
.connections[] |= if any(.sentinels[].host; index("Redis2.development.svc.cluster.local"))
then .sentinelPassword = $password else . end'
However, if possible, it would probably be better to invoke jq just once.
Related
I am trying to get certain value in the json with some for each and if condition.
code looks like the following
for i in "$(jq -r '.value[]' test.json)"; do
result=$(echo $i | jq -r .result)
if [ "$result" == "succeeded" ]
then
name=$(echo $i | jq -r .agent.name)
echo "$name"
fi
done
json file - test.json
{
"count":3,
"value":[
{
"location":"CA",
"result":"failed",
"agent":{
"id":97833,
"name":"Brad"
},
"priority":0
},
{
"location":"TX",
"result":"failed",
"agent":{i
"id":15232,
"name":"Tom"
},
"priority":0
},
{
"location":"CO",
"result":"succeeded",
"agent":{
"id":13412,
"name":"John"
},
"priority":0
}
]
}
I am trying to loop through the json file using jq to get the name of the agent if the result was "succeeded". but the result i get from result=$(echo $i | jq -r .result) seems like it's returning string array #("failed","failed","succeeded").
========================update==============================
jq ' # get the name of the agent if the result was "succeeded".
.value[]
| select(.result == "succeeded")
| var1=.agent.name
| echo $var1
' test.json
if I want to save the .agent.name into a variable and use it with a different command, what do I have to do?
There is no need to use a shell loop.
With your test.json,
jq ' # get the name of the agent if the result was "succeeded".
.value[]
| select(.result == "succeeded")
| .agent.name
' test.json
produces:
"John"
I have a JSON endpoint which I can fetch value with curl and yml local file. I want to get the difference and delete it with id of name present on JSON endpoint.
JSON's endpoint
[
{
"hosts": [
"server1"
],
"id": "qz9o847b-f07c-49d1-b1fa-e5ed0b2f0519",
"name": "V1_toto_a"
},
{
"hosts": [
"server2"
],
"id": "a6aa847b-f07c-49d1-b1fa-e5ed0b2f0519",
"name": "V1_tata_b"
},
{
"hosts": [
"server3"
],
"id": "a6d9ee7b-f07c-49d1-b1fa-e5ed0b2f0519",
"name": "V1_titi_c"
}
]
files.yml
---
instance:
toto:
name: "toto"
tata:
name: "tata"
Between JSON's endpoint and local file, I want to delete it with id of tata, because it is the difference between the sources.
declare -a arr=(_a _b _c)
ar=$(cat files.yml | grep name | cut -d '"' -f2 | tr "\n" " ")
fileItemArray=($ar)
ARR_PRE=("${fileItemArray[#]/#/V1_}")
for i in "${arr[#]}"; do local_var+=("${ARR_PRE[#]/%/$i}"); done
remote_var=$(curl -sX GET "XXXX" | jq -r '.[].name | #sh' | tr -d \'\")
diff_=$(echo ${local_var[#]} ${remote_var[#]} | tr ' ' '\n' | sort | uniq -u)
output = titi
the code works, but I want to delete the titi with id dynamically
curl -X DELETE "XXXX" $id_titi
I am trying to delete with bash script, but I have no idea to continue...
Your endpoint is not proper JSON as it has
commas after the .name field but no following field
no commas between the elements of the top-level array
If this is not just a typo from pasting your example into this question, then you'd need to address this first before proceeding. This is how it should look like:
[
{
"hosts": [
"server1"
],
"id": "qz9o847b-f07c-49d1-b1fa-e5ed0b2f0519",
"name": "toto"
},
{
"hosts": [
"server2"
],
"id": "a6aa847b-f07c-49d1-b1fa-e5ed0b2f0519",
"name": "tata"
},
{
"hosts": [
"server3"
],
"id": "a6d9ee7b-f07c-49d1-b1fa-e5ed0b2f0519",
"name": "titi"
}
]
If your endpoint is proper JSON, try the following. It extracts the names from your .yml file (just as you do - there are plenty of more efficient and less error-prone ways but I'm trying to adapt your approach as much as possible) but instead of a Bash array generates a JSON array using jq which for Bash is a simple string. For your curl output it's basically the same thing, extracting a (JSON) array of names into a Bash string. Note that in both cases I use quotes <var>="$(…)" to capture strings that may include spaces (although I also use the -c option for jq to compact it's output to a single line). For the difference between the two, everything is taken over by jq as it can easily be fed with the JSON arrays as variables, perform the subtraction and output in your preferred format:
fromyml="$(cat files.yml | grep name | cut -d '"' -f2 | jq -Rnc '[inputs]')"
fromcurl="$(curl -sX GET "XXXX" | jq -c 'map(.name)')"
diff="$(jq -nr --argjson fromyml "$fromyml" --argjson fromcurl "$fromcurl" '
$fromcurl - $fromyml | .[]
')"
The Bash variable diff now contains a list of names only present in the curl output ($fromcurl - $fromyml), one per line (if, other than in your example, there happens to be more than one). If the curl output had duplicates, they will still be included (use $fromcurl - $fromyml | unique | .[] to get rid of them):
titi
As you can see, this solution has three calls to jq. I'll leave it to you to further reduce that number as it fits your general workflow (basically, it can be put together into one).
Getting the output of a program into a variable can be done using read.
perl -M5.010 -MYAML -MJSON::PP -e'
sub get_next_file { local $/; "".<> }
my %filter = map { $_->{name} => 1 } values %{ Load(get_next_file)->{instance} };
say for grep !$filter{$_}, map $_->{name}, #{ decode_json(get_next_file) };
' b.yaml a.json |
while IFS= read -r id; do
curl -X DELETE ..."$id"...
done
I used Perl here because what you had was no way to parse a YAML file. The snippet requires having installed the YAML Perl module.
'm new to jq, and hoping to convert JSON below so that, for each object in the records array , the "Account" object is deleted and replaced with an "AccountID" field which has a the value of Account.Id.
Assuming I don't know what the name of the field (eg. Account ) is prior to executing, so it Has to be dynamically included as an argument to --arg.
Contacts.json:
{
"records": [
{
"attributes": {
"type": "Contact",
"referenceId": "ContactRef1"
},
"Account": {
"attributes": {
"type": "Account",
"url": "/services/data/v51.0/sobjects/Account/asdf"
},
"Id": "asdf"
}
},
{
"attributes": {
"type": "Contact",
"referenceId": "ContactRef2"
},
"Account": {
"attributes": {
"type": "Account",
"url": "/services/data/v51.0/sobjects/Account/qwer"
},
"Id": "qwer"
}
}
]
}
to
{
"records": [
{
"attributes": {
"type": "Contact",
"referenceId": "ContactRef1"
},
"AccountID": "asdf"
}
},{
"attributes": {
"type": "Contact",
"referenceId": "ContactRef2"
},
"AccountID": "qwer"
}
}
]
}
This example above is a little contrived because in actuality, I need to be able to dynamically name the ID field to be able to port the new JSON structure into destination system. For my use case, it's not always valid to tack "ID" onto the field name ( eg. Account .. ID ), so I passed the field names to --arg .
This is as close as I got.. but it's not quite there. and I suspect there is better way.
jq -c --arg field "Account" --arg field_name_id "AccountID" '. |= . + if .records?[]?[$field] != null then { "\($field_name_id)" : .records[][$field].Id } else empty end | if .records?[]?[$field] != null then del(.records[][$field]) else empty end' Contacts.json
I've wrestled with this quite a while, but this is as far as I'm able to manage without running into tons of syntax errors. I really appreciate any help to add an AccountID field on each object in the records array.
Here's the actual bash script where jq is being run ( relevant parts are where FIELD(S) is being used )
#! /bin/bash
# This script takes a of soql file as first and only argument
# The main purpose is to tweak the json results from an sfdx:data:tree:export so the json is compatible with sfdx:data:tree:import
# This is needed because sfdx export & import are inadequate to use whne relationships more than 2 levels deep in the export query.
# grab all unique object names within the soql file for any objects where the ID field is being SELECTed ( eg. "Account Iteration__r Profile UserRole" )
FIELDS=`grep -oe '\([A-Za-z_]\+\)\.[iI][dD]' $1 | cut -f 1 -d . - | sort -u`
#find all json files in file and rewrite the relationship FIELDS blocks into someting sfdx can import
for FIELD in $FIELDS;
do
if [[ $FIELD =~ __r ]]
then
FIELD_NAME_ID=`sed 's/__r/__c/' <<< $FIELD`
else
FIELD_NAME_ID="${FIELD}ID"
fi
JSON_FILES=`ls *.json`
#Loop all json files in direcotry
for DATA_FILE in $JSON_FILES
do
#replace any email addresses left in custom data( just in case )
#using gsed becuse Mac lacks -i flag for in-place substitution
gsed -i 's/[^# "]*#[^#]*\.[^# ,"]*/fake#test.com/g' $DATA_FILE
# make temporary file to hold the rewritten json
TEMP_FILE="temp-${DATA_FILE}.bk"
echo $DATA_FILE $FIELD $FIELD_NAME_ID
#For custom relationship jttrs. change __r to __c to get the name of Id field, otherwise just add "ID".
jq -c --arg field $FIELD --arg field_name_id $FIELD_NAME_ID '. |= . + if .records?[]?[$field] != null then { "\($field_name_id)" : .records[][$field].Id } else empty end | if .records?[]?[$field] != null then del(.records[][$field]) else empty end' $DATA_FILE 1> ./$TEMP_FILE 2> modify-json.errors
# if TEMP_FILE is not empty, then jq revised it, so replace contents the original JSON DATA_FILE
if [[ -s ./$TEMP_FILE ]]
then
#JSON format spacing/line-breaks
jq '.' $TEMP_FILE > $DATA_FILE
fi
rm $TEMP_FILE
done
done
The key to a simple solution is |=. Here's one using map:
.records |= map( .Account.Id as $x
| del(.Account)
| . + {AccountID: $x} )
which can be simplified to:
.records |= map( . + {AccountID: .Account.Id}
| del(.Account) )
Either of these can easily be adapted to the case where the two field names are passed in as arguments, or if they must be inferred from the "owner" of "Id".
Adapting peak's answer to use the dynamic field name:
jq -c --arg field "Account" \
--arg field_name_id "AccountID" '
.records |= map(.[$field].Id as $x
| del(.[$field])
| . + {($field_name_id): $x})
'
I am trying to create JSON from scratch using bash.
The final structure needs to be like:
{
"hosts": {
"a_hostname" : {
"ips" : [
1,
2,
3
]
},
{...}
}
}
First I'm creating an input file with the format:
hostname ["1.1.1.1","2.2.2.2"]
host-name2 ["3.3.3.3","4.4.4.4"]
This is being created by:
for host in $( ansible -i hosts all --list-hosts ) ; \
do echo -n "${host} " ; \
ansible -i hosts $host -m setup | sed '1c {' | jq -r -c '.ansible_facts.ansible_all_ipv4_addresses' ; \
done > hosts.txt
The key point here is that the IP list/array, is coming from a JSON file and being extracted by jq. This extraction outputs an already valid / quoted JSON array, but as a string in a txt file.
Next I'm using jq to parse the whole text file into the desired JSON:
jq -Rn '
{ "hosts": [inputs |
split("\\s+"; "g") |
select(length > 0 and .[0] != "") |
{(.[0]):
{ips:.[1]}
}
] | add }
' < ~/hosts.txt
This is almost correct, everything except for the IPs value which is treated as a string and quoted leading to:
{
"hosts": {
"hostname1": {
"ips": "[\"1.1.1.1\",\"2.2.2.2\"]"
},
"host-name2": {
"ips": "[\"3.3.3.3\",\"4.4.4.4\"]"
}
}
}
I'm now stuck at this final hurdle - how to insert the IPs without causing them to be quoted again.
Edit - quoting solved by using {ips: .[1] | fromjson }} instead of {ips:.[1]}.
However this was completely negated by #CharlesDuffy's help suggesting converting to TSV.
Original Q body:
So far I've got to
jq -n {hosts:{}} | \
for host in $( ansible -i hosts all --list-hosts ) ; \
do jq ".hosts += {$host:{}}" | \
jq ".hosts.$host += {ips:[1,2,3]}" ; \
done ;
([1,2,3] is actually coming from a subshell but including it seemed unnecessary as that part works, and made it harder to read)
This sort of works, but there seems to be 2 problems.
1) Final output only has a single host in it containg data from the first host in the list (this persists even if the second problem is bypassed):
{
"hosts": {
"host_1": {
"ips": [
1,
2,
3
]
}
}
}
2) One of the hostnames has a - in it, which causes syntax and compiler errors from jq. I'm stuck going around quote hell trying to get it to be interpreted but also quoted. Help!
Thanks for any input.
Let's say your input format is:
host_1 1 2 3
host_2 2 3 4
host-with-dashes 3 4 5
host-with-no-addresses
...re: edit specifying a different format: Add #tsv onto the JQ command producing the existing format to generate this one instead.
If you want to transform that to the format in question, it might look like:
jq -Rn '
{ "hosts": [inputs |
split("\\s+"; "g") |
select(length > 0 and .[0] != "") |
{(.[0]): .[1:]}
] | add
}' <input.txt
Which yields as output:
{
"hosts": {
"host_1": [
"1",
"2",
"3"
],
"host_2": [
"2",
"3",
"4"
],
"host-with-dashes": [
"3",
"4",
"5"
],
"host-with-no-addresses": []
}
}
I want to convert my sqlite data from my database to JSON format.
I would like to use this syntax:
sqlite3 -line members.db "SELECT * FROM members LIMIT 3" > members.txt
OUTPUT:
id = 1
fname = Leif
gname = Håkansson
genderid = 1
id = 2
fname = Yvonne
gname = Bergman
genderid = 2
id = 3
fname = Roger
gname = Sjöberg
genderid = 1
How to do this with nice and structur code in a for loop?
(Only in Bash)
I have tried some awk and grep but not with a great succes yet.
Would be nice with some tips.
I want a result similar to this:
[
{
"id":1,
"fname":"Leif",
"gname":"Hakansson",
"genderid":1
},
{
"id":2,
"fname":"Yvonne",
"gname":"Bergman",
"genderid":2
},
{
"id":3,
"fname":"Roger",
"gname":"Sjberg",
"genderid":1
}
}
If your sqlite3 is compiled with the json1 extension (or if you can obtain a version of sqlite3 with the json1 extension), then you can use it to generate JSON objects (one JSON object per row). For example:
select json_object('id', id, 'fname', fname, 'gname', gname, 'genderid', genderid) ...
You can then use a tool such as jq to convert the stream of objects into an array of objects, e.g. pipe the output of the sqlite3 to jq -s ..
(A less tiresome alternative might be to use the sqlite3 function json_array(), which produces an array, which you can reassemble into an object using jq.)
If the json1 extension is unavailable, then you could use the following as a starting point:
awk 'BEGIN { print "["; }
function out() {if (n++) {print ","}; if (line) {print "{" line "}"}; line="";}
function trim(x) { sub(/^ */, "", x); sub(/ *$/, "", x); return x; }
NF==0 { out(); next};
{if (line) {line = line ", " }
i=index($0,"=");
line = line "\"" trim(substr($0,1,i-1)) ": \"" substr($0, i+2) "\""}
END {out(); print "]"} '
Alternatively, you could use the following jq script, which converts numeric strings that occur on the RHS of "=" to numbers:
def trim: sub("^ *"; "") | sub(" *$"; "");
def keyvalue: index("=") as $i
| {(.[0:$i] | trim): (.[$i+2:] | (tonumber? // .))};
[foreach (inputs, "") as $line ({object: false, seed: {} };
if ($line|trim) == "" then { object: .seed, seed : {} }
else {object: false,
seed: (.seed + ($line | keyvalue)) }
end;
.object | if . and (. != {}) then . else empty end ) ]
Just type -json argument with SQLite 3.33.0 or higher and get json output:
$ sqlite3 -json database.db "select * from TABLE_NAME"
from SQLite Release 3.33.0 note:
...
CLI enhancements:
Added four new output modes: "box", "json", "markdown", and "table".
The "column" output mode automatically expands columns to contain the longest output row and automatically turns ".header" on if it has
not been previously set.
The "quote" output mode honors ".separator"
The decimal extension and the ieee754 extension are built-in to the CLI
...
I think I would prefer to parse sqlite output with a single line per record rather than the very wordy output format you suggested with sqlite3 -line. So, I would go with this:
sqlite3 members.db "SELECT * FROM members LIMIT 3"
which gives me this to parse:
1|Leif|Hakansson|1
2|Yvonne|Bergman|2
3|Roger|Sjoberg|1
I can now parse that with awk if I set the input separator to | with
awk -F '|'
and pick up the 4 fields on each line with the following and save them in an array like this:
{ id[++i]=$1; fname[i]=$2; gname[i]=$3; genderid[i]=$4 }
Then all I need to do is print the output format you need at the end. However, you have double quotes in your output and they are a pain to quote in awk, so I temporarily use another pipe symbol (|) as a double quote and then, at the very end, I get tr to replace all the pipe symbols with double quotes - just to make the code easier on the eye. So the total solution looks like this:
sqlite3 members.db "SELECT * FROM members LIMIT 3" | awk -F'|' '
# sqlite output line - pick up fields and store in arrays
{ id[++i]=$1; fname[i]=$2; gname[i]=$3; genderid[i]=$4 }
END {
printf "[\n";
for(j=1;j<=i;j++){
printf " {\n"
printf " |id|:%d,\n",id[j]
printf " |fname|:|%s|,\n",fname[j]
printf " |gname|:|%s|,\n",gname[j]
printf " |genderid|:%d\n",genderid[j]
closing=" },\n"
if(j==i){closing=" }\n"}
printf closing;
}
printf "]\n";
}' | tr '|' '"'
Sqlite-utils does exactly what you're looking for. By default, the output will be JSON.
Better late than never to plug jo.
Save sqlite3 to a text file.
Get jo (jo's also available in distro repos)
and use this bash script.
while read line
do
id=`echo $line | cut -d"|" -f1`
fname=`echo $line | cut -d"|" -f2`
gname=`echo $line | cut -d"|" -f3`
genderid=`echo $line | cut -d"|" -f4`
jsonline=`jo id="$id" fname="$fname" gname="$gname" genderid="$genderid"`
json="$json $jsonline"
done < "$1"
jo -a $json
Please don't create (or parse) json with awk. There are dedicated tools for this. Tools like xidel.
While first and foremost a html, xml and json parser, xidel can also parse plain text.
I'd like to offer a very elegant solution using this tool (with much less code than jq).
I'll assume your 'members.txt'.
First to create a sequence of each json object to-be:
xidel -s members.txt --xquery 'tokenize($raw,"\n\n")'
Or...
xidel -s members.txt --xquery 'tokenize($raw,"\n\n") ! (position(),.)'
1
id = 1
fname = Leif
gname = Håkansson
genderid = 1
2
id = 2
fname = Yvonne
gname = Bergman
genderid = 2
3
id = 3
fname = Roger
gname = Sjöberg
genderid = 1
...to better show you the individual items in the sequence.
Now you have 3 multi-line strings. To turn each item/string into another sequence where each item is a new line:
xidel -s members.txt --xquery 'tokenize($raw,"\n\n") ! x:lines(.)'
(x:lines(.) is a shorthand for tokenize(.,'\r\n?|\n'))
Now for each line tokenize on the " = " (which creates yet another sequence) and save it to a variable. For the first line for example this sequence is ("id","1"), for the second line ("fname","Leif"), etc.:
xidel -s members.txt --xquery 'tokenize($raw,"\n\n") ! (for $x in x:lines(.) let $a:=tokenize($x," = ") return ($a[1],$a[2]))'
Finally remove leading whitespace (normalize-space()), create a json object ({| {key-value-pair} |}) and put all json objects in an array ([ ... ]):
xidel -s members.txt --xquery '[tokenize($raw,"\n\n") ! {|for $x in x:lines(.) let $a:=tokenize($x," = ") return {normalize-space($a[1]):$a[2]}|}]'
Prettified + output:
xidel -s members.txt --xquery '
[
tokenize($raw,"\n\n") ! {|
for $x in x:lines(.)
let $a:=tokenize($x," = ")
return {
normalize-space($a[1]):$a[2]
}
|}
]
'
[
{
"id": "1",
"fname": "Leif",
"gname": "Håkansson",
"genderid": "1"
},
{
"id": "2",
"fname": "Yvonne",
"gname": "Bergman",
"genderid": "2"
},
{
"id": "3",
"fname": "Roger",
"gname": "Sjöberg",
"genderid": "1"
}
]
Note: For xidel-0.9.9.7173 and newer --json-mode=deprecated is needed to create a json array with [ ]. The new (XQuery 3.1) way to create a json array is to use array{ }.