Let's say that with a bash script I want to create a file, so to the command to create the file will be something like this:
myscript hostgroup_one 2
hostgroup_one and the number 2 are parameters.
How can I insert the parameters in the lines below and output all the lines as a file?
{
"bool": {
"should": [
{
"match_phrase": {
"hostgroup.keyword": "$1"
}
}
],
"minimum_should_match": $2
}
}
I'd use jq to build the JSON:
jq -n \
--arg hostgroup "$1" \
--argjson minimum "$2" \
'{bool: {should: [{match_phrase: {"hostgroup.keyword": $hostgroup}}], minimum_should_match: $minimum}}'
will produce your desired output.
While jq is a great tool for manipulating json, as glenn jackman recommends, if you don't have it and your sysadmin won't install it (?!?)...
You can use a "here document"
Your myscript could look something like this:
#!/bin/bash
echo "dollar-1 is '${1}' dollar-2 is '${2}' dollar-3 is '${3}'"
cat <<EOF >"${1}"
{
"bool": {
"should": [
{
"match_phrase": {
"hostgroup.keyword": "${2}"
}
}
],
"minimum_should_match": ${3}
}
}
EOF
echo "done"
I've made the output filename the first parameter, and then your two parameters, so it would be run like:
myscript output.json hostgroup_one 2
You don't need to do that, you could use 2 params and redirect output:
myscript hostgroup_one 2 > output.json
(note you don't have to use EOF as your here-document delimiter, I just like it)
Of course you don't need the echo statements, and you should have error checking (does ${#} equal 3 parameters?)
Related
I do not seem to be able to find an answer, but have seen enough to know there is likely a better way of doing what I want to do.
Problem: I have a bash array. For each element in the bash array, I want to update a JSON array.
The JSON looks like the below. I am wanting to update the fruit array.
"foods": {
"perishable": {
"fruit": []
I'll get an array of length n, for example:
fruit_array=("banana" "orange")
It should look something like this:
"foods": {
"perishable": {
"fruit": [
{
"001": {
"002": "banana"
}
},
{
"001": {
"002": "orange"
}
}
]
Is there a nice way of doing this? At the moment I am trying the below:
#!/bin/bash
fruit_array=("banana" "orange")
for fruit in "${fruit_array[#]}"; do
jq \
--arg fruit $fruit \
'.foods.perishables.fruit += [{"001": {"002": $fruit}}]' \
template.json > template_with_fruit.json
done
This doesn't work for the obvious reason that the template is being re-read, but I have messed around to get it consuming the output of the previous iteration and nothing comes out at the end. I am only able to update the template once.
However, I know this seems a little dodgy and suspect there is a cleaner, more jq way.
A previous - aborted - attempt went something like this:
jq \
--argjson fruit "$(printf '{"001": {"002": "%s"}}\n' \"${fruit_array[#]}\" | jq -nR '[inputs]')" \
'.foods.perishables.fruit += $fruit' \
Which produced a escaped string which I couldn't do anything with, but at least hinted that there might be a neater solution to the standard bash loop.
I am missing something.
Any help would, as always, be appreciated.
JQ can do all that on its own; you don't need a loop or anything.
jq '.foods.perishable.fruit += (
$ARGS.positional
| map({"001": {"002": .}})
)' template.json --args "${fruit_array[#]}" >template_with_fruit.json
If you pass your array as a space delimited string, you can use JQ like so:
jq --arg fruits "$fruit_array" \
'.foods.perishable.fruit |= ($fruits | split(" ") | map({ "001": { "002": . } }))' input
{
"foods": {
"perishable": {
"fruit": [
{
"001": {
"002": "banana"
}
},
{
"001": {
"002": "orange"
}
}
]
}
}
}
Create whole json string at once with printf:
fruit_array=("banana" "orange")
printf -v fruits '{"001": {"002": %s}},' "${fruit_array[#]}"
$ echo $fruits
{"001": {"002": banana}},{"001": {"002": orange}},
Then add it to your template removing last comma:
$ echo ${fruits%,}
{"001": {"002": banana}},{"001": {"002": orange}}
"Numbers" could also be set like this:
fruit_array=("1 2 banana" "3 4 orange")
printf '{"%.3d": {"%.3d": %s}},' ${fruit_array[#]}
{"001": {"002": banana}},{"003": {"004": orange}},
I am trying to create JSON from scratch using bash.
The final structure needs to be like:
{
"hosts": {
"a_hostname" : {
"ips" : [
1,
2,
3
]
},
{...}
}
}
First I'm creating an input file with the format:
hostname ["1.1.1.1","2.2.2.2"]
host-name2 ["3.3.3.3","4.4.4.4"]
This is being created by:
for host in $( ansible -i hosts all --list-hosts ) ; \
do echo -n "${host} " ; \
ansible -i hosts $host -m setup | sed '1c {' | jq -r -c '.ansible_facts.ansible_all_ipv4_addresses' ; \
done > hosts.txt
The key point here is that the IP list/array, is coming from a JSON file and being extracted by jq. This extraction outputs an already valid / quoted JSON array, but as a string in a txt file.
Next I'm using jq to parse the whole text file into the desired JSON:
jq -Rn '
{ "hosts": [inputs |
split("\\s+"; "g") |
select(length > 0 and .[0] != "") |
{(.[0]):
{ips:.[1]}
}
] | add }
' < ~/hosts.txt
This is almost correct, everything except for the IPs value which is treated as a string and quoted leading to:
{
"hosts": {
"hostname1": {
"ips": "[\"1.1.1.1\",\"2.2.2.2\"]"
},
"host-name2": {
"ips": "[\"3.3.3.3\",\"4.4.4.4\"]"
}
}
}
I'm now stuck at this final hurdle - how to insert the IPs without causing them to be quoted again.
Edit - quoting solved by using {ips: .[1] | fromjson }} instead of {ips:.[1]}.
However this was completely negated by #CharlesDuffy's help suggesting converting to TSV.
Original Q body:
So far I've got to
jq -n {hosts:{}} | \
for host in $( ansible -i hosts all --list-hosts ) ; \
do jq ".hosts += {$host:{}}" | \
jq ".hosts.$host += {ips:[1,2,3]}" ; \
done ;
([1,2,3] is actually coming from a subshell but including it seemed unnecessary as that part works, and made it harder to read)
This sort of works, but there seems to be 2 problems.
1) Final output only has a single host in it containg data from the first host in the list (this persists even if the second problem is bypassed):
{
"hosts": {
"host_1": {
"ips": [
1,
2,
3
]
}
}
}
2) One of the hostnames has a - in it, which causes syntax and compiler errors from jq. I'm stuck going around quote hell trying to get it to be interpreted but also quoted. Help!
Thanks for any input.
Let's say your input format is:
host_1 1 2 3
host_2 2 3 4
host-with-dashes 3 4 5
host-with-no-addresses
...re: edit specifying a different format: Add #tsv onto the JQ command producing the existing format to generate this one instead.
If you want to transform that to the format in question, it might look like:
jq -Rn '
{ "hosts": [inputs |
split("\\s+"; "g") |
select(length > 0 and .[0] != "") |
{(.[0]): .[1:]}
] | add
}' <input.txt
Which yields as output:
{
"hosts": {
"host_1": [
"1",
"2",
"3"
],
"host_2": [
"2",
"3",
"4"
],
"host-with-dashes": [
"3",
"4",
"5"
],
"host-with-no-addresses": []
}
}
I am trying to use a bash variable to store json.
testConfig=",{
\"Classification\": \"mapred-site\",
\"Properties\": {
\"mapreduce.map.java.opts\": \"-Xmx2270m\",
\"mapreduce.map.memory.mb\": \"9712\"
}
}"
echo $testConfig
Output: ,{
If I give it in a single line it works.
But i would like to store values in my variable in a clean format.
I tried using cat >>ECHO
That didn't work either
Any help is appreciated as to how I can store this in order to get the output in an expected format.
Thanks.
You may use a here doc as described here:
read -r -d '' testConfig <<'EOF'
{
"Classification": "mapred-site",
"Properties": {
"mapreduce.map.java.opts": "-Xmx2270m",
"mapreduce.map.memory.mb": "9712"
}
}
EOF
# Just to demonstrate that it works ...
echo "$testConfig" | jq .
Doing so you can avoid quoting.
You can use read -d
read -d '' testConfig <<EOF
,{ /
"Classification": "mapred-site",
"Properties": {/
"mapreduce.map.java.opts": "-Xmx2270m",
"mapreduce.map.memory.mb": "9712"
}
}
EOF
echo $testConfig;
I've to call a file and pass a json as parameters in this way
(suppose that my file is called test.sh), from bash I need to do something like this:
./test.sh "[{\"username\":\"user1\",\"password\":\"pwd1\",\"group\":\"usergroup1\"},{\"username\":\"user2\",\"password\":\"pwd2\",\"group\":\"usergroup2\"},{\"username\":\"user3\",\"password\":\"pwd3\",\"group\":\"usergroup3\"}]"
and the content of test.sh is the following
#!/bin/bash
#read the json
system_user="$1"
printf "$system_user"
accounts=($(jq -s ".[]" <<< $system_user))
printf "$accounts"
for account in "${accounts[#]}"
do
printf "\n\n$account\n\n"
done
the output of -> printf "$system_user" is
[{"username":"user1","password":"pwd1","group":"usergroup1"},{"username":"user2","password":"pwd2","group":"usergroup2"},{"username":"user3","password":"pwd3","group":"usergroup3"}]
but the output of -> printf "$accounts" is something like this
[
[
{
"username":
"user1"
etc. etc. one object for each token :-(
and so on, but what I was expecting is an array of three object (like you can test on jqplay.org)
{
"username": "user1",
"password": "pwd1",
"group": "usergroup1"
}
{
"username": "user2",
"password": "pwd2",
"group": "usergroup2"
}
{
"username": "user3",
"password": "pwd3",
"group": "usergroup3"
}
In this way I can make a foreach on ${accounts[#]}
What I'm doing wrong?
Thank you
With the -c option, you can print each JSON object on a single line, making it easier to populate the array you want.
$ readarray -t arr < <(jq -c '.[]' <<< "[{\"username\":\"user1\",\"password\":\"pwd1\",\"group\":\"usergroup1\"},{\"username\":\"user2\",\"password\":\"pwd2\",\"group\":\"usergroup2\"},{\"username\":\"user3\",\"password\":\"pwd3\",\"group\":\"usergroup3\"}]")
$ printf "Object: %s\n" "${arr[#]}"
Object: {"username":"user1","password":"pwd1","group":"usergroup1"}
Object: {"username":"user2","password":"pwd2","group":"usergroup2"}
Object: {"username":"user3","password":"pwd3","group":"usergroup3"}
You are interchanging bash arrays and JSON arrays. When you are creating accounts array, bash splits the elements per each whitespace. That's why you don't get what you expect. You can try the following:
declare -A accounts
while IFS="=" read -r key value
do
accounts[$key]="$value"
done < <(jq -r "to_entries|map(\"\(.key)=\(.value)\")|.[]" <<< $system_user)
for account in "${accounts[#]}"
do
printf "$account\n"
done
(stolen from here: https://stackoverflow.com/a/26717401/328977)
to get the following output:
{"username":"user1","password":"pwd1","group":"usergroup1"}
{"username":"user2","password":"pwd2","group":"usergroup2"}
{"username":"user3","password":"pwd3","group":"usergroup3"}
I'm curling an endpoint:
#!/bin/bash
instance_info=$(curl -sk https://internal.admin.com/app/instance)
which gives a json response:
{
"basePath": "/install",
"metadata": {
"deployed_artifact": "app01",
"docker": "True",
"http_port": "7471",
"url": "www.google.com"
},
"name": "app-01",
"server": "webserver1"
}
I'm trying to avoid curling more than once to get the variables I need from the json using JQ.
Using bash I'd really appreciate if someone can show me how to store the response as another var and then use this to variablize name: server: url: http_port:
The following seems to run the curl twice:
#!/bin/bash
instance_info=$(curl -sk https://internal.admin.com/app/instance)
server_name=$(echo instance_info | /usr/bin/jq --raw-output '.server')
url=$(echo instance_info | /usr/bin/jq --raw-output '.url')
You are calling curl once and this suffices. Then you have the content in a variable, so you can access it without calling curl again.
Regarding your code, your approach is fine but you are missing $ when you are echoing the variable:
server_name=$(echo $instance_info | /usr/bin/jq --raw-output '.server')
# ^
See a sample. Here I hardcode the JSON:
your_json='
{
"basePath": "/install",
"metadata": {
"deployed_artifact": "app01",
"docker": "True",
"http_port": "7471",
"url": "www.google.com"
},
"name": "app-01",
"server": "webserver1"
}'
for the server:
$ echo "$your_json" | jq --raw-output '.server'
webserver1
For the url you need to indicate the block where it lies on. That is, metadata:
$ echo "$your_json" | jq --raw-output '.metadata.url'
www.google.com
To store into a variable, say:
your_field=$(echo "$your_json" | jq --raw-output 'XXXX')
# ^^^^
Here is a script which demonstrates how to use jq's #sh formatting directive along with bash eval to set bash variables using the output of a filter. In this case we hardcode the json which presumably would have come from curl.
#!/bin/bash
instance_info='
{
"basePath": "/install",
"metadata": {
"deployed_artifact": "app01",
"docker": "True",
"http_port": "7471",
"url": "www.google.com"
},
"name": "app-01",
"server": "webserver1"
}'
eval "$(jq -M -r '#sh "server_name=\(.server) url=\(.metadata.url)"' <<< "$instance_info")"
echo $server_name
echo $url
When run this produces the output
webserver1
www.google.com
The Extract data and set shell variables section of the JQ Cookbook has more examples of #sh.
sed -rn 's/[ ]*"([^"]*)"[^"]*("[^"]*").*/\1=\2/gp'' file.json
O/P:
basePath="/install"
deployed_artifact="app01"
docker="True"
http_port="7471"
url="www.google.com"
name="app-01"
server="webserver1"
eval this output you can get the variable
eval $(sed -rn 's/[ ]*"([^"]*)"[^"]*("[^"]*").*/\1=\2/gp' file.json )
echo $name $server
O/p:
app-01 webserver1
Read the values with one jq pass then there's no need to store the json:
read name server url http_port < <(
curl -sk https://internal.admin.com/app/instance | \
jq --raw-output '[.name, .server, .metadata.url, .metadata.http_port] | join(" ")'
)
echo -e "\$name: $name\n\$server: $server\n\$url: $url\n\$http_port $http_port"
$name: app-01
$server: webserver1
$url: www.google.com
$http_port 7471