Creating a Select Menu in bash using JSON input - json

I am trying to create a menu using the select function in bash. I am accessing an API which will return its output in json format. I will then process what the API returns into a select statement where a user can then interact with.
Here is the API call and how I parse the output:
curl -H "Authorization:Bearer $ACCESS_TOKEN" https://api.runscope.com/buckets \
| python -mjson.tool > output.json
This will send the output from the curl through python's json parsing tool and finally into the output.json file.
I then create an array using this json blob. I had to set IFS to \n in order to parse the file properly:
IFS=$'\n'
BUCKETS=("$(jq '.data | .[].name' output.json)")
I then add an exit option to the array so that users have a way to quit the selection menu:
BUCKETS+=("Exit")
Finally, I create the menu:
select BUCKET in $BUCKETS;
do
case $BUCKET in
"Exit")
echo "Exiting..."
break;;
esac
echo "You picked: $BUCKET"
done
Unfortunately, this does not create the exit option. I am able to see a menu consisting of every other option I want, except the exit option. Every option in the menu and in the array has quotes around them. How do I get the Exit option to show up?

$BUCKETS is expanding to the first element of the BUCKETS array.
Which is then being word-split and used as your select entries.
This makes sense since you wrapped the jq subshell in double quotes which prevented word-splitting from happening there (and means the IFS change isn't doing anything I believe).
Iff your entries can contain spaces and you want that assigned to an array properly the way to do that is by reading the output of jq with a while IFS= read -r entry; do loop.
BUCKETS=()
while IFS= read -r entry; do
BUCKETS+=("$entry")
done < <(jq '.data | .[].name' output.json)
Then appending your exit item to the array.
BUCKETS+=(Exit)
and then using
select BUCKET in "${BUCKETS[#]}"; do
(Both select a in l; do or select a in l\ndo there's no need for ; and \n there.)
That all being said unless you need output.json for something else you can avoid that too.
BUCKETS=()
while IFS= read -r entry; do
BUCKETS+=("$entry")
done < <(curl -H "Authorization:Bearer $ACCESS_TOKEN" https://api.runscope.com/buckets | python -mjson.tool | jq '.data | .[].name')

Instead of
jq '.data | .[].name' output.json
try
jq -r '.data | .[].name' output.json
(-r : raw data without quotes)
And the most important part :
select BUCKET in "${BUCKETS[#]}"
^^^
ARRAY syntax

Related

extra characters at end of command in shell

I am trying to run the following script:
sed -E -n '/"data"/,/}/{/[{}]/d;s/^[[:space:]]*"([^"]+)":[[:space:]]*"([^"]+)".*$/\1|\2/g;p}' /tmp/data.json | while IFS="|" read -r item val;do item="${item^^}"; item="${val}"; export "${item}"; echo ${item}; done
This basically exports data from inside a JSON as environment variables.
That is,
Here, the key data will have a list (of different lengths) of key-value pairs within itself wherein the key is not fixed. Now, I want to read every key in the list and export its value. For example, I want these commands to be executed as part of the shell script.
export HELLO1
export SAMPLEKEY
However, when I run this, it gives the error: sed: 1: "/"data"/,/}/{/[{}]/d;s/ ...": extra characters at the end of p command. What might be the reason for this?
Rather than trying to use sed to parse .json files (which can rapidly grow beyond reasonable sed parsing), instead use a tool made for parsing json (like jq -- json query). You can easily obtain the keys for values under data, and then parse with your shell tools.
(note: your questions should be tagged bash since you use the parameter expansion for character-case which is a bashism, e.g. ${item^^})
Using jq, you could do something like the following:
jq '.data' /tmp/data.json | tail -n+2 | head -n-1 |
while read -r line; do line=${line#*\"}; line=${line%%\"*}; \
printf "export %s " ${line^^}; done; echo ""
Which results in the output:
export HELLO1 export SAMPLEKEY
(there are probably even cleaner way to do this with jq -- and there was)
You can have jq output the keys for data one per line with:
jq -r '.data | to_entries[] | (.key|ascii_upcase)' /tmp/data.json
This allows you to shorten your command to generate export in from of the keys with:
while read -r key; do \
printf "export %s " $key; \
done < <(jq -r '.data | to_entries[] | (.key|ascii_upcase)' /tmp/data.json); \
echo ""
(note: to effect your actual environment, you would need to export the values as part of your shell startup)

bash script complains about file name too long

So I have a script that does this (jq is a command line JSON processor):
echo "Getting LB Node IDs"
echo $LB_STATUS | jq '.loadBalancer.nodes[] .id'
The output of the last line is:
1
2
3
But when I try and assign it to an array:
echo "Creating an Array"
nodeIdArray=($($LB_STATUS | jq '.loadBalancer.nodes[] .id'))
I get this error:
./myShellScript.sh: line 53: {"loadBalancer":{"name":"lbName","id":1,"protocol":"HTTP","port":80,"algorithm":"WEIGHTED_LEAST_CONNECTIONS","status":"ACTIVE","cluster":{"name":"ztm-n22.dfw1.lbaas.rackspace.net"},"nodes":[{"address":"1.2.3.4","id":1,"type":"PRIMARY","port":80,"status":"ONLINE","condition":"ENABLED","weight":1},{"address":"1.2.3.4","id":2,"type":"PRIMARY","port":80,"status":"ONLINE","condition":"ENABLED","weight":1},{"address":"1.2.3.4","id":3,"type":"PRIMARY","port":80,"status":"ONLINE","condition":"ENABLED","weight":1}],"timeout":30,"created":{"time":"2016-06-28T22:14:24Z"},"healthMonitor":{"type":"CONNECT","delay":10,"timeout":5,"attemptsBeforeDeactivation":2},"sslTermination":...<A BOAT LOAD MORE JSON I CUT OUT FOR BREVITY'S SAKE>: File name too long
SO $LB_STATUS | jq '.loadBalancer.nodes[] .id' produces a few numbers while trying to assign those numbers to an array doesn't work
What Went Wrong
$variable | something doesn't pass the text in variable as input to something -- instead, it runs the contents of $variable as a command. Presumably you wanted echo "$variable" | something instead (but see below!)
Even if that were fixed, the array=( $(some-command) ) idiom is itself buggy. See BashPitfalls #50 describing why it shouldn't be used, and the various alternatives.
What To Do Instead
When feeding content from a variable as an input to a command, it's idiomatic to use a herestring: somecommand <<<"$variable". These aren't free (as they create temporary files), but they're cheaper than pipelines (which fork off subshells).
If you have bash 4.x or newer, you have readarray:
readarray -t nodeIdArray < <(jq -r '.loadBalancer.nodes[].id' <<<"$LB_STATUS")
If you need compatibility with bash 3.x, read -a can do the job:
IFS=$'\n' read -r -d '' -a nodeIdArray \
< <(jq -r '.loadBalancer.nodes[].id' <<<"$LB_STATUS" && printf '\0')
...which also has the advantage of causing read to return a nonzero exit status if the jq command fails.
You left out the echo.
Change
nodeIdArray=($($LB_STATUS | jq '.loadBalancer.nodes[] .id'))
to
nodeIdArray=($( echo $LB_STATUS | jq '.loadBalancer.nodes[] .id' ))

Ignore Unparseable JSON with jq

I'm using jq to parse some of my logs, but some of the log lines can't be parsed for various reasons. Is there a way to have jq ignore those lines? I can't seem to find a solution. I tried to use the --seq argument that was recommended by some people, but --seq ignores all the lines in my file.
Assuming that each log entry is exactly one line, you can use the -R or --raw-input option to tell jq to leave the lines unparsed, after which you can prepend fromjson? | to your filter to make jq try to parse each line as JSON and throw away the ones that error.
I have log stream where some messages are in json format.
I want to pipe the json messages through jq, and just echo the rest.
The json messages are on a single line.
Solution: use grep and tee to split the lines in two streams, those starting with "^{" pipe through jq and the rest just echo to terminal.
kubectl logs -f web-svjkn | tee >(grep -v "^{") | grep "^{" | jq .
or
cat logs | tee >(grep -v "^{") | grep "^{" | jq .
Explanation:
tee generates 2nd stream, and grep -v prints non json info, 2nd grep only pipes what looks like json opening bracket to jq.
This is an old thread, but here's another solution fully in jq. This allows you to both process proper json lines and also print out non-json lines.
jq -R . as $line | try (fromjson | <further processing for proper json lines>) catch $line'
There are several Q&As on the FAQ page dealing with the topic of "invalid JSON", but see in particular the Q:
Is there a way to have jq keep going after it hits an error in the input file?
In particular, this shows how to use --seq.
However, from the the sparse details you've given (SO recommends a minimal example be given), it would seem it might be better simply to use inputs. The idea is to process one JSON entity at a time, using "try/catch", e.g.
def handle: inputs | [., "length is \(length)"] ;
def process: try handle catch ("Failed", process) ;
process
Don't forget to use the -n option when invoking jq.
See also Processing not-quite-valid JSON.
If JSON in curly braces {}:
grep -Pzo '\{(?>[^\{\}]|(?R))*\}' | jq 'objects'
If JSON in square brackets []:
grep -Pzo '\[(?>[^\[\]]|(?R))*\]' | jq 'arrays'
This works if there are no []{} in non-JSON lines.

Bash script traversing a multi-line JSON object using jq

I have to curl to a site (statuscake.com) that sends multiple items back in a JSON, each line of which contains multiple items. I want to extract from each line two of them, WebsiteName and TestID, so I can check if WebsiteName matches the one I'm interested in, get the TestID out and pass this to a second curl statement to delete the test.
Although it's more complex, the JSON that comes back is essentially of the form
[{"TestID": 123, "WebsiteName": "SomeSite1"}, {"TestID": 1234, "WebsiteName": "SomeSite2"}]
I can't seem to find a magic jq command to do it all in one - if there is one, I'd be really happy to see it.
I've got
cat $data | jq '[.[] | .WebsiteName]'
to get an array of the website names (and a very similar one for the TestIDs, but I think I've done something daft. data is the information coming back from the curl to get the JSON and that's populated OK.
I want to be able to assign these to two arrays, names and ids, then search names for the index of the relevant name, grab the id from ids and pass that to the curl. Unless there's a better way.
Any advice please?
My Xidel can do it all at once by selecting the JSON with a XPath-like query:
E.g. return all ids where the WebsiteName contains "site2" from an array of objects:
xidel /tmp/x.json -e '$json()[contains((.).WebsiteName, "site2")]/TestID'
Or e.g. to download the original JSON and then make the HTTP request with the ids:
xidel http://statuscake.com/your-url... -f '$json()[contains((.).WebsiteName, "site2")]/TestID!x"/your-delete-url{.}..."'
If I'm getting your question right, it sounds like what you want is to, for each element, select those where .WebsiteName == "needle", and then get .TestID from it. You can do just that:
.[] | select(.WebsiteName == "needle") | .TestID
If you want an array as the result, you can wrap the above script in square brackets.
The jq filters startswith and endswith may be of interest to you. If you're going to pass the result back to cURL, you may also be interested in the #sh formatting filter and the -r command-line flag.
Assuming you have a bash 4+ and assuming the json is valid (does not contain newlines in strings, etc.) this works:
$ echo "$data"
[{"TestID": 123, "WebsiteName": "SomeSite1"}, {"TestID": 1234, "WebsiteName":
"SomeSite2"}, {"TestID": 555, "WebsiteName": "foo*ba#r blah[54]quux{4,5,6}"}]
$ declare -A arr
$ while IFS= read -r line; do
eval "$line"
done < <(jq -M -r '.[] | #sh "arr[\(.WebsiteName)]+=\(.TestID)"' <<<"$data")
$ declare -p arr
declare -A arr='(["foo*ba#r blah[54]quux{4,5,6}"]="555" [SomeSite2]="1234" [SomeSite1]="123" )'
Here is a solution using only jq primitives.
.[]
| if .WebsiteName == "SomeSite1" then .TestID else empty end
This is essentially the same as Santiago's answer but if you are new to jq it may be informative because select/1 is defined as
def select(f): if f then . else empty end;

Read a json file with bash

I kwould like to read the json file from http://freifunk.in-kiel.de/alfred.json in bash and separate it into files named by hostname of each element in that json string.
How do I read json with bash?
How do I read json with bash?
You can use jq for that. First thing you have to do is extract the list of hostnames and save it to a bash array. Running a loop on that array you would then run again a query for each hostname to extract each element based on them and save the data through redirection with the filename based on them as well.
The easiest way to do this is with two instances of jq -- one listing hostnames, and another (inside the loop) extracting individual entries.
This is, alas, a bit inefficient (since it means rereading the file from the top for each record to extract).
while read -r hostname; do
[[ $hostname = */* ]] && continue # paranoia; see comments
jq --arg hostname "$hostname" \
'.[] | select(.hostname == $hostname)' <alfred.json >"out-${hostname}.json"
done < <(jq -r '.[] | .hostname' <alfred.json)
(The out- prefix prevents alfred.json from being overwritten if it includes an entry for a host named alfred).
You can use python one-liner in similar way like (I haven't checked):
curl -s http://freifunk.in-kiel.de/alfred.json | python -c '
import json, sys
tbl=json.load(sys.stdin)
for t in tbl:
with open(tbl[t]["hostname"], "wb") as fp:
json.dump(tbl[t], fp)
'