I am trying to use a grep to search a JSON output, I used a curl command to return the data from a particular codeship build and I want to use GREP to store said ID value in a variable. However after I run the command and try to echo out the value of the variable its blank.
Below are the commands:
export API_KEY=abc123
export PROJECT_ID=123456
export LAST_BUILD_ID=$(curl -s https://codeship.com/api/v1/projects/$PROJECT_ID.json?api_key=$API_KEY | grep -Eo '"builds":\[{"id":\d+' | grep -Eo --color=never '\d+' | tail -1)
export LAST_BUILD_URL=$(echo "https://codeship.com/api/v1/builds/$LAST_BUILD_ID/restart.json?api_key=$API_KEY")
My response : never use grep nor regex to parse json.
Instead, use a proper json parser.
In shell, take a look to jq.
Example, adapt it a bit :
#!/bin/bash
API_KEY=abc123
PROJECT_ID=123456
html=$(curl -s https://codeship.com/api/v1/projects/$PROJECT_ID.json?api_key=$API_KEY)
LAST_BUILD_ID=$(jq '.builds | .[] | .never' <<< "$html") # just guessing
LAST_BUILD_URL=$(echo "https://codeship.com/api/v1/builds/$LAST_BUILD_ID/restart.json?api_key=$API_KEY")
Note
If you provide the JSON, I will be able to be more specific with the jq command
Related
I have a bunch of secrets (key/value) pairs stored in AWS Secrets Manager. I tried to parse the secrets using jq as:
aws secretsmanager get-secret-value --secret-id <secret_bucket_name> | jq --raw-output '.SecretString' | jq -r .PASSWORD
It retrieves the value stored in .PASSWORD, but the problem is I not only want to retrieve the value stored in key but also want to retrieve the key/value in the following manner:
KEY_1="1234"
KEY_2="0000"
.
.
.
so on...
By running the above command I am not able to parse in this format and also for every key/value I have to run this command many times which is tedious. Am I doing something wrong or is there a better way of doing this?
This isn't related to python, but more related to behaviour of aws cli and jq. I come up with something like this.
aws secretsmanager get-secret-value --secret-id <secret_name> --output text --query SecretString | jq ".[]"
There are literally hundred different ways to format something like this.
aws cli itself has lot of options to filter output using --query option https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-output.html
Exact conversion you are looking for would require somwthing like this:
aws secretsmanager get-secret-value --secret-id <secret_name> --output text --query SecretString \
| jq -r 'to_entries[] | [.key, "=", "\"", .value, "\"" ] | #tsv' \
| tr -d "\t"
There has to be some better way of doing this!!
Try the snippet below. I tend to put these little helper filters into their own shell function <3
tokv() {
jq -r 'to_entries|map("\(.key|ascii_upcase)=\"\(.value|tostring)\"")|.[]'
}
$ echo '{"foo":"bar","baz":"fee"}' | tokv
FOO="bar"
BAZ="fee"
I am running a puppet bolt command query certain information from a set of servers in json format. I am piping it to jq.. Below is what I get
$ bolt command run "cat /blah/blah" -n #hname.txt -u uid --no-host-key-check --format json |jq -jr '.items[]|[.node],[.result.stdout]'
[
"node-name"
][
"stdout data\n"
]
What do I need to do to make it appear like below
["nodename":"stdout data"]
If you really want output that is not valid JSON, you will have to construct the output string, which can easily be done using string interpolation, e.g.:
jq -r '.items[] | "[\"\(.node)\",\"\(.result.stdout)\"]"'
#peak thank you.. that helped. Below is how it looks like
$ bolt command run "cat /blah/blah" -n #hname.txt -u UID --no-host-key-check --format json |jq -r '.items[] | "[\"\(.node)\",\"\(.result.stdout)\"]"'
["node name","stdout data
"]
I used a work around to get the data I needed by using the #csv flag to the command itself. Sharing with you below what worked.
$ bolt command run "cat /blah/blah" -n #hname.txt -u uid --no-host-key-check --format json |jq -jr '.items[]|[.node],[.result.stdout]|#csv'
""node-name""stdout.data
"
find dir1 | grep .json | python -mjson.tool $1 > /dev/null
I am using the above command but this doesn't take the files as inputs. What should i do to check for all the json files in a folder and validate whether its a proper json.
I found this a while back and have been using it:
for j in *.json; do python -mjson.tool "$j" > /dev/null || echo "INVALID $j" >&2; done;
I'm sure it compares similarly to dasho-o's answer, but am including it as another option.
You need to use 'xarg'.
The pipe find/grep will place the file names of the json file to STDIN. You need to create a command line with those file names. This is what xargs does:
find dir1 | grep .json | xargs -L1 python -mjson.tool > /dev/null
Side notes:
Since 'find' has filtering and execution predicates, more compact line can be created
find dir1 -name '*.json' -exec python -mjson.tool '{}' ';'
Also consider using 'jq' as light weight alternative to validating via python.
I want to parse a value or convert a value into json format.
I have no idea how to do it.
echo -e $(kubectl get pods "test-pod" -o jsonpath="{range .status.containerStatuses[*]}{.state}"\\n"{end}")
map[running:map[startedAt:2019-06-07T00:51:34Z]]
map[running:map[startedAt:2019-06-07T00:51:40Z]]
map[running:map[startedAt:2019-06-07T00:51:44Z]]
map[waiting:map[message:Back-off 5m0s restarting failed container=con4 pod=test-pod_test(609c90e4-88be-11e9-ba5f-fa163e9a67be) reason:CrashLoopBackOff]]
I would like to get only all containers' status like [running, running, running, waiting].
Thanks in advance.
You can achieve it using the jq and keys[] command in jq. The following will be the command to use:
kubectl get pods kube-dns-86f4d74b45-khd4z -n kube-system -o json | jq -r '.status.containerStatuses[].state | keys[]'
The above command will give the following output of all container running or waiting or any state
I have a file containing:
{"id":1,"jsonrpc":"2.0","result":{"speed":0}}
How would I be able to grep "0" after "speed":"?
I have tried 'grep -o -P "speed":{1}', not what I am looking for.
You should use jq (sudo apt-get install jq on raspbian) for this task.
echo '{"id":1,"jsonrpc":"2.0","result":{"speed":0}}' | jq .result.speed
Result: 0
Since you said in your question that you have a file "containing" this line, you might want to use grep first to get only the line you're interested in, otherwise jq might throw an error.
Example file:
abc
{"id":1,"jsonrpc":"2.0","result":{"speed":0}}
123
Running grep "speed" yourfile.txt | jq .result.speed would output 0.