Retrieve secrets from AWS Secrets Manager - json

I have a bunch of secrets (key/value) pairs stored in AWS Secrets Manager. I tried to parse the secrets using jq as:
aws secretsmanager get-secret-value --secret-id <secret_bucket_name> | jq --raw-output '.SecretString' | jq -r .PASSWORD
It retrieves the value stored in .PASSWORD, but the problem is I not only want to retrieve the value stored in key but also want to retrieve the key/value in the following manner:
KEY_1="1234"
KEY_2="0000"
.
.
.
so on...
By running the above command I am not able to parse in this format and also for every key/value I have to run this command many times which is tedious. Am I doing something wrong or is there a better way of doing this?

This isn't related to python, but more related to behaviour of aws cli and jq. I come up with something like this.
aws secretsmanager get-secret-value --secret-id <secret_name> --output text --query SecretString | jq ".[]"
There are literally hundred different ways to format something like this.
aws cli itself has lot of options to filter output using --query option https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-output.html
Exact conversion you are looking for would require somwthing like this:
aws secretsmanager get-secret-value --secret-id <secret_name> --output text --query SecretString \
| jq -r 'to_entries[] | [.key, "=", "\"", .value, "\"" ] | #tsv' \
| tr -d "\t"
There has to be some better way of doing this!!

Try the snippet below. I tend to put these little helper filters into their own shell function <3
tokv() {
jq -r 'to_entries|map("\(.key|ascii_upcase)=\"\(.value|tostring)\"")|.[]'
}
$ echo '{"foo":"bar","baz":"fee"}' | tokv
FOO="bar"
BAZ="fee"

Related

Filtering array using curl (qBittorrent Web API)

Hoping someone can help me. I'm trying to understand the qBittorrent Web API. At the moment I'm listing all the paused torrents with:
curl -i http://localhost:8080/api/v2/torrents/info?category=test
The problem is that lists the whole JSON array - my question is can I just display the "name" or "hash" fields? This is all using curl through cmd, but I've tried this in Git Bash & Powershell:
[{"eta":8640000,"f_l_piece_prio":false,"force_start":false,"hash":"8419d48d86a14335c83fdf4930843438a2f75a6b","last_activity":1664863523,"magnet_uri":"","max_seeding_time":0,"**name**":"TestTorrentName","num_complete":12,"num_incomplete":1,"num_leechs":0,"num_seeds":0,"priority":0,"progress":1,"ratio":0,"ratio_limit":-2,"save_path":"F:\\Completed\\test\\","seeding_time":0,"seeding_time_limit":-2,"seen_complete":1664863523,"seq_dl":false,"size":217388295,"state":"pausedUP","super_seeding":false,"tags":"","time_active":569,"total_size":217388295,"tracker":"udp://open.stealth.si:80/announce","trackers_count":10,"up_limit":-1,"uploaded":0,"uploaded_session":0,"upspeed":0}]
I've tried the following that should work according to https://jqplay.org/ - see screenshot
curl -i http://localhost:8080/api/v2/torrents/info?category=test | jq --raw-output '.[] | .name'
But unfortunately I'm getting the following error:
curl -i http://localhost:8080/api/v2/torrents/info?category=test | jq --raw-output '.[] | .name'
% Total % Received % Xferd Average Speed Time '.name'' is not recognized as an internal or external command,
operable program or batch file.
Ti
curl -i http://localhost:8080/api/v2/torrents/info?category=test | jq --raw-output '.[] | .name'
The -i let curl give some header info, that is parsed to jq, but jq can only parse JSON end therefore fails.
Remove the -i and optionally replace it with -s to remove the stats:
curl -s http://localhost:8080/api/v2/torrents/info?category=test | jq --raw-output '.[] | .name'

Parsing json output using jq -jr

I am running a puppet bolt command query certain information from a set of servers in json format. I am piping it to jq.. Below is what I get
$ bolt command run "cat /blah/blah" -n #hname.txt -u uid --no-host-key-check --format json |jq -jr '.items[]|[.node],[.result.stdout]'
[
"node-name"
][
"stdout data\n"
]
What do I need to do to make it appear like below
["nodename":"stdout data"]
If you really want output that is not valid JSON, you will have to construct the output string, which can easily be done using string interpolation, e.g.:
jq -r '.items[] | "[\"\(.node)\",\"\(.result.stdout)\"]"'
#peak thank you.. that helped. Below is how it looks like
$ bolt command run "cat /blah/blah" -n #hname.txt -u UID --no-host-key-check --format json |jq -r '.items[] | "[\"\(.node)\",\"\(.result.stdout)\"]"'
["node name","stdout data
"]
I used a work around to get the data I needed by using the #csv flag to the command itself. Sharing with you below what worked.
$ bolt command run "cat /blah/blah" -n #hname.txt -u uid --no-host-key-check --format json |jq -jr '.items[]|[.node],[.result.stdout]|#csv'
""node-name""stdout.data
"

How to get public dns name from aws ec2 describe-instances

I'm trying to make simple script which queries my ec2 instances and gets public dns name of instances which matches my filter. There is my first shot:
#!/bin/bash
aws ec2 describe-instances \
--filters "Name=tag:app,Values=swarm-cluster" \
"Name=tag:role,Values=manager" \
--query "Reservations[*].Instances[*].PublicDnsName"
It almost works but I get something ugly:
[
[
""
],
[
"ec2-xxx-xxx-xxx-xxx.venus-central-1.compute.amazonaws.com"
]
]
I want just list of FQDNs, one per line. How to format output?
I know, I can do it with tr, sed and so on but I'd like use more sophisticated way. :)
You can just append --output text to your CLI call to get a text output.
Ref - https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-output.html
By using jq you can parse the JSON response to get what you want.
Example:
#!/bin/bash
aws ec2 describe-instances \
--filters "Name=tag:app,Values=swarm-cluster" \
"Name=tag:role,Values=manager" \
--query "Reservations[*].Instances[*].PublicDnsName" | jq ".[0][1]"
That will give you:
"ec2-xxx-xxx-xxx-xxx.venus-central-1.compute.amazonaws.com"
for i in `cat kafka_instance_names`
do
aws ec2 describe-instances --filters "Name=tag:Name,Values=$i" --query "Reservations[*].Instances[*].PublicDnsName" | grep ec
done
"ec2-54-166-168-168.compute-1.amazonaws.com"
"ec2-52-72-30-88.compute-1.amazonaws.com"

jq filter for AWS for description without calling jq twice?

I have this to filter out snapshots with Jenkins in the description. Is there a more efficient way to do the same thing?
aws --region eu-west-1 ec2 describe-snapshots | jq '.Snapshots[] |\ select(.Description | contains("Jenkins"))' | jq -r '.SnapshotId'
Maybe something like this, You can use JMESPath Query inside your cli statement.
aws --region eu-west-1 ec2 describe-snapshots --query 'Snapshots[?contains(Description, `Jenkins`) == `false`]'

Store Codeship build ID in variable using GREP

I am trying to use a grep to search a JSON output, I used a curl command to return the data from a particular codeship build and I want to use GREP to store said ID value in a variable. However after I run the command and try to echo out the value of the variable its blank.
Below are the commands:
export API_KEY=abc123
export PROJECT_ID=123456
export LAST_BUILD_ID=$(curl -s https://codeship.com/api/v1/projects/$PROJECT_ID.json?api_key=$API_KEY | grep -Eo '"builds":\[{"id":\d+' | grep -Eo --color=never '\d+' | tail -1)
export LAST_BUILD_URL=$(echo "https://codeship.com/api/v1/builds/$LAST_BUILD_ID/restart.json?api_key=$API_KEY")
My response : never use grep nor regex to parse json.
Instead, use a proper json parser.
In shell, take a look to jq.
Example, adapt it a bit :
#!/bin/bash
API_KEY=abc123
PROJECT_ID=123456
html=$(curl -s https://codeship.com/api/v1/projects/$PROJECT_ID.json?api_key=$API_KEY)
LAST_BUILD_ID=$(jq '.builds | .[] | .never' <<< "$html") # just guessing
LAST_BUILD_URL=$(echo "https://codeship.com/api/v1/builds/$LAST_BUILD_ID/restart.json?api_key=$API_KEY")
Note
If you provide the JSON, I will be able to be more specific with the jq command