I try to use the AWS secrets manager in the linux system. I could use aws cli command
aws secretsmanager get-secret-value --secret-id abc_account --version-stage AWSCURRENT
to get following output
{
"ARN": "arn:aws:secretsmanager:us-east-1:123456789:secret:abc_account-XhteiW",
"Name": "abc_account",
"VersionId": "89637ef4-4594-4c63-9887-3f7d2c7ccc6f",
"SecretString": "{\"username\":\"abc_account\",\"password\":\"PASSWORD111\"}",
"VersionStages": [
"AWSCURRENT"
],
"CreatedDate": "2021-02-08T23:57:58.325000-05:00"
}
what I need is to save the password PASSWORD111 into a variable var1 in the linux. something like
var1=$(aws secretsmanager get-secret-value --secret-id svc_vma_insights_data_platform --version-stage AWSCURRENT | awk XXXXXX )
or
var1=$(aws secretsmanager get-secret-value --secret-id svc_vma_insights_data_platform --version-stage AWSCURRENT | grep XXXXXX )
This is extracting the secret string from the JSON output, and then extracting the password from that JSON:
passwd=$(aws ... | jq -r '.SecretString' | jq -r '.password')
On linux you may try this gnu grep:
var1=$(aws ... | grep -oP 'password\W+\K[^"\\]+')
echo "$var1"
PASSWORD111
Command regex:
password\W+: Match text password followed by 1+ non-word characters
\K: Reset match info
[^"\\]+: Match 1+ of any character that is not a " and not a \
Related
I am calling python command which will return data is JSON key-value pair.
I have put python command and other command in one shell script named as - a.sh
Code (a.sh):
cd /home/drg/Code/dth
a=$(python3 main.py -z shell -y droub -i 56)
echo "$a"
When I am calling this script I am getting output as:
{'password': 'XYZ', 'name': 'Stguy', 'port': '5412', 'host': 'igtet', 'db_name': 'test3'}
And after getting this output I want to pass the output value like password, name to psql command to run postgresql query.
So, what I want is that I should be able to store password value in one variable, name in one variable like:
a= xyz
b=Stguy
p= port
So, that I can use this variables to pass in psql query as:
psql -h $a -p $p -U $b -d $db -c "CREATE SCHEMA IF NOT EXISTS ${sname,,};"
Can someone please help me with this?
Note: Env is linux(Centos 8)
Thanks in advance!
One way of solving this could be a combination of jq for value extraction and shell-builtin read for multiple variable assignment:
JSON='{"name": "Stguy", "port": 5412, "host": "igtet", "db_name": "test3"}'
read -r a b c <<<$( echo $JSON | jq -r '"\(.host) \(.port) \(.name)"' )
echo "a: $a, b: $b, c: $c"
doing jq string interpolation "\( )" to print result in one line
You can aslo go with sed or awk:
PSQL="$( python3 main.py -z shell -y droub -i 56 | sed "s/^[^:]*: *'\([^']*\)'[^:]*: *'\([^']*\)'[^:]*: *'\([^']*\)'[^:]*: *'\([^']*\)'[^:]*: *'\([^']*\)'}/psql -h '\4' -p '\1' -U '\2' -d '\5'/")"
[ "${PSQL:0:5}" = "psql " ] && ${PSQL} -c "CREATE SCHEMA IF NOT EXISTS ${sname,,};"
For security consideration, i urge you anyway to avoid passing account data (user passwd) through environment variables.
It would be better if your python script had an option to directly launch psql with required parameters.
azure cli az group list return data like
[
"demo3",
"demo",
"demo2",
"NetworkWatcherRG"
]
I'd like to foreach it's value on ubuntu bash then printing below result
demo3
demo
demo2
NetworkWatcherRG
What I've tried :
I tried below script
jq -c '.[]' $(az group list) | while read i; do echo $i ;done
but get image's error
Your command expands to this (see it for yourself with set -x):
jq -c '.[]' '[' '"demo3",' '"demo",' '"demo2",' '"NetworkWatcherRG"' ']'
The command substitution is replaced with the command output, but jq doesn't expect a JSON body as a parameter – either files containing JSON, or a stream in standard input. Since all the parameters ([, "demo3" etc.) are not filenames, you see the errors you do.
You could have Bash make it look like it's a file with process substitution:
jq -c '.[]' <(az group list)
or, more portably, use pipes:
az group list | jq -c '.[]'
Notice that quoting wouldn't help here either: if you ran
jq -c '.[] "$(az group list)"
it would expand to
jq -c '.[]' '[
"demo3",
"demo",
"demo2",
"NetworkWatcherRG"
]'
and jq would try to open a file with the name
[
"demo3",
"demo",
"demo2",
"NetworkWatcherRG"
]
which does not exist.
My secrets are stored in AWS Secrets Manager in the following format:
KEY_1 "XXXX"
KEY_2 0
I am trying to retrieve the secrets in the following format:
KEY_1="XXXX"
KEY_2=0
that means the same way they're stored in Secrets Manager with = symbol. I am using jq to parse in the following manner:
aws secretsmanager get-secret-value --secret-id <secret_name> --output text --query SecretString \
| jq -r 'to_entries[] | [.key, "=", "\"", .value, "\"" ] | #tsv' \
| tr -d "\t"
This command is working fine but with a small problem. This command is adding an extra double quotation mark ("") in the value field. It is giving the following output:
KEY_1=""XXXX""
KEY_2="0"
But I do not want these extra double quotation marks to be added. I want the output should be in the same format as described above while retrieving above i.e, without extra quotation marks.
You are outputting as text, what is the need for jq ?
--output text
If you want to output as json you can use
--output json
I receive some json that I process until it becomes just text lines. In the first line there's a value that I would like to keep in a variable and all the rest after the first line should be displayed with less or other utils.
Can I do this without using a temporary file?
The context is this:
aws logs get-log-events --log-group-name "$logGroup" --log-stream-name "$logStreamName" --limit "$logSize" |
jq '{message:.nextForwardToken}, .events[] | .message' |
sed 's/^"//g' | sed 's/"$//g'
In the first line there's the nextForwardToken that I want to put in the variable and all the rest is log messages.
The json looks like this:
{
"events": [
{
"timestamp": 1518081460955,
"ingestionTime": 1518081462998,
"message": "08.02.2018 09:17:40.955 [SimpleAsyncTaskExecutor-138] INFO o.s.b.c.l.support.SimpleJobLauncher - Job: [SimpleJob: [name=price-update]] launched with the following parameters: [{time=1518081460875, sku=N-W7ZLH9U737B|N-XIBH22XQE87|N-3EXIRFNYNW0|N-U19C031D640|N-6TQ1847FQE6|N-NF0XCNG0029|N-UJ3H0OZROCQ|N-W2JKJD4S6YP|N-VEMA4QVV3X1|N-F40J6P2VM01|N-VIT7YEAVYL2|N-PKLKX1PAUXC|N-VPAK74C75DP|N-C5BLYC5HQRI|N-GEIGFIBG6X2|N-R0V88ZYS10W|N-GQAF3DK7Y5Z|N-9EZ4FDDSQLC|N-U15C031D668|N-B8ELYSSFAVH}]"
},
{
"timestamp": 1518081461095,
"ingestionTime": 1518081462998,
"message": "08.02.2018 09:17:41.095 [SimpleAsyncTaskExecutor-138] INFO o.s.batch.core.job.SimpleStepHandler - Executing step: [index salesprices]"
},
{
"timestamp": 1518082421586,
"ingestionTime": 1518082423001,
"message": "08.02.2018 09:33:41.586 [upriceUpdateTaskExecutor-3] DEBUG e.u.d.a.j.d.b.StoredMasterDataReader - Reading page 1621"
}
],
"nextBackwardToken": "b/33854347851370569899844322814554152895248902123886870536",
"nextForwardToken": "f/33854369274157730709515363051725446974398055862891970561"
}
I need to put in a variable this:
f/33854369274157730709515363051725446974398055862891970561
and display (or put in an other variable) the messages:
08.02.2018 09:17:40.955 [SimpleAsyncTaskExecutor-138] INFO o.s.b.c.l.support.SimpleJobLauncher - Job: [SimpleJob: [name=price-update]] launched with the following parameters: [{time=1518081460875, sku=N-W7ZLH9U737B|N-XIBH22XQE87|N-3EXIRFNYNW0|N-U19C031D640|N-6TQ1847FQE6|N-NF0XCNG0029|N-UJ3H0OZROCQ|N-W2JKJD4S6YP|N-VEMA4QVV3X1|N-F40J6P2VM01|N-VIT7YEAVYL2|N-PKLKX1PAUXC|N-VPAK74C75DP|N-C5BLYC5HQRI|N-GEIGFIBG6X2|N-R0V88ZYS10W|N-GQAF3DK7Y5Z|N-9EZ4FDDSQLC|N-U15C031D668|N-B8ELYSSFAVH}]
08.02.2018 09:17:41.095 [SimpleAsyncTaskExecutor-138] INFO o.s.batch.core.job.SimpleStepHandler - Executing step: [index salesprices]
08.02.2018 09:33:41.586 [upriceUpdateTaskExecutor-3] DEBUG e.u.d.a.j.d.b.StoredMasterDataReader - Reading page 1621
Thanks in advance for your help.
You might consider it a bit of trick, but you can use tee to pipe all the output to stderr and fetch the one line you want for your variable with head:
var="$(command | tee /dev/stderr | head -n 1)"
Or you can solve this with a bit of scripting:
first=true
while read -r line; do
if $first; then
first=false
var="$line"
fi
echo "$line"
done < <(command)
If you are interested in storing the contents to variables, use mapfile or read on older bash versions.
Just using read to get the first line do. I've added -r flag to jq print output without quotes
read -r token < <(aws logs get-log-events --log-group-name "$logGroup" --log-stream-name "$logStreamName" --limit "$logSize" | jq -r '{message:.nextForwardToken}, .events[] | .message')
printf '%s\n' "$token"
Or using mapfile
mapfile -t output < <(aws logs get-log-events --log-group-name "$logGroup" --log-stream-name "$logStreamName" --limit "$logSize" | jq -r '{message:.nextForwardToken}, .events[] | .message')
and loop through the array. The first element will always contain the token-id you want.
printf '%s\n' "${output[0]}"
Rest of the elements can be iterated over,
for ((i=1; i<${#output[#]}; i++)); do
printf '%s\n' "${output[i]}"
done
Straightforwardly:
aws logs get-log-events --log-group-name "$logGroup" \
--log-stream-name "$logStreamName" --limit "$logSize" > /tmp/log_data
-- set nextForwardToken variable:
nextForwardToken=$(jq -r '.nextForwardToken' /tmp/log_data)
echo $nextForwardToken
f/33854369274157730709515363051725446974398055862891970561
-- print all message items:
jq -r '.events[].message' /tmp/log_data
08.02.2018 09:17:40.955 [SimpleAsyncTaskExecutor-138] INFO o.s.b.c.l.support.SimpleJobLauncher - Job: [SimpleJob: [name=price-update]] launched with the following parameters: [{time=1518081460875, sku=N-W7ZLH9U737B|N-XIBH22XQE87|N-3EXIRFNYNW0|N-U19C031D640|N-6TQ1847FQE6|N-NF0XCNG0029|N-UJ3H0OZROCQ|N-W2JKJD4S6YP|N-VEMA4QVV3X1|N-F40J6P2VM01|N-VIT7YEAVYL2|N-PKLKX1PAUXC|N-VPAK74C75DP|N-C5BLYC5HQRI|N-GEIGFIBG6X2|N-R0V88ZYS10W|N-GQAF3DK7Y5Z|N-9EZ4FDDSQLC|N-U15C031D668|N-B8ELYSSFAVH}]
08.02.2018 09:17:41.095 [SimpleAsyncTaskExecutor-138] INFO o.s.batch.core.job.SimpleStepHandler - Executing step: [index salesprices]
08.02.2018 09:33:41.586 [upriceUpdateTaskExecutor-3] DEBUG e.u.d.a.j.d.b.StoredMasterDataReader - Reading page 1621
I believe the following meets the stated requirements, assuming a bash-like environment:
x=$(aws ... |
tee >(jq -r '.events[] | .message' >&2) |
jq .nextForwardToken) 2>&1
This makes the item of interest available as the shell variable $x.
Notice that the string manipulation using sed can be avoided by using the -r command-line option of jq.
Calling jq just once
x=$(aws ... |
jq -r '.nextForwardToken, (.events[] | .message)' |
tee >(tail -n +2 >&2) |
head -n 1) 2>&1
echo "x=$x"
I was wondering how to parse the CURL JSON output from the server into variables.
Currently, I have -
curl -X POST -H "Content: agent-type: application/x-www-form-urlencoded" https://www.toontownrewritten.com/api/login?format=json -d username="$USERNAME" -d password="$PASSWORD" | python -m json.tool
But it only outputs the JSON from the server and then have it parsed, like so:
{
"eta": "0",
"position": "0",
"queueToken": "6bee9e85-343f-41c7-a4d3-156f901da615",
"success": "delayed"
}
But how do I put - for example the success value above returned from the server into a variable $SUCCESS and have the value as delayed & have queueToken as a variable $queueToken and 6bee9e85-343f-41c7-a4d3-156f901da615 as a value?
Then when I use-
echo "$SUCCESS"
it shows this as the output -
delayed
And when I use
echo "$queueToken"
and the output as
6bee9e85-343f-41c7-a4d3-156f901da615
Thanks!
Find and install jq (https://stedolan.github.io/jq/). jq is a JSON parser. JSON is not reliably parsed by line-oriented tools like sed because, like XML, JSON is not a line-oriented data format.
In terms of your question:
source <(
curl -X POST -H "$content_type" "$url" -d username="$USERNAME" -d password="$PASSWORD" |
jq -r '. as $h | keys | map(. + "=\"" + $h[.] + "\"") | .[]'
)
The jq syntax is a bit weird, I'm still working on it. It's basically a series of filters, each pipe taking the previous input and transforming it. In this case, the end result is some lines that look like variable="value"
This answer uses bash's "process substitution" to take the results of the jq command, treat it like a file, and source it into the current shell. The variables will then be available to use.
Here's an example of Extract a JSON value from a BASH script
#!/bin/bash
function jsonval {
temp=`echo $json | sed 's/\\\\\//\//g' | sed 's/[{}]//g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w $prop`
echo ${temp##*|}
}
json=`curl -s -X GET http://twitter.com/users/show/$1.json`
prop='profile_image_url'
picurl=`jsonval`
`curl -s -X GET $picurl -o $1.png`
A bash script which demonstrates parsing a JSON string to extract a
property value. The script contains a jsonval function which operates
on two variables, json and prop. When the script is passed the name of
a twitter user it attempts to download the user's profile picture.
You could use perl module on command line:
1st, ensure they is installed, under debian based, you could
sudo apt-get install libjson-xs-perl
But for other OS, you could install perl modules via CPAN (the Comprehensive Perl Archive Network):
cpan App::cpanminus
cpan JSON::XS
Note: You may have to run this with superuser privileges.
then:
curlopts=(-X POST -H
"Content: apent-type: application/x-www-form-urlencoded"
-d username="$USERNAME" -d password="$PASSWORD")
curlurl=https://www.toontownrewritten.com/api/login?format=json
. <(
perl -MJSON::XS -e '
$/=undef;my $a=JSON::XS::decode_json <> ;
printf "declare -A Json=\047(%s)\047\n", join " ",map {
"[".$_."]=\"".$a->{$_}."\""
} qw|queueToken success eta position|;
' < <(
curl "${curlopts[#]}" $curlurl
)
)
The line qw|...| let you precise which variables you want to be driven... This could be replaced by keys $a, but could have to be debugged as some characters is forbiden is associative arrays values names.
echo ${Json[queueToken]}
6bee9e85-343f-41c7-a4d3-156f901da615
echo ${Json[eta]}
0