I want to filter and project some data out of the list of secrets in an Azure keyvault. Actually, I want to filter by contentType and return the name. I've tried to do this using the --query option of az, but I didn't manage it.
So, I decided it to do it the traditional Unix way using a pipe and awk:
az keyvault secret list
--vault-name pickkeyvault
--query '[].[contentType,name]'
--output tsv
| awk '/^sftpuser/ { print $2}'
However, I at least would like to understand whether this is possible using az only.
Can anyone help me? Thanks!
Related
I am new to regex and Linux, I want to know how we can fetch JSON/XML data using regex using a Linux terminal. I am a windows user so currently working on the windows subsystem for Linux. I know I should use jq for JSON but the requirement is to use regex, the pattern that I will have will be same in every report so regex can be used even though it is not really recommended.
I wanna know two things
How I can test my regex in windows subsystem for Linux.
how I can add it in the shell script, as of now I am using jq.
This is how I am using jq to fetch data in shell script
cat abc.json | jq -r '.url'
So how i can achieve the same thing using regex ?
my abc.json is as below
{"url":"http://www.netcharles.com/orwell/essays.htm",
"domain":"netcharles.com",
"title":"Orwell Essays & Journalism Section - Charles' George Orwell Links",
"tags":["orwell","writing","literature","journalism","essays","politics","essay","reference","language","toread"],
"index":2931,
"time_created":1345419323,
"num_saves":24}
I tried this in windows subsystem for linux
sed -E '(url|title|tags)":"((\"|[^"])*)' abc.json
got this error
sed: -e expression #1, char 1: unknown command: `('
Expected output
\\"url\\":\\"http://www.netcharles.com/orwell/essays.htm\\",\\"domain\\":\\"netcharles.com\\",\\"title\\":\\"Orwell Essays & Journalism Section - Charles\\' George Orwell Links\\",\\"tags\\":[\\"orwell\\",\\"writing\\",\\"literature\\",\\"journalism\\",\\"essays\\",\\"politics\\",\\"essay\\",\\"reference\\",\\"language\\",\\"toread\\"],\\"index\\":2931,\\"time_created\\":1345419323,\\"num_saves\\":24
or could someone please tell me what would be regex for accessing something like this. ps - the value of first_name will be a string
cat user.json | jq -r '.user_data.username.first_name'
This question already has answers here:
How to remove double-quotes in jq output for parsing json files in bash?
(2 answers)
Closed 1 year ago.
The Problem:
I am trying to fetch credentials from AWS Secrets Manager in my terminal, however the Keys and Values I want needs to be in JSON, however they come with a lot of escape characters due to quotes.
The Scenario:
After I fire the aws secretsmanager get-secret-value --secret-id snowflake-access-uat command, I get the credentials as below:
{
"ARN": "arn:aws:secretsmanager:ap-regionnm-1:111111111111:secret:my-secret",
"Name": "snowflake-access-uat",
"VersionId": "dont-care",
"SecretString": "{\"sf-user\":\"USER_123_ADMIN\",\"sf-password\":\"FooBaarPassword\",\"sf-db\":\"MY_SPL_DB\",\"wh_name\":\"JOB_EXECUTOR\",\"sf-role\":\"JOB_EXECUTOR_ROLE\",\"sf-account\":\"icy-party\"}",
"VersionStages": [
"AWSCURRENT"
],
"CreatedDate": 1627104812.142
}
However, I am interested in Secret String only, for which I fire aws secretsmanager get-secret-value --secret-id snowflake-programmatic-access-uat | jq '.SecretString' command and receive this:
"{\"sf-user\":\"USER_123_ADMIN\",\"sf-password\":\"FooBaarPassword\",\"sf-db\":\"MY_SPL_DB\",\"wh_name\":\"JOB_EXECUTOR\",\"sf-role\":\"JOB_EXECUTOR_ROLE\",\"sf-account\":\"icy-party\"}"
But since it has multiple escape characters, I am unable to leverage it with jq tree. I tried to get from this link for reference but I'm unable to make it work. Besides, I need the Keys and Values to be variables in my bash session.
NOTE: I cannot use any third party tools, since I need to automate this on CodeBuild (Run time fresh instance will be selected)
The escape characters are there because you don't use -r with jq '.SecretString'. Change it to jq -r '.SecretString' and your output will instead be:
{"sf-user":"USER_123_ADMIN","sf-password":"FooBaarPassword","sf-db":"MY_SPL_DB","wh_name":"JOB_EXECUTOR","sf-role":"JOB_EXECUTOR_ROLE","sf-account":"icy-party"}
...which, being valid JSON, you can feed back into jq -r to retrieve individual fields.
SecretStringJson=$(... | jq -r '.SecretString')
### one jq call per field isn't the most efficient possible way but it's easy
sfUser=$(jq -r '.["sf-user"]' <<<"$SecretStringJson")
sfDb=$(jq -r '.["sf-db"]' <<<"$SecretStringJson")
# ...etc
I wrote a script to create sqs resource on local stack. I wanted to pass a value that I get from one cli command to the next but inside an inline json. Following is the section of the script in question.
arn=$(aws --endpoint-url=http://localhost:4576 sqs get-queue-attributes \
--queue-url http://localhost:4576/my_dead_letter_queue_url \
--query 'Attributes.QueueArn' \
--output text)
aws --endpoint-url=http://localhost:4576 sqs create-queue \
--queue-name my_queue \
--attributes \
'{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$arn\", \"maxReceiveCount\":\"5\"}"}'
So I'm trying to pass that "arn" variable but the cli is taking that as a string and trying to find a sqs with url "$arn" and fails. I also tried removing the quote. In that case, the error is malformed string.
Instead of the arn variable, if I use the arn value as string there, it works.
Can someone please show me how to pass that variable inside that inline json if it is possible?
Thank you for reading :)
Shahed
I was able to do the following with successful results, grant it it doesn't process the json (for that I'm just replacing tokens via sed), but I updated my example and tested it at least in bash with what I was doing:
#!/bin/bash
export awscmd="aws --region us-east-1 iam"
function setArn() {
${awscmd} list-policies --query 'Policies[?PolicyName==`'${1}'`].{ARN:Arn}' --output text
}
arn=$(setArn "some-policy-name")
echo '{"RedrivePolicy":"{"deadLetterTargetArn":"'$arn'", "maxReceiveCount":"5"}"}'
$ ./somearntest.sh
{"RedrivePolicy":"{"deadLetterTargetArn":"arn:aws:iam::############:policy/some-policy-name", "maxReceiveCount":"5"}"}
Notice the use of single tics to concatenate the output result outside of the string. This is in bash 4 and I removed the escaped \"s as I think that was added in error; ymmv.
The problem here is you are trying to expand a bash variable inside single quotes. Using single quotes like this is usually to pass a bunch of strings and unqoutable stuff as one argument. If you can't replace them with double quotes you'll have to resort to dirty eval hacks, which I do not recommend.
Here is an example:
$ arn=foobar
$ echo '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$arn\", \"maxReceiveCount\":\"5\"}"}'
{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$arn\", \"maxReceiveCount\":\"5\"}"}
$ eval echo '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$arn\", \"maxReceiveCount\":\"5\"}"}'
{RedrivePolicy:{"deadLetterTargetArn":"foobar", "maxReceiveCount":"5"}}
For more information I suggest to check How eval works and Expansion of variables inside single quotes
jq is a lightweight and flexible command-line JSON processor.
https://stedolan.github.io/jq/
Is there a jq command line tool or wrapper which lets you pipe output into it and interactively explore jq, with the JSON input in one pane and your interactively updating result in another pane, similar to jmespath.terminal ?
I'm looking for something similar to the JMESPath Terminal jpterm
"JMESPath exploration tool in the terminal"
https://github.com/jmespath/jmespath.terminal
I found this project jqsh but it's not maintained and it appears to produce a lot of errors when I use it.
https://github.com/bmatsuo/jqsh
I've used https://jqplay.org/ and it's a great web based jq learning tool. However, I want to be able to, in the shell, pipe the json output of a command into an interactive jq which allows me to explore and experiment with jq commands.
Thanks in advance!
I've been using jiq and I'm pretty happy with it.
https://github.com/fiatjaf/jiq
It's jid with jq.
You can drill down interactively by using jq filtering queries.
jiq uses jq internally, and it requires you to have jq in your PATH.
Using the aws cli
aws ec2 describe-regions --region-names us-east-1 us-west-1 | jiq
jiq output
[Filter]> .Regions
{
"Regions": [
{
"Endpoint": "ec2.us-east-1.amazonaws.com",
"RegionName": "us-east-1"
},
{
"Endpoint": "ec2.us-west-1.amazonaws.com",
"RegionName": "us-west-1"
}
]
}
https://github.com/simeji/jid
n.b. I'm not clear how strictly it follows jq syntax and feature set
You may have to roll-your-own.
Of course, jq itself is interactive in the sense that if you invoke it without specifying any JSON input, it will process STDIN interactively.
If you want to feed the same data to multiple programs, you could easily write your own wrapper. Over at github, there's a bash script named jqplay that has a few bells and whistles. For example, if the input command begins with | then the most recent result is used as input.
Example 1
./jqplay -c spark.json
Enter a jq filter (possibly beginning with "|"), or blank line to terminate:
.[0]
{"name":"Paddington","lovesPandas":null,"knows":{"friends":["holden","Sparky"]}}
.[1]
{"name":"Holden"}
| .name
"Holden"
| .[0:1]
"H"
| length
1
.[1].name
"Holden"
Bye.
Example 2
./jqplay -n
Enter a jq filter (possibly beginning and/or ending with "|"), or blank line to terminate:
?
An initial | signifies the filter should be applied to the previous jq
output.
A terminating | causes the next line that does not trigger a special
action to be appended to the current line.
Special action triggers:
:exit # exit this script, also triggered by a blank line
:help # print this help
:input PATHNAME ...
:options OPTIONS
:save PN # save the most recent output in the named file provided
it does not exist
:save! PN # save the most recent output in the named file
:save # save to the file most recently specified by a :save command
:show # print the OPTIONS and PATHNAMEs currently in effect
:! PN # equivalent to the sequence of commands
:save! PN
:input PN
? # print this help
# # ignore this line
1+2
3
:exit
Bye.
If you're using Emacs (or willing to) then JQ-mode allows you to run JQ filters interactively on the current JSON document buffer:
https://github.com/ljos/jq-mode
There is a new one: https://github.com/PaulJuliusMartinez/jless
JLess is a command-line JSON viewer designed for reading, exploring, and searching through JSON data.
JLess will pretty print your JSON and apply syntax highlighting.
Expand and collapse Objects and Arrays to grasp the high- and low-level structure of a JSON document. JLess has a large suite of vim-inspired commands that make exploring data a breeze.
JLess supports full text regular-expression based search. Quickly find the data you're looking for in long String values, or jump between values for the same Object key.
running aws ec2 describe-instances will return a json text similar to the following:
{
"Reservations": [
{
"Instances": [
"PublicDnsName": "ec2..."
"VpcId": "vpc-...",
...
"Instances": [
I know for each "Instance" I can extract the contents of a single field, for example PublicDnsName, using jq as follows:
jq '.Reservations[].Instances[].PublicDnsName' which will list the dns names for my instances
But how do I extract two or more fields and separate them by a space or comma or something? I want PublicDnsName and VpcId to be listed side-by-side for each Instance.
Specifically what I'm looking for is a list of instances where VpcId is null, undefined, or non-existent. In other words I'd like a list of my Classic instances and I need this through api so I can process the results.
Working with AWS CLI alone
I love jq and it's really really helpful to combine with aws cli but you can already do a lot simply with aws cli in your case.
But how do I extract two or more fields and separate them by a space or comma or something? I want PublicDnsName and VpcId to be listed side-by-side for each Instance.
aws ec2 describe-instances --query '
Reservations[*].Instances[].{PublicDnsName:PublicDnsName,VpcId:VpcId}' --output text
Specifically what I'm looking for is a list of instances where VpcId is null, undefined, or non-existent
If you search for empty VpcId, the following will work:
aws ec2 describe-instances --query '
Reservations[].Instances[?VpcId==``].[PublicDnsName]' --output text
Working with jq
As the original question is
How to tell jq to print multiples fields(keys) from aws ec2 cli
sure you can work with jq, you can get multiple fields with the following by separating by a comma
aws ec2 describe-instances |\
jq '.Reservations[].Instances[].PublicDnsName, .Reservations[].Instances[].VpcId'
You can better write it by factoring Reservations[].Instances:
aws ec2 describe-instances |\
jq '.Reservations[].Instances[] | .PublicDnsName, .VpcId'
and jq supports transformation into csv but it needs to take an array and to avoid issues with double-quotes, you can work with raw data:
aws ec2 describe-instances |\
jq -r '.Reservations[].Instances[]
| [.PublicDnsName, .VpcId]
| #csv'
The last will just product the expected result:
"ec2-some_id_region.amazonaws.com","vpc-some_other_id"