Extracting some fields from an AWS JSON using Powershell - json

Now that Powershell is Open Source and cross platform (with Powershell Core), I thought I'd give it a try again.
I had used in the past and at some point had figured how pipelines work, but it's not super intuitive. It should be, but it isn't, so I'm somewhat stuck...
Task at hand: parse and print several fields from the JSON output of a command. I could probably do it the old fashioned way, with external commands and string processing, à la bash, but I want to learn how to do it the Powershell way™.
And to clarify, I want to do it interactively, in a pipeline. I don't want to write a script, I want to learn how to do this kind of processing in a one-liner (or 2, at most, but basically with Powershell working as a REPL, not as a script tool).
Bash command:
aws cloudformation describe-stack-events --stack-name some-stack-here | jq ".StackEvents[] | [.Timestamp, .ResourceStatus, .ResourceType, .ResourceStatusReason] | join(\" \")"
What this does is take the input JSON and print just 3 fields I'm interested in.
Input JSON:
{
"StackEvents": [
{
"StackId": "arn:aws:cloudformation:some-region-here:some-number-here:stack/some-stack-here/some-id-here",
"EventId": "some-event-id-here",
"ResourceStatus": "UPDATE_COMPLETE",
"ResourceType": "AWS::CloudFormation::Stack",
"Timestamp": "some-date-here",
"StackName": "some-stack-here",
"PhysicalResourceId": "arn:aws:cloudformation:some-region-here:some-number-here:stack/some-stack-here/some-id-here",
"LogicalResourceId": "some-stack-here"
},
{
"StackId": "arn:aws:cloudformation:some-region-here:some-number-here:stack/some-stack-here/some-id-here",
"EventId": "some-event-id-here",
"ResourceStatus": "UPDATE_COMPLETE_CLEANUP_IN_PROGRESS",
"ResourceType": "AWS::CloudFormation::Stack",
"Timestamp": "some-date-here",
"StackName": "some-stack-here",
"PhysicalResourceId": "arn:aws:cloudformation:some-region-here:some-number-here:stack/some-stack-here/some-id-here",
"LogicalResourceId": "some-stack-here"
}
]
}
Command output:
"some-date-here UPDATE_COMPLETE AWS::CloudFormation::Stack "
"some-date-here UPDATE_COMPLETE_CLEANUP_IN_PROGRESS AWS::CloudFormation::Stack "
(I think this should be on Stackoverflow because the topic involves a pretty solid understanding of Powershell concepts, including .NET objects, much closer to programming than to sysadmining, i.e. SuperUser or so.)

You can do:
$j = aws cloudformation describe-stack-events --stack-name some-stack-here | ConvertFrom-Json
$j.StackEvents | % { "{0} {1} {2}" -f $_.Timestamp, $_.Resourcestatus, $_.ResourceType }
Use the ConvertFrom-Json cmdlet to store the command result into a powershell object, then you can select the StackEvents array and loop it to select the required values.
If you wanted to do it in one line:
(aws cloudformation describe-stack-events --stack-name some-stack-here | ConvertFrom-Json).StackEvents | %
{ "{0} {1} {2}" -f $_.Timestamp, $_.Resourcestatus, $_.ResourceType }

Related

JQ write each object to subdirectory file

I'm new to jq (around 24 hours). I'm getting the filtering/selection already, but I'm wondering about advanced I/O features. Let's say I have an existing jq query that works fine, producing a stream (not a list) of objects. That is, if I pipe them to a file, it produces:
{
"id": "foo"
"value": "123"
}
{
"id": "bar"
"value": "456"
}
Is there some fancy expression I can add to my jq query to output each object individually in a subdirectory, keyed by the id, in the form id/id.json? For example current-directory/foo/foo.json and current-directory/bar/bar.json?
As #pmf has pointed out, an "only-jq" solution is not possible. A solution using jq and awk is as follows, though it is far from robust:
<input.json jq -rc '.id, .' | awk '
id=="" {id=$0; next;}
{ path=id; gsub(/[/]/, "_", path);
system("mkdir -p " path);
print >> path "/" id ".json";
id="";
}
'
As you will need help from outside jq anyway (see #peak's answer using awk), you also might want to consider using another JSON processor instead which offers more I/O features. One that comes to my mind is mikefarah/yq, a jq-inspired processor for YAML, JSON, and other formats. It can split documents into multiple files, and since its v4.27.2 release it also supports reading multiple JSON documents from a single input source.
$ yq -p=json -o=json input.json -s '.id'
$ cat foo.json
{
"id": "foo",
"value": "123"
}
$ cat bar.json
{
"id": "bar",
"value": "456"
}
The argument following -s defines the evaluation filter for each output file's name, .id in this case (the .json suffix is added automatically), and can be manipulated to further needs, e.g. -s '"file_with_id_" + .id'. However, adding slashes will not result in subdirectories being created, so this (from here on comparatively easy) part will be left over for post-processing in the shell.

ws ec2 create-launch-template-version passing variables into json array

I have 4 environment variables on my laptop, ami_id, instance_type, key_name and security_group_ids. I am trying to create a launch template version using these variables but I do not know how to pass them into the JSON array properly
aws ec2 create-launch-template-version --launch-template-id lt-xxx --launch-template-data '{"ImageId":"$ami_id", "InstanceType": "$instance_type", "KeyName": "$key_name", "SecurityGroupIds": ["$security_group_ids"]}'
An error occurred (InvalidAMIID.Malformed) when calling the CreateLaunchTemplateVersion operation: The image ID '$ami_id' is not valid. The expected format is ami-xxxxxxxx or ami-xxxxxxxxxxxxxxxxx.
Using a here-document allows you to feed some readable text into a variable while expanding the shell variables, like this:
#!/bin/sh
ami_id=ami1234
instance_type=t3.nano
key_name=key1
security_group_ids=sg123,sg456
template_data=$(cat <<EOF
{
"ImageId":"$ami_id",
"InstanceType": "$instance_type",
"KeyName": "$key_name",
"SecurityGroupIds": ["$security_group_ids"]
}
EOF
)
echo "$template_data"
You can then test the JSON syntax with jq:
./template.sh | jq -c
{"ImageId":"ami1234","InstanceType":"t3.nano","KeyName":"key1","SecurityGroupIds":["sg123,sg456"]}

Search and extract value using JQ command line processor

I have a JSON file very similar to the following:
[
{
"uuid": "832390ed-58ed-4338-bf97-eb42f123d9f3",
"name": "Nacho"
},
{
"uuid": "5b55ea5e-96f4-48d3-a258-75e152d8236a",
"name": "Taco"
},
{
"uuid": "a68f5249-828c-4265-9317-fc902b0d65b9",
"name": "Burrito"
}
]
I am trying to figure out how to use the JQ command line processor to first find the UUID that I input and based on that output the name of the associated item. So for example, if I input UUID a68f5249-828c-4265-9317-fc902b0d65b9 it should search the JSON file, find the matching UUID and then return the name Burrito. I am doing this in Bash. I realize it may require some outside logic in addition to JQ. I will keep thinking about it and put an update here in a bit. I know I could do it in an overly complicated way, but I know there is probably a really simple JQ method of doing this in one or two lines. Please help me.
https://shapeshed.com/jq-json/#how-to-find-a-key-and-value
You can use select:
jq -r --arg query Burrito '.[] | select( .name == $query ) | .uuid ' tst.json

Convert JSON from AWS SSM to environment variables using jq

I have done some research on this and feel as if I'm about 80% there but struggling to adjust the jq output as required due to splitting one of the strings.
I'm trying to convert the JSON output from AWS SSM to environment variables.
AWS command
aws ssm get-parameters-by-path \
--path /qa/es \
--with-decryption \
--query 'Parameters[*].{Name:Name,Value:Value}' \
Output
[
{
"Name": "/qa/es/AWS_ACCESS_KEY_ID",
"Value": "ABC123"
},
{
"Name": "/qa/es/AWS_SECRET_ACCESS_KEY",
"Value": "abcdefghijkl"
},
{
"Name": "/qa/es/ENDPOINT",
"Value": "https://amazonaws.com"
}
]
My required output from jq, note I'm only after the environment variable AFTER the last /. There may be cases where this could be /qa/es/something/nested/ENV_VAR
AWS_ACCESS_KEY_ID=ABC123
AWS_SECRET_ACCESS_KEY=abcdefghijkl
ENDPOINT=https://amazonaws.com
Once I have this I can utilise the answer here to set the environment variables. Exporting JSON to environment variables
The closest I have got is
jq -r "map(\"\(try(.Name |= split(\"/\")))=\(.Value|tostring)\")|.[]" params.json
Which gives me
{"Name":["","qa","es","AWS_ACCESS_KEY_ID"],"Value":"ABC123"}=ABC123
{"Name":["","qa","es","AWS_SECRET_ACCESS_KEY"],"Value":"abcdefghijkl"}=abcdefghijkl
{"Name":["","qa","es","ENDPOINT"],"Value":"https://amazonaws.com"}=https://amazonaws.com
Close, but not close enough! Can anyone point me in the right direction here?
With the -r command-line option,
.[]
| "\(.Name|split("/")|.[-1])=\(.Value)"
yields:
AWS_ACCESS_KEY_ID=ABC123
AWS_SECRET_ACCESS_KEY=abcdefghijkl
ENDPOINT=https://amazonaws.com
This seems to correspond to what you've asked for, but this approach has the potential disadvantage that it assumes something about "=", so please be careful!
As the previous comment but use '#sh' to escape the value
| "\(.Name|split("/")|.[-1])=\(.Value | #sh)"

Pass bash variable (string) as jq argument to walk JSON object

This is seemingly a very basic question. I am new to JSON, so I am prepared for the incoming facepalm.
I have the following JSON file (test-app-1.json):
"application.name": "test-app-1",
"environments": {
"development": [
"server1"
],
"stage": [
"server2",
"server3"
],
"production": [
"server4",
"server5"
]
}
The intent is to use this as a configuration file and reference for input validation.
I am using bash 3.2 and will be using jq 1.4 (not the latest) to read the JSON.
The problem:
I need to return all values in the specified JSON array based on an argument.
Example: (how the documentation and other resources show that it should work)
APPLICATION_ENVIRONMENT="developement"
jq --arg appenv "$APPLICATION_ENVIRONMENT" '.environments."$env[]"' test-app-1.json
If executed, this returns null. This should return server1.
Obviously, if I specify text explicitly matching the JSON arrays under environment, it works just fine:
jq 'environments.development[]' test-app-1.json returns: server1.
Limitations: I am stuck to jq 1.4 for this project. I have tried the same actions in 1.5 on a different machine with the same null results.
What am I doing wrong here?
jq documentation: https://stedolan.github.io/jq/manual/
You have three issues - two typos and one jq filter usage issue:
set APPLICATION_ENVIRONMENT to development instead of developement
use variable name consistently: if you define appenv, use $appenv, not $env
address with .environments[$appenv]
When fixed, looks like this:
$ APPLICATION_ENVIRONMENT="development"
$ jq --arg appenv "$APPLICATION_ENVIRONMENT" '.environments[$appenv][]' test-app-1.json
"server1"