Reference value from state’s input, using JSONPath syntax in a SSMSendCommand API step through parameter which expects an array - json

I have on a AWS state machine the following step defined for api aws-sdk:ssm:sendCommand
{
"Type": "Task",
"Parameters": {
"DocumentName.$": "$.result.DocumentName",
"InstanceIds.$": "$..Dimensions[?(#.Name=~/.*InstanceId.*/)].Value",
"MaxErrors": "0",
"MaxConcurrency": "100%",
"CloudWatchOutputConfig": {
"CloudWatchLogGroupName": "diskspace-log",
"CloudWatchOutputEnabled": true
},
"Parameters": {
"workingDirectory": [
""
],
"executionTimeout": [
"3600"
],
"commands": [
"echo -------------------Mounting volume without signals $..Dimensions[?(#.Name=~/.*device.*/)].Value---------------------",
"echo",
"mount $..Dimensions[?(#.Name=~/.*device.*/)].Value"
]
}
}
}
The section: "commands": [] expects an array.
"commands" should accept input reference as any other parameter in the schema, so in theory will be posible to use json path parameters (Example: "size.$": "$.product.details.size") for referencing needed parameters from input.
https://docs.aws.amazon.com/step-functions/latest/dg/input-output-inputpath-params.html
This following example works without using input referencing :
"commands": [
"echo -------------------Mounting /dev/ebs---------------------",
"echo",
"mount /dev/ebs"
]
But I need to reference from input, hardcoded values won't work for me. I tried, but no working.
"commands": [
"echo -------------------Mounting volume without signals $..Dimensions[?(#.Name=~/.*device.*/)].Value---------------------",
"echo",
"mount $..Dimensions[?(#.Name=~/.*device.*/)].Value"
]
Also tried, not working also:
"commands.$": "States.Array(States.Format('echo -------------------Mounting volume without signals {} ---------------------', $..Dimensions[?(#.Name=~/.*device.*/)].Value),'echo',States.Format('mount {}', $..Dimensions[?(#.Name=~/.*device.*/)].Value))"
I believe some of the provided intrinsic functions will help on achieving the expected result but I'm lost on how to properly set up the syntax.
https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-intrinsic-functions.html#asl-intrsc-func-arrays
The step calls a RunShellScript type of documentCommand.
And executes the commands provided on parameters in the step of the state machine.
I got on the output:
States.Format('echo -------------------Mounting volume without signals {} ---------------------', $..Dimensions[?(#.Name=~/.*device.*/)].Value)'
Its not detecting the input reference, I expect to output.
-------------------Mounting volume without signals /dev/ebs ---------------------
and in the background execute:
mount /dev/ebs

I was able to send the commands through a Pass State Flow, here is the definition:
{
"Type": "Pass",
"Next": "SendCommand",
"ResultPath": "$.ForArgs",
"Parameters": {
"Params": {
"Args": [
{
"Arg1": "ec2-metadata -i"
},
{
"Arg2": "echo"
},
{
"Arg3.$": "States.Format('echo -------------------Mounting volume without signals {} ---------------------', States.ArrayGetItem($..Dimensions[?(#.Name=~/.*device.*/)].Value, 0))"
},
{
"Arg4": "echo"
},
{
"Arg5.$": "States.Format('mount {}', States.ArrayGetItem($..Dimensions[?(#.Name=~/.*device.*/)].Value, 0))"
},
{
"Arg6.$": "States.Format('echo Checking if device {} is mounted', States.ArrayGetItem($..Dimensions[?(#.Name=~/.*device.*/)].Value, 0))"
},
{
"Arg7.$": "States.Format('if findmnt --source \"{}\" >/dev/null', States.ArrayGetItem($..Dimensions[?(#.Name=~/.*device.*/)].Value, 0))"
},
{
"Arg8": "\tthen echo device is mounted"
},
{
"Arg9": "\telse echo device is not mounted"
},
{
"Arg10": "fi"
}
]
}
}
}
Next on the sendCommandApi:
"commands.$": "$.ForArgs.Params.Args[*][*]"

Related

Integrate json values into another file

I'm trying to update an existing json file from values in another json file using jq in a bash shell.
I've got a settings json file
{
"Logging": {
"MinimumLevel": {
"Default": "Information",
"Override": "Warning"
},
"WriteTo": [
{
"Name": "File",
"Args": {
"path": "./logs/log-.txt",
"rollingInterval": "Day"
}
}
]
},
"Settings": {
"DataServerUrl": "https://address.to.server.com",
"ServerKey": "1f969476798adfe95114dd28ed3a3ff"
"ServerTimeZone": "Mountain Standard Time",
"MaxOccupantCount": 6
}
}
In an integration step, I'm attempting to incorporate values for specific environments (think dev/staging/prod) from an external json file with limited setting values. An example of such a file is
{
"DataServerUrl": "https://dev.server.addr.com",
"ServerKey": "2a4d99233efea456b95114aa23ed342ae"
}
I can get to the data using jq. I can update the data using jq if I hard-code the updates. I'm looking for something general to take in any environment settings values and update them in the base settings file. My searches suggest I can do this in a single step without knowing the specific values. A command similar to
jq -r 'to_entries[]' settings.dev.json |
while IFS= read -r key value; do
jq -r '.[$key] |= [$value]' settings.json
done
What happens is I get error messages stating jq: error: $key is not defined at <top-level> (as well as the same message for $value). The messages appear several times in pairs. settings.json is not changed. Now, this makes partial sense because the output from just jq -r 'to_entries[]' settings.dev.json looks like (empty space in this output is included as produced by the command).
"key": "DataServerUrl",
"value": "https://dev.server.addr.com"
"key": "ServerKey",
"value": "2a4d99233efea456b95114aa23ed342ae"
How do I go about iterating over the values in the environment settings file such that I can use those values to update the base settings file for further processing (i.e., publishing to the target environment)?
The simplest way is to provide both files and address the second one using input. That way, all you need is the assignment:
jq '.Settings = input' settings.json insert.json
{
"Logging": {
"MinimumLevel": {
"Default": "Information",
"Override": "Warning"
},
"WriteTo": [
{
"Name": "File",
"Args": {
"path": "./logs/log-.txt",
"rollingInterval": "Day"
}
}
]
},
"Settings": {
"DataServerUrl": "https://dev.server.addr.com",
"ServerKey": "2a4d99233efea456b95114aa23ed342ae"
}
}
Demo
You could do something like
jq -s '.[1] as $insert | .[0].Settings |= $insert | .[0]' settings.json insert.json
Where we :
slurp both files
Save insert.json to a variable called $insert
Append (|=) $insert to .[0].Settings
Show only the first file .[0]
So the output will become:
{
"Logging": {
"MinimumLevel": {
"Default": "Information",
"Override": "Warning"
},
"WriteTo": [
{
"Name": "File",
"Args": {
"path": "./logs/log-.txt",
"rollingInterval": "Day"
}
}
]
},
"Settings": {
"DataServerUrl": "https://dev.server.addr.com",
"ServerKey": "2a4d99233efea456b95114aa23ed342ae"
}
}

jq output is empty when tag name does not exist

When I run the jq command to parse a json document from the amazon cli I have the following problem.
I’m parsing through the IP address and a tag called "Enviroment". The enviroment tag in the instance does not exist therefore it does not throw me any result.
Here's an example of the relevant output returned by the AWS CLI
{
"Reservations": [
{
"Instances": [
{
"PrivateIpAddress": "10.0.0.1",
"Tags": [
{
"Key": "Name",
"Value": "Balance-OTA-SS_a"
},
{
"Key": "Environment",
"Value": "alpha"
}
]
}
]
},
{
"Instances": [
{
"PrivateIpAddress": "10.0.0.2",
"Tags": [
{
"Key": "Name",
"Value": "Balance-OTA-SS_a"
}
]
}
]
}
]
}
I’m running the following command
aws ec2 describe-instances --filters "Name=tag:Name,Values=Balance-OTA-SS_a" | jq -c '.Reservations[].Instances[] | ({IP: .PrivateIpAddress, Ambiente: (.Tags[]|select(.Key=="Environment")|.Value)})'
## output
empty
How do I show the IP address in the output of the command even if the enviroment tag does not exist?
Regards,
Let's assume this input:
{
"Reservations": [
{
"Instances": [
{
"PrivateIpAddress": "10.0.0.1",
"Tags": [
{
"Key": "Name",
"Value": "Balance-OTA-SS_a"
},
{
"Key": "Environment",
"Value": "alpha"
}
]
}
]
},
{
"Instances": [
{
"PrivateIpAddress": "10.0.0.2",
"Tags": [
{
"Key": "Name",
"Value": "Balance-OTA-SS_a"
}
]
}
]
}
]
}
This is the format returned by describe-instances, but with all the irrelevant fields removed.
Note that tags is always a list of objects, each of which has a Key and a Value. This format is perfect for from_entries, which can transform this list of tags into a convenient mapping object. Try this:
.Reservations[].Instances[] |
{
IP: .PrivateIpAddress,
Ambiente: (.Tags|from_entries.Environment)
}
{"IP":"10.0.0.1","Ambiente":"alpha"}
{"IP":"10.0.0.2","Ambiente":null}
That answers how to do it. But you probably want to understand why your approach didn't work.
.Reservations[].Instances[] |
{
IP: .PrivateIpAddress,
Ambiente: (.Tags[]|select(.Key=="Environment")|.Value)
}
The .[] filter you're using on the tags can return zero or multiple results. Similarly, the select filter can eliminate some or all items. When you apply this inside an object constructor (the expression from { to }), you're causing that whole object to be created a variable number of times. You need to be very careful where you use these filters, because often that's not what you want at all. Often you instead want to do one of the following:
Wrap the expression that returns multiple results in an array constructor [ ... ]. That way instead of outputting the parent object potentially zero or multiple times, you output it once containing an array that potentially has zero or multiple items. E.g.
[.Tags[]|select(.Key=="Environment")]
Apply map to the array to keep it an array but process its contents, e.g.
.Tags|map(select(.Key=="Environment"))
Apply first(expr) to capture only the first value emitted by the expression. If the expression might emit zero items, you can use the comma operator to provide a default, e.g.
first((.Tags[]|select(.Key=="Environment")),null)
Apply some other array-level function, such as from_entries.
.Tags|from_entries.Environment
You can either use an if ... then ... else ... end construct, or //. For example:
.Reservations[].Instances[]
| {IP: .PrivateIpAddress} +
({Ambiente: (.Tags[]|select(.Key=="Environment")|.Value)}
// null)

How to use jq to extract a particular field from a terraform state file?

Here is a simplified json file of a terraform state file (let's call it dev.ftstate)
{
"version": 4,
"terraform_version": "0.12.9",
"serial": 2,
"lineage": "ba56cc3e-71fd-1488-e6fb-3136f4630e70",
"outputs": {},
"resources": [
{
"module": "module.rds.module.reports_cpu_warning",
"mode": "managed",
"type": "datadog_monitor",
"name": "alert",
"each": "list",
"provider": "module.rds.provider.datadog",
"instances": []
},
{
"module": "module.rds.module.reports_lag_warning",
"mode": "managed",
"type": "datadog_monitor",
"name": "alert",
"each": "list",
"provider": "module.rds.provider.datadog",
"instances": []
},
{
"module": "module.rds.module.cross_region_replica_lag_alert",
"mode": "managed",
"type": "datadog_monitor",
"name": "alert",
"each": "list",
"provider": "module.rds.provider.datadog",
"instances": []
},
{
"module": "module.rds",
"mode": "managed",
"type": "aws_db_instance",
"name": "master",
"provider": "provider.aws",
"instances": [
{
"schema_version": 0,
"attributes": {
"address": "dev-database.123456.us-east-8.rds.amazonaws.com",
"allocated_storage": 10,
"password": "",
"performance_insights_enabled": false,
"tags": {
"env": "development"
},
"timeouts": {
"create": "6h",
"delete": "6h",
"update": "6h"
},
"timezone": "",
"username": "admin",
"vpc_security_group_ids": [
"sg-1234"
]
},
"private": ""
}
]
}
]
}
There are many modules at the same level of module.rds inside the instances. I took out many of them to create the simplified version of the raw data. The key takeway: do not assume the array index will be constant in all cases.
I wanted to extract the password field in the above example.
My first attempt is to use equality check to extract the relevant modules
` jq '.resources[].module == "module.rds"' dev.tfstate`
but it actually just produced a list of boolean values. I don't see any mention of builtin functions like filter in jq's manual
then I tried to just access the field:
> jq '.resources[].module[].attributes[].password?' dev.tfstate
then it throws the following error
jq: error (at dev.tfstate:1116): Cannot iterate over string ("module.rds")
So what is the best way to extract the value? Hopefully it can only focus on the password attribute in module.rds module only.
Edit:
My purpose is to detect if a password is left inside a state file. I want to ensure the passwords are exclusively stored in AWS secret manager.
You can extract the module you want like this.
jq '.resources[] | select(.module == "module.rds")'
I'm not confident that I understand the requirements for the rest of the solution. So this might not only not be the best way of doing what you want; it might not do what you want at all!
If you know where password will be, you can do this.
jq '.resources[] | select(.module == "module.rds") | .instances[].attributes.password'
If you don't know exactly where password will be, this is a way of finding it.
jq '.resources[] | select(.module == "module.rds") | .. | .password? | values'
According to the manual under the heading "Recursive Descent," ..|.a? will "find all the values of object keys “a” in any object found “below” ."
values filters out the null results.
You could also get the password value out of the state file without jq by using Terraform outputs. Your module should define an output with the value you want to output and you should also output this at the root module.
Without seeing your Terraform code you'd want something like this:
modules/rds/main.tf
resource "aws_db_instance" "master" {
# ...
}
output "password" {
value = aws_db_instance.master.password
sensitive = true
}
example/main.tf
module "rds" {
source = "../modules/rds"
# ...
}
output "rds_password" {
value = module.rds.password
sensitive = true
}
The sensitive = true parameter means that Terraform won't print the output to stdout when running terraform apply but it's still held in plain text in the state file.
To then access this value without jq you can use the terraform output command which will retrieve the output from the state file and print it to stdout. From there you can use it however you want.

how to loop through json input - aws step function / state machine

Is it possible to create a loop in aws step function and loop through json input array?
I have a function generateEmails that creates array with n number of objects:
{
"emails": [
{
"to": [
"willow1#aaa.co.uk"
]
},
{
"to": [
"willow2#aaa.co.uk"
]
}, {
"to": [
"willow3#aaa.co.uk"
]
}
]
}
and now I want to call next function sendEmail for each object in emails array with something like this:
{
"email": {
"to": [
"willow#aaa.co.uk"
]
}
}
step function code:
{
"Comment": "A state machine that prepares and sends confirmation email ",
"StartAt": "generateEmails",
"States": {
"generateEmails": {
"Type": "Task",
"Resource": "arn:aws:lambda::prepare-confirmation-email",
"Next": "sendEmail"
},
"sendEmail": {
"Type": "Task",
"Resource": "arn:aws:lambda::function:template-service",
"End" : true
}
}
}
Is that possible to achieve?
Thanks!
Yes, the Step Functions Map state makes this easy.
https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-map-state.html
Map allows you to run the same set of operations to each item in an array. If you set the MaxConcurrency field to something larger than 1, then it will do these in parallel. Or you can set to 1 and it will iterate through sequentially.
For the scenario you described, the number of items will probably mean that "Inline" Map will work just fine. But if that list is larger and you want to fan out to higher concurrency, the recently launched Distributed Map feature will let you do so.
https://aws.amazon.com/blogs/aws/step-functions-distributed-map-a-serverless-solution-for-large-scale-parallel-data-processing/

How to use user variables with file provisioner in Packer?

I have a packer json like:
"builders": [{...}],
"provisioners": [
{
"type": "file",
"source": "packer/myfile.json",
"destination": "/tmp/myfile.json"
}
],
"variables": {
"myvariablename": "value"
}
and myfile.json is:
{
"var" : "{{ user `myvariablename`}}"
}
The variable into the file does get replaced, is a sed replacement with shell provisioner after the file the only option available here?
Using packer version 0.12.0
You have to pass these as environment variables. For example:
"provisioners": [
{
"type": "shell"
"environment_vars": [
"http_proxy={{user `proxy`}}",
],
"scripts": [
"some_script.sh"
],
}
],
"variables": {
"proxy": null
}
And in the script you can use $http_proxy
So far I've come just with the solution to use file & shell provisioner. Upload file and then replace variables in file via shell provisioner which can be fed from template variables provided by e.g. HashiCorp Vault
Yo may use OS export function to set environment and pass it to Packer
Here is a config using OS ENV_NAME value to choose local folder to copy from
export ENV_NAME=dev will set local folder to dev
{
"variables": {
...
"env_folder": "{{env `ENV_NAME`}}",
},
"builders": [{...}]
"provisioners": [
{
"type": "file",
"source": "files/{{user `env_folder`}}/",
"destination": "/tmp/"
},
{...}
]
}
User variables must first be defined in a variables section within your template. Even if you want a user variable to default to an empty string, it must be defined. This explicitness helps reduce the time it takes for newcomers to understand what can be modified using variables in your template.
The variables section is a key/value mapping of the user variable name to a default value. A default value can be the empty string. An example is shown below:
{
"variables": {
"aws_access_key": "",
"aws_secret_key": ""
},
"builders": [{
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
// ...
}]
}
check this link for more information