Is it possible to create a loop in aws step function and loop through json input array?
I have a function generateEmails that creates array with n number of objects:
{
"emails": [
{
"to": [
"willow1#aaa.co.uk"
]
},
{
"to": [
"willow2#aaa.co.uk"
]
}, {
"to": [
"willow3#aaa.co.uk"
]
}
]
}
and now I want to call next function sendEmail for each object in emails array with something like this:
{
"email": {
"to": [
"willow#aaa.co.uk"
]
}
}
step function code:
{
"Comment": "A state machine that prepares and sends confirmation email ",
"StartAt": "generateEmails",
"States": {
"generateEmails": {
"Type": "Task",
"Resource": "arn:aws:lambda::prepare-confirmation-email",
"Next": "sendEmail"
},
"sendEmail": {
"Type": "Task",
"Resource": "arn:aws:lambda::function:template-service",
"End" : true
}
}
}
Is that possible to achieve?
Thanks!
Yes, the Step Functions Map state makes this easy.
https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-map-state.html
Map allows you to run the same set of operations to each item in an array. If you set the MaxConcurrency field to something larger than 1, then it will do these in parallel. Or you can set to 1 and it will iterate through sequentially.
For the scenario you described, the number of items will probably mean that "Inline" Map will work just fine. But if that list is larger and you want to fan out to higher concurrency, the recently launched Distributed Map feature will let you do so.
https://aws.amazon.com/blogs/aws/step-functions-distributed-map-a-serverless-solution-for-large-scale-parallel-data-processing/
Related
I have on a AWS state machine the following step defined for api aws-sdk:ssm:sendCommand
{
"Type": "Task",
"Parameters": {
"DocumentName.$": "$.result.DocumentName",
"InstanceIds.$": "$..Dimensions[?(#.Name=~/.*InstanceId.*/)].Value",
"MaxErrors": "0",
"MaxConcurrency": "100%",
"CloudWatchOutputConfig": {
"CloudWatchLogGroupName": "diskspace-log",
"CloudWatchOutputEnabled": true
},
"Parameters": {
"workingDirectory": [
""
],
"executionTimeout": [
"3600"
],
"commands": [
"echo -------------------Mounting volume without signals $..Dimensions[?(#.Name=~/.*device.*/)].Value---------------------",
"echo",
"mount $..Dimensions[?(#.Name=~/.*device.*/)].Value"
]
}
}
}
The section: "commands": [] expects an array.
"commands" should accept input reference as any other parameter in the schema, so in theory will be posible to use json path parameters (Example: "size.$": "$.product.details.size") for referencing needed parameters from input.
https://docs.aws.amazon.com/step-functions/latest/dg/input-output-inputpath-params.html
This following example works without using input referencing :
"commands": [
"echo -------------------Mounting /dev/ebs---------------------",
"echo",
"mount /dev/ebs"
]
But I need to reference from input, hardcoded values won't work for me. I tried, but no working.
"commands": [
"echo -------------------Mounting volume without signals $..Dimensions[?(#.Name=~/.*device.*/)].Value---------------------",
"echo",
"mount $..Dimensions[?(#.Name=~/.*device.*/)].Value"
]
Also tried, not working also:
"commands.$": "States.Array(States.Format('echo -------------------Mounting volume without signals {} ---------------------', $..Dimensions[?(#.Name=~/.*device.*/)].Value),'echo',States.Format('mount {}', $..Dimensions[?(#.Name=~/.*device.*/)].Value))"
I believe some of the provided intrinsic functions will help on achieving the expected result but I'm lost on how to properly set up the syntax.
https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-intrinsic-functions.html#asl-intrsc-func-arrays
The step calls a RunShellScript type of documentCommand.
And executes the commands provided on parameters in the step of the state machine.
I got on the output:
States.Format('echo -------------------Mounting volume without signals {} ---------------------', $..Dimensions[?(#.Name=~/.*device.*/)].Value)'
Its not detecting the input reference, I expect to output.
-------------------Mounting volume without signals /dev/ebs ---------------------
and in the background execute:
mount /dev/ebs
I was able to send the commands through a Pass State Flow, here is the definition:
{
"Type": "Pass",
"Next": "SendCommand",
"ResultPath": "$.ForArgs",
"Parameters": {
"Params": {
"Args": [
{
"Arg1": "ec2-metadata -i"
},
{
"Arg2": "echo"
},
{
"Arg3.$": "States.Format('echo -------------------Mounting volume without signals {} ---------------------', States.ArrayGetItem($..Dimensions[?(#.Name=~/.*device.*/)].Value, 0))"
},
{
"Arg4": "echo"
},
{
"Arg5.$": "States.Format('mount {}', States.ArrayGetItem($..Dimensions[?(#.Name=~/.*device.*/)].Value, 0))"
},
{
"Arg6.$": "States.Format('echo Checking if device {} is mounted', States.ArrayGetItem($..Dimensions[?(#.Name=~/.*device.*/)].Value, 0))"
},
{
"Arg7.$": "States.Format('if findmnt --source \"{}\" >/dev/null', States.ArrayGetItem($..Dimensions[?(#.Name=~/.*device.*/)].Value, 0))"
},
{
"Arg8": "\tthen echo device is mounted"
},
{
"Arg9": "\telse echo device is not mounted"
},
{
"Arg10": "fi"
}
]
}
}
}
Next on the sendCommandApi:
"commands.$": "$.ForArgs.Params.Args[*][*]"
In Azure Data Factory, I'm attempting to add params to the body of a copy task (connected to a REST API post request as the source). I'm wanting to use dynamic content to do so, but I'm struggling trying to find the real solution for the proper nomenclature. Here's what I have so far.
copy task
dynamic content
{
"datatable":
{
"start":0,
"length": 10000,
"filters": [
{
"name": "Arrival Dates",
"start": "pipeline().parameters.pDate1",
"end": "pipeline().parameters.pDate2"
}
],
"sort": [
{
"name": "start_date",
"order": "ASC"
}
]
}
}
You'll notice that I've added params for dates. Is this the correct nomenclature for trying to add dynamic content? The autocorrect tried to add the # sign in the beginning of the code block, which will cause the entire thing to error out. I've tried adding it before each parameter, but that isn't actually reading the dynamic values either.
This is not correct. You need to use concat to concatenate the different variables. Something like this :
#concat('{ "datatable": { "start":0, "length": 10000, "filters": [ { "name": "Arrival Dates", "start": "',pipeline().parameters.pDate1,'", "end": "',pipeline().parameters.pDate2,'" } ], "sort": [ { "name": "start_date", "order": "ASC" } ] } }')
This is also documented in the SO question.
{
"metadata": {
"id": "2",
"uri": "3",
"type": "2"
},
"Number": "2323600002913",
"Date": "04/21/2009",
"postingDate": "00/00/0000",
"ata": {
"results": [
{
"metadata": {
"id": "r",
"uri": "e2",
"type": "s2"
},
"item": "000010",
"data":"ad"
}
]
}
}
want to remove metadata property from above json message and output should be like below
{
"Number": "2323600002913",
"Date": "04/21/2009",
"postingDate": "00/00/0000",
"ata": {
"results": [
{
"item": "000010",
"data":"ad"
}
]
}
}
I tried with removeProperty() which is working for root level metadata but inside metadata not removed.
how to use replace() in this case or anything else to only remove metadata.
The simplest way is use inline code, cause even with removeProperty() expression to remove the metadata under results, it will return the results array data not the whole json data. Then you will have to combine them, it's not a convenient way.
And with inline code you could refer to my below picture. The variable json is the value from triggerbody, then just delete the node or key and return the json variable. And with this way, even you want to delete many metadata in the array, you could add a for loop to delete it, just think of it as plain js code.
Update:if you want to get value from variable,cause no support expression to get value from variable so use the below expression.
var json =wworkflowContext.actions.Initialize_variable.inputs.variables[0].value;
And about how to loop the array in the json refer to my below pic.
We have a heavily nested json document containing server metrcs, the document contains > 1000 fields some of which are completely irrelevant to us for analytic purposes so i would like to remove them before indexing the document in Elastic.
However i am unable to find the correct filter to use as the fields i want to remove have common names in multiple different objects within the document.
The source document looks like this ( reduced in size for brevity)
[
{
"server": {
"is_master": true,
"name": "MYServer",
"id": 2111
},
"metrics": {
"Server": {
"time": {
"boundary": {},
"type": "TEXT",
"display_name": "Time",
"value": "2018-11-01 14:57:52"
}
},
"Mem_OldGen": {
"used": {
"boundary": {},
"display_name": "Used(mb)",
"value": 687
},
"committed": {
"boundary": {},
"display_name": "Committed(mb)",
"value": 7116
}
"cpu_count": {
"boundary": {},
"display_name": "Cores",
"value": 4
}
}
}
}
]
The data is loaded into logstash using the http_poller input plugin and needs to be processed before sending to Elastic for indexing.
I am trying to remove the fields that are not relevant for us to track for analytical purposes, these include the "display_name" and "boundary" fields from each json object in the different metrics.
I have tried using the mutate filter to remove the fields but because they exist in so many different objects it requires to many coded paths to be added to the logstash config.
I have also looked at the ruby filter, which seems promising as it can look the event, but i am unable to get it to crawl the entire json document, or more importantly actually remove the fields.
Here is what i was trying as a test
filter {
split{
field => "message"
}
ruby {
code => '
event.get("[metrics][Mem_OldGen][used]").to_hash.keys.each { |k|
logger.info("field is:", k)
if k.include?("display_name")
event.remove(k)
end
if k.include?("boundary")
event.remove(k)
end
}
'
}
}
It first splits the input at the message level to create one event per server, then tries to remove the fields from a specific metric.
Any help you be greatly appreciated.
If I get the point, you want to keep just the value key.
So, considering the response hash:
response = {
"server": {
"is_master": true,
"name": "MYServer",
"id": 2111
},
"metrics": {
...
You could do:
response[:metrics].transform_values { |hh| hh.transform_values { |h| h.delete_if { |k,v| k != :value } } }
#=> {:server=>{:is_master=>true, :name=>"MYServer", :id=>2111}, :metrics=>{:Server=>{:time=>{:value=>"2018-11-01 14:57:52"}}, :Mem_OldGen=>{:used=>{:value=>687}, :committed=>{:value=>7116}, :cpu_count=>{:value=>4}}}}
I am working on a IVR solution for small businesses in my local area but I am having trouble wrapping my head around how Node will handle menus. I could make a seperate Node server for each of my customers but I would like to have a single server that pulls each customer's IVR setup from a Mongo database or file when their number is called. I have an idea on how to save the menu structure in JSON but I am lost when it comes to turning that JSON into responses to <gather> inputs. I was thinking I could use a JSON structure like this in the DB (or maybe as a .json file on Amazon S3):
{
"menu": {
"id": 1,
"name": "Main",
"script": "Thank you for calling Local Company. To speak to sales press 1, ...",
"options": [
{
"name": "",
"action": "",
"value": "",
"next": ""
},
{
"name": "Sales",
"action": "dial",
"value": 12345678901,
"next": ""
},
{
"name": "Support",
"action": "dial",
"value": 12345678902,
"next": ""
},
{
"name": "Directions",
"action": "say",
"value": "Our offices are located at...",
"next": 1
},
{
"name": "Mailbox",
"action": "mailbox",
"value": "main",
"next": 1
}
]
}
}
Twilio developer evangelist here.
If you can return the JSON based on the number a user is dialling, then you could do something like this:
const Twilio = require('twilio');
app.post('/voice', (req, res) => {
const dialledNumber = req.body.To;
getIVRObjectFromPhoneNumber(dialledNumber, (IVRObject) => {
const twiml = Twilio.twiml.VoiceResponse();
if (typeof req.body.Digits !== 'undefined') {
// A user has pressed a digit, do the next thing!
const action = IVRObject.menu.options[req.body.Digits]
twiml[action.action](action.value);
} else {
// No digits yet, return the <Gather>
const gather = twiml.gather({
numDigits: 1
});
gather.say(IVRObject.script);
}
res.send(twiml.toString());
});
});
This doesn't quite use all of your object, I'm not sure what the values for next mean, but hopefully it's a start. The getIVRObjectFromPhoneNumber method is my made up, asynchronous method that returns a JavaScript object parsed from your example JSON above.
Let me know if this helps at all.