AWS stepfunctions - passing and reading variables from lambda function - json

I am trying to read output from my lambda function into a variable in my step function. The lambdas default output is
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}
What I would like to return is a json object like this
{
"version": version,
"bucket": bucket
}
where the version and bucket name gets passed from the lambda. in my step function, I am trying to capture this and insert it into an s3 url like this:
"S3Uri.$": "States.Format('s3://{}/path/to/script/{}/script.py',$.bucket, $.version)"
However, I am struggling with having the correct output from the lambda and how to grab the value in step functions. I've tried
return {
'statusCode': 200,
'body': json.dumps({
"version": version,
"bucket": bucket
}) }
And various ways of constructing the json object as a string to the body, eg
"{\"version\": \"" + version + "\",\"bucket\": \"" + bucket + "\"}"
But I can't find the right combo, and the job keeps failing. Here is an example error message
The JsonPath argument for the field '$.bucket' could not be found in
the input '{"statusCode": 200, "body": "{\"version\":
\"v0-1\", \"bucket\":
\"sagemaker-us-west-2-removed\"}"}'"
How should I construct the lambda output, and the corresponding step function variable to have the values pass through? Again, I want the lambda to tell step functions what bucket and version was used, and then have step functions insert these values into an s3 url string.
EDIT: here is the full error message for one of the attempts
{
"error": "States.Runtime",
"cause": "An error occurred while executing the state 'Postproc' (entered at the event id #38). The function 'States.Format('s3://{}/AAPM/AAPM_2207/prod/{}/meta/scripts/preproc/aapm-postproc.py',$.bucket, $.version)' had the following error: The JsonPath argument for the field '$.bucket' could not be found in the input '{\"statusCode\": 200, \"body\": \"{\\\"version\\\": \\\"v0-1\\\", \\\"bucket\\\": \\\"sagemaker-us-west-2-removed\\\"}\"}'"
}

As in this answer the trick was just to get rid of json dumps.
return {
'statusCode': 200,
'body': {
"version": version,
"bucket": bucket
}
}
and I could access them fine with $.bucket, $.version

Related

How to parse nested JSON, within a string, using Kusto

I have a Python Azure Function that produces custom logging messages when the Function executes. I need to pull out some of the JSON values nested in the logging message.
How can I use Kusto to access the nested JSON within the logging message string?
Example logging message:
Desired values marked with <----------
####### EventGrid trigger processing an event:
{
"id": "long-guid",
"data": {
"api": "FlushWithClose",
"requestId": "long-guid",
"eTag": "long-guid",
"contentType": "application/octet-stream",
"contentLength": 16264, <----------------------
"contentOffset": 0,
"blobType": "BlockBlob",
"blobUrl": "https://function.blob.core.windows.net/parentdir/childdir/file.name",
"url": "https://function.dfs.core.windows.net/parentdir/childdir/file.name", <---- JUST FILE.NAME here
"sequencer": "long-guid",
"identity": "long-guid",
"storageDiagnostics": {
"batchId": "long-guid"
}
},
"topic": "/subscriptions/long-guid/resourceGroups/resourceGroup/providers/Microsoft.Storage/storageAccounts/accountName",
"subject": "/blobServices/default/containers/containerName/blobs/childDir/file.name",
"event_type": "Microsoft.Storage.BlobCreated"
} #######
I imagine it has something to do with the Kusto extend function, but piping in...
| extend parsedMessage = todynamic(message)
| project timestamp, test = parsedMessage["id"]
...yields only an empty test column
message in your specific case isn't a valid JSON payload - as it has the ###... EventGrid trigger processing an event: prefix (and a somewhat similar suffix).
That is why todynamic() isn't able to process it and why you're not able to reference properties in the JSON payload that's included in it.
Ideally, you would change the payload you ingest to be a valid JSON payload, and re-type the target column to dynamic instead of string.
If you can't do that, you can use the substring() function or parse operator to get everything but the aforementioned prefix/suffix, and parse the output of that using todynamic()
though note that doing that each time you query the data bears runtime overhead that could be avoided by following the advice above.

AWS Step function string/json concatenation

I have orchestrated a data pipe line using AWS Step function.
In last state I want to send a custom notification. I'm using an Intrinsic function States.Format to format my message and subject. It works fine for Context object element. Here, I have tested that in Message parameter.
But it doesn't work with input JSON. This is my input JSON
{
"job-param":{
"pipe-line-name":"My pipe line name", "other-keys":"other values"
}
}
"Success State": {
"Type": "Task",
"Resource": "arn:aws:states:::sns:publish",
"Parameters": {
"Message.$": "States.Format('Execution Id:{}, completed successfully!', $$.Execution.Id)",
"Subject.$": "States.Format('[INFO] {} completed successfully!', $.job-param.pipe-line-name)",
"TopicArn": "arn:aws:sns:us-east-1:************:sns-topic"
},
"End": true
}
While saving this state machine, it gives me following error message:
The value for the field 'Subject.$' must be a valid JSON Path
I checked Input and Result path. They have this value. I can directly use this value as parameter. This is working fine. But I can't format with other string.
"Subject.$": "$.job-param.pipe-line-name"
Alternate approach would be to call lambda to customize and trigger SNS. But I want to avoid that.
Can I request some suggestions to fix this error?
Thanks in advance!
If you want to use any name with - in your JSON then you can write your JSON Path like this:
"Subject.$": "States.Format('[INFO] {} completed successfully!', $['job-param']['pipe-line-name'])",
But it would be easier if you change your input JSON and replace - with _:
"Subject.$": "States.Format('[INFO] {} completed successfully!', $.job_param.pipe_line_name)",

Google Apps Script API Request with GraphQL parse error

I'm trying to make API request with using Apps Script. API is GraphQL based.
I'm using JSON.stringify function. But API returns an error.
Request:
payload={"query":"{mutation {change_column_value (board_id: 177955, item_id: 287466289, column_id:\"phone\", value: \"{phone : 15065332974, countryShortName: \"US\" }\") {name column_values{ id value}}}"}}
I'm getting error;
{"errors":[{"message":"Parse error on \" }\" (STRING) at [1, 148]","locations":[{"line":1,"column":148}]}]}
Apps Script Code:
var us = "US"
var column_values = '{ \
mutation {change_column_value (board_id: 177955, item_id: '+ 287466289 +', column_id:"phone", value: "{phone : 15065332974, countryShortName: \"' + us +'\" }") {name column_values{ id value}} \
}';
settings = {
"method": "POST",
"headers": {"Content-Type" : "application/json","Authorization" : "eyJhbGciOiJIXXXXXXXXXX"},
"payload" : JSON.stringify({query:column_values})
}
response= UrlFetchApp.fetch(url, settings);
Brackets in a GraphQL operation indicate a selection set -- one or more fields that are being requested. The operation itself (whether a query or a mutation) is not a field, so you don't wrap the whole operation in brackets.
Correct:
mutation {
doSomething
}
Incorrect:
{
mutation {
doSomething
}
}
The only time you will see brackets that look like they are on the "outside" is when using query shorthand.
Provided that value is a String, then your use of backslashes to escape the double quotes inside that string should be fine.
You might also consider using variables, which would clean up your code considerably and make it less error prone.
You do not need to escape special characters inside a JSON request.
The syntax implies using key-value pairs assigned to each other with a colon, whereby strings are located inside quotes (not escaped with a backslash!), while numbers and variable names are not.
The brackets should be used in function of your desired API request as documented.

Could not parse request body into json: Unexpected character (\'-\' (code 45)) AWS Lambda + API + Postman

I have been trying for a few days to get a parameter sent from the API Gateway in AWS to a Lambda function and I am having no success.
I decided to start from the beginning so I followed their walkthrough (http://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started.html#getting-started-new-lambda)
I have checked this walkthrough twice and I have followed the steps to the letter.
Problem
When I test the API from Postman or in Swift I am getting the error:
{"message": "Could not parse request body into json: Unexpected character (\'-\' (code 45)) in numeric value: expected digit (0-9) to follow minus sign, for valid numeric value\n at [Source: [B#c036d15; line: 1, column: 3]"}
In postman, When I view the result as JSON I just get
Bad String
Lambda Function
The function is the basic example from the Walkthrough:
console.log('Loading event');
exports.handler = function(event, context) {
var name = (event.name === undefined ? 'No-Name' : event.name);
console.log('"Hello":"' + name + '"');
context.done(null, {"Hello":name}); // SUCCESS with message
};
When Tested from the Lambda Console and with the Test data I get the result:
{
"Hello": "TestUser123"
}
When Tested from the API Gateway Test, The result is also:
{
"Hello": "TestUser123"
}
Can anyone see why both test consoles are allowing this work but when tested with POSTMAN or used within a Swift Script it does not work ?
Edit 1
In postman, I have set the content-type to application/json
The script returns the default:
{
"Hello": "user"
}
However, When I add in the parameters name and TestUser123 in POSTMAN, this is when it returns the error.
Update 1
Ok, so I changed the mapping template to one that I found on another answer:
{ "name": "$input.params('name')" }
Now the result is:
{
"Hello": ""
}
Any Ideas why it is not getting the name?
I just got stuck with this today.
your mapping template is:
{ "name": "$input.params('name')" }
AWS uses AWS Velocity templates, which; even though looks like JSON, is different.
if you use
{ "name": $input.params('name') } // notice no quotes
for the mapping template right at the integration request step, then it should work as expected.
Read the error message very carefully, it actually tells you the problem. For example, I got
Could not parse request body into json: Unexpected character (\'\"\' (code 34)): was expecting comma to separate Object entries
So the problem is that I'm missing a comma. I check my Lambda's Integration Request - Body Mapping Template:
{
"age" : $input.json('$.persondata.age'),
"income" : $input.json('$.persondata.income')
"height" : $input.json('$.persondata.height')
}
Can you spot the problem? I am missing a comma after the income line.
Here is another example.
Could not parse request body into json: Unexpected character (\'}\' (code 125)): expected a value
When I look at the Integration Request - Body Mapping Template:
#set($inputRoot = $input.path('$'))
{
"age" : $inputRoot.age,
"height" : $inputRoot.height,
"income" : $inputRootincome
}
Can you spot the problem? I am missing a dot in $inputRootincome.
Error Message :
Could not parse request body into json: Could not parse payload into json: Unrecognized token \' \': was expecting (\'true\', \'false\' or \'null\')
Cause of the error : When string values inside the json are not assigned using double quotations in the aws mapping template the error occurs.
Solution : (Please Note : This example is for application/json type request template)
Actually the solution for the problem is, if you are using a value of type string in json then its value should be assigned inside a ("" - double quotation marks) in the mapping template.
The below shown example has the following attributes :
customerId - string (Please note : this value comes from a query parameter)
customerName - string
customerAge - integer
isPermanentEmployee - boolean
customerAddress - string (Please note this is an optional parameter)
And the mapping template should be defined like the example shown below
Refer the example below :
#set($inputRoot = $input.path('$'))
{
"CustomerId": "$input.params('customerId')",
"CustomerName": "$inputRoot.customerName",
"CustomerAge": $inputRoot.customerAge,
"IsPermanentEmployee": $inputRoot.isPermanentEmployee
"CustomerAddress ": #if($inputRoot.customerAddress == "") "" #elseif($inputRoot.customerAddress != "") "$inputRoot.customerAddress" #end
}
If you note the above mapping template, I would have given string values inside double quotation marks("") which will solve the error
Also this example contains how to handle optional parameters in aws mapping templates using #if#else statements.
It is likely that you had copy-pasted multiple lines in your "Integration Request" in the API gateway.
When copying a line and pasting it below, you might have copied the hidden character '\n' at the end of that line. This is probably causing issues at the lambda function.
Example: Copying the line containing age and pasting it twice and modifying them to have height and income respectively.
#set($inputRoot = $input.path('$'))
{
"age" : $inputRoot.age,
"height": $inputRoot.height,
"income": $inputRoot.income
}
Instead of copy-pasting, just type the line out for height and income.

Rally JSON I/O error creating a test case result

I am trying to create a test case result using a REST client, but get this error:
"Errors": ["Cannot parse input stream due to I/O error as JSON document: Parse error: expected '{' but saw '\uFFFF' [ chars read = >>>\uFFFF<<< ]"]
I get the same error when the name of the object, testcaseresult is not specified in the request body. Here are the steps to create a test case result using a browser REST client:
a) Generate the authorize key using "GET" method and the following URL:
https://rally1.rallydev.com/slm/webservice/v2.0/security/authorize
This is the response that I get back, with the security token: "123abc..."
{"OperationResult": {"_rallyAPIMajor": "2", "_rallyAPIMinor": "0", "Errors": [], "Warnings": [], "SecurityToken": "abc123..."}}
b) Use "POST" method, and the following URL:
https://rally1.rallydev.com/slm/webservice/v2.0/testcaseresult/create?key=abc123...
notice the security token in the end.
c) here is an example of a request body:
{
"testcaseresult":
{
"Build":"1",
"Tester":"/user/777",
"Date":"2010-09-04T19:56:05.000Z",
"TestCase":"/testcase/1111",
"Verdict":"Pass"
}
}
Only the required fields and the fields you want to set need to be referenced. Notice the outer key/value pair
{
"testcaseresult":{}
}
The fields that point to a full object, like "Tester" (points to User object) and "TestCase" (points to a TestCase object that owns the result) have to be referenced by their ObjectIDs:
"Tester":"/user/777",
"TestCase":"/testcase/1111",