How do I modify parameters in a Container Overrides section of a Step Functions machine? - json

I have a Step Functions Machine the definition file of which looks as such:
"ContainerOverrides": [
{
"Name": "Foo",
"Environment": [
{
"Name": "Foo"
"Value": "Bar"
},
],
"Command.$": "States.Array($.Foo,$.Foo,$.Bar,$.Bar,$.Bar)"
}
]
},
that I am trying to rewrite into Typescript (CDK). I've gotten the following few lines.
containerOverrides: [{
containerDefinition: Foo,
environment: [
{ name: 'Foo', value: 'Bar'},
],
command: ['States.Array($.Foo,$.Foo,$.Bar,$.Bar,$.Bar)'],
}],
I'm a bit confused about how to go about this.
When I deploy the above CDK code, I get as output:
"Command": [
"States.Array($.Foo,$.Foo,$.Bar,$.Bar,$.Bar)"
],
My confusion is in regards to the following: The ContainerOverrides method doesn't accept parameters, but I need to modify a parameter (Command.$), so how can I possibly do that? I came across this post where somebody seems to have a similar issue, but when I try to apply the proposed solution, of simply writing
command: JsonPath.arrayAt('States.Array($.Foo,$.Foo,$.Bar,$.Bar,$.Bar)'
I get told that ''Cannot use JsonPath fields in an array, they must be used in objects''

TL;DR The current implementation of EcsRunTask doesn't permit this. The general-purpose CallAwsService construct does.
The EcsRunTask construct is the CDK's implementation of the ECS optimised integration. The construct only accepts an array of strings as override commands. It cannot produce substitutable output like "Command.$": "$.commands" that's needed to read the override command from the execution input. This is a limitation of the CDK implementation, not of the ECS optimized integration itself.
The cleanest solution is to use the CallAwsService construct, which implements the SDK service integration. It requires manual configuration. The API-specific config goes in the parameters prop. The prop is loosely typed as { [string]: any }. It's flexible, but it's your job to provide the expected syntax for the ecs:RunTask SDK call. Here is the relevant bit for your question:
parameters {
Overrides: {
ContainerOverrides: [
{ Command: sfn.JsonPath.array("sh", "-c", sfn.JsonPath.stringAt("$.cmd")), },
],
},
}
It produces the expected command override in the Step Functions task definition:
"Command.$": "States.Array('sh', '-c', $.cmd)"

Related

How to pass a list as an environment variable to an AWS Lambda function from a JSON config file?

I have a JSON file that is going to contain a number of different lists per client that I am deploying for. These lists are going to serve as container overrides for an ECS task that my Lambda function will be invoking. The JSON config file would look something like this:
{
"clientName": {
"environment": [
{
"name": "name1",
"value": "value1"
}
]
}
}
And my serverless.yml would look something like this:
environment:
CONTAINER_ENVIRONMENT: ${file(serverlessConfig.json):${env:CLIENT_NAME}.environment}
Which results in the following error:
Could not resolve "CONTAINER_ENVIRONMENT" environment variable: Unsupported environment variable format:
[
{
"name": "name1",
"value": "value1"
}
]
I've tried using CloudFormation intrinsic functions such as Fn::Join and Fn::ToJsonString. These both threw an error when trying to run locally using sls invoke local (the errors were the same as the above). After some digging it seems that these functions aren't compatible with the serverless environment property.
The only thing that has worked so far is storing the list as a string in a .env file, but that's not really ideal since these configs could take a number of different environment objects.
Is there any way to get this to work with the setup that I have going?

Does config.GetSection not work in Azure Functions? And what is the recommended alternative?

Azure Function with a complex (List of objects) configuration type is working locally (with that complex type in local.settings.json) but fails to read / create list of objects in Azure (with that complex type in Azure Function configuration settings). I'm looking for the recommended / optimal way to support that across both platforms / methods of access.
This works great in my local.settings.json where I use the configuration builder and pull data out like
var myList = config.GetSection("ConfigurationList").Get<List<MyType>>();
however this doesn't seem to work in Azure Functions?? Now I think that is because in local.settings.json it is a json file and looks like
"ConfigurationList" : [ { "Name": "A", "Value": 2 }, { "Name": "B", "Value": 3 }]
while in Azure Functions it is a setting "ConfigurationList" with the value
[ { "Name": "A", "Value": 2 }, { "Name": "B", "Value": 3 }]
(so there isn't really a "section" in Azure Functions?)
It seems like the "easy" solution to this is to just change the .json to be a quoted string and deserialize the string (and then it would work the same in both places); but that doesn't seem like it would be the "best" (or "recommended" solution)
i.e. something like
"ConfigurationList" : "[ { \"Name\": \"A\", \"Value\": 2 }, { \"Name\": \"B\", \"Value\": 3 }]"
var myList = (List<MyType>)JsonConvert.DeserializeObject(config["ConfigurationList"], typeof(List<MyType>));
Which isn't the worst; but makes the json a bit "not as nice" and doesn't "flow" across the two platforms ... if it is what I have to do, fine; but hoping for a more standard approach / recommendation
As I metioned in the comment, on local you can process local.settings.json as a json file, but when on azure, the value in configuration settings is environment variable. There is no section, it just string.
Please notice that only string values are allowed, and that anything nested will break. Learn how to use nest settings on azure web app(azure functon is based on azure app service sandbox, so it is the same.):
https://learn.microsoft.com/en-us/archive/blogs/waws/asp-net-core-settings-for-azure-app-service
For example, if this is the json structure:
{
"Parent": {
"ChildOne": "C1 from secrets.json",
"ChildTwo": "C2 from secrets.json"
}
}
Then in web app, you should save it like this:
(source: windows.net)
Not sure if you are looking something like this , it seems a list but if it is a simple JObject like
"ConfigurationList" : {
"Name": "A",
"Value": 2
}
Then you can declare ConfigurationList:Name , ConfigurationList:Value in the configuration settings of function app

The name cannot contain any of the following symbols: '[, ], .'.'. for Compose in Azure Logic App

I have below JSON which I need to update inside a logic app
{
"name": "SampleDoc",
"type": "123",
"properties": {
"GP.Test": "M1",
"MG.Test": "C1"
}
}
I have used following setProperty syntax: -
#setProperty(variables('ResponseBody'),'properties', setProperty(variables('ResponseBody')['properties'], 'test','abc'),
setProperty(variables('ResponseBody')['properties'], 'GP.Test','M2'))
My desired JSON output should be
{
"name": "SampleDoc",
"type": "123",
"properties": {
"GP.Test": "M2",
"MG.Test": "C1"
}
}
But when I am running this, I am getting this error: -
InvalidTemplate. Unable to process template language expressions in action 'Compose' inputs at line '1' and column '2617': 'The provided property name 'GP.Test' has these invalid characters '.'. The name cannot contain any of the following symbols: '[, ], .'.'.
Could anyone suggest if we can handle '.' inside compose or any other way for achieving this?
Yes that is correct behavior that is occurring in the logic apps. The reason is that you have used the set property function to set the value of the GP.Test property. When working with the expressions in logic apps, the '.' operator is reserved operator and will be used to access sub properties etc of the expressions, functions etc. Hence you get the error. The solution to this is actually simple, you use the compose action directly without using the set property. Sample screenshot below.
Or if you want complex transformations, then using the liquid transformations through the integration account is the way to go

getDegree()/isOutgoing() funcitons don't work in graphAware/neo4j-to-elasticsearch mapping.json

Neo4j Version: 3.2.2
Operating System: Ubuntu 16.04
I use getDegree() function in mapping.json file, but the return would always be null; I'm using the dataset neo4j tutorial Movie/Actor dataset.
Output from elasticsearch request
mapping.json
{
"defaults": {
"key_property": "uuid",
"nodes_index": "default-index-node",
"relationships_index": "default-index-relationship",
"include_remaining_properties": true
},
"node_mappings": [
{
"condition": "hasLabel('Person')",
"type": "getLabels()",
"properties": {
"getDegree": "getDegree()",
"getDegree(type)": "getDegree('ACTED_IN')",
"getDegree(direction)": "getGegree('OUTGOING')",
"getDegree('type', 'direction')": "getDegree('ACTED_IN', 'OUTGOING')",
"getDegree-degree": "degree"
}
}
],
"relationship_mappings": [
{
"condition": "allRelationships()",
"type": "type",
}
]
}
Also if I use isOutgoing(), isIncoming(), otherNode function in relationship_mappings properties part, elasticsearch would never load the relationship data from neo4j. I think I probably have some misunderstanding of this sentence only when one of the participating nodes "looking" at the relationship is provided on this page https://github.com/graphaware/neo4j-framework/tree/master/common#inclusion-policies
mapping.json
{
"defaults": {
"key_property": "uuid",
"nodes_index": "default-index-node",
"relationships_index": "default-index-relationship",
"include_remaining_properties": true
},
"node_mappings": [
{
"condition": "allNodes()",
"type": "getLabels()"
}
],
"relationship_mappings": [
{
"condition": "allRelationships()",
"type": "type",
"properties": {
"isOutgoing": "isOutgoing()",
"isIncomming": "isIncomming()",
"otherNode": "otherNode"
}
}
]
}
BTW, is there any page that list all of the functions that we can use in mapping.json? I know two of them
github.com/graphaware/neo4j-framework/tree/master/common#inclusion-policies
github.com/graphaware/neo4j-to-elasticsearch/blob/master/docs/json-mapper.md
but it seems there are more, since I can use getType(), which hasn't been listed in any of the above pages.
Please let me know if I can provide any further help to solve the problem
Thanks!
The getDegree() function is not available to use, in contrary to getType(). I will explain why :
When the mapper (the part responsible to create a node or relationship representation as ES document ) is doing its job, it receive a DetachedGraphObject being a detached node or relationship.
The meaning of detached is that it is happening outside of a transaction and thus query operations are not available against the database anymore. The getType() is available because it is part of the relationship metadata and it is cheap, however if we would want to do the same for getDegree() this can be seriously more costly during the DetachedObject creation (which happen in a tx) depending on the number of different types etc.
This is however something we are working on, by externalising the mapper in a standalone java application coupled with a broker like kafa, rabbit,.. between neo and this app. We would not, however offer the possibilty to requery the graph in the current version of the module as it can have serious performance impacts if the user is not very careful.
As last, the only suggestion I can give you is to keep a property on your node with the updates of degrees you need to replicate to ES.
UPDATE
Regarding this part of the documentation :
For Relationships only when one of the participating nodes "looking" at the relationship is provided:
This is used only when not using the json definition, so you can use one or the other. the json definition has been added later as addition and both cannot be used together.
For answering this part, it means that the nodes of the incoming or outgoing side, depending on the definition, should be included in the inclusion policy for nodes, like hasLabel('Employee') || hasProperty('form') || getProperty('age', 0) > 20 . If you have an allNodes policy then it is fine.

Play application.conf : Where to consult the list of all possible variable?

Where can i find the list of all possible variable that is possible to set in play application.conf ?
I can't find this information on playframework website.
Thank you
If you use IDE such as eclipse or IntelliJ, you can inspect Play.application().configuration() at runtime while debugging and it will contain all possble configuration key/value pairs. It briefly looks as follows:
{
"akka":{ },
"application":{ },
"applyEvolutions":{ },
"awt":{ },
"db":{ },
"dbplugin":"disabled",
"evolutionplugin":"enabled",
"file":{ },
"java":{ },
"jline":{ },
"line":{ },
"logger":{ },
"os":{ },
"path":{ },
"play":{ },
"promise":{ },
"report":{ },
"sbt":{ },
"sun":{ },
"user":{ }
}
There is no such list of all possible variables, since the application.conf is arbitrarily extensible by all sorts of tools and components, most of them third party, and can contain any config the user wants.
For example: the configuration detailing Play's thread pools is really just Akka configuration.
The key things (DB config, languages, evolutions) are in the template, either with default values or commented out, when you initialise a new Play application.
The config page on the site discusses some additional configuration you might need, but this mostly relates to concerns external to the application, like launching and logging.