Related
I was searching a way to run ecs task. I already have a cluster and task definition settings. I just wanted to trigger a task using CloudFormation template. I know that I can run a task by clicking on the console and it works fine. For cfn, approach needs to be define properly.
Check the attached screenshots. I wanted to run that task using CloudFormation and pass container override environment variables. As per my current templates, it is not allowing me to do same like I can do using console. Using console I just need to select the following options
1. Launch type
2. Task Definition
Family
Revision
3. VPC and security groups
4. Environment variable overrides rest of the things automatically selected
It starts working with console but with cloudformaton template how can we do that. Is it possible to do or there is no such feature?
"taskdefinition": {
"Type" : "AWS::ECS::TaskDefinition",
"DependsOn": "DatabaseMaster",
"Properties" : {
"ContainerDefinitions" : [{
"Environment" : [
{
"Name" : "TARGET_DATABASE",
"Value" : {"Ref":"DBName"}
},
{
"Name" : "TARGET_HOST",
"Value" : {"Fn::GetAtt": ["DatabaseMaster", "Endpoint.Address"]}
}
]
}],
"ExecutionRoleArn" : "arn:aws:iam::xxxxxxxxxx:role/ecsTaskExecutionRole",
"Family" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"TaskRoleArn" : "arn:aws:iam::xxxxxxxxxxxxxxx:role/xxxxxxxxxxxxxxx-XXXXXXXXX"
}
},
"EcsService": {
"Type" : "AWS::ECS::Service",
"Properties" : {
"Cluster" : "xxxxxxxxxxxxxxxxx",
"LaunchType" : "FARGATE",
"NetworkConfiguration" : {
"AwsvpcConfiguration" : {
"SecurityGroups" : ["sg-xxxxxxxxxxx"],
"Subnets" : ["subnet-xxxxxxxxxxxxxx"]
}
},
"TaskDefinition" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}
}
There is no validity error in the code however, I am talking about the approach. I added image name container name but now it is asking for memory and cpu, it should not ask as it is already defined we just need to run a task.
Edited
I wanted to run a task after creation of my database and wanted to pass those database values to the task to run and complete a job.
Here is the working example of what you can do if you wanted to pass variable and run a task. In my case, I wanted to run a task after creation of my database but with environment variables, directly AWS does not provide any feature to do so, this is the solution which can help to trigger you ecs task.
"IAMRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"events.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Description": "Allow CloudWatch Events to trigger ECS task",
"Policies": [
{
"PolicyName": "Allow-ECS-Access",
"PolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecs:*",
"iam:PassRole",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "*"
}
]
}
}
],
"RoleName": { "Fn::Join": [ "", ["CloudWatchTriggerECSRole-", { "Ref": "DBInstanceIdentifier" }]]}
}
},
"DummyParameter": {
"Type" : "AWS::SSM::Parameter",
"Properties" : {
"Name" : {"Fn::Sub": "${AWS::StackName}-${DatabaseMaster}-EndpointAddress"},
"Type" : "String",
"Value" : {"Fn::GetAtt": "DatabaseMaster.Endpoint.Address"}
},
"DependsOn": "TaskSchedule"
},
"TaskSchedule": {
"Type": "AWS::Events::Rule",
"Properties": {
"Description": "Trigger ECS task upon creation of DB instance",
"Name": { "Fn::Join": [ "", ["ECSTaskTrigger-", { "Ref": "DBName" }]]},
"RoleArn": {"Fn::GetAtt": "IAMRole.Arn"},
"EventPattern": {
"source": [ "aws.ssm" ],
"detail-type": ["Parameter Store Change"] ,
"resources": [{"Fn::Sub":"arn:aws:ssm:eu-west-1:XXXXXXX:parameter/${AWS::StackName}-${DatabaseMaster}-EndpointAddress"}],
"detail": {
"operation": ["Create"],
"name": [{"Fn::Sub": "${AWS::StackName}-${DatabaseMaster}-EndpointAddress"}],
"type": ["String"]
}
},
"State": "ENABLED",
"Targets": [
{
"Arn": "arn:aws:ecs:eu-west-1:xxxxxxxx:cluster/NameOf-demo",
"Id": "NameOf-demo",
"RoleArn": {"Fn::GetAtt": "IAMRole.Arn"},
"EcsParameters": {
"LaunchType": "FARGATE",
"NetworkConfiguration": {
"AwsVpcConfiguration": {
"SecurityGroups": {"Ref":"VPCSecurityGroups"},
"Subnets": {"Ref":"DBSubnetName"}
}
},
"PlatformVersion": "LATEST",
"TaskDefinitionArn": "arn:aws:ecs:eu-west-1:XXXXXXXX:task-definition/NameXXXXXXXXX:1"
},
"Input": {"Fn::Sub": [
"{\"containerOverrides\":[{\"name\":\"MyContainerName\",\"environment\":[{\"name\":\"VAR1\",\"value\":\"${TargetDatabase}\"},{\"name\":\"VAR2\",\"value\":\"${TargetHost}\"},{\"name\":\"VAR3\",\"value\":\"${TargetHostPassword}\"},{\"name\":\"VAR4\",\"value\":\"${TargetPort}\"},{\"name\":\"VAR5\",\"value\":\"${TargetUser}\"},{\"name\":\"VAR6\",\"value\":\"${TargetLocation}\"},{\"name\":\"VAR7\",\"value\":\"${TargetRegion}\"}]}]}",
{
"VAR1": {"Ref":"DBName"},
"VAR2": {"Fn::GetAtt": ["DatabaseMaster", "Endpoint.Address"]},
"VAR3": {"Ref":"DBPassword"},
"VAR4": "5432",
"VAR5": {"Ref":"DBUser"},
"VAR6": "value6",
"VAR7": "eu-west-2"
}
]}
}
]
}
}
For Fargate task, we need to specify in CPU in Task Definition. and memory or memory reservation in either task or container definition.
and environment variables should be passed to each container as ContainerDefinitions and overrided when task is run from ecs task-run from console or cli.
{
"ContainerTaskdefinition": {
"Type": "AWS::ECS::TaskDefinition",
"Properties": {
"Family": "SomeFamily",
"ExecutionRoleArn": !Ref RoleArn,
"TaskRoleArn": !Ref TaskRoleArn,
"Cpu": "256",
"Memory": "1GB",
"NetworkMode": "awsvpc",
"RequiresCompatibilities": [
"EC2",
"FARGATE"
],
"ContainerDefinitions": [
{
"Name": "container name",
"Cpu": 256,
"Essential": "true",
"Image": !Ref EcsImage,
"Memory": "1024",
"LogConfiguration": {
"LogDriver": "awslogs",
"Options": {
"awslogs-group": null,
"awslogs-region": null,
"awslogs-stream-prefix": "ecs"
}
},
"Environment": [
{
"Name": "ENV_ONE_KEY",
"Value": "Valu1"
},
{
"Name": "ENV_TWO_KEY",
"Value": "Valu2"
}
]
}
]
}
}
}
EDIT(from discussion in comments):
ECS Task Run is not a cloud-formation resource, it can only be run from console or CLI.
But if we choose to run from a cloudformation resource, it can be done using cloudformation custom resource. But once task ends, we now have a resource in cloudformation without an actual resource behind. So, custom resource needs to do:
on create: run the task.
on delete: do nothing.
on update: re-run the task
Force an update by changing an attribute or logical id, every time we need to run the task.
I tried to get the example from https://learn.microsoft.com/en-us/microsoftteams/platform/get-started/get-started-dotnet to run.
I successfully built and published (on Azure) the webapp.
Then I tried
- manually editing the json file in the visual studio solution similarly to what the docs said, and then import the resulting zip in App Studio
- adding the App directly in App Studio through the manifest editor (this works, but it fails when installing/testing it)
Both give me an error without any more info "something went wrong".
Any way to actually figure out what I did wrong (if anything)?
Either way, maybe you guys can figure it out from the json file's contents:
{
"$schema": "https://statics.teams.microsoft.com/sdk/v1.0/manifest/MicrosoftTeams.schema.json",
"manifestVersion": "1.0",
"version": "1.0.0",
"id": " 1CC58D17-1E95-443C-958F-E1F14D4CA3B4",
"packageName": "com.contoso.helloworld",
"developer": {
"name": "Contoso",
"websiteUrl": "https://www.microsoft.com",
"privacyUrl": "https://www.microsoft.com/privacy",
"termsOfUseUrl": "https://www.microsoft.com/termsofuse"
},
"name": {
"short": "Hello World",
"full": "Hello World App for Microsoft Teams"
},
"description": {
"short": "Hello World App for Microsoft Teams",
"full": "This sample app provides a very simple app for Microsoft Teams. You can extend this to add more content and capabilities."
},
"icons": {
"outline": "contoso20x20.png",
"color": "contoso96x96.png"
},
"accentColor": "#60A18E",
"staticTabs": [
{
"entityId": "com.contoso.helloworld.hellotab",
"name": "Hello Tab",
"contentUrl": "https://microsoftteamssampleshelloworldweb20181022032046.azurewebsites.net/hello",
"scopes": [
"personal"
]
}
],
"configurableTabs": [
{
"configurationUrl": "https://microsoftteamssampleshelloworldweb20181022032046.azurewebsites.net/configure",
"canUpdateConfiguration": true,
"scopes": [
"team"
]
}
],
"bots": [
{
"botId": "00000000-0000-0000-0000-000000000000",
"needsChannelSelector": false,
"isNotificationOnly": false,
"scopes": [
"team",
"personal"
]
}
],
"composeExtensions": [
{
"botId": "00000000-0000-0000-0000-000000000000",
"scopes": [
"personal",
"team"
],
"commands": [
{
"id": "getRandomText",
"description": "Gets some random text and images that you can insert in messages for fun.",
"title": "Get some random text for fun",
"initialRun": true,
"parameters": [
{
"name": "cardTitle",
"description": "Card title to use",
"title": "Card title"
}
]
}
]
}
],
"permissions": [],
"validDomains": []
}
Any suggestions?
There is a whitespace in your app Id in shared manifes:
"id": " 1CC58D17-1E95-443C-958F-E1F14D4CA3B4"
Could you please remove it and try and let us know if it works? Also you can remove bots and composeExtensions section if you want.
I can't comment yet (not enough reputation points) but could you go through the instructions again?
I believe something went wrong with either the GUID, or one of the URL's. The instructions also advise to use ngrok, which is usefull for debugging.
If you can't find a clear error message, i advise you follow those instructions.
When using the "Open Folder" functionality of Visual Studio, the IDE searches for project settings and configurations in a special json file. For CPP projects, this could be CppProperties.json. For CMake projects, this could be CMakeSettings.json.
This json file contains a collection of one or more "configurations," such as "Debug" or "Release". I will use a recent CMake project as an example:
"configurations": [
{
"name": "ARM-Debug",
"generator": "Ninja",
"configurationType": "Debug",
"inheritEnvironments": [
"gcc-arm"
],
"buildRoot": "${env.USERPROFILE}\\CMakeBuilds\\${workspaceHash}\\build\\${name}",
"installRoot": "${env.USERPROFILE}\\CMakeBuilds\\${workspaceHash}\\install\\${name}",
"cmakeCommandArgs": "",
"buildCommandArgs": "-v",
"ctestCommandArgs": "",
"intelliSenseMode": "linux-gcc-arm",
"variables": [
{
"name": "CMAKE_TOOLCHAIN_FILE",
"value": "${workspaceRoot}/cmake/arm-none-eabi-toolchain.cmake"
}
]
},
{
"name": "ARM-Release",
"generator": "Ninja",
"configurationType": "Release",
"inheritEnvironments": [
"gcc-arm"
],
"buildRoot": "${env.USERPROFILE}\\CMakeBuilds\\${workspaceHash}\\build\\${name}",
"installRoot": "${env.USERPROFILE}\\CMakeBuilds\\${workspaceHash}\\install\\${name}",
"cmakeCommandArgs": "",
"buildCommandArgs": "-v",
"ctestCommandArgs": "",
"intelliSenseMode": "linux-gcc-arm",
"variables": [
{
"name": "CMAKE_TOOLCHAIN_FILE",
"value": "${workspaceRoot}/cmake/arm-none-eabi-toolchain.cmake"
}
]
}
As you can see, I have two configurations with nearly identical properties.
My question: is it possible to define these common/shared properties once, in such a way as to allow the configurations to inherit them and avoid repeating myself?
The easier way is to define an environment at global level (outside of any configuration), such as:
{
"environments": [
{
"namespace" : "env",
"varName": "varValue"
}
],
Then you can reuse that wherever you need to, e.g.:
"cmakeCommandArgs": "${env.varName}",
You can also have multiple environments, and reuse them, like this:
{
"environments": [
{
"environment": "env1",
"namespace": "env",
"varName": "varValueEnv1"
},
{
"environment": "env2",
"namespace": "env",
"varName": "varValueEnv2"
}
],
"configurations": [
{
"name": "x64-Release",
"inheritEnvironments": [
"msvc_x64_x64", "env2"
],
"cmakeCommandArgs": "${env.varName}",
.....
}
]
the 'x64-Release' will inherit the variables's value in the environment called "env2" (namespace 'env')
We are upgrading our Data pipeline version from 3.3.2 to 5.8, so those bootstrap actions on old AMI release have changed to be setup using configuration and specifying them under classification / property definition.
So my Json looks like below
{
"enableDebugging": "true",
"taskInstanceBidPrice": "1",
"terminateAfter": "2 Hours",
"name": "ExportCluster",
"taskInstanceType": "m1.xlarge",
"schedule": {
"ref": "Default"
},
"emrLogUri": "s3://emr-script-logs/",
"coreInstanceType": "m1.xlarge",
"coreInstanceCount": "1",
"taskInstanceCount": "4",
"masterInstanceType": "m3.xlarge",
"keyPair": "XXXX",
"applications": ["hadoop","hive", "tez"],
"subnetId": "XXXXX",
"logUri": "s3://pipelinedata/XXX",
"releaseLabel": "emr-5.8.0",
"type": "EmrCluster",
"id": "EmrClusterWithNewEMRVersion",
"configuration": [
{ "ref": "configureEmrHiveSite" }
]
},
{
"myComment": "This object configures hive-site xml.",
"name": "HiveSite Configuration",
"type": "HiveSiteConfiguration",
"id": "configureEmrHiveSite",
"classification": "hive-site",
"property": [
{"ref": "hive-exec-compress-output" }
]
},
{
"myComment": "This object sets a hive-site configuration
property value.",
"name":"hive-exec-compress-output",
"type": "Property",
"id": "hive-exec-compress-output",
"key": "hive.exec.compress.output",
"value": "true"
}
],
"parameters": []
With the above Json file it gets loaded into Data Pipeline but throws an error saying
Object:HiveSite Configuration
ERROR: 'HiveSiteConfiguration'
Object:ExportCluster
ERROR: 'configuration' values must be of type 'null'. Found values of type 'null'
I am not sure what this really means and could you please let me know if i am specifying this correctly which i think i am according to http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html
The below block should have the name as "EMR Configuration" only then its recognized correctly by the AWS Data pipeline and the Hive-site.xml is being set accordingly.
{
"myComment": "This object configures hive-site xml.",
"name": "EMR Configuration",
"type": "EmrConfiguration",
"id": "configureEmrHiveSite",
"classification": "hive-site",
"property": [
{"ref": "hive-exec-compress-output" }
]
},
I have this JSON file:
{
"vulnerable_configuration_cpe_2_2": [
"cpe:/o:apple:apple_tv:9.1",
"cpe:/o:apple:watchos:2.1",
"cpe:/o:apple:iphone_os:9.2",
"cpe:/o:apple:mac_os_x:10.11.2"
],
"vulnerable_configuration": [
{
"title": "cpe:2.3:o:apple:apple_tv:9.1",
"id": "cpe:2.3:o:apple:apple_tv:9.1"
},
{
"title": "cpe:2.3:o:apple:watchos:2.1",
"id": "cpe:2.3:o:apple:watchos:2.1"
},
{
"title": "cpe:2.3:o:apple:iphone_os:9.2",
"id": "cpe:2.3:o:apple:iphone_os:9.2"
},
{
"title": "cpe:2.3:o:apple:mac_os_x:10.11.2",
"id": "cpe:2.3:o:apple:mac_os_x:10.11.2"
}
],
"summary": "The Disk Images component in Apple iOS before 9.2.1, OS X before 10.11.3, and tvOS before 9.1.1 allows local users to gain privileges or cause a denial of service (memory corruption) via unspecified vectors.",
"references": [
"https://support.apple.com/HT206168",
"https://support.apple.com/HT205732",
"https://support.apple.com/HT205731",
"https://support.apple.com/HT205729",
"http://lists.apple.com/archives/security-announce/2016/Mar/msg00001.html",
"http://lists.apple.com/archives/security-announce/2016/Jan/msg00005.html",
"http://lists.apple.com/archives/security-announce/2016/Jan/msg00003.html",
"http://lists.apple.com/archives/security-announce/2016/Jan/msg00002.html"
],
"Modified": "2016-03-29T19:01:26.810-04:00",
"Published": "2016-02-01T06:59:01.090-05:00",
"access": {
"vector": "LOCAL",
"complexity": "LOW",
"authentication": "NONE"
},
"cvss": 7.2,
"cvss-time": "2016-03-29T09:43:55.537-04:00",
"cwe": "CWE-119",
"id": "CVE-2016-1717",
"impact": {
"integrity": "COMPLETE",
"confidentiality": "COMPLETE",
"availability": "COMPLETE"
}
I want to delete the following data using JQ or any Linux Tool
access (including vector, complexity, authentication)
references: only first or 1 reference
cvss-time
vulnerable_configuration_cpe_2_2 (Incluide cpe:/o:apple:apple_tv:9.1,
etc)
The result, I want it to be this:
{
"vulnerable_configuration": [
{
"title": "cpe:2.3:o:apple:apple_tv:9.1",
"id": "cpe:2.3:o:apple:apple_tv:9.1"
},
{
"title": "cpe:2.3:o:apple:watchos:2.1",
"id": "cpe:2.3:o:apple:watchos:2.1"
},
{
"title": "cpe:2.3:o:apple:iphone_os:9.2",
"id": "cpe:2.3:o:apple:iphone_os:9.2"
},
{
"title": "cpe:2.3:o:apple:mac_os_x:10.11.2",
"id": "cpe:2.3:o:apple:mac_os_x:10.11.2"
}
],
"summary": "The Disk Images component in Apple iOS before 9.2.1, OS X before 10.11.3, and tvOS before 9.1.1 allows local users to gain privileges or cause a denial of service (memory corruption) via unspecified vectors.",
"references": [
"https://support.apple.com/HT206168",
],
"Modified": "2016-03-29T19:01:26.810-04:00",
"Published": "2016-02-01T06:59:01.090-05:00",
"cvss": 7.2,
"cwe": "CWE-119",
"id": "CVE-2016-1717",
"impact": {
"integrity": "COMPLETE",
"confidentiality": "COMPLETE",
"availability": "COMPLETE"
}
I try
cat file.json | jq ('del(.cvss-time)' and 'access')
but no work
What is the command JQ I have to use? Or what tool can I use?
Tanks!
The following filter for jq >= 1.5 meets your requirements:
del(.access)
| .references |= [.[0]]
| del(."cvss-time")
| del(.vulnerable_configuration_cpe_2_2)
If you are using jq 1.4, use del(.["cvss-time"]) instead of del(."cvss-time").