Merging JSON files using jenkins groovy - json

I have multiple JSON files with the following structures
FILE1.json
{
"Plugins": [
{
"Name": "Async Http Client",
"Installation": [
{
"Version": "1.7.24.2",
"Enabled": true,
"URL": "http://wiki.jenkins-ci.org/display/JENKINS/Async+Http+Client+Plugin",
"Server": "Server1"
}
]
},
{
"Name": "Common API for Blue Ocean",
"Installation": [
{
"Version": "1.10.2",
"Enabled": true,
"URL": "https://wiki.jenkins-ci.org/display/JENKINS/Blue+Ocean+Plugin",
"Server": "Server1"
}
]
},
.
.
.
]
}
FILE2.json
{
"Plugins": [
{
"Name": "Async Http Client",
"Installation": [
{
"Version": "1.7.24.1",
"Enabled": true,
"URL": "http://wiki.jenkins-ci.org/display/JENKINS/Async+Http+Client+Plugin",
"Server": "Server2"
}
]
},
{
"Name": "Common API for Blue Ocean",
"Installation": [
{
"Version": "1.10.2",
"Enabled": true,
"URL": "https://wiki.jenkins-ci.org/display/JENKINS/Blue+Ocean+Plugin",
"Server": "Server2"
}
]
},
.
.
.
]
}
FILE3.json
{
"Plugins": [
{
"Name": "Common API for Blue Ocean",
"Installation": [
{
"Version": "1.10.1",
"Enabled": true,
"URL": "https://wiki.jenkins-ci.org/display/JENKINS/Blue+Ocean+Plugin",
"Server": "Server3"
}
]
},
{
"Name": "Support Core Plugin",
"Installation": [
{
"Version": "2.54",
"Enabled": true,
"URL": "https://wiki.jenkins.io/display/JENKINS/Support+Core+Plugin",
"Server": "Server3"
}
]
},
.
.
.
]
}
I want to merge them into a single file with the following structure using groovy
MergedFile.json
{
"Plugins": [
{
"Name": "Async Http Client",
"Installation": [
{
"Version": "1.7.24.2",
"Enabled": true,
"URL": "http://wiki.jenkins-ci.org/display/JENKINS/Async+Http+Client+Plugin",
"Server": "Server1"
},
{
"Version": "1.7.24.1",
"Enabled": true,
"URL": "http://wiki.jenkins-ci.org/display/JENKINS/Async+Http+Client+Plugin",
"Server": "Server2"
}
]
},
{
"Name": "Common API for Blue Ocean",
"Installation": [
{
"Version": "1.10.2",
"Enabled": true,
"URL": "https://wiki.jenkins-ci.org/display/JENKINS/Blue+Ocean+Plugin",
"Server": "Server1"
},
{
"Version": "1.10.2",
"Enabled": true,
"URL": "https://wiki.jenkins-ci.org/display/JENKINS/Blue+Ocean+Plugin",
"Server": "Server2"
},
{
"Version": "1.10.1",
"Enabled": true,
"URL": "https://wiki.jenkins-ci.org/display/JENKINS/Blue+Ocean+Plugin",
"Server": "Server3"
}
]
},
{
"Name": "Support Core Plugin",
"Installation": [
{
"Version": "2.54",
"Enabled": true,
"URL": "https://wiki.jenkins.io/display/JENKINS/Support+Core+Plugin",
"Server": "Server3"
}
]
},
.
.
.
]
}
I am looking for a jenkins pipeline script using jenkins groovy. The file structure of the 3 files are the same. Only the file content maybe different. If there is any common "Name", "Installation" is merged under the same "Name" like mentioned above.
Any help is appreciated!
Edit:
I've tried something like this..
files = findFiles(glob: "*.json")
def input
def mergedjson = readJSON file:"${files[0]}"
for (f=1;f<files.size();f++)
{
input=readJSON file:"${files[f]}"
i = input.Plugins.size()
j = mergedjson.Plugins.size()
for (x=0;x<i;x++)
{
def json_status=0
for (y=0;y<j;y++)
{
if (input.Plugins[x].Name == mergedjson.Plugins[y].Name)
{
mergedjson.Plugins[y].Installation += input.Plugins[x].Installation
json_status=1
break;
}
}
if (json_status == 0)
mergedjson.Plugins[j++] = input.Plugins[x]
}
}
inventory=JsonOutput.prettyPrint(JsonOutput.toJson(mergedjson))
writeFile([file: "Inventory.json", text: inventory])
I am getting the following error
java.lang.StackOverflowError
at java.lang.reflect.InvocationTargetException.(InvocationTargetException.java:72)
at sun.reflect.GeneratedMethodAccessor544.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaBeanProperty.getProperty(MetaBeanProperty.java:59)
at groovy.lang.PropertyValue.getValue(PropertyValue.java:42)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.getProperties(DefaultGroovyMethods.java:389)
at groovy.json.JsonOutput.getObjectProperties(JsonOutput.java:327)
at groovy.json.JsonOutput.writeObject(JsonOutput.java:320)
at groovy.json.JsonOutput.writeMap(JsonOutput.java:458)
at groovy.json.JsonOutput.writeObject(JsonOutput.java:321)
at groovy.json.JsonOutput.writeMap(JsonOutput.java:458)
at groovy.json.JsonOutput.writeObject(JsonOutput.java:321)
at groovy.json.JsonOutput.writeMap(JsonOutput.java:458)
at groovy.json.JsonOutput.writeObject(JsonOutput.java:321)
.
.
.

Related

Unable to parse JSON data for Oracle Integration Cloud using JQ

need help to parse the JSON data received from Oracle Integration Cloud. The expected output is mentioned below alongwith the command i am trying to use.
JQ command
jq '[{id: .id},{integrations: [.integrations[]|{code: .code, version: .version, dependencies: .dependencies|{connections: .connections[]|{id: .id, status: .status}}, .dependencies|{lookups: .lookups}}]}]' output.json
Error :
jq: error: syntax error, unexpected FIELD (Unix shell quoting issues?) at , line 1:
[{id: .id},{integrations: [.integrations[]|{code: .code, version: .version, dependencies: .dependencies|{connections: .connections[]|{id: .id, status: .status}}, .dependencies|{lookups: .lookups}}]}]
Note : If i run below command to fetch only connections data it works fine
jq '[{id: .id},{integrations: [.integrations[]|{code: .code, version: .version, dependencies: .dependencies|{connections: .connections[]|{id: .id, status: .status}}}]}]' output.json
Expected Output:
[
{
"id": "SAMPLE_PACKAGE"
},
{
"integrations": [
{
"code": "HELLO_INTEGRATION",
"version": "01.00.0000",
"dependencies": {
"connections": {
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
}
}
},
{
"code": "HELLO_INTEGRATIO_LOOKUP",
"version": "01.00.0000",
"dependencies": {
"connections": {
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
},
"lookups": {
"name": "COMMON_LOOKUP_VARIABLES",
"status": "CONFIGURED"
}
}
},
{
"code": "HI_INTEGRATION",
"version": "01.00.0000",
"dependencies": {
"connections": {
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
}
}
}
]
}
]
output.json file contains
{
"bartaType": "DEVELOPED",
"countOfIntegrations": 3,
"id": "SAMPLE_PACKAGE",
"integrations": [
{
"code": "HELLO_INTEGRATION",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"lockedFlag": false,
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED",
"type": "rest",
"usage": 6
}
]
},
"description": "",
"eventSubscriptionFlag": false,
"filmstrip": [
{
"code": "HELLO_WORLD1",
"iconUrl": "/images/rest/rest_icon_46.png",
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED"
}
],
"id": "HELLO_INTEGRATION|01.00.0000",
"lockedFlag": false,
"name": "HELLO_INTEGRATION",
"pattern": "Orchestration",
"patternDescription": "Map Data",
"payloadTracingEnabledFlag": true,
"publishFlag": false,
"scheduleApplicable": false,
"scheduleDefined": false,
"status": "ACTIVATED",
"style": "FREEFORM",
"styleDescription": "Orchestration",
"tempCopyExists": false,
"tracingEnabledFlag": true,
"version": "01.00.0000",
"warningMsg": "ACTIVATE_PUBLISH_NO_CONN"
},
{
"code": "HELLO_INTEGRATIO_LOOKUP",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"lockedFlag": false,
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED",
"type": "rest",
"usage": 6
}
],
"lookups": [
{
"lockedFlag": false,
"name": "COMMON_LOOKUP_VARIABLES",
"status": "CONFIGURED",
"usage": 1
}
]
},
"description": "",
"eventSubscriptionFlag": false,
"filmstrip": [
{
"code": "HELLO_WORLD1",
"iconUrl": "/images/rest/rest_icon_46.png",
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED"
}
],
"id": "HELLO_INTEGRATIO_LOOKUP|01.00.0000",
"lockedFlag": false,
"name": "HELLO_INTEGRATION_LOOKUP",
"pattern": "Orchestration",
"patternDescription": "Map Data",
"payloadTracingEnabledFlag": true,
"publishFlag": false,
"scheduleApplicable": false,
"scheduleDefined": false,
"status": "ACTIVATED",
"style": "FREEFORM",
"styleDescription": "Orchestration",
"tempCopyExists": false,
"tracingEnabledFlag": true,
"version": "01.00.0000",
"warningMsg": "ACTIVATE_PUBLISH_NO_CONN"
},
{
"code": "HI_INTEGRATION",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"lockedFlag": false,
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED",
"type": "rest",
"usage": 6
}
]
},
"description": "",
"eventSubscriptionFlag": false,
"filmstrip": [
{
"code": "HELLO_WORLD1",
"iconUrl": "/images/rest/rest_icon_46.png",
"name": "Hello World1",
"role": "SOURCE",
"status": "CONFIGURED"
}
],
"id": "HI_INTEGRATION|01.00.0000",
"lockedFlag": false,
"name": "HI_INTEGRATION",
"pattern": "Orchestration",
"patternDescription": "Map Data",
"payloadTracingEnabledFlag": true,
"publishFlag": false,
"scheduleApplicable": false,
"scheduleDefined": false,
"status": "ACTIVATED",
"style": "FREEFORM",
"styleDescription": "Orchestration",
"tempCopyExists": false,
"tracingEnabledFlag": true,
"version": "01.00.0000",
"warningMsg": "ACTIVATE_PUBLISH_NO_CONN"
}
],
"isCloneAllowed": false,
"isViewAllowed": false,
"name": "SAMPLE_PACKAGE",
"type": "DEVELOPED"
}
The problem is that the lookups key is not always present so, you cannot use the [] on it. So, instead you can use the map function and provide a default before piping to the map function like below
[
{ id: .id },
{
integrations: [
.integrations[]|{
id: .id,
code: .code,
dependencies: {
connections: (.dependencies.connections//[]|map({id,status}))[0],
lookups: (.dependencies.lookups//[]|map({name,status}))[0]
}
}
]
}
]
The (.dependencies.lookups//[]|map({name,status}))[0] has the effect of passing an empty array to the map function which results in a null value when accessing the first element.
See in action https://jqplay.org/s/zQBkHtnzOd1
The provided JQ statement works fine for single elements in the array , but incase the array contains multiple elements it only fetches the first element. Also i updated the dependencies object to capture all the arrays ( connections,lookups,certificates,libraries,integrations)
Below is the modified one. Please suggest for any better options.
[
{ id: .id },
{
integrations: [
.integrations[]|{
id: .id,
code: .code,
dependencies: {
connections: (.dependencies.connections//[]|map({id,status})),
lookups: (.dependencies.lookups//[]|map({name,status})),
certificates: (.dependencies.certificates//[]|map({id,status})),
libraries: (.dependencies.libraries//[]|map({code,status,version})),
integrations: (.dependencies.integrations//[]|map({code,version}))
}
}
]
}
]|del(..|select(.==[]))
Note: To remove the empty arrays del function is added which is giving the below output :
[
{
"id": "SAMPLE_PACKAGE"
},
{
"integrations": [
{
"id": "HELLO_INTEGRATION|01.00.0000",
"code": "HELLO_INTEGRATION",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
},
{
"id": "HELLO_WORLD2",
"status": "CONFIGURED"
}
]
}
},
{
"id": "HELLO_INTEGRATIO_LOOKUP|01.00.0000",
"code": "HELLO_INTEGRATIO_LOOKUP",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
}
],
"lookups": [
{
"name": "COMMON_LOOKUP_VARIABLES",
"status": "CONFIGURED"
}
]
}
},
{
"id": "HI_INTEGRATION|01.00.0000",
"code": "HI_INTEGRATION",
"dependencies": {
"connections": [
{
"id": "HELLO_WORLD1",
"status": "CONFIGURED"
}
]
}
}
]
}
]

Change json object with jq

how can I update servers object in my json using jq?
"tags": [],
"servers": [],
"components": {
"securitySchemes": {
"cookie": {
"type": "apiKey",
"in": "cookie",
"name": "store",
"description": "Store API"
}
},
I need to have servers like that:
"servers": [
{
"url": "http://localhost:5000",
"description": "Localhost server"
}
],
echo '{"tags": [],
"servers": [],
"components": {
"securitySchemes": {
"cookie": {
"type": "apiKey",
"in": "cookie",
"name": "store",
"description": "Store API"
}
}
}}' | jq '.servers = [{url:"http://localhost:5000", description: "Localhost server"}]'
Or, for a fixed json file (adding missing }):
jq '.servers = [{url:"http://localhost:5000", description: "Localhost server"}]' file.json

Kubernetes + jq - retrieving containers list per pod yields cartesian product

Im trying to use jq on kubernetes json output, to create new json object containing list of objects - container and image per pod, however im getting cartesian product.
my input data (truncated from sensitive info):
{
"apiVersion": "v1",
"items": [
{
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"creationTimestamp": "2021-06-30T12:45:40Z",
"name": "pod-1",
"namespace": "default",
"resourceVersion": "757679286",
"selfLink": "/api/v1/namespaces/default/pods/pod-1"
},
"spec": {
"containers": [
{
"image": "image-1",
"imagePullPolicy": "Always",
"name": "container-1",
"resources": {},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"volumeMounts": [
{
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
"readOnly": true
}
]
},
{
"image": "image-2",
"imagePullPolicy": "Always",
"name": "container-2",
"resources": {},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"volumeMounts": [
{
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
"readOnly": true
}
]
}
],
"dnsPolicy": "ClusterFirst",
"enableServiceLinks": true,
"priority": 0,
"restartPolicy": "Always",
"schedulerName": "default-scheduler",
"securityContext": {},
"serviceAccount": "default",
"serviceAccountName": "default",
"terminationGracePeriodSeconds": 30,
"tolerations": [
{
"effect": "NoExecute",
"key": "node.kubernetes.io/not-ready",
"operator": "Exists",
"tolerationSeconds": 300
},
{
"effect": "NoExecute",
"key": "node.kubernetes.io/unreachable",
"operator": "Exists",
"tolerationSeconds": 300
}
],
"volumes": [
{
"name": "default-token-b954f",
"secret": {
"defaultMode": 420,
"secretName": "default-token-b954f"
}
}
]
},
"status": {
"conditions": [
{
"lastProbeTime": null,
"lastTransitionTime": "2021-06-30T12:45:40Z",
"status": "True",
"type": "Initialized"
},
{
"lastProbeTime": null,
"lastTransitionTime": "2021-06-30T12:45:40Z",
"message": "containers with unready status: [container-1 container-2]",
"reason": "ContainersNotReady",
"status": "False",
"type": "Ready"
},
{
"lastProbeTime": null,
"lastTransitionTime": "2021-06-30T12:45:40Z",
"message": "containers with unready status: [container-1 container-2]",
"reason": "ContainersNotReady",
"status": "False",
"type": "ContainersReady"
},
{
"lastProbeTime": null,
"lastTransitionTime": "2021-06-30T12:45:40Z",
"status": "True",
"type": "PodScheduled"
}
],
"containerStatuses": [
{
"image": "image-1",
"imageID": "",
"lastState": {},
"name": "container-1",
"ready": false,
"restartCount": 0,
"started": false,
"state": {
"waiting": {
"message": "Back-off pulling image \"image-1\"",
"reason": "ImagePullBackOff"
}
}
},
{
"image": "image-2",
"imageID": "",
"lastState": {},
"name": "container-2",
"ready": false,
"restartCount": 0,
"started": false,
"state": {
"waiting": {
"message": "Back-off pulling image \"image-2\"",
"reason": "ImagePullBackOff"
}
}
}
],
"qosClass": "BestEffort",
"startTime": "2021-06-30T12:45:40Z"
}
}
],
"kind": "List",
"metadata": {
"resourceVersion": "",
"selfLink": ""
}
}
my command:
jq '.items[] | { "name": .metadata.name, "containers": [{ "name": .spec.containers[].name, "image": .spec.containers[].image }]} '
desired output:
{
"name": "pod_1",
"containers": [
{
"name": "container_1",
"image": "image_1"
},
{
"name": "container_2",
"image": "image_2"
}
]
}
output I get:
{
"name": "pod-1",
"containers": [
{
"name": "container-1",
"image": "image-1"
},
{
"name": "container-1",
"image": "image-2"
},
{
"name": "container-2",
"image": "image-1"
},
{
"name": "container-2",
"image": "image-2"
}
]
}
Could anyone explain what am I doing wrong?
Best Regards, Piotr.
The problem is "name": .spec.containers[].name and "image": .spec.containers[].image:
Both expressions generate a sequence of each value for name and image which will than be combined.
Simplified example of why you get a Cartesian product:
jq -c -n '{name: ("A", "B"), value: ("C", "D")}'
outputs:
{"name":"A","value":"C"}
{"name":"A","value":"D"}
{"name":"B","value":"C"}
{"name":"B","value":"D"}
You get the desired output using this jq filter on your input:
jq '
.items[]
| {
"name": .metadata.name,
"containers": .spec.containers
| map({name, image})
}'
output:
{
"name": "pod-1",
"containers": [
{
"name": "container-1",
"image": "image-1"
},
{
"name": "container-2",
"image": "image-2"
}
]
}

Cloudformation template to create EMR cluster

I am trying to create EMR-5.30.1 clusters with applications such as Hadoop, livy, Spark, ZooKeeper, and Hive with the help of the CloudFormation template. But the issue is with this template is I am able the cluster with only one application from the above list of applications.
below is the CloudFormation Template
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Best Practice EMR Cluster for Spark or S3 backed Hbase",
"Parameters": {
"EMRClusterName": {
"Description": "Name of the cluster",
"Type": "String",
"Default": "emrcluster"
},
"KeyName": {
"Description": "Must be an existing Keyname",
"Type": "String",
"Default": "keyfilename"
},
"MasterInstanceType": {
"Description": "Instance type to be used for the master instance.",
"Type": "String",
"Default": "m5.xlarge"
},
"CoreInstanceType": {
"Description": "Instance type to be used for core instances.",
"Type": "String",
"Default": "m5.xlarge"
},
"NumberOfCoreInstances": {
"Description": "Must be a valid number",
"Type": "Number",
"Default": 1
},
"SubnetID": {
"Description": "Must be Valid public subnet ID",
"Default": "subnet-ee15b3e0",
"Type": "String"
},
"LogUri": {
"Description": "Must be a valid S3 URL",
"Default": "s3://aws/elasticmapreduce/",
"Type": "String"
},
"S3DataUri": {
"Description": "Must be a valid S3 bucket URL ",
"Default": "s3://aws/elasticmapreduce/",
"Type": "String"
},
"ReleaseLabel": {
"Description": "Must be a valid EMR release version",
"Default": "emr-5.30.1",
"Type": "String"
},
"Applications": {
"Description": "Please select which application will be installed on the cluster this would be either Ganglia and spark, or Ganglia and s3 backed Hbase",
"Type": "String",
"AllowedValues": [
"Spark",
"Hbase",
"Hive",
"Livy",
"ZooKeeper"
]
}
},
"Mappings": {},
"Conditions": {
"Spark": {
"Fn::Equals": [
{
"Ref": "Applications"
},
"Spark"
]
},
"Hbase": {
"Fn::Equals": [
{
"Ref": "Applications"
},
"Hbase"
]
},
"Hive": {
"Fn::Equals": [
{
"Ref": "Applications"
},
"Hive"
]
},
"Livy": {
"Fn::Equals": [
{
"Ref": "Applications"
},
"Livy"
]
},
"ZooKeeper": {
"Fn::Equals": [
{
"Ref": "Applications"
},
"ZooKeeper"
]
}
},
"Resources": {
"EMRCluster": {
"DependsOn": [
"EMRClusterServiceRole",
"EMRClusterinstanceProfileRole",
"EMRClusterinstanceProfile"
],
"Type": "AWS::EMR::Cluster",
"Properties": {
"Applications": [
{
"Name": "Ganglia"
},
{
"Fn::If": [
"Spark",
{
"Name": "Spark"
},
{
"Ref": "AWS::NoValue"
}
]
},
{
"Fn::If": [
"Hbase",
{
"Name": "Hbase"
},
{
"Ref": "AWS::NoValue"
}
]
},
{
"Fn::If": [
"Hive",
{
"Name": "Hive"
},
{
"Ref": "AWS::NoValue"
}
]
},
{
"Fn::If": [
"Livy",
{
"Name": "Livy"
},
{
"Ref": "AWS::NoValue"
}
]
},
{
"Fn::If": [
"ZooKeeper",
{
"Name": "ZooKeeper"
},
{
"Ref": "AWS::NoValue"
}
]
}
],
"Configurations": [
{
"Classification": "hbase-site",
"ConfigurationProperties": {
"hbase.rootdir":{"Ref":"S3DataUri"}
}
},
{
"Classification": "hbase",
"ConfigurationProperties": {
"hbase.emr.storageMode": "s3"
}
}
],
"Instances": {
"Ec2KeyName": {
"Ref": "KeyName"
},
"Ec2SubnetId": {
"Ref": "SubnetID"
},
"MasterInstanceGroup": {
"InstanceCount": 1,
"InstanceType": {
"Ref": "MasterInstanceType"
},
"Market": "ON_DEMAND",
"Name": "Master"
},
"CoreInstanceGroup": {
"InstanceCount": {
"Ref": "NumberOfCoreInstances"
},
"InstanceType": {
"Ref": "CoreInstanceType"
},
"Market": "ON_DEMAND",
"Name": "Core"
},
"TerminationProtected": false
},
"VisibleToAllUsers": true,
"JobFlowRole": {
"Ref": "EMRClusterinstanceProfile"
},
"ReleaseLabel": {
"Ref": "ReleaseLabel"
},
"LogUri": {
"Ref": "LogUri"
},
"Name": {
"Ref": "EMRClusterName"
},
"AutoScalingRole": "EMR_AutoScaling_DefaultRole",
"ServiceRole": {
"Ref": "EMRClusterServiceRole"
}
}
},
"EMRClusterServiceRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"elasticmapreduce.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"ManagedPolicyArns": [
"arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole"
],
"Path": "/"
}
},
"EMRClusterinstanceProfileRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"ec2.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"ManagedPolicyArns": [
"arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role"
],
"Path": "/"
}
},
"EMRClusterinstanceProfile": {
"Type": "AWS::IAM::InstanceProfile",
"Properties": {
"Path": "/",
"Roles": [
{
"Ref": "EMRClusterinstanceProfileRole"
}
]
}
}
},
"Outputs": {}
}
Also, I want to add a bootstrap script in this template as well, Can anyone please help me with the issue.
As per my knoweldge and understanding, Applications in your case should be an array like below, as mentioned in documentation
"Applications" : [ Application, ... ],
In you case, you can list applications like
"Applications" : [
{"Name" : "Spark"},
{"Name" : "Hbase"},
{"Name" : "Hive"},
{"Name" : "Livy"},
{"Name" : "Zookeeper"},
]
For more arguments other than Name to individual application dictionary , see detail here, you can pass Args, Additional_info etc
You can use following way:-
If you set "ReleaseLabel" then there is no need to mention versions of applications
"Applications": [{
"Name": "Hive"
},
{
"Name": "Presto"
},
{
"Name": "Spark"
}
]
For bootstrap:-
"BootstrapActions": [{
"Name": "setup",
"ScriptBootstrapAction": {
"Path": "s3://bucket/key/Bootstrap.sh"
}
}]
Define like this to create all applications at once.
{
"Type": "AWS::EMR::Cluster",
"Properties": {
"Applications": [
{
"Name": "Ganglia"
},
{
"Name": "Spark"
},
{
"Name": "Livy"
},
{
"Name": "ZooKeeper"
},
{
"Name": "JupyterHub"
}
]
}
}

Elasticsearch watcher and Microsoft Teams webhook

I have been trying desperately for 5 days to create an elasticsearch watcher alert that sends a notification on an incoming webhook teams. However, the answer I receive is "Bad payload received by generic incoming webhook". I do not understand why it does not work.
{
"trigger": {
"schedule": {
"interval": "2m"
}
},
"input": {
"simple": {
"summary": "Test Nom",
"text": "test"
}
},
"condition": {
"always": {}
},
"actions": {
"MS_TEAMS_PORT443": {
"webhook": {
"scheme": "https",
"host": "outlook.office.com",
"port": 443,
"method": "post",
"path": "/webhook/XYZ",
"params": {},
"headers": {
"content-type": "application/json"
},
"body": "{{#toJson}}ctx.payload.summary{{/toJson}}"
}
}
}
}
And this is the response when I launch it:
{
"watch_id": "_inlined_",
"node": "QUApyNq4S5GyhHF-CuNjfg",
"state": "executed",
"status": {
"state": {
"active": true,
"timestamp": "2019-10-21T08:40:39.802Z"
},
"last_checked": "2019-10-21T08:40:39.802Z",
"last_met_condition": "2019-10-21T08:40:39.802Z",
"actions": {
"MS_TEAMS_PORT443": {
"ack": {
"timestamp": "2019-10-21T08:40:39.802Z",
"state": "awaits_successful_execution"
},
"last_execution": {
"timestamp": "2019-10-21T08:40:39.802Z",
"successful": false,
"reason": "received [400] status code"
}
}
},
"execution_state": "executed",
"version": -1
},
"trigger_event": {
"type": "manual",
"triggered_time": "2019-10-21T08:40:39.802Z",
"manual": {
"schedule": {
"scheduled_time": "2019-10-21T08:40:39.802Z"
}
}
},
"input": {
"simple": {
"summary": "Test Nom",
"text": "test"
}
},
"condition": {
"always": {}
},
"metadata": {
"name": "testJsonALaMano",
"xpack": {
"type": "json"
}
},
"result": {
"execution_time": "2019-10-21T08:40:39.802Z",
"execution_duration": 125,
"input": {
"type": "simple",
"status": "success",
"payload": {
"summary": "Test Nom",
"text": "test"
}
},
"condition": {
"type": "always",
"status": "success",
"met": true
},
"actions": [
{
"id": "MS_TEAMS_PORT443",
"type": "webhook",
"status": "failure",
"reason": "received [400] status code",
"webhook": {
"request": {
"host": "outlook.office.com",
"port": 443,
"scheme": "https",
"method": "post",
"path": "/webhook/XYZ",
"headers": {
"content-type": "application/json"
},
"body": "Test Nom"
},
"response": {
"status": 400,
"headers": {
"date": [
"Mon, 21 Oct 2019 08:40:38 GMT"
],
"content-length": [
"49"
],
"expires": [
"-1"
],
"x-beserver": [
"VI1PR07MB5053"
],
"x-aspnet-version": [
"4.0.30319"
],
"x-proxy-backendserverstatus": [
"400"
],
"x-cafeserver": [
"VE1PR03CA0023.EURPRD03.PROD.OUTLOOK.COM"
],
"x-calculatedbetarget": [
"VI1PR07MB5053.eurprd07.prod.outlook.com"
],
"request-id": [
"6d651f70-74b5-4010-a2a6-662666fa9985"
],
"pragma": [
"no-cache"
],
"x-msedge-ref": [
"Ref A: C6E8A3DCFF9541DD95D63FD71ACD695C Ref B: PAR02EDGE0513 Ref C: 2019-10-21T08:40:39Z"
],
"x-feserver": [
"VE1PR03CA0023"
],
"x-powered-by": [
"ASP.NET"
],
"x-backendhttpstatus": [
"400"
],
"content-type": [
"text/plain; charset=utf-8"
],
"cache-control": [
"no-cache"
]
},
"body": "Bad payload received by generic incoming webhook."
}
}
}
]
},
"messages": []
}
Wrong body statement. Need to send text in body. Change your body to be like this
"body": "{\"text\": \"{{ctx.payload.summary}}\"}"
source property is your friend here:
"body": {
"source": {
"text" : "Test Nom"
}
}