I've a simple JSON look like:
{
"account_login" : "google#gmail.com",
"view_id" : 1868715,
"join_id" : "utm_campaign=toyota&utm_content=multiformat_sites&utm_medium=cpc&utm_source=facebook",
"start_date" : "2020-02-03",
"end_date" : "2020-08-30"
}
With following Groovy script (from this answer):
def content = """
{
"account_login" : "google#gmail.com",
"view_id" : 1868715,
"join_id" : "utm_campaign=toyota&utm_content=multiformat_sites&utm_medium=cpc&utm_source=facebook",
"start_date" : "2020-02-03",
"end_date" : "2020-08-30"
}
"""
def slurped = new JsonSlurper().parseText(content)
def builder = new JsonBuilder(slurped)
builder.content.join_id = builder.content.join_id.split("\\s*&\\s*") //# to array
.collectEntries{
//# convert each item to map entry
String[] utmMarks = it.trim().split("\\s*=\\s*")
utmMarks[0] = [
"utm_medium" : "ga:medium",
"utm_campaign" : "ga:campaign",
"utm_source" : "ga:source",
"utm_content" : "ga:adContent",
"utm_term" : "ga:keyword",
].get( utmMarks[0] )
utmMarks
}
.findAll{
k,v-> k && v!=null //# filter out empty/null keys
}
//builder.content.filters = ...
println(builder.toPrettyString())
I'll get:
{
"account_login": "google#gmail.com",
"view_id": 1868715,
"join_id": {
"ga:campaign": "toyota",
"ga:adContent": "multiformat_sites",
"ga:medium": "cpc",
"ga:source": "facebook"
},
"start_date": "2020-02-03",
"end_date": "2020-08-30"
}
I want to update this script (or write new) and add new property: array filters to modified json above. Expected output:
{
"account_login":"google#gmail.com",
"view_id":1868715,
"join_id":{
"ga:campaign":"toyota",
"ga:adContent":"multiformat_sites",
"ga:medium":"cpc",
"ga:source":"facebook"
},
"start_date":"2020-02-03",
"end_date":"2020-08-30",
"converted_utm_marks":"ga:campaign=toyota&ga:adContent=multiformat_sites&ga:medium=cpc&ga:source=facebook",
"filters":[
{
"dimensionName":"ga:medium",
"operator":"EXACT",
"expressions":[
"cpc"
]
},
{
"dimensionName":"ga:adContent",
"operator":"EXACT",
"expressions":[
"multiformat_sites"
]
},
{
"dimensionName":"ga:campaign",
"operator":"EXACT",
"expressions":[
"toyota"
]
},
{
"dimensionName":"ga:source",
"operator":"EXACT",
"expressions":[
"facebook"
]
}
]
}
But the problem is that the set of filters for each JSON will be different. This set depends directly on the join_id set. If JSON join_id will contain:
"join_id": {
"ga:campaign": "toyota",
"ga:keyword": "car"
}
filters array should be:
[
{
"dimensionName":"ga:campaign",
"operator":"EXACT",
"expressions":[
"toyota"
]
},
{
"dimensionName":"ga:keyword",
"operator":"EXACT",
"expressions":[
"car"
]
}
]
operator is always equals EXACT. Property dimensionName - is a join_id.propety name. Expressions is a join_id.property value. So, property filters based on join_id and I need to loop through join_id property and build filters array with described structure. How to achieve expected output? JOLT configuration appreciated also.
I can't even simple iterate through join_id map:
slurped.join_id.each { println "Key: $it.key = Value: $it.value" }
I got the error:
/home/jdoodle.groovy: 24: illegal colon after argument expression;
solution: a complex label expression before a colon must be parenthesized # line 24, column 28.
.collect { [it.ga:campaign] }
UPDATE
I found out how to build this array:
def array =
[
filters: slurped.join_id.collect {key, value ->
[
dimensionName: key,
operator: "EXACT",
expressions: [
value
]
]
}
]
Seems like i got it:
def slurped = new JsonSlurper().parseText(content)
def builder = new JsonBuilder(slurped)
builder.content.filters = builder.content.join_id.collect {key, value ->
[
dimensionName: key,
operator: "EXACT",
expressions: [
value
]
]
}
Are there any better solutions?
def slurped = new JsonSlurper().parseText(content)
def builder = new JsonBuilder(slurped)
builder.content.filters = builder.content.join_id.collect {key, value ->
[
dimensionName: key,
operator: "EXACT",
expressions: [
value
]
]
}
Related
I am trying to access the first key on a given tuple. The key's name is dynamic and may change so it cannot be accessed using a static value and has to be done either through a for loop or through the use of terraform fuctions.
I've created a small local resource that outputs the following section
skip_empty_mails = { for key, value in local.j.settings.tasks : key => value.email_notifications if value.email_notifications != {} }
The output sent back is
{
"3" = {
"on_start" = [
"foo1#aligntech.com",
"foo2#aligntech.com",
"foo3#aligntech.com",
"foo4#aligntech.com",
]
}
"4" = {
"no_alert_for_skipped_runs" = false
"on_start" = [
"foo21#aligntech.com",
"foo22#aligntech.com",
"foo23#aligntech.com",
"foo24#aligntech.com",
]
]
"on_start" = [
"foo21#aligntech.com",
"foo22#aligntech.com",
"foo23#aligntech.com",
"foo24#aligntech.com",
]
"on_success" = [
"foo21#aligntech.com",
"foo22#aligntech.com",
"foo23#aligntech.com",
"foo24#aligntech.com",
]
}
}
As seen above the key that holds all the values needs to be accessed in a way that will give me the ability to attach it to a string and use string.on_start to pull its values.
The issue is that our key's name is dynamic and may vary.
I've tried following the terraform function documentation But haven't found anything that might be of use in this case.
You may be able to replicate using the following code
locals {
json = {
"3" = {
"on_start" = [
"foo1#aligntech.com",
"foo2#aligntech.com",
"foo3#aligntech.com",
"foo4#aligntech.com",
]
},
"4" = {
"no_alert_for_skipped_runs" = false
"on_failure" = [
"foo21#aligntech.com",
"foo22#aligntech.com",
"foo23#aligntech.com",
"foo24#aligntech.com",
]
"on_start" = [
"foo21#aligntech.com",
"foo22#aligntech.com",
"foo23#aligntech.com",
"foo24#aligntech.com",
]
"on_success" = [
"foo1#foo.com",
"foo2#foo.com",
"foo3#foo.com",
"foo4#foo.com",
]
}
}
}
You can try with a combination of values, element or flatten see the documentation:
https://developer.hashicorp.com/terraform/language/functions/values
https://developer.hashicorp.com/terraform/language/functions/element
https://developer.hashicorp.com/terraform/language/functions/flatten
Below are samples:
First key extraction
locals {
json = {
"4" = {
"no_alert_for_skipped_runs" = false
"on_failure" = [
"foo1#foo.com",
"foo2#foo.com",
]
"on_start" = [
"foo1#foo.com",
"foo2#foo.com",
]
}
}
}
output "data" {
value = element(values(local.json), 1).on_start
}
the Terraform plan will be:
Changes to Outputs:
+ data = [
+ "foo1#foo.com",
+ "foo2#foo.com",
]
Extract and combine on_start item from all
locals {
json = {
"3" = {
"on_start" = [
"foo1#aligntech.com",
"foo2#aligntech.com",
]
},
"4" = {
"on_start" = [
"foo1#foo.com",
"foo2#foo.com",
]
}
}
}
output "data" {
value = flatten(values(local.json)[*].on_start)
}
the Terraform plan will be:
Changes to Outputs:
+ data = [
+ "foo1#aligntech.com",
+ "foo2#aligntech.com",
+ "foo1#foo.com",
+ "foo2#foo.com",
]
I have another question about JSON parsing. Imagine I have this JSON:
{
"all_dogs" :[
{
"name": "foo",
"groups": ["morning", "evening"]
},
{
"name": "bar",
"groups": ["evening", "saturday"]
},
{
"name": "feet",
"groups": ["afternoon"]
}
]
}
I can extract all the groups like this:
locals {
all_dogs = jsondecode(file("${path.module}/dogs.json"))
all_groups = toset(flatten(local.all_dogs.all_dogs[*].groups))
}
Now, I’m trying to create a MAP. Each group is a key of the map and values of the maps are the different dogs in that group.
So I would like to create a map like this:
afternoon = [feet]
evening= [foo, bar]
morning= [foo]
saturday= [bar]
I'm trying with something like this and I tried several options... But I can't make it work.
output "ex" {
value = flatten([
for group in local.all_groups: [
for dog in local.all_dogs : {
group = group
dog = dog
}
]
]
)
}
Later on, I would like to use that map to provision some resources. Is this eventually possible?
locals {
all_dogs = jsondecode(file("${path.module}/dogs.json"))
groups = flatten([for d in local.all_dogs.all_dogs : [for g in d.groups : { key : g, value : d.name }]])
}
The value for local groups will be a list of tuples and it will look something like this:
groups = [
{
"key" = "morning"
"value" = "foo"
},
{
"key" = "evening"
"value" = "foo"
},
{
"key" = "evening"
"value" = "bar"
},
{
"key" = "saturday"
"value" = "bar"
},
{
"key" = "afternoon"
"value" = "feet"
},
]
Now we have to create a map from this list:
output "my_map" {
value = {
for g in local.groups : g.key => g.value...
}
}
This will produce the following output:
my_map= {
"afternoon" = [
"feet",
]
"evening" = [
"foo",
"bar",
]
"morning" = [
"foo",
]
"saturday" = [
"bar",
]
I would like to parse the below nested JSON in Groovy and get the values of "it","ft","stg","prd" for each application and store in separate array.Can someone help please ?
{
"Application1": {
"environments": {
"it": [
"server1"
],
"ft": [
"server2"
],
"stg": [
"server3"
],
"prd": [
"server4"
]
},
"war-path" : "/opt/tomcat/",
"war-name" : "Application1"
},
"Application2": {
"environments": {
"it": [
"serverA"
],
"ft": [
"serverB"
],
"stg": [
"serverC"
],
"prd": [
"serverD"
]
},
"war-path" : "/var/lib",
"war-name" : "Application2"
}
}
}
Expected output something like below in separate list for each environment. Also the 1st level(Application1,Application2..) will be dynamic always
it = [server1,serverA]
ft = [server2,serverB]
stg = [server3, serverC]
prd = [server4,serverD]
Updated: After deriving expected answer with the inputs from Philip Wrage.
def projects = readJSON file: "${env.WORKSPACE}//${infrafile}"
def json_str = JsonOutput.toJson(projects)
def json_beauty = JsonOutput.prettyPrint(json_str)
def envlist = ["it","ft","stg","prd"]
def fileListResult = []
for (envname in envlist) {
servers=serverlist(json_beauty,envname)
println(servers)
}
def serverlist(json_beauty,envname){
def jsonSlurper = new JsonSlurper()
def jsonMap = jsonSlurper.parseText(json_beauty)
assert jsonMap instanceof Map
jsonMap.each { appName, appDetails ->
assert appDetails instanceof Map
appDetails.environments.each { envName, servers ->
for (items in servers{
if (envName == "${envname}"){
fileListResult.add(items)
}
}
}
}
return fileListResult
}
As suggested by #Chris, you can use the built-in JsonSlurper to parse the JSON and then navigate the parsed results as a Map.
In the following example I show how you can simply print the results to the console. However, using this as a guide, you can see how to extract the data into whatever kind of data structure that suits your purpose.
Assume that you have the JSON in a String variable jsonText. You may be pulling in from a file or an HTTP POST or whatever your app requires. I used the following code to set this value for testing.
def jsonText = """
{
"Application1": {
"environments": {
"it": [
"server1"
],
"ft": [
"server2"
],
"stg": [
"server3"
],
"prd": [
"server4"
]
},
"war-path" : "/opt/tomcat/",
"war-name" : "Application1"
},
"Application2": {
"environments": {
"it": [
"serverA"
],
"ft": [
"serverB"
],
"stg": [
"serverC"
],
"prd": [
"serverD"
]
},
"war-path" : "/var/lib",
"war-name" : "Application2"
}
}
"""
The meat of the solution then follows. Parse the JSON text into a Map, and then iterate over the entries in that Map, performing whatever operations you require. The servers for each environment will already be contained within a List.
import groovy.json.JsonSlurper
def jsonSlurper = new JsonSlurper()
def jsonMap = jsonSlurper.parseText(jsonText)
assert jsonMap instanceof Map
jsonMap.each { appName, appDetails ->
println "Application: $appName"
assert appDetails instanceof Map
appDetails.environments.each { envName, servers ->
assert servers instanceof List
println "\tEnvironment: $envName"
println "\t\t$servers"
}
}
From this code I obtain the following console output.
Application: Application1
Environment: it
[server1]
Environment: ft
[server2]
Environment: stg
[server3]
Environment: prd
[server4]
Application: Application2
Environment: it
[serverA]
Environment: ft
[serverB]
Environment: stg
[serverC]
Environment: prd
[serverD]
EDIT (based on clarification of required output):
import groovy.json.JsonSlurper
def jsonText = "\n{\n \"Application1\": {\n \"environments\": {\n \"it\": [\n \"server1\"\n ],\n \"ft\": [\n \"server2\"\n ],\n \"stg\": [\n \"server3\"\n ],\n \"prd\": [\n \"server4\"\n ]\n },\n \"war-path\" : \"/opt/tomcat/\",\n \"war-name\" : \"Application1\"\n},\n \"Application2\": {\n \"environments\": {\n \"it\": [\n \"serverA\"\n ],\n \"ft\": [\n \"serverB\"\n ],\n \"stg\": [\n \"serverC\"\n ],\n \"prd\": [\n \"serverD\"\n ]\n },\n \"war-path\" : \"/var/lib\",\n \"war-name\" : \"Application2\"\n}\n}\n"
def jsonSlurper = new JsonSlurper()
def jsonMap = jsonSlurper.parseText(jsonText)
def result = [:]
jsonMap.each { appName, appDetails ->
appDetails.environments.each { envName, servers ->
if ( !result.containsKey(envName) ) {
result.put(envName, [])
}
result.get(envName).addAll(servers)
}
}
println result
Results are a Map where the keys are the various environments specified within JSON file, and the values are Lists of servers associated with those environments across all applications. You can access any List individually with something like result.stg, or assign these the different variables later if that's desired/required (def stg = result.stg).
[it:[server1, serverA], ft:[server2, serverB], stg:[server3, serverC], prd:[server4, serverD]]
If you want all environments, you can use the spread operator to take
all environments from the values. Next you have to merge the maps on
the keys. E.g.
def json = """ { "Application1": { "environments": { "it": [ "server1" ], "ft": [ "server2" ], "stg": [ "server3" ], "prd": [ "server4" ] }, }, "Application2": { "environments": { "it": [ "serverA" ], "ft": [ "serverB" ], "stg": [ "serverC" ], "prd": [ "serverD" ] }, } } }"""
def data = new groovy.json.JsonSlurper().parseText(json)
println data.values()*.environments.inject{ a, b ->
b.inject(a.withDefault{[]}) { m, kv ->
// with groovy 2.5+: m.tap{ get(kv.key).addAll(kv.value) }
m[kv.key].addAll(kv.value); m
}
}
// → [it:[server1, serverA], ft:[server2, serverB], stg:[server3, serverC], prd:[server4, serverD]]
I am new to groovy and trying
1) from the output of prettyPrint(toJson()), I am trying to get a list of values from a specific key inside an json array using groovy. Using the below JSON output from prettyPrint example below, I am trying to create a list which consists only the values of the name key.
My Code:
def string1 = jiraGetIssueTransitions(idOrKey: jira_id)
echo prettyPrint(toJson(string1.data))
def pretty = prettyPrint(toJson(string1.data))
def valid_strings = readJSON text: "${pretty}"
echo "valid_strings.name : ${valid_strings.name}"
Output of prettyPrint(toJson(string1.data))is below JSON:
{
"expand": "places",
"places": [
{
"id": 1,
"name": "Bulbasaur",
"type": {
"grass",
"poison"
}
},
{
"id": 2,
"name": "Ivysaur",
"type": {
"grass",
"poison"
}
}
}
Expected result
valid_strings.name : ["Bulbasaur", "Ivysaur"]
Current output
valid_strings.name : null
The pretty printed JSON content is invalid.
If the JSON is valid, then names can be accessed as follows:
import groovy.json.JsonSlurper
def text = """
{
"expand": "places",
"places": [{
"id": 1,
"name": "Bulbasaur",
"type": [
"grass",
"poison"
]
},
{
"id": 2,
"name": "Ivysaur",
"type": [
"grass",
"poison"
]
}
]
}
"""
def json = new JsonSlurper().parseText(text)
println(json.places*.name)
Basically, use spray the attribute lookup (i.e., *.name) on the appropriate object (i.e., json.places).
I've used something similar to print out elements within the response in ReadyAPI
import groovy.json.*
import groovy.util.*
def json='[
{ "message" : "Success",
"bookings" : [
{ "bookingId" : 60002172,
"bookingDate" : "1900-01-01T00:00:00" },
{ "bookingId" : 59935582,
"bookingDate" : "1900-01-01" },
{ "bookingId" : 53184048,
"bookingDate" : "2019-01-15",
"testId" : "12803798123",
"overallScore" : "PASS" },
{ "bookingId" : 53183765,
"bookingDate" : "2019-01-15T13:45:00" },
{ "bookingId" : 52783312,
"bookingDate" : "1900-01-01" }
]
}
]
def response = context.expand( json )
def parsedjson = new groovy.json.JsonSlurper().parseText(response)
log.info parsedjson
log.info " Count of records returned: " + parsedjson.size()
log.info " List of bookingIDs in this response: " + parsedjson.bookings*.bookingId
I have a json document like this on my mongodb collection:
Updated document:
{
"_id" : ObjectId("59da4aef8c5d757027a5a614"),
"input" : "hi",
"output" : "Hi. How can I help you?",
"intent" : "[{\"intent\":\"greeting\",\"confidence\":0.8154089450836182}]",
"entities" : "[]",
"context" : "{\"conversation_id\":\"48181e58-dd51-405a-bb00-c875c01afa0a\",\"system\":{\"dialog_stack\":[{\"dialog_node\":\"root\"}],\"dialog_turn_counter\":1,\"dialog_request_counter\":1,\"_node_output_map\":{\"node_5_1505291032665\":[0]},\"branch_exited\":true,\"branch_exited_reason\":\"completed\"}}",
"user_id" : "50001",
"time_in" : ISODate("2017-10-08T15:57:32.000Z"),
"time_out" : ISODate("2017-10-08T15:57:35.000Z"),
"reaction" : "1"
}
I need to perform group by on intent.intent field and I'm using Rstudio with mongolite library.
What I have tried is :
pp = '[{"$unwind": "$intent"},{"$group":{"_id":"$intent.intent", "count": {"$sum":1} }}]'
stats <- chat$aggregate(
pipeline=pp,
options = '{"allowDiskUse":true}'
)
print(stats)
But it's not working, output for above code is
_id count
1 NA 727
If intent attribute type is string and keep the object as string.
We can split it to array with \" and use third item of array.
db.getCollection('test1').aggregate([
{ "$project": { intent_text : { $arrayElemAt : [ { $split: ["$intent", "\""] } ,3 ] } } },
{ "$group": {"_id": "$intent_text" , "count": {"$sum":1} }}
])
Result:
{
"_id" : "greeting",
"count" : 1.0
}