Parsing nested JSON in groovy - json

I would like to parse the below nested JSON in Groovy and get the values of "it","ft","stg","prd" for each application and store in separate array.Can someone help please ?
{
"Application1": {
"environments": {
"it": [
"server1"
],
"ft": [
"server2"
],
"stg": [
"server3"
],
"prd": [
"server4"
]
},
"war-path" : "/opt/tomcat/",
"war-name" : "Application1"
},
"Application2": {
"environments": {
"it": [
"serverA"
],
"ft": [
"serverB"
],
"stg": [
"serverC"
],
"prd": [
"serverD"
]
},
"war-path" : "/var/lib",
"war-name" : "Application2"
}
}
}
Expected output something like below in separate list for each environment. Also the 1st level(Application1,Application2..) will be dynamic always
it = [server1,serverA]
ft = [server2,serverB]
stg = [server3, serverC]
prd = [server4,serverD]
Updated: After deriving expected answer with the inputs from Philip Wrage.
def projects = readJSON file: "${env.WORKSPACE}//${infrafile}"
def json_str = JsonOutput.toJson(projects)
def json_beauty = JsonOutput.prettyPrint(json_str)
def envlist = ["it","ft","stg","prd"]
def fileListResult = []
for (envname in envlist) {
servers=serverlist(json_beauty,envname)
println(servers)
}
def serverlist(json_beauty,envname){
def jsonSlurper = new JsonSlurper()
def jsonMap = jsonSlurper.parseText(json_beauty)
assert jsonMap instanceof Map
jsonMap.each { appName, appDetails ->
assert appDetails instanceof Map
appDetails.environments.each { envName, servers ->
for (items in servers{
if (envName == "${envname}"){
fileListResult.add(items)
}
}
}
}
return fileListResult
}

As suggested by #Chris, you can use the built-in JsonSlurper to parse the JSON and then navigate the parsed results as a Map.
In the following example I show how you can simply print the results to the console. However, using this as a guide, you can see how to extract the data into whatever kind of data structure that suits your purpose.
Assume that you have the JSON in a String variable jsonText. You may be pulling in from a file or an HTTP POST or whatever your app requires. I used the following code to set this value for testing.
def jsonText = """
{
"Application1": {
"environments": {
"it": [
"server1"
],
"ft": [
"server2"
],
"stg": [
"server3"
],
"prd": [
"server4"
]
},
"war-path" : "/opt/tomcat/",
"war-name" : "Application1"
},
"Application2": {
"environments": {
"it": [
"serverA"
],
"ft": [
"serverB"
],
"stg": [
"serverC"
],
"prd": [
"serverD"
]
},
"war-path" : "/var/lib",
"war-name" : "Application2"
}
}
"""
The meat of the solution then follows. Parse the JSON text into a Map, and then iterate over the entries in that Map, performing whatever operations you require. The servers for each environment will already be contained within a List.
import groovy.json.JsonSlurper
def jsonSlurper = new JsonSlurper()
def jsonMap = jsonSlurper.parseText(jsonText)
assert jsonMap instanceof Map
jsonMap.each { appName, appDetails ->
println "Application: $appName"
assert appDetails instanceof Map
appDetails.environments.each { envName, servers ->
assert servers instanceof List
println "\tEnvironment: $envName"
println "\t\t$servers"
}
}
From this code I obtain the following console output.
Application: Application1
Environment: it
[server1]
Environment: ft
[server2]
Environment: stg
[server3]
Environment: prd
[server4]
Application: Application2
Environment: it
[serverA]
Environment: ft
[serverB]
Environment: stg
[serverC]
Environment: prd
[serverD]
EDIT (based on clarification of required output):
import groovy.json.JsonSlurper
def jsonText = "\n{\n \"Application1\": {\n \"environments\": {\n \"it\": [\n \"server1\"\n ],\n \"ft\": [\n \"server2\"\n ],\n \"stg\": [\n \"server3\"\n ],\n \"prd\": [\n \"server4\"\n ]\n },\n \"war-path\" : \"/opt/tomcat/\",\n \"war-name\" : \"Application1\"\n},\n \"Application2\": {\n \"environments\": {\n \"it\": [\n \"serverA\"\n ],\n \"ft\": [\n \"serverB\"\n ],\n \"stg\": [\n \"serverC\"\n ],\n \"prd\": [\n \"serverD\"\n ]\n },\n \"war-path\" : \"/var/lib\",\n \"war-name\" : \"Application2\"\n}\n}\n"
def jsonSlurper = new JsonSlurper()
def jsonMap = jsonSlurper.parseText(jsonText)
def result = [:]
jsonMap.each { appName, appDetails ->
appDetails.environments.each { envName, servers ->
if ( !result.containsKey(envName) ) {
result.put(envName, [])
}
result.get(envName).addAll(servers)
}
}
println result
Results are a Map where the keys are the various environments specified within JSON file, and the values are Lists of servers associated with those environments across all applications. You can access any List individually with something like result.stg, or assign these the different variables later if that's desired/required (def stg = result.stg).
[it:[server1, serverA], ft:[server2, serverB], stg:[server3, serverC], prd:[server4, serverD]]

If you want all environments, you can use the spread operator to take
all environments from the values. Next you have to merge the maps on
the keys. E.g.
def json = """ { "Application1": { "environments": { "it": [ "server1" ], "ft": [ "server2" ], "stg": [ "server3" ], "prd": [ "server4" ] }, }, "Application2": { "environments": { "it": [ "serverA" ], "ft": [ "serverB" ], "stg": [ "serverC" ], "prd": [ "serverD" ] }, } } }"""
def data = new groovy.json.JsonSlurper().parseText(json)
println data.values()*.environments.inject{ a, b ->
b.inject(a.withDefault{[]}) { m, kv ->
// with groovy 2.5+: m.tap{ get(kv.key).addAll(kv.value) }
m[kv.key].addAll(kv.value); m
}
}
// → [it:[server1, serverA], ft:[server2, serverB], stg:[server3, serverC], prd:[server4, serverD]]

Related

Looping through groovy object(List) using each{it} and pass the the elements into a json payload in Jenkins

I have a list containing the name of workspaces in groovy Jenkinsfile. I wrote an each() loop to iterate through the list and use the names in the endpoint below to get the workspace ID from the api response.
def getWorkspaceId() {
def result = []
Listworkspace.each{
def response = httpRequest(
customHeaders: [
[ name: "Authorization", value: "Bearer " + env.BEARER_TOKEN ],
[ name: "Content-Type", value: "application/vnd.api+json" ]
],
url: "https://app.terraform.io/api/v2/organizations/${TF_ORGNAME}/workspaces/$it
)
def data = new JsonSlurper().parseText(response.content)
println ("Workspace Id: " + data.data.id)
result << data.data.id
}
return result
}
After getting the IDs, I want to pass them as part of a json payload.
def buildPayload() {
def workspaceID = new JsonSlurper().parseText(getWorkspaceId())
workspaceID.each{
def payload = """
{
"data": {
"attributes": {
"is-destroy":false,
"message":
},
"type":"runs",
"relationships": {
"workspace": {
"data": [
{"id": "$it", "type": "workspaces"}
]
}
}
}
}
}
"""
return payload
}
Is there a way I can iterate through the list of IDs returned and append each json object for the key "data" after iteration. See the code below
"relationships": {
"workspace": {
"data": [
{"id": "id1", "type": "workspaces"},
{"id": "id2", "type": "workspaces"},
{"id": "id3", "type": "workspaces"}
]
When making the api call, it throws a 400 Bad request error. I tried to print the payload and I found out it attaches the whole list of IDs to the payload.
Any suggestion will be greatly appreciated. Thank you.
def buildPayload() {
def workspaceID = new JsonSlurper().parseText(getWorkspaceId())
workspaceID.each{
def payload = """
{
"data": {
"attributes": {
"is-destroy":false,
"message":
},
"type":"runs",
"relationships": {
"workspace": {
"data": [
[id1, id2, id3]
]
}
}
}
}
}
"""
return payload
}
I'd recommend using the JsonOutput class to make your life easier. In essence, as long as your getWorkspaceId() method is returning a list of ids, you can do something like this:
import groovy.json.JsonOutput
def buildPayload(def ids) {
def payload = [
data: [
attributes: [
"is-destroy": false,
"message" : "",
],
type: "runs",
relationships: [
workspace: [
data: ids.collect {
return [id: it, type: "workspaces"]
}
]
]
]
]
return JsonOutput.toJson(payload)
}
This will take each id in your ids list and build a map where each id is identified by its number and the type: workspaces key pair. This is all then included in the larger payload.

Preparing JSON array of Objects from multiple array list

I am very new to the Groovy scripts and would like to build a JSON output from the below JSON input. Kindly help!
My JSON input looks like this:
{
"id":"1222",
"storageNode": {
"uuid": "22255566336",
"properties": {
"BuinessUnit": [
"Light",
"Fan",
"Watch"
],
"Contact": [
"abc#gmail.com",
"fhh#gmail.com"
],
"Location": [
"Banglore",
"Surat",
"Pune"
]
}
}
}
Expected Output:
[
{
"BuinessUnit": "Light",
"Contact": "abc#gmail.com",
"Location": "Banglore"
},
{
"BuinessUnit": "Fan",
"Contact": "fhh#gmail.com",
"Location": "Surat"
},
{
"BuinessUnit": "Watch",
"Contact": "",
"Location": "Pune"
}
]
Please note that in case any array is not matching the value count that will always be the last one and in that case, a blank value ("") has to be populated. The "BusinessUnit" object can be referred for array size validation.
My code looks like this:
import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
import groovy.json.*;
def Message processData(Message message) {
//Body
def body = message.getBody(String.class);
def jsonSlurper = new JsonSlurper()
def list = jsonSlurper.parseText(body)
String temp
def BU = list.storageNode.properties.get("BusinessUnit")
def builder = new JsonBuilder(
BU.collect {
[
BusinessUnit: it
]
}
)
message.setBody(builder.toPrettyString())
return message
}
It is only returning this:
[
{
"BusinessUnit": "Light"
},
{
"BusinessUnit": "Fan"
},
{
"BusinessUnit": "Watch"
}
]
Now how will I add other parts to it? Please help!
I have come up with the following solution that converts source JSON string to the target JSON string:
import groovy.json.JsonBuilder
import groovy.json.JsonSlurper
def json = '''
{
"id":"1222",
"storageNode": {
"uuid": "22255566336",
"properties": {
"BusinessUnit": [
"Light",
"Fan",
"Watch"
],
"Contact": [
"abc#gmail.com",
"fhh#gmail.com"
],
"Location": [
"Banglore",
"Surat",
"Pune"
]
}
}
}
'''
println convert(json)
String convert(String json) {
def list = new JsonSlurper().parseText(json)
List<String> units = list.storageNode.properties.BusinessUnit
List<String> contacts = list.storageNode.properties.Contact
List<String> locations = list.storageNode.properties.Location
def result = []
units.eachWithIndex { unit, int index ->
result << [
BusinessUnit: unit,
Contact : contacts.size() > index ? contacts[index] : '',
Location : locations.size() > index ? locations[index] : '',
]
}
return new JsonBuilder(result).toPrettyString()
}
I've omitted the logic of getting string from the message and packaging transformed JSON into message.
I hope it will help you to move forward. Please let me know if you need further assistance here.
You can use the built-in Groovy facilities, like transpose():
import groovy.json.*
def json = new JsonSlurper().parseText '''{ "id":"1222", "storageNode": { "uuid": "22255566336", "properties": {
"BuinessUnit": [ "Light", "Fan", "Watch" ],
"Contact": [ "abc#gmail.com", "fhh#gmail.com" ],
"Location": [ "Banglore", "Surat", "Pune" ] } } }'''
def names = json.storageNode.properties*.key
def values = json.storageNode.properties*.value
int maxSize = values*.size().max()
// pad lists with trainiling spaces
values.each{ v -> ( maxSize - v.size() ).times{ v << '' } }
def result = values.transpose().collect{ tuple -> [ names, tuple ].transpose().collectEntries{ it } }
assert result.toString() == '[[BuinessUnit:Light, Contact:abc#gmail.com, Location:Banglore], [BuinessUnit:Fan, Contact:fhh#gmail.com, Location:Surat], [BuinessUnit:Watch, Contact:, Location:Pune]]'
This piece of code can process everything under storageNode.properties.

Modifying JSON in Groovy (or JOLT)

I've a simple JSON look like:
{
"account_login" : "google#gmail.com",
"view_id" : 1868715,
"join_id" : "utm_campaign=toyota&utm_content=multiformat_sites&utm_medium=cpc&utm_source=facebook",
"start_date" : "2020-02-03",
"end_date" : "2020-08-30"
}
With following Groovy script (from this answer):
def content = """
{
"account_login" : "google#gmail.com",
"view_id" : 1868715,
"join_id" : "utm_campaign=toyota&utm_content=multiformat_sites&utm_medium=cpc&utm_source=facebook",
"start_date" : "2020-02-03",
"end_date" : "2020-08-30"
}
"""
def slurped = new JsonSlurper().parseText(content)
def builder = new JsonBuilder(slurped)
builder.content.join_id = builder.content.join_id.split("\\s*&\\s*") //# to array
.collectEntries{
//# convert each item to map entry
String[] utmMarks = it.trim().split("\\s*=\\s*")
utmMarks[0] = [
"utm_medium" : "ga:medium",
"utm_campaign" : "ga:campaign",
"utm_source" : "ga:source",
"utm_content" : "ga:adContent",
"utm_term" : "ga:keyword",
].get( utmMarks[0] )
utmMarks
}
.findAll{
k,v-> k && v!=null //# filter out empty/null keys
}
//builder.content.filters = ...
println(builder.toPrettyString())
I'll get:
{
"account_login": "google#gmail.com",
"view_id": 1868715,
"join_id": {
"ga:campaign": "toyota",
"ga:adContent": "multiformat_sites",
"ga:medium": "cpc",
"ga:source": "facebook"
},
"start_date": "2020-02-03",
"end_date": "2020-08-30"
}
I want to update this script (or write new) and add new property: array filters to modified json above. Expected output:
{
"account_login":"google#gmail.com",
"view_id":1868715,
"join_id":{
"ga:campaign":"toyota",
"ga:adContent":"multiformat_sites",
"ga:medium":"cpc",
"ga:source":"facebook"
},
"start_date":"2020-02-03",
"end_date":"2020-08-30",
"converted_utm_marks":"ga:campaign=toyota&ga:adContent=multiformat_sites&ga:medium=cpc&ga:source=facebook",
"filters":[
{
"dimensionName":"ga:medium",
"operator":"EXACT",
"expressions":[
"cpc"
]
},
{
"dimensionName":"ga:adContent",
"operator":"EXACT",
"expressions":[
"multiformat_sites"
]
},
{
"dimensionName":"ga:campaign",
"operator":"EXACT",
"expressions":[
"toyota"
]
},
{
"dimensionName":"ga:source",
"operator":"EXACT",
"expressions":[
"facebook"
]
}
]
}
But the problem is that the set of filters for each JSON will be different. This set depends directly on the join_id set. If JSON join_id will contain:
"join_id": {
"ga:campaign": "toyota",
"ga:keyword": "car"
}
filters array should be:
[
{
"dimensionName":"ga:campaign",
"operator":"EXACT",
"expressions":[
"toyota"
]
},
{
"dimensionName":"ga:keyword",
"operator":"EXACT",
"expressions":[
"car"
]
}
]
operator is always equals EXACT. Property dimensionName - is a join_id.propety name. Expressions is a join_id.property value. So, property filters based on join_id and I need to loop through join_id property and build filters array with described structure. How to achieve expected output? JOLT configuration appreciated also.
I can't even simple iterate through join_id map:
slurped.join_id.each { println "Key: $it.key = Value: $it.value" }
I got the error:
/home/jdoodle.groovy: 24: illegal colon after argument expression;
solution: a complex label expression before a colon must be parenthesized # line 24, column 28.
.collect { [it.ga:campaign] }
UPDATE
I found out how to build this array:
def array =
[
filters: slurped.join_id.collect {key, value ->
[
dimensionName: key,
operator: "EXACT",
expressions: [
value
]
]
}
]
Seems like i got it:
def slurped = new JsonSlurper().parseText(content)
def builder = new JsonBuilder(slurped)
builder.content.filters = builder.content.join_id.collect {key, value ->
[
dimensionName: key,
operator: "EXACT",
expressions: [
value
]
]
}
Are there any better solutions?
def slurped = new JsonSlurper().parseText(content)
def builder = new JsonBuilder(slurped)
builder.content.filters = builder.content.join_id.collect {key, value ->
[
dimensionName: key,
operator: "EXACT",
expressions: [
value
]
]
}

Can I combine Json.Builder with JsonOutput in groovy? CSV to JSON

I have to parse csv file (with some modifications) to json in groovy.When I'm trying to execute this code i have a problem with spliting some values.
Content of csv file:
TestKey;Finished;user;status
RWS.PT.001;2020-07-20T23:01:21+02:00;admin;PASS;
RWS.PT.002;2020-07-20T23:02:21+02:00;admin;PASS;
my code in groovy:
import groovy.json.*
def builder = new groovy.json.JsonBuilder()
def root = builder {
testExecutionKey 'DEMO-303'
info (
user: 'admin')
tests 'rows':'ghgh','uuuuu'
}
print JsonOutput.prettyPrint(JsonOutput.toJson(root))
def csvfile = new File('C:/temp/raportTest.csv').readLines()
def keys = csvfile[0].split(';')
def rows = csvfile[1..-1].collect { line ->
def i = 0, vals = line.split(';')
keys.inject([:]) { map, key -> map << ["$key": vals[i++]] }
}
print JsonOutput.prettyPrint(JsonOutput.toJson(rows))
my target file should looks like this:
{
"testExecutionKey": "DEMO-303",
"info" : {
"user" : "admin"
},
"tests" : [
{
"testKey" : "RWS.PT.001",
"finished" : "2020-07-20T23:01:21+02:00",
"status" : "PASS"
},
{
"testKey" : "RWS.PT.002",
"finished" : "2020-07-20T23:01:21+02:00",
"status" : "PASS"
}
]
}
Now I have:
{
"testExecutionKey": "DEMO-303",
"info": {
"user": "admin"
},
"tests": [
{
"rows": "ghgh"
},
"uuuuu"
]
}[
{
"TestKey": "RWS.PT.001",
"Finished": "2020-07-20T23:01:21+02:00",
"user": "admin",
"status": "PASS"
},
{
"TestKey": "RWS.PT.002",
"Finished": "2020-07-20T23:02:21+02:00",
"user": "admin",
"status": "PASS"
}
]
How can I input code from def rows (from JsonOutput) into JsonBuilder (instead of "rows": "ghgh").
Please help!
You over-engineered it a bit. You actually don't have to combine anything, just use the JsonBuilder and some Groovy magic:
import groovy.json.*
def kindaFile = '''
TestKey;Finished;user;status
RWS.PT.001;2020-07-20T23:01:21+02:00;admin;PASS;
RWS.PT.002;2020-07-20T23:02:21+02:00;admin;PASS;
'''.trim()
def keys
def testList = []
//parse CSV
kindaFile.splitEachLine( /;/ ){ parts ->
if( !keys )
keys = parts
else{
def test = [:]
parts.eachWithIndex{ val, ix -> test[ keys[ ix ] ] = val }
testList << test
}
}
def builder = new JsonBuilder()
def root = builder {
testExecutionKey 'DEMO-303'
info user: 'admin'
tests testList
}
println JsonOutput.prettyPrint(JsonOutput.toJson(root))
prints
{
"testExecutionKey": "DEMO-303",
"info": {
"user": "admin"
},
"tests": [
{
"TestKey": "RWS.PT.001",
"Finished": "2020-07-20T23:01:21+02:00",
"user": "admin",
"status": "PASS"
},
{
"TestKey": "RWS.PT.002",
"Finished": "2020-07-20T23:02:21+02:00",
"user": "admin",
"status": "PASS"
}
]
}

Groovy: How to parse the json specific key's value into list/array

I am new to groovy and trying
1) from the output of prettyPrint(toJson()), I am trying to get a list of values from a specific key inside an json array using groovy. Using the below JSON output from prettyPrint example below, I am trying to create a list which consists only the values of the name key.
My Code:
def string1 = jiraGetIssueTransitions(idOrKey: jira_id)
echo prettyPrint(toJson(string1.data))
def pretty = prettyPrint(toJson(string1.data))
def valid_strings = readJSON text: "${pretty}"
echo "valid_strings.name : ${valid_strings.name}"
Output of prettyPrint(toJson(string1.data))is below JSON:
{
"expand": "places",
"places": [
{
"id": 1,
"name": "Bulbasaur",
"type": {
"grass",
"poison"
}
},
{
"id": 2,
"name": "Ivysaur",
"type": {
"grass",
"poison"
}
}
}
Expected result
valid_strings.name : ["Bulbasaur", "Ivysaur"]
Current output
valid_strings.name : null
The pretty printed JSON content is invalid.
If the JSON is valid, then names can be accessed as follows:
import groovy.json.JsonSlurper
def text = """
{
"expand": "places",
"places": [{
"id": 1,
"name": "Bulbasaur",
"type": [
"grass",
"poison"
]
},
{
"id": 2,
"name": "Ivysaur",
"type": [
"grass",
"poison"
]
}
]
}
"""
def json = new JsonSlurper().parseText(text)
println(json.places*.name)
Basically, use spray the attribute lookup (i.e., *.name) on the appropriate object (i.e., json.places).
I've used something similar to print out elements within the response in ReadyAPI
import groovy.json.*
import groovy.util.*
def json='[
{ "message" : "Success",
"bookings" : [
{ "bookingId" : 60002172,
"bookingDate" : "1900-01-01T00:00:00" },
{ "bookingId" : 59935582,
"bookingDate" : "1900-01-01" },
{ "bookingId" : 53184048,
"bookingDate" : "2019-01-15",
"testId" : "12803798123",
"overallScore" : "PASS" },
{ "bookingId" : 53183765,
"bookingDate" : "2019-01-15T13:45:00" },
{ "bookingId" : 52783312,
"bookingDate" : "1900-01-01" }
]
}
]
def response = context.expand( json )
def parsedjson = new groovy.json.JsonSlurper().parseText(response)
log.info parsedjson
log.info " Count of records returned: " + parsedjson.size()
log.info " List of bookingIDs in this response: " + parsedjson.bookings*.bookingId