Save result to a new csv file instead of println Groovy - json

How can I save result of groovy script to a new file? C:/temp/all1.csv. I want to parse json file to csv, script is working fine but I don't know how can I save result in a new file. Please help.
import groovy.json.*
import java.io.File
def json ='''
{
"expand": "schema,names",
"startAt": 0,
"maxResults": 50,
"total": 21,
"issues": [
{
"expand": "operations,versionedRepresentations",
"id": "217580",
"self": "issue/217580",
"key": "ART-4070",
"fields": {"summary": "#[ART] Pre.3 Verification \\"S\\""}
},
{
"expand": "operations,versionedRepresentations",
"id": "217579",
"self": "issue/217579",
"key": "ART-4069",
"fields": {"summary": "Verification \\"C\\""}
},
{
"expand": "operations,versionedRepresentations",
"id": "217577",
"self": "issue/217577",
"key": "ART-4068",
"fields": {"summary": "#[ART] Enum type"}
}
]
}
'''
File csvFile = new File( 'C:/temp/all1.csv')
def jsonSlurper = new JsonSlurper()
def config = [ // header -> extractor
"key": { it.key },
"summary": { it.fields.summary }
]
def encode(e) { // help with nulls; quote the separator
(e ?: "").replaceAll(";", "\\;")
}
def csvLine(items) { // write items as "CSV"
println(items.collect{ encode it }.join(";"))
}
def obj = new JsonSlurper().parseText(json)
csvLine(config.keySet())
obj.issues.each{ issue ->
csvLine(config.values().collect{ f -> f issue })
}
result:
key;summary
ART-4070;#[ART] Pre.3 Verification "S"
ART-4069;Verification "C"
ART-4068;#[ART] Enum type

To go with the current code, you could use csvFile.append(...) instead of println inside your
csvLine function and depending on your amount of real data, this might
be a good compromise between performance and resource.
Or you can write the whole CSV at once. E.g.
// prepare whole table
def data = [config.keySet()]
data.addAll(
obj.issues.collect{ issue ->
config.values().collect{ f -> f issue }
}
)
// write table as csv
def csvFile = "/tmp/out.csv" as File
csvFile.text = data.collect{
it.collect{ encode it }.join(";")9
}.join("\n")

Related

Preparing JSON array of Objects from multiple array list

I am very new to the Groovy scripts and would like to build a JSON output from the below JSON input. Kindly help!
My JSON input looks like this:
{
"id":"1222",
"storageNode": {
"uuid": "22255566336",
"properties": {
"BuinessUnit": [
"Light",
"Fan",
"Watch"
],
"Contact": [
"abc#gmail.com",
"fhh#gmail.com"
],
"Location": [
"Banglore",
"Surat",
"Pune"
]
}
}
}
Expected Output:
[
{
"BuinessUnit": "Light",
"Contact": "abc#gmail.com",
"Location": "Banglore"
},
{
"BuinessUnit": "Fan",
"Contact": "fhh#gmail.com",
"Location": "Surat"
},
{
"BuinessUnit": "Watch",
"Contact": "",
"Location": "Pune"
}
]
Please note that in case any array is not matching the value count that will always be the last one and in that case, a blank value ("") has to be populated. The "BusinessUnit" object can be referred for array size validation.
My code looks like this:
import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
import groovy.json.*;
def Message processData(Message message) {
//Body
def body = message.getBody(String.class);
def jsonSlurper = new JsonSlurper()
def list = jsonSlurper.parseText(body)
String temp
def BU = list.storageNode.properties.get("BusinessUnit")
def builder = new JsonBuilder(
BU.collect {
[
BusinessUnit: it
]
}
)
message.setBody(builder.toPrettyString())
return message
}
It is only returning this:
[
{
"BusinessUnit": "Light"
},
{
"BusinessUnit": "Fan"
},
{
"BusinessUnit": "Watch"
}
]
Now how will I add other parts to it? Please help!
I have come up with the following solution that converts source JSON string to the target JSON string:
import groovy.json.JsonBuilder
import groovy.json.JsonSlurper
def json = '''
{
"id":"1222",
"storageNode": {
"uuid": "22255566336",
"properties": {
"BusinessUnit": [
"Light",
"Fan",
"Watch"
],
"Contact": [
"abc#gmail.com",
"fhh#gmail.com"
],
"Location": [
"Banglore",
"Surat",
"Pune"
]
}
}
}
'''
println convert(json)
String convert(String json) {
def list = new JsonSlurper().parseText(json)
List<String> units = list.storageNode.properties.BusinessUnit
List<String> contacts = list.storageNode.properties.Contact
List<String> locations = list.storageNode.properties.Location
def result = []
units.eachWithIndex { unit, int index ->
result << [
BusinessUnit: unit,
Contact : contacts.size() > index ? contacts[index] : '',
Location : locations.size() > index ? locations[index] : '',
]
}
return new JsonBuilder(result).toPrettyString()
}
I've omitted the logic of getting string from the message and packaging transformed JSON into message.
I hope it will help you to move forward. Please let me know if you need further assistance here.
You can use the built-in Groovy facilities, like transpose():
import groovy.json.*
def json = new JsonSlurper().parseText '''{ "id":"1222", "storageNode": { "uuid": "22255566336", "properties": {
"BuinessUnit": [ "Light", "Fan", "Watch" ],
"Contact": [ "abc#gmail.com", "fhh#gmail.com" ],
"Location": [ "Banglore", "Surat", "Pune" ] } } }'''
def names = json.storageNode.properties*.key
def values = json.storageNode.properties*.value
int maxSize = values*.size().max()
// pad lists with trainiling spaces
values.each{ v -> ( maxSize - v.size() ).times{ v << '' } }
def result = values.transpose().collect{ tuple -> [ names, tuple ].transpose().collectEntries{ it } }
assert result.toString() == '[[BuinessUnit:Light, Contact:abc#gmail.com, Location:Banglore], [BuinessUnit:Fan, Contact:fhh#gmail.com, Location:Surat], [BuinessUnit:Watch, Contact:, Location:Pune]]'
This piece of code can process everything under storageNode.properties.

JSONBuilder in Groovy adds wrong quotation marks

I'm trying to read a json file, edit some parts of it and then parse it back to a json file. The goal is to change the value of a confluence page. I'm using the groovy code in a Jenkins pipeline. Here it is:
def changeValue(){
def json_map = readJSON file: '/tmp/updater.json'
def body_content = '{"storage":{"value":'
body_content += '"<h1>test</h1>"'
body_content += ',"representation":"storage"}}'
json_map.body = body_content
json_as_string = new JsonBuilder(json_map).toPrettyString().replaceAll("\\\\", "") // It also adds unneccesary escapes
print json_as_string
}
This is the contents of the updater.json:
{
"id":"redacted",
"type":"page",
"title":"redacted",
"space":{"key":"redacted"},
"body":{"storage":{"value":"<h1>wrong</h1>","representation":"storage"}},
"version":{
"number":6
}
}
That is what I get:
{
"id": "redacted",
"type": "page",
"title": "redacted",
"space": {
"key": "redacted"
},
"body": "{"storage":{"value":"<h1>test</h1>","representation":"storage"}}",
"version": {
"number": 6
}
}
As you can see, it added quotation marks around the block of the body. How can I get rid of them?
The result is as expected, you update the body with a new String.
If you want to update only the value use this based on this answer
import groovy.json.JsonBuilder
import groovy.json.JsonSlurper
def jsn = """
{
"id":"redacted",
"type":"page",
"title":"redacted",
"space":{"key":"redacted"},
"body":{"storage":{"value":"<h1>wrong</h1>","representation":"storage"}},
"version":{
"number":6
}
}"""
def slp= new JsonSlurper().parseText(jsn)
bld.content.body.storage.value = '<h1>test</h1>'
println(bld.toPrettyString())
result
{
"id": "redacted",
"type": "page",
"title": "redacted",
"space": {
"key": "redacted"
},
"body": {
"storage": {
"value": "<h1>test</h1>",
"representation": "storage"
}
},
"version": {
"number": 6
}
}

Combine multiple JSON files, and parse into CSV

I have about 100 JSON files, all titled with different dates and I need to merge them into one CSV file that has headers "date", "real_name", "text".
There are no dates listed in the JSON itself, and the real_name is nested. I haven't worked with JSON in a while and am a little lost.
The basic structure of the JSON looks more or less like this:
Filename: 2021-01-18.json
[
{
"client_msg_id": "xxxx",
"type": "message",
"text": "THIS IS THE TEXT I WANT TO PULL",
"user": "XXX",
"user_profile": {
"first_name": "XXX",
"real_name": "THIS IS THE NAME I WANT TO PULL",
"display_name": "XXX",
"is_restricted": false,
"is_ultra_restricted": false
},
"blocks": [
{
"type": "rich_text",
"block_id": "yf=A9",
}
]
}
]
So far I have
import glob
read_files = glob.glob("*.json")
output_list = []
all_items = []
for f in read_files:
with open(f, "rb") as infile:
output_list.append(json.load(infile))
data = {}
for obj in output_list[]
data['date'] = f
data['text'] = 'text'
data['real_name'] = 'real_name'
all_items.append(data)
Once you've read the JSON object, just index into the dictionaries for the data. You might need obj[0]['text'], etc., if your JSON data is really in a list in each file, but that seems odd and I'm assuming your data was pasted from output_list after you'd collected the data. So assuming your file content is exactly like below:
{
"client_msg_id": "xxxx",
"type": "message",
"text": "THIS IS THE TEXT I WANT TO PULL",
"user": "XXX",
"user_profile": {
"first_name": "XXX",
"real_name": "THIS IS THE NAME I WANT TO PULL",
"display_name": "XXX",
"is_restricted": false,
"is_ultra_restricted": false
},
"blocks": [
{
"type": "rich_text",
"block_id": "yf=A9",
}
]
}
test.py:
import json
import glob
from pathlib import Path
read_files = glob.glob("*.json")
output_list = []
all_items = []
for f in read_files:
with open(f, "rb") as infile:
output_list.append(json.load(infile))
data = {}
for obj in output_list:
data['date'] = Path(f).stem
data['text'] = obj['text']
data['real_name'] = obj['user_profile']['real_name']
all_items.append(data)
print(all_items)
Output:
[{'date': '2021-01-18', 'text': 'THIS IS THE TEXT I WANT TO PULL', 'real_name': 'THIS IS THE NAME I WANT TO PULL'}]

Parsing JSON to CSV in Groovy

I'm trying to parse json file to csv. Could you help?
example json:
{
"expand": "schema,names",
"startAt": 0,
"maxResults": 50,
"total": 21,
"issues": [
{
"expand": "operations,versionedRepresentations",
"id": "217580",
"self": "issue/217580",
"key": "ART-4070",
"fields": {"summary": "#[ART] Pre.3 Verification \\"S\\""}
},
{
"expand": "operations,versionedRepresentations",
"id": "217579",
"self": "issue/217579",
"key": "ART-4069",
"fields": {"summary": "Verification \\"C\\""}
},
{
"expand": "operations,versionedRepresentations",
"id": "217577",
"self": "issue/217577",
"key": "ART-4068",
"fields": {"summary": "#[ART] Enum type"}
}
]
}
result csv should be like:
key;summary
ART-4070;#[ART] Pre.3 Verification \"S\"
ART-4069;Verification \"C\"
ART-4068;#[ART] Enum type
I've tried such a code:
import groovy.json.*
def jsonSlurper = new JsonSlurper()
def json = '''
{
"expand": "schema,names",
"startAt": 0,
"maxResults": 50,
"total": 21,
"issues": [
{
"expand": "operations,versionedRepresentations",
"id": "217580",
"self": "issue/217580",
"key": "ART-4070",
"fields": {"summary": "#[ART] Pre.3 Verification \\"S\\""}
},
{
"expand": "operations,versionedRepresentations",
"id": "217579",
"self": "issue/217579",
"key": "ART-4069",
"fields": {"summary": "Verification \\"C\\""}
},
{
"expand": "operations,versionedRepresentations",
"id": "217577",
"self": "issue/217577",
"key": "ART-4068",
"fields": {"summary": "#[ART] Enum type"}
}
]
}
'''
def obj = jsonSlurper.parse(json)
def columns = obj.issues*.keySet().flatten().unique()
// remove nulls
def encode = { e -> e == null ? '' : e }
// Print all the column names
println columns.collect { c -> encode( c ) }.join( ';' )
// create all the rows
println obj.issues.collect { row ->
// A row at a time
columns.collect { colName -> encode( row[ colName ] ) }.join( ';' )
}.join( '\n' )
but result is wrong:
expand;id;self;key;fields
operations,versionedRepresentations;217580;issue/217580;ART-4070;[summary:#[ART] Pre.3 Verification "S"]
operations,versionedRepresentations;217579;issue/217579;ART-4069;[summary:Verification "C"]
operations,versionedRepresentations;217577;issue/217577;ART-4068;[summary:#[ART] Enum type]
how can i extract only what i want from json file? I need only two columns:key,summary and values for them.
You want to extract only specific information from your list of issues
and you need different strategies to extract those. So I'd use
a "configuration" to describe the extraction (see the map config
below). Then the code is quite close to your original one (extracted
some common code etc)
import groovy.json.*
def config = [ // header -> extractor
"key": { it.key },
"summary": { it.fields.summary }
]
def encode(e) { // help with nulls; quote the separator
(e ?: "").replaceAll(";", "\\;")
}
def csvLine(items) { // write items as "CSV"
println(items.collect{ encode it }.join(";"))
}
// main
def obj = new JsonSlurper().parse("data.json" as File)
csvLine(config.keySet())
obj.issues.each{ issue ->
csvLine(config.values().collect{ f -> f issue })
}

how to parse CSV to JSON from 2 CSV Files in Groovy

Please help with parse CSV to JSON from 2 CSV Files in groovy
For example :
CSV1:
testKey,status
Name001,PASS
Name002,PASS
Name003,FAIL
CSV2:
Kt,Pd
PT-01,Name001
PT-02,Name002
PT-03,Name003
PT-04,Name004
I want to input in "testlist" data from CSV2.val[1..-1],CSV1.val[1..-1]
Result should be like :
{
"testExecutionKey": "DEMO-303",
"info": {
"user": "admin"
},
"tests": [
{
"TestKey": "PT-01",
"status": "PASS"
},
{
"TestKey": "PT-02",
"status": "PASS"
},
{
"TestKey": "PT-03",
"status": "FAIL"
}
]
code without this modification (from only 1 csv):
import groovy.json.*
def kindaFile = '''
TestKey;Finished;user;status
Name001;PASS;
Name002;PASS;
'''.trim()
def keys
def testList = []
//parse CSV
kindaFile.splitEachLine( /;/ ){ parts ->
if( !keys )
keys = parts
else{
def test = [:]
parts.eachWithIndex{ val, ix -> test[ keys[ ix ] ] = val }
testList << test
}
}
def builder = new JsonBuilder()
def root = builder {
testExecutionKey 'DEMO-303'
info user: 'admin'
tests testList
}
println JsonOutput.prettyPrint(JsonOutput.toJson(root))
Your sample JSON doesn't match the CSV definition. It looks lile you're using fields [1..-1] from CSV 1, as you stated, but fields [0..-2] from CSV 2. As you only have 2 fields in each CSV that's the equivalent of csv1[1] and csv2[0]. The example below uses [0..-2]. Note that if you always have exactly two fields in your input files then the following code could be simplified a little. I've given a more generic solution that can cope with more fields.
Load both CSV files into lists
File csv1 = new File( 'one.csv')
File csv2 = new File( 'two.csv')
def lines1 = csv1.readLines()
def lines2 = csv2.readLines()
assert lines1.size() <= lines2.size()
Note the assert. That's there as I noticed you have 4 tests in CSV2 but only 3 in CSV1. To allow the code to work with your sample data, it iterates through through CSV1 and adds the matching data from CSV2.
Get the field names
fieldSep = /,[ ]*/
def fieldNames1 = lines1[0].split( fieldSep )
def fieldNames2 = lines1[0].split( fieldSep )
Build the testList collection
def testList = []
lines1[1..-1].eachWithIndex { csv1Line, lineNo ->
def mappedLine = [:]
def fieldsCsv1 = csv1Line.split( fieldSep )
fieldsCsv1[1..-1].eachWithIndex { value, fldNo ->
String name = fieldNames1[ fldNo + 1 ]
mappedLine[ name ] = value
}
def fieldsCsv2 = lines2[lineNo + 1].split( fieldSep )
fieldsCsv2[0..-2].eachWithIndex { value, fldNo ->
String name = fieldNames2[ fldNo ]
mappedLine[ name ] = value
}
testList << mappedLine
}
Parsing
You can now parse the list of maps with your existing code. I've made a change to the way the JSON string is displayed though.
def builder = new JsonBuilder()
def root = builder {
testExecutionKey 'DEMO-303'
info user: 'admin'
tests testList
}
println builder.toPrettyString()
JSON Output
Running the above code, using your CSV1 and CSV 2 data, gives the JSON that you desire.
for CSV1:
testKey,status
Name001,PASS
Name002,PASS
Name003,FAIL
and CSV2:
Kt,Pd
PT-01,Name007
PT-02,Name001
PT-03,Name003
PT-05,Name002
PT-06,Name004
PT-07,Name006
result is:
{
"testExecutionKey": "DEMO-303",
"info": {
"user": "admin"
},
"tests": [
{
"status": "PASS",
"testKey": "PT-01"
},
{
"status": "PASS",
"testKey": "PT-02"
},
{
"status": "FAIL",
"testKey": "PT-03"
}
]
}
but I need exactly the same values for testKey (testKey from CSV1=Kt from CSV2)
{
"testExecutionKey": "DEMO-303",
"info": {
"user": "admin"
},
"tests": [
{
"testKey": "PT-02",
"status": "PASS"
},
{
"testKey": "PT-05",
"status": "PASS"
},
{
"testKey": "PT-03",
"status": "FAIL"
}
]
}