I think I've read every Groovy parsing question on here but I can't seem to find my exact scenario so I'm reaching out for help - please be kind, I'm new to Groovy and I've really bitten off more than I can chew in this latest endeavor.
So I have this XML Response:
<?xml version="1.0" encoding="UTF-8"?>
<worklogs date_from="2020-04-19 00:00:00" date_to="2020-04-25 23:59:59" number_of_worklogs="60" format="xml" diffOnly="false" errorsOnly="false" validOnly="false" addDeletedWorklogs="true" addBillingInfo="false" addIssueSummary="true" addIssueDescription="false" duration_ms="145" headerOnly="false" userName="smm288" addIssueDetails="false" addParentIssue="false" addUserDetails="true" addWorklogDetails="false" billingKey="" issueKey="" projectKey="" addApprovalStatus="true" >
<worklog>
<worklog_id></worklog_id>
<jira_worklog_id></jira_worklog_id>
<issue_id></issue_id>
<issue_key></issue_key>
<hours></hours>
<work_date></work_date>
<username></username>
<staff_id />
<billing_key></billing_key>
<billing_attributes></billing_attributes>
<activity_id></activity_id>
<activity_name></activity_name>
<work_description></work_description>
<parent_key></parent_key>
<reporter></reporter>
<external_id />
<external_tstamp />
<external_hours></external_hours>
<external_result />
<customField_11218></customField_11218>
<customField_12703></customField_12703>
<customField_12707></customField_12707>
<hash_value></hash_value>
<issue_summary></issue_summary>
<user_details>
<full_name></full_name>
<email></email>
<user-prop key="auto_approve_timesheet"></user-prop>
<user-prop key="cris_id"></user-prop>
<user-prop key="iqn_gl_string"></user-prop>
<user-prop key="is_contractor"></user-prop>
<user-prop key="is_employee"></user-prop>
<user-prop key="it_leadership"></user-prop>
<user-prop key="primary_role"></user-prop>
<user-prop key="resource_manager"></user-prop>
<user-prop key="team"></user-prop>
</user_details>
<approval_status></approval_status>
<timesheet_approval>
<status></status>
<status_date></status_date>
<reviewer></reviewer>
<actor></actor>
<comment></comment>
</timesheet_approval>
</worklog>
....
....
</worklogs>
And I'm retrieving this XML Response from an API call so the response is held within an object. NOTE: The sample XML above is from Postman.
What I'm trying to do is the following:
1. Only retrieve certain values from this response from all the nodes.
2. Write the values collected to a .json file.
I've created a map but now I'm kind of stuck on how to parse through it and create a .json file out of the fields I want.
This is what I have thus far
#Grab('org.codehaus.groovy.modules.http-builder:http-builder:0.7.1')
#Grab('oauth.signpost:signpost-core:1.2.1.2')
#Grab('oauth.signpost:signpost-commonshttp4:1.2.1.2')
import groovyx.net.http.RESTClient
import groovyx.net.http.Method
import static groovyx.net.http.ContentType.*
import groovyx.net.http.HttpResponseException
import groovy.json.JsonBuilder
import groovy.json.JsonOutput
import groovy.json.*
// User Credentials
def jiraAuth = ""
// JIRA Endpoints
def jiraUrl = "" //Dev
def jiraUrl = "" //Production
// Tempo API Tokens
//def tempoApiToken = "" //Dev
//def tempoApiToken = "" //Production
// Define Weekly Date Range
def today = new Date()
def lastPeriodStart = today - 8
def lastPeriodEnd = today - 2
def dateFrom = lastPeriodStart.format("yyyy-MM-dd")
def dateTo = lastPeriodEnd.format("yyyy-MM-dd")
def jiraClient = new RESTClient(jiraUrl)
jiraClient.ignoreSSLIssues()
def headers = [
"Authorization" : "Basic " + jiraAuth,
"X-Atlassian-token": "no-check",
"Content-Type" : "application/json"
]
def response = jiraClient.get(
path: "",
query: [
tempoApiToken: "${tempoApiToken}",
format: "xml",
dateFrom: "${dateFrom}",
dateTo: "${dateTo}",
addUserDetails: "true",
addApprovalStatus: "true",
addIssueSummary: "true"
],
headers: headers
) { response, worklogs ->
println "Processing..."
// Start building the Output - Creates a Worklog Map
worklogs.worklog.each { worklognodes ->
def workLog = convertToMap(worklognodes)
// Print out the Map
println (workLog)
}
}
// Helper Method
def convertToMap(nodes) {
nodes.children().collectEntries {
if (it.name() == 'user-prop') {
[it['#key'], it.childNodes() ? convertToMap(it) : it.text()]
} else {
[it.name(), it.childNodes() ? convertToMap(it) : it.text()]
}
}
}
I'm only interested in parsing out the following fields from each node:
<worklogs>
<worklog>
<hours>
<work_date>
<billing_key>
<customField_11218>
<issue_summary>
<user_details>
<full_name>
<user-prop key="auto_approve_timesheet">
<user-prop key="it_leadership">
<user-prop key="resource_manager">
<user-prop key="team">
<user-prop key="cris_id">
<user-prop key="iqn_id">
<approval_status>
</worklog>
...
</worklogs>
I've tried the following:
1. Converting the workLog to a json string (JsonOutput.toJson) and then converting the json string to prettyPrint (JsonOutput.prettyPrint) - but this just returns a collection of .json responses which I can't do anything with (thought process was, this is as good as I can get and I'll just use a .json to .csv converter and get rid of what I don't want) - which is not the solution I ultimately want.
2. Printing the map workLog just returns little collections which I can't do anything with either
3. Create a new file using File and creating a .json file of workLog but again, it doesn't translate well.
The results of the println for workLog is here (just so everyone can see that the response is being held and the map matches the XML response).
[worklog_id: , jira_worklog_id: , issue_id: , issue_key: , hours: , work_date: , username: , staff_id: , billing_key: , billing_attributes: , activity_id: , activity_name: , work_description: , parent_key: , reporter: , external_id:, external_tstamp:, external_hours: , external_result:, customField_11218: , hash_value: , issue_summary: , user_details:[full_name: , email: , auto_approve_timesheet: , cris_id: , iqn_gl_approver: , iqn_gl_string: , iqn_id: , is_contractor: , is_employee: , it_leadership: , primary_role: , resource_manager: , team: ], approval_status: , timesheet_approval:[status: ]]
I would so appreciate it if anyone could offer some insights on how to move forward or even documentation that has good examples of what I'm trying to achieve (Apache's documentation is sorely lacking in examples, in my opinion).
It's not all of the way there. But, I was able to get a JSON file created with the XML and Map. From there I can just use the .json file to create a .csv and then get rid of the columns I don't want.
// Define Weekly Date Range
def today = new Date()
def lastPeriodStart = today - 8
def lastPeriodEnd = today - 2
def dateFrom = lastPeriodStart.format("yyyy-MM-dd")
def dateTo = lastPeriodEnd.format("yyyy-MM-dd")
def jiraClient = new RESTClient(jiraUrl)
jiraClient.ignoreSSLIssues()
// Creates and Begins the File
File file = new File("${dateFrom}_RPT05.json")
file.write("")
file.append("[\n")
// Defines the File
def arrplace = 0
def arrsize = 0
def headers = [
"Authorization" : "Basic " + jiraAuth,
"X-Atlassian-token": "no-check",
"Content-Type" : "application/json"
]
def response = jiraClient.get(
path: "/plugins/servlet/tempo-getWorklog/",
query: [
tempoApiToken: "${tempoApiToken}",
format: "xml",
dateFrom: "${dateFrom}",
dateTo: "${dateTo}",
addUserDetails: "true",
addApprovalStatus: "true",
addIssueSummary: "true"
],
headers: headers
) { response, worklogs ->
println "Processing..."
// Gets Size of Array
worklogs.worklog.each { worklognodes ->
arrsize = arrsize+1 }
// Start Building the Output - Creates a Worklog Map
worklogs.worklog.each { worklognodes ->
worklognodes = convertToMap(worklognodes)
// Convert Map to a JSON String
def json_str = JsonOutput.toJson(worklognodes)
// Adds Row to File
file.append(json_str)
arrplace = arrplace+1
if(arrplace<arrsize)
{file.append(",")}
file.append("\n")
print "."
}
}
file.append("]")
// Helper Method
def convertToMap(nodes) {
nodes.children().collectEntries {
if (it.name() == 'user-prop') {
[it['#key'], it.childNodes() ? convertToMap(it) : it.text()]
} else {
[it.name(), it.childNodes() ? convertToMap(it) : it.text()]
}
}
}
The output is a collection/array of worklogs.
I want to create an endpoint that receives JSON data and should parse it as an array of strings.
POST /
{
"keys": ["foo", "bar"]
}
I'm running into problems with the type system. This is what I tried (.as(Array(String))) but it does not compile:
require "kemal"
def print_keys(keys : Array(String))
puts "Got keys: #{keys}"
end
post "/" do |env|
keys = env.params.json["keys"].as(Array(String)) # <-- ERROR
print_keys(keys)
end
Kemal.run
The error message is:
8 | keys = env.params.json["keys"].as(Array(String)) # <-- ERROR
^
Error: can't cast (Array(JSON::Any) | Bool | Float64 | Hash(String, JSON::Any) | Int64 | String | Nil) to Array(String)
If I change the code to parse not Array(String) but instead String, it compiles without problems. Why does it make a difference in the .as method that the type is Array(String) instead of String?
How can the code be changed to parse arrays of strings?
I found an example in the documentation, which uses JSON.mapping. In my concrete example, it could be written as follows:
require "kemal"
def print_keys(keys : Array(String))
puts "Got keys: #{keys}"
end
class KeyMappings
JSON.mapping({
keys: Array(String)
})
end
post "/" do |env|
json = KeyMappings.from_json env.request.body.not_nil!
print_keys(json.keys)
end
Kemal.run
I have a problem with a method in my Jenkinsfile when i tried to convert xml to json. That is the method, and the pipeline.
I tried to pass the method directly to the echo, but it gives me an error and the pipeline fails
Sorry but i don't know that details i could give about the error, because i start to learning and this is the first time that i see this code.
ERROR: org.xml.sax.SAXParseException; lineNumber: 2; columnNumber: 1; Content is not allowed in prolog.
I edit my question and i add a bat in stage OWASP dependencies testing. This bat create automatically the xml, i put in a validator xml and this did not errors. So i don't know if the problem is with the code of Jenkinsfile or xml, because the error is the same. I put part of xml code because it's very long, but
the error is still in the second line.
XML Code:
<?xml version="1.0" encoding="UTF-8"?>
<analysis xmlns="https://jeremylong.github.io/DependencyCheck/dependency-check.2.2.xsd">
<scanInfo>
<engineVersion>5.2.2</engineVersion>
<dataSource>
<name>NVD CVE Checked</name>
<timestamp>2019-11-25T09:01:51</timestamp>
</dataSource>
<datasource>...</datasource>
</scanInfo>
....................
</analysis>
import groovy.json.*;
def getDependencyResumeFromXML(pathReport){
def xml = bat(script:'type ' + pathReport, returnStdout:true);
def x = new XmlParser().parseText(xml);
def nDep = x.dependencies.dependency.size();
def dependencies = [:];
for(def i=0;i<nDep;i++){
dependencies[i] = [fileName: x.dependencies.dependency[i].fileName.text(),description:x.dependencies.dependency[i].description.text(),vulnerabilities:[:]];
def nVul = x.dependencies.dependency[i].vulnerabilities.vulnerability.size();
for(def j=0;j<nVul;j++){
dependencies[i].vulnerabilities[j] = [
name:x.dependencies.dependency[i].vulnerabilities.vulnerability[j].name.text(), cvssScore:x.dependencies.dependency[i].vulnerabilities.vulnerability[j].cvssScore.text(),
severity:x.dependencies.dependency[i].vulnerabilities.vulnerability[j].severity.text(),
cwe:x.dependencies.dependency[i].vulnerabilities.vulnerability[j].cwe.text(),
description:x.dependencies.dependency[i].vulnerabilities.vulnerability[j].description.text(),
];
}
}
return dependencies;
}
pipeline{
.......
stages{
stage('OWASP dependencies testing'){
steps{
script{
bat 'mvn org.owasp:dependency-check-maven:check';
def pathReport = 'C:\\tmp\\workspace\\umbrella-pipeline-prueba\\target\\dependency-check\\dependency-check-report.xml';
def xml = bat(script:'type ' + pathReport, returnStdout:true);
echo '------------------ 1';
echo xml;
echo '------------------ 2';
echo '--------------------------------'
def dependencias = getDependencyResumeFromXML(pathReport);
echo '------------- 3';
echo dependencias;
echo '------------- 4';
}
}
}
}
}
Current logback.xml
<appender name="FILEOUT3" class="ch.qos.logback.core.FileAppender">
<file>D:/${byDay}.log</file>
<append>true</append>
<encoder>
<Pattern>%d{HH:mm:ss} %-5level %msg%replace(%xException){"\n", ">> "}%nopex%n</Pattern>
</encoder>
</appender>
Current result :
play.api.Configuration$$anon$1: Configuration error[Cannot connect to database [default]]
>> at play.api.Configuration$.play$api$Configuration$$configError(Configuration.scala:92) ~[play_2.10-2.2.0.jar:2.2.0]
>> at play.api.Configuration.reportError(Configuration.scala:570) ~[play_2.10-2.2.0.jar:2.2.0]
>> at play.api.db.BoneCPPlugin$$anonfun$onStart$1.apply(DB.scala:252) ~[play-jdbc_2.10-2.2.0.jar:2.2.0]
I want result :
[12:43:16.454] play.api.Configuration$$anon$1: Configuration error[Cannot connect to database [default]]
[12:43:16.454] at play.api.Configuration$.play$api$Configuration$$configError(Configuration.scala:92) ~[play_2.10-2.2.0.jar:2.2.0]
[12:43:16.454] at play.api.Configuration.reportError(Configuration.scala:570) ~[play_2.10-2.2.0.jar:2.2.0]
[12:43:16.454] at play.api.db.BoneCPPlugin$$anonfun$onStart$1.apply(DB.scala:252) ~[play-jdbc_2.10-2.2.0.jar:2.2.0]
:
:
more 40 lines
:
[12:43:16.454] at play.api.db.BoneCPPlugin$$anonfun$onStart$1.apply(DB.scala:252) ~[play-jdbc_2.10-2.2.0.jar:2.2.0]
IMPORTANT!
Same Time Print
I want know.
How to change logback.xml?
You cannot do this by changing only logback.xml, because ReplacingCompositeConverter, which is called by %replace, uses only static strings as replacements (no dates or any other conversion words of PatternLayout).
To reach your goal you should create custom converter (if you'll use mine, notice that it should be placed in ch.qos.logback.core.pattern package)
package ch.qos.logback.core.pattern;
import ch.qos.logback.classic.PatternLayout;
import ch.qos.logback.core.pattern.parser.Node;
import ch.qos.logback.core.pattern.parser.Parser;
import ch.qos.logback.core.spi.ScanException;
public class ReplacingAndParsingCompositeConverter<E> extends ReplacingCompositeConverter<E> {
#Override
protected String transform(E event, String in) {
if (!started) {
return in;
}
String parsedReplacement;
try {
Parser<E> p = new Parser<E>(replacement);
p.setContext(getContext());
Node t = p.parse();
Converter<E> c = p.compile(t, PatternLayout.defaultConverterMap);
ConverterUtil.setContextForConverters(getContext(), c);
ConverterUtil.startConverters(c);
StringBuilder buf = new StringBuilder();
while (c != null) {
c.write(buf, event);
c = c.getNext();
}
parsedReplacement = buf.toString();
} catch (ScanException e) {
e.printStackTrace();
parsedReplacement = replacement;
}
return pattern.matcher(in).replaceAll(parsedReplacement);
}
}
Then you should declare this converter in logback.xml with <conversionRule/> and use it instead of old %replace.
<configuration ...>
<conversionRule conversionWord="replaceAndParse" converterClass="ch.qos.logback.core.pattern.ReplacingAndParsingCompositeConverter" />
<appender name="FILEOUT3" class="ch.qos.logback.core.FileAppender">
<file>D:/${byDay}.log</file>
<append>true</append>
<encoder>
<Pattern>[%d{HH:mm:ss.SSS}] %-5level %msg%replaceAndParse(%xException){"(\r?\n)", "$1[%d{HH:mm:ss.SSS}]"}%nopex%n</Pattern>
</encoder>
</appender>
....
</configuration>
I created the followind application.conf:
akka {
actor {
prio-dispatcher {
type = "Dispatcher"
mailbox-type = "my.package.PrioritizedMailbox"
}
}
}
when dumping configuration with
actorSystem = ActorSystem.create()
println(actorSystem.settings)
I'm getting the output:
# application.conf: 5
"prio-dispatcher" : {
# application.conf: 7
"mailbox-type" : "my.package.PrioritizedMailbox",
# application.conf: 6
"type" : "Dispatcher"
},
and later on
[WARN] [08/30/2012 22:44:54.362] [default-akka.actor.default-dispatcher-3] [Dispatchers] Dispatcher [prio-dispatcher] not configured, using default-dispatcher
What am I missing here?
UPD Found the solution here, had to use the name "akka.actor.prio-dispatcher"
The configuration above dictates that name of mailbox is akka.actor.prio-dispatcher
Description of the problem: http://groups.google.com/group/akka-user/browse_thread/thread/678f2ae1c068e0fa