Reading json Payload as part of a CSV file in dataweave - json

Question

I have a web service which exports a CSV file which has one of the columns as a JSON payload. After the execution of the web service, I store the values in a local variable for transformation. Every time when I read the values from that column I am missing the values and only "}" is returned. Not sure why this is happening. I want to preserve the JSON payload as is and persist to a file after some processing. Please advise
I am using the code below to get the value of the attribute column and it always returns a "}". The rest of the contents are ignored
CSV Fragment
-------------
id,name,attributes
1,name1,{"Choice Number":"0","Campaign Run Id":"234"}
2,name2,{"Choice Number":"1","Campaign Run Id":"235"}
3,name3,{"Choice Number":"2","Campaign Run Id":"236"}
4,name4,{"Choice Number":"3","Campaign Run Id":"236"}
Code
----
%dw 1.0
%output application/java
---
flowVars.activityData map ((actData) -> {
"playerId": actData.id,
"name": actData.name,
"attributes": actData.attributes
})
I was expecting that the full JSON payload from the attributes column will be returned and that is not the case. One thing that I noticed here is that there is no escaping of characters in the JSON payload in the input. But I don't have any control on that as well. How do I extract the information from the attribute column in this case
Since I cannot share the whole project, created a sample project and using the inputs from #machaval with http object receiving the csv file. Marked the mimetype as text/csv and sending the payload
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns:tracking="http://www.mulesoft.org/schema/mule/ee/tracking" xmlns:file="http://www.mulesoft.org/schema/mule/file" xmlns:dw="http://www.mulesoft.org/schema/mule/ee/dw" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd
http://www.mulesoft.org/schema/mule/ee/dw http://www.mulesoft.org/schema/mule/ee/dw/current/dw.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/ee/tracking http://www.mulesoft.org/schema/mule/ee/tracking/current/mule-tracking-ee.xsd">
<http:listener-config name="HTTP_Listener_Configuration" host="0.0.0.0" port="8081" doc:name="HTTP Listener Configuration"/>
<flow name="CSVFlow">
<http:listener config-ref="HTTP_Listener_Configuration" path="/process" doc:name="HTTP"/>
<object-to-string-transformer doc:name="Object to String"/>
<dw:transform-message doc:name="Transform Message">
<dw:set-payload><![CDATA[%dw 1.0
%output application/json
---
//First remove the header
(payload splitBy "\n")[1 to -1]
map ((item, index) -> using(commas = item find ",")
{
id: item[0 to commas[0] - 1],
name: item[commas[0] + 1 to commas[1] - 1],
attributes: item[commas[1] + 1 to -1]
}
)]]></dw:set-payload>
</dw:transform-message>
</flow>
</mule>

Answer 1

Hi Rajeev the problem with your data is that is not a CSV the json needs to be escaped somehow. So the way I solved your problem was by handling your input as a String (this can be easily done by using the object to string mp) and parsing it by hand. My assumption is that there are not more tricks on your format.
%dw 1.0
output application/json
---
//First remove the header
(payload splitBy "\n")[1 to -1]
map ((item, index) -> using(commas = item find ",")
{
id: item[0 to commas[0] - 1],
name: item[commas[0] + 1 to commas[1] - 1],
attributes: item[commas[1] + 1 to -1]
}
)

Answer 2

there is one problem - special characters in csv(, ") are surrounded by quotes. if there are more than one columns with multiple quotes and commas in the values the above solution will fail. i took the liberty to modify the solution and add some tweaks to it:
%dw 2.0
output application/java
import * from dw::core::Arrays
var headerRow = (data) -> ((data splitBy "\n")[0]) splitBy ","
var dataRows = (data) -> (data splitBy "\n")[1 to -1]
---
dataRows(payload) map (dataRow,index) ->
do
{
var commas = dataRow find ","
var indices = flatten(dataRow find /(?<!")"(?!")/)
var quoteposition =
indices map
(
(item, index) ->
(
{
(start:item) if isEven(index),
(end:indices[index + 1]) if isEven(index)
}
)
) filter $ != null and $ != {}
fun removeCommasinQuotes(c: Array<Number>, q: Array<{|start?: Number, end?: Number|}>) = c filter (item,index) -> !(q some (item > $.start and item < $.end))
var separator = removeCommasinQuotes(commas,quoteposition)
---
{
(headerRow(payload)[0]): dataRow[0 to separator[0] - 1],
(headerRow(payload)[1]):dataRow[separator[0] + 1 to separator[1] - 1],
(headerRow(payload)[2]):dataRow[separator[1] + 1 to separator[2] - 1],
(headerRow(payload)[3]):dataRow[separator[2] + 1 to separator[3] - 1],
(headerRow(payload)[4]):dataRow[separator[3] + 1 to separator[4] - 1],
(headerRow(payload)[5]):dataRow[separator[4] + 1 to separator[5] - 1],
(headerRow(payload)[6]):dataRow[separator[5] + 1 to separator[6] - 1],
(headerRow(payload)[7]):dataRow[separator[6] + 1 to separator[7] - 1],
(headerRow(payload)[8]):dataRow[separator[7] + 1 to -1]
}
}