Mule 3.8.3 Studio 6.4.4
I am receiving a XML payload that is a collection of customer numbers. I need to end up pulling each number out and sending it to a messaging queue.
Sample Incoming Data:
<request func="">
<data>
<transactions time='1539262470'>
<transaction set='customers' notifyid='WMS_NADC_CUSTOMERS' type='update'>
<customers>
<customer id="CIT_1113-11" t="1539257721" y="U" w="WebUser"></customer>
<customer id="C42998-2" t="1539261561" y="N" w="WebUser"></customer>
<customer id="C42998" t="1539262040" y="U" w="WebUser"> </customer>
</customers>
</transaction>
</transactions>
</data>
</request>
After receiving this I use weave to transform into json in an attempt to more easily access the id's.
%dw 1.0
%output application/json
---
{
customers: payload.request.data.transactions.transaction.customers.*customer map (cust, indexOfCustomer) ->{
customer: cust.#id as :string
}
}
The transformed payload now looks like
{
"customers": [
{
"customer": "CIT_1113-11"
},
{
"customer": "C42998-2"
},
{
"customer": "C42998"
}
]
}
At this point, I am trying to loop through the payload. Setting the for each to payload.get('customers') takes me into a jackson.node.ArrayNode.
I haven't been able to figure out how to access each individual object inside the list. Can one of you please tell me how to do that?
I want to end up placing a payload onto a message queue that looks like
{
"customer": "C42998"
}
The simplest way would be to use a combination of DataWeave to get a Java Iteratable, and a for-each scope. Check out this example (assuming the incoming payload is your XML).
<dw:transform-message doc:name="Transform Message">
<dw:set-payload><![CDATA[%dw 1.0
%output application/java
%var customers = payload.request.data.transactions.transaction.customers.*customer
---
customers map (customer) ->{
customer: customer.#id as :string
}]]></dw:set-payload>
</dw:transform-message>
<foreach doc:name="For Each">
<json:object-to-json-transformer doc:name="Object to JSON"/>
<logger message="#[payload]" level="INFO" doc:name="Logger"/>
</foreach>
In Mule 3, the for-each won't accept JSON or XML, even if they obviously represent a structure that could be iterated over (like a JSON array, for example). This is why we need %output application/java in the DataWeave transformer. Later, within the for-each scope, we can transform this back to JSON. In your case, just replace the logger with your queue connector to send the messages where needed.
Related
Testing a POC and wondering what the quickest/simplest way to pretty print a JSON payload in the Anypoint Studio console. I'd rather not figure out how to use a Groovy component to accomplish this.
I've tried setting the application/json indent property to true as below, but it still prints the entire object on a single line (which is indented).
output application/json indent=true
Any lightweight solutions for quick POC work?
The console in Studio shows what it is being logged. If you are using DataWeave to create the JSON, as evidenced by the script you shared, then the default value for the indent writer property is true. So when logging it, the generated JSON will be pretty-printed by default.
Example:
<ee:transform doc:name="Transform Message" >
<ee:message >
<ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
{
a:"aaaa",
b: {c:1, d:"dddd"},
e: "ccc"
}]]></ee:set-payload>
</ee:message>
</ee:transform>
<logger level="INFO" doc:name="Logger" message="Payload: #[payload]"/>
Output (omitting all the logging information for clarity):
Payload: {
"a": "aaaa",
"b": {
"c": 1,
"d": "dddd"
},
"e": "ccc"
}
I have a json file which contains a collection of numerous JSON objects. A sample format is given below:
{"ID": 123,"Name": "TEST-1","val11": {},"url": "test1.com","val12": []}
{"ID": 456,"Name": "TEST-2","val21": {},"url": "test2.com","val22": []}
{"ID": 789,"Name": "TEST-3","val31": {},"url": "test3.com","val32": []}
As you see, it is not an array ([ ] and commas missing). I need to convert this into a valid JSON array.
The code that I tried is:
%dw 2.0
output application/json
var PayloadSplit = payload splitBy('\n')
var PayloadArray = (PayloadSplit map (value, index) -> read(value, 'application/json'))
---
PayloadArray
This works fine for a small sized payload. However, if I try to perform this on the entire file (size about 320 MB with ~20k JSON objects), it fails with a java.lang.OutOfMemoryError: Java heap space error. Is there a way to overcome this? Or can I split the main file into multiple files and then try this (in a ForEach Loop perhaps?). Please advise
Edit1 - Attaching the mule flow below:
<flow name="convert-object-to-array-test1Flow" doc:id="0645e9bd-7f77-4b1e-93d0-dedd9d154ef7" >
<http:listener doc:name="Listener" doc:id="551cd3b6-e4c8-4b7a-aff3-305effbe8a8b" config-ref="HTTP_Listener_config" path="/file"/>
<file:read doc:name="Read" doc:id="21a310c1-5887-4bc0-83b9-b8968e145f0d" path="C:\Desktop\NDJsonSample.json" outputMimeType="application/ndjson" />
<ee:transform doc:name="Transform Message" doc:id="95235c56-2f5a-4f39-ba96-8be7c4e501b5" >
<ee:message >
<ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
payload]]></ee:set-payload>
</ee:message>
</ee:transform>
<logger level="INFO" doc:name="Logger" doc:id="935530dd-17fd-41c9-8fe0-1561ba3de703" />
</flow>
DW already have support for this format. It is called ndjson. Please visit the documentation. You just need to set application/ndjson to the payload.
I have a web service which exports a CSV file which has one of the columns as a JSON payload. After the execution of the web service, I store the values in a local variable for transformation. Every time when I read the values from that column I am missing the values and only "}" is returned. Not sure why this is happening. I want to preserve the JSON payload as is and persist to a file after some processing. Please advise
I am using the code below to get the value of the attribute column and it always returns a "}". The rest of the contents are ignored
CSV Fragment
-------------
id,name,attributes
1,name1,{"Choice Number":"0","Campaign Run Id":"234"}
2,name2,{"Choice Number":"1","Campaign Run Id":"235"}
3,name3,{"Choice Number":"2","Campaign Run Id":"236"}
4,name4,{"Choice Number":"3","Campaign Run Id":"236"}
Code
----
%dw 1.0
%output application/java
---
flowVars.activityData map ((actData) -> {
"playerId": actData.id,
"name": actData.name,
"attributes": actData.attributes
})
I was expecting that the full JSON payload from the attributes column will be returned and that is not the case. One thing that I noticed here is that there is no escaping of characters in the JSON payload in the input. But I don't have any control on that as well. How do I extract the information from the attribute column in this case
Since I cannot share the whole project, created a sample project and using the inputs from #machaval with http object receiving the csv file. Marked the mimetype as text/csv and sending the payload
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns:tracking="http://www.mulesoft.org/schema/mule/ee/tracking" xmlns:file="http://www.mulesoft.org/schema/mule/file" xmlns:dw="http://www.mulesoft.org/schema/mule/ee/dw" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd
http://www.mulesoft.org/schema/mule/ee/dw http://www.mulesoft.org/schema/mule/ee/dw/current/dw.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/ee/tracking http://www.mulesoft.org/schema/mule/ee/tracking/current/mule-tracking-ee.xsd">
<http:listener-config name="HTTP_Listener_Configuration" host="0.0.0.0" port="8081" doc:name="HTTP Listener Configuration"/>
<flow name="CSVFlow">
<http:listener config-ref="HTTP_Listener_Configuration" path="/process" doc:name="HTTP"/>
<object-to-string-transformer doc:name="Object to String"/>
<dw:transform-message doc:name="Transform Message">
<dw:set-payload><![CDATA[%dw 1.0
%output application/json
---
//First remove the header
(payload splitBy "\n")[1 to -1]
map ((item, index) -> using(commas = item find ",")
{
id: item[0 to commas[0] - 1],
name: item[commas[0] + 1 to commas[1] - 1],
attributes: item[commas[1] + 1 to -1]
}
)]]></dw:set-payload>
</dw:transform-message>
</flow>
</mule>
Hi Rajeev the problem with your data is that is not a CSV the json needs to be escaped somehow. So the way I solved your problem was by handling your input as a String (this can be easily done by using the object to string mp) and parsing it by hand. My assumption is that there are not more tricks on your format.
%dw 1.0
output application/json
---
//First remove the header
(payload splitBy "\n")[1 to -1]
map ((item, index) -> using(commas = item find ",")
{
id: item[0 to commas[0] - 1],
name: item[commas[0] + 1 to commas[1] - 1],
attributes: item[commas[1] + 1 to -1]
}
)
there is one problem - special characters in csv(, ") are surrounded by quotes. if there are more than one columns with multiple quotes and commas in the values the above solution will fail. i took the liberty to modify the solution and add some tweaks to it:
%dw 2.0
output application/java
import * from dw::core::Arrays
var headerRow = (data) -> ((data splitBy "\n")[0]) splitBy ","
var dataRows = (data) -> (data splitBy "\n")[1 to -1]
---
dataRows(payload) map (dataRow,index) ->
do
{
var commas = dataRow find ","
var indices = flatten(dataRow find /(?<!")"(?!")/)
var quoteposition =
indices map
(
(item, index) ->
(
{
(start:item) if isEven(index),
(end:indices[index + 1]) if isEven(index)
}
)
) filter $ != null and $ != {}
fun removeCommasinQuotes(c: Array<Number>, q: Array<{|start?: Number, end?: Number|}>) = c filter (item,index) -> !(q some (item > $.start and item < $.end))
var separator = removeCommasinQuotes(commas,quoteposition)
---
{
(headerRow(payload)[0]): dataRow[0 to separator[0] - 1],
(headerRow(payload)[1]):dataRow[separator[0] + 1 to separator[1] - 1],
(headerRow(payload)[2]):dataRow[separator[1] + 1 to separator[2] - 1],
(headerRow(payload)[3]):dataRow[separator[2] + 1 to separator[3] - 1],
(headerRow(payload)[4]):dataRow[separator[3] + 1 to separator[4] - 1],
(headerRow(payload)[5]):dataRow[separator[4] + 1 to separator[5] - 1],
(headerRow(payload)[6]):dataRow[separator[5] + 1 to separator[6] - 1],
(headerRow(payload)[7]):dataRow[separator[6] + 1 to separator[7] - 1],
(headerRow(payload)[8]):dataRow[separator[7] + 1 to -1]
}
}
I need to convert Pipe Delimited text file to JSON in Mule 4. I tried with below code.
`<file:listener doc:name="Fetch Input from Geometry SFTP" doc:id="1151a602-6748-43b7-b491-2caedf6f7010" directory="C:\Vijay\My Projects\AM-Mexico\US\AIM-65\Input" autoDelete="true" outputMimeType="text/csv; separator=|" recursive="false">
<non-repeatable-stream />
<scheduling-strategy >
<fixed-frequency frequency="10" timeUnit="SECONDS"/>
</scheduling-strategy>
</file:listener>
<ee:transform doc:name="Transform Message" doc:id="79dcddf9-34a0-4005-88b4-3a395544be8c" >
<ee:message >
<ee:set-payload ><![CDATA[%dw 2.0
output application/json
input payload text/csv
---
payload]]></ee:set-payload>
</ee:message>
</ee:transform>`
If I try to execute the code i am getting exception as below.
Message : "Internal execution exception while executing the script this is most probably a bug.
Caused by:
java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:147)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at $java.io.InputStream$$FastClassByCGLIB$$31b19c4e.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.mule.extension.file.common.api.stream.AbstractFileInputStream.lambda$createLazyStream$0(AbstractFileInputStream.java:41)
at $java.io.InputStream$$EnhancerByCGLIB$$55e4687e.read(<generated>)
at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:98)
at org.mule.weave.v2.io.DefaultSeekableStream.readUntil(SeekableStream.scala:193)
at org.mule.weave.v2.io.DefaultSeekableStream.delegate$lzycompute(SeekableStream.scala:202)
at org.mule.weave.v2.io.DefaultSeekableStream.delegate(SeekableStream.scala:201)
at org.mule.weave.v2.io.DefaultSeekableStream.seek(SeekableStream.scala:234)
at org.mule.weave.v2.io.SeekableStream.resetStream(SeekableStream.scala:17)
at org.mule.weave.v2.io.SeekableStream.resetStream$(SeekableStream.scala:16)
at org.mule.weave.v2.io.DefaultSeekableStream.resetStream(SeekableStream.scala:186)
at org.mule.weave.v2.model.values.BinaryValue$.getBytesFromSeekableStream(BinaryValue.scala:84)
at org.mule.weave.v2.model.values.BinaryValue$.getBytes(BinaryValue.scala:68)
at org.mule.weave.v2.model.values.BinaryValue.equals(BinaryValue.scala:26)
at org.mule.weave.v2.model.values.BinaryValue.equals$(BinaryValue.scala:25)
at org.mule.weave.v2.module.pojo.reader.JavaBinaryValue.equals(JavaBinaryValue.scala:11)
at org.mule.weave.v2.model.values.wrappers.DelegateValue.equals(DelegateValue.scala:38)
at org.mule.weave.v2.model.values.wrappers.DelegateValue.equals$(DelegateValue.scala:37)
at org.mule.weave.v2.model.values.wrappers.LazyValue.equals(DelegateValue.scala:65)
at org.mule.weave.v2.model.types.Type.$anonfun$acceptsSchema$2(Type.scala:203)
at org.mule.weave.v2.model.types.Type.$anonfun$acceptsSchema$2$adapted(Type.scala:198)
any help on this would be higly appreciated.
You need to set the reader properties for separator:
%dw 2.0
input payload application/csv separator='|'
output application/json
---
payload
Relevant documentation here.
I am getting an error while using a dataweave transformation on a JSON Payload. The JSON Payload is
{
"requestId": "13431#1638a2abfb8",
"result": [
{
"batchId": 1028,
"importId": "1028",
"status": "Queued"
}
],
"success": true
}
The above payload is returned by a RESTful service and I have converted that to a object using byteArray to Object transformer before applying the following dataweave transformation
%dw 1.0
%output application/json
---
batchexecution:
{
batchid:payload.result[0].batchid,
status: payload.result[0].status,
success:payload.success
} when ((payload.result != null) and (sizeOf payload.result > 0))
otherwise
{
batchid: 0,
status:"Not queued",
success:false
}
I am expecting only one record for the result object and I have a check to see whether the array is null or its size is >0. I get the following error when I execute the transformation code. Not sure what is wrong here.
I am expecting the following output for the transformation but I am getting the error while executing the transformation code
{
"batchexecution": {
"batchId": 1028,
"status": "Queued",
"success": true
}
}
But I am getting the following error as You cannot compare a value of type ::array.
Message : Exception while executing:
{"requestId":"64b3#1638e55058c","result":[{"batchId":1037,"importId":"1037","status":"Queued"}],"success":true}
^
You cannot compare a value of type ::array.
Payload : {"requestId":"64b3#1638e55058c","result":[{"batchId":1037,"importId":"1037","status":"Queued"}],"success":true}
Payload Type : java.lang.String
Element : /marketing-dbmkt-etl-marketoFlow/processors/8 # marketing-dbmkt-etl-marketo:marketing-dbmkt-etl-marketo.xml:69 (Transform Message)
Element XML : <dw:transform-message doc:name="Transform Message" metadata:id="90448cfd-5884-441a-a989-e32e4877ac24">
<dw:input-payload mimeType="application/json" doc:sample="sample_data\batchreturnObject.dwl"></dw:input-payload>
<dw:set-payload>%dw 1.0%output application/json---batchexecution:{batchid:payload.result[0].batchid,status: payload.result[0].status,success:payload.success} when ((payload.result != null) and (sizeOf payload.result > 0))otherwise{batchid: 0,status:"Not queued",success:false}</dw:set-payload>
</dw:transform-message>
--------------------------------------------------------------------------------
Root Exception stack trace:
com.mulesoft.weave.mule.exception.WeaveExecutionException: Exception while executing:
{"requestId":"64b3#1638e55058c","result":[{"batchId":1037,"importId":"1037","status":"Queued"}],"success":true}
^
You cannot compare a value of type ::array.
at com.mulesoft.weave.mule.exception.WeaveExecutionException$.apply(WeaveExecutionException.scala:10)
at com.mulesoft.weave.mule.WeaveMessageProcessor.execute(WeaveMessageProcessor.scala:121)
at com.mulesoft.weave.mule.WeaveMessageProcessor.process(WeaveMessageProcessor.scala:67)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:108)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:98)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.interceptor.AbstractEnvelopeInterceptor.processBlocking(AbstractEnvelopeInterceptor.java:58)
at org.mule.processor.AbstractRequestResponseMessageProcessor.process(AbstractRequestResponseMessageProcessor.java:47)
at org.mule.processor.AsyncInterceptingMessageProcessor.processNextTimed(AsyncInterceptingMessageProcessor.java:129)
at org.mule.processor.AsyncInterceptingMessageProcessor$AsyncMessageProcessorWorker$1.process(AsyncInterceptingMessageProcessor.java:213)
at org.mule.processor.AsyncInterceptingMessageProcessor$AsyncMessageProcessorWorker$1.process(AsyncInterceptingMessageProcessor.java:206)
at org.mule.execution.ExecuteCallbackInterceptor.execute(ExecuteCallbackInterceptor.java:16)
at org.mule.execution.CommitTransactionInterceptor.execute(CommitTransactionInterceptor.java:35)
at org.mule.execution.CommitTransactionInterceptor.execute(CommitTransactionInterceptor.java:22)
at org.mule.execution.HandleExceptionInterceptor.execute(HandleExceptionInterceptor.java:30)
at org.mule.execution.HandleExceptionInterceptor.execute(HandleExceptionInterceptor.java:14)
at org.mule.execution.BeginAndResolveTransactionInterceptor.execute(BeginAndResolveTransactionInterceptor.java:67)
at org.mule.execution.ResolvePreviousTransactionInterceptor.execute(ResolvePreviousTransactionInterceptor.java:44)
at org.mule.execution.SuspendXaTransactionInterceptor.execute(SuspendXaTransactionInterceptor.java:50)
at org.mule.execution.ValidateTransactionalStateInterceptor.execute(ValidateTransactionalStateInterceptor.java:40)
at org.mule.execution.IsolateCurrentTransactionInterceptor.execute(IsolateCurrentTransactionInterceptor.java:41)
at org.mule.execution.ExternalTransactionInterceptor.execute(ExternalTransactionInterceptor.java:48)
at org.mule.execution.RethrowExceptionInterceptor.execute(RethrowExceptionInterceptor.java:28)
at org.mule.execution.RethrowExceptionInterceptor.execute(RethrowExceptionInterceptor.java:13)
at org.mule.execution.TransactionalErrorHandlingExecutionTemplate.execute(TransactionalErrorHandlingExecutionTemplate.java:110)
at org.mule.execution.TransactionalErrorHandlingExecutionTemplate.execute(TransactionalErrorHandlingExecutionTemplate.java:30)
at org.mule.processor.AsyncInterceptingMessageProcessor$AsyncMessageProcessorWorker.doRun(AsyncInterceptingMessageProcessor.java:205)
at org.mule.work.AbstractMuleEventWork.run(AbstractMuleEventWork.java:53)
at org.mule.work.WorkerContext.run(WorkerContext.java:301)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
********************************************************************************
The problem here is not obvious, but I have come across the same issue before - it's related to the sizeOf function and the poor way that Mule applies precedence to some of it's operators. When you say (sizeOf payload.result > 0) it's first trying to attempt to resolve the payload.result > 0 expression - hence the error you're seeing (it's trying to compare an array to 0).
Please use ((sizeOf payload.result) > 0) instead (I always make a point of wrapping sizeOf in parentheses for this reason).
As a side note, you have batchid:payload.result[0].batchid - it should be batchId:payload.result[0].batchId (capitalisation in batchId)
whenever you are using any function like sizeOf in dataweave try to encapsulate it with round braces to avoid these kinds of errors.
#ghoshyTech in your case
when ((payload.result != null) and ((sizeOf payload.result) > 0))