I need to convert Pipe Delimited text file to JSON in Mule 4. I tried with below code.
`<file:listener doc:name="Fetch Input from Geometry SFTP" doc:id="1151a602-6748-43b7-b491-2caedf6f7010" directory="C:\Vijay\My Projects\AM-Mexico\US\AIM-65\Input" autoDelete="true" outputMimeType="text/csv; separator=|" recursive="false">
<non-repeatable-stream />
<scheduling-strategy >
<fixed-frequency frequency="10" timeUnit="SECONDS"/>
</scheduling-strategy>
</file:listener>
<ee:transform doc:name="Transform Message" doc:id="79dcddf9-34a0-4005-88b4-3a395544be8c" >
<ee:message >
<ee:set-payload ><![CDATA[%dw 2.0
output application/json
input payload text/csv
---
payload]]></ee:set-payload>
</ee:message>
</ee:transform>`
If I try to execute the code i am getting exception as below.
Message : "Internal execution exception while executing the script this is most probably a bug.
Caused by:
java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:147)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at $java.io.InputStream$$FastClassByCGLIB$$31b19c4e.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.mule.extension.file.common.api.stream.AbstractFileInputStream.lambda$createLazyStream$0(AbstractFileInputStream.java:41)
at $java.io.InputStream$$EnhancerByCGLIB$$55e4687e.read(<generated>)
at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:98)
at org.mule.weave.v2.io.DefaultSeekableStream.readUntil(SeekableStream.scala:193)
at org.mule.weave.v2.io.DefaultSeekableStream.delegate$lzycompute(SeekableStream.scala:202)
at org.mule.weave.v2.io.DefaultSeekableStream.delegate(SeekableStream.scala:201)
at org.mule.weave.v2.io.DefaultSeekableStream.seek(SeekableStream.scala:234)
at org.mule.weave.v2.io.SeekableStream.resetStream(SeekableStream.scala:17)
at org.mule.weave.v2.io.SeekableStream.resetStream$(SeekableStream.scala:16)
at org.mule.weave.v2.io.DefaultSeekableStream.resetStream(SeekableStream.scala:186)
at org.mule.weave.v2.model.values.BinaryValue$.getBytesFromSeekableStream(BinaryValue.scala:84)
at org.mule.weave.v2.model.values.BinaryValue$.getBytes(BinaryValue.scala:68)
at org.mule.weave.v2.model.values.BinaryValue.equals(BinaryValue.scala:26)
at org.mule.weave.v2.model.values.BinaryValue.equals$(BinaryValue.scala:25)
at org.mule.weave.v2.module.pojo.reader.JavaBinaryValue.equals(JavaBinaryValue.scala:11)
at org.mule.weave.v2.model.values.wrappers.DelegateValue.equals(DelegateValue.scala:38)
at org.mule.weave.v2.model.values.wrappers.DelegateValue.equals$(DelegateValue.scala:37)
at org.mule.weave.v2.model.values.wrappers.LazyValue.equals(DelegateValue.scala:65)
at org.mule.weave.v2.model.types.Type.$anonfun$acceptsSchema$2(Type.scala:203)
at org.mule.weave.v2.model.types.Type.$anonfun$acceptsSchema$2$adapted(Type.scala:198)
any help on this would be higly appreciated.
You need to set the reader properties for separator:
%dw 2.0
input payload application/csv separator='|'
output application/json
---
payload
Relevant documentation here.
Related
I have a CSV file with a header and these values:
"20000160";"20000160";"177204930";"Zusammendruck ""Blumen"" nk 01.03.07";"2021";"01";"EUR";"599.000";"599,000";"599.00";"599,00 EUR";"EUR";"0.00";"0,00 EUR";"0.00";"0,00 EUR";"EUR"
"20000000";"20000000";"1013";"Einschreiben";"2021";"01";"EUR";"0.000";"0,000";"22.80";"22,80 EUR";"EUR";"0.00";"0,00 EUR";"0.00";"0,00 EUR";"EUR"
"20000000";"20000000";"1018";"Rückschein";"2021";"01";"EUR";"0.000";"0,000";"6.60";"6,60 EUR";"EUR";"0.00";"0,00 EUR";"0.00";"0,00 EUR";"EUR"
"8003325905";"8003325905";"233800118";"Prof.Services: Datenmanagement;Pauschale";"2021";"01";"EUR";"0.000";"0,000";"600.00";"600,00 EUR";"EUR";"0.00";"0,00 EUR";"108.00";"108,00 EUR";"EUR"
I configured File Read connector to escape "Zusammendruck ""Blumen"" nk 01.03.07", and it is working:
<file:read doc:name="Read CSV, Set MIME Type" doc:id="bb378f83-d0ea-4951-8253-8253953ed9e7" path="${outputCSV}" outputMimeType='application/csv; streaming=true; quote="\""; separator=";"; escape="\""' outputEncoding="UTF-8" />
But I also have to escape ; to correctly parse "Prof.Services: Datenmanagement;Pauschale". I tried to configure the pattern as escape="\"|;" but I got an warn:
WARN 2021-12-20 16:59:34,604 [[MuleRuntime].uber.26:
[test].upload.BLOCKING #27454a45] [processor: ; event:
eb625e31-61b5-11ec-a30d-00090ffe0001]
org.mule.weave.v2.model.service.DefaultLoggingService$: Option
escape="|; expects a value of length 1 but got "|;. Only the
first character is going to be used and the rest is going to be
ignored.
How can I read and parse data correctly, considering the example data?
The default CSV parser in DataWeave has some limitations. For this reason I have developed a Mule module based on Apache Commons CSV. You can find it on GitHub: https://github.com/rbutenuth/csv-module The dependency is available on Maven Central, so you don't need to compile it yourself.
It can parse your CSV with the following settings:
<csv:config name="Csv_Config" doc:name="Csv Config" recordSeparator="\n" withHeaderLine="false" escape="">
<csv:columns >
<csv:column columnName="column_01" type="TEXT" />
<csv:column columnName="column_02" type="TEXT" />
<!-- columnss 3 to 16 ommitted -->
<csv:column columnName="column_17" type="TEXT" />
</csv:columns>
</csv:config>
I have used column_01 to column_17 as column names, as I could not guess meaningfull names from the content.
You can achieve it in your case with DataWeave with the following settings:
application/csv separator=";",quoteValues=true,quote="\"",escape="\""
Testing a POC and wondering what the quickest/simplest way to pretty print a JSON payload in the Anypoint Studio console. I'd rather not figure out how to use a Groovy component to accomplish this.
I've tried setting the application/json indent property to true as below, but it still prints the entire object on a single line (which is indented).
output application/json indent=true
Any lightweight solutions for quick POC work?
The console in Studio shows what it is being logged. If you are using DataWeave to create the JSON, as evidenced by the script you shared, then the default value for the indent writer property is true. So when logging it, the generated JSON will be pretty-printed by default.
Example:
<ee:transform doc:name="Transform Message" >
<ee:message >
<ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
{
a:"aaaa",
b: {c:1, d:"dddd"},
e: "ccc"
}]]></ee:set-payload>
</ee:message>
</ee:transform>
<logger level="INFO" doc:name="Logger" message="Payload: #[payload]"/>
Output (omitting all the logging information for clarity):
Payload: {
"a": "aaaa",
"b": {
"c": 1,
"d": "dddd"
},
"e": "ccc"
}
I have a json file which contains a collection of numerous JSON objects. A sample format is given below:
{"ID": 123,"Name": "TEST-1","val11": {},"url": "test1.com","val12": []}
{"ID": 456,"Name": "TEST-2","val21": {},"url": "test2.com","val22": []}
{"ID": 789,"Name": "TEST-3","val31": {},"url": "test3.com","val32": []}
As you see, it is not an array ([ ] and commas missing). I need to convert this into a valid JSON array.
The code that I tried is:
%dw 2.0
output application/json
var PayloadSplit = payload splitBy('\n')
var PayloadArray = (PayloadSplit map (value, index) -> read(value, 'application/json'))
---
PayloadArray
This works fine for a small sized payload. However, if I try to perform this on the entire file (size about 320 MB with ~20k JSON objects), it fails with a java.lang.OutOfMemoryError: Java heap space error. Is there a way to overcome this? Or can I split the main file into multiple files and then try this (in a ForEach Loop perhaps?). Please advise
Edit1 - Attaching the mule flow below:
<flow name="convert-object-to-array-test1Flow" doc:id="0645e9bd-7f77-4b1e-93d0-dedd9d154ef7" >
<http:listener doc:name="Listener" doc:id="551cd3b6-e4c8-4b7a-aff3-305effbe8a8b" config-ref="HTTP_Listener_config" path="/file"/>
<file:read doc:name="Read" doc:id="21a310c1-5887-4bc0-83b9-b8968e145f0d" path="C:\Desktop\NDJsonSample.json" outputMimeType="application/ndjson" />
<ee:transform doc:name="Transform Message" doc:id="95235c56-2f5a-4f39-ba96-8be7c4e501b5" >
<ee:message >
<ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
payload]]></ee:set-payload>
</ee:message>
</ee:transform>
<logger level="INFO" doc:name="Logger" doc:id="935530dd-17fd-41c9-8fe0-1561ba3de703" />
</flow>
DW already have support for this format. It is called ndjson. Please visit the documentation. You just need to set application/ndjson to the payload.
I am getting an error while using a dataweave transformation on a JSON Payload. The JSON Payload is
{
"requestId": "13431#1638a2abfb8",
"result": [
{
"batchId": 1028,
"importId": "1028",
"status": "Queued"
}
],
"success": true
}
The above payload is returned by a RESTful service and I have converted that to a object using byteArray to Object transformer before applying the following dataweave transformation
%dw 1.0
%output application/json
---
batchexecution:
{
batchid:payload.result[0].batchid,
status: payload.result[0].status,
success:payload.success
} when ((payload.result != null) and (sizeOf payload.result > 0))
otherwise
{
batchid: 0,
status:"Not queued",
success:false
}
I am expecting only one record for the result object and I have a check to see whether the array is null or its size is >0. I get the following error when I execute the transformation code. Not sure what is wrong here.
I am expecting the following output for the transformation but I am getting the error while executing the transformation code
{
"batchexecution": {
"batchId": 1028,
"status": "Queued",
"success": true
}
}
But I am getting the following error as You cannot compare a value of type ::array.
Message : Exception while executing:
{"requestId":"64b3#1638e55058c","result":[{"batchId":1037,"importId":"1037","status":"Queued"}],"success":true}
^
You cannot compare a value of type ::array.
Payload : {"requestId":"64b3#1638e55058c","result":[{"batchId":1037,"importId":"1037","status":"Queued"}],"success":true}
Payload Type : java.lang.String
Element : /marketing-dbmkt-etl-marketoFlow/processors/8 # marketing-dbmkt-etl-marketo:marketing-dbmkt-etl-marketo.xml:69 (Transform Message)
Element XML : <dw:transform-message doc:name="Transform Message" metadata:id="90448cfd-5884-441a-a989-e32e4877ac24">
<dw:input-payload mimeType="application/json" doc:sample="sample_data\batchreturnObject.dwl"></dw:input-payload>
<dw:set-payload>%dw 1.0%output application/json---batchexecution:{batchid:payload.result[0].batchid,status: payload.result[0].status,success:payload.success} when ((payload.result != null) and (sizeOf payload.result > 0))otherwise{batchid: 0,status:"Not queued",success:false}</dw:set-payload>
</dw:transform-message>
--------------------------------------------------------------------------------
Root Exception stack trace:
com.mulesoft.weave.mule.exception.WeaveExecutionException: Exception while executing:
{"requestId":"64b3#1638e55058c","result":[{"batchId":1037,"importId":"1037","status":"Queued"}],"success":true}
^
You cannot compare a value of type ::array.
at com.mulesoft.weave.mule.exception.WeaveExecutionException$.apply(WeaveExecutionException.scala:10)
at com.mulesoft.weave.mule.WeaveMessageProcessor.execute(WeaveMessageProcessor.scala:121)
at com.mulesoft.weave.mule.WeaveMessageProcessor.process(WeaveMessageProcessor.scala:67)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:108)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:98)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.interceptor.AbstractEnvelopeInterceptor.processBlocking(AbstractEnvelopeInterceptor.java:58)
at org.mule.processor.AbstractRequestResponseMessageProcessor.process(AbstractRequestResponseMessageProcessor.java:47)
at org.mule.processor.AsyncInterceptingMessageProcessor.processNextTimed(AsyncInterceptingMessageProcessor.java:129)
at org.mule.processor.AsyncInterceptingMessageProcessor$AsyncMessageProcessorWorker$1.process(AsyncInterceptingMessageProcessor.java:213)
at org.mule.processor.AsyncInterceptingMessageProcessor$AsyncMessageProcessorWorker$1.process(AsyncInterceptingMessageProcessor.java:206)
at org.mule.execution.ExecuteCallbackInterceptor.execute(ExecuteCallbackInterceptor.java:16)
at org.mule.execution.CommitTransactionInterceptor.execute(CommitTransactionInterceptor.java:35)
at org.mule.execution.CommitTransactionInterceptor.execute(CommitTransactionInterceptor.java:22)
at org.mule.execution.HandleExceptionInterceptor.execute(HandleExceptionInterceptor.java:30)
at org.mule.execution.HandleExceptionInterceptor.execute(HandleExceptionInterceptor.java:14)
at org.mule.execution.BeginAndResolveTransactionInterceptor.execute(BeginAndResolveTransactionInterceptor.java:67)
at org.mule.execution.ResolvePreviousTransactionInterceptor.execute(ResolvePreviousTransactionInterceptor.java:44)
at org.mule.execution.SuspendXaTransactionInterceptor.execute(SuspendXaTransactionInterceptor.java:50)
at org.mule.execution.ValidateTransactionalStateInterceptor.execute(ValidateTransactionalStateInterceptor.java:40)
at org.mule.execution.IsolateCurrentTransactionInterceptor.execute(IsolateCurrentTransactionInterceptor.java:41)
at org.mule.execution.ExternalTransactionInterceptor.execute(ExternalTransactionInterceptor.java:48)
at org.mule.execution.RethrowExceptionInterceptor.execute(RethrowExceptionInterceptor.java:28)
at org.mule.execution.RethrowExceptionInterceptor.execute(RethrowExceptionInterceptor.java:13)
at org.mule.execution.TransactionalErrorHandlingExecutionTemplate.execute(TransactionalErrorHandlingExecutionTemplate.java:110)
at org.mule.execution.TransactionalErrorHandlingExecutionTemplate.execute(TransactionalErrorHandlingExecutionTemplate.java:30)
at org.mule.processor.AsyncInterceptingMessageProcessor$AsyncMessageProcessorWorker.doRun(AsyncInterceptingMessageProcessor.java:205)
at org.mule.work.AbstractMuleEventWork.run(AbstractMuleEventWork.java:53)
at org.mule.work.WorkerContext.run(WorkerContext.java:301)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
********************************************************************************
The problem here is not obvious, but I have come across the same issue before - it's related to the sizeOf function and the poor way that Mule applies precedence to some of it's operators. When you say (sizeOf payload.result > 0) it's first trying to attempt to resolve the payload.result > 0 expression - hence the error you're seeing (it's trying to compare an array to 0).
Please use ((sizeOf payload.result) > 0) instead (I always make a point of wrapping sizeOf in parentheses for this reason).
As a side note, you have batchid:payload.result[0].batchid - it should be batchId:payload.result[0].batchId (capitalisation in batchId)
whenever you are using any function like sizeOf in dataweave try to encapsulate it with round braces to avoid these kinds of errors.
#ghoshyTech in your case
when ((payload.result != null) and ((sizeOf payload.result) > 0))
I am using Anypoint Studio 6.1 and Mule 3.8.1 and have the following JSON response from a Mule workflow:
{
"Description": "Top 10 games:
- Mario Kart
- Legend of Zelda"
}
This fails validation with the following error:
Error: Parse error on line 2:
{ "Description": "Top 10 games: - M
-----------------^
Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '[', got 'undefined'
I have tried escaping the special characters put it still does not work. How can I change the response to pass JSON v4 validation. The JSON validator I have been using is http://jsonlint.com/
Thanks
The root cause of the issue is carriage return in the JSON response. The valid JSON should be as below
{
"Description": "Top 10 games:\r\n- Mario Kart\r\n- Legend of Zelda"
}
Please provide more details on where you are receiving this description. is it from XML or any other source. Also it will be helpful if you post your DWL code as well.