ISO-8859-1 Input to UTF-8 Output via VFS - esb

Question

We receive a XML file (not well-formed) encoded as ISO-8859-1 on a Linux system. An ESB Proxy picks this up and uses a Sequence to forward with SAMBA VFS to a Windows share.
On Windows certain characters are not displayed properly because I can't get it to encode as UTF-8. I have tried several approaches all without success. Using xxd these are the characters I have problems with:
0000b70: 4c55 45ef bfbd 5344 204a 503c 2f69 7373 LUEï.½SD JP</iss
0000bf0: 5445 4749 4320 5641 4c55 45ef bfbd 5344 TEGIC VALUEï.½SD
The half symbol. The proxy is:
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse" name="Proxy" transports="vfs" statistics="enable" trace="disable" startOnLoad="true">
<target faultSequence="ErrorSequence">
<inSequence>
<property name="OUT_ONLY" value="true"/>
<clone sequential="true">
<target sequence="Sequence"/>
</clone>
</inSequence>
</target>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.PollInterval">60</parameter>
<parameter name="transport.vfs.MoveAfterProcess">file:///data/process</parameter>
<parameter name="transport.vfs.FileURI">file:///data/incoming</parameter>
<parameter name="transport.vfs.MoveAfterFailure">file:///data/failed</parameter>
<parameter name="transport.vfs.FileNamePattern">security.*</parameter>
<parameter name="transport.vfs.ContentType">text/plain; charset="UTF-8"</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
<description/>
</proxy>
The sequence is:
<sequence xmlns="http://ws.apache.org/ns/synapse" name="Sequence" onError="ErrorSequence" trace="enable">
<xslt key="UTF8Convert">
<property name="encoding" value="UTF-8"></property>
<property name="messageType" value="text/xml"></property>
<property name="ContentType" value="text/xml; charset=UTF-8</property>
</xslt>
<property xmlns:ns="http://org.apache.synapse/xsd" name="filename" expression="(get-property('transport', 'FILE_NAME'))"></property>
<property name="transport.vfs.ContentType" value="text/xml; charset="UTF-8"" scope="transport" type="STRING"></property
<property xmlns:ns="http://org.apache.synapse/xsd" name="transport.vfs.ReplyFileName" expression="get-property('filename')" scope="transport"></property>
<property name="OUT_ONLY" value="true"></property>
<property xmlns:ns="http://org.apache.synapse/xsd" name="serverurl" expression="fn:concat('vfs:smb://',fn:normalize-space(get-property('DOMAIN')),'\',fn:normalize-space(get-property('USERNAME')),':',fn:normalize-space(get-property('PWD')),'#',fn:normalize-space(get-property('DESTINATION')),'/',fn:normalize-space(get-property('DIR')),'/')"></property>
<header xmlns:ns="http://org.apache.synapse/xsd" name="To" expression="fn:concat(get-property('serverurl'),get-property('filename'))"></header>
<send>
<endpoint>
<default></default>
</endpoint>
</send>
</sequence>
The XSLT is:
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:output method="xml" version="1.0" encoding="UTF-8"></xsl:output>
<xsl:template match="#*|node()">
<xsl:copy>
<xsl:apply-templates select="#*|node()"></xsl:apply-templates>
</xsl:copy>
</xsl:template>
</xsl:stylesheet>
The trace log is like this:
00:35:09,924 [-] [vfs-Worker-18] INFO TRACE_LOGGER Start : Sequence<ApexXMLToEPMSequence>
00:35:09,924 [-] [vfs-Worker-18] INFO TRACE_LOGGER Setting the onError handler : ErrorSequence for the sequence : Sequence
00:35:09,924 [-] [vfs-Worker-18] INFO TRACE_LOGGER Sequence <SequenceMediator> ::mediate()
00:35:09,925 [-] [vfs-Worker-18] INFO TRACE_LOGGER Mediation started from mediator position : 0
00:35:09,925 [-] [vfs-Worker-18] INFO TRACE_LOGGER Start : XSLT mediator
00:35:09,927 [-] [vfs-Worker-18] INFO TRACE_LOGGER Setting parameter 'encoding' to 'UTF-8'
00:35:09,927 [-] [vfs-Worker-18] INFO TRACE_LOGGER Setting parameter 'messageType' to 'text/xml'
00:35:09,927 [-] [vfs-Worker-18] INFO TRACE_LOGGER Setting parameter 'ContentType' to 'text/xml; charset=UTF-8'
00:35:09,927 [-] [vfs-Worker-18] INFO TRACE_LOGGER output method: xml; encoding: UTF-8
00:35:09,927 [-] [vfs-Worker-18] INFO TRACE_LOGGER Using org.apache.synapse.util.jaxp.StreamSourceBuilder
00:35:09,932 [-] [vfs-Worker-18] INFO TRACE_LOGGER Transformation completed - processing result
00:35:09,933 [-] [vfs-Worker-18] INFO TRACE_LOGGER Replace node with result
00:35:09,933 [-] [vfs-Worker-18] INFO TRACE_LOGGER End : XSLT mediator
00:35:09,933 [-] [vfs-Worker-18] INFO TRACE_LOGGER Start : Property mediator
00:35:09,933 [-] [vfs-Worker-18] INFO TRACE_LOGGER Setting property : filename at scope : default to : security.2016046.113132 (i.e. result of expression : (get-property('transport', 'FILE_NAME')))
00:35:09,933 [-] [vfs-Worker-18] INFO TRACE_LOGGER End : Property mediator
00:35:09,933 [-] [vfs-Worker-18] INFO TRACE_LOGGER Start : Property mediator
00:35:09,933 [-] [vfs-Worker-18] INFO TRACE_LOGGER Setting property : transport.vfs.ContentType at scope : transport to : text/xml; charset="UTF-8" (i.e. constant : text/xml; charset="UTF-8")
00:35:09,933 [-] [vfs-Worker-18] INFO TRACE_LOGGER End : Property mediator
00:35:09,933 [-] [vfs-Worker-18] INFO TRACE_LOGGER Start : Property mediator
00:35:09,934 [-] [vfs-Worker-18] INFO TRACE_LOGGER Setting property : transport.vfs.ReplyFileName at scope : transport to : security.2016046 (i.e. result of expression : get-property('filename'))
00:35:09,934 [-] [vfs-Worker-18] INFO TRACE_LOGGER End : Property mediator
00:35:09,934 [-] [vfs-Worker-18] INFO TRACE_LOGGER Start : Property mediator
00:35:09,934 [-] [vfs-Worker-18] INFO TRACE_LOGGER Setting property : OUT_ONLY at scope : default to : true (i.e. constant : true)
00:35:09,934 [-] [vfs-Worker-18] INFO TRACE_LOGGER End : Property mediator
00:35:09,934 [-] [vfs-Worker-18] INFO TRACE_LOGGER Start : Property mediator
00:35:09,934 [-] [vfs-Worker-18] INFO TRACE_LOGGER Setting property : serverurl at scope : default to : vfs:smb://AC\user:passw#eserver/folder/ (i.e. result of expression : fn:concat('vfs:smb://',fn:normalize-space(get-property('DOMAIN')),'\',fn:normalize-space(get-property('USERNAME')),':',fn:normalize-space(get-property('PWD')),'#',fn:normalize-space(get-property('DESTINATION')),'/', fn:normalize-space(get-property('DIR')),'/'))
00:35:09,934 [-] [vfs-Worker-18] INFO TRACE_LOGGER End : Property mediator
00:35:09,934 [-] [vfs-Worker-18] INFO TRACE_LOGGER Start : Header mediator
00:35:09,934 [-] [vfs-Worker-18] INFO TRACE_LOGGER Set SOAP header : To to : vfs:smb://AC\user:passw#server/folder/security.2016046
00:35:09,934 [-] [vfs-Worker-18] INFO TRACE_LOGGER End : Header mediator
00:35:09,934 [-] [vfs-Worker-18] INFO TRACE_LOGGER Start : Send mediator
00:35:09,934 [-] [vfs-Worker-18] INFO TRACE_LOGGER Sending message through endpoint : null resolving to address = vfs:smb://AC\user:passw#server/folder/security.2016046
00:35:09,934 [-] [vfs-Worker-18] INFO TRACE_LOGGER SOAPAction: urn:mediate
00:35:09,934 [-] [vfs-Worker-18] INFO TRACE_LOGGER WSA-Action: urn:mediate
00:35:09,935 [-] [vfs-Worker-18] INFO TRACE_LOGGER End : Send mediator
00:35:09,935 [-] [vfs-Worker-18] INFO TRACE_LOGGER End : Sequence <Sequence>
The output on Windows shows as the input above with no coding to UTF-8. How do I make the ESB re-encode please?

Answer 1

If your xml file is encoded as ISO-8859-1, why do you configure your VFS proxy to read it as UTF-8 with : <parameter name="transport.vfs.ContentType">text/plain; charset="UTF-8"</parameter> ?
It seems that you read it as UTF-8 and write it as UTF-8
If you want to force output encoding with VFS, try to set a property named "CHARACTER_SET_ENCODING" before send

Answer 2

If you try to decode ISO-8859-1 encoded files using UTF-8, you will end up with some symbolic character representations (like accented characters). So best thing is Send the file as it is.
Use the VFS parameter like below,
<parameter name="transport.vfs.ContentType">text/plain; charset=iso-8859-1</parameter>
Add the following property and try your scenario.
<property name="messageType" value="text/plain; charset=iso-8859-1" scope="axis2"/>