WSO2 - ESB Pass through Proxy Post Request Failure - esb

Question

Hi guys can you help me out with my WSO2 - ESB problem.
I am experiencing the following error in our UAT environment but not in my SIT environment. I need a hint on why this is happening so I can change my CAR file or add the proper configuration to our server if needed.
For some reason when the Client doing a Post request I am experiencing a timeout error. Upon checking the logs I see that their is an error that states that the Payload could not be written as JSON. This does not happen in a GET request so I am at a lost. I can also confirm that the request is being sent in the back end using either GET/POST.
Please note that the following servers does not exist so app03/app04 so failure to connect to those server is expected (We only have 2 UAT servers but 4 in our PRD environment).
What I do not understand is why this only happens in Post request and if i need to modify my code or set something in the WSO2-ESB server.
For any clarification or additional information please do not hesitate to ask.
Thanks for your help
Carbon Logs
TID: [-1234] [] [2017-09-22 09:12:03,608] INFO {org.apache.synapse.mediators.builtin.LogMediator} - Request: Target URL = /services/XXXX-ws_PS/payments/validate, Client IP = 172.22.222.156, Request ID = urn:uuid:0df0d876-6093-4479-9537-70bbbdf4d649 {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2017-09-22 09:12:04,029] INFO {org.apache.synapse.mediators.builtin.LogMediator} - Response: To = /services/XXXX-ws_PS/payments/validate, IP = 172.22.222.156, RequestID = urn:uuid:0df0d876-6093-4479-9537-70bbbdf4d649 {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2017-09-22 09:12:04,029] ERROR {org.apache.synapse.commons.json.JsonUtil} - #writeAsJson. Payload could not be written as JSON. MessageID: urn:uuid:3b69a0e3-f6b3-4e49-8d14-23d7596c2366 {org.apache.synapse.commons.json.JsonUtil}
TID: [-1234] [] [2017-09-22 09:12:04,029] ERROR {org.apache.synapse.transport.passthru.PassThroughHttpSSLSender} - Failed to submit the response {org.apache.synapse.transport.passthru.PassThroughHttpSSLSender}
org.apache.axis2.AxisFault: Payload could not be written as JSON.
at org.apache.synapse.commons.json.JsonUtil.writeAsJson(JsonUtil.java:263)
at org.apache.synapse.commons.json.JsonStreamFormatter.writeTo(JsonStreamFormatter.java:61)
at org.apache.synapse.transport.passthru.PassThroughHttpSender.submitResponse(PassThroughHttpSender.java:555)
at org.apache.synapse.transport.passthru.PassThroughHttpSender.invoke(PassThroughHttpSender.java:264)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
at org.apache.synapse.core.axis2.Axis2Sender.sendBack(Axis2Sender.java:222)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:531)
at org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:118)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:97)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:59)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:158)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:337)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.handleMessage(SynapseCallbackReceiver.java:554)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.receive(SynapseCallbackReceiver.java:188)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ClientWorker.run(ClientWorker.java:261)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
TID: [-1234] [] [2017-09-22 09:12:04,042] ERROR {org.apache.synapse.core.axis2.Axis2Sender} - <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body/></soapenv:Envelope> Unexpected error sending message back {org.apache.synapse.core.axis2.Axis2Sender}
org.apache.axis2.AxisFault: Failed to submit the response
at org.apache.synapse.transport.passthru.PassThroughHttpSender.handleException(PassThroughHttpSender.java:613)
at org.apache.synapse.transport.passthru.PassThroughHttpSender.invoke(PassThroughHttpSender.java:266)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
at org.apache.synapse.core.axis2.Axis2Sender.sendBack(Axis2Sender.java:222)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:531)
at org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:118)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:97)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:59)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:158)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:337)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.handleMessage(SynapseCallbackReceiver.java:554)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.receive(SynapseCallbackReceiver.java:188)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ClientWorker.run(ClientWorker.java:261)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.axis2.AxisFault: Payload could not be written as JSON.
at org.apache.synapse.commons.json.JsonUtil.writeAsJson(JsonUtil.java:263)
at org.apache.synapse.commons.json.JsonStreamFormatter.writeTo(JsonStreamFormatter.java:61)
at org.apache.synapse.transport.passthru.PassThroughHttpSender.submitResponse(PassThroughHttpSender.java:555)
at org.apache.synapse.transport.passthru.PassThroughHttpSender.invoke(PassThroughHttpSender.java:264)
... 16 more
TID: [-1234] [] [2017-09-22 09:12:04,047] ERROR {org.apache.synapse.mediators.builtin.LogMediator} - Error: To = /services/XXXX-ws_PS/payments/validate, IP = 172.22.222.156, RequestID = urn:uuid:0df0d876-6093-4479-9537-70bbbdf4d649 {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2017-09-22 09:12:20,541] INFO {org.apache.synapse.mediators.builtin.LogMediator} - Request: Target URL = /services/XXXX-ws_PS/institutions/details/00239, Client IP = 172.22.222.156, Request ID = urn:uuid:e5aca458-03a6-41b3-81b5-d409c5eaa66a {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2017-09-22 09:12:20,542] WARN {org.apache.synapse.transport.passthru.ConnectCallback} - Connection refused or failed for : app03.XXXX.uat.XXX.com.ph:50209 {org.apache.synapse.transport.passthru.ConnectCallback}
TID: [-1234] [] [2017-09-22 09:12:20,542] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : AnonymousEndpoint with address https://app03.XXXX.uat.XXX.com.ph:50209/XXXX-ws/ will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2017-09-22 09:12:20,542] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : AnonymousEndpoint with address https://app03.XXXX.uat.XXX.com.ph:50209/XXXX-ws/ - last suspend duration was : 300000ms and current suspend duration is : 300000ms - Next retry after : Fri Sep 22 09:17:20 PHT 2017 {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2017-09-22 09:12:20,542] WARN {org.apache.synapse.endpoints.LoadbalanceEndpoint} - Endpoint [XXXX-ws_LBEP] Detect a Failure in a child endpoint : AnonymousEndpoint {org.apache.synapse.endpoints.LoadbalanceEndpoint}
TID: [-1] [] [2017-09-22 09:12:20,545] WARN {org.apache.synapse.transport.passthru.ConnectCallback} - Connection refused or failed for : app04.XXXX.uat.XXX.com.ph:50209 {org.apache.synapse.transport.passthru.ConnectCallback}
TID: [-1234] [] [2017-09-22 09:12:20,545] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : AnonymousEndpoint with address https://app04.XXXX.uat.XXX.com.ph:50209/XXXX-ws/ will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2017-09-22 09:12:20,546] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : AnonymousEndpoint with address https://app04.XXXX.uat.XXX.com.ph:50209/XXXX-ws/ - last suspend duration was : 300000ms and current suspend duration is : 300000ms - Next retry after : Fri Sep 22 09:17:20 PHT 2017 {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2017-09-22 09:12:20,546] WARN {org.apache.synapse.endpoints.LoadbalanceEndpoint} - Endpoint [XXXX-ws_LBEP] Detect a Failure in a child endpoint : AnonymousEndpoint {org.apache.synapse.endpoints.LoadbalanceEndpoint}
TID: [-1234] [] [2017-09-22 09:12:20,785] INFO {org.apache.synapse.mediators.builtin.LogMediator} - Response: To = /services/XXXX-ws_PS/institutions/details/00239, IP = 172.22.222.156, RequestID = urn:uuid:e5aca458-03a6-41b3-81b5-d409c5eaa66a {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2017-09-22 09:15:03,802] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-49847 Socket Timeout : 180000 Remote Address : h02610a24.ho.ad.XXX/172.22.222.156:58409 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [-1234] [] [2017-09-22 09:30:09,430] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: /, MessageID: urn:uuid:d233e911-f717-4d42-a24c-eca0e7c32768, Direction: request, Envelope: <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"><soapenv:Body/></soapenv:Envelope> {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2017-09-22 09:47:24,248] ERROR {org.apache.synapse.transport.passthru.SourceHandler} - I/O error: Unrecognized SSL message, plaintext connection? {org.apache.synapse.transport.passthru.SourceHandler}
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.security.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:156)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:868)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:245)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:280)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:410)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:119)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:159)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:338)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:586)
at java.lang.Thread.run(Thread.java:745)
TID: [-1234] [] [2017-09-22 10:00:14,936] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: /, MessageID: urn:uuid:d5e87ee5-d83b-4ded-8110-b64dd01ce08f, Direction: request, Envelope: <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"><soapenv:Body/></soapenv:Envelope> {org.apache.synapse.mediators.builtin.LogMediator}
Proxy Code
<?xml version="1.0" encoding="UTF-8"?>
<proxy name="XXXX-ws_PS" startOnLoad="true" transports="https http" xmlns="http://ws.apache.org/ns/synapse">
<target>
<inSequence>
<property description="targetURL" expression="get-property('To')" name="targetURL" scope="default" type="STRING"/>
<property description="clientIP" expression="get-property('axis2','REMOTE_ADDR')" name="clientIP" scope="default" type="STRING"/>
<property description="requestID" expression="get-property('MessageID')" name="requestID" scope="default" type="STRING"/>
<log description="requestLog" level="custom">
<property expression="get-property('targetURL')" name="Request: Target URL"/>
<property expression="get-property('clientIP')" name="Client IP"/>
<property expression="get-property('requestID')" name="Request ID"/>
</log>
<send description="">
<endpoint key="XXXX-ws_LBEP"/>
</send>
</inSequence>
<outSequence>
<log description="resposeLog" level="custom">
<property expression="get-property('targetURL')" name="Response: To"/>
<property expression="get-property('clientIP')" name="IP"/>
<property expression="get-property('requestID')" name="RequestID"/>
</log>
<send/>
</outSequence>
<faultSequence>
<log category="ERROR" description="errorLog" level="custom">
<property expression="get-property('targetURL')" name="Error: To"/>
<property expression="get-property('clientIP')" name="IP"/>
<property expression="get-property('requestID')" name="RequestID"/>
</log>
</faultSequence>
</target>
</proxy>
Endpoint Code
<?xml version="1.0" encoding="UTF-8"?>
<endpoint name="XXXX-ws_LBEP" xmlns="http://ws.apache.org/ns/synapse">
<loadbalance algorithm="org.apache.synapse.endpoints.algorithms.RoundRobin">
<endpoint>
<http uri-template="https://app01.XXXX.uat.XXX.com.ph:50209/XXXX-ws/">
<timeout>
<duration>600000</duration>
<responseAction>fault</responseAction>
</timeout>
<suspendOnFailure>
<initialDuration>300000</initialDuration>
<progressionFactor>1.0</progressionFactor>
<maximumDuration>1800000</maximumDuration>
</suspendOnFailure>
<markForSuspension>
<retriesBeforeSuspension>3</retriesBeforeSuspension>
<retryDelay>30000</retryDelay>
</markForSuspension>
</http>
</endpoint>
<endpoint>
<http uri-template="https://app02.XXXX.uat.XXX.com.ph:50209/XXXX-ws/">
<timeout>
<duration>600000</duration>
<responseAction>fault</responseAction>
</timeout>
<suspendOnFailure>
<initialDuration>300000</initialDuration>
<progressionFactor>1.0</progressionFactor>
<maximumDuration>1800000</maximumDuration>
</suspendOnFailure>
<markForSuspension>
<retriesBeforeSuspension>3</retriesBeforeSuspension>
<retryDelay>30000</retryDelay>
</markForSuspension>
</http>
</endpoint>
<endpoint>
<http uri-template="https://app03.XXXX.uat.XXX.com.ph:50209/XXXX-ws/">
<timeout>
<duration>600000</duration>
<responseAction>fault</responseAction>
</timeout>
<suspendOnFailure>
<initialDuration>300000</initialDuration>
<progressionFactor>1.0</progressionFactor>
<maximumDuration>1800000</maximumDuration>
</suspendOnFailure>
<markForSuspension>
<retriesBeforeSuspension>3</retriesBeforeSuspension>
<retryDelay>30000</retryDelay>
</markForSuspension>
</http>
</endpoint>
<endpoint>
<http uri-template="https://app04.XXXX.uat.XXX.com.ph:50209/XXXX-ws/">
<timeout>
<duration>600000</duration>
<responseAction>fault</responseAction>
</timeout>
<suspendOnFailure>
<initialDuration>300000</initialDuration>
<progressionFactor>1.0</progressionFactor>
<maximumDuration>1800000</maximumDuration>
</suspendOnFailure>
<markForSuspension>
<retriesBeforeSuspension>3</retriesBeforeSuspension>
<retryDelay>30000</retryDelay>
</markForSuspension>
</http>
</endpoint>
</loadbalance>
</endpoint>

Answer 1

The use of 'targetURL' is not correct. Instead define the endpoint in the <send> So something like
<send>
<endpoint key="XXXX-ws_LBEP" />
</send>