Random 502 Bad gateway error using Azure application Gateway with Traefik Ingress for MultiSite - kubernetes-ingress

I have created an Azure application gateway WAF V2 tier, which is connected to Traefik ingress controller in AKS and the ingress forward traffic to the applications in AKS.
The backend-pool point to Treafik ingress private IP and the Http Setting and Listener have been satup for multisite.
I have also sat up HealthProbe which is tested successfully.
The timeout has been increased to 180 seconds.
Sending a get requests to app host works fine for a few minutes then it starts to give 502 bad gateway response for some seconds and later on starts to give 200 ok reponse. So as you can see it is not stable.
The moment that it gives 502 error, Backend-health and HealthProbe works fine. There are not much info in logs which you can see it here:
Application Gateway (WAF) -> Traefik Ingress Controller -> services
AzureDiagnostics | where ResourceProvider == "MICROSOFT.NETWORK" and
Category has "ApplicationGatewayAccessLog" and httpStatus_d == 502
TimeGenerated [UTC]
2021-10-08T08:05:21.533Z
ResourceId
/SUBSCRIPTIONS/.../RESOURCEGROUPS/..../PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/MyAPPGATEWAY
Category
ApplicationGatewayAccessLog
ResourceGroup
RG-APP-SANDBOX
SubscriptionId
......
ResourceProvider
MICROSOFT.NETWORK
Resource
MyAPPGATEWAY
ResourceType
APPLICATIONGATEWAYS
OperationName
ApplicationGatewayAccess
requestUri_s
/
userAgent_s
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0
ruleName_s
vnet-sandbox-rqrt-https
httpMethod_s
GET
instanceId_s
appgw_0
httpVersion_s
HTTP/1.1
clientIP_s
......
host_s
hellopython-aks.mydomain.io
sslEnabled_s
on
clientPort_d
52260
httpStatus_d
502
receivedBytes_d
495
sentBytes_d
366
timeTaken_d
0.004
SourceSystem
Azure
timeStamp_t [UTC]
2021-10-08T08:04:42Z
transactionId_g
.....
listenerName_s
vnet-sandbox-listener-https
backendPoolName_s
vnet-sandbox-bp
backendSettingName_s
vnet-sandbox-hts-https
originalRequestUriWithArgs_s
/
sslCipher_s
ECDHE-RSA-.....
sslProtocol_s
TLSv1.2
sslClientVerify_s
NONE
serverRouted_s
MyIngressPrivateIP:443
serverStatus_s
502
serverResponseLatency_s
0.004
originalHost_s
hellopython-aks.mydomain.io
Type
AzureDiagnostics
_ResourceId
/subscriptions/...../resourcegroups/...../providers/microsoft.network/applicationgateways/myappgateway
Does anyone know why this happens and how I can fix it?

Related

Ingress rewrite string is being ignored

The requirement is to access the burger service in https://meals.food.com/burger2.
The context path within the app is /burger.
Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /burger/$2
spec:
rules:
- host: meals.food.com
http:
paths:
- backend:
service:
name: burger
port:
number: 80
path: /burger2(/|$)(.*)
pathType: Prefix
Upon checking the ingress controller logs:
[05/Jan/2022:13:54:11 +0000] "GET // HTTP/1.1" 304 0 "-" "Mozilla/5.0
(X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/96.0.4664.110 Safari/537.36" 957 0.002 [anotherservice-80] []
x.x.x.x:80 0 0.002 304 230200x023
Is my ingress config correct?
My suspicion is that something is altering the request between my request from the browser to ingress-controller.
Is my ingress config correct? My suspicion is that something is altering the request between my request from the browser to ingress-controller.
Your ingress config looks OK. I do not see any errors in it. He will act as follows:
The example address meals.food.com/burger2/blah-blah-blah will be rewirted to meals.food.com/burger/blah-blah-blah. If that was your intention then config is fine.
However you have got 304 HTTP code.
The HTTP 304 Not Modified client redirection response code indicates that there is no need to retransmit the requested resources. It is an implicit redirection to a cached resource. This happens when the request method is safe, like a GET or a HEAD request, or when the request is conditional and uses a If-None-Match or a If-Modified-Since header.
The equivalent 200 OK response would have included the headers Cache-Control, Content-Location, Date, ETag, Expires, and Vary.
In other words
When the browser receives a request, but does not know whether it has the latest version of a write, it sends a conditional validation request, communicating the last modified date and time to the server via the If-Modified-Since or If-None-Match header.
The server then checks these headers and determines if their values are the same. If so - the server will send back the HTTP 304 code and the browser will use the cached copy of the resource. If not, it means that the file has been modified, so the browser will save a new copy by sending HTTP 200 code.
In your case it looks as if someone tried to download the same (unchanged) resource multiple times and therefore got the code 304. If so, everything is fine.

Error 404 in Nginx reverse proxy with iframe

I've been trying to run a reverse proxy to manage a graph of Grafana in an iframe. I'm using Apache on port 80 with the html page, Grafana server on port 3000 and Nginx with the reverse proxy on 8081.
The problem is that the proxy gives me a 404 error when I try to access the contents of the iframe. To test everything, I tried a local Grafana server configuration and a remote server configuration with 2 different iframes. The one connected to the local Grafana gives me a generic error of Grafana and a 404 when uploading the file, the remote one only 404.
Obviously each link taken individually works if I insert it on the browser, and without using the proxy it works the same way on the iframe.
I am not an Nginx expert and there will probably be some errors in the configuration. As a configuration file I'm using the Nginx default.conf file, so I haven't touched the nginx.conf because I don't think I need it for this purpose.
index.html
<h4>grafana local</h4>
<iframe src="http://localhost:8081/grafana/d-solo/KXusIR0Mk/test?orgId=1&from=1606211757397&to=1606233357397&panelId=2" width="450" height="200" frameborder="0"></iframe>
<h4>grafana remote</h4>
<iframe src="http://localhost:8081/salvo/d-solo/s-kg75yZz/stazione-meteo?orgId=1&panelId=8&from=1606215708391&to=1606237308391" width="450" height="200" frameborder="0"></iframe>
default.conf
server {
listen 8081;
listen [::]:8081;
server_name 127.0.0.1;
location /grafana/ {
#rewrite (grafana\/).* d-solo/KXusIR0Mk/test?orgId=1&from=1606211757397&to=1606233357397&panelId=2 break;
proxy_pass http://localhost:3000/;
}
location /salvo/ {
#rewrite (salvo\/).* d-solo/s-kg75yZz/stazione-meteo?orgId=1&panelId=8&from=1606215708391&to=1606237308391 break;
proxy_pass http://<ip_remoteGrafana>:<port>/;
}
}
PS: the ip and the port of the remote Grafana are not mine so I removed them, but in the code they are there ;)
access.log
::1 - - [26/Nov/2020:14:17:41 +0100] "GET /salvo/d-solo/s-kg75yZz/stazione-meteo?orgId=1&panelId=8&from=1606215708391&to=1606237308391 HTTP/1.1" 302 29 "http://localhost/grafana/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"
::1 - - [26/Nov/2020:14:17:41 +0100] "GET /login HTTP/1.1" 404 197 "http://localhost/grafana/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"
error.log
2020/11/26 14:17:41 [error] 101110#101110: *1 open() "/usr/share/nginx/html/login" failed (2: No such file or directory), client: ::1, server: 127.0.0.1, request: "GET /login HTTP/1.1", host: "localhost:8081", referrer: "http://localhost/grafana/"
What is this due to?
Apparently proxied Grafana needs authentication (redirect to /login), which is not compatible with your context path (it must be /salvo/login in your case). You will need redirect/content rewrite, which fill update all path to your used context /salvo/. That's complicated and not very reliable.
But you may somehow do that, but then you may discover that target Grafana doesn't allow to be embedded in the iframe or that Grafana cookies config (e.g. samesite, secure) may be also problem for your http app or Grafana auth is a problem. I would work with simple / context first to discover issues early and later also with more complicated context paths e.g. /salvo/.

Creating NGSI Source subscription in wirecloud mashup

I must admit I'm missing some points in creating an NGSI Source subscription for a wirecloud mashup. I tried the Wirecloud Basic History Info tutorial, tried my own deployed instance of Orion CB, tried creating subscription for both Manchester and Eindhoven cities' orion endpoint but nothing actually work: Here's the attempts I made for each case:
Wirecloud User Guide Basic Tutorial (History Info -settings created as described here):
NGSI server URL: http://orion.lab.fiware.org:1026/
NGSI proxy URL: https://ngsiproxy.lab.fiware.org
Use the FIWARE credentials of the user: Enabled
Use the FIWARE credentials of the workspace owner: Disabled
NGSI tenant/service: Emtpy
NGSI scope: /
NGSI entity types: Node, AMMS, Regulator
Id pattern: Empty
Monitored NGSI attributes: Latitud, Longitud, presence, batteryCharge, illuminance, ActivePower, ReactivePower, electricPotential, electricalCurrent
Unfortunately, after adding the NGSI Entity to PoI and creating settings as described in the Guide, PoI are not displayed on map as it is on the guide. Instead, I get the following error (NGSI Source operator logs):
Error creating subscription in the context broker server: Unexpected response from WireCloud's proxy
2. Own instance of Orion CB:
Added the following settings to the NGSI Source operator (NB: my orion instance is hosted by server ip 193.136.xx.xx):
NGSI server URL: http://193.136.xx.xx:53149/
NGSI proxy URL: https://ngsiproxy.lab.fiware.org
FIWARE-Service: urbansense
FIWARE-SERvicePath: /basic
NGSI Entity types: WeatherObserved
Monitored NGSI Attributes: location, temperature, relativeHumidity
I then get following error as well:
Error creating subscription in the context broker server: Connection Error
What exactly am i doing wrong here please?
EDIT: Using own instance of the NGSI-proxy:
NGSI server ULR http://193.136.29.17:53149/
NGSI proxy URL: http://127.0.0.1:3000
FIWARE-Service urbansense
FIWARE-ServicePath /basic
NGSI entity types WeatherObserved
Monitored NGSI attributes location, temperature, relativeHumidity
Howver, the error still remains:
Error creating subscription in the context broker server: Connection Error
Checking entities/attributes availability:
$curl -H 'fiware-service: urbansense' -H 'fiware-servicepath: /basic' http://193.136.29.17:53149/v2/entities
[{"id":"001","type":"WeatherObserved","dateObserved":{"type":"Text","value":"2014-06-05T21:47:06+01:00","metadata":{}},"illuminance":{"type":"Number","value":2.80323,"metadata":{}},"location":{"type":"StructuredValue","value":{"coordinates":[-8.594765,41.178296]},"metadata":{}},"name":{"type":"Text","value":"FEUP I322 Shannon","metadata":{}},"relativeHumidity":{"type":"None","value":null,"metadata":{}},"temperature":{"type":"None","value":null,"metadata":{}}},{"id":"003","type":"AirQualityObserved","O3":{"type":"Number","value":2.48661,"metadata":{}},"dateObserved":{"type":"Text","value":"2014-06-05T20:34:06+01:00","metadata":{}},"location":{"type":"StructuredValue","value":{"coordinates":[-8.594765,41.178296]},"metadata":{}},"name":{"type":"Text","value":"FEUP I322 Shannon","metadata":{}},"particles":{"type":"None","value":null,"metadata":{}}},{"id":"002","type":"NoiseLevelObserved","dateObserved":{"type":"Text","value":"2016-05-27T07:28:44+01:00","metadata":{}},"location":{"type":"StructuredValue","value":{"coordinates":[-8.590856,41.162781]},"metadata":{}},"measurand":{"type":"Number","value":67.7,"metadata":{}},"name":{"type":"Text","value":"Testbed Velasquez","metadata":{}},"sonometerClass":{"type":"Number","value":1,"metadata":{}}}]
NGSI-proxy log:
$ ngsi-proxy
ngsi-proxy server listening on port 3000
::ffff:127.0.0.1 - - [23/May/2019:10:58:26 +0000] "OPTIONS /eventsource HTTP/1.1" 204 - "http://193.136.xx.xx:53152/arilwan/urbansense-history-info" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36"
Created connection with id: b0eb3540-7d49-11e9-8abb-f5559f5e609e
::ffff:127.0.0.1 - - [23/May/2019:10:58:26 +0000] "POST /eventsource HTTP/1.1" 201 135 "http://193.136.xx.xx:53152/arilwan/urbansense-history-info" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36"
::ffff:127.0.0.1 - - [23/May/2019:10:58:26 +0000] "OPTIONS /callbacks HTTP/1.1" 204 - "http://193.136.xx.xx:53152/arilwan/urbansense-history-info" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36"
Created callback with id: b1039f40-7d49-11e9-8abb-f5559f5e609e
::ffff:127.0.0.1 - - [23/May/2019:10:58:26 +0000] "POST /callbacks HTTP/1.1" 201 131 "http://193.136.xx.xx:53152/arilwan/urbansense-history-info" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36"
::ffff:127.0.0.1 - - [23/May/2019:10:58:57 +0000] "OPTIONS /callbacks/b1039f40-7d49-11e9-8abb-f5559f5e609e HTTP/1.1" 204 - "http://193.136.xx.xx:53152/arilwan/urbansense-history-info" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36"
Deleting callback b1039f40-7d49-11e9-8abb-f5559f5e609e
Deleted callback with id: b1039f40-7d49-11e9-8abb-f5559f5e609e
::ffff:127.0.0.1 - - [23/May/2019:10:58:57 +0000] "DELETE /callbacks/b1039f40-7d49-11e9-8abb-f5559f5e609e HTTP/1.1" 204 - "http://193.136.xx.xx:53152/arilwan/urbansense-history-info" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36"
NGINX error log(nginx_logs/error.log):
2019/05/23 10:57:04 [warn] 6#6: *171 upstream sent more data than specified in "Content-Length" header while reading upstream, client: 193.136.38.232, server: , request: "GET /showcase/media/CoNWeT/map-viewer/2.6.2/index.html?entrypoint=true&v=31cb4ca4c8751f6d04c4242d0b52b176a2c7bc9b&theme=wirecloud.defaulttheme HTTP/1.1", upstream: "http://192.168.224.10:8000/showcase/media/CoNWeT/map-viewer/2.6.2/index.html?entrypoint=true&v=31cb4ca4c8751f6d04c4242d0b52b176a2c7bc9b&theme=wirecloud.defaulttheme", host: "193.136.xx.xx:53152", referrer: "http://193.136.xx.xx:53152/arilwan/urbansense-history-info"
2019/05/23 10:57:34 [error] 6#6: *179 upstream prematurely closed connection while reading response header from upstream, client: 193.136.38.232, server: , request: "POST /cdp/http/193.136.xx.xx:53149/v2/subscriptions HTTP/1.1", upstream: "http://192.168.224.10:8000/cdp/http/193.136.xx.xx:53149/v2/subscriptions", host: "193.136.xx.xx:53152", referrer: "http://193.136.xx.xx:53152/arilwan/urbansense-history-info"
You're not making anything wrong, provided that on the chosen Context Broker, with that tenant ID (FIWARE-Service) and on that Service Path, there exist such Entity Types.
But the connection error is not due to your Context Broker instances (your own one or the public instance on FIWARE lab), but to the NGSI-proxy. The FIWARE Lab instance of NGSI-proxy (https://ngsiproxy.lab.fiware.org) seems not to be working properly. While it is reported and respawned, you can try your own instancce of an NGSI-proxy by following instructions here: https://github.com/conwetlab/ngsi-proxy.
Bear in mind that the NGSI-Source operator (I assume you're using it) uses the NGSI-Proxy for the reception of the notifications on subscriptions (the operator subscribes for any change in the monitored NGSI Attributes), and since the proxy is not working properly, the subscription fails.
Best regards

How to debug JSON binding errors in REST

I have a Spring MVC application with a few REST interfaces exposed and is running on an Apache Tomcat instance.
I'm wondering if there are any ways to troubleshoot the data binding errors when trying to POST to an end point. So when I post some JSON I would like to find out why the data binding is failing (when Spring converts this to an object at the Controller). I might have to do some config changes to the apache server to increase the debugging level to show more info or increase the logging level in my application (log4j) to display any errors that Spring throws out. Wondering if anyone can help me find ways to debug databinding issues.
Here is the request/response log from the Chrome's dev console for the failed request. If I simplify the JSON structure, it does seem to bind but I'd like to find out which specific JSON attributes are causing the problem.
Request URL:http://localhost:8080/entryrive-service/user/6/entry/
Request Method:POST
Status Code:400 Bad Request
Request Headersview source
Accept:application/json, text/plain, */*
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Content-Length:667
Content-Type:application/json;charset=UTF-8
Host:localhost:8080
Origin:http://localhost:8383
Referer:http://localhost:8383/entryrive-client/app/index.html
User-Agent:Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
Request Payloadview source
{id:null, title:Sprouts Farmers Market, entryDate:1388559600000, entryPayerId:[6], notes:test,…}
entryCreaterId: null
entryDate: 1388559600000
entryItemEntry: []
entryPayerId: [6]
entrySimpleEntry: {id:null, itemDescription:null, simpleUserIdAndLiableCost:[,…]}
entryTotal: "30"
groupId: 3
id: null
notes: "test"
title: "Sprouts Farmers Market"
userId: null
Response Headersview source
Access-Control-Allow-Credentials:true
Access-Control-Allow-Origin:http://localhost:8383
Connection:close
Content-Language:en
Content-Length:968
Content-Type:text/html;charset=utf-8
Date:Sun, 05 Jan 2014 19:50:05 GMT
Server:Apache-Coyote/1.1
Increasing the logging level of Spring framework did seem to show up what Spring has been upto
log4j.logger.org.springframework=ALL

Unable to tunnle to vmc mysql service due to memory limitation

I am unable to tunnle to my free hosted instance of a rails app on cloudfoundry inftrastructure.
When I run 'vmc tunnel mysql-service', I get the below:
1: none
2: mysql
3: mysqldump
Which client would you like to start?> 2
Opening tunnel on port 10000... FAILED
CFoundry::AccountNotEnoughMemory: 600: Not enough memory capacity, you're allowed: 2048M
For more information, see ~/.vmc/crash
Checking the ~/.vmc/crash logs I see:
Time of crash:
2013-03-13 18:16:54 -0400
CFoundry::AccountNotEnoughMemory: 600: Not enough memory capacity, you're allowed: 2048M
<<<
REQUEST: PUT https://api.cloudfoundry.com/apps/caldecott
REQUEST_HEADERS:
Authorization : bearer eyJhbGciOiJSUzI1NiJ9.eyJleHAiOjEzNjM4MTc3OTgsInVzZXJfbmFtZSI6ImhzdWVpbmczQGdtYWlsLmNvbSIsInNjb3BlIjpbImNsb3VkX2NvbnRyb2xsZXIucmVhZCIsIm9wZW5pZCIsInBhc3N3b3JkLndyaXRlIl0sImVtYWlsIjoiaHN1ZWluZzNAZ21haWwuY29tIiwiYXVkIjpbIm9wZW5pZCIsImNsb3VkX2NvbnRyb2xsZXIiLCJwYXNzd29yZCJdLCJqdGkiOiJkMzZjNDI3MS02ZDJkLTRjN2EtOThmYS1kNzc2MjhiZDFiNmMiLCJ1c2VyX2lkIjoiODY0OWZkMzEtY2JiNy00N2YyLTkyNmItODM5Y2MzNWFlMTlmIiwiY2xpZW50X2lkIjoidm1jIn0.Lt1Bw7mBP55Hi9MIPTn90s0RXkJcJwGZXZcqDep4BBnnwjrAOAPQPGlIwBA-Ovy9K5BazMXqnQCOv8kxpK8o4wo3vG6RAJPvF7p76JgZDq0C_n_PUV1LaxGrldnpc2PLawR0FHHChb7tKCJP4cf26lK8A8vg5GEwi8HWO5OJCERI-3CKKiGJB5mVj2rWGmE39-ihAWmT5LpS5jAEZ-XVvo4VDEKknJ8SQC6693FzdCZ2AJBHkAgNxRoCsBtvkxOgKkspI-IkcaMZx884BT24cGbseZ5XY3bj6ZjAb499AfbIFe97Hme4axtpWo8qn1grkrJxyI3gmYAVMHVgo1M1IQ
Content-Length : 310
Content-Type : application/json
REQUEST_BODY: {"name":"caldecott","instances":1,"state":"STARTED","staging":{"model":"sinatra","stack":"ruby19"},"resources":{"memory":64,"disk":2048,"fds":256},"env":["CALDECOTT_AUTH=43ae7176-67f6-41ac-8cff-bf21b4249a49"],"uris":["caldecott-d9149.cloudfoundry.com"],"services":["mysql-service"],"console":null,"debug":null}
RESPONSE: [403]
RESPONSE_HEADERS:
cache-control : no-cache
connection : keep-alive
content-type : application/json; charset=utf-8
date : Wed, 13 Mar 2013 22:16:54 GMT
keep-alive : timeout=20
server : nginx
transfer-encoding : chunked
x-ua-compatible : IE=Edge,chrome=1
RESPONSE_BODY:
{
"code": 600,
"description": "Not enough memory capacity, you're allowed: 2048M"
}
>
cfoundry-0.5.2/lib/cfoundry/baseclient.rb:156:in handle_error_response'
cfoundry-0.5.2/lib/cfoundry/baseclient.rb:135:inhandle_response'
cfoundry-0.5.2/lib/cfoundry/baseclient.rb:85:in request'
cfoundry-0.5.2/lib/cfoundry/baseclient.rb:74:input'
cfoundry-0.5.2/lib/cfoundry/v1/model_magic.rb:55:in block (2 levels) in define_client_methods'
cfoundry-0.5.2/lib/cfoundry/v1/model.rb:91:inupdate!'
cfoundry-0.5.2/lib/cfoundry/v1/app.rb:131:in update!'
cfoundry-0.5.2/lib/cfoundry/v1/app.rb:121:instart!'
tunnel-vmc-plugin-0.2.2/lib/tunnel-vmc-plugin/tunnel.rb:173:in start_helper'
tunnel-vmc-plugin-0.2.2/lib/tunnel-vmc-plugin/tunnel.rb:89:increate_helper'
tunnel-vmc-plugin-0.2.2/lib/tunnel-vmc-plugin/tunnel.rb:28:in open!'
tunnel-vmc-plugin-0.2.2/lib/tunnel-vmc-plugin/plugin.rb:41:inblock in tunnel'
interact-0.5.2/lib/interact/progress.rb:98:in with_progress'
tunnel-vmc-plugin-0.2.2/lib/tunnel-vmc-plugin/plugin.rb:40:intunnel'
mothership-0.5.1/lib/mothership/base.rb:66:in run'
mothership-0.5.1/lib/mothership/command.rb:72:inblock in invoke'
What actions should I take to resolve this?
To offer further background below are a few details about the env. my app is running in:
vmc stats logoff
Using manifest file manifest.yml
Getting stats for logoff... OK
instance cpu memory disk
0 0.1% 74.2K of 2G 63.3M of 2G
vmc env logoff
Using manifest file manifest.yml
Getting env for logoff... OK
vmc services
Getting services... OK
name service version
mysql-service mysql 5.1
This is because you have used all of your allotted 2Gb of RAM. To tunnel to a service, vmc needs to deploy a small Ruby application called Caldecott, this uses 64Mb. So in short, you need to free up 64Mb!