I am attempting to use purely https with my compute engine. I have a network load balancer created that forwards to a pool with my instance in it. However, the pool has constantly failing health checks because it won't let me configure a health check that uses https.
I'm using apache to redirect 80 to 443. Does anyone know how to either create an https health check or have the http health check follow the redirect?
Thanks for any help.
--edit--
I finally came across some documentation at http://googlecloudplatform.blogspot.com/2015/07/Debugging-Health-Checks-in-Load-Balancing-on-Google-Compute-Engine.html.
Failure 5: Not answering directly with a 200 response code The web server may be configured to redirect to a page that returns an HTTP 200 response code. The health check will not follow the redirect; it expects the health check page to return a 200 directly.
This basic capability has been supported at every other hosting provider we've been on. Why can't this be done? What am I missing?
I spent the whole day trying to configure a purely https based load balancer in GCloud for a Kubernetes cluster with an ingress controller.
I finally got it working, so maybe I share my experience with people that struggle with the same configuration. If the health-check fails for the instances you will usually see the following accessing your websites URL.
Error: Server Error
The server encountered a temporary error and could not complete your request.
Please try again in 30 seconds.
1) Protocol: GCloud introduced new health checks which can be configured for HTTPS, SSLTCP, SSL, HTTP, HTTPS, or HTTP/2 probing. This can help the original problem to prevent a redirect from port 80 to port 443.
2) Path: The most common issue is a that the "/" path of your application will not return a 200 OK and thus let the health issue fail. This can be prevented by adding a path argument to your health check e.g. "/index".
3) Ingress HTTPS: This is relatively simple. Adding a secret or a pre-shared-cert to your ingress.yaml will automatically result in an HTTPS Load Balancer instead of HTTP. Further information to follow are here
Lastly, the guide from the docs for Setting up HTTP Load Balancing with Ingress .
However, even though the new HTTPS Health checks seem to work, they are still in the beta phase and bugs are reported in the issue tracker. The documentation for the gcloud-ingress-controller can be found here.
Related
Our load balancer is returning 502 errors for some requests. It is just a very low percentage of the total requests, we have around 36000 request per hour and about 40 errors per hour, so just a 0,01% of the requests returns an error.
The instances are healthy when the error occurs and we have added this forwarding rule to the firewall for the load balancer: 130.211.0.0/22 tcp:1-5000 Apply to all targets
It is not a very serious problem because the application tolerates such errors, but I would like to know why they are given.
Any help will be apreciated.
It seems that there are no an easy solution for this.
As Mike Fotinakis explains in this blog (thank you for this info JasonG :)):
It turns out that there is a race condition between the Google Cloud HTTP(S) Load Balancer and NGINX’s default keep-alive timeout of 65 seconds. The NGINX timeout might be reached at the same time the load balancer tries to re-use the connection for another HTTP request, which breaks the connection and results in a 502 Bad Gateway response from the load balancer.
In my case I'm using Apache with the mpm_prefork module. The solution proposed is to increase the connection keepalive timeout to 650s, but this is not possible because each connection opens one new process (so this would represent a great waste of resources).
UPDATE:
It seems that there are some new documentation about this problem on the official load balancer documentation page (search for "Timeouts and retries"): https://cloud.google.com/compute/docs/load-balancing/http/
They recommend to set the KeepAliveTimeout value to 620 in both cases (Apache and Nginx).
I had an issue w/ 502s that was unexplainable after recreating a load balancer and backend config. I recreated my backend & instance group for unmanaged instances and this seemed to fix the issue for me. I wasn't able to identify any issues in my configuration in GCP :(
But I had a lot more errors - 1/10. There are load balancer logs that will tell you what the cause is and docs explain the causes.
Eg mine were:
jsonPayload: { statusDetails: "failed_to_pick_backend" #type: "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry" }
If you're using nginx and it's on POSTS and the error is reported as "backend_connection_closed_before_data_sent_to_client" it may be fixed by changing your nginx timeouts. See this excellent blog post:
https://blog.percy.io/tuning-nginx-behind-google-cloud-platform-http-s-load-balancer-305982ddb340#.btzyusgi6
We are trying to test using the HTTP Load Balancing instead of the Network Load Balancing. When we try and go to http://beta.stubwire.com/ we get a 400 error saying we are speaking plain HTTP to an SSL-enabled server port but then if I go to https://beta.stubwire.com it works fine. I have included some screen shots of the load balancer below, is the error in here or is it the way the server is configured?
Finally able to figure this out. Under the "Backend services" of the load balancer you can click on Edit. Then you can select the different "Backend" that you wanted it to contain. On this we had specified "80, 443" as it mentioned we could list both. But we had a backend service created for secure and non secure pages. So only specifying the port that that service uses fixed that issue.
We're having significant issues with our http load balancer from certain IPs only.
I've seen a few other posts here about this. We've made sure the firewall is ok, I've even deleted and recreated the forwarding rules. Which is blasted annoying since the IP changes.
Still no joy. The problem only affects certain IP addresses - and if I post the same data to the IP of one of the servers, I have no problem.
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>502 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>
<h2></h2>
</body></html>
EDIT
We use cloudflare - usually this is actually disabled for this host however, I have just re-enabled it and now traffic is accepted again. Presumably since the traffic originates from a CF ip.
A 502 error is a "bad gateway" response. Have you checked the health check status of your instances at the time that the 502 errors are occurring?
You haven't mentioned whether you're running backends in more than one region. It's possible that your backends in one region are all being marked unhealthy at once, which is causing your failures.
Are your backend services using the default HTTP health check, or have you customized it? If the former, you might consider defining a more lenient health check for your backends (though this may mask actual application server failures). The default is to check backends every 5s on "/" with a 5s timeout, and require 2 consecutive failures or successes to change the state.
I've seen similar 502 errors when the backend service is too busy to handle the incoming traffic. Try adding more instances and it should go away.
However, I see this as a bandaid fix as this does not really solve the issue
I've configured a HTTP(S) Load balancer as per the documentation on https://cloud.google.com/compute/docs/load-balancing/http/
When I try to access the site via the Public IP address associated with the Load balancer. I'm getting a 502 response with the message:
Error: Server Error
The server encountered a temporary error and could not complete your request.
Please try again in 30 seconds.
I believe this is coming from the load balancer.
Anyone have any insight into what might be going on, what more I should be looking at?
I'm trying to set up an HTTPS load balancer for GKE using HTTPS L7 load balancer but for some reason is not working. Even the HTTP load balancer in the HTTP Load Balancing walkthrough. The forwarding rule's IP address is created and I'm able to ping and telnet to port 80. But when request via curl it give me a error.
<title>502 Server Error</title> </head> <body text=#000000
bgcolor=#ffffff> <h1>Error: Server Error</h1> <h2>The server
encountered a temporary error and could not complete your request.
<p>Please try again in 30 seconds.</h2> <h2></h2> </body></html>
All the steps were fine and I created a firewall without any tags for the ${NODE_PORT} but it didn't work.
Has anyone encountered this problem?
I had the same problem with my application, the problem is that we did not have an endpoint returning "Success" and the health checks were always failing.
It seems that the HTTP/HTTPS load balancer will not send the request to the cluster nodes if the health checks are not passing, so my solution was to create an endpoint that always returns 200 OK, and as soon as the health checks were passing, the LB started working.
I just walked through the example and (prior to opening up a firewall for $NODE_PORT) saw the same 502 error.
If you look in the cloud console at
https://console.developers.google.com/project/<project>/loadbalancing/http/backendServices/details/web-map-backend-service
you should see that the backend shows 0 out of ${num_nodes_in_cluster} as healthy.
For your firewall definition, make sure that you set the source filter to 130.211.0.0/22 to allow traffic from the the load balancing service and set the allowed protocols and ports to tcp:$NODE_PORT.
I use GKE, and I just walked through the example and it works fine, but when I route to my own service, it does not work. (my service is a rest api service)
I found that the biggest difference between my service and the example, is that: the example got a root endpoint("/"), but I do not support it.
So, I solved this problem in this way: add a root endpoint("/") to my service, and just return success(an empty endpoint that returns nothing), and then re-create the ingress, and waited for several minutes, and then the ingress works!!
I think this problem should be caused by healthy checker UNHEALTHY instances do not receive new connections.
Here is a link for Healthy checks: https://cloud.google.com/compute/docs/load-balancing/health-checks
The issue resolved after a few minutes (like 5-10 minutes) in my case.
If using an ingress, there may be additional information in the events relating to the ingress. To view these:
kubectl describe ingress example
In my case, the load balancer was returning this error because there was no web server running on my instances and instance-groups to handle the network request.
I installed nginx on all the machines and then it started working.
From now on, I made a point to add nginx in my startup script while creating the vm/instance.
If you are using nginx behind your loadbalancer then it's important that the default_server is returning 200 or some other 2**. That means that if you for example have a rewrite rule that returns 301 then it will fail.
The solution is to set default_server on your main server:
server {
# Rewrite calls to www
listen 443;
server_name example.com;
return 301 https://www.example.com$request_uri;
}
server {
listen 443 default_server;
server_name www.example.com;
...
Adding a Firewall Rule for Source: 130.211.0.0/22(the Load Balancer range on GCP) for the tcp:$NODEPORTIP fixed this for me.
I created an
endpoint to all request that contain 'GoogleHC' in the user-agent.
so,
server{
server_name example.com www.example.com
if ($http_user_agent ~* 'GoogleHC.*') {
return 200 'isaac newton';
}
}
Our server setup is the following:
a proxy and load balancer directs all the requests to its machines behind. The problem is, that these machines behind do not know where they are. If the proxy gets the request for
www.bridge.de/m01
he redirects to machine01.
Machine01 only knows its local path
m01
For an application solution for a password reset functionality I considered several opportunities.
We decided to pass the value of URL from 'before proxy' to the database of machine01. So machine01 'knows' its external context for that specific requests.
My question is: Is there a better way to pass external URL context to machines behind a proxy? We are using JavaEE, JSP and MySql for our application. Virtual machines running with CentOS.
Thanks for any suggestions! :D
Your question is not fully clear.
I assume you have the issue, what your load balancer terminates the connection and forwards you the request.
Usually your balancer provides you the origin URL of the request, since you may need it from time to time.
In this case you can check your http headers. If it is not provided, you have to reconfigure your balancer to provide you the needed details.
check this: Strategies for dealing with URIs when building an application that sits behind a reverse proxy