Unhealthy Ingress services - google-compute-engine

I am trying to deploy an application via GKE. As far I created two services and two deployments for the front and the back for the App .
I created an ingress ressource using "gce" controller and I mapped the services as shown
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
labels:
app: app
part: ingress
name: my-irool-ingress
annotations:
kubernetes.io/ingress.class: "gce"
kubernetes.io/ingress.global-static-ip-name: my-ip
spec:
backend:
serviceName: client-svc
servicePort: 3000
rules:
- http:
paths:
- path: /back
backend:
serviceName: back-svc
servicePort: 9000
- http:
paths:
- path: /back/*
backend:
serviceName: back-svc
servicePort: 9000
It worked almost fine ( not all the root where mapped correctly but it worked). I added modification on the code ( only the code of the application ) and I rebuild the images and recreated the services, but the ingress seemed angry with the modifications I have added and
all my services became in the unhealthy state
This is the front service
apiVersion: v1
kind: Service
metadata:
labels:
app: app
part: front
name: client
namespace: default
spec:
type: NodePort
ports:
- nodePort: 32585
port: 3000
protocol: TCP
selector:
app: app
part: front
when I do a describe , I got nothing beside that my services are unhealthy.
And in the moment of creation I keep getting
Warning GCE 6m loadbalancer-controller
googleapi: Error 409: The resource
'[project/idproject]/global/healthChecks/k8s-be-32585--17c7......01'
already exists, alreadyExists
My question is:
What is wrong about the code showed above? Should I map all the services to the port 80 ( default ingress port so it could work? )
What are the readinessProbe and livenessProbe? Should I add them or mapping one to the services to default backend should be enough ?

For your first question, deleting and re-creating the ingress may resolve the issue. For the second question, you can review the full steps of configuring Liveness and Readiness probes here. Furthermore, as defined here (as an example for a pod):
livenessProbe: Indicates whether the Container is running. If the
liveness probe fails, the kubelet kills the Container, and the
Container is subjected to its restart policy. If a Container does not
provide a liveness probe, the default state is Success.
And readinessProbe: Indicates whether the Container is ready to
service requests. If the readiness probe fails, the endpoints
controller removes the Pod’s IP address from the endpoints of all
Services that match the Pod. The default state of readiness before the
initial delay is Failure. If a Container does not provide a readiness
probe, the default state is Success.

Related

How to use Traefik+MetalLB to Expose Kubernetes API (apiserver)

I have a microk8s running on my raspberry pi and I'm hoping to use a traefik ingressroute to expose kubernetes API to my subdomain
below is my ingressroute:
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: kube-api
spec:
entryPoints:
- web
- websecure
routes:
- match: Host(`kubernetes.mydomain.com`)
kind: Rule
services:
- kind: Service
name: kubernetes
port: 16443 # have also tried 443
tls:
secretName: kubernetes.mydomain.com
This works fine for my other services+ingressroute but not with the api.
For the kubernetes api I'm only able to see my certificate was successfully generated but the page just displays 'Internal Server Error'
Please let me know what additional information I can provide and I will gladly do so!
This issue was because traefik was trying to connect with kube-apiserver over https.
I had to use a serverTransport to allow insecure communication between traefik and kube-apiserver. This is not a security concern as communication to traefik will verify ssl.
The way to do this can be found at the very bottom of this page.
https://doc.traefik.io/traefik/v2.4/routing/providers/kubernetes-crd/#kind-serverstransport

Path based routing fails to match propertly with Traefik Ingress in Kubernetes

I have a kubernetes ingress for an application where i'm using path based routing.
The cluster is running on Google Cloud Kubernetes Engine and my ingress controller is Traefik v2.4.
Some of my links are:
https://www.kwetter.org/ -> Homepage (Frontend)
https://www.kwetter.org/profile -> Profile page (Frontend)
https://www.kwetter.org/messages-> Messages page (Frontend)
https://www.kwetter.org/api/auth/connect -> OAuth endpoints (IdentityServer)
https://www.kwetter.org/api/auth/users -> User endpoints (IdentityServer)
The logic that I want is to have anything matching the path /* going to the frontend, and anything matching /api/auth/* to be routed to identity server.
However, only exact paths are routed, https://www.kwetter.org/ works, https://www.kwetter.org/profile doesnt.
Same for the other service, https://www.kwetter.org/api/auth works, https://www.kwetter.org/api/auth/users doesn't.
My ingress looks like this:
kind: Ingress
apiVersion: networking.k8s.io/v1beta1
metadata:
name: traefik-ingress
annotations:
networking.gke.io/managed-certificates: kwetter-certificate
traefik.ingress.kubernetes.io/router.entrypoints: web,websecure
spec:
rules:
- host: kwetter.org
http:
paths:
- path: /
backend:
serviceName: kwetter-web-app
servicePort: 80
- path: /api/auth
pathType: Prefix
backend:
serviceName: kwetter-identity-server
servicePort: 80
- host: www.kwetter.org
http:
paths:
- path: /
backend:
serviceName: kwetter-web-app
servicePort: 80
- path: /api/auth
pathType: Prefix
backend:
serviceName: kwetter-identity-server
servicePort: 80
The page is loaded fine for the frontend, but the static files return a 404, with the traefik message "response 404 (backend NotFound), service rules for the path non-existent". The full url is https://kwetter.org/static/js/2.2217857e.chunk.js and with pathType: Prefix, this should match to the "/" path.
Can anybody tell me where i'm going wrong?
Edit for Solution:
I have tried the re-write target based solution, which conflicted with my API controllers at the service they reached.
Eventually I just tried to put a star in the path:
path: /*
path: /api/auth/*
This solved the whole routing issue, didn't know this was even possible.
The logic that I want is to have anything matching the path /* going to the frontend, and anything matching /api/auth/* to be routed to identity server.
You need to use regular expressions and rewrite-target annotation in your .yaml files. Look at the example Ingress .yaml file:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
name: rewrite
namespace: default
spec:
rules:
- host: rewrite.bar.com
http:
paths:
- backend:
serviceName: http-svc
servicePort: 80
path: /something(/|$)(.*)
In this ingress definition, any characters captured by (.*) will be assigned to the placeholder $2, which is then used as a parameter in the rewrite-target annotation.
For example, the ingress definition above will result in the following rewrites:
rewrite.bar.com/something rewrites to rewrite.bar.com/
rewrite.bar.com/something/ rewrites to rewrite.bar.com/
rewrite.bar.com/something/new rewrites to rewrite.bar.com/new
You can find more information about rewrite-target annotation here.
You can find similar tips in the traefik documentation.
# Replace path with regex
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: test-replacepathregex
spec:
replacePathRegex:
regex: ^/foo/(.*)
replacement: /bar/$1
But in this situation you may notice some differences in yaml. If you want to create regular expressions for traefik, you can test the solution here.

Is there any way to control OpenShift routes for co-working with horizontal pod auto-scaler?

I'm using Horizontal Pod Autoscaler to scale my pods in an OpenShift environment. I have a web application running in pods. As the pod scales, I got an HTTP status code 404 error in the first few seconds of an HTTP request. Is this because routes is sending a request to a pod that is in the process of being launched? If so, is there any way to prevent the error? I've tried setting router.openshift.io/haproxy.health.check.interval to a small value, but I still can't avoid this error.
It seems you did not configure your readiness checks correctly. Check the documentation on how to add readiness and liveness checks to your Deployment.
A readiness probe determines if a container is ready to accept service requests.
A liveness probe determines if a container is still running.
In newer versions of OpenShift / Kubernetes there is now also the startupProbe, which may help you in your case.
Here is an example of a Deployment with a liveness and a readiness probe:
kind: Deployment
apiVersion: apps/v1
...
spec:
...
template:
spec:
containers:
- name: example
readinessProbe:
tcpSocket:
port: 8080
livenessProbe:
tcpSocket:
port: 8080
...

Static global IP on GKE using Nginx Ingress?

I am trying to setup an nginx ingress controller in my GKE cluster and I'd like to use a static global IP address but I am struggling figuring it out how.
After a lot of research, most guides/stackoverflow/blogs just say "use the kubernetes.io/ingress.global-static-ip-name annotation on your ingress resource" however that does not do anything.
Below is an example of my Ingress resource
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
namespace: my-namespace
annotations:
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.class: "nginx"
kubernetes.io/ingress.allow-http: "false"
nginx.org/websocket-services: "ws-svc"
kubernetes.io/ingress.global-static-ip-name: my-global-gce-ip
spec:
tls:
- secretName: my-secret
hosts:
- mysite.com
rules:
- host: mysite.com
http:
paths:
- path: /
backend:
serviceName: web
servicePort: 80
The service always get's an ephemeral IP address which is thrown away whenever I recreate the controller.
I suspect the issue at hand here is that annotation only works for GCE type of Ingress, not nginx (Even though this is stated nowhere)
Next I attempted setting the IP manually in my ingress resource as showsn in this guide yet when I look at the service created, the external IP address just shows as pending which some github issues seem to point is due to the fact that it is a global and not a regional IP.
With all this in mind, is there any way to have a static global ip on a GKE cluster using an nginx ingress controller?
You have to set the static IP as loadBalancerIP in nginx ingress controller, not in ingress-resource (as you did). As per the documentation, Load Balancer IP is the IP address to assign to load balancer (if supported).
https://github.com/helm/charts/tree/master/stable/nginx-ingress
spec:
...
externalTrafficPolicy: Cluster
loadBalancerIP: [your static IP]
sessionAffinity: None
type: LoadBalancer
And make sure your IP is regional and not global. Only GCP load balancers (GCP built-in ingress controller) support global IP.

ingress with both rules and default backend in Google Container Engine

I'm running on the Google Container Engine platform have an ingress that I would like to have a default backend service for almost all of my domains (there are quite a few, but have another, specific service for one domain on it. Going by my understanding of the ingress user guide (scan for "Default Backends:" in there), the config below should work correctly.
However, it doesn't ever create the second backend. Running kubectl describe ingress on the ingress made and when looking at the LB in the Google console site, only the first "default" backend service is listed. Changing the default one into a rule one fixes the problem but means I have to explicitly list all of the domains I want to support.
So, I'm assuming I have a bug in the config below. If so, what is it?
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: boringsites
spec:
backend:
serviceName: boringsites
servicePort: 80
tls:
- secretName: boringsites-tls
rules:
- host: subdomain.example.com
http:
paths:
- backend:
serviceName: other-svc
servicePort: 80
I just created https://gist.github.com/bprashanth/9f4533b19fd864b723ba0720a3648fa3#file-default-basic-yaml-L94 on kubernetes 1.3 and it works as expected. Perhaps you can debug backwards? Where are you running kube and what version are you using? There is a known and fixed race in 1.2 that you might be running into, especially if you updated the ingress. Also note that you need services of type=nodeport, or the ingress controller on gce will ignore the service you plugged into the reasource.