I m using helm charts to deploy an app in Azure Kubernetes Service with Ingress-nginx
My helm charts used to work fine last month, but when I created something new with my charts, the public IP of my ingress nginx is not working anymore.
PS C:\Zetaris\HelmDeployment> kubectl get ingress -n zetaris
NAME CLASS HOSTS ADDRESS PORTS AGE
xx-gui-nginx <none> uisaas.xxxxxxxxxxxxxx 20.167.72.8 80, 443 53m
xx-rest-nginx <none> restsaas.xxxxx 20.167.72.8 80, 443 55m
Here is my yaml:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: lightning-gui-nginx
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: letsencrypt-enterprise
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "server: hide";
more_set_headers "X-Content-Type-Options: nosniff";
more_set_headers "X-Frame-Options: DENY";
more_set_headers "X-Xss-Protection: 1";
more_set_headers "referrer-policy: no-referrer";
more_set_headers "Content-Security-Policy: frame-ancestors 'none'";
spec:
tls:
- hosts:
- {{ .Values.ingress.guiurl }}
secretName: tls-secret-gui
rules:
- host: {{ .Values.ingress.guiurl }}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: xx-gui-svc
port:
number: 80
---
apiVersion: v1
kind: Service
metadata:
name: xx-gui-svc
namespace: {{ .Values.namespace }}
spec:
ports:
- port: 80
targetPort: 9001
type: ClusterIP
selector:
app: {{ .Values.deployment.name }}
I can reach my app on port 9001 if I port-forward directly my pod.
The error is coming from the IP:
telnet 20.167.72.8 80
Connecting To 20.167.72.8...Could not open connection to the host, on port 80: Connect failed
As far as i know, there no IP restrictions or firewalls in AKS.
Do you have any idea on why would my ingress IP would not be reachable on internet?
Tried to create another app with same helm chart, uninstall and reinstall ingress service, public IP is never reachable.
Never mind I found the issue, needed to add:
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz"
When creating the ingress-nginx
Depending on the result of ssl_client_verify, I want to use a different upstream. So if $ssl_client_verify = SUCCESS route to yes-mtls-backend:80 K8s service, otherwise route to no-mtls-backend:80 K8s service. The following did not work:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: dynamic-upstream
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/auth-tls-secret: default/ca-cert
nginx.ingress.kubernetes.io/auth-tls-verify-client: optional
nginx.ingress.kubernetes.io/auth-tls-verify-depth: "3"
nginx.ingress.kubernetes.io/configuration-snippet: |
if ($ssl_client_verify = SUCCESS) {
set $proxy_host "default-yes-mtls-backend-80";
}
spec:
rules:
- host: xyz.com
http:
paths:
- backend:
service:
name: no-mtls-backend
port:
number: 80
path: /
pathType: Prefix
Note that auth-tls-verify-client is optional so that Nginx will not ask for a client certificate.
How to get this working? Is $proxy_host the wrong thing to set or this whole approach is flawed?
This worked:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: dynamic-upstream
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/auth-tls-secret: default/ca-cert
nginx.ingress.kubernetes.io/auth-tls-verify-client: optional
nginx.ingress.kubernetes.io/auth-tls-verify-depth: "3"
nginx.ingress.kubernetes.io/configuration-snippet: |
if ($ssl_client_verify = SUCCESS) {
set $service_name "yes-mtls-backend";
set $service_port "80";
set $proxy_upstream_name "default-yes-mtls-backend-80";
}
spec:
rules:
- host: xyz.com
http:
paths:
- backend:
service:
name: no-mtls-backend
port:
number: 80
path: /
pathType: Prefix
- host: xyz.com
http:
paths:
- backend:
service:
name: yes-mtls-backend
port:
number: 80
path: /
pathType: Prefix
In my version of Nginx, upstreams are dynamically loaded using Lua, so duplicating the rule will tell Lua to load yes-mtls-backend in the upstream block. That way, when overwriting $service_name, $service_port and $proxy_upstream_name there will be a backend to route traffic to.
I need to configure a new listener from the Ingress.yaml manifest, currently the aws loadbalancer driver version is 2.4.4, when I perform the process from the AWS console it allows me to add the new listener and redirect the traffic to HTTPS without problem but after a few minutes it disappears, I perform the configuration directly in the ingress manifest with the annotations but the listener does not come out correctly in the AWS console.
Manifiest:
`
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":{ "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:xxx-xxxxx:xxxxx:certificate/xxxx-xxxxxxx
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/scheme: internal
kubernetes.io/ingress.class: alb
name: xxxxxx
namespace: xxxxx
spec:
rules:
- host: xxxxxxx
http:
paths:
- backend:
service:
name: service
port:
number: 80
path: /*
pathType: ImplementationSpecific
listener configured from maniefiest ingress
enter image description here
listener configuring it manually from AWS Console
enter image description here
I have deployed my Kubernetes cluster on EKS. I have an ingress-nginx which is exposed via load balancer to route traffic to different services. In ingress-nginx first request goes to auth service for authentication and if it is a valid request then I allow it to move forward. This is done using ingress-nginx annotation nginx.ingress.kubernetes.io/auth-url.
Auth service is developed using FastAPI. In case of 401 response from fastAPI look like this
FASTAPI
But when I use ingress-nginx the response look like this
INGRESS_NGINX
Is there a way to get JSON respone from Ingress-nginx?
Ingress File
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.class: 'nginx'
nginx.ingress.kubernetes.io/use-regex: 'true'
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/auth-response-headers: item_id
nginx.ingress.kubernetes.io/auth-method: POST
nginx.ingress.kubernetes.io/auth-url: http://pth-auth.default.svc.cluster.local:8000/item/1
# UPDATE THIS LINE ABOVE
spec:
rules:
- http:
paths:
- path: /?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: client-cluster-ip-service
servicePort: 3000
- path: /api/?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: server-cluster-ip-service
servicePort: 5000
- path: /pth-auth/?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: pth-auth
servicePort: 8000
Here's a solution that worked for me. It allows the auth service to return a custom error message for each request.
The caveat is that because nginx can't access auth response body, the pth-auth service needs to put the data in Pth-Auth-Error header (base64-encoded).
This example handles 401, 500, and a special case when pth-auth service is unavailable.
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.class: 'nginx'
nginx.ingress.kubernetes.io/use-regex: 'true'
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/auth-response-headers: item_id
nginx.ingress.kubernetes.io/auth-method: POST
nginx.ingress.kubernetes.io/auth-url: http://pth-auth.default.svc.cluster.local:8000/item/1
# UPDATE THIS LINE ABOVE
nginx.ingress.kubernetes.io/configuration-snippet: |
# Redirect auth errors to custom named locations
error_page 401 = #ingress_service_custom_error_401;
error_page 500 = #ingress_service_custom_error_500;
# Grab data from auth error response
auth_request_set $pth_auth_error $upstream_http_pth_auth_error;
auth_request_set $pth_auth_error_content_type $upstream_http_content_type;
auth_request_set $pth_auth_status $upstream_status;
nginx.ingress.kubernetes.io/server-snippet: |
location #ingress_service_custom_error_401 {
internal;
# Decode auth response header
set_decode_base64 $pth_auth_error_decoded $pth_auth_error;
# Return the error from pth-auth service if any
if ($pth_auth_error_decoded != ""){
add_header Content-Type $pth_auth_error_content_type always;
return 401 $pth_auth_error_decoded;
}
# Fall back to default nginx response
return 401;
}
location #ingress_service_custom_error_500 {
internal;
# Decode auth response header
set_decode_base64 $pth_auth_error_decoded $pth_auth_error;
# Return the error from pth-auth service if any
if ($pth_auth_error_decoded != ""){
add_header Content-Type $pth_auth_error_content_type always;
return 500 $pth_auth_error_decoded;
}
# Return a hardcoded error in case no pth-auth pods are available
if ($pth_auth_status = 503){
add_header Content-Type application/json always;
return 503 "{\"msg\":\"pth-auth service is unavailable\"}";
}
# Fall back to default nginx response
return 500;
}
spec:
rules:
- http:
paths:
- path: /?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: client-cluster-ip-service
servicePort: 3000
- path: /api/?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: server-cluster-ip-service
servicePort: 5000
- path: /pth-auth/?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: pth-auth
servicePort: 8000
Inspired by: https://stackoverflow.com/a/31485557/99237
Troubleshooting tips:
Here's the template nginx ingress uses when transforming the ingress annotations into nginx config file.
Connect to the ingress controller pod and look at /etc/nginx/nginx.conf to view the generated nginx config.
This worked for me, took reference from here
https://github.com/kubernetes/ingress-nginx/issues/2292
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.class: 'nginx'
nginx.ingress.kubernetes.io/use-regex: 'true'
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/auth-response-headers: item_id
nginx.ingress.kubernetes.io/auth-method: POST
nginx.ingress.kubernetes.io/auth-url: http://pth-auth.default.svc.cluster.local:8000/items/1
nginx.ingress.kubernetes.io/server-snippet: |
location = /error/401 {
proxy_method POST;
proxy_pass http://pth-auth.default.svc.cluster.local:8000/error/401;
}
location = /error/403 {
proxy_method POST;
proxy_pass http://pth-auth.default.svc.cluster.local:8000/error/403;
}
nginx.ingress.kubernetes.io/configuration-snippet: |
error_page 401 /error/401;
error_page 403 /error/403;
# UPDATE THIS LINE ABOVE
spec:
rules:
- http:
paths:
- path: /?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: client-cluster-ip-service
servicePort: 3000
- path: /api/?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: server-cluster-ip-service
servicePort: 5000
- path: /pth-auth/?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: pth-auth
servicePort: 8000
You just need to tell nginx in case of error route traffic to this location and their your function will handle specific errors. In my case, function is error/{error_code}.
I am dockering our current application and deploying on kubernetes cluster.
We have 2 services, namely, service-A and service-B. One of our services(example service-A) uses websocket. we have configured a rule in ingress to route the websocket request directly to service-A on port 8080. Also have a rule to route other requests to service-B on port 443. But ingress controller always route the websocket request to service-B instead of routing to service-A.
So, I removed the service-B rule from ingress, but still its routed as tls request and request never reaches service-A. Not sure why its rerouted as TLS instead of http request upgraded to websocket connection.
Please find my ingress configuration below:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
field.cattle.io/publicEndpoints: '[{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"cluster42:service-B","ingressName":"cluster42:my-ingress","hostname":"cluster42-phase-0 ","path":"/","allNodes":false},{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"cluster42:service-A","ingressName":"cluster42:my-ingress","hostname":"cluster42-phase-0 ","path":"/ws-service","allNodes":false}]'
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/rewrite-target":"/","nginx.ingress.kubernetes.io/ssl-passthrough":"true"},"labels":{"app":"ingress","chart":" myapplication-chart","heritage":"Tiller","release":"installation-cluster42"},"name":"my-ingress","namespace":"cluster42"},"spec":{"rules":[{"host":"cluster42-phase-0 ","http":{"paths":[{"backend":{"serviceName":"service-B","servicePort":443},"path":"/"}]}}],"tls":[{"hosts":["cluster42-phase-0 "]}]}}
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.org/websocket-services: "service-A"
creationTimestamp: "2019-05-24T11:46:40Z"
generation: 31
labels:
app: ingress
chart: myapplication-chart
heritage: Tiller
release: installation-cluster42
name: my-ingress
namespace: cluster42
resourceVersion: "57549362"
selfLink: /apis/extensions/v1beta1/namespaces/cluster42/ingresses/my-ingress
uid: 98784b1f-7e19-11e9-b2f1-005056b0b58e
spec:
rules:
- host: cluster42-phase-0
http:
paths:
- backend:
serviceName: service-B
servicePort: 443
path: /
- backend:
serviceName: service-A
servicePort: 8080
path: /ws-service
tls:
- hosts:
- cluster42-phase-0
status:
loadBalancer:
ingress:
- {}
I expect the request to be routed to service-A instead of service-B. Can you please let me know if I am missing something in my configuration or doing anything wrong.
Thanks in Advance.
Check if destination port are opened and not used.
Then check if you have enough right to access by tiller to kube-system namespace.
otherwise you have to create RBAC and special service.
More infromation you can find here: tiller-rbac.
You have some mistakes in your ingress configuration file in spec section.
Take notice that currently the Ingress only supports a single TLS port, 443, and assumes TLS termination. So it's obvious that destination service will be service B with port 443. So you can delete tls section from configuration file.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
field.cattle.io/publicEndpoints: '[{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"cluster42:service-B","ingressName":"cluster42:my-ingress","hostname":"cluster42-phase-0 ","path":"/","allNodes":false},{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"cluster42:service-A","ingressName":"cluster42:my-ingress","hostname":"cluster42-phase-0 ","path":"/ws-service","allNodes":false}]'
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/rewrite-target":"/","nginx.ingress.kubernetes.io/ssl-passthrough":"true"},"labels":{"app":"ingress","chart":" myapplication-chart","heritage":"Tiller","release":"installation-cluster42"},"name":"my-ingress","namespace":"cluster42"},"spec":{"rules":[{"host":"cluster42-phase-0 ","http":{"paths":[{"backend":{"serviceName":"service-B","servicePort":443},"path":"/"}]}}],"tls":[{"hosts":["cluster42-phase-0 "]}]}}
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.org/websocket-services: "service-A"
creationTimestamp: "2019-05-24T11:46:40Z"
generation: 31
labels:
app: ingress
chart: myapplication-chart
heritage: Tiller
release: installation-cluster42
name: my-ingress
namespace: cluster42
resourceVersion: "57549362"
selfLink: /apis/extensions/v1beta1/namespaces/cluster42/ingresses/my-ingress
uid: 98784b1f-7e19-11e9-b2f1-005056b0b58e
spec:
rules:
- host: cluster42-phase-0
http:
paths:
- path: /
backend:
serviceName: service-B
servicePort: 443
- path: /ws-service
backend:
serviceName: service-A
servicePort: 8080
status:
loadBalancer:
ingress:
- {}