Ingress creates location / (root path match all) in nginx config on specific servername, avoiding path to be found on default servername - kubernetes-ingress

I am running into this problem (or behave) of Ingress I can't understand:
If I have a path: /api1 configed in ingess without a specified host name, like this:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: api1
namespace: somenamespace
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/proxy-body-size: 100m
nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "300"
spec:
rules:
- http:
paths:
- path: /api1
pathType: ImplementationSpecific
backend:
serviceName: service-for-api1
servicePort: 1234
Then I can access this path in any hostname or IP address as long as it points to my server, e.g. my-server.com/api1 or 1.2.3.4/api1
And If I added another ingress config with a specified host name, like: /api2
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: api2
namespace: somenamespace
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/proxy-body-size: 100m
nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "300"
spec:
rules:
- host: my-server.com # Only difference is this one has a host field
http:
paths:
- path: /api2
pathType: ImplementationSpecific
backend:
serviceName: service-for-api2
servicePort: 1234
Then I can access my-server.com/api2 all right, But when accessing my-server.com/api1, I got a 404 not found error. If try to access through IP, 1.2.3.4/api1, It works all right.
I read the nginx config generated by ingress, found out when no host is specified in ingress config, the path rule (/api1) will be added to servername _ which is the default server of nginx. And once a hostname is provided in ingress config, It (/api2) will be added to servername hostname block in nginx configfile. Also there will be a location / rule to match all path in this hostname, preventing all paths fallback to default server. So when access my-server.com/api2, the route is okey but when access my-server.com/api1, It matches location / in servername my-server.com block, preventing default server functioning at all.
Why do ingress do this??
I was believing not specify any hostname in ingress config means host: *, thus any host can match the rule. But it behaves like only hostname unknown to ingress can match host: *.
The nginx config I looked at is like this (forgive my for only copying part of it):
## start server _
server {
server_name _ ;
listen 80 default_server reuseport backlog=65535 ;
listen [::]:80 default_server reuseport backlog=65535 ;
listen 443 default_server reuseport backlog=65535 ssl http2 ;
listen [::]:443 default_server reuseport backlog=65535 ssl http2 ;
set $proxy_upstream_name "-";
...
location /api1 {
...
}
...
}
## end server _
## start server my-server.com
server {
server_name my-server.com
listen 80 ;
listen [::]:80 ;
listen 443 ssl http2 ;
listen [::]:443 ssl http2 ;
set $proxy_upstream_name "-";
location /api2 {
....
}
location / {
set $namespace "somenamespace";
set $ingress_name "service-for-api2";
set $service_name "";
set $service_port "";
set $location_path "/";
set $global_rate_limit_exceeding n;
...
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0;
proxy_next_upstream_tries 3;
...
}
}
## end server my-server.com
When copying I noticed the "try the next upstream server before returning an error", Is it not work properly so I got a 404 instead of actually trying servername _ ? When I looked the nginx log, it said the 404 request is sent to service [upstream-default-backend], I don't know what it means.
After I discover this problem I deleted the specified hostname in /api2 and it work allright. But I jsut can't help wondering why ingress does this behave...

Related

AKS Ingress Nginx external IP unreachable

I m using helm charts to deploy an app in Azure Kubernetes Service with Ingress-nginx
My helm charts used to work fine last month, but when I created something new with my charts, the public IP of my ingress nginx is not working anymore.
PS C:\Zetaris\HelmDeployment> kubectl get ingress -n zetaris
NAME CLASS HOSTS ADDRESS PORTS AGE
xx-gui-nginx <none> uisaas.xxxxxxxxxxxxxx 20.167.72.8 80, 443 53m
xx-rest-nginx <none> restsaas.xxxxx 20.167.72.8 80, 443 55m
Here is my yaml:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: lightning-gui-nginx
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: letsencrypt-enterprise
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "server: hide";
more_set_headers "X-Content-Type-Options: nosniff";
more_set_headers "X-Frame-Options: DENY";
more_set_headers "X-Xss-Protection: 1";
more_set_headers "referrer-policy: no-referrer";
more_set_headers "Content-Security-Policy: frame-ancestors 'none'";
spec:
tls:
- hosts:
- {{ .Values.ingress.guiurl }}
secretName: tls-secret-gui
rules:
- host: {{ .Values.ingress.guiurl }}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: xx-gui-svc
port:
number: 80
---
apiVersion: v1
kind: Service
metadata:
name: xx-gui-svc
namespace: {{ .Values.namespace }}
spec:
ports:
- port: 80
targetPort: 9001
type: ClusterIP
selector:
app: {{ .Values.deployment.name }}
I can reach my app on port 9001 if I port-forward directly my pod.
The error is coming from the IP:
telnet 20.167.72.8 80
Connecting To 20.167.72.8...Could not open connection to the host, on port 80: Connect failed
As far as i know, there no IP restrictions or firewalls in AKS.
Do you have any idea on why would my ingress IP would not be reachable on internet?
Tried to create another app with same helm chart, uninstall and reinstall ingress service, public IP is never reachable.
Never mind I found the issue, needed to add:
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz"
When creating the ingress-nginx

Route request based on ssl_client_verify in Nginx ingress controller

Depending on the result of ssl_client_verify, I want to use a different upstream. So if $ssl_client_verify = SUCCESS route to yes-mtls-backend:80 K8s service, otherwise route to no-mtls-backend:80 K8s service. The following did not work:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: dynamic-upstream
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/auth-tls-secret: default/ca-cert
nginx.ingress.kubernetes.io/auth-tls-verify-client: optional
nginx.ingress.kubernetes.io/auth-tls-verify-depth: "3"
nginx.ingress.kubernetes.io/configuration-snippet: |
if ($ssl_client_verify = SUCCESS) {
set $proxy_host "default-yes-mtls-backend-80";
}
spec:
rules:
- host: xyz.com
http:
paths:
- backend:
service:
name: no-mtls-backend
port:
number: 80
path: /
pathType: Prefix
Note that auth-tls-verify-client is optional so that Nginx will not ask for a client certificate.
How to get this working? Is $proxy_host the wrong thing to set or this whole approach is flawed?
This worked:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: dynamic-upstream
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/auth-tls-secret: default/ca-cert
nginx.ingress.kubernetes.io/auth-tls-verify-client: optional
nginx.ingress.kubernetes.io/auth-tls-verify-depth: "3"
nginx.ingress.kubernetes.io/configuration-snippet: |
if ($ssl_client_verify = SUCCESS) {
set $service_name "yes-mtls-backend";
set $service_port "80";
set $proxy_upstream_name "default-yes-mtls-backend-80";
}
spec:
rules:
- host: xyz.com
http:
paths:
- backend:
service:
name: no-mtls-backend
port:
number: 80
path: /
pathType: Prefix
- host: xyz.com
http:
paths:
- backend:
service:
name: yes-mtls-backend
port:
number: 80
path: /
pathType: Prefix
In my version of Nginx, upstreams are dynamically loaded using Lua, so duplicating the rule will tell Lua to load yes-mtls-backend in the upstream block. That way, when overwriting $service_name, $service_port and $proxy_upstream_name there will be a backend to route traffic to.

redirect http to https using a kubernetes ingress controller on Amazon EKS

I need to configure a new listener from the Ingress.yaml manifest, currently the aws loadbalancer driver version is 2.4.4, when I perform the process from the AWS console it allows me to add the new listener and redirect the traffic to HTTPS without problem but after a few minutes it disappears, I perform the configuration directly in the ingress manifest with the annotations but the listener does not come out correctly in the AWS console.
Manifiest:
`
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":{ "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:xxx-xxxxx:xxxxx:certificate/xxxx-xxxxxxx
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/scheme: internal
kubernetes.io/ingress.class: alb
name: xxxxxx
namespace: xxxxx
spec:
rules:
- host: xxxxxxx
http:
paths:
- backend:
service:
name: service
port:
number: 80
path: /*
pathType: ImplementationSpecific
listener configured from maniefiest ingress
enter image description here
listener configuring it manually from AWS Console
enter image description here

Custom response for Ingress-Nginx External Authentication

I have deployed my Kubernetes cluster on EKS. I have an ingress-nginx which is exposed via load balancer to route traffic to different services. In ingress-nginx first request goes to auth service for authentication and if it is a valid request then I allow it to move forward. This is done using ingress-nginx annotation nginx.ingress.kubernetes.io/auth-url.
Auth service is developed using FastAPI. In case of 401 response from fastAPI look like this
FASTAPI
But when I use ingress-nginx the response look like this
INGRESS_NGINX
Is there a way to get JSON respone from Ingress-nginx?
Ingress File
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.class: 'nginx'
nginx.ingress.kubernetes.io/use-regex: 'true'
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/auth-response-headers: item_id
nginx.ingress.kubernetes.io/auth-method: POST
nginx.ingress.kubernetes.io/auth-url: http://pth-auth.default.svc.cluster.local:8000/item/1
# UPDATE THIS LINE ABOVE
spec:
rules:
- http:
paths:
- path: /?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: client-cluster-ip-service
servicePort: 3000
- path: /api/?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: server-cluster-ip-service
servicePort: 5000
- path: /pth-auth/?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: pth-auth
servicePort: 8000
Here's a solution that worked for me. It allows the auth service to return a custom error message for each request.
The caveat is that because nginx can't access auth response body, the pth-auth service needs to put the data in Pth-Auth-Error header (base64-encoded).
This example handles 401, 500, and a special case when pth-auth service is unavailable.
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.class: 'nginx'
nginx.ingress.kubernetes.io/use-regex: 'true'
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/auth-response-headers: item_id
nginx.ingress.kubernetes.io/auth-method: POST
nginx.ingress.kubernetes.io/auth-url: http://pth-auth.default.svc.cluster.local:8000/item/1
# UPDATE THIS LINE ABOVE
nginx.ingress.kubernetes.io/configuration-snippet: |
# Redirect auth errors to custom named locations
error_page 401 = #ingress_service_custom_error_401;
error_page 500 = #ingress_service_custom_error_500;
# Grab data from auth error response
auth_request_set $pth_auth_error $upstream_http_pth_auth_error;
auth_request_set $pth_auth_error_content_type $upstream_http_content_type;
auth_request_set $pth_auth_status $upstream_status;
nginx.ingress.kubernetes.io/server-snippet: |
location #ingress_service_custom_error_401 {
internal;
# Decode auth response header
set_decode_base64 $pth_auth_error_decoded $pth_auth_error;
# Return the error from pth-auth service if any
if ($pth_auth_error_decoded != ""){
add_header Content-Type $pth_auth_error_content_type always;
return 401 $pth_auth_error_decoded;
}
# Fall back to default nginx response
return 401;
}
location #ingress_service_custom_error_500 {
internal;
# Decode auth response header
set_decode_base64 $pth_auth_error_decoded $pth_auth_error;
# Return the error from pth-auth service if any
if ($pth_auth_error_decoded != ""){
add_header Content-Type $pth_auth_error_content_type always;
return 500 $pth_auth_error_decoded;
}
# Return a hardcoded error in case no pth-auth pods are available
if ($pth_auth_status = 503){
add_header Content-Type application/json always;
return 503 "{\"msg\":\"pth-auth service is unavailable\"}";
}
# Fall back to default nginx response
return 500;
}
spec:
rules:
- http:
paths:
- path: /?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: client-cluster-ip-service
servicePort: 3000
- path: /api/?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: server-cluster-ip-service
servicePort: 5000
- path: /pth-auth/?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: pth-auth
servicePort: 8000
Inspired by: https://stackoverflow.com/a/31485557/99237
Troubleshooting tips:
Here's the template nginx ingress uses when transforming the ingress annotations into nginx config file.
Connect to the ingress controller pod and look at /etc/nginx/nginx.conf to view the generated nginx config.
This worked for me, took reference from here
https://github.com/kubernetes/ingress-nginx/issues/2292
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.class: 'nginx'
nginx.ingress.kubernetes.io/use-regex: 'true'
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/auth-response-headers: item_id
nginx.ingress.kubernetes.io/auth-method: POST
nginx.ingress.kubernetes.io/auth-url: http://pth-auth.default.svc.cluster.local:8000/items/1
nginx.ingress.kubernetes.io/server-snippet: |
location = /error/401 {
proxy_method POST;
proxy_pass http://pth-auth.default.svc.cluster.local:8000/error/401;
}
location = /error/403 {
proxy_method POST;
proxy_pass http://pth-auth.default.svc.cluster.local:8000/error/403;
}
nginx.ingress.kubernetes.io/configuration-snippet: |
error_page 401 /error/401;
error_page 403 /error/403;
# UPDATE THIS LINE ABOVE
spec:
rules:
- http:
paths:
- path: /?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: client-cluster-ip-service
servicePort: 3000
- path: /api/?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: server-cluster-ip-service
servicePort: 5000
- path: /pth-auth/?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: pth-auth
servicePort: 8000
You just need to tell nginx in case of error route traffic to this location and their your function will handle specific errors. In my case, function is error/{error_code}.

How to make websocket working in kubernetes ingress?

I am dockering our current application and deploying on kubernetes cluster.
We have 2 services, namely, service-A and service-B. One of our services(example service-A) uses websocket. we have configured a rule in ingress to route the websocket request directly to service-A on port 8080. Also have a rule to route other requests to service-B on port 443. But ingress controller always route the websocket request to service-B instead of routing to service-A.
So, I removed the service-B rule from ingress, but still its routed as tls request and request never reaches service-A. Not sure why its rerouted as TLS instead of http request upgraded to websocket connection.
Please find my ingress configuration below:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
field.cattle.io/publicEndpoints: '[{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"cluster42:service-B","ingressName":"cluster42:my-ingress","hostname":"cluster42-phase-0 ","path":"/","allNodes":false},{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"cluster42:service-A","ingressName":"cluster42:my-ingress","hostname":"cluster42-phase-0 ","path":"/ws-service","allNodes":false}]'
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/rewrite-target":"/","nginx.ingress.kubernetes.io/ssl-passthrough":"true"},"labels":{"app":"ingress","chart":" myapplication-chart","heritage":"Tiller","release":"installation-cluster42"},"name":"my-ingress","namespace":"cluster42"},"spec":{"rules":[{"host":"cluster42-phase-0 ","http":{"paths":[{"backend":{"serviceName":"service-B","servicePort":443},"path":"/"}]}}],"tls":[{"hosts":["cluster42-phase-0 "]}]}}
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.org/websocket-services: "service-A"
creationTimestamp: "2019-05-24T11:46:40Z"
generation: 31
labels:
app: ingress
chart: myapplication-chart
heritage: Tiller
release: installation-cluster42
name: my-ingress
namespace: cluster42
resourceVersion: "57549362"
selfLink: /apis/extensions/v1beta1/namespaces/cluster42/ingresses/my-ingress
uid: 98784b1f-7e19-11e9-b2f1-005056b0b58e
spec:
rules:
- host: cluster42-phase-0
http:
paths:
- backend:
serviceName: service-B
servicePort: 443
path: /
- backend:
serviceName: service-A
servicePort: 8080
path: /ws-service
tls:
- hosts:
- cluster42-phase-0
status:
loadBalancer:
ingress:
- {}
I expect the request to be routed to service-A instead of service-B. Can you please let me know if I am missing something in my configuration or doing anything wrong.
Thanks in Advance.
Check if destination port are opened and not used.
Then check if you have enough right to access by tiller to kube-system namespace.
otherwise you have to create RBAC and special service.
More infromation you can find here: tiller-rbac.
You have some mistakes in your ingress configuration file in spec section.
Take notice that currently the Ingress only supports a single TLS port, 443, and assumes TLS termination. So it's obvious that destination service will be service B with port 443. So you can delete tls section from configuration file.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
field.cattle.io/publicEndpoints: '[{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"cluster42:service-B","ingressName":"cluster42:my-ingress","hostname":"cluster42-phase-0 ","path":"/","allNodes":false},{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"cluster42:service-A","ingressName":"cluster42:my-ingress","hostname":"cluster42-phase-0 ","path":"/ws-service","allNodes":false}]'
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/rewrite-target":"/","nginx.ingress.kubernetes.io/ssl-passthrough":"true"},"labels":{"app":"ingress","chart":" myapplication-chart","heritage":"Tiller","release":"installation-cluster42"},"name":"my-ingress","namespace":"cluster42"},"spec":{"rules":[{"host":"cluster42-phase-0 ","http":{"paths":[{"backend":{"serviceName":"service-B","servicePort":443},"path":"/"}]}}],"tls":[{"hosts":["cluster42-phase-0 "]}]}}
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.org/websocket-services: "service-A"
creationTimestamp: "2019-05-24T11:46:40Z"
generation: 31
labels:
app: ingress
chart: myapplication-chart
heritage: Tiller
release: installation-cluster42
name: my-ingress
namespace: cluster42
resourceVersion: "57549362"
selfLink: /apis/extensions/v1beta1/namespaces/cluster42/ingresses/my-ingress
uid: 98784b1f-7e19-11e9-b2f1-005056b0b58e
spec:
rules:
- host: cluster42-phase-0
http:
paths:
- path: /
backend:
serviceName: service-B
servicePort: 443
- path: /ws-service
backend:
serviceName: service-A
servicePort: 8080
status:
loadBalancer:
ingress:
- {}