Custom response for Ingress-Nginx External Authentication - json

Question

I have deployed my Kubernetes cluster on EKS. I have an ingress-nginx which is exposed via load balancer to route traffic to different services. In ingress-nginx first request goes to auth service for authentication and if it is a valid request then I allow it to move forward. This is done using ingress-nginx annotation nginx.ingress.kubernetes.io/auth-url.
Auth service is developed using FastAPI. In case of 401 response from fastAPI look like this
FASTAPI
But when I use ingress-nginx the response look like this
INGRESS_NGINX
Is there a way to get JSON respone from Ingress-nginx?
Ingress File
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.class: 'nginx'
nginx.ingress.kubernetes.io/use-regex: 'true'
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/auth-response-headers: item_id
nginx.ingress.kubernetes.io/auth-method: POST
nginx.ingress.kubernetes.io/auth-url: http://pth-auth.default.svc.cluster.local:8000/item/1
# UPDATE THIS LINE ABOVE
spec:
rules:
- http:
paths:
- path: /?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: client-cluster-ip-service
servicePort: 3000
- path: /api/?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: server-cluster-ip-service
servicePort: 5000
- path: /pth-auth/?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: pth-auth
servicePort: 8000

Answer 1

Here's a solution that worked for me. It allows the auth service to return a custom error message for each request.
The caveat is that because nginx can't access auth response body, the pth-auth service needs to put the data in Pth-Auth-Error header (base64-encoded).
This example handles 401, 500, and a special case when pth-auth service is unavailable.
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.class: 'nginx'
nginx.ingress.kubernetes.io/use-regex: 'true'
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/auth-response-headers: item_id
nginx.ingress.kubernetes.io/auth-method: POST
nginx.ingress.kubernetes.io/auth-url: http://pth-auth.default.svc.cluster.local:8000/item/1
# UPDATE THIS LINE ABOVE
nginx.ingress.kubernetes.io/configuration-snippet: |
# Redirect auth errors to custom named locations
error_page 401 = #ingress_service_custom_error_401;
error_page 500 = #ingress_service_custom_error_500;
# Grab data from auth error response
auth_request_set $pth_auth_error $upstream_http_pth_auth_error;
auth_request_set $pth_auth_error_content_type $upstream_http_content_type;
auth_request_set $pth_auth_status $upstream_status;
nginx.ingress.kubernetes.io/server-snippet: |
location #ingress_service_custom_error_401 {
internal;
# Decode auth response header
set_decode_base64 $pth_auth_error_decoded $pth_auth_error;
# Return the error from pth-auth service if any
if ($pth_auth_error_decoded != ""){
add_header Content-Type $pth_auth_error_content_type always;
return 401 $pth_auth_error_decoded;
}
# Fall back to default nginx response
return 401;
}
location #ingress_service_custom_error_500 {
internal;
# Decode auth response header
set_decode_base64 $pth_auth_error_decoded $pth_auth_error;
# Return the error from pth-auth service if any
if ($pth_auth_error_decoded != ""){
add_header Content-Type $pth_auth_error_content_type always;
return 500 $pth_auth_error_decoded;
}
# Return a hardcoded error in case no pth-auth pods are available
if ($pth_auth_status = 503){
add_header Content-Type application/json always;
return 503 "{\"msg\":\"pth-auth service is unavailable\"}";
}
# Fall back to default nginx response
return 500;
}
spec:
rules:
- http:
paths:
- path: /?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: client-cluster-ip-service
servicePort: 3000
- path: /api/?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: server-cluster-ip-service
servicePort: 5000
- path: /pth-auth/?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: pth-auth
servicePort: 8000
Inspired by: https://stackoverflow.com/a/31485557/99237
Troubleshooting tips:
Here's the template nginx ingress uses when transforming the ingress annotations into nginx config file.
Connect to the ingress controller pod and look at /etc/nginx/nginx.conf to view the generated nginx config.

Answer 2

This worked for me, took reference from here
https://github.com/kubernetes/ingress-nginx/issues/2292
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.class: 'nginx'
nginx.ingress.kubernetes.io/use-regex: 'true'
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/auth-response-headers: item_id
nginx.ingress.kubernetes.io/auth-method: POST
nginx.ingress.kubernetes.io/auth-url: http://pth-auth.default.svc.cluster.local:8000/items/1
nginx.ingress.kubernetes.io/server-snippet: |
location = /error/401 {
proxy_method POST;
proxy_pass http://pth-auth.default.svc.cluster.local:8000/error/401;
}
location = /error/403 {
proxy_method POST;
proxy_pass http://pth-auth.default.svc.cluster.local:8000/error/403;
}
nginx.ingress.kubernetes.io/configuration-snippet: |
error_page 401 /error/401;
error_page 403 /error/403;
# UPDATE THIS LINE ABOVE
spec:
rules:
- http:
paths:
- path: /?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: client-cluster-ip-service
servicePort: 3000
- path: /api/?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: server-cluster-ip-service
servicePort: 5000
- path: /pth-auth/?(.*)
# UPDATE THIS LINE ABOVE
backend:
serviceName: pth-auth
servicePort: 8000
You just need to tell nginx in case of error route traffic to this location and their your function will handle specific errors. In my case, function is error/{error_code}.