k3s, traefik, awx-operator and ingress - k3s

I have a three node, on-prem, k3 cluster. I have installed AWX per these instructions: https://github.com/ansible/awx-operator
I can access the AWX login screen by executing this command on my laptop:
[red#mac.local ~]$ kubectl -n kube-system port-forward deployment/traefik 7080
Forwarding from 127.0.0.1:7080 -> 8052
Forwarding from [::1]:7080 -> 8052
Handling connection for 7080
...
And I can point my browser to http://localhost:7080 and I see the AWX login screen.
But I haven't a clue how to set up an ingress for AWX. What have I tried? A whole bunch of things and none of those have worked.
My latest attempt to create a ingress used this yaml:
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: awx-demo-ingress
# namespace: awx
spec:
rules:
- host: rocky
http:
paths:
- path: /awx
pathType: Exact
backend:
service:
name: awx-demo-service
port:
number: 80
This creates an ingress ...
kube-system awx-demo-ingress traefik localhost 10.0.3.51,10.0.3.52,10.0.3.53 80 12m
But rocky does not resolve to any IP address and if I point my browsers to http://10.0.3.51/ or http://10.0.3.51/awx results in 404 error.
I am close but I still got something wrong ...
$ kubectl describe ingress awx-demo-ingress -n kube-system
Name: awx-demo-ingress
Labels: <none>
Namespace: kube-system
Address: 10.0.3.51,10.0.3.52,10.0.3.53
Ingress Class: traefik
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
rocky-k3-1.XXXXXXXXX.com
/login awx-demo-service:80 (<error: endpoints "awx-demo-service" not found>)
Annotations: <none>
Events: <none>
I also tried this ...
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: awx-demo-ingress
namespace: awx
spec:
rules:
- host: rocky-k3-1.XXXXXXXX.com
http:
paths:
- path: /login
pathType: Exact
backend:
service:
name: awx-demo-service
port:
number: 80
$ kubectl describe ingress awx-demo-ingress
Name: awx-demo-ingress
Labels: <none>
Namespace: awx
Address: 10.0.3.51,10.0.3.52,10.0.3.53
Ingress Class: traefik
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
rocky-k3-1.XXXXXXXXXX.com
/login awx-demo-service:80 (10.42.1.9:8052)
Annotations: <none>
Events: <none>
But if I do a curl like this ...
$ curl http://rocky-k3-1.XXXXXXXX.com/login/
404 page not found
... I get 404 errors.

It turned out that I was using the wrong path in my ingress.yml file.
- path: /login
needed to change to
- path: /

Related

AKS Ingress Nginx external IP unreachable

I m using helm charts to deploy an app in Azure Kubernetes Service with Ingress-nginx
My helm charts used to work fine last month, but when I created something new with my charts, the public IP of my ingress nginx is not working anymore.
PS C:\Zetaris\HelmDeployment> kubectl get ingress -n zetaris
NAME CLASS HOSTS ADDRESS PORTS AGE
xx-gui-nginx <none> uisaas.xxxxxxxxxxxxxx 20.167.72.8 80, 443 53m
xx-rest-nginx <none> restsaas.xxxxx 20.167.72.8 80, 443 55m
Here is my yaml:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: lightning-gui-nginx
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: letsencrypt-enterprise
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "server: hide";
more_set_headers "X-Content-Type-Options: nosniff";
more_set_headers "X-Frame-Options: DENY";
more_set_headers "X-Xss-Protection: 1";
more_set_headers "referrer-policy: no-referrer";
more_set_headers "Content-Security-Policy: frame-ancestors 'none'";
spec:
tls:
- hosts:
- {{ .Values.ingress.guiurl }}
secretName: tls-secret-gui
rules:
- host: {{ .Values.ingress.guiurl }}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: xx-gui-svc
port:
number: 80
---
apiVersion: v1
kind: Service
metadata:
name: xx-gui-svc
namespace: {{ .Values.namespace }}
spec:
ports:
- port: 80
targetPort: 9001
type: ClusterIP
selector:
app: {{ .Values.deployment.name }}
I can reach my app on port 9001 if I port-forward directly my pod.
The error is coming from the IP:
telnet 20.167.72.8 80
Connecting To 20.167.72.8...Could not open connection to the host, on port 80: Connect failed
As far as i know, there no IP restrictions or firewalls in AKS.
Do you have any idea on why would my ingress IP would not be reachable on internet?
Tried to create another app with same helm chart, uninstall and reinstall ingress service, public IP is never reachable.
Never mind I found the issue, needed to add:
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz"
When creating the ingress-nginx

nginx path base routing it not working / is not responding

I have nginx deployed with helm
[rke#k8gui apps]$ kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-controller-fcrj2 1/1 Running 0 113m
pod/ingress-nginx-controller-jkpsm 1/1 Running 0 113m
pod/ingress-nginx-controller-v2dfh 1/1 Running 0 113m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller LoadBalancer 10.43.132.34 192.168.33.100 80:30345/TCP,443:31347/TCP 113m
service/ingress-nginx-controller-admission ClusterIP 10.43.84.182 <none> 443/TCP 113m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/ingress-nginx-controller 3 3 3 3 3 kubernetes.io/os=linux 113m
The external IP is received via metallb.
I am trying to test path base nginx routing but not working
apple.yaml
kind: Pod
apiVersion: v1
metadata:
name: apple-app
labels:
app: apple
spec:
containers:
- name: apple-app
image: hashicorp/http-echo
args:
- "-text=apple"
---
kind: Service
apiVersion: v1
metadata:
name: apple-service
spec:
selector:
app: apple
ports:
- port: 5678 # Default port for image
banana.yaml
kind: Pod
apiVersion: v1
metadata:
name: banana-app
labels:
app: banana
spec:
containers:
- name: banana-app
image: hashicorp/http-echo
args:
- "-text=banana"
---
kind: Service
apiVersion: v1
metadata:
name: banana-service
spec:
selector:
app: banana
ports:
- port: 5678 # Default port for image
ing.yaml
[rke#k8gui apps]$ more ing.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress
annotations:
ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
rules:
- http:
paths:
- path: /apple
pathType: Prefix
backend:
service:
name: apple-service
port:
number: 5678
- path: /banana
pathType: Prefix
backend:
service:
name: banana-service
port:
number: 5678
[rke#k8gui apps]$ kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
example-ingress nginx * 192.168.33.100 80 12m
but curling 192.168.33.100/apple or 192.168.33.100/banana is not working
any idea?
curling to the pod IP is working fine any idea what is missing?

Problem with ALB Ingress Controller in redirecting to right path

I have done the setup of ALB (Application Load Balancer) using Ingress Controller (version -> docker.io/amazon/aws-alb-ingress-controller:v1.1.8) for my AWS EKS cluster (v 1.20) running with Fargate profile.
I can access my service using the load balancer link:-
http://5e07dbe1-default-nginxingr-29e9-1260427999.us-east-1.elb.amazonaws.com/
I have 2 different services configured in my Ingress as shown below:-
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: "nginx-ingress"
namespace: "default"
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/security-groups: sg-014b302d73097d083
# alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
# alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:195725532069:certificate/b6a9e691-b807-4f10-a0bf-0449730ecdf4
# alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
# alb.ingress.kubernetes.io/backend-protocol: HTTPS
#alb.ingress.kubernetes.io/load-balancer-attributes: "60"
#alb.ingress.kubernetes.io/rewrite-target: /
labels:
app: nginx-ingress
spec:
rules:
- http:
paths:
# - path: /*
# pathType: Prefix
# backend:
# service:
# name: ssl-redirect
# port:
# number: use-annotation
- path: /foo
pathType: Prefix
backend:
service:
name: "nginx-service"
port:
number: 80
- path: /*
pathType: Prefix
backend:
service:
name: "mydocker-svc"
port:
number: 8080
Now the problem is if I put /foo at the end of LB link then nothing happens and I get 404 not found error:-
Both my services are fine with respective Pods running behind their respective Kubernetes NodePort services but they are not accessible using the Ingress. If I swap the path to /* from /foo for the other service (nginx-service), I can then access that but then it will break my previous service (mydocker-svc).
Please let me know where I'm the mistake so that I can fix this issue. Thank you
ALB Controller:-
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/name: alb-ingress-controller
name: alb-ingress-controller
namespace: kube-system
spec:
selector:
matchLabels:
app.kubernetes.io/name: alb-ingress-controller
template:
metadata:
labels:
app.kubernetes.io/name: alb-ingress-controller
spec:
containers:
- name: alb-ingress-controller
args:
- --ingress-class=alb
- --cluster-name=eks-fargate-alb-demo
- --aws-vpc-id=vpc-0dc46d370e38de475
- --aws-region=us-east-1
image: docker.io/amazon/aws-alb-ingress-controller:v1.1.8
serviceAccountName: alb-ingress-controller
Nginx service:-
apiVersion: v1
kind: Service
metadata:
annotations:
alb.ingress.kubernetes.io/target-type: ip
name: "nginx-service"
namespace: "default"
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
type: NodePort
selector:
app: "nginx"
mydocker-svc:-
apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
labels:
eks.amazonaws.com/fargate-profile: fp-default
run: mydocker
name: mydocker-svc
annotations:
alb.ingress.kubernetes.io/target-type: ip
spec:
ports:
- port: 8080
protocol: TCP
targetPort: 8080
selector:
eks.amazonaws.com/fargate-profile: fp-default
run: mydocker
type: NodePort
status:
loadBalancer: {}
TargetGroups become unhealthy, if the annotation in Kubernetes NodePort service like alb.ingress.kubernetes.io/target-type: IP is missing:-
You can try this out one i am using as reference
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-usermgmt-restapp-service
labels:
app: usermgmt-restapp
annotations:
# Ingress Core Settings
kubernetes.io/ingress.class: "alb"
alb.ingress.kubernetes.io/scheme: internet-facing
# Health Check Settings
alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
alb.ingress.kubernetes.io/healthcheck-port: traffic-port
#Important Note: Need to add health check path annotations in service level if we are planning to use multiple targets in a load balancer
#alb.ingress.kubernetes.io/healthcheck-path: /usermgmt/health-status
alb.ingress.kubernetes.io/healthcheck-interval-seconds: '15'
alb.ingress.kubernetes.io/healthcheck-timeout-seconds: '5'
alb.ingress.kubernetes.io/success-codes: '200'
alb.ingress.kubernetes.io/healthy-threshold-count: '2'
alb.ingress.kubernetes.io/unhealthy-threshold-count: '2'
spec:
rules:
- http:
paths:
- path: /app1/*
backend:
serviceName: app1-nginx-nodeport-service
servicePort: 80
- path: /app2/*
backend:
serviceName: app2-nginx-nodeport-service
servicePort: 80
- path: /*
backend:
serviceName: usermgmt-restapp-nodeport-service
servicePort: 8095
Read more at : https://www.stacksimplify.com/aws-eks/aws-alb-ingress/kubernetes-aws-alb-ingress-context-path-based-routing/

AWS EKS ALB ingress controller is not working, and HOST is not populating

I am trying to implement a simple "hello world" on eks with alb Ingress controller.
My goal is to ..
Create a cluster
Deploy an Ingress to access using ELB
Following things have been done
Created EKS cluster
added "alb ingress controller"
C:\workspace\eks>kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
alb-ingress-controller-5f96d7df77-mdrw2 1/1 Running 0 4m1s
Created application as below
apiVersion: apps/v1
kind: Deployment
metadata:
name: "2048-deployment"
namespace: "2048-game"
labels:
app: "2048"
spec:
replicas: 1
selector:
matchLabels:
app: "2048"
template:
metadata:
labels:
app: "2048"
spec:
containers:
- image: alexwhen/docker-2048
imagePullPolicy: Always
name: "2048"
ports:
- containerPort: 80
Serveice is as following
apiVersion: v1
kind: Service
metadata:
name: "service-2048"
namespace: "2048-game"
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
type: NodePort
selector:
app: "2048"
Ingress controller is as below
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: "2048-ingress"
namespace: "2048-game"
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
labels:
app: 2048-ingress
spec:
rules:
- http:
paths:
- path: /*
backend:
serviceName: "service-2048"
servicePort: 80
output is as below, not getting Host addess as ELB .and not able to access from outside
C:\sample>kubectl get ingress/2048-ingress -n 2048-game
NAME HOSTS ADDRESS PORTS AGE
2048-ingress * 80 71s
Update :
Found following error in alb-ingress-controller-5f96d7df77-mdrw2 logs.
Not able to find how to change
kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to resolve 2 qualified subnet for ALB. Subnets must contains these tags: 'kubernetes.io/cluster/ascluster': ['shared' or 'owned'] and 'kubernetes.io/role/elb': ['' or '1']. See https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/controller/config/#subnet-auto-discovery for more details. Resolved qualified subnets: '[]'" "controller"="alb-ingress-controller" "request"={"Namespace":"default","Name":"ingress-default-dev"}
The subnets where eks nodes resides should be tagged with the following
https://docs.aws.amazon.com/eks/latest/userguide/network_reqs.html#vpc-subnet-tagging
If your subnets are not tagged with kubernetes.io/cluster/<cluster-name>=shared etc....
you can also try passing subnets in ingress file annotations like below
alb.ingress.kubernetes.io/subnets: subnet-xxxxxx, subnet-xxxxxx

How to make websocket working in kubernetes ingress?

I am dockering our current application and deploying on kubernetes cluster.
We have 2 services, namely, service-A and service-B. One of our services(example service-A) uses websocket. we have configured a rule in ingress to route the websocket request directly to service-A on port 8080. Also have a rule to route other requests to service-B on port 443. But ingress controller always route the websocket request to service-B instead of routing to service-A.
So, I removed the service-B rule from ingress, but still its routed as tls request and request never reaches service-A. Not sure why its rerouted as TLS instead of http request upgraded to websocket connection.
Please find my ingress configuration below:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
field.cattle.io/publicEndpoints: '[{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"cluster42:service-B","ingressName":"cluster42:my-ingress","hostname":"cluster42-phase-0 ","path":"/","allNodes":false},{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"cluster42:service-A","ingressName":"cluster42:my-ingress","hostname":"cluster42-phase-0 ","path":"/ws-service","allNodes":false}]'
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/rewrite-target":"/","nginx.ingress.kubernetes.io/ssl-passthrough":"true"},"labels":{"app":"ingress","chart":" myapplication-chart","heritage":"Tiller","release":"installation-cluster42"},"name":"my-ingress","namespace":"cluster42"},"spec":{"rules":[{"host":"cluster42-phase-0 ","http":{"paths":[{"backend":{"serviceName":"service-B","servicePort":443},"path":"/"}]}}],"tls":[{"hosts":["cluster42-phase-0 "]}]}}
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.org/websocket-services: "service-A"
creationTimestamp: "2019-05-24T11:46:40Z"
generation: 31
labels:
app: ingress
chart: myapplication-chart
heritage: Tiller
release: installation-cluster42
name: my-ingress
namespace: cluster42
resourceVersion: "57549362"
selfLink: /apis/extensions/v1beta1/namespaces/cluster42/ingresses/my-ingress
uid: 98784b1f-7e19-11e9-b2f1-005056b0b58e
spec:
rules:
- host: cluster42-phase-0
http:
paths:
- backend:
serviceName: service-B
servicePort: 443
path: /
- backend:
serviceName: service-A
servicePort: 8080
path: /ws-service
tls:
- hosts:
- cluster42-phase-0
status:
loadBalancer:
ingress:
- {}
I expect the request to be routed to service-A instead of service-B. Can you please let me know if I am missing something in my configuration or doing anything wrong.
Thanks in Advance.
Check if destination port are opened and not used.
Then check if you have enough right to access by tiller to kube-system namespace.
otherwise you have to create RBAC and special service.
More infromation you can find here: tiller-rbac.
You have some mistakes in your ingress configuration file in spec section.
Take notice that currently the Ingress only supports a single TLS port, 443, and assumes TLS termination. So it's obvious that destination service will be service B with port 443. So you can delete tls section from configuration file.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
field.cattle.io/publicEndpoints: '[{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"cluster42:service-B","ingressName":"cluster42:my-ingress","hostname":"cluster42-phase-0 ","path":"/","allNodes":false},{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"cluster42:service-A","ingressName":"cluster42:my-ingress","hostname":"cluster42-phase-0 ","path":"/ws-service","allNodes":false}]'
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/rewrite-target":"/","nginx.ingress.kubernetes.io/ssl-passthrough":"true"},"labels":{"app":"ingress","chart":" myapplication-chart","heritage":"Tiller","release":"installation-cluster42"},"name":"my-ingress","namespace":"cluster42"},"spec":{"rules":[{"host":"cluster42-phase-0 ","http":{"paths":[{"backend":{"serviceName":"service-B","servicePort":443},"path":"/"}]}}],"tls":[{"hosts":["cluster42-phase-0 "]}]}}
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.org/websocket-services: "service-A"
creationTimestamp: "2019-05-24T11:46:40Z"
generation: 31
labels:
app: ingress
chart: myapplication-chart
heritage: Tiller
release: installation-cluster42
name: my-ingress
namespace: cluster42
resourceVersion: "57549362"
selfLink: /apis/extensions/v1beta1/namespaces/cluster42/ingresses/my-ingress
uid: 98784b1f-7e19-11e9-b2f1-005056b0b58e
spec:
rules:
- host: cluster42-phase-0
http:
paths:
- path: /
backend:
serviceName: service-B
servicePort: 443
- path: /ws-service
backend:
serviceName: service-A
servicePort: 8080
status:
loadBalancer:
ingress:
- {}