I have a kubernetes ingress manifest YAML, looking like next:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
certmanager.k8s.io/acme-http01-edit-in-place: "true"
certmanager.k8s.io/cluster-issuer: letsencrypt
name: staging
namespace: dev
spec:
rules:
- host: staging.domain.com
http:
paths:
- backend:
serviceName: task-11111
servicePort: 80
path: /task-11111/*
- backend:
serviceName: task-22222
servicePort: 80
path: /task-22222/*
- backend:
serviceName: task-33333
servicePort: 80
path: /task-33333/*
tls:
- hosts:
- staging.domain.com
secretName: staging-domain-com
What I'm trying to achieve is to add (if not present) or remove (if present) a backend block. What I have now:
yq -y '.spec.rules[].http.paths += [{ "backend": { "serviceName": "'${CI_COMMIT_REF_NAME}'", "servicePort": 80}, "path": "/'${CI_COMMIT_REF_NAME}'/*"}]'
(adds a new block with variable value, but doesn't bother if it already exists)
yq -y 'del(.. | .paths? // empty | .[] | select(.path |contains("'${CI_COMMIT_REF_NAME}'")) )'
(fails with jq: error (at <stdin>:0): Invalid path expression with result {"backend":{"serviceName":...)
So rules may look like this after deletion (assume that CI_COMMIT_REF_NAME = task-33333):
spec:
rules:
- host: staging.domain.com
http:
paths:
- backend:
serviceName: task-11111
servicePort: 80
path: /task-11111/*
- backend:
serviceName: task-22222
servicePort: 80
path: /task-22222/*
or like this after adding (assume that CI_COMMIT_REF_NAME = task-44444):
spec:
rules:
- host: staging.domain.com
http:
paths:
- backend:
serviceName: task-11111
servicePort: 80
path: /task-11111/*
- backend:
serviceName: task-22222
servicePort: 80
path: /task-22222/*
- backend:
serviceName: task-33333
servicePort: 80
path: /task-33333/*
- backend:
serviceName: task-44444
servicePort: 80
path: /task-44444/*
Any help is greatly appreciated.
[The following has been revised to reflect the update to the question.]
Assuming CI_COMMIT_REF_NAME is available to jq as $CI_COMMIT_REF_NAME, which can be done using jq with the command-line argument:
--arg CI_COMMIT_REF_NAME "$CI_COMMIT_REF_NAME"
an appropriate jq filter would be along the following lines:
.spec.rules[0].http.paths |=
(map(select(.path | index($CI_COMMIT_REF_NAME) | not)) as $new
| if ($new | length) == length
then . + [{ "backend": { "serviceName": $CI_COMMIT_REF_NAME, "servicePort": 80}, "path": ($CI_COMMIT_REF_NAME + "/*") }]
else $new
end )
You can test this with the following jq invocation:
jq --arg CI_COMMIT_REF_NAME task-4444 -f program.jq input.json
where of course input.json is the JSON version of your YAML.
(I'd use index in preference to contains if at all possible.)
Related
I assume more than two domains to be used for my multi-tenant code, domain1.com, mydomain2.com each will have a unique identifier and an ID would be maintained for each
I want my common app myApp hosted at myappdomain.com to serve both domains using CNAME and the service container should get a request prefix in the path based on the domain like:
domain1.com/home should translate the myApp request like myApp/111/home where 111 is the id
domain2.com/home should translate the myApp request like myApp/222/home where 222 is the id
This is working when the request is formed like domain1.com/myApp/111/home with the config below, but I want to use a generic rewrite rule to mask domain1.com/home to myApp/111/home & domain2.com/home to myApp/222/home
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: myapp-custom-ingress
namespace: myapp-dev
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /myapp/$1
spec:
rules:
- host: myappdomain.com
http:
paths:
- path: /myapp/(.*)
pathType: ImplementationSpecific
backend:
serviceName: ui-myapp-proxy-mgmt-webapp
servicePort: 80
- host: domain1.com
http:
paths:
- path: /myapp/(.*)
pathType: ImplementationSpecific
backend:
serviceName: ui-myapp-proxy-mgmt-webapp
servicePort: 80
Note: currently myApp is deployed in html\myApp directory of apache in the service image
I am trying to use microservices with my frontend application through nginx ingress controller.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.48.1/deploy/static/provider/aws/deploy.yaml
above is the command which we have followed to deploy nginx-controller.
reference - https://kubernetes.github.io/ingress-nginx/deploy/#aws
------ My deployment.yaml & service.yaml for integrations-api is as below -------
'''
apiVersion: apps/v1
kind: Deployment
metadata:
name: integrations-api
labels:
app: integrations-api
spec:
replicas: 1
selector:
matchLabels:
app: integrations-api
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: integrations-api
spec:
containers:
- image: "###imagepath####"
imagePullPolicy: Always
name: integrations-api
ports:
- containerPort: 8083
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: integrations-api
annotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp,http"
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "###certpath###"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
spec:
type: LoadBalancer
selector:
app: integrations-api
ports:
- name: http
port: 80
targetPort: 8083
protocol: TCP
- name: https
port: 443
targetPort: 8083
protocol: TCP
'''
------ My ingress.yaml looks like this --------
'''
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: example-ingress
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: integrations-api
servicePort: 80
- path: /
backend:
serviceName: user-api
servicePort: 80
'''
IN my node js integraion-api code we have added testing api path as below
'''
app.get('/camps', (req, res) => {
let obj = {}
res.send(obj);
});
'''
When i am vising endpoint of nginx-controller(here it is load balancer endpoint) https://####NLB-endpoint###/camps
i am getting response.
same configuration like deployment.yaml, service.yaml & nodejs code is written for user-services api. but i am not getting response for user-api
https://####NLB-endpoint###/users
Note, When i am shuffeling the ingress file as below i am getting response of https://####NLB-endpoint###/users but not for https://####NLB-endpoint###/camps. looks inress is taking path which is mentioned in first place only.
'''
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: example-ingress
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: user-api
servicePort: 80
- path: /
backend:
serviceName: integrations-api
servicePort: 80
'''
Any clue how can i fix this ?
Thanks in advance. it would great help from your side if someone guide us on the same.
This response may be too late. You are trying to creating a path based routing with multiple microservices. In this case, you need to set a path. For user-api, you need to set /user as path and then for integrations-api, you need to specify another path like /camps.
I am using alb ingress controller and the ingress yaml file is pasted below.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
namespace: default
name: alb-ingress
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:123456789123:certificate/xxxxxxx-yyyy-zzzz-abce-ffffffffffff
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/security-groups: sg-xxxxxxxxxxxxxx
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
spec:
rules:
- http:
paths:
- path: /*
backend:
serviceName: ssl-redirect
servicePort: use-annotation
- host: domain.com
http:
paths:
- backend:
serviceName: greensvc
servicePort: 80
path: /green
- backend:
serviceName: redsvc
servicePort: 80
path: /red
- host: mongo.domain.com
http:
paths:
- backend:
serviceName: mongo-express-service
servicePort: 8081
corresponding rules in ALB is
ALB was deployed successfully but the logs from ingress "alb-ingress" reported two error
/* ssl-redirect:use-annotation (<error: endpoints "ssl-redirect" not found>)
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
what I am trying to achieve is to create a single default action (no need of any other rules) on port 80 which is forcing the https traffic. A default action in https port that is forwarded to a service, Along with other host or path based routing as per the scrren shot. Currently both default action is set to return a fixed responce 404.
How can I modify current yaml file such that the ALB rules will be updated as per my interest. Also why the ingress is logging the annotation error for ssl-redirect.
Using apiVersion: networking.k8s.io/v1 worked for me.
You have to be aware of some changes needed to be done as you are using a different apiVersion as listed here https://kubernetes.io/docs/reference/using-api/deprecation-guide/#ingress-v122
My final code was like these:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
namespace: ...
name: ...
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/actions.response-420: >
{"type":"fixed-response","fixedResponseConfig":{"contentType":"text/plain","statusCode":"420","messageBody":"..."}}
spec:
rules:
- http:
paths:
- path: /myPath
pathType: Prefix
backend:
service:
name: response-420
port:
name: use-annotation
I've not tested this myself, so please evaluate this closely before applying directly.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
namespace: default
name: alb-ingress
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:123456789123:certificate/xxxxxxx-yyyy-zzzz-abce-ffffffffffff
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/security-groups: sg-xxxxxxxxxxxxxx
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
alb.ingress.kubernetes.io/actions.response-404: >
{"type":"fixed-response","fixedResponseConfig":{"contentType":"text/plain","statusCode":"404","messageBody":"Not Found"}}
alb.ingress.kubernetes.io/actions.green-svc: >
{"type":"forward","forwardConfig":{"targetGroups":[{"serviceName":"green-k8s-service","servicePort":80,"weight":100}]}}
alb.ingress.kubernetes.io/conditions.green-svc: >
[{"field":"host-header","hostHeaderConfig":{"values":["domain.com"]}},{"field":"path-pattern","pathPatternConfig":{"values":["/green"]}}]
alb.ingress.kubernetes.io/actions.red-svc: >
{"type":"forward","forwardConfig":{"targetGroups":[{"serviceName":"red-k8s-service","servicePort":80,"weight":100}]}}
alb.ingress.kubernetes.io/conditions.red-svc: >
[{"field":"host-header","hostHeaderConfig":{"values":["domain.com"]}},{"field":"path-pattern","pathPatternConfig":{"values":["/red"]}}]
alb.ingress.kubernetes.io/actions.mongo-svc: >
{"type":"forward","forwardConfig":{"targetGroups":[{"serviceName":"mongo-k8s-service","servicePort":8081,"weight":100}]}}
alb.ingress.kubernetes.io/conditions.mongo-svc: >
[{"field":"host-header","hostHeaderConfig":{"values":["mongo.domain.com"]}}]
spec:
backend:
serviceName: response-404
servicePort: use-annotation
rules:
- http:
paths:
- backend:
serviceName: ssl-redirect
servicePort: use-annotation
- backend:
serviceName: green-svc
servicePort: use-annotation
- backend:
serviceName: red-svc
servicePort: use-annotation
- backend:
serviceName: mongo-svc
servicePort: use-annotation
The default ALB rule is specified directly under the spec object but before the rules object.
For the annotations, actions and conditions are interpreted together when they share the same name.
The Ingress Annotations doc page for AWS LoadBalancer Controller has a lot of great examples that should be able to help you work out any other tweaks or changes you might want to make. Specifically, check out Traffic Routing
I am bit confused the way path resolves the endpoint, does it show in any logs the final endpoint it creates. I am stuck with this now. Below is the endpoint which I wanted to call:-
https://hostname/api/orders/employees. And to call this endpoint through Ingress application gateway, this is how I configured but it always return 502 bad gateway error.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ordersapi
namespace: orders
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/appgw-ssl-certificate: "wildcard.apps.com"
appgw.ingress.kubernetes.io/ssl-redirect: "true"
spec:
rules:
- host: orders.apps.com
http:
paths:
- path: /api/orders/employees
backend:
serviceName: orderservice
servicePort: 80
Finally, there are two solution to this problem:-
First
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ordersapi
namespace: orders
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/appgw-ssl-certificate: "wildcard.apps.com"
appgw.ingress.kubernetes.io/ssl-redirect: "true"
spec:
rules:
- host: orders.apps.com
http:
paths:
- path: /api/orders/employees
backend:
serviceName: orderservice
servicePort: 80
- path: /api/product/products
backend:
serviceName: orderservice
servicePort: 80h
Second
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ordersapi
namespace: orders
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/backend-path-prefix: "/api/"
appgw.ingress.kubernetes.io/appgw-ssl-certificate: "wildcard.apps.com"
appgw.ingress.kubernetes.io/ssl-redirect: "true"
spec:
rules:
- host: orders.apps.com
http:
paths:
- path: /api/*
backend:
serviceName: orderservice
servicePort: 80
You seem to have enabled SSL redirect but your service is serving on a non ssl port.
this could explain the bad gateway.
Often, the Azure AppGW will return 502 Bad Gateway when there are bad Certs involved, the health check for the backend service is wrong, and other reason
You should look at this:
https://support.microsoft.com/en-ca/help/4504111/azure-application-gateway-with-bad-gateway-502-errors
and this
https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-troubleshooting-502
I got same issue. My ingress setup was like:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-name
annotations:
kubernetes.io/ingress.class: azure/application-gateway
spec:
rules:
- host: <myhostname>
http:
paths:
- path: /api*
backend:
serviceName: backend-app
servicePort: 80
- path: /
backend:
serviceName: frontend-app
servicePort: 80
If I defined path like /api/todolist, the endpoint worked fine. On the other hand if I went with /api* my requests were redirected to frontend app.
The problem was that endpoint /api/todolist existed on my backend and returned status was 200, for the /api endpoint I did not setup anything so I got 404 status.
In my case I needed to add healthcheck under /api edpoint, that returned proper http status :) For me returning string "Healthy" was enough.
#djsly #Jean-Philippe Bond - Thanks for your response and pointing the URL that helped me to investigate further. Having the backend application deployed on port 80 had a reason as SSL terminates at application gateway and listener redirects the request to backend application running on port 80, which works fine.
After further investigation, I added backend path prefix in ingress file (appgw.ingress.kubernetes.io/backend-path-prefix: "/api/orders/employees") which resolved the problem for one endpoint but not for all.
To describe the problem in details, application contains some of the restful services mentioned below and their endpoints are such as -
http://hostname/api/orders/employees
http://hostname/api/Lookup/officeHierarchy
http://hostname/api/Department/codes
http://hostname/api/position/members
Now if you see, these different endpoints starts with prefix "/api/" and then the controller name and actions.
Here Expected Result is
If any of these endpoints are called (via HTTP Get), data should be returned but it fails.
Investigation done so far
I added the prefix and did some changes to it. Thus If I configure my ingress like below, it returns the result successfully for only one specific endpoint - >
curl -v http://orders.apps.com/api/orders/employees returns 200 but fails for others.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ordersapi
namespace: orders
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/backend-path-prefix: "/api/orders/employees"
appgw.ingress.kubernetes.io/appgw-ssl-certificate: "wildcard.apps.com"
appgw.ingress.kubernetes.io/ssl-redirect: "true"
spec:
rules:
- host: orders.apps.com
http:
paths:
- path: /api/orders/employees
backend:
serviceName: orderservice
servicePort: 80
Thus , to make all endpoints works, I did the below changes in the above mentioned ingress file but calling
curl -v http://orders.apps.com/api/orders/employees returns 404. And the same goes with other endpoints like
curl -v http://orders.apps.com/api/department/codes returns 404.
As per my understanding, by doing the below changes - "path - /api/*" should be overwritten to the path - /api/orders/employees being called but it does not.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ordersapi
namespace: orders
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/backend-path-prefix: "/api/"
appgw.ingress.kubernetes.io/appgw-ssl-certificate: "wildcard.apps.com"
appgw.ingress.kubernetes.io/ssl-redirect: "true"
spec:
rules:
- host: orders.apps.com
http:
paths:
- path: /api*
backend:
serviceName: orderservice
servicePort: 80
Your suggestions are appreciated.
Thanks
Defining multiple paths without backend-path-prefix worked, e.g.
http:
paths:
path: /api/order/employees
backend:
serviceName: orderservice
servicePort: 80
path: /api/position/members
backend:
serviceName: positionservice
servicePort: 80
This seems to be a complex way but so far I found this the only way which worked.
For me the solution was to just add "*" to paths, e.g.
rules:
- http:
paths:
- pathType: Prefix
path: /api/*
backend:
service:
name: api
port:"
number: 80
- pathType: Prefix
path: /graphql*
backend:
service:
name: api
port:
number: 80
without any extra annotations. For whatever reason it does not work without "*" at the end.
I am using AKS and I need to make the ingress controller accessible by host and IP (to be accessed traffic manager), I am using Nginx Ingress rules as below:
for example using https://foo.bar.com and https://14.15.16.222
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: foo
namespace: default
spec:
tls:
- hosts:
- foo.bar.com
secretName: foo-secret
rules:
- host: foo.bar.com
http:
paths:
- backend:
serviceName: echoheaders-x
servicePort: 80
path: /
The below configuration did the trick eventhough I get a certificate error with the IP access.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: foo
namespace: default
spec:
tls:
- hosts:
- foo.bar.com
secretName: foo-secret
rules:
- host: foo.bar.com
http:
paths:
- backend:
serviceName: echoheaders-x
servicePort: 80
path: /
http:
paths:
- backend:
serviceName: echoheaders-x
servicePort: 80
path: /