I have private Azure Kubernetes cluster with installed Nginx Ingress (using internal Load Balancer)
This is non-production cluster and during weekends we plan stop it. But when we start it - it can't be finished successfully and after 30 minutes AKS cluster is in Failed state
After research I found that it happens only if Ingress is installed on private AKS with restricted outbound access
Any ideas how can it be solved?
one thing you can do is upgrade your Kubernetes cluster:
Check the upgrades available for your cluster
az aks get-upgrades --resource-group <resoure-group-name> --name <cluster-name> --output table
Then upgrade your cluster
az aks upgrade \
--resource-group <resoure-group-name> \
--name <cluster-name> \
--kubernetes-version <kubernetes_verion>
Replace the Kubernetes version by a version you got from the first command
Related
I am using kubernetes (by windows 10 - docker desktop).
I am using mysql, that is running by helm 3 (loaded from bitnami repository).
I am creating another application.
For now, I am testing on docker (not in kubernetes yet).
Everything is fine, but when trying to connect the database from my project
(BTW - Project works fine, but not when running on docker).
Something like:
docker run --name test-docker --rm my-image:tag --db "root:12345#tcp(127.0.0.1:3306)/test"
(db is a parameter to to connect to db).
I get the message:
2022-02-21T12:18:17.205Z FATAL failed to open db: could not setup schema: cannot create jobs table: dial tcp 127.0.0.1:3306: connect: connection refused
I have investigate a little, and find that the problem may be because the dockers running need to run on the same network.
(Nonetheless, they are actually dockers, when one is running by helm tool for K8S).
this is on:
kubernetes networking
When I run:
nsenter -t your-container-pid -n ip addr
the pid is not directory, so I get the message:
/proc/<pid>/ns/net - No such file or directory
How can I eventually run my project that can use the mysql (running in dockers on K8S)?
Thanks.
Docker containers are isolated from other containers and the external network by default. There are several options to establish connection between Docker containers:
Docker sets up a default bridge network automatically, through which the communication is possible between containers and between containers and the host machine. Both your containers should be on the bridge network - for container with your project to connect to your DB container by referring to it's name. More details on this approach and how it can be set up is here.
You can also create user-defined bridge network - basically, your own custom bridge network - and attach your Docker containers to it. In this way, both containers won't be connected to the default bridge network at all. Example of this approach is described in details here.
First, user-defined network should be created:
docker network create <network-name>
List your newly created network and check with inspect command its IP address and that no containers are connected to it:
docker network ls
docker network inspect <network-name>
You can either connect your containers on their start with --network flag:
docker run -dit --name <container-name1> --network <network-name>
docker run -dit --name <container-name2> --network <network-name>
Or attach running containers by their name or by their ID to your newly created network by docker network connect - more options are listed here:
docker network connect <network-name> <container-name1>
docker network connect <network-name> <container-name2>
To verify that your containers are connected to the network, check again the docker network inspect command.
Once connected in network, containers can communicate with each other, and you can connect to them using another container’s IP address or name.
EDIT: As suggested by #Eitan, when referring to the network instead of a changing IP address in root:12345#tcp(127.0.0.1:3306)/test, special DNS name host.docker.internal can be used - it resolves to the internal IP address used by the host.
I have set up a Kubernetes cluster using Minikube in an Ubuntu VM. I cloned this GitHub repo and created the namespace, deployment, service and ingress.
I have also enabled ingress addon by running minikube addons enable ingress.
When I run kubectl get svc -n ingress-nginx, the external ip is none.
When I run kubectl get ingress -n sample, the address is empty.
Please advise how to set up k8s ingress.
PS: I had minikube tunnel running.
PS 2: kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create--1-* 0/1 Completed 0 11m
ingress-nginx-admission-patch--1-* 0/1 Completed 1 11m
ingress-nginx-controller-* 1/1 Running 0 11m
Thanks to this SO post. It worked after I downgraded Minikube to v1.11.0. I used --driver=none.
I have 2 home PCs with windows 10. I want to create test ipfs private network between 2 nodes.
I downloaded go-ipfs.
ipfs version 0.6.0
I executed
ipfs init
ipfs bootstrap rm --all
I made swarm.key and added node to config.
ipfs bootstrap add /ip4/62.122.132.55/tcp/4001/ipfs/QmYY9U3JFuMykjRjKd1jg6aPCmcf2UYpbaS8fHpwkAVvia
I checked if port 4001 is listening.
TCP [::]:4001 [::]:0 LISTENING
The same I made on the second node.
I started nodes
ipfs daemon
The result was
Initializing daemon...
go-ipfs version: 0.6.0
Repo version: 10
System version: amd64/windows
Golang version: go1.14.4
Swarm is limited to private network of peers with the swarm key
Swarm key fingerprint: 4c11fea5d8ab67ccf08b22a4ee220db3
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/192.168.1.102/tcp/4001
Swarm listening on /ip6/::1/tcp/4001
Swarm listening on /p2p-circuit
Swarm announcing /ip4/127.0.0.1/tcp/4001
Swarm announcing /ip4/192.168.1.102/tcp/4001
Swarm announcing /ip6/::1/tcp/4001
API server listening on /ip4/127.0.0.1/tcp/5001
WebUI: http://127.0.0.1:5001/webui
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready
On second node I tried execute
c:\ipfs>ipfs swarm connect /ip4/62.122.132.55/tcp/4001/ipfs/QmYY9U3JFuMykjRjKd1jg6aPCmcf2UYpbaS8fHpwkAVvia
Error: connect QmYY9U3JFuMykjRjKd1jg6aPCmcf2UYpbaS8fHpwkAVvia failure: \
failed to dial QmYY9U3JFuMykjRjKd1jg6aPCmcf2UYpbaS8fHpwkAVvia: all dials failed
* [/ip4/62.122.132.55/tcp/4001] dial tcp4 0.0.0.0:4001->62.122.132.55:4001: i/o timeout
c:\ipfs>
After some time daemon showed
Daemon is ready
2020-07-19T08:43:50.560+0300 [31mERROR[0m reprovider.simple simple/reprovide.go:108 \
failed to reprovide: failed to find any peer in table
What I missed?
can you run the following commands?
ipfs id
https://docs.ipfs.io/reference/cli/#ipfs-id
ipfs swarm peers
https://docs.ipfs.io/reference/cli/#ipfs-swarm-peers
ipfs boostrap list
https://docs.ipfs.io/reference/cli/#ipfs-bootstrap
Please can someone let me know which latest version of openshift-ansible (origin) is stable enough to install on Centos 7?
I am looking for successful multi-node install experience and any tips that was used.
Thanks
the latest stable release is 3.9
git clone https://github.com/openshift/openshift-ansible
cd openshift-ansible
git checkout release-3.9
and follow the Advanced Installation guide
https://docs.openshift.org/latest/install_config/install/advanced_install.html
It is now working.
After enabling openshift_repos_enable_testing=true, I did not run the pre-requisite playbook before the deploy_cluster playbook, which was why it was still giving the error of not finding the packages.
I believe that v3.11.0 version of OpenShift OKD/Origin (latest 3.x release at time) meets your needs. In this answer is a complete roadmap for installing OpenShift OKD/Origin as a single node cluster service.
Some information transposed from the OKD website about OpenShift OKD/Origin...
The Community Distribution of Kubernetes that powers Red Hat
OpenShift. Built around a core of OCI container packaging and
Kubernetes container cluster management, OKD is also augmented by
application lifecycle management functionality and DevOps tooling. OKD
provides a complete open source container application platform.
OKD is a distribution of Kubernetes optimized for continuous
application development and multi-tenant deployment. OKD adds
developer and operations-centric tools on top of Kubernetes to enable
rapid application development, easy deployment and scaling, and
long-term lifecycle maintenance for small and large teams. OKD is a
sibling Kubernetes distribution to Red Hat OpenShift.
OKD embeds Kubernetes and extends it with security and other
integrated concepts. OKD is also referred to as Origin in github and
in the documentation.
If you are looking for enterprise-level support, or information on
partner certification, Red Hat also offers Red Hat OpenShift Container
Platform.
So I recommend starting with OpenShift OKD/Origin using the roadmap below to install on CentOS 7. Then you can explore other possibilities ("multi-node", for example).
However, if you want to test the OpenShift (OKD) 4.X application the guide and the right way to do this is at this link Install the OpenShift (OKD) 4.X cluster (UPI/"bare-metal"). It is a long way and with a reasonable level of complexity.
PLUS:
Informations about OpenShift Ansible on GitHub and RedHat Ansible;
You can take a look at the OpenShift Installer (NOT OKD/Origin!).
OpenShift Origin (OKD) - Open source container application platform:
OpenShift is a family of containerization software products developed by Red Hat. Its flagship product is the OpenShift Container Platform - an on-premises platform as a service built around Docker containers orchestrated and managed by Kubernetes on a foundation of Red Hat Enterprise Linux. The family's other products provide this platform through different environments: OKD serves as the community-driven upstream (akin to the way that Fedora is upstream of Red Hat Enterprise Linux), OpenShift Online is the platform offered as software as a service, and Openshift Dedicated is the platform offered as a managed service.
The OpenShift Console has developer and administrator oriented views. Administrator views allow one to monitor container resources and container health, manage users, work with operators, etc. Developer views are oriented around working with application resources within a namespace. OpenShift also provides a CLI that supports a superset of the actions that the Kubernetes CLI provides.
The OpenShift Origin (OKD) is the comunity driven version of OpenShift (non-enterprise-level). That means you can host your own PaaS (Platform as a service) for free and almost with no hassle.
[Ref(s).: https://en.wikipedia.org/wiki/OpenShift ,
https://www.openshift.com/blog/openshift-ecosystem-get-started-openshift-origin-gitlab ]
Setup Local OpenShift Origin (OKD) Cluster on CentOS 7
All commands in this setup must be performed with the "root" user.
Update CentOS 7
Updating your CentOS 7 server...
yum -y update
Install and Configure Docker
OpenShift required docker engine on the host machine for running containers. Install Docker and other dependencies on CentOS 7 using the commands below...
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum -y install git-core
yum -y install wget
yum -y install yum-utils
yum -y install device-mapper-persistent-data
yum -y install lvm2
yum -y install docker-ce
yum -y install docker-ce-cli
yum -y install containerd.io
Add logged in user account to docker group...
usermod -aG docker $USER
newgrp docker
Create necessary folders...
mkdir "/etc/docker"
mkdir "/etc/containers"
Create "registries.conf" file with an insecure registry parameter ("172.30.0.0/16") to the Docker daemon...
tee "/etc/containers/registries.conf" << EOF
[registries.insecure]
registries = ['172.30.0.0/16']
EOF
Create "daemon.json" file with configurations...
tee "/etc/docker/daemon.json" << EOF
{
"insecure-registries": [
"172.30.0.0/16"
]
}
EOF
We need to reload systemd and restart the Docker daemon after editing the config...
systemctl daemon-reload
systemctl restart docker
Enable Docker to start at boot...
systemctl enable docker
Then enable "IP forwarding" on your system...
tee "/etc/sysctl.d/ip_forward.conf" << EOF
net.ipv4.ip_forward=1
EOF
sysctl -w net.ipv4.ip_forward=1
Configure Firewalld.
Add the necessary firewall permissions...
firewall-cmd --zone=public --add-port=80/tcp --permanent
firewall-cmd --zone=public --add-port=443/tcp --permanent
firewall-cmd --zone=public --add-port=8443/tcp --permanent
firewall-cmd --zone=public --add-port=53/udp --permanent
firewall-cmd --zone=public --add-port=8053/udp --permanent
firewall-cmd --reload
NOTE: Allows containers access to the OpenShift master API (8443/tcp), DNS (53/udp) endpoints and add others permissions.
Download OpenShift
Download the OpenShift binaries from GitHub and move them to the "/usr/local/bin/" folder...
wget https://github.com/openshift/origin/releases/download/v3.11.0/openshift-origin-client-tools-v3.11.0-0cbc58b-linux-64bit.tar.gz
tar -zxvf openshift-origin-client-tools-v3.11.0-0cbc58b-linux-64bit.tar.gz
cd ./openshift-origin-client-tools-v3.11.0-0cbc58b-linux-64bit
mv ./oc /usr/local/bin/
mv ./kubectl /usr/local/bin/
rm -rf ./openshift-origin-client-tools-v3.11.0-0cbc58b-linux-64bit*
Verify installation of OpenShift client utility...
oc version
Start OpenShift Origin (OKD) Local Cluster
Now bootstrap a local single server OpenShift Origin cluster by running the following command...
oc cluster up --public-hostname="<YOUR_SERVER_IP_OR_NAME>"
... or...
oc cluster up --public-hostname="$(ip route get 1 | awk '{print $NF;exit}')"
This one above will get the primary IP address of the local machine dynamically.
[Ref(s).: https://stackoverflow.com/a/25851186/3223785 ]
TIP: In case of error, try perform the command oc cluster down and repeat the command above.
NOTE: Insufficient hardware configuration (mainly CPU and RAM) will cause timeout on the command above.
IMPORTANT: If the parameter --public-hostname="<YOUR_SERVER_IP_OR_NAME>" is not informed, then calls to the web service ("web console") at URL <YOUR_SERVER_IP_OR_NAME> will be redirected to the local IP "127.0 .0.1".
[Ref(s).: https://github.com/openshift/origin/issues/19699 , https://github.com/openshift/origin/issues/19699#issuecomment-854069124 , https://github.com/openshift/origin/issues/20726 ,
https://github.com/openshift/origin/issues/20726#issuecomment-498078849 , https://hayardillasenlared.blogspot.com/2020/06/instalar-openshift-origin-ubuntu.html , https://www.a5idc.net/helpview_526.html , https://thecodeshell.wordpress.com/ , https://www.techrepublic.com/article/how-to-install-openshift-origin-on-ubuntu-18-04/ ]
The command above will...
Start OKD Cluster listening on the interface informed (<YOUR_SERVER_IP_OR_NAME>:8443);
Start a web console listening on all interfaces at "/console" (<YOUR_SERVER_IP_OR_NAME>:8443);
Launch Kubernetes system components;
Provisions registry, router, initial templates and a default project;
The OpenShift cluster will run as an all-in-one container on a Docker host.
On a successful installation, you should get output similar to below...
[...]
Login to server ...
Creating initial project "myproject" ...
Server Information ...
OpenShift server started.
The server is accessible via web console at:
https://<YOUR_SERVER_IP_OR_NAME>:8443
You are logged in as:
User: developer
Password: <any value>
To login as administrator:
oc login -u system:admin
TIPS:
There are a number of options which can be applied when setting up Openshift Origin. View them with oc cluster up --help;
Command model using custom options...
MODEL
oc cluster up --public-hostname="<PUBLIC_HOSTNAME_OR_IP>" --routing-suffix="<PUBLIC_HOSTNAME_OR_IP>.<SUFFIX>"
EXAMPLE
oc cluster up --public-hostname="192.168.56.124" --routing-suffix="192.168.56.124.nip.io"
;
The OpenShift Origin cluster configuration files will be located inside the "~/openshift.local.clusterup" directory. The "~" is the logged in user home directory.
If your cluster setup was successful the command...
oc cluster status
... will give you a positive output like this...
Web console URL: https://<YOUR_SERVER_IP_OR_NAME>:8443/console/
Config is at host directory
Volumes are at host directory
Persistent volumes are at host directory /root/openshift.local.clusterup/openshift.local.pv
Data will be discarded when cluster is destroyed
Run OpenShift as a single node cluster service on system startup
Create OpenShift service file...
read -r -d '' FILE_CONTENT << 'HEREDOC'
BEGIN
[Unit]
Description=OpenShift oc cluster up service
After=docker.service
Requires=docker.service
[Service]
ExecStart=/usr/bin/bash -c "/usr/local/bin/oc cluster up --public-hostname=\"$(ip route get 1 | awk '{print $NF;exit}')\""
ExecStop=/usr/bin/bash -c "/usr/local/bin/oc cluster down"
Restart=no
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=occlusterup
User=root
Type=oneshot
RemainAfterExit=yes
TimeoutSec=300
[Install]
WantedBy=multi-user.target
END
HEREDOC
echo -n "${FILE_CONTENT:6:-3}" > '/etc/systemd/system/openshift.service'
NOTE: For some reason without the workaround /usr/bin/bash -c "<SOME_COMMAND>" we were unable to start the OpenShift cluster. Additional information about parameters for the oc cluster up command can be seen in the references immediately below.
[Ref(s).: https://avinetworks.com/docs/18.1/avi-vantage-openshift-installation-guide/ ,
https://github.com/openshift/origin/issues/7177#issuecomment-391478549 ,
https://github.com/minishift/minishift/issues/1910#issuecomment-375031172 ]
[Ref(s).: https://tobru.ch/openshift-oc-cluster-up-as-systemd-service/ , https://eenfach.de/gitblit/blob/RedHatTraining!agnosticd.git/af831991c7c752a1215cfc4cff6a028e31f410d7/ansible!configs!rhte-oc-cluster-vms!files!oc-cluster.service.j2 ]
Start and enable (start at boot) the OpenShift service and see the log output in sequence...
systemctl enable openshift.service
systemctl start openshift.service
journalctl -u openshift.service -f --no-pager | less
Using OpenShift OKD/Origin Admin Console
OKD includes a web console which you can use for creation and other management actions. This web console is accessible on server IP/hostname on the port 8443 via https...
https://<IP_OR_HOSTNAME>:8443/console
NOTE: You should see an OpenShift Origin page with username and password form (USERNAME: developer / PASSWORD: developer ).
Deploy a test application in the Cluster
Login to Openshift cluster as "regular developer" user (USERNAME: developer / PASSWORD: developer )...
oc login
TIP: You begin logged in as "developer".
Create a test project using oc "new-project" command...
MODEL
oc new-project <PROJECT_NAME> --display-name="<PROJECT_DISPLAY_NAME>" --description="<PROJECT_DESCRIPTION>"
EXAMPLE
oc new-project test-project --display-name="Test Project" --description="My cool Test Project."
NOTE: All commands below involving the "deployment-example" parameter value will be linked to the "test-project" because after create this project it will be selected as the project for the subsequent settings. To confirm this login as administrator using the oc login -u system:admin command and see the output of the oc status command. For more information, see the oc project <PROJECT_NAME> command in the "Some OpenShift Origin Cluster Useful Commands" section.
Tag an application image from Docker Hub registry...
oc tag --source=docker openshift/deployment-example:v2 deployment-example:latest
Deploy application to OpenShift...
MODEL
oc new-app <DEPLOYMENT_NAME>
EXAMPLE
oc new-app "deployment-example"
Allow external access to the deployed application...
MODEL
oc expose "svc/<DEPLOYMENT_NAME>"
EXAMPLE
oc expose "svc/deployment-example"
Show application deployment status...
oc status
Show pods status...
oc get pods
Get service detailed information...
oc get svc
Test Application local access...
NOTE: See <CLUSTER_IP> on command oc get svc output above.
curl http://<CLUSTER_IP>:8080
See external access route to the deployed application...
oc get routes
Test external access to the application...
Open the URL <HOST_PORT> on your browser.
MODEL
http://<HOST_PORT>
EXAMPLE
http://deployment-example-test-project.192.168.56.124.nip.io
NOTES:
See <HOST_PORT> on oc get routes output;
The wildcard DNS record *.<IP_OR_HOSTNAME>.nip.io points to OpenShift Origin server IP address.
Delete test project...
MODEL
oc delete project "<PROJECT_NAME>"
EXAMPLE
oc delete project "test-project"
[Ref(s).: https://docs.openshift.com/container-platform/4.2/applications/projects/working-with-projects.html#deleting-a-project-using-the-CLIprojects ]
Delete test deployment...
MODEL
oc delete all -l app="<DEPLOYMENT_NAME>"
EXAMPLE
oc delete all -l app="deployment-example"
Check pods status after deleting the project and the deployment...
oc get pods
TIP: Completely recreate the cluster...
oc cluster down
rm -rf ~/openshift.local.clusterup
. May be necessary reboot the server to delete the above folder;
. The "~" is the logged in user home directory.
Some OpenShift Origin Cluster Useful Commands
To login as an administrator use...
oc login -u system:admin
As administrator ("system:admin") user you can see informations such as node status...
oc get nodes
To get more detailed information about a specific node, including the reason for the current condition...
MODEL
oc describe node "<NODE_NAME>"
EXAMPLE
oc describe node "localhost"
To display a summary of the resources you created...
oc status
Select a project to perform CLI operations...
oc project "<PROJECT_NAME>"
NOTE: The selected project will be used in all subsequent operations that manipulate project-scoped content.
[Ref(s).: https://docs.openshift.com/container-platform/4.2/applications/projects/working-with-projects.html#viewing-a-project-using-the-CLI_projects ]
To return to the "regular developer" user (USERNAME: developer / PASSWORD: developer )...
oc login
To check who is the logged in user...
oc whoami
Thanks! =D
I have Wildfly 10 running on Openshift Origin 3 in AWS with an elastic ip.
I setup a Route in Openshift to map / to the wildfly service. This is working fine. If I go to http://my.ip.address I get the WildFly welcome page.
But if I map a different path, say /wf01, it doesn't work. I get a 404 Not Found error.
My guess is the router is passing along the /wf01 to the service? If that's the case, can I stop it from doing it? Otherwise how can I map http://my.ip.address/wf01 to my wildfly service?
I also want the wildfly console to be accessible from outside (this is a demo server for my own use). I added "-bmanagement","0.0.0.0" to the deploymentconfig but looking at the wildfly logs it is still binding to 127.0.0.1:
02:55:41,483 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051:
Admin console listening on http://127.0.0.1:9990
A router today cannot remap/rewrite the incoming HTTP path to another path value before passing it along. A workaround is to mount another route+service at the root that handles the root and redirects / forwards.
You can also use port-forward :
oc port-forward -h
Forward 1 or more local ports to a pod
Usage:
oc port-forward POD [LOCAL_PORT:]REMOTE_PORT [...[LOCAL_PORT_N:]REMOTE_PORT_N] [options]
Examples:
# Listens on ports 5000 and 6000 locally, forwarding data to/from ports 5000 and 6000 in the pod
$ oc port-forward -p mypod 5000 6000
# Listens on port 8888 locally, forwarding to 5000 in the pod
$ oc port-forward -p mypod 8888:5000
# Listens on a random port locally, forwarding to 5000 in the pod
$ oc port-forward -p mypod :5000
# Listens on a random port locally, forwarding to 5000 in the pod
$ oc port-forward -p mypod 0:5000