I try to install Consul HA to configure my Vault HA on OpenShift platform. I am first using Consul Helm chart to install it and I can successfully install it using some SCC for my client and server serviceAccounts.
Later on, I wanted to deploy consul with ACL init through this helm but I am facing an error.
First installing consul with helm :
helm upgrade -i consul-ha --namespace vault consul/ --set ui.enabled=true --set global.acls.manageSystemACLs=true
Then my pods quickly go into error :
oc get pods
NAME READY STATUS RESTARTS AGE
consul-ha-consul-622vr 0/1 Init:0/1 0 43s
consul-ha-consul-kxj4n 0/1 Init:0/1 0 43s
consul-ha-consul-server-0 0/1 Running 0 43s
consul-ha-consul-server-1 0/1 ContainerCreating 0 43s
consul-ha-consul-server-2 0/1 Pending 0 43s
consul-ha-consul-server-acl-init-8jf44 0/1 Error 0 33s
consul-ha-consul-server-acl-init-cleanup-dg5dk 0/1 ContainerCreating 0 14s
consul-ha-consul-server-acl-init-cleanup-xfq4m 0/1 Error 0 42s
consul-ha-consul-server-acl-init-l86r6 0/1 Error 0 43s
consul-ha-consul-wz4mf 0/1 Init:0/1 0 43s
I get this error on describing acl-init pod :
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned vault/consul-ha-consul-server-acl-init-hrsk2 to compute-1
Warning Failed 2m12s kubelet, compute-1 Error: container create failed: time="2020-10-05T07:46:12Z" level=warning msg="signal: killed"
time="2020-10-05T07:46:12Z" level=error msg="container_linux.go:349: starting container process caused \"process_linux.go:449: container init caused \\\"read init-p: connection reset by peer\\\"\""
container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"read init-p: connection reset by peer\""
Warning Failed 2m3s kubelet, compute-1 Error: container create failed: time="2020-10-05T07:46:22Z" level=warning msg="signal: killed"
time="2020-10-05T07:46:22Z" level=error msg="container_linux.go:349: starting container process caused \"process_linux.go:365: sending config to init process caused \\\"write init-p: broken pipe\\\"\""
container_linux.go:349: starting container process caused "process_linux.go:365: sending config to init process caused \"write init-p: broken pipe\""
Warning Failed 114s kubelet, compute-1 Error: container create failed: time="2020-10-05T07:46:31Z" level=fatal msg="join_namespaces:542 nsenter: failed to open /proc/1372777/ns/ipc: No such file or directory"
time="2020-10-05T07:46:31Z" level=fatal msg="nsexec:724 nsenter: failed to sync with child: next state: Invalid argument"
time="2020-10-05T07:46:31Z" level=error msg="container_linux.go:349: starting container process caused \"process_linux.go:319: getting the final child's pid from pipe caused \\\"EOF\\\"\""
container_linux.go:349: starting container process caused "process_linux.go:319: getting the final child's pid from pipe caused \"EOF\""
Warning Failed 103s kubelet, compute-1 Error: container create failed: time="2020-10-05T07:46:42Z" level=warning msg="signal: killed"
time="2020-10-05T07:46:42Z" level=error msg="container_linux.go:349: starting container process caused \"process_linux.go:449: container init caused \\\"read init-p: connection reset by peer\\\"\""
container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"read init-p: connection reset by peer\""
Warning Failed 94s kubelet, compute-1 Error: container create failed: time="2020-10-05T07:46:51Z" level=warning msg="signal: killed"
time="2020-10-05T07:46:51Z" level=error msg="container_linux.go:349: starting container process caused \"process_linux.go:365: sending config to init process caused \\\"write init-p: broken pipe\\\"\""
container_linux.go:349: starting container process caused "process_linux.go:365: sending config to init process caused \"write init-p: broken pipe\""
Warning Failed 81s kubelet, compute-1 Error: container create failed: time="2020-10-05T07:47:04Z" level=fatal msg="join_namespaces:542 nsenter: failed to open /proc/1376195/ns/ipc: No such file or directory"
time="2020-10-05T07:47:04Z" level=fatal msg="nsexec:724 nsenter: failed to sync with child: next state: Invalid argument"
time="2020-10-05T07:47:04Z" level=error msg="container_linux.go:349: starting container process caused \"process_linux.go:319: getting the final child's pid from pipe caused \\\"EOF\\\"\""
container_linux.go:349: starting container process caused "process_linux.go:319: getting the final child's pid from pipe caused \"EOF\""
Warning Failed 73s kubelet, compute-1 Error: container create failed: time="2020-10-05T07:47:12Z" level=fatal msg="join_namespaces:542 nsenter: failed to open /proc/1377778/ns/ipc: No such file or directory"
time="2020-10-05T07:47:12Z" level=fatal msg="nsexec:724 nsenter: failed to sync with child: next state: Invalid argument"
time="2020-10-05T07:47:12Z" level=error msg="container_linux.go:349: starting container process caused \"process_linux.go:319: getting the final child's pid from pipe caused \\\"EOF\\\"\""
container_linux.go:349: starting container process caused "process_linux.go:319: getting the final child's pid from pipe caused \"EOF\""
Normal SandboxChanged 62s (x8 over 2m12s) kubelet, compute-1 Pod sandbox changed, it will be killed and re-created.
Warning Failed 62s kubelet, compute-1 Error: container create failed: time="2020-10-05T07:47:23Z" level=warning msg="signal: killed"
time="2020-10-05T07:47:23Z" level=error msg="container_linux.go:349: starting container process caused \"process_linux.go:449: container init caused \\\"read init-p: connection reset by peer\\\"\""
container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"read init-p: connection reset by peer\""
Normal Pulled 56s (x9 over 2m19s) kubelet, compute-1 Container image "hashicorp/consul-k8s:0.18.1" already present on machine
I can't figure out what's going on. Any idea on this ?
Related
I have a galera cluster with 3 nodes in QA environement.. There was a abrupt shut down of the nodes and I restarted and bootstrapped the first nodewhich is working fine. The second node when I try to start gives the error , "rsyncd[30958]: unknown module 'rsync_sst-undo_dir' tried from XXX" .
In mysql log :
[Warning] WSREP: 0.0 : State transfer to 1.0 (x) failed: -255 (Unknown error 255) [ERROR] WSREP: /home/buildbot/buildbot/build/gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1207: Will never receive state. Need to abort.
Not sure what causes this error,
rsyncd[30958]: unknown module 'rsync_sst-undo_dir' tried from XXX systemd[1]: mariadb.service: main process exited, code=killed, status=6/ABRT unknown module 'rsync_sst-undo_dir' "
Any pointers would be very helpful .
Thanks
Any pointers would be very helpful .
previously my MySQL pod stuck at terminating status, and then I tried to force delete using command like this
kubectl delete pods <pod> --grace-period=0 --force
Later I tried to helm upgrade again, my pod was stuck at containercreating status, and this event from pod
17s Warning FailedMount pod/db-mysql-primary-0 MountVolume.SetUp failed for volume "pvc-f32a6f84-d897-4e35-9595-680302771c54" : kubernetes.io/csi: mount
er.SetUpAt failed to check for STAGE_UNSTAGE_VOLUME capability: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix
/var/lib/kubelet/plugins/dobs.csi.digitalocean.com/csi.sock: connect: no such file or directory"
17s Warning FailedMount pod/db-mysql-secondary-0 MountVolume.SetUp failed for volume "pvc-61fc6eda-97fa-455f-ac2c-df8ebcb90f1c" : kubernetes.io/csi: mount
er.SetUpAt failed to check for STAGE_UNSTAGE_VOLUME capability: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix
/var/lib/kubelet/plugins/dobs.csi.digitalocean.com/csi.sock: connect: no such file or directory"
anyone please can help me to resolve this issue, thanks a lot.
When you run the command
kubectl delete pods <pod> --grace-period=0 --force
you ask kubernetes to forget the Pod, not to delete it. You have to be careful while using this command. You have to make sure that the containers of the Pod are not running in the host especially when they are mounted to a PVC. Probably the containers is still running and attached to the PVC.
pool-product-8jd40 0
spec:
drivers: null
and on my some pool the driver csi not ready (null), it's supposed to be equal 1 (ready)
*sorry i can't attach the image yet
I am trying to deploy django web application on alibabacloud everything seems to be working perfectly(running gunicorn --bind 0.0.0.0:8000 project_name.wsgi on virtual environment)
Then after deactivating the virtual environment and setting up
[Unit]
Description=gunicorn daemon
After=network.target
[Service]
User=admin
Group=www-data
WorkingDirectory=/home/admin/project_name
ExecStart=/home/admin/project_name/myprojectenv/bin/gunicorn --access-logfile - --workers 3 --bind
unix:/home/admin/project_name/project_name.sock project_name.wsgi:application
in /etc/systemd/system/gunicorn.service
then running sudo systemctl start gunicorn I keep getting the error
Failed to start gunicorn.service: Unit gunicorn.service is masked.
Please how can I fix this?
I have tried systemctl unmask gunicorn.socket but it keeps showing me the error
Unit gunicorn.socket does not exist, proceeding anyway.
Failed to unmask unit: The name org.freedesktop.PolicyKit1 was not provided
by any .service files
I have installed mongodb 2.4.14 using this guide.
But After Installing the mongodb I'm getting the errors just before completion. I have searched a lot but unable to eliminate those errors. Below is the error
initctl: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused
insserv: warning: script 'mongodb' missing LSB tags and overrides
insserv: Default-Start undefined, assuming empty start runlevel(s) for script mongodb'
insserv: Default-Stop undefined, assuming empty stop runlevel(s) for scriptmongodb'
initctl: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused
insserv: warning: script 'mongodb' missing LSB tags and overrides
insserv: Default-Start undefined, assuming empty start runlevel(s) for script mongodb'
insserv: Default-Stop undefined, assuming empty stop runlevel(s) for scriptmongodb'
And after installing mysql I am also getting errors
Renaming removed key_buffer and myisam-recover options (if present)
initctl: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused
insserv: warning: script 'mongodb' missing LSB tags and overrides
insserv: Default-Start undefined, assuming empty start runlevel(s) for script mongodb'
insserv: Default-Stop undefined, assuming empty stop runlevel(s) for scriptmongodb'
Setting up mysql-server (5.7.15-0ubuntu0.16.04.1) ...
Processing triggers for libc-bin (2.23-0ubuntu3) ...
Processing triggers for systemd (229-4ubuntu11) ...
Processing triggers for ureadahead (0.100.0-19) ...
Please Help
The problem is the upstart has replaced the init prior to ubuntu 15 so either you can edit the upstart script or you can install 14.04
mariadb 10.1.x galera cluster setting.
first node 192.168.159.132
/etc/mysql/my.cnf
[galera]
# Mandatory settings
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_address=gcomm://192.168.159.132,192.168.159.135,192.168.159.136
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
first node 192.168.159.132
$ sudo service mysql bootstrap
* Bootstrapping the cluster mysqld
Starting mysql (via systemctl): mysql.service
Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.
$ systemctl status mariadb.service
[ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -110 (Connection timed out)
[ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1379: Failed to open channel 'my_wsrep_cluster' at 'gcomm://192.168.32.251,192.168.32.252,192.168.32...ection timed out)
[ERROR] WSREP: gcs connect failed: Connection timed out
[ERROR] WSREP: wsrep::connect(gcomm://192.168.32.251,192.168.32.252,192.168.32.253) failed: 7
[ERROR] Aborting
systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: Failed to start MariaDB database server.
systemd[1]: mariadb.service: Unit entered failed state.
systemd[1]: mariadb.service: Failed with result 'exit-code'.
Why "Galera Cluster" not Start?
How To checking "Connection Time Out"?
There is a Error in your config. After gcom you only use one : like:
wsrep_cluster_address=gcomm://192.168.159.132,192.168.159.135,192.168.159.136
their is an error:
use format as below:
wsrep_cluster_address="gcomm://first_ip,second_ip,third_ip"
The galera configuration looks fine. But you have to provide an extra argument while bootstrapping the first node. Try this:
mysqld --wsrep-new-cluster
For the first node wsrep_cluster_address can be empty. For next nodes you have to provide at least one ip address that is already a part of cluster. But it's a good practice to provide the list of all the nodes that you are going to use. It wont give you any error for mentioning a node in cluster-address which is not present in the cluster yet. So you can use, wsrep_cluster_address=gcomm://192.168.159.132,192.168.159.135,192.168.159.136 for every node.
You are getting "connection timed out" because the node is trying to connect to the other nodes listed in wsrep_cluster_address, which aren't online as this is the very first node in the cluster. So, during bootstrap you should start the first node with empty wsrep_cluster_address :.wsrep_cluster_address='gcomm://'.