First time posting on StackOverflow so please be gentle!
I'm setting up a new RHEL8 server to run Podman. Previously, I've done this on a pretty vanilla server but this one is setup in line with our corporate image. This means a homedir that is mounted over NFS.
When I try a simple podman command such as podman run centos, I get a couple of errors (see below). According to https://github.com/containers/podman/blob/main/rootless.md, Podman non-root is known to have problems with NFS homedirs.
Output from podman run centos (and others):
❯ podman run centos
Resolved "centos" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull quay.io/centos/centos:latest...
Getting image source signatures
Copying blob 7a0437f04f83 done
Error: writing blob: adding layer with blob "sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621": Error processing tar file(exit status 1): Error setting up pivot dir: mkdir /home/me/.local/share/containers/storage/overlay/2653d992f4ef2bfd27f94db643815aa567240c37732cae1405ad1c1309ee9859/diff/.pivot_root926823499: permission denied
No, my username isn't really 'me'
Is there a way to use podman non-root in this setup? I'd prefer to avoid creating a local user account to run things under (this is my dev server and isn't where the application will actually be running but will involve me building, running, destroying regularly so I'd rather avoid having to do anything 'clever')
Output of podman info:
❯ podman info
host:
arch: amd64
buildahVersion: 1.23.1
cgroupControllers: []
cgroupManager: cgroupfs
cgroupVersion: v1
conmon:
package: conmon-2.0.32-1.module+el8.5.0+13852+150547f7.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.0.32, commit: 4b12bce835c3f8acc006a43620dd955a6a73bae0'
cpus: 1
distribution:
distribution: '"rhel"'
version: "8.5"
eventLogger: file
hostname: servername
idMappings:
gidmap:
- container_id: 0
host_id: 2000
size: 1
uidmap:
- container_id: 0
host_id: 10279927
size: 1
kernel: 4.18.0-348.12.2.el8_5.x86_64
linkmode: dynamic
logDriver: k8s-file
memFree: 1881419776
memTotal: 3918233600
ociRuntime:
name: runc
package: runc-1.0.3-1.module+el8.5.0+13556+7f055e70.x86_64
path: /usr/bin/runc
version: |-
runc version 1.0.3
spec: 1.0.2-dev
go: go1.16.7
libseccomp: 2.5.1
os: linux
remoteSocket:
path: /run/user/10279927/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: true
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.1.8-1.module+el8.5.0+12582+56d94c81.x86_64
version: |-
slirp4netns version 1.1.8
commit: d361001f495417b880f20329121e3aa431a8f90f
libslirp: 4.4.0
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.1
swapFree: 4294963200
swapTotal: 4294963200
uptime: 2h 45m 20.28s (Approximately 0.08 days)
plugins:
log:
- k8s-file
- none
- journald
network:
- bridge
- macvlan
volume:
- local
registries:
search:
- registry.fedoraproject.org
- registry.access.redhat.com
- registry.centos.org
- docker.io
store:
configFile: /home/me/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions:
overlay.mount_program:
Executable: /usr/bin/fuse-overlayfs
Package: fuse-overlayfs-1.8-1.module+el8.5.0+13754+92ec836b.x86_64
Version: |-
fusermount3 version: 3.2.1
fuse-overlayfs: version 1.8
FUSE library version 3.2.1
using FUSE kernel interface version 7.26
graphRoot: /home/me/.local/share/containers/storage
graphStatus:
Backing Filesystem: nfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "false"
imageStore:
number: 0
runRoot: /run/user/10279927/containers
volumePath: /home/me/.local/share/containers/storage/volumes
version:
APIVersion: 3.4.2
Built: 1642068949
BuiltTime: Thu Jan 13 10:15:49 2022
GitCommit: ""
GoVersion: go1.16.7
OsArch: linux/amd64
Version: 3.4.2
Thank you!
Based on this article, https://www.redhat.com/sysadmin/rootless-podman-nfs, podman and nfs home directories don't mix well together.
This is worked around by changing the graphroot(which is described in the above article) to write to a local, non-nfs, location.
Related
I'm trying to use Kubernetes with Docker. My image runs with Docker. I have one master-node and two worker-nodes. I also created a local registry like this $ docker run -d -p 5000:5000 --restart=always --name registry registry:2 and pushed my image into it. Everything worked fine so far.
I added { "insecure-registries":["xxx.xxx.xxx.xxx:5000"] } to the daemon.json file at /etc/docker. And I also changed the content of the docker-file at /etc/default/to DOCKER_OPTS="--config-file=/etc/docker/daemon.json". I made the changes on all nodes and I restarted the docker daemon afterwards.
I am able to pull my image from every node with the following command:
sudo docker pull xxx.xxx.xxx.xxx:5000/helloworldimage
I try to create my container from the master node with the command bellow:
sudo kubectl run test --image xxx.xxx.xxx.xxx:5000/helloworldimage
Than I get the following error:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned default/test-775f99f57-m9r4b to rpi-2
Normal BackOff 18s (x2 over 44s) kubelet, rpi-2 Back-off pulling image "xxx.xxx.xxx.xxx:5000/helloworldimage"
Warning Failed 18s (x2 over 44s) kubelet, rpi-2 Error: ImagePullBackOff
Normal Pulling 3s (x3 over 45s) kubelet, rpi-2 Pulling image "xxx.xxx.xxx.xxx:5000/helloworldimage"
Warning Failed 3s (x3 over 45s) kubelet, rpi-2 Failed to pull image "xxx.xxx.xxx.xxx:5000/helloworldimage": rpc error: code = Unknown desc = failed to pull and unpack image "xxx.xxx.xxx.xxx:5000/helloworldimage:latest": failed to resolve reference "xxx.xxx.xxx.xxx:5000/helloworldimage:latest": failed to do request: Head https://xxx.xxx.xxx.xxx:5000/v2/helloworldimage/manifests/latest: http: server gave HTTP response to HTTPS client
Warning Failed 3s (x3 over 45s) kubelet, rpi-2 Error: ErrImagePull
This is the docker version I use:
Client: Docker Engine - Community
Version: 19.03.5
API version: 1.40
Go version: go1.12.12
Git commit: 633a0ea
Built: Wed Nov 13 07:37:22 2019
OS/Arch: linux/arm
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.5
API version: 1.40 (minimum version 1.12)
Go version: go1.12.12
Git commit: 633a0ea
Built: Wed Nov 13 07:31:17 2019
OS/Arch: linux/arm
Experimental: false
containerd:
Version: 1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683
This is the Kubernetes version I use:
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0+k3s.1", GitCommit:"0f644650f5d8e9f091629f860b342f221c46f6d7", GitTreeState:"clean", BuildDate:"2020-01-06T23:20:30Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/arm"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0+k3s.1", GitCommit:"0f644650f5d8e9f091629f860b342f221c46f6d7", GitTreeState:"clean", BuildDate:"2020-01-06T23:20:30Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/arm"}```
Kubernetes: Failed to pull image. Server gave HTTP response to HTTPS client.
{ "insecure-registries":["xxx.xxx.xxx.xxx:5000"] }
to the daemon.json file at /etc/docker.
I solved this problem by configuring it on all kubernetes nodes.
It appears that in some situations solution described here solved the problem:
sudo systemctl edit docker
Add below lines:
[Service]
ExecStart=/usr/bin/dockerd -H fd:// --insecure-registry registry:5000
sudo systemctl daemon-reload
systemctl restart docker
systemctl status docker
using openshift, and one pod keep pending, because nfs server cannot be mounted (nfs server is able to be mounted by mannually using command line, but cannot be mounted from the Pod)
I have installed nfs-common, so it's not the root cause. I trying to install nfs-utils, but I was failed, the error message is:
E: Unable to locate package: nfs-utils.
I also tried libnfs12 and libnfs-utils, they were the same as nfs-utils. I also used apt-get install upgade and update to solve the package locating problem, but they were useless.
I'm going to show the yaml file for connecting the nfs server
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-test01
lables:
disktype: baas
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteMany
nfs:
path: /baas
server: 9.111.140.47
readOnly: false
persistentVolumeReclaimPolicy: Recycle
after using "oc describe pod/mypod" for the pending Pod, below is the feedback:
Warning FailedMount 14s kubelet, localhost MountVolume.SetUp failed for volume "pv-test01" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/origin/cluster-up/root/openshift.local.clusterup/openshift.local.volumes/pods/267db6f2-d875-11e9-80ba-005056bc3ce0/volumes/kubernetes.io~nfs/pv-test01 --scope -- mount -t nfs 9.111.140.47:/baas /var/lib/origin/cluster-up/root/openshift.local.clusterup/openshift.local.volumes/pods/267db6f2-d875-11e9-80ba-005056bc3ce0/volumes/kubernetes.io~nfs/pv-test01
Output: Running scope as unit run-28094.scope.
mount: wrong fs type, bad option, bad superblock on 9.111.140.47:/baas,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount.<type> helper program)
In some cases useful info is found in syslog - try
dmesg | tail or so.
so how can I mount to nfs server from the Pod? should I keep installing nfs-utils? If yes, how can I install it?
I need some inputs/help to bring up my container(customized) on OpenShift
[mag-vm#mag-vm-centos-2 ~]$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
mag_main latest e039447d7212 13 days ago 1.316 GB
[mag-vm#mag-vm-centos-2 ~]$ oc import-image mag_main:latest
error: no image stream named "mag_main" exists, pass --confirm to create and import
[mag-vm#mag-vm-centos-2 ~]$ oc import-image mag_main:latest --confirm
The import completed with errors.
Name: mag_main
Namespace: cirrus
Created: Less than a second ago
Labels: <none>
Annotations: openshift.io/image.dockerRepositoryCheck=2017-06-07T16:24:49Z
Docker Pull Spec: 172.30.124.119:5000/cirrus/mag_main
Unique Images: 0
Tags: 1
latest
tagged from mag_main:latest
! error: Import failed (Unauthorized): you may not have access to the Docker image "mag_main:latest"
Less than a second ago
[mag-vm#mag-vm-centos-2 ~]$
Could you pls help me to overcome this issue, is it something to do with the "secret" settings?
Thanks in Advance.
Also as an additional input, I am able to bring up the container for this docker image using "docker run" command
STEP #1 : sudo docker run -t mag_main:latest /bin/bash
STEP #2 : Once the container is up, I used "./bin/karaf" to run the services inside this docker container
Pls let me know, how can I do the same from the OpenShift.
OpenShift Details;
[mag-vm#mag-vm-centos-2 ~]$ oc version
oc v1.5.1+7b451fc
kubernetes v1.5.2+43a9be4
features: Basic-Auth GSSAPI Kerberos SPNEGO
Server https://10.100.71.160:8443
openshift v1.5.1+7b451fc
kubernetes v1.5.2+43a9be4
[mag-vm#mag-vm-centos-2 ~]$
Docker Details;
[mag-vm#mag-vm-centos-2 ~]$ sudo docker version
Client:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:23:11 2016
OS/Arch: linux/amd64
Server:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:23:11 2016
OS/Arch: linux/amd64
[mag-vm#mag-vm-centos-2 ~]$
I used docker-machine with Azure as the driver to spin up a VM. I then deployed a simple nginx test container on to the host. My issue is that when I try to set and endpoint I am getting the following error:
azure vm endpoint create huldra 80 32769
info: Executing command vm endpoint create
+ Getting virtual machines
+ Reading network configuration
+ Updating network configuration
error: Parameter 'ConsoleScreenshotBlobUri' should not be set.
info: Error information has been recorded to /Users/ryan/.azure/azure.err
error: vm endpoint create command failed
When I look at the error log it pretty much repeats what the console said Parameter 'ConsoleScreenshotBlobUri' should not be set.
Here are my docker and azure environment details:
❯ docker info
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 3
Server Version: 1.10.2
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 21
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 4.2.0-18-generic
Operating System: Ubuntu 15.10
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.636 GiB
Name: huldra
ID: PHUY:JRE3:DOJO:NNWO:JBBH:42H2:56ZO:HVSB:MZDE:QLOI:GO6F:SCC5
WARNING: No swap limit support
Labels:
provider=azure
~/Projects/dockerswarm master*
❯ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ce51127b2bb8 nginx "nginx -g 'daemon off" 11 minutes ago Up 11 minutes 0.0.0.0:32769->80/tcp, 0.0.0.0:32768->443/tcp machinenginx
❯ azure --version
0.9.17 (node: 5.8.0)
❯ azure vm list
info: Executing command vm list
+ Getting virtual machines
data: Name Status Location DNS Name IP Address
data: ------ --------- -------- ------------------- -------------
data: huldra ReadyRole West US huldra.cloudapp.net x.x.x.x
info: vm list command OK
As previously reported here, two pods can't mount the same disk even though one of them tries to do it as read-only mode.
This is supposed to be allowed from Kubernetes documentation.
Mounting scheme is:
UniqueCluster/PodA has successfully mounted gdeDisk1 as read-write
UniqueCluster/PodB fails to start when mounting gdeDisk1 as read-only
Node description:
Name: gke-zupcat-cluster-8fd35d81-node-1zr4
Labels: kubernetes.io/hostname=gke-zupcat-cluster-8fd35d81-node-1zr4
CreationTimestamp: Wed, 22 Jul 2015 14:47:56 -0300
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
Ready True Thu, 23 Jul 2015 12:06:18 -0300 Wed, 22 Jul 2015 22:53:34 -0300 kubelet is posting ready status
Addresses: 10.240.17.72,146.148.79.174
Capacity:
cpu: 2
memory: 7679608Ki
pods: 40
Version:
Kernel Version: 3.16.0-0.bpo.4-amd64
OS Image: Debian GNU/Linux 7 (wheezy)
Container Runtime Version: docker://Unknown
Kubelet Version: v1.0.1
Kube-Proxy Version: v1.0.1
PodCIDR: 10.108.0.0/24
ExternalID: 11953122931827361742
Pods: (5 in total)
Namespace Name
default fastrwdiskpod-yu517
kube-system fluentd-cloud-logging-gke-zupcat-cluster-8fd35d81- node-1zr4
kube-system kube-dns-v8-i3h20
kube-system kube-ui-v1-8zdrq
kube-system monitoring-heapster-v5-e1zmi
No events.
Products versions:
Client Version: version.Info{Major:"1", Minor:"0", GitVersion:"v1.0.0", GitCommit:"cd821444dcf3e1e237b5f3579721440624c9c4fa", GitTreeState:"clean"}
Server Version: version.Info{Major:"1", Minor:"0", GitVersion:"v1.0.1", GitCommit:"6a5c06e3d1eb27a6310a09270e4a5fb1afa93e74", GitTreeState:"clean"}
docker version Docker version 1.7.1, build 786b29d
According to the GCE persistent disk documentation: "if you attach a persistent disk to multiple instances, all instances must attach the persistent disk in read-only mode."
The Kubernetes documentation for GCE PD volumes also explains this limitation: "A feature of PD is that they can be mounted as read-only by multiple consumers simultaneously. This means that you can pre-populate a PD with your dataset and then serve it in parallel from as many pods as you need. Unfortunately, PDs can only be mounted by a single consumer in read-write mode - no simultaneous readers allowed."