I am trying the "ctr" cli with containerd, I have a use case where I want to migrate/clone/copy the running container. I figured "checkpoint" is something I should look into, didn't find much on official documentation. Also, there are 2 subcommands for the checkpoint, for container and task.
Container
root#aks:/# ctr container checkpoint --help
NAME:
ctr containers checkpoint - checkpoint a container
USAGE:
ctr containers checkpoint [command options] CONTAINER REF
OPTIONS:
--rw include the rw layer in the checkpoint
--image include the image in the checkpoint
--task checkpoint container task
Task
spawner git:(config_in_vault) ctr task checkpoint --help
NAME:
ctr tasks checkpoint - checkpoint a container
USAGE:
ctr tasks checkpoint [command options] [flags] CONTAINER
OPTIONS:
--exit stop the container after the checkpoint
--image-path value path to criu image files
--work-path value path to criu work files and logs
Any idea what's the difference?
PS- I asked the same question on containerd repo:
https://github.com/containerd/containerd/discussions/7088
Related
My customer asked me if openshift can provide the same control on memory usage as docker can, for example, docker run can have the following parameters to control memory usage when running a container:
--kernel-memory
--memory
--memory-reservation
While I searched the corresponding part in openshift, I found ResoureQuota and LimitRange should work for that, but what if a pod claims itself will use 100Mi memory by using LimitRange but actually it will consume 500Mi memory instead? the memory can still be used "illegally", seems docker with --memory can control this situation more better.
In openshift, is there any method for controlling real memory usage instead of checking what a pod claimed in LimitRange or using "oc set resources dc hello --requests=memory=256Mi"?
Best regards
Lan
As far as my experience with Openshift I have not come across the situation where the POD has consumed more memory or CPU for which it has configured. If in case it reaches the threshold, the POD automatically will be killed and restarts.
You can set the POD resource limits in the Deployment config:
resources:
limits:
cpu: 750m
memory: 1024Mi
The resources can be monitored in the metrics section of the respective POD:
Apart from the indiviual POD settings you can define your own overall project settings for each container in the POD.
$ oc get limits
NAME
limits
$ oc describe limits <NAME>
Name: <NAME>
Namespace: <NAME_SPACE>
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Pod memory 256Mi 32Gi - - -
Pod cpu 125m 6400m - - -
Container cpu 125m 6400m 125m 750m -
Container memory 256Mi 32Gi 512Mi 1Gi -
For more information on resource settings refer here.
If you only use --requests=memory=256Mi, you set QoS level to "burstable", which means pod can request at least 256Mi memory without upper limit except reaching project quota. If you want to limit pod memory, use --limit=memory=256Mi instead.
I am trying to run an sonatype/nexus3 on openshift online v3 pro. If I just use the web console to create a new app from image it assigns it only 512Mi and it dies with OOM. It did get created though and logged a lot of java output before it died of out of memory. When using the web console there doesnt appear a way to set the memory on the image. When I try to edited the yaml of the pod it doesn't let me edited the memory limit.
Reading the docs about memory limits it suggests that I can run with this:
oc run nexus333 --image=sonatype/nexus3 --limits=memory=750Mi
Then it doesn't even start. It dies with:
{kubelet ip-172-31-59-148.ec2.internal} Error: Error response from
daemon: {"message":"create
c30deb38b3c26252bf1218cc898fbf1c68d8fc14e840076710c211d58ed87a59:
mkdir
/var/lib/docker/volumes/c30deb38b3c26252bf1218cc898fbf1c68d8fc14e840076710c211d58ed87a59:
permission denied"}
More information from oc get events:
FIRSTSEEN LASTSEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE
16m 16m 1 nexus333-1-deploy Pod Normal Scheduled {default-scheduler } Successfully assigned nexus333-1-deploy to ip-172-31-50-97.ec2.internal
16m 16m 1 nexus333-1-deploy Pod spec.containers{deployment} Normal Pulling {kubelet ip-172-31-50-97.ec2.internal} pulling image "registry.reg-aws.openshift.com:443/openshift3/ose-deployer:v3.6.173.0.21"
16m 16m 1 nexus333-1-deploy Pod spec.containers{deployment} Normal Pulled {kubelet ip-172-31-50-97.ec2.internal} Successfully pulled image "registry.reg-aws.openshift.com:443/openshift3/ose-deployer:v3.6.173.0.21"
15m 15m 1 nexus333-1-deploy Pod spec.containers{deployment} Normal Created {kubelet ip-172-31-50-97.ec2.internal} Created container
15m 15m 1 nexus333-1-deploy Pod spec.containers{deployment} Normal Started {kubelet ip-172-31-50-97.ec2.internal} Started container
15m 15m 1 nexus333-1-rftvd Pod Normal Scheduled {default-scheduler } Successfully assigned nexus333-1-rftvd to ip-172-31-59-148.ec2.internal
15m 14m 7 nexus333-1-rftvd Pod spec.containers{nexus333} Normal Pulling {kubelet ip-172-31-59-148.ec2.internal} pulling image "sonatype/nexus3"
15m 10m 19 nexus333-1-rftvd Pod spec.containers{nexus333} Normal Pulled {kubelet ip-172-31-59-148.ec2.internal} Successfully pulled image "sonatype/nexus3"
15m 15m 1 nexus333-1-rftvd Pod spec.containers{nexus333} Warning Failed {kubelet ip-172-31-59-148.ec2.internal} Error: Error response from daemon: {"message":"create 3aa35201bdf81d09ef4b09bba1fc843b97d0339acfef0c30cecaa1fbb6207321: mkdir /var/lib/docker/volumes/3aa35201bdf81d09ef4b09bba1fc843b97d0339acfef0c30cecaa1fbb6207321: permission denied"}
I am not sure why if I use the web console I cannot assign more memory. I am not sure why running it with oc run dies with the mkdir error. Can anyone tell me how to run sonatype/nexus3 on openshift online pro?
Looking in the documentation I see that it is a Java VM solution.
When using Java 8, memory usage can be DRAMATICALLY IMPROVED using only the following 2 runtime Java VM options:
... "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap" ...
I just deployed my container (Spring Boot JAR) that consumed over 650 MB RAM. With just these two (new) options RAM consumption dropped to just 270 MB!!!
So, with these 2 runtime settings all OOM's are left far behind! Enjoy!
You may want to also follow along with the tutorial that is in the OpenShift docs https://docs.openshift.com/online/dev_guide/app_tutorials/maven_tutorial.html
I have had success deploying this in OpenShift Online Pro
Okay the mkdir /var/lib/docker/volumes/ permission denied seems to be that the image needs a /nexus-data mount and that is refused. I saw that by deploying from the web console (dies with OOM) but the edit yaml for the created pod to see the generated volume mount.
Creating the image with the following yaml using cat nexus3_pod.ephemeral.yaml | oc create -f - with the volume mount and explicit memory settings the container will now start up:
apiVersion: "v1"
kind: "Pod"
metadata:
name: "nexus3"
labels:
name: "nexus3"
spec:
containers:
-
name: "nexus3"
resources:
requests:
memory: "1200Mi"
limits:
memory: "1200Mi"
image: "sonatype/nexus3"
ports:
-
containerPort: 8081
name: "nexus3"
volumeMounts:
- mountPath: /nexus-data
name: nexus3-1
volumes:
- emptyDir: {}
name: nexus3-1
Notes
The mage sets -Xmx1200m as documented at sonatype/docker-nexus3. So if you assign memory less than 1200Mi it will crash with OOM when the heap grows over the limit. You may as well set requested and max to be the max heap side anything.
When the allocated memory was too low it crashed die just as it was setting up the DB which corrupted the db log which meant it then got in a crash loop "couldn't load 4 byte from 0 byte file" when I recreated it with more memory. It seems that with an emptyDir the files hang around between crash restarts and memory changes (that's documented behaviour I think). I had to recreate a pod with a different name to get a clean emptyDir and assigned memory of 1200Mi to get it to all start.
Description of problem:
On Centos machine, if we try to delete the SCSI disk image for running vm (which is added via virt-manager) through virsh vol-delete command, getting "cannot unlink file 'XXX': Success" error.
This error occurs intermittently and occurs only if we try to delete disk image when vm is running.
Version-Release number of selected component (if applicable):
OS:CentOS Linux release 7.2.1511
Libvirt:
[root#CV-HJ-CentOS7-02 images]# virsh version
Compiled against library: libvirt 1.2.17
Using library: libvirt 1.2.17
Using API: QEMU 1.2.17
Running hypervisor: QEMU 1.5.3
Steps to Reproduce:
Add SCSI disk from virt manager to VM
Start VM from virt-manager and confirm disk is attached as SCSI.
Try to delete the newly added SCSI disk using virsh vol-delete command:
virsh # vol-delete /var/lib/libvirt/images/.img
Actual results:
It is giving Following error:
error: Failed to delete vol /var/lib/libvirt/images/.img
error: cannot unlink file '/var/lib/libvirt/images/.img': Success
It looks like you're trying to delete the disk image before detaching it from the running VM which isn't allowed. You'll need to detach the disk first, then do a pool refresh and then you'll be able to delete it.
Here's an example using "f23-tst_default" as the name of my VM (domain) and a disk named "f23-test_default.qcow2" which I want to remove:
# virsh domblklist f23-tst_default
Target Source
------------------------------------------------
vda /var/lib/libvirt/images/f23-tst_default.img
sda /var/lib/libvirt/images/f23-tst_default.qcow2
# virsh detach-disk f23-tst_default --target sda
Disk detached successfully
# virsh domblklist f23-tst_default
Target Source
------------------------------------------------
vda /var/lib/libvirt/images/f23-tst_default.img
# virsh pool-refresh default
Pool default refreshed
# virsh vol-delete --pool default f23-tst_default.qcow2
Vol f23-tst_default.qcow2 deleted
If you don't do a 'pool-refresh' then virsh doesn't realize that the domain is no longer using the volume and, therefore, won't allow you to remove it.
I faced the same issue and it was a file permissions problem. I too faced it after upgrading from Centos 7.1 to 7.2.
To resolve make sure that the owner of the directory where the image is stored (default pool is /var/lib/libvirt/images) is the one defined in "user" option in /etc/libvirt/qemu.conf (default user is qemu).
If you haven't touched the defaults then:
# chown qemu:qemu /var/lib/libvirt/images
Then create a new image and try to delete it. It should succeed.
I'd like to write a script that can be run from a Google Compute instance, which creates a disk and mounts it. The disks I've created and mounted so far have been done through the web console. The problem I'm having is in figuring out the paramaters for safe_format_and_mount (and possibly in some step before).
From within the instance, here is my attempt so far:
ami#snowflake:~$ gcloud compute disks create foo --zone europe-west1-c
Created [https://www.googleapis.com/compute/v1/projects/snowflake- 1056/zones/europe-west1-c/disks/foo].
NAME ZONE SIZE_GB TYPE STATUS
foo europe-west1-c 500 pd-standard READY
ami#snowflake:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 10G 0 disk
`-sda1 8:1 0 10G 0 part /
sdb 8:16 0 500G 0 disk /mnt/work
sdc 8:32 0 2T 0 disk /mnt/data1
The docs for safe_format_and_mount give now this sequence:
$ sudo mkdir MOUNT_POINT
$ sudo /usr/share/google/safe_format_and_mount -m "mkfs.ext4 -F" DISK_LOCATION MOUNT_POINT
However, I have no idea what DISK_LOCATION is, nor does lsblock's output give anything that seems pertinent.
Don't forget that you need to attach the disk to your instance before you can use it:
gcloud compute instances attach-disk myinstance --disk foo \
--zone europe-west1-c --device-name foo
The --device-name option allows you to specify the device name the guest operating system will see. If you use the same name as the disk name, the disk location will be /dev/disk/by-id/google-foo.
I am following this guide https://developers.google.com/compute/docs/troubleshooting#ssherrors specifically the section about recovering your persistent disk with another vm.
I am trying to follow this part:
mount /dev/disk/by-id/scsi-0Google_PersistentDisk_myinstance-debugging /mnt/myinstance
This is the error I get:
root#debugger:~# mount /dev/disk/by-id/scsi-0Google_PersistentDisk_marty-wll-debugging /mnt/marty-wll
mount: you must specify the filesystem type
I am unsure of the filesystem due to google-compute disks being used, and the system has already been deleted and attached to another machine following the google developers guide I referenced above.
parted scsi-0Google_PersistentDisk_marty-wll-debugging -l
root#debugger:/dev/disk/by-id# parted scsi-0Google_PersistentDisk_marty-wll-debugging -l
Model: Google PersistentDisk (scsi)
Disk /dev/sda: 10.7GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Number Start End Size Type File system Flags
1 1049kB 10.7GB 10.7GB primary ext4
Model: Google PersistentDisk (scsi)
Disk /dev/sdb: 10.7GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Number Start End Size Type File system Flags
1 1049kB 10.7GB 10.7GB primary ext4
gave me the information that its "ext4".
although when I issue the following command I still get an error:
root#debugger:~# mount -t ext4 /dev/disk/by-id/scsi-0Google_PersistentDisk_marty-wll-debugging /mnt/marty-wll
mount: wrong fs type, bad option, bad superblock on /dev/sdb,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
dmesg of syslog said :
[ 2452.205447] EXT4-fs (sdb): VFS: Can't find ext4 filesystem
any ideas?
Thanks for pointing this out, I will update the docs. Try adding -part1 to the end of your device name. This will mount the partition, instead of the disk. For your specific case:
mount /dev/disk/by-id/scsi-0Google_PersistentDisk_myinstance-debugging-part1 /mnt/myinstance
Also, there are cleaner aliases, so this should work as well:
mount /dev/disk/by-id/google-myinstance-debugging-part1 /mnt/myinstance