The task I want to complete: I need to run a python package inside of a singularity-container that is asking to open at least some 9704 files. This is the first I have heard of it and searching around this has something to do with a system’s ulimit.
What I currently have is the following def file.
I am setting the * hard nofile flag and the * soft nofile flag to 15 thousand. The sed line does edit the conf file but within the singularity shell my ulimit is still the default 1024.
Bootstrap: docker
From: fedora
%post
dnf -y update
dnf -y install nano pip wget libXcomposite libXcursor libXi libXtst libXrandr alsa-lib mesa-libEGL libXdamage mesa-libGL libXScrnSaver
wget -c https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh
/bin/bash Anaconda3-2020.02-Linux-x86_64.sh -bfp /usr/local
conda config --file /.condarc --add channels defaults
conda config --file /.condarc --add channels conda-forge
conda update conda
sed -i '2s/#/\n* hard nofile 15000\n* soft nofile 15000\n\n#/g' /etc/security/limits.conf
bash
%runscript
python /Users/lamsal/count_of_monte_cristo/orthofinder_run/OrthoFinder_source/orthofinder.py -f /Users/lamsal/count_of_monte_cristo/orthofinder_run/concatanated_FAs/
I am following the “official” instuctions to change the ulimits for a RHEL based system from IBM’s webpage here: https://www.ibm.com/docs/en/rational-clearcase/9.0.2?topic=servers-increasing-number-file-handles-linux-workstations
Is the sed line not the right way to change ulimits for a singularity image?
Short answer:
Change the value on the host OS.
Long answer:
In this instance, running a singularity container is best thought of as any other binary you're executing in your host OS. It creates its own separate environment, but otherwise it follows the rules and restrictions of the user running it. Here, the ulimit is taken from the host kernel and completely ignores any configs that may exist in the container itself.
Compare the output from the following:
# check the ulimit on the host
ulimit -n
# check the ulimit in the singularity container
singularity exec -e image.sif ulimit -n
# docker only cares about container config settings
docker run --rm fedora:latest ulimit -n
# change your local ulimit
ulimit -n 4096
# verify it has changed
ulimit -n
# singularity has changed
singularity exec -e image.sif ulimit -n
# ... but docker hasn't
docker run --rm fedora:latest ulimit -n
To have a persistent fix, you'll need to modify the setting on your host OS. Assuming you're on MacOS this answer should take care of that.
If you don't have root privs or you're only using this intermittently you can run ulimit by before running singularity. Alternatively, you could use a wrapper script to run the image and set it in there.
Related
I have a binary installed on a Docker container that I have been trying to run via Singularity:
singularity run docker://repo/container_image ./repository/bin --flag
The problem is that with such command it sources my .bashrc, which is causing some problems with the binary.
So I tried running it with --no-home and flaged repositories to be mounted with -B:
singularity run --no-home -B /hostrepo01:/data,/hostrepo02:/results docker://repo/container_image ./repository/bin --flag
This still imports some paths form my host os, for instance if I open a Singularity shell with the options bellow and do a cd, the shell tries to access the adress I have for my home on the host OS.
singularity run --no-home -B /hostrepo01:/data,/hostrepo02:/results docker://repo/container_image
How can I execute a command inside a singularity container that do not interact/sources options from the host OS, other than what I specify with the -B flag?
You can use the --contain flag
-c, --contain use minimal /dev and empty other
directories (e.g. /tmp and $HOME) instead
of sharing filesystems from your host
singularity run --contain -B /hostrepo01:/data,/hostrepo02:/results docker://repo/container_image
I am trying to achieve something like
docker run -it <image_name> bash
I want to specify the image to run and do not care about anything else.
crictl requires config files for both a container and a pod for the run command, if I am not mistaken.
[hbaba#ip-XX-XX-XXX misc]$ sudo crictl -r /run/crio/crio.sock run -h
....
USAGE:
crictl run [command options] container-config.[json|yaml] pod-config.[json|yaml]
I am looking for the simplest way of starting a container, possibly with only a specified image.
I am running Docker on a Macintosh, and have installed the MariaDB image. I would like to access it from another machine on the LAN.
I understand that the solution is to enable bind-address=0.0.0.0 (or something similar) in /etc/mysql/my.cnf. I executed docker exec -it mariadb bash, installed Joe text editor (because I am much more familiar with it than Vi or Nano), and edited the file.
The problem is that when I restart the Docker image,it has forgotten all the changes, and it doesn’t work.
Am I missing a step, or is this not the way to go about it?
Containers are throw-away by design and, as you noticed, any modifications are lost when you run fresh one.
You have two options:
First one is described here: Docker: editing my.cnf in when building the image (just mount your custom config and be done).
Second option is to make your custom container image based on official image + your modification, something like this:
Dockerfile:
# Lets say mariadb v10.3.28... Change for what you want.
FROM mariadb:10.3.28
# there is already `#bind-address=0.0.0.0` in /etc/mysql/my.cnf
# we use sed and replace it with `bind-address=0.0.0.0`)
RUN sed -i "s/#bind-address=0.0.0.0/bind-address=0.0.0.0/g" /etc/mysql/my.cnf && \
# and, for example, lets change `max_allowed_packet` too.
sed -i "s/max_allowed_packet.*/max_allowed_packet=512M/g" /etc/mysql/my.cnf;
(rule of thumbs is "make as many steps in single RUN as possible" to save image layers)
then build it:
$ cd /where/my/dockerfile/is
$ docker build . -t mymysql
and run it:
# In newer mariadb it should be `-e MARIADB_ROOT_PASSWORD=`
# And maybe you should mount datadir somewhere `-v /my/own/datadir:/var/lib/mysql`
$ docker run -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=my-secret-pw mymysql
I was creating an own docker image based on the latest rakudo-star docker image. I wanted to use DBIish to connect to a mysql database. Unfortunately I am not able to get the DBDish::mysql to work.
I've installed default-libmysqlclient-dev as you can see in
# find / -name 'libmysqlclient*.so'
/usr/lib/x86_64-linux-gnu/libmysqlclient_r.so
/usr/lib/x86_64-linux-gnu/libmysqlclient.so
The error i am facing is:
# perl6 -Ilib -e 'use DBDish::mysql; DBDish::mysql.connect()'
Cannot locate native library 'mysqlclient': mysqlclient: cannot open shared object file: No such file or directory
in method setup at /usr/share/perl6/sources/24DD121B5B4774C04A7084827BFAD92199756E03 (NativeCall) line 289
in method CALL-ME at /usr/share/perl6/sources/24DD121B5B4774C04A7084827BFAD92199756E03 (NativeCall) line 539
in method connect at /root/DBIish/lib/DBDish/mysql.pm6 (DBDish::mysql) line 12
in block <unit> at -e line 1
Short answer: you need the package libmysqlclient20 (I added the documentation request to a similar DBIish issue). Debian 9 (stable at the moment) uses and older version than Ubuntu 18.04 (stable at the moment) and Debian Unstable. It also refers to mariadb instead of mysql. Pick libmariadbclient18 on images based on Debian Stable and create a link with the mysql name (see below).
On Debian Testing/Unstable and recent derivatives:
$ sudo apt-get install libmysqlclient20
$ dpkg -L libmysqlclient20
/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.9
/usr/share
/usr/share/doc
/usr/share/doc/libmysqlclient20
/usr/share/doc/libmysqlclient20/NEWS.Debian.gz
/usr/share/doc/libmysqlclient20/changelog.Debian.gz
/usr/share/doc/libmysqlclient20/copyright
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
On Debian 9 and derivatives:
$ dpkg -L libmariadbclient18
/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libmariadbclient.so.18.0.0
/usr/lib/x86_64-linux-gnu/mariadb18
/usr/lib/x86_64-linux-gnu/mariadb18/plugin
/usr/lib/x86_64-linux-gnu/mariadb18/plugin/client_ed25519.so
/usr/lib/x86_64-linux-gnu/mariadb18/plugin/dialog.so
/usr/lib/x86_64-linux-gnu/mariadb18/plugin/mysql_clear_password.so
/usr/share
/usr/share/doc
/usr/share/doc/libmariadbclient18
/usr/share/doc/libmariadbclient18/changelog.Debian.gz
/usr/share/doc/libmariadbclient18/copyright
/usr/lib/x86_64-linux-gnu/libmariadbclient.so.18
Create the link:
$ sudo ln -s /usr/lib/x86_64-linux-gnu/libmariadbclient.so.18 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18
In order to illustrate this, I created an Ubuntu 18.04 container for the occasion*:
docker run -ti --rm --entrypoint=bash rakudo/ubuntu-amd64-18.04
And the abbreviated commands and output:
# apt-get install -y libmysqlclient20 build-essential
# zef install DBIish
# perl6 -e 'use DBDish::mysql; DBDish::mysql.connect()'
Cannot look up attributes in a DBDish::mysql type object
[...]
The error is because I didn't pass the correct parameters for connect as I didn't have a db running. The important thing is that no .so file is missing.
*: I uploaded it to the Docker Hub, a normal run will put you right in the REPL:
$ docker run -ti --rm rakudo/ubuntu-amd64-18.04
To exit type 'exit' or '^D'
>
(I didn't use the Star image when debugging, but it does not matter because this is a more generic problem.)
My objective is to build a Docker image that includes MySQL prefilled with the tables and data produced by an Alembic migration. Unfortunately, Alembic can't produce the necessary data without an active MySQL instance, nor can it independently create a SQL dump to be loaded by MySQL on first run.
I've attempted to use multi-stage builds to use both the mysql and python containers for this, but the MySQL daemon is brought down again as soon as the Python stage begins.
# start MySQL daemon
FROM mysql:5.6
RUN docker-entrypoint.sh
# install and run Alembic
FROM python:2.7-alpine
# [install Alembic]
COPY ./alembic-migrations /alembic-migrations
# [run migrations]
I'm not hung up on this particular solution, but it seemed like the simplest option. Is there a way to do what I'm attempting? Should I resign myself to installing Python and Alembic in the MySQL container?
It'll probably make some Docker evangelist's eyes bleed, but this is how I was able to accomplish the behaviour I was looking for. It was actually simpler and runs faster than I'd expected.
FROM python:2.7-alpine as python
FROM mysql:5.6
# set up a functional chroot of the Python image at /python
COPY --from=python / /python
RUN set -ex; \
cp /etc/resolv.conf /python/etc/resolv.conf; \
mknod -m 0644 /python/dev/random c 1 8; \
mknod -m 0644 /python/dev/urandom c 1 9;
# install software depedencies in chroot jail
RUN set -ex; \
chroot /python apk --no-cache --virtual add [some dependencies]
# install Python libraries
COPY ./requirements.txt /python/tmp/requirements.txt
RUN chroot /python pip install -r /tmp/requirements.txt;
# apply Alembic migrations and remove the Python chroot jail
COPY ./usr/local/bin/build.sh /usr/local/bin/
COPY ./alembic /python/alembic
RUN build.sh && rm -rf /python;
ENTRYPOINT ["docker-entrypoint.sh", "--datadir=/var/lib/mysql-persist"]
EXPOSE 3306
CMD ["mysqld"]
The build.sh script simply forks the docker-entrypoint.sh script from the MySQL container, then invokes the Alembic-specific code within the Python chroot.
#!/bin/sh
docker-entrypoint.sh --datadir=/var/lib/mysql-persist 2>/dev/null &
chroot /python build.sh
Note that I'm setting a custom data directory (/var/lib/mysql-persist) because the upstream mysql container defines VOLUME /var/lib/mysql, which I can't override.
The result is a built image that contains MySQL, complete with database, but does not contain any traces of the Python container or Alembic scripts. It can now be distributed via a registry and fetched by docker-compose, avoiding the need for all users to execute the Alembic migrations independently.