Singularity

What is Singularity?

Singularity is a Linux containerisation tool suitable for HPC environments. It uses its own container format and also has features to import Docker containers.

What is Docker?

Docker is a platform that employs features of the Linux kernel to run software in a container. The software housed in a docker container is not just a single program but an entire OS distribution, or at least enough of the OS to enable the program to work.

Docker can be thought of as somewhat like a software distribution mechanism like yum or apt. It also can be thought of as an expanded version of a chroot jail, or a reduced version of a virtual machine.

Read more about Docker on the official web site.

Docker vs Singularity

Here are some important differences between Docker and Singularity.

  • Docker and Singularity have their own container formats
  • Docker containers may be imported to run via Singularity. A full example is below.
  • Docker containers usually run as root. However as a Melbourne Bioinformatics (formerly VLSCI) user you have a regular user account which does not have root privileges.
  • Singularity allows for containers that run as a regular user. When importing a Docker container, Singularity removes any elements which can only run as root. The resulting containers are therefore able to run as a regular user.

Importing a Docker image

If you are working on a Docker container, you may use Singularity to convert this container to the Singularity format. Once done you may upload your Singularity container to a Melbourne Bioinformatics cluster such as snowy or barcoo, and run jobs using your container. The container is kept in your project's own storage area. This is different from Shifter which imports containers via Docker hub.

Here is an example. Note that most of these steps are performed on your local system, not a Melbourne Bioinformatics cluster!

Melbourne Bioinformatics filesystems

If you need to use the Melbourne Bioinformatics filesystems, hsm, scratch and vlsci inside your container, you will need to make sure folders exist for those filesystems to be mounted on.

Start your container (in this example we will use ubuntu:latest) and create those three directories.

$ sudo docker run -it ubuntu:latest bash
root@119565e80448:/# cd /
root@119565e80448:/# mkdir hsm scratch vlsci

Exporting your Docker image

Find the name of your Docker image using docker ps

$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              NAMES
119565e80448        ubuntu:latest       "bash"              2 minutes ago       Up 2 minutes        boring_nobel

In this example the name is boring_nobel. Export this image to a tarball

$ sudo docker export boring_nobel > ubuntu.tar

Preparing a Singularity image

You will need to have Singularity installed on your local computer in order to prepare your image. This is because the 'create' and 'import' operations of Singularity require root, which you do not have on Melbourne Bioinformatics systems.

Create an empty singularity image, and then import the exported docker image into it:

$ sudo singularity create ubuntu.img
Creating a sparse image with a maximum size of 1024MiB...
Using given image size of 1024
Formatting image (/sbin/mkfs.ext3)
Done. Image can be found at: ubuntu.img
$ sudo singularity import ubuntu.img ubuntu.tar

Uploading your Singularity image

You should upload your new singularity image to a Melbourne Bioinformatics cluster using your usual method. For example, to use scp to upload to snowy you might do:

$ scp ubuntu.img yourusername@snowy.melbournebioinformatics.org.au:

Using your Singularity image

From here onwards, the steps are performed on a Melbourne Bioinformatics cluster. We previously transferred ubuntu.img to snowy.

If you ssh into snowy and load the singularity module, you can then run commands inside your container.

$ ssh yourusername@snowy.melbournebioinformatics.org.au
$ module load singularity
$ singularity exec ubuntu.img /bin/bash
$ cat /etc/issue
Ubuntu 16.04.1 LTS \n \l

If you created the three directories for Melbourne Bioinformatics filesystems you should find they are mounted inside your container.

$ mount | grep gpfs
/dev/projects on /vlsci type gpfs (rw,relatime)
/dev/hsm on /hsm type gpfs (rw,relatime)
/dev/scratch on /scratch type gpfs (rw,relatime)

WARNING If your container mounts the /vlsci, /hsm or /scratch directories, software inside the container may be able to destroy any data on these filesystems for which you have write permissions. You should proceed with caution.

Using Singularity containers inside a SLURM job

It's straightforward to run your jobs inside a Singularity container.

In your sbatch script, ensure you load the singularity module. You can run any job inside the container by prefacing it with singularity exec <your image name>.

Here is a trivial example which we will save as singularity_test.slurm.

#!/bin/bash
#SBATCH --job-name=singularity_test
#SBATCH -p main
#SBATCH --ntasks=1

module load singularity

# where are we running?
hostname
singularity exec ubuntu.img hostname

# which OS is this?
cat /etc/issue
singularity exec ubuntu.img cat /etc/issue

If you run this with sbatch:

$ sbatch singularity_test.slurm
Submitted batch job 915487

Once the job completes, let's have a look at the result.

$ cat slurm-915487.out 
snowy006
snowy006
Red Hat Enterprise Linux Server release 6.8 (Santiago)
Kernel \r on an \m

Ubuntu 16.04.1 LTS \n \l

Lines 1 and 3 of the output were run directly on the OS of the compute node which handled your job. Lines 2 and 4 were run on the same compute node but inside the Singularity container.

Questions

If you have any questions about Singularity, the Melbourne Bioinformatics team is always able to help.