Containers on HPC Clusters

Container technologies provide a way to package software and its dependencies in a portable and reproducible manner. They offer an effective solution for deploying complex applications with specific software requirements across diverse computing environments.

Containers on our HPC clusters allow you to:

• Run applications with complex dependencies
• Ensure reproducibility of your scientific workflows
• Import workflows developed elsewhere into our HPC environment
• Easily share your software environment with collaborators
• Use pre-built images from container registries

Singularity/Apptainer

Singularity (now also known as Apptainer) is the container technology available on our HPC clusters. It is specifically designed for scientific computing in multi-user HPC environments, providing the benefits of containerization while addressing the security concerns associated with traditional container engines like Docker.

Key Features

• User space execution - Singularity runs containers without requiring root privileges
• Integration with host resources - Seamlessly access host file systems and GPU resources
• MPI support - Run MPI-based applications using the high-performance host interconnect
• Docker compatibility - Easily convert and use Docker containers

Getting Started

Singularity is available by default on all our HPC clusters. You can verify the installation by running:

singularity --version

or

apptainer --version

Building Containers

While Singularity containers can be run on our HPC clusters, building containers directly on the login nodes is generally not recommended due to resource constraints and potential impact on other users. There are several options for building Singularity/Apptainer containers:

Option 1: Pull from a Container Registry

The easiest approach is to pull pre-built containers from public registries:

singularity pull docker://tensorflow/tensorflow:latest-gpu

Option 2: Build Locally and Transfer

Build your container on your local machine and transfer it to the HPC cluster:

singularity build mycontainer.sif mydef.def
scp mycontainer.sif [email protected]:~/containers/

Option 3: Convert Docker Images

Convert Docker images to Singularity format:

singularity pull docker://ubuntu:20.04

Singularity Definition Files

You can create custom containers using Singularity definition files. Here's a simple example:

Bootstrap: docker
From: ubuntu:20.04
%post
    apt-get update
    apt-get install -y python3 python3-pip
    pip3 install numpy scipy pandas
%runscript
    python3 "$@"
%labels
    Author Your Name
    Version v0.1

Running Containers

There are several ways to run Singularity containers on our HPC clusters:

Basic Execution

singularity run mycontainer.sif

Shell into Container

singularity shell mycontainer.sif

Execute Commands

singularity exec mycontainer.sif python3 myscript.py

Using in Job Scripts

To run containerized applications in batch jobs, include the singularity command in your job script:

#!/bin/bash
#SBATCH --job-name=container_job
#SBATCH --partition=standard
#SBATCH --time=01:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=4
module load singularity
singularity exec mycontainer.sif ./my_application

Binding Host Directories

To access files from the host system, bind directories into your container:

singularity exec --bind /scratch/myproject:/data mycontainer.sif python3 /data/analysis.py

MPI Containers

Running MPI applications within Singularity containers requires a hybrid approach where the container uses the host's MPI implementation:

MPI Binding

Singularity automatically binds the host MPI libraries into the container. To run an MPI application:

mpirun -n 4 singularity exec mpi_container.sif /path/to/mpi/executable

Ensure that the MPI version in the container is compatible with the host MPI implementation.

Example Slurm MPI Job

#!/bin/bash
#SBATCH --job-name=mpi_container
#SBATCH --partition=standard
#SBATCH --time=01:00:00
#SBATCH --nodes=2
#SBATCH --ntasks=16
module load openmpi
module load singularity
srun -n $SLURM_NTASKS singularity exec mpi_container.sif /app/mpi_program

GPU Containers

Singularity provides native support for NVIDIA GPUs through the --nv flag:

singularity exec --nv gpu_container.sif python3 /app/train_model.py

This flag enables the container to access the host's NVIDIA GPU drivers and hardware.

GPU Job Example

#!/bin/bash
#SBATCH --job-name=gpu_container
#SBATCH --partition=gpu
#SBATCH --time=02:00:00
#SBATCH --nodes=1
#SBATCH --gpus=2
module load singularity
singularity exec --nv tensorflow.sif python3 /path/to/tensorflow_script.py

Best Practices

• Keep containers lightweight - Include only the software and dependencies you need
• Use read-only containers - Use the SIF (Singularity Image Format) whenever possible
• Organize data properly - Use host bind mounts for data instead of including large datasets in your containers
• Document your containers - Include metadata and documentation about the container's contents and usage
• Test before scaling - Test your containerized workflow with small datasets before running large-scale computations
• Version control your definition files - Store Singularity definition files in version control systems for reproducibility

Limitations

While containers provide many benefits, there are some limitations to be aware of:

• No Docker or Podman - These container engines are not available on our HPC clusters due to security considerations
• Performance overhead - Containers may introduce a small performance overhead compared to bare-metal execution
• Specialized hardware access - Some specialized hardware may require additional configuration to work within containers

Frequently Asked Questions

Why can't I use Docker on the HPC clusters?

Docker requires root privileges to run, which presents security concerns in multi-user environments. Singularity provides similar functionality without requiring privileged access.

Can I convert Docker images to use with Singularity?

Yes, Singularity can directly pull and convert Docker images:

singularity pull docker://ubuntu:20.04

How do I share files between the host and the container?

Use the --bind option to mount host directories into the container:

singularity exec --bind /path/on/host:/path/in/container mycontainer.sif command

Where should I store my container images?

For small containers, your home directory is appropriate. For larger containers, consider storing them in your project directory or scratch space.

Can I install software inside a running container?

Singularity SIF files are immutable. To make changes, you need to rebuild the container or use a sandbox mode for development purposes.