Updated Container Info on README

README.md

@@ -79,7 +79,149 @@ ipython kernel install --user --name tfbenchmark
 
 ## Singularity (i.e., a Docker-style workflow that operates on shared cluster systems)
 
-Singularity is another way to build your environment, allowing even greater portability across systems at different universities. A full tutorial on using Singularity can be found [here](https://docs.s3it.uzh.ch/cluster/singularity_tutorial/).
+### First, some background:
+
+#### What is a container? 
+
+#### How does a container differ from a virtual machine?
+
+### or from a virtual environment (such as conda or mamba)?
+
+Singularity is another way to build your environment, allowing even greater portability across systems at different universities or computing centers.
+
+###### [A short introduction to container concepts from Pawsey Supercomputing Centre](https://pawseysc.github.io/singularity-containers/11-containers-intro/index.html)  
+
+### When should you use a container? Some possibilities:
+When you are using ScienceCluster, but you have an application to build/install with many or complicated system dependencies (ubuntu packages).<br>
+When you have an application that you wish to build and run on multiple systems.<br>
+If someone has distributed an application as a docker or singularity container image.<br>
+
+A full tutorial on using Singularity on ScienceCluster can be found [**here**](https://docs.s3it.uzh.ch/cluster/singularity_tutorial/).
+
+Guide to the singularity definition file, for building your own custom singularity container:
+[definition file](https://docs.sylabs.io/guides/3.5/user-guide/definition_files.html)
+
+### Bonus example using OpenMPI parallelism
+
+Containers using (MPI)[https://docs.sylabs.io/guides/3.5/user-guide/mpi.html] parallelism (Message Passing Interface) can be more complicated, and there are multiple ways to configure singularity plus MPI.
+
+Here is an example using the **bind** approach.  A definition file could look like below.  It assumes that you have already compiled and built the executable mpi_hello_world from mpi_hello_world.c, borrowed from (here)[https://github.com/mpitutorial/mpitutorial/tree/gh-pages/tutorials/mpi-hello-world]:
+```
+Bootstrap: docker
+From: ubuntu:20.04
+
+%files
+    ./mpi_hello_world /opt/bin/mpi_hello_world
+
+
+%environment
+    export MPI_DIR="/apps/opt/spack/linux-ubuntu20.04-x86_64/gcc-9.3.0/openmpi-4.1.3-hxw6m2yyxelnba6ruvojic4oohxsbdl7"
+       ## ScienceCluster openmpi bind paths
+    export PATH="$MPI_DIR/bin:$PATH"
+    export LD_LIBRARY_PATH="$MPI_DIR/lib:$LD_LIBRARY_PATH"
+    export LIBRARY_PATH="$MPI_DIR/lib:$LIBRARY_PATH"
+
+%post
+    echo "Installing base packages..."
+    apt-get update -y  && . /etc/environment
+    apt-get install hwloc -y
+    apt-get install libnl-3-200 -y
+    mkdir -p /opt/bin
+```
+
+In this mode, the application in the container calls the MPI library on the host (ScienceCluster) for communication between tasks. A slurm script to run this container using 4 MPI tasks on a single node could look like this:
+```
+#!/bin/bash
+#SBATCH --ntasks=4  ## number of MPI tasks
+#SBATCH --cpus-per-task=1  ## number of cores per task
+#SBATCH --time=00:01:00
+#SBATCH --mem 3000M 
+#SBATCH --nodes=1
+#SBTACH --ntasks-per-node=4
+
+
+module load openmpi
+module load singularityce
+
+### run the container in bind mode
+
+echo 'start'
+
+srun --exclusive --ntasks=4 --cpus-per-task=1 --nodes=1  --mem 3000M singularity exec --bind "$MPI_DIR" --bind /apps ./mpi_hello_world /opt/bin/mpi_hello_world
+
+echo 'finished'
+```
+Here is the same example using the **hybrid** approach and the definition file, hellompi_hybrid.def:
+```
+Bootstrap: docker
+From: ubuntu:20.04
+
+%files
+    mpi_hello_world.c /opt
+    makefile /opt
+
+%environment
+    export OMPI_DIR=/opt/ompi
+    export SINGULARITY_OMPI_DIR=$OMPI_DIR
+    export SINGULARITYENV_APPEND_PATH=$OMPI_DIR/bin
+    export SINGULAIRTYENV_APPEND_LD_LIBRARY_PATH=$OMPI_DIR/lib
+
+%post
+    echo "Installing base packages..."
+    apt-get update -y  && . /etc/environment
+    apt-get install wget gcc bash g++ make libsysfs2 libsysfs-dev libevent-dev libpmix-dev -y
+
+    echo "Installing Open MPI"
+    export OMPI_DIR=/opt/ompi
+    export OMPI_VERSION=4.1.3
+    export OMPI_URL="https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-$OMPI_VERSION.tar.bz2"
+    mkdir -p /tmp/ompi
+    mkdir -p /opt
+    ##  Download
+    cd /tmp/ompi && wget -O openmpi-$OMPI_VERSION.tar.bz2 $OMPI_URL && tar -xjf openmpi-$OMPI_VERSION.tar.bz2
+    # Compile and install
+    cd /tmp/ompi/openmpi-$OMPI_VERSION && ./configure --enable-orterun-prefix-by-default --with-pmix=/usr/lib/x86_64-linux-gnu/pmix/ --prefix=$OMPI_DIR && make install
+    # Set env variables so we can compile our application
+    export PATH=$OMPI_DIR/bin:$PATH
+    export LD_LIBRARY_PATH=$OMPI_DIR/lib:$LD_LIBRARY_PATH
+    export MANPATH=$OMPI_DIR/share/man:$MANPATH
+
+    echo "Compiling the MPI application..."
+    cd /opt
+    make
+```
+Notice that Open MPI is installed directly into the container. The files 'mpi_hello_world.c', 'makefile', and hellompi_hybrid.def would be in the current working directory.  Because of the sudo requirement, you would need to create the container on a ScienceCloud virtual machine (we have images with singularity pre-installed) or on your own machine. Then the command to build the container image is:
+```
+sudo singularity build hellompi_hybrid.sif hellompi_hybrid.def
+```
+You would then transfer the .sif image file to ScienceCluster.  Then, after expanding the image with:
+```
+singularity build --sandbox ./hellompi_hybrid ./hellompi_hybrid.sif
+```
+The executable within the image can be run with the following slurm script:
+```
+#!/bin/bash
+#SBATCH --ntasks=4  ## number of MPI tasks
+#SBATCH --cpus-per-task=1  ## number of cores per task
+#SBATCH --time=00:01:00
+#SBATCH --mem 3000M 
+#SBATCH --nodes=1
+#SBTACH --ntasks-per-node=4
+
+
+module load openmpi/4.1.3
+module load singularityce
+
+
+echo 'start'
+
+srun singularity exec ./hellompi_hybrid  /opt/mpi_hello_world
+
+echo 'finished'
+```
+
+
+## Prepare the tensorflow benchmark using singularity
 
 To use Singularity, create the Singularity Image file from the definition file using:
 ```