Powered By GitBook
README
Learn how to use Cloud Build to build Singularity containers for HPC workloads.
This tutorial shows you how to use Cloud Build to build Singularity containers. In constrast to Docker, the Singularity container binary is designed specifically for high performance computing (HPC) workloads.
This diagram illustrates the process the tutorial takes you through:
diagram
​​
​
​
You may also view this tutorial on the Google Cloud Platform Community site.

Before you begin

Install the Cloud SDK​

The Cloud SDK installs commands and libraries for interacting with Google Cloud Platform (GCP) from the command line. This tutorial makes extensive use of the gcloud and gsutil commands to create and manage GCP resources.
Even if you alread have the Cloud SDK installed, it is a good idea to update to the lastest version using this command:
1
gcloud components update
Copied!

Set the required environment variables

Edit the file env.sh and replace the placeholders with the following:
    [YOUR_ORG]: the name of the organization that will own your project
    [YOUR_BILLING_ACCOUNT_NAME]: the name of the account responsible for any costs associated with your project
    [NAME_FOR_THE_PROJECT_YOU_WILL_CREATE]: the name of your project
    [COMPUTE_ZONE_YOU_WANT_TO_USE]: the name of the GCP compute zone that will contain your project
The organization and billing account should already exist. If you need to create an organization, see Creating and managing organizations.
Use these gcloud commands to retrieve the values required by env.sh:
1
gcloud organizations list
Copied!
1
gcloud alpha billing accounts list
Copied!
1
gcloud projects list
Copied!
1
gcloud compute zones list
Copied!
If you are creating a new project, use the name you intend to give it.

Source env.sh to define the environment variables in your current shell session

1
source ./env.sh
Copied!

(Optional) Create a new project

To use a new project rather than an existing one, create the new project with the following commands:
1
gcloud projects create $PROJECT --organization=$ORG
Copied!
1
gcloud beta billing projects link $PROJECT --billing-account=$(gcloud beta billing accounts list | grep $BILLING_ACCOUNT | awk '{print $1}')
Copied!
1
gcloud config configurations create -- activate $PROJECT
Copied!

Use the desired project and zone

1
gcloud config set core/project $PROJECT
Copied!
1
gcloud config set compute/zone $ZONE
Copied!

Enable the necessary GCP APIs

The following commands enable Compute Engine, Cloud Build, and Container Registry:
1
gcloud services enable compute.googleapis.com
Copied!
1
gcloud services enable cloudbuild.googleapis.com
Copied!
1
gcloud services enable containerregistry.googleapis.com
Copied!

Build a Singularity container

Cloud Build supports the definition and use of custom build steps to extend the range of tasks it can handle. In the first step of this section you create a Singularity custom build step. In the third step you use the Singularity custom build step to build a Singularity container and save it in Cloud Storage. Once the Singuarlity custom build step has been created in your project you can use it to create as many Singularity containers as you like.

Create a Singularity builder

The singularity-buildstep directory contains the files you use to create the Singularity custom build step.
    builder.yaml is the recipe to create the Singularity builder. Specifically, it's a configuration file you will submit to build the custom build step.
    Dockerfile contains the steps required to download and build Singularity in a go container.
Move to that directory:
1
cd singularity-buildstep
Copied!
Then use this command to build the custom build step:
1
gcloud builds submit --config=builder.yaml --substitutions=_SINGULARITY_VERSION="3.0.2" .
Copied!
At the time of this writing, the latest stable version of Singularity is 3.0.2. To use a different (possibly newer) version, modify the value of the _SINGULARITY_VERSION substitution accordingly; it should match a release tag on the sylabs/singularity repository.

Create a bucket

You need a place to store the containers the Singularity build step creates. In this tutorial you will use a Cloud Storage bucket to hold your Singularity containers. You can use an existing bucket you have access to or use this command to create a new one:
1
gsutil mb gs://${PROJECT}-singularity
Copied!

Build a Singularity container

Singularity uses a Singularity definition file as a blueprint for building a container. The definition file contains a number of sections which control how the singularity build command constructs a container.
Go back up one directory to look at julia.def:
1
cd ..
Copied!
1
cat julia.def
Copied!
1
Bootstrap: library
2
From: centos:7
3
​
4
%runscript
5
# run your script here.
6
​
7
# check if there any arguments,
8
if [ -z "[email protected]" ]; then
9
# if there are none, test julia:
10
echo 'println("hello world from julia container!!!")' | julia
11
else
12
# if there is an argument, then run it! and hope its a julia script :)
14
fi
15
​
16
​
17
%environment
18
PATH=/julia-1.0.3/bin:$PATH
19
LD_LIBRARY_PATH=/julia-1.0.3/lib:/julia-1.0.3/lib/julia:$LD_LIBRARY_PATH
20
LC_ALL=C
21
export PATH LD_LIBRARY_PATH LC_ALL
22
​
23
%post
24
yum -y update
25
​
26
# install some basic tools
27
yum -y install curl tar gzip
28
​
29
# now, download and install julia
30
curl -sSL "https://julialang-s3.julialang.org/bin/linux/x64/1.0/julia-1.0.3-linux-x86_64.tar.gz" > julia.tgz
31
tar -C / -zxf julia.tgz
32
rm -f julia.tgz
Copied!
The julia.def file is a simple example of a Singularity definition file. It creates a CentOS 7 container with the Julia programming language installed. When you run the container, the entry point is the Julia executable. This means that if the user provides a script, Julia will execute it. If not, Julia will print a "hello world" message. The file has three sections:
    %runscript: The contents of this section are written to a file within the container and executed when the container
    is run.
    %environment: This section defines environment variables that will be set at runtime (shell, run, exec).
    %post: These commands are executed after the base operating system has been installed at build time.
To create a Singularity container from the julia.def definition file, use the custom Singularity build step created above. The cloudbuild.yaml file specifies the steps required to build the container and store the result in a Cloud Storage bucket.
1
cat cloudbuild.yaml
Copied!
1
# In this directory, run the following command to build this builder.
2
# $ gcloud builds submit . --config=cloudbuild.yaml --substitutions=_SINGULARITY_VERSION=3.0.0."
3
​
4
steps:
5
- name: gcr.io/$PROJECT_ID/singularity-${_SINGULARITY_VERSION}
6
args: ['build', 'julia-centos.sif', 'julia.def']
7
artifacts:
8
objects:
9
location: 'gs://${PROJECT_ID}-singularity'
10
paths: ['julia-centos.sif']
Copied!
The first step (the only step in this example) runs the singularity command to build the julia-centos.sif container from the julia.def definition file. The artifacts section directs Cloud Build to store the resulting container in the Cloud Storage bucket you created above.
Use this command to execute the build:
1
gcloud builds submit --config=cloudbuild.yaml --substitutions=_SINGULARITY_VERSION="3.0.2" .
Copied!
After the build completes, verify that the container was created using the list (ls) command.
1
gsutil ls gs://${PROJECT}-singularity/julia-centos.sif
Copied!
If the build was successful, you should see a response similar to the following:
1
gs://my-project-singularity/julia-centos.sif
Copied!

Test the container

Verify that your container is working properly by downloading and executing it. You will need a compute instance with singularity installed. This means that you have two options. If you are familiar with Singularity and have it installed locally, you can download the container and execute it. If you don't have Singularity and want to bring up a cloud instance with Singularity ready to go, you can launch a Google Cloud Compute Instance.

Run the container locally

You can do the equivalent of a singularity pull by using gsutil to copy the container from storage back down to your local machine:
1
gsutil cp gs://${PROJECT}-singularity/julia-centos.sif .
Copied!
Now run the container:
1
singularity run julia-centos.sif
Copied!
The response should be this:
1
hello world from julia container!!!
Copied!

Run the container on a Compute Engine instance

This command will create the required Compute Engine instance.
1
gcloud compute instances create singularity-test \
2
--machine-type=n1-standard-1 \
3
--scopes=https://www.googleapis.com/auth/cloud-platform \
4
--metadata-from-file startup-script=startup.sh
Copied!
When the new instance is running, download the julia-centos.sif container using the command:
1
gcloud compute ssh singularity-test --command "gsutil cp gs://${PROJECT}-singularity/julia-centos.sif ."
Copied!
The startup.sh startup script downloads and builds the Singularity binary from scratch. This can take several minutes. Use the following command to determine if the build is complete:
1
gcloud compute ssh singularity-test --command "which singularity"
Copied!
When the response is /usr/local/bin/singularity, you are ready to proceed. Now you can run the container with this command:
1
gcloud compute ssh singularity-test --command "singularity run julia-centos.sif"
Copied!
The response should be this:
1
hello world from julia container!!!
Copied!
If you prefer, you can also log into the Compute Engine instance and verify the container you built from the command line there:
1
gcloud compute ssh singularity-test
Copied!
1
singularity run julia-centos.sif
Copied!
1
exit
Copied!

Clean up

Delete the test Compute Engine instance:
1
gcloud compute instances delete singularity-test
Copied!
Delete the container bucket:
1
gsutil rm -r gs://${PROJECT}-singularity
Copied!
Last modified 6mo ago