Using GPUs on Clipper

Clipper is equipped with 22 NVIDIA GPUs for research use. This article details how to request GPUs in Slurm jobs.

Open OnDemand includes a Short GPU Instance Specification for interactive GPU workloads. Short GPU sessions are limited to 4 hours and are intended for development, testing, visualization, and exploratory analysis.

Available GPUs

* NVIDIA’s CUDA compute capability indicates the features and generation of a specific GPU. See: https://developer.nvidia.com/cuda-gpus

What option is right for my research?

The NVIDIA H100 NVL is one of the latest generations of NVIDIA cards and excels at all workloads. It will be the fastest of all options available in Clipper, regardless of task. It is rated to be about 3.5 times faster than the Tesla V100s in double-precision calculations. The H100 NVL has 94 GB of high-speed HBM3 memory versus the V100s' 32 GB of HBM2 memory.

The Quadro cards lack double-precision performance, maxing out at around 500 GFLOPS (billion FLOPS) for double-precision calculations. That’s about 16 times slower than the Tesla V100s.

Single-precision performance between the v100s and RTX 8000 is similar.

CUDA 12/CUDA 13

CUDA 13 has dropped support for the Tesla V100s GPUs. The V100s nodes use an older NVIDIA driver and can only run code compiled on CUDA 12 or earlier. Both CUDA 12 and CUDA 13 are installed on the cluster. If you wish to use the Tesla V100s and need to compile code, it is recommended to do so in an interactive session on one of the V100s nodes. 

Third-party packages such as PyTorch may eventually drop support for CUDA 12 entirely.

Single vs. Double Precision

Single-precision calculations use 32 bits to represent a number. Double-precision calculations use 64 bits to represent a number.

Single-precision offers acceptable accuracy for tasks like graphics or machine learning, where some error tolerance is possible. Double-precision calculations provide a wider range of numbers and higher precision, crucial for scientific simulations where tiny differences matter.

Requesting GPUs in a Slurm Job

GPUs are not available in a Slurm job without explicitly requesting them.

Requesting Whole GPUs

To request one or more GPUs per node in a Slurm job, use the following in your submission:

--gpus-per-node=[type]:number

For example, to request a single GPU:

--gpus-per-node=tesla_v100s:1
--gpus-per-node=quadro_rtx_8000:1
--gpus-per-node=nvidia_h100_nvl:1

More Information

• Generic Resource (GRES) Scheduling - Slurm Offical Documentation
• NVIDIA GPUs - Compute Compatibility