GVSU HPC Cluster uses Slurm to manage job scheduling and resource allocation. If your job isn't starting or seems delayed in starting compared to others, this guide might help you understand why, how Slurm prioritizes jobs and what you can do to troubleshoot your jobs.
Key Concepts in Slurm Scheduling
Priority
Priority is the core mechanism Slurm uses to decide the order in which all jobs are started. Every queued job gets a priority score, and the scheduler (Slurm) starts the highest priority jobs first when resource become available. Job priority is calculates using some of the following factors:
- Fairshare – Users who have consumed fewer resources recently receive a higher priority.
- Job Age – Jobs gain priority the longer they wait in the queue.
Note: These factors only apply to jobs waiting in the queue. Once a job is running, it continues until completion (or until canceled, fails or it hits its wall time).
Fairshare
Fairshare is a dynamic priority system for Slurm that helps ensure equitable access to cluster resources such as CPUs, memory, and GPUs. Instead of using a strict “first come, first served” model, Slurm continuously tracks each user’s recent resource usage and adjusts job priorities accordingly.
- Users (or accounts) that have consumed more than their fair share of resources will see their new jobs assigned a lower priority until others have had the opportunity to run.
- Users (or accounts) that have used less than their fair share will see their jobs receive a higher priority boost.
- The fairshare factor is time-decayed, meaning older usage gradually “fades” and stops impacting current scheduling.
Fairshare helps with the following:
- Prevents a small number of users from monopolizing cluster resources.
- Keep resource distribution aligned with the allocations/policies set by the cluster administrators.
- Ensure both heavy and light users have fair opportunities to run jobs over time.
Half Life Decay
Slurm uses something called Half Life Decay which means over time your priority score will start to come back. GVSU cluster is set to 1 day so after one day you will get half of your score back. On average this makes it so if you don’t submit a job for a whole week your fairshare score should go back up to 1. Your fairshare score is a number between 0 and 1, with 1 being the highest score.