When should I use Clipper?
The High Performance Computing (HPC) cluster is used to solve advanced computational problems that require significantly more resources not found in the average desktop or lab computing environment. If you would like access to the HPC cluster please fill out a ticket by using this service request.
Who can use Clipper?
Any GVSU researcher can submit jobs to the cluster. Jobs are queued up in the order they are submitted.
Procedure
Globus
Logging In
Basic Job Submission
Your Job Script
Choosing a Queue/Partition
Partition Details
Specifying the Amount of Time Your Job Will Run
Specifying Node and Core Requirements
Specifying Memory Requirements
Storage
WinSCP
Globus is our file transfer node used to send large amounts of data to and from the cluster. Globus can help with moving large amounts of data for research that is 100's of GB if not more. To get access to Globus please reach out to us by sending us an email to arcit@gvsu.edu.
(Note: You have to be on campus or using the VPN in order to log into the Cluster)
(Note: STUDENTS you need to either be on VPN or hard wired in when on campus in order to sign in, you cannot use WIFI)
Using Command Line
- Open up Command Prompt
- Enter the following “ssh username@clipper.gvsu.edu”
- You will then be prompted with “Are you sure you want to continue connecting”, verify it is clipper.gvsu.edu you are trying to connect to and type “yes” and hit enter
- You will then be prompted for your password, enter your password.
- Then select the option you want to use to verify it is you using DUO.
- After DUO is successful, you should now be connected to the cluster.
Using Putty
To login into the cluster you can use Putty. The steps provided below will show you how to login after you have installed Putty.
- Open up Putty
- Make sure you are on the Session tab on the left and then enter “clipper.gvsu.edu” into the “Host Name (or IP address)” section. Make sure “Connection type:” is set to “SSH”
- Leave everything else as default and then click “Open”.
- You will now get a command line screen to show up. From here type in your gvsu id name (example: mccrakei).
- Then type in your password
- You will then be prompted to use DUO. Choose the option you want and then you should be logged in.
Users generally submit jobs by writing a job script file and submitting the job to Slurm with the sbatch command. The sbatch command takes a number of options (some of which can be omitted or defaulted). These options define various requirements of the job, which are used by the scheduler to figure out what is needed to run your job, and to schedule it to run as soon as possible, subject to the constraints on the system, usage policies, and considering the other users of the cluster.
The options to sbatch can be given on the command line, or in most cases inside the job script. When given inside the job script, the option is placed alone on a line starting with #SBATCH (you must include a space after the SBATCH). These #SBATCH lines SHOULD come before any non-comment/non-blank line in the script --- any #SBATCH lines AFTER a non-comment/non-blank line in the script might get ignored by the scheduler. The # at the start of these lines means they will be ignored by the shell; i.e. only the sbatch command will read them.
This section will show you how to submit a job using the cluster and all of the parameters you need to set in your file that you are submitting. Default values are shown below if you do not set the parameters.
Submitting a job:
- Log into the cluster
- Create a new file called test-script by using the command below.
vi test-script.sh
Copy and paste in the lines below into the file.
#!/bin/bash
#SBATCH --nodes=1 ##Number of nodes I want to use
#SBATCH --mem=1024 ##Memory I want to use in MB
#SBATCH --time=00:05:00 ## time it will take to complete job
#SBATCH --partition=all ##Partition I want to use
#SBATCH --ntasks=1 ##Number of task
#SBATCH --job-name=test-jo ## Name of job
#SBATCH --output=test-job.%j.out ##Name of output file
- Type “:wq” (this will save and exit the file).
- Now type ls in the command line you should now see your test-script.sh file.
- Now type “sbatch test-script.sh” and then your job will be submitted.
You can use the command squeue -u (your username) to see your submitted job in the queue.
If you do not specify what parameters you need then what is listed below will be set as a default when submitting the job.
When choosing a Queue/Partitions there are few things you will want to keep in mind. Grand Valley currently has 7 queues to submit your jobs to and those 7 queues use the resources from 3 partitions.
|
Priority (1-5, 1 being highest priority)
|
Queue Name
|
Max Run Time (hrs)
|
Max Jobs
|
Max Nodes
|
Description
|
|
1
|
debug
|
2
|
|
2
|
This queue provides priority access to shared nodes for testing of jobs with short wall time and limited resources.
|
|
2
|
short
|
24
|
|
3
|
This queue is meant for short jobs with low resources.
|
|
3
|
bigmem
|
120
|
|
4
|
Use only for jobs with use of a lot of memory (only uses the bigmem partition nodes).
|
|
3
|
cpu
|
120
|
|
7
|
This queue provides priority for CPU usage.
|
|
3
|
gpu
|
120
|
|
4
|
Use only for jobs with use of GPU’s (only uses the gpu partition nodes).
|
|
4
|
all
|
120
|
|
15
|
This is the default queue that has access to all the partitions.
|
|
5
|
class
|
24
|
|
4
|
For students to use for classroom jobs.
|
|
Name of Partition
|
Processor Model Per Node
|
Number of Processors Per Node
|
Number of Cores Per Processor
|
Clock Speed
|
GPU
Card
|
Memory Per Node
|
|
Bigmem
|
Intel(R) Xeon(R) Gold 6226 CPU
|
4
|
12
|
2.70GHz
|
N/A
|
1.4 TB
|
|
CPU
|
Intel(R) Xeon(R) Gold 6248 CPU
|
2
|
20
|
2.50GHz
|
N/A
|
180GB
|
|
GPU
|
Intel(R) Xeon(R) Gold 6226 CPU
|
2
|
12
|
2.70GHz
|
Tesla V100S-PCIE-32GB (2 per node)
|
180GB
|
When submitting a job, make sure that you specify the amount of time it will take to complete the job. Give yourself a buffer as well. If you think a job will take an hour to complete you may want to put 2 hours to allow for any unexpected issues and prevent termination due to a time restraint. You can use the following parameter --time=TIME or -t TIME in any of the following formats:
M (M minutes)
M:S (M minutes, S seconds)
H:M:S (H hours, M minutes, S seconds)
D-H (D days, H hours)
D-H:M (D days, H hours, M minutes)
D-H:M:S (D days, H hours, M minutes, S seconds)
NOTE: The default time is 5 minutes so if you do not set the time parameter than your job will terminate in 5 minutes.
Slurm provides many options for specifying your node and core requirements, and we only cover the basics here. More details can be found at the official Slurm site.
From your job's perspective, of most concern is the number of CPU cores and how they are distributed over the node. We let N represent the number of MPI tasks, and M represent the number of threads needed by the job. Most jobs then fall into one of these categories:
sequential jobs: these jobs will run on a single CPU core (and therefore a single node). In this case N = M = 1
shared memory parallel jobs: These jobs use some sort of multithreading, and so require M CPU cores on a single node. Here M depends on the code and/or the problem being solved, and N=1.
The sbatch parameters you will use the most will be:
-n N or --ntasks=N: This sets the number of MPI tasks for the job, which should be one for the sequential and pure multithreaded cases.
-c M or --cpus-per-task=M: This sets the number of CPU cores to use for each task. All the CPU cores for a given task will be allocated on the same node (although cores from more than one tasks might also be allocated on the same node, as long as all the cores from both tasks fit on the same node). This defaults to one. For sequential jobs and pure MPI jobs, this should be set to one. For pure multithreaded jobs, this should be set to the number of threads desired.
If you want to set the specific amount of memory to use try using the following below:
#SBATCH --ntasks=8
#SBATCH –mem-per-cpu=1024
This will request 8 cores with at least 1GB of memory per core.
NOTE: for --mem-per-cpu, the specified memory size must be in MB.
| |
Home
|
Active
|
Archive
|
Scratch
|
|
Primary
|
Private files or data usage for each user.
|
Can be used for private files or shared files depending on who has access to the project folder.
|
Can be used for private files or shared files depending on who has access to the project folder.
|
Temporary data storage for users and groups.
|
|
Access Location
|
Automatic Login
$home
Or
/home/(your username)
|
/active/(your project folder)
|
/archive/(your project folder)
|
/tmp
|
|
Size
|
Each user has a home folder that has a max storage of 20GB.
|
Each project folder has a max storage of 2TB. Some projects can have more upon request.
|
Each project folder has a max storage of 2TB. Some projects can have more upon request.
|
Each Node has a 850GB scratch storage.
|
|
Purge Policy
|
NO
|
NO
|
NO
|
This is done manually if the scratch disk space gets full.
|
|
Backed Up
|
Yes
|
NO
|
NO
|
NO
|
WinSCP
If you would like to be able to move files around or upload files to the Cluster you can use a software called WinSCP. The instructions below will provide a download link and show you how to setup a connection to the cluster.
- Download WinSCP using this link.
- After the sotware is installed go a head and run the application
- You should get a screen that looks like this after starting it.
- For the Host name: you will want to put clipper.gvsu.edu and then hit save.
- After clicking save the new login site should be selected on the left and you can now click login.
- You will be prompted for a username and password
- After typing in both of those you will have a new window asking for Duo Two-Factor. Select an option you want to use.
- After connecting you will then see the cluster files on the right and your computer you are using on the left.
- From here you can pull files down from your folder on the cluster to your local computer. Or upload ones.