Great Lakes Cluster Defaults
| Cluster Defaults | Default Value |
|---|---|
| Default walltime | 60 minutes |
| Default memory Per CPU | 768 MB |
| Default number of CPUs | no memory specified: 1 core Memory specified: memory/768 = # of cores (rounded down) |
| /scratch file deletion policy | 60 days without being accessed. (see POLICIES ) |
| /scratch quotas per root account | 10 TB storage and 1 million inode limit (see POLICIES ) |
| /home quota per user | 80 GB |
| Max queued jobs per user per account | 5,000 |
| Shell timeout if idle: | 2 hours |
Great Lakes Partition Limits
| Partition Limit | standard | gpu | gpu_mig40 | spgpu | largemem | standard-oc | viz* | debug* |
|---|---|---|---|---|---|---|---|---|
| Max walltime | 2 weeks | 2 hours | 4 hours | |||||
| Max running Mem per root account | 3,500 GB | 1.5 TB | 660 GB | 3,500 GB | 40 GB | |||
| Max running CPUs per root account | 500 cores | 36 cores | 132 cores | 500 cores | 8 cores | |||
| Max running GPUs per root account | n/a | 5 | n/a | n/a | 5 | n/a | ||
*there is only one gpu in each viz node, and these are only accessible through Open OnDemand’s Remote Desktop application
*all debug limits are per user, and only one job can run at a time. largemem and standard-oc limits are per account
Please see the section on Partition Policies for more details.
Great Lakes Storage: /home and /scratch
On the Great Lakes cluster, users have access to two main types of storage: /home and /scratch. Each serves a different purpose:
- /home is for personal, non-project work like scripts, small data files, and software builds.
- /scratch is for active project data needed while running jobs. It’s larger and faster but temporary.
Both directories have specific rules about who can access your files.
File Permissions:
To protect your data and the system, files or folders that are set to be readable or writable by “other” (users outside your account or group) will have those permissions automatically removed. Always set appropriate permissions to limit access to just yourself or your project group. If you need a group to manage access to a directory or files, let us know.
Home Directories
Every user is assigned a /home directory for non-project work. This space is intended for storing scripts, small data files, software builds, and other personal files unrelated to specific funded projects.
- Each /home directory has a hard quota of 80 GB.
- Files stored here should only be accessible by the user and their primary group(s).
- Files or directories with global (other) read or write permissions will have those permissions automatically removed to maintain system security and data integrity.
Use this space for development, light workflows, and other general-purpose computing needs. For large datasets or collaborative work, use the /scratch directory structure described below.
Scratch Directories
Every user has a /scratch folder for each Slurm account they belong to. This space is designed for fast access to active data and should only be used for files needed to run current jobs on the cluster.
- /scratch uses a high-performance file system (GFPS), which is faster than other storage volumes.
- It is not backed up, so do not store important or long-term files here. Move important data to a durable location when your jobs are done.
- Each Slurm account also has a shared_data folder where all users in the account can share files.
- The folders are set up so that files can be accessed by anyone in the same Slurm account group, to make working together easier.
Example:
/scratch/myproject_root/myproject1
/scratch/myproject_root/myproject1/bob
/scratch/myproject_root/myproject1/shared_dataSlurm Accounts and Funding:
Each Slurm account may match a different source of funding:
- myproject0 → UMRCP (free allocation)
- myproject1 → Research Allocation X (paid)
- myproject2 → Research Allocation Y (paid)
Make sure to use the correct account for each project so your usage is tracked properly.