Cluster Defaults, Partition Limits, and Storage

Armis2 Cluster Defaults

Cluster Defaults Default Value
Default Walltime 60 minutes
Default Memory Per CPU 768 MB
Default Number of CPUs no memory specified: 1 core
Memory specified: memory/768 = # of cores (rounded down)
/scratch file deletion policy 60 days without being accessed (see Scratch Storage Policies below)
/scratch quota per root account 10 TB storage limit (see Scratch Storage Policies below)
/home quota per user 80 GB
Max queued jobs per user per account 5,000 
Shell timeout if idle: 15 Minutes

Armis2 Partition Limits

Partition Limit standard gpu largemem
Max Walltime 2 weeks
Max running Mem per root account 5160 GB 2210 GB
Max running CPUs per root account 1032 cores 84 cores
Max running GPUs per root account n/a 10  n/a

Armis2 Login Limits

NOTICE: The login nodes are for interacting with the Slurm scheduler or code and data management; they are not for running workloads. For sessions on the login nodes, users are limited to:

  • 2 cores
  • 4G of memory

If you need additional resources for testing:

  • Submit an interactive job with salloc: ( salloc –account=[your account] -p debug –time=10:00 –nodes=1 –ntasks-per-node=4 (this will create an interactive session on 1 node with 4 processors for 10 minutes)
  • Use Open OnDemand.
  • Use the debug partition if your testing does not require more than 8 CPUs and 40G of memory.

Armis2 Storage

Every user has a /scratch directory for every Slurm account they are a member of.  Additionally for that account, there is a shared data directory for collaboration with other members of that account.  The account directory group ownership is set using the Slurm account-based UNIX groups, so all files created in the /scratch directory are accessible by any group member, to facilitate collaboration.

Example:
/scratch/msbritt_root
/scratch/msbritt_root/msbritt
/scratch/msbritt_root/shared_data

Please see the section on Storage Policies for more details.