<style type="text/css"> code {color: #78c9ff} small {opacity: .5} .reveal li li {opacity: .75; font-size: .75em} .container { background: url(https://pad.gwdg.de/uploads/63eb96a9-f217-4842-98d5-f08f1afd607b.png); background-size: cover; overflow-y: hidden; } .footer { display: none; } </style> # GöHPCoffee 2022-01-19 ## Slurm Common Issues and Random Tips and Tricks! Marcus Boden --- ## My job vanished without a trace! - take a look with `sacct -j <jobid>` - it should always show up there - did you use scratch? Did you submit from scratch? --- ## Why doesn't my job run already ### There are empty nodes! - Slurm calculates internal priority of jobs - Your job might just not have the highest priority. - You can check with `sprio` - limit to partition with e.g. `-p medium` - customize format `-o "%.15i %.10Y %.10u` - sort by prio `-S Y` --- ## Reproducibility - Slurm carries environment (e.g. modules) at submit into job - disbabled on Emmy - Controlled by the [--export argument](https://slurm.schedmd.com/sbatch.html#OPT_export). - recommended: --export=none - clean slate each time - no accidental modules/settings carried into job - You need to set everything in the jobscript --- ## command line vs #SBATCH vs Environment Order of precendence: 1. CLI parameters `sbatch -p standard96 <jobscript>` 3. [Environment Variables](https://slurm.schedmd.com/sbatch.html#SECTION_INPUT-ENVIRONMENT-VARIABLES) `SBATCH_PARTITION=standard96` 5. Jobscript `#SBATCH -p standard96` --- ## Quick tests and more - `sbatch --wrap="sleep 300"` - No jobscript required - `srun/sbatch --test-only` - gets estimated waiting time and placement - `squeue --me` - You don't need `squeue -u $USER` anymore - Useful for debugging - `sacct -B -j <jobid>` shows jobscripts - `sacct --env -j <jobid>` shows Environment - `sacct -j <jobid> -o MaxRSS` shows used mem - `sacct` has a lot of options. See `sacct -e` --- ## Customize your squeue ### (or other Slurm commands) - The output of `squeue` is can be controlled - on the command line - `-o <FORMAT>` - via Environment Variables - `SQUEUE_FORMAT="<FORMAT>"` - Take a look at the manpage for options - Example on Emmy: ```SQUEUE_FORMAT="%.18i %.12P %.12j %.8u %.8a %.9T %.10M %.5D %R"```
{"title":"GöHPCoffee","author":"Marcus Boden","type":"slide","slideOptions":{"transition":"none","mouseWheel":true,"keyboard":{"38":"prev","40":"next"}}}