Slurm HOWTO for Administrators

ADMINISTRATION

Note: Most commands in this section are restricted to system administrators.

How to stop Slurm from scheduling jobs?

You can stop Slurm from scheduling jobs on a per partition basis by setting that partition's state to DOWN. Set its state UP to resume scheduling. For example:

$ scontrol update PartitionName=foo State=DOWN
$ scontrol update PartitionName=foo State=UP

How to reboot (all) nodes as soon as they become idle?

$ scontrol reboot ASAP nextstate=RESUME <node1>,<node2>  # specific nodes
$ scontrol reboot ASAP nextstate=RESUME ALL              # all nodes

How to cancel pending reboot of nodes?

$ scontrol cancel_reboot <node1>,<node2>

How to check current node status?

$ scontrol show node <node>

How to instruct all Slurm daemons to re-read the configuration file

$ scontrol reconfigure

How to prevent a user from submitting new jobs?

Use the following sacctmgr command:

$ sacctmgr update user <username> set maxsubmitjobs=0

Notes:

• Job submission is then rejected with the following message:

$ sbatch job.slurm
sbatch: error: AssocMaxSubmitJobLimit
sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)

• Use the following command to release the limit:

$ sacctmgr update user <username> set maxsubmitjobs=-1

How to drain node(s)?

$ scontrol update NodeName=<node1>,<node2> State=DRAIN Reason="Some Reason"

Notes:

• Reason is mandatory.

• Do not just set state DOWN to drain nodes. This will kill any active jobs that may run on that nodes.

How to list reason for nodes being drained or down?

$ sinfo -R

How to resume node state?

$ scontrol update NodeName=<node1>,<node2> State=RESUME

How to create a reservation on nodes?

Suggested reading: https://slurm.schedmd.com/reservations.html

$ scontrol create reservation user=root starttime=now duration=UNLIMITED flags=maint,ignore_jobs nodes=ALL
$ scontrol create reservation user=root starttime=2020-12-24T17:00 duration=12:00:00 flags=maint,ignore_jobs nodes=<node1>,<node2>
$ scontrol show reservation

Note: Add "FLEX" flag to allow jobs that qualify for the reservation to start before the reservation begins (and continue after it starts). Add "MAGNETIC" flag to attract jobs that qualify for the reservation to run in that reservation without having requested it at submit time.

How to create a floating reservation on nodes?

Use the flag "TIME_FLOAT" and a start time that is relative to the current time (use the keyword "now"). In the example below, the nodes are prevented from starting any jobs exceeding a walltime of 2 days.

$ scontrol create reservation user=root starttime=now+2days duration=UNLIMITED flags=maint,ignore_jobs,time_float nodes=<node1>,<node2>

Note: Floating reservations are not intended to run jobs, but to prevent long running jobs from being initiated on specific nodes. Attempts by users to make use of a floating reservation will be rejected. When ready to perform the maintenance, place the nodes in DRAIN state and delete the reservation.

How to use a reservation?

$ sbatch --reservation=foo_6 ... script.slurm

How to delete a reservation?

$ scontrol delete ReservationName=foo_6

How to get node oriented information similar to 'mdiag -n'?

$ sinfo -N -l

Fields can be individually customized. See sinfo man page. For example:

$ sinfo -N --format="%8N %12P %.4C %.8O %.6m %.6e %.8T %.20E"

NODELIST PARTITION    CPUS CPU_LOAD MEMORY FREE_M    STATE               REASON
n0001    standard*    0/16     0.01 128000 120445     idle                 none
n0002    standard*    0/16     0.01 128000 120438     idle                 none
n0003    standard*    0/0/      N/A 128000    N/A    down*       Not responding

How to get node oriented information similar to 'pbsnodes'?

$ scontrol show nodes                     # One paragraph per node (all nodes)
$ scontrol show nodes <node1>,<node2>     # One paragraph per node (specified nodes) 
$ scontrol -o show nodes                  # One line per node (all nodes)
$ scontrol -o show nodes <node1>,<node2>  # One line per node (specified nodes)

How to update multiple jobs of a user with a single scontrol command?

Not possible. But you can, e.g., use squeue to build the script taking advantage of its filtering and formatting options.

$ squeue -tpd -h -o "scontrol update jobid=%i priority=1000" >my.script

You can also identify the list of jobs and add them to the JobID all at once, for example:

$ scontrol update JobID=123 qos=reallylargeqos
$ scontrol update JobID=123,456,789 qos=reallylargeqos
$ scontrol update JobID=[123-400],[500-600] qos=reallylargeqos

Another option is to use the JobName, if all the jobs have the same name.

$ scontrol update JobName="foobar" UserID=johndoe qos=reallylargeqos

However, Slurm does not allow the UserID filter alone.

How to create a new account?

Add account at top level in association tree:

$ sacctmgr add account <accountname> Cluster=justus Description="Account description" Organization="none"

Add account as child of some parent account in association tree:

$ sacctmgr add account <accountname> parent=<parent_accountname>

How to move account to another parent?

$ sacctmgr modify account name=<accountname> set parent=<new_parent_accountname>

How to delete an account?

$ sacctmgr delete account name=<accountname>

How to add a new user?

$ sacctmgr add user <username> DefaultAccount=<accountname>

How to add/remove users from an account?

$ sacctmgr add user <username> account=<accountname>                  # Add user to account
$ sacctmgr add user <username> account=<accountname2>                 # Add user to a second account
$ sacctmgr remove user <username> where account=<accountname>         # Remove user from this account

How to change default account of a user?

  
$ sacctmgr modify user where user=<username> set DefaultAccount=<default_account>

Note: The user must already be associated with the account you want to set as default.

How to show configuration settings of the slurm database daemon?

$ sacctmgr show configuration

How to implement user resource throttling policies?

Quoting from https://bugs.schedmd.com/show_bug.cgi?id=3600#c4

With Slurm, the associations are meant to establish base limits on the
defined partitions, accounts and users. Because limits propagate down
through the association tree, you only need to define limits at a high
level and those limits will be applied to all partitions, accounts and
users that are below it (parent to child). You can also override those
high level (parent) limits by explicitly setting different limits at
any lower level (on the child). So using the association tree is the
best way to get some base limits applied that you want for most cases. 
QOS's are meant to override any of those base limits for exceptional
cases. Like Maui, you can use QOS's to set a different priority.
Again, the QOS would be overriding the base priority that could be set
in the associations.

How to set a resource limit for an individual user?

Suggested reading: https://slurm.schedmd.com/resource_limits.html

Example:

$ sacctmgr modify user <username> set maxjobs=1            # Limit maximum number of running jobs for user
$ sacctmgr list assoc user=<username> format=user,maxjobs  # Show that limit
$ sacctmgr modify user <username> set maxjobs=-1           # Remove that limit

How to fix/reset a user's RawUsage value?

$ sacctmgr modify user <username> where Account=<account> set RawUsage=<number>

How to create/modify/delete QOSes?

Suggested reading: https://slurm.schedmd.com/qos.html

Examples:

$ sacctmgr show qos                                      # Show existing QOSes
$ sacctmgr add qos verylong                              # Create new QOS verylong
$ sacctmgr modify qos verylong set MaxWall=28-00:00:00   # Set maximum walltime limit
$ sacctmgr modify qos verylong set MaxTRESPerUser=cpu=4  # Set maximum maximum number of CPUS a user can allocate at a given time
$ sacctmgr modify qos verylong set flags=denyonlimit     # Prevent submission if job requests exceed any limits of QOS
$ sacctmgr modify user <username> set qos+=verylong      # Add a QOS to a user account
$ sacctmgr modify user <username> set qos-=verylong      # Remove a QOS from a user account
$ sacctmgr delete qos verylong                           # Delete that QOS

How to find (and fix) runaway jobs?

$ sacctmgr show runaway

Notes:

• Runaway jobs are orphaned jobs that don't exist in the Slurm controller but have a start and no end time in the Slurm data base. Runaway jobs mess with accounting and affects new jobs of users who have too many runaway jobs.
• If there are jobs in this state this command will also provide an option to fix them. This will set the end time for each job to the latest out of the start, eligible, or submit times, and set the state to completed.

How to show a history of database transactions?

$ sacctmgr list transactions

Note: Useful to get timestamps for when a user/account/qos has been created/modified/removed etc.