Slurm - A workload manager
- Intro
- End user commands
- Queues, Partitions, Slurm scripts
- Examples - Interactive, GPU Jobs
Slurmctld - A controller
- Accounting, Synced UID/GID
- Auth setups - Munge
- Database setup - MariaDB
- Node Types & Practices
Monitoring Jobs & Activity
Check out this git page for monitoring the slurm jobs, resources usage in each jobs and overall activity running on the cluster.
Sync groups/users (not yet set)
No set up has been done so far to automatically sync new user or group accounts. To be done by 'LDAP' protocol for central authentication.
Before diving into slurm setup, what follows next is a bare bones job scheduler with cgroup enabled kernel, system-run and task spooler. Combining and piping these packages we will be achieving a fraction of functionality what full fledged scheduler like slurm does.
Enabling CPU Sharing with cgroups
Intro on cgroups
Cgroup allows system to restrict usage of system resources. When CPU is shared with cgroups, job schedulars can allocate resources to each jobs limiting CPUs. Current allocations can be checked by,
$ systemd-cgtop
CGroup Tasks %CPU Memory Input/s Output/s
/user.slice 42 5.5 362M - -
Command id can be used to print the user ID and activity.
$ id
uid =1000( rpi2 ) gid =1000( rpi2 ) groups =1000( rpi2 )
Manual restriction of CPU and memory usuage
If we dont restrict memory/CPU usage, the system can take all memory for a jobs, hence its importance to have allocation set for running jobs within limits. We manually tell cgroups using “my.slice” script to constrain the use of memory and CPU.
This can be done by launching systemd-run with the my.slice script. For further examples, refer arch wiki.
Custom allocation
Using a custom cgroup slice file and systemd-run
$ cat /etc/systemd/system/my.slice
[Slice]
CPUQuota=10%
AllowedCPUs=0-2
MemoryHigh=100M
Creating services with a slice script
System service are defined in here,
$ /usr/lib/systemd/system/*.service
[Service]
Slice=my.slice
my.slice, it will be restricted to only 10% CPU, 100 MB memory and with CPU cores 0,1,2.
Running Interactive Command
systemd-run can be run directly with a slice file, instead of creating a service for accessing resources,
$ systemd-run --scope --user --slice=my.slice <command>
What --scope does in command,
oval "immediately run" fit; arrow; oval "systemd/resource usage" fit; arrow; oval "kills after command competition" fit
Ex 1: This runs python simulation with resources defined in my.slice
$ systemd-run --scope --user --slice=my.slice python simulation.py
$ systemd-run --scope -p MemoryMax=2G --user <command> # scope for async execution
Information of Jobs Processing
Below commands helps to monitor resource usage
$ systemd-cgtop
$ systemd-cgls
Raspbian Default Configuration
To print the Raspbian kernel ring buffer,
$ dmseg | less
[Sun Feb 22 00:45:37 2026] Kernel command line: coherent_pool=1M 8250.nr_uarts=0 snd_bcm283
5.enable_headphones=0 cgroup_disable=memory snd_bcm2835.enable_headphones=1 snd_bcm2835.ena
ble_hdmi=1 snd_bcm2835.enable_hdmi=0 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000
We notice here, the cgroup memory allocation is disabled. For sharable memory we to enable it,
Cgroup_V1 Obsolete
Older Linux used to cgroup v1 at
/proc/cgroupsModern Linux systems use cgroup v2 at
/sys/fs/cgroup
Glimpse of Primitive Scheduler
To manage a queue of tasks & run in background and tells run details
- Combing Cgroup with task spooling
tsp$ tsp sleep 5 $ tsp sleep 10 && notifiy-send "Sending this" - Checking the queue,
$ tsp ID State Output E-Level Times(r/u/s) Command [run=1/1] 4 running /tmp/ts-out.xf6eMJ sleep 15 3 finished /tmp/ts-out.q9oe9G 0 5.01/0.00/0.00 sleep 5
Details in Raspbian
Refer issue for enabling details and background on cgroup_memory in cgroupV2.
Intro Resource for cgroupV2
Quick intro about cgroupV2 - pdf. To enable the cgroup memory control, edit a parameter on cmdline text file.
File can be found on,
$ /boot/firmware/cmdline.txt # append at end
.. cgroup_enable=memory
After reboot, make sure memory is sharable. Done using dmesg logs,
[Tue Mar 3 00:38:54 2026] cgroup: Disabling memory control group subsystem
[Tue Mar 3 00:38:54 2026] cgroup: Enabling memory control group subsystem
Additionally, the controller file from path /sys/fs/cgroup/ should have an entry for 'memory',
$ cat /sys/fs/cgroup/cgroup.controllers
cpuset cpu io memory pids
This confirms the memory controller is enabled by cgroup.
Setting up NFS share
NFS (Network File System) protocol allows directories from one Linux machine to be mounted on others over the network.
We need a shared directory, make one using “mkdir” and give it permission by “chmod 777”.
$ sudo mkdir /work
$ sudo chown nobody:nogroup -R /work
$ sudo chmod 777 -R /work
Mount SD card/external drive on /work directory. Frist check connected devices using lsblk and system UUID using blkid to mount these.
$ lsblk
$ blkid # finding uuid
Add UUID in the /etc/fstab file for persistence, Refernce: fstab usage, mount follows,
<device> <dir> <type> <options> <dump> <fsck>
UUID=3f8c216d-f80a-49bf-9b31-435eb5fade07 /work ext4 defaults 0 2
Note, here 0 is for disabling dump check and 2 is for fsck at boot. So when the system boots, SD card will be automatically mounted at /work path.
Reload the daemon, remount and set up the permission. This allows compute nodes to read/write to the shared NFS directory /work
$ systemctl daemon-reload
$ sudo mount -a
$ sudo chown nobody:nogroup -R /work # initial liberal ownership
$ sudo chmod -R 766 /work
Update NFS shared directory
NFS directory config are present in /etc/exports, Refer Arch wiki - Diskless system
Add shared directory /work in the config file /etc/exports on all the nodes connected to head node.
# comment out the example with right path
/work *(rw,sync,no_subtree_check,no_root_squash)
Enable the drive with sudo exportfs -ra, this can be checked with showmount -e
Mounting Shared on Compute Nodes,
(specific to Debian-based)
*Remove the old deb sources, sudo mv /etc/apt/sources.list.d/debian.sources /tmp/
* And install 'nfs-common' sudp apt install nfs-common
* Now mounts shared /work on compute nodes using NFS defaults.
$ echo "rpi2:/work /work nfs defaults 0 0" >> /etc/fstab
User Slurm and Group Sync Database
SLURM works using several background services (daemons) with different purpose.
- slurmdbd - database deamon (file node)
- slurmctld - controller deamon (head node)
- slurmd - client deamon
By default, installing slurm service and slurm client creates corresponding user and group with uid=64030 (default across debian based machine)
Installing SLURM packages,
$ sudo apt install slurmd slurm-client
Setting up database on the head node, done by installing MariaDB server - Reference
$ sudo apt install mariadb-server
Open the SQL shell, and create and account. Set up password and grant all privileges.
$ sudo mysql -u root
create database slurm_acct_db;
create user 'slurm'@'localhost';
set password for 'slurm'@'localhost' = password('xxxxxpass');
grant usage on *.* to 'slurm'@'localhost';
grant all privileges on slurm_acct_db.* to 'slurm'@'localhost';
flush privileges;
Copy Slurm Database /usr/share/doc/slurmd/examples/slurmdbd.conf.simple → /etc/slurm/slurmdbd.conf.
only thing that needs to be address
StoragePass=xxxxxpass
Set permission only for slurm user and group
Secure the configuration file using, (file permission are important for slum service to run)
$ sudo chmod 600 /etc/slurm/slurmdbd.conf
$ sudo chown slurm: /etc/slurm/slurmdbd.conf
Before enabling the database /usr/lib/systemd/system/slurmdbd.service, stop “slurmd” “slurmctld” services.
$ sudo systemctl stop slurmd slurmctld
$ sudo systemctl enable slurmdbd
Setting up cgroupV2
Check this Source
Print cgroup for a single line (0::/init.scope) if not, configure the systemd.
$ cat /proc/1/cgroup
0::/init.scope
Setting up slurm cgroup file
- Remove following lines from the sample config,
# following needs to be removed # CgroupAutomount= # CgroupMountpoint= # CgroupPlugin=cgroup/v1 # -------------------------- # Add CgroupPlugin=autodetect - Check this, "Slurm daemons in a systemd Unit with the special systemd option Delegate=yes."
- On the head node, start the control daemon first,
$ sudo systemctl start slurmctld - If this head node is utilized as a compute node, then
cgroups.confandcgroup_allowd_devices_file.conffiles should be setup as earlier. - One could check if all the services are running with
$ sudo systemctl | grep slurm $ sudo journalctl -xeu slurmctld.service # to debug - Incase of cluster id mismatch,
rm /var/lib/slurm/slurmctld/clustername. - Copy the configs to shared directory,
# Nodes and Shared directory$ cp slurm.conf /work $ cp cgroup.conf /work $ cp cgroup_allowed_devices_file.conf /work - To making sure memory control is enabled with cgroups, run,
cat /sys/fs/cgroup/cgroup.controllers cpuset cpu io pids memory - Overwrite the config from shared directory to
/etc/and set the opt permissions formungekeys.
Accounting with Slurm
- Setting up on the login node, Creating a Slurm Account using
sacctmgrcommand,$ sudo sacctmgr add account TestDrive description="Initial Testing" Organization=home - Checking Accounts details using below command. It prints description, accounts and root.
$ sudo sacctmgr show account Account Descr Org ---------- -------------------- -------------------- root default root account root testdrive initial testing home
Now one can associate users under the Account “testdrive”, but these user and their groups must be synced across the nodes.
- A common group corresponding to the slurm account and users following that, Creating a Common User Group using,
sudo groupadd -g 8500 td for i in {1..5}; do sudo useradd -u 850$i -g 8500 -m \\ -p $(echo td0$i | openssl passwd -1 -stdin)\\ td0$i; done - Corresponding slurm users with group id ( 8500),
for i in {1..5}; do sudo echo "td0$i:td0$i" | chpasswd; done for i in {1..5}; do sudo sacctmgr add user td0$i account=testdrive; done
Debugging SLURM
- Check, if sbatch script is not working,
$ slurmd -C Exception caught: rsmi_init. Configure “slurm.conf” accordingly.
$ cat /var/log/slurmd.conf warning: restricted to a subset of cpusTaking a node down,
$ scontrol show node rpi5 NodeName=rpi5 Arch=aarch64 CoresPerSocket=4 CPUAlloc=0 CPUEfctv=4 CPUTot=4 CPULoad=0.00 AvailableFeatures=(null) ActiveFeatures=(null) Gres=(null)`` NodeAddr=10.61.135.5 NodeHostName=rpi5 Version=24.11.5 OS=Linux 6.12.62+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.12.62-1+rpt1 (2025-12-18) RealMemory=900 AllocMem=0 FreeMem=628 Sockets=1 Boards=1 State=IDLE+DRAIN ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=long BootTime=2026-03-03T07:38:48 SlurmdStartTime=2026-03-03T20:33:38 LastBusyTime=2026-03-03T20:33:38 ResumeAfterTime=None CfgTRES=cpu=4,mem=900M,billing=4 AllocTRES= CurrentWatts=0 AveWatts=0 Reason=Low RealMemory (reported:905 < 100.00% of configured:906) [slurm@2026-03-03T16:28:46]$ sudo scontrol update NodeName=rpi5 State=DOWN Reason=hung_completing $ sinfo -R REASON USER TIMESTAMP NODELIST hung_completing root 2026-03-03T20:37:12 rpi5Resuming the nodes, and reload slurm configuration using below command,
$ sudo scontrol update NodeName=rpi5 State=RESUME $ sudo scontrol reconfigureTo know system information, run
$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST short up 5:00 2 idle rpi[2,7] long* up infinite 4 idle rpi[2,4-5,7]
Set up additional nodes if necessary and check all the nodes are configured properly
Packaging system
Easybuild tutorial here from Ghent University,
Supports lua's environment module, pull up lua and it's dependencies. Check this source
To install lua based module system,
$ sudo apt install lmodCheck installed version using
$ lua -v 5.3You can install other dependencies using same,
$ sudo apt install liblua5.3-dev tcl-dev lua-posix-dev
In the Debian system, lmod directory is /usr/share/lmod/lmod/. The command moduel will be sourced once /etc/profile.d/lmod.sh is
read.