eessi-arch-guard
eessi-arch-guard¶
eessi-arch-guard is a Slurm TaskProlog that reloads an EESSI module stack when the task runs on a CPU or GPU other than the target recorded at module load. The EESSI module chooses that target once. Lmod keeps the recorded MODULEPATH and the EBROOT* paths until the stack is purged and loaded again. The prolog does that purge and load on the compute node, before execve, and prints the environment delta for slurmstepd to apply.
Version 0.2.0.
Why¶
A job submitted from a login node carries that node’s EESSI paths. Slurm delivers the same environment to the allocated node. A binary from the login-node tree then executes on a different microarchitecture: the process can die with SIGILL, take a slow generic path, or open a GPU tree that is absent on the node. Running module load EESSI/<version> again leaves the stack in place, because Lmod evaluates a loaded module file once.
The prolog is the step that still has the task environment and the executing node’s view of the CPU and the GPU. A batch script can purge and reload in the shell. srun ./a.out starts the binary directly, so the corrected variables have to be in the environment before execve.
Python 3.9 or newer on the prolog host, and a virtual environment on a login node.
Reload one stack that was loaded for a different CPU, and read the prolog lines.
Install the TaskProlog, write its config file, and correct a failed reload.
How detection, the waitpkg check, and the reload fit together.
CLI, environment variables, config keys, failure tokens, and the Python API.
Copyright (c) eessi-arch-guard developers