Job Failure Codes

The documentation for Slurm Failure codes can be found here: Slurm Exit Codes.

A job's exit code (aka exit status, return code and completion code) is captured by Slurm and saved as part of the job record. For sbatch jobs, the exit code that is captured is the output of the batch script. For salloc jobs, the exit code will be the return value of the exit call that terminates the salloc session. For srun, the exit code will be the return value of the command that srun executes.

Any non-zero exit code will be assumed to be a job failure and will result in a Job State of FAILED with a Reason of "NonZeroExitCode".

The exit code is an 8 bit unsigned number ranging between 0 and 255. While it is possible for a job to return a negative exit code, Slurm will display it as an unsigned value in the 0 - 255 range.

To view a jobs exit code, the sacct command can be used.

sacct -j <JobID> --format=JobID,JobName,State,ExitCode,DerivedExitCode,Comment

Users can also use the sjobexitmod command to view job submission completed data.

sjobexitmod -l <JobID>

Users can also use the sjobexitmod command to leave comments for failed jobs.

sjobexitmod -r "out of memory" <JobID>