Slurm has numerous reason codes for jobs remaining in queue. A complete list can be found here: Job Reason Codes.
squeue
The squeue command can be used to look up job pending codes.
Examples:
squeue -j <jobid>
squeue --me
Common Errors
- AssocGrp* — The job's association has reached an aggregate limit.
- AssocMax* — A portion of the job request exceeds a maximum limit (e.g., PerJob, PerNode) for the requested association.
- BeginTime — The job's earliest start time has not yet been reached.
- Dependency — This job has a dependency on another job that has not been satisfied.
- Max*PerAccount — A portion of the job request exceeds the per-Account limit on the job's QOS.
- Priority — One of more higher priority jobs exist for the partition associated with the job or for the advanced reservation.
- QOSGrp* — The job's QOS has reached an aggregate limit.
- QOSMax* — A portion of the job request exceeds a maximum limit (e.g., PerJob, PerNode) for the requested QOS.
- Resources — The resources requested by the job are not available (e.g., already used by other jobs)
GPU Failures
If you happen to run squeue --me and receive a job reason code of "AssocGrpGRES" or "AssocGrpGRESMinutes" then your account is using all it's available GPUs. To view how many GPUs are available for your account use:sacctmgr show -s account name=<ACCOUNT> format=grptres%40,grptresrunmins%50
To see how many GPUs are actually in use for an account use:
squeue -A <account> -O StateCompact:3,JobID:10,UserName,tres-alloc:70,ReasonList:20,TimeLimit,TimeLeft | grep gres\/gpu