When will my job start?

A very common question is “when will my job start?”. Unfortunately, it is very difficult to answer for most jobs.

The job queue is not a FIFO. It is continually re-sorted by job priority, so that the highest priority job is always considered first whenever nodes become available to start new jobs.

At any given time, the job scheduler can tell you when jobs would be started under perfect conditions, but that estimated start time is only valid if the following conditions are met:

• No new higher-priority jobs are submitted before the job is started. If higher-priority jobs are submitted, the actual start time of the job will be later than expected. It is common for this to significantly affect the start time of lower-priority jobs (since most projects and thus most new jobs will have a higher priority).

• All jobs run for exactly the time (TimeLimit, -t) they have asked for. If running jobs finish ahead of schedule, the actual start time of your job will be sooner than expected. It is common for this to significantly affect the start time of a job (since many jobs run for much less time than they asked for).

• The same number of compute nodes are available. If nodes are taken offline or online for any reason, the actual start time of jobs will be delayed. It is rare for this to significantly affect the start time of a job since the number of such nodes are usually small relative to the total size of the cluster.

• No user uses nsc-boost-priority to increase the priority of a queued job. It is rare for this to significantly affect the start time of a job, since projects can only boost a small amount of jobs.

In practice, this means that the estimated start time is only accurate for a handful of the highest priority jobs in the queue. For low-priority jobs, the estimated start time (if it is even shown at all, see below) is more or less unusable.

You can use the command squeue to show you the estimated start time of jobs using the format option %S. Example (shows estimated start time of all your pending jobs):

squeue -o "%.12i %.24S"  --state=PD -u $USER

Please note that many jobs will not even have an estimated start time shown. This happens in two cases:

• The job was submitted so recently that the job scheduler has not yet had a chance to look at it.

• The job has such a low priority that the job scheduler did not even consider it for starting. For performance reasons the job scheduler will only consider a certain number of jobs (in priority order) to be started before it has to start all over again. This means that low priority jobs will not be looked at, and will not have a start time set. In practice, this does not matter much, as jobs with such a low priority will not be started anytime soon, so the estimated start time would be useless anyway.