Extending the time limit of a job

You can extend the time limit of a running job by using the command nsc-boost-timelimit on the login node. nsc-boost-timelimit --help will show the available options.

Due to technical limitations in the job scheduler, boosting the time limit of a job is only possible once it has started. If you want to be notified of your job having started, you can use the --mail-type=BEGIN option to sbatch.

The time limit you specify with the -t option is the new total time limit (e.g to extend a one hour job by one hour, use nsc-boost-timelimit -t 2:0:0).

You can also use the -a option to add to the existing timelimit, e.g nsc-boost-timelimit -a 1.5 to add 1.5 hours to the existing timelimit.

Example: check the cost of extending the time limit of a job to 8 hours

[x_makro@tetralith1 ~]$ nsc-boost-timelimit -c -t 8:0:0 14399940
The cost for boosting the timelimit of job 14399940 is 1200.0 tokens
Project nsc-guest has 29554.8 tokens available

Example: extend the time limit of a job to 8 hours

[x_makro@tetralith1 ~]$ nsc-boost-timelimit -t 8:0:0 14399940
Timelimit changed, cost: 1200.0 tokens.
Project nsc-guest now has 28354.8 tokens available

Example: extend the time limit of a job to 5 days

[x_makro@tetralith1 ~]$ nsc-boost-timelimit -t 5-0:0:0 14399940
Timelimit changed, cost: 17920.0 tokens.
Project nsc-guest now has 10434.8 tokens available

Cost

The base cost of extending the time limit of a job is currently 10 tokens per core hour of extension¹ (e.g extending the time limit of a 16-core, one hour job to two hours costs 160 tokens).

Limitations on extending the time limit

The timelimit can only be extended as far into the future as the cluster’s normal limit on job length (3 days on Sigma and 7 days on Tetralith). This is for your own protection. NSC typically provides a week’s warning of upcoming maintenance etc, and any longer-running jobs might then be killed at the start of the maintenance period. If you want to accept the risk of your job being killed by NSC in such situations, you can use the --accept-the-risks option and set any time limit you want (as long as you have enough tokens). Another option is to extend your job several times (e.g 7 days at a time).

Example: fail to extend the time limit of a job to 8 days, then using the --accept-the-risks option to extend it anyway

[kronberg@tetralith1 ~]$ nsc-boost-timelimit -t 8-0:0:0 17462797
==== FAIL ====
Your request could not be fulfilled: You may only extend a job 168 hours into the future (unless using the "--accept-the-risks" option)!

[kronberg@tetralith1 ~]$ nsc-boost-timelimit --accept-the-risks -t 8-0:0:0 17462797
Timelimit changed, cost: 30560.0 tokens.
Project nsc now has 969440.0 tokens available

You cannot use the tool to extend the time limit on a job if one or more of the compute nodes it uses are being taken offline (“draining”) for urgent security updates, or if one or more of the nodes is reserved for some other purpose. If you absolutely need such a job extended, contact NSC Support and explain why that job is extra important to you.