What will happen?

• During March, April and the first half of May users have access to a separate upgraded login node (nebula2) and compute nodes running the new OS for testing.

• On May 15, Nebula will have a few hours downtime where all compute nodes are switched and where the “nebula” login node alias is redirected to point to a login node with the new OS by default.

• Some time in the August - September timeframe there will be 1-2 days of downtime for each system to finalize the switchover.

What do I need to do?

To continue using Nebula without interruption, you should move your jobs to the upgraded part of Nebula sometime between now and April 30.

You yourself choose when to do this, but we recommend that you do it sooner rather than later to leave as much time as possible for you (and possibly NSC) to fix any problems you may encounter.

Typically, it will look something like this:

1. Make some basic tests to see what you need to change and verify that at least one job seems to start correctly in the upgraded part of Nebula (i.e when submitted from nebula2.nsc.liu.se (hostname “nebula2”)).
2. Leave existing running and queued jobs as-is (they can run in the non-upgraded part until April 30). You may of course cancel queued jobs and resubmit them in the new part of the system if you prefer.
3. Submit any new jobs from nebula2.nsc.liu.se (hostname “nebula2”) with possibly modified settings/job scripts/modules/…
4. Monitor the first few new jobs and once they have completed, check that the results are as expected.

Once your jobs are working in the upgraded part of Nebula, please stop using the non-upgraded part. NSC will monitor the demand for nodes in both parts of the system and adjust the number of nodes in each part accordingly, i.e it will not be easier to run jobs in the old part after the first few days.

Storage (/prod*, /home etc) are not affected (i.e no need to move your data).

Software migration guide

This section covers how to get software you have been running on CentOS 7 to run on Rocky Linux 9.

Applications provided by NSC

Most applications provided by NSC will continue to work.

Initially we will provide the latest and most commonly used versions of the most common applications.

Additional applications and versions will be added later, but only if requested via NSC Support.

If you don’t find your application/version in the output from module avail (on upgraded nodes) and it’s not mentioned on this page already, please contact NSC Support and ask about it.

Some of the NSC-provided applications will run natively in the new operating system, some will have been recompiled and others will be run in a CentOS 7 container using Apptainer. We aim to make all this as transparent to the user as possible, but minor adjustments to job scripts may be needed (and documented on this page).

Compiled applications that you or your project have installed yourself on CentOS 7 to run on compute nodes

If you or someone in your project has built or installed your own application you will need to choose a suitable way to run the application in the new environment.

There are several ways to do this (recompile, run as-is, run in a container). Which one to use depends on the application. NSC Support can assist you with this. Some documentation is provided below, more will follow later.

Inside a supercomputer job, please try the following steps (i.e., inside a submit script, or after issuing interactive). Note that it is a good idea to verify not just that the application starts, but also to try an example job to check that it performs as expected.

1. Run your application as usual, i.e., as mpprun <application>. Some applications just work without any additional steps. However, you may see error messages about “missing symbols” or “missing libraries”. In that case, go to the next step.

2. Try mpprun --compat el7 <application>. This runs your application inside a “compatibility configuration” designed to mimic CentOS 7 as closely as possible. The compatibility consists of running your application, leveraging modern OS features, inside an NSC provided OS container image of the old CentOS 7 OS mimicking the old software environment to very high fidelity. (Note: if you have already recompiled your binary on Rocky 9, you should NOT run it with the --compat el7 flag.)

3. If step 2 also fails, the next step is to try to rebuild the application with one of the build environments provided in the Rocky Linux 9 environment. Load an appropriate buildenv-<something> module and follow the usual instructions for how to install software. Note: make sure to completely rebuild the application, i.e., do a make dist-clean, make clean, or equivalent as the first step to make sure all components are rebuilt from scratch.

Compiled applications that you or your project have installed yourself on CentOS 7 to run on the login nodes

If you or your project have installed a software application that you like to use on the login node for data analysis etc., please try the following steps:

1. Try to run your application as usual, e.g., as ./<application>. Some applications just work without any additional steps. However, you may see error messages about “missing symbols” or “missing libraries”. In that case, go to the next step.

2. Try hpc_compat_el7 <application>. This runs your application inside a “compatibility configuration” designed to mimic CentOS 7 as closely as possible. The compatibility solution is the same as that used for mpprun --compat el7. (Note: if you have already recompiled your binary on Rocky 9, you should NOT run it via hpc_compat_el7.) `
3. If step 2 also fails, the next step is to try to rebuild the application with one of the build environments provided in the Rocky Linux 9 environment. Load an appropriate buildenv-<something> module and follow the usual instructions for how to install software. Note: make sure to completely rebuild the application, i.e., do a make dist-clean, make clean, or equivalent as the first step to make sure all components are rebuilt from scratch.

If your application includes a GUI:

• 2D graphics: test using the “compatibility configuration” (step 2 above), if this fails please try to use an interactive session. If this also fails you may have to rebuild your application (step 3 above).

• 3D graphics: Your application will have to be reinstalled (step 3 above)

Python applications

The Rocky Linux 9 environment provides Anaconda and Condaforge modules that work the same way as those available in CentOS 7. You should be able to load these modules and just keep using the conda environments you installed under CentOS 7.

If you’d rather use the modules provided under Python/, the Rocky Linux 9 environment provides a similar Python 3 module that you can try. However, users who find that the changes in dependency libraries compared to the CentOS 7 modules cause issues are directed to set up the necessary environments using the Anaconda or Condaforge modules.

Note that this particular part of the backwards compatibility for Python is tested on other NSC clusters, but still under construction on Nebula as of 2024-03-25. If old environments you need don’t work please contact NSC Support.

Updated mpprun

Note that the mpprun application differs substantially on Rocky Linux 9 compared to CentOS 7. Use mpprun -h to see the help, and the option mpprun -i <binary> can be helpful for power users to understand more precisely how mpprun will launch a binary through the various MPI-specific launchers.

Old build environments

The old build environments of CentOS 7, buildenv-<something>/<old-version>, will not be carried over to EL9, but there will be newer version replacement modules available corresponding to them. If you can not make use of these refreshed modules, please contact NSC Support for assistance on your migration to these environments.

Application performance

NSC has run applications on the upgraded part of Nebula (both recompiled, run as-is, and run through a CentOS 7 container) and not seen any significant negative performance impact. In fact, some applications are even running faster.

If you see a significant performance loss in the upgraded part of Nebula, please let NSC Support know as soon as possible.

If you cannot get your application working on Rocky 9

1. Ask NSC Support for help. Do this early, do not wait until April 30!
2. If the application cannot be made to work even with assistance from NSC, we have the option to leave some non-upgraded compute nodes running for a while (but no longer than June 30th, 2024). However, only users that have asked for help and that NSC has not been able to help will be allowed to use such nodes. The number of such nodes will likely be very limited.

Some more technical details

Users can choose which part to use by logging in to the corresponding login node (using SSH or Thinlinc):

• Upgraded/Rocky 9: nebula2.nsc.liu.se (a.k.a nebula2, available from November 13th)
• Non-upgraded/CentOS 7: nebula.nsc.liu.se (a.k.a nebula1, available until January 8th)

The number of upgraded compute nodes will gradually increase during the upgrade window. The rate at which this happens will be adjusted based on how fast users are moving their jobs.

Why are we doing this upgrade?

The current operating system CentOS 7 (which is based on RedHat Enterprise Linux 7) will not receive any security updates after 2024-06-30.

As Nebula is planned to continue operating longer than this, we need to upgrade the operating system.