Tetralith large-scale data analysis nodes

This service provides access to four “fat” (384 GiB RAM) compute nodes in Tetralith for large-scale data analysis.

This service is currently limited to members in NAISS Large Storage projects (and existing SNIC Large Storage proejcts) with allocations larger than 200 TiB.

This rather arbitrary limit was chosen to enable the same job scheduling policy used on Vagn (and later DCS) to be used (at least initially) for the analysis nodes. Since only four nodes are available (with separate funding from NAISS) for this service, they cannot be effectively shared by too many users, and thus NSC has decided to limit access to the largest storage projects.

Please note that access to these nodes is not automatic. A NAISS Large Storage project that wishes to use these nodes should contact NSC Support to get access.

Using the analysis nodes

Four Tetralith compute nodes with 384 GB RAM each are dedicated for the DCS projects for analysis use.

Since the number of nodes is limited, please make sure to only request as much resources as you need. I.e don’t request a full node (“-N1”) if you can make do with just a few cores (e.g “-n2”).

To use the analysis nodes, you need to use our batch scheduler/resource manager. There are two main reasons for this:

• Make it possible to automate data processing on a large scale using batch jobs.
• Make it possible for many users to share the analysis nodes without interfering with each other (e.g by using up all memory on a node).

The analysis nodes are in a “reservation” on Tetralith. This means that they are separate from the other compute nodes - you must specify that you want to use them, and you must be authorized to use them.

To use analysis nodes, you must:

1. Use the compute project your Large Storage project has designated. If you don’t know which project this is, ask the project PI or NSC Support

2. State that you want to use the analysis reservation: --reservation=lsda

Running batch jobs without annoying other users

The analysis nodes are primarily intended for interactive use. It is permitted to run batch jobs on them as long as you make sure that you don’t use all available resources.

Note: one user can easily block all analysis nodes for other users by submitting a large volume of jobs at once. Doing this is not acceptable! Analysis nodes must be available for interactive use, i.e it is not acceptable for most users to have to wait for several hours to get an interactive shell. Please use common sense!

Common sense can be assisted by checking the queue status before submitting a large volume of jobs. To get an overview of running and queued jobs, use the squeue command with suitable options, e.g squeue --reservation=dcs -o "%.10i %.9P %.8u %.8T %.11L %.11l %.8N %.10m".