File systems on MET resources

The Stratus and Nebula clusters have access to several file systems. These file systems serve different purposes and are described below.

Home directories (/home)

On the Stratus and Nebula clusters, each user has a home directory. The home directories are not shared between the two clusters: The /home file system on Stratus is different from the /home file system on Nebula. If your user name is sm_abcde, your home directory will be /home/sm_abcde. If you need to refer to your home directory in a script, use $HOME instead of hardcoding the name.

Your home directory is backed up to tape at least weekly. Use your home directory for reasonable amounts of important data that cannot be recreated easily. If you need something restored from a backup, please contact . Sometimes you can also restore deleted files yourself by browsing the file system snapshots.

On Nebula, as default you can only store 100 GiB in your home directory, due to user quotas. To see what your quota is, and how much of it you have left, use the command nscquota. Your quota can be increased if you need it to. If so, please contact .

/nobackup/forsk (on Nebula)

All Nebula users have a personal directory in /nobackup/forsk. If your user name is sm_abcde, your personal directory on /nobackup/forsk will be /nobackup/forsk/sm_abcde). This is where you should store the bulk of your data.

User quotas are used on the forsk file system. The default quota is currently 10 TiB. To see what your quota is, and how much of it you have left, use the command nscquota. Your quota can be increased if you need it to. If so, please contact .

/nobackup/prod1 and /nobackup/prod2 (on Stratus and read-only on Nebula)

On Stratus, NWP production data is stored in /nobackup/prod1 and /nobackup/prod2. These file systems are also available from Nebula, but read-only.

Group quotas are used on the prod1 and prod2 file systems.

Node-local scratch storage (/scratch/local)

Each compute node has a scratch file system that can be used for temporary storage on that node while your job is running. The data will be deleted when the job finishes.

On Stratus and Nebula, you have approximately 200 GiB available per thin node, and 870 GiB per fat node. You need to use the subdirectory created for you by the system and pointed to by the $SNIC_TMP environment variable.

Installed software (/software)

The /software file system contains software installed by NSC. Users cannot write to that file system.

Most of the software is made available through the "module" system and can be listed using module avail. Some libraries may not have modules associated with them, so you might find it useful to browse the /software/apps directory for them.

The Stratus and Nebula clusters have their own /software file systems. The directory is not shared between the two clusters.

Snapshots

The /home, /nobackup/forsk, and /software file systems use snapshots. A snapshot is a read-only view of some previous point in time on the file system. Snapshots can be used to restore deleted files.

You will find the snapshots for a particular file system in the directory .snapshots of the file system's top directory. For instance, the snapshots for /home is in the directory /home/.snapshots. In the snapshot directory, snapshots are named in the format YYYY-MM-DDTHHMM+0000. The suffix +0000 indicates that the time is in UTC.

Below is an example on how the user sm_abcde would restore the state of the file foo.c from their home directory to how it was 4 days ago. In this example, 2017-06-10 was 4 days ago.

$ date
2017-06-14 08:02 UTC
$ cp /home/.snapshots/2017-06-10T0800+0000/sm_abcde/foo.c /home/sm_abcde/

Files created and deleted in the time between when two snapshots were taken cannot be restored. Files that were deleted too long ago – before the currently oldest snapshot was taken – cannot be restored from snapshots. The oldest snapshot is usually 7 days old, except for the /nobackup/forsk file system where the oldest snapshot is usually 4 days old.

Note that snapshots are not guaranteed to exist and do not serve as a backup. If some file system runs out of space, snapshots will be deleted, and if the server suffers a catastrophic failure, snapshots will be also affected. Actual backups of the /home file systems are taken to tape at least weekly. If you need something restored from a backup, please contact .

Displaying quota and usage information

The command nscquota on the Nebula cluster shows information about your quota on the /home and /nobackup/forsk file systems. It can also show quota information for all Nebula users if the -a flag is specified: nscquota -a.

The command nscquota is also available on the Stratus cluster. It can show quota information for all Stratus groups if the -g flag is specified: nscquota -g. On file systems on Stratus where quotas are not enforced (like on /home), nscquota will still show usage information.

Run nscquota --help on either cluster for more information.