What happens to data in a project directory (/proj/XXXX) when the project ends?

When the project that “owns” a project storage directory ends, the project no longer has the right to continue using the NSC Centre Storage system to store their data. At that point, the process of removing the data starts.

As NAISS will at least initially keep working as SNIC did, this policy remains in place.

If you plan to remove all your data from the project directory within 30 days of the project end date, you can stop reading now.

Why remove data at all? Why such a complicated process?

According to the SNIC storage policy (SNIC funds the NSC Centre Storage system), the project PI should save project data no later than 90 days after the project ends. Remaining data will then be deleted by the SNIC Centre.

The easiest way to implement this policy would be for NSC to automatically delete all remaining data after 90 days. However, we believe that would often lead to loss of valuable research data.

Instead, we have decided on the method that is described below. In short:

• Send repeated email reminders to the PI and project members telling them to save their data.

• Make the directory more and more inaccessible to force users of the data to notice that something is about to happen.

• Make a reasonable effort to get confirmation from the PI that the directory can be deleted.

• Finally: delete any remaining data.

Phase 1: “30 days of normal use + email warnings”

When a project ends, the project directory remains writable for 30 days.

This period allows running jobs to finish (jobs are allowed to start up until the time the project ends) and lets you do limited post-processing (e.g package files into zip or tar archives) and transfer data away from NSC.

Please plan to finish all post-processing and remove your data no later than 30 days after your project ends! Any time given beyond that is intended as an extra margin in case of illness, technical problems, etc.

During this period, NSC will send out automated emails weekly to the project PI and all members that have data stored in the project directory.

Please note that we cannot guarantee that you will receive those emails. Common problems are:

• The email address you have registered in SUPR is no longer valid, or you don’t read it very often.

• The automated emails are classified as spam and end up in your spam folder.

If you know that a project you are a member of has ended more than a week ago and you have not received these emails, look for them in your spam folder or contact support@nsc.liu.se. The emails are sent from support@nsc.liu.se and have the subject “Important information regarding /proj/… on NSC Centre Storage”.

If you don’t know what to do with your data after the project ends, talk to the project PI or contact NSC Support. In some cases Swestore can be used.

Phase 2: “another 30 days, the data is delete-only + email warnings”

From 30 to 60 days after the project ends, the data remains on disk, but you cannot write new data, only read and delete old data.

By making the data “delete-only”, any jobs that use the project directory for output will fail, and that user will notice something is about to happen.

During this period, NSC will continue to send out the automated email reminders.

Phase 3: “another 30 days, data unavailable but can still be restored”

From 60 to 90 days after the project ends, the data remains on disk, but is hidden from view.

The data is hidden to make sure any users or applications that use the data (e.g as a read-only data set) notices that the data is gone and acts before the data is actually deleted.

No automated email reminders are sent out during this period. Instead NSC will make a reasonable effort¹ to reach the project PI to verify that he/she is aware that the remaining data will be deleted and that the project has saved any important data.

If NSC cannot reach the PI with a reasonable effort, we reserve the right to delete the remaining data in the project directory 90 days after the project ends (or later) or to keep it on disk for up to a year, depending on the circumstances².

Phase 4: deletion

Once NSC gets an OK from the PI to delete the remaining data, the project directory will be deleted even if less than 90 days have passed since the project ended. Once this is done, the automated emails will stop.

If NSC cannot reach the PI we reserve the right to delete the data anyway, but we will not do so until at least 90 days after the project ended.

Once NSC deletes the data, there is no way to get it back. There are no backup copies on tape, etc.

If you cannot move the data in time

If you are a project member: talk to the project PI, or if it’s getting close to the 90-day limit, contact NSC directly.

If you are the project PI: talk to NSC (email support@nsc.liu.se), we will usually give projects a reasonable extension to allow you time to move your data. But there has to be a reason for it, and extensions are always limited in time.

Adopted storage

Sometimes a project will continue under another name (e.g when a 12-month compute project is replaced by another one). To avoid having to move all data and edit job scripts to use the new location, the project PI can decide to adopt the directory of a previous project. This is done in SUPR when applying for the new project by choosing the same directory name as the previous project used.

If a new project that uses the same directory name as an old one starts during the 90-day window described on this page, the directory is immediately and automatically taken over by the new project and restored to normal writable status with the quota of the new project, and no further email reminders are sent out.

If an active project takes over an old project directory name after the 90-day window, the project gets a new empty directory. Since NSC does not automatically delete the data after exactly 90 days (there is some manual work involved), there might still be a possibility to recover the data. If this happens to you (and you want your old data back), contact NSC Support immediately.

If you have applied for a new project

We sometimes get requests like “I have now applied for a new project, please don’t delete my old data” or “I will apply for a new project …”. Here is our standard reply:

• NSC normally processes Proposals for Small compute or storage projects within a few working days.

• For Medium compute or storage projects there is a deadline on the 15th of each month³. Proposals submitted before the deadline will be processed by a NAISS working group sometime during the last half of the month. If approved, the new project will start on the first of the coming month.

• Proposals for Large compute or storage projects are processed centrally by NAISS twice per year with start of allocations July 1st and January 1st.

computing time that becomes available when old projects end (usually on the last day of the month) to new projects. The deadline is normally the 15th of each month but can be earlier, e.g during December or during the summer (due to vacations). If you need to know the exact date, contact NSC Support.

Based on the above, you should be able to predict when your new project can be expected to start (if it is granted).

If the expected start time of your new project is less than 90 days after the end of your previous project, no special handling is needed, your existing project directory will be linked to your new project when the new project starts, and will then be made visible in the file system with the correct quota. In this situation, NSC will usually not grant an extension of the previous project.

If your project is denied you are welcome to contact NSC Support and ask for a short extension of your previous project if that is needed to allow you to move your data away from NSC.

If the processing of your Proposal is delayed you are welcome to contact NSC Support and ask for a short extension of your previous project so the data is protected from deletion until the Proposal is granted or denied.

Please note that you can avoid this problem altogether by applying for your next project well before the previous one ends. SUPR will send a reminder to the PI a month before the project ends, but as emails can easily get lost or forgotten about, we also recommend a calendar reminder

What happens when a user is removed from the project?

Data belonging to a user who is removed from a project stays in the project directory, but will no longer be accessible by that user. It is then up to the project PI what to do with the data (delete, keep, change owner, …). If the project PI cannot access or delete the data, it is possible to request help from NSC.

Data in a user’s home directory is not affected by project membership.