Is my data safe?

We consider the storage system to be very reliable. It is based on the same proven technology (IBM Spectrum Scale (formerly GPFS)) as the previous system (which we consider to have been very reliable and ran from 2007 to 2014 without major problems), but has been improved in several ways, e.g:

• It uses end-to-end check sums and version numbers to detect, locate and correct data corruption that would be undetectable in older systems.
• Rebuild times after a disk failure are much shorter than in older traditional RAID systems. This is due to using “declustered RAID”. This minimizes the window where the system is vulnerable to additional disk failures, and also minimizes the performance impact to users during the rebuild following a disk failure.

Data on the system is protected against multiple disk failures using “8+2 Reed-Solomon” or better (i.e two disks out of a group of 10 can fail without affecting access to data). Combined with the short rebuild times after a disk failure we consider the risk of losing data due to disk failures to be very low.

We also use “snapshots” to protect against you (or NSC) accidentally deleting files.

However, we do not protect you against all types of failures. Some events can lead to loss of data, e.g

• Data in /proj will automatically be deleted 90 days after the project ends. It is your responsibility to copy or move your data from NSC before this happens.
• Various types of physical “disasters”: flooding, fire, theft, vandalism, …
• Software bugs in the storage system
• NSC staff mistakes
• Intrusions
• Data being accidentally being deleted, and this not being discovered until much later (when the data no longer exists in a snapshot)

After considering the value of our user’s data (which often can be recreated by re-running compute jobs), the cost of making off-site backups (which could protect against most disasters and some software bugs, some mistakes and some intrusions) and the low risk of data loss due to the above risks, we have decided only to perform limited tape backups of home directories (weekly) and do no tape backups of project storage.

Put differently: for a fixed amount of money available for storage, we bought hard drives, not backup tapes.

If your data is very valuable or irreplaceable, we recommend that you keep copies outside NSC. If you cannot store that data at your home university, we can recommend Swestore.