Service Outages from Research Computing: Large File Storage (LFS)

We've now resolved the incident.

• the /lifesci filesystem is now back online and healthy
• it is now accessible via the usual means - NFS and SMB.
• we have added some additional space and now have 60TB available for data

Thanks for your patience.

The Team continue to work on the LS Shares issue. We are sorry for any inconvenience this may be causing.

The team are still working on the fs shares issue.

Please note that this issue is currently impacting the use of the Darca Freezer Alarm system.

We have had to shut down the lifesci network shares again, as some of the disks in the storage are offline. We don’t expect any data loss but the disks being offline will cause problems with reading data. The LFS, DIF and Compute cluster shares are unaffected but most of the SLS group shares are currently not accessible.

The Research Computing Team are continuing to work on the storage space issue as our highest priority.

Progress is being made - we will continue to provide regular updates.

Research Computing staff are continuing to work on the storage space issue as our highest priority.

We have cleared some space on all the filesystems - some more than others to give us a bit of breathing room, but the situation is still severe.

We are working with suppliers to bring some additional capacity online and we will update the community very early next week.

We have managed to clear some additional space on fs/lifesci which means we have been able to restore access to the research group file shares.

We are in the process of bringing extra capacity online. This is expected to take the team several days to accomplish. In the meantime, there is very little free space available (we only have 6TB left) and quotas are in place. Please let us know if you have an urgent need to store large amounts of data.

We are aware that this will continue to have impact on your day to day work. Please be assured that we are fully committed to resolving this as soon as possible.

The Research Computing team are treating this as our top priority and are working with our supplier to recover the service.  

We will keep you updated.

If you are affected by this issue but have not yet logged a ticket, you can do so by visiting the Self Service Portal and clicking on the Broad Service Disruption fs.lifesci Storage Issue, then clicking on "I am affected by this disruption"

Access to the lfs.lifesci fileshare is currently removed (as of 13.00 on Monday 22nd May 2023) while our engineers work on a resolution for this issue. This means that you will not be able to access your network share to fs.lifesci while this work is carried out.

This will include all instrument machines that mount a fs.lifesci share directly.  You will still be able to save files to OneDrive. 

We are not able to provide a timescale at the moment.  We anticipate that this issue will take at least a day to resolve.  We will issue regular updates to keep you informed. 

We are aware that this will have impact on your day to day work.  Please be assured that we are fully committed to resolving this as soon as possible. 

We're currently investigating the issue and will update this post with progress.

We've had reports of service outages with our LFS (Large File Storage) filesystem. The LFS filesystem is partially broken and some tape restores will not be accessible, and users will be unable to retrieve their data.

We're currently investigating the issue and will update this post with progress.