AFN has been experiencing problems with filesystems filling up, causing email and web access failures. AFN admins are working on cleaning up the offending filesystems, and hope to restart mail service by the end the day.
The RAID array has been rebuilt and data restored from the last known-good backup. The email spool has been cleaned up and all user services and data should be available once again.
AFN management is working on obtaining a new server and new disk space, which should provide a stable hardware platform with sufficient capaicty for the forseeable future. When the new hardware arrives there will be a couple of periods of downtime to allow migration of data and services off of the current collection of servers.
As of Tuesday afternoon, the RAID array has been reinitialized with the four remaining functional drives. The home filesystems /afn/homes0 and /afn/homes1 have been restored from the last usable backup, which dates from April 2008. The mail spool has also been restored from the same backup. /The /afn/homes2 filesystem is currently being restored; as it was the largest filesystem, it will take some time for the restore to complete.
On Friday evening, March 27, the RAID5 array which houses the email spool and most of the user data suffered a series of failures which resulted in the loss of two drives from the array. All email in inboxes and most user data was lost. AFN admins are attempting to rebuild the RAID array and will attempt to restore as much user data as can be recovered. As of Monday afternoon, March 30, the RAID array is reinitializing on its surviving drives; this process will take several hours.
Until further notice, email and user data remain inaccessible. We apologize for the inconvenience and are making every effort to restore AFN to functionality, but it is expected that this will take some time and result in some user data being lost.