The email system (postfix smtp server and dovecot imap server) may be up and down several times over the next few days as we deal with another bout of compromised userids and spam. Please bear with us.
AFN will shut down briefly at 12:00noon Thursday, July 13. The University of Florida has to perform power maintenance this coming weekend, and AFN's severs need to move to a different power feed. Total downtime should be less than an hour.
We have restart the Postfix email server and the Dovecot IMAP mail delivery server. We believe that we have sufficient free disk space to resume normal email service on this server until we can resolve the configuration issues on the new server and migrate to it. We thank you for your patience, and hope that we can keep a better grip on the spam situation now that the massive buildup has been eradicated.
The existing AFN email server has more or less collapsed under its own weight. We had some gaps in our spam-proofing, and AFN's kind of fallen into a positive feedback loop resulting in tens of millions of unprocessed messages in the incoming queue. The system's run short of disk space to the point that we can't even run the queue mangement tools to *look at* the contents of the queue, much less do anything *useful* to the contents. We tried moving the incoming queue to new, larger, disk space yesterday, but the copy was taking far too long, and without having a way of even telling how many millions of messages were being moved, we couldn't be sure that the new space wouldn't get filled.
So, after discussing amongst ourselves, the systems administrators have concluded that the only viable approach to getting AFN's email server back up and processing email again is to purge the incoming queue. That purge job is running now, and is doing miraculous things to the disk space situation. Once the purge job finishes, we will make some tweaks to the spamproofing, and bring the mail server back up again. This will mean that users may lose a day or two of incoming email, but we should finally be out from under the self-perpatuating backlog that pretty much overwhelmed us.
We've hit some snags configuring the fabled new AFN server, so we're taking steps to keep the existing hardware cooking along. We've moved a bunch of home directories around in order to scare up more disk space for email processing. While we're working on this email access to AFN is shut off. We should be done and have mail access back on by late this afternoon. We apologize for the inconvenience, and thank you for your patience while we beat our heads against a brick wall around here!
We're drowing in email again. We're working on purging a bunch of dead emails, spams, bounce messages, etc., and have the Postfix mail server daemon shut down until we can get the queues purged. Please bear with us.
We are hoping to begin the long prophesied moved to newer server hardware in the next few weeks. When that happens, we'll have at least a couple of periods of several hours' system unavailability as we move userids, files, mail spools and such to the new server. Further updates once the dates are nailed down.
Would everyone please go through their email inboxes and purge any messages no longer needed? If you wish to retain messages, please create a "mail" directory in your home directory and move messages there. This will help reduce usage on the incoming mail directory, which is short on space.
We're currently purging several thousand spam emails generated against a compromised userid from the mail queue. To expedite the process, we have the mail system sending agent (Postfix) shut down. Once the purge jobs are finished running we will bring Postfix back up. At the rate things are going, we should be done by around 4:00pm.
We are evaluating the current state of the AFN email service, with an eye to getting us off the blacklists (e.g. Yahoo and Hotmail are currently rejecting afn.org email), enhancing protection against incoming spam, and improving the performance of the mail server. As we work with the server configuration there may be periodic interruptions in email access and delivery. We will work to minimize any such disruptions, and apologize for any inconvenience caused by this work.
We have disabled SSLv2 and SSLv3 on incoming and outgoing email and web services, as both protocols are vulnerable to various attacks. STARTTLS will still work.
New domain name ssh.afn.org has been added to replace ftp.afn.org, since the FTP protocol has been discontinued. Using ssh.afn.org is preferred to explicitly naming freenet1.afn.org or other freenet machine, as it will be kept current regardless of which machine is hosting the SSH service.
Apologies for the server move running over the end of the maintenance window. When we moved the server farm to its new rack, we discovered that the power in the rack was 208V instead of 120V. We also discovered that our Promise Ultratrak RM8000 RAID array did NOT have autoswitching power supplies. There were loud bangs, flashes of light, and small clouds of acrid smoke.
Fortunately, the University of Florida's Astronomy department had a surplus Ultratrak RM8000, which they were very gracious to loan us (thanks, Matt!). After moving the drives to the new array housing and TRIPLE checking that the power supplies were set to 230V, we were able to bring the system back up at about 4:20pm.
The AFN DNS nameserver will move to a different machine in the AFN cluster. The change should not be service affecting.
The AFN server farm will be relocated to a different rack in the UF SSRB Data Center. All AFN services will be unavailable during the server moves. We do not anticipate the moves taking more than an hour, but maintenance window is scheduled for two hours in case we run into any snags. We have to vacate our current rack and reduce our in-room footprint to make room for new Data Center equipment.
Due to spammers finding and sending to firstname.lastname@example.org,
the address has been changed. Send requests for assistance to "request" at afn.org.
Most systems are working now. Unfortunately, the current system as rebuilt will require
everyone to have their account password reset. It can be reset to the previously used password,
but the new system uses a different password encryption method, making the updates necessary.
HOWEVER, SOME USERS HAVE REPORTED SUCCESSFUL CONNECTIONS WITHOUT HAVING TO UPDATE THEIR PASSWORD.
Please try to connect as usual before requesting a password update.
The authentication process after log on may take up to 10 seconds. To have your password reset,
send an email message (or have a friend send for you) to
email@example.com (see news above for
April 21st) with your name and a phone number where you can be reached; someone will text or call
that number within a few days to coordinate the password reset. Let us know in your email
if you can't receive text messages.
Several changes have been made to enhance security and performance. Telnet and FTP are not available, as those older services pass usernames and passwords in clear text, unencrypted. Instead, SSH replaces telnet and SFTP/SCP replaces FTP. There are several applications available for this, such as PuTTY for SSH, with brief instructions, and WinSCP for SCP for Windows; Mac Terminal utility with SSH and Fugu for Mac OS X. Note that fugu-1.2.1pre1 in the Unstable folder is the only option now available for Mavericks. Use ssh.afn.org as the domain name to connect to either service.
IMAP for email uses TLS security, and clients must be configured with TLS for connecting to imap.afn.org for receiving and smtp.afn.org for sending. POP email protocol is no longer available.
We're currently working on getting authentication working again for email. The IMAP server will come and go through the weekend until we get the server and the LDAP directory happy with each other. We're not there yet, but we're getting closer!
Good news! Looks like we've got the web server back! Users should be able to see web pages originating from their public_html directories. Email is still not there, but we're getting closer. Stay tuned!
New servers have been installed and configuration is ongoing. As soon as we can retrieve some configuration files from one of the dead nodes, we ought to have email and web services restored, hopefully within the next day or two.
Once everything is back up and stable, we'll let things be for a few weeks, then schedule a maintenance window to migrate user services and data off the interim server onto the permanent replacement server.
AFN has suffered a series of hardware hassles over the last several days, culminating in several tired servers more or less simultaneously giving up the ghost. This has lent a certain previously lacking urgency to our long standing plans to migrate off off the hodge-podge of elderly scrounged and donated servers.
AFN sysadmins are now engaged in first phase of a multi-step process to move to newer hardware and more recent versions of server software. The interactive, email/fileserver/web, and DNS services are being moved this weekend. Assuming the installation and configruation tasks go smoothly, and the interactive node's user and password files can be recovered relatively painlessly from the dead drives, we should have interactive, email and web services back up by the first part of the week of March 10th.
The next step in the process will be to move the users' home directories onto a much newer, more capable, more reliable fileserver. Once the home directories have migrated, the last phase of the project, migrating services onto the big server will take place. At that juncture, AFN should be on a much better hardware and software footing going forward.
We do apologize for the disruption in service caused by the raft of system issues and their mitigation. We expect the situation to be much improved after we reach the end of the migration project.