MXroute - Notice history

We want to stress that the only outage here is of our billing and support portal. All customer services are online. However, we do consider our billing and support portal to be quite important, even if (and it is) secondary to actual customer mail servers. With that said:

We've been granted a license by our billing software vendor that will allow us to run a failover setup moving forward. However, we still need to transfer the data from the location behind the cut fiber. We still believe at this hour that the lesser evil is waiting for that repair crew to splice the fiber, and that the worse choice is to restore a 24 hour old backup and then rebuild the last 24 hours of changes in the database from logs that we're not prepared to automate against in any shorter amount of time than we could rebuild it by manual data entry.

This was a perfect alignment of variables made worse by exponential growth of our business and a failure on our part to adjust the backup strategy to compensate for that growth. When that backup strategy was introduced, rebuilding 24 hours of the database by manual data entry wasn't such a large job.

This is a great time to migrate our billing/support portal. If it's effectively down, we can migrate it by the time it would be back up. That's what we're doing right now.

A fiber cut is impacting network connectivity in Spokane, WA where accounts.mxroute.com is currently hosted. Continuing to monitor.

We were seeing webmail outages up to 5 minutes at a time on the Safari server yesterday, increasing in frequency throughout the day. The decision was made to move the Apache includes to the SSD array currently housing MySQL, to speed up config reloads. For us that means moving the DA user data. This is proving to take longer than predicted, there's a strange blocker to determining progress other than "done" or "not done."

Giving an ETA for DirectAdmin to be back online would be irresponsible. Best course of action is see this through. No email access or delivery is harmed by this. This is only administration control panel access.

From Jeff. The man, myth, legend:

"Arelion/Telia is having an outage caused by an equipment failure in Atlanta that is impacting East/West traffic in the US. Arelion plans to replace the equipment shortly (they said at about midnight Eastern time). We've turned off Arelion in both Ashburn and LA temporarily until the issue is resolved."