We want to stress that the only outage here is of our billing and support portal. All customer services are online. However, we do consider our billing and support portal to be quite important, even if (and it is) secondary to actual customer mail servers. With that said:
We've been granted a license by our billing software vendor that will allow us to run a failover setup moving forward. However, we still need to transfer the data from the location behind the cut fiber. We still believe at this hour that the lesser evil is waiting for that repair crew to splice the fiber, and that the worse choice is to restore a 24 hour old backup and then rebuild the last 24 hours of changes in the database from logs that we're not prepared to automate against in any shorter amount of time than we could rebuild it by manual data entry.
This was a perfect alignment of variables made worse by exponential growth of our business and a failure on our part to adjust the backup strategy to compensate for that growth. When that backup strategy was introduced, rebuilding 24 hours of the database by manual data entry wasn't such a large job.
