MXroute - Packet loss to accounts.mxroute.com – Incident details

All systems operational

Packet loss to accounts.mxroute.com

Resolved
Major outage
Started 2 months agoLasted about 13 hours

Affected

Billing Panel

Operational from 1:13 AM to 8:34 AM, Major outage from 8:34 AM to 2:29 PM

Support

Operational from 1:13 AM to 8:34 AM, Major outage from 8:34 AM to 2:29 PM

Updates
  • Resolved
    Resolved
    This incident has been resolved.
  • Update
    Update

    We want to stress that the only outage here is of our billing and support portal. All customer services are online. However, we do consider our billing and support portal to be quite important, even if (and it is) secondary to actual customer mail servers. With that said:

    We've been granted a license by our billing software vendor that will allow us to run a failover setup moving forward. However, we still need to transfer the data from the location behind the cut fiber. We still believe at this hour that the lesser evil is waiting for that repair crew to splice the fiber, and that the worse choice is to restore a 24 hour old backup and then rebuild the last 24 hours of changes in the database from logs that we're not prepared to automate against in any shorter amount of time than we could rebuild it by manual data entry.

    This was a perfect alignment of variables made worse by exponential growth of our business and a failure on our part to adjust the backup strategy to compensate for that growth. When that backup strategy was introduced, rebuilding 24 hours of the database by manual data entry wasn't such a large job.

  • Update
    Update

    This is a great time to migrate our billing/support portal. If it's effectively down, we can migrate it by the time it would be back up. That's what we're doing right now.

  • Identified
    Identified

    A fiber cut is impacting network connectivity in Spokane, WA where accounts.mxroute.com is currently hosted. Continuing to monitor.

  • Investigating
    Investigating

    We are seeing high packet loss to accounts.mxroute.com. This appears to be a network issue a layer above us. We are monitoring the situation. This does not impact services, it does impact billing and support.