MXroute - Notice history

This incident has been resolved.

Physical presence in route to handle on-site issue.

The fix requires collaboration with a third party, and that is taking some time. SMTP is fully functioning on this server in the meantime, and everything else is a bit more questionable. Inbound email is being held until the end of the event.

This incident has been resolved.

Considering this resolved. We're forcing queues to push out emails that were delayed.

While continuing to work on this, we're changing the status to reflect that all outbound email is currently delayed for just a few more minutes.

Bringing our spare up to speed to replace the server experiencing packet loss. Only some emails are being held back from outbound delivery right now, and those will clear in a few moments when we get this settled.

This incident has been resolved.

It's just a simple network issue introduced by an "apt upgrade" but the IPMI is making the process far more difficult than it needs to be. It's going to be just fine, just give me a minute.

Lucy is back online. A full account of events should follow a bit later.

There's an admin dream team working on this box right now with us.

We are continuing to work on a fix for this incident.

With activity split between getting the production server to boot it's OS, and deploying a replacement server, suddenly JetBackup doesn't work on a new server. So we opened a ticket with JetBackup. Whatever supernatural entity is responsible for this incredible sequence of events, even you have to sleep eventually.

We're still working on the OS. During this process, we've been preparing a replacement server to restore backups to, as we'd like that process to be moving along just in case we need it. If it's time wasted, and we sure hope it is, it was a worthy cause.

By all accounts we should either be online or right on the edge of being online. But the OS is still giving us hell. You can vent in tickets if you want, and we get it, but you won't speed this up. If we had a magic button we'd have pressed it already.

Hardware replaced, now it's just a matter of getting the OS to deal with the hardware changes and boot properly.

Still working on this. Note there is a reason it says 'Operational' above and it's too long to explain here, but no service on the lucy.mxrouting.net server is online until the text of these updates says otherwise.

RAID controller replacement didn't help. Was it replaced wrong? Is it the riser card? Stay tuned to find out.

We are continuing to work with remote hands on the raid controller replacement.

Frustration is understood. We're waiting to hear back from remote hands at the datacenter. Here are some facts:

1. No inbound email will be missed, it will flow in after the server is back online.
2. Many reasonable redundancies were in place here. It is not common for raid controllers to suddenly fail.
3. This server is actively backed up, no matter what else happens.

We expect to pop in the new controller, import the raid, and work fine under the expectation that this hardware configuration is perfect. This being a statistical anomaly alone, we don't expect a repeat event.

The server is not detecting the RAID controller. We're going to swap out the controller with the spare.

The Lucy server is having trouble booting, so that's interesting. We're working on it.

This incident has been resolved.

Routing issue in Ashburn may be causing some issues. Understanding scope and reviewing mitigation options.

There was an issue with an upstream network provider. This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

There is currently a network issue at the datacenter, working to resolve.

This incident has been resolved.

There appears to be no available on-site staff to resolve the current issue with the longhorn.mxrouting.net server. The server may be offline for some time until we are able to get someone on-site to resolve the issue. Migration of this server to a new facility, to prevent this issue, will begin a few hours after that resolution.

We are continuing to work on a fix for this incident.

This incident has been resolved.

Intermittent packet loss over some routes may indicate unmitigated DDOS attack, continuing to investigate.

This incident has been resolved.