MXroute - Notice history

Maintenance is considered complete. Any remaining issues should first be assumed to be client side DNS issues, only otherwise when ruled out.

This incident has been resolved.

We are considering this "resolved." Everything is working but Crossbox, which we do not consider necessary to call the server "working" in production. We're still monitoring for any unforeseen issues.

We're seeing more users with functional service than users without, and we expect this entire thing to be over in less than an hour, increasingly so as each minute passes (as the rsync finalizes each user's home directory one by one, in alphabetical order). Crossbox (mail.mxlogin.com) will indeed not be functional at the time that we declare this "resolved."

Things are moving very smoothly at this stage. We see users sending out mail from the server without issue, so at least that is working right now with no hiccups. Inbound mail is still being queued, to be delivered when user data is copied. Crossbox (mail.mxlogin.com) will likely not work for some time after we consider this "resolved" for reasons not particularly interesting to anyone, but it will be a work in progress.

DNS has been moved to the new server while the migration is underway. This does not indicate everything is working, but we should begin accepting emails into our queue and delivering them to users soon. Outbound mail should start working almost immediately, and "some" users should find their accounts usable to some degree or another. It shouldn't be considered odd to find anything/everything not working until the event is declared resolved.

Most of the required parts are in place, but we need more user data to sync before we can swap DNS over.

Still waiting on DirectAdmin to build all of it's services, as it seems to do this in the background after it claims installation is done now (strange?). In the meantime, the bulk of the data is being copied over so that we can hit the ground running, or as close to that as possible.

New OS is ready to begin the process, we'll see how long it takes to finalize. The RAID is doing a sync right now so that might slow things down a bit.

Hardware ready, OS installation in progress.

The current server is ready for migration to the new system, the new system is a work in progress.

Right now the work is split between two tasks. First is getting the new server online and ready to prep the OS for migration. Second is preparing the current production server for data transfer (booting into ISO, enabling networking, mounting disk).

The issue with the operating system is beyond reasonable repair and seems to relate to the fact that we migrated an OS out of OVH's SoYouStart brand, which appears to have a unique image for Ubuntu 20. We are preparing a new server next to this one, which we will prep for deployment and begin migrating everyone's data. Because the server is close in proximity, we expect the data migration to move fairly quickly. We could be wrong on that.

The operating system on the wednesday.mxrouting.net server is pretty well hosed, and we're working on a plan to get it back online. This isn't going to be a quick process.

The server did not take well to being rebooted. We're looking at a kernel panic on boot right now. Best case scenario it's a bad kernel upgrade. Worse case scenario, the file system is hosed. Current estimate for repair is anywhere from 3 minutes to 3 days, so probably best not to draw any conclusions yet.

This incident has been resolved.

Maintenance complete. Users may see oddities for a few minutes, nothing more.

Maintenance complete. If you still see any problems, they should clear up quickly.

Blizzard server is back online. Any emails you didn't receive during the outage should begin pouring in shortly (from now up to a few hours from now). The next step is to migrate this server to new hardware to never experience this again, as this was the result of a poor hardware configuration on our part. More on that later, for now we're online.

The previous status remains the current status, though 30-45 minutes may have been slightly under the target.

Still going well. If everything goes as planned, this is the home stretch before we bring it back up. ETA 30-45 minutes? It's a fairly rough guess. If it's dramatically wrong, that could change to anywhere from 1 hour to 12 hours, that's how crucial this step is in determining the result, and the result can't be known until it's finished.

Still working on this box. All signs point to positive results, but the time it takes is one of those "it takes what it takes and anything less than what it takes is complete failure" kind of things so moving as quick as possible.

So far things are looking good, hoping to get it back online soon. This is a real time update, things can change, but we'd like you to be as optimistic as we are at this moment.

This is a hardware failure event. Working to provide a quick resolution so that we can follow it up with a longer one. The primary goal is to get everything back online in as little time as possible. Whether or not we can do that in a timely manner is, frankly, not yet known.

Resolved.