All systems operational

Billing Panel

100 % uptime
Jul 2023100% uptime
Aug 2023100% uptime
Sep 2023100% uptime
DirectAdmin Panel
100 % uptime
Jul 2023100% uptime
Aug 2023100% uptime
Sep 2023100% uptime

IMAP

100 % uptime
Jul 2023100% uptime
Aug 2023100% uptime
Sep 2023100% uptime

SMTP

100 % uptime
Jul 2023100% uptime
Aug 2023100% uptime
Sep 2023100% uptime
Webmail
100 % uptime
Jul 2023100% uptime
Aug 2023100% uptime
Sep 2023100% uptime

Support

100 % uptime
Jul 2023100% uptime
Aug 2023100% uptime
Sep 2023100% uptime

Notice history

Sep 2023

Lucy server
  • Resolved

    Lucy is back online. A full account of events should follow a bit later.

  • Identified

    There's an admin dream team working on this box right now with us.

  • Identified

    We are continuing to work on a fix for this incident.

  • Identified

    With activity split between getting the production server to boot it's OS, and deploying a replacement server, suddenly JetBackup doesn't work on a new server. So we opened a ticket with JetBackup. Whatever supernatural entity is responsible for this incredible sequence of events, even you have to sleep eventually.

  • Identified

    We're still working on the OS. During this process, we've been preparing a replacement server to restore backups to, as we'd like that process to be moving along just in case we need it. If it's time wasted, and we sure hope it is, it was a worthy cause.

  • Identified

    By all accounts we should either be online or right on the edge of being online. But the OS is still giving us hell. You can vent in tickets if you want, and we get it, but you won't speed this up. If we had a magic button we'd have pressed it already.

  • Identified

    Hardware replaced, now it's just a matter of getting the OS to deal with the hardware changes and boot properly.

  • Identified

    Still working on this. Note there is a reason it says 'Operational' above and it's too long to explain here, but no service on the lucy.mxrouting.net server is online until the text of these updates says otherwise.

  • Identified

    RAID controller replacement didn't help. Was it replaced wrong? Is it the riser card? Stay tuned to find out.

  • Identified

    We are continuing to work with remote hands on the raid controller replacement.

  • Identified

    Frustration is understood. We're waiting to hear back from remote hands at the datacenter. Here are some facts:

    1. No inbound email will be missed, it will flow in after the server is back online.
    2. Many reasonable redundancies were in place here. It is not common for raid controllers to suddenly fail.
    3. This server is actively backed up, no matter what else happens.

    We expect to pop in the new controller, import the raid, and work fine under the expectation that this hardware configuration is perfect. This being a statistical anomaly alone, we don't expect a repeat event.

  • Identified

    The server is not detecting the RAID controller. We're going to swap out the controller with the spare.

  • Identified

    The Lucy server is having trouble booting, so that's interesting. We're working on it.

  • Investigating

    We are currently investigating this incident.

Possible routing issues
  • Resolved

    This incident has been resolved.

  • Investigating

    Routing issue in Ashburn may be causing some issues. Understanding scope and reviewing mitigation options.

  • Resolved

    There was an issue with an upstream network provider. This incident has been resolved.

  • Investigating

    Some users may be seeing routing issues to some of our servers. While we have not been able to replicate it ourselves, we are seeing reports of outages which suggests a potential upstream issue that may be more significant to a wider portion of the internet than just us.

aus.mxroute.com
  • Resolved

    This incident has been resolved.

  • Investigating

    We are currently investigating this incident.

aus.mxroute.com server
  • Resolved

    This incident has been resolved.

  • Identified

    There is currently a network issue at the datacenter, working to resolve.

  • Investigating

    We are currently investigating this incident.

Longhorn Server
  • Resolved

    This incident has been resolved.

  • Identified

    There appears to be no available on-site staff to resolve the current issue with the longhorn.mxrouting.net server. The server may be offline for some time until we are able to get someone on-site to resolve the issue. Migration of this server to a new facility, to prevent this issue, will begin a few hours after that resolution.

  • Identified

    We are continuing to work on a fix for this incident.

  • Investigating

    We are currently investigating this incident.

shadow.mxrouting.net
  • Resolved

    This incident has been resolved.

  • Identified

    There is an issue with the network switch that shadow.mxrouting.net is connected to. Resolution in progress.

Aug 2023

aus.mxroute.com server
  • Resolved

    Hardware migration complete. Crossbox will be reinstalled soon, but everything else is working. Remember that Crossbox is only one webmail option, others are at https://aus.mxroute.com:2096

  • Identified

    Hardware migration will most likely take place within the next 4 hours. Users may see a few rejected connections during the transition. This should clear up on it's own in a very short amount of time.

  • Identified

    We'll be pulling the trigger on hardware migration for the Aus server tonight. No downtime, unlikely customers will notice anything other than a brief hiccup. The hiccup in question is expected to be less troubling than your cell phone switching towers with a rough handshake.

  • Identified

    While the aus server has been online since our last update, we're not confident in it's ability to stay online. We are migrating the system to new hardware right now. Steps have been taken to ensure data integrity in the event of a hardware failure, should it occur prior to completion of our migration.

  • Identified

    The Aus server is online. This shouldn't be celebrated yet. Here's to copying data as fast as physically possible before a degraded RAID1 array which failed to boot multiple times says it's had enough.

  • Identified

    The RAID controller is still reporting a fault, but we have more eyes on the issue on site than we've had in the previous hours.

  • Identified

    We are continuing to work on a fix for this incident.

  • Identified

    We are continuing to work on a fix for this incident.

  • Identified

    We are continuing to work on a fix for this incident.

  • Identified

    We are continuing to work on a fix for this incident. The RAID controller has failed, and we're waiting for on-site staff to further investigate possible solutions with available hardware.

  • Identified

    We are continuing to work on a fix for this incident.

  • Identified

    There appears to be a hardware issue with the aus.mxroute.com server. We are working with on-site staff to determine the scope and best path to resolution.

  • Investigating

    We are currently investigating this incident.

Path.net just shit the bed
Acadia Server
  • Resolved

    This incident has been resolved.

  • Identified

    Intermittent packet loss over some routes may indicate unmitigated DDOS attack, continuing to investigate.

  • Investigating

    We are currently investigating this incident.

Longhorn Server
  • Resolved

    This incident has been resolved.

  • Investigating

    We are currently investigating this incident. No email will be lost due to their event.

Jul 2023

Spam filters on Arrow server
  • Resolved

    This incident has been resolved.

  • Identified

    Spam filters on the arrow.mxrouting.net server are not performing as desired after upgrading to the new system. More spam is coming in than is desirable. We are working on this.

Arrow Spam Filtering
  • Completed
    July 17, 2023 at 7:05 PM

    Spam filtering on the Arrow server should be better and improved. An email will be sent to users on the Arrow server explaining further.

  • In progress
    July 17, 2023 at 6:48 PM

    Spam filtering might be a bit "wonky" for a while as we perfect these improvements, but the worst result should be receiving an email that you prefer not to accept. Please do not open support tickets about spam received that shouldn't have been until this maintenance is closed.

  • Planned
    July 17, 2023 at 6:24 PM

    We are expecting a temporary increase in inbound spam for users on the arrow.mxrouting.net server while we work on improvements to it's spam filtering.

Witcher Server
  • Resolved

    This incident has been resolved.

  • Investigating

    We are currently investigating an issue with the witcher.mxrouting.net server.

Jul 2023 to Sep 2023