Support experiencing degraded performance

Billing Panel

100.0% uptime
Jun 2023 · 100.0%Jul · 100.0%Aug · 100.0%
Jun 2023100.0% uptime
Jul 2023100.0% uptime
Aug 2023100.0% uptime
DirectAdmin Panel
100.0% uptime
Jun 2023 · 100.0%Jul · 100.0%Aug · 100.0%
Jun 2023100.0% uptime
Jul 2023100.0% uptime
Aug 2023100.0% uptime

IMAP

100.0% uptime
Jun 2023 · 100.0%Jul · 100.0%Aug · 100.0%
Jun 2023100.0% uptime
Jul 2023100.0% uptime
Aug 2023100.0% uptime

SMTP

100.0% uptime
Jun 2023 · 100.0%Jul · 100.0%Aug · 100.0%
Jun 2023100.0% uptime
Jul 2023100.0% uptime
Aug 2023100.0% uptime
Webmail
100.0% uptime
Jun 2023 · 100.0%Jul · 100.0%Aug · 100.0%
Jun 2023100.0% uptime
Jul 2023100.0% uptime
Aug 2023100.0% uptime

Support

100.0% uptime
Jun 2023 · 100.0%Jul · 100.0%Aug · 100.0%
Jun 2023100.0% uptime
Jul 2023100.0% uptime
Aug 2023100.0% uptime

Notice history

Aug 2023

aus.mxroute.com server
  • Resolved
    Resolved

    Hardware migration complete. Crossbox will be reinstalled soon, but everything else is working. Remember that Crossbox is only one webmail option, others are at https://aus.mxroute.com:2096

  • Identified
    Update

    Hardware migration will most likely take place within the next 4 hours. Users may see a few rejected connections during the transition. This should clear up on it's own in a very short amount of time.

  • Identified
    Update

    We'll be pulling the trigger on hardware migration for the Aus server tonight. No downtime, unlikely customers will notice anything other than a brief hiccup. The hiccup in question is expected to be less troubling than your cell phone switching towers with a rough handshake.

  • Identified
    Update

    While the aus server has been online since our last update, we're not confident in it's ability to stay online. We are migrating the system to new hardware right now. Steps have been taken to ensure data integrity in the event of a hardware failure, should it occur prior to completion of our migration.

  • Identified
    Update

    The Aus server is online. This shouldn't be celebrated yet. Here's to copying data as fast as physically possible before a degraded RAID1 array which failed to boot multiple times says it's had enough.

  • Identified
    Update

    The RAID controller is still reporting a fault, but we have more eyes on the issue on site than we've had in the previous hours.

  • Identified
    Update

    We are continuing to work on a fix for this incident.

  • Identified
    Update

    We are continuing to work on a fix for this incident.

  • Identified
    Update

    We are continuing to work on a fix for this incident.

  • Identified
    Update

    We are continuing to work on a fix for this incident. The RAID controller has failed, and we're waiting for on-site staff to further investigate possible solutions with available hardware.

  • Identified
    Update

    We are continuing to work on a fix for this incident.

  • Identified
    Identified

    There appears to be a hardware issue with the aus.mxroute.com server. We are working with on-site staff to determine the scope and best path to resolution.

  • Investigating
    Investigating

    We are currently investigating this incident.

Path.net just shit the bed
  • Resolved
    Resolved

    This incident has been resolved.

  • Identified
    Identified

    We are continuing to work on a fix for this incident. Some updates here: https://status.path.net/incidents/50t7b6ts412f

  • Investigating
    Investigating

    We are currently investigating this incident. It looks like path.net just took down a good portion of the internet.

Acadia Server
  • Resolved
    Resolved

    This incident has been resolved.

  • Identified
    Identified

    Intermittent packet loss over some routes may indicate unmitigated DDOS attack, continuing to investigate.

  • Investigating
    Investigating

    We are currently investigating this incident.

Longhorn Server
  • Resolved
    Resolved

    This incident has been resolved.

  • Investigating
    Investigating

    We are currently investigating this incident. No email will be lost due to their event.

Jul 2023

Spam filters on Arrow server
  • Resolved
    Resolved

    This incident has been resolved.

  • Identified
    Identified

    Spam filters on the arrow.mxrouting.net server are not performing as desired after upgrading to the new system. More spam is coming in than is desirable. We are working on this.

Arrow Spam Filtering
  • Completed
    July 17, 2023 at 7:05 PM
    Completed
    July 17, 2023 at 7:05 PM

    Spam filtering on the Arrow server should be better and improved. An email will be sent to users on the Arrow server explaining further.

  • In progress
    July 17, 2023 at 6:48 PM
    In progress
    July 17, 2023 at 6:48 PM

    Spam filtering might be a bit "wonky" for a while as we perfect these improvements, but the worst result should be receiving an email that you prefer not to accept. Please do not open support tickets about spam received that shouldn't have been until this maintenance is closed.

  • Planned
    July 17, 2023 at 6:24 PM
    Planned
    July 17, 2023 at 6:24 PM

    We are expecting a temporary increase in inbound spam for users on the arrow.mxrouting.net server while we work on improvements to it's spam filtering.

Witcher Server
  • Resolved
    Resolved

    This incident has been resolved.

  • Investigating
    Investigating

    We are currently investigating an issue with the witcher.mxrouting.net server.

Jun 2023

Outbound Relays
  • Resolved
    Update

    This morning one of our outbound relay servers ran out of memory. This caused some outbound email to be stuck in queue for longer than intended. While we keep a hot spare ready for this particular server at all times, it was not engaged as the event did not trigger the right type of alert. We've fixed this and all of the delayed outbound mail is processing, likely done by the time this update is posted.

    Measures will be taken to ensure first that this does not happen again, and second that the length of the event is significantly reduced if we fail to prevent it from happening again.

  • Resolved
    Resolved

    This incident has been resolved.

  • Investigating
    Investigating

    We are currently investigating a possible issue with outbound relays.

Tuesday Server Maintenance
  • Completed
    June 09, 2023 at 1:27 AM
    Completed
    June 09, 2023 at 1:27 AM

    Fixing Crossbox and then done.

  • Planned
    June 09, 2023 at 1:09 AM
    Planned
    June 09, 2023 at 1:09 AM

    We are performing emergency maintenance on the tuesday.mxrouting.net server in order to prevent extended downtime tomorrow. Our datacenter provider for that server (the only server we have with that provider) has informed us that they are moving physical locations tomorrow, and that they expect a 3-4 hour outage. We're not doing that. We're migrating the server into our "home" datacenter in Virginia immediately.

    During this maintenance you can expect brief hiccups that appear to resolve themselves very quickly. We've gotten very good at migrating these servers in production, and being one of our smallest (in disk usage) servers, this should go over very smoothly.

    We will:

    • Disable all services on the server
    • Run one final rsync of data
    • Switch DNS
    • Bring services back online

    This will result in no loss of data, and only the most brief of outages. Typically an outage of this length goes unnoticed by our customers, this status page is merely a formality.

Banshee Server Issue
  • Resolved
    Resolved

    This incident has been resolved. Postmortem and email to customers will be a bit delayed while the details are worked out.

  • Identified
    Update

    We're still performing an rsync of the data, and it's intended that the next step takes much less time than this one. When this is over we'll detail in a postmortem report what went wrong, how we'll prevent it from happening to such a degree in the future, and we'll talk about compensation (the latter will be in an email to the relevant customers).

  • Identified
    Update

    The rsync of data to the new server is still ongoing. It's roughly 3/4 of the way done.

  • Identified
    Update

    The fastest path to resolution right now is for us to migrate the Banshee server to another system. This is unfortunately going to take a significant number of hours, and any attempt to tell you how many hours that would be would have to be a lie, it simply can't be known right now. It'll take no less than the time it takes, and we'll make every effort to make that number as small as possible.

    The Banshee server is a legacy cPanel server, and this wasn't how we wanted to retire it's hardware. There is in fact no hardware issue, but for some reason the OS is hosed beyond reasonable repair. All data is intact, and all inbound emails should be held and received after this is complete.

  • Identified
    Identified

    Now it just keeps booting to BIOS config and it won't do anything else. BIOS config is strangely lacking in the proper options that it should contain, options which might help to overcome this. So we're sending datacenter techs back out to see just what in the hell. Everything should be fine soon, no reason to suspect that this will be a long term outage.

  • Investigating
    Update

    The server goes offline almost immediately after coming back online. This server lacks an IPMI, so we are having a KVM attached. With that said, while booted into rescue mode all disks appear fine, this does not appear to be a hardware issue. We just can't get any strong insight into it without a KVM.

  • Investigating
    Investigating

    Problem reappeared. We are currently investigating this incident.

  • Resolved
    Resolved

    This incident has been resolved.

Jun 2023 to Aug 2023