Support experiencing degraded performance

Billing Panel

100.0% uptime
Aug 2023 · 100.0%Sep · 100.0%Oct · 100.0%
Aug 2023100.0% uptime
Sep 2023100.0% uptime
Oct 2023100.0% uptime
DirectAdmin Panel
100.0% uptime
Aug 2023 · 100.0%Sep · 100.0%Oct · 100.0%
Aug 2023100.0% uptime
Sep 2023100.0% uptime
Oct 2023100.0% uptime

IMAP

100.0% uptime
Aug 2023 · 100.0%Sep · 100.0%Oct · 100.0%
Aug 2023100.0% uptime
Sep 2023100.0% uptime
Oct 2023100.0% uptime

SMTP

99.98% uptime
Aug 2023 · 100.0%Sep · 100.0%Oct · 99.93%
Aug 2023100.0% uptime
Sep 2023100.0% uptime
Oct 202399.93% uptime
Webmail
100.0% uptime
Aug 2023 · 100.0%Sep · 100.0%Oct · 100.0%
Aug 2023100.0% uptime
Sep 2023100.0% uptime
Oct 2023100.0% uptime

Support

100.0% uptime
Aug 2023 · 100.0%Sep · 100.0%Oct · 100.0%
Aug 2023100.0% uptime
Sep 2023100.0% uptime
Oct 2023100.0% uptime

Notice history

Oct 2023

Sep 2023

Lucy server
  • Resolved
    Resolved

    Lucy is back online. A full account of events should follow a bit later.

  • Identified
    Update

    There's an admin dream team working on this box right now with us.

  • Identified
    Update

    We are continuing to work on a fix for this incident.

  • Identified
    Update

    With activity split between getting the production server to boot it's OS, and deploying a replacement server, suddenly JetBackup doesn't work on a new server. So we opened a ticket with JetBackup. Whatever supernatural entity is responsible for this incredible sequence of events, even you have to sleep eventually.

  • Identified
    Update

    We're still working on the OS. During this process, we've been preparing a replacement server to restore backups to, as we'd like that process to be moving along just in case we need it. If it's time wasted, and we sure hope it is, it was a worthy cause.

  • Identified
    Update

    By all accounts we should either be online or right on the edge of being online. But the OS is still giving us hell. You can vent in tickets if you want, and we get it, but you won't speed this up. If we had a magic button we'd have pressed it already.

  • Identified
    Update

    Hardware replaced, now it's just a matter of getting the OS to deal with the hardware changes and boot properly.

  • Identified
    Update

    Still working on this. Note there is a reason it says 'Operational' above and it's too long to explain here, but no service on the lucy.mxrouting.net server is online until the text of these updates says otherwise.

  • Identified
    Update

    RAID controller replacement didn't help. Was it replaced wrong? Is it the riser card? Stay tuned to find out.

  • Identified
    Update

    We are continuing to work with remote hands on the raid controller replacement.

  • Identified
    Update

    Frustration is understood. We're waiting to hear back from remote hands at the datacenter. Here are some facts:

    1. No inbound email will be missed, it will flow in after the server is back online.
    2. Many reasonable redundancies were in place here. It is not common for raid controllers to suddenly fail.
    3. This server is actively backed up, no matter what else happens.

    We expect to pop in the new controller, import the raid, and work fine under the expectation that this hardware configuration is perfect. This being a statistical anomaly alone, we don't expect a repeat event.

  • Identified
    Update

    The server is not detecting the RAID controller. We're going to swap out the controller with the spare.

  • Identified
    Identified

    The Lucy server is having trouble booting, so that's interesting. We're working on it.

  • Investigating
    Investigating

    We are currently investigating this incident.

Aug 2023

aus.mxroute.com server
  • Resolved
    Resolved

    Hardware migration complete. Crossbox will be reinstalled soon, but everything else is working. Remember that Crossbox is only one webmail option, others are at https://aus.mxroute.com:2096

  • Identified
    Update

    Hardware migration will most likely take place within the next 4 hours. Users may see a few rejected connections during the transition. This should clear up on it's own in a very short amount of time.

  • Identified
    Update

    We'll be pulling the trigger on hardware migration for the Aus server tonight. No downtime, unlikely customers will notice anything other than a brief hiccup. The hiccup in question is expected to be less troubling than your cell phone switching towers with a rough handshake.

  • Identified
    Update

    While the aus server has been online since our last update, we're not confident in it's ability to stay online. We are migrating the system to new hardware right now. Steps have been taken to ensure data integrity in the event of a hardware failure, should it occur prior to completion of our migration.

  • Identified
    Update

    The Aus server is online. This shouldn't be celebrated yet. Here's to copying data as fast as physically possible before a degraded RAID1 array which failed to boot multiple times says it's had enough.

  • Identified
    Update

    The RAID controller is still reporting a fault, but we have more eyes on the issue on site than we've had in the previous hours.

  • Identified
    Update

    We are continuing to work on a fix for this incident.

  • Identified
    Update

    We are continuing to work on a fix for this incident.

  • Identified
    Update

    We are continuing to work on a fix for this incident.

  • Identified
    Update

    We are continuing to work on a fix for this incident. The RAID controller has failed, and we're waiting for on-site staff to further investigate possible solutions with available hardware.

  • Identified
    Update

    We are continuing to work on a fix for this incident.

  • Identified
    Identified

    There appears to be a hardware issue with the aus.mxroute.com server. We are working with on-site staff to determine the scope and best path to resolution.

  • Investigating
    Investigating

    We are currently investigating this incident.

Path.net just shit the bed
  • Resolved
    Resolved

    This incident has been resolved.

  • Identified
    Identified

    We are continuing to work on a fix for this incident. Some updates here: https://status.path.net/incidents/50t7b6ts412f

  • Investigating
    Investigating

    We are currently investigating this incident. It looks like path.net just took down a good portion of the internet.

Aug 2023 to Oct 2023