MXroute - Notice history

All systems operational

Billing Panel - Operational

100% - uptime
Aug 2024 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2024
Sep 2024
Oct 2024

DirectAdmin Panel - Operational

100% - uptime
Aug 2024 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2024
Sep 2024
Oct 2024

IMAP - Operational

100% - uptime
Aug 2024 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2024
Sep 2024
Oct 2024

SMTP - Operational

100% - uptime
Aug 2024 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2024
Sep 2024
Oct 2024

Webmail - Operational

100% - uptime
Aug 2024 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2024
Sep 2024
Oct 2024

Support - Operational

100% - uptime
Aug 2024 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2024
Sep 2024
Oct 2024

Notice history

Oct 2024

Software updates to fleet
  • Completed
    October 12, 2024 at 1:09 AM
    Completed
    October 12, 2024 at 1:09 AM

    IMAP server back up on Pixel. That wasn't supposed to happen. This software upgrade path is flawed and needs to be reconsidered. Halting updates.

  • Update
    October 12, 2024 at 12:14 AM
    In progress
    October 12, 2024 at 12:14 AM

    The IMAP server on pixel.mxrouting.net was supposed to go down for a few seconds, but it's taking longer. SMTP is not down, IMAP is.

  • In progress
    October 11, 2024 at 11:45 PM
    In progress
    October 11, 2024 at 11:45 PM

    We're running some necessary software updates to the fleet. Random things might flap up/down just a bit but nothing too bad or for very long.

    If a webmail instance goes down for a bit on one of these boxes during the MySQL/PHP upgrades, feel free to use https://webmail.mxroute.com or any other email client you already have.

Crossbox on safari.mxrouting.net maintenance
  • Completed
    October 06, 2024 at 7:00 PM
    Completed
    October 06, 2024 at 7:00 PM
    Maintenance has completed successfully
  • Update
    October 06, 2024 at 4:35 PM
    In progress
    October 06, 2024 at 4:35 PM

    All indicators right now point to this taking longer than scheduled, but the reason for the maintenance is the same reason that I cannot presently tell how much longer that might be.

  • In progress
    October 06, 2024 at 5:00 AM
    In progress
    October 06, 2024 at 5:00 AM
    Maintenance is now in progress
  • Planned
    October 06, 2024 at 5:00 AM
    Planned
    October 06, 2024 at 5:00 AM

    We will be performing maintenance on the Crossbox (mail.mxlogin.com) webmail client at 12:00AM US/Central on October 6th, 2024. We are scheduling a 12 hour outage and expecting it to take much less. This does not impact email service at all, only one of our webmail choices. If you need a suggestion on a different webmail to use during this time, try using webmail.mxlogin.com which will function with any email account that is hosted on our systems.

    If for some reason you plan to use webmail.mxlogin.com to login to an email account that is not pointing it's MX records to us, you can add a TXT record to the DNS of that customer domain which has the name "mxwebmail" and the contents "safari.mxrouting.net" inside it.

Sep 2024

DirectAdmin Temporarily Disabled on Safari server
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Update

    We were seeing webmail outages up to 5 minutes at a time on the Safari server yesterday, increasing in frequency throughout the day. The decision was made to move the Apache includes to the SSD array currently housing MySQL, to speed up config reloads. For us that means moving the DA user data. This is proving to take longer than predicted, there's a strange blocker to determining progress other than "done" or "not done."

    Giving an ETA for DirectAdmin to be back online would be irresponsible. Best course of action is see this through. No email access or delivery is harmed by this. This is only administration control panel access.

  • Identified
    Identified

    We are performing emergency maintenance that does not impact any email service on the safari.mxrouting.net server. Part of this process involves temporarily disabling DirectAdmin. We will keep you posted.

safari.mxrouting.net hardware upgrade
  • Completed
    September 13, 2024 at 3:05 AM
    Completed
    September 13, 2024 at 3:05 AM
    Maintenance has completed successfully.
  • Update
    September 13, 2024 at 2:17 AM
    In progress
    September 13, 2024 at 2:17 AM

    Server has been back up for 20 minutes but now we're moving MySQL. This means your email clients are working fine, webmail clients like webmail.mxroute.com are working fine, but Crossbox and Roundcube on the Safari server itself will be back up once we finish moving MySQL data to the new SSD array.

  • Update
    September 13, 2024 at 1:16 AM
    In progress
    September 13, 2024 at 1:16 AM

    Services taken down for power down.

  • Update
    September 13, 2024 at 1:06 AM
    In progress
    September 13, 2024 at 1:06 AM

    Shutting down MySQL

  • In progress
    September 13, 2024 at 12:00 AM
    In progress
    September 13, 2024 at 12:00 AM
    Maintenance is now in progress
  • Planned
    September 13, 2024 at 12:00 AM
    Planned
    September 13, 2024 at 12:00 AM

    The safari.mxrouting.net server will be going down sometime between 7PM and 11PM US/Central time today for the addition of two SSD drives that will be used to migrate the MySQL installation off of the same array housing the rest of customer data. This is intended to address the root cause of a performance issue that had been reported a while back. While we've been fine on performance for some time, that event did teach us what was necessary to prevent it from occurring again in the future, and this is it. We do not expect the server to be down during the 4 hour window that we've specified. We expect it to occur sometime during that window. Unfortunately we are not able to schedule this to a more specific time or to a further date in the future, today is when the stars aligned.

Packet loss over a few routes to Ashburn
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    From Jeff. The man, myth, legend:

    "Arelion/Telia is having an outage caused by an equipment failure in Atlanta that is impacting East/West traffic in the US. Arelion plans to replace the equipment shortly (they said at about midnight Eastern time). We've turned off Arelion in both Ashburn and LA temporarily until the issue is resolved."

  • Investigating
    Investigating

    Some users are reporting high packet loss when connecting to servers in our Ashburn location. A check from ping.pe shows only a few routes impacted, most can reach it just fine. Looks to have a correlation with GTT, unclear. Investigating.

Crossbox down for users on wednesday.mxrouting.net server
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    The application reports that the license is expired for this server. It isn't. We've done everything we can, it's in the hands of their developers as of right now. You can use any other email client or webmail client in the meantime, webmail.mxroute.com is a popular choice.

Aug 2024

Restarting Crossbox
  • Completed
    August 31, 2024 at 8:21 PM
    Completed
    August 31, 2024 at 8:21 PM

    Done.

  • In progress
    August 31, 2024 at 8:08 PM
    In progress
    August 31, 2024 at 8:08 PM

    We are restarting instances of the Crossbox webmail application across the fleet.

Echo server reboot
  • Completed
    August 27, 2024 at 8:46 PM
    Completed
    August 27, 2024 at 8:46 PM

    Been back up for a while, forgot to close status.

  • In progress
    August 27, 2024 at 8:30 PM
    In progress
    August 27, 2024 at 8:30 PM
    Maintenance is now in progress
  • Planned
    August 27, 2024 at 8:30 PM
    Planned
    August 27, 2024 at 8:30 PM

    The echo.mxrouting.net server will be rebooted at 8:30 PM GMT on August 27th, 2024. The expectation is that this takes about 5-10 minutes and then everything is back up and running. Secondary plans will be prepared by that time in the case of failure to come back online.

Outbound email delivery delays
  • Resolved
    Resolved

    This incident has been resolved. Postmortem:

    In certain events this config block caused exim to give an empty HELO to postfix:

    helo_data = ${if eq{$sender_address_domain}{}\

    {${lookup{${lc:${domain:$h_from:}}}lsearch,ret=key{/etc/virtual/domainowners}{$value}}}\

    {${if exists{/etc/virtual/helo_data}{${lookup{$sending_ip_address}iplsearch{/etc/virtual/helo_data}{$value}{$primary_hostname}}}{$primary_hostname}}}\

    }

    That config block has been updated:

    helo_data = ${if exists{/etc/virtual/helo_data}{${lookup{$sending_ip_address}iplsearch{/etc/virtual/helo_data}{$value}{$primary_hostname}}}{$primary_hostname}}

  • Identified
    Identified

    We're seeing some intermittent outbound email delivery delays due to a strange issue with postfix. Currently investigating the problem a bit deeper with an increased log level to see if we can shake loose some more details.

Aug 2024 to Oct 2024

Next