Support experiencing degraded performance

Billing Panel

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
DirectAdmin Panel
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

IMAP

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

SMTP

99.98% uptime
Sep 2023 · 100.0%Oct · 99.93%Nov · 100.0%
Sep 2023100.0% uptime
Oct 202399.93% uptime
Nov 2023100.0% uptime
Webmail
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

Support

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

Notice history

Nov 2023

friday.mxlogin.com
  • Completed
    December 05, 2023 at 1:35 AM
    Completed
    December 05, 2023 at 1:35 AM

    We are finally considering this to be fully resolved. There should be no abnormalities or parts that are not working as intended.

  • Update
    November 28, 2023 at 9:26 PM
    In progress
    November 28, 2023 at 9:26 PM

    Migrating Crossbox now for the friday.mxlogin.com server, as we prepare to close this maintenance today.

  • Update
    November 28, 2023 at 3:43 PM
    In progress
    November 28, 2023 at 3:43 PM

    Converting mdbox to maildir on accounts on the friday.mxlogin.com server to enable better troubleshooting for users still experiencing issues.

  • Update
    November 28, 2023 at 3:28 AM
    In progress
    November 28, 2023 at 3:28 AM

    The expectation is that all users on the friday.mxlogin.com server are in a relatively good spot, with a few exceptions:

    1. Some custom mail/webmail subdomains lack SSL certificates. Why didn't these copy over from the JetBackup clone? Great question. Working on it. Most are fixed already. (Expected resolved as of Nov 28 12:00AM US/Central)

    2. MySQL migration is taking much longer than expected but that just means no Crossbox until we're finished with that, and users on this server haven't had Crossbox for quite some time now so at least it's not a new development.

    3. There is a small amount of email data on the old server that isn't on the new, and that's clearing up over time right now. We're not getting any more complaints about this, it's unclear if anything is missing that any users even notice. Users missing anything vital, please open a support ticket so we can rush the final sync of your account from the old server (as our last sync is around 1-2 days old depending on account position). (Expected almost entirely resolved as of Nov 28 12:00AM US/Central)

    4. If you are having trouble with webmail, use webmail.mxroute.com.

    5. If your cPanel password needs to be reset, the process is virtually identical to our DA servers: https://mxroutedocs.com/directadmin/resetpass/

    Overall we've learned that migration cPanel servers is no longer as simple as it used to be. It wasn't long ago that cPanel servers migrated simply, and our DirectAdmin servers were the ones that we had trouble migrating smoothly. The two seem to have switched position on us, some of which is quite likely our fault (ex. some creative symlinking used to fix a previous issue). We're still plugging away at this and expect everything to be done soon.

  • Update
    November 27, 2023 at 8:08 PM
    In progress
    November 27, 2023 at 8:08 PM

    If no other webmail is functioning for users on the friday.mxlogin.com server during the finalization of this maintenance, please try webmail.mxroute.com. We are continuing to finalize this and expect to be done in a few minutes.

  • Update
    November 27, 2023 at 7:58 PM
    In progress
    November 27, 2023 at 7:58 PM

    DNS swap for friday.mxlogin.com has just occurred. Working to fix SSL and migrate Crossbox over to get it re-enabled. May be some MySQL errors in this process as well.

  • Update
    November 19, 2023 at 5:11 AM
    In progress
    November 19, 2023 at 5:11 AM

    The end of this maintenance is in sight, but it's still a little unclear how quickly we're moving toward that end. We'll know more as it progresses. Some things remain true at this stage:

    1. Crossbox (mail.mxlogin.com) is offline for users of the friday.mxlogin.com server (ONLY that server). Use webmail.mxroute.com if needed.

    2. Some users will see IMAP or SMTP errors intermittently which appear to resolve very quickly. The number of users experiencing this are a fraction of what they were when this maintenance began, and we'll be at zero soon enough.

    3. As this isn't an "outage" we've been working to make sure the resolution isn't worse than the problem, and we're almost there.

  • Update
    November 16, 2023 at 4:42 PM
    In progress
    November 16, 2023 at 4:42 PM

    We're extending the maintenance period again, as we're still perfecting this fix. We continue to mitigate the issues. Only a handful of customers on the friday.mxlogin.com server should continue to see occasional IMAP errors at this time.

  • Update
    November 13, 2023 at 6:14 PM
    In progress
    November 13, 2023 at 6:14 PM

    The initial clone of the server has completed. Next we're going to sync that with the latest data from the primary, then switch DNS to the secondary, then run another sync to pick up anything left from the primary. No ETA.

  • Update
    November 09, 2023 at 10:00 AM
    In progress
    November 09, 2023 at 10:00 AM

    As this long maintenance cycle continues, there may be some improvements to mitigation. Let's restate the issues as we currently see them:

    1. Crossbox (mail.mxlogin.com) is intentionally down and will remain down until the end of this maintenance. ONLY for users on the friday.mxlogin.com server. NOT users of ANY other server. You have many ways to check your email but if you want a quick recommendation, try webmail.mxroute.com.

    2. IMAP errors will occur at random for the more highly active users on the server. The time to resolution after each error should now be reduced to almost instant.

    3. Some users will see delayed inbound emails, though this delay should be almost unnoticeable as of right now.

    The hope is that with changes to mitigation this morning, the issues will be even more minor as we continue to work toward the total resolution. This is still not an outage, and no outage of this server is planned. This is intermittent minor issues which may annoy a fair portion of users on the server until resolved.

  • Update
    November 07, 2023 at 5:30 PM
    In progress
    November 07, 2023 at 5:30 PM

    Everything is still going according to plan on the migration of the friday.mxlogin.com server. Due to extremely unique conditions there will continue to be intermittent IMAP and SMTP errors until this process is complete, and we'll keep trying to minimize them. Most customers will not notice any issues during this time, but our more active users will. There is no short term relief for those users. Crossbox (mail.mxlogin.com) will remain down for users on this server, users will be encouraged to use their own mail clients (as most already are), or use webmail.mxroute.com in the meantime.

    If you are not certain whether or not this message impacts your service, it almost certainly does not.

  • In progress
    November 06, 2023 at 2:31 AM
    In progress
    November 06, 2023 at 2:31 AM

    This migration does not require downtime of the server, though it is being associated with downtime of one of our webmail options on the server (Crossbox - mail.mxlogin.com). You can continue to use other webmail options, as well as webmail.mxroute.com, to access your email. You can continue using all third party email clients without issue. There may be occasional errors from this server, as this is a symptom of why this maintenance is taking place.

  • Planned
    November 03, 2023 at 8:36 PM
    Planned
    November 03, 2023 at 8:36 PM

    Intermittent issues with IMAP and SMTP can only be resolved on the friday.mxlogin.com server by migration to new hardware. It sounds strange, but it's true. We're in the process of doing this in production with the aim of not causing any downtime. It is expected that this could take the entire weekend as a result. Crossbox (mail.mxlogin.com) may be inaccessible for much, it not all, of this process for users on this server.

Oct 2023

Sep 2023

Lucy server
  • Resolved
    Resolved

    Lucy is back online. A full account of events should follow a bit later.

  • Identified
    Update

    There's an admin dream team working on this box right now with us.

  • Identified
    Update

    We are continuing to work on a fix for this incident.

  • Identified
    Update

    With activity split between getting the production server to boot it's OS, and deploying a replacement server, suddenly JetBackup doesn't work on a new server. So we opened a ticket with JetBackup. Whatever supernatural entity is responsible for this incredible sequence of events, even you have to sleep eventually.

  • Identified
    Update

    We're still working on the OS. During this process, we've been preparing a replacement server to restore backups to, as we'd like that process to be moving along just in case we need it. If it's time wasted, and we sure hope it is, it was a worthy cause.

  • Identified
    Update

    By all accounts we should either be online or right on the edge of being online. But the OS is still giving us hell. You can vent in tickets if you want, and we get it, but you won't speed this up. If we had a magic button we'd have pressed it already.

  • Identified
    Update

    Hardware replaced, now it's just a matter of getting the OS to deal with the hardware changes and boot properly.

  • Identified
    Update

    Still working on this. Note there is a reason it says 'Operational' above and it's too long to explain here, but no service on the lucy.mxrouting.net server is online until the text of these updates says otherwise.

  • Identified
    Update

    RAID controller replacement didn't help. Was it replaced wrong? Is it the riser card? Stay tuned to find out.

  • Identified
    Update

    We are continuing to work with remote hands on the raid controller replacement.

  • Identified
    Update

    Frustration is understood. We're waiting to hear back from remote hands at the datacenter. Here are some facts:

    1. No inbound email will be missed, it will flow in after the server is back online.
    2. Many reasonable redundancies were in place here. It is not common for raid controllers to suddenly fail.
    3. This server is actively backed up, no matter what else happens.

    We expect to pop in the new controller, import the raid, and work fine under the expectation that this hardware configuration is perfect. This being a statistical anomaly alone, we don't expect a repeat event.

  • Identified
    Update

    The server is not detecting the RAID controller. We're going to swap out the controller with the spare.

  • Identified
    Identified

    The Lucy server is having trouble booting, so that's interesting. We're working on it.

  • Investigating
    Investigating

    We are currently investigating this incident.

Sep 2023 to Nov 2023