Support experiencing degraded performance

friday.mxlogin.com

Completed
Scheduled for November 03, 2023 at 8:36 PM – December 05, 2023 at 1:35 AM

Affects

IMAP
SMTP
Webmail
Updates
  • Completed
    December 05, 2023 at 1:35 AM
    Completed
    December 05, 2023 at 1:35 AM

    We are finally considering this to be fully resolved. There should be no abnormalities or parts that are not working as intended.

  • Update
    November 28, 2023 at 9:26 PM
    In progress
    November 28, 2023 at 9:26 PM

    Migrating Crossbox now for the friday.mxlogin.com server, as we prepare to close this maintenance today.

  • Update
    November 28, 2023 at 3:43 PM
    In progress
    November 28, 2023 at 3:43 PM

    Converting mdbox to maildir on accounts on the friday.mxlogin.com server to enable better troubleshooting for users still experiencing issues.

  • Update
    November 28, 2023 at 3:28 AM
    In progress
    November 28, 2023 at 3:28 AM

    The expectation is that all users on the friday.mxlogin.com server are in a relatively good spot, with a few exceptions:

    1. Some custom mail/webmail subdomains lack SSL certificates. Why didn't these copy over from the JetBackup clone? Great question. Working on it. Most are fixed already. (Expected resolved as of Nov 28 12:00AM US/Central)

    2. MySQL migration is taking much longer than expected but that just means no Crossbox until we're finished with that, and users on this server haven't had Crossbox for quite some time now so at least it's not a new development.

    3. There is a small amount of email data on the old server that isn't on the new, and that's clearing up over time right now. We're not getting any more complaints about this, it's unclear if anything is missing that any users even notice. Users missing anything vital, please open a support ticket so we can rush the final sync of your account from the old server (as our last sync is around 1-2 days old depending on account position). (Expected almost entirely resolved as of Nov 28 12:00AM US/Central)

    4. If you are having trouble with webmail, use webmail.mxroute.com.

    5. If your cPanel password needs to be reset, the process is virtually identical to our DA servers: https://mxroutedocs.com/directadmin/resetpass/

    Overall we've learned that migration cPanel servers is no longer as simple as it used to be. It wasn't long ago that cPanel servers migrated simply, and our DirectAdmin servers were the ones that we had trouble migrating smoothly. The two seem to have switched position on us, some of which is quite likely our fault (ex. some creative symlinking used to fix a previous issue). We're still plugging away at this and expect everything to be done soon.

  • Update
    November 27, 2023 at 8:08 PM
    In progress
    November 27, 2023 at 8:08 PM

    If no other webmail is functioning for users on the friday.mxlogin.com server during the finalization of this maintenance, please try webmail.mxroute.com. We are continuing to finalize this and expect to be done in a few minutes.

  • Update
    November 27, 2023 at 7:58 PM
    In progress
    November 27, 2023 at 7:58 PM

    DNS swap for friday.mxlogin.com has just occurred. Working to fix SSL and migrate Crossbox over to get it re-enabled. May be some MySQL errors in this process as well.

  • Update
    November 19, 2023 at 5:11 AM
    In progress
    November 19, 2023 at 5:11 AM

    The end of this maintenance is in sight, but it's still a little unclear how quickly we're moving toward that end. We'll know more as it progresses. Some things remain true at this stage:

    1. Crossbox (mail.mxlogin.com) is offline for users of the friday.mxlogin.com server (ONLY that server). Use webmail.mxroute.com if needed.

    2. Some users will see IMAP or SMTP errors intermittently which appear to resolve very quickly. The number of users experiencing this are a fraction of what they were when this maintenance began, and we'll be at zero soon enough.

    3. As this isn't an "outage" we've been working to make sure the resolution isn't worse than the problem, and we're almost there.

  • Update
    November 16, 2023 at 4:42 PM
    In progress
    November 16, 2023 at 4:42 PM

    We're extending the maintenance period again, as we're still perfecting this fix. We continue to mitigate the issues. Only a handful of customers on the friday.mxlogin.com server should continue to see occasional IMAP errors at this time.

  • Update
    November 13, 2023 at 6:14 PM
    In progress
    November 13, 2023 at 6:14 PM

    The initial clone of the server has completed. Next we're going to sync that with the latest data from the primary, then switch DNS to the secondary, then run another sync to pick up anything left from the primary. No ETA.

  • Update
    November 09, 2023 at 10:00 AM
    In progress
    November 09, 2023 at 10:00 AM

    As this long maintenance cycle continues, there may be some improvements to mitigation. Let's restate the issues as we currently see them:

    1. Crossbox (mail.mxlogin.com) is intentionally down and will remain down until the end of this maintenance. ONLY for users on the friday.mxlogin.com server. NOT users of ANY other server. You have many ways to check your email but if you want a quick recommendation, try webmail.mxroute.com.

    2. IMAP errors will occur at random for the more highly active users on the server. The time to resolution after each error should now be reduced to almost instant.

    3. Some users will see delayed inbound emails, though this delay should be almost unnoticeable as of right now.

    The hope is that with changes to mitigation this morning, the issues will be even more minor as we continue to work toward the total resolution. This is still not an outage, and no outage of this server is planned. This is intermittent minor issues which may annoy a fair portion of users on the server until resolved.

  • Update
    November 07, 2023 at 5:30 PM
    In progress
    November 07, 2023 at 5:30 PM

    Everything is still going according to plan on the migration of the friday.mxlogin.com server. Due to extremely unique conditions there will continue to be intermittent IMAP and SMTP errors until this process is complete, and we'll keep trying to minimize them. Most customers will not notice any issues during this time, but our more active users will. There is no short term relief for those users. Crossbox (mail.mxlogin.com) will remain down for users on this server, users will be encouraged to use their own mail clients (as most already are), or use webmail.mxroute.com in the meantime.

    If you are not certain whether or not this message impacts your service, it almost certainly does not.

  • In progress
    November 06, 2023 at 2:31 AM
    In progress
    November 06, 2023 at 2:31 AM

    This migration does not require downtime of the server, though it is being associated with downtime of one of our webmail options on the server (Crossbox - mail.mxlogin.com). You can continue to use other webmail options, as well as webmail.mxroute.com, to access your email. You can continue using all third party email clients without issue. There may be occasional errors from this server, as this is a symptom of why this maintenance is taking place.

  • Planned
    November 03, 2023 at 8:36 PM
    Planned
    November 03, 2023 at 8:36 PM

    Intermittent issues with IMAP and SMTP can only be resolved on the friday.mxlogin.com server by migration to new hardware. It sounds strange, but it's true. We're in the process of doing this in production with the aim of not causing any downtime. It is expected that this could take the entire weekend as a result. Crossbox (mail.mxlogin.com) may be inaccessible for much, it not all, of this process for users on this server.