Notice history

Jul 2024

Aug 2024

Sep 2024

Jul 2024

Aug 2024

Sep 2024

Jul 2024

Aug 2024

Sep 2024

Jul 2024

Aug 2024

Sep 2024

Jul 2024

Aug 2024

Sep 2024

Jul 2024

Aug 2024

Sep 2024

Sep 2024

Resolved
September 30, 2024 at 10:10 AM
Resolved
September 30, 2024 at 10:10 AM
This incident has been resolved.
Update
September 30, 2024 at 9:25 AM
Update
September 30, 2024 at 9:25 AM
We were seeing webmail outages up to 5 minutes at a time on the Safari server yesterday, increasing in frequency throughout the day. The decision was made to move the Apache includes to the SSD array currently housing MySQL, to speed up config reloads. For us that means moving the DA user data. This is proving to take longer than predicted, there's a strange blocker to determining progress other than "done" or "not done."
Giving an ETA for DirectAdmin to be back online would be irresponsible. Best course of action is see this through. No email access or delivery is harmed by this. This is only administration control panel access.
Identified
September 30, 2024 at 6:36 AM
Identified
September 30, 2024 at 6:36 AM
We are performing emergency maintenance that does not impact any email service on the safari.mxrouting.net server. Part of this process involves temporarily disabling DirectAdmin. We will keep you posted.

safari.mxrouting.net hardware upgrade

Completed
September 13, 2024 at 3:05 AM
Completed
September 13, 2024 at 3:05 AM
Maintenance has completed successfully.
Update
September 13, 2024 at 2:17 AM
In progress
September 13, 2024 at 2:17 AM
Server has been back up for 20 minutes but now we're moving MySQL. This means your email clients are working fine, webmail clients like webmail.mxroute.com are working fine, but Crossbox and Roundcube on the Safari server itself will be back up once we finish moving MySQL data to the new SSD array.
Update
September 13, 2024 at 1:16 AM
In progress
September 13, 2024 at 1:16 AM
Services taken down for power down.
Update
September 13, 2024 at 1:06 AM
In progress
September 13, 2024 at 1:06 AM
Shutting down MySQL
In progress
September 13, 2024 at 12:00 AM
In progress
September 13, 2024 at 12:00 AM
Maintenance is now in progress
Planned
September 13, 2024 at 12:00 AM
Planned
September 13, 2024 at 12:00 AM
The safari.mxrouting.net server will be going down sometime between 7PM and 11PM US/Central time today for the addition of two SSD drives that will be used to migrate the MySQL installation off of the same array housing the rest of customer data. This is intended to address the root cause of a performance issue that had been reported a while back. While we've been fine on performance for some time, that event did teach us what was necessary to prevent it from occurring again in the future, and this is it. We do not expect the server to be down during the 4 hour window that we've specified. We expect it to occur sometime during that window. Unfortunately we are not able to schedule this to a more specific time or to a further date in the future, today is when the stars aligned.

Resolved
September 10, 2024 at 5:46 AM
Resolved
September 10, 2024 at 5:46 AM
This incident has been resolved.
Monitoring
September 10, 2024 at 3:15 AM
Monitoring
September 10, 2024 at 3:15 AM
From Jeff. The man, myth, legend:
"Arelion/Telia is having an outage caused by an equipment failure in Atlanta that is impacting East/West traffic in the US. Arelion plans to replace the equipment shortly (they said at about midnight Eastern time). We've turned off Arelion in both Ashburn and LA temporarily until the issue is resolved."
Investigating
September 10, 2024 at 3:11 AM
Investigating
September 10, 2024 at 3:11 AM
Some users are reporting high packet loss when connecting to servers in our Ashburn location. A check from ping.pe shows only a few routes impacted, most can reach it just fine. Looks to have a correlation with GTT, unclear. Investigating.

Resolved
September 02, 2024 at 7:08 PM
Resolved
September 02, 2024 at 7:08 PM
This incident has been resolved.
Identified
September 01, 2024 at 4:16 AM
Identified
September 01, 2024 at 4:16 AM
The application reports that the license is expired for this server. It isn't. We've done everything we can, it's in the hands of their developers as of right now. You can use any other email client or webmail client in the meantime, webmail.mxroute.com is a popular choice.

Aug 2024

Restarting Crossbox

Completed
August 31, 2024 at 8:21 PM
Completed
August 31, 2024 at 8:21 PM
Done.
In progress
August 31, 2024 at 8:08 PM
In progress
August 31, 2024 at 8:08 PM
We are restarting instances of the Crossbox webmail application across the fleet.

Echo server reboot

Completed
August 27, 2024 at 8:46 PM
Completed
August 27, 2024 at 8:46 PM
Been back up for a while, forgot to close status.
In progress
August 27, 2024 at 8:30 PM
In progress
August 27, 2024 at 8:30 PM
Maintenance is now in progress
Planned
August 27, 2024 at 8:30 PM
Planned
August 27, 2024 at 8:30 PM
The echo.mxrouting.net server will be rebooted at 8:30 PM GMT on August 27th, 2024. The expectation is that this takes about 5-10 minutes and then everything is back up and running. Secondary plans will be prepared by that time in the case of failure to come back online.

Resolved
August 12, 2024 at 4:35 PM
Resolved
August 12, 2024 at 4:35 PM
This incident has been resolved. Postmortem:

In certain events this config block caused exim to give an empty HELO to postfix:

helo_data = ${if eq{$sender_address_domain}{}\
{${lookup{${lc:${domain:$h_from:}}}lsearch,ret=key{/etc/virtual/domainowners}{$value}}}\
{${if exists{/etc/virtual/helo_data}{${lookup{$sending_ip_address}iplsearch{/etc/virtual/helo_data}{$value}{$primary_hostname}}}{$primary_hostname}}}\
}

That config block has been updated:

helo_data = ${if exists{/etc/virtual/helo_data}{${lookup{$sending_ip_address}iplsearch{/etc/virtual/helo_data}{$value}{$primary_hostname}}}{$primary_hostname}}
Identified
August 12, 2024 at 4:25 PM
Identified
August 12, 2024 at 4:25 PM
We're seeing some intermittent outbound email delivery delays due to a strange issue with postfix. Currently investigating the problem a bit deeper with an increased log level to see if we can shake loose some more details.

Jul 2024

Identified
July 23, 2024 at 11:01 PM
Identified
July 23, 2024 at 11:01 PM
The Crossbox instance is currently down on the banshee.mxlogin.com server. This server has no doubt had a bit of a back and forth relationship with it, at one point being the only server to consistently run without it. In the meantime feel free to use webmail.mxroute.com or your own email clients. This appears to be a continuation of this problem: https://support.cpanel.net/hc/en-us/articles/24546938662039-Upstream-issue-MySQL-8-0-38-does-not-start-Segmentation-Fault

Resolved
July 11, 2024 at 10:25 PM
Resolved
July 11, 2024 at 10:25 PM
This incident has been resolved.
Investigating
July 11, 2024 at 8:04 PM
Investigating
July 11, 2024 at 8:04 PM
The Friday server experienced a load exceeding 500. The system was rebooted to try to bring down the load to troubleshoot the cause. As of this posting, the server has not come back online.

Resolved
July 18, 2024 at 1:24 AM
Resolved
July 18, 2024 at 1:24 AM
This incident has been resolved for several days, we just wanted to leave the issue open until we felt good about it.
Investigating
July 11, 2024 at 1:15 PM
Investigating
July 11, 2024 at 1:15 PM
There are instances of "condition check lookup defer" and "temporary local problem" on this server. cPanel has assigned this an incident number, https://support.cpanel.net/hc/en-us/articles/4410449210519-Dovecot-restart-causes-message-delivery-failure-with-error-condition-check-lookup-defer

Resolved
July 07, 2024 at 10:48 PM
Resolved
July 07, 2024 at 10:48 PM
This incident has been resolved.
Investigating
July 07, 2024 at 10:46 PM
Investigating
July 07, 2024 at 10:46 PM
We are currently investigating this incident.

Resolved
July 09, 2024 at 7:19 PM
Resolved
July 09, 2024 at 7:19 PM
This incident has been resolved.
Update
July 09, 2024 at 7:09 PM
Update
July 09, 2024 at 7:09 PM
MySQL is working fine now on Ocean, Aus, Acadia, and Friday cPanel servers. The Banshee server is the only one remaining to be repaired. This one may take more time.
Update
July 04, 2024 at 5:49 PM
Update
July 04, 2024 at 5:49 PM
The problem appears to be spreading in our cPanel servers. This is not a Crossbox problem. This is a cPanel problem which impacts Crossbox. It seems that cPanel pushed a bad MySQL update which broke a wealth of servers, not just ours. We're evaluating and testing options, we don't want to risk causing larger issues as this impact is not what we consider to be service breaking.
Identified
July 02, 2024 at 10:30 PM
Identified
July 02, 2024 at 10:30 PM
Currently users are unable to access Crossbox (or anything else which uses MySQL) on these cPanel servers: Banshee, Aus, Friday, Ocean. This is specific to our cPanel servers, we are not seeing any similar behavior on our DirectAdmin servers. It may take some time to get a handle on this, it's quite odd. As it is impacting a small number of customers this status page will continue to say that everything is operational, as we do not want to give the rest of our customers a reason to not open a support ticket if they are experiencing an issue with any other webmail.

Jul 2024 to Sep 2024

MXroute - Notice history

All systems operational

Notice history

Sep 2024

Aug 2024

Jul 2024