MXroute - Notice history

We are currently investigating this incident.

Our Las Vegas location, where the Longhorn and Acadia servers are located, is currently experiencing an outage. We are working on this.

Wait times for support escalations are not within an optimal time range. This is a temporary issue and we will keep the status page updated as they begin to level out.

A very small subset of outbound email has been seeing delays up to 1 hour. This is estimated to impact around 400 outbound emails per day, across various servers. While this is a very small portion of the email that our customers send, an unnecessary delay is in fact a problem.

The issue is that Rspamd on the outbound filter stops logging multiple times per day. This is not impacted by rsyslog, or by replacing rsyslog. This issue impacts only Rspamd, and Rspamd presents no errors relating to it. To resume logging, we have to restart the service. While the service is restarting, there is a very tiny window (roughly 1 second or less) where Postfix rejects connections from the primary servers (the ones you connect to and send mail from). When Postfix rejects the connection, Exim on the primary server that was rejected holds the email in queue and stops trying to connect to the filter server at all, until retry time is reached. The answer here is that we need to fix Rspamd, that's where this all comes from. We're working on building out a new deployment to completely replace it. This will be done in production with no downtime for any service. This status will be updated when we finish that new deployment.

The Echo server failed ping checks, was found unresponsive. We are currently investigating this incident.

This morning users on the Shadow server briefly saw the message "Your recipient address has been blocked by admins" when sending emails to any recipient @gmail.com. It is difficult to say precisely what caused this, but we can attribute the result to our automation. It was resolved relatively quickly, but not without users seeing the problem first.

We have seen an unusual increase in users on our service sending emails to invalid recipient domains. We realize that you cannot always prevent this, so we are taking steps to reduce the volume of it on our side. This means we are parsing logs for invalid recipient domains, adding them to a list, and preventing emails from even attempting to send to those domains later. Unfortunately, gmail.com was caught in one of these automated sweeps. The actual logic for how and why it ended up there is what is presently unknown, as the simple answer is mitigated by the fact that this only occurred on the Shadow server (while the same DNS service used to experience the temporary failure is shared across every server).

The automation is dead, it won't be coming back unless and until we add failsafes.