Past Incidents

If your email account hits disk quota, your sender will not receive a bounce email or error indicating that the email failed to deliver. In fact, despite our extreme disagreement with the practice, it will be silently accepted and never delivered unless you increase your quota within a day or so of hitting said quota. This is problematic but complicated. For the moment, this edge case impacts less users than any other issue we've publicized to date, and can be mitigated by not giving email accounts tiny quotas. For that reason, we're taking the time to weigh options, experiment with solutions, and see where we land.

Let's go over the cause, the reasons for the cause, and the potential resolutions on the table.

Cause: We accept the email and hold it in queue, retrying for about a day before we give up and declare it a failure. The inbound servers are not allowed to send bounce emails, the outbound filters will reject them.

Reason 1: The default behavior of the stack will accept some email and reject it before delivery under certain edge cases, or it will accept email to be forwarded and then the outbound filters reject it as spam (because the outbound filters are more strict, and this keeps IP reputation and customer complaints to a healthy medium). The inbound servers being allowed to send bounce emails can cause them to be utilized for backscatter attacks by spammers, courtesy of you and us (thus lowering both our reputation and yours together).

Reason 2: Customers sometimes make redirect filters which redirect all email to third party services. If, for some reason, the third party rejects a redirected email then the bounce can come back to the user, and because the redirect filter is less intelligent than a forwarder, it redirects the bounce email to the third party. The third party often rejects the email for the same reason, which bounces, which redirects, which bounces, which redirects, which bounces, which redirects, and suddenly we have an email queue of 20,000 emails and we're headed toward a meltdown of the infrastructure.

So fixing this is not a simple task. It may mean that we need to pick one of these solutions:

1. We need to REJECT emails when you hit disk quota, NOT bounce them. They won't be held in queue and retried later, they will fail outright.

2. We need to get better at filtering bounce emails to allow certain ones through while others fail.

3. We need to hold emails for much longer when they fail due to quota, and continue retrying for longer. This may be the messiest solution.

Emergency router maintenance could cause rolling outages of up to 30 minutes for each server in our Germany location. No inbound email will be lost during this time, though it may be delayed if it attempts delivery during the outage. Servers that may experience this: Ocean, Friday, Banshee, Safari, Lucy, Arrow, Echo, Blizzard, Sunfire, Taylor, Pixel, Shadow

This event will last between 02:30 and 07:00 UTC. We're expecting that not all servers will be down at once, and that any down will not exceed 30 minutes. For the status page, we're just marking every server in Germany for partial outage during this time frame.

Maintenance is rolling through the network in Germany. Brief outages are unavoidable. No inbound mail will be missed. We’ll talk more about this event, thoughts, and reactions in #postmortems at chat.mxroute.com in the next 24 hours.

Arrow, Lucy, Safari, and Friday are back online. Postmortem in the #postmortems channel at chat.mxroute.com.