Some systems are experiencing issues

Past Incidents

16th January 2021

No incidents reported

15th January 2021

Crossbox Crossbox repeatedly asking to update

Fixed

14th January 2021

Crossbox iOS app failing on Taylor server

iOS app is unable to connect to the Taylor server, reason unclear at this time

Crossbox Crossbox repeatedly asking to update

Crossbox is currently presenting a popup asking for you to update. When you click to allow it to update, it repeats. Reported to devs.

Charlotte Relay Charlotte Relay Outbound Delays

For what looks to be somewhere in the range of an hour, roughly 1/3rd of outbound email saw a delay due to an issue on the Charlotte relay server. The root cause of the issue has not yet been identified, but the server has been pulled from rotation pending further investigation.

13th January 2021

No incidents reported

12th January 2021

Crossbox Crossbox management service

A management service used by Crossbox has been failing randomly on some servers without restarting itself as needed. A mitigation has been put in place that may be less than ideal, but should make the issue mostly unnoticeable for users until it can be patched in the coming week(s).

11th January 2021

Longhorn Longhorn Quotas

The mentioned problem does not appear to be what it was thought to be, and may not actually be a problem at all.

Longhorn Longhorn Quotas

Longhorn quotas are not functioning properly. Upgrading kernel and prepping for reboot.

10th January 2021

Outbound Filters SBL'd IP in headers

One of our outbound filters had been listed at Spamhaus for reasons unrelated to our service. The IP belongs to a service provider, our filters are split between two service providers. While that IP does not send mail, it is recorded in headers and some software (SpamAssassin being one) may check the reputation of all IPs in the headers, which could lead to increased spam folder delivery. The server in question was pulled from rotation pending resolution.

Blizzard Blizzard Server Issues

The root partition claimed 100% utilization, which prevented exim from writing /var/spool, and prevented emails from queuing. This caused exim to return a 4xx error, which is good and what it should have done to ensure that no email is lost. The issue will have delayed inbound email, but it will eventually be received so long as the sending servers function according to spec, and there is no reason to expect otherwise.

However, the root partition was never at 100% utilization despite reporting that. Currently, I have no explanation for that, but will continue investigating.