Block storage in Vegas

This issue is resolved for now.

This morning we experienced an outage of block storage in our Vegas location. Unfortunately this did not trigger an outage alert, as the systems were not down. Rather, their home partitions were unable to read/write, which was only visible to users when attempting to read their mailboxes over IMAP. E-mails received during this time should have been accepted and failed to deliver internally, leaving them queued in exim on the root partition until the problem was resolved. This should result in no e-mail lost.

Moving forward we’re certain of two things:

1. This cannot be a regular occurrence but:
2. We must find a way to monitor for this type of outage to ensure that we can quickly work to resolve if it does occur. Monitoring for this type of behavior is not simple, and we’re not yet sure what the best path to that will be.