MXroute - Notice history

This has been resolved for most of the day, considering resolved but continuing to monitor.

Performance is starting to level back out. We're still seeing iowait spikes but not nearly to the same degree. Cause still unclear. Performance may continue to vary, not going to mark the issue resolved until it has been resolved for hours.

There may be another reboot in this server's future to test the disks outside of the OS, as 80% iowait seems unwarranted.

Reboot finished. Continuing to work on server.

Rebooting this server is going to be a necessary troubleshooting step. Doing that now.

The motherboard failed on the server. The disks had to be migrated to a new box. Everything is fine, no further issue expected.

Server is online. Waiting for postmortem from on-site techs.

We are continuing to wait for an update from on-site technicians.

Datacenter technicians have been working on the server for a bit to determine why it is inaccessible. It is not believed to be a hardware issue, this appears to be a network issue.

Mail flow fully restored. It will take a bit longer to push all of the email that was queued during this, but it is flowing.

What happened: Our cloud provider rebooted the node without notice. Our interfaces did not come up as intended, monitoring system did not trip because BGP didn't actually go down, ZoneMTA couldn't create outbound connections. Connections from the same /24 to the same /24 no longer worked over port 2525.

What we did: Bring interfaces up, get email flowing again. Changed postfix bind address on filter server to cloud-provided IP to compensate for failure to connect over 2525 from/to the same /24 (as this "appears" to be outside of our ability to impact). Adjusted DNS for filter006.mxroute.com for Received header reverse DNS checks. Flushed mail queue. Changed monitoring system to ping the /24 that sends mail rather than just monitor it's BGP session.

Outbound mail flow is restored, but it is taking a bit of time to get all of the queued emails out.