MXroute - Notice history

This incident has been resolved. Postmortem:

In certain events this config block caused exim to give an empty HELO to postfix:

helo_data = ${if eq{$sender_address_domain}{}\

{${lookup{${lc:${domain:$h_from:}}}lsearch,ret=key{/etc/virtual/domainowners}{$value}}}\

{${if exists{/etc/virtual/helo_data}{${lookup{$sending_ip_address}iplsearch{/etc/virtual/helo_data}{$value}{$primary_hostname}}}{$primary_hostname}}}\

}

That config block has been updated:

helo_data = ${if exists{/etc/virtual/helo_data}{${lookup{$sending_ip_address}iplsearch{/etc/virtual/helo_data}{$value}{$primary_hostname}}}{$primary_hostname}}

MySQL is working fine now on Ocean, Aus, Acadia, and Friday cPanel servers. The Banshee server is the only one remaining to be repaired. This one may take more time.

The problem appears to be spreading in our cPanel servers. This is not a Crossbox problem. This is a cPanel problem which impacts Crossbox. It seems that cPanel pushed a bad MySQL update which broke a wealth of servers, not just ours. We're evaluating and testing options, we don't want to risk causing larger issues as this impact is not what we consider to be service breaking.