Dovecot on Chocobo server

Affected Systems

Postmortem

A user on the Chocobo server reported missing emails. I sent test messages in to confirm, and Dovecot's LMTP logged "saved mail to INBOX" successfully for each one, yet not a single file was ever on disk. The user's previously stored mail had also been quietly disappearing for some unknown stretch before today, with no IMAP, POP3, Sieve, replication, or administrative action accountable for it. After hours of investigation through logs, index dumps, and process state, the smoking gun showed up in the mailbox's index transaction journal. Every delivery was being committed normally, then immediately followed by an internal flag change marking the message for deletion, then expunged. The flag change carried a marker that only gets set by Dovecot's own internal code paths, never by IMAP clients. No external action had triggered any of it. The Dovecot process was eating the user's mail seconds after each message landed. A Dovecot restart cleared the in-process state for that mailbox, and new mail to the account stopped vanishing. That is a workaround, not a fix. Chocobo is the only server in the MXroute fleet running Dovecot 2.4.3. Every other server runs 2.4.2 or earlier, and none of them has ever shown this pattern in 13 years of operation. This is also the second unique-to-Chocobo Dovecot incident in recent weeks. The first was the SNI cert reload issue I wrote about previously, which culminated in an E2BIG error and a 75 minute outage. Two unprecedented Dovecot bugs, both on the only server running 2.4.3, is enough signal for me to act. I performed an emergency maintenance to downgrade Dovecot from 2.4.3 to 2.4.2, matching the rest of the fleet. The downgrade was clean, indexes regenerated on first access as expected, and no mailboxes are showing the deliver-then-expunge pattern after the rollback. The mail lost from the affected user is not recoverable. Those messages were unlinked from disk by the bug as they arrived, with no copy elsewhere. As of right now I can prove that this issue did not impact all users on Chocobo, but I cannot prove that it only impacted one. I am sorry to that user, and to anyone else who may have been quietly affected by this on Chocobo before today and not noticed in time to tell me about it. I will be staying on Dovecot 2.4.2 across the entire fleet until I am confident that a future release is stable on our workload. I owe this server, and the people relying on it, more careful caution about adopting new versions. Lesson learned, again.