All software is buggy and it’s always a bit hypocritical for a developer to complain about bugs. I’m certainly aware that MailMate has many bugs and some of them are incredibly frustrating for some users. For those users it might be a comfort to know that an email client developer has to deal with one of the most frustrating types of bugs: IMAP server bugs.

An IMAP server bug can often be characterized as follows:

  • It only affects IMAP email clients and not a webmail client or some proprietary email client for the IMAP service involved.
  • It makes the email client seem like it has a bug.
  • There’s no way to report it.
  • It is probably not hard to fix.
  • It is never fixed.

(Even in the rare case of a fix, old versions of the IMAP server continue to be used.)

Most users (understandably) see no difference between email client bugs and server bugs. The email client takes the blame. Whatever the problem is I must either fix a bug in MailMate or add some workaround for an IMAP server bug.

UPDATE: Well, apparently I owe Yahoo an apology. I reached out to Yahoo on Twitter and I was then contacted by email. Apparently, they fixed some issues last week which they believe fixes the UID mismatch issues described below. I really hope the issue is fixed, but I remain a bit skeptical. The first time a MailMate user reported a UID mismatch issue for Yahoo was more than 2 years ago. But for now: I’m really sorry, Yahoo, for believing that this issue would never be fixed.

Some IMAP servers are worse than others. Today, I’m writing about Yahoo IMAP, but it could just as well be Exchange or some other buggy IMAP implementation. In any case, if you are searching for a new IMAP server then please consider some of the alternatives.

Yahoo IMAP has many problems. The inbox is incorrectly named Inbox instead of INBOX as required by the RFC, it can more or less randomly reject authentication making it seem like the password is incorrect, and it can suddenly become unavailable (not a bug but a stability issue) with various types of temporary errors. Most of these problems are handled gracefully by MailMate. The bug driving me mad is what is best described as “schizophrenic emails” — emails with more than 1 identity.

An IMAP mailbox has two different ways of enumerating emails. There’s a sequential way where emails are numbered 1 to N if the mailbox contains N messages and there’s so-called UIDs which stay the same even when messages are added or removed from the mailbox. The latter is very important for email clients caching emails (which means almost all desktop email clients).

Unfortunately, Yahoo IMAP has a bug where it messes up the mapping between sequence numbers and UIDs. The following is the relevant parts of an example of this happening:

06:32:51 C: C17 SELECT Trash
06:32:51 S: * 1381 EXISTS
...
06:32:51 S: C17 OK [READ-WRITE] SELECT completed; now in selected state
...
06:32:53 C: C21 UID FETCH 1:211045 (UID FLAGS)
06:33:07 S: * 1 FETCH (FLAGS ($Junk) UID 209612)
...
06:33:07 S: * 8 FETCH (FLAGS (\Seen) UID 209619)
06:33:07 S: * 9 FETCH (FLAGS (\Seen $NotJunk) UID 209665)
06:33:07 S: * 10 FETCH (FLAGS (\Seen $NotJunk) UID 209666)
06:33:07 S: * 11 FETCH (FLAGS (\Seen $NotJunk) UID 209667)
06:33:07 S: * 12 FETCH (FLAGS (\Seen $NotJunk) UID 209668)
06:33:07 S: * 13 FETCH (FLAGS (\Seen $NotJunk) UID 209669)
...
06:33:07 S: C21 OK UID FETCH completed

At this point everything is fine. In particular, note that message number 12 has UID 209668.

...
06:33:15 C: G23 UID FETCH 209612:209666 (UID BODY.PEEK[] RFC822.SIZE)
06:33:16 S: * 12 FETCH (UID 209665 RFC822.SIZE 3848 BODY[] {3869}
...
06:33:16 Error: Detected mismatch in UIDs received from server (previous value: 209668, new value: 209665).
06:33:16 Ignored mismatch to work around Yahoo server bug. Using previous UID value.

What? Message number 12 has UID 209665. We can even see that this UID belongs to message number 9. MailMate detects the issue and, as an experimental workaround, MailMate ignores the change and keeps the old UID. Yes, this is a bit risky.

06:33:16 S: * 13 FETCH (UID 209666 RFC822.SIZE 67708 BODY[] {67731}
...
06:33:17 Error: Detected mismatch in UIDs received from server (previous value: 209669, new value: 209666).
06:33:17 Ignored mismatch to work around Yahoo server bug. Using previous UID value.
06:33:17 S: G23 NO [UNAVAILABLE] UID FETCH Server error while fetching messages
...

It happens again, message 13 is now mapped to UID 209666 which actually belongs to message 10. MailMate ignores it again, but now the server bails completely with an UNAVAILABLE error. I suspect that something crashed…

At this point I’m thinking that the server might have forgotten to tell MailMate about some deleted messages and in that case the experimental workaround (ignoring the UID mismatch) would be a bad idea, but this is not the case. The retry attempt shortly after reveals that the UIDs are back to “normal”.

06:33:21 S: * 9 FETCH (UID 209665)
...
06:33:21 S: * 13 FETCH (UID 209669)

And while we’re at it. Here’s another example of weird Yahoo IMAP behavior:

06:33:00 C: G3 SELECT "Bulk Mail"
...
06:33:00 S: G3 OK [READ-WRITE] SELECT completed; now in selected state
06:33:00 C: G5 UID FETCH 1:* (UID)
06:33:00 S: * 1 FETCH (UID 209538)
06:33:00 S: * 2 FETCH (UID 209567)
06:33:00 S: * 3 FETCH (UID 209568)
06:33:00 S: * 4 FETCH (UID 209569)
06:33:00 S: * 5 FETCH (UID 209570)
06:33:00 S: * 6 FETCH (UID 209571)
06:33:00 S: G5 OK UID FETCH completed
...
06:33:01 C: G7 UID FETCH 1:209571 (UID FLAGS)
06:33:01 S: * 1 FETCH (FLAGS ($Junk) UID 209538)
06:33:01 S: * 2 FETCH (FLAGS ($Junk) UID 209567)
06:33:01 S: * 3 FETCH (FLAGS ($Junk) UID 209568)
06:33:01 S: * 4 FETCH (FLAGS ($Junk) UID 209569)
06:33:01 S: * 5 FETCH (FLAGS ($Junk) UID 209570)
06:33:01 S: G7 OK UID FETCH completed
06:33:01 C: G8 UID FETCH 209538 (UID BODY.PEEK[] RFC822.SIZE)
06:33:01 S: G8 OK UID FETCH completed

Can you spot the problem? The server claims the existence of 5 messages. When MailMate asks for the data of the first one then nothing is returned. Sigh…

I’ve had various variations of this bug reported by several users and I’m pretty tired of it. Some of these reports also indicate that it’s not always safe to ignore the sudden UID changes (I’ve attempted to improve the workaround to make this less likely to be a problem). In order to better handle future “bug” reports, I’ve written a wiki page such that I can quickly provide users with more information.

I would of course very much like to report these issues to Yahoo, but I cannot find any way to do that. The closest I’ve found is this help page which states:

What if my account is working outside of the app? This means there’s something wrong with the app. You can download the Yahoo Mail app from the app store and use that instead […]

I guess that means I shouldn’t feel bad about recommending users to find a different IMAP provider.

Yahoo, please prove me wrong and fix these issues. Consider this my bug report since I couldn’t find anywhere else to put it.

(I repeat, MailMate is also buggy and I do see the double standard in play here. That said, I’m pretty sure Yahoo is not spending any time implementing workarounds for MailMate bugs…)