A Developer's Slow Descent into Madness: Yahoo IMAP

All software is buggy and it’s always a bit hypocritical for a developer to complain about bugs. I’m certainly aware that MailMate has many bugs and some of them are incredibly frustrating for some users. For those users it might be a comfort to know that an email client developer has to deal with one of the most frustrating types of bugs: IMAP server bugs.

An IMAP server bug can often be characterized as follows:

  • It only affects IMAP email clients and not a webmail client or some proprietary email client for the IMAP service involved.
  • It makes the email client seem like it has a bug.
  • There’s no way to report it.
  • It is probably not hard to fix.
  • It is never fixed.

(Even in the rare case of a fix, old versions of the IMAP server continue to be used.)

Most users (understandably) see no difference between email client bugs and server bugs. The email client takes the blame. Whatever the problem is I must either fix a bug in MailMate or add some workaround for an IMAP server bug.


UPDATE: Well, apparently I owe Yahoo an apology. I reached out to Yahoo on Twitter and I was then contacted by email. Apparently, they fixed some issues last week which they believe fixes the UID mismatch issues described below. I really hope the issue is fixed, but I remain a bit skeptical. The first time a MailMate user reported a UID mismatch issue for Yahoo was more than 2 years ago. But for now: I’m really sorry, Yahoo, for believing that this issue would never be fixed.


Some IMAP servers are worse than others. Today, I’m writing about Yahoo IMAP, but it could just as well be Exchange or some other buggy IMAP implementation. In any case, if you are searching for a new IMAP server then please consider some of the alternatives.

Yahoo IMAP has many problems. The inbox is incorrectly named Inbox instead of INBOX as required by the RFC, it can more or less randomly reject authentication making it seem like the password is incorrect, and it can suddenly become unavailable (not a bug but a stability issue) with various types of temporary errors. Most of these problems are handled gracefully by MailMate. The bug driving me mad is what is best described as “schizophrenic emails” — emails with more than 1 identity.

An IMAP mailbox has two different ways of enumerating emails. There’s a sequential way where emails are numbered 1 to N if the mailbox contains N messages and there’s so-called UIDs which stay the same even when messages are added or removed from the mailbox. The latter is very important for email clients caching emails (which means almost all desktop email clients).

Unfortunately, Yahoo IMAP has a bug where it messes up the mapping between sequence numbers and UIDs. The following is the relevant parts of an example of this happening:

06:32:51 C: C17 SELECT Trash
06:32:51 S: * 1381 EXISTS
...
06:32:51 S: C17 OK [READ-WRITE] SELECT completed; now in selected state
...
06:32:53 C: C21 UID FETCH 1:211045 (UID FLAGS)
06:33:07 S: * 1 FETCH (FLAGS ($Junk) UID 209612)
...
06:33:07 S: * 8 FETCH (FLAGS (\Seen) UID 209619)
06:33:07 S: * 9 FETCH (FLAGS (\Seen $NotJunk) UID 209665)
06:33:07 S: * 10 FETCH (FLAGS (\Seen $NotJunk) UID 209666)
06:33:07 S: * 11 FETCH (FLAGS (\Seen $NotJunk) UID 209667)
06:33:07 S: * 12 FETCH (FLAGS (\Seen $NotJunk) UID 209668)
06:33:07 S: * 13 FETCH (FLAGS (\Seen $NotJunk) UID 209669)
...
06:33:07 S: C21 OK UID FETCH completed

At this point everything is fine. In particular, note that message number 12 has UID 209668.

...
06:33:15 C: G23 UID FETCH 209612:209666 (UID BODY.PEEK[] RFC822.SIZE)
06:33:16 S: * 12 FETCH (UID 209665 RFC822.SIZE 3848 BODY[] {3869}
...
06:33:16 Error: Detected mismatch in UIDs received from server (previous value: 209668, new value: 209665).
06:33:16 Ignored mismatch to work around Yahoo server bug. Using previous UID value.

What? Message number 12 has UID 209665. We can even see that this UID belongs to message number 9. MailMate detects the issue and, as an experimental workaround, MailMate ignores the change and keeps the old UID. Yes, this is a bit risky.

06:33:16 S: * 13 FETCH (UID 209666 RFC822.SIZE 67708 BODY[] {67731}
...
06:33:17 Error: Detected mismatch in UIDs received from server (previous value: 209669, new value: 209666).
06:33:17 Ignored mismatch to work around Yahoo server bug. Using previous UID value.
06:33:17 S: G23 NO [UNAVAILABLE] UID FETCH Server error while fetching messages
...

It happens again, message 13 is now mapped to UID 209666 which actually belongs to message 10. MailMate ignores it again, but now the server bails completely with an UNAVAILABLE error. I suspect that something crashed…

At this point I’m thinking that the server might have forgotten to tell MailMate about some deleted messages and in that case the experimental workaround (ignoring the UID mismatch) would be a bad idea, but this is not the case. The retry attempt shortly after reveals that the UIDs are back to “normal”.

06:33:21 S: * 9 FETCH (UID 209665)
...
06:33:21 S: * 13 FETCH (UID 209669)

And while we’re at it. Here’s another example of weird Yahoo IMAP behavior:

06:33:00 C: G3 SELECT "Bulk Mail"
...
06:33:00 S: G3 OK [READ-WRITE] SELECT completed; now in selected state
06:33:00 C: G5 UID FETCH 1:* (UID)
06:33:00 S: * 1 FETCH (UID 209538)
06:33:00 S: * 2 FETCH (UID 209567)
06:33:00 S: * 3 FETCH (UID 209568)
06:33:00 S: * 4 FETCH (UID 209569)
06:33:00 S: * 5 FETCH (UID 209570)
06:33:00 S: * 6 FETCH (UID 209571)
06:33:00 S: G5 OK UID FETCH completed
...
06:33:01 C: G7 UID FETCH 1:209571 (UID FLAGS)
06:33:01 S: * 1 FETCH (FLAGS ($Junk) UID 209538)
06:33:01 S: * 2 FETCH (FLAGS ($Junk) UID 209567)
06:33:01 S: * 3 FETCH (FLAGS ($Junk) UID 209568)
06:33:01 S: * 4 FETCH (FLAGS ($Junk) UID 209569)
06:33:01 S: * 5 FETCH (FLAGS ($Junk) UID 209570)
06:33:01 S: G7 OK UID FETCH completed
06:33:01 C: G8 UID FETCH 209538 (UID BODY.PEEK[] RFC822.SIZE)
06:33:01 S: G8 OK UID FETCH completed

Can you spot the problem? The server claims the existence of 5 messages. When MailMate asks for the data of the first one then nothing is returned. Sigh…

I’ve had various variations of this bug reported by several users and I’m pretty tired of it. Some of these reports also indicate that it’s not always safe to ignore the sudden UID changes (I’ve attempted to improve the workaround to make this less likely to be a problem). In order to better handle future “bug” reports, I’ve written a wiki page such that I can quickly provide users with more information.

I would of course very much like to report these issues to Yahoo, but I cannot find any way to do that. The closest I’ve found is this help page which states:

What if my account is working outside of the app? This means there’s something wrong with the app. You can download the Yahoo Mail app from the app store and use that instead […]

I guess that means I shouldn’t feel bad about recommending users to find a different IMAP provider.

Yahoo, please prove me wrong and fix these issues. Consider this my bug report since I couldn’t find anywhere else to put it.

(I repeat, MailMate is also buggy and I do see the double standard in play here. That said, I’m pretty sure Yahoo is not spending any time implementing workarounds for MailMate bugs…)


MailMate 1.9.7 Released

MailMate 1.9.7 brings a long list of improvements and fixes as documented in the detailed release notes. Here follows a short list of the most notable highlights:

  • Improved single message window behavior. It is now “connected” to the main window such that undo/redo behavior is better and features (key bindings) such as “Move to Mailbox…” work as expected.
  • The “Apply Rules” menu item is now a menu which allows applying rules to either selected messages or the entire mailbox.
  • “Export ▸ Copy to Folder” command which exports .eml files into a folder hierarchy matching the original accounts. Duplicates are automatically avoided and the command can be used as an action in mailbox rules.
  • Internal changes to provide an action for bundles which allows changing header values of existing emails. This feature is currently used to implement “Command ▸ MailMate ▸ Change Subject”.
  • Email addresses can now be blacklisted such that they are not used for completions in the Composer.
  • Low-level, the message list supports coloring and styling lines. This feature is described in more detail here.
  • Numerous new hidden preferences for users that like to tinker with the details.
  • Manual pages updated to include some of the previously undocumented features.
  • Improved behavior for various problematic IMAP servers including Gmail and Exchange IMAP.
  • Uses the IMAP RENAME command when an IMAP mailbox is renamed (also when a hierarchy is involved). This is both more robust and more efficient than before.
  • Now doing Gmail “OAuth2” authorization through an external browser.
  • Improved composer text view including fixing various display issues.
  • Improved autoscroll in the composer preview
  • Improved and fixed S/MIME handling which now also allows separate certificates for signing/encrypting.
  • Improved behavior for window tabs on Sierra and later macOS versions.
  • Fixed various issues on High Sierra.

ScreenCastsOnline Tutorial for MailMate

ScreenCastsOnline has published a tutorial for MailMate. It’s 45 minutes covering the basics of MailMate. Thanks to Mike Schmitz for taking the time to create it.

Note that ScreenCastsOnline is a membership based service, but you can also see the tutorial as part of a free trial period. See more on the preview page.

Alternatively, there is a ScreenCastsOnline monthly magazine available on the iTunes App Store. This also includes a free trial period, allowing you to download this month’s issue, including the MailMate video tutorial. If you like, you can cancel the free trial after downloading this months issue.


Application Specific Passwords (iCloud Accounts)

A while ago Apple announced that they are going to require that all applications accessing iCloud data (including emails in iCloud email accounts) must do so using a so-called application specific password. This happens a week from now, that is, on June 15.

I’m expecting this to be a surprise for some MailMate users and therefore I’m, preemptively, going to answer what is likely to be frequently asked questions:

  • Can I still use MailMate for my iCloud account(s)?

    In most cases, the answer is yes.

  • MailMate is rejecting my password, what can I do?

    You need to enable two factor authentication and then setup an application specific password for MailMate. Update: The great people at BusyMac made a video which shows how to do this. It’s mostly the same for MailMate.

  • What if two factor authentication is not available to me?

    Sorry, I don’t know. As far as I know Apple has not provided a solution, but we’ll have to see what shows up next Thursday. Maybe some accounts are excluded from the new “rule”. If there is no solution and if your license key is not very old then I’ll refund your purchase.

  • Why don’t you just do like Apple Mail, it works fine with my iCloud account?

    Apple uses an authentication method only available to applications created by Apple. All third party applications are going to require an application specific password if they need to access an iCloud account.

  • But security is much better now, right?

    It’s certainly safer to use two factor authentication when possible, but the current implementation of application specific passwords is a bit of a hack. They are not much better than the main iCloud password since they cannot be limited to access specific parts of iCloud data.

  • Do you think Apple could have done it differently?

    Well, let’s just say I’m sure Apple would have done it differently if the change affected their own applications…

Use “Help ▸ Send Feedback” within MailMate if you have other iCloud related questions or maybe corrections to the above. (I’ll add any frequently asked questions to this blog post.)


MailMate Screencasts (Episodes 3 and 4)

As described in the previous blog post, a very dedicated MailMate user, Matt Petrowsky, has created a series of screencasts about how he uses MailMate. Last week I commented on the first two episodes and today I’ll comment on episodes 3 and 4.

Here’s a link to the entire list of screencasts: MailMate: The Killer Email App for Mac OS.

Note that since last week, Matt also created an additional screencast covering a specific new (minor) feature of MailMate.

Episode 3: Working with Messages (8:12)

I don’t really have much to comment on for this one. It might be interesting to know that the red link indicator that Matt mentions is triggered whenever a textually displayed link address does not match the actual page linked to. For example, it would be triggered by HTML like this:

<a href="http://example.com/potentially_bad_hidden_link">http://example.com/the_displayed_link</a>

Matt applauds the ability to display multiple messages in the message view, but he also notes that it does not work well for HTML messages. I completely agree with that. Ironically, a fix to this problem would very likely break the ability to select the text of all displayed emails in one go.

Episode 4: Key Bindings (9:31)

Matt mentions Xcode, but it’s not really needed for editing custom key bindings. A plain text editor is fine since the property list files used by MailMate are mostly in a readable and easily editable plain text format. (After using the property list editor distributed with Xcode to save the file then this has likely changed to an XML format.)

Another detail is that Matt disables the default Gmail key bindings file. He doesn’t really need to do that. He could have put his additions/changes into a file named, e.g., Matt and then configured key bindings (in the General preferences pane) with a comma separated list of files like this: Matt, Gmail. This is not very important, but it would ensure that changes I do to the default Gmail file are not ignored.

If anything, this screencast illustrates that it would be very nice if MailMate had an embedded editor for changing custom key bindings.

As Matt mentions, the main resource for key bindings is the manual page. I’d like to also emphasize the list of available key binding selectors.

Matt also mentions the powerful HTML features of MailMate. Here I’d like to emphasize that even though MailMate can do a lot of things with HTML then most users are still going to experience that MailMate has a very basic plain text composer. The rationale for this can be found in the manual. I’m primarily mentioning this, because it’s likely to be a deal breaker for many trial users of MailMate.

That’s it for today. Thanks again, Matt!


MailMate Screencasts (Episodes 1 and 2)

One of the MailMate patrons sent me (and you) a special New Years gift. Matt Petrowsky decided to make a series of screencasts about how he uses MailMate. The screencasts are high quality and they cover a lot of aspects of MailMate. It is, by far, the most detailed general visual introduction available for MailMate.

And here’s the big surprise: He made 13 episodes totalling more than 2 hours of video.

I’m simply amazed by this effort and it is greatly appreciated. Thanks Matt!

I should have shared it a month ago, but I first wanted to look through all of it myself to see if I had any important comments — and then a month passed. Shame on me! I’ve now decided to split this task into smaller subtasks. Today, I’m commenting on the first 2 episodes.

Here’s a link to the entire list of screencasts: MailMate: The Killer Email App for Mac OS.

Update February 10th: Matt created an additional video which can be found here.

Note that MailMate has been updated to revision 5344 (it’s version number is still 1.9.6). See the release notes for the quite long list of details.

Episode 1: Introduction (7:39)

Matt previously used the Gmail web interface and this is his starting point. Therefore he didn’t notice that if he had accounts configured in Apple Mail or Thunderbird then MailMate would have automatically triggered “File ▸ Import Accounts…”. It is, by the way, quite a hassle to make this work since there is no standard way to obtain settings from other email clients and Apple, in particular, keeps on changing how they save account settings. This is not a complaint, but it helps explain why this feature tends to be broken on a regular basis.

My favorite quote of the introduction is that “MailMate is an application that you really must configure based on how you work”. That might also be a nice way to say that MailMate has a somewhat steep learning curve.

Episode 2: Account Setup (7:29)

Matt compares MailMate with the Gmail web interface which also makes this a nice introduction to the differences between webmail and desktop email clients. Desktop email clients use IMAP, POP3, or Exchange to access emails. MailMate only supports IMAP (and Exchange when IMAP access is enabled on the server).

It is clearly described how to enable IMAP access for a Gmail account, but I’m pretty sure this is actually not necessary. I tried to disable IMAP, but I was still able to access my Gmail account from MailMate. I believe this is because the setting is ignored when using OAuth2 to authenticate an account. The OAuth2 setting is also described by Matt and this is certainly recommended for Gmail accounts, because otherwise all kinds of issues may arise. My thoughts on OAuth2 for desktop email clients can be found here.

There’s a very nice description of using multiple email addresses for a single IMAP account. With respect to Gmail you need to be careful though. If Google does not like the “From” address of an email (when using their SMTP server) then it doesn’t reject the email, it just rewrites the “From” header. I cannot find this documented anywhere, but Google describes how to add additional email addresses on this support page.

Some minor comments:

  • Never ever disable SSL. This setting only exists for the few users needing it for, e.g., a local IMAP server, but I’ve considered hiding it by default. If you have an email provider which does not support SSL (or does not enforce it) then it’s time to consider an alternative.
  • Matt is nice to say that it’s really easy to add an account. Just for the record, I do think it could be a lot easier.

That’s it for now. Thanks again Matt!


Personalized Subscriber Emails

To mark the end of 2016 I sent out an email to all those subscribing to be notified about news about MailMate (notice the “Keep Me Posted” text field available on this page). This is the first time I’ve done this even though it’s almost 2 years ago that the first people registered for these emails.

I created the emails using MailMate with the use of some still undocumented and incomplete features. I created a draft in MailMate with some template values and then used a bundle command to create a personalized email for each subscriber in my database. The personalized parts were the recipient address, the month/year of subscribing to the emails, and the optional comment/request registered with the address. The last part was only included if I had a reply/comment to that comment. Yes, some of those replies are coming with almost 2 years of a delay. There are probably better (and certainly faster) ways to create such emails, but it was an interesting learning experience.

You can see the raw message as I wrote it here and you can see a generated one here. It’s mainly a review of what is new in MailMate in 2016.

I also wrote a private “thank you” email to all of the MailMate patrons in a similar way. If you are a registered user then you can learn more about being a patron using the “MailMate ▸ Registration ▸ Become a MailMate Patron…” menu item.

It was interesting going through all the subscriber comments which were mostly feature requests. In particular, some things were requested more often than others. These could be divided into two groups:

  • Requests I really would like to implement/do, but I keep on postponing:
    • A “simpler” 1-level threading mode, e.g., grouping all emails belonging to the same thread, optionally based on the subject of the messages.
    • Message list with multiline entries (previewing message text).
    • Correspondence view (multiple emails in a single message view).
    • Documentation for the creation of custom bundles.
  • Requests I’m not going to implement:
    • A WYSIWYG HTML editor.
    • Local (“On My Mac”) mailboxes and/or POP3 support.
    • Native Exchange support.

Somewhere in between is variations of “make MailMate prettier”, but that’s subjective and I don’t intentionally make MailMate ugly. I do think it got a little prettier in 2016, but I admit that I’m usually more concerned about the look (and adherence to standards) of the emails that MailMate sends (something which seems to be a bit unorthodox among email client developers).

Finally, there are always requests for MailMate 2.0 and this was my standard answer: “Don’t put too much into version numbers. Version 1.9.6 is more than I originally expected 2.0 to be. Also, MailMate is iteratively updated. Bumping to 2.0 would just mean version 3.0 would be getting closer (which would be a paid upgrade).”

The release notes is always the best resource for figuring out exactly what has changed in MailMate.

Happy New Year!


MailMate 1.9.6 Released

MailMate 1.9.6 is primarily a maintenance release. Here are the most notable items of the much longer and detailed release notes:

  • New: “Edit ▸ Add Link” menu item (⌘K). Inserts Markdown code to create a link based on the current word. Also inserts a URL if it’s on the pasteboard.
  • New: URL scheme for doing toolbar-like searches: mlmt:quicksearch?string=s%20mlmt
  • New: “New Message With Attachment” and “Copy Attachment” in the context sensitive menu for an attachment.
  • New: MmSMTPFixedHostname
  • Changed: Generation of the HTML alternative for plain text (non-Markdown) messages is now handled using a dedicated script. For compatibility reasons this uses whitespace:normal and <p> instead of <div>. This should resolve various issues when corresponding with Outlook users.
  • Changed: Undoing move actions now re-selects messages if they exist in the current mailbox.
  • Fixed: Saving a search as a smart mailbox frequently failed.
  • Fixed: Various issues with HTML embedding when forwarding HTML messages with attachments.
  • Fixed: Issue with the choice of Drafts mailbox (and implicitly SMTP server) when an identity could not be derived and an explicit one had not been configured.
  • Fixed: Drag’n’drop emails to the Finder on Sierra.

MailMate 1.9.5 Released

It’s a minor version bump, but it’s not a minor update. MailMate 1.9.5 is a major update as documented in the detailed release notes. Here follows a short list of the most notable highlights:

  • Embed arbitrary HTML when replying/forwarding.
  • MailMate icon replaced (designed by Eli Schiff).
  • Toolbar icons added (designed by Eli Schiff).
  • Style outgoing messages using CSS.
  • Code styling using Pygments.
  • Math styling using ASCIIMath or TeXMath (converted to MathML which is not supported by all recipient email clients).
  • Numerous additions to the Composer preferences pane to control the HTML related features above.
  • Fixed various issues with macOS 10.12 (Sierra).
  • Support for tags and colored flags in the toolbar.
  • “Format ▸ Bold/Italic” menu items which inserts/removes Markdown syntax to emphasize text.
  • Network code now uses CFNetwork instead of OpenSSL. This implicitly means proxy support (System Preferences), IPv6 support, and TLS 1.2 support.
  • Redesigned headers view.
  • Fixed an issue with sleep mode and/or being offline. MailMate failed to send all pending messages in Drafts (until relaunching MailMate).

If you are a happy MailMate user then also note this addition:

  • Added “MailMate ▸ Registration ▸ Become a MailMate Patron…” menu item (only enabled for registered users).

MailMate 1.9.4 Released

This release includes a lot of small improvements and bug fixes. The most interesting changes are various new hidden preferences, including MmSMTPAlternativeEnabled and MmSMTPUTF8Enabled. More information in the manual and in the detailed release notes.