Mavericks and Apple Mail

Mavericks (OS X 10.9) has been released and I’ve just read a very interesting article (TidBITS) about how Apple Mail has changed behavior with regard to how it handles Gmail accounts. Apparently this has, so far, not been a great success. Having been through the same implementation problems as the Apple developers I would like to share some of my thoughts on the issue of handling Gmail accounts.

I have a love/hate relationship with Gmail with an emphasis on the latter and I’m sure the same is true for the Apple Mail developers. Both Apple Mail and MailMate have tried to treat Gmail accounts as if they used the IMAP standard, but this is tricky since it is very non-standard — and I only use the term “non-standard”, because Google does this on purpose. Objectively, when they claim to provide IMAP access then the correct wording should be “extremely buggy”.

Nevertheless, Gmail is here to stay and both the Apple Mail developers and I know that we have to do whatever we can to allow Gmail accounts to nicely co-exist with standard IMAP accounts (yes, alternatives to Gmail do exist).

Duplicate messages

The main problem with Gmail is that labels are mapped to IMAP mailboxes. This means that any message with multiple labels is present in multiple IMAP mailboxes. A standards-compliant (offline) email client is going to handle that by fetching the message multiple times. Bandwidth is wasted, space is wasted, and weird things can happen when you move or copy messages. The worst Gmail mailbox is the “[Gmail]/All Mail” mailbox. This contains all messages not located in the trash or spam mailboxes. In MailMate (and previously in Apple Mail) it was best to ignore this mailbox and that is what happens by default in MailMate. If all messages have at least one label then all messages are still available in MailMate. Unfortunately, this is not always the case. I’ll get back to this issue further below.

Ignoring “All Mail” does not solve the problem with messages with multiple labels. Some users never use multiple labels and simply move messages as if the labels were simple mailboxes. This works fine in both Apple Mail (earlier versions) and MailMate, but it’s not a general solution working for all users.

Multiple labels in MailMate

Recently, I implemented a workaround in MailMate for the problem of multiple labels. It is currently an experimental solution and you can enable it following the instructions in the release notes for version 1.7. In short, MailMate can be told to handle specific Gmail labels as if they were tags. If a label is handled as a tag then MailMate automatically ignores the corresponding IMAP mailbox. Tags are already supported by MailMate (no integration with Mavericks yet) and uses IMAP keywords on standard IMAP servers. An added benefit of this is that you can move messages between Gmail accounts and other IMAP accounts and the tags are preserved. This is great if you are planning to migrate…

The “All Mail” problem

The remaining problem in MailMate is the “[Gmail]/All Mail” mailbox. It might contain messages which are not present in any other mailbox. Such messages are not available in MailMate. The only workaround is if the user makes sure that no such messages exist — and this can be tricky when using other email clients than MailMate (such as iOS Mail).

Apple Mail has now “solved” this problem by fetching all these messages and then identify duplicates across mailboxes to make sure they only store each message once. They are essentially trying to make Apple Mail behave like Gmail itself does. Personally, I think this is a dangerous path. Gmail IMAP behavior is non-standard, largely undocumented, and controlled by a single company. In other words, handle with care.

I’ve been bitten by sudden Gmail behavioral changes in the past and I won’t go down the same path as Apple Mail (and other email clients), but I’ve got an idea as to how to solve this remaining problem without major changes in MailMate.

An untested theoretical solution

When fetching a message, Gmail also allows the email client to fetch all the Gmail labels for the message. This would allow MailMate to identify the messages which are present in any other mailbox(es). If the IMAP code treated these messages as non-existing then the “All Mail” mailbox could become a mailbox containing the exact set of messages not present in any other mailboxes. The only minor problem would be that “All Mail” is not really “all mail”, but MailMate already has the “All Messages” virtual mailbox.

Maybe I’ll have time to implement and test this idea in the future. It might require a bit of funding. For now, I just wanted to share my thoughts on the issue.

Update October 27th: I implemented this solution in the latest test version of MailMate. Handle with care: You can fetch the test version by holding down ⌥ when clicking “Check Now” in the Software Update preferences pane. After that you need to subscribe to “[Gmail]/All Mail”. Details are provided in a mailing list message.