Mavericks, Gmail, Apple Mail, and MailMate

Mavericks and Apple Mail

Mavericks (OS X 10.9) has been released and I’ve just read a very interesting article (TidBITS) about how Apple Mail has changed behavior with regard to how it handles Gmail accounts. Apparently this has, so far, not been a great success. Having been through the same implementation problems as the Apple developers I would like to share some of my thoughts on the issue of handling Gmail accounts.

I have a love/hate relationship with Gmail with an emphasis on the latter and I’m sure the same is true for the Apple Mail developers. Both Apple Mail and MailMate have tried to treat Gmail accounts as if they used the IMAP standard, but this is tricky since it is very non-standard — and I only use the term “non-standard”, because Google does this on purpose. Objectively, when they claim to provide IMAP access then the correct wording should be “extremely buggy”.

Nevertheless, Gmail is here to stay and both the Apple Mail developers and I know that we have to do whatever we can to allow Gmail accounts to nicely co-exist with standard IMAP accounts (yes, alternatives to Gmail do exist).

Duplicate messages

The main problem with Gmail is that labels are mapped to IMAP mailboxes. This means that any message with multiple labels is present in multiple IMAP mailboxes. A standards-compliant (offline) email client is going to handle that by fetching the message multiple times. Bandwidth is wasted, space is wasted, and weird things can happen when you move or copy messages. The worst Gmail mailbox is the “[Gmail]/All Mail” mailbox. This contains all messages not located in the trash or spam mailboxes. In MailMate (and previously in Apple Mail) it was best to ignore this mailbox and that is what happens by default in MailMate. If all messages have at least one label then all messages are still available in MailMate. Unfortunately, this is not always the case. I’ll get back to this issue further below.

Ignoring “All Mail” does not solve the problem with messages with multiple labels. Some users never use multiple labels and simply move messages as if the labels were simple mailboxes. This works fine in both Apple Mail (earlier versions) and MailMate, but it’s not a general solution working for all users.

Multiple labels in MailMate

Recently, I implemented a workaround in MailMate for the problem of multiple labels. It is currently an experimental solution and you can enable it following the instructions in the release notes for version 1.7. In short, MailMate can be told to handle specific Gmail labels as if they were tags. If a label is handled as a tag then MailMate automatically ignores the corresponding IMAP mailbox. Tags are already supported by MailMate (no integration with Mavericks yet) and uses IMAP keywords on standard IMAP servers. An added benefit of this is that you can move messages between Gmail accounts and other IMAP accounts and the tags are preserved. This is great if you are planning to migrate…

The “All Mail” problem

The remaining problem in MailMate is the “[Gmail]/All Mail” mailbox. It might contain messages which are not present in any other mailbox. Such messages are not available in MailMate. The only workaround is if the user makes sure that no such messages exist — and this can be tricky when using other email clients than MailMate (such as iOS Mail).

Apple Mail has now “solved” this problem by fetching all these messages and then identify duplicates across mailboxes to make sure they only store each message once. They are essentially trying to make Apple Mail behave like Gmail itself does. Personally, I think this is a dangerous path. Gmail IMAP behavior is non-standard, largely undocumented, and controlled by a single company. In other words, handle with care.

I’ve been bitten by sudden Gmail behavioral changes in the past and I won’t go down the same path as Apple Mail (and other email clients), but I’ve got an idea as to how to solve this remaining problem without major changes in MailMate.

An untested theoretical solution

When fetching a message, Gmail also allows the email client to fetch all the Gmail labels for the message. This would allow MailMate to identify the messages which are present in any other mailbox(es). If the IMAP code treated these messages as non-existing then the “All Mail” mailbox could become a mailbox containing the exact set of messages not present in any other mailboxes. The only minor problem would be that “All Mail” is not really “all mail”, but MailMate already has the “All Messages” virtual mailbox.

Maybe I’ll have time to implement and test this idea in the future. It might require a bit of funding. For now, I just wanted to share my thoughts on the issue.

Update October 27th: I implemented this solution in the latest test version of MailMate. Handle with care: You can fetch the test version by holding down ⌥ when clicking “Check Now” in the Software Update preferences pane. After that you need to subscribe to “[Gmail]/All Mail”. Details are provided in a mailing list message.

4 comments.

  1. I too have a love/hate relationship with Google. I love their services but just cannot seem to adjust my workflow to their Web client and non-standard IMAP implementation. I keep wishing they would offer a very standard IMAP implementation; however, this is highly unlikely to ever happen. I therefore make very limited use of my Google Apps account for email, only forwarding a copy of some of my regular messages to Google when I travel. All this being said, I should confess that I have not tried your current experimental tag work-around, but I think it makes more sense that your suggested theoretical solution. If I were to use MailMate for my Google Apps mail, I know that I would want the ability to move messages between Google and non-Google accounts. It’s not clear to me by your description of the theoretical solution whether or not this would be possible.

  2. What’s your opinion on Fastmail’s implementation of IMAP?

  3. Fastmail runs Cyrus IMAP, one of (if not) the premier protocol implementations. They also contribute significantly to the project.

  4. @Scott: I think you misunderstood my theoretical solution. It was only to handle the remaining problem of the “[Gmail]/All Mail” folder. Labels would still be handled as tags (if told to do so by the user).

    @Jeff: I agree with the reply from Andrew. MailMate has no problems with Fastmail. (My personal favorite IMAP server is Dovecot.)

Post a comment.

Crowd Funding Campaign

Reviews

Macworld wrote:
  • As email search abilities go, this seems less like bringing a gun to a knife fight, and more like thundering into that particular duel at the controls of a helicopter gunship.
  • Needless to say, the program never crashed, glitched, or gave me any sort of trouble during my tests. MailMate does not know weakness.
About.com wrote:
  • MailMate is a refreshingly efficient way to deal with mail in IMAP accounts, all with handy keyboard shortcuts, superior search and precise smart folders.
  • As a testament to its raw power, MailMate does not stop at From: or body text, of course; instead, you can have it match just names or first names, email addresses, domain names, parts of domain names, quoted text and what not.
Lifehacker wrote:
  • Search is amazing, providing you with a ridiculous amount of control.
  • Practically everything can be accessed using the keyboard.

User Quotes

Rob Schumann (MacUpdate) wrote:
  • I […] felt immediately at home with its much more capable filtering system that can be directed at any folder, not just the inbox, and the ability to construct complex any/all rules that can be nested multiple levels deep.
  • […] in general it just feels significantly faster than Mail.app
David Levy (email) wrote:
  • I've been stress testing MailMate with a variable and highly intermittent connection. […] My overall assessment is that MailMate has been markedly superior [to Apple Mail] in this use modality […]
Tom Borowski (blog) wrote:
  • […]this way of handling signatures is far superior to any other approach I’ve seen so far. Why hardly anyone implements it in this way is a mystery to me.
TheBrix2008 (MacUpdate) wrote:
  • […] on first run, MailMate spotted my Mail.app IMAP settings, prompted for import, then got to work without fuss.
  • it has a phenomenal array of filtering, statistics and views […] My favourite is “Correspondence”, which picks out all email from your current correspondent and lists it in a separate pane… instantly.
  • The author should be rewarded for having the courage to start from scratch.