MailMate 1.11.1 Released

This is mainly a maintenance release. Details are in the release notes.

The release is particularly interesting for users preferring the original IMAP approach to deleting emails, that is, marking emails as deleted and then expunging them (permanently) at a later time:

  • New: Hidden preference MmDeleteBehavior: moveToDeletedMessages, markAsDeleted, expunge, archive.
  • New: Hidden preference MmAutomaticExpungeBehavior: mailboxSwitch, never.
  • New: An Expunge toolbar button (which reuses the trash icon).
  • New: Accounts can be given their own deleteBehavior overriding the global default.

MailMate 1.11 Released

The MailMate 1.11 release has many new features, improvements, and fixes. As always all the details can be found in the release notes. I’m particularly happy about some non-visible “under the hood” changes which are going to make it easier for me to work on new features and optimizations in the future.

Here follows a short list of the most notable highlights:

  • New: The conditions pane (also in rules) now have a “Message is” menu and a “Message is not” menu. These have submenus to easily match the “read” state, (colored) flags, and various other frequently used search conditions. It is not new that these searches can be made, but they should be much easier to do now.
  • New: GUI setting for automatically adding trusted S/MIME certificates to the keychain. This includes telling the user when it happens and warning if a certificate already exists for the same email address.
  • New: A dedicated “Two Panes” layout with no message preview (see the “View ▸ Layout” menu).
  • New: The context sensitive menu of an IMAP account displays the exact storage quota usage and limit. If multiple so-called quota roots exist then they are shown in a submenu. This only works for IMAP servers with QUOTA support as described in RFC2087.
  • New: The MmQuotaWarningLevel hidden preference can be used to control when the used quota percentage is shown for IMAP accounts in the mailbox list itself. The default is 80.0.
  • New: Forwarding (as an attachment) is now available as a rule action.
  • Changed: Major refactoring of all so-called rule editors (mailboxes pane, conditions and rule actions). This should make it much easier to add features to these in the future.
  • Changed: Major refactoring of the internal query system (used by smart mailboxes and many other parts of MailMate). This includes bug fixes and simpler (more robust) code.
  • Changed: Improved support for tags in the Composer window (including a toolbar popup button).
  • Fixed: Several security related issues.
  • And much more…

Developer Documentation for Bundles

MailMate has a very flexible plugin-system available via so-called bundles. There’s already a large set of bundles available, for example, for various task managers and text editors, but you can also create your own bundles with your own personal features.

Bundles have existed for MailMate for a long time, but the creation of bundles has been largely undocumented. This changed recently when I created a documentation page for bundles. This should make it easier to create personal bundles, but you are also very welcome to create bundles to be shared with other MailMate users. Companies with multiple MailMate users might also want to create their own company-wide bundles which could, for example, integrate with local issue tracking systems or provide easy creation of standard replies. If there’s something you would like to do which is not possible using bundles then let me know. I’d like to make it even more flexible than it already is.

Here’s a quote from the documentation which is related to the work MailMate has to do to help you avoid having to know anything about how to parse an email:

Parsing emails is really hard. You need to handle MIME parts, encrypted parts, various encodings like quoted-printable, base64, format=flowed, header encoded words, and various non- or semi-standard formats like tnef (winmail.dat). But much worse, you also have to work around numerous bugs and legacy types of behavior. Ideally, you need to be able to parse any email created by any version of any email client since the early 1970s.

If MailMate only allowed you to get the raw data of an email then it would be very hard to do anything useful and it would be close to impossible to make it robust. Fortunately, MailMate allows you to get the data you need after it has been decoded and after MailMate has worked around all the bugs and legacy stuff of the past.

On a more philosophical note, the main goal of MailMate is to provide input which takes care of as many of the existing email intricacies as possible before handing over data to commands or other parts of the interface. The many problems concerning the conversion of headers and bodies to any kind of canonical data should be handled by MailMate to keep everything else as simple as possible. In other words, canonicalization is to be MailMate’s side of the fence.

Implicitly, this also means that if you have an email which MailMate cannot parse as you would expect it to do so (even if it’s a buggy one) then let me know and I’ll look into yet another workaround.


MailMate 1.10 Released (and Some Thoughts on Version Numbers)

If you are like me then the version number, 1.10, feels a bit weird because it’s not just single digits. That doesn’t make any sense, but it’s actually the only reason that MailMate has been using 1.9.x version numbers since March 2015. It took longer to get from version 1.9 to 1.10 than it took to get from 1.5 to 1.9. Theoretically, this shouldn’t matter much, but in practice small changes of the version number means that updates are going to be considered to be minor updates. Unfortunately, this also means less publicity when updates are released.

You might wonder why I don’t just bump it to 2.0, but I’m saving that for when it really feels like 2.0 to me. That might also not make sense and, who knows, maybe that’ll never happen. Just remember, it’s just numbers.

The following are some of the notable items when comparing 1.10 with the first 1.9.7 release. Some of them were part of a public release which did not involve a version bump at all (r5425). You’ll need to dig into the release notes to get the full story, in particular, if you haven’t tried MailMate since version 1.9.

  • New: Improved handling of a spoofed email address in the name part of the “From” header. Also fixed bugs related to mailsploit.
  • New: Hidden preference to always launch in an offline state: MmInitialOfflineStateEnabled
  • New: Key binding for toggling all accounts offline/online: toggleOnlineStateOfAllAccounts:
  • New: Hidden preference for sending without using the X-Mailer header: MmStripXMailerBeforeSending
  • New: Bundle commands can request attachments (or any other message parts) to be provided as files.
  • New: Bundle commands can return attachments to be included in created emails (both new ones and replies).
  • New: Bundle commands can return actions in the JSON format.
  • New: The primary dock badge counter now looks the same as in other applications.
  • New: Hidden preference MmCommandsIncludeCollapsedMessages which changes the behavior when running a command on collapsed message threads. This is likely to become default behavior.
  • New: Hidden preference MmOpenMessageURLInContext to tell MailMate to handle message: URLs by opening a window with the IMAP mailbox of the message selected.
  • Changed: Added script for more easily creating a new bundle (SharedSupport/bin/create_bundle).
  • Changed: Collapsed thread still shows attachment icon in message list if any attachments exist in the thread.
  • Changed: An outlined flag is now shown when a collapsed thread contains one or more flagged messages.
  • Changed: Optimized some date related computations (this should make date-based smart mailboxes much faster).
  • Changed: Undo/Redo now works for attachments added in the Composer.
  • Changed: Added a warning sheet when pasting in the Composer results in attaching one or more files.
  • Changed: Disabled dragging attachment file names on High Sierra since I cannot make it work correctly. Drag the icon instead.
  • Fixed: Some layout and coloring issues in the mailbox editor (High Sierra).
  • Fixed: Group names (from Contacts) in the Composer were forgotten if closing/sending without first hitting return.
  • Fixed: Some issues with auto-scrolling the preview related to code/math segments.
  • Fixed: Spotlight and the custom location feature now use relative paths. The previous approach would lead to problems if the username of the account was changed.
  • Fixed: Some issues with the Spotlight importer and the sharing extension on High Sierra.
  • Fixed: MmOpenMessageURLInContext also works when opening .eml files.
  • Fixed: The sharing extension is now a bit better at handling text input, e.g., when sharing a note in the Notes app. (Only plain text handled for now.)
  • Fixed: Enabling/disabling some types of mailbox/search conditions would clear the text field or fill it with some old value.

Email Spoofing

A few days ago several MailMate users made me aware of the mailsploit.com site. The site describes a type of phishing attack known as email spoofing and it provides some examples which makes numerous email clients1 fail to display the real email address of the sender — as given in the “From:” header of the email.

If you are in a hurry then all you need to know is that MailMate did fail on some of these examples (although for different reasons than other email clients) and that I have fixed it in the latest test release of MailMate. Hold down ⌥ when clicking “Check Now” in the Software Update preferences pane to try it out. You should get revision 5441 or newer. Note that revision 5440 also fixes the issue, but in practice this doesn’t help much in terms of making spoofing harder. More on that below if you have time for some more details.

The technical part of an exploit based on “mailsploit” is described well at the web site. In short, it tricks some email clients into finding the wrong email address within an email address header like “From”. The email client would then display the wrong sender. The definition of “wrong” here is based on RFC5322.

It is important to understand that spoofing a “From” header has always been easy and, in my opinion, it is still easy. One attempt to change this is DMARC and this is where “mailsploit” shows that such an attempt is doomed to fail if email clients are not improved.

After I fixed the “mailsploit” related bugs in MailMate I realized that it didn’t really fix a more basic problem in MailMate. This is because many parts of the interface only display the name part of an address header. The email address is only shown if there is no name part. This makes it easy to spoof a sender like this:

From: "potus@whitehouse.gov" <evil@example.com>

You can even make it more believable like this:

From: "Donald Trump <potus@whitehouse.gov>" <evil@example.com>

In MailMate, you’ll only see the first part in the message list “From” column, but both are shown in the message view (although not in the compact headers mode). The same is true for both Apple Mail and Gmail (I didn’t check any other email clients). Apple Mail won’t even show you the real address in the message view unless you open a context sensitive menu.

Nothing here is new, but most email client users might not be aware of how little they should trust a “From” header. Bugs or no bugs.

In the most recent test release of MailMate I’ve added the following improvement: Whenever the name part of an address header contains a @ then it’s replaced with a skull (💀). That should at least make the user aware of simple attempts to spoof an address header.

  1. The reason so many email clients have parser bugs is likely because it’s ridiculously hard to parse an email address header. An entire book could be written on this subject. Once again I’ll emphasize that an email client should be able to parse any email generated by any (buggy) version of any email client/generator ever used. 


A Developer's Slow Descent into Madness: Exchange IMAP

I recently wrote about a frustrating issue with Yahoo IMAP and I was quite surprised (and a bit embarrassed) when I actually got into contact with someone at Yahoo working on fixing it. Now I’m thinking it would be nice if I could get the same result if I write about another frustrating server issue.

But first I’ll have to quote myself from my previous blog post:

All software is buggy and it’s always a bit hypocritical for a developer to complain about bugs. I’m certainly aware that MailMate has many bugs and some of them are incredibly frustrating for some users. For those users it might be a comfort to know that an email client developer has to deal with one of the most frustrating types of bugs: IMAP server bugs.

The target of this rant is Exchange IMAP which has an annoying habit of rejecting uploaded messages without any information about why it happens. In the past I’ve seen this happen for emails with one or more very long lines. In that case, the error message is:

BAD Command Error. 10

I was able to work around it by detecting this particular response and then re-encoding its MIME parts before retrying the upload. The subject of this blog post is a slightly different error message:

BAD Command Argument Error. 11

A MailMate user reported this issue when trying to upload a sent message (created by MailMate). This message included another message forwarded as an attachment. Fortunately, I was able to reproduce the issue with an outlook.com account and I could start debugging. I simplified the message step by step and in the end it appears to require two things to happen:

  • A message including a message forwarded as an attachment.
  • The attachment includes an encoded header line which Exchange cannot parse.

Here’s an example:

From: Test@example.com
To: Test@example.com
Subject: Test message pushing Exchange to fail
Message-ID: <54259AAF-8FE0-44A4-8DCE-032F020A7C13@example.com>
Content-Type: multipart/mixed; boundary="=_MailMate_1C634E1D-F7F7-4E44-BD38-1E95A63DE5B6_="


--=_MailMate_1C634E1D-F7F7-4E44-BD38-1E95A63DE5B6_=
Content-Type: message/rfc822

Subject: =?UNKNOWN?Q?n=E9cessaire?=

Body of forwarded message.

--=_MailMate_1C634E1D-F7F7-4E44-BD38-1E95A63DE5B6_=--

The UNKNOWN charset used in the “Subject” line doesn’t make sense and was probably introduced or caused by some other buggy email software. It’s a minor issue, but apparently it’s a major issue for the Exchange IMAP implementation.

After spending a lot of time to track down the source of the problem, I now have to figure out how to work around it. You might wonder why a workaround is needed at all since the bug is clearly in the server implementation, but if I chose this path for every IMAP server bug encountered then many MailMate users would quickly consider MailMate to be unstable. Most users just need things to work. Furthermore, users might not have the option of switching to a more robust IMAP server.

The workaround I chose was to let MailMate re-encode all header lines in all MIME parts of the email before retrying the upload. This only happens for the uploaded copy. The worst part of this is that the user is not notified about the problem. (Some day I would like to add a feature which allows the user to review all the workarounds taking place while using MailMate.)

Now, I’d like to quote my previous blog post again:

An IMAP server bug can often be characterized as follows:

  • It only affects IMAP email clients and not a webmail client or some proprietary email client for the IMAP service involved.
  • It makes the email client seem like it has a bug.
  • There’s no way to report it.
  • It is probably not hard to fix.
  • It is never fixed.

Yahoo proved me wrong on the above. It’ll be interesting to see if Microsoft can do the same. In practice (for me), it doesn’t matter much because Microsoft is not in control of all installations of Exchange IMAP. Even if a bug is fixed then servers with the bug will exist many years into the future.

Some times I like to state that an email client must be able to handle any email ever generated by any kind of email software at any time of the history of emails. I should probably add that it should also be able to work around any bug that exists in any live IMAP server.

Oh, and I promise that my next blog post won’t be rant :)


A Developer's Slow Descent into Madness: Yahoo IMAP

All software is buggy and it’s always a bit hypocritical for a developer to complain about bugs. I’m certainly aware that MailMate has many bugs and some of them are incredibly frustrating for some users. For those users it might be a comfort to know that an email client developer has to deal with one of the most frustrating types of bugs: IMAP server bugs.

An IMAP server bug can often be characterized as follows:

  • It only affects IMAP email clients and not a webmail client or some proprietary email client for the IMAP service involved.
  • It makes the email client seem like it has a bug.
  • There’s no way to report it.
  • It is probably not hard to fix.
  • It is never fixed.

(Even in the rare case of a fix, old versions of the IMAP server continue to be used.)

Most users (understandably) see no difference between email client bugs and server bugs. The email client takes the blame. Whatever the problem is I must either fix a bug in MailMate or add some workaround for an IMAP server bug.


UPDATE: Well, apparently I owe Yahoo an apology. I reached out to Yahoo on Twitter and I was then contacted by email. Apparently, they fixed some issues last week which they believe fixes the UID mismatch issues described below. I really hope the issue is fixed, but I remain a bit skeptical. The first time a MailMate user reported a UID mismatch issue for Yahoo was more than 2 years ago. But for now: I’m really sorry, Yahoo, for believing that this issue would never be fixed.


Some IMAP servers are worse than others. Today, I’m writing about Yahoo IMAP, but it could just as well be Exchange or some other buggy IMAP implementation. In any case, if you are searching for a new IMAP server then please consider some of the alternatives.

Yahoo IMAP has many problems. The inbox is incorrectly named Inbox instead of INBOX as required by the RFC, it can more or less randomly reject authentication making it seem like the password is incorrect, and it can suddenly become unavailable (not a bug but a stability issue) with various types of temporary errors. Most of these problems are handled gracefully by MailMate. The bug driving me mad is what is best described as “schizophrenic emails” — emails with more than 1 identity.

An IMAP mailbox has two different ways of enumerating emails. There’s a sequential way where emails are numbered 1 to N if the mailbox contains N messages and there’s so-called UIDs which stay the same even when messages are added or removed from the mailbox. The latter is very important for email clients caching emails (which means almost all desktop email clients).

Unfortunately, Yahoo IMAP has a bug where it messes up the mapping between sequence numbers and UIDs. The following is the relevant parts of an example of this happening:

06:32:51 C: C17 SELECT Trash
06:32:51 S: * 1381 EXISTS
...
06:32:51 S: C17 OK [READ-WRITE] SELECT completed; now in selected state
...
06:32:53 C: C21 UID FETCH 1:211045 (UID FLAGS)
06:33:07 S: * 1 FETCH (FLAGS ($Junk) UID 209612)
...
06:33:07 S: * 8 FETCH (FLAGS (\Seen) UID 209619)
06:33:07 S: * 9 FETCH (FLAGS (\Seen $NotJunk) UID 209665)
06:33:07 S: * 10 FETCH (FLAGS (\Seen $NotJunk) UID 209666)
06:33:07 S: * 11 FETCH (FLAGS (\Seen $NotJunk) UID 209667)
06:33:07 S: * 12 FETCH (FLAGS (\Seen $NotJunk) UID 209668)
06:33:07 S: * 13 FETCH (FLAGS (\Seen $NotJunk) UID 209669)
...
06:33:07 S: C21 OK UID FETCH completed

At this point everything is fine. In particular, note that message number 12 has UID 209668.

...
06:33:15 C: G23 UID FETCH 209612:209666 (UID BODY.PEEK[] RFC822.SIZE)
06:33:16 S: * 12 FETCH (UID 209665 RFC822.SIZE 3848 BODY[] {3869}
...
06:33:16 Error: Detected mismatch in UIDs received from server (previous value: 209668, new value: 209665).
06:33:16 Ignored mismatch to work around Yahoo server bug. Using previous UID value.

What? Message number 12 has UID 209665. We can even see that this UID belongs to message number 9. MailMate detects the issue and, as an experimental workaround, MailMate ignores the change and keeps the old UID. Yes, this is a bit risky.

06:33:16 S: * 13 FETCH (UID 209666 RFC822.SIZE 67708 BODY[] {67731}
...
06:33:17 Error: Detected mismatch in UIDs received from server (previous value: 209669, new value: 209666).
06:33:17 Ignored mismatch to work around Yahoo server bug. Using previous UID value.
06:33:17 S: G23 NO [UNAVAILABLE] UID FETCH Server error while fetching messages
...

It happens again, message 13 is now mapped to UID 209666 which actually belongs to message 10. MailMate ignores it again, but now the server bails completely with an UNAVAILABLE error. I suspect that something crashed…

At this point I’m thinking that the server might have forgotten to tell MailMate about some deleted messages and in that case the experimental workaround (ignoring the UID mismatch) would be a bad idea, but this is not the case. The retry attempt shortly after reveals that the UIDs are back to “normal”.

06:33:21 S: * 9 FETCH (UID 209665)
...
06:33:21 S: * 13 FETCH (UID 209669)

And while we’re at it. Here’s another example of weird Yahoo IMAP behavior:

06:33:00 C: G3 SELECT "Bulk Mail"
...
06:33:00 S: G3 OK [READ-WRITE] SELECT completed; now in selected state
06:33:00 C: G5 UID FETCH 1:* (UID)
06:33:00 S: * 1 FETCH (UID 209538)
06:33:00 S: * 2 FETCH (UID 209567)
06:33:00 S: * 3 FETCH (UID 209568)
06:33:00 S: * 4 FETCH (UID 209569)
06:33:00 S: * 5 FETCH (UID 209570)
06:33:00 S: * 6 FETCH (UID 209571)
06:33:00 S: G5 OK UID FETCH completed
...
06:33:01 C: G7 UID FETCH 1:209571 (UID FLAGS)
06:33:01 S: * 1 FETCH (FLAGS ($Junk) UID 209538)
06:33:01 S: * 2 FETCH (FLAGS ($Junk) UID 209567)
06:33:01 S: * 3 FETCH (FLAGS ($Junk) UID 209568)
06:33:01 S: * 4 FETCH (FLAGS ($Junk) UID 209569)
06:33:01 S: * 5 FETCH (FLAGS ($Junk) UID 209570)
06:33:01 S: G7 OK UID FETCH completed
06:33:01 C: G8 UID FETCH 209538 (UID BODY.PEEK[] RFC822.SIZE)
06:33:01 S: G8 OK UID FETCH completed

Can you spot the problem? The server claims the existence of 5 messages. When MailMate asks for the data of the first one then nothing is returned. Sigh…

I’ve had various variations of this bug reported by several users and I’m pretty tired of it. Some of these reports also indicate that it’s not always safe to ignore the sudden UID changes (I’ve attempted to improve the workaround to make this less likely to be a problem). In order to better handle future “bug” reports, I’ve written a wiki page such that I can quickly provide users with more information.

I would of course very much like to report these issues to Yahoo, but I cannot find any way to do that. The closest I’ve found is this help page which states:

What if my account is working outside of the app? This means there’s something wrong with the app. You can download the Yahoo Mail app from the app store and use that instead […]

I guess that means I shouldn’t feel bad about recommending users to find a different IMAP provider.

Yahoo, please prove me wrong and fix these issues. Consider this my bug report since I couldn’t find anywhere else to put it.

(I repeat, MailMate is also buggy and I do see the double standard in play here. That said, I’m pretty sure Yahoo is not spending any time implementing workarounds for MailMate bugs…)


MailMate 1.9.7 Released

MailMate 1.9.7 brings a long list of improvements and fixes as documented in the detailed release notes. Here follows a short list of the most notable highlights:

  • Improved single message window behavior. It is now “connected” to the main window such that undo/redo behavior is better and features (key bindings) such as “Move to Mailbox…” work as expected.
  • The “Apply Rules” menu item is now a menu which allows applying rules to either selected messages or the entire mailbox.
  • “Export ▸ Copy to Folder” command which exports .eml files into a folder hierarchy matching the original accounts. Duplicates are automatically avoided and the command can be used as an action in mailbox rules.
  • Internal changes to provide an action for bundles which allows changing header values of existing emails. This feature is currently used to implement “Command ▸ MailMate ▸ Change Subject”.
  • Email addresses can now be blacklisted such that they are not used for completions in the Composer.
  • Low-level, the message list supports coloring and styling lines. This feature is described in more detail here.
  • Numerous new hidden preferences for users that like to tinker with the details.
  • Manual pages updated to include some of the previously undocumented features.
  • Improved behavior for various problematic IMAP servers including Gmail and Exchange IMAP.
  • Uses the IMAP RENAME command when an IMAP mailbox is renamed (also when a hierarchy is involved). This is both more robust and more efficient than before.
  • Now doing Gmail “OAuth2” authorization through an external browser.
  • Improved composer text view including fixing various display issues.
  • Improved autoscroll in the composer preview
  • Improved and fixed S/MIME handling which now also allows separate certificates for signing/encrypting.
  • Improved behavior for window tabs on Sierra and later macOS versions.
  • Fixed various issues on High Sierra.

ScreenCastsOnline Tutorial for MailMate

ScreenCastsOnline has published a tutorial for MailMate. It’s 45 minutes covering the basics of MailMate. Thanks to Mike Schmitz for taking the time to create it.

Note that ScreenCastsOnline is a membership based service, but you can also see the tutorial as part of a free trial period. See more on the preview page.

Alternatively, there is a ScreenCastsOnline monthly magazine available on the iTunes App Store. This also includes a free trial period, allowing you to download this month’s issue, including the MailMate video tutorial. If you like, you can cancel the free trial after downloading this months issue.


Application Specific Passwords (iCloud Accounts)

A while ago Apple announced that they are going to require that all applications accessing iCloud data (including emails in iCloud email accounts) must do so using a so-called application specific password. This happens a week from now, that is, on June 15.

I’m expecting this to be a surprise for some MailMate users and therefore I’m, preemptively, going to answer what is likely to be frequently asked questions:

  • Can I still use MailMate for my iCloud account(s)?

    In most cases, the answer is yes.

  • MailMate is rejecting my password, what can I do?

    You need to enable two factor authentication and then setup an application specific password for MailMate. Update: The great people at BusyMac made a video which shows how to do this. It’s mostly the same for MailMate.

  • What if two factor authentication is not available to me?

    Sorry, I don’t know. As far as I know Apple has not provided a solution, but we’ll have to see what shows up next Thursday. Maybe some accounts are excluded from the new “rule”. If there is no solution and if your license key is not very old then I’ll refund your purchase.

  • Why don’t you just do like Apple Mail, it works fine with my iCloud account?

    Apple uses an authentication method only available to applications created by Apple. All third party applications are going to require an application specific password if they need to access an iCloud account.

  • But security is much better now, right?

    It’s certainly safer to use two factor authentication when possible, but the current implementation of application specific passwords is a bit of a hack. They are not much better than the main iCloud password since they cannot be limited to access specific parts of iCloud data.

  • Do you think Apple could have done it differently?

    Well, let’s just say I’m sure Apple would have done it differently if the change affected their own applications…

Use “Help ▸ Send Feedback” within MailMate if you have other iCloud related questions or maybe corrections to the above. (I’ll add any frequently asked questions to this blog post.)