MailMate 1.12 Released

A detailed list of the many changes, fixes, and new features can, as always, be found in the release notes. Most importantly, MailMate 1.12 includes better handling of smart mailbox and searches based on dates. MailMate is (finally) able to do searches relative to the current day, week, month, and year (also minute and hour if anyone should find that useful). Smart mailboxes, for example, can be created which only shows the emails of today instead of the emails of the past 24 hours. For convenience, the “Message is” menu in the conditions editor includes shortcuts to creates searches for messages received “Today/Yesterday/This Week/This Month/This Year”, but all date headers allow any kind of date relative search if something more specific is needed.

Most importantly, the toolbar search field supports a new modifier to do date based searches. The details are in the manual, but here are a few of the examples:

Search string Meaning
d 2005 Received in year 2005
d 2005-4-1 Received April 1st 2005
d <2005-4 Received before April 2005
d 7 The most recent day 7 (this or previous month)
d 7-4 The most recent April 7th (this or previous year)
d 7-4-2005 Received on April 7th 2005
d 1y Received this year
d !3d Not received within the past 3 days
d 5y !2y Received 2-5 years ago
d today Received today
d yesterday Received yesterday
d 2005 or 2007 or 2y Received 2005 or 2007 or within the past 2 years

The search syntax is also available via the “View Search Syntax” menu item in the toolbar search field menu.


MailMate 1.11.3 Released

It’s a minor version bump, but there’s a lot of changes and fixes — including a few new features. As always, the details can be found in the release notes. In particular, it is now possible to configure any mailbox to show the messages of its submailboxes. Any early macOS Mojave users should also appreciate the many changes done to make MailMate work better in the new macOS dark mode (this is work-in-progress).


EFAIL

You might have read about the EFAIL project today and you might wonder what my thoughts are on the subject. Or more precisely, you want to know whether or not MailMate is affected.

The short answer is that MailMate is affected, but that MailMate 1.11 (released mid March) includes the fixes and workarounds needed. That is, if I didn’t miss something. This doesn’t mean that all email encryption is then safe, because that depends on both the sender and the recipient(s). I should also note that the EFAIL paper only tests MailMate 1.10 (released mid December). As far as I know, test results for MailMate 1.11 or later versions do not exist.

The long answer is on the mailing list since I also wanted to provide an open forum for this subject when/if issues are reported that I might not have fixed.


MailMate 1.11.2 Released

Two weeks ago MailMate 1.11.1 was released, but it was primarily a maintenance release. MailMate 1.11.2 released today includes some new features. Most of them are related to the Rules pane in the mailbox editor, but there are also other new features and some changes/fixes. As always, the details can be found in the release notes:

  • New: Rules can be reordered and dragged to other rule lists. Hold down ⌥ to copy.
  • New: Rules can be hierarchical (use drag’n’drop to make subrules or use the new actions menu).
  • New: Rules work with cut/copy/paste.
  • New: Undo/redo works for rule reordering/dragging/cutting/pasting/deleting.
  • New: Actions menu in the rule editor which includes “Duplicate” and “Add Subrule…” menu items.
  • New: Boolean setting includeSubmailboxMessages = 1; can be used by manually editing Mailboxes.plist. A GUI for this feature is going to be added later.
  • New: Hidden preference MmAutomaticallyExpandThreadsEnabled (experimental).
  • New: Hidden preference MmAutomaticallyExpandOnlyWhenCounted (experimental).

Also note that a new bundle is now available: Brian Reiter added support for using VS Code as an external editor. The other editors supported are Atom, BBEdit, MacVim, Sublime Text, and TextMate. Instructions for using an external editor can be found here.


MailMate 1.11.1 Released

This is mainly a maintenance release. Details are in the release notes.

The release is particularly interesting for users preferring the original IMAP approach to deleting emails, that is, marking emails as deleted and then expunging them (permanently) at a later time:

  • New: Hidden preference MmDeleteBehavior: moveToDeletedMessages, markAsDeleted, expunge, archive.
  • New: Hidden preference MmAutomaticExpungeBehavior: mailboxSwitch, never.
  • New: An Expunge toolbar button (which reuses the trash icon).
  • New: Accounts can be given their own deleteBehavior overriding the global default.

MailMate 1.11 Released

The MailMate 1.11 release has many new features, improvements, and fixes. As always all the details can be found in the release notes. I’m particularly happy about some non-visible “under the hood” changes which are going to make it easier for me to work on new features and optimizations in the future.

Here follows a short list of the most notable highlights:

  • New: The conditions pane (also in rules) now have a “Message is” menu and a “Message is not” menu. These have submenus to easily match the “read” state, (colored) flags, and various other frequently used search conditions. It is not new that these searches can be made, but they should be much easier to do now.
  • New: GUI setting for automatically adding trusted S/MIME certificates to the keychain. This includes telling the user when it happens and warning if a certificate already exists for the same email address.
  • New: A dedicated “Two Panes” layout with no message preview (see the “View ▸ Layout” menu).
  • New: The context sensitive menu of an IMAP account displays the exact storage quota usage and limit. If multiple so-called quota roots exist then they are shown in a submenu. This only works for IMAP servers with QUOTA support as described in RFC2087.
  • New: The MmQuotaWarningLevel hidden preference can be used to control when the used quota percentage is shown for IMAP accounts in the mailbox list itself. The default is 80.0.
  • New: Forwarding (as an attachment) is now available as a rule action.
  • Changed: Major refactoring of all so-called rule editors (mailboxes pane, conditions and rule actions). This should make it much easier to add features to these in the future.
  • Changed: Major refactoring of the internal query system (used by smart mailboxes and many other parts of MailMate). This includes bug fixes and simpler (more robust) code.
  • Changed: Improved support for tags in the Composer window (including a toolbar popup button).
  • Fixed: Several security related issues.
  • And much more…

Developer Documentation for Bundles

MailMate has a very flexible plugin-system available via so-called bundles. There’s already a large set of bundles available, for example, for various task managers and text editors, but you can also create your own bundles with your own personal features.

Bundles have existed for MailMate for a long time, but the creation of bundles has been largely undocumented. This changed recently when I created a documentation page for bundles. This should make it easier to create personal bundles, but you are also very welcome to create bundles to be shared with other MailMate users. Companies with multiple MailMate users might also want to create their own company-wide bundles which could, for example, integrate with local issue tracking systems or provide easy creation of standard replies. If there’s something you would like to do which is not possible using bundles then let me know. I’d like to make it even more flexible than it already is.

Here’s a quote from the documentation which is related to the work MailMate has to do to help you avoid having to know anything about how to parse an email:

Parsing emails is really hard. You need to handle MIME parts, encrypted parts, various encodings like quoted-printable, base64, format=flowed, header encoded words, and various non- or semi-standard formats like tnef (winmail.dat). But much worse, you also have to work around numerous bugs and legacy types of behavior. Ideally, you need to be able to parse any email created by any version of any email client since the early 1970s.

If MailMate only allowed you to get the raw data of an email then it would be very hard to do anything useful and it would be close to impossible to make it robust. Fortunately, MailMate allows you to get the data you need after it has been decoded and after MailMate has worked around all the bugs and legacy stuff of the past.

On a more philosophical note, the main goal of MailMate is to provide input which takes care of as many of the existing email intricacies as possible before handing over data to commands or other parts of the interface. The many problems concerning the conversion of headers and bodies to any kind of canonical data should be handled by MailMate to keep everything else as simple as possible. In other words, canonicalization is to be MailMate’s side of the fence.

Implicitly, this also means that if you have an email which MailMate cannot parse as you would expect it to do so (even if it’s a buggy one) then let me know and I’ll look into yet another workaround.


MailMate 1.10 Released (and Some Thoughts on Version Numbers)

If you are like me then the version number, 1.10, feels a bit weird because it’s not just single digits. That doesn’t make any sense, but it’s actually the only reason that MailMate has been using 1.9.x version numbers since March 2015. It took longer to get from version 1.9 to 1.10 than it took to get from 1.5 to 1.9. Theoretically, this shouldn’t matter much, but in practice small changes of the version number means that updates are going to be considered to be minor updates. Unfortunately, this also means less publicity when updates are released.

You might wonder why I don’t just bump it to 2.0, but I’m saving that for when it really feels like 2.0 to me. That might also not make sense and, who knows, maybe that’ll never happen. Just remember, it’s just numbers.

The following are some of the notable items when comparing 1.10 with the first 1.9.7 release. Some of them were part of a public release which did not involve a version bump at all (r5425). You’ll need to dig into the release notes to get the full story, in particular, if you haven’t tried MailMate since version 1.9.

  • New: Improved handling of a spoofed email address in the name part of the “From” header. Also fixed bugs related to mailsploit.
  • New: Hidden preference to always launch in an offline state: MmInitialOfflineStateEnabled
  • New: Key binding for toggling all accounts offline/online: toggleOnlineStateOfAllAccounts:
  • New: Hidden preference for sending without using the X-Mailer header: MmStripXMailerBeforeSending
  • New: Bundle commands can request attachments (or any other message parts) to be provided as files.
  • New: Bundle commands can return attachments to be included in created emails (both new ones and replies).
  • New: Bundle commands can return actions in the JSON format.
  • New: The primary dock badge counter now looks the same as in other applications.
  • New: Hidden preference MmCommandsIncludeCollapsedMessages which changes the behavior when running a command on collapsed message threads. This is likely to become default behavior.
  • New: Hidden preference MmOpenMessageURLInContext to tell MailMate to handle message: URLs by opening a window with the IMAP mailbox of the message selected.
  • Changed: Added script for more easily creating a new bundle (SharedSupport/bin/create_bundle).
  • Changed: Collapsed thread still shows attachment icon in message list if any attachments exist in the thread.
  • Changed: An outlined flag is now shown when a collapsed thread contains one or more flagged messages.
  • Changed: Optimized some date related computations (this should make date-based smart mailboxes much faster).
  • Changed: Undo/Redo now works for attachments added in the Composer.
  • Changed: Added a warning sheet when pasting in the Composer results in attaching one or more files.
  • Changed: Disabled dragging attachment file names on High Sierra since I cannot make it work correctly. Drag the icon instead.
  • Fixed: Some layout and coloring issues in the mailbox editor (High Sierra).
  • Fixed: Group names (from Contacts) in the Composer were forgotten if closing/sending without first hitting return.
  • Fixed: Some issues with auto-scrolling the preview related to code/math segments.
  • Fixed: Spotlight and the custom location feature now use relative paths. The previous approach would lead to problems if the username of the account was changed.
  • Fixed: Some issues with the Spotlight importer and the sharing extension on High Sierra.
  • Fixed: MmOpenMessageURLInContext also works when opening .eml files.
  • Fixed: The sharing extension is now a bit better at handling text input, e.g., when sharing a note in the Notes app. (Only plain text handled for now.)
  • Fixed: Enabling/disabling some types of mailbox/search conditions would clear the text field or fill it with some old value.

Email Spoofing

A few days ago several MailMate users made me aware of the mailsploit.com site. The site describes a type of phishing attack known as email spoofing and it provides some examples which makes numerous email clients1 fail to display the real email address of the sender — as given in the “From:” header of the email.

If you are in a hurry then all you need to know is that MailMate did fail on some of these examples (although for different reasons than other email clients) and that I have fixed it in the latest test release of MailMate. Hold down ⌥ when clicking “Check Now” in the Software Update preferences pane to try it out. You should get revision 5441 or newer. Note that revision 5440 also fixes the issue, but in practice this doesn’t help much in terms of making spoofing harder. More on that below if you have time for some more details.

The technical part of an exploit based on “mailsploit” is described well at the web site. In short, it tricks some email clients into finding the wrong email address within an email address header like “From”. The email client would then display the wrong sender. The definition of “wrong” here is based on RFC5322.

It is important to understand that spoofing a “From” header has always been easy and, in my opinion, it is still easy. One attempt to change this is DMARC and this is where “mailsploit” shows that such an attempt is doomed to fail if email clients are not improved.

After I fixed the “mailsploit” related bugs in MailMate I realized that it didn’t really fix a more basic problem in MailMate. This is because many parts of the interface only display the name part of an address header. The email address is only shown if there is no name part. This makes it easy to spoof a sender like this:

From: "potus@whitehouse.gov" <evil@example.com>

You can even make it more believable like this:

From: "Donald Trump <potus@whitehouse.gov>" <evil@example.com>

In MailMate, you’ll only see the first part in the message list “From” column, but both are shown in the message view (although not in the compact headers mode). The same is true for both Apple Mail and Gmail (I didn’t check any other email clients). Apple Mail won’t even show you the real address in the message view unless you open a context sensitive menu.

Nothing here is new, but most email client users might not be aware of how little they should trust a “From” header. Bugs or no bugs.

In the most recent test release of MailMate I’ve added the following improvement: Whenever the name part of an address header contains a @ then it’s replaced with a skull (💀). That should at least make the user aware of simple attempts to spoof an address header.

  1. The reason so many email clients have parser bugs is likely because it’s ridiculously hard to parse an email address header. An entire book could be written on this subject. Once again I’ll emphasize that an email client should be able to parse any email generated by any (buggy) version of any email client/generator ever used. 


A Developer's Slow Descent into Madness: Exchange IMAP

I recently wrote about a frustrating issue with Yahoo IMAP and I was quite surprised (and a bit embarrassed) when I actually got into contact with someone at Yahoo working on fixing it. Now I’m thinking it would be nice if I could get the same result if I write about another frustrating server issue.

But first I’ll have to quote myself from my previous blog post:

All software is buggy and it’s always a bit hypocritical for a developer to complain about bugs. I’m certainly aware that MailMate has many bugs and some of them are incredibly frustrating for some users. For those users it might be a comfort to know that an email client developer has to deal with one of the most frustrating types of bugs: IMAP server bugs.

The target of this rant is Exchange IMAP which has an annoying habit of rejecting uploaded messages without any information about why it happens. In the past I’ve seen this happen for emails with one or more very long lines. In that case, the error message is:

BAD Command Error. 10

I was able to work around it by detecting this particular response and then re-encoding its MIME parts before retrying the upload. The subject of this blog post is a slightly different error message:

BAD Command Argument Error. 11

A MailMate user reported this issue when trying to upload a sent message (created by MailMate). This message included another message forwarded as an attachment. Fortunately, I was able to reproduce the issue with an outlook.com account and I could start debugging. I simplified the message step by step and in the end it appears to require two things to happen:

  • A message including a message forwarded as an attachment.
  • The attachment includes an encoded header line which Exchange cannot parse.

Here’s an example:

From: Test@example.com
To: Test@example.com
Subject: Test message pushing Exchange to fail
Message-ID: <54259AAF-8FE0-44A4-8DCE-032F020A7C13@example.com>
Content-Type: multipart/mixed; boundary="=_MailMate_1C634E1D-F7F7-4E44-BD38-1E95A63DE5B6_="


--=_MailMate_1C634E1D-F7F7-4E44-BD38-1E95A63DE5B6_=
Content-Type: message/rfc822

Subject: =?UNKNOWN?Q?n=E9cessaire?=

Body of forwarded message.

--=_MailMate_1C634E1D-F7F7-4E44-BD38-1E95A63DE5B6_=--

The UNKNOWN charset used in the “Subject” line doesn’t make sense and was probably introduced or caused by some other buggy email software. It’s a minor issue, but apparently it’s a major issue for the Exchange IMAP implementation.

After spending a lot of time to track down the source of the problem, I now have to figure out how to work around it. You might wonder why a workaround is needed at all since the bug is clearly in the server implementation, but if I chose this path for every IMAP server bug encountered then many MailMate users would quickly consider MailMate to be unstable. Most users just need things to work. Furthermore, users might not have the option of switching to a more robust IMAP server.

The workaround I chose was to let MailMate re-encode all header lines in all MIME parts of the email before retrying the upload. This only happens for the uploaded copy. The worst part of this is that the user is not notified about the problem. (Some day I would like to add a feature which allows the user to review all the workarounds taking place while using MailMate.)

Now, I’d like to quote my previous blog post again:

An IMAP server bug can often be characterized as follows:

  • It only affects IMAP email clients and not a webmail client or some proprietary email client for the IMAP service involved.
  • It makes the email client seem like it has a bug.
  • There’s no way to report it.
  • It is probably not hard to fix.
  • It is never fixed.

Yahoo proved me wrong on the above. It’ll be interesting to see if Microsoft can do the same. In practice (for me), it doesn’t matter much because Microsoft is not in control of all installations of Exchange IMAP. Even if a bug is fixed then servers with the bug will exist many years into the future.

Some times I like to state that an email client must be able to handle any email ever generated by any kind of email software at any time of the history of emails. I should probably add that it should also be able to work around any bug that exists in any live IMAP server.

Oh, and I promise that my next blog post won’t be rant :)