History

The first attempt to standardize email headers can be seen in RFC 561. This is September 1973, the document is 2 pages long, and not surprisingly it does not mention the “Re:” prefix. Note that this does not mean that the “Re:” prefix was not in use at the time. Many RFCs have been written after the fact and simply document what is already the prevailing behavior of implementations. This is emphasized by the first use of “Re:” in an RFC in 1977 where it is only present as part of an example of an email reply. This example lives on in RFC 733 (1977) and the classic RFC 822 (1982). It takes almost 20 years before RFC 822 is updated, but for the first time the “Re:” prefix is explicitly described in RFC 2822:

When used in a reply, the field body MAY start with the string “Re: ” (from the Latin “res”, in the matter of) followed by the contents of the “Subject:” field body of the original message.

It even includes an explanation of what “Re:” means although I’m with this guy on that one, but the Latin interpretation serves a purpose as we’ll see further below. For the sake of completion, the meaning of “Re:” was changed (corrected?) to be from the Latin “in re” in RFC 5322.

Ok, that was a nice bit of history and it actually started before I was born which means I don’t have to feel so old today. A similar history exists for many other of the standard email headers, but in the special case of the “Re:” prefix you might ask: Why standardize it at all? I hope that is going to be evident at the end of this blog post.

Theory

Originally, the “Re:” prefix served the simple purpose of making it easy to see when a message was a reply to another message. The use of it might even pre-date the introduction of the “In-Reply-To” header which could be seen as an alternative solution. In retrospect, it should have been standardized with the following rules:

• Add “Re:” to the subject of replies and replies only.
• Do not add another prefix when replying to a reply.

As we’ll see further below then the following clarification would also have been nice: Do not use any localized variants of “Re:”.

Practice

It might not have helped if the reply prefix had been standardized, but the lack of standardization certainly did not help. Variations of the prefix were invented and they resulted in various problems. I’ll describe 3 prefix related problems below.

The first problem is how to handle a reply to a reply. The naive solution is to just add another “Re:” prefix, but this is ugly and it can quickly increase the length of a subject line. A better solution is to only add the prefix if one does not already exist. Some email clients decided to do better than that and they introduced a counter in the prefix:

Re[4]: This correspondence now involves 4 messages.

That is not a bad idea, but since not all email clients supported it then a reply would often look like this:

Re: Re[4]: This correspondence now involves 4+1 messages.

The “smart” email client might be able to merge the prefixes when replying and therefore the above could be an acceptable side effect until all email clients supported the counters, but this now seems unlikely to ever happen. It does not help that this has never been standardized.

The second problem is localization. Some email clients (none mentioned none forgotten) insert a localized prefix instead of “Re:”, for example, in Danish this could be “Sv:” as an abbreviation of “Svar” (Reply). The result is predictable:

Re: Sv: Re: Sv: I don't think we are using the same email clients...

This is probably why it was decided that “Re:” is a Latin abbreviation.

The third problem is mailing list subject prefixes, for example, the MailMate mailing list prefixes subject lines with “[MlMt]” (I’ll use “blob” to refer to it as done in RFC 5256). This can confuse both email clients and mailing list software, in particular, if combined with the other problems described. The result could be a subject line like this:

Re: Sv: Re: [MlMt] Re: This is starting to get silly...

Things get even more complicated when also considering forwarding related prefixes and other semi-standard conventions. Now, the question is, what can MailMate do about this?

The heuristic solution

No perfect solution exists for this problem. The following is an excerpt from RFC 5256 where the suggested solution is to simply ignore most of the variations described:

Translations of the “re” or “fw”/”fwd” tokens are not specified for removal in the base subject extraction process. An attempt to add such translated tokens would result in a geometrically complex, and ultimately unimplementable, task.

I agree with the sentiment of this, but MailMate tries to be a little bit more pragmatic. The heuristic solution in MailMate is simple: A regular expression is used to identify the prefix of a subject line. This regular expression and many others can be found in the specifiers.plist file within the MailMate application bundle. The essence of the regular expression is as follows:

((?:\s*\p{Alpha}{2,3}(?:\[\d+\])?[:：])+)

This is geek language for a (possibly repeated) sequence of two or three Unicode letters followed by an optional counter and a mandatory colon. The colon can actually be one of two kinds of colon. The usual one and a so-called full-width colon. The latter is used in the Chinese prefix “回信：” and possibly by other languages as well.

Why MailMate needs to know the prefix

MailMate is quite liberal with respect to what it identifies as a prefix. It is therefore also important that the prefix is handled with care. As we’ll see below it is mostly used for improving the display and sorting of subject lines. An exception is the generation of a subject line for a reply for which MailMate tries to never add more than a single “Re:”. This strategy adheres to Postel’s law: “Be liberal in what you accept, and conservative in what you send”.

The identification of the prefix, the blob, and the body of a subject line is used for the following purposes in MailMate:

• Display all subject lines with the same order of elements and without extraneous whitespace between these elements. The blob and, in particular, the order of the prefix and the blob is not standardized and therefore varies a lot.
• Proper sorting of messages by subject.
• Only compare the body of a subject when trying to determine whether or not the subject has changed between a message and its reply (MailMate warns about such a change before sending a reply).
• Only prefix a subject line with a single “Re:” when generating a reply.
• Allow searches for messages with a particular subject blob or subject body.

The behavior of MailMate is far from perfect, but the solution to this and similar header parsing problems is very general and flexible. For example, the sorting of the messages outline by subject is handled by defining the following sortKey for the subject column of the messages outline (outlineColumns.plist):

sortKey = "subject.blob,subject.body,subject.prefix";

And the display of the subject line in the messages outline is handled by the following format string (outlineColumns.plist):

formatString = "${subject.prefix:+${subject.prefix} }${subject.blob:+[${subject.blob}] }${subject.body}";

If you don’t like that (and you feel adventurous) then you can create your own columns or override the existing ones using low-level customizations.

This post became much longer than I anticipated and it didn’t even cover all the details. Now, please don’t get me started on the intricacies of the address related email headers…

• Supports inline images in Markdown mode. Markup for referencing an attached image is inserted when adding an image to the textview (dragging or pasting).
• Opens links in the composer when hitting the enter key.
• Now derives a name from other headers if a Reply-To header does not include the name of the recipient.
• Added “Reply List” menu item (⌥⌘R).
• Added hidden preference (MmMessagesOutlineMoveStrategy) to control how the message selection changes when moving messages out of the currently selected mailbox.
• Handles (removes) Re:-prefixes in Chinese and other languages when generating the subject line for a reply.

Mac OS X Snow Leopard is supported, but you should still expect version 1.6 to require Mac OS X Lion.

Here are the most interesting changes since version 1.5.1:

• New “Synchronization Schedule” submenu for the Mailbox menu to configure how often individual mailboxes are synchronized.
• “Send and Archive”, “Reply All”, and “Flag” are now available as optional toolbar buttons.
• Inlined images are now rotated according to EXIF data in images (Mountain Lion only).
• Added “Reply to …” to the context sensitive menu of addresses in the headers view.
• Now respects the first/last name ordering of individual contacts in the “Address Book” (“Contacts” on Mountain Lion).
• Automatic handling of winmail.dat (TNEF) files. This is an experimental feature for expert users. Feedback is welcome.
• Low level feature can be used to add custom email headers to outgoing messages.
• Improved handling of OpenPGP and S/MIME. Now MailMate properly handles signed/encryped body parts in multipart/mixed MIME messages.
• Various IMAP fixes/workarounds including a number of fixes to handle ‘\‘ and ‘"‘ correctly.
• SMTP code now correctly handles lines prefixed with one or more dots.
• And more…

Detailed release notes can be seen here.

The review on the subject of searching/smart mailboxes:

As email search abilities go, this seems less like bringing a gun to a knife fight, and more like thundering into that particular duel at the controls of a helicopter gunship.

As any software developer I most often receive feedback from users with problems, so this was nice to read:

Needless to say, the program never crashed, glitched, or gave me any sort of trouble during my tests. MailMate does not know weakness.

The implementation of tags is based on IMAP keywords which means that they are easily and efficiently synchronized with an IMAP server (some servers may have limited support). The user interface is an unobtrusive token based text field shown when hitting a simple shortcut. It is very similar to the recipient address fields of the composer including automatic completion of tag names. It is easy to create new tags and a new preferences pane allows you to review and edit existing tags and their IMAP keyword equivalents. Read more about tags in the manual.

Other changes include:

• When failing to encrypt (OpenPGP) a message to an untrusted recipient then “Trust Once” is now provided as an option (when trying to send).
• Untrusted S/MIME certificates are now editable (via the “Show Details” button), that is, the trust settings can be changed within MailMate.
• Holding down ⌥ can be used to expunge (delete) selected messages immediately (instead of moving them to the trash mailbox of the IMAP account). An alert is shown which can optionally be suppressed.
• Improved line wrapping of headers of outgoing messages (standards compliant behavior).
• Various IMAP related fixes and workarounds.
• Fixed problem when using mailto: with attachments (only possible via AppleScript).

This release also includes a workaround for a problem under 10.7.5 which would make MailMate and the authorization interface to the system keychain hang when changing the trust settings of a server certificate. The problem is an Apple bug related to code signing.

Version 1.5 is likely to be the last major release with support for Mac OS X Snow Leopard (10.6). Expect version 1.6 to require Mac OS X Lion.

As usual detailed release notes are available. Here are some of the highlights:

• Support for S/MIME and OpenPGP.
• Updated to use Growl Framework 2.0 for notifications. This means MailMate now supports Mac OS X Notifications on Mountain Lion with or without Growl installed.
• Codesigned with a Developer ID certificate from Apple making MailMate work with Gatekeeper (Mountain Lion security feature).
• Support for searching LDAP servers when autocompleting names/addresses in the composer. Currently no GUI for this feature. Read more about it in the manual page about “Hidden Preferences”.
• Using AppleScript it is now possible to create (and optionally send) messages with attachments.
• Numerous IMAP fixes including several workarounds for buggy IMAP servers.
• Workaround for a bug in OS X 10.8.2 triggered when using AESendMessage to send AppleScript events. This affected users of SpamSieve. Every message sent to SpamSieve would hang and then time out (after 2 minutes) making MailMate seem extremely slow.
• HTML body parts in emails are now properly indexed for searching.
• Numerous performance optimizations.

Since Apple released the 10.8.2 update for Mountain Lion I have received several reports about MailMate being slow and unable to detect spam. The problem turns out to be that the AppleScript communication between MailMate and SpamSieve breaks down. Each request sent by MailMate to SpamSieve hangs for 2 minutes before failing with a timeout error. This happens for every incoming message. See further below for more technical details and a bit of insight into the detective work some times needed in the software development process.

The Solution

If you think you have this problem with MailMate/SpamSieve then you should fetch the current test version of MailMate by holding down ⌥ when clicking “Check Now” in the Software Update preferences pane of MailMate (note that, in general, test versions are not always stable versions of MailMate and they may include incomplete features).

If you have a Mac App Store license then you cannot update to a test version. Instead you should contact me directly.

You can also temporarily fix the problem by killing a process named appleeventsd. You can find and kill it in the Activity Monitor or you can paste the following in a Terminal window (requires a password):

sudo killall -KILL appleeventsd

The Technical Details

The time line of an investigation of this problem can be seen in this ticket. In short, Michael Tsai (SpamSieve developer) joined the discussion and we found a workaround involving a different approach to the AppleScript communication between MailMate and SpamSieve. But a much better understanding of the problem and a better workaround came about when Michael was contacted by Brian Webster. You can read about Brian’s findings here.

This is what is currently “known” about the problem:

• The problem is related to the OS X 10.8.2 update.
• The problem is related to the use of the AESendMessage/AESend functions used with typeApplicationBundleID or typeApplSignature for the target type.
• Using typeKernelProcessID for the target type still works (thanks goes to Brian Webster for this crucial fact). This is what is used for the workaround in MailMate.
• NSAppleScript and osascript are not affected by the problem.
• Killing the appleeventsd daemon fixes the problem temporarily (Antonin Hildebrand discovered this fact).

The above indicates that the problem is likely to be some kind of stale cache used by appleeventsd for mapping bundle ids and 4-letter application signatures to process ids. It also indicates that triggering this problem involves relaunching the target application (in order to give it a new process id). This has been confirmed by a user (Torsten Grust) in the context of MailMate/SpamSieve.

Thanks to everyone who helped track down this problem. Note that I am still not able to reproduce the problem myself which means that feedback on the solutions above is still appreciated.

Speaking of reviews, MailMate has also received a few user reviews in the Mac App Store (in the American and Australian stores). At the time of writing this, they are all positive 5 star reviews except for a single 1 star review. The exception is a good example of why Apple should provide some way to contact unhappy app store buyers (it could work anonymously). I don’t mind negative reviews, but in this case, a buyer has an “out-of-the-box” problem and I cannot do anything to help. Others have written wise words on this subject.

You are very welcome to provide a review/rating in the Mac App Store, but if you have some specific problem then please write a feedback email first (use the “Help ▸ Send Feedback” menu item). I may not be able to help you, but feedback is never ignored.

@mailmateapp Any chance we can permanently tag an email address as ‘not junk’ so as not to be forced to click that button every time?

It is a good question and it deserves more than a 140 character answer.

Image blocking

By default, MailMate does not display external images for any messages arriving in the inbox of any account. This is a security feature which ensures that no one can track if or when you read a given message. When viewing a message it is possible to allow loading external images once or allow it permanently by explicitly marking the message as “Not Junk”.

Image blocking can be configured in the Security preferences pane. Most importantly, it is possible to change the set of messages for which image blocking is enabled. This brings in all the power of smart mailboxes to the configuration of image blocking and we can use this to “implement” an answer to the Twitter question above.

The Answer

We need to create two smart mailboxes. The first one simply matches all messages marked as “Not Junk”. Base it on “All Messages” and give it the following condition:

The other mailbox is more interesting. We’ll name it “Image blocking” and we’ll make the smart mailbox above a child of this mailbox. This is not important, but it emphasizes that the two mailboxes are related.

The “Image blocking” mailbox can be based on any mailbox (the universal Inbox would be a natural choice) and it only needs a single condition:

In other words, this mailbox matches any message for which the sender’s address does not match any sender’s address in the set of messages marked as “Not Junk”.

Finally, we need to change the setting in the Security preferences pane to block images for messages in the smart mailbox named “Image blocking”.

The end result is that whenever a message is marked as “Not Junk” then all other messages from the same sender are treated as if they were marked as “Not Junk” as well.

This may all seem a bit complicated and it certainly could be made easier to enable something like the above, but the beauty is in the flexibility. You can make image blocking behave in almost any way you can imagine.

@dmorel: I hope this answers your question.