Markdown in MailMate

MailMate 1.4 is the first version of MailMate capable of using Markdown to generate rich text (HTML) messages. How it currently works is documented in the manual. The main purpose of this blog post is to provide some of the thoughts behind the implementation.

Markdown example

HTML in emails

The main reason that MailMate only supports plain text in its composer is that I’m really not a big fan of HTML in emails. I can live with the size of HTML emails, the privacy/security issues (since MailMate can handle that), and the ugliness of a raw HTML email, but it is difficult to accept the visual appearance of many HTML emails.

Depending on the email client used to write an email, the look of quoted paragraphs, the font used, the font size(s), and even the font color are under the control of the originating email client. At its extreme, I’ve seen an email client insert images of correspondents in a redundant recap of all correspondence in a message thread, but it is bad enough when someone thinks that the recipient of an email is going to appreciate a blue Comic Sans font. Reading through email can some times feel like browsing through a set of amateur web pages, each with its own unique visual appearance.

Essentially, I would like the visual appearance of an email to be under the control of the recipient and not the sender. The typical workaround when viewing emails is to configure the email client to ignore the HTML body part of a message when possible (we’ll get back to how that works further below). In MailMate, you can do it by enabling the “Prefer Plain Text” option in the “Preferences ▸ Viewer” pane.

The problem, for me, with preferring plain text for both composing and viewing is that I do like rich text formatting as long as it is related to the semantics of the message. For example, when using emphasized words, bullet lists, tables, or verbatim text (code) with a non-proportional font. A nicely emphasized word does look better and is easier to read than the traditional starred *word*.

So, the question is, if MailMate is going to support rich text formatting when composing an email then how should it work? The end result has to involve HTML in order for it to work in most email clients. And it should also work well in email clients with no support for HTML. The answer is Markdown.

Markdown

Markdown is a plain text formatting syntax created by John Gruber in cooperation with Aaron Swartz. Gruber also created a Perl-script for converting Markdown to HTML, but this is no longer the important part of his creation since numerous (and better) implementations now exist. Markdown is based on the traditional use of simple markup in plain text emails such as using asterisks to emphasize a *word*. John Gruber expanded and (loosely) formalized the syntax of such “readable” markup thereby allowing it to be parsed and then converted to HTML. Even though Markdown was designed for conversion to HTML, it could be used for other formats as well (disregarding the use of inlined HTML). If you do not know Markdown then I encourage you to try the “dingus” provided by Gruber.

Using intuitive simple plain text markup symbols Markdown allows you to easily emphasize words, write outlines, use headers and subheaders, quote text, make code blocks, link to URLs, embed images (no floating images), and little more than that. It’s all about semantics. At the same time Markdown does not encourage you to do any styling such as using custom fonts and colors.

The best (and essential) aspect of Markdown is that the definition of quoted text is also based on the style traditionally used in emails. This means that quoting Markdown text works as expected. It does not ruin lists, headers, code blocks, or anything else. Markdown can therefore be just as useful for replies as it is for new messages.

Even though Markdown was inspired by email plain text syntax it has not been used much in that setting. The primary use has been for writing web content. For example, I am writing this blog post in Markdown. A Google search did reveal a few examples though.

I found MarkdownMail for iOS which allows Markdown to be used for the creation of HTML for use in both emails and blog posts, and I found hacks for Emacs and Mutt on how to create both plain text and HTML for an email (more examples are welcome in the comments).

The structure of a raw email

In order to discuss the implementation of Markdown in MailMate, we need a quick crash course on the structure of a raw email. You can skip it if you already know about MIME body parts.

Originally, an email only had two parts; an envelope (from, to, subject, etc.) and a body (the message itself). When the need for attachments arose the problem was solved by encoding them in plain text and making them part of the body with some plain text barriers to indicate the start and end of these attachments. This approach evolved over the years and finally became the MIME standard. MIME is an acronym for “Multipurpose Internet Mail Extensions” and it is described in a series of RFCs, the first one being RFC 2045. Note that the first (and now obsoleted RFC) describing MIME is from June 1992 (RFC 1341) and almost 20 years seems to have been enough to make MIME widely supported (well, maybe not for newsgroup clients).

Using MIME an email can be constructed as a tree of body parts in which each body part has its own set of headers. There are numerous advantages. An attachment is a separate body part, an email can be embedded as a body part, a digital signature can be put in its own body part, and so forth.

The type of each body part is specified in a header named Content-Type. A simple plain text message has the following header (actually the header is not needed in this case since the values are the defaults):

Content-Type: text/plain; charset=us-ascii

Note that text is denoted as the type of the message and plain is the subtype.

A message with multiple body parts could have a “root” body part with the following header:

Content-Type: multipart/alternative

In this case the subtype alternative (RFC 2046) means that the body parts in the message should be interpreted as alternatives for message display with the last body part being the best one. In practice this is almost only used to provide two alternatives; a text/plain alternative and a text/html (or multipart/related) alternative. The latter is shown by email clients which support HTML (unless configured otherwise) and all other email clients can show the plain text alternative.

Markdown is not an alternative

It may be tempting to introduce a new subtype and then use a MIME type named text/markdown, but this would miss the point of Markdown being designed to be simple readable plain text. In particular, a text/plain alternative to the Markdown text would just be the Markdown text itself. It would be redundant. Markdown is more like an attribute of plain text. A better way to convey this information would be to use a content type parameter:

Content-Type: text/plain; charset=us-ascii; markup=markdown

This is similar to the format=flowed parameter introduced in RFC 3676 (which is also supported by MailMate).

You may wonder why this parameter is needed at all since an HTML body part is already generated and that is what most receiving email clients are going to display. This parameter is useful when replying to a Markdown message since it tells the receiving email client that it is safe to assume that the reply can be written using Markdown as well. Of course, MailMate is currently the only email client able to use this information.

Taking it one step further

Now, I’m still not a big fan of including HTML in emails which are essentially plain text. The solution outlined above is acceptable, but there is still going to be issues with how other email clients display the HTML generated and how it is handled in a reply (there is a lack of standardization in this area). Currently, MailMate generates simple HTML with no styling, but it is likely that this is not going to be good enough in practice. At least not for quoted text.

We could take it all one step further if we either know that the recipient has a Markdown-capable email client or if we simply don’t care. When MailMate displays a plain text email (or body part) with the markup=markdown parameter/value then it automatically converts the plain text to HTML before displaying it. The beauty of this is the simplicity of a raw message. Here is an example:

From: "Freron Software" <mm-feedback@freron.com>
To: "Freron Software" <mm-feedback@freron.com>
Subject: Taking it one step further...
Date: Wed, 21 Dec 2011 13:35:00 +0100
Message-ID: <DB451A45-1077-4C4D-90A2-44A18EEE68C9@freron.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed; markup=markdown
Content-Transfer-Encoding: 8bit
X-Mailer: MailMate (1.4r2651)

# Markdown in [MailMate][]

Note that *this* message has no HTML body part. Based on the
`Content-Type` header, the receiving email client could do the
conversion, but if it is not able to do so then this is still readable
plain text.

In the eyes of an email client developer, this is a beautiful raw email.
Nicely sorted headers and no hideous quoted-printable encoding (or
base64) even though long lines and non-ASCII like &aelig;&oslash;&aring; is used.

[MailMate]: https://freron.com

The responsibility of generating the HTML is now on the receiving email client. The obvious disadvantage is that it would not be supported by most email clients, but Markdown is also perfectly readable as plain text. Nicely formatted messages can be sent this way without using more space than normally used for a simple plain text message, but email clients recognizing Markdown can both display these messages nicely and create nicely formatted replies using Markdown as well.

Note that the use of Markdown does not exclude the possibility of a WYSIWYG editor, but it does require that the features of such an editor should be limited to what can be expressed as basic Markdown.

Generating an HTML body part is optional in MailMate. If you want to play with the Markdown settings in MailMate then note that the composer has a custom layout (see the “View ▸ Layout” menu) which makes previewing easy. If you do this make sure you also use the “View ▸ Show Raw Message” (⌥⌘U) and “View ▸ Show HTML Source” (⌃⌘U) menu items.

Standardization

When receiving a Markdown message MailMate uses sundown (a Markdown converter) to generate the HTML displayed. For now, HTML in the Markdown text is stripped, but the Markdown extensions fenced code and tables are allowed. Most importantly, sundown has been configured to respect newlines. This is different than the typical use of Markdown, but it ensures that email clients with no knowledge of Markdown can still handle paragraphs correctly. It also avoids problems with the concept of lazy blockquotes.

The choices above (made by me) illustrate the need for standardization if this feature should ever go beyond the small community of MailMate users. In particular, when HTML is not generated the sending and receiving email clients must agree upon the style of Markdown used. Essentially, 2 specifications are needed:

  • A specification of the markup parameter for the Content-Type header.
  • A specification of the Markdown language as used in emails.

The first one is fairly straightforward while the second one looks like a can of worms due to the many existing variations of Markdown and the current lack of strictly formal specification.

To properly introduce a markup parameter, it has to be registered and approved by IESG as described in section 3.1 in RFC 4288:

Registrations in the standards tree MUST be
approved by the IESG and MUST correspond to a formal publication by a
recognized standards body.

In the same RFC there is also the following relevant comment about parameters in section 4.3:

New parameters SHOULD NOT be defined as a way to introduce new
functionality in types registered in the standards tree, although new
parameters MAY be added to convey additional information that does
not otherwise change existing functionality.  An example of this
would be a "revision" parameter to indicate a revision level of an
external specification such as JPEG.

Time will tell if markup=markdown has a future beyond MailMate. For now, MailMate is a proof-of-concept which can be used to examine any practical problems which needs to be considered if standardizing either the markup parameter or Markdown.

Random thoughts and open questions

  • Inlining images works fine with external resources, but it could be made to work with embedded images as well by using content ids (cid:).
  • Should embedded HTML be allowed?
  • Should a default embedded CSS stylesheet be allowed?
  • It would be great if the editor had syntax highlighting, shortcuts for common actions, and so on.
  • MailMate currently does not handle replying to HTML-only messages very well. These could be converted to Markdown text before replying.

You are welcome to continue the list in the comments.