Thoughts on Writing Emails using Markdown

Markdown in MailMate

MailMate 1.4 is the first version of MailMate capable of using Markdown to generate rich text (HTML) messages. How it currently works is documented in the manual. The main purpose of this blog post is to provide some of the thoughts behind the implementation.

Markdown example

HTML in emails

The main reason that MailMate only supports plain text in its composer is that I’m really not a big fan of HTML in emails. I can live with the size of HTML emails, the privacy/security issues (since MailMate can handle that), and the ugliness of a raw HTML email, but it is difficult to accept the visual appearance of many HTML emails.

Depending on the email client used to write an email, the look of quoted paragraphs, the font used, the font size(s), and even the font color are under the control of the originating email client. At its extreme, I’ve seen an email client insert images of correspondents in a redundant recap of all correspondence in a message thread, but it is bad enough when someone thinks that the recipient of an email is going to appreciate a blue Comic Sans font. Reading through email can some times feel like browsing through a set of amateur web pages, each with its own unique visual appearance.

Essentially, I would like the visual appearance of an email to be under the control of the recipient and not the sender. The typical workaround when viewing emails is to configure the email client to ignore the HTML body part of a message when possible (we’ll get back to how that works further below). In MailMate, you can do it by enabling the “Prefer Plain Text” option in the “Preferences ▸ Viewer” pane.

The problem, for me, with preferring plain text for both composing and viewing is that I do like rich text formatting as long as it is related to the semantics of the message. For example, when using emphasized words, bullet lists, tables, or verbatim text (code) with a non-proportional font. A nicely emphasized word does look better and is easier to read than the traditional starred *word*.

So, the question is, if MailMate is going to support rich text formatting when composing an email then how should it work? The end result has to involve HTML in order for it to work in most email clients. And it should also work well in email clients with no support for HTML. The answer is Markdown.

Markdown

Markdown is a plain text formatting syntax created by John Gruber in cooperation with Aaron Swartz. Gruber also created a Perl-script for converting Markdown to HTML, but this is no longer the important part of his creation since numerous (and better) implementations now exist. Markdown is based on the traditional use of simple markup in plain text emails such as using asterisks to emphasize a *word*. John Gruber expanded and (loosely) formalized the syntax of such “readable” markup thereby allowing it to be parsed and then converted to HTML. Even though Markdown was designed for conversion to HTML, it could be used for other formats as well (disregarding the use of inlined HTML). If you do not know Markdown then I encourage you to try the “dingus” provided by Gruber.

Using intuitive simple plain text markup symbols Markdown allows you to easily emphasize words, write outlines, use headers and subheaders, quote text, make code blocks, link to URLs, embed images (no floating images), and little more than that. It’s all about semantics. At the same time Markdown does not encourage you to do any styling such as using custom fonts and colors.

The best (and essential) aspect of Markdown is that the definition of quoted text is also based on the style traditionally used in emails. This means that quoting Markdown text works as expected. It does not ruin lists, headers, code blocks, or anything else. Markdown can therefore be just as useful for replies as it is for new messages.

Even though Markdown was inspired by email plain text syntax it has not been used much in that setting. The primary use has been for writing web content. For example, I am writing this blog post in Markdown. A Google search did reveal a few examples though.

I found MarkdownMail for iOS which allows Markdown to be used for the creation of HTML for use in both emails and blog posts, and I found hacks for Emacs and Mutt on how to create both plain text and HTML for an email (more examples are welcome in the comments).

The structure of a raw email

In order to discuss the implementation of Markdown in MailMate, we need a quick crash course on the structure of a raw email. You can skip it if you already know about MIME body parts.

Originally, an email only had two parts; an envelope (from, to, subject, etc.) and a body (the message itself). When the need for attachments arose the problem was solved by encoding them in plain text and making them part of the body with some plain text barriers to indicate the start and end of these attachments. This approach evolved over the years and finally became the MIME standard. MIME is an acronym for “Multipurpose Internet Mail Extensions” and it is described in a series of RFCs, the first one being RFC 2045. Note that the first (and now obsoleted RFC) describing MIME is from June 1992 (RFC 1341) and almost 20 years seems to have been enough to make MIME widely supported (well, maybe not for newsgroup clients).

Using MIME an email can be constructed as a tree of body parts in which each body part has its own set of headers. There are numerous advantages. An attachment is a separate body part, an email can be embedded as a body part, a digital signature can be put in its own body part, and so forth.

The type of each body part is specified in a header named Content-Type. A simple plain text message has the following header (actually the header is not needed in this case since the values are the defaults):

Content-Type: text/plain; charset=us-ascii

Note that text is denoted as the type of the message and plain is the subtype.

A message with multiple body parts could have a “root” body part with the following header:

Content-Type: multipart/alternative

In this case the subtype alternative (RFC 2046) means that the body parts in the message should be interpreted as alternatives for message display with the last body part being the best one. In practice this is almost only used to provide two alternatives; a text/plain alternative and a text/html (or multipart/related) alternative. The latter is shown by email clients which support HTML (unless configured otherwise) and all other email clients can show the plain text alternative.

Markdown is not an alternative

It may be tempting to introduce a new subtype and then use a MIME type named text/markdown, but this would miss the point of Markdown being designed to be simple readable plain text. In particular, a text/plain alternative to the Markdown text would just be the Markdown text itself. It would be redundant. Markdown is more like an attribute of plain text. A better way to convey this information would be to use a content type parameter:

Content-Type: text/plain; charset=us-ascii; markup=markdown

This is similar to the format=flowed parameter introduced in RFC 3676 (which is also supported by MailMate).

You may wonder why this parameter is needed at all since an HTML body part is already generated and that is what most receiving email clients are going to display. This parameter is useful when replying to a Markdown message since it tells the receiving email client that it is safe to assume that the reply can be written using Markdown as well. Of course, MailMate is currently the only email client able to use this information.

Taking it one step further

Now, I’m still not a big fan of including HTML in emails which are essentially plain text. The solution outlined above is acceptable, but there is still going to be issues with how other email clients display the HTML generated and how it is handled in a reply (there is a lack of standardization in this area). Currently, MailMate generates simple HTML with no styling, but it is likely that this is not going to be good enough in practice. At least not for quoted text.

We could take it all one step further if we either know that the recipient has a Markdown-capable email client or if we simply don’t care. When MailMate displays a plain text email (or body part) with the markup=markdown parameter/value then it automatically converts the plain text to HTML before displaying it. The beauty of this is the simplicity of a raw message. Here is an example:

From: "Freron Software" <mm-feedback@freron.com>
To: "Freron Software" <mm-feedback@freron.com>
Subject: Taking it one step further...
Date: Wed, 21 Dec 2011 13:35:00 +0100
Message-ID: <DB451A45-1077-4C4D-90A2-44A18EEE68C9@freron.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed; markup=markdown
Content-Transfer-Encoding: 8bit
X-Mailer: MailMate (1.4r2651)

# Markdown in [MailMate][]

Note that *this* message has no HTML body part. Based on the 
`Content-Type` header, the receiving email client could do the 
conversion, but if it is not able to do so then this is still readable 
plain text.

In the eyes of an email client developer, this is a beautiful raw email. 
Nicely sorted headers and no hideous quoted-printable encoding (or 
base64) even though long lines and non-ASCII like æøå is used.

[MailMate]: http://freron.com

The responsibility of generating the HTML is now on the receiving email client. The obvious disadvantage is that it would not be supported by most email clients, but Markdown is also perfectly readable as plain text. Nicely formatted messages can be sent this way without using more space than normally used for a simple plain text message, but email clients recognizing Markdown can both display these messages nicely and create nicely formatted replies using Markdown as well.

Note that the use of Markdown does not exclude the possibility of a WYSIWYG editor, but it does require that the features of such an editor should be limited to what can be expressed as basic Markdown.

Generating an HTML body part is optional in MailMate. If you want to play with the Markdown settings in MailMate then note that the composer has a custom layout (see the “View ▸ Layout” menu) which makes previewing easy. If you do this make sure you also use the “View ▸ Show Raw Message” (⌥⌘U) and “View ▸ Show HTML Source” (⌃⌘U) menu items.

Standardization

When receiving a Markdown message MailMate uses sundown (a Markdown converter) to generate the HTML displayed. For now, HTML in the Markdown text is stripped, but the Markdown extensions fenced code and tables are allowed. Most importantly, sundown has been configured to respect newlines. This is different than the typical use of Markdown, but it ensures that email clients with no knowledge of Markdown can still handle paragraphs correctly. It also avoids problems with the concept of lazy blockquotes.

The choices above (made by me) illustrate the need for standardization if this feature should ever go beyond the small community of MailMate users. In particular, when HTML is not generated the sending and receiving email clients must agree upon the style of Markdown used. Essentially, 2 specifications are needed:

  • A specification of the markup parameter for the Content-Type header.
  • A specification of the Markdown language as used in emails.

The first one is fairly straightforward while the second one looks like a can of worms due to the many existing variations of Markdown and the current lack of strictly formal specification.

To properly introduce a markup parameter, it has to be registered and approved by IESG as described in section 3.1 in RFC 4288:

Registrations in the standards tree MUST be
approved by the IESG and MUST correspond to a formal publication by a
recognized standards body.

In the same RFC there is also the following relevant comment about parameters in section 4.3:

New parameters SHOULD NOT be defined as a way to introduce new
functionality in types registered in the standards tree, although new
parameters MAY be added to convey additional information that does
not otherwise change existing functionality.  An example of this
would be a "revision" parameter to indicate a revision level of an
external specification such as JPEG.

Time will tell if markup=markdown has a future beyond MailMate. For now, MailMate is a proof-of-concept which can be used to examine any practical problems which needs to be considered if standardizing either the markup parameter or Markdown.

Random thoughts and open questions

  • Inlining images works fine with external resources, but it could be made to work with embedded images as well by using content ids (cid:).
  • Should embedded HTML be allowed?
  • Should a default embedded CSS stylesheet be allowed?
  • It would be great if the editor had syntax highlighting, shortcuts for common actions, and so on.
  • MailMate currently does not handle replying to HTML-only messages very well. These could be converted to Markdown text before replying.

You are welcome to continue the list in the comments.

13 comments.

  1. Hi Benny,

    I just wanted to let you know that I bought mailmate because of the markdown feature, and am loving it, your choice of giving the option of sending rich text or raw markdown is very much appreciated.

    Regarding your random thoughts, I don’t see a reason to include CSS, same with embedding HTML. It seems to me that enough functionality is currently covered, after all – email is mainly about writing words, not so much about styling them.

    As to what to add to the list of open points, I saw you mention tagging with keywords on MacUpdate. This would be my number one request.

    Wishing you a successful year.

    Regards, -Martin

  2. @martin: Thanks for the support. What is your use case for markdown without HTML generation? (Given that only MailMate supports it when receiving a message.)

    I have already had several requests for better control of how the HTML is styled (when generated), but I’m glad you like the markdown-only option.

    Note the rudimentary tagging available via custom key bindings. But I do know tagging could/should have much better support.

  3. Benny,

    I’m using it with HTML generation, my point is that I don’t have a great interest in advanced generation features, given that it’s text that I am concerned with. That being said, I can see that people would be interested in more control via CSS, but personally I would rather that you add more ‘critical’ functionality instead of putting effort into sugarcoating emails ;D.

  4. One quick response — the problems with replying to HTML-only email are pretty severe, IMHO. Probably enough for me not to purchase at the end of my trial. I (mostly) love the product otherwise. Biggest complaint is that the UI often hangs when polling big mailboxes. Apple’s Mail client is the only one I’ve found that doesn’t do that.

  5. @Mike: Is your problem with HTML email that you cannot compose HTML replies or is it ‘just’ that the HTML is not converted into plain text (or Markdown)? The latter I would very much like to do something about, but an HTML composer is a very low priority.

    With respect to the UI hangs then I would appreciate a feedback email with additional details. In particular, I would like you to try out a recent test build of MailMate.

    Thanks for the feedback!

  6. @Benny — the HTML problem is that when I reply to an HTML-only email, the reply is empty, which deletes all the previous conversation. That just doesn’t work, in my workplace and I suspect in many others. I’m not bothered about composing HTML email — in fact, I quite like the markdown composer (although I wish there was a preview function to check I got the syntax right) — but HTML-only emails need to be converted to text so they can be included in the reply.

    Re the interface hangs, all I can say is from time to time it hangs (it’s doing it right now). The best explanation I can think of is that it’s polling my mailboxes. I’m on day 30 of my trial today, but I’d be happy to try a test build.

  7. @Mike: Ok, the plan is to convert HTML to plain text (Markdown) although I’m not yet sure how well it’ll work in practice. Time will tell. As a workaround, it is possible to select the text of a message before replying, but it won’t always work well since this conversion is a bit primitive (which is why this is not currently done by default for HTML-only messages).

    Note that ⌘1 in the composer provides a preview view in the composer (updated when saving).

    I’ll write to you privately about a test build.

  8. I just got my trial license of MailMate today after hearing that Sparrow, which I didn’t really like all that much in the first place, was acquired by Google and that future development would cease. I haven’t ran into the replying to HTML only email issue, but then I’ve only been using MailMate for less than a day. However, as I was reading through these comments I noticed that you plan on adding HTML to Markdown conversion. I recently came across a tool that does just that, which may be helpful in your development of this feature. It’s included in Brett Terpestra’s Markdown Services: http://is.gd/FGet7d

    Keep up the great work!

  9. @Michael: Thanks for the link and thanks for trying out MailMate. MailMate currently uses html2text by Aaron Swartz. It is a modified version since an email client cannot quite use Markdown as-is.

  10. Just because you don’t like how some people design their e-mails, you make one waste all that time!

    This is a huge deal breaker for me!

    Why not give little HTML control like making text bold, lists e.t.c. like Mail.app does.

    And maybe in the hidden preferences full access to HTML editing.

  11. @Erich: I’m sorry your expectations of MailMate did not match reality. If implementing an HTML editor was as easy as flipping a switch then I would be willing to offer it as a preference right now, but it’s far from that easy. It’s a major feature and if implemented it would always be a broken feature, because HTML for emails is a broken concept.

    That said, I’m aware that it’s important to many potential users of MailMate and I might find some way to at least narrow the gap, but you should not expect an HTML editor like the one in Apple Mail.

  12. Concerning using markdown in replies, would it be possible to set MailMate to ignoring problems, rather than disabling markdown or displaying a warning?

  13. @Balder: I’m not yet willing to do that. It’s important that the user does not send quoted text unintentionally interpreted as Markdown. (Note 1: If you edit the message in a way that removes the problem then the banner automatically disappears. Note 2: In any case, you don’t need to dismiss the banner. It can be ignored.)

Post a comment.

Crowd Funding Campaign

Reviews

Macworld wrote:
  • As email search abilities go, this seems less like bringing a gun to a knife fight, and more like thundering into that particular duel at the controls of a helicopter gunship.
  • Needless to say, the program never crashed, glitched, or gave me any sort of trouble during my tests. MailMate does not know weakness.
About.com wrote:
  • MailMate is a refreshingly efficient way to deal with mail in IMAP accounts, all with handy keyboard shortcuts, superior search and precise smart folders.
  • As a testament to its raw power, MailMate does not stop at From: or body text, of course; instead, you can have it match just names or first names, email addresses, domain names, parts of domain names, quoted text and what not.
Lifehacker wrote:
  • Search is amazing, providing you with a ridiculous amount of control.
  • Practically everything can be accessed using the keyboard.

User Quotes

Rob Schumann (MacUpdate) wrote:
  • I […] felt immediately at home with its much more capable filtering system that can be directed at any folder, not just the inbox, and the ability to construct complex any/all rules that can be nested multiple levels deep.
  • […] in general it just feels significantly faster than Mail.app
David Levy (email) wrote:
  • I've been stress testing MailMate with a variable and highly intermittent connection. […] My overall assessment is that MailMate has been markedly superior [to Apple Mail] in this use modality […]
Tom Borowski (blog) wrote:
  • […]this way of handling signatures is far superior to any other approach I’ve seen so far. Why hardly anyone implements it in this way is a mystery to me.
TheBrix2008 (MacUpdate) wrote:
  • […] on first run, MailMate spotted my Mail.app IMAP settings, prompted for import, then got to work without fuss.
  • it has a phenomenal array of filtering, statistics and views […] My favourite is “Correspondence”, which picks out all email from your current correspondent and lists it in a separate pane… instantly.
  • The author should be rewarded for having the courage to start from scratch.