GENERAL FAQ

Given 15 seconds, what should I know?

What is MarkMail?

What powers MarkMail?

Why did you create MarkMail?

Will you load more lists?

Why would I use MarkMail instead of Google or some other search engine?

Many reasons:

1. Scope. Many of the emails we host aren't in Google's index. Or at least they weren't in Google's index before we went online. But even now with Google and the rest spidering us you'll still want to use MarkMail because...

2. Speciality. Search engines only index public email if it appears on an HTML page somewhere. Google doesn't know why the words are on the page - doesn't know the sender, the date, the subject, what's in the body of the attachments, and so on. We do. So if you want to search for an issue with Apache James involving respooling, with MarkMail you type list:james respool. And hey, if you remember the email you want came from your friend James you type list:james from:james respool and you're there. You can be very specific because MarkMail is a site built specifically for email.

3. Structure. We know about the structure of email. This lets us exclude searching those annoying "copyright notice" footers at the bottom of emails, add relevance weight to the important message headers, reduce the importance of quoted message text, or let you exclude quoted message text if you'd prefer (see the opt:noquote feature). We even understand the structure inside attachment files.

4. Analytics. Where else have you seen a chart showing the historical activity corresponding to any arbitrary query you type in? It's a ton of fun to watch activity trends for lists, people, and keywords, or any mix of the three. Don't forget to use a minus sign to negate a query: javaone -from:sun.com.

5. Attachments. Do a search for emails with PowerPoint attachments (hint, the query is ext:ppt), click on the attachment link, and watch how you can view the attachment without leaving the search results! Same goes for Word files and PDFs. If you include a search term, we'll even show you which slides include the term: ext:ppt axis.

6. Convenience. We've worked to build the MarkMail site as an immersive experience. We don't like how regular search engines make you click, read, hit the Back button, only to click, read, and hit the Back button again. With MarkMail all the important information stays in front of you all the time, results right next to hits.

How do I post a reply to a message I've found?

MarkMail hosts list archives in read-only mode and doesn't yet provide a mechanism for you to participate in mailing list discussions directly. To post an email to one of the lists, you need to use your normal mail program (Outlook, Thunderbird, etc) to send an mail to the list.

You can derive the listname from our archive name. If the archive name is com.domain.project.list then the list's public email address is list@project.domain.xcom.

Is there a discussion forum?

I found a bug, how do I file it?

What's next for the site?

How will I know when you add new features or launch new stuff?

How many emails and lists do you manage?

The archive includes many millions of emails across thousands of lists. Current counts are always displayed on the home page. New messages to those lists are added continuously throughout the day, and are immediately available for search. We also run a version of MarkMail inside Mark Logic for our internal mailing lists (but it's a lot smaller).

How many senders does that make?

In the original 4,000,000 Apache archives we counted a little over 150,000 unique poster names. In 2007 we saw almost 20,000 posters not seen in years past (and 12,000 repeat posters).

How many emails have attachments?

We find roughly 1% of emails have attachments.

Which browsers do you support?

We strive to work with Firefox 1.5+, Internet Explorer 6+, Opera 9+, and Safari 1.2+. More recent browsers sometimes have more features. The site works best if you leave your font size normal.

How can I help?

Can I buy one?

What search syntax are you using, and what can it do?

If you're a beginner, just type your word or phrase into the search box. We'll show you relevant results, and you can always use the dynamic faceted navigation links in the left side analytics pane to get more specific based on list, sender, attachment types, and message types.

If you're using the MarkMail gadget on iGoogle.com, the search box can be found in the Edit Settings dialog. Clicking on any message or thread will bring you through to markmail.org. The tabs in the gadget let you see some of the same analytics you can see in the faceted navigation at MarkMail.

Over time you may want to learn our search syntax, explained below. We support searching for...

Search terms are case insensitive. They are also stemmed, meaning a search for "issue" will additionally match "issues" and vice-versa.

Constraints are ANDed together except in the case of multiple fielded constraints of the same type which will be OR'd together. So for example the query list:tomcat list:struts includes both tomcat and struts lists.

Any of the above constraints may be negated with a minus sign indicating that matching messages should be excluded from the results. For example, the following query finds Tomcat traffic excluding CVS or SVN commits: list:tomcat -type:checkins. Negations are not OR'd; more negations always exclude more messages.

By default search results are ordered by relevance, most relevant first. If the search string contains order:date-forward the results will be sorted chronologically, oldest to newest; if the string contains order:date-backward it's the reverse. If that's too much typing you can try order:df and order:db. Searches that don't have a meaningful relevance score, like list:tomcat, are by default sorted date-backward.

If the search string contains the special opt:noquote argument the results will avoid searching text that appears in quoted messages. This can be used to find messages where a particular person said something.

Lastly, the opt:nostem argument in a search indicates you want the search without "stemming", meaning a search for run won't match runs or ran as would normally happen. This lets you be a bit more precise in what you're looking for.

Are there any useful keyboard shortcuts?

We're geeks, so we love keyboard shortcuts. Here's a list:

Can I put a MarkMail search box on my site?

Sure, and that can be convenient if you want to give people the opportunity to search messages without visiting the MarkMail site first. Just copy and paste this HTML into your site:

<form action="http://markmail.org/search/list:tomcat"> Search MarkMail: <input type="text" name="q" size="50"/> <input type="submit" value="Search"/> </form>

You should replace "tomcat" with the name of your list or project. You can adjust the default search constraint too. For example, the following searches messages sent by a particular person, in this case Sam Ruby:

<form action="http://markmail.org/search/from:rubys"> Search MarkMail: <input type="text" name="q" size="50"/> <input type="submit" value="Search"/> </form>

Replace "rubys" with the name of your desired poster. You can even combine terms, even negations, into the default search constraint.

How do I add MarkMail to my browser's search box?

How do I link to MarkMail?

We love incoming links. You'll probably want to link to one of our project-based homepages:

(You get the idea.)

How do I link to an email or thread?

Can you load my private email?

Can I just browse the lists and messages?

Any known issues?

There are a few known issues:

• Use of large fonts or larger DPI settings can cause text to exceed the allotted bounds
• Firefox can report "Transferring data from markmail.org..." in the browser's status bar even after all data has been transferred

How do I request that content be removed?

TECHIE FAQ

What's hard about searching email?

Email doesn't work well in a relational model because there's too much free text (all those words in the message). It doesn't work well in a search engine either because there's too much ad hoc structure (headers, footers, paragraphs) and hierarchy (attachment files containing pages containing paragraphs). And if you try to marry relational with search you only get into trouble with slow joins across the systems and indexes being out of sync and slow to update.

We've found email works naturally as XML, with the structure represented by elements and hierarchy by the nesting of the elements. We're hoping that on this solid base we'll be able to push the envelope for email management.

How do you store the emails?

Each email is stored an XML document inside MarkLogic Server. Every document representing a message has a <message> root element with attributes to hold the message's unique id, thread id, list name, date, and so on.

Underneath the root, there's a <headers> element containing more elements, one for each of the email's headers. For special headers like <from> we examine the content and extract it in a normalized format, so for example <from> has @address and @personal attributes holding the sender's email and real name. We use these attributes for search and display instead of the raw From: header value.

Following <headers> there resides an optional <attachments> element that holds information about the email's attachments - pointers to their original versions, any image representations, and (for attachments we know about like DOC, PPT, and PDF) the content on each page. The binary files reside in MarkLogic Server also.

Lastly, under the root there's a <body> element that hold the message content. The body isn't held as simple text but rather as <para> elements and <quotepara> elements with attributes indicating the quote level. We also have <footer> elements. Each footer has a @type attribute to let us know if it's a person's signature block, a confidentiality statement, a footer added by a free hosting provider, a list subscription management line, or several other classifications. Signature footers we display in italics, confidentiality footers we display in gray, and some footers aren't displayed at all. Inline we've added markup for recognized emails and URLs. This enables targeted searching against these items as well as custom display (i.e. email obfuscation). A fun query we can do with all this markup is: given a person's name find the email address embedded within their most recent footer. (Note that we haven't exposed this query to the public yet. Let us know if you want us to.)

What generates the charts?

The raw data for the charts comes from a MarkLogic Server product feature called lexicons. Lexicons enable fast calculation of the distinct values for a specific element or attribute along with the occurrence frequency of each value. To generate the historical activity chart, we ask for the distinct month values and frequencies across the emails satisfying the user's query.

We pass that raw data as XML to a commercial Flash charting library whose code we have modified to better suit our purposes.

How do you load the emails?

A Java program converts email to XML and loads them into MarkLogic Server. The program can load emails from an mbox archive or catch new emails as they come across the wire. We subscribe a user to each archived list in order to receive the new messages.

The delay between message receipt and searchability is less than a second. You're always searching the very latest.

How do you do the search?

What runs the front-end?

We actually use MarkLogic Server to generate the HTML pages. Well, technically they're XHTML. Our back-end data model holds emails as XML, each visitor's browser requires (X)HTML, and in between we have the XML-centric scripting language XQuery. It seems wasteful and unnecessarily difficult to marshall XML into objects or strings and then marshall it back out as XML. Instead we query XML, process XML in an XML-aware scripting language, and output XML as XHTML. It's the only time in our web lives we haven't suffered an impedence mismatch at some level of a web architecture. And as a bonus, the XQuery language always outputs well-formed XHTML.

How will this scale?

At this point we're running on a small MarkLogic Server cluster, utilizing the inbuilt capabilities of MarkLogic Server to scale as necessary for user load and email traffic. The largest clusters deployed at some of our customer projects exceed 100 machines. We have loads of runway left.

This sounds like fun, are you hiring?