With all respect for the people working at Mozilla/Thunderbird and fully understand the limitation they are faced with, I would like to see a more detailed mission statement for the products (TB/LG) and the future of it. Only expressing TB is for individual users, SOHO and not for the Enterprise is a very vague  statement

I was going to respond to that in the thread, but I got wordy so I posted this blog entry instead.

At the recently completed Mozilla Summit, variations of this request were made by many people that are close to the Thunderbird project (including myself). But after listening to several days of Firefox people extolling the virtues of moving everything to the browser, and being “more like the web”, I have a new appreciation of how difficult the development of a vision statement is for the Thunderbird team.

The standard game plan that Mozilla projects are expected to follow is to develop an application with a significantly high market share so that they can use their market influence to fight for the rights of individual users. Mozilla is a fascinating organization as a hybrid commercial/public interest organization, and they take their values quite seriously.

Unfortunately Thunderbird, which is the only real product that MoMo currently has, hovers around 6 million users, which is much lower than the number they believe are necessary to have the influence they would like (I have heard 100 million users as a goal). Nobody currently has a concrete plan to develop a product with 100 million users. So the current strategy (as I see it) is to try a series of experiments to try to develop some concepts that might be used to specify the 100 million user product. I can’t resist naming things (my wife calls me “an Adam”), so let’s call this future product by the code name “Gigabird”.

One such experiment is Raindrop. Other experiments are going on in extensions to Thunderbird, which seem to be focusing on changes to the user interface. Right now, that is where the vast majority of the developer resources are focused at MoMo.

So if your real strategic mission is to develop Gigabird, what do you do with your legacy product Thunderbird? The big problem is that the “ordinary users” that are the primary focus of Firefox (and by implication also MoMo) are migrating away from email for many forms of messaging to other media – Twitter, Facebook, text messages, web forums, blog comments, etc. The hardcore users of email (who are likely to continue to use a desktop client) are sitting in office cubicles, yet going after these “enterprise” users is counter-cultural for Mozilla.

So what are the strategic choices available to MoMo?

the HailMary

The goal here is to try to come up with one or more really clever innovations that will form the basis of Gigabird. This is, after all, the way that some of the new messaging formats have occurred, with Twitter as the poster child. Using these innovations as a base, the basic plan for Gigabird will be formulated at some point in the future. This is the current MoMo strategy, at least as I see it. Given the existing Mozilla culture, I would probably do this as well. (Warning: should this strategy every become publicly revealed, the director will disavow any knowledge of these actions.)

the AboutFace

Here you notice that the values-driven direction that you felt so passionate about is actually not going to get you anywhere, so you make a major readjustment in values to allow you to pragmatically accept a new direction. Such moves have been done by Mozilla in the past, and are part of the standard corporate Myth propagated by Mitchell Baker (the story about how in the early days they were adamant about never shipping a binary). The application to MoMo could be to accept that what they have is an email product, and their future users are going to be sitting in cubicles. Users in cubicles should have rights too, so there could be a valid Mozilla Foundation purpose in fighting for the rights of these “enterprise users” and let Thunderbird develop into an enterprise product.

the SlowPlod

This is the direction that existing Thunderbird users are hoping for. The ultimate goal is to slowly improve Thunderbird until it is undeniably the best email client around. You fix any important bugs.  You support all of the hot new messaging concepts. You spiff up the user interface, incrementally adding new features that provide small improvements to usability. You keep your power uses happy with lots of extensions for specialized purposes. It’s pretty clear that dmose does not believe that he has sufficient resources to pursue this strategy, nor is he likely to have them in the foreseeable future. The Thunderbird code base is also really hard to adapt to these new media (witness the struggles that I have had or jcranmer’s blog ). I think that the MoMo team wishes us well, but believes that the future lies elsewhere.

the VacuumTube

Just because you can’t change the direction of humanity does not mean that you have nothing. Vacuum tubes are long gone – yet the guitar player at my church proudly uses his fancy amp with glowing tubes showing through plexiglass. The company that bought one of my previous businesses had also previously purchased a manufacturer of vacuum tubes, which had morphed into specialized purposes like lamps for spectroscopy, and nuclear-warfare-resistant cathode ray tubes. Email clients will be with us forever, and in the hands of people who love them could have a useful future in various niches.

My Prediction

MoMo will pursue the HailMary until they have enough ideas to formulate a real plan. At that point, they will want to devote all of their resources to Gigabird, and be looking for an honorable way to retreat from Thunderbird – which will be a variation of the VacuumTube. The likely retreat will probably be some sort of future custodianship by a conglomeration of companies that provide a freemium strategy. So if there was a basic, free Thunderbird product that could be enhanced with addons with commercial value (like my Exchange Web Services product, or Postbox as a Thunderbird addon), then MoMo could pursue their vision without abandoning their Thunderbird users, and let companies like MesQuilla and Postbox support Thunderbird.

EWS messages do not come from the server in RFC-822 format, so it seems like a pity to store them that way, though that is the common method used in the rest of the mailnews codebase. Instead, I decided to implement a local storage scheme based on SQLite and Mozilla’s Storage interface. Andrew Sutherland has done a lot of great work setting up an environment similar to this for the gloda database, so there are lots of good examples to pull from. Also, because the datamodel for EWS includes not only messages but also Calendar and Contact items, I can have a common database infrastucture that I can leverage over those other pieces once I get it working for the messaging part.

I’ve now replaced my previous in-memory datastore for message metadata with an SQLite version. This is equivalent to the datastore module in gloda, and the data it is storing is like the RFC-822 headers. I still have to do the storage for the body, and also hook this up with folder change state so that the code knows that it has data it can trust.

As I have done this, I’ve had a new set of insights into the relationship of the various objects in the Mozilla mailnews world (which I sometimes call Skink). Previously, I had sort of expected that the natural progression of gloda would be to slowly displace the role of the message summary database, nsIMsgDBHdr.  But now I see that a more natural progression would be for SQLite to be used as a replacement for the local mailstore (currently mbox, with maildir support moving forward as well.) Really the main issue is the async nature of the SQLite calls, which sort of precludes its easy use as a replacement for nsIMsgDBHdr. But the datastores are typically accessed async anyway. If the message metadata in the message stores was stored primarily in SQLite format, as I will be doing, then it would be much easier to hookup an SQLite-based global search facility to all of these databases. Yes that is what gloda does now, but it has to go through all of the work to maintain a separate version of everything. Why have three copies of everything (Mork, MBox, and Gloda) when you could only have two (Mork and Gloda)?

As another insight, while looking through the gloda code I noticed that a JSON object was being saved to store some of the items. I though that was a good idea at first – but then I tried to write a simple serializer to convert from my internal native format to JSON objects, and saw that it was not going to be an easy project. But then I remembered that SOAP is really just a mechanism to serialize typed objects, and I already have a SOAP encoder and decoder! So instead of using JSON, I use objects serialized with my SOAP XML encoder to store unindexed items in my SQLite store. So a message (sans body) ends up looking like this as a TEXT item in SQLite:

<Message xmlns="http://schemas.microsoft.com/exchange/services/2006/types">
 <Subject>Postini First Junk Email Safely Quarantined</Subject>
 <DateTimeReceived>2010-06-04T22:19:30Z</DateTimeReceived>
 <Size>2612</Size>
 <Importance>Normal</Importance>
 <DisplayTo>Kent James</DisplayTo>
 <Culture>en-US</Culture>
 <Sender>
  <Mailbox>
   <Name>Postini Support</Name>
   <EmailAddress>noreply@hostedmsexchange.com</EmailAddress>
   <RoutingType>SMTP</RoutingType>
  </Mailbox>
 </Sender>
 <ToRecipients>
  <Mailbox>
   <Name>Kent James</Name>
   <EmailAddress>rkentjames@caspia.org</EmailAddress>
   <RoutingType>SMTP</RoutingType>
  </Mailbox>
 </ToRecipients>
 <From>
  <Mailbox>
   <Name>Postini Support</Name>
   <EmailAddress>noreply@hostedmsexchange.com</EmailAddress>
   <RoutingType>SMTP</RoutingType>
  </Mailbox>
 </From>
 <InternetMessageId>&lt;0c34b5a4-5f3c-4654-bf9d-99c9a8cb439b@HUB02.4emm.local&gt;
 </InternetMessageId>
 <IsRead>1</IsRead>
</Message>

At first it bothered me to save what is essentially a duplicate of what is coming over the wire, but why not? It’s not conceptually any different than RFC-822, or JSON, in function.

Something I’ve found myself doing at work is creating a new filter for every folder I create. I work on technical cases and for each new case number I create a new folder and have all emails with that case number go into that folder.  The crappy part about it is that I literally have hundreds of cases I deal with, and hence hundreds of filters.  You’ve already got the regex match criteria in filtaqulla, I’d love to be able to take that match criteria and use whatever string it matches as the destination folder action criteria. Sample subject line contains the following: 2010-0609-518

One of the features added to my extension FiltaQuilla recently is the ability to add custom javascript actions. I thought that I would give a shot at doing this request as a custom javascript action. The author of the email was expecting this to get linked to the regex search term, but that is not the easiest way to do it. It is easier to just let the filter action also do the regex search.

After installing FiltaQuilla, you need to enable the custom javascript action, which is done on the Addon options page. After that, you can add a javascript custom action in the filter editor.

Then you need to write the javascript code for the action. I’ve done similar work in the past, so that took me about an hour to get correct. Then insert the code into the action field for the javascript action, and it’s ready to go! You have to have some sort of search in the filter as well. You could just search for everything if you wanted, or you could add a regex search for the precise term if you want. It doesn’t matter much, as all of the work is really being done in the action. You just need to make sure that the action gets called for each message that might need to be moved. There’s more documentation of this feature available at the FiltaQuilla page on this site.

The code I came up with is:

let digitsRegex = /20\d\d\-[0-1]\d[0-3]\d\-\d\d\d/;
let acctmgr = Cc["@mozilla.org/messenger/account-manager;1"]
                .getService(Ci.nsIMsgAccountManager);
let copyService = Cc["@mozilla.org/messenger/messagecopyservice;1"]
                    .getService(Ci.nsIMsgCopyService);
for (let index = 0; index < msgHdrs.length; index++)
{
  let hdr = msgHdrs.queryElementAt(index, Ci.nsIMsgDBHdr);
  let theDigits = digitsRegex.exec(hdr.subject);
  let folders = acctmgr.allFolders;
  let length = folders.length;
  for (let i = 0; i < length; i++) {
    let folder = folders.queryElementAt(i, Ci.nsIMsgFolder);
    if (folder.name == theDigits) {
      let messages = Cc["@mozilla.org/array;1"]
                       .createInstance(Ci.nsIMutableArray);
      messages.appendElement(hdr, false);
      copyService.CopyMessages(hdr.folder, messages, folder, true,
                               null, msgWindow, false);
    }
  }
}

With this filter, if you create a folder with a name like “2010-0616-001″, and your message has that in the subject, the message will get moved to that folder.

If you are having trouble getting it to work, an easy way to debug is just to insert statements like this into the javascript:

Cu.reportError("I am here");

Those printouts will show up on the error console, so you should be able to see if your filter action code is working or not.

Why no header information? Because the header summary, for reasons that I cannot explain, reads the message file directly – and assumes that the message is in RFC 2822 format. But Exchange Server has already done all of that parsing, and separated the message into its components, including metadata (like subject), body, and attachments.

An important issue going forward for the Mozilla Mailnews (Skink) codebase is, what exactly is the thingy that Thunderbird purports to be? Is it a processor of RFC 2822 messages? Or is it something more, that easily incorporates other forms of communications? I hope to write a longer post on that issue soon.

]]> http://mesquilla.com/2010/06/04/reading-the-body-mailnews-exchange-support/feed/ 7 http://mesquilla.com/2010/06/01/lessons-from-google-thunderbird-as-a-firefox-extension/ http://mesquilla.com/2010/06/01/lessons-from-google-thunderbird-as-a-firefox-extension/#comments Tue, 01 Jun 2010 19:39:16 +0000 rkent http://mesquilla.com/?p=829 During my current trek to the Bay Area, I seem to be bombarded with news about Google. I was particularly interested in the different way that Google views its business model, and that got me thinking about how Thunderbird fits into Mozilla’s business model.

In an extensive article in Atlantic magazine (I always read the Atlantic while travelling), James Fallow describes efforts that Google is undertaking to try to revive the viability of news reporting as a professional activity. A lot of this revolves around different methods of bundling content with monetization schemes. In the traditional newspaper model, news is one of many parts of the bundle, with monetization provided by various advertising streams. The internet has provided many ways to unbundle the various pieces of the newspaper, with the result that the hard news content no longer has a sufficient bundling with the monetization schemes.

Google may be viewed as a new kind of bundle, with its many services (search, maps, email, news, etc.) all part of the bundle of services that are attached to the monetization scheme of search advertising. The Atlantic article notes that:

Virtually all of Google’s (enormous) revenue comes from a tiny handful of its activities: mainly the searches people conduct when they’re looking for something to buy. That money subsidizes all the other services the company offers …

That requires a new way of thinking about understanding the relationship of traditional “cost centers” to “revenue”. When Google’s Andy Rubin was asked to justify the expense of Android development in this interview in the San Jose Mercury News, he responded:

So all we have to focus on are those types of innovations that scale for large audiences, and … the revenue crank just turns.

So what is Mozilla’s bundle and monetization scheme, and how does Thunderbird contribute to that bundle?

Mozilla’s main bundle that is attached to a monetization scheme is the FireFox browser, which is monetized through people’s use of a particular search provider when connected through the bundle. Thunderbird (and its related Mozilla Messaging cost center) do not currently contribute anything directly to that bundle and its monetization scheme.

I don’t believe that Mozilla’s subsidization of Thunderbird can continue indefinitely. Perhaps that was possible for awhile, but Mozilla’s bundle can be expected to come under continual competitive attack, which will eventually force them to marshall the resources needed to defend the bundle. Mozilla Messaging and Thunderbird need to be part of the solution, and not just a cost drag, when the tough decisions are being made to defend the bundle.

If we think like Google, it’s pretty obvious what the step is that is needed for Thunderbird to become a major contributor to Mozilla’s main business of “Firefox the bundle”.  We need to figure out ways to attract people to the bundle, and get them to linger as long as possible on the bundle while the “revenue crank just turns.” The obvious solution is that the underlying communications code that drives Thunderbird needs to be repackaged as a FireFox extension. (Since I call that codebase Skink, let’s temporarily use the code name SkinkFox to describe the Thunderbird-as-Firefox-extension product.)  SkinkFox could have the same relationship to Thunderbird that Lightning has to Sunbird: the same product packaged both standalone, and as an extension.

“That’s just SeaMonkey!” you say. Well no it is not. If you think like an engineer and focus on features, then I guess SkinkFox is similar to SeaMonkey. But thinking like a marketer, the goal of this is to build the FireFox brandname and revenue stream, and SeaMonkey does not contribute to that while SkinkFox does.

Technically, great efforts are already being made to reduce the barriers that keep the mailnews (Skink) codebase separate from FireFox. I don’t think it would be a huge leap to figure out how to merge Thunderbird’s user interface and tab system into FireFox’s.

As Thunderbird enters a critical rethinking in the next few months in the aftermath of Thunderbird 3.0/3.1, SkinkFox is one of a variety of thoughts I will be giving on possible futures for messaging at Mozilla.

So here’s a picture to show where I am at. The two messages are real messages that exist on the Exchange server, and have been read by my extension. The message body is not real:

One problem that I had to solve is to make the nsMsgDatabase object extendable. The standard code for that does lots of static casts to the pointer for the XPCOM object, which either caused crashes, or duplicate database caches, depending on the approach that I took to get around that.  I ended up writing a patch for the base code to solve this, which presumably I will land for TB 3.2. I still think there must be some way to cleverly avoid this problem without patching the base code, but I could not figure it out.

I’m not real happy about requiring a base patch for this work, because I had hoped to be able to release a prototype for this on Thunderbird 3.1 / comm-central 1.9.2, even though the main release should be in Thunderbird 3.2/ comm-central 1.9.3. Maybe I can still figure out a solution that does not require a base patch.

Fun with Branding

I’ve talked about the Colonial layer and the Native layer in my application. What’s above the Colonial layer? I want to be able to distinguish between the code shared with Firefox, and the specific code that is used by messaging applications (Thunderbird and SeaMonkey), because I want portions of my work to be loadable into a FireFox extension (just in case). It could be called by the name of the main directory (mailnews) or that directory’s parent (comm-central). Personally I find mailnews very unattractive as a brand. (A bug asks “What’s the name of MailNews?”)

Although I got no traction when I suggested it on m.d.a.thunderbird, I’m going to experiment personally with calling the Mozilla code that implements messaging functionality, and is not included in Firefox, as the Skink layer instead of “mailnews” (following Mozilla’s lizard analogy and Gecko). Yes I am being bullheaded, but this is my personal blog, so I can do what I want, right?

I also managed to register the domain exquilla.com, as ExQuilla is the logical brand for my Exchange Server integration functionality.

]]> http://mesquilla.com/2010/05/21/thread-pane-and-nsimsgdbhdrnsimsgdatabase-mailnews-exchange-support/feed/ 2 http://mesquilla.com/2010/05/20/javascript-custom-filter-action-in-new-filtaquilla/ http://mesquilla.com/2010/05/20/javascript-custom-filter-action-in-new-filtaquilla/#comments Thu, 20 May 2010 17:00:05 +0000 rkent http://mesquilla.com/?p=788 I recently released a new version of FiltaQuilla for Thunderbird and SeaMonkey, and one of its cool new features is a custom filter action that runs arbitrary JavaScript. This relies on new backend features, so you’ll need TB 3.1 beta2 or later for this to work.

Mitra recently pinged bug 249191 asking for a filter action that affects remote content. That looked like a good application of a javascript filter action, so I tried it out.

Looking at the underlying code in nsMsgContentPolicy.cpp, there is a message header property “remoteContentPolicy” that can be used to enable showing of remote content for a message. Unfortunately disallowing content appears more complex, as the address book whitelisting appears to override the message header value. But really I think that allowing remote content on a per message basis is probably the more common action need, so let me demo that.

In my documentation for FiltaQuilla, I show a simple basic javascript example of a custom action that affects a message header subject property:

for (let index = 0; index < msgHdrs.length; index++)
{
  let hdr = msgHdrs.queryElementAt(index, Ci.nsIMsgDBHdr);
  hdr.subject = "[Hello, world] " + hdr.subject;
}

It’s fairly easy to modify this to set the remoteContentPolicy instead. The result is:

for (let index = 0; index < msgHdrs.length; index++)
{
 let hdr = msgHdrs.queryElementAt(index, Ci.nsIMsgDBHdr);
 hdr.setUint32Property("remoteContentPolicy", 2);
}

So if you enter this as a JavaScript custom filter in JunQuilla, you can enable remote content on messages controlled by a filter.

I tested it on a spam message that had “Exxon” in the subject. The resulting filter, including the javascript, looks like this:

I’m sure that defining these filters is beyond the skill of all but a few advanced users, but once the code is written it’s pretty much just cut and paste to get this to work. If I collect a few more examples, I suppose I should create a webpage to show them.

Thanks to jcranmer, Standard8, and bienvenu on IRC channel #maildev for helping to point me to the base code that manages remote content policy.

The account manager interface was apparently written by an ancient advanced intelligence (oops I mean here), that used mysterious technologies that are no longer understood by humankind. Renowned cybernetic archaeologist Joshua Cranmer has tried to unravel some of its mysteries, but that provided me more of a hint of where to look for issues than a roadmap of what I needed to really do. So I was forced into my last resort – reading the code and trying to figure it out. There were some surprises.

First, a brief description of terms. When you first create a new account type (unless you are using the new email interface for Thunderbird 3.0), you enter into the account wizard. That sets up up all of the initial values. After that, if you try to edit those you end up in the account manager. They are closely-related, but distinct, interfaces.

First, how do you add a new account type at all to these interfaces? It turns out that kill-rdf missed a spot, and it is still possible to use rdf to add a new hook into the account wizard. Extensions can have a subdirectory “isp”, and into that subdirectory you can inject an rdf file that contains some information about your account manager hook. The rss implementation gives some hints of how that is done.

Once you add the rdf file, your new account type shows up in the account setup under “add other account …”:

The rdf file that you add contains all sorts of information about the account that you are trying to setup. Through some magic, the rdf items get linked directly (at least in theory) to default values for the main account objects.

But unfortunately, extensibility of the account manager has never really been carefully supported, so things just don’t work out quite the way you would like. Some examples:

• My account type represents a new protocol type, “ews” instead of “imap” or “pop3″. But at one point the server page looks into the standard file “messenger.properties” for strings like “serverType-imap=IMAP Mail Server” which of course do not exist for “serverType-ews”.
• The pages that show in the account wizard are partly hard wired for the existing account types.
• Although the account manager has some attempts at extensibility, most of its options are setup as a list of account types to exclude when showing certain screens. This is maddening if you are adding a new account type, because since your account type previously did not exist, of course it does not show up in all of the exclusion lists.

So this intricately designed infrastructure that the ancients designed is basically unusable at present to add new account types, as far as I can tell. For each little option that you want to add, you need to trace for hours through very abstract code to figure out whether it will work, or if there is some hidden gotcha. Ultimately, it really is easier just to throw it out and start over again.

Fortunately, through the wonders of javascript function replacements, I was able to create enough hooks to get my new account added, without requiring a patch to the core code. Here is a rough outline of the steps that I took:

1. First, the account wizard seems to use the existence of identities to decide if your have a mail account type, and if you do it forces you to use the standard mail account wizard types. I managed to override this in javascript.
2. After you do that, through some bizarre quirk the rdf property “NC:wizardShortName” is actually a list of the wizard pages that you want to appear for custom account types. So I can add a line to the rdf file:

   <NC:wizardShortName>identitypage,ewsServerPage</NC:wizardShortName>

   and that will allow me to specify precisely which pages are shown in the wizard. “ewsServerPage” is a custom page that I add though an overlay.

3. In the account manager, there is no easy way to control which subpages appear in the account tree for a custom account type. Once again I can use a javascript function hook to manually adjust the available pages. I wanted to remove the default server page (which asks if I want pop3 or imap), and add my own page in its place. I also wanted to be able to remove pages I did not want. That lets me control the list precisely, so that I now get this account tree:

I suppose dmose would like me to organize all of this into a sample extension that others could use. One of the reasons that it is hard to get motivated to do this is that there is no real plan for the future of any of this, yet I hear rumors that a lot of this is scheduled for demolition (rdf, account wizard, even binary extensions that my entire application relies on.) Plus I’m not sure any sane person would want to do the kludges that I am doing to get this to work. Is there even any real demand for custom account types in the mailnews code, beyond what I am doing?

Anyway, the good news for the Exchange Web Services integration project is that I now have basic hooks into the account manager interface, so that I can easily add custom changes to the account manager setup to support Exchange Server. The most basic information (user name and server address) can be entered through the account manager interface, and used to access the correct Exchange server.

serverType-imap=IMAP Mail Server

Mailnews Traditional Filter and Search – Big Issues for Enhancements

1. Integrate full boolean search into Search Term editor

• Bug 297852 – Search/Filter and Boolean expressions

The mailnews backend traditional search logic supports full boolean expressions, but the user interface has never been updated to allow this. With asuth’s pending update to the quick search UI, making extensive use of boolean expressions internally, we are getting to the point where quick search can express certain searches that so-called “advanced search” cannot (and vice versa.)

2. Store filters globally, and apply them to selected account/folder trees/contexts

• Bug 11039 – Filter outgoing messages (perhaps to use a different Sent/FCC folder)
• Bug 166842 – [RFE] Ability to import and export message filters
• Bug 129883 – Filters Should be Shareable Across Multiple E-mail Accounts
• Bug 59365 – Filters should have triggers
• Bug 34973 – (GlobalFilters) global message filter option

Filters are currently stored per account (except in news, where they can also be per-folder). But when people define a particular filter, they frequently want it to apply to more than one account. There is no easy way currently to do this, short of recreating all filters on each account. Moving filters is also a pain when accounts are reorganized.

I would store filters in a single location, and provide some sort of UI to define where and when each filter is applied. That would not only be the account or folder, but also the context (such as incoming, manual, sending, or triggered.)

3. Eliminate context and account dependencies of search and filter options

• Bug 67421 – Missing possibility to filter on text in body for IMAP
• Bug 223591 – Junk classification should execute on each message as it’s received, not after all have been fetched
• Bug 196036 – Add junk mail status to search/filter criteria

A necessary prerequisite of “Store filters globally” is to eliminate the current complex dependencies that search terms and filter actions have on the precise capabilities of a particular account, folder, or context. Some examples:

1. Body search is not available for IMAP unless “download for offline use” is selected for the account
2. Junk search terms are not available unless the filter context is “after classification”.
3. News has a much more restricted group of available search terms and filter actions than mail.

A major reason that we currently store and define filters the way that we do is to support these restrictions. Yet somehow we manage to silently ignore them without grave consequences in saved searches – you can define a particular saved search, and then expand its application to include folders and account where the underlying search terms perhaps make no sense.

Unfortunately this creates major problems for the user, who has no way of discovering that in order to activate a junk-based filter, he has to define it as “after classification”. Or before she can do a body search on IMAP, “download for offline use” must be enabled. It also means that the whole concept of a global filter, which is much desired, goes against the current philosophy of precise enabling of available features.

I will confess that I have been a major contributor to these issues in the last couple of years, in my zeal to maximimize the availability of features while minimizing the impact on existing behavior. But I don’t really like the result. I think, instead, we should just let the user define what they want, and have the software silently change the behavior to match (download the IMAP body if they want to search on it, delay filters until after bayes classification if they want to search on junk terms.) For cases where something makes no sense at all (like asking to search a news folder for cc) we should define reasonable defaults, and just ignore that the search does not make sense. Perhaps if the filter is being defined only on a news folder, then we could partially hide terms that don’t make sense (or gray them out). But a global filter should have all terms available, even if they don’t make sense in all circumstances.

4. Centralize storage of message metadata related to MIME processing

• Bug 105169 – Filter for Attachments
• Bug 224392 – Feature request: Add ability to filter by attachment name (extension)

Currently, MIME processing is done on demand when a message is displayed, or as part of gloda indexing. But MIME processing is relatively expensive as well as async, which means that its results are not available in traditional search and filtering. We really need to perform MIME processing on all messages, and store basic MIME-related metadata so that it only needs to be done once, and that information would then be available to all consumers, including traditional filter and search.

5. Provide discoverable path to optional search and filter features

• Bug 19442 – Regular expressions in mail and news filters
• Bug 9309 – option to have attachments auto saved to chosen location
• Bug 438441 – (bifftones) Custom bifftones for mail filters (like custom ringtones)
• Bug 200252 – filter action “print message”
• Bug 397009 – A filter will let me tag, but not untag

These are a sample of requests that have been implemented in my extensions. There will always be lots more possibilities for search terms and filter actions than make sense to implement in core. I am not convinced that the current method of expecting users to search for extensions to do things is optimal. In the case of custom filter actions and search terms, it might make sense to have a whole category of normally disabled custom actions and terms that can be discovered by all users, added to by extensions, that enable turning on rarely needed actions and terms.

Bug-related issues

There are another class of important filter and search bugs that are problems rather than enhancements, and whose fix rely on solving difficult core issues. Some of these are:

6. Resolve architectural issues of new message notification

• Bug 11040 – Allow filters to control biff UI (i.e. only notify me of “important” msg)
• Bug 222068 – Reading filtered mails does not clear “New Mail” icon
• Bug 228168 – Getting new mail notification with 0 mails when mail is filtered to local folder from IMAP (“<account> has 0 new messages”)
• Bug 233929 – Still get notification on junk if messages left in Inbox
• Bug 378582 – Filter which moves messages from IMAP account to Local Folder prevents new mail alert

As I have said before, new message notification is hopelessly confused in mailnews, and needs refactoring.  There are lots of notification-related bugs that I an afraid to touch until that is done.

7. Queue operations for busy folders

• Bug 196732 – Some messages correctly identified as junk are not moved to the Junk Mail folder
• Bug 168648 – Cannot filter (move) messages to Inbox that is busy (fetching its own mail)
• Bug 139215 – mail not filtered while compacting folders

Currently if a folder is unavailable for some reason, then filtering just fails. This creates a constant source of unreliable filtering. We really need some explicit method to handle this, perhaps a queue of operations to apply to a folder when it is available again.

Next steps

I don’t really have the time to implement any of these issues for the foreseeable future. I don’t know for sure what MoMo’s priorities are, but I would guess that filtering is generally not high on the list. So I guess I’ll need to put out a help wanted sign (but don’t believe the “reward” part!).

The account “rkent@caspia.org” is an Exchange account, and is finding the standard first-level Exchange folders – including some like “Calendar” and “Contacts” that I suppose should be suppressed here, and shown instead in their correct location in Thunderbird’s Calendar and Address Book.

The main issue in getting this phase to work was trying to understand how mailnews startup really works, which was complicated because my usual fountain of all knowledge (David Bienvenu) was on vacation when I needed to make progress. I think that I have figured out that nsIMsgFolder->GetAllSubfoders has been overloaded with the side-effect of initialization, and that happens only at startup, but there are still lots of holes in my understanding of Thunderbird’s IMAP initialization, which is the closest parallel to Exchange within the Mozilla Mailnews Environment (MME). It did become really clear to me though that the IMAP implementation is very much a problem of synchronization betweem two data stores, though that synchronization is fairly invisible in normal modes of using IMAP. All I am doing now is, on expanding the account’s root folder, looking for MME objects representing each of the folders that exist in EWS, creating them if needed. (I am also supposed to be deleting any MME folders that are not in EWS, but you can see from the “rerer” folder that that is not working for some reason.)

This is currently a barely working implementation (like it asserts with a corrupt database the second time I call it). but the point of this was really to better understand the scope of the entire project, and to provide a basis for an initial development of the architecture, followed by a serious evaluation prior to continuing with the bulk of the project.

Having now completed this initial goal, I need to turn for the next few weeks to landing some bugs that sorely need fixing in Thunderbird’s filter code prior to 3.1, as I have been ignoring that for way too long. But before I do, let me record some thoughts on critical issues that I need to face when I return to this.

1. Code division between C++ and JavaScript, internal and external linkage, and comm-central dependencies.

My current implementation is written entirely in C++ as XPCOM objects, but tested using xpcshell unit tests written in JavaScript. The code is split into two extensions.  “webservices” implements the SOAP layer, resides in /mozilla/extensions, and uses external linkage. “exchangews” resides in /mailnews/extensions, consists of both the exchange-specific mailnews objects, as well as the Exchange Native layer, and uses internal linkage. The dependencies are setup so that the SOAP layer is independent, the Exchange Native layer needs the SOAP layer, and the mailnews layer needs the Native layer, and (very weakly) the SOAP layer.

I would like to consider implementing both the SOAP layer and Native layers using external linkage with frozen Mozilla interfaces, so that they could be provided without version dependencies. Then perhaps I could add some generic factories for starter MME objects to the core Thunderbird code in version 3.2, which I could extend using JavaScript to provide the Exchange-specific mailnews objects. That way I could greatly reduce my dependencies on versions in the shipped code.

2. Error logging

The Mozilla framework does not really have a robust centralized error logging and reporting facility, instead there seems to be a multitude of different approaches that don’t cooperate. My current sprinkling of “NS_WARNING” printouts won’t really help in an imagined future where customers encounter failures. I need to reach some decision on how to do this, so that I don’t have to redo the error management when I want to release something.

3. Synchronization

This is very much a synchronization application. I need to consider the overall synchronization strategy, and how much to follow the lead of MME’s existing IMAP implementation.

4. SOAP layer

Partly as a personal education in SOAP, and against the recommendation of several people more experienced than myself, I updated and used the old abandoned Mozilla webservices code as the SOAP layer. Although that is now generally working OK for me, others have suggested that I could accomplish the same thing in an easier, more robust manner using standard XML manipulation routines. I want to attempt to replace one of my SOAP calls using standard XML routines, to see which I prefer, before I implement the 50 or so calls that I will need to glue my Exchange Native layer to the SOAP layer.

5. People

So far, this project has been entirely my own personal effort. But in the future, I envision a small company of 3 – 5 people who provide and support this code, and are able to make a living doing so. The hope is that I could be independent of, but cooperative with, the main Mozilla Messaging team that supports Thunderbird. (That is, I hope to answer “no” to my question “Do MEALS need a fork?“) At what point should I be adding people to this team? A one-person development team can be quite efficient if it can get the job done, as you eliminate all of the effort associated with maintaining relationships and communication. I think there is a reasonable (50/50) chance that I can get this project done on my own by the end of 2010 and/or in sync with Thunderbird 3.2, as is my current goal. I’ll need to firm this up by the end of June. But I would be open now to beginning to partner with one of the experienced volunteer mailnews developers (you know who you are) who has similar interests.