You’ll need to install my addon JunQuilla to enable one critical piece of user interface. JunQuilla supports a folder property that lets you selectively enable or disable junk processing for a tree of folders. So after you’ve installed JunQuilla, enable processing of junk for a newsgroup:

This will run future posts sent to the newsgroup m.d.a.thunderbird through the bayes junk filter in Thunderbird. After this is enabled, some of the junk management controls on the folder should be enabled. Try “Run Junk Controls on Folder” to process existing posts for junk.

But nothing will appear to happen when you do, because there is no functionality to delete or hide the junk messages for newsgroups built in. Still, you can see the results using JunQuilla’s “junk percent” and “junk status +” columns:

The next step is to remove those junk messages from your view. The easiest thing to do is to create a virtual folder that shows the messages in the newsgroup without the junk messages. This is where the user interface fights against you. But you can trick it in the following manner.

Create a virtual folder on a message folder with search criteria “Junk Percent < 80″. Save the folder as a subfolder of a local mail folder. Now open the virtual folder and change the folders that it scans, removing the original folder and replacing it with the newsgroup folder:

Now the spam posts are hidden from this virtual folder:

You may need to specifically train the junk filter using a few junk and good messages in news. But the good news is that newsgroup spammers are not really optimizing against bayes filters, so it seems to be a lot easier to detect newsgroup spam than mail spam.

Perhaps it would be interesting to show how I hooked into the sending function in the user interface. I asked the usual suspects, and it was not clear to anyone that it could be done without adding backend hooks – which I would like to avoid as much as possible to increase my chances of getting some initial alphas of this released to work with existing TB 3.1 users.

It turned out to be fairly straight forward. In MsgComposeCommands.js there is an observer notification that occurs, called “mail:composeOnSend”. This occurs right before the UI is ready to call SendMsg on the nsIMsgCompose object gMsgCompose. So what I needed to do was to intercept that call, and implement my own version of SendMsg rather than the SMTP/Rfc822-focused standard C++ code. To do that, I create a new object when I receive the notification with the old object as its prototype, then include a custom SendMsg that only applies if the sending account is an Exchange server. The overlay code ends up looking like this (greatly simplified):

function observe(subject, topic, data) {
 // wrap the gMsgCompose object so that we can detect attempts to
 //  send using ews.
 let newCompose = new ewsCompose(gMsgCompose);
 gMsgCompose = newCompose;
}

// ewsCompose provides a wrapper around the compose object, so that we
//  can override functions.
function ewsCompose(oldCompose) {
 this.oldCompose = oldCompose;
 this.SendMsg = function ewsSendMsg
     (msgType, identity, currentAccountKey, msgWindow, progress) {
   if (incomingServer instanceof Ci.msqIEwsIncomingServer) {
     // sending using Exchange Web Services (details not shown)
     // ...
     ewsCompose.sendMsg();
     return;
   }
   else
     return this.oldCompose.SendMsg
       (msgType, identity, currentAccountKey, msgWindow, progress);
 }
 this.__proto__ = oldCompose;
}

This seems to work just fine. It was a little tricky getting the compose code to believe that I had actually sent the message, so it could quit complaining of an unsent message. What I ended up doing in my “sending succeeded” callback is to add a few shutdown calls:

let ewsEventListener = {
  // msqIEwsEventListener implementation
  onEvent: function onEvent(aItem, aEvent, aData, aResult) {
    if (aEvent == 'StopRequest') {
      // ewsProgress was saved from the "progress" variable
      //   in nsMsgComposeCommands.js
      if (ewsProgress)
        ewsProgress.closeProgressDialog(aResult == Cr.NS_OK ? false : true);
      stateListener.ComposeProcessDone(aResult);
      MsgComposeCloseWindow(true);
    }
  }
}

With the ability now to send and receive messages, I’ve completed a single vertical pass through all of the key functionality. There are many, many details that I have passed over in the process. But it’s time to start thinking about what I would need to make work to get a usable alpha release of this, perhaps in about six weeks.

EWS messages do not come from the server in RFC-822 format, so it seems like a pity to store them that way, though that is the common method used in the rest of the mailnews codebase. Instead, I decided to implement a local storage scheme based on SQLite and Mozilla’s Storage interface. Andrew Sutherland has done a lot of great work setting up an environment similar to this for the gloda database, so there are lots of good examples to pull from. Also, because the datamodel for EWS includes not only messages but also Calendar and Contact items, I can have a common database infrastucture that I can leverage over those other pieces once I get it working for the messaging part.

I’ve now replaced my previous in-memory datastore for message metadata with an SQLite version. This is equivalent to the datastore module in gloda, and the data it is storing is like the RFC-822 headers. I still have to do the storage for the body, and also hook this up with folder change state so that the code knows that it has data it can trust.

As I have done this, I’ve had a new set of insights into the relationship of the various objects in the Mozilla mailnews world (which I sometimes call Skink). Previously, I had sort of expected that the natural progression of gloda would be to slowly displace the role of the message summary database, nsIMsgDBHdr.  But now I see that a more natural progression would be for SQLite to be used as a replacement for the local mailstore (currently mbox, with maildir support moving forward as well.) Really the main issue is the async nature of the SQLite calls, which sort of precludes its easy use as a replacement for nsIMsgDBHdr. But the datastores are typically accessed async anyway. If the message metadata in the message stores was stored primarily in SQLite format, as I will be doing, then it would be much easier to hookup an SQLite-based global search facility to all of these databases. Yes that is what gloda does now, but it has to go through all of the work to maintain a separate version of everything. Why have three copies of everything (Mork, MBox, and Gloda) when you could only have two (Mork and Gloda)?

As another insight, while looking through the gloda code I noticed that a JSON object was being saved to store some of the items. I though that was a good idea at first – but then I tried to write a simple serializer to convert from my internal native format to JSON objects, and saw that it was not going to be an easy project. But then I remembered that SOAP is really just a mechanism to serialize typed objects, and I already have a SOAP encoder and decoder! So instead of using JSON, I use objects serialized with my SOAP XML encoder to store unindexed items in my SQLite store. So a message (sans body) ends up looking like this as a TEXT item in SQLite:

<Message xmlns="http://schemas.microsoft.com/exchange/services/2006/types">
 <Subject>Postini First Junk Email Safely Quarantined</Subject>
 <DateTimeReceived>2010-06-04T22:19:30Z</DateTimeReceived>
 <Size>2612</Size>
 <Importance>Normal</Importance>
 <DisplayTo>Kent James</DisplayTo>
 <Culture>en-US</Culture>
 <Sender>
  <Mailbox>
   <Name>Postini Support</Name>
   <EmailAddress>noreply@hostedmsexchange.com</EmailAddress>
   <RoutingType>SMTP</RoutingType>
  </Mailbox>
 </Sender>
 <ToRecipients>
  <Mailbox>
   <Name>Kent James</Name>
   <EmailAddress>rkentjames@caspia.org</EmailAddress>
   <RoutingType>SMTP</RoutingType>
  </Mailbox>
 </ToRecipients>
 <From>
  <Mailbox>
   <Name>Postini Support</Name>
   <EmailAddress>noreply@hostedmsexchange.com</EmailAddress>
   <RoutingType>SMTP</RoutingType>
  </Mailbox>
 </From>
 <InternetMessageId>&lt;0c34b5a4-5f3c-4654-bf9d-99c9a8cb439b@HUB02.4emm.local&gt;
 </InternetMessageId>
 <IsRead>1</IsRead>
</Message>

At first it bothered me to save what is essentially a duplicate of what is coming over the wire, but why not? It’s not conceptually any different than RFC-822, or JSON, in function.

Thunderbird dies a horrible death if I click on any of the actions in the header though. Got to work on that.

]]> http://mesquilla.com/2010/06/10/showing-the-header-mailnews-exchange-support/feed/ 0 http://mesquilla.com/2010/06/04/reading-the-body-mailnews-exchange-support/ http://mesquilla.com/2010/06/04/reading-the-body-mailnews-exchange-support/#comments Fri, 04 Jun 2010 22:58:26 +0000 rkent http://mesquilla.com/?p=850 Just as a status update, my Exchange Server extension can now read message bodies. But note that there is no header information displayed with the message:

Why no header information? Because the header summary, for reasons that I cannot explain, reads the message file directly – and assumes that the message is in RFC 2822 format. But Exchange Server has already done all of that parsing, and separated the message into its components, including metadata (like subject), body, and attachments.

An important issue going forward for the Mozilla Mailnews (Skink) codebase is, what exactly is the thingy that Thunderbird purports to be? Is it a processor of RFC 2822 messages? Or is it something more, that easily incorporates other forms of communications? I hope to write a longer post on that issue soon.

]]> http://mesquilla.com/2010/06/04/reading-the-body-mailnews-exchange-support/feed/ 7 http://mesquilla.com/2010/05/21/thread-pane-and-nsimsgdbhdrnsimsgdatabase-mailnews-exchange-support/ http://mesquilla.com/2010/05/21/thread-pane-and-nsimsgdbhdrnsimsgdatabase-mailnews-exchange-support/#comments Fri, 21 May 2010 22:23:28 +0000 rkent http://mesquilla.com/?p=800 Now that I have a state machine to handle situations that need multiple SOAP calls, it was not a big leap forward to hookup the SOAP requests to the mailnews summary database. That’s what gets the messages displayed in Thunderbird’s thread pane.

So here’s a picture to show where I am at. The two messages are real messages that exist on the Exchange server, and have been read by my extension. The message body is not real:

One problem that I had to solve is to make the nsMsgDatabase object extendable. The standard code for that does lots of static casts to the pointer for the XPCOM object, which either caused crashes, or duplicate database caches, depending on the approach that I took to get around that.  I ended up writing a patch for the base code to solve this, which presumably I will land for TB 3.2. I still think there must be some way to cleverly avoid this problem without patching the base code, but I could not figure it out.

I’m not real happy about requiring a base patch for this work, because I had hoped to be able to release a prototype for this on Thunderbird 3.1 / comm-central 1.9.2, even though the main release should be in Thunderbird 3.2/ comm-central 1.9.3. Maybe I can still figure out a solution that does not require a base patch.

Fun with Branding

I’ve talked about the Colonial layer and the Native layer in my application. What’s above the Colonial layer? I want to be able to distinguish between the code shared with Firefox, and the specific code that is used by messaging applications (Thunderbird and SeaMonkey), because I want portions of my work to be loadable into a FireFox extension (just in case). It could be called by the name of the main directory (mailnews) or that directory’s parent (comm-central). Personally I find mailnews very unattractive as a brand. (A bug asks “What’s the name of MailNews?”)

Although I got no traction when I suggested it on m.d.a.thunderbird, I’m going to experiment personally with calling the Mozilla code that implements messaging functionality, and is not included in Firefox, as the Skink layer instead of “mailnews” (following Mozilla’s lizard analogy and Gecko). Yes I am being bullheaded, but this is my personal blog, so I can do what I want, right?

I also managed to register the domain exquilla.com, as ExQuilla is the logical brand for my Exchange Server integration functionality.

]]> http://mesquilla.com/2010/05/21/thread-pane-and-nsimsgdbhdrnsimsgdatabase-mailnews-exchange-support/feed/ 2 http://mesquilla.com/2010/05/10/mailbox-state-machine-exchange-support/ http://mesquilla.com/2010/05/10/mailbox-state-machine-exchange-support/#comments Mon, 10 May 2010 21:41:15 +0000 rkent http://mesquilla.com/?p=756 Previously I have described interfaces that allowed me to display Exchange Web Server objects in Thunderbird’s folder tree and account manager. My goal lately has been to display lists of messages in the thread pane. To do that, I have to acquire, store and interface message metadata (the message summary database) to the Mozilla mailnews interfaces.

Unfortunately an issue has loomed too large to ignore: the need to sequence a number of SOAP requests when I need to send multiple requests to meet a single goal. Previously I had simply kludged around this, but I needed to make some decisions to be able to move forward in an orderly manner.

I’m also having enormous problems deciding what to name things. So before I can even talk about what I decided to do, I need to give a little lesson on the naming that surrounds my architecture.

Overall Architecture

Since my last diagram, I have clarified things a little, so the structure now looks like this:

The architectural layers are:

Mailnews: the existing Mozilla Mailnews code and objects.

Colonial: the Colonial layer provides the glue between objects represented in Mozilla mailnews format, and Exchange server format. The “colonial” analogy is an extension of the “native” layer analogy presented earlier. That is, if we picture Mozilla mailnews as the mother country, trying to make sense of some aboriginal “native” tribe, then there exists a colonial administration that pretends to have the same laws as the mother country, but has to translate all of those laws into some sort of actions involving native people. So the objects that pretend to be following the laws of the mother country (Mozilla mailnews), but in fact really exist in some other culture, are Colonial objects.

In my architecture, the Colonial objects are subclasses of base Mailnews objects like nsMsgDatabase, nsMsgDBFolder, nsMsgIncomingServer, etc.

Native: This layer’s objects exist as XPCOM objects on the Mozilla platform, but are represented in the native “Exchange” format rather than in Mailnews format. So for example they are identified primarily by string object ids (and not integer keys like Mailnews objects), and they are located in the Exchange tree. For example, folders containing contacts, the inbox, and tasks are all subfolders of a root mailbox folder, as in Exchange Server.

EWS: This layer provides translation of methods to perform operations on native-layer objects (like get subfolders, or sync message metadata) with objects that can be directly translated into SOAP web calls. (I actually use an intermediate representation, called a “property list”, that is very similar to an nsIPropertyBag, but supports ordered, multiple-valued lists such as are present in the SOAP protocols. But these property list objects can be translated directly into the SOAP protocol using the schema provided by Exchange Web Services).

SOAP: As discussed earlier, this is an updated version of Mozilla’s abandoned webservices package.

Exchange: The Exchange Messaging Server itself, implementing a SOAP interface.

State Machine to implement Mailnews Object methods

Let me now describe the problem, and what I decided to do. This is the msqEwsMachine class in the diagram.

The interface to Mozilla mailnews frequently consists of methods that are executed on mailnews interfaces, for example nsIMsgFolder.getNewMessages(). Such methods will get translated into a sequence of requests to a web server, which leave the SOAP layer as a sequence of nsXMLHttpRequests.

Which object is responsible for taking the method call, managing the series of requests to the web server, and generating appropriate events and callbacks when the method has been completed? That has been the issue of the last two weeks.

Originally, I had it done by the colonial-layer representation of nsIMsgIncomingServer, msqEwsIncomingServer. But I did not like how that forced the incoming server object to know way too much about the details of the SOAP calls, when at that layer all it really needed to know was the data results. I also tested using the service layer objects, either the colonial layer object that implements nsIMsgMessageService, or its Native-layer counterpart. But I found that everything that I did needed access to the root of the Native layer data representation, which is the native mailbox object.

So that is where the management ended up: methods on an object, msqEwsMachine, which is fully owned by the the native-layer mailbox object (The mailbox is the native-layer analog of nsIMsgIncomingServer).

A simple request to the mailbox layer would be “discover subfolders”, which requests that the Exchange server provide a list of all defined folders, their parent/child relationship, and some basic folder properties such as the display name. That requires two SOAP requests: 1) a first to translate the so-called “DistinguishedFolderId” such as “inbox” into the guid equivalent for the limited number of distinguished folders, and 2) a second call where the root folder is asked for all of its children recursively. At the end of the whole process, an “onStop” event is generated, which will ultimately be translated into the appropriate mailnews notifications.

Next step: message metadata and nsIMsgDBHdr

Yes this has been a rather boring post, but I didn’t want anyone to think nothing was happening! As the next step, I am now ready to start generating a list of messages per folder, and their corresponding nsIMsgDBHdr objects. Then I can show messages in the thread pane.

The account manager interface was apparently written by an ancient advanced intelligence (oops I mean here), that used mysterious technologies that are no longer understood by humankind. Renowned cybernetic archaeologist Joshua Cranmer has tried to unravel some of its mysteries, but that provided me more of a hint of where to look for issues than a roadmap of what I needed to really do. So I was forced into my last resort – reading the code and trying to figure it out. There were some surprises.

First, a brief description of terms. When you first create a new account type (unless you are using the new email interface for Thunderbird 3.0), you enter into the account wizard. That sets up up all of the initial values. After that, if you try to edit those you end up in the account manager. They are closely-related, but distinct, interfaces.

First, how do you add a new account type at all to these interfaces? It turns out that kill-rdf missed a spot, and it is still possible to use rdf to add a new hook into the account wizard. Extensions can have a subdirectory “isp”, and into that subdirectory you can inject an rdf file that contains some information about your account manager hook. The rss implementation gives some hints of how that is done.

Once you add the rdf file, your new account type shows up in the account setup under “add other account …”:

The rdf file that you add contains all sorts of information about the account that you are trying to setup. Through some magic, the rdf items get linked directly (at least in theory) to default values for the main account objects.

But unfortunately, extensibility of the account manager has never really been carefully supported, so things just don’t work out quite the way you would like. Some examples:

• My account type represents a new protocol type, “ews” instead of “imap” or “pop3″. But at one point the server page looks into the standard file “messenger.properties” for strings like “serverType-imap=IMAP Mail Server” which of course do not exist for “serverType-ews”.
• The pages that show in the account wizard are partly hard wired for the existing account types.
• Although the account manager has some attempts at extensibility, most of its options are setup as a list of account types to exclude when showing certain screens. This is maddening if you are adding a new account type, because since your account type previously did not exist, of course it does not show up in all of the exclusion lists.

So this intricately designed infrastructure that the ancients designed is basically unusable at present to add new account types, as far as I can tell. For each little option that you want to add, you need to trace for hours through very abstract code to figure out whether it will work, or if there is some hidden gotcha. Ultimately, it really is easier just to throw it out and start over again.

Fortunately, through the wonders of javascript function replacements, I was able to create enough hooks to get my new account added, without requiring a patch to the core code. Here is a rough outline of the steps that I took:

1. First, the account wizard seems to use the existence of identities to decide if your have a mail account type, and if you do it forces you to use the standard mail account wizard types. I managed to override this in javascript.
2. After you do that, through some bizarre quirk the rdf property “NC:wizardShortName” is actually a list of the wizard pages that you want to appear for custom account types. So I can add a line to the rdf file:

   <NC:wizardShortName>identitypage,ewsServerPage</NC:wizardShortName>

   and that will allow me to specify precisely which pages are shown in the wizard. “ewsServerPage” is a custom page that I add though an overlay.

3. In the account manager, there is no easy way to control which subpages appear in the account tree for a custom account type. Once again I can use a javascript function hook to manually adjust the available pages. I wanted to remove the default server page (which asks if I want pop3 or imap), and add my own page in its place. I also wanted to be able to remove pages I did not want. That lets me control the list precisely, so that I now get this account tree:

I suppose dmose would like me to organize all of this into a sample extension that others could use. One of the reasons that it is hard to get motivated to do this is that there is no real plan for the future of any of this, yet I hear rumors that a lot of this is scheduled for demolition (rdf, account wizard, even binary extensions that my entire application relies on.) Plus I’m not sure any sane person would want to do the kludges that I am doing to get this to work. Is there even any real demand for custom account types in the mailnews code, beyond what I am doing?

Anyway, the good news for the Exchange Web Services integration project is that I now have basic hooks into the account manager interface, so that I can easily add custom changes to the account manager setup to support Exchange Server. The most basic information (user name and server address) can be entered through the account manager interface, and used to access the correct Exchange server.

serverType-imap=IMAP Mail Server

Mailnews Traditional Filter and Search – Big Issues for Enhancements

1. Integrate full boolean search into Search Term editor

• Bug 297852 – Search/Filter and Boolean expressions

The mailnews backend traditional search logic supports full boolean expressions, but the user interface has never been updated to allow this. With asuth’s pending update to the quick search UI, making extensive use of boolean expressions internally, we are getting to the point where quick search can express certain searches that so-called “advanced search” cannot (and vice versa.)

2. Store filters globally, and apply them to selected account/folder trees/contexts

• Bug 11039 – Filter outgoing messages (perhaps to use a different Sent/FCC folder)
• Bug 166842 – [RFE] Ability to import and export message filters
• Bug 129883 – Filters Should be Shareable Across Multiple E-mail Accounts
• Bug 59365 – Filters should have triggers
• Bug 34973 – (GlobalFilters) global message filter option

Filters are currently stored per account (except in news, where they can also be per-folder). But when people define a particular filter, they frequently want it to apply to more than one account. There is no easy way currently to do this, short of recreating all filters on each account. Moving filters is also a pain when accounts are reorganized.

I would store filters in a single location, and provide some sort of UI to define where and when each filter is applied. That would not only be the account or folder, but also the context (such as incoming, manual, sending, or triggered.)

3. Eliminate context and account dependencies of search and filter options

• Bug 67421 – Missing possibility to filter on text in body for IMAP
• Bug 223591 – Junk classification should execute on each message as it’s received, not after all have been fetched
• Bug 196036 – Add junk mail status to search/filter criteria

A necessary prerequisite of “Store filters globally” is to eliminate the current complex dependencies that search terms and filter actions have on the precise capabilities of a particular account, folder, or context. Some examples:

1. Body search is not available for IMAP unless “download for offline use” is selected for the account
2. Junk search terms are not available unless the filter context is “after classification”.
3. News has a much more restricted group of available search terms and filter actions than mail.

A major reason that we currently store and define filters the way that we do is to support these restrictions. Yet somehow we manage to silently ignore them without grave consequences in saved searches – you can define a particular saved search, and then expand its application to include folders and account where the underlying search terms perhaps make no sense.

Unfortunately this creates major problems for the user, who has no way of discovering that in order to activate a junk-based filter, he has to define it as “after classification”. Or before she can do a body search on IMAP, “download for offline use” must be enabled. It also means that the whole concept of a global filter, which is much desired, goes against the current philosophy of precise enabling of available features.

I will confess that I have been a major contributor to these issues in the last couple of years, in my zeal to maximimize the availability of features while minimizing the impact on existing behavior. But I don’t really like the result. I think, instead, we should just let the user define what they want, and have the software silently change the behavior to match (download the IMAP body if they want to search on it, delay filters until after bayes classification if they want to search on junk terms.) For cases where something makes no sense at all (like asking to search a news folder for cc) we should define reasonable defaults, and just ignore that the search does not make sense. Perhaps if the filter is being defined only on a news folder, then we could partially hide terms that don’t make sense (or gray them out). But a global filter should have all terms available, even if they don’t make sense in all circumstances.

4. Centralize storage of message metadata related to MIME processing

• Bug 105169 – Filter for Attachments
• Bug 224392 – Feature request: Add ability to filter by attachment name (extension)

Currently, MIME processing is done on demand when a message is displayed, or as part of gloda indexing. But MIME processing is relatively expensive as well as async, which means that its results are not available in traditional search and filtering. We really need to perform MIME processing on all messages, and store basic MIME-related metadata so that it only needs to be done once, and that information would then be available to all consumers, including traditional filter and search.

5. Provide discoverable path to optional search and filter features

• Bug 19442 – Regular expressions in mail and news filters
• Bug 9309 – option to have attachments auto saved to chosen location
• Bug 438441 – (bifftones) Custom bifftones for mail filters (like custom ringtones)
• Bug 200252 – filter action “print message”
• Bug 397009 – A filter will let me tag, but not untag

These are a sample of requests that have been implemented in my extensions. There will always be lots more possibilities for search terms and filter actions than make sense to implement in core. I am not convinced that the current method of expecting users to search for extensions to do things is optimal. In the case of custom filter actions and search terms, it might make sense to have a whole category of normally disabled custom actions and terms that can be discovered by all users, added to by extensions, that enable turning on rarely needed actions and terms.

Bug-related issues

There are another class of important filter and search bugs that are problems rather than enhancements, and whose fix rely on solving difficult core issues. Some of these are:

6. Resolve architectural issues of new message notification

• Bug 11040 – Allow filters to control biff UI (i.e. only notify me of “important” msg)
• Bug 222068 – Reading filtered mails does not clear “New Mail” icon
• Bug 228168 – Getting new mail notification with 0 mails when mail is filtered to local folder from IMAP (“<account> has 0 new messages”)
• Bug 233929 – Still get notification on junk if messages left in Inbox
• Bug 378582 – Filter which moves messages from IMAP account to Local Folder prevents new mail alert

As I have said before, new message notification is hopelessly confused in mailnews, and needs refactoring.  There are lots of notification-related bugs that I an afraid to touch until that is done.

7. Queue operations for busy folders

• Bug 196732 – Some messages correctly identified as junk are not moved to the Junk Mail folder
• Bug 168648 – Cannot filter (move) messages to Inbox that is busy (fetching its own mail)
• Bug 139215 – mail not filtered while compacting folders

Currently if a folder is unavailable for some reason, then filtering just fails. This creates a constant source of unreliable filtering. We really need some explicit method to handle this, perhaps a queue of operations to apply to a folder when it is available again.

Next steps

I don’t really have the time to implement any of these issues for the foreseeable future. I don’t know for sure what MoMo’s priorities are, but I would guess that filtering is generally not high on the list. So I guess I’ll need to put out a help wanted sign (but don’t believe the “reward” part!).

The account “rkent@caspia.org” is an Exchange account, and is finding the standard first-level Exchange folders – including some like “Calendar” and “Contacts” that I suppose should be suppressed here, and shown instead in their correct location in Thunderbird’s Calendar and Address Book.

The main issue in getting this phase to work was trying to understand how mailnews startup really works, which was complicated because my usual fountain of all knowledge (David Bienvenu) was on vacation when I needed to make progress. I think that I have figured out that nsIMsgFolder->GetAllSubfoders has been overloaded with the side-effect of initialization, and that happens only at startup, but there are still lots of holes in my understanding of Thunderbird’s IMAP initialization, which is the closest parallel to Exchange within the Mozilla Mailnews Environment (MME). It did become really clear to me though that the IMAP implementation is very much a problem of synchronization betweem two data stores, though that synchronization is fairly invisible in normal modes of using IMAP. All I am doing now is, on expanding the account’s root folder, looking for MME objects representing each of the folders that exist in EWS, creating them if needed. (I am also supposed to be deleting any MME folders that are not in EWS, but you can see from the “rerer” folder that that is not working for some reason.)

This is currently a barely working implementation (like it asserts with a corrupt database the second time I call it). but the point of this was really to better understand the scope of the entire project, and to provide a basis for an initial development of the architecture, followed by a serious evaluation prior to continuing with the bulk of the project.

Having now completed this initial goal, I need to turn for the next few weeks to landing some bugs that sorely need fixing in Thunderbird’s filter code prior to 3.1, as I have been ignoring that for way too long. But before I do, let me record some thoughts on critical issues that I need to face when I return to this.

1. Code division between C++ and JavaScript, internal and external linkage, and comm-central dependencies.

My current implementation is written entirely in C++ as XPCOM objects, but tested using xpcshell unit tests written in JavaScript. The code is split into two extensions.  “webservices” implements the SOAP layer, resides in /mozilla/extensions, and uses external linkage. “exchangews” resides in /mailnews/extensions, consists of both the exchange-specific mailnews objects, as well as the Exchange Native layer, and uses internal linkage. The dependencies are setup so that the SOAP layer is independent, the Exchange Native layer needs the SOAP layer, and the mailnews layer needs the Native layer, and (very weakly) the SOAP layer.

I would like to consider implementing both the SOAP layer and Native layers using external linkage with frozen Mozilla interfaces, so that they could be provided without version dependencies. Then perhaps I could add some generic factories for starter MME objects to the core Thunderbird code in version 3.2, which I could extend using JavaScript to provide the Exchange-specific mailnews objects. That way I could greatly reduce my dependencies on versions in the shipped code.

2. Error logging

The Mozilla framework does not really have a robust centralized error logging and reporting facility, instead there seems to be a multitude of different approaches that don’t cooperate. My current sprinkling of “NS_WARNING” printouts won’t really help in an imagined future where customers encounter failures. I need to reach some decision on how to do this, so that I don’t have to redo the error management when I want to release something.

3. Synchronization

This is very much a synchronization application. I need to consider the overall synchronization strategy, and how much to follow the lead of MME’s existing IMAP implementation.

4. SOAP layer

Partly as a personal education in SOAP, and against the recommendation of several people more experienced than myself, I updated and used the old abandoned Mozilla webservices code as the SOAP layer. Although that is now generally working OK for me, others have suggested that I could accomplish the same thing in an easier, more robust manner using standard XML manipulation routines. I want to attempt to replace one of my SOAP calls using standard XML routines, to see which I prefer, before I implement the 50 or so calls that I will need to glue my Exchange Native layer to the SOAP layer.

5. People

So far, this project has been entirely my own personal effort. But in the future, I envision a small company of 3 – 5 people who provide and support this code, and are able to make a living doing so. The hope is that I could be independent of, but cooperative with, the main Mozilla Messaging team that supports Thunderbird. (That is, I hope to answer “no” to my question “Do MEALS need a fork?“) At what point should I be adding people to this team? A one-person development team can be quite efficient if it can get the job done, as you eliminate all of the effort associated with maintaining relationships and communication. I think there is a reasonable (50/50) chance that I can get this project done on my own by the end of 2010 and/or in sync with Thunderbird 3.2, as is my current goal. I’ll need to firm this up by the end of June. But I would be open now to beginning to partner with one of the experienced volunteer mailnews developers (you know who you are) who has similar interests.