MesQuilla

Mailbox state machine (Mailnews Exchange support)

Previously I have described interfaces that allowed me to display Exchange Web Server objects in Thunderbird’s folder tree and account manager. My goal lately has been to display lists of messages in the thread pane. To do that, I have to acquire, store and interface message metadata (the message summary database) to the Mozilla mailnews interfaces.

Unfortunately an issue has loomed too large to ignore: the need to sequence a number of SOAP requests when I need to send multiple requests to meet a single goal. Previously I had simply kludged around this, but I needed to make some decisions to be able to move forward in an orderly manner.

I’m also having enormous problems deciding what to name things. So before I can even talk about what I decided to do, I need to give a little lesson on the naming that surrounds my architecture.

Overall Architecture

Since my last diagram, I have clarified things a little, so the structure now looks like this:

The architectural layers are:

Mailnews: the existing Mozilla Mailnews code and objects.

Colonial: the Colonial layer provides the glue between objects represented in Mozilla mailnews format, and Exchange server format. The “colonial” analogy is an extension of the “native” layer analogy presented earlier. That is, if we picture Mozilla mailnews as the mother country, trying to make sense of some aboriginal “native” tribe, then there exists a colonial administration that pretends to have the same laws as the mother country, but has to translate all of those laws into some sort of actions involving native people. So the objects that pretend to be following the laws of the mother country (Mozilla mailnews), but in fact really exist in some other culture, are Colonial objects.

In my architecture, the Colonial objects are subclasses of base Mailnews objects like nsMsgDatabase, nsMsgDBFolder, nsMsgIncomingServer, etc.

Native: This layer’s objects exist as XPCOM objects on the Mozilla platform, but are represented in the native “Exchange” format rather than in Mailnews format. So for example they are identified primarily by string object ids (and not integer keys like Mailnews objects), and they are located in the Exchange tree. For example, folders containing contacts, the inbox, and tasks are all subfolders of a root mailbox folder, as in Exchange Server.

EWS: This layer provides translation of methods to perform operations on native-layer objects (like get subfolders, or sync message metadata) with objects that can be directly translated into SOAP web calls. (I actually use an intermediate representation, called a “property list”, that is very similar to an nsIPropertyBag, but supports ordered, multiple-valued lists such as are present in the SOAP protocols. But these property list objects can be translated directly into the SOAP protocol using the schema provided by Exchange Web Services).

SOAP: As discussed earlier, this is an updated version of Mozilla’s abandoned webservices package.

Exchange: The Exchange Messaging Server itself, implementing a SOAP interface.

State Machine to implement Mailnews Object methods

Let me now describe the problem, and what I decided to do. This is the msqEwsMachine class in the diagram.

The interface to Mozilla mailnews frequently consists of methods that are executed on mailnews interfaces, for example nsIMsgFolder.getNewMessages(). Such methods will get translated into a sequence of requests to a web server, which leave the SOAP layer as a sequence of nsXMLHttpRequests.

Which object is responsible for taking the method call, managing the series of requests to the web server, and generating appropriate events and callbacks when the method has been completed? That has been the issue of the last two weeks.

Originally, I had it done by the colonial-layer representation of nsIMsgIncomingServer, msqEwsIncomingServer. But I did not like how that forced the incoming server object to know way too much about the details of the SOAP calls, when at that layer all it really needed to know was the data results. I also tested using the service layer objects, either the colonial layer object that implements nsIMsgMessageService, or its Native-layer counterpart. But I found that everything that I did needed access to the root of the Native layer data representation, which is the native mailbox object.

So that is where the management ended up: methods on an object, msqEwsMachine, which is fully owned by the the native-layer mailbox object (The mailbox is the native-layer analog of nsIMsgIncomingServer).

A simple request to the mailbox layer would be “discover subfolders”, which requests that the Exchange server provide a list of all defined folders, their parent/child relationship, and some basic folder properties such as the display name. That requires two SOAP requests: 1) a first to translate the so-called “DistinguishedFolderId” such as “inbox” into the guid equivalent for the limited number of distinguished folders, and 2) a second call where the root folder is asked for all of its children recursively. At the end of the whole process, an “onStop” event is generated, which will ultimately be translated into the appropriate mailnews notifications.

Next step: message metadata and nsIMsgDBHdr

Yes this has been a rather boring post, but I didn’t want anyone to think nothing was happening! As the next step, I am now ready to start generating a list of messages per folder, and their corresponding nsIMsgDBHdr objects. Then I can show messages in the thread pane.

Posted in Exchange Web Services (EWS), Mailnews development, Planet MozillaMessaging | 3 Comments »

Mailnews Exchange Support: Account Manager

To ease my transition back into ews after getting my brain all wrapped around filters for awhile, I thought I would tackle something that is mostly internal to the mailnews code. So I took on the task of getting a basic custom Account Manager interface up for the exchange web services account type.

The account manager interface was apparently written by an ancient advanced intelligence (oops I mean here), that used mysterious technologies that are no longer understood by humankind. Renowned cybernetic archaeologist Joshua Cranmer has tried to unravel some of its mysteries, but that provided me more of a hint of where to look for issues than a roadmap of what I needed to really do. So I was forced into my last resort – reading the code and trying to figure it out. There were some surprises.

First, a brief description of terms. When you first create a new account type (unless you are using the new email interface for Thunderbird 3.0), you enter into the account wizard. That sets up up all of the initial values. After that, if you try to edit those you end up in the account manager. They are closely-related, but distinct, interfaces.

First, how do you add a new account type at all to these interfaces? It turns out that kill-rdf missed a spot, and it is still possible to use rdf to add a new hook into the account wizard. Extensions can have a subdirectory “isp”, and into that subdirectory you can inject an rdf file that contains some information about your account manager hook. The rss implementation gives some hints of how that is done.

Once you add the rdf file, your new account type shows up in the account setup under “add other account …”:

The rdf file that you add contains all sorts of information about the account that you are trying to setup. Through some magic, the rdf items get linked directly (at least in theory) to default values for the main account objects.

But unfortunately, extensibility of the account manager has never really been carefully supported, so things just don’t work out quite the way you would like. Some examples:

• My account type represents a new protocol type, “ews” instead of “imap” or “pop3″. But at one point the server page looks into the standard file “messenger.properties” for strings like “serverType-imap=IMAP Mail Server” which of course do not exist for “serverType-ews”.
• The pages that show in the account wizard are partly hard wired for the existing account types.
• Although the account manager has some attempts at extensibility, most of its options are setup as a list of account types to exclude when showing certain screens. This is maddening if you are adding a new account type, because since your account type previously did not exist, of course it does not show up in all of the exclusion lists.

So this intricately designed infrastructure that the ancients designed is basically unusable at present to add new account types, as far as I can tell. For each little option that you want to add, you need to trace for hours through very abstract code to figure out whether it will work, or if there is some hidden gotcha. Ultimately, it really is easier just to throw it out and start over again.

Fortunately, through the wonders of javascript function replacements, I was able to create enough hooks to get my new account added, without requiring a patch to the core code. Here is a rough outline of the steps that I took:

1. First, the account wizard seems to use the existence of identities to decide if your have a mail account type, and if you do it forces you to use the standard mail account wizard types. I managed to override this in javascript.
2. After you do that, through some bizarre quirk the rdf property “NC:wizardShortName” is actually a list of the wizard pages that you want to appear for custom account types. So I can add a line to the rdf file:

   <NC:wizardShortName>identitypage,ewsServerPage</NC:wizardShortName>

   and that will allow me to specify precisely which pages are shown in the wizard. “ewsServerPage” is a custom page that I add though an overlay.

3. In the account manager, there is no easy way to control which subpages appear in the account tree for a custom account type. Once again I can use a javascript function hook to manually adjust the available pages. I wanted to remove the default server page (which asks if I want pop3 or imap), and add my own page in its place. I also wanted to be able to remove pages I did not want. That lets me control the list precisely, so that I now get this account tree:

I suppose dmose would like me to organize all of this into a sample extension that others could use. One of the reasons that it is hard to get motivated to do this is that there is no real plan for the future of any of this, yet I hear rumors that a lot of this is scheduled for demolition (rdf, account wizard, even binary extensions that my entire application relies on.) Plus I’m not sure any sane person would want to do the kludges that I am doing to get this to work. Is there even any real demand for custom account types in the mailnews code, beyond what I am doing?

Anyway, the good news for the Exchange Web Services integration project is that I now have basic hooks into the account manager interface, so that I can easily add custom changes to the account manager setup to support Exchange Server. The most basic information (user name and server address) can be entered through the account manager interface, and used to access the correct Exchange server.

serverType-imap=IMAP Mail Server

Posted in Exchange Web Services (EWS), Mailnews development, Planet Mozilla, Planet MozillaMessaging | 6 Comments »

What Filters Need

I’ve just completed a two-week hiatus from my work on Exchange Web Services (EWS) integration project to focus on fixing a few filter issues in mailnews core (Thunderbird and SeaMonkey) prior to TB’s 3.1 feature freeze. I will now return to EWS as my primary focus. But before I do, I think it would be useful to give an overview of some big-picture issues in filter and traditional search (which are closely related), providing something resembling a roadmap but without any real commitment by anyone to move forward.

Mailnews Traditional Filter and Search – Big Issues for Enhancements

1. Integrate full boolean search into Search Term editor

• Bug 297852 – Search/Filter and Boolean expressions

The mailnews backend traditional search logic supports full boolean expressions, but the user interface has never been updated to allow this. With asuth’s pending update to the quick search UI, making extensive use of boolean expressions internally, we are getting to the point where quick search can express certain searches that so-called “advanced search” cannot (and vice versa.)

2. Store filters globally, and apply them to selected account/folder trees/contexts

• Bug 11039 – Filter outgoing messages (perhaps to use a different Sent/FCC folder)
• Bug 166842 – [RFE] Ability to import and export message filters
• Bug 129883 – Filters Should be Shareable Across Multiple E-mail Accounts
• Bug 59365 – Filters should have triggers
• Bug 34973 – (GlobalFilters) global message filter option

Filters are currently stored per account (except in news, where they can also be per-folder). But when people define a particular filter, they frequently want it to apply to more than one account. There is no easy way currently to do this, short of recreating all filters on each account. Moving filters is also a pain when accounts are reorganized.

I would store filters in a single location, and provide some sort of UI to define where and when each filter is applied. That would not only be the account or folder, but also the context (such as incoming, manual, sending, or triggered.)

3. Eliminate context and account dependencies of search and filter options

• Bug 67421 – Missing possibility to filter on text in body for IMAP
• Bug 223591 – Junk classification should execute on each message as it’s received, not after all have been fetched
• Bug 196036 – Add junk mail status to search/filter criteria

A necessary prerequisite of “Store filters globally” is to eliminate the current complex dependencies that search terms and filter actions have on the precise capabilities of a particular account, folder, or context. Some examples:

1. Body search is not available for IMAP unless “download for offline use” is selected for the account
2. Junk search terms are not available unless the filter context is “after classification”.
3. News has a much more restricted group of available search terms and filter actions than mail.

A major reason that we currently store and define filters the way that we do is to support these restrictions. Yet somehow we manage to silently ignore them without grave consequences in saved searches – you can define a particular saved search, and then expand its application to include folders and account where the underlying search terms perhaps make no sense.

Unfortunately this creates major problems for the user, who has no way of discovering that in order to activate a junk-based filter, he has to define it as “after classification”. Or before she can do a body search on IMAP, “download for offline use” must be enabled. It also means that the whole concept of a global filter, which is much desired, goes against the current philosophy of precise enabling of available features.

I will confess that I have been a major contributor to these issues in the last couple of years, in my zeal to maximimize the availability of features while minimizing the impact on existing behavior. But I don’t really like the result. I think, instead, we should just let the user define what they want, and have the software silently change the behavior to match (download the IMAP body if they want to search on it, delay filters until after bayes classification if they want to search on junk terms.) For cases where something makes no sense at all (like asking to search a news folder for cc) we should define reasonable defaults, and just ignore that the search does not make sense. Perhaps if the filter is being defined only on a news folder, then we could partially hide terms that don’t make sense (or gray them out). But a global filter should have all terms available, even if they don’t make sense in all circumstances.

4. Centralize storage of message metadata related to MIME processing

• Bug 105169 – Filter for Attachments
• Bug 224392 – Feature request: Add ability to filter by attachment name (extension)

Currently, MIME processing is done on demand when a message is displayed, or as part of gloda indexing. But MIME processing is relatively expensive as well as async, which means that its results are not available in traditional search and filtering. We really need to perform MIME processing on all messages, and store basic MIME-related metadata so that it only needs to be done once, and that information would then be available to all consumers, including traditional filter and search.

5. Provide discoverable path to optional search and filter features

• Bug 19442 – Regular expressions in mail and news filters
• Bug 9309 – option to have attachments auto saved to chosen location
• Bug 438441 – (bifftones) Custom bifftones for mail filters (like custom ringtones)
• Bug 200252 – filter action “print message”
• Bug 397009 – A filter will let me tag, but not untag

These are a sample of requests that have been implemented in my extensions. There will always be lots more possibilities for search terms and filter actions than make sense to implement in core. I am not convinced that the current method of expecting users to search for extensions to do things is optimal. In the case of custom filter actions and search terms, it might make sense to have a whole category of normally disabled custom actions and terms that can be discovered by all users, added to by extensions, that enable turning on rarely needed actions and terms.

Bug-related issues

There are another class of important filter and search bugs that are problems rather than enhancements, and whose fix rely on solving difficult core issues. Some of these are:

6. Resolve architectural issues of new message notification

• Bug 11040 – Allow filters to control biff UI (i.e. only notify me of “important” msg)
• Bug 222068 – Reading filtered mails does not clear “New Mail” icon
• Bug 228168 – Getting new mail notification with 0 mails when mail is filtered to local folder from IMAP (“<account> has 0 new messages”)
• Bug 233929 – Still get notification on junk if messages left in Inbox
• Bug 378582 – Filter which moves messages from IMAP account to Local Folder prevents new mail alert

As I have said before, new message notification is hopelessly confused in mailnews, and needs refactoring.  There are lots of notification-related bugs that I an afraid to touch until that is done.

7. Queue operations for busy folders

• Bug 196732 – Some messages correctly identified as junk are not moved to the Junk Mail folder
• Bug 168648 – Cannot filter (move) messages to Inbox that is busy (fetching its own mail)
• Bug 139215 – mail not filtered while compacting folders

Currently if a folder is unavailable for some reason, then filtering just fails. This creates a constant source of unreliable filtering. We really need some explicit method to handle this, perhaps a queue of operations to apply to a folder when it is available again.

Next steps

I don’t really have the time to implement any of these issues for the foreseeable future. I don’t know for sure what MoMo’s priorities are, but I would guess that filtering is generally not high on the list. So I guess I’ll need to put out a help wanted sign (but don’t believe the “reward” part!).

Posted in Mailnews development, Planet Mozilla, Planet MozillaMessaging | 8 Comments »

Mailnews Exchange Support: First Folder Discovery

I set a goal for myself a few weeks ago to perform a single vertical slice through my Mailnews Exchange Support project, focusing on displaying Microsoft Exchange folders in the Thunderbird folderpane. I have now filled in the missing glue from the architecture in my last post, and have my first display of Exchange Web Services (EWS) objects in the Thunderbird user interface.

The account “rkent@caspia.org” is an Exchange account, and is finding the standard first-level Exchange folders – including some like “Calendar” and “Contacts” that I suppose should be suppressed here, and shown instead in their correct location in Thunderbird’s Calendar and Address Book.

The main issue in getting this phase to work was trying to understand how mailnews startup really works, which was complicated because my usual fountain of all knowledge (David Bienvenu) was on vacation when I needed to make progress. I think that I have figured out that nsIMsgFolder->GetAllSubfoders has been overloaded with the side-effect of initialization, and that happens only at startup, but there are still lots of holes in my understanding of Thunderbird’s IMAP initialization, which is the closest parallel to Exchange within the Mozilla Mailnews Environment (MME). It did become really clear to me though that the IMAP implementation is very much a problem of synchronization betweem two data stores, though that synchronization is fairly invisible in normal modes of using IMAP. All I am doing now is, on expanding the account’s root folder, looking for MME objects representing each of the folders that exist in EWS, creating them if needed. (I am also supposed to be deleting any MME folders that are not in EWS, but you can see from the “rerer” folder that that is not working for some reason.)

This is currently a barely working implementation (like it asserts with a corrupt database the second time I call it). but the point of this was really to better understand the scope of the entire project, and to provide a basis for an initial development of the architecture, followed by a serious evaluation prior to continuing with the bulk of the project.

Having now completed this initial goal, I need to turn for the next few weeks to landing some bugs that sorely need fixing in Thunderbird’s filter code prior to 3.1, as I have been ignoring that for way too long. But before I do, let me record some thoughts on critical issues that I need to face when I return to this.

1. Code division between C++ and JavaScript, internal and external linkage, and comm-central dependencies.

My current implementation is written entirely in C++ as XPCOM objects, but tested using xpcshell unit tests written in JavaScript. The code is split into two extensions.  “webservices” implements the SOAP layer, resides in /mozilla/extensions, and uses external linkage. “exchangews” resides in /mailnews/extensions, consists of both the exchange-specific mailnews objects, as well as the Exchange Native layer, and uses internal linkage. The dependencies are setup so that the SOAP layer is independent, the Exchange Native layer needs the SOAP layer, and the mailnews layer needs the Native layer, and (very weakly) the SOAP layer.

I would like to consider implementing both the SOAP layer and Native layers using external linkage with frozen Mozilla interfaces, so that they could be provided without version dependencies. Then perhaps I could add some generic factories for starter MME objects to the core Thunderbird code in version 3.2, which I could extend using JavaScript to provide the Exchange-specific mailnews objects. That way I could greatly reduce my dependencies on versions in the shipped code.

2. Error logging

The Mozilla framework does not really have a robust centralized error logging and reporting facility, instead there seems to be a multitude of different approaches that don’t cooperate. My current sprinkling of “NS_WARNING” printouts won’t really help in an imagined future where customers encounter failures. I need to reach some decision on how to do this, so that I don’t have to redo the error management when I want to release something.

3. Synchronization

This is very much a synchronization application. I need to consider the overall synchronization strategy, and how much to follow the lead of MME’s existing IMAP implementation.

4. SOAP layer

Partly as a personal education in SOAP, and against the recommendation of several people more experienced than myself, I updated and used the old abandoned Mozilla webservices code as the SOAP layer. Although that is now generally working OK for me, others have suggested that I could accomplish the same thing in an easier, more robust manner using standard XML manipulation routines. I want to attempt to replace one of my SOAP calls using standard XML routines, to see which I prefer, before I implement the 50 or so calls that I will need to glue my Exchange Native layer to the SOAP layer.

5. People

So far, this project has been entirely my own personal effort. But in the future, I envision a small company of 3 – 5 people who provide and support this code, and are able to make a living doing so. The hope is that I could be independent of, but cooperative with, the main Mozilla Messaging team that supports Thunderbird. (That is, I hope to answer “no” to my question “Do MEALS need a fork?“) At what point should I be adding people to this team? A one-person development team can be quite efficient if it can get the job done, as you eliminate all of the effort associated with maintaining relationships and communication. I think there is a reasonable (50/50) chance that I can get this project done on my own by the end of 2010 and/or in sync with Thunderbird 3.2, as is my current goal. I’ll need to firm this up by the end of June. But I would be open now to beginning to partner with one of the experienced volunteer mailnews developers (you know who you are) who has similar interests.

Posted in Exchange Web Services (EWS), Mailnews development, Planet Mozilla, Planet MozillaMessaging | 1 Comment »

Mailnews Exchange Support: the Ews Native layer

In the last week, I realized that my Exchange integration project needed another layer in the architecture. Previously, I have discussed a layer that does SOAP calls, and a second layer that extends Mozilla mailnews objects. (Let me call the Mozilla-specific world of mailnews objects MMO “Mozilla mailnews objects” for short). I had thought that I would be able to glue these two layers together through a single method that would send and receive Exchange Web Services (EWS) messages, but that did not work out. The complexity of the data that is returned from EWS is too great to be represented in simple callbacks from the EWS SOAP calls. I also did not want to try to reinterpret that data into MMO terms in the relatively low-level SOAP calls.

So what I have done is to add a new layer of objects, that I call the “Ews Native” layer, that contains data in the format that EWS expects. As a simple example of what I mean by that, folders are identified uniquely in EWS using a very long string called FolderId, while in MMO they are a URI which represents their name and location. The Ews Native layer knows nothing about that URI, it just works with FolderIds.

The Ews Native layer currently consists of the following components:

NativeFolder: contains for example FolderId, ParentFolderId, DisplayName (“Inbox”), FolderClass (“IPF.Note”), and various counts.

NativeMailbox: This would correspond to an “account” in MMO. In addition to account information (name, serverURI, password) it contains a cache that owns the native folder instances.

NativeService: This generates references to native mailbox objects, and owns those objects.

The glue between the Ews native objects and the SOAP layer consists of a method that takes as inputs references to the Ews native objects, and generates specific SOAP calls to operate on those objects. The results of the call are then placed in the Ews Native objects. So, for example, there is a “listSubfolders” call, which uses an EWS FindFolder command to generate a list of FolderIds for the children of a particular folder. The request call then updates the native folder object for the parent, adding (or updating) an internal array of strings containing those ids.

In the unit tests, a call to get the children of the inbox looks like this:

// get children of the inbox
 let request = createSoapRequest(gNativeMailbox);
 response = new EwsSoapResponse();
 request.listSubfolders(response, gInboxFolder, null);
 yield false;

Here gInboxFolder was previously created so that its FolderId was set.

“EwsSoapResponse” is just a js object that implements an interface msqIEwsSoapResponse, that has onStartRequest, onStopRequest methods modelled on nsIRequestObserver. I may eventually implement the full nsIRequest support for my calls, not sure yet.

The request is a C++ object that does the SOAP work, interpreting its results into the Ews native object. The request contains lots of information interesting in debugging (such as the call and response XML), but the actual useful data is stored in the Ews native objects.

In the request object, I have four methods implemented currently: getFolder (which updates folder properties), createFolder, deleteFolder, and listSubfolders. The most complex unit test finds the inbox, creates a subfolder, lists the inbox subfolders to confirm the new folder exists, deletes the new folder, then lists the inbox subfolders again to confirm that it is gone.

Using Outlook Web Services, I can see the results:

The “SubFolder nnnnnn” folders were created by my extension unit tests.

The next step is now to glue the EWS native objects to the EWS-specific MMO objects, and use those to start to show some of the real Exchange Server data in the Thunderbird user interface. This is all part of an attempt to do a complete vertical slice through the overall project, focused on the narrow task of showing EWS objects in the Thunderbird folder pane, so that I can develop and evaluate the overall architecture of the project.

Posted in Exchange Web Services (EWS), Mailnews development, Planet Mozilla, Planet MozillaMessaging | 2 Comments »