The most current version of ExQuilla is available from the MesQuilla site here. A slightly older version (due to review delays) is available from Mozilla’s addon site here.

This version supports access to messages and personal contacts from a Microsoft Exchange Server 2007 or 2010 [...]]]> I am pleased to announce that ExQuilla, a Thunderbird addon that provides access to messages and contacts in Microsoft Exchange Server, is now available. This is a beta-quality release.  ExQuilla is currently only available for Windows, in English, and supports Thunderbird version 9, 10, and 11. Linux and Mac versions are planned for the future.

The most current version of ExQuilla is available from the MesQuilla site here. A slightly older version (due to review delays) is available from Mozilla’s addon site here.

This version supports access to messages and personal contacts from a Microsoft Exchange Server 2007 or 2010 using Exchange Web Services (EWS). Calendar support is not currently enabled, but is under development.

For an overview of ExQuilla, visit the ExQuilla support site.

Kent James

As review, SkinkGlue is my attempt to add an interface layer to Thunderbird to allow new account types to be added using javascript. SkinkGlue provides the necessary C++ interfaces to allow standard Thunderbird objects to be created without writing C++ code. SkinkGlue is licensed inder the standard Mozilla licenses, so it could be used by [...]]]> New account types in Javascript for Thunderbird (Part 4)

Now that I am essentially feature complete in adding Address Book features to my Exchange Web Services implementation ExQuilla, I’d like to discuss how SkinkGlue was used to allow this to mostly be done in JavaScript.

As review, SkinkGlue is my attempt to add an interface layer to Thunderbird to allow new account types to be added using javascript. SkinkGlue provides the necessary C++ interfaces to allow standard Thunderbird objects to be created without writing C++ code. SkinkGlue is licensed inder the standard Mozilla licenses, so it could be used by other extensions, or implemented as a Thunderbird standard feature. The current source is available here.

Adding Exchange Server contacts to the address book

While ExQuilla itself is mostly written in C++, I thought I would try to do the interface to the address book mostly in javascript.

At first I thought that would be straightforward, as the address book interfaces are much simpler than the mail interfaces, and there was supposed to be an existing extension that already did this. But I could not make this work reliably by trying to create a full javascript implementation of the nsIAbDirectory interface.

The main culprit is that the address book implementation seems to require that directory objects also derive from nsRDFResource. The underlying interface nsIRdfResource has the non-scriptable method GetValueConst, which is used by core Mozilla code in accessing RDF items. Perhaps I could have figured out how to work around this, but given the work I had already done in SkinkGlue, it seemed easier to just add a SkinkGlue type that allowed me to create an nsIAbDirectory object.  Another advantage of this is that I could take advantage of the existing full  generic C++ implementations, and then only override methods that I needed to change.

So in the end, I created a SkinkGlue object to be used for address book directories that can be extended using javascript. The C++ implementation of this derives from both nsRDFResource as well as nsAbDirProperty, and provides the minimum needed interfaces. This is actually quite parallel to the mailnews SkinkGlue objects, as the address book object nsAbDirProperty is similar in role to the mail object nsMsgDBFolder. Both provide a generic C++ implementation of key interfaces, and are used in standard mailnews C++ objects as a parent class. By adding this to SkinkGlue, I am simply allowing javascript users to follow the same methods that C++ users employ.

For address book cards, this was not necessary. There is an existing object “@mozilla.org/addressbook/cardproperty;1″ which works just fine.

The result is that the standard Exchange Server “Contacts” folder and subfolders appear in the Thunderbird address book:

Note this is a Mork-free implementation, unlike the mailnews SkinkGlue objects.

Since the address book interface does not support a tree of directories, I had to implement the subfolder “Test subfolder” as a top-level address book with an extended name.

I hope to demonstrate this publically soon by adding “address book” support to my TweeQuilla Twitter extension, where an address directory would be interpreted as a list of followers.

Changes needed to Address Book to make this easier

The address book interfaces are much cleaner on average than the mail interfaces, but unfortunately not completely there yet. What would make this easier to do?

1. Remove address book dependencies on RDF

The use of RDF is ultimately what forces me to create a C++ object to backup my javascript implementation.

2. Add async access capability to key interfaces

Most new address book accounts that might be supported would be remote sources. While the existing LDAP implementation manages to work around that, it is not at all clear from the interfaces how this is done. Really async access needs to be thought through and applied consistently. In my implementation, I essentially end up loading all of the contacts into memory, which will not scale well to large implementations.

This release adds support for most primary email features, but not the EWS address book or calendar. However, this is still Alpha quality code, which means that many of the features have minor glitches.

Further information is [...]]]> Today I release alpha version 0.2 of ExQuilla Exchange Web Service. This may be downloaded at the AMO site for the ExQuilla.

This release adds support for most primary email features, but not the EWS address book or calendar. However, this is still Alpha quality code, which means that many of the features have minor glitches.

Major Additions since last release

• Cross-account message copies and moves
• Message Filters
• Junk Processing (though moves of junk messages are not working)
• Virtual Folders
• Save message as a file
• Show message source
• Message print
• Improvements in handling of Unicode character sets

Further information is available at the ExQuilla page on this site.

This is alpha quality software, which means it is usable if you are brave. (I have been using it for a month with my rkent@mesquilla.com email account).

There are many, many non-implemented features. Rather than repeat all of that here, I’ll just refer you to the extension description page that I have put up. The most severe restrictions are that this is email only, and there is no attempt to support localization [...]]]> Today I released to Mozilla’s addon site a preview release of my Exchange Web Services extension, which I am calling ExQuilla (see the addons page here to download).

This is alpha quality software, which means it is usable if you are brave. (I have been using it for a month with my rkent@mesquilla.com email account).

There are many, many non-implemented features. Rather than repeat all of that here, I’ll just refer you to the extension description page that I have put up. The most severe restrictions are that this is email only, and there is no attempt to support localization or internationalization (so your Russian or Thai spam will look really strange.) Also, this build only supports Windows users running Thunderbird 3.1 Use at your own risk!

There is one major bug that I am aware of. For reasons that I have been unable to determine, sometimes the extension quits working, and sends a variety of messages to the error console. You need to restart to solve this. Hopefully I’ll figure out soon what is causing this.

The initial download of messages can be quite slow, which seems to be a limitation of Exchange Server (at least for the provider that I am using). But after the initial sync of messages in a particular folder, future use is generally quite fast, so be patient.

I would be interested to hear people’s experience in trying to get this to work with their own Exchange implementations. It really is too early though to give detailed feedback or bug reports, but I would be interested in anything that prevents it from being used at all for your needs.

I’ll be away most of next week (going to a prisoner re-entry conference and visiting a reentry facility affiliated with my church) so it may be November until I can get back to you if you have issues.

Originally I had assummed that there would be changes required in core to support what I am doing. But so far, that has not been absolutely necessary. I had problems earlier with the database code, but I’ve managed to get around that by just using a standard nsMailDatabase object instead of doing my own version.

So I tried to build a version that could [...]]]> Things have been a little slower lately, what with summer and family visits and so forth, but my project to support Exchange server in Thunderbird proceeds nevertheless. I’m starting to think seriously of an Alpha release.

Originally I had assummed that there would be changes required in core to support what I am doing. But so far, that has not been absolutely necessary. I had problems earlier with the database code, but I’ve managed to get around that by just using a standard nsMailDatabase object instead of doing my own version.

So I tried to build a version that could run on current Thunderbird 3.1 releases. That required staring at the build code for a couple of weeks, which I guess is the price that I pay for writing much of this extension in C++.  In the end, the magic that I needed was surprisingly simple:

ifndef MOZ_STATIC_MAIL_BUILD
 EXTRA_DSO_LIBS = msgbsutl
 else
 SHARED_LIBRARY_LIBS =
   $(DEPTH)/mailnews/base/util/$(LIB_PREFIX)msgbsutl_s.$(LIB_SUFFIX)
endif

I am now running my EWS extension with my normal Thunderbird profile – but I have not had the courage yet to use the email address for anything real. Fortunately, the spammers of the world have discovered my test address rkentjames@caspia.org, and are kindly providing me with a wide variety of test emails.

The question I have now is what is the minimum functionality required for an alpha release. I’ve recently added the ability to read attachments from my Exchange messages, but I don’t yet have the ability to create emails that contain attachments. I think that after I add that ability, I will have a basic email read/write capability, and will stop adding features and concentrate on fixing annoyances prior to releasing an alpha.

I’ll still be missing a zillion features, including:

• using the exchange address book
• copying and moving messages to other folders
• message filters

and a whole lot more. But the alpha application could be used to do a basic check of someone’s work email from a home Thunderbird installation.

Perhaps it would be interesting to show how I hooked into the sending function in the user interface. I asked the usual suspects, and it was not clear to anyone that it could be done without adding backend hooks – which I would like to avoid as much as possible to increase my chances of getting some initial alphas of this released to work with existing TB 3.1 users.

It turned out to be fairly straight forward. In MsgComposeCommands.js there is an observer notification that occurs, called [...]]]> I can now send a message through Exchange server from my Thunderbird installation.

Perhaps it would be interesting to show how I hooked into the sending function in the user interface. I asked the usual suspects, and it was not clear to anyone that it could be done without adding backend hooks – which I would like to avoid as much as possible to increase my chances of getting some initial alphas of this released to work with existing TB 3.1 users.

It turned out to be fairly straight forward. In MsgComposeCommands.js there is an observer notification that occurs, called “mail:composeOnSend”. This occurs right before the UI is ready to call SendMsg on the nsIMsgCompose object gMsgCompose. So what I needed to do was to intercept that call, and implement my own version of SendMsg rather than the SMTP/Rfc822-focused standard C++ code. To do that, I create a new object when I receive the notification with the old object as its prototype, then include a custom SendMsg that only applies if the sending account is an Exchange server. The overlay code ends up looking like this (greatly simplified):

function observe(subject, topic, data) {
 // wrap the gMsgCompose object so that we can detect attempts to
 //  send using ews.
 let newCompose = new ewsCompose(gMsgCompose);
 gMsgCompose = newCompose;
}

// ewsCompose provides a wrapper around the compose object, so that we
//  can override functions.
function ewsCompose(oldCompose) {
 this.oldCompose = oldCompose;
 this.SendMsg = function ewsSendMsg
     (msgType, identity, currentAccountKey, msgWindow, progress) {
   if (incomingServer instanceof Ci.msqIEwsIncomingServer) {
     // sending using Exchange Web Services (details not shown)
     // ...
     ewsCompose.sendMsg();
     return;
   }
   else
     return this.oldCompose.SendMsg
       (msgType, identity, currentAccountKey, msgWindow, progress);
 }
 this.__proto__ = oldCompose;
}

This seems to work just fine. It was a little tricky getting the compose code to believe that I had actually sent the message, so it could quit complaining of an unsent message. What I ended up doing in my “sending succeeded” callback is to add a few shutdown calls:

let ewsEventListener = {
  // msqIEwsEventListener implementation
  onEvent: function onEvent(aItem, aEvent, aData, aResult) {
    if (aEvent == 'StopRequest') {
      // ewsProgress was saved from the "progress" variable
      //   in nsMsgComposeCommands.js
      if (ewsProgress)
        ewsProgress.closeProgressDialog(aResult == Cr.NS_OK ? false : true);
      stateListener.ComposeProcessDone(aResult);
      MsgComposeCloseWindow(true);
    }
  }
}

With the ability now to send and receive messages, I’ve completed a single vertical pass through all of the key functionality. There are many, many details that I have passed over in the process. But it’s time to start thinking about what I would need to make work to get a usable alpha release of this, perhaps in about six weeks.

EWS messages do not come from the server in RFC-822 format, so it seems like a pity to store them that way, though that is the common method used in the rest of the [...]]]> My project to provide Exchange Web Services (EWS) support to applications based on the Mozilla mailnews codebase entered a new phase this week, where I am starting to consider the issue of local persistence of data downloaded from the server. (In the previous week, I got two other things working: display of HTML emails, and updating of UNREAD status from the local app to the server).

EWS messages do not come from the server in RFC-822 format, so it seems like a pity to store them that way, though that is the common method used in the rest of the mailnews codebase. Instead, I decided to implement a local storage scheme based on SQLite and Mozilla’s Storage interface. Andrew Sutherland has done a lot of great work setting up an environment similar to this for the gloda database, so there are lots of good examples to pull from. Also, because the datamodel for EWS includes not only messages but also Calendar and Contact items, I can have a common database infrastucture that I can leverage over those other pieces once I get it working for the messaging part.

I’ve now replaced my previous in-memory datastore for message metadata with an SQLite version. This is equivalent to the datastore module in gloda, and the data it is storing is like the RFC-822 headers. I still have to do the storage for the body, and also hook this up with folder change state so that the code knows that it has data it can trust.

As I have done this, I’ve had a new set of insights into the relationship of the various objects in the Mozilla mailnews world (which I sometimes call Skink). Previously, I had sort of expected that the natural progression of gloda would be to slowly displace the role of the message summary database, nsIMsgDBHdr.  But now I see that a more natural progression would be for SQLite to be used as a replacement for the local mailstore (currently mbox, with maildir support moving forward as well.) Really the main issue is the async nature of the SQLite calls, which sort of precludes its easy use as a replacement for nsIMsgDBHdr. But the datastores are typically accessed async anyway. If the message metadata in the message stores was stored primarily in SQLite format, as I will be doing, then it would be much easier to hookup an SQLite-based global search facility to all of these databases. Yes that is what gloda does now, but it has to go through all of the work to maintain a separate version of everything. Why have three copies of everything (Mork, MBox, and Gloda) when you could only have two (Mork and Gloda)?

As another insight, while looking through the gloda code I noticed that a JSON object was being saved to store some of the items. I though that was a good idea at first – but then I tried to write a simple serializer to convert from my internal native format to JSON objects, and saw that it was not going to be an easy project. But then I remembered that SOAP is really just a mechanism to serialize typed objects, and I already have a SOAP encoder and decoder! So instead of using JSON, I use objects serialized with my SOAP XML encoder to store unindexed items in my SQLite store. So a message (sans body) ends up looking like this as a TEXT item in SQLite:

<Message xmlns="http://schemas.microsoft.com/exchange/services/2006/types">
 <Subject>Postini First Junk Email Safely Quarantined</Subject>
 <DateTimeReceived>2010-06-04T22:19:30Z</DateTimeReceived>
 <Size>2612</Size>
 <Importance>Normal</Importance>
 <DisplayTo>Kent James</DisplayTo>
 <Culture>en-US</Culture>
 <Sender>
  <Mailbox>
   <Name>Postini Support</Name>
   <EmailAddress>noreply@hostedmsexchange.com</EmailAddress>
   <RoutingType>SMTP</RoutingType>
  </Mailbox>
 </Sender>
 <ToRecipients>
  <Mailbox>
   <Name>Kent James</Name>
   <EmailAddress>rkentjames@caspia.org</EmailAddress>
   <RoutingType>SMTP</RoutingType>
  </Mailbox>
 </ToRecipients>
 <From>
  <Mailbox>
   <Name>Postini Support</Name>
   <EmailAddress>noreply@hostedmsexchange.com</EmailAddress>
   <RoutingType>SMTP</RoutingType>
  </Mailbox>
 </From>
 <InternetMessageId>&lt;0c34b5a4-5f3c-4654-bf9d-99c9a8cb439b@HUB02.4emm.local&gt;
 </InternetMessageId>
 <IsRead>1</IsRead>
</Message>

At first it bothered me to save what is essentially a duplicate of what is coming over the wire, but why not? It’s not conceptually any different than RFC-822, or JSON, in function.

Thunderbird dies a horrible death if I click on any of the actions in the header though. Got to [...]]]> In my last post, I thought it was going to be tricky to get the message header to display in my Thunderbird Exchange Web Services extension. Turned out it wasn’t so hard after all. After I display the body, I just had to pretend like I was the Mime processor, and spit out headers to an nsIMsgHeaderSink that is listening on the message window. The details of what to do are in nsMimeHtmlEmitter.cpp. The result:

Thunderbird dies a horrible death if I click on any of the actions in the header though. Got to work on that.

Why no header information? Because the header summary, for reasons that I cannot explain, reads the message file directly – and assumes that the message is in RFC 2822 format. But Exchange Server has already done all of that parsing, and separated the message into its components, including metadata (like subject), body, and attachments.

An important issue going forward for the Mozilla Mailnews (Skink) codebase is, what [...]]]> Just as a status update, my Exchange Server extension can now read message bodies. But note that there is no header information displayed with the message:

Why no header information? Because the header summary, for reasons that I cannot explain, reads the message file directly – and assumes that the message is in RFC 2822 format. But Exchange Server has already done all of that parsing, and separated the message into its components, including metadata (like subject), body, and attachments.

An important issue going forward for the Mozilla Mailnews (Skink) codebase is, what exactly is the thingy that Thunderbird purports to be? Is it a processor of RFC 2822 messages? Or is it something more, that easily incorporates other forms of communications? I hope to write a longer post on that issue soon.

So here’s a picture to show where I am at. The two messages are real messages that exist on the Exchange server, and have been read by my extension. The message body is not real:

One problem that I had to solve is to make the nsMsgDatabase object extendable. The standard code for [...]]]> Now that I have a state machine to handle situations that need multiple SOAP calls, it was not a big leap forward to hookup the SOAP requests to the mailnews summary database. That’s what gets the messages displayed in Thunderbird’s thread pane.

So here’s a picture to show where I am at. The two messages are real messages that exist on the Exchange server, and have been read by my extension. The message body is not real:

One problem that I had to solve is to make the nsMsgDatabase object extendable. The standard code for that does lots of static casts to the pointer for the XPCOM object, which either caused crashes, or duplicate database caches, depending on the approach that I took to get around that.  I ended up writing a patch for the base code to solve this, which presumably I will land for TB 3.2. I still think there must be some way to cleverly avoid this problem without patching the base code, but I could not figure it out.

I’m not real happy about requiring a base patch for this work, because I had hoped to be able to release a prototype for this on Thunderbird 3.1 / comm-central 1.9.2, even though the main release should be in Thunderbird 3.2/ comm-central 1.9.3. Maybe I can still figure out a solution that does not require a base patch.

Fun with Branding

I’ve talked about the Colonial layer and the Native layer in my application. What’s above the Colonial layer? I want to be able to distinguish between the code shared with Firefox, and the specific code that is used by messaging applications (Thunderbird and SeaMonkey), because I want portions of my work to be loadable into a FireFox extension (just in case). It could be called by the name of the main directory (mailnews) or that directory’s parent (comm-central). Personally I find mailnews very unattractive as a brand. (A bug asks “What’s the name of MailNews?”)

Although I got no traction when I suggested it on m.d.a.thunderbird, I’m going to experiment personally with calling the Mozilla code that implements messaging functionality, and is not included in Firefox, as the Skink layer instead of “mailnews” (following Mozilla’s lizard analogy and Gecko). Yes I am being bullheaded, but this is my personal blog, so I can do what I want, right?

I also managed to register the domain exquilla.com, as ExQuilla is the logical brand for my Exchange Server integration functionality.