In my last report, I was testing and updating the old Mozilla SOAP framework for use in communicating with Exchange Web Services (EWS). Using the framework in a Thunderbird 3.0 environment, I could talk to the BING search SOAP interface. It took a couple of more weeks to resolve issues associated with communicating with EWS. There were a number of features missing from the old Mozilla SOAP, most importantly:

• attributes in SOAP elements.
• arrays of choices with unlimited number of elements. The property bag of the previous interface can’t handle multiple items with the same name, so I did my own “property list” replacement for it.
• authenticated access to the SOAP service.

I’ve successfully sent a “Get Folder” call to an Exchange Server. The code to define the specific Exchange call looks like this:

let getFolder = new EwsGetFolder;
getFolder.invoke(responseListener);

function EwsGetFolder()
{
 // defaults
 this.FolderBaseShape = "Default";
 this.FolderIds = "inbox";
}

EwsGetFolder.prototype = new SoapBase(kMessages2007sp1Schema);

EwsGetFolder.prototype.invoke =
  function EwsGetFolder_invoke(aSOAPResponseListener)
{
 let FolderShapeList =
   objectPropertyList({BaseShape: this.FolderBaseShape});
 let FolderIdsList = objectPropertyList(
     {DistinguishedFolderId: objectPropertyList(
       {$attributes: objectPropertyList({Id: this.FolderIds}) })});
 let soapCall = Cc["@mozilla.org/xmlextras/soap/call;1"]
                  .createInstance(Ci.nsISOAPCall);
 let parameters = [];
 let FolderShapeType =
   this._schema.collection.getType(
     'FolderResponseShapeType',
     'http://schemas.microsoft.com/exchange/services/2006/types');
 let FolderIdsType =
   this._schema.collection.getType(
     'NonEmptyArrayOfBaseFolderIdsType',
     'http://schemas.microsoft.com/exchange/services/2006/types');
 parameters.push(
   new soapParameter(
     "FolderShape",
     FolderShapeList,
     FolderShapeType,
     "http://schemas.microsoft.com/exchange/services/2006/messages"));
 parameters.push(
   new soapParameter(
     'FolderIds',
     FolderIdsList,
     FolderIdsType,
     "http://schemas.microsoft.com/exchange/services/2006/messages"));
 soapCall.encode(
   Ci.nsISOAPMessage.VERSION_1_1,
   'GetFolder',
   this._schema.targetNamespace,
   0, null, // header blocks
   parameters.length, parameters);
 soapCall.transportURI = kAccount.url;
 soapCall.asyncInvoke(aSOAPResponseListener,
                      kAccount.user, kAccount.password);
}

So I can now successfully do SOAP communication to an Exchange Server (2007 SP1), using the standard Exchange schemas to encode the messages. There is still much needed to clean this up, but I’ve got enough working for now.

So now I turn my attention to the setup of a new account type in Thunderbird. This is the same issue that jcranmer is working on, but he is specifically trying to do everything in javascript, while I am content (for now) to work partly in C++. So I have to build the darn thing.

Ah, the Mozilla build system. Gotta love it. You have to know the difference between :, ::, :=, =~, (), {}, in a zillion different languages (perl, python, autoconf, automake, bash, make at least). When it works, it works like magic. But sometimes you wave your magic wand, say your magic incantation, and nothing happens. That’s what makes it so hard. You spend hours seeing if you can get the spell to work, but sometimes you just have to give up and do it the old fashioned way, knowing that there is probably some trivial change you could have done to make the magic work. But I guess I have no right to complain. Everyone said to NOT do it this way, but did I listen? I’ve been in build hell for a week.

But it turned out to be just build purgatory! Now that I am out, what wonders heaven holds! I can use a real debugger on my code, and not Venkman and dump()! The vast majority of the existing mailnews code is in C++, so I can see what local folders, IMAP, and newsgroups do, and very easily adapt it to my needs. I can subclass the existing implementations!

So as of today, I have a series of subclassed C++ files that contain what will be the future EWS-based objects – though I have not attempted yet to glue them to EWS, just simulations. I have new C++ files for the folder, message service, incoming server, and summary database – along with some basic unit tests. (This is very similar to the approach that the RSS implementation takes).

So what can I do? I can just display an account, its subfolders, and a message header (all simulated not real). It looks like this:

As you can see, I have not faced the issue yet of registering my protocols, so when I click on the message I get a complaint. I’ll be doing that next, along with trying to get the real information from EWS instead of this simulation.

OK, I know, you are not impressed – one simulated message, and that does not even work?  “Everything is hard to do in Thunderbird” is my defense, so cut me some slack, all right?

rkent

====
Check tinderbox for current checkin status and burns

(cd pristine/src) [This is a directory containing a clone of comm-central from mozilla]
hg pull –u

(cd mozpush/src) [This is a directory containing a clone of my local pristine repository]

hg pull –u

(If there are problems, create a new mozpush directory, and do
hg clone /s/tb/pristine/src src)

hg import /f/tb/patch/name.patch
(above starts notepad, give bug & approvals and save)
hg outgoing -p
(check that the patch and comment are correct)
hg push

(cd pristine/src)
hg pull –rebase
(if changes, you may need hg update, or hg rebase -s revision -d revision)
hg outgoing -p

(if everything looks good)
hg push

Go to http://hg.mozilla.org/comm-central/shortlog and get the diff link for the push, something like:

http://hg.mozilla.org/comm-central/rev/324e7b178ad7

(or http://hg.mozilla.org/releases/comm-1.9.1/shortlog )

Go to bug, comment on checked-in attachment with above link.

Mark resolved/fixed, and add the appropriate pristine/src Target Milestone.

(when you checkin to comm-1.9.1, please set the status-thunderbird3.0 field to “.1-fixed” (or whatever the appropriate dot release is)

Follow tinderboxes.

I considered a variety of approaches to this. One extension “Asertiva Thunderbird Extension for Sugar” uses the IBM/Prototype js library for SOAP access. Others recommended that I consider one of the open source SOAP libraries, such as a python-based library, or Apache’s AXIS2 library. Or that I cooperate with the existing project to provide an open-source method of accessing Exchange server.

But I’m not sure of how “open source” I want all of this to be. From my perspective, “open source” as a charitable activity is not successful. We all need to eat, and so the revenue model needs to be clear if a project is going to be more than a phase of life I am going through at the moment. So I would rather keep my options open until I understand that better. Anyway, that’s a long discussion which is beyond the scope of the current posting, which is supposed to be a status update.

I am still in an education phase, trying to understand SOAP and the related protocols, and to figure out what exactly I gain from using any existing library versus doing things more directly from the raw XML. So as both a trial and education step (and against the recommendation of others I might add) I’ve tried to update the old Gecko webservices extension to work in current Gecko 1.9.2, and to work with some current Microsoft SOAP protocols.

Rather than start with EWS, I started with the simpler BING search calls. I used existing Microsoft demos in C#, and could capture the communications using Wireshark to see what I was supposed to be sending and receiving.

Updating the abandoned webservices extension to Gecko 1.9.2

After testing webservices some under an old Firefox 2 build, I upgraded portions of the code to work under a current comm-central trunk build, using Gecko 1.9.2. My requirements are somewhat simpler than the original extension:

1. Most importantly, my target is chrome-based extensions rather than browser code, so a lot of the security issues that FF folks worried about were not important to me.
2. I had no interest also in allowing native JS creation of components, as I could rely on using .createInstance calls instead.
3. At least initially, I am not supporting the reading of wsdl files, nor the associated automatic creation of proxy calls and interfaces. Instead, I read in a schema file, and generate my own code for each method. My understanding is that Microsoft, in their EWS libraries, also does not actually automatically generate method calls from a WSDL-based proxy, but follows this same approach of starting with the schema files.

That allowed me to avoid about half of the existing code, and focus on the /schema and /soap directories of the webservices extension.

Looking at my hg logs, it took one week of development time, and 17 patches to get to the point where I could write a unit test under Gecko 1.9.2, and confirm that I could create webservices components using a unit test. (I’m doing things in a test-driven development fashion, writing XPCSHELL unit tests to try out different features of webservices.)

Learning and testing Gecko webservices

Although there was a little old documentation around on Gecko webservices, ultimately I just needed to learn things the old fashioned way, reading the code and its interfaces and experimenting to see what worked. I took little baby steps, starting first with an XML schema primer and eventually working my way toward duplicating the functionality in some Microsoft Bing search demo C# code. This phase, starting from the first demonstration of loading of SOAP components under Gecko 1.9.2 through testing of encoding and decoding of a Bing search message, took about 2 weeks of coding and 23 patches, with the creation of 22 unit tests in the process.

The main issue that I had to deal with in the existing SOAP code is that it did not support maxOccurs>1 schema types, such as this one from the Bing schema:

<xsd:complexType name="ArrayOfNewsArticle">
  <xsd:sequence>
    <xsd:element minOccurs="0" maxOccurs="unbounded"
                 name="NewsArticle" type="tns:NewsArticle" />
  </xsd:sequence>
</xsd:complexType>

I solved this by using an nsIArray to hold multiple values of the same element.

Sample Code

So now I can do a Bing search in an XPCSHELL test, and decode and test the results. I want to show some of the js that I use, to give some idea of the complexity (or lack thereof) of using this.

I create a class BingSearch, then encode some values to setup the search. A basic call for a search looks like this:

function run_test()
{
 do_test_pending();
 getNews = new BingSearch();
 getNews.Query = "obama";
 getNews.Options = ["EnableHighlighting"];
 getNews.Sources = ["News"];
 getNews.News = {Offset: 0, SortBy: "Relevance", Count: 1};
 getNews.invoke(getNewsListener);
}

The “BingSearch” object is presumably what a sophisticated library would create automatically from the WSDL file. Instead, I create it by hand. Here’s my partial implementation, that does not support all of the allowed inputs to a Bing search, but works for my tests:

function BingSearch()
{
 // defaults
 this.Version = "2.0";
 this.AppId = "<you get this from Microsoft for your application>";
 this.Market = "en-us";
}

BingSearch.prototype = new BingBase();

BingSearch.prototype.invoke = function BingSearch_invoke(aSOAPResponseListener)
{
 let parametersBag = objectPropertyBag({
   Version: this.Version,
   Market: this.Market,
   Query: this.Query,
   AppId: this.AppId,
   Options: arrayPropertyBag("SearchOption", this.Options),
   Sources: arrayPropertyBag("SourceType", this.Sources),
   News: objectPropertyBag(this.News)
 });

 // soap message component
 let soapCall = Cc["@mozilla.org/xmlextras/soap/call;1"]
                  .createInstance(Ci.nsISOAPCall);
 let parameters = [];
 parameters.push(new soapParameter("parameters",
                   parametersBag,
                   this._schema.getTypeByName('SearchRequest')));
 soapCall.encode(Ci.nsISOAPMessage.VERSION_1_2,
                 'SearchRequest',
                 this._schema.targetNamespace,
                 0, null, // header blocks
                 parameters.length, parameters);
 soapCall.transportURI = "http://api.bing.net:80/soap.asmx";
 soapCall.encoding = this._encoding;
 soapCall.asyncInvoke(aSOAPResponseListener);
}

const kSchema = {
 file: 'data/bing20.xsd',
 schemaNamespace: 'http://www.w3.org/2001/XMLSchema',
 targetNamespace: 'http://schemas.microsoft.com/LiveSearch/2008/03/Search'
 }

function BingBase()
{
 if (typeof this._schema == "undefined")
 {
   this._schema = getSchema(kSchema);
   // use the 2001 SOAP encoder
   this._encoding = Cc["@mozilla.org/xmlextras/soap/encoding;1"]
                      .createInstance(Ci.nsISOAPEncoding);
   this._encoding.getAssociatedEncoding("http://www.w3.org/2001/09/soap-encoding", true);
   this._encoding.schemaCollection = this._schema.collection;
 }
}

So far, the complexity of this does not seem unmanageable to me. I’ve only shown the endoding step. Decoding the response consists of a creating a call-specific translator similar to the “BingSearch.prototype.invoke” function above, which relies on the webservices soap library decode method. All of the other functions I’ve created (such as arrayPropertyBag) are not at all specific to the nature of the SOAP interface being used. I am not seeing the need to process the WSDL file automatically and generate proxy functions.

I’m not yet convinced that this resurrection of the old webservices library is the right approach, but I am not seeing any obstacles to using it either. I can generate and decode soap calls fairly efficiently, debug issues that arise, plus I have code that will integrate fairly easily with either javascript or C++ code in a Gecko chrome environment.

Next steps

I’m trying to decide on the next step toward moving this forward. I’m leaning toward attempting a specific EWS application, such as read-only access to an Exchange calendar as a Lightning extension. Another option might be to add some core mailnews hooks to allow me to create either message accounts or addressbooks using an extension – though I’m hoping jcranmer will beat me to it.

The issue here is that I see lots of good work going on by people who are mostly involved in QA, such as wsmwk, WADA, and Ludo, but I as a developer don’t really know how to make the best use of that work.

I assume there is supposed to be a waterfall here, from (bug reporter)->(QA)->(developer)->(code reviewer)->(bug landing). I understand all of the steps of the process except this (QA)->(developer) handoff. I would be curious to hear from people involved in QA about what they view the main outcome of their work is supposed to be, as viewed by a developer.

Here’s what my understanding is of the current process. After bugs are submitted, QA has three main responsibilities: 1) get the bug in the correct component, 2) move the status to NEW or one of the inactive states (DUPE, INVALID, etc.) 3) clarify the bug information to get clear steps to reproduce.

Is this accurate?

Let’s look to see how that is working by looking at my recent work. In the last three months, I fixed eight bugs (that’s a little off my desired pace, but we were frozen a lot of that time). What brought those bugs to my attention?

(3) bugs I reported myself, either due to issues I observed or as a result of following support forums.

(3) bugs are fixes of crashes that wsmwk reported from crash stats

(2) bugs were filed earlier by others. If I recall correctly, both of those bugs were items that I noticed first in support forums, then located the bug in Bugzilla and fixed it.

In no cases did the standard QA waterfall process play a significant role in bringing a bug to my attention. And that concerns me, because I see some very competent and dedicated people working hard on QA, but I don’t seem to be making effective use of that work.

Am I somehow not following the process that I am supposed to be following, or is that process flawed? In theory I am probably in a better position than most people here, because I primarily track items that appear in the mailnews core/filters component.

I wish there was a clear way for the QA people to bring a limited number of bugs to my attention that are 1) important, 2) clearly defined and reproducible, and 3) likely to be fairly easy to fix.

The mailnews core/filters category currently has 326 NEW/ASSIGNED/REOPENED bugs in it. I could probably fix a few per month that were brought to my attention. A reasonable expectation might be that 10% of those are addressed in the next year. How are the QA folks supposed to influence the selection of which 32 bugs actually get my attention?

(posted to http://mesquilla.com and m.d.a.thunderbird, followup to m.d.a.thunderbird please.)

rkent

In addition to some new filter actions (print, add sender to address list, and save attachments to a folder) this release introduces “custom search terms” for the first time. This is a new feature that has been added recently to the mailnews core code, and is part of the TB 3.0 and SM 2.0 releases.

The search I am talking about is the old-style Thunderbird search, not the newer global database (gloda) search that was added to Thunderbird. Gloda gets all of the press, but we’ve also taught the old style search a few tricks as well! These are particularly useful in saved searchs (also called virtual folders).

I think that the most interesting new capability is that you can define a search (and therefore a virtual folder) by adding a few lines of javascript code that does precisely what you want. Let me give a very simple example.

Let’s assume that you want a virtual  folder to contain all of the active items that you currently need to process involving projects. Incoming emails are marked with tags, either manually or by some sort of filter. You define new tags for each project, and each tag begins with “pro” and ends with some sort of project marker, say a number or word. You want to have your active folder contain messages that have a tag containing “pro”, but NOT include messages that are tagged “done”.

The standard mailnews tag search forces you to enter the tag name for each tag that you want. There is no way to search for tags by the characters in the tags. Plus, searching for several tags is an OR function, and saying to not include DONE messages is an AND function. But the standard mailnews search does not handle complex boolean searches.

The javascript custom search comes to the rescue! Go to the folder that contains the messages that you want, and select the Javascript search term:

Click on the script icon and you will get a small editor window, where you can enter a few lines of javascript. All that we need to do is grab the string that has the tags in it, and make sure it includes “pro” but does not include “done”. Our javacript is given a variable “message” which is the database header object that can be used to get message properties. We need to execute an expression as the last statement, whose value is either true if we want the message, or false if we do not. That’s just the following two lines of javascript:

let tags = message.getStringProperty('keywords');
(/pro/.test(tags) && !(/done/.test(tags));

So here’s what our javascript window and code looks like:

Now save this as a virtual folder, and you have the exact virtual folder that you want!

OK, this is just for geeks, but it is really powerful in letting you define folders that can precisely define the workflow that you want. There’s also a few other geek-friendly search terms, including the much-requested regular expression search by subject or other header. For details, see the updated FiltaQuilla page on this site.

FiltaQuilla is still in experimental status, though I have now nominated it to be public. Still it may be a few weeks before it gets there. If you are an existing FiltaQuilla user, you will need to go to the download page directly and download and install the new version.

Enjoy!

Join the forum discussion on this post - (1) Posts

I don’t have any illusions that this has a significant chance of being implemented, and I’m not even sure it’s a good idea myself. But I ask you to suspend disbelief for a minute, and imagine a change to the development culture and process.

An email client is different from a web client in many ways, but one significant way is that there is no real need for a fat uniform core product that developers can target (such as web developers for FireFox). So we are free to allow wide changes in our product configuration that would not make sense for FireFox. There is really no fundamental need for Thunderbird to be presented as a single, fat, feature-laden client.

Instead, ship Thunderbird as a minimal base with a collection of extensions. The extensions could be in a variety of statuses. At one status extreme, “Core” extensions would be enabled by default, would be fully localized, and their updates would be shipped with updates to the core product, rather than through AMO. Many existing core features would be converted into “Core” extensions that could be disabled if desired (for example bayes junk processing, or gloda.) At the other extreme, “Pilot” extensions would be shipped with the core product in a disabled state, would be updated by AMO, and not fully localized. There would also be “Standard” extensions that are shipped with the product, not maintained through AMO, but would not be enabled by default. Lightning might be one example of this. FiltaQuilla or JunQuilla could easily get added to this category in the future, or popular extensions like ThunderBrowse.

So why would you do such a crazy thing? For several reasons.

First, you would have a path to add features to the program that is not as generally disruptive as has been, for example, gloda or the new message header. By not using a new extension, an existing user would not see changes to their workflow that they did not want or appreciate. Also, new features need not delay the release of new versions of Thunderbird, as “Pilot” status extensions could be updated through AMO.

Second, new complex features like gloda, even though they are developed by the core team (well mostly asuth) are in a state of rapid flux, and would really benefit from allowing updates more frequently than even the accelerated release process will allow.

Third, you provide a natural path for outside developers to add contribution to the product without having to completely submerge themselves in the Mozilla culture, or give up complete control of their creation.

Fourth, this really recognizes that the use of an email client is highly personal. Basic users could be presented a basic email client. Advanced users could easily add advanced features. (Existing AMO-based extensions are also good for this, but the quality is not uniform, and they frequently are not kept up to date. And the standard product is still very fat with lots of features that are unneeded by most users.)

Fifth, this would solve the serious issue with Thunderbird of how hard it is for the average user to install addons (because the most popular and important addons would be shipped with the product).

There’s another dimension to this, and that is the developer’s relationship with his or her extension. I know that I feel a responsibility for my extensions that is beyond the responsibility I feel for any core code. You can see that in the documentation that I provide, and my reliability in responding to issues. I think that many other extension developers are like that as well. I’m guessing that they would be delighted to see a higher level of promotion of their work, without the need to cede complete control that incorporation in core might involve.

I suppose I could write a book on what this might look like, but for now let me leave it here.

rkent

JunQuilla is my attempt to extend the user interface in the Mozilla mailnews product to provide the information that I believe is needed to properly manage the bayesian junk filter. I suppose that most of these features should really be in the core product, but I found that support for that was not very strong, so I decided to do most of this in an extension instead. These backend features have only been added to the core code in the last couple of years, so this extension will only work on newer versions of the Mozilla email clients (Thunderbird 3.* versions, and SeaMonkey 2.* versions.)

Version 1.0.0 fixes some bugs that have been reported in previous releases, provides partial support for SeaMonkey (except for the “Uncertain” folders), and adds a number of new features:

Junk Options

You can set critical overall junk options in the standard junk options screen (previously, this was only possible in the more obscure addons/options area). In Thunderbird, select Tools/Options/Security/Junk. In SeaMonkey, select Edit/Preferences/Mail & Newsgoups/JunQuilla. There you will see a display like this:

“Junk threshold” is the percentage value as calculated by the bayes classifier for each message, above which a message will be classified as junk. This should be set as low as possible, though always high enough to avoid having any real messages classified as junk. The default value of 90 is much too high for a well-trained bayes classifier.

The “Maximum token count” is a measure of the resources that the junk classifier will use. The higher it is set, the more accurate your classifications will be. The default value of 100,000 is probably too low for good classification performance. I’ve had good results with 300,000 – and JunQuilla will set your value to this when first installed. If this value is too high, and you have trained a lot of messages, then memory usage may be excessive.

The other parameters are read only, and are displays of values from your training file. The “Current token count” shows how many junk training tokens (which are like words) are currently in use. You probably won’t get good performance until this number is over 10,000 – and it really should be more like 100,000. “Good” and “Junk” messages trained shows how many messages have been used to train the junk filter. Ideally the number of junk and good messages should be more or less equal. If they are not, then pick some previously untrained messages and train them.

When the number of tokens exceeds the maximum value, then Mozilla mailnews will prune the training file in a large chunk, typically reducing both the number of trained messages, and the number of tokens, in about half.

Disable/enable junk processing for a folder.

You can set an “inherited folder property” to allow you to selectively enable or disable junk processing for folders. This has two main uses.

• If you have server-side filters that process email in IMAP, then you may already know that certain folders contain either junk mail or good mail, and don’t want to waste time processing them locally – or take the risk that they will be processed incorrectly.

• Mozilla mailnews core code now supports junk processing of RSS and News folders. You can select certain RSS or News folders, and then junk processing will run on new posts to those folders. This will also enable the standard user interface features that allow you to train messages as good or junk in those folders.

To set this, right click on a folder in the folder tree, and select Properties, then the “General Information” tab. At the bottom, you will see this:

“Analyze Junk” is an inherited folder property. What that means is that each folder can either gets its value from its parent, or can be set locally. The default value depends on the characteristics of the folder itself. So for example, this would be disabled in News by default, but enabled in IMAP. To change the value, first reset the “Inherit” checkbox, then set the value that you want in “Enabled”. If you change a value for a folder, then the value will also change for the children of that folder (assumming that they have the default “Inherit” checked.)

Toolbar “Is Junk” and “Is Good” button

You can add two new buttons to your toolbar – “Is Junk” and “Is Good”. Here’s what they look like, next to a standard “Junk/Not Junk” icon:

To add these buttons, right click on a toolbar, select “Customize”, then drag the buttons to the desired location.

The standard Junk button will always show as “Junk” when it thinks a message is good, and “Good” when it thinks that a message is junk. But that means that we can only classify a message as “Good” when it has been falsely classified as junk, and we never want our junk filter to do that. The “Is Good” button is meant to be used in the “Uncertain” folders to give you a means to train a message as “Good” there.

Join the forum discussion on this post - (3) Posts]]> http://mesquilla.com/2009/11/16/junquilla-version-1-0-0-released/feed/ 0 http://mesquilla.com/2009/11/06/inherited-folder-properties-revisited/ http://mesquilla.com/2009/11/06/inherited-folder-properties-revisited/#comments Fri, 06 Nov 2009 19:50:52 +0000 rkent http://mesquilla.com/?p=317 In a previous posting, I introduced the concept of inherited folder properties in the Mozilla mailnews products (Thunderbird and SeaMonkey). In the months since, I have incorporated these into my extensions quite significantly, so here I would like to show the UI I am currently using for this, and also discuss some of the issues that I face.

(All references to extensions in this posting refer to the 1.0.0 versions, which as of this writing have not been posted to AMO yet. But they should be available in a few weeks.)

Implemented UI

Briefly, inherited properties are a property that can be defined globally, at the server, or at the folder, and its characteristics will be propagated to child objects. This make it easy to specify precisely how the property is applied.

As an example, I have recently implemented a feature “Index in Global Database” in GlodaQuilla which can be used to selectively suppress certain accounts or folders from being accessed by the global database indexer. In the account manager, where indexing can be disabled for an entire account, the UI looks like this:

Each inherited property has default values which are typically set by the base code. In the case of the gloda database indexer, everything but newgroups are indexed by default. Initially each inherited property is set to just use the standard default processing, but if I clear the “default” checkbox, then I can turn off gloda indexing for this account.

If I do that, then go to a first-level folder in the account, I see the following under folder properties:

At the folder level, because I disabled global indexing on the account, it is now shown as disabled on the folder. I could clear the inherit box and selectively enable it on just this folder and its children if I wanted.

This particular UI merges naturally with the existing methods of setting properties in mailnews, but I’m not sure it is optimum for an inherited property. The inherited nature could be more clearly shown, and a particular feature more quickly configured, if I showed a tree of accounts and folders, with checkboxes next to each account to enable or inherit the feature. Maybe in a future version.

Implemented properties

Here are some of the implementations of inherited properties that exist in my extensions:

1. (GlodaQuilla) Index in Global Database – suppresses the running of the global database indexer
2. (FiltaQuilla) Apply Filters to Folder – for Imap folders, allow incoming filters to run on that folder
3. (JunQuilla) Analyze Junk – allow junk processing to be turned on or off. This also allows junk processing to run on RSS or news folders.
4. (TaQuilla) Analyze particular automatic tags.

Issue: Existing mechanisms

Ideally, the inherited property would be the one and only way to manage a program feature. But for existing features, the existing mechanisms remain, which can lead to possible confusion. For example, with JunQuilla’s “Analyze Junk” property, there is existing UI to enable junk processing at the account level. Here the inherited property will always override the default mechanism (but that is mostly because I implemented it in core that way, and I have a little influence on how junk processing is handled in core.) For GlodaQuilla’s “Index in Global Database” the behaviour is different. Existing UI will only allow this to be enabled or disabled globally, and the inherited property does not override this. The inherited property code uses the default server preference as a global enable/disable for a property, so if gloda used that same mechanism instead of an independent preference, this issue would go away. I guess I could say the same thing about junk processing as well.

For FiltaQuilla’s “Apply Filters to Folder”, there is a subtle issue in the inherited nature. I did not implement in core the ability of the inherited property to override the existing default as applied to the Inbox, so incoming filters always run on the inbox. That creates 2 ui issues. First, although I show at the account level the “Apply Filters” option, it does not actually suppress application to the inbox as one would expect. Second, I currently do not show a folder property for “Apply Filters to Folder” for the IMAP inbox since it would not make sense there, so that also means there is no way to enable processing of filters for the children of the inbox. Maybe I should call this feature instead “Apply Filters to non-Inbox Folders” to solve this, or change the core code so that the feature also applied to the Inbox.

Issue: My RDF-inspired property for junk management

Looking ahead to a world where a number of extensions might try to define bayes filter traits, in code I recommended that properties used to manage junk processing use an RDF-inspired globally unique identifier. Then I followed my own advice and defined the identifier that controls junk processing on a folder as: “dobayes.mailnews@mozilla.org#junk” Unfortunately, the existing account manager code does not allow periods in property names, which means I could not use the account manager to manage this. I’ve filed bug 525024 on this issue, and perhaps that can be incorporated after TB 3.0 / SM 2.0.

Issue: Missing inheritance levels

I’ve heard others comment that often they want to set a property on a particular class of folders, say on all Trash folders, or all Sent folders. I’ve considered implementing another level in the inherited properties feature, that would be a folder type. So you would then set a property that would be inherited by any folder of a particular type, and its children – and of course also overridden by the local folder property.

Issue: UI for global property

All of the inherited properties could also be enabled globally using the “mail.server.default.<property>” preference, but I did not give any UI for that in my extensions. I thought that would be too confusing for the user to show those preferences, which would be very similar to existing mechanisms. This is not an issue for properties that use the preference system for server-level issues, but none of the existing server-level preferences are also inherited properties. Perhaps we could move that direction in the future.

The default spam processing that is available without using filters (whitelisting, move or delete messages with a sufficiently high threshold) should be sufficient for most users. But for people with special requirements you can now implement those requirements in a filter with customizations. Let me give examples in this posting.

Using the “after classification” filters

Proper care and feeding of spam really needs to classify messages in three ways. Some messages can be easily detected as spam, and should never be looked at. Others are clearly ham, and should be treated as real. But those in the middle need some handling, which may be either training, or perhaps examined weekly to make sure no false positives are there. Default Mozilla mailnews (which is the generic term for features that are available in any of the applications created from this codebase, including Thunderbird and SeaMonkey) junk management doesn’t provide any capability to manage these uncertain emails. My JunQuilla extension provides an Uncertain folder which is focused on the training issue, but with the new  filter features you can have more precise control of this. (Currently you can’t install JunQuilla in SeaMonkey, but I will fix that eventually).

First, let’s see what is new and how it can be enabled.

Create a new filter by selecting Tools/Message Filters … then New. Open up the search attributes menu, and you’ll see something like this:

No junk options! To get those, you’ll need to first select one of the “after classification” contexts from “Apply Filter When”. Then you’ll see something like this:

If “Checking mail (after classification)” is disabled, then you probably are trying to set an after-classification filter on a POP3 account that is actually sending its email to another location (the so-called deferred-to server). You need instead to set the “after classification” filter on the “deferred-to” server, which is typically Local Folders.

Let me explain each of these search attributes.

Junk Percent is the score returned from the bayes filter when classifying the message, with 100 being the most likely to be junk, and 0 the least likely. The default setting in Thunderbird classifies a message as junk when this score is 90 or greater. It is sort of a probablility, but not really because too many false assumptions are made in the Naive Bayesian Classifier for this to really be a probability. Just treat it like a score. Unfortunately default installs of Thunderbird and SeaMonkey do not provide you with any way to see the value of this on typical messages. JunQuilla though provides a custom column that shows this on each message so you can get a feel for what typical values are.

Junk Status is pretty simple, it either Is or Isnt Junk. In the normal case where the internal bayes filter is used to classify the message, this means it had a junk percent of greater than 90.

Junk Score Origin shows you who classified the message. Its values are:

Plugin: the bayes filter.

User: you manually classified this message as junk or good (not useful in an incoming filter, but maybe in a manual filter or search).

Filter: a previous filter action set the junk status.

Whitelist: the spam processing decided this message was good because it was from someone in your address book.

IMAP Flag: this message was classified by another system, so we know it is junk or good, but don’t know why the other system classified it this way. You might see this if you access mail from more than one computer.

Default Thunderbird does not support any way to see the junk score origin on individual messages, though JunQuilla provides a Junk Status + column which uses different icons for each junk score origin.

Classifying messages as uncertain

So let’s design a filter that will move messages to an Uncertain folder if we want to examine them, but not have them clutter the inbox. That’s pretty easy, we’ll just move messages with a junk percent in a certain range to that folder:

The order of message processing in Mozilla mailnews is:

1. Run normal filters (on each message as it is received)
2. Check whitelisting (on a message batch, this and subsequent steps)
3. Run bayes classifier on non-whitelisted messages, and mark messages as junk or good.
4. Apply “after classification” filters.
5. Apply junk message moves using default junk processing.

So at least in theory, you can apply the “after classification” filter to the Uncertain messages, and still let the default junk processing move junk messages to a Junk folder. (Testing of this is welcomed!)

Weak Whitelisting

As a more complex example, spammers are starting to send out emails that have spoofed From addresses that match the domain of your email, figuring that there is a chance that you have these other addresses whitelisted, so you’ll get the spam. To fight this, we’ll setup a filter that does a whitelist that is slightly weaker than the usual all-or-nothing whitelist on those easily spoofed addresses. Because whitelisting occurs before spam processing, and no score will be assigned if the message is whitelisted, you will need to disable the default whitelisting functionality, and rely entirely on message filters for this to work.

We’ll add the following search terms, all of which must match to apply our weak whitelist:

1. From address appears in an address book (this is normal whitelisting)
2. My domain appears in the address (because that is easily spoofed)
3. Junk Score Origin is Plugin (this prevents the filter from running on messages that we classified, in case we run it manually on existing folders).
4. Junk Status is Junk (we’ll only whitelist if the bayes filter thought it was junk. I only do this so that I can see that the filter decided to override the decision of the bayes processor, which needs the Junk Status + column from JunQuilla to see.)
5. Junk Percent is less than 95 (since the bayes filter only marked messages as junk with the percent > 90, this means that we will override the bayes decision for messages between 90 and 95 in score).

Putting this all together, you get a filter that looks like this:

You would also need to define a filter that is applied after this one, that whitelists any messages that meet the normal whitelist criteria, but were marked as junk by the bayes filter.

I’m not necessarily recommending this filter, it was meant as a demonstration. But I hope you can see that the new ability to use the bayes filter in combination with other message criteria in a filter provides lots of new possibilities for more precise handling of possible spam messages.

So here are things that are new in Thunderbird 3.0 beta 3 (Seamonkey 2.0 beta 1) that involve message filtering:

1. Imap filtering on folders that are not the inbox.

If you have a server filter that moves IMAP messages to a folder other than the inbox, previously you could not use TB filters on it. Now you can, controlled by an inherited folder property. There is no user interface for this feature at the moment, but I will add some to FiltaQuilla. See bug 257415.

2. “From, To, Cc, or Bcc” filter/search term

This is a new search term that can be used in advanced search or filters. It does two things: combines all of the normal address fields into a single term, plus adds support for Bcc for the first time.

The support for Bcc needs some comments. Just in case it isn’t clear, Bcc information is not added to outgoing emails that you send (or the incoming emails that you receive), so the only place it really shows up is in emails that you sent, and have kept your own copy of. So the Bcc term does nothing for emails that you receive, only emails in your sent folder. Also, we currently do not have any mechanism to automatically apply filters to messages that you send, so really the main place Bcc will show up in filter/search is in Advanced search, saved searches (virtual folders), or in manual filters (filter-after-the-fact).

3. Junk fields in manual filters

We have now enabled the junk-related search terms of “Junk Status”, “Junk Percent”, and “Junk Status Origin” in manual filters.

There is a subtle change to the behavior of search term editing that was implemented that affected this. Previously, if you caused the filter editor to select an invalid search term (typically by changing the filter context away from “Manual”), that was ignored, and you could save the invalid filter. Now, the invalid terms are grayed out, and attempts to save the filter will result in an error dialog. This behavior was important to allow adding of more complex filtering options.

4. Custom Search Terms

The biggest change that affects filters is the ability for extensions to add custom search terms. This is a complement to the addition of custom filter actions, which were available in beta 2, and are the main point of FiltaQuilla. I expect to do a blog post soon detailing the code necessary for an extension to add a custom search term, plus I will add some custom search terms to FiltaQuilla (probably a regex search on a header).

See bug 495519 for details, including a demonstration extension that adds custom terms.

5. Fix of message copy/move bug.

There has been a long-standing bug involving failed moves or copies, that typically occurred when there were automatic copies or moves of emails, such as are done in filters or junk processing. Bug 497622 hopefully solves this, which should improve the reliability of moves and copies.

What did’t make beta 3, but should be in TB3.0

There are two big changes that have mostly working patches, that did not make beta 3 but hopefully will be available in beta 4.

1. IMAP body filters.

IMAP messages are now downloaded by default, so in theory it should be easy to filter on their bodies. This is bug 127250 (or possibly the closely related bug 67421). This bug is really critical in combination with the new custom search attributes, as it would be very valuable to write filters that analyze each message body, and then take a particular action when something is found there.

2. Post-bayes incoming filter context

Currently, it is not possible to filter on any of the message characteristics that are determined by the bayes filter (including junk status, and the closely related custom traits) automatically while a message is incoming. This is because the bayes analysis, and its related filtering actions, are applied after the standard filters. A post-bayes filter context will allow the user to specify that a particular filter should be applied after the bayes analysis, not before, and then we will be able to enable the junk-related search terms (and an extension could add a filter term that used a custom bayes trait). See bug 198100.