In the Thunderbird 3.0 / SeaMonkey 2.0 series, I snuck in a little hidden preference to allow modifications to the way that the TB bayes filter does it tokenization. The main point of this was to allow better transfer of information between SA and TB. I’d like to describe here how to use that feature.

SA’s decision process involves two key steps. First, they evaluate the message with zillions of rules, and tag the message with each rule that the message hits. Second, they have a method of combining all of those tags into a final junk score for the message, which is used to decide whether to tag the overall message as spam.

SA’s rules include many tests that are not done by TB (score one point for SA). Yet when they take all of their rules and combine them together, in most cases they are using broad measures of goodness and spaminess that apply to some concept of a universal email. In contrast, TB’s bayes algorithm will precisely tune the junk analysis to the particular set of emails that a user gets. So if you are in the Viagra business, you can tune your local TB filter to accept those. (score one point for TB).

Wouldn’t it be great if you could combine together the superior rule set of SA, with the superior decision making customization of TB? That is the point of the hidden preference.

SA communicates its message tags to TB in the form of a custom header, X-SPAM-STATUS. A sample value of that from a spam message that I got is:

X-Spam-Status: No, score=4.9 required=5.0 tests=HTML_IMAGE_ONLY_16,
HTML_IMAGE_RATIO_02,HTML_MESSAGE,HTML_SHORT_LINK_IMG_2,
MIME_HTML_ONLY, SPF_PASS shortcircuit=no autolearn=disabled version=3.2.5

This message was not marked as spam by SA, as it just barely missed the score=5.0 cutoff point, so arrives in TB as a suspicious but nevertheless good message. Yet there is lots of information about the tests that SA did in that header. TB should be able to take advantage of that information.

In the default TB configuration, the entire content of the X-Spam-Status header is treated as a single token. So TB will only respond to exact matches to that header. Yet those tests can appear in a variety of combinations, so that is not really taking advantage of the information effectively.

The answer provided in TB3 / SM2 is to provide a hidden preference that allows you to tell TB to break that particular header up into lots of separate tokens, and analyze each of those separately. That is documented in the code here. In the preference, you specify a particular header, and give a list of delimiters for that tokenization. So to process the x-spam-status header as individual tokens including both space and comma as delimiters, you add the preference:

mailnews.bayesian_spam_filter.tokenizeheader.x-spam-status

and assign it the string value ” ,\t\n\r\f”

(That can be hard to set with all of those special characters in it). That includes space, comma, tab, line feed, carriage return, and form feed as delimiters.

You really can’t do this to an existing training corpus without getting strange results. So you must reset the training data and retrain after you do that.

As a sample, the tokens used by TB to process an email with my example x-spam-status is:

This image uses the “Junk Analysis Detail” report from my JunQuilla extension.

Things to notice:

1. The tokens that TB is using to analyze this email are mostly these x-spam-status tokens, as this is a mostly-image spam message with little text for TB to analyze. So the use of the SA tokens greatly enhances the power of TB’s internal filter.
2. The final score for this, 84, ended up with the message marked as junk for me. I run a cutoff of 75, while the default cutoff is 90.
3. Many of the x-spam-status tokens are actually working against us here. For example, x-spam-status:required=5.0 has a score of 23, which moves the message toward the “good” status. Yet that token appears on every single email I receive, so should be neutral. What is going on here?

   This is the issue that I reported in a previous post “bad effects on junk training corpus from change” That is, recently I have been automatically training more messages as Good than Junk. “x-spam-status:required=5.0″ must not exist in my older emails for some reason, so this bias is appearing because 1) the token changed at some point in time, and 2) there is a mismatch in the number of junk and good emails being trained recently. I really need to figure out a solution to this. What I will probably do is to add features to FiltaQuilla to allow me to precisely match the training of junk and good to remove this bias.

I still need to do some tests of this to directly compare the performance of the spam filters with and without the tokenization to be sure this is really a good idea. That is quite tricky to do unfortunately.

In spite of these issues, I would recommend to new installations that you use this alternate tokenization when you are using a Spam Assassin front-end to Thunderbird or SeaMonkey.

Frankly, I’ve struggled to find a good personal use of TaQuilla for use in my dogfooding. I’ve tried using it to categorize “interesting” posts, but I can’t even agree myself from day-to-day what is “interesting”, and the soft tagging is even more indecisive. But I’ve finally hit on a good use for it in my workflow – rejecting of sports articles in newsfeeds!

I have an RSS feed that subscribes to local news for the “Seattle Times” newspaper, but I find a lot of the articles are sports related. Now I am a certified geek, and not really interested in those types of articles. So what I did is to create a tag “Sports”, then I setup TaQuilla soft tags for “Sports” on the RSS feed, create a virtual folder that filters out articles tagged with “Sports”, and voila I can read the newspaper feed without all of those annoying sports articles. It’s particularly useful since many of the same articles get updated multiple times, and the updates are very efficiently rejected with the bayes filter if I tag the original.

In addition to some new filter actions (print, add sender to address list, and save attachments to a folder) this release introduces “custom search terms” for the first time. This is a new feature that has been added recently to the mailnews core code, and is part of the TB 3.0 and SM 2.0 releases.

The search I am talking about is the old-style Thunderbird search, not the newer global database (gloda) search that was added to Thunderbird. Gloda gets all of the press, but we’ve also taught the old style search a few tricks as well! These are particularly useful in saved searchs (also called virtual folders).

I think that the most interesting new capability is that you can define a search (and therefore a virtual folder) by adding a few lines of javascript code that does precisely what you want. Let me give a very simple example.

Let’s assume that you want a virtual  folder to contain all of the active items that you currently need to process involving projects. Incoming emails are marked with tags, either manually or by some sort of filter. You define new tags for each project, and each tag begins with “pro” and ends with some sort of project marker, say a number or word. You want to have your active folder contain messages that have a tag containing “pro”, but NOT include messages that are tagged “done”.

The standard mailnews tag search forces you to enter the tag name for each tag that you want. There is no way to search for tags by the characters in the tags. Plus, searching for several tags is an OR function, and saying to not include DONE messages is an AND function. But the standard mailnews search does not handle complex boolean searches.

The javascript custom search comes to the rescue! Go to the folder that contains the messages that you want, and select the Javascript search term:

Click on the script icon and you will get a small editor window, where you can enter a few lines of javascript. All that we need to do is grab the string that has the tags in it, and make sure it includes “pro” but does not include “done”. Our javacript is given a variable “message” which is the database header object that can be used to get message properties. We need to execute an expression as the last statement, whose value is either true if we want the message, or false if we do not. That’s just the following two lines of javascript:

let tags = message.getStringProperty('keywords');
(/pro/.test(tags) && !(/done/.test(tags));

So here’s what our javascript window and code looks like:

Now save this as a virtual folder, and you have the exact virtual folder that you want!

OK, this is just for geeks, but it is really powerful in letting you define folders that can precisely define the workflow that you want. There’s also a few other geek-friendly search terms, including the much-requested regular expression search by subject or other header. For details, see the updated FiltaQuilla page on this site.

FiltaQuilla is still in experimental status, though I have now nominated it to be public. Still it may be a few weeks before it gets there. If you are an existing FiltaQuilla user, you will need to go to the download page directly and download and install the new version.

Enjoy!

Join the forum discussion on this post - (1) Posts

I don’t have any illusions that this has a significant chance of being implemented, and I’m not even sure it’s a good idea myself. But I ask you to suspend disbelief for a minute, and imagine a change to the development culture and process.

An email client is different from a web client in many ways, but one significant way is that there is no real need for a fat uniform core product that developers can target (such as web developers for FireFox). So we are free to allow wide changes in our product configuration that would not make sense for FireFox. There is really no fundamental need for Thunderbird to be presented as a single, fat, feature-laden client.

Instead, ship Thunderbird as a minimal base with a collection of extensions. The extensions could be in a variety of statuses. At one status extreme, “Core” extensions would be enabled by default, would be fully localized, and their updates would be shipped with updates to the core product, rather than through AMO. Many existing core features would be converted into “Core” extensions that could be disabled if desired (for example bayes junk processing, or gloda.) At the other extreme, “Pilot” extensions would be shipped with the core product in a disabled state, would be updated by AMO, and not fully localized. There would also be “Standard” extensions that are shipped with the product, not maintained through AMO, but would not be enabled by default. Lightning might be one example of this. FiltaQuilla or JunQuilla could easily get added to this category in the future, or popular extensions like ThunderBrowse.

So why would you do such a crazy thing? For several reasons.

First, you would have a path to add features to the program that is not as generally disruptive as has been, for example, gloda or the new message header. By not using a new extension, an existing user would not see changes to their workflow that they did not want or appreciate. Also, new features need not delay the release of new versions of Thunderbird, as “Pilot” status extensions could be updated through AMO.

Second, new complex features like gloda, even though they are developed by the core team (well mostly asuth) are in a state of rapid flux, and would really benefit from allowing updates more frequently than even the accelerated release process will allow.

Third, you provide a natural path for outside developers to add contribution to the product without having to completely submerge themselves in the Mozilla culture, or give up complete control of their creation.

Fourth, this really recognizes that the use of an email client is highly personal. Basic users could be presented a basic email client. Advanced users could easily add advanced features. (Existing AMO-based extensions are also good for this, but the quality is not uniform, and they frequently are not kept up to date. And the standard product is still very fat with lots of features that are unneeded by most users.)

Fifth, this would solve the serious issue with Thunderbird of how hard it is for the average user to install addons (because the most popular and important addons would be shipped with the product).

There’s another dimension to this, and that is the developer’s relationship with his or her extension. I know that I feel a responsibility for my extensions that is beyond the responsibility I feel for any core code. You can see that in the documentation that I provide, and my reliability in responding to issues. I think that many other extension developers are like that as well. I’m guessing that they would be delighted to see a higher level of promotion of their work, without the need to cede complete control that incorporation in core might involve.

I suppose I could write a book on what this might look like, but for now let me leave it here.

rkent

So far, I have added the following new search terms:

BCC – locate items in the BCC field

Subject Regex – search the subject using a javascript regular expression

Header Regex – search any specific header using a javascript regular expression

Javascript – load javascript in a text field, and program your own search given an nsIMsgDBHdr object

Tag of Thread Head – match a tag in the head of a message’s thread

Tag of Thread Messages – match a tag near the message in its thread (within +/- 10 messages by default)

Address in Thread – match an address near the message in its thread (within +/- 10 messages by default)

This stuff can be useful outside of filters, in fact I am mostly using them personally to define virtual folders. So I’ll probably move them to a new extension, and try to get FiltaQuilla out the door finally.

rkent

JunQuilla is my attempt to extend the user interface in the Mozilla mailnews product to provide the information that I believe is needed to properly manage the bayesian junk filter. I suppose that most of these features should really be in the core product, but I found that support for that was not very strong, so I decided to do most of this in an extension instead. These backend features have only been added to the core code in the last couple of years, so this extension will only work on newer versions of the Mozilla email clients (Thunderbird 3.* versions, and SeaMonkey 2.* versions.)

Version 1.0.0 fixes some bugs that have been reported in previous releases, provides partial support for SeaMonkey (except for the “Uncertain” folders), and adds a number of new features:

Junk Options

You can set critical overall junk options in the standard junk options screen (previously, this was only possible in the more obscure addons/options area). In Thunderbird, select Tools/Options/Security/Junk. In SeaMonkey, select Edit/Preferences/Mail & Newsgoups/JunQuilla. There you will see a display like this:

“Junk threshold” is the percentage value as calculated by the bayes classifier for each message, above which a message will be classified as junk. This should be set as low as possible, though always high enough to avoid having any real messages classified as junk. The default value of 90 is much too high for a well-trained bayes classifier.

The “Maximum token count” is a measure of the resources that the junk classifier will use. The higher it is set, the more accurate your classifications will be. The default value of 100,000 is probably too low for good classification performance. I’ve had good results with 300,000 – and JunQuilla will set your value to this when first installed. If this value is too high, and you have trained a lot of messages, then memory usage may be excessive.

The other parameters are read only, and are displays of values from your training file. The “Current token count” shows how many junk training tokens (which are like words) are currently in use. You probably won’t get good performance until this number is over 10,000 – and it really should be more like 100,000. “Good” and “Junk” messages trained shows how many messages have been used to train the junk filter. Ideally the number of junk and good messages should be more or less equal. If they are not, then pick some previously untrained messages and train them.

When the number of tokens exceeds the maximum value, then Mozilla mailnews will prune the training file in a large chunk, typically reducing both the number of trained messages, and the number of tokens, in about half.

Disable/enable junk processing for a folder.

You can set an “inherited folder property” to allow you to selectively enable or disable junk processing for folders. This has two main uses.

• If you have server-side filters that process email in IMAP, then you may already know that certain folders contain either junk mail or good mail, and don’t want to waste time processing them locally – or take the risk that they will be processed incorrectly.

• Mozilla mailnews core code now supports junk processing of RSS and News folders. You can select certain RSS or News folders, and then junk processing will run on new posts to those folders. This will also enable the standard user interface features that allow you to train messages as good or junk in those folders.

To set this, right click on a folder in the folder tree, and select Properties, then the “General Information” tab. At the bottom, you will see this:

“Analyze Junk” is an inherited folder property. What that means is that each folder can either gets its value from its parent, or can be set locally. The default value depends on the characteristics of the folder itself. So for example, this would be disabled in News by default, but enabled in IMAP. To change the value, first reset the “Inherit” checkbox, then set the value that you want in “Enabled”. If you change a value for a folder, then the value will also change for the children of that folder (assumming that they have the default “Inherit” checked.)

Toolbar “Is Junk” and “Is Good” button

You can add two new buttons to your toolbar – “Is Junk” and “Is Good”. Here’s what they look like, next to a standard “Junk/Not Junk” icon:

To add these buttons, right click on a toolbar, select “Customize”, then drag the buttons to the desired location.

The standard Junk button will always show as “Junk” when it thinks a message is good, and “Good” when it thinks that a message is junk. But that means that we can only classify a message as “Good” when it has been falsely classified as junk, and we never want our junk filter to do that. The “Is Good” button is meant to be used in the “Uncertain” folders to give you a means to train a message as “Good” there.

Join the forum discussion on this post - (3) Posts]]> http://mesquilla.com/2009/11/16/junquilla-version-1-0-0-released/feed/ 0 http://mesquilla.com/2009/11/12/glodaquilla-version-0-3-0-released/ http://mesquilla.com/2009/11/12/glodaquilla-version-0-3-0-released/#comments Thu, 12 Nov 2009 23:46:02 +0000 rkent http://mesquilla.com/?p=425 This release is intended to update GlodaQuilla to work with Thunderbird version 3.0 (including release candidate RC1). You can download the new version of GlodaQuilla here. (GlodaQuilla’s main function is to provide access to information about the indexing status of each Thunderbird message in the “gloda” global database.)

In addition to the gloda columns, this release adds an experimental feature to allow overriding of the default gloda configuration concerning which messages are indexed. This new inherited folder property is called “Index in Global Database”.  Access to inherited properties such as this are discussed in the post “Inherited Folder Properities (revisited)”

You can use this property to disable indexing of servers or folders that are normally allowed, or to enable indexing on folders that are normally disabled (such as trash or junk folders). In my system, for example, for testing purposes I download my email both as IMAP and as POP3, but I disable the indexing of IMAP to prevent double indexing of the same messages.

This new property does not make any attempt to fix existing indexing when it is changed. Minimal support for this hopefully will be added for the 1.0.0 release of GlodaQuilla.

Join the forum discussion on this post - (1) Posts

In this release, I’ve fixed some bugs, plus added support for some new sound formats. If your operating system will launch a .mp3 file in a local player, you can now ask it to do that as part of the filter action. I also support .ogg files using Mozilla’s standard Ogg Vorbis player, though my experience has been that this not reliable enough yet for real usage, at least on Windows XP.

You can download this from AMO, see a detailed description of ToneQuilla here, or see details of changes in this revision from the releases page.

I’ve also now submitted this for official review by the folks at AMO, so hopefully after a month this will be able to leave the experimental sandbox, and things like automatic updates should also work.

Make sure that you make a filter to play “We Are in Love” for the emails that you receive from your significant other!

rkent

(All references to extensions in this posting refer to the 1.0.0 versions, which as of this writing have not been posted to AMO yet. But they should be available in a few weeks.)

Implemented UI

Briefly, inherited properties are a property that can be defined globally, at the server, or at the folder, and its characteristics will be propagated to child objects. This make it easy to specify precisely how the property is applied.

As an example, I have recently implemented a feature “Index in Global Database” in GlodaQuilla which can be used to selectively suppress certain accounts or folders from being accessed by the global database indexer. In the account manager, where indexing can be disabled for an entire account, the UI looks like this:

Each inherited property has default values which are typically set by the base code. In the case of the gloda database indexer, everything but newgroups are indexed by default. Initially each inherited property is set to just use the standard default processing, but if I clear the “default” checkbox, then I can turn off gloda indexing for this account.

If I do that, then go to a first-level folder in the account, I see the following under folder properties:

At the folder level, because I disabled global indexing on the account, it is now shown as disabled on the folder. I could clear the inherit box and selectively enable it on just this folder and its children if I wanted.

This particular UI merges naturally with the existing methods of setting properties in mailnews, but I’m not sure it is optimum for an inherited property. The inherited nature could be more clearly shown, and a particular feature more quickly configured, if I showed a tree of accounts and folders, with checkboxes next to each account to enable or inherit the feature. Maybe in a future version.

Implemented properties

Here are some of the implementations of inherited properties that exist in my extensions:

1. (GlodaQuilla) Index in Global Database – suppresses the running of the global database indexer
2. (FiltaQuilla) Apply Filters to Folder – for Imap folders, allow incoming filters to run on that folder
3. (JunQuilla) Analyze Junk – allow junk processing to be turned on or off. This also allows junk processing to run on RSS or news folders.
4. (TaQuilla) Analyze particular automatic tags.

Issue: Existing mechanisms

Ideally, the inherited property would be the one and only way to manage a program feature. But for existing features, the existing mechanisms remain, which can lead to possible confusion. For example, with JunQuilla’s “Analyze Junk” property, there is existing UI to enable junk processing at the account level. Here the inherited property will always override the default mechanism (but that is mostly because I implemented it in core that way, and I have a little influence on how junk processing is handled in core.) For GlodaQuilla’s “Index in Global Database” the behaviour is different. Existing UI will only allow this to be enabled or disabled globally, and the inherited property does not override this. The inherited property code uses the default server preference as a global enable/disable for a property, so if gloda used that same mechanism instead of an independent preference, this issue would go away. I guess I could say the same thing about junk processing as well.

For FiltaQuilla’s “Apply Filters to Folder”, there is a subtle issue in the inherited nature. I did not implement in core the ability of the inherited property to override the existing default as applied to the Inbox, so incoming filters always run on the inbox. That creates 2 ui issues. First, although I show at the account level the “Apply Filters” option, it does not actually suppress application to the inbox as one would expect. Second, I currently do not show a folder property for “Apply Filters to Folder” for the IMAP inbox since it would not make sense there, so that also means there is no way to enable processing of filters for the children of the inbox. Maybe I should call this feature instead “Apply Filters to non-Inbox Folders” to solve this, or change the core code so that the feature also applied to the Inbox.

Issue: My RDF-inspired property for junk management

Looking ahead to a world where a number of extensions might try to define bayes filter traits, in code I recommended that properties used to manage junk processing use an RDF-inspired globally unique identifier. Then I followed my own advice and defined the identifier that controls junk processing on a folder as: “dobayes.mailnews@mozilla.org#junk” Unfortunately, the existing account manager code does not allow periods in property names, which means I could not use the account manager to manage this. I’ve filed bug 525024 on this issue, and perhaps that can be incorporated after TB 3.0 / SM 2.0.

Issue: Missing inheritance levels

I’ve heard others comment that often they want to set a property on a particular class of folders, say on all Trash folders, or all Sent folders. I’ve considered implementing another level in the inherited properties feature, that would be a folder type. So you would then set a property that would be inherited by any folder of a particular type, and its children – and of course also overridden by the local folder property.

Issue: UI for global property

All of the inherited properties could also be enabled globally using the “mail.server.default.<property>” preference, but I did not give any UI for that in my extensions. I thought that would be too confusing for the user to show those preferences, which would be very similar to existing mechanisms. This is not an issue for properties that use the preference system for server-level issues, but none of the existing server-level preferences are also inherited properties. Perhaps we could move that direction in the future.