You’ll need to install my addon JunQuilla to enable one critical piece of user interface. JunQuilla supports a folder property that lets you selectively enable or disable junk processing for a tree of folders. So after you’ve installed JunQuilla, enable processing of junk for a newsgroup:

This will run future posts sent to the newsgroup m.d.a.thunderbird through the bayes junk filter in Thunderbird. After this is enabled, some of the junk management controls on the folder should be enabled. Try “Run Junk Controls on Folder” to process existing posts for junk.

But nothing will appear to happen when you do, because there is no functionality to delete or hide the junk messages for newsgroups built in. Still, you can see the results using JunQuilla’s “junk percent” and “junk status +” columns:

The next step is to remove those junk messages from your view. The easiest thing to do is to create a virtual folder that shows the messages in the newsgroup without the junk messages. This is where the user interface fights against you. But you can trick it in the following manner.

Create a virtual folder on a message folder with search criteria “Junk Percent < 80″. Save the folder as a subfolder of a local mail folder. Now open the virtual folder and change the folders that it scans, removing the original folder and replacing it with the newsgroup folder:

Now the spam posts are hidden from this virtual folder:

You may need to specifically train the junk filter using a few junk and good messages in news. But the good news is that newsgroup spammers are not really optimizing against bayes filters, so it seems to be a lot easier to detect newsgroup spam than mail spam.

Something I’ve found myself doing at work is creating a new filter for every folder I create. I work on technical cases and for each new case number I create a new folder and have all emails with that case number go into that folder.  The crappy part about it is that I literally have hundreds of cases I deal with, and hence hundreds of filters.  You’ve already got the regex match criteria in filtaqulla, I’d love to be able to take that match criteria and use whatever string it matches as the destination folder action criteria. Sample subject line contains the following: 2010-0609-518

One of the features added to my extension FiltaQuilla recently is the ability to add custom javascript actions. I thought that I would give a shot at doing this request as a custom javascript action. The author of the email was expecting this to get linked to the regex search term, but that is not the easiest way to do it. It is easier to just let the filter action also do the regex search.

After installing FiltaQuilla, you need to enable the custom javascript action, which is done on the Addon options page. After that, you can add a javascript custom action in the filter editor.

Then you need to write the javascript code for the action. I’ve done similar work in the past, so that took me about an hour to get correct. Then insert the code into the action field for the javascript action, and it’s ready to go! You have to have some sort of search in the filter as well. You could just search for everything if you wanted, or you could add a regex search for the precise term if you want. It doesn’t matter much, as all of the work is really being done in the action. You just need to make sure that the action gets called for each message that might need to be moved. There’s more documentation of this feature available at the FiltaQuilla page on this site.

The code I came up with is:

let digitsRegex = /20\d\d\-[0-1]\d[0-3]\d\-\d\d\d/;
let acctmgr = Cc["@mozilla.org/messenger/account-manager;1"]
                .getService(Ci.nsIMsgAccountManager);
let copyService = Cc["@mozilla.org/messenger/messagecopyservice;1"]
                    .getService(Ci.nsIMsgCopyService);
for (let index = 0; index < msgHdrs.length; index++)
{
  let hdr = msgHdrs.queryElementAt(index, Ci.nsIMsgDBHdr);
  let theDigits = digitsRegex.exec(hdr.subject);
  let folders = acctmgr.allFolders;
  let length = folders.length;
  for (let i = 0; i < length; i++) {
    let folder = folders.queryElementAt(i, Ci.nsIMsgFolder);
    if (folder.name == theDigits) {
      let messages = Cc["@mozilla.org/array;1"]
                       .createInstance(Ci.nsIMutableArray);
      messages.appendElement(hdr, false);
      copyService.CopyMessages(hdr.folder, messages, folder, true,
                               null, msgWindow, false);
    }
  }
}

With this filter, if you create a folder with a name like “2010-0616-001″, and your message has that in the subject, the message will get moved to that folder.

If you are having trouble getting it to work, an easy way to debug is just to insert statements like this into the javascript:

Cu.reportError("I am here");

Those printouts will show up on the error console, so you should be able to see if your filter action code is working or not.

Mitra recently pinged bug 249191 asking for a filter action that affects remote content. That looked like a good application of a javascript filter action, so I tried it out.

Looking at the underlying code in nsMsgContentPolicy.cpp, there is a message header property “remoteContentPolicy” that can be used to enable showing of remote content for a message. Unfortunately disallowing content appears more complex, as the address book whitelisting appears to override the message header value. But really I think that allowing remote content on a per message basis is probably the more common action need, so let me demo that.

In my documentation for FiltaQuilla, I show a simple basic javascript example of a custom action that affects a message header subject property:

for (let index = 0; index < msgHdrs.length; index++)
{
  let hdr = msgHdrs.queryElementAt(index, Ci.nsIMsgDBHdr);
  hdr.subject = "[Hello, world] " + hdr.subject;
}

It’s fairly easy to modify this to set the remoteContentPolicy instead. The result is:

for (let index = 0; index < msgHdrs.length; index++)
{
 let hdr = msgHdrs.queryElementAt(index, Ci.nsIMsgDBHdr);
 hdr.setUint32Property("remoteContentPolicy", 2);
}

So if you enter this as a JavaScript custom filter in JunQuilla, you can enable remote content on messages controlled by a filter.

I tested it on a spam message that had “Exxon” in the subject. The resulting filter, including the javascript, looks like this:

I’m sure that defining these filters is beyond the skill of all but a few advanced users, but once the code is written it’s pretty much just cut and paste to get this to work. If I collect a few more examples, I suppose I should create a webpage to show them.

Thanks to jcranmer, Standard8, and bienvenu on IRC channel #maildev for helping to point me to the base code that manages remote content policy.

In the Thunderbird 3.0 / SeaMonkey 2.0 series, I snuck in a little hidden preference to allow modifications to the way that the TB bayes filter does it tokenization. The main point of this was to allow better transfer of information between SA and TB. I’d like to describe here how to use that feature.

SA’s decision process involves two key steps. First, they evaluate the message with zillions of rules, and tag the message with each rule that the message hits. Second, they have a method of combining all of those tags into a final junk score for the message, which is used to decide whether to tag the overall message as spam.

SA’s rules include many tests that are not done by TB (score one point for SA). Yet when they take all of their rules and combine them together, in most cases they are using broad measures of goodness and spaminess that apply to some concept of a universal email. In contrast, TB’s bayes algorithm will precisely tune the junk analysis to the particular set of emails that a user gets. So if you are in the Viagra business, you can tune your local TB filter to accept those. (score one point for TB).

Wouldn’t it be great if you could combine together the superior rule set of SA, with the superior decision making customization of TB? That is the point of the hidden preference.

SA communicates its message tags to TB in the form of a custom header, X-SPAM-STATUS. A sample value of that from a spam message that I got is:

X-Spam-Status: No, score=4.9 required=5.0 tests=HTML_IMAGE_ONLY_16,
HTML_IMAGE_RATIO_02,HTML_MESSAGE,HTML_SHORT_LINK_IMG_2,
MIME_HTML_ONLY, SPF_PASS shortcircuit=no autolearn=disabled version=3.2.5

This message was not marked as spam by SA, as it just barely missed the score=5.0 cutoff point, so arrives in TB as a suspicious but nevertheless good message. Yet there is lots of information about the tests that SA did in that header. TB should be able to take advantage of that information.

In the default TB configuration, the entire content of the X-Spam-Status header is treated as a single token. So TB will only respond to exact matches to that header. Yet those tests can appear in a variety of combinations, so that is not really taking advantage of the information effectively.

The answer provided in TB3 / SM2 is to provide a hidden preference that allows you to tell TB to break that particular header up into lots of separate tokens, and analyze each of those separately. That is documented in the code here. In the preference, you specify a particular header, and give a list of delimiters for that tokenization. So to process the x-spam-status header as individual tokens including both space and comma as delimiters, you add the preference:

mailnews.bayesian_spam_filter.tokenizeheader.x-spam-status

and assign it the string value ” ,\t\n\r\f”

(That can be hard to set with all of those special characters in it). That includes space, comma, tab, line feed, carriage return, and form feed as delimiters.

You really can’t do this to an existing training corpus without getting strange results. So you must reset the training data and retrain after you do that.

As a sample, the tokens used by TB to process an email with my example x-spam-status is:

This image uses the “Junk Analysis Detail” report from my JunQuilla extension.

Things to notice:

1. The tokens that TB is using to analyze this email are mostly these x-spam-status tokens, as this is a mostly-image spam message with little text for TB to analyze. So the use of the SA tokens greatly enhances the power of TB’s internal filter.
2. The final score for this, 84, ended up with the message marked as junk for me. I run a cutoff of 75, while the default cutoff is 90.
3. Many of the x-spam-status tokens are actually working against us here. For example, x-spam-status:required=5.0 has a score of 23, which moves the message toward the “good” status. Yet that token appears on every single email I receive, so should be neutral. What is going on here?

   This is the issue that I reported in a previous post “bad effects on junk training corpus from change” That is, recently I have been automatically training more messages as Good than Junk. “x-spam-status:required=5.0″ must not exist in my older emails for some reason, so this bias is appearing because 1) the token changed at some point in time, and 2) there is a mismatch in the number of junk and good emails being trained recently. I really need to figure out a solution to this. What I will probably do is to add features to FiltaQuilla to allow me to precisely match the training of junk and good to remove this bias.

I still need to do some tests of this to directly compare the performance of the spam filters with and without the tokenization to be sure this is really a good idea. That is quite tricky to do unfortunately.

In spite of these issues, I would recommend to new installations that you use this alternate tokenization when you are using a Spam Assassin front-end to Thunderbird or SeaMonkey.

Frankly, I’ve struggled to find a good personal use of TaQuilla for use in my dogfooding. I’ve tried using it to categorize “interesting” posts, but I can’t even agree myself from day-to-day what is “interesting”, and the soft tagging is even more indecisive. But I’ve finally hit on a good use for it in my workflow – rejecting of sports articles in newsfeeds!

I have an RSS feed that subscribes to local news for the “Seattle Times” newspaper, but I find a lot of the articles are sports related. Now I am a certified geek, and not really interested in those types of articles. So what I did is to create a tag “Sports”, then I setup TaQuilla soft tags for “Sports” on the RSS feed, create a virtual folder that filters out articles tagged with “Sports”, and voila I can read the newspaper feed without all of those annoying sports articles. It’s particularly useful since many of the same articles get updated multiple times, and the updates are very efficiently rejected with the bayes filter if I tag the original.

In addition to some new filter actions (print, add sender to address list, and save attachments to a folder) this release introduces “custom search terms” for the first time. This is a new feature that has been added recently to the mailnews core code, and is part of the TB 3.0 and SM 2.0 releases.

The search I am talking about is the old-style Thunderbird search, not the newer global database (gloda) search that was added to Thunderbird. Gloda gets all of the press, but we’ve also taught the old style search a few tricks as well! These are particularly useful in saved searchs (also called virtual folders).

I think that the most interesting new capability is that you can define a search (and therefore a virtual folder) by adding a few lines of javascript code that does precisely what you want. Let me give a very simple example.

Let’s assume that you want a virtual  folder to contain all of the active items that you currently need to process involving projects. Incoming emails are marked with tags, either manually or by some sort of filter. You define new tags for each project, and each tag begins with “pro” and ends with some sort of project marker, say a number or word. You want to have your active folder contain messages that have a tag containing “pro”, but NOT include messages that are tagged “done”.

The standard mailnews tag search forces you to enter the tag name for each tag that you want. There is no way to search for tags by the characters in the tags. Plus, searching for several tags is an OR function, and saying to not include DONE messages is an AND function. But the standard mailnews search does not handle complex boolean searches.

The javascript custom search comes to the rescue! Go to the folder that contains the messages that you want, and select the Javascript search term:

Click on the script icon and you will get a small editor window, where you can enter a few lines of javascript. All that we need to do is grab the string that has the tags in it, and make sure it includes “pro” but does not include “done”. Our javacript is given a variable “message” which is the database header object that can be used to get message properties. We need to execute an expression as the last statement, whose value is either true if we want the message, or false if we do not. That’s just the following two lines of javascript:

let tags = message.getStringProperty('keywords');
(/pro/.test(tags) && !(/done/.test(tags));

So here’s what our javascript window and code looks like:

Now save this as a virtual folder, and you have the exact virtual folder that you want!

OK, this is just for geeks, but it is really powerful in letting you define folders that can precisely define the workflow that you want. There’s also a few other geek-friendly search terms, including the much-requested regular expression search by subject or other header. For details, see the updated FiltaQuilla page on this site.

FiltaQuilla is still in experimental status, though I have now nominated it to be public. Still it may be a few weeks before it gets there. If you are an existing FiltaQuilla user, you will need to go to the download page directly and download and install the new version.

Enjoy!

I don’t have any illusions that this has a significant chance of being implemented, and I’m not even sure it’s a good idea myself. But I ask you to suspend disbelief for a minute, and imagine a change to the development culture and process.

An email client is different from a web client in many ways, but one significant way is that there is no real need for a fat uniform core product that developers can target (such as web developers for FireFox). So we are free to allow wide changes in our product configuration that would not make sense for FireFox. There is really no fundamental need for Thunderbird to be presented as a single, fat, feature-laden client.

Instead, ship Thunderbird as a minimal base with a collection of extensions. The extensions could be in a variety of statuses. At one status extreme, “Core” extensions would be enabled by default, would be fully localized, and their updates would be shipped with updates to the core product, rather than through AMO. Many existing core features would be converted into “Core” extensions that could be disabled if desired (for example bayes junk processing, or gloda.) At the other extreme, “Pilot” extensions would be shipped with the core product in a disabled state, would be updated by AMO, and not fully localized. There would also be “Standard” extensions that are shipped with the product, not maintained through AMO, but would not be enabled by default. Lightning might be one example of this. FiltaQuilla or JunQuilla could easily get added to this category in the future, or popular extensions like ThunderBrowse.

So why would you do such a crazy thing? For several reasons.

First, you would have a path to add features to the program that is not as generally disruptive as has been, for example, gloda or the new message header. By not using a new extension, an existing user would not see changes to their workflow that they did not want or appreciate. Also, new features need not delay the release of new versions of Thunderbird, as “Pilot” status extensions could be updated through AMO.

Second, new complex features like gloda, even though they are developed by the core team (well mostly asuth) are in a state of rapid flux, and would really benefit from allowing updates more frequently than even the accelerated release process will allow.

Third, you provide a natural path for outside developers to add contribution to the product without having to completely submerge themselves in the Mozilla culture, or give up complete control of their creation.

Fourth, this really recognizes that the use of an email client is highly personal. Basic users could be presented a basic email client. Advanced users could easily add advanced features. (Existing AMO-based extensions are also good for this, but the quality is not uniform, and they frequently are not kept up to date. And the standard product is still very fat with lots of features that are unneeded by most users.)

Fifth, this would solve the serious issue with Thunderbird of how hard it is for the average user to install addons (because the most popular and important addons would be shipped with the product).

There’s another dimension to this, and that is the developer’s relationship with his or her extension. I know that I feel a responsibility for my extensions that is beyond the responsibility I feel for any core code. You can see that in the documentation that I provide, and my reliability in responding to issues. I think that many other extension developers are like that as well. I’m guessing that they would be delighted to see a higher level of promotion of their work, without the need to cede complete control that incorporation in core might involve.

I suppose I could write a book on what this might look like, but for now let me leave it here.

rkent

So far, I have added the following new search terms:

BCC – locate items in the BCC field

Subject Regex – search the subject using a javascript regular expression

Header Regex – search any specific header using a javascript regular expression

Javascript – load javascript in a text field, and program your own search given an nsIMsgDBHdr object

Tag of Thread Head – match a tag in the head of a message’s thread

Tag of Thread Messages – match a tag near the message in its thread (within +/- 10 messages by default)

Address in Thread – match an address near the message in its thread (within +/- 10 messages by default)

This stuff can be useful outside of filters, in fact I am mostly using them personally to define virtual folders. So I’ll probably move them to a new extension, and try to get FiltaQuilla out the door finally.

rkent

JunQuilla is my attempt to extend the user interface in the Mozilla mailnews product to provide the information that I believe is needed to properly manage the bayesian junk filter. I suppose that most of these features should really be in the core product, but I found that support for that was not very strong, so I decided to do most of this in an extension instead. These backend features have only been added to the core code in the last couple of years, so this extension will only work on newer versions of the Mozilla email clients (Thunderbird 3.* versions, and SeaMonkey 2.* versions.)

Version 1.0.0 fixes some bugs that have been reported in previous releases, provides partial support for SeaMonkey (except for the “Uncertain” folders), and adds a number of new features:

Junk Options

You can set critical overall junk options in the standard junk options screen (previously, this was only possible in the more obscure addons/options area). In Thunderbird, select Tools/Options/Security/Junk. In SeaMonkey, select Edit/Preferences/Mail & Newsgoups/JunQuilla. There you will see a display like this:

“Junk threshold” is the percentage value as calculated by the bayes classifier for each message, above which a message will be classified as junk. This should be set as low as possible, though always high enough to avoid having any real messages classified as junk. The default value of 90 is much too high for a well-trained bayes classifier.

The “Maximum token count” is a measure of the resources that the junk classifier will use. The higher it is set, the more accurate your classifications will be. The default value of 100,000 is probably too low for good classification performance. I’ve had good results with 300,000 – and JunQuilla will set your value to this when first installed. If this value is too high, and you have trained a lot of messages, then memory usage may be excessive.

The other parameters are read only, and are displays of values from your training file. The “Current token count” shows how many junk training tokens (which are like words) are currently in use. You probably won’t get good performance until this number is over 10,000 – and it really should be more like 100,000. “Good” and “Junk” messages trained shows how many messages have been used to train the junk filter. Ideally the number of junk and good messages should be more or less equal. If they are not, then pick some previously untrained messages and train them.

When the number of tokens exceeds the maximum value, then Mozilla mailnews will prune the training file in a large chunk, typically reducing both the number of trained messages, and the number of tokens, in about half.

Disable/enable junk processing for a folder.

You can set an “inherited folder property” to allow you to selectively enable or disable junk processing for folders. This has two main uses.

• If you have server-side filters that process email in IMAP, then you may already know that certain folders contain either junk mail or good mail, and don’t want to waste time processing them locally – or take the risk that they will be processed incorrectly.

• Mozilla mailnews core code now supports junk processing of RSS and News folders. You can select certain RSS or News folders, and then junk processing will run on new posts to those folders. This will also enable the standard user interface features that allow you to train messages as good or junk in those folders.

To set this, right click on a folder in the folder tree, and select Properties, then the “General Information” tab. At the bottom, you will see this:

“Analyze Junk” is an inherited folder property. What that means is that each folder can either gets its value from its parent, or can be set locally. The default value depends on the characteristics of the folder itself. So for example, this would be disabled in News by default, but enabled in IMAP. To change the value, first reset the “Inherit” checkbox, then set the value that you want in “Enabled”. If you change a value for a folder, then the value will also change for the children of that folder (assumming that they have the default “Inherit” checked.)

Toolbar “Is Junk” and “Is Good” button

You can add two new buttons to your toolbar – “Is Junk” and “Is Good”. Here’s what they look like, next to a standard “Junk/Not Junk” icon:

To add these buttons, right click on a toolbar, select “Customize”, then drag the buttons to the desired location.

The standard Junk button will always show as “Junk” when it thinks a message is good, and “Good” when it thinks that a message is junk. But that means that we can only classify a message as “Good” when it has been falsely classified as junk, and we never want our junk filter to do that. The “Is Good” button is meant to be used in the “Uncertain” folders to give you a means to train a message as “Good” there.

Join the forum discussion on this post