You’ll need to install my addon JunQuilla to enable one critical piece of user interface. JunQuilla supports a folder property that lets you selectively enable or disable junk processing for a tree of folders. So after you’ve installed JunQuilla, enable processing of junk for a newsgroup:

This will run future posts sent to the newsgroup m.d.a.thunderbird through the bayes junk filter in Thunderbird. After this is enabled, some of the junk management controls on the folder should be enabled. Try “Run Junk Controls on Folder” to process existing posts for junk.

But nothing will appear to happen when you do, because there is no functionality to delete or hide the junk messages for newsgroups built in. Still, you can see the results using JunQuilla’s “junk percent” and “junk status +” columns:

The next step is to remove those junk messages from your view. The easiest thing to do is to create a virtual folder that shows the messages in the newsgroup without the junk messages. This is where the user interface fights against you. But you can trick it in the following manner.

Create a virtual folder on a message folder with search criteria “Junk Percent < 80″. Save the folder as a subfolder of a local mail folder. Now open the virtual folder and change the folders that it scans, removing the original folder and replacing it with the newsgroup folder:

Now the spam posts are hidden from this virtual folder:

You may need to specifically train the junk filter using a few junk and good messages in news. But the good news is that newsgroup spammers are not really optimizing against bayes filters, so it seems to be a lot easier to detect newsgroup spam than mail spam.

In the Thunderbird 3.0 / SeaMonkey 2.0 series, I snuck in a little hidden preference to allow modifications to the way that the TB bayes filter does it tokenization. The main point of this was to allow better transfer of information between SA and TB. I’d like to describe here how to use that feature.

SA’s decision process involves two key steps. First, they evaluate the message with zillions of rules, and tag the message with each rule that the message hits. Second, they have a method of combining all of those tags into a final junk score for the message, which is used to decide whether to tag the overall message as spam.

SA’s rules include many tests that are not done by TB (score one point for SA). Yet when they take all of their rules and combine them together, in most cases they are using broad measures of goodness and spaminess that apply to some concept of a universal email. In contrast, TB’s bayes algorithm will precisely tune the junk analysis to the particular set of emails that a user gets. So if you are in the Viagra business, you can tune your local TB filter to accept those. (score one point for TB).

Wouldn’t it be great if you could combine together the superior rule set of SA, with the superior decision making customization of TB? That is the point of the hidden preference.

SA communicates its message tags to TB in the form of a custom header, X-SPAM-STATUS. A sample value of that from a spam message that I got is:

X-Spam-Status: No, score=4.9 required=5.0 tests=HTML_IMAGE_ONLY_16,
HTML_IMAGE_RATIO_02,HTML_MESSAGE,HTML_SHORT_LINK_IMG_2,
MIME_HTML_ONLY, SPF_PASS shortcircuit=no autolearn=disabled version=3.2.5

This message was not marked as spam by SA, as it just barely missed the score=5.0 cutoff point, so arrives in TB as a suspicious but nevertheless good message. Yet there is lots of information about the tests that SA did in that header. TB should be able to take advantage of that information.

In the default TB configuration, the entire content of the X-Spam-Status header is treated as a single token. So TB will only respond to exact matches to that header. Yet those tests can appear in a variety of combinations, so that is not really taking advantage of the information effectively.

The answer provided in TB3 / SM2 is to provide a hidden preference that allows you to tell TB to break that particular header up into lots of separate tokens, and analyze each of those separately. That is documented in the code here. In the preference, you specify a particular header, and give a list of delimiters for that tokenization. So to process the x-spam-status header as individual tokens including both space and comma as delimiters, you add the preference:

mailnews.bayesian_spam_filter.tokenizeheader.x-spam-status

and assign it the string value ” ,\t\n\r\f”

(That can be hard to set with all of those special characters in it). That includes space, comma, tab, line feed, carriage return, and form feed as delimiters.

You really can’t do this to an existing training corpus without getting strange results. So you must reset the training data and retrain after you do that.

As a sample, the tokens used by TB to process an email with my example x-spam-status is:

This image uses the “Junk Analysis Detail” report from my JunQuilla extension.

Things to notice:

1. The tokens that TB is using to analyze this email are mostly these x-spam-status tokens, as this is a mostly-image spam message with little text for TB to analyze. So the use of the SA tokens greatly enhances the power of TB’s internal filter.
2. The final score for this, 84, ended up with the message marked as junk for me. I run a cutoff of 75, while the default cutoff is 90.
3. Many of the x-spam-status tokens are actually working against us here. For example, x-spam-status:required=5.0 has a score of 23, which moves the message toward the “good” status. Yet that token appears on every single email I receive, so should be neutral. What is going on here?

   This is the issue that I reported in a previous post “bad effects on junk training corpus from change” That is, recently I have been automatically training more messages as Good than Junk. “x-spam-status:required=5.0″ must not exist in my older emails for some reason, so this bias is appearing because 1) the token changed at some point in time, and 2) there is a mismatch in the number of junk and good emails being trained recently. I really need to figure out a solution to this. What I will probably do is to add features to FiltaQuilla to allow me to precisely match the training of junk and good to remove this bias.

I still need to do some tests of this to directly compare the performance of the spam filters with and without the tokenization to be sure this is really a good idea. That is quite tricky to do unfortunately.

In spite of these issues, I would recommend to new installations that you use this alternate tokenization when you are using a Spam Assassin front-end to Thunderbird or SeaMonkey.

JunQuilla is my attempt to extend the user interface in the Mozilla mailnews product to provide the information that I believe is needed to properly manage the bayesian junk filter. I suppose that most of these features should really be in the core product, but I found that support for that was not very strong, so I decided to do most of this in an extension instead. These backend features have only been added to the core code in the last couple of years, so this extension will only work on newer versions of the Mozilla email clients (Thunderbird 3.* versions, and SeaMonkey 2.* versions.)

Version 1.0.0 fixes some bugs that have been reported in previous releases, provides partial support for SeaMonkey (except for the “Uncertain” folders), and adds a number of new features:

Junk Options

You can set critical overall junk options in the standard junk options screen (previously, this was only possible in the more obscure addons/options area). In Thunderbird, select Tools/Options/Security/Junk. In SeaMonkey, select Edit/Preferences/Mail & Newsgoups/JunQuilla. There you will see a display like this:

“Junk threshold” is the percentage value as calculated by the bayes classifier for each message, above which a message will be classified as junk. This should be set as low as possible, though always high enough to avoid having any real messages classified as junk. The default value of 90 is much too high for a well-trained bayes classifier.

The “Maximum token count” is a measure of the resources that the junk classifier will use. The higher it is set, the more accurate your classifications will be. The default value of 100,000 is probably too low for good classification performance. I’ve had good results with 300,000 – and JunQuilla will set your value to this when first installed. If this value is too high, and you have trained a lot of messages, then memory usage may be excessive.

The other parameters are read only, and are displays of values from your training file. The “Current token count” shows how many junk training tokens (which are like words) are currently in use. You probably won’t get good performance until this number is over 10,000 – and it really should be more like 100,000. “Good” and “Junk” messages trained shows how many messages have been used to train the junk filter. Ideally the number of junk and good messages should be more or less equal. If they are not, then pick some previously untrained messages and train them.

When the number of tokens exceeds the maximum value, then Mozilla mailnews will prune the training file in a large chunk, typically reducing both the number of trained messages, and the number of tokens, in about half.

Disable/enable junk processing for a folder.

You can set an “inherited folder property” to allow you to selectively enable or disable junk processing for folders. This has two main uses.

• If you have server-side filters that process email in IMAP, then you may already know that certain folders contain either junk mail or good mail, and don’t want to waste time processing them locally – or take the risk that they will be processed incorrectly.

• Mozilla mailnews core code now supports junk processing of RSS and News folders. You can select certain RSS or News folders, and then junk processing will run on new posts to those folders. This will also enable the standard user interface features that allow you to train messages as good or junk in those folders.

To set this, right click on a folder in the folder tree, and select Properties, then the “General Information” tab. At the bottom, you will see this:

“Analyze Junk” is an inherited folder property. What that means is that each folder can either gets its value from its parent, or can be set locally. The default value depends on the characteristics of the folder itself. So for example, this would be disabled in News by default, but enabled in IMAP. To change the value, first reset the “Inherit” checkbox, then set the value that you want in “Enabled”. If you change a value for a folder, then the value will also change for the children of that folder (assumming that they have the default “Inherit” checked.)

Toolbar “Is Junk” and “Is Good” button

You can add two new buttons to your toolbar – “Is Junk” and “Is Good”. Here’s what they look like, next to a standard “Junk/Not Junk” icon:

To add these buttons, right click on a toolbar, select “Customize”, then drag the buttons to the desired location.

The standard Junk button will always show as “Junk” when it thinks a message is good, and “Good” when it thinks that a message is junk. But that means that we can only classify a message as “Good” when it has been falsely classified as junk, and we never want our junk filter to do that. The “Is Good” button is meant to be used in the “Uncertain” folders to give you a means to train a message as “Good” there.

Join the forum discussion on this post

(All references to extensions in this posting refer to the 1.0.0 versions, which as of this writing have not been posted to AMO yet. But they should be available in a few weeks.)

Implemented UI

Briefly, inherited properties are a property that can be defined globally, at the server, or at the folder, and its characteristics will be propagated to child objects. This make it easy to specify precisely how the property is applied.

As an example, I have recently implemented a feature “Index in Global Database” in GlodaQuilla which can be used to selectively suppress certain accounts or folders from being accessed by the global database indexer. In the account manager, where indexing can be disabled for an entire account, the UI looks like this:

Each inherited property has default values which are typically set by the base code. In the case of the gloda database indexer, everything but newgroups are indexed by default. Initially each inherited property is set to just use the standard default processing, but if I clear the “default” checkbox, then I can turn off gloda indexing for this account.

If I do that, then go to a first-level folder in the account, I see the following under folder properties:

At the folder level, because I disabled global indexing on the account, it is now shown as disabled on the folder. I could clear the inherit box and selectively enable it on just this folder and its children if I wanted.

This particular UI merges naturally with the existing methods of setting properties in mailnews, but I’m not sure it is optimum for an inherited property. The inherited nature could be more clearly shown, and a particular feature more quickly configured, if I showed a tree of accounts and folders, with checkboxes next to each account to enable or inherit the feature. Maybe in a future version.

Implemented properties

Here are some of the implementations of inherited properties that exist in my extensions:

1. (GlodaQuilla) Index in Global Database – suppresses the running of the global database indexer
2. (FiltaQuilla) Apply Filters to Folder – for Imap folders, allow incoming filters to run on that folder
3. (JunQuilla) Analyze Junk – allow junk processing to be turned on or off. This also allows junk processing to run on RSS or news folders.
4. (TaQuilla) Analyze particular automatic tags.

Issue: Existing mechanisms

Ideally, the inherited property would be the one and only way to manage a program feature. But for existing features, the existing mechanisms remain, which can lead to possible confusion. For example, with JunQuilla’s “Analyze Junk” property, there is existing UI to enable junk processing at the account level. Here the inherited property will always override the default mechanism (but that is mostly because I implemented it in core that way, and I have a little influence on how junk processing is handled in core.) For GlodaQuilla’s “Index in Global Database” the behaviour is different. Existing UI will only allow this to be enabled or disabled globally, and the inherited property does not override this. The inherited property code uses the default server preference as a global enable/disable for a property, so if gloda used that same mechanism instead of an independent preference, this issue would go away. I guess I could say the same thing about junk processing as well.

For FiltaQuilla’s “Apply Filters to Folder”, there is a subtle issue in the inherited nature. I did not implement in core the ability of the inherited property to override the existing default as applied to the Inbox, so incoming filters always run on the inbox. That creates 2 ui issues. First, although I show at the account level the “Apply Filters” option, it does not actually suppress application to the inbox as one would expect. Second, I currently do not show a folder property for “Apply Filters to Folder” for the IMAP inbox since it would not make sense there, so that also means there is no way to enable processing of filters for the children of the inbox. Maybe I should call this feature instead “Apply Filters to non-Inbox Folders” to solve this, or change the core code so that the feature also applied to the Inbox.

Issue: My RDF-inspired property for junk management

Looking ahead to a world where a number of extensions might try to define bayes filter traits, in code I recommended that properties used to manage junk processing use an RDF-inspired globally unique identifier. Then I followed my own advice and defined the identifier that controls junk processing on a folder as: “dobayes.mailnews@mozilla.org#junk” Unfortunately, the existing account manager code does not allow periods in property names, which means I could not use the account manager to manage this. I’ve filed bug 525024 on this issue, and perhaps that can be incorporated after TB 3.0 / SM 2.0.

Issue: Missing inheritance levels

I’ve heard others comment that often they want to set a property on a particular class of folders, say on all Trash folders, or all Sent folders. I’ve considered implementing another level in the inherited properties feature, that would be a folder type. So you would then set a property that would be inherited by any folder of a particular type, and its children – and of course also overridden by the local folder property.

Issue: UI for global property

All of the inherited properties could also be enabled globally using the “mail.server.default.<property>” preference, but I did not give any UI for that in my extensions. I thought that would be too confusing for the user to show those preferences, which would be very similar to existing mechanisms. This is not an issue for properties that use the preference system for server-level issues, but none of the existing server-level preferences are also inherited properties. Perhaps we could move that direction in the future.

The default spam processing that is available without using filters (whitelisting, move or delete messages with a sufficiently high threshold) should be sufficient for most users. But for people with special requirements you can now implement those requirements in a filter with customizations. Let me give examples in this posting.

Using the “after classification” filters

Proper care and feeding of spam really needs to classify messages in three ways. Some messages can be easily detected as spam, and should never be looked at. Others are clearly ham, and should be treated as real. But those in the middle need some handling, which may be either training, or perhaps examined weekly to make sure no false positives are there. Default Mozilla mailnews (which is the generic term for features that are available in any of the applications created from this codebase, including Thunderbird and SeaMonkey) junk management doesn’t provide any capability to manage these uncertain emails. My JunQuilla extension provides an Uncertain folder which is focused on the training issue, but with the new  filter features you can have more precise control of this. (Currently you can’t install JunQuilla in SeaMonkey, but I will fix that eventually).

First, let’s see what is new and how it can be enabled.

Create a new filter by selecting Tools/Message Filters … then New. Open up the search attributes menu, and you’ll see something like this:

No junk options! To get those, you’ll need to first select one of the “after classification” contexts from “Apply Filter When”. Then you’ll see something like this:

If “Checking mail (after classification)” is disabled, then you probably are trying to set an after-classification filter on a POP3 account that is actually sending its email to another location (the so-called deferred-to server). You need instead to set the “after classification” filter on the “deferred-to” server, which is typically Local Folders.

Let me explain each of these search attributes.

Junk Percent is the score returned from the bayes filter when classifying the message, with 100 being the most likely to be junk, and 0 the least likely. The default setting in Thunderbird classifies a message as junk when this score is 90 or greater. It is sort of a probablility, but not really because too many false assumptions are made in the Naive Bayesian Classifier for this to really be a probability. Just treat it like a score. Unfortunately default installs of Thunderbird and SeaMonkey do not provide you with any way to see the value of this on typical messages. JunQuilla though provides a custom column that shows this on each message so you can get a feel for what typical values are.

Junk Status is pretty simple, it either Is or Isnt Junk. In the normal case where the internal bayes filter is used to classify the message, this means it had a junk percent of greater than 90.

Junk Score Origin shows you who classified the message. Its values are:

Plugin: the bayes filter.

User: you manually classified this message as junk or good (not useful in an incoming filter, but maybe in a manual filter or search).

Filter: a previous filter action set the junk status.

Whitelist: the spam processing decided this message was good because it was from someone in your address book.

IMAP Flag: this message was classified by another system, so we know it is junk or good, but don’t know why the other system classified it this way. You might see this if you access mail from more than one computer.

Default Thunderbird does not support any way to see the junk score origin on individual messages, though JunQuilla provides a Junk Status + column which uses different icons for each junk score origin.

Classifying messages as uncertain

So let’s design a filter that will move messages to an Uncertain folder if we want to examine them, but not have them clutter the inbox. That’s pretty easy, we’ll just move messages with a junk percent in a certain range to that folder:

The order of message processing in Mozilla mailnews is:

1. Run normal filters (on each message as it is received)
2. Check whitelisting (on a message batch, this and subsequent steps)
3. Run bayes classifier on non-whitelisted messages, and mark messages as junk or good.
4. Apply “after classification” filters.
5. Apply junk message moves using default junk processing.

So at least in theory, you can apply the “after classification” filter to the Uncertain messages, and still let the default junk processing move junk messages to a Junk folder. (Testing of this is welcomed!)

Weak Whitelisting

As a more complex example, spammers are starting to send out emails that have spoofed From addresses that match the domain of your email, figuring that there is a chance that you have these other addresses whitelisted, so you’ll get the spam. To fight this, we’ll setup a filter that does a whitelist that is slightly weaker than the usual all-or-nothing whitelist on those easily spoofed addresses. Because whitelisting occurs before spam processing, and no score will be assigned if the message is whitelisted, you will need to disable the default whitelisting functionality, and rely entirely on message filters for this to work.

We’ll add the following search terms, all of which must match to apply our weak whitelist:

1. From address appears in an address book (this is normal whitelisting)
2. My domain appears in the address (because that is easily spoofed)
3. Junk Score Origin is Plugin (this prevents the filter from running on messages that we classified, in case we run it manually on existing folders).
4. Junk Status is Junk (we’ll only whitelist if the bayes filter thought it was junk. I only do this so that I can see that the filter decided to override the decision of the bayes processor, which needs the Junk Status + column from JunQuilla to see.)
5. Junk Percent is less than 95 (since the bayes filter only marked messages as junk with the percent > 90, this means that we will override the bayes decision for messages between 90 and 95 in score).

Putting this all together, you get a filter that looks like this:

You would also need to define a filter that is applied after this one, that whitelists any messages that meet the normal whitelist criteria, but were marked as junk by the bayes filter.

I’m not necessarily recommending this filter, it was meant as a demonstration. But I hope you can see that the new ability to use the bayes filter in combination with other message criteria in a filter provides lots of new possibilities for more precise handling of possible spam messages.

Major changes are planned for FiltaQuilla (new custom searches) and ToneQuilla (switching to ogg Vorbis as the standard format). JunQuilla should have bug fixes, TaQuilla and GlodaQuilla will be virtually unchanged.

• ToneQuilla mostly gets an important bug fix to stop playing from ending prematurely.
• JunQuilla gets the new Junk Detail Analysis view that was mentioned in a previous post.
• FiltaQuilla gets three new filter actions: Run a Process with parameters, Train as Good, and Train as Junk.

The description page for each of the extensions has updated descriptions of the extensions, including new features.

ToneQuilla

The published version has a bug that interrupts the playing in some cases. I’ll fix that, but otherwise no changes. Prior to TB3 though I would like to implement support for Ogg Vorbis files in TB core, and add that to ToneQuilla as well. I’ll try to add some more tones at the same time – though possibly in a separate extension so that the main download does not get bloated.

JunQuilla

Changes in the backend to the database interface have made portions of JunQuilla quit working with recent Thunderbird builds. I’ll fix that. Also, I will be able to add the junk detail views shown in my recent post since the required backend code has now landed. There’s also a small error that appears in the console on startup that I hope to find and eliminate.

There will also be a bug fix in beta2 in the backend bayes filter code that could make a big difference in the effectiveness of the filter. Previously, the mozilla-added header that indicated whether a message was read or unread was used as a bayes token. But that makes no sense, since marking as read is a post-receipt action, all incoming messages are unread. So depending on how you trained your messages, there could be an unexpected biasing of unread messages either toward good or junk. In my case, unread messages were biased toward good, tending to lower the junk score by about 7 percentage points – which is a lot.

FiltaQuilla

As with JunQuilla, some functions quit working in recent builds (at least Do Not Notify, possibly others). I’ll need to fix those. I don’t know yet if I will have time to add additional requested features. I see that the Penelope team (Mozilla-based Eudora client) is hoping to add the detach attachments action to the core, so I probably will not do that here. I’ll try to get a preliminary version of the nsIProcess interface up so that you could launch a program with parameters.

TaQuilla

The backend changes that I needed to allow this to work have now landed, so I should be able to release a preliminary version of this soft tagging extension.

Here’s a typical result for a spam email, using a modified version of JunQuilla:

The columns may need a little explaining. “Token” is the string that was detected as a token. If it is shown as a word followed by a colon (“subject:melissa,”) then it was found in the header field  shown to the left of the colon. “skip” is an exception, it means that a large token was found of the length given by the number.

“Token %” is the percent likelihood that the particular token is found in a spam email. “Total %” is a running calculation of the likelihood that the entire message is junk, using only the shown token and those stronger. The entire list is sorted in order of decreasing token strength, so that you can see the importance of each of the tokens to the overall calculation.

Just in this one email, several interesting things come out. First, the strongest token was “skip:* 40″ which meant that a long token was found in the body of length 40. Instead of storing such long tokens, TB creates an artificial token that just shows the length. This was an incorrect indicator, as that token was a strong measure of “goodness”.

The next three indicators include “Melissa”. For some strange reason, I get lots of spam email bound for someone named Melissa Barnes, that I think lives in Florida, but that email is directed at my main caspia.com email address. So you can see the effect of lots of training that “Melissa Barnes” is junk.

Looking down some more, there is a token for the “x-spam-status” header, which is given by an upstream Spam Assassin provider. The x-spam-status header often has a lot of interesting information in it, representing tests done by Spam Assassin in its attempts to categorize the email. But those separate tests are not listed as tokens that can be analyzed separately by TB’s bayes filter. Currently, only the “subject” header is broken up into separate tokens, all other headers are listed as a single complete token. I have a bug pending (bug 476389) which will allow me to break specific headers into separate tokens, with a custom set of delimiters, which should allow me to get at that valuable information.

I’ve also learned some things about some difficulties that the bayes filter is having. The x-mozilla headers that TB adds to keep track of message information such as unread, is being added to the token list. This can have some unexpected negative results. If, for example, in the past you trained “good” messages that were read, but “junk” messages that were unread, then “unread” becomes a strong indicator of a message being spam. Yet all new emails are unread! This is not a good thing. I’m going to remove adding those headers in the future in bug 472005.

Another issue is how the bayes filter deals with tokens that are neutral, that is have probabilities near 50%. Although the bayes filter does not use tokens with probability >40% and <60%, often in a long spam email there are a lot of random words that are are in the 30s and 60s percent. These words, appearing randomly, seem to force the calculation closer to 50% – which will not be detected as spam (by default TB uses 90% as the spam detection limit. I use 75%). I need to investigate this further, as you should not be able to defeat the spam filter by filling you message with a lot of common words.

See http://mesquilla.com/extensions/junquilla for details.