In addition to some new filter actions (print, add sender to address list, and save attachments to a folder) this release introduces “custom search terms” for the first time. This is a new feature that has been added recently to the mailnews core code, and is part of the TB 3.0 and SM 2.0 releases.

The search I am talking about is the old-style Thunderbird search, not the newer global database (gloda) search that was added to Thunderbird. Gloda gets all of the press, but we’ve also taught the old style search a few tricks as well! These are particularly useful in saved searchs (also called virtual folders).

I think that the most interesting new capability is that you can define a search (and therefore a virtual folder) by adding a few lines of javascript code that does precisely what you want. Let me give a very simple example.

Let’s assume that you want a virtual  folder to contain all of the active items that you currently need to process involving projects. Incoming emails are marked with tags, either manually or by some sort of filter. You define new tags for each project, and each tag begins with “pro” and ends with some sort of project marker, say a number or word. You want to have your active folder contain messages that have a tag containing “pro”, but NOT include messages that are tagged “done”.

The standard mailnews tag search forces you to enter the tag name for each tag that you want. There is no way to search for tags by the characters in the tags. Plus, searching for several tags is an OR function, and saying to not include DONE messages is an AND function. But the standard mailnews search does not handle complex boolean searches.

The javascript custom search comes to the rescue! Go to the folder that contains the messages that you want, and select the Javascript search term:

Click on the script icon and you will get a small editor window, where you can enter a few lines of javascript. All that we need to do is grab the string that has the tags in it, and make sure it includes “pro” but does not include “done”. Our javacript is given a variable “message” which is the database header object that can be used to get message properties. We need to execute an expression as the last statement, whose value is either true if we want the message, or false if we do not. That’s just the following two lines of javascript:

let tags = message.getStringProperty('keywords');
(/pro/.test(tags) && !(/done/.test(tags));

So here’s what our javascript window and code looks like:

Now save this as a virtual folder, and you have the exact virtual folder that you want!

OK, this is just for geeks, but it is really powerful in letting you define folders that can precisely define the workflow that you want. There’s also a few other geek-friendly search terms, including the much-requested regular expression search by subject or other header. For details, see the updated FiltaQuilla page on this site.

FiltaQuilla is still in experimental status, though I have now nominated it to be public. Still it may be a few weeks before it gets there. If you are an existing FiltaQuilla user, you will need to go to the download page directly and download and install the new version.

Enjoy!

Join the forum discussion on this post - (1) Posts

So far, I have added the following new search terms:

BCC – locate items in the BCC field

Subject Regex – search the subject using a javascript regular expression

Header Regex – search any specific header using a javascript regular expression

Javascript – load javascript in a text field, and program your own search given an nsIMsgDBHdr object

Tag of Thread Head – match a tag in the head of a message’s thread

Tag of Thread Messages – match a tag near the message in its thread (within +/- 10 messages by default)

Address in Thread – match an address near the message in its thread (within +/- 10 messages by default)

This stuff can be useful outside of filters, in fact I am mostly using them personally to define virtual folders. So I’ll probably move them to a new extension, and try to get FiltaQuilla out the door finally.

rkent

(All references to extensions in this posting refer to the 1.0.0 versions, which as of this writing have not been posted to AMO yet. But they should be available in a few weeks.)

Implemented UI

Briefly, inherited properties are a property that can be defined globally, at the server, or at the folder, and its characteristics will be propagated to child objects. This make it easy to specify precisely how the property is applied.

As an example, I have recently implemented a feature “Index in Global Database” in GlodaQuilla which can be used to selectively suppress certain accounts or folders from being accessed by the global database indexer. In the account manager, where indexing can be disabled for an entire account, the UI looks like this:

Each inherited property has default values which are typically set by the base code. In the case of the gloda database indexer, everything but newgroups are indexed by default. Initially each inherited property is set to just use the standard default processing, but if I clear the “default” checkbox, then I can turn off gloda indexing for this account.

If I do that, then go to a first-level folder in the account, I see the following under folder properties:

At the folder level, because I disabled global indexing on the account, it is now shown as disabled on the folder. I could clear the inherit box and selectively enable it on just this folder and its children if I wanted.

This particular UI merges naturally with the existing methods of setting properties in mailnews, but I’m not sure it is optimum for an inherited property. The inherited nature could be more clearly shown, and a particular feature more quickly configured, if I showed a tree of accounts and folders, with checkboxes next to each account to enable or inherit the feature. Maybe in a future version.

Implemented properties

Here are some of the implementations of inherited properties that exist in my extensions:

1. (GlodaQuilla) Index in Global Database – suppresses the running of the global database indexer
2. (FiltaQuilla) Apply Filters to Folder – for Imap folders, allow incoming filters to run on that folder
3. (JunQuilla) Analyze Junk – allow junk processing to be turned on or off. This also allows junk processing to run on RSS or news folders.
4. (TaQuilla) Analyze particular automatic tags.

Issue: Existing mechanisms

Ideally, the inherited property would be the one and only way to manage a program feature. But for existing features, the existing mechanisms remain, which can lead to possible confusion. For example, with JunQuilla’s “Analyze Junk” property, there is existing UI to enable junk processing at the account level. Here the inherited property will always override the default mechanism (but that is mostly because I implemented it in core that way, and I have a little influence on how junk processing is handled in core.) For GlodaQuilla’s “Index in Global Database” the behaviour is different. Existing UI will only allow this to be enabled or disabled globally, and the inherited property does not override this. The inherited property code uses the default server preference as a global enable/disable for a property, so if gloda used that same mechanism instead of an independent preference, this issue would go away. I guess I could say the same thing about junk processing as well.

For FiltaQuilla’s “Apply Filters to Folder”, there is a subtle issue in the inherited nature. I did not implement in core the ability of the inherited property to override the existing default as applied to the Inbox, so incoming filters always run on the inbox. That creates 2 ui issues. First, although I show at the account level the “Apply Filters” option, it does not actually suppress application to the inbox as one would expect. Second, I currently do not show a folder property for “Apply Filters to Folder” for the IMAP inbox since it would not make sense there, so that also means there is no way to enable processing of filters for the children of the inbox. Maybe I should call this feature instead “Apply Filters to non-Inbox Folders” to solve this, or change the core code so that the feature also applied to the Inbox.

Issue: My RDF-inspired property for junk management

Looking ahead to a world where a number of extensions might try to define bayes filter traits, in code I recommended that properties used to manage junk processing use an RDF-inspired globally unique identifier. Then I followed my own advice and defined the identifier that controls junk processing on a folder as: “dobayes.mailnews@mozilla.org#junk” Unfortunately, the existing account manager code does not allow periods in property names, which means I could not use the account manager to manage this. I’ve filed bug 525024 on this issue, and perhaps that can be incorporated after TB 3.0 / SM 2.0.

Issue: Missing inheritance levels

I’ve heard others comment that often they want to set a property on a particular class of folders, say on all Trash folders, or all Sent folders. I’ve considered implementing another level in the inherited properties feature, that would be a folder type. So you would then set a property that would be inherited by any folder of a particular type, and its children – and of course also overridden by the local folder property.

Issue: UI for global property

All of the inherited properties could also be enabled globally using the “mail.server.default.<property>” preference, but I did not give any UI for that in my extensions. I thought that would be too confusing for the user to show those preferences, which would be very similar to existing mechanisms. This is not an issue for properties that use the preference system for server-level issues, but none of the existing server-level preferences are also inherited properties. Perhaps we could move that direction in the future.

Major changes are planned for FiltaQuilla (new custom searches) and ToneQuilla (switching to ogg Vorbis as the standard format). JunQuilla should have bug fixes, TaQuilla and GlodaQuilla will be virtually unchanged.

So here are things that are new in Thunderbird 3.0 beta 3 (Seamonkey 2.0 beta 1) that involve message filtering:

1. Imap filtering on folders that are not the inbox.

If you have a server filter that moves IMAP messages to a folder other than the inbox, previously you could not use TB filters on it. Now you can, controlled by an inherited folder property. There is no user interface for this feature at the moment, but I will add some to FiltaQuilla. See bug 257415.

2. “From, To, Cc, or Bcc” filter/search term

This is a new search term that can be used in advanced search or filters. It does two things: combines all of the normal address fields into a single term, plus adds support for Bcc for the first time.

The support for Bcc needs some comments. Just in case it isn’t clear, Bcc information is not added to outgoing emails that you send (or the incoming emails that you receive), so the only place it really shows up is in emails that you sent, and have kept your own copy of. So the Bcc term does nothing for emails that you receive, only emails in your sent folder. Also, we currently do not have any mechanism to automatically apply filters to messages that you send, so really the main place Bcc will show up in filter/search is in Advanced search, saved searches (virtual folders), or in manual filters (filter-after-the-fact).

3. Junk fields in manual filters

We have now enabled the junk-related search terms of “Junk Status”, “Junk Percent”, and “Junk Status Origin” in manual filters.

There is a subtle change to the behavior of search term editing that was implemented that affected this. Previously, if you caused the filter editor to select an invalid search term (typically by changing the filter context away from “Manual”), that was ignored, and you could save the invalid filter. Now, the invalid terms are grayed out, and attempts to save the filter will result in an error dialog. This behavior was important to allow adding of more complex filtering options.

4. Custom Search Terms

The biggest change that affects filters is the ability for extensions to add custom search terms. This is a complement to the addition of custom filter actions, which were available in beta 2, and are the main point of FiltaQuilla. I expect to do a blog post soon detailing the code necessary for an extension to add a custom search term, plus I will add some custom search terms to FiltaQuilla (probably a regex search on a header).

See bug 495519 for details, including a demonstration extension that adds custom terms.

5. Fix of message copy/move bug.

There has been a long-standing bug involving failed moves or copies, that typically occurred when there were automatic copies or moves of emails, such as are done in filters or junk processing. Bug 497622 hopefully solves this, which should improve the reliability of moves and copies.

What did’t make beta 3, but should be in TB3.0

There are two big changes that have mostly working patches, that did not make beta 3 but hopefully will be available in beta 4.

1. IMAP body filters.

IMAP messages are now downloaded by default, so in theory it should be easy to filter on their bodies. This is bug 127250 (or possibly the closely related bug 67421). This bug is really critical in combination with the new custom search attributes, as it would be very valuable to write filters that analyze each message body, and then take a particular action when something is found there.

2. Post-bayes incoming filter context

Currently, it is not possible to filter on any of the message characteristics that are determined by the bayes filter (including junk status, and the closely related custom traits) automatically while a message is incoming. This is because the bayes analysis, and its related filtering actions, are applied after the standard filters. A post-bayes filter context will allow the user to specify that a particular filter should be applied after the bayes analysis, not before, and then we will be able to enable the junk-related search terms (and an extension could add a filter term that used a custom bayes trait). See bug 198100.

• ToneQuilla mostly gets an important bug fix to stop playing from ending prematurely.
• JunQuilla gets the new Junk Detail Analysis view that was mentioned in a previous post.
• FiltaQuilla gets three new filter actions: Run a Process with parameters, Train as Good, and Train as Junk.

The description page for each of the extensions has updated descriptions of the extensions, including new features.

ToneQuilla

The published version has a bug that interrupts the playing in some cases. I’ll fix that, but otherwise no changes. Prior to TB3 though I would like to implement support for Ogg Vorbis files in TB core, and add that to ToneQuilla as well. I’ll try to add some more tones at the same time – though possibly in a separate extension so that the main download does not get bloated.

JunQuilla

Changes in the backend to the database interface have made portions of JunQuilla quit working with recent Thunderbird builds. I’ll fix that. Also, I will be able to add the junk detail views shown in my recent post since the required backend code has now landed. There’s also a small error that appears in the console on startup that I hope to find and eliminate.

There will also be a bug fix in beta2 in the backend bayes filter code that could make a big difference in the effectiveness of the filter. Previously, the mozilla-added header that indicated whether a message was read or unread was used as a bayes token. But that makes no sense, since marking as read is a post-receipt action, all incoming messages are unread. So depending on how you trained your messages, there could be an unexpected biasing of unread messages either toward good or junk. In my case, unread messages were biased toward good, tending to lower the junk score by about 7 percentage points – which is a lot.

FiltaQuilla

As with JunQuilla, some functions quit working in recent builds (at least Do Not Notify, possibly others). I’ll need to fix those. I don’t know yet if I will have time to add additional requested features. I see that the Penelope team (Mozilla-based Eudora client) is hoping to add the detach attachments action to the core, so I probably will not do that here. I’ll try to get a preliminary version of the nsIProcess interface up so that you could launch a program with parameters.

TaQuilla

The backend changes that I needed to allow this to work have now landed, so I should be able to release a preliminary version of this soft tagging extension.

Under stock Thunderbird, notification of arriving emails (sometimes called BIFF) may be controlled in two ways – and both of these are system-wide settings. These features are controlled from the General/Options dialog:

“Show an alert” adds a small popup in the system tray when an email is received, while “Play a sound” can play a single system-wide sound.

Now if you have a complex setup, then this is not nearly enough control to be practical. I for example have some high-volume emails lists that I receive, plus for testing purposes I keep both IMAP and POP versions of my main account. I really don’t want to be notified in any way for the email lists, nor for the IMAP account receipts. Also, I like to hear sounds only on special emails, not on everything. So, how do you setup a more practical method of controlling notification?

We have 2 extensions that can help with this, ToneQuilla and FiltaQuilla. FiltaQuilla adds a new filter action, “Do Not Notify” that prevents an alert popup for occurring when a message matching that filter is received. ToneQuilla plays a particular sound as a filter action. So the way to have control of your notifications is to turn on the “Show an Alert” notification in Thunderbird (and suppress it for particular emails with a filter), and turn off the “Play a Sound” notification (and add it back for particular types of emails with a filter).

So, for example, for the email list ICCM, my filter is configured like this:

That is, I turn off notification (which is the little popup) and then move the message to a folder.

When I want to play sounds, I turn on sounds for particular classes of emails using ToneQuilla. For example, I like to know when I receive a bugmail message, so bugmail has the following filter:

My wife particularly likes the “we are in Love” sound bite that plays whenever I get an email from her!

With this combination of features, I think that you will find notification in Thunderbird can be controlled now in a useful manner.

• Launch a file in the operating system
• Remove Star
• Remove a Keyword
• Suppress Biff notification
• Append text to the subject
• Mark message Unread
• Mark message replied
• Copy the message to a new folder, marking Read in the new folder.

FiltaQuilla is available here on Mozilla’s addon site. More details are also available on this site.