What Filters Need

I’ve just completed a two-week hiatus from my work on Exchange Web Services (EWS) integration project to focus on fixing a few filter issues in mailnews core (Thunderbird and SeaMonkey) prior to TB’s 3.1 feature freeze. I will now return to EWS as my primary focus. But before I do, I think it would be useful to give an overview of some big-picture issues in filter and traditional search (which are closely related), providing something resembling a roadmap but without any real commitment by anyone to move forward.

Mailnews Traditional Filter and Search – Big Issues for Enhancements

1. Integrate full boolean search into Search Term editor

• Bug 297852 – Search/Filter and Boolean expressions

The mailnews backend traditional search logic supports full boolean expressions, but the user interface has never been updated to allow this. With asuth’s pending update to the quick search UI, making extensive use of boolean expressions internally, we are getting to the point where quick search can express certain searches that so-called “advanced search” cannot (and vice versa.)

2. Store filters globally, and apply them to selected account/folder trees/contexts

• Bug 11039 – Filter outgoing messages (perhaps to use a different Sent/FCC folder)
• Bug 166842 – [RFE] Ability to import and export message filters
• Bug 129883 – Filters Should be Shareable Across Multiple E-mail Accounts
• Bug 59365 – Filters should have triggers
• Bug 34973 – (GlobalFilters) global message filter option

Filters are currently stored per account (except in news, where they can also be per-folder). But when people define a particular filter, they frequently want it to apply to more than one account. There is no easy way currently to do this, short of recreating all filters on each account. Moving filters is also a pain when accounts are reorganized.

I would store filters in a single location, and provide some sort of UI to define where and when each filter is applied. That would not only be the account or folder, but also the context (such as incoming, manual, sending, or triggered.)

3. Eliminate context and account dependencies of search and filter options

• Bug 67421 – Missing possibility to filter on text in body for IMAP
• Bug 223591 – Junk classification should execute on each message as it’s received, not after all have been fetched
• Bug 196036 – Add junk mail status to search/filter criteria

A necessary prerequisite of “Store filters globally” is to eliminate the current complex dependencies that search terms and filter actions have on the precise capabilities of a particular account, folder, or context. Some examples:

1. Body search is not available for IMAP unless “download for offline use” is selected for the account
2. Junk search terms are not available unless the filter context is “after classification”.
3. News has a much more restricted group of available search terms and filter actions than mail.

A major reason that we currently store and define filters the way that we do is to support these restrictions. Yet somehow we manage to silently ignore them without grave consequences in saved searches – you can define a particular saved search, and then expand its application to include folders and account where the underlying search terms perhaps make no sense.

Unfortunately this creates major problems for the user, who has no way of discovering that in order to activate a junk-based filter, he has to define it as “after classification”. Or before she can do a body search on IMAP, “download for offline use” must be enabled. It also means that the whole concept of a global filter, which is much desired, goes against the current philosophy of precise enabling of available features.

I will confess that I have been a major contributor to these issues in the last couple of years, in my zeal to maximimize the availability of features while minimizing the impact on existing behavior. But I don’t really like the result. I think, instead, we should just let the user define what they want, and have the software silently change the behavior to match (download the IMAP body if they want to search on it, delay filters until after bayes classification if they want to search on junk terms.) For cases where something makes no sense at all (like asking to search a news folder for cc) we should define reasonable defaults, and just ignore that the search does not make sense. Perhaps if the filter is being defined only on a news folder, then we could partially hide terms that don’t make sense (or gray them out). But a global filter should have all terms available, even if they don’t make sense in all circumstances.

4. Centralize storage of message metadata related to MIME processing

• Bug 105169 – Filter for Attachments
• Bug 224392 – Feature request: Add ability to filter by attachment name (extension)

Currently, MIME processing is done on demand when a message is displayed, or as part of gloda indexing. But MIME processing is relatively expensive as well as async, which means that its results are not available in traditional search and filtering. We really need to perform MIME processing on all messages, and store basic MIME-related metadata so that it only needs to be done once, and that information would then be available to all consumers, including traditional filter and search.

5. Provide discoverable path to optional search and filter features

• Bug 19442 – Regular expressions in mail and news filters
• Bug 9309 – option to have attachments auto saved to chosen location
• Bug 438441 – (bifftones) Custom bifftones for mail filters (like custom ringtones)
• Bug 200252 – filter action “print message”
• Bug 397009 – A filter will let me tag, but not untag

These are a sample of requests that have been implemented in my extensions. There will always be lots more possibilities for search terms and filter actions than make sense to implement in core. I am not convinced that the current method of expecting users to search for extensions to do things is optimal. In the case of custom filter actions and search terms, it might make sense to have a whole category of normally disabled custom actions and terms that can be discovered by all users, added to by extensions, that enable turning on rarely needed actions and terms.

Bug-related issues

There are another class of important filter and search bugs that are problems rather than enhancements, and whose fix rely on solving difficult core issues. Some of these are:

6. Resolve architectural issues of new message notification

• Bug 11040 – Allow filters to control biff UI (i.e. only notify me of “important” msg)
• Bug 222068 – Reading filtered mails does not clear “New Mail” icon
• Bug 228168 – Getting new mail notification with 0 mails when mail is filtered to local folder from IMAP (“<account> has 0 new messages”)
• Bug 233929 – Still get notification on junk if messages left in Inbox
• Bug 378582 – Filter which moves messages from IMAP account to Local Folder prevents new mail alert

As I have said before, new message notification is hopelessly confused in mailnews, and needs refactoring.  There are lots of notification-related bugs that I an afraid to touch until that is done.

7. Queue operations for busy folders

• Bug 196732 – Some messages correctly identified as junk are not moved to the Junk Mail folder
• Bug 168648 – Cannot filter (move) messages to Inbox that is busy (fetching its own mail)
• Bug 139215 – mail not filtered while compacting folders

Currently if a folder is unavailable for some reason, then filtering just fails. This creates a constant source of unreliable filtering. We really need some explicit method to handle this, perhaps a queue of operations to apply to a folder when it is available again.

Next steps

I don’t really have the time to implement any of these issues for the foreseeable future. I don’t know for sure what MoMo’s priorities are, but I would guess that filtering is generally not high on the list. So I guess I’ll need to put out a help wanted sign (but don’t believe the “reward” part!).