Which is just to say, if you’re not planning to ship against Thunderbird 3.1, don’t write off SQLite changing the game.

Which is not to say it will be a “surprise! here’s a fully formed implementation” kind of thing, just that it will be “here’s a coherent set of ideas, some of which are already working and all suggest the proposed future steps are not insane, and here’s what you can do to provide insight/feedback/hack on things.”

I should probably also point out that I’m expecting to see you, bienvenu, jcranmer, and other contributors in a week in-person at the mozilla summit. I want to leverage that high-bandwidth reduced-chance-for-misunderstandings environment. If that wasn’t coming up, I would be much more likely to solicit public feedback. (I’m actually working on different stuff (UI framework) between now and then, so there’s little lost opportunity.)

It’s the kind of thing I am reluctant to blog about until I am further along in the implementation process. At the current stage, it is a speculative and contentious plan whose most likely immediate outcome would be at least a low-level flame war.

Which is to say, it would likely make a good number of people unhappy and concerned about where Thunderbird is (or at least I am) headed without any evidence to reassure them or help them change their minds. The people who would be enthused by the direction would likely be happy, but things are not yet at a stage where anyone can really contribute meaningfully. Also, I feel like I’ve already expressed the gist of my gameplan to those would be enthused. For example, if you found this strategy on my part wholly surprising, I would, in turn, be surprised.

It’s very important to note that this is not going to be a short-term kind of effort and the 3-pane view as it currently exists is certainly not going away anytime I can see. In fact, bienvenu’s thinking is much more aligned with your own than mine, and he’s pretty bad-ass, so we could end up in a situation where the 3-pane and ‘skink’ in general evolve to the next level and continue to exist much as they do today.

Yeah, efficient single record lookups should generally result in acceptable performance as long as the header cache is large. The trick is that if you ever want to do more expensive queries, you have to be very careful not to do things in such a way that you block your main-thread synchronous reader.

Probably the safest way to accomplish that would be to use a combination of 1) using a read-only second connection for long running reads when the user is non-idle and 2) only ever performing writes asynchronously on your ‘main’ connection and when the user is idle.

The rationale is that multiple connections can be open and performing concurrent read-only operations without interfering with each other. A single connection can be used on multiple threads, but a mutex means that only one thread can be in a sqlite3_step at a time, which means that a complex query on the async thread (or any other thread) could block the main thread with a complex query, even if it is read-only.

From a write perspective, you definitely do not want to be performing writes on the main thread because you will clog up the event loop. It’s potentially inadvisable to perform writes from a different connection than your main one just because the file locking makes all reads impossible until the transaction has completed (which again will likely result in blocking the main thread), whereas if you’re writing from your main connection then you have a chance to get a read operation in from the main thread even while stuff is happening on the other thread. (Although you are then in danger of getting inconsistent reads.)

So, to summarize, doing any synchronous access from the main thread is not really a great idea, but if you’re working with the existing UI framework, you don’t have much of a choice.

In my implementation, I’m allowing sync grabbing of single records from a simple indexed key, but not of the queries (like “get a list of all item ids that need updating”). I was going to ask you if you thought that was OK, since the mozStorage MDC article “strongly discourages” that, it sounds like you think it is.

“account/message types to just exist at the gloda level”

That must mean that you are tring to get the standard display logic (folder and message panes) to work with some sort of message abstraction other than nsIMsgDBHdr objects. It would be great if you could blog about that.

You can use SQLite synchronously via mozStorage if you want. It’s only a serious problem when you have potentially long-running operations. As long as you stay away from using the full-text search mechanism or complex queries you should be sufficiently fine.

Note that I’m not suggesting that it’s a good idea long term, but if your goal is to work with nsMsgDBView and existing message reader and friends, it’s a better idea than the convolutions that would be required to try and make them work with async.

Vision-wise, my plan is for new account/message types to just exist at the gloda level and never touch the ‘skink’ level, as you so dub it. The work to support that is under way and not proven yet, so nothing is set in stone. Functionality developed at the skink level should still be able to be lofted to the gloda level as will continue to be the case for POP/IMAP for at least the medium term.