QA -> Developers communication
January 22, 2010 – 10:44 amA few weeks ago on IRC dmose and I discussed the general issue of how QA communicates priorities to developers. I’d like to hear some comments on that from others, and possibly participate in some sort of trial of improvements.
The issue here is that I see lots of good work going on by people who are mostly involved in QA, such as wsmwk, WADA, and Ludo, but I as a developer don’t really know how to make the best use of that work.
I assume there is supposed to be a waterfall here, from (bug reporter)->(QA)->(developer)->(code reviewer)->(bug landing). I understand all of the steps of the process except this (QA)->(developer) handoff. I would be curious to hear from people involved in QA about what they view the main outcome of their work is supposed to be, as viewed by a developer.
Here’s what my understanding is of the current process. After bugs are submitted, QA has three main responsibilities: 1) get the bug in the correct component, 2) move the status to NEW or one of the inactive states (DUPE, INVALID, etc.) 3) clarify the bug information to get clear steps to reproduce.
Is this accurate?
Let’s look to see how that is working by looking at my recent work. In the last three months, I fixed eight bugs (that’s a little off my desired pace, but we were frozen a lot of that time). What brought those bugs to my attention?
(3) bugs I reported myself, either due to issues I observed or as a result of following support forums.
(3) bugs are fixes of crashes that wsmwk reported from crash stats
(2) bugs were filed earlier by others. If I recall correctly, both of those bugs were items that I noticed first in support forums, then located the bug in Bugzilla and fixed it.
In no cases did the standard QA waterfall process play a significant role in bringing a bug to my attention. And that concerns me, because I see some very competent and dedicated people working hard on QA, but I don’t seem to be making effective use of that work.
Am I somehow not following the process that I am supposed to be following, or is that process flawed? In theory I am probably in a better position than most people here, because I primarily track items that appear in the mailnews core/filters component.
I wish there was a clear way for the QA people to bring a limited number of bugs to my attention that are 1) important, 2) clearly defined and reproducible, and 3) likely to be fairly easy to fix.
The mailnews core/filters category currently has 326 NEW/ASSIGNED/REOPENED bugs in it. I could probably fix a few per month that were brought to my attention. A reasonable expectation might be that 10% of those are addressed in the next year. How are the QA folks supposed to influence the selection of which 32 bugs actually get my attention?
(posted to http://mesquilla.com and m.d.a.thunderbird, followup to m.d.a.thunderbird please.)
rkent
I’m coming to this problem from the other side (as a triager, and not a very active one), and there’s a lot here that baffles me too. I think there are two possible mechanisms to bring bugs to developers’ attention, and that neither is perfect:
- If the developer is particularly interested in bugs concerning some Product/Component pair, he should “watch” the default QA address for that Product/Component (the way I, as a triager, watch general@seamonkey.bugs which is where most new bugs get filed). He will then get bugmail wherever a new bug is filed in that P/C or reassigned to it.
- If the triager knows (or thinks) that a bug is in some particular developer’s (or developers’) ballpark, he should add the corresponding entry or entries to the bug’s Cc list in the hope of getting that dev’s/s’ attention.
- One flaw of the above system is that it is easy to become flooded by a lot of bugmail, to the point of not paying attention to all of them. So what should we do? I don’t know. Maybe timeless, who AFAICT gets at least one (and sometimes several) bugmail message(s) for *every* bug filed or changed at b.m.o., would have something better to contribute than I can.
The two methods that you mention are redundant if the developer is watching the appropriate component. And cc can be used for lots of things, including “please comment on this issue”. A search for all of the bugs that I am cc’d on will not give a meaningful list of bugs that I reasonably should consider fixing soon.
So all of what you describe is in fact the current system, and IMHO is not very effective at helping me to decide “which bug should I try to solve next?”
It seems like we need a way to identify bugs which are clearly reproducible. Determining difficulty of fixing takes a developers time and seems hard to estimate; not that it isn’t very useful. But it seems that at least if bugs have been well triaged with STRs then a developer can much more easily fix the problem. I don’t know that we have a way which marks bugs as highly triaged w/ STRs such that a qualified developer could start on a fix. It would be really good to get some resolution to this issue.