A view on bug report voting systems: 0/1 vs. 0-x

MSDN Product Feedback Center Blog explains how bug votes and Dr. Watson reports affect bug priorities inside Microsoft. The story discusses the differences between "this is important for me"-style vote (0/1 votes) versus the current MSDN style (where you can give a bug an importance score of 1..5 – or a 0-x voting system from a more generic perspective).

The aim of both approaches is to get a community opinion on which bugs are the most critical ones. Dr. Watson's error reports (just like f.e. Mozilla Quality Feedback Agent's similar ones) act as 0/1 votes – either a user experiences a crash or not. So, for crash bugs, Watson measures the amount of occurrences in the user base. For non-critical bugs, a good measure is harder to find, since you usually cannot get automated problem reporting.
A 0/1 vote on bugs is slightly problematic, because sorting by the vote count doesn't really tell you about the severity of the issue. An enhancement request to a really visible part of the product can bypass a dataloss bug in a less popular module. Even if you trim out enhancements requests (RFEs), you're still left with a pretty flat view of purely quantitative measurements.
The bad part here is that a 0-x voting system is usually no less troublesome. While you do get qualitative opinions on the importance, they're likely to be misleading. Most normal people vote maximum importance on any bug they've encountered regardless of its objective severity. A particularly serious bunch of reporters can probably constrain their voting habits a bit, but with big publicity products 0-x voting easily degrades into a 0/5 system.
Don't believe that? Go to PFC and search for bugs reported in last couple of days, filtering by "rating 5". As a result, I noted that 33 of 88 bugs are of rating 5. Sure, at least 20 of those drop from the 5.0 rating because somebody else considers them irrelevant and votes 1 – but not all of them. People aren't very active at downvoting reports they don't consider important, even though (at least theoretically) the best way to make your own bug report stand out would be to vote 1 on every other report.
Eventually the system ends up with numerous irrelevant reports at the top of the list, and loss of confidence in the whole rating system. How about filtering by vote count then? Sure, but you'll be losing information _and_ gathering a base of rating 5 bugs which you'll never see on your radar. Doesn't really look good from a customer perspective.
Oh all right, I'm pretty pessimistic here. Still, past years have strengthened my belief in one model: 0/1 voting and an easy way to post verbal comments. You'll get whining ("Fix this, it's the main reason my friends won't use X!!!"), but the freetext entry also gathers the most informative views and the best arguments – that's far more effective and credible qualitative analysis than a 0-5 vote. It's harder (less automated) for the developers, but definitely better for the product.

August 9, 2004 · Jouni Heikniemi · No Comments
Posted in: General