Public vs. private beta testing

In late July Microsoft published IE 7 beta 1. But what? It's not public! Now it's time to scream! Really, reading the IE team blog's post with all its comments is… enlightening. I'm a bit surprised we've got to the point where people start whining when a beta isn't public – it's not like public betas had been the standard operating procedure all along anyway. Of course, open source projects have a different concept of beta releases, but why should it be so for commercial applications?
A public beta is making your users test the software for you. This is something people loathed at a time, saying that software houses should do the testing stuff themselves. Of course, a beta is also a chance to see a preview and get ready for new technology, but still – early betas rarely always provide truly useful insight into the interesting new things. It's mostly about testing and reporting bugs, really.
Contrary to what one might initially think, getting as many bug reports as possible is not beneficial for the quality of the product. Getting as many bugs reported as possible is what you want. Although more testers provide more feedback and thus more useful findings, the signal/noise ratio drops dramatically as the number of eager testers rises. Remember some of the Mozilla suite's releases when bugzilla.mozilla.org was flooded with duplicate/invalid bug reports?
Now, if Microsoft has the testing department that's sufficient for the product's current phase AND they know IE7 is unfinished enough not to be ready for everyday use, why should they release a public beta that's bound to get them kazillion bug reports on issues they already know about?
Looking at the various reports people have written on IE7, I'm inclined to believe this wouldn't have been the phase for a public beta. But as a more general phenomenon: Microsoft's first betas have often been rather shoddy tech previews. Beta 2s, on the other hand, have often represented nigh-production quality. It'll certainly be interesting to see both IE7B2's quality and release model. At least a lot of promises have already been made.
There are several roads to quality software, and not all of them involve slipping out pre-production software to everybody who happens to want it. Different resources and methodologies give room for different approaches. At the end, judgement by final product is the only fair way.

August 6, 2005 В· Jouni Heikniemi В· Comments Closed
Posted in: General