Quality in open source and commercial apps, my take

Lately, I've read several interesting blog posts and news articles on the differences between open source and traditional closed-source commercial software. To mention three firestarters, here are some articles that led to me posting this: The Luxury of Ignorance by Eric S. Raymond, Judging the quality of open source projects by Ron Jacobs and now recently, How can I trust Firefox? by Peter Torr (this has to be one of the most commented blog posts I've ever seen).
Those articles are reasonably lengthy, so I'll summarize: ESR talks about the fact that open source projects often lack the UI design and customer-oriented thinking that f.e. Microsoft products have. There may be an easy-looking installation wizard, but it's actually incredibly hard to use. Ron says the quality of an open source project is very hard to measure, although it doesn't become quite clear why is it harder to measure than for traditional closed source. Josh Carlisle continues from Ron's post by outlining his suggestion on how to evaluate open source projects. And then, Peter Torr stabs at Firefox, pointing out several fundamental flaws in the installation procedure.
Where does all this lead? Now, on to my opinions…

Are quality comparisons different between categories of software?

The main dilemma left unanswered by the open source quality critics is, as already mentioned, that there aren't much better ways to evaluate the quality of closed-source products either. The fact that something is open source only gives you access to the source; that should, by definition, add to your ability of measuring the quality. Then again, OS projects vary heavily in stability; things like Apache and Linux are widely considered to be fairly stable and reliable, while many of the smaller projects are fueled by the enthusiasm of a few developers. Evaluating the sturdiness and health of and open source project may perhaps be even harder than it is for a commercial application.
Josh's blog entry outlines a "scorecard" for an OS project. While I don't believe in summing up the quality of a project by measurements such as "how many unit tests you have" (for obvious reasons) and "do you pass FxCop clearly" (FxCop's view of good code certainly doesn't represent a universally constant expression of quality), I agree this sort of measurements would aid both in development and in evaluating the software. Again, this particular facet applies similarly to all software. It would certainly be interesting to see the scorecard for, just as an example, Farpoint's Spread, one of the buggiest closed-source components I've had the pleasure to work with.
Many of the writers point out that an important facet for an OS project is its response to the actual (not just perceived, as ESR reminds) customer needs. Josh formalizes this to 'Average Time from Defect to Resolution'. Of course, the simplistic numeric evaluation of this is as misleading as statistics go – there are various good reasons to leave bugs unfixed, and good software management does not encourage rushing to every bug at every point. Still, the point is valid; support is fundamentally important for both OS and traditional commercial projects. Closed source vendors tend to provide more timely but also less useful support. "Less useful", because the coders aren't often using the code themselves on a daily basis, and because the release schedules often dictate the rate of fixes.

Some thoughts on the quality of OS vs. commercial software

The category of "open source projects" is so broad that any conclusive statements are likely to hold for nothing but a meaningless fraction of all projects. Thus, it's perhaps best to focus on the more serious OS projects. In the same spirit, it's often not even relevant to evaluate the most marginal shareware components when you're looking at a commercial solution – you do want some stability. Between those groups, I believe some summarizations are possible. The following ones are, of course, my opinions only.

OS code quality beats commercial software

I believe the average code quality for strong open source projects is higher than for many of the serious commercial solutions. By "code quality" I refer to factors such as coding standards, security, unit functionality et cetera. So why do I believe this? I have a couple of main points here.
First, it is a known fact that time constraints and excessively tight schedules form a considerable quality risk. Open source projects have much less release pressures and thus enjoy the liberty of refusing bad-quality code. Again, this statement only applies for OS projects that have a notion of "release" instead of just tarballing random builds like some smallish projects do. Even though there are very professional and very quality-oriented businesses doing commercial software, I haven't yet seen the organization where the release pressure wouldn't occasionally cause bugs to occur. This happens in OS projects too – larger projects such as Mozilla approach the tightly scheduled release model of a business product anyway. However, the scale of the problems is considerably smaller.
Second, OS developers in serious projects tend to be very talented (as all of the referred blog posts also mention). They have often gathered much experience from business projects, but got frustrated by the lack of time and flexibility to do things "the right way" there. The OS projects are a way for them to practice their skills and polish their code to the extreme. This does lead to some excellent engineering solutions. It's often noted that the OS community also has beginner developers in their teams. This is true; but again, successful OS projects have and maintain the time it takes to review and correct the beginner's code prior to checkin. Most commercial product teams also get their share of beginners, but there's much less time and desire to educate them properly. Of course, there are vast differences between organizations here – best companies tutor their newbies better than any open source project could.
Third, the publicity of the code is a factor. We all know closed-source projects get some really ugly – and even risky – hacks thrown in as nobody will see them (this is often related to the pre-release panic). Open source projects can afford less of these, as the code is public and fame is pretty easy to lose. It's also much more personal – many open source projects have a very clear notion of who did and what. The personal responsibility of the product is on a level totally uncomparable to many of the commercial projects.
Fourth, open source projects are often more agile to adopt new architectures, methodologies and so on. They carry less burden from bureaucracy and policy issues. This is also supported by the ability to fork and merge projects more freely – developers and users have a much easier time trying out new features separately from the main installation, thus bringing together more ideas, more experiences and more alternatives than a single-path commercial application would do.
On yet another note: Code quality isn't as much a question of resources as one would easily think. In the end, it only takes one developer to write excellent code. He won't write as much code as a team of 50 full-time coders will, but he will probably write better code – just because he's alone, and the effort of coordinating different parts of the codebase are minimal. Of course, lonely programming isn't optimal even for code quality, but the same argumentation does work for team sizes (to a certain extent).

Product quality is a much tougher one

Most of the projects don't live just by their code. Measuring the code quality as written out above isn't the key factor for most product selection scenarios. When it comes to support, documentation, user interface issues etc., it's very rare to find an open source project to beat even the average commercial product. Now resources come into play.
If you're a coder, you can use open source tools pretty easily. In the .NET world, NUnit, NAnt, NDoc, SharpDevelop and CruiseControl among others are excellent examples of quite usable open source software – because you don't need much else than the code. Some of the projects even have decent documentation. But when you pick up Linux and OpenOffice and install it on a newbie's computer, you're in for a world of hurt. Although even OS desktop software has taken giant steps from its early days, very few open source applications can give their user the Luxury of Ignorance. When you don't have it, you need support and manuals. Too bad you often don't have them either, so you're screwed. Firefox has usability issues (even Peter's provocative post about the installation problems isn't entirely pointless), but it isn't from the worst end of the scale.
All this tends to tip the scales towards the direction of the commercial solutions. The closer to a newbie user you go, the more of the open source's bad sides you encounter. From the other perspective, the software companies' money (hey, you can actually get somebody to do the tech writing job the OS coders consider too boring) and the designated management / decision making system tend to produce more solid byproducts. Also, it is true that a commercial company may actually follow its customers' needs better than an open source project, where the key feature decisions are often based on the developers' personal needs.

Some concluding remarks and notes for the future

Putting open source and commercial projects on the same line isn't easy. Even though these different categories of software origin come from very different backgrounds, they sort of compete. However, the open source movement isn't just another vendor. If you're lucky, you can take an OS project and use it just like you would use a commercial product. In these cases, the competition is fairly balanced. This works well for extermely well-polished OS projects (rather few) and the ones where you just need the binaries (many simpler code libraries, developer tools etc.). Normal home users naturally make their choice based on the product – for them, the development and maintenance philosophy rarely plays a significant role.
However, if you're making a bigger strategic choice, the playfield is much broader. An open source solution may not provide all the service you could get from a commercial product, but if you're ready to work some, you will also gain benefits you could never have from an external vendor. In a way, using an open-source product with an evolutive approach is a cross-over between using a shrink-wrapped package and writing (or having written) your own tools. This is the key part many organizations seem to ignore. Sometimes they have to, often they just don't think. The openness of open source provides the possibility to be your own tech support. That is not just a bad thing, even if it certainly requires some work.
The amount of the "well-polished OS-based projects", i.e. those ready to compete with commercial products, is growing. In the future, there will be increasing amount of choices to make. From what I have seen, I'd say the involvement of even one serious mid-sized company in an OS app's development can work miracles. Movement attracts movement, and suddenly you have a pretty nice team of developers maintaining not just their own code, but also yours. I've seen companies take use of Bugzilla in this way, and BZ is far from being the best example.
For business use, comparing open source apps with traditional software packages is like comparing the work of your own employees and the result of an outsourcing deal. Sure you can just look at the work results (did our offices get properly cleaned, or what does the software product look like?), but then you're ignoring much of the fundamental differences in the business model required (why did you go outsourcing anyway?). Deeper analysis is required for real results. Open source isn't always the winner – actually, it doesn't usually seem to be these days. But things are changing, and the scales are becoming more balanced. Again, good understanding and unbiased, honest evaluation of what's going on helps you benefit from the change.

December 23, 2004 В· Jouni Heikniemi В· 3 Comments
Posted in: General

3 Responses

  1. Josh Carlisle - .Net Brain Freeze - December 28, 2004

    More Thoughts on evaluating open source.

  2. Josh Carlisle - December 28, 2004

    I enjoyed your comments Jouni!
    In the tradition of blogs: :)
    http://viewfusion.com/blog/jcarlisle/archive/2004/12/28/200.aspx

  3. My DasBlog! - May 17, 2005

    More Thoughts on evaluating open source.

Leave a Reply