Original article: What we can Learn from Two Linux vs. Microsoft Studies
The question of which platform is better for running a business, Windows or Linux, has inspired debates that are nearly religious in their vehemence. Two studies came out this year that purport to settle the question, at least when it comes to issues of security and reliability. Do they really provide a final answer, or just more fuel for the fire?
Two white papers that compared two platforms, Microsoft and Linux, have been generating a bit of buzz lately. The papers are the result of tests conducted by Security Innovation, a provider of application software services, and are dated June and November of 2005. The first one is titled “Role Comparison Security Report – Database Server Role.” It compares the relative security of three different platforms used for the database server role:
1. Microsoft Windows Server 2003 running Microsoft SQL Server 2000 Service Pack 3 database server.
2. Red Hat Enterprise Linux 3.0 running MySQL database server.
3. Red Hat Enterprise Linux 3.0 running Oracle 10g database server.
Databases are vital to practically every business these days, and with so much more of a business’ information available online, security is an increasingly important consideration. Many companies must weigh the cost of a particular solution, but this kind of study can tell them whether they are making a decision that will be a false economy down the road.
The second paper is titled “Reliability – Analyzing Solution Uptime as Business Needs Change.” Its goal was to examine reliability and manageability differences between Linux- and Windows-based solutions. Security Innovation created an e-commerce scenario in which experienced administrators started from either Windows Server 2000 or Novell SuSe Enterprise Linux 8, then had to upgrade to Windows Server 2003 or Novell SuSe Enterprise Linux 9, respectively.
Any administrator will tell you that upgrades can stress the system (to say nothing of administrators!). This is one of a number of reasons that upgrading gets put off. Eventually, it must be done, though, and everyone hopes for the seamless, relatively invisible upgrade where everything just works and the system is down for as short a period of time as possible. Again, there is no question that this is the kind of study that many business owners should find useful to their decision making process.
There are, however, certain questions about the two studies. They were conducted by an “independent” company – the aforementioned Security Innovation. But both studies were paid for by Microsoft. Also, if you check out SI’s partner’s list on their website, it mentions Microsoft – but no open source related companies. You could stretch a point by saying that IBM and HP are on the list, but Novell and Red Hat are conspicuously absent. Not to put too fine a point on it, that would lead me to wonder just how “independent” this study really was. It certainly calls for a closer look.
The first report notes that database servers must “manage, store and retrieve data in a highly available way.” This should lead to concerns about security from the get-go, yet “security has been conspicuously absent” from the list of criteria that IT decision makers have used when choosing an appropriate platform. To study these issues, Security Innovation set up three systems as noted in the first section of this article, and considered vulnerabilities that were patched over the course of a year (from March 1, 2004 through February 28, 2005).
It isn’t quite an apples to apples comparison. The company was able to use the “minimal install” option for Linux and the MySQL database server, which, they note, “yields a smaller attack surface.” In the case of Oracle, SI went with the recommended configuration. With Microsoft, SI needed to do a “complete” installation because “there are many components which are difficult or impossible to completely remove from the operating system…” I leave it as an exercise for the reader whether such differences in the installations will yield meaningful results.
And what were those results? “Looking at just the database applications by themselves, our study found that SQL Server 2000 had zero vulnerabilities in the one year time period, MySQL had 7 vulnerabilities and Oracle 10g had the most with 30 vulnerabilities.” The full server stack results are even more shocking for open source supporters: 63 vulnerabilities for the Windows-based solution, 116 for the MySQL solution, and a whopping 207 for Red Hat running Oracle.
The study also examined what it called “days of risk.” That is the time between a vulnerability being publicly announced and the vendor releasing a patch. That time period was, on average, 32 days for the Windows-based solution, more than 61 days for the Linux-MySQL platform, and more than 38 days for Linux-Oracle.
One aspect of this study that bothers me a little is that I can’t really tell how much it took into account the fact that volunteer support for Linux is strong enough that its forums will have fixes for bugs literally within hours of users becoming aware of them. In the executive summary, SI states that it worked with Mark Cox of Red Hat to resolve discrepancies in vulnerability disclosure dates, but still, Microsoft’s parallel is the monthly bug/security fixes it gives out. That should affect “days of risk.”
Perhaps one of the assumptions made by the study, for comparison purposes, sheds some light on this: “The user requires the features, trust, support and professional maintenance provided by a trusted software distributor.” In the case of Linux, this means that “users will only install versions of the OS components blessed and released by the OS vendor, so that their support contract remains valid.” But Linux administrators are used to seeking support in the various user forums; not accounting for that may cripple the study. Indeed, it is specifically stated, further down, that “It is assumed that Red Hat customers only install patches released by Red Hat…and are taking other similar steps to ensure they comply with their maintenance contract. Similarly, Windows customers only utilize fixes released by Microsoft.”
Correct me if I’m wrong, but that methodology seems guaranteed to maximize “days of risk” for the Linux system, and minimize vulnerabilities discovered and taken care of in the Windows system. If you are interested in reading more about this report, you can check out the full 44 pages here.
Security is only one potential headache to deal with when handling your company’s computing platform. Interestingly, SI tried to measure “IT pain” in the creation of the study released in November 2005. It stated that the pain stems from two scenarios. The first deals with business solutions not being available when you need them – not only uptime for day to day needs, but also adding new capabilities when those business needs change (such as adding personalization to an e-commerce site). The second scenario is familiar to far too many IT professionals: “IT being able to meet business needs only through a combination of ‘heroic efforts’ to overcome unpredictable or failure-prone behavior by technologies,” leading to long hours and/or short term fixes that solve the problem this time, but don’t take the long term into account.
So what sort of metric do you use to measure IT pain? SI discussed this issue with a number of chief officers at companies to understand what factors caused business solutions to fail, and what challenges IT departments faced. SI learned that “the key to managing reliability was to choose platforms and applications that enabled IT to be efficient and facilitated a simpler environment over time.”
The methodology is interesting, because SI wanted to simulate an evolving business as closely as possible, rather than rely on benchmarks that don’t truly test a system as a whole. The firm pitted Windows 2000 Server against SuSE Linux Enterprise Server 8, simulating the year from July 1, 2004, to June 30, 2005. They also simulated an evolving e-commerce company with changing business requirements; security maintenance was also taken into consideration. At the end of the year, both systems were transitioned to more recent versions of their respective operating system.
So what kind of results did SI see? One result was that Linux’s famed modularity and granularity of control led administrators to choose “vastly different paths to resolve dependence conflicts that arose when new components were installed. The result was solutions that grew in complexity and heterogeneity rapidly over time.” Given that the study stated at the outset that simpler systems reduce IT pain, Linux would not receive high marks. Contrast that with the Microsoft side: “During the experiment, all Windows administrators followed a fairly homogeneous route to both install patches and apply component upgrades for the simulated changing business requirements.”
No wonder the study found that the Windows platform is “more consistent, predictable, and easier to manage than Linux.” On average, the Linux administrators were significantly slower than the Microsoft ones in fulfilling business objectives – partly because they experienced more system failures and needed to apply a greater number of patches to their systems. If you would like to examine the 47-page report, you can find it here.
What we can Learn from Two Linux vs. Microsoft Studies - Why Doesn’t it Match My Experience?
(Page 4 of 4 )
When these reports were linked to by Slashdot, particularly the second one, they caused more than a bit of controversy. Many, many Linux administrators chimed in to state their own experiences were much different; indeed, one talked about uptimes in excess of 700 days, including a Linux box that had been running for six years without requiring a reboot. Microsoft boxes, on the other hand, weren’t showing two years of uptime until 2003, and it was very rare at that, according to another reader. A third one pointed out that it might take longer to implement a new procedure on a Linux box, “But here’s the thing: Once it’s done, it’s done…if you count all the down-time and set-backs which can happen after implementation, you probably ultimately save a lot of time by going with a Linux-based enterprise.”
Another Slashdot reader observed that “You can write good as well as bad code both on Linux and Windows, and there are more than enough examples for both on both platforms.” It’s an insightful observation, if not entirely useful. On the other hand, when it comes to pain and ease of installation, another commenter said that “I remote-install and distro-upgrade Linux boxes (imagine updating Win2k to Win2K3 on the run without physical access) routinely. I don’t know of anyone who installs or upgrades MS-Windows that way.”
Of course, this again brings up the whole apples and oranges comparison, and whether using the two platforms is so different that they can’t really be compared. One Linux advocate pointed out that “the core difference in Windows and Linux is that most shops do a LOT more on one Linux/Unix box than one Windows box. Most Windows shops (ours included), have a Windows server for one specific task, perhaps two tasks. Most Linux and Unix boxes run many different tasks and as such you need far less of them.” That offers a level of simplification that the SI report likely did not take into account.
This brings us to an important point. Microsoft is selling a “one size fits all” product, while Linux’s modularity lets the user custom-fit the solution to what the business in question needs. Intuitively, we know that no one solution is the best one for all situations. There may certainly be cases in which Microsoft’s offering is the right answer, but as long as businesses have different needs, it won’t be the answer for everyone.
So if the two systems are so different, despite fulfilling similar functions, why do these studies exist? One Slashdot reader seems to have hit the nail on the head: “I think the flood of Microsoft biased studies in the last year go a long way toward bolstering Linux’s claims. If they weren’t to some extent true, Microsoft wouldn’t be trying so hard to discredit them.” Fortunately for Microsoft users, the company has also focused on improving its software, and indeed, it has become better in the last few years. But, to quote one last commenter, “it just doesn’t matter anymore. They blew it. Linux is here, it’s a lot more flexible, and it’s not going away.”
