Is open source the new monoculture?

Date:May 4, 2004 / year-entry #172
Tags:other
Orig Link:https://blogs.msdn.microsoft.com/oldnewthing/20040504-00/?p=39523
Comments:    63
Summary:Okay I know I'm going to get into a lot of trouble for even bringing up this topic... This past weekend, Ulf Harnhammar discovered two buffer overflow and two directory traversal vulnerabilities in LHA, a library of data compression functions. Since the code for this is public, it has been copied all over the place....

Okay I know I'm going to get into a lot of trouble for even bringing up this topic...

This past weekend, Ulf Harnhammar discovered two buffer overflow and two directory traversal vulnerabilities in LHA, a library of data compression functions.

Since the code for this is public, it has been copied all over the place. At least one commercial archive management program and at least one commercial mail antivirus program are vulnerable. A denial of service attack is already under way against the mail antivirus program; all you have to do is attach a malformed LHA file to a message, causing the scanner to crash when it attempts to scan the attachment. When the administrator restarts the mail server, the scanner will resume where it left off... and crash again. (Somebody with more time on their hands could craft a more clever LHA file attack that takes over the mail server itself.)

The fact that the code itself was public meant that everybody didn't have to write their own LHA functions. This is a good thing. However, it also means that everybody has the same security vulnerabilities. This is a bad thing.

So we have one bug that can take down large numbers of machines, even though they're all running different software.

How do you track all the versions? Is there a solution to this? Is it even a problem?


Comments (63)
  1. Serge Wautier says:

    We develop and sell an industrial automation system with its built-in Web server (No, no embedded Linux. The thingie has 48KB of flash for its OS and it does a damn lot more than simply being a Web server…).

    "How secure is it ?" do clients ask before they connect our devices in the field to a phone line.

    "More than Apache or Windows IIS" we say !

    Of course it doesn’t reach the ankle of IIS or Apache in terms of functionality and reliability.

    Of course it has 10 times more vulnerabilities. BUT, they are not the same. And nobody knows them. So, yes, it’s more secure !

    PS: And, yes, we are switching to existing implementations in our new developments where we have a decent amount of memory and processing power. So, yes, the advantage of re-using existing implementations far outweights the security issue. (at least for us. But our devices don’t control any nuclear reactor.)

  2. Ingo Rammer says:

    Hmmm … but hasn’t this been always a problem? I can fondly remember ping-of-death which IMHO showed this common-source-tree problem for the first time at large.

    -Ingo

  3. Alternatively, would it be possible to make operating systems secure even when they are running insecure code?

    Perhaps, Windows (and other OSs) would be more secure if it was common practice to run as much code as possible in processes with minimal security. For example, instead of Outlook Express using full privileges when rendering HTML and trying to make every line the HTML rendering engine bullet proof, it could create a process with minimal privileges to render the page for it. Then, instead of one dodgy email taking down the whole system, in the worst case it could kill one helper process.

    This is one of the things that really annoys me about Windows. There is lots of very good security code at the core of the operating system to protect processes and the file system and the registry but most of it is either turned off by default or isn’t used as well as it could be.

    Perhaps a lot of the current security problems with Windows could be avoided if it were made extremely easy to do everything at minimal privileges by default. Either a helper library or a manifest file that applications could use to describe the sand box they want to play in. The current system of trying to make every line of every program totally secure seems like more work that it should be.

  4. Anon says:

    Jonathan that’s a pretty cool idea, running Outlook in a sandbox. It’s also so obvious I can’t believe that Microsoft haven’t thought of it.

    Are there any flaws to this approach?

  5. Scott says:

    Heck, if it wasn’t open source, Ulf would never have been able to find the vulnerability!

    Also, when’s the last time anyone has ever seen LHA files? I think I remember them from the BBS days, but that was at least 10-15 years ago.

  6. Anon,

    I don’t claim to have thought of this idea – it is the normal way of running ‘risky’ services (like web servers) on Unix (and possibly Windows).

    I am guessing the reason Microsoft haven’t implemented something like this for more applications is because it would be hard to separate out the parts of each program that need to run in different security contexts (for example, displaying a web page might need no privileges at all while saving the web page to disk would require the ability to write to the target folder).

    But in the same way const correctness makes programming more difficult, it would seem like the right way to go as it forces a more secure design and the boundary points between different security contexts would become far more obvious.

    Jonathan

  7. Scott,

    >> Also, when’s the last time anyone has ever seen LHA files?

    try ‘-atrium filetype:lha’ in google. theres like 300.000 files out there that google knows about :-)

    WM_CHEERS

    thomas woelfer

  8. Eric Lippert says:

    Code Access Security in .NET is about restricting the rights of individual assemblies based on evidence about where they came from.

    > I am guessing the reason Microsoft haven’t implemented something like this for more applications is because it would be hard to separate out the parts of each program that need to run in different security contexts

    We did implement that. GOogle "software restriction policies" and you’ll find the documentation.

    And yes, one of the hard parts of writing that code is dealing with in-proc COM objects that have different rights than the executable calling them.

    SRP is certainly not the beautiful, deeply baked-in wonderfulness of the CLR security system, but it is a step in the direction you’re suggesting.

  9. Mat Hall says:

    On the whole, OSS software is as vulnerable as anything coming from a large closed-source organisation. Although the visible nature of the source may make vulnerabilities easier to find, the fact that anyone who has the time can patch the hole means fix-times can be much shorter. (And as it’s taken an awfully long time for anyone to spot the LHA bug, who’s to say a similar problem doesn’t exist in, say, the CAB extractor that ships with Windows and just hasn’t been spotted yet?)

    (And LHA is still the archive format of preference in the Amiga world. Take a look at ftp://wuarchive.wustl.edu/pub/aminet — there are at last count 59,400 LHAs on there.)

  10. quanta says:

    No, open source is not monoculture. Since the source is open, bug tracking, testing and repair is done collectively by the community.

    Users are not at the mercy of one single entity if a problem occurs. They could conceivably even fix the code themselves (assuming they had the expertise).

    It’s interesting you’ve brought this up, since my entire office (all running Win2k) was nailed by the Sasser worm yesterday. We wasted a whole morning of productivity. There was only one source for the necessary patch – a 7MB file at Microsoft.com – and that site and Windows Update was being slashdotted by world+dog.

  11. James Day says:

    Open source isn’t the new monoculture. It’s just shard code and has comparable vulnerabilities than any other shared code systems, such as the abstration layers provided by an operating system or services running on top of it.

    The open source shared code model has some significant advantages compared to closed source: more eyes and often a lack of need to depend on the support of the vendor, so updates are more likely to be available. Not always, of course. Embedded code is one issue here. The extra eyes are also a potential disadvantage, of course, for the attackers can use the open source code… but that also means that confidence in the lack of serious flaws is likely to rise rapidly as those exploits fail to appear.

    Given the choice I trust open source over closed source. I’ve more confidence in its security, predictability, reliability, cross-platform availability and future availability of fixes. That means that, given a choice between most Microsoft and many non-Microsoft solutions, I’ll often choose the non-Microsoft solution. And where I don’t, the Microsoft solution generally starts out as the loser, with me looking for reasons to favor it. One example was strsafe. Handy idea, looked at the Microsoft solution, used the non-Microsoft solution – I wanted to be sure my code isn’t obsoleted by a change at Microsoft and to be able potentially to use it on other platforms.

    One hurdle for Microsoft in competing for mindshare from me is short product life-cycle and support periods – well under a decade, in general. Then I look at something like Linux and see future support which appears to be and probably is going to be as long as there’s any significant base of hardware to run a particular version, which appears likely to be in the fifty plus year range, in part because groups of users can be expected to gather together to handle key issues long after active development of a particular version ceases. I don’t see much sign of Microsoft trying to address this sort of long term reliability issue with its software platforms.

    This from someone whose primary development target has always been and remains Microsoft platforms. None of this means I don’t have considerable respect for the abilities and quality of Microsoft people and products – both are routinely superb. It’s simply honest feedback on how the mindshare war is being lost. Not trouble… but perhaps unpleasant anyway.

    Now, you did ask some questions. How to track all versions? Does Microsoft want to get into the buiness of trying to get open-source community buy-in to a Microsoft-led standard (you probably know how hard that would be). It’s conceivable to track components used in products down to the function level but I’m not yet aware of anone doing it. And would Microsoft want to disclose that information about its own use of code, some of it open source? It is a problem but I don’t see much prospect of Microsoft solving it. A solution seems more likely from the open source community, which is already used to disclosing which components it uses. I can envisage applications and operating systems coming with component dependency lists for use in automated vulnerability fixes. Which, for some of its own software, Microsoft already does at some levels, with significant benefits for consumers of its products. But not for third party products, which is why I don’t see a comprehensive solution coming from Microsoft, for Microsoft would have to knowingly commit to an infrastructure which supports the products of its competitors – and to do so for a much longer time span than it uses for its own mass market products – into decades. A solution developing from the Linux package management groups seems more likely.

  12. Chris Walker says:

    What would be a good way to decide if some piece of software was a monoculture? One definition would be if a bug was found on one instance in a group, that it was also found in greater than, say, 90 percent of the others. Argue over the percent, but keep it above 30.

    If everyone in one group is running the same version of some *nix OS, then in that case, you definitely have a monoculture. Whether the OS is written under Open Source licenses or not doesn’t change the fact.

    The fact that gurus can get source patches for security bugs fast just means that people who do other things with their lives will be at risk. Studies have shown up to 30% of security patches never get installed.

    Dean Engler at Stanford has been using static analysis to look for bugs. He used Open Source because, well, he has access to it. The error rates he was detecting and the types of errors were no different than what we observed in Windows NT sources.

    The question of tracking source code is an interesting one not only in the Open Source community, but in the not so open community. Code gets copied and pasted and bugs move with them. There is some work being done in this area that is a spin off of work done in academics to find cases of plagiarism. An interesting example of code being a monoculture in itself is the JPEG code. It had some buffer overruns, and this code was in tons of products.

  13. grey says:

    i seem to recall something called package management systems – something windows is completely devoid of.

    shared libraries, applications, configurations – anything – under linux can be controlled through RPM or apt-get

    this being said, an application can REQUIRE such and such version of such and such library. use rpm -q with some choice command line options and bam – you’ve found all the applications that depend on that library. then you just update the stinking library with a patch RPM.

    not that hard. copying code == bad kojo

  14. Mat Hall says:

    "…you’re not paying for the OS so much as you’re paying for them to make sure the OS isn’t terribly vulnerable over the time you own it."

    Unless, of course, you own it past the date the manufacturer has decided to kill it off. At least with OSS software, someone somewhere can keep it alive, but what happens if a new Win95 vuln is discovered? The answer is "nothing" — MS no longer actively support it, and as there’s no way of fixing it yourself you’re effectively screwed. OTOH, if I still wanted to run a really old version of Linux (perhaps for some mission-critical app compatability) and a hole was found, at least I could patch it myself, or find someone to patch it for me…

    Don’t get me wrong, MS have done some wonderful things, but sometimes having the source code can be A Good Thing.

  15. Paul Tyng says:

    Mat Hall "Although the visible nature of the source may make vulnerabilities easier to find, the fact that anyone who has the time can patch the hole means fix-times can be much shorter. (And as it’s taken an awfully long time for anyone to spot the LHA bug, who’s to say a similar problem doesn’t exist in, say, the CAB extractor that ships with Windows and just hasn’t been spotted yet?)"

    Fix times shorter because anyone who has the time can patch and it took an awfully long time for anyone to spot the LHA bug? Hmmm, interesting.

  16. I actually wrote about this here:http://weblogs.asp.net/larryosterman/archive/2004/04/29/123090.aspx

    In my case, I was talking about the zlib compression library, which had a similar vulnerability in it.

    "Many Eyes Make Shallow Bugs" is one of the greatest falacies of the open source movement IMHO. The problem is that now, as a result of this vulnerability, things that you didn’t even know about have vulnerabilities and need to be patched.

    Yesterday I attended my annual security training refresher (yes, Microsoft employees are REQUIRED to go an annual security refresher). One of Michael Howards comments during the refresher was "Beware the Giblets" – the little pieces of code that are shipped as a part of your application that you include and then forget. You need to track all the giblets in your application and understand how you will deal with vulnerabilities in the giblets.

    Unfortunately, sometimes people forget these things.

  17. Scott: "When was the last time anyone saw an LHA file anyway?". I’m willing to bet real money that you’ll be seeing a whole lot more of them in the near future :)

  18. Peter says:

    We may get a "monoculture" in a close source environment as a sideeffect of success. However, in Open Source the "monoculture" is present by design, philisophy and actively promoted.

    P.S. The whole "monoculture" meme is ludicrous. The analogy to the biological concept is so far streched on this it basically reduces semantically to " unspecified bad thing", nothing more)

  19. russ says:

    Jeez…..Microsoft honks.

    As someone else pointed out, a symptom of code-reuse. And open-source != security we can call agree on also but to twist this into monoculture? puleez… to even imply that OS even promotes any kind of unified culture to me is preposterous. It’s a landscape full of rabid change in tact and direction and rampant reinvention of the wheel … but that’s what makes it viable. It’s the mutation and variation that makes it strong. OS to me wins because hands-down because of flexibility. it’s about choice. period.

  20. Rodrigo Strauss says:

    There’s something I would like to know:

    Why everybody thinks twice before saying something bad about open source software?

    Mmmmm… Maybe because everybody knows that there’s an army of fanatics just waiting a chance to start a flame war. :-)

  21. Fess Up when you F-Up says:

    Quanta,

    You said, "It’s interesting you’ve brought this up, since my entire office (all running Win2k) was nailed by the Sasser worm yesterday. We wasted a whole morning of productivity. There was only one source for the necessary patch – a 7MB file at Microsoft.com – and that site and Windows Update was being slashdotted by world+dog."

    Fine. So it’s Microsoft’s fault that you wasted a day trying to get the patch they released THREE WEEKS ago? What were you doing all that time? Talking with your fellow office mates about how bitchin’ open source is instead of paying attention to the message that Win2K pops up when there are new patches to get? My Win2K box tells me when MS has new patches and I install them. I had ZERO problem getting the patches because I did it weeks ago instead of during the crisis. Boy, I guess you’re right. MS is to blame for your lost productivity.

  22. James Day says:

    I wrote "I don’t see much sign of Microsoft trying to address this sort of long term reliability issue with its software platforms".

    That’s unfair to the considerable work Microsoft and its employees do to maintain platform compatibility across releases. I was considering the lifetime of a minimally upgraded platform – that is, from the point of view of a buy it and use it for 10-20+ years situation, where a well-considered business application or appliance may have lifetime of that duration.

    Larry, your post is interesting but open source is better than a closed source system. At least I can use a search tool on the source code to see if I’m vulnerable. It’s really hard to do that on a typical closed source application, so you’re left simply trusting the vendor to do the right thing for a product which may have ceased being supported ten years ago. When was the last time Microsoft shipped a critical security fix for Windows 3.1? Windows 95? Would it ship one today for either? Maybe 95, but Windows Update today doesn’t even deliver critical security fixes for stock Windows 98 systems (it no longer supports IE4), so I doubt it. It’s worse still for products where the company is no longer in business – I’d better have the source code or I’m sunk or relying on binary patches.

  23. Wilhelm Svenselius says:

    In response to "Fess Up when you F-Up" – I couldn’t agree more. This patch has been out for weeks, any Windows systems administrator worth his title would have installed it on all vulnerable systems by now. The ones who haven’t, deserve what they get.

  24. James: "At least I can use a search tool on the source code…."

    But can you really do this? Think about it. There are DAILY vulnerabilities announced on bugtraq. Do you really grep through the ENTIRE source tree of your distro to verify that you’re vulnerable? If so, how do you get your "real" work done? Or do you rely on the distribution owner to do that for you?

    Bottom line: IMHO, there’s no magic bullet here. Open Source isn’t it, neither is Closed Source. We ALL need to be better at security than we used to be.

  25. James Day says:

    Mat,

    Nobody here has yet espoused a socialist philosophy. If you think open source is socialist, you probably don’t understand it well enough. It’s about self-interest. One of the few bits of open source writing I’ve done was for an anti-spam system. It was good but not as good as I wanted, so I wrote some components needed to help improve it to its current level. That some 90,000 other Windows users benefit is not the most important point to me. That it solved the spam problem for me is. In the open source world, there ususally is some suitably skilled person or group with a direct interest in fixing a problem because that problem directly affects them.

    Looking further back in hitory, Richard Stallman created the GPL out of self-interest. He was kind and let a company use something without a fee but they then refused to let him have the changes they made. So his self-interest in getting those changes in the future led him to develop the GPL.

    I’m no expert on Linux history but I think a desire for self-control over the operating system was a significant part in what caused the initial development of Linux. Self-interest again.

    It’s all good capitalist philosophy, though some like a veneer of being good to others on top of it. That’s fine.

  26. James Day says:

    Larry,

    Not me, if I have a choice – there are other things I prefer to do with my time. But I do know that there are others who will and do when it comes to open source software. For me, doing it myself is the last resort.

    For anything I really need I’ll hope to join together with others such that I can persuade others to do the work so I can do other things. I expect most to work that way and rely on packaging groups to do the work if they are using open source or closed source products. Actually getting involved in doing it is a fallback option, not the preferred course.

    My own primary OS distribution owner is currently Microsoft. In my limited involvement with open source development, I’m the developer who verifies that builds work fine with Microsoft development tools and runs the MS-compiled builds full time to ensure it (well, and because that’s convenient for me – self-interest is never far away:)).

    Looking ahead, I don’t expect it to take as long as five years before open source has something more capable than Windows Update. I don’t expect it to be a single vendor solution like Windows Update.

  27. quanta says:

    Fess Up when you F-Up:

    Actually, most of my office doesn’t have admin rights, and our sysadmins generally push updates on a monthly basis. They do need to test them.

    Perhaps the update windows should be tightened. Perhaps the sysadmins should be better trained. We are, after all, only human.

    My only point was to point out that we were dependant on a single entity to bail us out – that’s what most people are talking about when they discuss the dangers of monoculture.

    Also notice how I worded my response without resorting to ad hominem attacks.

  28. Just because something is open-source doesn’t mean it is not commercial. When you buy something from Red Hat, you know they are going to fix when it breaks. How is that different from buying something from a company that sells closed-source software?

  29. Philip Storry says:

    Monocultures seem inevitable.

    In a free market, many companies will soon be reduced (through bankruptcies, mergers and takeovers) to just a handful. This has happened in almost every market.

    And in a closed or managed market (China, we’re looking at you here! Well, you and all the other socialist republics…), you end up with only one or two options anyway – because it would be inefficient to offer many different-but-the-same products, and they’ll go with the economies of scale that one national provider can give them. (More throughput AND more corruption! How can you resist? *grins*)

    Open Source is a completely free market, really. So often, there will only be one or two providers. For instance, although I can name an alternative OSS webserver to Apache, I don’t know how many other people could. I suspect not many, certainly not without resorting to a well-phrased Google search anyway.

    The important difference here is that a new competitor can break through. Apache is not an example I use likely – remember that its very name comes from "a patchy server" – a server created by patching vulnerabilities in an existing server, until it was no longer recognisable from the project that begat it.

    This will happen whenever a platform or program is vulnerable. The free market will ensure that people use it, which means that people will also develop on/for it and (unfortunately) attack it. The openness makes it easier to manage, though.

    Contrast this with a sitaution where the market is dominated by a monopolistic company that attacks those trying to breakthrough by means other than just having a better product – e.g. contractual means, marketing, and so forth. That monoculture will never be as healthy as a free market monoculture which deals purely in technical merit, and for obvious reasons. (I’m not going to say more about that – draw your own conclusions, and could other visitors try not to sound like a raving slashdot squatter if you must post comments in reply…)

    I think that management of the versions is something that the developers must handle. It’s not something that affects OSS any more than it affects closed source software. Remember that worms which hit Microsoft’s database layers affected not only Microsoft’s products, but other companies as well. Companies are using those database layers to store information for backup systems, antivirus systems, systems management software, and more. I’m sure that there are lots of systems with "embedded" SQL servers and the like.

    And the developers of those products have to stay in touch with the maintainers of the tools they’re using.

    OSS fosters communities for thise anyway, and uses tools like mailing lists to announce changes and highlight important bugfixes.

    Microsoft has MSDN, which has fostered an excellent community. But perhaps if they want an edge over OSS, they could consider offering mailing lists which inform of updates for their redistributable tools?

    The onus here is definitely on the developer, but that’s not an excuse for the maintainer of the developer’s tools to make the information hard to find, or to simply stay quiet until asked.

  30. Eric says:

    Jonathan, I don’t know if you’ve looked into .Net’s Code Access Security Features at all, but that sort of sandboxing is built into .Net; you can label specific modules with specific permission requirements (including permissions the code *should not* be allowed to exercise).

    I will be pretty disappointed if the next version of Outlook etc. isn’t in .Net and doesn’t use CAS.

  31. Peter Evans says:

    Boy Raymond,

    You sure opened up the feed trough for the zealots today. Vunerable code is what it is. Its the distribution mechanisms that make it a problem. I don’t think versioning every little code meme is practical. The true solution seems to be the immunities built into the distribution mechanisms. I don’t think the openness of the source is the issue.

  32. Santhosh says:

    Even though the openness of the source is not the issue, it definitley contributed to the problem, as it allowed countless developers to use those components. And I do not believe these kinds of issues will be solved through a future distribution mechansim because many of these ‘clients’ will be ‘closed-source’ communities who may not have interest in the product anymore.

  33. Mat says:

    My only counterpoint to those that pointed out the problems with sasser (one of my laptops got it) is this:

    If it was a bug in *nix, do you think there would have been as simple a fix up as quickly as it was? In fact, since WU prevented it, do you think that a *nix patch would have been available in short notice to prevent something like this?

    You guys pounded me over usability and how people don’t care about what things say; they just want them to work. Well, I challenge your current debate the same way — outside of gov’t, academia, and we nerds, 99.9% of anyone using a computer doesn’t give a flying f-bomb of their OS’s source code’s preference. Whether it’s in the closet or it has come out, no one cares.

    But what they do care about is having someone/company who has a *vested interest* in fixing vulnerabilities. The socialist paradigm was mentioned… so, if there’s a patch that needs to be written for open source stuff, who’s gonna do it? Your IT dept.? Gonna wait till one day you get lucky and google it? I know there’s some support out there for *nix, but I’ll take the resources of a commercial OS maker to watch their code. Microsoft (and Apple!) have that vested interest in fixing things. If you want to look at it my way, you’re not paying for the OS so much as you’re paying for them to make sure the OS isn’t terribly vulnerable over the time you own it.

    </soapbox>

  34. Raymond Chen says:

    Of course the same logic applies to open source. Who knows how many people took advantage of the LHA buffer overrun before Ulf found it?

    My original point was that once a vulnerability is found, you have to track down everybody who copied the source code and patch them. Some companies may be slower to issue a patch than others. And open source means that a vulnerability in commonly-copied code may never be eradicated completely.

  35. Actually, there was a fascinating article on bugtraq today about this:

    http://www.securityfocus.com/archive/1/362109/2004-05-01/2004-05-07/0

    Mark Maifrett of eEye comments in the post that nowadays, virus companies (and other security companies) watch things like processes crashing on machines so closely that they can quite quickly diagnose that there’s a new worm on the loose. For sasser, for example, the posts about people starting to see probes on the affected port started coming in on Friday (or maybe saturday) when the worm was first released.

    So it’s pretty likely that the worm was really released when people think it was.

  36. Philip Storry says:

    *sighs*

    I knew it. Raymond was right – there would be trouble over this topic. :-)

    An interesting idea struck me, though. People have mentioned WU and *NIX package managers, saying that these help users ensure that everything’s up to date.

    But that’s not the side of the fence we’re necessarily interested in. Sure, it works if you’re using a shared library. But not if you’ve embedded the code into your app. Developers have been known to do either.

    So how about a package management system for developers? A CVS/RCS system that can actually connect to remote systems, and check for updates for you. RSS could be used to allow this – you can "subscribe" to the libraries you want. It fetches the sources remotely, and caches them locally, and you’re away – you simply build from the local cache. An update command checks all your remote dependancies for updates, and downloads them for you. And it should have a little flag that says "don’t build", so that it a fix is pending but not yet available you can be made aware when the update occurs, and choose yourself whether you want to build anyway.

    Hmmm. I gotta get me to a patent office. ;-)

  37. Brian Smith says:

    Edward: I dont know if you are new to the internet or not, but if Microsoft was to release a patch without doing thurough testing to make sure that the patch worked on numerous different platforms under many different hardware configurations, the OSS crowd would go crazy.

  38. asdf says:

    Fess Up when you F-Up – because sometimes people don’t download all the patches early because of things like:

    http://support.microsoft.com/default.aspx?scid=kb;EN-US;841382

    http://www.eweek.com/article2/0,1759,1578752,00.asp

  39. Mat says:

    James, well worded. I don’t mean to sound ingorant of the benefits of writing a program. Your spam stuff is probably wonderful, and I bet I’d even use it. However, my own point is that this is apples and oranges.

    Security is different than a regular program. We all know this, but let me state the difference anyhow. When building a program that will generally accomplish a task, open source can be great. It allows collaboration to the Nth degree. When you’re dealing with security, it allows exposure to the Nth degree. That’s a failing of open, /by its own definition/ it’s not a good thing.

    Open source, everyone can try to fix problems, and everyone can try to find weaknesses.

    Closed source, only those making it can try to fix problems, and it’s harder to find weaknesses.

    On balance, speaking on security ONLY, I’ll take closed. Ask anyone in sports, war, whatever — the bluff and the unknown are securities themselves. American football players conceal their play until its running. Why? If the opposing team knows where the ball is going, it’s easy to attack that point and stop it. And 1000 more analogies.

    For programs, it very much is a question of capitalism, James. For OS’s, I think this is the appropriate frame.

  40. Andreas Häber says:

    Phillip Storry: Nice idea, but it’s already implemented (sorry if I’ve missed an important point!!).

    As you may know in products such as MS Visual Sourcesafe you have the same thing *locally* with the ‘share project’ feature. Same for CVS. Now, as I understood you you

    But other SCM products in the market seems to do this. After a quick google for ‘clearcase distributed'(I assumed Rational ClearCase would have support for something like this since it’s quite expensive :)) I found this document[1]. Here you’ll see this paragraph:

    "The ClearCase multisite feature is an attempt to decentralize a centralized system and it doesn’t work as well as a truly distributed system. The basic idea is that each site gets a branch, that branch is writable by that site only, the other site’s branches are read only. BitKeeper has no such restrictions, all sites can work on the same branch at the same time. "

    [1] http://www.bitkeeper.com/Comparisons.ClearCase.html

  41. Scott says:

    If you had a security bug in VBRUN300.DLL, how are all the programs that shipped with it supposed to get updates?

  42. Philip Storry says:

    Andreas Häber> I think that the tools you’re pointing at are meant more for internal development, albeit across multiple projects. However, I could be wrong. :-)

    Part of what I was talking about is a community thing, really. A technological solution is only half-way there – project maintainers need to produce some kind of "update feed" (which is why I suggested using tweaked RSS to deliver notifications of updates/changes), so that people can be aware of what’s going on – an automated way for people to check their giblets, as Mr. Osterman would say. :-)

    However, those tools you pointed at are a step in the right direction, albeit an expensive one. ;-)

    James Day> "Looking ahead, I don’t expect it to take as long as five years before open source has something more capable than Windows Update."

    Thanks for the laugh that gave me. :-)

    In my personal opinion, Windows Update is seriously lacking. Try the following simple experiment to see what I mean:

    Install a copy of a slightly older OS, and update it.

    If you do that with a Linux distribution, then either rpm or apt-get will almost certainly be able to update every bit of software on the machine in one hit. A large hit, but just one. So you’ll only need one restart (or change of runlevels!) to get things going with new, shiny code.

    If you do that with Windows Update, you’re in for a world of pain. I did it a short while ago with Windows 2000 Server. The fastest way to accomplish it is to unselect everything except the Windows Service Pack, and install that – the reboot. Then hit Windows Update again, and deselect everything except the latest Internet Explorer update – install that, and you’ll probably have to reboot again. Now, you’re probably looking at just 30 or 40 patches to install. Unfortunately, a couple of them conflict with each other – if you try to install them all at once, they’ll fail. This isn’t noted, and could just be that the two patches were never tested in this was and marked for exclusive install. So you’ve probably got one or two reboots ahead of you yet,as you slowly work your way through each one that failed.

    By the way, all that unselecting is manual. There’s no unselect all button that I’ve seen (On Win2K Server WU), so you’re going to be clicking on each and very update – probably about 70 of them for the first run, and going down slowly with each subsequent run.

    And after all of that, you’ve only updated Windows. Not any of your apps.

    rpm/apt-get aren’t perfect. But the idea that Windows Update is better just doesn’t fit with my experience – I’d say that Windows Update might catch up with rpm/apt-get in five years time, not the other way around. ;-)

    Their main advantage, of course, is that they’re package managers. In the case of apt-get on a debian system, it can often know everything about your computer’s software setup. So it can ensure that it gets all dependencies sorted out beforehand – and that it only updates dependancies once, before it updates the packages that have those dependancies. Package managers make the whole process pretty easy, and I’d rather rely upon apt-get or YaST’s auto-updater than Windows Update.

    To be fair, Windows Update’s troubles aren’t necessarily its fault – the nature of the patches it has to install are partly to blame. (Boy, was I relieved to find that the latest MDAC patch didn’t require a reboot! Rejoice!)

    But you’d think that Windows Update could at least spot when a service pack is in the list, and automatically select only that one item for you. Especially as it actually has it marked as being an exclusive install item – it just won’t let you install the Service Packs and other patches at the same time anyway. Windows Update’s interface and logic when selecting updates are severely flawed, IMHO.

  43. Scott: If a security bug’s found in vbrun100.dll, then it’ll probably show up on WindowsUpdate. Not a perfect solution, but…

    However, vbrun100.dll’s a giblet (see http://weblogs.asp.net/larryosterman/archive/2004/05/04/126054.aspx), and part of your responsibility when using a giblet is that you need to understand how to update them.

    This is true if the giblet is vbrun100.dll, or if the giblet is LHA, or the giblet is openssl, or the giblet is zlib or the giblet is the SQL desktop engine. Regardless of the giblet, you need to understand what your strategy is to deal with security bugs in your giblets.

    CAVEAT: I don’t know if Microsoft would ship an update to vbrun100.dll via windows update. I don’t even know if it’s even feasable to do it. Do NOT under sny circumstances take this comment as a commitment to put vbrun100.dll on windows update in the event that a security hole is found in vbrun100.dll. Instead consult the MSDN documentation to determine how this is handled (and no, I don’t know where that would be, sorry :().

  44. Oops: s/vbrun100/vbrun300/g

  45. Jeremy Morton says:

    But you’d think that Windows Update could at least spot when a service pack is in the list, and automatically select only that one item for you. Especially as it actually has it marked as being an exclusive install item – it just won’t let you install the Service Packs and other patches at the same time anyway.

    Philip, my experience with WU, is that if everything is selected with an exclusive item and you click Install, it will say that one of the items is an exclusive one; if you choose to continue, it will only install the exclusive one. As such, you don’t need to unselect updates unless you don’t want to ever install them.

  46. Edward says:

    Mat "If it was a bug in *nix, do you think there would have been as simple a fix up as quickly as it was? In fact, since WU prevented it, do you think that a *nix patch would have been available in short notice to prevent something like this?"

    I don’t know what you think of as short notice but by all accounts this vulnerability was discovered by eEye months ago and reported to Microsoft and agreed to keep quiet until they had written and regression tested a patch.

    http://www.eeye.com/html/Press/PR20040413.html

    Who knows if anyone else discovered this problem, developed some expliot code that didn’t crash LSASS and then hacked into any number of 2k and XP boxes without anyone noticing?

    They might have corrupted or stolen valuable data and the businesses running windows would not have known that they were even at risk.

    You trust Microsoft to keep patching because its in their business interest, but isn’t it in your business interest to make sure you are as secure as possible, regardless of whether Microsoft decides it can admit to it’s failings.

  47. Anon says:

    If you’d like more info on how Linux can do this without apt-get check out http://www.ximian.com/products/redcarpet/features.html

    Although obviously this is now part of Novell.

  48. Karl Barrus says:

    <i>But what they do care about is having someone/company who has a *vested interest* in fixing vulnerabilities. The socialist paradigm was mentioned… so, if there’s a patch that needs to be written for open source stuff, who’s gonna do it?</i>

    This is fine and dandy in theory, that companies have a vested interest, blah blah blah, but in reality it doesn’t work out like that. Check debian or gentoo – there are fixes ALL THE TIME for small issues. Stuff that in many ways aren’t even problems – issues that can’t be exploited because they are completely theoretical in nature, etc. This stuff gets fixed because somebody feels it reflects on them personally that it isn’t fixed.

    Contrast that sense of responsibility with fixes from a big company (i.e. Microsoft). Somebody ultimately makes the bean-counting call whether it is better to fix it and possibly break things, or let it go. Thus, only the high-risk items get fixed.

  49. Peter Torr says:

    asdf: But there are also several work-arounds listed for the LSASS vulnerability (being used by Sasser), such as:

    * Create a read-only text file in the windowsdebug directory

    * Block affected ports with a firewall (which you should be doing anyway)

    Granted the second one won’t help if a "rogue laptop" enters your system, but the first one is pretty easy to do relative to the clean-up costs.

  50. "But what they do care about is having someone/company who has a *vested interest* in fixing vulnerabilities."

    Of course, Microsoft has no legal responsibility to fix vulnerabilities; read your EULAs. It may have PR or professional pride interests in fixing them, but certainly isn’t something I’d describe as "vested".

  51. Tomas says:

    Serge Wautier: "Of course it has 10 times more vulnerabilities. BUT, they are not the same. And nobody knows them. So, yes, it’s more secure !"

    That’s almost the same as saying: As long as nobody uses it (and runs into those vulnerabilities), it’s secure!. :-)

    I think you’re maybe confusing obscurity with security. A webserver on a 48K machine is cool though :-)

  52. Philip Storry says:

    Jeremy> I’ve never tried that, I must admit.

    But what about when there’s more than one exclusive install item? That’s what happens at the start of my scenario. Windows Update doesn’t seem intelligent enough to handle it gracefully and obviously. :-(

  53. Skywing says:

    "One hurdle for Microsoft in competing for mindshare from me is short product life-cycle and support periods – well under a decade, in general. Then I look at something like Linux and see future support which appears to be and probably is going to be as long as there’s any significant base of hardware to run a particular version, which appears likely to be in the fifty plus year range"

    I don’t think this is very accurate. How much support is there for Linux 1.0, or even Linux 2.0 today?

    More than that, most of the Linux desktop vendors seem to provide or have historically provided far shorter product licecycles than Microsoft (e.g. Red Hat).

    Now, if you say that you can just upgrade to a new version of the Linux kernel instead of sticking with, say, 2.0 – that’s not really so different from upgrading to the next Windows version.

  54. Fess Up when you F-Up says:

    ADSF:

    Fear of patching early because you’re afraid the patch might be problematic is not an issue of open versus closed source. You can have a bad patch in either case. Please don’t talk about "shallow bugs" and "IF you have the software chops to read/fix the code" as reasons for open source greatness. I get bug reports from Redhat pretty much every week. Who cares if the patch is open or closed source? People either trust the patch or they don’t. I’m always told that open source folks can turn a patch around in hours, hence it’s super keen to use open source. Does this mean that since the patch was was written and released in hours (with all the testing that implies) I’m supposed to trust it more so I won’t wait weeks to install it? I can guarantee you that I personally will trust it less. I have yet to see anyone who can give a percentage (based on anything other than random thoughts and hopes) of the people who read and comprehend the source code before applying the Linux patch.

    QUANTA:

    Glad to see you starting to take some responsibility for your share of the blame instead of the crying about the inherent unfairness of MS and it’s mighty monopoly. It’s shows solid growth on your behalf. See? No ad hominem attack this time. I praised you. Funny how it works. Do something (eg "take responsibility") to show you deserve some respect and, lo and behold, you get some.

    Feel better now?

  55. Cesar Eduardo Barros says:

    Skywing:

    The 2.0 branch of the linux kernel is still being actively maintained. Look at http://www.kernel.org/ and you’ll see the latest version of the 2.0 kernel is 2.0.40, which was released 2004-02-08. Since 2.0 is from 1996-06-08, it’s almost eight years. Look at http://www.kernel.org/pub/linux/kernel/v2.0/ to see more dates.

    As long as there are enough people interested in maintaining something, someone will maintain it.

  56. Anon says:

    Fess-up:

    "IF you have the software chops to read/fix the code" as reasons for open source greatness.

    The whole point of this is not that *I* (individual) need the chops, it is that I can pay *anyone* who does have them to make changes for me (as a company, say).

    And to me that creates a little more competitiveness in some market places – which everyone agrees is a good thing, no?

  57. Open Source is the new Monoculture? No – this applies more to one or two small, widely used libraries (LHA and ZLib so far — funny that they’re both compression libs) than to Open Source in general.

    The premise in the question is bogus. You might as well ask "is generic-X the new monoculture?".

    The problem is more that these libs were written for use in another context, but have been adapted and re-used in a different context. (Dig the "giblets" idea though — very useful concept to keep in mind).

    As several people have pointed out, patch management is the bigger problem here – and patches on patches open up new and exciting opportunities in multi-dimensional matrix computations for the test team.

    Didn’t know you now have yearly security refreshers. Good for you.

  58. quanta says:

    "Glad to see you starting to take some responsibility for your share of the blame instead of the crying about the inherent unfairness of MS and it’s mighty monopoly. It’s shows solid growth on your behalf. See? No ad hominem attack this time. I praised you. Funny how it works."

    I am as unimpressed with your previous personal attacks as I am of your condescending "praise".

    I did not ask for either.

  59. Quantam says:

    I’m going to have to agree with Fess Up, as loud as he is. Think about this, logically. If it’s the sysadmin’s job to update Windows, whose job is it to update Linux, or something else that is open source?

    The fact that just anybody can make patches for open source software means that the patches are that much more likely to be bad than [insert MS product name here] patches, where MS has financial motive (we all know how greedy MS is, really) to make patches that aren’t bad. Obviously, even still there will be bad patches from MS; they’ll just be few and far between.

    This lower degree of reliability of open-source patches means that they will have to be tested by your sysadmins for even longer periods before they can be used, making it even more likely that some worm or virus will take advantage of some hole that’s in a patch you haven’t got installed weeks after it was released.

    Of course, this is all assuming that your sysadmins know what they’re doing, and are not delaying installing patches due to their own laziness/cluelessness. In this case, Windows would stand even further above any open-source OS (although the principle holds true for things other than just operating systems), as clueless sysadmins would be more likely to install bad patches, as the percentage of bad open-source patches are higher.

    But in either case, it’s the sysadmins’ job to ensure that systems are working correctly. Granted, if Microsoft releases a bad patch that cripples the system there’s nothing they can do about it. But it is the sysadmins’ job to weigh the risk of a bad patch and the harm it may do against the potential for harm from the thing the patch fixes. In the last 9 years of using Windows, I’ve never had a ‘bad’ patch be more than a nuisance until a later patch fixed it. Your office getting hit so hard by this virus was your overly-cautious/lazy/clueless sysadmins’ fault, and nothing more.

  60. Ed says:

    When Microsoft releases a patch, they tend to fix dozens of things; sometimes hundreds.

    When the OS world rushes out a patch, it fixes one thing – the reported security hole.

    It’s much more feasible to test a five-line patch that targets one specific problem in an hour than it is to test 15 fixes to unrelated components in a month, especially since there underlying code base may change during the time of that testing.

    Every time I have to do an upgrade due to a security hole, I perform a code review on that update before I install it.

    Most of the time, the change is to add an additional boundary condition check, with an error. If the boundary condition fails, it behaves almost exactly the same as another potential boundary condition failure near that location in the code. The only ways these could possibly blow up are ‘wrong boundary condition’ and ‘calling code assumes there will never be a new error code’ (which is poor design on the third party app’s part.)

    Much of the time when it’s not a ‘boundary check for a new error code’, the change is to add an additional boundary condition check, with correction code. Any software that uses the function will not notice any change, so long as the correction code is correct.

    Most of the bugs I’ve heard about having been introduced in Microsoft patches appear to be completely unrelated to the actual security hole fixed; rather, they’re part of the other code. Buffer overruns are easy to fix – just check the size of what you’re putting into the buffer first, and react accordingly.

    The problem really isn’t that Microsoft is closed source. The problem is that Microsoft doesn’t use good software practices for security patch releases.

    Microsoft does have some pretty good motivation to do this – they’re continually working on new features; if they did patches the way the OS community does them, they’d need to apply them to their source tree at each stable version, and on the development version, same as the OS community does. However, that’s a lot of additional work, and it’s much easier to just fix the dev version and ship it. After all, it’s not their applications that die – after all, they identify situations where that happens, and include updates for those apps in the update (adding in a few more scope creep items in the process. repeat.)

    Open Source is better than closed source merely because it attracts capable programmers motivated by desire for fame, instead of people motivated by greed. If peer review does not happen in the development process, it generally doesn’t happen sufficiently – I review all the security patches I apply, but I don’t have time to review all of the feature patches I apply.

  61. Peter Lund says:

    Chris Walker: /Dawson/ Engler, not /Dean/ Engler.

    -Peter

    PS: I think Raymond will find the papers on Exokernels he cowrote with Frans Kaashoek et al interesting…

  62. Raymond Chen says:

    Commenting on this topic has been closed.

Comments are closed.


*DISCLAIMER: I DO NOT OWN THIS CONTENT. If you are the owner and would like it removed, please contact me. The content herein is an archived reproduction of entries from Raymond Chen's "Old New Thing" Blog (most recent link is here). It may have slight formatting modifications for consistency and to improve readability.

WHY DID I DUPLICATE THIS CONTENT HERE? Let me first say this site has never had anything to sell and has never shown ads of any kind. I have nothing monetarily to gain by duplicating content here. Because I had made my own local copy of this content throughout the years, for ease of using tools like grep, I decided to put it online after I discovered some of the original content previously and publicly available, had disappeared approximately early to mid 2019. At the same time, I present the content in an easily accessible theme-agnostic way.

The information provided by Raymond's blog is, for all practical purposes, more authoritative on Windows Development than Microsoft's own MSDN documentation and should be considered supplemental reading to that documentation. The wealth of missing details provided by this blog that Microsoft could not or did not document about Windows over the years is vital enough, many would agree an online "backup" of these details is a necessary endeavor. Specifics include:

<-- Back to Old New Thing Archive Index