| Date: | November 15, 2005 / year-entry #349 |
| Tags: | other |
| Orig Link: | https://blogs.msdn.microsoft.com/oldnewthing/20051115-13/?p=33333 |
| Comments: | 42 |
| Summary: | When you install Windows on a machine that already has a copy of Windows, you get the option of performing a "clean" install (installing the operating system from scratch, ignoring the existing one) or an "upgrade" install (upgrading the existing operating system to the new one). And in real life, it turns out, surveys show... |
When you install Windows on a machine that already has a copy of Windows, you get the option of performing a "clean" install (installing the operating system from scratch, ignoring the existing one) or an "upgrade" install (upgrading the existing operating system to the new one). And in real life, it turns out, surveys show that nobody upgrades their servers. Really. Sure, IT administrators may get new hardware, they may upgrade the software, they may even install a new version of the operating system, but our research says that, to within experimental error, nobody does an "upgrade install". They always do a "clean install" and reinstall any applications that they need. Since this is a server, there aren't many applications that need to be installed, so reinstalling all the applications doesn't take quite so long. What is the lesson here? You have to dogfood intelligently. If you're self-hosted on a server build and you install the latest build via an "upgrade install", you're not really helping to test setup, since no real customer does what you're doing. This type of disconnect has historically been a problem in the operating systems group, where people typically run through the interactive setup (or if they're really clever, hand it an unattend script to answer the questions automatically), even though research shows that less than ten percent of all Windows systems are set up that way. The vast majority of the systems are installed via a process known as Sysprep. But inside Microsoft, very few people use this Sysprep install mechanism. Result: More attention is paid to improving the quality of manual setup at the expense of Sysprep because more internal bugs are filed against manual setup than against Sysprep. Even though Sysprep method is more commonly used in real life. In other words, we failed to dogfood setup. Fortunately, with Windows Vista, the setup team recognized this disconnect between what gets used at Microsoft and what gets used in the real world and attacked it head-on: The two setup models have been merged into one. (And, if you count "clean install" and "upgrade install" as two different models, which they sort of are, then in fact three setup models have merged to one.) The three types of setup (mass-produced, manual clean install, and manual upgrade install) are now just minor variations on a single engine. The dogfood is back. (Related: Why is there no all-encompassing superset version of Windows?) |
Comments (42)
Comments are closed. |
We do it because if you install over most installers just plain suck and if one thing is not right it will finish and not work, or you have mixed versioned files, or if it fails and rolls back some of the stuff you tried to do screws up somthing else by leaving some component of the newer version installed. Better to start clean and work up, less hassle and better restults. Less downtime cause if something screws up on an upgrade install you gotta troubleshoot it and many 24/7 servers cost big bucks to be down for extended time.
I agree with typhoon, I tried an upgrade from 3.5 to NT 4, server was so hosed spent a week trying to trouble shoot, I ended up reformatting. Tried 4.0 to 2000, this time I called microsoft support as we had it, it took 2 days to get a response back, which then they had to send me to an engineer, which took 2 days, then I had to compose all kinds of logs and dumps. Finally after a week of midnight support calls for me to reboot the server and so on. I came in that weekend reformatted and reloaded. That Monday an engineer finally got back to me to try something, but it was now too late. Server 2003 comes around, reformat reload, no problems never even attempt an upgrade.
I have tried desktop system and stuff as well, even 2000 pro to XP pro had wierd problems like File Save in any app would work, File save as on any app using common dialogs would crash the app with work unsaved. They bounced me from OS support to Office suport on that one for a while until I just reloaded. Anyway, I guess through the years I have learned, you never ever ever upgrade a Microsoft OS, it causes more problems than a fresh install. That will be the challenge I think for MS to overcome. When Vista rolls around, I am not daring enough to trust it to upgrade when I have yet to see an upgrade work.
if installation is meant to match modern reality, why didnt the vista beta install onto clean hardware, or better yet, ship with vmware and VPC images?
Nowadays the primary means of testing an unstable OS is not "find a scratch machine to install it on" but "lets host it under a new VM".
This is true – but only for Windows. We routinely upgrade Linux and OS X systems because the process is reliable; my personal experience, and that of the other Windows admins I know has been that people like the theory of upgrading Windows but have been so badly burned by systems which developed bizarre instabilities that they’ve given up on the process.
There are two problems which make Windows so much more expensive to manage (at least in our environment): one is that the Windows Installer is harder to get right than, say, the Debian package management system which makes install automation much harder than it should be. That’s a relatively minor problem, however, as the real killer is that flawed as MSI may be it’s far better than the demented installers (or extremely poorly-designed MSI packages if they do use them) most vendors insist on using which tend to be very poorly tested and fail as soon as you venture outside of the extremely limited QA process small companies like Adobe or Corel can afford.
I don’t trust upgrad installs for any OS – server or desktop – I always perform a clean install. I run a lot of apps, but I have all my software installers on a network share, so it is quick and easy to re-install all the apps on my workstation…even with the large number of apps that I run.
Is dogfood now a verb?
"We routinely upgrade Linux and OS X systems because the process is reliable"
Er-HUH? I’ve been burned by RPM & apt-get often enough. They’re ok for a package or two, but not for going from maybe RH 5.0 to 6.0
All installers suck, whether it’s UNIX or Windows.
I figure if it’s going to step on all my customizations anyway, I might as well install from scratch. And from source.
Heck, the system I’m posting from is so old it doesn’t even have RPM anymore, I just install from source tarballs.
(Granted it took me 5-1/2 hours this weekend to figure out how to make hotplug and udevd play well together to automount a USB stick, but think of all the time I’ll save not having to type "mount /mnt/stick" any more!! :-) but it was fun finding out and I know my system better. I discovered 3 config files, a set of utilities, and an important directory I didn’t know about.)
Sysprep is the dominant setup for Windows in general, not for servers in particular. I used the server example as a lead-in to the dogfood angle (the shift in focus occurred at "What is the lesson here?")
"Why didn’t Vista beta ship as VPC images?" Because that completely misses the point of my article. Shipping as VPCs would be testing the 0% case. The purpose of a beta test is not to "find bugs in the beta installer". The purpose of a beta is to "find bugs in the final product". And the final product does not ship as VPCs. Shipping the beta as VPCs means that no bugs in the installer would be found.
I think this is because upgrading over windows is concidered by most to be "A bad idea"(tm).
"Is dogfood now a verb?"
These days, people often shortcut their language usage by verbing nouns – it helps them to showcase their thoughts :-).
In the old times, this was not true. You can check the Bob Supnik paper "Bug, Feature, or Code Rot? Adventures in OS Debugging" (http://simh.trailing-edge.com/docs/bugfeature.pdf). Many tentatives to do a clean install of old operating systems like TENEX or ITS on hardware simulators aborted.
The later builds of these OS were never tried to be clean installed, as new hardware was not available anymore.
I’d be happy if the Vista installer was smarter about USB devices. I tried installing Windows XP Pro using a REV drive on a legacy-free computer with no CD or floppy, and I found it impossible. Sure, you can put the XP ISO on the Rev disk, sure the computer boots from the Rev drive just fine, heck, Setup even formats the drive perfectly and copies all its files. Then it reboots and all hell breaks loose.
For some reason, Setup, which was running PERFECTLY FINE off a Rev drive 30 seconds ago, now has absolutely no knowledge of USB. It doesn’t assign the Rev drive a letter, so when it asks you to find asetupfile.cab (or whatever), you’re completely sunk.
We ended up having to buy a $50 CD drive because Windows XP Setup loads USB drivers sometimes, and then sometimes it doesn’t. What an ass-pain that was.
I think Calvin and Hobbes said it best: "Verbing words weirds language."
Another example of upgrade problems: If you upgrade from Win2K Pro to Win XP Pro then no users can Remote Desktop in. It looks like everything is setup correctly but doesn’t work. Doing enough Google searching eventually leads to a knowledgebase article with some settings to tweak. The time saved by doing an upgrade was wasted by troubleshooting things that should work but don’t.
The biggest problem I have is that new releases contain many new features. If you do an upgrade install you typically aren’t asked about these features and only end up with the features you already had upgraded.
Yes, I am the person who always chooses "Custom" when installing software ….
Uh oh, looks like there are two "Bryan"s running around here. I’d probably better change my name to (hopefully) disambiguate.
Anyway. In my experience helping fix Linux problems (at justlinux.com), it is true that many distros don’t easily upgrade from one version to another. RPM-based distros, in particular, are crappy at this. Windows seems to fare about as well as the RPM distros at this particular task.
However, apt-get seems to be able to handle it OK, as long as it doesn’t have to install too many updates at any one time. If the upgrade happens over several runs of apt-get, it’s been OK so far.
But I haven’t tried upgrading from stable to testing (for instance) in one shot, so maybe that’s just as broken as RPMs and Windows.
What I want to eventually find out (and I probably will — in 2007 or 2008, once we get some Vista media here to play with… we always seem to be at least a year or two behind) is how the whole "mass-produced setup" thing works. Does it still require a sysprep tool?
A few months ago my sister’s Windows Me somehow got hosed (I’m surprised it lasted this long!), and I had to help her install XP over the phone. We had to do an upgrade in order to not lose any of her stuff.
Surprisingly, it worked without a hitch. But, once I get a chance to fly down to her place with a spare hard drive, I’m going to copy off all her stuff and do a clean install. Even though things appear to be OK, I definitely wouldn’t trust a Me-to-XP upgrade for long term.
Besides the fact that I want to get her set up with a nonadmin account (when you upgrade from 9x, your 9x account is always made into an adminstrator).
I do "upgrades".
But only with same OS version as installed ;-)
Sometimes this is the only way to get server back to life for a while (to move data and format / reinstall ;-)
I upgraded at least 10 times from 2000 server to 2003 server, including online production servers. With no issues…
Oh my, what I just knocked my head against. That makes quite a lot of sense in terms of sysprep that MS isn’t using it internally.
Sysprep is seemingly an under-documented utility that makes life so much easier for a number of us. I wish more of it was made public – like what does it really do under the covers to "sysprep" (verbifying I guess. =P) an image aside from changing the SID? Or why is it that I must be careful to delete hardware or else be left with ghosted references to them. Why is there support for discovering some drivers but not others…. the list goes on.
As for clean installs, that’s me the whole way. I don’t believe in most upgrades…perhaps its only because I’ve never had any luck.
In practice Sysprep is 90% there, with the other 10% being a cumbersome setup and a support structure that is tricky at best. Did you know that Sysprep will reboot infinitely if the telephony service is disabled? AND sysprep doesn’t check this before syspreping. Also sysprep queries the SMB lanmanserver service to enumerate the OS version. WHY??? I can think of 15 different methods that are better, and don’t require a service that Sysprep doesn’t use as intended.
I believe Sysprep would be a wonderful candidate for Open Source, similar to WIX (even if Vista fixes this by adding features to it’s install, the Windows Vista release is a long time away, never mind the time it will take for corporate infrastructure and end users to migrate… a realistic time frame may be 2.5-3 years from now before adoption is at a reasonable level vs. XP). This is a utility that many people will use, Microsoft will continue giving away, and truthfully people outside Microsoft are more interested in seeing work right.
Eating one’s own dogfood is a good thing for everyone. Sadly, "hundenahrung" doesn’t roll off the tongue quite as easily.
I’d have to disagree on the Linux scenario. The primary reason everyone recommends that you install /var and /usr on different partitions is so that doing a clean install to upgrade is convienent.
Sysprep would be one of the WORST Microsoft programs to open source.
It does too much low level voodoo to open source. Raymond would have blog entries (and nightmares) for the next decade if third party developers started copying "good" ideas from an open sysprep.
The reason I always do a clean install (server or desktop) is that I keep the old boot disk as insurance in case the upgrade fails. If anything goes wrong, or we run out of time, or whatever, we just boot it off the old disk and we’re exactly back where we started. Disks are cheap compared to my time. Although now I’m using mirrored boot disks in most boxes I can afford to risk one of them, I still haven’t tried the upgrade approach.
Mind you, I’m a custom install fan, and upgrade relatively infrequently, so there tend to be now program versions too. Vista will be like this I think – so many things will have a Vista version that it’s probably not going to be worth upgrading the OS, then reinstalling/upgrading all my software.
I’m pretty curious about this – although it hasn’t been explicitly said, the article seems to suggest that Sysprep is the dominant method of setting up Windows *servers*.
Is that really true? My own experience is that Sysprep/Ghost (and Ghost-like) methods are extremely common with desktop systems, but *not* so common with server systems, which tend to be installed via the manual install or unattended methods.
Or am I in the 10%?
(FWIW, I totally agree that no server admin I know would trust an upgrade install for production use – we’ve all been burned at least once by upgrade installs! It’s possible that upgrades work great by now, but server admins tend to be pretty cautious, once burned twice shy kind of people.)
Windows…I only upgrade when it’s on top of the current version.
Otherwise, everything gets wiped. A plain install is just easier, keeps the cruft that windows installs too often get clean.
Ever tried to do surgery on a registry after an upgrade? *shudder*
Open BSD is reinstalled–site.tgz is awesome for that. ;)
The Ubuntu boxes I have, however, get apt-get dist-upgrade.
I’ve had a dist-upgrade work so well it fixed locale information I accidentally destroyed (and I’m fairly sure I had the option to keep my broken settings)
Personally, i do not upgrade on desktop because of third party applications. And the other reason is that, using automated installs, along with AD or SMS based software deployment makes it much simpler and less time consuming than any upgrade.
On the servers though, it is almost always paranoia, from 2K to 2K3 it was IIS settings, certified drivers from dell (in my case), recommendation from server manafacturers to upgrade firmware, drivers etc. ahead of time. Which almost always make the upgrade more complicated. AND most of all, it is easier to transfer settings/permissions/scripts than to fix them when an unexpected failure occurs.
"We routinely upgrade Linux and OS X systems because the process is reliable"
There’s been a Linux reply, so here’s an OS X one. I just don’t trust simple upgrades either – not that I’ve even tried them, they just give me a bad feeling! However, OS X has the really nice ‘Archive and Install’ option. It works like this: all the non-OS data is left alone; the old OS stuff is moved to a backup folder; the new OS is clean installed. The pleasing result is: all your users are still there and untouched; the OS is virgin clean and all those little customisations you made to various config files are automatically backed up, letting you restore your tweaks as need be.
We don’t upgrade because upgrades don’t work!
Operating system developers need to spend more time testing upgrades, not less.
Matt – There is hardly voodoo magic in Sysprep. Sysinternals even has a similar tool to Sysprep for changing SIDs that has full source available, except it really isn’t geared for Sysprep’s level of system cloning. If you think open source projects need to be simple, please refer to emacs, the linux kernel, and even WIX. There are open source shell replacements for windows that have nearly 100% coverage of explorer’s features while adding new features of their own. I reiterate sysprep would be a wonderful open source candidate.
As always, this was a well-reasoned and well-written article. I have one small issue with Windows server installs in general: why is it so impossible to boot the installer over the network? Analogous processes would be jumpstarting a Solaris box or netbooting the OpenBSD installer
Microsoft employees, in their "doing IT our way" road show, described how every production server has a PE cd inserted for building and recovery (and why is PE so underdocumented?).
i hate this god damn windows anyway, i cant customize it, becouse some jerk decided that probably i would need some features, for example
SendTo Menu -> Send to Floppy
if you’re so smart, tell me, how to remove it, becouse it seems it’s hardcoded in the shell32.dll
It’s not just that we don’t upgrade when we install a new version of Windows on the desktop because the experience has been so horrendous in the past, it’s that we don’t even do a clean install of the new version until the hardware breaks down because of the fear that loads of things won’t work. With what we had before at least we’ve learned what doesn’t work.
‘SendTo Menu -> Send to Floppy’
What OS are you using. I’m on XP SP2 and the SendTo menu only has what I want in it – including my favourite a shortcut to SendTo itself. Take a look in:
Docu..&…tings/<user>/SendTo/ (*)
and see what’s in it.
(*) I HATE the outrageously long folder name too. ‘/usr’ or ‘/Users’ and ‘cd ~’ please.
[Begin rambling anecdotes]
About a decade ago, I worked for a very large company’s headquarters in their IS department, doing basic PC maintenance throughout the building. They had about a thousand PCs (give or take) at the time; no doubt they have more today. And at the time, the company had the policy that each department was responsible for paying for its own PCs — they all had to be bought through IS, but IS didn’t have a PC budget: Individual departments did.
This meant that the individual departments (and sometimes individual employees) would be responsible for deciding what we would do to their computers: Upgrade, replace, or the horror that was trickle-down (boss gets a new computer; his secretary gets his old one; her friend from accounting gets her old one; his old one goes to the intern; the intern’s goes to the temp guy… etc.). In my time there, I installed dozens of apps and at least three OSes, several thousand times.
My experience was that most OS installs don’t work right until the third time, and all upgrades NEVER work right. I can remember as far back as being forced to install Win3.1 on brand-new Compaq ‘386es where the installer would guaranteeably crash and reboot at semi-random intervals.
Since then, I’ve installed a *lot* of software, and you guys at Microsoft have improved your OS installers somewhat. But, even so, the rule of thumb that the install doesn’t work until the third time you try it is *still* true: Even when installing WinXP, I’ve encountered major glitches with PCI support, or with the USB system being broken, or with the built-in video or sound drivers producing a BSOD, right out of the box. And this applies to many different computers from many different manufacturers: This very computer, whose hardware is a mere six months old, has no power-management support in Windows, because setup would BSOD five minutes into booting the CD if I enabled power-management in the BIOS. (Seems to be a bug in Setup’s drivers for PCI Express.)
[End rambling anecdotes]
I think my point is that while I’ll eventually install Vista just like so many others here, I’m going to (as usual) expect it to burst into flames the first few times I try to install it, because that’s simply what I’ve come to expect from Windows installers. Maybe Vista’s installer will be better; maybe it’s been improved: But Microsoft’s reputation precedes it on this one, and it’s a reputation of "crash-and-burn."
That said, I’ve done a lot of Linux installs too, and while they’ve gotten a lot better than the all-night 36-hour install-a-thon I did in 1995, they still have a ways to go too. ;-)
A few years ago now Red Hat shipped a version of their OS installer (Anaconda) which couldn’t handle NTFS partitions. It didn’t ignore them, or accidentally reformat them or anything like that, it just crashed. I think you got a Python backtrace or something equally useless.
And the reason seems to have been exactly what’s described here. lots of Red Hat’s customers (especially desktop users) dual-boot Windows, but Red Hat’s engineers aren’t running Windows, so they never saw the trouble during testing.
FWIW I’ve done a lot of Red Hat upgrades and they were very good, definitely better value than the install & restore dance. I think the earlier poster was correct to cite package management integration as a key issue here. Microsoft’s OS upgrades probably work a lot better if you only use Microsoft software, but who does that?
You know, Sean W. seems right on the money. Now that I think back, my brand new gaming box required that I restart the XP install about 3 times before it "took".
In this case, none of it was the installers fault. It was fighting the fact that the shop had plugged my SATA drive into the wrong socket. Some other USB incompatability I cant quite recall, and, finally, a bundled Symantic firewall that came with the PC that caused it to bluescreen anytime any app – incluidng the upgrade wizard – tried to access the internet.
The only time I ever did an upgrade install was when we were running SBS. In that case you are sort of forced to do it. Now I always do clean installs to remove all the old junk that has built up on the machine.
No kidding.
Upgrades mean "leave all sorts of stuff around and hope it doesn’t cause problems". If you’re dealing with servers, and you have the time, clean installs are obviously safer.
i upgrade servers all the time.
I’ve had a couple fail, but for the most part it works just fine. It does take some care and feeding to take care of lingering errors and unused software.
but to state "nobody upgrades their server" is wrong.
PingBack from http://mattjp.wordpress.com/2005/12/14/sysprep-server-deployment-and-pain/