| Date: | March 21, 2006 / year-entry #102 |
| Tags: | other |
| Orig Link: | https://blogs.msdn.microsoft.com/oldnewthing/20060321-00/?p=31843 |
| Comments: | 57 |
| Summary: | Windows File Protection works by replacing files after they have been overwritten. Why didn't Windows just apply ACLs to deny write permission to the files? We tried that. It didn't work. Programs expect to be able to overwrite the files. A program's setup would run and it decided that it needed to "update" some system... |
Windows File Protection works by replacing files after they have been overwritten. Why didn't Windows just apply ACLs to deny write permission to the files? We tried that. It didn't work. Programs expect to be able to overwrite the files. A program's setup would run and it decided that it needed to "update" some system file and attempt to overwrite it. If the system tried to stop the file from being overwritten, the setup program would halt and report that it was unable to install the file. Even if the operating system detected that somebody was trying to overwrite a system file and instead gave them a handle to The solution was to let the program think it had won, and then, when it wasn't looking, put the original back. Now that Windows File Protection has been around for a few years, software installers have learned that it's not okay to overwrite system files (and trying to do it won't work anyway), so starting in Windows Vista, the Windows File Protection folks have started taking stronger steps to protect system files, and this includes using ACLs to make the files harder to replace. Presumably, they will have compatibility plans in place to accomodate programs whose setup really wants to overwrite a file. |
Comments (57)
Comments are closed. |
So with Windows File Protection user is allowed to believe that the application will function OK (since it installed just fine), but then WFP will replace the file(s) back, thus potentially breaking it? How is this better than just disallowing to install the app in the first place?
If the choice is between windows operating and an individual application operating the choice is clear.
I’m guessing Valdas’ point is that he’d rather have a 3rd option – block the App install. I don’t think that’s any better – instead of the app being potentially unstable (and Windows getting blamed), the app will simply fail to install in the first place (and Windows gets blamed).
There’s no "right" way to deal with such apps – the providers need to fix their installers so they don’t need to replace/modify/expect a certain version of system files (if such a need is even accurate in the first place, which I think Raymond is hinting is often not the case).
I think Valdas does have one point in his favor – if the app INSTALL breaks, at least you know exactly which app is attempting to mess things up for everyone else (And, with a little web search, why). With WFP, it may not be obvious which of multiple recently-installed apps was the culprit (since, with XP at least, I can do an awful lot of installs without having to reboot, which seems to be when I usually got WFP messages in the past).
Well the whole mess is really 100% MSFT’s own fault. For a long time, MSFT motivated the developers to do OS’s dirty work — "if your program uses another version of coolDlg.DLL and not the one delivered with OS, no problem-o, go on and copy the newer one to System or System32 with your installation program".
Instead, the OS should have had an API call which would be the only way to modify the files in System32, and which would clearly declare that it can but it doesn’t have to really change something there.
One version of the installer that I deeply hate (the InstallShield monstrosity) even managed to overwrite some NT DLL with a 9x one completely lobotomizing NT.
Blocking an install is easier said than done.
Your two options are
1. Block it before it can run. This involves a great number of difficulties as the install may be a custom created install not relying on MSI and thus can’t be scanned in advance to determine whether the program about to be executed is both a setup program and wants to update a protected file.
2. Block it as it is running. This one is equally infeasible as it would imply that Windows be able to track Undo information for every program on the off chance that the program running is trying to install something and later on during the install process tries to install over a protected file.
"Well the whole mess is really 100% MSFT’s own fault. For a long time, MSFT motivated the developers to do OS’s dirty work — "if your program uses another version of coolDlg.DLL and not the one delivered with OS, no problem-o, go on and copy the newer one to System or System32 with your installation program"."
Huh? The well-known way of pulling this off is to put "your version" of the DLL/whatever in the same directory as your application. That’s what most applications still do today. If your installer is copying a "common" or "system" file to the operating system’s directory, it better be darn careful about it. Why oh why don’t installers check their versions before copying? I don’t know how many times I’ll go to install drivers for something and get a "the file you are attempting to override is newer than the one you are copying" dialog…
"the providers need to fix their installers so they don’t need to replace/modify/expect a certain version of system files"
If one could trust that some behaviour your software relied on by part X of the OS would not change, and that isn’t so. If you were involved in writing installer software for example you might find that your attempts to replace a file were no longer successful…
The idea that one can run old apps on newer operating systems is silly, IMO.
Moi:
"The idea that one can run old apps on newer operating systems is silly, IMO."
So everytime you upgrade your OS, you will need to buy all new software and games?
That is even more silly.
In other words, if you have a festering wound covered with a dirty tattered bandage, you wouldn’t take it off, clean and disinfect the wound, and then put a clean bandaged on it. You would put still another bandage on top to hide the pus and blood that’s draining out. Then when the infected leg falls off, you would offer a service pack to fix it or, better, sell an upgrade.
The real fix is for ALL third parties to not install files in the system area. All drivers, dll’s, and configuration files should be installed in the application’s directory. While this is quite possible and rather easy to do, the documentation provided by Microsoft suggests you do otherwise.
Is it any wonder that Windows takes a hit when problems crop up? Microsoft has invited the stupidities from the get go. It deserves any hit it gets from the problems they cause.
How times change.
"Important
An application’s installation program must install CTL3D.DLL in the Windows SYSTEM directory or, if running on a networked Windows installation, the Windows directory. CTL3D.DLL must not be installed in the application’s own directory or any other directory […] CTL3DV2.DLL will not produce 3-D effects unless it is running from either windowssystem or the Windows directory. It is hoped this will encourage developers to correctly install CTL3DV2.DLL with their applications." — CTL3D documentation, by Wes Cherry and Kyle Marsh, August 1993.
Over the years I have cringed at the thought that rogue elements could overwrite system binaries, bypassing Windows File Protection with use of tools like SysInternal’s handle.exe. I always wondered WHY they didn’t have tighter ACLs on the files, and
I like the way WFP is implemented, with a "true" copy in the system32dllcache folder.
There have been times when I really wanted to make a replacement (usually putting XP’s notepad in Win 2000), where I’ve been able to circumvent WFP by putting the new notepad in the dllcache first, and then elsewhere.
The neat thing here is, this way, it protects my PC from rogue installers, yet allows me to mess with my PC when I want to.
If Windows had built-in installer right from the start…
"The real fix is for ALL third parties to not install files in the system area."
Anyone know why in the heck Flash gets installed in a "macromed" folder inside system32? What is wrong with Program Files?!
Tom Bowen:
> The issues Raymond is talking about had to be
> decided long before there was anything like the
> web, much less a web search.
The issue he is talking about is Windows File Protection, which didn’t exist until Windows XP. "The Web" was already going strong by this point.
—
Back in the 3.1 days (and possibly the 9x days), if you tried to keep your new version in your own directory, and the DLL was something that other programs running on the system were already using, you would not be able to load your new version, no matter how hard you tried. The system would only load a DLL once, and then share that instance of the DLL with any other programs that tried loading it later on. This is the whole reason programs were encouraged to write new system file versions into the system directories in the first place.
I do not quite see how Microsoft can be blamed for this issue, given the history behind it.
Basically, Microsoft has a difficult problem to, if not resolve, at least minimize. A very large number of software developers distribute things like msvcrt.dll (the C runtime DLL provided by Microsoft’s older compilers) with a version they know works for their system. They fail to examine the version of the old & new DLL to ensure they aren’t downgrading the file (usually out of a combination of slothful setup practices, or ignorance of the problem). Microsoft winds up looking bad when some other program fails to run because the supporting DLL no longer provides the function required by a previously installed application.
What does one do?
AC: There were a number of times I wish MS had added an InstallSystemDll() call that took the path to a DLL and checked if it already existed.
If the dll did not already exist, copy it in place and set the share count to 1.
If it did exist, and a version check (using the version info inside the dll) shows the requested version is newer than the existing one, then overwrite the old one (including whatever magic is necessary to overwrite on reboot if the old one is in use) and increment the share count.
Finally, if it did exist, but the new version was older, just increment the share count on the existing dll and return a success code.
That would have alleviated so much of the dll hell associated with system dlls, we might not be in this mess now. (Applications would still install a private version to their application directory if they really did need a specific version, or if the lib authors were not competent enough to maintain ABI backwards-compatibility between minor/patch releases)
For all the bad press WFP takes, I believe it to be a major step for system stability. I worry a lot less about crap installs on servers than I used to. Better than alternatives.
"Windows File Protection, which didn’t exist until Windows XP." Windows File Protection first arrived in Windows 95. In 1995, the web was only just getting off the ground.
"Instead, the OS should have had an API call…" Oh you mean VerInstallFile?
Except (AFAICT) VerInstallFile() doesn’t do the "replace on reboot if destination file is in use" magic, doesn’t increment the share count, and doesn’t return success if a newer file already exists.
If a dev can’t use this to replace dlls that are in use, and they need to roll that bit themselves, they can easily end up using the replace on reboot to overwrite newer versions. Oops.
If the function doesn’t return success on a newer file existing, and the caller doesn’t realise that one of the error codes is actually a success(!), what is the caller likely to do on error? Oh yeah, set up a replace on reboot, overwriting the newer version again. Oops.
(Why would anyone ever want to call VerInstallFile() with VIFF_FORCEINSTALL set to override the versioning of VerInstallFile()? Surely they’d just copy the file. I suppose part of the problem there is that VIFF_FORCEINSTALL isn’t granular enough and overrides too many things. I want to not care about different code pages as I’m installing an "en-GB" over an "en-US" file? Oops – just overwrote a newer version as well. Doh!)
IIRC versions of office around 1995 wrote more stuff in the Windows and System directories than in their program directories. Never did find out why.
The missing piece here is shared components. It’s always been a little … confusing … about the rules around redistributing components which ship with the operating system.
One of the keys of Windows’ success was to provide ways for other people to make money. ISVs of the metaphorical/mythical shrink-wrap software are one set of people but also having a rich environment for ISVs to provide shared components is similarly important.
Sheridan used to provide a grid control when I first started working at MSFT so I use it as my canonical example.
COM registration (until XP with application and component manifests) is system-wide so you couldn’t really carry a COM component private to an application. So you have to put it somewhere and you need to establish protocol about how it gets updated.
This is a good example of a component which could be privatized to the application but who owns servicing the binary? The application? The component provider?
These are difficult questions. The pendulum was towards "sharing is great! save memory!". Now, for applications, the pendulum is swinging the other way (privatization of everything) but imagine the Macromedia was privatized to each web site that used it. How could Macromedia ever produce a security patch for its engine – instead your system would be vulnerable to every privatized copy of the library.
Foo:
> You’d get the same thing on Unix if you had dumb installer and an app that uses shared libraries.
No, Unix is better than that.
You can link against a major version of the lib. This is a link to the installed version. Multiple versions of the same lib can coexist.
If you absolutely need a specific version of a lib, you can stick it in /opt/app/lib or wherever and fix LD_LIBRARY_PATH to point at it first. If you’re confused about where an app gets its libs, ‘ldd app_name’ will tell you.
Of course you can bugger a box by installing stuff over the existing libs, but there’s no reason to.
"Windows File Protection first arrived in Windows 95."
Unless Google and MSDN are incorrect, WFP first arrived in Windows 2000 and Me. Maybe you’re thinking of the System File Checker tool 98 had?
In Windows 95, it didn’t have a name. It was just a dialog box called "System File Warning".
WFP was not around in Windows 95. I worked on a product that replaced winsock.dll and it worked on 95 and NT 4.0. No warning dialog box.
The system file warning dialog in Windows 95 didn’t monitor all DLLs, just the ones that were known to cause problems at the time.
I’m from Linux, but my understanding of how Windows works *today* is that attempts to change system libraries cause your app to use the library in question as part of a private assembly, but keeps all other apps using the system-supplied version.
I recognize this problem isn’t as easy to work out as it appears on first glance. I think Microsoft has done about the best it can now by giving the new app what it’s actually asking for (that a certain library be available to it) without mucking with the system. There are drawbacks with this approach (eg., wasted disk space or the fact that the "upgraded" library may have security issues that the system version doesn’t), but there would be drawbacks on any approach.
Now, yes, IMO it was a mistake to allow/encourage developers to muck with system libraries. But, then again, developers should be able to realize it’s no longer 1993.
This "system file warning" dialog may have existed in Windows 95, but it sounds like a far cry from WFP, with the dllcache and all. I was wrong about it not being in 2K, and perhaps ME, but I still don’t think Windows 95’s "system file warning" counts as WFP.
True, it wasn’t very full-featured, but the basic idea was there.
I use Linux
Linux does what I tell it to do.
And if Windows had a decent process for installing applications, this wouldn’t be a problem. Instead it allows third party applications to do anything they like to the system during install (Kazaa anyone?).
At least if the ACL route had been taken, the installers would have had to be fixed – and fixed they would have been, when the vendors realised they didn’t run on XP.
It’s possible to work around file protection (I’ve lost count of how many times I’ve done it to uxtheme.dll) but it’s not very nice having the OS slyly replace files behind my back.
"So everytime you upgrade your OS, you will need to buy all new software and games?"
No, you do not. You should first question why you need to "upgrade your OS". The applications are what is important as they are what provide you with productivity – if you can achieve what you want with them, why should you want to change the OS?
Why do apps need to install DLL’s in the system directory anyway ?
The idea of DLL’s was that they would be shared between applications, but the current state is that most applications that install system-wide DLL’s are the only apps that use them. The only reason I see for sharing DLL’s is saving disk space, maybe important in 1995 but nowadays it is no longer an issue.
Since Vista is Microsoft’s attempt to catch up to OS X 10.4 Tiger why not also use the idea of bundles, just package the whole application with all it’s libraries and everything else it needs (except it’s settings) into a single bundle. I switched from Linux to OS X about a year ago and the concept of bundles is one of the things that make OS X a joy to use.
Windows could really do with a package manager. (No, msi does not count.)
It would also benefit from a repository system.
A package (deb, rpm, etc.) on linux is a compressed archive of the files to be installed, combined with a list of dependancies and optional configuration scripts.
When a package is installed, the package manager checks the dependencies and makes sure there are no file conflicts and then installs the files.
When combined with a repository system, it can download any missing dependancies. This goes a long way to reducing ‘dll hell’.
On Windows, there would be a Microsoft repository, including the latest versions of all MS redist components, although installers could include required packages for use ONLY when an internet connection is not available.
Third parties could also add repositories.
It could also be combined with automatic updates (as it already is on most linux distributions) so that getting new updates would be as simple as scanning the repositories for updated packages.
If third parties can also add repositories, then it would also add a much better solution for automatic updates for third party applications.
It is still possible to install programs outside the package manager on linux, as it would be on Windows. In fact, if WFP was also combined with the package manager, it would provide a mechanism for preventing overwrites, even of non-MS shared components.
Aaargh: It’s not just saving disk space. The other main two reasons, both more important than saving HD space are saving memory, and security.
Saving memory comes from the fact that if there are two applications running on a system using the same dll, then the actual code and read-only data (e.g. bitmap resources) only need to be loaded into memory once, and each app shares the data. While memory is cheaper than it was in ’95, it’s not as cheap as disk space, and the data we’re loading has got *big*. Icons instead of being 256-color 16×16 and 32×32 can now be (on some systems) 32-bit color 128 x 128 bitmaps, or vector graphics. And the types of resources used, such as animations, sounds, has increased as well.
The security comes from plugging holes. If someone finds a vulnerability in a shared library (say, zlib) then if all the applications that need that functionality use the shared version, you only need to upgrade one copy of the library to fix all the applications that use it. If each application has its own copy, or statically linked it, or even just used the source code directly (ahem, mozilla) then securing your system is an awful lot harder.
I love when an interface lies to me.
Honestly, this post made me feel a lot better about WFP. I had no idea that it had any purpose other than to block me from doing what I wanted with my system. I’m sure you can imagine how annoyed I was when I finally figured out that Windows was invisibly correcting my "mistakes," not bothering to tell me because users are too dumb to learn better.
I have a copy of VB3.0. It comes whith a cut-down installer generator, which I didn’t use as it was very nearly impossible tom stop it writing *all* your DLLs to windows/system.
That was the conventional wisdom "dlls go in windows/system".
Will this work on FAT32 partitions? How?
If Vista breaks old apps by resisting those to write to system32, why not at the same time rename the system directory to system64 in win64?
Wasn’t the name (system32) preserved because old apps should be able to write to it?
Do you 2 apps take this lawful compiled shared library with bugs & patches till disk crash do you part?
Hmm I notice this as I did a test install of an app I installed using Wise (version 7 IIRC) that the acl got in the way of Wise doing a version check. Tne problem is there isn’t an easy way around this for older programs.
Dunno if the Vista team have changed the rules so you can at least do a version check without getting an ACL violation.
Thankfully this isn’t my problem anymore for me as I’ve left the company concerned BUT the app was Windows XP Gold certified so we were very careful over version checking etc.
Andreas Magnusson, I use GNU/Linux and I’m 19. Not only do I get to read TONT, I can have as much Wine as I want ;)
BTW, system files ARE in fact protected by ACL’s. Just try (re)moving anything in system32. It only works when logged in as an administrator, which already pwns the box anyway.
As long as Microsoft covers up the independent software vendor (ISV) mistakes, they won’t bother to learn how to do things correctly. Simply because it makes business sense: why should the ISV waste time learning how to install their app correctly, if MS will cover up their mistake?
All the appcompat cruft makes for a better user experience but it just perpuates crappy software development. Many problems can be avoided by simply RTFM – the SDK docs, the API calls, etc. Many developers don’t bother.
MS really needs to stop covering for hardware and software bugs. Of course, it would be nice if their own applications didn’t try to overwrite kernel sreaming modules on Win2000 every time. :=)
BTW Raymond, your blog should be required reading for all Windows developers, cause it points out the mistakes others have made in the past. Other docs do not do that.
Thursday, March 23, 2006 2:27 PM by Driver Guy
> Many problems can be avoided by simply RTFM –
> the SDK docs, the API calls, etc.
In that spirit, I used to correspond with a certain famous blogger who used to solicit bug reports so that subsequent RTFM would yield more accurate information.
In that spirit I occasionally still submit bug reports directly to MSDN. Sometimes they reply that content is acquired from MSDN and who knows if it will ever get fixed. Sometimes they don’t reply and who knows if it will ever get fixed. Sometimes they reply that they’re fixing it but it still doesn’t get fixed.
Well, yes, sometimes RTFM does avoid problems. I just wish that were true more often. Meanwhile I RTFM a lot.
They should use GetSystemDirectory.
http://blogs.msdn.com/oldnewthing/archive/2004/10/08/239767.aspx
Linux doesn’t solve DLL Hell, it just replaces it with dependency hell instead. So the system won’t let you install conflicting libraries – great – now you can’t install (and therefore run) your application at all. Hardly a leap forward, all you did was swap one problem for another.
The approach Windows uses works absolutely fine, the problems with DLL Hell are due to its history and not any problem with the current design (in which programs can happily ship their own libraries privately).
Does someone know of good articles on updating system files and keeping WPF happy, including COM servers (OCXs), and using the side-by-side sharing hack that MS introduced in 98SE?
M: If all else fails ($LD_PRELOAD, $LD_LIBRARY_PATH), then there’s still chroot. I haven’t seen Windows installing a small version of Windows in a directory to safely install software in, seperate from the real installation. But debootstrap and yast can do that.
But if you stick to the distro package repositories, dependencies will be satisfied unless there are conflicting files. However, it’s no problem for packages of the same library but different versions to coexist (for instance libxml1 and libxml2).
PingBack from http://blogs.msdn.com/oldnewthing/archive/2006/04/03/567318.aspx