| Date: | September 19, 2005 / year-entry #268 |
| Tags: | other |
| Orig Link: | https://blogs.msdn.microsoft.com/oldnewthing/20050919-13/?p=34153 |
| Comments: | 30 |
| Summary: | One reaction to my story about the oldest computer at Microsoft still doing useful work was shock (shock!) that Microsoft suffers from power outages. In the Pacific Northwest, winter windstorms are quite common, and it is not unexpected that a windstorm blow down tall trees (which are also quite common) which in turn take out... |
One reaction to my story about the oldest computer at Microsoft still doing useful work was shock (shock!) that Microsoft suffers from power outages. In the Pacific Northwest, winter windstorms are quite common, and it is not unexpected that a windstorm blow down tall trees (which are also quite common) which in turn take out power lines. And if those power lines supply Microsoft main campus, then main campus loses power. All the critical computers have UPSs so that they can make a soft landing when the power goes out, but it's hardly the case that every single computer in every office and lab gets a UPS. That would be prohibitively expensive and wouldn't accomplish much anyway. Sure, each of the five computers in your office might stay alive for another fifteen minutes, but this assumes that you're actually in your office to shut them down cleanly when the power goes out. If your machine is frozen into the debugger, no amount of software-automated shutdown will help. (A frozen machine cannot shut itself down.) In other words, the cost-benefit of giving every employee a UPS for each machine in their office simply doesn't pan out. In the last few days of 1999, the main Windows development building was prepared for a wholesale catastrophe. Generator trucks were brought in so that the entire building could be kept up and running should the power fail as part of a worldwide Year 2000 meltdown. Those trucks were huge and no doubt extremely expensive. And thankfully were never needed. Those who were in Los Angeles last week for the PDC might be amused to learn that the PDC technical staff, fearing a repeat of Monday's blackout, rented a generator truck to provide emergency backup power for all the machines on stage for Bill Gates' and Jim Allchin's keynote addresses. The power may go out in Los Angeles, but the PDC keynote must go on! More musings about power outages next time. |
Comments (30)
Comments are closed. |
". If your machine is frozen into the debugger, no amount of software-automated shutdown will help. (A frozen machine cannot shut itself down.)"
While I don’t disagree with you, it seems as though you’re dismissing a 90% solution due to a 10% problem.
I’ve never understood the mentality behind not using a UPS for a PC. The arguement that it isn’t economic is unsupportable when you compare the value of a person’s time and the reliability of the data that is being used.
UPSes provide three services; 1) Power when the source is disrupted, 2) Surge and brownout prevention and 3) Power conditioning – ensuring a quality of power that is supplied to the hardware is free of electric ‘noise’ and other malformations (like hot cross-wired against cold). In addition to these three services, a UPS also ensures that your computer will perfectly survive a transient power disruption. If the lights flicker for a few moments during a storm (or even a car accident that takes out one of many supply point powerpoles), the UPS ensures that your PC continues to operate as if nothing ever happened.
While a ‘frozen’ PC won’t automatically shut-down during a long-term, catestrophic power outage, most of your PCs will be able to shut-down should a major power failure event happen.
One risk that isn’t considered in your article is the corruption of data, or the ‘hanging’ of a perfectly fine PC during a bad-power / no power event. The corruption of data has the potential of costing far more than the value of a simple UPS. Even the cost of a developer’s time (or any person who uses a PC) to re-create what was lost in the event of a power failure likely far exceeds the cost of a typical UPS for a desk-top unit (an APC 350 UPS with USB connection is $50).
So many companies expect that telephone system will continue to work during a power outage (and often it does), but are happy to have the computing infrastructure go away if the power blips. Consider the cost to any company to suddenly have your entire staff instantly unproductive during a power bump while they reboot and attempt to regenerate / redo all the work that was not saved when the power went bad. With a UPS, people keep right on working, never loosing data, never being distracted or interrupted. A $50 UPS begins to look very affordable.
It seems your reasoning would apply equally to most other desktop PC users. Based on my experience, I cannot agree with your analysis.
If an inexpensive UPS costs $100, how many hours of lost productivity does it have to save per year? Not very many.
If power outages are at all common, and your time has any value at all, a UPS on your workstation is a good investment.
I had a UPS but never hooked up the serial cable because I didn’t trust the software that came with it. In addition to the cost of the unit, you also have to include the cost of training the IT department helpdesk how to set them up, a particularly gnarly task since we run all sorts of beta operating systems here. But if an individual employee can justify the cost for the benefit, they can order one for themselves. Personally I’ve never lost more than a few minutes’ work. In fact, I’ve lost more work due to a bad UPS than I’ve had saved by a working one.
Where I work, everyone has laptops now. No need for personal UPSs.
We still get killed by power outages though; the network infrastructure isn’t fully UPSed and we lose connectivity quickly.
My company provides a uninterruptable power for every employee, and there isn’t a UPS in sight until you leave the building.
There you’ll see two big diesel generators. They provide power to a seperate "UPS bus" in the building. Anything that has to be kept alive is plugged into that bus.
Under normal situations, that bus is suppled by our power provider. When power goes out (as it does from time to time in Florida) the generators kick in.
The generators are provided by the building’s owner, rather than the company itself (we lease) but it does appear that it is economical else it would not be done.
Don’t forget that UPSs have "moving parts" too and the protection they provide is as only as good as their health! I’ve been burned by a bad UPS once. We had several single-machine UPSs left over after a lab consolidation, so I grabbed on and put my main workstation on it. Apparently the battery was starting to get a bit old. We must have had a power failure or a spike over the weekend. Monday morning I came in and saw that my PC was rebooting over and over–halfway into the reboot, the UPS would issue a shrill shriek, and the reboot would start over. Lord knows how long that went on over the weekend. Inrush current, over and over….
I’ll probably get burned by a real power failure some day but since we’ve averaged 3 power failures/year over the last 4 years at this location, I figure I can live with the odds for now.
I’m surprised the msft campus doesn’t have it’s own emergency generator.
At my previous job the power would go out about once a week during the summer. Management would not provide UPSes so I bought one on EBay and installed new batteries.
On the next outage, after everyone quit swearing, they started to hear the beep from my UPS and came to investigate.
The CEO also came and was extremely annoyed at me since he told me that I couldn’t expense a UPS and now I obviously had one.
It took ten minutes of explanation from a VP to get him to understand that I bought the UPS with my own money.
A few weeks later, the company started to purchase UPSes.
Once I fix a bug, it’s out of my mind. Fixing a bug that I just fixed is harder than fixing a bug that I haven’t worked on yet.
Microsoft could at least give a UPS for dev machines. It can’t possibly be "prohibitively expensive" to provide that extra bit of reliability for everyone’s key machine. There will still be plenty of dirty shutdowns for the file system guys to debug. ;) Isn’t there still an ergonomic benefit, or was that canned due to cutbacks? You could argue the UPS might prevent repetitive stress when you have to recreate lost work.
I’m quite surprised that the main Microsoft campus doesn’t have any kind of emergency generator. The last three companies I’ve worked at in London have all had huge great generators in the basement of various types to hold them through a power outage.
At my current company I’ve got four plug sockets under my desk (with a 4-gang or 2 hanging off them, which Facilities turn a blind eye too :) ), three are standard 3 pin power plugs and the fourth is coloured red and has different shaped pins.
Anything plugged into that fourth socket is plugged into ‘VIP’ bus and is on the hot-start generators, with no downtime in the case of power problems (the other sockets are on the bigger generators that take a few minutes to warm up).
With a trading floor downstairs, any downtime through power problems would be inexcusable.
It takes three semi-trailers to power an entire building. (This includes all the testing labs with thousands of computers in them.) Would you want one of these parked outside your building spewing diesel fumes so you can save 15 minutes of work once a year?
UPSes aren’t a panacea. When I was at Enron, we lost all of the servers supporting our internal trading apps when the UPS crashed.
It took a couple of hours to get the UPS out of the floor and stable, but once they did so, everything came right back up. (Although Oracle’s REALLY slow while the box it’s running on is doing consistancy checks on its RAID arrays.)
Don’t most developers automatically press Ctrl+S every few minutes anyway? It’s a reflex action for me.
One thing that you’re missing here is the idea that either there’s a UPS at everyone’s desk, or Microsoft has no backup power supplies at all.
There’s a middle ground: the computer labs, where critical dev and test equipment reside, generally have data center-class UPS systems.
Though I don’t have direct experience here, I’m sure that the internal Microsoft IT group, that hosts line-of-business apps, including the source control systems, has robust power backup for the servers they manage, too.
If the data or processes are critical, get them onto an IT-supported machine, or into your group’s lab. There the data will be protected against more than just power failures: the hardware will be monitored and replaced as it fails, there will be data backups, etc.
Diesel generators do not run all the time, they only kick in when the power goes out. That’s why they’re so huge, the generator itself is tiny, most of the space are the huge batteries to supply power for the half a minute or so while the generator starts. We have those at work as well and imho it’s a much better solution than individual UPS for each computer/worker once you get over few of those (we have somewhere on the order of 1,000 or so computers). And yes, I’d rather have a box outside the building than to have everyone scramble after a power failure.
Some of us need to retake a class in economics.
$100 UPS to save 15 minutes of work that may be worth $50.
That is not cost efficient. Only when the work being saved is worth more than the cost of the UPS should the purchase of the UPS be justified.
James
No matter how frequent the interval between saves, no matter how vigilant the autosave feature might be, power outages invariably occur immediately prior to the save that *would have* preserved the most important work you’ve produced so far.
Always have a UPS. They’re now too cheap to go without
My television and TiVo are on their own, dedicated UPS, a cheap, 500va UPS from MicroCenter. I think I may have paid $39 for it, if that much. I never miss shows now.
My computers are all plugged into a surplus 3000va rackmount UPS, a holdover from when the web hosting company I had previously worked for changed over from many individual UPSes to a few room-sized units assisted by diesel generators.
And that’s at my house.
My desktop PC at work, and those of all the other developers where I work, are plugged into individual Belkin 750va units. They’ve saved us numerous times already, more than paying for themselves during the stormy summer season already. The potential lost productivity that we *didn’t* experience paid for the UPSes the very first time the power went out.
Do the math: IF ((Your hourly rate)*(hours spent recreating lost work)) – (cost of a small UPS) > 0 THEN get yer butt out to the store. End If.
For most IT professionals, if you spent more than two hours recovering or recreating lost work, you’ve completely paid for a small UPS that will last several years. Over the lifetime of the UPS, you’ll likely experience multiple power outages that’ll total more than two hours of downtime and have the potential to wipe out far more hours’ worth of work.
In my opinion, if you have a modern computer, you absolutely need a keyboard, mouse, monitor, and UPS.
Hm, for me in southern Germany "power outage" is actually a pretty alien concept. I vaguelly remember one scheduled maintenance outage a few years back, but apart from that my server always ran for years without reboot (current uptime only 296 days because that’s when I set up the new system). Therefore never really saw the need for a UPS. No idea, perhaps it’s because all power lines are run underground or whatever.
I think your all missing a "dogfood" scenario here. Windows is probably less likely to corrupt data in a devastating way when your average consumer gets an outage at home because it’s developers have to suffer under the same circumstances at work.
And MS obviously make allowances at critical times e.g. the RTM of Win2000
Re: Economics.
>$100 UPS to save 15 minutes of work that may be worth $50.
In this case, you’ve proven that the UPS won’t pay for itself in a single blackout.
Now you’ve just got to ensure that you’ll never have but one blackout in the lifetime of the UPS and you’ll have finally proven your point.
Even if the UPS had a miserable lifetime of only 2 years, how many blackouts are you likely to suffer during that time? These things add up.
Even if you save your data continuously, can you guarantee that you’ll never suffer from hardware failure as a result of a blackout, brownout, or surge, which could possibly destroy not only your UNsaved date, but all your SAVED data as well? What, you say you backup your data to disk/tape/network on a regular basis? How often might that be?
Your statement "Only when the work being saved is worth more than the cost of the UPS should the purchase of the UPS be justified. " is indeed true.
Your statement "That is not cost efficient." is irrelevant because it defines ‘cost efficient’ only as saving enough money to pay for the UPS in one shot.
What if that fifteen minutes’ worth of work wasn’t just some random accident at a convenient time when you’re doing nothing but typing leisurely away. What it it struck at a *time-critical* moment.
If power failures waited until it was convenient for you to happen, they’d only happen after you’ve left work.
But if you don’t want to take my advice, then fine. Don’t. You sit there, retyping your lost code, proud of saving $50 or $100 by not buying that UPS. I’m *certainly* not going to be taking *your* advice and risking doing my work twice because you think I might save a buck or two.
For all we know, we might be competitors…So, keep up the good work.
Monday, September 19, 2005 11:19 AM by Eric K.
> power outages invariably occur immediately
> prior to the save that *would have*
> preserved the most important work you’ve
> produced so far.
Nope, that’s second place. First place is *during* the save, so your file ends up with some sectors containing binary zeroes instead of either the old or new contents.
Monday, September 19, 2005 12:24 PM by John Davies
> It took ten minutes of explanation from a VP
> to get him to understand that I bought the
> UPS with my own money.
And once he understood that, he didn’t fire you on the spot? And he didn’t fire the VP who knew about it? Wow, what an enlightened CEO. But watch out. If your company survives it will get taken over by someone with more traditional management practices and you’ll never be allowed to get away with something like that again.
If you live in Western Australia, you can almost guarantee there will at least be a dozen or so micro-brown outs lasting a few hundreds of milliseconds through out the summer.
The wonders of peak power usage being driven by AC usage, and the power utility not keeping up with the demand.
At Microsoft’s campus in Las Colinas, we have power outages that last a half-second or so. (They’re much less frequent now than they were when the building was new; we had ’em as often as twice a month.)
UPSes would absolutely save lots of work (and money, I believe) for power line hiccups like that.
Which is why many of my peers and I went out and bought our own UPSes.
Norman-
You’ve obviously never worked for a small company. At a small company your personal property is constantly being brought into work to supplement the small number of items provided by the company.
At various jobs, I’ve brought in power tools, computer parts, cables, test equipment, photographic equipment, and office supplies. Most of the people I worked with did the same.
My home-based part-time consulting company has better tools than any company I ever worked for.
Wednesday, September 21, 2005 12:27 PM by John Davies
> Norman- You’ve obviously never worked for a
> small company.
I have and I do now, but I haven’t worked for one with an enlightened CEO.
> At a small company your personal property
> is constantly being brought into work to
> supplement the small number of items provided
> by the company.
Sure, like pens and some of my personally purchased MSDN stuff. (In some companies even the latter is a firing offence.) But in your situation, the company had already told you not to purchase a UPS, and they hadn’t told you that a personal purchase could be an exception. You’re a brave man.
@Norman
What do you mean by "enlightened CEO" ? Are you implying that "normal CEO" would just fire the guy that bought the UPS?
If so, why? Just because he had a brain and spent his OWN money in benefit of company work??
Also, why those MSDN discs would be a firing offense?
@Norman
My experience working for companies is exatcly ZERO! :)
I work for the government, fortunately! I can use my brains and even though it could cause embarrassment to someone, I can’t be fired just because of this!
I can understand that prohibitting personal soft on work machines makes a lot of sense, as it avoids ppl installing unlicensed programs there! But I think this should apply only to non technical staff, devs and IT personnel should be allowed to, if he or she can prove the soft is legally licensed!
I tried to VPN in to work today to see how my stress tests were running. Unfortunately Mother Nature…