| Date: | October 12, 2007 / year-entry #376 |
| Tags: | no-good-deed-goes-unpunished;other |
| Orig Link: | https://blogs.msdn.microsoft.com/oldnewthing/20071012-00/?p=24803 |
| Comments: | 36 |
| Summary: | There's no such thing as a free lunch. Or free code either. Many years ago, I wrote some samples for the SDK as a favor, and at each major SDK release, I am reminded that no good deed goes unpunished. I can tell when a major SDK release is coming, because I get a piece... |
There's no such thing as a free lunch. Or free code either. Many years ago, I wrote some samples for the SDK as a favor, and at each major SDK release, I am reminded that no good deed goes unpunished. I can tell when a major SDK release is coming, because I get a piece of mail that goes something like this:
Yup, it's another episode of "You're not my manager." This gazillion-item checklist goes something like this:
It's the gift that punishes the giver. And not just once, but repeatedly, with no end in sight. If I'd know that writing an SDK sample would be so much recurring hassle, I might not have done it. Every time this happens, I have to remind the SDK folks that if they want people to do something as a favor to them, they have to make it easy. This means setting up machines with Windows XP and Windows XP 64-bit edition, installing Visual Studio Super Awesome Edition and all the additional tools, and letting people use Remote Desktop to connect to them, compile their samples, and run tests X, Y and Z. Each time I recommend that they do this, the response is "Hey, that's a good idea." It's like they forget that they did this same thing last time. |
Comments (36)
Comments are closed. |
Personally, I think they should test the SDK sample. It is always better to get some clueless third party to check if it works. If they can’t get it going, then it is likely to frustrate some poor outside user who is then going to come back and gripe about MS giving bad / useless code.
Raymond, while I agree they could make it easier by providing a machine for you to remote desktop into and quickly verify your sample works, I do think it’s unfair that you say you’re doing a favor to the SDK team. Samples provided by the SDK are my primary way of learning and having an avenue (connect) to post issues or questions is invaluable. Putting it on your blog is welcome, but since there’s no quality control, or version upkeep, and if you happened to leave Microsoft there’s a chance that sample would never get updated on your blog either. So you’re doing a favor for me, your customer, who continues buying yet another piece of Microsoft software each time something else nice comes out and continues developing on windows because the samples work and are written (and maintained) by the people who set the guidelines for how it should be done.
So while I hope the SDK team makes it easier for you to update/maintain/QC your samples, please don’t give up and pull your samples from the SDK!
And in reply to Richard’s post on the SDK team testing the sample, my impression is that they weren’t staffed up to test the thousands of samples they put out in the SDK and depend on the individual sample owners to ensure they not only work, but also are designed with the latest guidelines/technology updates from whatever team submitted that sample. Think of the logistical nightmare of having a test team that’s large enough to have tester who know every single technology of every sample that they ship. While "does it build/run" is an easy test, who verifies that the code still makes sense, or even more importantly, that the code does not have any recently found security holes? I would hope that the person who wrote the sample has to do it, and yes, for every major release, or it would have the same trust level as any sample found on a random google group.
Or are there no more Ray entries in the SDK help ?
I see Richard’s point about having the SDK folks test it, but I worry about a clueless 3rd party knowing whether the results/output of the sample code is correct or not.
If the demo code is from some feature that Ray developed, I’d strongly disagree with the "not my job" mentality. You built it, you should be responsible for showing folks how to use it, even if your example/demo is handed off.
If Ray’s just bitching about the process, well, gee, I sure test a lot of my stuff on various platforms/compilers to make sure it works, why should example code be any different ? Someone has to be responsible for it, you’re not just throwing it over the wall and bailing. That’s lazy.
Raymond’s right. The samples are something that can be tested fast, once you have a preconfigured computer. SDK folks must have the current configuration. But the chance that contributor has the same configuration is null.
As we all know, installing Windows XP (or newer) is damn slow process, ditto for Visual Studio. Requesting from somebody to spend a day installing environment only to do a test that takes a few minutes is rude, and if they still do that, I’m surprised that there are any samples in the SDK left.
The minimum they should provide is the Virtual PC images with the fully installed SDK.
In fact, I believe that installing preconfigured Windows should be also as easy and fast as copying image for Virtual PC. It would be possible to make things like that, if there would be such an initiative. But that’s another story. It’s obviousl not something that will get somebody raise in MSFT, even if millions of people would benefit, including, as we see, the MSFT itself.
Why is the whole procedure not automated at this point ?!?
Seems like most of those steps could and should be automated. You should only get email if your sample fails an automated regression test. After all, the SDK samples are generally small and independent–perfect for little unit tests.
Did anybody notice the word "Free" ????
Yeah, you’d think that the SDK folks would provide "free" VHDs of "Windows XP with all service packs and patches plus Visual Studio Super Awesome Edition plus the latest build of the Platform SDK plus Visual Studio settings…". Surely they have ones they use themselves.
64-bit would be more difficult, being that Virtual PC / Server doesn’t support 64-bit guest OSs, at least not until Windows Server Virtualization reaches a certain level of maturity. Unless you’re allowed to use The VPC Competitor Who Must Not Be Named (rhymes with "rare").
BTW, a contributor who don’t test their sample at all is stupid.
So, why do are you still doing it for each release?
I’d get rid of "Start with a clean install of Windows XP with all service packs and patches.", because it is almost unlikely that the contributer does not have one.
It just boils down to respecting people’s time.
A few months ago when resolving an issue sent to a ‘hot’ alias I noticed on the web site that there were a couple 1-1.5 year-old stress issues assigned to me so I resolved them. A few moments later they get reactivated and I get notified. I close them again, making a note that they are ancient edge-case stress bugs for a particular internal build that have been fixed, and to not reactivate. Moments later it gets reactivated again, saying I’m resolving it wrong — some minor bookkeeping reason.
After going back and forth we finally found that to resolve the issue "correctly", instead of doing it via. the web site I had to install some heavy local application, dig up some old info, find the relevant new owners, etc. I put this on my ‘todo’ list…
…I hope they’re not still holding the machines.
I’m going to be ruthless here and say that if the code doesnt work in today’s systems, it shouldn’t be in the SDK docs. Otherwise end users paste the stuff in, the SDK team field the bugreps. And its not just functionality; there’s security, i18n and lots of other potholes in old code.
-the real problem here is not that raymond is expected to do the work, but that this should all somehow be automated; every code snippet in the docs should be built and tested on the released platforms and the nightly build of the forthcoming docs. This goes for content in MSDN articles as well as classic SDK stuff.
Is this hard? Well, all the Java code in (product placement time) Ant in Action is tested against the nightly builds of Ant and its dependencies, albeit on a stable OS release. And for the Ruby books, the Pragmatic Programmers have somehow got the text integrated with the compiler so that not only is every code snippet built on demand, the output of the runs gets pasted into the book.
Documentation is just another form of software artifact, one that needs maintenance. Too bad that raymond didnt hand off ownership, or worse, that the SDK team don’t maintain their own examples.
Finally, note that the Java APIs have a lot less code snippets in them. Maybe it’s to keep costs down.
A fan of the musical <em>Wicked</em>?
WTF: Not realizing what a good deed Raymond has done, the SDK team has this completely backwards!
Raymond has done a good deed by contributing to a cannon of recommended practices for the SDK team to test their changes against.
The code has been widely distributed and needs to be supported for backward compatibility. It is not the samples that need to be tested against the SDK for correctness, it is the SDK that needs to be tested against the samples for correctness.
The notion that if Raymond is unavailable to do their testing for them then the samples will be removed is equivalent to saying that if the original alpha/beta tester of a feature is unavailable then support for the feature will be discontinued without notice.
Instead, Raymond should expect an occasional piece of mail that goes something like this: "Hi. A bug in the latest test build of the SDK has been revealed by your sample code, but we can’t seem to figure out what we did wrong. Can you make time to lend us some of your psychic debugging powers?"
Holy crap. And all this time I thought the samples were written by co-op/interns: http://msdn2.microsoft.com/en-us/library/aa365917.aspx
Why is the automation in place to hassle original SDK sample authors, but not to actually test the code? Oh yeah, it’s easier.
I agree that the SDK team should be doing it themselves, and that sample code provided by developers are a "favor". The SDK team should be doing that all themselves. And why hassle the original developer with form email, with the threat of removing the sample? Whats the incentive? To have the sample be automagically removed after the task wasn’t completed and thus not be harassed by more form mails? That’s only going to hurt users that are able to use the samples.
"Each time I recommend that they do this, the response is Hey, that’s a good idea.’ It’s like they forget that they did this same thing last time."
I’d be curious to know if you are recommending this to the same individuals. I’d guess that this particular task is given to low-level who are promoted/transferred/fired at a relatively high rate, preventing any sort of institutional learning.
High rates of org-chart churn make institutional learning counter-productive for the individual. "Hmm. I can improve processes, which will have no observable effect for a couple of years, or I can manually send 1,000 emails and tell my boss how hard I’m working…what to do, what to do…"
Well, it is better going through that checkpoint list than distributing broken samples like those in WTL 8.0 FINAL (they wrote it in caps, not me, oh the irony).
Lots of talk here about what the SDK team should do with regards to testing the samples, but not from anyone that actually does it.
So here it is.
There are currently 1461 managed (.NET) and 722 unmanaged (Win32) samples in the Windows SDK. Raymond owns 5 of these.
Every day, we build all of these samples on all the major supported OSes (XP, Vista, Server) & hardware platforms (x86, x64, IA64) the SDK supports on machines configured with all the necessary prerequisites (DirectX SDK, Office, .NET). The data from these tests generate about 2.5GB worth of information. We have a system in place to sift through this to find build errors that we still have to find a fix for (it’s not obvious sometimes). But this is purely build information, and there are other tests we have to run on the samples that generates even more data that are requirements for shipping the SDK.
Believe it or not, sample code has to meet the same quality guidelines as binaries do when they ship. They are run through the same security tests, tests for obsolete APIs, tests for term usage, etc. that all shipping products at Microsoft must pass. I wish this could be totally automated, but sometimes we need help from sample contributors to help us check off these requirements. We have processes to provide machines so that contributors don’t need to go through a ‘gagazillion-item checklist’ to test their samples.
Sample creators are the experts of their technologies. We heavily depend on them to know how to write good sample code and keep it up to date and relevant to show customers how to use their technologies. We try to ensure samples build cleanly, demonstrate their intentions, and use good coding practices so others may use them as solid examples.
It is not our intention to burden sample contributors with extra work every time we ship an SDK, but to make sure we give customers what they need to properly use the APIs we publish.
-Brian
Windows SDK Team
Brian, I have found the problem. Your release plan is missing 2183 line items of approximately 40 hours each for certification and review of the code samples.
Your sample "owners" may or may not be willing and able to test these samples for you. But why they would want to volunteer their personal time to provide free work for their employer is hard to figure out. You see, each and every one of those "owners" has a manager who almost certainly assumes he has the full-time attention of his staff.
I’m sure they appreciate that it isn’t your intention to create extra work for them, and I’m sure you’ll appreciate that with no samples, the SDK will be easier to test and ship.
Ok, I admit I am being cranky about this.
I appreciate that the requisite testing and patching is a costly and painful exercise. Nevertheless, in relation to the gazillions of lines of customer codebase it is not a disproportionate burden. It is not only a matter of shipping samples that work, but also ensuring customers are not inadvertently or unnecessarily forced to fix what is not broken.
I recognize that some samples will be found to be using techniques that simply must be deprecated or disallowed. In such situations there is a decision whether to rework or expunge the code. Why is this the responsibility of the original author? Does anybody even know who wrote the first "hello world" program?
Raymond, I’m sure the people you talk to agree that’s a good idea, cause it obviously is. But think about what *their* manager would say about that. If there are several 1000 code samples per product floating around, which each get a frequent SDK update it must be a small logistical nightmare to allocate enough computers to ensure a large amount of developers can work on their code sample test at the same time. Nobody like the "sorry, no boxes available right now, come back when you don’t have time anymore".
If there are several products that get an update that would mean even more resources, etc etc .. so it’s of course handy to let the sample writer take care of everything.
I’m not saying that’s the way to go though, I’m more in favour of a solution that’s not a nightmare to set up and still have the sample writers be able to quickly test their code.
They might as well have a (small?) department that does this for *all* samples for *all* products. If a sample doesn’t work the way is required, the writer gets informed. Sure, that’ll cost money too and the workers will need to get exact instructions on how to test each product, but still .. it’s centralised, less resources need to be allocated at the same time and the writer doesn’t get informed if everything works out
Kernighan? Or perhaps Ritchie?
Yuhong: The point about starting with a *clean* fully patched WinXP system is that most of us *don’t* have such a machine: we will all have installed extra things on it – e-mail clients, VPN software, browser plugins, anti-virus software – any one of which could bring in an extra DLL or patch which affects the functionality of your app.
Testing *the sample you submit* is obvious – what Raymond is complaining about (and rightly so, IMO) is needing to re-test it every time the SDK is updated. If the SDK guys could reduce this to "could you all please re-build/re-test your sample under the virtual machine at \sdktesttestvmbuild1234testmachine.vhd" it would be much less of a pain for everyone: drop in your sample files, compile, run. A couple of minutes, max, unless something has actually broken with the SDK update.
Raymond’s suggestion of a physical machine and Remote Desktop has the downside of contention, but everyone could run a copy of a standard virtual machine at once with no impact. A virtual machine might interfere with a few specific examples – but probably less so than running over Remote Desktop Connection would.
Of course, Raymond could always build such a VM and donate it to the SDK team…
Worse still, I occasionally get people spotting posts of mine on developer mailing lists, sometimes from years earlier – and sending me completely unrelated support queries. "You’re obviously an expert in Widget, how do I make it blow bubbles across a WAN?"
Waitaminute…Visual Studio Super Awesome Edition? Where do I get that one:? All I can get through my msdn subscription is Visual Studio Mediocre Edition, which seems to have trouble just compiling code.
The question I have is: does Raymond currently own the API that the sample is for? If he does, then his manager should be accommodating this request, and if not, whoever now owns the API should assume ownership of the sample. It seems to me that the group that owns the API should also be maintaining the sample code — it is both documentation and a test case. If there is a change in recommended practice that requires updating the samples, then the API documentation will need to be updated with the new recommendation anyway. If the current API owner isn’t willing to maintain the sample, the he has less room to complain when his API is abused or broken.
I’d assume if Raymond owns five code samples, he’s the "designated expert" on the things those five samples relate to, the most likely person in the company to know if standards for those things change.
I don’t see him complaining about this at all in the original article.
What he appears to have an issue with is that the email senders are unaware or failing to follow Brian’s "processes to provide machines": not just once, but every single time.
He also grumbles about the request being sent to him, but I see this as his own fault: the samples will have his contact details on. If he wanted his manager to be contacted instead, he should’ve put that in the contact info.
He also seems to take issue with the silly amount of work being expected of him. Personally, if I got such a poorly-thought-out request, I’d replt "I’m willing to donate no more than [X time] per release to your cause. If you’re serious about decent docs, improve your processes considerably: otherwise, delete the samples, I’ll pop them on my blog instead."
Isn’t this the best example of Virtual Machine use?
@Steve Loughran
"I’m going to be ruthless here and say that if the code doesnt work in today’s systems, it shouldn’t be in the SDK docs. […] And its not just functionality; there’s security, i18n and lots of other potholes in old code."
You raise an interesting point here: How much "produciton quality" should a sample have?
For me, a sample is to illustrate one single point, it should do that as conscice as possible. Respecting requirements of production code (like putting strings into resources) complicates examples immensely – especially if this is Win32 code.
OTOH, Examples may be the one MSDN thing that even the most hasty coder reads (including guys like me who think they already know everything…), so they’d be a good place to educate.
@Brian:
You test them every day?!? Why not after every keystroke in editor? It would waste your time more easily.
Okay, maybe there’s a relatively simple solution here.
First, withdraw your samples from the SDK. That way, you don’t have to test them every time a new Visual Studio Mega Super Awesome Pro Media Joint Special Edition for Home Enterprise (Now with Additional Flair!) gets released.
Then, bundle ’em up in zip files and post ’em on your own web site with a disclaimer that says in big letters, "Free samples — untested — unofficial — these may not work — questions about these will be ignored." That way, the public doesn’t lose out on your samples, and you’re off the hook for code quality. You can even post new stuff there, and it’s basically as-is.
Shoot, the SysInternals guys basically did that for years, and their stuff was consistently highly-rated by everybody who used it.
The only people who lose out in this are the SDK guys, and if they’re regularly causing you extra grief, they deserve to lose out, I think ;-)
@Sean W.: You’re overlooking that the SysInternals guys worked just as hard on their stuff as the SDK people. They didn’t just toss stuff on the web and then forgot about it.
Some of the SDK samples (especially the ones dealing with functions that have to be tickled in just the right order) cannot possibly be divined by a programmer using only the docs on each individual function. At the least, they are great time savers.
That’s exactly where the value of the samples kicks in. It’s good and appropriate to recognize that they have this value, and make an effort to guarantee it. Just tossing everything in a big "unsupported" pile is absolutely the easier way, but it’s not a "solution". Otherwise you’d end up with an SDK devoid of samples and lots of blogs where everything is spread out. Not something that would make me happy.
Now, that Raymond and the SDK folks need to figure out a better working relationship for that is another matter… But someone who recognizes the value should make sure Raymond can be committed to maintaining his SDK samples with as little hassle as possible.
The great big Internet and its prophet Google are certainly invaluable allies, but I’m a part of that old-school group who would still like the stuff you get with the offline installation to go above and beyond the call of simply saying "these are the function parameters, good luck". Don’t get me started on reading the headers (or disassembling some hairy bit of code). I feel slightly disappointed every time I have to dig up the things that aren’t documented. Sure, you’ll learn lots, but the barrier is a lot higher, and you never know when something you’ve inferred from the internals will be invalidated. I love learning, but my boss likes finished products. If I can get something done without groping around in innards, so much the better.
As for Raymond’s stuff being more useful: it is if you come across one of those things you really need to know. I imagine most hackers just assimilate as much of this inside information as they can just in case it ever comes in handy, as some of it surely will. I’d be the last to argue that Raymond (and all those other Microsoft bloggers who are close to the metal) aren’t providing an invaluable service. But at the same time, those are things I read in my (semi-)spare time. Few of it will directly apply to what I’m doing. Without the SDK, Raymond and friends would be of little help.
That’s where the meat-and-potatoes SDK comes in, and I don’t mind some garnish on the side if it saves me firing up my browser and trawling through Google hits to find out who’s in the know about something (and it’s not all gold out there). When we say the SDK is "official", that’s not to imply that the information in there should be the most complete or useful — just that it’s the documentation that people should be able to turn to and use for production stuff, even without the gold nuggets of the experts. I think code samples are very much a part of that.
As for the choice you mention, I’d like to think that we could have our cake and eat it too if the procedures are streamlined. Otherwise I’d have to ask you what you wanted more: Raymond posting stuff for the Old New Thing or fixing bugs in the next release of Windows? It’s not really a fair choice.
@JM: No, they didn’t toss up everything and forget about it. Some of it was quite well maintained. But there were also some old utilities and code samples that they really *did* just decide to leave up for "historical reasons": Download at your own risk. Check their archives; they’ve released a heck of a lot more stuff than just Process Explorer and Rootkit Revealer, and some of it’s even visibly marked as old and unsupported.
And while I’ve read through the SDK samples, I’ve found that I can usually get better information by (A) reading the header files and (B) searching the web. The days of the "single repository of knowledge" are long gone; this is the era of Google, and if you’re not using it, you’re missing out on a *lot* of better information than MSDN and the SDK usually provide, including some stuff that MSDN and the SDK leave out. Yes, it might not be "official", but so what? It doesn’t really matter *who* gives you an answer as long as it’s the correct one. The fact that you’re even reading this blog already demonstrates that you understand the value of that scattered knowledge: The information Raymond posts here is a "big unsupported pile", but I’d argue that a single entry in the Old New Thing is often more useful than half the documentation in the SDK.
If I have to choose between Raymond spending his time on (A) recompiling ancient samples for the umpteenth time or (B) fixing bugs in the next release of Windows, I’ll pick B every time. Raymond, just post the samples and forget about them, and spend your time on things that matter.
It looks so nice at first.