There are some basic ground rules that apply to all system programming, so obvious that most documentation does not bother explaining them because these rules should have been internalized by practitioners of the art to the point where they need not be expressed. In the same way that when plotting driving directions you wouldn't even consider taking a shortcut through somebody's backyard or going the wrong way down a one-way street, and in the same way that an experienced chess player doesn't even consider illegal moves when deciding what to do next, an experienced programmer doesn't even consider violating the following basic rules without explicit permission in the documentation to the contrary:
- Everything not defined is undefined. This may be a tautology, but it is a useful one. Many of the rules below are just special cases of this rule.
- All parameters must be valid. The contract for a function applies only when the caller adheres to the conditions, and one of the conditions is that the parameters are actually what they claim to be. This is a special case of the "everything not defined is undefined" rule.
- Pointers are not
NULL unless explicitly permitted otherwise.
- Pointers actually point to what they purport to point to. If a function accepts a pointer to a
CRITICAL_SECTION, then you really have to pass pointer to a valid CRITICAL_SECTION.
- Pointers are properly aligned. Pointer alignment is a fundamental architectural requirement, yet something many people overlook having been pampered by a processor architecture that is very forgiving of alignment errors.
- The caller has the right to use the memory being pointed to. This means no pointers to memory that has been freed or memory that the caller does not have control over.
- All buffers are valid to the size declared or implied. If you pass a pointer to a buffer and say that it is ten bytes in length, then the buffer really needs to be ten bytes in length.
- Handles refer to valid objects that have not been destroyed. If a function wants a window handle, then you really have to pass a valid window handle.
- All parameters are stable.
- You cannot change a parameter while the function call is in progress.
- If you pass a pointer, the pointed-to memory will not be modified by another thread for the duration of the call.
- You can't free the pointed-to memory either.
- The correct number of parameters is passed with the correct calling convention. This is another special case of the "everything not defined is undefined" rule.
- Thank goodness modern compilers refuse to pass the wrong number of parameters, though you'd be surprised how many people manage to sneak the wrong number of parameters past the compiler anyway, usually by devious casting.
- When invoking a method on an object, the
this parameter is the object. Again, this is something modern compilers handle automatically, though people using COM from C (and yes they exist) have to pass the this parameter manually, and occasionally they mess up.
- Function parameter lifetime.
- The called function can use the parameters during the execution of the function.
- The called function cannot use the parameters once the function has returned. Of course, if the caller and the callee have agreed on a means of extending the lifetime, then those rules apply.
- The lifetime of a parameter that is a pointer to a COM object can be extended by the use of the
IUnknown::AddRef method.
- Many functions are passed parameters with the express intent that they be used after the function returns. It is then the caller's responsibility to ensure that the lifetime of the parameter is at least as long as the function needs it. For example, if you register a callback function, then the callback function needs to be valid until you deregister the callback function.
- Input buffers.
- A function is permitted to read from the full extent of the buffer provided by the caller, even if not all of the buffer is required to determine the result.
- Output buffers.
- An output buffer cannot overlap an input buffer or another output buffer.
- A function is permitted to write to the full extent of the buffer provided by the caller, even if not all of the buffer is required to hold the result.
- If a function needs only part of a buffer to hold the result of a function call, the contents of the unused portion of the buffer are undefined.
- If a function fails and the documentation does not specify the buffer contents on failure, then the contents of the output buffer are undefined. This is a special case of the "everything not defined is undefined" rule.
- Note that COM imposes its own rules on output buffers. COM requires that all output buffers be in a marshallable state even on failure. For objects that require nontrivial marshalling (interface pointers and BSTRs being the most common examples), this means that the output pointer must be
NULL on failure.
(Remember, every statement here is a basic ground rule, not an absolute inescapable fact. Assume every sentence here is prefaced with "In the absence of indications to the contrary". If the caller and callee have agreed on an exception to the rule, then that exception applies. For example, a pointer is prototyped as volatile is explicitly marked as "This value can change from another thread," so the rule against modifying function parameters does not apply to such a pointer.)
Coming up with this was hard, in the same way it's hard to come up with a list of illegal chess moves. The rules are so automatic that they aren't really rules so much as things that simply are and it would be crazy even to consider otherwise. As a result, I'm sure there are other "rules so obvious they need not be said" that are missing. (For example, "You cannot terminate a thread while it is inside somebody else's function.")
One handy rule of thumb for what you can do to a function call is to ask, "How would I like it if somebody did that to me?" (This is a special case of the "Imagine if this were possible" test.)
A very useful cluestick to hit people with who really should know this (or stop coding for a living) is:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dncomg/html/msdn_therules.asp
It covers a number of the most common rules that I’ve found other people break, apart from one. I can’t seem to find anywhere on MSDN the rule that says that NULL is a valid BSTR, and must be accepted as if it were a BSTR of zero length. I’ve got it in a couple of books, seen it around on the net (in particular Eric Lippert’s blog), and know it to be true, but I can’t find an authoritative (i.e. official documentation) URL to point people who aren’t in the same office as me at.
Any help anyone?
This is one of my favorites.
"An output buffer cannot overlap an input buffer or another output buffer."
On a project I used to work on, I debugged several intermittent crashes caused by calls like this:
strFoo.Format("prefix %s suffix", strFoo);
(where strFoo is ATL::CString). At a glance, it’s not immediately obvious that you’re overlapping input and output buffers. In fact, for some string implementations, this wouldn’t be an overlap.
Adam: "A null pointer is a valid value for a BSTR variable. By convention, it is always treated the same as a pointer to a BSTR that contains zero characters."
http://msdn.microsoft.com/library/en-us/wcedcom/html/cerefStringManipulationFunctions.asp
Clearly, Raymond has no sense of adventure.
I have to admit that I occasionally find code that is so incomprehensibly inane that I wonder how the code every managed to work in the first place. Sometimes it violates some of these rules. Sometimes it’s code I wrote ten years ago…
Imagine if you were first handed the keys to a car in an unpopulated city with no instruction. You very well might drive on the lawns. I think that programmers need to be taught what is right many times, taught what is wrong a few times, or learn it on their own, once.
I know plenty of programmers who will never need to care about argument volatility because they will never be charged with handling threading on their own. This is, honestly, probably a good thing.
On the flip side, functions that declare parameters as const MUST NOT change them. I remember tracking down a crash in a call to a Microsoft multimedia library routine many years ago. The author of the routine apparently decided that const meant it was okay to change a byte in the caller’s const string as long as he changed it back before returning. Since the string was declared in a read-only segment this didn’t work too well, even in a single-threaded environment. Imagine how much fun it would have been to track down that bug in a multi-threaded situation if the string had been writeable.
"Pointers are not NULL unless explicitly permitted otherwise."
I guess it’s assumed that most, if not all functions that accept a handle to a device context, allow the pointer to be NULL, or else programs would have to start checking the return value of BeginPaint before passing the DC handle along. Although "indoctrinated" developers may have no problem understanding (accepting) this, more critical people may question why there’s no mention of this anywhere in the documentation?
Example: The function TextOut states only this (hDC): "[in] Handle to the device context." There’s no mention of a NULL hDC.
My favourite rule, from over a decade of telling people how to write Winsock programs, is that a "flags" value that is documented that it "may be any of the following values" can also be zero, which gives the default behaviour. Only when a flag "must be a combination of one or more of these values" do you have to pick at least one.
Quoting Raymond:
"All parameters are stable.
* You cannot change a parameter while the function call is in progress."
I don’t understand why my function can’t change a value of a parameter if the parameter is passed by value and is thus equivalent to a local variable whose lifetime is limited by a function execution time? Thanks.
Raymond II: NULL is indeed an invalid value for TextOut’s HDC parameter. (And which the chk version of Windows reports to the debugger.)
I already explained in this post why it isn’t mention that you can’t pass invalid parameters.
Dmitry: If you pass it by value, then the value you’re changing isn’t the one that you passed to the function. (The function got a copy, and you’re not changing the copy.)
Amazingly, I’ve seen C compilers accept the wrong number of arguments without a warning.
It was some time ago, so I can’t remember whether the header containing the function prototype was correctly included, but even that should generate a warning.
(I can’t remember if it was an old GCC or an (older?) IBM RS6000 ‘cc’. I used both at the time)
There are, of course, interesting times when you really want to do interesting things, when you can get away with ignoring the rules. For example, there’s a lovely perl module, Win32::API, which lets you call arbitrary (winapi calling convention) functions without complining C for every call you want to do. It works by using inline assembler to push the arguments to the stack, then calls the function with too few arguments on purpose.
Is it fragile code? Yes. Is it the least fragile way possible to do what you want done? In this case, yes. Breaking the rules is sometimes neccessary to make the impossible possible.
oldnewthing: "NULL is indeed an invalid value for TextOut’s HDC parameter."
I mentioned checking the return value of BeginPaint for a reason. I haven’t seen any application do this, despite the fact that the programmers would know nothing about the behaviour when passing an invalid value, other than "oh well, it appears to work."
It’s easy when you know the answer. Some aren’t even aware of the questions.
All parameters must be valid.
does it means that the fonction doesnt need to check for range for exemple ?
"does it means that the fonction doesnt need to check for range for exemple?"
It means that a program that passes invalid parameters will experience undefined behavior. The function can do anything it wants. That’s what "undefined" means.
The perl module that conses up a callstack is ultimately following the rules – it just does it in a strange way. When the called function gets control, the correct number of parameters are on the stack.
Noob:
There are different schools of thought on that one. Many people say that routine should always check parameters, but that just leads to slow code. An alternative is to assert parameter checks so they aren’t in release builds.
IMHO, anytime you are jumping a wall (for example, a programmer calling one of your functions inside your library), you should check the input parameters. However, once checked, any routines in your library that you call from inside your library should not check the input parameters since you assume that you know how to call you own code. This helps to protect you from external parameter bugs while not slowing your internal code down with a lot of parameter checks.
Purplet: I think you’re wrong. I’ve read the Kernighan-Ritchie C book (the updated version that follows the ANSI standard), and it does contain the using of "…" in parameter lists, plus how the callee can access the extra parameters.
Now that’s a nice URL – in the future, I’m gonna send everyone to http://blogs.msdn.com/555511.aspx :D
You forgot the sine qua non of correct function calls:
Every time you write a function call, read the function’s documentation.
Obviously I’m exaggerating a little, but not much.
I seem to remember a maxim from the old days: constants arn’t, variables won’t.
Also with Fortran compilers passing a constant (say 1) to a function could result in the constant being changed globally, or at least for that compilation unit, (to say 2). This leads to some very bizarre behavior.
http://www.ibiblio.org/pub/languages/fortran/ch1-8.html#01
Nate: Type "alignment" into the search box.
Of course, "Pointers are not NULL unless explicitly permitted otherwise." makes sense to a C programmer, but a C++ developer community could reasonably assume that a reference would be used unless something interesting is going on.
There are really two types of rule here: those that have no exception, which are better enumerated by the language and platform definitions, and those that are specific to a culture or developers. Code within a project should be consistent, but pretending such rules are broader than that is foolish. Since it doesn’t specify a scope, your list is foolish.
Overall, it’s better to document "p != NULL" or "The range (a, a+N] is readable and disjoint from the range (b, b+N], which is writable" than to assume that they’re "so obvious they don’t need stating!". free and fread differ on the first point; memcpy and memmove differ on the latter. If you think there’s a general case, you need to get out of your box.
While I’m not arguing against this guideline I do have a rant against people doing this "no pointer can be null" rule blindly:
The problem is that the C standard uses pointers to mean both pointers to objects and array iterators but they didn’t provide a way to define a low overhead 0 sized array. The C++ standard requires (T)0+0==(T)0 and (T)0-(T)0==0 and many vector implementations take advantage of this instead of trying to allocate some dummy argument to support this. The C standard isn’t so smart in this area and most standard functions are undefined if you pass it a null pointer even you pass in the size as 0.
Moral of this story, if taking an array and allowing 0 size to be valid, do (size!=0 && ptr==0) instead of (ptr==0) or just don’t do any pointer checks at all. Your users will thank you for it.
Yay, I feel famous! I’m the CornedBee in that linked thread.
By the way, is what I said about SuspendThread correct?
"does it means that the fonction doesnt need to check for range for exemple?"
No. There’s a concept from the current Fortran standards that’s very useful here — the distinction between requirements on the user and requirements on the provider of the interface (which, in the Fortran standard, is the compiler).
That is, when you are using a function, you should not assume that it will check for invalid range, unless told otherwise.
However, when you are writing a function, that’s a different set of constraints. In my opinion, if you’re writing something for general use, you should check for invalid inputs unless you’ve got a good reason (e.g., performance or excessive code complication) not to.
And then, having written the function, you should document whether or not it checks for valid inputs.
Nick, you’re unfairly slighting Fortran compilers. That problem only happened on a few very ancient compilers. Even the worst Fortran compilers from two decades ago are smart enough not to do such things. (The problem, fundamentally, is that Fortran is normally implemented as pass-by-reference, and the compilers were passing a reference to a reused pooled constant rather than a copy of it, and there was no protection to keep it from being overwritten.)
This is compounded by the fact that Fortran programs can be compiled in multiple separate pieces, with the older versions of the language having no C-header-like methods for transferring interface information from one piece to another so the compiler can check it. (This has been much improved in newer versions of Fortran.)
I’m surprised that the problems with lack of compiler-checking for argument matchup don’t happen in C programs, too. Sure, you include the function prototypes in a header file in everything you compile, but what if you change the prototype, recompile only half the code, and then link it all? Or what if you simply use the wrong header file that happens to have a function of the same name in it?
Then meet my &sword++;
:-)
Brooks:
In C, those type of issues do happen a lot. Old C compilers didn’t require you to prototype routines. So they assumed you knew what you were doing. Even after prototypes were required (by either the standard or by compiler warnings), there are things you can do to mess up. For example if you declare your function in one module as taking 2 arguments but in another module declare the prototype with only one argument, C will compile and link just fine. The big difference between C and C++ are the mangled names used by the linker in C++. Those mangled names contain all the argument information thus preventing the prototype error . C doesn’t mangle the names. Thus "void Bob (int)" in one module and "int Bob (char *)" in another both compile down to a routine name of "_Bob".
>> Pointers are properly aligned.
I think (this is not my evening so think is the proper word) that he’s saying that a pointer should point to a memory location which is correctly aligned for the data type you will access.
Given typical sizes for types (16bit for short and wchar_t, 32bit for int and float, 64bit for double) a short* and a wchar_t* should point to a 16bit aligned address (that is, an even number); an int* and a float* should point to 32bit aligned addresses; double* should point to 64bit aligned addresses.
Also care must be taken for pointers.. on 64bit machines, void** should be 64bit aligned.
Raymond is referring to forgivin processors because x86 processors handle misaligned data accesses themselves (if the AC flag is 0, which it usually is) and only a small, often negligible, performance impact incours. Other processors (MIPS, PowerPC, Alpha) may raise an exception on misaligned data access and thus your program might fail to work.
Never pass a pointer to a goto label as an argument.
Nick,
It isn’t a question about "on purpose". In large systems, without well managed prototype files, it happens a fair amount.
In C++ you have to go out of your way to do it. In C, it happens by mistake.
VC has been using natural alignment for a very long time. The default is eight byte alignment which means that data will be aligned by size up to eight bytes in size. In a more geek term, (address MOD size) == 0 for anything eight bytes and smaller.
In general, you don’t have to worry about alignment unless you are packing data into memory or reading structures from disk. If you don’t want to worry about alignment but still want your structures and classes to be small, then order your member variables by size. The biggest problem is having small member variables mixed in with large ones. You end up with gaps in the structure.
“if you declare your function in one module as taking 2 arguments but in another module declare the prototype with only one argument, C will compile and link just fine”
But you can only do this "on purpose", that is to say by having two declarations of the function which are hidden from each other (you’ll get a compiler warning or error if it sees both, depending on the context). It’s possible to conjure the same mistake in C++ by judicious casting. Cast the function to the wrong type in your headers, then call it, and you get the wrong answers.
int (* wrongBob)(char *) = (int (*)(char *)) Bob;
int result = wrongBob("my input"); /* unexpected results */
I’m not sure its necessary to wait for the user to swing the gun round and stuff it into their own mouth before agreeing to shoot them in the foot if they so ask.
Thank you for this! I wish I had this set of rules at my last job. We often had well-meaning but overzealous testers who wanted us to handle invalid pointers and other garbage inputs in APIs that weren’t designed for it. In one project, my team was forced to catch NULL pointers as errors because the testers assumed that the programmers using our APIs were too lazy to check for them (and yet they would somehow handle error returns properly).
Brooks Moses:
> what if you change the prototype, recompile only half the code, and then link it all?
That’s what make(1) is for. You tell it which C files depend on which header files (or you use the compiler to do it for you, as in gcc’s -Mx arguments), you tell it that any .o file can be compiled from a .c file of the same name (or you depend on the default rule that says this), and you tell it which .o files you want to link. Then when you change a header, make will rebuild all the C files for you, before it relinks the final executable.
nmake probably works similarly, though I don’t know what kind of default rules it has, and I don’t know whether the VC compiler can generate header dependencies automatically or not.
"That is, when you are using a function, you should not assume that it will check for invalid range, unless told otherwise.
However, when you are writing a function…you should check for invalid inputs "
So, why are we as developers admonished to provide idiot-proof APIs when the OS developers are apparently held to no such standard?
PingBack from http://www.synesthesia.co.uk/blog/archives/2006/03/22/links-35/
"So, why are we as developers admonished to provide idiot-proof APIs when the OS developers are apparently held to no such standard?"
If you can’t answer this question, just play it safe and idiot-proof your own APIs and assume that no other API is idiot-proof. This is a rule of thumb (http://en.wikipedia.org/wiki/Rule_of_thumb), not any sort of standard.
"though you’d be surprised how many people manage to sneak the wrong number of parameters past the compiler anyway"
VB6 made doing this easy. Try adding a class, and add a public function to that class that has no arguments and returns a variant.
I remember finding you could pass whatever you like to this function and it compiles fine…
But in VB, you could also skip arguments, like somefunction(arg1,,arg3) iirc
Three weeks in one this time… Here’s my links of interest for the three weeks ending April 8th, 2006… Daring Fireball: Windows: The New Classic – John GruberGreat commentary and analysis on Apple’s release of BootCamp