The mechanism for keeping track of window hooks was very different in 16-bit Windows. The functions involved were SetWindowsHook, UnhookWindowsHook and DefHookProc. The first two functions still exist today, but the third one has been replaced with a macro:

// 16-bit prototype
DWORD WINAPI DefHookProc(int nCode, WPARAM wParam,
                         LPARAM lParam, HHOOK FAR *phk);

// 32-bit macro
#define DefHookProc(nCode, wParam, lParam, phhk)\
        CallNextHookEx(*phhk, nCode, wParam, lParam)

Disclaimer: All code below is "reconstructed from memory". The spirit of the code is intact, but the precise details may be off.

To install a windows hook in 16-bit Windows, you started by calling SetWindowsHook:

HHOOK g_hhkPrev;

g_hhkPrev = SetWindowsHook(WH_WHATEVER, MyHookProc);

The return value from SetWindowsHook must be saved in a global variable, which we gave the somewhat provocative name g_hhkPrev. The hook procedure itself went something like this:

// In Win16, hook procedures returned a DWORD, not an LRESULT.
DWORD CALLBACK MyHookProc(int nCode, WPARAM wParam, LPARAM lParam)
{
  if (nCode >= 0) { ... }
  return DefHookProc(nCode, wParam, lParam, &g_hhkPrev);
}

And then when you were finished, you removed the hook by calling UnhookWindowsHook:

UnhookWindowsHook(WH_WHATEVER, MyhookProc);
g_hhkPrev = NULL;

Internally, the chain of hook functions was managed as a linked list, but instead of using some internal data structure to keep track of the hooks, the linked list was managed inside the HHOOK variables themselves.

The internal implementation of SetWindowsHook was simply this:

// This array is initialized with a bunch
// of "do nothing" hook procedures.
HOOKPROC g_rgHook[NUMHOOKS];

HHOOK WINAPI SetWindowsHook(int nType, HOOKPROC pfnHookProc)
{
 HHOOK hhkPrev = (HHOOK)g_rgHook[nType];
 g_rgHook[nType] = pfnHookProc;
 return hhkPrev;
}

Installing a hook merely set your hook procedure as the head of the hook chain, and it returned the previous head. Invoking a hook was a simple matter of calling the hook at the head of the chain:

DWORD CallHook(int nType, int nCode, WPARAM wParam, LPARAM lParam)
{
 return g_rgHook[nType](nCode, wParam, lParam);
}

Each hook procedure did its work and then sent the call down the hook chain by calling DefHookProc, passing the HHOOK by address.

DWORD WINAPI DefHookProc(int nCode, WPARAM wParam,
                         LPARAM lParam, HHOOK FAR *phk)
{
 HOOKPROC pfnNext = (HOOKPROC)*phk;
 if (nCode >=0) {
  return pfnNext(nCode, wParam, lParam);
 }
 ... more to come ...
}

As you can see, it's all blindingly simple: Invoking a hook calls the first hook procedure, which then calls DefHookProc, which knows that a HHOOK is just a HOOKPROC, and it forwards the call down the chain by merely calling the next hook procedure directly.

The real magic happens when somebody wants to unhook. Recall that the rule for hook procedures is that a negative hook code should be passed straight to DefHookProc (or in modern times, CallNextHookEx). This convention allows the hook system to use negative codes to manage its own internal bookkeeping. In this case, we're using -1 as the "unhook this hook procedure" code.

BOOL WINAPI UnhookWindowsHook(int nType, HOOKPROC pfnHookProc)
{
 return DefHookProc(-1, 0, (LPARAM)pfnHookProc,
                    (HHOOK FAR*)&g_rgHook[nType]);
}

And then the real magic begins:

DWORD WINAPI DefHookProc(int nCode, WPARAM wParam,
                         LPARAM lParam, HHOOK FAR *phk)
{
 HOOKPROC pfnNext = (HOOKPROC)*phk;
 if (nCode >=0) {
  return pfnNext(nCode, wParam, lParam);
 }
 switch (nCode) {
 case -1: // trying to unhook a node
  if (pfnNext == (HOOKPROC)lParam) { // found it
   *phk = (HHOOK)pfnNext(-2, 0, 0);
   return TRUE;
  }
  // else keep looking
  return pfnNext(nCode, wParam, lParam);

 case -2: // report the next hook procedure
   return (DWORD)*phk;
 }
 return 0;
}

And there you have it, the entire window hook system in two dozen lines of code. You have to give 16-bit Windows credit for being small.

Let's walk through hook installation, dispatch, and removal to see how this all works. Suppose there is one WH_KEYBOARD hook in the system. Our variables are therefore set up like this:

// In USER
g_rgHook[WH_KEYBOARD] = Hook1;

// In HOOK1.DLL
HHOOK g_hhkPrev1 = DoNothingHookProc;

DWORD CALLBACK Hook1(int nCode, WPARAM wParam, LPARAM lParam)
{
 if (nCode >= 0) { ... work ... }
 return DefHookProc(nCode, wParam, lParam, &g_hhkPrev1);
}

Now suppose you want to install a new hook, Hook2.

// In HOOK2.DLL
HHOOK g_hhkPrev2;

g_hhkPrev = SetWindowsHook(WH_KEYBOARD, Hook2);

The SetWindowsHook function just puts your function in as the new "head" hook function and returns the old one.

// In USER
g_rgHook[WH_KEYBOARD] = Hook2;

// In HOOK2.DLL
HHOOK g_hhkPrev2 = Hook1;

DWORD CALLBACK Hook2(int nCode, WPARAM wParam, LPARAM lParam)
{
 if (nCode >= 0) { ... work ... }
 return DefHookProc(nCode, wParam, lParam, &g_hhkPrev2);
}

// In HOOK1.DLL
HHOOK g_hhkPrev1 = DoNothingHookProc;

DWORD CALLBACK Hook1(int nCode, WPARAM wParam, LPARAM lParam)
{
 if (nCode >= 0) { ... work ... }
 return DefHookProc(nCode, wParam, lParam, &g_hhkPrev1);
}

Now suppose the window manager decides it's time to fire the WH_KEYBOARD hook. It starts with CallHook which calls g_rgHook[WH_KEYBOARD] that takes us to Hook2. That hook function does its work, then calls DefHookProc(..., &g_hhkPrev2), which dispatches the hook to g_hhkPrev2 == Hook1. Similarly, the hook travels through Hook1, then DefHookProc(..., &g_hhkPrev1), where it finally reaches the DoNothingHookProc which does nothing and ends the hook chain.

Now suppose that HOOK1.DLL decides to uninstall its hook. It therefore calls UnhookWindowsHook(WH_KEYBOARD, Hook1). This starts off the hook chain with the internal hook code -1 and &g_rgHook[WH_KEYBOARD] as the first hook pointer. This activates the case -1 in DefHookProc code path, which dereferences its phk parameter and obtains g_rgHook[WH_KEYBOARD] == Hook2. Since this is not equal to Hook1, the call forwards down the chain to Hook2.

Like a good hook function, Hook2 reacts to the negative hook code by handing the call directly to DefHookProc(-1, ..., &g_hhkPrev2). This time, *phk == g_hhkPrev2 == Hook1, so the test succeeds and we dispatch the hook down the chain with a new internal code of -2, which means, "Tell me what the next hook procedure is".

This dispatch calls Hook1 which (since the notification code is negative) immediately passes the call to DefHookProc(-2, ..., &g_hhkPrev1). This now triggers the case -2 code path, which just returns *phk == g_hhkPrev1 == DoNothingHookProc. This value is returned to the DefHookProc(-1, ...) which stores the result into *phk == g_hhkPrev2; the result is that you have g_hhkPrev2 = DoNothingHookProc. Finally, DefHookProc returns TRUE to indicate that the hook was successfully uninstalled. This value is then returned out from all the nested function calls to the original caller of UnhookWindowsHook.

Observe that at the end of this unhook exercise, we get the desired result:

// In USER
g_rgHook[WH_KEYBOARD] = Hook2; // unchanged

// In HOOK2.DLL
g_hhkPrev2 = DoNothingHookProc; // updated!

DWORD CALLBACK Hook2(int nCode, WPARAM wParam, LPARAM lParam)
{
 if (nCode >= 0) { ... work ... }
 return DefHookProc(nCode, wParam, lParam, &g_hhkPrev2);
}

And Hook1 is out of the hook chain, as we desired.

This really isn't all that complicated. All we did was delete a node from a linked list. It's just that this particular linked list cannot be traversed by just dereferencing pointers. Instead, we have to issue a function call and ask the recursive function to perform the work on the "next" node for us. That's what the negative nCode values are for.

Every time I work through this exercise, I am impressed by how compactly 16-bit Windows was written. In just two dozen lines of code, we managed a linked list of function calls, including a dispatching system as well as arbitrary deletion from the middle of the linked list, and all without any memory allocation.

(And because I know people are going to try to change the topic: Remember, I'm talking about 16-bit Windows, not 32-bit window hooks.)

Next time, we'll look at one way people abused this simple system.