Unusual uses for a ball-point pen: Breaking into the debugger

Date:January 30, 2007 / year-entry #33
Orig Link:https://blogs.msdn.microsoft.com/oldnewthing/20070130-00/?p=28223
Comments:    27
Summary:Before PCI there was ISA. The pinout for ISA slots had one very convenient property: If you shorted the last two pins in an open ISA slot, you generated an NMI. (This trick is alluded to in passing in this discussion of generating crash dumps from hardware.) It so happened that the tip of a...

Before PCI there was ISA. The pinout for ISA slots had one very convenient property: If you shorted the last two pins in an open ISA slot, you generated an NMI. (This trick is alluded to in passing in this discussion of generating crash dumps from hardware.)

It so happened that the tip of a ball-point pen was the ideal shape for accomplishing this. You inserted the tip of the pen at the very end of an open slot (the end nearest the back of the computer) and slowly dragged it towards the front of the computer until it shorted the nearest pair of pins.


If you had the Windows 95 debugger connected to the system (known as WDEB386), it caught the NMI and broke into the debugger. Doesn't matter if interrupts were disabled; you got your debugger. NMI stands for "non-maskable interrupt" and it's the "non-maskable" part that is important here, for it means that there's nothing that will prevent it from happening¹. Very handy if a device driver got itself stuck in an infinite loop. You could tell the real device driver developers from the posers by seeing who carried a ball-point pen with them. (I'll talk more about NMI in a future entry.)

I was reminded of this little hardware trivia by a colleague who pointed out that his first encounter with me was me trying to break into a machine that had wedged by asking him to perform this little maneuver. Over the phone. From 1600 miles away. He was convinced it was some sort of prank.

Bonus ISA trivia: During the transition period from ISA to PCI, motherboards supported both buses. If you looked at a computer from this era, you saw eight ISA slots and eight PCI slots... and eight brackets in the cover for the cards to mount into. How did that work? Shouldn't there be sixteen mounting brackets?

Those PCI folks were so clever. If you looked at an ISA card from above, with the back plate of the computer facing away from you, it looked like this:  The board itself formed the vertical stroke, and the little metal mounting bracket formed the horizontal stroke. The mounting bracket on the case therefore sat slightly to the right of the ISA slot. The PCI folks made their cards a mirror image of the ISA card: If you looked at a PCI card from above, it looked like this:  The mounting bracket on the case sat to the right of the PCI slot.

In this way, you were able to accommodate eight ISA slots and eight PCI slots with only eight mounting brackets. Each mounting bracket supported the PCI slot to its left or the ISA slot to its right.

¹For pedants: Yes, you could actually get a machine so badly wedged that even the NMI had no effect, but you needed the help of poorly-behaving hardware devices.

Comments (27)
  1. jwf says:

    I guess real developers used ball point pens. I recall having a nice red button at the end of a cord that looped into the back of the PC – the periscope switch.

  2. BryanK says:

    I believe SMIs (and perhaps SMM in general?) also disable the NMI.  (This may depend on the interrupt controller, but I doubt it.)  No idea whether Windows has support for SMM, but I don’t see why it wouldn’t.

    And yes, I’ve seen one machine get "so badly wedged" that an NMI had no effect.  It did require "help" from the hardware: I eventually tracked it down to the sound card not liking something about the dual-core CPU (or perhaps SMP in general?).  Dumping the sound card and changing over to the onboard sound chip prevented the hard-lockups.  Disabling one core also helped, but I didn’t want to be limited to one core.

    (How did I know it stopped NMIs from working?  Well, relatively easily: I turned on the NMI watchdog in Linux (most IO-APICs have the ability to set up an NMI watchdog), figuring it was a driver running into a deadlock.  But it never saw an NMI from the watchdog when it was hung.)

  3. Xan says:

    NMIs can be disabled also by outing some values in I/O ports of the chipset (the processor could still listen to them on its NMI pin, but if the wire is cut …)

  4. Hayden says:

    Aha, memories.

    I once was working on an 80188 embedded system using TDREM, the embedded version of the Turbo Debugger remote client. The serial port was a plug-in board, so I arranged the serial interrupt to be NMI and not one of the INT pins.

    If the code was in a loop, you could just press CTRL-break on the PC running TD, and you had control.

  5. BOFH says:

    Ah yes, the NMI…

    I just remembered I installed a switch on the front panel of my Amiga 500+ and connected it to the NMI pins somewhere on the motherboard.

    Next to it I had another switch installed that would short out the crystal, effectively turning off the CPU clock.

    Now why did I have that..?

    I also remember have the Hardware Reference Manual permanently open by the computer those years.

    Uh, the flashbacks…

  6. And of course there was always the IBM PC Jr, which used NMI for the keyboard interrupt.

    When Microsoft commented that it was a bad idea to the IBM hardware guys, they were couldn’t figure out why we were so upset – after all, one interrupt was as good as another, right?

  7. I made my own "Periscope switch" for my original IBM 5150 five-slot PC. I took the game adapter card and soldered a thin coax cable to NMI and ground with a pushbutton at the other end of the cable. A lot of people made these – it was a quick and easy project.

    I think I finally threw this card out a couple of years ago, after it sat in a drawer for ten years!

  8. Ben Cooke says:

    One annoying effect of the inversion of the PCI cards is that the connectors on them are usually "upside down" when installed in a tower case.

    Now, I realise that the correct way up for a D-SUB connector depends on your point of view, but as someone who grew up with connectors with the "big side" up — on my Amiga, on my games consoles and even the on-board connectors on my more modern PCs — having the "small side" uppermost is something I just can’t get used to.

  9. sergio says:

    I still have somewhere the hand-made extension that connected to the ZX Spectrum bus and provided the NMI switch plus some circuit to trick processor to think shortly that it has less memory. That way it "rebooted" and initialized only first X KB leaving the rest intact. Very cool for avoiding to load again the machine programs which stuck (it was not a pleasure loading anything from the tape). The "normal" reset switch I had built-in. Quite a feat for the small case. Both thanks to the magazines of that time and to my friend who was "the hardware guy" (he previously soldered for him all the circuits of his first DIY home computer).

    (And you try and tell the young people of today that ….. they won’t believe you.


  10. Mike Dunn says:

    In my youth, I created NMIs by with RUN/STOP-RESTORE.

  11. Marc Bernard says:

    Reminds me of the time I used a pencil to configure a station…


  12. Stu says:

    I still have several machines with both ISA and PCI slots around and in all of them, the PCI slots are above the ISA slots, such that only one mounting bracket is "shared".

    Were there really ever motherboards where all of them were shared?

    Looking at the design of my computers, there is not enough space between the slots for an extra one, so the case design would have to have been modified.

  13. Erwin Alva says:

    Mike Dunn:

    In my youth, I created NMIs by with RUN/STOP-RESTORE.

    Ah, yes. Those good old VIC-20 and C-64 days…

  14. Jules says:

    "Were there really ever motherboards where all of them were shared?"

    I used to regularly work on Compaq machines that had (IIRC) 4xPCI and 3xISA, and all of the ISAs share with a PCI.  And two of the pairs were the wrong way up, because they branched out from both sides of a daughterboard.

    "One annoying effect of the inversion of the PCI cards is that the connectors on them are usually "upside down" when installed in a tower case."

    A question on this topic: are PCIe cards upside-down or are they back to being the right way up again?  I don’t yet have a PCIe machine, so can’t easily check…

  15. Mike Dunn says:

    I hated those machines where the slots were on a daughterboard. I once had a Dell in a whacky desktop case, where in order to add/remove any cards, you had to pull a lever and pull out the whole assembly: daughterboard, all the cards, and the backplate. And of course, the attached cables always got in the way.

  16. James says:

    This reminds me of the Power Computing PPC systems, which had NMI buttons on the front for debugging purposes, and of AGP, which uses what used to be the ISA position to share a space with a PCI slot.

    Meanwhile, we’ve progressed from using ballpoint pens on ISA slots to using pencils on the CPU itself: one of the overclocking tricks is to use pencil markings to short the cut traces on Athlon CPU packaging, thus unlocking the CPU frequency multiplier.

    Raymond, are you sure you need hardware to be ‘misbehaving’ to block NMIs? Setting the top bit of I/O port 70h is supposed to have that effect without an actual malfunction. (Various other things could also stop the NMI handler firing as expected, of course: trashing the stack it tries to use, installing your own NMI handler in its place, corrupting the handler code…)

  17. steveg says:

    I used to reset the good ol’ 6502 with a screwdriver across the correct pins. Hmmm… just did a quick google, looks like I was shorting RST + CLK, oh well, did the trick.

  18. Jon says:

    Jules: unless you have a BTX machine (Dell, Apple or Gateway), in which the slots become "right side up" again, for cooling purposes. Under some circumstances (difficult in routing the traces), the data lanes become reversed or otherwise switched and the PHY is supposed to fix this. This is referred to as lane reversal or lane swizzling.

    The higher-end Dell servers all have physical NMI switches. They’re disabled by default in the BIOS.

  19. Feroze says:

    Win9x and Wdeb remind me of something. Whenever windows breaks into debugger,and then resumed by pressing ‘g’,it would lose the mouse. This was very annoying, until somebody told me of a trick to get the machine to redetect the mouse. It consisted of the following commands in the debugger:

    o 64 d4

    o 60 e9

    i 60

    i 60

    i 60


    I had no idea why this worked, but it did (most of the time)

  20. KJK::Hyperion says:

    A PCI "death button" is possible as well, but rather more complicated:


    (with circuit schematics!)

  21. Sean W. says:

    Jules:  Glancing into the window in the side of this computer to double-check, the answer is yes, PCIe cards are still "upside-down."

    My personal favorite way of testing device drivers (and new code in an OS kernel), though, is the crashes-vs.-reboots test, which is sorta like printf debugging, only a *lot* harder to suffer through.  On the plus side, it works even if you don’t have a kernel debugger at your disposal.  Just insert code in your device driver at the spots where you think the bugs are such that the driver pulls high the system’s reset line on one side of a conditional test, and falls into an infinite loop on the other side.  (Obviously, you wanna remove that code in the production version; and a little psychic debugging can help a lot here too.)  Thus you now have a boolean status output where you had no status before:  If the machine reboots, it’s a 1, and if it hangs, it’s a 0.

    I don’t recommend doing debugging that way if you can absolutely avoid it; it’s slow and painful.  But it *does* work.

    (Actually, back in the day, I used to keep an ancient parallel dot-matrix printer around for driver debugging, since it was so easy to feed status info out to it, and the output didn’t disappear on a crash or reboot.  But it’s been a long time since I did driver work, and I’m not sure what happened to that old printer now.)

  22. Jules says:

    "o 64 d4

    o 60 e9

    i 60

    i 60

    i 60"

    Port 64 is the keyboard controller. ‘d4’ is the command to send the next byte of output to the mouse.

    Port 60 is the keyboard interface. ‘e9’ is the byte sent the mouse which is described as "Status request. The mouse responds with "acknowledge" then sends the following 3-byte status packet (then resets its movement counters.)"

    You then read 3 bytes from the mouse, which should leave just the last byte of the status packet in the buffer (containing the mouse’s resolution).

    According to the docs, the mouse also sends a single byte status code when it is powered on.  Presumably, on resumption from the debugger, the driver is reinitialised (?) and checks for the presence of the mouse by attempting to read that status byte, and if there isn’t anything there assumes there is no mouse.  Your process leaves a single byte in the right register so the driver is tricked into believing that a just-powered-on mouse is connected.

    Useful references:

    http://maven.smith.edu/~thiebaut/ArtOfAssembly/CH20/CH20-2.html – how the keyboard controller works

    http://www.computer-engineering.org/ps2mouse/ – how a PS/2 mouse works

  23. Mouser says:

    o 64 d4

    This means write next byte to mouse


    o 60 e9

    This means status request, reset counters in mouse


    E9h (Status Request) – The mouse responds with "acknowledge" then sends the following 3-byte status packet (then resets its movement counters.):

    i 60

    i 60

    i 60


    I had no idea why this worked, but it

    did (most of the time)

    I guess if you break in the middle of  reading the mouse, either the mouse or the controller get into a wierd state. This sequence works to reinit the mouse and controller.

  24. strik says:

    Mike Dunn:

    In my youth, I created NMIs by with RUN/STOP-RESTORE.

    No, RESTORE generated the NMI. The RUN/STOP was only needed as the C64 was programmed to ignore the NMI if that key was not pressed.

  25. Been there, done that says:

    Reminds me of the "good ol’ days", when men were men, and…anyway, one fellow I worked with did all sorts of electrical trouble-shooting and never needed a voltmeter….he’d just start prodding the open terminals looking for a "tingle". He told me that if he couldn’t get a good feel he’d grab hold of a water pipe with one hand and keep prodding at the circuitry until he felt something bite at him. He swore he only did this on 110 AC because  that was "safe", but not so much on DC or 480 vac….(go figure)

    I don’t think he did too good on computer mother boards…the voltage was too low to give him a good tingle.

    I guess I was a wimp…I always used a meter.

  26. Your memory is probably bad.

Comments are closed.

*DISCLAIMER: I DO NOT OWN THIS CONTENT. If you are the owner and would like it removed, please contact me. The content herein is an archived reproduction of entries from Raymond Chen's "Old New Thing" Blog (most recent link is here). It may have slight formatting modifications for consistency and to improve readability.

WHY DID I DUPLICATE THIS CONTENT HERE? Let me first say this site has never had anything to sell and has never shown ads of any kind. I have nothing monetarily to gain by duplicating content here. Because I had made my own local copy of this content throughout the years, for ease of using tools like grep, I decided to put it online after I discovered some of the original content previously and publicly available, had disappeared approximately early to mid 2019. At the same time, I present the content in an easily accessible theme-agnostic way.

The information provided by Raymond's blog is, for all practical purposes, more authoritative on Windows Development than Microsoft's own MSDN documentation and should be considered supplemental reading to that documentation. The wealth of missing details provided by this blog that Microsoft could not or did not document about Windows over the years is vital enough, many would agree an online "backup" of these details is a necessary endeavor. Specifics include:

<-- Back to Old New Thing Archive Index