| Date: | July 18, 2006 / year-entry #238 |
| Tags: | other |
| Orig Link: | https://blogs.msdn.microsoft.com/oldnewthing/20060718-32/?p=30483 |
| Comments: | 27 |
| Summary: | The designers of 32-bit Windows didn't have to worry quite so much about squeezing everything into 256KB of memory. Since modules in Win32 are based on demand-paging, all you have to do is map the entire image into memory and then run around accessing the parts you need. There is no distinction between resident and... |
The designers of 32-bit Windows didn't have to worry quite so much about squeezing everything into 256KB of memory. Since modules in Win32 are based on demand-paging, all you have to do is map the entire image into memory and then run around accessing the parts you need. There is no distinction between resident and non-resident names; the names of exported functions are just stored in the image, with a pointer (well, a relative virtual address) to the name stored in the export table. Unlike the 16-bit ordinal export table, the 32-bit ordinal export table is not sparse. If your DLL exports two functions, one as ordinal 10 and one as ordinal 1000, you will have a 991-entry table that consists of two actual function pointers and a lot of zeros. Thus, you should try not to have large gaps in your ordinal exports, or you will be wasting space in your DLL's export table. As I noted above, there is only one exported names table, so you don't have to distinguish between resident and non-resident names. The exported names table functions in the same manner as the exported names table in 16-bit Windows, mapping names to ordinals. Unlike the 16-bit named export tables, where order is irrelevant, the exported names table in 32-bit Windows is kept sorted so that a more efficient binary search can be used to locate functions. As with 16-bit Windows, every named function is assigned an ordinal. If the programmer didn't assign one in the module definition file, the linker will make one up for you, and as with 16-bit Windows, the value the linker makes up can vary from build to build. However, there is a major difference between the two models: Recall that named exports in 16-bit Windows were discouraged (on efficiency grounds), and as a result, every exported function was explicitly assigned an ordinal, which was the preferred way of linking to the function. On the other hand, named exports in 32-bit Windows are the norm, with no explicit ordinal assignment. This means that the ordinal for a named export is not fixed. For example, let's look at the ordinal that got assigned to the kernel32 function
Now, some people are in the habit of reverse-engineering import libraries, probably because they can't be bothered to download the Platform SDK and get the real import libraries. The problem with generating the import library manually is that you can't tell whether the ordinal that was assigned to, say, the This unfortunate choice on the part of the import library generation tools to live dangerously has created compatibility problems for the DirectX team. (I don't know why DirectX got hit by this harder than other teams. Perhaps because game developers don't have the time to learn the fine details of Win32; they just want to write their game.) Since they used one of these tools, they ended up linking to DirectX functions like There are other reasons why you cannot generate an import library from a DLL; I'll pick up those topics later when I talk about import libraries in more detail. Next time, forwarders. |
Comments (27)
Comments are closed. |
I had the same problem as the DirectX team. We were stuck using the Borland development environment which uses a different object file format, so we had to use their implib tool to generate import libraries for certain Microsoft DLLs. When it came time to develop our own in-house DLL for our applications, we ran into the "import by ordinal problem", so I had to use the .DEF file to explicitly assign each exported function an ordinal.
After a while I got tired of manually editing the .DEF file, so I made a BASH script (running under cygwin, but I’m sure it will work on a direct WIN32 port of bash) to process the DLL’s header files and automatically assign recently added functions to the .DEF file with the appropriate ordinals.
MFC’s runtime DLLs do their exporting by ordinal with no name in order to save space. So many functions are exported by the DLLs (6,443 of them in MFC 7.1) that the name table would be massive, especially since they would be C++ decorated names.
(Any idea why the ordinals start from 256?)
You can of course use ordinals with GetProcAddress, but I don’t recall ever doing it. I’d thought that Windows CE’s coredll.dll only offered ordinals but then recalled that passing names to GetProcAddress worked, so that must have been a faulty recollection.
So how are “mere mortals” supposed to use all those unnamed, undocumented ordinals in Windows DLLs that are used by IE, Office, etc? (Rhertorical question, of course the official answer is that we are not supposed to use them.)
If Microsoft insists on using what amounts to their own private API, programmers will reverse engineer that API, even if it does create “unsafe” applications.
As for DirectX, wasn’t the SDK originally only available to owners of Visual Studio (i.e. $600 a pop)? Maybe that’s why games programmers generated their own import libraries? At least in the early days, most games programmers were using DOS-based development tools (as most games were DOS-based before DirectX 5). Many of these tools had compilers that could generate Win32 executables, so I doubt many games shops would want to spend thousands of dollars for Visual Studio licences (plus all the lost productivity while programmers learn VC++’s awful GUI) when they could just generate their own import libraries for free.
No, anyone could get the DirectX SDK, no purchase required.
Now, I do recall that some older compiler tools from other vendors had problems with the supplied import libraries. If you were too cheap to upgrade to newer tools, then I suppose that might encourage you to reverse engineer an import library from the DLL.
However, that’s still not really an excuse, since what you should be doing is linking by name, not by ordinal. The ordinals were never documented (for exactly the reason Raymond describes here), so linking by ordinal was really asking for trouble. It’s perfectly possible to build an import generator which does everything by name – it just seems that’s not what people did.
I suspect what happened is that the less clueful developers just got some random import library generator tool they found on Teh Internets and didn’t really investigate what it was actually doing. It appeared to work, so that was good enough. They were blissfully unaware that they were relying on something undocumented until the next version came out and horked them.
Mike: Are they that many? There are some that are not mentioned in the import libraries and some have names that are not descriptive at all, but I still think they have a name, just not a good one.
@Tony Cox: The problem was that linking by ordinal was a recommended practice for improving load times, as was rebasing DLLs to reduce loader fixups. I went looking for the proper references in the MSDN but the first one I came across was Matt Pietrek’s Under the Hood article <http://msdn.microsoft.com/msdnmag/issues/0500/hood/>. In it he says "[Optimizing executables typically] means basing and binding, but might also include importing functions by ordinal or changing executable page alignment. Intuitively, these strategies should make your code load faster." He goes on to perform some real world tests and finds that on Windows 2000 binding by ordinal only improved load times by four percent. So perhaps this "common knowledge" about importing by ordinal is simply a hang-over from the 16-bit days when DLL exports were not sorted.
What is counted as part of IE and what is part of Windows?
Is shell32.dll part of windows or IE? What about shlwapi.dll? Comctl32.dll?
Both Internet Explorer and windows shell components make a LOT of calls into those dlls and I suspect some of them are not documented.
[If there are any undocumented calls from IE or Office into Windows, I want to know about them. (Note, however, that I am not interested in internal calls from IE to itself or Office to itself. Programs certainly allowed to use their own internal functions. That’s why they’re internal! And of course if there’s an internal Windows API that IE doesn’t call, then clearly that’s not interesting either.) -Raymond]
[The formal separation is buried in some 500-bazillion-page legal document which I have no desire to read. -Raymond]
Fine, then how can you expect us mere users and developers on windows to know where the boudary lies.
On another note, has anybody actually made the header files in the platform SDK compile with gcc.
I certainly hand problem doing about 2 years ago.
Surely implib should be fixed to generate bind by name libs if binding by ordinal is deprecated.
"Fine, then how can you expect us mere users and developers on windows to know where the boudary lies."
Why should you care?
The separation between Windows and e.g. IE for what is clearly Windows-components (such as common controls)… simply isn’t anymore.
I’ve always disliked this practice of Microsoft, to even let the application groups modify system components – if it’s installed with the operating system, and in the case of common controls just about everything usermode depends on it, then by $DIETY it is a system component. What’s perhaps even worse is that they change even how the system (from the users experience) behaves.
Just as a thought experiment, imagine Mozilla deciding to overwrite system components, e.g. shipping their own shell32.dll.
I’ve said it before, and I’ll probably say it again, making Explorer.exe use HTML and putting this crap into the core system was one of the worst things MS ever did – both to itself, but even worse; to all its users. The flood of viruses exploiting this crap (’cause crap code it was, else it wouldn’t be the security nightmare, nay disaster, it still is).
And all this just to destroy Netscape…
Raymond: "If there are … from IE"? Just run depends on the IE5 executable shipped with NT5. shlwapi.dll? That’s a system DLL (it’s documented so in its version information).
"Fine, then how can you expect us mere users and developers on windows to know where the boudary lies."
Easy – Everything on MSDN is yours to play with, everything else is not.
I think that game designers have a very different perspective on software development than do, for example, enterprise warehouse management or financial control developers. In business, software has to be reliable and it is worth the money to test it and to "do things the right way." But in most game development it isn’t about getting it "right" but getting it done. If software fails you lose a single sale but do not face a lawsuit or massive loss of revenue. Entertainment software has a very different focus even though this might not be the best way for the video game industry to make good products. Must like that time period when a very large portion of the video game playing population had switched to Windows 2000 and Windows XP but the 9x family was still the primary target of video game shops.
Labeling shlwapi.dll as a system dll just because it has a productname of "Microsoft® Windows® Operating System" is wrong, even mshtml.dll (clearly part of IE) has a productname of "Microsoft® Windows® Operating System"
For many (but not all) dlls, you CAN generate an import lib from the dll:
echo EXPORTS > %DEF%
for /f "skip=19 tokens=4" %%i in (‘dumpbin /exports "%DLL%"’) do echo %%i >> %DEF%
lib /def:%DEF% /out:%LIB%
IOW, I eagerly await the eventual blog post on the subject. :)
Does the Office team get a "special" SDK then? Are the MSDN libs sanitized versions of some internally distributed Win32 SDK?
Chris: no, they aren’t sanitized. Pretty much all import libraries contain all of the undocumented imports. To use them in your code you just need to declare the prototype, which often is public albeit undocumented – and Raymond said the Office team used to reverse-engineer them long before they had access to Windows internals. I think the only exception are the SystemFunctionXXX exports of advapi32.dll
PingBack from http://blogs.msdn.com/oldnewthing/archive/2006/07/19/671238.aspx
Stu: most unnamed ordinals can be resolved to names thanks to the public PDB symbols, like the huge amount exported by shell32.dll and shlwapi.dll. Just, don’t actually use them :-) That is, unless they have already been documented as part of the EU settlement (you’ll find a number of them are still undocumented, but most are Unicode stubs for Windows 95/98, and now there’s unicows.dll for that)
Yeah, sure. Office uses them. That’s why Office applications are honor guests in the hall of shame of the Application Compatibility database. Do you aspire to that? :-)
Jonathan: Internet Explorer *is* Windows. I mean, Internet Explorer 7 is distributed as a Windows patch!
Mike: Mac OSX uses HTML, CSS, and Javascript in its Dashboard. Do you consider this to be a bad idea? Would inventing a new language be better in the overall scheme of things?
> Now, some people are in the habit of reverse-engineering import libraries, probably because
>they can’t be bothered to download the Platform SDK and get the real import libraries.
AFAICT, the usual reason is that the EULA for the Platform SDK doesn’t allow redistribution of the import libraries, so if you’re packaging up a compiler, you don’t really have much choice but to make your own version by reverse engineering the system DLLs.
> it was only by browsing some of the Mozilla source code that it dawned on me what the difference between SW_NORMAL and SW_RESTORE is
I find that the WINE source code is invaluable for this kind of thing. I had a piece of code a while back that just wasn’t working right. My user-defined data parameter to a window I was creating was getting munged in some way that I couldn’t quite figure out. The MSDN docs were silent on the issue, I had no idea what was going wrong. In desperation I turned to the WINE source code for CreateWindowEx. Just doing a quick dry-run over it revealed the problem: if you have WS_EX_MDICHILD in your extended styles, the user-defined data is replaced with an internal structure that’s documented not in CreateWindowEx but in CreateMDIChildWindow (or whatever the API is called). I could have spent weeks trying to figure out what was going on if I didn’t have access to that source code, or if WINE hadn’t implemented the behaviour. This just isn’t documented *anywhere*. (Or at least wasn’t when I was trying to figure it out, sometime around 2002 I think)
Regarding josh’s post above, here’s some more info on creating import libraries from DLLs: http://spiff.tripnet.se/~iczelion/importlib.html
Preserving the spirit while accommodating separate address spaces and new processors.
How were functions exported in windows 3.1 (enhanced mode and win32s)? As in 16-bit real/standard mode or as native win32?
For reference.
PingBack from http://blogs.msdn.com/oldnewthing/archive/2006/07/28/681473.aspx
I found this list of article on Raymond's blog . Raymond's blog is one of the more interesting