Using fibers to simplify enumerators, part 2: When life is easier for the caller

Date:December 30, 2004 / year-entry #439
Orig Link:
Comments:    16
Summary:Last time, we looked at how a directory tree enumerator function would have been written if the person writing the enumerator (the producer) got to write the spec. Now let's look at what it would look like if the person consuming the enumerator wrote the spec: #include #include #include #include enum...

Last time, we looked at how a directory tree enumerator function would have been written if the person writing the enumerator (the producer) got to write the spec. Now let's look at what it would look like if the person consuming the enumerator wrote the spec:

#include <windows.h>
#include <shlwapi.h>
#include <stdio.h>
#include <strsafe.h>

enum FEFOUND {
 FEF_FILE,          // found a file
 FEF_DIR,           // found a directory
 FEF_LEAVEDIR,      // leaving a directory
 FEF_DONE,          // finished

 FER_CONTINUE,      // continue enumerating
                    // (if directory: recurse into it)
 FER_SKIP,          // skip directory (do not recurse)

class DirectoryTreeEnumerator {
  DirectoryTreeEnumerator(LPCTSTR pszDir);

  FEFOUND Next();
  void SetResult(FERESULT fer);
  void Skip() { SetResult(FER_SKIP); }

  LPCTSTR GetCurDir();
  LPCTSTR GetCurPath();
  const WIN32_FIND_DATA* GetCurFindData();
    ... implementation ...

Under this design, the enumerator spits out files, and the caller tells the enumerator when to move on to the next one, optionally indicating that an enumerated directory should be skipped rather than recursed into.

Notice that there is no FER_STOP result code. If the consumer wants to stop enumerating, it will merely stop calling Next().

With this design, our test function that computes the inclusive and exclusive sizes of each directory is quite simple:

ULONGLONG FileSize(const WIN32_FIND_DATA *pwfd)
    ((ULONGLONG)pwfd->nFileSizeHigh << 32) +

ULONGLONG TestWalk(DirectoryTreeEnumerator* penum)
 ULONGLONG ullSizeSelf = 0;
 ULONGLONG ullSizeAll = 0;
 for (;;) {
  FEFOUND fef = penum->Next();
  switch (fef) {
  case FEF_FILE:
   ullSizeSelf += FileSize(penum->GetCurFindData());

  case FEF_DIR:
   ullSizeAll += TestWalk(penum);

   ullSizeAll += ullSizeSelf;
   printf("Size of %s is %I64d (%I64d)\n",
    penum->GetCurDir(), ullSizeSelf, ullSizeAll);
   return ullSizeAll;

  case FEF_DONE:
   return ullSizeAll;
 /* notreached */

int __cdecl main(int argc, char **argv)
 DirectoryTreeEnumerator e(TEXT("."));
 return 0;

Of course, this design puts all the work on the enumerator. Instead of letting the producer walking the tree and calling the callback as it finds things, the caller calls Next() repeatedly, and each time, the enumerator has to find the next file and return it. Since the enumerator returns, it can't store its state in the call stack; instead it has to mimic the call stack manually with a stack data structure.

class DirectoryTreeEnumerator {
 DirectoryTreeEnumerator(LPCTSTR pszDir);

 FEFOUND Next();
 void SetResult(FERESULT fer)
  { m_es = fer == FER_SKIP ? ES_SKIP : ES_NORMAL; }
 void Skip() { SetResult(FER_SKIP); }

 LPCTSTR GetCurDir()
    { return m_pseCur->m_szDir; }
 LPCTSTR GetCurPath()
    { return m_szPath; }
 const WIN32_FIND_DATA* GetCurFindData()
    { return &m_pseCur->m_wfd; }

 struct StackEntry {
  StackEntry *m_pseNext;
  HANDLE m_hfind;
  WIN32_FIND_DATA m_wfd;
  TCHAR m_szDir[MAX_PATH];

 StackEntry* Push(LPCTSTR pszDir);
 void StopDir();
 bool Stopped();
 void Pop();

 enum EnumState {

 StackEntry *m_pseCur;
 EnumState m_es;
 TCHAR m_szPath[MAX_PATH];

    LPCTSTR pszDir)
 StackEntry* pse = new StackEntry();
 if (pse &&
                 MAX_PATH, pszDir)) &&
     PathCombine(m_szPath, pse->m_szDir,
                  TEXT("*.*")) &&
     (pse->m_hfind = FindFirstFile(m_szPath,
       &pse->m_wfd)) != INVALID_HANDLE_VALUE) {
  pse->m_pseNext = m_pseCur;
  m_es = ES_FIRST;
  m_pseCur = pse;
 } else {
  delete pse;
  pse = NULL;
 return pse;

void DirectoryTreeEnumerator::StopDir()
 StackEntry* pse = m_pseCur;
 if (pse->m_hfind != INVALID_HANDLE_VALUE) {
  pse->m_hfind = INVALID_HANDLE_VALUE;

bool DirectoryTreeEnumerator::Stopped()
 return m_pseCur->m_hfind == INVALID_HANDLE_VALUE;

void DirectoryTreeEnumerator::Pop()
 StackEntry* pse = m_pseCur;
 m_pseCur = pse->m_pseNext;
 delete pse;

 while (m_pseCur) {

    DirectoryTreeEnumerator(LPCTSTR pszDir)
 : m_pseCur(NULL)

FEFOUND DirectoryTreeEnumerator::Next()
 for (;;) {
  /* Anything to enumerate? */
  if (!m_pseCur) return FEF_DONE;

  /* If just left a directory, pop */
  if (Stopped()) {
   m_es = ES_NORMAL;

  /* If accepted a directory, recurse */
  else if (m_es == ES_NORMAL &&
      (m_pseCur->m_wfd.dwFileAttributes &
                      FILE_ATTRIBUTE_DIRECTORY)) {

  /* Any more files in this directory? */
  if (m_es != ES_FIRST &&
             &m_pseCur->m_wfd)) {
   return FEF_LEAVEDIR;

  /* Don't recurse into . or .. */
  if (lstrcmp(m_pseCur->m_wfd.cFileName,
                   TEXT(".")) == 0 ||
                   TEXT("..")) == 0 ||
      !PathCombine(m_szPath, m_pseCur->m_szDir,
                   m_pseCur->m_wfd.cFileName)) {
   m_es = ES_NORMAL;

  /* Return this found item */
  m_es = ES_NORMAL; /* default state */
  if (m_pseCur->m_wfd.dwFileAttributes &
                      FILE_ATTRIBUTE_DIRECTORY) {
   return FEF_DIR;
  } else {
   return FEF_FILE;
 /* notreached */

Yuck-o-rama. The simple recursive function has turned into this horrible mess of state management.

Wouldn't it be great if we could have it both ways? The caller would see a simple enumerator that spits out files (or directories). But the enumerator sees a callback that it can throw files into.

We'll build that next time.

Comments (16)
  1. Nate says:

    I’ve always been baffled by the purpose of fibers, but it seems pretty clear where Raymond is going and I like what I see.

    Do other platforms (UNIX, Mac) have constructs comparable to fibers? I suppose that you could build fibers on top of threads using a couple of semaphores but that seems pretty roundabout.

  2. Igor Tandetnik says:

    "it seems pretty clear where Raymond is going"

    Probably the same place Matthew Wilson has gone in C++ Users Journal’s February 2004 article "Callback Enumeration APIs & the Input Iterator Concept"

    (paid subscription required).

    In this article, Matthew uses fibers to wrap EnumWindows API into STL-like iterator.

    "Do other platforms (UNIX, Mac) have constructs comparable to fibers?"

    POSIX has (deprecated) functions getcontext, setcontext, swapcontext and makecontext. These are effectively equivalent to fibers.

    Then there’s GNU Pth library that supports non-preemtive cooperatively scheduled "threads" (which are closer to Windows fibers than to Windows or POSIX threads)

    Disclaimer: I have not personally used Windows fibers, nor getcontext et al, nor Pth

  3. Ben Cooke says:

    I’m getting a hankering for co-routines.

    (I’m also starting to get the idea that Fibers are basically just co-routines, but I’ll wait until tomorrow to see if I’m right!)

  4. Memet says:

    Amen Raymond. I’ve been waiting to see a use of Fibers all my (programming) life.

    Enough with the previews already, Get on with the show! =)

  5. Ovidiu says:

    Besides SQL Server, is there any other product on the market that can actually use fibers for something useful?

  6. UNIX has ucontext, but it is not implemented everywhere:

    This paper describes a portable method to do the above:

  7. Joe White says:

    Indy can use fibers. Indy is an open-source TCP/IP library (with classes for just about every Internet protocol you’ve ever heard of, both clients and servers), written in Delphi, usable from both Win32 (in Delphi) and .NET (from any .NET language). Last I knew, though, the fiber support was only in the Win32 version. I’m pretty sure Indy-with-fibers is used in some high-end products, because the fiber support took them a lot of engineering. I couldn’t offhand tell you which high-end apps it’s used in, but you could check out their Web site (

    I heard Kudzu (Chad Hower, one of the main coders on Indy) talk about this stuff at last year’s BorCon, so I know a decent bit about it. If you compile Indy with fiber support, then it uses manually-scheduled fibers together with some sort of low-level OS calls (might be I/O completion ports, but don’t quote me on that) to build very high-performance TCP servers. (Transparent, too — you write the same code either way, then just compile Indy with fibers or not.) I don’t remember all the details, but if you’ve got hundreds or thousands of simultaneous connections, it’s supposed to be a substantial performance boost. (On the other hand, if you’ve only got a few connections, it actually slows things down compared to the non-fiber implementation, so you have to plan accordingly. Why can’t optimization ever be easy? (grin))

  8. I’m looking forward to seeing tomorrows blog. While we are waiting, here is a way to do this sort of thing in C cleanly using the preprocessor. It still requires a state machine, but the state machine is generated for you automatically:

  9. Fibers let both sides think they’re in control.

  10. Cooney says:


    (On the other hand, if you’ve only got a few connections, it actually slows things down compared to the non-fiber implementation, so you have to plan accordingly. Why can’t optimization ever be easy? (grin))

    Is it really a problem? With 10 connections, the relative overhead may be more, but does it impact performance?

  11. Brian says:

    David: An interesting article. A downside not addressed there is that by using static storage instead of a second stack, you pick up reentrancy issues. But if that’s not a concern, it’s a clever hack.

  12. Richard says:

    I’m getting a hankering for co-routines.

    You have seen this… using fibers to implement co-routines?

  13. Raymond Chen says:

    Noting that the article uses undocumented CLR functions and even admits that it can deadlock with the garbage collector. Makes you wonder why anybody would follow the advice in that article in a real program. The last sentence of the article should scare everybody.

  14. A while back there was an article in MSDN magazine about wrapping up the unmaged fibers API to implement…

Comments are closed.

*DISCLAIMER: I DO NOT OWN THIS CONTENT. If you are the owner and would like it removed, please contact me. The content herein is an archived reproduction of entries from Raymond Chen's "Old New Thing" Blog (most recent link is here). It may have slight formatting modifications for consistency and to improve readability.

WHY DID I DUPLICATE THIS CONTENT HERE? Let me first say this site has never had anything to sell and has never shown ads of any kind. I have nothing monetarily to gain by duplicating content here. Because I had made my own local copy of this content throughout the years, for ease of using tools like grep, I decided to put it online after I discovered some of the original content previously and publicly available, had disappeared approximately early to mid 2019. At the same time, I present the content in an easily accessible theme-agnostic way.

The information provided by Raymond's blog is, for all practical purposes, more authoritative on Windows Development than Microsoft's own MSDN documentation and should be considered supplemental reading to that documentation. The wealth of missing details provided by this blog that Microsoft could not or did not document about Windows over the years is vital enough, many would agree an online "backup" of these details is a necessary endeavor. Specifics include:

<-- Back to Old New Thing Archive Index