Win32 System Programming

Critical Comments

This section contains a variety of comments criticizing different aspects of the Win32 design and functionality. These comments are my own or, in some cases, have come from readers and course participants. Disclaimer: I generally like Win32. The comments here should be taken in that spirit.

Send your Opinions

Dissenting or additional opinions on any of these subjects are welcome, and I'd be happy to post them here. UNIX (or MacOS, or … ) vs. Win32 technical opinions would be especially welcome, but will go in a separate section. Send your comments directly to jmhart@world.std.com.

Introduction

The Win32 API is, in general, fairly consistent, logical, and easy to learn. However, there are times when I wondered whether someone was asleep during the design reviews. I say this with all due respect for the Win32 designers and for the tremendous pressure under which they must have operated. Furthermore, as the length of this book's errata sheet proves, perfection is difficult to achieve. Nonetheless, after you use the API for a while, you start to see things that could be more logical, consistent, and easier to use.

Now that the API is established, however, programmers will have to deal with it for many years to come, and is seems fair to make a few critical comments. These comments and observations might help programmers avoid some simple mistakes and assumptions. Some comments are rather minor, even petty, but some are more serious. Many of the comments are merely a matter of aesthetics and taste. Some can be fixed with new functions, but in other cases we are stuck with the design.

Next to each comment, I've put a personal "seriousness grade" of A (most serious) to C (least serious). The comments are grouped into several sections.

Handles - A

Win32 is to be praised for using the HANDLE data type to refer to nearly anything. In general, we call CreateWidget to get a HANDLE for a hypothetical "widget" object. After calls to ManipulateWidgetThisWay and ManipulateWidgetThatWay (always using the HANDLE), we finally call CloseHandle. We can secure these objects at the time of the Create call, change the security, and duplicate the handles. There are a few exceptions to this otherwise commendable consistency, however, and the exceptions are often inexplicable. In some cases, one might justify the inconsistencies by noting that the objects are not sharable across processes.

Legacy Handles. DLL handles (see LoadLibrary and GetProcAddress) are an example of "legacy handles", left over from Win16, that are treated differently. HWND handles, are, of course, totally different beasts.
Heap Handles. Heaps are created with HeapCreate (rather than CreateHeap, which would be more consistent) and closed with HeapDestroy.
File Search Handles. FindFirstFile returns a handle, which is then closed with FindClose rather than CloseHandle.
Lock Handles. LockFile and LockFileEx would be more convenient to program, in my opinion, if they returned handles. As it is, you are required to retain the lock range.
Critical Sections. There is CRITICAL_SECTION data type, but there is never a HANDLE for them. And, since when is "delete" (as in DeleteCriticalSection) the opposite of "initialize" (as in InitializeCriticalSection).
The Registry. Here, we have HKEY throughout.

Invalid Handle Values - B

A Create call returns a handle to a specified object, normally as a function value, but, as in the case of CreateProcess and DuplicateHandle, the returned handle can be in a parameter or structure. At any rate, it would be convenient if either NULL or INVALID_HANDLE_VALUE were used consistently in all cases. This could almost be rated an "A severity" problem as it is all too easy to test for the wrong value, thus letting errors slip by.

This problem does, in fact, occur in programs such as pipeNP.c. CreateNamedPipe does not operate under Windows 95/98, but it will return a NULL, leading the program to think that the handle is valid. It appears as if the Windows 95/98 implementation is at fault here; it must test to see whether the function is supported and returns a NULL immediately if it is not supported.

JDH reports that GetCurrentProcess(), should it fail, will return INVALID_HANDLE_VALUE under NT and some other value (NULL?) under W95. Note that the documentation does not say which value you should get in the case of failure. This reader, after the call, then called DuplicateHandle on the GetCurrentProcess result, obtaining INVALID_HANDLE_VALUE under W95 and a valid handle under NT.

Ex Functions - B

The Win32 API designers must have had dialogues along the following lines:

Speaker A: "The DoThisToThat function can't perform a 'long this' operation. There is no parameter to specify the operation's range."

Speaker B: "No problem. Just specify a new DoThisToThatEx function and put the range in an additional parameter.

Speaker A: "Sounds good. The specification will be complete this afternoon. And, I'll be certain that the additional parameters are used in a totally new way that no one has ever seen before."

This scenario must have been enacted with ReadFile, WriteFile, Sleep, WaitForMultipleObjects, and WaitForSingleObject to give us "Extended I/O" for files which have the FILE_FLAG_OVERLAPPED flag set. LockFile and UnlockFile were similarly enhanced. The registry functions then received the full treatment to the point where you have to be very careful to remember when the "Ex" suffix is and is not required.

Will there be a DoThisToThatExEx function in the future?

How Many Bytes do You Need? - A

Numerous functions return values whose size cannot be known prior to the call. Such values may be strings, SIDs, Security Descriptors, and so on. There are several ways in which the size can be returned to the calling program, including: 1) the function value, 2) a function "call by address" parameter where the actual buffer size provided by the user is a separate parameter value, 3) a function "call by address" parameter where the current size provided by the calling program is modified by the function, or 4) a separate structure. Win32 seems to mix these styles more or less randomly.

While this may be only an annoyance to some, I have rated it an "A" because of the problem's pervasiveness and the resulting lack of coherence. The security functions are the most seriously impacted.

Here are some examples:

Function Value

GetCurrentDirectory is a "pure" function of the "how many bytes do you need" variety.

Call by Address

LookupAccountName has two separate call by address parameters, one for the SID size and one for the domain name. LookupAccountSid is similar in this way. GetUserName has a single such parameter.

Mixed Call by Address and Call by Value

GetFileSecurity has two parameters for the security descriptor size.

Mixed Call by Address, Call by Value, and Function Value

SetFilePointer is my favorite, even though it relates to file pointers and not the size of returned objects. This function has something for everybody. The high-order file pointer value is set in a call by address parameter, and the function modifies the parameter with the resulting high-order pointer. The low-order pointer is a value, and the resulting pointer is returned as a function value. To top it all off, the return value is overloaded with the error indicator. GetFileSize shows some of the same strangeness. One could argue that the functions were designed to be simple when dealing with "short" files and difficult when dealing with "long" files. Will we see a new class of "Ex" functions in the future?

Separate Structure

GetAclInformation specifies an ACL_INFORMATION_CLASS structure that has a member for the number of bytes already used by the Acl and one for the number of remaining bytes in the programmer-specified buffer.

Annoying Inconsistencies

GetTempPath and GetCurrentDirectory specify the buffer length first and the buffer second. Other functions, such as GetUserName, arrange the parameters in the opposite order.

64-Bit File Sizes and Pointers - B

See the comments above regarding SetFilePointer and GetFileSize. There really will need to be better versions of these two functions as large files become more common. Let's hope that they are not called SetFilePointerEx and GetFileSizeEx.
LockFileEx and UnlockFileEx specify the lock region length with separate parameters and the lock region start address in the overlapped structure.
CreateFileMapping and MapViewOfFile each have two value parameters.
Why not just use a single parameter (a LONGLONG, PULONGLONG, or PLONGLONG) with functions requiring file pointers.

Missing Functionality - A, B, and C

Multiple Wait Semaphores - A

It really would be convenient to have the ability to wait atomically for multiple semaphore units. You need this functionality at times. Take a look at Multiple Wait Semaphore for my solution to this problem. A standard, Microsoft-supported Win32 functions set would be very desirable. It appears that the designers did not want to destroy the coherent architecture that applies to events, semaphores, and mutexes.

Thread Handles - A

There appears to be no way to find a thread's handle from the thread ID. There is a GetThreadHandle function, but it is not what we need. I've been told that there is a method to get a thread handle and will track it down. See Page 207 comments for a method to obtain the thread handle with an undocumented function."

Interprocess Signaling - B

Console control handlers are good, and you can send signals between processes that share the same console using GenerateConsoleControlEvent. It really would be convenient to be able to send signals between unrelated processes, as the console-sharing requirement is far too restrictive. As it stands, cooperating processes can have threads listening on a mailslot or named pipe to shut down a process, but this is not the most convenient method.

HANDLE Types - B

There appears to be no way to determine a HANDLE's type. That is, does a specific handle refer to a thread, process, file, etc? The GetHandleInformation function does not provide this information.

HANDLE Reference Count - C

There appears to be no method to determine how many handles are open on a given object. Additional information could distinguish between references in other processes and in the current process.

File Length by Name - C

There is no convenient way to get the file length by name, rather than handle. I've used GetCompressedFileSize to find out whether a file is empty, but in other cases I'd either have to get a file handle using CreateFile, call GetFileSize, and then close the hanlde. FindFirstFile is another alternative.

fork() - C

There is no equivalent to the UNIX lone fork() function, and it is not at all easy to emulate. This has not bothered me, as CreateProcess has met my needs (hence the C- rating), and threads can be used to get much the same effect as fork(). However, it is extremely difficult to obtain an accurate emulation of the UNIX semantics, and there may be some situations that really do require the lone fork(). Any opinions out there among you readers?

JDH is happy about this situation, reporting that, under UNIX, if you fork() when many threads are running and one or more hold locks, you get an incredible mess. NT does not have this problem.

Pointers - The Long and Short of it - B

Win32 gives us a 32-bit operating system, which is good. There is no need at all for "short" 16-bit pointers or any of the Win16 memory models that grew out of the old Intel memory architecture. So why do we sometimes have LP (as in LPVOID, LPCTSTR, and LPSECURITY_ATTRIBUTES)? And why is the L sometimes omitted (my preference, as in PHANDLE and PSECURITY_DESCRIPTOR). Why not just get rid of LP once and for all and give us a consistent L for everything? Get rid of the legacy stuff and create an environment for programmers who don't care about the old 16-bit world. Do the same for the lp (lower case) that appears with so many parameter names.

Incidentally, let's hope that we don't enter a world of "long long pointers" (LLPVOID, perhaps?) as Win32 evolves to Win64.

How Many Ways Can You Say "Forever" - B

No, this is not the name of a pop song. It is a game you can play when you look at the various functions that may wait and require a timeout period. Here are three examples; you may also want to play the same game with no time out (0 would be good, but what about NMPWAIT_NOWAIT?).

INFINITY. This would be my preferred name, and we can use it with WaitForSingleObject and WaitForMultipleObjects.
NMPWAIT_WAIT_FOREVER is used with WaitNamedPipe and CallNamedPipe.
MAILSLOT_WAIT_FOREVER is used with CreateMailSlot.

Program Safety - B (Perhaps an A)

As programmers, we are, quite properly, admonished to avoid any and all situations that could "crash" a program. Thus, for example, we should be certain that we do not de-reference a NULL pointer. However, many functions that take handles as arguments will cause an exception if the handle variable has not been initialized. Occasionally, you can get an exception if you close the same handle twice in succession. (This is, of course, a foolish thing to do, but Win32 should check the parameters and not generate an exception.)

JDH reports: "The exception that is obtained by closing the same handle twice is C0000008 and has the symbol EXCEPTION_INVALID_HANDLE, which can be found nowhere in the documentation (that I know of); it lives in the Win32 headers, of course. I have noted that I get this only when running in the debugger on WinNT, and I can't figure out why this should be so."

File Links - A

File links of some sort are extremely useful, even necessary. The NTFS actually supports links, but Win32 does not take advantage of this capability. Lack of links is a common headache in UNIX to NT ports.

Commands and Utilities - B

Fortunately, UNIX-style commands, utilities, and shells can be obtained inexpensively (or even for free), and perhaps it is just as well that they are not bundled with Windows. These programs really are indispensable.

How Do I Release the Memory? - C

Several functions, such as FormatMessage and BuildSecurityDescriptor (see InitBuildSD.c) allocate memory for the objects that they create. The programmer ultimately should release the memory to avoid a memory leak. The on-line help advises using LocalFree, which is a "legacy" function. Why not use HeapFree using the process heap?

Create or Open? - C

CreateFile has a flag parameter to specify whether the named file is expected to exist or not and whether an existing file should be overwritten. We are urged not to use the "legacy" OpenFile function. Yet, for file mapping objects, mutexes, events, and semaphores, we have separate Create and Open calls, where the Open only succeeds if the object already exists.

Little Endian and Big Endian - C

Win32, to my knowledge, is only supported on "little endian" machines (does anyone know the situation with Windows CE?), so the low-order portion of a LONGLONG (64-bit integer) should always precede, in memory address, the high-order portion. OVERLAPPED structures do this properly (another picky complaint: why is the field called Offset rather than OffsetLow?), but the WIN32_FIND_DATA structure has it backwards. Also, several function calls, such as CreateFileMapping and MapViewOfFile, require you to specify the two pointer components with the high-order component first.

Remove or Delete? - C

This is one of those little inconsistencies, but it's worth mentioning at least one. I call DeleteFile when I want to delete a file, but I call RemoveDirectory to delete a directory. Why not have, instead, a DeleteDirectory or RemoveFile?

32-bit, 64-bit Addressing, and Windows NT Version 5.0 - Severity TBD

Large scale applications really will need a large virtual address space in the very near future, if not now. Many UNIX implementations already support true 64-bit memory architectures.

Windows 2000 takes some very small, but not very satisfactory, steps in the direction of supporting larger address spaces. Among other things, it will support "Very Large Memory" (VLM) so that a system can have large amounts of physical memory to divide up among processes. Process address spaces can also be larger (up to 8GB, I believe), but this extra address space is not swappable; it appears as if it must be hard-allocated to a process. A large database vendor is already promising to exploit this feature to achieve higher performance.

As I obtain and digest additional information, I'll add a section on NT 5.0. At this time, is looks as if we are going to be revisiting new forms of "extended" and "expanded" memory (remember them?). A true large virtual address space is the only solution in the long run. I'd go so far as to say the NT 5.0 is extremely disappointing in this respect (it does have some good features, however, especially the "zero administration" functionality).

Things I Could Do Without - C

This is only a personal opinion, but I feel that the functionality of any system should be both complete and reasonably orthogonal (that is, there should not be several ways to do the same thing for no good reason). For example, the synchronization functions (see the section below) generally meet these criteria.

It is also desirable not to have capabilities that are not really useful to anyone; there is more to learn, more to maintain, and more ways to make errors. Here are my candidates for functionality that I could do without (feel free to disagree).

LP and Other Remnants of the Past

It would be nice to remove the LPVOID, etc definitions once and for all, replacing them with PVOID, etc. consistently throughout. While we're at it, it would be nice to have a windows.h include file that only exposes the Win32 32-bit functions and does not allow any legacy functions (LocalAlloc, OpenFile, and so on). This would force programmers to use 32-bit functions only (SQA managers should insist on this before they release any code). While we're at it, FARPROC really should be relegated to the dust bin of history.

Asynchronous I/O

Asynchronous I/O is difficult to program, and you can easily achieve the same effect with threads. Furthermore, at least in the tests in the book, the performance is not very good. I/O completion ports may be one example of where overlapped I/O is beneficial. Even then, one could use a semaphore to limit the number of running threads and, again, the programming would be simpler.

_beginthreadex

It would be nice if the thread-safe standard C library were written is such a way that it could initialize its own thread-specific storage. As a result, we could just use CreateThread rather than the rather ugly _beginthreadex.

CreateRemoteThread

Want to have some fun? Loop through process IDs starting from 1. Then use OpenProcess to get a process handle based upon the process ID. If you succeed, then you have a valid process handle. Now call CreateRemoteThread using the process handle and a random address for the thread routine in that process (or, perhaps, a deviously-selected address). Watch the process crash or otherwise misbehave.

Conclusion: TerminateProcess is bad enough, but CreateRemoteThread is a real security and robustness threat. Of course, processes can protect themselves by denying PROCESS_CREATE_THREAD rights, but this takes some effort and is not the default. Sooner or later, if Windows NT is to be truly secure in the enterprise, this issue will need to be addressed.

Warning: I really recommend that you DO NOT carry out the above experiment other than as a thought experiment. The resulting thread could do nearly anything; it might, for example, delete a few files that you are fond of.

JDH suggests, for additional entertainment, that you pass the address of ExitProcess as the thread function, even supplying a reasonable termination code. Notice, of course, that this is the address of ExitProcess in the target process, not the one executing.the CreateRemoteThread call.

HEAP_NO_SYNCHRONIZE

This is a preliminary judgement, subject to revision after I perform some additional tests. I have a colleague, whom I admire, who thinks that this flag (used with HeapAlloc and friends) is terrific. I have found that it has a minimal positive performance impact, at least in simple cases, and using this flag creates certain dangers. There may be situations where the flag is helpful, but there must be other ways to improve performance that are safer. Furthermore, no other functions have such a flag.

A Final Note of Praise where Praise Is Due - Threads and Synchronization

Now that I've been critical, it's time to say something nice.

I feel that the thread management and synchronization functions are well-designed and implemented and that they are very easy (even fun) to use, generally working as expected. A common opinion is that threads and synchronization are difficult subjects and the programming is complex. I disagree; the functions all make sense, the functionality is nearly complete (I would still like to see an atomic multiple semaphore wait function), and the functionality is exactly what you want. Semaphores, mutexes, and events all have a role to play, even though they are not totally independent. (For example, it is possible to create a semaphore out of a mutex and an event. See the example.)

Furthermore, threads and synchronization are a standard part of Win32 and have been there from the beginning. By way of contrast, threads arrived very late in UNIX, and it took a long time for them to become standardized. Many early implementations were not really kernel threads but were more like fibers. Finally, I find pthreads to be clumsy in many respects; for example, why do we need the pthread_ prefix on all the functions? This naming destroys the much-praised UNIX name brevity. And why is it necessary to have a mutex-condition variable pair?

A final point. One reason that threads and synchronization may be considered difficult is that the risks of deadlocks, race conditions, resource starvation, etc. are high. Bear in mind, however, that these risks are inherent to the programming model and have nothing whatsoever to do with the Win32 implementation or design. If fact, Win32 has several facilities (such as WaitForMultipleObjects and abandoned mutexes) that, to some extent, make it easier to avoid these difficulties. Criticizing Win32 because it allows deadlocks makes no more sense than criticizing ANSI C for allowing infinite loops.

I wrote the above before I was aware of the critical section timing issues, dicussed elsewhere. It would also be nice to have the defects fixed. Even so, I feel that the thread design and implementation is fairly sound and the irritations will be removed over time.