What exactly is a subsystem? - terminology

Question

I'm reading a book which says there are these subsystems:
win32,os/2,posix,etc..
But I don't have any perceptual knowledge with these notations, can you explain it in short words?

Answer 1

I get the feeling the concept of a "subsystem" is somewhat ill-defined, or at least used with different meanings in different contexts.
According to MSDN documentation:
Environment subsystems are Windows NT processes that emulate different operating system environments. The Windows NT executive provides generic services that all environment subsystems can call to perform basic operating system functions.
Windows Internals book talks about the following two subsystems:
Windows subsystem of which it says - "this [subsystem] is special in that Windows can't run without it. (It owns the keyboard, mouse and display and it is required to be present even on server systems with no interactive users logged in. In fact the other two (which two?) subsystems are configured to start on demand, whereas the Windows subsystem must always be running."
Subsystem for Unix-based Applications, also known as SUA[POSIX] subsystem
Now, the /SUBSYSTEM option that can be sent to the Microsoft VS C++ linker in its documentation says and I quote
You can specify any of the following subsystems:
BOOT_APPLICATION
An application that runs in the Windows boot environment. For more information about boot applications, see About the BCD WMI Provider.
CONSOLE
A Windows character-mode application. The operating system provides a console for console applications.
Extensible Firmware Interface (EFI) Image
The EFI subsystem options describe executable images that run in the Extensible Firmware Interface environment. This environment is typically provided with the hardware and executes before the operating system is loaded. The major differences between EFI image types are the memory location that the image is loaded into and the action that's taken when the call to the image returns. An EFI_APPLICATION image is unloaded when control returns. An EFI_BOOT_SERVICE_DRIVER or EFI_RUNTIME_DRIVER is unloaded only if control returns with an error code. An EFI_ROM image is executed from ROM. For more information, see the specifications on the Unified EFI Forum website.
NATIVE
Code that runs without a subsystem environment—for example, kernel mode device drivers and native system processes. This option is usually reserved for Windows system features.
POSIX
An app that runs in the POSIX subsystem in Windows.
WINDOWS
An app that runs in the Windows graphical environment. This includes both desktop apps and Windows Store apps.
WINDOWSCE
The WINDOWSCE subsystem indicates that the app is intended to run on a device that has a version of the Windows CE kernel. Versions of the kernel include PocketPC, Windows Mobile, Windows Phone 7, Windows CE V1.0-6.0R3, and Windows Embedded Compact 7.
So there you go. Finally, people sometimes talk about the "Win32" subsystem, which I don't know if I should take to mean the "windows" subsystem or the "console" subsystem in the linker option sense.
Back to the Windows Internals book, it further says "each executable image (.exe) is bound to one and only one subsystem" which would explain the need to specify the subsystem your app is for at link-time.

Answer 2

Windows starting from NT (NT 3.1) is able to support semantics of different operating systems (or OS families) that existed at that time (1993). Microsoft called them Subsystems (today they would probably call them emulation layers).
When linking against a subsystem, it decides how your semantics will be. For the Win32 subsystem, for example, filenames are case insensitive (foo.txt and fOo.Txt refer to the same file) and device files (like con or nul) exist in every directory. For the POSIX subsystem, file names are case sensitive and device files exist only at one place. By linking existing (legacy) applications against a subsystem different from Win32, these apps "feel" more like the respective OSes and porting work is reduced.
If you want to know the subsystem of an EXE/DLL, you can open it in DependencyWalker - if it (directly or indirectly) depends on KERNEL32.DLL it is Win32 subsystem, if it (directly) depends on NTDLL.DLL it is native subsystem (Note that KERNEL32.DLL will itself depend on NTDLL.DLL, providing the compatibility layer for the Win32 subsystem).
This is mostly obsolete today. I say mostly as Microsoft included a new "Linux subsystem" in Windows 10 Anniversary update (which is a subsystem like Native, Win32 or POSIX), that behaves binary equivalent to Linux, making it easy to compile Linux applications to be run on Windows (or more precisely, its Linux subsystem).
The /SUBSYSTEM linker switch started out to do exactly the same, but was augmented with more options later (/SUBSYSTEM:CONSOLE also compiles for the Win32 subsystem but the application will allocate a console window if it did not inherit one from its parent process, /SUBSYSTEM:EFI_APPLICATION will compile an executable that cannot run on Windows at all, but will run in the Exensible Firmware Interface (EFI/UEFI) boot environment, etc.).

Answer 3

It might help if we knew which book you're referring to!
More generally, Win32 (which is 32-bit Windows, i.e. Windows NT 3.5 or later), OS/2 and the POSIX family are all operating systems. (POSIX is a standard family of APIs into the UNIX-like operating systems - see here for more.)
It sounds like what you describe is a program that can run on many different operating systems and which has operating-system specific components -- these would be the "subsystems".
However, creating an application in this way does sound like the kind of thing that was done fifteen or twenty years ago. That's about the time that people used to refer to those three families of operating systems, too...