How Unix Won
Unix has won in every conceivable way. And in true mythic style, it contains the seeds of its own eclipse. This is my subjective historical narrative of how that happened.
I’m using the name “Unix” to include the entire family of operating systems descended from it, or that have been heavily influenced by it. That includes Linux, SunOS, Solaris, BSD, Mac OS X, and many, many others.
Both major mobile OSs, Android and iOS, have Unix roots. Their billions of users dwarf those using clunky things like laptops and desktops, but even there, Windows is only the non-Unix viable OS. Almost everything running server-side in giant datacenters is Linux.
How did Unix win?
It was built by programmers, for programmers. If you read the early papers describing Unix, you will see how the key abstractions (hierarchical filesystems, permissions, processes, interactive shells, pipes) have lasted conceptually unsullied for decades. That could only have happened if it exerted such a force field over geek minds that they propagated it.
The majority of it was written not in assembly language, but in C, a higher-level language, hence making it portable. This enabled relatively easy ports to a wide variety of hardware.
A freak historical accident. AT&T Labs, where it was developed, was forbidden under its anti-trust settlement from commercializing products unrelated to its core telecom business. Hence, Unix was licensed very cheaply to universities, including UC Berkeley, which subsequently built one of the more influential branches of the Unix family tree—BSD. Apparently, AT&T classified Unix as industrial waste for tax purposes when licensing it!
Unix spread through academia, and those students spread it through corporations after they graduated. It’s a strategy that has been used by every major tech company, only in this case it was organic.
And then came the Internet, and the whole universe of daemons, tools, protocols, and utilities that undergirded it was built natively on Unix. That wasn’t a huge surprise because a lot of the people that built the Net were Unix natives. BSD open-sourced its TCP/IP stack, kicking off its wide adoption outside the military.
By the late 90s and early 2000s, Linux started taking over the server-side. It has become the ultimate virtuous cycle in open source. When picking a new kernel one is virtually forced to go with Linux because of the huge community and massive engineering that has gone into solidifying it. That’s probably why Android picked Linux even though it was running on the other end of the hardware spectrum.
So how and why would we move past Unix?
Unix, which got standardized into the POSIX spec, has accreted a tremendous amount of complexity over the decades. The POSIX spec is 3000 pages long. Linux now has nearly 400 system calls. What started as a clean, pure, minimal and elegant set of system abstractions has become a complex beast.
The very idea of the OS providing general-purpose abstractions with wide applicability is being challenged. When Unix was created, IO was orders of magnitude slower than CPU. There were enough CPU cycles to burn to provide these high-level abstractions, like a complex hierarchical filesystem. Now IO can easily saturate CPU. At least on the server side, folks just want to get the fastest performance out of their hardware. That’s leading to the rise of frameworks like SPDK and DPDK that bypass heavy OS abstractions for storage and networking in favor of applications directly accessing the raw hardware, and rolling whatever abstraction they do need on their own.
The entire ecosystem in which an OS exists is changing. Application-level experiences are hermetic and make the underlying OS more or less irrelevant. Just look at Android or iOS or ChromeOS. Programming to a virtual machine (like the Java VM) makes programmers much more invested in their PL/runtime than the OS.
The above two points raise an interesting question: what does a modern operating system want? Look, for example, at Fuchsia, which is going in the direction of a microkernel with capabilities, that pushes most drivers and OS services out to userspace. These are ideas that have been floating in academic OS research for decades, but could never gain real mainstream acceptance because of the high barrier to making a viable real-world OS. This is the effect Rob Pike was talking about in his “Systems Research is Irrelevant” talk. But the prior two trends are finally dislodging the iron grip of Unix, and OSs are becoming interesting again!