The programming assembly-line
I read Matthew Crawford’s book, “Shop Class as Soul Craft”1, nearly 2 years ago, and in that time no other book has been on my mind more. There are things about it that ring deeply true to me, and others I think are deeply misunderstood. What’s worse, the topic is so raw and close to my heart that the thought of writing about it makes me singe. Anyway, here goes.
The book is a philosophical meditation on the concept of work. What makes work meaningful, satisfying and enjoyable? What makes it dull, boring and soul crushing? He talks about how blue-collar work can be intellectually challenging and satisfying, and how white-collar “knowledge work” can be repetitive and stultifying. Whether or not you agree with his conclusions, his examination of the underlying issues is stimulating, and is peppered with colorful and entertaining episodes from his own life. He was the director of a Washington think tank, but ended up as the proprietor of a bike repair shop specializing in Japanese motor bikes. So he speaks from experience on both sides of the fence, which already gives him more credibility than most authors on the topic. If you want a shorter introduction to some of the core issues discussed in the book, read these two2 essays3 by him.
The primary culprit for the dehumanization of work, according to Crawford, is Taylorism. So we must look at that first, and then we will loop back to what all this has to do with programmers and programming.
Taylorism
Taylorism is so deeply ingrained in every aspect of not just modern commerce but life in general that it is almost impossible to identify it. Just like a fish doesn’t even know that it is in water. If you haven’t read his “Principles of Scientific Management”4, it is a short and easy read, and now available free on Project Gutenberg. After reading it, you will realize that it is one of the founding documents of modern life, as well as the progenitor of the “business book” category.
Briefly, Taylorism has two central tenets:
- Measurement: associate metrics with all aspects of work.
- The separation of thinking and doing: An educated class of managers measures and plans all the work, and is responsible for the overall process, while a class of laborers carries out the implementation of those plans.
Taylor was convinced that the laborers he was studying were “soldiering”–knowingly working slowly to maintain work and wages for all their brethren.
Consider this dialogue that Taylor had with a worker in order to convince him to move more pig iron:
“Schmidt, are you a high-priced man?”
“Vell, I don’t know vat you mean.”
“Oh, come now, you answer my questions. what I want to find out is whether you are a high-priced man or one of these cheap fellows here. What I want to find out is whether you want to earn $1.85 a day or whether you are satisfied with $1.15, just the same as all those cheap fellows are getting.”
“Did I vant $1.85 a day? Vas dot a high-priced man? Vell, yes, I vas a high-priced man.”
“Well, if you are a high-priced man, you will load that pig iron on that car to-morrow for $1.85. Now do wake up and answer my question. Tell me whether you are a high-priced man or not.”
“Vell–did I got $1.85 for loading dot pig iron on dot car to-morrow?”
“Yes, of course you do, and you get $1.85 for loading a pile like that every day right through the year. That is what a high-priced man does, and you know it just as well as I do.”
“Vell, den, I vas a high-priced man.”
“Now, hold on, hold on. You know just as well as I do that a high-priced man has to do exactly as he’s told from morning till night. Well, if you are a high-priced man, you will do exactly as this man tells you to-morrow, from morning till night. When he tells you to pick up a pig and walk, you pick it up and you walk, and when he tells you to sit down and rest, you sit down. You do that right straight through the day. And what’s more, no back talk. Now a high-priced man does just what he’s told to do, and no back talk. Do you understand that? When this man tells you to walk, you walk; when he tells you to sit down, you sit down, and you don’t talk back at him. Now you come on to work here to-morrow morning and I’ll know before night whether you are really a high-priced man or not.”
(emphasis mine)
The net effect of all this was that the worker could, under the appropriate supervision, now load 47 tons of iron a day, up from his previous 12 ½ tons a day. He got a 60% pay raise, while the company moving the iron decreased its labor per ton by about the same percentage (from 9.2 c/ton to 3.9 c/ton). But another windfall for the company was the saving in time. Previously, moving 100 tons using 2 workers would have taken 4 days, whereas now it would take just a little over 1 day.
The assembly line
The earliest well-known enterprise to go whole-hog Taylorist was Ford, with its assembly line for manufacturing cars. The world never looked back.
The assembly line with narrowly specialized roles for each worker is a consequence of the thinking/doing separation. The smarts are in the process, the assembly line. The workers operating it are fungible. Of course, the workers did not like this at all. Skilled craftsmen that built entire artifacts were replaced with machines that required only semi-skilled operators churning out one tiny part of the whole. The pace of their work was determined by the speed of the assembly-line, which was set by managers. They had lost all autonomy and control over their work. Their work had been de-skilled. The transfer of power from skilled, knowledgeable craft workers to “the management” was sealed. Ford made this palatable by paying his workers high wages.5
In his book, Dan Pink6 talks about the three factors that make work meaningful: autonomy, mastery and purpose. The fungible non-distinguishable worker manning the assembly line was stripped of all three.
The drive to specialize
The seeds of Taylorism go back to 1817, when David Ricardo outlined the principle of comparitive advantage. He explained how two parties could gain by making goods they were efficient at and then trading with each other, even if each of them could make the traded good on its own. So even if the other party is worse than you at making something, you are better off trading and focusing on your specialty, because of the saved opportunity costs. (See the Wikipedia article for a number of concrete examples7).
The undeniable math of comparitive advantage is the natural driver for specialization. Taylor brought the concept down from the level of countries to individual workers.
What does this have to do with writing software?
The field of software engineering was built on firmly Taylorist principles. It had no other choice, because that was the default school of thought.
In 1968, NATO called a conference on the subject of “software engineering” (the field did not have a name back then, and NATO’s pick of terminology stuck)8. The industry was having a lot of trouble creating large and complex software, and it was about time something was done about it.
Doug McIlroy expressed what became the rallying cry for the field: “The software industry is not industrialized”!9 He said:
Software production today appears in the scale of industrialization somewhere below the more backward construction industries. I think its proper place is considerably higher, and would like to investigate the prospects for mass-production techniques in software. In the phrase “mass production techniques,” my emphasis is on “techniques” and not on mass production plain. Of course mass production, in the sense of limitless replication of a prototype, is trivial for software. But certain ideas from industrial technique I claim are relevant. The idea of subassemblies carries over directly and is well exploited. The idea of interchangeable parts corresponds roughly to our term `modularity,’ and is fitfully respected. The idea of machine tools has an analogue in assembly programs and compilers. Yet this fragile analogy is belied when we seek for analogues of other tangible symbols of mass production. There do not exist manufacturers of standard parts, much less catalogues of standard parts. One may not order parts to individual specifications of size, ruggedness, speed, capacity, precision or character set.
More than 40 years later that still remains the Holy Grail of software engineering.
McIlroy envisioned a world where we would be constructing software by picking components off a shelf, and snapping them together like Legos. The hard work would be building the right blocks, and then it would be easy to snap them together.
Later, Osterweil played a neat recursive trick and argued that the process of making software could be expressed as software too10. The implication was that it could be executed, just like software.
In some ways this vision has come true, in other ways, it hasn’t.
The biggest example of how the dream of software components has come to pass is web applications. All the components are quite well-understood and, by now, standard. The database. The object-relational mapping. The web server. The Javascript frontend. The webapp developer can pick and choose from a wide variety of choices for each of those. In fact, some grumble that the number of choices adds complexity. All the same, unless you’re building something that must scale to the planet (like Google, Amazon etc), it is ridiculously easy to cobble together a web application that meets 80% of your vision within a day.
Post-modern programming11 starts with Googling, and ends with glue code.
In fact, this componentization has come far enough that many developers complain that the plumbing of pre-existing parts has drained the joy and skill out of their profession12. Does that sound familiar? The craftsman-turned-assembly-line worker of the 20th century had exactly the same lament!
John D. Cook argues that calling somebody “plumber programmer”13 is actually a compliment, because it’s the plumbing and connecting and impedance matching that is hard. But I suspect today that’s true because it’s only the plumbing that is left to do.
How many times can you build a cookie-cutter CRUD app with a web frontend that fetches and displays data from a SQL backend before you feel like all you are doing is cranking a lever? If your Java IDE generates half your code for you, and you auto-complete (Ctrl-Space!) most of the rest, do you feel like you are operating machinery, and not really engaging in expression?14
The pyramid
In other ways, McIlroy’s vision is still a dream.
Massive development effort still goes into applications that are young and not standardized. There are no components that will let you build your own web-scale search engine quickly, for example.
Programmers are certainly not fungible assembly-line operators15.
As in all things in life, there is a hierarchy to these things. There is a small number of elite wizard programmers who do write significant things from scratch. Think of those who have implemented language compilers and runtimes, database engines, search engines, operating system kernels and the like. Unfortunately, such projects come along rarely, and the bar for contribution is extremely high.
(By the way, if you are tired of plumbing together databases and PHP and really want to experience the wild West of programming, go contribute patches to the Linux kernel. No fancy languages or libraries that make you soft in userspace. Just plain old C.)
Some problems, such as building webapps backed by databases, are well understood, and their solutions have become templatized. Others, such as indexing a billion documents, or storing a bazillion petabytes across a gajillion machines, are at the edge of our knowledge, and their solutions are still being actively explored.
If you think you are writing cookie-cutter code, it is because you are working on a cookie-cutter problem.
Gratitude
But we must all realize that at the end of the day, we work on something that is in economic demand (in other words, software engineers are well-paid), intellectually challenging (at least, on most days) and in some small way improves the world and helps people. Even all this self-examination and criticism is a luxury that many do not have.
So, be grateful.