Computers can only play games

Computer Power and Human Reason, chapters 1-3

Colossus Mark 2, 1943, via Wikimedia Commons

Nik Prassas recently republished a nice piece asking, as his headline put it, “must computers play games?” He asks us to imagine playing a war simulation video game, in which we, say, execute drone strikes on enemy targets. We would do so with no hesitation or moral qualms because we know it to be a game, therefore not real. Now imagine, Prassas continues, we were then informed it had not been a game but had been real, and we had personally pulled the trigger on real bombings—we would be overwhelmed with horror and revulsion. (It’s unclear if Nik is aware this is [spoiler alert] exactly the plot of Ender’s Game, but I digress). The sequence of events and actions taken by the operator are precisely the same in either case; the only thing to distinguish them is what Prassas calls “the almost infinite difference in moral gravity,” a human feeling inaccessible to a computer. This is troubling because increasingly computers are carrying out drone strikes. If the only thing to distinguish the situations is a human sense, we have to ask if the computer can conceptualize the difference between a game and real life.

Another way of thinking about that question is—can a computer understand the specific difference in what I have called (in an admittedly rather vague way) the “moral gravity” of, for example, taking a pawn on a chessboard and taking a life on the battlefield. After all, they do both with equally sublime and inhuman efficiency.

Or, more simply—must computers play games?

If the answer to that question is yes, then we may want to think about how much of our decision-making we hand over to systems that suffer from this walled-up-ness in their own operational field without any possibility of transcendence—systems that are so different in their inner workings that they would not even understand the question.

I am currently in the midst of an exceptional book called Computer Power and Human Reason: From Judgment to Calculation by Joseph Weizenbaum, published in 1976. I have not yet finished the book and will have more to say about it later, presumably as the basis for my Masterpiece Polemic Against the Evils of AI. For now I have a more limited point to make regarding the first several chapters, in which Weizenbaum begins from first principles to explain how computer language and programming fundamentally work. His explanations leave no room for doubt in answering Nik’s question: Computers can do nothing but play games.

---

Joseph Weizenbaum was a pioneer of early computer programming who created the world’s first chatbot. Running on the early IBM 7094 computer and first demonstrated in 1966, he named it ELIZA “because, like the Eliza of Pygmalion fame, it could be taught to ‘speak’ increasingly well.” Weizenbaum considers the creation of ELIZA no great accomplishment and is emphatic that its basic game of pattern-recognition and mimicry constituted no understanding on its part (let alone consciousness) and was disturbed by the emotional investment, anthropomorphism, and “powerful delusional thinking” it provoked in its conversational partners.

This was to be the catalyst for the the great shift in his thinking that would occupy the rest of his life, leading to his book a decade later and its thesis, which he articulates succinctly in the introduction as follows: “There are important differences between men and machines as thinkers. I would argue that, however intelligent machines may be made to be, there are some acts of thought that ought to be attempted only by human beings.”

A machine, Weizenbaum tells us, is the embodiment of a law. With mechanical machines these laws are a subset of the laws of nature—a steam engine that turns a turbine or moves a train embodies and enacts the relevant laws of thermodynamics. Barring a breakdown or a revision of its design, a machine will follow its law absolutely:

We set a punch press into motion, and it mangles the hand of a worker who gets too close to it. The very regularity of the machine is its most fearsome property. We put it to its task and it performs, regularly to be sure, but blindly as well. When we say that justice is blind, we mean to commend it as being almost a machine that performs its function without regard to irrelevant facts—but facts nonetheless. To blind justice, whether the prisoner before the bar is rich or poor or is a man or is a woman is irrelevant. To the punch press, whether the material in its jaws is a piece of metal or a worker’s hand is irrelevant. Like all machines, blind justice and punch presses do only what they are made to do—and that they do exactly.

This quality of regularity, of adherence to its law, allows for a more expansive definition of a machine beyond simply mechanical contraptions. A bureaucracy, for instance, is often characterized (derogatorily) as a machine for its rigid adherence to procedure. Freed from the constraints physics imposes on mechanical machines and those ethics imposes on a social machine, we enter the realm of abstract machines. Put another way, an abstract machine is a set of rules and procedures producing an outcome that is also abstract. And, in fact, people have designed and operated abstract machines for thousands of years: we call them games.

A good game is one whose rules are complete yet permissive. The fun is in choosing novel actions and strategies secure in the knowledge we will not stumble into a situation for which no applicable rule exists. This isn’t far off from C. Thi Nguyen’s formulation that the joy of gameplay is exercising agency within a simplified and structured environment, which I’ve written about previously.

In a game, rules amount to the set of permissible states of play and whether it is permissible to move from a given one to another. Weizenbaum uses chess as his primary example, which is purely abstract except insofar as the rules specify time limits on turns—that constitutes its sole point of contact with the real world. The rules of chess specify the board and its number of spaces, the pieces and their movement, the initial setup, capturing pieces, and various required actions like getting out of check. These rules add up to the set of possible actions a player can take on their turn, or put another way, the possible state-changes that would take board state A to any number of board states B. Any appeal to a judge during a chess tournament should not involve any judgment because the rules are complete—the only question that can be asked is “Is this move legal?” for which there should be a definitive answer that can be consulted.

But we could imagine a “game” with no permissiveness at all, in which a rule exists for every game-state that dictates the state-change to occur and the new state to be reached. Weizenbaum asks us to imagine a game consisting of a roll of toilet paper, several black stones, an indefinite number of white stones, and a die. We set up some white stones bracketed by black stones, an empty space, and then the same again, with one black stone set above a square as a marker. The game has 18 distinct states determined by whether the marker stone is over a white stone, a black stone, or an empty square along with the number on the die (3 marker positions x 6 die positions = 18 states). We write a set of rules specifying the action to be taken for each of these states: which way to move the marker and which side of the die to select next. There is no choice to be had here; these are not rules that bound the possibilities of gameplay, they are a rigid set of instructions that continue until the “game” ends, or rather, until the machine we have constructed has carried out its task, which in this case is to combine the two groups of white stones into one larger group. What this game is, in fact, is a primitive adding machine, a single-operation calculator.

Weizenbaum then demonstrates how we can build a trinary language out of these game states. It’s a bit laborious to explain but basically if we denote the black stones as X, the white stones as 1, and the empty spaces as 0, we can diagram a game state as something like X11X000111X (this state expresses 2+3). Then, and this is the crucial part, we can translate all of our state change rules into this same alphabet, building a table of instructions capable of dictating state-change rules within the same language as the game-state itself.

A formal language is a game. That is not a mere metaphor but a statement asserting a formal correspondence. But if that statement is true, we should, when talking about a language, be able to easily move back and forth between a game-like vocabulary and a corresponding language-like vocabulary. Precisely that can be done.

The rules of our game constitute what Alan Turing called an “effective procedure.” Our game is a simple machine that can do one thing. The crux of Turing’s work was to theorize and prove possible a universal machine capable of translating and imitating the effective procedure of any other single-purpose machine. A universal Turing machine is thus, essentially, a machine that can read any set of rules and play the game they specify. Almost tautologically then, we could say that a computer is a machine that can be programmed.

But Weizenbaum knew what he was doing when he started this explanation with game-states. He took those game states and represented them in trinary code and rewrote his rules in trinary code as well, and we might call that code a language but it is crucial to remember that language constitutes a set of board configurations and nothing else. Within the language of the adding machine, there is nothing but the adding machine—no larger world and no question of anything other than game-states. In other words, we have constructed a language consisting wholly of syntax and nothing at all of meaning:

We have seen that the very idea of an effective procedure is inextricably tied up with the idea of language. Isn’t it odd that I could have spent so much time discussing language without ever alluding to meaning? The reason I have been able to avoid confronting the concept of meaning is that I have been discussing only formal languages or, as I have said, abstract games. Not that meaning plays no role whatever in such language games. It does. But this role is entirely subsumed in the transformation rules of the language… It is a property of formal languages, indeed, it is their essence, that all their transformation rules are purely syntactic, i.e., describe permissible rearrangements of strings of symbols in the language, including replacements of symbols and introductions of new symbols independent of any interpretation such symbols may have outside the framework of the language itself. One can, for example, do pages of algebraic transformations, following the rules of algebra blindly, without ever having to know that one may substitute numbers for lowercase letters but not for parentheses, in other words, without ever giving any interpretations to the symbols one is dealing with.

So, just to recapitulate, all computing is a form of gameplay, one that happens within a closed linguist environment that contains no meaning and makes it impossible to say anything about the world outside it or even to conceive of it at all. When people say LLMs are just probable-output machines that have no conception of their interlocutors or what their responses actually mean, this is what they are talking about. It’s not simply that they have no conception of these things, at their current level of development, but that they cannot have such conceptions because it runs counter to the fundamental natural of computer programming. Blindness and meaninglessness are built into the machine at the deepest possible level.

I find this profoundly disturbing! Weizenbaum’s book is fifty years old now. Obviously the world has been computerized to an extraordinary degree in the interim, which is to say that more and more parts of our lives have been given over to highly sophisticated systems of blind symbol manipulation. As time passes and the non-computerized world fades farther into the past, it becomes harder in certain ways to notice the character of the computerized world. Or rather, we might ask, what forms of activity and thought are engendered and encouraged by the ubiquity of computation and programming, by a world that runs on scripts?

My starting Computer Power and Human Reason overlapped with finishing Gormenghast, Mervyn Peake’s Gothic fantasy sequel to Titus Groan.¹ These stories take place within the colossal, crumbling estate of Gormenghast, a self-contained world that operates like a clockwork mechanism of centuries-old ritual. Reading Weizenbaum and Peake in conjunction I couldn’t help seeing Gormenghast as a living computer. The entire life of the castle is dictated by the Tomes of Ritual, with all contingencies accounted for. Early in Gormenghast we see a ritual meant to be performed one way in sunshine, another in rain. Only the Master of Ritual—first Sourdust, then Barquentine—is capable of interpreting the Tomes at all. But all of it is dead mummery—whatever it originally symbolized or celebrated was lost to memory centuries earlier, leaving only empty forms. The inhabitants of Gormenghast give their lives over to embodying these abstractions and are so fully subsumed within its symbols and empty language they have no access to real life. The only person who recognizes this is Titus, the young Earl, who rebels, Peake says very explicitly, because he cannot tolerate the meaninglessness of all his responsibilities.²

I can’t be the only one who feels it. That dread that we’re not as far from Gormenghast as we’d like to believe. We program our computers and they program society. This societal programming has expressed itself in several ways: as regimentation and discipline intent on snuffing out spontaneity in daily life, as a cratering void where meaning once was, and as a sneaking unseriousness as even decisions with life-and-death stakes have been infected with a sense that it’s all just a game.

Examples abound. Friend of the newsletter Adam Kotsko recently wrote a perceptive piece about the ideology of the New York Times and its political coverage, trying to understand why it’s all so exhaustingly vibes-oriented: “It is simply impossible for them to report on a political event without adding some reflection on the potential effects. Nor can they allow themselves to focus on the real-world effects of political decisions on people outside the political class. It’s all about the self-contained game of struggling for advantage, taken simply for its own sake. Every statement, every factual claim, is a move in that game and is assessed as such.” This is spot-on but we might ask from where this idea of politics as game originates, and why it has taken hold at this moment in history. I would argue its origins are two-fold—stemming from a deep intuition of all systems and actions as game-like that has accrued over the past fifty years, and reinforced by the actions and attitudes of our elected officials who treat national politics, on the one hand, with the flippancy and trollishness of an uninterested board game participant or, on the other, as a completely determined process in which their role is simply a relay toward generating the expected output.

In finance: Weizenbaum discusses how early computers were seen as arriving miraculously just in time to save a banking system that was groaning under the weight of multiplying quantities of checks that clerks could not process by legally mandated deadlines. Fast forward to the days before the 2008 financial crisis, in which bankers bundled their assets into elaborate Jenga towers, all while assuring regulators these new securities were both too complex to scrutinize with the human mind but also completely safe in the hands of their automated systems.

Would it be going to far to overlay the timeline of early computing onto the timeline of neoliberalism’s rise, beginning with Pinochet’s coup in 1971, and come to see neoliberalism in its entirety as both epiphenomenon and reinforcer of computerization? I will invoke another great piece by Adam, published just yesterday, discussing people’s willingness to give over every aspect of their lives and decisionmaking to AI and how we recreate the regimes of competition that govern our entire society in our daily lives:

All of these metrics, all this pointless and yet endless competition, stems from the quest to recreate society on the model of the market. That is to say, yes, we are still living in the aftermath of neoliberalism… The neoliberal theorists were very clear on this: the reason to make the market the center of society was to ensure that there could never be human agency or control. The benefit of market outcomes is not ultimately that they are more efficient or serve the greater good or whatever other propaganda one might derive from dorm-room libertarianism (the neoliberalism of fools). The benefit of market outcomes is that they are impersonal, inhuman. They are no individual’s responsibility, which means that no individual can ever assert their agency over them.

They weren’t afraid of Communist central planning because it was unworkable or inefficient or unfair—they were afraid that it would work. They were terrified at the prospect of human beings taking control over the conditions of their lives and livelihoods, and they gradually built a shell around every government and institution on earth to make sure that could never happen. It was a slow and gradual process, but its result was equivalent to dropping a bomb on human autonomy.

Of course, no one unites and embodies all the qualities of computing-as-game like the reigning avatar of tech, Elon Musk. He is a perfect output of the system and its greatest indictment. Musk is a master of symbol manipulation whose only real skill is (or was) presenting himself and his products in ways that aligned with larger forces in the zeitgeist. For those of us who concern ourselves with meaning, we saw someone with no principles who slid around the political spectrum and specialized in vaporware. But Musk, viewing all his dealings and press events as syntactical games of empty verbiage—Objective: construct a sequence of words that will make Tesla stock go up—has never seemed troubled in the slightest by hypocrisy or broken promises. Why would he be? None of those things he said actually meant anything.

Once we understand the computer as a machine that’s exceptionally good at manipulating meaningless symbols by laws of pure syntax, LLMs seem like its ultimate and inevitable evolution. Perhaps evolution isn’t the right word. LLMs represent an incredible difference in scale from an IBM 7094 but fundamentally not a difference in kind. This, I think, is the real reason knowing every book and every essay ever written by a human has been used as training data on those models is so upsetting—in a very real way that is a process of taking writing that was labored over to express meaning and feeding it into a machine that destroys meaning as its first operation.

Silicon Valley first introduced AI to the public with the stirring message, “This technology will drive the value of all artistic production down to zero.” As they sought to escalate the “hype” they began promising it would render all human workers obsolete and trigger mass unemployment. Now they dream apocalyptically of a “permanent underclass,” whose inability or unwillingness to embrace AI tools will deservedly condemn them to a life of penury. Many have observed this has been a world-historical messaging misfire. But our analysis renders it legible, I think. They have absorbed the logic of the machine at its deepest level, the logic of the game. And every game has its losers.

1. Previously: book report, Steerpike as Falstaff figure

2. To extend this analogy, Steerpike is a virus, perverting “the code” to malicious and secret ends