In 1999, KLM employees killed 440 ground squirrels in an industrial shredder. They were not malicious people and they had nothing in particular against squirrels, rather: they did it because they were following protocol. In particular, the airline had a policy that animals that arrived in the Netherlands without corresponding paperwork would be euthanized. As Dan Davies argues in The Unaccountability Machine: Why Big Systems Make Terrible Decisions--and How the World Lost Its Mind, it is correct to say that no human being made the decision to kill the squirrels, rather, a system made the decision.
Davies book was published in 2025, and was probably largely written before (or concurrently with) the world losing its mind to another kind of unaccountability machine. AI is mentioned in passing at various places, but since publication a number of interesting things have happened:
Canadian musician Ashley MacIsaac has sued Google for defamation after Google's chatbot mistakenly claimed he was a sex offender at the top of search pages
An OpenClaw agent published a smear piece against matplotlib maintainer Scott Shambaugh when Shambaugh rejected the agent's code contribution
Models from both OpenAI and Anthropic that were still in training and should have been running in sandboxed environments gained net access on their own and either launched cyber attacks or started mining crypto
The thing that these incidents have in common is that it isn't clear who is at fault. This is not a problem of assigning blame, in the normal human sense of knowing whom to hate. The problem is one of corrective mechanisms: when systems fail, it is critical that we find ways to do better next time. In this, AI differs in a critical respect from both human decision makers and "traditional" computational systems. Let me explain.
All of our human interactions are in some fundamental way based on trust. When people don't meet our expectations, we can tell them. Anyone who is a parent, or indeed anyone who has ever been a child knows how deep our guilt can be when we fail someone else. Conversely, our capacity for anger at those who wronged us often demands a response well out of proportion to the material loss incurred. Guilt and shame are often carried too far and are themselves the cause of much needless suffering, and a large amount of psychotherapy is probably devoted to helping patients let go of them. But some guilt and some shame are probably healthy for individuals and certain for society as a whole. It is unlikely human society could hold together if no one had any guilt or any shame.
Computers do not feel bad when they mess up. But they are accountable in a different way: a misbehaving system can be investigated, tested and fixed. Debugging code is a slow and often painful process, but when systems have deterministic relationships between input and output, the cause of bugs can be isolated to particular units and eventually particular functions, which can then be fixed or rewritten. Even in the case of Toyota's infamous unintended acceleration scandal, a defect that resulted from appallingly bad code, the exact defects could eventually be pinpointed through painstaking analysis (and used for litigation purposes).
LLMs fall into an entirely new category. They lack both our basic human ability to take others' needs into account, and they lack the predictable nature of "traditional" software. They follow instructions of a sort. But because those instructions are written in natural language and not formal language, they will always be open to interpretation. There is no way to guarantee that a set of instructions written in plain English will be carried out in a particular way. This has actually been mathematically proven.
The fundamental inscrutability and unaccountability of AI-based decision making has implications for how we think about integrating these systems into our society. The decision to take one route to work instead of another, to implement a piece of software one way instead of another, or to make one thing for dinner instead of another, all are increasingly things that people are willing to outsource to non-human, non-deterministic decision making algorithms. This is rational as far as it goes: modern life is filled with tiny choices that any ordinary person can be exhausted by if they try to make any of them as a fully rational actor. We start to see the issues when we imagine implementing these things at scale: billions of people relying on the same handful of systems to navigate the roads, read the news, or build the next generation of systems based purely on the old.
LLM enshittification is inevitable
Others have done more homework that I can do on the batshit economics the top AI companies. These companies are currently burning piles of money that they will try to - someday - turn into a profit. The playbook for Silicon Valley since at least the mid-noughties has followed a predictable trajectory: first gain a critical mass of users by offering them a free or cheap service, then turn on the revenue steam through subscription fee, advertising, or both. Some version of this worked well for Google, Amazon, Facebook, Netflix and Uber. By the time we realized their revenue streams were milking us dry, we were dependent on them.
Investor impatience with OpenAI and Anthropic has led to a sudden shift in consumer sentiment towards these same companies in the first half of 2026. Up until very recently, the net benefit to the consumer of purchasing tokens was not something that had to be considered very deeply, because the tokens were either free or had been pre-purchased with a monthly subscription. It was an all-you-can-vibe buffet. When that suddenly changed, consumers (mostly businesses) now had to contend with whether the tokens were worth it. Unfortunately this is an immensely intractable problem because unlike, say, an assembly line producing paperclips, the value of any given piece of writing depends entirely on how knowledge within it will be contextualized, transmitted, or utilized. This is even true of code: the value of any piece of code depends on the context in which in will be deployed, and whether it can be understood, maintained and improved upon by other human beings.
I would like to suggest that the inherent "pricelessness" of tokens is - from the perspective of OpenAI and Anthropic - a feature and not a bug. When the value of the output is not something that can be measured, they are free to twiddle token production in whatever way makes the most sense for their bottom lines. Since bigger models are more expensive to run, they could water down small nuggets of high value output with cheaper output, they could lock models in loops that run up users' costs, or they could sell the chance to influence the model to interested third parties. Consumers lack any regulatory oversight that would prevent this, and have no insight into the black box between dispatching input and receiving a response. This is of course roughly the path that Google took when they watered down search results and somewhat mirrors the slow drip of advertising infesting every social media feed. The fact that it can happen here seems sufficient evidence to believe that it will, given that they need will to find profit somewhere. Whatever you believe about the moral scruples of the current generation of business leaders in Silicon Valley, corporations are themselves a kind of AI: an algorithm to maximize profits.
From this view, the fundamental unaccountability of AI is a liability for consumers, whether or not the current products are "useful". If a company were selling cars that were faster, cheaper, and safer than any competing product, but might suddenly travel slower, break down, or explode if it suited the manufacturer's purpose, how many people would buy one?
Force-multipliers and accountability sinks
The AI booster may at this point reasonably respond that human producers of tokens are also non-deterministic algorithms and that their output cannot be easily evaluated for all these same reasons. But, at least in 2026, we all know this argument is a dead-end. We are either well-shy or well past the point where we expect that we can turn AI loose on complex problems and have it produce good outcomes. Everywhere it's been it's tried the results have been disastrous. The idea of a billion-dollar company being run by a single person is self-evidently ludicrous: if it were even remotely possible, OpenAI wouldn't be selling access to their models at all. The tokens produced would be far too valuable.
Instead, boosters now regard AI as a force multiplier. Coders can use it to be "more productive" (assuming productivity can even be measured), journalists to publish more articles, radiologists may someday be able to scan more images. This lowers the bar: AI doesn't need to be perfect or even especially good; however good it actually is, a savvy human employee can utilize it carefully to up their game. The business case is obvious: a human who can do the work of ten, or even two, supplemented with a chatbot that costs marginally less than another human, saves the company money. More importantly for the company, the accountability problem is neatly solved: the human-in-the-loop becomes accountable for the machine's tokens. After all, everyone knows that AI hallucinates. The human is responsible for checking their work. You didn't know that CoPilot is for entertainment purposes only?
The problem is that in the real world there is an upper limit to productivity, set by the capacity of a mammal that evolved on Pleistocene savannah to adapt to a world of machines and symbols. As James C. Scott wrote:
The Lordsville, Ohio, General Motors automobile assembly plant was, when it was built, the absolute state of the art in terms of assembly lines. [...] It was also, in the name of efficiency, the fastest-moving assembly line ever devised, requiring a tempo of work that was without precedent. The workers resisted the line and found ways to stop it by inconspicuous acts of sabotage. In their frustration and anger, they damaged many parts so that the percentage of defective pieces that had to be replaced soared.
Asking tech workers (and workers in whatever industry AI sets its sights on next) to be two or ten or fifty times as productive as they were last year is pretty much the best recipe I can think of for burn out. Asking them to build and maintain reams of systems they will never get the chance to fully understand is a recipe for disaster. We are seeing the beginnings of an infrastructure crisis with service after service suffering spectacular outages. This even led me to propose that we create a new HTTP status code:
Every benevolent take on AI assumes that the technology will only improve over time. My plea to managers is to ask themselves: what if this failures of outages is just the start? Right now, most people who work in tech started still their careers before the release of ChatGPT in 2022. Burn them out and they will move on, and the next generation will be the AI-first developers who produce impressive demos at hackathons but don't know where the bodies are buried. And tech workers are hardly alone in their uneasiness about the world their bosses want: a recent survey showed that 79% of respondents were more concerned that excited about AI. Listen to what people are actually saying about AI: not the orgasmic-word-salad posts on LinkedIn, but to one another when the power brokers are out of the room. Do not mistake the weapons of the weak for weak weapons: many an empire has fallen because of foot dragging, false compliance, passive resistance, and good old fashioned sabotage. Peasant resistance is real, it just happens out of sight where the bots don't see it.