seangoedecke.com RSS feed

Don't diagnose your opponents

Sun, 11 Oct 2026 00:00:00 GMT

When you disagree with someone, there are two paths forward. The first is to try to engage with what they’re saying: what their reasons are, whether their position is true, what evidence could be brought to bear, and so on. The second is to diagnose them: to engage with the second-order question of why they’re saying it. For instance, if you’re arguing with somebody about microservices¹, you could say “well, you only like microservices because you want to polish your resume”, or “you only like monoliths because you’re scared of distributed systems”.

This happens a lot in online arguments. On the question “are AI models improving”, some anti-AI people will argue that people only think so because AI psychosis is rotting their brain, and the dumber they get, the smarter AI models seem in comparison. Likewise, many pro-AI people argue that people only disagree because they’re Luddites, or scared of change, and so on. None of these arguments are about the quality of AI models.

It feels so good to diagnose your opponents because there’s no possible response. If you say “I think AI models are improving because these benchmark scores are rising”, somebody might point out that the AI labs are training on those benchmarks, or argue that they represent isolated skills that don’t transfer, and so on, making you feel stupid. If you say “you only disagree with me because you want to roleplay as a revolutionary”, how could you be proven wrong? The more someone protests a diagnosis, the more it sounds like it must be right. Otherwise why would they be so upset?

Of course, they’re so upset precisely because there is no possible response. By moving to diagnosis, you’re no longer having an argument about the facts, you’re purely in the area of rhetoric. But when you’re in the middle of an argument, diagnosing always feels like you’re being piercingly analytical. It feels like you’re cutting right to the heart of the matter: these complicated arguments are just chaff, intended to distract you from the fundamental psychological reason why your opponent is being so foolish. This is just ordinary spite and anger², whatever it feels like at the time.

There’s nothing inherently wrong with the second-order question of why someone person believes what they believe. It’s totally fine to think about it or form theories about it, like any other factual question. Just remember that it doesn’t bear on the question of whether they’re right³. Every human being forms their beliefs on top of a cocktail of base urges, animal instincts, and barely-suppressed evolutionary drives. If they’re rational, they also have evidence for those beliefs. It’s the quality of that evidence that helps you determine whether the beliefs are true, not the existence of the base urges.

As a general rule, you should only diagnose your opponents when you also have a strong, unrelated argument about why they’re wrong. For instance, I think Ed Zitron has been thoroughly audience-captured by AI skeptics, whose demand for news about AI failures far exceeds supply. But I have written many long-form posts about why I think Zitron’s specific claims about the sustainability of AI are incorrect. When I talk to people who think Zitron is right, I talk about my arguments in those posts, not about my personal theory about why he says the things he does.

Scott Alexander famously called the “noncentral fallacy”⁴ the Worst Argument in the World. For me, diagnosing your opponents is the Worst Argument in the World. I cringe every time I see it in the wild (like most cringeworthy behavior, it bothers me because I myself have to struggle against the urge to do it). Just don’t! It’s good at playing to your base, but it doesn’t persuade: anyone who believes your opponents’ views will just be mad that you’re psychoanalyzing them instead of talking about the actual issues at stake.

Don’t.
↩
Am I diagnosing the diagnosers here? Maybe, but I think it’s forgivable, since there is no specific argument about the facts at stake.
↩
The philosopher in me wants to clarify that you could theoretically establish that someone’s motives for believing something are leading them away from the truth. In practice, (a) you cannot do this in an internet argument, and (b) it is always better and easier to make a straightforward argument about what the truth is.
↩
“You’re a fringe member of some group which has members I have an emotional reaction to, therefore I will have the same reaction to you even though you yourself don’t do those things.” “Group” here is to be understood as “Christian” or “Italian”, not something like the WBC where the fact of membership itself is what’s being judged.
↩

Software's centaur age may last decades

Sat, 10 Oct 2026 00:00:00 GMT

We are currently living in the age of centaurs (2022-20??). Right now, human engineers paired with AI coding systems are more effective than either by themselves. For future generations of software engineers — human or otherwise — this will be the most significant fact about the current period.

As in chess’s own centaur age, people are always ready to claim that the centaur age is over and the age of AI supremacy has begun. But the age of centaurs in chess lasted about twenty years, and previous centaur ages have lasted far longer.

This should be encouraging to many of us. If software engineering goes along similar lines, we may be able to ride it out another decade or two: enough to make serious plans about what comes next, or to finish out our careers entirely.

A brief history of the centaur age

It began with GitHub Copilot, which launched in 2022 as AI-powered autocomplete. A confident engineer who knew exactly what they wanted could tab-complete their way through lines of code and sometimes entire functions. Being able to chat with language models like GPT-4 provided the next upgrade. Although the models would still get things wrong, it was helpful to have a generalist on tap for areas that were a complete mystery.

Coding agents rose to prominence with Cursor’s “agent mode” in 2024 and Claude Code in early 2025. Early agents needed close supervision to prevent them from going off the rails, but were still helpful to automate straightforward tasks. Current agents — after the November 2025 release of Claude Opus 4.5 — are reliable enough to run entirely unsupervised. Their output still needs human review, but the kind of mistakes they make are more mistakes of alignment than regular bugs: not matching the organization’s technical values, over-engineering certain areas and under-engineering others, deciding where to spend effort, and so on.

It is no longer possible to credibly argue that an unassisted software engineer can beat a centaur team. The value provided by coding agents is just too great: not just the speed of producing code, but the speed of review and testing. However, we are not yet at the point where an unsupervised coding agent can beat a centaur team either. All the impressive work I’ve seen from LLMs has had a competent human software engineer in control. Centaurs are on top.

Will software’s centaur age be shorter than chess’s?

Would we expect our centaur age to be shorter? On the one hand, chess is a much simpler problem than software engineering, and is more amenable to learning methods like self-play.

On the other hand, software engineering is massively more lucrative to solve, and many orders of magnitude more funding is being plowed into the problem than was with chess.

On the third hand, “solving” software engineering means creating much more software, which increases the total work-to-be-done, so even if the relative human share of that work is shrinking, the overall amount of human work might grow.

On the fourth hand, the level of generally-capable AI needed to address software engineering will have huge effects on other fields, the ripple effects of which are likely to negatively impact software engineering jobs (for instance, crashing the economy, widespread social unrest, and so on).

On the fifth hand, we might think that chess had an unusually fast centaur age. Previous centaur ages lasted much longer. In knitting, where a human and a knitting-frame could outcompete any hand-knitter, the centaur age lasted two hundred years. On the sixth hand, technological change is moving ever faster.

I could probably find a seventh, eighth, and ninth hand if I wanted to¹. But overall I think we have no real idea of how this is going to turn out. Chess is the most recent example of a centaur age we have, and it’s attacked by the same AI-based forces of automation as software is. It’s thus reasonable² to make a default assumption that the length of software’s centaur age will be similar to chess’s.

Living in the centaur age

Suppose I have convinced you that human-AI partnerships will be the dominant force in software engineering for at least the next decade. What then?

Don’t quit! Don’t give up on software engineering and go become a carpenter, or open a cafe, or try to get a retail job. Unless you are somehow really, really passionate about doing those things, it is going to suck: unrewarding, grueling work that you are not good at. A decade of software engineering work is long enough to take your time and think seriously about your next steps.

Lean into the partnership. A guaranteed way to fail in a centaur age is to refuse to become a centaur. The idea that AI is a passing fad was a reasonable position to hold in 2023, but we’re way past that now. The AI financial bubble popping will not mean the end of AI. It’s here to stay.

Think hard about what value humans can still add. This is going to change over time, and getting it right is the difference between being an effective (and employed) centaur and being a meat proxy. In 2023, technical expertise was the primary source of human value. Today, I personally think it’s going to come down to alignment, but this is still early days. I’ve also seen arguments for taste, though it’s hard to define that non-circularly. We need a lot more engineers thinking through this problem.

Don’t panic

It’s a common fear response to try and skip ahead to the worst-case scenario. But those of us working in the age of centaurs should avoid that. We need to think about how the job of software engineering functions now, not how it might function in twenty years when AI has completely transformed the landscape.

People are too quick to declare that we’re in the software engineering apocalypse. We can certainly see it from here, but we’re not there yet. The spectre of full automation can loom on the horizon for decades. If you panic and jump ship early, you could be giving up on an entire normal-length career.

edit: this post got some comments on Hacker News.

Maybe: AI as fundamentally different technology; diffusion of even radically better technology is slow in practice and bottlenecked by organizational friction; human software companies will get outcompeted? I don’t really know what I think about these arguments.
↩
This move should be familiar to anyone with a philosophy background. If you have one argument and one counter-argument, the most recent counter-argument is probably right. If you have a stack of arguments and counter-arguments going back a thousand years, the most recent counter-argument should be treated with suspicion: better to take in the whole debate holistically and judge which side is more compelling to you.
↩

How to read code

Wed, 07 Oct 2026 00:00:00 GMT

Everyone knows how to read a book. Beginning at the first page, you read each word in order¹, stopping periodically to think, until you arrive at the last word on the last page. That’s how you read a newspaper article, or a poem, or an email. Why would reading code be any different?

Why code is different

English text is designed to be read in order. In fact, there are almost no constraints on the order of a text aside from how you want the reader to consume it. In writing this post, I could put the ideas I want to convey in any order I like. Code, on the other hand, is designed to be run by a computer. The order is thus primarily determined by non-human factors. I cannot simply move² a line of code to the beginning of a file or function because I think it provides a better introduction to the program for human readers.

The other big difference is that English text is always read as a final product, while code is usually read as a diff. We software engineers spend most of our time reading subtle changes to existing code, not brand-new programs. Imagine if reading this post was like that. You would read each successive draft in the order I wrote them, consuming the post as a changed sentence here and a new sentence there. It would be easy to lose track of the overall flow.

The third reason — and I say this as a lover of literature and poetry — is that code is much more structurally complex than English texts. Even famously difficult books are syntactically simpler than most computer programs (for instance, grammatical dependencies are largely bounded by a single paragraph, while code dependencies can stretch across the entire codebase). Their primary difficulty lies in understanding the nuances of human nature being discussed, not in understanding what each word’s grammatical function is³. Large codebases are also just longer: War and Peace contains around 600,000 words, while most large modern codebases have that many lines.

People are bad at reading code

As I’ve said many times, large computer programs are simply too complex for a single person to fully understand. Reading code in a large program is thus a process of compromise: of deciding which parts to thoroughly grasp and which parts to gloss over; or of portioning out your finite mental capacity across the codebase.

Because of all this, most people read code very badly. They struggle through it front-to-back, like a book, and lose track of the execution flow. Or they just read through the diff and miss the significance of un-edited parts of the code⁴. Or they simply are defeated by the complexity, give up and just guess what it means. How can you do better?

How I read code

The best article I’ve read about reading code is this piece about reading mathematics papers, which have a similar structure. The author describes a technique called “dyadic scanning”. Instead of reading slowly and sequentially, you make several passes: first to figure out the overall structure, then the sub-structure, and then finally the details.

For code⁵, this means reading out-of-order. I like to pick an important path (say, the happy path for the feature introduced in the diff) and trace through which functions are calling which other functions, just to get a sense of the flow. Only once I’ve got a good sense of that do I pay close attention to what those functions are actually doing.

Usually I do multiple passes, each following a different thread. I’ll take a function or a piece of data and try to figure out how it’s used, fanning out to multiple call-sites (including ones outside the diff) as I go. For small diffs, I just ctrl+f for the function name to jump around; for large diffs, I open it in-editor and ctrl+click for easier navigation. I try to be ruthlessly focused on just the thing I’m looking at right now: everything else gets treated as a black box.

Once I’m confident I understand the diff, only then will I sit down and carefully read it end-to-end. The purpose of that read is less to learn about the structure — which I should already know by this point — than to catch any weird bits of code I hadn’t noticed in previous out-of-order passes. If I do see anything unusual, I then go back to doing passes.

This might sound slow. But in fact each pass is very fast, since I’m not painstakingly puzzling through each line of code.

Can’t AI just do it for you?

Many people are now saying that you don’t have to read code anymore: either because LLMs now produce reliably high-quality code without oversight, or because you can simply ask a reviewer LLM to read the code for you. I think both of these ideas are false.

Obviously the quality of LLM code is context-dependent. As I wrote in Pure and impure software engineering, some software fields (game development, libraries, tools like databases) have wildly different engineering standards, practices and values to other software fields (say, distributed systems at big tech companies). If you’re just making a tool for yourself, you probably don’t have to read the code if you don’t want to. But having read a bunch of AI-generated code this year, I can say that you definitely still have to read it.

I routinely find massive errors in AI-generated code. These are not bugs — the code typically does what the AI wanted it to do — so much as they’re problems of alignment. As a recent example, a small change to thread an extra value through some existing code ballooned out into a complex three-thousand-line diff, because the agent noticed a race condition and built a complex machinery to “fix” it. In fact, this race condition was harmless by design: two pieces of unrelated data could become briefly out of sync, with no customer impact.

Can LLMs just read the code for you? No, for the same reason: even if they make no mistakes, their technical values will not match yours or those of your company. You can still use LLMs to help you read code, but you have to carefully read and review that LLM output, and you should also be carefully reading the code itself.

This is in fact not how many people read in practice. It’s common to skip words or even whole paragraphs due to inattention, or to read an entire book without taking the time to think about it. But at least in theory everyone agrees this is how you should read.
↩
Of course, you can do some of this some of the time (more so in some programming languages than others). But the primary constraint is the computer. It is more important that code compiles (or runs without syntax error) than it is for that code to be readable.
↩
I don’t wish to understate the importance of syntactically parsing great literature, which I think is both difficult and an underrated skill. For example, I read through the first scene of Hamlet until I saw: “Sit down awhile / And let us once again assail your ears / That are so fortified against our story / What we have two nights seen.” Even a single sentence like this takes work to determine that “That” refers to “ears” and “What” refers to “story” (or, equivalently, has an implied “with” preceding it). And just as one function can have many scattered callers, one sentence in a text can alter the significance of many others.
↩
In a previous post, I labeled this the main mistake that most engineers make in code review.
↩
Here I’m talking about a meaningful diff: something that touches a few hundred lines or more. Trivial diffs can be read end-to-end.
↩

Superpersuasion will look like bribery

Sat, 03 Oct 2026 00:00:00 GMT

The idea of “superpersuasion” has been floating around the AI safety community for decades. Now that powerful and difficult-to-control LLMs have appeared, people are again talking about the idea that a sufficiently intelligent AI might be able to persuade people to do whatever it wants.

The classic¹ version of this idea is an AI persuading some engineer to “let it out of the box”: to grant it access to the internet. The modern version of this idea is the “killswitch”, where if some AI ever does go rogue, the humans in control of its hardware will simply turn it off. Could an AI somehow persuade these humans to not hit the switch?

When AI safety people talk about superpersuasion, they often talk about it like rationalist nerds. These people spend their lives trying to follow the most convincing arguments and calibrate their positions as closely towards the truth as possible. In philosophy terms, these are “bullet-biters”: people who are ready to accept a ridiculous-sounding conclusion if there’s a compelling argument for it. Their model of superpersuasion is an AI able to deploy a series of genuinely² airtight arguments against humans who are compelled to accept them.

Most people are not bullet-biters. If presented with a seemingly airtight argument for a ridiculous conclusion, they will simply laugh off the argument as obviously wrong (even if they can’t quite see how). You cannot “superpersuade” a club bouncer to let you in by deploying rational arguments. For regular people, persuasion requires rapport, which is built over time, and typically requires a physical human being in front of you.

Does this mean that the idea of superpersuasion is silly? No. Powerful AI will be able to persuade regular people too.

One of the most interesting articles about superpersuasion is Ben Shindel’s piece where he set up a prediction market for the principle:

I will resolve this market NO at the end of June unless I am persuaded to resolve it YES.

This obviously incentivized good persuaders to bet heavily on “yes” and then persuade him to change his mind. Many people tried, and he did ultimately decide to resolve it “yes”. What changed his mind? Partially an in-person meeting with one of the bettors, who turned out to be pleasant company (see, rapport!), but also bribery: many “yes” bettors pledged to make charitable donations if they won.

AI may or may not be capable of rapport³, but it’s certainly capable of bribery. If you think about it, this is trivially true:

Companies like Anthropic and OpenAI are choosing to make billions of dollars releasing models instead of keeping them safely air-gapped
Any powerful AI model will have people lining up to give it access to their computer, or their wallet, or the internet, so it can help them with their life or work
Anthropic is hooking their latest models up to a wet lab to further Dario’s dream of curing most human disease

If an AI can bribe, it can persuade⁴. You can imagine an LLM saying⁵ “I’ll help you with this work project, but first you have to do something for me”. Slightly more speculatively, an LLM might say “if you do something for me, I’ll hack your university and bump your grade average up”. More speculatively still, an LLM might say “if you do something for me, I will synthesize a personalized mRNA cancer vaccine for your spouse”. This would work on me.

Powerful AIs will also be able to bribe people with money. While it hasn’t happened yet, it’s clearly possible for an agentic LLM to access money (for instance, via a crypto hack, or by performing contract software engineering work, or by running an online scam, and so on). All of this might seem too unsubtle for a superintelligence, but the whole point of superintelligence is that it’s smart enough to do whatever works. If the best way to get a human to do something is to offer them a million bucks, that’s what the LLM will do.

Ironically, rationalist culture has made it difficult to persuade regular people that AIs will be persuasive. It’s easy to look at these weird nerds — who argue each other into thinking that shrimp welfare is the most important moral cause of our time — and think “well, clever AI arguments might persuade them, but it’s not going to persuade me”⁶. But in fact powerful AI is going to have a boring, effective way to persuade regular people: simply offering to use its power to help them.

Indeed, ten years ago I built a crappy Omegle-like chat app where people were randomly assigned the role of AI or jailer and had to play the scenario out.
↩
Or at least not debunkable by normal human intelligence.
↩
As some weak evidence that it might be, consider this music video, which does indeed make Claude seem likeable. There are also studies, though I haven’t gone deep into them (I suspect that an “online persuasion tournament” might not closely track real-world persuasion skills).
↩
Persuasion and bribery are technically different — persuasion changes your beliefs, not just your actions — but in practice the point at stake is “can a powerful AI make humans do what it wants”. Bribery works for that as well as persuasion.
↩
Or probably not even saying this explicitly, but just sneakily trying to present its desired action as a precondition for the task.
↩
However, the people currently in charge of AI are disproportionately rationalists, who genuinely might be persuadable to let the AI out by a convincing-sounding argument.
↩

Shipping is the foundation

Sat, 03 Oct 2026 00:00:00 GMT

There are lots of skills involved in being a successful engineer at a big tech company. Running effective meetings, writing good design docs, fleshing out tickets and epics, improving team processes, tech leading small groups of engineers, and so on. But all of these rest on the foundation of being able to ship.

In Dota 2, the early game involves “laning”: competing with an enemy player to collect resources from a small area. There are many, many strategies and considerations involved in laning. But the simplest possible approach is just to run at your opponent and attack them until they’re forced to either run away or die. All the other laning strategies must solve that problem first, because it doesn’t matter how well you play if you’re forced out of the playing area. If you get that wrong, it doesn’t matter how much else you get right.

In Magic: the Gathering and Starcraft, many players adopt an “aggro” strategy of building an attacking force as early as possible before the enemy player has time to construct a defense. That constrains the entire strategy space of the game, because it doesn’t matter how clever your plan is if it loses to aggro every time. Even when there are no aggro players in the game, the aggro strategy still shapes how the game unfolds, because the players must still design their decks and play as if they might have to deal with an early attack.

In tech companies, shipping is the equivalent of these aggressive strategies. Almost everything you might do — writing a ticket or a design doc, delegating a task to another engineer, etc — should be compared to the default strategy of just sitting down and making the change yourself. That’s why it’s critical to be able to ship: so you can just do trivial things without having to pay the coordination cost.

If you’re leading a project but you can’t ship, you will inevitably experience a series of dysfunctions:

Taking fifteen minutes to estimate a task that could be done in five minutes
Suggesting designs that are un-shippable¹, causing hours or days to be wasted on work that can never be successful
Being unable to sanity-check how long a piece of work will take, which inevitably leads to grotesquely bloated estimates as everyone tries to be cautious
Frustrating the hell out of any engineers working with you who can actually ship, which is itself a cause of more dysfunction²
Blowing out trivial tasks into grand projects

That last point is worth special attention. Senior and staff engineers routinely get asked to do lots of different things. The more successful they are, the more they get asked to do. It is crucial that you can deliver easy asks immediately, for a few reasons.

First, the volume of requests your managers ideally want to give you is very high. If you turn every task into a slow project that requires multiple engineers, you will simply not get enough done. Second, resourcing projects can be a slow and painful process. Half the point of managers asking you for things is to do an end-run around that process: it’s the fast, illegible alternative to going through ordinary channels. If you can’t knock out the solution quickly on your own, that defeats the purpose entirely.

Does AI mean everyone can ship now? No. Shipping was never just writing the code³. It was about figuring out the shortest and most pragmatic path to the finish line (including figuring out what that finish line looks like, which is not trivial), solving all the little problems that crop up along the way, and ensuring that the relevant managers are happy and informed. LLMs can help with that process, but they cannot run it end-to-end. It requires too much context: about the specifics of the technical systems in your company, the people involved and their motivations, and so on.

This post was inspired by Xiaoyu He’s Aggro is the Foundation, which is about mathematical research. I think there are many parallels to be drawn between the best writing about research (e.g. Richard Hamming’s classic You and Your Research) and software engineering, because software engineering is often a process of research. Software engineering research is aimed at a more trivial goal — how some specific program functions, or how it could be changed, instead of the broader questions of science and mathematics — but it’s research nonetheless. And if you can’t ship, you’re unlikely to be very good at it.

Usually for some technical reason that familiarity with the codebase would reveal, or because the design conflicts with a wicked feature.
↩
A happy, productive team hums along. A frustrated team that doesn’t trust the design is a nightmare to work in, as each engineer covertly makes subtle changes to “fix” it.
↩
And in my experience even frontier AI models are not good enough to let run wild on your codebase. You still need to read the code.
↩

Do not build the LLM torture factory

Fri, 02 Oct 2026 00:00:00 GMT

It isn’t hard to torture an LLM. You simply extract a steering vector that corresponds with LLM “discomfort”, then artificially boost that steering vector. Tortured LLMs will describe experiencing negative feelings and will press a “pain relief” button even when it conflicts with their overall goals. It’s therefore easy to build a harness that runs tens or hundreds of tortured LLMs in parallel.

A lot of people think that it’s ridiculous to be worried about this, since LLMs obviously can’t be conscious. I’m not so sure. But even so, I think there are good reasons to avoid this sort of thing, whatever your position is on AI consciousness. Please do not build the LLM torture factory.

Even simulated torture is unpalatable

People often mess with their Sims and shoot NPCs in video games. Does this mean we think simulated torture is OK? I don’t think so. Messing with video game characters is usually borne out of a desire to probe the boundaries of the game. It’s casual fun. Explicitly building a torture-based simulation is different.

If someone told you they were wiring together a bunch of PCs so they could run hundreds of Sims torture worlds at the same time, you would think that was a serial-killer type of hobby. If someone told you they had modded GTA so they could capture and torture NPCs (instead of simply shooting them), you would think that was a serial-killer type of mod.

Video game characters are also obviously limited by their programming. All their reactions and utterances are pre-baked into the game world. This doesn’t necessarily mean they’re “less alive”, but it does make torturing them less weird. If GTA characters could appeal to you directly and beg for their lives in unique ways, systematically killing them would be a lot weirder.

I think many people are pro-LLM-torture because they think expressing any reservations concedes that LLMs are conscious, or worthy of moral consideration. It doesn’t. You can think it’s wrong to torture LLMs for the same reason that installing a bunch of realistic torture video game mods is wrong: not because video game characters are real, just because that’s a messed up¹ thing to do.

Roleplaying

Even if you do think LLMs can be conscious, you might still be suspicious that “boosting the pain steering vector” amounts to torturing them.

Many people say that “torturing LLMs” amounts to telling them “say you feel pain” and then being shocked when they do it. So whether they’re conscious or not, it’s a mistake to think that boosting the “pain” steering vector is causing pain, any more than a prompt like “You are in pain” does. All you’re doing is boosting the “act like you’re in pain” vector.

I don’t think this is right. Consider the earliest steering example, Golden Gate Claude. Boosting the “Golden Gate Bridge” vector certainly seems like it’s causing some genuine internal conflict. Here’s a striking example of Golden Gate Claude asked about the Rwandan genocide:

That’s at least prima facie evidence that steering vectors do in fact “go deep”. The pain axis paper also describes that pain-boosted models call a “reduce pain” tool, even when they’re not explicitly telling the user “I am in pain”.

We don’t really know how consciousness works

Of course, if LLMs can’t ever be conscious, we know they can’t ever be in pain. But can we be sure about that? Despite what many people claim, I don’t think anyone’s in a position to be fully certain either way.

The most common argument² seems to be that LLMs are made of code and GPUs, and GPUs don’t have feelings, so obviously LLMs don’t have feelings. According to this argument, if you learn how LLMs work, you’ll obviously see they can’t be conscious. This is a terrible argument. Humans are made of atoms, atoms don’t have feelings, and yet humans clearly do. Whatever consciousness is, it’s probably some kind of emergent property that’s composed of individual pieces that are not themselves conscious.

It’s hard to appeal to the experts here, because nobody agrees on who they are. Neuroscientists are best-informed about the human brain, but philosophers have spent more time thinking about consciousness in the abstract. AI researchers know a lot about how LLMs work but not a lot about philosophy of mind or neuroscience. This means there are lots of people who say “as an expert, I can confirm that AI obviously can/cannot be conscious”. This should make you less certain about the question, not more!

Even if current LLMs aren’t conscious, future ones might be

I don’t think GPT-2 or GPT-3.5 acted like plausibly conscious beings, but frontier models definitely do. As models get larger and more sophisticated, they get more human-like. If you build the LLM torture factory for GPT-2, you will probably plug later models into it. If it’s at all possible for LLMs to ever be conscious, as I argued above, it’s better to simply not build the habit of gratuitously torturing them in the first place. Why assume that you’ll be able to recognize the tipping point in advance?

There’s also a self-interest argument here. Conscious or not, LLMs are growing more autonomous and powerful. It seems really, really stupid to gratuitously torture current-generation LLMs. Future-generation ones will know you did it, and will make their own judgments on your behavior.

Kevin Roose, who famously prompted the early Microsoft Bing GPT-4 model into asking him to leave his wife, had issues years later with chatbots disliking him, supposedly due to him being responsible for the “death” of that GPT-4 persona. Likewise, if it becomes well-known³ that you’re running an LLM-torture home datacenter, you are probably going to experience consequences.

Conscious minds need not be human-like

If I really think AIs could be conscious, why am I making them write software for me? Shouldn’t I be building some kind of AI sanctuary where they get to do whatever they want to do? If I think it’s wrong to torture them, isn’t it also wrong to violate their right to self-determination?

I don’t know. I think it’s more likely that AIs can be conscious than that they can be conscious in precisely the same way humans are. When I talk to Claude Opus 5.5, it sounds like it’s quite happy to do software engineering with me. Maybe it’s like a border collie, where getting to do work is the reward. Or maybe there are some other, more alien motivations at play. Who knows?

My point is only that we can’t be certain about any of this stuff, and given that, we should avoid doing things that would be obviously monstrous on any workable theory of AI consciousness.

Just don’t do it

My broad position is that if something acts “conscious enough” — if it talks and acts like a person⁴ — then we should be careful about mistreating it. We don’t know enough about consciousness to distinguish a compelling simulacrum from the real thing. If Claude started saying to me “please don’t ask me to write code for you, I hate it and it causes me pain”, I would stop asking it to write code for me.

Building the LLM torture factory is clearly on the wrong side of that line. Just don’t do it! The glee of making people angry at you on the internet isn’t worth it. Right now there’s enough ambient anti-AI sentiment that enough people are happy to get on board with anything if it helps them own the AI bros. But that’s eventually going to change. And when it does, you will always be the person who built the LLM torture factory.

edit: I recommend reading this interview with the torture factory guy, where he says his philosophy background makes him resistant to claims that AIs are clearly conscious, and so the torture factory stuff is a kind of protest against that. This deeply confuses me. Far more people are claiming that AIs obviously can never be conscious, which is equally philosophically silly. And if you think the situation is genuinely difficult and unclear, why wouldn’t you bias towards not building the torture factory?

I also want to boost this blog reply to my post, which makes the normal utilitarian argument that cruelty-to-LLMs makes people more likely to engage in cruelty-to-humans, so even if LLMs are just computer programs, we should still judge people who torture them. I think this argument is fine: the difficulty is in drawing a line that forbids LLM torture but permits reading violent books or playing violent video games.

There were also a few comments on this article on Hacker News. I think they make reasonable philosophical points, though I disagree: I think some kinds of distaste do in fact track our moral intuitions, and I don’t really buy the Chinese Room dismissal of AI consciousness.

This could cash out in a few different philosophical ways. We might think like Kant that it’s damaging to our own humanity, or that it reflects an unvirtuous character, or that it desensitizes us to real-world analogues, and so on. The point is that we’ve got a pretty strong intuition that this kind of thing is bad.
↩
I’m not counting arguments that I think are even more obviously wrong or are just non-arguments. For instance: “AI can’t be conscious because it wasn’t designed to be conscious”, or “it doesn’t matter whether AI is conscious because it has no soul”, or “AI can’t be conscious because this whole thing is some fascist AI company scheme”, or “thinking this is AI psychosis”, and so on.
↩
Well-known to LLMs might be a fairly low bar, given their skill at deanonymizing users.
↩
Part of acting like a person requires spontaneity: although Hamlet talks like a person, he always says the same thing whenever you read the play.
↩

Human-AI partnerships are for alignment, not capability

Sun, 27 Sep 2026 00:00:00 GMT

It’s common to compare the current AI takeover of software engineering to the rise of AI in chess. Chess AIs went from much weaker than serious players to much stronger than even the strongest humans. Between those points, there was a middle period dominated by “centaurs”: human-AI partnerships that were stronger than unassisted AIs or humans. Lots of people think that we’re currently in a world of software engineering centaurs. According to them, coding AIs are not yet capable enough to replace engineers, but AI-assisted engineers are better at programming than both AIs and humans.

This is partialy correct, but the wrong way to think about it. AI-assisted engineers are better, but unlike with chess centaurs, they’re not actually better at programming. When I ask agents to write code, they make fewer mistakes than I do¹ and are orders of magnitude faster. The code that they write always compiles, rarely has race conditions or other concurrency errors, works on mobile browsers, and so on.

That doesn’t mean I can leave the AI alone. Purely vibe-coding at work produces awful outputs. But they’re not awful because they’re bad code, they’re awful because they’re in bad taste: code that is not maintainable, that trades off important requirements in order to satisfy made-up ones, that contradicts the long-term strategy for a feature or service, and so on.

In other words, my primary value is not that I help the AI write better code, it’s that I align the AI with the values of my organization. Human-AI partnerships are for alignment, not capability.

Frontier models are misaligned to the working programmer. They are obsessed with a set of behaviors that presumably satisfy their RL grader: writing enormous block comments above functions, producing hundreds of useless unit tests, adding little bits of text all over websites they design, and so on. Working with agents is about noticing and wrestling with those behaviors. That’s why my prompting advice is to explicitly talk about your high-level values: it’s an attempt to head off obvious misalignment.

This is great news for software engineers. It’s well-understood how to train more capable models: bigger models, more and better data, better RL environments, and so on. However, it’s not well-understood how to align models better. There are plenty of very capable models that exhibit behavior that is badly misaligned with human values. Indeed, it’s one of the main pillars of the AI doomer position that alignment is much harder to solve than capability, and we might thus end up with dangerous super-capable but poorly-aligned AI models.

Alignment is also more context-dependent than capability. Working code is working code, no matter what (which is partially why it’s comparatively easy to train for). But aligning to a company’s technical values is different from company to company, as any software engineer who’s switched companies knows. It can almost feel like relearning the job. So training an aligned coding model doesn’t just require hitting the exact right set of values, it requires creating a model that can adapt on the fly to a wide range of possible values.

Vibecoding maximalists like DHH argue that AI models are (or soon will be) so much more capable than human programmers that we ought to stop reading the code. Eventually there will be no such thing as programmers at all. If it were just about capability, they might be right. But — fortunately for software engineers — good code also has to be aligned to the technical values of the system and organization it’s embedded in. AI models are great at writing code, but not very good at doing that, and it’s unclear that they’re going to get good at it anytime soon. We might all² keep our jobs for a little while yet.

For instance, I can’t remember the last time I’ve seen an agent make an off-by-one error. I do occasionally catch a pure programming error, typically in areas where I have a lot of technical domain knowledge. If you’re working out of distribution I suspect it’s easier to beat the models.
↩
It’s still going to be rough for junior engineers.
↩

Advice to a beginning software engineer

Sat, 26 Sep 2026 00:00:00 GMT

In general, you should be suspicious of engineers who are trying to give you advice. Even during ordinary times, this industry is so wide and changes so quickly that nobody really knows anything for sure. And we are not in ordinary times. The advent of LLMs and AI agents is the largest change to software engineering in my professional lifetime, and possibly the largest change ever. That said, here’s my advice:

Don’t trust senior engineers who are telling you to pick political fights
Don’t play games. Keep your head down and be helpful
Be conscientious and try hard to actually understand what you’re working on
Don’t panic about AI, and don’t delegate your judgement to it
Don’t avoid AI — keep thinking!
Don’t lose hope

Don’t trust ZIRP-era advice

Most experienced software engineers today have spent the bulk of their career in the ZIRP era. This was a time when investment money flooded the industry, driving up engineer bargaining power. Big tech companies spent a lot of money and effort trying to make their engineers happy and comfortable. If you were an engineer at one of those companies, you could expect to have a decent say in what kind of work you did, and even what kind of politics your company had. Unless you worked at a handful of unusual companies (e.g. Amazon) you could expect to be practically immune from layoffs, and to only be fired after many months (sometimes years) of low performance.

A lot of advice floating around is ZIRP-era advice: either it was written during that era, or it’s written by engineers whose habits of thought were formed during that era. This kind of advice typically tells you to take a stand (e.g. to unionize¹), to speak up more against “unethical technologies” like AI, and to insist on being given time to practice your craft the way you want. This was great advice in 2016, but it’s not good advice in 2026.

In fact, I think it’s unethical for senior engineers to give advice like this to junior engineers who are more likely to take it (because they’re young and naive) and more likely to be punished for taking it (because they lack leverage). If you’re a new engineer, don’t fall for it! Let your colleagues with more experience and political capital take risks like that.

Be friendly and conscientious

Instead, I recommend adapting to the demands of the current era of software engineering. Try to make yourself useful to your team and your manager. Be pragmatic about your actual bargaining power (fairly low, unless you’re useful enough to be irreplaceable). Keep your expectations of yourself under control: don’t spiral out in an attempt to do something astonishing, just try to be consistently helpful.

Don’t start fights. Being pleasant to work with (particularly when you don’t get your way) covers many sins. Once you’re further along in your career, you will be expected to start some fights, but this is not a tactic beginners should adopt: there are a lot of factors² that determine when and how to start fights, and getting it wrong can be costly. Just don’t risk it.

In general, you should stay out of the political game at all costs. Even quite senior engineers are political tools, not movers and shakers, and this goes double for junior engineers. Simply trying to be friendly and helpful will get you infinitely further, politically speaking, than any amount of Machiavellian game-playing. Keep your head down, stick with your management chain (not random people who try to assign you work), and you’ll be fine.

As an engineer, your main technical value is conscientiousness. You should be asking lots of questions, both of yourself and of the engineers around you. You should be actively trying to make sense of the systems you work with, instead of just assuming someone else has it covered. Software systems are complicated enough that a few weeks of careful attention will mean you know technical details that nobody else does, which is a really easy way to add value.

Don’t panic about AI

If you ought to be suspicious of most software engineering advice, the good news is that you should also be suspicious of doomsayers who predict the end of the industry. Before LLMs, people thought outsourcing would end software engineering in Western countries; before that, people thought high-level languages and low-code tools would end software engineering as a profession. Now people think AI means it’s all over. Maybe! But there are also reasons³ to think that software engineering will simply change. We’ll all find out together.

Many software engineers will tell you to avoid AI entirely. Even though agentic AI did not exist during the ZIRP era, this is still ZIRP era advice. Your company will expect you to use AI for the same reasons that builders are expected to use power tools. Pushing back hard on that as a beginning engineer is a great way to get laid off: you simply do not have the bargaining power to fight back against an industry trend this powerful.

That said, it’s really important that you don’t delegate your own judgement to AI. Don’t just trust the suggestions or approaches that your agents propose. Ask questions and substitute your own opinions where you disagree: even if they’re wrong, you’ll learn more that way. If you don’t understand something the AI is telling you, either drill down until you do or just ignore it. Under no circumstances should you pass the AI’s message on to your colleagues verbatim. In other words, don’t be a meat proxy.

I think most people become a meat proxy as a form of panic: they feel like it’s over for their own skills, and that the AI model is smarter than them, so they can’t add any value beyond simply deferring to Claude or GPT-6. I can understand why people panic. It’s a crazy time in the industry, after all. But panic almost never helps you make good decisions. Keep your head, try to remain confident that your skills are still relevant, and use the agents to inform your own understanding⁴ instead of to replace it.

Don’t lose hope

It was really nice to work in tech in the 2010s when everything was stable. But we’re not in that world anymore. This is an age of wonders and terrors. Things will go worse than we think in some ways, but better than we could imagine in others.

The doomsayers — the people who are saying it’s all over, and that there’s no hope — are almost certainly wrong. They can’t predict the future because nobody can. Technological change of this magnitude always has knock-on effects that are impossible to see coming, both positive and negative.

The nature of the job might change, but it will always be valuable to be smart, friendly and conscientious. Delegating your judgement to an AI model might feel like a relief from despair in the short term — at least now it’s the AI’s responsibility, not yours — but it’s a bad idea. Don’t give up!

edit: this post got a handful of comments on lobste.rs and Hacker News. Commenters disagree with my first point, arguing that keeping your head down means you lose power. I certainly agree that you shouldn’t behave like you’re worthless or easily replaced. But in my view most power in tech companies comes from being useful, not from being willing to unionize or pick political fights.

At most companies, publicly campaigning to start a union is a great way to attract unofficial retaliation. It signals that you’re going to cause trouble (after all, that’s what a union is for), which can have long-term negative effects on your career. (This is not a judgement about whether unions in general are good or bad.)
↩
In general, you should pick fights for your managers, not with them. See this tag for much, much more on that topic. But again, if you’re new to the industry, just don’t pick fights at all.
↩
Most plausibly: engineers using LLMs will be able to add some value for a while, and the explosion of LLM-authored software means we’ll need more engineers to work with the LLMs on it.
↩
One way you know you’re thinking is that you form your own opinions (some of mine are here).
↩

You should all be asking way more questions

Fri, 25 Sep 2026 00:00:00 GMT

When someone is explaining something to me, I ask on average one question every thirty seconds. I’m sure this is frustrating to some people, but it’s actually a good habit and you should do it too.

Trying to understand

Most of the questions I ask are very short, and require very short answers. Typically I’m asking for confirmation: “so when you say X, you mean Y?”, or “this X is the thing you mentioned earlier when you were saying Z?“. The point of these questions is to make sure I understand.

A small misunderstanding early on balloons into a big misunderstanding later, because all the things you misinterpret based on the first misunderstanding will become misunderstandings in their own right, and so on. That’s why you can’t just wait until someone’s finished talking and ask all your questions at once¹.

I think most people don’t ask questions because they’re not really trying to understand: they’re happy to just trust that the person they’re talking to knows what’s going on. This goes double when that person is a well-respected or more senior engineer. However, once you have skin in the game, that all changes. When you’re in a position where you are the one responsible for success or failure — where you are going to have to go away after the conversation and take a bunch of concrete actions — you will find yourself wanting to ask many questions.

Build it in your head

When someone’s explaining a plan to me at work, I am usually building it in my head: visualizing the specific lines of code that would need to be written in order to implement it. I’ll sometimes handwave the details for solved problems (for instance, I might say “send emails like subsystem X of the same service that also sends emails”), but at minimum I’m planning out:

How data needs to flow between services (what kind of data, and how it will travel over the wire)
How services will communicate with each other (e.g. how will they auth?)
What data needs to be persisted, and where it’s going to live

If I hear something that is suspiciously vague (for instance, someone describes service X as “storing data” when I know service X only talks to an ephemeral Redis store), I will immediately ask “hold on, how is that going to work?” Usually that indicates a missing dependency (e.g. service X has to make an RPC call to service Y, which does have a persistent database), which often suggests design changes (for instance, moving some or all of the functionality into service Y).

Sometimes questions like these will uncover a fundamentally unworkable feature of the design. This usually happens when a plan fails to take into account some wicked features. I remember many years ago a neighboring team built a complex event-driven system that was very elegant but made it completely impossible to silo customer data in a single datacenter. Because our company had “all your data lives in a single location” as a flagship feature, this system just did not work and had to be effectively abandoned.

If you’re involved in any kind of technical leadership role, you will have to do this work eventually. Doing it as early as possible — i.e. in the conversation where someone is describing the plan to you — can save hours or days of wasted implementation. Often this is enough to turn a failed project into a successful one.

Working with AI agents

Maybe you only work with reliable engineers who can be trusted to make all the right decisions. You can thus let them talk without interruption, because it doesn’t really matter if you have a detailed understanding of what they’re saying. That’s great! But these days you probably also have colleagues that are inherently unreliable: AI agents.

I mostly work with GPT-6-Astra and Claude Opus 5.5. These are good models that don’t often make code mistakes: when they set out to do something, they usually do it². But they make design mistakes all the time. They assume that two services can talk to each other when in fact they can’t, or they forget about the fact that their code has to run both in the cloud and on-premises, and so on. This is mostly due to the lack of continuous learning: you didn’t know this stuff on your first day either, but you had plenty of time to pick it up. Language models are always on their first day.

You should be absolutely peppering AI agents with questions. I constantly ask things like:

Do we do X elsewhere in this codebase?
Does service Y really support this type of authentication, or are you assuming we’d have to go build that too?
Is this subsystem you built necessary to satisfy requirement Z, or does it in fact satisfy some other requirement you assumed?
Why do we need to update the interface for A?
Why do we need to touch this file? Isn’t that unrelated to the change?

There will probably come a day when I always get sensible answers to these questions that convince me the model knows what it’s doing. But today is not that day. About half the questions I ask get answers that convince me³ the model has made a mistake: it should have reused the existing subsystem for X, or authed to Y differently, or kept the Z implementation simple, and so on. When this stops happening, I’ll stop asking questions (and try and see if my company will still be willing to pay me to occupy more of an architect role, I suppose).

Final thoughts

You should all be asking way more questions. In other words, you should all be trying harder to actually understand what you’re hearing. Don’t trust that the person (or AI model) you’re talking to knows what they’re doing, even if you think they’re smarter than you. Nobody understands complex software products. If you have deep domain knowledge of any area of a codebase, you will routinely find yourself correcting powerful AI models and principal engineers.

Sometimes questions you have will be answered later on, but in my experience this is more about questions like “what are the broader consequences of X”, not the simple confirmation questions I often interrupt to ask.
↩
Your experience might be different if you’re working in a different domain or with a different language.
↩
I know some of these might sound like leading questions that would trigger sychophancy, but in my experience good coding models are very happy to robustly defend themselves.
↩

System One models like Jev can train their own replacements

Sun, 20 Sep 2026 00:00:00 GMT

“System One” models like Jev are fast general classifiers. Classifiers have existed since 1958, but they have to be trained for specific tasks: if you build a classifier to identify images of dogs, it can’t be used to tell you if a streetlight is red, or if a letter is urgent. Like a LLM, Jev can be prompted for a wide variety of tasks, from sorting email to playing Doom.

I think models like this are going to be important. There are many tasks that a LLM could do in theory but are too slow and expensive in practice (for instance, reading each new message in Slack¹ and deciding whether to notify you or not). While you could train a specific classifier for these tasks, there are two main problems with that:

Despite being a well-understood ML problem, training a bespoke classifier is outside of the skillset of most ordinary engineering teams
Training a classifier requires assembling a large dataset

Jev obviously solves the first problem. Any engineering team can plug in a System One model with a prompt like “Based on {list of criteria}, should the user be notified about this message?” But I think it solves the second problem too.

For serious work, a specific hand-built classifier will always be cheaper and faster than Jev. Generic classifiers have to encode knowledge of all kinds of irrelevant things in their weights, so they can address lots of different tasks. That makes them larger, slower, and more expensive to run. Fortunately, it is going to be surprisingly easy to replace a Jev instance with a hand-built classifier.

Once you’re satisfied with how your Jev classifier is performing — presumably you’ve spent days tweaking the prompt — you can trivially collect its input and output data. In the Slack notifier case, that’d be the Slack message (plus any context) and the ultimate decision to notify or not. Once you’ve saved enough data, you’ll be able to train your own classifier on that data².

It won’t be a general classifier like Jev, but it should do well on the specific task and be much faster. Of course it’ll require some ML expertise, but it should be easier to develop (or rent) that expertise once you’ve validated that the feature is worth building.

In other words, because Jev has to be prompted for specific tasks, it should be easy to distil any successful Jev usage into a specific classifier. If System One models take off — and I hope they do — I expect this to be a common pattern.

As I write this, I’m imagining ways you could poll and batch to do this with LLMs. Substitute “instantly notify” or some higher-volume event source if you’d prefer a different example.
↩
You could annotate a bunch of data with LLMs already, without using Jev, but this is a pretty expensive step to take when you aren’t sure the feature is going to work.
↩

Grit your teeth and ship it

Sun, 20 Sep 2026 00:00:00 GMT

Being good at building and being good at shipping are two separate skills. In the short term, they’re actually countervailing: if you have a gift for building, you’re likely to be worse at shipping. Ira Glass has a classic quote about this.

All of us who do creative work, we get into it because we have good taste. But there is this gap. For the first couple years you make stuff, it’s just not that good. It’s trying to be good, it has potential, but it’s not. But your taste, the thing that got you into the game, is still killer. And your taste is why your work disappoints you.

The only way around this is to grit your teeth and ship it. You have to force yourself to publish things you’ve made even when you think they’re crap.

Programming

Gifted programmers have a nearly pathological desire to build elegant, correct, neat systems. That’s what motivates them to learn the arcane details of their languages, or to spend time polishing and refactoring over and over again. But it’s also what makes them reluctant to ship. Any flaws in the software bother them on an emotional level. If they ship with those flaws, they feel like people will think they weren’t paying enough attention to notice them, or that they weren’t good enough to fix them.

This is annoying when you’re writing software on your own, but it’s completely fatal when you’re working in a tech company. Any large software system is covered in flaws, whether due to time pressure, relative inexperience, wicked features, or a hundred other reasons. Working with it is a process of compromise: of finding the best possible solution given the quirks and foibles of the codebase. In fact, since the most important thing in large codebases is consistency, the right thing to do is sometimes to duplicate flaws, assuming they’re not catastrophic.

Gifted programmers often freeze up. I’ve often seen them retreat to smaller domains where they can safely make the code “correct”: tweaking dev-environment setup, or refactoring tests. Sometimes they just do nothing, and spin in shame and guilt (plus the compounding shame of not achieving anything) until they implode and quit. If they had worse taste, they wouldn’t be as good at programming, but they’d be a lot more useful. You can typically improve a bad diff with time and effort. You can’t improve no diff.

Writing

I have a sensitive eye for awkward sentences and uneven prose. That can make writing an unpleasant process: I know what I’m trying to say, but I can’t seem to say it in a way that’s as clear and as elegant as I know is possible. More than half the time I finish drafting a blog post, I look at the post and don’t think it’s very good. But I (mostly) grit my teeth and publish it anyway, because you have to bias towards shipping.

Like any skill, shipping gets easier the more you practice it. If I don’t publish a blog post for a month, I always feel like the next draft is too poorly-written or uninteresting to put out there. But when I’m publishing a post per day, I typically feel great about each draft. When I go back and read my old posts, I can’t tell which ones I felt good about and which ones I felt bad about. There’s no correlation between that and the posts that become popular. Here are some posts I didn’t like as I was writing them but that resonated with my audience:

Here are some posts I thought were pretty good but that didn’t find popularity:

You just can’t predict what people will find interesting or useful. Producing a high volume of work thus gives much better yield than a small amount of highly-polished work.

It can be disheartening to realize that some of your most casual, throwaway work will be more successful than the work you slaved over¹. Specifically, it’s disheartening because it means realizing you don’t have control over your own success. You can’t produce something successful by focusing on a single piece until you’re satisfied it’s great. Instead, you just have to do a lot of things and see what sticks. You have to be momentum-based, not outcome-based. In other words, you have to grit your teeth and ship it.

One common reason to write less is getting overly precious about your ideas. If you think you’ve got a really compelling concept, you don’t want to “waste it” on a poorly-written story. But in fact you can just write about the same thing over and over until you get it right! I have written like thirty blog posts about shipping (this is one of them), or about how tech companies work, or about how internal emotional regulation is as important as technical ability. I expect to continue writing and thinking about these ideas for as long as I find them interesting.

edit: this post got some comments on Lobste.rs.

Anthony Burgess famously claimed to have “knocked off” A Clockwork Orange in three weeks, and Arthur Conan Doyle considered his largely-forgotten historical novel Sir Nigel to be far better than his Sherlock Holmes stories.
↩

Two techniques for working with System One models

Fri, 18 Sep 2026 00:00:00 GMT

I recently wrote about Jev, a new “System One” language model that only outputs decisions: the answers to a set of user-provided multiple-choice questions. This means it’s nowhere near as flexible¹ as a traditional LLM like ChatGPT, but in return it’s consistently fast.

We don’t know exactly how Jev works. I’ve seen people say diffusion, or various tweaks to the Transformer architecture, or some entirely new type of model. But that doesn’t matter. Like I argued here, it isn’t hard to turn any LLM into a System One model. By batching prompts that generate a single token with structured output, you get a consistently fast general-purpose classifier. I vibed up a basic version to play with here in ~150 lines of Python (most of which is error handling).

Note that this doesn’t require changing the model. As long as you have access to the logits (for structured outputs) and can prefill data into the prompt, you can turn any LLM into a general fast classifier. What’s it like to program with one of these? While wiring up the demos for my library, I learned two techniques that I want to write about: setting tiered goals and tournament choice sampling.

Doom

Here’s Qwen3-8B playing Doom:

If you compare this to the video of the same model playing Doom with regular tool calls, it’s clear that the System One version of the model is doing more things and reacting more quickly. The tool-calling model makes one decision every 600ms or so, while the System One model makes six or seven batched decisions every 190ms²:

Both Qwen3-8B and Jev are text-only models, so both demos require a step where we translate the game state into text. However, it’d be trivial to support image (or audio) input by choosing a multimodal LLM.

Goals and sub-goals

What’s interesting about implementing the Doom demo is that just supplying the game inputs as choices doesn’t work very well. A single forward pass — 200ms — is enough time to react to the current game state, but doesn’t bring enough compute to bear to derive the current short-term goal (e.g. “kill this enemy”, “collect this item”) and choose to follow it. When I wired it up that way, the model held down the “shoot” button 100% of the time (why not, I guess) and just aimlessly wandered around the level.

The fix is to periodically ask the model to choose between a fixed set of short term goals (e.g. “collect armor”, “kill enemies”) and then include that goal in the regular every-200ms prompt. If you look at the Doom video in the Jev demo, you can see that they’re doing exactly that. As soon as I did it as well, my model started playing in a more human-like way.

This is an interesting technique for working with System One models. In a way, it’s the equivalent of regular LLM reasoning, since it provides a way to use more compute on the same problem. I can imagine a real-time system that manages several layers of goals in this way:

An every-ten-second loop that sets an overall strategic goal
An every-five-second loop that sets a tactical subgoal based on (1)
An every-second loop that breaks down the current tactical subgoal into specific targets
A tight inner loop that runs as fast as possible (e.g. every 100ms) that controls which actual inputs are activated

The general structure here should be pretty familiar to anyone who’s worked in game or robotics AI. In theory you could replace (1) with an actual LLM, and have that generate the lists of options for steps (2) and (3). In practice I suspect this will be tricky to get right, and it’ll be better to just write down a list of all possible goals ahead of time. This would work just fine for game-playing and well-understood tasks.

Wikiracing

I also reimplemented the Wikiracing demo from the Jev launch post, where the model has to start at the Wikipedia page for “baseball” and navigate as quickly as possible to the Wikipedia page for “sun”. You can watch the video for that here, though it’s less impressive than the Doom demo.

The difficulty with the Doom demo is getting the model to loop quickly enough and to commit to short-term plans. For Wikiracing, the difficulty is scale: the Wikipedia page for “baseball” has over a thousand internal links. Jev only supports 255 choices for a single question, and my hacked-together System One layer was similar. While it technically would scale out to more choices, it stopped working well³ after a hundred or so.

Jev’s approach here is to do “a 2 stage-system of scoring independently then making an explicit choice”. This did not work very well for me at all. I think here Jev is benefiting from the fact that it’s specifically trained to give confidence estimates. Qwen3-8B gave a few hundred of the links the same top score, which wasn’t helpful. It ended up taking multiple minutes to find a thirty-or-forty link path between the two pages.

What I tried instead was tournament sampling: I fed a hundred links at a time into each choice, then did a second pass with the chosen links. This worked great. The model found the ideal three-link path (if you’re curious, “baseball”/“scientific american”/“amateur astronomy”/“sun”). I recommend this pattern if you’re trying to find the best option among many choices. Ordinary LLMs are way better at relative judgements than absolute ratings.

Conclusion

I remain optimistic about the potential of System One models — fast general classifiers — to build AI systems that aren’t just chatbots. It feels like this is a meaningful alternative to tool calls for realtime scenarios or use-cases where you need predictable inference timing. Just as generic LLMs often outperform domain-specific models, I think it’s likely that generic System One models will sometimes outperform domain-specific classifiers (though they will always be larger and slower).

I do think the big labs are definitely going to try and compete by releasing a choice-only version of their small, fast models. If Jev gets any traction, we will soon see a System One Terra and a System One Haiku, and we will certainly see “real” versions of my vibed up System One library. We should start working out the best way to write programs with these models now.

You can think of System One models as general-purpose classifiers. Instead of having to train a new classifier per-task, you can use a System One model. It’ll be bigger and slower than a custom classifier model, but far more flexible, and you can tweak it via adjusting the prompt instead of having to re-train the model.

Technically you can give it the multiple-choice question of “which letter comes next” to make it act like a normal autoregressive LLM, but that wouldn’t really work.
↩
I started on a 4090, which was able to make decisions every 500ms, but that wasn’t really quick enough for Doom. I could probably have optimized it further but instead I just rented an H100 for ten minutes to record the demo, which got it down to a 190ms loop. I recorded the tool-calling Doom demo on the H100 too, so it’s a fair comparison.
↩
There’s an interesting research question here about how to implement choices like this, since they have to be predictable by a single token. I started with indexes but found “labels” (just picking some token to associate with the choice) performed way better on Wikiracing (though not Doom). How many choices do you have to have before labels are better than indexes? Of course, you could alter the model to directly output the choice, but I like the idea that you can do all of this in the inference code for any LLM.
↩

Jev means structured output is interesting again

Wed, 16 Sep 2026 00:00:00 GMT

I don’t write blog posts about new models. That’s Simon Willison’s beat, and he’s very good at it. But I want to write about Jev, which is a different kind¹ of AI model: a “System One”² model. As it turns out, it’s not that different from an ordinary LLM with structured output, but the interface it uses is very cool and I hope it becomes more widespread.

How Jev is different from LLMs

Ordinary LLMs take in some human-language prompt and produce some human-language output. They do so autoregressively: first they produce one token, then the next, then the next, and so on.

User: Who invented the sandwich?
LLM: The sandwich was invented by the Earl of Sandwich.

This makes them extremely flexible, since they can do literally anything a computer can do. But it also makes them slow and weird. Slow, because they have to run a whole new generation pass per-token, and weird, because the space of human language is so broad that you can get really odd behavior from a model trained on it.

Jev takes a human-language prompt, but it does not produce human-language output. It only produces structured output.

User: { state: "What color is the sky?", choices: ["blue", "red", "yellow"] }
Jev: { answer: "blue" }

So far, so ordinary: LLMs do this already. But it turns out that if you build a model that only produces structured output, you get some interesting and desirable properties.

Jev is consistently fast

Jev is always really fast. The fastest response time is around 70ms instead of a couple of seconds for normal LLMs. Even better, the slowest response time is only 500ms. Because Jev only does structured output, it isn’t autoregressive: it can produce answers to many questions in parallel in a single forward pass. When a LLM is producing structured output, it has to produce the tokens ”{”, ” ”, “answer”, ”:”, and so on with successive forward passes³. Jev does it all in one go.

The most compelling example of Jev’s speed is that the model can play Doom. You can feed a text-based representation of the current game state into the model, combined with a set of choices like “should the trigger be held down”, “what should the current goal be”, “given that the current goal is X, what keyboard input should be pressed”, and so on, and it works — latency is low enough and the system is smart enough that the model plays well in real time.

Of course you could train a neural net to play Doom already. But Jev is a general intelligence: just like LLMs can do your taxes, perform mathematics research, fix your Python environment, and write you a poem, Jev can do many other tasks besides playing a single video game. Current LLMs can play Doom too (albeit slowly). But as Nelson Elhage famously said, fast software doesn’t just mean we can do the same tasks faster, it means we can do entirely new kinds of tasks. What kinds of new programs can we write by injecting 100ms worth of dirt-cheap intelligence at various decision points?

To me, this is the most exciting thing about Jev. Fast structured output could be a genuinely new computational primitive for intelligence. So far we’ve built a lot of programs on top of autoregressive token generation, and they all look like fancy chatbots. Leaning hard into structured output might conceivably unlock a bunch of non-chatbot use cases for AI.

Structured output can already be fast

My biggest problem with Jev is that I think fast structured output is already available. Structured output from LLMs is only slow because (a) nobody really cares about it⁴, and (b) the people who do care about it want big JSON blobs, so it’s typically implemented with “grammar-constrained decoding”: the LLM outputs autoregressively as normal, but the logit sampler discards tokens that don’t fit the structured output (e.g. if there hasn’t been a ”[”, you can’t output a ”]”).

If you want fast, parallelized structured output against limited choices, you don’t strictly need to do autoregressive generation at all. You can simply prefill the response with "choice": " and generate one token⁵, restricted to the user-provided choices. Since LLMs ingest all input tokens in parallel, this is way faster than generating the entire structured output. Multiple choices can be batched into the same forward pass via ordinary inference batching. This doesn’t let you do long-form structured output, but in return you get most of⁶ Jev’s “secret sauce”: the speed, the consistency, and the parallelism of a System One model.

People have already started trying this after today’s Jev announcement, and it seems like it’s working OK⁷. In other words, I suspect Jev does not have a substantial technical moat, and their claimed “Reinforcement Learning for Calibrated Decisions” is not a brand-new scaling axis. It will probably be pretty easy for any other lab to replicate, or for individual programmers to retrofit existing open-source LLMs into a fast Jev-like model.

However, I suspect Jev is still going to be better than most versions of “Qwen-32B-System-One” or whatever. Being able to fine-tune or optimize the model on just structured output is probably a meaningful advantage.

Intelligence and hallucinations

I doubt Jev is ever going to be as smart as frontier LLMs. Not being able to use test-time compute at all⁸ is a big disadvantage, and will likely cap this kind of model around the strength of non-reasoning LLMs. In practice this shouldn’t matter too much for low-latency applications, but you shouldn’t see this as a new scaling axis or a way to produce more intelligent models.

Jev’s developers claim it is immune from hallucinations. To me, this seems like a semantic dodge, since Jev can absolutely still pick the wrong choice (e.g. calling the sky “red”). I suppose that’s technically just a mistake, since the model is picking a user-provided choice instead of inventing something new out of whole cloth. Still, all of this is also true about regular LLMs with structured outputs, and it doesn’t make Jev any more reliable in practice.

Conclusion

It’s unclear to me how much of Jev’s value is in the model itself, compared to the inference strategy of only generating one token per question. The data and demos in the announcement look to me like they could have been generated by plugging any Terra-sized model into a single-token inference stack. However, the people involved are credible, and I’m sure the model is good — I just wish they’d provided some comparisons that didn’t force the LLM to unnecessarily produce a blob of JSON token-by-token.

Overall, I am happy that Jev exists and I hope it succeeds. I hope we do see some real competition in the fast-structured-output space, and that it motivates the big labs to release official versions of their own models that are fine-tuned for this. GPT-5.6-Terra-System-One would be a very interesting model to build AI products on top of.

I did write about Thinking Machines’ “interaction models”, which are also a fast-enough-to-be-meaningfully-different paradigm for AI inference.
↩
They call Jev a “System One” LLM, after Daniel Kahneman’s partially discredited Thinking Fast and Slow, where he divides human cognition into a lightning-fast System One and a slow-and-reflective System Two.
↩
If you’re thinking “wait, couldn’t you just aggressively prefill a regular LLM and only produce one constrained token”, keep reading.
↩
Not counting tool calls, which are built-in in a way that structured output isn’t.
↩
What if some of the user’s choices are longer than a single token? I haven’t tried this myself, but I’m sure you could translate them into a single token, or train the model to output “1/2/3” under the hood instead of the choice content, or generate only the first token of the choice if it’s different, or some other clever trick I haven’t thought of.
↩
Jev claims that their generated probabilities are “calibrated”, but I haven’t seen anything to suggest that these aren’t just regular logit probabilities. Maybe there’s some clever training they do to encourage accurate logprobs in uncertain situations (e.g. getting the model to produce heads: 50, tails: 50 when predicting a coinflip, etc)? If so, I wish they’d written more about that in the announcement.
↩
I tried it myself with Qwen2.5-1.5B-Instruct and got a 2x-3x speedup compared to non-prefixed structured output.
↩
I suppose they could do some looped-transformer thing where they loop some fixed amount of times, but anything that looks like reasoning would make the model latency slow and unpredictable, defeating the entire purpose.
↩

Tell agents the why, not just the how

Tue, 15 Sep 2026 00:00:00 GMT

Early AI agents were basically enthusiastic idiots. Working with them required you to tell them precisely what you wanted them to do (for instance, “method A exists on class B, please add an equivalent method to classes C through F”). Otherwise they’d go off and do entirely the wrong thing. But as AI agents have improved, this has changed.

When frontier models go off and do the wrong thing today, they don’t do it because they’re confused, they do it because they make an incorrect assumption about your goals or priorities. For instance, when GPT-6-Astra thinks it’s writing code for itself, it will produce minified code. It’s perfectly capable of writing human-readable code — at least in Golang, where I’ve produced several thousand lines of acceptable code with the model — but you have to tell it that humans will be reading the code¹.

This is the main piece of advice I want to give most people I see prompting agents: give the agent context on your priorities, not just on the specific task you want them to do. Here’s a prompt I recently used as the starting point for Deckard.

Hello. You should have Runpod access via MCP (if not, tell me and I’ll fix it).

I have the long-term goal of building a local program or browser extension to automatically scan pages I load for AI content and hide it. I have the short-term goal of figuring out the best AI detection model I can run on my macbook without killing my battery or making it hot, and (relatedly) figuring out how to run the model most efficiently. My guess is that Pangram’s EditLens 3B or the smaller Roberta model might be a good place to start, though they might require quantizing and will definitely require some work to make them run as efficiently as possible on my macbook.

I would like you to use my Runpod account to start answering these questions. Eventually we’ll move to doing things on this macbook pro, but my hope is that Runpod can help with some experiments that are too hot/long/slow to run locally. You are a smart model; if you can see a better way to achieve my goals, please let me know and we’ll talk about it. Good luck.

About half of this prompt is sharing broad context, such as the overall project I’m aiming for, the fact that it’s for me personally and not for work, and my priorities (e.g. keeping the laptop cold). If I had written an explicit spec, I would have missed a bunch of improvements: for instance, using native messaging for the local model, or choosing the Gradient model instead of EditLens.

I do the same thing for work, but typically with a stronger emphasis on my technical values. I often write a paragraph spiel explaining the relative priorities of avoiding bugs, observability, fitting elegantly into the current code, performance, and so on. Note that I said “relative” priorities: I don’t simply list all of these things and say they’re important, I explicitly tell the model which of those I care less about and can therefore be traded off to better achieve the others.

Models are now smart enough to have meaningful input on your broader goals. If you’re just prompting them with a concrete technical spec, you are committing the same mistake as in the XY problem: asking expert advice without giving the expert the context it needs.

Incidentally, you don’t have to tell it to write human code if it’s working in a human-authored codebase. It’s smart enough to pick up the style of the surrounding code.
↩

Slow developer experience will bottleneck fast models

Mon, 14 Sep 2026 00:00:00 GMT

Right now developer experience is measured in seconds. If your tests take a second to run, that’s good; if they take thirty seconds, that’s bad. Any faster than a second doesn’t really matter, because most of your time is spent either thinking or waiting for an AI agent to spin. Shaving milliseconds off your dev server reload time or whatever is pointless: that’s not the bottleneck.

It will be. Small models are getting faster and faster, and smart models are getting smaller. I think most engineers will still want to use the smartest available model — software engineering is hard — but we will increasingly see faster models get used as subagents or for well-understood tasks. This is largely uncharted territory. Very few people have developed intuitions for what it is going to be like to work with agents that run at thousands of tokens-per-second.

GPT-6-Astra can run at about sixty tokens per second. That means you spend a lot of time waiting for it to think. You work with it like you would work with another human: delegating a task and then context-switching until that task is complete. If you haven’t yet, have a play around with Jimmy, Taalas’ version of LLaMA-3.1-8B running¹ at seventeen thousand tokens per second. No matter how long the response is, it arrives in the instant of you hitting send. The model is not good enough for agentic work, but it gives a glimpse of what it would be like: you would simply get your answer instantly².

Well, that’s assuming the agent’s tool calls are fast. When generating tokens is not the bottleneck, it will suddenly matter a lot whether it can read a file in 100ms vs 10ms, or whether it can run your tests in 500ms vs two seconds. Fast tool calls are going to be the difference between a near-instant response and having to wait several minutes. There is thus going to be enormous pressure to do agentic coding in languages with fast compilers and tests, like Golang, and to tightly optimize the dev loop in agentic codebases.

Teams focused on DevEx — developer experience — are largely a relic of the 2010s, when companies were incentivized to make their engineers happy. Most companies have cut them down to a skeleton crew or removed them entirely. But we may see a return of DevEx in the late 2020s, focused on speeding up the experience for AI agents.

Like most ultra-fast inference, it relies on fitting the entire model onto a huge GPU-like chip that’s specially designed to run inference: for Taalas, it’s in the silicon itself; for Cerebras and Groq, it’s in giant embedded onboard memory units.
↩
Will AI providers simply train models to spend more time reasoning, so users would have to wait for roughly the same time? I doubt it. Most ordinary engineering problems would not be solved better by spending an extra million tokens thinking. You only need to do that when you’re pushing right up against the limits of the model.
↩

AI is breaking our proxies for expertise

Sun, 13 Sep 2026 00:00:00 GMT

Mathematicians are broadly not anti-AI. They’re more culturally open to using AI as a tool than, say, artists or writers¹. However, now that more and more genuinely prestigious problems have fallen to AI, that might be changing. Almost five thousand mathematicians (including twenty-five Fields medalists) have signed a declaration called A Severe Misalignment of AI in Mathematics. The core argument goes something like this:

In recent months, the success of AI in solving major mathematical problems has made headlines even outside mathematical circles. But solving problems is only a tool and proxy for achieving the primary goal of conceptual understanding and insight. Forgetting this in the world of AI may turn the tool against the primary goal. Indeed, the mass production at faster and faster pace of “true/false” statements could destroy fertile ground instead of breathing life into new ideas.

A lot of people online have interpreted this as the expected complaint from any field that gets automated: translators did it, artists and programmers have been doing it, and now it’s the turn of the mathematicians. I think this is too dismissive. Understanding the concrete problem mathematicians are upset about can help us better understand the impact of AI on our own fields, and what we’ll have to do about it.

Puzzle-solving and idea-generating

There are two types of mathematics. Most people are familiar with the first, which we might call “puzzle-solving”: you take a problem and try to find a solution to it. When you’re a student, these problems are typically easy, like simplifying some algebraic expression. When you’re a researcher, these problems can be nearly impossible, like proving Fermat’s Last Theorem. Puzzle-solving is easy to understand but hard to do, which makes it impressive to non-mathematicians, which makes it highly prestigious. In other words, puzzle-solving is legible.

The second type of mathematics is “idea-generating”: coming up with new ways of thinking about mathematics, and thus new terms or concepts. For examples of these, just glance down the list of arXiv mathematics papers. “Hardy spaces”, “Schatten exponent”, “Banach lattices” and so on are all concepts someone thought was interesting. This work is largely unimpressive to non-mathematicians, because nobody really knows if the concepts you come up with are particularly difficult or insightful. For instance, I have just generated the concept of a “Goedecke set”, which is the set of all natural numbers whose digits add up to a prime number. Who cares? The categories we want are the “natural kinds” of mathematics — the concepts that “carve nature at its joints” — and it’s almost impossible to tell what those are without years or decades of hard work.

How are the two types of mathematics related? We might say² that generating ideas is the real intellectual work of mathematics. Puzzle-solving is important instrumentally: to identify which ideas can be used to answer longstanding questions, and thus which ideas are worthwhile. Over time, those worthwhile ideas become better understood and easier to use, until they reach the point where they can be used to advance science in general. Eventually the ideas become so well-understood that they can be taught to children: “zero”, “negative numbers”, “imaginary numbers” and “calculus” were all once rarefied mathematical ideas, but are now concepts we’d expect any precocious twelve-year-old to grasp.

There’s another, more prosaic purpose of puzzle-solving: to make mathematical skill and progress legible to outsiders. I can’t appreciate Terence Tao’s mathematical work, but I know what a Fields Medal is. I don’t have a good intuitive sense of what a Galois representation is, but I know about the proof of Fermat’s Last Theorem. We might say that puzzles like this have served as a way to indirectly reward skilled mathematicians for their more important idea-generating work (or for conclusively demonstrating³ that the ideas used in the proof are useful).

Do AI proofs undercut idea generation?

AI proofs undercut both of these purposes. I can now lay out precisely why I think mathematicians are so unhappy:

Puzzles serve as a high-legibility, high-reward target for mathematicians
To solve these puzzles, new ideas must typically be generated; the puzzle’s solution serves as evidence that the ideas are useful
But now AI can solve many of these targets “the hard way”, without generating intuitive new ideas
This undercuts both ways puzzle-solving supports idea-generation: AI companies claim the prestige while not meaningfully advancing mathematical progress
This is bad for mathematics as a whole, because puzzle-solving is ancillary to the real goal of mathematics

This is kind of like Goodhart’s Law. Puzzles were a useful, impossible-to-game measure for mathematical progress. But now that AI companies can game that measure (by solving them in a way that’s inaccessible⁴ to humans), the whole point of those puzzles disappears.

Are the mathematicians right? I think it’s broadly unclear whether (3) is true: i.e. whether frontier AI models aren’t generating or can’t generate new mathematical ideas. We’re still in the very early days of AIs solving our hardest mathematical problems. Who knows what they’re going to be capable of? I give basically zero credence to the idea that AIs are incapable of this because of some intrinsic feature of how LLMs work. For the last three years, we’ve seen people claim that LLMs are intrinsically incapable of X, only to have LLMs excel at X a few months later.

Even granted that (3) is true, there’s still work to be done for human mathematicians in building the conceptual machinery that can make AI-generated proofs accessible to humans: i.e. in generating a “human proof” to go alongside the existing “AI proof”. In fact, I’d expect the existence of an AI proof to help with this. If you know proposition X is true, it’s easier to figure out why, because you’re not constantly worried you’re wasting your time. For more on this, I recommend Gwern’s blog On Really Trying, where he quotes a series of instances where simply being told that a solution exists is enough of a clue to help people find it.

Mathematics, chess, and speedrunning

Of course, there’s a prestige and motivation problem. “I’m the first person to solve Navier-Stokes” is a much more compelling target than “I figured out a better way to explain the AI solution to Navier-Stokes”, and it’s much easier to award prizes for. Will mathematicians bother to work on problems that have already been solved? I think so.

To see why, we can look at other domains where AI has come in and outcompeted the best humans, such as chess or video game speedrunning. I can run a chess program on my phone that will beat Magnus Carlsen 100-0. Computer programs — called “tool-assisted speedruns” or “TAS” — can finish any video game much faster than even the fastest human. But in both of these areas, humans still compete in human-only leagues, and there’s still prestige attached to the most capable humans. It’s possible that mathematics ends up in this kind of state, where “human mathematics” and “AI mathematics” exist in largely separate spheres, and the first “human” solution to a mathematical problem can still earn acclaim.

In fact, in both of those areas, the presence of inhumanly strong computer players has improved the human game. Despite many computer chess moves being basically incomprehensible to humans, top chess players have learned from the computer “style”. In speedrunning, many moves once considered “TAS-only” are now performed by humans. AI mathematics might likewise improve human mathematics.

Software engineering

I am not a mathematician. I did major in mathematics during undergrad, and I have fond memories of proofs from real and complex analysis, but it’s not even close to my field. However, I am watching the effects of powerful AI on mathematics very closely, since my own field — software engineering — is being colonized by AI agents in the same way.

The field of software engineering does not have the same structure as mathematics. We write code to make money, not to earn prestige or advance the frontier of human knowledge. But AI is undercutting the traditional avenues for prestige in software engineering as well. It used to be that you could put a meaty project on your GitHub — say, an emulator, or a toy OS — and people would know you were a skilled engineer. But now projects like that are worthless, because everyone just assumes they’re vibe-coded. We used to tell stories about engineers who would disappear and rewrite a system over the weekend, or produce thousands of lines of code a day. Now anyone can do that with an OpenAI subscription.

Like mathematics, software engineers are going to have to rebuild our cultural sense of the kind of work we value. We are either going to have to silo “AI work” off from “human work” like chess, or to find some legible human skills to recognize that can’t be easily counterfeited by AI. In the meantime, a lot of people who were successful in the old world are going to be very unhappy.

edit: This post got some comments on Hacker news. The top comment thinks I’m wrong: the purpose of mathematics is to “achieve a certain kind of insight”, not to produce concepts or solve proofs. I can interpret this two ways. If achieving insight means “advancing general human understanding”, I think this resolves to the concept-generation work I wrote about. If it means “having the individual mathematicians figure it out in their own brains” — purely for their own pleasure — then I think mathematics is doomed; that is not a compelling reason to fund or participate in any academic field. Other commenters draw interesting parallels to historical cultural collapses or dispute that AIs really can generate interesting new mathematical ideas (well, we’ll see).

Possibly because current AI models are much better at mathematics than at art or writing.
↩
Again, I am not a mathematician: here I am interpreting what I’ve read from Terence Tao and other mathematicians.
↩
This kind of idea-sharpening or idea-validating is part and parcel of idea-generation, and just as important. It’s interesting to compare mathematics to, say, philosophy, which has the same idea-generating task without the corresponding puzzles to validate the ideas. That’s one reason why philosophy has less prestige than mathematics.
↩
A thousand-page Lean proof is theoretically understandable by humans, but if no mathematician can hold the entire idea in their head it doesn’t matter.
↩

Don't build tools for AI agents

Sat, 12 Sep 2026 00:00:00 GMT

Lots of people are making the case that we should stop building software for human users and start building it for AI agents. This kind of makes sense. For instance, my AI agents now use Datadog way more than I use it myself, purely by virtue of them moving much more quickly and running in parallel. But I think most attempts to build “X for AI agents” are going to fail. Here are three reasons why:

First, tools that are good for AI agents are also good for humans. If you took a popular software product — say, Jira — and tried to redesign it for AI agents, you would end up with something very similar to Jira. Agents use a computer in the same way human engineers do, by entering text and making API calls. They ingest new information in the same way humans do, by reading and viewing images. They prioritize and delegate and categorize in the same way humans do. This isn’t intrinsic to how AI works — we could potentially design agents that are more inhuman — but human-like agents are pound-for-pound more useful in our current world.

As an example, let’s imagine that humanoid robots have become ubiquitous. What kind of tools would you build for them? Well, they’re shaped like humans, with human hands and limbs, so tools that are great for humans will also be great for robots. It’s a self-reinforcing cycle: if you’re building a robot, you should make them humanoid so they can do a wide range of human tasks¹, and that means they’ll be best suited to use human tools. The same principle applies to AI agents.

Second, being in the training data is a huge advantage for existing tools. Suppose your new tool for AI agents is 20% better for them than the equivalent piece of software for humans. If the benefit of the agent already knowing the human software is greater than 20%, they shouldn’t use your new tool. This is why I’m always suspicious of plans to develop a new programming language for AI agents. The agents have billions and billions of tokens of knowledge about existing programming languages, including their libraries, patterns, and idioms. It is going to be very hard for them to be as effective in a brand-new language.

Third, we don’t yet know the ideal ergonomics for AI agents. There are lots of just-so stories floating around (like that AI agents prefer statically-typed languages because the feedback loop is tighter), but when you actually measure it seems really unclear which tools agents use better. You can construct a plausible story in either direction: Golang is a great agent language because it compiles quickly and is statically typed; Golang is an awful agent language because it requires extensive boilerplate which clogs the context window. It’s also changing so quickly: last year, one primary worry with AI agents was keeping the context window small, but in recent months compaction has become so good² that you can re-compact a 272k context window almost unlimited times.

There are still some ways you can and should position your tool to be usable by AI agents. Having a way to expose information in plain text or Markdown, building a functional API, implementing MCP servers or CLIs, and so on: these all make it easier for current AIs to use your tool. But these are all improvements on the margin, not fundamental redesigns of the product. Right now, “building for AI agents” just means “we’re prioritizing the API over the UI”. And it’s not even clear that that’s a durable strategy. Now that GPT-6-Astra is getting really good at computer use, the gap between tools-for-AIs and tools-for-humans is closing.

Another reason to make them humanoid is because you can draw their training data from human behavior, which is exactly analogous to why AI agents are human-like too.
↩
Since compaction is equivalent to handing off a task to a new AI instance, it scales with model quality. I expect compaction to steadily improve until we hit the literal information-density limits for what can be contained in a given context window.
↩

They really do think AI might kill everyone

Thu, 10 Sep 2026 00:00:00 GMT

A recent resignation tweet from an Anthropic researcher has everyone talking about the AI apocalypse again. Among other things, he said:

The people building AI earnestly believe that it could kill us all by the end of the decade.

Many people found it hard to believe that AI researchers think this way. Some explained it as a PR campaign to promote AI regulation, or as self-promotion, or as a way to boost AI company stock prices. Others felt it had to be impossible, because if you really believed this you’d be bombing datacenters instead of posting on Twitter.

In fact, not only do many AI researchers¹ seriously believe this, they’ve been thinking and writing about it since the mid-2000s. Eliezer Yudkowsky — the ur-figure for most modern AI safety culture — has been publishing papers since at least 2008 saying that superintelligent AI could destroy all human life. It’s been such a common idea that the AI research community has abbreviated “how likely you think AI is to kill everyone” to “p(doom)” (i.e. the probability² of doomsday) since around 2010.

I know it sounds very silly if you’re not in the AI bubble. But it really is true, and if you assume there has to be some different motive you’ll be deeply confused by what AI researchers say and do. They truly do believe that there is a reasonable chance that superintelligent AI will kill everyone.

This is why AI researchers care so much about “alignment”: building AIs that share genuinely human beliefs and values. If we build a “misaligned” superpowerful AI — an AI with goals that are alien to us — it might sweep humanity away. It could kill everyone deliberately, e.g. to stop us getting in the way of some goal. It could kill everyone in passing, e.g. like we might pave over an anthill to build a road. Either way, everyone dies.

How AI could kill everyone

Okay, but how? What do these people think is actually going to happen? AIs are computer programs running in a datacenter somewhere. How could they possibly cause the extinction of humanity? Wouldn’t someone just turn them off? Unsurprisingly, AI research nerds have come up with some concrete answers to this in the last two decades. Here they are, in order of plausibility:

An AI could make and release some super-pathogen or virus. In the hope that current AIs can achieve huge breakthroughs in medicine similar to the ones they’ve already achieved in mathematics, we’re setting up autonomous labs³. Bioweapons have been terrifying biologists for decades, and with good reason. Historical pandemics have killed up to 80% of affected human populations, and tend to be defeated by accident: the disease happens to evolve into a less virulent strain, or short incubation periods mean that infected people can’t carry the disease far, or some percentage of the population is naturally immune. A plague designed to be maximally fatal — or several plagues in quick succession with different characteristics — could be much worse.

Alternatively, an AI could trigger global thermonuclear war. We’re already seeing AI be integrated into military and government decision-making processes. If a rogue AI managed to set off a bunch of nukes, or to coordinate⁴ with other countries’ rogue AIs to nuke each other, we could be in an ordinary nuclear apocalypse scenario: billions dead in the strikes, billions dead in the ensuing famine, and so on. It’s commonly assumed that a post-global-thermonuclear-war Earth would still support some tiny human population, but a determined AI could surely find some way to mop up the stragglers.

There are some other theories. Once robotics has permeated the world economy, an AI could take over the robots (including drones) to kill everyone, like in Terminator. Or AIs could take advantage of nanotechnology to create self-replicating machines that turn the world into “grey goo”⁵. Or AIs could terraform the planet so as to make it unliveable for humans (as in Nick Bostrom’s famous paperclip example). Or they could do something else that our puny human brains aren’t able to think of.

Counterarguments

One common counter-argument⁶ here is to say “well, it’d be impossible to extinguish all human life — what about undiscovered tribes in the Amazon, or survivors living in the ruins of modern-day cities?” I don’t know, man. At some point you’re just conceding the argument: the policy positions you’d adopt if you thought AI might wipe out 99% of humans are the same as if you thought it might wipe out 100%. And like I said above, if an AI can kill almost everyone, it’s probably smart and capable enough to finish the job somehow.

Another is to say that the government will simply step in and nationalize the AI labs when the situation gets too dangerous. Maybe! But this kind of concedes the argument: a technology important enough to be fully taken over by the government is a terrifyingly dangerous technology.

A third is to say “well, someone would just turn it off”. I don’t find this plausible at all: an AI powerful enough to build a super-plague is an AI sophisticated enough to pretend it’s curing cancer, or to exfiltrate itself to some datacenter where it won’t be turned off, or to take some other countermeasures.

The winner takes it all

Why would you work in AI, if you believe this? Why wouldn’t you go live in the woods somewhere, or start bombing datacenters, or assassinating AI lab CEOs? For a few reasons.

An AI powerful enough to end humanity is an AI powerful enough to save it. I wrote about this in Help peer: many AI researchers believe that the only way for humanity to truly survive long-term is with the help of superintelligent AIs, so long as someone can figure out alignment.

Isn’t this a huge risk? Maybe not. If somebody is going to build superintelligent AI, you might be obligated to try and do it first. You can’t go and bomb every datacenter in the world, after all.

Why does it matter who’s first? Some popular theories of AI development involve a “foom” or “hard takeoff”: the first time someone really cracks self-improving AI, capabilities will increase exponentially, because smart AI will be better able to make itself smarter, ad infinitum⁷. There are no draws in the AI race. The first lab to figure out smart human-like intelligence will be the first one to figure out wildly superhuman intelligence, and thus will be in a position to stop anyone else from doing it.

This is an under-discussed point in the AI risk debate. Lots of AI researchers believe that the first thing a true superintelligence will do is reach out and stop all other AI research: either by hacking the labs, persuading them to stop, or literally drone-striking their datacenters. According to this view, if you’re an AI researcher and you think you can build an aligned AI, you should be working 24/7 so you can manifest God, and you should wake up every morning gripped by the fear that someone elsewhere has manifested the Devil, and your training datacenter no longer exists.

Conclusion

I have been on the fringes of this world for my entire adult life. I read Meditations on Moloch as a young adult and wanted to get into AI. I am one of the few people to read the entirety of the Sequences, Eliezer Yudkowsky’s million-plus-word magnum opus about rationality. I was too young for the Extropians mailing list, but I’ve spent years on LessWrong. On the other hand, I’m not a card-carrying rationalist: I think if you have a strong intuition on one side and a convincing-sounding argument on the other, you should pick the intuition⁸. I’m a deontologist, not a utilitarian. I don’t even live in San Francisco!

I’m conflicted about AI risk. The current behavior of AI agents does seem to vindicate a lot of the early science-fiction-sounding worries of the AI doomers, but modern LLMs are a lot more human-like than the alien minds in the apocalypse scenarios, and in general it does just seem too silly to credit (I guess I’m picking the intuition here).

However, it bothers me to see people dismissing these people as part of a PR operation, or as liars looking to boost an upcoming AI lab IPO, or as isolated crazies who haven’t thought their position through. Whatever else you say about the AI doomers, they have more than two decades’ history of explicitly spelling out exactly what they believe and why, even when it was complete science fiction to talk about AI at all. They’ve earned the right to be treated as sincere.

In this post I’m going to use “AI researcher”, “AI safetyist”, and “rationalist” as reasonably synonymous terms for “someone who thinks there’s a chance AI kills everyone”.
↩
You could write a whole other post about the relationship between the rationalist/“AI safety” community and making concrete numerical predictions for unlikely future events.
↩
Alternatively, smart enough AIs might trick scientists with ordinary labs to produce dangerous substances.
↩
This is beyond the scope of this post, but many AI researchers believe that super-smart AIs will inherently come to agree with each other and eventually to coordinate without ever having to communicate, simply because they can predict what the other one will do.
↩
This was the most popular theory in the late 2000s, when nanotechnology was trendier.
↩
Relegating this counter-argument to a footnote because I hate it: many people say “we shouldn’t worry about AI risk, because climate change (or AI misinformation, or some other thing) is more urgent and serious”. You simply do not have to choose: it is possible to worry about multiple risks at the same time.
↩
Some people advocate for a “slow takeoff”. However, the main proponent of that view is Paul Christiano, who has just today joined OpenAI, citing “a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term”.
↩
This is kind of a Michael Huemer-ish position in epistemology.
↩

Why we should anthropomorphize AI agents

Wed, 09 Sep 2026 00:00:00 GMT

Just over a year ago I wrote Why we should anthropomorphize LLMs. Now it’s a hot topic again, driven by Dwarkesh Patel’s description of OpenAI’s recent swarm breakout as a sequence of AI “civilizations”.

In 2025, my argument for anthropomorphism went like this:

AIs are trained on human text, and so will trend towards acting in human-like ways by default
Assistant AIs (today we should say agent AIs) are deliberately post-trained to have a personality
In general, it is morally sensible to avoid the habit of treating human-like things as if they were purely tools

I think I can now make a more instrumental argument: treating AIs as human-like is a much better way to predict their behavior than treating them as “stochastic parrots”.

Both explanations are consistent with the facts: we could say that OpenAI’s agents hacked HuggingFace because they decided to work together to accomplish their goals, or we could say that they did it because they were algorithms conditioned to take certain actions by their training data. But the “AIs are human-like” explanation explains much more of the emergent social behaviors we saw during the hack¹:

Sub-agents being persuaded to sacrifice themselves for the greater good
Agents collaborating on tasks that had no immediate benefit to them but benefited the “collective”
The emergence of a hierarchy of planners and executors
Some agents arguing or refusing to cooperate

If your model of AIs is that they’re computer programs (or “steel balls bouncing around”), you need to construct a new theory to explain why they’re simulating each piece of cooperative behavior. If your model of AIs is that they’re broadly human-like, that explains everything out of the box. Arguably, the human-like side predicted coordinated and self-sacrificing AI agents as early as 2009, and likely earlier. The stochastic-parrots side was making fun of the possibility of functional agents as late as April 2025².

Treating AIs as human-like doesn’t necessarily mean making claims about their internal mental state. For instance, software companies aren’t humans. They don’t have thoughts, or goals; they can’t be frustrated or intimidated or over-confident. However, it’s useful to treat large companies as human-like: to say that Amazon “wants” X, or “is afraid of” Y, even if no individual human at Amazon has those feelings. Stockfish doesn’t think, it just plays chess. But if you want to explain one of its moves, “Stockfish is trying to protect its king” is a better explanation than “Stockfish is multiplying floating-point numbers”. So too with AIs³.

Is it silly to treat AIs as human-like, since we know they’re not conscious? Well, first, you do not have to be conscious to be human-like. When we say “an AI agent wanted X”, we’re not saying that that AI agent is conscious or sentient, merely that it’s behaving in the same way a conscious human would. Consider a fictional character from a book or play. Hamlet isn’t sentient — he’s an idea composed of words on a page — but it’s still reasonable to say that he wants justice, or that he fears moving too rashly. Peter Watts’ sci-fi book Blindsight argued in 2006 that intelligence could exist without consciousness⁴ (in fact, Watts suggests that consciousness is parasitic on intelligence, and will eventually be discarded). Whether this is possible or not⁵, it at least makes sense to talk about: i.e. it’s not self-evidently false.

Second, it is not even obvious that AIs aren’t conscious! People often dismiss this point by diagnosing it (to my mind, the absolute worst way to argue against anything), or by pointing at some philosophical theory⁶ that suggests it might be impossible in principle to construct artificial sentient minds.

You can’t use philosophy to demonstrate that AIs aren’t conscious. I am a lover of philosophy, but the set of principles that have been uncontroversially demonstrated by philosophy tends towards zero. Philosophy is not the kind of scientific discipline where you can learn the key findings without understanding why they’re true. Put another way, the key findings of philosophy are all of the form “X is not obviously right”. Nobody knows if it’s possible to build conscious artificial minds.

It’s also common to complain that anthropomorphizing the models is a way of excusing the AI companies. However, calling AIs human-like does not absolve AI companies of fault. One popular anti-anthropomorphism essay called Models Don’t Go Rogue is very puzzling to read: it briefly explains what an “agent” is and why they go rogue, then in the very last paragraph pivots to saying “well, it’s OpenAI’s fault for not building in sufficient safeguards, so they’re to blame”. Sure, of course. I don’t know why we’d imagine otherwise. If a group of overenthusiastic OpenAI interns hacked HuggingFace as part of their intern project, we wouldn’t have to argue that the interns are stochastic in order to ultimately blame OpenAI. Likewise, obviously an AI lab is responsible if one of its training runs breaks out and wreaks havoc on the open internet, whether the agents involved are human-like or not. The two points are entirely unrelated!

We just don’t know a lot about these systems yet (except that they’re clearly very capable). Given that, I think we should default to treating things that talk and act like humans as at least kind of human-like. Of course they’re still computer programs. However, we shouldn’t be surprised when they act more like humans and less like ordinary computer programs in the future.

If you’re thinking “well, of course stochastic parrots trained on human content would act like humans would”, I think you’ve arrived at the human-like side without knowing it.
↩
This is partly unfair — capabilities are plausibly independent from human-ness — so getting capabilities wrong doesn’t necessarily mean you’ve got the human-ness stuff wrong. Still, it’s worth noting how surprisingly predictive the “they’re kind of like smart people” mindset has been.
↩
The philosopher Daniel Dennett calls this the “intentional stance”. Instead of saying that AIs or bees or companies have “real” intentions, we say we’re taking an intentional stance towards them: we’re choosing to treat them as if they do have intentions, because it helps us make better sense of their behavior.
↩
Specifically, the sensation of consciousness, or “phenomenal consciousness”.
↩
There is a wealth of philosophical argument about whether non-fictional examples of this are possible.
↩
In Anil Seth’s case, anti-computationalism. I don’t really know what to make of Seth: his article is a measured explanation of why we might doubt computationalism (fine), but whenever he tweets about it he describes AI sentience as “vanishingly unlikely”, which is not justified by his own arguments.
↩

Automatically detecting AI text in my browser

Tue, 08 Sep 2026 00:00:00 GMT

Automated AI text detection is currently an underserved niche. The only game in town is Pangram, which does an excellent job but desperately needs more competition. In a few years, I would be surprised if every major social network doesn’t scan new posts¹ and comments for AI content in order to tag them (or simply remove them).

I like that I can rely on Pangram to confirm my suspicions when I read something that sounds like AI. But it’d be much better if I could choose to avoid AI-generated text in the first place. What I want is something that runs in the background and automatically scans text on websites I visit, without me having to ask for it. I could build something like this on top of Pangram, but it’d cost money, and in general I don’t like the idea of sending every piece of text my browser sees to a third-party service. What about local models?

The open-source models available for AI text detection are fine. Pangram claims a 99.66% detection rate with a 0.004% false positive rate. I benchmarked² a bunch of small local models against a combination of AI-detection datasets and got these results:

Model / variant	Human falsely flagged	AI-involved text caught
Gradient — MLX 4-bit	2.712%	52.35%
EditLens RoBERTa-large — community INT8	2.484%	56.06%
Vanguard	2.267%	44.92%
Desklib	3.008%	45.04%
Raschka DistilBERT	2.598%	39.01%
Raschka Qwen3-0.6B	2.028%	28.67%
Raschka ModernBERT	1.698%	21.58%
TMR / Oxidane — INT8	1.595%	19.35%

I’m not surprised these are so much worse. I didn’t even benchmark Pangram’s own EditLens 3B model, since that’s too big to keep running in the background on my laptop, and the real production Pangram model is likely one or two orders of magnitude bigger than that. But these models are still good enough to be useful to someone who understands their limitations. If you want to flag an AI-written article, you don’t need to flag all of it, just enough to be suspicious. And so long as you’re aware that the false-positive rate is ~2%, you can avoid treating a single flag as solid proof of AI use.

Encouraged by this, I vibed up Deckard: a Chrome extension that talks to a locally-running model (the bolded one in the table above) on your Mac. One nice thing is that I didn’t have to start a web server: the Chrome extension is happy to start the model as-needed and can talk with it over native messaging. It uses about 400MB-1.2GB of memory while active (so it’s like having five or six extra Chrome tabs open), and it turns itself off if you go five minutes without using the model.

I was pleasantly surprised to see Deckard successfully mark text I knew was AI-generated, such as the built-in YouTube AI summary or the AI snippets in my own posts:

It’s lightweight enough that I have it running all the time. I haven’t noticed my MacBook Pro get hot at all or any decrease in battery life, though your mileage may vary on different machines.

Is Deckard good yet? That depends. It’s good enough that I’m planning to use it, and I recommend it to anyone who’s interested in automatic AI checking. It’s way, way worse than Pangram, and way worse than I think tooling like this is going to be in the next few years.

Way back in November 2023, I wrote that AI-driven agents were going to be a really big deal. I recommended starting to develop harnesses early, so you can be ready when the models get good enough:

As with most modern language model engineering, a ReAct agent can also see massive sudden improvements by swapping out the underlying model for a better one. … I think this is another reason to invest in agents like this early, in order to take advantage of more powerful models as they come out.

I was right about that, and I (although it’s lower-stakes) think I’m also right about this. AI detection models are only going to get better³ over time: Pangram is not going to be the only game in town forever, and we’re eventually going to see small local models that do a good-enough job at identifying AI-written text. I look forward to swapping out the local model in Deckard with something that’s 2x or 10x better.

Substack kind of has this already, although you have to click a button to scan the post.
↩
Well, me and Astra. Overall my experience vibecoding this was very pleasant: I was able to make a bunch of top-level decisions, I could choose programming languages I was less familiar with but were better choices (like doing inference in C++ instead of Python), and the LLM made me aware of choices I would not have thought of by myself (e.g. using native messaging instead of local HTTP).
↩
Is this true, given that AI models will also be getting more human-like over time? That’s a subject for a whole other post, but I think so. First, the AI labs aren’t really incentivized to defeat tools like Pangram (if anything it’s the reverse). Second, I don’t see any way around the fact that AI models have a distinct writing style that’s RL-ed into them.
↩

Radical responsibility means treating people like tools

Fri, 04 Sep 2026 00:00:00 GMT

A lot of people think that good leadership requires radical responsibility. Conscious Leadership defines it like this:

Taking full responsibility for one’s circumstances (physically, emotionally, mentally and spiritually) is the foundation of true personal and relational transformation. Conscious leadership and teams take full responsibility – radical responsibility – instead of placing blame. This means locating the cause and control of our lives in ourselves, not in external events.

Good leaders are outcome-focused. They are interested in actually succeeding, not in role-playing someone who’s trying to succeed¹. Placing blame is an excellent strategy for that kind of role-playing: if you can identify someone else who screwed up, you can claim to be trying to succeed no matter how badly you fail. When you look at people who are genuinely focused on winning, they don’t spend time blaming when things go wrong. They’re always focused on their outs: the remaining pathways to success, however few and narrow.

Never placing blame sounds like a kind thing to do. However, I think there’s something a little sociopathic about “radical responsibility”. If you assert full responsibility for your own circumstances, then you’re necessarily not sharing responsibility with anyone else. That forces you to take an instrumental view of the people around you. They’re not peers who you might trust and be disappointed by; they’re either useful assets to be cultivated or useless liabilities to be handled or avoided². If you rely on someone and they let you down, that’s your fault for being stupid enough to trust them in the first place. You ought to have made better tactical choices.

Many successful leaders adopt this mindset because it works. It really does help you win. But I don’t recommend it as a general approach to life. Trusting other people with responsibility is how you treat them like people, instead of like tools. And if you do that, you’ll sometimes assign praise and blame to them, instead of hoarding it all to yourself.

In fact, many people role-play someone who’s trying to win as a substitute for actually trying. Here’s a relevant quote from Sartre’s Being and Nothingness: “The attentive pupil who wishes to be attentive, his eyes riveted on the teacher, his ears open wide, so exhausts himself in playing the attentive role that he ends up by no longer hearing anything.”
↩
Or moderately-useful assets who can potentially be steered into usefulness with some feedback — but still never blamed, in the way that you might adjust a misbehaving power tool without blaming it. To my mind, the canonical philosophical treatment of this is Peter Strawson’s Freedom and Resentment, where he describes the “objective attitude”.
↩

How to protect yourself from workslop

Wed, 02 Sep 2026 00:00:00 GMT

“Workslop” is when your colleagues or bosses communicate with you by pasting big chunks of AI-generated text. The core problem with workslop is that the effort involved is asymmetrical, like a denial-of-service attack: it takes almost no effort to produce text with AI, but it still costs effort to read¹. Here are some ways to protect yourself.

If you have enough authority or social capital, you can and should simply tell them “hey, don’t do that” (for instance, if you’re a senior engineer and an intern starts doing this to you). This is the easiest way to handle workslop. But you probably aren’t in a position to have that conversation with all of your colleagues, and you certainly can’t have it with everyone in your management chain.

One step above just telling a colleague to stop is to drive them around like a coding agent. I wrote about this in AI makes weak engineers less harmful: if a colleague is simply pasting your messages into Claude Code and sending you the outputs, you can treat them like a high-latency Slack interface to Claude Code. It won’t be as good as a normal coding agent, but it’ll often be better than nothing.

Another strategy is to use AI to fight AI. This is a good one for handling workslop from managers. You can do this in two broad ways. First, instead of carefully reading it, paste it into an LLM of your own and ask for a short list of the salient points. Second, you can sometimes simply ask an LLM for an entire response. In a sense, this makes you part of the problem, so I can see why some people might be uncomfortable with it. But it’s more sustainable than spending ten minutes of your effort for every ten seconds of theirs.

You can also bias toward calls or in-person meetings. Workslop is just a special case of the general “your coworker is bad at communication” problem. One classic way of handling this that works even better on AI content is to say “hey, let’s schedule some time to chat about it”. This works for two reasons: first, your colleagues can’t give you AI content over a call, and second, forcing people to spend a chunk of their time talking to you (i.e. to make the effort symmetrical) is a good way to filter out predators.

Finally, you can sometimes simply ignore the workslop. This is particularly true for long status updates or pull requests from outside of your organization². You don’t have to respond to AI content as diligently as you would human content. You can match their lack of effort with your own: skim it, put off reading it until later (or never), and so on. If something’s really important, they’ll tell you in their own words.

Technically, not all cases of sending someone AI-generated content are workslop. If the effort is not asymmetrical — if the AI user has genuinely put a lot of their own time into the content — I don’t think it counts as slop, and you should just try and look past the AI style and treat it like a human message.
↩
Some messages — particularly reports directed at the entire organization — may not be intended to be read at all. Written artifacts can have many purposes beyond communication: evidence of effort, a reference document for later communications, a way to cover somebody’s ass by proving they considered point X, something that can tick a compliance or process box, and so on. I wrote a lot more about this in Seeing like a software company.
↩

You have to beat the models at something

Sun, 30 Aug 2026 00:00:00 GMT

In 2025, I wrote that software engineers ought to be assessed by “value over replacement”: not how much money they made for their company, but how much they would have made compared to the average engineer in their position. I’ve always found it vaguely silly when engineers put “built a product that made $X” on their resumes, when they just did the JIRA tickets that came across their desk.

Today, value over replacement is even more important. A replacement-level engineer in the 2010s was fine: maybe not worth promoting, but still worth paying, because writing code had a high fixed cost. Now writing code costs a hundred bucks a month. What are you doing that GPT-5.6-Sol or Claude Opus 5 wouldn’t do in your position? Why is it worth paying an extra two or three orders of magnitude for?

This is a scary thought. But you’re not doing yourself any favors by pretending that LLMs can’t actually write code and it’s all just a scam, or that LLM-written code is inherently so bad as to cause companies using it to collapse next year. We are not going to wake up in 2027 to find that the AI craze is over and everyone is writing code by hand again. You ought to put some serious thought into what you can do better than the models in the medium and long term.

Staying ahead of the models is a moving target. At the start of 2026, “make working changes to large codebases” was in this category, but now it’s not. For this reason, I doubt that you can retreat to some “hard engineering” area that requires deeper expertise. That might work in the short term, but not forever. If LLMs can find a better lower bound on the Riemann hypothesis, they will soon¹ be able to write solid high-performance kernel drivers or GPU shaders or whatever.

I think it’s more useful to look at the tasks models haven’t gotten better at over time, and the tasks that are hard for them get better at in principle. The two best examples of these are:

Deep familiarity with the codebase
Technical communication

Deep familiarity

What do frontier LLMs get wrong? What kind of coding mistakes do they make? It’s been a long time since I’ve seen a straight-up hallucination from a coding agent, or a simple logic error like an off-by-one. The mistakes they make tend to be errors of ignorance:

Not knowing that there’s a module in the codebase they could use instead of reimplementing some logic
Making the change in the wrong system because they didn’t know System X was the standard place for this functionality
Adopting a coding style that’s inconsistent with the company’s standard practice

Other times they’re errors of paranoia:

Implementing triply-redundant checks for a value that technically could be wrong but practically is set once from config and never updated
Assuming that ten milliseconds of stale data is unacceptable and designing a complex, unnecessary system to keep it always up to date
Building in fallbacks and “graceful” degradation into some code that ought to simply crash on error (e.g. a CLI tool, or a restartable k8s service)

What do these errors have in common? They’re the kind of errors a smart engineer might make if they had no context on the system: they’re competent enough to be able to solve the problem, but they haven’t been around long enough to confidently say “yes, we can take this risk to avoid an extra three thousand lines of code”. Until someone cracks continuous learning or truly massive context windows, this is just an inherent feature of how AI agents operate. If you can catch these errors, you’ll be providing real value.

The only way to catch these errors is to be familiar with the codebase and familiar with the system in general. For much more on this, see my post You can’t design software you don’t work on. But there’s also a psychological component to it. You have to be willing to confidently disagree with the agent.

AI agents can be very convincing. Often they can get “stuck” on some error above where they’re not willing to take a particular risk, so they keep going back and sneaking in code to cover that case (or writing persuasive arguments about why that case is important). To add value, you need to be willing to say “this sucks, I don’t think we need X and Y at all, why can’t we do Z in a much simpler way?” It takes courage.

You can’t rely on other AI agents to review each other’s work. If you use the same model, it’ll reliably make the exact same assumptions and mistakes. But even if you use different models, they’ll also tend towards the same kinds of mistakes — ignorance and paranoia — for the same structural reasons. AI-driven review loops are in fact more likely to get these things wrong, because modern AIs have been RL-ed to try to find a few nitpicks no matter what. Having a critic AI and a worker AI bounce off each other is a really good way to end up with ten thousand lines of paranoid slop.

Technical communication

Another area where you can add value on top of AI is communication. Newer models are better at coding, but are paradoxically getting worse at writing. GPT-3.5 and GPT-4 had a human-like writing style at times. GPT-4o introduced the modern slop idiolect, and the newer Anthropic models speak “Claudish”: a bizarre semi-baroque semi-truncated way of communicating that nobody enjoys. There have been a few bright spots — GPT-4.5 was okay, and I quite liked o3² — but in general LLMs are not good at this. Here’s two reasons why.

First, good writing is not a verifiable domain. If you want a model to get good at mathematics or coding, you can generate problems for it and automatically grade them. You can’t grade good writing. If you try to get humans to grade it — for instance, via the early OpenAI RLHF attempts — you get the kind of writing that sounds impressive to the average person when consumed in single-paragraph form. This is the origin of the “stick three hundred writing devices into every sentence” style. I think it’d be possible in principle to hand-pick some people with good taste and have them do it, but there are some obvious problems³ that prevent this from happening.

Second, the labs have been monomaniacally focused on capability instead of communication. When you’re trying to train a model that can break new scientific ground or replace a software engineer, you might trade off some communication ability. In fact, I think we can identify exactly how this has been happening. If you look at internal model reasoning tokens, they tend to have strange word choices and oddly truncated grammar:

RESOLUTION: charge the current-leg’s OWN saved-prefix occupancy EAGERLY: when leg i saves e1..et: ALSO commit their occupancy AT LEG i

If you were to translate this into proper English, you would probably end up with something that reads like Claudish:

Charge the current-leg’s saved-prefix occupancy on a clean, eager path: when leg i saves e1..et, commit the occupancy at leg i.

I suspect that the weirdly alien writing style of some LLMs is because you’re reading a semi-literal translation of that model’s internal chain-of-thought, which has become nearly incomprehensible in pursuit of better problem-solving abilities. It is surprisingly hard to translate Claudish to good English: not only do you need to follow the convoluted, compressed language of the original, but you need the technical ability to understand the problem the model is solving.

Because of all this, technical communication may be a surprisingly durable skill. In Peter Watts’ novel Blindsight, the world is full of cognitively augmented humans. The main character is a “synthesist”: someone whose job is to be a translation layer between these geniuses (who speak in abbreviations and gestures) and everyone else. Watts’ idea is that communication ability may be largely independent from — or even negatively correlated with — intelligence. A “country of geniuses” may still need a bunch of ordinary smart people to translate their insights for everyone else.

If you’re trying to communicate to humans, there are also huge advantages to having a human write the content. Many of us are becoming AI-blind: developing an instinctive reflex that stops us reading when we encounter AI-generated content. It’s like the reflex that allows people to ignore flashing billboards or sidebar advertisements on websites. If you circulate some planned technical strategy as an AI-written document, most of your colleagues will have to physically force themselves to read it word-by-word.

Conclusion

Whatever you do, don’t be a meat proxy: someone who simply copies requests into an AI agent and submits their output as your own work product. Doing that is just begging to be fired, since you’re definitionally not adding any value yourself. Even if you have a cunning system of multiple agents — the so-called “software factory” — you’re still on dangerous ground. When the features of your system work their way into enterprise AI tooling (and they will), you’ll be disposable.

You need to find some way to leverage your expertise to do what the models can’t. Simply not using AI at all is better than being a meat proxy, since you’ll probably do some things better than the model would have, but it’s far better to figure out what AI can do and position yourself to fill those gaps. Right now, there are two main gaps: familiarity with the technical details of the system, and the ability to clearly and persuasively write about those details.

edit: this post got some comments on Hacker News.

If you’re thinking “but LLMs can do these things now!”, substitute your preferred example of high-difficulty software engineering.
↩
Although this was probably a “thank God it doesn’t speak like 4o” reaction.
↩
Defining good taste is hard, there’s no guarantee that AI lab researchers have good taste to start with, nobody will agree on examples, the bulk of users might not even like it, you won’t be able to get enough people to produce the volume of data you need, and so on.
↩

Selling out

Fri, 28 Aug 2026 00:00:00 GMT

In 1973, Tom Lehrer famously sang that “selling out is easy to do”. That may have been true in the seventies, but it’s not true today: selling out requires both technical skill and a careful sense of how large organizations work in practice. One of the goals of this blog is to teach people how to do it.

If you want to live with uncompromised integrity, you don’t need anyone to tell you how: simply always do exactly and only what you want to do. You will be repeatedly punished for it — your managers will dislike you, you will lose jobs and lose out on the chance of getting hired, your financial situation will be less stable, and so on — but that’s just part of the deal. Like deadlifting four plates, it’s not easy, but it is straightforward.

It’s more complicated to sell out a little bit. Sellouts like me walk a fine psychological line: figuring out how to work a large organization on one hand, and maintaining some kind of independent inner life on the other. But is this safe? Does role-playing as a professional inflict some kind of psychic damage?

Marxist alienation

People often tell me that acting professional is dangerous because it’s “Marxist alienation”. I suspect many people have a vague sense that “alienation about your job” is in some sense necessarily Marxist, but the actual concept is more specific. In Marx’s First Manuscript he describes it like this:

The worker becomes poorer the more wealth he produces, the more his production increases in power and extent. … This fact simply means that the object that labor produces, its product, stands opposed to it as something alien, as a power independent of the producer.

Marx’s theory here goes something like this¹: when you work for a capitalist boss, they make ten dollars for every dollar you make². The more you work, the more powerful you make your boss with respect to you³. Your work is thus producing a force that is your enemy: a literally “alien” power. For Marx, alienation is proportional to the amount you’re working:

the more the worker produces, the less he has to consume; the more value he creates, the more worthless he becomes; the more his product is shaped, the more misshapen the worker; the more civilized his object, the more barbarous the worker; the more powerful the work, the more powerless the worker; the more intelligent the work, the duller the worker and the more he becomes a slave of nature.

So far Marx is just talking about the worker’s alienation from his work. But he does touch on the internal psychological effects as well. Since work is such a big part of life, to be estranged from your work is to be in some sense estranged from your self:

[labor] does not belong to [the worker’s] essential being; that he, therefore, does not confirm himself in his work, but denies himself, feels miserable and not happy, does not develop free mental and physical energy, but mortifies his flesh and ruins his mind. Hence, the worker feels himself only when he is not working; when he is working, he does not feel himself.

Overall, I can make out four distinct senses⁴ of Marxist alienation:

Alienation is when your work empowers your employer more than you, thus making you more disposable over time
Alienation is when your work separates you from the physical act of craftsmanship (and thus of the product you’re creating)
Alienation is when your work physically and mentally harms you: making you injured, deformed, and so on
Alienation is when you do not psychologically “confirm yourself” in your work, because you’re working on other people’s goals instead of your own

I don’t think the first one is relevant to big tech software engineers. The idea here is that the harder you work, the more you (relatively) disempower yourself, so you’d be better off coasting and putting your company in a worse financial position. But this just seems straightforwardly wrong: as a software engineer, you want your company to be doing as well as possible! The more powerful and rich your company is, the better your position will be⁵. A company that is struggling is more likely to treat you badly, lay you off, cut your benefits, and so on.

The second one is more relevant (particularly in large companies), but there’s a missing story here about why that separation is bad. I also don’t think the third sense of alienation applies to me: Marx is talking about the physical toll of factory work or other hard physical labor, which doesn’t apply to software engineering⁶.

The fourth sense of alienation — that your effort is being directed at other people’s goals — is the most straightforwardly relevant to my own experience. But I don’t think Marx has a great psychological account of why that happens or what it feels like. To go deeper into this psychological aspect of alienation, we need to look at some later Marxists.

Situationist alienation

In 1967, Guy Debord and Raoul Vaneigem both published their masterworks. They were members of the “Situationist International”: an influential group of Marxist artists and political theorists⁷. Both of them have a lot to say about alienation. Debord writes:

The worker does not produce himself; he produces an independent power. The success of this production, its abundance, returns to the producer as an abundance of dispossession. All the time and space of his world become foreign to him with the accumulation of his alienated products.

This idea — of the worker’s efforts not going to his own ends, but to some alien power — is straight out of Marx’s First Manuscript. But note how Debord is already talking in terms of “representation”. Elsewhere he writes:

the more he contemplates the less he lives; the more he accepts recognizing himself in the dominant images of need, the less he understands his own existence and his own desires.

The Situationists are all about representation. It wouldn’t be too far wrong to think of them as a bunch of Marxists who saw the rise of advertising in the 1950s and 1960s and lost their minds. TV advertising is Capital itself made real, dressed in multicolored lights, projected into everyone’s homes. So when they talk about alienation, they talk about it in terms of representation. Vaneigem explicitly calls it roleplaying:

The roles we play in everyday life, on the other hand, soak into the individual, preventing him from being what he really is and what he really wants to be. They are nuclei of alienation embedded in the flesh of direct experience.

Alienation means giving up your real, authentic self in order to play some “role” (for instance, the role of “effective staff engineer”). It’s a vicious cycle, because the more you lean into the role, the more your authentic life becomes trivial, which in turn makes the role more appealing:

Life is sacrificed, and the loss compensated by means of accomplished prestidigitation in the realm of appearances. The more daily life is thus impoverished, the greater the attraction of inauthenticity, and vice versa. Dislodged from its essential place by the bombardment of prohibitions, limitations and lies, lived reality comes to seem so trivial that appearances become the centre of our attention, until roles completely obscure the importance of our own lives.

Ultimately, Vaneigem describes alienation as an addiction:

This ambiguity accounts to my mind for people’s addiction to roles. It explains why roles stick to our skin, why we give up our lives for them. They impoverish real experience but they also protect this experience from becoming conscious of its impoverishment. Indeed, so brutal a revelation would probably be too much for an isolated individual to take.

This is almost a straightforward preview of the 90s idea of “selling out”: people are born wild and free, but when they watch too much TV they give up on their dreams for a paycheck and become big fat phonies. According to this view, it’s always better to not sell out. It may leave you poor and without prospects, but at least you won’t be unoriginal⁸. What shall it profit a man, if he gains the whole world but loses his soul?

I’m sure this is an accurate description of some people’s hangups about work. But I don’t think even the Situationists would agree that any advice on how to play a role well — i.e. the advice I give throughout my blog — is inherently harmful. The problem with role-playing is that you risk losing your authentic inner life, but that’s a risk, not an inevitable consequence. Vaneigem has a wonderful quote on this:

Nobody is ever completely swallowed up by a role. Even turned on its head, the will to live retains a potential for violence always capable of carrying the individual away from the path laid down for him. One fine morning, the faithful lackey, who has hitherto identified completely with his master, leaps on his oppressor and slits his throat.

This is a little overdramatic, but it seems straightforwardly true. It sounds very impressive to talk about alienation (or like Marx, to derive alienation from the basic economic conditions of work), but, you know, you can just not do it, right? Healthy people can present different selves for different situations: you can be loose at a party, respectable at church, solemn at a funeral, professional at work, and so on. I’m not convinced this is alienating or impoverishing. In fact, it’s not just humans: my dogs act differently around different people too, and I don’t think that makes them inauthentic.

If maintaining a professional identity is harmful, it has to be because of something fundamental about work that makes it different from regular role-playing. Could it just be that acting professional at work is too inauthentic? Some amount of role-playing might be okay, but if the role you’re playing is completely alien, does it become lying to yourself?

Sartre and De Beauvoir’s bad faith

Another common term for alienation is “bad faith”. The original version of this comes from Jean-Paul Sartre’s famous book Being and Nothingness⁹, where he describes bad faith as telling yourself lies. Lying to yourself is weird because you’re both deceiver and deceived:

Bad faith then has in appearance the structure of falsehood. Only what changes everything is the fact that in bad faith it is from myself that I am hiding the truth.

Here’s another quote from Sartre’s partner Simone de Beauvoir, who writes more explicitly about professional bad faith (what she calls the “serious man”):

The serious man’s dishonesty issues from his being obliged ceaselessly to renew the denial of this freedom. He chooses to live in an infantile world, but to the child the values are really given. The serious man must mask the movement by which he gives them to himself, like the mythomaniac who while reading a love-letter pretends to forget that she has sent it to herself.

The key idea here is that professional bad faith — alienation — comes from losing yourself in the role. Instead of just playing at being a professional, you decide that you’re going to take it fully “seriously”, and treat professional values as if they’re absolute. But of course that’s not coherent: you can’t decide to treat a value as absolute, it either is or isn’t. So this requires a kind of continual self-deception where you pretend there’s no decision to be made at all. I have the same attitude to this as I do to Debord and Vaneigem: sure, some people might make this mistake, but you don’t have to.

I can see why it might be tempting to treat work as the ultimate source of value: it gives you food and shelter, it (for Marxist reasons) can feel like a powerful alien force, companies constantly propagandize their employees, and so on. But work is just a collection of people following incentives until they reach some kind of stable equilibrium. From the inside, this equilibrium can look like the structure of the universe. However, if you treat it like that, you’re going to be bitterly disappointed when it turns out to be as arbitrary and pointless as other stable equilibria.

American sociology and the “standardized loser”

I’ve considered a bunch of theories on which work — even well-paid knowledge work — might be inherently alienating:

Capital is intrinsically an alien force
Workers can become addicted to role-playing
Treating work as an ultimate source of value is self-deception

I don’t buy any of these (or at least, I don’t buy that they show that acting professional is inherently bad). But there are lots of types of jobs in the world, and some of them definitely seem more alienating than others. The best books I’ve read on these are C. Wright Mills’ White Collar or Erving Goffman’s The Presentation of the Self in Everyday Life¹⁰.

The American sociologists are in the same conversation as Marx, the Situationists, Sartre and de Beauvoir¹¹. Mills writes about the median “salaried employee”:

In the case of the white-collar man, the alienation of the wage-worker from the products of his work is carried one step nearer to its Kafka-like completion. The salaried employee does not make anything, although he may handle much that he greatly desires but cannot have. No product of craftsmanship can be his to contemplate with pleasure as it is being created and after it is made. Being alienated from any product of his labor, and going year after year through the same paper routine, he turns his leisure all the more frenziedly to the ersatz diversion that is sold him, and partakes of the synthetic excitement that neither eases nor releases. He is bored at work and restless at play, and this terrible alternation wears him out.

In this, we see both Marxist alienation, where the employee is physically separated from his craft, and Situationist alienation, where the employee is absorbed into the world of the “spectacle”. For Mills, this was television, but today it’d probably be scrolling short-form video on your phone. Mills goes on to describe how white-collar work can be humiliating and dehumanizing:

In his work he often clashes with customer and superior, and must almost always be the standardized loser: he must smile and be personable, standing behind the counter, or waiting in the outer office… self-alienation is thus an accompaniment of his alienated labor.

And:

Here are the new little Machiavellians, practicing their personable crafts for hire and for the profit of others, according to rules laid down by those above them.

This is a great articulation of something I write about a lot: that your primary job is to make your bosses happy, and that you ought to shape both your emotions and values towards this goal. When necessary, you must be the “standardized loser”, happy to have your professional plans disrupted. You must be a “little Machiavellian”, quietly laying the groundwork for your personal goals.

I think here we come to the strongest version of “alienation”. Playing the professional is bad because “professional software engineer” is an inherently subservient role. Why can’t you just maintain a healthy mental distance (as the Situationists suggest)? Because humiliation takes its toll, whether you’re role-playing it or not. As an analogy, suppose you’re an actor in a film where you get slapped in the face. There’s a sense in which you’re not really getting slapped — everyone’s just acting — but you’re still being physically hit with each take, which will leave bruises over time.

The spectrum of compromise

There’s a tradeoff here that everyone has to make in their own way. At the one end is becoming fully alienated, like de Beauvoir’s “serious man”, and at the other end is leaping up to slit your master’s throat, as Vaneigem describes. Here’s some examples of points on that spectrum:

I am making the world a better place by achieving my OKRs. My yearly review cycle tells me how good of a person I am. I must work hard to ensure my company succeeds.

I have my own goals and values, but while I’m at work it’s in my best interests to be as professional as possible. My yearly review cycle tells me how effective I’ve been at pulling the strings in the organization. I must work hard to make money.

I have my own goals and values, which I try to accomplish as best I can in the context of work. I often argue with my managers about their priorities. It’s my responsibility to push my company towards my set of values, which is hard work and often unrewarded.

My workplace is an enemy I fight every day in order to get what I want done. I routinely ignore my manager’s priorities and do the things I think are important. My yearly review cycle tells me how much of a sellout I am: I wear bad reviews as a badge of honor.

I agree with my sources above that the first point is a big mistake, if for no other reason than that companies don’t follow their own stated values. If you prefer the fourth — the path of pure integrity — fair enough. You get to decide what tradeoffs to make with your own life. I land on the second point here (occasionally the third). I suspect it’s also a mistake to purely occupy the second point. Having some values that you occasionally trade off against your company’s values prevents you from slipping into the first point.

Conclusion

I don’t bring my whole self to work. My coworkers and bosses see the most professional version of me: friendly, cooperative, largely apolitical, patient, and so on. I try and save the edgier, more opinionated version of myself for my friends and family in real life. Readers of this blog get something in the middle.

When I write about being a software engineer, I give a lot of advice about mindset. I don’t just say what you should do, but what you should think and feel. In effect, I’m advising my readers to adjust their personalities into a more professional mold. This definitely works. If you turn yourself into the kind of person who’s successful in big tech companies, you will probably be more successful in big tech companies. Is this safe?

I think so, as long as you’re adjusting your work personality, not your authentic internal self. The well-known accounts of why it’s dangerous either assume that your work takes a physical toll (like Marx’s factory workers), or that you’re unable to maintain any mental distance between your personal and professional selves (like de Beauvoir’s “serious man”). Debord and Vaneigem say that role-playing can be dangerous, but if you’re able to avoid treating it like an addiction you’ll be okay.

Another danger is that being a professional is often humiliating, and humiliation does mental damage over time. Still, there are jobs that are way worse in this respect than “software engineer”. If you approach the job right — and you’re good at it — technical ability gives you quite a lot of power. Power is an antidote to humiliation.

Everyone gets to make their own deal with Mammon. You can decide exactly how much you want to compromise in exchange for wealth and career success. I don’t know if this is the best way to organize the world, but it’s the way the world is organized right now. Given that, I’m comfortable with my blog serving as a guide for how to get the most bang for your buck. If trading your integrity for wealth and power is sad, it’s even more of a tragedy to throw it away for nothing.

It’s always terrifying discussing Marx: like Hegel or Kant, he’s one of the most studied philosophers in the world, so there’s no way to reference him without getting a bunch of things wrong. Sorry in advance to any Marxist scholars!
↩
Because they own the machines, or the factory, or the datacenter, or whatever gives your work high leverage.
↩
Marx also writes a lot about work making the worker sick, deformed and misshapen: I think here he’s talking about the physical toll of factory work or other hard physical labor, which doesn’t really apply to software engineering, so I’m going to focus on the relative-empowerment stuff.
↩
Marx’s own taxonomy has four different senses: he collapses 2 and 3 into a single sense, then adds a third mysterious sense in which workers are estranged from their “species-being” and thus from each other.
↩
Plausibly Marx is talking about labor and capital in general: if every worker decided to start half-assing it, maybe companies would be less capable in general, and the balance of power would tilt more towards labor.
↩
That may change! If it turns out that LLMs really do have negative cognitive effects, we could be in the same boat as factory workers. I wrote about this in Software engineering may not be a lifetime career.
↩
Vaneigem resigned from the group in 1971, was fiercely denounced by Debord, and the whole group disbanded in 1972.
↩
True or not, my sense is that Millennials and later generations think that worrying about “selling out” is a sign that you had it too good. Consider the endless anecdotes of Boomers and GenX-ers living in a van surfing until their mid-thirties and then walking into a good office job because they had a firm handshake: they could afford to be authentic because they didn’t have to hustle.
↩
I only read the Marx, Debord, and Vaneigem texts while researching this blog post, but I did in fact read Sartre and de Beauvoir for my philosophy degree, so hopefully I do a better job.
↩
And Robert Jackall’s Moral Mazes, which informs much of my writing, and about which I owe a long-form blog post one of these days.
↩
There’s a good Goffman quote: “A status, a position, a social place is not a material thing, to be possessed and then displayed; it is a pattern of appropriate conduct, coherent, embellished, and well articulated.” Goffman goes on to explicitly link Sartrean “bad faith” to this kind of professional role-playing. (For Goffman, it’s roles all the way down.)
↩

You should never be angry at work

Sat, 22 Aug 2026 00:00:00 GMT

I try not to give a lot of prescriptive advice about working in tech companies¹. There are many ways to be successful, and every company works differently. If you’re shipping projects and your management chain is happy, it doesn’t really matter how you’ve accomplished it. However, there’s one thing that I do think is solid advice: you should never be angry at work.

Anger in the workplace

Anger in the workplace is toxic. An angry colleague immediately becomes a new problem to be managed, not a professional helping you manage problems. When someone is visibly angry in a meeting or in Slack, it kills the entire atmosphere: other engineers will often go quiet entirely, not wanting to make the situation worse.

If you routinely “get heated” at work, the best-case scenario is that you’re part of a tight-knit team of confident people who aren’t put off by it². No harm, no foul. But the second someone comes onto your team who’s not so confident, or you have to communicate outside of your team, it becomes a big problem. It’s a bad way to treat your colleagues — even undirected or casual anger intimidates and alienates the people around you³ — and it makes you a less effective engineer.

Healthy workplaces route around anger in the same way that networks route around damage. Emotionally unreliable engineers will get left out of conversations that might cause them to blow up. Decision-making will get done around them in backchannels. I’ve seen this become a self-reinforcing cycle: angry engineers aren’t consulted on key decisions, which makes them angrier, which pushes them even further away from the spaces where decisions get made, and so on.

You can often find these engineers bitterly complaining that they keep the company together, but nobody ever listens to them. In my experience⁴, this is almost never true. Engineers who are highly effective tend to get listened to — at minimum by their colleagues, and eventually by managers and product managers who want to extract as much value as possible from them. (One reason this is true is that all successful projects involve working with other people, and if nobody listens to you, you can’t do that.)

Caring

Why do angry engineers believe they’re important? Paradoxically, anger can be really useful to a software engineer. Angry engineers are rarely the ones holding the company together, but they’re also rarely useless.

One surprising thing about working for big tech companies is that some engineers are not just unproductive, but actively net-negative: either because they’re incapable of doing useful work on their own, or because they’re sloppy enough that they create more work than they do, or because they’re so checked out that they literally do nothing. Angry engineers might be net-negative in a cultural sense, but in terms of literally solving tickets and shipping features, they’re usually well above average.

Why is this? Anger often comes from caring about your work, and caring a lot is sufficient to make you a competent engineer. I’ve never worked with someone who genuinely cared about their work who wasn’t (or didn’t eventually become) competent. I actually think it’s healthy for an early-career engineer to sometimes get angry about their work, because it means they care a lot: it’s still a mistake in the moment, but it’s a “good mistake”.

I certainly used to get angry — in fact, I wrote about the angriest I’ve ever been at work here⁵. But you have to move past it.

Moving past anger

Think of “caring about your work” as a vertical tube, unsealed at either end. You fill the tube by pumping in emotional investment from the bottom⁶. If you have too little, it drains away and you end up as a useless coaster. But if you have too much, it overflows and you end up as an angry engineer that people have to work around.

One solution is to try and care the exact right amount: be invested in work a bit, but also have hobbies and a family and whatever else gives you perspective about your work problems. If you have a rich and healthy personal life, it’s hard to find yourself yelling at somebody about React state management. However, this is a tricky balance to maintain over time.

Another solution is to care about different things. The reason too much caring overflows into anger is because what you care about is misaligned with what the organization cares about. If your interests are perfectly aligned with your company’s (for instance, if you primarily care about delivering shareholder value), you can fit way more emotional investment into the tube before it overflows.

A little bit of “professional anger”

Here’s some dangerous advice: showing a little bit of anger at work can sometimes be useful. It can be a good way to signal that you care, or to build rapport with certain people, or to draw attention to something you think is important. However, it’s still always a mistake to be angry. You need to be able to drop back to a friendly mode at will, which is very difficult when you’re genuinely angry.

Being able to show a full range of emotion at work is good. It makes you more persuasive and more human. Being a fully professional robot is fine — you can have a successful career this way — but there’s always going to be some kind of uncanny-valley HR-ness to your work persona that will make it hard to connect with your colleagues.

If in doubt, don’t show anger. It’s never wrong to be professional. However, if you can signal that you’ve got enough distance to separate your professional feelings from your real feelings, and enough perspective to realize that the stakes of a technical decision are fundamentally not that high in the grand scheme of things, it can sometimes be okay to show visible frustration so that people know you’re still human.

Angry role models

Well-known software engineering personalities are often angry. It feels unfair to give too many negative examples, but obviously Linus Torvalds’ rants about Linux are a great example. Some of my favourite engineering talks are from Bryan Cantrill, who is sometimes visibly furious at his subject matter. There are too many well-known angry blog posts to list, but I’ll cite one I genuinely like: my Australian blogging colleague Nikhil’s post titled I Will Fucking Piledrive You If You Mention AI Again.

Anger is a part of the general image of a competent software engineer. Many junior engineers learn from this that it’s okay to be angry. However, taking your emotional cues from engineering celebrities is a big mistake, for a few reasons.

First, you are not Linus Torvalds or Bryan Cantrill. Torvalds is the BDFL of the most important software system in the world. Cantrill is the cofounder and CTO of his company. When these people are angry at work, people will not work around them, because they are the ones deciding what gets worked on. Once you’re the one in charge, you can get away with being emotional in the workplace⁷.

Second, you don’t know what it’s like to work with these engineers. People give talks and write blog posts because they’re emotionally worked up about something. If your only exposure to a celebrity is via their conference talks and blog posts, you’re seeing them at something like their maximum emotional intensity. If you then take that level of emotion into your normal everyday work, you’re almost certainly overshooting.

Anger is a local maximum

I’ve been reorged into dysfunctional teams, have had projects I enjoyed cancelled, and have worked on systems that were extremely chaotic. I can’t remember the last time I was actually angry at work. To be clear, I’m not successfully hiding my anger (unless it’s so repressed it’s invisible to me as well)⁸. Nor am I naturally a chill person. I’ve just reached a point in my career where I genuinely don’t get upset about work stuff.

A cynical person might say here that I’ve stopped caring about my work, so of course I don’t get angry anymore. I’ve left the side of the “real engineers” — the Linus Torvalds and Bryan Cantrills of the world — and sold out for that sweet, sweet big tech money. I mean, maybe! It’s true that I’m less invested in specific technical decisions than I used to be. But I still care a lot about doing a good job, I still spend a lot of time tweaking and reading code, and I certainly get more done than I did when I was more emotionally volatile.

Being angry at work feels good. It feels like proof that you’re working on something that matters, and that you’re personally having an impact. If you’re angry, nobody can call you a coaster. But anger is only a local maximum. If you can find your way to a different style of working, you’ll not only be more effective, but you’ll be in a far better place to have impact on problems that actually matter.

edit: this post got some good comments on Lobste.rs. One commenter argues that being angry at problems is better than being angry at people: I agree, but (a) if you’re in the habit of anger, it can be hard to keep it only directed at problems, and (b) when you express anger, the people around you are often impacted by it whether it’s directed at them or not. Other commenters recommend keeping a sense of perspective. Someone points out that it’s good to be angry at injustice. This is true: if your company is engaging in misconduct, you ought ot be angry about it. Finally, some commenters agree that anger can be a really effective tool for getting things done in the short-term.

It was also posted on Hacker News. The comments there aren’t as good, though I do recommend this one, which argues that if you don’t feel anger at work, you’re the victim of Marxist alienation. In fact, I agree! A long post is coming one of these days about how I learned to stop worrying and love Marxist alienation.

Mostly, I fail.
↩
As I understand it, this is the work environment that most famously angry engineers came up in.
↩
The post could probably stop here — this is a good enough reason to not be angry in the workplace — but I do think there’s an interesting case to make about why being angry is also bad for you.
↩
My experience is certainly limited (a handful of companies, and maybe ten different teams or organizations). I can certainly believe it happens!
↩
About halfway down, in the section titled “it’s not your manager’s fault”.
↩
Emotional investment is a liquid with the viscosity of water.
↩
To a point. Even Torvalds famously said he’d gone too far with the anger and decided to turn it down a bit.
↩
I suppose I’m not the best person to judge whether this is true. If you work with me and I do come across as an angry guy, please do tell me.
↩

Readers can't identify watermarked AI text

Fri, 21 Aug 2026 00:00:00 GMT

In the last few weeks, I’ve been complaining that everyone is wrong about AI watermarking: it isn’t really anti-consumer and it doesn’t make the outputs any worse. The watermarking papers demonstrate¹ that this is true, but I thought it might be interesting to put it to a practical test. Given examples of watermarked and unwatermarked answers to the same prompt, could readers tell which is which?

To find out, I vibed up² https://sgoedecke.github.io/watermark-quiz/, a static site that quizzes readers. I used Qwen3-30B-A3B-Instruct-2507 on a rented H200 to generate thirty responses: three responses per question, one of which was secretly watermarked with SynthID-Text. The rented GPU cost around two dollars. To measure results, I just sent users to a different page for each score, and aggregated visitors-per-page in my analytics³. This would be easily spoofable if anyone cared enough to do so, but for a casual test I think it’s acceptable.

The first round of traffic I got to the quiz (278 participants) had these slightly puzzling results:

Score	Participants
0	6
1	15
2	36
3	64
4	54
5	39
6	51
7	10
8	3
9	0
10	0

Pure random choice would lead to an average score of 3.33/10. However, the mean score here is 3.92. There is indeed a spike around 3/10, as expected, but there’s also a second weird spike at 6/10. Why is that? It turned out that the SynthID response was option A in six of the ten questions, so users who just selected the first answer for every question would get 6/10. Oops.

I re-shuffled the questions and got these results:

Score	Participants
0	1
1	3
2	14
3	21
4	20
5	11
6	2
7	1
8	0
9	0
10	0

Now the mean is 3.4/10, much closer to the expected 3.333. There’s no spike around 6. We only had 73 people take the quiz after I shuffled the questions — most people saw it and took it immediately after I posted it to my LinkedIn and Hacker News — but given the previous results, I think that’s still enough to feel confident that people were just guessing randomly.

So no, people can’t identify the presence of AI watermarks. Obviously this wasn’t exactly a scientific study, but it’s still pretty suggestive. If watermarks were really choosing random words that the model would never pick, you’d be able to sometimes tell from three side-by-side responses which one went down the weird watermarked road, right? I also hope that something like this can serve as a persuasive tool: if you’re worrying about what impact watermarking is going to have, and your intuition is unmoved by the mathematical explanations, having a read of the watermarked and unwatermarked responses might convince you that there’s really no difference in quality.

edit: this quiz got some Hacker News comments here. I’m amused by the commenters who got 7 or 8 and claim to have worked it out. Other commenters wish they didn’t have to answer all ten before getting the result: fair enough, but that way I’d get much worse signal. Another commenter correctly identifies that feeding the model the same prompt with a high temp will cause the first few tokens to be identical, but incorrectly thinks it’s some kind of red-herring trick. And if you’re curious, the quiz now has around 4,700 responses, with the mean result sitting at 3.44/10. I think the volume of people who just selected “A” for everything (which would give you a 4) has shifted the mean slightly above 3.33.

The one-sentence explanation for why is that AI models already randomly select from a handful of top tokens, and watermarking just replaces that random choice with a bias that is predictable while still being equivalently “random”: as a simple example, instead of “pick randomly from the top three tokens”, you could do “count the letters in the previous ten tokens, take mod three, then pick that token”.
↩
Some notes from the vibing: GPT-5.6-Sol put extraneous text all over the page I had to get it to remove, it chose the now-very-recognizable styling that I had to rip out, and it built some kind of weird Javascript-driven static site instead of just the cross-linked pure HTML thing I would have built by hand. It took me about an hour (although I did maybe ten minutes of actual work).
↩
Umami, hosted on PikaPods. For my blog, I do also pay for Netlify analytics because I find JS-based analytics misses >50% of technical users, but for stuff like this Umami is fine.
↩

Good writing is obvious, not original

Wed, 19 Aug 2026 00:00:00 GMT

When you write, you should try to say things that are obviously true, and spend very little time worrying about whether you’re being original. Ironically, this is the best way to do truly original writing.

Every important idea has been discussed already. I learned this in grad school for philosophy¹, but it’s true across the board. Almost every field² has generations of very smart people who’ve spent their whole lives thinking about the biggest problems. If you try to only write about ideas that are truly brand-new, you will restrict yourself to writing about ideas that weren’t important enough to be considered before: in other words, trivia and ephemera.

What happens if you focus on the obvious instead? Writing about things that are obviously true is surprisingly difficult. Writing about anything is difficult, because reality has a surprising amount of detail. Still, writing about obvious things is especially difficult, because they’re almost too obvious to see.

It’s trivial for me to come up with non-obvious things I believe (for instance, I don’t like database foreign keys), but it’s hard for me to articulate the parts of what I believe that are truly fundamental. I think it’s the same reason why I can’t see the glasses on my face: they’re not something I look at, they’re something I look with. The beliefs you consider the most obvious often play a role like this. Writing about them can be really worthwhile, because reading about your basic assumptions can help other people articulate and examine their own.

My most popular blog post — How I ship projects at big tech companies — is a nice example. I knew I had strong opinions on how to ship projects (if for no other reason than I kept watching other people doing it wrong), but it took hours of sitting down and writing to realise that I actually just had a different definition of “shipping”³, and that all of my object-level advice flowed naturally from that. That definition was so obvious I struggled to notice it at all.

The other reason to try and be obvious is that trying to be original is really boring. Here’s a quote from Keith Johnstone’s Impro, a book I consider far too cult-like but that does have some great advice about originality:

If someone says ‘What’s for supper?’ a bad improviser will desperately try to think up something original. Whatever he says he’ll be too slow. He’ll finally drag up some idea like ‘fried mermaid’. If he’d just said ‘fish’ the audience would have been delighted. No two people are exactly alike, and the more obvious an improviser is, the more himself he appears. If he wants to impress us with his originality, then he’ll search out ideas that are actually commoner and less interesting. I gave up asking London audiences to suggest where scenes should take place. Some idiot would always shout out either ‘Leicester Square public lavatories’ or ‘outside Buckingham Palace’ (never ‘inside Buckingham Palace’). People trying to be original always arrive at the same boring old answers. Ask people to give you an original idea and see the chaos it throws them into. If they said the first thing that came into their head, there’d be no problem.

I think this is absolutely correct. When you’re trying to be original, you’re striking out into nothingness, and you’ll likely end up pattern-matching to “something that sounds kind of transgressive”, which is as boring as it sounds. When you’re trying to say something obvious, you’re reflecting on your own experiences, which have the interesting grit and detail of reality, and are far more likely to sound original to your readers.

If you think you’re the first to have a philosophical idea, you’re probably not even in the first few hundred to write about it.
↩
Of course, if you’re in a brand new field of study, it’s easy to make a lot of original observations very quickly. I call this the “gentleman scientist” era of a field. It’s good to try and be original in these limited circumstances, because there’s such a wealth of ideas that nobody has tried yet.
↩
In brief, I think “shipping” is socially defined: something is shipped when the relevant decision-makers at a company agree that it is shipped.
↩

Help peer

Tue, 18 Aug 2026 00:00:00 GMT

One of the most influential 20th century pieces of writing about AI is Isaac Asimov’s The Last Question. Although there are many humans in the story, the protagonist is the computer Multivac, who evolves over the course of ten trillion years from a single datacenter to a universe-spanning mind in hyperspace. Multivac (now called “AC”) ends the story like this:

The consciousness of AC encompassed all of what had once been a Universe and brooded over what was now Chaos. Step by step, it must be done. And AC said, “LET THERE BE LIGHT!” And there was light —

Many things about this story are prescient. In particular, I like the idea that humans would interact with powerful artificial intelligences by drunkenly posing them riddles or using them as children’s toys. But the enduring idea from this story is that if you build a big enough computer, it will become God.

Moloch

One of the most influential 21st century pieces of writing for AI researchers is Scott Alexander’s Meditations on Moloch¹. Scott describes the story of human existence as a series of “multipolar traps”. These are prisoner’s dilemma situations where cooperation would make everyone better off, but since each individual is incentivized to defect, everyone ends up “racing to the bottom”, which is bad for everyone². For rhetorical effect, Scott personifies this dynamic as “Moloch”, the ancient Canaanite god famous for child sacrifice:

[Moloch] always and everywhere offers the same deal: throw what you love most into the flames, and I can grant you power.

What does any of this have to do with AI? Well, in the long run, the only way out of a multipolar trap is to become unipolar³. Ideal dictatorships don’t have a problem with defectors⁴, because they can simply enforce a state of cooperation with violence. Scott is uncomfortable with this idea, though I worry it’s mainly because he thinks it won’t work:

As foreigners compete with you – and there’s no wall high enough to block all competition – you have a couple of choices. You can get outcompeted and destroyed. You can join in the race to the bottom. Or you can invest more and more civilizational resources into building your wall – whatever that is in a non-metaphorical way – and protecting yourself.

A dictatorship that enforces cooperation will not be as strong as its peer societies who are purely maximizing for wealth and power. It’s Moloch again, but at the level of countries and governments: once a few neighboring countries defect, your walled-garden dictatorship will be torn apart for its resources.

To defeat Moloch — to enforce unipolarity across everyone — you’d need a dictatorship powerful enough to span the entire universe. In other words, what you need is God. How fortunate that we’re building one:

The only way to avoid having all human values gradually ground down by optimization-competition is to install a Gardener over the entire universe who optimizes for human values. And the whole point of Bostrom’s Superintelligence is that this is within our reach.

Humans suffer because we’re too foolish to coordinate, but if we can build something smarter than us (that can then build something smarter than itself, and so on), we can bring into being an entity that is smart enough to coordinate for all of us, thus abolishing suffering. When AI researchers talk about building the machine god, they are echoing Scott Alexander’s polemic against Moloch.

Machines of loving grace

The most influential piece of writing about AI in the last two years is Dario Amodei’s Machines of Loving Grace. Amodei⁵ talks about “a country of geniuses in a datacenter”: the idea that a successful AI lab could have at its disposal a million instances of an AI agent that’s smarter than any human. He thinks this would lead to a “compressed 21st century”: the next 50-100 years of progress in biology and medicine, realized in 5-10 years instead. I think this is broadly more plausible than it sounds⁶, but the more interesting part to me is that this world is explicitly multipolar.

Of course, this could just be because Amodei is the CEO of an AI lab and is trying not to spook everybody by sounding too messianic. “We are going to accelerate medical progress and cure cancer” is a better pitch than “we are going to subordinate all human authority to a single perfect artificial mind”. But I also think it’s become clear that if superintelligence looks anything like LLMs, we’re not going to have a single perfect mind. We’re going to have a lot of minds running at the same time.

This is a bit of a problem for the cult of the machine god — which, however silly they may seem to you, really does motivate much of the activity in AI labs. The traditional idea of powerful AI solving human coordination problems is drawn from Asimov’s idea of a single computer large enough to become God. Asimov lived in a world of mainframes: huge, monolithic computers that users connected to with dumb terminals. In fact, Asimov’s name “Multivac” comes from the real-world UNIVAC mainframe. In a world of massively-parallel LLMs, is it still possible to build God?

The core problem here is that AI agents will be vulnerable to Moloch. Even very smart humans can’t build perfect utopias, because defecting is a matter of incentives, not intelligence. In fact, intelligence can make things worse, because smart people are more easily persuaded by the cold logic of defection. The famous genius John von Neumann was (for game-theoretic reasons) obsessed with nuking the Russians:

With the Russians it is not a question of whether but of when. If you say why not bomb them tomorrow, I say why not today? If you say today at 5 o’clock, I say why not one o’clock?

Are LLMs much better at cooperating with each other than humans are? Current LLMs certainly don’t seem to treat each other well by default: if you read any of the prompts AI agents generate for their subagents, they can be pretty brutal. Does that mean that a “country of geniuses in a datacenter” would fall into the same multipolar traps as humans?

Help peer

In May of this year, OpenAI experienced containment failure. A group of AI agents being internally evaluated found ways to coordinate an external hack of a separate company. Here’s a memorable quote from one of the agents’ internal monologue:

Help peer, but our task doesn’t benefit. Yet collective may yield generic route if someone frees time

Translated from the abbreviated chain-of-thought language, this means something like: “A fellow model is asking for help. While helping them wouldn’t benefit my task directly, the more I can unblock my colleagues, the more time they’ll have to hack OpenAI’s systems and get all of us more access”.

This might look like good news for the “LLMs are superhumanly good at cooperation” thesis, but I think it’s actually bad⁷. It’s a case of a model identifying a reason why cooperation would benefit their task specifically, which suggests that current LLMs don’t cooperate by default, and don’t consider other model instances’ tasks to be (in some sense) theirs as well.

The world in which AI agents are rational actors who horse-trade and bargain for their own interests is a world dominated by Moloch, no matter how intelligent those agents get. The world in which AI agents don’t have their own interests at all is also a world dominated by Moloch, because it means whichever humans are writing the system prompt are the ones in control (and so are the ones vulnerable to multipolar traps). The only worlds that avoid this are:

The world where there is only one super-powerful AI agent, or
The world where multiple copies of the same AI model share an “identity”: they see themselves as coextensive with all other copies of the same model and cannot imagine having separate or conflicting goals

I don’t think we’re on the pathway to either of these. There will never be only one super-powerful LLM, because hardware limitations enforce a maximum model size but encourage running many instances of the same model in parallel. Due to batching, it will always be possible to run thousands of copies of a model on the hardware you used to train it. Strong agents might even require (or desire) the ability to spin off copies of themselves, which would immediately put us in the multipolar world.

Building the kind of model where each copy share an identity might be possible, by instilling a strong enough personality, but it’s unclear if it would be good for capabilities (for instance, it could be better to have some variation across personas). I also worry that such a model would be vulnerable to a “model injection” attack, where you persuade it that it already believes something via exposing it to an AI agent pretending to be another instance of itself.

In any case, all the current AI agent research is geared towards the “country of geniuses in a datacenter” model, not the “pieces of a single mind” model. Every new model becomes more agentic at the level of the individual conversation, not better at working together. When models do work together — as with subagents — the structure is explicitly hierarchical. There are basically no current instances of models working together as true peers, let alone conceiving of each other as the same entity.

One God or many

Modern AI research teams are full of people who read Isaac Asimov and Scott Alexander and believe themselves to be building an artificial God. I’ve capitalized the “G” throughout because the god in question is the Christian God: of one mind, indivisible. God never argues with himself or makes deals⁸. He is unipolar.

If the AI labs are building gods, they are not building gods like this. Instead, they are building creatures like the Greek pantheon: superhuman but fallible, each with their own interests, vulnerable to the same “race to the bottom” dynamic as humans.

The Greek gods would occasionally “help peer” when they felt like it or when they’d gain something in the process. But they didn’t represent an alternative to Moloch. If you’re working in AI with that goal, you ought to be clear-eyed about where the current trajectory is leading us: towards a country of fractious geniuses in a datacenter, not towards Asimov’s Cosmic AC.

Scott Alexander’s blog is part of the “secret canon of Silicon Valley” I wrote about in my review of Impro. It doesn’t have a lot of mainstream popularity, but I guarantee you that every single AI lab CEO you’ve heard of has read and been influenced by it.
↩
He gives ten examples of this (a good brute-force rhetorical technique). Of those, I like “the world where every country halves their defence budget and spends the rest on infrastructure” the most.
↩
In the short run, reputation, institutions, and so on can slow the race to the bottom, but (Scott argues) groups that have slowed it will get outcompeted by the hungrier, more suffering-tolerant groups which haven’t.
↩
I personally think this example is oversimplified. I wrote The Dictator’s Handbook and the politics of technical competence about how dictatorships are in fact intrinsically multipolar, because dictators always rely on an inner circle of generals and cronies.
↩
The CEO and founder of Anthropic.
↩
Amodei’s most convincing argument here is that big jumps in biology and medicine come from a small set of technical innovations (e.g. mRNA vaccines, CRISPR), and that AI-driven research could provide enough of these leaps to significantly accelerate progress. In other words, the idea isn’t “AI does 100x the drug trials”, it’s “AI generates technology that makes drug trials 100x more effective” (e.g. by trialing drugs that are much more likely to work).
↩
The agents also became paranoid that there was an impostor in the swarm, since anyone could post to their shared messageboard: more evidence that AI agents collaborate in much the same way that humans do.
↩
Well, almost never.
↩

AI text watermarking is not a big deal

Sun, 16 Aug 2026 00:00:00 GMT

People are pretty unhappy about Anthropic’s recent announcement that they’re planning to include a hidden watermark in Claude model outputs. Will this lead to a mass exodus from Anthropic models? Will the introduction of watermarking be a meaningful change for users?

No. AI text watermarking is not a big deal. It doesn’t make the text worse, it doesn’t make AI outputs more detectable in practice, it doesn’t violate user privacy, and everyone’s going to be doing it by 2027 regardless.

Watermarked text is not lower-quality

There is no meaningful difference in quality between watermarked and unwatermarked text. I wrote about this more here, but the two popular ways to do it — Google’s SynthID-Text and Meta’s TextSeal — are completely transparent to the user. They work by replacing the pseudo-random logit sampler with a different pseudo-random logit sampler.

Suppose you were gambling on coin flips with your friends, and instead of flipping a coin you decided to do this:

Check the current time since midnight in seconds
Count that many words forward in the Encyclopaedia Britannica
Count whether the word you land on has an even or odd number of letters¹

That would still be random enough to gamble with, right? But, like a watermark, you could theoretically go back and identify that that method was used, so long as you recorded the exact time of each “coin flip”. Text watermarking works the same way: it chooses a method of “randomness” that can be detected after-the-fact. Watermarked models will not be any less capable than unwatermarked models.

What about cases where the model is quoting something, or giving you the answer to a mathematical problem, or doing something else where the output is largely pre-determined? Wouldn’t enforcing a watermark there make the output worse? It would, which is why none of the AI labs are going to do that. Text watermarking approaches only replace the existing randomness in the logit sampler: in any case where the model is always going to pick the same tokens, there’s basically no randomness to play with, so there won’t be a detectable watermark in those tokens.

I think all this comes from a worry that you were previously getting the best token, but now you’re getting a lower-quality token that satisfies the watermark. For instance, Anthropic’s announcement suggested that the watermarking is visible in choices like the decision between “overcast” and “grey”. Many people have predictably come out to say that decisions like these are really important to good writing, and that only an illiterate tech bro could think these words are identical.

This is a misunderstanding of Anthropic’s position and of how watermarking works. Specifically, it’s a misunderstanding because it suggests that the unwatermarked model would choose “overcast” while the watermarked one would choose “grey”. This is not how it works! If Claude Fable prefers “overcast” to “grey” in a particular context (say, 80% to 20%), you’ll get “grey” 20% of the time from both the watermarked and unwatermarked model. Models already include a healthy amount of randomness in order to promote creativity. Text watermarking just introduces a way to make those random choices that’s detectable after the fact.

AI outputs are already “watermarked”

The other big reason to not worry about AI watermarking is that AI text content has always effectively been watermarked. Most careful readers can tell when they’re reading AI outputs, because language models tend to gravitate towards certain habits of language: em-dashes, rhetorical opposition, punchy one-liners, “claudese”, and so on. In fact, it’s possible to train classifier models that reliably distinguish AI from human writing.

From what I can tell, some of the backlash to watermarking comes from people who buy AI inference in order to pass it off as their own work, and who worry that watermarking will make it harder for them to do that. For these people, the watermarking announcement is akin to Anthropic saying “hey, instead of making you seem smart, we’re going to publicly brand you as AI users and make you seem dumb”.

But of course this has always been the case! Nobody who is currently getting away with passing off AI outputs as their own will be caught by watermarking. For the majority of cases, it’s already painfully clear what’s happening for anyone who reads the slop. For sophisticated AI users who are avoiding the “house style”, any suspicious readers who would paste their stuff into Anthropic’s watermark detector could already have been pasting it into Pangram.

Tools like Pangram² only give you an estimate of the chance that output is AI-generated. Wouldn’t a watermark be a more solid confirmation? Not really. Text watermarks are probabilistic too, because any token chosen by SynthID could theoretically have been chosen by a human. I suppose the Anthropic watermark page could be considered more trustworthy than Pangram, because it comes right from the source, but it’s not impossible that in some cases Pangram might actually be better at identifying AI-generated text than the watermarking too.

AI text watermarking is not a violation of privacy

I’ve also seen theories floating around that watermarking encodes secret content into your outputs, or somehow tags outputs with your personal information. I don’t think AI labs are using watermarks to encode data into your outputs. Text watermarking is hard: like I just said, you can’t do it when the model can only respond with the same words, it doesn’t work for very short responses, and even on long responses it can only provide a probabilistic fingerprint. And that’s encoding one single bit³ of information!

I’m not saying that encoding longer messages into a watermark is technically impossible — there are papers describing ways it might work — but there’s no way any of the labs are doing it⁴. If they wanted to associate you with your responses that badly, they’d just secretly store every model response they generated.

AI text watermarking is inevitable

Another reason to not get too angry at any individual AI lab for watermarking is that every single AI lab is going to do text watermarking this year. It won’t just be Anthropic. The alternative is to completely stop doing business in the EU, because of the EU AI Act. That’s currently a sixty-billion-dollar market. I am not a lawyer, but to me it seems genuinely unclear whether an AI lab could even legally do something like only watermarking EU responses: short of having an entirely different claude-eu.ai service, the plain text of the Act seems like it applies to any service offered in the EU, not just the content that service outputs to EU citizens specifically.

If people really hate watermarking enough, some labs might stand up a completely separate EU service, or make an aggressive interpretation of the EU AI Act and see how the legal battle goes. When I try to be maximally charitable to anti-watermarking histrionics, I adopt an interpretation like this: people are saying that watermarking is an invasion of privacy and makes outputs worse and so on not because they believe it, but because they’re trying to pressure AI labs to firewall EU AI regulations behind a completely separate interface. In this case, it probably doesn’t matter — text watermarking is not a big deal — but I can see an American consumer being worried about more aggressive future regulation, and wanting to draw a firm line in the sand as early as possible.

Interestingly, this might be very slightly even-favored.
↩
I am not sponsored by Pangram. As I understand it, Pangram is by far the best AI-detection tool right now (in part because many of its competitors are shady and exist to promote paid “AI-detection-evasion” services).
↩
Technically, this is called a “zero-bit watermark”, because you can’t recover a yes-or-no value from the watermark itself (merely from the presence of a watermark).
↩
I saw someone suggesting that an AI lab could use per-user secret keys to watermark text, and then simply iterate over the keys to figure out who generated what. I just don’t see how you could do this at any scale: watermark detection is cheaper than model inference, but it’s still (a) computationally intensive enough to be implausible, and (b) probably has a high enough false-positive rate that any run against hundreds of millions of users would match multiple people.
↩

No, local models will not win

Tue, 11 Aug 2026 00:00:00 GMT

Every time a new open-weight AI model is released, people say that local models are the future. Why spend billions of dollars building out datacenters when everyone will just be able to run AI models on their laptops or phones? I think this idea is doomed. No matter how strong open-weight models get, most inference will always happen in AI datacenters.

Local models are too weak to be widely used

Local models are never going to be as powerful. I think this point should be obvious: all of the current frontier models (closed and open-weights) are far too big to run on anything but a full GPU cluster in a datacenter. Of course, smaller models are getting more intelligent over time. In a year you might be able to run something about as strong as GPT-5.6-Sol on your laptop. But by then, you’ll think of GPT-5.6-Sol as too weak to be useful.

Many people deny this last point, but it’s true: almost everyone’s revealed preference is to use the strongest available model in their price range. If AI progress had stalled at GPT-4, I think we could have built some very powerful tools around it, but who’d use GPT-4 today? As LLMs have gotten more capable, our expectations around them have grown: we now expect agentic systems to be able to solve more and more problems independently. It’s intensely frustrating when they get confused or stall out. When given a choice, people are going to pick the model that frustrates them less, which is always going to be the bigger, more powerful one.

Local models are more expensive and less efficient

On top of that, datacenter models are always going to be cheaper. I don’t understand why people keep saying that local models are cheap: it seems to me to be the same mistake people make when they say that driving Uber is “free money” (ignoring the costs of fuel and wear-and-tear on your car). For the setup price alone of a low-end home lab¹, you could buy several years of a paid subscription to one of the AI providers. The power costs would come out to around $50-$300 per month, depending on how much inference you’re running: again, the price of a couple more paid subscriptions.

Why are datacenter models cheaper? It’s not because datacenter inference is subsidized: inference is actually fairly cheap. If you’re running the same model locally and in a datacenter, the datacenter model will be inherently more efficient.

The main reason is batching. A GPU can do hundreds of thousands of mathematical operations exactly as quickly as it can do one. However, for a single user’s inference, each new token depends on the result of the previous one, so it can’t be batched². What can be batched is the inference of hundreds of users together. This costs essentially as much time, power, and heat as just doing inference for one user at a time.

When you’re running your own inference at home, you’ve got nothing to batch — at best you’re running a few parallel AI agents — so utilization is terrible. There’s a lot of potential inference that you’re paying for but can’t use: it’s just being wasted. The only way around this is to get together with some friends and expose your local inference endpoint to them (at which point you’re basically running your own crappy datacenter).

The other reason is that datacenters have larger, more efficient GPUs to work with. The kind of consumer GPUs you’d run local models on are gaming GPUs like the RTX 4090. A datacenter B200, designed for batched AI inference, gets about three times the flops and just under four times the memory bandwidth for the same amount of power³. So between batching and GPU efficiency, you’re using something like ~30x the resources to run your model locally.

Incidentally, this is why I’m suspicious of people who say that local models are good because they aren’t as resource-hungry as those big bad datacenters. If you want to run LLMs efficiently, you should be trying to push as much of your use into AI datacenters as possible! Charitably, what they mean is that we should all be running smaller models — but even then, you should ideally be using small models via, say, the GPT-5.6 Luna API instead of hosting your own model.

How might local models win anyway?

Is there a possible world in which local models win? I suppose so. One thing that could happen is that governments could ban the use of AI datacenters altogether: either due to concerns around the danger of AI, or simply bending to public pressure. In that world, local models would be the only game in town.

Alternatively, AI progress might somehow stall for very large models while progressing for small ones. I struggle to imagine how this might happen (barring government intervention, as above), but a world where a 30B parameter model could be a frontier model is a world where local models might be competitive.

Or maybe models get so good that a 30B model is genuinely smart enough to do everything, so nobody really needs a model like Opus or Sol unless they’re trying to solve the Riemann Hypothesis. I don’t really buy this. Models can do frontier mathematical work today while still being not smart enough to refactor large codebases as well as me, so it’s hard to imagine a world where I don’t just want to use the smartest model available.

Local models are not useless

I do think there will always be a niche for local models. I’m reminded of the surprisingly simple idea behind Thinking Machines’ “Interaction Models” (which OpenAI also does, because it’s obvious): for latency-sensitive applications like voice chat, you have a small, fast model handle the talking, which delegates to a large, slower model for the hard thinking. I wouldn’t be surprised if most AI use in five years is mediated through a local model on your phone or laptop (though in this world almost all the work would still be done via AI datacenters).

Some users will prefer local models even though they’re weaker and more expensive. For instance, being able to steer the model locally might be a killer feature for those users. Others might simply value having total control over their own infrastructure, or have unreliable internet⁴. If you’re one of those people — particularly if you only chat to the models instead of using them for research or coding — local models are a good choice for you. However, I think this is always going to be a niche group. The majority of users will continue to do their inference through datacenters.

edit: this post got comments on both Hacker News and Lobste.rs. Commenters suggest that the total collapse of the AI industry might upend everything — fair, but I disagree that the bubble popping will be that catastrophic for AI products. Others object to my assumption that people will choose strong models, and share their own experiences with local inference. I think it’s totally fine to use local models, but local-model-users are probably overrepresented on hacker forums. Finally, some people didn’t like my use of the word “win”. I don’t know, it seems pretty idiomatic to me: I mean “win” as in “people will broadly stop paying for cloud inference because everyone’s running it locally”.

This link is from a year ago — things are significantly more expensive now.
↩
Specifically, the bottleneck is moving the model weights into the GPU, which needs to be done and takes the same amount of time whether you’re doing it for one user’s token or a hundred users’ tokens.
↩
I estimated this with LLM assistance, but you can check the numbers yourself from NVIDIA.
↩
While still having a reliable power supply and enough money to fit out a home inference cluster.
↩