Many people claim that AI inference is unprofitable to serve, and thus must be subsidized by an ocean of dumb money from investors who believe that some future AI model will come to dominate the world economy. When that dumb money goes away, so will AI products. According to this view, LLMs are just inherently too expensive (in terms of money, power, and water) to be used in consumer products. In fact, they can only be used today by externalizing the costs: money onto VC funds and now retail ETF investors, power onto electric utility consumers, and water onto the communities where datacenters are built.
\nThere are good reasons to dislike AI, but this really isn’t one of them. In fact, AI inference is obviously profitable.
\nFrontier AI providers are reporting 70%-80% gross margins on inference, but maybe we can’t trust them. Let’s do some very rough estimates on the actual cost.
\nA Nvidia A100 consumes 400W of power under full load. In practice, even a carefully-tuned inference server will not be at full load all the time, but it’s at least an upper bound. Suppose you’re running a dense 70B model1, which will fit comfortably (unquantized) on four A100s at around 2M tokens per hour. At industrial power prices, that’s about 13c/hr in the USA. Suppose (pessimistically) cooling is the same cost. That’s about 13 cents per million output tokens2.
\nLet’s amortize the cost of the GPUs, since that’s going to be the most expensive part. An A100 costs about $20k. If each A100 lasts around five years3, you’ll have to make 16k/yr in profit to recoup your capital investment (or $1.80 per hour). At lower utilization, it’ll take longer to recoup, but your GPUs will also last longer. Either way, your overall inference costs are at about one dollar per million tokens.
\nGPT-5.4-mini charges $4.50 per million tokens, and stronger OpenAI or Anthropic models are three to six times as expensive. It’s hard to make a direct comparison because we don’t know the size of OpenAI or Anthropic models, but the claimed 70% or 80% profit margin is extremely plausible.
\nWhat if you don’t trust my estimates either? Let’s look at the pricing of open-weights Chinese LLMs. DeepSeek have claimed a bit over 80% profit margin on inference for DeepSeek-R1. Since their API pricing for R1 is less than half that of OpenAI or Anthropic4, that suggests that my estimates above for inference cost might be too expensive. Cooling at scale is probably cheaper than power, R1 only has half the active parameters of a dense 70B model, modern GPUs are more efficient than the A100, and there are significant economies of scale in inference.
\nSince DeepSeek’s models are available for anyone to download, they can’t get away with extracting a large profit margin. One of the other inference providers would undercut them with the same model. Inference costs for DeepSeek-V4-Pro on the market are around 87 cents per million output tokens, which is probably pretty close to the actual cost of serving the model.
\nAll of this doesn’t mean that OpenAI or Anthropic are profitable. Those companies are making huge capital investments that may or may not pan out, and are spending enormous amounts of money on talent and compute to train brand-new models and retain users.
\nThey’re doing crazy things like offering per-month subscription models for nearly unlimited inference, which is almost certainly not profitable. If you used an API token instead of your Anthropic subscription in Claude Code, you’d pay ten times the cost. But that doesn’t mean API-based Claude Code couldn’t be a good deal. Some people are already using DeepSeek’s inference API for agentic coding, because once you take away the huge profit margin it’s cheaper than the relative per-month subscription.
\nWhy won’t OpenAI or Anthropic lower their prices? Supposedly OpenAI has thought about it, but for an AI lab, inference has to subsidize training costs. A company like OpenAI has to fund the production of new models from the inference margins on existing models (at least partially). That’s why the margins on inference are so high: the AI labs are trying to squeeze out every dollar so they can stay alive in the training arms race.
\nHowever, inference only has to subsidize training costs for an AI lab. If you’re merely an inference provider, you don’t have to do any training at all. Therefore, even if OpenAI and Anthropic go out of business, whoever snaps up the rights to their frontier models will be able to continue selling Opus and GPT inference at a profit5. The AI bubble popping will not mean the end of the inference business, because AI inference is obviously profitable.
\nExpensive frontier models are probably mixture-of-experts, not dense, which is tougher to estimate. However, I think a 70B dense model and a MoE with 70B active params will come out to basically the same numbers at scale (though the MoE will require more GPU memory and thus a greater upfront cost). Are frontier models around 70B params? Nobody outside the AI labs really knows, but my guess is that 70B is probably larger than a Haiku/mini class model.
\n↩\nI think it’s reasonable to estimate the cost of output tokens only, since they’re by far the most expensive part of serving inference. Input tokens are cheaper for two reasons: transformers let you prefill them in parallel, and for most real-world use cases they can be aggressively cached in the KV cache.
\n↩\nIt’s common (and wrong) to estimate GPU lifespan at three years. I wrote a lot about this in AI GPUs probably live longer than three years.
\n↩\nAgain, this is just an guess, since we don’t know what OpenAI or Anthropic model is equivalent in size to R1.
\n↩\nI do wonder if Anthropic would be able to prevent other people from being able to access the model if the company goes out of business. Anthropic is currently in debt to Broadcom, Google, and a bunch of private equity firms. Would they get the Mythos and Opus weights, over Dario’s protestations?
\n↩\nPeople who think current AI use is unsustainable often rely on the claim that inference GPUs only last “three years at the most” under load. The idea here is that once the AI bubble money drains away, current infrastructure will rapidly become obsolete, and there won’t be enough money floating around to buy a whole slate of brand-new GPUs. Inference costs would thus rapidly become way too expensive for current AI products to make any financial sense.
Continue reading...