{"componentChunkName":"component---src-templates-blog-post-js","path":"/jev-means-structured-output-is-interesting-again/","result":{"data":{"site":{"siteMetadata":{"title":"sean goedecke"}},"markdownRemark":{"id":"0a94f647-5ace-5853-888b-9761616a2970","excerpt":"I don’t write blog posts about new models. That’s Simon Willison’s beat, and he’s very good at it. But I want to write about Jev, which is a different kind of…","html":"

I don’t write blog posts about new models. That’s Simon Willison’s beat, and he’s very good at it. But I want to write about Jev, which is a different kind1 of AI model: a “System One”2 model. As it turns out, it’s not that different from an ordinary LLM with structured output, but the interface it uses is very cool and I hope it becomes more widespread.

\n

How Jev is different from LLMs

\n

Ordinary LLMs take in some human-language prompt and produce some human-language output. They do so autoregressively: first they produce one token, then the next, then the next, and so on.

\n
User: Who invented the sandwich?\nLLM: The sandwich was invented by the Earl of Sandwich.
\n

This makes them extremely flexible, since they can do literally anything a computer can do. But it also makes them slow and weird. Slow, because they have to run a whole new generation pass per-token, and weird, because the space of human language is so broad that you can get really odd behavior from a model trained on it.

\n

Jev takes a human-language prompt, but it does not produce human-language output. It only produces structured output.

\n
User: { state: \"What color is the sky?\", choices: [\"blue\", \"red\", \"yellow\"] }\nJev: { answer: \"blue\" }
\n

So far, so ordinary: LLMs do this already. But it turns out that if you build a model that only produces structured output, you get some interesting and desirable properties.

\n

Jev is consistently fast

\n

Jev is always really fast. The fastest response time is around 70ms instead of a couple of seconds for normal LLMs. Even better, the slowest response time is only 500ms. Because Jev only does structured output, it isn’t autoregressive: it can produce answers to many questions in parallel in a single forward pass. When a LLM is producing structured output, it has to produce the tokens ”{”, ” ”, “answer”, ”:”, and so on with successive forward passes3. Jev does it all in one go.

\n

The most compelling example of Jev’s speed is that the model can play Doom. You can feed a text-based representation of the current game state into the model, combined with a set of choices like “should the trigger be held down”, “what should the current goal be”, “given that the current goal is X, what keyboard input should be pressed”, and so on, and it works — latency is low enough and the system is smart enough that the model plays well in real time.

\n

\n \n