The Forecast Is Not the Weather

A weather forecast is not the weather. It has no wind, no rain, no cold front sweeping across the ocean. It is only a compressed model of what might happen. Sometimes it is eerily accurate. Sometimes you leave home without an umbrella and get soaked.

That does not make the model useless. It only means you should never confuse the forecast with reality itself.

AI agents are now falling into the same trap.

To train them at scale, researchers need rich environments where agents can search, click, write files, call APIs, make reservations, break things, and try again. Real environments are too expensive, slow, fragile, and dangerous for that kind of experimentation. You cannot let an experimental agent repeatedly transfer real money or delete production data just to collect more training trajectories.

So they build synthetic worlds instead.

One recent paper proposes generating those worlds mostly through language. Another insists they should be executable, code-driven, and backed by real databases. Both approaches are impressive. Both are still simulations. And both force the same uncomfortable question: when an agent learns inside a model of the world, what exactly has it learned — the world, or just a convincing reflection of it?

Teaching a model how the world talks

The first paper, Qwen-AgentWorld, attacks a basic gap. Agents are trained to choose actions, but they rarely learn to predict what those actions will actually cause. Instead of optimizing only state to action, the authors train language models on (state, action) to next observation. The model learns to simulate terminal output, browser responses, tool results, Android screens, and operating-system interactions across millions of trajectories in seven domains.

The idea is powerful: before teaching an agent to act, teach it to expect consequences. An agent modifying a repository should anticipate which files will change. One buying a ticket should anticipate whether seats actually decrease and whether the reservation will persist. This kind of world modeling can serve as a simulator or as pre-training for the agent itself, and the authors report improvements on several benchmarks.

But there is a catch. Predicting what the world will say is not the same as preserving what the world must become.

The shadows on the wall

A language world model can generate plausible responses: “Reservation confirmed,” “File deleted,” “Payment successful.” It can imitate the visible surface of reality extremely well. The problem is what it may fail to maintain behind that surface. Did the account actually get charged? Did the seat inventory decrease? Is the deleted file still gone five steps later? Does the reservation remain valid after another tool call?

The model can produce convincing shadows while the underlying state quietly falls apart. It learns how the world talks without fully learning what its words commit the world to doing.

Plato’s cave offers a useful image here. The prisoners become experts at recognizing patterns in the shadows moving across the wall. They can predict which shadow appears next. But predicting the next shadow is not the same as understanding the machinery that produces them. A language world model risks becoming an extraordinary forecaster of environmental shadows while remaining uncertain about the consistent reality that must keep producing them.

The alternative: make the world execute

The contrasting paper, Agent World Model from ICML 2026, takes a different path. Instead of asking a language model to improvise the next observation, it uses language models to help generate executable environments backed by SQL databases and real code. They scale this to one thousand synthetic environments with persistent state and verifiable transitions.

When an agent books a flight, the stored state actually changes. When it deletes a file, later queries reflect that deletion. The advantage is clear: actions have inspectable consequences. Reward functions can verify real outcomes instead of trusting a final message that merely claims success. The world becomes more accountable.

But executable environments have their own limitation. They can only produce the consequences their designers remembered to encode. Anything omitted from the schema effectively does not exist. They risk becoming narrow, rigid, and blind to the messy variables that make real environments difficult.

Two imperfect kinds of world

We are not choosing between reality and illusion. Neither approach creates the real world. We are choosing between two different models.

Language-generated worlds are broad, flexible, and rich in linguistic variation and long-tail situations, but they are vulnerable to producing locally plausible transitions that fail to remain globally consistent. Executable worlds are persistent, verifiable, and rule-governed, but they risk being structurally narrow and missing the disordered parts of reality.

One can invent too much world. The other can forbid too much world.

The interesting question is not which one wins. It is what each one should be allowed to control. Language is excellent at generating the visible surface of a world — messages, descriptions, ambiguity, personality, and unexpected situations. Code is better at preserving the commitments that must remain true: balances, permissions, inventory, identity, and irreversible change.

That suggests a simple division of labor: generate the surface, execute the consequence.

The hybrid that should exist

The most practical path is a hybrid: keep a small executable causal core and let language generate everything else. Call it, provisionally, an Executable-Core Language World.

It would have four components. An executable state holds only the variables that matter for rewards, permissions, irreversible consequences, and future consistency — balances, inventory, reservations, file existence, and authorization. A transition engine modifies that state according to strict rules that no sentence can override. A language renderer receives a read-only view of the resulting state and generates varied, natural observations. And a consistency validator checks that the generated text does not contradict the protected state.

Consider an agent attempting to book a cancelled flight. The executable core records that the status is cancelled and seats are zero. The language model can generate “Sorry, that flight has just sold out” or “Reservation failed: no remaining inventory.” It may not generate “Reservation confirmed.” The surface can vary. The consequences cannot.

This is not an attempt to rebuild the entire world in code. It is an attempt to keep explicit only what must remain true after the agent acts.

The cheapest experiment worth running

Testing this idea does not require a massive model or a thousand environments. Build one small flight-booking world in three versions: pure executable templates with deterministic transitions, a pure language simulator, and the hybrid. Then test normal bookings, stale prices, cancelled flights, conflicting signals, and longer sequences of dependent actions.

Measure task success, invariant violations, contradictions between visible observations and hidden state, state divergence over time, and latency. This small experiment would not prove that the hybrid creates better agents. It would test a narrower but crucial claim: can a minimal executable kernel preserve causal commitments while a language model provides environmental richness around it?

That alone is worth knowing. It is cheap, visible, falsifiable, and easy for others to challenge.

The hybrid may be unnecessary

The proposal should not be treated as correct merely because it sounds architecturally sensible. The experiment may reveal that the hybrid adds no meaningful value. If a pure language simulator preserves state and long-horizon consistency just as well, the executable core may be unnecessary. If executable templates already provide enough variation, the language layer may be little more than decoration. If the consistency validator rejects too many responses, the system may become slow and repetitive.

Those are legitimate outcomes. If a simpler system performs just as well, we should use the simpler system. The experiment must retain the right to embarrass the proposal.

Back outside the cave

Language world models may become extraordinary predictors of the shadows cast by terminals, websites, tools, and users. Executable environments may preserve the objects and rules that keep those shadows coherent. But neither is reality.

The goal is not to escape simulation. It is to know which parts of the simulation are safe predictions, which parts are binding commitments, and which parts can safely be wrong.

A forecast can guide us. A shadow can teach us. A model can help an agent anticipate the world. But none of them becomes reality merely by sounding convincing.

The problem is not that the agent lives in a simulation. The problem starts when the agent — and sometimes the researcher — mistakes the forecast for the weather.

References:

Qwen-AgentWorld: Language World Models for General Agents

Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning