Safety world models: predicting harm before a machine acts
A world model is a learned predictor of how a scene will unfold. Most are built to improve capability. This work studies the world model built for safety: one that forecasts the futures in which a person gets hurt and lets a machine rule those actions out before it commits to them.
Most safety today is reactive. A machine senses its current state and triggers a stop or a swerve once something is already wrong, which is often too late when the other body in the scene is a person who moves, hesitates, and changes direction. Acting in time requires anticipation: a model of how the scene will evolve over the next few seconds, including how the people in it are likely to behave. That model is a world model, and most of the ones being built right now are aimed at capability, used to train policies, generate data, and improve planning.
A safety world model answers a different question. Before the machine commits to an action, it asks what distribution of futures that action opens up, and in particular which of those futures contain harm. It rolls out candidate actions against predicted human and object behavior, weighs the plausible outcomes, and rejects actions whose tail of futures includes someone getting hurt. The properties that make such a model trustworthy are not the ones that make a capable one:
- Calibrated uncertainty: it must know when it does not know, because an overconfident forecast of a safe future is more dangerous than no forecast at all
- Long-tail coverage: the rare, safety-critical futures it most needs to predict are exactly the ones least represented in its training data
- Verifiability: a generative predictor of the future has to be validated well enough to rest a safety case on, which is far harder than measuring average accuracy
- Latency: a prediction is only useful if it arrives in time to change the action
This work studies world models built specifically as a safety layer rather than a capability one. It asks what such a model must predict, how its uncertainty has to be calibrated and bounded, and how it can be validated enough to carry real safety claims. A safety world model sits directly on top of perception: it inherits whatever the machine got right or wrong about the scene, and turns that read into a forecast a safety decision can actually stand on.
What you're reading is the abstract.