Different bets on autonomy: how training and data choices decide what can go wrong

Full autonomy has no settled recipe. Beneath the product announcements, the real disagreement is about how a machine should learn to act in the world, and that choice of training and data paradigm is the bet a company is making. The main bets in public view today are:

• End-to-end imitation learning on large real-world fleet data, where the system learns to map sensors to actions directly from human driving or operation
• Modular stacks with explicit, separately trained and separately verifiable perception, prediction, and planning
• Simulation and sim-to-real transfer, where most experience is generated in a simulator before deployment
• Teleoperated demonstration data, where human operators puppet the machine to collect examples of the task, common in humanoid programs
• Internet-scale video and multimodal pretraining, where a vision-language-action model inherits broad priors before any task-specific training

These are not interchangeable means to the same end. Each paradigm has a safety signature: a set of failures it makes rare and a set it makes hard to see or hard to bound. Imitation learning inherits the long tail and the covariate-shift problem, performing well on the distribution it was shown and degrading silently outside it. Sim-to-real lets you train on dangerous scenarios no fleet could safely collect, but it inherits the reality gap, so a system can be confidently wrong exactly where the simulator and the world diverge. Teleoperated data directly demonstrates the correct action but is expensive to collect and tends to under-sample the rare, safety-critical events that matter most. End-to-end models are hard to verify because they expose no interpretable intermediate to audit, while modular stacks expose verifiable interfaces but propagate upstream errors downstream. Foundation-model approaches gain generalization at the cost of unpredictable, hard-to-bound failure modes.

This work surveys the dominant training and data paradigms across autonomous vehicle and humanoid programs and maps each to its safety implications: the failure modes it suppresses, the ones it conceals, and the verification and assurance methods that are actually compatible with it. The aim is a practitioner's map of which safety claims each paradigm can and cannot support, so that the choice of how to train a machine is made with its safety consequences in full view rather than discovered after deployment.