For decades, robots were programmed for each task: precise motions for a specific part on a specific production line. Teaching a robot something new meant engineering a new program. Vision-language-action (VLA) models promise a different approach: a single model that sees through a camera, understands natural-language instructions and outputs robot actions — and brings knowledge learned from the internet into the physical world.

From language models to robot actions

A VLA model starts from the same ingredients as multimodal chatbots: a model trained on vast amounts of web images and text, so it knows what objects are, what words mean and how things relate. It is then trained further on robot data — camera images paired with the actions a robot took.

The key trick, popularized by Google DeepMind’s RT-2 (Robotics Transformer 2) in 2023, is to represent robot actions as tokens, just like words. The model “writes” a sequence of numbers describing how to move the arm and gripper, in the same way it would write text.

Why web knowledge helps

Because the model learned from web data, it can understand instructions involving concepts that never appeared in the robot training data. The RT-2 paper showed robots performing tasks such as picking up an object described abstractly — for example choosing the toy dinosaur when asked for an “extinct animal” — or reasoning about which object could be used as an improvised hammer. The authors reported that RT-2 substantially improved generalization to new objects and instructions compared with its predecessor.

Sharing data across robots

A major obstacle in robotics is data: each lab has its own robot and collects relatively little data. The Open X-Embodiment project, a collaboration of many research institutions, pooled data from numerous robot types into a shared dataset. Models trained on this mixture (RT-X) performed better than models trained on each robot’s data alone, suggesting that, as in language, more diverse data helps.

What came after

Since RT-2, many groups have released VLA models, including open models trained on large robot datasets, and companies building humanoid and industrial robots have adopted similar architectures. Common directions include:

  • combining a slower “reasoning” model with a fast low-level controller;
  • generating smooth continuous motions instead of discrete action tokens;
  • training extensively in simulation;
  • learning from videos of humans doing tasks.

The limits

  • Speed: large models can be too slow for fast, reactive control on their own.
  • Precision: fine manipulation still challenges these systems.
  • Data scarcity: robot data remains tiny compared with internet text.
  • Safety: an instruction-following robot must reliably refuse unsafe actions and handle mistakes in the physical world, where errors break things.
  • Evaluation: comparing systems is hard because results depend on specific robots, objects and environments.

Even so, VLA models have changed the conversation: robotics increasingly looks like an extension of foundation-model research rather than a separate field.

Why data is the bottleneck

Unlike text, robot interaction data must be collected physically, making large datasets slow and expensive to build.

What to watch next

  • Open VLA models. More openly released models and datasets will make robot learning research reproducible.
  • Learning from human video. Using the vast supply of videos of people doing tasks could ease the robot data shortage.
  • Fast controllers. Architectures that pair a reasoning model with a high-frequency motor controller.
  • Standard benchmarks. Shared evaluation tasks to compare robot models fairly.
  • Real deployments. Factories and warehouses using instruction-following robots outside of demos.

Key takeaways

  • VLA models take camera images and language instructions and output robot actions.
  • RT-2 represented actions as tokens, letting web-scale knowledge guide robot behavior.
  • Web knowledge helps robots handle objects and instructions not seen in robot training data.
  • Open X-Embodiment pooled data from many robot types and improved generalization.
  • Speed, precision, scarce robot data, safety and evaluation remain open challenges.

Frequently asked questions

Is a VLA model the same as a chatbot controlling a robot?

They share foundations, but VLA models are trained on robot data to output motor actions directly, rather than only producing text plans.

Why is robot data so scarce?

Collecting it requires physical robots operating in the real world, which is slow and expensive compared with gathering text or images from the web.

Can these models run on the robot itself?

Smaller models can, but large ones often need powerful onboard computers or remote servers, which affects latency.