To drive safely, a vehicle must know where everything around it is, what each object is, and what it is likely to do next — in rain, at night and at highway speed. Autonomous vehicles combine several types of sensors with AI to build that picture many times per second.

Cameras

Cameras capture rich visual detail: colors, traffic lights, road markings, signs, and the appearance of pedestrians and vehicles. They are inexpensive, and modern computer vision models extract a great deal from them.

Strengths: reading signs and signals, recognizing objects, high resolution. Weaknesses: no direct distance measurement (it must be estimated), and performance suffers in glare, darkness, heavy rain or fog.

Lidar

Lidar (light detection and ranging) emits laser pulses and measures how long they take to bounce back, building a precise 3D point cloud of the surroundings.

Strengths: accurate distance and shape measurement, works in darkness. Weaknesses: historically expensive (though prices have fallen substantially), lower resolution for reading signs, and performance can degrade in heavy rain, snow or fog.

Radar

Radar uses radio waves to detect objects and, through the Doppler effect, directly measures their speed.

Strengths: works well in rain, fog and darkness; measures velocity; long range. Weaknesses: lower resolution, making it harder to distinguish object shapes. Newer “imaging radar” improves detail.

Other inputs

  • Ultrasonic sensors for very short ranges, such as parking.
  • High-definition maps with detailed lane information, used by many robotaxi systems.
  • GPS and inertial sensors for positioning and motion tracking.
  • Microphones in some designs, to detect emergency vehicle sirens.

Sensor fusion and the AI stack

No single sensor is perfect, so many systems fuse their data. A typical software stack includes:

  1. Perception: detect and classify objects, lanes and free space from sensor data.
  2. Localization: determine the vehicle’s exact position.
  3. Prediction: forecast what other road users will do.
  4. Planning: choose a safe, comfortable path and speed.
  5. Control: steer, accelerate and brake.

Increasingly, companies use large neural networks that handle several of these steps together, trained on huge amounts of driving data and refined in simulation.

Camera-only vs. multi-sensor

The industry is divided. Robotaxi operators such as Waymo use cameras, lidar and radar together, arguing that redundancy improves safety. Tesla has pursued a camera-centric approach, arguing that humans drive with vision and that AI can do the same at lower cost and scale. Each approach has trade-offs in cost, robustness and validation, and the debate continues as both evolve.

Sensor fusion in practice

Most autonomous systems combine sensors so the weaknesses of one are covered by another: cameras read signs and colors, lidar provides precise distances, and radar works in rain and fog and measures speed directly. Software fuses these inputs into a single model of the world around the vehicle, tracking objects and predicting their movements many times per second.

What to watch next

  • Falling lidar prices. Cheaper solid-state lidar could make multi-sensor designs affordable for consumer cars, not only robotaxis.
  • Imaging radar. Higher-resolution radar may take over some tasks from lidar, especially in bad weather.
  • End-to-end driving models. Large neural networks that map sensor input more directly to driving decisions are a major research and industry trend; validating their safety is the open question.
  • Expansion of operating areas. Robotaxi services moving into new cities, highways and climates are the clearest real-world test of each sensor strategy.
  • Regulation. Rules on safety reporting, data recording and approval of driverless systems will shape which approaches can scale.

Key takeaways

  • Cameras provide rich visual detail but must estimate distance and struggle in poor visibility.
  • Lidar builds precise 3D maps using laser pulses.
  • Radar measures speed directly and works well in bad weather.
  • Sensor fusion combines strengths; AI handles perception, prediction and planning.
  • Companies disagree on camera-only versus multi-sensor designs.

Frequently asked questions

Why do robotaxis have spinning devices on the roof?

Those are typically lidar sensors, which scan the surroundings with laser pulses to build a 3D view.

Can self-driving cars drive in snow?

Snow is challenging: it can obscure lane markings and affect sensors. Many deployments started in mild climates and expand gradually.

What is sensor fusion?

Combining data from different sensors so the system benefits from each one’s strengths and compensates for their weaknesses.