From perception to planning, everything happens in real-time. “Zoox vehicles must be able to react to any environmental change as they occur,” said He. “When our vehicles encounter a new scenario, they will approach cautiously, not unlike what a human driver would do. Our vehicles also understand when there are temporary environmental changes, such as unmapped construction zones.”
Optimal perception, performance, and safety
Designing an autonomous vehicle from the ground up is challenging and time-consuming, but it does provide benefits. One is having complete control of the perception systems. Zoox has implemented three complementary real-time perception systems that are independent and redundant of the others, running simultaneously. The first is the main AI system, which provides our main perception output.
“The first system uses a variety of machine learning models and obtains a sophisticated understanding of the world around the vehicle,” said He. “This includes our vision, lidar, and radar sensors—all needed for detection, tracking, and segmentation. We also use a motion modeling system that gives our system an understanding of how agents move through the world.”
The second and third systems act as collision-avoidance systems. They both focus on checking for possible obstructions in the Zoox vehicle’s direct path that might lead to a collision. One of these systems consists of geometric, interpretable algorithms that operate on sensor data and are responsible for detecting objects in our intended driving path. The other is a machine-learned algorithm, which we call Safety Net, that performs both detection and prediction of future movement in a short time horizon, 360 degrees around our vehicle.
These two systems are architecturally different from the central AI system to avoid common-cause failures. They are also optimized for low end-to-end latency, allowing our vehicle to react to sudden obstructions. For example, a pedestrian running in front of our vehicle after being occluded by a vehicle on the side of the road. If the future collision probability meets a certain threshold, the system will trigger the vehicle to stop.