Every journey needs a plan
How our vehicles choose the best way forward.
If you’ve been following Zoox for a while, you might know how our Perception system lets our vehicle see and hear its environment and how our Prediction technology enables it to understand the behaviors of road users around it. But there’s another step, without which we wouldn’t be going anywhere. Enter Planner.
Planner takes all the data from the rest of the autonomous stack and turns it into a plan of action. It handles the countless big and small decisions that go into every journey.
The Zoox robotaxi driving autonomously in Foster City, California.
Decisions, decisions
Before the journey begins, our vehicle has to figure out the quickest route from A to B. Then, once it’s on the road, there are a lot of real-time decisions it needs to make. Avoiding the pedestrian who just stepped off the curb, reacting to a red light, or slowing down for the car that wants to merge.
Planner tracks and controls the precise movements of the vehicle to make sure it travels the route accurately and safely. It relies on data from our state-of-the-art sensors to compare its actual movement against its projected trajectory and then adjusts in real time as necessary.
Choosing a way forward
Each time our vehicle makes a decision, it is looking to achieve at least five goals:
Navigate safely through its environment
Follow the rules of the road
Complete the journey
Get there efficiently
Keep the riders as comfortable as possible
At every decision point, there are multiple things our vehicle could do. To choose the best course of action, it determines one or more “costs” for each of the above categories associated with multiple possible solutions and then picks the solution with the lowest cost.
At the same time, it knows not to go past certain predefined limits no matter what. For example, driving off the drivable surface or into the wrong direction of a one-way street.
ShareRick Zhang, Senior Manager, Planner Interactions"The cost of each decision is formulated against our planned route. We compare all the possible options and choose the one that best balances safety, progress, and comfort."
Supporting our safety mission
When it’s driving, our vehicle updates its plan of action multiple times a second. But there’s a ruggedized backup system underneath that is even faster: our Collision Avoidance System (CAS).
Where Planner excels at almost all driving tasks, CAS is there for ultra-fast decisions—like hitting the brakes if someone steps out right in front of the vehicle. It constantly interacts with Planner to maintain safety and can even take over if something goes wrong with the rest of the system.
Humans use data from past experiences to inform in-the-moment decision-making when driving. Similarly, Zoox uses data to decide its path. Our Planner can build a risk profile for situations beforehand, recognize them when they occur, and choose what the data tells us is the safest response. As individual vehicles encounter more and more situations, they’re able to pass on their learnings to the entire fleet through future software releases.
Building the ultimate driver
We deploy highly trained drivers in our test vehicles to create “ideal driving” data sets for our AI to learn from. Then we use simulations to compare those professional driving miles with what our vehicle would do. By looking at the overall difference, we can close the gap between our AI and the “nominal” human driver. In closing that gap, we create an AI driver that outperforms the average human. Not to mention that even professional drivers can make mistakes, and we can remove those from the training set.
A Zoox test vehicle retrofitted with an almost-identical sensor architecture as our purpose-built robotaxi.
On top of this, the Zoox sensor architecture lets our AI see its environment in 360 degrees, using everything from infrared to radar. It has an awareness of its surroundings that humans simply don’t, letting it make more informed decisions at every turn.
ShareMarc Wimmershoff, Senior Director, Planning & Control and Autonomy Software Integration"We ask our test drivers to drive like chauffeurs. Then we take the best features of every drive and craft the data set to make a new standard. If you can match that standard, you have a superhuman driver."
You go first
As human drivers, a lot of our decisions depend on what the people around us are doing. Want to switch lanes? You start indicating. The driver behind slows down to leave space. Even if the gap is small, you move over, because you know they’re aware of your intentions.
These situations can be difficult for AI to understand. Relying on prediction alone doesn't cut it, because everyone's behavior is interdependent: what other drivers do depends on what you do and vice versa. The latest version of our Planner is closely coupled with Prediction to account for that. It doesn’t just anticipate how the environment will change, but how it will change given possible actions of our vehicle and other agents interacting.
As our vehicles drive, Planner generates options for how they can move through the world. It runs multiple times per second, generating these options each time. After the possible paths are created, Planner pieces them together to create the final trajectory. These options can be generated by either traditional motion planning techniques or from the latest machine learning approaches. This allows our system to take the best of either approach on every execution, even mixing and matching among them, in order to create the optimal plan.
This approach gives our vehicles the adaptability required to respond to unpredictable urban environments. That’s the driving philosophy behind Planner: give our vehicles the tools to prepare for every situation. Including the ones that are difficult to plan for.