ALT-Pilot: Autonomous navigation with Language augmented Topometric maps

Mohammad Omama¹, Pranjal Paul^*2, Pranav Inani^*2, Sarat Chandra Yellapragada², Krishna Murthy Jatavallabhula^†3, Sandeep Chinchali^†1, Madhava Krishna^†2

¹The University of Texas at Austin ²International Institute of Information Technology ³Massachusetts Institute of Technology

^*Co-second Authors ^† Equal Advising

Paper arXiv Video Code (Coming Soon!)

Language Augmented Topometric Maps

Multimodal Localization

Abstract

We present an autonomous navigation system that operates without assuming HD LiDAR maps of the environment. Our system, ALT-Pilot, relies only on publicly available road network information and a sparse (and noisy) set of crowdsourced language landmarks. With the help of onboard sensors and a language-augmented topometric map, ALT-Pilot autonomously pilots the vehicle to any destination on the road network. We achieve this by leveraging vision- language models pre-trained on web-scale data to identify potential landmarks in a scene, incorporating vision-language features into the recursive Bayesian state estimation stack to generate global (route) plans, and a reactive trajectory planner and controller operating in the vehicle frame. We implement and evaluate ALT-Pilot in simulation and on a real, full-scale autonomous vehicle and report improvements over state-of- the-art topometric navigation systems by a factor of 3× on localization accuracy and 5× on goal reachability.

ALT-Pilot: Pipeline

[1] We allow users to augment topometric maps by clicking anywhere on the map and adding a language-only landmark – a brief text tag describing a distinct static object in the scene, such as a street sign or a roundabout.

[2] At the core of ALT-Pilot is a probabilistic multimodal localization algorithm that bridges LiDAR, image, and language to estimate a posterior distribution of vehicle poses in the map frame (top-right). This allows us to account for erroneously specified landmarks, and provides a global localization estimate useful for route (global) planning, which we couple with onboard sensors and perception for trajectory planning and control.

[3] From the uncertainty profile plot, notice how ALT-Pilot is more certain over the entire trajectory; compared to Maplite, a state-of-the-art topometric navigation approach for AD.