Katrin Renz

Publications

DriveLM: Driving with Graph Visual Question Answering

Chonghao Sima*Katrin Renz*Kashyap Chitta Li Chen Hanxue Zhang Chengen Xie Ping Luo Andreas Geiger Hongyang Li

ArXiv, 2023 (Part of: CVPR 2024 Autonomous Driving Challenge)

Prior work in 3D object detection evaluates models using offline metrics like average precision since closed-loop online evaluation on the downstream driving task is costly. However, it is unclear how indicative offline results are of driving performance. In this work, we perform the first empirical evaluation measuring how predictive different detection metrics are of driving performance when detectors are integrated into a full self-driving stack. We conduct extensive experiments on urban driving in the CARLA simulator using 16 object detection models. We find that the nuScenes Detection Score has a higher correlation to driving performance than the widely used average precision metric. In addition, our results call for caution on the exclusive reliance on the emerging class of `planner-centric' metrics.

@article{Sima2023ArXiv,
                    author      = {Chonghao Sima and Katrin Renz and Kashyap Chitta and Li Chen and Hanxue Zhang and Chengen Xie and Ping Luo and Andreas Geiger and Hongyang Li},
                    title       = {DriveLM: Driving with Graph Visual Question Answering},
                    booktitle   = {arXiv preprint arXiv:2312.14150},
                    year        = {2023}
                    }

On Offline Evaluation of 3D Object Detection for Autonomous Driving

Tim SchreierKatrin RenzAndreas Geiger Kashyap Chitta

ICCVW, 2023

Paper Abstract Bibtex

@INPROCEEDINGS{Schreier2023ICCVW,
                    author      = {Tim Schreier and Katrin Renz and Andreas Geiger and Kashyap Chitta},
                    title       = {On Offline Evaluation of 3D Object Detection for Autonomous Driving},
                    booktitle   = {International Conference on Computer Vision (ICCV) Workshops},
                    year        = {2023}
                    }

PlanT: Explainable Planning Transformers via Object-Level Representations

Katrin RenzKashyap Chitta Otniel-Bogdan Mercea A. Sophia Koepke Zeynep Akata Andreas Geiger

CoRL, 2022

Paper Project page Code Abstract Bibtex

Planning an optimal route in a complex environment requires efficient reasoning about the surrounding scene. While human drivers prioritize important objects and ignore details not relevant to the decision, learning-based planners typically extract features from dense, high-dimensional grid representations of the scene containing all vehicle and road context information. In this paper, we propose PlanT, a novel approach for planning in the context of self-driving that uses a standard transformer architecture. PlanT is based on imitation learning with a compact object-level input representation. With this representation, we demonstrate that information regarding the ego vehicle’s route provides sufficient context regarding the road layout for planning. On the challenging Longest6 benchmark for CARLA, PlanT outperforms all prior methods (matching the driving score of the expert) while being 5.3× faster than equivalent pixel-based planning baselines during inference. Combining PlanT with an off-the-shelf per- ception module provides a sensor-based driving system that is more than 9 points better in terms of driving score than the existing state of the art. Furthermore, we propose an evaluation protocol to quantify the ability of planners to identify relevant objects, providing insights regarding their decision-making. Our results indicate that PlanT can reliably focus on the most relevant object in the scene, even when this object is geometrically distant.

@INPROCEEDINGS{Renz2022CORL,
                    author      = {Katrin Renz and Kashyap Chitta and Otniel-Bogdan Mercea and A. Sophia Koepke and Zeynep Akata and Andreas Geiger},
                    title       = {PlanT: Explainable Planning Transformers via Object-Level Representations},
                    booktitle   = {Conference on Robotic Learning (CoRL)},
                    year        = {2022}
                    }

KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients

Niklas HanselmannKatrin RenzKashyap Chitta Apratim Bhattacharyya Andreas Geiger

ECCV, 2022 (Oral)

Paper Project page Code Abstract Bibtex

Simulators offer the possibility of safe, low-cost development of self-driving systems. However, current driving simulators exhibit naïve behavior models for background traffic. Hand-tuned scenarios are typically added during simulation to induce safety-critical situations. An alternative approach is to adversarially perturb the background traffic trajectories. In this paper, we study this approach to safety-critical driving scenario generation using the CARLA simulator. We use a kinematic bicycle model as a proxy to the simulator's true dynamics and observe that gradients through this proxy model are sufficient for optimizing the background traffic trajectories. Based on this finding, we propose KING, which generates safety-critical driving scenarios with a 20% higher success rate than black-box optimization. By solving the scenarios generated by KING using a privileged rule-based expert algorithm, we obtain training data for an imitation learning policy. After fine-tuning on this new data, we show that the policy becomes better at avoiding collisions. Importantly, our generated data leads to reduced collisions on both held-out scenarios generated via KING as well as traditional hand-crafted scenarios, demonstrating improved robustness.

@INPROCEEDINGS{Hanselmann2022ECCV,
                    author      = {Niklas Hanselmann and Katrin Renz and Kashyap Chitta and Apratim Bhattacharyya and Andreas Geiger},
                    title       = {KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients},
                    booktitle   = {European Conference on Computer Vision (ECCV)},
                    year        = {2022}
                    }

TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving

Kashyap Chitta Aditya Prakash Bernhard Jaeger Zehao YuKatrin RenzAndreas Geiger

T-PAMI, 2022

Paper Code Abstract Bibtex

How should we integrate representations from complementary sensors for autonomous driving? Geometry-based fusion has shown promise for perception (e.g. object detection, motion forecasting). However, in the context of end-to-end driving, we find that imitation learning based on existing sensor fusion methods underperforms in complex driving scenarios with a high density of dynamic agents. Therefore, we propose TransFuser, a mechanism to integrate image and LiDAR representations using self-attention. Our approach uses transformer modules at multiple resolutions to fuse perspective view and bird's eye view feature maps. We experimentally validate its efficacy on a challenging new benchmark with long routes and dense traffic, as well as the official leaderboard of the CARLA urban driving simulator. At the time of submission, TransFuser outperforms all prior work on the CARLA leaderboard in terms of driving score by a large margin. Compared to geometry-based fusion, TransFuser reduces the average collisions per kilometer by 48%.

@article{Chitta2022ARXIV,
                    author  = {Chitta, Kashyap and Prakash, Aditya and Jaeger, Bernhard and Yu, Zehao and Renz, Katrin and Geiger, Andreas},
                    title   = {TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving},
                    journal = {arXiv},
                    volume  = {2205.15997},
                    year    = {2022}
                    }

Sign Segmentation with Changepoint-Modulated Pseudo-Labelling

Katrin RenzNicolaj C. Stache Neil Fox Gül Varol Samuel Albanie

CVPRW, 2021

Paper Project page Abstract Bibtex

The objective of this work is to find temporal boundaries between signs in continuous sign language. Motivated by the paucity of annotation available for this task, we propose a simple yet effective algorithm to improve segmentation performance on unlabelled signing footage from a domain of interest. We make the following contributions: (1) We motivate and introduce the task of source-free domain adaptation for sign language segmentation, in which labelled source data is available for an initial training phase, but is not available during adaptation. (2) We propose the Changepoint-Modulated Pseudo-Labelling (CMPL) algorithm to leverage cues from abrupt changes in motion-sensitive feature space to improve pseudo-labelling quality for adaptation. (3) We showcase the effectiveness of our approach for category-agnostic sign segmentation, transferring from the BSLCORPUS to the BSL-1K and RWTH-PHOENIX-Weather 2014 datasets, where we outperform the prior state of the art.

@INPROCEEDINGS{Renz2021signsegmentation_b,
                    "title     = {Sign Segmentation with Changepoint-Modulated Pseudo-Labelling},
                    "author    = {Katrin Renz  and Nicolaj C. Stache and Neil Fox and G{"u}l Varol and Samuel Albanie,
                    "booktitle = {CVPRW},
                    "year      = {2021}"
                    }

Sign Language Segmentation with Temporal Convolutional Networks

Katrin RenzNicolaj C. Stache Samuel Albanie Gül Varol

ICASSP, 2021

Paper Project page Abstract Bibtex

The objective of this work is to determine the location of temporal boundaries between signs in continuous sign language videos. Our approach employs 3D convolutional neural network representations with iterative temporal segment refinement to resolve ambiguities between sign boundary cues. We demonstrate the effectiveness of our approach on the BSLCORPUS, PHOENIX14 and BSL-1K datasets, showing considerable improvement over the state of the art and the ability to generalise to new signers, languages and domains.

@INPROCEEDINGS{Renz2021signsegmentation_a,
                    "title     = {Sign Language Segmentation with Temporal Convolutional Networks},
                    "author    = {Renz, Katrin and Stache, Nicolaj C. and Albanie, Samuel and Varol, G{"u}l},
                    "booktitle = {ICASSP},
                    "year      = {2021}"
                    }

Katrin Renz

News

Publications

(Co-)Mentored Students