Projecte matriculat
Títol: Automatic lidar-camera calibration leveraging 3D and 2D object detections in highways for scenario extraction
Director/a: GONZÁLEZ ARBESU, JOSÉ MARIA
Departament: TSC
Títol: Automatic lidar-camera calibration leveraging 3D and 2D object detections in highways for scenario extraction
Data inici oferta: 05-02-2026 Data finalització oferta: 13-02-2026
Estudis d'assignació del projecte:
GR ENG TELEMÀTICA
| Tipus: Individual | |
| Lloc de realització: Fora UPC | |
| Supervisor/a extern: Marc Perez Quintana | |
| Institució/Empresa: IDIADA Technology, S.A.) | |
| Titulació del Director/a: Graduado Matemáticas e Informática | |
| Paraules clau: | |
| Object detection, lidar, computer vision, calibration techniques, point clouds, images, data fusion | |
| Descripció del contingut i pla d'activitats: | |
| IDIADA is developing an object-detection approach that fuses images and point clouds, designed to transfer to unseen domains without access to annotated target data and without assuming camera-lidar synchronization. Image-based detectors tend to generalize better across domains thanks to large-scale pretraining on diverse cameras and scenes (e.g., ImageNet). In contrast, learning-based lidar methods often generalize less effectively because they are commonly trained on a single dataset with one sensor setup and limited domain variability. Motivated by this gap, we propose leveraging the strong cross-domain generalization of 2D CNN-based image detectors and lifting their 2D outputs into 3D detections using lidar in a model-based pipeline that requires no training data and does not rely on synchronized sensors. For each incoming lidar point cloud, we run a clustering algorithm to generate 3D bounding-box proposals, then use the image modality to filter and assign classes to these proposals. Specifically, for each camera we obtain 2D detections and track IDs from the preceding and following images using YOLOv12 for detection and ByteTrack for tracking. We temporally align these tracks to the point-cloud timestamp by interpolating the 2D bounding boxes. Next, for every 3D box proposal produced by clustering, we project its 3D corners onto the image plane using the camera intrinsic matrix and the lidar-to-camera extrinsic calibration, and compute the smallest axis-aligned 2D bounding box that encloses all projected corners. Camera intrinsics are estimated with Zhang's calibration method, while the lidar-camera extrinsics are obtained via an adaptation of Velo2Cam, using circles painted with high-reflectivity paint (rather than circular holes) to localize the calibration pattern in the lidar point cloud.
Driving on highways with the current sensor mounting system leads to small variations in the lidar-camera calibration that affect the performance of the perception system. Therefore, in this work, we aim to develop an algorithm for automatic lidar-camera calibration leveraging an initial coarse calibration and recorded point clouds and images from highway field operational tests. The main idea is to start with the coarse calibration to associate projected 3D bounding boxes to 2D bounding boxes, keeping only those matches with an intersection over union (IoU) above 80%. For those matches we can then optimize the lidar-camera calibration so that it maximizes the intersection over union of the matches. This refined calibration can be used to re-generate new matches, and we can iteratively refine the lidar-camera calibration considering more matches, potentially decreasing the threshold on the IoU. Additionally, we will propose a method for Lidar-vehicle calibration, something that is not present in IDIADA's toolkit at the moment. Optionally, we might leverage this refined calibration to improve the scenario extraction pipeline by refining the multimodal object detectors or lane detectors. |
|
| Orientació a l'estudiant: | |
| Horari d'atenció a estudiants per a l'assignació de projecte: |
|