CBL - Campus del Baix Llobregat

Projecte llegit

Títol: Calibración automática entre LiDAR y cámara basada en detecciones de objetos 3D y 2D

Estudiants que han llegit aquest projecte:

Director/a: GONZÁLEZ ARBESU, JOSÉ MARIA

Departament: TSC

Títol: Calibración automática entre LiDAR y cámara basada en detecciones de objetos 3D y 2D

Data inici oferta: 05-02-2026     Data finalització oferta: 13-02-2026


Estudis d'assignació del projecte:
    GR ENG TELEMÀTICA
Tipus: Individual
 
Lloc de realització: Fora UPC    
 
        Supervisor/a extern: Marc Perez Quintana
        Institució/Empresa: IDIADA Technology, S.A.
        Titulació del Director/a: Graduado Matemáticas e Informática
 
Paraules clau:
Object detection, lidar, computer vision, calibration techniques, point clouds, images, data fusion
 
Descripció del contingut i pla d'activitats:
IDIADA is developing an object-detection approach that fuses images and point clouds, designed to transfer to unseen domains without access to annotated target data and without assuming camera-lidar synchronization. Image-based detectors tend to generalize better across domains thanks to large-scale pretraining on diverse cameras and scenes (e.g., ImageNet). In contrast, learning-based lidar methods often generalize less effectively because they are commonly trained on a single dataset with one sensor setup and limited domain variability. Motivated by this gap, we propose leveraging the strong cross-domain generalization of 2D CNN-based image detectors and lifting their 2D outputs into 3D detections using lidar in a model-based pipeline that requires no training data and does not rely on synchronized sensors. For each incoming lidar point cloud, we run a clustering algorithm to generate 3D bounding-box proposals, then use the image modality to filter and assign classes to these proposals. Specifically, for each camera we obtain 2D detections and track IDs from the preceding and following images using YOLOv12 for detection and ByteTrack for tracking. We temporally align these tracks to the point-cloud timestamp by interpolating the 2D bounding boxes. Next, for every 3D box proposal produced by clustering, we project its 3D corners onto the image plane using the camera intrinsic matrix and the lidar-to-camera extrinsic calibration, and compute the smallest axis-aligned 2D bounding box that encloses all projected corners. Camera intrinsics are estimated with Zhang's calibration method, while the lidar-camera extrinsics are obtained via an adaptation of Velo2Cam, using circles painted with high-reflectivity paint (rather than circular holes) to localize the calibration pattern in the lidar point cloud.

Driving on highways with the current sensor mounting system leads to small variations in the lidar-camera calibration that affect the performance of the perception system. Therefore, in this work, we aim to develop an algorithm for automatic lidar-camera calibration leveraging an initial coarse calibration and recorded point clouds and images from highway field operational tests. The main idea is to start with the coarse calibration to associate projected 3D bounding boxes to 2D bounding boxes, keeping only those matches with an intersection over union (IoU) above 80%. For those matches we can then optimize the lidar-camera calibration so that it maximizes the intersection over union of the matches. This refined calibration can be used to re-generate new matches, and we can iteratively refine the lidar-camera calibration considering more matches, potentially decreasing the threshold on the IoU. Additionally, we will propose a method for Lidar-vehicle calibration, something that is not present in IDIADA's toolkit at the moment. Optionally, we might leverage this refined calibration to improve the scenario extraction pipeline by refining the multimodal object detectors or lane detectors.
 
Overview (resum en anglès):
This Bachelor's Thesis studies the refinement of the extrinsic calibration between a LiDAR and a camera for autonomous driving and advanced driver-assistance systems. The camera and the LiDAR provide complementary information about the environment. However, combining the information from both sensors correctly requires accurate knowledge of the transformation between their coordinate systems. During driving, vibrations or small changes in the sensor positions may cause calibration errors and affect the projection of 3D objects onto the image.

This work does not estimate the extrinsic calibration from zero. Instead, it proposes a refinement method based on objects detected in driving scenes. The intrinsic parameters of the camera are obtained using the camera calibration tool provided by Autoware. The initial extrinsic calibration is calculated through an adaptation of Velo2Cam, using ArUco markers and highly reflective circles. The LiDAR point cloud is then filtered and clustered to generate 3D bounding boxes, while YOLOv12 and ByteTrack provide the 2D detections and object trajectories. To compensate for the lack of synchronization, the 2D boxes are temporally interpolated to the timestamp of each point cloud.

The 3D boxes are projected onto the image and associated with the 2D detections using Intersection over Union (IoU). Based on these associations, the optimizer modifies the three rotation parameters and the three translation parameters to increase the average overlap between the boxes. The updated transformation can then be used to generate new associations and repeat the process iteratively.

Two optimization strategies are compared. The first uses IPOPT with a smoothed objective function, which makes gradient-based optimization possible. The second uses CMA-ES through Optuna and performs the search without gradients. The evaluation is carried out in a synthetic environment with known reference extrinsic parameters and with real data collected by a vehicle equipped with a LiDAR and a camera. The experiments study the recovery of perturbations, the influence of errors in the 2D and 3D boxes, and the computational cost.

The results show that the procedure can improve the alignment between the 2D detections and the projected 3D boxes when the initial calibration is sufficiently accurate and the associations are reliable. However, a higher IoU does not always mean that the real extrinsic calibration has been recovered accurately, because different combinations of rotation and translation may produce similar projections. Therefore, the method is suitable as a tool for calibration refinement and detecting processing errors, but it cannot replace an initial calibration based on accurate geometric correspondences.

© CBLTIC Campus del Baix Llobregat - UPC