CBL - Campus del Baix Llobregat

Projecte llegit

Títol: Generative-AI-based pipeline for small-UAV-based 3D reconstruction of static objects

Estudiants que han llegit aquest projecte:

Director/a: ROYO CHIC, PABLO

Departament: DAC

Títol: Generative-AI-based pipeline for small-UAV-based 3D reconstruction of static objects

Data inici oferta: 25-01-2024     Data finalització oferta: 25-09-2024


Estudis d'assignació del projecte:
    DG ENG AERO/TELEMÀT
Tipus: Individual
 
Lloc de realització: Fora UPC    
 
        Supervisor/a extern: Filip Lemic
        Institució/Empresa: I2Cat
        Titulació del Director/a: Senior researcher
 
Nom del segon director/a (UPC): Sergi Abadal
Departament 2n director/a:
 
Paraules clau:
NERF, Gaussian Splatting, 3D, drones, reconstruction, position
 
Descripció del contingut i pla d'activitats:
This BSc thesis targets the design, implementation, and evaluation of NeRF-based pipeline for 3D digital reconstruction of static objects using their pictures taken by a fleet of small Unmanned Aerial Vehicles (UAVs). For that purpose, a NeRF will be designed, implemented, and trained for enabling the 3D reconstruction functionality. Experimental assessment using an existing small drone setup will be performed to benchmark the solution against the existing baseline. The developed pipeline is envisioned to eventually be contributed to the existing small UAV setup for enhancing its performance and functionalities.

The project will have 3 phases:

i) The first phase will focus on the design and development of a NERF to be applied for 3D object reconstruction.

ii) The second phase will focus on the training, testing, and optimization of the proposed NeRF-based object reconstruction pipeline.

iii) The third phase of the project the NeRF-based pipeline will be integrated with the existing small UAV setup and its reconstruction performance will be assessed experimentally and compared against an existing baseline based on the Structure-from-Motion (SfM) algorithm. Within this phase, the applicability of the NeRF-based baseline for anomaly detection in 3D static objects will also be assessed and demonstrated.
 
Overview (resum en anglès):
Unmanned aerial vehicles (UAVs) are becoming more and more common, and this trend is predicted to continue in the coming years. Although there are many uses for UAVs, this bachelor's thesis will focus on the three-dimensional (3D) digital reconstruction of still real-world objects. On the one hand, there are UAVs that are able to obtain 3D reconstruction of environments and large objects using complex systems like LiDAR (Light Detection and Ranging) sensors or cameras with high resolution and quality. On the other hand, there are UAVs that are used to enable 3D reconstructions inside houses or structures using the aforementioned systems or more complex ones. In this work, the focus will be on testing the capabilities of UAVs with limited resources and smaller than the rest (in this case the smallest UAV on the market will be tested).

Using Structure from Motion (SfM) systems and the support of Generative AI approaches, the aim of this thesis is to prove that, with limited capabilities, a coherent and precise 3D reconstruction is possible. The focus will be on proving that low-capability UAVs can be competent enough to habilitate anomaly detection on small objects. This will enable new applications to the UAVs on zones that are of hard reachability. These smaller UAVs can also be used to obtain 3D reconstructions of structures easily with UAVs that have less economical cost to the end user, generating a pipeline for obtaining the whole reconstruction.

The Bitcraze framework's autonomous positioning feature for the UAV had to be enabled before the experiments could begin. For this, an anchor-based LoCo positioning system from Bitcraze had to be set up in the testing environment. The objective was to have the UAV (a Crazyflie 2.1) follow a static trajectory, capturing images of an object that is at the centre of this path. These tests considered different scenarios.

After that, the Generative Artificial Intelligence (GenAI) pipeline was used. For using this pipeline, three approaches were used: Nerfacto, Instant-ngp and Splatfacto. On one hand, these 3 approaches were compared using 1 UAV, and after that they were compared using 2 UAVs. Here it is important to say that Colmap, another SfM tool, was used for the processing of the data. On the other hand, after that, the positions obtained by the UAV at the moment of taking the images were used to later align the estimated positions by Colmap. All of this was tested to see which is the best model, see if the corrected positions helped with the 3D reconstruction, and finally, but most importantly, to see if, with these 3D reconstructions, it is possible to enable the anomaly detection of small objects feature for small UAVs with limited capabilities. Finally, some results of performance of the mesh extraction, point cloud generation and of the render quality of the different models. These were done and successfully made, fullfiling the objectives and generate new challenges for future works.

© CBLTIC Campus del Baix Llobregat - UPC