Open Semester Projects and Thesis

Marigold, a recent diffusion model and associated fine-tuning protocol shows impressive results for monocular depth estimation. Its core principle is to leverage the rich visual knowledge in Stable Diffusion, and fine-tune with a small amount of synthetic data. Although showing very great results, there are many exciting future directions to explore, e.g.: How to speed up? How to adapt it for other modalities? High-resolution? How to predict depth in metrics, etc. **Requirements** - Programming in Python and Pytorch - Good knowledge of deep learning - [Preferable] have experience in diffusion models **Reference** Marigold paper: https://arxiv.org/abs/2312.02145

Develop the next generation of the SOTA monocular predictor

Songyou Peng (songyou.peng@inf.ethz.ch) Anton Obukhov (anton.obukhov@gmail.com) Bingxin Ke (bingxin.ke@geod.baug.ethz.ch)

In previous work at the Photogrammetry and Remote Sensing Group, we have made the first step to synthesize realistic LiDAR scans by optimizing a neural field scene representation. This prototype [1] opens up many directions to be explored. For example, we would like to extend it to handle dynamic scenes [2] and adverse weather. In the student project, we look into ways to exploit existing 3D assets for LiDAR view synthesis. There are at least two way to use digital object models for our purposes: (1) in real LiDAR scans only those parts of the object are captured that face the scanner, so we are missing information when rendering them from a different viewpoint. The first part of the project will investigate how to construct an a-priori model of object shape from existing 3D assets to improve reconstruction quality. (2) 3D assets can also be used to improve the diversity of simulated scans, by inserting new, dynamic objects - for instance inserting additional cars or trucks to create more complex traffic scenarios. To do this in a realistic manner, one cannot simply cast rays and intersect them with the surfaces of the new object: it is also necessary to correctly simulate point cloud noise, ray drops due to mirror reflections, attenuation etc., and multi-returns on the object boundary. The second part of the project will explore ways to directly learn these patterns from large volumes of data and to transfer them to synthetic scans. Students should be motivated, have previous experience with PyTorch, and ideally also with point cloud processing. Knowledge of generative and 3D AI tools like NeRF, GANs and diffusion models is a plus. If successful the project may lead to a publication in a reputable robotics or computer vision conference. The project is suitable for MSc project or MSc thesis. Group work (up to two people) s possible, and strongly encouraged for the MSc project. [1] Huang, Shengyu, et al. "Neural LiDAR Fields for Novel View Synthesis." ICCV 2023. [2] Huang, Shengyu, et al. “Dynamic Neural LiDAR Fields for Novel View Synthesis." to appear. [3] Manivasagam, Sivabalan, et al. "Towards Zero Domain Gap: A Comprehensive Study of Realistic LiDAR Simulation for Autonomy Testing." ICCV 2023. Settings for applications: Experience with PyTorch required Practical experience with point clouds preferred Ways to stand out: knowledge of generative and 3D AI tools like NeRF, GANs and diffusion models

1) Explore learning methods to leverage the information from existing digital 3D assets and construct a shape prior for object reconstruction from LiDAR scans 2) Develop methods to insert 3D objects into synthetic LiDAR scans by transferring scanning patterns from real scans.

Shengyu Huang, shengyu.huang@geod.baug.ethz.ch

Depth completion is a computer vision task where a scene is recorded with a sparse set of depth measurements (for example with LIDAR) and an RGB camera, and one needs to complete the unobserved pixels. Depth completion task are useful for various domains like autonomous driving, AR/VR, and remote sensing. Recently, anisotropic diffusion has proven effective for guided super-resolution, especially when combined with deep learning [1]. In this project, we plan to extend the anisotropic diffusion concept to the task of depth completion. [1] https://openaccess.thecvf.com/content/CVPR2023/html/Metzger_Guided_Depth_Super-Resolution_by_Deep_Anisotropic_Diffusion_CVPR_2023_paper.html Image credits: https://paperswithcode.com/task/depth-completion

The student is expected to (1) perform a review of the most recent prior art in the domain, (2) reproduce given code bases (inference and training), (3) propose experiment objectives and code changes, and (4) organize findings in the final report. Stretch goals include submission to a top-tier conference (CVPR, ICCV, ECCV) and organizing the proposed solution into a high-impact utility, such as a code repository or a Python package. Settings for applications Python, PyTorch, Linux shell; MSc-level knowledge of (deep) machine learning and computer vision/image analysis

Nando Metzger (nando.metzger@geod.baug.ethz.ch), Photogrammetry and Remote Sensing, ETH Zürich Alex Becker (alexander.becker@geod.baug.ethz.ch), Photogrammetry and Remote Sensing, ETH Zürich