Geometric Deep Learning for Point Cloud Processing

Deep learning techniques have demonstrated remarkable achievements for many computer vision tasks. This project aims to focus on the challenges and opportunities that come with deep learning applied to 3D data, especially 3D point clouds. Point clouds are a widely used 3D data representation, which is the output of many 3D sensing systems like depth sensors or laser scanners. However, point clouds are usually not directly usable, and must therefore be converted to higher-level representations. One such higher-level representation is a wireframe, i.e., a graph-structured edge or skeletal representation of an object, which makes the salient contours and their connectivity explicit. Furthermore, the conversion of point clouds into precise 3D CAD models, where the surfaces are mathematically represented, is the ultimate goal in computer-aided manufacturing applications. In order to obtain these types of compact and accurate models from raw point clouds, we need to analyze geometric features and topological relationships of point cloud data.

The goal of the project is to develop new deep learning methods for point cloud processing. Specifically, we focus on inferring structural information from raw, not grid-structured point clouds in order to best complement the current 3D object reconstruction schemes. Our long-term goal is to leap-frog multi-step, hierarchical point cloud processing, which is brittle to set up and error-prone, by directly predicting 3D CAD models from raw input point clouds.

We start by inferring a graph-structured wireframe representations (Figure.1) from raw point clouds of polyhedral objects, since many man-made objects are (approximately) polyhedral and can be described by corners and straight edges. To this end, we depart from edge point detection, which is a traditional low-level vision problem, but instead learn wireframe structures directly. In order to explore point cloud data scanned from 3D objects of arbitrary shapes, we furthermore plan to develop novel generative methods to abstract a high-level graph structure of arbitrary shape. A promising technique to solve this task are graph neural networks that are (in principle) capable of learning graphs of arbitrary structure. With this building blocks, we hope to make progress towards the final research question: how to infer comprehensive 3D CAD models from point clouds, which includes handling vertices, edges and surfaces jointly; with the ultimate goal of being amenable to applications in manufacturing, metrology, quality inspection, visualisation, animation, or rendering.