2024.08–Present: Deep Learning-Based Semantic Segmentation of LiDAR Point Clouds for Civil Infrastructure

Project Summary

This research investigates supervised deep learning methods for the semantic segmentation of high-resolution LiDAR point clouds, specifically targeting horizontal civil infrastructure such as pavements and bridges. Traditional workflows rely heavily on manual classification, making the process labor-intensive, error-prone, and incompatible with the demands of large-scale digital infrastructure management. This project addresses those challenges by developing advanced Deep Learning-powered tools that significantly reduce human workload while improving accuracy and repeatability.

The approach integrates 3D semantic segmentation models with 2D image-based classification using multi-view projections. We combine digital models, annotated datasets, and modern computer vision libraries (e.g., OpenCV, TorchVision, CLIP) to evaluate both 3D native and 2D surrogate techniques. Furthermore, we investigate the effect of fusing RGB information from photogrammetric imagery with LiDAR geometry, enhancing segmentation clarity and aiding in the downstream integration with civil engineering tools (e.g., BIM, FEM).

Objectives

• Develop Deep Learning-based classification tools for automated segmentation of structural elements in LiDAR point clouds.
• Integrate RGB-augmented multi-view projections for scene-level classification.
• Build supervised learning pipelines using civil-specific annotations and ground truth datasets.
• Quantify the impact of RGB and geometric fusion in improving classification accuracy.
• Prepare guidelines and workflows for seamless adoption in digital twin environments.

Key Results (In Progress)

• Built preliminary datasets combining RGB-colored LiDAR scans from terrestrial and aerial sources.
• Implemented dual-mode training pipelines using 2D semantic segmentation (e.g., Mask2Former, DeepLabV3+) and 3D point-wise networks (e.g., PointNet++, SPFormer).
• Designed a benchmarking suite to test the scalability and generalization of segmentation models across diverse structural geometries.
• Created prototype software tools to generate structure-specific digital twins from raw point clouds.

Related Publications

Yan, X., Fascetti, A.* (in prep.). Review and Experimental Evaluation of Deep Learning-Based 3D Point Cloud Scene Understanding for Civil Infrastructure. Target journal: Automation in Construction.

Funding

This research is funded through the Improved Infrastructure Systems and Evaluation (IRISE) Consortium under the Year 4 and Year 6 Research Programs at the University of Pittsburgh, with additional support from the Pennsylvania Department of Transportation (PennDOT).

Visual Summary