PhD Final Defense – Shengyi Wang

Condition Assessment, Deterioration Prediction, and Network-Level Bridge Maintenance Decision Making Using Computer Vision, Multimodal Data Integration, and Knowledge Graphs

Advisor: Professor Nora El-Gohary

Abstract

The aging and deterioration of bridge infrastructure increase the need for effective inspection, condition

assessment, deterioration prediction, and maintenance decision making. Data-driven analysis of bridge

inspection and inventory data could improve the efficiency, consistency, and safety of bridge inspection

and management while helping transportation agencies allocate limited resources. However, three major

knowledge gaps remain. First, existing methods have limited capability and generalizability for extracting

bridge components, defects, and their relationships from diverse inspection images. Second, methods for

integrating multimodal bridge data, including structured data, textual inspection reports, inspection images,

traffic data, and weather data, remain limited. Third, integrated multimodal bridge data and knowledge

graphs have not been sufficiently utilized to support network-level maintenance decisions. To address these

gaps, this research develops a multimodal bridge inspection data analytics framework for knowledge graphbased

network-level bridge condition assessment, deterioration prediction, and maintenance decision

making.

To achieve these goals, the research methodology comprised six primary tasks: (1) a literature review on

related topics such as data-driven bridge condition assessment and deterioration prediction, deep learningbased

computer vision and semantic segmentation, image captioning and vision-language models, graph

neural networks for infrastructure, and ordinal regression and domain-informed time-series prediction; (2)

a vision foundation model-based semi-supervised method for pixel-level bridge component semantic

segmentation using self-distillation with no labels version 2 (DINOv2), segment anything model (SAM)

feature fusion, confidence-aware pseudo-labeling, and cost-sensitive learning; (3) a vision-language modelbased

image captioning method that leverages bootstrapping language-image pre-training (BLIP), a hybrid

convolutional neural network and transformer feature extraction framework, adaptive feature weighting,

and contrastive learning to generate free-form descriptions of bridge components, defects, and their

relationships; (4) a semantic-guided, frequency-decoupled dual-pathway method that integrates Mambabased

context modeling, invertible neural networks, and domain knowledge for multiclass bridge damage

segmentation; (5) a multimodal knowledge graph-driven graph neural network method that integrates

structured, textual, visual, and bridge-network information to assess current deck, superstructure, and

substructure condition ratings; and (6) a multimodal knowledge graph-based time-series mixture learning

method for next-cycle deterioration prediction and maintenance decision making.

The developed models were evaluated individually. The experimental results demonstrated the

effectiveness of the proposed methods across the evaluated tasks. Collectively, this research establishes an

integrated multimodal bridge analytics framework that connects automated information extraction from

inspection images, multimodal data fusion for network-level condition assessment and deterioration

prediction, and maintenance decision making. It also highlights the potential of knowledge graphs and large

vision and language models to support more informed bridge management.