CEE Seminars and Conferences

View Full Calendar

PhD Final Defense for Jiayi Luo

Event Type
Civil and Environmental Engineering
Newmark Quade Conference Room
Nov 30, 2023   1:00 pm  

Ballast Condition and Degradation Evaluation using Computer Vision Techniques: Algorithms and Applications

Advisor: Professor Erol Tutumluer


Railroads serve as one of the most efficient means of transportation within the U.S. The track

substructure plays a significant and irreplaceable role in the overall performance of a railway

track system in response to repeated train loading. A majority of the railroad structures are

ballasted tracks, which may contain up to six basic components including rail, crossties, ballast,

subballast, subgrade, and embankment. Among these, railroad ballast stands as a key component

of the railroad track substructure. It is comprised of large, uniformly graded, and fully crushed

coarse aggregate materials placed between and immediately underneath the crossties, providing

drainage and structural support for the track system. Ballast degradation denotes the process

where the material deteriorates as the voids in unbound aggregate layers are filled with relatively

finer materials, or fouling agents, commonly resulting from the breakdown of ballast,

contamination, and subgrade soil intrusion. As the ballast ages, it is progressively subjected to

fouling and degradation caused by particle breakage and abrasion, thus leading to poor drainage,

rapid and excessive settlement, and reduced lateral stability. To measure ballast fouling and

degradation levels, and correlate it with ballast and track performance, several ballast fouling

indices have been proposed. These indices fall into different categories: weight-based, volumebased

and imaging-based. Depending on the specific analysis scenarios, different indices should

be employed to provide accurate estimation of the ballast condition.

Current ballast evaluation methods mainly rely on visual inspections, laboratory analysis of

sampled ballast, and Ground Penetrating Radar (GPR) techniques. While visual inspections are

subjective and sample collection has its limitations, GPR has been effective in assessing ballast

conditions to some extent but falls short in detailed geotechnical analysis. Over the years, field

inspection systems have evolved, with vision-based methods being developed for analyzing track

conditions, but these primarily focus on track superstructure components, lacking in substructure

evaluations like ballast fouling evaluation. Non-vision methods like GPR and Light Detection

and Ranging (LiDAR) have been used for substructure inspections; however, they lack detailed

geotechnical information about the ballast, accompanied by various resolution concerns and


This doctoral research study focused on developing a novel ballast evaluation system by

leveraging deep learning-based computer vision techniques to evaluate ballast condition and

degradation level in the field. The proposed system entails the development of both software and

hardware components to facilitate automated field ballast data collection and in-depth ballast

condition evaluation. An imaging-based index, Percent Degraded Segments (PDS), is introduced

to correlate image segmentation results with ballast degradation indicators, primarily Fouling

Index (FI). The deep learning algorithms are developed and implemented, specifically Mask

Region-based Convolutional Neural Network (Mask R-CNN), for precise ballast instance

segmentation and establishing regression models to link PDS and FI. An automated Ballast

Scanning Vehicle (BSV) is designed and constructed to acquire high-quality field data during

operation, which is then analyzed using a companion data analysis software, I-BALLAST, for

streamlining the evaluation procedure and minimizing user-dependency. Model improvements

are pursued through (i) exploring advanced deep learning architectures like Vision Transformer,

and (ii) a semi-supervised learning framework, Ballast Semi-Supervised Framework (BSSF), to

enhance ballast segmentation performance. Several field experiments have been conducted

during the development and validation of the ballast evaluation system. Two field experiments,

in conjunction with Ballast Shoulder Cleaner (SBC), in Indiana and Ohio Norfolk Southern (NS)

railroads, have been conducted to test the functionalities of the developed BSV and collect field

data. The fully developed ballast evaluation system, after integrating the BSV and I-BALLAST,

is successfully validated through the last field experiment at the High Tonnage Loop (HTL) in

the Transportation Technology Center (TTC), demonstrating its potential to provide accurate and

robust ballast condition evaluations and geometric analyses, crucial for optimizing rail

maintenance and rehabilitation planning on-site. In the longer term, the fully developed ballast

evaluation system could serve as the key component of a comprehensive Ballast Management

System (BMS) to study the deterioration mechanisms and improvement of ballasted track

designs and ballast maintenance practices.

link for robots only