Automated Field Compliance Checking for Improved Construction Safety using Natural Language Processing and Computer Vision
Advisor: Professor Nora El-Gohary
Co-Advisor: Professor Mani Golparvar-Fard
Construction safety regulations and standards contain a massive number of fall protection requirements with respect to
different equipment, facilities, and operations. A large number of site accidents occur due to field noncompliance with
such regulations and standards, particularly the Occupational Safety and Health Administration (OSHA) regulations.
However, current jobsite safety checking practices rely heavily on manual inspection to identify violations. To reduce the
time, cost, and errors of such a manual and exhaustive process, there is a need for automated field compliance checking,
which seeks to automate the process of extracting safety requirements from OSHA, capturing site information, and
identifying violations. To address this need, this thesis proposes a new deep learning-based method that utilizes natural
language processing (NLP) and computer vision (CV) techniques to detect field noncompliances to OSHA, with minimum
human assistance. First, a set of computational models and NLP methods to automatically extract requirements from
safety regulations are developed, including entity extraction, relation extraction, and knowledge graph-based query
representation. Second, a set of computational models and CV methods to automatically detect site condition information
from site images are developed, including object detection, attribute recognition, visual relation detection, and scene graph
representation. Third, the detected site condition information is compared with applicable safety requirements, using
knowledge graph-based reasoning, to detect noncompliances and generate documentation.
To achieve the aforementioned goals, the research methodology included seven primary research tasks. First, conducting
a comprehensive literature review on related topics such as automated compliance checking software and research efforts
as well as machine learning models and methods for information extraction, object detection, attribute recognition, visual
relation detection, and generative models. Second, developing a machine learning-based approach to extract entities that
describe fall protection requirements from OSHA. Third, developing a machine learning-based method to extract relations
that describe fall protection requirements from OSHA and represent the extracted requirements as query graphs. Fourth,
developing a machine learning-based method to identify fall-related site objects, with multiple attributes, to provide
intricate site condition information. Fifth, developing a machine learning-based method to detect visual relations among
fall-related site objects and generate scene graphs to represent the detected site information in a similar structure to the
query graphs. Sixth, developing a machine learning-based method to retrieve applicable safety clauses according to the
site scenes and conduct knowledge graph-based compliance reasoning. And seventh, integrating the developed
computational models and methods into a prototype system and comparing the proposed approach with a general end-toend
approach that utilizes multimodal foundation models.
The developed computational models and methods for automated field compliance checking, including regulatory
information extraction and representation, site information detection and representation, as well as graph-based clause
classification and automated reasoning were individually tested, each achieving over 80.0% in precision and recall. The
final integrated automated field compliance checking prototype system was evaluated using a set of fall-related site images,
as well as fall-related requirements from OSHA. The prototype showed effective performance, achieving an average
precision of 71.2%, recall of 73.9%, and F-1 measure of 72.5%, respectively, for noncompliance detection. The generated
documentation showed reasonable chain-of-thoughts, precise summarization, and proper suggestions of corrective
measures. These experimental results demonstrate the good potential of the proposed automated field compliance
checking approach.