COLLOQUIUM: Phuong Cao, "Dependable AI Systems on Supercomputers: Measuring GPU Failures and Preempting Attacks"

Refreshments Provided.

Abstract:
AI training and scientific computing now run on shared GPU supercomputers with many accelerators, tenants, and network entry points; Delta and DeltaAI are the most requested GPU resources for AI/ML research in the NSF ACCESS program. An AI model or scientific result is as dependable as the GPUs that compute it and the shared software stack (login nodes, schedulers, and networks) that hosts it. Production monitoring makes these systems a longitudinal measurement platform that few, if any, research labs can offer: i) GPU resilience traces spanning GPU generations, from the Blue Waters supercomputer to the 1,056 A100 and H100 GPUs across NCSA's Delta and DeltaAI, where 11.7 million GPU hours revealed 3.2x lower per-GPU mean time between uncorrectable memory errors on H100 than on A100; and ii) security incident logs spanning 1999 to the present, alongside an SSH honeypot that has captured 15 billion attack attempts. This talk uses production supercomputers as a laboratory to build failure- and attack-resilient AI systems, drawing on a security data lake embedded in a security operations center for live evaluations of attacks and faults:

1. Measuring GPU failures: How do GPU hardware errors propagate into AI training and scientific jobs? A measurement study of Hopper H100 and Ampere A100 GPUs in the Delta production supercomputer characterizes how these errors affect jobs, with implications for fault-tolerant training and job recovery.

2. Preempting attacks: Can attacks on shared supercomputers be stopped before they cause damage? Measurements of attack attempts reveal repeated patterns and build prior knowledge for factor graph-based detection models to preempt multi-stage attacks during their initial phases.

Future work will anticipate failure-induced attacks: Can attackers exploit or trigger GPU failures, from a single faulty GPU to a system-wide blackout? Correlating GPU error logs with security incident records will separate malicious faults from benign ones, building on reliability and security monitoring with hardware architectural invariants.

Bio:
Phuong Cao is a Research Scientist at the National Center for Supercomputing Applications (NCSA). Cao’s research secures HPC cyberinfrastructure against accidental failures and intentional cyberattacks; deployed in NCSA production systems, it protects open-science workloads from GPU disruptions and attacks using real-world measurements from NSF Mid-scale research infrastructure (FABRIC testbed) and DOE labs. His research has been supported by several awards from the NSF Office of the Director and the Office of Advanced Cyberinfrastructure. His collaborations have produced publications at Supercomputing, USENIX Security, NSDI, CoNEXT, and DSN (Best Paper Award). Cao is an NSF Trusted CI Fellow and NCSA’s Deputy Executive Director of the Joint Laboratory for Extreme Scale Computing (JLESC), an international, virtual organization dedicated to fostering collaborative research in high-performance computing (HPC). He holds a Ph.D. in Electrical and Computer Engineering from the University of Illinois.

Part of the Siebel School Speakers Series. Faculty Host: Klara Nahrstedt

Meeting ID: 813 1197 3737
Passcode: csillinois

If accommodation is required, please email <erink@illinois.edu> or <communications@cs.illinois.edu>. Someone from our staff will contact you to discuss your specific needs