Efficient I/O: HDF5 and DLIO Workshop

Event Type

Conference/Workshop

Sponsor

NCSA

Date

Mar 12, 2024   1:00 - 3:00 pm  

Registration

Registration form

Contact

Bruno Abreu

E-Mail

babreu@illinois.edu

Views

17

Originating Calendar

NCSA Research Consulting Training Events

This workshop will include two talks about optimizing, profiling, and benchmarking input/output operations using HDF5 and DLIO.

• Optimizing Your I/O Workload: Techniques for Effective HDF5 Usage - Scot Breitenfeld, The HDF Group

HDF5 is a widely used data model, file format, and I/O library, particularly in HPC applications for managing and storing large amounts of simulation data. This talk will focus on HDF5 usage on NCSA's Delta system and provide a brief overview of HDF5 (serial and parallel) with an emphasis on HDF5 HPC tuning techniques such as collective metadata I/O, data aggregation, asynchronous I/O, and other HDF5 tuning parameters and features. During the presentation, we will discuss various storage options available on Delta for HDF5 files and multiple post-simulation storage options, including cloud storage and other tools.

• Deep Learning I/O: Benchmark and Profiling - Hariharan Devarajan, Lawrence Livermore National Laboratory

In this talk, we will be taking a dive into the unique features of I/O in deep learning application. We will present our ongoing efforts to understand the I/O characteristics of DL workloads using our DLIO profiler. Finally, we will present the DLIO Benchmark which is built to accurately represent the I/O characteristics in deep learning workloads.