This webinar included two talks about optimizing, profiling, and benchmarking input/output operations using HDF5 and DLIO.
- Optimizing Your I/O Workload: Techniques for Effective HDF5 Usage - Scot Breitenfeld, The HDF Group
- View presentation slides
HDF5 is a widely used data model, file format, and I/O library, particularly in HPC applications for managing and storing large amounts of simulation data. This talk focused on HDF5 usage on NCSA's Delta system and gave a brief overview of HDF5 (serial and parallel) with an emphasis on HDF5 HPC tuning techniques such as collective metadata I/O, data aggregation, asynchronous I/O, and other HDF5 tuning parameters and features. Various storage options available on Delta for HDF5 files and multiple post-simulation storage options, including cloud storage and other tools were also discussed.
- Deep Learning I/O: Benchmark and Profiling - Hariharan Devarajan, Lawrence Livermore National Laboratory
- View presentation slides
This talk described the unique features of I/O in deep learning applications and presented ongoing efforts to understand the I/O characteristics of DL workloads using the DLIO profiler. Finally, the DLIO Benchmark which is built to accurately represent the I/O characteristics in deep learning workloads was presented.