The HDF Group
Parallel HDF5 Quincey Koziol
[email protected] The HDF Group CScADS’11 July 25, 2011 July 25, 2011
CScADS'11
1
www.hdfgroup.org
What is HDF? • HDF stands for Hierarchical Data Format • A file format for managing any kind of data • Software system to manage data in the format • Designed for high volume or complex data • Designed for every size and type of system • Open format and software library, tools • There are two HDF s: HDF4 and HDF5 • Today we focus on HDF5 July 25, 2011
CScADS'11
2
www.hdfgroup.org
Brief History of HDF 1987
At NCSA (University of Illinois), a task force formed to create an architecture-independent format and library: AEHOO (All Encompassing Hierarchical Object Oriented format) Became HDF
Early NASA adopted HDF for Earth Observing System project 1990 s 1996
DOE s ASC (Advanced Simulation and Computing) Project began collaborating with the HDF group (NCSA) to create Big HDF (Increase in computing power of DOE systems at LLNL, LANL and Sandia National labs, required bigger, more complex data files). Big HDF became HDF5.
1998
HDF5 was released with support from National Labs, NASA, NCSA
2006
The HDF Group spun off from University of Illinois as non-profit corporation
July 25, 2011
CScADS'11
3
www.hdfgroup.org
The HDF Group • Established in 1988 • 18 years at University of Illinois’ National Center for Supercomputing Applications • 5 years as independent non-profit company, “The HDF Group”
• The HDF Group owns HDF4 and HDF5 • Basic HDF4 and HDF5 formats, libraries, and tools are open and free
• Currently employ 36 FTEs
July 25, 2011
CScADS'11
4
www.hdfgroup.org
Goals of The HDF Group • Maintain and evolve HDF for sponsors and communities that depend on it • Provide support to the HDF communities through consulting, training, tuning, development, research • Sustain the company for the long term to assure data access over time July 25, 2011
CScADS'11
5
www.hdfgroup.org
HDF5 Philosophy A single platform with multiple uses • One general format • One library, with • Options to adapt I/O and storage to data needs • Layers on top and below
• Ability to interact well with other technologies • Attention to past, present, future compatibility July 25, 2011
CScADS'11
6
www.hdfgroup.org
HDF5 Data Model • Groups – provide structure among objects • Datasets – where the primary data goes • Data arrays • Rich set of datatype options • Flexible, efficient storage and I/O
• Attributes, for metadata
Everything else is built essentially from these parts. July 25, 2011
CScADS'11
7
www.hdfgroup.org
Structures to organize objects Groups /
(root)
3-‐D array
/foo lat | lon | temp -‐-‐-‐-‐|-‐-‐-‐-‐-‐|-‐-‐-‐-‐-‐ 12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6
palette
Table
Raster image Raster image
Datasets July 25, 2011
CScADS'11
2-‐D array
8
www.hdfgroup.org
Users of HDF5 Software Most data consumers are here. ScienEfic/engineering applicaEons. Domain-‐specific libraries/API, tools.
Tools & Applications HDF5 Application Programming Interface Virtual file layer (VFL)
ApplicaEons, tools use this API to create, read, write, query, etc. Power users (consumers) Modules to adapt I/O to specific features of system, or do I/O in some special way.
File system, MPI-‐IO, SAN, other layers File could be on parallel system, in memory, collecEon of files, etc.
HDF5 File July 25, 2011
CScADS'11
9
www.hdfgroup.org
Layers – parallel example Application
I/O flows through many layers from application to disk.
Parallel computing system (Linux cluster) Compute
node
Compute
Compute
Compute
node
node
node
I/O library (HDF5) Parallel I/O library (MPI-I/O) Parallel file system (GPFS) Switch network/I/O servers Disk architecture & layout of data on disk
July 25, 2011
CScADS'11
10
www.hdfgroup.org
Parallel HDF5 Now • New DOE Funding: • “ExaHDF5” – Project w/LBNL & PNL to enhance HDF5 and aim for exascale platforms • “Scalable HDF5” – Contract w/LLNL to enhance HDF5 and explore high-performance, non-MPI-I/O solutions • “Damsel” – Project w/ANL, NWU & ORNL to design and implement a next generation file format and I/O middleware package
July 25, 2011
CScADS'11
11
www.hdfgroup.org
ExaHDF5 Tasks • Remove “collective” restriction for metadata modifications (if possible)* • Including supporting compressed datasets
• Add metadata and raw data indexing to HDF5 • Add support for asynchronous parallel I/O • Design and implement file system autotuning mechanism • Support “ordered updates” in parallel July 25, 2011
CScADS'11
12
www.hdfgroup.org
Single-Writer/Multiple-Reader Access • Situation: A long-running process is modifying an HDF5 file and simultaneously other processes want to inspect data in the file. • Solution: Single-Writer/Multiple-Reader (SWMR) File Access, using “ordered updates” • Allows simultaneous reading of HDF5 file while the file is being modified by another process • No inter-process coordination necessary
• Bonus! Crash-proofs file also!
July 25, 2011
CScADS'11
13
www.hdfgroup.org
Scalable HDF5 Tasks • Explore and implement alternate scalable I/O approaches: • “Poor man’s parallel I/O” (PMPIO) (from LLNL) • “Reduced-Blocking I/O” (rbIO) (from ANL)
• Design new Virtual File Drivers tuned for “modern” parallel file systems • Metadata aggregation & alignment in file • Advanced page buffering within library • Deferred/staged/segregated object creation
July 25, 2011
CScADS'11
14
www.hdfgroup.org
Other Planned HDF5 Tasks • Design and implement “Virtual Object Layer” within HDF5 • Allows creation of plugins operating at higherlevel of abstraction that Virtual File Layer • HDF5 data model, without using HDF5 file • Can we merge HDF5 with [parallel] file system?
• Expand HDF5 data model • Support “shared” dataspaces • Attributes on datatypes and dataspaces (allows units on datatypes, etc.)
• “Append-only” library and file format optimizations July 25, 2011
CScADS'11
15
www.hdfgroup.org
Parallel HDF5 Challenges • We are implementing file system on top of MPI-I/O! • Not enough support in MPI for necessary locking operations, etc. • Difficult to create production-quality software in a portable and cost-effective way
• Need more funding • Support and reach out to HPC application development teams • Keep up with research efforts: ADIOS, pnetCDF, etc. July 25, 2011
CScADS'11
16
www.hdfgroup.org
