Merge pull request #704 in HDFFV/hdf5 from optimize_superblock_read to develop

* commit '3dde6d0e32461f46630f814a2fdfbd4c813703bf':
  Updated the code and RELEASE.txt note per comments from John Mainzer
  Try to address most of the issues raised by Dana in the code review
  Made edits suggested by John as part of the code review
  Update the MANIFEST and release_docs/RELEASE files
  Add a test for parallel reads of independent files using MPI subgroups
  Edits to the file open optimization and associated test code to bring them closer to the HDF5 library's unwritten coding standards.  Also bug fix to repair a hang in testphdf5.
  The initial coding for the superblock read optization
This commit is contained in:
Richard Warren
2017-10-13 07:58:46 -05:00
6 changed files with 1022 additions and 44 deletions

View File

@@ -62,6 +62,31 @@ New Features
Parallel Library:
-----------------
- Optimize parallel open/location of the HDF5 super-block
Previous releases of PHDF5 required all parallel ranks to
search for the HDF5 superblock signature when opening the
file. As this is accomplished more or less as a synchronous
operation, a large number of processes can experience a
slowdown in the file open due to filesystem contention.
As a first step in improving the startup/file-open performance,
we allow MPI rank 0 of the associated MPI communicator to locate
the base offset of the super-block and then broadcast that result
to the remaining ranks in the parallel group. Note that this
approach is utilized ONLY during file opens which employ the MPIO
file driver in HDF5 by previously having called H5Pset_fapl_mpio().
HDF5 parallel file operations which do not employ multiple ranks
e.g. specifiying MPI_COMM_SELF (whose MPI_Comm_size == 1)
as opposed to MPI_COMM_WORLD, will not be affected by this
optimization. Conversely, parallel file operations on subgroups
of MPI_COMM_WORLD are allowed to be run in parallel with each
subgroup operating as an independant collection of processes.
(RAW 2017/10/10, HDFFV-10294)
- Large MPI-IO transfers
Previous releases of PHDF5 would fail when attempting to