4.1 KiB
VFD SWMR Punch List
This document lives at https://bitbucket.hdfgroup.org/users/dyoung/repos/vchoi_fork/browse/doc/VFD_SWMR_Punch_List.md?at=refs%2Fheads%2Ffeature%2Fvfd_swmr
19 November 2019
-
Design optimizations in index writes. Perhaps write deltas only, with a full index every n ticks?
-
Design work for NFS and object store versions.
-
Design work for journaling variation.
-
Add support for opening multiple files in either VFD SWMR writer or VFD SWMR writer mode. See addition of the EOT queue in section 3.2.2 of the RFC, and related changes in sections 3.3 and 3.3.2.
-
Add the pb_expansion_threshold field to the H5F_vfd_swmr_config_t structure, and update H5Pset_vfd_swmr_config()and H5Pget_vfd_swmr_config()accordingly.
-
Implement end tick now API call. See section 3.1.2 of the RFC for specifications.
-
Implement enable / disable EOT call. See section 3.1.3 of the RFC for specifications.
-
Implement option to flush raw data as part of EOT -- write performance test to characterize the performance hit.
-
Move VFD SWMR specific H5F code to its own file -- say H5Fvfd_swmr.c.
-
Update the new page buffer to support the pb_expansion_threshold, and trigger an early end of tick if the threshold is exceeded.
-
Modify metadata file write call to allow the location of the index to float and thus be of arbitrary size.
-
Add code to remove entries from the index after they have been written to the HDF5 file, and have not been modified for at least max_lag ticks.
-
Tidy short cuts in the initial implementation. These include:
-
Odd behavior in the superblock refresh routine (see comments in code). Figure out what is going on, and then either bypass the issue or fix it as seems appropriate.
-
Comment H5F_vfd_swmr_config_t in H5Fpublic.h properly.
-
Cleanup EOA hack in H5FD_read().
-
Address file open failures in SWMR tests. These appear to be cases in which the writer finishes before the reader opens -- causing the reader to fail as the metadata file no longer exists. If so, handle this more gracefully.
-
-
Add support for specifying the VFD that sits under the VFD SWMR reader VFD. Probably do this as part of the pluggable VFD design that Jake is working on.
-
Modify the metadata cache so that we don't allocate space for the page hash table unless the file is opened in VFD SWMR reader mode.
-
Implement the logging facility (section 3.14 RFC)
-
Test code to expose existing page buffer bugs and fix same. Note that they seem to appear primarily on Jelly.
-
DLL pre remove sanity check assertion failure
-
vfd_swmr_addrem_writer: H5PB.c:2981: H5PB__mark_entry_dirty: Assertion '(entry_ptr)->delay_write_until > (pb_ptr)->cur_tick' failed.
-
-
Flesh out designs for unit, integration and performance tests suites as outlined in the RFC. Implement same.
-
Fix memory leak in sparse-reader test.
-
Test John's patch that repairs the superblock flags mismatch that crashes the reader.
-
Investigate a potential time-of-check, time-of-use race condition involving EOA/EOF and the skip_read variable in some of the H5PB routines.
-
Understand use of H5F_t on branch feature/vfd_swmr instead of H5C_t as on develop branch.
-
Temporarily reserve a new superblock flag for VFD SWMR for development purposes.
-
Prior to feature merge, permanently reserve a new superblock flag.
-
Revisit the global heap. The global heap receives data types (a form of metadata) and variable-length (VL) data such as strings. It is translated to raw data on its way down the HDF5 software stack. If the global heap is modified/replaced as currently planned, then VFD SWMR does not have to deal with it. However, if the global heap overhaul does not take place, then we have more work to do.