had finished its work and closed the .h5 file, thus removing the shadow file.
Make the sparse writer wait to close the .h5 file for a signal from
testvfdswmr.sh. In testvfdswmr.sh, send the signal when the readers have all
finished.
cells in a matrix in an arbitrary order, first it chooses a random
starting `offset` in [0, rows * columns - 1]. Then it chooses a
random `increment` that's relatively prime to `rows * columns`.
Then it visits every cell in `rows * columns` steps:
for (i = 0; i < rows * columns; i++) {
visit(cell[offset / columns][offset % columns]);
offset = (increment + offset) % (rows * columns);
}
By moving the HDrandom() calls outside of the main loop and visiting
each cell only once, this probably speeds things up quite a bit. It's
also more resilient to a crummy random sequence. The new code visits
cells in an order that's probably arbitrary enough for testing purposes.
to end with whitespace padding rather than newlines. My introduction of
variadic TESTING() got rid of the padding. I have straightened this out
by newline-terminating the stdout lines in the test program and in its
expected out. I also add some newlines to the program's standard error
output so that the expected error output still matches.
comment to myself that I need to reduce code duplication with the MPMDE
test.
In vfd_read_each_equals(), print the correct expected value when there
is a discrepancy.
No functional change intended: correct a comment in
vfd_read_each_equals. Fix indentation in the test_raw_data_handling()
header comment.
buffer's treatment of multipage metadata entries (MPMDEs). Mention why
an H5PB_flush() is not necessary for MPMDEs to reach visibility at the
VFD layer.
macro magic. Use namebases and namebase, instead.
Extract a bunch of copy-and-paste VFD SWMR setup into a new subroutine,
swmr_fapl_augment().
Make sure that the metadata reads all-0s until it reads all -1s.
Extract a subroutine, vfd_read_each_equals(), that reads and compares a
region with one of its arguments.
Rename from test_basic_metadata_handling() to
test_metadata_delay_basic(), since that gets at what we're testing
better.
Don't perform an H5PB_flush(), it's not necessary for this test because
H5Fvfd_swmr_end_tick() has done essentially the same thing.
anticipate comparing the written buffer with the read buffer.
Don't initialize variables prematurely so that the compiler has a chance
to warn about variables read before they are written.
Repeatedly flush the page buffer, once each time we end the tick.
Write errors to stdout instead of stderr.
Combine the VFD SWMR and non-VFD SWMR raw-data test into one
routine that takes a bool parameter to switch on VFD SMWR.
Update my description of the to-be-written metadata test for VFD SWMR.
output, then it tested $? for an error exit. $? told the error status of
`tee`, though, not the test programs! So no test failures were counted, even
when some tests clearly failed. I changed the test script to use a shell
subroutine, `catch_out_err_and_rc`, to catch test programs' output and result
code.
(1) Increase the # of records to write (Nrecords) in testvfdswmr.sh.in so as to ensure the writer
will not exit before the reader
(2) Use H5E_BEGIN_TRY/END_TRY when H5Fopen() the test file in reader tests
(3) Add "READER" or "WRITER" to debugging messages
(4) Misc cleanup
the memory manager -- details shown below.
Note that there are other issues as well -- this is not a
working version.
[mainzer@jelly test]$ ./vfd_swmr
Testing Configure VFD SWMR with fapl PASSED
Testing VFD SWMR configuration for the file and fapl PASSED
Testing H5Fvfd_swmr_end_tick() for VFD SWMR PASSED
Testing Create/Open/Flush an HDF5 file for VFD SWMR PASSED
Testing Verify the metadata file for VFD SWMR writer vfd_swmr: H5MVsection.c:233: H5MV__sect_can_merge: Assertion `((sect1->sect_info.addr)!=((haddr_t)(long)(-1)) && (sect2->sect_info.addr)!=((haddr_t)(long)(-1)) && (sect1->sect_info.addr)<(sect2->sect_info.addr))' failed.
Abort (core dumped)
[mainzer@jelly test]$
(1) Assertion failure in the vfd_swmr test
(2) Reader error in the vfd swmr concurrent tests
Also fixes for:
(a) Use H5MV_alloc() to allocate space for md_pages_reserved when creating the metadata file in H5F__vfd_swmr_init()
(b) Remove a multi-page (when vfd_swmr_writer is true) from the page buffer in H5MF_xfree()
(2) Test files for encoding/decoding property lists
(3) Fix test failures for PB statistics in test/page_buffer.c
(Will double check with John later about PB statistics collection)
--src/H5PB.c: checks for size >= page size
--src/H5MF.c: disable/enable page buffering in H5MF_tidy_self_referential_fsm_hack()
--src/H5MFsection.c: call H5PB_remove_entry() for both raw/metadata pages in H5MF__sect_small_merge()
(B) Port and modify existing concurrent swmr tests to VFD SWMR. Also modify the following:
--remove flushes from VFD SWMR writer tests
--set Nreaders to 0 in test/testvfdswmr.sh.in to test for writers only
Please enter the commit message for your changes. Lines starting
Tested serial / debug on Charis and Jelly.
Two known issues:
1) New page buffer seems to expose issues in the accumulator code.
For whatever reason, fheap with the new page buffer exposes corruption
issues if the page buffer uses H5F__accum_read/write(), but the problems
go away if the page buffers uses H5FD_read/write() instead.
Need to either chase this or dis-able page bufffer in combination with
the accumulator.
2) Encountered a number of assertion failures that are explainable by the
free space manager code not telling the page buffer to discard pages
when they are freed.
Wrote code to handle this -- once the free space manager is modified,
this code should be removed and the original assertions restored.
1) Free space manager for the metadata file
2) Delayed free space release linked list
3) H5F_update_vfd_swmr_metadata_file()
3) VFD SWMR driver: read callback
4) Flushing for VFD SWMR
5) Port one concurrent test from swmr test set
6) Bug fixes and refactoring
1) Define driver for the VFD SWMR reader
2) Implement VFD SWMR open callback
3) Implement H5FD_vfd_swmr_get_tick_and_idx()
4) Load and decode metadata file header and index
4) Closing for VFD SWMR
* commit 'c834d9f99d45e5b9752e8525fe8761ea5592bf2c': (41 commits)
HDFFV-10568 fix hdf5_java library dependency
Remove another extra path var
Move muti-config dir setting to root process
Fix one more intermediate location
Use different variable
Cleanup and add intermediate dir for java
Java must use shared libs to allow dlopen calls
Correct names
Call new function
Correct default API version for develop to 112.
Fix typo
HD prefix and whitespace
Update RELEASE.txt with suggested changes
Update MANIFEST file for new t_coll_md_read.c file
Remove now-unused local variable
Add fix for HDFFV-10501
Revert testfile FILE change
change FILE path
Add testfiles to data copy
Same changes needed for examples as test
...
1) Public routines: H5Pget/set_vfd_swmr_config
2) Public routine: H5Fvfd_swmr_end_tick
3) Initialization when opening a file with VFD SWMR writer
4) Tests in test/vfd_swmr.c for VFD SWMR
5) Fix a bug in src/H5Fint.c: when error is encountered after the root group is created
Note the following:
--This is WORK IN PROGRESS and will subject to change as implementation goes.
--There is test failure form enc_dec_plist.c: I will fix this when changes to the property list are settled.
--The branch is updated with the latest from develop as of 8/14/2018
Description:
Applied the typo fixes from user's report.
The previous pull request couldn't be merged because it was too old,
and it was too complicated for me to resolve conflicts.
Platform tested:
Linux/64 (jelly) - very minor
