* commit '1bbec8d69d8c7d3aa46e44b5448733ef396eece1':
Deleted extraneous files from doc, and updated the contact email address in doc/vfd-swmr-user-guide.phg to swmr@forum.hdfgroup.org
* commit '47b1c667c64677d2e28c4128fc027cfdfe60b354':
More minor edits to users guide.
Updated users guide for location of alpha 1 release & repaired several typos.
Added .pdf and .docx versions of the most recent version of the VFD SWMR RFC. Updates on the RFC are cursory, and mostly in the introduction. The design section is dated -- added a warning to this effect.
VFD SWMR RFC. Updates on the RFC are cursory, and mostly in the
introduction. The design section is dated -- added a warning to
this effect.
Reviewed and editied the users guide -- specifically:
Verified instructions for build and test, and also for demos.
Removed mention of call to autogen.sh, as that will already be
run on the friendly user release version.
Re-worked the introduction to better discribe VFD SWMR, and
to outline current limitations.
Changed references to "shadow file" to "metadata file" to match
field names.
Still need to update pointer to the bit-bucket repository for the
friendly user release.
Also, must verify that the demo code mentioned in the user guide
is publicly available.
* commit 'aebc84053f9a592aac765fbd3b26f2941015d2e5':
Limit the repeat rate for duplicate zoo warnings to once every five seconds.
Move below_speed_limit() from vfd_swmr_bigset_writer.c to vfd_swmr_common.c, document it, and fix a bug.
Document some of the functions in here. Update the comment at the top of the file. NFCI.
In the `vfd_swmr_create_fapl()` dissection, change the /** **/
comments in the literal code to plain markdown paragraphs.
Slightly change wording and markdown elsewhere.
H5Dopen fails, rapidly retry up to 9,999 times. Log H5Dopen failures,
but log no more than once every five seconds to avoid spamming the
terminal.
With these changes, it's easier for the reader to open the last dataset
before the writer created it, but the reader recovers instead of
quitting with an error. It should only be necessary to retry opening
the *last* dataset; all previous datasets should open on one try if the
last is open.
call H5F__vfd_swmr_writer_md_test(), actually sleep for more than `max_lag`
ticks so that the deferred-frees queue is actually flushed before we force new
items onto the queue and count them.
In H5F__vfd_swmr_writer_md_test(), check that the number of deferred
shadow-space frees is *precisely* the number `nshadow_defrees`, since that
seems to be what the tests that call this routine intend.
using the new routine H5FDdeduplicate().
Simplify H5F_open() a bit, pushing some of the configuration checks into the
SMWR VFD. For example, check that page buffering is enabled in
H5FD_vfd_swmr_open() instead of in H5F_open(). Compare VFD SWMR configurations
in H5FD_vfd_swmr_dedup() instead of in H5F_open().
Clone the default file-access property list at a new file-access property list
ID, H5P_FILE_ACCESS_ANY_VFD. The new ID is used to indicate that if the file
that's being opened is already open under an existing virtual file, and if that
virtual file would not ordinarily be opened with the default FAPL, then it's ok
to use that virtual file.
Add a new optional method, `dedup`, to H5FD_class_t, and use it to customize a
VFD's deduplication.
Customize the SWMR VFD's deduplication. Make it honor H5P_FILE_ACCESS_ANY_VFD
Embed the VFD SWMR configuration in the H5FD_vfd_swmr_t to facilitate
comparison of configuration between new and old SWMR virtual files.
In H5F__sfile_search(), match using a pointer comparison instead of H5FD_cmp(),
because we will only ever enter H5F__sfile_search() with a deduplicated
H5FD_t *.
Fix HDFFV-11120 and HDFFV-11121 (CVE-2018-13870 and CVE-2018-13869)
* commit '707e30c6be1954c0027374124207e46caae68cbc':
Fixed typos in error messages.
Fix HDFFV-11120 and HDFFV-11121 (CVE-2018-13870 and CVE-2018-13869)
* commit 'e4603ff8a171e82de6373ba6eddcf59d5eeab5aa':
Updated the Java tests to deal with H5Sset_extent_none changes
Fixed additional typos in tselect.c comments.
Fixes typo in tselect.c
Updates H5Sset_extent_none() to set H5S_NULL
* commit '5857bb817019b8513f796c9564de3f1845f9a44a':
Only one line needed to compare
Revert err file
revert error-stack enable
Add release notes for tools
add test file ext
Correct file name in test
Tools refactor
The API call used to set the internal H5S_NO_CLASS value which
produced errors when such a dataspace was passed to many other API
calls.
Fixes HDFFV-11027
Description:
When a buffer overflow occurred because a name length was corrupted
and became very large, h5dump produced a segfault on one file and a
memcpy parameter overlap on another file. This commit added checks
that detect a read pass the end of the buffer to prevent these error
conditions.
Platforms tested:
Linux/64 (jelly)
SunOS 5.11 (emu)
* commit '9d3ea366d4bf3278e8260f4d5cdcc88cb4b61c56':
Update MANIFEST. Add new test to Cmake.
Fix for jira issue HDFFV-11080: (1) Patch up the file pointer when reading attribute of variable length datatype (2) Test to verify the fix when doing multiple threads
* commit '3023b34272cc6ada84aebfa7441a6a55afa3621c':
Update MANIFEST. Add new test to Cmake.
Fix for jira issue HDFFV-11080: (1) Patch up the file pointer when reading attribute of variable length datatype (2) Test to verify the fix when doing multiple threads
multiple referring H5F_t on the EOT queue. This stops the VFD SWMR writer from
advancing the shadow file's tick number too fast when we're using virtual
datasets (VDS) with VFD SWMR.
Move error_stack setup to tools library
Add structure for h5dump properties
Eliminate duplicated h5diff functions
Use properties structure in place of argument lists.
* commit '806f5b5b2a280072f62f58c48e93acdc84f2b3ae':
Removed POSIX #defines from configure on Solaris.
Fallback to older time functions in new timer code.
* commit 'b8013380792a3503003f0a2bd60d7541656b03c7':
Removed POSIX #defines from configure on Solaris.
Fallback to older time functions in new timer code.
if H5FD__vfd_swmr_header_deserialize() succeeds, then the header that was
passed in was actually initialized. This squashes a used-before-initialized
warning from GCC.
* commit 'b480d0a38b1e265c55c74262bf384d255a33e07b':
Remove path to szip header file from AM_CPPFLAGS when configure check of libsz fails. Fix for HDFFV-10830.
HDFFV-11116 Add option for install location of Fortran MOD files
* commit '800f93f7c994f88dfa716746153ded4b1e690e3a':
Remove path to szip header file from AM_CPPFLAGS when configure check of libsz fails. Fix for HDFFV-10830.
multiple opens of the same file with VFD SWMR---i.e., twice for writing, or for
reading and for writing. In the long run, this will help me encapsulate more
of the SWMR functionality in the VFD, too.
the lower virtual file's `exc_owner` field. While I'm here, remove a
gratuitous assertion.
This is part of a changeset that helps us avoid creating multiple H5F_shared_t
for one file when virtual datasets are used with VFD SWMR. The old code for
deduplicating VFD SWMR H5F_shared_t instances did not work correctly with VFD
SWMR, so we'd end up with multiple H5F_shared_t all active on the same file.
member to all virtual files. Add a routine, H5FD_has_conflict(), that returns
true if a new virtual file is identical to an existing virtual file that has an
exclusive owner. Establish an exclusive owner for a VFD SWMR virtual file's
lower virtual file.
Rename bsdqueue.h to H5queue.h and install it, since it's used by H5FDpublic.h.
This is part of a changeset that helps us avoid creating multiple H5F_shared_t
for one file when virtual datasets are used with VFD SWMR. The old code for
deduplicating VFD SWMR H5F_shared_t instances did not work correctly with VFD
SWMR, so we'd end up with multiple H5F_shared_t all active on the same file.
* commit '188c43b8ee78d830c22f30fb9adb802e46722fcd':
Change load path in libhdf5_java.dylib to unversioned libhdf5.dylib for junit tests on Macs.
Fixes Autotools builds on Solaris after monotonic timer changes
clock_gettime() wasn't being found, so some configure changes
were needed to set an appropriate POSIX level and use a C99
compiler. This required several compiler flag changes.
Autotools only
* commit '4c2071bfd3270dc3a688914b4a058147b94b7023':
Lib files needed are now copied to a subdirectory in the java/test directory, and on Macs the loader path for libhdf5.xxxs.so is changed in the temporary copy of libhdf5_java.dylib. Fix for HDFFV-11063
* commit 'fc69de0e8ae30e80c1c3c542b156226a75444525':
Lib files needed are now copied to a subdirectory in the java/test directory, and on Macs the loader path for libhdf5.xxxs.so is changed in the temporary copy of libhdf5_java.dylib. Fix for HDFFV-11063
* commit '7371c83f9777b34b31909e99e052398b93c31bed':
Remove non-existent example
Add new source files to CMake build
Clean up warnings
va_arg -> HDva_arg
Refactor code to remove remaining checks for H5_HAVE_GETTIMEOFDAY scattered around in various places. Also clean up iopipe.c.
Correct mistake in H5_now_usec calculation for clock_gettime.
Remove detection for mach/mach/time.h, since we're no longer using the time routines from that header.
Update H5_now_usec to prefer using clock_gettime.
Changes to make timers within the library monotonic.
* commit '32c483cb86031962a09ecdb49dcb331d28dddee9':
Copy lib files for junit.sh to temporary subdirectory of <build directory>/java/test instead of to the prefix/lib directory. Fix for HDFFV-11063.
* commit '30776671fb0a27a98b0640aa968f850241a862ce': (31 commits)
* Fix intermittent error with Splitter VFD. Mismatch in time of test file creation was creating false negatives.
Added note for HDFFV-10591.
Fix HDFFV-10591
Reduce overhead for H5open, which is involved in the public symbols like H5T_NATIVE_INT, etc.
Remove unnecessary version conditions for Clang compilers.
Fix for HDFFV-10961: Couple changes to the tests' usage output files for h5clear.
Fix for HDFFV-10961: add description to usage for --increment option.
Add c++ to --enable-sanitize-checks option.
Remove duplicate entries in MANIFEST.
Make changes to CMake CompilerFlags.cmake files so extra flags are loaded for non-GNU compilers.
Fix MANIFEST
Add tests for all version to H5_NO_DEPRECATED_SYMBOLS section and to section for current version, with and without default API version flags.
HDFFV-11000: update-testh5cc.sh.in to test sample versioned functions in HDF5 1.10, 1.12 and develop.
Remember the info for the last ID looked up for a given ID type.
Eliminate allocating file & memory offset & length arrays when performing I/O on a single element.
Update new clang files to not pick up clang as vendor for pgCC. Add new files to MANIFEST Temporary demotion of 2 -Werror warning flags that fail on macos 10.12 Remove Production flag unknown to Apple clang.
Remove redundant metadata cache tagging from some low-level internal chunk functions. The metadata cache tagging has already been done by routines further up the call stack.
Clean up code to get clang version in config/linux-gnulibc1
Minor normalizations with HDF5 1.10.
Add flags from config/clang-warnings/*general files to H5 C and CXX flags for all versions of Clang and Clang++ compilers. Switched from cut to awk in testcheck_version.sh.in to avoid dependence on tab vs. " " in version definitions in H5public.h.
...
* commit 'e1215177b6b87887233a90992a0264f11d78b88c':
* Fix intermittent error with Splitter VFD. Mismatch in time of test file creation was creating false negatives.
Mismatch in time of test file creation was creating false negatives.
Add file-duplication routine: `h5_duplicate_file_by_bytes()`.
* Change library calls in `h5test.c:h5_compare_file_bytes()` to their HD-prefixed equivalents.
Assert index entries are in sorted order earlier in the loop over
old and new indices.
When looping over the remaining new index entries, just do
`entries_added++` to match the other loops.
In the final log entry in H5F_vfd_swmr_reader_end_of_tick(), mention
whether the call will exit with success or failure.
apply it individually to each dataset instead of setting the chunk-cache
parameters on the file. Alas, it didn't make any difference, but I'll keep the
change.
Description:
h52gif produced a segfault when a buffer overflow occurred because
the data size was corrupted and became very large. This commit added
a check on the data size against the buffer size to prevent the segfault.
It also added error reporting to h52gif to display an error message
instead of silently exiting when the failure occurred.
Platforms tested:
Linux/64 (jelly)
SunOS 5.11 (emu)
H5F_vfd_swmr_reader_end_of_tick: delete superfluous assertions and
extract a com mon subexpression into a H5FD_t * variable.
Carry on with HGOTO_ERROR() cleanup. Delete superfluous parentheses to
reduce visual clutter. Delete superfluous casts. Delete out-of-date
comment: the index size is not fixed any longer.
statement-ify, changing
HGOTO_ERROR(..., \
)
to
HGOTO_ERROR(...,
);
Remove blank lines between if-clause and HGOTO_ERROR. Add some curly
braces to if-statements where that clarifies things.
NFCI.
that makes assertions fail.
Add an optional `close` method to the `H5D_chunk_ops_t`, and use that to
release "holds" on metadata cache (MDC) entries.
For extensible arrays and v2 B-trees, use the existing `dest`(roy)
method to implement `close`. For v1 B-trees and other chunk indices,
don't provide `close`: we cannot safely close the v1 B-tree index, and
the other indices don't have a meaningful presence in the MDC.
Revert my first attempt at making v1 B-tree chunk indices closeable
with `dest`.
Put my comment about the stopgap fix for VFD SWMR at the right place
in src/H5Dchunk.c.
the entries in a hash bucket after evicting tagged entries. Evicting
tagged entries can can affect both entries before and after the current
entry in the bucket's linked list, so we cannot be sure that either
`entry_ptr` or `follow_ptr` is valid.
This stops the assertion (entry_ptr->page != page) ||
(entry_ptr->refreshed_in_tick == tick) from failing in the test
`testvfdswmr.sh many_small`.
H5D__chunk_index_close(), and call it in H5D__chunk_read() after reading
a chunked dataset. In this way, indices based on extensible arrays and
v2 B-trees do not leave pinned/tagged entries in the metadata cache that
we cannot evict/refresh when we load changes from the shadow file.
Make some changes to the v1 B-tree code that set the pointer to the
closed B-tree to NULL and, further, tolerate a NULL pointer where
previously that was impossible.
* commit '01b632c69ca1c97c2b0e1aaeeab1dd2362b580a6':
Fix for HDFFV-10961: Couple changes to the tests' usage output files for h5clear.
Fix for HDFFV-10961: add description to usage for --increment option.
* commit '5c0bd670d6e9919c463fbc05de99e2ba55b0761e':
Add tests for all version to H5_NO_DEPRECATED_SYMBOLS section and to section for current version, with and without default API version flags.
HDFFV-11000: update-testh5cc.sh.in to test sample versioned functions in HDF5 1.10, 1.12 and develop.
* commit '20eab963899841c9a003baebead8e3bc07d9e127':
Remove duplicate entries in MANIFEST.
Make changes to CMake CompilerFlags.cmake files so extra flags are loaded for non-GNU compilers.
Update new clang files to not pick up clang as vendor for pgCC. Add new files to MANIFEST Temporary demotion of 2 -Werror warning flags that fail on macos 10.12 Remove Production flag unknown to Apple clang.
* commit '7f0c0134166275c0745e0e65f9d81b4de72d5ce2':
Fix MANIFEST
Remember the info for the last ID looked up for a given ID type.
Eliminate allocating file & memory offset & length arrays when performing I/O on a single element.
Remove redundant metadata cache tagging from some low-level internal chunk functions. The metadata cache tagging has already been done by routines further up the call stack.
* commit 'a8a4e9f27b67ae2e22970e42859225adcc2e5cbb':
Eliminate allocating file & memory offset & length arrays when performing I/O on a single element.
* commit '9e8899c3b47a2041242dc4539ca8b1c5e151a371':
Remove redundant metadata cache tagging from some low-level internal chunk functions. The metadata cache tagging has already been done by routines further up the call stack.
* commit 'c7626f79fdee4eee13925e240ef41e61f3367eab':
Add flags from config/clang-warnings/*general files to H5 C and CXX flags for all versions of Clang and Clang++ compilers. Switched from cut to awk in testcheck_version.sh.in to avoid dependence on tab vs. " " in version definitions in H5public.h.
Add files for adding warning flags for clang compilers in autotools configure.
* commit 'aa338c3a87a4544ccb164832422e3e2462a57b88':
Detect when there's the same-shaped selection of a single block of elements on both selections, but with different selection types (i.e. one selection defined as an 'all' type and the other as a hyperslab or point type), without falling into the generic selection iteration case.
Normalization of H5T.c with 1.10.
Normalization of H5D.c with 1.10.
* commit 'de54cff50c419751e0080b6a732facac43f87ba6':
Detect when there's the same-shaped selection of a single block of elements on both selections, but with different selection types (i.e. one selection defined as an 'all' type and the other as a hyperslab or point type), without falling into the generic selection iteration case.
both selections, but with different selection types (i.e. one selection defined
as an 'all' type and the other as a hyperslab or point type), without falling
into the generic selection iteration case.
below than 80 columns: use the malloc idiom,
```
type *p;
p = malloc(sizeof(*p));
```
instead of
```
type *p;
p = (type *)malloc(sizeof(type));
```
Make a similar change to some `memset` calls. NFCI.
HGOTO_ERROR() instead of HDONE_ERROR() so that we jump to the `done`
label right away. This ought to fix the problem Vailin was seeing,
where the library left H5F_vfd_swmr_reader_end_of_tick() prematurely for
seemingly no reason.
each to appear.
Add but do not yet perform tests on many small extensible datasets and a few
big extensible datasets. Vailin is working on a bug that causes both tests to
fail virtually always.
each dataset every `steps` steps.
Update usage message.
Add a cast to `time_t` to quiet a compiler warning.
Replace two occurrences of a debug statement in `verify_extensible_dset()` with
one occurrence in `verify_chunk()`.
Replace the anonymous constant `2` with `hang_back` and increase `hang_back` to
3. XXX Now that I've fixed a bug, reduce `hang_back` to 2, again.
Verify datasets in the reverse of the order they are written so that we spend
less time re-verifying datasets written in the same step.
Description:
A superblock pointer was set to a temporary structure, as a kluge,
and meant to be reset back to NULL after the processing was completed.
However, in the case of failure before the completion, this setting
caused the superblock pointer to be unpinned twice in the error
recovery process. As a result, that generated a segfault in h5clear.
The fix resets the superblock pointer to NULL after H5F__super_ext_remove_msg
fails so that h5clear can fail properly.
Note: After the fix, when built in debug mode, there will be an
assertion failure with the user's file, which appeared to be corrupted:
H5MM_final_sanity_check: Assertion `0 == H5MM_curr_alloc_bytes_s' failed.
This did not happen on a good file or on the corrupted file with the
library built in production mode. The un-freed memory were allocated
during error recovery.
Platforms tested:
Linux/64 (jelly)
by the added entries. This fixes a bug in the vfd_swmr_bigset_writer/_reader
test, where the reader would read metadata with a bad checksum. This may also
fix an error that Vailin is seeing.
when handling an I/O request on a metadata entry that has been sub-allocated
from a larger file space allocation (i.e. fixed and extensible array), and
that crosses at least one page boundary. .
This required modifying the metadata cache to provide the type of the
metadata cache entry in the current I/O request. For now, this is done
with a function call. Once we are sure this works, it may be appropriate
to convert this to a macro, or to add a flags parameter to the H5F block
read/write calls.
Also updated the metadata cache to report whether a read request is
speculative -- again via a function call. This allowed me to remove
the last address static variable in the H5PB_read() call, which is
necessary to support multiple files opened in VFD SWMR mode.
Also re-wrote the H5PB_remove_entries() call to handle release
of large metadata file space allocations that have been sub-allocated
into multiple metadata entries. Also modified the call to
H5PB_remove_entries() in H5MF__xfree_impl() to invoke it whenever
the page buffer is enabled and the size of the space to be freed is
of page size or larger.
Tested serial / debug on charis and Jelly.
Found a bug in H5MF_xfree_impl(), in which the call to H5PB_remove_entries()
is skipped due to HGOTO_DONE calls earlier in the function. While the
obvious action is to move the call earlier in the function, best to
consult with Vailin first, as there is much going on and it would be
best to avoid making the situation worse. If nothing else, there are
some error management issues.
to integer of different size [-Werror=pointer-to-int-cast]` and
`test/snapshots-hdf5/current/test/swmr_sparse_reader.c:129:100:
error: cast from pointer to integer of different size
[-Werror=pointer-to-int-cast]`.
g++ (older than gcc/g++ 4.2).
Correct gnu-cxxflags to determine warnings flags to be added based on
C++ compiler version instead of C compiler version.
* Rename server-stop utility to mirror_server_stop.
* Remove external dependency on bzero().
* Modify test/use_common to use only the public API.
* Rename internal bitswap macro to follow convention.
* "Simultaneous and equivalent" Read-Write and Write-Only channels for
file I/O.
* Only supports drivers with the H5FD_FEAT_DEFAULT_VFD_COMPATIBLE flag for
now, preventing issues with multi-file drivers.
Add Mirror VFD to library.
* Write-only operations over a network.
* Uses TCP/IP sockets.
* Server and auxiliary server-shutdown programs provided in a new directory,
`utils/mirror_vfd`.
* Automated testing via loopback ("remote" of localhost).
commit 8963c3bf756f8f8ec21beea9bd29a767e77675a8
Author: Larry Knox <lrknox@hdfgroup.org>
Date: Wed Apr 8 16:52:27 2020 -0500
Commit changes to gnu-cxxflags to remove unmatched " and to gnu-fflags
to not add C warnings flags to H5_FCFLAGS.
This is triggered by the tests for revised references when the libver bounds setting does not allow version
4 datatype message to be created. The test failure is abort core dumped.
This is due to the datatype initialization fails before the datatype ID is registered.
The datatype cleanup code should provide for the above situation.
The code to fix the problem is the same as what is done in H5D__open_oid().
Delete the comment questioning whether pthread_mutex_lock is allowed
in a key destructor, since pthread_key_create(3) provides the answer:
There is no notion of a destructor-safe function. If an application
does not call pthread_exit() from a signal handler, or if it blocks any
signal whose handler may call pthread_exit() while calling async-unsafe
functions, all functions may be safely called from destructors.
Delete redundant comment.
Modify h5repack to integrate with VOL connectors
Update tools library to accomodate VOL connectors
Update logic in h5tools_fopen for VOL connectors
Add command-line options to h5repack for specifying in/out VOL
connectors
Implement h5tools_set_vol_fapl
Fix library shutdown issue
Integrate ROS3 and HDFS VFDs into new h5tools_get_fapl() scheme
Avoid H5Ocopy in h5repack when using different VOL connectors
Update h5tools_test_utils.c for ROS3 and HDFS integration
there's no need to check if the metadata cache is flushing if we already
know the file is closing, because the condition we rely on is "closing
OR flushing." Further, the cache may have already gone away, so
sometimes calling into the cache to see if it's flushing will crash the
program.
could block at the barrier, wake and exit the barrier, re-acquire the barrier
lock and increase `nentered` before the other blocked threads woke and checked
`nentered % count == 0`. Then the other blocked threads would check `nentered
% count == 0` and, finding it false, go back to sleep in the barrier. This new
implementation waits for a looser condition to obtain so that threads don't go
back to sleep in the barrier.
all configurations.
Previously it was neither declared nor defined in --disable-threadsafety
builds. The compiler's warning got lost in the noise---I first saw the issue
because my -Werror branch stopped compiling cold---and the tests still linked
and ran.
Reduce gratuitous casts---e.g., (size_t)1.
Use the right format string for a pointer.
In the H5C sanity checks, change a "size increase" variable from ssize_t
(too narrow) to int64_t (wide enough).
Parenthesize every appearance of `storage` in the macro
`H5D_CHUNK_STORAGE_INDEX_CHK(storage)` so that you can pass in an
expression like &sc and it works properly.
Disallow re-assignment of the `dset` parameter to H5D__chunk_init()
because it helped assure me that it's safe to replace the repeating
expression `&dset->shared->layout.storage.u.chunk` with `sc` throughout.
Replace lengthy expressions such as
`&dset->shared->layout.storage.u.chunk` with `sc` throughout several
functions in H5Dchunk.c ISTR that the compiler warned that `sc` was
declared but unused in a couple of functions, and then I found that `sc`
could be used in many places. Maybe the disused `sc` appeared because a
bunch of code was copied and pasted, I don't know. Anyway, it's a lot
tighter code now that I use `sc`.
In H5D__chunk_update_old_edge_chunks() and H5D__chunk_delete()
I actually expand `sc` and another temporary variable, `pline`,
because they're used only in !defined(NDEBUG) code. This squashes
unused-variable warnings in the defined(NDEBUG) configuration.
Don't drop the `volatile` qualification with a cast in
tools/src/h5import/h5import.c.
previous warnings PR. An array element was assigned to
itself---shape[2]Â =Â shape[2];---instead of being assigned to
chunk[2].
fortran/src/H5Pf.c: move conditional compilation controlled by
H5_NO_DEPRECATED_SYMBOLS outside of a function for readability.
fortran/src/H5match_types.c: put a variable's declaration under the same
conditional compilation (H5_FORTRAN_HAVE_C_LONG_DOUBLE) as its
use.
For now, skip compilation of some unused debug dump routines in the JNI.
While I'm in the JNI, delete a set-but-unused variable.
src/H5Z.c: condition a variable declaration on H5_NO_DEPRECATED_SYMBOLS
so that it's not declared but unused or vice versa.
test/cache_common.h: add an #include in to get some symbols we need to
avoid implicit declaration warnings.
test/dsets.c: use a more conventional conditional-compilation syntax.
test/dt_arith.c, test/fillval.c: initialize a bunch of uninitialized
variables before use.
test/vfd.c: pass the expected type of `void **` to posix_memalign(3)
instead of `int **`.
testpar/t_bigio.c: explicitly compare with 0 instead of using ! when
"equal to 0?" is the question not "is false?" Repair some
indentation while I'm here.
testpar/testpar.h: repair misaligned line-continuation backslashes in a
macro that probably should be a function so that we don't have
to fiddle with the line continuation to begin with.
tools/src/h5repack/h5repack_main.c: fix some compiler fussing about
enums.
tools/test/perform/pio_engine.c: the compiler fusses if you cast a
function call returning double directly to off_t. It's ok if
you cast a variable that's a double to off_t, however. Write
and use a new function, sqrto(), to avoid the cast warnings.
of the va_list, so it's at least possible for another connector to know what
the operation is and decide whether to implement it or not.
Added a new VOL sub-class called "introspect" where callbacks that report
information about the connector or container can be placed. Added an
'opt_query' callback to this sub-class, for a connector to report back
to the library whether a particular optional callback operation is supported.
Also added a 'get_conn_cls' introspection callback, to retrieve the H5VL_class_t
of a connector (either the "current" connector, H5VL_GET_CONN_LVL_CURR, or
the terminal connector, H5VL_GET_CONN_LVL_TERM).
Moved the "post open" operation from a file 'specific' operation to a file
'optional' operation, now that it's possible to detect (with the 'opt_query'
introspection callback) whether a VOL connector implements an optional
operation, without just returning an error.
Added new internal VOL helper routines: H5VL_object_is_native, to determine
if an object is in (or is a) native file, and H5VL_file_is_same, to determine
if two objects are in (or are) the same terminal VOL connector's container.
(And moved the special handling for FILE_IS_EQUAL operation out of internal VOL
callback routine into H5VL_file_is_same)
Made new dataset 'get' operation for H5Dvlen_get_buf_size, aligning it better
with other 'get' operations in API.
Fixed several issues with pass-through connectors, which are now passing the
'make check-passthrough-vol' tests again.
A bunch of warning and style cleanups as well.
structs containing those arrays. Encapsulating the arrays in this way
makes it easier to write and think about pointers to these types, casts
to/from these types, etc.
An interesting side-effect that we probably should *not* rely on is
that the struct-encapsulation changes the alignment so that some GCC
warnings about casts that increase the alignment requirement of the
operand go away. Warnings like that have to be taken seriously: I will
add -Werror=cast-align to the default compiler flags so that they stop
the build quickly.
GCC warnings led me to some surprising casts in test/trefer.c. I found
that it was possible to make many simplifications after introducing the
struct-encapsulation that I described, above.
In test objcopy_ref `same_file` is assigned but never used. Delete it.
procedure for checking it consistent across selection types and between
_s and _u, ensures it is always is performed even when called within the H5S package, and removes the redundant check that would occur when callins H5S_select_adjust_s() from outside the H5S package.
structs containing those arrays. Encapsulating the arrays in this way
makes it easier to write and think about pointers to these types, casts
to/from these types, etc.
An interesting side-effect that we probably should *not* rely on is
that the struct-encapsulation changes the alignment so that some GCC
warnings about casts that increase the alignment requirement of the
operand go away. Warnings like that have to be taken seriously: I will
add -Werror=cast-align to the default compiler flags so that they stop
the build quickly.
GCC warnings led me to some surprising casts in test/trefer.c. I found
that it was possible to make many simplifications after introducing the
struct-encapsulation that I described, above.
of `hsize_t`, `start`, to `long long`, but I think the way that I have
rewritten it, it probably produces a more useful result? As a bonus,
GCC has stopped warning about it.
algorithm to (optionally) avoid sharing selection data structures.
Tested internal code (including with valgrind) by setting VDS code to
avoid sharing selection, has since been changed to share selection for
performance, so this code is not yet tested in regression tests. API
has not been tested.
"obsolete" / POSIX "basic" regular expressions. Also, not every version of
`sed` out there supports the `-E` option. So delete the -E flag and use
the regex `[^/][^/]*` instead of `[^/]+`.
Add config/netbsd to the MANIFEST.
location set to be in a file. Only meant to be used by VOL connectors.
Implement H5VLpeek_connector_id() to support connectors querying their
own IDs. Fix app_ref with connector IDs in a couple places (external
VOLs registered as default through ENV should be visible to the
application). Modify vlen and reference interfaces to work with
arbitrary VOL connectors. Implement file "post open" specific
callback, to enable connectors to update their file structs after a
wrap context has been set.
original installation prefix from the examples prefix. Use that
relative path to locate the current installation prefix, always. Fall
back to an absolute installation prefix if the relative path cannot be
derived.
original installation prefix from the examples prefix. Use that
relative path to locate the current installation prefix, always. Fall
back to an absolute installation prefix if the relative path cannot be
derived.
This is handy for NetBSD where HDF5 examples are installed
by convention in $prefix/share/examples/hdf5/ rather than in
${prefix}/share/hdf5_examples/, which is the HDF5 default.
Place hdf5_examples/ under ${datarootdir} which on most systems will be
${prefix}/share/, anyway.
* commit 'fcd87d2569ca4d8396c7db4eddb46cab04010abf':
Revise gnu-*flags and cmake/HDF*CompilerFlags.cmake files to add warning flags for GCC compilers version 4.8 and above. Removed files from gnu-warnings that only apply to versions < 4.8. Consolidated warnings from versions < 4.8 that apply to versions >= 4.8 into the 4.8 warnings files.
flags for GCC compilers version 4.8 and above.
Removed files from gnu-warnings that only apply to versions < 4.8.
Consolidated warnings from versions < 4.8 that apply to versions >= 4.8
into the 4.8 warnings files.
* commit 'ae07c9ba2429a9c49291bcd9f02c0ce36b0a60ce':
Replaced a few calls to HDmemcpy with H5MM_memcpy, which does overlap checking.
Removed HDgets macros since gets was deprecated in C99 and removed in C11 (gets() is not used in our repo).
columns. If the number of datasets is greater than the number of steps,
then only pause between steps, do not pause between individual datasets
written/verified. Otherwise, pause between each dataset written/verified.
that are longer than the buffer that the caller supplied: the checksum
usually will fail, but that's not actually a fatal condition, and
usually we will have another opportunity to verify the checksum.
In H5FD_vfd_swmr_read(), remove a bunch of disused code.
In H5FD_vfd_swmr_read(), do not re-read a shadow image that has a
bad checksum, because a bad checksum indicates a serious problem
(writer outran reader, OS defect, hardware failure) from which
H5FD_vfd_swmr_read() cannot recover.
Rationale: the writer write(2)s new shadow images before the new index,
and the new index before the new header. In H5FD_vfd_swmr_read(),
the reader has read(2) both the index and the header in full. POSIX
semantics indicate that in these circumstances, the last shadow image
write(2) MUST be completely visible when we read(2). That is, the index
write(2) & read(2) and the header write(2) & read(2) pair cannot
divide a preceding shadow-image write(2).
The reader may see a "torn" image at this juncture if, for example,
the writer got max_lag ticks ahead of it and reused the storage for
this shadow image. Even if the reader "recovers" by re-reading the
image until its checksum is correct, it cannot be sure that the
image thus read is the right one for the HDF5 address passed to
H5FD_vfd_swmr_read(), and it cannot be sure that the image thus read is
not stale, because it's operating with an out-of-date shadow index.
Add log outlets swmr_read, swmr_read_exception, and swmr_read_err.
Log to `swmr_read` on entry to H5FD_vfd_swmr_read(), log to
`swmr_read_exception` when checksums are skipped for exceptional
conditions (page buffer not configured, buffer shorter than shadow
image), and log to `swmr_read_err` when the checksum fails.
VFD_SWMR_LEAVE() for use by FUNC_ENTER_API/_LEAVE_API macros. In the
macros, don't HGOTO_ERROR(), since that will jump back to the `out`
label, but HDONE_ERROR() on error, instead.
* commit 'f82348f80940e6ea197b7913f6357a5bbde3c43d':
Comment out pull-request action and enable fail fast
Correct filename
Change to use alias
Fix name setting
OESS-65 fix packaging config files
* commit 'deadf2546561caa01537334dbb209e60c28d26c5':
Align contents of CMake and autotools testfile cleanup lists.
Avoid allocating a chunk index for datasets with 0-sized dimensions, until the dataset is extended.
output streams using STDIN_PATH and STDOUT_PATH. I will use that for
the zoo reader and writer. Move redirection of standard error output to
the standard output stream outside of the curly braces, since usually
I want to save the `echo` and `wait` output, too, and it makes the
redirection of the supervised program a little easier to follow, I
think.
* commit '3f3363177d5cc1efc8fcfc70e1feed7a293da2d4':
Range check # of 'extra' command-line parameters against array size.
Refactor "extra" command-line arguments, and clean up code so the tool doesn't assert when exiting.
Extend h5debug to support 3-D datasets for chunks.
* commit 'c03ee563f46013d22f36a1895664a9ba876558e9':
Further updates to the tools warnings fixes from code review.
Updates to tools warning PR from code review.
Fixes for warnings in the tools code.
* commit 'd16eb45d536bd13ca3cf1b0bdc76c58d95d13f17':
For compatibility with non-C99 Visual Studio versions, use "%" PRIuMAX instead of "%ju".
Fix `test/swmr_sparse_reader.c:118:77: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]` and `test/snapshots-hdf5/current/test/swmr_sparse_reader.c:129:100: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]`.
Fix `src/H5FDdirect.c:1346:5: error: ISO C90 forbids mixed declarations and code [-Werror=declaration-after-statement]`.
to integer of different size [-Werror=pointer-to-int-cast]` and
`test/snapshots-hdf5/current/test/swmr_sparse_reader.c:129:100:
error: cast from pointer to integer of different size
[-Werror=pointer-to-int-cast]`.
* commit '5ea58acf8a2bdf8ca19d4c86d9ef94fbc4ac449f':
Add missing ')'.
Add C++ warnings treated as error for autotools builds.
Move macro ADD_H5_CFLAGS to cmake_ext_mod/HDFMacros.cmake and remove duplicate versions.
Address PR suggestions and add overlooked GCC compiler version flags.
Don't add general warnings flags for unsupported old versions of gcc and g++ (older than gcc/g++ 4.2). Correct gnu-cxxflags to determine warnings flags to be added based on C++ compiler version instead of C compiler version.
* commit '43424a1392b662a29542bb70014c253162ccfe48':
Add missing ')'.
Add C++ warnings treated as error for autotools builds.
Move macro ADD_H5_CFLAGS to cmake_ext_mod/HDFMacros.cmake and remove duplicate versions.
Address PR suggestions and add overlooked GCC compiler version flags.
Don't add general warnings flags for unsupported old versions of gcc and g++ (older than gcc/g++ 4.2). Correct gnu-cxxflags to determine warnings flags to be added based on C++ compiler version instead of C compiler version.
* commit '1db039d1beee05d7349e153cef4cf642c907cc1e':
Fixed h5diff command-line parameters to use 1 and 2 instead of src and dst.
Added VOL command-line options to (p)h5diff, h5ls, h5dump, and h5mkgrp.
* commit '785bd0975ba13c3033820d0fc809da963fcd54ca':
Fixed h5diff command-line parameters to use 1 and 2 instead of src and dst.
Added VOL command-line options to (p)h5diff, h5ls, h5dump, and h5mkgrp.
compact datasets.
Bundle the zoo-test configuration into a new type, zoo_config_t.
Add a couple new "zoo" test phases, "delete" and "validate-deletion", to
the existing "create" and "verify" phases. Give names and numbers to
all phases with the new `enum`, `phase_t`, and refactor so that tend_zoo
runs a selection of phases at each step.
Stub the "delete" and "validate-deletion" phases for most test steps.
Actually implement for compact dataset (ds_cpt_i) test.
In tend_zoo(), delay for 50 milliseconds after running all steps.
Really, this should delay after each step....
Implement vfd_swmr_writer_may_increase_tick_to() and
vfd_swmr_reader_did_increase_tick_to() with a file that reader and
writer share. The reader saves its current tick number in the shared
file. The writer does not advance its tick number past the reader's.
Collect some statistics in vfd_swmr_writer_may_increase_tick_to() and
print them before the writer exits.
Add option flags for skipping compact dataset tests (-C) and for
printing error stacks (-e). Update the usage message, which was
stale before the new options were added.
Delete some dead code.
Add #if 0'd-out code for the reader to wait for the writer before
running "delete" and "validate-deletion" steps.
tells whether the call may wait for the reader tick to catch up.
Add stub routines vfd_swmr_writer_may_increase_tick_to() and
vfd_swmr_reader_did_increase_tick_to() for tests---e.g.,
vfd_swmr_zoo_writer/_reader---to use to coordinate their tick numbers.
vfd_swmr_writer_may_increase_tick_to(new_tick, wait_for_reader) returns
true if the writer may increase its tick number to `new_tick` without
overrunning the reader.
A reader uses vfd_swmr_reader_did_increase_tick_to() to tell a writer
that its tick number has increased.
g++ (older than gcc/g++ 4.2).
Correct gnu-cxxflags to determine warnings flags to be added based on
C++ compiler version instead of C compiler version.
* commit '9ecca22dd1852145810280e8c54aa6b0ab26b604':
Fixed problems with S3 and HDFS VFDs. * Updated info struct parameter to correct info_string * Fixed Makefile.am where the S3 VFD was listed twice, causing duplicated symbols errors.
`mdc_invalidation`, and use it to log a message when
H5C_evict_or_refresh_all_entries_in_page() does not find any affected
entries.
Pass a page length to H5C_evict_or_refresh_all_entries_in_page() so that
it can assert if a multipage eviction overlaps a single-page entry,
which had better not happen.
Fix a bug in H5F_vfd_swmr_reader_end_of_tick() and heavily rework it:
remove page-table entries and evict/refresh MDC entries that overlap
*added* shadow-index entries. Because we didn't do that before, in the
zoo test, the reader didn't see all of the changes made by the writer
until the writer closed the file: MDC entries covered the new content
in the shadow file.
In H5F_vfd_swmr_reader_end_of_tick(), log changes to the shadow index
with the new outlet, `shadow_index_update`.
Convert a some of John's disused diagnostic printfs to an
`hlog_fast(eot, ...)` call.
end-of-tick processing macro. H5F_vfd_swmr_process_eot_queue() looks
for files due for end-of-tick processing and calls either the reader or
writer EOT routine.
Always call the reader/writer EOT routines with an actual H5F_t instead
of NULL.
* commit 'da565fba7b37c0e6806adab1f59416bb0ac9041e':
Update h5repack long arguments for id-->value changes.
Changed identifiers and command-line options from "id" to "value" in tools code to be in agreement with H5VL API terminology.
* commit '2188e7d731199cac08c6e6150ccd2fb43227ec32':
Update h5repack long arguments for id-->value changes.
Changed identifiers and command-line options from "id" to "value" in tools code to be in agreement with H5VL API terminology.
pagebuffer entry size. If the sizes are different, then release the old
shadow space (using the correct size) and set the shadow page number to
0 so that H5F_update_vfd_swmr_metadata_file() will allocate new shadow
space with the right size.
Vailin says that this fixes the bug she found, where a 4096-byte buffer
allocated by H5MV_alloc() is released with H5MV_free() as if it was 8192
bytes long.
Add Splitter VFD and Mirror VFD.
* commit '664fc6af47e369152727ff66d68d8532feb0e210':
Add enable-mirror-vfd flag to configure options.
Tidying of Mirror VFD.
Add function header comment for `mirror_writer:run_writer()`.
Add Splitter VFD to library.
commit 8963c3bf756f8f8ec21beea9bd29a767e77675a8
Author: Larry Knox <lrknox@hdfgroup.org>
Date: Wed Apr 8 16:52:27 2020 -0500
Commit changes to gnu-cxxflags to remove unmatched " and to gnu-fflags
to not add C warnings flags to H5_FCFLAGS.
* commit 'b27a20f8c0e286ea7c2836605632ea2ee5522d03': (59 commits)
Hide Frortran directive warnings
Updated test ddl files so CMake passes after h5dump failures.
Remove flag from gfort-general in gfort-5, add /EHsc to Win C++
Update printf pointer syntax and revert fortran to f2003
Reduce window fortran standard to f90
Minor updates and fix settings for fortran flags
Update intel flags
Narrowed the scope of ISO_C_BINDING with ONLY additions
Fix missing endif
Correct improper file move
Move intel warnings to subfolder, update autotools files
Corrected note
Remove overzealous additions
Correct whitespace in flags, force list mode
Flag construction must be after compiler detection
TRILAB-192 add c++ and fortran warnings build systems one file
Skip testing of passthrough VOL connector in a Family File tools test
Update release note for CMake warnings
Restore previous tools library driver name behavior with passthrough VOL connector
Add missing header include
...
that don't count as VFD SWMR test failures.
The variable-length string tests are nondeterministic, so we expect them to
fail, and they do fail quite often on `jelly`, for example.
my investigation of shadow-file freespace leaks.
Copy the tick number from the H5F_shared_t to a temporary variable
and use the temporary instead of spelling out `shared->tick_num`.
Assert that every entry on the tick list is dirty, since clean
entries will not appear on the tick list. Mark each corresponding
shadow-index entry dirty, and set its "tick of last change" to the
current tick and "tick of last flush" to 0. This makes the scan
for entries that do not appear on the tick list more believable.
Lower staircase in the loop that scans for shadow-index entries that
did not appear on the tick list. Log indicies that are marked for
reclamation.
* Rename server-stop utility to mirror_server_stop.
* Remove external dependency on bzero().
* Modify test/use_common to use only the public API.
* Rename internal bitswap macro to follow convention.
* commit 'f362e14add76bc137c8f9bb8542a697e48900b91':
Hide Frortran directive warnings
Remove flag from gfort-general in gfort-5, add /EHsc to Win C++
Update printf pointer syntax and revert fortran to f2003
Reduce window fortran standard to f90
Minor updates and fix settings for fortran flags
Update intel flags
Fix missing endif
Correct improper file move
Move intel warnings to subfolder, update autotools files
Corrected note
Remove overzealous additions
Correct whitespace in flags, force list mode
Flag construction must be after compiler detection
TRILAB-192 add c++ and fortran warnings build systems one file
Update release note for CMake warnings
Add missing header include
* commit 'f6514c35ea3786206a8db73e1cad8fce8fe6a715':
Updated test ddl files so CMake passes after h5dump failures.
Modifications based on PR review feedback.
Add two routines gen_ref_files() and gen_sel_files() to generate test files related to references and selections. These test files will be copied to 1.12, 1.10 and 1.8 for compatibility testing.
* commit 'f6514c35ea3786206a8db73e1cad8fce8fe6a715': (48 commits)
Updated test ddl files so CMake passes after h5dump failures.
Narrowed the scope of ISO_C_BINDING with ONLY additions
Skip testing of passthrough VOL connector in a Family File tools test
Restore previous tools library driver name behavior with passthrough VOL connector
Updated tools dump output when dataset offset is undefined.
Fixed unnecessary H5R casts in the tools that were raising warnings.
Added checks for native optional call support in some of the tools.
Changed default dataset shared struct to initialize hid_t IDs to H5I_INVALID_HID.
Correct extra flags
Modifications based on PR review feedback.
Minor renaming in the tools code.
Misc changes for h5dump VOL changes.
Fix for HDFFV-11065.
Fix latent bug in h5repack options file reading Add 'enable-error-stack' option to h5format_convert
TRILAB-192 remove comment
Minor renaming in h5dump (addr --> token)
TRILAB-192 add comparable clang flags
Fix some missing symbols issues in tools library
A few small tweaks to the tools from review
Replace usage of some standard library functions with HD- versions
...
* commit '3f081c99376ad562dcb6bbf7d6c11216f682aca7':
Modifications based on PR review feedback.
Add two routines gen_ref_files() and gen_sel_files() to generate test files related to references and selections. These test files will be copied to 1.12, 1.10 and 1.8 for compatibility testing.
* commit '4830a1796441abbb5a4fbd2cff9a651f2a0d066e':
Updated tools dump output when dataset offset is undefined.
Added checks for native optional call support in some of the tools.
Misc changes for h5dump VOL changes.
* commit '110cafb9c7b5b9f9170340432062295d630f3ee6':
Updated tools dump output when dataset offset is undefined.
Added checks for native optional call support in some of the tools.
Misc changes for h5dump VOL changes.
eventually reclaimed.
Defer reclamation of raw data filespace, only.
XXX Deferring only raw-data reclamation isn't *quite* sufficient because
XXX a writer could conceivably reuse metadata space as raw-data space
XXX before max_lag ticks have elapsed. Readers could see metadata
XXX corrupted by raw data.
Once a file starts to close, stop deferring reclamation.
In H5MF_free_aggrs(), perform all deferred reclamations if the file is
closing.
* commit '5d1590079ea0061a7b86d7210cb796b6d58a17c7':
Skip testing of passthrough VOL connector in a Family File tools test
Restore previous tools library driver name behavior with passthrough VOL connector
* commit 'a5a2fb44296327787476407a249d9470b3c80427':
Skip testing of passthrough VOL connector in a Family File tools test
Restore previous tools library driver name behavior with passthrough VOL connector
* commit 'bae05235a2d87acb0d6b3a2e1c32f3b6f48cf203':
Changed default dataset shared struct to initialize hid_t IDs to H5I_INVALID_HID.
A fix in the cleaning up code for datatype when datatype initialization via H5D__init_type() fails. This is triggered by the tests for revised references when the libver bounds setting does not allow version 4 datatype message to be created. The test failure is abort core dumped. This is due to the datatype initialization fails before the datatype ID is registered. The datatype cleanup code should provide for the above situation. The code to fix the problem is the same as what is done in H5D__open_oid().
Remove tongue-in-cheek credit for Rusty Shackleford and Dale Alvin Gribble.
Follow HDF5 conventions.
* commit 'afdcac28b85d690eebc80ab568fa5266081baaaa':
A fix in the cleaning up code for datatype when datatype initialization via H5D__init_type() fails. This is triggered by the tests for revised references when the libver bounds setting does not allow version 4 datatype message to be created. The test failure is abort core dumped. This is due to the datatype initialization fails before the datatype ID is registered. The datatype cleanup code should provide for the above situation. The code to fix the problem is the same as what is done in H5D__open_oid().
validate_zoo()/check_zoo(), instead of checking/modifying the global
variable `pass`, just return `false` on failure, `true` on success.
Update test `cache_image` to match.
available to other files. Make the tick number a parameter instead of
using the f->shared->tick_num, so that we can pass a maximal tick number
and that way reclaim all of the deferred frees.
* commit 'f5cb547825e462c3299b6f41d03a71aa6faef950': (29 commits)
Minor renaming in h5dump (addr --> token)
Correct macro check
Add extensive warnings to tools executables
Reorg comments
Correct warning flag form
Because of tools macro, this needs to be a warning
unused set-variable in macro
Fix Werror issues in JNI and tools
TRILAB-192 fix JNI shadow warning
TRILAB-192 restrict extensive warnings to libraries
TRILAB-24 use CHECK_STRUCT_HAS_MEMBER
TRILAB-192 add release note
TRILAB-192 Identify warnings that fail as errors
Correct failure when allocation tracking are disabled.
Update the MANIFEST for the autoconf/cmake shared warnings files.
Correct syntax
Update CMake compiler flags to use same set of warnings
Revise API for H5get_alloc_stats() to take a struct instead of separate values.
Allow use of INTEL icl/icl++ for clang
Move the GNU warnings configuration under config/gnu-warnings/ and trim the filename prefixes. Update config/gnu-flags to suit.
...
* commit 'd5bbf3d4086d7fa0300325a08a608f1cc7b4ce33':
A few small tweaks to the tools from review
Replace usage of some standard library functions with HD- versions
Update Tools library to be better compatible with VOL connectors
* commit 'ea952b25ba0fd5c4b5f60a94e75a1b5b78e66172':
A few small tweaks to the tools from review
Replace usage of some standard library functions with HD- versions
Update Tools library to be better compatible with VOL connectors
the first portion of a metadata cache entry in speculative reads.
This is necessary for VFD SWMR as it presumes that metadata entries
are read and written atomically. See comments in H5C.c / H5C_load_entry()
for further details.
* commit '27b23c8d52e89e9ce15dc9cb9685be85793a5d82':
Add extensive warnings to tools executables
Reorg comments
Correct warning flag form
Because of tools macro, this needs to be a warning
unused set-variable in macro
Fix Werror issues in JNI and tools
TRILAB-192 fix JNI shadow warning
TRILAB-192 restrict extensive warnings to libraries
TRILAB-192 add release note
TRILAB-192 Identify warnings that fail as errors
Correct syntax
Update CMake compiler flags to use same set of warnings
Allow use of INTEL icl/icl++ for clang
* commit 'd3ef866a2963ce93d0b95184534a70849b6fa11d':
Revise API for H5get_alloc_stats() to take a struct instead of separate values.
Move the GNU warnings configuration under config/gnu-warnings/ and trim the filename prefixes. Update config/gnu-flags to suit.
Fixed the RELEASE.txt note for HDFFV-11057
Fix shutdown errors when using the HDF5_VOL_CONNECTOR environment variable to set a dynamically loaded plugin as the default VOL connector. Fixes HDFFV-11057
Fix threadsafe for new test
Fix issues when deserializing point/all/none selection with version beyond the library's supported version: (1) Verify the decoded version before proceeding further with deserialization (2) Close the dataspace if errors occurred after opening the dataspace
Very minor comment change in H5VLconnector.h.
Add routines to query the library's free list sizes and allocation stats.
By default, demote warnings promoted to errors back to warnings.
Incorporate the H5_ECFLAGS into the library CFLAGS. Now a bunch of errors will occur. Next commit will demote the warnings promoted to errors back to warnings again.
Break out warnings into more files that autoconf and CMake can share. This change temporarily disables the warnings that were promoted to errors, but I will add those warnings back as warnings, not errors, in the next commit.
Avoid a crash, don't check if we're flushing when the file is closing: there's no need to check if the metadata cache is flushing if we already know the file is closing, because the condition we rely on is "closing OR flushing." Further, the cache may have already gone away, so sometimes calling into the cache to see if it's flushing will crash the program.
* commit 'd3ef866a2963ce93d0b95184534a70849b6fa11d':
Revise API for H5get_alloc_stats() to take a struct instead of separate values.
Add routines to query the library's free list sizes and allocation stats.
* commit 'b51585a9110c4322c52f9dcdc60cd26c3ea44546':
Revise API for H5get_alloc_stats() to take a struct instead of separate values.
Add routines to query the library's free list sizes and allocation stats.
* commit 'e916acd018d5a307e7690a7978a1b73e83508f2b':
Move the GNU warnings configuration under config/gnu-warnings/ and trim the filename prefixes. Update config/gnu-flags to suit.
By default, demote warnings promoted to errors back to warnings.
Incorporate the H5_ECFLAGS into the library CFLAGS. Now a bunch of errors will occur. Next commit will demote the warnings promoted to errors back to warnings again.
Break out warnings into more files that autoconf and CMake can share. This change temporarily disables the warnings that were promoted to errors, but I will add those warnings back as warnings, not errors, in the next commit.
* commit '2f09d8f3c8a4a44932c1b8ffb0f6ea2219059a5c':
Move the GNU warnings configuration under config/gnu-warnings/ and trim the filename prefixes. Update config/gnu-flags to suit.
By default, demote warnings promoted to errors back to warnings.
Incorporate the H5_ECFLAGS into the library CFLAGS. Now a bunch of errors will occur. Next commit will demote the warnings promoted to errors back to warnings again.
Break out warnings into more files that autoconf and CMake can share. This change temporarily disables the warnings that were promoted to errors, but I will add those warnings back as warnings, not errors, in the next commit.
This is triggered by the tests for revised references when the libver bounds setting does not allow version
4 datatype message to be created. The test failure is abort core dumped.
This is due to the datatype initialization fails before the datatype ID is registered.
The datatype cleanup code should provide for the above situation.
The code to fix the problem is the same as what is done in H5D__open_oid().
objects as the cache_image test does. The zoo writer is a work in
progress.
This version is useful as a reproducer for the hang in the global heap
that I stumbled over, yesterday. I run this to reproduce,
env HLOG="pbio=on" ./vfd_swmr_zoo_writer -W -a
the tick number in H5F_vfd_swmr_close_or_flush() when just flushing,
because that made the assertion in H5PB_vfd_swmr__set_tick(), that the
tick number had not increased by more than one, fail.
* commit 'ce7936d2b256bdd5e61a5ef018f35e9562667cac':
Fixed the RELEASE.txt note for HDFFV-11057
Fix shutdown errors when using the HDF5_VOL_CONNECTOR environment variable to set a dynamically loaded plugin as the default VOL connector. Fixes HDFFV-11057
Very minor comment change in H5VLconnector.h.
with multiple pages. Update statistics to maintain consistency.
Refactor a bit: in H5PB__write_meta(), move code that's in both the if-
and else- branch to either before the if-else or after and de-duplicate.
In H5PB_vfd_swmr__update_index(), always update the length of a
shadow-index entry to the current size of its corresponding page-table
entry.
In H5PB_vfd_swmr__update_index(), disregard shadow-index entries that
await garbage collection.
Modify h5repack to integrate with VOL connectors
Update tools library to accomodate VOL connectors
Update logic in h5tools_fopen for VOL connectors
Add command-line options to h5repack for specifying in/out VOL
connectors
Implement h5tools_set_vol_fapl
Fix library shutdown issue
Integrate ROS3 and HDFS VFDs into new h5tools_get_fapl() scheme
Avoid H5Ocopy in h5repack when using different VOL connectors
Update h5tools_test_utils.c for ROS3 and HDFS integration
* commit '81b1ed4e1724b8a6a731ab2b8fb03234a8b49d15':
Fix issues when deserializing point/all/none selection with version beyond the library's supported version: (1) Verify the decoded version before proceeding further with deserialization (2) Close the dataspace if errors occurred after opening the dataspace
* "Simultaneous and equivalent" Read-Write and Write-Only channels for
file I/O.
* Only supports drivers with the H5FD_FEAT_DEFAULT_VFD_COMPATIBLE flag for
now, preventing issues with multi-file drivers.
Add Mirror VFD to library.
* Write-only operations over a network.
* Uses TCP/IP sockets.
* Server and auxiliary server-shutdown programs provided in a new directory,
`utils/mirror_vfd`.
* Automated testing via loopback ("remote" of localhost).
* commit '093c2c7e69bc4a40e9d49eece1d6fdcf54e8ecb0':
Avoid a crash, don't check if we're flushing when the file is closing: there's no need to check if the metadata cache is flushing if we already know the file is closing, because the condition we rely on is "closing OR flushing." Further, the cache may have already gone away, so sometimes calling into the cache to see if it's flushing will crash the program.
* commit '9abbdeaa66c70a00b6a7bedee9c76d2493a8e947': (31 commits)
Remove system command from valgrind reports
Whitespace
Fix incorrect FUNC_LEAVE macro (should match FUNC_ENTER_*_TAG).
Add semicolons to more PASSED() invocations.
So that I can use PASSED(); anywhere a statement can go, #define PASSED() with a do-while wrapper.
revert type cast
Switch the 'get offset' operation from a dataset 'get' callback to a [native] dataset 'optional' operation.
Modfy expression to match mpicc build with Intel as intel compiler, not gcc.
Complete the comment on thread_main(), explaining why the barrier is used.
Fixed issue building HDF5 with NAG Fortran 7.0.
The first implementation seemed to allow for the possibility that a thread could block at the barrier, wake and exit the barrier, re-acquire the barrier lock and increase `nentered` before the other blocked threads woke and checked `nentered % count == 0`. Then the other blocked threads would check `nentered % count == 0` and, finding it false, go back to sleep in the barrier. This new implementation waits for a looser condition to obtain so that threads don't go back to sleep in the barrier.
Add java version and reference libsettings
Test the right condition for the EBUSY return in pthread_barrier_destroy().
s/exit_failure/EXIT_FAILURE/g
Implement pthread_barrier(3) for Darwin using a counter, condition variable, and mutex. Untested.
Update examples and fix old version references.
Move MODEL check to before usage in CTestScript.cmake.
Copy generated files instead creating twice
Fix standalone link
Fix Fortran macro use and jni comment
...
page buffer. Now variable-length (VL) data such as VL strings work with VFD
SWMR. This change also makes the library more consistent in its treatment of
global-heap storage, since it's always been allocated as metadata, not raw
data.
* commit 'b53eb6fb29e556c482ce20fa7a21485c99beea74':
Add semicolons to more PASSED() invocations.
So that I can use PASSED(); anywhere a statement can go, #define PASSED() with a do-while wrapper.
unsigned integer value that tells at which index to inject a failure
for testing purposes. I used this to establish that I can start the
shadow-index lookup test with a previous seed and see the same tests run
again.
* commit '38af6ee3d59a8e836f4c62dd533ddabd914c7abb':
Switch the 'get offset' operation from a dataset 'get' callback to a [native] dataset 'optional' operation.
* commit '803d805c74466a9d736455930b17de2d9f5cb02d':
Complete the comment on thread_main(), explaining why the barrier is used.
The first implementation seemed to allow for the possibility that a thread could block at the barrier, wake and exit the barrier, re-acquire the barrier lock and increase `nentered` before the other blocked threads woke and checked `nentered % count == 0`. Then the other blocked threads would check `nentered % count == 0` and, finding it false, go back to sleep in the barrier. This new implementation waits for a looser condition to obtain so that threads don't go back to sleep in the barrier.
Test the right condition for the EBUSY return in pthread_barrier_destroy().
s/exit_failure/EXIT_FAILURE/g
Implement pthread_barrier(3) for Darwin using a counter, condition variable, and mutex. Untested.
could block at the barrier, wake and exit the barrier, re-acquire the barrier
lock and increase `nentered` before the other blocked threads woke and checked
`nentered % count == 0`. Then the other blocked threads would check `nentered
% count == 0` and, finding it false, go back to sleep in the barrier. This new
implementation waits for a looser condition to obtain so that threads don't go
back to sleep in the barrier.
* commit '13f5b3aee20d1d65f56dc08f088e0f218da5cf37':
Update examples and fix old version references.
Move MODEL check to before usage in CTestScript.cmake.
time in H5PB_dest(). While we're in H5PB_dest(), mark deleted shadow-index
entries as "garbage" and skip the O(n) shadow index-entries copy.
Rename shadow index-entry member `moved_to_hdf5_file` to `moved_to_lower_file`
while I'm in here---NFCI.
* commit 'cf58730177d60fd7311582a29539d87f10a7d88e':
Update Windows platforms
HDFFV-11036 add release note
HDFFV-11036 add file compare test process
Correct usage of add_compile_definitions
messages to them.
Move and update a comment about removing (all) items from the deferred queue
before processing them.
Bug fix: don't leak file space, add back to the deferred queue all items that
were not disposed of.
* commit '380fe7cfdfd98461cb1bff54bad4de0ee9904ad6':
Fix always true issue because member is not dynamically allocated
CMake cleanup
HDFFV-11032 fix JNI call
* commit '158ba17be0ad40e3e47a951c1340710608347772':
Replace the old H5F_DECODE_LENGTH() implementation with one that initializes the variable it's loading on every path.
* commit '50aac126d7c4082c2aeb0f3a68cad26bd1cd9783':
Start to document the function/parameter/variable attribute macros.
Filter all of the new _USED / _UNUSED type modifiers.
H5_HAVE_PARALLEL, _DEBUG_API, etc. Add attributes to some variables and parameters that are unused under some configurations.
mysteriously printing "ding ding!" when the shadow index is enlarged.
While I'm here, place the new index into place regardless of whether we succeed
at the deferred free of the old index's shadow-file space.
H5F_shared_t's queue before processing any, instead of removing just one,
processing it, removing another, processing it, and so on. While we processed
the first entry on the queue, we often called H5MF_xfree() again, which called
process_deferred_frees() again, which processed the first entry, calling
H5MF_xfree() again, and on and on, until the deferred frees list was exhausted.
This deep recursion showed up as a wide, tall stack in my flame graphs. Taking
all deferred entries off of the queue to start definitely breaks the recursion
and saves processing time.
Merged per discussion with Elena.
* commit 'b4697f969295245840350e17d75d92a0fdd7e7a9':
Use HD prefix.
Delete unhelpful comment per Jordan's question.
Make sure that H5TS_thread_id() is available as either a function or a macro in all configurations.
Provide local copies of err(3)- and errx(3)-alike functions for Visual Studio compatibility.
Provide C99/POSIX.1 format strings PRI[doux]{8,16,32,64,MAX,PTR} on systems that are missing <inttypes.h>.
Merged per discussion with Elena.
* commit '3f903a441ad84001ea66589728bd8b036b6fdfca':
Take out the temporary performance tests.
Make calls through a function pointer. Use the same number of arguments, always.
Increase iterations, provide a baseline for no-op, simplify the overhead case a bit.
Temporarily add some code that measures the time to run the simplest possible H5T__copy_all()-like routine 10 million times and then measures the version with FUNC_ENTER_STATIC/_LEAVE_NOAPI and a HGOTO_ERROR() statement.
H5T_copy() constification plus Quincey's contributions.
there's no need to check if the metadata cache is flushing if we already
know the file is closing, because the condition we rely on is "closing
OR flushing." Further, the cache may have already gone away, so
sometimes calling into the cache to see if it's flushing will crash the
program.
Delete the comment questioning whether pthread_mutex_lock is allowed
in a key destructor, since pthread_key_create(3) provides the answer:
There is no notion of a destructor-safe function. If an application
does not call pthread_exit() from a signal handler, or if it blocks any
signal whose handler may call pthread_exit() while calling async-unsafe
functions, all functions may be safely called from destructors.
Delete redundant comment.
implementation, rename hlog_log() to hlog(), hlog_vlog() to vhlog(), et cetera.
Rename hlog_lazy() to hlog_fast().
Define some log sinks and use them in the page buffer and in VFD SWMR.
* commit '1b937c69f4b8187401eeb8eef195bbcc2ea58312':
HDFFV-11014, fix the h5repack issue that misses a few attributes during the repacking. The flag that checks the object reference attribute is not updated properly. The fix is trivial. Just need to move the flag update line into the inner loop. Tested at Jelly. Also update the release.txt.
all configurations.
Previously it was neither declared nor defined in --disable-threadsafety
builds. The compiler's warning got lost in the noise---I first saw the issue
because my -Werror branch stopped compiling cold---and the tests still linked
and ran.
syntax highlighting colors some (but not all) unescaped underscores red like
there is some problem. It's possible there is some problem, since underscores
are used to indicate some kind of emphasis---probably underlining.
shadow pages. Reduce casts by choosing correct format strings and compatible
variable types.
Poison writes to addr by making it const. Don't increase addr in the read(2)
loop because it's never used afterward.
Delete some more dead code.
Rename read_ptr as p and declare it much closer to its use. Change its type to
`char *` so that no casts are necessary to increase it.
for them did not help me keep track of what they were for.
For brevity, I will call a deferred free record a "defree" in the code.
The deferred_free_queue_t becomes a lower_defree_queue_t, and each record on
the queue becomes a lower_defree_t. A lower_defree_t tracks one deferred free
on the lower VFD---that is, the one under the SWMR VFD.
The old_image_queue_t becomes a shadow_defree_queue_t, and a record therein is
a shadow_defree_t. A shadow_defree_t tracks one deferred free on the shadow
file.
* commit 'a0a75e7b6e157b40bbe09c7955153835ce200f79':
If H5_HAVE_THREADSAFE is not #defined, define nothing but a stub implementation of H5TS_thread_id().
* commit 'cefacee21b5ad569e2394b32793648e1b80f3d6b':
src/H5Eint.c: #include H5TSprivate.h for H5TS_thread_id() definitions.
Change thread IDs to uint64_t from unsigned long, per Quincey's suggestion.
Add thread_id.c to the MANIFEST and the CMakeLists.txt per Allen's request.
Replace pthread_self_ulong() with H5TS_thread_id(). The POSIX Threads implementation ought to be portable to any system that has POSIX Threads. On Windows, I use the same API call as before.
Add to the H5F_shared_t (!) a new member that tells the index in
the shadow file where the index should be written.
Allocate shadow filespace for the header and the index separately so
that the index can float. Update tests to match the expected original
location of the index.
Introduce vfd_swmr_enlarge_shadow_index(), a routine that allocates space in
the shadow file for a new index that has (up to) twice as many entries as the
old index, allocates a new in-core index of the same size, and copies the old
in-core index to the new. Call vfd_swmr_enlarge_shadow_index() in
H5PB_vfd_swmr__update_index() when the in-core index has too few slots.
In the comment at the top of H5FD__vfd_swmr_load_hdr_and_idx(), describe the
protocol that it follows, now, when it reads the shadow header and index.
Delete some dead code in the function and add a bit of diagnostic code.
TBD quiet the diagnostic code.
In H5F_vfd_swmr_init(), follow the protocol: write the index, first, then
the header.
Modify property-list checks and tests to reserve no fewer than two pages at the
front of the shadow file for the header and index.
an H5F_shared_t because the new routine that will relocate the index (which
will be in a future commit) has to pass an H5F_t to the filespace allocator.
* commit '10301154215fb6a22590e80f7b5ed0005f3e1786':
Minor refactoring to the VFD info free call.
Updated the 'const memory free' changes based on PR feedback.
Added a free wrapper that lets us free constant pointers without generating warnings.
* commit 'c5e2d975734d6e7a88fb48b76ec7e11d0a7763de':
Small fixes for tools and skip autotools tests for no filter
Correctly manage warnings for new tests
Ext libs ZLIB and SZIP warnings are separate concerns
* commit '30ca0cdc9a0b1f7fc1521007c65d9baeb69c9a23':
fix bad function cast warning
initialization discards const warning
testpar/t_2Gio.c: Fix a typo that I think was introduced by a previous warnings PR. An array element was assigned to itself---shape[2]Â =Â shape[2];---instead of being assigned to chunk[2].
Reduce casts of HDcalloc()/HDmalloc() that -Wc++-compat required.
Reduce differences between my -Werror branch and `develop`:
fix unused function warning
* commit 'f19e06b59ee53f17465a7b5974c25c3245a40d9a':
testpar/t_2Gio.c: Fix a typo that I think was introduced by a previous warnings PR. An array element was assigned to itself---shape[2]Â =Â shape[2];---instead of being assigned to chunk[2].
previous warnings PR. An array element was assigned to
itself---shape[2]Â =Â shape[2];---instead of being assigned to
chunk[2].
fortran/src/H5Pf.c: move conditional compilation controlled by
H5_NO_DEPRECATED_SYMBOLS outside of a function for readability.
fortran/src/H5match_types.c: put a variable's declaration under the same
conditional compilation (H5_FORTRAN_HAVE_C_LONG_DOUBLE) as its
use.
For now, skip compilation of some unused debug dump routines in the JNI.
While I'm in the JNI, delete a set-but-unused variable.
src/H5Z.c: condition a variable declaration on H5_NO_DEPRECATED_SYMBOLS
so that it's not declared but unused or vice versa.
test/cache_common.h: add an #include in to get some symbols we need to
avoid implicit declaration warnings.
test/dsets.c: use a more conventional conditional-compilation syntax.
test/dt_arith.c, test/fillval.c: initialize a bunch of uninitialized
variables before use.
test/vfd.c: pass the expected type of `void **` to posix_memalign(3)
instead of `int **`.
testpar/t_bigio.c: explicitly compare with 0 instead of using ! when
"equal to 0?" is the question not "is false?" Repair some
indentation while I'm here.
testpar/testpar.h: repair misaligned line-continuation backslashes in a
macro that probably should be a function so that we don't have
to fiddle with the line continuation to begin with.
tools/src/h5repack/h5repack_main.c: fix some compiler fussing about
enums.
tools/test/perform/pio_engine.c: the compiler fusses if you cast a
function call returning double directly to off_t. It's ok if
you cast a variable that's a double to off_t, however. Write
and use a new function, sqrto(), to avoid the cast warnings.
Reduce gratuitous casts---e.g., (size_t)1.
Use the right format string for a pointer.
In the H5C sanity checks, change a "size increase" variable from ssize_t
(too narrow) to int64_t (wide enough).
Parenthesize every appearance of `storage` in the macro
`H5D_CHUNK_STORAGE_INDEX_CHK(storage)` so that you can pass in an
expression like &sc and it works properly.
Disallow re-assignment of the `dset` parameter to H5D__chunk_init()
because it helped assure me that it's safe to replace the repeating
expression `&dset->shared->layout.storage.u.chunk` with `sc` throughout.
Replace lengthy expressions such as
`&dset->shared->layout.storage.u.chunk` with `sc` throughout several
functions in H5Dchunk.c ISTR that the compiler warned that `sc` was
declared but unused in a couple of functions, and then I found that `sc`
could be used in many places. Maybe the disused `sc` appeared because a
bunch of code was copied and pasted, I don't know. Anyway, it's a lot
tighter code now that I use `sc`.
In H5D__chunk_update_old_edge_chunks() and H5D__chunk_delete()
I actually expand `sc` and another temporary variable, `pline`,
because they're used only in !defined(NDEBUG) code. This squashes
unused-variable warnings in the defined(NDEBUG) configuration.
Don't drop the `volatile` qualification with a cast in
tools/src/h5import/h5import.c.
* commit '4276e37eab01c4d67e0d669fff28d3eac31cc806':
Remove commented warnings line
Note for common warnings files
Create common build system files for warnings
* commit '71c050f837149a0c11e0936e661047c091deaa2f':
rename macro
fix issues from test fail
change condition
pick up from Dave's fix
leave Wswitch-default for later fix
fix Wredundant-decls, Wswitch-default, Wdeclaration-after-statement, Wsign-compare, Wmisleading-indentation, Wshadow
* commit '38d3834c54ee1dfa7366ae1c581909f0cf7fa000':
Add another warning variation
Correct ignore warnings
Only use sanitizer with Clang
Remove cmake macros from valgrind tests
* commit '3b804074bd358b4a1f5b8ace30f53c4a5b250bf8':
Add another warning variation
Correct ignore warnings
Only use sanitizer with Clang
Remove cmake macros from valgrind tests
the way that the shadow header and shadow index are loaded.
In H5FD__vfd_swmr_load_hdr_and_idx(), adopt a new protocol for reading
the shadow file:
0 If the maximum number of retries have been attempted, then exit
with an error.
1 Try to read the shadow file *header*. If successful, continue to 2.
If there is a hard failure, then return an error. If there is a failure
that may be transient, then sleep and retry at 0.
2 If the tick number in the header is less than the tick last read by the VFD,
then return an error.
3 If the tick number in the header is equal to the last tick read by the
VFD, then exit without doing anything.
4 Try to read the shadow file *index*. If successful, continue to 5.
If there is a hard failure, then return an error. If there is a failure
that may be transient, then sleep and retry at 0.
5 If a different tick number was read from the index than from the index,
then continue at 0.
6 Try to *re-read* the shadow file *header*. If successful, continue to 7.
If there is a hard failure, then return an error. If there is a failure
that may be transient, then sleep and retry at 0.
7 Compare the header that was read previously with the new header. If
the new header is different than the old, then we may not have read
the index at the right shadow-file offset, or the index may have been
read in an inconsistent state, so sleep and retry at 0. Otherwise,
return success.
Simplify H5FD__vfd_swmr_header_deserialize() and
H5FD__vfd_swmr_index_deserialize(). Remove their retry loops. Make
each return TRUE on success, FALSE on an error that may be transient,
and FAIL on an irrecoverable error.
In H5FD__vfd_swmr_header_deserialize(), do not check the size of the
shadow file with fstat(2), since the read(2) will fail if the file is
too small. This saves us a system call.
Lightly consti-ify H5FD__vfd_swmr_index_deserialize() arguments.
In H5FD__vfd_swmr_load_hdr_and_idx():
Consolidate all of the retry-looping. Increase the initial
retry delay from 1ns to 1/10s. Delete the disused maximum-retry
constants.
Use #if 0 to disable some error-checking code that ought to be
unnecessary under the new protocol.
Don't memset() the header and index header, but make sure
they're fully initialized with real content, instead.
* commit '590aaff33046df99a4d88ba59e4b461e060b36e4':
Optimized the floating point comparisons a little bit.
Fix for failing h5diff tests involving floating-point compares.
Removed H5_DEC_ENUM
Revert "Revert "Moved -Wunsuffixed-float-constants to the developer warnings.""
Revert "Moved -Wunsuffixed-float-constants to the developer warnings."
Moved -Wunsuffixed-float-constants to the developer warnings.
Fixed a bug in testpar/t_cache.c concerning checking expected vs. actual cache entry reads and writes.
* commit '9475ee5d59e9ac916271a64c5372dfc28f709417':
Optimized the floating point comparisons a little bit.
Fix for failing h5diff tests involving floating-point compares.
* commit '6486e06545bf637d46e18a787395542f4fca143a':
Revert "Moved -Wunsuffixed-float-constants to the developer warnings."
Moved -Wunsuffixed-float-constants to the developer warnings.
Fixed a bug in testpar/t_cache.c concerning checking expected vs. actual cache entry reads and writes.
* commit 'edcba1ce2e11f80d20fcce8f4cfacae1308bf839':
more cleanup
Max Library "index" should be 7.
not use hdf5 private header
fix float type cmp warning
fix float type cmp warning
fix missing prototype warning
fix uninitizlized warning
change it back
pick up missing piece
fix unused related warnings
removed unused parameter
more fix and address comments
remove unsed var,function,macro, etc
* commit '145ef3ceee20c28a443bd11507c58858bea3f889':
more cleanup
change it back
pick up missing piece
fix unused related warnings
removed unused parameter
more fix and address comments
remove unsed var,function,macro, etc
* commit 'a250af4916b0625bfda2ca259d58d2ca41ed0a88':
Remove irrelevant sentence from comment.
Remove duplicate lines noted on forum by Eric Bavier
Add v111 version for H5O functions to enable version1 default for HDF5 1.8-1.10 and version 3 default for HDF5 1.12. Version 2 functions are available (deprecated) but will not be the default for any version.
* commit 'a92cfed71e0da0ff61af2550640e38bf6d415fee':
Remove irrelevant sentence from comment.
Remove duplicate lines noted on forum by Eric Bavier
Add v111 version for H5O functions to enable version1 default for HDF5 1.8-1.10 and version 3 default for HDF5 1.12. Version 2 functions are available (deprecated) but will not be the default for any version.
* commit 'a63f22e27cad74c9bb98ca2b4d2a5fd118ef3571':
Only 2 versions of H5O_info_t and H5O_iterate_t.
PATH_MAX fix in efc test for Windows.
H5vers.txt has versions v10 that should be v110 and some missing v110 entries.
* commit '66a94df13b3801547523f1ea90bc84fe2ee6824b':
Removed UIUC from copyright, added helpful comment.
Updated MANIFEST
Added a config file for ARM (Tested on a Raspberry Pi 3B+ w/ gcc 6).
Tidying from code review.
Fixed stack and frame size warnings. Not complete, but fixes most of the easier cases.
* commit '44739ccd4715cc9e15595575700e945341fcdbcc':
Tidying from code review.
Fixed stack and frame size warnings. Not complete, but fixes most of the easier cases.
* commit 'c7292efc1879cdbbbeae9503fbaf170bd19c8383':
Removed UIUC from copyright, added helpful comment.
Updated MANIFEST
Added a config file for ARM (Tested on a Raspberry Pi 3B+ w/ gcc 6).
* commit 'a7648879d729c1b75bd32f3a151831e9cbfbe31c':
Add test for reference shutdown issue
H5R: set app ref when incrementing ref_count on location held by reference (fix HDFFV-10992)
* commit 'f73103745e9b9165e9d399ddb5b9991d7fb2a9e5':
HDFFV-11001 fix if block
HDFFV-11001 need to qualify all by parallel or serial types
HDDFV-11001 add note
HDFFV-11001 Add fine control over testing
Adjust regex for warnings
Update tools hid_t declarations with H5I_INVALID_HID
simplify H5FD__vfd_swmr_index_deserialize(): reuse
h5_retry_init()/h5_retry_next() for retry loops.
Don't wait for the fstat(2) to read the correct size, because the
read(2) will return short if the file isn't long enough. (This change
should save at least one system call, always.)
Leave a bunch of comments about the changes that I will have to make so
that the shadow index will float.
NFCI: do not cast H5MM_malloc() return values, this is not C++.
compares both new and old shadow indices and calls H5PB_remove_entry()
on each entry that was in the old index but is not in the new.
Ever since H5PB_remove_entry() started removing shadow index
entries, it has been possible for H5F_vfd_swmr_reader_end_of_tick()
to walk past the end of the new shadow index or even to skip entries
in the new index. Sometimes an assertion failed when that happened.
I have restructured the code in H5F_vfd_swmr_reader_end_of_tick()
so that it compares the old and new indices, gathering a list of
removed pages, in one step. In the next step, it processes the
list of removed pages, calling H5PB_remove_entry() on each page.
In the step after that, it notifies the metadata cache of each
removed page. This fixes the bug I described, above.
`if ((p = allocate(...)) == NULL) { }` into two statements, `p =
allocate(...); if (p == NULL) { }`, put a semicolon at the end of an
HGOTO_ERROR(), remove comments /* end if */, /* end for */ after closing
curly braces.
pointer to the metadata index instead of copying the index itself. Use
struct assignment instead of copying individual struct members. Lower a
staircase.
names. While I am here, do not copy the last element of the index over
the element that's being deleted, because in the very next step I'm
shifting all elements over by one.
`uint64_t` to `size_t` because it describes the size of an in-core
structure as well as an on-disk one, and `size_t` is wide enough
to store the size of any in-core structure, while `uint64_t` may
be much too wide. Check that `index_length` is no more than SIZE_MAX
after we read it.
* commit '21f3d628d702bfd6c9cbb4b90f18842023ea40b3':
Separate result variables for the function performed
Check for actual content before using file
HDFFV-10995 - add missing import
HDFFV-10995 - need to add import
HDFFV-10995 - Update all time instances
HDFFV-10995 adjust testing to tolerate non-US locale
* commit '0a2bb11b248df6841daabca3970df5d8504adfc7':
address problems from comments
fix and address comments
change according to previous comments
add missing piece
remove unnecessary check macro
fix intel compile warnings
Revert "fix warnings from Intel compiler"
Revert "fix warnings and some text alignment"
Revert "let hdf5 pick up the right compiler in Intel environment"
Revert "fix issues from previous PR comments"
Revert "using a different MACRO"
using a different MACRO
fix issues from previous PR comments
let hdf5 pick up the right compiler in Intel environment
fix warnings and some text alignment
fix warnings from Intel compiler
* commit '1a50056dafc8e22facda675acb43a71705fdd15c':
Changed -Wformat-truncation to level 2 in CMake w/ gcc.
Changed -Wformat-truncation's level from 2 to 1.
* commit 'b6bc749eac68335004ad80edc0f4e28322631d15':
Grep needs to search error file
Correct flaws in test scripts and files
grep for failure text in error stack
Add list to function arg
Test files for new ref types
h5diff test script files
Add ref changes to test script
* commit 'c2ca33dcfa340af603399036dfddad0ebbd72f6f':
Remove unnecessary H5CX call
Refactor H5Dvlen_get_buf_size to use optional dataset operation, with generic fallback for VOL connectors that don't implement operation
Cleanups from PR reviews
Refactor all the 'H5VL_*_optional' callbacks to move the type of operation out of the va_list, so it's at least possible for another connector to know what the operation is and decide whether to implement it or not.
* commit 'b55a584fd5b3aac6f2514fba41a6182030b497ae':
In portable shell scripts (using #!/bin/sh) we have to use single square brackets, [ ], instead of the bash-ism double square brackets, [[ ]].
Make this script work on hosts where PERL is not installed at /usr/bin/perl.
of the va_list, so it's at least possible for another connector to know what
the operation is and decide whether to implement it or not.
Added a new VOL sub-class called "introspect" where callbacks that report
information about the connector or container can be placed. Added an
'opt_query' callback to this sub-class, for a connector to report back
to the library whether a particular optional callback operation is supported.
Also added a 'get_conn_cls' introspection callback, to retrieve the H5VL_class_t
of a connector (either the "current" connector, H5VL_GET_CONN_LVL_CURR, or
the terminal connector, H5VL_GET_CONN_LVL_TERM).
Moved the "post open" operation from a file 'specific' operation to a file
'optional' operation, now that it's possible to detect (with the 'opt_query'
introspection callback) whether a VOL connector implements an optional
operation, without just returning an error.
Added new internal VOL helper routines: H5VL_object_is_native, to determine
if an object is in (or is a) native file, and H5VL_file_is_same, to determine
if two objects are in (or are) the same terminal VOL connector's container.
(And moved the special handling for FILE_IS_EQUAL operation out of internal VOL
callback routine into H5VL_file_is_same)
Made new dataset 'get' operation for H5Dvlen_get_buf_size, aligning it better
with other 'get' operations in API.
Fixed several issues with pass-through connectors, which are now passing the
'make check-passthrough-vol' tests again.
A bunch of warning and style cleanups as well.
* commit 'ec4418a75716b7f2ce53e627de28d139b281b545':
In portable shell scripts (using #!/bin/sh) we have to use single square brackets, [ ], instead of the bash-ism double square brackets, [[ ]].
Make this script work on hosts where PERL is not installed at /usr/bin/perl.
* commit '817235bb60e1f79c4b22b4a6116a1594aa75b81d': (145 commits)
Remove const
Fix compile error - declaration after executable statement
Adjust cache.c only variables.
Fix include to correct memory calls - big-endian issue.
Update h5debug to retrieve file pointer through VOL framework
Minor whitespace
Remove duplicate instance
Revert and move declaration
Correct struct access
Fix duplicate and varname
Fix compile and test issues from DT
Modify H5VL initialization routines to initialize all VOL-managed object types. Modify H5VLwrap_register() to reject non-VOL-managed object types. Also fix overisights in h5trace.c from previous changes.
Add release note for sanitizer support
HDFFV-10979 cleanup globals
TRILABS-135 Add clang analyzers
HDFFV-10979 fix global name clash
Fix issues found with ONLY_SHARED_LIBS option
Fix 2010 compile issues
Change from using H5Dcreate to H5Dcreate2
Latest date first in RELEASE.txt
...
Delete the little-used free-list length, dl_len, and just count up the list
entries when diagnostic code needs the length.
Extract the code for deferring shadow-image free into a new subroutine,
`vfd_swmr_idx_entry_defer_free()`.
Rename type `deferred_free_head_t` as `deferred_free_queue_t`.
Remove the disused H5F__LL_{REMOVE,PREPEND} macros.
Add some diagnostic code and #if 0'd assertions.
Change `qsort(ptr, n, sizeof(type), cmp)` to
`qsort(ptr, n, sizeof(*ptr), cmp)`.
Use a `continue` statement to lower a staircase in
H5F_update_vfd_swmr_metadata_file().
Add vfd_swmr_mdf_idx_entry_remove() to delete a shadow index entry and add the
image at that entry to a deferred-free list. Call it whenever a page is
evicted.
Update the comment in H5PB_remove_entry() that asks if we need to remove shadow
index entries: now we *do* remove them. Remove shadow index entries in
H5PB__evict_entry(). Also mention in the comment that the index-entry removal
performed by H5PB__evict_entry() ought to be sufficient.
alignment. The VFD SWMR code had always assumed that the regions were aligned
to page size. It would blithely round the start addresses of regions to the
next lower page. When the region was freed, the freespace manager (H5MV) would
suffer an assertion or corruption.
HDF5 file and in the shadow file. I had added assertions that the page numbers
were unique, and this caused those assertions to fail. I don't know if I'll
keep the assertions, but this is an inexpensive change that makes the test more
realistic.
* commit '9ee8c599d7b8cd495efb28cd9f068192712ae086':
More fixes for previous committed PR #2079 dated Dec 5 2019. (1) H5O_dtype_ver_bounds[] for V112 should be H5O_DTYPE_VRESION_4 (2) The tests for the new reference types should work for V112 and beyond
* commit 'b42325e8f5ff3b6bfa2ce446af5b6dc5cbbff666':
More fixes for previous committed PR #2079 dated Dec 5 2019. (1) H5O_dtype_ver_bounds[] for V112 should be H5O_DTYPE_VRESION_4 (2) The tests for the new reference types should work for V112 and beyond
* commit 'dc2cbdaf6720b6c3af72846989dfdbec4f207208':
Revert and move declaration
Correct struct access
Fix duplicate and varname
Fix compile and test issues from DT
Add release note for sanitizer support
HDFFV-10979 cleanup globals
TRILABS-135 Add clang analyzers
* commit 'ce653ff82b7dd52b8e36a5eff7b1728c4b75d9f7':
Modify H5VL initialization routines to initialize all VOL-managed object types. Modify H5VLwrap_register() to reject non-VOL-managed object types. Also fix overisights in h5trace.c from previous changes.
* commit 'c087ef41b637686ae5e783bfb0a48e91dbe3f58e': (28 commits)
Add H5Rdestroy calls to cleanup
Remove problem tests for now
Add new test reference
HDFFV-10876 fix compare reference
HDFFV-10876 ignore old reference API tests
Add missing quotes
Fix windows 10 compile error
Correct indentation
Revert declaration used in macro.
Revert declaration deletion.
HDFFV-10876 add note
Address compile warnings
Fix compile errors - mostly in jni
HDFFV-10876 Update dump to match DDL spec
Add extra line
Adjust whitespace
Update h5ls help and rework dump_mem routines for attrs
Update create calls
Add debug compile option as comment
Update reference calls
...
including the merge of `hdffv/hdf5/develop`, back to the branch that Vailin and
I share.
Now I need to put this branch on a fork with a less confusing name than
vchoi_fork!
* commit '9c3900d0bf05a65e59cce6dc613b38b240599d93':
Add note to RELEASE.txt for H5Sselect_adjust.
Add RELEASE.txt note for H5Sselect_project_interesection
Minor tweaks noticed while going over VOL documentation.
* commit 'b816f4c4aaf9bc80dbde35158b520e693f135fa2':
Fixed PR issues pointed out by Allen
Remove the Copyright UofI per Larry's guidance
Adds a new t_2Gio.c MPI test
* commit '7dfeab4863a25a79999fa1e5ede12a1aa685ca97':
Fixed PR issues pointed out by Allen
Remove the Copyright UofI per Larry's guidance
Adds a new t_2Gio.c MPI test
* commit '107bcbd3dfff51895c99ea3ff71afa0ce693a956':
Implement support for using H5Dvlen_get_buf_size with non-native VOL connectors.
Add H5VL_MAP_OPTIONAL operation id (unused currently).
Fix H5VL_token_t type and fix H5VL_loc_by_token to use H5VL_token_t *
Change hdset_reg_ref_t and H5R_ref_t from arrays of unsigned char to structs containing those arrays. Encapsulating the arrays in this way makes it easier to write and think about pointers to these types, casts to/from these types, etc.
* commit '107bcbd3dfff51895c99ea3ff71afa0ce693a956':
Implement support for using H5Dvlen_get_buf_size with non-native VOL connectors.
Two fixes: (1) Set the version for reference datatype messge to H5O_DTYPE_VERSION_4. (2) Verify the decoded version for hyperslab selection.
Add H5VL_MAP_OPTIONAL operation id (unused currently).
Fix H5VL_token_t type and fix H5VL_loc_by_token to use H5VL_token_t *
Change hdset_reg_ref_t and H5R_ref_t from arrays of unsigned char to structs containing those arrays. Encapsulating the arrays in this way makes it easier to write and think about pointers to these types, casts to/from these types, etc.
* commit '7b03a1c03633d695b487642e54f897c715f8622e':
Fix H5VL_token_t type and fix H5VL_loc_by_token to use H5VL_token_t *
Change hdset_reg_ref_t and H5R_ref_t from arrays of unsigned char to structs containing those arrays. Encapsulating the arrays in this way makes it easier to write and think about pointers to these types, casts to/from these types, etc.
* commit '1e10b3212ed5904b69d9f4c4bcb4d0122dfd7c77':
Two fixes: (1) Set the version for reference datatype messge to H5O_DTYPE_VERSION_4. (2) Verify the decoded version for hyperslab selection.
structs containing those arrays. Encapsulating the arrays in this way
makes it easier to write and think about pointers to these types, casts
to/from these types, etc.
An interesting side-effect that we probably should *not* rely on is
that the struct-encapsulation changes the alignment so that some GCC
warnings about casts that increase the alignment requirement of the
operand go away. Warnings like that have to be taken seriously: I will
add -Werror=cast-align to the default compiler flags so that they stop
the build quickly.
GCC warnings led me to some surprising casts in test/trefer.c. I found
that it was possible to make many simplifications after introducing the
struct-encapsulation that I described, above.
In test objcopy_ref `same_file` is assigned but never used. Delete it.
* commit 'aa0b54e27e5d6f0b7c23fe5662d5c8f30898d7f4':
Fixed missing blob callbacks in test VOL connectors.
Add support for scalar dataspaces to H5Sproject_intersection. Add checking for ank validity.
Fix issues with using H5Pget_fill_value() with file datatypes retrieved from H5VLget_file_type().
* commit 'ca34de56cd336f58c4bcf4ac9e1460f898fba3c9':
Add OAPL parameter to H5Rcreate_ APIs
Fix failing test and retain semantics for chunk cache properties in non-chunked datasets
Add two missing calls to H5I_dec_ref for new dapl_id
Community-proposed fix
* commit '3664cea5d6f87a7e8e708191f9cea36077424a80':
Fix failing test and retain semantics for chunk cache properties in non-chunked datasets
Add two missing calls to H5I_dec_ref for new dapl_id
Community-proposed fix
Check for smaller or larger section size after merging and shrinking a section,
for this case is the section that is smaller than threshold (see H5MF_xfree() in H5MF.c).
It is possible for the section to be smaller after merging/shrinking (see H5MF__sect_large_shrink()
in H5MFsection.c).
* commit '0772b975d1d2bfa15aedeb4b6e2c2aac78c61a2f':
Fix missing free in H5T__ref_mem_read()
Fix bugs in H5VL file comparison code. Add short circuit success to H5VL_cmp_connector_cls().
Implement file comparison VOL callback. Other changes to allow references to work with non-native connectors. There is a bug somewhere.
Add support for point selections to H5S_select_project_intersection.
Remove H5VL_NATIVE_FILE_GET_FILE_ID and add H5VL_OBJECT_GET_FILE
Revert "Make a squash commit of 'Quiet some warnings by adjusting warnings level and fixing some code.' (commit 5c911d8baf)"
Revert "Oops, remove more C99 designated initializers for VS 2010 compatibility."
H5R: fix H5Tconv to check for null references
Trivial parameter renaming in VOL API calls.
Move checking for zero offset in selection adjust calls to the selection callbacks. This makes the procedure for checking it consistent across selection types and between _s and _u, ensures it is always is performed even when called within the H5S package, and removes the redundant check that would occur when callins H5S_select_adjust_s() from outside the H5S package.
Replace H5Sselect_adjust_u() and H5Shyper_adjust_s() with H5Sselect_adjust. Implement "adjust_s" callback for all selection types. Add range checking to H5Sselect_adjust().
* commit '40b2658b4b266b4dc65ec3db2897e0aaf54f8e42':
Fix missing free in H5T__ref_mem_read()
Fix bugs in H5VL file comparison code. Add short circuit success to H5VL_cmp_connector_cls().
Implement file comparison VOL callback. Other changes to allow references to work with non-native connectors. There is a bug somewhere.
* commit '0108072f7a746b809952d5cc560577f931b64f71':
Revert "Make a squash commit of 'Quiet some warnings by adjusting warnings level and fixing some code.' (commit 5c911d8baf)"
Revert "Oops, remove more C99 designated initializers for VS 2010 compatibility."
* commit '4e12984b77cdd7615843d94f8de8d54db27476ac':
Move checking for zero offset in selection adjust calls to the selection callbacks. This makes the procedure for checking it consistent across selection types and between _s and _u, ensures it is always is performed even when called within the H5S package, and removes the redundant check that would occur when callins H5S_select_adjust_s() from outside the H5S package.
Replace H5Sselect_adjust_u() and H5Shyper_adjust_s() with H5Sselect_adjust. Implement "adjust_s" callback for all selection types. Add range checking to H5Sselect_adjust().
* commit 'a8892bb42d6f6e4fbc30fae0eb2b957f81c938b9': (45 commits)
Oops, remove more C99 designated initializers for VS 2010 compatibility.
Add an #include to get a function declaration.
Don't use C99 designated initializers, they're not compatible with Visual Studio 2010.
Quiet some more maybe-uninitialized warnings---each is a false positive, *sigh*. This is more code that may not compile with VS2010, *sigh sigh*.
Always warn on maybe-uninitialized. -Wincompatible-pointer-types was not available until GCC 5, so enable it only if that's the GCC version we're using.
Only promote maybe-uninitialized warnings to errors on GCC 8. Even on GCC 8, there may be false positives at low optimization levels? I need to check.
Only use -Werror=cast-function-type with GCC 8 and later.
Put all of the -W options back into the order I found them in so that it's easier to compare old and new config/gnu-flags.
Add new source files to CMakeLists.txt.
Mention the -Werror= flags in libhdf5.settings.in.
free -> HDfree
Promote decleration-after-statement warnings to errors.
Quiet decleration-after-statement warnings.
Move a statement under some declarations since some vintages of Visual Studio don't like declarations after statements.
Document H5D__chunk_mem_xfree_wrapper().
Undo accidental test deletion.
Oops, delete a debug printf that snuck in here.
Undo my changes to the HD macros, hadn't really intended those to be on this branch....
Make errors of some more warnings. Move disabled warnings to DEVELOPER_WARNING_CFLAGS. Put just one warning option on a line, and sort some of the options.
Cast to the parameter type, H5VL_token_t *, instead of to unsigned char *.
...
* commit '62208b056a09c01855fbac7f75146be58ad6bfe5': (44 commits)
Add an #include to get a function declaration.
Don't use C99 designated initializers, they're not compatible with Visual Studio 2010.
Quiet some more maybe-uninitialized warnings---each is a false positive, *sigh*. This is more code that may not compile with VS2010, *sigh sigh*.
Always warn on maybe-uninitialized. -Wincompatible-pointer-types was not available until GCC 5, so enable it only if that's the GCC version we're using.
Only promote maybe-uninitialized warnings to errors on GCC 8. Even on GCC 8, there may be false positives at low optimization levels? I need to check.
Only use -Werror=cast-function-type with GCC 8 and later.
Put all of the -W options back into the order I found them in so that it's easier to compare old and new config/gnu-flags.
Add new source files to CMakeLists.txt.
Mention the -Werror= flags in libhdf5.settings.in.
free -> HDfree
Promote decleration-after-statement warnings to errors.
Quiet decleration-after-statement warnings.
Move a statement under some declarations since some vintages of Visual Studio don't like declarations after statements.
Document H5D__chunk_mem_xfree_wrapper().
Undo accidental test deletion.
Oops, delete a debug printf that snuck in here.
Undo my changes to the HD macros, hadn't really intended those to be on this branch....
Make errors of some more warnings. Move disabled warnings to DEVELOPER_WARNING_CFLAGS. Put just one warning option on a line, and sort some of the options.
Cast to the parameter type, H5VL_token_t *, instead of to unsigned char *.
Change hdset_reg_ref_t and H5R_ref_t from arrays of unsigned char to structs containing those arrays. Encapsulating the arrays in this way makes it easier to write and think about pointers to these types, casts to/from these types, etc.
...
* commit 'b8a56718e40efaf6954a760984b9d513dbff8f7d':
Add documentation for H5CX_get_ohdr_flags
Add object header flags to API context
Use major error code of calling package
Handle H5CX_state_t for lcpl
Add character encoding to lcpl context
Change LC property name from 'create_intermediate_group' to 'intermediate_group'
Use API context to store/retrieve LCPL when creating intermediate groups
* commit '4ce9c10c568ff81ebd4eb7585039cbe3c40d1718':
Add documentation for H5CX_get_ohdr_flags
Add object header flags to API context
Use major error code of calling package
Handle H5CX_state_t for lcpl
Add character encoding to lcpl context
Change LC property name from 'create_intermediate_group' to 'intermediate_group'
Use API context to store/retrieve LCPL when creating intermediate groups
* commit '33d173d672f4affc804cf6e3c43b92693bd3ed68':
Modify H5Ssel_iter_get_seq_list() to accept iterators that have reached the end of iteration (returning zero sequences). Previously it could cause an assertion failure.
Given that the VFD SWMR configuration FAPL property is set, the writer field must
be consistent with the flags passed in the H5Fopen() (either H5F_ACC_RDWR for the
VFD SWMR writer, or H5F_ACC_RDONLY for the VFD SWMR readers).
* commit 'c104f4cb87a1558ab6a772967887738dc9800f0d':
Modify H5Ssel_iter_get_seq_list() to accept iterators that have reached the end of iteration (returning zero sequences). Previously it could cause an assertion failure.
* commit '13456df8a47fdcd8f96f170c9218e57a03066120':
Minor correction
Update list of tested platforms
Add VS2019 support
Fixed the H5_mpi_get_bigio_count function prototype
procedure for checking it consistent across selection types and between
_s and _u, ensures it is always is performed even when called within the H5S package, and removes the redundant check that would occur when callins H5S_select_adjust_s() from outside the H5S package.
The test "driver_addr != sblock->driver_addr" is failing for superblock version 2 & 3.
Fix: there is no driver_addr in superblock version 2 & 3.
It should decode the root group object header address (root_addr) and verify accordingly.
(A) #5: Add the "pb_expansion_threshold" field to the "H5F_vfd_swmr_config_t" structure
and update H5Pset_vfd_swmr_config() and H5Pget_vfd_swmr_config() accordingly
(B) #13 bullet 2: Comment H5F_vfd_swmr_config_t in H5Fpublic.h properly
(copied from John's description in the RFC)
(C) Change the field name "vfd_swmr_writer" to "writer" in "struct H5F_vfd_swmr_config_t"
(as indicated on page 11 in the RFC) and all references to it
* commit '07b59c0b63041f1be964aee2a27f5c748c29d688':
Implement public H5Sselect_project_intersection(). Updated internal algorithm to (optionally) avoid sharing selection data structures. Tested internal code (including with valgrind) by setting VDS code to avoid sharing selection, has since been changed to share selection for performance, so this code is not yet tested in regression tests. API has not been tested.
structs containing those arrays. Encapsulating the arrays in this way
makes it easier to write and think about pointers to these types, casts
to/from these types, etc.
An interesting side-effect that we probably should *not* rely on is
that the struct-encapsulation changes the alignment so that some GCC
warnings about casts that increase the alignment requirement of the
operand go away. Warnings like that have to be taken seriously: I will
add -Werror=cast-align to the default compiler flags so that they stop
the build quickly.
GCC warnings led me to some surprising casts in test/trefer.c. I found
that it was possible to make many simplifications after introducing the
struct-encapsulation that I described, above.
* commit 'db6eab893e5f027f0b535316e0e7a9abe9ff8619':
Implement public H5Sselect_project_intersection(). Updated internal algorithm to (optionally) avoid sharing selection data structures. Tested internal code (including with valgrind) by setting VDS code to avoid sharing selection, has since been changed to share selection for performance, so this code is not yet tested in regression tests. API has not been tested.
algorithm to (optionally) avoid sharing selection data structures.
Tested internal code (including with valgrind) by setting VDS code to
avoid sharing selection, has since been changed to share selection for
performance, so this code is not yet tested in regression tests. API
has not been tested.
of `hsize_t`, `start`, to `long long`, but I think the way that I have
rewritten it, it probably produces a more useful result? As a bonus,
GCC has stopped warning about it.
* commit 'e08b69d3d2e572de4707b1fbf908111d29f40a1b':
Use HD prefix.
Fix issue where H5R__reopen_file did not make the file "post open" callback.
Apparently, + has no special meaning, and neither does \+, in so-called "obsolete" / POSIX "basic" regular expressions. Also, not every version of `sed` out there supports the `-E` option. So delete the -E flag and use the regex `[^/][^/]*` instead of `[^/]+`.
Implement H5VLget_file_type() to return a copy of a datatype with the location set to be in a file. Only meant to be used by VOL connectors. Implement H5VLpeek_connector_id() to support connectors querying their own IDs. Fix app_ref with connector IDs in a couple places (external VOLs registered as default through ENV should be visible to the application). Modify vlen and reference interfaces to work with arbitrary VOL connectors. Implement file "post open" specific callback, to enable connectors to update their file structs after a wrap context has been set.
Fix HDFFV-10937: use a more reliable (and probably faster) scheme for visiting all elements of a matrix in an arbitrary order.
* commit '4d834adba4aeb1a0174bddb83212b7073b64e269':
Use HD prefix.
Fix HDFFV-10937: use a more reliable (and probably faster) scheme for visiting all elements of a matrix in an arbitrary order.
* commit 'f1b39ad80e2fc0d0f4f6ecae2a9fb0ea436c34cc':
Apparently, + has no special meaning, and neither does \+, in so-called "obsolete" / POSIX "basic" regular expressions. Also, not every version of `sed` out there supports the `-E` option. So delete the -E flag and use the regex `[^/][^/]*` instead of `[^/]+`.
"obsolete" / POSIX "basic" regular expressions. Also, not every version of
`sed` out there supports the `-E` option. So delete the -E flag and use
the regex `[^/][^/]*` instead of `[^/]+`.
Add config/netbsd to the MANIFEST.
* commit '1dbec40d465072540f8507c524ae7b909a6b1cf0':
Implement H5VLget_file_type() to return a copy of a datatype with the location set to be in a file. Only meant to be used by VOL connectors. Implement H5VLpeek_connector_id() to support connectors querying their own IDs. Fix app_ref with connector IDs in a couple places (external VOLs registered as default through ENV should be visible to the application). Modify vlen and reference interfaces to work with arbitrary VOL connectors. Implement file "post open" specific callback, to enable connectors to update their file structs after a wrap context has been set.
location set to be in a file. Only meant to be used by VOL connectors.
Implement H5VLpeek_connector_id() to support connectors querying their
own IDs. Fix app_ref with connector IDs in a couple places (external
VOLs registered as default through ENV should be visible to the
application). Modify vlen and reference interfaces to work with
arbitrary VOL connectors. Implement file "post open" specific
callback, to enable connectors to update their file structs after a
wrap context has been set.
* commit '0b721858e46a317c370a24115032d5be41688f67':
Make these scripts relocatable again: derive a relative path for the original installation prefix from the examples prefix. Use that relative path to locate the current installation prefix, always. Fall back to an absolute installation prefix if the relative path cannot be derived.
Get the path to prefix right: needs a ../ to back out of subdirectory c/.
Make this script relocatable again: derive a relative path for the original installation prefix from the examples prefix. Use that relative path to locate the current installation prefix, always. Fall back to an absolute installation prefix if the relative path cannot be derived.
Let us override the examples directory using --with-examplesdir=DIR. This is handy for NetBSD where HDF5 examples are installed by convention in $prefix/share/examples/hdf5/ rather than in ${prefix}/share/hdf5_examples/, which is the HDF5 default.
Follow longstanding execv convention for compatibility with NetBSD.
Under the examples directories, always find the installed HDF5 executables and scripts using @prefix@ instead of a relative path, because the number of ../ in the relative path will be different on NetBSD than on other systems.
Make the HDF5 configure script grok NetBSD.
For portability, insulate the HDF5 library from some system macros.
Not every system has perl installed in /usr/bin/, so change the shebang (#!) line to `/usr/bin/env perl` to locate perl on the PATH.
For portability, use the POSIX sh(1) string-comparison operator `=` instead of `==`.
original installation prefix from the examples prefix. Use that
relative path to locate the current installation prefix, always. Fall
back to an absolute installation prefix if the relative path cannot be
derived.
original installation prefix from the examples prefix. Use that
relative path to locate the current installation prefix, always. Fall
back to an absolute installation prefix if the relative path cannot be
derived.
* commit '299c9ed5fe19edac37cab27bc04d587fa67568c2':
Removed set_bigio_count prototype since it has been added to H5private.h
Change API names for get/set bitio_count. Also added H5_mpi_set_bigio_count to H5private.h.
had finished its work and closed the .h5 file, thus removing the shadow file.
Make the sparse writer wait to close the .h5 file for a signal from
testvfdswmr.sh. In testvfdswmr.sh, send the signal when the readers have all
finished.
This lets test/testflushrefresh.sh pass again. It was timing out while it
waited for expected failures to occur because the retry loop ran for way too
long.
cells in a matrix in an arbitrary order, first it chooses a random
starting `offset` in [0, rows * columns - 1]. Then it chooses a
random `increment` that's relatively prime to `rows * columns`.
Then it visits every cell in `rows * columns` steps:
for (i = 0; i < rows * columns; i++) {
visit(cell[offset / columns][offset % columns]);
offset = (increment + offset) % (rows * columns);
}
By moving the HDrandom() calls outside of the main loop and visiting
each cell only once, this probably speeds things up quite a bit. It's
also more resilient to a crummy random sequence. The new code visits
cells in an order that's probably arbitrary enough for testing purposes.
Fix encode and decode of deprecated object reference addresses
Make H5Rdeprec.c use tokens instead of haddr_t
Fix H5Oopen_by_addr() to serialize addr to token
* commit '78268762bcf9cb949f711fa0e5290f96957cb60a':
Improvements in response to Allen's suggestions.
Corrections and improvements to testh5dump.sh.in changes.
Fix errors in testh5dump.sh tests that grep for reference error messages.
to end with whitespace padding rather than newlines. My introduction of
variadic TESTING() got rid of the padding. I have straightened this out
by newline-terminating the stdout lines in the test program and in its
expected out. I also add some newlines to the program's standard error
output so that the expected error output still matches.
* commit '4c558700ab33934e7358483c5d20fea4823baf9e':
Update RELEASE.txt for reference changes
Fix reference type comparison in h5dump
Make wrappers, tests and tools use H5Treclaim() instead of H5Dvlen_reclaim()
Add new H5R API that abstracts object, region and attribute reference types
Remove ability to loc by ref from H5VL layer
Add support for retrieving object name by token
Add H5VL_OBJECT_GET_TYPE to get object type
Add H5VL_MAX_TOKEN_SIZE and H5VL_token_t
Adapt Jerome's "file info" H5VL 'get' query to retrieve container token info.
Fix H5VL_blob_get to return size of blob
Add 'blob' callbacks to VOL, along with a native implementation to store them in the global heap, and changed the VL datatype conversion code to use blobs.
Remove debugging logic from the new t_bigio test
Expanded t_bigio.c to include Jordan's test from HDFFV-10539
Another cleanup pass as suggested by the reviewers.
Fix some typos and remove an unused prototype from H5Sprivate.h
At the suggestion of the PR reviewers, moved the mpio_create_large_type to H5mpi.c and renamed the function appropriately. Also moved some support functions to set and get the vvalue where we transistion to using derived datatypes.
Made code review edit suggested by Jerome, plus various code updates to files that I touched to eliminate compile warnings (on my Linux box).
No functional changes, just removed some tab characters
Make the initial bug fixes to allow >2GB writes with Independent IO
* commit '9ed9762889fcd05dc230d6652b92fddc5ec880a4':
Update RELEASE.txt for reference changes
Fix reference type comparison in h5dump
Make wrappers, tests and tools use H5Treclaim() instead of H5Dvlen_reclaim()
Add new H5R API that abstracts object, region and attribute reference types
Remove ability to loc by ref from H5VL layer
Add support for retrieving object name by token
Add H5VL_OBJECT_GET_TYPE to get object type
Add H5VL_MAX_TOKEN_SIZE and H5VL_token_t
Adapt Jerome's "file info" H5VL 'get' query to retrieve container token info.
Fix H5VL_blob_get to return size of blob
Add 'blob' callbacks to VOL, along with a native implementation to store them in the global heap, and changed the VL datatype conversion code to use blobs.
* commit 'abb43d3d3ac2e802d0efbe21e028381ed99722f9':
Remove debugging logic from the new t_bigio test
Expanded t_bigio.c to include Jordan's test from HDFFV-10539
Another cleanup pass as suggested by the reviewers.
Fix some typos and remove an unused prototype from H5Sprivate.h
At the suggestion of the PR reviewers, moved the mpio_create_large_type to H5mpi.c and renamed the function appropriately. Also moved some support functions to set and get the vvalue where we transistion to using derived datatypes.
Made code review edit suggested by Jerome, plus various code updates to files that I touched to eliminate compile warnings (on my Linux box).
No functional changes, just removed some tab characters
Make the initial bug fixes to allow >2GB writes with Independent IO
Also support references to external files
Add new H5T_REF type and type conversion routines
Support conversion from H5T_REF_OBJ/DSET_REG to H5T_REF
Add H5Treclaim() API to reclaim memory of vlen/reference types
Deprecate H5Dvlen_reclaim()
Fix H5T_vlen_reclaim() and H5T_reclaim() to use private callback
Add H5T_ref_reclaim()
Move previous H5R APIs to H5Rdeprec.c
Clean up H5Ocopy
Separate H5O_copy_expand_ref() to H5Ocopy_ref()
Add support for copying new reference types
Clean up deprecated routines to go through VOL and same code path
Fix return codes in existing trefer.c test
Rename trefer.c to trefer_deprec.c
trefer.c is for new references
Add performance test for trefer
Add additional obj_copy_ref test
Make use of tokens and blobs to store references
Skip blob encoding for object references
Start adding new reference examples
Fix const in blob API
Add H5HG_HEAP_ID_SIZE macro to return native blob size
Add maximum size for blobs
Fix blob API callbacks to pass VOL file object
Add public wrappers for blob VOL API
Implement passthrough blob callbacks
Update H5Tvlen after callback changes
Update trace information for H5VL blob routines
Fix public header inclusion in native and passthru headers
in the global heap, and changed the VL datatype conversion code to use blobs.
Move encode/decode of sequence lengths into VL datatype callbacks, from native
VOL blob routines.
* commit 'ff3278f1a0d57663d73caa6aaa7f209864e9a9bd':
Fixed a bug (HDFFV-10899) where the behavior of the deprecated H5Gget_objinfo() call had changed and passing in a non-existing soft link with a statbuf of NULL passed instead of failed (as it should as per the RM). The HDF5 1.8 and 1.10 behavior was restored and H5Gget_objinfo() will now fail and return -1 as before.
H5Gget_objinfo() call had changed and passing in a non-existing
soft link with a statbuf of NULL passed instead of failed (as
it should as per the RM). The HDF5 1.8 and 1.10 behavior was
restored and H5Gget_objinfo() will now fail and return -1 as
before.
This is handy for NetBSD where HDF5 examples are installed
by convention in $prefix/share/examples/hdf5/ rather than in
${prefix}/share/hdf5_examples/, which is the HDF5 default.
Place hdf5_examples/ under ${datarootdir} which on most systems will be
${prefix}/share/, anyway.
executables and scripts using @prefix@ instead of a relative
path, because the number of ../ in the relative path will be
different on NetBSD than on other systems.
Examples on NetBSD are installed at ${prefix}/share/examples/hdf5/
instead of at ${prefix}/share/hdf5_examples/, by convention. It may be
the same on other BSDs, I'm not sure.
(#!) line to `/usr/bin/env perl` to locate perl on the PATH.
Everything after the first pathname in the shebang line is treated as
a single argument to the command interpreter (/usr/bin/env "perl -w"),
and there is not ordinarily any such program as "perl -w". So if the
old shebang line used an option such as `-w`, add a `use warnings;`
statement to the script---note that the semantics change slightly.
`bin/destdep` uses a trick to pass `-p` to `/usr/bin/env perl`. It
couldn't hurt to use the same trick to pass `-w`.
With these changes, `sh autogen.sh` runs on NetBSD. It ought to still
work on every other system HDF5 supports, too.
comment to myself that I need to reduce code duplication with the MPMDE
test.
In vfd_read_each_equals(), print the correct expected value when there
is a discrepancy.
No functional change intended: correct a comment in
vfd_read_each_equals. Fix indentation in the test_raw_data_handling()
header comment.
buffer's treatment of multipage metadata entries (MPMDEs). Mention why
an H5PB_flush() is not necessary for MPMDEs to reach visibility at the
VFD layer.
macro magic. Use namebases and namebase, instead.
Extract a bunch of copy-and-paste VFD SWMR setup into a new subroutine,
swmr_fapl_augment().
Make sure that the metadata reads all-0s until it reads all -1s.
Extract a subroutine, vfd_read_each_equals(), that reads and compares a
region with one of its arguments.
Rename from test_basic_metadata_handling() to
test_metadata_delay_basic(), since that gets at what we're testing
better.
Don't perform an H5PB_flush(), it's not necessary for this test because
H5Fvfd_swmr_end_tick() has done essentially the same thing.
* commit '7924eee0e5ee0745b784c635042b8633886fb799':
(1) Address the feedback from the PR review (2) Add release notes
Fix for HDFFV-10585 investigate hyperslab slowness: 1) Improve hyperslab performance when doing I/O from 1-d disjoint file dataspace to 1-d contiguous memory dataspace. 2) Move coding in H5D__chunk_io_init() that is constructing the chunk mappings to a separate routine.
* commit '4fb64c89fe52a309150ca2d5bb97eb4763e598e9':
Converted H5O MD cache cork calls to use the VOL.
Marked up H5Fget/set_mpi_atomicity() to use the VOL.
anticipate comparing the written buffer with the read buffer.
Don't initialize variables prematurely so that the compiler has a chance
to warn about variables read before they are written.
Repeatedly flush the page buffer, once each time we end the tick.
Write errors to stdout instead of stderr.
* commit '0ea77179d507bd2c29f3b9b357a0c35d4077b29c': (39 commits)
Fix the errors when updating the versions in the new 1.13 develop branch.
add missing version to list
Add new versioned filename to list
Update N_FSINFO_VERSION_BOUNDS to H5F_LIBVER_NBOUND.
Remove file script
Update versioning to next major version
Update develop branch version to 1.13.0 after creation of hdf5_1_12 branch.
Merge Binh-Minh's changes for code improvement to chunk_info.c (PR #1942) to develop branch.
Update to current external file population.
Revert else-if to single line.
Fix missing option and incorrect close
Revert "Moved NDEBUG guards to H5EA_DEBUG in H5EA package."
Revert "Fix for H5EA debug package when built in release mode."
Change print calls to HD-wrapped. Minor formatting tweaks (inc. regressions and in-file style-matching).
Refactor h5repackgentest to have more modular internals, for easier addition of new tests. Update manifest and test script for new hdf5 external-storage .h5 files.
Remove obsolete symbol
small syntax changes
Renamed get/set_time() calls in the tools library to avoid name clash when building static parallel HDF5 w/ static linking to OpenMPI.
Change unused variable to generic form
Whitespace cleanup
...
gettimeofday() alternate. Perform nanoseconds arithmetic using uint64_t
instead of long to avoid unwanted overflows on 32-bit systems like
my i386 (!) development box.
* commit '9a388325ebaec339f67ddd210ca31ddcc3ede973':
Update to current external file population.
Revert else-if to single line.
Change print calls to HD-wrapped. Minor formatting tweaks (inc. regressions and in-file style-matching).
Refactor h5repackgentest to have more modular internals, for easier addition of new tests. Update manifest and test script for new hdf5 external-storage .h5 files.
Refactoring.
Add copyright/disclaimer to repack gentest. Minor formatting tweaks.
Minor formatting adjustments.
Undo some formatting modifications. Update MANIFEST.
Add forgotten files to include from h5repack testfiles for tests.
Fix failing automated reporting. Add 8-bit big-endian testfile.
First pass at repack consolidating external storage if layout given. Checks out manually, tests still complain.
Formatting changes, mostly.
Add initial failing test for h5repack copying external data into new file.
Modify code comments for clarity. Some minor reformatting.
Some minor reformatting. Replace tabs with spaces.
Add test that uses the family member FAPL setting.
refactor h5repack_main.c -- remove some redundancy, formatting
Fit help message to 80 character width and clarify some details.
* commit '6eb506eff76b5455f8aac8889c6809aca2d8be52':
Revert "Moved NDEBUG guards to H5EA_DEBUG in H5EA package."
Revert "Fix for H5EA debug package when built in release mode."
Combine the VFD SWMR and non-VFD SWMR raw-data test into one
routine that takes a bool parameter to switch on VFD SMWR.
Update my description of the to-be-written metadata test for VFD SWMR.
* commit '93e8bbd9f47d6a2eace95f2ed3b2dd4b6336940a':
Renamed get/set_time() calls in the tools library to avoid name clash when building static parallel HDF5 w/ static linking to OpenMPI.
* commit 'a6930a2bcfd37ea3610baafb608faec883286315':
Renamed get/set_time() calls in the tools library to avoid name clash when building static parallel HDF5 w/ static linking to OpenMPI.
* commit '8cddc96f73e8f90f67792405f5142b419e9061d7':
Change unused variable to generic form
Whitespace cleanup
Add missing HD prefix
HDFFV-10740 - skip copying dirs in for loops
Correct HDF5 options to tristate values.
HDFFV-10905 fix test scripts for no zlib
HDFFV-10903 - fix VFD constants in JNI
1) Improve hyperslab performance when doing I/O from 1-d disjoint file dataspace
to 1-d contiguous memory dataspace.
2) Move coding in H5D__chunk_io_init() that is constructing the chunk mappings to
a separate routine.
an assertion from firing:
"entry_ptr->delay_write_until == 0" failed: file "../../../vchoi_fork/src/H5PB.c", line 4093, function "H5PB__write_raw"
In a comment, mention a change that has to be made to accommodate parallel
mode.
* commit '4d4a88a9f3775d1c6be820208a2699ae38d586b4': (29 commits)
Fix for H5EA debug package when built in release mode.
Moved NDEBUG guards to H5EA_DEBUG in H5EA package.
Added MPICH_SKIP_MPICXX and OMPI_SKIP_CXX to H5public.h to avoid inadvertant linking to the deprecated MPI C++ wrappers. Fixes HDFFV-10893.
Added a release note for HDFFV-10892 (fcntl lock bug).
Fix segfault after H5VL_loc_params_t fix merge
HDFFV-10901 add missing config options
HDFFV-10854 add release note for windows
Fix get home dir for windows
Use variable for test output
Allow tests to dump output if not comparing
revert as CHECK will first check for exception
Clear exception for getting JNI string to clear the exception
Correct Windows check
revert windows change
Windows ssl headers are not in system loacation
Add include dirs and default lib for windows
Add policy to use <package>_ROOT
Add statement for label if code unimplemented
Update Windows require for CURL to CMake 3.13
Fix unused var
...
* commit 'ab52a0effa605ad6d75cb77cf708bba795cba641':
Added MPICH_SKIP_MPICXX and OMPI_SKIP_CXX to H5public.h to avoid inadvertant linking to the deprecated MPI C++ wrappers. Fixes HDFFV-10893.
* commit 'f5a84c9856888446e329dd36bff842e5a2db708c':
Added MPICH_SKIP_MPICXX and OMPI_SKIP_CXX to H5public.h to avoid inadvertant linking to the deprecated MPI C++ wrappers. Fixes HDFFV-10893.
* commit 'c1c522f6ce8f6d7e44e6ea99860b19228f46952a':
Fix issue in H5Lcreate_hard and H5Olink where entire H5VL_loc_params_t structure was passed via varargs
* commit '0de0965c61823186ac61c0d6876d4d77578e5bcc':
Fix issue in H5Lcreate_hard and H5Olink where entire H5VL_loc_params_t structure was passed via varargs
* commit '05f49d5ffdbd4cf9e967d60a3a0efd0ca7f2c820': (22 commits)
HDFFV-10854 add release note for windows
Fix get home dir for windows
Use variable for test output
Allow tests to dump output if not comparing
revert as CHECK will first check for exception
Clear exception for getting JNI string to clear the exception
Correct Windows check
revert windows change
Windows ssl headers are not in system loacation
Add include dirs and default lib for windows
Add policy to use <package>_ROOT
Add statement for label if code unimplemented
Update Windows require for CURL to CMake 3.13
Fix unused var
Add updated find module for earlier CMake versions
Correct label and var names
Add back missing java implementation
use h5test.h
add test library
special windows override
...
* commit '2456a36f46a15ca44840cb64c7aac213f0ca8bf8': (53 commits)
Fixed some exit calls.
Reworked the previously refactorred code.
Improvement of test code
Fix CMake build of test needing compression libs
Fixed error code and error messages Tested on Jelly (very minor)
Changed H5VL_NATIVE_DATASET_GET_CHUNK_INFO_BY_COOR to H5VL_NATIVE_DATASET_GET_CHUNK_INFO_BY_COORD per a review comment.
Add comment to add tests later to verify conflicting values for MPI-specific properties on multiple opens.
Updated comments
Updated based on reviews and refactored test code
Move ext_path field in H5F_t to H5F_shared_t. Remove unused routine H5F_set_coll_md_read().
Move coll_md_read and coll_md_write fields in H5F_t to H5F_shared_t.
Fix namespace usage
Fix EXISTS test syntax
Update settings
Add 32-bit cross-compile toolchain
Code cleanup and refactor
Re-ordered items w.r.t. date.
Whitespace and HD prefix changes
Fixed typo
Added notes about HDFFV-10677 and HDFFV-10661
...
Description:
Fixed tests to handle the case when a requested compression filter
is not available.
Platform tested
Linux/64 (jelly)
Darwin (osx1011test)
- Added new chunk query functions: H5Dget_num_chunks(), H5Dget_chunk_info(), and H5Dget_chunk_info_by_coord()
- Fixed iterator issue in H5EA_iterate(), H5FA_iterate(), and H5D__none_idx_iterate()
* commit '96dab622e3bbe75d85b0cef3ca2bd5288cf4ac32':
Fixed error code and error messages Tested on Jelly (very minor)
Changed H5VL_NATIVE_DATASET_GET_CHUNK_INFO_BY_COOR to H5VL_NATIVE_DATASET_GET_CHUNK_INFO_BY_COORD per a review comment.
Updated comments
Updated based on reviews and refactored test code
Code cleanup and refactor
Re-ordered items w.r.t. date.
Fixed typo
Added notes about HDFFV-10677 and HDFFV-10661
HDFFV-10677 and HDFFV-10661
* commit '62750441cdfbf17f7fa5db3bbf1320c1dc5be838':
Add comment to add tests later to verify conflicting values for MPI-specific properties on multiple opens.
Move ext_path field in H5F_t to H5F_shared_t. Remove unused routine H5F_set_coll_md_read().
Move coll_md_read and coll_md_write fields in H5F_t to H5F_shared_t.
* commit '1efd23ff02b9bc1a3d303d53395245d1366588be':
Whitespace and HD prefix changes
fix get gethostname for windows
OESS-29 Fix HD prefix in perform and enable test compile
HDFFV-10876 h5dump OPAQUE size should be optional
output, then it tested $? for an error exit. $? told the error status of
`tee`, though, not the test programs! So no test failures were counted, even
when some tests clearly failed. I changed the test script to use a shell
subroutine, `catch_out_err_and_rc`, to catch test programs' output and result
code.
H5FDvfd_swmr_private.h.
Perform tick processing in FUNC_ENTER_API_NOCLEAR, where it was missing.
Track the number of times the HDF5 library has been entered/exited through its
public API. Only perform tick processing on the first entry and last exit.
This stops us from performing tick processing in API calls invoked by
application callbacks. Performing tick processing in nested API calls led to
crashes.
Note well: FUNC_LEAVE_API now performs tick processing even on an error exit!
Previously, it did not. I'm not sure if the change is ok.
Description:
- Added functions to query chunk information:
H5Dget_num_chunks(dset_id, fspace_id, *nchunks)
Gets the number of written chunks that intersect with the given
dataspace. However, in this version, the intersection is not
yet completed. Thus, the number of all written chunks will be
returned.
H5Dget_chunk_info_by_coord(dset_id, *offset, *filter_mask, *addr, *size)
Given a chunk's logical coordinates, returns the chunk's filter,
address, and size.
H5Dget_chunk_info(dset_id, fspace_id, index, *offset, *filter_mask, *addr, *size)
Given a chunk's index, returns the chunk's logical coordinates, filter,
address, and size. The chunk belongs to a set of chunks that have
nonempty intersection with the specified dataspace. However, in
this version, the intersection is not yet completed, and the index
is of all the written chunks.
These functions comply with VOL.
- Fixed some oversights found in the library for the tests in chunk_info.c
to work correctly. The returned value from a callback function was not
checked in H5EA_iterate(), H5FA_iterate(), and H5D__none_idx_iterate().
This oversight caused a callback function to continue iterating even though
it's supposed to stop.
Platforms tested:
Linux/64 (jelly)
Linux/64 (platypus)
Darwin (osx1011test)
* commit 'fb8296371cf45654a9252ed01b635519205da3de':
Minor tweaks to new H5P MPI code based on code review feedback.
Fixed a bug in the cache image code that was introduced by the HD changes.
Added H5Pset/get_mpi_params calls and unified them with the MPI-I/O VFD info in H5FDmpio.c.
* commit '937616872cf228f838fd1eb2853ce6110f3da10b':
Minor tweaks to new H5P MPI code based on code review feedback.
Fixed a bug in the cache image code that was introduced by the HD changes.
Added H5Pset/get_mpi_params calls and unified them with the MPI-I/O VFD info in H5FDmpio.c.
* commit 'd42de626b604b75963fd593a145fd9c9b9e20602':
Fixed a bug where we incorrectly pass a lock struct to fcntl for file locking instead of a pointer.
* commit 'ddf04a9a727252f40197c48da541212dff646046':
Add a 'closing' flag on the shared file struct, and switch several of the "internal" data structures to use it.
Fixed the assertion failure for HDFFV-10873.
* commit '3090ac81506094ee510ae89c31749b5c1632aa34':
Add a 'closing' flag on the shared file struct, and switch several of the "internal" data structures to use it.
* commit 'f42c7cc0d85cc5b232afd621711fdbd64f6e19d9':
Fix major error codes
Switch shared file struct name from 'H5F_file_t' to 'H5F_shared_t', to match naming convention for shared structs in rest of library data structures.
Small tidy up.
Progress toward movig H5MF* API to use shared file pointers instead of top file pointers.
Progress on moving the H5MF* interface to using shared file pointers instead of top file pointers.
Begin converting the H5MF interface to use shared file pointers instead of top file pointers.
Progress toward moving the dataset routines to using the 'shared' file pointer instead of the 'top' file pointer.
Begin converting dataset code to use shared file pointer instead of top file pointer.
Finish converting H5PB_* routines to use shared file pointer, instead of top file pointer.
First pass of converting H5PB_* routines to use shared file pointers.
Rest of changes for 'top' file pointer to 'shared' file pointer for H5F__accum routines.
Partial H5F__accum_* routines.
* commit '92f103cfd4b19b61a0ddf6e62e1692d2b3eeb4d9':
Add H5_DLL to all s3comms function prototypes to stop Windows unresolved external symbols errors. HDFFV-10889.
* commit '9081fb266cc15198e0c75a77f23f2b252c6ee79f':
Add missing HD prefix
This test cannot use HD prefix macro with printf of fprintf
Update HD prefix and reconcile hl/cpp test
Add HD prefix to tests
* commit 'dcf084aff42d3abafaa82b356493a2d1585afb85': (25 commits)
Changes based on PR feedback.
Correct name of folder
Mostly whitespace in testpar, addl changes for tmp dir
Whitespace cleanup
Added public H5Sselect_adjust_u and H5Shyper_adjust_s calls.
Some refactoring prior to implementing new H5P MPI functions. * Macro cleanup and obvious warning fixes in parallel code. * Moved H5FD comm and info dup/free wrapper code to a new H5mpi.c file and separated it to deal with each MPI type separately.
Fix compiler warnings with diabled ROS3 and HDFS VFDs. Fix misuse of [HD]strncpy and [HD]snprintf. Minor formatting changes. Fix test cleanup issue with java groups example.
Changed the directory name left out from the previous commit.
Changed the name of the temporary directory from 'tmp' to a special name to avoid possible conflicts.
Fix H5F_get_file_id and H5F__get_file_id to take app_ref parameter
HDFFV-10879 fix test varname
Fix for HDFFV-10813 H5Fset_metadata_read_retry_info() test fails on jelly with PGI/19.
Added map API build status to libhdf5.settings.
Updated h5trav.c to emit output that doesn't require ddl changes for map support.
HDFFV-10879 set default API version
Fixed Map API CMake option
Renamed H5Mdelete_key to H5Mdelete and H5Mset to H5Mput.
Remove old code
Correct error test spelling and use
Added ifdef for intel compilers, __GNUC__ has to be defined for header files which use it to work with intel compilers. This gives a warning for intel compiler about fallthrough, so added a check not to include fallthrough if using intel compilers.
...
* commit 'dcf084aff42d3abafaa82b356493a2d1585afb85':
Changes based on PR feedback.
Fix for HDFFV-10813 H5Fset_metadata_read_retry_info() test fails on jelly with PGI/19.
* commit '538f743c145fbbfa7b1119693efd2ff61b34efae':
Changes based on PR feedback.
Fix for HDFFV-10813 H5Fset_metadata_read_retry_info() test fails on jelly with PGI/19.
* commit '3a504899eef1a66031c6f52623c24bb4e51ca51e':
Correct name of folder
Mostly whitespace in testpar, addl changes for tmp dir
Whitespace cleanup
Added public H5Sselect_adjust_u and H5Shyper_adjust_s calls.
Some refactoring prior to implementing new H5P MPI functions. * Macro cleanup and obvious warning fixes in parallel code. * Moved H5FD comm and info dup/free wrapper code to a new H5mpi.c file and separated it to deal with each MPI type separately.
Fix compiler warnings with diabled ROS3 and HDFS VFDs. Fix misuse of [HD]strncpy and [HD]snprintf. Minor formatting changes. Fix test cleanup issue with java groups example.
Changed the directory name left out from the previous commit.
Changed the name of the temporary directory from 'tmp' to a special name to avoid possible conflicts.
Fix H5F_get_file_id and H5F__get_file_id to take app_ref parameter
HDFFV-10879 fix test varname
* commit 'bd8da502cf5b763433e3c628a4043a4264529cec':
Some refactoring prior to implementing new H5P MPI functions. * Macro cleanup and obvious warning fixes in parallel code. * Moved H5FD comm and info dup/free wrapper code to a new H5mpi.c file and separated it to deal with each MPI type separately.
* Macro cleanup and obvious warning fixes in parallel code.
* Moved H5FD comm and info dup/free wrapper code to a new H5mpi.c file
and separated it to deal with each MPI type separately.
* commit '2e5cd3d2b2d953420252c80ab5e32ac95ac0c5b3':
Fix compiler warnings with diabled ROS3 and HDFS VFDs. Fix misuse of [HD]strncpy and [HD]snprintf. Minor formatting changes. Fix test cleanup issue with java groups example.
* commit '0b868668bac0d268120844adb9ad216a9a8e1395':
Changed the directory name left out from the previous commit.
Changed the name of the temporary directory from 'tmp' to a special name to avoid possible conflicts.
* commit 'aeb145428f5e6bc51742fff031b0cf8cf6fbefbb':
Updated h5trav.c to emit output that doesn't require ddl changes for map support.
Fixed Map API CMake option
Renamed H5Mdelete_key to H5Mdelete and H5Mset to H5Mput.
Added ifdef for intel compilers, __GNUC__ has to be defined for header files which use it to work with intel compilers. This gives a warning for intel compiler about fallthrough, so added a check not to include fallthrough if using intel compilers.
Added the map (H5M) API
* commit 'd95745235a468b17f697843adb3d0e54bd919893':
Updated h5trav.c to emit output that doesn't require ddl changes for map support.
Fixed Map API CMake option
Renamed H5Mdelete_key to H5Mdelete and H5Mset to H5Mput.
Added the map (H5M) API
* commit 'df7ce9a5c933315af34dd402cfb2ca394003b56b':
Added ifdef for intel compilers, __GNUC__ has to be defined for header files which use it to work with intel compilers. This gives a warning for intel compiler about fallthrough, so added a check not to include fallthrough if using intel compilers.
* commit 'c3c044cc003d6a4039635e638e1cd1c2487be177':
Escape { and } in regex as required by sh in updated cori.nersc.gov. Tested change on jam, jelly, emu, ostrich and osx1011test.
* commit '5fe1216a4e6af8feef0551f7bece8a0e193310a9':
Fix return type for H5Sselect_intersect_block().
Updated H5TRACE macro.
Add H5Sselect_shape_same and H5Sselect_intersect_block API routines, along with tests and minor cleanups and refactorings.
* commit '905d40aa6f6eb603c5507e7967130760e3a2e43c':
Fix return type for H5Sselect_intersect_block().
Updated H5TRACE macro.
Add H5Sselect_shape_same and H5Sselect_intersect_block API routines, along with tests and minor cleanups and refactorings.
* commit 'ff8d8a9edb3b8012fa02d2b9fa3c823fee7575fa':
Fix conflicting types for reset_raw_data_files() parameter in external_common.c changed from int to hbool_t. Change 0 and 1 to FALSE and TRUE in calls to reset_raw_data_files().
* commit '4227b8948d7091205b4f8dc1271d7dcae01bb0fb':
Reverts the removal of config/conclude_fc.am while keeping the changes to config/gnu-fflags.
Fixed a couple of typos.
* commit '363217073750d07e0eeb0aac2b404a799fda9f7a':
Update COPYING file. Remove hdfs from list of vfds for check-vfd testing as it is read-only.
Fixed the MANIFEST
* commit 'd169391529f253a9903b8e6cacf38c59ceefab07':
Parallel Fortran tests now use the MPI module instead of including mpif.h.
First stab at gfortran versions. Tested on gfortran 7-9 so far.
Moved the -std=f2008 option to gfortran 9 for now.
Updated gfortran options. * Removed conclude_fc.am, which was causing verbose builds. Need to investigate more, though. * Added gfortran 9 warnings and flags. This will need to be separated into other version blocks based on when the options first appeared. * The standard is now explicitly set at 2008.
* commit '7d291d059da6287f769951c4854c566da0b93ce8':
Parallel Fortran tests now use the MPI module instead of including mpif.h.
First stab at gfortran versions. Tested on gfortran 7-9 so far.
Moved the -std=f2008 option to gfortran 9 for now.
Updated gfortran options. * Removed conclude_fc.am, which was causing verbose builds. Need to investigate more, though. * Added gfortran 9 warnings and flags. This will need to be separated into other version blocks based on when the options first appeared. * The standard is now explicitly set at 2008.
* commit 'f97e11e7635a0cd8728d4604ca5dceb3925ba44c':
Update comment and check for strtoumax.
Modify CMakeLists.txt file for renamed h5tools_test_utils files.
Add HD to string functions. Switched strtoul to strtoumax in H5FDs3comms.c. Removed unused functions and variables in s3 and hdfs TestH5Pfapl*.java. Update Copyright headers.
Squashed commit of the following:
* commit '8aa9043837ea217280098b4b32da8d98ac18f58b':
Update comment and check for strtoumax.
Modify CMakeLists.txt file for renamed h5tools_test_utils files.
Add HD to string functions. Switched strtoul to strtoumax in H5FDs3comms.c. Removed unused functions and variables in s3 and hdfs TestH5Pfapl*.java. Update Copyright headers.
Squashed commit of the following:
* commit 'e19b0302cc38d1850ada7a00431511343cddc8a6': (31 commits)
HDFFV-10845 update note
Correct format of drive check
Use generator expr to get correct name
HDFFV-10845 Allow mingw to find functionality
HDFFV-10845 fix comment syntax
HDFFV-10845 Windows drive can only goto Z
HDFFV-10845 make flags private
Correct java load library name access
Correct syntax
Fix typo
Correct OUTPUT_NAME usage
CMake generator expr cannot be used in get_property cmd
HDFFV-1045 gcc static must be link flags
HDFFV-10845 quote multiple strings in cmake link command
HDFFV-10845 update mingw cmake changes
Add missing target root path, check other export macro
HDFFV-10845 skip test if CMake command doesn't support ignore EOL
HDFFV-10845 copy files with windows EOL
HDFFV-10845 separate test used by copy windows files
HDFFV-10845 mingw needs special windows reference files
...
* Removed conclude_fc.am, which was causing verbose builds. Need to
investigate more, though.
* Added gfortran 9 warnings and flags. This will need to be separated
into other version blocks based on when the options first appeared.
* The standard is now explicitly set at 2008.
* commit 'd5031a55192ac6b8ca81a0e5315cb4739af98896':
Removed support for gcc 4.3 and earlier.
Fix errant quotes
* Fixed a bug in the gcc warning parsing * Changed version parsing from == gcc 4.x to < to handle ancient compilers.
Reworked the config/gnu-flags file to be more organized and robust.
Converted duplicated flags code to a set of if statements.
* commit '22d78bdfa4d560a0b8a2dbb547574bfe61bf64f0':
Updates to warnhist script, along with a few cleanups, and add some comments to warning cleanups that are a bit obscure.
Cleaned up code that genericizes warning output, and add "-u" option to leave unique types in warnings (i.e. don't genericize output). Also fixed the '-S' option to work correctly.
Added GCC9 flags to CMake build
Add support for GCC9, update warnhist script, and clean up warnings.
* commit '73df02f13c2b1e0b256cdbff1c7c584e84548df4':
Updates to warnhist script, along with a few cleanups, and add some comments to warning cleanups that are a bit obscure.
Cleaned up code that genericizes warning output, and add "-u" option to leave unique types in warnings (i.e. don't genericize output). Also fixed the '-S' option to work correctly.
Added GCC9 flags to CMake build
Add support for GCC9, update warnhist script, and clean up warnings.
* commit '88ab4b144bcf8454be9eff8800706c0abb404de1':
fix var name
Fix compile error with ifdef H5_HAVE_GETHOSTNAME
perf program can only be built on unix platforms
Declaration needed for parallel standalone
Correct test use of library name
Change how name is retrieved
Move before null check
HDFFV-10581 remove special name checking
HDFFV-10581 fix locations for static pdb files
Fixed libhdf5 settings file to be consistent for autotools & CMake
* commit '5e86326e0a731505095013aaf2a766bfa6cc1dda':
Fixed Build HDF Tests name in CMake's libhdf5 settings
Fixed libhdf5 settings file for both autotools & CMake
* commit '6ced6457c3048bd10bf6f470b329b4810d9be826':
Move the -Wformat-nonliteral warning to the developer flags. Fix bugs I introduced in the last commit.
Updated configure & CMake compiler flags for GCC 8.x, along with corresponding changes to warnhist script (and some extra improvements for condensing C++ and Java warnings), and fixed a bunch of warnings.
* commit 'c69b1025f58ec8467be1e056214510c70ed00867':
(1) Add/remove comments. (2) A temporary fix to address the test/objcopy.c: test_copy_group_deep() test failure with the family driver. The test failure occurs with these configurations in objcopy.c: --with shared src messages (CONFIG_SHARE_SRC) --without shared dst messages (CONFIG_SHARE_DST) --with latest format for source file (CONFIG_SRC_NEW_FORMAT) --without dense attributes (CONFIG_DENSE) --with latest format for destination file (CONFIG_DST_NEW_FORMAT) The temporary fix is in src/H5MFaggr.c (see comments above #ifdef REPLACE/#endif).
Checkin for new shutting down free-space manager.
(2) A temporary fix to address the test/objcopy.c: test_copy_group_deep() test failure with
the family driver. The test failure occurs with these configurations in objcopy.c:
--with shared src messages (CONFIG_SHARE_SRC)
--without shared dst messages (CONFIG_SHARE_DST)
--with latest format for source file (CONFIG_SRC_NEW_FORMAT)
--without dense attributes (CONFIG_DENSE)
--with latest format for destination file (CONFIG_DST_NEW_FORMAT)
The temporary fix is in src/H5MFaggr.c (see comments above #ifdef REPLACE/#endif).
* commit '3768566139df18928aa29ece0eff3010b224633b':
Add release notes.
Correct hid_t error value to H5I_INVALID_HID in tests related to libver_bounds.
Fix for HDFFV-10808 H5Pset_file_space_strategy succeeds when using H5Pset_libver_bounds v18,v18. Fails file creation when non-default free-space info is set in fcpl and the library version high bound is less than v110 because free-space info message is introduced in library release v110.
Fails file creation when non-default free-space info is set in fcpl and the library version
high bound is less than v110 because free-space info message is introduced in library release v110.
* commit '1d8f7bf297100ec11204442708a7f670a89f3f02':
Updated the Java attribute test so that attribute access property lists are created and used correctly.
Add test cleanup
Need to shift off two arguments
HDFFV-9407 add autotools test
Move pragma statements outside of routines, to make older compilers happy.
HDFFV-9407 Add test
* commit 'f4015bed9e7ee43468240e734232f9114b749d4f':
Updated the Java attribute test so that attribute access property lists are created and used correctly.
* commit 'f6ddfd7dda8aa95b33ae89b87c70493d5c644dbe':
Add test cleanup
Need to shift off two arguments
HDFFV-9407 add autotools test
HDFFV-9407 Add test
* commit 'e35effff7da40fa8b0dda273be9bbdfa9c8c3624':
Move pragma statements outside of routines, to make older compilers happy.
Updated H5Tcopy() to get the dataset's datatype through the VOL when that is passed in as the object ID.
* commit '35fd0ec8ceffe96cee352187154da15c967fb990':
Updated H5Tcopy() to get the dataset's datatype through the VOL when that is passed in as the object ID.
Fix fortran test and test library linking
Update GCC 6 & 7 flags for CMake builds
Move -Wnormalized down into GCC 6.x flags
Put the memcpy overlap check back into H5MM.
Fixed the heap overflow in t_filters_parallel
Fixed some low-hanging fruit from -fsanitize in t_filters_parallel.
Add lib dir for testing plugins
Add support for GCC 7.x warnings, update warnhist script to account for them, clean up warnings.
Add H5_HLDLL prefix for windows link
HDFFV-10805 cleanup examples and test code
Fix minor typo in H5S_select_iter_release
Add missing fortran mods
HDFFV-10805 Fix test of library libinfo
Cleanup possible CMake target conflicts
Correct CMake issues
* commit '35fd0ec8ceffe96cee352187154da15c967fb990':
Updated H5Tcopy() to get the dataset's datatype through the VOL when that is passed in as the object ID.
Fix fortran test and test library linking
Update GCC 6 & 7 flags for CMake builds
Move -Wnormalized down into GCC 6.x flags
Put the memcpy overlap check back into H5MM.
Fixed the heap overflow in t_filters_parallel
Fixed some low-hanging fruit from -fsanitize in t_filters_parallel.
Add lib dir for testing plugins
Add support for GCC 7.x warnings, update warnhist script to account for them, clean up warnings.
Add H5_HLDLL prefix for windows link
HDFFV-10805 cleanup examples and test code
Add missing fortran mods
HDFFV-10805 Fix test of library libinfo
Cleanup possible CMake target conflicts
Correct CMake issues
* commit '8b00d921d7313cd21947992ab4a007d593c49207':
Updated H5Tcopy() to get the dataset's datatype through the VOL when that is passed in as the object ID.
Updated CMake bulids, too.
* commit '17ae90917856e4cbab5a2f1c597197ee98802fc7':
Update GCC 6 & 7 flags for CMake builds
Move -Wnormalized down into GCC 6.x flags
Add support for GCC 7.x warnings, update warnhist script to account for them, clean up warnings.
* commit 'c24f11070aea56331110628d7645332534e4f227':
Add H5_HLDLL prefix for windows link
HDFFV-10805 cleanup examples and test code
Add missing fortran mods
HDFFV-10805 Fix test of library libinfo
Cleanup possible CMake target conflicts
Correct CMake issues
* commit '2e973f7e12301975f9d8dfdd5534682525554e77':
Fix minor typo in H5S_select_iter_release
Fix memory leak in objcopy test after H5Aread of vlen data
* commit 'ca1ee7fae118c8d4137b1b5efe57660d2e0ff9b1': (64 commits)
Fix memory leak in objcopy test after H5Aread of vlen data
HDFFV-10616 add h5fc script
Fixed a problem in the ohdr test where hard-coded strings passed to H5Awrite() triggered -fsanitize failures.
Yanked qsort assert
Switched to HGOTO_DONE() in the links code.
Updated the HDqsort() macro to ensure we don't pass NULL buffers to qsort(3) in the future.
Fixed a bug in the links code where iterating over an empty group would pass a NULL pointer to qsort(3), which is undefined behavior.
Reset the other convenience pointer because why no
Fixed a memory bug where a pointer is used after it's been freed.
fixed pass_through vol
HDFFV-10805 Add option to only build shared targets
Fixed a warning in H5CX.c concerning incorrect use of freeing VOL connector info.
Add H5S_SEL_ITER_SHARE_WITH_DATASPACE selection iterator creation flag, to share dataspace's selection with iterator (and with caution about not modifying or closing the dataspace while the iterator is open).
Added a warning summary script for gcc to bin
Fix misc. typos, etc. from code review
Fixed problem with H5Fdelete "test" and split VFD.
New hyperslab selection routines and new public selection iterator routines.
Added the creation and use of an attribute access property list to testhdf5.
Update MANIFEST for move of h5cc.in and h5redeploy.in to bin directory.
Added missing DELETE enum value.
...
* commit '4d20db43bea4bc4588b20abebef5b0d043dcf31d':
Fixed a problem in the ohdr test where hard-coded strings passed to H5Awrite() triggered -fsanitize failures.
* commit '6a1e9a1fbe9606d0dd3452092cab17794e131e64':
Fixed a problem in the ohdr test where hard-coded strings passed to H5Awrite() triggered -fsanitize failures.
* commit 'c6e21cede788790b39af4a46a8bc15e49e382346':
Yanked qsort assert
Switched to HGOTO_DONE() in the links code.
Updated the HDqsort() macro to ensure we don't pass NULL buffers to qsort(3) in the future.
Fixed a bug in the links code where iterating over an empty group would pass a NULL pointer to qsort(3), which is undefined behavior.
* commit '0a75da70a61d41a0bf20a5c617f3eb678ed0b103':
Yanked qsort assert
Switched to HGOTO_DONE() in the links code.
Updated the HDqsort() macro to ensure we don't pass NULL buffers to qsort(3) in the future.
Fixed a bug in the links code where iterating over an empty group would pass a NULL pointer to qsort(3), which is undefined behavior.
* commit '10535e0376d88e218cab782322bfc06f35835f31':
Reset the other convenience pointer because why no
Fixed a memory bug where a pointer is used after it's been freed.
* commit '8d67b440f7d94f1b191ce154a41685bcaa90fd12':
Reset the other convenience pointer because why no
Fixed a memory bug where a pointer is used after it's been freed.
* commit '712a2a13d8add828d053135a172abb2e819d59ad':
Fixed a warning in H5CX.c concerning incorrect use of freeing VOL connector info.
Add H5S_SEL_ITER_SHARE_WITH_DATASPACE selection iterator creation flag, to share dataspace's selection with iterator (and with caution about not modifying or closing the dataspace while the iterator is open).
Fix misc. typos, etc. from code review
New hyperslab selection routines and new public selection iterator routines.
* commit 'd20139399750246f55c902940987e0048a40c268':
Add H5S_SEL_ITER_SHARE_WITH_DATASPACE selection iterator creation flag, to share dataspace's selection with iterator (and with caution about not modifying or closing the dataspace while the iterator is open).
Fix misc. typos, etc. from code review
New hyperslab selection routines and new public selection iterator routines.
* commit 'dfdc27b04b2e8bcd1985ba90ce6553d8b3805fda':
Fix release notes based on feedback from pull request.
Fix for HDFFV-10800 H5Ocopy failure: The value for the H5F_LIBVER_V18 index in H5O_fill_ver_bounds[], the format version bounds array for fill value message, should be version 3 not 2.
* commit '9ef56483f6447b60cf7568e2022aa5d8f193427c':
Added missing DELETE enum value.
Split VOL connector routines into separate headers: * H5VLconnector.h for terminal connector things * H5VLconnector_passthru.h for passthrough connector things
* commit '04dc4ef436b087cfc45030b9183386598d1b4bae':
Moved h5cc.in and h5redeploy.in from tools/misc to bin directory. They should always be built and installed whether tools are enabled or disabled. Also added Makefile.am to bin to build h5redeploy and to install and uninstall them. h5cc is created from h5cc.in by configure.
should always be built and installed whether tools are enabled or
disabled. Also added Makefile.am to bin to build h5redeploy and
to install and uninstall them. h5cc is created from h5cc.in by
configure.
* commit 'bb5e7822f447b8a175e807e2e7bad5e0192394e5':
Fixed an error string typo
Added H5Fdelete call and VOL support (but no VFD/native implementation).
* commit '0742c15fca95965ed4eb87c6d54f41bb07bd4526':
makefile clean-up
move the vol_info to an optional parameter
added H5Pset_vol
added vol connector
added test for vol_connector
removed H5VL constants
removed VOLL NULL connector and replaced it with native VOL
added connector_name test
added missing files
add tests
added more tests
added missing files
Implemented VOL APIs and C constants, no tests
Implemented VOL APIs and C constants, no tests
initial API impl.
* commit '6e1dc507240fba447c79827897b099e91b9dd5c1':
Add release note for enable-tools and enable-tests options.
Modified Makefile.ams for h5repack, h5watch and gif2h5 to allow disabling tests. Moved h5cc.in from tools/src/misc to src directory to always create h5cc whether or not tools are enabled. Added configuration status of tools and tests to libhdf5.settings.
Add options to enable or disable building tools and tests. The default is enabled for each.
disabling tests.
Moved h5cc.in from tools/src/misc to src directory to always create h5cc
whether or not tools are enabled.
Added configuration status of tools and tests to libhdf5.settings.
* commit '0cf52525d5b45cc7cf6f89267da76b46ffa75f90':
H5A__free() was renamed to H5A__shared_free() and now requires attr->shared to not be NULL.
Fixed a memory issue where unfreed shared attribute dataspace memory tripped an assert in our memory sanity checks.
Fixed a memory issue where unfreed shared attribute dataspace memory tripped an assert in our memory sanity checks.
* commit '931abe94244c94e106835250e2f723f634a61c8c':
Move set (MAX_PROC_COUNT 8) back to HPC options files.
Move check for MPI defined to enable parallel from options scripts in HPC to config/cmake/HDF5options.cmake.
* commit '9557087c0ed7811301610155654bf58d6583e38c':
Minor warning fixes: * Fixed an error return value in H5Aint.c (NULL --> FAIL) * Added H5VL_FILE_GET_FILENO to H5trace.c
* commit '66699f2c7d889e725ced0fba579621ef97726b41': (49 commits)
Minor fixes/improvements for VDS performance improvement (HDFFV-10693).
Refactor H5S__hyper_project_intersection to operate directly on span trees, improving performance and removing the conditionn that the extents be equal.
Refactor H5S__hyper_project_intersection to operate directly on span trees, improving performance and removing the conditionn that the extents be equal.
Added code that disallows unregistering the native VOL connector.
Rename H5VL_ID_GET_NAME to H5VL_OBJECT_GET_NAME
Correct message command syntax
Assign NULL to func_name/file_name/desc error entry pointers when clearing error stack
Avoid clearing error stack when calling H5Eget_msg()
Reorder lines in HDF5config.cmake so that additions to end of $HPC-HDF5options scripts will take precedence.
updated -E option
remove -M option
added -M HPC to ctest command
fixed missing paralle test
Adding cross-compilation for theta (ANL) and qsub (COBALT) batch controls.
Add note about CMake and spaces in paths
Putting ctest command in variable CMD did not work for skipping failing tests.
Code improvement Description: Fixed potential division by zero occurrences and changed an assert to if statement. Platforms tested: Linux/64 (jelly) Linux/64 (platypus) Darwin (osx1011test)
Combine "collective group and dataset write" and "independent group and dataset read" tests into one test. The second test is dependent on the file created by the first test, and will not pass when run by itself.
Add final list of single testphdf5 tests
Pull a test from default testphdf5
...
Complete rework of H5S__hyper_project_intersection. Addresses VDS performance issues raised in HDFFV-10693.
* commit 'd3e242eba1ba98726ed64b14e75f6adee724bea7':
Minor fixes/improvements for VDS performance improvement (HDFFV-10693).
Refactor H5S__hyper_project_intersection to operate directly on span trees, improving performance and removing the conditionn that the extents be equal.
Refactor H5S__hyper_project_intersection to operate directly on span trees, improving performance and removing the conditionn that the extents be equal.
* commit '8a57a683391eed79aca80a7325ef152bb76ad0a6':
Code improvement Description: Fixed potential division by zero occurrences and changed an assert to if statement. Platforms tested: Linux/64 (jelly) Linux/64 (platypus) Darwin (osx1011test)
* H5VLconnector.h for terminal connector things
* H5VLconnector_passthru.h for passthrough connector things
Note that these headers are arranged such that they are included in
hdf5.h so VOL connectors only need to include that. The separation into
multiple headers is mainly for readability.
* commit '6eaef5ebd61b8019f66f51d3fe74ec3c1b1d4f0f':
Reorder lines in HDF5config.cmake so that additions to end of $HPC-HDF5options scripts will take precedence.
updated -E option
remove -M option
added -M HPC to ctest command
fixed missing paralle test
Adding cross-compilation for theta (ANL) and qsub (COBALT) batch controls.
* commit '9406278103a7b72d3eacea9fbeaadbf34bf7e43a':
Reorder lines in HDF5config.cmake so that additions to end of $HPC-HDF5options scripts will take precedence.
Description:
Fixed potential division by zero occurrences and changed an assert to
if statement.
Platforms tested:
Linux/64 (jelly)
Linux/64 (platypus)
Darwin (osx1011test)
* commit '8a4f679fda20577c4528977d0ee9a7e4c5bee055':
Combine "collective group and dataset write" and "independent group and dataset read" tests into one test. The second test is dependent on the file created by the first test, and will not pass when run by itself.
* commit '08b009cc4ab2a8d8d289697d6d4688c0e5c53c6e':
Clean up if-else chain.
Remove thread-local copy of hyperslab operation generation value.
Core changes to selection code from the hyperslab_updates branch.
* commit '53ffbf4bf89350b5d6b90c11585f39af94209754':
Cast enum to int, for strict varargs handling.
Refactor group creation parameters to take them out of the group creation property list and pass them as normal function parameters.
Warning cleanup
Refactor link creation parameters from being passing as properties in the link creation property list to vararg function parameters.
Minor cleanups
Refactor attribute creation parameters to move parameters out of creation property list and into function parameters.
A few minor cleanups
Refactor dataset creation parameters out of the dataset creation property list and pass them as regular function parameters.
* commit 'e66bf948a7acba0ddc76239f6d02f17c8b10eb0f':
Fix for split VFD and file cleanup when testing H5Fis_hdf5 and H5Fis_accessible in tfile.c.
Improvements to the H5Fis_accessible() and H5Fis_hdf5() tests in tfile.c. * General code tidying. * Expanded tests to work with multi-file VFDs.
* commit '2c9025dcb96e5445ce878fa9e1d05fca738ae328':
HDFFV-10760 Skipping external_env.c and vds_env.c for CMake to do VOL check and VFD check. Temporarily skipping external_env.c and vds_env.c for VFD check and VOL check in CMake following the example of links_env. Leave them for future work.
* commit '62198c2436ac3787afb49fa990d1ba8204264dba':
Add an 'unwrap' VOL callback, so that connectors can unwrap an underlying object without closing it. (Especially needed for pass-through connectors)
Description:
Added wrappers for H5Pset/get_create_intermediate_group:
// Specifies in property list whether to create missing
// intermediate groups
void setCreateIntermediateGroup(bool crt_intmd_group) const;
// Determines whether property is set to enable creating missing
// intermediate groups
bool getCreateIntermediateGroup() const;
Platforms tested:
Linux/64 (jelly)
Linux/64 (platypus)
Darwin (osx1011test)
* commit 'f42b6ace1899bd1e720bbdf4873e6f6e61bee39b':
Changing output file names to avoid possible conflict during parallel build and test, especially on jelly.
* commit '53a5c3be903ce7c24d02a6bd3a8d9b07a3f82404':
Change the checking to H5I_INVALID_HID in test_attr_dense_rename().
Fix for HDFFV-10579 H5Arename fails when creation order of attributes is tracked. The attribute needs to be removed from the creation order index v2 B-tree before re-inserting the renamed attribute via H5A__dense_insert().
* commit 'b5ef82a1786605ae86502bc82086047720b7d4ca': (21 commits)
Moving the handling of null prefix into H5_combine_path.
Changing the prefix of external file and VDS from empty string to null for performance improvement.
Improving the condition checking of empty string.
Replacing string operation strdup with assignment for empty string.
Adding back links_env.out which I accidentally removed in my previous commit.
Taking out two unnecessary diff output files.
Taking out unnecessary diff files for output.
Adding some comments.
Some coding style changes.
Adding the standard output files for the external_env.c and vds_env.c tests.
Small correction for my previous commit.
Forgot to add external_common.c and external_common.h.
Updated CMake for the splitting of external.c and vds.c.
Minor fix: removal of unnecessary enum values.
Minor fixes: updating the test vds_env.c according to the set up of vds.c.
Left out this file in previous commit.
HDFFV-10658 - setting and getting properties in API context: 1. switched to use the existing H5F_prefix_open_t for enum type; 2. put the common private function used by external.c and external_env.c into external_common.c
This commit basically has the following changes: 1. restored the datatype, dataspace, and LCPL of the dataset for VOL connector back to the properties. 2. splitted external.c and vds.c because they called HDsetenv in the program, instead using shell scripts to set the environment variables. 3. changed H5CX_get_vds_prefix and H5CX_get_ext_file_prefix to use H5P_peek instead of H5P_get.
HDFFV-10658: I left out this file in my previous commit.
HDFFV-10658: 1. moving HDgetenv to dataset initialization stage to reduce the overhead; 2. restoring the retrieval of three vol properties to H5P_get instead of using API context to prepare for Quincey's upcoming refactoring work.
...
* commit '659e212e119b4920d2bd41b397cfb0e958817eba':
Correct set extent operation on VDS to iterate over # of used sub-datasets instead of # of allocated sub-datasets.
* commit '9ee3d472113fc3e66ef34e40a0990ed07a2824bc':
Renamed the HDF5_TEST_VOL option to HDF5_TEST_PASSTHROUGH_VOL.
Renamed the autotools check-vol target to check-passthrough-vol.
* commit 'd7e1464058515d07b838741f65a77977224814de':
Add C++, Java, and FORTRAN wrappers and tests for H5Fget_fileno
Add trace macro.
Add H5Fget_fileno() API routine.
* commit '52276f3713eec584044bc72d4724507848cfeba0':
Modifications based on PR feedback: (1) Add H5Sdeprec.c to src/CMakeLists.txt (2) Add test for H5Sencode1.
Modification for num_elem based on PR feedback.
HDFFV-10365: Changes as described in the RFC: H5Sencode/H5Sdecode Format Change. This also addresses HDFFV-10255: H5Sencode/decode performance issue.
1. switched to use the existing H5F_prefix_open_t for enum type;
2. put the common private function used by external.c and external_env.c into external_common.c
* commit '3ccc98e256587c43f6ba5a31a2cf9d922f4e1773':
Modifications based on PR feedback: (1) Add H5Sdeprec.c to src/CMakeLists.txt (2) Add test for H5Sencode1.
Modification for num_elem based on PR feedback.
HDFFV-10365: Changes as described in the RFC: H5Sencode/H5Sdecode Format Change. This also addresses HDFFV-10255: H5Sencode/decode performance issue.
* commit '227b7131d1e4b5edce1c3eecc9a7e7f560363093':
Move dataspace selection-specific coding to the callbacks as preparation for the H5Sencode changes.
* commit '4142d8b7d4ac48cffcc2b92af4a54352b329865b':
Move dataspace selection-specific coding to the callbacks as preparation for the H5Sencode changes.
1. restored the datatype, dataspace, and LCPL of the dataset for VOL connector back to the properties.
2. splitted external.c and vds.c because they called HDsetenv in the program, instead using shell scripts to set the environment variables.
3. changed H5CX_get_vds_prefix and H5CX_get_ext_file_prefix to use H5P_peek instead of H5P_get.
* commit '5774e451dde6410ab2491acda309c43517a49935':
Add help info
Update for namespace
Add namespace option to dependencies
Correct examples for packaging
HDFFV-10741 - add option to skip tool error stack tests
* commit '844f38c6462f42a9a3a6002de6a4acb0520b8fb7':
Correct entry
Snapshot version 1.11 release 4. Update version to 1.11.5.
Allow option to select NAMESPACE
Correct examples for packaging
* commit 'fe104cc38ffbdb39d3e04da107d86ebfc7e8b622':
Test improvement Description Moved the new tests to a more appropriate test function. Platforms tested: Linux/64 (jelly)
Fixed HDFFV-10210 and HDFFV-10587 Description: - Added parameter validation (HDFFV-10210) - Added detection of division by zero (HDFFV-10587 - CVE-2018-17438) - Fixed typos in various tests Platforms tested: Linux/64 (jelly) Linux/64 (platypus) Darwin (osx1011test)
* commit 'b66df06dbf607741127137ebfda373890cb37649':
Changes that show the right way to iterate over enums.
Changes that show the right way to iterate over enums.
Used the H5_INC_ENUM macro to squash enum value increment warnings.
Description:
- Added parameter validation (HDFFV-10210)
- Added detection of division by zero (HDFFV-10587 - CVE-2018-17438)
- Fixed typos in various tests
Platforms tested:
Linux/64 (jelly)
Linux/64 (platypus)
Darwin (osx1011test)
* commit '0c20c65e2f3abf390ad87c9167daca4cdff2de39':
Added the HDopen work-around on windows to pio_engine.c
Adds _wopen support on Windows so that files with UTF-8 names can be opened.
Fix issues with direct chunk operations and the single chunk index.
* commit 'c812a6c9ec5ff440dc405167c54b86fdb18dcfa4':
Fix issue with direct chunk write not updating the "last chunk" index cache. Fix issues involving datasets being "no allocated" when they contain cached raw data.
(Primarily for use in the async VOL connector, which has to schedule API
operations for future execution and then restore the state of the library when
the operation actually executes)
This implicitly adds support for changing the VOL connector for command-line
tools or any application linked with the library.
Also, add 'make check-vol' support for all directories, clearing up necessary
issues in testing scripts, etc.
* commit '6d07eb126e63b025199a5fe68f41a42269b5921b':
Update standard check
Fix typo
HDFFV-10703 Update version match logic
HDFFV-10703 revert pre-3.11 option
HDFFV-10703 macro vars need quotes
HDFFV-10703 Update CMake commands to latest standard
Use correct variable name
* commit '5ad3891d9b861593ebe25d540bed2d913eb83aba':
Remove ' ' (typo).
Code improvement
Removed an extra "using" statement
Adding documentation
More changes to align with incoming selection improvements.
Added C++ wrapper for H5Ovisit2
Adding a C++ wrapper
C++ wrapper for H5Ovisit2
* commit '6c212353c45ef84b5c977c6019e4d72263534aea':
Code improvement
Removed an extra "using" statement
Adding documentation
Added C++ wrapper for H5Ovisit2
Adding a C++ wrapper
Description:
- Added a wrapper for H5Ovisit2 to class H5Object
// Recursively visit elements reachable from this object.
void visit(H5_index_t idx_type, H5_iter_order_t order,
visit_operator_t user_op, void *op_data, unsigned int fields);
- Fixed various typos in documentation
Platforms tested:
Linux/64 (jelly)
Linux/64 (platypus)
Darwin (osx1011test)
* commit 'b5a4f470cdf7113c0691cca6fb201672c4e1d86d':
Use STREQUAL
Add command to build options and check defines
Skip testing of cache_image test
CMake if statement correction
* commit 'edcbc9966f5c742970ad32eb50a7925fcf7ab677':
Refactor parallel test names
Rename parallel tests to start with PAR_
Distinguish parallel performance tests
Set the full generator name
Add new cmake copy file tests to ignore
Use correct values for other file versions
CMake script changes to simplify execution
* commit '36e3d1600a3f9d771cfb9b4475f23c57d4b10280':
Refactor out timeout values
Correct name of variable
Remove F2003 Option and references
Remove obsolete file
Update examples version
Update to example reference version
Switch OFF if compression source is not available
Fail configure if SZIP/ZLIB is requested but not found
* commit '7f5741825db0737db5ac46c43009681508a73002':
Add release notes information.
Modification based on feedback from pull request.
Modification based on feedback from pull request.
There is performance issue when closing an object. The slow down is due to the search of the "tag_list" to find out the "corked" status of an object. The fix: (1) Add a counter "num_objs_corked" in the cache structure to track the number of "corked" objects. (2) Skip the search of "tag_list" if the counter is zero i.e. no "corked" objects.
the "tag_list" to find out the "corked" status of an object.
The fix:
(1) Add a counter "num_objs_corked" in the cache structure to track the number
of "corked" objects.
(2) Skip the search of "tag_list" if the counter is zero i.e. no "corked" objects.
* commit '6c2a4b728f3c8eadb18d4e10d41bdbd7d1c8f3e8':
Add LOCAL_BATCH_SCRIPT_ARG variable for submitting account information and other batch command variations with ctest command.
Add batchscript options that work when running ctest directly.
* commit 'c348d453b583ec9092152e6aef1f7fa9854cf5d9':
Correct H5DIFF to PH5DIFF in *ctestS.sl.in.cmake list of parallel tests reserved for *ctestP.sl.cmake. Revise options to run batch tests and alternative options for cross compile in HPC/*-HDF5options.cmake.
Change examples file to zip for CMake zip file.
Update bin/release for filename changes and HPC-CMake tar file corrections.
Add new files to MANIFEST. Update permissions for raybsub to 755.
* commit '055208b71c573be3d4b1047b840d4d8b255bc25e':
Minor tweak to CMake preadwrite entry.
Updated sec2, log, and core VFDs to use pread/pwrite when available (can be controlled via a configure/CMake option)
* commit '5f1f231310bcb3278f11a51e1ea47782f6ee3235':
Updated H5Dio.c comment.
Fixed a valgrind issue with the unprotect log message. The logging call was placed after the point where the cache entry was freed. The fix was to just use the entry's address directly in the log call instead of the entry itself.
Fixed a bug revealed by the dtransform test. A datatype transform buffer needed to be calloc'd.
* commit 'bae744199b69eb7c3f5eda2441cca30cb8c6175f':
Add more specific batch scripts. Add script raybsub to handle submitting .lsf files on ray with bsub < script.lsf syntax that we couldn't handle with CMake. Add hpc-cmake-tgz option for bin/release.
Account info can be submitted as an argument to run the batch scripts. Special versions for sandia are not needed.
Add HDF5options.cmake files for serial and parallel tests on HPC machines (where tests are to be run as batch jobs. Add README_HPC file with instructions for setting up configuration to use batch scripts for running tests on HPC machines and some pointers for cross compiling.
Remove knl configuration from ctestS.sl.in.cmake. Add sandia batch files with account info.
Add batch files with batch parameters specific to cori and ray. Release.txt entry for skipping long double dt_arith tests on ppc64.
Add script raybsub to handle submitting .lsf files on ray with bsub <
script.lsf syntax that we couldn't handle with CMake.
Add hpc-cmake-tgz option for bin/release.
(1) Increase the # of records to write (Nrecords) in testvfdswmr.sh.in so as to ensure the writer
will not exit before the reader
(2) Use H5E_BEGIN_TRY/END_TRY when H5Fopen() the test file in reader tests
(3) Add "READER" or "WRITER" to debugging messages
(4) Misc cleanup
call was placed after the point where the cache entry was freed.
The fix was to just use the entry's address directly in the log
call instead of the entry itself.
open in the VFD SWMR reader case.
Note that the following failures in testvfdswmr.sh:
1) Unable to find metadata file on VFD SWMR reader open.
2) Occasional sanity check failures in the page buffer on raw data write.
3) Filter failures on raw data read in VFD SWMR readers when compression
is enabled.
4) Unexpected data errors in VFD SWMR readers when compression is
disabled.
Note that I expect that items 3 & 4 two aspects of the same issues -- the
fact that we don't guarantee that raw data is consistent with metadata.
Item 2) must be addressed, but it is so infrequent that it isn't doesn't
affect the conclusion VFD SWMR seems to work, and thus it can wait until
phase 2.
I am given to understand that Vailin has largely addressed item 1),
and will be checking in her solution to this soon.
Tested on Charis and Jelly.
machines (where tests are to be run as batch jobs.
Add README_HPC file with instructions for setting up configuration
to use batch scripts for running tests on HPC machines and some
pointers for cross compiling.
* commit '2fe69e7639198b174da393e47a029d0ac35b31c8':
Added test for HDFFV-10588
Fixed HDFFV-10684
Fixed HDFFV-10586 and HDFFV-10588 Description: HDFFV-10586 CVE-2018-17434 Divide by zero inh5repack_filters Added a check for zero value HDFFV-10588 CVE-2018-17437 Memory leak in H5O_dtype_decode_helper This is actually an Invalid read issue. It was found that the attribute name length in an attribute message was corrupted, which caused the buffer pointer to be advanced too far and later caused an invalid read. Added a check to detect attribute name and its length mismatch. The fix is not perfect, but it'll reduce the chance of this issue when a name length is corrupted or the attribute name is corrupted. Platforms tested: Linux/64 (jelly) Linux/64 (platypus) Darwin (osx1010test)
HDFFV-10586 and HDFFV-10588
* commit '25cd1ab02b9ddaf58a4f5422f4ab4fde411e050a':
Added test for HDFFV-10588
Fixed HDFFV-10684
Fixed HDFFV-10586 and HDFFV-10588 Description: HDFFV-10586 CVE-2018-17434 Divide by zero inh5repack_filters Added a check for zero value HDFFV-10588 CVE-2018-17437 Memory leak in H5O_dtype_decode_helper This is actually an Invalid read issue. It was found that the attribute name length in an attribute message was corrupted, which caused the buffer pointer to be advanced too far and later caused an invalid read. Added a check to detect attribute name and its length mismatch. The fix is not perfect, but it'll reduce the chance of this issue when a name length is corrupted or the attribute name is corrupted. Platforms tested: Linux/64 (jelly) Linux/64 (platypus) Darwin (osx1010test)
* commit 'f033214796c383a9018dd760e92257dc27d7b852':
Add clear objects to ignore tests
Add clear objects to ignore tests
Add new clear tests to ignore list
Cleanup variable names
HDFFV-10685 Correct syntax
HDFFV-10685 Corrected version
HDFFV-10685 Always set Mac version flags - use soversion
Description:
The test for HDFFV-10588 has revealed a bug in H5Ewalk.
H5Ewalk did not stop midway even when the call back function returns
H5_ITER_STOP. This is because a condition is missing from the for
loops in H5E__walk causing the callback functions unable to stop until
all the errors in the stack are iterated. Quincey advised on the final
fix. In this fix, "status" is switched to "ret_value" and HGOTO_ERROR
to HERROR, and the for loops won't continue when "ret_value" is not 0.
Platforms tested:
Linux/64 (jelly)
Linux/64 (platypus)
Darwin (osx1010test)
in un-related tests (i.e. earray, fheap, etc.).
On jelly and charis, vfd_swmr now passes.
testvfdswmr.sh displas the following failures:
1) occasionally ccan't open the metadata file. This shows
up more on jelly than charis.
2) occasional complaints about incorrect raw data
3) occasional complaints from Quincey's evict tagged
entries code that it can't evict all the tagged entries.
4) Numerous filter failures. At a guess, this is an artifact of
raw data not making it to file in sync with the metadata.
I didn't see this on charis, as I don't have compression
configured.
5) An assertion failure in the page buffer in which a sanity check
is failing in the code to update the replacement policy.
This is worrying --- I'll need to look into it on my return.
Description:
HDFFV-10586 CVE-2018-17434 Divide by zero inh5repack_filters
Added a check for zero value
HDFFV-10588 CVE-2018-17437 Memory leak in H5O_dtype_decode_helper
This is actually an Invalid read issue. It was found that the
attribute name length in an attribute message was corrupted,
which caused the buffer pointer to be advanced too far and later
caused an invalid read.
Added a check to detect attribute name and its length mismatch. The
fix is not perfect, but it'll reduce the chance of this issue when a
name length is corrupted or the attribute name is corrupted.
Platforms tested:
Linux/64 (jelly)
Linux/64 (platypus)
Darwin (osx1010test)
* commit '90d13bef33f9e2e80b23996a0c39f16f7c34ecf8':
Fixed typo Platforms tested: Darwin (osx1010test)
Refixed HDFFV-10578 Description: Applied Neil's fix for this issue after removing previous attempt. The resources are now released in init_objs() when failure occurs there. Neil will fix HDFFV-10676 separately. Platforms tested: Linux/64 (jelly) Linux/64 (platypus) Darwin (osx1010test)
Removed previous change in H5O__chunk_deserialize().
Removed the previous change in H5O__chunk_deserialize()
Removed previous change in table_list_add().
Removed the previous change in table_list_add()
Updated per review Description: HDFFV-10676 - CVE-2018-13873 Changed the new assert to if statement, per Dana's comment. Platforms tested: Linux/64 (jelly)
HDFFV-10578 and HDFFV-10676 Description: HDFFV-10578 - CVE-2018-17234 The file has some issue, however, there was a bug in h5dump that caused memory leaks after the problem in the file was encountered. The bug was that an if statement was missing in the function table_list_add() resulting in the memory not being freed at a later time. After the fix had been applied, there were no more leaks after h5dump detected the issue in the file and reported the error.
* commit 'b02de315b93ac29d2483a91d526b110a25073505':
NNSA Tri-LabsTRILAB-98: Another two test cases out.
NNSA Tri-Labs TRILAB-98: Taking out a few more test cases.
NNSA Tri-Labs TRILAB-98 dt_arith and cpp_testhdf5 tests fail on sierra.llnl.gov: According to the group decision, simply provide a macro to disable some failing test cases on sierra (IBM power9 cpu). All failing cases involve long double data type.
* commit 'b4d4d371a03158c39f120b1bde6c4bd51f1b2eb6':
HDFFV-10443: Add "field" parameter to H5Oinfo* and H5Ovisit* APIs.
Added chunked dataset, H5S_ALL test
typo
typo
updated defined constants
Documented HDFFV-10652
HDFFV-10652 Implemented a process-0 read and then broadcast for collective read of full datasets (H5S_ALL) by all the processes in the file communicator.
Removed reason for breaking read-proc0-and-bcast
switched to using CX instead of a global var.
HDFFV-10652 Implemented a process-0 read and then broadcast for collective read of full datasets (H5S_ALL) by all the processes in the file communicator.
* commit 'e27df5a5fe365480b0809ce523b0e9cd1096a43d':
Add variables to set up module switch between configure and build for cross compiling on CrayXC40.
Remove install-exec-hook from hl/fortran/Makefile.am. Add check for existing links in hl/fortran/src/Makefile.am.
Configure build directory in batch scripts.
Add libhdf5_hl_fortran symlinks for libhdf5hl_fortran files. Set ctest*.lsf.in.cmake scripts for sierra. Correct src/CMakelists.txt syntax error.
Correct syntax error in CMakeLists.txt.
Add .lsf batch scripts and insert the build directory into the scripts during configuration.
Description:
Applied Neil's fix for this issue after removing previous
attempt. The resources are now released in init_objs() when
failure occurs there.
Neil will fix HDFFV-10676 separately.
Platforms tested:
Linux/64 (jelly)
Linux/64 (platypus)
Darwin (osx1010test)
Avoid potential invalid read when decoding unknown object header message.
* commit '685288beabc4f4383d69ec07be05c067fab78544':
Delay checking if decoded message's "shareable" flag is appropriate for the message type until we've verified we understand the message type. Reduce size of H5O_msg_class_g to *not* include space for H5O_BOGUS_INVALID. Make bogus messages shareable. Add new bogus message test with shareable messages to cover the formerly problematic code. Re-run gen_bogus.c to add this test case and also to fix the bogus_invalid messages that were no longer H5O_BOGUS_INVLAID due to a new message class being added in a previous commit. Added comment to remind developers to run gen_bogus.c when adding a new message class.
* commit 'ab5fe769ab711f736238abc89ef215a6ecff5f7e':
HDFFV-10625 -- Implemented a process-0 read and then broadcast for collective read of full (HS_ALL), contiguous, atomic datasets by all the processes in the file communicator.
indent change
changed logic statement in if
Added chunking test, fixed issue with CX set
Correct another git merge failure.
Correct misplaced line from git merge.
Updated and refined version of Scot's "rank 0 bcast" changes.
the message type until we've verified we understand the message type.
Reduce size of H5O_msg_class_g to *not* include space for
H5O_BOGUS_INVALID. Make bogus messages shareable. Add new bogus
message test with shareable messages to cover the formerly problematic
code. Re-run gen_bogus.c to add this test case and also to fix the
bogus_invalid messages that were no longer H5O_BOGUS_INVLAID due to a
new message class being added in a previous commit. Added comment to
remind developers to run gen_bogus.c when adding a new message class.
* commit '7bf6d847ae36940f1b1147820356795c7764dbeb':
Remove incorrect comment.
Updated error stacks for h5dump testing.
Corrected comment in src/H5VLint.c, fixed pass-through info size in src/H5VLpassthru.c, switched to stashing VOL connector ID & info in API context (in src/H5CX.c, src/H5CXprivate.h, src/H5F.c, src/H5Fint.c, and src/H5Fefc.c), patched up all sorts of issues in the tests, to make them work with 'check-vfd' (and 'check-vol' again).
Clean up style issues, make function callbacks static, correct debug statements, etc. No functionality changes at all.
* commit 'f0656a57dc3baf02709e1ad5e496cf63cf820877':
Remove incorrect comment.
Updated error stacks for h5dump testing.
Corrected comment in src/H5VLint.c, fixed pass-through info size in src/H5VLpassthru.c, switched to stashing VOL connector ID & info in API context (in src/H5CX.c, src/H5CXprivate.h, src/H5F.c, src/H5Fint.c, and src/H5Fefc.c), patched up all sorts of issues in the tests, to make them work with 'check-vfd' (and 'check-vol' again).
* commit '8a5d2c130a93a665933cda8db8554d006b7ca093':
Clean up style issues, make function callbacks static, correct debug statements, etc. No functionality changes at all.
Description:
HDFFV-10578 - CVE-2018-17234
The file has some issue, however, there was a bug in h5dump that caused
memory leaks after the problem in the file was encountered. The bug
was that an if statement was missing in the function table_list_add()
resulting in the memory not being freed at a later time.
After the fix had been applied, there were no more leaks after h5dump
detected the issue in the file and reported the error.
In H5O__chunk_deserialize, replaced an assert with an if statement
and reporting error, per Neil's recommendation
HDFFV-10676 - CVE-2018-13873
Also in H5O__chunk_deserialize, added an assertion to detect
out of bound ids
src/H5VLpassthru.c, switched to stashing VOL connector ID & info in
API context (in src/H5CX.c, src/H5CXprivate.h, src/H5F.c, src/H5Fint.c, and
src/H5Fefc.c), patched up all sorts of issues in the tests, to make them work
with 'check-vfd' (and 'check-vol' again).
* commit '0e34f0feaaeb6d2286f87f695f25fae45c070a42': (26 commits)
HDFFV-10664 update reference file
HDFFV-10664 update reference files
HDFFV-10664 add missing function and check for restriction
Remove "bad" statements at end of TEST_ERROR. Formerly arguments to FAIL_PUTS_ERROR
Add RELEASE.txt entry for HDFFV-10596.
Add semicolon at end of TEST_ERROR() invocation for consistency.
Uninstall should remove the libhdf5_hl_fortran links.
Add symlinks named libhdf5_hl_fortran* to libhdf5hl_fortran in Makefile.am to match cmake install and the name pattern for other hl lib files.
Change FAIL_PUTS_ERROR to TEST_ERROR.
Cleaned up the parallel flush test and set t_pflush1 to always fail. Also set CMake to handle this.
Update calls: H5Dcreate to H5Dcreate2, H5Acreate to H5Acreate2.
Removed a comment.
Fix comment
Align H5Lcreate_ud behavior with documentation for NULL udata pointer
Update toolchain comment
Use c99 standard cmake variable
Add PGI toolchain
Correction based on code review.
HDFFV-10664 add check for state before set call
HDFFV-10546 refactor variable name
...
* commit 'f4c9ec5d41aa53829617274d3286f1e9a45ae57e':
Add RELEASE.txt entry for HDFFV-10596.
Uninstall should remove the libhdf5_hl_fortran links.
Add symlinks named libhdf5_hl_fortran* to libhdf5hl_fortran in Makefile.am to match cmake install and the name pattern for other hl lib files.
* commit 'f7115c31363e80e850c0a228e63a5a035c48b74a':
Remove "bad" statements at end of TEST_ERROR. Formerly arguments to FAIL_PUTS_ERROR
Add semicolon at end of TEST_ERROR() invocation for consistency.
* commit 'd59aa1d75cb6d0ae4f5964d017e275aa53394eb6':
Cleaned up the parallel flush test and set t_pflush1 to always fail. Also set CMake to handle this.
* commit '763db8178e673fdcae4a370fe2e6d345f905d5f1':
Change FAIL_PUTS_ERROR to TEST_ERROR.
Update calls: H5Dcreate to H5Dcreate2, H5Acreate to H5Acreate2.
the memory manager -- details shown below.
Note that there are other issues as well -- this is not a
working version.
[mainzer@jelly test]$ ./vfd_swmr
Testing Configure VFD SWMR with fapl PASSED
Testing VFD SWMR configuration for the file and fapl PASSED
Testing H5Fvfd_swmr_end_tick() for VFD SWMR PASSED
Testing Create/Open/Flush an HDF5 file for VFD SWMR PASSED
Testing Verify the metadata file for VFD SWMR writer vfd_swmr: H5MVsection.c:233: H5MV__sect_can_merge: Assertion `((sect1->sect_info.addr)!=((haddr_t)(long)(-1)) && (sect2->sect_info.addr)!=((haddr_t)(long)(-1)) && (sect1->sect_info.addr)<(sect2->sect_info.addr))' failed.
Abort (core dumped)
[mainzer@jelly test]$
Merged to catch daily test.
* commit '40c55f24ecbdd322fba527d768689fbdbd268374':
Removed a comment.
Fixed documentation - typo only
Fixed CVE division-by-zero issues Description: Fixed HDFFV-10577 and similar issues found in H5Dchunk.c. All the occurrences are in: H5D__create_chunk_map_single H5D__create_chunk_file_map_hyper H5D__chunk_allocate H5D__chunk_update_old_edge_chunks H5D__chunk_prune_by_extent H5D__chunk_copy_cb H5D__chunk_collective_fill Platforms tested: Linux/64 (jelly) Linux/64 (platypus) Darwin (osx1010test)
* commit 'ed0c468902409ae7fa8fb91c5f5a3a3093846736':
HDFFV-10664 add check for state before set call
HDFFV-10546 refactor variable name
HDFFV-10664 minimized dataset headers for java interface
* commit 'd3dff6efe6f769b219f9dcccebd057afe75ed3c7':
Correction based on code review.
Changes based on feedback from pull request.
Fix for HDFFV-10659: The library abort with "infinite loop closing library" after deleting attributes in densed storage. The fix: When deleting attribute nodes from the name index v2 B-tree, if an attribute is found in the intermediate B-tree nodes, which may be merged/redistributed in the process, we need to free the dynamically allocated spaces for the intermediate decoded attribute.
* commit 'af21ce90d9465e805807f17da21f93e64e03492a': (46 commits)
Changed to Unix line endings and chmod -x.
Fix runtest case for data to stdout in tool
Updated the parallel install docs.
Eliminated the need for a separate script variable.
Flipped swapped testing strings.
Specify variable type. Remove unnecessary whitespace.
Added test_vol_plugin.sh to the list of scripts to clean
Fixed a CMake build issue (CMake still doesn't run the VOL plugin tests)
Added more sub-tests to the VOL plugin test.
Added a simple test for registration of VOL connector plugins. Autotools only for right now, but this will be fleshed out in future work.
OHDR tests now accept h5_fileaccess() fapls. Formatting, informative comments, and minor renaming.
Fix some CMake listings
Add error checking to the minimized dset header size calculation. Update printf->HDprintf statements.
Remove `#if 0` block of deprecated code.
Fix mistake with H5E_BEGIN_TRY {...} H5E_END_TRY block containing ERROR-raising macros. Formatting tweaks.
Added a helpful message to the flush script.
Added a shell script so we can run the parallel flush test on OpenMPI.
Add "compact" storage test to relative header size comparisons.
Formatting adjustments.
Formatting adjustments
...
* commit 'b30a1fe44aa804a67f2291005bf8135ac5c2dc99':
Updated the parallel install docs.
Eliminated the need for a separate script variable.
Added a helpful message to the flush script.
Added a shell script so we can run the parallel flush test on OpenMPI.
* commit 'd6c2a96ac2f103d90b96d5b39814810e6a31ef99':
Updated the parallel install docs.
Eliminated the need for a separate script variable.
Added a helpful message to the flush script.
Added a shell script so we can run the parallel flush test on OpenMPI.
* commit '3ca19cca5395d79be69209f8d7d0a2b06834a648':
Flipped swapped testing strings.
Added test_vol_plugin.sh to the list of scripts to clean
Fixed a CMake build issue (CMake still doesn't run the VOL plugin tests)
Added more sub-tests to the VOL plugin test.
Added a simple test for registration of VOL connector plugins. Autotools only for right now, but this will be fleshed out in future work.
Dataset object header minimization
Jira TRILAB-45
* commit 'f54fb420d08ea0a7e7b026150c0ab559803e0acd': (34 commits)
Specify variable type. Remove unnecessary whitespace.
OHDR tests now accept h5_fileaccess() fapls. Formatting, informative comments, and minor renaming.
Fix some CMake listings
Add error checking to the minimized dset header size calculation. Update printf->HDprintf statements.
Remove `#if 0` block of deprecated code.
Fix mistake with H5E_BEGIN_TRY {...} H5E_END_TRY block containing ERROR-raising macros. Formatting tweaks.
Add "compact" storage test to relative header size comparisons.
Formatting adjustments.
Formatting adjustments
Move H5Fset_dset_no_attrs_hint VOL operations to native. Move minimzied object header tests from separate file to test/ohdr.c Some formatting changes.
Re-format test/ohdr_mindset.c Fix a few transcription errors in other test files.
formatting adjustments
Reformat to be more consistent with existing code. Fix a few typos.
add missing paramter - macro seemingly unused, but absence results in compiler complaint
fix reference; move declaration in file
Remove unused debugging print in '#if 0' block
Fix typo. Fix CHECK of wrong ID (dset[2|3]_id).
change test file name and apply 'h5_fixname' to it
Sidestep and hide&flag minor issues causing test failures.
Incorporate minimized dset ohdr tests into extant suite.
...
after deleting attributes in densed storage.
The fix: When deleting attribute nodes from the name index v2 B-tree,
if an attribute is found in the intermediate B-tree nodes, which may be
merged/redistributed in the process, we need to free the dynamically
allocated spaces for the intermediate decoded attribute.
* commit 'b4fe787bb9fe4bfc4709a124df59dd987c9a09d2':
Add test for H5Arename NULL/empty attribute name fix
align H5Arename behavior with H5Arename_by_name
Description:
Fixed HDFFV-10577 and similar issues found in H5Dchunk.c. All
the occurrences are in:
H5D__create_chunk_map_single
H5D__create_chunk_file_map_hyper
H5D__chunk_allocate
H5D__chunk_update_old_edge_chunks
H5D__chunk_prune_by_extent
H5D__chunk_copy_cb
H5D__chunk_collective_fill
Platforms tested:
Linux/64 (jelly)
Linux/64 (platypus)
Darwin (osx1010test)
* commit '991996b25122ff010b9af1bbb6cc05be491ce038':
Remove unused CMake files
Update option text
Add reference file to list
HDFFV-10656 remove moved source
HDFFV-10656 Add CHECK_VOL support to CMake
* commit '75a03ea0c36f24a6a3982b75a8c52325879bb569':
Moved private native VOL connector functions to H5VLnative_private.h.
Moved the remainder of the code into separate files.
Split the native VOL connector code into multiple files and moved the attribute code over.
(1) Assertion failure in the vfd_swmr test
(2) Reader error in the vfd swmr concurrent tests
Also fixes for:
(a) Use H5MV_alloc() to allocate space for md_pages_reserved when creating the metadata file in H5F__vfd_swmr_init()
(b) Remove a multi-page (when vfd_swmr_writer is true) from the page buffer in H5MF_xfree()
* commit '70c8c629ce1d7b1eb98220135ddcca1bf9c5b7fc':
Added missing H5O calls.
Fixed a missing H5R call.
Made hbool_t processing consistent in va_lists.
Fixed an enums-in-va_list issue in the passthru VOL connector.
Updated enums in VOL va_lists to be passed as ints.
* commit '3597bd469ff0073134b31765d9b87660cd3e0039':
Revert change to MSC_VER, in favor of Dana's
Remove H5VL_FILE_CACHE_VOL_CONN from file specific callbacks, as it turns out not to be necessary.
* commit 'e21c1cf7c6ee6dba5dd5bfd1a525227ac1304b53':
Removed H5I_REFERENCE from the library. It has always been unused and has been marked 'deprecated' since 1.10.0. Fixes HDFFV-10252.
* commit 'a7618f432b671ad041ae63a83bf9b9bb08096dbe':
Updated with extra sanity check and improved comment.
Fixed an issue in the passthru Windows va_copy hack.
* commit '9f175afce8c08bf6193e45ce391e05e4198dd61c':
Updated va_arg-->HDva_arg in the native VOL connector.
Added HD to va_* calls in a couple of library files.
* commit 'b0457117d867d9850d042d7c3f71a747899b932e':
Further va_list improvements.
Added a helpful comment to the va_copy() Windows hack.
Added a work-around for va_copy() on pre-2013 versions of Visual Studio.
* commit '9ae9b3ef513613bb2fca15f96461a156148c2627':
Moved the native VOL connector's optional enums to the public headers and renamed to include native/NATIVE in the name.
* commit '50ca26f3ec428de408040335f2a2999e750d98b3':
Updated the passthru VOL connector: * snprintf --> sprintf (for Windows compat) * tabs --> spaces * removed spurious parens from returns * Changed the logging macro to something less clash-prone
Make sure to free copy of VOL connector's name when H5VL_register_connector fails
* commit '833a4f73fd8dc696f8bf500cd842d88949bb1519':
Fix plugin code from referencing invalid key ID value, and also switch from strtok() to strtok_r() to avoid possible interference with / to application use of strtok().
* commit '76c0d8c434186107176fc83a0ce4e94b584196ae':
Add test for H5Aget_info(_by_name/_by_idx) NULL info pointer
align H5Aget_info behavior with H5Gget_info when null attribute pointer is supplied
* commit 'deb7d2fb9714acc8222815328f71b0aae7e74275':
Use version 2 functions in test file to address compile errors for --with-default-api-version=v16 option.
* commit '139a12d13cda046fd3cd64a4f097eb3485f50173':
HDFFV-10632 correct number
HDFFV-10632 stder/stdout change note
Add stamp files for generated files
HDFFV-10632 update autotools test scripts to match cmake
HDFFV=10632 test cleanup
Fix special macro stdout as empty
Need nonexistent file to skip stdout compare
HDFFV-10632 - split stderr and stdout in cmake
* commit '4643860aa940fb52da251f7023ec2e48d7d10f34':
HDFFV-10635: Some minor changes to the test case and the comments in the library.
HDFFV-10635: add a test case.
HDFFV-10635: Allowing to write the same variable-length element more than once.
* commit '3e8599591504c95d8a97100b9546174f6132dc97':
HDFFV-10635: Some minor changes to the test case and the comments in the library.
HDFFV-10635: add a test case.
HDFFV-10635: Allowing to write the same variable-length element more than once.
* commit '7796cbd4174e85aa211dbffa562ada7413f1dbf7':
Fix build errors VOL changes, when using --disable-deprecated-symbols.
Removed RELEASE.txt entry concerning --enable-debug/production configure flags as those changes were pushed to HDF5 1.10.
Remove reference to h5_vol_external_log_native.
* commit 'dd23dac577c70d7fc0aa3fe21c1ebf52a18975d2':
Removed RELEASE.txt entry concerning --enable-debug/production configure flags as those changes were pushed to HDF5 1.10.
* commit '55e87e5b86aacbb37c57614cd2e869541291caad': (51 commits)
Correct Windows build.
Refactor allocating & releasing pass through wrapper objects.
Fix typo
Remove duplicated comment.
Add a couple of missing prototypes for static routines, along with updating a comment from VOL plugin -> connector.
Switch H5VL_class_value_t from unsigned to int, and add "invalid" ID value.
Correct "make check-vol" regression test target.
Switch H5VL_class_value_t from enum to unsigned integer
Add support for "make check-vol", along with a few minor cleanups, etc.
Refactor infrastructure for setting FAPL information from environment variables during testing, including connecting native, pass-through, and dynamically loaded VOL connectors. Also bring native and pass-through VOL connectors into alignment, removing the "H5VLnative_private.h" header.
Remove example VOL connector, since it's been superceded by the pass-through VOL connector in the src subdirectory.
Add pass-through VOL connector
Update API tracing info for VOL functions and regenerate trace macros for VOL API calls.
Added 'notify' callback for async requests; switched VOL class and info comparison to return comparison value as parameter, so they can return error values; "cancelled" -> "canceled"; switched order of 'wrap_object' and 'free_wrap_ctx' management callbacks.
Add using FAPL from h5_fileaccess() to more tests.
Remove unused test for unimplemented routine.
Switch loc_params to VOL callbacks to pass struct by pointer instead of by value.
Add info_to_str and str_to_info "management" callbacks for serializing and deserializing a connector's info object.
Revert some of the changes to support the original property list value of metadata read attempts.
Add VOL connector info to the flie access property list returned from H5Fget_access_plist(). Also, other misc. cleanups, etc.
...
* commit 'c58d84e721ef9fbda63abc834767a28dc978d121': (50 commits)
Refactor allocating & releasing pass through wrapper objects.
Fix typo
Remove duplicated comment.
Add a couple of missing prototypes for static routines, along with updating a comment from VOL plugin -> connector.
Switch H5VL_class_value_t from unsigned to int, and add "invalid" ID value.
Correct "make check-vol" regression test target.
Switch H5VL_class_value_t from enum to unsigned integer
Add support for "make check-vol", along with a few minor cleanups, etc.
Refactor infrastructure for setting FAPL information from environment variables during testing, including connecting native, pass-through, and dynamically loaded VOL connectors. Also bring native and pass-through VOL connectors into alignment, removing the "H5VLnative_private.h" header.
Remove example VOL connector, since it's been superceded by the pass-through VOL connector in the src subdirectory.
Add pass-through VOL connector
Update API tracing info for VOL functions and regenerate trace macros for VOL API calls.
Added 'notify' callback for async requests; switched VOL class and info comparison to return comparison value as parameter, so they can return error values; "cancelled" -> "canceled"; switched order of 'wrap_object' and 'free_wrap_ctx' management callbacks.
Add using FAPL from h5_fileaccess() to more tests.
Remove unused test for unimplemented routine.
Switch loc_params to VOL callbacks to pass struct by pointer instead of by value.
Add info_to_str and str_to_info "management" callbacks for serializing and deserializing a connector's info object.
Revert some of the changes to support the original property list value of metadata read attempts.
Add VOL connector info to the flie access property list returned from H5Fget_access_plist(). Also, other misc. cleanups, etc.
Switch tool error output to match VOL changes within library
...
(2) Test files for encoding/decoding property lists
(3) Fix test failures for PB statistics in test/page_buffer.c
(Will double check with John later about PB statistics collection)
variables during testing, including connecting native, pass-through, and
dynamically loaded VOL connectors. Also bring native and pass-through
VOL connectors into alignment, removing the "H5VLnative_private.h" header.
comparison to return comparison value as parameter, so they can return error
values; "cancelled" -> "canceled"; switched order of 'wrap_object' and
'free_wrap_ctx' management callbacks.
* commit '8cf3bfb14bd2a80f13d269a9e84cd99f86f19254':
Yanked all MPI-1 calls
Updated H5S to use the MPI-2 function MPI_Type_get_exten() where available. OpenMPI 4.0 removed the deprecated MPI-1 MPI_type_extent() call by default, so this avoids needing a special OpenMPI build.
* commit 'd517bdf9d36f423e7abad34e109b6ce694fd03c8':
Pulled the RELEASE.txt note concerning HDFFV-10633 since that change was pushed to hdf5_1_10 in case we release the Windows VFD in 1.10.5.
* commit '7bf6e98bd921c4ef1899144ab8a1f0d66a141141':
Move wait_H5init.cmake up to config/cmake.
Changes to CMake code to pause cmake/ctest, waiting for batch jobs to run and finish.
Really don't run parallel tests without option specifying script.
Add script and changes to wait for H5Tinit.c Add more sample batch scripts, specifically for sbatch, not for knl cross compile. Don't run parallel tests when no parallel test script is configured in HDF5options.cmake.
* commit 'c923cdad6e515c842f3795a5b6d754ad94021e09':
HDFFV-10571: Minor format changes.
HDFFV-10571: Minor change - reformatting the error check.
HDFFV-10571: Minor change - adding the error check right after decoding of chunk dimension for safeguard.
HDFFV-10571: Minor change - revised the comment to be clearer.
HDFFV-10571 Divided by Zero vulnerability. Minor fix: I added an error check to make sure the chunk size is not zero.
* commit '4f5a52e595194562644517b7600f3b96d4008c17':
HDFFV-10601: Adding the new hdf5/tools/test/perform/chunk_cache.c to MANIFEST.
HDFFV-10601: Some compiler doesn't support clock_gettime(). So I changed to use gettimeofday (HDgettimeofday in the HDF5 library).
* commit 'cd13d24e5140578a880aebe4e2d8b899179d0870':
HDFFV-10601: I added error checking to the HDF5 functions.
HDFFV10601: Adding performance test to verify the improvement.
HDFFV-10601: I changed to a better way to calculate the number of chunks in a dataset.
HDFFV-10601 Issues with chunk cache hash value calcuation:
--src/H5PB.c: checks for size >= page size
--src/H5MF.c: disable/enable page buffering in H5MF_tidy_self_referential_fsm_hack()
--src/H5MFsection.c: call H5PB_remove_entry() for both raw/metadata pages in H5MF__sect_small_merge()
(B) Port and modify existing concurrent swmr tests to VFD SWMR. Also modify the following:
--remove flushes from VFD SWMR writer tests
--set Nreaders to 0 in test/testvfdswmr.sh.in to test for writers only
Please enter the commit message for your changes. Lines starting
Add more sample batch scripts, specifically for sbatch, not for knl
cross compile.
Don't run parallel tests when no parallel test script is configured in
HDF5options.cmake.
* commit '4616c1e863ab756d22af696672ba9bc2fe0eea25':
Add custom target to force wait on generated file
TRILAB-81 fix typo
TRILAB-82 fixed errors found in review
Update windows test machine info
TRILAB-81 Output error if filename length is greater then 255
TRILAB-81 check for func success before using value
TRILAB-81 Coverity fix
Move options to root CMakeLists.txt
TRILAB-34 fix required through testing
TRILAB-34 add batch option (also patch2 from 10633)
TRILAB-81 coverity fixes
* commit '1652a60fcbe9894d7c34a778fb24dce1979fb9a9':
HDFFV-10605 Only test plugins if SHARED enabled
Correct platforms
Correct extlib depends
Commit version changes for additional files for 1.11.3 snapshot release.
* commit 'e6545faa5699aae4d5dca23ed40e3080fa9dc72d':
Update HDF5-Examples version to be included in CMake-* tar/zip files.
Snapshot version 1.11 release 3.
* commit 'e6545faa5699aae4d5dca23ed40e3080fa9dc72d':
Update HDF5-Examples version to be included in CMake-* tar/zip files.
Snapshot version 1.11 release 3.
* commit 'be0d6b206aaa04f7d9cb406a71aeeb458ba67975':
Update HDF5-Examples version to be included in CMake-* tar/zip files.
Snapshot version 1.11 release 3.
* commit '20ad15a1a8e9008103fe7d30401408eaa2aff6e5':
Fixed exit() to HDexit() in use_append_chunk.c and use_append_mchunks.c
Fixed fprintf to HDfprintf in the following routines: 1) read_uc_file() in use_common.c 2) main() of use_append_chunk.c and use_append_mchunks.c
Fix for HDFFV-10554 use_append_chunks in swmr use case test failure Fix the timing issue of the test by moving the open/close of the test file for the writer to use_append_chunk.c and use_append_mchunks.c.
* commit '20ad15a1a8e9008103fe7d30401408eaa2aff6e5':
Fixed exit() to HDexit() in use_append_chunk.c and use_append_mchunks.c
Fixed fprintf to HDfprintf in the following routines: 1) read_uc_file() in use_common.c 2) main() of use_append_chunk.c and use_append_mchunks.c
Fix for HDFFV-10554 use_append_chunks in swmr use case test failure Fix the timing issue of the test by moving the open/close of the test file for the writer to use_append_chunk.c and use_append_mchunks.c.
Tested serial / debug on Charis and Jelly.
Two known issues:
1) New page buffer seems to expose issues in the accumulator code.
For whatever reason, fheap with the new page buffer exposes corruption
issues if the page buffer uses H5F__accum_read/write(), but the problems
go away if the page buffers uses H5FD_read/write() instead.
Need to either chase this or dis-able page bufffer in combination with
the accumulator.
2) Encountered a number of assertion failures that are explainable by the
free space manager code not telling the page buffer to discard pages
when they are freed.
Wrote code to handle this -- once the free space manager is modified,
this code should be removed and the original assertions restored.
1. H5D__chunk_hash_val: When the number of chunks in the fastest changing dimension is larger than the number of slots in the hash table, H5D__chunk_hash_val abandons the normal hash value calculation algorithm and simply uses the scaled dimension. This will cause chunks a selection that cuts across chunks in dimensions other than the fastest changing to all have the same hash value, and they will therefore always evict each other from the cache, having an obvious major performance impact. Eliminated the check for the number of slots in this function and always use the full algorithm.
2. H5D__chunk_init: When the scaled dimensions (number of chunks in each dimension) are calculated in H5D__chunk_init, a simple divide ("/") operator is used with the dataset size in elements and the chunk size in elements. While this is fine when the dataset size is an exact multiple of the chunk size, in other cases, since "/" rounds down, it results in a scaled dimension one less than it should (it ignores the partial edge chunk). This has trickle down effects on hash value calculation that can cause excess hash value collisions and therefore performance issues. Changed the calculation to (((dataset_size - 1) / chunk_size) + 1).
Tested the build with Autotool and CMake.
* commit '51076d40617aa6f0236809c2c919cc6921b3bbc1':
Fixed exit() to HDexit() in use_append_chunk.c and use_append_mchunks.c
Fixed fprintf to HDfprintf in the following routines: 1) read_uc_file() in use_common.c 2) main() of use_append_chunk.c and use_append_mchunks.c
Fix for HDFFV-10554 use_append_chunks in swmr use case test failure Fix the timing issue of the test by moving the open/close of the test file for the writer to use_append_chunk.c and use_append_mchunks.c.
* commit '99c1e577da3a2dd45fbfee748220986c42962921':
Removed a couple of extraneous commas from VOL example.
Cleaned up VOL example.
Updated MANIFEST file.
Fixed VOL example to use a platform-independent format specifier for size_t.
* commit '99c1e577da3a2dd45fbfee748220986c42962921':
Removed a couple of extraneous commas from VOL example.
Cleaned up VOL example.
Updated MANIFEST file.
Fixed VOL example to use a platform-independent format specifier for size_t.
* commit 'ee12aff2ca89e92b99590f29096794a84c6bacdc':
Removed a couple of extraneous commas from VOL example.
Cleaned up VOL example.
Updated MANIFEST file.
Fixed VOL example to use a platform-independent format specifier for size_t.
* commit 'ff34cd0a0506ca16f4971cf03bd1b6e33bfa132c':
Updated RELEASE.txt to reflect HDFFV-10609 work (H5Xpkg.h files being incorrectly installed).
Add files and changes to cross compile with CrayLinuxEnvironment and run ctest tests with a batch job on knl compute nodes.
* commit 'b24259ed8cdb273273f82c2b3f3a2dcac60b32ba':
Add files and changes to cross compile with CrayLinuxEnvironment and run ctest tests with a batch job on knl compute nodes.
and CMake.
* CMake will no longer install private generated headers and tools
library headers.
* Several empty public header files (which represent internal packages)
were removed. These were only installed by CMake.
* Autotools installs will install H5FDwindows.h.
Fixes HDFFV-10614.
* commit '47f30b474bdc498c20bd6d2a0ba7e8947ab389f0':
HDFFV-10608 Move toolchain files into a subfolder
TRILAB-36 add option for test script batch command
Fixed warnings in the VOL example and run it from the script.
Split H5VLnative.h into public and private files and updated
* commit '096279caf76a542306b840fb90e1e753e8bbf748':
HDFFV-10608 Move toolchain files into a subfolder
TRILAB-36 add option for test script batch command
* commit 'b33a2e7dfe76a2947c12462adb40c88af9c92e75':
Fixed warnings in the VOL example and run it from the script.
Split H5VLnative.h into public and private files and updated
* commit '5f99891e6dc59524fb6e43ca96a86df22b403c61':
Move options to root cmake file
EED-258 Add modules uses toolchain files
Updates to the VOL ID and object API calls.
This brings the H5VL code in line with the H5I code regarding
naming, parameter order, etc.
Several public API calls were affected by this change.
These changed names to reflect their use with VOL drivers:
H5VLregister() --> H5VLregister_driver()
H5VLregister_by_name() --> H5VLregister_driver_by_name()
H5VLunregister() --> H5VLunregister_driver()
H5VLis_registered() --> H5VLis_driver_registered()
This call was renamed to match H5Iregister(). The order of the first
two parameters also reversed and the object pointer is now const.
H5VLobject_register() --> H5VLregister()
* commit '1e8ef703cdc057211b3587be776a6e9f62e8f76f':
Fixed a C++ style commenting issue and removed an unused field from H5F_trav_obj_ids_t.
Fixed a memory issue in H5Drefresh() where the dataset's H5VL_object_t gets closed but we try to use it to find the driver.
* commit 'c3a813a37fb700d2d1039d6fdc54d8e377412b76':
Fixed a memory issue in H5Drefresh() where the dataset's H5VL_object_t gets closed but we try to use it to find the driver.
* commit '75b84b09bd249b36a063fd51ecd3383801b0118b':
Updated the H5I debug dump to not segfault when types are not initialized or contain zero IDs in the skip list.
The H5I debug ID dump function is now always available instead of hidden behind an H5_DEBUG_OUTPUT ifdef.
* commit '2a95ab5acbe032f58891b653f56b9db8a4938d95':
TRILABS-21 add toolchain related info and files
TRILABS-21 Add intel compiler support
Fix undef for Xl compilers
TRILAB-21 Intel flags
1) Free space manager for the metadata file
2) Delayed free space release linked list
3) H5F_update_vfd_swmr_metadata_file()
3) VFD SWMR driver: read callback
4) Flushing for VFD SWMR
5) Port one concurrent test from swmr test set
6) Bug fixes and refactoring
* commit 'd26bb4d26ce5a23804a80c16dcb53e5f3eb957e4':
HDFFV-10594 - mac must be space separated
HDFFV-10594 - Add rpath configuration note
Add extra path option for build location
Use relative RPATH paths
* commit '2618dc314b9e4875e6ece7ee8696846ac2d02183':
Remove extra unneeded
Remove libsettings from pregenerated block
CMake changes for SKIPPED and flush tests
Correct $withval usage in configure
HDFFV-10332 Adjust checks and generation of files
HDFFV-10332 Use pre-generated files
* commit '96d5da7f1ad681573d42d2a4fd9e422fe049442d':
Remove extra unneeded
Remove libsettings from pregenerated block
Correct $withval usage in configure
HDFFV-10332 Adjust checks and generation of files
HDFFV-10332 Use pre-generated files
* commit 'e96bd2d46e208e0eb6c62cfcc4f86141f933884e':
CMake changes for SKIPPED and flush tests
H5S adjust callbacks now return an herr_t instead of void.
Normalization with vol_integration (property lists, file drivers, other misc).
* commit 'bbafe0debae9886988e4b7ec31bb8ef7292e43f9':
Remove extra constant
Fix typo
Add Constants to all ERR STK checks
Use constants for test comparisons
Develop normalization with vol_integration. Mostly peripheral things like the tools and wrappers, with just enough core library code to support that.
* commit '59f36c5cdef1899742feb1c06ee09886e2d0a220':
Remove extra constant
Fix typo
Add Constants to all ERR STK checks
Use constants for test comparisons
* commit 'eb78fd88325d58a95c026129f1e2b4c7c50f621a':
Develop normalization with vol_integration. Mostly peripheral things like the tools and wrappers, with just enough core library code to support that.
* commit 'ec53237fb849042b4a5b276a3e66a98ebb3710ee':
Removed 'VOL' versions of FUNC_ENTER macros as well as empty functions orginally intended to support the full SWMR feature.
* commit '680cf4d946fd10d50e0f7acdc5bc93f91037fb4f':
Removed 'VOL' versions of FUNC_ENTER macros as well as empty functions orginally intended to support the full SWMR feature.
* commit '6c78869c19a88edfd7b6a2a99a8fb8f5febf0b7a':
Fix whitespace and exit status
HDFFV-9059 take file as argument
HDFFV-9059 Allow detect to take a file as an argument
1) Define driver for the VFD SWMR reader
2) Implement VFD SWMR open callback
3) Implement H5FD_vfd_swmr_get_tick_and_idx()
4) Load and decode metadata file header and index
4) Closing for VFD SWMR
* commit 'c834d9f99d45e5b9752e8525fe8761ea5592bf2c': (41 commits)
HDFFV-10568 fix hdf5_java library dependency
Remove another extra path var
Move muti-config dir setting to root process
Fix one more intermediate location
Use different variable
Cleanup and add intermediate dir for java
Java must use shared libs to allow dlopen calls
Correct names
Call new function
Correct default API version for develop to 112.
Fix typo
HD prefix and whitespace
Update RELEASE.txt with suggested changes
Update MANIFEST file for new t_coll_md_read.c file
Remove now-unused local variable
Add fix for HDFFV-10501
Revert testfile FILE change
change FILE path
Add testfiles to data copy
Same changes needed for examples as test
...
* commit 'fddbbc158d170b4e3a7ffd663f9799fd7e01a6a0':
HDFFV-10568 fix hdf5_java library dependency
Remove another extra path var
Move muti-config dir setting to root process
Fix one more intermediate location
Use different variable
Cleanup and add intermediate dir for java
* commit '763b95c6b0f027025a5a5bfb203b77317ffa2368':
Update RELEASE.txt with suggested changes
Update MANIFEST file for new t_coll_md_read.c file
Remove now-unused local variable
Add fix for HDFFV-10501
* commit 'a02414cc24e01096a239fc774ce94c38655f53ce':
Revert testfile FILE change
change FILE path
Add testfiles to data copy
Same changes needed for examples as test
Correct name of test jar
Only remove test jars
Remove obsolete var
Correct name of osx installer
DT location incorrect
* commit 'd20fd310e42b73e9037fb1a3bcc5e55d5e5c5162':
Change prefix in example scripts to relative path to bin. This was done for 1.8, and works wherever installed without the need to replace the original prefix.
1) Public routines: H5Pget/set_vfd_swmr_config
2) Public routine: H5Fvfd_swmr_end_tick
3) Initialization when opening a file with VFD SWMR writer
4) Tests in test/vfd_swmr.c for VFD SWMR
5) Fix a bug in src/H5Fint.c: when error is encountered after the root group is created
Note the following:
--This is WORK IN PROGRESS and will subject to change as implementation goes.
--There is test failure form enc_dec_plist.c: I will fix this when changes to the property list are settled.
--The branch is updated with the latest from develop as of 8/14/2018
* commit '5647dea421be9dc8429f08632aa72a8a22904292': (47 commits)
Rearrange issues by date order
RELEASE.txt changes for MPI updates
Update Drop Site options and Coverage settings
Reorder bugfix release notes from latest to earliest, and miscellaneous format cleanup.
Add RELEASE.txt entry for HDFFV-10475
HDFFV-10544 Add more descriptive text
HDFFV-10544 Correct var name
HDFFV-10544 remove native from class function
HDFFV-10544 correct typo
HDFFV-10544 add release note
HDFFV-10544 add class name to error text
HDFFV-10544 exception variable as local class
Improve error handling of exceptions
Typo fix
Set CMAKE_REQUIRED_INCLUDES instead of using path in call
Add Autotools and CMake checks for big I/O MPI-3 functions
Add hdf5settings section for parallel compression status in CMake builds
HDFFV-10508 rework sentence
HDFFV-10508 clarify library differences
HDFFV-10508 spelling and grammer
...
* commit 'a3385675b3cdc2102457a9f91877cc27778ab29a':
Reorder bugfix release notes from latest to earliest, and miscellaneous format cleanup.
Add RELEASE.txt entry for HDFFV-10475
* commit '6e4c036d5dc476396428f4b044e23043a80e8df7':
HDFFV-10544 Correct var name
HDFFV-10544 remove native from class function
HDFFV-10544 correct typo
HDFFV-10544 add release note
HDFFV-10544 add class name to error text
HDFFV-10544 exception variable as local class
Improve error handling of exceptions
* commit '9aa2eaeb9ed9c94f63973cb55f13de256558497c':
Add Autotools and CMake checks for big I/O MPI-3 functions
Add hdf5settings section for parallel compression status in CMake builds
Switch to CheckSymbolExists in CMake
Add configure check for MPI_Mprobe and MPI_Imrecv functions
* commit '272eb4b8bffd2ab732d268c421583587317b8dc2':
RELEASE.txt updates for HDFFV-10467 and HDFFV-10509
Update HDF5 examples to correct version.
Correct a few typos.
Correct typo in comment in config/gnu-flags.
Update bine/release to create batch scripts and build-unix-sh, and to put files in a subdirectory.
Add line to libhdf5settings file for status of Parallel writes to
filtered datasets status
Surround Parallel Compression code in MPI_VERSION >= 3 checks
Add disabled message for Parallel Compression built w/ MPI-2
Modify Parallel Compression tests to only run the parallel filtered read
tests when parallel filtered writes are disabled
Update big I/O code to handle being built with MPI-2
Add checks to CMakeLists.txt for MPI_Mprobe and MPI_Imrecv
message(FATAL_ERROR"Set to use libhdfs library, but could not find or use libhdfs. Please verify that the path to HADOOP_HOME is valid, and/or reconfigure without HDF5_ENABLE_HDFS")
// Class LinkAccPropList represents the HDF5 file access property list and
// inherits from DataType.
#ifndef __H5LinkAccPropList_H
#define __H5LinkAccPropList_H
@@ -22,15 +19,15 @@ namespace H5 {
/*! \class LinkAccPropList
\brief Class LinkAccPropList inherits from PropList and provides
wrappers for the HDF5 file access property list.
wrappers for the HDF5 link access property list.
*/
// Inheritance: PropList -> IdComponent
classH5_DLLCPPLinkAccPropList:publicPropList{
public:
///\brief Default file access property list.
///\brief Default link access property list.
staticconstLinkAccPropList&DEFAULT;
// Creates a file access property list.
// Creates a link access property list.
LinkAccPropList();
///\brief Returns this class name.
@@ -39,7 +36,7 @@ class H5_DLLCPP LinkAccPropList : public PropList {
// Copy constructor: same as the original LinkAccPropList.
LinkAccPropList(constLinkAccPropList&original);
// Creates a copy of an existing file access property list
// Creates a copy of an existing link access property list
// using the property list id.
LinkAccPropList(consthid_tplist_id);
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
