Merge changes that correspond to the 64-bit ID changes (without the actual
switch to 64-bit IDs) to the 1.8 release branch. (Plus a few minor cleanups
and alignments with the trunk that aren't on the branch)
Tested on:
Mac OSX/64 10.9.4 (amazon) w/C++ & FORTRAN
(h5committested on branch already for a week)
Bring r23529 from trunk to 1.8 branch:
Stop aliasing property to indicate internal collective metadata operations
with property to perform collective raw data operations from the application.
Tested on:
Mac OSX/64 10.8.3 (amazon) w/parallel
(daily tests on trunk)
t_cache.c - define needed for windows
t_posix_compliant.c - #ifdef around include <unistd.h>
t_shapesame.c - HD prefix needed for stat function
Tested: local linux
error and wanted to exit the test program. This was not good since if only a
subset of processes called MPI_Finalize(), the other processes will likely
hang. That happened in AIX that it would waited till the alarm signal to kill
the processes. Definitely a waste of time.
Solution: Changed it to call MPI_Abort.
That showed another problem. HDF5 has setup atexit post-process to try to close
unclose objects, release resources, etc. But if the MPI processes have
encountered an error and has been aborted, it is not likely any more MPI calls
can function properly. E.g., it would attempt to free some communicators in
the HDF5 MPIO file handle. It would again hang.
Solution: need to call H5dont_atexit() to disable any atexit post-processing.
This must be done early, like before calling H5open. This is added to each
parallel test main program.
testphdf5.h:
Changed macros VRFY and MESG. Added comments too.
testphdf5.c:
t_mpi.c:
t_cache.c:
t_shapesame.c:
Added H5dont_atexit.
Tested: h5committest.
Initialize loop variable that caused failures in certain circumstances.
Also clean up compiler warnings and release MPI datatype.
Tested on:
Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in debug mode
Rename H5AC_set() to H5AC_insert_entry()
Get rid of H5C_set_skip_flags() & related flags
Tested on:
Mac OS X/32 10.6.4 (amazon) w/debug
(too simple to require h5committest)
Bring r18917 from trunk to 1.8 branch:
Bring r18911 (plus some adaptions to match the code on the trunk)
from the metadata journaling "merging" branch to the trunk:
More general changes to align trunk with eventual changes from
metadata journaling branch.
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (jam) w/PGI compilers, w/default API=1.8.x,
w/C++ & FORTRAN, w/threadsafe, in debug mode
Linux/64-amd64 2.6 (amani) w/Intel compilers, w/default API=1.6.x,
w/C++ & FORTRAN, in production mode
Bring r18864 from trunk to 1.8 branch:
General cleanups to align metadata journaling branch and trunk
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
(h5committested on trunk)
Bring r18768 from trunk to 1.8 branch:
Correct a mis-merged patch when marking an entry dirty.
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode
(h5committested on trunk)
Bring r18740 from trunk to 1.8 branch:
Bring r18738 from metadata journaling 'merging' branch to trunk:
Switch H5AC_resize_pinned_entry() to work on protected entries as well,
and rename to H5AC_resize_entry()
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
(h5committested on trunk)
Bring r18725 from trunk to 1.8 branch:
Bring r18724 from metadata journaling 'merging' branch to trunk:
Rename H5[A]C_rename to H5[A]C_move_entry.
Tested on:
FreeBSD/32 6.3 (duty) w/debug
(h5committested on trunk)
Bring r18721 from trunk to 1.8 branch:
Bring r18720 from metadata journaling merging branch to trunk:
Bring changes from metadata journaling branch to 'merging' branch:
Rename H5[A]C_mark_pinned_or_protected_entry_dirty() to
H5[A]C_mark_entry_dirty() and get rid of H5[A]C_mark_pinned_entry_dirty().
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
(h5committested on trunk)
Bring r18702 from trunk to 1.8 branch:
Bring r18672 from metadata journaling "merging" branch to trunk:
Mostly changes to move to only using one 'user data' parameter for
calls to H5AC_protect(), along with some minor reformatting code cleanups.
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (jam) w/PGI compilers, w/default API=1.8.x,
w/C++ & FORTRAN, w/threadsafe, in debug mode
Linux/64-amd64 2.6 (amani) w/Intel compilers, w/default API=1.6.x,
w/C++ & FORTRAN, in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode
Mac OS X/32 10.6.3 (amazon) in debug mode
Mac OS X/32 10.6.3 (amazon) w/C++ & FORTRAN, w/threadsafe,
in production mode
Bring r18171 from trunk to 1.8 branch (with appropriate tweaks):
Bring r18169 from 'merge_metadata_journaling' branch back to trunk:
Converge changes on metadata_journaling branch and trunk:
- Remove H5F_t* parameter from H5HL_unprotect()
- Remove H5F_t* parameter from H5HL_dirty()
- Remove H5F_t* parameter from H5O_unprotect()
- Bring changes to metadata caching routines back:
- H5AC_pin_protected_entry()
- H5AC_resize_pinned_entry()
- H5AC_unpin_entry()
- H5AC_mark_pinned_entry_dirty()
- H5AC_mark_pinned_or_protected_entry_dirty()
- Revise internal H5C routines to drop H5F_t* and/or H5C_t* as
parameter where possible
- Revise tests to follow internal routine changes
Also, other minor changes to speedup the 'cache' test.
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (jam) w/PGI compilers, w/default API=1.8.x,
w/C++ & FORTRAN, w/threadsafe, in debug mode
Linux/64-amd64 2.6 (amani) w/Intel compilers, w/default API=1.6.x,
w/C++ & FORTRAN, in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in debug mode
Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode
Mac OS X/32 10.6.2 (amazon) in debug mode
Mac OS X/32 10.6.2 (amazon) w/C++ & FORTRAN, w/threadsafe,
Bring r18031 from trunk to 1.8 branch:
Bring r18030 from merge_metadata_journaling branch to trunk:
Bring "brush clearing" changes (whitespace & style issues, mostly) from
metadata_journaling branch to the "merging" branch, to converge the trunk and
the metadata_journaling branch.
Also, some other minor cleanups along the way.
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
(h5committested on trunk)
Bring r17408, 17411, 17412, 17413 & 17414 from trunk back to 1.8 branch:
r17408:
Move flush operation on mounted file hierarchy into H5Fmount module.
r17411:
Make H5AC_flush just flush the cache and make H5AC_dest perform the
proper parallel synchronization before destroying the cache. Also, further
discriminate between 'closing' and 'non-closing' actions in H5F_flush.
r17412:
Seperate 'flush' functionality from 'destroy' functionality at the
H5F level also.
r17413:
Remove vestigial intermediate routine for flushing file and move it
into API routine. Also, remove private, unused (now) 'H5F_FLUSH_DOWN' symbol
from public header file.
r17414:
Flush the core VFD's buffer before closing the file, also flush the
metadata accumulator before reseting it. Write the driver info message out
in the superblock flush routine more directly, instead of using wrapper
routine, since the wrapper routine won't work when the superblock is being
shutdown.
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
(h5committest performed on trunk)
Bring r17365 from trunk to 1.8 branch:
Final merge of changes from sblock_mdc branch back to trunk. The
superblock is now managed by the metadata cache.
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
(h5committest performed on trunk)
The failure was caused by some over active sanity checking code in
unlock_entry(). In essence the code did not consider the possibility
that under certain, very unusual circumstances, an entry could be flushed
to disk during the H5AC_unprotect() call. Instead, it simply failed
if a dirty entry was marked clean after the call to H5AC_unprotect().
This bug in the test code was exposed by recent changes to the default
cache configuration made as part of the "metadata blizard" bug fix.
Fixed the bug by adding code to detect when an entry is flushed during
the call to H5AC_unprotect(), and not trigger a failure if a dirty entry
is marked clean after a call to H5AC_unprotect() if the entry has been
flushed.
In passing also found and fixed another test bug in which expunged
entries were erroneously marked as dirty in the test code's independant
register of entry status.
Tested parallel on Phoenix (AMD64 Linux) and Jam.
No further testing, as the fix was tested extensively prior to checkin
on the 1.9 branch.
Bring r15800 back from trunk:
- File free space branch changes through r15794
Tested on:
Mac OS X/32 10.5.5 (amazon) in debug mode
Mac OS X/32 10.5.5 (amazon) w/C++ & FORTRAN, w/threadsafe,
in production mode
FreeBSD/32 6.3 (duty) in debug mode
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/Intel compilers w/default API=1.6.x,
w/C++ & FORTRAN, in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode
Remove trailing whitespace from C/C++ source files, with the following
script:
foreach f (*.[ch] *.cpp)
sed 's/[[:blank:]]*$//' $f > sed.out && mv sed.out $f
end
Tested on:
Mac OS X/32 10.5.5 (amazon)
No need for h5committest, just whitespace changes...
large entry, or to a large increases in the size of an existing entry.
This required some additions to the cache configuration structure, and
thus will require changes in the metadata cache documentation.
The basic idea is to monitor the size of entries as they are loaded,
inserted, or increased in size. If the size of the entry (or increase)
exceeds some user selected fraction of the size of the cache, increase
the size of the cache.
Note that this fix was designed quickly -- while it deals with the
use case that exposed the problem, we may have to revisit the issue
later.
Tested serial and parallel on Phoenix, and h5committest.
Tested platform:
Kagiso only since it is only a comment block change. If it works in one
machine, it should work in all, I hope. Still need to check the parallel
build on copper.
1) Added trace file support to the metadata cache. This allows capture
of all metadata cache calls in trace files for purposes of optimization
and debuging.
2) Added an expunge entry function. This allows an entry to be deleteded
from the cache without writing it to disk even if it is dirty.
3) Added a function call to resize pinned entries.
4) Added code to deal with entries that are dirty on load. This is
needed in support of a bug fix which can alter object headers on
load to repair files.
5) Added progress reporting code to the "MDC API smoke check" test in
cache_api.c. To enable the progress reporting, set report_progress
to TRUE in mdc_api_call_smoke_check().
Tested with h5committest, and a parallel test on phoenix (dual athelon
linux box).
Code cleanup
Description:
Trim trailing whitespace in Makefile.am and C/C++ source files to make
diffing changes easier.
Platforms tested:
None necessary, whitespace only change
1) Check in potential fix to unreproduceable bug in t_cache observed on
Cobalt
2) Check in code supporting added pinned entry features in the metadata
cache.
Description:
1) Elena encountered a bug in t_cache when running on Cobalt. On
investigation I found a potential race condition in t_cache that
could explain her observation.
2) Quincey requested additions to the metadata cache allowing renaming
of pinned entryies, and marking of pinned or protected entries as
dirty.
Solution:
1) Modified t_cache.c to prevent the potential race condition. Elena
was unable to reproduce the bug on Cobalt, so we don't know if my
fix actually addressed the problem.
2) Added code supporting the requested functions. The changes were
relatively trivial, but required substantial new test code.
Platforms tested:
h5committest, serial test on heping, parallel test on phoenix.
Misc. update:
Bug fix
Description:
Fix two test bugs:
1) Unitialized variable bug in test of H5C_get_entry_status()
(in test/cache.c)
2) Incorrect initialization of constant in t_cache when running
with MPE. (in testpar/t_cache.c)
Solution:
1) Don't test uninitialize variable.
2) Correct initialzation of constant in MPE case.
Platforms tested:
Tested on Heping and Copper.
Misc. update:
Code cleanup/feature twist
Description:
Adjust recent H5AC routines to take H5F_t pointers instead of H5C_t
pointers, to match the rest of the H5AC routines.
This change propagated into a few of the tests, which also had some
compiler warnings cleaned up...
Platforms tested:
FreeBSD 4.11 (sleipnir) w/parallel
Linux 2.4/64 (mir) w/C++ & FORTRAN
Add pinned entry capability to cache.
Description:
For frequently accessed cache entries, the protect/unprotect overhead
is sometimes a bottleneck.
Solution:
Allow entries to be pinned in the cache. Pinned entries can't be
evicted, but can be flushed or modified.
Platforms tested:
h5committested -- minus one small typo in test/cache.c whose fix was
tested on copper and heping only.
Misc. update:
bug fix.
Description:
The MPE_Stop_log did not work in copper. It spewed out MPE errors
and ended in infinite looping.
Solution:
Changed to a smaller test size to avoid generating huge MPE log files
when MPE is configured in.
Platforms tested:
Copper (mpe)
Misc. update:
Bug fix.
Description:
When MPE is used, the test generates huge Clog files in /tmp that
could fill up the disk (like in Copper.)
Solution:
Turned off MPE logging.
Platforms tested:
heping (pp) and copper(pp)
Misc. update:
Attempt to ensure that the parallel cache test runs at a reasonable
speed with large numbers of processors.
Description:
In some cases, the number of random locks and unlocks was a multiple of
the MPI rank.
Solution:
Use rank % 4 instead of simply rank.
Platforms tested:
copper
Misc. update:
Add a file that I forgot in my last checkin.
Description:
t_cache.c is the source file for the new parallel metadata cache test.
Solution:
See above.
Platforms tested:
h5committested before the last checkin. Will run another
h5committest shortly, but it shouldn't be necessary as there
are no new changes.
Misc. update:
