Flush the core VFD's buffer before closing the file, also flush the
metadata accumulator before reseting it. Write the driver info message out
in the superblock flush routine more directly, instead of using wrapper
routine, since the wrapper routine won't work when the superblock is being
shutdown.
Tested on:
FreeBSD/32 6.3 (duty) in debug mode, w/check-vfd
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode, w/check-vfd
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode, w/check-vfd
Linux/64-amd64 2.6 (smirom) w/Intel compilers w/default API=1.6.x,
w/C++ & FORTRAN, in production mode, w/check-vfd
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode, w/check-vfd
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode, w/check-vfd
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode, w/check-vfd
Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode, w/check-vfd
Mac OS X/32 10.5.8 (amazon) in debug mode, w/check-vfd
Mac OS X/32 10.5.8 (amazon) w/C++ & FORTRAN, w/threadsafe,
in production mode, w/check-vfd
Changed max macro in earray.c to avoid Windows conflict.
Changed if(test) to assert(test) in H5FDcore.c - H5FD_core_open()
Tested on:
Windows: XP32, XP64, Vista32, Vista64
H5committest.
to the beginning of the file. Otherwise, the file might be re-extended later on Open VMS.
Also updated the return value for the HDlseek to be more appropriate.
Tested 1.8 on Open VMS.
Various bug fixes & cleanups with VFDs and file free space changes,
along with cleaning up compiler warnings.
Enable more VFDs (for 'make check-vfd' testing) for many tests.
Tested on:
Mac OS X/32 10.5.5 (amazon) in debug mode
Mac OS X/32 10.5.5 (amazon) w/C++ & FORTRAN, w/threadsafe,
in production mode
FreeBSD/32 6.3 (duty) in debug mode
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/Intel compilers w/default API=1.6.x,
w/C++ & FORTRAN, in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode
Fix various problems with a the core & sec2 VFDs.
Improve the h5_get_file_size() routine to handle files created with
VFDs that use multiple files.
Tested on:
Mac OS X/32 10.5.5 (amazon) in debug mode
Mac OS X/32 10.5.5 (amazon) w/C++ & FORTRAN, w/threadsafe,
in production mode
FreeBSD/32 6.3 (duty) in debug mode
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/Intel compilers w/default API=1.6.x,
w/C++ & FORTRAN, in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode
Bring file free space branch changes through r15795 into trunk, which
includes a fair bit of code cleanup & rearrangement along with a couple of
bug fixes also.
Tested on:
Mac OS X/32 10.5.5 (amazon) in debug mode
Mac OS X/32 10.5.5 (amazon) w/C++ & FORTRAN, w/threadsafe,
in production mode
FreeBSD/32 6.3 (duty) in debug mode
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/Intel compilers w/default API=1.6.x,
w/C++ & FORTRAN, in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode
Remove trailing whitespace from C/C++ source files, with the following
script:
foreach f (*.[ch] *.cpp)
sed 's/[[:blank:]]*$//' $f > sed.out && mv sed.out $f
end
Tested on:
Mac OS X/32 10.5.5 (amazon)
No need for h5committest, just whitespace changes...
> 1.
Description: Added a new field 'app_count' to H5I_id_info_t struct, to track
the reference count on an id due to the application. the old 'count' field
tracks the total. Generally any id visible to the application gets placed
in app_count. Added app_ref boolean parameter to H5I_inc_ref, H5I_dec_ref,
H5I_register, H5I_clear_type, and a few other functions, to specify whether
the operation(s) being performed on the id(s) are due to the application
(TRUE) or not (FALSE). Test added for this case.
Tested: kagiso, smirom, linew (h5committest)
Change back to clearing memory buffers by default, but give users option
to disable this, for performance reasons (--disable-clear-file-buffers).
Also, add configure option (--enable-using-memchecker) to conveniently
control making the library "play nice" with memory checking tools like valgrind
and purify.
Tested on:
Mac OS X/32 10.4.9 (amazon)
FreeBSD/32 6.2 (duty)
Linux/32 2.6 (kagiso)
Tested platform:
Kagiso only since it is only a comment block change. If it works in one
machine, it should work in all, I hope. Still need to check the parallel
build on copper.
Finish new version of the I/O pipeline message, which is much smaller than
the previous version. This version is used with the "use the latest version
of the format" flag.
Closed several memory leaks/overruns (found with valgrind).
Also, lots of compiler & formatting cleanups.
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
New feature
Description:
Check in baseline for compact group revisions, which radically revises the
source code for managing groups and object headers.
WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!!
WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!!
This initiates the "unstable" phase of the 1.7.x branch, leading up
to the 1.8.0 release. Please test this code, but do _NOT_ keep files created
with it - the format will change again before the release and you will not
be able to read your old files!!!
WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!!
WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!!
Solution:
There's too many changes to really describe them all, but some of them
include:
- Stop abusing the H5G_entry_t structure and split it into two separate
structures for non-symbol table node use within the library: H5O_loc_t
for object locations in a file and H5G_name_t to store the path to
an opened object. H5G_entry_t is now only used for storing symbol
table entries on disk.
- Retire H5G_namei() in favor of a more general mechanism for traversing
group paths and issuing callbacks on objects located. This gets us out
of the business of hacking H5G_namei() for new features, generally.
- Revised H5O* routines to take a H5O_loc_t instead of H5G_entry_t
- Lots more...
Platforms tested:
h5committested and maybe another dozen configurations.... :-)
Code cleanup
Description:
Trim trailing whitespace, which is making 'diff'ing the two branches
difficult.
Solution:
Ran this script in each directory:
foreach f (*.[ch] *.cpp)
sed 's/[[:blank:]]*$//' $f > sed.out && mv sed.out $f
end
Platforms tested:
FreeBSD 4.11 (sleipnir)
Too minor to require h5committest
Code cleanup
Description:
Clean up various warnings reported by the Windows team.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Too minor to require h5committest
Description: Removed PABLO from the source
Solution:
Platforms tested: arabica with 64-bit, copper with parallel,
heping with GNU C and C++ and PGI fortran (but
I disabled hl, there is some weird problem only
on heping: F9XMODFLAG is not
propagated to the Makefile files
Misc. update:
Bug fix
Description:
Another attempt to fix the address overflow in the core VFL, hopefully one
that works on 64-bit platforms.
Platforms tested:
AIX 5.1 (copper)
Bug fix.
Description:
Fix off-by-one error in Core VFL driver which would cause spurious address
or size overflow errors when an odd valued address or size was checked.
Platforms tested:
FreeBSD 4.10 (sleipnir)
Linux 2.4 (heping)
Solaris 2.7 (arabica)
Code cleanup
Description:
Various minor tweaks to clean code up and bring it into closer
syncronization with the release branch.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel
h5committested
IRIX64 6.5 (modi4)
Purpose:
Bug Fix
Description:
If an HDF5 file grows larger than its address space, it dies and is unable to
write any data. This is more likely to happen since users are able to change
the number of bytes used to store addresses in the file.
Solution:
HDF5 now throws an error instead of dying. In addition, it "reserves" address
space for the local heap and for object headers (which do not allocate space
immediately). This ensures that after the error occurs, there is enough address
space left to flush the entire file to disk, so no data is lost.
A more complete explanation is at /doc/html/TechNotes/ReservedFileSpace.html
Platforms tested:
sleipnir, copper (parallel), verbena, arabica, Windows (Visual Studio 7)
Solution:
Platforms tested:
Misc. update:
Code cleanup & minor optimization
Description:
Re-work the way interface initialization routines are specified in the
library to avoid the overhead of checking for them in routines where there is
no interface initialization routine. This cleans up warnings with gcc 3.4,
reduces the library binary size a bit (about 2-3%) and should speedup the
library's execution slightly.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/gcc34
h5committest
Code cleanup
Description:
Add destructor to match constructor fr VFLs when they are shut down by the
library.
Solution:
Added H5FD_*_term() routines to "undo" changes made in H5FD_*_init()
routines.
Platforms tested:
IBM p690 (copper)
too minor to require h5committest
Bug fix/optimization
Description:
Address slowdown in MPI-I/O file metadata operations that was introduced
mid-stream. We now _require_ a POSIX compliant parallel file system for the
MPI-I/O file driver (as well as for the MPI-POSIX file driver).
Also optimized file open operation when the file is being created by
reducing the number of collective & syncronizing calls.
Additionally, refactor the MPI routines into a common place, eliminating
duplicated code.
Platforms tested:
FreeBSD 4.9 (sleipnir) w/parallel
h5committest
Code cleanup & bug fix
Description:
More de-linting...
Also correct a bug that lint exposed which was incrementing the incorrect
driver ID of a file driver and also no saving the correct driver ID for the
newly opened file.
Platforms tested:
FreeBSD 4.8 (sleipnir)
too minor for h5committest
New Feature
Description:
Added support for the lock and unlock function calls for file
drivers. Only FPHDF5 uses this feature. All of these drivers don't
define lock or unlock methods.
Platforms tested:
Linux, Modi4, Sol
Misc. update:
Bug Fix
Description:
Metadata cache in parallel I/O can cause hangs in applications which
perform independent I/O on chunked datasets, because the metadata cache
can attempt to flush out dirty metadata from only a single process, instead
of collectively from all processes.
Solution:
Pass a dataset transfer property list down from every API function which
could possibly trigger metadata I/O.
Then, split the metadata cache into two sets of entries to allow dirty
metadata to be set aside when a hash table collision occurs during
independent I/O.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir) serial & parallel
Misc. update:
Updated release_docs/RELEASE
Bug Fix
Description:
When calling H5Fopen with the core VFL driver, but without the
H5F_ACC_CREAT flag goes ahead and creates a memory file.
Solution:
Check for the H5F_ACC_CREAT flag before allowing the memory file to be
created.
Platforms tested:
FreeBSD 4.7 (sleipnir)
Code cleanup/new feature.
Description:
Split FUNC_LEAVE into API and non-API specific versions. This allows a
solution to compiling this branch with C++, as well as reducing the size
of the binaries produced.
Platforms tested:
FreeBSD 4.7 (sleipnir) w/serial, parallel (including MPE) & thread-safe
Purpose:
a bug fix
Description:
When a file is opened for a second time, file close degree is supposed to
be either the same as the first open, or default as the first open is also
default.
Platforms tested:
Linux 2.2(eirene), Solaris 2.7(arabica), IRIX64 6.5(modi4)
Purpose:
New API functions
Description:
Added API functions to return pointer to low-level file handle
(H5Fget_vfd_handle and H5FDget_vfd_handle) and related property list
setting functions(H5Pset_family_offset and H5Pset_multi_type).
Platforms tested:
Linux 2.2(eirene), Solaris 2.7(arabica), IRIX64 6.5(modi4)
Code cleanup
Description:
Combined H5P_isa_class and H5I_object functionality into a new internal
H5P API function: H5P_object_verify, which checks that a property list is
the appropriate class and then returns the property list object associated
with the property list ID.
This reduces the source code by about 200 LOC and trims the library binary
some more.
Platforms tested:
FreeBSD 4.6 (sleipnir)
Code cleanup
Description:
Changed the last HRETURN* statements in the FUNC_ENTER macros into HGOTO*
macros, which reduces the size of the library binary in certain
configurations by another 10%
Platforms tested:
FreeBSD 4.6 (sleipnir) serial & parallel, IRIX64 6.5 (modi4) serial &
parallel
Code cleanup
Description:
Change most (all?) HRETURN_ERROR macros to HGOTO_ERROR macros, along with
HRETURN macros to HGOTO_DONE macros. This unifies the error return path
from functions and reduces the size of the library by up to 10% on some
platforms.
Additionally, I improved a lot of the error cleanup code in many routines.
Platforms tested:
FreeBSD 4.6 (sleipnir) serial & parallel and IRIX64 6.5 (modi4) serial &
parallel.
Code cleanup
Description:
Broke the FUNC_ENTER macro into several macros, with more specialized
uses (which followup mail will describe). This was designed to move
most/all of the checks which could be done at compile time to that point,
instead of needlessly performing them (over & over :-) at run-time.
This reduces the library's size (and thus staticly linked binaries) and
has a minor speedup effect also.
Platforms tested:
IRIX64 6.5 (modi4) with parallel & FORTRAN enabled, and additional testing
on FreeBSD and Solaris immediately after the checkin.
Code cleanup.
Description:
Took Robb's recent ideas for improving the FUNC_ENTER/FUNC_LEAVE macros
equivalents in the SAF library and adapted them to our library. I added
an additional macro which is equivalent to FUNC_ENTER:
FUNC_ENTER_NOINIT - Has the API tracing code, etc. from FUNC_ENTER but
none of the library or interface initialization code. This is to
be used _only_ for static functions and those which explicitly
cannot have the library or interface initialization code enabled
(like the API termination routines, etc.).
This allowed many more of the functions in the library [but not all yet :-(]
to be wrapped with FUNC_ENTER[_NOINIT]/FUNC_LEAVE pairs.
It also reduced the size of the library and executables (by cutting out a
bunch of code which was never executed), I'll e-mail the exact results when
I've finished editing it.
Platforms tested:
IRIX64 6.5 (modi4)
New Feature
Description:
The VFL flush function is called immediately before a file is closed.
This can cause duplicate syncronization actions to occur, if the VFL
close function also performs them.
Solution:
Added 'closing' parameter to VFL 'flush' operation. This allows the VFL
flush function to bypass operations that will be duplicated within the VFL
close function.
Additionally, use the 'closing' parameter to bypass calls to MPI_File_sync()
when set. Since MPI_File_close() also syncronizes the file, this avoids
the terrible performance hit taken when calling MPI_File_sync() as the file
is closing.
Platforms tested:
IRIX64 6.5 (modi4)
Purpose:
Modify H5Fclose behavior
Description:
The HDF5 actual file close behaves in several ways in terms of if there
are still objects(dataset, group, datatype) opened in file.
Solution:
Added a new file access property, file close degree. It has four values,
H5F_CLOSE_DEFAULT
H5F_CLOSE_WEAK
H5F_CLOSE_SEMI
H5F_CLOSE_STRONG
The way a file is closed is decided by these values.
Platforms tested:
IRIX64 6.5, SunOS 5.6, FreeBSD 4.4
Code cleanup
Description:
Windows is generating hundreds of warnings from some of the practices in
the library. Mostly, they are because size_t is 32-bit and hsize_t is
64-bit on Windows and we were carelessly casting the larger values down to
the smaller ones without checking for overflow.
Also, some other small code cleanups,etc.
Solution:
Re-worked some algorithms to eliminate the casts and also added more
overflow checking for assignments and function parameters which needed
casts.
Kent did most of the work, I just went over his changes and fit them into
the the library code a bit better.
Platforms tested:
FreeBSD 4.4 (hawkwind)
Code cleanup
Description:
Get rid of IDs from internal function calls and some small cleanups from
the old-stype => generic property list conversion.
Platforms tested:
FreeBSD 4.4 (hawkwind)
