Bug fix
Description:
Rewrite code for mounting files to clean up layers of kludges and implement
a much cleaner and more maintainable design.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4
Feature
Description:
Added "support" for UTF-8 character encoding.
Solution:
Wrote tests to check that UTF-8 can be used in a number of places in
HDF5 (object names, data, etc.). These tests live in test/tunicode.c.
Added a new UTF-8 character encoding for datatypes.
Platforms tested:
mir, modi4, heping
Misc. update:
Code cleanup
Description:
Clean up a few warnings flagged by the Intel C compiler.
Platforms tested:
FreeBSD 4.11 (sleipnir) w/Intel C
Too minor to require h5committest
Description: Removed PABLO from the source
Solution:
Platforms tested: arabica with 64-bit, copper with parallel,
heping with GNU C and C++ and PGI fortran (but
I disabled hl, there is some weird problem only
on heping: F9XMODFLAG is not
propagated to the Makefile files
Misc. update:
Bug Fix/Code Cleanup/Doc Cleanup/Optimization/Branch Sync :-)
Description:
Generally speaking, this is the "signed->unsigned" change to selections.
However, in the process of merging code back, things got stickier and stickier
until I ended up doing a big "sync the two branches up" operation. So... I
brought back all the "infrastructure" fixes from the development branch to the
release branch (which I think were actually making some improvement in
performance) as well as fixed several bugs which had been fixed in one branch,
but not the other.
I've also tagged the repository before making this checkin with the label
"before_signed_unsigned_changes".
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel & fphdf5
FreeBSD 4.10 (sleipnir) w/threadsafe
FreeBSD 4.10 (sleipnir) w/backward compatibility
Solaris 2.7 (arabica) w/"purify options"
Solaris 2.8 (sol) w/FORTRAN & C++
AIX 5.x (copper) w/parallel & FORTRAN
IRIX64 6.5 (modi4) w/FORTRAN
Linux 2.4 (heping) w/FORTRAN & C++
Misc. update:
Bug fix
Description:
Correct typedef for dataset region references to avoid struct alignment
issues on Crays.
Solution:
Change the typedef for hdset_reg_ref_t from a struct to an array of
unsigned char's of the correct size and propagate the appropriate adjustments
around the code.
Platforms tested:
FreeBSD 4.10 (sleipnir)
Linux 2.4 (verbena) w/fortran
Cray T90 (subzero) w/fortran
Cray SV1 (wind) w/fortran & parallel
Cray T3E (cyclone) w/fortran & parallel
Code cleanup
Description:
Tweak recent "forward compatibility" changes to the H5E* API (which allowed
for the old H5E API functions to remain unchanged) by allowing for the error
stack callback function (H5E_auto_t) to also remain unchanged from the 1.6
branch. This required changing the H5E{get|set}_auto routines to have the
old style H5E_auto_t type (which didn't have a stack ID parameter) and the new
H5E{get|set}_auto_stack routines to have a newer "H5E_auto_stack_t" type (which
has a stack ID parameter). This should make the H5E API changes as forwardly
compatible as possible.
One side-affect of this change was that it was impossible to determine if
the current auto error callback was the old style (H5E_auto_t) or the new style
(H5E_auto_stack_t) of callback, so a new API function (H5Eauto_is_stack) was
adde to query this.
Platforms tested:
FreeBSD 4.10 (sleipnir)
IRIX64 6.5 (modi4)
h5committest
Description: Restore 6 old error API functions back to the library to be backward
compatible with v1.6. They are H5Epush, H5Eprint, H5Ewalk, H5Eclear, H5Eset_auto,
H5Eget_auto. These functions do not have error stack as parameter.
Solution: Internally, these functions use default error stack.
Platforms tested: h5committest and fuss.
Misc. update: RELEASE.txt
Description:
Replaced the old metadata cache with a cache with a modified LRU
replacement policy. This should improve the hit rate.
Solution:
Since we want to flush cache entries in increasing address order, I
used the threaded binary B-tree code to store the cache entries.
There is a fair bit of overhead here, so we may want to consider
other options.
While the code is designed to allow the support of other replacement
algorithms, at present, only a modified version of LRU is supported.
The modified LRU algorithm requires that a user selectable portion
of the cache entries be clean. The clean entries are evicted first
when writes are not permitted. If the pool of clean entries is used
up, the cache grows beyond its user specified maximum size. The
cache can also exceed its maximum size if the combined size of the
protected (or locked) entries exceeds the maximum size of the cache.
Platforms tested:
eirene (serial, parallel, fp), h5committested
Misc. update:
Code cleanup & small bug fix
Description:
Regenerate dependency files
Add htri_t as separate type from hbool_t for code tracing purposes.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel
too minor to require h5committest
Code cleanup
Description:
Reduce compiler warnings on SGI IRIX
Platforms tested:
FreeBSD 4.10 (sleipnir)
SGI IRIX6 (Cheryl's machine)
Too minor to require full h5committest
Code cleanup & minor optimization
Description:
Re-work the way interface initialization routines are specified in the
library to avoid the overhead of checking for them in routines where there is
no interface initialization routine. This cleans up warnings with gcc 3.4,
reduces the library binary size a bit (about 2-3%) and should speedup the
library's execution slightly.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/gcc34
h5committest
Code optimization
Description:
Revised dataspace selections to use a more "object oriented" mechanism
to set the function pointers for each selection and selection iterator. This
reduces the amount and number of times that dataspace selection info has to
be copied.
Additionally, change hyperslab selection information to be dynamically
allocated instead of an inline struct.
Platforms tested:
Solaris 2.7 (arabica)
FreeBSD 4.10 (sleipnir) w/parallel
Too minor to require h5committest
Refactor code
Description:
Move chunk and contiguous cached raw data from file information to dataset
information. This simplifies a number of internal interfaces, aligns the
code with it's purpose better and should allow more optimizations to the
chunked data I/O performance.
Platforms tested:
Solaris 2.7 (arabica)
FreeBSD 4.10 (sleipnir)
h5committest
Misc. update:
Code optimization
Description:
Expand the use of macros to inline trivial function pointer lookup and
calls to reduce the overall number of functions invoked during normal operation
of the library.
Platforms tested:
Solaris 2.7 (arabica)
FreeBSD 4.9 (sleipnir) w/parallel
Too minor to require h5committest
Code cleanup/optimization
Description:
Don't clear the error stack when calling H5open(), it's unnecessary.
Platforms tested:
Solaris 2.7 (arabica)
FreeBSD 4.9 (sleipnir)
too minor to require h5committest
feature
Description:
H5check_version will print warning messages if the version does not
match or if the lib version string is not consistent, even when
$HDF5_DISABLE_VERSION_CHECK is set to 1. This will mess up
tests that try to match output.
Changed it so that if $HDF5_DISABLE_VERSION_CHECK is set to 2, no
warning about version mismatch is printed.
The lib version string warning is suppressed if $HDF5_DISABLE_VERSION_CHECK
is set to any non-zero value.
Platforms tested:
Tested in sol only but pretty comprehensive to make the warnings do get
suppressed.
Misc. update:
Bug fix
Description:
Address the failure in the FORTRAN builds by making some more of the
MPI-related typedefs in the library available outside of parallel builds.
Platforms tested:
Linux 2.4 (verbena) w/FORTRAN & C++
Bug fix
Description:
Fix a couple of bugaboos in the serial build that crept in
Platforms tested:
FreeBSD 4.9 (sleipnir)
too minor to require h5committest
Code cleanup
Description:
Clean up compiler warnings, especially the 'FUNC' variable not used which
comes out in production mode.
Solution:
Had to add a new FUNC_ENTER_NOAPI_NOINIT_NOFUNC macro for those non-API
functions which don't need the 'FUNC' variable defined. (This will be _so_
much easier when C99 is standard on all our supposed platforms, since it has a
__FUNC__ macro... )
Platforms tested:
FreeBSD 4.9 (sleipnir)
too minor for h5committest (although there were lots of files changed, the
change was minor in each one)
Code cleanup/bug fix
Description:
Re-enabled calling the H5E termination routine, now that the bugs are
fixed.
Removed tracing support for H5E_major_t and H5E_minor_t, now that they have
been removed from the library.
Cleaned up printing of IDs in tracing routine.
Platforms tested:
h5committested
Code cleanup
Description:
Add another dependency to the API shutdown code to not shut down the ID
management code until the other APIs which use IDs are shut down.
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committest
Bug fix
Description:
When objects were left open in a file, the API shutdown code would end
up closing property lists "out from underneath" files that were left open
pending the objects in the being closed. This caused problems later when
file objects would try to access their property list and fail to close,
causing an infinite loop in the library shutdown.
Solution:
Create some dependencies in the order that APIs are shut down, trying to
close "higher" level APIs before closing "low" level APIs. This still isn't
precise, but it does work correctly now.
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committest
New feature/Bug fix
Description:
Add new fill time value - H5D_FILL_TIME_IFSET which writes the fill value
to a dataset if the user has defined one, otherwise not writing the fill value
to the dataset.
Platforms tested:
FreeBSD 4.8 (sleipnir) serial & parallel
h5committest
Code cleanup
Description:
Limit the scope on more function prototypes/macros/typedefs.
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committest not necessary.
Bug fix.
Description:
If the buffer holding the list of interfaces that are attempting to be
closed down becomes full, the library stops recording that an infinite loop has
occurred when shutting down the interfaces.
Solution:
Always return the correct "pending" information.
Also added ellipsis to the string of interfaces to try to make it more
obvious that the buffer became full.
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committested
Code Improvements/Bug Fixes
Description:
Comparison of equality of a double/float variable to 0.0 is not
guaranteed to work and is bad practice.
In H5Fcontig.c, a warning was given by a statement like:
x = (++x) % y;
This could be confusing to a compiler I suppose.
In H5RS.c, a typedef of a structure was being tagged by the compiler
as "useless" because it had the form:
typedef struct foo {
int var1;
/* ... */
}; /* <--- note no name for this typedef */
The statement "typedef struct foo foo" is already in the header file.
Solution:
Test that the absolute value of the variable is < a very small
positive number.
Changed "x = (++x) % y" to "++x; x %= y;" instead.
Removed the "typedef" from the structure in the H5RS.c file.
Platforms tested:
Modi4 (Parallel & Fortran)
Verbena (C++ & Fortran)
Arabica (Fortran)
Misc. update:
Code cleanup
Description:
Clean up miscellaneous warnings which have crept into the code.
Fix "_POSIX_C_SOURCE not defined" warning on FreeBSD.
Adjust gcc compiler flags to be more concise for production mode.
Refactor the H5O code so that there is a stronger boundary between code
in the H5O package and code in the library which just calls H5O routines.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir) serial & parallel and gcc 2.95.4 & gcc 3.2.2
Misc. update:
Update MANIFEST if you add or remove any file.
Bug Fix
Description:
Metadata cache in parallel I/O can cause hangs in applications which
perform independent I/O on chunked datasets, because the metadata cache
can attempt to flush out dirty metadata from only a single process, instead
of collectively from all processes.
Solution:
Pass a dataset transfer property list down from every API function which
could possibly trigger metadata I/O.
Then, split the metadata cache into two sets of entries to allow dirty
metadata to be set aside when a hash table collision occurs during
independent I/O.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir) serial & parallel
Misc. update:
Updated release_docs/RELEASE
Code cleanup/new feature.
Description:
Split FUNC_LEAVE into API and non-API specific versions. This allows a
solution to compiling this branch with C++, as well as reducing the size
of the binaries produced.
Platforms tested:
FreeBSD 4.7 (sleipnir) w/serial, parallel (including MPE) & thread-safe
Lots of performance improvements & a couple new internal API interfaces.
Description:
Performance Improvements:
- Cached file offset & length sizes in shared file struct, to avoid
constantly looking them up in the FCPL.
- Generic property improvements:
- Added "revision" number to generic property classes to speed
up comparisons.
- Changed method of storing properties from using a hash-table
to the TBBT routines in the library.
- Share the propery names between classes and the lists derived
from them.
- Removed redundant 'def_value' buffer from each property.
- Switching code to use a "copy on write" strategy for
properties in each list, where the properties in each list
are shared with the properties in the class, until a
property's value is changed in a list.
- Fixed error in layout code which was allocating too many buffers.
- Redefined public macros of the form (H5open()/H5check, <variable>)
internally to only be (<variable>), avoiding innumerable useless
calls to H5open() and H5check_version().
- Reuse already zeroed buffers in H5F_contig_fill instead of
constantly re-zeroing them.
- Don't write fill values if writing entire dataset.
- Use gettimeofday() system call instead of time() system when
checking the modification time of a dataset.
- Added reference counted string API and use it for tracking the
names of objects opening in a file (for the ID->name code).
- Removed redundant H5P_get() calls in B-tree routines.
- Redefine H5T datatype macros internally to the library, to avoid
calling H5check redundantly.
- Keep dataspace information for dataset locally instead of reading
from disk each time. Added new module to track open objects
in a file, to allow this (which will be useful eventually for
some FPH5 metadata caching issues).
- Remove H5AC_find macro which was inlining metadata cache lookups,
and call function instead.
- Remove redundant memset() calls from H5G_namei() routine.
- Remove redundant checking of object type when locating objects
in metadata cache and rely on the address only.
- Create default dataset object to use when default dataset creation
property list is used to create datasets, bypassing querying
for all the property list values.
- Use default I/O vector size when performing raw data with the
default dataset transfer property list, instead of querying for
I/O vector size.
- Remove H5P_DEFAULT internally to the library, replacing it with
more specific default property list based on the type of
property list needed.
- Remove redundant memset() calls in object header message (H5O*)
routines.
- Remove redunant memset() calls in data I/O routines.
- Split free-list allocation routines into malloc() and calloc()-
like routines, instead of one combined routine.
- Remove lots of indirection in H5O*() routines.
- Simplify metadata cache entry comparison routine (used when
flushing entire cache out).
- Only enable metadata cache statistics when H5AC_DEBUG is turned
on, instead of always tracking them.
- Simplify address comparison macro (H5F_addr_eq).
- Remove redundant metadata cache entry protections during dataset
creation by protecting the object header once and making all
the modifications necessary for the dataset creation before
unprotecting it.
- Reduce # of "number of element in extent" computations performed
by computing and storing the value during dataspace creation.
- Simplify checking for group location's file information, when file
has not been involving in file-mounting operations.
- Use binary encoding for modification time, instead of ASCII.
- Hoist H5HL_peek calls (to get information in a local heap)
out of loops in many group routine.
- Use static variable for iterators of selections, instead of
dynamically allocation them each time.
- Lookup & insert new entries in one step, avoiding traversing
group's B-tree twice.
- Fixed memory leak in H5Gget_objname_idx() routine (tangential to
performance improvements, but fixed along the way).
- Use free-list for reference counted strings.
- Don't bother copying object names into cached group entries,
since they are re-created when an object is opened.
The benchmark I used to measure these results created several thousand
small (2K) datasets in a file and wrote out the data for them. This is
Elena's "regular.c" benchmark.
These changes resulted in approximately ~4.3x speedup of the
development branch when compared to the previous code in the
development branch and ~1.4x speedup compared to the release
branch.
Additionally, these changes reduce the total memory used (code and
data) by the development branch by ~800KB, bringing the development
branch back into the same ballpark as the release branch.
I'll send out a more detailed description of the benchmark results
as a followup note.
New internal API routines:
Added "reference counted strings" API for tracking strings that get
used by multiple owners without duplicating the strings.
Added "ternary search tree" API for text->object mappings.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
Other platforms/configurations tested?
FreeBSD 4.7 (sleipnir) serial & parallel
Solaris 2.6 (baldric) serial
new feature
Description:
Added MPICH/MPE instrumentation support.
All source code are bracketed by the macro H5_HAVE_MPE.
Use "--enable-mpe" to configure it in.
Currently only worked in Eirene because the MPE library is
not installed in all machines yet.
The added file, H5MPprivate.h, holds HDF5/MPE related defintions.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}? Yep.
Other platforms/configurations tested?
--enable-mpe feature tested in Eirene.
Purpose:
New Feature; Optimization; Clean-up
(Merged from 1.4 branch)
Description:
There is no symbolic constant to pass to functions that take an
optional object ID for when the caller wants to indicate no object
ID. In the past the caller always passed a negative integer.
GPFS performs poorly.
The h5ls tool decides whether to list the file name in the output
based on a compile-time choice, which isn't always optimal at run
time.
Solution:
Added a symbolic constant H5I_INVALID_HID.
Added code to tell the mmfsd of GPFS to forego byte range token
prefetching.
h5ls decides whether to print the file name at runtime based on
the number of objects being listed.
Platforms tested:
SuSE Linux (arborea), gcc and mpich-1.2.4
SunOS (baldric), gcc
2002-12-03 23:23:45 Robb Matzke <matzke@arborea.spizella.com>
* H5_trace: Added printing for H5S_sel_type, code "St".
2002-12-03 22:45:11 Robb Matzke <matzke@arborea.spizella.com>
*: Removed a little white space. Other changes are probably due to
additional API tracing statements inserted
automatically.
New feature.
Description:
Added check for "HDF5_DISABLE_VERSION_CHECK" environment variable to
disable the version checking code between the headers and the library.
Tested:
FreeBSD 4.7 (sleipnir) - no triple check necessary.
Code cleanup
Description:
Changed the last HRETURN* statements in the FUNC_ENTER macros into HGOTO*
macros, which reduces the size of the library binary in certain
configurations by another 10%
Platforms tested:
FreeBSD 4.6 (sleipnir) serial & parallel, IRIX64 6.5 (modi4) serial &
parallel
Purpose:
Merged from 1.4 branch
Description:
API tracing improvements
Platforms tested:
Linux (--disable-hsizet didn't work before my changes and still doesn't
work, but --enable-hsizet is fine).
Note: This checkin includes temporary code in H5FDmpiposix.c to turn
off GPFS byte range token prefetches on ASCI/Blue. Once the HDF5 API
supports the necessary prerequisite functionality this temporary code
can be migraged up above HDF5.
