Update
Description:
Big change to the H5FD.c module:
- Split apart H5FD_alloc and H5FD_free. H5FD_alloc was huge and
H5FD_free had a freeing of the freelist part which I needed to
call from the SAP.
- Added support for FPHDF5. If it's a client, then it sends the
allocation or free request to the SAP. The SAP will call the
same code, but it'll actually do the allocation/freeing in that
case.
Platforms tested:
Linux & Modi4
Update
Description:
Added support for the allocation and freeing of space in the file.
This information is kept on the Server.
Platforms tested:
Linux & Modi4
Code cleanup
Description:
Changed hard-coded magic number (32) for the maximum number of filters
in a filter pipeline to use a symbolic constant (H5Z_MAX_NFILTERS)
instead. This limit could (and probably should) be removed to allow an
unlimited number of filters in a pipeline.
Platforms tested:
FreeBSD 4.7 (sleipnir)
Code cleanup
Description:
Clean up miscellaneous warnings which have crept into the code.
Fix "_POSIX_C_SOURCE not defined" warning on FreeBSD.
Adjust gcc compiler flags to be more concise for production mode.
Refactor the H5O code so that there is a stronger boundary between code
in the H5O package and code in the library which just calls H5O routines.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir) serial & parallel and gcc 2.95.4 & gcc 3.2.2
Misc. update:
Update MANIFEST if you add or remove any file.
Update
Description:
Since metadata doesn't use a "set view" to do I/O, I moved the check
for the use_set_view variable down into the "real" write routine.
Also added a check for when the server is dumping metadata to the
file. in that case, don't write this information to the SAP...
Platforms tested:
Linux
Update
Description:
Added the flush function.
Modified so that it calls the FPHDF5 code for reading and writing.
This involved splitting the write function up into three different
parts to avoid lame goto's. There's some code which will copy a data
xfer property list and add in there that we're "dumping" the data so
that we can recall this layer when the SAP dumps metadata to a
process...
Platforms tested:
Linux
Update
Description:
Added a "metadata flush" function which allows the clients to force a
dump of the metadata from the SAP. This should be done before closing
the file.
Modified the information stored on an "open" of the file. The
filename is unnecessary, so I got rid of it.
More integration with the File Driver code: Passing in a data xfer
property list.
Platforms tested:
Linux
Bug Fix
Description:
Some of the error conditions in the H5GOTO_ERROR macros were NULL
instead of FAIL.
Solution:
Changed accordingly.
Platforms tested:
Linux
Code cleanup.
Description:
Break up the ~9350 line H5T.c module into smaller pieces, which contain
code for a particular feature or support for a datatype class.
This should make the "main" H5T code (still in H5T.c) easier to support,
as well as removing some of the "minor" routines from the user applications
which don't use them (my rough estimates show about 4% reduction (~30K on
a FreeBSD machine) in optimized, staticly-linked binaries for very simple
programs)
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir)
Misc. update:
Update MANIFEST
Bug fix.
Description:
Missed adding the dxpl to the 'flush' VFL callback in the stream driver,
causing the C++ compiles to fail.
Solution:
Added the dxpl parameter.
Platforms tested:
FreeBSD 4.7 (sleipnir) w/CC=g++
Update
Description:
Folded in Quincey's changes to the caching stuff. (Stole the code
from the H5FDmpio driver).
Roughed in some code for doing a read from the SAP. Also roughed in
code for doing a write. However, the write requires an OID, which I'm
not sure how to pass down into the driver (maybe via the
dxpl_id?...but then it has always to be set before calling one of
these routines...).
Removed some of the global variables which were there because of the
FPHDF5 stuff...
Removed the H5Ofphdf5.* stuff from the Makefile.in, since I'm pretty
sure it's going away and I don't want to waste time updating that
module if that's the case...so just don't compile it.
Platforms tested:
Linux
Fix
Description:
The extra field I put in this structure wasn't needed.
Solution:
Placed it in the file driver's structure instead.
Platforms tested:
Linux
Update
Description:
Removed some obsolete fields - such as the AC_subst_t object ID -
which isn't needed. Fixed so that there are fewer global variables.
There are still some left, however. Modified the client and server
code so that it handles read requests better. There were some flaws
in how this was done before (it was calling H5FD_read() instead of
just returning the status that it couldn't find the metadata in the
cache).
Platforms tested:
Linux
Bug Fix
Description:
Metadata cache in parallel I/O can cause hangs in applications which
perform independent I/O on chunked datasets, because the metadata cache
can attempt to flush out dirty metadata from only a single process, instead
of collectively from all processes.
Solution:
Pass a dataset transfer property list down from every API function which
could possibly trigger metadata I/O.
Then, split the metadata cache into two sets of entries to allow dirty
metadata to be set aside when a hash table collision occurs during
independent I/O.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir) serial & parallel
Misc. update:
Updated release_docs/RELEASE
Bug fix
Description:
The "system scope" for threads isn't supported on all platforms.
Solution:
Add detection of this feature to the configure script and check for
"H5_HAVE_SYSTEM_SCOPE_THREADS" in the appropriate places.
Platforms tested:
modi4 w/threadsafe
New feature for developers.
Description:
Added "function stack" tracing to library. This allows developers (there
is no public API) to call H5FS_print within the library and get a listing
of the functions traversed to reach that point in the library. Eventually,
I may add support for reporting the parameters to each function also...
Mainly for debugging parallel I/O programs, but I think it will come in
handy in other cases also.
The function stack tracking is controlled with a configure switch:
--enable-funcstack, which defaults to enabled currently. When we branch
for 1.6, we should change the default setting on the branch to be disabled.
Also, added a destructor to the thread-specific keys when thread-safety is
turned on in the library. Otherwise, they were leaking memory and causing
difficult to debug errors in threaded programs (like the test/ttsafe test).
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir) w/thread-safety enabled.
Misc. update:
Updated MANIFEST with new files added (src/H5FS.c & src/H5FDprivate.h)
Update release_docs/RELEASE with thread-safety bug fix.
Field Addition
Description:
Added a File ID field to the H5FD_t structure so that, way deep down
in the H5FD_* functions, I'll know what ID the SAP wants for this
particular file.
This is #ifdef'd out so that if you don't have FPHDF5 enabled, then
it won't be there...
Platforms tested:
Linux
Update
Description:
H5FP.c, H5FPclient.c, H5FPprivate.h, H5FPserver.c:
Update. More progression towards the SAP as metadata cache. It
only lacks the ability to take care of metadata allocations.
H5FDfphdf5.[ch]:
Start of a new driver for FPHDF5. Not fully implemented just
yet...
Platforms tested:
Linux
Update
Description:
Converted the "FUNC_LEAVE" macros to "FUNC_LEAVE_NOAPI" to be
consistent with the rest of the library.
Note: This is probably completely useless as the file is going to go
away with the new model of the SAP as metadata cache...But it helps
me compile...
Platforms tested:
Linux (FPHDF5)
Update
Description:
Added support for defining what the haddr type is in terms of MPI
types.
Solution:
Include some #defines when we typedef haddr...
Platforms tested:
Linux (For FPHDF5 stuff)...
Update
Description:
Changes:
- Added support for the server dumping metadata writes to a
client.
- Some of the code wasn't handling allocated buffers correctly
(freeing twice).
- Modified server so that it handles metadata only.
Platforms tested:
Linux
Update
Description:
This is the first conversion of the FPHDF5 code to be a metadata
cache. There's an extra error message. I rewrote the sync/change code
to be read metadata/write metadata instead.
I still need to hook these changes into the HDF5 code so that it
looks at the SAP first before checking the file for metadata.
Platforms tested:
Linux
Bug Fix
Description:
When calling H5Fopen with the core VFL driver, but without the
H5F_ACC_CREAT flag goes ahead and creates a memory file.
Solution:
Check for the H5F_ACC_CREAT flag before allowing the memory file to be
created.
Platforms tested:
FreeBSD 4.7 (sleipnir)
Updated
Description:
Updated Copyright notice.
Replaced HGOTO_ERROR calls involving MPI calls with HMPI_GOTO_ERROR.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}? Tested on Eirene (PP) only since
the code is in the MPIO module only.
Bug fix
Description:
Currently, when the library encounters an object header message that isn't
know, it fails to open that object in the file.
Solution:
Allow the library to skip over the unknown object header message and
continue to process the remaining messages, in the hope that the skipped
message isn't important later. If it is important, it will be caught at
a higher level of the library.
Platforms tested:
FreeBSD 4.7 (sleipnir)
This is generated by autoheader.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}? YES
Other platforms/configurations tested? eirene with --enable-mpe
Code cleanup & bug fix
Description:
Cleanup another set of warnings on Windows and also fix mis-placed assertion
that caused the daily tests to fail.
Platforms tested:
IRIX64 6.5 (modi4) w/-n32
Purpose:
Change some macros to make windows happy
Description:
Currently no srandom and random functions on windows,
Function gethostname cannot be resolved when DLL turned on
Solution:
use srand and rand to replace srandom and random
turn off the option to check gethostname
Platforms tested:
windows 2000, linux 2.2.18smp
Misc. update:
Update MANIFEST if you add or remove any file.
Update release_docs/RELEASE for bug fixes, new features, etc.
Update applicable document files too.
Code cleanup
Description:
Add 'H5_DLL' macro to H5FL macros, in order to allow Windows builds to
work.
Platforms tested:
FreeBSD 4.7 (sleipnir)
Visual Studio 6.0
Code cleanup
Description:
Various code cleanups to allow the development branch to be compiled with
a C++ compiler (i.e. CC=g++ )
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir) C++
Code improvement
Description:
Add extra pair of braces to API versions of FUNC_ENTER/FUNC_LEAVE macros,
to make API <-> non-API mismatches obvious.
Platforms tested:
FreeBSD 4.7 (sleipnir)
Code cleanup/new feature.
Description:
Split FUNC_LEAVE into API and non-API specific versions. This allows a
solution to compiling this branch with C++, as well as reducing the size
of the binaries produced.
Platforms tested:
FreeBSD 4.7 (sleipnir) w/serial, parallel (including MPE) & thread-safe
Lots of performance improvements & a couple new internal API interfaces.
Description:
Performance Improvements:
- Cached file offset & length sizes in shared file struct, to avoid
constantly looking them up in the FCPL.
- Generic property improvements:
- Added "revision" number to generic property classes to speed
up comparisons.
- Changed method of storing properties from using a hash-table
to the TBBT routines in the library.
- Share the propery names between classes and the lists derived
from them.
- Removed redundant 'def_value' buffer from each property.
- Switching code to use a "copy on write" strategy for
properties in each list, where the properties in each list
are shared with the properties in the class, until a
property's value is changed in a list.
- Fixed error in layout code which was allocating too many buffers.
- Redefined public macros of the form (H5open()/H5check, <variable>)
internally to only be (<variable>), avoiding innumerable useless
calls to H5open() and H5check_version().
- Reuse already zeroed buffers in H5F_contig_fill instead of
constantly re-zeroing them.
- Don't write fill values if writing entire dataset.
- Use gettimeofday() system call instead of time() system when
checking the modification time of a dataset.
- Added reference counted string API and use it for tracking the
names of objects opening in a file (for the ID->name code).
- Removed redundant H5P_get() calls in B-tree routines.
- Redefine H5T datatype macros internally to the library, to avoid
calling H5check redundantly.
- Keep dataspace information for dataset locally instead of reading
from disk each time. Added new module to track open objects
in a file, to allow this (which will be useful eventually for
some FPH5 metadata caching issues).
- Remove H5AC_find macro which was inlining metadata cache lookups,
and call function instead.
- Remove redundant memset() calls from H5G_namei() routine.
- Remove redundant checking of object type when locating objects
in metadata cache and rely on the address only.
- Create default dataset object to use when default dataset creation
property list is used to create datasets, bypassing querying
for all the property list values.
- Use default I/O vector size when performing raw data with the
default dataset transfer property list, instead of querying for
I/O vector size.
- Remove H5P_DEFAULT internally to the library, replacing it with
more specific default property list based on the type of
property list needed.
- Remove redundant memset() calls in object header message (H5O*)
routines.
- Remove redunant memset() calls in data I/O routines.
- Split free-list allocation routines into malloc() and calloc()-
like routines, instead of one combined routine.
- Remove lots of indirection in H5O*() routines.
- Simplify metadata cache entry comparison routine (used when
flushing entire cache out).
- Only enable metadata cache statistics when H5AC_DEBUG is turned
on, instead of always tracking them.
- Simplify address comparison macro (H5F_addr_eq).
- Remove redundant metadata cache entry protections during dataset
creation by protecting the object header once and making all
the modifications necessary for the dataset creation before
unprotecting it.
- Reduce # of "number of element in extent" computations performed
by computing and storing the value during dataspace creation.
- Simplify checking for group location's file information, when file
has not been involving in file-mounting operations.
- Use binary encoding for modification time, instead of ASCII.
- Hoist H5HL_peek calls (to get information in a local heap)
out of loops in many group routine.
- Use static variable for iterators of selections, instead of
dynamically allocation them each time.
- Lookup & insert new entries in one step, avoiding traversing
group's B-tree twice.
- Fixed memory leak in H5Gget_objname_idx() routine (tangential to
performance improvements, but fixed along the way).
- Use free-list for reference counted strings.
- Don't bother copying object names into cached group entries,
since they are re-created when an object is opened.
The benchmark I used to measure these results created several thousand
small (2K) datasets in a file and wrote out the data for them. This is
Elena's "regular.c" benchmark.
These changes resulted in approximately ~4.3x speedup of the
development branch when compared to the previous code in the
development branch and ~1.4x speedup compared to the release
branch.
Additionally, these changes reduce the total memory used (code and
data) by the development branch by ~800KB, bringing the development
branch back into the same ballpark as the release branch.
I'll send out a more detailed description of the benchmark results
as a followup note.
New internal API routines:
Added "reference counted strings" API for tracking strings that get
used by multiple owners without duplicating the strings.
Added "ternary search tree" API for text->object mappings.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
Other platforms/configurations tested?
FreeBSD 4.7 (sleipnir) serial & parallel
Solaris 2.6 (baldric) serial
