Description:
When an attribute was created with a datatype or dataspace that was shared in
the same object header that the attribute was in, the attribute could not be
deleted. Changes made to ensure that the attribute can be deleted both when the
attribute is in the object header and when it is shared in the heap. Object
header message decode routines now take an "open_oh" parameter to enable them to
avoid opening the same object header twice.
Tested: jam, smirom (h5committest)
Remove trailing whitespace from C/C++ source files, with the following
script:
foreach f (*.[ch] *.cpp)
sed 's/[[:blank:]]*$//' $f > sed.out && mv sed.out $f
end
Tested on:
Mac OS X/32 10.5.5 (amazon)
No need for h5committest, just whitespace changes...
datatype versions are encountered.
Description: The library now recognizes some problems with datatype versions in
H5O_decode_helper(), and, if not performing strict format checks, automatically
corrects them. Framework added for other message decode routines to
automatically correct file errors. Datatype version information added to
h5debug.
Tested: kagiso, smirom, linew (h5committest)
Description:
As part of our Windows cleanup, we try to remove windows-specific tweaks in the source code. There are many instances where Windows code is introduces via ifdef's. We re-evaluate whether they are still required, and found that many of them are not. Others we change to "feature"-specific code, rather than Windows-specific.
Tested:
VS2005 on WinXP
VS.NET on WinXP
h5committest (kagisopp, smirom, linew)
Check in "unique, but sharable" optimization to ISOHM code, which
allows object header messages that are only used in one object to remain in
the sole user's header, but migrates messages that are used in more than one
header into the ISOHM heap.
Tested on:
Mac OS X/32 10.4.9 (amazon)
FreeBSD/32 6.2 (duty)
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Tested platform:
Kagiso only since it is only a comment block change. If it works in one
machine, it should work in all, I hope. Still need to check the parallel
build on copper.
Remove the "scaffolding" for shared message method invocation and
simplify the way shared messages are dealt with in general.
Tested on:
FreeBSD/32 6.2 (duty)
Mac OS X/32 10.4.8 (amazon)
Add "set_crt_index" and "get_crt_index" methods for the object header
message class.
Unify fractal heap definitions for shared messages and attributes, under
"object header" fractal heap definitions.
Initial code for adding creation order index to object header messages.
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Added can_share callback for OH messages. This determines whether the
message is allowed to be shared in the heap (committed and immutable datatypes
can't be).
Fixed a bug in the dense attribute storage that tried to open the shared message
heap when it hadn't been created yet.
Made the test to extend shared dataspace messages a bit more robust.
Refactored the code the searches a shared message list index to be a little
more efficient.
A few other minor changes.
Tested on smirom, kagiso, Windows, and juniper.
Add first pass of "dense" attribute storage to objects. Lots of parts of
this are stubbed out, but all the tests are passing and I'll work on the corner
cases soon.
Eliminated several unused parameters from object header message callback
routines.
Other, miscellaneous code cleanups, etc. (and probably some things I've
forgotten about... :-)
Tested on:
FreeBSD/32 4.11 (sleipnir)
Linux/32 2.4 (heping)
AIX/32 5.? (copper)
Finish removing library's internal code that uses H5G_get_objinfo() and
retarget it at either getting the link information or the object information,
as appropriate. (Still need to add user-level tests for H5Oget_info(), but
since several internal components of the library depend on the internal version,
it appears to be working correctly).
Tested on:
FreeBSD/32 4.11 (sleipnir)
Linux/322.4 (heping)
Linux/64 2.4 (mir)
AIX/32 5.? (copper)
Mac OS X/32 10.4.8 (amazon)
This feature is still in progress; Shared Object Header Messages are not
complete as a feature and are not thoroughly tested. There are still
"TODO" comments in the code (comments with the word "JAMES" in them,
so as not to be confused with other TODO comments).
Hopefully this checkin will reduce the liklihood of conflicts as I finish
implementing this feature.
All current tests pass on juniper, copper (parallel), heping, kagiso, and mir.
New feature
Description:
Add in a combination of Peter's & my code to support copying
variable-length data from one file to another, although currently only
supported with contiguous data storage.
Platforms tested:
FreeBSD 4.11 (sleipnir)
h5committest
New feature
Description:
Check in baseline for compact group revisions, which radically revises the
source code for managing groups and object headers.
WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!!
WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!!
This initiates the "unstable" phase of the 1.7.x branch, leading up
to the 1.8.0 release. Please test this code, but do _NOT_ keep files created
with it - the format will change again before the release and you will not
be able to read your old files!!!
WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!!
WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!!
Solution:
There's too many changes to really describe them all, but some of them
include:
- Stop abusing the H5G_entry_t structure and split it into two separate
structures for non-symbol table node use within the library: H5O_loc_t
for object locations in a file and H5G_name_t to store the path to
an opened object. H5G_entry_t is now only used for storing symbol
table entries on disk.
- Retire H5G_namei() in favor of a more general mechanism for traversing
group paths and issuing callbacks on objects located. This gets us out
of the business of hacking H5G_namei() for new features, generally.
- Revised H5O* routines to take a H5O_loc_t instead of H5G_entry_t
- Lots more...
Platforms tested:
h5committested and maybe another dozen configurations.... :-)
New feature
Description:
Add in baseline "object copy" code from Peter [in the form of a new API
routine: H5Gcopy()]. There's still some work to do (like handling variable-
length datatypes and possibly support for references) and it hasn't been tested
on mounted files yet, but the core functionality is there and working
correctly.
I've also got a set of patches to update the 1.6 branch with tweaks to
keep the branches mostly in sync, but Elena will kill me if I import them
before the 1.6.5 release is out... :-)
Platforms tested:
FreeBSD 4.11 (sleipnir)
h5committested
Code cleanup
Description:
Merge back changes from "compact group" work that improve the
infrastructure of the library and may impact others. In this round of
merging, that includes:
- Move datatype allocation into single internal routine, instead of
duplicated code that was spread out in a dozen or so places.
- Clean up guts of object header routines (H5O_*) to allow for some of
the fancieroperations that need to be performed on groups, along with
some general improvements.
- Added a new error code
- Some minor cleanups in other code....
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4
Mac OS X
Code cleanup
Description:
Trim trailing whitespace, which is making 'diff'ing the two branches
difficult.
Solution:
Ran this script in each directory:
foreach f (*.[ch] *.cpp)
sed 's/[[:blank:]]*$//' $f > sed.out && mv sed.out $f
end
Platforms tested:
FreeBSD 4.11 (sleipnir)
Too minor to require h5committest
Description: Removed PABLO from the source
Solution:
Platforms tested: arabica with 64-bit, copper with parallel,
heping with GNU C and C++ and PGI fortran (but
I disabled hl, there is some weird problem only
on heping: F9XMODFLAG is not
propagated to the Makefile files
Misc. update:
Bug Fix/Code Cleanup/Doc Cleanup/Optimization/Branch Sync :-)
Description:
Generally speaking, this is the "signed->unsigned" change to selections.
However, in the process of merging code back, things got stickier and stickier
until I ended up doing a big "sync the two branches up" operation. So... I
brought back all the "infrastructure" fixes from the development branch to the
release branch (which I think were actually making some improvement in
performance) as well as fixed several bugs which had been fixed in one branch,
but not the other.
I've also tagged the repository before making this checkin with the label
"before_signed_unsigned_changes".
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel & fphdf5
FreeBSD 4.10 (sleipnir) w/threadsafe
FreeBSD 4.10 (sleipnir) w/backward compatibility
Solaris 2.7 (arabica) w/"purify options"
Solaris 2.8 (sol) w/FORTRAN & C++
AIX 5.x (copper) w/parallel & FORTRAN
IRIX64 6.5 (modi4) w/FORTRAN
Linux 2.4 (heping) w/FORTRAN & C++
Misc. update:
Code cleanup & optimization
Description:
Improve ADF/CGNS benchmark by reducing the number of internal attribute
copies made during creations, opens and writes.
Added new H5O_iterate() routine for iterating through messages of a certain
type in the object header (attributes are the only message currently that can
have multiple instances in the object header).
Cross-pollinated various minor code cleanups to reduce diffs between
branches.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel
Solaris 2.7 (arabica)
Too minor to require h5committest
bug fix
Description:
The H5O_mtime_decode function was not handling properly the case for the
Code Warrior compiler
Solution:
Platforms tested:
Code Warrior
Misc. update:
Purpose:
Bug fix
Description:
Modification time test (mtime) would die silently on some systems. This is
because the code is very system-dependant (it relies on getting the current
time and the timezone from the OS).
Solution:
mtime test now uses TEST_ERROR macro to print "FAILED" and to output where the
failure occurred. Configure script is a little smarter about whether
gettimeofday() function returns the timezone correctly.
Further bugs will need to be addressed on a system-by-system basis.
Platforms tested:
sleipnir, arabica, verbena, copper, windows (VC7)
Code cleanup
Description:
Reduce compiler warnings on SGI IRIX
Platforms tested:
FreeBSD 4.10 (sleipnir)
SGI IRIX6 (Cheryl's machine)
Too minor to require full h5committest
Code cleanup & minor optimization
Description:
Re-work the way interface initialization routines are specified in the
library to avoid the overhead of checking for them in routines where there is
no interface initialization routine. This cleans up warnings with gcc 3.4,
reduces the library binary size a bit (about 2-3%) and should speedup the
library's execution slightly.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/gcc34
h5committest
Bug fixes and code cleanup
Description:
Lots of changes here:
- Fixed bug #691 - when shared datatypes are used in attributes they
are incorrectly copied into the attribute instead of referring
the the named datatype in the file. This required bumping the
version of the attribute message. The new version of the attribute
message is only written out when a shared datatype is used in
the attribute. [Also, this format change made the size of the
attribute smaller.]
- Added information to attribute debugging routine so that shared
datatypes are displayed correctly with the h5debug tool.
- Refactored the H5O* routines to extract code that was common to
several routines into subroutines to call.
- Added 'link' method for H5O message sub-classes, which increments
the link count on shared objects when a message is created which
shares them.
- Corrected [unreported] bug where the link count was not being
decremented on the shared object when a object header message
with a reference to that object was deleted from the file.
- Reduced size of shared message from 49 bytes (which was incorrect
anyway and should have been 48 bytes) to 10 bytes, which required
bumping the version of "shared" messages.
- Refactored some of the shared datatype routines to allow for easier
queries of "committedness" internally to the library and also
added routine to easily increment/decrement the reference count of
a shared datatype.
Platforms tested:
FreeBSD 4.9 (sleipnir)
h5committest
New feature, sorta
Description:
Track changes to various internal APIs
Platforms tested:
FreeBSD 4.8 (sleipnir) w/C++
Linux 2.4 (burrwhite) w/FORTRAN
Solaris 2.7 (arabica) w/FORTRAN
IRIX64 6.5 (modi4) w/parallel & FORTRAN
(h5committest not run due to my ongoing difficulties with C++ on burrwhite).
Code cleanup
Description:
Clean up miscellaneous warnings which have crept into the code.
Fix "_POSIX_C_SOURCE not defined" warning on FreeBSD.
Adjust gcc compiler flags to be more concise for production mode.
Refactor the H5O code so that there is a stronger boundary between code
in the H5O package and code in the library which just calls H5O routines.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir) serial & parallel and gcc 2.95.4 & gcc 3.2.2
Misc. update:
Update MANIFEST if you add or remove any file.
Bug Fix
Description:
Metadata cache in parallel I/O can cause hangs in applications which
perform independent I/O on chunked datasets, because the metadata cache
can attempt to flush out dirty metadata from only a single process, instead
of collectively from all processes.
Solution:
Pass a dataset transfer property list down from every API function which
could possibly trigger metadata I/O.
Then, split the metadata cache into two sets of entries to allow dirty
metadata to be set aside when a hash table collision occurs during
independent I/O.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir) serial & parallel
Misc. update:
Updated release_docs/RELEASE
Code cleanup/new feature.
Description:
Split FUNC_LEAVE into API and non-API specific versions. This allows a
solution to compiling this branch with C++, as well as reducing the size
of the binaries produced.
Platforms tested:
FreeBSD 4.7 (sleipnir) w/serial, parallel (including MPE) & thread-safe
Lots of performance improvements & a couple new internal API interfaces.
Description:
Performance Improvements:
- Cached file offset & length sizes in shared file struct, to avoid
constantly looking them up in the FCPL.
- Generic property improvements:
- Added "revision" number to generic property classes to speed
up comparisons.
- Changed method of storing properties from using a hash-table
to the TBBT routines in the library.
- Share the propery names between classes and the lists derived
from them.
- Removed redundant 'def_value' buffer from each property.
- Switching code to use a "copy on write" strategy for
properties in each list, where the properties in each list
are shared with the properties in the class, until a
property's value is changed in a list.
- Fixed error in layout code which was allocating too many buffers.
- Redefined public macros of the form (H5open()/H5check, <variable>)
internally to only be (<variable>), avoiding innumerable useless
calls to H5open() and H5check_version().
- Reuse already zeroed buffers in H5F_contig_fill instead of
constantly re-zeroing them.
- Don't write fill values if writing entire dataset.
- Use gettimeofday() system call instead of time() system when
checking the modification time of a dataset.
- Added reference counted string API and use it for tracking the
names of objects opening in a file (for the ID->name code).
- Removed redundant H5P_get() calls in B-tree routines.
- Redefine H5T datatype macros internally to the library, to avoid
calling H5check redundantly.
- Keep dataspace information for dataset locally instead of reading
from disk each time. Added new module to track open objects
in a file, to allow this (which will be useful eventually for
some FPH5 metadata caching issues).
- Remove H5AC_find macro which was inlining metadata cache lookups,
and call function instead.
- Remove redundant memset() calls from H5G_namei() routine.
- Remove redundant checking of object type when locating objects
in metadata cache and rely on the address only.
- Create default dataset object to use when default dataset creation
property list is used to create datasets, bypassing querying
for all the property list values.
- Use default I/O vector size when performing raw data with the
default dataset transfer property list, instead of querying for
I/O vector size.
- Remove H5P_DEFAULT internally to the library, replacing it with
more specific default property list based on the type of
property list needed.
- Remove redundant memset() calls in object header message (H5O*)
routines.
- Remove redunant memset() calls in data I/O routines.
- Split free-list allocation routines into malloc() and calloc()-
like routines, instead of one combined routine.
- Remove lots of indirection in H5O*() routines.
- Simplify metadata cache entry comparison routine (used when
flushing entire cache out).
- Only enable metadata cache statistics when H5AC_DEBUG is turned
on, instead of always tracking them.
- Simplify address comparison macro (H5F_addr_eq).
- Remove redundant metadata cache entry protections during dataset
creation by protecting the object header once and making all
the modifications necessary for the dataset creation before
unprotecting it.
- Reduce # of "number of element in extent" computations performed
by computing and storing the value during dataspace creation.
- Simplify checking for group location's file information, when file
has not been involving in file-mounting operations.
- Use binary encoding for modification time, instead of ASCII.
- Hoist H5HL_peek calls (to get information in a local heap)
out of loops in many group routine.
- Use static variable for iterators of selections, instead of
dynamically allocation them each time.
- Lookup & insert new entries in one step, avoiding traversing
group's B-tree twice.
- Fixed memory leak in H5Gget_objname_idx() routine (tangential to
performance improvements, but fixed along the way).
- Use free-list for reference counted strings.
- Don't bother copying object names into cached group entries,
since they are re-created when an object is opened.
The benchmark I used to measure these results created several thousand
small (2K) datasets in a file and wrote out the data for them. This is
Elena's "regular.c" benchmark.
These changes resulted in approximately ~4.3x speedup of the
development branch when compared to the previous code in the
development branch and ~1.4x speedup compared to the release
branch.
Additionally, these changes reduce the total memory used (code and
data) by the development branch by ~800KB, bringing the development
branch back into the same ballpark as the release branch.
I'll send out a more detailed description of the benchmark results
as a followup note.
New internal API routines:
Added "reference counted strings" API for tracking strings that get
used by multiple owners without duplicating the strings.
Added "ternary search tree" API for text->object mappings.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
Other platforms/configurations tested?
FreeBSD 4.7 (sleipnir) serial & parallel
Solaris 2.6 (baldric) serial
Purpose:
Fix bugs for windows platform.
Description:
1. Update HDlseek for windows
2. Update H5Otime.c for normal time(non-daylight time)
3. Add H5_DLL in front of two functions so that windows DLL can work.
Solution:
Platforms tested:
windows 2000, eirene(since most updates have WIN32 around it, not need to test other platforms)
Misc. update:
Update MANIFEST if you add or remove any file.
Update release_docs/RELEASE for bug fixes, new features, etc.
Update applicable document files too.
Code cleanup
Description:
Changed the last HRETURN* statements in the FUNC_ENTER macros into HGOTO*
macros, which reduces the size of the library binary in certain
configurations by another 10%
Platforms tested:
FreeBSD 4.6 (sleipnir) serial & parallel, IRIX64 6.5 (modi4) serial &
parallel
Code cleanup
Description:
Change most (all?) HRETURN_ERROR macros to HGOTO_ERROR macros, along with
HRETURN macros to HGOTO_DONE macros. This unifies the error return path
from functions and reduces the size of the library by up to 10% on some
platforms.
Additionally, I improved a lot of the error cleanup code in many routines.
Platforms tested:
FreeBSD 4.6 (sleipnir) serial & parallel and IRIX64 6.5 (modi4) serial &
parallel.
Code cleanup
Description:
Broke the FUNC_ENTER macro into several macros, with more specialized
uses (which followup mail will describe). This was designed to move
most/all of the checks which could be done at compile time to that point,
instead of needlessly performing them (over & over :-) at run-time.
This reduces the library's size (and thus staticly linked binaries) and
has a minor speedup effect also.
Platforms tested:
IRIX64 6.5 (modi4) with parallel & FORTRAN enabled, and additional testing
on FreeBSD and Solaris immediately after the checkin.
Purpose:
turn on daylight feature on windows
Description:
currently we have to hard code daylight feature of windows to make it pass mtime test.
Solution:
Platforms tested:
linux 2.2.18
Purpose:
1. temporarily fixed mtime test when daylight saving changes on windows.
2. move H5_inline defination for win32 to windows H5pubconf.h at H5private.h.
Description:
1. the mtime test is not working when the daylight saving ends. Hard-coded back to the correct time. Need to find a universal solution later.
2. Move H5_inline defination for win32 at H5private.h into windows specificed H5pubconf.h so that windows maintenance can be more easy to handle.
Solution:
see above.
Platforms tested:
windows 2000, confirmed at Linux Red Hat 6.2.
Code cleanup for better compatibility with C++ compilers
Description:
C++ compilers are choking on our C code, for various reasons:
we used our UNUSED macro incorrectly when referring to pointer types
we used various C++ keywords as variables, etc.
we incremented enum's with the ++ operator.
Solution:
Changed variables, etc.to avoid C++ keywords (new, class, typename, typeid,
template)
Fixed usage of UNUSED macro from this:
char UNUSED *c
to this:
char * UNUSED c
Switched the enums from x++ to x=x+1
Platforms tested:
FreeBSD 4.4 (hawkwind)
Code cleanup (sorta)
Description:
When the first versions of the HDF5 library were designed, I remembered
vividly the difficulties of porting code from a 32-bit platform to a 16-bit
platform and asked that people use intn & uintn instead of int & unsigned
int, respectively. However, in hindsight, this was overkill and
unnecessary since we weren't going to be porting the HDF5 library to
16-bit architectures.
Currently, the extra uintn & intn typedefs are causing problems for users
who'd like to include both the HDF5 and HDF4 header files in one source
module (like Kent's h4toh5 library).
Solution:
Changed the uintn & intn's to unsigned and int's respectively.
Platforms tested:
FreeBSD 4.4 (hawkwind)
Code Cleanup
Description:
Changed POSIX function calls to the HD* way of doing things. Checked
with checkposix and tested on Linux.
Platforms tested:
Linux
Purpose:
a bug fix
Description:
forget considering daylight factor for mtime test on windows platform
Solution:
adding daylight factor in this file
Platforms tested:
windows 2000, confirmed at eirene
Update
Description:
Changed
#include <hdf_file.h>
construct to
#include "hdf_file.h"
so that the GNU compiler can more easily pick up the dependencies
which it places in the .depend and Dependencies files. Also
regenerated the Dependencies to go along with this.
Platforms tested:
Linux
Purpose:
a bug fix
Description:
delete the line for unused defined variables on windows.
Solution:
delete the line.
Platforms tested:
windows 2000
[machines you have tested the changed version. This is absolute
important. Test it out on at least two or three different platforms
such as Big-endian-32bit (SUN/IRIX), little-endian-32(LINUX) and
64-bit (IRIX64/UNICOS/DEC-ALPHA) would be good.]
Purpose:
make mtime test available on windows
Description:
calculating time zone difference on windows
Solution:
1.Use /#if WIN32 #endif/ to make sure the modification will not affect other platforms.
2. use <sys/types.h> and <sys/timeb.h> on NT to obtain time zone difference.
Platforms tested:
windows 2000
