Repair synchronization bug in the metadata cache in PHDF5
Also repair numerous other bugs that surfaced in testing the
bug fix.
Description:
While operations modifying metadata must be collective, we allow
independant reads. This allows metadata caches on different processes
to adjust to different sizes, and to place the entries on their dirty
lists in different orders. Since only process 0 actually writes
metadata to disk (all other processes thought they did, but the writes
were discarded on the theory that they had to be collective), this made
it possible for another process to modify metadata, flush it, and then
read it back in in its original form (pre-modification) form. The
possibilities for file corruption should be obvious.
Solution:
Make the policy that only process 0 can write to file explicit, and
visible to the metadata caches. Thus only process 0 may flush dirty
entries -- all other caches must retain dirty entries until they are
informed by process 0 that the entries are clean.
Synchronization is handled by counting the bytes of dirty cache entries
created, and then synching up between the caches whenever the sum
exceeds an (eventually user specified) limit. Dirty metadata creation
is consistent across all processes because all operations modifying
metadata must be collective.
This change uncovered may bugs which are repaired in this checkin.
It also required modification of H5HL and H5O to allocate file space
on insertion rather than on flush from cache.
Platforms tested:
H5committest, heping(parallel & serial)
Misc. update:
new features
Description:
add support for compiling the library and testphdf5 in Windows
Solution:
Platforms tested:
Linux
AIX
Solaris
Windows VC6
Misc. update:
Feature.
Description:
Added ALARM_ON, ALARM_OFF feature to terminate tests that run more than
the default time limit. So far, the feature is applied in the standard test
frame.
Platforms tested:
heping PP.
New feature.
Description:
Added the time command to the make check target to report time usage
of the execute of each test and test scripts. This gives us some idea
how long each test takes and some vague idea it is compute bound or
not.
powerpc-ibm-aix5.x:
Change $RUNPARALLEL default setting to allow it being invoked by the
time command.
Platforms tested:
h5committested.
Description:
On VMS dsets test failed screaming about corrupted stack.
Debugging didn't show any obvious problems. Careful examination
of compilation warning messages (something like "function is used as
data in dsets") made me to search for duplicated names.
It turned out that two variables H5Z_SET_LOCAL and H5Z_CAN_APPLY
were confused with H5Z_set_local and H5Z_can_apply functions from
H5Z.c file.
Solution:
Changed variable names to H5Z_SET_LOCAL_TEST and H5Z_CAN_APPLY_TEST.
Platforms tested: Alpha VMS, heping; minor change
Misc. update:
Feature
Description:
Added H5_CFLAGS, etc. to 1.7 branch.
Now compilation flags can be put in H5_*FLAGS and they'll be used when
building hdf5 but not in h5cc.
Platforms tested:
mir, sleipnir, modi4
Misc. update:
Bug fix
Description:
Disabled C++ shared libraries for Sun Workshop compiler.
Solution:
This bug only seems to happen when using the -xarch=v9 flag to compile in
64-bit mode, but disabling shared libraries entirely for this compiler is
an easier fix (I don't know how to detect 64 bit mode from the command line).
The framework for disabling shared libraries for other C++ compilers is
in place.
Platforms tested:
sol, mir, sleipnir, modi4
Bug fix/feature
Description:
Disabled shared libraries for a number of Fortran compilers that don't
support them.
This allows other compilers to support shared Fortran libraries.
Solution:
Added a conditional, SHARED_FORTRAN_CONDITIONAL, which is true if
Fortran supports shared libraries. It is set in configure.in.
Platforms tested:
mir, sleipnir, colonelk, heping
Description: H5T_register() replaces any existing data conversion path
with a new path.
Solution: Added a parameter to H5T_register() and H5T_path_find() to
signal the library whether the new conversion path is from API function
H5Tregister() or from private function like H5T_init_interface(). If it
is from H5Tregister(), replace existing path. If it's from H5T_init_interface()
and the library is trying to register default hard conversion functions,
don't replace existing path because the path is registered by the library.
For example, the library registered H5T_conv_int_float() and is trying to
register H5T_conv_long_float(). The size of long is equal to int. There's
no need to replace the path H5T_conv_int_float().
Platforms tested: h5committest and fuss.
Description: The API function H5Tis_hard() which was recently checked
in has been renamed to H5Tcompiler_conv() to be more descriptive. No
other changes.
Platforms tested: fuss - simple change.
Misc. update: config/lt_vers.am to update libtool version.
Bug fix
Description:
Building with --disable-static seems to have been broken on several platforms.
Fixed this.
Configure will now disable shared libraries automatically when using pgf90
or when building on Cygwin.
Solution:
To avoid errors when using shared libraries with pgf90 (which I had not
realized were compiler-specific), I had manually set convenience libraries
to use the -static linking flag. Apparently, this is not necessary, and
caused these libraries to be created as empty archives when --disable-static
was used.
Also, some libraries were including other libraries, which was not
necessary.
I also fixed code in configure.in to correctly detect whether shared
or static libraries are enabled, and moved code that disables shared libraries
to before libtool is created (rather than editing libtool after the fact).
Despite repeated warnings that only one shared library can be linked at a
time, I have yet to have shared libraries break the linking of tests on
any system. We'll see if the Daily Tests turn up anything.
Platforms tested:
mir (fortran, C++), sleipnir (C++), modi4 (fortran, C++, parallel),
sol (fortran, C++), cygwin
Description: Brought back changes from SX-6.
Solution: Tests will not compile. Fixed.
Platforms tested: SX-6, heping and mir with new g95
Misc. update:
Description: Quincey added the condition branch (else if (*((ST*)S) != (ST)((DT)(*((ST*)S)))))
in the definition of the macro H5T_CONV_Fx_CORE. It handles a special situation
when the source is "float" and assigned the value of "INT_MAX". Compilers do roundup
making this value "INT_MAX+1". This branch is to check that situation and return
exception for some compilers, mainly GCC.
The branch if (*((ST*)S) > (DT)(D_MAX) || (S_PREC < D_PREC && *((ST*)S) ==
(DT)(D_MAX))) is for some compilers like Sun, HP, IBM, and SGI where under
the same situation the "int" doesn't overflow.
A test was added to dt_arith.c to check 2 conversions, from float to int where the
float is INT_MAX, and from double to signed char where the double is SCHAR_MAX.
Platforms tested: h5committest and fuss.
Code cleanup/reorganization
Description:
Merge back some more changes extracted from the "compact group" set.
This bunch cleans up and prepares the H5G_* routines for eventual import of
new features.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4
Mac OS X.4
Code cleanup
Description:
Merge back changes from "compact group" work that improve the
infrastructure of the library and may impact others. In this round of
merging, that includes:
- Move datatype allocation into single internal routine, instead of
duplicated code that was spread out in a dozen or so places.
- Clean up guts of object header routines (H5O_*) to allow for some of
the fancieroperations that need to be performed on groups, along with
some general improvements.
- Added a new error code
- Some minor cleanups in other code....
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4
Mac OS X
Bug fix
Description:
Incorrectly terminated the received string in getenv_all
Solution:
Tasks would incorrectly terminat the environment string they received from the
root task in getenv_all.
They did the equivalent of:
env[strlen(str) + 1] = '\0'.
This resulted in a single "garbage" character inserted
at the end of the string. This wasn't noticed until now because it seems that on most
platforms this garbage character was a 0 anyway.
This has been corrected to;
env[strlen(str)] = '\0'.
Platforms tested:
Red Storm (where the bug was discovered)
heping (pp)
Misc. update:
Description: Put in a new API funciton, H5Tis_hard. It checks whether
the conversion function from a native type to another native type is a
compiler (hard) conversion. Also checked a test in test/dt_arith.c.
Platforms tested: h5committest and fuss.
Bug fix & code cleanup
Description:
Address most of datatype conversion exception handling bug that Ed
Hartnett reported. (He's reported a different problem now, but we're closer
at least).
Also, condense exception handling #ifdef's into one location instead of
spread out in so many places.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Too minor to require h5committest
Code cleanup
Description:
Changed some scale+offset filter symbols from H5_SO_... -> H5Z_SO_...
to indicate that they are in the H5Z package.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Too minor to require h5committest
update dsets.c for scale-offset filter.
Description:
H5Psetscaleoffset has changed parameters.
Solution:
Platforms tested:
sol 2.7, Linux 2.4, AIX 6.4(32-bit and 64-bit)
Misc. update:
tidy up.
Description:
MPE option created *.clog file whenever it is executed in MPI.
The cleanup of *.clog files were done in individual Makefile.in.
Often, it is forgotten.
Solution:
Moved the cleaning of *.clog files to CHECK_CLEANFILES in commence.am
so that it is applied whenever check-clean is called.
Platforms tested:
heping pp using MPE.
Bug fixes
Description:
This checkin fixes an occasional error on kelgia on sol during distclean.
It also causes test scripts to depend properly on the programs they're
supposed to be testing.
Solution:
The kelgia bug was due to some files being cleaned by automake and manually.
Removed the manual cleaning in src/Makefile.am.
Test script dependencies now need to be specified manually, since the
makefile can't guess what they test from their name. Currently all test
scripts in a given directory have a single list of dependencies--this was
easy and seems to be sufficient.
These dependencies are listed in the SCRIPT_DEPEND variable in the Makefile.am.
Platforms tested:
heping, mir, modi4, sol
Misc. update:
Description: Added 2 new configure options, --enable-exception and --enable-accuracy.
--enable-exception lets the library check whether user's exception handling
functions are present during compiler data conversions and use them if they are.
When it's disabled, this step is skipped to improve conversion speed. This
step isn't implemented yet for soft conversions because there would be little
gain in speed.
--enable-accuracy guarantees data accuracy during data conversions. It means
the library will choose compiler conversions only if the accurate data is
secured. Otherwise, the library will go for the library's own conversions. If
this option is disabled, the library uses compiler conversions in favor of
their speed as long as they work even if data can be incorrect.
Platforms tested: h5committest and fuss. Some systems may fail after this
checkin.
Code cleanup
Description:
Clean up internals of group creation & iteration code.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Mac OS X (nile)
Too minor to require h5committest
Code cleanup (sorta)
Description:
Adjust H5G_stat_t to prepare for eventual "external link" information.
This also puts the information for regular objects and soft links into more
obviously separate places.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4
minor tidy up.
Description:
Changed both test programs and test scripts to use the same suffixes
(.chkexe and .chklog). Changed from .log to .chklog to avoid running
into potential conflicts by other "things" (e.g., config.log).
Tested:
Heping.
Code cleanup
Description:
Prevent memory leak when too many links are traversed. Also add test to
traverse too many links.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4
Improvement for Windows
Description:
thread number can not be printed out on windows. Add a special
macro for WIN32 to print out some helpful messages
Solution:
Platforms tested:
Heping and MSVS 6.0 on Windows XP
Misc. update:
Bug fix
Description:
Fix "strong" file closing to handle a dangling dataset which uses a
dangling named datatype.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Too minor too require h5committest
Code cleanup
Description:
Clean up code somewhat:
- Move from HDmemset() -> H5G_ent_reset() to clear out group entry info
- Simplify H5G_unlink() call
- Use portability macros instead of direct system calls in more places
- Improve readbility by neatening whitespace, etc.
- Move some macros into source code module instead of header files
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4
New debugging feature
Description:
Add some code to track where memory from the internal free list allocator
is allocated within the library. It's not quite ready for "prime time" yet,
but it's close enough to be useful.
This is disabled by default and requires the H5FL_TRACK macro in
src/H5FLprivate.h to be uncommented to activate during debugging.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4
Too minor to require full h5committest
Bug fix
Description:
Fix core dump when checking whether to invalidate the name of an object
when unlinking an object in a group opened through an object reference.
Solution:
Check if names of various objects are valid before comparing them, etc.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Too minor to require h5committest
Code cleanup/bug fix
Description:
Add missing files to manifest
Remove C++ style single-line comment from recent checkin and also re-indent
in a few places.
Platforms tested:
FreeBSD 4.11 (sleipnir) w/thread-safe
Not configured w/h5committest
Improvement
Description:
ttsafe cancel failed on Windows because pthread mutex and cond are not initialized
Add some assertion statements to locate the errors when they happen
Solution:
Initialize pthread mutex and cond to the default attributes.
Platforms tested:
MSVS 6.0 on windows XP and heping(with pthread)
Misc. update:
Improvement
Description:
Add some assertion statements to locate the errors when they happen
Solution:
Platforms tested:
MSVS 6.0 on windows and heping(with pthread)
Misc. update:
Improvement
Description:
Add some assertion statements to locate the error when errors happen
Solution:
Platforms tested:
MSVS 6.0 on windows XP and heping(enable threadsafe with pthread)
Misc. update:
Code cleanup
Description:
Trim trailing whitespace, which is making 'diff'ing the two branches
difficult.
Solution:
Ran this script in each directory:
foreach f (*.[ch] *.cpp)
sed 's/[[:blank:]]*$//' $f > sed.out && mv sed.out $f
end
Platforms tested:
FreeBSD 4.11 (sleipnir)
Too minor to require h5committest
Code cleanup
Description:
Collapse the various tests on groups opened by dereferencing into one test.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Too minor to require h5committest
Big fix
Description:
A group opened by dereferencing a object reference would not work for
H5Giterate(), due to the local heap & B-tree information not being cached.
Solution:
Get the local heap & B-tree info & point to that structure instead of
the group entry for the group.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Too minor to require h5committest
Bug fix & code cleanup
Description:
Fix another bug in the file mounting code and refactor the unmount
code that it is simpler.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4
Too minor to require h5committest
Bug fix and improvement.
Description:
The test output were not displayed for parallel tests.
The serial tests output were always displayed whether the test was
actually ran this time or not.
Solution:
Moved the display of test output inside the target where the actual
tests are run.
Platforms tested:
Tested in heping using pp and fortran.
Bug fix and code minor cleanup.
Description:
The attempt to redirect stderr together to the log files were
done in the wrong order such that stderr output goes to where
stdout WAS. Fixed that.
Code minor cleanup--use shell variables to hold the name of the
log file and the test instead of generating them repeatedly. This
makes easier code reading, less crowded and avoid typos.
Only conclude.in is changed. The Makefile.in changes are all
derived from automake.
Platforms tested:
h5committested.
Makefile feature
Description:
'make check-clean' now cleans *.h5 files created by tests as well as
.log and .chkexe files.
Solution:
check-clean is now a separate target in its own right, which cleans less
than mostlyclean (check-clean < mostlyclean < clean < distclean).
Platforms tested:
mir, heping, modi4 (serial and parallel)
Feature: check-clean target
Description:
'make check-clean' cleans up output files from tests.
Solution:
Tests create foo.chkexe and foo.log files. Scripts create foo.chksh and
foo.logsh files. 'make check-clean' will clean these files up so that
the tests can be re-run.
Also suppressed some not-very-useful output of Makefiles when it would
echo commands.
Platforms tested:
mir, sleipnir, modi4
Misc. update:
Bug fix
Description:
Correct problems when querying information about a group that was opened
by dereferencing an object reference.
Solution:
Read in symbol table information instead of rely on it being cached.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Too minor to require h5committest
Bug fix
Description:
If a named datatype is copied and the copy is used to create a dataset,
the dataset would inadvertantly refer to the original named datatype instead
of a local (possibly modified) copy of the named datatype.
Solution:
Fixed datatype copying routine.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Too minor to require h5committest
