* 2000-10-31 Robb Matzke <matzke@llnl.gov> (H5F_dest)
Do not call H5FL_FREE() if the root group
object is null. This fixes a bug in h5ls when that tool is given
a file which is not an hdf5 file.
Bug fix
Description:
There is typo in the H5D_write function which reported the
optimized write failure as a H5E_READERROR.
Solution:
Replaced it with the correct H5E_WRITEERROR code.
Platforms tested:
modi4 parallel (compiled H5D.o only).
Purpose:
Detect whether the system has the netinet/tcp.h header file.
Description:
The Stream VFD uses setsockopt(2) to set TCP_NODELAY on an opened
socket. This constant is defined normally in netinet/tcp.h
except for GNU cc under Windows where this header doesn't exist.
This template header file just contains the
#undef HAVE_NETINET_TCP_H
Platforms tested:
Windows NT, GNU cc
Purpose:
Port to Windows platform
Bug fix
Description:
The Stream VFD is ported to Windows now.
Also fixed a bug where an application terminated when it got a SIGPIPE
due to sending/receiving on a closed socket.
Solution:
The socket stuff is treated different under Windows when using
the MS compilers to build HDF5. They define their own socket datatype
and have closesocket() instead of close(2) to close sockets.
Also there are different header files for all the socket stuff.
So I introduced my own socket decriptor datatype in H5FDstream.h
which should be used to pass in external sockets. This datatype
is mapped to either 'int' (UNIX-type sockets) or 'SOCKET' (Windows).
In the code the error code checking was adapted according to the
socket datatype used. Also, for Windows you have to call a routine
to initialize the Socket layer before using it.
As a kind of bug fix, the process signal mask is now set to ignore
SIGPIPE signals which otherwise cause the application to terminate.
The driver read/write routines catch this and return an error code.
Platforms tested:
Windows NT, both with MS Visual C++ compiler and with GNU cc
It is interesting that when compiling with GNU cc under Windows
it is possible to use both Windows and UNIX-type sockets (either
one or the other). So I check for GNU cc and go for UNIX sockets
if possible.
Bug fix
Description:
Forgot to decrement the number of objects in the group when removing
each one...
Solution:
Decrement the ID count when successfully removing each object from a
group.
Platforms tested:
None yet.
Bug fix
Description:
Fixed a very subtle bug that was corrupting the objects in a group when
H5I_clear_group was called and not all the objects in the group were able
to be cleared.
Solution:
Track whether an object as been deleted from each hash location's linked
list and only destroy the list when all the objects on the list are
actually removed.
Platforms tested:
None yet.
Code optimization
Description:
Minor tweaks throughout the optimized regular hyperslab code to increase
speed. This set of improvements increase the benchmark time from taking
~5.46 seconds to ~4.50 seconds, or around a 20% further speedup.
Platforms tested:
Solaris 2.6 (baldric)
Bug fix
Description:
Recent changes to H5FD_read() added a new argument of type of
data to be written but the MPIO file driver call to H5FD_read()
was not updated. Also, the prototype of H5FD_read() in H5Fprivate.h
was "screened out", thus the compiler could not detect the inconsistency.
With the mismatched arguments, MPIO failed badly.
Solution:
Update the H5FD_read() call with the new parameter. Since the
call is used by H5Dread call so far, it is hardcoded to use
H5FD_MEM_DRAW as the value. If the call is used besides for
H5Dread, this parameter needs to be better defined.
(Still need to fix the prototype being blocked off.)
Platforms tested:
O2K -64 parallel.
Feature symmetry
Description:
A while ago I needed to get the 'type' of data being accessed during writes
to the VFL driver, so I put in code to get the information down there.
Albert asked for the same information during reads, so I've added that in.
Tested:
FreeBSD 4.1.1 (hawkwind)
Increase constant for H5S_MAX_RANK from 31 -> 32
Description:
To better align the HDF5 library's maximum dimenions with HDF4 and netCDF,
the maximum number of dimensions has been increased to 32.
Platforms Tested:
FreeBSD 4.1.1 (hawkwind)
Solaris 2.5 (baldric)
Bug fix.
Description:
When the maximum number of elements in a fixed size, free-list managed array
was allocated, the free-list code was writing off the end of the array.
Solution:
Changed definition of free-list managed arrays to allocate correct space for
the array.
Platforms tested:
Solaris 2.5 (baldric) w/Purify
Optimization for parallel I/O
Description:
When contiugous hyperslabs are defined (i.e. with the block=stride), the
library was only aggregating the hyperslabs together for the fastest
changing dimension.
Solution:
Add some extra code to detect when contiguous hyperslabs span more than one
row and output the entire contiguous section at once.
Platforms tested:
FreeBSD 4.1.1 (hawkwind)
Code cleanup
Description:
There's been a compiler warning about modifying a 'const' object in the
H5T_copy routine for several years now, which a few users have reported.
Solution:
Removed call to sort the old datatype (the const object) and changed the
code to work with unsorted datatypes.
Platforms tested:
FreeBSD 4.1.1 (hawkwind)
Bug fix
Description:
"Time" datatypes (H5T_UNIX_D*) were not being stored and retrieved in
the datatype object header message correctly.
Solution:
Store endian-ness and precision in the datatype object header message and
added test to continue to track them working correctly.
This fixes bug #512.
Platforms tested:
FreeBSD 4.1.1 (hawkwind)
Bug fix
Description:
H5Sget_simple_extent_npoints return type is hsize_t and the error value is
0, but it is possible to have 0 points in a dataset with an unlimited
dimension, but no data written yet.
Solution:
Changed H5Sget_simple_extent_npoints return type to hssize_t and the error
value to -1
Platforms tested:
FreeBSD 4.1.1 (hawkwind)
Bug fix (bug #446)
Description:
H5Tget_offset return type was size_t and the error return value was 0, but
0 is also a valid successful return value.
Solution:
Changed the return type to int and the error return value to -1
Platforms tested:
FreeBSD 4.1.1 (hawkwind)
Bug fix.
Description:
Previously, it has been possible to dereference deleted objects in a file.
Obviously, this is incorrect and could cause all sorts of problems if the
object being dereferenced had been partially over-written with other
information. - This is documented in Bug #493.
Solution:
Check the link count for objects being dereferenced and don't allow any
objects with link counts of zero to be dereferenced.
This fixes bug #493.
Platforms tested:
FreeBSD 4.1.1 (hawkwind)
Purpose:
Just add a few lines so that it will not break windows platform.
Description:
netdb.h cannot be recognized by windows, use winsock.h in windows instead.
Solution:
adding the following lines at the beginning of H5FDstream.c.
If you don't like, you may do it in another way, just keep it work for windows.
#ifdef WIN32
> #include <winsock.h>
> #else
> #endif
Platforms tested:
I am only testing whether it will bother building objects on windows. I haven't tested whether it
will affect the testing results. Hopefully not.
windows 2000, eirene,arabica,gondolin,paz,opus.
Purpose:
Parallel Bug Fixes
Description:
Was out of sync with header file re-arrangements I checked in last night.
Solution:
Fixed to use new header files, etc.
Platforms tested:
O2K (modi4)
Bug Fix
Description:
When parallel I/O is turned on, there were some macros used in the H5D
routines which poked around in the H5F_t structure. This breaks the
privacy of that structure and ties the H5D code too tightly to the H5F_t
struct.
Solution:
Added a small function to retrieve the the value (driver_id) needed from
the H5F_t function.
Platforms tested:
Eyeballed only, Albert needs this right away...
Maintainance & performance enhancements
Description:
Re-arranged header files to protect private symbols better.
Changed optimized regular hyperslab I/O to compute the offsets more
efficiently from previous method of using matrix operations.
Added sequential I/O operations at a more abstract level (at the same level
as H5F_arr_read/write), to support the optimized hyperslab I/O.
Platforms tested:
Solaris 2.6 (baldric) & FreeBSD 4.1.1 (hawkwind)
Bug fiX
Description:
H5S_hyper_select_valid would report hyperslab invalid if the one of
the count values is zero. The verifying algorithm did not take into
consideration that block or count can contain zeros to indicate no
element is wanted.
Solution:
Added code to test if block or count is zero. If so, skip the rest
of the validity check.
Platforms tested:
IRIX64 -64.
Bug fix (done by Kim Yates)
Description:
The optimized mpio code was broken and when read was done, it hanged.
Solution:
H5FDmpio.c:
In H5FD_mpio_write, moved the 16-line block of code in which
all procs other than p0 skip the actual write
to be just before the call to MPI_File_write_at.
Previously, the values of the local vars that controlled
"allsame" were not always set correctly when the moved block
was reached.
H5S.c:
Changed default value of H5_mpi_opt_types_g to TRUE, so that
the MPI-IO hyperslab code is executed by default in parallel HDF5,
rather than executing the serial hyperslab code.
H5Smpio.c:
In function H5S_mpio_hyper_type, added a call to free
an intermediate type. Cures a small memory leak.
Added code for cases of empty hyperslab
Changed displacements to be MPI_Aint
Platforms tested:
modi4 -64: worked fine with mpich 1.2.0 but failed with the messages
saying it ran out of entries for MPI_Types during the collective_read
test. After tracing the code all the way to the collective read, all
MPI Types have been freed properly. It aborted with the above message
when it executed the line
if (MPI_SUCCESS!= MPI_File_read_at_all(file->f, mpi_off, buf, size_i, buf_type, &mpi_stat ))
Could not see any problem with this line. It could be a bug in the
SGI version of MPI.
Purpose:
[is this a bug fix? feature? ...]
Description:
[describe the bug, or describe the new feature, etc]
Solution:
[details about the changes, algorithm, etc...]
[Please as detail as you can since your own explanation is
better than others guessing it from the code.]
Platforms tested:
[machines you have tested the changed version. This is absolute
important. Test it out on at least two or three different platforms
such as Big-endian-32bit (SUN/IRIX), little-endian-32(LINUX) and
64-bit (IRIX64/UNICOS/DEC-ALPHA) would be good.]
Bug Fix
Description:
Use H5FD_get_eoa instead of H5FD_get_eof to check for reading off the end
of the allocated file space. Using H5FD_get_eof was causing the Stream
VFD to fail.
Solution:
Switched from using H5FD_get_eof to H5FD_get_eoa
Platforms tested:
FreeBSD 4.1.1 (hawkwind)
Purpose:
Bugfix
Description:
The Stream VFD was leaking memory on every opened file.
Solution:
In H5FD_stream_close(), finally free the file structure used to describe
the closed file.
Platforms tested:
Linux, SGI
Rearrange code
Description:
The data sieve buffering code for contiguously stored datasets was
wedged in the H5F_arr_read/H5F_arr_write routines.
Solution:
Created a new H5Fcontig.c to hold I/O routines for contiguously stored
datasets (like H5Fistore.c for chunked dataset I/O routines) and moved
data sieving code into those routines.
Platforms tested:
Solaris 2.6 (i.e. baldric)
Code Optimization.
Description:
The optimized routines for copying regular hyperslabs in memory have been
using the same matrix routines to copy their hyperslab pieces as the
routines for irregularly shaped hyperslabs. This ends up imposing lots of
extra overhead on the optimized routine, since it basically "knows" all the
matrix information it needs.
Solution:
Keep track of the [small] amount of matrix information necessary to perform
the regular hyperslab copies in the optimized routines themselves instead of
using the matrix routines. This improves the performance for the benchmark
I'm running from ~18 seconds to ~12 seconds and should apply to parallel
I/O situations also.
Platforms tested:
Solaris 2.6 (i.e. baldric)
Code Optimization
Description:
The matrix operations are currently the hot-spot in the library code
for regular hyperslab operations.
Solution:
Unrolled loops for 3 of the more heavily used functions
(H5V_stride_optimize2, H5V_hyper_stride & H5V_hyper_copy) for the common
cases (i.e. up to 3-D datasets). This squeezes some more blood out of
the stone (turnip? :-) and improves the h5hypers.c benchmark on baldric
by another 20-25%.
Platforms tested:
Solaris 2.6 (i.e. baldric)
Bug Fix
Description:
The core and log VFL drivers were leaking small amounts of memory when they
were used.
Solution:
Free the appropriate memory block (for the core driver) and don't allocate
a block (for the log driver).
Platforms tested:
Solaris 2.6 (i.e. baldric)
Implemented new feature
Description:
Added data sieve buffering code to raw I/O data path. This is enabled for
all the VFL drivers except the mpio & core drivers. Also added two new
API functions to control the sieve buffer size: H5Pset_sieve_buf_size() and
H5Pget_sieve_buf_size().
Platforms tested:
Solaris 2.6 (i.e. baldric)
Fix compiler warning
Description:
"HUGE_VAL" (a double value) was being put into a float type and generating
a warning during compile time.
Solution:
Replaced "HUGE_VAL" with "FLT_MAX"
Platforms tested:
FreeBSD 4.1
Small Code Cleanup
Description:
Code to optimize adjacent (i.e. contiguous) hyperslab was ugly and used too
many temporary variables.
Solution:
Computed the optimized hyperslabs slightly differently and got rid of
unnecessary temporary variables.
Platforms tested:
FreeBSD 4.1
Bug fix (sorta)
Description:
When the stride and block size of a hyperslab selection are equal, the
blocks that are selected are contiguous in the dataset. Prior to my
hyperslab optimizations, this situation used to be detected and somewhat
optimized to improve performance. I've added more code to optimize for
this situation and integrated it with the new hyperslab optimization that
weren't very efficient for that case as they should have been.
Solution:
Detect contiguous hyperslab selections (i.e. block size in a dimension is
the same as the stride in that dimension) and store the optimized,
contiguous version of that hyperslab. We also store the original, un-
optimized version of the hyperslab to give back to the user if they query
the hyperslab selection they just made.
Platforms tested:
FreeBSD 4.1
Bug Fix
Description:
The prototype for the H5Pregister function has a variable named
`class'. This is a reserved word in C++ and causes the C++
compiler to freak.
Solution:
This variable's name was changed to cls_id in the .c file, so I
changed it in the header file to cls_id to match.
Platforms tested:
Linux
Bug Fix.
Description:
An assertion in the local heap code was mistakenly checking against too
large of a value for the size of new local heap created. When used with
larger-sized (>10KB) variable-length objects, it was failing the check.
Solution:
Corrected to check against the actual size of the heap allocated, without
the heap header.
Platforms tested:
FreeBSD 4.1
Restore file
Description:
It appears that Robb's checkin earlier today erroneously overwrote this
file with an older version... *grumble*
Solution:
Found another copy of newest version, verified that it is operating
correctly and re-checked it in.
Platforms tested:
FreeBSD 4.1
H5FDstream.h needs to be installed.
Description:
H5FDstream.h is included in the hdf5.h file and needs to be
installed with the other public headers.
Solution:
Added it to the rest of the install headers.
Fix Irix pmake bugs
Description:
Build fails on Irix when builddir != srcdir
Solution:
* acconfig.h
* src/H5config.h.in [REGENERATED]
Added definition for HAVE_STREAM
* config/conclude.in
* config/depend1.in
* config/depend2.in
* config/depend3.in
* config/depend4.in
The `Dependencies' file is located in the source
tree. This fixes bugs for Irix pmake when compiling
outside the source tree. Hopefully it still preserves
Albert's changes which allow concurrent compilations
to not stomp on each other's Dependencies files.
* examples/Dependencies [REGENERATED]
* src/Dependencies [REGENERATED]
* test/Dependencies [REGENERATED]
* tools/Dependencies [REGENERATED]
Regenerated for testing purposes.
Platforms:
i686-pc-linux
mips-sgi-irix6.5
sparc-sun-solaris2.6
Add the Stream VFD sources to the appropriate makefile variables.
Description:
Added H5FDstream.c to the LIB_SRC variable and H5FDstream.h
to the PUB_HDR variable for building the Stream VFD.
Define HAVE_STREAM.
Description:
If the Stream VFD was configured the configured script
will expand this into
'#define HAVE_STREAM 1' in H5config.h and
'#define H5_STREAM 1' in H5pubconf.h.
Added source files for the Stream Virtual File Driver.
Description:
The Stream VFD allows users to stream complete HDF5 files
via socket connections between different applications.
Files which were created anew are flushed to any connected client
on each H5Fflush() or H5Fclose() operation.
Files which are opened as read-only will be read from a socket
on a H5Fopen() call.
The driver's H5FDset_fapl_stream() routine allows to pass in
several parameters such as an external socket descriptor,
some socket options, and flags for broadcasting a received file.
If an external socket is provided the Stream VFD would use that
for the socket calls. Otherwise it parses the filename argument
in H5Fcreate()/H5Fopen() for a 'hostname::port' parameter.
All files processed by the Stream VFD are kept in memory
(same way as the core VFD does).
Platforms:
Tested so far under Linux, Irix 32/64bit, OSF1, Solaris, Cray Unicos,
Hitachi SR8000, IBM AIX.
Not tested under Windows yet.
Fix last couple of errors from introducing "regular" hyperslab feature
into the library.
Description:
Code was blindly dereferencing data structures which aren't defined when
operating on regular hyperslabs.
Solution:
Check for regular hyperslab defined and retrieve information from regular
hyperslab info instead of mucking about in other hyperslab information.
Platforms:
Solaris 2.6
