Code cleanup/new features
Description:
Switch over to a new style for registering filters with the library -
instead passing in an ID, a string and a callback function to H5Zregister,
the client should pass in a single pointer to a H5Z_claass_t struct which
contains the ID, the description string and all the function callbacks as
fields.
Added support for a new "can apply" callback for each filter, which is
called when a dataset is created to check whether the parameters for that
filter apply correctly to the combination of the datatype and the chunk size
(i.e. dataspace) for the dataset.
Added support for a new "set local" callback for each filter, which
is called when a dataset is created (after the "can apply" filter callback)
and sets filter parameters that are specific to that particular dataset.
Switched the filters we ship over to use the new H5Z_class_t struct for
their internal registrations and also added "set local" callbacks to the
szip and shuffle filters and a "can apply" callback to the szip filter.
Lots of other code cleanups, etc. also
Solution:
Platforms tested:
FreeBSD 4.8 (sleipnir) w/szip
Linux 2.4 (sleipnir) w/szip
Solaris 2.7 (arabica) w/FORTRAN
IRIX64 6.5 (modi4) w/szip, FORTRAN & parallel
Misc. update:
Code cleanup & new features
Description:
Added three new API functions for working on I/O filters: H5Pmodify_filter,
H5Pget_filter_by_id and H5Pall_filters_avail. Documentation on these will be
added for the 1.6.0 release, until then, we'll have to use the source code.
Also, simplified the H5Pset_szip and H5Pset_shuffle API interfaces, since
those filters are using the new "set local" callback facility to set dataset-
specific parameters.
Probably, some of the changes to internal H5Z routines are captured in
these changes also.
Solution:
Platforms tested:
FreeBSD 4.8 (sleipnir) w/szip
Linux 2.4 (sleipnir) w/szip
Solaris 2.7 (arabica) w/FORTRAN
IRIX64 6.5 (modi4) w/szip, FORTRAN & parallel
Misc. update:
Code cleanup
Description:
Got rid of anonymous inline struct for filter information in a pipeline
and replaced with typedef'd type.
Solution:
Platforms tested:
FreeBSD 4.8 (sleipnir) w/szip
Linux 2.4 (sleipnir) w/szip
Solaris 2.7 (arabica) w/FORTRAN
IRIX64 6.5 (modi4) w/szip, FORTRAN & parallel
Misc. update:
Code cleanup
Description:
Array declaration was using hard-coded constant for maximum number of
dimensions.
Solution:
Changed to use H5O_LAYOUT_NDIMS.
Platforms tested:
FreeBSD 4.8 (sleipnir) w/szip
Linux 2.4 (sleipnir) w/szip
Solaris 2.7 (arabica) w/FORTRAN
IRIX64 6.5 (modi4) w/szip, FORTRAN & parallel
Misc. update:
New feature
Description:
Hooked up internal dataset creation call (H5D_create) to the new "can apply"
and "set local" callbacks for filters.
Solution:
Platforms tested:
FreeBSD 4.8 (sleipnir) w/szip
Linux 2.4 (sleipnir) w/szip
Solaris 2.7 (arabica) w/FORTRAN
IRIX64 6.5 (modi4) w/szip, FORTRAN & parallel
Misc. update:
code warrior port
Description:
added some type casts (char*) for CW portability
Solution:
Platforms tested:
w2000(CW , MS)
linux
arabica
modi4
Misc. update:
Code cleanup & bug fix
Description:
Cleaned up comments & copyright statement.
Blocks which were unable to be compressed were missing the "length" prefix,
causing the decompression routine to fail.
Solution:
Always add "length" prefix to all blocks, even ones the fail to compress.
Platforms tested:
FreeBSD 4.8 (sleipnir)
IRIX64 6.5 (modi4) w/parallel
Misc. update:
Code cleanup
Description:
Added lots of comments to existing filters, so they are easier to understand
and use as examples for future filters. Cleaned up various bits of code, etc.
Solution:
Platforms tested:
FreeBSD 4.8 (sleipnir)
IRIX64 6.5 (modi4) w/parallel
Misc. update:
Code cleanup (sorta)
Description:
The H5Pset_szip API call was forcing users to pass in their parameters
in the same way as the internal representation of the filter parameters used
by the library (as an array of unsigned values).
Solution:
Changed to pass in separate parameters in a more user-friendly format.
Platforms tested:
FreeBSD 4.8 (sleipnir)
IRIX64 6.5 (modi4) w/parallel
Misc. update:
Bug Fix
Description:
Was using the wrong pointer into a structure...
Solution:
Used the correct pointer to get the file_id...
Platforms tested:
Modi4
Misc. update:
Update & Bug Fix
Description:
The "free" protocol was missing. Added that to the server side.
When doing a "create" of a file (with no other data structures
created), the freespace in the file wasn't being reclaimed.
Solution:
After adding the free protocol, we put the burden of running through
the FD_free function on the SAP instead of each client.
Platforms tested:
Linux
Misc. update:
To support szip compression in HDF5
Description:
This is where szip filter function is located.
Solution:
The filter function composes of "encode" and "decode" part, which
is similar as deflate filter.
One critical difference is:
when szip decompresses the data, it needs to know the buffer size to hold
the decompressed data in advance.
Currently it's hard for HDF5 lib to give the buffer size easily. So to avoid
this problem, in each chunk, we add a small header to hold the buffer size of
each chunk. The code will use UINT32 ENCODE and UINT32 DECODE to finish this
part of work.
Platforms tested:
Since there are changes of configure.in and configure,I didn't use h5committest.
I tested with four platforms.
1) Linux 2.4 (eirene)
2) Solaris 2.7(arabica)
3) windows 2000(VS 6.0)
4) SGI IRIX6.5-64(modi4)
For test 1)-3), only basic C tests were done
For modi4 test, I tested 64-bit C,parallel and fortran.
All tests passed, except a warning message from szip library when checksum is used in some order, which doesn't
e any real problems.
Misc. update:
To support szip compression in HDF5
Description:
szip compression support is required by NASA ESDIS. The compression algorithm
is good for scientific data. In HDF5, we add another filter
function to make szip as a default compression package as we did for gzip(or zlib).
In this check-in, a new function called H5Pset_szip was added into H5Pdcpl.c, this
function is very similar to H5Pset_deflate. The only difference is SZIP needs
four parameters from the user to get the compression done while gzip needs only one.
So we pass a pointer to this function instead of an integer number.
The description of the four parameters of szip should be in different
documents.
H5Z.c and other header files were simply modified to have HDF5 know szip.
Solution:
See the decription
Platforms tested:
Since there are changes of configure.in and configure,I didn't use h5committest.
I tested with four platforms.
1) Linux 2.4 (eirene)
2) Solaris 2.7(arabica)
3) windows 2000(VS 6.0)
4) SGI IRIX6.5-64(modi4)
For test 1)-3), only basic C tests were done
For modi4 test, I tested 64-bit C,parallel and fortran.
All tests passed, except a warning message from szip library when checksum is used in some order, which doesn't cause any real problems.
Misc. update:
To support szip compression in HDF5
Description:
szip compression support is required by NASA ESDIS. The compression algorithm
is a good compression algorithm for scientific data. In HDF5, we add another filter
function to make szip as a default compression package as we did for gzip(or zlib).
H5config.in needs to be regenerated by autoheader for SZIP flags.
Solution:
A new file called H5Zszip.c will be added in the Makefile.in
Flags like: HAVE_FILTER_SZIP should be auto-generated by autoheader.
Platforms tested:
Since there are changes of configure.in and configure,I didn't use h5committest. I tested with four platforms.
1) Linux 2.4 (eirene)
2) Solaris 2.7(arabica)
3) windows 2000(VS 6.0)
4) SGI IRIX6.5-64(modi4)
For test 1)-3), only basic C tests were done
For modi4 test, I tested 64-bit C,parallel and fortran. All tests passed, except a warning message from szip library when checksum is used in some order, which doesn't
cause any real problems.
Misc. update:
Bug Fix & Update
Description:
FPHDF5 was creating files which didn't have the EOA field in the
superblock set correctly. It turns out that the SAP was keeping this
information to itself instead of giving it to the client processes.
Naughty SAP!
Solution:
Have the SAP send this information back to the clients so that they
can update the superblock as necessary. This now creates a file (with
just the root group) that looks correct! Only problem is that there's
extra file space being allocated. Also, at program termination,
there's an infinite loop...
Platforms tested:
H5committests (run by hand on burrwhite, arabica, and modi4)
Misc. update:
Bug fix/code cleanup/new feature
Description:
Fix h5debug to work correctly again, with all the changes over the past
few months.
Improved and cleaned up debugging output available in h5debug.
Platforms tested:
FreeBSD 4.7 (sleipnir)
Bug fix and Update
Description:
From Quincey's comments on the code I checked in last night:
- In H5F_close call, the "private" processes should call the
H5F_flush with the "CLEAR_ONLY" flag.
- There's no need for a special case for FPHDF5 in the
FD_real_alloc function since FPHDF5 doesn't define an alloc
function.
- The return type of H5Pset_fapl_fphdf5 should be herr_t instead
of hid_t. I don't know how it got that way in the first place.
- The variable names for MPI types and the structure typedefs
should be switched: H5FP_request/H5FP_request_t to
H5FP_request_t/H5FP_request and so on.
- In the H5FP.c module, I was commiting the H5FP_request MPI
datatype but using the wrong offset field...
Platforms tested:
Linux...will test on others, but these are mostly FPHDF5 changes.
Misc. update:
bug Fix
Description:
MPI stuff was getting into the non-parallel headers.
Solution:
#ifdefed the headers out if H5_HAVE_FPHDF5 isn't defined...
Platforms tested:
Sol, Modi4
Misc. update:
Update
Description:
When doing an allocation, check to see, if it's an FPHDF5
driver, if only the captain should be performing the allocation
Platforms tested:
Linux, Modi4, Sol
Misc. update:
Update
Description:
Have the FPHDF5 module initialized on startup. Have only the captain
process perform certain functions (like flushing during an F_close
call or allocating the superblock).
H5Fistore needed a few checks to see if it was working with an FPHDF5
driver. Done similarly to how MPI and MPIPOSIX drivers are checked..
Platforms tested:
Linux, Modi4, Sol
Misc. update:
Why not?
Description:
Lots of changes. This is basically a "commit because there are a lot
of changes" commit.
The server now acts more like a metadata caching server. The dumping
mechanism finally works (I think). At least in the test I did it
seemed to write things back. There's a new MPI TAG to use
(H5FP_TAG_DUMP) when dumping things. Changed a lot of the functions
from passing a structure around to just passing a pointer to that
structure. I have no clue why I did it the previous way...
Um...And more synchronization stuff between the client and server.
The protocol between the two was being broke in a couple of places.
We're getting closer!
Platforms tested:
Linux (probably Modi4 too)
Misc. update:
Bug Fix
Description:
The MPI datatypes we were committing weren't using the correct
structure.
Also, we needed to know the rank of the captain process inside of the
BARRIER COMM.
Solution:
Changed to use the correct structure.
Got the Captain process's rank and Bcast it to the "privates".
Platforms tested:
Linux
Misc. update:
Update
Description:
More changes to the FPHDF5 file driver. Too many to mention here.
Some highlights:
- Addition of lock/unlock methods to the driver, though they
aren't implemented fully just yet.
- Extra query functions to determine various things about the
file. (If this is the captain process, etc)
- Fixes to make things work...
Platforms tested:
Linux
Misc. update:
New Feature
Description:
Added support for the lock and unlock function calls for file
drivers. Only FPHDF5 uses this feature. All of these drivers don't
define lock or unlock methods.
Platforms tested:
Linux, Modi4, Sol
Misc. update:
Cleanup
Description:
The H5O_FPHDF5 object was left in this header file by mistake when
the H5Ofphdf5 module was removed
Solution:
Removed it.
Platforms tested:
Linux
Finish code cleanup
Description:
Wrap up the conversion of H5F_flush's multiple boolean flags into a single
bitfield of flags by pushing the flags down into the H5AC_flush and
H5F_istore_flush routines.
Also, changed the flags from H5_FLUSH_<foo> to H5F_FLUSH_<foo> to be more
consistent with rest of library.
And reverted the changes to H5FDflush and H5FD_flush routines.
Platforms tested:
FreeBSD 4.7 (sleipnir)
Solaris 5.8 (sol)
IRIX64 6.5 (modi4) w/parallel
Misc. update:
Refactoring of Flush Logic
Description:
The Flushing logic passed in multiple flags to indicate what type of
flush it was: closing, invalidate, alloc only. This made extending
the function of the flush logic to handle other flags annoying.
Solution:
I changed it to be just one bitmasked flag to indicate what type of
flushing to do. I also added the CLEAR_ONLY flag, which will be used
in the FPHDF5 stuff.
Platforms tested:
h5committest doesn't work for me (my environment isn't setup on the
other machines I guess). I tested it manually:
Linux parallel & C++
Sol Fortran
Modi4 parallel & Fortran.
Potential bug fix.
Description:
The code to rotate the metadata writing process may not be compiled
correctly in all cases.
Solution:
Change from using "++mpi_round" to "mpi_round+1"
Platforms tested:
Eyeballed...
New internal feature
Description:
Add internal API for building and working with heaps (H5HP). This will be
used for the LRU algorithm in the new metadata cache code.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir)
Code/comment cleanup
Description:
Made more information about reference-counted strings private to the
H5RS package by trimming the typedef used in H5RSprivate.h and switching
from using macros for a couple of operations to using functions.
Updated copyright information for a few of these files.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
Bug Fix
Description:
Private header file was included in a public header file by mistake.
Solution:
Put the private header file into the .c module instead.
Platforms tested:
Linux 2.4 (see, I really did test it).
Update
Description:
Changed the H5FPinit() function to accept pointers to what the SAP
Comm and SAP Barrier Comm should be assigned to. This is the first
step in hopefully getting rid of the global versions of them
altogether (but we'll see).
Platforms tested:
Linux
Update
Description:
Final support addition for FPHDF5. Wherever there was an equivalent
MPIO or MPIO/POSIX call/check, I placed an FPHDF5 call/check there as
well, with the appropriate #ifdefs in place.
The hdf5.h is updated with the H5FDfphdf5.h driver #include.
Platforms tested:
Linux & Modi4
