HDFFV-8214 - h5repack failed converting small chunked dataset (size < 1K) to contiguous layout.
Description:
h5repack failed converting small chunked dataset (size < 1K) to contiguous layout.
The first case was when chunk dim is bigger than the dataset dim (at leat one), h5repack failed with displaying error stacks.
The other case is when chunk dim is smaller than the dataset dim, h5repack failed to change layout.
Merged from HDF5 trunk 23407
Tested:
jam (linux32-LE), koala (linux64-LE), ostrich (linuxppc64-BE), emu (solaris-BE), fred (mac64-LE), Windows (32-LE cmake), cmake (jam)
HDFFV-8012 - h5repack changes max dims and cause failure if only "-f none" is used without changing layout for chunked dataset when a chunk dim is bigger than a dataset dim
Description:
"h5repack -f <obj>:NONE <file.h5> out.h5" command failed if source file contains chunked dataset and a chunk dim is bigger than a dataset dim.
Another issue is that the command changed max dims if chunk dim is smaller than the dataset dim.
These issue occurred when dataset size is smaller than 64k (compact size limit)
Fixed them.
Merged from HDF5 trunk r22805
Tested:
jam (linux32-LE), koala (linux64-LE), ostrich (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Windows (32-LE cmake), cmake (jam)
Convert configure.in to configure.ac.
Align TESTS usage with autotools convention.
Update all references from configure.in to configure.ac
Tested: h5committest
HDFFV-8107 testh5diff will fail if build/test in HDF5 source tree
Description:
This is sub-task for "HDFFV-8105 testh5diff.sh uses the wrong operator (-a) in an if statement."
From the HDFFV-8105's update, h5diff test failed if build&test is performed in HDF5 source tree because 'cp' try to copy test files to self dir.
It's addressed by skipping if cp's src dir and dest dir is same.
Also this applied for all other tools under src/tools dir.
No change to the CMakeLists.txt files because CMake cautions/demands that in-source builds be avoided.
Merged from HDF5 trunk r22617.
Tested:
jam (linux32-LE), koala (linux64-LE), ostrich (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), some manual tests as well
Fix for HDFFV-7993 - h5repack fails with error "chunk size must be <= maximum dimension size for fixed-sized dimensions"
Description:
Fixed a failure when change the chunk size of a specified chunked dataset with unlimited max dims.
Also took care of converting to contiguous and compact from the dataset.
Test cases were added and tagged with jira#.
Merged from HDF5 trunk r22277.
Tested:
jam (linux32-LE), koala (linux64-LE), ostrich (linuxppc64-BE), tejeda (mac32-LE), Windows (32-LE cmake), Cmake (jam)
Bring r22180, r22181 and r22214 from trunk to 1.8 branch:
Add new "metadata block size" command line option ('-M <x>' or
'--metadata_block_size=<x>') for h5repack.
[With release managers blessing]
Tested on:
Mac OSX/64 10.7.3 (amazon) w/debug)
(h5committest upcoming)
Fix for HDFFV-7840 h5repack: memory leak over one of the h5diff test file
Description:
Turned out that there were two causes of memory leaks.
1. for handling variable length string in attribute.
2. for handling compound type with non-reference members.
The first issue is fixed in copy_attr() which is updated to use h5tools_detect_vlen to take care of vlen string as well as vlen data.
The second is fixed in copy_refs_attr() of compound handling code.
Merged from HDF5 trunk r21869.
Tested:
jam (linux32-LE), koala (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Windows (32-LE cmake), Cmake (jam)
h5dump was used in test script but was not invoked by RUNSERIAL.
THis does not work in batch machines like Blue Gene in LLNL.
Solution:
Added $RUNSERIAL to invoke $H5DUMP_BIN.
Tested:
LLNL BlueGene (udawn)
HDFFV-5932 - h5repack breaks files with dimension scales
Description:
- Fixed h5repack to update values of references(object and region) of
attributes in h5repack for 1) references, 2) ARRAY of references,
3) VLEN of references, and 4) COMPOUND of references.
- Merged from HDF5 trunk 21393, 21382, 21386, 21389. (support Peter)
Tested:
jam (linux32-LE), koala (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE)
Work for HDFFV-7602 - HDF5 command tools: Provide framework for reusable
test files among tools
Description:
Merged from HDF5 trunk r21057
Provide framework to share test files among tools for tools test.
Tested:
jam (linux32-LE), koala (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE)
Fix for Bug1896 h5repack - changing layout to COMPACT does not work
Description:
Merged from hdf5 trunk (r19389)
Make h5repack be able to convert a layout to COMPACT for small size dataset as default. Also add verifying layout changes in our test script.
Tested:
jam, amani
Bring r19179 from trunk to 1.8 branch:
Bring changes from Coverity branch back to trunk:
r19079 & 19080:
[BZ1942] h5dump -u to generate XML, it does not respect the -m option
xml version of dump_data function didn't check for use of fp_format variable.
Added new test expected file for committed bug 1942
r19103, 19104 & 19105:
[BZ1821] h5repack -v did not display correct output for a selected compression. Needed new test for comparing output of -v option.
Added new test file for solution to BZ1821
BZ1821 - Bring test changes from the shell script actually used.
Tested on:
Mac OS X/32 10.6.4 (amazon) debug & production
(h5committested on branch)
Fix for bug1814 - NPOESS: h5repack doesn't handle references to the groups
as an element of a dataset
Description:
Merged from hdf5 trunk r18425
handles object reference to named-datatype as well.
Add test cases.
Tested:
jam
Add a run to the h5repack shell script to read a family file
The file used for input is located in the common source tools for testfiles, in tools/testfiles
Modified the h5repack shell script to read files from this location (h5repack reads its input files from a dedicated testfiles location in h5repack/testfiles)
Changed the h5diff open file call to use h5tools_fopen, so that it can open all file drivers
Tested: linux
Description:
h5repack previously would not take named datatypes into consideration when copying
datasets and attributes. This would cause extra anonymous datatypes in the target file
at best, and cause errors halfway through the repacking at worst. h5repack should now
always handle named datatypes correctly. Named datatypes are also now converted to the
native type when -n is given.
Tested: jam, linew, smirom (h5committest)
1. #1501 (B1) tools bug if dataset is larger than H5TOOLS_BUFSIZE limit.
ISSUE : the tools use the following formula to read by hyperslabs: hyperslab_size[i] = MIN( dim_size[i], H5TOOLS_BUFSIZE / datum_size) where H5TOOLS_BUFSIZE is a constant defined of 1024K. This is OK as long as the datum_size does not exceed 1024K, otherwise we have a hyperslab size of 0 (since 1024K/(greater than 1024K) = 0). This affects h5dump. h5repack, h5diff
SOLUTION: add a check for a 0 size and define as 1 if so.
TEST FOR H5DUMP: Defined a case in the h5dump test generator program of such a type (an array type of doubles with a large array dimension, that was the case the user reported). Since the written file commited in svn would be around 1024K, opted for not writing the data (the part of the code where the hyperslab is defined is executed, since h5dump always reads the files). Defined a macro WRITE_ARRAY to enable such writing if needed. Added a run on the h5dump shell script. Added 2 new files to svn: tools/testfiles/tarray8.ddl, tools/testfiles/tarray8.h5. NOTE: while doing this I thought of adding this dataset case to an existing file, but that would add the large array output to those files (the ddls). The issue is that the file list is increasing.
TEST FOR H5DIFF: for h5diff the check for reading by hyperslabs is H5TOOLS_MALLOCSIZE (128 * H5TOOLS_BUFSIZE) or 128 Mb. This makes it not possible to add such a file to svn, so used the same method as h5dump (only write the dataset if WRITE_ARRAY is defined). As opposed to h5dump, the hyperslab code is NOT executed when the dataset is empty (dataset is not read). Added the new dataset to existing files and shell run (tools/h5diff/testfiles/h5diff_dset1.h5 and tools/h5diff/testfiles/h5diff_dset2.h5 and output in tools/h5diff/testfiles/h5diff_80.txt).
TEST FOR H5REPACK: similar issue as h5diff with the difference that the hyperslab code is run. Added a run to the shell script (with a filter, otherwise the code uses H5Ocopy).
FURTHER ISSUES: the type in question ("double") has a different output cross platforms (e.g on liberty some garbage number is printed at some array locations)
SOLUTION: defined an "int" type for this test. However the printing of such an array has a bogus output at least in one platform (FreeBsd), so eliminated the test run altogether and filed a bug report on this
Bring r16401 back from trunk:
Correct error introduced in r16353 with layout version, and add test
so it gets caught earlier.
Tested on:
FreeBSD/32 6.3 (duty)
Too minor to require h5committest
-t T, --threshold=T Threshold value for H5Pset_alignment
-a A, --alignment=A Alignment value for H5Pset_alignment
2) bug fix
the printing of the dataset name was not done for references (verbose mode)
tested: windows, linux
Add a userblock to an HDF5 file during the repack. The user gives
give a filename and userblock size as command line parameters to
h5repack and the contents of that file are stored in the
userblock for the HDF5 file created by h5repack.
New flags to handle this -u and -b
Tested : windows, linux
Summary: when using h5diff to compare the results of h5repack (or other tools that copy one HDF5 file to another), a new option is needed to allow h5diff to make an "absolute" comparison of the 2 files. This is the "contents" mode explained in the usage below.
If this mode is present, objects in both files must match (must be exactly the same). If this does not happen, the tool returns an error code of 1 (instead of the success code of 0)
Changes to the h5repack test script: the call to h5diff was changed to include -c (maintaining the previous -q).
tested: windows, linux
if these are detected this syntax is used, otherwise the one in usage is used
there was another -i option for
-i L2, --indexed=L2 Minimum number of links in the indexed format
That was changed to -d
-d L2, --indexed=L2 Minimum number of links in the indexed format
Tested: windows, linux
Add regression test for h5repack with userblock
Tested on:
FreeBSD/32 6.2 (duty) in debug mode
FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
AIX/32 5.3 (copper) w/FORTRAN, w/parallel, in production mode
Mac OS X/32 10.4.10 (amazon) in debug mode
h5repack revision:
1. added a new test due to the introduction of H5Ocopy in the copy of objects (compressed dataset with references, that still must go a second sweep of the file to be regenerated).
2. Moved all the source files from the h5repack test program to a new file h5repacktst.c and removed the old ones (testh5repack*.c).
3. Renamed the binary files from test*.h5 to h5repack*.h5 for easy reference.
4. Modified the shell script to use variables for file names instead of hard coded names
1) added a new parameter to the h5diff function diff_array that contains
the beginning position of the hyperslab, so that the total position in
the array is printed correctly when reading by hyperslabs.
2) added a new test to h5diff that reads and diffs by hyperslabs. The
test reads a 1GB dataset, from which a 1KB hyperslab was written with
differences .
3) added the generation of 2 files to the generator program to test the
h5diff hyperslab read.
4) changed the h5diff binary pre-generated file names to be more
descriptive (e.g, instead of file1.h5, made it h5diff_basic1.h5)
5) changed the name of the h5repack options text file to info.h5repack
Fixes for bugs 676, 228
676: both h5repack and h5diff use H5Dread. In the case of a "big"
dataset, use read/write by hyperslabs the same way h5dump uses. An
arbitrary value of 1GB was defined for "big", i.e, if the dataset is
greater than 1GB, then read/write by hyperslabs
228: use the file type in read/write by default. A new switch -n was
introduced if the user wants to use a native type, which was the
previous use by default.
Added a new test for h5repack that repacks a 1GB dataset
Tested: heping (serial, parallel), sol, copper
