Merge 1.8 and h5dump/tools and tests based on tools library from trunk.
Reduced warnings.
HDFFV-7949:
Remove duplicated functions in h5ls
Tested: local linux,h5committest
- HDFFV-5928 - GMQS: h5diff problem and improvement on comparsing the same objects
Description:
Merged from HDF5 trunk r20676.
Fixed:
1) adding h5tools_is_obj_same() function to check if two given IDs or paths point to the same object. This function can be very useful for other tools and applications.
2) using h5tools_is_obj_same() at h5diff() and diff() in h5diff.c. If two paths point to the same object, there is no need to check the details of the object since we know there is no difference. The fix will increase the performance by skipping the content comparison. It also fixed the problem of reporting difference for some cases of comparing the same file, e.g. empty files or files with incomparable objects the same file.
Test update:
Updat prvious test cases (171, 172, 530) affected by this fix, so they
still perfrom originally intended testing without bypassing.
Tested:
jam (linux32-LE), koala (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), cmake
Bring r20538 & r20539 from trunk to 1.8 branch:
Update with new warning flags for gcc 4.6.x
Bring Coverity changes back to trunk:
r20276:
Add recursive is vlen string function.
Cleanup resource leaks for issues: 200,202,329,688,811,812
r20277:
Check types and close by adding error section: issue 687
r20278:
Replaced implicit pointer conversion with (ocrt_info.new_obj != NULL).
r20280:
Addressed coverity issues 927-929 & 583. The real issue is failure to check
file name length -- at least at the H5FD interface level. This needs more work, but at least I have dealt with the issue in H5FDfamily.c
r20337:
H5O_type_t obj_type = H5O_TYPE_UNKNOWN;
r20338:
Added udata.name = NULL; to prevent potential uninitialized use after done: label.
r20339:
coverity issues: 686,828,1670-1673,1707-1711
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
FreeBSD/32 8.2 (loyalty) w/gcc4.6, in debug mode
FreeBSD/64 8.2 (freedom) w/gcc4.6, in debug mode
Linux/32 2.6 (jam) w/PGI compilers, w/default API=1.8.x,
w/C++ & FORTRAN, w/threadsafe, in debug mode
Linux/64-amd64 2.6 (amani) w/Intel compilers, w/default API=1.6.x,
w/C++ & FORTRAN, in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, w/threadsafe, in production mode
Linux/PPC 2.6 (heiwa) w/C++ & FORTRAN, w/threadsafe, in debug mode
Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in debug mode
Bring r20480 from trunk to 1.8 branch:
Remove global variable for the path name in the group traversal code. Also
clean up a bunch of compiler warnings.
Tested on:
Mac OS X/32 10.6.7 (amazon) w/debug & production
(h5committested on trunk)
Fixed Bug 2216 - GMQS: h5diff - memory leak when compares vlen string in
dataset or attributes
Description:
Merged from HDF5 trunk r20266, r20270 and r20285.
Test for dataset : valgrind --leak-check=full ./h5diff -v h5diff_dset1.h5 h5diff_dset2.h5 /g1/VLstring
Test for attr : valgrind --leak-check=full ./h5diff h5diff_attr1.h5 h5diff_attr2.h5
Both test cases are in testing script.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), Cmake - jam
Added back test for H5Tdetect_class of H5T_VLEN after each instance of above function to catch all VLEN types in h5dump.
bring back from trunk r20247
Tested: local linux
Fixing Bug 2092 - h5dump does not display index for a dataset
Description:
Merged from HDF5 trunk r20108 and r20132.
h5dump skip displaying array indices every certain number when the
array type dataset is relatively big. The certain number varies
according to the size of each array.
This checkin fix the skipping array indices problem. This fix also
correct indentation of the dataset data output.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), Cmake - jam
Bring r19654 from trunk to 1.8 branch:
Bring Coverity revisions from branch back to trunk, and clean up some other
misc. compiler warnings also.
r19500:
Fix coverity items 1446 and 1447. Moved up calls to memset in test_cont in
ohdr.c so the test never tries to close uninitialized locations.
r19501:
Fix coverity items 1398-1445. Various uninitialized variable errors in fheap.c.
r19502:
Fixed coverity issue 579 and some additional warnings in the file as well.
r19503:
Bug fix: This fix addressed the "RESOURCE_LEAK" problems #789 and 790, run 26
r19504:
minor mods to try to keep coverity from flagging false positives.
r19505:
Fixed coverity issues 566 - 571. Declared variables that are passed to functions that use them as arrays to be arrays of size 1.
Tested on:
Mac OS X/32 10.6.4 (amazon) w/debug, production & parallel
(h5committested on coverity branch)
Bring r19252 & 19253 from trunk to 1.8 branch:
Bring Coverity changes from branch to trunk:
r19161:
Fixed the part for matching the subset info with dataset
r19189:
BZ1646: h5dump does not check number of dimensions for subsetting parameters against the dataset
Changed subset_t structure from holding hsize_t pointers to holding new subset_d pointers, which hold the original hsize_t pointer + len. this len is then checked against dataset ndims in the handle_dataset function of h5dump.
Changed all references to use new data structure.
Added tests for each subset parameter.
r19190:
Added new h5dump ddl files
Tested on:
Mac OS X/32 10.6.4 (amazon) w/debug & production
(h5committested on Coverity branch)
Bring r19237 from trunk to 1.8 branch:
Clean up misc. compiler warnings.
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
Mac OS X/32 10.6.4 (amazon) w/C++ & FORTRAN, w/threadsafe,
in production mode
Mac OS X/32 10.6.4 (amazon) w/parallel, in debug mode
any more. Changed algorithm of creating mask which now sits
at the least significant bits. Display data by down shift
first and then mask it for display. Changed the variable
names to reflect the new purpose better.
Tested: AlbertPax (linux)
Install packed-bits feature for h5dump.
tools/h5dump/h5dump.c:
Added lots of error checking into parse_mask_list() to catch and
flagged packed-bits argument errors. For now, the packed bits
are limited to 1 byte (8 bits) range.
tools/h5dump/testh5dump.sh.in
Added 10 new packed-bits test to verify normal operation and argument
error detection.
tools/lib/h5tools.h:
tools/lib/h5tools_str.c:
tools/lib/h5tools.c:
Only name changes of the packed-bit global variables to reflect
the nature of the variable better, IMO.
Tested: albertPax (linux)
------------------------------------------------------------------------
Install packed-bits feature for h5dump.
Allen has installed the feature in the NPOESS feature branch.
I ported it back to the v1.8 branch for v1.8.5 release.
tools/h5dump/testh5dump.sh.in:
Added tests for the packed-bits feature.
tools/h5dump/h5dump.c
tools/h5dump/h5dump.h
Added code to parse and handled the packed-bits request.
tools/lib/h5tools.h
tools/lib/h5tools_str.c
tools/lib/h5tools.c
Added code to print packed-bits.
tools/testfiles/tpackedbits2.ddl
tools/testfiles/tnofilename-with-packed-bits.ddl
tools/testfiles/tpackedbits.ddl
Expected output files for packed-bits feature.
Tested:
h5committested the NPOESS branch.
Tested in the linux VM machine in my Dove.
Bring r18109 from trunk to 1.8 branch:
Bring r18076-18096 from hdf5_1_8_coverity branch to trunk:
r18076:
Correct Coverity issue #1 by removing dead code
r18077:
Fix coverity item 142. When an error occurred while copying a linked list in
H5S_point_copy, the library would not free the partially allocated list. Added
code to free the list in this case.
r18078:
Correct Coverity issue #2 by removing impossible to reach code.
r18079:
Correct #3 by removing impossible to reach code.
r18080:
Correct Coverity issue #4 by removing impossible to reach code.
r18081:
fix coverity 26 , check (dblik->parent) before calls H5HF_man_iblock_detach().
r18082:
Fixed coverity issues 321 and 316.
321: freed sm_buf in error handling to remove resource leak.
Also set sm_buf to NULL after other instances in which
it is freed to prevent double free.
316: initialized nmembs to 0.
r18083:
Correct Coverity issue #6 by removing debugging knob from error reporting code.
r18084:
Fix coverity item 269 + others. When a error occurred in a function using the
h5tools error framework, the "past_catch" variable would not be set to true
because that statement was before the label that goto jumped to. This could
cause a failure in the cleanup section to go back to the start of the section,
freeing variables twice, etc.
Moved the label infront of past_catch=TRUE.
r18085:
fixed coverity #27, check if (heap) before use heap->obj....
r18086:
fixed coverity #28, check curr_span not null before use it at if(curr_span &&
(io_bytes_left==0 || curr_seq>=maxseq))
r18087:
Correct Coverity issue #7 by cleaning up correctly on error
r18088:
Correct Coverity #8 by removing unchanged variable checking code.
r18089:
Correct Coverity issue #9 - remove impossible to reach code.
r18090:
Correct Coverity issue #11 by removing impossible to reach code. Also clean
up some minor style issues.
r18091:
Fix coverity items 314 and 318. Changed the improper assertion of the return
value of a library function to a check, and a return(void) on failure.
r18092:
Fix coverity item 70. Changed the improper assertion of the return value of a
library function to a check, and a return(void) on failure.
r18093:
Correct Coverity issue #12 by removing dead code.
r18094:
Correct Coverity issue #16 by removing debugging code.
r18095:
Fixed coverity issue # 271.
Removed redundant checking and freeing of sm_buf1 and sm_buf2.
r18096:
Correct Coverity issue #17 by refactoring test to remove dead code.
Also, removed previous "coverity" statements in comments, we'll review
those issues again and see if we can figure them out, now that we have more
experience with Coverity.
Tested on:
Mac OS X/32 10.6.2 (amazon) debug & prod
321: freed sm_buf in error handling to remove resource leak.
Also set sm_buf to NULL after other instances in which
it is freed to prevent double free.
316: initialized nmembs to 0.
Bring r18011 from trunk to 1.8 branch:
Bring Coverity changes into the trunk: (also other minor cleanups)
r17991:
Fix Coverity items 175 and 176. Fixed memory leak on error in print_enum in
H5LT.c.
r17993: (r17992 was not a Coverity change)
Close Coverity issue #206: inconsistently checking whether dt->shared was
non-NULL after H5T_alloc() returned a valid 'dt' value (which should guarantee
that dt->shared is valid).
r17994:
Fix Coverity item 149. Fixed file handle leak on error in H5FD_stdio_open.
r17995:
Fixed Coverity issues 154 to 161:
Added H5MP_close routine to error handling in the event *mp has not been
freed before error.
r17996:
Close Coverity issue #126: potentially leaking merged_spans on routine failure.
r17997:
Fix Coverity items 147 and 148. Fixed resource leaks on error in H5FDloc.c.
r17998:
Coverity issue 269-272:
Added integer result variable to functions that could return negative.
Assigned to unsigned after checking.
Added H5E_BEGIN_TRY block around H5Tclose and removed H5E_THROW in the
catch block.
Checked buffer is NULL before free. Changed HGOTO_ERROR outside of the if
block to H5E_THROW.
r17999:
Close Coverity issue #127: release temporary spans in more generic manner.
(Also add error checking to previous fix)
r18000:
Resolved Coverity issues 211 and 212 in H5T.c. Added comments to ignore
Coverity warning regarding not checking pointer for NULL, as we are using
an assert which catches the issue.
r18001:
Fix Coverity item 146. Fixed resource leak on error in H5O_layout_copy.
r18002:
Fix Coverity items 143 and 145. Fixed resource leaks on error in
H5D_compact_copy and H5D_contig_copy.
r18003:
Close Coverity issue #192: close file on error
r18004:
Fix Coverity issue #125: release temporary spans on error
r18005:
Resolved Coverity issues 5, 25, and 83 (in H5T.c):
Separated embedded functions in order to check for NULL on return of
H5I_object before passing into H5T_copy.
Check to see if new_dt is NULL within error handling before
dereferencing it.
Ignore Coverity's dead code warnings as the checks that lead to the code
are machine dependent.
r18006:
Coverity 63,70,73: Checked result of function before assigning to an unsigned
variable.
r18007:
Coverity 78,79: added continue statement if H5Pget_filter2 returns negative.
r18008:
Fixed Coverity issue # 138:
Added support in error handling to free dst pointer (if allocated) on error.
r18009:
Whitespace & coding style cleanup
Tested on:
Mac OS X/32 10.6.2 (amazon) w/debug & production
(h5committested on trunk)
Bring r17953 from trunk to 1.8 branch:
Bring Coverity changes into the trunk:
r17877:
Error 266: Uninitialized memspace set to -1. Changed malloc and free to HDmalloc and HDfree. Removed unused dtype var.
r17878:
Error 265: Uninitialized mem_space set to -1. Changed malloc and free to HDmalloc and HDfree.
Error 267: Uninitialized smspace set to -1. Changed malloc and free to HDmalloc and HDfree.
r17879:
Error 242: Uninitialized aid set to -1.
Error 243: Uninitialized sid set to -1.
Uninitialized tid set to -1 for consistency
r17880:
Error 242: reinitialized aid to -1 after close to be ready for reuse.
Error 243: reinitialized sid to -1 after close to be ready for reuse.
reinitialized tid to -1 for consistency after close to be ready for reuse.
r17881:
use valgrind to check there is a memory leak. The fix is to free ptrstr in line 5838 at xml_dump_group() of h5dump.c after it is used. run the valgrind after the fix, no memory leak for that part of the code.
r17882:
Fix Coverity items 256 and 269.
r17883:
Error 222-230: initialized hid_t vars to -1 after close to be ready for reuse.
Also added H5Tclose for tid in gent_bigdims
r17884:
Bug fix (Coverity run2, view 23, dead code)
(this is pair-program done by Albert, Elena and Larry).
Confirmed and fixed the deadcode in hyperslab read branch of function
diff_datasetid.
(Discovered other bad code that diff_datasetid() should be recoded.
Bug 1693 is entered for this.)
r17906:
Fix Coverity item 260.
r17907:
262: Initialized hid_t's dtype, dtype_tmp and file to -1. Initialized H5T_t * dt to NULL.
r17908:
Fix Coverity item 261.
r17909:
Fix Coverity item 248.
r17910:
Revise fix for Coverity item 248.
r17911:
Resolved coverity issues #s 263, 162, 163, 164. All issues in dsets.c. Initialized fid and did hid_t's. filter_corrupt function was returning in the middle of an if statement, bypassing free calls. Updated error handling to free buffers and hid_t's appropriately.
r17912:
(done by Larry and Albert)
Cleanup Coverity view warnings (#231-241) about
using uninitialized variables. Initialized all
of them.
r17913:
Resolved issue 251 and 264. Initialized tid2 to -1 and initialized buffers that were freed in case of an error.
r17914:
Resolved coverity issues 66, 220, and 221:
66: Negative Return assignment ignored
220,221: Initialized hid_t's.
r17915:
Fix Coverity item 247.
r17916:
Fix Coverity item 246.
r17917:
Fix Coverity item 245.
r17918:
Fix Coverity item 244.
r17919:
Coverity Issue #84: Moved asserts in H5T_cmp to the top of the function, and converted them to HDassert. Coverity complaining about using potentially NULL pointer without checking it. Want to see if Coverity will accept Assertions as acceptable checking before using the value.
Tested on:
FreeBSD/32 6.3 (duty) w/debug
(h5committested on trunk)
1. #1501 (B1) tools bug if dataset is larger than H5TOOLS_BUFSIZE limit.
ISSUE : the tools use the following formula to read by hyperslabs: hyperslab_size[i] = MIN( dim_size[i], H5TOOLS_BUFSIZE / datum_size) where H5TOOLS_BUFSIZE is a constant defined of 1024K. This is OK as long as the datum_size does not exceed 1024K, otherwise we have a hyperslab size of 0 (since 1024K/(greater than 1024K) = 0). This affects h5dump. h5repack, h5diff
SOLUTION: add a check for a 0 size and define as 1 if so.
TEST FOR H5DUMP: Defined a case in the h5dump test generator program of such a type (an array type of doubles with a large array dimension, that was the case the user reported). Since the written file commited in svn would be around 1024K, opted for not writing the data (the part of the code where the hyperslab is defined is executed, since h5dump always reads the files). Defined a macro WRITE_ARRAY to enable such writing if needed. Added a run on the h5dump shell script. Added 2 new files to svn: tools/testfiles/tarray8.ddl, tools/testfiles/tarray8.h5. NOTE: while doing this I thought of adding this dataset case to an existing file, but that would add the large array output to those files (the ddls). The issue is that the file list is increasing.
TEST FOR H5DIFF: for h5diff the check for reading by hyperslabs is H5TOOLS_MALLOCSIZE (128 * H5TOOLS_BUFSIZE) or 128 Mb. This makes it not possible to add such a file to svn, so used the same method as h5dump (only write the dataset if WRITE_ARRAY is defined). As opposed to h5dump, the hyperslab code is NOT executed when the dataset is empty (dataset is not read). Added the new dataset to existing files and shell run (tools/h5diff/testfiles/h5diff_dset1.h5 and tools/h5diff/testfiles/h5diff_dset2.h5 and output in tools/h5diff/testfiles/h5diff_80.txt).
TEST FOR H5REPACK: similar issue as h5diff with the difference that the hyperslab code is run. Added a run to the shell script (with a filter, otherwise the code uses H5Ocopy).
FURTHER ISSUES: the type in question ("double") has a different output cross platforms (e.g on liberty some garbage number is printed at some array locations)
SOLUTION: defined an "int" type for this test. However the printing of such an array has a bogus output at least in one platform (FreeBsd), so eliminated the test run altogether and filed a bug report on this
Remove trailing whitespace from C/C++ source files, with the following
script:
foreach f (*.[ch] *.cpp)
sed 's/[[:blank:]]*$//' $f > sed.out && mv sed.out $f
end
Tested on:
Mac OS X/32 10.5.5 (amazon)
No need for h5committest, just whitespace changes...
add a check for block overlap after the command line parsing
* Algorithm
*
* In a inner loop, the parameters from SSET are translated into temporary
* variables so that 1 row is printed at a time (getting the coordinate indices
* at each row).
* We define the stride, count and block to be 1 in the row dimension to achieve
* this and advance until all points are printed.
* An outer loop for cases where dimensionality is greater than 2D is made.
* In each iteration, the 2D block is displayed in the inner loop. The remaining
* slower dimensions above the first 2 are incremented one at a time in the outer loop
*
* The element position is obtained from the matrix according to:
* Given an index I(z,y,x) its position from the beginning of an array
* of sizes A(size_z, size_y,size_x) is given by
* Position of I(z,y,x) = index_z * size_y * size_x
* + index_y * size_x
* + index_x
*
tested: windows, linux
