Commit Graph

222 Commits

Author SHA1 Message Date
David Young
dc97b9a165 Merge branch 'aug-develop' into aug-merge-attempt/feature/vfd_swmr 2020-08-04 10:22:47 -05:00
vchoi
4bc49d8b2b Fix for jira issue HDFFV-11080:
(1) Patch up the file pointer when reading attribute of variable length datatype
(2) Test to verify the fix when doing multiple threads
2020-07-14 10:56:33 -05:00
Quincey Koziol
e767f44e95 Merge remote-tracking branch 'origin/develop' into monotonic_timer 2020-06-26 18:57:38 -05:00
David Young
f8d469ca38 Add a test writer that creates groups under the root and a reader that polls
until all of the expected groups appear.
2020-05-27 10:47:04 -05:00
David Young
abced7dc4e Rebase repair: add stubs.c back to the test library sources. 2020-05-20 12:06:00 -05:00
Quincey Koziol
a96e6246d0 Align contents of CMake and autotools testfile cleanup lists. 2020-05-20 09:37:20 -05:00
Quincey Koziol
4ad7dca124 Avoid allocating a chunk index for datasets with 0-sized dimensions, until
the dataset is extended.
2020-05-20 09:37:20 -05:00
Jacob Smith
b11015c4cc Add Splitter VFD to library.
* "Simultaneous and equivalent" Read-Write and Write-Only channels for
  file I/O.
* Only supports drivers with the H5FD_FEAT_DEFAULT_VFD_COMPATIBLE flag for
  now, preventing issues with multi-file drivers.

Add Mirror VFD to library.

* Write-only operations over a network.
* Uses TCP/IP sockets.
* Server and auxiliary server-shutdown programs provided in a new directory,
  `utils/mirror_vfd`.
* Automated testing via loopback ("remote" of localhost).
2020-05-20 09:34:20 -05:00
David Young
e3ce3bf76a Take out the temporary performance tests. 2020-05-20 09:31:54 -05:00
David Young
66334f899d Temporarily add some code that measures the time to run the simplest possible
H5T__copy_all()-like routine 10 million times and then measures the version
with FUNC_ENTER_STATIC/_LEAVE_NOAPI and a HGOTO_ERROR() statement.
2020-05-20 09:31:54 -05:00
David Young
06dfd6025d Replace pthread_self_ulong() with H5TS_thread_id(). The POSIX Threads
implementation ought to be portable to any system that has POSIX
Threads.  On Windows, I use the same API call as before.
2020-05-20 09:31:54 -05:00
David Young
77144641df Revert "Make a squash commit of 'Quiet some warnings by adjusting warnings level and fixing some code.' (commit 5c911d8baf)" 2020-05-20 09:20:17 -05:00
David Young
b1be318a62 Undo accidental test deletion. 2020-05-20 09:19:37 -05:00
David Young
6b80036fde Straggler from previous commit: build and link the new .c files where I
moved some previously-static variables and functions that were causing
unused-variable/function warnings.
2020-05-20 09:19:37 -05:00
David Young
2b863a8119 Change some GCC warnings to errors. Fix code to quiet some warnings. 2020-05-20 09:17:27 -05:00
David Young
5d4b109abf Create a reader for the extensible datasets tests. 2020-05-07 12:09:15 -05:00
David Young
f8e494ba80 Add my work-in-progress dataset test. It writes a handful of datasets
that expand in one or two dimensions, depending on the setting of the -d
option argument.
2020-05-06 15:01:29 -05:00
Quincey Koziol
deadf25465 Align contents of CMake and autotools testfile cleanup lists. 2020-05-02 20:42:33 -05:00
Quincey Koziol
ac069841f3 Avoid allocating a chunk index for datasets with 0-sized dimensions, until
the dataset is extended.
2020-05-01 17:59:57 -05:00
David Young
9b5384b436 Add a reader personality to vfd_swmr_zoo_writer. Remove the disused
`-t` option.  Disable the error stack across create_/validate_zoo()
calls.
2020-03-31 13:14:59 -05:00
David Young
cb5e1ead29 Let tests provide a "hook" that runs after each step in create_zoo().
Provide a hook in vfd_swmr_zoo_writer that calls H5Fvfd_swmr_end_tick().
2020-03-23 14:55:07 -05:00
David Young
51b8fa3ffe Take a stab at a "zoo" writer for VFD SWMR: write all of the same
objects as the cache_image test does.  The zoo writer is a work in
progress.

This version is useful as a reproducer for the hang in the global heap
that I stumbled over, yesterday.  I run this to reproduce,

env HLOG="pbio=on" ./vfd_swmr_zoo_writer -W -a
2020-03-20 15:20:51 -05:00
Jacob Smith
b65405439d Add Splitter VFD to library.
* "Simultaneous and equivalent" Read-Write and Write-Only channels for
  file I/O.
* Only supports drivers with the H5FD_FEAT_DEFAULT_VFD_COMPATIBLE flag for
  now, preventing issues with multi-file drivers.

Add Mirror VFD to library.

* Write-only operations over a network.
* Uses TCP/IP sockets.
* Server and auxiliary server-shutdown programs provided in a new directory,
  `utils/mirror_vfd`.
* Automated testing via loopback ("remote" of localhost).
2020-03-13 17:13:17 -05:00
David Young
ae351c2376 Add tests for the two expected failure modes for variable-length (VL) strings
in VFD SWMR mode.
2020-03-13 14:16:40 -05:00
David Young
4e4c1c3850 Add my work in progress on variable-length string test for VFD SWMR. 2020-03-10 17:48:32 -05:00
David Young
986c7451a0 Replace pthread_self_ulong() with H5TS_thread_id(). The POSIX Threads
implementation ought to be portable to any system that has POSIX
Threads.  On Windows, I use the same API call as before.
2020-02-03 11:59:14 -06:00
David Young
c8f533cfc3 Merge all of my changes from merge-back-to-feature-vfd_swmr-attempt-1,
including the merge of `hdffv/hdf5/develop`, back to the branch that Vailin and
I share.

Now I need to put this branch on a fork with a less confusing name than
vchoi_fork!
2019-12-09 10:30:58 -06:00
David Young
0108072f7a Revert "Make a squash commit of 'Quiet some warnings by adjusting warnings level and fixing some code.' (commit 5c911d8baf)" 2019-11-27 11:26:02 -06:00
David Young
bd77af437f Undo accidental test deletion. 2019-11-13 13:41:05 -06:00
David Young
943e601b10 Straggler from previous commit: build and link the new .c files where I
moved some previously-static variables and functions that were causing
unused-variable/function warnings.
2019-11-13 11:49:15 -06:00
David Young
34eff2aae5 Change some GCC warnings to errors. Fix code to quiet some warnings. 2019-11-05 12:14:26 -06:00
Jerome Soumagne
ae490016b9 Add new H5R API that abstracts object, region and attribute reference types
Also support references to external files

Add new H5T_REF type and type conversion routines

Support conversion from H5T_REF_OBJ/DSET_REG to H5T_REF

Add H5Treclaim() API to reclaim memory of vlen/reference types

Deprecate H5Dvlen_reclaim()

Fix H5T_vlen_reclaim() and H5T_reclaim() to use private callback

Add H5T_ref_reclaim()

Move previous H5R APIs to H5Rdeprec.c

Clean up H5Ocopy

Separate H5O_copy_expand_ref() to H5Ocopy_ref()

Add support for copying new reference types

Clean up deprecated routines to go through VOL and same code path

Fix return codes in existing trefer.c test

Rename trefer.c to trefer_deprec.c

trefer.c is for new references

Add performance test for trefer

Add additional obj_copy_ref test

Make use of tokens and blobs to store references

Skip blob encoding for object references

Start adding new reference examples
2019-10-08 14:30:24 -05:00
Binh-Minh Ribler
e06bf9d263 HDFFV-10677 and HDFFV-10661
Description:
    - Added functions to query chunk information:
        H5Dget_num_chunks(dset_id, fspace_id, *nchunks)
            Gets the number of written chunks that intersect with the given
            dataspace.  However, in this version, the intersection is not
            yet completed.  Thus, the number of all written chunks will be
            returned.
        H5Dget_chunk_info_by_coord(dset_id, *offset, *filter_mask, *addr, *size)
            Given a chunk's logical coordinates, returns the chunk's filter,
            address, and size.
        H5Dget_chunk_info(dset_id, fspace_id, index, *offset, *filter_mask, *addr, *size)
            Given a chunk's index, returns the chunk's logical coordinates, filter,
            address, and size.  The chunk belongs to a set of chunks that have
            nonempty intersection with the specified dataspace.  However, in
            this version, the intersection is not yet completed, and the index
            is of all the written chunks.
        These functions comply with VOL.

    - Fixed some oversights found in the library for the tests in chunk_info.c
      to work correctly.  The returned value from a callback function was not
      checked in H5EA_iterate(), H5FA_iterate(), and H5D__none_idx_iterate().
      This oversight caused a callback function to continue iterating even though
      it's supposed to stop.
Platforms tested:
    Linux/64 (jelly)
    Linux/64 (platypus)
    Darwin (osx1011test)
2019-08-28 10:08:12 -05:00
Allen Byrne
98c8c6e45b Mostly whitespace in testpar, addl changes for tmp dir 2019-08-15 10:28:40 -05:00
Larry Knox
ab6679315c Update COPYING file.
Remove hdfs from list of vfds for check-vfd testing as it is read-only.
2019-07-30 16:37:57 -05:00
Larry Knox
8008294578 Squashed commit of the following:
Merge changes from update_merged_S3_HDFS branch into develop.

commit d5034315aea88629929ac0c9c59ebfafd5f21a31
Merge: 9c48823 d3fdcd8
Author: Larry Knox <lrknox@hdfgroup.org>
Date:   Thu Jul 25 08:24:53 2019 -0500

    Merge branch 'develop' into update_merged_S3_HDFS
2019-07-25 11:47:12 -05:00
Songyu Lu
1d1e96f1dd Some coding style changes. 2019-04-11 15:51:21 -05:00
Songyu Lu
477dda3c0d HDFFV-10658 - setting and getting properties in API context:
1. switched to use the existing H5F_prefix_open_t for enum type;
2. put the common private function used by external.c and external_env.c into external_common.c
2019-04-09 16:20:19 -05:00
Songyu Lu
50d9a397ab This commit basically has the following changes:
1. restored the datatype, dataspace, and LCPL of the dataset for VOL connector back to the properties.
2. splitted external.c and vds.c because they called HDsetenv in the program, instead using shell scripts to set the environment variables.
3. changed H5CX_get_vds_prefix and H5CX_get_ext_file_prefix to use H5P_peek instead of H5P_get.
2019-04-04 11:03:05 -05:00
Vailin Choi
d3dff6efe6 Merge branch 'develop' of https://bitbucket.hdfgroup.org/scm/hdffv/hdf5 into develop
Resolve conflicts:
Conflicts:
	test/Makefile.am
2019-01-02 13:23:54 -06:00
Dana Robinson
fba448b145 Merge pull request #1413 in HDFFV/hdf5 from ~DEROBINS/hdf5_der:yay_plugins to develop
* commit '3ca19cca5395d79be69209f8d7d0a2b06834a648':
  Flipped swapped testing strings.
  Added test_vol_plugin.sh to the list of scripts to clean
  Fixed a CMake build issue (CMake still doesn't run the VOL plugin tests)
  Added more sub-tests to the VOL plugin test.
  Added a simple test for registration of VOL connector plugins. Autotools only for right now, but this will be fleshed out in future work.
2018-12-31 10:28:25 -06:00
Vailin Choi
f808c108ed Fix for HDFFV-10659: The library abort with "infinite loop closing library"
after deleting attributes in densed storage.
The fix: When deleting attribute nodes from the name index v2 B-tree,
if an attribute is found in the intermediate B-tree nodes, which may be
merged/redistributed in the process, we need to free the dynamically
allocated spaces for the intermediate decoded attribute.
2018-12-28 16:49:11 -06:00
Dana Robinson
3a05e7109a Added test_vol_plugin.sh to the list of scripts to clean 2018-12-27 21:42:51 -08:00
Dana Robinson
621014be28 Added a simple test for registration of VOL connector plugins.
Autotools only for right now, but this will be fleshed out in future
work.
2018-12-27 15:50:04 -08:00
Jacob Smith
99bc714c4b Move H5Fset_dset_no_attrs_hint VOL operations to native.
Move minimzied object header tests from separate file to test/ohdr.c
Some formatting changes.
2018-12-18 13:42:03 -06:00
Jacob Smith
ea73325250 Merge branch 'develop' into dset_ohdr_minimize 2018-12-12 17:26:08 -06:00
Jacob Smith
496de6922f change test file name and apply 'h5_fixname' to it 2018-12-11 17:45:12 -06:00
Quincey Koziol
8c606f586e Correct "make check-vol" regression test target. 2018-11-28 22:54:46 -06:00
Quincey Koziol
d442fe2d98 Add support for "make check-vol", along with a few minor cleanups, etc. 2018-11-28 19:07:30 -06:00
Vailin Choi
ce2748f011 (A) Fixes for the assertion failures described in issue #2 (see John Mainzer's last checkin message):
--src/H5PB.c: checks for size >= page size
--src/H5MF.c: disable/enable page buffering in H5MF_tidy_self_referential_fsm_hack()
--src/H5MFsection.c: call H5PB_remove_entry() for both raw/metadata pages in H5MF__sect_small_merge()
(B) Port and modify existing concurrent swmr tests to VFD SWMR. Also modify the following:
--remove flushes from VFD SWMR writer tests
--set Nreaders to 0 in test/testvfdswmr.sh.in to test for writers only
Please enter the commit message for your changes. Lines starting
2018-11-14 15:35:35 -06:00