+ +\section sec_learn Learning HDF5 +There are several resources for learning about HDF5. The HDF Group provides an on-line HDF5 tutorial, +documentation, examples, and videos. There are also tutorials provided by other organizations that are very useful for learning about HDF5. + +\subsection subsec_learn_intro The HDF Group Resources +For a quick introduction to HDF5 see the following: +
| +@ref IntroHDF5 + | ++A very brief introduction to HDF5 and the HDF5 programming model and APIs + | +
| +@ref LearnHDFView + | ++A tutorial for learning how to use HDFView. NO programming involved! + | +
| +@ref LearnBasics + | ++Step by step instructions for learning HDF5 that include programming examples + | +
| +Using the High Level APIs + | ++\ref H5LT \ref H5IM \ref H5TB \ref H5PT \ref H5DS + | +
| +Introduction to Parallel HDF5 + | ++A brief introduction to Parallel HDF5. If you are new to HDF5 please see the @ref LearnBasics topic first. + | +
| +\ref ViewTools + | ++\li @ref LearnHDFView +\li @ref ViewToolsCommand +\li @ref ViewToolsJPSS + | +
| +HDF5-1.10 New Features + | ++\li Introduction to the Virtual Dataset - VDS +\li Introduction to Single-Writer/Multiple-Reader (SWMR) + | +
| +Example Programs + | ++\ref HDF5Examples + | +
| +Videos + | ++\li Introduction to HDF5 +\li Parallel HDF5 + | +
+Navigate back: \ref index "Main" + +*/ diff --git a/doxygen/dox/IntroHDF5.dox b/doxygen/dox/IntroHDF5.dox new file mode 100644 index 0000000000..ec46217c4a --- /dev/null +++ b/doxygen/dox/IntroHDF5.dox @@ -0,0 +1,627 @@ +/** @page IntroHDF5 Introduction to HDF5 + +Navigate back: \ref index "Main" / \ref GettingStarted +
+ +\section sec_intro_desc HDF5 Description +HDF5 consists of a file format for storing HDF5 data, a data model for logically organizing and accessing +HDF5 data from an application, and the software (libraries, language interfaces, and tools) for working with this format. + +\subsection subsec_intro_desc_file File Format +HDF5 consists of a file format for storing HDF5 data, a data model for logically organizing and accessing HDF5 data from an application, +and the software (libraries, language interfaces, and tools) for working with this format. + +\subsection subsec_intro_desc_dm Data Model +The HDF5 Data Model, also known as the HDF5 Abstract (or Logical) Data Model consists of +the building blocks for data organization and specification in HDF5. + +An HDF5 file (an object in itself) can be thought of as a container (or group) that holds +a variety of heterogeneous data objects (or datasets). The datasets can be images, tables, +graphs, and even documents, such as PDF or Excel: + +
| +\image html fileobj.png + | +
| +\image html group.png + | +
| +\image html dataset.png + | +
| +\image html datatype.png + | +
-
+
-
+Pre-Defined Datatypes: These are datatypes that are created by HDF5. They are actually opened (and closed)
+by HDF5 and can have different values from one HDF5 session to the next. There are two types of pre-defined datatypes:
+
-
+
- +Standard datatypes are the same on all platforms and are what you see in an HDF5 file. Their names are of the form +H5T_ARCH_BASE where ARCH is an architecture name and BASE is a programming type name. For example, #H5T_IEEE_F32BE +indicates a standard Big Endian floating point type. + +
- +Native datatypes are used to simplify memory operations (reading, writing) and are NOT the same on different platforms. +For example, #H5T_NATIVE_INT indicates an int (C). + +
+ -
+Derived Datatypes: These are datatypes that are created or derived from the pre-defined datatypes.
+An example of a commonly used derived datatype is a string of more than one character. Compound datatypes
+are also derived types. A compound datatype can be used to create a simple table, and can also be nested,
+in which it includes one more other compound datatypes.
+
+
+This is an example of a dataset with a compound datatype. Each element in the dataset consists +of a 16-bit integer, a character, a 32-bit integer, and a 2x3x2 array of 32-bit floats (the datatype). +It is a 2-dimensional 5 x 3 array (the dataspace). The datatype should not be confused with the dataspace. + ++ ++\image html cmpnddtype.png + +
+
| +\image html dataspace1.png + | +
| +\image html dataspace.png + | +
| +\image html properties.png + | +
h5dump
+The h5dump utility displays the contents of an HDF5 file in Data Description Language (\ref DDLBNF110). +Below is an example of h5dump output for an HDF5 file that contains no objects: +\code +$ h5dump file.h5 + HDF5 "file.h5" { + GROUP "/" { + } + } +\endcode + +With large files and datasets the output from h5dump can be overwhelming. +There are options that can be used to examine specific parts of an HDF5 file. +Some useful h5dump options are included below: +\code + -H, --header Display header information only (no data) + -dh5cc, h5fc, h5c++
+The built HDF5 binaries include the h5cc, h5fc, h5c++ compile scripts for compiling applications. +When using these scripts there is no need to specify the HDF5 libraries and include files. +Compiler options can be passed to the scripts. + +HDFView
+The HDFView tool allows browsing of data in HDF (HDF4 and HDF5) files. + +\section sec_intro_pm Introduction to the HDF5 Programming Model and APIs +The HDF5 Application Programming Interface is extensive, but a few functions do most of the work. + +To introduce the programming model, examples in Python and C are included below. The Python examples +use the HDF5 Python APIs (h5py). See the Examples from "Learning the Basics" page for complete examples +that can be downloaded and run for C, FORTRAN, C++, Java and Python. + +The general paradigm for working with objects in HDF5 is to: +\li Open the object. +\li Access the object. +\li Close the object. + +The library imposes an order on the operations by argument dependencies. For example, a file must be +opened before a dataset because the dataset open call requires a file handle as an argument. Objects +can be closed in any order. However, once an object is closed it no longer can be accessed. + +Keep the following in mind when looking at the example programs included in this section: +-
+
-
+
-
+
- +C routines begin with the prefix “H5*” where * is a single letter indicating the object on which the +operation is to be performed. + +
- +FORTRAN routines are similar; they begin with “h5*” and end with “_f”. + +
- +Java routines are similar; the routine names begin with “H5*” and are prefixed with “H5.” as the class. Constants are +in the HDF5Constants class and are prefixed with "HDF5Constants.". The function arguments +are usually similar, @see @ref HDF5LIB + +
-
+
-
+File Interface:
- #H5Fopen (C)
- h5fopen_f (FORTRAN)
- H5.H5Fopen (Java)
+ -
+Dataset Interface:
- #H5Dopen (C)
- h5dopen_f (FORTRAN)
- H5.H5Dopen (Java)
+ -
+Dataspace interface:
- #H5Sclose (C)
- h5sclose_f (FORTRAN)
- H5.H5Sclose (Java)
+
+ -
+For portability, the HDF5 library has its own defined types. Some common types that you will see
+in the example code are:
+
-
+
- +#hid_t is used for object handles + +
- +hsize_t is used for dimensions + +
- +#herr_t is used for many return values + +
+ -
+Language specific files must be included in applications:
+
-
+
-
+Python: Add
"import h5py / import numpy"+
+ -
+C: Add
"#include hdf5.h"+
+ -
+FORTRAN: Add
"USE HDF5"and call h5open_f and h5close_f to initialize and close the HDF5 FORTRAN interface +
+ -
+Java: Add
"import hdf.hdf5lib.H5; + import hdf.hdf5lib.HDF5Constants;"+
+
+ -
+Python: Add
| +\image html crtf-pic.png + | +
| +Python +\code + import h5py + file = h5py.File (‘file.h5’, ‘w’) + file.close () +\endcode + | +
| +C +\code + #include “hdf5.h” + + int main() { + hid_t file_id; + herr_t status; + + file_id = H5Fcreate ("file.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); + status = H5Fclose (file_id); + } +\endcode + | +
| +\image html crtdset.png + | +
| +Python +\code + dataset = file.create_dataset("dset",(4, 6), h5py.h5t.STD_I32BE) +\endcode + | +
| +C +\code + // Create the dataspace for the dataset. + dims[0] = 4; + dims[1] = 6; + + dataspace_id = H5Screate_simple(2, dims, NULL); + + // Create the dataset. + dataset_id = H5Dcreate (file_id, "/dset", H5T_STD_I32BE, dataspace_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); + + // Close the dataset and dataspace + status = H5Dclose(dataset_id); + status = H5Sclose(dataspace_id); +\endcode + | +
| +Python +\code + data = np.zeros((4,6)) + for i in range(4): + for j in range(6): + data[i][j]= i*6+j+1 + + dataset[...] = data <-- Write data to dataset + data_read = dataset[...] <-- Read data from dataset +\endcode + | +
| +C +\code + status = H5Dwrite (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, dset_data); + status = H5Dread (dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, dset_data); +\endcode + | +
| +\image html crtgrp.png + | +
| +Python +\code + import h5py + file = h5py.File('dset.h5', 'r+') + group = file.create_group ('MyGroup') + file.close() +\endcode + | +
| +C +\code + group_id = H5Gcreate (file_id, "MyGroup", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); + status = H5Gclose (group_id); +\endcode + | +
| +\image html crtatt.png + | +
| +Python +\code + dataset.attrs["Units"] = “Meters per second” <-- Create string + attr_data = np.zeros((2,)) + attr_data[0] = 100 + attr_data[1] = 200 + dataset.attrs.create("Speed", attr_data, (2,), “i”) <-- Create Integer +\endcode + | +
| +C +\code + hid_t attribute_id, dataspace_id; // identifiers + hsize_t dims; + int attr_data[2]; + herr_t status; + + ... + + // Initialize the attribute data. + attr_data[0] = 100; + attr_data[1] = 200; + + // Create the data space for the attribute. + dims = 2; + dataspace_id = H5Screate_simple(1, &dims, NULL); + + // Create a dataset attribute. + attribute_id = H5Acreate2 (dataset_id, "Units", H5T_STD_I32BE, + dataspace_id, H5P_DEFAULT, H5P_DEFAULT); + + // Write the attribute data. + status = H5Awrite(attribute_id, H5T_NATIVE_INT, attr_data); + + // Close the attribute. + status = H5Aclose(attribute_id); + + // Close the dataspace. + status = H5Sclose(dataspace_id); +\endcode + | +
+Navigate back: \ref index "Main" / \ref GettingStarted + + +@page HDF5Examples HDF5 Examples +Example programs of how to use HDF5 are provided below. +For HDF-EOS specific examples, see the examples +of how to access and visualize NASA HDF-EOS files using IDL, MATLAB, and NCL on the +HDF-EOS Tools and Information Center page. + +\section secHDF5Examples Examples +\li \ref LBExamples +\li Examples by API +\li Examples in the Source Code +\li Other Examples + +\section secHDF5ExamplesCompile How To Compile +For information on compiling in C, C++ and Fortran, see: \ref LBCompiling + +\section secHDF5ExamplesOther Other Examples +IDL, MATLAB, and NCL Examples for HDF-EOS +Examples of how to access and visualize NASA HDF-EOS files using IDL, MATLAB, and NCL. + +Miscellaneous Examples +These (very old) examples resulted from working with users, and are not fully tested. Most of them are in C, with a few in Fortran and Java. + +Using Special Values +These examples show how to create special values in an HDF5 application. + +*/ diff --git a/doxygen/dox/LearnBasics.dox b/doxygen/dox/LearnBasics.dox new file mode 100644 index 0000000000..298672d185 --- /dev/null +++ b/doxygen/dox/LearnBasics.dox @@ -0,0 +1,183 @@ +/** @page LearnBasics Learning the Basics + +Navigate back: \ref index "Main" / \ref GettingStarted +
+ +\section secIntro Introduction +The following topics cover the basic features in HDF5. The topics build on each other and are +intended to be completed in order. Some sections use files created in earlier sections. The +examples used can also be found on the \ref LBExamples +page and in the HDF5 source code (C, C++, Fortran). + +\section Topics Topics +\li @subpage LBFileOrg +\li @subpage LBAPI +\li @subpage LBProg +\li @subpage LBFileCreate +\li @subpage LBDsetCreate +\li @subpage LBDsetRW +\li @subpage LBAttrCreate +\li @subpage LBGrpCreate +\li @subpage LBGrpCreateNames +\li @subpage LBGrpDset +\li @subpage LBDsetSubRW +\li @subpage LBDatatypes +\li @subpage LBPropsList +\li @subpage LBDsetLayout +\li @subpage LBExtDset +\li @subpage LBComDset +\li @subpage LBContents +\li @subpage LBQuiz +\li @subpage LBQuizAnswers +\li @subpage LBCompiling +\li @subpage LBTraining + +
+Navigate back: \ref index "Main" / \ref GettingStarted + + +@page LBExamples Examples from Learning the Basics + +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBExamples +These examples are used in the \ref LearnBasics topic. See \ref LBCompiling for details on compiling them. +PLEASE NOTE that the example programs are listed in the order they are expected to be run. Some example +programs use files created in earlier examples. + +\section secLBExamplesSrc HDF5 Source Code Examples +These examples (C, C++, Fortran) are provided in the HDF5 source code and (Unix) binaries. +
| Feature + | +Examples + | +Comments + | +
|---|---|---|
| Create a file + | +C Fortran C++ Java Python + | ++ | +
| Create a dataset + | +C Fortran C++ Java Python + | ++ | +
| Read and write to a dataset + | +C Fortran C++ Java Python + | ++ | +
| Create an attribute + | +C Fortran C++ Java Python + | ++ | +
| Create a group + | +C Fortran C++ Java Python + | ++ | +
| Create groups in a file using absolute and relative paths + | +C Fortran C++ Java Python + | ++ | +
| Create datasets in a group + | +C Fortran C++ Java Python + | ++ | +
| Create a file and dataset and select/read a subset from the dataset + | +C Fortran C++ Java Python + | +Also see examples to Write by row (and column) below. + | +
| Create an extendible (unlimited dimension) dataset + | +C Fortran C++ Java Python + | +Also see examples to Extend by row (and column) below + | +
| Create a chunked and compressed dataset + | +C Fortran C++ Java Python + | ++ | +
| Feature + | +Examples + | +
|---|---|
| Write by row + | +C Fortran + | +
| Write by column + | +C Fortran + | +
| Extend by row + | +C Fortran + | +
| Extend by column + | +C Fortran + | +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +*/ diff --git a/doxygen/dox/LearnBasics1.dox b/doxygen/dox/LearnBasics1.dox new file mode 100644 index 0000000000..a9b6d0e71f --- /dev/null +++ b/doxygen/dox/LearnBasics1.dox @@ -0,0 +1,1023 @@ +/** @page LBFileOrg HDF5 File Organization + +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBFileOrg HDF5 file +An HDF5 file is a container for storing a variety of scientific data and is composed of two primary types of objects: groups and datasets. + +\li HDF5 group: a grouping structure containing zero or more HDF5 objects, together with supporting metadata +\li HDF5 dataset: a multidimensional array of data elements, together with supporting metadata + +Any HDF5 group or dataset may have an associated attribute list. An HDF5 attribute is a user-defined HDF5 structure +that provides extra information about an HDF5 object. + +Working with groups and datasets is similar in many ways to working with directories and files in UNIX. As with UNIX +directories and files, an HDF5 object in an HDF5 file is often referred to by its full path name (also called an absolute path name). + +\li
/ signifies the root group.
+
+\li /foo signifies a member of the root group called foo.
+
+\li /foo/zoo signifies a member of the group foo, which in turn is a member of the root group.
+
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics
+
+@page LBAPI The HDF5 API
+
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics
++ +\section secLBAPI HDF5 C API +The HDF5 library provides several interfaces, or APIs. These APIs provide routines for creating, +accessing, and manipulating HDF5 files and objects. + +The library itself is implemented in C. To facilitate the work of FORTRAN 90, C++ and Java programmers, +HDF5 function wrappers have been developed in each of these languages. This tutorial discusses the use +of the C functions and the FORTRAN wrappers. + +All C routines in the HDF5 library begin with a prefix of the form H5*, where * is one or two uppercase +letters indicating the type of object on which the function operates. +The FORTRAN wrappers come in the form of subroutines that begin with h5 and end with _f. +Java routine names begin with “H5*” and are prefixed with “H5.” as the class. Constants are +in the HDF5Constants class and are prefixed with "HDF5Constants.". +The APIs are listed below: +
| API + | +DESCRIPTION + | +
|---|---|
| H5 + | +Library Functions: general-purpose H5 functions + | +
| H5A + | +Annotation Interface: attribute access and manipulation routines + | +
| H5D + | +Dataset Interface: dataset access and manipulation routines + | +
| H5E + | +Error Interface: error handling routines + | +
| H5F + | +File Interface: file access routines + | +
| H5G + | +Group Interface: group creation and operation routines + | +
| H5I + | +Identifier Interface: identifier routines + | +
| H5L + | +Link Interface: link routines + | +
| H5O + | +Object Interface: object routines + | +
| H5P + | +Property List Interface: object property list manipulation routines + | +
| H5R + | +Reference Interface: reference routines + | +
| H5S + | +Dataspace Interface: dataspace definition and access routines + | +
| H5T + | +Datatype Interface: datatype creation and manipulation routines + | +
| H5Z + | +Compression Interface: compression routine(s) + | +
+ +Keep the following in mind when looking at the example programs included in this tutorial: + +\section LBProgAPI APIs vary with different languages +\li C routines begin with the prefix “H5*” where * is a single letter indicating the object on which the operation is to be performed: +
| File Interface: | +#H5Fopen | +
| Dataset Interface: | +#H5Dopen | +
| File Interface: | +h5fopen_f | +
| Dataset Interface: | +h5dopen_f | +
| File Interface: | +H5.H5Fopen | +
| Dataset Interface: | +H5.H5Dopen | +
-
+
-
+Python: Add
"import h5py / import numpy"+
+ -
+C: Add
"#include hdf5.h"+
+ -
+FORTRAN: Add
"USE HDF5"and call h5open_f and h5close_f to initialize and close the HDF5 FORTRAN interface +
+ -
+Java: Add
"import hdf.hdf5lib.H5; + import hdf.hdf5lib.HDF5Constants;"+
+
+ +An HDF5 file is a binary file containing scientific data and supporting metadata. +\section secLBFileCreate HDF5 File Access +To create an HDF5 file, an application must specify not only a file name, but a file access mode, +a file creation property list, and a file access property list. These terms are described below: +
-
+
- File access mode:
+When creating a file, the file access mode specifies the action to take if the file already exists: +-
+
- #H5F_ACC_TRUNC specifies that if the file already exists, the current contents will be deleted so +that the application can rewrite the file with new data. + +
- #H5F_ACC_EXCL specifies that the open will fail if the file already exists. If the file does not +already exist, the file access parameter is ignored. + +
+Note that there are two different access modes for opening existing files: +-
+
- #H5F_ACC_RDONLY specifies that the application has read access but will not be allowed to write any data. + +
- #H5F_ACC_RDWR specifies that the application has read and write access. + +
+ - File creation property list:
The file creation property list is used to +control the file metadata. File metadata contains information about the size of the user-block*, +the size of various file data structures used by the HDF5 library, etc. In this tutorial, the +default file creation property list, #H5P_DEFAULT, is used.
+ *The user-block is a fixed-length block of data located at the beginning of the file which is +ignored by the HDF5 library. The user-block may be used to store any data or information found +to be useful to applications. +
+ - File access property list:
The file access property list is used to +control different methods of performing I/O on files. It also can be used to control how a file +is closed (whether or not to delay the actual file close until all objects in a file are closed). +The default file access property list, #H5P_DEFAULT, is used in this tutorial. +
+
-
+
- Specify the file creation and access property lists, if necessary. +
- Create the file. +
- Close the file, and if necessary, close the property lists. +
hdf5.h contains definitions and declarations and must be included
+in any program that uses the HDF5 library.
++In FORTRAN: The module
HDF5 contains definitions and declarations and must be used in any
+program that uses the HDF5 library. Also note that #H5open MUST be called at the beginning of an HDF5 Fortran
+application (prior to any HDF5 calls) to initialize the library and variables. The #H5close call MUST be at
+the end of the HDF5 Fortran application.
+\li #H5Fcreate creates an HDF5 file and returns the file identifier.+For Fortran, the file creation property list and file access property list are optional. They can be omitted if the +default values are to be used.
+The root group is automatically created when a file is created. Every file has a root group and the path name of +the root group is always
/.
+\li #H5Fclose terminates access to an HDF5 file.+When an HDF5 file is no longer accessed by a program, #H5Fclose must be called to release the resources used by the file. +This call is mandatory.
+Note that if #H5Fclose is called for a file, but one or more objects within the file remain open, those objects will +remain accessible until they are individually closed. This can cause access problems for other users, if objects were +inadvertently left open. A File Access property controls how the file is closed. + +\subsection subsecLBFileExampleCont File Contents +The HDF Group has developed tools for examining the contents of HDF5 files. The tool used throughout the HDF5 tutorial +is the HDF5 dumper,
h5dump, which displays the file contents in human-readable form. The output of h5dump is an ASCII
+display formatted according to the HDF5 DDL grammar. This grammar is defined, using Backus-Naur Form, in the
+\ref DDLBNF110.
+
+To view the HDF5 file contents, simply type:
+\code
+h5dump | +\image html imgLBFile.gif + | +
file.h5, as generated by h5dump. The HDF5 file called file.h5
+contains a group called /, or the root group. (The file called filef.h5, created by the FORTRAN version of the example,
+has the same output except that the filename shown is filef.h5.)
+\code
+HDF5 "file.h5" {
+ GROUP "/" {
+ }
+ }
+\endcode
+
+\subsection subsecLBFileExampleDDL File Definition in DDL
+The simplified DDL file definition for creating an HDF5 file. For simplicity, a simplified DDL is used in this tutorial. A
+complete and more rigorous DDL can be found in the \ref DDLBNF110.
+
+The following symbol definitions are used in the DDL:
+\code
+ ::= defined as
+ + +A dataset is a multidimensional array of data elements, together with supporting metadata. To create +a dataset, the application program must specify the location at which to create the dataset, the +dataset name, the datatype and dataspace of the data array, and the property lists. + +\section secLBDsetCreateDtype Datatypes +A datatype is a collection of properties, all of which can be stored on disk, and which, when taken as +a whole, provide complete information for data conversion to or from that datatype. + +There are two categories of datatypes in HDF5: +
-
+
- Pre-defined: These datatypes are opened and closed by HDF5.
+Pre-defined datatypes can be atomic or composite: +- Atomic datatypes cannot be decomposed into smaller datatype units at the API level. For example: integer, float, reference, string. +
- Composite datatypes are aggregations of one or more datatypes. For example: array, variable length, enumeration, compound.
+ - Derived: These datatypes are created or derived from the pre-defined types.
+A simple example of creating a derived datatype is using the string datatype, H5T_C_S1, to create strings of more than one character:
+\code + hid_t strtype; // Datatype ID + herr_t status; + + strtype = H5Tcopy (H5T_C_S1); + status = H5Tset_size (strtype, 5); // create string of length 5 +\endcode +
+
| Datatype | +Description | +
|---|---|
| H5T_STD_I32LE | +Four-byte, little-endian, signed, two's complement integer | +
| H5T_STD_U16BE | +Two-byte, big-endian, unsigned integer | +
| H5T_IEEE_F32BE | +Four-byte, big-endian, IEEE floating point | +
| H5T_IEEE_F64LE | +Eight-byte, little-endian, IEEE floating point | +
| H5T_C_S1 | +One-byte, null-terminated string of eight-bit characters | +
| Native Datatype | +Corresponding C or FORTRAN Type | +
|---|---|
| C | +|
| H5T_NATIVE_INT | +int | +
| H5T_NATIVE_FLOAT | +float | +
| H5T_NATIVE_CHAR | +char | +
| H5T_NATIVE_DOUBLE | +double | +
| H5T_NATIVE_LDOUBLE | +long double | +
| Fortran | +|
| H5T_NATIVE_INTEGER | +integer | +
| H5T_NATIVE_REAL | +real | +
| H5T_NATIVE_DOUBLE | +double precision | +
| H5T_NATIVE_CHARACTER | +character | +
+When creating a dataset, HDF5 allows the user to specify how raw data is organized and/or compressed on +disk. This information is stored in a dataset creation property list and passed to the dataset interface. +The raw data on disk can be stored contiguously (in the same linear way that it is organized in memory), +partitioned into chunks, stored externally, etc. In this tutorial, we use the default dataset creation +property list (contiguous storage layout and no compression). For more information about dataset creation +property lists, see \ref sec_dataset in the \ref UG. +\li Link Creation Property List
+The link creation property list governs creation of the link(s) by which a new dataset is accessed and the +creation of any intermediate groups that may be missing. +\li Dataset Access Property List
+Dataset access property lists are properties that can be specified when accessing a dataset. + +\section secLBDsetCreateSteps Steps to Create a Dataset +To create an empty dataset (no data written) the following steps need to be taken: +
-
+
- Obtain the location identifier where the dataset is to be created. +
- Define or specify the dataset characteristics:
+
-
+
- Define a datatype or specify a pre-defined datatype. +
- Define a dataspace. +
- Specify the property list(s) or use the default. +
+ - Create the dataset. +
- Close the datatype, the dataspace, and the property list(s) if necessary. +
- Close the dataset. +
dset.h5
+in the C version (dsetf.h5 in Fortran), defines the dataset dataspace, creates a
+dataset which is a 4x6 integer array, and then closes the dataspace, the dataset, and the file.
+
+For details on compiling an HDF5 application: [ \ref LBCompiling ]
+
+\subsection subsecLBDsetCreateProgRem Remarks
+#H5Screate_simple creates a new simple dataspace and returns a dataspace identifier.
+#H5Sclose releases and terminates access to a dataspace.
+
+C
+\code
+ dataspace_id = H5Screate_simple (rank, dims, maxdims);
+ status = H5Sclose (dataspace_id );
+\endcode
+
+FORTRAN
+\code
+ CALL h5screate_simple_f (rank, dims, dataspace_id, hdferr, maxdims=max_dims)
+ or
+ CALL h5screate_simple_f (rank, dims, dataspace_id, hdferr)
+
+ CALL h5sclose_f (dataspace_id, hdferr)
+\endcode
+
+#H5Dcreate creates an empty dataset at the specified location and returns a dataset identifier.
+#H5Dclose closes the dataset and releases the resource used by the dataset. This call is mandatory.
+
+C
+\code
+ dataset_id = H5Dcreate(file_id, "/dset", H5T_STD_I32BE, dataspace_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
+ status = H5Dclose (dataset_id);
+\endcode
+
+FORTRAN
+\code
+ CALL h5dcreate_f (loc_id, name, type_id, dataspace_id, dset_id, hdferr)
+ CALL h5dclose_f (dset_id, hdferr)
+\endcode
+
+Note that if using the pre-defined datatypes in FORTRAN, then a call must be made to initialize and terminate access to the pre-defined datatypes:
+\code
+ CALL h5open_f (hdferr)
+ CALL h5close_f (hdferr)
+\endcode
+
+H5open must be called before any HDF5 library subroutine calls are made;
+H5close must be called after the final HDF5 library subroutine call.
+
+See the programming example for an illustration of the use of these calls.
+
+\subsection subsecLBDsetCreateContent File Contents
+The contents of the file dset.h5 (dsetf.h5 for FORTRAN) are shown below:
+| +\image html imgLBDsetCreate.gif + | +
| dset.h5 in DDL | +dsetf.h5 in DDL | +
|---|---|
| +\code +HDF5 "dset.h5" { +GROUP "/" { + DATASET "dset" { + DATATYPE { H5T_STD_I32BE } + DATASPACE { SIMPLE ( 4, 6 ) / ( 4, 6 ) } + DATA { + 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0 + } + } +} +} +\endcode + | ++\code +HDF5 "dsetf.h5" { +GROUP "/" { + DATASET "dset" { + DATATYPE { H5T_STD_I32BE } + DATASPACE { SIMPLE ( 6, 4 ) / ( 6, 4 ) } + DATA { + 0, 0, 0, 0, + 0, 0, 0, 0, + 0, 0, 0, 0, + 0, 0, 0, 0, + 0, 0, 0, 0, + 0, 0, 0, 0 + } + } +} +} +\endcode + | +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBDsetRW Reading From and Writing To a Dataset +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBDsetRW Dataset I/O Operation +During a dataset I/O operation, the library transfers raw data between memory and the file. The data in memory +can have a datatype different from that of the file and can also be of a different size (i.e., the data in +memory is a subset of the dataset elements, or vice versa). Therefore, to perform read or write operations, +the application program must specify: +\li The dataset +\li The dataset's datatype in memory +\li The dataset's dataspace in memory +\li The dataset's dataspace in the file +\li The dataset transfer property list
+
-
+
- (The dataset transfer property list controls various aspects of the I/O operations, such as the number +of processes participating in a collective I/O request or hints to the library to control caching of raw +data. In this tutorial, we use the default dataset transfer property list.) +
-
+
- Obtain the dataset identifier. +
- Specify the memory datatype. +
- Specify the memory dataspace. +
- Specify the file dataspace. +
- Specify the transfer properties. +
- Perform the desired operation on the dataset. +
- Close the dataset. +
- Close the dataspace, datatype, and property list if necessary. +
/dset, writes the dataset to the file, then reads
+the dataset back. It then closes the dataset and file.
+
+Note that #H5S_ALL is passed in for both the memory and file dataspace parameters in the read and write calls.
+This indicates that the entire dataspace of the dataset will be read or written to. #H5S_ALL by itself does not
+necessarily have this meaning. See the \ref RM entry for #H5Dread or #H5Dwrite for more information on using #H5S_ALL.
+
+For details on compiling an HDF5 application:
+[ \ref LBCompiling ]
+
+\subsection secLBDsetRWExRem Remarks
+#H5Fopen opens an existing file and returns a file identifier.
+
+#H5Dopen opens an existing dataset with the specified name and location.
+
+#H5Dwrite writes raw data from an application buffer to the specified dataset, converting from the datatype and
+dataspace of the dataset in memory to the datatype and dataspace of the dataset in the file. Specifying #H5S_ALL
+for both the memory and file dataspaces indicates that the entire dataspace of the dataset is to be written to.
+#H5S_ALL by itself does not necessarily have this meaning. See the \ref RM entry for #H5Dwrite for more information
+on using #H5S_ALL.
+
+#H5Dread reads raw data from the specified dataset to an application buffer, converting from the file datatype and
+dataspace to the memory datatype and dataspace. Specifying #H5S_ALL for both the memory and file dataspaces
+indicates that the entire dataspace of the dataset is to be read. #H5S_ALL by itself does not necessarily have
+this meaning. See the \ref RM entry for #H5Dread for more information on using #H5S_ALL.
+
+\subsection secLBDsetRWExCont File Contents
+
+Shown below is the contents of dset.h5 (created by the C program).
+
+dset.h5 in DDL
+\code
+ HDF5 "dset.h5" {
+ GROUP "/" {
+ DATASET "dset" {
+ DATATYPE { H5T_STD_I32BE }
+ DATASPACE { SIMPLE ( 4, 6 ) / ( 4, 6 ) }
+ DATA {
+ 1, 2, 3, 4, 5, 6,
+ 7, 8, 9, 10, 11, 12,
+ 13, 14, 15, 16, 17, 18,
+ 19, 20, 21, 22, 23, 24
+ }
+ }
+ }
+ }
+\endcode
+
+Shown below is the contents of dsetf.h5 (created by the FORTRAN program).
+
+dsetf.h5 in DDL
+\code
+ HDF5 "dsetf.h5" {
+ GROUP "/" {
+ DATASET "dset" {
+ DATATYPE { H5T_STD_I32BE }
+ DATASPACE { SIMPLE ( 6, 4 ) / ( 6, 4 ) }
+ DATA {
+ 1, 7, 13, 19,
+ 2, 8, 14, 20,
+ 3, 9, 15, 21,
+ 4, 10, 16, 22,
+ 5, 11, 17, 23,
+ 6, 12, 18, 24
+ }
+ }
+ }
+ }
+\endcode
+
++Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBAttrCreate Creating an Attribute +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +Attributes are small datasets that can be used to describe the nature and/or the intended usage of +the object they are attached to. In this section, we show how to create, read, and write an attribute. + +\section secLBAttrCreate Creating an attribute +Creating an attribute is similar to creating a dataset. To create an attribute, the application must +specify the object which the attribute is attached to, the datatype and dataspace of the attribute +data, and the attribute creation property list. + +The steps to create an attribute are as follows: +
-
+
- Obtain the object identifier that the attribute is to be attached to. +
- Define the characteristics of the attribute and specify the attribute creation property list.
+
-
+
- Define the datatype. +
- Define the dataspace. +
- Specify the attribute creation property list. +
+ - Create the attribute. +
- Close the attribute and datatype, dataspace, and attribute creation property list, if necessary. +
-
+
- Obtain the attribute identifier. +
- Specify the attribute's memory datatype. +
- Perform the desired operation. +
- Close the memory datatype if necessary. +
dset.h5
+in C (dsetf.h5 in FORTRAN), obtains the identifier of the dataset /dset, defines
+the attribute's dataspace, creates the dataset attribute, writes the attribute, and then closes the attribute's
+dataspace, attribute, dataset, and file.
+
+For details on compiling an HDF5 application:
+[ \ref LBCompiling ]
+
+\subsection secLBAttrCreateRWExRem Remarks
+#H5Acreate creates an attribute which is attached to the object specified by the first parameter, and returns an identifier.
+
+#H5Awrite writes the entire attribute, and returns the status of the write.
+
+When an attribute is no longer accessed by a program, #H5Aclose must be called to release the attribute from use.
+An #H5Aclose/h5aclose_f call is mandatory.
+
+\subsection secLBAttrCreateRWExCont File Contents
+
+Shown below is the contents and the attribute definition of dset.h5 (created by the C program).
+
+dset.h5 in DDL
+\code
+HDF5 "dset.h5" {
+GROUP "/" {
+DATASET "dset" {
+DATATYPE { H5T_STD_I32BE }
+DATASPACE { SIMPLE ( 4, 6 ) / ( 4, 6 ) }
+DATA {
+ 1, 2, 3, 4, 5, 6,
+ 7, 8, 9, 10, 11, 12,
+ 13, 14, 15, 16, 17, 18,
+ 19, 20, 21, 22, 23, 24
+}
+ATTRIBUTE "attr" {
+ DATATYPE { H5T_STD_I32BE }
+ DATASPACE { SIMPLE ( 2 ) / ( 2 ) }
+ DATA {
+ 100, 200
+ }
+}
+}
+}
+}
+\endcode
+
+Shown below is the contents and the attribute definition of dsetf.h5 (created by the FORTRAN program).
+
+dsetf.h5 in DDL
+\code
+HDF5 "dsetf.h5" {
+GROUP "/" {
+DATASET "dset" {
+DATATYPE { H5T_STD_I32BE }
+DATASPACE { SIMPLE ( 6, 4 ) / ( 6, 4 ) }
+DATA {
+ 1, 7, 13, 19,
+ 2, 8, 14, 20,
+ 3, 9, 15, 21,
+ 4, 10, 16, 22,
+ 5, 11, 17, 23,
+ 6, 12, 18, 24
+}
+ATTRIBUTE "attr" {
+ DATATYPE { H5T_STD_I32BE }
+ DATASPACE { SIMPLE ( 2 ) / ( 2 ) }
+ DATA {
+ 100, 200
+ }
+}
+}
+}
+}
+\endcode
+
+\subsection secLBAttrCreateRWExDDL Attribute Definition in DDL
+
+HDF5 Attribute Definition
+\code
++Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +*/ diff --git a/doxygen/dox/LearnBasics2.dox b/doxygen/dox/LearnBasics2.dox new file mode 100644 index 0000000000..ffcb971840 --- /dev/null +++ b/doxygen/dox/LearnBasics2.dox @@ -0,0 +1,1159 @@ +/** @page LBGrpCreate Creating an Group +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBGrpCreate Creating an group +An HDF5 group is a structure containing zero or more HDF5 objects. The two primary HDF5 objects are groups and datasets. To create a group, the calling program must: +
-
+
- Obtain the location identifier where the group is to be created. +
- Create the group. +
- Close the group. +
group.h5 in C
+(groupf.h5 for FORTRAN), creates a group called MyGroup in the root group, and then closes the group and file.
+
+For details on compiling an HDF5 application:
+[ \ref LBCompiling ]
+
+\subsection secLBGrpCreateRWExCont File Contents
+
+Shown below is the contents and the definition of the group of group.h5 (created by the C program).
+(The FORTRAN program creates the HDF5 file groupf.h5 and the resulting DDL shows the filename
+groupf.h5 in the first line.)
+| +\image html imggrpcreate.gif + | +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBGrpCreateNames Creating Groups using Absolute and Relative Names +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +Recall that to create an HDF5 object, we have to specify the location where the object is to be created. +This location is determined by the identifier of an HDF5 object and the name of the object to be created. +The name of the created object can be either an absolute name or a name relative to the specified identifier. +In the previous example, we used the file identifier and the absolute name
/MyGroup to create a group.
+
+In this section, we discuss HDF5 names and show how to use absolute and relative names.
+
+\section secLBGrpCreateNames Names
+HDF5 object names are a slash-separated list of components. There are few restrictions on names: component
+names may be any length except zero and may contain any character except slash (/) and the null terminator.
+A full name may be composed of any number of component names separated by slashes, with any of the component
+names being the special name . (a dot or period). A name which begins with a slash is an absolute name which
+is accessed beginning with the root group of the file; all other names are relative names and and the named
+object is accessed beginning with the specified group. A special case is the name / (or equivalent) which
+refers to the root group.
+
+Functions which operate on names generally take a location identifier, which can be either a file identifier
+or a group identifier, and perform the lookup with respect to that location. Several possibilities are
+described in the following table:
+
+| Location Type | +Object Name | +Description | +
|---|---|---|
| File identifier | +/foo/bar | +The object bar in group foo in the root group. | +
| Group identifier | +/foo/bar | +The object bar in group foo in the root group of the file containing the specified group. +In other words, the group identifier's only purpose is to specify a file. | +
| File identifier | +/ | +The root group of the specified file. | +
| Group identifier | +/ | +The root group of the file containing the specified group. | +
| Group identifier | +foo/bar | +The object bar in group foo in the specified group. | +
| File identifier | +. | +The root group of the file. | +
| Group identifier | +. | +The specified group. | +
| Other identifier | +. | +The specified object. | +
MyGroup in the root group of the specified file.
+
+The second #H5Gcreate/h5gcreate_f creates the group Group_A in the group MyGroup in the root group of the specified
+file. Note that the parent group (MyGroup) already exists.
+
+The third #H5Gcreate/h5gcreate_f creates the group Group_B in the specified group.
+
+\subsection secLBGrpCreateNamesExCont File Contents
+
+Shown below is the contents and the definition of the group of groups.h5 (created by the C program).
+(The FORTRAN program creates the HDF5 file groupsf.h5 and the resulting DDL shows the filename
+groupsf.h5 in the first line.)
+| +\image html imggrps.gif + | +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBGrpDset Creating Datasets in Groups +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBGrpDset Datasets in Groups +We have shown how to create groups, datasets, and attributes. In this section, we show how to create +datasets in groups. Recall that #H5Dcreate creates a dataset at the location specified by a location +identifier and a name. Similar to #H5Gcreate, the location identifier can be a file identifier or a +group identifier and the name can be relative or absolute. The location identifier and the name +together determine the location where the dataset is to be created. If the location identifier and +name refer to a group, then the dataset is created in that group. + +\section secLBGrpDsetEx Programming Example + +\subsection secLBGrpDsetExDesc Description +See \ref LBExamples for the examples used in the \ref LearnBasics tutorial. + +The example shows how to create a dataset in a particular group. It opens the file created in the previous example and creates two datasets: + +For details on compiling an HDF5 application: +[ \ref LBCompiling ] + +\subsection secLBGrpDsetExCont File Contents + +Shown below is the contents and the definition of the group of
groups.h5 (created by the C program).
+(The FORTRAN program creates the HDF5 file groupsf.h5 and the resulting DDL shows the filename
+groupsf.h5 in the first line.)
+| +\image html imggrpdsets.gif + | +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBDsetSubRW Reading From or Writing To a Subset of a Dataset +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBDsetSubRW Dataset Subsets +There are two ways that you can select a subset in an HDF5 dataset and read or write to it: +
- +Hyperslab Selection: The #H5Sselect_hyperslab call selects a logically contiguous +collection of points in a dataspace, or a regular pattern of points or blocks in a dataspace. +
- +Element Selection: The #H5Sselect_elements call selects elements in an array. +
| +\image html LBDsetSubRWProg.png + | +
| +\image html imgLBDsetSubRW11.png + | +
| +\image html imgLBDsetSubRW12.png + | +
| +\image html imgLBDsetSubRW31.png + | +
| +\image html imgLBDsetSubRW32.png + | +
| +\image html imgLBDsetSubRW33.png + | +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBDatatypes Datatype Basics +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBDtype What is a Datatype? +A datatype is a collection of datatype properties which provide complete information for data conversion to or from that datatype. + +Datatypes in HDF5 can be grouped as follows: +\li Pre-Defined Datatypes: These are datatypes that are created by HDF5. They are actually opened +(and closed) by HDF5, and can have a different value from one HDF5 session to the next. +\li Derived Datatypes: These are datatypes that are created or derived from the pre-defined datatypes. +Although created from pre-defined types, they represent a category unto themselves. An example of a commonly used derived +datatype is a string of more than one character. + +\section secLBDtypePre Pre-defined Datatypes +The properties of pre-defined datatypes are: +\li Pre-defined datatypes are opened and closed by HDF5. +\li A pre-defined datatype is a handle and is NOT PERSISTENT. Its value can be different from one HDF5 session to the next. +\li Pre-defined datatypes are Read-Only. +\li As mentioned, other datatypes can be derived from pre-defined datatypes. + +There are two types of pre-defined datatypes, standard (file) and native. + +
Standard
+A standard (or file) datatype can be: +-
+
- Atomic: A datatype which cannot be decomposed into smaller datatype units at the API level.
+The atomic datatypes are:
+
-
+
- integer +
- float +
- string (1-character) +
- date and time +
- bitfield +
- reference +
- opaque +
+ - Composite: An aggregation of one or more datatypes.
+Composite datatypes include:
+
-
+
- array +
- variable length +
- enumeration +
- compound datatypes +
+
| Notes | +
|---|
| +\li Standard pre-defined datatypes are the SAME on all platforms. +\li They are the datatypes that you see in an HDF5 file. +\li They are typically used when creating a dataset. + | +
Native
+Native pre-defined datatypes are used for memory operations, such as reading and writing. They are +NOT THE SAME on different platforms. They are similar to C type names, and are aliased to the +appropriate HDF5 standard pre-defined datatype for a given platform. + +For example, when on an Intel based PC, #H5T_NATIVE_INT is aliased to the standard pre-defined type, +#H5T_STD_I32LE. On a MIPS machine, it is aliased to #H5T_STD_I32BE. +| Notes | +
|---|
| +\li Native datatypes are NOT THE SAME on all platforms. +\li Native datatypes simplify memory operations (read/write). The HDF5 library automatically converts as needed. +\li Native datatypes are NOT in an HDF5 File. The standard pre-defined datatype that a native datatype corresponds +to is what you will see in the file. + | +
Pre-Defined
+The following table shows the native types and the standard pre-defined datatypes they correspond +to. (Keep in mind that HDF5 can convert between datatypes, so you can specify a buffer of a larger +type for a dataset of a given type. For example, you can read a dataset that has a short datatype +into a long integer buffer.) + +| C Type | +HDF5 Memory Type | +HDF5 File Type* | +
|---|---|---|
| Integer | +||
| int | +#H5T_NATIVE_INT | +#H5T_STD_I32BE or #H5T_STD_I32LE | +
| short | +#H5T_NATIVE_SHORT | +#H5T_STD_I16BE or #H5T_STD_I16LE | +
| long | +#H5T_NATIVE_LONG | +#H5T_STD_I32BE, #H5T_STD_I32LE, + #H5T_STD_I64BE or #H5T_STD_I64LE | +
| long long | +#H5T_NATIVE_LLONG | +#H5T_STD_I64BE or #H5T_STD_I64LE | +
| unsigned int | +#H5T_NATIVE_UINT | +#H5T_STD_U32BE or #H5T_STD_U32LE | +
| unsigned short | +#H5T_NATIVE_USHORT | +#H5T_STD_U16BE or #H5T_STD_U16LE | +
| unsigned long | +#H5T_NATIVE_ULONG | +#H5T_STD_U32BE, #H5T_STD_U32LE, + #H5T_STD_U64BE or #H5T_STD_U64LE | +
| unsigned long long | +#H5T_NATIVE_ULLONG | +#H5T_STD_U64BE or #H5T_STD_U64LE | +
| Float | +||
| float | +#H5T_NATIVE_FLOAT | +#H5T_IEEE_F32BE or #H5T_IEEE_F32LE | +
| double | +#H5T_NATIVE_DOUBLE | +#H5T_IEEE_F64BE or #H5T_IEEE_F64LE | +
| F90 Type | +HDF5 Memory Type | +HDF5 File Type* | +
|---|---|---|
| integer | +H5T_NATIVE_INTEGER | +#H5T_STD_I32BE(8,16) or #H5T_STD_I32LE(8,16) | +
| real | +H5T_NATIVE_REAL | +#H5T_IEEE_F32BE or #H5T_IEEE_F32LE | +
| double-precision | +#H5T_NATIVE_DOUBLE | +#H5T_IEEE_F64BE or #H5T_IEEE_F64LE | +
| * Note that the HDF5 File Types listed are those that are most commonly created. + The file type created depends on the compiler switches and platforms being + used. For example, on the Cray an integer is 64-bit, and using #H5T_NATIVE_INT (C) + or H5T_NATIVE_INTEGER (F90) would result in an #H5T_STD_I64BE file type. | +
-
+
- Make a copy of the pre-defined datatype: +\code + tid = H5Tcopy (H5T_STD_I32BE); +\endcode + +
- Change the datatype. +
Creating
+The function #H5Tarray_create creates a new array datatype object. Parameters specify +\li the base datatype of each element of the array, +\li the rank of the array, i.e., the number of dimensions, +\li the size of each dimension, and +\li the dimension permutation of the array, i.e., whether the elements of the array are listed in C or FORTRAN order. + +Working with existing array datatypes
+When working with existing arrays, one must first determine the the rank, or number of dimensions, of the array. + +The function #H5Tget_array_dims returns the rank of a specified array datatype. + +In many instances, one needs further information. The function #H5Tget_array_dims retrieves the +permutation of the array and the size of each dimension. + +\subsection subsecLBDtypeSpecCmpd Compound + +\subsubsection subsubsecLBDtypeSpecCmpdProp Properties of compound datatypes +A compound datatype is similar to a struct in C or a common block in Fortran. It is a collection of +one or more atomic types or small arrays of such types. To create and use of a compound datatype +you need to refer to various properties of the data compound datatype: +\li It is of class compound. +\li It has a fixed total size, in bytes. +\li It consists of zero or more members (defined in any order) with unique names and which occupy non-overlapping regions within the datum. +\li Each member has its own datatype. +\li Each member is referenced by an index number between zero and N-1, where N is the number of members in the compound datatype. +\li Each member has a name which is unique among its siblings in a compound datatype. +\li Each member has a fixed byte offset, which is the first byte (smallest byte address) of that member in a compound datatype. +\li Each member can be a small array of up to four dimensions. + +Properties of members of a compound datatype are defined when the member is added to the compound type and cannot be subsequently modified. + +\subsubsection subsubsecLBDtypeSpecCmpdDef Defining compound datatypes +Compound datatypes must be built out of other datatypes. First, one creates an empty compound +datatype and specifies its total size. Then members are added to the compound datatype in any order. + +Member names. Each member must have a descriptive name, which is the key used to uniquely identify +the member within the compound datatype. A member name in an HDF5 datatype does not necessarily +have to be the same as the name of the corresponding member in the C struct in memory, although +this is often the case. Nor does one need to define all members of the C struct in the HDF5 +compound datatype (or vice versa). + +Offsets. Usually a C struct will be defined to hold a data point in memory, and the offsets of the +members in memory will be the offsets of the struct members from the beginning of an instance of the +struct. The library defines the macro to compute the offset of a member within a struct: +\code + HOFFSET(s,m) +\endcode +This macro computes the offset of member m within a struct variable s. + +Here is an example in which a compound datatype is created to describe complex numbers whose type +is defined by the complex_t struct. +\code +typedef struct { + double re; /*real part */ + double im; /*imaginary part */ +} complex_t; + +complex_t tmp; /*used only to compute offsets */ +hid_t complex_id = H5Tcreate (H5T_COMPOUND, sizeof tmp); +H5Tinsert (complex_id, "real", HOFFSET(tmp,re), H5T_NATIVE_DOUBLE); +H5Tinsert (complex_id, "imaginary", HOFFSET(tmp,im), H5T_NATIVE_DOUBLE); +\endcode + +\subsection subsecLBDtypeSpecRef Reference +There are two types of Reference datatypes in HDF5: +\li \ref subsubsecLBDtypeSpecRefObj +\li \ref subsubsecLBDtypeSpecRefDset + +\subsubsection subsubsecLBDtypeSpecRefObj Reference to objects +In HDF5, objects (i.e. groups, datasets, and named datatypes) are usually accessed by name. +There is another way to access stored objects -- by reference. + +An object reference is based on the relative file address of the object header in the file +and is constant for the life of the object. Once a reference to an object is created and +stored in a dataset in the file, it can be used to dereference the object it points to. +References are handy for creating a file index or for grouping related objects by storing +references to them in one dataset. + +Creating and storing references to objects
+The following steps are involved in creating and storing file references to objects: +-
+
- Create the objects or open them if they already exist in the file. +
- Create a dataset to store the objects' references, by specifying #H5T_STD_REF_OBJ as the datatype +
- Create and store references to the objects in a buffer, using #H5Rcreate. +
- Write a buffer with the references to the dataset, using #H5Dwrite with the #H5T_STD_REF_OBJ datatype. +
Reading references and accessing objects using references
+The following steps are involved: +-
+
- Open the dataset with the references and read them. The #H5T_STD_REF_OBJ datatype must be used to describe the memory datatype. +
- Use the read reference to obtain the identifier of the object the reference points to using #H5Rdereference. +
- Open the dereferenced object and perform the desired operations. +
- Close all objects when the task is complete. +
Creating and storing references to dataset regions
+The following steps are involved in creating and storing references to a dataset region: +\li Create a dataset to store the dataset region (selection), by passing in #H5T_STD_REF_DSETREG for the datatype when calling #H5Dcreate. +\li Create selection(s) in existing dataset(s) using #H5Sselect_hyperslab and/or #H5Sselect_elements. +\li Create reference(s) to the selection(s) using #H5Rcreate and store them in a buffer. +\li Write the references to the dataset regions in the file. +\li Close all objects. + +Reading references to dataset regions
+The following steps are involved in reading references to dataset regions and referenced dataset regions (selections). +-
+
- Open and read the dataset containing references to the dataset regions. +The datatype #H5T_STD_REF_DSETREG must be used during read operation. +
- Use #H5Rdereference to obtain the dataset identifier from the read dataset region reference. + OR + Use #H5Rget_region to obtain the dataspace identifier for the dataset containing the selection from the read dataset region reference. + +
- With the dataspace identifier, the \ref H5S interface functions, H5Sget_select_*, +can be used to obtain information about the selection. +
- Close all objects when they are no longer needed. +
| Function | +Description | +
|---|---|
| #H5Sget_select_npoints | +Returns the number of elements in the hyperslab | +
| #H5Sget_select_hyper_nblocks | +Returns the number of blocks in the hyperslab | +
| #H5Sget_select_hyper_blocklist | +Returns the "lower left" and "upper right" coordinates of the blocks in the hyperslab selection | +
| #H5Sget_select_bounds | +Returns the coordinates of the "minimal" block containing a hyperslab selection | +
| #H5Sget_select_elem_npoints | +Returns the number of points in the element selection | +
| #H5Sget_select_elem_pointlist | +Returns the coordinates of points in the element selection | +
-
+
- Ragged arrays: Multi-dimensional ragged arrays can be implemented with the last (fastest changing) +dimension being ragged by using a VL datatype as the type of the element stored. (Or as a field in a compound datatype.) + +
- Fractal arrays: If a compound datatype has a VL field of another compound type with VL fields +(a nested VL datatype), this can be used to implement ragged arrays of ragged arrays, to whatever +nesting depth is required for the user. + +
- Polygon lists: A common storage requirement is to efficiently store arrays of polygons with +different numbers of vertices. VL datatypes can be used to efficiently and succinctly describe an +array of polygons with different numbers of vertices. + +
- Character strings: Perhaps the most common use of VL datatypes will be to store C-like VL character +strings in dataset elements or as attributes of objects. + +
- Indices: An array of VL object references could be used as an index to all the objects in a file
+which contain a particular sequence of dataset values. Perhaps an array something like the following:
+\code
+ Value1: Object1, Object3, Object9
+ Value2: Object0, Object12, Object14, Object21, Object22
+ Value3: Object2
+ Value4:
+ Value5: Object1, Object10, Object12 + . + . +\endcode +
+ - Object Tracking: An array of VL dataset region references can be used as a method of tracking
+objects or features appearing in a sequence of datasets. Perhaps an array of them would look like:
+\code
+ Feature1: Dataset1:Region, Dataset3:Region, Dataset9:Region
+ Feature2: Dataset0:Region, Dataset12:Region, Dataset14:Region,
+ Dataset21:Region, Dataset22:Region
+ Feature3: Dataset2:Region
+ Feature4:
+ Feature5: Dataset1:Region, Dataset10:Region, Dataset12:Region + . + . +\endcode +
+
Creation
+VL datatypes are created with the #H5Tvlen_create function as follows: +\code +type_id = H5Tvlen_create(hid_t base_type_id); +\endcode +The base datatype will be the datatype that the sequence is composed of, characters for character +strings, vertex coordinates for polygon lists, etc. The base datatype specified for the VL datatype +can be of any HDF5 datatype, including another VL datatype, a compound datatype, or an atomic datatype. + +Querying base datatype of VL datatype
+It may be necessary to know the base datatype of a VL datatype before memory is allocated, etc. +The base datatype is queried with the #H5Tget_super function, described in the \ref H5T documentation. + +Querying minimum memory required for VL information
+It order to predict the memory usage that #H5Dread may need to allocate to store VL data while +reading the data, the #H5Dvlen_get_buf_size function is provided: +\code +herr_t H5Dvlen_get_buf_size(hid_t dataset_id, hid_t type_id, hid_t space_id, hsize_t *size) +\endcode +This routine checks the number of bytes required to store the VL data from the dataset, using +the \em space_id for the selection in the dataset on disk and the \em type_id for the memory representation +of the VL data in memory. The *\em size value is modified according to how many bytes are required +to store the VL data in memory. + +Specifying how to manage memory for the VL datatype
+The memory management method is determined by dataset transfer properties passed into the +#H5Dread and #H5Dwrite functions with the dataset transfer property list. + +Default memory management is set by using #H5P_DEFAULT for the dataset transfer +property list identifier. If #H5P_DEFAULT is used with #H5Dread, the system \em malloc and \em free +calls will be used for allocating and freeing memory. In such a case, #H5P_DEFAULT should +also be passed as the property list identifier to #H5Dvlen_reclaim. + +The rest of this subsection is relevant only to those who choose not to use default memory management. + +The user can choose whether to use the system \em malloc and \em free calls or user-defined, or custom, +memory management functions. If user-defined memory management functions are to be used, the +memory allocation and free routines must be defined via #H5Pset_vlen_mem_manager(), as follows: +\code +herr_t H5Pset_vlen_mem_manager(hid_t plist_id, H5MM_allocate_t alloc, void *alloc_info, H5MM_free_t free, void *free_info) +\endcode +The \em alloc and \em free parameters identify the memory management routines to be used. If the user +has defined custom memory management routines, \em alloc and/or \em free should be set to make those +routine calls (i.e., the name of the routine is used as the value of the parameter); if the user +prefers to use the system's \em malloc and/or \em free, the \em alloc and \em free parameters, respectively, should be set to \em NULL + +The prototypes for the user-defined functions would appear as follows: +\code +typedef void *(*H5MM_allocate_t)(size_t size, void *info) ; typedef void (*H5MM_free_t)(void *mem, void *free_info) ; +\endcode +The \em alloc_info and \em free_info parameters can be used to pass along any required information to +the user's memory management routines. + +In summary, if the user has defined custom memory management routines, the name(s) of the routines +are passed in the \em alloc and \em free parameters and the custom routines' parameters are passed in the +\em alloc_info and \em free_info parameters. If the user wishes to use the system \em malloc and \em free functions, +the \em alloc and/or \em free parameters are set to \em NULL and the \em alloc_info and \em free_info parameters are ignored. + +Recovering memory from VL buffers read in
+The complex memory buffers created for a VL datatype may be reclaimed with the #H5Dvlen_reclaim +function call, as follows: +\code +herr_t H5Dvlen_reclaim(hid_t type_id, hid_t space_id, hid_t plist_id, void *buf); +\endcode + +The \em type_id must be the datatype stored in the buffer, \em space_id describes the selection for the +memory buffer to free the VL datatypes within, \em plist_id is the dataset transfer property list +which was used for the I/O transfer to create the buffer, and \em buf is the pointer to the buffer +to free the VL memory within. The VL structures (#hvl_t) in the user's buffer are modified to zero +out the VL information after it has been freed. + +If nested VL datatypes were used to create the buffer, this routine frees them from the bottom up, +releasing all the memory without creating memory leaks. + ++Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +*/ diff --git a/doxygen/dox/LearnBasics3.dox b/doxygen/dox/LearnBasics3.dox new file mode 100644 index 0000000000..2fe0f5249d --- /dev/null +++ b/doxygen/dox/LearnBasics3.dox @@ -0,0 +1,1015 @@ +/** @page LBPropsList Property Lists Basics +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBPList What is a Property (or Property List)? +In HDF5, a property or property list is a characteristic or feature associated with an HDF5 object. +There are default properties which handle the most common needs. These default properties are +specified by passing in #H5P_DEFAULT for the Property List parameter of a function. Default properties +can be modified by use of the \ref H5P interface and function parameters. + +The \ref H5P API allows a user to take advantage of the more powerful features in HDF5. It typically +supports unusual cases when creating or accessing HDF5 objects. There is a programming model for +working with Property Lists in HDF5 (see below). + +For examples of modifying a property list, see these tutorial topics: +\li \see \ref LBDsetLayout +\li \see \ref LBExtDset +\li \see \ref LBComDset + +There are many Property Lists associated with creating and accessing objects in HDF5. See the +\ref H5P Interface documentation in the HDF5 \ref RM for a list of the different +properties associated with HDF5 interfaces. + +In summary: +\li Properties are features of HDF5 objects, that can be changed by use of the Property List API and function parameters. +\li Property lists provide a mechanism for adding functionality to HDF5 calls without increasing the number of arguments used for a given call. +\li The Property List API supports unusual cases when creating and accessing HDF5 objects. + +\section secLBPListProg Programming Model +Default properties are specified by simply passing in #H5P_DEFAULT (C) / H5P_DEFAULT_F (F90) for +the property list parameter in those functions for which properties can be changed. + +The programming model for changing a property list is as follows: +\li Create a copy or "instance" of the desired pre-defined property type, using the #H5Pcreate call. This +will return a property list identifier. Please see the \ref RM entry for #H5Pcreate, for a comprehensive +list of the property types. +\li With the property list identifier, modify the property, using the \ref H5P APIs. +\li Modify the object feature, by passing the property list identifier into the corresponding HDF5 object function. +\li Close the property list when done, using #H5Pclose. + +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBDsetLayout Dataset Storage Layout +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBDsetLayoutDesc Description of a Dataset + +\section secLBDsetLayout Dataset Storage Layout +The storage information, or storage layout, defines how the raw data values in the dataset are +physically stored on disk. There are three ways that a dataset can be stored: +\li contiguous +\li chunked +\li compact + +See the #H5Pset_layout/#H5Pget_layout APIs for details. + +\subsection subsecLBDsetLayoutCont Contiguous +If the storage layout is contiguous, then the raw data values will be stored physically adjacent +to each other in the HDF5 file (in one contiguous block). This is the default layout for a dataset. +In other words, if you do not explicitly change the storage layout for the dataset, then it will +be stored contiguously. +
| +\image html tutr-locons.png + | +
| +\image html tutr-lochk.png + | +
| +\image html tutr-lochks.png + | +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + + +@page LBExtDset Extendible Datasets +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBExtDsetCreate Creating an Extendible Dataset +An extendible dataset is one whose dimensions can grow. HDF5 allows you to define a dataset to have +certain initial dimensions, then to later increase the size of any of the initial dimensions. + +HDF5 requires you to use chunking to define extendible datasets. This makes it possible to extend +datasets efficiently without having to excessively reorganize storage. (To use chunking efficiently, +be sure to see the advanced topic, Chunking in HDF5.) + +The following operations are required in order to extend a dataset: +\li Declare the dataspace of the dataset to have unlimited dimensions for all dimensions that might eventually be extended. +\li Set dataset creation properties to enable chunking. +\li Create the dataset. +\li Extend the size of the dataset. + +\section secLBExtDsetProg Programming Example + +\subsection subsecLBExtDsetProgDesc Description +See \ref LBExamples for the examples used in the \ref LearnBasics tutorial. + +The example shows how to create a 3 x 3 extendible dataset, write to that dataset, extend the dataset +to 10x3, and write to the dataset again. + +For details on compiling an HDF5 application: +[ \ref LBCompiling ] + +\subsection subsecLBExtDsetProgRem Remarks +\li An unlimited dimension dataspace is specified with the #H5Screate_simple call, by passing in +#H5S_UNLIMITED as an element of the maxdims array. +\li The #H5Pcreate call creates a new property as an instance of a property list class. For creating +an extendible array dataset, pass in #H5P_DATASET_CREATE for the property list class. +\li The #H5Pset_chunk call modifies a Dataset Creation Property List instance to store a chunked +layout dataset and sets the size of the chunks used. +\li To extend an unlimited dimension dataset use the the #H5Dset_extent call. Please be aware that +after this call, the dataset's dataspace must be refreshed with #H5Dget_space before more data can be accessed. +\li The #H5Pget_chunk call retrieves the size of chunks for the raw data of a chunked layout dataset. +\li Once there is no longer a need for a Property List instance, it should be closed with the #H5Pclose call. + +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBComDset Compressed Datasets +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBComDsetCreate Creating a Compressed Dataset +HDF5 requires you to use chunking to create a compressed dataset. (To use chunking efficiently, +be sure to see the advanced topic, Chunking in HDF5.) + +The following operations are required in order to create a compressed dataset: +\li Create a dataset creation property list. +\li Modify the dataset creation property list instance to enable chunking and to enable compression. +\li Create the dataset. +\li Close the dataset creation property list and dataset. + +For more information on compression, see the FAQ question on Using Compression in HDF5. + +\section secLBComDsetProg Programming Example + +\subsection subsecLBComDsetProgDesc Description +See \ref LBExamples for the examples used in the \ref LearnBasics tutorial. + +The example creates a chunked and ZLIB compressed dataset. It also includes comments for what needs +to be done to create an SZIP compressed dataset. The example then reopens the dataset, prints the +filter information, and reads the dataset. + +For details on compiling an HDF5 application: +[ \ref LBCompiling ] + +\subsection subsecLBComDsetProgRem Remarks +\li The #H5Pset_chunk call modifies a Dataset Creation Property List instance to store a chunked layout +dataset and sets the size of the chunks used. +\li The #H5Pset_deflate call modifies the Dataset Creation Property List instance to use ZLIB or DEFLATE +compression. The #H5Pset_szip call modifies it to use SZIP compression. There are different compression +parameters required for each compression method. +\li SZIP compression can only be used with atomic datatypes that are integer, float, or char. It cannot be +applied to compound, array, variable-length, enumerations, or other user-defined datatypes. The call +to #H5Dcreate will fail if attempting to create an SZIP compressed dataset with a non-allowed datatype. +The conflict can only be detected when the property list is used. + +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBContents Discovering the Contents of an HDF5 File +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBContents Discovering what is in an HDF5 file +HDFView and h5dump are standalone tools which cannot be called within an application, and using +#H5Dopen and #H5Dread require that you know the name of the HDF5 dataset. How would an application +that has no prior knowledge of an HDF5 file be able to determine or discover the contents of it, +much like HDFView and h5dump? + +The answer is that there are ways to discover the contents of an HDF5 file, by using the +\ref H5G, \ref H5L and \ref H5O APIs: +\li The \ref H5G interface (covered earlier) consists of routines for working with groups. A group is +a structure that can be used to organize zero or more HDF5 objects, not unlike a Unix directory. +\li The \ref H5L interface consists of link routines. A link is a path between groups. The \ref H5L interface +allows objects to be accessed by use of these links. +\li The \ref H5O interface consists of routines for working with objects. Datasets, groups, and committed +datatypes are all objects in HDF5. + +Interface routines that simplify the process: +\li #H5Literate traverses the links in a specified group, in the order of the specified index, using a +user-defined callback routine. (A callback function is one that will be called when a certain condition +is met, at a certain point in the future.) +\li #H5Ovisit / #H5Lvisit recursively visit all objects/links accessible from a specified object/group. + + +\section secLBContentsProg Programming Example + +\subsection subsecLBContentsProgUsing Using #H5Literate, #H5Lvisit and #H5Ovisit +For example code, see the \ref HDF5Examples page. +Specifically look at the Examples by API. +There are examples for different languages, where examples of using #H5Literate and #H5Ovisit/#H5Lvisit are included. + +The h5ex_g_traverse example traverses a file using H5Literate: +\li C: h5ex_g_traverse.c +\li F90: h5ex_g_traverse_F03.f90 + +The h5ex_g_visit example traverses a file using H5Ovisit and H5Lvisit: +\li C: h5ex_g_visit.c +\li F90: h5ex_g_visit_F03.f90 + +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBQuiz Learning the basics QUIZ +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\ref LBFileOrg +
-
+
- Name and describe the two primary objects that can be stored in an HDF5 file. + +
- What is an attribute? + +
- Give the path name for an object called
harrythat is a member of a group calleddick, which, in turn, is a member of the root group. +
+
-
+
- Describe the purpose of each of the following HDF5 APIs: +\code + H5A, H5D, H5E, H5F, H5G, H5T, H5Z +\endcode + +
-
+
- What two HDF5 routines must be called to create an HDF5 file? + +
- What include file must be included in any file that uses the HDF5 library? + +
- An HDF5 file is never completely empty because as soon as it is created, it automatically contains a certain primary object. What is that object? + +
-
+
- Name and describe two major datatype categories. + +
- List the HDF5 atomic datatypes. Give an example of a predefined datatype. How would you create a string dataset? + +
- What does the dataspace describe? What are the major characteristics of the simple dataspace? + +
- What information needs to be passed to the #H5Dcreate function, i.e., what information is needed to describe a dataset at creation time? + +
-
+
- What are six pieces of information which need to be specified for reading and writing a dataset? + +
- Why are both the memory dataspace and file dataspace needed for read/write operations, while only the memory datatype is required? + +
- In Figure 6.1, what does this line mean? +\code +DATASPACE { SIMPLE (4 , 6 ) / ( 4 , 6 ) } +\endcode + +
-
+
- What is an attribute? + +
- Can partial I/O operations be performed on attributes? + +
-
+
- What are the two primary objects that can be included in a group? + +
-
+
- Group names can be specified in two ways. What are these two types of group names? + +
- You have a dataset named
mooin the groupboo, which is in the groupfoo, which, in turn, +is in therootgroup. How would you specify an absolute name to access this dataset? +
+
-
+
- Describe a way to access the dataset moo described in the previous section +(question 2) using a relative name. Describe a way to access the same dataset using an absolute name. + +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBQuizAnswers Learning the basics QUIZ with Answers +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\ref LBFileOrg +
-
+
- Name and describe the two primary objects that can be stored in an HDF5 file.
+
+
++ +Answer + +Group: A grouping structure containing zero or more HDF5 objects, together with supporting metadata. +
+Dataset: A multidimensional array of data elements, together with supporting metadata. +
+ - What is an attribute?
+
+
++ +Answer + +An HDF5 attribute is a user-defined HDF5 structure that provides extra information about an HDF5 object. + +
+ - Give the path name for an object called
harrythat is a member of a group calleddick, which, in turn, is a member of the root group. ++
++ +Answer + +/dick/harry + +
+
-
+
- Describe the purpose of each of the following HDF5 APIs:
+\code
+ H5A, H5D, H5E, H5F, H5G, H5T, H5Z
+\endcode
+
+
++ +Answer + +H5A: Attribute access and manipulation routines + +
+H5D: Dataset access and manipulation routines +
+H5E: Error handling routines H5F: File access routines +
+H5G: Routines for creating and operating on groups +
+H5T: Routines for creating and manipulating the datatypes of dataset elements +
+H5Z: Data compression routines +
+
-
+
- What two HDF5 routines must be called to create an HDF5 file?
+
+
++ +Answer + +#H5Fcreate and #H5Fclose. + +
+ - What include file must be included in any file that uses the HDF5 library?
+
+
++ +Answer + +hdf5.h must be included because it contains definitions and declarations used by the library. + +
+ - An HDF5 file is never completely empty because as soon as it is created, it automatically contains a certain primary object. What is that object?
+
+
++ +Answer + +The root group. + +
+
-
+
- Name and describe two major datatype categories.
+
+
++ +Answer + +Atomic datatype: An atomic datatype cannot be decomposed into smaller units at the API level. + +
+Compound datatype: A compound datatype is a collection of atomic and compound datatypes, or small arrays of such types. +
+ - List the HDF5 atomic datatypes. Give an example of a predefined datatype. How would you create a string dataset?
+
+
++ +Answer + +There are six HDF5 atomic datatypes: integer, floating point, date and time, character string, bit field, and opaque. + +
+Examples of predefined datatypes include the following:
+\li #H5T_IEEE_F32LE - 4-byte little-endian, IEEE floating point +\li #H5T_NATIVE_INT - native integer + +You would create a string dataset with the #H5T_C_S1 datatype, and set the size of the string with the #H5Tset_size call. +
+ - What does the dataspace describe? What are the major characteristics of the simple dataspace?
+
+
++ +Answer + +The dataspace describes the dimensionality of the dataset. A simple dataspace is characterized by its rank and dimension sizes. + +
+ - What information needs to be passed to the #H5Dcreate function, i.e., what information is needed to describe a dataset at creation time?
+
+
++ +Answer + +The dataset location, name, dataspace, datatype, and dataset creation property list. + +
+
-
+
- What are six pieces of information which need to be specified for reading and writing a dataset?
+
+
++ +Answer + +The dataset identifier, the dataset's datatype and dataspace in memory, the dataspace in the file, +the dataset transfer property list, and a data buffer. + +
+ - Why are both the memory dataspace and file dataspace needed for read/write operations, while only the memory datatype is required?
+
+
++ +Answer + +A dataset's file datatype is not required for a read/write operation because the file datatype is specified +when the dataset is created and cannot be changed. Both file and memory dataspaces are required for dataset +subsetting and for performing partial I/O operations. + +
+ - In Figure 6.1, what does this line mean?
+\code
+DATASPACE { SIMPLE (4 , 6 ) / ( 4 , 6 ) }
+\endcode
+
+
++ +Answer + +It means that the dataset dset has a simple dataspace with the current dimensions (4,6) and the maximum size of the dimensions (4,6). + +
+
-
+
- What is an attribute?
+
+
++ +Answer + +An attribute is a dataset attached to an object. It describes the nature and/or the intended usage of the object. + +
+ - Can partial I/O operations be performed on attributes?
+
+
++ +Answer + +No. + +
+
-
+
- What are the two primary objects that can be included in a group?
+
+
++ +Answer + +A group and a dataset. + +
+
-
+
- Group names can be specified in two ways. What are these two types of group names?
+
+
++ +Answer + +Relative and absolute. + +
+ - You have a dataset named
mooin the groupboo, which is in the groupfoo, which, in turn, +is in therootgroup. How would you specify an absolute name to access this dataset? ++
++ +Answer + +/foo/boo/moo + +
+
-
+
- Describe a way to access the dataset moo described in the previous section
+(question 2) using a relative name. Describe a way to access the same dataset using an absolute name.
+
+
++ +Answer + +Access the group /foo and get the group ID. Access the group boo using the group ID obtained in Step 1. +Access the dataset moo using the group ID obtained in Step 2. +\code +gid = H5Gopen (file_id, "/foo", 0); /* absolute path */ +gid1 = H5Gopen (gid, "boo", 0); /* relative path */ +did = H5Dopen (gid1, "moo"); /* relative path */ +\endcode +Access the group /foo and get the group ID. Access the dataset boo/moo with the group ID just obtained. +\code +gid = H5Gopen (file_id, "/foo", 0); /* absolute path */ +did = H5Dopen (gid, "boo/moo"); /* relative path */ +\endcode +Access the dataset with an absolute path. +\code +did = H5Dopen (file_id, "/foo/boo/moo"); /* absolute path */ +\endcode + +
+
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBCompiling Compiling HDF5 Applications +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +\section secLBCompiling Tools and Instructions on Compiling +Compiling applications to use the HDF5 Library can be as simple as executing: +\code +h5cc -o myprog myprog.c +\endcode + +As an application's file base evolves, there are better solutions using autotools and makefiles or +CMake and CMakeLists.txt files. Many tutorials and references can be found with a simple search. + +This tutorial section will discuss the use of compile scripts on Linux. +See the \ref secLBCompilingVS section for compiling with Visual Studio. + +\section secLBCompilingLinux Compile Scripts +When the library is built, the following compile scripts are included: +\li h5cc: compile script for HDF5 C programs +\li h5fc: compile script for HDF5 F90 programs +\li h5c++: compile script for HDF5 C++ programs + +These scripts are easilye used to compile single file applications, such as those included in the tutorial. +
| Warning + | +The h5cc/h5fc/h5c++ compile scripts are included when building with configure. Versions of +these compile scripts have also been added to CMake for Linux ONLY. The CMake versions rely on pkgconfig files. + | +
|---|
Examples of Using the Unix Compile Scripts:
+Following are examples of compiling and running an application with the Unix compile scripts: +\code + h5fc myprog.f90 + ./a.out + + h5cc -o myprog myprog.c + ./myprog +\endcode + +To see how the libraries linked in with a compile script were configured and built, use the +-showconfig option. For example, if using h5cc type: +\code + h5cc -showconfig +\endcode + +Detailed Description of Unix Compile Scripts:
+The h5cc, h5c++, and h5fc compile scripts come with the HDF5 binary distributions (include files, +libraries, and utilities) for the platforms we support. The h5c++ and h5fc utilities are ONLY present +if the library was built with C++ and Fortran. + +\section secLBCompilingVS Using Visual Studio + + 1. If you are building on 64-bit Windows, find the "Platform" dropdown + and select "x64". Also select the correct Configuration (Debug, Release, RelWithDebInfo, etc) + + 2. Set up path for external headers + + The HDF5 install path settings will need to be in the project property sheets per project. + Go to "Project" and select "Properties", find "Configuration Properties", + and then "C/C++". + + 2.1 Add the header path to the "Additional Include Directories" setting. Under "C/C++" + find "General" and select "Additional Include Directories". Select "Edit" from the dropdown + and add the HDF5 install/include path to the list. + (Ex: "C:\Program Files\HDF_Group\HDF5\1.10.9\include") + + 2.2 Building applications with the dynamic/shared hdf5 libraries requires + that the "H5_BUILT_AS_DYNAMIC_LIB" compile definition be used. Under "C/C++" + find "Preprocessor" and select "Preprocessor Definitions". Select "Edit" from the dropdown + and add "H5_BUILT_AS_DYNAMIC_LIB" to the list. + + 3. Set up path for external libraries + + The HDF5 install path/lib settings will need to be in the project property sheets per project. + Go to "Project" and select "Properties", find "Configuration Properties", + and then "Linker". + + 3.1 Add the libraries to the "Additional Dependencies" setting. Under "Linker" + find "Input" and select "Additional Dependencies". Select "Edit" from the dropdown + and add the required HDF5 install/lib path to the list. + (Ex: "C:\Program Files\HDF_Group\HDF5\1.10.9\lib\hdf5.lib") + + 3.2 For static builds, the external libraries should be added. + For example, to compile a C++ application, enter: + libhdf5_cpp.lib libhdf5.lib libz.lib libszaec.lib libaec.lib + +\section secLBCompilingLibs HDF5 Libraries +Following are the libraries included with HDF5. Whether you are using the Unix compile scripts or +Makefiles, or are compiling on Windows, these libraries are or may need to be specified. The order +they are specified is important on Linux: + +| Library | +Linux Name | +Mac Name | +Windows Name | +
|---|---|---|---|
| +\code +HDF5 High Level C++ APIs +HDF5 C++ Library +HDF5 High Level Fortran APIs +HDF5 Fortran Library +HDF5 High Level C APIs +HDF5 C Library +\endcode + | ++\code +libhdf5_hl_cpp.a +libhdf5_cpp.a +libhdf5hl_fortran.a +libhdf5_fortran.a +libhdf5_hl.a +libhdf5.a +\endcode + | ++\code +libhdf5_hl_cpp.a +libhdf5_cpp.a +libhdf5hl_fortran.a +libhdf5_fortran.a +libhdf5_hl.a +libhdf5.a +\endcode + | ++Windows +\code +libhdf5_hl_cpp.lib +libhdf5_cpp.lib +libhdf5hl_fortran.lib +libhdf5_fortran.lib +libhdf5_hl.lib +libhdf5.lib +\endcode + |
| Library | +Linux Name | +Mac Name | +Windows Name | +
|---|---|---|---|
| +\code +HDF5 High Level C++ APIs +HDF5 C++ Library +HDF5 High Level Fortran APIs +HDF5 Fortran Library +HDF5 High Level C APIs +HDF5 C Library +\endcode + | ++\code +libhdf5_hl_cpp.so +libhdf5_cpp.so +libhdf5hl_fortran.so +libhdf5_fortran.so +libhdf5_hl.so +libhdf5.so +\endcode + | ++\code +libhdf5_hl_cpp.dylib +libhdf5_cpp.dylib +libhdf5hl_fortran.dylib +libhdf5_fortran.dylib +libhdf5_hl.dylib +libhdf5.dylib +\endcode + | ++\code +hdf5_hl_cpp.lib +hdf5_cpp.lib +hdf5hl_fortran.lib +hdf5_fortran.lib +hdf5_hl.lib +hdf5.lib +\endcode + |
| Library | +Linux Name | +Mac Name | +Windows Name | +
|---|---|---|---|
| +\code +SZIP Compression Library +SZIP Compression Library +ZLIB or DEFLATE Compression Library +\endcode + | ++\code +libszaec.a +libaec.a +libz.a +\endcode + | ++\code +libszaec.a +libaec.a +libz.a +\endcode + | ++\code +libszaec.lib +libaec.lib +libz.lib +\endcode + | +
How do you use find_package with HDF5?
+To use find_package you will first need to make sure that HDF5_ROOT is set correctly. For setting this +environment variable see the Preconditions in the USING_HDF5_CMake.txt file in the share directory. + +See the CMakeLists.txt file provided with these examples for how to use find_package with HDF5. + +Please note that the find_package invocation changed to require "shared" or "static": +\code + FIND_PACKAGE(HDF5 COMPONENTS C HL NO_MODULE REQUIRED shared) + FIND_PACKAGE(HDF5 COMPONENTS C HL NO_MODULE REQUIRED static) +\endcode + +Previously, the find_package invocation was: +\code + FIND_PACKAGE(HDF5 COMPONENTS C HL NO_MODULE REQUIRED) +\endcode + +My platform/compiler is not included. Can I still use the configuration files?
+Yes, you can but you will have to edit the HDF5_Examples.cmake file and update the variable: +\code + CTEST_CMAKE_GENERATOR +\endcode + +The generators for your platform can be seen by typing: +\code + cmake --help +\endcode + +What do I do if the build fails?
+I received an error during the build and the application binary is not in the +build directory as I expected. How do I determine what the problem is? + +If the error is not clear, then the first thing you may want to do is replace the -V (Dash Uppercase Vee) +option for ctest in the build script to -VV (Dash Uppercase Vee Uppercase Vee). Then remove the build +directory and re-run the build script. The output should be more verbose. + +If the error is still not clear, then check the log files. You will find those in the build directory. +For example, on Unix the log files will be in: +\code + build/Testing/Temporary/ +\endcode +There are log files for the configure, test, and build. + ++Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +@page LBTraining Training Videos +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
+ +Training Videos + +
+Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +*/ diff --git a/doxygen/dox/LearnHDFView.dox b/doxygen/dox/LearnHDFView.dox new file mode 100644 index 0000000000..b1f632caed --- /dev/null +++ b/doxygen/dox/LearnHDFView.dox @@ -0,0 +1,472 @@ +/** @page LearnHDFView Learning HDF5 with HDFView + +Navigate back: \ref index "Main" / \ref GettingStarted +
+ +This tutorial enables you to get a feel for HDF5 by using the HDFView browser. It does NOT require +any programming experience. + +\section sec_learn_hv_install HDFView Installation +\li Download and install HDFView. It can be downloaded from the Download HDFView page. +\li Obtain the storm1.txt text file, used in the tutorial. + +\section sec_learn_hv_begin Begin Tutorial +Once you have HDFView installed, bring it up and you are ready to begin the tutorial. + +
+This tutorial requires that the default HDFView File Access Mode be Read / Write. If fields are greyed out so that you cannot select them, then the File Access Mode is Read Only.
+
+To change the File Access Mode follow these steps:
+
|
+
-
+
- @ref subsec_learn_hv_topics_file +
- @ref subsec_learn_hv_topics_image +
- @ref subsec_learn_hv_topics_attr +
- @ref subsec_learn_hv_topics_compress +
- @ref subsec_learn_hv_topics_subset +
- @ref subsec_learn_hv_topics_table +
-
+
- Select the File pull-down menu at the top left, and then select New -> HDF5. +
- Specify a location and type in storm.h5 for the name of your file, and click on the Save button.
+You will see the storm.h5 file in the TableView:
+
+
++ ++\image html storm.png + +
+ - Right click on storm.h5, and select New -> Group. +
- Enter Data for the name of the group and then click the Ok button. You will see the group Data in the TableView.
+
+
++ ++\image html DataGroup.png + +
+ - Right click on the group Data and select New -> Dataset. +
- A window pops up on the right. Fill in the information as follows, and then click Ok (leave the
+Datatype information as is):
+
+
++ +Dataset Name + +Storm + ++ +Under Dataspace, Current size + +57x57 + ++ +Layout + +Contiguous (default) + +
+ - Click to expand the Data group in the tree view to see the Storm dataset:
+
+
++ ++\image html StormDataset.png + +
+ - Double left click on the Storm dataset in the tree view. A window with an empty spreadsheet pops open. +
- Copy the data from the storm1.txt file into the dataset.
+
+If you downloaded storm1.txt,
+then click on the Import/Export Data menu and select Import Data from -> Text File.
+Specify a location, select storm1.txt
+and click on the Open button. Answer Yes in the dialog box that
+pops up (which asks if you wish to paste the selected data).
+
+Alternately, you can copy/paste directly. Select and copy the data in a separate window. Position your
+cursor at (0,0) in your table, and select Paste from the Table menu.
+
+The values will be entered into the spreadsheet.
+
+
++ ++\image html datasetwdata.png + +
+ - Table -> Close the dataset, and save the data. +
-
+
- Right click on Storm in the tree view, and select Open As. +
- Select the Image button under Display As (near the top) in the Dataset Selection window that pops
+up. Then click OK at the bottom of the window to display the image.
+
+
++ ++\image html showasimage.png + +
+ - The rainbow icon brings you to the Image Palette window. Click on that to play with the palette +(GrayWave probably is the best choice). Close. +
-
+
- Click on the /Data folder in the tree view. You will see two tabs, Object Attribute Info and
+General Object Info, in the pane on the right site of the HDFView window.
+
+
++ ++\image html noattrs.png + +
+ - With the left mouse button, select the Add Attribute button. +
- Select the Add Attribute button to add an attribute with these values: +
- Select the Ok button. The attribute will show up under the Object Attribute Info tab. +
- Double-click the BatchID attribute line to open the data table for BatchID. +
- Click in the first cell and enter 3343 followed by the enter key. +
- Table -> Close, answer Yes in the dialog box that +pops up (which asks if you wish to paste the selected data). +
| Name + | +BatchID + | +
|---|---|
| Type + | +INTEGER + | +
| Size (bits) + | +32 + | +
-
+
- Left mouse click on the /Storm dataset in the tree view. You will see the Object Attribute +Info and General Object Info tabs on the right +
- In the Object Attribute Info pane select the Add Attribute button and enter an attribute with +these values. (Be sure to add a String Length or the string will be truncated to one character!): +
- Select the Ok button. The attribute will show up under the Object Attribute Info tab. +
- Double-click the Units attribute line to open the data table for Units. +
- Click in the first cell and enter m/s followed by the enter key. +
- Table -> Close, answer Yes in the dialog box that
+pops up (which asks if you wish to paste the selected data).
+
+
++ ++\image html scarletletter.png + +
+
| Name + | +Units + | +
|---|---|
| Type + | +STRING + | +
| String Length + | +3 + | +
-
+
- Right click on storm.h5. Select New -> Group. +
- Enter Image for the name of the group, and click the OK button to create the group.
+
+
++ ++\image html newgroupimage.png + +
+ - Right click on the Image group, and select New -> Dataset. +
- Enter the following information for the dataset. Leave the Datatype as is (INTEGER):
+
+
+You will see the Another Storm dataset in the Image group: ++ +Dataset name + +Another Storm + ++ +Under Dataspace, Current size + +57x57 + ++ +Storage Layout + +Chunked + ++ +Chunk Size + +20x20 + ++ +Compression + +gzip + ++ Compression Level + +9 + ++
++ ++\image html hdfview-anthrstrm.png + +
+ - Double left-mouse click on the Another Storm dataset to display the spreadsheet:
+
+
++ ++\image html hdfview-anthrstrm-sprdsht.png + +
+ - Copy the data from the storm1.txt file into the dataset. (See the previous topic for copying +storm1.txt into a dataset.) +
- Table -> Close, and save the data. +
- Right click on Another Storm, and select Open As. +
- Select the Image button in the Dataset Selection window that pops up. Click the Ok button at the
+bottom of the window to view the dataset as an image.
+
+
++ ++\image html hdfview-anthrstrm-img.png + +
+
-
+
- Right click on the Data group and select New -> Image. +
- A window pops up on the right. Enter the following and then click Ok: +
- Close the dataset. +
- Expand the Data group to see its contents. You will see the Storm Image dataset.
+
+
++ ++\image html hdfview-imgicon.png + +
+ -
+Add data to the Storm Image dataset as was shown previously:
+
-
+
- Right click on Storm Image, and select Open As to open the Dataset Selection window. +
- Click on the Spreadsheet button at the top left of the Dataset Selection window to view the image +as a spreadsheet. +
- Copy the data from the storm1.txt file into the dataset. +
- Close the dataset and save the data. +
+ - Left double click on Storm Image to see the image. Close the dataset. +
- Right click on Storm Image and select Open As to bring up the Data Selection window. +
- Select a subset by clicking the left mouse on the image in the window and dragging the mouse.
+Notice that the Height and Width values change. Select to display it as an image. Click Ok.
+
+
++ ++\image html hdfview-imgsubset.png + +
+ - Position the cursor in the middle of the image. Press Shift+Left Mouse button and hold, and then +drag the mouse to select another subset. +
- Select Image->Write Selection to Image. Enter Subset for the new image name. Click Ok. The Subset +image will appear in the tree view on the left. +
- Left double click on the image Subset to bring it up on the right.
+
+
++ ++\image html hdfview-newimgsubset.png + +
+ - Close the Subset image. +
| Image name + | +Storm Image + | +
|---|---|
| Height + | +57 + | +
| Width + | +57 + | +
-
+
- Right button click on the group Data. Select New -> Compound DS. +
- A window pops up. Only fill in the following fields:
+
+
+ ++ +Dataset name + +Table + ++ +Dataspace (Current size only) + +4 + ++ +Compound Datatype Properties: + +
Number of Members +3 + ++ +Compound Datatype Properties: + +
Name / Datatype / Size +Description / string / 4 + +
Temperature / float / 1 +
Pressure / double / 1 ++
++ ++\image html hdfview-newcmpd.png + +
+ - Click Ok at the bottom. +
- Open the Data group (if it is not open) and double left click on the Table object.
+
+
++ ++\image html hdfview-table.png + +
+ - Close the dataset. +
+Navigate back: \ref index "Main" / \ref GettingStarted + +*/ diff --git a/doxygen/dox/ReferenceManual.dox b/doxygen/dox/ReferenceManual.dox index df0c747483..790092530a 100644 --- a/doxygen/dox/ReferenceManual.dox +++ b/doxygen/dox/ReferenceManual.dox @@ -1,53 +1,32 @@ /** \page RM HDF5 Reference Manual -The functions provided by the HDF5 C-API are grouped into the following +The functions provided by the HDF5 API are grouped into the following \Emph{modules}:
| Modules | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
-
+ +\section secToolsBasic Basic Facts about HDF5 +The following are basic facts about HDF5 files to keep in mind while completing these tutorial topics: +\li All HDF5 files contain a root group "/". +\li There are two primary objects in HDF5, a group and a dataset: + Groups allow objects to be organized into a group structure, such as a tree. + Datasets contain raw data values. +\li Additional information about an HDF5 object may optionally be stored in attributes attached to the object. + +\section secToolsTopics Tutorial Topics +
+Navigate back: \ref index "Main" / \ref GettingStarted + +@page ViewToolsCommand Command-line Tools +Navigate back: \ref index "Main" / \ref GettingStarted + + +\section secViewToolsCommandObtain Obtain Tools and Files (Optional) +Pre-built binaries for Linux and Windows are distributed within the respective HDF5 binary release +packages, which can be obtained from the Download HDF5 page. + +HDF5 files can be obtained from various places such as \ref HDF5Examples and HDF-EOS and Tools and +Information Center. Specifically, the following examples are used in this tutorial topic: +\li HDF5 Files created from compiling the \ref LBExamples +\li HDF5 Files on the Examples by API page +\li NPP JPSS files, SVM01_npp.. (gzipped) +and SVM09_npp.. (gzipped) +\li HDF-EOS OMI-Aura file + +\section secViewToolsCommandTutor Tutorial Topics +A variety of command-line tools are included in the HDF5 binary distribution. There are tools to view, +edit, convert and compare HDF5 files. This tutorial discusses the tools by their functionality. It +does not cover all of the HDF5 tools. + +
+Navigate back: \ref index "Main" / \ref GettingStarted + +@page ViewToolsView Command-line Tools For Viewing HDF5 Files +Navigate back: \ref index "Main" / \ref GettingStarted / \ref ViewToolsCommand + + +\section secViewToolsViewTOC Contents +
-h
+or --help option:
+\code
+h5dump -h
+\endcode
+
+The following h5dump options can be helpful in viewing the content and structure of a file:
+
h5dump using the -d D option and specifying the entire
+path and name of the dataset for D. The path is important in identifying the correct dataset,
+as there can be multiple datasets with the same name. The path can be determined by looking at
+the objects in the file with h5dump -n.
+
+The following example uses the groups.h5 file that is created by the
+\ref LBExamples
+example h5_crtgrpar.c. To display dset1 in the groups.h5 file below, specify dataset
+/MyGroup/dset1. The -H option is used to suppress printing of the data values:
+
+Contents of groups.h5
+\code
+ $ h5dump -n groups.h5
+ HDF5 "groups.h5" {
+ FILE_CONTENTS {
+ group /
+ group /MyGroup
+ group /MyGroup/Group_A
+ dataset /MyGroup/Group_A/dset2
+ group /MyGroup/Group_B
+ dataset /MyGroup/dset1
+ }
+ }
+\endcode
+
+Display dataset "dset1"
+\code
+ $ h5dump -d "/MyGroup/dset1" -H groups.h5
+ HDF5 "groups.h5" {
+ DATASET "/MyGroup/dset1" {
+ DATATYPE H5T_STD_I32BE
+ DATASPACE SIMPLE { ( 3, 3 ) / ( 3, 3 ) }
+ }
+ }
+\endcode
+
+\subsubsection subsubsecViewToolsViewDset_h5dumpEx5 Example 5
+The -p option is used to examine the the dataset filters, storage layout, and fill value properties of a dataset.
+
+This option can be useful for checking how well compression works, or even for analyzing performance
+and dataset size issues related to chunking. (The smaller the chunk size, the more chunks that HDF5
+has to keep track of, which increases the size of the file and potentially affects performance.)
+
+In the file shown below the dataset /DS1 is both chunked and compressed:
+\code
+ $ h5dump -H -p -d "/DS1" h5ex_d_gzip.h5
+ HDF5 "h5ex_d_gzip.h5" {
+ DATASET "/DS1" {
+ DATATYPE H5T_STD_I32LE
+ DATASPACE SIMPLE { ( 32, 64 ) / ( 32, 64 ) }
+ STORAGE_LAYOUT {
+ CHUNKED ( 4, 8 )
+ SIZE 5278 (1.552:1 COMPRESSION)
+ }
+ FILTERS {
+ COMPRESSION DEFLATE { LEVEL 9 }
+ }
+ FILLVALUE {
+ FILL_TIME H5D_FILL_TIME_IFSET
+ VALUE 0
+ }
+ ALLOCATION_TIME {
+ H5D_ALLOC_TIME_INCR
+ }
+ }
+ }
+\endcode
+
+You can obtain the h5ex_d_gzip.c program that created this file, as well as the file created,
+from the Examples by API page.
+
+\subsection subsecViewToolsViewDset_h5ls h5ls
+Specific datasets can be specified with h5ls by simply adding the dataset path and dataset after the
+file name. As an example, this command displays dataset dset2 in the groups.h5
+file used in @ref subsubsecViewToolsViewDset_h5dumpEx4 :
+\code
+h5ls groups.h5/MyGroup/Group_A/dset2
+\endcode
+
+Just the dataspace information gets displayed:
+\code
+dset2 Dataset {2, 10}
+\endcode
+
+The following options can be used to see detailed information about a dataset.
+
-v is shown below:
+\code
+ $ h5ls -v groups.h5/MyGroup/Group_A/dset2
+ Opened "groups.h5" with sec2 driver.
+ dset2 Dataset {2/2, 10/10}
+ Location: 1:3840
+ Links: 1
+ Storage: 80 logical bytes, 80 allocated bytes, 100.00% utilization
+ Type: 32-bit big-endian integer
+\endcode
+
+The output of using -d is shown below:
+\code
+ $ h5ls -d groups.h5/MyGroup/Group_A/dset2
+ dset2 Dataset {2, 10}
+ Data:
+ (0,0) 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
+\endcode
+
+\section secViewToolsViewGrps Groups
+Both h5dump and h5ls can be used to view specific groups in a file.
+
h5dump options that are useful for examining groups are:
+
HDFEOS group in the OMI file mentioned previously, you can specify the path and name of the group as follows:
+\code
+h5dump -g "/HDFEOS" -H -A 0 OMI-Aura.he5
+\endcode
+
+The -A 0 option suppresses attributes and -H suppresses printing of data values:
+\code
+ HDF5 "OMI-Aura.he5" {
+ GROUP "/HDFEOS" {
+ GROUP "ADDITIONAL" {
+ GROUP "FILE_ATTRIBUTES" {
+ }
+ }
+ GROUP "GRIDS" {
+ GROUP "OMI Column Amount O3" {
+ GROUP "Data Fields" {
+ DATASET "ColumnAmountO3" {
+ DATATYPE H5T_IEEE_F32LE
+ DATASPACE SIMPLE { ( 720, 1440 ) / ( 720, 1440 ) }
+ }
+ DATASET "RadiativeCloudFraction" {
+ DATATYPE H5T_IEEE_F32LE
+ DATASPACE SIMPLE { ( 720, 1440 ) / ( 720, 1440 ) }
+ }
+ DATASET "SolarZenithAngle" {
+ DATATYPE H5T_IEEE_F32LE
+ DATASPACE SIMPLE { ( 720, 1440 ) / ( 720, 1440 ) }
+ }
+ DATASET "ViewingZenithAngle" {
+ DATATYPE H5T_IEEE_F32LE
+ DATASPACE SIMPLE { ( 720, 1440 ) / ( 720, 1440 ) }
+ }
+ }
+ }
+ }
+ }
+ }
+\endcode
+
+\subsection subsecViewToolsViewGrps_h5ls h5ls
+You can view the contents of a group with h5ls/ by specifying the group after the file name.
+To use h5ls to view the contents of the /HDFEOS group in the OMI-Aura.he5 file, type:
+\code
+h5ls -r OMI-Aura.he5/HDFEOS
+\endcode
+
+The output of this command is:
+\code
+ /ADDITIONAL Group
+ /ADDITIONAL/FILE_ATTRIBUTES Group
+ /GRIDS Group
+ /GRIDS/OMI\ Column\ Amount\ O3 Group
+ /GRIDS/OMI\ Column\ Amount\ O3/Data\ Fields Group
+ /GRIDS/OMI\ Column\ Amount\ O3/Data\ Fields/ColumnAmountO3 Dataset {720, 1440}
+ /GRIDS/OMI\ Column\ Amount\ O3/Data\ Fields/RadiativeCloudFraction Dataset {720, 1440}
+ /GRIDS/OMI\ Column\ Amount\ O3/Data\ Fields/SolarZenithAngle Dataset {720, 1440}
+ /GRIDS/OMI\ Column\ Amount\ O3/Data\ Fields/ViewingZenithAngle Dataset {720, 1440}
+\endcode
+
+If you specify the -v option, you can also see the attributes and properties of the datasets.
+
+\section secViewToolsViewAttr Attributes
+
+\subsection subsecViewToolsViewAttr_h5dump h5dump
+Attributes are displayed by default if using h5dump. Some files contain many attributes, which
+can make it difficult to examine the objects in the file. Shown below are options that can help
+when using h5dump to work with files that have attributes.
+
+\subsubsection subsubsecViewToolsViewAttr_h5dumpEx6 Example 6
+The -a A option will display an attribute. However, the path to the attribute must be included
+when specifying this option. For example, to see the ScaleFactor attribute in the OMI-Aura.he5 file, type:
+\code
+h5dump -a "/HDFEOS/GRIDS/OMI Column Amount O3/Data Fields/SolarZenithAngle/ScaleFactor" OMI-Aura.he5
+\endcode
+
+This command displays:
+\code
+ HDF5 "OMI-Aura.he5" {
+ ATTRIBUTE "ScaleFactor" {
+ DATATYPE H5T_IEEE_F64LE
+ DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
+ DATA {
+ (0): 1
+ }
+ }
+ }
+\endcode
+
+How can you determine the path to the attribute? This can be done by looking at the file contents with the -n 1 option:
+\code
+h5dump -n 1 OMI-Aura.he5
+\endcode
+
+Below is a portion of the output for this command:
+\code
+ HDF5 "OMI-Aura.he5" {
+ FILE_CONTENTS {
+ group /
+ group /HDFEOS
+ group /HDFEOS/ADDITIONAL
+ group /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/EndUTC
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/GranuleDay
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/GranuleDayOfYear
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/GranuleMonth
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/GranuleYear
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/InstrumentName
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/OrbitNumber
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/OrbitPeriod
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/PGEVersion
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/Period
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/ProcessLevel
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/StartUTC
+ attribute /HDFEOS/ADDITIONAL/FILE_ATTRIBUTES/TAI93At0zOfGranule
+
+ ...
+\endcode
+
+There can be multiple objects or attributes with the same name in a file. How can you make sure
+you are finding the correct object or attribute? You can first determine how many attributes
+there are with a specified name, and then examine the paths to them.
+
+The -N option can be used to display all objects or attributes with a given name.
+For example, there are four attributes with the name ScaleFactor in the OMI-Aura.he5 file,
+as can be seen below with the -N option:
+\code
+h5dump -N ScaleFactor OMI-Aura.he5
+\endcode
+
+It outputs:
+\code
+HDF5 "OMI-Aura.he5" {
+ATTRIBUTE "ScaleFactor" {
+ DATATYPE H5T_IEEE_F64LE
+ DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
+ DATA {
+ (0): 1
+ }
+}
+ATTRIBUTE "ScaleFactor" {
+ DATATYPE H5T_IEEE_F64LE
+ DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
+ DATA {
+ (0): 1
+ }
+}
+ATTRIBUTE "ScaleFactor" {
+ DATATYPE H5T_IEEE_F64LE
+ DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
+ DATA {
+ (0): 1
+ }
+}
+ATTRIBUTE "ScaleFactor" {
+ DATATYPE H5T_IEEE_F64LE
+ DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
+ DATA {
+ (0): 1
+ }
+}
+}
+\endcode
+
+\subsection subsecViewToolsViewAttr_h5ls h5ls
+If you include the -v (verbose) option for h5ls, you will see all of the attributes for the
+specified file, dataset or group. You cannot display individual attributes.
+
+\section secViewToolsViewSub Dataset Subset
+
+\subsection subsecViewToolsViewSub_h5dump h5dump
+If you have a very large dataset, you may wish to subset or see just a portion of the dataset.
+This can be done with the following h5dump options.
+
START (s), STRIDE (S), COUNT (c), and BLOCK (k) options
+define the shape and size of the selection. They are arrays with the same number of dimensions as the rank
+of the dataset's dataspace, and they all work together to define the selection. A change to one of
+these arrays can affect the others.
+
+When specifying these h5dump options, a comma is used as the delimiter for each dimension in the
+option value. For example, with a 2-dimensional dataset, the option value is specified as "H,W",
+where H is the height and W is the width. If the offset is 0 for both dimensions, then
+START would be specified as follows:
+\code
+-s "0,0"
+\endcode
+
+There is also a shorthand way to specify these options with brackets at the end of the dataset name:
+\code
+-d DATASETNAME[s;S;c;k]
+\endcode
+
+Multiple dimensions are separated by commas. For example, a subset for a 2-dimensional dataset would be specified as follows:
+\code
+-d DATASETNAME[s,s;S,S;c,c;k,k]
+\endcode
+
+For a detailed understanding of how selections works, see the #H5Sselect_hyperslab API in the \ref RM.
+
+The dataset SolarZenithAngle in the OMI-Aura.he5 file can be used to illustrate these options. This
+dataset is a 2-dimensional dataset of size 720 (height) x 1440 (width). Too much data will be displayed
+by simply viewing the specified dataset with the -d option:
+\code
+h5dump -d "HDFEOS/GRIDS/OMI Column Amount O3/Data Fields/SolarZenithAngle" OMI-Aura.he5
+\endcode
+Subsetting narrows down the output that is displayed. In the following example, the first
+15x10 elements (-c "15,10") are specified, beginning with position (0,0) (-s "0,0"):
+\code
+ h5dump -A 0 -d "HDFEOS/GRIDS/OMI Column Amount O3/Data Fields/SolarZenithAngle" -s "0,0" -c "15,10" -w 0 OMI-Aura.he5
+\endcode
+
+If using the shorthand method, specify:
+\code
+ h5dump -A 0 -d "HDFEOS/GRIDS/OMI Column Amount O3/Data Fields/SolarZenithAngle[0,0;;15,10;]" -w 0 OMI-Aura.he5
+\endcode
+
+Where,
+\par The -d option must be specified
+
+before
+\par subsetting options (if not using the shorthand method).
+
+The -A 0 option suppresses the printing of attributes.
+
+The -w 0 option sets the number of columns of output to the maximum allowed value (65535).
+This ensures that there are enough columns specified for displaying the data.
+
+Either command displays:
+\code
+ HDF5 "OMI-Aura.he5" {
+ DATASET "HDFEOS/GRIDS/OMI Column Amount O3/Data Fields/SolarZenithAngle" {
+ DATATYPE H5T_IEEE_F32LE
+ DATASPACE SIMPLE { ( 720, 1440 ) / ( 720, 1440 ) }
+ SUBSET {
+ START ( 0, 0 );
+ STRIDE ( 1, 1 );
+ COUNT ( 15, 10 );
+ BLOCK ( 1, 1 );
+ DATA {
+ (0,0): 79.403, 79.403, 79.403, 79.403, 79.403, 79.403, 79.403, 79.403, 79.403, 79.403,
+ (1,0): 79.071, 79.071, 79.071, 79.071, 79.071, 79.071, 79.071, 79.071, 79.071, 79.071,
+ (2,0): 78.867, 78.867, 78.867, 78.867, 78.867, 78.867, 78.867, 78.867, 78.867, 78.867,
+ (3,0): 78.632, 78.632, 78.632, 78.632, 78.632, 78.632, 78.632, 78.632, 78.632, 78.632,
+ (4,0): 78.429, 78.429, 78.429, 78.429, 78.429, 78.429, 78.429, 78.429, 78.429, 78.429,
+ (5,0): 78.225, 78.225, 78.225, 78.225, 78.225, 78.225, 78.225, 78.225, 78.225, 78.225,
+ (6,0): 78.021, 78.021, 78.021, 78.021, 78.021, 78.021, 78.021, 78.021, 78.021, 78.021,
+ (7,0): 77.715, 77.715, 77.715, 77.715, 77.715, 77.715, 77.715, 77.715, 77.715, 77.715,
+ (8,0): 77.511, 77.511, 77.511, 77.511, 77.511, 77.511, 77.511, 77.511, 77.511, 77.511,
+ (9,0): 77.658, 77.658, 77.658, 77.307, 77.307, 77.307, 77.307, 77.307, 77.307, 77.307,
+ (10,0): 77.556, 77.556, 77.556, 77.556, 77.556, 77.556, 77.556, 77.556, 77.102, 77.102,
+ (11,0): 78.408, 78.408, 78.408, 78.408, 78.408, 78.408, 78.408, 78.408, 77.102, 77.102,
+ (12,0): 76.34, 78.413, 78.413, 78.413, 78.413, 78.413, 78.413, 78.413, 78.413, 78.413,
+ (13,0): 78.107, 78.107, 78.107, 78.107, 78.107, 78.107, 78.107, 78.107, 78.107, 77.195,
+ (14,0): 78.005, 78.005, 78.005, 78.005, 78.005, 78.005, 76.991, 76.991, 76.991, 76.991
+ }
+ }
+ }
+ }
+\endcode
+
+What if we wish to read three rows of three elements at a time (-c "3,3"), where each element
+is a 2 x 3 block (-k "2,3") and we wish to begin reading from the second row (-s "1,0")?
+
+You can do that with the following command:
+\code
+ h5dump -A 0 -d "HDFEOS/GRIDS/OMI Column Amount O3/Data Fields/SolarZenithAngle"
+ -s "1,0" -S "2,3" -c "3,3" -k "2,3" -w 0 OMI-Aura.he5
+\endcode
+
+In this case, the stride must be specified as 2 by 3 (or larger) to accommodate the reading of 2 by 3 blocks.
+If it is smaller, the command will fail with the error,
+\code
+h5dump error: wrong subset selection; blocks overlap.
+\endcode
+
+The output of the above command is shown below:
+\code
+ HDF5 "OMI-Aura.he5" {
+ DATASET "HDFEOS/GRIDS/OMI Column Amount O3/Data Fields/SolarZenithAngle" {
+ DATATYPE H5T_IEEE_F32LE
+ DATASPACE SIMPLE { ( 720, 1440 ) / ( 720, 1440 ) }
+ SUBSET {
+ START ( 1, 0 );
+ STRIDE ( 2, 3 );
+ COUNT ( 3, 3 );
+ BLOCK ( 2, 3 );
+ DATA {
+ (1,0): 79.071, 79.071, 79.071, 79.071, 79.071, 79.071, 79.071, 79.071, 79.071,
+ (2,0): 78.867, 78.867, 78.867, 78.867, 78.867, 78.867, 78.867, 78.867, 78.867,
+ (3,0): 78.632, 78.632, 78.632, 78.632, 78.632, 78.632, 78.632, 78.632, 78.632,
+ (4,0): 78.429, 78.429, 78.429, 78.429, 78.429, 78.429, 78.429, 78.429, 78.429,
+ (5,0): 78.225, 78.225, 78.225, 78.225, 78.225, 78.225, 78.225, 78.225, 78.225,
+ (6,0): 78.021, 78.021, 78.021, 78.021, 78.021, 78.021, 78.021, 78.021, 78.021
+ }
+ }
+ }
+ }
+\endcode
+
+\section secViewToolsViewDtypes Datatypes
+
+\subsection subsecViewToolsViewDtypes_h5dump h5dump
+The following datatypes are discussed, using the output of h5dump with HDF5 files from the
+Examples by API page:
+
/DS1. It has a datatype of #H5T_STD_I32LE (32-bit Little-Endian Integer)
+and is a 4 by 7 array:
+\code
+$ h5dump h5ex_d_rdwr.h5
+HDF5 "h5ex_d_rdwr.h5" {
+GROUP "/" {
+ DATASET "DS1" {
+ DATATYPE H5T_STD_I32LE
+ DATASPACE SIMPLE { ( 4, 7 ) / ( 4, 7 ) }
+ DATA {
+ (0,0): 0, -1, -2, -3, -4, -5, -6,
+ (1,0): 0, 0, 0, 0, 0, 0, 0,
+ (2,0): 0, 1, 2, 3, 4, 5, 6,
+ (3,0): 0, 2, 4, 6, 8, 10, 12
+ }
+ }
+}
+}
+\endcode
+
+Contrast that with the following dataset that has both an Array datatype and is an array:
+\code
+$ h5dump h5ex_t_array.h5
+HDF5 "h5ex_t_array.h5" {
+GROUP "/" {
+ DATASET "DS1" {
+ DATATYPE H5T_ARRAY { [3][5] H5T_STD_I64LE }
+ DATASPACE SIMPLE { ( 4 ) / ( 4 ) }
+ DATA {
+ (0): [ 0, 0, 0, 0, 0,
+ 0, -1, -2, -3, -4,
+ 0, -2, -4, -6, -8 ],
+ (1): [ 0, 1, 2, 3, 4,
+ 1, 1, 1, 1, 1,
+ 2, 1, 0, -1, -2 ],
+ (2): [ 0, 2, 4, 6, 8,
+ 2, 3, 4, 5, 6,
+ 4, 4, 4, 4, 4 ],
+ (3): [ 0, 3, 6, 9, 12,
+ 3, 5, 7, 9, 11,
+ 6, 7, 8, 9, 10 ]
+ }
+ }
+}
+}
+\endcode
+
+In this file, dataset /DS1 has a datatype of
+\code
+H5T_ARRAY { [3][5] H5T_STD_I64LE }
+\endcode
+and it also has a dataspace of
+\code
+SIMPLE { ( 4 ) / ( 4 ) }
+\endcode
+In other words, it is an array of four elements, in which each element is a 3 by 5 array of #H5T_STD_I64LE.
+
+This dataset is much more complex. Also note that subsetting cannot be done on Array datatypes.
+
+See this FAQ for more information on the Array datatype.
+
+\subsubsection subsubsecViewToolsViewDtypes_objref Object Reference
+An Object Reference is a reference to an entire object (dataset, group, or named datatype).
+A dataset with an Object Reference datatype consists of one or more Object References.
+An Object Reference dataset can be used as an index to an HDF5 file.
+
+The /DS1 dataset in the following file (h5ex_t_objref.h5) is an Object Reference dataset.
+It contains two references, one to group /G1 and the other to dataset /DS2:
+\code
+$ h5dump h5ex_t_objref.h5
+HDF5 "h5ex_t_objref.h5" {
+GROUP "/" {
+ DATASET "DS1" {
+ DATATYPE H5T_REFERENCE { H5T_STD_REF_OBJECT }
+ DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
+ DATA {
+ (0): GROUP 1400 /G1 , DATASET 800 /DS2
+ }
+ }
+ DATASET "DS2" {
+ DATATYPE H5T_STD_I32LE
+ DATASPACE NULL
+ DATA {
+ }
+ }
+ GROUP "G1" {
+ }
+}
+}
+\endcode
+
+\subsubsection subsubsecViewToolsViewDtypes_regref Region Reference
+A Region Reference is a reference to a selection within a dataset. A selection can be either
+individual elements or a hyperslab. In h5dump you will see the name of the dataset along with
+the elements or slab that is selected. A dataset with a Region Reference datatype consists of
+one or more Region References.
+
+An example of a Region Reference dataset (h5ex_t_regref.h5) can be found on the
+Examples by API page,
+under Datatypes. If you examine this dataset with h5dump you will see that /DS1 is a
+Region Reference dataset as indicated by its datatype, highlighted in bold below:
+\code
+$ h5dump h5ex_t_regref.h5
+HDF5 "h5ex_t_regref.h5" {
+GROUP "/" {
+ DATASET "DS1" {
+ DATATYPE H5T_REFERENCE { H5T_STD_REF_DSETREG }
+ DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
+ DATA {
+ DATASET /DS2 {(0,1), (2,11), (1,0), (2,4)},
+ DATASET /DS2 {(0,0)-(0,2), (0,11)-(0,13), (2,0)-(2,2), (2,11)-(2,13)}
+ }
+ }
+ DATASET "DS2" {
+ DATATYPE H5T_STD_I8LE
+ DATASPACE SIMPLE { ( 3, 16 ) / ( 3, 16 ) }
+ DATA {
+ (0,0): 84, 104, 101, 32, 113, 117, 105, 99, 107, 32, 98, 114, 111, 119,
+ (0,14): 110, 0,
+ (1,0): 102, 111, 120, 32, 106, 117, 109, 112, 115, 32, 111, 118, 101,
+ (1,13): 114, 32, 0,
+ (2,0): 116, 104, 101, 32, 53, 32, 108, 97, 122, 121, 32, 100, 111, 103,
+ (2,14): 115, 0
+ }
+ }
+}
+}
+\endcode
+
+It contains two Region References:
+\li A selection of four individual elements in dataset /DS2 : (0,1), (2,11), (1,0), (2,4)
+See the #H5Sselect_elements API in the \ref UG for information on selecting individual elements.
+\li A selection of these blocks in dataset /DS2 : (0,0)-(0,2), (0,11)-(0,13), (2,0)-(2,2), (2,11)-(2,13)
+See the #H5Sselect_hyperslab API in the \ref UG for how to do hyperslab selection.
+
+
+If you look at the code that creates the dataset (h5ex_t_regref.c) you will see that the
+first reference is created with these calls:
+\code
+ status = H5Sselect_elements (space, H5S_SELECT_SET, 4, coords[0]);
+ status = H5Rcreate (&wdata[0], file, DATASET2, H5R_DATASET_REGION, space);
+\endcode
+
+where the buffer containing the coordinates to select is:
+\code
+ coords[4][2] = { {0, 1},
+ {2, 11},
+ {1, 0},
+ {2, 4} },
+\endcode
+
+The second reference is created by calling,
+\code
+ status = H5Sselect_hyperslab (space, H5S_SELECT_SET, start, stride, count, block);
+ status = H5Rcreate (&wdata[1], file, DATASET2, H5R_DATASET_REGION, space);
+\endcode
+where start, stride, count, and block have these values:
+\code
+ start[2] = {0, 0},
+ stride[2] = {2, 11},
+ count[2] = {2, 2},
+ block[2] = {1, 3};
+\endcode
+
+These start, stride, count, and block values will select the elements shown in bold in the dataset:
+\code
+84 104 101 32 113 117 105 99 107 32 98 114 111 119 110 0
+102 111 120 32 106 117 109 112 115 32 111 118 101 114 32 0
+116 104 101 32 53 32 108 97 122 121 32 100 111 103 115 0
+\endcode
+
+If you use h5dump to select a subset of dataset
+/DS2 with these start, stride, count, and block values, you will see that the same elements are selected:
+\code
+$ h5dump -d "/DS2" -s "0,0" -S "2,11" -c "2,2" -k "1,3" h5ex_t_regref.h5
+HDF5 "h5ex_t_regref.h5" {
+DATASET "/DS2" {
+ DATATYPE H5T_STD_I8LE
+ DATASPACE SIMPLE { ( 3, 16 ) / ( 3, 16 ) }
+ SUBSET {
+ START ( 0, 0 );
+ STRIDE ( 2, 11 );
+ COUNT ( 2, 2 );
+ BLOCK ( 1, 3 );
+ DATA {
+ (0,0): 84, 104, 101, 114, 111, 119,
+ (2,0): 116, 104, 101, 100, 111, 103
+ }
+ }
+}
+}
+\endcode
+
+For more information on selections, see the tutorial topic on
+@ref LBDsetSubRW. Also see the
+\ref secViewToolsViewSub tutorial topic on using h5dump to view a subset.
+
+\subsubsection subsubsecViewToolsViewDtypes_string String
+There are two types of string data, fixed length strings and variable length strings.
+
+Below is the h5dump output for two files that have the same strings written to them. In one file,
+the strings are fixed in length, and in the other, the strings have different sizes (and are variable in size).
+
+Dataset of Fixed Length Strings
+\code
+HDF5 "h5ex_t_string.h5" {
+GROUP "/" {
+ DATASET "DS1" {
+ DATATYPE H5T_STRING {
+ STRSIZE 7;
+ STRPAD H5T_STR_SPACEPAD;
+ CSET H5T_CSET_ASCII;
+ CTYPE H5T_C_S1;
+ }
+ DATASPACE SIMPLE { ( 4 ) / ( 4 ) }
+ DATA {
+ (0): "Parting", "is such", "sweet ", "sorrow."
+ }
+ }
+}
+}
+\endcode
+
+Dataset of Variable Length Strings
+\code
+HDF5 "h5ex_t_vlstring.h5" {
+GROUP "/" {
+ DATASET "DS1" {
+ DATATYPE H5T_STRING {
+ STRSIZE H5T_VARIABLE;
+ STRPAD H5T_STR_SPACEPAD;
+ CSET H5T_CSET_ASCII;
+ CTYPE H5T_C_S1;
+ }
+ DATASPACE SIMPLE { ( 4 ) / ( 4 ) }
+ DATA {
+ (0): "Parting", "is such", "sweet", "sorrow."
+ }
+ }
+}
+}
+\endcode
+
+You might wonder which to use. Some comments to consider are included below.
+\li In general, a variable length string dataset is more complex than a fixed length string. If you don't
+specifically need a variable length type, then just use the fixed length string.
+\li A variable length dataset consists of pointers to heaps in different locations in the file. For this
+reason, a variable length dataset cannot be compressed. (Basically, the pointers get compressed and
+not the actual data!) If compression is needed, then do not use variable length types.
+\li If you need to create an array of of different length strings, you can either use fixed length strings
+along with compression, or use a variable length string.
+
++Navigate back: \ref index "Main" / \ref GettingStarted / \ref ViewToolsCommand + +*/ diff --git a/doxygen/dox/ViewTools2.dox b/doxygen/dox/ViewTools2.dox new file mode 100644 index 0000000000..4d8788a81f --- /dev/null +++ b/doxygen/dox/ViewTools2.dox @@ -0,0 +1,786 @@ +/** @page ViewToolsEdit Command-line Tools For Editing HDF5 Files +Navigate back: \ref index "Main" / \ref GettingStarted / \ref ViewToolsCommand + + +\section secViewToolsEditTOC Contents +
h5repack tool can be used to remove unused space in an HDF5
+file. If no options other than the input and output HDF5 files are specified on the
+h5repack command line, it will write the file to the new
+file, getting rid of the unused space:
+\code
+h5repack I. Introduction-
|
+ |
+  |
+ |
+ Figure 2: HDF5 objects --
+ datasets, datatypes, or dataspaces
+ |
+ |
+ The format of an HDF5 file on disk encompasses several - key ideas of the HDF4 and AIO file formats as well as - addressing some shortcomings therein. The new format is - more self-describing than the HDF4 format and is more - uniformly applied to data objects in the file. +The format of an HDF5 file on disk encompasses several key ideas + of the HDF4 and AIO file formats as well as addressing some + shortcomings therein. The new format is more self-describing than the + HDF4 format and is more uniformly applied to data objects in the file. -An HDF5 file appears to the user as a directed graph. - The nodes of this graph are the higher-level HDF5 objects - that are exposed by the HDF5 APIs: +An HDF5 file appears to the user as a directed graph. The nodes + of this graph are the higher-level HDF5 objects that are exposed by the + HDF5 APIs: -
At the lowest level, as information is actually written to the disk, - an HDF5 file is made up of the following objects: -
At the lowest level, as information is actually written to the + disk, an HDF5 file is made up of the following objects: +
The HDF5 Library uses these low-level objects to represent the - higher-level objects that are then presented to the user or - to applications through the APIs. For instance, a group is an - object header that contains a message that points to a local - heap (for storing the links to objects in the group) and to a - B-tree (which indexes the links). A dataset is an object header - that contains messages that describe datatype, dataspace, layout, - filters, external files, fill value, and other elements with the - layout message pointing to either a raw data chunk or to a - B-tree that points to raw data chunks. +The HDF5 Library uses these low-level objects to represent the + higher-level objects that are then presented to the user or to + applications through the APIs. For instance, a group is an object + header that contains a message that points to a local heap (for storing + the links to objects in the group) and to a B-tree (which indexes the + links). A dataset is an object header that contains messages that + describe datatype, dataspace, layout, filters, external files, fill + value, and other elements with the layout message pointing to either a + raw data chunk or to a B-tree that points to raw data chunks. I.A. This Document-This document describes the lower-level data objects; - the higher-level objects and their properties are described - in the HDF5 User’s Guide. ++ This document describes the lower-level data objects; the higher-level + objects and their properties are described in the HDF5 + User Guide. + -Three levels of information comprise the file format. - Level 0 contains basic information for identifying and - defining information about the file. Level 1 information contains - the information about the pieces of a file shared by many objects - in the file (such as a B-trees and heaps). Level 2 is the rest - of the file and contains all of the data objects, with each object - partitioned into header information, also known as - metadata, and data. ++ Three levels of information comprise the file format. Level 0 contains + basic information for identifying and defining information about the + file. Level 1 information contains the information about the pieces of + a file shared by many objects in the file (such as a B-trees and + heaps). Level 2 is the rest of the file and contains all of the data + objects, with each object partitioned into header information, also + known as metadata, and data. + -The sizes of various fields in the following layout tables are - determined by looking at the number of columns the field spans - in the table. There are three exceptions: (1) The size may be - overridden by specifying a size in parentheses, (2) the size of - addresses is determined by the Size of Offsets field - in the superblock and is indicated in this document with a - superscripted ‘O’, and (3) the size of length fields is determined - by the Size of Lengths field in the superblock and is - indicated in this document with a superscripted ‘L’. ++ The sizes of various fields in the following layout tables are + determined by looking at the number of columns the field spans in the + table. There are three exceptions: (1) The size may be overridden by + specifying a size in parentheses, (2) the size of addresses is + determined by the Size of Offsets field in the superblock and + is indicated in this document with a superscripted ‘O’, and + (3) the size of length fields is determined by the Size of + Lengths field in the superblock and is indicated in this document with + a superscripted ‘L’. + -Values for all fields in this document should be treated as unsigned - integers, unless otherwise noted in the description of a field. - Additionally, all metadata fields are stored in little-endian byte - order. - +Values for all fields in this document should be treated as + unsigned integers, unless otherwise noted in the description of a + field. Additionally, all metadata fields are stored in little-endian + byte order. -All checksums used in the format are computed with the - Jenkins’ - lookup3 algorithm. - ++ All checksums used in the format are computed with the Jenkins’ + lookup3 algorithm. + -Whenever a bit flag or field is mentioned for an entry, bits are - numbered from the lowest bit position in the entry. - +Whenever a bit flag or field is mentioned for an entry, bits are + numbered from the lowest bit position in the entry. -Various tables in this document aligned with “This space inserted - only to align table nicely”. These entries in the table are just - to make the table presentation nicer and do not represent any values - or padding in the file. - +Various tables in this document aligned with “This space + inserted only to align table nicely”. These entries in the table + are just to make the table presentation nicer and do not represent any + values or padding in the file. I.B. Changes for HDF5 1.10-As of October 2015, changes in the file format for HDF5 1.10 - have not yet been finalized. +As of October 2015, changes in the file format for HDF5 1.10 have + not yet been finalized. - -II. Disk Format: Level 0 - File Metadata++ II. Disk Format: Level 0 - File Metadata +- -II.A. Disk Format: Level 0A - Format Signature and Superblock++ II.A. Disk Format: Level 0A - Format + Signature and Superblock +-The superblock may begin at certain predefined offsets within - the HDF5 file, allowing a block of unspecified content for - users to place additional information at the beginning (and - end) of the HDF5 file without limiting the HDF5 Library’s - ability to manage the objects within the file itself. This - feature was designed to accommodate wrapping an HDF5 file in - another file format or adding descriptive information to an HDF5 - file without requiring the modification of the actual file’s - information. The superblock is located by searching for the - HDF5 format signature at byte offset 0, byte offset 512, and at - successive locations in the file, each a multiple of two of - the previous location; in other words, at these byte offsets: - 0, 512, 1024, 2048, and so on. +The superblock may begin at certain predefined offsets within the + HDF5 file, allowing a block of unspecified content for users to place + additional information at the beginning (and end) of the HDF5 file + without limiting the HDF5 Library’s ability to manage the objects + within the file itself. This feature was designed to accommodate + wrapping an HDF5 file in another file format or adding descriptive + information to an HDF5 file without requiring the modification of the + actual file’s information. The superblock is located by searching + for the HDF5 format signature at byte offset 0, byte offset 512, and at + successive locations in the file, each a multiple of two of the + previous location; in other words, at these byte offsets: 0, 512, 1024, + 2048, and so on. -The superblock is composed of the format signature, followed by a - superblock version number and information that is specific to each - version of the superblock. - Currently, there are three versions of the superblock format. - Version 0 is the default format, while version 1 is basically the same - as version 0 with additional information when a non-default B-tree ‘K’ - value is stored. Version 2 is the latest format, with some fields - eliminated or compressed and with superblock extension and checksum - support. +The superblock is composed of the format signature, followed by a + superblock version number and information that is specific to each + version of the superblock. Currently, there are three versions of the + superblock format. Version 0 is the default format, while version 1 is + basically the same as version 0 with additional information when a + non-default B-tree ‘K’ value is stored. Version 2 is the + latest format, with some fields eliminated or compressed and with + superblock extension and checksum support. -Version 0 and 1 of the superblock are described below: +Version 0 and 1 of the superblock are described below: -
-
-
-
-
-
- Version 2 of the superblock is described below: - -
-
-
-
-
-
+
|
+ Driver Information Block AddressO + Root Group Symbol Table Entry |
+
- -II.B. Disk Format: Level 0B - File Driver Info- -The driver information block is an optional region of the - file which contains information needed by the file driver - to reopen a file. The format is described below: - - -
-
-
|
+ Version Number of the File’s Free Space + Information
+ | This value is used to determine the format of the + file’s free space information. ++ The only value currently valid in this field is ‘0’, + which indicates that the file’s free space is as described below. + --
-
+
+
+ Version 2 of the superblock is described below: + +
+
+
+
+
+
+
+
+ + II.B. Disk Format: Level 0B - File Driver + Info ++ ++ The driver information block is an optional region of the file + which contains information needed by the file driver to reopen a file. + The format is described below: + + + +
+
+
+
+
+
+
- The two drivers encoded in the Driver Identification field are as follows: -
The format of the Driver Information field for the - above two drivers are described below: +The two drivers encoded in the +Driver Identification field are as follows: +
+ The format of the Driver Information field for the above two + drivers are described below: + -
-
-
-
-
-
-
-
-
-
-
+
|
+ Name of Member File N |
+ + This field is the null-terminated name of the member file and its + length should be multiples of 8 bytes. Additional bytes will be + padded with NULLs. The default naming convention is %s-X.h5, + where X is one of the letters s (for superblock), + b (for B-tree), r (for raw data), g (for + global heap), l (for local heap), and o (for + object header). The name of the whole HDF5 file will substitute the + %s in the string. + - -II.C. Disk Format: Level 0C - Superblock Extension+
+
+
+
+
+
+
+
+ + II.C. Disk Format: Level 0C - Superblock + Extension ++ ++ The superblock extension is used to store superblock metadata + which is either optional, or added after the version of the superblock + was defined. Superblock extensions may only exist when version 2+ of + superblock is used. A superblock extension is an object header which + may hold the following messages: + +
- -III. Disk Format: Level 1 - File Infrastructure++ III. Disk Format: Level 1 - File + Infrastructure +- -III.A. Disk Format: Level 1A - B-trees and B-tree Nodes++ III.A. Disk Format: Level 1A - B-trees and B-tree + Nodes +-B-trees allow flexible storage for objects which tend to grow - in ways that cause the object to be stored discontiguously. B-trees - are described in various algorithms books including “Introduction to - Algorithms” by Thomas H. Cormen, Charles E. Leiserson, and Ronald - L. Rivest. B-trees are used in several places in the HDF5 file format, - when an index is needed for another data structure. +B-trees allow flexible storage for objects which tend to grow in + ways that cause the object to be stored discontiguously. B-trees are + described in various algorithms books including “Introduction to + Algorithms” by Thomas H. Cormen, Charles E. Leiserson, and Ronald + L. Rivest. B-trees are used in several places in the HDF5 file format, + when an index is needed for another data structure. -The version 1 B-tree structure described below is the original index - structure, but are limited by some bugs in our implementation (mainly in - how they handle deleting records). The version 1 B-trees are being phased - out in favor of the version 2 B-trees described below, although both - types of structures may be found in the same file, depending on - application settings when creating the file. +The version 1 B-tree structure described below is the original + index structure, but are limited by some bugs in our implementation + (mainly in how they handle deleting records). The version 1 B-trees are + being phased out in favor of the version 2 B-trees described below, + although both types of structures may be found in the same file, + depending on application settings when creating the file. - -III.A.1. Disk Format: Level 1A1 - Version 1 B-trees (B-link Trees)++ III.A.1. Disk Format: Level 1A1 - Version 1 + B-trees (B-link Trees) +-Version 1 B-trees in HDF5 files an implementation of the B-link tree, - in which the sibling nodes at a particular level in the tree are stored - in a doubly-linked list, is described in the “Efficient Locking for - Concurrent Operations on B-trees” paper by Phillip Lehman and S. Bing Yao - as published in the ACM Transactions on Database Systems, - Vol. 6, No. 4, December 1981. ++ Version 1 B-trees in HDF5 files an implementation of the B-link tree, + in which the sibling nodes at a particular level in the tree are stored + in a doubly-linked list, is described in the “Efficient Locking + for Concurrent Operations on B-trees” paper by Phillip Lehman and + S. Bing Yao as published in the ACM Transactions on + Database Systems, Vol. 6, No. 4, December 1981. + -The B-link trees implemented by the file format contain one more - key than the number of children. In other words, each child - pointer out of a B-tree node has a left key and a right key. - The pointers out of internal nodes point to sub-trees while - the pointers out of leaf nodes point to symbol nodes and - raw data chunks. - Aside from that difference, internal nodes and leaf nodes - are identical. +The B-link trees implemented by the file format contain one more + key than the number of children. In other words, each child pointer out + of a B-tree node has a left key and a right key. The pointers out of + internal nodes point to sub-trees while the pointers out of leaf nodes + point to symbol nodes and raw data chunks. Aside from that difference, + internal nodes and leaf nodes are identical. -
-
-
-
-
-
|
- Signature
- |
- The ASCII character string “ |
- Node Type
- |
- Each B-link tree points to a particular type of data. - This field indicates the type of data as well as - implying the maximum degree K of the tree and - the size of each Key field. - - -
|
- Node Level
- |
- The node level indicates the level at which this node - appears in the tree (leaf nodes are at level zero). Not - only does the level indicate whether child pointers - point to sub-trees or to data, but it can also be used - to help file consistency checking utilities reconstruct - damaged trees. - - |
- Entries Used
- |
- This determines the number of children to which this - node points. All nodes of a particular type of tree - have the same maximum degree, but most nodes will point - to less than that number of children. The valid child - pointers and keys appear at the beginning of the node - and the unused pointers and keys appear at the end of - the node. The unused pointers and keys have undefined - values. - - |
- Address of Left Sibling
- |
- This is the relative file address of the left sibling of - the current node. If the current - node is the left-most node at this level then this field - is the undefined address. - - |
- Address of Right Sibling
- |
- This is the relative file address of the right sibling of - the current node. If the current - node is the right-most node at this level then this - field is the undefined address. - - |
- Keys and Child Pointers
- |
- Each tree has 2K+1 keys with 2K - child pointers interleaved between the keys. The number - of keys and child pointers actually containing valid - values is determined by the node’s Entries Used field. - If that field is N then the B-link tree contains - N child pointers and N+1 keys. - - |
- Key
- |
- The format and size of the key values is determined by - the type of data to which this tree points. The keys are - ordered and are boundaries for the contents of the child - pointer; that is, the key values represented by child - N fall between Key N and Key - N+1. Whether the interval is open or closed on - each end is determined by the type of data to which the - tree points. - - -- The format of the key depends on the node type. - For nodes of node type 0 (group nodes), the key is formatted as - follows: - -
- For nodes of node type 1 (chunked raw data nodes), the key is - formatted as follows: - -
|
- Child Pointer
- |
- The tree node contains file addresses of subtrees or - data depending on the node level. Nodes at Level 0 point - to data addresses, either raw data chunks or group nodes. - Nodes at non-zero levels point to other nodes of the - same B-tree. - -For raw data chunk nodes, the child pointer is the address - of a single raw data chunk. For group nodes, the child pointer - points to a symbol table, which contains - information for multiple symbol table entries. - -- Conceptually, each B-tree node looks like this: -
- - where child[i] is a pointer to a sub-tree (at a level - above Level 0) or to data (at Level 0). - Each key[i] describes an item stored by the B-tree - (a chunk or an object of a group node). The range of values - represented by child[i] is indicated by key[i] - and key[i+1]. - - - The following question must next be answered: - “Is the value described by key[i] contained in - child[i-1] or in child[i]?” - The answer depends on the type of tree. - In trees for groups (node type 0) the object described by - key[i] is the greatest object contained in - child[i-1] while in chunk trees (node type 1) the - chunk described by key[i] is the least chunk in - child[i]. - -That means that key[0] for group trees is sometimes unused; - it points to offset zero in the heap, which is always the - empty string and compares as “less-than” any valid object name. - -And key[N] for chunk trees is sometimes unused; - it contains a chunk offset which compares as “greater-than” - any other chunk offset and has a chunk byte size of zero - to indicate that it is not actually allocated. +- -III.A.2. Disk Format: Level 1A2 - Version 2 B-trees- -Version 2 B-trees are “traditional” B-trees, with one major difference. - Instead of just using a simple pointer (or address in the file) to a - child of an internal node, the pointer to the child node contains two - additional pieces of information: the number of records in the child - node itself, and the total number of records in the child node and - all its descendants. Storing this additional information allows fast - array-like indexing to locate the nth record in the B-tree. - -The entry into a version 2 B-tree is a header which contains global - information about the structure of the B-tree. The root node - address - field in the header points to the B-tree root node, which is either an - internal or leaf node, depending on the value in the header’s - depth field. An internal node consists of records plus - pointers to further leaf or internal nodes in the tree. A leaf node - consists of solely of records. The format of the records depends on - the B-tree type (stored in the header). - -
-
-
-
-
+
+
Conceptually, each B-tree node looks like this: +
where child[ +i] is a pointer to a sub-tree (at a level above Level 0) or to +data (at Level 0). Each key[ +i] describes an +item stored by the B-tree (a chunk or an object of a group node). +The range of values represented by child[ +i] is indicated by key[ +i] and key[ +i+1]. + + + + The following question must next be answered: “Is the value + described by key[i] contained in child[i-1] or in child[i]?” + The answer depends on the type of tree. In trees for groups (node type + 0) the object described by key[i] is the greatest object + contained in child[i-1] while in chunk trees (node type 1) the + chunk described by key[i] is the least chunk in child[i]. + + +That means that key[0] for group trees is sometimes unused; it + points to offset zero in the heap, which is always the empty string and + compares as “less-than” any valid object name. + ++ And key[N] for chunk trees is sometimes unused; it contains a + chunk offset which compares as “greater-than” any other + chunk offset and has a chunk byte size of zero to indicate that it is + not actually allocated. + + ++ + III.A.2. Disk Format: Level 1A2 - Version 2 + B-trees ++ ++ Version 2 B-trees are “traditional” B-trees, with one major + difference. Instead of just using a simple pointer (or address in the + file) to a child of an internal node, the pointer to the child node + contains two additional pieces of information: the number of records in + the child node itself, and the total number of records in the child + node and all its descendants. Storing this additional information + allows fast array-like indexing to locate the nth record in + the B-tree. + + ++ The entry into a version 2 B-tree is a header which contains global + information about the structure of the B-tree. The root node + address field in the header points to the B-tree root node, which is + either an internal or leaf node, depending on the value in the + header’s depth field. An internal node consists of + records plus pointers to further leaf or internal nodes in the tree. A + leaf node consists of solely of records. The format of the records + depends on the B-tree type (stored in the header). + + +
+
+
+
+
+
-
- -
-
+
+
+ +
+
+
|
+
+ Child Node Pointer 0O + |
+ Number of Records N0 for Child + Node 0 (variable size) |
+ Total Number of Records for Child Node 0 + (optional, variable size) |
+ Child Node Pointer 1O + |
+ Number of Records N1 for Child + Node 1 (variable size) |
+ Total Number of Records for Child Node 1 + (optional, variable size) ... |
+ |
+ Child Node Pointer NO + |
+ Number of Records Nn for Child + Node N (variable size) |
+ Total Number of Records for Child Node N + (optional, variable size) Checksum |
+
-
-
-
- -
-
-
Version |
+ Type |
+ Record 0, 1, 2...N-1 (variable size) |
+
+ Checksum |
+ -
-
+
+
|
+ Checksum
+ |
+ This is the checksum for this node. +- The record layout for each stored (in other words, non-testing) + + The record layout for each stored (in other words, non-testing) B-tree type is as follows: -
-
+
-
|
+ (Items marked with an ‘L’ in the above table are
+ of the size specified in “Size of Lengths” field in the
+ superblock.) |
+
+ -
-
-
- -
-
+
-
|
+ (Items marked with an ‘L’ in the above table are
+ of the size specified in “Size of Lengths” field in the
+ superblock.) |
+
+ -
-
-
- -
-
+
-
|
+ (Items marked with an ‘L’ in the above table are
+ of the size specified in “Size of Lengths” field in the
+ superblock.) |
+
+ -
-
-
- -
-
+
-
|
+ (Items marked with an ‘L’ in the above table are
+ of the size specified in “Size of Lengths” field in the
+ superblock.) |
+
+ -
-
-
- -
-
-
-
-
-
- -
-
-
-
-
-
- -
-
-
-
-
-
- -
-
+
-
|
+ (Items marked with an ‘O’ in the
+ above table are of the size specified in “Size of
+ Offsets” field in the superblock.) |
+
+ -
-
-
- -
-
-
-
-
-
- -
-
-
-
-
- -III.B. Disk Format: Level 1B - Group Symbol Table Nodes++ III.B. Disk Format: Level 1B - Group Symbol + Table Nodes +-A group is an object internal to the file that allows - arbitrary nesting of objects within the file (including other groups). - A group maps a set of link names in the group to a set of relative - file addresses of objects in the file. Certain metadata for an object to - which the group points can be cached in the group’s symbol table entry in - addition to being in the object’s header. +A group is an object internal to the file that allows arbitrary + nesting of objects within the file (including other groups). A group + maps a set of link names in the group to a set of relative file + addresses of objects in the file. Certain metadata for an object to + which the group points can be cached in the group’s symbol table + entry in addition to being in the object’s header. -An HDF5 object name space can be stored hierarchically by - partitioning the name into components and storing each - component as a link in a group. The link for a - non-ultimate component points to the group containing - the next component. The link for the last - component points to the object being named. +An HDF5 object name space can be stored hierarchically by + partitioning the name into components and storing each component as a + link in a group. The link for a non-ultimate component points to the + group containing the next component. The link for the last component + points to the object being named. -One implementation of a group is a collection of symbol table nodes - indexed by a B-link tree. Each symbol table node contains entries - for one or more links. If an attempt is made to add a link to an already - full symbol table node containing 2K entries, then the node is - split and one node contains K symbols and the other contains - K+1 symbols. ++ One implementation of a group is a collection of symbol table nodes + indexed by a B-link tree. Each symbol table node contains entries for + one or more links. If an attempt is made to add a link to an already + full symbol table node containing 2K entries, then the node is + split and one node contains K symbols and the other contains K+1 + symbols. + -
-
-
-
+
- -III.C. Disk Format: Level 1C - Symbol Table Entry- -Each symbol table entry in a symbol table node is designed - to allow for very fast browsing of stored objects. - Toward that design goal, the symbol table entries - include space for caching certain constant metadata from the - object header. - -
-
+ + III.C. Disk Format: Level 1C - Symbol + Table Entry ++ +Each symbol table entry in a symbol table node is designed to + allow for very fast browsing of stored objects. Toward that design + goal, the symbol table entries include space for caching certain + constant metadata from the object header. + +
+
-
-
-
-
+
+
+
+
+
+
|
+ Object Header Address
+ |
+ Every object has an object header which serves as a permanent + location for the object’s metadata. In addition to appearing + in the object header, some of the object’s metadata can be + cached in the scratch-pad space. + |
+ Cache Type
+ |
+ The cache type is determined from the object header. It also + determines the format for the scratch-pad space: +
|
+ Reserved
+ |
+ These four bytes are present so that the scratch-pad space is + aligned on an eight-byte boundary. They are always set to zero. + |
+ Scratch-pad Space
+ |
+ This space is used for different purposes, depending on the + value of the Cache Type field. Any metadata about an object + represented in the scratch-pad space is duplicated in the object + header for that object. +Furthermore, no data is cached in the group entry scratch-pad + space if the object header for the object has a link count greater + than one. +Format of the Scratch-pad Space-The symbol table entry scratch-pad space is formatted - according to the value in the Cache Type field. +The symbol table entry scratch-pad space is formatted according + to the value in the Cache Type field. -If the Cache Type field contains the value zero
-
+ If the Cache Type field contains the value zero
+ If the Cache Type field contains the value one
-
+ If the Cache Type field contains the value one
+
-
-
-
-
-
-
|
- Address of B-tree
- |
- This is the file address for the root of the - group’s B-tree. - - |
- Address of Name Heap
- |
- This is the file address for the group’s local - heap, in which are stored the group’s symbol names. - -- If the Cache Type field contains the value two
-
-
-
-
-
-
+
- -III.D. Disk Format: Level 1D - Local Heaps- -A local heap is a collection of small pieces of data that are particular - to a single object in the HDF5 file. Objects can be - inserted and removed from the heap at any time. - The address of a heap does not change once the heap is created. - For example, a group stores addresses of objects in symbol table nodes - with the names of links stored in the group’s local heap. - - -
-
-
-
-
-
Objects within a local heap should be aligned on an 8-byte boundary. - -- -III.E. Disk Format: Level 1E - Global Heap- -Each HDF5 file has a global heap which stores various types of - information which is typically shared between datasets. The - global heap was designed to satisfy these goals: - -
The implementation of the heap makes use of the memory management - already available at the file level and combines that with a new - object called a collection to achieve goal B. The global heap - is the set of all collections. Each global heap object belongs to - exactly one collection and each collection contains one or more global - heap objects. For the purposes of disk I/O and caching, a collection is - treated as an atomic object, addressing goal A. - - -When a global heap object is deleted from a collection (which occurs - when its reference count falls to zero), objects located after the - deleted object in the collection are packed down toward the beginning - of the collection and the collection’s global heap object 0 is created - (if possible) or its size is increased to account for the recently - freed space. There are no gaps between objects in each collection, - with the possible exception of the final space in the collection, if - it is not large enough to hold the header for the collection’s global - heap object 0. These features address goal C. - - -The HDF5 Library creates global heap collections as needed, so there may - be multiple collections throughout the file. The set of all of them is - abstractly called the “global heap”, although they do not actually link - to each other, and there is no global place in the file where you can - discover all of the collections. The collections are found simply by - finding a reference to one through another object in the file. For - example, data of variable-length datatype elements is stored in the - global heap and is accessed via a global heap ID. The format for - global heap IDs is described at the end of this section. - - -
-
-
-
-
-
-
-
- -
-
-
-
-
-
-
-
- - The format for the ID used to locate an object in the global heap is - described here: - -
-
-
-
-
-
+
- -III.F. Disk Format: Level 1F - Fractal Heap- -- Each fractal heap consists of a header and zero or more direct and - indirect blocks (described below). The header contains general - information as well as - initialization parameters for the doubling table. The Root - Block Address in the header points to the first direct or - indirect block in the heap. - - -- Fractal heaps are based on a data structure called a doubling - table. A doubling table provides a mechanism for quickly - extending an array-like data structure that minimizes the number of - empty blocks in the heap, while retaining very fast lookup of any - element within the array. More information on fractal heaps and - doubling tables can be found in the RFC - “Private - Heaps in HDF5.” - - -- The fractal heap implements the doubling table structure with - indirect and direct blocks. - Indirect blocks in the heap do not actually contain data for - objects in the heap, their “size” is abstract - - they represent the indexing structure for locating the - direct blocks in the doubling table. - Direct blocks - contain the actual data for objects stored in the heap. - - -- All indirect blocks have a constant number of block entries in each - row, called the width of the doubling table (stored in - the heap header). - - The number - of rows for each indirect block in the heap is determined by the - size of the block that the indirect block represents in the - doubling table (calculation of this is shown below) and is - constant, except for the “root” - indirect block, which expands and shrinks its number of rows as - needed. - - -- Blocks in the first two rows of an indirect block - are Starting Block Size number of bytes in size, - and the blocks in each subsequent row are twice the size of - the blocks in the previous row. In other words, blocks in - the third row are twice the Starting Block Size, - blocks in the fourth row are four times the - Starting Block Size, and so on. Entries for - blocks up to the Maximum Direct Block Size point to - direct blocks, and entries for blocks greater than that size - point to further indirect blocks (which have their own - entries for direct and indirect blocks). - +
+ If the Cache Type field contains the value two
+
- The number of rows of blocks, nrows, in an
- indirect block of size iblock_size is given by the
- following expression:
-
+
+
+
+
+
+
+
+ + III.D. Disk Format: Level 1D - Local Heaps ++ +A local heap is a collection of small pieces of data that are + particular to a single object in the HDF5 file. Objects can be inserted + and removed from the heap at any time. The address of a heap does not + change once the heap is created. For example, a group stores addresses + of objects in symbol table nodes with the names of links stored in the + group’s local heap. + +
+
+
+
+
+
+
+
Objects within a local heap should be aligned on an 8-byte + boundary. + ++ + III.E. Disk Format: Level 1E - Global Heap ++ +Each HDF5 file has a global heap which stores various types of + information which is typically shared between datasets. The global heap + was designed to satisfy these goals: + +
+ The implementation of the heap makes use of the memory management + already available at the file level and combines that with a new object + called a collection to achieve goal B. The global heap is the + set of all collections. Each global heap object belongs to exactly one + collection and each collection contains one or more global heap + objects. For the purposes of disk I/O and caching, a collection is + treated as an atomic object, addressing goal A. + + +When a global heap object is deleted from a collection (which + occurs when its reference count falls to zero), objects located after + the deleted object in the collection are packed down toward the + beginning of the collection and the collection’s global heap + object 0 is created (if possible) or its size is increased to account + for the recently freed space. There are no gaps between objects in each + collection, with the possible exception of the final space in the + collection, if it is not large enough to hold the header for the + collection’s global heap object 0. These features address goal C. + + +The HDF5 Library creates global heap collections as needed, so + there may be multiple collections throughout the file. The set of all + of them is abstractly called the “global heap”, although + they do not actually link to each other, and there is no global place + in the file where you can discover all of the collections. The + collections are found simply by finding a reference to one through + another object in the file. For example, data of variable-length + datatype elements is stored in the global heap and is accessed via a + global heap ID. The format for global heap IDs is described at the end + of this section. + +
+
+
+
+
+
+
+
+ +
+
+
+
+
+
+
+
+ The format for the ID used to locate an object in the global heap + is described here: + +
+
+
+
+
+
-
- Using the computed values for nrows and
- max_dblock_rows, along with the Width of the
- doubling table, the number of direct and indirect block entries
- (K and N in the indirect block description, below)
- in an indirect block can be computed:
- + + III.F. Disk Format: Level 1F - Fractal Heap +-- The size indirect blocks on disk is determined by the number - of rows in the indirect block (computed above). The size of direct - blocks on disk is exactly the size of the block in the doubling - table. - ++ Each fractal heap consists of a header and zero or more direct and + indirect blocks (described below). The header contains general + information as well as initialization parameters for the doubling + table. The Root Block Address in the header points to the + first direct or indirect block in the heap. + -
-
-
+
+
+
|
+ Number of Huge Objects in Heap
+ |
+ This is the number of huge objects in the heap. +
|
+ Number of Tiny Objects in Heap
+ |
+
- This is the number of tiny objects that are packed in heap + IDs. + |
+ Table Width
+ |
+ This is the number of columns in the doubling table for + managed blocks. This value must be a power of two. +-
-
+
- -
-
+
-
|
+ (Items marked with an ‘O’ in the
+ above table are of the size specified in “Size of
+ Offsets” field in the superblock.) |
+
+ -
-
-
- -
-
+
+
+
+
+
Field Name |
+ Description |
+
- |
+ Signature
+ |
+
+ The ASCII character string “
+ -
-
-
-
- An object in the fractal heap is identified by means of a fractal heap ID, - which encodes information to locate the object in the heap. - Currently, the fractal heap stores an object in one of three ways, - depending on the object’s size: - -
-
-
-
-
The specific format for each type of heap ID is described below: - - -
-
-
-
-
-
+
+
+ An object in the fractal heap is identified by means of a fractal + heap ID, which encodes information to locate the object in the heap. + Currently, the fractal heap stores an object in one of three ways, + depending on the object’s size: + +
+
+
+
+
The specific format for each type of heap ID is described below: + + +
+
+
|
+ Data (variable size) - -
-
++
+
+
+ +
+
-
+
+
+
- -
-
+
+
|
+ (Items marked with an ‘L’ in the
+ above table are of the size specified in “Size of
+ Lengths” field in the superblock.) |
+
+ +
+
+ +
+
+
+
+
+
-
-
- -
-
-
-
-
-
+
+
+ +
+
+
+
+
+
-
-
- -
-
-
-
-
-
-
-
- -
-
-
-
-
-
+
- -III.G. Disk Format: Level 1G - Free-space Manager- -- Free-space managers are used to describe space within a heap or - the entire HDF5 file that is not currently used for that heap or - file. - - -- The free-space manager header contains metadata information - about the space being tracked, along with the address of the list - of free space sections which actually describes the free - space. The header records information about free-space sections being - tracked, creation parameters for handling free-space sections of a - client, and section information used to locate the collection of - free-space sections. - - -- The free-space section list stores a collection of - free-space sections that is specific to each client of the - free-space manager. - - For example, the fractal heap is a client of the free space manager - and uses it to track unused space within the heap. There are 4 - types of section records for the fractal heap, each of which has - its own format, listed below. - - -
-
-
-
-
-
- The free-space sections being managed are stored in a - free-space section list, described below. The sections - in the free-space section list are stored in the following way: - a count of the number of sections describing a particular size of - free space and the size of the free-space described (in bytes), - followed by a list of section description records; then another - section count and size, followed by the list of section - descriptions for that size; and so on. - - -
-
-
-
-
-
-
-
- - The section-type specific data for each free-space section record is - described below: - - -
-
-
-
- -
-
-
-
- -
-
-
-
- -
-
-
-
- -
-
-
-
-
-
+
- -III.H. Disk Format: Level 1H - Shared Object Header Message Table- -- The shared object header message table is used to locate - object - header messages that are shared between two or more object headers - in the file. Shared object header messages are stored and indexed - in the file in one of two ways: indexed sequentially in a - shared header message list or indexed with a v2 B-tree. - The shared messages themselves are either stored in a fractal - heap (when two or more objects share the message), or remain in an - object’s header (when only one object uses the message currently, - but the message can be shared in the future). - - -- The shared object header message table - contains a list of shared message index headers. Each index header - records information about the version of the index format, the index - storage type, flags for the message types indexed, the number of - messages in the index, the address where the index resides, - and the fractal heap address if shared messages are stored there. - - -- Each index can be either a list or a v2 B-tree and may transition - between those two forms as the number of messages in the index - varies. Each shared message record contains information used to - locate the shared message from either a fractal heap or an object - header. The types of messages that can be shared are: Dataspace, - Datatype, Fill Value, Filter Pipeline and Attribute. - - -- The shared object header message table is pointed to - from a shared message table message - in the superblock extension for a file. This message stores the - version of the table format, along with the number of index headers - in the table. - - -
-
+
+
-
-
+
+
+ + III.G. Disk Format: Level 1G - + Free-space Manager ++ +Free-space managers are used to describe space within a heap or + the entire HDF5 file that is not currently used for that heap or file. + + ++ The free-space manager header contains metadata information + about the space being tracked, along with the address of the list of free + space sections which actually describes the free space. The header + records information about free-space sections being tracked, creation + parameters for handling free-space sections of a client, and section + information used to locate the collection of free-space sections. + + ++ The free-space section list stores a collection of free-space + sections that is specific to each client of the free-space + manager. For example, the fractal heap is a client of the free space + manager and uses it to track unused space within the heap. There are 4 + types of section records for the fractal heap, each of which has its + own format, listed below. + + +
+
+
+
+
+
+
+
+ + The free-space sections being managed are stored in a free-space + section list, described below. The sections in the free-space section + list are stored in the following way: a count of the number of sections + describing a particular size of free space and the size of the + free-space described (in bytes), followed by a list of section + description records; then another section count and size, followed by + the list of section descriptions for that size; and so on. + + + +
+
+
+
+
+
+ The defined record types for a fractal heap client are: - - An index can track more than one type of message, but each type - of message can only by in one index. - - - + +
|
- Number of Messages for index #N
- |
+ The number of shared messages being tracked for the index. - - |
+ Record Set #N Section #K Data
+ |
This is the section-type specific information for each record + in the record set, described below. + |
- Index Address for index #N
- |
+ This field is the address of the list or v2 B-tree where the - index nodes reside. - - |
+ Checksum
+ |
+ + This is the checksum for the Free-space Section List. + ++ The section-type specific data for each free-space section record + is described below: + +
+
+
+
+ +
+
+
+
+ +
+
+
+
+ +
+
+
+
+ +
+
-
-
- - Shared messages are indexed either with a shared message record - list, described below, or using a v2 B-tree (using record type 7). - The number of records in the shared message record list is - determined in the index’s entry in the shared object header message - table. - - -
-
+
+
+
+
+
+
+ + III.H. Disk Format: Level 1H - Shared Object + Header Message Table ++ ++ The shared object header message table is used to locate + object header messages that are shared between two or more object + headers in the file. Shared object header messages are stored and + indexed in the file in one of two ways: indexed sequentially in a shared + header message list or indexed with a v2 B-tree. The shared messages + themselves are either stored in a fractal heap (when two or more + objects share the message), or remain in an object’s header (when + only one object uses the message currently, but the message can be + shared in the future). + + ++ The shared object header message table contains a list of + shared message index headers. Each index header records information + about the version of the index format, the index storage type, flags + for the message types indexed, the number of messages in the index, the + address where the index resides, and the fractal heap address if shared + messages are stored there. + + ++ Each index can be either a list or a v2 B-tree and may transition + between those two forms as the number of messages in the index varies. + Each shared message record contains information used to locate the + shared message from either a fractal heap or an object header. The + types of messages that can be shared are: Dataspace, Datatype, + Fill Value, Filter Pipeline and Attribute. + + ++ The shared object header message table is pointed to from a shared message table message in the + superblock extension for a file. This message stores the version of the + table format, along with the number of index headers in the table. + + +
+
-
-
-
-
-
-
- - The record for each shared message in an index is stored in one of the - following forms: - - -
-
-
-
-
-
-
-
- -
-
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
- | - | - (Items marked with an ‘O’ in the above table are of the size - specified in “Size of Offsets” field in the superblock.) - |
-
| Field Name | -Description | +
|---|
| Field Name | +Description | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Message Location |
-
- This has a value of 1 indicating that the message is stored in - an object header. - - |
+ Signature |
+
+
+ The ASCII character string “
+ |
||||||||||||||
Hash Value |
-
- This is the hash value for the message. - - |
+ Version for index #N |
+
+ This is the version number for the list of shared object + header message indexes and this document describes version 0. + |
||||||||||||||
Message Type |
-
- This is the message type in the object header. - - |
+ Index Type for index #N |
+
+ The type of index can be an unsorted list or a v2 B-tree. + |
||||||||||||||
Creation Index |
-
- This is the creation index of the message within the object - header. - - |
+ Message Type Flags for index #N |
+
+ This field indicates the type of messages tracked in the + index, as follows: +
An index can track more than one type of message, but each + type of message can only by in one index. + |
||||||||||||||
Object Header Address |
-
- This is the address of the object header where the message is - located. - - |
+ Minimum Message Size for index #N |
+
+ This is the message size sharing threshold for the index. If + the encoded size of the message is less than this value, the + message is not shared. + |
List Cutoff for index #N
This is the cutoff value for the indexing of messages to + switch from a list to a v2 B-tree. If the number of messages is + greater than this value, the index should be a v2 B-tree.
+v2 B-tree Cutoff for index #N
This is the cutoff value for the indexing of messages to + switch from a v2 B-tree back to a list. If the number of messages + is less than this value, the index should be a list.
+Number of Messages for index #N
The number of shared messages being tracked for the index.
+Index Address for index #N
This field is the address of the list or v2 B-tree where the + index nodes reside.
+Fractal Heap Address for index #N
This field is the address of the fractal heap if shared + messages are stored there.
+Checksum
This is the checksum for the table.
++
+ Shared messages are indexed either with a shared message + record list, described below, or using a v2 B-tree (using record type + 7). The number of records in the shared message record list is + determined in the index’s entry in the shared object + header message table. +
+ +| byte | +byte | +byte | +byte | +
|---|---|---|---|
| Signature | +|||
| Shared Message Record #0 | +|||
| Shared Message Record #1 | +|||
| ... | +|||
| Shared Message Record #N-1 | +|||
| Checksum | +|||
+
| Field Name | +Description | +
|---|---|
Signature |
+
+
+ The ASCII character string “
+ |
+
Shared Message Record #N |
+
+ + The record for locating the shared message, either in the fractal + heap for the index, or an object header (see format for index + nodes below). + + |
+
Checksum |
+
+ This is the checksum for the list. + |
+
+
The record for each shared message in an index is stored in one + of the following forms:
+ +| byte | +byte | +byte | +byte | +
|---|---|---|---|
| Message Location | +This space inserted + only to align table nicely | +||
| Hash Value | +|||
| Reference Count | +|||
Fractal Heap ID + |
+ |||
+
| Field Name | +Description | +
|---|---|
Message Location |
+
+ This has a value of 0 indicating that the message is stored + in the heap. + |
+
Hash Value |
+
+ This is the hash value for the message. + |
+
Reference Count |
+
+ This is the number of times the message is used in the file. + + |
+
Fractal Heap ID |
+
+ This is an 8-byte fractal heap ID for the message as stored + in the fractal heap for the index. + |
+
+
+
| byte | +byte | +byte | +byte | +
|---|---|---|---|
| Message Location | +This space inserted + only to align table nicely | +||
| Hash Value | +|||
| Reserved | +Message Type | +Creation Index | +|
Object Header AddressO + |
+ |||
| + | (Items marked with an ‘O’ in the + above table are of the size specified in “Size of + Offsets” field in the superblock.) | +
+
| Field Name | +Description | +
|---|---|
Message Location |
+
+ This has a value of 1 indicating that the message is stored + in an object header. + |
+
Hash Value |
+
+ This is the hash value for the message. + |
+
Message Type |
+
+ This is the message type in the object header. + |
+
Creation Index |
+
+ This is the creation index of the message within the object + header. + |
+
Object Header Address |
+
+ This is the address of the object header where the message is + located. + |
+
-
-IV. Disk Format: Level 2 - Data Objects
++ IV. Disk Format: Level 2 - Data Objects +
-Data objects contain the “real” user-visible information in the file. - These objects compose the scientific data and other information which - are generally thought of as “data” by the end-user. All the - other information in the file is provided as a framework for - storing and accessing these data objects. -
+Data objects contain the “real” user-visible + information in the file. These objects compose the scientific data and + other information which are generally thought of as “data” + by the end-user. All the other information in the file is provided as a + framework for storing and accessing these data objects.
-A data object is composed of header and data - information. The header information contains the information - needed to interpret the data information for the object as - well as additional “metadata” or pointers to additional - “metadata” used to describe or annotate each object. -
+A data object is composed of header and data information. The + header information contains the information needed to interpret the + data information for the object as well as additional + “metadata” or pointers to additional “metadata” + used to describe or annotate each object.
-
-IV.A. Disk Format: Level 2A - Data Object Headers
++ IV.A. Disk Format: Level 2A - Data Object + Headers +
-The header information of an object is designed to encompass - all of the information about an object, except for the data itself. - This information includes the dataspace, the datatype, information - about how the data is stored on disk (in external files, compressed, - broken up in blocks, and so on), as well as other information used - by the library to speed up access to the data objects or maintain - a file’s integrity. Information stored by user applications - as attributes is also stored in the object’s header. The header - of each object is not necessarily located immediately prior to the - object’s data in the file and in fact may be located in any - position in the file. The order of the messages in an object header - is not significant.
+The header information of an object is designed to encompass all + of the information about an object, except for the data itself. This + information includes the dataspace, the datatype, information about how + the data is stored on disk (in external files, compressed, broken up in + blocks, and so on), as well as other information used by the library to + speed up access to the data objects or maintain a file’s + integrity. Information stored by user applications as attributes is + also stored in the object’s header. The header of each object is + not necessarily located immediately prior to the object’s data in + the file and in fact may be located in any position in the file. The + order of the messages in an object header is not significant.
-Object headers are composed of a prefix and a set of messages. The - prefix contains the information needed to interpret the messages and - a small amount of metadata about the object, and the messages contain - the majority of the metadata about the object. -
+Object headers are composed of a prefix and a set of messages. + The prefix contains the information needed to interpret the messages + and a small amount of metadata about the object, and the messages + contain the majority of the metadata about the object.
-
-IV.A.1. Disk Format: Level 2A1 - Data Object Header Prefix
++ IV.A.1. Disk Format: Level 2A1 - Data + Object Header Prefix +
-
-IV.A.1.a. Version 1 Data Object Header Prefix
++ IV.A.1.a. Version 1 Data Object + Header Prefix +
-Header messages are aligned on 8-byte boundaries for version 1 - object headers. -
+Header messages are aligned on 8-byte boundaries for version 1 + object headers.
-| byte | -byte | -byte | -byte | -
|---|---|---|---|
| Version | -Reserved (zero) | -Total Number of Header Messages | -|
| Object Reference Count | -|||
| Object Header Size | -|||
| Header Message Type #1 | -Size of Header Message Data #1 | -||
| Header Message #1 Flags | -Reserved (zero) | -||
Header Message Data #1 |
- |||
| . . . |
- |||
| Header Message Type #n | -Size of Header Message Data #n | -||
| Header Message #n Flags | -Reserved (zero) | -||
Header Message Data #n |
- |||
-
| Field Name | -Description | -||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Version |
-
- This value is used to determine the format of the - information in the object header. When the format of the - object header is changed, the version number - is incremented and can be used to determine how the - information in the object header is formatted. This - is version one (1) (there was no version zero (0)) of the - object header. - - |
- ||||||||||||||||||
Total Number of Header Messages |
-
- This value determines the total number of messages listed in - object headers for this object. This value includes the messages - in continuation messages for this object. - - |
- ||||||||||||||||||
Object Reference Count |
-
- This value specifies the number of “hard links” to this object - within the current file. References to the object from external - files, “soft links” in this file and object references in this - file are not tracked. - - |
- ||||||||||||||||||
Object Header Size |
-
- This value specifies the number of bytes of header message data - following this length field that contain object header messages - for this object header. This value does not include the size of - object header continuation blocks for this object elsewhere in the - file. - - |
- ||||||||||||||||||
Header Message #n Type |
-
- This value specifies the type of information included in the - following header message data. The message types for - header messages are defined in sections below. - - |
- ||||||||||||||||||
Size of Header Message #n Data |
-
- This value specifies the number of bytes of header - message data following the header message type and length - information for the current message. The size includes - padding bytes to make the message a multiple of eight - bytes. - - |
- ||||||||||||||||||
Header Message #n Flags |
-
- This is a bit field with the following definition: -
|
- ||||||||||||||||||
Header Message #n Data |
-
- The format and length of this field is determined by the - header message type and size respectively. Some header - message types do not require any data and this information - can be eliminated by setting the length of the message to - zero. The data is padded with enough zeroes to make the - size a multiple of eight. - - |
-
-
-IV.A.1.b. Version 2 Data Object Header Prefix
- -Note that the “total number of messages” field has been dropped from - the data object header prefix in this version. The number of messages - in the data object header is just determined by the messages encountered - in all the object header blocks.
- -Note also that the fields and messages in this version of data object - headers have no alignment or padding bytes inserted - they are - stored packed together.
- -| byte | -byte | -byte | -byte | -
|---|---|---|---|
| Signature | -|||
| Version | -Flags | -This space inserted only to align table nicely | -|
| Access time (optional) | -|||
| Modification Time (optional) | -|||
| Change Time (optional) | -|||
| Birth Time (optional) | -|||
| Maximum # of compact attributes (optional) | -Minimum # of dense attributes (optional) | -||
| Size of Chunk #0 (variable size) | -This space inserted only to align table nicely | -||
| Header Message Type #1 | -Size of Header Message Data #1 | -Header Message #1 Flags | -|
| Header Message #1 Creation Order (optional) | -This space inserted only to align table nicely | -||
Header Message Data #1 |
- |||
| . . . |
- |||
| Header Message Type #n | -Size of Header Message Data #n | -Header Message #n Flags | -|
| Header Message #n Creation Order (optional) | -This space inserted only to align table nicely | -||
Header Message Data #n |
- |||
| Gap (optional, variable size) | -|||
| Checksum | -|||
-
| Field Name | -Description | -||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Signature |
-
- The ASCII character string “ |
- ||||||||||||||||||||||||
Version |
-
- This field has a value of 2 indicating version 2 of the object header. - - |
- ||||||||||||||||||||||||
Flags |
-
- This field is a bit field indicating additional information - about the object header. -
|
- ||||||||||||||||||||||||
Access Time |
-
- This 32-bit value represents the number of seconds after the - UNIX epoch when the object’s raw data was last accessed - (in other words, read or written). - -This field is present if bit 5 of flags is set. - - |
- ||||||||||||||||||||||||
Modification Time |
-
- This 32-bit value represents the number of seconds after - the UNIX epoch when the object’s raw data was last - modified (in other words, written). - -This field is present if bit 5 of flags is set. - - |
- ||||||||||||||||||||||||
Change Time |
-
- This 32-bit value represents the number of seconds after the - UNIX epoch when the object’s metadata was last changed. - -This field is present if bit 5 of flags is set. - - |
- ||||||||||||||||||||||||
Birth Time |
-
- This 32-bit value represents the number of seconds after the - UNIX epoch when the object was created. - -This field is present if bit 5 of flags is set. - - |
- ||||||||||||||||||||||||
Maximum # of compact attributes |
-
- This is the maximum number of attributes to store in the compact - format before switching to the indexed format. - -This field is present if bit 4 of flags is set. - - |
- ||||||||||||||||||||||||
Minimum # of dense attributes |
-
- This is the minimum number of attributes to store in the indexed - format before switching to the compact format. - -This field is present if bit 4 of flags is set. - - |
- ||||||||||||||||||||||||
Size of Chunk #0 |
-
- - This unsigned value specifies the number of bytes of header - message data following this field that contain object header - information. - -- This value does not include the size of object header - continuation blocks for this object elsewhere in the file. - -- The length of this field varies depending on bits 0 and 1 of - the flags field. - - |
- ||||||||||||||||||||||||
Header Message #n Type |
-
- Same format as version 1 of the object header, described above. - - |
- ||||||||||||||||||||||||
Size of Header Message #n Data |
-
- This value specifies the number of bytes of header - message data following the header message type and length - information for the current message. The size of messages - in this version does not include any padding bytes. - - |
- ||||||||||||||||||||||||
Header Message #n Flags |
-
- Same format as version 1 of the object header, described above. - - |
- ||||||||||||||||||||||||
Header Message #n Creation Order |
-
- This field stores the order that a message of a given type - was created in. - -This field is present if bit 2 of flags is set. - - |
- ||||||||||||||||||||||||
Header Message #n Data |
-
- Same format as version 1 of the object header, described above. - - |
- ||||||||||||||||||||||||
Gap |
-
- A gap in an object header chunk is inferred by the end of the - messages for the chunk before the beginning of the chunk’s - checksum. Gaps are always smaller than the size of an - object header message prefix (message type + message size + - message flags). - -Gaps are formed when a message (typically an attribute message) - in an earlier chunk is deleted and a message from a later - chunk that does not quite fit into the free space is moved - into the earlier chunk. - - |
- ||||||||||||||||||||||||
Checksum |
-
- This is the checksum for the object header chunk. - - |
+ byte | +byte | +byte | +byte |
|---|
The header message types and the message data associated with - them compose the critical “metadata” about each object. Some - header messages are required for each object while others are - optional. Some optional header messages may also be repeated - several times in the header itself, the requirements and number - of times allowed in the header will be noted in each header - message description below. -
+Header Message Data #1
+
.
.
Header Message Data #n
+
+
| Field Name | +Description | +||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Version |
+
+ This value is used to determine the format of the information + in the object header. When the format of the object header is + changed, the version number is incremented and can be used to + determine how the information in the object header is formatted. + This is version one (1) (there was no version zero (0)) of the + object header. + |
+ ||||||||||||||||||
Total Number of Header Messages |
+
+ This value determines the total number of messages listed in + object headers for this object. This value includes the messages in + continuation messages for this object. + |
+ ||||||||||||||||||
Object Reference Count |
+
+ This value specifies the number of “hard links” + to this object within the current file. References to the object + from external files, “soft links” in this file and + object references in this file are not tracked. + |
+ ||||||||||||||||||
Object Header Size |
+
+ This value specifies the number of bytes of header message + data following this length field that contain object header + messages for this object header. This value does not include the + size of object header continuation blocks for this object elsewhere + in the file. + |
+ ||||||||||||||||||
Header Message #n Type |
+
+ This value specifies the type of information included in the + following header message data. The message types for header + messages are defined in sections below. + |
+ ||||||||||||||||||
Size of Header Message #n Data |
+
+ This value specifies the number of bytes of header message + data following the header message type and length information for + the current message. The size includes padding bytes to make the + message a multiple of eight bytes. + |
+ ||||||||||||||||||
Header Message #n Flags |
+
+ This is a bit field with the following definition: +
|
+ ||||||||||||||||||
Header Message #n Data |
+
+ The format and length of this field is determined by the + header message type and size respectively. Some header message + types do not require any data and this information can be + eliminated by setting the length of the message to zero. The data + is padded with enough zeroes to make the size a multiple of eight. + + |
+
+
+ IV.A.1.b. Version 2 Data Object + Header Prefix +
+ +Note that the “total number of messages” field has + been dropped from the data object header prefix in this version. The + number of messages in the data object header is just determined by the + messages encountered in all the object header blocks.
+ ++ Note also that the fields and messages in this version of data object + headers have no alignment or padding bytes inserted - they are + stored packed together. +
+ +| byte | +byte | +byte | +byte | +
|---|---|---|---|
| Signature | +|||
| Version | +Flags | +This space inserted + only to align table nicely | +|
| Access time (optional) | +|||
| Modification Time (optional) | +|||
| Change Time (optional) | +|||
| Birth Time (optional) | +|||
| Maximum # of compact attributes (optional) | +Minimum # of dense attributes (optional) | +||
| Size of Chunk #0 (variable size) | +This space inserted + only to align table nicely | +||
| Header Message Type #1 | +Size of Header Message Data #1 | +Header Message #1 Flags | +|
| Header Message #1 Creation Order (optional) | +This space inserted + only to align table nicely | +||
Header Message Data #1 + |
+ |||
| . . . |
+ |||
| Header Message Type #n | +Size of Header Message Data #n | +Header Message #n Flags | +|
| Header Message #n Creation Order (optional) | +This space inserted + only to align table nicely | +||
Header Message Data #n + |
+ |||
| Gap (optional, variable size) | +|||
| Checksum | +|||
+
| Field Name | +Description | +||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Signature |
+
+
+ The ASCII character string “
+ |
+ ||||||||||||||||||||||||
Version |
+
+ This field has a value of 2 indicating version 2 of the + object header. + |
+ ||||||||||||||||||||||||
Flags |
+
+ This field is a bit field indicating additional information + about the object header. +
|
+ ||||||||||||||||||||||||
Access Time |
+
+ This 32-bit value represents the number of seconds after the + UNIX epoch when the object’s raw data was last accessed (in + other words, read or written). ++ This field is present if bit 5 of flags is set. + + |
+ ||||||||||||||||||||||||
Modification Time |
+
+ This 32-bit value represents the number of seconds after the + UNIX epoch when the object’s raw data was last modified (in + other words, written). ++ This field is present if bit 5 of flags is set. + + |
+ ||||||||||||||||||||||||
Change Time |
+
+ This 32-bit value represents the number of seconds after the + UNIX epoch when the object’s metadata was last changed. ++ This field is present if bit 5 of flags is set. + + |
+ ||||||||||||||||||||||||
Birth Time |
+
+ This 32-bit value represents the number of seconds after the + UNIX epoch when the object was created. ++ This field is present if bit 5 of flags is set. + + |
+ ||||||||||||||||||||||||
Maximum # of compact attributes |
+
+ This is the maximum number of attributes to store in the + compact format before switching to the indexed format. ++ This field is present if bit 4 of flags is set. + + |
+ ||||||||||||||||||||||||
Minimum # of dense attributes |
+
+ This is the minimum number of attributes to store in the + indexed format before switching to the compact format. ++ This field is present if bit 4 of flags is set. + + |
+ ||||||||||||||||||||||||
Size of Chunk #0 |
+
+ This unsigned value specifies the number of bytes of header + message data following this field that contain object header + information. +This value does not include the size of object header + continuation blocks for this object elsewhere in the file. ++ The length of this field varies depending on bits 0 and 1 of the flags + field. + + |
+ ||||||||||||||||||||||||
Header Message #n Type |
+
+ Same format as version 1 of the object header, described + above. + |
+ ||||||||||||||||||||||||
Size of Header Message #n Data |
+
+ + This value specifies the number of bytes of header message data + following the header message type and length information for the + current message. The size of messages in this version does not + include any padding bytes. + + |
+ ||||||||||||||||||||||||
Header Message #n Flags |
+
+ Same format as version 1 of the object header, described + above. + |
+ ||||||||||||||||||||||||
Header Message #n Creation Order |
+
+ This field stores the order that a message of a given type + was created in. ++ This field is present if bit 2 of flags is set. + + |
+ ||||||||||||||||||||||||
Header Message #n Data |
+
+ Same format as version 1 of the object header, described + above. + |
+ ||||||||||||||||||||||||
Gap |
+
+ A gap in an object header chunk is inferred by the end of the + messages for the chunk before the beginning of the chunk’s + checksum. Gaps are always smaller than the size of an object header + message prefix (message type + message size + message flags). +Gaps are formed when a message (typically an attribute + message) in an earlier chunk is deleted and a message from a later + chunk that does not quite fit into the free space is moved into the + earlier chunk. + |
+ ||||||||||||||||||||||||
Checksum |
+
+ This is the checksum for the object header chunk. + |
+
The header message types and the message data associated with + them compose the critical “metadata” about each object. + Some header messages are required for each object while others are + optional. Some optional header messages may also be repeated several + times in the header itself, the requirements and number of times + allowed in the header will be noted in each header message description + below.
-
-IV.A.2. Disk Format: Level 2A2 - Data Object Header Messages
++ IV.A.2. Disk Format: Level 2A2 - + Data Object Header Messages +
-Data object header messages are small pieces of metadata that are - stored in the data object header for each object in an HDF5 file. - Data object header messages provide the metadata required to describe - an object and its contents, as well as optional pieces of metadata - that annotate the meaning or purpose of the object. -
+Data object header messages are small pieces of metadata that are + stored in the data object header for each object in an HDF5 file. Data + object header messages provide the metadata required to describe an + object and its contents, as well as optional pieces of metadata that + annotate the meaning or purpose of the object.
-Data object header messages are either stored directly in the data - object header for the object or are shared between multiple objects - in the file. When a message is shared, a flag in the Message Flags - indicates that the actual Message Data - portion of that message is stored in another location (such as another - data object header, or a heap in the file) and the Message Data - field contains the information needed to locate the actual information - for the message. -
++ Data object header messages are either stored directly in the data + object header for the object or are shared between multiple objects in + the file. When a message is shared, a flag in the Message + Flags indicates that the actual Message Data portion of that + message is stored in another location (such as another data object + header, or a heap in the file) and the Message Data field + contains the information needed to locate the actual information for + the message. +
-- The format of shared message data is described here:
+The format of shared message data is described here:
-| byte | -byte | -byte | -byte | -
|---|---|---|---|
| Version | -Type | -Reserved (zero) | -|
| Reserved (zero) | -|||
AddressO |
- |||
| - | - (Items marked with an ‘O’ in the above table are of the size - specified in “Size of Offsets” field in the superblock.) - |
-
| Field Name | -Description | -||||||
|---|---|---|---|---|---|---|---|
Version |
- The version number is used when there are changes in the format - of a shared object message and is described here: -
|
- ||||||
Type |
- The type of shared message location: -
|
- ||||||
Address |
- The address of the object header - containing the message to be shared. - |
-
-
-
| byte | -byte | -byte | -byte | -
|---|---|---|---|
| Version | -Type | -This space inserted only to align table nicely | -|
AddressO |
- |||
| - | - (Items marked with an ‘O’ in the above table are of the size - specified in “Size of Offsets” field in the superblock.) - |
-
| Field Name | -Description | -||||||
|---|---|---|---|---|---|---|---|
| Reserved (zero) | +|||||||
Version |
- The version number is used when there are changes in the format - of a shared object message and is described here: -
| ||||||
2 |
- Used by the library of version 1.6.1 and after. - | -||||||
| + | (Items marked with an ‘O’ in the + above table are of the size specified in “Size of + Offsets” field in the superblock.) | +
Type
The type of shared message location: -
| Value | -Description | -
|---|---|
0 |
- Message stored in another object’s header (a committed - message). - | -
Address
The address of the object header - containing the message to be shared.
-
-
| byte | -byte | -byte | -byte | -
|---|---|---|---|
| Version | -Type | -This space inserted only to align table nicely | -|
| Location (variable size) | -|||
-
| Field Name | -Description | -||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
Version |
- The version number indicates changes in the format of shared - object message and is described here: -
|
- ||||||||||
Type |
- The type of shared message location: -
|
- ||||||||||
Location |
- This field contains either a Size of Offsets-bytes - address of the object header - containing the message to be shared, or an 8-byte fractal heap ID - for the message in the file’s shared object header message - heap. - - |
-
The following is a list of currently defined header messages: -
+-
IV.A.2.a. The NIL Message
+| Field Name | +Description | +||||||
|---|---|---|---|---|---|---|---|
Version |
+ The version number is used when there are changes in + the format of a shared object message and is described here: +
|
+ ||||||
Type |
+ The type of shared message location: +
|
+ ||||||
Address |
+ The address of the object header containing the + message to be shared. |
+
+
+
| byte | +byte | +byte | +byte | +
|---|---|---|---|
| Version | +Type | +This space inserted + only to align table nicely | +|
AddressO + |
+ |||
| + | (Items marked with an ‘O’ in the + above table are of the size specified in “Size of + Offsets” field in the superblock.) | +
+
| Field Name | +Description | +||||
|---|---|---|---|---|---|
Version |
+ The version number is used when there are changes in + the format of a shared object message and is described here: +
|
+ ||||
Type |
+ The type of shared message location: +
|
+ ||||
Address |
+ The address of the object header containing the + message to be shared. |
+
+
+
| byte | +byte | +byte | +byte | +
|---|---|---|---|
| Version | +Type | +This space inserted + only to align table nicely | +|
| Location (variable size) | +|||
+
| Field Name | +Description | +||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
Version |
+ The version number indicates changes in the format of + shared object message and is described here: +
|
+ ||||||||||
Type |
+ The type of shared message location: +
|
+ ||||||||||
Location |
+ + This field contains either a Size of Offsets-bytes address + of the object header containing the message to be shared, or an + 8-byte fractal heap ID for the message in the file’s shared + object header message heap. + |
+
The following is a list of currently defined header messages:
+ ++
+ IV.A.2.a. The NIL Message +
+ + +| Header Message Name: NIL | |
| Header Message Type: 0x0000 | |
| Length: Varies | |
| Status: Optional; may be repeated. | |
| Description: | -The NIL message is used to indicate a message which is to be - ignored when reading the header messages for a data object. - [Possibly one which has been deleted for some reason.] - |
| Format of Data: Unspecified | |
.
- You will be asked to provide the following information: