Commit 2bb96c6f authored by Marc Howison's avatar Marc Howison

updated NEWS with details about 1.6 changes; changed hsize_t type in...

updated NEWS with details about 1.6 changes; changed hsize_t type in H5PartSetViewIndices; fixes to the build system; fixed bugs that were preventing the use of H5PartSetView during writes; added open object checks to regression test
parent 181c0300
#### H5PART 1.6 ##############################################################
Updated Documentation
---------------------
The Fortran API has been added to the doxygen documentation. Most entries have
a reference to their respective C API call.
Regression Test Suite
---------------------
The test subdirectory has been reorganized to include a systematic series
of regression tests of common use cases for the API. The goal is to have both
complete coverage of the API (every call is exercised) and testing of some
unlikely or unsuspected values and cases.
Currently, the C test covers the entire H5Part API and the Fortran test is
still in development.
Benchmark Utilities
-------------------
Two benchmarking utilities, H5PartBench and H5BlockBench, have been added to
the tools collection. Both require the parallel library and have functionality
and syntax similar to the IOR benchmark:
http://sourceforge.net/projects/ior-sio/
File Mode Flags
---------------
Previously, the only flags used were H5PART_READ, H5PART_WRITE, and
H5PART_APPEND, and these were mutually exclusive. Additional flags have been
added to select the underlying virtual file driver (VFD) that HDF5 uses in
parallel mode and to accommodate the lustre filesystem:
H5PART_VFD_MPIPOSIX (use the MPI-POSIX driver, which bypasses MPI-IO)
H5PART_VFD_MPIIO_IND (use MPI-IO in independent mode)
H5PART_FS_LUSTRE (activate H5P tunings for the lustre stripe size)
These flags can be specified using the typical bitwise OR method, e.g.
char flag = H5PART_WRITE | H5PART_VFD_MPIPOSIX | H5PART_FS_LUSTRE;
will select write mode with these two additional options.
The parallel library now defaults to using the MPI-IO collective mode driver
(previously it used independent mode). The collective buffering algorithm in
the Cray XT4/5 environment has been substantially improved in the last year,
and defaulting to collective mode allows us to take advantage of these
improvements automatically.
There are also new open calls H5PartOpenAlign and H5PartOpenParallelAlign that
take an additional 'alignment' value. This value is passed to HDF5 and used to
pad out objects so that they align to filesystem boundaries. For instance, it
would make sense to use the stripe size as the alignment value when writing to
a lustre filesystem.
Unified 'View' Model
--------------------
Views are now supported in both read and write mode (previously only supported
in read mode). It is possible to specify non-sensical views: you could for
instance specify a read view that is larger than the dataset on disk. Or you
could create a write view that is larger than the data in memory. In those
cases, you will encounter an error when you try to perform a read or write
operation on a dataset.
There are two new methods for selecting views. H5PartSetNumParticlesStrided
selects a view with $n$ particles per processor, but such that the data in
memory is expected to have a 'stride' factor. For instance, if you have
particle data with fields $x$ and $y$ and a single array with entries
$x1,y1,x2,y2...$, then the stride factor is 2. Striding works for both reads
and writes and only affects the view of memory: individual fields are still
stored as individual arrays on disk.
The second new method is H5PartSetViewIndices, which allows for point
selections of datasets. You can pass a list of indices for the points you
want to select. This is useful, for instance, when interfacing with FastBit
to perform queries that select a small subset of non-contiguous particles.
Because views are now supported on write, it is possible to write a dataset
using multiple 'passes' or to leave some values unwritten.
Internal Handling of Dataspaces
-------------------------------
H5PartSetNumParticles and H5PartSetNumParticlesStrided now share the same
HDF5 dataspace state in the H5PartFile struct with the calls that modify
the view (H5PartSetView etc.). Previously, the memory and disk dataspaces
were regenerated from the view on every read access, which was unnecessary.
Autogeneration of API Calls
---------------------------
Python scripts have been added to automatically generate read/write call
variants for different data types.
Renamed Fortran Include
-----------------------
The automake system no longer greps the *F90.inc files to generate an include
called 'H5Part.inc'. Instead, the *F90.inc files have been renamed to *.f90,
and awk is used to generate an 'H5PartF.h' file (the same naming convention
as in MPI's 'mpif.h').
The change to *.f90 was necessary to facilitate integration of the Fortran
API into doxygen. Note that the *.f90 files will not compile, nor can they
be included directly. Simply use
include 'H5PartF.h'
in your Fortran code to include all the definitions for Fortran H5Part/H5Block
calls.
64 Char Limit on Dataset Names
------------------------------
Previously, a user could overrun internal buffers for dataset names. Now, a
fixed limit of 64 chars is imposed. Dataset names that are longer than this are
truncated and a warning is printed. We expect that most users are using
short canoncical names like x, px, id, etc.
#### H5PART 1.4 ##############################################################
Error Handling
--------------
Now all functions, in which an error could occure, are returning a
......
......@@ -390,6 +390,7 @@ if test "X$USE_PARALLEL" = "Xyes"; then
fi
fi
H5P_LIB_NAME="-lpH5Part"
MTARGET="libpH5Part.a"
TTARGET="test"
......@@ -417,6 +418,7 @@ if test "X$USE_PARALLEL" = "Xyes"; then
FC=${MPIFC}
fi
H5P_LIB_NAME="-lpH5PartF"
MTARGET="${MTARGET} libpH5PartF.a"
TTARGET="${TTARGET} testf"
fi
......@@ -424,10 +426,12 @@ if test "X$USE_PARALLEL" = "Xyes"; then
else # --enable-parallel=no
AC_MSG_RESULT([no])
H5P_LIB_NAME="-lH5Part"
MTARGET="libH5Part.a"
TTARGET="test"
if test "X$USE_FORTRAN" = "Xyes"; then
H5P_LIB_NAME="-lH5PartF"
MTARGET="${MTARGET} libH5PartF.a"
TTARGET="${TTARGET} testf"
fi
......@@ -596,6 +600,7 @@ AC_SUBST(CFLAGS)
AC_SUBST(FFLAGS)
AC_SUBST(STDCXX)
AC_SUBST(H5P_LIB_LOC)
AC_SUBST(H5P_LIB_NAME)
AC_SUBST(UNDERSCORE_H)
AC_SUBST(BUILD_TOOLS)
AC_SUBST(LDFLAGS)
......
This diff is collapsed.
......@@ -205,7 +205,7 @@ H5PartSetView (
h5part_int64_t
H5PartSetViewIndices (
H5PartFile *f, /*!< [in] Handle to open file */
const hsize_t *indices, /*!< [in] List of indices */
const h5part_int64_t *indices, /*!< [in] List of indices */
h5part_int64_t nelems /*!< [in] Size of list */
);
......
......@@ -24,6 +24,7 @@ typedef unsigned long MPI_Comm;
#endif
#define H5PART_STEPNAME_LEN 64
#define H5PART_DATANAME_LEN 64
struct H5BlockFile;
......
......@@ -4,6 +4,11 @@ OBJEXT=o
INCLUDES = -I@HDF5ROOT@/include @MPIINC@
EXTRA_HEADERS = H5PartPrivate.h \
H5BlockPrivate.h \
H5MultiBlockPrivate.h \
H5PartAttrib.h
# Extra files that I wish to include in the dist tar ball.
EXTRA_DIST = TestUnderscoreC.c \
TestUnderscore.f \
......@@ -13,7 +18,8 @@ EXTRA_DIST = TestUnderscoreC.c \
H5Part.f90 \
H5PartAttrib.f90 \
H5Block.f90 \
H5BlockReadWrite.f90
H5BlockReadWrite.f90 \
$(EXTRA_HEADERS)
# Files that I don't want to include in the dist tar ball
nodist_include_HEADERS = H5PartF.h @UNDERSCORE_H@
......@@ -40,12 +46,6 @@ include_HEADERS = H5Part.h \
H5PartF.h \
@UNDERSCORE_H@
# Listing of all possible headers that I may include
EXTRA_HEADERS = H5PartPrivate.h \
H5BlockPrivate.h \
H5MultiBlockPrivate.h \
H5PartAttrib.h
# Listing of sources
libH5Part_a_SOURCES = H5Part.c \
H5PartAttrib.c \
......@@ -54,7 +54,8 @@ libH5Part_a_SOURCES = H5Part.c \
H5MultiBlock.c \
H5MultiBlockReadWrite.c
libH5PartF_a_SOURCES = H5PartF.c \
libH5PartF_a_SOURCES = $(libH5Part_a_SOURCES) \
H5PartF.c \
H5PartAttribF.c \
H5BlockF.c \
H5BlockReadWriteF.c
......
......@@ -3,7 +3,7 @@
OBJEXT=o
HDFLIB = -L@HDF5ROOT@/lib -lhdf5 -lz @SZLIB@
LIBS = -L@H5P_LIB_LOC@ -lH5PartF -lH5Part $(HDFLIB) -lm
LIBS = -L@H5P_LIB_LOC@ @H5P_LIB_NAME@ $(HDFLIB) -lm
INCLUDES = -I@H5P_LIB_LOC@ -I@HDF5ROOT@/include @MPIINC@
......@@ -18,7 +18,8 @@ test_SOURCES = test.c \
testframe.c \
write.c \
read.c \
testframe.h
testframe.h \
params.h
testf_SOURCES = testf.F90
......
......@@ -2,7 +2,7 @@
#define _H5PART_TEST_PARAMS_H_
#define FILENAME "test.h5"
#define LONGSTEPNAME "thisisaverylongstepnamethatshouldexceedthelimitof64charcausingawarningtoprint"
#define LONGNAME "thisisaverylongnamethatshouldexceedthelimitof64charcausingawarningtoprint"
#define NTIMESTEPS 10
/* do not decrease this value below 99, or it will break assumptions
......
......@@ -61,10 +61,10 @@ int main(int argc, char **argv)
//if (GetTestCleanup() && !getenv("HDF5_NOCLEANUP"))
// TestCleanup();
//return GetTestNumErrs();
#ifdef PARALLEL_IO
return MPI_Finalize();
#else
return GetTestNumErrs();
#endif
}
......@@ -649,3 +649,38 @@ test_is_valid(H5PartFile *file)
RETURN(status, H5PART_SUCCESS, "H5PartFileIsValid");
}
void
test_open_objects(H5PartFile *file, int max_objects)
{
ssize_t nopen = H5Fget_obj_count(file->file, H5F_OBJ_ALL);
if (nopen > max_objects)
{
TestErrPrintf( "*** TOO MANY OBJECTS OPEN: %d > %d "
"at line %4d in %s\n", nopen, max_objects,
(int)__LINE__, __FILE__ );
hid_t *list = malloc(sizeof(hid_t)*nopen);
H5Fget_obj_ids(file->file, H5F_OBJ_ALL, nopen, list);
H5O_info_t info;
int i;
for (i=0; i<nopen; i++) {
H5Oget_info(list[i], &info);
switch (info.type) {
case H5O_TYPE_GROUP:
TestErrPrintf("obj%d has type GROUP\n", i);
break;
case H5O_TYPE_DATASET:
TestErrPrintf("obj%d has type DATASET\n", i);
break;
case H5O_TYPE_NAMED_DATATYPE:
TestErrPrintf("obj%d has type NAMED_DATATYPE\n", i);
break;
default:
TestErrPrintf("obj%d has unknown type\n", i);
}
}
free(list);
}
}
......@@ -184,5 +184,8 @@ get_attr_name(char *name, char *tag, int id);
void
test_is_valid(H5PartFile *file);
void
test_open_objects(H5PartFile *file, int max_objects);
#endif
......@@ -83,6 +83,10 @@ test_write_data64(H5PartFile *file, int nparticles, int step)
status = H5PartSetNumParticlesStrided(file, nparticles, -1);
RETURN(status, H5PART_SUCCESS, "H5PartSetNumParticlesStrided");
/* invalid nparticles will produce an error */
status = H5PartSetNumParticlesStrided(file, -1, 2);
RETURN(status, H5PART_ERR_INVAL, "H5PartSetNumParticlesStrided");
#if PARALLEL_IO
TEST("Setting throttle");
status = H5PartSetThrottle(file, 2);
......@@ -195,6 +199,7 @@ test_write_data32(H5PartFile *file, int nparticles, int step)
{
int i,t;
h5part_int32_t status;
int rank, nprocs;
float *x,*y,*z;
float *px,*py,*pz;
......@@ -216,6 +221,12 @@ test_write_data32(H5PartFile *file, int nparticles, int step)
TEST("Setting throttle");
status = H5PartSetThrottle(file, 2);
RETURN(status, H5PART_SUCCESS, "H5PartSetThrottle");
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
#else
rank = 0;
nprocs = 1;
#endif
TEST("Writing 32-bit data");
......@@ -236,8 +247,11 @@ test_write_data32(H5PartFile *file, int nparticles, int step)
status = H5PartSetStep(file, t);
RETURN(status, H5PART_SUCCESS, "H5PartSetStep");
status = H5PartSetNumParticles(file, nparticles);
RETURN(status, H5PART_SUCCESS, "H5PartSetNumParticles");
/* test a two-part write using views */
status = H5PartSetView(file,
rank*nparticles,
rank*nparticles + 31);
RETURN(status, H5PART_SUCCESS, "H5PartSetView");
status = H5PartWriteDataFloat32(file, "x", x);
RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
......@@ -259,7 +273,34 @@ test_write_data32(H5PartFile *file, int nparticles, int step)
status = H5PartWriteDataFloat32(file, "pz", pz);
RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
status = H5PartWriteDataInt32(file, "id", id);
status = H5PartWriteDataInt32(file, LONGNAME, id);
RETURN(status, H5PART_SUCCESS, "H5PartWriteDataInt32");
/* the second write phase... */
status = H5PartSetView(file,
rank*nparticles + 32,
rank*nparticles + nparticles - 1);
RETURN(status, H5PART_SUCCESS, "H5PartSetView");
/* offset the input arrays */
status = H5PartWriteDataFloat32(file, "x", x+32);
RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
status = H5PartWriteDataFloat32(file, "y", y+32);
RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
status = H5PartWriteDataFloat32(file, "z", z+32);
RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
status = H5PartWriteDataFloat32(file, "px", px+32);
RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
status = H5PartWriteDataFloat32(file, "py", py+32);
RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
status = H5PartWriteDataFloat32(file, "pz", pz+32);
RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
status = H5PartWriteDataInt32(file, LONGNAME, id+32);
RETURN(status, H5PART_SUCCESS, "H5PartWriteDataInt32");
}
}
......@@ -274,9 +315,6 @@ test_write_strided_data32(H5PartFile *file, int nparticles, int step)
data=(float*)malloc(6*nparticles*sizeof(float));
//status = H5PartSetNumParticlesStrided(file, nparticles, 6);
//RETURN(status, H5PART_SUCCESS, "H5PartSetNumParticlesStrided");
TEST("Writing 32-bit strided data");
for (t=step; t<step+NTIMESTEPS; t++)
......@@ -332,6 +370,8 @@ void test_write1(void)
test_write_data32(file1, NPARTICLES, 1);
test_write_file_attribs(file1, 0);
_H5Part_close_hdf_ids(file1);
test_open_objects(file1, 1);
status = H5PartCloseFile(file1);
RETURN(status, H5PART_SUCCESS, "H5PartCloseFile");
}
......@@ -372,7 +412,7 @@ void test_write3(void)
test_is_valid(file1);
TEST("Redefining step name");
status = H5PartDefineStepName(file1, LONGSTEPNAME, 16);
status = H5PartDefineStepName(file1, LONGNAME, 16);
RETURN(status, H5PART_SUCCESS, "H5PartDefineStepName");
test_write_strided_data64(file1, NPARTICLES, 0);
......@@ -401,10 +441,10 @@ void test_write4(void)
test_is_valid(file2);
TEST("Redefining step name");
status = H5PartDefineStepName(file1, LONGSTEPNAME, 16);
status = H5PartDefineStepName(file1, LONGNAME, 16);
RETURN(status, H5PART_SUCCESS, "H5PartDefineStepName");
status = H5PartDefineStepName(file2, LONGSTEPNAME, 16);
status = H5PartDefineStepName(file2, LONGNAME, 16);
RETURN(status, H5PART_SUCCESS, "H5PartDefineStepName");
test_write_data64(file1, NPARTICLES, NTIMESTEPS);
......
......@@ -3,7 +3,7 @@
OBJEXT=o
HDFLIB = -L@HDF5ROOT@/lib -lhdf5 -lz @SZLIB@
LIBS = -L@H5P_LIB_LOC@ -lH5Part $(HDFLIB) -lm @STDCXX@
LIBS = -L@H5P_LIB_LOC@ @H5P_LIB_NAME@ $(HDFLIB) -lm @STDCXX@
INCLUDES = -I@H5P_LIB_LOC@ -I@HDF5ROOT@/include @MPIINC@
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment