updated NEWS with details about 1.6 changes; changed hsize_t type in...

updated NEWS with details about 1.6 changes; changed hsize_t type in H5PartSetViewIndices; fixes to the build system; fixed bugs that were preventing the use of H5PartSetView during writes; added open object checks to regression test

NEWS

+#### H5PART 1.6 ##############################################################
+
+Updated Documentation
+---------------------
+
+The Fortran API has been added to the doxygen documentation. Most entries have
+a reference to their respective C API call.
+
+Regression Test Suite
+---------------------
+
+The test subdirectory has been reorganized to include a systematic series
+of regression tests of common use cases for the API. The goal is to have both
+complete coverage of the API (every call is exercised) and testing of some
+unlikely or unsuspected values and cases.
+
+Currently, the C test covers the entire H5Part API and the Fortran test is
+still in development.
+
+Benchmark Utilities
+-------------------
+
+Two benchmarking utilities, H5PartBench and H5BlockBench, have been added to
+the tools collection. Both require the parallel library and have functionality
+and syntax similar to the IOR benchmark:
+
+  http://sourceforge.net/projects/ior-sio/
+
+File Mode Flags
+---------------
+
+Previously, the only flags used were H5PART_READ, H5PART_WRITE, and
+H5PART_APPEND, and these were mutually exclusive. Additional flags have been
+added to select the underlying virtual file driver (VFD) that HDF5 uses in
+parallel mode and to accommodate the lustre filesystem:
+
+H5PART_VFD_MPIPOSIX (use the MPI-POSIX driver, which bypasses MPI-IO)
+H5PART_VFD_MPIIO_IND (use MPI-IO in independent mode)
+H5PART_FS_LUSTRE (activate H5P tunings for the lustre stripe size)
+
+These flags can be specified using the typical bitwise OR method, e.g.
+
+  char flag = H5PART_WRITE | H5PART_VFD_MPIPOSIX | H5PART_FS_LUSTRE;
+
+will select write mode with these two additional options.
+
+The parallel library now defaults to using the MPI-IO collective mode driver
+(previously it used independent mode). The collective buffering algorithm in
+the Cray XT4/5 environment has been substantially improved in the last year,
+and defaulting to collective mode allows us to take advantage of these
+improvements automatically.
+
+There are also new open calls H5PartOpenAlign and H5PartOpenParallelAlign that
+take an additional 'alignment' value. This value is passed to HDF5 and used to
+pad out objects so that they align to filesystem boundaries. For instance, it
+would make sense to use the stripe size as the alignment value when writing to
+a lustre filesystem.
+
+Unified 'View' Model
+--------------------
+
+Views are now supported in both read and write mode (previously only supported
+in read mode). It is possible to specify non-sensical views: you could for
+instance specify a read view that is larger than the dataset on disk. Or you
+could create a write view that is larger than the data in memory. In those
+cases, you will encounter an error when you try to perform a read or write
+operation on a dataset.
+
+There are two new methods for selecting views. H5PartSetNumParticlesStrided
+selects a view with $n$ particles per processor, but such that the data in
+memory is expected to have a 'stride' factor. For instance, if you have
+particle data with fields $x$ and $y$ and a single array with entries
+$x1,y1,x2,y2...$, then the stride factor is 2. Striding works for both reads
+and writes and only affects the view of memory: individual fields are still
+stored as individual arrays on disk.
+
+The second new method is H5PartSetViewIndices, which allows for point
+selections of datasets. You can pass a list of indices for the points you
+want to select. This is useful, for instance, when interfacing with FastBit
+to perform queries that select a small subset of non-contiguous particles.
+
+Because views are now supported on write, it is possible to write a dataset
+using multiple 'passes' or to leave some values unwritten.
+
+Internal Handling of Dataspaces
+-------------------------------
+
+H5PartSetNumParticles and H5PartSetNumParticlesStrided now share the same
+HDF5 dataspace state in the H5PartFile struct with the calls that modify
+the view (H5PartSetView etc.). Previously, the memory and disk dataspaces
+were regenerated from the view on every read access, which was unnecessary.
+
+Autogeneration of API Calls
+---------------------------
+
+Python scripts have been added to automatically generate read/write call
+variants for different data types.
+
+Renamed Fortran Include
+-----------------------
+
+The automake system no longer greps the *F90.inc files to generate an include
+called 'H5Part.inc'. Instead, the *F90.inc files have been renamed to *.f90,
+and awk is used to generate an 'H5PartF.h' file (the same naming convention
+as in MPI's 'mpif.h').
+
+The change to *.f90 was necessary to facilitate integration of the Fortran
+API into doxygen. Note that the *.f90 files will not compile, nor can they
+be included directly. Simply use
+
+  include 'H5PartF.h'
+
+in your Fortran code to include all the definitions for Fortran H5Part/H5Block
+calls.
+
+64 Char Limit on Dataset Names
+------------------------------
+
+Previously, a user could overrun internal buffers for dataset names. Now, a
+fixed limit of 64 chars is imposed. Dataset names that are longer than this are
+truncated and a warning is printed. We expect that most users are using
+short canoncical names like x, px, id, etc.
+
+
+#### H5PART 1.4 ##############################################################
+
 Error Handling
 --------------
  Now all functions, in which an error could occure, are returning a

configure.ac

@@ -390,6 +390,7 @@ if test "X$USE_PARALLEL" = "Xyes"; then
         fi
     fi
 
+    H5P_LIB_NAME="-lpH5Part"
     MTARGET="libpH5Part.a"
     TTARGET="test"
 
@@ -417,6 +418,7 @@ if test "X$USE_PARALLEL" = "Xyes"; then
 	    FC=${MPIFC}
 	fi
  
+        H5P_LIB_NAME="-lpH5PartF"
         MTARGET="${MTARGET} libpH5PartF.a"
         TTARGET="${TTARGET} testf"
     fi
@@ -424,10 +426,12 @@ if test "X$USE_PARALLEL" = "Xyes"; then
 else  # --enable-parallel=no
     AC_MSG_RESULT([no])
 
+    H5P_LIB_NAME="-lH5Part"
     MTARGET="libH5Part.a"
     TTARGET="test"
 
     if test "X$USE_FORTRAN" = "Xyes"; then
+        H5P_LIB_NAME="-lH5PartF"
         MTARGET="${MTARGET} libH5PartF.a"
         TTARGET="${TTARGET} testf"
     fi
@@ -596,6 +600,7 @@ AC_SUBST(CFLAGS)
 AC_SUBST(FFLAGS)
 AC_SUBST(STDCXX)
 AC_SUBST(H5P_LIB_LOC)
+AC_SUBST(H5P_LIB_NAME)
 AC_SUBST(UNDERSCORE_H)
 AC_SUBST(BUILD_TOOLS)
 AC_SUBST(LDFLAGS)

src/H5Part.h

@@ -205,7 +205,7 @@ H5PartSetView (
 h5part_int64_t
 H5PartSetViewIndices (
 	H5PartFile *f,			/*!< [in]  Handle to open file */
-	const hsize_t *indices,		/*!< [in]  List of indices */
+	const h5part_int64_t *indices,	/*!< [in]  List of indices */
 	h5part_int64_t nelems		/*!< [in]  Size of list */
 	);
 

src/H5PartTypes.h

@@ -24,6 +24,7 @@ typedef unsigned long		MPI_Comm;
 #endif
 
 #define H5PART_STEPNAME_LEN	64
+#define H5PART_DATANAME_LEN	64
 
 struct H5BlockFile;
 

src/Makefile.am

@@ -4,6 +4,11 @@ OBJEXT=o
 
 INCLUDES = -I@HDF5ROOT@/include @MPIINC@
 
+EXTRA_HEADERS = 	H5PartPrivate.h \
+			H5BlockPrivate.h \
+			H5MultiBlockPrivate.h \
+			H5PartAttrib.h
+
 # Extra files that I wish to include in the dist tar ball.
 EXTRA_DIST =		TestUnderscoreC.c \
 			TestUnderscore.f \
@@ -13,7 +18,8 @@ EXTRA_DIST =		TestUnderscoreC.c \
 			H5Part.f90 \
 			H5PartAttrib.f90 \
 			H5Block.f90 \
-			H5BlockReadWrite.f90
+			H5BlockReadWrite.f90 \
+			$(EXTRA_HEADERS)
 
 # Files that I don't want to include in the dist tar ball
 nodist_include_HEADERS = H5PartF.h @UNDERSCORE_H@
@@ -40,12 +46,6 @@ include_HEADERS = 	H5Part.h \
 			H5PartF.h \
 			@UNDERSCORE_H@
 
-# Listing of all possible headers that I may include
-EXTRA_HEADERS =		H5PartPrivate.h \
-			H5BlockPrivate.h \
-			H5MultiBlockPrivate.h \
-			H5PartAttrib.h
-
 # Listing of sources
 libH5Part_a_SOURCES =	H5Part.c \
 			H5PartAttrib.c \
@@ -54,7 +54,8 @@ libH5Part_a_SOURCES =	H5Part.c \
 			H5MultiBlock.c \
 			H5MultiBlockReadWrite.c
 
-libH5PartF_a_SOURCES =	H5PartF.c \
+libH5PartF_a_SOURCES =	$(libH5Part_a_SOURCES) \
+			H5PartF.c \
 			H5PartAttribF.c \
 			H5BlockF.c \
 			H5BlockReadWriteF.c

test/Makefile.am

@@ -3,7 +3,7 @@
 OBJEXT=o
 
 HDFLIB = -L@HDF5ROOT@/lib -lhdf5 -lz @SZLIB@
-LIBS = -L@H5P_LIB_LOC@ -lH5PartF -lH5Part $(HDFLIB) -lm
+LIBS = -L@H5P_LIB_LOC@ @H5P_LIB_NAME@ $(HDFLIB) -lm
 
 INCLUDES = -I@H5P_LIB_LOC@ -I@HDF5ROOT@/include @MPIINC@
 
@@ -18,7 +18,8 @@ test_SOURCES =	test.c \
 		testframe.c \
 		write.c \
 		read.c \
-		testframe.h
+		testframe.h \
+		params.h
 
 testf_SOURCES = testf.F90
 

test/params.h

@@ -2,7 +2,7 @@
 #define _H5PART_TEST_PARAMS_H_
 
 #define FILENAME "test.h5"
-#define LONGSTEPNAME "thisisaverylongstepnamethatshouldexceedthelimitof64charcausingawarningtoprint"
+#define LONGNAME "thisisaverylongnamethatshouldexceedthelimitof64charcausingawarningtoprint"
 #define NTIMESTEPS 10
 
 /* do not decrease this value below 99, or it will break assumptions

test/test.c

@@ -61,10 +61,10 @@ int main(int argc, char **argv)
 	//if (GetTestCleanup() && !getenv("HDF5_NOCLEANUP"))
         //    TestCleanup();
 
-	//return GetTestNumErrs();
-
 #ifdef PARALLEL_IO
 	return MPI_Finalize();
+#else
+	return GetTestNumErrs();
 #endif
 }
 

test/testframe.c

@@ -649,3 +649,38 @@ test_is_valid(H5PartFile *file)
 	RETURN(status, H5PART_SUCCESS, "H5PartFileIsValid");
 }
 
+void
+test_open_objects(H5PartFile *file, int max_objects)
+{
+	ssize_t nopen = H5Fget_obj_count(file->file, H5F_OBJ_ALL);
+	if (nopen > max_objects)
+	{
+		TestErrPrintf(	"*** TOO MANY OBJECTS OPEN: %d > %d "
+				"at line %4d in %s\n", nopen, max_objects,
+				(int)__LINE__, __FILE__ );
+
+		hid_t *list = malloc(sizeof(hid_t)*nopen);
+		H5Fget_obj_ids(file->file, H5F_OBJ_ALL, nopen, list);
+
+                H5O_info_t info;
+		int i;
+		for (i=0; i<nopen; i++) {
+			H5Oget_info(list[i], &info);
+                        switch (info.type) {
+                        case H5O_TYPE_GROUP: 
+			TestErrPrintf("obj%d has type GROUP\n", i);
+                        break;
+                        case H5O_TYPE_DATASET: 
+			TestErrPrintf("obj%d has type DATASET\n", i);
+                        break;
+                        case H5O_TYPE_NAMED_DATATYPE: 
+			TestErrPrintf("obj%d has type NAMED_DATATYPE\n", i);
+                        break;
+                        default:
+			TestErrPrintf("obj%d has unknown type\n", i);
+                        }
+		}
+		free(list);
+	}
+}
+

test/testframe.h

@@ -184,5 +184,8 @@ get_attr_name(char *name, char *tag, int id);
 void
 test_is_valid(H5PartFile *file);
 
+void
+test_open_objects(H5PartFile *file, int max_objects);
+
 #endif
 

test/write.c

@@ -83,6 +83,10 @@ test_write_data64(H5PartFile *file, int nparticles, int step)
 	status = H5PartSetNumParticlesStrided(file, nparticles, -1);
 	RETURN(status, H5PART_SUCCESS, "H5PartSetNumParticlesStrided");
 
+	/* invalid nparticles will produce an error */
+	status = H5PartSetNumParticlesStrided(file, -1, 2);
+	RETURN(status, H5PART_ERR_INVAL, "H5PartSetNumParticlesStrided");
+
 #if PARALLEL_IO
 	TEST("Setting throttle");
 	status = H5PartSetThrottle(file, 2);
@@ -195,6 +199,7 @@ test_write_data32(H5PartFile *file, int nparticles, int step)
 {
 	int i,t;
 	h5part_int32_t status;
+	int rank, nprocs;
 
 	float *x,*y,*z;
 	float *px,*py,*pz;
@@ -216,6 +221,12 @@ test_write_data32(H5PartFile *file, int nparticles, int step)
 	TEST("Setting throttle");
 	status = H5PartSetThrottle(file, 2);
 	RETURN(status, H5PART_SUCCESS, "H5PartSetThrottle");
+
+	MPI_Comm_rank(MPI_COMM_WORLD, &rank);
+	MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
+#else
+	rank = 0;
+	nprocs = 1;
 #endif
 
 	TEST("Writing 32-bit data");
@@ -236,8 +247,11 @@ test_write_data32(H5PartFile *file, int nparticles, int step)
 		status = H5PartSetStep(file, t);
 		RETURN(status, H5PART_SUCCESS, "H5PartSetStep");
 
-		status = H5PartSetNumParticles(file, nparticles);
-		RETURN(status, H5PART_SUCCESS, "H5PartSetNumParticles");
+		/* test a two-part write using views */
+		status = H5PartSetView(file,
+			rank*nparticles,
+			rank*nparticles + 31);
+		RETURN(status, H5PART_SUCCESS, "H5PartSetView");
 
 		status = H5PartWriteDataFloat32(file, "x", x);
 		RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
@@ -259,7 +273,34 @@ test_write_data32(H5PartFile *file, int nparticles, int step)
 		status = H5PartWriteDataFloat32(file, "pz", pz);
 		RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
 		
-		status = H5PartWriteDataInt32(file, "id", id);
+		status = H5PartWriteDataInt32(file, LONGNAME, id);
+		RETURN(status, H5PART_SUCCESS, "H5PartWriteDataInt32");
+
+		/* the second write phase... */
+		status = H5PartSetView(file,
+			rank*nparticles + 32,
+			rank*nparticles + nparticles - 1);
+		RETURN(status, H5PART_SUCCESS, "H5PartSetView");
+		/* offset the input arrays */
+		status = H5PartWriteDataFloat32(file, "x", x+32);
+		RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
+
+		status = H5PartWriteDataFloat32(file, "y", y+32);
+		RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
+
+		status = H5PartWriteDataFloat32(file, "z", z+32);
+		RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
+		
+		status = H5PartWriteDataFloat32(file, "px", px+32);
+		RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
+
+		status = H5PartWriteDataFloat32(file, "py", py+32);
+		RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
+
+		status = H5PartWriteDataFloat32(file, "pz", pz+32);
+		RETURN(status, H5PART_SUCCESS, "H5PartWriteDataFloat32");
+		
+		status = H5PartWriteDataInt32(file, LONGNAME, id+32);
 		RETURN(status, H5PART_SUCCESS, "H5PartWriteDataInt32");
 	}
 }
@@ -274,9 +315,6 @@ test_write_strided_data32(H5PartFile *file, int nparticles, int step)
 
 	data=(float*)malloc(6*nparticles*sizeof(float));
 
-	//status = H5PartSetNumParticlesStrided(file, nparticles, 6);
-	//RETURN(status, H5PART_SUCCESS, "H5PartSetNumParticlesStrided");
-
 	TEST("Writing 32-bit strided data");
 
 	for (t=step; t<step+NTIMESTEPS; t++)
@@ -332,6 +370,8 @@ void test_write1(void)
 	test_write_data32(file1, NPARTICLES, 1);
 	test_write_file_attribs(file1, 0);
 
+	_H5Part_close_hdf_ids(file1);
+	test_open_objects(file1, 1);
 	status = H5PartCloseFile(file1);
 	RETURN(status, H5PART_SUCCESS, "H5PartCloseFile");
 }
@@ -372,7 +412,7 @@ void test_write3(void)
 	test_is_valid(file1);
 
 	TEST("Redefining step name");
-	status = H5PartDefineStepName(file1, LONGSTEPNAME, 16);
+	status = H5PartDefineStepName(file1, LONGNAME, 16);
 	RETURN(status, H5PART_SUCCESS, "H5PartDefineStepName");
 
 	test_write_strided_data64(file1, NPARTICLES, 0);
@@ -401,10 +441,10 @@ void test_write4(void)
 	test_is_valid(file2);
 
 	TEST("Redefining step name");
-	status = H5PartDefineStepName(file1, LONGSTEPNAME, 16);
+	status = H5PartDefineStepName(file1, LONGNAME, 16);
 	RETURN(status, H5PART_SUCCESS, "H5PartDefineStepName");
 
-	status = H5PartDefineStepName(file2, LONGSTEPNAME, 16);
+	status = H5PartDefineStepName(file2, LONGNAME, 16);
 	RETURN(status, H5PART_SUCCESS, "H5PartDefineStepName");
 
 	test_write_data64(file1, NPARTICLES, NTIMESTEPS);

tools/Makefile.am

@@ -3,7 +3,7 @@
 OBJEXT=o
 
 HDFLIB = -L@HDF5ROOT@/lib -lhdf5 -lz @SZLIB@
-LIBS = -L@H5P_LIB_LOC@ -lH5Part $(HDFLIB) -lm @STDCXX@
+LIBS = -L@H5P_LIB_LOC@ @H5P_LIB_NAME@ $(HDFLIB) -lm @STDCXX@
 
 INCLUDES = -I@H5P_LIB_LOC@ -I@HDF5ROOT@/include @MPIINC@