Skip to content

Add generic filtering option #558

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
markcmiller86 wants to merge 1 commit into
base: 4.12RC
Choose a base branch
from
Open

Add generic filtering option #558

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 30 additions & 13 deletions docs/globals.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -823,37 +823,38 @@ However, a file's settings can be adjusted independently from the library's glob
Compression features are controlled by an arbitrary string, whose contents are described in more detail below.
By default, the Silo library does not have compression enabled.
A number of different compression techniques are available.
Some operate in a mesh and variable and data-type agnostic way while others depend on the type of data and sometimes even the type of mesh.
Some operate in a mesh-and-variable-and-data-type-agnostic way while others depend on the type of data and sometimes even the type of mesh.

Some compression settings are global to all compression methods.
There are two global parameters that control behavior of compression algorithms.
These must appear in the compression `options` string before any compression-specific parameters.
These must appear in the compression `options` string *before* any compression-specific parameters.

The first is the error mode (`"ERRMODE=<word>"` which controls how the Silo library responds when it encounters an error during compression and/or is unable to compress the data.
The two options for `<word>` are `FALLBACK` or `FAIL`. Including `"ERRMODE=FALLBACK"` in the compression options string tells Silo that whenever compression fails, it should simply fallback to writing uncompressed data.
The first is the error mode (`"ERRMODE=<word>"`) which controls how the Silo library responds when it encounters an error during compression and/or is unable to compress the data.
The two options for `<word>` are `FALLBACK` or `FAIL`.
Including `"ERRMODE=FALLBACK"` in the compression `options` string tells Silo that whenever compression fails, it should simply fallback to writing uncompressed data.
Including `"ERRMODE=FAIL"` in the compression `options` string tells Silo to fail the write and return `E_COMPRESSION` error for the operation.

The second is the minimum compression ratio to be achieved by compressing the data.
It is specified as `"MINRATIO=<float>"`.
For example, including `"MINRATIO=2.5"` in the compression options string tells Silo that all data must be compressed by at least a factor of 2.5:1.
For example, including `"MINRATIO=2.5"` in the compression `options` string tells Silo that all data must be compressed by at least a factor of 2.5:1.
If it is unable the compress by at least this amount, Silo will either fallback or fail the write depending on the `ERRMODE` setting.

The remaining paragraphs describe compression algorithm specific options.

GZIP compression
: is enabled using `"METHOD=GZIP"` in the options string.
: is enabled using `"METHOD=GZIP"` in the `options` string.

GZIP recognizes the `LEVEL=<int>`, compression parameter.
The compression level is an integer from 0 to 9, where 0 results in the fastest compression performance but at the expense of lower compression ratios.
Likewise, a level of 9 results in the slowest compression performance but with possibly better compression ratios.
If the `"LEVEL=<int>"` keyword does not appear in the options string or specifies invalid values, the default is level one (1).
If the `"LEVEL=<int>"` keyword does not appear in the `options` string or specifies invalid values, the default is level one (1).
The GZIP method of compression is applied independently to float and integer data for all types of meshes and variables.
It is also guaranteed to be available to all Silo clients.

When GZIP compression is applied to floating point data, a secondary [HDF5 shuffle](https://docs.hdfgroup.org/hdf5/v1_14/group___d_c_p_l.html#ga31e09cb0bf2da2893eed8a72220e6521) filter is also applied to improve compressibilty of the data.

SZIP compression:
: is enabled using `"METHOD=SZIP"` in the options string.
: is enabled using `"METHOD=SZIP"` in the `options` string.
The SZIP compression algorithm is designed specifically for scientific data.
SZIP recognizes the `BLOCK=<int>`, and `MASK={EC|NN}` parameters.
The `BLOCK=<int>`, takes an integer value from 0 to 32, which is a blocking size and must be even and not greater than 32, with typical values being 8, 10, 16, or 32.
Expand All @@ -870,7 +871,7 @@ However, a file's settings can be adjusted independently from the library's glob
Like GZIP, SZIP compression is applied to float and integer data independently of the types of meshes and variables.

FPZIP compression
: is enabled using `"METHOD=FPZIP"` in the options string.
: is enabled using `"METHOD=FPZIP"` in the `options` string.
The FPZIP compression algorithm was developed by [Peter Lindstrom](https://github.com/lindstro) at LLNL and is also designed for high speed compression of regular arrays of data.
FPZIP recognizes the `"LOSS=0|1|2|3"` parameter which specifies the amount of loss that is tolerable in the result in terms of quarters of full precision.
For example, `"LOSS=3"` indicates that a loss of 3/4 of full precision is tolerable (resulting in 8 bit floats or 16 bit doubles).
Expand All @@ -881,19 +882,22 @@ However, a file's settings can be adjusted independently from the library's glob
So, it is not always guaranteed to exist.

HZIP compression
: is enabled using `"METHOD=HZIP"` in the options string.
: is enabled using `"METHOD=HZIP"` in the `options` string.
The HZIP compression algorithm was developed by [Peter Lindstrom](https://github.com/lindstro) at LLNL and is designed for high-speed compression of unstructured meshes of quad or hex elements and node-centered variables (it does not yet support zone-centered variables) defined on a mesh.
Before applying this compression method to any given Silo mesh or variable object, the Silo library checks for compatibility with the constraints of the compression algorithm.
If the mesh or variable object is compatible, the object will be written with compression enabled.
Otherwise, compression will be silently ignored.
It is possible to build the Silo library without HZIP compression support.
So, it is not always guaranteed to exist.

Note that FPZIP and HZIP compression features are **not** available in a BSD Licensed release of Silo library.
They are available only in a Legacy licensed release of the Silo library.
Note that FPZIP and HZIP compression features are **not** available in a BSD Licensed installation of Silo library.
They are available only in a Legacy licensed installation of the Silo library.
Also note that the FPZIP and HZIP compression libraries built into Silo here are unfortunately not name-mangled to avoid possible name collision with any versions of those libraries being independently used as standalone libraries.
For applications using FPZIP and HZIP libraries independently of Silo, the Silo library will have to have been configured and installed **without** FPZIP and HZIP.
Finally, note that the FPZIP and HZIP compression filters built into Silo here are named `"silo-fpzip"` and `"silo-hzip"` as far as HDF5 is concerned to avoid confusion with any versions of those filters possibly being independently used as standalone HDF5 plugins.

ZFP compression
: is enabled using `"METHOD=ZFP"` in the options string.
: is enabled using `"METHOD=ZFP"` in the `options` string.
ZFP compression in the Silo library uses a built-in version of the [H5Z-ZFP](https://h5z-zfp.readthedocs.io/en/latest/) compression filter.
Data compressed with Silo using ZFP is fully compatible with any reader using the H5Z-ZFP compression filter.
Use `"RATE=<float>"` to set compression mode to use ZFP's rate-based compression.
Expand All @@ -904,6 +908,19 @@ However, a file's settings can be adjusted independently from the library's glob
Note that all other ZFP related parameters having to do with data type and array dimensions are handled by Silo automatically during each
`DBPutXxx()` call.

The [H5Z-ZFP]() compression filter and ZFP library is built into Silo and uses the exact same source code in Silo as the standalone version.
However, that filter's source code and ZFP library is name-mangled when built into Silo to avoid any collision with any versions of H5Z-ZFP and/or ZFP independently available as standalone shared libraries.

HDF5 Plugin compression
: is enabled using `"METHOD=HDF5_PLUGIN"` in the `options` string.
One of either the `"ID=..."` or `"NAME=..."` options below is required.
Use `"ID=<filter-id>"` to set the specific HDF5 filter to be used.
Use `"NAME=<filter-name>"` to set the specific HDF5 filter to be used.
Use `"CDVALS=<uint0>,<uint1>,...,<uintK>"` to specify filter parameters.
Optionally, use `"PATH=<path-to-plugin-dir>"` to specify a path where the plugin shared library can be found.

This last option allows Silo data producers to use any HDF5 compression filter plugin available but not necessarily built into the Silo library.

{{ EndFunc }}

## `DBGetCompression()`
Expand Down
16 changes: 12 additions & 4 deletions src/hdf5_drv/H5Zzfp_plugin.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,14 @@
#define H5Z_FILTER_ZFP_VERSION_MINOR 0
#define H5Z_FILTER_ZFP_VERSION_PATCH 1

/*
HDF5 generic cd_vals[] memory layout (6 unsigned ints) for
controlling H5Z-ZFP behavior as a plugin. NOTE: These cd_vals
used to pass properties in-memory from caller to filter via HDF5
generic interface are NOT THE SAME AS the cd_vals[] that
ultimately get stored to the file for the filter "header" data.
*/

#define H5Z_ZFP_MODE_RATE 1
#define H5Z_ZFP_MODE_PRECISION 2
#define H5Z_ZFP_MODE_ACCURACY 3
Expand Down Expand Up @@ -67,10 +75,10 @@ do { \
if ((N>=6)&&(CD[0] == H5Z_ZFP_MODE_EXPERT)) \
{ \
unsigned int *p; int *q; \
p = MiB; *p = CD[2]; \
p = MaB; *p = CD[3]; \
p = MaP; *p = CD[4]; \
q = MiE; *q = (int) CD[5]; \
p = &MiB; *p = CD[2]; \
p = &MaB; *p = CD[3]; \
p = &MaP; *p = CD[4]; \
q = &MiE; *q = (int) CD[5]; \
} \
} while(0)

Expand Down
143 changes: 115 additions & 28 deletions src/hdf5_drv/silo_hdf5.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3333,6 +3333,24 @@ hdf2hdf_type(hid_t ftype)
return mtype;
}

static int
_filter_present(hid_t dcpl, H5Z_filter_t filter_id)
{
int nfilters = H5Pget_nfilters(dcpl);

for (int i = 0; i < nfilters; i++)
{
#if defined H5_USE_16_API || !HDF5_VERSION_GE(1,8,0)
if (H5Pget_filter(dcpl,(unsigned)i,0,0,0,0,0) == filter_id)
#else
if (H5Pget_filter(dcpl,(unsigned)i,0,0,0,0,0,NULL) == filter_id)
#endif
return 1;
}

return 0;
}

/*-------------------------------------------------------------------------
* Function: db_hdf5_set_compression
*
Expand Down Expand Up @@ -3363,42 +3381,26 @@ db_hdf5_set_compression(DBfile *dbfile, int flags)
char *check;
int level, block, nfilters;
int nbits, prec;
int have_gzip, have_szip, have_fpzip, have_hzip, have_zfp, i;
int have_gzip = FALSE;
int have_szip = FALSE;
int have_fpzip = FALSE;
int have_hzip = FALSE;
int have_zfp = FALSE;
int i;
H5Z_filter_t filtn;
unsigned int filter_config_flags, opt_flag;

/* Check what filters already exist */
have_gzip = FALSE;
have_szip = FALSE;
have_fpzip = FALSE;
have_hzip = FALSE;
have_zfp = FALSE;
if ((nfilters = H5Pget_nfilters(P_ckcrprops))<0)
{
db_perror("H5Pget_nfilters", E_CALLFAIL, me);
return (-1);
}
for (i=0; i<nfilters; i++) {
#if defined H5_USE_16_API || !HDF5_VERSION_GE(1,8,0)
filtn = H5Pget_filter(P_ckcrprops,(unsigned)i,0,0,0,0,0);
#else
filtn = H5Pget_filter(P_ckcrprops,(unsigned)i,0,0,0,0,0,NULL);
#endif
if (H5Z_FILTER_DEFLATE==filtn)
have_gzip = TRUE;
have_gzip = _filter_present(P_ckcrprops, H5Z_FILTER_DEFLATE);
#ifdef H5_HAVE_FILTER_SZIP
if (H5Z_FILTER_SZIP==filtn)
have_szip = TRUE;
have_szip = _filter_present(P_ckcrprops, H5Z_FILTER_SZIP);
#endif
if (DB_HDF5_FPZIP_ID==filtn)
have_fpzip = TRUE;
if (DB_HDF5_HZIP_ID==filtn)
have_hzip = TRUE;
have_fpzip = _filter_present(P_ckcrprops, DB_HDF5_FPZIP_ID);
have_hzip = _filter_present(P_ckcrprops, DB_HDF5_HZIP_ID);
#ifdef HAVE_ZFP
if (H5Z_FILTER_ZFP==filtn)
have_zfp = TRUE;
have_zfp = _filter_present(P_ckcrprops, H5Z_FILTER_ZFP);
#endif
}

#ifndef _MSC_VER
#warning WHAT ABOUT NULL RETURN FROM DBGETCOMPRESSION
#endif
Expand All @@ -3423,6 +3425,7 @@ db_hdf5_set_compression(DBfile *dbfile, int flags)
{
float mcr;
(void)strncpy(chararray, ptr+9, 5);
chararray[5] = '\0';
mcr = (float) strtod(chararray, &check);
if (mcr > 1.0)
SILO_Globals.compressionMinratio = mcr;
Expand All @@ -3441,6 +3444,7 @@ db_hdf5_set_compression(DBfile *dbfile, int flags)
{
unsigned int minsize;
strncpy(chararray, ptr+8, 8);
chararray[8] = '\0';
minsize = strtol(chararray, &check, 10);
if (minsize > 0)
SILO_Globals.compressionMinsize = minsize;
Expand Down Expand Up @@ -3468,6 +3472,7 @@ db_hdf5_set_compression(DBfile *dbfile, int flags)
"LEVEL=")) != (char *)NULL)
{
(void)strncpy(chararray, ptr+6, 1);
chararray[1] = '\0';
level = (int) strtol(chararray, &check, 10);
if ((chararray != check) && (level >= 0) && (level <=9))
{
Expand Down Expand Up @@ -3515,6 +3520,7 @@ db_hdf5_set_compression(DBfile *dbfile, int flags)
"BLOCK=")) != (char *)NULL)
{
(void)strncpy(chararray, ptr+6, 2);
chararray[2] = '\0';
block = (int) strtol(chararray, &check, 10);
if ((chararray != check) && (block >= 0) && (block <=32))
{
Expand Down Expand Up @@ -3601,6 +3607,7 @@ db_hdf5_set_compression(DBfile *dbfile, int flags)
"BITS=")) != (char *)NULL)
{
(void)strncpy(chararray, ptr+5, 2);
chararray[2] = '\0';
nbits = (int) strtol(chararray, &check, 10);
if ((chararray != check) && (nbits >= 0) && (nbits <=64))
{
Expand Down Expand Up @@ -3637,6 +3644,7 @@ db_hdf5_set_compression(DBfile *dbfile, int flags)
"LOSS=")) != (char *)NULL)
{
(void)strncpy(chararray, ptr+5, 2);
chararray[2] = '\0';
prec = (int) strtol(chararray, &check, 10);
if ((chararray != check) && (prec >= 0) && (prec <=3))
{
Expand Down Expand Up @@ -3672,6 +3680,8 @@ db_hdf5_set_compression(DBfile *dbfile, int flags)
"RATE=")) != (char *)NULL)
{
strncpy(chararray, ptr+5, 8);
chararray[8] = '\0';
errno = 0;
tmpdbl = strtod(chararray, &check);
if (chararray != check && errno == 0 && tmpdbl > 0)
H5Pset_zfp_rate_cdata(tmpdbl, cd_nelmts, cd_values);
Expand All @@ -3680,6 +3690,8 @@ db_hdf5_set_compression(DBfile *dbfile, int flags)
"PRECISION=")) != (char *)NULL)
{
strncpy(chararray, ptr+10, 2);
chararray[2] = '\0';
errno = 0;
tmpuint = (uint) strtoul(chararray, &check, 10);
if (chararray != check && errno == 0 && tmpuint > 0)
H5Pset_zfp_precision_cdata(tmpuint, cd_nelmts, cd_values);
Expand All @@ -3688,6 +3700,8 @@ db_hdf5_set_compression(DBfile *dbfile, int flags)
"ACCURACY=")) != (char *)NULL)
{
strncpy(chararray, ptr+9, 8);
chararray[8] = '\0';
errno = 0;
tmpdbl = strtod(chararray, &check);
if (chararray != check && errno == 0 && tmpdbl > 0)
H5Pset_zfp_accuracy_cdata(tmpdbl, cd_nelmts, cd_values);
Expand All @@ -3697,6 +3711,8 @@ db_hdf5_set_compression(DBfile *dbfile, int flags)
{
int nvals, minexp; unsigned int minbits, maxbits, maxprec;
strncpy(chararray, ptr+7, 20);
chararray[20] = '\0';
errno = 0;
nvals = sscanf(chararray, "%u,%u,%u,%d", &minbits, &maxbits, &maxprec, &minexp);
if (nvals == 4 && errno == 0)
H5Pset_zfp_expert_cdata(minbits, maxbits, maxprec, minexp, cd_nelmts, cd_values);
Expand All @@ -3720,6 +3736,77 @@ db_hdf5_set_compression(DBfile *dbfile, int flags)
}
}
#endif
else if ((ptr=(char *)strstr(DBGetCompressionFile(dbfile),
"METHOD=HDF5_PLUGIN")) != (char *)NULL)
{
uint tmpuint = 0;
H5Z_filter_t filtid = 0;
char filtname[64] = "";
unsigned int cdvals[100];
int const cdmaxvals = sizeof(cdvals)/sizeof(cdvals[0]);
int cdnvals = 0;

if ((ptr=(char *)strstr(DBGetCompressionFile(dbfile),
"ID=")) != (char *)NULL)
{
strncpy(chararray, ptr+3, 5);
chararray[5] = '\0';
errno = 0;
filtid = (H5Z_filter_t) strtol(chararray, &check, 10);
if (chararray == check || errno != 0 || filtid < 0 || filtid > 64000)
{
db_perror(DBGetCompressionFile(dbfile), E_COMPRESSION, me);
return (-1);
}
}

if (!_filter_present(P_ckcrprops, filtid))
{
if ((ptr=(char *)strstr(DBGetCompressionFile(dbfile),
"NAME=")) != (char *)NULL)
{
int len = (int) strcspn(ptr+5, " ")+1;
if (len >= (int) sizeof(filtname))
{
db_perror(DBGetCompressionFile(dbfile), E_COMPRESSION, me);
return (-1);
}
strncpy(filtname, ptr+5, len);
}

if ((ptr=(char *)strstr(DBGetCompressionFile(dbfile),
"CDVALS=")) != (char *)NULL)
{
int i = 0;
errno = 0;
for (char *tok = strtok(ptr+7, ",");
(tok != NULL) && (i < cdmaxvals) && (errno==0);
tok = strtok(NULL, ","), i++)
cdvals[i] = (uint) strtoul(tok, NULL, 10);
cdnvals = errno == 0 ? i : -1;
}

if ((ptr=(char *)strstr(DBGetCompressionFile(dbfile),
"PATH=")) != (char *)NULL)
{
char filtpath[1024] = "";
int len = (int) strcspn(ptr+5, " ")+1;
if (len >= (int) sizeof(filtpath))
{
db_perror(DBGetCompressionFile(dbfile), E_COMPRESSION, me);
return (-1);
}
strncpy(filtpath, ptr+5, len);
H5PLprepend(filtpath);
}

if (H5Pset_filter(P_ckcrprops, filtid, opt_flag, cdnvals, cdvals)<0)
{
db_perror("H5Pset_filter", E_CALLFAIL, me);
return (-1);
}
}
}
else
{
db_perror(DBGetCompressionFile(dbfile), E_COMPRESSION, me);
Expand Down
3 changes: 2 additions & 1 deletion tests/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -533,10 +533,11 @@ if(SILO_ENABLE_HDF5 AND HDF5_FOUND)
LABELS "hdf5")
set(COMPRESSION_TESTS "compression-hdf5")
if(ZLIB_FOUND)
add_test(NAME compression-generic COMMAND $<TARGET_FILE:compression> generic)
add_test(NAME compression-gzip COMMAND $<TARGET_FILE:compression> gzip)
add_test(NAME compression-gzip-read COMMAND $<TARGET_FILE:compression> readonly)
set_tests_properties(compression-gzip-read PROPERTIES DEPENDS "compression-gzip")
list(APPEND COMPRESSION_TESTS compression-gzip compression-gzip-read)
list(APPEND COMPRESSION_TESTS compression-gzip compression-gzip-read compression-generic)
endif()
if(HDF5_ENABLE_SZIP_SUPPORT AND SZIP_FOUND)
add_test(NAME compression-szip COMMAND $<TARGET_FILE:compression> szip)
Expand Down
Loading
Loading