SZ: Fast Error-Bounded Scientific Data Compressor

Keywords: floating point data compressor, lossy compressor, error bounded compression

Key developers: Sheng Di, Dingwen Tao, Xin Liang ; Other contributors: Ali M. Gok (Pastri version), Sihuan Li (Time-based compression for HACC simulation) ; Supervisor: Franck Cappello

Today’s HPC applications are producing extremely large amounts of data, thus it is necessary to use an efficient compression before storing them to parallel file systems.

We developed the error-bounded HPC data compressor, by proposing a novel HPC data compression method that works very effectively on compressing large-scale HPC data sets.

The key features of SZ are listed below. 

1. Usage: 

    Compression: Input: a data set (or a floating-point array with any dimensions); Output: the compressed byte stream

    Decompression: input: the compressed byte stream; Output: the original data set with the compression error of each data point being within a pre-specified error bound ∆.

2. Environment: SZ supports C, Fortran, and Java. It has been tested on Linux and Mac, with different architectures (x86, x64, ppc, etc.).

3. Error control: SZ supports many types of error bounds. The users can set absolute error bound, value-range based relative error bound, or a combination of the two bounds(with operator AND or OR). The users can also error bound mode to be PNSR-fixed, the point-wise relative error bound, etc. More details can be found in the configuration file (sz.config). 

4. SZ supports two compression modes (similar to Gzip): SZ_BEST_SPEED and SZ_BEST_COMPRESSION. SZ_BEST_SPEED results in the fastest compression. The best compression factor will be reached when using SZ_BEST_COMPRESSION and ZSTD_FAST_SPEED meanwhile. The default setting is SZ_BEST_COMPRESSION + Zstd.

5. User guide: More detailed usage and examples can be found under the directories doc/user-guide.pdf and example/ respectively, in the package. 

6. Citations: If you mention SZ in your paper, please cite the following references.

7. Download

Version SZ 2.0.2.0

-->>> Package Download (including everything) <<<--

-->>> Github of SZ <<<--

-->>> User Guide , hands-on-document <<<---

(Contact: disheng222@gmail.com or sdi1@anl.gov)

If you download the code, please let us know who you are. We are very keen of helping you using the SZ library.

8. Publications: 

9. Version history: We recommend the latest version.

10. Other versions are available upon request