SZ: Fast Error-Bounded Lossy HPC Data Compressor

Sheng Di, Dingwen Tao, Franck Cappello

Today’s HPC applications are producing extremely large amounts of data, thus it is necessary to use an efficient compression before storing them to parallel file systems.

We developed the error-bounded HPC data compressor, by proposing a novel HPC data compression method that works very effectively on compressing large-scale HPC data sets.

The key features of SZ are listed below. 

1. Compression: Input: a data set (or a floating-point array with any dimensions) ; Output: the compressed byte stream

    Decompression: input: the compressed byte stream ; Output: the original data set with the compression error of each data point being within a pre-specified error bound ∆.

2. SZ supports C, Fortran, and Java. 

3. SZ supports two types of error bounds. The users can set either absolute error bound or relative error bound, or a combination of the two bounds (with operator AND or OR).

4. Detailed usage and examples can be found under the directories doc/user-guide.pdf and example/ respectively, in the package. 

5. Version history: We recommend the latest version SZ 1.3

6. Download

Version SZ 1.3

-->>> Code download <<<--

(Contact: disheng222@gmail.com or sdi1@anl.gov)

If you download the code, please let us know who you are. We are very keen of helping you using the SZ library.

A paper describing SZ is to appear in IPDPS16, and its technical report is available to download.