graph TD
A[封装请求接口] --> B{"选择算法"}
B -->|"DEFLATE"| C[执行DEFLATE压缩]
B -->|"LZ4"| D[执行LZ4压缩]
B -->|"ZSTD"| E[执行ZSTD压缩]
B -->|"BROTLI"| F[执行BROTLI压缩]
C --> H[数据流输出]
D --> H[数据流输出]
E --> H[数据流输出]
F --> H[数据流输出]
Optimized for highest compression ratios on each compressor, where possible.
Optimized for fastest decompression times on each decompressor, where possible.
Configurable, redistributable, self-contained, amalgamated and cross-platform (C++03).
Optional benchmark infrastructure (C++11).
ZLIB/LibPNG licensed.
Bundle stream format
[0x00 ...] Optional zero padding (N bits)
[0x70 0x??] Header (8 bits). De/compression algorithm (8 bits)
enum { RAW, SHOCO, LZ4F, MINIZ, LZIP, LZMA20, ZPAQ, LZ4, // 0..7
BROTLI9, ZSTD, LZMA25, BSC, BROTLI11, SHRINKER, CSC20, // 7..14
ZSTDF, BCM, ZLING, MCM, TANGELO, ZMOLLY, CRUSH, LZJB, // 15..22
BZIP2 // 23..
};
[vle_unpacked_size] Unpacked size of the stream (N bytes). Data is stored in a variable
length encoding value, where bytes are just shifted and added into a
big accumulator until MSB is found.
[vle_packed_size] Packed size of the stream (N bytes). Data is stored in a variable
length encoding value, where bytes are just shifted and added into a
big accumulator until MSB is found.
[bitstream] Compressed bitstream (N bytes). As returned by compressor.
If possible, header-less bitstreams are preferred.
Bundle .bun archive format
- Files/datas are packed into streams by using any compression method (see above)
- Streams are archived into a standard ZIP file:
- ZIP entry compression is (0) for packed streams and (1-9) for unpacked streams.
- ZIP entry comment is a serialized JSON of (file) meta-datas (@todo).
- Note: you can mix streams of different algorithms into the very same ZIP archive.
Showcase
#include <cassert>
#include "bundle.h"
int main() {
using namespace bundle;
using namespace std;
// 23 mb dataset
string original( "There's a lady who's sure all that glitters is gold" );
for (int i = 0; i < 18; ++i) original += original + string( i + 1, 32 + i );
// pack, unpack & verify all encoders
vector<unsigned> libs {
RAW, SHOCO, LZ4F, MINIZ, LZIP, LZMA20,
ZPAQ, LZ4, BROTLI9, ZSTD, LZMA25,
BSC, BROTLI11, SHRINKER, CSC20, BCM,
ZLING, MCM, TANGELO, ZMOLLY, CRUSH, LZJB
};
for( auto &lib : libs ) {
string packed = pack(lib, original);
string unpacked = unpack(packed);
cout << original.size() << " -> " << packed.size() << " bytes (" << name_of(lib) << ")" << endl;
assert( original == unpacked );
}
cout << "All ok." << endl;
}
On choosing compressors (on a regular basis)
Rank
Compression ratio
Fastest compressors
Fastest decompressors
Average speed
Memory efficiency
1st
91.15% ZPAQ
958.18MB/s RAW
2231.20MB/s RAW
1340.63MB/s RAW
tbd
2nd
90.71% MCM
358.41MB/s LZ4F
993.68MB/s LZ4
508.50MB/s LZ4F
tbd
3rd
90.02% TANGELO
240.87MB/s SHRINKER
874.83MB/s LZ4F
334.57MB/s SHRINKER
tbd
4th
88.31% BSC
223.28MB/s LZJB
547.62MB/s SHRINKER
267.57MB/s LZJB
tbd
5th
87.74% LZMA25
210.74MB/s ZSTDF
382.52MB/s MINIZ
246.66MB/s ZSTDF
tbd
6th
87.74% LZIP
159.59MB/s SHOCO
380.39MB/s ZSTD
209.32MB/s SHOCO
tbd
7th
87.63% BROTLI11
40.19MB/s ZLING
333.76MB/s LZJB
65.40MB/s ZLING
tbd
8th
87.50% CSC20
33.67MB/s CRUSH
304.06MB/s SHOCO
60.29MB/s CRUSH
tbd
9th
87.15% BCM
13.73MB/s ZSTD
297.34MB/s ZSTDF
26.51MB/s ZSTD
tbd
10th
86.44% ZMOLLY
09.00MB/s BSC
287.83MB/s CRUSH
13.44MB/s BZIP2
tbd
11th
86.17% LZMA20
08.51MB/s BZIP2
287.58MB/s BROTLI9
11.51MB/s BROTLI9
tbd
12th
86.05% BROTLI9
06.77MB/s ZMOLLY
246.88MB/s BROTLI11
10.78MB/s BSC
tbd
13th
85.27% BZIP2
05.87MB/s BROTLI9
175.54MB/s ZLING
08.13MB/s LZ4
tbd
14th
85.24% ZSTD
05.21MB/s BCM
118.49MB/s LZMA25
07.24MB/s MINIZ
tbd
15th
82.89% ZLING
04.08MB/s LZ4
108.71MB/s LZMA20
06.73MB/s ZMOLLY
tbd
16th
81.68% MINIZ
03.65MB/s MINIZ
72.72MB/s CSC20
05.27MB/s LZMA20
tbd
17th
77.93% ZSTDF
02.70MB/s LZMA20
57.05MB/s LZIP
04.90MB/s LZMA25
tbd
18th
77.57% LZ4
02.50MB/s LZMA25
31.88MB/s BZIP2
04.83MB/s CSC20
tbd
19th
77.37% CRUSH
02.50MB/s CSC20
13.44MB/s BSC
04.65MB/s BCM
tbd
20th
67.30% SHRINKER
02.25MB/s MCM
06.68MB/s ZMOLLY
04.13MB/s LZIP
tbd
21th
63.30% LZ4F
02.14MB/s LZIP
04.20MB/s BCM
02.29MB/s MCM
tbd
22th
59.37% LZJB
01.15MB/s TANGELO
02.34MB/s MCM
01.17MB/s TANGELO
tbd
23th
06.42% SHOCO
00.24MB/s BROTLI11
01.18MB/s TANGELO
00.48MB/s BROTLI11
tbd
24th
00.00% RAW
00.23MB/s ZPAQ
00.21MB/s ZPAQ
00.22MB/s ZPAQ
tbd
Note: SHOCO is a text compressor intended to be used for plain ascii IDs only.
Charts
@mavam has an awesome R script that plots some fancy graphics in his compbench repository. The following CC images are a few of his own showcasing an invocation for a 10,000 packet PCAP trace:
API - data
namespace bundle
{
// low level API (raw pointers)
bool is_packed( *ptr, len );
bool is_unpacked( *ptr, len );
unsigned type_of( *ptr, len );
size_t len( *ptr, len );
size_t zlen( *ptr, len );
const void *zptr( *ptr, len );
bool pack( unsigned Q, *in, len, *out, &zlen );
bool unpack( unsigned Q, *in, len, *out, &zlen );
// medium level API, templates (in-place)
bool is_packed( T );
bool is_unpacked( T );
unsigned type_of( T );
size_t len( T );
size_t zlen( T );
const void *zptr( T );
bool unpack( T &, T );
bool pack( unsigned Q, T &, T );
// high level API, templates (copy)
T pack( unsigned Q, T );
T unpack( T );
}
评论