extlz4 0.2.4.2 → 0.3.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +5 -5
- data/HISTORY.ja.md +25 -0
- data/README.md +49 -41
- data/bin/extlz4 +1 -1
- data/contrib/lz4/INSTALL +1 -0
- data/contrib/lz4/Makefile.inc +87 -0
- data/contrib/lz4/NEWS +89 -0
- data/contrib/lz4/README.md +42 -36
- data/contrib/lz4/build/README.md +55 -0
- data/contrib/lz4/build/VS2010/datagen/datagen.vcxproj +169 -0
- data/contrib/lz4/build/VS2010/frametest/frametest.vcxproj +176 -0
- data/contrib/lz4/build/VS2010/fullbench-dll/fullbench-dll.vcxproj +180 -0
- data/contrib/lz4/build/VS2010/fullbench/fullbench.vcxproj +176 -0
- data/contrib/lz4/build/VS2010/fuzzer/fuzzer.vcxproj +173 -0
- data/contrib/lz4/build/VS2010/liblz4-dll/liblz4-dll.rc +51 -0
- data/contrib/lz4/build/VS2010/liblz4-dll/liblz4-dll.vcxproj +179 -0
- data/contrib/lz4/build/VS2010/liblz4/liblz4.vcxproj +175 -0
- data/contrib/lz4/build/VS2010/lz4.sln +98 -0
- data/contrib/lz4/build/VS2010/lz4/lz4.rc +51 -0
- data/contrib/lz4/build/VS2010/lz4/lz4.vcxproj +189 -0
- data/contrib/lz4/build/VS2017/datagen/datagen.vcxproj +173 -0
- data/contrib/lz4/build/VS2017/frametest/frametest.vcxproj +180 -0
- data/contrib/lz4/build/VS2017/fullbench-dll/fullbench-dll.vcxproj +184 -0
- data/contrib/lz4/build/VS2017/fullbench/fullbench.vcxproj +180 -0
- data/contrib/lz4/build/VS2017/fuzzer/fuzzer.vcxproj +177 -0
- data/contrib/lz4/build/VS2017/liblz4-dll/liblz4-dll.rc +51 -0
- data/contrib/lz4/build/VS2017/liblz4-dll/liblz4-dll.vcxproj +183 -0
- data/contrib/lz4/build/VS2017/liblz4/liblz4.vcxproj +179 -0
- data/contrib/lz4/build/VS2017/lz4.sln +103 -0
- data/contrib/lz4/build/VS2017/lz4/lz4.rc +51 -0
- data/contrib/lz4/build/VS2017/lz4/lz4.vcxproj +164 -0
- data/contrib/lz4/build/cmake/CMakeLists.txt +235 -0
- data/contrib/lz4/lib/README.md +98 -34
- data/contrib/lz4/lib/liblz4-dll.rc.in +35 -0
- data/contrib/lz4/lib/lz4.c +1698 -681
- data/contrib/lz4/lib/lz4.h +546 -235
- data/contrib/lz4/lib/lz4frame.c +608 -378
- data/contrib/lz4/lib/lz4frame.h +315 -83
- data/contrib/lz4/lib/lz4frame_static.h +4 -100
- data/contrib/lz4/lib/lz4hc.c +1090 -282
- data/contrib/lz4/lib/lz4hc.h +276 -141
- data/contrib/lz4/lib/xxhash.c +371 -235
- data/contrib/lz4/lib/xxhash.h +128 -93
- data/contrib/lz4/ossfuzz/Makefile +78 -0
- data/contrib/lz4/ossfuzz/compress_frame_fuzzer.c +48 -0
- data/contrib/lz4/ossfuzz/compress_fuzzer.c +58 -0
- data/contrib/lz4/ossfuzz/compress_hc_fuzzer.c +64 -0
- data/contrib/lz4/ossfuzz/decompress_frame_fuzzer.c +75 -0
- data/contrib/lz4/ossfuzz/decompress_fuzzer.c +62 -0
- data/contrib/lz4/ossfuzz/fuzz.h +48 -0
- data/contrib/lz4/ossfuzz/fuzz_data_producer.c +77 -0
- data/contrib/lz4/ossfuzz/fuzz_data_producer.h +36 -0
- data/contrib/lz4/ossfuzz/fuzz_helpers.h +94 -0
- data/contrib/lz4/ossfuzz/lz4_helpers.c +51 -0
- data/contrib/lz4/ossfuzz/lz4_helpers.h +13 -0
- data/contrib/lz4/ossfuzz/ossfuzz.sh +23 -0
- data/contrib/lz4/ossfuzz/round_trip_frame_fuzzer.c +43 -0
- data/contrib/lz4/ossfuzz/round_trip_fuzzer.c +57 -0
- data/contrib/lz4/ossfuzz/round_trip_hc_fuzzer.c +44 -0
- data/contrib/lz4/ossfuzz/round_trip_stream_fuzzer.c +302 -0
- data/contrib/lz4/ossfuzz/standaloneengine.c +74 -0
- data/contrib/lz4/ossfuzz/travisoss.sh +26 -0
- data/contrib/lz4/tmp +0 -0
- data/contrib/lz4/tmpsparse +0 -0
- data/ext/blockapi.c +5 -5
- data/ext/extlz4.c +2 -0
- data/ext/extlz4.h +5 -0
- data/ext/frameapi.c +1 -1
- data/ext/hashargs.c +2 -2
- data/ext/hashargs.h +1 -1
- data/ext/lz4_amalgam.c +0 -23
- data/gemstub.rb +5 -16
- data/lib/extlz4.rb +51 -3
- data/lib/extlz4/oldstream.rb +1 -1
- data/test/common.rb +2 -2
- metadata +73 -16
- data/contrib/lz4/circle.yml +0 -39
- data/contrib/lz4/lib/lz4opt.h +0 -366
- data/lib/extlz4/version.rb +0 -3
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
|
-
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: f1060d0000526f937ddf36e247c7e2041fbdf7241382d38a88a85bd39738d958
|
|
4
|
+
data.tar.gz: 61e5321caa13e409b8bd1af20429c184fca8fcc87f73ce31b1cbd4caf1acde24
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 487f4884327c7111ad64877190fb07a25a38ff1de84b1544ec131a238d6b1f617daf89d6ff5a9776ec05783d78139c41c481f8b2fd7b2dfba56c92019fbce413
|
|
7
|
+
data.tar.gz: 00c9a7ccdec52f3a7f60aeae9626af3f83ef8a7a8a456fe267bb4244e930fe977d2c9af72137e2e6fb1cc6906b5ab7a84eb01a393c03e69cc83d393284eae710
|
data/HISTORY.ja.md
CHANGED
|
@@ -1,3 +1,28 @@
|
|
|
1
|
+
# extlz4-0.3.1 (令和2年6月7日)
|
|
2
|
+
|
|
3
|
+
* lz4-1.9.2 に更新
|
|
4
|
+
* `extlz4.so` が `<extlz4-dir>/lib` へ置かれない環境に対応
|
|
5
|
+
* https://github.com/dearblue/ruby-extlz4/issues/2
|
|
6
|
+
|
|
7
|
+
|
|
8
|
+
# extlz4-0.3 (平成31年4月)
|
|
9
|
+
|
|
10
|
+
* lz4-1.9.0 に更新
|
|
11
|
+
* `LZ4` モジュールをリファインメントとして利用できるように機能を追加
|
|
12
|
+
* `Object` および `String` クラスで `.to_lz4frame` `.unlz4frame` メソッドが利用できるようになりました。
|
|
13
|
+
* `String` クラスで `.to_lz4block` `.unlz4block` メソッドが利用できるようになりました。
|
|
14
|
+
|
|
15
|
+
|
|
16
|
+
# extlz4-0.2.5 (平成30年1月14日 日曜日)
|
|
17
|
+
|
|
18
|
+
* lz4 ライブラリを [1.8.1](https://github.com/lz4/lz4/releases/tag/v1.8.1) に更新
|
|
19
|
+
|
|
20
|
+
|
|
21
|
+
# extlz4-0.2.4.3 (平成30年1月13日 土曜日)
|
|
22
|
+
|
|
23
|
+
* LZ4::Decoder.read でバッファ領域が確保されていなかった問題を修正
|
|
24
|
+
|
|
25
|
+
|
|
1
26
|
# extlz4-0.2.4.2 (平成30年1月7日 日曜日)
|
|
2
27
|
|
|
3
28
|
* バージョン番号の改定のみ
|
data/README.md
CHANGED
|
@@ -5,28 +5,28 @@
|
|
|
5
5
|
|
|
6
6
|
LZ4 データストリームを圧縮・伸張できます。lz4-cli で扱うことが出来ます。
|
|
7
7
|
|
|
8
|
-
```
|
|
8
|
+
```shell:shell
|
|
9
9
|
$ dmesg | ruby -r extlz4 -e 'LZ4.encode_file($stdin.binmode, $stdout.binmode)' | lz4c -d | more
|
|
10
10
|
```
|
|
11
11
|
|
|
12
12
|
ほかの ruby 向けの lz4 バインディングライブラリとしては KOMIYA Atsushi さんによる [lz4-ruby (http://rubygems.org/gems/lz4-ruby)](http://rubygems.org/gems/lz4-ruby) があります。
|
|
13
13
|
|
|
14
14
|
|
|
15
|
-
##
|
|
16
|
-
|
|
17
|
-
*
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
*
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
*
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
*
|
|
28
|
-
|
|
29
|
-
|
|
15
|
+
## FEATURES (機能)
|
|
16
|
+
|
|
17
|
+
* Generic LZ4 frame data process (for `.lz4` file format)
|
|
18
|
+
* Decode LZ4 Frame data : `LZ4.decode`
|
|
19
|
+
* Encode LZ4 Frame data : `LZ4.encode`
|
|
20
|
+
* Generic LZ4 frame data file process (for `.lz4` file format)
|
|
21
|
+
* Decode LZ4 Frame data file : `LZ4.decode_file`
|
|
22
|
+
* Encode LZ4 Frame data file : `LZ4.encode_file`
|
|
23
|
+
* LZ4 block data process
|
|
24
|
+
* Decode LZ4 block data : `LZ4.block_decode`
|
|
25
|
+
* Encode LZ4 block data : `LZ4.block_encode` (supporting high compression level)
|
|
26
|
+
* Streaming Decode LZ4 block data : `LZ4.block_stream_decode` and `LZ4::BlockDecoder#update`
|
|
27
|
+
* Streaming Encode LZ4 block data : `LZ4.block_stream_encode` and `LZ4::BlockEncoder#update` (supporting high compression level)
|
|
28
|
+
|
|
29
|
+
See [Quick reference](QUICKREF.md) for more details.
|
|
30
30
|
|
|
31
31
|
|
|
32
32
|
## ATTENTIONS (注意事項)
|
|
@@ -59,21 +59,6 @@ extlz4-0.1.1 でその不具合の修正を行いました。
|
|
|
59
59
|
修復できるのはあくまで extlz4-0.1 のこの不具合に起因するファイルのみとなります。
|
|
60
60
|
|
|
61
61
|
|
|
62
|
-
## FEATURES (機能)
|
|
63
|
-
|
|
64
|
-
* Generic LZ4 frame data process
|
|
65
|
-
* Decode LZ4 Frame data : LZ4.decode
|
|
66
|
-
* Encode LZ4 Frame data : LZ4.encode
|
|
67
|
-
* Generic LZ4 frame data file process
|
|
68
|
-
* Decode LZ4 Frame data file : LZ4.decode\_file
|
|
69
|
-
* Encode LZ4 Frame data file : LZ4.encode\_file
|
|
70
|
-
* LZ4 block data process
|
|
71
|
-
* Decode LZ4 block data : LZ4.block\_decode
|
|
72
|
-
* Encode LZ4 block data : LZ4.block\_encode (supporting high compression level)
|
|
73
|
-
* Streaming Decode LZ4 block data : LZ4.block\_stream\_decode and LZ4::BlockDecoder#update
|
|
74
|
-
* Streaming Encode LZ4 block data : LZ4.block\_stream\_encode and LZ4::BlockEncoder#update (supporting high compression level)
|
|
75
|
-
|
|
76
|
-
|
|
77
62
|
## ABOUT TAINT STATE AND SECURITY (汚染状態とセキュリティについて)
|
|
78
63
|
|
|
79
64
|
extlz4 はオブジェクト間での汚染状態を一方向伝播します。
|
|
@@ -92,25 +77,25 @@ First, load extlz4. (最初に extlz4 を読み込んでください)
|
|
|
92
77
|
require "extlz4"
|
|
93
78
|
```
|
|
94
79
|
|
|
95
|
-
###
|
|
80
|
+
### One shot decoding from LZ4 Frame (LZ4 Frame 伸張処理)
|
|
96
81
|
|
|
97
82
|
``` ruby:ruby
|
|
98
83
|
uncompressed_data_string = LZ4.decode(compressed_data_string)
|
|
99
84
|
```
|
|
100
85
|
|
|
101
|
-
###
|
|
86
|
+
### One shot encoding to LZ4 Frame (LZ4 Frame 通常圧縮処理)
|
|
102
87
|
|
|
103
88
|
``` ruby:ruby
|
|
104
89
|
compressed_data_string = LZ4.encode(uncompressed_data_string)
|
|
105
90
|
```
|
|
106
91
|
|
|
107
|
-
###
|
|
92
|
+
### One shot high compression encoding to LZ4 Frame (LZ4 Frame 高圧縮処理)
|
|
108
93
|
|
|
109
94
|
``` ruby:ruby
|
|
110
95
|
compressed_data_string = LZ4.encode(uncompressed_data_string, 9)
|
|
111
96
|
```
|
|
112
97
|
|
|
113
|
-
###
|
|
98
|
+
### Stream decoding to LZ4 Frame
|
|
114
99
|
|
|
115
100
|
``` ruby:ruby
|
|
116
101
|
File.open("sample.txt.lz4", "rb") do |file|
|
|
@@ -122,7 +107,7 @@ File.open("sample.txt.lz4", "rb") do |file|
|
|
|
122
107
|
end
|
|
123
108
|
```
|
|
124
109
|
|
|
125
|
-
###
|
|
110
|
+
### Stream encoding by high compression to LZ4 Frame
|
|
126
111
|
|
|
127
112
|
``` ruby:ruby
|
|
128
113
|
File.open("sample.txt.lz4", "wb") do |file|
|
|
@@ -133,7 +118,7 @@ File.open("sample.txt.lz4", "wb") do |file|
|
|
|
133
118
|
end
|
|
134
119
|
```
|
|
135
120
|
|
|
136
|
-
###
|
|
121
|
+
### Stream encoding without block to LZ4 Frame
|
|
137
122
|
|
|
138
123
|
``` ruby:ruby
|
|
139
124
|
file = File.open("sample.txt.lz4", "wb")
|
|
@@ -142,7 +127,7 @@ lz4 << "abcdefghijklmnopqrstuvwxyz\n"
|
|
|
142
127
|
lz4.close # VERY IMPORTANT!
|
|
143
128
|
```
|
|
144
129
|
|
|
145
|
-
###
|
|
130
|
+
### One shot block data processing to/from LZ4 Block (fast compression encoding and decoding)
|
|
146
131
|
|
|
147
132
|
``` ruby:ruby
|
|
148
133
|
src = "abcdefg" * 100
|
|
@@ -151,7 +136,7 @@ data = LZ4.block_decode(lz4data)
|
|
|
151
136
|
p src == data # => true
|
|
152
137
|
```
|
|
153
138
|
|
|
154
|
-
###
|
|
139
|
+
### One shot block data processing to/from LZ4 Block (high compression encoding and decoding)
|
|
155
140
|
|
|
156
141
|
``` ruby:ruby
|
|
157
142
|
src = "abcdefg" * 100
|
|
@@ -161,7 +146,7 @@ data = LZ4.block_decode(lz4data)
|
|
|
161
146
|
p src == data # => true
|
|
162
147
|
```
|
|
163
148
|
|
|
164
|
-
###
|
|
149
|
+
### One shot block data processing to/from LZ4 Block (high speed encoding)
|
|
165
150
|
|
|
166
151
|
``` ruby:ruby
|
|
167
152
|
src = "abcdefg" * 100
|
|
@@ -169,7 +154,7 @@ level = -19 # transform to one's complement as acceleration
|
|
|
169
154
|
lz4data = LZ4.block_encode(level, src)
|
|
170
155
|
```
|
|
171
156
|
|
|
172
|
-
### Block stream data processing (high compression encoding and decoding)
|
|
157
|
+
### Block stream data processing to/from LZ4 Block (high compression encoding and decoding)
|
|
173
158
|
|
|
174
159
|
``` ruby:ruby
|
|
175
160
|
level = 8 # (OPTIONAL PARAMETER)
|
|
@@ -194,6 +179,11 @@ p src2 == data # => true
|
|
|
194
179
|
```
|
|
195
180
|
|
|
196
181
|
|
|
182
|
+
## Support `Ractor` with Ruby3
|
|
183
|
+
|
|
184
|
+
Ruby3 で追加された `Ractor` に対応しています。
|
|
185
|
+
|
|
186
|
+
|
|
197
187
|
## BONUS (おまけ)
|
|
198
188
|
|
|
199
189
|
コマンドラインプログラムとして ``extlz4`` が追加されます。
|
|
@@ -201,3 +191,21 @@ p src2 == data # => true
|
|
|
201
191
|
これは lz4 と同程度の機能を持ちます (車輪の再発明とも言う)。
|
|
202
192
|
|
|
203
193
|
とはいえ、引数のとり方を変えてあり、gzip のような形で利用できます。
|
|
194
|
+
|
|
195
|
+
|
|
196
|
+
## SPECIFICATION (仕様)
|
|
197
|
+
|
|
198
|
+
- package name: extlz4
|
|
199
|
+
- author: dearblue (mailto:dearblue@users.osdn.me)
|
|
200
|
+
- project page: <https://github.com/dearblue/ruby-extlz4>
|
|
201
|
+
- how to install: `gem install extlz4`
|
|
202
|
+
- version: 0.3.2
|
|
203
|
+
- product quality: technical preview
|
|
204
|
+
- licensing: [2 clause BSD License](LICENSE)
|
|
205
|
+
- dependency gems: none
|
|
206
|
+
- dependency external c libraries: none
|
|
207
|
+
- bundled external c libraries (git submodules):
|
|
208
|
+
- [lz4](https://github.com/lz4/lz4)
|
|
209
|
+
[version 1.9.3](https://github.com/lz4/lz4/tree/v1.9.3)
|
|
210
|
+
under [2 clause BSD license](https://github.com/lz4/lz4/blob/v1.9.3/LICENSE)
|
|
211
|
+
by [Yann Collet](https://github.com/Cyan4973)
|
data/bin/extlz4
CHANGED
data/contrib/lz4/INSTALL
CHANGED
|
@@ -8,6 +8,7 @@ make install # this command may require root access
|
|
|
8
8
|
|
|
9
9
|
LZ4's `Makefile` supports standard [Makefile conventions],
|
|
10
10
|
including [staged installs], [redirection], or [command redefinition].
|
|
11
|
+
It is compatible with parallel builds (`-j#`).
|
|
11
12
|
|
|
12
13
|
[Makefile conventions]: https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html
|
|
13
14
|
[staged installs]: https://www.gnu.org/prep/standards/html_node/DESTDIR.html
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
ifeq ($(V), 1)
|
|
2
|
+
Q =
|
|
3
|
+
else
|
|
4
|
+
Q = @
|
|
5
|
+
endif
|
|
6
|
+
|
|
7
|
+
TARGET_OS ?= $(shell uname)
|
|
8
|
+
ifeq ($(TARGET_OS),)
|
|
9
|
+
TARGET_OS ?= $(OS)
|
|
10
|
+
endif
|
|
11
|
+
|
|
12
|
+
ifneq (,$(filter Windows%,$(TARGET_OS)))
|
|
13
|
+
LIBLZ4 = liblz4-$(LIBVER_MAJOR)
|
|
14
|
+
LIBLZ4_EXP = liblz4.lib
|
|
15
|
+
WINBASED = yes
|
|
16
|
+
else
|
|
17
|
+
LIBLZ4_EXP = liblz4.dll.a
|
|
18
|
+
ifneq (,$(filter MINGW%,$(TARGET_OS)))
|
|
19
|
+
LIBLZ4 = liblz4
|
|
20
|
+
WINBASED = yes
|
|
21
|
+
else
|
|
22
|
+
ifneq (,$(filter MSYS%,$(TARGET_OS)))
|
|
23
|
+
LIBLZ4 = msys-lz4-$(LIBVER_MAJOR)
|
|
24
|
+
WINBASED = yes
|
|
25
|
+
else
|
|
26
|
+
ifneq (,$(filter CYGWIN%,$(TARGET_OS)))
|
|
27
|
+
LIBLZ4 = cyglz4-$(LIBVER_MAJOR)
|
|
28
|
+
WINBASED = yes
|
|
29
|
+
else
|
|
30
|
+
LIBLZ4 = liblz4.$(SHARED_EXT_VER)
|
|
31
|
+
WINBASED = no
|
|
32
|
+
EXT =
|
|
33
|
+
endif
|
|
34
|
+
endif
|
|
35
|
+
endif
|
|
36
|
+
endif
|
|
37
|
+
|
|
38
|
+
ifeq ($(WINBASED),yes)
|
|
39
|
+
EXT = .exe
|
|
40
|
+
WINDRES = windres
|
|
41
|
+
endif
|
|
42
|
+
|
|
43
|
+
#determine if dev/nul based on host environment
|
|
44
|
+
ifneq (,$(filter MINGW% MSYS% CYGWIN%,$(shell uname)))
|
|
45
|
+
VOID := /dev/null
|
|
46
|
+
else
|
|
47
|
+
ifneq (,$(filter Windows%,$(OS)))
|
|
48
|
+
VOID := nul
|
|
49
|
+
else
|
|
50
|
+
VOID := /dev/null
|
|
51
|
+
endif
|
|
52
|
+
endif
|
|
53
|
+
|
|
54
|
+
ifneq (,$(filter Linux Darwin GNU/kFreeBSD GNU OpenBSD FreeBSD NetBSD DragonFly SunOS Haiku MidnightBSD MINGW% CYGWIN% MSYS%,$(shell uname)))
|
|
55
|
+
POSIX_ENV = Yes
|
|
56
|
+
else
|
|
57
|
+
POSIX_ENV = No
|
|
58
|
+
endif
|
|
59
|
+
|
|
60
|
+
# Avoid symlinks when targetting Windows or building on a Windows host
|
|
61
|
+
ifeq ($(WINBASED),yes)
|
|
62
|
+
LN_S = cp -p
|
|
63
|
+
LN_SF = cp -p
|
|
64
|
+
else
|
|
65
|
+
ifneq (,$(filter MINGW% MSYS% CYGWIN%,$(shell uname)))
|
|
66
|
+
LN_S = cp -p
|
|
67
|
+
LN_SF = cp -p
|
|
68
|
+
else
|
|
69
|
+
ifneq (,$(filter Windows%,$(OS)))
|
|
70
|
+
LN_S = cp -p
|
|
71
|
+
LN_SF = cp -p
|
|
72
|
+
else
|
|
73
|
+
LN_S = ln -s
|
|
74
|
+
LN_SF = ln -sf
|
|
75
|
+
endif
|
|
76
|
+
endif
|
|
77
|
+
endif
|
|
78
|
+
|
|
79
|
+
ifneq (,$(filter $(shell uname),SunOS))
|
|
80
|
+
INSTALL ?= ginstall
|
|
81
|
+
else
|
|
82
|
+
INSTALL ?= install
|
|
83
|
+
endif
|
|
84
|
+
|
|
85
|
+
INSTALL_PROGRAM ?= $(INSTALL) -m 755
|
|
86
|
+
INSTALL_DATA ?= $(INSTALL) -m 644
|
|
87
|
+
INSTALL_DIR ?= $(INSTALL) -d -m 755
|
data/contrib/lz4/NEWS
CHANGED
|
@@ -1,3 +1,92 @@
|
|
|
1
|
+
v1.9.3
|
|
2
|
+
perf: highly improved speed in kernel space, by @terrelln
|
|
3
|
+
perf: faster speed with Visual Studio, thanks to @wolfpld and @remittor
|
|
4
|
+
perf: improved dictionary compression speed, by @felixhandte
|
|
5
|
+
perf: fixed LZ4_compress_HC_destSize() ratio, detected by @hsiangkao
|
|
6
|
+
perf: reduced stack usage in high compression mode, by @Yanpas
|
|
7
|
+
api : LZ4_decompress_safe_partial() supports unknown compressed size, requested by @jfkthame
|
|
8
|
+
api : improved LZ4F_compressBound() with automatic flushing, by Christopher Harvie
|
|
9
|
+
api : can (de)compress to/from NULL without UBs
|
|
10
|
+
api : fix alignment test on 32-bit systems (state initialization)
|
|
11
|
+
api : fix LZ4_saveDictHC() in corner case scenario, detected by @IgorKorkin
|
|
12
|
+
cli : `-l` legacy format is now compatible with `-m` multiple files, by Filipe Calasans
|
|
13
|
+
cli : benchmark mode supports dictionary, by @rkoradi
|
|
14
|
+
cli : fix --fast with large argument, detected by @picoHz
|
|
15
|
+
build: link to user-defined memory functions with LZ4_USER_MEMORY_FUNCTIONS, suggested by Yuriy Levchenko
|
|
16
|
+
build: contrib/cmake_unofficial/ moved to build/cmake/
|
|
17
|
+
build: visual/* moved to build/
|
|
18
|
+
build: updated meson script, by @neheb
|
|
19
|
+
build: tinycc support, by Anton Kochkov
|
|
20
|
+
install: Haiku support, by Jerome Duval
|
|
21
|
+
doc : updated LZ4 frame format, clarify EndMark
|
|
22
|
+
|
|
23
|
+
v1.9.2
|
|
24
|
+
fix : out-of-bound read in exceptional circumstances when using decompress_partial(), by @terrelln
|
|
25
|
+
fix : slim opportunity for out-of-bound write with compress_fast() with a large enough input and when providing an output smaller than recommended (< LZ4_compressBound(inputSize)), by @terrelln
|
|
26
|
+
fix : rare data corruption bug with LZ4_compress_destSize(), by @terrelln
|
|
27
|
+
fix : data corruption bug when Streaming with an Attached Dict in HC Mode, by @felixhandte
|
|
28
|
+
perf: enable LZ4_FAST_DEC_LOOP on aarch64/GCC by default, by @prekageo
|
|
29
|
+
perf: improved lz4frame streaming API speed, by @dreambottle
|
|
30
|
+
perf: speed up lz4hc on slow patterns when using external dictionary, by @terrelln
|
|
31
|
+
api: better in-place decompression and compression support
|
|
32
|
+
cli : --list supports multi-frames files, by @gstedman
|
|
33
|
+
cli: --version outputs to stdout
|
|
34
|
+
cli : add option --best as an alias of -12 , by @Low-power
|
|
35
|
+
misc: Integration into oss-fuzz by @cmeister2, expanded list of scenarios by @terrelln
|
|
36
|
+
|
|
37
|
+
v1.9.1
|
|
38
|
+
fix : decompression functions were reading a few bytes beyond input size (introduced in v1.9.0, reported by @ppodolsky and @danlark1)
|
|
39
|
+
api : fix : lz4frame initializers compatibility with c++, reported by @degski
|
|
40
|
+
cli : added command --list, based on a patch by @gabrielstedman
|
|
41
|
+
build: improved Windows build, by @JPeterMugaas
|
|
42
|
+
build: AIX, by Norman Green
|
|
43
|
+
|
|
44
|
+
v1.9.0
|
|
45
|
+
perf: large decompression speed improvement on x86/x64 (up to +20%) by @djwatson
|
|
46
|
+
api : changed : _destSize() compression variants are promoted to stable API
|
|
47
|
+
api : new : LZ4_initStream(HC), replacing LZ4_resetStream(HC)
|
|
48
|
+
api : changed : LZ4_resetStream(HC) as recommended reset function, for better performance on small data
|
|
49
|
+
cli : support custom block sizes, by @blezsan
|
|
50
|
+
build: source code can be amalgamated, by Bing Xu
|
|
51
|
+
build: added meson build, by @lzutao
|
|
52
|
+
build: new build macros : LZ4_DISTANCE_MAX, LZ4_FAST_DEC_LOOP
|
|
53
|
+
install: MidnightBSD, by @laffer1
|
|
54
|
+
install: msys2 on Windows 10, by @vtorri
|
|
55
|
+
|
|
56
|
+
v1.8.3
|
|
57
|
+
perf: minor decompression speed improvement (~+2%) with gcc
|
|
58
|
+
fix : corruption in v1.8.2 at level 9 for files > 64KB under rare conditions (#560)
|
|
59
|
+
cli : new command --fast, by @jennifermliu
|
|
60
|
+
cli : fixed elapsed time, and added cpu load indicator (on -vv) (#555)
|
|
61
|
+
api : LZ4_decompress_safe_partial() now decodes exactly the nb of bytes requested (feature request #566)
|
|
62
|
+
build : added Haiku target, by @fbrosson, and MidnightBSD, by @laffer1
|
|
63
|
+
doc : updated documentation regarding dictionary compression
|
|
64
|
+
|
|
65
|
+
v1.8.2
|
|
66
|
+
perf: *much* faster dictionary compression on small files, by @felixhandte
|
|
67
|
+
perf: improved decompression speed and binary size, by Alexey Tourbin (@svpv)
|
|
68
|
+
perf: slightly faster HC compression and decompression speed
|
|
69
|
+
perf: very small compression ratio improvement
|
|
70
|
+
fix : compression compatible with low memory addresses (< 0xFFFF)
|
|
71
|
+
fix : decompression segfault when provided with NULL input, by @terrelln
|
|
72
|
+
cli : new command --favor-decSpeed
|
|
73
|
+
cli : benchmark mode more accurate for small inputs
|
|
74
|
+
fullbench : can bench _destSize() variants, by @felixhandte
|
|
75
|
+
doc : clarified block format parsing restrictions, by Alexey Tourbin (@svpv)
|
|
76
|
+
|
|
77
|
+
v1.8.1
|
|
78
|
+
perf : faster and stronger ultra modes (levels 10+)
|
|
79
|
+
perf : slightly faster compression and decompression speed
|
|
80
|
+
perf : fix bad degenerative case, reported by @c-morgenstern
|
|
81
|
+
fix : decompression failed when using a combination of extDict + low memory address (#397), reported and fixed by Julian Scheid (@jscheid)
|
|
82
|
+
cli : support for dictionary compression (`-D`), by Felix Handte @felixhandte
|
|
83
|
+
cli : fix : `lz4 -d --rm` preserves timestamp (#441)
|
|
84
|
+
cli : fix : do not modify /dev/null permission as root, by @aliceatlas
|
|
85
|
+
api : `_destSize()` variant supported for all compression levels
|
|
86
|
+
build : `make` and `make test` compatible with `-jX`, reported by @mwgamera
|
|
87
|
+
build : can control LZ4LIB_VISIBILITY macro, by @mikir
|
|
88
|
+
install: fix man page directory (#387), reported by Stuart Cardall (@itoffshore)
|
|
89
|
+
|
|
1
90
|
v1.8.0
|
|
2
91
|
cli : fix : do not modify /dev/null permissions, reported by @Maokaman1
|
|
3
92
|
cli : added GNU separator -- specifying that all following arguments are files
|
data/contrib/lz4/README.md
CHANGED
|
@@ -2,74 +2,68 @@ LZ4 - Extremely fast compression
|
|
|
2
2
|
================================
|
|
3
3
|
|
|
4
4
|
LZ4 is lossless compression algorithm,
|
|
5
|
-
providing compression speed
|
|
5
|
+
providing compression speed > 500 MB/s per core,
|
|
6
6
|
scalable with multi-cores CPU.
|
|
7
7
|
It features an extremely fast decoder,
|
|
8
8
|
with speed in multiple GB/s per core,
|
|
9
9
|
typically reaching RAM speed limits on multi-core systems.
|
|
10
10
|
|
|
11
11
|
Speed can be tuned dynamically, selecting an "acceleration" factor
|
|
12
|
-
which trades compression ratio for
|
|
12
|
+
which trades compression ratio for faster speed.
|
|
13
13
|
On the other end, a high compression derivative, LZ4_HC, is also provided,
|
|
14
14
|
trading CPU time for improved compression ratio.
|
|
15
15
|
All versions feature the same decompression speed.
|
|
16
16
|
|
|
17
|
+
LZ4 is also compatible with [dictionary compression](https://github.com/facebook/zstd#the-case-for-small-data-compression),
|
|
18
|
+
both at [API](https://github.com/lz4/lz4/blob/v1.8.3/lib/lz4frame.h#L481) and [CLI](https://github.com/lz4/lz4/blob/v1.8.3/programs/lz4.1.md#operation-modifiers) levels.
|
|
19
|
+
It can ingest any input file as dictionary, though only the final 64KB are used.
|
|
20
|
+
This capability can be combined with the [Zstandard Dictionary Builder](https://github.com/facebook/zstd/blob/v1.3.5/programs/zstd.1.md#dictionary-builder),
|
|
21
|
+
in order to drastically improve compression performance on small files.
|
|
22
|
+
|
|
23
|
+
|
|
17
24
|
LZ4 library is provided as open-source software using BSD 2-Clause license.
|
|
18
25
|
|
|
19
26
|
|
|
20
27
|
|Branch |Status |
|
|
21
28
|
|------------|---------|
|
|
22
|
-
|master | [![Build Status][travisMasterBadge]][travisLink] [![Build status][AppveyorMasterBadge]][AppveyorLink] [![coverity][coverBadge]][coverlink] |
|
|
23
29
|
|dev | [![Build Status][travisDevBadge]][travisLink] [![Build status][AppveyorDevBadge]][AppveyorLink] |
|
|
24
30
|
|
|
25
|
-
[travisMasterBadge]: https://travis-ci.org/lz4/lz4.svg?branch=master "Continuous Integration test suite"
|
|
26
31
|
[travisDevBadge]: https://travis-ci.org/lz4/lz4.svg?branch=dev "Continuous Integration test suite"
|
|
27
32
|
[travisLink]: https://travis-ci.org/lz4/lz4
|
|
28
|
-
[AppveyorMasterBadge]: https://ci.appveyor.com/api/projects/status/github/lz4/lz4?branch=master&svg=true "Windows test suite"
|
|
29
33
|
[AppveyorDevBadge]: https://ci.appveyor.com/api/projects/status/github/lz4/lz4?branch=dev&svg=true "Windows test suite"
|
|
30
34
|
[AppveyorLink]: https://ci.appveyor.com/project/YannCollet/lz4-1lndh
|
|
31
|
-
[coverBadge]: https://scan.coverity.com/projects/4735/badge.svg "Static code analysis of Master branch"
|
|
32
|
-
[coverlink]: https://scan.coverity.com/projects/4735
|
|
33
35
|
|
|
34
|
-
> **Branch Policy:**
|
|
35
|
-
> - The "master" branch is considered stable, at all times.
|
|
36
|
-
> - The "dev" branch is the one where all contributions must be merged
|
|
37
|
-
before being promoted to master.
|
|
38
|
-
> + If you plan to propose a patch, please commit into the "dev" branch,
|
|
39
|
-
or its own feature branch.
|
|
40
|
-
Direct commit to "master" are not permitted.
|
|
41
36
|
|
|
42
37
|
Benchmarks
|
|
43
38
|
-------------------------
|
|
44
39
|
|
|
45
40
|
The benchmark uses [lzbench], from @inikep
|
|
46
|
-
compiled with GCC
|
|
47
|
-
The reference system uses a Core i7-
|
|
41
|
+
compiled with GCC v8.2.0 on Linux 64-bits (Ubuntu 4.18.0-17).
|
|
42
|
+
The reference system uses a Core i7-9700K CPU @ 4.9GHz (w/ turbo boost).
|
|
48
43
|
Benchmark evaluates the compression of reference [Silesia Corpus]
|
|
49
44
|
in single-thread mode.
|
|
50
45
|
|
|
51
46
|
[lzbench]: https://github.com/inikep/lzbench
|
|
52
47
|
[Silesia Corpus]: http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia
|
|
53
48
|
|
|
54
|
-
| Compressor
|
|
55
|
-
| ----------
|
|
56
|
-
| memcpy
|
|
57
|
-
|**LZ4
|
|
58
|
-
|
|
59
|
-
|
|
|
60
|
-
|
|
|
61
|
-
|
|
|
62
|
-
| LZF v3.6
|
|
63
|
-
|
|
|
64
|
-
|
|
65
|
-
| [zlib] deflate 1.2.
|
|
66
|
-
|**LZ4 HC -9 (v1.7.3)** |**2.720**| 34 MB/s | **3240 MB/s** |
|
|
67
|
-
| [zlib] deflate 1.2.8 -6| 3.099 | 33 MB/s | 390 MB/s |
|
|
49
|
+
| Compressor | Ratio | Compression | Decompression |
|
|
50
|
+
| ---------- | ----- | ----------- | ------------- |
|
|
51
|
+
| memcpy | 1.000 | 13700 MB/s | 13700 MB/s |
|
|
52
|
+
|**LZ4 default (v1.9.0)** |**2.101**| **780 MB/s**| **4970 MB/s** |
|
|
53
|
+
| LZO 2.09 | 2.108 | 670 MB/s | 860 MB/s |
|
|
54
|
+
| QuickLZ 1.5.0 | 2.238 | 575 MB/s | 780 MB/s |
|
|
55
|
+
| Snappy 1.1.4 | 2.091 | 565 MB/s | 1950 MB/s |
|
|
56
|
+
| [Zstandard] 1.4.0 -1 | 2.883 | 515 MB/s | 1380 MB/s |
|
|
57
|
+
| LZF v3.6 | 2.073 | 415 MB/s | 910 MB/s |
|
|
58
|
+
| [zlib] deflate 1.2.11 -1| 2.730 | 100 MB/s | 415 MB/s |
|
|
59
|
+
|**LZ4 HC -9 (v1.9.0)** |**2.721**| 41 MB/s | **4900 MB/s** |
|
|
60
|
+
| [zlib] deflate 1.2.11 -6| 3.099 | 36 MB/s | 445 MB/s |
|
|
68
61
|
|
|
69
62
|
[zlib]: http://www.zlib.net/
|
|
70
63
|
[Zstandard]: http://www.zstd.net/
|
|
71
64
|
|
|
72
|
-
LZ4 is also compatible and
|
|
65
|
+
LZ4 is also compatible and optimized for x32 mode,
|
|
66
|
+
for which it provides additional speed performance.
|
|
73
67
|
|
|
74
68
|
|
|
75
69
|
Installation
|
|
@@ -77,27 +71,39 @@ Installation
|
|
|
77
71
|
|
|
78
72
|
```
|
|
79
73
|
make
|
|
80
|
-
make install # this command may require root
|
|
74
|
+
make install # this command may require root permissions
|
|
81
75
|
```
|
|
82
76
|
|
|
83
77
|
LZ4's `Makefile` supports standard [Makefile conventions],
|
|
84
78
|
including [staged installs], [redirection], or [command redefinition].
|
|
79
|
+
It is compatible with parallel builds (`-j#`).
|
|
85
80
|
|
|
86
81
|
[Makefile conventions]: https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html
|
|
87
82
|
[staged installs]: https://www.gnu.org/prep/standards/html_node/DESTDIR.html
|
|
88
83
|
[redirection]: https://www.gnu.org/prep/standards/html_node/Directory-Variables.html
|
|
89
84
|
[command redefinition]: https://www.gnu.org/prep/standards/html_node/Utilities-in-Makefiles.html
|
|
90
85
|
|
|
86
|
+
Building LZ4 - Using vcpkg
|
|
87
|
+
|
|
88
|
+
You can download and install LZ4 using the [vcpkg](https://github.com/Microsoft/vcpkg) dependency manager:
|
|
89
|
+
|
|
90
|
+
git clone https://github.com/Microsoft/vcpkg.git
|
|
91
|
+
cd vcpkg
|
|
92
|
+
./bootstrap-vcpkg.sh
|
|
93
|
+
./vcpkg integrate install
|
|
94
|
+
vcpkg install lz4
|
|
95
|
+
|
|
96
|
+
The LZ4 port in vcpkg is kept up to date by Microsoft team members and community contributors. If the version is out of date, please [create an issue or pull request](https://github.com/Microsoft/vcpkg) on the vcpkg repository.
|
|
91
97
|
|
|
92
98
|
Documentation
|
|
93
99
|
-------------------------
|
|
94
100
|
|
|
95
101
|
The raw LZ4 block compression format is detailed within [lz4_Block_format].
|
|
96
102
|
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
Interoperable versions of LZ4 must respect
|
|
103
|
+
Arbitrarily long files or data streams are compressed using multiple blocks,
|
|
104
|
+
for streaming requirements. These blocks are organized into a frame,
|
|
105
|
+
defined into [lz4_Frame_format].
|
|
106
|
+
Interoperable versions of LZ4 must also respect the frame format.
|
|
101
107
|
|
|
102
108
|
[lz4_Block_format]: doc/lz4_Block_format.md
|
|
103
109
|
[lz4_Frame_format]: doc/lz4_Frame_format.md
|