extlz4 0.2.4.2 → 0.3.2

Sign up to get free protection for your applications and to get access to all the features.
Files changed (79) hide show
  1. checksums.yaml +5 -5
  2. data/HISTORY.ja.md +25 -0
  3. data/README.md +49 -41
  4. data/bin/extlz4 +1 -1
  5. data/contrib/lz4/INSTALL +1 -0
  6. data/contrib/lz4/Makefile.inc +87 -0
  7. data/contrib/lz4/NEWS +89 -0
  8. data/contrib/lz4/README.md +42 -36
  9. data/contrib/lz4/build/README.md +55 -0
  10. data/contrib/lz4/build/VS2010/datagen/datagen.vcxproj +169 -0
  11. data/contrib/lz4/build/VS2010/frametest/frametest.vcxproj +176 -0
  12. data/contrib/lz4/build/VS2010/fullbench-dll/fullbench-dll.vcxproj +180 -0
  13. data/contrib/lz4/build/VS2010/fullbench/fullbench.vcxproj +176 -0
  14. data/contrib/lz4/build/VS2010/fuzzer/fuzzer.vcxproj +173 -0
  15. data/contrib/lz4/build/VS2010/liblz4-dll/liblz4-dll.rc +51 -0
  16. data/contrib/lz4/build/VS2010/liblz4-dll/liblz4-dll.vcxproj +179 -0
  17. data/contrib/lz4/build/VS2010/liblz4/liblz4.vcxproj +175 -0
  18. data/contrib/lz4/build/VS2010/lz4.sln +98 -0
  19. data/contrib/lz4/build/VS2010/lz4/lz4.rc +51 -0
  20. data/contrib/lz4/build/VS2010/lz4/lz4.vcxproj +189 -0
  21. data/contrib/lz4/build/VS2017/datagen/datagen.vcxproj +173 -0
  22. data/contrib/lz4/build/VS2017/frametest/frametest.vcxproj +180 -0
  23. data/contrib/lz4/build/VS2017/fullbench-dll/fullbench-dll.vcxproj +184 -0
  24. data/contrib/lz4/build/VS2017/fullbench/fullbench.vcxproj +180 -0
  25. data/contrib/lz4/build/VS2017/fuzzer/fuzzer.vcxproj +177 -0
  26. data/contrib/lz4/build/VS2017/liblz4-dll/liblz4-dll.rc +51 -0
  27. data/contrib/lz4/build/VS2017/liblz4-dll/liblz4-dll.vcxproj +183 -0
  28. data/contrib/lz4/build/VS2017/liblz4/liblz4.vcxproj +179 -0
  29. data/contrib/lz4/build/VS2017/lz4.sln +103 -0
  30. data/contrib/lz4/build/VS2017/lz4/lz4.rc +51 -0
  31. data/contrib/lz4/build/VS2017/lz4/lz4.vcxproj +164 -0
  32. data/contrib/lz4/build/cmake/CMakeLists.txt +235 -0
  33. data/contrib/lz4/lib/README.md +98 -34
  34. data/contrib/lz4/lib/liblz4-dll.rc.in +35 -0
  35. data/contrib/lz4/lib/lz4.c +1698 -681
  36. data/contrib/lz4/lib/lz4.h +546 -235
  37. data/contrib/lz4/lib/lz4frame.c +608 -378
  38. data/contrib/lz4/lib/lz4frame.h +315 -83
  39. data/contrib/lz4/lib/lz4frame_static.h +4 -100
  40. data/contrib/lz4/lib/lz4hc.c +1090 -282
  41. data/contrib/lz4/lib/lz4hc.h +276 -141
  42. data/contrib/lz4/lib/xxhash.c +371 -235
  43. data/contrib/lz4/lib/xxhash.h +128 -93
  44. data/contrib/lz4/ossfuzz/Makefile +78 -0
  45. data/contrib/lz4/ossfuzz/compress_frame_fuzzer.c +48 -0
  46. data/contrib/lz4/ossfuzz/compress_fuzzer.c +58 -0
  47. data/contrib/lz4/ossfuzz/compress_hc_fuzzer.c +64 -0
  48. data/contrib/lz4/ossfuzz/decompress_frame_fuzzer.c +75 -0
  49. data/contrib/lz4/ossfuzz/decompress_fuzzer.c +62 -0
  50. data/contrib/lz4/ossfuzz/fuzz.h +48 -0
  51. data/contrib/lz4/ossfuzz/fuzz_data_producer.c +77 -0
  52. data/contrib/lz4/ossfuzz/fuzz_data_producer.h +36 -0
  53. data/contrib/lz4/ossfuzz/fuzz_helpers.h +94 -0
  54. data/contrib/lz4/ossfuzz/lz4_helpers.c +51 -0
  55. data/contrib/lz4/ossfuzz/lz4_helpers.h +13 -0
  56. data/contrib/lz4/ossfuzz/ossfuzz.sh +23 -0
  57. data/contrib/lz4/ossfuzz/round_trip_frame_fuzzer.c +43 -0
  58. data/contrib/lz4/ossfuzz/round_trip_fuzzer.c +57 -0
  59. data/contrib/lz4/ossfuzz/round_trip_hc_fuzzer.c +44 -0
  60. data/contrib/lz4/ossfuzz/round_trip_stream_fuzzer.c +302 -0
  61. data/contrib/lz4/ossfuzz/standaloneengine.c +74 -0
  62. data/contrib/lz4/ossfuzz/travisoss.sh +26 -0
  63. data/contrib/lz4/tmp +0 -0
  64. data/contrib/lz4/tmpsparse +0 -0
  65. data/ext/blockapi.c +5 -5
  66. data/ext/extlz4.c +2 -0
  67. data/ext/extlz4.h +5 -0
  68. data/ext/frameapi.c +1 -1
  69. data/ext/hashargs.c +2 -2
  70. data/ext/hashargs.h +1 -1
  71. data/ext/lz4_amalgam.c +0 -23
  72. data/gemstub.rb +5 -16
  73. data/lib/extlz4.rb +51 -3
  74. data/lib/extlz4/oldstream.rb +1 -1
  75. data/test/common.rb +2 -2
  76. metadata +73 -16
  77. data/contrib/lz4/circle.yml +0 -39
  78. data/contrib/lz4/lib/lz4opt.h +0 -366
  79. data/lib/extlz4/version.rb +0 -3
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: 79289896ae954100abb7a0a20312abefc1206134
4
- data.tar.gz: 2328b0e92f682b0ec3b006eb88e70cd221eda360
2
+ SHA256:
3
+ metadata.gz: f1060d0000526f937ddf36e247c7e2041fbdf7241382d38a88a85bd39738d958
4
+ data.tar.gz: 61e5321caa13e409b8bd1af20429c184fca8fcc87f73ce31b1cbd4caf1acde24
5
5
  SHA512:
6
- metadata.gz: 4cede1dccaf6955f460f8e458dadf0dc8f27ad11a76dcb5438086320d79bc5f159ea2f9b183224110a89c93061e364feb35053bc938e60881ced44e44d5d4b95
7
- data.tar.gz: 2e867e0857b9c253c3e4260e31913685429eadbe9785db89576dc556fcb184507ffd5f8b095e5fecc754ae57f1d40f28fbb2a401d459be09b1e93e1dd24ba614
6
+ metadata.gz: 487f4884327c7111ad64877190fb07a25a38ff1de84b1544ec131a238d6b1f617daf89d6ff5a9776ec05783d78139c41c481f8b2fd7b2dfba56c92019fbce413
7
+ data.tar.gz: 00c9a7ccdec52f3a7f60aeae9626af3f83ef8a7a8a456fe267bb4244e930fe977d2c9af72137e2e6fb1cc6906b5ab7a84eb01a393c03e69cc83d393284eae710
data/HISTORY.ja.md CHANGED
@@ -1,3 +1,28 @@
1
+ # extlz4-0.3.1 (令和2年6月7日)
2
+
3
+ * lz4-1.9.2 に更新
4
+ * `extlz4.so` が `<extlz4-dir>/lib` へ置かれない環境に対応
5
+ * https://github.com/dearblue/ruby-extlz4/issues/2
6
+
7
+
8
+ # extlz4-0.3 (平成31年4月)
9
+
10
+ * lz4-1.9.0 に更新
11
+ * `LZ4` モジュールをリファインメントとして利用できるように機能を追加
12
+ * `Object` および `String` クラスで `.to_lz4frame` `.unlz4frame` メソッドが利用できるようになりました。
13
+ * `String` クラスで `.to_lz4block` `.unlz4block` メソッドが利用できるようになりました。
14
+
15
+
16
+ # extlz4-0.2.5 (平成30年1月14日 日曜日)
17
+
18
+ * lz4 ライブラリを [1.8.1](https://github.com/lz4/lz4/releases/tag/v1.8.1) に更新
19
+
20
+
21
+ # extlz4-0.2.4.3 (平成30年1月13日 土曜日)
22
+
23
+ * LZ4::Decoder.read でバッファ領域が確保されていなかった問題を修正
24
+
25
+
1
26
  # extlz4-0.2.4.2 (平成30年1月7日 日曜日)
2
27
 
3
28
  * バージョン番号の改定のみ
data/README.md CHANGED
@@ -5,28 +5,28 @@
5
5
 
6
6
  LZ4 データストリームを圧縮・伸張できます。lz4-cli で扱うことが出来ます。
7
7
 
8
- ``` shell:shell
8
+ ```shell:shell
9
9
  $ dmesg | ruby -r extlz4 -e 'LZ4.encode_file($stdin.binmode, $stdout.binmode)' | lz4c -d | more
10
10
  ```
11
11
 
12
12
  ほかの ruby 向けの lz4 バインディングライブラリとしては KOMIYA Atsushi さんによる [lz4-ruby (http://rubygems.org/gems/lz4-ruby)](http://rubygems.org/gems/lz4-ruby) があります。
13
13
 
14
14
 
15
- ## SUMMARY (概要)
16
-
17
- * package name: extlz4
18
- * author: dearblue (mailto:dearblue@users.noreply.github.com)
19
- * report issue to: <https://github.com/dearblue/ruby-extlz4/issues>
20
- * how to install: `gem install extlz4`
21
- * version: 0.2.4.2
22
- * product quality: technical preview
23
- * licensing: BSD-2-Clause License
24
- * dependency gems: none
25
- * dependency external c libraries: none
26
- * bundled external c libraries:
27
- * lz4-1.8 <https://github.com/lz4/lz4/tree/v1.8.0>
28
- under [BSD 2-Clause license](https://github.com/lz4/lz4/tree/v1.8.0/LICENSE)
29
- by [Yann Collet](https://github.com/Cyan4973)
15
+ ## FEATURES (機能)
16
+
17
+ * Generic LZ4 frame data process (for `.lz4` file format)
18
+ * Decode LZ4 Frame data : `LZ4.decode`
19
+ * Encode LZ4 Frame data : `LZ4.encode`
20
+ * Generic LZ4 frame data file process (for `.lz4` file format)
21
+ * Decode LZ4 Frame data file : `LZ4.decode_file`
22
+ * Encode LZ4 Frame data file : `LZ4.encode_file`
23
+ * LZ4 block data process
24
+ * Decode LZ4 block data : `LZ4.block_decode`
25
+ * Encode LZ4 block data : `LZ4.block_encode` (supporting high compression level)
26
+ * Streaming Decode LZ4 block data : `LZ4.block_stream_decode` and `LZ4::BlockDecoder#update`
27
+ * Streaming Encode LZ4 block data : `LZ4.block_stream_encode` and `LZ4::BlockEncoder#update` (supporting high compression level)
28
+
29
+ See [Quick reference](QUICKREF.md) for more details.
30
30
 
31
31
 
32
32
  ## ATTENTIONS (注意事項)
@@ -59,21 +59,6 @@ extlz4-0.1.1 でその不具合の修正を行いました。
59
59
  修復できるのはあくまで extlz4-0.1 のこの不具合に起因するファイルのみとなります。
60
60
 
61
61
 
62
- ## FEATURES (機能)
63
-
64
- * Generic LZ4 frame data process
65
- * Decode LZ4 Frame data : LZ4.decode
66
- * Encode LZ4 Frame data : LZ4.encode
67
- * Generic LZ4 frame data file process
68
- * Decode LZ4 Frame data file : LZ4.decode\_file
69
- * Encode LZ4 Frame data file : LZ4.encode\_file
70
- * LZ4 block data process
71
- * Decode LZ4 block data : LZ4.block\_decode
72
- * Encode LZ4 block data : LZ4.block\_encode (supporting high compression level)
73
- * Streaming Decode LZ4 block data : LZ4.block\_stream\_decode and LZ4::BlockDecoder#update
74
- * Streaming Encode LZ4 block data : LZ4.block\_stream\_encode and LZ4::BlockEncoder#update (supporting high compression level)
75
-
76
-
77
62
  ## ABOUT TAINT STATE AND SECURITY (汚染状態とセキュリティについて)
78
63
 
79
64
  extlz4 はオブジェクト間での汚染状態を一方向伝播します。
@@ -92,25 +77,25 @@ First, load extlz4. (最初に extlz4 を読み込んでください)
92
77
  require "extlz4"
93
78
  ```
94
79
 
95
- ### Decoding (伸張処理)
80
+ ### One shot decoding from LZ4 Frame (LZ4 Frame 伸張処理)
96
81
 
97
82
  ``` ruby:ruby
98
83
  uncompressed_data_string = LZ4.decode(compressed_data_string)
99
84
  ```
100
85
 
101
- ### Encoding (通常圧縮処理)
86
+ ### One shot encoding to LZ4 Frame (LZ4 Frame 通常圧縮処理)
102
87
 
103
88
  ``` ruby:ruby
104
89
  compressed_data_string = LZ4.encode(uncompressed_data_string)
105
90
  ```
106
91
 
107
- ### High compression encoding (高圧縮処理)
92
+ ### One shot high compression encoding to LZ4 Frame (LZ4 Frame 高圧縮処理)
108
93
 
109
94
  ``` ruby:ruby
110
95
  compressed_data_string = LZ4.encode(uncompressed_data_string, 9)
111
96
  ```
112
97
 
113
- ### Frame decoding
98
+ ### Stream decoding to LZ4 Frame
114
99
 
115
100
  ``` ruby:ruby
116
101
  File.open("sample.txt.lz4", "rb") do |file|
@@ -122,7 +107,7 @@ File.open("sample.txt.lz4", "rb") do |file|
122
107
  end
123
108
  ```
124
109
 
125
- ### Frame encoding by high compression
110
+ ### Stream encoding by high compression to LZ4 Frame
126
111
 
127
112
  ``` ruby:ruby
128
113
  File.open("sample.txt.lz4", "wb") do |file|
@@ -133,7 +118,7 @@ File.open("sample.txt.lz4", "wb") do |file|
133
118
  end
134
119
  ```
135
120
 
136
- ### Frame encoding without block
121
+ ### Stream encoding without block to LZ4 Frame
137
122
 
138
123
  ``` ruby:ruby
139
124
  file = File.open("sample.txt.lz4", "wb")
@@ -142,7 +127,7 @@ lz4 << "abcdefghijklmnopqrstuvwxyz\n"
142
127
  lz4.close # VERY IMPORTANT!
143
128
  ```
144
129
 
145
- ### Block data processing (fast compression encoding and decoding)
130
+ ### One shot block data processing to/from LZ4 Block (fast compression encoding and decoding)
146
131
 
147
132
  ``` ruby:ruby
148
133
  src = "abcdefg" * 100
@@ -151,7 +136,7 @@ data = LZ4.block_decode(lz4data)
151
136
  p src == data # => true
152
137
  ```
153
138
 
154
- ### Block data processing (high compression encoding and decoding)
139
+ ### One shot block data processing to/from LZ4 Block (high compression encoding and decoding)
155
140
 
156
141
  ``` ruby:ruby
157
142
  src = "abcdefg" * 100
@@ -161,7 +146,7 @@ data = LZ4.block_decode(lz4data)
161
146
  p src == data # => true
162
147
  ```
163
148
 
164
- ### Block data processing (high speed encoding)
149
+ ### One shot block data processing to/from LZ4 Block (high speed encoding)
165
150
 
166
151
  ``` ruby:ruby
167
152
  src = "abcdefg" * 100
@@ -169,7 +154,7 @@ level = -19 # transform to one's complement as acceleration
169
154
  lz4data = LZ4.block_encode(level, src)
170
155
  ```
171
156
 
172
- ### Block stream data processing (high compression encoding and decoding)
157
+ ### Block stream data processing to/from LZ4 Block (high compression encoding and decoding)
173
158
 
174
159
  ``` ruby:ruby
175
160
  level = 8 # (OPTIONAL PARAMETER)
@@ -194,6 +179,11 @@ p src2 == data # => true
194
179
  ```
195
180
 
196
181
 
182
+ ## Support `Ractor` with Ruby3
183
+
184
+ Ruby3 で追加された `Ractor` に対応しています。
185
+
186
+
197
187
  ## BONUS (おまけ)
198
188
 
199
189
  コマンドラインプログラムとして ``extlz4`` が追加されます。
@@ -201,3 +191,21 @@ p src2 == data # => true
201
191
  これは lz4 と同程度の機能を持ちます (車輪の再発明とも言う)。
202
192
 
203
193
  とはいえ、引数のとり方を変えてあり、gzip のような形で利用できます。
194
+
195
+
196
+ ## SPECIFICATION (仕様)
197
+
198
+ - package name: extlz4
199
+ - author: dearblue (mailto:dearblue@users.osdn.me)
200
+ - project page: <https://github.com/dearblue/ruby-extlz4>
201
+ - how to install: `gem install extlz4`
202
+ - version: 0.3.2
203
+ - product quality: technical preview
204
+ - licensing: [2 clause BSD License](LICENSE)
205
+ - dependency gems: none
206
+ - dependency external c libraries: none
207
+ - bundled external c libraries (git submodules):
208
+ - [lz4](https://github.com/lz4/lz4)
209
+ [version 1.9.3](https://github.com/lz4/lz4/tree/v1.9.3)
210
+ under [2 clause BSD license](https://github.com/lz4/lz4/blob/v1.9.3/LICENSE)
211
+ by [Yann Collet](https://github.com/Cyan4973)
data/bin/extlz4 CHANGED
@@ -1,7 +1,7 @@
1
1
  #!ruby
2
2
  #vim: set fileencoding:utf-8
3
3
 
4
- # AUTHOR: dearblue <dearblue@users.noreply.github.com>
4
+ # AUTHOR: dearblue <dearblue@users.osdn.me>
5
5
  # Only this program is usable as public domain software.
6
6
 
7
7
  require "extlz4"
data/contrib/lz4/INSTALL CHANGED
@@ -8,6 +8,7 @@ make install # this command may require root access
8
8
 
9
9
  LZ4's `Makefile` supports standard [Makefile conventions],
10
10
  including [staged installs], [redirection], or [command redefinition].
11
+ It is compatible with parallel builds (`-j#`).
11
12
 
12
13
  [Makefile conventions]: https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html
13
14
  [staged installs]: https://www.gnu.org/prep/standards/html_node/DESTDIR.html
@@ -0,0 +1,87 @@
1
+ ifeq ($(V), 1)
2
+ Q =
3
+ else
4
+ Q = @
5
+ endif
6
+
7
+ TARGET_OS ?= $(shell uname)
8
+ ifeq ($(TARGET_OS),)
9
+ TARGET_OS ?= $(OS)
10
+ endif
11
+
12
+ ifneq (,$(filter Windows%,$(TARGET_OS)))
13
+ LIBLZ4 = liblz4-$(LIBVER_MAJOR)
14
+ LIBLZ4_EXP = liblz4.lib
15
+ WINBASED = yes
16
+ else
17
+ LIBLZ4_EXP = liblz4.dll.a
18
+ ifneq (,$(filter MINGW%,$(TARGET_OS)))
19
+ LIBLZ4 = liblz4
20
+ WINBASED = yes
21
+ else
22
+ ifneq (,$(filter MSYS%,$(TARGET_OS)))
23
+ LIBLZ4 = msys-lz4-$(LIBVER_MAJOR)
24
+ WINBASED = yes
25
+ else
26
+ ifneq (,$(filter CYGWIN%,$(TARGET_OS)))
27
+ LIBLZ4 = cyglz4-$(LIBVER_MAJOR)
28
+ WINBASED = yes
29
+ else
30
+ LIBLZ4 = liblz4.$(SHARED_EXT_VER)
31
+ WINBASED = no
32
+ EXT =
33
+ endif
34
+ endif
35
+ endif
36
+ endif
37
+
38
+ ifeq ($(WINBASED),yes)
39
+ EXT = .exe
40
+ WINDRES = windres
41
+ endif
42
+
43
+ #determine if dev/nul based on host environment
44
+ ifneq (,$(filter MINGW% MSYS% CYGWIN%,$(shell uname)))
45
+ VOID := /dev/null
46
+ else
47
+ ifneq (,$(filter Windows%,$(OS)))
48
+ VOID := nul
49
+ else
50
+ VOID := /dev/null
51
+ endif
52
+ endif
53
+
54
+ ifneq (,$(filter Linux Darwin GNU/kFreeBSD GNU OpenBSD FreeBSD NetBSD DragonFly SunOS Haiku MidnightBSD MINGW% CYGWIN% MSYS%,$(shell uname)))
55
+ POSIX_ENV = Yes
56
+ else
57
+ POSIX_ENV = No
58
+ endif
59
+
60
+ # Avoid symlinks when targetting Windows or building on a Windows host
61
+ ifeq ($(WINBASED),yes)
62
+ LN_S = cp -p
63
+ LN_SF = cp -p
64
+ else
65
+ ifneq (,$(filter MINGW% MSYS% CYGWIN%,$(shell uname)))
66
+ LN_S = cp -p
67
+ LN_SF = cp -p
68
+ else
69
+ ifneq (,$(filter Windows%,$(OS)))
70
+ LN_S = cp -p
71
+ LN_SF = cp -p
72
+ else
73
+ LN_S = ln -s
74
+ LN_SF = ln -sf
75
+ endif
76
+ endif
77
+ endif
78
+
79
+ ifneq (,$(filter $(shell uname),SunOS))
80
+ INSTALL ?= ginstall
81
+ else
82
+ INSTALL ?= install
83
+ endif
84
+
85
+ INSTALL_PROGRAM ?= $(INSTALL) -m 755
86
+ INSTALL_DATA ?= $(INSTALL) -m 644
87
+ INSTALL_DIR ?= $(INSTALL) -d -m 755
data/contrib/lz4/NEWS CHANGED
@@ -1,3 +1,92 @@
1
+ v1.9.3
2
+ perf: highly improved speed in kernel space, by @terrelln
3
+ perf: faster speed with Visual Studio, thanks to @wolfpld and @remittor
4
+ perf: improved dictionary compression speed, by @felixhandte
5
+ perf: fixed LZ4_compress_HC_destSize() ratio, detected by @hsiangkao
6
+ perf: reduced stack usage in high compression mode, by @Yanpas
7
+ api : LZ4_decompress_safe_partial() supports unknown compressed size, requested by @jfkthame
8
+ api : improved LZ4F_compressBound() with automatic flushing, by Christopher Harvie
9
+ api : can (de)compress to/from NULL without UBs
10
+ api : fix alignment test on 32-bit systems (state initialization)
11
+ api : fix LZ4_saveDictHC() in corner case scenario, detected by @IgorKorkin
12
+ cli : `-l` legacy format is now compatible with `-m` multiple files, by Filipe Calasans
13
+ cli : benchmark mode supports dictionary, by @rkoradi
14
+ cli : fix --fast with large argument, detected by @picoHz
15
+ build: link to user-defined memory functions with LZ4_USER_MEMORY_FUNCTIONS, suggested by Yuriy Levchenko
16
+ build: contrib/cmake_unofficial/ moved to build/cmake/
17
+ build: visual/* moved to build/
18
+ build: updated meson script, by @neheb
19
+ build: tinycc support, by Anton Kochkov
20
+ install: Haiku support, by Jerome Duval
21
+ doc : updated LZ4 frame format, clarify EndMark
22
+
23
+ v1.9.2
24
+ fix : out-of-bound read in exceptional circumstances when using decompress_partial(), by @terrelln
25
+ fix : slim opportunity for out-of-bound write with compress_fast() with a large enough input and when providing an output smaller than recommended (< LZ4_compressBound(inputSize)), by @terrelln
26
+ fix : rare data corruption bug with LZ4_compress_destSize(), by @terrelln
27
+ fix : data corruption bug when Streaming with an Attached Dict in HC Mode, by @felixhandte
28
+ perf: enable LZ4_FAST_DEC_LOOP on aarch64/GCC by default, by @prekageo
29
+ perf: improved lz4frame streaming API speed, by @dreambottle
30
+ perf: speed up lz4hc on slow patterns when using external dictionary, by @terrelln
31
+ api: better in-place decompression and compression support
32
+ cli : --list supports multi-frames files, by @gstedman
33
+ cli: --version outputs to stdout
34
+ cli : add option --best as an alias of -12 , by @Low-power
35
+ misc: Integration into oss-fuzz by @cmeister2, expanded list of scenarios by @terrelln
36
+
37
+ v1.9.1
38
+ fix : decompression functions were reading a few bytes beyond input size (introduced in v1.9.0, reported by @ppodolsky and @danlark1)
39
+ api : fix : lz4frame initializers compatibility with c++, reported by @degski
40
+ cli : added command --list, based on a patch by @gabrielstedman
41
+ build: improved Windows build, by @JPeterMugaas
42
+ build: AIX, by Norman Green
43
+
44
+ v1.9.0
45
+ perf: large decompression speed improvement on x86/x64 (up to +20%) by @djwatson
46
+ api : changed : _destSize() compression variants are promoted to stable API
47
+ api : new : LZ4_initStream(HC), replacing LZ4_resetStream(HC)
48
+ api : changed : LZ4_resetStream(HC) as recommended reset function, for better performance on small data
49
+ cli : support custom block sizes, by @blezsan
50
+ build: source code can be amalgamated, by Bing Xu
51
+ build: added meson build, by @lzutao
52
+ build: new build macros : LZ4_DISTANCE_MAX, LZ4_FAST_DEC_LOOP
53
+ install: MidnightBSD, by @laffer1
54
+ install: msys2 on Windows 10, by @vtorri
55
+
56
+ v1.8.3
57
+ perf: minor decompression speed improvement (~+2%) with gcc
58
+ fix : corruption in v1.8.2 at level 9 for files > 64KB under rare conditions (#560)
59
+ cli : new command --fast, by @jennifermliu
60
+ cli : fixed elapsed time, and added cpu load indicator (on -vv) (#555)
61
+ api : LZ4_decompress_safe_partial() now decodes exactly the nb of bytes requested (feature request #566)
62
+ build : added Haiku target, by @fbrosson, and MidnightBSD, by @laffer1
63
+ doc : updated documentation regarding dictionary compression
64
+
65
+ v1.8.2
66
+ perf: *much* faster dictionary compression on small files, by @felixhandte
67
+ perf: improved decompression speed and binary size, by Alexey Tourbin (@svpv)
68
+ perf: slightly faster HC compression and decompression speed
69
+ perf: very small compression ratio improvement
70
+ fix : compression compatible with low memory addresses (< 0xFFFF)
71
+ fix : decompression segfault when provided with NULL input, by @terrelln
72
+ cli : new command --favor-decSpeed
73
+ cli : benchmark mode more accurate for small inputs
74
+ fullbench : can bench _destSize() variants, by @felixhandte
75
+ doc : clarified block format parsing restrictions, by Alexey Tourbin (@svpv)
76
+
77
+ v1.8.1
78
+ perf : faster and stronger ultra modes (levels 10+)
79
+ perf : slightly faster compression and decompression speed
80
+ perf : fix bad degenerative case, reported by @c-morgenstern
81
+ fix : decompression failed when using a combination of extDict + low memory address (#397), reported and fixed by Julian Scheid (@jscheid)
82
+ cli : support for dictionary compression (`-D`), by Felix Handte @felixhandte
83
+ cli : fix : `lz4 -d --rm` preserves timestamp (#441)
84
+ cli : fix : do not modify /dev/null permission as root, by @aliceatlas
85
+ api : `_destSize()` variant supported for all compression levels
86
+ build : `make` and `make test` compatible with `-jX`, reported by @mwgamera
87
+ build : can control LZ4LIB_VISIBILITY macro, by @mikir
88
+ install: fix man page directory (#387), reported by Stuart Cardall (@itoffshore)
89
+
1
90
  v1.8.0
2
91
  cli : fix : do not modify /dev/null permissions, reported by @Maokaman1
3
92
  cli : added GNU separator -- specifying that all following arguments are files
@@ -2,74 +2,68 @@ LZ4 - Extremely fast compression
2
2
  ================================
3
3
 
4
4
  LZ4 is lossless compression algorithm,
5
- providing compression speed at 400 MB/s per core,
5
+ providing compression speed > 500 MB/s per core,
6
6
  scalable with multi-cores CPU.
7
7
  It features an extremely fast decoder,
8
8
  with speed in multiple GB/s per core,
9
9
  typically reaching RAM speed limits on multi-core systems.
10
10
 
11
11
  Speed can be tuned dynamically, selecting an "acceleration" factor
12
- which trades compression ratio for more speed up.
12
+ which trades compression ratio for faster speed.
13
13
  On the other end, a high compression derivative, LZ4_HC, is also provided,
14
14
  trading CPU time for improved compression ratio.
15
15
  All versions feature the same decompression speed.
16
16
 
17
+ LZ4 is also compatible with [dictionary compression](https://github.com/facebook/zstd#the-case-for-small-data-compression),
18
+ both at [API](https://github.com/lz4/lz4/blob/v1.8.3/lib/lz4frame.h#L481) and [CLI](https://github.com/lz4/lz4/blob/v1.8.3/programs/lz4.1.md#operation-modifiers) levels.
19
+ It can ingest any input file as dictionary, though only the final 64KB are used.
20
+ This capability can be combined with the [Zstandard Dictionary Builder](https://github.com/facebook/zstd/blob/v1.3.5/programs/zstd.1.md#dictionary-builder),
21
+ in order to drastically improve compression performance on small files.
22
+
23
+
17
24
  LZ4 library is provided as open-source software using BSD 2-Clause license.
18
25
 
19
26
 
20
27
  |Branch |Status |
21
28
  |------------|---------|
22
- |master | [![Build Status][travisMasterBadge]][travisLink] [![Build status][AppveyorMasterBadge]][AppveyorLink] [![coverity][coverBadge]][coverlink] |
23
29
  |dev | [![Build Status][travisDevBadge]][travisLink] [![Build status][AppveyorDevBadge]][AppveyorLink] |
24
30
 
25
- [travisMasterBadge]: https://travis-ci.org/lz4/lz4.svg?branch=master "Continuous Integration test suite"
26
31
  [travisDevBadge]: https://travis-ci.org/lz4/lz4.svg?branch=dev "Continuous Integration test suite"
27
32
  [travisLink]: https://travis-ci.org/lz4/lz4
28
- [AppveyorMasterBadge]: https://ci.appveyor.com/api/projects/status/github/lz4/lz4?branch=master&svg=true "Windows test suite"
29
33
  [AppveyorDevBadge]: https://ci.appveyor.com/api/projects/status/github/lz4/lz4?branch=dev&svg=true "Windows test suite"
30
34
  [AppveyorLink]: https://ci.appveyor.com/project/YannCollet/lz4-1lndh
31
- [coverBadge]: https://scan.coverity.com/projects/4735/badge.svg "Static code analysis of Master branch"
32
- [coverlink]: https://scan.coverity.com/projects/4735
33
35
 
34
- > **Branch Policy:**
35
- > - The "master" branch is considered stable, at all times.
36
- > - The "dev" branch is the one where all contributions must be merged
37
- before being promoted to master.
38
- > + If you plan to propose a patch, please commit into the "dev" branch,
39
- or its own feature branch.
40
- Direct commit to "master" are not permitted.
41
36
 
42
37
  Benchmarks
43
38
  -------------------------
44
39
 
45
40
  The benchmark uses [lzbench], from @inikep
46
- compiled with GCC v6.2.0 on Linux 64-bits.
47
- The reference system uses a Core i7-3930K CPU @ 4.5GHz.
41
+ compiled with GCC v8.2.0 on Linux 64-bits (Ubuntu 4.18.0-17).
42
+ The reference system uses a Core i7-9700K CPU @ 4.9GHz (w/ turbo boost).
48
43
  Benchmark evaluates the compression of reference [Silesia Corpus]
49
44
  in single-thread mode.
50
45
 
51
46
  [lzbench]: https://github.com/inikep/lzbench
52
47
  [Silesia Corpus]: http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia
53
48
 
54
- | Compressor | Ratio | Compression | Decompression |
55
- | ---------- | ----- | ----------- | ------------- |
56
- | memcpy | 1.000 | 7300 MB/s | 7300 MB/s |
57
- |**LZ4 fast 8 (v1.7.3)**| 1.799 |**911 MB/s** | **3360 MB/s** |
58
- |**LZ4 default (v1.7.3)**|**2.101**|**625 MB/s** | **3220 MB/s** |
59
- | LZO 2.09 | 2.108 | 620 MB/s | 845 MB/s |
60
- | QuickLZ 1.5.0 | 2.238 | 510 MB/s | 600 MB/s |
61
- | Snappy 1.1.3 | 2.091 | 450 MB/s | 1550 MB/s |
62
- | LZF v3.6 | 2.073 | 365 MB/s | 820 MB/s |
63
- | [Zstandard] 1.1.1 -1 | 2.876 | 330 MB/s | 930 MB/s |
64
- | [Zstandard] 1.1.1 -3 | 3.164 | 200 MB/s | 810 MB/s |
65
- | [zlib] deflate 1.2.8 -1| 2.730 | 100 MB/s | 370 MB/s |
66
- |**LZ4 HC -9 (v1.7.3)** |**2.720**| 34 MB/s | **3240 MB/s** |
67
- | [zlib] deflate 1.2.8 -6| 3.099 | 33 MB/s | 390 MB/s |
49
+ | Compressor | Ratio | Compression | Decompression |
50
+ | ---------- | ----- | ----------- | ------------- |
51
+ | memcpy | 1.000 | 13700 MB/s | 13700 MB/s |
52
+ |**LZ4 default (v1.9.0)** |**2.101**| **780 MB/s**| **4970 MB/s** |
53
+ | LZO 2.09 | 2.108 | 670 MB/s | 860 MB/s |
54
+ | QuickLZ 1.5.0 | 2.238 | 575 MB/s | 780 MB/s |
55
+ | Snappy 1.1.4 | 2.091 | 565 MB/s | 1950 MB/s |
56
+ | [Zstandard] 1.4.0 -1 | 2.883 | 515 MB/s | 1380 MB/s |
57
+ | LZF v3.6 | 2.073 | 415 MB/s | 910 MB/s |
58
+ | [zlib] deflate 1.2.11 -1| 2.730 | 100 MB/s | 415 MB/s |
59
+ |**LZ4 HC -9 (v1.9.0)** |**2.721**| 41 MB/s | **4900 MB/s** |
60
+ | [zlib] deflate 1.2.11 -6| 3.099 | 36 MB/s | 445 MB/s |
68
61
 
69
62
  [zlib]: http://www.zlib.net/
70
63
  [Zstandard]: http://www.zstd.net/
71
64
 
72
- LZ4 is also compatible and well optimized for x32 mode, for which it provides an additional +10% speed performance.
65
+ LZ4 is also compatible and optimized for x32 mode,
66
+ for which it provides additional speed performance.
73
67
 
74
68
 
75
69
  Installation
@@ -77,27 +71,39 @@ Installation
77
71
 
78
72
  ```
79
73
  make
80
- make install # this command may require root access
74
+ make install # this command may require root permissions
81
75
  ```
82
76
 
83
77
  LZ4's `Makefile` supports standard [Makefile conventions],
84
78
  including [staged installs], [redirection], or [command redefinition].
79
+ It is compatible with parallel builds (`-j#`).
85
80
 
86
81
  [Makefile conventions]: https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html
87
82
  [staged installs]: https://www.gnu.org/prep/standards/html_node/DESTDIR.html
88
83
  [redirection]: https://www.gnu.org/prep/standards/html_node/Directory-Variables.html
89
84
  [command redefinition]: https://www.gnu.org/prep/standards/html_node/Utilities-in-Makefiles.html
90
85
 
86
+ Building LZ4 - Using vcpkg
87
+
88
+ You can download and install LZ4 using the [vcpkg](https://github.com/Microsoft/vcpkg) dependency manager:
89
+
90
+ git clone https://github.com/Microsoft/vcpkg.git
91
+ cd vcpkg
92
+ ./bootstrap-vcpkg.sh
93
+ ./vcpkg integrate install
94
+ vcpkg install lz4
95
+
96
+ The LZ4 port in vcpkg is kept up to date by Microsoft team members and community contributors. If the version is out of date, please [create an issue or pull request](https://github.com/Microsoft/vcpkg) on the vcpkg repository.
91
97
 
92
98
  Documentation
93
99
  -------------------------
94
100
 
95
101
  The raw LZ4 block compression format is detailed within [lz4_Block_format].
96
102
 
97
- To compress an arbitrarily long file or data stream, multiple blocks are required.
98
- Organizing these blocks and providing a common header format to handle their content
99
- is the purpose of the Frame format, defined into [lz4_Frame_format].
100
- Interoperable versions of LZ4 must respect this frame format.
103
+ Arbitrarily long files or data streams are compressed using multiple blocks,
104
+ for streaming requirements. These blocks are organized into a frame,
105
+ defined into [lz4_Frame_format].
106
+ Interoperable versions of LZ4 must also respect the frame format.
101
107
 
102
108
  [lz4_Block_format]: doc/lz4_Block_format.md
103
109
  [lz4_Frame_format]: doc/lz4_Frame_format.md