extlz4 0.2.4.2 → 0.3.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (79) hide show
  1. checksums.yaml +5 -5
  2. data/HISTORY.ja.md +25 -0
  3. data/README.md +49 -41
  4. data/bin/extlz4 +1 -1
  5. data/contrib/lz4/INSTALL +1 -0
  6. data/contrib/lz4/Makefile.inc +87 -0
  7. data/contrib/lz4/NEWS +89 -0
  8. data/contrib/lz4/README.md +42 -36
  9. data/contrib/lz4/build/README.md +55 -0
  10. data/contrib/lz4/build/VS2010/datagen/datagen.vcxproj +169 -0
  11. data/contrib/lz4/build/VS2010/frametest/frametest.vcxproj +176 -0
  12. data/contrib/lz4/build/VS2010/fullbench-dll/fullbench-dll.vcxproj +180 -0
  13. data/contrib/lz4/build/VS2010/fullbench/fullbench.vcxproj +176 -0
  14. data/contrib/lz4/build/VS2010/fuzzer/fuzzer.vcxproj +173 -0
  15. data/contrib/lz4/build/VS2010/liblz4-dll/liblz4-dll.rc +51 -0
  16. data/contrib/lz4/build/VS2010/liblz4-dll/liblz4-dll.vcxproj +179 -0
  17. data/contrib/lz4/build/VS2010/liblz4/liblz4.vcxproj +175 -0
  18. data/contrib/lz4/build/VS2010/lz4.sln +98 -0
  19. data/contrib/lz4/build/VS2010/lz4/lz4.rc +51 -0
  20. data/contrib/lz4/build/VS2010/lz4/lz4.vcxproj +189 -0
  21. data/contrib/lz4/build/VS2017/datagen/datagen.vcxproj +173 -0
  22. data/contrib/lz4/build/VS2017/frametest/frametest.vcxproj +180 -0
  23. data/contrib/lz4/build/VS2017/fullbench-dll/fullbench-dll.vcxproj +184 -0
  24. data/contrib/lz4/build/VS2017/fullbench/fullbench.vcxproj +180 -0
  25. data/contrib/lz4/build/VS2017/fuzzer/fuzzer.vcxproj +177 -0
  26. data/contrib/lz4/build/VS2017/liblz4-dll/liblz4-dll.rc +51 -0
  27. data/contrib/lz4/build/VS2017/liblz4-dll/liblz4-dll.vcxproj +183 -0
  28. data/contrib/lz4/build/VS2017/liblz4/liblz4.vcxproj +179 -0
  29. data/contrib/lz4/build/VS2017/lz4.sln +103 -0
  30. data/contrib/lz4/build/VS2017/lz4/lz4.rc +51 -0
  31. data/contrib/lz4/build/VS2017/lz4/lz4.vcxproj +164 -0
  32. data/contrib/lz4/build/cmake/CMakeLists.txt +235 -0
  33. data/contrib/lz4/lib/README.md +98 -34
  34. data/contrib/lz4/lib/liblz4-dll.rc.in +35 -0
  35. data/contrib/lz4/lib/lz4.c +1698 -681
  36. data/contrib/lz4/lib/lz4.h +546 -235
  37. data/contrib/lz4/lib/lz4frame.c +608 -378
  38. data/contrib/lz4/lib/lz4frame.h +315 -83
  39. data/contrib/lz4/lib/lz4frame_static.h +4 -100
  40. data/contrib/lz4/lib/lz4hc.c +1090 -282
  41. data/contrib/lz4/lib/lz4hc.h +276 -141
  42. data/contrib/lz4/lib/xxhash.c +371 -235
  43. data/contrib/lz4/lib/xxhash.h +128 -93
  44. data/contrib/lz4/ossfuzz/Makefile +78 -0
  45. data/contrib/lz4/ossfuzz/compress_frame_fuzzer.c +48 -0
  46. data/contrib/lz4/ossfuzz/compress_fuzzer.c +58 -0
  47. data/contrib/lz4/ossfuzz/compress_hc_fuzzer.c +64 -0
  48. data/contrib/lz4/ossfuzz/decompress_frame_fuzzer.c +75 -0
  49. data/contrib/lz4/ossfuzz/decompress_fuzzer.c +62 -0
  50. data/contrib/lz4/ossfuzz/fuzz.h +48 -0
  51. data/contrib/lz4/ossfuzz/fuzz_data_producer.c +77 -0
  52. data/contrib/lz4/ossfuzz/fuzz_data_producer.h +36 -0
  53. data/contrib/lz4/ossfuzz/fuzz_helpers.h +94 -0
  54. data/contrib/lz4/ossfuzz/lz4_helpers.c +51 -0
  55. data/contrib/lz4/ossfuzz/lz4_helpers.h +13 -0
  56. data/contrib/lz4/ossfuzz/ossfuzz.sh +23 -0
  57. data/contrib/lz4/ossfuzz/round_trip_frame_fuzzer.c +43 -0
  58. data/contrib/lz4/ossfuzz/round_trip_fuzzer.c +57 -0
  59. data/contrib/lz4/ossfuzz/round_trip_hc_fuzzer.c +44 -0
  60. data/contrib/lz4/ossfuzz/round_trip_stream_fuzzer.c +302 -0
  61. data/contrib/lz4/ossfuzz/standaloneengine.c +74 -0
  62. data/contrib/lz4/ossfuzz/travisoss.sh +26 -0
  63. data/contrib/lz4/tmp +0 -0
  64. data/contrib/lz4/tmpsparse +0 -0
  65. data/ext/blockapi.c +5 -5
  66. data/ext/extlz4.c +2 -0
  67. data/ext/extlz4.h +5 -0
  68. data/ext/frameapi.c +1 -1
  69. data/ext/hashargs.c +2 -2
  70. data/ext/hashargs.h +1 -1
  71. data/ext/lz4_amalgam.c +0 -23
  72. data/gemstub.rb +5 -16
  73. data/lib/extlz4.rb +51 -3
  74. data/lib/extlz4/oldstream.rb +1 -1
  75. data/test/common.rb +2 -2
  76. metadata +73 -16
  77. data/contrib/lz4/circle.yml +0 -39
  78. data/contrib/lz4/lib/lz4opt.h +0 -366
  79. data/lib/extlz4/version.rb +0 -3
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: 79289896ae954100abb7a0a20312abefc1206134
4
- data.tar.gz: 2328b0e92f682b0ec3b006eb88e70cd221eda360
2
+ SHA256:
3
+ metadata.gz: f1060d0000526f937ddf36e247c7e2041fbdf7241382d38a88a85bd39738d958
4
+ data.tar.gz: 61e5321caa13e409b8bd1af20429c184fca8fcc87f73ce31b1cbd4caf1acde24
5
5
  SHA512:
6
- metadata.gz: 4cede1dccaf6955f460f8e458dadf0dc8f27ad11a76dcb5438086320d79bc5f159ea2f9b183224110a89c93061e364feb35053bc938e60881ced44e44d5d4b95
7
- data.tar.gz: 2e867e0857b9c253c3e4260e31913685429eadbe9785db89576dc556fcb184507ffd5f8b095e5fecc754ae57f1d40f28fbb2a401d459be09b1e93e1dd24ba614
6
+ metadata.gz: 487f4884327c7111ad64877190fb07a25a38ff1de84b1544ec131a238d6b1f617daf89d6ff5a9776ec05783d78139c41c481f8b2fd7b2dfba56c92019fbce413
7
+ data.tar.gz: 00c9a7ccdec52f3a7f60aeae9626af3f83ef8a7a8a456fe267bb4244e930fe977d2c9af72137e2e6fb1cc6906b5ab7a84eb01a393c03e69cc83d393284eae710
data/HISTORY.ja.md CHANGED
@@ -1,3 +1,28 @@
1
+ # extlz4-0.3.1 (令和2年6月7日)
2
+
3
+ * lz4-1.9.2 に更新
4
+ * `extlz4.so` が `<extlz4-dir>/lib` へ置かれない環境に対応
5
+ * https://github.com/dearblue/ruby-extlz4/issues/2
6
+
7
+
8
+ # extlz4-0.3 (平成31年4月)
9
+
10
+ * lz4-1.9.0 に更新
11
+ * `LZ4` モジュールをリファインメントとして利用できるように機能を追加
12
+ * `Object` および `String` クラスで `.to_lz4frame` `.unlz4frame` メソッドが利用できるようになりました。
13
+ * `String` クラスで `.to_lz4block` `.unlz4block` メソッドが利用できるようになりました。
14
+
15
+
16
+ # extlz4-0.2.5 (平成30年1月14日 日曜日)
17
+
18
+ * lz4 ライブラリを [1.8.1](https://github.com/lz4/lz4/releases/tag/v1.8.1) に更新
19
+
20
+
21
+ # extlz4-0.2.4.3 (平成30年1月13日 土曜日)
22
+
23
+ * LZ4::Decoder.read でバッファ領域が確保されていなかった問題を修正
24
+
25
+
1
26
  # extlz4-0.2.4.2 (平成30年1月7日 日曜日)
2
27
 
3
28
  * バージョン番号の改定のみ
data/README.md CHANGED
@@ -5,28 +5,28 @@
5
5
 
6
6
  LZ4 データストリームを圧縮・伸張できます。lz4-cli で扱うことが出来ます。
7
7
 
8
- ``` shell:shell
8
+ ```shell:shell
9
9
  $ dmesg | ruby -r extlz4 -e 'LZ4.encode_file($stdin.binmode, $stdout.binmode)' | lz4c -d | more
10
10
  ```
11
11
 
12
12
  ほかの ruby 向けの lz4 バインディングライブラリとしては KOMIYA Atsushi さんによる [lz4-ruby (http://rubygems.org/gems/lz4-ruby)](http://rubygems.org/gems/lz4-ruby) があります。
13
13
 
14
14
 
15
- ## SUMMARY (概要)
16
-
17
- * package name: extlz4
18
- * author: dearblue (mailto:dearblue@users.noreply.github.com)
19
- * report issue to: <https://github.com/dearblue/ruby-extlz4/issues>
20
- * how to install: `gem install extlz4`
21
- * version: 0.2.4.2
22
- * product quality: technical preview
23
- * licensing: BSD-2-Clause License
24
- * dependency gems: none
25
- * dependency external c libraries: none
26
- * bundled external c libraries:
27
- * lz4-1.8 <https://github.com/lz4/lz4/tree/v1.8.0>
28
- under [BSD 2-Clause license](https://github.com/lz4/lz4/tree/v1.8.0/LICENSE)
29
- by [Yann Collet](https://github.com/Cyan4973)
15
+ ## FEATURES (機能)
16
+
17
+ * Generic LZ4 frame data process (for `.lz4` file format)
18
+ * Decode LZ4 Frame data : `LZ4.decode`
19
+ * Encode LZ4 Frame data : `LZ4.encode`
20
+ * Generic LZ4 frame data file process (for `.lz4` file format)
21
+ * Decode LZ4 Frame data file : `LZ4.decode_file`
22
+ * Encode LZ4 Frame data file : `LZ4.encode_file`
23
+ * LZ4 block data process
24
+ * Decode LZ4 block data : `LZ4.block_decode`
25
+ * Encode LZ4 block data : `LZ4.block_encode` (supporting high compression level)
26
+ * Streaming Decode LZ4 block data : `LZ4.block_stream_decode` and `LZ4::BlockDecoder#update`
27
+ * Streaming Encode LZ4 block data : `LZ4.block_stream_encode` and `LZ4::BlockEncoder#update` (supporting high compression level)
28
+
29
+ See [Quick reference](QUICKREF.md) for more details.
30
30
 
31
31
 
32
32
  ## ATTENTIONS (注意事項)
@@ -59,21 +59,6 @@ extlz4-0.1.1 でその不具合の修正を行いました。
59
59
  修復できるのはあくまで extlz4-0.1 のこの不具合に起因するファイルのみとなります。
60
60
 
61
61
 
62
- ## FEATURES (機能)
63
-
64
- * Generic LZ4 frame data process
65
- * Decode LZ4 Frame data : LZ4.decode
66
- * Encode LZ4 Frame data : LZ4.encode
67
- * Generic LZ4 frame data file process
68
- * Decode LZ4 Frame data file : LZ4.decode\_file
69
- * Encode LZ4 Frame data file : LZ4.encode\_file
70
- * LZ4 block data process
71
- * Decode LZ4 block data : LZ4.block\_decode
72
- * Encode LZ4 block data : LZ4.block\_encode (supporting high compression level)
73
- * Streaming Decode LZ4 block data : LZ4.block\_stream\_decode and LZ4::BlockDecoder#update
74
- * Streaming Encode LZ4 block data : LZ4.block\_stream\_encode and LZ4::BlockEncoder#update (supporting high compression level)
75
-
76
-
77
62
  ## ABOUT TAINT STATE AND SECURITY (汚染状態とセキュリティについて)
78
63
 
79
64
  extlz4 はオブジェクト間での汚染状態を一方向伝播します。
@@ -92,25 +77,25 @@ First, load extlz4. (最初に extlz4 を読み込んでください)
92
77
  require "extlz4"
93
78
  ```
94
79
 
95
- ### Decoding (伸張処理)
80
+ ### One shot decoding from LZ4 Frame (LZ4 Frame 伸張処理)
96
81
 
97
82
  ``` ruby:ruby
98
83
  uncompressed_data_string = LZ4.decode(compressed_data_string)
99
84
  ```
100
85
 
101
- ### Encoding (通常圧縮処理)
86
+ ### One shot encoding to LZ4 Frame (LZ4 Frame 通常圧縮処理)
102
87
 
103
88
  ``` ruby:ruby
104
89
  compressed_data_string = LZ4.encode(uncompressed_data_string)
105
90
  ```
106
91
 
107
- ### High compression encoding (高圧縮処理)
92
+ ### One shot high compression encoding to LZ4 Frame (LZ4 Frame 高圧縮処理)
108
93
 
109
94
  ``` ruby:ruby
110
95
  compressed_data_string = LZ4.encode(uncompressed_data_string, 9)
111
96
  ```
112
97
 
113
- ### Frame decoding
98
+ ### Stream decoding to LZ4 Frame
114
99
 
115
100
  ``` ruby:ruby
116
101
  File.open("sample.txt.lz4", "rb") do |file|
@@ -122,7 +107,7 @@ File.open("sample.txt.lz4", "rb") do |file|
122
107
  end
123
108
  ```
124
109
 
125
- ### Frame encoding by high compression
110
+ ### Stream encoding by high compression to LZ4 Frame
126
111
 
127
112
  ``` ruby:ruby
128
113
  File.open("sample.txt.lz4", "wb") do |file|
@@ -133,7 +118,7 @@ File.open("sample.txt.lz4", "wb") do |file|
133
118
  end
134
119
  ```
135
120
 
136
- ### Frame encoding without block
121
+ ### Stream encoding without block to LZ4 Frame
137
122
 
138
123
  ``` ruby:ruby
139
124
  file = File.open("sample.txt.lz4", "wb")
@@ -142,7 +127,7 @@ lz4 << "abcdefghijklmnopqrstuvwxyz\n"
142
127
  lz4.close # VERY IMPORTANT!
143
128
  ```
144
129
 
145
- ### Block data processing (fast compression encoding and decoding)
130
+ ### One shot block data processing to/from LZ4 Block (fast compression encoding and decoding)
146
131
 
147
132
  ``` ruby:ruby
148
133
  src = "abcdefg" * 100
@@ -151,7 +136,7 @@ data = LZ4.block_decode(lz4data)
151
136
  p src == data # => true
152
137
  ```
153
138
 
154
- ### Block data processing (high compression encoding and decoding)
139
+ ### One shot block data processing to/from LZ4 Block (high compression encoding and decoding)
155
140
 
156
141
  ``` ruby:ruby
157
142
  src = "abcdefg" * 100
@@ -161,7 +146,7 @@ data = LZ4.block_decode(lz4data)
161
146
  p src == data # => true
162
147
  ```
163
148
 
164
- ### Block data processing (high speed encoding)
149
+ ### One shot block data processing to/from LZ4 Block (high speed encoding)
165
150
 
166
151
  ``` ruby:ruby
167
152
  src = "abcdefg" * 100
@@ -169,7 +154,7 @@ level = -19 # transform to one's complement as acceleration
169
154
  lz4data = LZ4.block_encode(level, src)
170
155
  ```
171
156
 
172
- ### Block stream data processing (high compression encoding and decoding)
157
+ ### Block stream data processing to/from LZ4 Block (high compression encoding and decoding)
173
158
 
174
159
  ``` ruby:ruby
175
160
  level = 8 # (OPTIONAL PARAMETER)
@@ -194,6 +179,11 @@ p src2 == data # => true
194
179
  ```
195
180
 
196
181
 
182
+ ## Support `Ractor` with Ruby3
183
+
184
+ Ruby3 で追加された `Ractor` に対応しています。
185
+
186
+
197
187
  ## BONUS (おまけ)
198
188
 
199
189
  コマンドラインプログラムとして ``extlz4`` が追加されます。
@@ -201,3 +191,21 @@ p src2 == data # => true
201
191
  これは lz4 と同程度の機能を持ちます (車輪の再発明とも言う)。
202
192
 
203
193
  とはいえ、引数のとり方を変えてあり、gzip のような形で利用できます。
194
+
195
+
196
+ ## SPECIFICATION (仕様)
197
+
198
+ - package name: extlz4
199
+ - author: dearblue (mailto:dearblue@users.osdn.me)
200
+ - project page: <https://github.com/dearblue/ruby-extlz4>
201
+ - how to install: `gem install extlz4`
202
+ - version: 0.3.2
203
+ - product quality: technical preview
204
+ - licensing: [2 clause BSD License](LICENSE)
205
+ - dependency gems: none
206
+ - dependency external c libraries: none
207
+ - bundled external c libraries (git submodules):
208
+ - [lz4](https://github.com/lz4/lz4)
209
+ [version 1.9.3](https://github.com/lz4/lz4/tree/v1.9.3)
210
+ under [2 clause BSD license](https://github.com/lz4/lz4/blob/v1.9.3/LICENSE)
211
+ by [Yann Collet](https://github.com/Cyan4973)
data/bin/extlz4 CHANGED
@@ -1,7 +1,7 @@
1
1
  #!ruby
2
2
  #vim: set fileencoding:utf-8
3
3
 
4
- # AUTHOR: dearblue <dearblue@users.noreply.github.com>
4
+ # AUTHOR: dearblue <dearblue@users.osdn.me>
5
5
  # Only this program is usable as public domain software.
6
6
 
7
7
  require "extlz4"
data/contrib/lz4/INSTALL CHANGED
@@ -8,6 +8,7 @@ make install # this command may require root access
8
8
 
9
9
  LZ4's `Makefile` supports standard [Makefile conventions],
10
10
  including [staged installs], [redirection], or [command redefinition].
11
+ It is compatible with parallel builds (`-j#`).
11
12
 
12
13
  [Makefile conventions]: https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html
13
14
  [staged installs]: https://www.gnu.org/prep/standards/html_node/DESTDIR.html
@@ -0,0 +1,87 @@
1
+ ifeq ($(V), 1)
2
+ Q =
3
+ else
4
+ Q = @
5
+ endif
6
+
7
+ TARGET_OS ?= $(shell uname)
8
+ ifeq ($(TARGET_OS),)
9
+ TARGET_OS ?= $(OS)
10
+ endif
11
+
12
+ ifneq (,$(filter Windows%,$(TARGET_OS)))
13
+ LIBLZ4 = liblz4-$(LIBVER_MAJOR)
14
+ LIBLZ4_EXP = liblz4.lib
15
+ WINBASED = yes
16
+ else
17
+ LIBLZ4_EXP = liblz4.dll.a
18
+ ifneq (,$(filter MINGW%,$(TARGET_OS)))
19
+ LIBLZ4 = liblz4
20
+ WINBASED = yes
21
+ else
22
+ ifneq (,$(filter MSYS%,$(TARGET_OS)))
23
+ LIBLZ4 = msys-lz4-$(LIBVER_MAJOR)
24
+ WINBASED = yes
25
+ else
26
+ ifneq (,$(filter CYGWIN%,$(TARGET_OS)))
27
+ LIBLZ4 = cyglz4-$(LIBVER_MAJOR)
28
+ WINBASED = yes
29
+ else
30
+ LIBLZ4 = liblz4.$(SHARED_EXT_VER)
31
+ WINBASED = no
32
+ EXT =
33
+ endif
34
+ endif
35
+ endif
36
+ endif
37
+
38
+ ifeq ($(WINBASED),yes)
39
+ EXT = .exe
40
+ WINDRES = windres
41
+ endif
42
+
43
+ #determine if dev/nul based on host environment
44
+ ifneq (,$(filter MINGW% MSYS% CYGWIN%,$(shell uname)))
45
+ VOID := /dev/null
46
+ else
47
+ ifneq (,$(filter Windows%,$(OS)))
48
+ VOID := nul
49
+ else
50
+ VOID := /dev/null
51
+ endif
52
+ endif
53
+
54
+ ifneq (,$(filter Linux Darwin GNU/kFreeBSD GNU OpenBSD FreeBSD NetBSD DragonFly SunOS Haiku MidnightBSD MINGW% CYGWIN% MSYS%,$(shell uname)))
55
+ POSIX_ENV = Yes
56
+ else
57
+ POSIX_ENV = No
58
+ endif
59
+
60
+ # Avoid symlinks when targetting Windows or building on a Windows host
61
+ ifeq ($(WINBASED),yes)
62
+ LN_S = cp -p
63
+ LN_SF = cp -p
64
+ else
65
+ ifneq (,$(filter MINGW% MSYS% CYGWIN%,$(shell uname)))
66
+ LN_S = cp -p
67
+ LN_SF = cp -p
68
+ else
69
+ ifneq (,$(filter Windows%,$(OS)))
70
+ LN_S = cp -p
71
+ LN_SF = cp -p
72
+ else
73
+ LN_S = ln -s
74
+ LN_SF = ln -sf
75
+ endif
76
+ endif
77
+ endif
78
+
79
+ ifneq (,$(filter $(shell uname),SunOS))
80
+ INSTALL ?= ginstall
81
+ else
82
+ INSTALL ?= install
83
+ endif
84
+
85
+ INSTALL_PROGRAM ?= $(INSTALL) -m 755
86
+ INSTALL_DATA ?= $(INSTALL) -m 644
87
+ INSTALL_DIR ?= $(INSTALL) -d -m 755
data/contrib/lz4/NEWS CHANGED
@@ -1,3 +1,92 @@
1
+ v1.9.3
2
+ perf: highly improved speed in kernel space, by @terrelln
3
+ perf: faster speed with Visual Studio, thanks to @wolfpld and @remittor
4
+ perf: improved dictionary compression speed, by @felixhandte
5
+ perf: fixed LZ4_compress_HC_destSize() ratio, detected by @hsiangkao
6
+ perf: reduced stack usage in high compression mode, by @Yanpas
7
+ api : LZ4_decompress_safe_partial() supports unknown compressed size, requested by @jfkthame
8
+ api : improved LZ4F_compressBound() with automatic flushing, by Christopher Harvie
9
+ api : can (de)compress to/from NULL without UBs
10
+ api : fix alignment test on 32-bit systems (state initialization)
11
+ api : fix LZ4_saveDictHC() in corner case scenario, detected by @IgorKorkin
12
+ cli : `-l` legacy format is now compatible with `-m` multiple files, by Filipe Calasans
13
+ cli : benchmark mode supports dictionary, by @rkoradi
14
+ cli : fix --fast with large argument, detected by @picoHz
15
+ build: link to user-defined memory functions with LZ4_USER_MEMORY_FUNCTIONS, suggested by Yuriy Levchenko
16
+ build: contrib/cmake_unofficial/ moved to build/cmake/
17
+ build: visual/* moved to build/
18
+ build: updated meson script, by @neheb
19
+ build: tinycc support, by Anton Kochkov
20
+ install: Haiku support, by Jerome Duval
21
+ doc : updated LZ4 frame format, clarify EndMark
22
+
23
+ v1.9.2
24
+ fix : out-of-bound read in exceptional circumstances when using decompress_partial(), by @terrelln
25
+ fix : slim opportunity for out-of-bound write with compress_fast() with a large enough input and when providing an output smaller than recommended (< LZ4_compressBound(inputSize)), by @terrelln
26
+ fix : rare data corruption bug with LZ4_compress_destSize(), by @terrelln
27
+ fix : data corruption bug when Streaming with an Attached Dict in HC Mode, by @felixhandte
28
+ perf: enable LZ4_FAST_DEC_LOOP on aarch64/GCC by default, by @prekageo
29
+ perf: improved lz4frame streaming API speed, by @dreambottle
30
+ perf: speed up lz4hc on slow patterns when using external dictionary, by @terrelln
31
+ api: better in-place decompression and compression support
32
+ cli : --list supports multi-frames files, by @gstedman
33
+ cli: --version outputs to stdout
34
+ cli : add option --best as an alias of -12 , by @Low-power
35
+ misc: Integration into oss-fuzz by @cmeister2, expanded list of scenarios by @terrelln
36
+
37
+ v1.9.1
38
+ fix : decompression functions were reading a few bytes beyond input size (introduced in v1.9.0, reported by @ppodolsky and @danlark1)
39
+ api : fix : lz4frame initializers compatibility with c++, reported by @degski
40
+ cli : added command --list, based on a patch by @gabrielstedman
41
+ build: improved Windows build, by @JPeterMugaas
42
+ build: AIX, by Norman Green
43
+
44
+ v1.9.0
45
+ perf: large decompression speed improvement on x86/x64 (up to +20%) by @djwatson
46
+ api : changed : _destSize() compression variants are promoted to stable API
47
+ api : new : LZ4_initStream(HC), replacing LZ4_resetStream(HC)
48
+ api : changed : LZ4_resetStream(HC) as recommended reset function, for better performance on small data
49
+ cli : support custom block sizes, by @blezsan
50
+ build: source code can be amalgamated, by Bing Xu
51
+ build: added meson build, by @lzutao
52
+ build: new build macros : LZ4_DISTANCE_MAX, LZ4_FAST_DEC_LOOP
53
+ install: MidnightBSD, by @laffer1
54
+ install: msys2 on Windows 10, by @vtorri
55
+
56
+ v1.8.3
57
+ perf: minor decompression speed improvement (~+2%) with gcc
58
+ fix : corruption in v1.8.2 at level 9 for files > 64KB under rare conditions (#560)
59
+ cli : new command --fast, by @jennifermliu
60
+ cli : fixed elapsed time, and added cpu load indicator (on -vv) (#555)
61
+ api : LZ4_decompress_safe_partial() now decodes exactly the nb of bytes requested (feature request #566)
62
+ build : added Haiku target, by @fbrosson, and MidnightBSD, by @laffer1
63
+ doc : updated documentation regarding dictionary compression
64
+
65
+ v1.8.2
66
+ perf: *much* faster dictionary compression on small files, by @felixhandte
67
+ perf: improved decompression speed and binary size, by Alexey Tourbin (@svpv)
68
+ perf: slightly faster HC compression and decompression speed
69
+ perf: very small compression ratio improvement
70
+ fix : compression compatible with low memory addresses (< 0xFFFF)
71
+ fix : decompression segfault when provided with NULL input, by @terrelln
72
+ cli : new command --favor-decSpeed
73
+ cli : benchmark mode more accurate for small inputs
74
+ fullbench : can bench _destSize() variants, by @felixhandte
75
+ doc : clarified block format parsing restrictions, by Alexey Tourbin (@svpv)
76
+
77
+ v1.8.1
78
+ perf : faster and stronger ultra modes (levels 10+)
79
+ perf : slightly faster compression and decompression speed
80
+ perf : fix bad degenerative case, reported by @c-morgenstern
81
+ fix : decompression failed when using a combination of extDict + low memory address (#397), reported and fixed by Julian Scheid (@jscheid)
82
+ cli : support for dictionary compression (`-D`), by Felix Handte @felixhandte
83
+ cli : fix : `lz4 -d --rm` preserves timestamp (#441)
84
+ cli : fix : do not modify /dev/null permission as root, by @aliceatlas
85
+ api : `_destSize()` variant supported for all compression levels
86
+ build : `make` and `make test` compatible with `-jX`, reported by @mwgamera
87
+ build : can control LZ4LIB_VISIBILITY macro, by @mikir
88
+ install: fix man page directory (#387), reported by Stuart Cardall (@itoffshore)
89
+
1
90
  v1.8.0
2
91
  cli : fix : do not modify /dev/null permissions, reported by @Maokaman1
3
92
  cli : added GNU separator -- specifying that all following arguments are files
@@ -2,74 +2,68 @@ LZ4 - Extremely fast compression
2
2
  ================================
3
3
 
4
4
  LZ4 is lossless compression algorithm,
5
- providing compression speed at 400 MB/s per core,
5
+ providing compression speed > 500 MB/s per core,
6
6
  scalable with multi-cores CPU.
7
7
  It features an extremely fast decoder,
8
8
  with speed in multiple GB/s per core,
9
9
  typically reaching RAM speed limits on multi-core systems.
10
10
 
11
11
  Speed can be tuned dynamically, selecting an "acceleration" factor
12
- which trades compression ratio for more speed up.
12
+ which trades compression ratio for faster speed.
13
13
  On the other end, a high compression derivative, LZ4_HC, is also provided,
14
14
  trading CPU time for improved compression ratio.
15
15
  All versions feature the same decompression speed.
16
16
 
17
+ LZ4 is also compatible with [dictionary compression](https://github.com/facebook/zstd#the-case-for-small-data-compression),
18
+ both at [API](https://github.com/lz4/lz4/blob/v1.8.3/lib/lz4frame.h#L481) and [CLI](https://github.com/lz4/lz4/blob/v1.8.3/programs/lz4.1.md#operation-modifiers) levels.
19
+ It can ingest any input file as dictionary, though only the final 64KB are used.
20
+ This capability can be combined with the [Zstandard Dictionary Builder](https://github.com/facebook/zstd/blob/v1.3.5/programs/zstd.1.md#dictionary-builder),
21
+ in order to drastically improve compression performance on small files.
22
+
23
+
17
24
  LZ4 library is provided as open-source software using BSD 2-Clause license.
18
25
 
19
26
 
20
27
  |Branch |Status |
21
28
  |------------|---------|
22
- |master | [![Build Status][travisMasterBadge]][travisLink] [![Build status][AppveyorMasterBadge]][AppveyorLink] [![coverity][coverBadge]][coverlink] |
23
29
  |dev | [![Build Status][travisDevBadge]][travisLink] [![Build status][AppveyorDevBadge]][AppveyorLink] |
24
30
 
25
- [travisMasterBadge]: https://travis-ci.org/lz4/lz4.svg?branch=master "Continuous Integration test suite"
26
31
  [travisDevBadge]: https://travis-ci.org/lz4/lz4.svg?branch=dev "Continuous Integration test suite"
27
32
  [travisLink]: https://travis-ci.org/lz4/lz4
28
- [AppveyorMasterBadge]: https://ci.appveyor.com/api/projects/status/github/lz4/lz4?branch=master&svg=true "Windows test suite"
29
33
  [AppveyorDevBadge]: https://ci.appveyor.com/api/projects/status/github/lz4/lz4?branch=dev&svg=true "Windows test suite"
30
34
  [AppveyorLink]: https://ci.appveyor.com/project/YannCollet/lz4-1lndh
31
- [coverBadge]: https://scan.coverity.com/projects/4735/badge.svg "Static code analysis of Master branch"
32
- [coverlink]: https://scan.coverity.com/projects/4735
33
35
 
34
- > **Branch Policy:**
35
- > - The "master" branch is considered stable, at all times.
36
- > - The "dev" branch is the one where all contributions must be merged
37
- before being promoted to master.
38
- > + If you plan to propose a patch, please commit into the "dev" branch,
39
- or its own feature branch.
40
- Direct commit to "master" are not permitted.
41
36
 
42
37
  Benchmarks
43
38
  -------------------------
44
39
 
45
40
  The benchmark uses [lzbench], from @inikep
46
- compiled with GCC v6.2.0 on Linux 64-bits.
47
- The reference system uses a Core i7-3930K CPU @ 4.5GHz.
41
+ compiled with GCC v8.2.0 on Linux 64-bits (Ubuntu 4.18.0-17).
42
+ The reference system uses a Core i7-9700K CPU @ 4.9GHz (w/ turbo boost).
48
43
  Benchmark evaluates the compression of reference [Silesia Corpus]
49
44
  in single-thread mode.
50
45
 
51
46
  [lzbench]: https://github.com/inikep/lzbench
52
47
  [Silesia Corpus]: http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia
53
48
 
54
- | Compressor | Ratio | Compression | Decompression |
55
- | ---------- | ----- | ----------- | ------------- |
56
- | memcpy | 1.000 | 7300 MB/s | 7300 MB/s |
57
- |**LZ4 fast 8 (v1.7.3)**| 1.799 |**911 MB/s** | **3360 MB/s** |
58
- |**LZ4 default (v1.7.3)**|**2.101**|**625 MB/s** | **3220 MB/s** |
59
- | LZO 2.09 | 2.108 | 620 MB/s | 845 MB/s |
60
- | QuickLZ 1.5.0 | 2.238 | 510 MB/s | 600 MB/s |
61
- | Snappy 1.1.3 | 2.091 | 450 MB/s | 1550 MB/s |
62
- | LZF v3.6 | 2.073 | 365 MB/s | 820 MB/s |
63
- | [Zstandard] 1.1.1 -1 | 2.876 | 330 MB/s | 930 MB/s |
64
- | [Zstandard] 1.1.1 -3 | 3.164 | 200 MB/s | 810 MB/s |
65
- | [zlib] deflate 1.2.8 -1| 2.730 | 100 MB/s | 370 MB/s |
66
- |**LZ4 HC -9 (v1.7.3)** |**2.720**| 34 MB/s | **3240 MB/s** |
67
- | [zlib] deflate 1.2.8 -6| 3.099 | 33 MB/s | 390 MB/s |
49
+ | Compressor | Ratio | Compression | Decompression |
50
+ | ---------- | ----- | ----------- | ------------- |
51
+ | memcpy | 1.000 | 13700 MB/s | 13700 MB/s |
52
+ |**LZ4 default (v1.9.0)** |**2.101**| **780 MB/s**| **4970 MB/s** |
53
+ | LZO 2.09 | 2.108 | 670 MB/s | 860 MB/s |
54
+ | QuickLZ 1.5.0 | 2.238 | 575 MB/s | 780 MB/s |
55
+ | Snappy 1.1.4 | 2.091 | 565 MB/s | 1950 MB/s |
56
+ | [Zstandard] 1.4.0 -1 | 2.883 | 515 MB/s | 1380 MB/s |
57
+ | LZF v3.6 | 2.073 | 415 MB/s | 910 MB/s |
58
+ | [zlib] deflate 1.2.11 -1| 2.730 | 100 MB/s | 415 MB/s |
59
+ |**LZ4 HC -9 (v1.9.0)** |**2.721**| 41 MB/s | **4900 MB/s** |
60
+ | [zlib] deflate 1.2.11 -6| 3.099 | 36 MB/s | 445 MB/s |
68
61
 
69
62
  [zlib]: http://www.zlib.net/
70
63
  [Zstandard]: http://www.zstd.net/
71
64
 
72
- LZ4 is also compatible and well optimized for x32 mode, for which it provides an additional +10% speed performance.
65
+ LZ4 is also compatible and optimized for x32 mode,
66
+ for which it provides additional speed performance.
73
67
 
74
68
 
75
69
  Installation
@@ -77,27 +71,39 @@ Installation
77
71
 
78
72
  ```
79
73
  make
80
- make install # this command may require root access
74
+ make install # this command may require root permissions
81
75
  ```
82
76
 
83
77
  LZ4's `Makefile` supports standard [Makefile conventions],
84
78
  including [staged installs], [redirection], or [command redefinition].
79
+ It is compatible with parallel builds (`-j#`).
85
80
 
86
81
  [Makefile conventions]: https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html
87
82
  [staged installs]: https://www.gnu.org/prep/standards/html_node/DESTDIR.html
88
83
  [redirection]: https://www.gnu.org/prep/standards/html_node/Directory-Variables.html
89
84
  [command redefinition]: https://www.gnu.org/prep/standards/html_node/Utilities-in-Makefiles.html
90
85
 
86
+ Building LZ4 - Using vcpkg
87
+
88
+ You can download and install LZ4 using the [vcpkg](https://github.com/Microsoft/vcpkg) dependency manager:
89
+
90
+ git clone https://github.com/Microsoft/vcpkg.git
91
+ cd vcpkg
92
+ ./bootstrap-vcpkg.sh
93
+ ./vcpkg integrate install
94
+ vcpkg install lz4
95
+
96
+ The LZ4 port in vcpkg is kept up to date by Microsoft team members and community contributors. If the version is out of date, please [create an issue or pull request](https://github.com/Microsoft/vcpkg) on the vcpkg repository.
91
97
 
92
98
  Documentation
93
99
  -------------------------
94
100
 
95
101
  The raw LZ4 block compression format is detailed within [lz4_Block_format].
96
102
 
97
- To compress an arbitrarily long file or data stream, multiple blocks are required.
98
- Organizing these blocks and providing a common header format to handle their content
99
- is the purpose of the Frame format, defined into [lz4_Frame_format].
100
- Interoperable versions of LZ4 must respect this frame format.
103
+ Arbitrarily long files or data streams are compressed using multiple blocks,
104
+ for streaming requirements. These blocks are organized into a frame,
105
+ defined into [lz4_Frame_format].
106
+ Interoperable versions of LZ4 must also respect the frame format.
101
107
 
102
108
  [lz4_Block_format]: doc/lz4_Block_format.md
103
109
  [lz4_Frame_format]: doc/lz4_Frame_format.md