zstd-ruby 1.4.1.0 → 1.5.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (96) hide show
  1. checksums.yaml +4 -4
  2. data/.github/dependabot.yml +8 -0
  3. data/.github/workflows/ruby.yml +35 -0
  4. data/README.md +2 -2
  5. data/ext/zstdruby/libzstd/BUCK +5 -7
  6. data/ext/zstdruby/libzstd/Makefile +304 -113
  7. data/ext/zstdruby/libzstd/README.md +83 -20
  8. data/ext/zstdruby/libzstd/common/bitstream.h +59 -51
  9. data/ext/zstdruby/libzstd/common/compiler.h +150 -8
  10. data/ext/zstdruby/libzstd/common/cpu.h +1 -3
  11. data/ext/zstdruby/libzstd/common/debug.c +11 -31
  12. data/ext/zstdruby/libzstd/common/debug.h +22 -49
  13. data/ext/zstdruby/libzstd/common/entropy_common.c +201 -75
  14. data/ext/zstdruby/libzstd/common/error_private.c +3 -1
  15. data/ext/zstdruby/libzstd/common/error_private.h +8 -4
  16. data/ext/zstdruby/libzstd/common/fse.h +50 -42
  17. data/ext/zstdruby/libzstd/common/fse_decompress.c +149 -55
  18. data/ext/zstdruby/libzstd/common/huf.h +43 -39
  19. data/ext/zstdruby/libzstd/common/mem.h +69 -25
  20. data/ext/zstdruby/libzstd/common/pool.c +30 -20
  21. data/ext/zstdruby/libzstd/common/pool.h +3 -3
  22. data/ext/zstdruby/libzstd/common/threading.c +51 -4
  23. data/ext/zstdruby/libzstd/common/threading.h +36 -4
  24. data/ext/zstdruby/libzstd/common/xxhash.c +40 -92
  25. data/ext/zstdruby/libzstd/common/xxhash.h +12 -32
  26. data/ext/zstdruby/libzstd/common/zstd_common.c +10 -10
  27. data/ext/zstdruby/libzstd/common/zstd_deps.h +111 -0
  28. data/ext/zstdruby/libzstd/common/zstd_internal.h +230 -111
  29. data/ext/zstdruby/libzstd/common/zstd_trace.h +154 -0
  30. data/ext/zstdruby/libzstd/compress/fse_compress.c +47 -63
  31. data/ext/zstdruby/libzstd/compress/hist.c +41 -63
  32. data/ext/zstdruby/libzstd/compress/hist.h +13 -33
  33. data/ext/zstdruby/libzstd/compress/huf_compress.c +332 -193
  34. data/ext/zstdruby/libzstd/compress/zstd_compress.c +3614 -1696
  35. data/ext/zstdruby/libzstd/compress/zstd_compress_internal.h +546 -86
  36. data/ext/zstdruby/libzstd/compress/zstd_compress_literals.c +158 -0
  37. data/ext/zstdruby/libzstd/compress/zstd_compress_literals.h +29 -0
  38. data/ext/zstdruby/libzstd/compress/zstd_compress_sequences.c +441 -0
  39. data/ext/zstdruby/libzstd/compress/zstd_compress_sequences.h +54 -0
  40. data/ext/zstdruby/libzstd/compress/zstd_compress_superblock.c +572 -0
  41. data/ext/zstdruby/libzstd/compress/zstd_compress_superblock.h +32 -0
  42. data/ext/zstdruby/libzstd/compress/zstd_cwksp.h +662 -0
  43. data/ext/zstdruby/libzstd/compress/zstd_double_fast.c +43 -41
  44. data/ext/zstdruby/libzstd/compress/zstd_double_fast.h +2 -2
  45. data/ext/zstdruby/libzstd/compress/zstd_fast.c +85 -80
  46. data/ext/zstdruby/libzstd/compress/zstd_fast.h +2 -2
  47. data/ext/zstdruby/libzstd/compress/zstd_lazy.c +1184 -111
  48. data/ext/zstdruby/libzstd/compress/zstd_lazy.h +59 -1
  49. data/ext/zstdruby/libzstd/compress/zstd_ldm.c +333 -208
  50. data/ext/zstdruby/libzstd/compress/zstd_ldm.h +15 -3
  51. data/ext/zstdruby/libzstd/compress/zstd_ldm_geartab.h +103 -0
  52. data/ext/zstdruby/libzstd/compress/zstd_opt.c +228 -129
  53. data/ext/zstdruby/libzstd/compress/zstd_opt.h +1 -1
  54. data/ext/zstdruby/libzstd/compress/zstdmt_compress.c +151 -440
  55. data/ext/zstdruby/libzstd/compress/zstdmt_compress.h +32 -114
  56. data/ext/zstdruby/libzstd/decompress/huf_decompress.c +395 -276
  57. data/ext/zstdruby/libzstd/decompress/zstd_ddict.c +20 -16
  58. data/ext/zstdruby/libzstd/decompress/zstd_ddict.h +3 -3
  59. data/ext/zstdruby/libzstd/decompress/zstd_decompress.c +630 -231
  60. data/ext/zstdruby/libzstd/decompress/zstd_decompress_block.c +606 -380
  61. data/ext/zstdruby/libzstd/decompress/zstd_decompress_block.h +8 -5
  62. data/ext/zstdruby/libzstd/decompress/zstd_decompress_internal.h +39 -9
  63. data/ext/zstdruby/libzstd/deprecated/zbuff.h +9 -8
  64. data/ext/zstdruby/libzstd/deprecated/zbuff_common.c +2 -2
  65. data/ext/zstdruby/libzstd/deprecated/zbuff_compress.c +1 -1
  66. data/ext/zstdruby/libzstd/deprecated/zbuff_decompress.c +1 -1
  67. data/ext/zstdruby/libzstd/dictBuilder/cover.c +55 -46
  68. data/ext/zstdruby/libzstd/dictBuilder/cover.h +20 -9
  69. data/ext/zstdruby/libzstd/dictBuilder/divsufsort.c +1 -1
  70. data/ext/zstdruby/libzstd/dictBuilder/fastcover.c +43 -31
  71. data/ext/zstdruby/libzstd/dictBuilder/zdict.c +53 -30
  72. data/ext/zstdruby/libzstd/dll/example/Makefile +2 -1
  73. data/ext/zstdruby/libzstd/dll/example/README.md +16 -22
  74. data/ext/zstdruby/libzstd/legacy/zstd_legacy.h +4 -4
  75. data/ext/zstdruby/libzstd/legacy/zstd_v01.c +24 -14
  76. data/ext/zstdruby/libzstd/legacy/zstd_v01.h +1 -1
  77. data/ext/zstdruby/libzstd/legacy/zstd_v02.c +17 -8
  78. data/ext/zstdruby/libzstd/legacy/zstd_v02.h +1 -1
  79. data/ext/zstdruby/libzstd/legacy/zstd_v03.c +17 -8
  80. data/ext/zstdruby/libzstd/legacy/zstd_v03.h +1 -1
  81. data/ext/zstdruby/libzstd/legacy/zstd_v04.c +25 -11
  82. data/ext/zstdruby/libzstd/legacy/zstd_v04.h +1 -1
  83. data/ext/zstdruby/libzstd/legacy/zstd_v05.c +43 -32
  84. data/ext/zstdruby/libzstd/legacy/zstd_v05.h +2 -2
  85. data/ext/zstdruby/libzstd/legacy/zstd_v06.c +27 -19
  86. data/ext/zstdruby/libzstd/legacy/zstd_v06.h +1 -1
  87. data/ext/zstdruby/libzstd/legacy/zstd_v07.c +32 -20
  88. data/ext/zstdruby/libzstd/legacy/zstd_v07.h +1 -1
  89. data/ext/zstdruby/libzstd/libzstd.pc.in +2 -1
  90. data/ext/zstdruby/libzstd/{dictBuilder/zdict.h → zdict.h} +201 -31
  91. data/ext/zstdruby/libzstd/zstd.h +740 -153
  92. data/ext/zstdruby/libzstd/{common/zstd_errors.h → zstd_errors.h} +3 -1
  93. data/lib/zstd-ruby/version.rb +1 -1
  94. data/zstd-ruby.gemspec +1 -1
  95. metadata +21 -10
  96. data/.travis.yml +0 -14
@@ -1,15 +1,26 @@
1
+ /*
2
+ * Copyright (c) Facebook, Inc.
3
+ * All rights reserved.
4
+ *
5
+ * This source code is licensed under both the BSD-style license (found in the
6
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
7
+ * in the COPYING file in the root directory of this source tree).
8
+ * You may select, at your option, one of the above-listed licenses.
9
+ */
10
+
11
+ #ifndef ZDICT_STATIC_LINKING_ONLY
12
+ # define ZDICT_STATIC_LINKING_ONLY
13
+ #endif
14
+
1
15
  #include <stdio.h> /* fprintf */
2
16
  #include <stdlib.h> /* malloc, free, qsort */
3
17
  #include <string.h> /* memset */
4
18
  #include <time.h> /* clock */
5
- #include "mem.h" /* read */
6
- #include "pool.h"
7
- #include "threading.h"
8
- #include "zstd_internal.h" /* includes zstd.h */
9
- #ifndef ZDICT_STATIC_LINKING_ONLY
10
- #define ZDICT_STATIC_LINKING_ONLY
11
- #endif
12
- #include "zdict.h"
19
+ #include "../common/mem.h" /* read */
20
+ #include "../common/pool.h"
21
+ #include "../common/threading.h"
22
+ #include "../common/zstd_internal.h" /* includes zstd.h */
23
+ #include "../zdict.h"
13
24
 
14
25
  /**
15
26
  * COVER_best_t is used for two purposes:
@@ -142,6 +153,6 @@ void COVER_dictSelectionFree(COVER_dictSelection_t selection);
142
153
  * smallest dictionary within a specified regression of the compressed size
143
154
  * from the largest dictionary.
144
155
  */
145
- COVER_dictSelection_t COVER_selectDict(BYTE* customDictContent,
156
+ COVER_dictSelection_t COVER_selectDict(BYTE* customDictContent, size_t dictBufferCapacity,
146
157
  size_t dictContentSize, const BYTE* samplesBuffer, const size_t* samplesSizes, unsigned nbFinalizeSamples,
147
158
  size_t nbCheckSamples, size_t nbSamples, ZDICT_cover_params_t params, size_t* offsets, size_t totalCompressedSize);
@@ -1576,7 +1576,7 @@ note:
1576
1576
  /* Construct the inverse suffix array of type B* suffixes using trsort. */
1577
1577
  trsort(ISAb, SA, m, 1);
1578
1578
 
1579
- /* Set the sorted order of tyoe B* suffixes. */
1579
+ /* Set the sorted order of type B* suffixes. */
1580
1580
  for(i = n - 1, j = m, c0 = T[n - 1]; 0 <= i;) {
1581
1581
  for(--i, c1 = c0; (0 <= i) && ((c0 = T[i]) >= c1); --i, c1 = c0) { }
1582
1582
  if(0 <= i) {
@@ -1,3 +1,13 @@
1
+ /*
2
+ * Copyright (c) Facebook, Inc.
3
+ * All rights reserved.
4
+ *
5
+ * This source code is licensed under both the BSD-style license (found in the
6
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
7
+ * in the COPYING file in the root directory of this source tree).
8
+ * You may select, at your option, one of the above-listed licenses.
9
+ */
10
+
1
11
  /*-*************************************
2
12
  * Dependencies
3
13
  ***************************************/
@@ -6,15 +16,17 @@
6
16
  #include <string.h> /* memset */
7
17
  #include <time.h> /* clock */
8
18
 
9
- #include "mem.h" /* read */
10
- #include "pool.h"
11
- #include "threading.h"
12
- #include "cover.h"
13
- #include "zstd_internal.h" /* includes zstd.h */
14
19
  #ifndef ZDICT_STATIC_LINKING_ONLY
15
- #define ZDICT_STATIC_LINKING_ONLY
20
+ # define ZDICT_STATIC_LINKING_ONLY
16
21
  #endif
17
- #include "zdict.h"
22
+
23
+ #include "../common/mem.h" /* read */
24
+ #include "../common/pool.h"
25
+ #include "../common/threading.h"
26
+ #include "../common/zstd_internal.h" /* includes zstd.h */
27
+ #include "../compress/zstd_compress_internal.h" /* ZSTD_hash*() */
28
+ #include "../zdict.h"
29
+ #include "cover.h"
18
30
 
19
31
 
20
32
  /*-*************************************
@@ -23,7 +35,7 @@
23
35
  #define FASTCOVER_MAX_SAMPLES_SIZE (sizeof(size_t) == 8 ? ((unsigned)-1) : ((unsigned)1 GB))
24
36
  #define FASTCOVER_MAX_F 31
25
37
  #define FASTCOVER_MAX_ACCEL 10
26
- #define DEFAULT_SPLITPOINT 0.75
38
+ #define FASTCOVER_DEFAULT_SPLITPOINT 0.75
27
39
  #define DEFAULT_F 20
28
40
  #define DEFAULT_ACCEL 1
29
41
 
@@ -31,50 +43,50 @@
31
43
  /*-*************************************
32
44
  * Console display
33
45
  ***************************************/
46
+ #ifndef LOCALDISPLAYLEVEL
34
47
  static int g_displayLevel = 2;
48
+ #endif
49
+ #undef DISPLAY
35
50
  #define DISPLAY(...) \
36
51
  { \
37
52
  fprintf(stderr, __VA_ARGS__); \
38
53
  fflush(stderr); \
39
54
  }
55
+ #undef LOCALDISPLAYLEVEL
40
56
  #define LOCALDISPLAYLEVEL(displayLevel, l, ...) \
41
57
  if (displayLevel >= l) { \
42
58
  DISPLAY(__VA_ARGS__); \
43
59
  } /* 0 : no display; 1: errors; 2: default; 3: details; 4: debug */
60
+ #undef DISPLAYLEVEL
44
61
  #define DISPLAYLEVEL(l, ...) LOCALDISPLAYLEVEL(g_displayLevel, l, __VA_ARGS__)
45
62
 
63
+ #ifndef LOCALDISPLAYUPDATE
64
+ static const clock_t g_refreshRate = CLOCKS_PER_SEC * 15 / 100;
65
+ static clock_t g_time = 0;
66
+ #endif
67
+ #undef LOCALDISPLAYUPDATE
46
68
  #define LOCALDISPLAYUPDATE(displayLevel, l, ...) \
47
69
  if (displayLevel >= l) { \
48
- if ((clock() - g_time > refreshRate) || (displayLevel >= 4)) { \
70
+ if ((clock() - g_time > g_refreshRate) || (displayLevel >= 4)) { \
49
71
  g_time = clock(); \
50
72
  DISPLAY(__VA_ARGS__); \
51
73
  } \
52
74
  }
75
+ #undef DISPLAYUPDATE
53
76
  #define DISPLAYUPDATE(l, ...) LOCALDISPLAYUPDATE(g_displayLevel, l, __VA_ARGS__)
54
- static const clock_t refreshRate = CLOCKS_PER_SEC * 15 / 100;
55
- static clock_t g_time = 0;
56
77
 
57
78
 
58
79
  /*-*************************************
59
80
  * Hash Functions
60
81
  ***************************************/
61
- static const U64 prime6bytes = 227718039650203ULL;
62
- static size_t ZSTD_hash6(U64 u, U32 h) { return (size_t)(((u << (64-48)) * prime6bytes) >> (64-h)) ; }
63
- static size_t ZSTD_hash6Ptr(const void* p, U32 h) { return ZSTD_hash6(MEM_readLE64(p), h); }
64
-
65
- static const U64 prime8bytes = 0xCF1BBCDCB7A56463ULL;
66
- static size_t ZSTD_hash8(U64 u, U32 h) { return (size_t)(((u) * prime8bytes) >> (64-h)) ; }
67
- static size_t ZSTD_hash8Ptr(const void* p, U32 h) { return ZSTD_hash8(MEM_readLE64(p), h); }
68
-
69
-
70
82
  /**
71
- * Hash the d-byte value pointed to by p and mod 2^f
83
+ * Hash the d-byte value pointed to by p and mod 2^f into the frequency vector
72
84
  */
73
- static size_t FASTCOVER_hashPtrToIndex(const void* p, U32 h, unsigned d) {
85
+ static size_t FASTCOVER_hashPtrToIndex(const void* p, U32 f, unsigned d) {
74
86
  if (d == 6) {
75
- return ZSTD_hash6Ptr(p, h) & ((1 << h) - 1);
87
+ return ZSTD_hash6Ptr(p, f);
76
88
  }
77
- return ZSTD_hash8Ptr(p, h) & ((1 << h) - 1);
89
+ return ZSTD_hash8Ptr(p, f);
78
90
  }
79
91
 
80
92
 
@@ -451,20 +463,20 @@ typedef struct FASTCOVER_tryParameters_data_s {
451
463
  * This function is thread safe if zstd is compiled with multithreaded support.
452
464
  * It takes its parameters as an *OWNING* opaque pointer to support threading.
453
465
  */
454
- static void FASTCOVER_tryParameters(void *opaque)
466
+ static void FASTCOVER_tryParameters(void* opaque)
455
467
  {
456
468
  /* Save parameters as local variables */
457
- FASTCOVER_tryParameters_data_t *const data = (FASTCOVER_tryParameters_data_t *)opaque;
469
+ FASTCOVER_tryParameters_data_t *const data = (FASTCOVER_tryParameters_data_t*)opaque;
458
470
  const FASTCOVER_ctx_t *const ctx = data->ctx;
459
471
  const ZDICT_cover_params_t parameters = data->parameters;
460
472
  size_t dictBufferCapacity = data->dictBufferCapacity;
461
473
  size_t totalCompressedSize = ERROR(GENERIC);
462
474
  /* Initialize array to keep track of frequency of dmer within activeSegment */
463
- U16* segmentFreqs = (U16 *)calloc(((U64)1 << ctx->f), sizeof(U16));
475
+ U16* segmentFreqs = (U16*)calloc(((U64)1 << ctx->f), sizeof(U16));
464
476
  /* Allocate space for hash table, dict, and freqs */
465
- BYTE *const dict = (BYTE * const)malloc(dictBufferCapacity);
477
+ BYTE *const dict = (BYTE*)malloc(dictBufferCapacity);
466
478
  COVER_dictSelection_t selection = COVER_dictSelectionError(ERROR(GENERIC));
467
- U32 *freqs = (U32*) malloc(((U64)1 << ctx->f) * sizeof(U32));
479
+ U32* freqs = (U32*) malloc(((U64)1 << ctx->f) * sizeof(U32));
468
480
  if (!segmentFreqs || !dict || !freqs) {
469
481
  DISPLAYLEVEL(1, "Failed to allocate buffers: out of memory\n");
470
482
  goto _cleanup;
@@ -476,7 +488,7 @@ static void FASTCOVER_tryParameters(void *opaque)
476
488
  parameters, segmentFreqs);
477
489
 
478
490
  const unsigned nbFinalizeSamples = (unsigned)(ctx->nbTrainSamples * ctx->accelParams.finalize / 100);
479
- selection = COVER_selectDict(dict + tail, dictBufferCapacity - tail,
491
+ selection = COVER_selectDict(dict + tail, dictBufferCapacity, dictBufferCapacity - tail,
480
492
  ctx->samples, ctx->samplesSizes, nbFinalizeSamples, ctx->nbTrainSamples, ctx->nbSamples, parameters, ctx->offsets,
481
493
  totalCompressedSize);
482
494
 
@@ -607,7 +619,7 @@ ZDICT_optimizeTrainFromBuffer_fastCover(
607
619
  /* constants */
608
620
  const unsigned nbThreads = parameters->nbThreads;
609
621
  const double splitPoint =
610
- parameters->splitPoint <= 0.0 ? DEFAULT_SPLITPOINT : parameters->splitPoint;
622
+ parameters->splitPoint <= 0.0 ? FASTCOVER_DEFAULT_SPLITPOINT : parameters->splitPoint;
611
623
  const unsigned kMinD = parameters->d == 0 ? 6 : parameters->d;
612
624
  const unsigned kMaxD = parameters->d == 0 ? 8 : parameters->d;
613
625
  const unsigned kMinK = parameters->k == 0 ? 50 : parameters->k;
@@ -1,5 +1,5 @@
1
1
  /*
2
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
2
+ * Copyright (c) Yann Collet, Facebook, Inc.
3
3
  * All rights reserved.
4
4
  *
5
5
  * This source code is licensed under both the BSD-style license (found in the
@@ -23,9 +23,13 @@
23
23
  /* Unix Large Files support (>4GB) */
24
24
  #define _FILE_OFFSET_BITS 64
25
25
  #if (defined(__sun__) && (!defined(__LP64__))) /* Sun Solaris 32-bits requires specific definitions */
26
+ # ifndef _LARGEFILE_SOURCE
26
27
  # define _LARGEFILE_SOURCE
28
+ # endif
27
29
  #elif ! defined(__LP64__) /* No point defining Large file for 64 bit */
30
+ # ifndef _LARGEFILE64_SOURCE
28
31
  # define _LARGEFILE64_SOURCE
32
+ # endif
29
33
  #endif
30
34
 
31
35
 
@@ -37,17 +41,19 @@
37
41
  #include <stdio.h> /* fprintf, fopen, ftello64 */
38
42
  #include <time.h> /* clock */
39
43
 
40
- #include "mem.h" /* read */
41
- #include "fse.h" /* FSE_normalizeCount, FSE_writeNCount */
42
- #define HUF_STATIC_LINKING_ONLY
43
- #include "huf.h" /* HUF_buildCTable, HUF_writeCTable */
44
- #include "zstd_internal.h" /* includes zstd.h */
45
- #include "xxhash.h" /* XXH64 */
46
- #include "divsufsort.h"
47
44
  #ifndef ZDICT_STATIC_LINKING_ONLY
48
45
  # define ZDICT_STATIC_LINKING_ONLY
49
46
  #endif
50
- #include "zdict.h"
47
+ #define HUF_STATIC_LINKING_ONLY
48
+
49
+ #include "../common/mem.h" /* read */
50
+ #include "../common/fse.h" /* FSE_normalizeCount, FSE_writeNCount */
51
+ #include "../common/huf.h" /* HUF_buildCTable, HUF_writeCTable */
52
+ #include "../common/zstd_internal.h" /* includes zstd.h */
53
+ #include "../common/xxhash.h" /* XXH64 */
54
+ #include "../compress/zstd_compress_internal.h" /* ZSTD_loadCEntropy() */
55
+ #include "../zdict.h"
56
+ #include "divsufsort.h"
51
57
 
52
58
 
53
59
  /*-*************************************
@@ -61,14 +67,15 @@
61
67
 
62
68
  #define NOISELENGTH 32
63
69
 
64
- static const int g_compressionLevel_default = 3;
65
70
  static const U32 g_selectivity_default = 9;
66
71
 
67
72
 
68
73
  /*-*************************************
69
74
  * Console display
70
75
  ***************************************/
76
+ #undef DISPLAY
71
77
  #define DISPLAY(...) { fprintf(stderr, __VA_ARGS__); fflush( stderr ); }
78
+ #undef DISPLAYLEVEL
72
79
  #define DISPLAYLEVEL(l, ...) if (notificationLevel>=l) { DISPLAY(__VA_ARGS__); } /* 0 : no display; 1: errors; 2: default; 3: details; 4: debug */
73
80
 
74
81
  static clock_t ZDICT_clockSpan(clock_t nPrevious) { return clock() - nPrevious; }
@@ -99,6 +106,26 @@ unsigned ZDICT_getDictID(const void* dictBuffer, size_t dictSize)
99
106
  return MEM_readLE32((const char*)dictBuffer + 4);
100
107
  }
101
108
 
109
+ size_t ZDICT_getDictHeaderSize(const void* dictBuffer, size_t dictSize)
110
+ {
111
+ size_t headerSize;
112
+ if (dictSize <= 8 || MEM_readLE32(dictBuffer) != ZSTD_MAGIC_DICTIONARY) return ERROR(dictionary_corrupted);
113
+
114
+ { ZSTD_compressedBlockState_t* bs = (ZSTD_compressedBlockState_t*)malloc(sizeof(ZSTD_compressedBlockState_t));
115
+ U32* wksp = (U32*)malloc(HUF_WORKSPACE_SIZE);
116
+ if (!bs || !wksp) {
117
+ headerSize = ERROR(memory_allocation);
118
+ } else {
119
+ ZSTD_reset_compressedBlockState(bs);
120
+ headerSize = ZSTD_loadCEntropy(bs, wksp, dictBuffer, dictSize);
121
+ }
122
+
123
+ free(bs);
124
+ free(wksp);
125
+ }
126
+
127
+ return headerSize;
128
+ }
102
129
 
103
130
  /*-********************************************************
104
131
  * Dictionary training functions
@@ -508,6 +535,7 @@ static size_t ZDICT_trainBuffer_legacy(dictItem* dictList, U32 dictListSize,
508
535
  clock_t displayClock = 0;
509
536
  clock_t const refreshRate = CLOCKS_PER_SEC * 3 / 10;
510
537
 
538
+ # undef DISPLAYUPDATE
511
539
  # define DISPLAYUPDATE(l, ...) if (notificationLevel>=l) { \
512
540
  if (ZDICT_clockSpan(displayClock) > refreshRate) \
513
541
  { displayClock = clock(); DISPLAY(__VA_ARGS__); \
@@ -571,7 +599,7 @@ static void ZDICT_fillNoise(void* buffer, size_t length)
571
599
  unsigned const prime1 = 2654435761U;
572
600
  unsigned const prime2 = 2246822519U;
573
601
  unsigned acc = prime1;
574
- size_t p=0;;
602
+ size_t p=0;
575
603
  for (p=0; p<length; p++) {
576
604
  acc *= prime2;
577
605
  ((unsigned char*)buffer)[p] = (unsigned char)(acc >> 21);
@@ -588,12 +616,12 @@ typedef struct
588
616
 
589
617
  #define MAXREPOFFSET 1024
590
618
 
591
- static void ZDICT_countEStats(EStats_ress_t esr, ZSTD_parameters params,
619
+ static void ZDICT_countEStats(EStats_ress_t esr, const ZSTD_parameters* params,
592
620
  unsigned* countLit, unsigned* offsetcodeCount, unsigned* matchlengthCount, unsigned* litlengthCount, U32* repOffsets,
593
621
  const void* src, size_t srcSize,
594
622
  U32 notificationLevel)
595
623
  {
596
- size_t const blockSizeMax = MIN (ZSTD_BLOCKSIZE_MAX, 1 << params.cParams.windowLog);
624
+ size_t const blockSizeMax = MIN (ZSTD_BLOCKSIZE_MAX, 1 << params->cParams.windowLog);
597
625
  size_t cSize;
598
626
 
599
627
  if (srcSize > blockSizeMax) srcSize = blockSizeMax; /* protection vs large samples */
@@ -682,7 +710,7 @@ static void ZDICT_flatLit(unsigned* countLit)
682
710
 
683
711
  #define OFFCODE_MAX 30 /* only applicable to first block */
684
712
  static size_t ZDICT_analyzeEntropy(void* dstBuffer, size_t maxDstSize,
685
- unsigned compressionLevel,
713
+ int compressionLevel,
686
714
  const void* srcBuffer, const size_t* fileSizes, unsigned nbFiles,
687
715
  const void* dictBuffer, size_t dictBufferSize,
688
716
  unsigned notificationLevel)
@@ -717,7 +745,7 @@ static size_t ZDICT_analyzeEntropy(void* dstBuffer, size_t maxDstSize,
717
745
  memset(repOffset, 0, sizeof(repOffset));
718
746
  repOffset[1] = repOffset[4] = repOffset[8] = 1;
719
747
  memset(bestRepOffset, 0, sizeof(bestRepOffset));
720
- if (compressionLevel==0) compressionLevel = g_compressionLevel_default;
748
+ if (compressionLevel==0) compressionLevel = ZSTD_CLEVEL_DEFAULT;
721
749
  params = ZSTD_getParams(compressionLevel, averageSampleSize, dictBufferSize);
722
750
 
723
751
  esr.dict = ZSTD_createCDict_advanced(dictBuffer, dictBufferSize, ZSTD_dlm_byRef, ZSTD_dct_rawContent, params.cParams, ZSTD_defaultCMem);
@@ -731,7 +759,7 @@ static size_t ZDICT_analyzeEntropy(void* dstBuffer, size_t maxDstSize,
731
759
 
732
760
  /* collect stats on all samples */
733
761
  for (u=0; u<nbFiles; u++) {
734
- ZDICT_countEStats(esr, params,
762
+ ZDICT_countEStats(esr, &params,
735
763
  countLit, offcodeCount, matchLengthCount, litLengthCount, repOffset,
736
764
  (const char*)srcBuffer + pos, fileSizes[u],
737
765
  notificationLevel);
@@ -762,7 +790,7 @@ static size_t ZDICT_analyzeEntropy(void* dstBuffer, size_t maxDstSize,
762
790
  /* note : the result of this phase should be used to better appreciate the impact on statistics */
763
791
 
764
792
  total=0; for (u=0; u<=offcodeMax; u++) total+=offcodeCount[u];
765
- errorCode = FSE_normalizeCount(offcodeNCount, Offlog, offcodeCount, total, offcodeMax);
793
+ errorCode = FSE_normalizeCount(offcodeNCount, Offlog, offcodeCount, total, offcodeMax, /* useLowProbCount */ 1);
766
794
  if (FSE_isError(errorCode)) {
767
795
  eSize = errorCode;
768
796
  DISPLAYLEVEL(1, "FSE_normalizeCount error with offcodeCount \n");
@@ -771,7 +799,7 @@ static size_t ZDICT_analyzeEntropy(void* dstBuffer, size_t maxDstSize,
771
799
  Offlog = (U32)errorCode;
772
800
 
773
801
  total=0; for (u=0; u<=MaxML; u++) total+=matchLengthCount[u];
774
- errorCode = FSE_normalizeCount(matchLengthNCount, mlLog, matchLengthCount, total, MaxML);
802
+ errorCode = FSE_normalizeCount(matchLengthNCount, mlLog, matchLengthCount, total, MaxML, /* useLowProbCount */ 1);
775
803
  if (FSE_isError(errorCode)) {
776
804
  eSize = errorCode;
777
805
  DISPLAYLEVEL(1, "FSE_normalizeCount error with matchLengthCount \n");
@@ -780,7 +808,7 @@ static size_t ZDICT_analyzeEntropy(void* dstBuffer, size_t maxDstSize,
780
808
  mlLog = (U32)errorCode;
781
809
 
782
810
  total=0; for (u=0; u<=MaxLL; u++) total+=litLengthCount[u];
783
- errorCode = FSE_normalizeCount(litLengthNCount, llLog, litLengthCount, total, MaxLL);
811
+ errorCode = FSE_normalizeCount(litLengthNCount, llLog, litLengthCount, total, MaxLL, /* useLowProbCount */ 1);
784
812
  if (FSE_isError(errorCode)) {
785
813
  eSize = errorCode;
786
814
  DISPLAYLEVEL(1, "FSE_normalizeCount error with litLengthCount \n");
@@ -869,7 +897,7 @@ size_t ZDICT_finalizeDictionary(void* dictBuffer, size_t dictBufferCapacity,
869
897
  size_t hSize;
870
898
  #define HBUFFSIZE 256 /* should prove large enough for all entropy headers */
871
899
  BYTE header[HBUFFSIZE];
872
- int const compressionLevel = (params.compressionLevel == 0) ? g_compressionLevel_default : params.compressionLevel;
900
+ int const compressionLevel = (params.compressionLevel == 0) ? ZSTD_CLEVEL_DEFAULT : params.compressionLevel;
873
901
  U32 const notificationLevel = params.notificationLevel;
874
902
 
875
903
  /* check conditions */
@@ -915,7 +943,7 @@ static size_t ZDICT_addEntropyTablesFromBuffer_advanced(
915
943
  const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples,
916
944
  ZDICT_params_t params)
917
945
  {
918
- int const compressionLevel = (params.compressionLevel == 0) ? g_compressionLevel_default : params.compressionLevel;
946
+ int const compressionLevel = (params.compressionLevel == 0) ? ZSTD_CLEVEL_DEFAULT : params.compressionLevel;
919
947
  U32 const notificationLevel = params.notificationLevel;
920
948
  size_t hSize = 8;
921
949
 
@@ -944,16 +972,11 @@ static size_t ZDICT_addEntropyTablesFromBuffer_advanced(
944
972
  return MIN(dictBufferCapacity, hSize+dictContentSize);
945
973
  }
946
974
 
947
- /* Hidden declaration for dbio.c */
948
- size_t ZDICT_trainFromBuffer_unsafe_legacy(
949
- void* dictBuffer, size_t maxDictSize,
950
- const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples,
951
- ZDICT_legacy_params_t params);
952
975
  /*! ZDICT_trainFromBuffer_unsafe_legacy() :
953
- * Warning : `samplesBuffer` must be followed by noisy guard band.
976
+ * Warning : `samplesBuffer` must be followed by noisy guard band !!!
954
977
  * @return : size of dictionary, or an error code which can be tested with ZDICT_isError()
955
978
  */
956
- size_t ZDICT_trainFromBuffer_unsafe_legacy(
979
+ static size_t ZDICT_trainFromBuffer_unsafe_legacy(
957
980
  void* dictBuffer, size_t maxDictSize,
958
981
  const void* samplesBuffer, const size_t* samplesSizes, unsigned nbSamples,
959
982
  ZDICT_legacy_params_t params)
@@ -1090,8 +1113,8 @@ size_t ZDICT_trainFromBuffer(void* dictBuffer, size_t dictBufferCapacity,
1090
1113
  memset(&params, 0, sizeof(params));
1091
1114
  params.d = 8;
1092
1115
  params.steps = 4;
1093
- /* Default to level 6 since no compression level information is available */
1094
- params.zParams.compressionLevel = 3;
1116
+ /* Use default level since no compression level information is available */
1117
+ params.zParams.compressionLevel = ZSTD_CLEVEL_DEFAULT;
1095
1118
  #if defined(DEBUGLEVEL) && (DEBUGLEVEL>=1)
1096
1119
  params.zParams.notificationLevel = DEBUGLEVEL;
1097
1120
  #endif
@@ -1,10 +1,11 @@
1
1
  # ################################################################
2
- # Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
2
+ # Copyright (c) Yann Collet, Facebook, Inc.
3
3
  # All rights reserved.
4
4
  #
5
5
  # This source code is licensed under both the BSD-style license (found in the
6
6
  # LICENSE file in the root directory of this source tree) and the GPLv2 (found
7
7
  # in the COPYING file in the root directory of this source tree).
8
+ # You may select, at your option, one of the above-listed licenses.
8
9
  # ################################################################
9
10
 
10
11
  VOID := /dev/null
@@ -1,23 +1,21 @@
1
- ZSTD Windows binary package
2
- ====================================
1
+ # ZSTD Windows binary package
3
2
 
4
- #### The package contents
3
+ ## The package contents
5
4
 
6
- - `zstd.exe` : Command Line Utility, supporting gzip-like arguments
7
- - `dll\libzstd.dll` : The ZSTD dynamic library (DLL)
8
- - `dll\libzstd.lib` : The import library of the ZSTD dynamic library (DLL) for Visual C++
9
- - `example\` : The example of usage of the ZSTD library
10
- - `include\` : Header files required by the ZSTD library
5
+ - `zstd.exe` : Command Line Utility, supporting gzip-like arguments
6
+ - `dll\libzstd.dll` : The ZSTD dynamic library (DLL)
7
+ - `dll\libzstd.lib` : The import library of the ZSTD dynamic library (DLL) for Visual C++
8
+ - `example\` : The example of usage of the ZSTD library
9
+ - `include\` : Header files required by the ZSTD library
11
10
  - `static\libzstd_static.lib` : The static ZSTD library (LIB)
12
11
 
13
-
14
- #### Usage of Command Line Interface
12
+ ## Usage of Command Line Interface
15
13
 
16
14
  Command Line Interface (CLI) supports gzip-like arguments.
17
15
  By default CLI takes an input file and compresses it to an output file:
18
- ```
16
+
19
17
  Usage: zstd [arg] [input] [output]
20
- ```
18
+
21
19
  The full list of commands for CLI can be obtained with `-h` or `-H`. The ratio can
22
20
  be improved with commands from `-3` to `-16` but higher levels also have slower
23
21
  compression. CLI includes in-memory compression benchmark module with compression
@@ -25,36 +23,32 @@ levels starting from `-b` and ending with `-e` with iteration time of `-i` secon
25
23
  CLI supports aggregation of parameters i.e. `-b1`, `-e18`, and `-i1` can be joined
26
24
  into `-b1e18i1`.
27
25
 
28
-
29
- #### The example of usage of static and dynamic ZSTD libraries with gcc/MinGW
26
+ ## The example of usage of static and dynamic ZSTD libraries with gcc/MinGW
30
27
 
31
28
  Use `cd example` and `make` to build `fullbench-dll` and `fullbench-lib`.
32
29
  `fullbench-dll` uses a dynamic ZSTD library from the `dll` directory.
33
30
  `fullbench-lib` uses a static ZSTD library from the `lib` directory.
34
31
 
35
-
36
- #### Using ZSTD DLL with gcc/MinGW
32
+ ## Using ZSTD DLL with gcc/MinGW
37
33
 
38
34
  The header files from `include\` and the dynamic library `dll\libzstd.dll`
39
35
  are required to compile a project using gcc/MinGW.
40
36
  The dynamic library has to be added to linking options.
41
37
  It means that if a project that uses ZSTD consists of a single `test-dll.c`
42
38
  file it should be linked with `dll\libzstd.dll`. For example:
43
- ```
39
+
44
40
  gcc $(CFLAGS) -Iinclude\ test-dll.c -o test-dll dll\libzstd.dll
45
- ```
46
- The compiled executable will require ZSTD DLL which is available at `dll\libzstd.dll`.
47
41
 
42
+ The compiled executable will require ZSTD DLL which is available at `dll\libzstd.dll`.
48
43
 
49
- #### The example of usage of static and dynamic ZSTD libraries with Visual C++
44
+ ## The example of usage of static and dynamic ZSTD libraries with Visual C++
50
45
 
51
46
  Open `example\fullbench-dll.sln` to compile `fullbench-dll` that uses a
52
47
  dynamic ZSTD library from the `dll` directory. The solution works with Visual C++
53
48
  2010 or newer. When one will open the solution with Visual C++ newer than 2010
54
49
  then the solution will upgraded to the current version.
55
50
 
56
-
57
- #### Using ZSTD DLL with Visual C++
51
+ ## Using ZSTD DLL with Visual C++
58
52
 
59
53
  The header files from `include\` and the import library `dll\libzstd.lib`
60
54
  are required to compile a project using Visual C++.