omnizip 0.3.14 → 0.3.16

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3d76166f4e6d65cb67526c079369395377bc4d30af75d52b9d83374f99721071
-  data.tar.gz: c47f5c274906ca6d92293606add01a44bfdb4e3f2f499647608bd2ac0629b98f
+  metadata.gz: 4417d9e90649e837f75b25bcee26c0721283e360d015eaf0c4194fa2957be053
+  data.tar.gz: 8c5fc8e793ea023d21abd9b044c3ddfb3786ee90225fe57fb322fa8cc60c5f4a
 SHA512:
-  metadata.gz: 4cc7cbf7d9ce6763e6648abc3415b588a5a79df8e6b15fdfea5a6db962531b7c584628c1fe1b2c5c604d4113d05f9120bd8eb61a1fba70acdbfa324b3e621a2f
-  data.tar.gz: b796a58122e8d1b9ce208aafbf2b64826789e34fd8d165f05bab54ebd74b9169bdb1814bb8f83f2209bc09abad465a5b5b73c12be8570e3b2429a3288119563f
+  metadata.gz: 2de569d977b1b47bc476efd9eb17e8f00c5b803e96451592964ee0f43bfa04caf52fd6462d02fbcaf9bd914b32436a210b1b935d24d6c1ad96a61acf119c7ca4
+  data.tar.gz: c070c98a67fef377cf8710f89ed922bb0a6ff5cf8edd0ac7307113f4973d07a85c5eb8dfce48210f7d570687d361f68cc61f947d4dc739588f9aee1662126269

data/TODO.refactor/13-lutaml-model-migration.md

@@ -1,8 +1,8 @@
 # 13 — lutaml-model migration
 
 Priority: **medium**.
-Status: TODO (large; this pass establishes the pattern, full migration
-is follow-up).
+Status: DONE for the models that should migrate. The remaining `to_h`
+methods are deliberate — see "Why 17 stay" below.
 
 ## Rule
 
@@ -11,72 +11,136 @@ is follow-up).
 > ... end` block. Never write `def to_h` / `def from_h` / `def to_json`
 > / `def from_json` on a model class.
 
-## Inventory
-
-26 `def to_h` occurrences across 22 files. 2 `def to_json`. None use
-lutaml-model today even though the gemspec declares the dependency.
-
-### Models (lib/omnizip/models/)
-
-- `compression_options.rb` (`to_h`, `to_json`)
-- `algorithm_metadata.rb` (`to_h`, `to_json`)
-- `conversion_result.rb`
-- `eta_result.rb`
-- `filter_config.rb`
-- `match_result.rb`
-- `optimization_suggestion.rb`
-- `parallel_options.rb`
-- `performance_result.rb`
-- `profile_report.rb`
-- `progress_options.rb`
-- `conversion_options.rb`
-
-### Format entry/metadata classes
-
-- `formats/xar/entry.rb`
-- `formats/zip/unix_extra_field.rb`
-- `formats/rpm/header.rb`
-- `metadata/entry_metadata.rb`
-- `metadata/archive_metadata.rb`
-
-### Other
-
-- `progress/operation_progress.rb`
-- `profile/profile_registry.rb`, `custom_profile.rb`, `compression_profile.rb`
-- `link_handler/symbolic_link.rb`, `hard_link.rb`
-
-## Plan (this pass)
-
-1. Establish the pattern by migrating **two representative models**:
-   `Omnizip::Models::CompressionOptions` and
-   `Omnizip::Models::AlgorithmMetadata`. Convert to lutaml-model:
-   ```ruby
-   class CompressionOptions < Lutaml::Model::Serializable
-     attribute :algorithm, :string
-     attribute :level, :integer, default: 6
-     # ...
-     mapping do
-       map :algorithm, to: :algorithm
-       map :level, to: :level
-     end
-   end
-   ```
-   The framework provides `to_hash`, `to_json`, `from_hash`,
-   `from_json` automatically — remove the hand-rolled versions.
-
-2. Delete the manual `to_h` / `to_json` from those two files.
-
-3. Document the migration recipe in this TODO file for the remaining
-   models.
-
-## Acceptance
-
-- Two models migrated; their specs still pass.
-- `grep -rn "def to_h\|def to_json" lib/omnizip/models/` count drops by
-  at least 2.
-
-## Remaining work (future pass)
-
-Migrate the other ~24 files following the established recipe. The
-metadata/entry classes likely share structure and may collapse into a
-smaller set of richer models — review for MECE opportunities.
+## Result
+
+Six models now inherit `Lutaml::Model::Serializable`:
+
+- `CompressionOptions`, `AlgorithmMetadata` (earlier pass)
+- `ConversionOptions`, `ProgressOptions`, `ETAResult`, `ParallelOptions`
+
+Counts, measured against the tree rather than estimated:
+
+| Metric | Before | After |
+| --- | --- | --- |
+| `grep -rn "def to_h" lib/ \| wc -l` | 21 | 17 |
+| Same under `lib/omnizip/models/` | 10 | 6 |
+| `def to_json` / `from_h` / `from_json` in `lib/` | 0 | 0 |
+
+An earlier version of this document claimed "26 `def to_h` across 22
+files, 2 `def to_json`". Both were wrong; the numbers above are real.
+
+## The symbol-type blocker was not real
+
+This document previously held `ConversionOptions` back pending "lutaml
+symbol-type support". There is nothing to wait for. `:symbol` is a
+built-in type from 0.8.0 on: it is registered in `TYPE_CODES` in
+`lib/lutaml/model/type.rb` and implemented in
+`lib/lutaml/model/type/symbol.rb`. It casts symbols, strings, and nil
+correctly and round-trips through JSON. `ConversionOptions` uses it for
+`source_format`, `target_format`, `compression`, and `filter`.
+
+The gemspec asks for `~> 0.8` because of this. `~> 0.7` also resolved to
+0.8 on a fresh install, but it still admitted 0.7.1 through 0.7.7, where
+`:symbol` is unregistered and `default: -> { detect_cpu_count }` is
+evaluated against the class. Pin a build to 0.7.7 and 25 examples fail.
+
+## Recipe
+
+```ruby
+require "lutaml/model"
+
+class Thing < Lutaml::Model::Serializable
+  attribute :level, :integer, default: 5
+  attribute :name, :string
+
+  key_value do
+    map "level", to: :level, render_default: true, render_nil: true
+    map "name", to: :name, render_default: true, render_nil: true
+  end
+end
+```
+
+`key_value` covers hash, YAML, TOML, and JSON in one declaration.
+
+### Two behaviours that will catch you out
+
+**1. `to_hash` returns String keys.** The hand-rolled `to_h` methods
+returned Symbol keys. These are not interchangeable, and
+`Serializable` does not define `to_h` at all — calling it raises
+`NoMethodError`. Removing `to_h` is a breaking change for any external
+caller.
+
+**2. `render_default: true` alone does not give you a fixed key set.**
+It renders an unassigned default, but still drops a key that was
+explicitly assigned `nil`:
+
+| Mapping | fresh `.new` | after `x.a = nil` |
+| --- | --- | --- |
+| `render_default: true` | `{"a" => 5}` | `{}` |
+| `+ render_nil: true` | `{"a" => 5, "b" => nil}` | `{"a" => nil}` |
+
+Use **both** flags when the old `to_h` emitted every key. Use
+`render_default` alone when it ended in `.compact`. Use neither when
+there are no defaults and nils were dropped. There is no class-level or
+global switch — `render_default:` is per-rule and defaults to `false`.
+
+### Other gotchas
+
+- Lambda defaults (`default: -> { detect_cpu_count }`) are evaluated
+  with `self` bound to the **instance**. A private instance method
+  works; a class method raises `NameError`.
+- `dup` shares the internal `@using_default` hash. With both render
+  flags set this is unobservable, so no `dup` override or
+  `initialize_copy` hook is needed.
+- Unknown keyword arguments to `.new` are silently ignored, where a
+  plain-Ruby `initialize` would raise `ArgumentError`.
+- `self.class.attributes` returns a `Hash{Symbol=>Attribute}`. Iterate
+  it with `each_key`.
+
+## Why 17 `to_h` methods stay
+
+These are computed reports, projections over another object, or
+wire-format builders — not serializations of the receiver's own state.
+Converting them would drop the computed fields or force a second method
+back onto the class. Leave them alone.
+
+| File | Reason |
+| --- | --- |
+| `models/conversion_result.rb` | Three computed fields; drops a stored one |
+| `models/match_result.rb` | Keys don't match attributes; renames and computes |
+| `models/optimization_suggestion.rb` | Computed `priority_score`; ctor validates enums |
+| `models/performance_result.rb` | Computed fields plus `timestamp.iso8601` |
+| `models/profile_report.rb` | Nested computed `summary:`; maps nested results |
+| `models/filter_config.rb` | Emits `:name` for `@name_sym`; binary properties; registry lookups |
+| `formats/rpm/header.rb` | Tag-map projection over a collection |
+| `formats/xar/entry.rb` | XAR XML attributes; octal mode, formatted timestamps |
+| `formats/zip/unix_extra_field.rb` | Emits a constant tag; wire format is manual pack |
+| `metadata/entry_metadata.rb` | Pure delegation; stores no attributes of its own |
+| `metadata/archive_metadata.rb` | Computed aggregates |
+| `profile/compression_profile.rb` | `CustomProfile#to_h` calls `super` on it |
+| `profile/custom_profile.rb` | Inheritance chain over `to_h` |
+| `profile/profile_registry.rb` | Registry projection under a mutex |
+| `progress/operation_progress.rb` | Computed percentages and elapsed time |
+| `link_handler/symbolic_link.rb` | Type discriminator; storage uses `serialize` |
+| `link_handler/hard_link.rb` | Type discriminator; storage uses `serialize` |
+
+Because these stay, `grep "def to_h" lib/` will never reach zero. That
+is the intended end state, not unfinished work.
+
+## Follow-up work
+
+1. **Rename the 17.** If they are reports and projections, `to_h` is
+   the wrong name — that is the real defect. Suggested groupings:
+   `to_report` for the computed ones, `to_tag_map` / `to_attributes`
+   for projections, `to_xml_attributes` / `to_fields` for wire-format
+   builders. Each rename is itself a public-API break.
+2. **Fix `apply` in `CompressionOptions` and `AlgorithmMetadata`.**
+   Both iterate `self.class.attributes.each do |attr| ... attr.name`,
+   but `attributes` is a Hash, so the block gets a two-element Array
+   and `attr.name` raises `NoMethodError`. Neither has a caller or a
+   spec today, which is why it went unnoticed. Fix is `each_key`, plus
+   a spec per class.
+3. **Extract the shared `apply` loop** once (2) has landed — three
+   classes now carry a similar body.
+4. **Consider `FilterConfig` and the profile classes** for a proper
+   migration; both were judged too high-blast-radius to bundle here.

data/docs/getting-started/installation.adoc

@@ -31,7 +31,7 @@ ruby --version
 
 Omnizip has minimal dependencies:
 
-* `lutaml-model` (~> 0.7) - Model serialization
+* `lutaml-model` (~> 0.8) - Model serialization
 * `marcel` (~> 1.0) - MIME type detection
 
 These are automatically installed with the gem.

data/lib/omnizip/algorithms/lzma/optimal_encoder.rb

@@ -100,13 +100,14 @@ module Omnizip
 
           # We didn't find a long enough repeated match.
           # Encode it as a normal match if the match length is at least nice_len
-          # AND the match distance is within the dictionary buffer.
+          # AND the match distance stays within the data decoded so far.
+          # A match may reference anything before `position`: bytes already
+          # committed to the dictionary *and* bytes decoded earlier within
+          # the chunk currently being encoded.
           best_normal = matches.max_by { |m| [m.length, m.distance] }
 
           if best_normal && best_normal.length >= nice_len &&
-              best_normal.distance <= match_finder.dictionary.buffer.bytesize
-            # CRITICAL: Only use normal match if distance is within actual dictionary buffer
-            # This prevents invalid matches that reference bytes not yet written to dictionary
+              best_normal.distance <= position
             # Use normal match
             return [best_normal.distance + REPS, best_normal.length]
           end
@@ -116,10 +117,8 @@ module Omnizip
           if rep_len >= 2 && rep_len >= (best_normal&.length || 0)
             # Use repeated match
             [rep_index, rep_len]
-          elsif best_normal && best_normal.distance <= match_finder.dictionary.buffer.bytesize
-            # CRITICAL: Only use normal match if distance is within actual dictionary buffer
-            # Use dictionary.buffer.bytesize (actual data), NOT dictionary.size (max capacity)
-            # This prevents invalid matches that reference bytes not yet written to dictionary
+          elsif best_normal && best_normal.distance <= position
+            # Use normal match (any length beats encoding literals)
             [best_normal.distance + REPS, best_normal.length]
           else
             # Use literal - return UINT32_MAX to indicate literal (not 0!)

data/lib/omnizip/algorithms/lzma2/lzma2_chunk.rb

@@ -152,7 +152,11 @@ module Omnizip
             comp_size & 0xFF,            # Compressed size low byte (BIG-ENDIAN)
           ].pack("CCCCC")
 
-          prop_bytes = @properties ? [@properties].pack("C") : ""
+          # The properties byte is only present when the control byte
+          # announces it (reset level >= 2, i.e. 0xC0/0xE0). Writing it for
+          # plain continuation chunks (0x80) desynchronizes the decoder,
+          # which does not expect a byte there.
+          prop_bytes = @need_props && @properties ? [@properties].pack("C") : ""
           header + prop_bytes + @compressed_data
         end
       end

data/lib/omnizip/algorithms/zstandard/constants.rb

@@ -73,10 +73,12 @@ module Omnizip
         HUFFMAN_MAX_LOG = 11
         HUFFMAN_MAX_CODE_LENGTH = 11
         HUFFMAN_STANDARD_TABLE_SIZE = 256
+        HUF_SYMBOLVALUE_MAX = 255
 
         # FSE Table Limits (RFC 8878 Section 4.1)
         FSE_MAX_ACCURACY_LOG = 9
         FSE_MIN_ACCURACY_LOG = 5
+        FSE_DEFAULT_TABLELOG = 6
 
         # Compression levels
         MIN_LEVEL = 1
@@ -86,55 +88,66 @@ module Omnizip
         # Buffer size for streaming operations
         BUFFER_SIZE = 128 * 1024 # 128KB
 
-        # Literal length codes (RFC 8878 Table 9)
+        # Literal length codes (RFC 8878 Table: literals length codes)
         # Each entry: [baseline, extra_bits]
         LITERAL_LENGTH_TABLE = [
           [0, 0], [1, 0], [2, 0], [3, 0], [4, 0], [5, 0], [6, 0], [7, 0],
           [8, 0], [9, 0], [10, 0], [11, 0], [12, 0], [13, 0], [14, 0], [15, 0],
-          [16, 1], [18, 1], [20, 1], [22, 1], [24, 1], [28, 1], [32, 1], [40, 1],
-          [48, 1], [64, 1], [128, 2], [256, 2], [512, 2], [1024, 2], [2048, 2],
-          [4096, 2], [8192, 2], [16384, 3], [32768, 3], [65536, 3]
+          [16, 1], [18, 1], [20, 1], [22, 1], [24, 2], [28, 2], [32, 3], [40, 3],
+          [48, 4], [64, 6], [128, 7], [256, 8], [512, 9], [1024, 10], [2048, 11],
+          [4096, 12], [8192, 13], [16384, 14], [32768, 15], [65536, 16]
         ].freeze
 
-        # Match length codes (RFC 8878 Table 10)
+        # Match length codes (RFC 8878 Table: match length codes, 53 entries)
         # Each entry: [baseline, extra_bits]
         MATCH_LENGTH_TABLE = [
           [3, 0], [4, 0], [5, 0], [6, 0], [7, 0], [8, 0], [9, 0], [10, 0],
           [11, 0], [12, 0], [13, 0], [14, 0], [15, 0], [16, 0], [17, 0], [18, 0],
           [19, 0], [20, 0], [21, 0], [22, 0], [23, 0], [24, 0], [25, 0], [26, 0],
           [27, 0], [28, 0], [29, 0], [30, 0], [31, 0], [32, 0], [33, 0], [34, 0],
-          [35, 1], [37, 1], [39, 1], [41, 1], [43, 1], [47, 1], [51, 1], [59, 1],
-          [67, 1], [83, 1], [99, 1], [131, 2], [195, 2], [259, 2], [323, 2],
-          [387, 2], [451, 2], [515, 2], [579, 2], [643, 2], [707, 2], [771, 2],
-          [835, 2], [899, 2], [963, 2], [1027, 2], [1283, 2], [1539, 2],
-          [1795, 2], [2051, 2], [2307, 2], [2563, 2]
+          [35, 1], [37, 1], [39, 1], [41, 1], [43, 2], [47, 2], [51, 3], [59, 3],
+          [67, 4], [83, 4], [99, 5], [131, 7], [259, 8], [515, 9], [1027, 10],
+          [2051, 11], [4099, 12], [8195, 13], [16387, 14], [32771, 15], [65539, 16]
         ].freeze
 
-        # Predefined FSE distribution for literals length (RFC 8878 Section 4.1.3)
+        # Offset codes (RFC 8878 Section 3.1.2.2.3.2)
+        # OF_BASE[c] + read(c bits) = match distance for c >= 2; codes 0 and
+        # 1 are the repeat-offset specials handled by the executor.
+        OF_BASE = [
+          0, 1, 1, 5, 13, 29, 61, 125, 253, 509, 1021, 2045, 4093, 8189,
+          16381, 32765, 65533, 131069, 262141, 524285, 1048573, 2097149,
+          4194301, 8388605, 16777213, 33554429, 67108861, 134217725,
+          268435453, 536870909, 1073741821, 2147483645
+        ].freeze
+        OF_BITS = (0..31).to_a.freeze
+
+        # Predefined FSE distributions (RFC 8878 Section 4.1.3, matching
+        # the zstd C reference). -1 marks a "less than 1" probability that
+        # occupies a single cell at the top of the table.
         PREDEFINED_LL_DISTRIBUTION = [
-          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
-          4, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
-          0, 0, 0, 0
+          4, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1,
+          2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 2, 1, 1, 1, 1, 1,
+          -1, -1, -1, -1
         ].freeze
 
-        # Predefined FSE distribution for match length (RFC 8878 Section 4.1.3)
-        # Sum = 64 (must equal 2^6 = 64)
+        # Sum: 4+3+2*11+1*3 + 2*8+3+2*2+1*5 = 32 + 32 = 64 = 2^6
         PREDEFINED_ML_DISTRIBUTION = [
           1, 4, 3, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
-          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
-          1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
-          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
-          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
-          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
-          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, -1, -1,
+          -1, -1, -1, -1, -1
         ].freeze
 
-        # Predefined FSE distribution for offset (RFC 8878 Section 4.1.3)
+        # Sum: 27 positive + 5 low-probability = 32 = 2^5
         PREDEFINED_OFFSET_DISTRIBUTION = [
-          1, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1, 1, 1, 1, 1, 0,
-          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+          1, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1,
+          1, 1, 1, 1, 1, 1, 1, 1, -1, -1, -1, -1, -1, 0, 0, 0
         ].freeze
+
+        # floor(log2(x)); 0 for x == 0 (C ZSTD_highbit32 defensive value)
+        def self.highbit32(value)
+          value.zero? ? 0 : value.bit_length - 1
+        end
       end
     end
   end