omnizip 0.3.15 → 0.3.16

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 978a7f51735e4fd90dbb0432b933ccdd17009d74c682db94f6ee6730fe1f893a
-  data.tar.gz: c02f967a72d720edbbc204bd6b36dc60a4c4f773aed3e308800880351a32c218
+  metadata.gz: 4417d9e90649e837f75b25bcee26c0721283e360d015eaf0c4194fa2957be053
+  data.tar.gz: 8c5fc8e793ea023d21abd9b044c3ddfb3786ee90225fe57fb322fa8cc60c5f4a
 SHA512:
-  metadata.gz: 6664a917d2c82bca2779e62a082e1462c1dd8c36f3f4777a47a871feafd4d3bb25e7ae4eb492623a823aed8755777fce93374075a502b0a8beabe2becf9aed05
-  data.tar.gz: ccf0ff7279dc03c4c5aec2fd6a679419c3fa275966e3b7f076e3bdd4e5890e2141664f6137df46681d29738dbf7fb25ef95e1ec3e450c0e3cfc7509dd9f759d7
+  metadata.gz: 2de569d977b1b47bc476efd9eb17e8f00c5b803e96451592964ee0f43bfa04caf52fd6462d02fbcaf9bd914b32436a210b1b935d24d6c1ad96a61acf119c7ca4
+  data.tar.gz: c070c98a67fef377cf8710f89ed922bb0a6ff5cf8edd0ac7307113f4973d07a85c5eb8dfce48210f7d570687d361f68cc61f947d4dc739588f9aee1662126269

data/TODO.refactor/13-lutaml-model-migration.md

@@ -1,8 +1,8 @@
 # 13 — lutaml-model migration
 
 Priority: **medium**.
-Status: TODO (large; this pass establishes the pattern, full migration
-is follow-up).
+Status: DONE for the models that should migrate. The remaining `to_h`
+methods are deliberate — see "Why 17 stay" below.
 
 ## Rule
 
@@ -11,72 +11,136 @@ is follow-up).
 > ... end` block. Never write `def to_h` / `def from_h` / `def to_json`
 > / `def from_json` on a model class.
 
-## Inventory
-
-26 `def to_h` occurrences across 22 files. 2 `def to_json`. None use
-lutaml-model today even though the gemspec declares the dependency.
-
-### Models (lib/omnizip/models/)
-
-- `compression_options.rb` (`to_h`, `to_json`)
-- `algorithm_metadata.rb` (`to_h`, `to_json`)
-- `conversion_result.rb`
-- `eta_result.rb`
-- `filter_config.rb`
-- `match_result.rb`
-- `optimization_suggestion.rb`
-- `parallel_options.rb`
-- `performance_result.rb`
-- `profile_report.rb`
-- `progress_options.rb`
-- `conversion_options.rb`
-
-### Format entry/metadata classes
-
-- `formats/xar/entry.rb`
-- `formats/zip/unix_extra_field.rb`
-- `formats/rpm/header.rb`
-- `metadata/entry_metadata.rb`
-- `metadata/archive_metadata.rb`
-
-### Other
-
-- `progress/operation_progress.rb`
-- `profile/profile_registry.rb`, `custom_profile.rb`, `compression_profile.rb`
-- `link_handler/symbolic_link.rb`, `hard_link.rb`
-
-## Plan (this pass)
-
-1. Establish the pattern by migrating **two representative models**:
-   `Omnizip::Models::CompressionOptions` and
-   `Omnizip::Models::AlgorithmMetadata`. Convert to lutaml-model:
-   ```ruby
-   class CompressionOptions < Lutaml::Model::Serializable
-     attribute :algorithm, :string
-     attribute :level, :integer, default: 6
-     # ...
-     mapping do
-       map :algorithm, to: :algorithm
-       map :level, to: :level
-     end
-   end
-   ```
-   The framework provides `to_hash`, `to_json`, `from_hash`,
-   `from_json` automatically — remove the hand-rolled versions.
-
-2. Delete the manual `to_h` / `to_json` from those two files.
-
-3. Document the migration recipe in this TODO file for the remaining
-   models.
-
-## Acceptance
-
-- Two models migrated; their specs still pass.
-- `grep -rn "def to_h\|def to_json" lib/omnizip/models/` count drops by
-  at least 2.
-
-## Remaining work (future pass)
-
-Migrate the other ~24 files following the established recipe. The
-metadata/entry classes likely share structure and may collapse into a
-smaller set of richer models — review for MECE opportunities.
+## Result
+
+Six models now inherit `Lutaml::Model::Serializable`:
+
+- `CompressionOptions`, `AlgorithmMetadata` (earlier pass)
+- `ConversionOptions`, `ProgressOptions`, `ETAResult`, `ParallelOptions`
+
+Counts, measured against the tree rather than estimated:
+
+| Metric | Before | After |
+| --- | --- | --- |
+| `grep -rn "def to_h" lib/ \| wc -l` | 21 | 17 |
+| Same under `lib/omnizip/models/` | 10 | 6 |
+| `def to_json` / `from_h` / `from_json` in `lib/` | 0 | 0 |
+
+An earlier version of this document claimed "26 `def to_h` across 22
+files, 2 `def to_json`". Both were wrong; the numbers above are real.
+
+## The symbol-type blocker was not real
+
+This document previously held `ConversionOptions` back pending "lutaml
+symbol-type support". There is nothing to wait for. `:symbol` is a
+built-in type from 0.8.0 on: it is registered in `TYPE_CODES` in
+`lib/lutaml/model/type.rb` and implemented in
+`lib/lutaml/model/type/symbol.rb`. It casts symbols, strings, and nil
+correctly and round-trips through JSON. `ConversionOptions` uses it for
+`source_format`, `target_format`, `compression`, and `filter`.
+
+The gemspec asks for `~> 0.8` because of this. `~> 0.7` also resolved to
+0.8 on a fresh install, but it still admitted 0.7.1 through 0.7.7, where
+`:symbol` is unregistered and `default: -> { detect_cpu_count }` is
+evaluated against the class. Pin a build to 0.7.7 and 25 examples fail.
+
+## Recipe
+
+```ruby
+require "lutaml/model"
+
+class Thing < Lutaml::Model::Serializable
+  attribute :level, :integer, default: 5
+  attribute :name, :string
+
+  key_value do
+    map "level", to: :level, render_default: true, render_nil: true
+    map "name", to: :name, render_default: true, render_nil: true
+  end
+end
+```
+
+`key_value` covers hash, YAML, TOML, and JSON in one declaration.
+
+### Two behaviours that will catch you out
+
+**1. `to_hash` returns String keys.** The hand-rolled `to_h` methods
+returned Symbol keys. These are not interchangeable, and
+`Serializable` does not define `to_h` at all — calling it raises
+`NoMethodError`. Removing `to_h` is a breaking change for any external
+caller.
+
+**2. `render_default: true` alone does not give you a fixed key set.**
+It renders an unassigned default, but still drops a key that was
+explicitly assigned `nil`:
+
+| Mapping | fresh `.new` | after `x.a = nil` |
+| --- | --- | --- |
+| `render_default: true` | `{"a" => 5}` | `{}` |
+| `+ render_nil: true` | `{"a" => 5, "b" => nil}` | `{"a" => nil}` |
+
+Use **both** flags when the old `to_h` emitted every key. Use
+`render_default` alone when it ended in `.compact`. Use neither when
+there are no defaults and nils were dropped. There is no class-level or
+global switch — `render_default:` is per-rule and defaults to `false`.
+
+### Other gotchas
+
+- Lambda defaults (`default: -> { detect_cpu_count }`) are evaluated
+  with `self` bound to the **instance**. A private instance method
+  works; a class method raises `NameError`.
+- `dup` shares the internal `@using_default` hash. With both render
+  flags set this is unobservable, so no `dup` override or
+  `initialize_copy` hook is needed.
+- Unknown keyword arguments to `.new` are silently ignored, where a
+  plain-Ruby `initialize` would raise `ArgumentError`.
+- `self.class.attributes` returns a `Hash{Symbol=>Attribute}`. Iterate
+  it with `each_key`.
+
+## Why 17 `to_h` methods stay
+
+These are computed reports, projections over another object, or
+wire-format builders — not serializations of the receiver's own state.
+Converting them would drop the computed fields or force a second method
+back onto the class. Leave them alone.
+
+| File | Reason |
+| --- | --- |
+| `models/conversion_result.rb` | Three computed fields; drops a stored one |
+| `models/match_result.rb` | Keys don't match attributes; renames and computes |
+| `models/optimization_suggestion.rb` | Computed `priority_score`; ctor validates enums |
+| `models/performance_result.rb` | Computed fields plus `timestamp.iso8601` |
+| `models/profile_report.rb` | Nested computed `summary:`; maps nested results |
+| `models/filter_config.rb` | Emits `:name` for `@name_sym`; binary properties; registry lookups |
+| `formats/rpm/header.rb` | Tag-map projection over a collection |
+| `formats/xar/entry.rb` | XAR XML attributes; octal mode, formatted timestamps |
+| `formats/zip/unix_extra_field.rb` | Emits a constant tag; wire format is manual pack |
+| `metadata/entry_metadata.rb` | Pure delegation; stores no attributes of its own |
+| `metadata/archive_metadata.rb` | Computed aggregates |
+| `profile/compression_profile.rb` | `CustomProfile#to_h` calls `super` on it |
+| `profile/custom_profile.rb` | Inheritance chain over `to_h` |
+| `profile/profile_registry.rb` | Registry projection under a mutex |
+| `progress/operation_progress.rb` | Computed percentages and elapsed time |
+| `link_handler/symbolic_link.rb` | Type discriminator; storage uses `serialize` |
+| `link_handler/hard_link.rb` | Type discriminator; storage uses `serialize` |
+
+Because these stay, `grep "def to_h" lib/` will never reach zero. That
+is the intended end state, not unfinished work.
+
+## Follow-up work
+
+1. **Rename the 17.** If they are reports and projections, `to_h` is
+   the wrong name — that is the real defect. Suggested groupings:
+   `to_report` for the computed ones, `to_tag_map` / `to_attributes`
+   for projections, `to_xml_attributes` / `to_fields` for wire-format
+   builders. Each rename is itself a public-API break.
+2. **Fix `apply` in `CompressionOptions` and `AlgorithmMetadata`.**
+   Both iterate `self.class.attributes.each do |attr| ... attr.name`,
+   but `attributes` is a Hash, so the block gets a two-element Array
+   and `attr.name` raises `NoMethodError`. Neither has a caller or a
+   spec today, which is why it went unnoticed. Fix is `each_key`, plus
+   a spec per class.
+3. **Extract the shared `apply` loop** once (2) has landed — three
+   classes now carry a similar body.
+4. **Consider `FilterConfig` and the profile classes** for a proper
+   migration; both were judged too high-blast-radius to bundle here.

data/docs/getting-started/installation.adoc

@@ -31,7 +31,7 @@ ruby --version
 
 Omnizip has minimal dependencies:
 
-* `lutaml-model` (~> 0.7) - Model serialization
+* `lutaml-model` (~> 0.8) - Model serialization
 * `marcel` (~> 1.0) - MIME type detection
 
 These are automatically installed with the gem.

data/lib/omnizip/algorithms/zstandard/encoder.rb

@@ -25,10 +25,13 @@ module Omnizip
     class Zstandard
       # Pure Ruby Zstandard encoder (RFC 8878).
       #
-      # Emits a single-segment frame. Each 128 KiB chunk of input
-      # becomes one block, choosing whichever is smallest: RLE, a
-      # Compressed_Block (Huffman-coded literals + an empty sequences
-      # section), or a Raw_Block fallback for incompressible data.
+      # Emits a single-segment frame. Each 128 KiB chunk becomes one
+      # block, choosing whichever is smallest: RLE, a Compressed_Block
+      # (LZ77 match finding + Huffman literals + FSE-coded sequences),
+      # or a Raw_Block fallback for incompressible data. The match
+      # finder works over absolute positions, so blocks can reference
+      # earlier blocks; repeat-offset state is carried between blocks
+      # exactly as the decoder tracks it.
       class Encoder
         include Constants
 
@@ -83,53 +86,78 @@ module Omnizip
           end
         end
 
+        # rubocop:disable Metrics/MethodLength
+        # rubocop:disable-next Metrics/AbcSize
         def write_blocks(data)
           return write_empty_last_block if data.empty?
 
+          params = MatchFinder.params_for_level(@level, data.bytesize)
+          ms = MatchFinder::MatchState.new(params[:hash_log])
+          ms.enable_chain(params[:chain]) if params[:chain].positive?
+
+          # Wire repeat-offset state carried across blocks; matches the
+          # decoder's executor state. Raw and RLE blocks leave it
+          # untouched.
+          reps = [1, 4, 8]
+
           offset = 0
           while offset < data.bytesize
-            chunk = data.byteslice(offset, BLOCK_MAX_SIZE)
-            offset += chunk.bytesize
-            is_last = offset >= data.bytesize
-            write_chunk(chunk, is_last)
+            block_end = [offset + BLOCK_MAX_SIZE, data.bytesize].min
+            is_last = block_end == data.bytesize
+            chunk = data.byteslice(offset, block_end - offset)
+
+            if chunk.bytesize >= 4 && rle_chunk?(chunk)
+              write_block_header(is_last ? 1 : 0, BLOCK_TYPE_RLE,
+                                 chunk.bytesize)
+              @output_stream.putc(chunk.getbyte(0))
+            else
+              seq_store = MatchFinder::SeqStore.new(reps.dup)
+              MatchFinder.compress_range(data, offset, block_end, seq_store,
+                                         ms, params[:min_match],
+                                         params[:lazy])
+              content, new_reps = try_compressed(seq_store, reps)
+              if content.nil?
+                write_block_header(is_last ? 1 : 0, BLOCK_TYPE_RAW,
+                                   chunk.bytesize)
+                @output_stream.write(chunk)
+              else
+                write_block_header(is_last ? 1 : 0, BLOCK_TYPE_COMPRESSED,
+                                   content.bytesize)
+                @output_stream.write(content)
+                reps = new_reps
+              end
+            end
+
+            offset = block_end
           end
         end
+        # rubocop:enable Metrics/MethodLength
 
         def write_empty_last_block
           write_block_header(1, BLOCK_TYPE_RAW, 0)
         end
 
-        def write_chunk(chunk, is_last)
-          if chunk.bytesize >= 4 && rle_chunk?(chunk)
-            write_block_header(is_last ? 1 : 0, BLOCK_TYPE_RLE, chunk.bytesize)
-            @output_stream.putc(chunk.getbyte(0))
-            return
-          end
-
-          content = try_compressed(chunk)
-          if content.nil?
-            write_block_header(is_last ? 1 : 0, BLOCK_TYPE_RAW,
-                               chunk.bytesize)
-            @output_stream.write(chunk)
-          else
-            write_block_header(is_last ? 1 : 0, BLOCK_TYPE_COMPRESSED,
-                               content.bytesize)
-            @output_stream.write(content)
-          end
-        end
-
-        # Build a Compressed_Block content: literals section plus an
-        # empty sequences section (nbSeq = 0). Returns nil when the
-        # result is not smaller than the raw chunk.
-        def try_compressed(chunk)
-          literals_section = LiteralsEncoder.encode(chunk)
-          # A single 0x00 byte: Number_of_Sequences = 0.
-          content = literals_section + "\x00".b
-          return nil if content.bytesize >= chunk.bytesize
-
-          content
+        # Build a Compressed_Block content: literals section plus the
+        # sequences section. Returns [content, wire_reps], or [nil,
+        # nil] when the result is not smaller than the raw chunk.
+        #
+        # The chunk's decompressed size is the literal bytes plus every
+        # sequence's match length.
+        #
+        # @param initial_reps [Array<Integer>] wire rep state the
+        #   decoder holds before this block
+        def try_compressed(seq_store, initial_reps)
+          literals_section = LiteralsEncoder.encode(seq_store.literals)
+          sequences_section, wire_reps =
+            SequencesEncoder.encode_section(seq_store, initial_reps)
+          content = literals_section + sequences_section
+          chunk_size = seq_store.literals.bytesize +
+            seq_store.sequences.sum(&:match_length)
+          return [nil, nil] if content.bytesize >= chunk_size
+
+          [content, wire_reps]
         rescue Omnizip::CompressionError
-          nil
+          [nil, nil]
         end
 
         def rle_chunk?(chunk)

data/lib/omnizip/algorithms/zstandard/huffman_encoder.rb

@@ -105,61 +105,75 @@ module Omnizip
           lengths
         end
 
-        # Cap code lengths at max_len while preserving the Kraft
-        # inequality: shorten the longest codes and lengthen the
-        # shortest until the sum of 2^-len is <= 1.
+        # Cap code lengths at max_len and repair the Kraft sum to be
+        # EXACTLY 2^max_len, in integer arithmetic. An exact sum is
+        # what lets the decoder re-derive the dropped last weight as a
+        # clean power of two — a float-tolerance repair can leave the
+        # remainder non-power-of-two and the frame undecodable.
         #
         # @param lengths [Array<Integer>] mutated in place
         # @param max_len [Integer]
         # @param freqs [Array<Integer>]
         # @return [Array<Integer>]
+        # rubocop:disable Metrics/MethodLength
+        # rubocop:disable-next Metrics/AbcSize
         def limit_lengths(lengths, max_len, freqs)
-          loop do
-            cur_max = lengths.max || 0
-            break if cur_max <= max_len
+          lengths.map! { |l| [l, max_len].min }
 
-            longest = nil
-            lengths.each_with_index do |l, i|
-              next unless l == cur_max
-              next if longest && freqs[i] >= freqs[longest]
+          target = 1 << max_len
+          kraft = lengths.sum { |l| l.positive? ? 1 << (max_len - l) : 0 }
 
-              longest = i
+          # Over-subscribed: lengthen the least-frequent shortest
+          # codes until the sum fits.
+          while kraft > target
+            idx = kraft_adjust_index(lengths, freqs) do |l|
+              l.positive? && l < max_len
             end
+            lengths[idx] += 1
+            kraft -= 1 << (max_len - lengths[idx])
+          end
 
-            shortest = nil
+          # Under-subscribed: shorten the most-frequent code that fits
+          # without overshooting; if none fits, lengthen one and let
+          # the first loop re-fit.
+          while kraft < target
+            best = nil
             lengths.each_with_index do |l, i|
-              next unless l.positive? && l < max_len
-              next if shortest && freqs[i] <= freqs[shortest]
+              next unless l > 1
 
-              shortest = i
-            end
+              contribution = 1 << (max_len - l)
+              next if kraft + contribution > target
+              next if best && freqs[i] <= freqs[best]
 
-            if longest && shortest
-              lengths[longest] -= 1
-              lengths[shortest] += 1
+              best = i
+            end
+            if best.nil?
+              idx = kraft_adjust_index(lengths, freqs) do |l|
+                l.positive? && l < max_len
+              end
+              lengths[idx] += 1
+              kraft -= 1 << (max_len - lengths[idx])
             else
-              lengths.map! { |l| [l, max_len].min }
-              break
+              delta = 1 << (max_len - lengths[best])
+              lengths[best] -= 1
+              kraft += delta
             end
           end
 
-          loop do
-            kraft = lengths.sum { |l| l.positive? ? 2.0**-l : 0.0 }
-            break if kraft <= 1.0 + 1e-10
-
-            min_idx = nil
-            lengths.each_with_index do |l, i|
-              next unless l.positive? && l < max_len
-              next if min_idx && lengths[i] >= lengths[min_idx]
+          lengths
+        end
+        # rubocop:enable Metrics/MethodLength
 
-              min_idx = i
-            end
-            break if min_idx.nil?
+        # Index of the least-frequent symbol matching the block.
+        def kraft_adjust_index(lengths, freqs)
+          best = nil
+          lengths.each_with_index do |l, i|
+            next unless yield(l)
+            next if best && freqs[i] >= freqs[best]
 
-            lengths[min_idx] += 1
+            best = i
           end
-
-          lengths
+          best
         end
 
         # Serialize the weight table. Direct 4-bit weights when the