langextract 0.3.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 18f4df67c5283fcde932a3a75aa0d4c0a02978b90a56d68f44219dc89a8081fe
-  data.tar.gz: d0cc9f1c38f50535e365adbabfa91ee31d69713716a3ec0fd13f9f05b98cb606
+  metadata.gz: 36a0e06e39f5d20e93edcfb3ff08982019bf38f350b13145a850667763e10dae
+  data.tar.gz: b6249cecfd2b08bc6e98c1806df03b661336d9731638994468b92fdd58a09894
 SHA512:
-  metadata.gz: deb42f645fabcea65211bc2bc12348f6ba278fa3e1213c8d54c548b1006515e1d442c6aea9a09bb88431401812acaf1400a01aa97f23403af8977f9f59f370c1
-  data.tar.gz: d3d9f43beef6ace15dd65c24250bb991066d5b6f0d7dbf3a073e36f087f157e02633f1273b7476d738b8ea9ea1294b5b12718eedd3dba0d27d0d99a750ca4a06
+  metadata.gz: 18f077f29df06209acc5565692fb633801df636c758dde967a76003dc837f4426e17189fb1df75cb667ff0313626308eac95dab0b710c364f1d59b25fcb3e3f7
+  data.tar.gz: 28229dbda5f455dea01d1293d0f8c5297f5222a77d6c5040d821d62f72de95109b19be8f572d04e852e3e210ba7a96c772f82b9920ae4253d84d923becd6e5e3

data/CHANGELOG.md

@@ -2,6 +2,20 @@
 
 All notable changes to this project will be documented in this file.
 
+## [0.4.0] - 2026-07-16
+
+### Added
+- `LangExtract::ModelConfig` now exposes opt-in `structured_output: true` (default `false`) for RubyLLM schema-constrained responses via `with_schema`. Non-boolean values are rejected, and schema-constrained Hash/Array responses are normalized back to JSON for the existing format handler. RubyLLM 1.16.0 or newer is required for this provider path. ([#13](https://github.com/dpaluy/langextract/pull/13))
+- Fuzzy resolver alignment now operates on ordered token subsequences with per-token similarity, aggregate threshold, coverage and density gates, and sentence/negation barriers. Dash/space, standalone comma, apostrophe, and numeric-grouping variants ground symmetrically while preserving original offsets. One-edit tokens of 3–5 characters require another exact token in the same alignment, and every target-side negation must match an equivalent source negation. The default minimum density is 0.50. ([#14](https://github.com/dpaluy/langextract/pull/14))
+
+### Changed
+- Fuzzy alignment responsibilities are split across token-stream, similarity, index, planner, and aligner components. Indexed binary-search lookups replace repeated forward scans, and the historical 4,000-start truncation is no longer applied while candidate ordering and source offsets remain deterministic.
+- Fuzzy planning skips search ranges larger than 20,000 tokens instead of building an unbounded target-by-source table. Exact alignment and preferred chunk-range fuzzy alignment still run; oversized ranges are skipped as a whole rather than truncating later candidate starts.
+- Declared `logger` as a runtime dependency because it is no longer a default gem on Ruby 4.0; clean installations can load LangExtract without requiring applications to add it separately.
+
+### Tests
+- Added resolver grounding regressions for symmetric formatting variants, contextual short typos, coverage/density boundaries, and source- and target-side semantic barriers; complexity and performance guards for indexed alignment; expanded upstream parity fixtures; and provider/factory coverage for schema-constrained output.
+
 ## [0.3.0] - 2026-07-11
 
 ### Fixed

data/README.md

@@ -18,11 +18,32 @@ Use it when a Ruby or Rails app needs structured LLM output that can be traced b
 - **Format handling** — JSON and YAML output parsing with strict and lenient modes
 - **Provider-agnostic** — pluggable LLM providers via [RubyLLM](https://github.com/crmne/ruby_llm)
 
+## Why not just use RubyLLM directly?
+
+RubyLLM is a provider adapter: it sends a prompt to an LLM and hands you back text. LangExtract is an extraction pipeline that uses RubyLLM as one interchangeable layer at the bottom (`providers/`). The resolver and the rest of the core know nothing about providers.
+
+Reach for RubyLLM directly when you want an answer from a model: summarize, answer a question, classify into buckets. You do not need LangExtract's grounding machinery for those.
+
+Reach for LangExtract when you need structured spans provably tied to your source text. Used directly, RubyLLM leaves you to build everything below yourself:
+
+| Concern | RubyLLM directly | LangExtract gem |
+|---|---|---|
+| **Source grounding** | The model says "Acme Corp appears" but not *where*, and may paraphrase | Exact + token-level fuzzy alignment maps grounded extractions to precise char/token offsets in the source |
+| **Long input** | You chunk manually and hope offsets survive | Sentence-aware chunking with `max_char_buffer`, offsets preserved across chunks |
+| **Output parsing** | You parse JSON/YAML, fenced blocks, and malformed output yourself | `format_handler` with strict/lenient modes and fenced-output extraction |
+| **Prompting** | You write and maintain the prompt | Prompt builder with few-shot `ExampleData` and context-window handling |
+| **Hallucination control** | The model can invent text that isn't in the source | Alignment flags extractions that don't ground to real spans |
+| **Overlaps / dedup** | Your problem | Resolver handles overlapping and duplicate spans |
+| **Visualization** | None | Self-contained HTML highlighting extractions in the source |
+| **Persistence** | None | Lossless JSONL round-trip (`IO.save` / `IO.load`) |
+
+In short: RubyLLM gets you an answer from a model; LangExtract gets you structured spans tied to your source. RubyLLM is a dependency the gem deliberately keeps swappable, not the feature.
+
 ## Requirements
 
 - Ruby >= 4.0.5
 - Tested on Ruby 4.0.5
-- Optional live inference adapter: `ruby_llm` >= 1.0 when using `LangExtract::Factory.create_model`
+- Optional live inference adapter: `ruby_llm` >= 1.16.0 when using `LangExtract::Factory.create_model`
 
 ## Installation
 
@@ -69,6 +90,18 @@ model = LangExtract::Factory.create_model(
 
 If you omit `model`, RubyLLM's configured `default_model` is used.
 
+Set `structured_output: true` to request RubyLLM's schema-constrained extraction envelope. It defaults to `false`, preserving the normal free-form response path:
+
+```ruby
+model = LangExtract::Factory.create_model(
+  LangExtract::ModelConfig.new(
+    model: "gpt-4o-mini",
+    provider: "openai",
+    structured_output: true
+  )
+)
+```
+
 ### Rails
 
 Create `config/initializers/langextract.rb`:
@@ -121,6 +154,12 @@ first.char_interval.end_pos
 first.alignment_status
 ```
 
+### Source alignment
+
+LangExtract first searches for exact source text, then falls back to token-level fuzzy alignment. Fuzzy matching preserves token order and applies per-token similarity, coverage, density, and aggregate-threshold gates. Dash/space, standalone comma, apostrophe, and numeric-grouping variants can ground in either direction while retaining original offsets. One-edit tokens of 3–5 characters are accepted only when another aligned token is exact. Every target negation must match an equivalent source negation, and sentence boundaries or source-side negations cannot be crossed. Same-sentence gaps require at least 0.50 matched-token density.
+
+Grounded extractions report `exact` or `fuzzy`; when no candidate meets the gates, the extraction reports `ungrounded` without a source interval. Fuzzy planning is limited to 20,000 source tokens per search range to prevent unbounded CPU work on large documents. Exact matching remains available at any size, and the normal extraction pipeline first searches its preferred chunk range (2,000 characters by default). Oversized fuzzy ranges are skipped as a whole rather than truncating later candidates. Adjust the aggregate gate with `fuzzy_threshold:` when calling `LangExtract.extract` (default `0.78`).
+
 ### Document collections
 
 ```ruby

data/lib/langextract/core/fuzzy_aligner.rb

@@ -0,0 +1,123 @@
+# frozen_string_literal: true
+
+require_relative "data"
+require_relative "fuzzy_alignment_index"
+require_relative "fuzzy_alignment_policy"
+require_relative "fuzzy_alignment_planner"
+
+module LangExtract
+  module Core
+    # Coordinates indexed ordered-subsequence fuzzy alignment and candidate
+    # selection. Index construction and dynamic planning live separately.
+    class FuzzyAligner
+      # Historical limit retained for compatibility. Indexed planning no longer
+      # truncates candidate starts.
+      MAX_FUZZY_CANDIDATE_STARTS = 4_000
+      SubsequenceAlignment = FuzzyAlignmentPlanner::SubsequenceAlignment
+
+      def initialize(source_tokens:, fuzzy_threshold:, min_coverage:, min_density:, allow_overlaps:,
+                     alignment_index: nil)
+        @source_tokens = source_tokens
+        @fuzzy_threshold = fuzzy_threshold
+        @allow_overlaps = allow_overlaps
+        @alignment_index = alignment_index || FuzzyAlignmentIndex.new(source_tokens)
+        @policy = FuzzyAlignmentPolicy.new(source_tokens: source_tokens, fuzzy_threshold: fuzzy_threshold)
+        @planner = FuzzyAlignmentPlanner.new(
+          source_tokens: source_tokens,
+          alignment_index: @alignment_index,
+          fuzzy_threshold: fuzzy_threshold,
+          min_coverage: min_coverage,
+          min_density: min_density
+        )
+      end
+
+      # Returns [CharInterval, score] candidates ranked by score, then position.
+      def candidates(target_tokens, range, occupied)
+        alignments = all_subsequence_alignments(target_tokens)
+        return [] if alignments.empty?
+
+        found = alignments.filter_map do |alignment|
+          next unless planner.feasible?(alignment, target_tokens.length)
+
+          score = policy.score_for(target_tokens, alignment.token_indices)
+          next unless score
+
+          interval = interval_for_alignment(alignment.token_indices, range)
+          next unless interval
+
+          match = [interval, score]
+          return [match] if perfect_non_overlapping?(match, occupied)
+
+          match
+        end
+
+        ranked_unique_candidates(found)
+      end
+
+      private
+
+      attr_reader :source_tokens, :fuzzy_threshold, :allow_overlaps, :alignment_index, :planner, :policy
+
+      def all_subsequence_alignments(target_tokens)
+        start_flags = Array.new(source_tokens.length, false)
+        target_tokens.each do |target_token|
+          alignment_index.matching_positions(target_token).indices.each { |idx| start_flags[idx] = true }
+        end
+        valid_starts = start_flags.each_index.select { |idx| start_flags[idx] }
+        return [] if valid_starts.empty?
+
+        seen = {}
+        valid_starts.each_with_object([]) do |start, alignments|
+          alignment = align_from_start(target_tokens, start)
+          next unless alignment
+          next if seen[alignment.token_indices]
+
+          seen[alignment.token_indices] = true
+          alignments << alignment
+        end
+      end
+
+      def align_from_start(target_tokens, source_start)
+        planner.alignment_for(target_tokens, source_start)
+      end
+
+      def find_token_match(target_token, from_index)
+        alignment_index.best_match(target_token, from_index)
+      end
+
+      def norm_source
+        alignment_index.normalized_tokens
+      end
+
+      def interval_for_alignment(token_indices, range)
+        return nil if token_indices.empty?
+
+        first_token = source_tokens[token_indices.first]
+        last_token = source_tokens[token_indices.last]
+        start_pos = [first_token.char_interval.start_pos, range.begin].max
+        end_pos = [last_token.char_interval.end_pos, range.end].min
+        return nil if start_pos >= end_pos
+
+        CharInterval.new(start_pos: start_pos, end_pos: end_pos)
+      end
+
+      def perfect_non_overlapping?(match, occupied)
+        interval, score = match
+        score >= 1.0 && (allow_overlaps || !overlaps_any?(interval, occupied))
+      end
+
+      def ranked_unique_candidates(candidates)
+        best_by_interval = candidates.each_with_object({}) do |(interval, score), result|
+          key = [interval.start_pos, interval.end_pos]
+          result[key] = [interval, score] if result[key].nil? || score > result[key].last
+        end
+
+        best_by_interval.values.sort_by { |interval, score| [-score, interval.start_pos, interval.end_pos] }
+      end
+
+      def overlaps_any?(interval, occupied)
+        occupied.any? { |other| interval.overlaps?(other) }
+      end
+    end
+  end
+end

data/lib/langextract/core/fuzzy_alignment_index.rb

@@ -0,0 +1,81 @@
+# frozen_string_literal: true
+
+require_relative "token_similarity"
+
+module LangExtract
+  module Core
+    # Precomputed source-token lookup for fuzzy alignment. Similarity is
+    # evaluated once per distinct normalized source text and target token.
+    class FuzzyAlignmentIndex
+      MatchingIndex = Data.define(:indices, :scores, :best_indices)
+
+      def initialize(source_tokens)
+        @source_tokens = source_tokens
+      end
+
+      def normalized_tokens
+        @normalized_tokens ||= source_tokens.map { |token| TokenSimilarity.normalize(token.text) }
+      end
+
+      def matching_positions(target_token)
+        @matching_positions_cache ||= {}
+        normalized_target = TokenSimilarity.normalize(target_token)
+        @matching_positions_cache[normalized_target] ||= build_matching_index(normalized_target)
+      end
+
+      def best_match(target_token, from_index)
+        matching = matching_positions(target_token)
+        position = matching.indices.bsearch_index { |idx| idx >= from_index }
+        position && matching.best_indices[position]
+      end
+
+      private
+
+      attr_reader :source_tokens
+
+      def source_index
+        @source_index ||= normalized_tokens.each_with_index.with_object({}) do |(normalized_text, idx), index|
+          (index[normalized_text] ||= []) << idx
+        end
+      end
+
+      def build_matching_index(normalized_target)
+        pairs = matching_pairs(normalized_target).sort_by(&:first)
+        indices = pairs.map(&:first)
+        scores = pairs.map(&:last)
+        MatchingIndex.new(
+          indices: indices,
+          scores: scores,
+          best_indices: suffix_best_indices(indices, scores)
+        )
+      end
+
+      def matching_pairs(normalized_target)
+        source_index.each_with_object([]) do |(normalized_text, indices), matches|
+          next unless TokenSimilarity.similar?(normalized_target, normalized_text)
+
+          similarity = TokenSimilarity.char_similarity(normalized_target, normalized_text)
+          indices.each { |idx| matches << [idx, similarity] }
+        end
+      end
+
+      def suffix_best_indices(indices, scores)
+        best_indices = Array.new(indices.length)
+        best_position = nil
+        (indices.length - 1).downto(0) do |position|
+          best_position = better_score_position(position, best_position, indices, scores)
+          best_indices[position] = indices[best_position]
+        end
+        best_indices
+      end
+
+      def better_score_position(position, best_position, indices, scores)
+        return position if best_position.nil?
+        return position if scores[position] > scores[best_position]
+        return position if scores[position] == scores[best_position] && indices[position] < indices[best_position]
+
+        best_position
+      end
+    end
+  end
+end

data/lib/langextract/core/fuzzy_alignment_planner.rb

@@ -0,0 +1,239 @@
+# frozen_string_literal: true
+
+require_relative "fuzzy_alignment_policy"
+require_relative "token_similarity"
+
+module LangExtract
+  module Core
+    # Plans ordered fuzzy token alignments with coverage, density, score, and
+    # semantic-barrier constraints. The suffix table is shared by all starts.
+    class FuzzyAlignmentPlanner
+      MAX_PATH_ALTERNATIVES_PER_MATCH_COUNT = 2
+      SENTENCE_BOUNDARY_PUNCTUATION = /[.!?\u2026\u3002\uff01\uff1f]/u
+
+      SubsequenceAlignment = Data.define(:matched, :window_size, :token_indices, :score)
+      AlignmentPath = Data.define(:matched, :score_sum, :first_idx, :last_idx, :previous, :safe_gap)
+
+      def initialize(source_tokens:, alignment_index:, fuzzy_threshold:, min_coverage:, min_density:)
+        @source_tokens = source_tokens
+        @alignment_index = alignment_index
+        @fuzzy_threshold = fuzzy_threshold
+        @min_coverage = min_coverage
+        @min_density = min_density
+      end
+
+      def alignment_for(target_tokens, source_start)
+        paths = alignment_table(target_tokens)[source_start]
+        preferred_path(paths, target_tokens.length)
+      end
+
+      def feasible?(alignment, target_count)
+        return false if target_count.zero?
+
+        coverage = alignment.matched.to_f / target_count
+        density = alignment.matched.to_f / alignment.window_size
+
+        coverage >= min_coverage &&
+          density >= min_density &&
+          alignment.score >= fuzzy_threshold &&
+          safe_source_gap?(alignment.token_indices)
+      end
+
+      private
+
+      attr_reader :source_tokens, :alignment_index, :fuzzy_threshold, :min_coverage, :min_density
+
+      def alignment_table(target_tokens)
+        @alignment_table_cache ||= {}
+        key = target_tokens.map(&:to_s).freeze
+        @alignment_table_cache[key] ||= build_alignment_table(target_tokens)
+      end
+
+      def build_alignment_table(target_tokens)
+        next_row = Array.new(source_tokens.length + 1) { [] }
+        target_tokens.reverse_each do |target_token|
+          next_row = alignment_row(target_token, next_row)
+        end
+        next_row
+      end
+
+      def alignment_row(target_token, next_row)
+        matching = alignment_index.matching_positions(target_token)
+        options = alignment_options(matching, next_row)
+        best_match = suffix_best_paths(options)
+        row = Array.new(source_tokens.length + 1)
+        source_tokens.length.downto(0) do |source_position|
+          row[source_position] = merge_path_frontiers(best_match[source_position], next_row[source_position])
+        end
+        row
+      end
+
+      def alignment_options(matching, next_row)
+        matching.indices.each_with_index.flat_map do |idx, position|
+          suffixes = next_row[idx + 1]
+          base = path_with_match(idx, matching.scores[position], nil)
+          continuations = suffixes.to_a.flat_map do |suffixes_for_count|
+            next [] unless suffixes_for_count
+
+            suffixes_for_count.filter_map do |suffix|
+              next unless suffix.safe_gap && source_gap_safe?(idx, suffix.first_idx)
+
+              path_with_match(idx, matching.scores[position], suffix)
+            end
+          end
+          [base] + continuations
+        end
+      end
+
+      def path_with_match(index, score, suffix)
+        AlignmentPath.new(
+          matched: 1 + (suffix&.matched || 0),
+          score_sum: score + (suffix&.score_sum || 0.0),
+          first_idx: index,
+          last_idx: suffix ? suffix.last_idx : index,
+          previous: suffix,
+          safe_gap: suffix.nil? || (suffix.safe_gap && source_gap_safe?(index, suffix.first_idx))
+        )
+      end
+
+      def suffix_best_paths(options)
+        best_match = Array.new(source_tokens.length + 1) { [] }
+        option_position = options.length - 1
+        next_frontier = []
+        (source_tokens.length - 1).downto(0) do |source_position|
+          while option_position >= 0 && options[option_position].first_idx >= source_position
+            append_path(best_match[source_position], options[option_position])
+            option_position -= 1
+          end
+
+          if barrier_token?(source_position)
+            best_match[source_position] = prune_frontier(best_match[source_position])
+            next_frontier = []
+          else
+            best_match[source_position] = merge_path_frontiers(best_match[source_position], next_frontier)
+            next_frontier = best_match[source_position]
+          end
+        end
+        best_match
+      end
+
+      def merge_path_frontiers(left, right)
+        return right if left.empty?
+        return left if right.empty?
+
+        merged = Array.new([left.length, right.length].max) { [] }
+        [left, right].each do |frontier|
+          frontier.each_with_index do |paths, count|
+            merged[count] = prune_paths(merged[count] + paths.to_a)
+          end
+        end
+        merged
+      end
+
+      def append_path(frontier, path)
+        frontier[path.matched] ||= []
+        frontier[path.matched] << path
+      end
+
+      def prune_frontier(frontier)
+        frontier.map { |paths| prune_paths(paths.to_a) }
+      end
+
+      def prune_paths(paths)
+        candidates = paths.uniq { |path| path_signature(path) }
+        nondominated = candidates.reject do |candidate|
+          candidates.any? do |other|
+            other != candidate && dominates_path?(other, candidate)
+          end
+        end
+        return nondominated if nondominated.length <= MAX_PATH_ALTERNATIVES_PER_MATCH_COUNT
+
+        score_order = nondominated.sort_by { |path| [-path.score_sum, path.last_idx, path.first_idx] }
+        compact_order = nondominated.sort_by { |path| [path.last_idx, -path.score_sum, path.first_idx] }
+        (score_order.first(MAX_PATH_ALTERNATIVES_PER_MATCH_COUNT / 2) +
+          compact_order.first(MAX_PATH_ALTERNATIVES_PER_MATCH_COUNT / 2)).uniq.first(
+            MAX_PATH_ALTERNATIVES_PER_MATCH_COUNT
+          )
+      end
+
+      def path_signature(path)
+        [path.first_idx, path.last_idx, path.score_sum, path.safe_gap]
+      end
+
+      def dominates_path?(left, right)
+        left.score_sum >= right.score_sum &&
+          left.last_idx <= right.last_idx &&
+          left.first_idx <= right.first_idx &&
+          (!right.safe_gap || left.safe_gap)
+      end
+
+      def preferred_path(paths, target_count)
+        candidates = paths.compact.flat_map { |paths_for_count| paths_for_count }
+        return nil if candidates.empty?
+
+        alignments = candidates.map { |path| subsequence_alignment(path) }
+        feasible = alignments.select { |alignment| feasible?(alignment, target_count) }
+        return alignments.max_by { |alignment| fallback_alignment_priority(alignment) } if feasible.empty?
+
+        feasible.max_by { |alignment| alignment_priority(alignment) }
+      end
+
+      def subsequence_alignment(path)
+        SubsequenceAlignment.new(
+          matched: path.matched,
+          window_size: path.last_idx - path.first_idx + 1,
+          token_indices: path_indices(path),
+          score: path.score_sum / path.matched
+        )
+      end
+
+      def path_indices(path)
+        indices = Array.new(path.matched)
+        current = path
+        path.matched.times do |position|
+          indices[position] = current.first_idx
+          current = current.previous
+        end
+        indices
+      end
+
+      def alignment_priority(alignment)
+        [alignment.score, alignment.matched, -alignment.token_indices.first, -alignment.window_size]
+      end
+
+      def fallback_alignment_priority(alignment)
+        [alignment.matched, alignment.score, -alignment.token_indices.first, -alignment.window_size]
+      end
+
+      def safe_source_gap?(token_indices)
+        token_indices.each_cons(2).all? do |left_idx, right_idx|
+          source_gap_safe?(left_idx, right_idx)
+        end
+      end
+
+      def source_gap_safe?(left_idx, right_idx)
+        barrier_prefix[right_idx] == barrier_prefix[left_idx + 1]
+      end
+
+      def barrier_token?(index)
+        token = source_tokens[index]
+        sentence_boundary_punctuation?(token.text) || negation_token?(token.text)
+      end
+
+      def barrier_prefix
+        @barrier_prefix ||= source_tokens.each_with_object([0]) do |token, prefix|
+          barrier = sentence_boundary_punctuation?(token.text) || negation_token?(token.text)
+          prefix << (prefix.last + (barrier ? 1 : 0))
+        end
+      end
+
+      def sentence_boundary_punctuation?(text)
+        text.match?(SENTENCE_BOUNDARY_PUNCTUATION)
+      end
+
+      def negation_token?(text)
+        FuzzyAlignmentPolicy.negation_token?(text)
+      end
+    end
+  end
+end

data/lib/langextract/core/fuzzy_alignment_policy.rb

@@ -0,0 +1,124 @@
+# frozen_string_literal: true
+
+require_relative "token_similarity"
+
+module LangExtract
+  module Core
+    # Validates semantic constraints that depend on the target-to-source token
+    # mapping rather than only the selected source interval.
+    class FuzzyAlignmentPolicy
+      NEGATION_TOKENS = %w[
+        aint arent cannot cant couldnt deny denied denies denying didnt doesnt
+        dont hadnt hardly hasnt havent isnt mustnt neither neednt never no none
+        not shouldnt shant wasnt werent without wont wouldnt
+      ].freeze
+
+      MappingState = Data.define(
+        :last_target_index,
+        :score_sum,
+        :has_exact_match,
+        :has_short_typo,
+        :negation_mask
+      )
+
+      def self.negation_token?(text)
+        NEGATION_TOKENS.include?(TokenSimilarity.normalize(text))
+      end
+
+      def initialize(source_tokens:, fuzzy_threshold:)
+        @source_tokens = source_tokens
+        @fuzzy_threshold = fuzzy_threshold
+      end
+
+      # Returns the best semantically valid average score, or nil when the
+      # selected source tokens cannot be mapped safely to the target tokens.
+      def score_for(target_tokens, source_indices)
+        return nil if target_tokens.empty? || source_indices.empty?
+
+        states = [initial_state]
+        source_indices.each do |source_index|
+          states = advance_states(states, target_tokens, source_tokens.fetch(source_index).text)
+          return nil if states.empty?
+        end
+
+        best_valid_score(states, required_negation_mask(target_tokens), source_indices.length)
+      end
+
+      private
+
+      attr_reader :source_tokens, :fuzzy_threshold
+
+      def initial_state
+        MappingState.new(
+          last_target_index: -1,
+          score_sum: 0.0,
+          has_exact_match: false,
+          has_short_typo: false,
+          negation_mask: 0
+        )
+      end
+
+      def best_valid_score(states, required_negations, matched_count)
+        states.filter_map do |state|
+          next unless valid_mapping?(state, required_negations, matched_count)
+
+          score = state.score_sum / matched_count
+          score if score >= fuzzy_threshold
+        end.max
+      end
+
+      def valid_mapping?(state, required_negations, matched_count)
+        return false unless state.negation_mask.allbits?(required_negations)
+        return false if state.has_short_typo && (!state.has_exact_match || matched_count < 2)
+
+        true
+      end
+
+      def advance_states(states, target_tokens, source_text)
+        best_by_signature = {}
+        states.each do |state|
+          ((state.last_target_index + 1)...target_tokens.length).each do |target_index|
+            target_text = target_tokens[target_index]
+            next unless TokenSimilarity.similar?(target_text, source_text)
+
+            candidate = extend_state(state, target_text, target_index, source_text)
+            signature = state_signature(candidate)
+            current = best_by_signature[signature]
+            best_by_signature[signature] = candidate if current.nil? || candidate.score_sum > current.score_sum
+          end
+        end
+        best_by_signature.values
+      end
+
+      def extend_state(state, target_text, target_index, source_text)
+        target = TokenSimilarity.normalize(target_text)
+        source = TokenSimilarity.normalize(source_text)
+        exact = target == source
+        negation_mask = state.negation_mask
+        negation_mask |= (1 << target_index) if exact && negation_token?(target)
+
+        MappingState.new(
+          last_target_index: target_index,
+          score_sum: state.score_sum + TokenSimilarity.char_similarity(target, source),
+          has_exact_match: state.has_exact_match || exact,
+          has_short_typo: state.has_short_typo || TokenSimilarity.short_typo?(target, source),
+          negation_mask: negation_mask
+        )
+      end
+
+      def state_signature(state)
+        [state.last_target_index, state.has_exact_match, state.has_short_typo, state.negation_mask]
+      end
+
+      def required_negation_mask(target_tokens)
+        target_tokens.each_with_index.reduce(0) do |mask, (token, index)|
+          negation_token?(token) ? mask | (1 << index) : mask
+        end
+      end
+
+      def negation_token?(text)
+        self.class.negation_token?(text)
+      end
+    end
+  end
+end