langextract 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 0aeb0a461910ee77f31dade0f57e4007a6525e8c4a8a8f8791f7a58a1b472926
+  data.tar.gz: c96a48d1f86d5257f5107a67b03a57b30e051ce877fd52b9d4b352fe54b57ed4
+SHA512:
+  metadata.gz: 6c86ace0907fd9260cc93aad98c540aa269b5d6944d17d694775bc362d88865b6892b07fd1bc7a02794c25448ae724cd76b404c84274e427953ed6b092788565
+  data.tar.gz: af118cf15965bb7ef4d03fd8a2a5a07ba1b7631c84d346be42217e0d44bf0cafdc006dbceda56b255387eed87461c257680a197cdeb8662083fd2c2472043b85

data/CHANGELOG.md

@@ -0,0 +1,9 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [0.1.0] - 2026-04-19
+
+### Added
+- Initial Ruby gem scaffold for `langextract`.
+- Core data contracts, tokenization, chunking, prompting, parsing, resolver, provider routing, JSONL IO, and visualization.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 David
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,250 @@
+# LangExtract
+
+[![Gem Version](https://badge.fury.io/rb/langextract.svg)](https://badge.fury.io/rb/langextract)
+[![CI](https://github.com/dpaluy/langextract/actions/workflows/ci.yml/badge.svg)](https://github.com/dpaluy/langextract/actions/workflows/ci.yml)
+
+A Ruby gem for extracting structured information from unstructured text using LLMs with precise source grounding and interactive visualization.
+
+Ruby port of [LangExtract](https://github.com/google/langextract) v1.2.1.
+
+Use it when a Ruby or Rails app needs structured LLM output that can be traced back to exact source spans instead of ungrounded JSON blobs.
+
+## Features
+
+- **Source grounding** — every extraction includes character and token offsets back to the original text
+- **Structured outputs** — deterministic, serializable result objects with alignment status
+- **Long-document chunking** — sentence-aware chunking with sequential multi-pass extraction
+- **Interactive visualization** — self-contained HTML highlighting of extraction spans
+- **Format handling** — JSON and YAML output parsing with strict and lenient modes
+- **Provider-agnostic** — pluggable LLM providers via [RubyLLM](https://github.com/crmne/ruby_llm)
+
+## Requirements
+
+- Ruby >= 3.4.5
+- Tested on Ruby 3.4.5 and 4.0.2
+- Optional live inference adapter: `ruby_llm` >= 1.0 when using `LangExtract::Factory.create_model`
+
+## Installation
+
+```bash
+gem "langextract"
+```
+
+Provider calls go through RubyLLM. Add RubyLLM to applications that need live model inference:
+
+```bash
+bundle add ruby_llm
+```
+
+## Configuration
+
+Configure RubyLLM the same way you already do in the host app. LangExtract does not own API keys or provider credentials:
+
+```ruby
+require "langextract"
+require "ruby_llm"
+
+RubyLLM.configure do |config|
+  config.openai_api_key = ENV.fetch("OPENAI_API_KEY")
+  config.default_model = "gpt-4o-mini"
+end
+```
+
+LangExtract has only a small optional configuration surface:
+
+| Option | ENV variable | Default |
+|--------|--------------|---------|
+| `default_model` | `LANGEXTRACT_MODEL` | RubyLLM's `default_model` |
+
+Per-call model configuration can override the RubyLLM model or provider without touching credentials:
+
+```ruby
+model = LangExtract::Factory.create_model(
+  LangExtract::ModelConfig.new(
+    model: "gpt-4o-mini",
+    provider: "openai"
+  )
+)
+```
+
+If you omit `model`, RubyLLM's configured `default_model` is used.
+
+### Rails
+
+Create `config/initializers/langextract.rb`:
+
+```ruby
+RubyLLM.configure do |config|
+  config.openai_api_key = Rails.application.credentials.dig(:openai, :api_key)
+  config.default_model = "gpt-4o-mini"
+end
+```
+
+## Usage
+
+### Extract
+
+Build a provider and extract grounded fields:
+
+```ruby
+model = LangExtract::Factory.create_model
+
+result = LangExtract.extract(
+  text: "Apple Inc. reported revenue of $94.8 billion for Q1 2024.",
+  model: model,
+  prompt_description: "Extract company financial data",
+  examples: [
+    LangExtract::ExampleData.new(
+      text: "Microsoft earned $56.5 billion in Q2 2023.",
+      extractions: [
+        { text: "Microsoft", description: "company" },
+        { text: "$56.5 billion", description: "revenue" },
+        { text: "Q2 2023", description: "period" }
+      ]
+    )
+  ]
+)
+
+result.extractions.each do |extraction|
+  puts "#{extraction.text} (#{extraction.description}) #{extraction.char_interval}"
+end
+```
+
+Return value access pattern:
+
+```ruby
+first = result.extractions.first
+first.text
+first.extraction_class
+first.char_interval.start_pos
+first.char_interval.end_pos
+first.alignment_status
+```
+
+### Document collections
+
+```ruby
+documents = [
+  LangExtract::Document.new(id: "q1", text: "Apple reported revenue."),
+  LangExtract::Document.new(id: "q2", text: "Microsoft reported profit.")
+]
+
+annotated_documents = LangExtract.extract(
+  documents: documents,
+  model: model,
+  prompt_description: "Extract company names",
+  prompt_validation: :off
+)
+```
+
+### Visualization
+
+```ruby
+html = LangExtract.visualize(result)
+File.write("output.html", html)
+```
+
+`visualize` accepts a single `LangExtract::AnnotatedDocument`, an array of annotated documents, or a JSONL path.
+
+### JSONL persistence
+
+```ruby
+LangExtract::IO.save_annotated_documents("results.jsonl", documents)
+documents = LangExtract::IO.load_annotated_documents_jsonl("results.jsonl")
+```
+
+### Format schema validation
+
+`FormatHandler` can validate normalized extractions against a small JSON-schema-like contract:
+
+```ruby
+schema = {
+  required: %w[text extraction_class],
+  properties: {
+    text: { type: "string" },
+    extraction_class: { type: "string" },
+    attributes: { type: "object" }
+  }
+}
+
+LangExtract::Core::FormatHandler.new.parse(model_output, schema: schema)
+```
+
+## Error handling
+
+```ruby
+begin
+  LangExtract.extract(...)
+rescue LangExtract::InvalidModelConfigError => e
+  warn "Invalid model configuration: #{e.message}"
+rescue LangExtract::ProviderConfigError => e
+  warn "Provider failed: #{e.message}"
+rescue LangExtract::PromptValidationError, LangExtract::FormatParsingError => e
+  warn e.message
+rescue LangExtract::AlignmentError => e
+  warn "Could not ground extraction: #{e.message}"
+rescue LangExtract::IOFailure => e
+  warn "Could not read or write LangExtract data: #{e.message}"
+end
+```
+
+## API reference
+
+- Upstream project: [google/langextract](https://github.com/google/langextract)
+- Ruby API docs: [rubydoc.info/gems/langextract](https://rubydoc.info/gems/langextract)
+
+## Current parity status
+
+This is a Ruby gem slice against the Google LangExtract v1.2.1. It includes the core public contracts, an optional RubyLLM-backed provider adapter, and fixture-backed tests for deterministic local behavior.
+
+The upstream v1.2.1 tag was collected with pytest into `test/fixtures/upstream/v1_2_1_pytest_manifest.json`: 404 deterministic tests plus 11 live API tests and 4 Ollama integration tests. That collection does not match the older PRD snapshot count of 479 deterministic / 494 total, so the count discrepancy must be reconciled before a 1.0 parity claim.
+
+Deferred v1+ items:
+
+- Full expected-output parity conversion for every deterministic upstream case in the manifest
+- External plugin discovery from installed Ruby gems
+- Batch inference workflows
+- Concurrent provider calls
+- URL fetching
+
+## Development
+
+```bash
+bundle install
+bundle exec rake test
+bundle exec rubocop
+bundle exec rake build
+bundle exec yard
+```
+
+Run a single test:
+
+```bash
+bundle exec ruby -Itest test/langextract/core/resolver_test.rb
+bundle exec ruby -Itest test/langextract/core/resolver_test.rb -n test_aligns_exact_extraction_text_and_token_offsets
+```
+
+## Architecture
+
+LangExtract follows a strict layered architecture:
+
+```
+Orchestrator
+  ├── Prompting / Format Handling
+  ├── Chunking / Tokenization
+  ├── Resolver / Alignment          ← center of gravity
+  ├── Annotation
+  └── Provider (via RubyLLM)
+```
+
+**Core modules never depend on provider SDKs.** Provider output is normalized into an internal structure before reaching the resolver. The resolver handles exact and fuzzy alignment of extraction text back to source offsets — this is the most complex and critical module.
+
+Differential test fixtures derived from the upstream Python library are the source of truth for behavioral parity.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/dpaluy/langextract.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](LICENSE.txt).

data/lib/langextract/config.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+require "logger"
+
+module LangExtract
+  class Config
+    attr_accessor :default_model, :logger
+
+    def initialize
+      @default_model = ENV.fetch("LANGEXTRACT_MODEL", nil)
+      @logger = default_logger
+    end
+
+    private
+
+    def default_logger
+      defined?(Rails) ? Rails.logger : Logger.new($stderr)
+    end
+  end
+end

data/lib/langextract/core/annotation.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+require_relative "resolver"
+
+module LangExtract
+  module Core
+    class Annotator
+      def initialize(tokenizer: UnicodeTokenizer.new, fuzzy_threshold: Resolver::DEFAULT_FUZZY_THRESHOLD,
+                     suppress_alignment_errors: true)
+        @tokenizer = tokenizer
+        @fuzzy_threshold = fuzzy_threshold
+        @suppress_alignment_errors = suppress_alignment_errors
+      end
+
+      def annotate(document:, extraction_hashes:, preferred_interval: nil)
+        resolver = Resolver.new(
+          text: document.text,
+          tokenizer: tokenizer,
+          fuzzy_threshold: fuzzy_threshold,
+          suppress_alignment_errors: suppress_alignment_errors
+        )
+        extractions = resolver.resolve(
+          extraction_hashes,
+          document_id: document.id,
+          preferred_interval: preferred_interval
+        )
+
+        AnnotatedDocument.new(document: document, extractions: extractions)
+      end
+
+      private
+
+      attr_reader :tokenizer, :fuzzy_threshold, :suppress_alignment_errors
+    end
+  end
+end

data/lib/langextract/core/chunking.rb

@@ -0,0 +1,116 @@
+# frozen_string_literal: true
+
+require_relative "tokenizer"
+
+module LangExtract
+  module Core
+    Chunk = Data.define(:text, :char_interval, :token_interval, :index, :document_id) do
+      def to_h
+        {
+          "text" => text,
+          "char_interval" => char_interval.to_h,
+          "token_interval" => token_interval&.to_h,
+          "index" => index,
+          "document_id" => document_id
+        }
+      end
+    end
+
+    class SentenceAwareChunker
+      DEFAULT_MAX_CHAR_BUFFER = 2_000
+
+      def initialize(max_char_buffer: DEFAULT_MAX_CHAR_BUFFER, tokenizer: UnicodeTokenizer.new)
+        raise ArgumentError, "max_char_buffer must be positive" unless max_char_buffer.positive?
+
+        @max_char_buffer = max_char_buffer
+        @tokenizer = tokenizer
+      end
+
+      def chunks(document)
+        text = document.respond_to?(:text) ? document.text : document.to_s
+        document_id = document.respond_to?(:id) ? document.id : nil
+        sentences = sentence_intervals(text)
+        token_lookup = token_lookup(text)
+        build_chunks(text, sentences, token_lookup, document_id)
+      end
+
+      private
+
+      attr_reader :max_char_buffer, :tokenizer
+
+      def sentence_intervals(text)
+        intervals = []
+        start_pos = 0
+
+        text.to_enum(:scan, /[.!?]+[)"'\]]*(?:\s+|$)|\n{2,}/).each do
+          match = Regexp.last_match
+          end_pos = match.end(0)
+          intervals << CharInterval.new(start_pos: start_pos, end_pos: end_pos) if end_pos > start_pos
+          start_pos = end_pos
+        end
+
+        intervals << CharInterval.new(start_pos: start_pos, end_pos: text.length) if start_pos < text.length
+        intervals.reject { |interval| text[interval.start_pos...interval.end_pos].strip.empty? }
+      end
+
+      def build_chunks(text, sentences, token_lookup, document_id)
+        return [] if text.empty?
+
+        chunks = []
+        current_start = nil
+        current_end = nil
+
+        sentences.each do |sentence|
+          current_start ||= sentence.start_pos
+          if current_end && sentence.end_pos - current_start > max_char_buffer
+            chunks << build_chunk(text, current_start, current_end, token_lookup, chunks.length, document_id)
+            current_start = sentence.start_pos
+          end
+
+          current_end = sentence.end_pos
+
+          while current_end - current_start > max_char_buffer
+            split_end = split_position(text, current_start)
+            chunks << build_chunk(text, current_start, split_end, token_lookup, chunks.length, document_id)
+            current_start = split_end
+          end
+        end
+
+        if current_start && current_end
+          chunks << build_chunk(text, current_start, current_end, token_lookup, chunks.length, document_id)
+        end
+        chunks.freeze
+      end
+
+      def split_position(text, start_pos)
+        limit = [start_pos + max_char_buffer, text.length].min
+        whitespace = text.rindex(/\s/, limit)
+        return limit unless whitespace && whitespace > start_pos
+
+        whitespace + 1
+      end
+
+      def build_chunk(text, start_pos, end_pos, token_lookup, index, document_id)
+        interval = CharInterval.new(start_pos: start_pos, end_pos: end_pos)
+        Chunk.new(
+          text: text[start_pos...end_pos],
+          char_interval: interval,
+          token_interval: token_interval_for(interval, token_lookup),
+          index: index,
+          document_id: document_id
+        )
+      end
+
+      def token_lookup(text)
+        tokenizer.tokenize(text)
+      end
+
+      def token_interval_for(char_interval, tokens)
+        matching = tokens.select { |token| token.char_interval.overlaps?(char_interval) }
+        return nil if matching.empty?
+
+        TokenInterval.new(start_pos: matching.first.index, end_pos: matching.last.index + 1)
+      end
+    end
+  end
+end