ruby-spacy 0.4.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6185c586feb32fa51efcd4349398cd4ca9541280a5cc8a1b6a73eb93a987d4ac
-  data.tar.gz: a146a9c40e2d5293e2401cb16b8ac6866cbb577e11a10d9657c406f933e7a3aa
+  metadata.gz: b759c507a1cad8e726bd36ad39e4539894395407e2ac5d64a31b7b907a4e34d8
+  data.tar.gz: 0f02f9b8f6bcf39a49b50b1934905e72c0c13bb9652eee8fbaad3a3b7b58b7e7
 SHA512:
-  metadata.gz: bf558d4e9a7a6765fd7d088bbf8324a6ee0e4f4186962551d71e5a991e0aefd1e51a186f19c2824fabcc6afd0c83960771f082237febece52c2a522ccb39a5cf
-  data.tar.gz: 3a64559cf8c169d1ac1ecdef526d26e5776989b9cc203a8ed30e0dd5d87ff62a4d1b741aff30c8cb49e5ffb716c6068f9af3a12d50d0d4de8ad6f22ebe80ea0d
+  metadata.gz: 5b7d8078148e05c860f3ef698b5103cfca533fabfac5001c5afd12419efaae21d6492aa22fcebfba9d9d22bf8701fe2c6a21eb1ee879344cdbd5d897af9b8853
+  data.tar.gz: 4ffab4be8e0338e86c34aad0285cd9471a3d5c570fbf48b4214cdbe5ca5cc01c7d251d4f6d5edc778de2185d89fc3858e094c363499c81f89d5c3a61ab8859c2

data/CHANGELOG.md

@@ -1,5 +1,63 @@
 # Change Log
 
+## 0.5.0 - 2026-07-21
+### Added
+- `Language#with_llm(provider:)` — provider-neutral block-based LLM API
+  supporting `:openai`, `:anthropic` (Claude), and `:ollama` (local models)
+- `AnthropicHelper` / `AnthropicClient` — Anthropic Messages API support
+  using only net/http (default model: `claude-sonnet-5`; requires
+  `ANTHROPIC_API_KEY`)
+- `base_url:` option for OpenAI client/helper — works with any
+  OpenAI-compatible server (Ollama, LM Studio, llama.cpp server, vLLM,
+  OpenRouter, etc.)
+- `schema:` option for `chat` — Structured Outputs on both providers;
+  returns a validated, parsed Ruby Hash
+- New examples under `examples/llm/` (Anthropic, local Ollama, and a
+  spaCy-vs-LLM NER comparison with structured outputs)
+
+### Changed
+- `temperature` is now sent only when explicitly specified; if a model
+  rejects it, the request is retried once without it. The model-name
+  heuristic (`temperature_unsupported?`) has been removed, so new models
+  work without gem updates. Note: previously a default of 0.7 was sent to
+  models that supported it (also from `Doc#openai_query` /
+  `Doc#openai_completion`); now the API default applies unless you pass
+  `temperature:` yourself
+- Shared HTTP layer (`LLMClientBase`) extracted from `OpenAIClient`;
+  no behavior change for existing OpenAI usage
+- Truncated responses (`finish_reason: length` / `stop_reason: max_tokens`)
+  now emit a warning; with `schema:`, unparseable (e.g. truncated) JSON
+  returns nil with a warning instead of raising `JSON::ParserError`
+- LLM error messages are printed to stderr (`Kernel#warn`) instead of
+  stdout, keeping stdout clean for program output
+- Default model names are now public constants:
+  `OpenAIClient::DEFAULT_MODEL`, `OpenAIClient::DEFAULT_EMBEDDINGS_MODEL`,
+  `AnthropicClient::DEFAULT_MODEL`
+
+## 0.4.1 - 2026-07-19
+### Fixed
+- Gem packaging: normalize file permissions at build time so packaged files
+  are world-readable (0644 / 0755). Version 0.4.0 shipped 11 owner-only
+  (0600) files, which made the gem unusable after `sudo gem install`
+
+## 0.4.0 - 2026-02-23
+### Added
+- `Language#with_openai` block API with `OpenAIHelper` for streamlined OpenAI integration
+- `Doc#linguistic_summary` for JSON-formatted spaCy analysis output
+- `Token#idx`, `Span#text`/`Span#to_s`, `Language#memory_zone` (spaCy 3.8+)
+
+### Fixed
+- `Doc#ents` returns proper `Span` objects instead of raw Python objects
+
+### Changed
+- Model name validation in `Language#initialize` for security
+- OpenAI client: temperature support for o-series models, 429 retry with
+  exponential backoff, client reuse, `dimensions`/`response_format` parameters,
+  tool call depth limit
+- Improved `respond_to_missing?` across all wrapper classes
+- Added `instance_variables_to_inspect` for Ruby 4.0+ compatibility
+- Added `base64` gem dependency (required for Ruby 3.4+)
+
 ## 0.3.0 - 2025-01-06
 ### Added
 - Ruby 4.0 support

data/README.md

@@ -11,13 +11,15 @@
 | ✅ | Named entity recognition                           |
 | ✅ | Syntactic dependency visualization                 |
 | ✅ | Access to pre-trained word vectors                 |
-| ✅ | OpenAI Chat/Completion/Embeddings API integration  |
+| ✅ | LLM integration: OpenAI, Anthropic (Claude), and local models |
 
-Current Version: `0.3.0`
+Current Version: `0.5.0`
 
 - Ruby 4.0 supported
 - spaCy 3.8 supported
-- OpenAI GPT-5 API integration
+- Multi-provider LLM API: OpenAI, Anthropic (Claude), and local models via Ollama or any OpenAI-compatible server
+- Structured outputs (JSON Schema) support
+- Block-based LLM API with linguistic analysis
 
 ## Installation of Prerequisites
 
@@ -587,7 +589,9 @@ end
 
 ## OpenAI API Integration
 
-> ⚠️ This feature requires GPT-5 series models. Please refer to OpenAI's [API reference](https://platform.openai.com/docs/api-reference) for details. Note: GPT-5 models do not support the `temperature` parameter.
+> ⚠️ This feature requires GPT-5 series models. Please refer to OpenAI's [API reference](https://platform.openai.com/docs/api-reference) for details.
+
+> ℹ️ The `temperature` parameter is sent to the API only when you specify it explicitly. If the model does not support it (e.g., GPT-5 series and o-series models), the request is automatically retried once without it — no per-model configuration is needed.
 
 Easily leverage GPT models within ruby-spacy by using an OpenAI API key. When constructing prompts for the `Doc::openai_query` method, you can incorporate the following token properties of the document. These properties are retrieved through tool calls (made internally by GPT when necessary) and seamlessly integrated into your prompt. The available properties include:
 
@@ -843,6 +847,131 @@ Output:
 ...
 ```
 
+### Block-based OpenAI API
+
+The `Language#with_openai` block API provides a streamlined way to combine spaCy's linguistic analysis with OpenAI. The `Doc#linguistic_summary` method generates a JSON summary of spaCy's analysis (tokens, entities, noun chunks, etc.) that can be passed directly to the LLM as context.
+
+**Basic usage:**
+
+```ruby
+require "ruby-spacy"
+
+nlp = Spacy::Language.new("en_core_web_sm")
+doc = nlp.read("Apple Inc. was founded by Steve Jobs in California.")
+
+nlp.with_openai(model: "gpt-5-mini") do |ai|
+  result = ai.chat(
+    system: "You are a linguistic analyst. Analyze the given linguistic data.",
+    user: doc.linguistic_summary
+  )
+  puts result
+end
+```
+
+**Batch processing with `pipe`:**
+
+```ruby
+require "ruby-spacy"
+
+nlp = Spacy::Language.new("en_core_web_sm")
+texts = ["The bank approved the loan.", "I sat on the river bank."]
+
+nlp.with_openai(model: "gpt-5-mini") do |ai|
+  nlp.pipe(texts).each do |doc|
+    result = ai.chat(
+      system: "Identify the meaning of 'bank' in one word based on the linguistic context.",
+      user: doc.linguistic_summary
+    )
+    puts "#{doc.text} => #{result}"
+  end
+end
+```
+
+**Customizing linguistic summary:**
+
+```ruby
+# Include sentences and morphology, exclude noun chunks
+summary = doc.linguistic_summary(
+  sections: [:text, :tokens, :entities, :sentences],
+  token_attributes: [:text, :lemma, :pos, :dep, :head, :morphology]
+)
+```
+
+**Embeddings:**
+
+```ruby
+nlp.with_openai do |ai|
+  vector = ai.embeddings("Hello world")
+  puts vector.length  # => 1536
+end
+```
+
+### Multi-provider LLM API
+
+The `Language#with_llm` block API generalizes `with_openai` to multiple providers. The helper yielded to the block has the same `chat` interface for every provider.
+
+**Anthropic (Claude):**
+
+```ruby
+# Requires the ANTHROPIC_API_KEY environment variable
+# Default model: claude-sonnet-5 (override with model: "...")
+nlp.with_llm(provider: :anthropic) do |ai|
+  result = ai.chat(
+    system: "You are a linguistic analyst.",
+    user: doc.linguistic_summary
+  )
+  puts result
+end
+```
+
+**Local models via Ollama (no API key needed):**
+
+```ruby
+# Requires a running Ollama server: https://ollama.com
+nlp.with_llm(provider: :ollama, model: "llama3.2") do |ai|
+  puts ai.chat(user: doc.linguistic_summary)
+end
+```
+
+Any other OpenAI-compatible server (LM Studio, llama.cpp server, vLLM, OpenRouter, etc.) works via `base_url:`:
+
+```ruby
+nlp.with_llm(provider: :openai, base_url: "http://localhost:1234/v1",
+             access_token: "not-needed", model: "your-model") do |ai|
+  puts ai.chat(user: "Hello!")
+end
+```
+
+**Structured outputs (`schema:`):** pass a JSON Schema and receive a validated, parsed Ruby Hash — useful for comparing LLM output with spaCy's analysis programmatically. Works with both `:openai` and `:anthropic`. Objects in the schema must set `additionalProperties: false`.
+
+```ruby
+schema = {
+  type: "object",
+  properties: {
+    entities: {
+      type: "array",
+      items: {
+        type: "object",
+        properties: { text: { type: "string" }, label: { type: "string" } },
+        required: %w[text label],
+        additionalProperties: false
+      }
+    }
+  },
+  required: ["entities"],
+  additionalProperties: false
+}
+
+result = nlp.with_llm(provider: :openai) do |ai|
+  ai.chat(system: "Extract named entities.", user: doc.text, schema: schema)
+end
+result["entities"].each { |ent| puts "#{ent["text"]} (#{ent["label"]})" }
+```
+
+**Note on `temperature`:** for all providers, `temperature` is omitted from requests unless you pass it explicitly (`ai.chat(user: "...", temperature: 0.3)`). Models that reject the parameter (e.g., GPT-5 series, o-series, and current Claude models) are automatically retried once without it, so any model works without per-model configuration.
+
+See `examples/llm/` for complete scripts, including a spaCy-vs-LLM NER comparison.
+
 ## Advanced Usage
 
 ### Setting a Timeout

data/Rakefile

@@ -10,3 +10,16 @@ Rake::TestTask.new(:test) do |t|
 end
 
 task default: :test
+
+# Gem packaging preserves on-disk file modes; owner-only permissions here
+# produce gems whose files are unreadable after a sudo install. Executable
+# status is determined by path alone because local (Dropbox-synced) modes
+# are unreliable.
+task :normalize_permissions do
+  `git ls-files -z`.split("\x0").each do |f|
+    executable = f.start_with?("bin/", "exe/")
+    File.chmod(executable ? 0o755 : 0o644, f)
+  end
+end
+
+Rake::Task["build"].enhance([:normalize_permissions])

data/examples/llm/anthropic_chat.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
+require "ruby-spacy"
+
+# Requires the ANTHROPIC_API_KEY environment variable
+# Default model: claude-sonnet-5 (override with model: "...")
+
+nlp = Spacy::Language.new("en_core_web_sm")
+doc = nlp.read("The cat sat on the mat while the dog slept under the table.")
+
+result = nlp.with_llm(provider: :anthropic) do |ai|
+  ai.chat(
+    system: "You are a linguistics tutor. Explain the syntactic structure briefly.",
+    user: doc.linguistic_summary
+  )
+end
+
+puts result

data/examples/llm/local_llm_ollama.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
+require "ruby-spacy"
+
+# Requires a running Ollama server (https://ollama.com):
+#   ollama pull llama3.2
+#   ollama serve
+# No API key is needed.
+
+nlp = Spacy::Language.new("en_core_web_sm")
+doc = nlp.read("I sat on the bank of the river.")
+
+result = nlp.with_llm(provider: :ollama, model: "llama3.2") do |ai|
+  ai.chat(
+    system: "Identify the meaning of the word 'bank' in one short sentence.",
+    user: doc.linguistic_summary
+  )
+end
+
+puts result
+
+# Any OpenAI-compatible server works the same way via base_url, e.g.:
+#   nlp.with_llm(provider: :openai, base_url: "http://localhost:1234/v1",
+#                access_token: "not-needed", model: "your-model") { |ai| ... }

data/examples/llm/structured_ner_comparison.rb

@@ -0,0 +1,49 @@
+# frozen_string_literal: true
+
+# add path to ruby-spacy lib to load path
+$LOAD_PATH.unshift(File.expand_path("../../lib", __dir__))
+
+require "ruby-spacy"
+
+# Compares spaCy's named entity recognition with an LLM's, using structured
+# outputs so the LLM's answer comes back as a validated Ruby Hash.
+# Requires the OPENAI_API_KEY environment variable
+# (or swap in provider: :anthropic with ANTHROPIC_API_KEY).
+
+nlp = Spacy::Language.new("en_core_web_sm")
+text = "Mr. Best flew to New York on Saturday morning to meet Tim Cook of Apple."
+doc = nlp.read(text)
+
+schema = {
+  type: "object",
+  properties: {
+    entities: {
+      type: "array",
+      items: {
+        type: "object",
+        properties: {
+          text: { type: "string" },
+          label: { type: "string", enum: %w[PERSON ORG GPE DATE TIME OTHER] }
+        },
+        required: %w[text label],
+        additionalProperties: false
+      }
+    }
+  },
+  required: ["entities"],
+  additionalProperties: false
+}
+
+llm_result = nlp.with_llm(provider: :openai) do |ai|
+  ai.chat(
+    system: "Extract all named entities from the user's text.",
+    user: text,
+    schema: schema
+  )
+end
+
+puts "spaCy entities:"
+doc.ents.each { |ent| puts "  #{ent.text} (#{ent.label})" }
+
+puts "\nLLM entities:"
+llm_result["entities"].each { |ent| puts "  #{ent["text"]} (#{ent["label"]})" }

data/lib/ruby-spacy/anthropic_client.rb

@@ -0,0 +1,66 @@
+# frozen_string_literal: true
+
+require_relative "llm_client_base"
+
+module Spacy
+  # A lightweight Anthropic Messages API client without external dependencies.
+  class AnthropicClient < LLMClientBase
+    # Default Anthropic API endpoint
+    API_ENDPOINT = "https://api.anthropic.com/v1"
+    # Messages API version header value
+    ANTHROPIC_VERSION = "2023-06-01"
+    # Default model for chat requests
+    DEFAULT_MODEL = "claude-sonnet-5"
+    # (see LLMClientBase::APIError)
+    APIError = LLMClientBase::APIError
+
+    # @param access_token [String] Anthropic API key
+    # @param timeout [Integer] request timeout in seconds
+    # @param base_url [String] API endpoint override
+    def initialize(access_token:, timeout: DEFAULT_TIMEOUT, base_url: API_ENDPOINT)
+      super(base_url: base_url, timeout: timeout)
+      @access_token = access_token
+    end
+
+    # Sends a Messages API request.
+    #
+    # The +temperature+ parameter is sent only when explicitly given. Current
+    # Claude models (Opus 4.7+) reject it; the request is then retried once
+    # without it.
+    #
+    # @param model [String] The model to use (e.g., "claude-sonnet-5")
+    # @param messages [Array<Hash>] The conversation messages
+    # @param system [String, nil] System prompt (top-level parameter)
+    # @param max_tokens [Integer] Maximum tokens in the response (required by the API)
+    # @param temperature [Float, nil] Sampling temperature (omitted when nil)
+    # @param output_config [Hash, nil] Output configuration, e.g.
+    #   { format: { type: "json_schema", schema: {...} } } for structured outputs
+    # @return [Hash] The API response
+    def messages(model:, messages:, system: nil, max_tokens: 1000, temperature: nil, output_config: nil)
+      body = {
+        model: model,
+        max_tokens: max_tokens,
+        messages: messages
+      }
+      body[:system] = system if system
+      body[:temperature] = temperature unless temperature.nil?
+      body[:output_config] = output_config if output_config
+
+      post("/messages", body)
+    rescue APIError => e
+      raise unless body.key?(:temperature) && e.status_code == 400 && e.message.match?(/temperature/i)
+
+      warn "Warning: model does not support the temperature parameter; retrying without it"
+      post("/messages", body.reject { |key, _| key == :temperature })
+    end
+
+    private
+
+    def request_headers
+      {
+        "x-api-key" => @access_token,
+        "anthropic-version" => ANTHROPIC_VERSION
+      }
+    end
+  end
+end