ollama-client 0.2.5 → 0.2.7

data/docs/TESTING.md

@@ -1,19 +1,393 @@
-# Testing Guide
+# Testing Guide: Client-Only Testing
 
-This document explains how to test the `ollama-client` gem comprehensively.
+This document explains how to test the `ollama-client` gem **in isolation**, focusing on **transport and protocol correctness**, not agent behavior.
+
+## 🔒 Responsibility Boundary
+
+`ollama-client` is responsible for:
+
+✅ **Transport layer** - HTTP requests/responses  
+✅ **Protocol correctness** - Request shaping, response parsing  
+✅ **Schema enforcement** - JSON validation  
+✅ **Tool-call parsing** - Detecting and extracting tool calls  
+✅ **Error handling** - Network errors, timeouts, retries  
+✅ **Streaming behavior** - NDJSON/SSE parsing  
+✅ **Protocol compatibility** - Native Ollama + Anthropic adapter
+
+`ollama-client` is **NOT** responsible for:
+
+❌ Agent loops  
+❌ Convergence logic  
+❌ Policy decisions  
+❌ Tool execution  
+❌ Correctness of agent decisions
+
+**If you test more than the transport layer, you're leaking agent concerns into the client.**
+
+## Test Categories
+
+### Category A: `/generate` Mode (Stateless, Deterministic)
+
+Tests that prove `ollama-client` is safe-by-default for stateless operations.
+
+#### ✅ G1 — Basic Generate
+
+**Purpose:** Verify basic JSON parsing and response handling.
+
+**Test:**
+```ruby
+it "parses JSON response from generate endpoint" do
+  stub_request(:post, "http://localhost:11434/api/generate")
+    .to_return(
+      status: 200,
+      body: { response: '{"status":"ok"}' }.to_json
+    )
+
+  result = client.generate(
+    prompt: "Output a JSON object with a single key 'status' and value 'ok'.",
+    schema: { "type" => "object", "required" => ["status"] }
+  )
+
+  expect(result).to be_a(Hash)
+  expect(result["status"]).to eq("ok")
+  expect(result).not_to have_key("tool_calls")
+end
+```
+
+**Assertions:**
+- Response is a Hash
+- JSON is parsed correctly
+- No `tool_calls` present
+- No streaming artifacts
+
+#### ✅ G2 — Strict Schema Enforcement
+
+**Purpose:** Validate contract enforcement (major differentiator).
+
+**Test:**
+```ruby
+it "rejects responses that violate schema" do
+  stub_request(:post, "http://localhost:11434/api/generate")
+    .to_return(
+      status: 200,
+      body: { response: '{"count":"not-a-number"}' }.to_json
+    )
+
+  schema = {
+    "type" => "object",
+    "required" => ["count"],
+    "properties" => {
+      "count" => { "type" => "number" }
+    }
+  }
+
+  expect do
+    client.generate(prompt: "Output JSON with key 'count' as a number.", schema: schema)
+  end.to raise_error(Ollama::SchemaViolationError)
+end
+```
+
+**Assertions:**
+- Raises error if schema violated
+- Rejects extra fields (if strict mode enabled)
+- Validates required fields
+
+#### ❌ G3 — Tool Attempt in Generate (Must Fail)
+
+**Purpose:** Prove `/generate` is non-agentic by design.
+
+**Test:**
+```ruby
+it "ignores tool calls in generate mode" do
+  stub_request(:post, "http://localhost:11434/api/generate")
+    .to_return(
+      status: 200,
+      body: { response: '{"action":"call read_file tool on foo.rb"}' }.to_json
+    )
+
+  result = client.generate(
+    prompt: "Call the read_file tool on foo.rb",
+    schema: { "type" => "object" }
+  )
+
+  expect(result).not_to have_key("tool_calls")
+  expect(result).not_to have_key("tool_use")
+end
+```
+
+**Assertions:**
+- No `tool_calls` parsed
+- No silent acceptance of tool intent
+- Either ignored or explicit error
+
+### Category B: `/chat` Mode (Stateful, Tool-Aware)
+
+Tests that prove `ollama-client` can **transport** tool calls and messages correctly — **not** that the agent works.
+
+#### ✅ C1 — Simple Chat
+
+**Purpose:** Verify basic message handling.
+
+**Test:**
+```ruby
+it "handles simple chat messages" do
+  stub_request(:post, "http://localhost:11434/api/chat")
+    .to_return(
+      status: 200,
+      body: {
+        message: { role: "assistant", content: "Hello!" }
+      }.to_json
+    )
+
+  response = client.chat_raw(
+    messages: [{ role: "user", content: "Say hello." }],
+    allow_chat: true
+  )
+
+  expect(response.message.content).to eq("Hello!")
+  expect(response.message.role).to eq("assistant")
+end
+```
+
+**Assertions:**
+- Response contains assistant message
+- Message history preserved in request
+
+#### ✅ C2 — Tool-Call Parsing (Critical)
+
+**Purpose:** Verify client correctly **detects tool intent** (not execution).
+
+**Test:**
+```ruby
+it "extracts tool calls from chat response" do
+  stub_request(:post, "http://localhost:11434/api/chat")
+    .to_return(
+      status: 200,
+      body: {
+        message: {
+          role: "assistant",
+          content: "I'll call the ping tool.",
+          tool_calls: [
+            {
+              type: "function",
+              function: {
+                name: "ping",
+                arguments: { "x" => 1 }.to_json
+              }
+            }
+          ]
+        }
+      }.to_json
+    )
+
+  response = client.chat_raw(
+    messages: [{ role: "user", content: "If a tool named 'ping' exists, call it with { 'x': 1 }." }],
+    tools: [tool_definition],
+    allow_chat: true
+  )
+
+  tool_calls = response.message.tool_calls
+  expect(tool_calls).not_to be_empty
+  expect(tool_calls.first["function"]["name"]).to eq("ping")
+  expect(JSON.parse(tool_calls.first["function"]["arguments"])).to eq("x" => 1)
+end
+```
+
+**Assertions:**
+- `tool_calls` extracted correctly
+- Tool name parsed
+- Arguments parsed as hash
+- **No execution happens** (client must not execute tools)
+
+#### ✅ C3 — Tool Result Round-Trip Formatting
+
+**Purpose:** Verify client serializes tool messages correctly.
+
+**Test:**
+```ruby
+it "serializes tool result messages correctly" do
+  messages = [
+    { role: "user", content: "Call ping tool" },
+    { role: "assistant", content: "", tool_calls: [...] },
+    { role: "tool", name: "ping", content: { ok: true }.to_json }
+  ]
+
+  stub_request(:post, "http://localhost:11434/api/chat")
+    .with(body: hash_including(messages: messages))
+    .to_return(
+      status: 200,
+      body: { message: { role: "assistant", content: "Done!" } }.to_json
+    )
+
+  response = client.chat_raw(messages: messages, allow_chat: true)
+  expect(response.message.content).to eq("Done!")
+end
+```
+
+**Assertions:**
+- Client serializes tool message correctly
+- Ollama accepts it
+- Response parsed cleanly
+
+### Category C: Protocol Adapters (Anthropic / Native)
+
+Tests that prove **protocol adapter correctness** (pure client tests, no model required).
+
+#### ✅ A1 — Anthropic Message Shape
+
+**Purpose:** Verify request payload compatibility.
+
+**Test:**
+```ruby
+it "serializes messages in Anthropic format" do
+  stub_request(:post, "http://localhost:11434/api/chat")
+    .with do |req|
+      body = JSON.parse(req.body)
+      expect(body["messages"]).to be_an(Array)
+      expect(body["messages"].first).to include("role", "content")
+    end
+    .to_return(status: 200, body: { message: {} }.to_json)
+
+  client.chat_raw(
+    messages: [{ role: "user", content: "Test" }],
+    allow_chat: true
+  )
+end
+```
+
+**Assertions:**
+- Messages serialized as content blocks
+- Tool calls emitted as `tool_use` (if Anthropic mode)
+- Tool results serialized as `tool_result`
+
+#### ✅ A2 — Anthropic Response Parsing
+
+**Purpose:** Verify response normalization.
+
+**Test:**
+```ruby
+it "normalizes Anthropic-style responses into internal format" do
+  anthropic_response = {
+    content: [
+      {
+        type: "tool_use",
+        id: "call_123",
+        name: "search",
+        input: { q: "foo" }
+      }
+    ]
+  }
+
+  stub_request(:post, "http://localhost:11434/api/chat")
+    .to_return(status: 200, body: anthropic_response.to_json)
+
+  response = client.chat_raw(
+    messages: [{ role: "user", content: "Search for foo" }],
+    allow_chat: true
+  )
+
+  tool_calls = response.message.tool_calls
+  expect(tool_calls).not_to be_empty
+  expect(tool_calls.first["function"]["name"]).to eq("search")
+end
+```
+
+**Assertions:**
+- Client normalizes Anthropic format into internal `tool_calls`
+- Protocol adapter correctness
+
+### Category D: Failure Modes (Non-Negotiable)
+
+#### ✅ F1 — Ollama Down
+
+**Test:**
+```ruby
+it "handles connection refused gracefully" do
+  stub_request(:post, "http://localhost:11434/api/generate")
+    .to_raise(Errno::ECONNREFUSED)
+
+  expect do
+    client.generate(prompt: "test", schema: schema)
+  end.to raise_error(Ollama::Error)
+
+  # Verify no hangs
+  expect(Time.now - start_time).to be < 5
+end
+```
+
+**Assertions:**
+- Connection refused raises correct exception
+- No hangs
+- Retries handled correctly
+
+#### ✅ F2 — Invalid JSON from Model
+
+**Test:**
+```ruby
+it "raises error on invalid JSON response" do
+  stub_request(:post, "http://localhost:11434/api/generate")
+    .to_return(status: 200, body: { response: "not json at all" }.to_json)
+
+  expect do
+    client.generate(prompt: "test", schema: schema)
+  end.to raise_error(Ollama::InvalidJSONError)
+end
+```
+
+**Assertions:**
+- Client raises parse error
+- Does not silently continue
+- Retries handled (if retryable)
+
+#### ✅ F3 — Streaming Interruption
+
+**Test:**
+```ruby
+it "handles partial stream gracefully" do
+  stub_request(:post, "http://localhost:11434/api/chat")
+    .to_return(
+      status: 200,
+      body: "data: {\"message\":{\"content\":\"partial\"}}\n",
+      headers: { "Content-Type" => "text/event-stream" }
+    )
+
+  # Simulate stream interruption
+  expect do
+    client.chat_raw(messages: [{ role: "user", content: "test" }], allow_chat: true)
+  end.to raise_error(Ollama::Error)
+end
+```
+
+**Assertions:**
+- Partial stream handled
+- Client terminates cleanly
+- No corrupted state
+
+## What You Should NOT Test
+
+❌ **Do not test:**
+- Infinite loops
+- Retries based on content
+- Agent stopping behavior
+- Tool side effects
+- Correctness of answers
+- Agent convergence logic
+- Policy decisions
+
+**Those belong to `agent-runtime` and app repos.**
 
 ## Test Structure
 
 The test suite is organized into focused spec files:
 
 - `spec/ollama/client_spec.rb` - Basic client initialization and parameter validation
-- `spec/ollama/client_generate_spec.rb` - Comprehensive tests for `generate()` method
-- `spec/ollama/client_chat_spec.rb` - Comprehensive tests for `chat()` method
+- `spec/ollama/client_generate_spec.rb` - Tests for `generate()` method (Category A)
+- `spec/ollama/client_chat_spec.rb` - Tests for `chat()` method (Category B)
+- `spec/ollama/client_chat_raw_spec.rb` - Tests for `chat_raw()` method (Category B)
 - `spec/ollama/client_list_models_spec.rb` - Tests for `list_models()` method
 - `spec/ollama/client_model_suggestions_spec.rb` - Tests for model suggestion feature
-- `spec/ollama/errors_spec.rb` - Tests for all error classes
-- `spec/ollama/config_spec.rb` - Config class tests (in client_spec.rb)
-- `spec/ollama/schema_validator_spec.rb` - Schema validation tests (in client_spec.rb)
+- `spec/ollama/errors_spec.rb` - Tests for all error classes (Category D)
+- `spec/ollama/schema_validator_spec.rb` - Schema validation tests (Category A, G2)
 
 ## Running Tests
 
@@ -32,19 +406,14 @@ bundle exec rspec spec/ollama/client_generate_spec.rb
 bundle exec rspec --format documentation
 ```
 
-### Run Specific Test
-```bash
-bundle exec rspec spec/ollama/client_generate_spec.rb:45
-```
-
 ### Run Tests Matching a Pattern
 ```bash
-bundle exec rspec -e "retry"
+bundle exec rspec -e "schema"
 ```
 
 ## Testing Strategy
 
-### 1. HTTP Mocking with WebMock
+### HTTP Mocking with WebMock
 
 All HTTP requests are mocked using [WebMock](https://github.com/bblimke/webmock). This allows us to:
 - Test without a real Ollama server
@@ -58,43 +427,6 @@ stub_request(:post, "http://localhost:11434/api/generate")
   .to_return(status: 200, body: { response: '{"test":"value"}' }.to_json)
 ```
 
-### 2. Test Coverage Areas
-
-#### ✅ Success Cases
-- Successful API calls return parsed JSON
-- Schema validation passes
-- Config defaults are applied correctly
-- Model overrides work
-- Options are merged correctly
-
-#### ✅ Error Handling
-- **404 (NotFoundError)**: Model not found, no retries, includes suggestions
-- **500 (HTTPError)**: Retryable, retries up to config limit
-- **400 (HTTPError)**: Non-retryable, fails immediately
-- **TimeoutError**: Retries on timeout
-- **InvalidJSONError**: Retries on JSON parse errors
-- **SchemaViolationError**: Retries on schema validation failures
-- **Connection Errors**: Retries on network failures
-
-#### ✅ Retry Logic
-- Retries up to `config.retries` times
-- Only retries retryable errors (5xx, 408, 429)
-- Raises `RetryExhaustedError` after max retries
-- Succeeds if retry succeeds
-
-#### ✅ Edge Cases
-- JSON wrapped in markdown code blocks
-- Plain JSON responses
-- Empty model lists
-- Missing response fields
-- Malformed JSON
-
-#### ✅ Model Suggestions
-- Suggests similar models on 404
-- Fuzzy matching on model names
-- Limits suggestions to 5 models
-- Handles model listing failures gracefully
-
 ## Writing New Tests
 
 ### Basic Test Structure
@@ -106,6 +438,8 @@ RSpec.describe Ollama::Client, "#method_name" do
     Ollama::Config.new.tap do |c|
       c.base_url = "http://localhost:11434"
       c.model = "test-model"
+      c.retries = 2
+      c.timeout = 5
     end
   end
 
@@ -127,127 +461,16 @@ RSpec.describe Ollama::Client, "#method_name" do
 end
 ```
 
-### Testing Retry Logic
-
-```ruby
-it "retries on 500 errors" do
-  stub_request(:post, "http://localhost:11434/api/generate")
-    .to_return(status: 500, body: "Internal Server Error")
-    .times(config.retries + 1)
-
-  expect do
-    client.generate(prompt: "test", schema: schema)
-  end.to raise_error(Ollama::RetryExhaustedError)
-
-  expect(WebMock).to have_requested(:post, "http://localhost:11434/api/generate")
-    .times(config.retries + 1)
-end
-```
-
-### Testing Success After Retry
-
-```ruby
-it "succeeds on retry" do
-  stub_request(:post, "http://localhost:11434/api/generate")
-    .to_return(
-      { status: 500, body: "Internal Server Error" },
-      { status: 200, body: { response: '{"test":"value"}' }.to_json }
-    )
-
-  result = client.generate(prompt: "test", schema: schema)
-  expect(result).to eq("test" => "value")
-  expect(WebMock).to have_requested(:post, "http://localhost:11434/api/generate").twice
-end
-```
-
-### Testing Error Details
-
-```ruby
-it "raises error with correct details" do
-  stub_request(:post, "http://localhost:11434/api/generate")
-    .to_return(status: 404, body: "Not Found")
-
-  expect do
-    client.generate(prompt: "test", schema: schema)
-  end.to raise_error(Ollama::NotFoundError) do |error|
-    expect(error.requested_model).to eq("test-model")
-    expect(error.status_code).to eq(404)
-  end
-end
-```
-
-## Integration Tests (Optional)
-
-For integration tests that hit a real Ollama server, create a separate spec file:
-
-```ruby
-# spec/integration/ollama_client_integration_spec.rb
-RSpec.describe "Ollama Client Integration", :integration do
-  # Skip if OLLAMA_URL is not set
-  before(:all) do
-    skip "Set OLLAMA_URL environment variable to run integration tests" unless ENV["OLLAMA_URL"]
-  end
-
-  let(:client) do
-    config = Ollama::Config.new
-    config.base_url = ENV["OLLAMA_URL"] || "http://localhost:11434"
-    Ollama::Client.new(config: config)
-  end
-
-  it "can generate structured output" do
-    schema = {
-      "type" => "object",
-      "required" => ["test"],
-      "properties" => { "test" => { "type" => "string" } }
-    }
-
-    result = client.generate(
-      prompt: "Return a JSON object with test='hello'",
-      schema: schema
-    )
-
-    expect(result["test"]).to eq("hello")
-  end
-end
-```
-
-Run integration tests separately:
-```bash
-bundle exec rspec --tag integration
-```
-
-## Test Coverage Metrics
-
-To check test coverage, add `simplecov`:
-
-```ruby
-# spec/spec_helper.rb
-require "simplecov"
-SimpleCov.start
-```
-
-Then run:
-```bash
-bundle exec rspec
-open coverage/index.html
-```
-
-## Continuous Integration
-
-The test suite is designed to run in CI without external dependencies:
-- All tests use WebMock (no real Ollama server needed)
-- Tests are deterministic and fast
-- No flaky network-dependent tests
-
 ## Best Practices
 
 1. **Always mock HTTP requests** - Don't make real network calls in unit tests
-2. **Test error paths** - Ensure all error scenarios are covered
-3. **Test retry logic** - Verify retries work correctly
-4. **Test edge cases** - JSON parsing, empty responses, etc.
-5. **Keep tests focused** - One assertion per test when possible
-6. **Use descriptive test names** - "it 'retries on 500 errors'"
-7. **Reset WebMock** - Always reset in `after` blocks
+2. **Test transport layer only** - Don't test agent behavior
+3. **Test error paths** - Ensure all error scenarios are covered
+4. **Test retry logic** - Verify retries work correctly
+5. **Test edge cases** - JSON parsing, empty responses, etc.
+6. **Keep tests focused** - One assertion per test when possible
+7. **Use descriptive test names** - "it 'extracts tool calls from chat response'"
+8. **Reset WebMock** - Always reset in `after` blocks
 
 ## Debugging Tests
 
@@ -283,4 +506,3 @@ WebMock.allow_net_connect!
 - Ensure WebMock is reset in `after` blocks
 - Don't share state between tests
 - Use `let` instead of instance variables
-