llm_gateway 0.5.0 → 0.7.0

data/docs/migration_guide_0.6.0.md

@@ -0,0 +1,386 @@
+# Migration Guide: 0.5.0 to 0.6.0
+
+This guide covers user-facing changes between `v0.5.0` and the latest commit on the 0.6.0 branch.
+
+## Summary
+
+0.6.0 separates provider authentication/configuration from model selection.
+
+- Provider config now contains only provider/auth settings such as `provider`, `api_key`, `access_token`, and `account_id`.
+- `model_key` is no longer accepted in provider/client configuration.
+- Pass the model per request with `model:` when calling `chat`, `stream`, Responses/Codex methods, or embeddings.
+- Legacy provider keys such as `openai_apikey_responses` were removed. Use the shorter provider keys.
+- `LlmGateway::Prompt` now accepts/configures a provider and model separately, and uses `stream` internally.
+- The `client.model_key` reader was removed; track the selected model at the call site or read it from returned messages.
+- Streaming events now expose accumulated partial messages during the stream, while `:message_end` exposes the final message through `event.message`.
+- Non-final stream event hashes now include `partial`; normal stream consumers are unaffected, but strict `event.to_h` snapshots/comparisons may need updates.
+- Normalized usage counters were renamed to concise keys: `:input`, `:cache_write`, `:cache_read`, `:output`, and `:total`; `:reasoning_tokens` was removed.
+- Streamed assistant messages now include `timestamp` as Unix milliseconds.
+- Custom stream mappers must initialize with provider/API metadata and emit a final `:message_end` patch.
+
+## 1. Replace legacy provider keys
+
+0.6.0 removes the backward-compatible legacy provider registry entries.
+
+| 0.5.0 provider key | 0.6.0 provider key |
+|---|---|
+| `anthropic_apikey_messages` | `anthropic_messages` |
+| `anthropic_oauth_messages` | `anthropic_messages` |
+| `openai_apikey_completions` | `openai_completions` |
+| `openai_apikey_responses` | `openai_responses` |
+| `openai_oauth_codex` | `openai_codex` |
+| `groq_apikey_completions` | `groq_completions` |
+
+### Before
+
+```ruby
+adapter = LlmGateway.build_provider(
+  provider: "openai_apikey_responses",
+  api_key: ENV.fetch("OPENAI_API_KEY"),
+  model_key: "gpt-5.4"
+)
+```
+
+### After
+
+```ruby
+adapter = LlmGateway.build_provider(
+  provider: "openai_responses",
+  api_key: ENV.fetch("OPENAI_API_KEY")
+)
+```
+
+## 2. Move `model_key` from provider config to request calls
+
+`model_key` is no longer a provider option. Passing it to `LlmGateway.build_provider` raises:
+
+```text
+ArgumentError: model_key is no longer a provider option; pass model: to chat/stream instead
+```
+
+Pass `model:` on each request instead.
+
+### Streaming
+
+```ruby
+# Before
+adapter = LlmGateway.build_provider(
+  provider: "openai_apikey_responses",
+  api_key: ENV.fetch("OPENAI_API_KEY"),
+  model_key: "gpt-5.4"
+)
+result = adapter.stream("Write one short sentence about Ruby.")
+
+# After
+adapter = LlmGateway.build_provider(
+  provider: "openai_responses",
+  api_key: ENV.fetch("OPENAI_API_KEY")
+)
+result = adapter.stream("Write one short sentence about Ruby.", model: "gpt-5.4")
+```
+
+### Configure arrays
+
+```ruby
+# Before
+LlmGateway.configure([
+  {
+    name: "primary",
+    config: {
+      provider: "groq_apikey_completions",
+      api_key: ENV.fetch("GROQ_API_KEY"),
+      model_key: "openai/gpt-oss-120b"
+    }
+  }
+])
+
+# After
+LlmGateway.configure([
+  {
+    name: "primary",
+    config: {
+      provider: "groq_completions",
+      api_key: ENV.fetch("GROQ_API_KEY")
+    }
+  }
+])
+
+LlmGateway.configured_clients.fetch("primary").stream(
+  "Hello",
+  model: "openai/gpt-oss-120b"
+)
+```
+
+## 3. Update direct client usage
+
+Direct clients no longer take `model_key:` in their constructors.
+
+```ruby
+# Before
+client = LlmGateway::Clients::OpenAI.new(
+  api_key: ENV.fetch("OPENAI_API_KEY"),
+  model_key: "gpt-5.4"
+)
+client.stream(messages)
+
+# After
+client = LlmGateway::Clients::OpenAI.new(
+  api_key: ENV.fetch("OPENAI_API_KEY")
+)
+client.stream(messages, model: "gpt-5.4")
+```
+
+The same pattern applies to:
+
+- `LlmGateway::Clients::Anthropic#chat` / `#stream`
+- `LlmGateway::Clients::OpenAI#chat` / `#stream` / `#responses` / `#stream_responses`
+- `LlmGateway::Clients::OpenAI#chat_codex` / `#stream_codex`
+- `LlmGateway::Clients::Groq#chat` / `#stream`
+
+Embeddings also take a per-call model:
+
+```ruby
+client.generate_embeddings(input, model: "text-embedding-3-large")
+```
+
+If omitted, clients still provide default models.
+
+## 4. Update `LlmGateway::Prompt` classes
+
+`Prompt` no longer looks up a configured client by comparing a string to `client.model_key`. It now keeps the provider and model as separate values.
+
+If you previously called `Prompt.new("gpt-5.4")`, update that code. The first initializer argument is now a provider adapter, not a model lookup key. Configure a provider on the class or pass one to the initializer.
+
+### Class-level configuration
+
+```ruby
+class SummaryPrompt < LlmGateway::Prompt
+  self.provider = LlmGateway.build_provider(
+    provider: "openai_responses",
+    api_key: ENV.fetch("OPENAI_API_KEY")
+  )
+  self.model = "gpt-5.4"
+
+  def prompt
+    "Summarize this text."
+  end
+end
+
+SummaryPrompt.new.run
+```
+
+
+### Instance-level configuration
+
+```ruby
+provider = LlmGateway.build_provider(
+  provider: "anthropic_messages",
+  api_key: ENV.fetch("ANTHROPIC_API_KEY")
+)
+
+SummaryPrompt.new(provider, "claude-sonnet-4-20250514").run
+```
+
+### Per-call overrides
+
+```ruby
+prompt = SummaryPrompt.new(default_provider, "gpt-5.1")
+
+prompt.stream(
+  provider: other_provider,
+  model: "gpt-5.4",
+  reasoning: "high"
+)
+```
+
+If you subclassed `Prompt` and called or overrode `post`, migrate that code to `stream`. `run` now calls `stream` internally.
+
+## 5. Stop using `client.model_key`
+
+Direct clients no longer expose a `model_key` reader because model selection is no longer client/provider state.
+
+```ruby
+# Before
+client = LlmGateway::Clients::OpenAI.new(
+  api_key: ENV.fetch("OPENAI_API_KEY"),
+  model_key: "gpt-5.4"
+)
+puts client.model_key
+
+# After
+client = LlmGateway::Clients::OpenAI.new(
+  api_key: ENV.fetch("OPENAI_API_KEY")
+)
+model = "gpt-5.4"
+result = client.stream(messages, model: model)
+# Track `model` at the call site when you need it later.
+```
+
+## 6. OAuth provider names
+
+OAuth is now represented by credentials, not by separate legacy provider keys.
+
+```ruby
+# Before
+adapter = LlmGateway.build_provider(
+  provider: "openai_oauth_codex",
+  access_token: current_access_token,
+  model_key: "gpt-5.4"
+)
+
+# After
+adapter = LlmGateway.build_provider(
+  provider: "openai_codex",
+  access_token: current_access_token
+)
+
+adapter.stream("Hello from OAuth auth", model: "gpt-5.4")
+```
+
+For Anthropic OAuth, use `provider: "anthropic_messages"` with an `access_token`.
+
+## 7. Update stream callback handling
+
+The final `:message_end` stream callback event changed shape.
+
+In 0.5.x, `:message_end` was an `AssistantStreamMessageEvent` and exposed the accumulated message through `event.partial`.
+
+In 0.6.0, `:message_end` is an `AssistantStreamMessageEndEvent` and exposes the final complete `AssistantMessage` through `event.message`. It does not expose `partial`.
+
+```ruby
+response = adapter.stream("Hello", model: "gpt-5.4") do |event|
+  case event.type
+  when :text_delta
+    print event.delta
+  when :message_end
+    final_message = event.message
+    puts final_message.provider
+    puts final_message.api
+  end
+end
+
+# The stream return value is the same final AssistantMessage.
+response # => AssistantMessage
+```
+
+If you previously handled every event as if it had `partial`, branch on `event.type == :message_end` first or check `respond_to?(:partial)`.
+
+```ruby
+adapter.stream("Hello", model: "gpt-5.4") do |event|
+  if event.type == :message_end
+    persist(event.message.to_h)
+  elsif event.respond_to?(:partial)
+    update_ui(event.partial)
+  end
+end
+```
+
+## 8. Update usage accounting keys
+
+Normalized `AssistantMessage#usage` and final stream `event.usage` patches now use provider-independent concise keys plus `:raw` for the original provider usage/token payload:
+
+| 0.5.x key | 0.6.0 key |
+|---|---|
+| `:input_tokens` | `:input` |
+| `:cache_creation_input_tokens` | `:cache_write` |
+| `:cache_read_input_tokens` | `:cache_read` |
+| `:output_tokens` | `:output` |
+| computed normalized total | `:total` |
+| original provider usage payload | `:raw` |
+| `:reasoning_tokens` | removed |
+
+`reasoning_tokens` was removed because providers expose and calculate reasoning token counts inconsistently. Use the streamed/final `ReasoningContent` blocks for reasoning text, and treat usage as the normalized token buckets above.
+
+```ruby
+# Before
+result.usage[:input_tokens]
+result.usage[:cache_read_input_tokens]
+result.usage[:output_tokens]
+
+# After
+result.usage[:input]
+result.usage[:cache_read]
+result.usage[:output]
+```
+
+When checking cache behavior, use `usage[:cache_read]` and `usage[:cache_write]`. `usage[:total]` is computed as `input + cache_write + cache_read + output`. Use `usage[:raw]` when you need provider-specific token fields that are not part of the normalized counters.
+
+## 9. Account for timestamps on streamed messages
+
+`PartialAssistantMessage` and `AssistantMessage` now include a `timestamp` field in Unix milliseconds. Provider-supplied timestamps are preserved when available; otherwise the accumulator assigns one.
+
+```ruby
+response = adapter.stream("Hello", model: "gpt-5.4") do |event|
+  puts event.partial.timestamp if event.respond_to?(:partial)
+end
+
+puts response.timestamp
+puts response.to_h[:timestamp]
+```
+
+If you instantiate `PartialAssistantMessage` or `AssistantMessage` directly in tests or custom integrations, include `timestamp:`.
+
+## 10. Update custom stream mappers
+
+If you implemented a custom adapter or stream mapper, update it for the new final-message flow.
+
+`LlmGateway::Adapters::StreamMapper` now requires provider/API metadata:
+
+```ruby
+mapper = MyStreamMapper.new(provider: "openai", api: "responses")
+```
+
+`Adapter#stream` passes these values automatically when it instantiates the configured mapper, but direct mapper construction and custom initializers must accept/pass these keywords.
+
+Custom mappers must also push a final normalized end patch. Use the normalized usage keys shown above for final `usage`.
+
+```ruby
+push_patches([
+  { type: :message_delta, delta: { stop_reason: "stop" }, usage: { output: 12 } },
+  { type: :message_end }
+], &block)
+```
+
+`StreamMapper#result` now returns the final `AssistantMessage` created by the `:message_end` patch. If a custom mapper never emits `:message_end`, `adapter.stream` will not have a final message to return.
+
+## 11. Cross-provider handoff note
+
+Message sanitization for cross-provider/model handoffs now receives the target model from the request options. When replaying or handing off transcripts across providers/models, pass `model:` explicitly on the destination call so model-specific sanitizer behavior can run.
+
+```ruby
+next_response = target_adapter.stream(
+  transcript_from_another_provider,
+  model: "gpt-5.4"
+)
+```
+
+## 12. Stream event hash snapshots
+
+Non-final stream events now expose a `partial` assistant message, so `event.to_h` includes an additional `partial` field.
+
+This is additive for normal stream callback consumers:
+
+```ruby
+adapter.stream("Hello", model: "gpt-5.4") do |event|
+  puts event.type
+  puts event.delta if event.respond_to?(:delta)
+end
+```
+
+If your tests or application code compare full `event.to_h` hashes or snapshot serialized events, update those expectations to include or ignore `partial`.
+
+## Checklist
+
+- [ ] Replace all legacy provider keys with the new provider keys.
+- [ ] Remove `model_key:` from `build_provider`, `configure`, and direct client constructors.
+- [ ] Remove any direct reads of `client.model_key` / `adapter.client.model_key`.
+- [ ] Add `model:` to `chat`, `stream`, Responses/Codex, and embeddings calls where you need a specific model.
+- [ ] Update `Prompt` subclasses to configure `provider` and `model` separately.
+- [ ] Replace `Prompt.new("model-key")` model lookup usage with explicit provider/model configuration.
+- [ ] Replace custom `Prompt#post` usage with `Prompt#stream`.
+- [ ] Update stream callbacks to read `event.message` for `:message_end` and `event.partial` only for non-final events.
+- [ ] Rename normalized usage lookups to `:input`, `:cache_write`, `:cache_read`, `:output`, and `:total`; use `:raw` for provider-specific token fields; remove `:reasoning_tokens` handling.
+- [ ] Include/read `timestamp` on streamed partial and final assistant messages where you construct or persist those objects.
+- [ ] Update custom stream mappers to accept `provider:` / `api:`, emit normalized usage keys, and emit `{ type: :message_end }`.
+- [ ] For cross-provider handoffs, pass the target `model:` explicitly.
+- [ ] Update strict `event.to_h` stream event snapshots/comparisons for the new `partial` field.

data/docs/migration_guide_0.7.0.md

@@ -0,0 +1,193 @@
+# Migration guide: 0.7.0
+
+This release refactors `LlmGateway::Prompt` around the normalized streaming response model and adds first-class prompt-owned tool loops.
+
+## Breaking changes
+
+### `Prompt.new` uses keyword arguments
+
+Prompt instance configuration is now keyword-only:
+
+```ruby
+# Before
+SummaryPrompt.new(provider, "claude-sonnet-4-20250514").run
+
+# After
+SummaryPrompt.new(
+  provider: provider,
+  model: "claude-sonnet-4-20250514"
+).run
+```
+
+The same applies when overriding class defaults for `reasoning`, `cache_key`, or `cache_retention`.
+
+Class-level prompt defaults should be assigned with writer methods:
+
+```ruby
+class SummaryPrompt < LlmGateway::Prompt
+  self.provider = provider
+  self.model = "gpt-5.4"
+  self.reasoning = "medium"
+end
+```
+
+If you used the older setter-style calls (`provider value` or `model value`) in prompt subclasses, switch to `self.provider = value` / `self.model = value`.
+
+### `Prompt#run` uses `stream` and normalized `AssistantMessage`
+
+`run` now calls the configured provider's `stream` method and expects it to return a normalized `LlmGateway::AssistantMessage` with `content` blocks.
+
+If you use test doubles or custom providers with `Prompt`, update them from hash-like chat responses:
+
+```ruby
+# Before
+{ choices: [ { content: "hello" } ] }
+```
+
+To `AssistantMessage` responses:
+
+```ruby
+LlmGateway::AssistantMessage.new(
+  id: "msg_123",
+  model: "gpt-5.4",
+  role: "assistant",
+  stop_reason: "stop",
+  provider: "openai",
+  api: "responses",
+  timestamp: Time.now.to_i,
+  usage: {},
+  content: [ { type: "text", text: "hello" } ]
+)
+```
+
+`run` returns the final normalized `AssistantMessage` after tool handling is complete. It no longer extracts or concatenates text content for you; inspect `response.content` when you need text or other blocks.
+
+`after_execute` callbacks now receive only the final `AssistantMessage` instead of both the message and extracted text.
+
+Prompt callback storage now uses Rails-style `class_attribute` inheritance. Register callbacks with `before_execute` / `after_execute` or assign a duplicated callback array on the subclass; avoid mutating inherited callback arrays directly with `before_execute_callbacks << ...` because that can affect related classes.
+
+### `extract_response` and `parse_response` hooks were removed
+
+`Prompt#run` no longer calls custom `extract_response` or `parse_response` methods.
+
+Move response transformation outside the prompt call, or wrap `run` in your subclass:
+
+```ruby
+class JsonPrompt < LlmGateway::Prompt
+  def prompt
+    "Return JSON."
+  end
+
+  def run_json(**options)
+    response = run(**options)
+    text = response.content.select { |block| block.type == "text" }.map(&:text).join
+    JSON.parse(text)
+  end
+end
+```
+
+### Tools are declared with `TOOLS`
+
+Prompt tools are now class-level tool classes declared in a `TOOLS` constant. `Prompt#tools` returns their provider definitions.
+
+```ruby
+class AddTool < LlmGateway::Tool
+  name "add"
+  description "Adds two numbers"
+  input_schema(type: "object")
+  cache true # optional cache_control marker where supported
+
+  def execute(input)
+    input[:left] + input[:right]
+  end
+end
+
+class MathPrompt < LlmGateway::Prompt
+  TOOLS = [AddTool].freeze
+
+  def prompt
+    "What is 2 + 3? Use the add tool."
+  end
+end
+```
+
+If a prompt has no tools, `tools` now returns `[]` instead of `nil`.
+
+### `run` automatically loops over tool calls
+
+When the assistant returns `tool_use` content blocks, `Prompt#run` now:
+
+1. Finds the matching class in `TOOLS` by tool name.
+2. Executes `tool_class.new.execute(input)`.
+3. Appends the assistant message and a user `tool_result` message.
+4. Calls `stream` again.
+5. Repeats until the response has no `tool_use` blocks.
+
+Unknown tools and tool execution errors are returned to the model as `tool_result` content rather than raised.
+
+### Prompt input is resolved once per run
+
+`prompt` is evaluated once at the start of `run`. The same initial input is used when building follow-up messages for tool results, so dynamic or expensive prompt builders are not re-evaluated during a single run.
+
+### `Prompt#stream` accepts explicit input and forwards reasoning/cache options
+
+`stream` now has this signature:
+
+```ruby
+stream(input = prompt, provider: nil, model: nil, reasoning: nil, **options, &block)
+```
+
+You can still call `stream` with no input, but subclasses or callers can now provide a transcript directly:
+
+```ruby
+prompt.stream([{ role: "user", content: "Hello" }], model: "gpt-5.4")
+```
+
+`Prompt` also now forwards `reasoning:` when configured on the class, instance, `run`, or `stream` call.
+
+### Prompt-level cache options
+
+Prompt instances accept and forward cache options:
+
+```ruby
+SummaryPrompt.new(
+  provider: provider,
+  model: "gpt-5.4",
+  cache_key: "summary-v1",
+  cache_retention: "short"
+).run
+```
+
+These are passed to providers as managed `cache_key` / `cache_retention` stream options. For providers that support cache control on system/tool blocks, `cache_retention` may also apply cache metadata to the prompt-owned `system_prompt` and tool definitions. Tool classes can also opt into cache metadata with `cache true`.
+
+### Stream callbacks may see server-tool events and content blocks
+
+Provider-hosted tools (for example OpenAI code interpreter or Anthropic code execution) are normalized as distinct server-tool blocks:
+
+- `server_tool_use`
+- `server_tool_result`
+- provider-specific `*_tool_result` blocks during streaming/finalization
+
+Stream callbacks may now receive additional event types when server tools are used:
+
+- `:tool_result_start`
+- `:tool_result_delta`
+- `:tool_result_end`
+
+`tool_start` events also expose `event.tool_type`, which is either `"tool_use"` or `"server_tool_use"`.
+
+If your stream handler exhaustively switches on event/content types, add fallbacks or handlers for these server-tool cases. Cross-provider handoff sanitization may convert server-tool blocks to regular `tool_use` / `tool_result` blocks when replaying transcripts on a different provider/API.
+
+## Migration checklist
+
+- [ ] Replace positional `Prompt.new(provider, model)` calls with `Prompt.new(provider: provider, model: model)`.
+- [ ] Replace prompt class setter-style calls (`provider value`, `model value`) with `self.provider = value` / `self.model = value`.
+- [ ] Update custom provider/test doubles used by `Prompt` to return `AssistantMessage`.
+- [ ] Remove `extract_response` and `parse_response` hooks; inspect, parse, or transform the returned `AssistantMessage` after `run`.
+- [ ] Update `after_execute` callbacks to accept the final `AssistantMessage` only.
+- [ ] Replace direct mutations of `before_execute_callbacks` / `after_execute_callbacks` with the callback registration methods or explicit subclass assignments.
+- [ ] Move prompt tool definitions to a `TOOLS = [ToolClass]` constant.
+- [ ] Account for automatic tool-loop execution in `run`.
+- [ ] Update any `tools.nil?` checks; no-tool prompts now expose `[]`.
+- [ ] Use `cache_key:` / `cache_retention:` on prompt instances when prompt caching is needed.
+- [ ] Add stream/content handling for server-tool event types if your callback code is exhaustive.

data/lib/llm_gateway/adapters/adapter.rb

@@ -15,12 +15,15 @@ module LlmGateway
         raise LlmGateway::Errors::MissingMapperForProvider, "No stream_mapper configured" unless stream_mapper
 
         normalized_input = map_input({
-          messages: sanitize_messages(normalize_messages(message)),
+          messages: sanitize_messages(normalize_messages(message), target_model: options[:model]),
           tools: tools,
           system: normalize_system(system)
         })
 
-        mapper = stream_mapper.new
+        mapper = stream_mapper.new(
+          provider: LlmGateway::Client.provider_id_from_client(client),
+          api: api_name
+        )
 
         perform_stream(
           normalized_input[:messages],
@@ -31,12 +34,7 @@ module LlmGateway
           mapper.map(chunk, &block)
         end
 
-        AssistantMessage.new(
-          mapper.result.merge(
-            provider: LlmGateway::Client.provider_id_from_client(client),
-            api: api_name
-          )
-        )
+        mapper.result
       end
 
       def upload_file(filename:, content:, mime_type: "application/octet-stream", purpose: "assistants")
@@ -99,14 +97,13 @@ module LlmGateway
         nil
       end
 
-      def sanitize_messages(messages)
+      def sanitize_messages(messages, target_model: nil)
         return messages unless input_sanitizer
 
         target_provider = LlmGateway::Client.provider_id_from_client(client)
         target_api = api_name
-        target_model = client.model_key
 
-        return messages if target_provider.nil? || target_api.nil? || target_model.nil?
+        return messages unless target_provider.present? && target_api.present? && target_model.present?
 
         input_sanitizer.sanitize(
           messages,

data/lib/llm_gateway/adapters/anthropic/input_mapper.rb

@@ -26,6 +26,8 @@ module LlmGateway
             map_tool_use_content(content)
           when "tool_result"
             map_tool_result_content(content)
+          when "server_tool_result"
+            map_server_tool_result_content(content)
           when "thinking", "reasoning"
             map_reasoning_content(content)
           else
@@ -122,6 +124,28 @@ module LlmGateway
             }
           end
 
+          def map_server_tool_result_content(content)
+            {
+              type: native_server_tool_result_type(content),
+              tool_use_id: content[:tool_use_id],
+              content: content[:content]
+            }
+          end
+
+          def native_server_tool_result_type(content)
+            return content[:name] if content[:name] && content[:name] != "server_tool_result"
+
+            result_type = content.dig(:content, :type)
+            case result_type
+            when "bash_code_execution_result"
+              "bash_code_execution_tool_result"
+            when /^text_editor_code_execution_.*_result$/
+              "text_editor_code_execution_tool_result"
+            else
+              content[:name] || "server_tool_result"
+            end
+          end
+
           def map_reasoning_content(content)
             result = {
               type: "thinking",