llm_gateway 0.3.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 972dac306c8d8f59b41c462f0133d0ce415e689515fbd84d5503541a9fee6f93
-  data.tar.gz: 6014254f858af1b83906b29950811e2ce84940f0ff49301a9b36dd19f592b7a8
+  metadata.gz: ce9b9e4f2137a73474b1ed5f0876d8b1bf6185666ab8c756c3a3f67e99e9d86e
+  data.tar.gz: 5735832e4bd57946ffc0a251c5a3a0861af0fc12a989456e1f877675e08846ba
 SHA512:
-  metadata.gz: 388573aa9afe19a075c561f17d5e4a6b8e550de25b73c3e6d9583403096f1c26f13667055d5920408c2c06f9add888af81a0bf34434762b8ece4a58de32ee3c3
-  data.tar.gz: 42d0b37c296ee058109cb38daaeefc269ce128a39bfbb3b89745eb734653a2828ec8c54f8acd1a12242aa03d1706304507c627074a53e4e6cddc7eb27324d69d
+  metadata.gz: afd52f4ead29acf7a612a06456e203e295534d2cb2275a7ea99be5840da39a821f4727402687bd9c3696bc0081c12f09861aa1b7ad135f986054625c68341422
+  data.tar.gz: 9c033b13f91e9315aadedca98cb61e32c01584a4e6cbe4f05b3782eb84287d24e50dbbc5e1bc127d14bc362d2aac262a589b810614a01d432d02f72acb3013a7

data/.pi/skills/live-provider-testing/SKILL.md

@@ -0,0 +1,183 @@
+---
+name: live-provider-testing
+description: Use when adding or updating llm_gateway live provider integration tests, stream tests, recorded handoff fixtures, or handoff tests that replay outputs across provider/model pairs.
+---
+
+# Live Provider Testing
+
+Use this skill when working on live integration tests under `test/integration/live/`, especially tests that validate multiple provider/model pairs, record final stream outputs, or create handoff tests from those recordings.
+
+## Core pattern
+
+Live provider tests use a shared `PAIRS` constant and generate one test per provider/model pair.
+
+```ruby
+PAIRS = [
+  { provider: "openai_apikey_completions", model: "gpt-5.1" },
+  { provider: "anthropic_apikey_messages", model: "claude-sonnet-4-20250514" },
+  { provider: "openai_apikey_responses", model: "gpt-5.4" },
+  { provider: "anthropic_oauth_messages", model: "claude-sonnet-4-20250514" },
+  { provider: "openai_oauth_codex", model: "gpt-5.4" }
+].freeze
+```
+
+Define tests by iterating over `PAIRS`, not by manually repeating calls:
+
+```ruby
+def self.define_stream_tests_for(provider:, model:)
+  test "live_text_streaming_#{provider}_#{model}" do
+    with_vcr_adapter(provider:, model:) do |adapter|
+      response = basic_streaming_text_test(adapter)
+      record_live_handoff_result(test_file: __FILE__, provider:, model:, result: response)
+    end
+  end
+end
+
+PAIRS.each do |pair|
+  define_stream_tests_for(provider: pair[:provider], model: pair[:model])
+end
+```
+
+Always include `LiveTestHelper` and reset configuration in teardown:
+
+```ruby
+include LiveTestHelper
+
+def teardown
+  LlmGateway.reset_configuration!
+end
+```
+
+## Running adapters
+
+Use `with_vcr_adapter(provider:, model:)` for live/VCR-backed provider tests. It handles:
+
+- provider configuration
+- API key and OAuth credential lookup
+- VCR cassette naming
+- replay tokens for OAuth providers
+- authentication skips
+
+Do not construct provider clients directly inside live tests unless the test specifically targets client construction.
+
+## Stream tests that record outputs
+
+The stream tests currently record final results for later handoff tests:
+
+- `test/integration/live/stream_image_test.rb`
+- `test/integration/live/stream_reasoning_test.rb`
+- `test/integration/live/stream_test.rb`
+
+Their helper methods should return the final `AssistantMessage` response after assertions pass. Generated tests then call:
+
+```ruby
+record_live_handoff_result(test_file: __FILE__, provider:, model:, result: response)
+```
+
+This writes JSON under:
+
+```text
+test/fixtures/handoff/{source_test_name_without_rb}/{provider_model}.json
+```
+
+Example:
+
+```text
+test/fixtures/handoff/stream_test/openai_apikey_completions_gpt-5.1.json
+```
+
+The JSON file is keyed by the current Minitest test name with the `test_` prefix removed, so multiple generated tests for the same pair can share one pair file.
+
+## Handoff tests from recorded stream outputs
+
+Handoff stream tests are separate files and must not modify the existing handoff tests:
+
+- Existing handoff tests to leave alone:
+  - `test/integration/live/handoff_test.rb`
+  - `test/integration/live/handoff_media_test.rb`
+- Stream handoff tests:
+  - `test/integration/live/handoff_stream_image_test.rb`
+  - `test/integration/live/handoff_stream_reasoning_test.rb`
+  - `test/integration/live/handoff_stream_test.rb`
+
+Each stream handoff test should:
+
+1. Read `PAIRS` from the source stream test file.
+2. Load all recorded output JSON files from the matching fixture directory.
+3. Send those recorded outputs to each provider/model pair.
+4. Assert that the receiving model understood the previous outputs.
+
+Important: the prompt must not interpolate `records.length` or otherwise tell the model the count. The model should infer the count from the supplied JSON/transcript. It is fine for the assertion to compare against `records.length.to_s`.
+
+Example prompt style:
+
+```ruby
+prompt = <<~PROMPT
+  You are receiving recorded final outputs from previous image streaming tests.
+  Each output describes the same image. Read the JSON, infer what all previous assistants saw,
+  and answer in one short sentence. Include the shape, color, and the number of recorded outputs
+  as an Arabic numeral.
+
+  #{JSON.pretty_generate(records)}
+PROMPT
+```
+
+Example assertions:
+
+```ruby
+text = response.content.select { |block| block.type == "text" }.map(&:text).join(" ").downcase
+assert_includes text, "red"
+assert_includes text, "circle"
+assert_includes text, records.length.to_s
+```
+
+## Reading pairs from source tests
+
+When creating a handoff test from a source stream test, keep the provider matrix coupled to the source test by reading its `PAIRS` definition:
+
+```ruby
+SOURCE_TEST_PATH = File.expand_path("stream_image_test.rb", __dir__)
+PAIRS = eval(File.read(SOURCE_TEST_PATH).match(/PAIRS = (\[.*?\])\s*\.freeze/m)[1]).freeze
+```
+
+Only use this pattern for test code where the source file is trusted repository code.
+
+## Fixture loading pattern
+
+Handoff tests should skip cleanly if recordings do not exist:
+
+```ruby
+def load_recorded_outputs
+  skip "Missing fixture directory at #{FIXTURE_DIR}. Run stream_image_test live tests first." unless Dir.exist?(FIXTURE_DIR)
+
+  Dir.glob(File.join(FIXTURE_DIR, "*.json")).sort.map do |path|
+    {
+      pair: File.basename(path, ".json"),
+      result: JSON.parse(File.read(path))
+    }
+  end.tap do |records|
+    skip "No recorded outputs in #{FIXTURE_DIR}. Run stream_image_test live tests first." if records.empty?
+  end
+end
+```
+
+## Validation expectations
+
+Use assertions that prove semantic handoff, not exact wording:
+
+- Image handoff: assert the response mentions `red`, `circle`, and the inferred fixture count.
+- Reasoning handoff: assert the response mentions `69` and the inferred fixture count.
+- General stream handoff: assert the response mentions the inferred fixture count and expected tool/math results such as `42`, `714`, and `887`.
+
+Avoid brittle assertions against full response text.
+
+## Syntax and test checks
+
+After editing Ruby tests, at minimum run syntax checks:
+
+```bash
+ruby -c test/integration/live/<file>.rb
+ruby -c test/utils/live_test_helper.rb
+```
+
+Run the actual live tests only when requested or when credentials/VCR setup is available, since they may require API keys, OAuth credentials, and network access.

data/.pi/skills/options-development/SKILL.md

@@ -0,0 +1,131 @@
+---
+name: options-development
+description: Use when developing or updating provider option mappers in llm_gateway from an API reference URL and optional API hint. Guides managed option mapping, valid option whitelists, source comments, tests, and client behavior boundaries.
+---
+
+# Options Development
+
+Use this skill when the user asks to build, update, or audit an LLM provider/API option mapper. The user should provide:
+
+- an API reference URL, e.g. `https://platform.claude.com/docs/en/api/messages/create`
+- a hint identifying the API when a provider has multiple APIs, e.g. `Anthropic Messages`, `OpenAI Responses`, `OpenAI Chat Completions`, `Groq Chat Completions`, `OpenAI Codex`
+
+## Goal
+
+Keep option handling split into clear layers:
+
+1. **Managed options**: library-owned options that should work consistently across clients.
+2. **Option mapper**: translates managed options to provider/API option names and shapes. If an option is not managed but is a valid provider option, pass it through. Every mapper must validate the final transformed hash against a whitelist of valid provider options and reject unknown keys.
+3. **Client behavior**: clients receive already-mapped options and must not do additional option mapping. A client should behave exactly like the provider API docs for its endpoint.
+4. **Tools and transcript**: option mappers must not map tools, transcript, messages, or system content.
+5. **Current limitation**: mapped options are currently pushed only into the stream path; do not broaden this unless the task explicitly asks for architecture work.
+
+## Current managed options
+
+- `reasoning`
+  - supported values: `"none"`, `"low"`, `"medium"`, `"high"`, `"xhigh"`
+  - mapped differently depending on the provider/API
+- `max_completion_tokens`
+  - default is usually `20_480`
+  - Anthropic maps this to `max_tokens`
+  - OpenAI Responses maps this to `max_output_tokens`
+- `response_format`
+  - examples: `"text"`, `{ type: "json_object" }`, `{ type: "json_schema" }`
+- `cache_key`
+  - OpenAI maps this to `prompt_cache_key`
+- `cache_retention`
+  - OpenAI values: `"short"`, `"long"`, `"none"`
+  - Anthropic passes this through as `cache_retention`
+- `temperature`
+  - Groq defaults this to `0`
+
+These are **not** options and must not be handled by option mappers:
+
+- `message` / `messages` / transcript
+- `tools`
+- `system`
+
+## Repository locations
+
+Option mappers live under `lib/llm_gateway/adapters/`, including:
+
+- `lib/llm_gateway/adapters/anthropic_option_mapper.rb`
+- `lib/llm_gateway/adapters/groq/option_mapper.rb`
+- `lib/llm_gateway/adapters/openai/chat_completions/option_mapper.rb`
+- `lib/llm_gateway/adapters/openai/responses/option_mapper.rb`
+- `lib/llm_gateway/adapters/openai_codex/option_mapper.rb`
+
+Tests live under `test/unit/options/`.
+
+## Required workflow
+
+1. **Identify mapper and tests**
+   - Use the API hint to find the corresponding option mapper and test file.
+   - Inspect the related adapter/client only to verify mapping boundaries; do not move mapping into clients.
+
+2. **Read the provider API reference**
+   - Fetch/read the provided URL if network access is available, for example with `curl -L <url>`.
+   - Extract every request-body option accepted by the endpoint.
+   - Exclude non-option structural fields such as messages/input/transcript, tools, and system/developer instructions.
+   - If docs are not fetchable, say so and ask the user for the relevant request parameter list before coding.
+
+3. **Add/maintain a source comment at the option mapper**
+   - Near the valid-option whitelist in the mapper, add a comment containing:
+     - source URL
+     - API name/hint
+     - date accessed
+     - the full list of valid option keys copied from the API reference
+   - Keep this comment updated whenever the whitelist changes.
+
+4. **Implement managed option mapping in the mapper only**
+   - Match the coding style and structure of the closest existing mapper before changing behavior. For example, Anthropic and OpenAI Chat Completions use named default constants, `VALID_OPTIONS`, `MANAGED_OPTIONS`, a `map` method that builds `mapped_options`, explicit normalizer helpers, and `validate_options!` near the mapper.
+   - Prefer `mapped_options = options.reject { |key, _| MANAGED_OPTIONS.include?(key) }` when a mapper has multiple managed aliases; this makes alias removal explicit and keeps pass-through provider-native options obvious.
+   - Remove managed aliases after mapping so the final hash contains only provider-native option keys.
+   - Preserve valid provider-native options that are not managed.
+   - Apply provider-specific defaults only in the mapper.
+   - Unless the user explicitly asks for default behavior changes, do not modify existing defaults.
+   - Do not map tools, transcript/messages, or system.
+
+5. **Whitelist after transformation**
+   - Define a `VALID_OPTIONS`/equivalent whitelist from the API reference.
+   - After all transformations, reject any final key not in the whitelist.
+   - Prefer raising `ArgumentError` with a useful message listing unknown option keys and/or valid keys.
+   - Validate the returned hash, not merely the input hash, so bad mapped output is caught.
+
+6. **Tests**
+   - Match the structure and naming style of the closest existing provider/API option test before adding cases. For Anthropic and OpenAI Chat Completions, prefer a compact set of broad tests:
+     - one adapter-boundary test named like `passes mapped managed options and provider-native options through adapter to client`
+     - one unknown provider option rejection test
+     - one structural field rejection test
+     - one superset/final output mapper test
+   - Prefer testing option behavior at the adapter boundary by stubbing the provider client and asserting the exact options passed to the client's request/stream method. This verifies that managed options are mapped before the client and that valid provider-native options pass through the adapter layer unchanged.
+   - Keep pure mapper tests for validation/error behavior, final-output superset assertions, and small normalization helpers when useful, but avoid relying only on direct `OptionMapper.map(...)` assertions for passthrough behavior.
+   - Fake clients in adapter-boundary stream tests should yield enough realistic stream chunks for the adapter's stream mapper and accumulator to complete without errors; do not yield only terminal/usage chunks if the mapper expects started content/tool state.
+   - Add/update tests for:
+     - each managed option mapping relevant to the provider/API
+     - pass-through of valid provider-native options together with representative managed options in the same adapter-level test
+     - rejection of unknown options after transformation
+     - no handling of tools/messages/system in the mapper
+     - provider-specific defaults, if any
+   - For adapter-level option tests:
+     - instantiate the real adapter with a fake/stub client for the target provider/API
+     - call the public adapter method (`stream` today) with managed and provider-native options
+     - capture the keyword args received by the client method
+     - assert the final provider-native option hash, including mapped managed options and unchanged provider-native options
+     - ensure fake clients provide whatever minimal stream/result events are needed so the adapter can complete without network or VCR
+   - After making option-mapper changes, run the targeted option tests, then the broader test suite if practical.
+   - If VCR/cassette-backed tests fail and option changes are the only code changes, do not immediately re-record or mutate cassettes. First explain why the VCR is failing (for example: request body changed because a new/default option is now sent, unknown option rejection changed control flow, or provider-native option passthrough altered the recorded request). Ask for confirmation before taking further VCR steps.
+
+## Implementation checklist
+
+Before finishing, verify:
+
+- [ ] mapper has source/API/date/full-valid-option-list comment
+- [ ] mapper maps all relevant managed options and deletes aliases
+- [ ] valid provider-native options pass through unchanged
+- [ ] final returned options are whitelist-validated
+- [ ] clients do not perform additional option mapping
+- [ ] tools/transcript/messages/system are not mapped as options
+- [ ] defaults were not modified unless explicitly requested
+- [ ] tests cover mapping, pass-through, rejection, and defaults
+- [ ] tests were run after option changes; any VCR failure was explained before further cassette work

data/CHANGELOG.md

@@ -1,5 +1,48 @@
 # Changelog
 
+## [v0.5.0](https://github.com/Hyper-Unearthing/llm_gateway/tree/v0.5.0) (2026-05-20)
+
+[Full Changelog](https://github.com/Hyper-Unearthing/llm_gateway/compare/v0.4.0...v0.5.0)
+
+**Merged pull requests:**
+
+- Refactor stream mapper accumulation [\#61](https://github.com/Hyper-Unearthing/llm_gateway/pull/61) ([billybonks](https://github.com/billybonks))
+- feat\(groq\): add stream support for groq [\#60](https://github.com/Hyper-Unearthing/llm_gateway/pull/60) ([billybonks](https://github.com/billybonks))
+- test\(feat\): allow options to be passed in model pairs [\#59](https://github.com/Hyper-Unearthing/llm_gateway/pull/59) ([billybonks](https://github.com/billybonks))
+- Focus streaming and Claude client tests [\#58](https://github.com/Hyper-Unearthing/llm_gateway/pull/58) ([billybonks](https://github.com/billybonks))
+- feat\(test\): automatically delete unused vcrs [\#57](https://github.com/Hyper-Unearthing/llm_gateway/pull/57) ([billybonks](https://github.com/billybonks))
+- refactor: handoff test [\#56](https://github.com/Hyper-Unearthing/llm_gateway/pull/56) ([billybonks](https://github.com/billybonks))
+- Refactor/options clients [\#55](https://github.com/Hyper-Unearthing/llm_gateway/pull/55) ([billybonks](https://github.com/billybonks))
+- burn: all the old code [\#54](https://github.com/Hyper-Unearthing/llm_gateway/pull/54) ([billybonks](https://github.com/billybonks))
+- test: only skip actual auth errors [\#53](https://github.com/Hyper-Unearthing/llm_gateway/pull/53) ([billybonks](https://github.com/billybonks))
+- test: dont try refresh token when using vcr only when regenerating [\#52](https://github.com/Hyper-Unearthing/llm_gateway/pull/52) ([billybonks](https://github.com/billybonks))
+
+## [v0.4.0](https://github.com/Hyper-Unearthing/llm_gateway/tree/v0.4.0) (2026-05-17)
+
+[Full Changelog](https://github.com/Hyper-Unearthing/llm_gateway/compare/v0.3.0...v0.4.0)
+
+**Merged pull requests:**
+
+- docs: update docs new code [\#51](https://github.com/Hyper-Unearthing/llm_gateway/pull/51) ([billybonks](https://github.com/billybonks))
+- test: rework live tests to use vcr [\#50](https://github.com/Hyper-Unearthing/llm_gateway/pull/50) ([billybonks](https://github.com/billybonks))
+- refactor: update provider keys [\#49](https://github.com/Hyper-Unearthing/llm_gateway/pull/49) ([billybonks](https://github.com/billybonks))
+- Refactor/major internal organisation [\#48](https://github.com/Hyper-Unearthing/llm_gateway/pull/48) ([billybonks](https://github.com/billybonks))
+- cross provider handoff support [\#47](https://github.com/Hyper-Unearthing/llm_gateway/pull/47) ([billybonks](https://github.com/billybonks))
+- Refactor provider usage and especially oauth [\#46](https://github.com/Hyper-Unearthing/llm_gateway/pull/46) ([billybonks](https://github.com/billybonks))
+- Refactor/options [\#45](https://github.com/Hyper-Unearthing/llm_gateway/pull/45) ([billybonks](https://github.com/billybonks))
+- fix: map content blocks in tools as well [\#44](https://github.com/Hyper-Unearthing/llm_gateway/pull/44) ([billybonks](https://github.com/billybonks))
+- Streaming support [\#43](https://github.com/Hyper-Unearthing/llm_gateway/pull/43) ([billybonks](https://github.com/billybonks))
+- feat: try to simplify api and provider combination [\#42](https://github.com/Hyper-Unearthing/llm_gateway/pull/42) ([billybonks](https://github.com/billybonks))
+- feat: add reasoning effort parameter [\#41](https://github.com/Hyper-Unearthing/llm_gateway/pull/41) ([billybonks](https://github.com/billybonks))
+- Feat/backport streaming [\#40](https://github.com/Hyper-Unearthing/llm_gateway/pull/40) ([billybonks](https://github.com/billybonks))
+- fix: cache control [\#39](https://github.com/Hyper-Unearthing/llm_gateway/pull/39) ([billybonks](https://github.com/billybonks))
+- feat: configure [\#38](https://github.com/Hyper-Unearthing/llm_gateway/pull/38) ([billybonks](https://github.com/billybonks))
+- fix: alias clients to old classname until 1.0 [\#36](https://github.com/Hyper-Unearthing/llm_gateway/pull/36) ([billybonks](https://github.com/billybonks))
+- refactor: adapter to split clients from adapter [\#35](https://github.com/Hyper-Unearthing/llm_gateway/pull/35) ([billybonks](https://github.com/billybonks))
+- test: make tests less brittle when regenerating vcr [\#34](https://github.com/Hyper-Unearthing/llm_gateway/pull/34) ([billybonks](https://github.com/billybonks))
+- Clean up gems and delete sample code [\#33](https://github.com/Hyper-Unearthing/llm_gateway/pull/33) ([billybonks](https://github.com/billybonks))
+- feat: add clade subscription as a seperate provider [\#32](https://github.com/Hyper-Unearthing/llm_gateway/pull/32) ([billybonks](https://github.com/billybonks))
+
 ## [v0.3.0](https://github.com/Hyper-Unearthing/llm_gateway/tree/v0.3.0) (2025-08-19)
 
 [Full Changelog](https://github.com/Hyper-Unearthing/llm_gateway/compare/v0.2.0...v0.3.0)