vibe-sort 0.3.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/docs/README.md CHANGED
@@ -2,7 +2,7 @@
2
2
 
3
3
  Welcome to the VibeSort documentation! This directory contains comprehensive guides for using, understanding, and contributing to the VibeSort gem.
4
4
 
5
- ## 📚 Documentation Overview
5
+ ## Documentation Overview
6
6
 
7
7
  ### [Architecture Overview](architecture.md)
8
8
 
@@ -52,7 +52,7 @@ Welcome to the VibeSort documentation! This directory contains comprehensive gui
52
52
 
53
53
  ---
54
54
 
55
- ## 🚀 Quick Links
55
+ ## Quick Links
56
56
 
57
57
  ### For Users
58
58
 
@@ -76,7 +76,7 @@ Welcome to the VibeSort documentation! This directory contains comprehensive gui
76
76
 
77
77
  ---
78
78
 
79
- ## 📖 Reading Guide
79
+ ## Reading Guide
80
80
 
81
81
  ### I want to use VibeSort in my project
82
82
 
@@ -107,25 +107,25 @@ Welcome to the VibeSort documentation! This directory contains comprehensive gui
107
107
 
108
108
  ---
109
109
 
110
- ## 🏗️ Architecture at a Glance
110
+ ## Architecture at a Glance
111
111
 
112
112
  ```
113
113
  User Application
114
- ↓
114
+
115
115
  VibeSort::Client (validates input, handles errors)
116
- ↓
116
+
117
117
  VibeSort::Configuration (stores API key, settings)
118
- ↓
118
+
119
119
  VibeSort::Sorter (communicates with OpenAI)
120
- ↓
120
+
121
121
  OpenAI API (GPT model processes request)
122
- ↓
122
+
123
123
  Sorted Array (returned to user)
124
124
  ```
125
125
 
126
126
  ---
127
127
 
128
- ## 📝 Example Usage
128
+ ## Example Usage
129
129
 
130
130
  ```ruby
131
131
  require 'vibe_sort'
@@ -148,7 +148,7 @@ For more examples, see the [README](../README.md) or [API Reference](api_referen
148
148
 
149
149
  ---
150
150
 
151
- ## 🔧 Key Concepts
151
+ ## Key Concepts
152
152
 
153
153
  ### Configuration
154
154
 
@@ -175,7 +175,7 @@ VibeSort never raises exceptions to user code. All errors are caught and returne
175
175
 
176
176
  ---
177
177
 
178
- ## 🤝 Contributing
178
+ ## Contributing
179
179
 
180
180
  We welcome contributions! Please see:
181
181
 
@@ -185,22 +185,22 @@ We welcome contributions! Please see:
185
185
 
186
186
  ---
187
187
 
188
- ## 📞 Support
188
+ ## Support
189
189
 
190
- - 🐛 [Report a Bug](https://github.com/chayut/vibe-sort/issues/new?labels=bug)
191
- - 💡 [Request a Feature](https://github.com/chayut/vibe-sort/issues/new?labels=enhancement)
192
- - 📖 [Read the Docs](README.md)
193
- - 💬 [Discussions](https://github.com/chayut/vibe-sort/discussions)
190
+ - [Report a Bug](https://github.com/chayut/vibe-sort/issues/new?labels=bug)
191
+ - [Request a Feature](https://github.com/chayut/vibe-sort/issues/new?labels=enhancement)
192
+ - [Read the Docs](README.md)
193
+ - [Discussions](https://github.com/chayut/vibe-sort/discussions)
194
194
 
195
195
  ---
196
196
 
197
- ## 📜 License
197
+ ## License
198
198
 
199
199
  VibeSort is released under the [MIT License](../LICENSE.txt).
200
200
 
201
201
  ---
202
202
 
203
- ## 🙏 Acknowledgments
203
+ ## Acknowledgments
204
204
 
205
205
  - Built with [Faraday](https://lostisland.github.io/faraday/)
206
206
  - Powered by [OpenAI](https://openai.com/)
@@ -208,4 +208,4 @@ VibeSort is released under the [MIT License](../LICENSE.txt).
208
208
 
209
209
  ---
210
210
 
211
- **Happy Sorting! 🌀**
211
+ **Happy Sorting! **
@@ -6,7 +6,7 @@ The main public interface for the VibeSort gem.
6
6
 
7
7
  ### Class Methods
8
8
 
9
- #### `new(api_key:, temperature: 0.0, provider: :openai, model: nil)`
9
+ #### `new(api_key:, temperature: 0.0, provider: nil, model: nil, extra_params: {})`
10
10
 
11
11
  Creates a new VibeSort client instance.
12
12
 
@@ -18,24 +18,27 @@ Creates a new VibeSort client instance.
18
18
  - Lower values (0.0-0.3): More deterministic and consistent
19
19
  - Higher values (0.7-2.0): More random and creative
20
20
  - Ignored by providers whose current models do not accept it (Anthropic)
21
- - `provider` (Symbol or String, optional): LLM provider (default: `:openai`)
22
- - Supported: `:openai`, `:anthropic`, `:gemini`, `:groq`, `:spacexai`
21
+ - `provider` (Symbol or String, optional): LLM provider
22
+ - Supported: `:openai`, `:anthropic`, `:gemini`, `:groq`, `:spacexai`, `:openrouter`
23
+ - When omitted, inferred from the `api_key` prefix via `VibeSort::KeyDetector`; unrecognized formats default to `:openai`
24
+ - When explicit and the key prefix suggests a different provider, a one-line warning is printed to stderr and the explicit provider is used
23
25
  - `model` (String, optional): Model ID override (default: the provider's default model)
26
+ - `extra_params` (Hash, optional): Provider-native request parameters deep-merged into the payload last, so they can override anything the adapter builds (default: `{}`)
24
27
 
25
28
  **Returns:** `VibeSort::Client` instance
26
29
 
27
30
  **Raises:**
28
31
 
29
- - `ArgumentError`: If api_key is nil or empty, or the provider is unknown
32
+ - `ArgumentError`: If api_key is nil or empty, or an explicit provider is unknown
30
33
 
31
34
  **Example:**
32
35
 
33
36
  ```ruby
34
- # Basic initialization (OpenAI, default provider)
35
- client = VibeSort::Client.new(api_key: ENV['OPENAI_API_KEY'])
37
+ # Provider inferred from the key prefix (sk-ant-... => :anthropic)
38
+ client = VibeSort::Client.new(api_key: ENV['ANTHROPIC_API_KEY'])
36
39
 
37
- # Anthropic Claude
38
- client = VibeSort::Client.new(provider: :anthropic, api_key: ENV['ANTHROPIC_API_KEY'])
40
+ # Explicit provider
41
+ client = VibeSort::Client.new(provider: :openai, api_key: ENV['OPENAI_API_KEY'])
39
42
 
40
43
  # Google Gemini with a custom model
41
44
  client = VibeSort::Client.new(provider: :gemini, api_key: ENV['GEMINI_API_KEY'], model: 'gemini-2.5-pro')
@@ -45,6 +48,14 @@ client = VibeSort::Client.new(
45
48
  api_key: ENV['OPENAI_API_KEY'],
46
49
  temperature: 0.2
47
50
  )
51
+
52
+ # With extra provider-native request parameters
53
+ client = VibeSort::Client.new(
54
+ provider: :openrouter,
55
+ api_key: ENV['OPENROUTER_API_KEY'],
56
+ model: 'meta-llama/llama-3.3-70b-instruct',
57
+ extra_params: { max_tokens: 200 }
58
+ )
48
59
  ```
49
60
 
50
61
  ### Instance Methods
@@ -110,7 +121,7 @@ Configuration object for VibeSort. Usually not used directly by end users.
110
121
 
111
122
  ### Class Methods
112
123
 
113
- #### `new(api_key:, temperature: 0.0, provider: :openai, model: nil)`
124
+ #### `new(api_key:, temperature: 0.0, provider: nil, model: nil, extra_params: {})`
114
125
 
115
126
  Creates a new configuration object.
116
127
 
@@ -118,8 +129,9 @@ Creates a new configuration object.
118
129
 
119
130
  - `api_key` (String, required): API key for the selected provider
120
131
  - `temperature` (Float, optional): Model temperature (default: 0.0)
121
- - `provider` (Symbol or String, optional): `:openai` (default), `:anthropic`, `:gemini`, `:groq`, or `:spacexai`
132
+ - `provider` (Symbol, String, or nil, optional): `:openai`, `:anthropic`, `:gemini`, `:groq`, `:spacexai`, or `:openrouter`; nil (default) infers the provider from the api_key prefix, falling back to `:openai`
122
133
  - `model` (String, optional): Model ID override (nil uses the provider's default)
134
+ - `extra_params` (Hash, optional): Provider-native request parameters deep-merged into the payload last (default: `{}`)
123
135
 
124
136
  **Raises:**
125
137
 
@@ -161,6 +173,40 @@ Returns the configured model override, or nil when using the provider default (r
161
173
 
162
174
  **Returns:** String or nil
163
175
 
176
+ #### `extra_params`
177
+
178
+ Returns the configured extra request parameters hash (read-only).
179
+
180
+ **Returns:** Hash
181
+
182
+ ---
183
+
184
+ ## VibeSort::KeyDetector
185
+
186
+ Best-effort provider inference from API key prefixes. Prefixes are conventions, not contracts — detection is only a soft default and a source of non-fatal warnings, never validation.
187
+
188
+ ### Class Methods
189
+
190
+ #### `detect(api_key)`
191
+
192
+ **Parameters:**
193
+
194
+ - `api_key` (String, required): API key to inspect
195
+
196
+ **Returns:** Symbol (detected provider) or nil (unrecognized format)
197
+
198
+ **Prefix map (ordered — specific prefixes checked before `sk-`):**
199
+
200
+ | Prefix | Provider |
201
+ |---|---|
202
+ | `sk-ant-` | `:anthropic` |
203
+ | `sk-or-` | `:openrouter` |
204
+ | `sk-` | `:openai` |
205
+ | `gsk_` | `:groq` |
206
+ | `AIza` | `:gemini` (classic key) |
207
+ | `AQ.` | `:gemini` (newer key format) |
208
+ | `xai-` | `:spacexai` |
209
+
164
210
  ---
165
211
 
166
212
  ## VibeSort::Sorter
@@ -199,7 +245,7 @@ Performs the sorting operation via the configured provider's API.
199
245
 
200
246
  Maps provider symbols to adapter classes.
201
247
 
202
- **Value:** `{ openai:, anthropic:, gemini:, groq:, spacexai: }`
248
+ **Value:** `{ openai:, anthropic:, gemini:, groq:, spacexai:, openrouter: }`
203
249
 
204
250
  ---
205
251
 
@@ -214,11 +260,14 @@ Internal provider adapters. `Providers::Base` holds the shared prompt, Faraday H
214
260
  | `Providers::Gemini` | `https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent` | `gemini-2.5-flash` |
215
261
  | `Providers::Groq` | `https://api.groq.com/openai/v1/chat/completions` | `llama-3.3-70b-versatile` |
216
262
  | `Providers::SpaceXAI` | `https://api.x.ai/v1/chat/completions` | `grok-4` |
263
+ | `Providers::OpenRouter` | `https://openrouter.ai/api/v1/chat/completions` | `openai/gpt-4o-mini` |
217
264
 
218
265
  Notes:
219
266
 
220
267
  - The Anthropic adapter uses structured outputs (JSON schema) and does not send `temperature` (rejected by current Claude models)
221
- - The Groq and SpaceXAI adapters subclass `Providers::OpenAI` (OpenAI-compatible APIs)
268
+ - The Groq, SpaceXAI, and OpenRouter adapters subclass `Providers::OpenAI` (OpenAI-compatible APIs)
269
+ - The OpenRouter adapter sends OpenRouter's recommended attribution headers (`HTTP-Referer`, `X-Title`)
270
+ - `Providers::Base` deep-merges `config.extra_params` into every payload last, so it can override anything the adapter builds
222
271
 
223
272
  ---
224
273
 
@@ -307,7 +356,7 @@ All sorting operations return a consistent hash structure:
307
356
 
308
357
  ### API Errors
309
358
 
310
- The `{Provider}` prefix names the configured provider (OpenAI, Anthropic, Gemini, Groq, or SpaceXAI):
359
+ The `{Provider}` prefix names the configured provider (OpenAI, Anthropic, Gemini, Groq, SpaceXAI, or OpenRouter):
311
360
 
312
361
  | Error Message Pattern | Cause | Solution |
313
362
  |----------------------|-------|----------|
@@ -455,11 +504,12 @@ results = threads.map(&:value)
455
504
  Set the API key for whichever provider you use:
456
505
 
457
506
  ```bash
458
- export OPENAI_API_KEY='sk-your-key-here' # provider: :openai (default)
507
+ export OPENAI_API_KEY='sk-your-key-here' # provider: :openai (fallback default)
459
508
  export ANTHROPIC_API_KEY='your-key-here' # provider: :anthropic
460
509
  export GEMINI_API_KEY='your-key-here' # provider: :gemini
461
510
  export GROQ_API_KEY='your-key-here' # provider: :groq
462
511
  export XAI_API_KEY='your-key-here' # provider: :spacexai
512
+ export OPENROUTER_API_KEY='your-key-here' # provider: :openrouter
463
513
  ```
464
514
 
465
515
  **Usage:**
@@ -472,9 +522,9 @@ client = VibeSort::Client.new(api_key: ENV['OPENAI_API_KEY'])
472
522
 
473
523
  ## Version
474
524
 
475
- Current version: `0.3.0`
525
+ Current version: `0.4.0`
476
526
 
477
527
  ```ruby
478
528
  VibeSort::VERSION
479
- # => "0.3.0"
529
+ # => "0.4.0"
480
530
  ```
data/docs/architecture.md CHANGED
@@ -20,9 +20,10 @@ VibeSort follows a layered architecture with clear separation of concerns:
20
20
  ▼
21
21
  ┌─────────────────────────────────────────┐
22
22
  │ VibeSort::Configuration │
23
- │ - Provider selection │
23
+ │ - Provider selection (explicit or │
24
+ │ inferred via KeyDetector) │
24
25
  │ - API key management │
25
- │ - Model override │
26
+ │ - Model override, extra_params │
26
27
  │ - Temperature settings │
27
28
  └─────────────────┬───────────────────────┘
28
29
  │
@@ -37,10 +38,11 @@ VibeSort follows a layered architecture with clear separation of concerns:
37
38
  │ VibeSort::Providers::Base │
38
39
  │ - Shared prompt │
39
40
  │ - HTTP client (Faraday) │
41
+ │ - extra_params deep merge │
40
42
  │ - Response parsing & validation │
41
43
  ├─────────────────────────────────────────┤
42
44
  │ OpenAI │ Anthropic │ Gemini │ │
43
- │ Groq │ SpaceXAI │
45
+ │ Groq │ SpaceXAI │ OpenRouter │
44
46
  │ (request/response wire formats) │
45
47
  └─────────────────┬───────────────────────┘
46
48
  │
@@ -50,6 +52,7 @@ VibeSort follows a layered architecture with clear separation of concerns:
50
52
  │ api.openai.com │ api.anthropic.com │
51
53
  │ generativelanguage.googleapis.com │
52
54
  │ api.groq.com │ api.x.ai │
55
+ │ openrouter.ai │
53
56
  └─────────────────────────────────────────┘
54
57
  ```
55
58
 
@@ -67,7 +70,7 @@ VibeSort follows a layered architecture with clear separation of concerns:
67
70
  - Return standardized response hashes
68
71
 
69
72
  **Key Methods**:
70
- - `initialize(api_key:, temperature: 0.0, provider: :openai, model: nil)`: Creates client with configuration
73
+ - `initialize(api_key:, temperature: 0.0, provider: nil, model: nil, extra_params: {})`: Creates client with configuration
71
74
  - `sort(array)`: Sorts array and returns result hash
72
75
 
73
76
  **Return Format**:
@@ -84,19 +87,26 @@ VibeSort follows a layered architecture with clear separation of concerns:
84
87
  **Purpose**: Encapsulates configuration settings.
85
88
 
86
89
  **Responsibilities**:
87
- - Store provider selection and API key
88
- - Store model override and temperature setting
90
+ - Resolve the provider: explicit argument wins (with a non-fatal stderr warning on a mismatched key prefix); otherwise inferred from the key prefix via `KeyDetector`, defaulting to `:openai`
91
+ - Store API key, model override, temperature, and extra_params
89
92
  - Validate API key presence and provider name
90
93
 
91
94
  **Key Methods**:
92
- - `initialize(api_key:, temperature: 0.0, provider: :openai, model: nil)`: Creates configuration
93
- - Raises `ArgumentError` if API key is nil/empty or provider is unknown
95
+ - `initialize(api_key:, temperature: 0.0, provider: nil, model: nil, extra_params: {})`: Creates configuration
96
+ - Raises `ArgumentError` if API key is nil/empty or an explicit provider is unknown
94
97
 
95
98
  **Attributes**:
96
99
  - `api_key`: Provider API key (String)
97
- - `provider`: `:openai`, `:anthropic`, `:gemini`, `:groq`, or `:spacexai` (Symbol)
100
+ - `provider`: `:openai`, `:anthropic`, `:gemini`, `:groq`, `:spacexai`, or `:openrouter` (Symbol)
98
101
  - `model`: Model ID override, or nil for the provider default (String or nil)
99
102
  - `temperature`: Model temperature (Float, 0.0-2.0; not sent to Anthropic)
103
+ - `extra_params`: Provider-native request parameters deep-merged into the payload last (Hash)
104
+
105
+ ### VibeSort::KeyDetector
106
+
107
+ **Purpose**: Best-effort provider inference from API key prefixes (`sk-ant-` → `:anthropic`, `sk-or-` → `:openrouter`, `gsk_` → `:groq`, `AIza`/`AQ.` → `:gemini`, `xai-` → `:spacexai`, `sk-` → `:openai`).
108
+
109
+ Prefixes are conventions, not contracts — providers ship multiple key formats concurrently and new formats appear without notice. Detection is therefore only a soft default and a source of non-fatal warnings, never validation: `detect` returns nil for unrecognized formats, and callers fall back to `:openai`.
100
110
 
101
111
  ### VibeSort::Sorter
102
112
 
@@ -113,17 +123,19 @@ VibeSort follows a layered architecture with clear separation of concerns:
113
123
  **`Providers::Base` responsibilities**:
114
124
  - Hold the shared sorting prompt
115
125
  - Build the HTTP connection with Faraday
126
+ - Deep-merge `config.extra_params` into the adapter payload last (nested hashes merge recursively; arrays and scalars replace)
116
127
  - Parse and validate JSON responses (shared across providers)
117
128
  - Raise `ApiError` on failures
118
129
 
119
130
  **Adapter hooks** (implemented per provider): `provider_name`, `endpoint`, `headers`, `build_payload`, `extract_content`
120
131
 
121
132
  **Adapters and default models**:
122
- - `Providers::OpenAI` — Chat Completions, `gpt-4o-mini`
123
- - `Providers::Anthropic` — Messages API with structured outputs (JSON schema), `claude-opus-4-8`
124
- - `Providers::Gemini` — generateContent with JSON response mode, `gemini-2.5-flash`
125
- - `Providers::Groq` — OpenAI-compatible (subclasses `Providers::OpenAI`), `llama-3.3-70b-versatile`
126
- - `Providers::SpaceXAI` — OpenAI-compatible (subclasses `Providers::OpenAI`), `grok-4`
133
+ - `Providers::OpenAI`: Chat Completions, `gpt-4o-mini`
134
+ - `Providers::Anthropic`: Messages API with structured outputs (JSON schema), `claude-opus-4-8`
135
+ - `Providers::Gemini`: generateContent with JSON response mode, `gemini-2.5-flash`
136
+ - `Providers::Groq`: OpenAI-compatible (subclasses `Providers::OpenAI`), `llama-3.3-70b-versatile`
137
+ - `Providers::SpaceXAI`: OpenAI-compatible (subclasses `Providers::OpenAI`), `grok-4`
138
+ - `Providers::OpenRouter`: OpenAI-compatible (subclasses `Providers::OpenAI`), `openai/gpt-4o-mini`, adds OpenRouter's recommended attribution headers
127
139
 
128
140
  ### VibeSort::ApiError
129
141
 
@@ -244,7 +256,7 @@ VibeSort follows a layered architecture with clear separation of concerns:
244
256
  ## Dependencies
245
257
 
246
258
  ### Runtime
247
- - **faraday** (~> 2.0): HTTP client library — the only runtime dependency; all providers are called over plain HTTPS
259
+ - **faraday** (~> 2.0): HTTP client library, the only runtime dependency; all providers are called over plain HTTPS
248
260
 
249
261
  ### Development
250
262
  - **rspec** (~> 3.0): Testing framework
data/docs/development.md CHANGED
@@ -521,8 +521,8 @@ Document all public methods:
521
521
  # @return [Hash] Result hash with :success, :sorted_array, :error keys
522
522
  #
523
523
  # @example
524
- # client.sort([5, 2, 8])
525
- # # => { success: true, sorted_array: [2, 5, 8] }
524
+ # client.sort([5, 2, 8])
525
+ # # => { success: true, sorted_array: [2, 5, 8] }
526
526
  def sort(array)
527
527
  # ...
528
528
  end
@@ -563,4 +563,4 @@ Before submitting a PR:
563
563
  - Read the documentation
564
564
  - Ask in the community
565
565
 
566
- Happy coding! 🚀
566
+ Happy coding!
@@ -0,0 +1,134 @@
1
+ # Free-Tier Model Benchmark — 2026-07-12
2
+
3
+ An empirical snapshot of how free LLM tiers handle vibe-sort's sorting contract, gathered
4
+ while designing v0.4.0 (OpenRouter provider, key-format detection, `extra_params` —
5
+ see [v0.4.0_PLAN.md](v0.4.0_PLAN.md)). All requests used free models / free-tier quota only.
6
+
7
+ **TL;DR:** Google's current Gemini flash models all sort correctly through the gem's exact
8
+ payload, with `gemini-3.1-flash-lite` the clear free-tier pick. OpenRouter's `:free` pool was
9
+ too congested on a Saturday afternoon (AEST) to be usable — 14 of 17 tested models failed with
10
+ rate limits or upstream errors, and one burned 10 minutes and 20k tokens to return nothing.
11
+
12
+ ---
13
+
14
+ ## Methodology
15
+
16
+ - **Task:** sort `[42, "banana", 7, "Apple", 3.14, "cherry", 1]`
17
+ → expected `[1, 3.14, 7, 42, "Apple", "banana", "cherry"]`
18
+ - **Payloads:** byte-for-byte what the gem sends — `Providers::OpenAI#build_payload` for
19
+ OpenRouter (system prompt + `temperature: 0.0` + `response_format: {type: "json_object"}`),
20
+ `Providers::Gemini#build_payload` for Gemini (`systemInstruction` +
21
+ `generationConfig.responseMimeType: "application/json"`).
22
+ - **Success criteria:** HTTP 200, content parses as JSON with a `sorted_array` key
23
+ (the gem's contract), array exactly matches the expected sort.
24
+ - **Fallbacks tried on rejection:** OpenRouter — retry without `response_format`;
25
+ Gemini — retry with system prompt folded into the user turn and no JSON mime type.
26
+ - **Pacing/retries:** 3.5–8s between requests; 5xx retried up to 3×; 429 handling varied by
27
+ pass (see Lessons). Plain Ruby `net/http`, no SDKs.
28
+ - **Cost:** $0. OpenRouter restricted to `$0`-priced models; Gemini restricted to
29
+ free-tier models with non-zero quota.
30
+
31
+ ---
32
+
33
+ ## Gemini API (free tier)
34
+
35
+ 7 models tested — every model whose free-tier quota is non-zero.
36
+
37
+ | Model | Outcome | Latency | Tokens in/out (+thinking) | Notes |
38
+ |---|---|---|---|---|
39
+ | gemini-3.1-flash-lite | ✅ correct | **2.08s** | 87/35 (+0) | Fastest; no thinking tokens; best quota (15 RPM / 500 RPD) |
40
+ | gemini-2.5-flash | ✅ correct | 2.66s | 87/49 (+247) | Gem's current `DEFAULT_MODEL` — still works |
41
+ | gemini-3.5-flash | ✅ correct | 5.98s | 87/50 (+418) | Most thinking tokens |
42
+ | gemini-3-flash-preview | ✅ correct | 24.14s | 87/49 (+321) | Correct but slow |
43
+ | gemma-4-26b-a4b-it | ❌ bad JSON | 28.6s | 87/49 (+318) | Ignored JSON mime type; replied with markdown reasoning; 1×500 on the way |
44
+ | gemma-4-31b-it | ❌ bad JSON | 9.8s | 87/30 (+256) | Same failure shape; also 1×500 |
45
+ | gemini-2.5-flash-lite | ❌ HTTP 404 | — | — | "No longer available to new users" (still listed by the models endpoint) |
46
+
47
+ ### Observations
48
+
49
+ - **All four current Gemini-branded flash models pass the gem's contract as-is** — JSON mode
50
+ honored, sort correct, numbers-before-strings and case-sensitivity respected.
51
+ - **Gemma models are incompatible with the gem today:** they accept the request but ignore
52
+ `responseMimeType: "application/json"` and return reasoning-style markdown. They'd need
53
+ `extra_params` surgery plus response post-processing — not worth it.
54
+ - **The 500s are real:** 2 of 7 models needed a 5xx retry on their very first request.
55
+ Any Gemini integration should retry 5xx at least once.
56
+ - **429 retries are a trap on the free tier:** every retry counts against RPM *and* RPD, so a
57
+ retry loop converts one throttled request into three consumed ones. After the daily caps
58
+ (20 RPD on most flash models) tripped, throttling also turned intermittent — consecutive
59
+ requests to the same model went 429, 429, 200. Policy that survives: never auto-retry a
60
+ free-tier 429; skip and move on.
61
+ - An input-length scaling test (5/20/50/100 elements) was attempted but the RPD caps were
62
+ exhausted before it could run. Future work, pending quota reset — only
63
+ `gemini-3.1-flash-lite` (500 RPD) realistically supports this kind of testing for free.
64
+
65
+ ---
66
+
67
+ ## OpenRouter (free models)
68
+
69
+ 26 models on openrouter.ai were $0-priced with text input (of 345 listed). 17 were tested
70
+ before the sweep was aborted; 9 were never reached. Tested on a Saturday afternoon AEST —
71
+ peak congestion for the shared free pool, which colours everything below.
72
+
73
+ | Model | Outcome | Notes |
74
+ |---|---|---|
75
+ | nvidia/nemotron-3-super-120b-a12b:free | ❌ empty content | **600s, 20,673 completion tokens, no answer** — presumably all reasoning |
76
+ | nvidia/nemotron-3.5-content-safety:free | ❌ bad JSON | Replied "User Safety: safe" — it's a classifier, not a chat model |
77
+ | nvidia/nemotron-nano-12b-v2-vl:free | ❌ upstream timeout | "Upstream idle timeout exceeded" after 121s |
78
+ | nvidia/nemotron-3-nano-30b-a3b:free | ❌ upstream error | NVIDIA "ResourceExhausted: worker limit reached" |
79
+ | nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free | ❌ upstream error | Same |
80
+ | nvidia/nemotron-3-ultra-550b-a55b:free | ❌ upstream error | Same |
81
+ | nvidia/nemotron-nano-9b-v2:free | ⏹ aborted | Stalled the sweep twice with a slow-drip connection (>20 min) |
82
+ | liquid/lfm-2.5-1.2b-instruct:free | ❌ HTTP 502 | Upstream down |
83
+ | liquid/lfm-2.5-1.2b-thinking:free | ❌ HTTP 502 | Upstream down |
84
+ | google/lyria-3-clip-preview | ❌ HTTP 400 | Music model; listed as $0 text-input but not a chat model |
85
+ | google/lyria-3-pro-preview | ❌ HTTP 400 | Same |
86
+ | cognitivecomputations/dolphin-mistral-24b-venice-edition:free | ❌ HTTP 429 | Free pool congested (retried in a 2nd pass ~1h later — still 429) |
87
+ | cohere/north-mini-code:free | ❌ HTTP 429 | Explicit "High demand" RPM-limit message |
88
+ | google/gemma-4-26b-a4b-it:free | ❌ HTTP 429 | Congested both passes |
89
+ | google/gemma-4-31b-it:free | ❌ HTTP 429 | Congested both passes |
90
+ | meta-llama/llama-3.2-3b-instruct:free | ❌ HTTP 429 | Congested both passes |
91
+ | meta-llama/llama-3.3-70b-instruct:free | ❌ HTTP 429 | Congested both passes |
92
+ | nousresearch/hermes-3-llama-3.1-405b:free | ❌ HTTP 429 | Congested both passes |
93
+ | openai/gpt-oss-120b:free, openai/gpt-oss-20b:free, openrouter/free, poolside/laguna-m.1:free, poolside/laguna-xs-2.1:free, qwen/qwen3-coder:free, qwen/qwen3-next-80b-a3b-instruct:free, tencent/hy3:free | ⏸ not tested | Sweep aborted before reaching them (blocked behind the stalling Nemotron) |
94
+
95
+ ### Observations
96
+
97
+ - **Zero of 17 tested free models completed a correct sort** during this window. This is a
98
+ statement about the free pool's availability on a weekend afternoon, not about the models —
99
+ nearly every failure was infrastructure (429/5xx/upstream), not model output.
100
+ - **The transport layer of a vibe-sort OpenRouter adapter is validated**, though: auth,
101
+ routing, the OpenAI-compatible payload, and OpenRouter's error shape
102
+ (`error.message` — including passed-through upstream provider errors) all behaved exactly
103
+ as `Providers::Base#extract_error_message` expects.
104
+ - **Free-model quirks to document:** `$0` pricing does not imply chat-usable (Lyria = music,
105
+ content-safety = classifier); reasoning models can consume enormous token counts and still
106
+ return empty content; per-request wall time is unbounded in practice (slow-drip responses
107
+ blow through per-read timeouts), so an overall deadline is the only safe timeout.
108
+ - Free-model requests are also account-capped on OpenRouter (daily request limits tied to
109
+ credit balance) — fine for smoke tests, unusable for CI.
110
+
111
+ ---
112
+
113
+ ## Implications for vibe-sort v0.4.0
114
+
115
+ 1. **OpenRouter adapter: ship it.** The mechanics are proven even though the free pool was
116
+ congested; with any paid model (or an off-peak free window) it should behave like the
117
+ other OpenAI-compatible adapters. Don't default to a `:free` model — congestion would make
118
+ the gem look broken.
119
+ 2. **Gemini `DEFAULT_MODEL`:** `gemini-2.5-flash` still works, but Google has begun retiring
120
+ 2.5-era models for new users (2.5-flash-lite already 404s). `gemini-3.1-flash-lite` is
121
+ faster here, spends no thinking tokens on this task, and has the widest free quota —
122
+ a strong candidate for the next default.
123
+ 3. **`extra_params` earns its place:** overriding `response_format`, capping `max_tokens` on
124
+ reasoning models, and OpenRouter provider-routing preferences were all wanted during this
125
+ exercise.
126
+ 4. **Never auto-retry 429s against free tiers** in examples/docs; retries multiply quota
127
+ consumption and prolong the lockout.
128
+
129
+ ## Reproduction
130
+
131
+ Standalone Ruby scripts (plain `net/http`, no gem dependencies) sent the exact adapter
132
+ payloads; per-model raw results (latency, token usage, full error messages, returned arrays)
133
+ were captured as JSON. API keys were session-scoped test keys, not committed. Numbers above
134
+ are single-shot measurements on free tiers — directional, not statistical.