vibe-sort 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +26 -1
- data/README.md +105 -49
- data/docs/IMPLEMENTATION_PLAN.md +50 -50
- data/docs/README.md +20 -20
- data/docs/api_reference.md +66 -16
- data/docs/architecture.md +27 -15
- data/docs/development.md +3 -3
- data/docs/free_model_benchmark_2026-07-12.md +134 -0
- data/docs/v0.2.0_UPDATE.md +37 -37
- data/lib/vibe_sort/client.rb +22 -8
- data/lib/vibe_sort/configuration.rb +32 -8
- data/lib/vibe_sort/providers/base.rb +15 -1
- data/lib/vibe_sort/sorter.rb +2 -1
- data/lib/vibe_sort/version.rb +1 -1
- data/lib/vibe_sort.rb +2 -0
- metadata +10 -5
data/docs/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Welcome to the VibeSort documentation! This directory contains comprehensive guides for using, understanding, and contributing to the VibeSort gem.
|
|
4
4
|
|
|
5
|
-
##
|
|
5
|
+
## Documentation Overview
|
|
6
6
|
|
|
7
7
|
### [Architecture Overview](architecture.md)
|
|
8
8
|
|
|
@@ -52,7 +52,7 @@ Welcome to the VibeSort documentation! This directory contains comprehensive gui
|
|
|
52
52
|
|
|
53
53
|
---
|
|
54
54
|
|
|
55
|
-
##
|
|
55
|
+
## Quick Links
|
|
56
56
|
|
|
57
57
|
### For Users
|
|
58
58
|
|
|
@@ -76,7 +76,7 @@ Welcome to the VibeSort documentation! This directory contains comprehensive gui
|
|
|
76
76
|
|
|
77
77
|
---
|
|
78
78
|
|
|
79
|
-
##
|
|
79
|
+
## Reading Guide
|
|
80
80
|
|
|
81
81
|
### I want to use VibeSort in my project
|
|
82
82
|
|
|
@@ -107,25 +107,25 @@ Welcome to the VibeSort documentation! This directory contains comprehensive gui
|
|
|
107
107
|
|
|
108
108
|
---
|
|
109
109
|
|
|
110
|
-
##
|
|
110
|
+
## Architecture at a Glance
|
|
111
111
|
|
|
112
112
|
```
|
|
113
113
|
User Application
|
|
114
|
-
|
|
114
|
+
|
|
115
115
|
VibeSort::Client (validates input, handles errors)
|
|
116
|
-
|
|
116
|
+
|
|
117
117
|
VibeSort::Configuration (stores API key, settings)
|
|
118
|
-
|
|
118
|
+
|
|
119
119
|
VibeSort::Sorter (communicates with OpenAI)
|
|
120
|
-
|
|
120
|
+
|
|
121
121
|
OpenAI API (GPT model processes request)
|
|
122
|
-
|
|
122
|
+
|
|
123
123
|
Sorted Array (returned to user)
|
|
124
124
|
```
|
|
125
125
|
|
|
126
126
|
---
|
|
127
127
|
|
|
128
|
-
##
|
|
128
|
+
## Example Usage
|
|
129
129
|
|
|
130
130
|
```ruby
|
|
131
131
|
require 'vibe_sort'
|
|
@@ -148,7 +148,7 @@ For more examples, see the [README](../README.md) or [API Reference](api_referen
|
|
|
148
148
|
|
|
149
149
|
---
|
|
150
150
|
|
|
151
|
-
##
|
|
151
|
+
## Key Concepts
|
|
152
152
|
|
|
153
153
|
### Configuration
|
|
154
154
|
|
|
@@ -175,7 +175,7 @@ VibeSort never raises exceptions to user code. All errors are caught and returne
|
|
|
175
175
|
|
|
176
176
|
---
|
|
177
177
|
|
|
178
|
-
##
|
|
178
|
+
## Contributing
|
|
179
179
|
|
|
180
180
|
We welcome contributions! Please see:
|
|
181
181
|
|
|
@@ -185,22 +185,22 @@ We welcome contributions! Please see:
|
|
|
185
185
|
|
|
186
186
|
---
|
|
187
187
|
|
|
188
|
-
##
|
|
188
|
+
## Support
|
|
189
189
|
|
|
190
|
-
-
|
|
191
|
-
-
|
|
192
|
-
-
|
|
193
|
-
-
|
|
190
|
+
- [Report a Bug](https://github.com/chayut/vibe-sort/issues/new?labels=bug)
|
|
191
|
+
- [Request a Feature](https://github.com/chayut/vibe-sort/issues/new?labels=enhancement)
|
|
192
|
+
- [Read the Docs](README.md)
|
|
193
|
+
- [Discussions](https://github.com/chayut/vibe-sort/discussions)
|
|
194
194
|
|
|
195
195
|
---
|
|
196
196
|
|
|
197
|
-
##
|
|
197
|
+
## License
|
|
198
198
|
|
|
199
199
|
VibeSort is released under the [MIT License](../LICENSE.txt).
|
|
200
200
|
|
|
201
201
|
---
|
|
202
202
|
|
|
203
|
-
##
|
|
203
|
+
## Acknowledgments
|
|
204
204
|
|
|
205
205
|
- Built with [Faraday](https://lostisland.github.io/faraday/)
|
|
206
206
|
- Powered by [OpenAI](https://openai.com/)
|
|
@@ -208,4 +208,4 @@ VibeSort is released under the [MIT License](../LICENSE.txt).
|
|
|
208
208
|
|
|
209
209
|
---
|
|
210
210
|
|
|
211
|
-
**Happy Sorting!
|
|
211
|
+
**Happy Sorting! **
|
data/docs/api_reference.md
CHANGED
|
@@ -6,7 +6,7 @@ The main public interface for the VibeSort gem.
|
|
|
6
6
|
|
|
7
7
|
### Class Methods
|
|
8
8
|
|
|
9
|
-
#### `new(api_key:, temperature: 0.0, provider:
|
|
9
|
+
#### `new(api_key:, temperature: 0.0, provider: nil, model: nil, extra_params: {})`
|
|
10
10
|
|
|
11
11
|
Creates a new VibeSort client instance.
|
|
12
12
|
|
|
@@ -18,24 +18,27 @@ Creates a new VibeSort client instance.
|
|
|
18
18
|
- Lower values (0.0-0.3): More deterministic and consistent
|
|
19
19
|
- Higher values (0.7-2.0): More random and creative
|
|
20
20
|
- Ignored by providers whose current models do not accept it (Anthropic)
|
|
21
|
-
- `provider` (Symbol or String, optional): LLM provider
|
|
22
|
-
- Supported: `:openai`, `:anthropic`, `:gemini`, `:groq`, `:spacexai`
|
|
21
|
+
- `provider` (Symbol or String, optional): LLM provider
|
|
22
|
+
- Supported: `:openai`, `:anthropic`, `:gemini`, `:groq`, `:spacexai`, `:openrouter`
|
|
23
|
+
- When omitted, inferred from the `api_key` prefix via `VibeSort::KeyDetector`; unrecognized formats default to `:openai`
|
|
24
|
+
- When explicit and the key prefix suggests a different provider, a one-line warning is printed to stderr and the explicit provider is used
|
|
23
25
|
- `model` (String, optional): Model ID override (default: the provider's default model)
|
|
26
|
+
- `extra_params` (Hash, optional): Provider-native request parameters deep-merged into the payload last, so they can override anything the adapter builds (default: `{}`)
|
|
24
27
|
|
|
25
28
|
**Returns:** `VibeSort::Client` instance
|
|
26
29
|
|
|
27
30
|
**Raises:**
|
|
28
31
|
|
|
29
|
-
- `ArgumentError`: If api_key is nil or empty, or
|
|
32
|
+
- `ArgumentError`: If api_key is nil or empty, or an explicit provider is unknown
|
|
30
33
|
|
|
31
34
|
**Example:**
|
|
32
35
|
|
|
33
36
|
```ruby
|
|
34
|
-
#
|
|
35
|
-
client = VibeSort::Client.new(api_key: ENV['
|
|
37
|
+
# Provider inferred from the key prefix (sk-ant-... => :anthropic)
|
|
38
|
+
client = VibeSort::Client.new(api_key: ENV['ANTHROPIC_API_KEY'])
|
|
36
39
|
|
|
37
|
-
#
|
|
38
|
-
client = VibeSort::Client.new(provider: :
|
|
40
|
+
# Explicit provider
|
|
41
|
+
client = VibeSort::Client.new(provider: :openai, api_key: ENV['OPENAI_API_KEY'])
|
|
39
42
|
|
|
40
43
|
# Google Gemini with a custom model
|
|
41
44
|
client = VibeSort::Client.new(provider: :gemini, api_key: ENV['GEMINI_API_KEY'], model: 'gemini-2.5-pro')
|
|
@@ -45,6 +48,14 @@ client = VibeSort::Client.new(
|
|
|
45
48
|
api_key: ENV['OPENAI_API_KEY'],
|
|
46
49
|
temperature: 0.2
|
|
47
50
|
)
|
|
51
|
+
|
|
52
|
+
# With extra provider-native request parameters
|
|
53
|
+
client = VibeSort::Client.new(
|
|
54
|
+
provider: :openrouter,
|
|
55
|
+
api_key: ENV['OPENROUTER_API_KEY'],
|
|
56
|
+
model: 'meta-llama/llama-3.3-70b-instruct',
|
|
57
|
+
extra_params: { max_tokens: 200 }
|
|
58
|
+
)
|
|
48
59
|
```
|
|
49
60
|
|
|
50
61
|
### Instance Methods
|
|
@@ -110,7 +121,7 @@ Configuration object for VibeSort. Usually not used directly by end users.
|
|
|
110
121
|
|
|
111
122
|
### Class Methods
|
|
112
123
|
|
|
113
|
-
#### `new(api_key:, temperature: 0.0, provider:
|
|
124
|
+
#### `new(api_key:, temperature: 0.0, provider: nil, model: nil, extra_params: {})`
|
|
114
125
|
|
|
115
126
|
Creates a new configuration object.
|
|
116
127
|
|
|
@@ -118,8 +129,9 @@ Creates a new configuration object.
|
|
|
118
129
|
|
|
119
130
|
- `api_key` (String, required): API key for the selected provider
|
|
120
131
|
- `temperature` (Float, optional): Model temperature (default: 0.0)
|
|
121
|
-
- `provider` (Symbol or
|
|
132
|
+
- `provider` (Symbol, String, or nil, optional): `:openai`, `:anthropic`, `:gemini`, `:groq`, `:spacexai`, or `:openrouter`; nil (default) infers the provider from the api_key prefix, falling back to `:openai`
|
|
122
133
|
- `model` (String, optional): Model ID override (nil uses the provider's default)
|
|
134
|
+
- `extra_params` (Hash, optional): Provider-native request parameters deep-merged into the payload last (default: `{}`)
|
|
123
135
|
|
|
124
136
|
**Raises:**
|
|
125
137
|
|
|
@@ -161,6 +173,40 @@ Returns the configured model override, or nil when using the provider default (r
|
|
|
161
173
|
|
|
162
174
|
**Returns:** String or nil
|
|
163
175
|
|
|
176
|
+
#### `extra_params`
|
|
177
|
+
|
|
178
|
+
Returns the configured extra request parameters hash (read-only).
|
|
179
|
+
|
|
180
|
+
**Returns:** Hash
|
|
181
|
+
|
|
182
|
+
---
|
|
183
|
+
|
|
184
|
+
## VibeSort::KeyDetector
|
|
185
|
+
|
|
186
|
+
Best-effort provider inference from API key prefixes. Prefixes are conventions, not contracts — detection is only a soft default and a source of non-fatal warnings, never validation.
|
|
187
|
+
|
|
188
|
+
### Class Methods
|
|
189
|
+
|
|
190
|
+
#### `detect(api_key)`
|
|
191
|
+
|
|
192
|
+
**Parameters:**
|
|
193
|
+
|
|
194
|
+
- `api_key` (String, required): API key to inspect
|
|
195
|
+
|
|
196
|
+
**Returns:** Symbol (detected provider) or nil (unrecognized format)
|
|
197
|
+
|
|
198
|
+
**Prefix map (ordered — specific prefixes checked before `sk-`):**
|
|
199
|
+
|
|
200
|
+
| Prefix | Provider |
|
|
201
|
+
|---|---|
|
|
202
|
+
| `sk-ant-` | `:anthropic` |
|
|
203
|
+
| `sk-or-` | `:openrouter` |
|
|
204
|
+
| `sk-` | `:openai` |
|
|
205
|
+
| `gsk_` | `:groq` |
|
|
206
|
+
| `AIza` | `:gemini` (classic key) |
|
|
207
|
+
| `AQ.` | `:gemini` (newer key format) |
|
|
208
|
+
| `xai-` | `:spacexai` |
|
|
209
|
+
|
|
164
210
|
---
|
|
165
211
|
|
|
166
212
|
## VibeSort::Sorter
|
|
@@ -199,7 +245,7 @@ Performs the sorting operation via the configured provider's API.
|
|
|
199
245
|
|
|
200
246
|
Maps provider symbols to adapter classes.
|
|
201
247
|
|
|
202
|
-
**Value:** `{ openai:, anthropic:, gemini:, groq:, spacexai: }`
|
|
248
|
+
**Value:** `{ openai:, anthropic:, gemini:, groq:, spacexai:, openrouter: }`
|
|
203
249
|
|
|
204
250
|
---
|
|
205
251
|
|
|
@@ -214,11 +260,14 @@ Internal provider adapters. `Providers::Base` holds the shared prompt, Faraday H
|
|
|
214
260
|
| `Providers::Gemini` | `https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent` | `gemini-2.5-flash` |
|
|
215
261
|
| `Providers::Groq` | `https://api.groq.com/openai/v1/chat/completions` | `llama-3.3-70b-versatile` |
|
|
216
262
|
| `Providers::SpaceXAI` | `https://api.x.ai/v1/chat/completions` | `grok-4` |
|
|
263
|
+
| `Providers::OpenRouter` | `https://openrouter.ai/api/v1/chat/completions` | `openai/gpt-4o-mini` |
|
|
217
264
|
|
|
218
265
|
Notes:
|
|
219
266
|
|
|
220
267
|
- The Anthropic adapter uses structured outputs (JSON schema) and does not send `temperature` (rejected by current Claude models)
|
|
221
|
-
- The Groq and
|
|
268
|
+
- The Groq, SpaceXAI, and OpenRouter adapters subclass `Providers::OpenAI` (OpenAI-compatible APIs)
|
|
269
|
+
- The OpenRouter adapter sends OpenRouter's recommended attribution headers (`HTTP-Referer`, `X-Title`)
|
|
270
|
+
- `Providers::Base` deep-merges `config.extra_params` into every payload last, so it can override anything the adapter builds
|
|
222
271
|
|
|
223
272
|
---
|
|
224
273
|
|
|
@@ -307,7 +356,7 @@ All sorting operations return a consistent hash structure:
|
|
|
307
356
|
|
|
308
357
|
### API Errors
|
|
309
358
|
|
|
310
|
-
The `{Provider}` prefix names the configured provider (OpenAI, Anthropic, Gemini, Groq, or
|
|
359
|
+
The `{Provider}` prefix names the configured provider (OpenAI, Anthropic, Gemini, Groq, SpaceXAI, or OpenRouter):
|
|
311
360
|
|
|
312
361
|
| Error Message Pattern | Cause | Solution |
|
|
313
362
|
|----------------------|-------|----------|
|
|
@@ -455,11 +504,12 @@ results = threads.map(&:value)
|
|
|
455
504
|
Set the API key for whichever provider you use:
|
|
456
505
|
|
|
457
506
|
```bash
|
|
458
|
-
export OPENAI_API_KEY='sk-your-key-here' # provider: :openai (default)
|
|
507
|
+
export OPENAI_API_KEY='sk-your-key-here' # provider: :openai (fallback default)
|
|
459
508
|
export ANTHROPIC_API_KEY='your-key-here' # provider: :anthropic
|
|
460
509
|
export GEMINI_API_KEY='your-key-here' # provider: :gemini
|
|
461
510
|
export GROQ_API_KEY='your-key-here' # provider: :groq
|
|
462
511
|
export XAI_API_KEY='your-key-here' # provider: :spacexai
|
|
512
|
+
export OPENROUTER_API_KEY='your-key-here' # provider: :openrouter
|
|
463
513
|
```
|
|
464
514
|
|
|
465
515
|
**Usage:**
|
|
@@ -472,9 +522,9 @@ client = VibeSort::Client.new(api_key: ENV['OPENAI_API_KEY'])
|
|
|
472
522
|
|
|
473
523
|
## Version
|
|
474
524
|
|
|
475
|
-
Current version: `0.
|
|
525
|
+
Current version: `0.4.0`
|
|
476
526
|
|
|
477
527
|
```ruby
|
|
478
528
|
VibeSort::VERSION
|
|
479
|
-
# => "0.
|
|
529
|
+
# => "0.4.0"
|
|
480
530
|
```
|
data/docs/architecture.md
CHANGED
|
@@ -20,9 +20,10 @@ VibeSort follows a layered architecture with clear separation of concerns:
|
|
|
20
20
|
▼
|
|
21
21
|
┌─────────────────────────────────────────┐
|
|
22
22
|
│ VibeSort::Configuration │
|
|
23
|
-
│ - Provider selection
|
|
23
|
+
│ - Provider selection (explicit or │
|
|
24
|
+
│ inferred via KeyDetector) │
|
|
24
25
|
│ - API key management │
|
|
25
|
-
│ - Model override
|
|
26
|
+
│ - Model override, extra_params │
|
|
26
27
|
│ - Temperature settings │
|
|
27
28
|
└─────────────────┬───────────────────────┘
|
|
28
29
|
│
|
|
@@ -37,10 +38,11 @@ VibeSort follows a layered architecture with clear separation of concerns:
|
|
|
37
38
|
│ VibeSort::Providers::Base │
|
|
38
39
|
│ - Shared prompt │
|
|
39
40
|
│ - HTTP client (Faraday) │
|
|
41
|
+
│ - extra_params deep merge │
|
|
40
42
|
│ - Response parsing & validation │
|
|
41
43
|
├─────────────────────────────────────────┤
|
|
42
44
|
│ OpenAI │ Anthropic │ Gemini │ │
|
|
43
|
-
│ Groq │ SpaceXAI
|
|
45
|
+
│ Groq │ SpaceXAI │ OpenRouter │
|
|
44
46
|
│ (request/response wire formats) │
|
|
45
47
|
└─────────────────┬───────────────────────┘
|
|
46
48
|
│
|
|
@@ -50,6 +52,7 @@ VibeSort follows a layered architecture with clear separation of concerns:
|
|
|
50
52
|
│ api.openai.com │ api.anthropic.com │
|
|
51
53
|
│ generativelanguage.googleapis.com │
|
|
52
54
|
│ api.groq.com │ api.x.ai │
|
|
55
|
+
│ openrouter.ai │
|
|
53
56
|
└─────────────────────────────────────────┘
|
|
54
57
|
```
|
|
55
58
|
|
|
@@ -67,7 +70,7 @@ VibeSort follows a layered architecture with clear separation of concerns:
|
|
|
67
70
|
- Return standardized response hashes
|
|
68
71
|
|
|
69
72
|
**Key Methods**:
|
|
70
|
-
- `initialize(api_key:, temperature: 0.0, provider:
|
|
73
|
+
- `initialize(api_key:, temperature: 0.0, provider: nil, model: nil, extra_params: {})`: Creates client with configuration
|
|
71
74
|
- `sort(array)`: Sorts array and returns result hash
|
|
72
75
|
|
|
73
76
|
**Return Format**:
|
|
@@ -84,19 +87,26 @@ VibeSort follows a layered architecture with clear separation of concerns:
|
|
|
84
87
|
**Purpose**: Encapsulates configuration settings.
|
|
85
88
|
|
|
86
89
|
**Responsibilities**:
|
|
87
|
-
-
|
|
88
|
-
- Store model override and
|
|
90
|
+
- Resolve the provider: explicit argument wins (with a non-fatal stderr warning on a mismatched key prefix); otherwise inferred from the key prefix via `KeyDetector`, defaulting to `:openai`
|
|
91
|
+
- Store API key, model override, temperature, and extra_params
|
|
89
92
|
- Validate API key presence and provider name
|
|
90
93
|
|
|
91
94
|
**Key Methods**:
|
|
92
|
-
- `initialize(api_key:, temperature: 0.0, provider:
|
|
93
|
-
- Raises `ArgumentError` if API key is nil/empty or provider is unknown
|
|
95
|
+
- `initialize(api_key:, temperature: 0.0, provider: nil, model: nil, extra_params: {})`: Creates configuration
|
|
96
|
+
- Raises `ArgumentError` if API key is nil/empty or an explicit provider is unknown
|
|
94
97
|
|
|
95
98
|
**Attributes**:
|
|
96
99
|
- `api_key`: Provider API key (String)
|
|
97
|
-
- `provider`: `:openai`, `:anthropic`, `:gemini`, `:groq`, or `:
|
|
100
|
+
- `provider`: `:openai`, `:anthropic`, `:gemini`, `:groq`, `:spacexai`, or `:openrouter` (Symbol)
|
|
98
101
|
- `model`: Model ID override, or nil for the provider default (String or nil)
|
|
99
102
|
- `temperature`: Model temperature (Float, 0.0-2.0; not sent to Anthropic)
|
|
103
|
+
- `extra_params`: Provider-native request parameters deep-merged into the payload last (Hash)
|
|
104
|
+
|
|
105
|
+
### VibeSort::KeyDetector
|
|
106
|
+
|
|
107
|
+
**Purpose**: Best-effort provider inference from API key prefixes (`sk-ant-` → `:anthropic`, `sk-or-` → `:openrouter`, `gsk_` → `:groq`, `AIza`/`AQ.` → `:gemini`, `xai-` → `:spacexai`, `sk-` → `:openai`).
|
|
108
|
+
|
|
109
|
+
Prefixes are conventions, not contracts — providers ship multiple key formats concurrently and new formats appear without notice. Detection is therefore only a soft default and a source of non-fatal warnings, never validation: `detect` returns nil for unrecognized formats, and callers fall back to `:openai`.
|
|
100
110
|
|
|
101
111
|
### VibeSort::Sorter
|
|
102
112
|
|
|
@@ -113,17 +123,19 @@ VibeSort follows a layered architecture with clear separation of concerns:
|
|
|
113
123
|
**`Providers::Base` responsibilities**:
|
|
114
124
|
- Hold the shared sorting prompt
|
|
115
125
|
- Build the HTTP connection with Faraday
|
|
126
|
+
- Deep-merge `config.extra_params` into the adapter payload last (nested hashes merge recursively; arrays and scalars replace)
|
|
116
127
|
- Parse and validate JSON responses (shared across providers)
|
|
117
128
|
- Raise `ApiError` on failures
|
|
118
129
|
|
|
119
130
|
**Adapter hooks** (implemented per provider): `provider_name`, `endpoint`, `headers`, `build_payload`, `extract_content`
|
|
120
131
|
|
|
121
132
|
**Adapters and default models**:
|
|
122
|
-
- `Providers::OpenAI
|
|
123
|
-
- `Providers::Anthropic
|
|
124
|
-
- `Providers::Gemini
|
|
125
|
-
- `Providers::Groq
|
|
126
|
-
- `Providers::SpaceXAI
|
|
133
|
+
- `Providers::OpenAI`: Chat Completions, `gpt-4o-mini`
|
|
134
|
+
- `Providers::Anthropic`: Messages API with structured outputs (JSON schema), `claude-opus-4-8`
|
|
135
|
+
- `Providers::Gemini`: generateContent with JSON response mode, `gemini-2.5-flash`
|
|
136
|
+
- `Providers::Groq`: OpenAI-compatible (subclasses `Providers::OpenAI`), `llama-3.3-70b-versatile`
|
|
137
|
+
- `Providers::SpaceXAI`: OpenAI-compatible (subclasses `Providers::OpenAI`), `grok-4`
|
|
138
|
+
- `Providers::OpenRouter`: OpenAI-compatible (subclasses `Providers::OpenAI`), `openai/gpt-4o-mini`, adds OpenRouter's recommended attribution headers
|
|
127
139
|
|
|
128
140
|
### VibeSort::ApiError
|
|
129
141
|
|
|
@@ -244,7 +256,7 @@ VibeSort follows a layered architecture with clear separation of concerns:
|
|
|
244
256
|
## Dependencies
|
|
245
257
|
|
|
246
258
|
### Runtime
|
|
247
|
-
- **faraday** (~> 2.0): HTTP client library
|
|
259
|
+
- **faraday** (~> 2.0): HTTP client library, the only runtime dependency; all providers are called over plain HTTPS
|
|
248
260
|
|
|
249
261
|
### Development
|
|
250
262
|
- **rspec** (~> 3.0): Testing framework
|
data/docs/development.md
CHANGED
|
@@ -521,8 +521,8 @@ Document all public methods:
|
|
|
521
521
|
# @return [Hash] Result hash with :success, :sorted_array, :error keys
|
|
522
522
|
#
|
|
523
523
|
# @example
|
|
524
|
-
#
|
|
525
|
-
#
|
|
524
|
+
# client.sort([5, 2, 8])
|
|
525
|
+
# # => { success: true, sorted_array: [2, 5, 8] }
|
|
526
526
|
def sort(array)
|
|
527
527
|
# ...
|
|
528
528
|
end
|
|
@@ -563,4 +563,4 @@ Before submitting a PR:
|
|
|
563
563
|
- Read the documentation
|
|
564
564
|
- Ask in the community
|
|
565
565
|
|
|
566
|
-
Happy coding!
|
|
566
|
+
Happy coding!
|
|
@@ -0,0 +1,134 @@
|
|
|
1
|
+
# Free-Tier Model Benchmark — 2026-07-12
|
|
2
|
+
|
|
3
|
+
An empirical snapshot of how free LLM tiers handle vibe-sort's sorting contract, gathered
|
|
4
|
+
while designing v0.4.0 (OpenRouter provider, key-format detection, `extra_params` —
|
|
5
|
+
see [v0.4.0_PLAN.md](v0.4.0_PLAN.md)). All requests used free models / free-tier quota only.
|
|
6
|
+
|
|
7
|
+
**TL;DR:** Google's current Gemini flash models all sort correctly through the gem's exact
|
|
8
|
+
payload, with `gemini-3.1-flash-lite` the clear free-tier pick. OpenRouter's `:free` pool was
|
|
9
|
+
too congested on a Saturday afternoon (AEST) to be usable — 14 of 17 tested models failed with
|
|
10
|
+
rate limits or upstream errors, and one burned 10 minutes and 20k tokens to return nothing.
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
## Methodology
|
|
15
|
+
|
|
16
|
+
- **Task:** sort `[42, "banana", 7, "Apple", 3.14, "cherry", 1]`
|
|
17
|
+
→ expected `[1, 3.14, 7, 42, "Apple", "banana", "cherry"]`
|
|
18
|
+
- **Payloads:** byte-for-byte what the gem sends — `Providers::OpenAI#build_payload` for
|
|
19
|
+
OpenRouter (system prompt + `temperature: 0.0` + `response_format: {type: "json_object"}`),
|
|
20
|
+
`Providers::Gemini#build_payload` for Gemini (`systemInstruction` +
|
|
21
|
+
`generationConfig.responseMimeType: "application/json"`).
|
|
22
|
+
- **Success criteria:** HTTP 200, content parses as JSON with a `sorted_array` key
|
|
23
|
+
(the gem's contract), array exactly matches the expected sort.
|
|
24
|
+
- **Fallbacks tried on rejection:** OpenRouter — retry without `response_format`;
|
|
25
|
+
Gemini — retry with system prompt folded into the user turn and no JSON mime type.
|
|
26
|
+
- **Pacing/retries:** 3.5–8s between requests; 5xx retried up to 3×; 429 handling varied by
|
|
27
|
+
pass (see Lessons). Plain Ruby `net/http`, no SDKs.
|
|
28
|
+
- **Cost:** $0. OpenRouter restricted to `$0`-priced models; Gemini restricted to
|
|
29
|
+
free-tier models with non-zero quota.
|
|
30
|
+
|
|
31
|
+
---
|
|
32
|
+
|
|
33
|
+
## Gemini API (free tier)
|
|
34
|
+
|
|
35
|
+
7 models tested — every model whose free-tier quota is non-zero.
|
|
36
|
+
|
|
37
|
+
| Model | Outcome | Latency | Tokens in/out (+thinking) | Notes |
|
|
38
|
+
|---|---|---|---|---|
|
|
39
|
+
| gemini-3.1-flash-lite | ✅ correct | **2.08s** | 87/35 (+0) | Fastest; no thinking tokens; best quota (15 RPM / 500 RPD) |
|
|
40
|
+
| gemini-2.5-flash | ✅ correct | 2.66s | 87/49 (+247) | Gem's current `DEFAULT_MODEL` — still works |
|
|
41
|
+
| gemini-3.5-flash | ✅ correct | 5.98s | 87/50 (+418) | Most thinking tokens |
|
|
42
|
+
| gemini-3-flash-preview | ✅ correct | 24.14s | 87/49 (+321) | Correct but slow |
|
|
43
|
+
| gemma-4-26b-a4b-it | ❌ bad JSON | 28.6s | 87/49 (+318) | Ignored JSON mime type; replied with markdown reasoning; 1×500 on the way |
|
|
44
|
+
| gemma-4-31b-it | ❌ bad JSON | 9.8s | 87/30 (+256) | Same failure shape; also 1×500 |
|
|
45
|
+
| gemini-2.5-flash-lite | ❌ HTTP 404 | — | — | "No longer available to new users" (still listed by the models endpoint) |
|
|
46
|
+
|
|
47
|
+
### Observations
|
|
48
|
+
|
|
49
|
+
- **All four current Gemini-branded flash models pass the gem's contract as-is** — JSON mode
|
|
50
|
+
honored, sort correct, numbers-before-strings and case-sensitivity respected.
|
|
51
|
+
- **Gemma models are incompatible with the gem today:** they accept the request but ignore
|
|
52
|
+
`responseMimeType: "application/json"` and return reasoning-style markdown. They'd need
|
|
53
|
+
`extra_params` surgery plus response post-processing — not worth it.
|
|
54
|
+
- **The 500s are real:** 2 of 7 models needed a 5xx retry on their very first request.
|
|
55
|
+
Any Gemini integration should retry 5xx at least once.
|
|
56
|
+
- **429 retries are a trap on the free tier:** every retry counts against RPM *and* RPD, so a
|
|
57
|
+
retry loop converts one throttled request into three consumed ones. After the daily caps
|
|
58
|
+
(20 RPD on most flash models) tripped, throttling also turned intermittent — consecutive
|
|
59
|
+
requests to the same model went 429, 429, 200. Policy that survives: never auto-retry a
|
|
60
|
+
free-tier 429; skip and move on.
|
|
61
|
+
- An input-length scaling test (5/20/50/100 elements) was attempted but the RPD caps were
|
|
62
|
+
exhausted before it could run. Future work, pending quota reset — only
|
|
63
|
+
`gemini-3.1-flash-lite` (500 RPD) realistically supports this kind of testing for free.
|
|
64
|
+
|
|
65
|
+
---
|
|
66
|
+
|
|
67
|
+
## OpenRouter (free models)
|
|
68
|
+
|
|
69
|
+
26 models on openrouter.ai were $0-priced with text input (of 345 listed). 17 were tested
|
|
70
|
+
before the sweep was aborted; 9 were never reached. Tested on a Saturday afternoon AEST —
|
|
71
|
+
peak congestion for the shared free pool, which colours everything below.
|
|
72
|
+
|
|
73
|
+
| Model | Outcome | Notes |
|
|
74
|
+
|---|---|---|
|
|
75
|
+
| nvidia/nemotron-3-super-120b-a12b:free | ❌ empty content | **600s, 20,673 completion tokens, no answer** — presumably all reasoning |
|
|
76
|
+
| nvidia/nemotron-3.5-content-safety:free | ❌ bad JSON | Replied "User Safety: safe" — it's a classifier, not a chat model |
|
|
77
|
+
| nvidia/nemotron-nano-12b-v2-vl:free | ❌ upstream timeout | "Upstream idle timeout exceeded" after 121s |
|
|
78
|
+
| nvidia/nemotron-3-nano-30b-a3b:free | ❌ upstream error | NVIDIA "ResourceExhausted: worker limit reached" |
|
|
79
|
+
| nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free | ❌ upstream error | Same |
|
|
80
|
+
| nvidia/nemotron-3-ultra-550b-a55b:free | ❌ upstream error | Same |
|
|
81
|
+
| nvidia/nemotron-nano-9b-v2:free | ⏹ aborted | Stalled the sweep twice with a slow-drip connection (>20 min) |
|
|
82
|
+
| liquid/lfm-2.5-1.2b-instruct:free | ❌ HTTP 502 | Upstream down |
|
|
83
|
+
| liquid/lfm-2.5-1.2b-thinking:free | ❌ HTTP 502 | Upstream down |
|
|
84
|
+
| google/lyria-3-clip-preview | ❌ HTTP 400 | Music model; listed as $0 text-input but not a chat model |
|
|
85
|
+
| google/lyria-3-pro-preview | ❌ HTTP 400 | Same |
|
|
86
|
+
| cognitivecomputations/dolphin-mistral-24b-venice-edition:free | ❌ HTTP 429 | Free pool congested (retried in a 2nd pass ~1h later — still 429) |
|
|
87
|
+
| cohere/north-mini-code:free | ❌ HTTP 429 | Explicit "High demand" RPM-limit message |
|
|
88
|
+
| google/gemma-4-26b-a4b-it:free | ❌ HTTP 429 | Congested both passes |
|
|
89
|
+
| google/gemma-4-31b-it:free | ❌ HTTP 429 | Congested both passes |
|
|
90
|
+
| meta-llama/llama-3.2-3b-instruct:free | ❌ HTTP 429 | Congested both passes |
|
|
91
|
+
| meta-llama/llama-3.3-70b-instruct:free | ❌ HTTP 429 | Congested both passes |
|
|
92
|
+
| nousresearch/hermes-3-llama-3.1-405b:free | ❌ HTTP 429 | Congested both passes |
|
|
93
|
+
| openai/gpt-oss-120b:free, openai/gpt-oss-20b:free, openrouter/free, poolside/laguna-m.1:free, poolside/laguna-xs-2.1:free, qwen/qwen3-coder:free, qwen/qwen3-next-80b-a3b-instruct:free, tencent/hy3:free | ⏸ not tested | Sweep aborted before reaching them (blocked behind the stalling Nemotron) |
|
|
94
|
+
|
|
95
|
+
### Observations
|
|
96
|
+
|
|
97
|
+
- **Zero of 17 tested free models completed a correct sort** during this window. This is a
|
|
98
|
+
statement about the free pool's availability on a weekend afternoon, not about the models —
|
|
99
|
+
nearly every failure was infrastructure (429/5xx/upstream), not model output.
|
|
100
|
+
- **The transport layer of a vibe-sort OpenRouter adapter is validated**, though: auth,
|
|
101
|
+
routing, the OpenAI-compatible payload, and OpenRouter's error shape
|
|
102
|
+
(`error.message` — including passed-through upstream provider errors) all behaved exactly
|
|
103
|
+
as `Providers::Base#extract_error_message` expects.
|
|
104
|
+
- **Free-model quirks to document:** `$0` pricing does not imply chat-usable (Lyria = music,
|
|
105
|
+
content-safety = classifier); reasoning models can consume enormous token counts and still
|
|
106
|
+
return empty content; per-request wall time is unbounded in practice (slow-drip responses
|
|
107
|
+
blow through per-read timeouts), so an overall deadline is the only safe timeout.
|
|
108
|
+
- Free-model requests are also account-capped on OpenRouter (daily request limits tied to
|
|
109
|
+
credit balance) — fine for smoke tests, unusable for CI.
|
|
110
|
+
|
|
111
|
+
---
|
|
112
|
+
|
|
113
|
+
## Implications for vibe-sort v0.4.0
|
|
114
|
+
|
|
115
|
+
1. **OpenRouter adapter: ship it.** The mechanics are proven even though the free pool was
|
|
116
|
+
congested; with any paid model (or an off-peak free window) it should behave like the
|
|
117
|
+
other OpenAI-compatible adapters. Don't default to a `:free` model — congestion would make
|
|
118
|
+
the gem look broken.
|
|
119
|
+
2. **Gemini `DEFAULT_MODEL`:** `gemini-2.5-flash` still works, but Google has begun retiring
|
|
120
|
+
2.5-era models for new users (2.5-flash-lite already 404s). `gemini-3.1-flash-lite` is
|
|
121
|
+
faster here, spends no thinking tokens on this task, and has the widest free quota —
|
|
122
|
+
a strong candidate for the next default.
|
|
123
|
+
3. **`extra_params` earns its place:** overriding `response_format`, capping `max_tokens` on
|
|
124
|
+
reasoning models, and OpenRouter provider-routing preferences were all wanted during this
|
|
125
|
+
exercise.
|
|
126
|
+
4. **Never auto-retry 429s against free tiers** in examples/docs; retries multiply quota
|
|
127
|
+
consumption and prolong the lockout.
|
|
128
|
+
|
|
129
|
+
## Reproduction
|
|
130
|
+
|
|
131
|
+
Standalone Ruby scripts (plain `net/http`, no gem dependencies) sent the exact adapter
|
|
132
|
+
payloads; per-model raw results (latency, token usage, full error messages, returned arrays)
|
|
133
|
+
were captured as JSON. API keys were session-scoped test keys, not committed. Numbers above
|
|
134
|
+
are single-shot measurements on free tiers — directional, not statistical.
|