modellens-mcp 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- modellens_mcp-0.1.0/LICENSE +21 -0
- modellens_mcp-0.1.0/PKG-INFO +226 -0
- modellens_mcp-0.1.0/README.md +199 -0
- modellens_mcp-0.1.0/modellens.py +696 -0
- modellens_mcp-0.1.0/modellens_mcp.egg-info/PKG-INFO +226 -0
- modellens_mcp-0.1.0/modellens_mcp.egg-info/SOURCES.txt +9 -0
- modellens_mcp-0.1.0/modellens_mcp.egg-info/dependency_links.txt +1 -0
- modellens_mcp-0.1.0/modellens_mcp.egg-info/entry_points.txt +2 -0
- modellens_mcp-0.1.0/modellens_mcp.egg-info/top_level.txt +1 -0
- modellens_mcp-0.1.0/pyproject.toml +38 -0
- modellens_mcp-0.1.0/setup.cfg +4 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Greatbeing
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,226 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: modellens-mcp
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: See which of your models actually works. MCP server + CLI that discovers, probes, ranks and recommends models across OpenAI-compatible gateways.
|
|
5
|
+
Author: Greatbeing
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/Greatbeing/modellens
|
|
8
|
+
Project-URL: Repository, https://github.com/Greatbeing/modellens
|
|
9
|
+
Project-URL: Issues, https://github.com/Greatbeing/modellens/issues
|
|
10
|
+
Keywords: mcp,mcp-server,model-routing,llm,llm-observability,ai-agents
|
|
11
|
+
Classifier: Development Status :: 4 - Beta
|
|
12
|
+
Classifier: Environment :: Console
|
|
13
|
+
Classifier: Intended Audience :: Developers
|
|
14
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
15
|
+
Classifier: Operating System :: OS Independent
|
|
16
|
+
Classifier: Programming Language :: Python :: 3
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
22
|
+
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
23
|
+
Requires-Python: >=3.9
|
|
24
|
+
Description-Content-Type: text/markdown
|
|
25
|
+
License-File: LICENSE
|
|
26
|
+
Dynamic: license-file
|
|
27
|
+
|
|
28
|
+
# ModelLens
|
|
29
|
+
|
|
30
|
+
**See which of your models actually works.**
|
|
31
|
+
|
|
32
|
+
An MCP server that turns any OpenAI-compatible gateway into an observable,
|
|
33
|
+
rankable set of models. It discovers what a provider advertises, probes what
|
|
34
|
+
works *right now* — latency, success, cancellation, streaming, tool calling —
|
|
35
|
+
keeps the history, and answers **"which model should I use?"**
|
|
36
|
+
|
|
37
|
+
Zero third-party dependencies. Python 3.9+ standard library only.
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
|
|
41
|
+
## Why
|
|
42
|
+
|
|
43
|
+
If you run more than one model provider, you already have this problem:
|
|
44
|
+
|
|
45
|
+
- The gateway says it serves 89 models. How many actually answer?
|
|
46
|
+
- Which one is fast *today*? Yesterday's answer is already stale.
|
|
47
|
+
- Your router picks the first entry in a list — and the first entry is slow.
|
|
48
|
+
- Failures are invisible until a user complains.
|
|
49
|
+
|
|
50
|
+
ModelLens makes that visible. It is the difference between trusting a model
|
|
51
|
+
list and measuring it.
|
|
52
|
+
|
|
53
|
+
Real example from a live gateway (89 models, 16 probed):
|
|
54
|
+
|
|
55
|
+
```
|
|
56
|
+
magpie/kilo/nvidia/nemotron-3.5-super-vl:free 97.6 100.0% 0.0% 628ms healthy
|
|
57
|
+
magpie/auto/beam-free 96.6 100.0% 0.0% 909ms healthy
|
|
58
|
+
magpie/longcat/LongCat-2.5 88.4 100.0% 0.0% 3104ms mixed
|
|
59
|
+
magpie/group/daily 70.0 100.0% 0.0% 77345ms mixed
|
|
60
|
+
magpie/radeon-cloud/DeepSeek-V4.1-Flash 70.0 100.0% 0.0% 87388ms mixed
|
|
61
|
+
magpie/nvidia/deepseek-ai/deepseek-v4.1-flash 0.0 0.0% 100.0% - unreliable
|
|
62
|
+
magpie/modelscope/Qwen/Qwen3.8-27B 0.0 0.0% 0.0% - quota_blocked
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
The default routing group was putting an 87-second model first and a 0.9-second
|
|
66
|
+
model third. ModelLens found that in one pass.
|
|
67
|
+
|
|
68
|
+
---
|
|
69
|
+
|
|
70
|
+
## Install
|
|
71
|
+
|
|
72
|
+
```bash
|
|
73
|
+
# from PyPI
|
|
74
|
+
pip install modellens
|
|
75
|
+
modellens --config config.json --data history.json --once ranking
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
or straight from source — no dependencies, no install step:
|
|
79
|
+
|
|
80
|
+
```bash
|
|
81
|
+
curl -fsSL https://raw.githubusercontent.com/Greatbeing/modellens/main/modellens.py -o modellens.py
|
|
82
|
+
python3 modellens.py --config config.json --data history.json --once ranking
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
## Configure
|
|
86
|
+
|
|
87
|
+
```json
|
|
88
|
+
{
|
|
89
|
+
"endpoints": [
|
|
90
|
+
{"name": "magpie", "base_url": "http://127.0.0.1:3425/v1", "api_key": ""},
|
|
91
|
+
{"name": "openai", "base_url": "https://api.openai.com/v1", "api_key": "sk-..."},
|
|
92
|
+
{"name": "ollama", "base_url": "http://127.0.0.1:11434/v1"}
|
|
93
|
+
]
|
|
94
|
+
}
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
`api_key` may be empty for a localhost gateway. `models` optionally filters the
|
|
98
|
+
advertised list; `enabled: false` disables an endpoint without deleting it.
|
|
99
|
+
|
|
100
|
+
## Run as an MCP server
|
|
101
|
+
|
|
102
|
+
Point any MCP client at it:
|
|
103
|
+
|
|
104
|
+
```json
|
|
105
|
+
{
|
|
106
|
+
"mcpServers": {
|
|
107
|
+
"modellens": {
|
|
108
|
+
"command": "python3",
|
|
109
|
+
"args": ["/path/to/modellens.py",
|
|
110
|
+
"--config", "/path/to/config.json",
|
|
111
|
+
"--data", "/path/to/history.json"]
|
|
112
|
+
}
|
|
113
|
+
}
|
|
114
|
+
}
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
Works with Claude Desktop, Cursor, Windsurf, VS Code, and any other MCP client.
|
|
118
|
+
|
|
119
|
+
### DeepSeek Harness
|
|
120
|
+
|
|
121
|
+
Register it in the profile's `cordis.patch.yml` (note: a new entry needs
|
|
122
|
+
`- insert:`, not a bare `- id:` — a bare one means "modify an existing entry"):
|
|
123
|
+
|
|
124
|
+
```yaml
|
|
125
|
+
- insert:
|
|
126
|
+
- id: modellens
|
|
127
|
+
name: "@deepseek-ai/dsh-mcp-client"
|
|
128
|
+
config:
|
|
129
|
+
serverName: "modellens"
|
|
130
|
+
transport: "stdio"
|
|
131
|
+
command: "python3"
|
|
132
|
+
args:
|
|
133
|
+
- "/data/workspace/modellens/modellens.py"
|
|
134
|
+
- "--config"
|
|
135
|
+
- "/data/workspace/modellens/config.json"
|
|
136
|
+
- "--data"
|
|
137
|
+
- "/data/workspace/modellens/history.json"
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
Then restart `dsh web`. Verified: the server runs as a child process of
|
|
141
|
+
`dsh-web` (`pgrep -P $(systemctl --user show dsh-web -p MainPID)`).
|
|
142
|
+
|
|
143
|
+
### Tools
|
|
144
|
+
|
|
145
|
+
| Tool | What it does |
|
|
146
|
+
|---|---|
|
|
147
|
+
| `list_models` | What each endpoint advertises (optional substring filter) |
|
|
148
|
+
| `probe` | Real requests: latency, success, streaming, tool calls |
|
|
149
|
+
| `ranking` | Quality ranking from recorded history |
|
|
150
|
+
| `recommend` | Best model for a profile: `fast` \| `cheap` \| `capable` |
|
|
151
|
+
| `report` | The whole picture as markdown |
|
|
152
|
+
|
|
153
|
+
### Example
|
|
154
|
+
|
|
155
|
+
```
|
|
156
|
+
probe(limit=20, check_tools=true)
|
|
157
|
+
ranking(limit=10)
|
|
158
|
+
recommend(profile="fast")
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
---
|
|
162
|
+
|
|
163
|
+
## How it scores
|
|
164
|
+
|
|
165
|
+
Every probe is a real request, so the numbers are evidence, not estimates.
|
|
166
|
+
|
|
167
|
+
| Signal | Weight | Notes |
|
|
168
|
+
|---|---|---|
|
|
169
|
+
| Success rate | 50% | 0% success ⇒ score 0, regardless of anything else |
|
|
170
|
+
| Median latency | 30% | Linear to 0 at 8 s |
|
|
171
|
+
| Cancellation rate | 20% | Timeouts and client cancellations |
|
|
172
|
+
|
|
173
|
+
Verdicts: `healthy` (≥95% success, <3 s median) · `mixed` · `unreliable`
|
|
174
|
+
(≥30% cancelled) · `quota_blocked` (all 429) · `dead` (all 404) · `untested`.
|
|
175
|
+
|
|
176
|
+
History is kept per model (newest 200 probes) in `history.json`, so rankings
|
|
177
|
+
improve as you use it.
|
|
178
|
+
|
|
179
|
+
---
|
|
180
|
+
|
|
181
|
+
## CLI
|
|
182
|
+
|
|
183
|
+
The server doubles as a command-line tool:
|
|
184
|
+
|
|
185
|
+
```bash
|
|
186
|
+
python3 modellens.py --config config.json --data history.json --once probe --limit 20
|
|
187
|
+
python3 modellens.py --config config.json --data history.json --once ranking
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
---
|
|
191
|
+
|
|
192
|
+
## Use cases
|
|
193
|
+
|
|
194
|
+
- **Before you pick a model** — `recommend(profile="fast")`
|
|
195
|
+
- **When a provider goes quiet** — `ranking` shows it dropping before your users do
|
|
196
|
+
- **When your router misbehaves** — compare group members, find the slow one
|
|
197
|
+
- **Cost control** — spot the expensive model that is never chosen
|
|
198
|
+
- **Onboarding a new provider** — probe once, see what actually works
|
|
199
|
+
|
|
200
|
+
---
|
|
201
|
+
|
|
202
|
+
## Status and roadmap
|
|
203
|
+
|
|
204
|
+
Working today: discovery, probing, streaming/tool-call checks, scoring, ranking,
|
|
205
|
+
recommendation, markdown report, MCP stdio transport, CLI.
|
|
206
|
+
|
|
207
|
+
Roadmap: OpenAI-compatible `/v1/models` only for now; Anthropic and Gemini
|
|
208
|
+
transports, per-endpoint cost accounting, alerting webhooks, and a hosted
|
|
209
|
+
multi-team version.
|
|
210
|
+
|
|
211
|
+
---
|
|
212
|
+
|
|
213
|
+
## License
|
|
214
|
+
|
|
215
|
+
MIT. Use it, ship it, charge for your own hosted version.
|
|
216
|
+
|
|
217
|
+
---
|
|
218
|
+
|
|
219
|
+
## Honest notes
|
|
220
|
+
|
|
221
|
+
- **Probing costs tokens.** Each probe is a real completion. Keep `limit`
|
|
222
|
+
sensible and don't probe a hundred models every minute.
|
|
223
|
+
- **A single probe is weak evidence.** The score is meaningful after several
|
|
224
|
+
probes; the report says `n=1` so you can see the sample size.
|
|
225
|
+
- **Latency depends on you.** Your network, not just the provider, is in the
|
|
226
|
+
measurement. Run it from where your agents run.
|
|
@@ -0,0 +1,199 @@
|
|
|
1
|
+
# ModelLens
|
|
2
|
+
|
|
3
|
+
**See which of your models actually works.**
|
|
4
|
+
|
|
5
|
+
An MCP server that turns any OpenAI-compatible gateway into an observable,
|
|
6
|
+
rankable set of models. It discovers what a provider advertises, probes what
|
|
7
|
+
works *right now* — latency, success, cancellation, streaming, tool calling —
|
|
8
|
+
keeps the history, and answers **"which model should I use?"**
|
|
9
|
+
|
|
10
|
+
Zero third-party dependencies. Python 3.9+ standard library only.
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
## Why
|
|
15
|
+
|
|
16
|
+
If you run more than one model provider, you already have this problem:
|
|
17
|
+
|
|
18
|
+
- The gateway says it serves 89 models. How many actually answer?
|
|
19
|
+
- Which one is fast *today*? Yesterday's answer is already stale.
|
|
20
|
+
- Your router picks the first entry in a list — and the first entry is slow.
|
|
21
|
+
- Failures are invisible until a user complains.
|
|
22
|
+
|
|
23
|
+
ModelLens makes that visible. It is the difference between trusting a model
|
|
24
|
+
list and measuring it.
|
|
25
|
+
|
|
26
|
+
Real example from a live gateway (89 models, 16 probed):
|
|
27
|
+
|
|
28
|
+
```
|
|
29
|
+
magpie/kilo/nvidia/nemotron-3.5-super-vl:free 97.6 100.0% 0.0% 628ms healthy
|
|
30
|
+
magpie/auto/beam-free 96.6 100.0% 0.0% 909ms healthy
|
|
31
|
+
magpie/longcat/LongCat-2.5 88.4 100.0% 0.0% 3104ms mixed
|
|
32
|
+
magpie/group/daily 70.0 100.0% 0.0% 77345ms mixed
|
|
33
|
+
magpie/radeon-cloud/DeepSeek-V4.1-Flash 70.0 100.0% 0.0% 87388ms mixed
|
|
34
|
+
magpie/nvidia/deepseek-ai/deepseek-v4.1-flash 0.0 0.0% 100.0% - unreliable
|
|
35
|
+
magpie/modelscope/Qwen/Qwen3.8-27B 0.0 0.0% 0.0% - quota_blocked
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
The default routing group was putting an 87-second model first and a 0.9-second
|
|
39
|
+
model third. ModelLens found that in one pass.
|
|
40
|
+
|
|
41
|
+
---
|
|
42
|
+
|
|
43
|
+
## Install
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
# from PyPI
|
|
47
|
+
pip install modellens
|
|
48
|
+
modellens --config config.json --data history.json --once ranking
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
or straight from source — no dependencies, no install step:
|
|
52
|
+
|
|
53
|
+
```bash
|
|
54
|
+
curl -fsSL https://raw.githubusercontent.com/Greatbeing/modellens/main/modellens.py -o modellens.py
|
|
55
|
+
python3 modellens.py --config config.json --data history.json --once ranking
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
## Configure
|
|
59
|
+
|
|
60
|
+
```json
|
|
61
|
+
{
|
|
62
|
+
"endpoints": [
|
|
63
|
+
{"name": "magpie", "base_url": "http://127.0.0.1:3425/v1", "api_key": ""},
|
|
64
|
+
{"name": "openai", "base_url": "https://api.openai.com/v1", "api_key": "sk-..."},
|
|
65
|
+
{"name": "ollama", "base_url": "http://127.0.0.1:11434/v1"}
|
|
66
|
+
]
|
|
67
|
+
}
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
`api_key` may be empty for a localhost gateway. `models` optionally filters the
|
|
71
|
+
advertised list; `enabled: false` disables an endpoint without deleting it.
|
|
72
|
+
|
|
73
|
+
## Run as an MCP server
|
|
74
|
+
|
|
75
|
+
Point any MCP client at it:
|
|
76
|
+
|
|
77
|
+
```json
|
|
78
|
+
{
|
|
79
|
+
"mcpServers": {
|
|
80
|
+
"modellens": {
|
|
81
|
+
"command": "python3",
|
|
82
|
+
"args": ["/path/to/modellens.py",
|
|
83
|
+
"--config", "/path/to/config.json",
|
|
84
|
+
"--data", "/path/to/history.json"]
|
|
85
|
+
}
|
|
86
|
+
}
|
|
87
|
+
}
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
Works with Claude Desktop, Cursor, Windsurf, VS Code, and any other MCP client.
|
|
91
|
+
|
|
92
|
+
### DeepSeek Harness
|
|
93
|
+
|
|
94
|
+
Register it in the profile's `cordis.patch.yml` (note: a new entry needs
|
|
95
|
+
`- insert:`, not a bare `- id:` — a bare one means "modify an existing entry"):
|
|
96
|
+
|
|
97
|
+
```yaml
|
|
98
|
+
- insert:
|
|
99
|
+
- id: modellens
|
|
100
|
+
name: "@deepseek-ai/dsh-mcp-client"
|
|
101
|
+
config:
|
|
102
|
+
serverName: "modellens"
|
|
103
|
+
transport: "stdio"
|
|
104
|
+
command: "python3"
|
|
105
|
+
args:
|
|
106
|
+
- "/data/workspace/modellens/modellens.py"
|
|
107
|
+
- "--config"
|
|
108
|
+
- "/data/workspace/modellens/config.json"
|
|
109
|
+
- "--data"
|
|
110
|
+
- "/data/workspace/modellens/history.json"
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
Then restart `dsh web`. Verified: the server runs as a child process of
|
|
114
|
+
`dsh-web` (`pgrep -P $(systemctl --user show dsh-web -p MainPID)`).
|
|
115
|
+
|
|
116
|
+
### Tools
|
|
117
|
+
|
|
118
|
+
| Tool | What it does |
|
|
119
|
+
|---|---|
|
|
120
|
+
| `list_models` | What each endpoint advertises (optional substring filter) |
|
|
121
|
+
| `probe` | Real requests: latency, success, streaming, tool calls |
|
|
122
|
+
| `ranking` | Quality ranking from recorded history |
|
|
123
|
+
| `recommend` | Best model for a profile: `fast` \| `cheap` \| `capable` |
|
|
124
|
+
| `report` | The whole picture as markdown |
|
|
125
|
+
|
|
126
|
+
### Example
|
|
127
|
+
|
|
128
|
+
```
|
|
129
|
+
probe(limit=20, check_tools=true)
|
|
130
|
+
ranking(limit=10)
|
|
131
|
+
recommend(profile="fast")
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
---
|
|
135
|
+
|
|
136
|
+
## How it scores
|
|
137
|
+
|
|
138
|
+
Every probe is a real request, so the numbers are evidence, not estimates.
|
|
139
|
+
|
|
140
|
+
| Signal | Weight | Notes |
|
|
141
|
+
|---|---|---|
|
|
142
|
+
| Success rate | 50% | 0% success ⇒ score 0, regardless of anything else |
|
|
143
|
+
| Median latency | 30% | Linear to 0 at 8 s |
|
|
144
|
+
| Cancellation rate | 20% | Timeouts and client cancellations |
|
|
145
|
+
|
|
146
|
+
Verdicts: `healthy` (≥95% success, <3 s median) · `mixed` · `unreliable`
|
|
147
|
+
(≥30% cancelled) · `quota_blocked` (all 429) · `dead` (all 404) · `untested`.
|
|
148
|
+
|
|
149
|
+
History is kept per model (newest 200 probes) in `history.json`, so rankings
|
|
150
|
+
improve as you use it.
|
|
151
|
+
|
|
152
|
+
---
|
|
153
|
+
|
|
154
|
+
## CLI
|
|
155
|
+
|
|
156
|
+
The server doubles as a command-line tool:
|
|
157
|
+
|
|
158
|
+
```bash
|
|
159
|
+
python3 modellens.py --config config.json --data history.json --once probe --limit 20
|
|
160
|
+
python3 modellens.py --config config.json --data history.json --once ranking
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
---
|
|
164
|
+
|
|
165
|
+
## Use cases
|
|
166
|
+
|
|
167
|
+
- **Before you pick a model** — `recommend(profile="fast")`
|
|
168
|
+
- **When a provider goes quiet** — `ranking` shows it dropping before your users do
|
|
169
|
+
- **When your router misbehaves** — compare group members, find the slow one
|
|
170
|
+
- **Cost control** — spot the expensive model that is never chosen
|
|
171
|
+
- **Onboarding a new provider** — probe once, see what actually works
|
|
172
|
+
|
|
173
|
+
---
|
|
174
|
+
|
|
175
|
+
## Status and roadmap
|
|
176
|
+
|
|
177
|
+
Working today: discovery, probing, streaming/tool-call checks, scoring, ranking,
|
|
178
|
+
recommendation, markdown report, MCP stdio transport, CLI.
|
|
179
|
+
|
|
180
|
+
Roadmap: OpenAI-compatible `/v1/models` only for now; Anthropic and Gemini
|
|
181
|
+
transports, per-endpoint cost accounting, alerting webhooks, and a hosted
|
|
182
|
+
multi-team version.
|
|
183
|
+
|
|
184
|
+
---
|
|
185
|
+
|
|
186
|
+
## License
|
|
187
|
+
|
|
188
|
+
MIT. Use it, ship it, charge for your own hosted version.
|
|
189
|
+
|
|
190
|
+
---
|
|
191
|
+
|
|
192
|
+
## Honest notes
|
|
193
|
+
|
|
194
|
+
- **Probing costs tokens.** Each probe is a real completion. Keep `limit`
|
|
195
|
+
sensible and don't probe a hundred models every minute.
|
|
196
|
+
- **A single probe is weak evidence.** The score is meaningful after several
|
|
197
|
+
probes; the report says `n=1` so you can see the sample size.
|
|
198
|
+
- **Latency depends on you.** Your network, not just the provider, is in the
|
|
199
|
+
measurement. Run it from where your agents run.
|