modellens-mcp 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Greatbeing
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,226 @@
1
+ Metadata-Version: 2.4
2
+ Name: modellens-mcp
3
+ Version: 0.1.0
4
+ Summary: See which of your models actually works. MCP server + CLI that discovers, probes, ranks and recommends models across OpenAI-compatible gateways.
5
+ Author: Greatbeing
6
+ License: MIT
7
+ Project-URL: Homepage, https://github.com/Greatbeing/modellens
8
+ Project-URL: Repository, https://github.com/Greatbeing/modellens
9
+ Project-URL: Issues, https://github.com/Greatbeing/modellens/issues
10
+ Keywords: mcp,mcp-server,model-routing,llm,llm-observability,ai-agents
11
+ Classifier: Development Status :: 4 - Beta
12
+ Classifier: Environment :: Console
13
+ Classifier: Intended Audience :: Developers
14
+ Classifier: License :: OSI Approved :: MIT License
15
+ Classifier: Operating System :: OS Independent
16
+ Classifier: Programming Language :: Python :: 3
17
+ Classifier: Programming Language :: Python :: 3.9
18
+ Classifier: Programming Language :: Python :: 3.10
19
+ Classifier: Programming Language :: Python :: 3.11
20
+ Classifier: Programming Language :: Python :: 3.12
21
+ Classifier: Programming Language :: Python :: 3.13
22
+ Classifier: Topic :: Software Development :: Libraries :: Python Modules
23
+ Requires-Python: >=3.9
24
+ Description-Content-Type: text/markdown
25
+ License-File: LICENSE
26
+ Dynamic: license-file
27
+
28
+ # ModelLens
29
+
30
+ **See which of your models actually works.**
31
+
32
+ An MCP server that turns any OpenAI-compatible gateway into an observable,
33
+ rankable set of models. It discovers what a provider advertises, probes what
34
+ works *right now* — latency, success, cancellation, streaming, tool calling —
35
+ keeps the history, and answers **"which model should I use?"**
36
+
37
+ Zero third-party dependencies. Python 3.9+ standard library only.
38
+
39
+ ---
40
+
41
+ ## Why
42
+
43
+ If you run more than one model provider, you already have this problem:
44
+
45
+ - The gateway says it serves 89 models. How many actually answer?
46
+ - Which one is fast *today*? Yesterday's answer is already stale.
47
+ - Your router picks the first entry in a list — and the first entry is slow.
48
+ - Failures are invisible until a user complains.
49
+
50
+ ModelLens makes that visible. It is the difference between trusting a model
51
+ list and measuring it.
52
+
53
+ Real example from a live gateway (89 models, 16 probed):
54
+
55
+ ```
56
+ magpie/kilo/nvidia/nemotron-3.5-super-vl:free 97.6 100.0% 0.0% 628ms healthy
57
+ magpie/auto/beam-free 96.6 100.0% 0.0% 909ms healthy
58
+ magpie/longcat/LongCat-2.5 88.4 100.0% 0.0% 3104ms mixed
59
+ magpie/group/daily 70.0 100.0% 0.0% 77345ms mixed
60
+ magpie/radeon-cloud/DeepSeek-V4.1-Flash 70.0 100.0% 0.0% 87388ms mixed
61
+ magpie/nvidia/deepseek-ai/deepseek-v4.1-flash 0.0 0.0% 100.0% - unreliable
62
+ magpie/modelscope/Qwen/Qwen3.8-27B 0.0 0.0% 0.0% - quota_blocked
63
+ ```
64
+
65
+ The default routing group was putting an 87-second model first and a 0.9-second
66
+ model third. ModelLens found that in one pass.
67
+
68
+ ---
69
+
70
+ ## Install
71
+
72
+ ```bash
73
+ # from PyPI
74
+ pip install modellens
75
+ modellens --config config.json --data history.json --once ranking
76
+ ```
77
+
78
+ or straight from source — no dependencies, no install step:
79
+
80
+ ```bash
81
+ curl -fsSL https://raw.githubusercontent.com/Greatbeing/modellens/main/modellens.py -o modellens.py
82
+ python3 modellens.py --config config.json --data history.json --once ranking
83
+ ```
84
+
85
+ ## Configure
86
+
87
+ ```json
88
+ {
89
+ "endpoints": [
90
+ {"name": "magpie", "base_url": "http://127.0.0.1:3425/v1", "api_key": ""},
91
+ {"name": "openai", "base_url": "https://api.openai.com/v1", "api_key": "sk-..."},
92
+ {"name": "ollama", "base_url": "http://127.0.0.1:11434/v1"}
93
+ ]
94
+ }
95
+ ```
96
+
97
+ `api_key` may be empty for a localhost gateway. `models` optionally filters the
98
+ advertised list; `enabled: false` disables an endpoint without deleting it.
99
+
100
+ ## Run as an MCP server
101
+
102
+ Point any MCP client at it:
103
+
104
+ ```json
105
+ {
106
+ "mcpServers": {
107
+ "modellens": {
108
+ "command": "python3",
109
+ "args": ["/path/to/modellens.py",
110
+ "--config", "/path/to/config.json",
111
+ "--data", "/path/to/history.json"]
112
+ }
113
+ }
114
+ }
115
+ ```
116
+
117
+ Works with Claude Desktop, Cursor, Windsurf, VS Code, and any other MCP client.
118
+
119
+ ### DeepSeek Harness
120
+
121
+ Register it in the profile's `cordis.patch.yml` (note: a new entry needs
122
+ `- insert:`, not a bare `- id:` — a bare one means "modify an existing entry"):
123
+
124
+ ```yaml
125
+ - insert:
126
+ - id: modellens
127
+ name: "@deepseek-ai/dsh-mcp-client"
128
+ config:
129
+ serverName: "modellens"
130
+ transport: "stdio"
131
+ command: "python3"
132
+ args:
133
+ - "/data/workspace/modellens/modellens.py"
134
+ - "--config"
135
+ - "/data/workspace/modellens/config.json"
136
+ - "--data"
137
+ - "/data/workspace/modellens/history.json"
138
+ ```
139
+
140
+ Then restart `dsh web`. Verified: the server runs as a child process of
141
+ `dsh-web` (`pgrep -P $(systemctl --user show dsh-web -p MainPID)`).
142
+
143
+ ### Tools
144
+
145
+ | Tool | What it does |
146
+ |---|---|
147
+ | `list_models` | What each endpoint advertises (optional substring filter) |
148
+ | `probe` | Real requests: latency, success, streaming, tool calls |
149
+ | `ranking` | Quality ranking from recorded history |
150
+ | `recommend` | Best model for a profile: `fast` \| `cheap` \| `capable` |
151
+ | `report` | The whole picture as markdown |
152
+
153
+ ### Example
154
+
155
+ ```
156
+ probe(limit=20, check_tools=true)
157
+ ranking(limit=10)
158
+ recommend(profile="fast")
159
+ ```
160
+
161
+ ---
162
+
163
+ ## How it scores
164
+
165
+ Every probe is a real request, so the numbers are evidence, not estimates.
166
+
167
+ | Signal | Weight | Notes |
168
+ |---|---|---|
169
+ | Success rate | 50% | 0% success ⇒ score 0, regardless of anything else |
170
+ | Median latency | 30% | Linear to 0 at 8 s |
171
+ | Cancellation rate | 20% | Timeouts and client cancellations |
172
+
173
+ Verdicts: `healthy` (≥95% success, <3 s median) · `mixed` · `unreliable`
174
+ (≥30% cancelled) · `quota_blocked` (all 429) · `dead` (all 404) · `untested`.
175
+
176
+ History is kept per model (newest 200 probes) in `history.json`, so rankings
177
+ improve as you use it.
178
+
179
+ ---
180
+
181
+ ## CLI
182
+
183
+ The server doubles as a command-line tool:
184
+
185
+ ```bash
186
+ python3 modellens.py --config config.json --data history.json --once probe --limit 20
187
+ python3 modellens.py --config config.json --data history.json --once ranking
188
+ ```
189
+
190
+ ---
191
+
192
+ ## Use cases
193
+
194
+ - **Before you pick a model** — `recommend(profile="fast")`
195
+ - **When a provider goes quiet** — `ranking` shows it dropping before your users do
196
+ - **When your router misbehaves** — compare group members, find the slow one
197
+ - **Cost control** — spot the expensive model that is never chosen
198
+ - **Onboarding a new provider** — probe once, see what actually works
199
+
200
+ ---
201
+
202
+ ## Status and roadmap
203
+
204
+ Working today: discovery, probing, streaming/tool-call checks, scoring, ranking,
205
+ recommendation, markdown report, MCP stdio transport, CLI.
206
+
207
+ Roadmap: OpenAI-compatible `/v1/models` only for now; Anthropic and Gemini
208
+ transports, per-endpoint cost accounting, alerting webhooks, and a hosted
209
+ multi-team version.
210
+
211
+ ---
212
+
213
+ ## License
214
+
215
+ MIT. Use it, ship it, charge for your own hosted version.
216
+
217
+ ---
218
+
219
+ ## Honest notes
220
+
221
+ - **Probing costs tokens.** Each probe is a real completion. Keep `limit`
222
+ sensible and don't probe a hundred models every minute.
223
+ - **A single probe is weak evidence.** The score is meaningful after several
224
+ probes; the report says `n=1` so you can see the sample size.
225
+ - **Latency depends on you.** Your network, not just the provider, is in the
226
+ measurement. Run it from where your agents run.
@@ -0,0 +1,199 @@
1
+ # ModelLens
2
+
3
+ **See which of your models actually works.**
4
+
5
+ An MCP server that turns any OpenAI-compatible gateway into an observable,
6
+ rankable set of models. It discovers what a provider advertises, probes what
7
+ works *right now* — latency, success, cancellation, streaming, tool calling —
8
+ keeps the history, and answers **"which model should I use?"**
9
+
10
+ Zero third-party dependencies. Python 3.9+ standard library only.
11
+
12
+ ---
13
+
14
+ ## Why
15
+
16
+ If you run more than one model provider, you already have this problem:
17
+
18
+ - The gateway says it serves 89 models. How many actually answer?
19
+ - Which one is fast *today*? Yesterday's answer is already stale.
20
+ - Your router picks the first entry in a list — and the first entry is slow.
21
+ - Failures are invisible until a user complains.
22
+
23
+ ModelLens makes that visible. It is the difference between trusting a model
24
+ list and measuring it.
25
+
26
+ Real example from a live gateway (89 models, 16 probed):
27
+
28
+ ```
29
+ magpie/kilo/nvidia/nemotron-3.5-super-vl:free 97.6 100.0% 0.0% 628ms healthy
30
+ magpie/auto/beam-free 96.6 100.0% 0.0% 909ms healthy
31
+ magpie/longcat/LongCat-2.5 88.4 100.0% 0.0% 3104ms mixed
32
+ magpie/group/daily 70.0 100.0% 0.0% 77345ms mixed
33
+ magpie/radeon-cloud/DeepSeek-V4.1-Flash 70.0 100.0% 0.0% 87388ms mixed
34
+ magpie/nvidia/deepseek-ai/deepseek-v4.1-flash 0.0 0.0% 100.0% - unreliable
35
+ magpie/modelscope/Qwen/Qwen3.8-27B 0.0 0.0% 0.0% - quota_blocked
36
+ ```
37
+
38
+ The default routing group was putting an 87-second model first and a 0.9-second
39
+ model third. ModelLens found that in one pass.
40
+
41
+ ---
42
+
43
+ ## Install
44
+
45
+ ```bash
46
+ # from PyPI
47
+ pip install modellens
48
+ modellens --config config.json --data history.json --once ranking
49
+ ```
50
+
51
+ or straight from source — no dependencies, no install step:
52
+
53
+ ```bash
54
+ curl -fsSL https://raw.githubusercontent.com/Greatbeing/modellens/main/modellens.py -o modellens.py
55
+ python3 modellens.py --config config.json --data history.json --once ranking
56
+ ```
57
+
58
+ ## Configure
59
+
60
+ ```json
61
+ {
62
+ "endpoints": [
63
+ {"name": "magpie", "base_url": "http://127.0.0.1:3425/v1", "api_key": ""},
64
+ {"name": "openai", "base_url": "https://api.openai.com/v1", "api_key": "sk-..."},
65
+ {"name": "ollama", "base_url": "http://127.0.0.1:11434/v1"}
66
+ ]
67
+ }
68
+ ```
69
+
70
+ `api_key` may be empty for a localhost gateway. `models` optionally filters the
71
+ advertised list; `enabled: false` disables an endpoint without deleting it.
72
+
73
+ ## Run as an MCP server
74
+
75
+ Point any MCP client at it:
76
+
77
+ ```json
78
+ {
79
+ "mcpServers": {
80
+ "modellens": {
81
+ "command": "python3",
82
+ "args": ["/path/to/modellens.py",
83
+ "--config", "/path/to/config.json",
84
+ "--data", "/path/to/history.json"]
85
+ }
86
+ }
87
+ }
88
+ ```
89
+
90
+ Works with Claude Desktop, Cursor, Windsurf, VS Code, and any other MCP client.
91
+
92
+ ### DeepSeek Harness
93
+
94
+ Register it in the profile's `cordis.patch.yml` (note: a new entry needs
95
+ `- insert:`, not a bare `- id:` — a bare one means "modify an existing entry"):
96
+
97
+ ```yaml
98
+ - insert:
99
+ - id: modellens
100
+ name: "@deepseek-ai/dsh-mcp-client"
101
+ config:
102
+ serverName: "modellens"
103
+ transport: "stdio"
104
+ command: "python3"
105
+ args:
106
+ - "/data/workspace/modellens/modellens.py"
107
+ - "--config"
108
+ - "/data/workspace/modellens/config.json"
109
+ - "--data"
110
+ - "/data/workspace/modellens/history.json"
111
+ ```
112
+
113
+ Then restart `dsh web`. Verified: the server runs as a child process of
114
+ `dsh-web` (`pgrep -P $(systemctl --user show dsh-web -p MainPID)`).
115
+
116
+ ### Tools
117
+
118
+ | Tool | What it does |
119
+ |---|---|
120
+ | `list_models` | What each endpoint advertises (optional substring filter) |
121
+ | `probe` | Real requests: latency, success, streaming, tool calls |
122
+ | `ranking` | Quality ranking from recorded history |
123
+ | `recommend` | Best model for a profile: `fast` \| `cheap` \| `capable` |
124
+ | `report` | The whole picture as markdown |
125
+
126
+ ### Example
127
+
128
+ ```
129
+ probe(limit=20, check_tools=true)
130
+ ranking(limit=10)
131
+ recommend(profile="fast")
132
+ ```
133
+
134
+ ---
135
+
136
+ ## How it scores
137
+
138
+ Every probe is a real request, so the numbers are evidence, not estimates.
139
+
140
+ | Signal | Weight | Notes |
141
+ |---|---|---|
142
+ | Success rate | 50% | 0% success ⇒ score 0, regardless of anything else |
143
+ | Median latency | 30% | Linear to 0 at 8 s |
144
+ | Cancellation rate | 20% | Timeouts and client cancellations |
145
+
146
+ Verdicts: `healthy` (≥95% success, <3 s median) · `mixed` · `unreliable`
147
+ (≥30% cancelled) · `quota_blocked` (all 429) · `dead` (all 404) · `untested`.
148
+
149
+ History is kept per model (newest 200 probes) in `history.json`, so rankings
150
+ improve as you use it.
151
+
152
+ ---
153
+
154
+ ## CLI
155
+
156
+ The server doubles as a command-line tool:
157
+
158
+ ```bash
159
+ python3 modellens.py --config config.json --data history.json --once probe --limit 20
160
+ python3 modellens.py --config config.json --data history.json --once ranking
161
+ ```
162
+
163
+ ---
164
+
165
+ ## Use cases
166
+
167
+ - **Before you pick a model** — `recommend(profile="fast")`
168
+ - **When a provider goes quiet** — `ranking` shows it dropping before your users do
169
+ - **When your router misbehaves** — compare group members, find the slow one
170
+ - **Cost control** — spot the expensive model that is never chosen
171
+ - **Onboarding a new provider** — probe once, see what actually works
172
+
173
+ ---
174
+
175
+ ## Status and roadmap
176
+
177
+ Working today: discovery, probing, streaming/tool-call checks, scoring, ranking,
178
+ recommendation, markdown report, MCP stdio transport, CLI.
179
+
180
+ Roadmap: OpenAI-compatible `/v1/models` only for now; Anthropic and Gemini
181
+ transports, per-endpoint cost accounting, alerting webhooks, and a hosted
182
+ multi-team version.
183
+
184
+ ---
185
+
186
+ ## License
187
+
188
+ MIT. Use it, ship it, charge for your own hosted version.
189
+
190
+ ---
191
+
192
+ ## Honest notes
193
+
194
+ - **Probing costs tokens.** Each probe is a real completion. Keep `limit`
195
+ sensible and don't probe a hundred models every minute.
196
+ - **A single probe is weak evidence.** The score is meaningful after several
197
+ probes; the report says `n=1` so you can see the sample size.
198
+ - **Latency depends on you.** Your network, not just the provider, is in the
199
+ measurement. Run it from where your agents run.