poolprobe 1.0.0__tar.gz → 1.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {poolprobe-1.0.0 → poolprobe-1.2.0}/PKG-INFO +109 -16
- {poolprobe-1.0.0 → poolprobe-1.2.0}/README.md +108 -15
- {poolprobe-1.0.0 → poolprobe-1.2.0}/poolprobe/__init__.py +1 -1
- {poolprobe-1.0.0 → poolprobe-1.2.0}/poolprobe/adapter.py +142 -28
- {poolprobe-1.0.0 → poolprobe-1.2.0}/poolprobe/bench.py +11 -3
- {poolprobe-1.0.0 → poolprobe-1.2.0}/poolprobe/cli.py +117 -8
- poolprobe-1.2.0/poolprobe/compatcheck.py +270 -0
- {poolprobe-1.0.0 → poolprobe-1.2.0}/poolprobe/probe.py +6 -1
- {poolprobe-1.0.0 → poolprobe-1.2.0}/poolprobe/state.py +27 -1
- {poolprobe-1.0.0 → poolprobe-1.2.0}/poolprobe/taxonomy.py +117 -8
- {poolprobe-1.0.0 → poolprobe-1.2.0}/poolprobe.egg-info/PKG-INFO +109 -16
- {poolprobe-1.0.0 → poolprobe-1.2.0}/poolprobe.egg-info/SOURCES.txt +1 -0
- {poolprobe-1.0.0 → poolprobe-1.2.0}/pyproject.toml +1 -1
- {poolprobe-1.0.0 → poolprobe-1.2.0}/LICENSE +0 -0
- {poolprobe-1.0.0 → poolprobe-1.2.0}/poolprobe/__main__.py +0 -0
- {poolprobe-1.0.0 → poolprobe-1.2.0}/poolprobe/chains.py +0 -0
- {poolprobe-1.0.0 → poolprobe-1.2.0}/poolprobe/report.py +0 -0
- {poolprobe-1.0.0 → poolprobe-1.2.0}/poolprobe/safewrite.py +0 -0
- {poolprobe-1.0.0 → poolprobe-1.2.0}/poolprobe.egg-info/dependency_links.txt +0 -0
- {poolprobe-1.0.0 → poolprobe-1.2.0}/poolprobe.egg-info/entry_points.txt +0 -0
- {poolprobe-1.0.0 → poolprobe-1.2.0}/poolprobe.egg-info/requires.txt +0 -0
- {poolprobe-1.0.0 → poolprobe-1.2.0}/poolprobe.egg-info/top_level.txt +0 -0
- {poolprobe-1.0.0 → poolprobe-1.2.0}/setup.cfg +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: poolprobe
|
|
3
|
-
Version: 1.
|
|
3
|
+
Version: 1.2.0
|
|
4
4
|
Summary: A doctor for OpenAI-compatible LLM gateways: probe the pool, classify every failure, and only then talk about pruning.
|
|
5
5
|
License: MIT License
|
|
6
6
|
|
|
@@ -48,20 +48,31 @@ Provides-Extra: cliproxy
|
|
|
48
48
|
Requires-Dist: PyYAML>=6.0; extra == "cliproxy"
|
|
49
49
|
Dynamic: license-file
|
|
50
50
|
|
|
51
|
-
|
|
51
|
+

|
|
52
52
|
|
|
53
53
|
# poolprobe
|
|
54
54
|
|
|
55
|
-
**
|
|
56
|
-
|
|
55
|
+
**Probe first. Classify before pruning.**
|
|
56
|
+
|
|
57
|
+
Inspect OpenAI-compatible model pools, classify failures and produce terminal, JSON or HTML reports. Keep transient and credential failures separate from permanent-model prune candidates.
|
|
58
|
+
|
|
59
|
+
[Install](#install) · [Quickstart](#quickstart) · [Commands](#commands) · [Prune guards](#the-five-prune-guards) · [Contributing](CONTRIBUTING.md)
|
|
60
|
+
|
|
61
|
+
**Scope:** Live sweeps and benchmarks send model requests and may incur provider charges. Pruning requires all five guards; config writes require `--apply`. Offline selfcheck does not establish live gateway availability.
|
|
62
|
+
|
|
63
|
+
**Checks:** run `python -m poolprobe selfcheck` before pushing.
|
|
64
|
+
This is the repository's offline contract; it needs no live service or credentials.
|
|
65
|
+
|
|
66
|
+
<details>
|
|
67
|
+
<summary>Package and license badges</summary>
|
|
57
68
|
|
|
58
69
|
[](https://www.python.org/downloads/)
|
|
59
70
|
[](#install)
|
|
60
71
|
[](LICENSE)
|
|
61
72
|
|
|
62
|
-
</
|
|
73
|
+
</details>
|
|
63
74
|
|
|
64
|
-
|
|
75
|
+
## Why it exists
|
|
65
76
|
|
|
66
77
|
A pool of routed models rots quietly. An upstream drops a model, a key hits an
|
|
67
78
|
account tier, a reasoning model starts spending its whole token budget on
|
|
@@ -96,23 +107,98 @@ Everything in the tool obeys these. They are the reason it exists.
|
|
|
96
107
|
|
|
97
108
|
## Install
|
|
98
109
|
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
110
|
+
Use **Python 3.10+**. For a first run from a checkout, use a virtual environment
|
|
111
|
+
(the activation command below is for a POSIX shell):
|
|
112
|
+
|
|
113
|
+
```sh
|
|
114
|
+
git clone https://gitlab.com/ameobius-ai/poolprobe.git
|
|
115
|
+
cd poolprobe
|
|
116
|
+
python3 -m venv .venv
|
|
117
|
+
. .venv/bin/activate
|
|
118
|
+
python -m pip install .
|
|
119
|
+
poolprobe --version
|
|
102
120
|
```
|
|
103
121
|
|
|
104
|
-
|
|
122
|
+
Expected version output: `poolprobe 1.2.0` with exit code `0`. Installation may
|
|
123
|
+
fetch the build tooling declared in [pyproject.toml](pyproject.toml); installing
|
|
124
|
+
is not the offline check below. No gateway credentials or real config are needed.
|
|
125
|
+
|
|
126
|
+
When a release is available on PyPI, the package-install alternatives are:
|
|
105
127
|
|
|
106
128
|
```bash
|
|
107
|
-
|
|
108
|
-
|
|
129
|
+
pip install poolprobe # stdlib engine, generic gateway lane
|
|
130
|
+
pip install 'poolprobe[cliproxy]' # + PyYAML, to read a cliproxy-style config
|
|
109
131
|
```
|
|
110
132
|
|
|
111
133
|
The engine has **zero runtime dependencies**. PyYAML is optional and only needed
|
|
112
134
|
to attribute models to providers and to plan config edits from a gateway YAML.
|
|
135
|
+
For that existing adapter, install `python -m pip install '.[cliproxy]'` from the
|
|
136
|
+
checkout; it may fetch PyYAML. It is not needed for the offline first run.
|
|
137
|
+
|
|
138
|
+
### CLIProxyAPI config attribution
|
|
139
|
+
|
|
140
|
+
The adapter reads the legacy `openai-compatibility` / `claude-api-key` lists
|
|
141
|
+
and CLIProxyAPI v8 `api-keys.openai-compatibility` / `api-keys.claude` groups.
|
|
142
|
+
For each family, a present v8 list replaces the legacy one, including `[]`.
|
|
143
|
+
Claude key entries inherit group fields and may override models/prefixes.
|
|
144
|
+
The read contract follows [CLIProxyAPI v8.0.45](https://github.com/router-for-me/CLIProxyAPI/tree/ad8bdd3f20bef44f0684b8a6096a216b2c76040d);
|
|
145
|
+
it is not a claim about every installed gateway version or provider family.
|
|
146
|
+
|
|
147
|
+
Configured model IDs use the normalized one-segment `prefix`: normally both
|
|
148
|
+
`alias` and `prefix/alias`, or only the prefixed ID with `force-model-prefix`.
|
|
149
|
+
As upstream specifies, the unprefixed ID is also retained when it equals the
|
|
150
|
+
prefix. A present `routing.force-model-prefix` overrides the legacy flag,
|
|
151
|
+
including `false`. The catalog remains the union of declared IDs and `/models`;
|
|
152
|
+
a declaration is not proof that the running gateway publishes or can serve it.
|
|
153
|
+
|
|
154
|
+
**Reading is not editing permission.** v8/mixed, prefixed, empty or ambiguous
|
|
155
|
+
layouts are read-only; the existing text-edit path remains limited to recognized,
|
|
156
|
+
unambiguous, unprefixed legacy declarations. Layouts with no supported section,
|
|
157
|
+
or malformed supported sections, raise an error with `--adapter cliproxy`;
|
|
158
|
+
`auto` falls back to the generic read-only adapter. Other v8 provider families
|
|
159
|
+
are not attributed here. No live
|
|
160
|
+
gateway, removal plan or config-write safety is established by the offline checks.
|
|
113
161
|
|
|
114
162
|
## Quickstart
|
|
115
163
|
|
|
164
|
+
### First run without a gateway
|
|
165
|
+
|
|
166
|
+
With the virtual environment above active, run the installed entry point:
|
|
167
|
+
|
|
168
|
+
```sh
|
|
169
|
+
poolprobe selfcheck
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
Expected output:
|
|
173
|
+
|
|
174
|
+
```text
|
|
175
|
+
poolprobe selfcheck ok
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
A pass exits `0`; a failed selfcheck exits `1`. Keep Python assertions enabled
|
|
179
|
+
(do not use `python -O`). The equivalent checkout command is
|
|
180
|
+
`python -m poolprobe selfcheck`.
|
|
181
|
+
|
|
182
|
+
This runs built-in synthetic checks for chain rules, prune guards, argv
|
|
183
|
+
redaction, terminal/JSON/HTML rendering and atomic writes. It needs no API key,
|
|
184
|
+
gateway, user config or saved state. The atomic-write check creates and cleans
|
|
185
|
+
its own temporary directory; no report or config file should be kept from this
|
|
186
|
+
run. A pass does **not** establish live gateway/model availability or validate a
|
|
187
|
+
real YAML config. No `cliproxy` extra is needed for these checks.
|
|
188
|
+
|
|
189
|
+
Next safe step: `poolprobe --help` lists commands without sending requests.
|
|
190
|
+
See [the existing contribution checks](CONTRIBUTING.md#run-the-existing-contract)
|
|
191
|
+
for the module/installed-entry and optional-adapter check sequence.
|
|
192
|
+
|
|
193
|
+
### Live gateway examples — not offline checks
|
|
194
|
+
|
|
195
|
+
The examples below are a separate operational path, not the first-run check.
|
|
196
|
+
`ping` contacts the gateway even though it sends no chat request; sweeps,
|
|
197
|
+
benchmarks and **pruning without `--apply`** can send model requests and incur
|
|
198
|
+
charges. `--apply` guards config changes, not network access. Review
|
|
199
|
+
[commands](#commands) and [all five prune guards](#the-five-prune-guards) before
|
|
200
|
+
using an authorized gateway. Do not run these examples as an offline test.
|
|
201
|
+
|
|
116
202
|
```bash
|
|
117
203
|
# 1. Is the gateway even up, and what does it publish? (no chat request sent)
|
|
118
204
|
poolprobe ping -u http://127.0.0.1:8317/v1 -k "$KEY"
|
|
@@ -123,7 +209,7 @@ poolprobe sweep -u http://127.0.0.1:8317/v1 -k "$KEY" --html out.html
|
|
|
123
209
|
# 3. Give it memory: fail streaks across runs are what make `prune` safe
|
|
124
210
|
poolprobe sweep -u ... -k "$KEY" --state ~/.cache/poolprobe.json
|
|
125
211
|
|
|
126
|
-
# 4.
|
|
212
|
+
# 4. Probe and plan candidates; only config writes require --apply
|
|
127
213
|
poolprobe prune -u ... -k "$KEY" --state ~/.cache/poolprobe.json \
|
|
128
214
|
--adapter cliproxy --config ~/.config/gateway.yaml
|
|
129
215
|
|
|
@@ -138,7 +224,7 @@ every argv echo — reports are meant to be pasted into issues.
|
|
|
138
224
|
### What a sweep looks like
|
|
139
225
|
|
|
140
226
|
```
|
|
141
|
-
poolprobe 1.
|
|
227
|
+
poolprobe 1.2.0 · sweep
|
|
142
228
|
http://127.0.0.1:8317/v1 · generic adapter (read-only)
|
|
143
229
|
|
|
144
230
|
5 models · 4 alive · 0 starved · 1 dead · 0 prunable · median 5.43s · run #1
|
|
@@ -200,7 +286,10 @@ a tool like this, so the guards live in one function, checked in one place:
|
|
|
200
286
|
|
|
201
287
|
1. **dead verdict** — `EMPTY`, `ERR`, or `TIMEOUT` (`OK`/`EMPTY-LENGTH` never qualify);
|
|
202
288
|
2. **kind `PERMANENT`** — the upstream does not know this model;
|
|
203
|
-
3. **fail streak** — dead for at least `--streak-min` consecutive *runs* (needs `--state`)
|
|
289
|
+
3. **fail streak** — dead for at least `--streak-min` consecutive *runs* (needs `--state`)
|
|
290
|
+
**and** for at least `--min-dead-for` seconds of wall time (default 1800), so a
|
|
291
|
+
burst of runs during one outage counts as one observation (`--min-interval SECONDS`
|
|
292
|
+
is the matching probe budget: models probed that recently are skipped, not re-asked);
|
|
204
293
|
4. **not protected** — absent from `--protect` (a namespaced alias's base name matches too);
|
|
205
294
|
5. **not a generation modality** — image/video/tts/music models legitimately fail a chat probe.
|
|
206
295
|
|
|
@@ -221,12 +310,16 @@ ran out — alive) · `ERR` (non-2xx) · `TIMEOUT`.
|
|
|
221
310
|
|------|---------|:---------:|
|
|
222
311
|
| `ALIVE` | Answered. | no |
|
|
223
312
|
| `ALIVE-STARVED` | Answered `EMPTY-LENGTH` — raise `--max-tokens`. | no |
|
|
224
|
-
| `PERMANENT` |
|
|
313
|
+
| `PERMANENT` | Explicit model-error evidence, such as structured `model_not_found`; a bare/route `404` is not enough. | **yes** |
|
|
225
314
|
| `TRANSIENT` | Timeout, `429`, `5xx` — weather. Burst-confirm before believing it. | no |
|
|
226
315
|
| `CREDENTIAL` | `401`/`403`, "invalid api key" — rotate the key, never the model. | no |
|
|
227
316
|
| `MODALITY` | Model lives on another endpoint (`/v1/images/…`), not chat. | no |
|
|
228
317
|
| `UNKNOWN` | None of the above; reported, never acted on. | no |
|
|
229
318
|
|
|
319
|
+
Structured model-error codes are classified before the display note is truncated
|
|
320
|
+
to 160 characters. An ambiguous `404` stays `UNKNOWN`; a permanent classification
|
|
321
|
+
alone never bypasses the five prune guards or makes an adapter editable.
|
|
322
|
+
|
|
230
323
|
Three renderers, one report: the terminal table (ANSI, honours `NO_COLOR` /
|
|
231
324
|
`FORCE_COLOR` / `TERM=dumb` / isatty), a machine-readable `--json`, and a
|
|
232
325
|
self-contained `--html` you can attach to a post. They cannot disagree about
|
|
@@ -1,17 +1,28 @@
|
|
|
1
|
-
|
|
1
|
+

|
|
2
2
|
|
|
3
3
|
# poolprobe
|
|
4
4
|
|
|
5
|
-
**
|
|
6
|
-
|
|
5
|
+
**Probe first. Classify before pruning.**
|
|
6
|
+
|
|
7
|
+
Inspect OpenAI-compatible model pools, classify failures and produce terminal, JSON or HTML reports. Keep transient and credential failures separate from permanent-model prune candidates.
|
|
8
|
+
|
|
9
|
+
[Install](#install) · [Quickstart](#quickstart) · [Commands](#commands) · [Prune guards](#the-five-prune-guards) · [Contributing](CONTRIBUTING.md)
|
|
10
|
+
|
|
11
|
+
**Scope:** Live sweeps and benchmarks send model requests and may incur provider charges. Pruning requires all five guards; config writes require `--apply`. Offline selfcheck does not establish live gateway availability.
|
|
12
|
+
|
|
13
|
+
**Checks:** run `python -m poolprobe selfcheck` before pushing.
|
|
14
|
+
This is the repository's offline contract; it needs no live service or credentials.
|
|
15
|
+
|
|
16
|
+
<details>
|
|
17
|
+
<summary>Package and license badges</summary>
|
|
7
18
|
|
|
8
19
|
[](https://www.python.org/downloads/)
|
|
9
20
|
[](#install)
|
|
10
21
|
[](LICENSE)
|
|
11
22
|
|
|
12
|
-
</
|
|
23
|
+
</details>
|
|
13
24
|
|
|
14
|
-
|
|
25
|
+
## Why it exists
|
|
15
26
|
|
|
16
27
|
A pool of routed models rots quietly. An upstream drops a model, a key hits an
|
|
17
28
|
account tier, a reasoning model starts spending its whole token budget on
|
|
@@ -46,23 +57,98 @@ Everything in the tool obeys these. They are the reason it exists.
|
|
|
46
57
|
|
|
47
58
|
## Install
|
|
48
59
|
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
60
|
+
Use **Python 3.10+**. For a first run from a checkout, use a virtual environment
|
|
61
|
+
(the activation command below is for a POSIX shell):
|
|
62
|
+
|
|
63
|
+
```sh
|
|
64
|
+
git clone https://gitlab.com/ameobius-ai/poolprobe.git
|
|
65
|
+
cd poolprobe
|
|
66
|
+
python3 -m venv .venv
|
|
67
|
+
. .venv/bin/activate
|
|
68
|
+
python -m pip install .
|
|
69
|
+
poolprobe --version
|
|
52
70
|
```
|
|
53
71
|
|
|
54
|
-
|
|
72
|
+
Expected version output: `poolprobe 1.2.0` with exit code `0`. Installation may
|
|
73
|
+
fetch the build tooling declared in [pyproject.toml](pyproject.toml); installing
|
|
74
|
+
is not the offline check below. No gateway credentials or real config are needed.
|
|
75
|
+
|
|
76
|
+
When a release is available on PyPI, the package-install alternatives are:
|
|
55
77
|
|
|
56
78
|
```bash
|
|
57
|
-
|
|
58
|
-
|
|
79
|
+
pip install poolprobe # stdlib engine, generic gateway lane
|
|
80
|
+
pip install 'poolprobe[cliproxy]' # + PyYAML, to read a cliproxy-style config
|
|
59
81
|
```
|
|
60
82
|
|
|
61
83
|
The engine has **zero runtime dependencies**. PyYAML is optional and only needed
|
|
62
84
|
to attribute models to providers and to plan config edits from a gateway YAML.
|
|
85
|
+
For that existing adapter, install `python -m pip install '.[cliproxy]'` from the
|
|
86
|
+
checkout; it may fetch PyYAML. It is not needed for the offline first run.
|
|
87
|
+
|
|
88
|
+
### CLIProxyAPI config attribution
|
|
89
|
+
|
|
90
|
+
The adapter reads the legacy `openai-compatibility` / `claude-api-key` lists
|
|
91
|
+
and CLIProxyAPI v8 `api-keys.openai-compatibility` / `api-keys.claude` groups.
|
|
92
|
+
For each family, a present v8 list replaces the legacy one, including `[]`.
|
|
93
|
+
Claude key entries inherit group fields and may override models/prefixes.
|
|
94
|
+
The read contract follows [CLIProxyAPI v8.0.45](https://github.com/router-for-me/CLIProxyAPI/tree/ad8bdd3f20bef44f0684b8a6096a216b2c76040d);
|
|
95
|
+
it is not a claim about every installed gateway version or provider family.
|
|
96
|
+
|
|
97
|
+
Configured model IDs use the normalized one-segment `prefix`: normally both
|
|
98
|
+
`alias` and `prefix/alias`, or only the prefixed ID with `force-model-prefix`.
|
|
99
|
+
As upstream specifies, the unprefixed ID is also retained when it equals the
|
|
100
|
+
prefix. A present `routing.force-model-prefix` overrides the legacy flag,
|
|
101
|
+
including `false`. The catalog remains the union of declared IDs and `/models`;
|
|
102
|
+
a declaration is not proof that the running gateway publishes or can serve it.
|
|
103
|
+
|
|
104
|
+
**Reading is not editing permission.** v8/mixed, prefixed, empty or ambiguous
|
|
105
|
+
layouts are read-only; the existing text-edit path remains limited to recognized,
|
|
106
|
+
unambiguous, unprefixed legacy declarations. Layouts with no supported section,
|
|
107
|
+
or malformed supported sections, raise an error with `--adapter cliproxy`;
|
|
108
|
+
`auto` falls back to the generic read-only adapter. Other v8 provider families
|
|
109
|
+
are not attributed here. No live
|
|
110
|
+
gateway, removal plan or config-write safety is established by the offline checks.
|
|
63
111
|
|
|
64
112
|
## Quickstart
|
|
65
113
|
|
|
114
|
+
### First run without a gateway
|
|
115
|
+
|
|
116
|
+
With the virtual environment above active, run the installed entry point:
|
|
117
|
+
|
|
118
|
+
```sh
|
|
119
|
+
poolprobe selfcheck
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
Expected output:
|
|
123
|
+
|
|
124
|
+
```text
|
|
125
|
+
poolprobe selfcheck ok
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
A pass exits `0`; a failed selfcheck exits `1`. Keep Python assertions enabled
|
|
129
|
+
(do not use `python -O`). The equivalent checkout command is
|
|
130
|
+
`python -m poolprobe selfcheck`.
|
|
131
|
+
|
|
132
|
+
This runs built-in synthetic checks for chain rules, prune guards, argv
|
|
133
|
+
redaction, terminal/JSON/HTML rendering and atomic writes. It needs no API key,
|
|
134
|
+
gateway, user config or saved state. The atomic-write check creates and cleans
|
|
135
|
+
its own temporary directory; no report or config file should be kept from this
|
|
136
|
+
run. A pass does **not** establish live gateway/model availability or validate a
|
|
137
|
+
real YAML config. No `cliproxy` extra is needed for these checks.
|
|
138
|
+
|
|
139
|
+
Next safe step: `poolprobe --help` lists commands without sending requests.
|
|
140
|
+
See [the existing contribution checks](CONTRIBUTING.md#run-the-existing-contract)
|
|
141
|
+
for the module/installed-entry and optional-adapter check sequence.
|
|
142
|
+
|
|
143
|
+
### Live gateway examples — not offline checks
|
|
144
|
+
|
|
145
|
+
The examples below are a separate operational path, not the first-run check.
|
|
146
|
+
`ping` contacts the gateway even though it sends no chat request; sweeps,
|
|
147
|
+
benchmarks and **pruning without `--apply`** can send model requests and incur
|
|
148
|
+
charges. `--apply` guards config changes, not network access. Review
|
|
149
|
+
[commands](#commands) and [all five prune guards](#the-five-prune-guards) before
|
|
150
|
+
using an authorized gateway. Do not run these examples as an offline test.
|
|
151
|
+
|
|
66
152
|
```bash
|
|
67
153
|
# 1. Is the gateway even up, and what does it publish? (no chat request sent)
|
|
68
154
|
poolprobe ping -u http://127.0.0.1:8317/v1 -k "$KEY"
|
|
@@ -73,7 +159,7 @@ poolprobe sweep -u http://127.0.0.1:8317/v1 -k "$KEY" --html out.html
|
|
|
73
159
|
# 3. Give it memory: fail streaks across runs are what make `prune` safe
|
|
74
160
|
poolprobe sweep -u ... -k "$KEY" --state ~/.cache/poolprobe.json
|
|
75
161
|
|
|
76
|
-
# 4.
|
|
162
|
+
# 4. Probe and plan candidates; only config writes require --apply
|
|
77
163
|
poolprobe prune -u ... -k "$KEY" --state ~/.cache/poolprobe.json \
|
|
78
164
|
--adapter cliproxy --config ~/.config/gateway.yaml
|
|
79
165
|
|
|
@@ -88,7 +174,7 @@ every argv echo — reports are meant to be pasted into issues.
|
|
|
88
174
|
### What a sweep looks like
|
|
89
175
|
|
|
90
176
|
```
|
|
91
|
-
poolprobe 1.
|
|
177
|
+
poolprobe 1.2.0 · sweep
|
|
92
178
|
http://127.0.0.1:8317/v1 · generic adapter (read-only)
|
|
93
179
|
|
|
94
180
|
5 models · 4 alive · 0 starved · 1 dead · 0 prunable · median 5.43s · run #1
|
|
@@ -150,7 +236,10 @@ a tool like this, so the guards live in one function, checked in one place:
|
|
|
150
236
|
|
|
151
237
|
1. **dead verdict** — `EMPTY`, `ERR`, or `TIMEOUT` (`OK`/`EMPTY-LENGTH` never qualify);
|
|
152
238
|
2. **kind `PERMANENT`** — the upstream does not know this model;
|
|
153
|
-
3. **fail streak** — dead for at least `--streak-min` consecutive *runs* (needs `--state`)
|
|
239
|
+
3. **fail streak** — dead for at least `--streak-min` consecutive *runs* (needs `--state`)
|
|
240
|
+
**and** for at least `--min-dead-for` seconds of wall time (default 1800), so a
|
|
241
|
+
burst of runs during one outage counts as one observation (`--min-interval SECONDS`
|
|
242
|
+
is the matching probe budget: models probed that recently are skipped, not re-asked);
|
|
154
243
|
4. **not protected** — absent from `--protect` (a namespaced alias's base name matches too);
|
|
155
244
|
5. **not a generation modality** — image/video/tts/music models legitimately fail a chat probe.
|
|
156
245
|
|
|
@@ -171,12 +260,16 @@ ran out — alive) · `ERR` (non-2xx) · `TIMEOUT`.
|
|
|
171
260
|
|------|---------|:---------:|
|
|
172
261
|
| `ALIVE` | Answered. | no |
|
|
173
262
|
| `ALIVE-STARVED` | Answered `EMPTY-LENGTH` — raise `--max-tokens`. | no |
|
|
174
|
-
| `PERMANENT` |
|
|
263
|
+
| `PERMANENT` | Explicit model-error evidence, such as structured `model_not_found`; a bare/route `404` is not enough. | **yes** |
|
|
175
264
|
| `TRANSIENT` | Timeout, `429`, `5xx` — weather. Burst-confirm before believing it. | no |
|
|
176
265
|
| `CREDENTIAL` | `401`/`403`, "invalid api key" — rotate the key, never the model. | no |
|
|
177
266
|
| `MODALITY` | Model lives on another endpoint (`/v1/images/…`), not chat. | no |
|
|
178
267
|
| `UNKNOWN` | None of the above; reported, never acted on. | no |
|
|
179
268
|
|
|
269
|
+
Structured model-error codes are classified before the display note is truncated
|
|
270
|
+
to 160 characters. An ambiguous `404` stays `UNKNOWN`; a permanent classification
|
|
271
|
+
alone never bypasses the five prune guards or makes an adapter editable.
|
|
272
|
+
|
|
180
273
|
Three renderers, one report: the terminal table (ANSI, honours `NO_COLOR` /
|
|
181
274
|
`FORCE_COLOR` / `TERM=dumb` / isatty), a machine-readable `--json`, and a
|
|
182
275
|
self-contained `--html` you can attach to a post. They cannot disagree about
|
|
@@ -37,7 +37,7 @@ CLI use::
|
|
|
37
37
|
|
|
38
38
|
from __future__ import annotations
|
|
39
39
|
|
|
40
|
-
__version__ = "1.
|
|
40
|
+
__version__ = "1.2.0"
|
|
41
41
|
|
|
42
42
|
# Submodules first: probe, bench and adapter all resolve ``taxonomy`` through
|
|
43
43
|
# the package attribute at import time, and a partially initialised package must
|
|
@@ -8,7 +8,8 @@ Two adapters, one interface:
|
|
|
8
8
|
against OpenRouter, one-api, LiteLLM, vLLM, cliproxy or a bare vendor endpoint.
|
|
9
9
|
* :class:`CliproxyAdapter` — adds attribution (which upstream provider serves
|
|
10
10
|
this alias, is it disabled, what is the real model name) by reading the
|
|
11
|
-
gateway's own YAML config,
|
|
11
|
+
gateway's own YAML config. Only unambiguous, unprefixed legacy layouts can
|
|
12
|
+
use the existing removal planner; v8 attribution stays read-only.
|
|
12
13
|
|
|
13
14
|
The split between *reading* and *removing* is deliberate:
|
|
14
15
|
|
|
@@ -31,9 +32,16 @@ from pathlib import Path
|
|
|
31
32
|
|
|
32
33
|
from .probe import Gateway, GatewayError, list_models
|
|
33
34
|
|
|
34
|
-
#:
|
|
35
|
-
#: Kept as a tuple so a different gateway layout is a one-line extension.
|
|
35
|
+
#: Supported legacy sections and their v8 ``api-keys`` family names.
|
|
36
36
|
PROVIDER_SECTIONS: tuple[str, ...] = ("openai-compatibility", "claude-api-key")
|
|
37
|
+
V8_FAMILIES: dict[str, str] = {
|
|
38
|
+
"openai-compatibility": "openai-compatibility",
|
|
39
|
+
"claude-api-key": "claude",
|
|
40
|
+
}
|
|
41
|
+
_SHARED_KEY_FIELDS = frozenset({
|
|
42
|
+
"priority", "prefix", "proxy-url", "headers", "models", "excluded-models",
|
|
43
|
+
"disable-cooling", "request-retry", "request-scoped-errors",
|
|
44
|
+
})
|
|
37
45
|
|
|
38
46
|
#: A list item is a model entry only when its nearest lower-indented ancestor
|
|
39
47
|
#: line is one of these keys. Provider blocks also carry ``name:`` and would
|
|
@@ -72,6 +80,66 @@ def effective_alias(name: str, alias: str = "") -> str:
|
|
|
72
80
|
return (alias or name or "").strip()
|
|
73
81
|
|
|
74
82
|
|
|
83
|
+
def _v8_blocks(groups: object, family: str) -> list[dict]:
|
|
84
|
+
"""Expand the two supported v8 families, without interpreting credentials."""
|
|
85
|
+
if not isinstance(groups, list):
|
|
86
|
+
raise ConfigError(f"api-keys.{family} must be a list")
|
|
87
|
+
blocks = []
|
|
88
|
+
for group in groups:
|
|
89
|
+
if not isinstance(group, dict) or not isinstance(group.get("keys"), list):
|
|
90
|
+
raise ConfigError(f"api-keys.{family} groups need a keys list")
|
|
91
|
+
keys = group["keys"]
|
|
92
|
+
if any(not isinstance(key, dict) for key in keys):
|
|
93
|
+
raise ConfigError(f"api-keys.{family} keys must be mappings")
|
|
94
|
+
if family == "openai-compatibility":
|
|
95
|
+
# Upstream keeps the group's models/prefix on the provider block;
|
|
96
|
+
# individual keys do not override them for this family.
|
|
97
|
+
blocks.append({key: value for key, value in group.items() if key != "keys"})
|
|
98
|
+
continue
|
|
99
|
+
allowed = _SHARED_KEY_FIELDS | {"name", "base-url", "keys"}
|
|
100
|
+
if set(group) - allowed:
|
|
101
|
+
raise ConfigError(f"unsupported api-keys.{family} group fields")
|
|
102
|
+
for key in keys:
|
|
103
|
+
if "base-url" in key:
|
|
104
|
+
raise ConfigError(f"api-keys.{family}: base-url belongs to the group")
|
|
105
|
+
block = {
|
|
106
|
+
name: value for name, value in group.items()
|
|
107
|
+
if name == "base-url" or name in _SHARED_KEY_FIELDS
|
|
108
|
+
}
|
|
109
|
+
block.update({
|
|
110
|
+
name: value for name, value in key.items()
|
|
111
|
+
if name not in {"auth_index", "auth-index"} and value is not None
|
|
112
|
+
})
|
|
113
|
+
blocks.append(block)
|
|
114
|
+
return blocks
|
|
115
|
+
|
|
116
|
+
|
|
117
|
+
def _has_models(value: object) -> bool:
|
|
118
|
+
"""An unhandled models subtree must not be eligible for text surgery."""
|
|
119
|
+
pending = [value]
|
|
120
|
+
seen = set()
|
|
121
|
+
while pending:
|
|
122
|
+
item = pending.pop()
|
|
123
|
+
if id(item) in seen:
|
|
124
|
+
continue # YAML aliases may share or even cycle containers.
|
|
125
|
+
seen.add(id(item))
|
|
126
|
+
if isinstance(item, dict):
|
|
127
|
+
if "models" in item:
|
|
128
|
+
return True
|
|
129
|
+
pending.extend(item.values())
|
|
130
|
+
elif isinstance(item, list):
|
|
131
|
+
pending.extend(item)
|
|
132
|
+
return False
|
|
133
|
+
|
|
134
|
+
|
|
135
|
+
def _prefix(value: object) -> str:
|
|
136
|
+
if not isinstance(value, str):
|
|
137
|
+
raise ConfigError("model prefix must be a string")
|
|
138
|
+
prefix = value.strip().strip("/")
|
|
139
|
+
# CLIProxyAPI normalizes an empty or multi-segment prefix away.
|
|
140
|
+
return "" if not prefix or "/" in prefix else prefix.strip()
|
|
141
|
+
|
|
142
|
+
|
|
75
143
|
# --- attribution ------------------------------------------------------------
|
|
76
144
|
|
|
77
145
|
|
|
@@ -180,7 +248,7 @@ class CliproxyAdapter(Adapter):
|
|
|
180
248
|
"""
|
|
181
249
|
|
|
182
250
|
name = "cliproxy"
|
|
183
|
-
removable =
|
|
251
|
+
removable = False
|
|
184
252
|
|
|
185
253
|
def __init__(self, gateway: Gateway, config_path: str | Path):
|
|
186
254
|
self.gateway = gateway
|
|
@@ -192,51 +260,97 @@ class CliproxyAdapter(Adapter):
|
|
|
192
260
|
# -- reading ---------------------------------------------------------
|
|
193
261
|
|
|
194
262
|
def _parse(self) -> dict[str, Attribution]:
|
|
263
|
+
self.removable = False
|
|
195
264
|
doc = _load_yaml(self.config_path)
|
|
196
265
|
if not isinstance(doc, dict):
|
|
197
266
|
raise ConfigError(f"{self.config_path}: expected a mapping at top level")
|
|
267
|
+
if "config-version" in doc and (
|
|
268
|
+
type(doc["config-version"]) is not int or doc["config-version"] != 8
|
|
269
|
+
):
|
|
270
|
+
raise ConfigError("unsupported config-version (expected 8)")
|
|
271
|
+
families = doc.get("api-keys")
|
|
272
|
+
if families is not None and not isinstance(families, (dict, list)):
|
|
273
|
+
raise ConfigError("api-keys must be a client-key list or a v8 provider mapping")
|
|
274
|
+
v8 = families if isinstance(families, dict) else {}
|
|
275
|
+
force = doc.get("force-model-prefix", False)
|
|
276
|
+
routing = doc.get("routing", {})
|
|
277
|
+
if not isinstance(routing, dict):
|
|
278
|
+
raise ConfigError("routing must be a mapping")
|
|
279
|
+
if "force-model-prefix" in routing:
|
|
280
|
+
force = routing["force-model-prefix"]
|
|
281
|
+
if not isinstance(force, bool):
|
|
282
|
+
raise ConfigError("force-model-prefix must be a boolean")
|
|
198
283
|
|
|
199
284
|
attr: dict[str, Attribution] = {}
|
|
200
285
|
providers = 0
|
|
286
|
+
recognized = False
|
|
287
|
+
editable = (
|
|
288
|
+
"config-version" not in doc and not isinstance(families, dict) and not force
|
|
289
|
+
)
|
|
290
|
+
for key, value in doc.items():
|
|
291
|
+
if key not in PROVIDER_SECTIONS and _has_models(value):
|
|
292
|
+
editable = False
|
|
201
293
|
for section in PROVIDER_SECTIONS:
|
|
202
|
-
|
|
203
|
-
if
|
|
294
|
+
family = V8_FAMILIES[section]
|
|
295
|
+
if family in v8:
|
|
296
|
+
# Presence, not truthiness: [] replaces a legacy section too.
|
|
297
|
+
blocks = _v8_blocks(v8[family], family)
|
|
298
|
+
editable = False
|
|
299
|
+
elif section in doc:
|
|
300
|
+
blocks = doc[section]
|
|
301
|
+
else:
|
|
204
302
|
continue
|
|
303
|
+
recognized = True
|
|
304
|
+
if not isinstance(blocks, list):
|
|
305
|
+
raise ConfigError(f"{section} must be a list")
|
|
205
306
|
for block in blocks:
|
|
206
307
|
if not isinstance(block, dict):
|
|
207
|
-
|
|
308
|
+
raise ConfigError(f"{section} entries must be mappings")
|
|
309
|
+
if any(_has_models(value) for key, value in block.items() if key != "models"):
|
|
310
|
+
editable = False
|
|
208
311
|
providers += 1
|
|
209
312
|
provider = str(block.get("name") or section)
|
|
210
313
|
base_url = str(block.get("base-url") or block.get("base_url") or "")
|
|
211
314
|
disabled = bool(block.get("disabled"))
|
|
212
|
-
|
|
315
|
+
raw_prefix = block.get("prefix", "")
|
|
316
|
+
prefix = _prefix(raw_prefix)
|
|
317
|
+
if raw_prefix:
|
|
318
|
+
editable = False
|
|
319
|
+
entries = block.get("models", [])
|
|
213
320
|
if not isinstance(entries, list):
|
|
214
|
-
|
|
321
|
+
raise ConfigError(f"{section}.models must be a list")
|
|
215
322
|
for entry in entries:
|
|
216
323
|
if isinstance(entry, str):
|
|
217
324
|
name, alias = entry, ""
|
|
218
325
|
elif isinstance(entry, dict):
|
|
219
|
-
name =
|
|
220
|
-
alias =
|
|
326
|
+
name = entry.get("name") or ""
|
|
327
|
+
alias = entry.get("alias") or ""
|
|
328
|
+
if not isinstance(name, str) or not isinstance(alias, str):
|
|
329
|
+
raise ConfigError("model name and alias must be strings")
|
|
221
330
|
else:
|
|
222
|
-
|
|
331
|
+
raise ConfigError("models entries must be names or mappings")
|
|
223
332
|
key = effective_alias(name, alias)
|
|
224
333
|
if not key:
|
|
334
|
+
editable = False
|
|
225
335
|
continue
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
336
|
+
routes = [key] if not prefix or not force or prefix == key else []
|
|
337
|
+
if prefix:
|
|
338
|
+
routes.append(f"{prefix}/{key}")
|
|
339
|
+
for route in routes:
|
|
340
|
+
# Keep the first hint, but ambiguity cannot enable edits.
|
|
341
|
+
if route in attr:
|
|
342
|
+
editable = False
|
|
343
|
+
attr.setdefault(route, Attribution(
|
|
232
344
|
provider=provider,
|
|
233
345
|
upstream=name.strip(),
|
|
234
346
|
base_url=base_url,
|
|
235
347
|
disabled=disabled,
|
|
236
348
|
source="config",
|
|
237
|
-
)
|
|
238
|
-
|
|
349
|
+
))
|
|
350
|
+
if not recognized:
|
|
351
|
+
raise ConfigError("no supported CLIProxyAPI provider layout")
|
|
239
352
|
self._providers = providers
|
|
353
|
+
self.removable = editable and bool(attr)
|
|
240
354
|
return attr
|
|
241
355
|
|
|
242
356
|
def attribution_map(self) -> dict[str, Attribution]:
|
|
@@ -279,21 +393,21 @@ class CliproxyAdapter(Adapter):
|
|
|
279
393
|
# we never measured. An empty list (gateway down) is a real 0 and shows.
|
|
280
394
|
if self._live is not None:
|
|
281
395
|
head += f", {len(self._live)} published"
|
|
282
|
-
return head + ")"
|
|
396
|
+
return head + (")" if self.removable else ", read-only)")
|
|
283
397
|
|
|
284
398
|
# -- removal ---------------------------------------------------------
|
|
285
399
|
|
|
286
400
|
def plan_removal(self, aliases: Sequence[str]) -> EditPlan:
|
|
287
401
|
"""Cut model entries out of the config text, preserving everything else.
|
|
288
402
|
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
only
|
|
292
|
-
|
|
293
|
-
operator is the only authority. Aliases that appear nowhere in the text
|
|
294
|
-
(published by a running process, or already removed) come back in
|
|
295
|
-
``not_found`` and are left alone.
|
|
403
|
+
The existing text scan is limited to unambiguous, unprefixed legacy
|
|
404
|
+
layouts. v8, mixed, prefixed and otherwise ambiguous attribution is
|
|
405
|
+
read-only: successful parsing is not proof that this scanner can edit it.
|
|
406
|
+
Aliases absent from the text come back in ``not_found`` unchanged.
|
|
296
407
|
"""
|
|
408
|
+
self.attribution_map()
|
|
409
|
+
if not self.removable:
|
|
410
|
+
raise ConfigError("this config layout supports attribution only (read-only)")
|
|
297
411
|
wanted = [a for a in dict.fromkeys(aliases) if a]
|
|
298
412
|
before = read_text(self.config_path)
|
|
299
413
|
entries = {entry.alias: entry for entry in iter_model_entries(before)}
|