apiwells 0.1.0__tar.gz → 0.2.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- apiwells-0.2.1/CHANGELOG.md +42 -0
- apiwells-0.2.1/PKG-INFO +194 -0
- apiwells-0.2.1/README.md +174 -0
- apiwells-0.2.1/docs/COMPATIBILITY.md +41 -0
- apiwells-0.2.1/docs/RELEASE_CHECKLIST.md +106 -0
- apiwells-0.2.1/docs/TEST_PLAN.md +152 -0
- apiwells-0.2.1/docs/V0.2_SPEC.md +119 -0
- apiwells-0.2.1/docs/compatibility/compatibility-matrix.md +86 -0
- apiwells-0.2.1/docs/compatibility/model0001-deepseek-flash.md +102 -0
- apiwells-0.2.1/docs/compatibility/model0002-alibaba-qwen37-plus.md +102 -0
- apiwells-0.2.1/docs/compatibility/model0003-openrouter-ling30-flash-fin.md +109 -0
- apiwells-0.2.1/docs/releases/0.1.0/VALIDATION.md +37 -0
- {apiwells-0.1.0 → apiwells-0.2.1}/pyproject.toml +34 -27
- {apiwells-0.1.0 → apiwells-0.2.1}/src/apiwells/__init__.py +2 -2
- apiwells-0.2.1/src/apiwells/cli.py +588 -0
- apiwells-0.2.1/src/apiwells/config.py +61 -0
- apiwells-0.2.1/src/apiwells/models/__init__.py +11 -0
- apiwells-0.2.1/src/apiwells/models/errors.py +62 -0
- apiwells-0.2.1/src/apiwells/models/result.py +56 -0
- apiwells-0.2.1/src/apiwells/probes/__init__.py +27 -0
- apiwells-0.2.1/src/apiwells/probes/chat.py +236 -0
- apiwells-0.2.1/src/apiwells/probes/connectivity.py +557 -0
- apiwells-0.2.1/src/apiwells/probes/models.py +280 -0
- apiwells-0.2.1/src/apiwells/probes/streaming.py +722 -0
- apiwells-0.2.1/src/apiwells/probes/structured.py +817 -0
- apiwells-0.2.1/src/apiwells/probes/tools.py +749 -0
- apiwells-0.2.1/src/apiwells/probes/usage.py +92 -0
- apiwells-0.2.1/src/apiwells/reporting/__init__.py +21 -0
- apiwells-0.2.1/src/apiwells/reporting/aggregation.py +55 -0
- apiwells-0.2.1/src/apiwells/reporting/console.py +192 -0
- apiwells-0.2.1/src/apiwells/reporting/json_report.py +69 -0
- apiwells-0.2.1/src/apiwells/reporting/redaction.py +225 -0
- apiwells-0.2.1/src/apiwells/runner.py +217 -0
- apiwells-0.2.1/src/apiwells/transport.py +242 -0
- apiwells-0.2.1/src/apiwells.egg-info/PKG-INFO +194 -0
- apiwells-0.2.1/src/apiwells.egg-info/SOURCES.txt +74 -0
- apiwells-0.2.1/src/apiwells.egg-info/requires.txt +1 -0
- apiwells-0.2.1/tests/integration/mock_server/__init__.py +0 -0
- apiwells-0.2.1/tests/integration/mock_server/responses.py +147 -0
- apiwells-0.2.1/tests/integration/mock_server/scenarios.py +21 -0
- apiwells-0.2.1/tests/integration/mock_server/server.py +842 -0
- apiwells-0.2.1/tests/integration/test_basic_full_flow.py +398 -0
- apiwells-0.2.1/tests/integration/test_cli_matrix.py +369 -0
- apiwells-0.2.1/tests/integration/test_deep_full_flow.py +418 -0
- apiwells-0.2.1/tests/integration/test_mock_gap_closure.py +328 -0
- apiwells-0.2.1/tests/integration/test_mock_server.py +337 -0
- apiwells-0.2.1/tests/integration/test_reporter_contract.py +378 -0
- apiwells-0.2.1/tests/integration/test_secret_safety_full_flow.py +422 -0
- apiwells-0.2.1/tests/test_basic_runner.py +248 -0
- apiwells-0.2.1/tests/test_chat_probe.py +431 -0
- apiwells-0.2.1/tests/test_cli_reporting.py +285 -0
- apiwells-0.2.1/tests/test_config.py +43 -0
- apiwells-0.2.1/tests/test_connectivity_probe.py +810 -0
- apiwells-0.2.1/tests/test_console_reporter.py +270 -0
- apiwells-0.2.1/tests/test_deep_runner.py +792 -0
- apiwells-0.2.1/tests/test_errors.py +48 -0
- apiwells-0.2.1/tests/test_json_reporter.py +265 -0
- apiwells-0.2.1/tests/test_models_probe.py +342 -0
- apiwells-0.2.1/tests/test_redaction.py +157 -0
- apiwells-0.2.1/tests/test_reporting_aggregation.py +121 -0
- apiwells-0.2.1/tests/test_result.py +98 -0
- apiwells-0.2.1/tests/test_runner.py +75 -0
- apiwells-0.2.1/tests/test_secret_leak.py +397 -0
- apiwells-0.2.1/tests/test_streaming_probe.py +1022 -0
- apiwells-0.2.1/tests/test_structured_probe.py +610 -0
- apiwells-0.2.1/tests/test_tool_probe.py +733 -0
- apiwells-0.2.1/tests/test_transport.py +562 -0
- apiwells-0.2.1/tests/test_usage.py +157 -0
- apiwells-0.1.0/CHANGELOG.md +0 -6
- apiwells-0.1.0/PKG-INFO +0 -118
- apiwells-0.1.0/README.md +0 -101
- apiwells-0.1.0/docs/PUBLISH_ZH.md +0 -290
- apiwells-0.1.0/src/apiwells/cli.py +0 -178
- apiwells-0.1.0/src/apiwells.egg-info/PKG-INFO +0 -118
- apiwells-0.1.0/src/apiwells.egg-info/SOURCES.txt +0 -15
- {apiwells-0.1.0 → apiwells-0.2.1}/LICENSE +0 -0
- {apiwells-0.1.0 → apiwells-0.2.1}/MANIFEST.in +0 -0
- {apiwells-0.1.0 → apiwells-0.2.1}/setup.cfg +0 -0
- {apiwells-0.1.0 → apiwells-0.2.1}/src/apiwells/__main__.py +0 -0
- {apiwells-0.1.0 → apiwells-0.2.1}/src/apiwells.egg-info/dependency_links.txt +0 -0
- {apiwells-0.1.0 → apiwells-0.2.1}/src/apiwells.egg-info/entry_points.txt +0 -0
- {apiwells-0.1.0 → apiwells-0.2.1}/src/apiwells.egg-info/top_level.txt +0 -0
- {apiwells-0.1.0 → apiwells-0.2.1}/tests/test_doctor.py +0 -0
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## Unreleased
|
|
4
|
+
|
|
5
|
+
No unreleased changes are recorded yet.
|
|
6
|
+
|
|
7
|
+
## 0.2.1 — 2026-09-28
|
|
8
|
+
|
|
9
|
+
- Correct the PyPI-facing README and package metadata.
|
|
10
|
+
- Make the README and v0.2 specification patch-version-neutral.
|
|
11
|
+
- Add verified repository and issue project URLs.
|
|
12
|
+
- Replace repository-relative README documentation links with PyPI-safe
|
|
13
|
+
absolute GitHub links.
|
|
14
|
+
- No runtime diagnostic behavior changes.
|
|
15
|
+
|
|
16
|
+
## 0.2.0 — 2026-09-28
|
|
17
|
+
|
|
18
|
+
- Add Basic connectivity/authentication/models diagnostics and explicit-model
|
|
19
|
+
Deep capability diagnostics while retaining legacy `--chat` behavior.
|
|
20
|
+
- Validate Streaming/SSE and report client TTFT and total latency.
|
|
21
|
+
- Parse provider-reported Usage without estimating missing token counts.
|
|
22
|
+
- Validate Tool Calling through the complete local-tool/two-request round trip.
|
|
23
|
+
- Validate Structured Output against JSON Schema, distinguishing strict schema
|
|
24
|
+
support from JSON-object-only behavior.
|
|
25
|
+
- Separate probe execution status from capability support.
|
|
26
|
+
- Add a shared error taxonomy and centralized secret redaction.
|
|
27
|
+
- Add versioned structured JSON reporting alongside compatibility reports.
|
|
28
|
+
- Validate live endpoint compatibility with three preserved ED-031 records;
|
|
29
|
+
results remain point-in-time observations, including partial capabilities.
|
|
30
|
+
- Validate local wheel/sdist packaging and Windows clean installation, including
|
|
31
|
+
Basic and critical Deep smoke checks and 43 clean-wheel integration tests.
|
|
32
|
+
- Document Local-First/no-telemetry behavior, security boundaries, test coverage
|
|
33
|
+
and completed release validation.
|
|
34
|
+
- Complete RC/TestPyPI acceptance and cross-platform CI on Ubuntu, Windows and
|
|
35
|
+
macOS; publish to production PyPI and verify artifact identity, clean
|
|
36
|
+
installation, CLI/JSON contracts, real-provider Basic smoke, secret safety
|
|
37
|
+
and the five-minute onboarding target.
|
|
38
|
+
|
|
39
|
+
## 0.1.0
|
|
40
|
+
- Add a single-request models endpoint check and opt-in text chat smoke check.
|
|
41
|
+
- Add HTTP/network failure categories, response shape checks and sanitized JSON.
|
|
42
|
+
- Disable redirects, retries and implicit environment proxy use.
|
apiwells-0.2.1/PKG-INFO
ADDED
|
@@ -0,0 +1,194 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: apiwells
|
|
3
|
+
Version: 0.2.1
|
|
4
|
+
Summary: Endpoint Doctor for OpenAI-compatible model APIs
|
|
5
|
+
License-Expression: MIT
|
|
6
|
+
Project-URL: Repository, https://github.com/apiwells/endpoint-doctor
|
|
7
|
+
Project-URL: Issues, https://github.com/apiwells/endpoint-doctor/issues
|
|
8
|
+
Keywords: api,diagnostics,llm,endpoint
|
|
9
|
+
Classifier: Development Status :: 3 - Alpha
|
|
10
|
+
Classifier: Environment :: Console
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: Programming Language :: Python :: 3
|
|
13
|
+
Classifier: Operating System :: OS Independent
|
|
14
|
+
Classifier: Topic :: Software Development :: Testing
|
|
15
|
+
Requires-Python: >=3.10
|
|
16
|
+
Description-Content-Type: text/markdown
|
|
17
|
+
License-File: LICENSE
|
|
18
|
+
Requires-Dist: jsonschema<5,>=4.26
|
|
19
|
+
Dynamic: license-file
|
|
20
|
+
|
|
21
|
+
# ApiWells Endpoint Doctor
|
|
22
|
+
|
|
23
|
+
A Local-First CLI for diagnosing OpenAI-compatible model API endpoints from
|
|
24
|
+
your machine. Endpoint Doctor checks connectivity and protocol capabilities;
|
|
25
|
+
it does not score model quality. Python 3.10+ is required.
|
|
26
|
+
|
|
27
|
+
Runtime dependencies include `jsonschema` for Structured Output validation.
|
|
28
|
+
|
|
29
|
+
## Install
|
|
30
|
+
|
|
31
|
+
Install the latest release from PyPI:
|
|
32
|
+
|
|
33
|
+
```sh
|
|
34
|
+
python -m pip install --upgrade apiwells
|
|
35
|
+
apiwells --version
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
`apiwells --version` reports the installed package version.
|
|
39
|
+
|
|
40
|
+
Install this local checkout with:
|
|
41
|
+
|
|
42
|
+
```sh
|
|
43
|
+
python -m pip install .
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
## Basic Doctor
|
|
47
|
+
|
|
48
|
+
Set `APIWELLS_API_KEY` using your shell's secret-input mechanism. Do not put a
|
|
49
|
+
real key in a command, screenshot, issue or repository. Use `--api-key-env NAME`
|
|
50
|
+
to select another environment variable.
|
|
51
|
+
|
|
52
|
+
```sh
|
|
53
|
+
apiwells doctor --base-url https://YOUR-API-HOST/v1
|
|
54
|
+
apiwells doctor --base-url https://YOUR-API-HOST/v1 --v2-json
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
Basic is the default mode: **URL → DNS → TLS → HTTP → Authentication → /models**.
|
|
58
|
+
It sends no inference request. HTTP, authentication and model-list checks share
|
|
59
|
+
one GET to `BASE/models`. An empty model list can pass the response-shape check;
|
|
60
|
+
a model list alone does not prove inference works.
|
|
61
|
+
|
|
62
|
+
Supply the exact API base: `/v1` is not added automatically. For a gateway with
|
|
63
|
+
`/openai/v1`, include that prefix. Do not supply the full `/models` or
|
|
64
|
+
`/chat/completions` URL.
|
|
65
|
+
|
|
66
|
+
For a local unauthenticated development server:
|
|
67
|
+
|
|
68
|
+
```sh
|
|
69
|
+
apiwells doctor --base-url http://127.0.0.1:3000/v1 --anonymous
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
## Deep Doctor
|
|
73
|
+
|
|
74
|
+
```sh
|
|
75
|
+
apiwells doctor --base-url https://YOUR-API-HOST/v1 --deep --model YOUR-MODEL-ID
|
|
76
|
+
apiwells doctor --base-url https://YOUR-API-HOST/v1 --deep --model YOUR-MODEL-ID --v2-json
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
Deep requires an explicit, nonempty model ID and follows:
|
|
80
|
+
**Basic → Chat → Streaming/SSE → Usage → Tool Calling → Structured Output**.
|
|
81
|
+
Usage is extracted from Chat and Streaming responses, not a separate request.
|
|
82
|
+
Capability probes run only after Basic passes and the requested model is found.
|
|
83
|
+
|
|
84
|
+
Deep makes multiple potentially billable inference requests:
|
|
85
|
+
|
|
86
|
+
- Chat validates a nonempty assistant text response.
|
|
87
|
+
- Streaming validates SSE framing, output and terminal `[DONE]`; it reports
|
|
88
|
+
TTFT and total latency. TTFT is measured from request start to the first valid
|
|
89
|
+
model-output event. HTTP headers, keepalive, role-only, empty and metadata-only
|
|
90
|
+
events do not count.
|
|
91
|
+
- Usage contains provider-reported token counts only. Missing values remain
|
|
92
|
+
unavailable or null; ApiWells never estimates them.
|
|
93
|
+
- Tool Calling must complete a full round trip: tool call, validated arguments,
|
|
94
|
+
local diagnostic tool execution, tool result, second request and a final
|
|
95
|
+
response using that result. Merely accepting `tools` is insufficient.
|
|
96
|
+
- Structured Output must return JSON that passes the requested JSON Schema.
|
|
97
|
+
An explicit rejection of `json_schema` as unsupported can trigger one
|
|
98
|
+
additional, potentially billable `json_object` sub-test. JSON-object-only
|
|
99
|
+
success does not certify strict JSON Schema support.
|
|
100
|
+
|
|
101
|
+
There are no hidden billable retries or fallback models. The Tool Calling
|
|
102
|
+
second request and conditional Structured Output sub-test are diagnostic steps,
|
|
103
|
+
with request counts included in their results. `--max-tokens` controls applicable
|
|
104
|
+
generation budgets and is not a guaranteed total cost ceiling; diagnostic
|
|
105
|
+
sub-tests may use bounded request parameters. Provider support varies; not every
|
|
106
|
+
endpoint supports every capability.
|
|
107
|
+
|
|
108
|
+
## Legacy chat compatibility
|
|
109
|
+
|
|
110
|
+
```sh
|
|
111
|
+
apiwells doctor --base-url https://YOUR-API-HOST/v1 --chat --model YOUR-MODEL-ID --json
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
Legacy `--chat` remains a single, opt-in, potentially billable non-streaming
|
|
115
|
+
request to `BASE/chat/completions`, with `Reply OK.` and default `max_tokens=8`.
|
|
116
|
+
A pass requires nonempty assistant text, not exactly `OK`. Use `--max-tokens 32`
|
|
117
|
+
to change the budget; some models need more output tokens. It performs no
|
|
118
|
+
automatic retry or fallback model selection. `--chat` and `--deep` are mutually
|
|
119
|
+
exclusive.
|
|
120
|
+
|
|
121
|
+
## Results and timing
|
|
122
|
+
|
|
123
|
+
Probe execution status (`PASS`, `FAIL`, `PARTIAL`) is separate from capability
|
|
124
|
+
support (`SUPPORTED`, `UNSUPPORTED`, `UNKNOWN`, `NOT_APPLICABLE`). A failed or
|
|
125
|
+
partial probe with unknown support does not establish that a capability is
|
|
126
|
+
unsupported. Core failures make the overall result FAIL; non-passing
|
|
127
|
+
Streaming, Tool Calling or Structured Output probes make it PARTIAL when core
|
|
128
|
+
checks pass.
|
|
129
|
+
|
|
130
|
+
The exit-code contract remains:
|
|
131
|
+
|
|
132
|
+
| Code | Meaning |
|
|
133
|
+
|---|---|
|
|
134
|
+
| `0` | Selected check / overall diagnostics passed |
|
|
135
|
+
| `1` | Endpoint diagnostics did not pass, including an overall PARTIAL |
|
|
136
|
+
| `2` | Local usage or configuration error |
|
|
137
|
+
|
|
138
|
+
For Basic/Deep, `--v2-json` writes a versioned JSON report with
|
|
139
|
+
`schema_version: "1"`, `apiwells_version`, `overall_status` and `probes`.
|
|
140
|
+
Each probe includes status, support, summary, metrics, evidence and error code.
|
|
141
|
+
The report schema version is independent of the package version.
|
|
142
|
+
`--json` retains the compatibility report with `ok`; legacy `--chat --json`
|
|
143
|
+
retains its single-check report. Use `--json`, not `--v2-json`, for legacy chat.
|
|
144
|
+
Completed checks produce one JSON object on stdout; usage/configuration errors
|
|
145
|
+
go to stderr and do not produce a JSON report.
|
|
146
|
+
|
|
147
|
+
Error codes distinguish configuration, DNS/TLS/connectivity/timeout, HTTP/auth,
|
|
148
|
+
JSON/schema, SSE/interruption and capability validation failures. They are
|
|
149
|
+
diagnostic evidence, not definitive root-cause conclusions.
|
|
150
|
+
|
|
151
|
+
`--timeout 15` is a socket-operation timeout, not an overall deadline. Client
|
|
152
|
+
TTFT and total latency include transport effects and are not server-only
|
|
153
|
+
inference timings. Legacy `elapsed_ms` is request/response elapsed time, not
|
|
154
|
+
streaming TTFT.
|
|
155
|
+
|
|
156
|
+
## Security and boundaries
|
|
157
|
+
|
|
158
|
+
- Local-First execution; no telemetry or hidden upload of prompts, responses or
|
|
159
|
+
usage. Requested diagnostic traffic goes to the endpoint you select.
|
|
160
|
+
- API keys and Authorization header values must not be printed. Centralized
|
|
161
|
+
redaction protects console/JSON reports and error paths; raw provider bodies
|
|
162
|
+
are not dumped into reports. Review reports before sharing.
|
|
163
|
+
- HTTPS certificate verification stays enabled. Remote HTTP requires
|
|
164
|
+
`--allow-http`; literal loopback addresses and localhost are allowed for
|
|
165
|
+
development.
|
|
166
|
+
- All redirects, including same-host redirects, are blocked so credentials are
|
|
167
|
+
not forwarded to redirect destinations.
|
|
168
|
+
- Environment proxies are ignored unless `--use-env-proxy` is supplied.
|
|
169
|
+
- Test only endpoints you are authorized to test. Private/local destinations are
|
|
170
|
+
intentionally allowed; this CLI is not an unrestricted server-side URL fetcher.
|
|
171
|
+
|
|
172
|
+
See [Security](https://github.com/apiwells/endpoint-doctor/blob/main/SECURITY.md) for handling and reporting guidance.
|
|
173
|
+
|
|
174
|
+
## Compatibility and development
|
|
175
|
+
|
|
176
|
+
[Compatibility](https://github.com/apiwells/endpoint-doctor/blob/main/docs/COMPATIBILITY.md) indexes three point-in-time ED-031 live
|
|
177
|
+
certification records. These establish observed endpoint behavior, not
|
|
178
|
+
universal provider support or model quality.
|
|
179
|
+
|
|
180
|
+
[The release-facing specification](https://github.com/apiwells/endpoint-doctor/blob/main/docs/V0.2_SPEC.md) summarizes the frozen
|
|
181
|
+
v0.2 scope. Benchmark, scoring/ranking, recommendations, cost optimization,
|
|
182
|
+
Router, Gateway, long-term Monitoring, Web Dashboard, SaaS accounts, cloud
|
|
183
|
+
telemetry and complete Responses API coverage are outside that scope.
|
|
184
|
+
|
|
185
|
+
For test coverage and the observed QA harness dependencies, see
|
|
186
|
+
[the test plan](https://github.com/apiwells/endpoint-doctor/blob/main/docs/TEST_PLAN.md). With the test harness installed, run:
|
|
187
|
+
|
|
188
|
+
```sh
|
|
189
|
+
python -m pytest tests -q
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
See [the release checklist](https://github.com/apiwells/endpoint-doctor/blob/main/docs/RELEASE_CHECKLIST.md) for release evidence,
|
|
193
|
+
and [the changelog](https://github.com/apiwells/endpoint-doctor/blob/main/CHANGELOG.md) for version history and changes.
|
|
194
|
+
License: MIT.
|
apiwells-0.2.1/README.md
ADDED
|
@@ -0,0 +1,174 @@
|
|
|
1
|
+
# ApiWells Endpoint Doctor
|
|
2
|
+
|
|
3
|
+
A Local-First CLI for diagnosing OpenAI-compatible model API endpoints from
|
|
4
|
+
your machine. Endpoint Doctor checks connectivity and protocol capabilities;
|
|
5
|
+
it does not score model quality. Python 3.10+ is required.
|
|
6
|
+
|
|
7
|
+
Runtime dependencies include `jsonschema` for Structured Output validation.
|
|
8
|
+
|
|
9
|
+
## Install
|
|
10
|
+
|
|
11
|
+
Install the latest release from PyPI:
|
|
12
|
+
|
|
13
|
+
```sh
|
|
14
|
+
python -m pip install --upgrade apiwells
|
|
15
|
+
apiwells --version
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
`apiwells --version` reports the installed package version.
|
|
19
|
+
|
|
20
|
+
Install this local checkout with:
|
|
21
|
+
|
|
22
|
+
```sh
|
|
23
|
+
python -m pip install .
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
## Basic Doctor
|
|
27
|
+
|
|
28
|
+
Set `APIWELLS_API_KEY` using your shell's secret-input mechanism. Do not put a
|
|
29
|
+
real key in a command, screenshot, issue or repository. Use `--api-key-env NAME`
|
|
30
|
+
to select another environment variable.
|
|
31
|
+
|
|
32
|
+
```sh
|
|
33
|
+
apiwells doctor --base-url https://YOUR-API-HOST/v1
|
|
34
|
+
apiwells doctor --base-url https://YOUR-API-HOST/v1 --v2-json
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
Basic is the default mode: **URL → DNS → TLS → HTTP → Authentication → /models**.
|
|
38
|
+
It sends no inference request. HTTP, authentication and model-list checks share
|
|
39
|
+
one GET to `BASE/models`. An empty model list can pass the response-shape check;
|
|
40
|
+
a model list alone does not prove inference works.
|
|
41
|
+
|
|
42
|
+
Supply the exact API base: `/v1` is not added automatically. For a gateway with
|
|
43
|
+
`/openai/v1`, include that prefix. Do not supply the full `/models` or
|
|
44
|
+
`/chat/completions` URL.
|
|
45
|
+
|
|
46
|
+
For a local unauthenticated development server:
|
|
47
|
+
|
|
48
|
+
```sh
|
|
49
|
+
apiwells doctor --base-url http://127.0.0.1:3000/v1 --anonymous
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
## Deep Doctor
|
|
53
|
+
|
|
54
|
+
```sh
|
|
55
|
+
apiwells doctor --base-url https://YOUR-API-HOST/v1 --deep --model YOUR-MODEL-ID
|
|
56
|
+
apiwells doctor --base-url https://YOUR-API-HOST/v1 --deep --model YOUR-MODEL-ID --v2-json
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
Deep requires an explicit, nonempty model ID and follows:
|
|
60
|
+
**Basic → Chat → Streaming/SSE → Usage → Tool Calling → Structured Output**.
|
|
61
|
+
Usage is extracted from Chat and Streaming responses, not a separate request.
|
|
62
|
+
Capability probes run only after Basic passes and the requested model is found.
|
|
63
|
+
|
|
64
|
+
Deep makes multiple potentially billable inference requests:
|
|
65
|
+
|
|
66
|
+
- Chat validates a nonempty assistant text response.
|
|
67
|
+
- Streaming validates SSE framing, output and terminal `[DONE]`; it reports
|
|
68
|
+
TTFT and total latency. TTFT is measured from request start to the first valid
|
|
69
|
+
model-output event. HTTP headers, keepalive, role-only, empty and metadata-only
|
|
70
|
+
events do not count.
|
|
71
|
+
- Usage contains provider-reported token counts only. Missing values remain
|
|
72
|
+
unavailable or null; ApiWells never estimates them.
|
|
73
|
+
- Tool Calling must complete a full round trip: tool call, validated arguments,
|
|
74
|
+
local diagnostic tool execution, tool result, second request and a final
|
|
75
|
+
response using that result. Merely accepting `tools` is insufficient.
|
|
76
|
+
- Structured Output must return JSON that passes the requested JSON Schema.
|
|
77
|
+
An explicit rejection of `json_schema` as unsupported can trigger one
|
|
78
|
+
additional, potentially billable `json_object` sub-test. JSON-object-only
|
|
79
|
+
success does not certify strict JSON Schema support.
|
|
80
|
+
|
|
81
|
+
There are no hidden billable retries or fallback models. The Tool Calling
|
|
82
|
+
second request and conditional Structured Output sub-test are diagnostic steps,
|
|
83
|
+
with request counts included in their results. `--max-tokens` controls applicable
|
|
84
|
+
generation budgets and is not a guaranteed total cost ceiling; diagnostic
|
|
85
|
+
sub-tests may use bounded request parameters. Provider support varies; not every
|
|
86
|
+
endpoint supports every capability.
|
|
87
|
+
|
|
88
|
+
## Legacy chat compatibility
|
|
89
|
+
|
|
90
|
+
```sh
|
|
91
|
+
apiwells doctor --base-url https://YOUR-API-HOST/v1 --chat --model YOUR-MODEL-ID --json
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
Legacy `--chat` remains a single, opt-in, potentially billable non-streaming
|
|
95
|
+
request to `BASE/chat/completions`, with `Reply OK.` and default `max_tokens=8`.
|
|
96
|
+
A pass requires nonempty assistant text, not exactly `OK`. Use `--max-tokens 32`
|
|
97
|
+
to change the budget; some models need more output tokens. It performs no
|
|
98
|
+
automatic retry or fallback model selection. `--chat` and `--deep` are mutually
|
|
99
|
+
exclusive.
|
|
100
|
+
|
|
101
|
+
## Results and timing
|
|
102
|
+
|
|
103
|
+
Probe execution status (`PASS`, `FAIL`, `PARTIAL`) is separate from capability
|
|
104
|
+
support (`SUPPORTED`, `UNSUPPORTED`, `UNKNOWN`, `NOT_APPLICABLE`). A failed or
|
|
105
|
+
partial probe with unknown support does not establish that a capability is
|
|
106
|
+
unsupported. Core failures make the overall result FAIL; non-passing
|
|
107
|
+
Streaming, Tool Calling or Structured Output probes make it PARTIAL when core
|
|
108
|
+
checks pass.
|
|
109
|
+
|
|
110
|
+
The exit-code contract remains:
|
|
111
|
+
|
|
112
|
+
| Code | Meaning |
|
|
113
|
+
|---|---|
|
|
114
|
+
| `0` | Selected check / overall diagnostics passed |
|
|
115
|
+
| `1` | Endpoint diagnostics did not pass, including an overall PARTIAL |
|
|
116
|
+
| `2` | Local usage or configuration error |
|
|
117
|
+
|
|
118
|
+
For Basic/Deep, `--v2-json` writes a versioned JSON report with
|
|
119
|
+
`schema_version: "1"`, `apiwells_version`, `overall_status` and `probes`.
|
|
120
|
+
Each probe includes status, support, summary, metrics, evidence and error code.
|
|
121
|
+
The report schema version is independent of the package version.
|
|
122
|
+
`--json` retains the compatibility report with `ok`; legacy `--chat --json`
|
|
123
|
+
retains its single-check report. Use `--json`, not `--v2-json`, for legacy chat.
|
|
124
|
+
Completed checks produce one JSON object on stdout; usage/configuration errors
|
|
125
|
+
go to stderr and do not produce a JSON report.
|
|
126
|
+
|
|
127
|
+
Error codes distinguish configuration, DNS/TLS/connectivity/timeout, HTTP/auth,
|
|
128
|
+
JSON/schema, SSE/interruption and capability validation failures. They are
|
|
129
|
+
diagnostic evidence, not definitive root-cause conclusions.
|
|
130
|
+
|
|
131
|
+
`--timeout 15` is a socket-operation timeout, not an overall deadline. Client
|
|
132
|
+
TTFT and total latency include transport effects and are not server-only
|
|
133
|
+
inference timings. Legacy `elapsed_ms` is request/response elapsed time, not
|
|
134
|
+
streaming TTFT.
|
|
135
|
+
|
|
136
|
+
## Security and boundaries
|
|
137
|
+
|
|
138
|
+
- Local-First execution; no telemetry or hidden upload of prompts, responses or
|
|
139
|
+
usage. Requested diagnostic traffic goes to the endpoint you select.
|
|
140
|
+
- API keys and Authorization header values must not be printed. Centralized
|
|
141
|
+
redaction protects console/JSON reports and error paths; raw provider bodies
|
|
142
|
+
are not dumped into reports. Review reports before sharing.
|
|
143
|
+
- HTTPS certificate verification stays enabled. Remote HTTP requires
|
|
144
|
+
`--allow-http`; literal loopback addresses and localhost are allowed for
|
|
145
|
+
development.
|
|
146
|
+
- All redirects, including same-host redirects, are blocked so credentials are
|
|
147
|
+
not forwarded to redirect destinations.
|
|
148
|
+
- Environment proxies are ignored unless `--use-env-proxy` is supplied.
|
|
149
|
+
- Test only endpoints you are authorized to test. Private/local destinations are
|
|
150
|
+
intentionally allowed; this CLI is not an unrestricted server-side URL fetcher.
|
|
151
|
+
|
|
152
|
+
See [Security](https://github.com/apiwells/endpoint-doctor/blob/main/SECURITY.md) for handling and reporting guidance.
|
|
153
|
+
|
|
154
|
+
## Compatibility and development
|
|
155
|
+
|
|
156
|
+
[Compatibility](https://github.com/apiwells/endpoint-doctor/blob/main/docs/COMPATIBILITY.md) indexes three point-in-time ED-031 live
|
|
157
|
+
certification records. These establish observed endpoint behavior, not
|
|
158
|
+
universal provider support or model quality.
|
|
159
|
+
|
|
160
|
+
[The release-facing specification](https://github.com/apiwells/endpoint-doctor/blob/main/docs/V0.2_SPEC.md) summarizes the frozen
|
|
161
|
+
v0.2 scope. Benchmark, scoring/ranking, recommendations, cost optimization,
|
|
162
|
+
Router, Gateway, long-term Monitoring, Web Dashboard, SaaS accounts, cloud
|
|
163
|
+
telemetry and complete Responses API coverage are outside that scope.
|
|
164
|
+
|
|
165
|
+
For test coverage and the observed QA harness dependencies, see
|
|
166
|
+
[the test plan](https://github.com/apiwells/endpoint-doctor/blob/main/docs/TEST_PLAN.md). With the test harness installed, run:
|
|
167
|
+
|
|
168
|
+
```sh
|
|
169
|
+
python -m pytest tests -q
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
See [the release checklist](https://github.com/apiwells/endpoint-doctor/blob/main/docs/RELEASE_CHECKLIST.md) for release evidence,
|
|
173
|
+
and [the changelog](https://github.com/apiwells/endpoint-doctor/blob/main/CHANGELOG.md) for version history and changes.
|
|
174
|
+
License: MIT.
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
# v0.2 Compatibility
|
|
2
|
+
|
|
3
|
+
The ED-031 compatibility records below were captured before the package version
|
|
4
|
+
was synchronized to 0.2.0. Their recorded package/build versions are preserved
|
|
5
|
+
as historical point-in-time evidence and must not be rewritten to match the
|
|
6
|
+
later final release version. Endpoint Doctor v0.2.0 was subsequently released
|
|
7
|
+
on **2026-09-28**. Production Basic smoke and cross-platform CI do not replace
|
|
8
|
+
these historical capability observations.
|
|
9
|
+
|
|
10
|
+
ED-031 real endpoint/provider live validation is **CLOSED / PASS**. This page is
|
|
11
|
+
a release-facing index; the historical
|
|
12
|
+
[compatibility matrix](compatibility/compatibility-matrix.md) and individual
|
|
13
|
+
records remain the authoritative detailed evidence. An overall PARTIAL record
|
|
14
|
+
does not mean that the ED-031 validation work package failed.
|
|
15
|
+
|
|
16
|
+
| Record | Provider | Model | Test date | ApiWells version | Overall |
|
|
17
|
+
|---|---|---|---|---|---|
|
|
18
|
+
| [MODEL-0001](compatibility/model0001-deepseek-flash.md) | DeepSeek Official | `deepseek-flash` | 2026-09-24 | 0.1.0 (v0.2 development build) | PARTIAL |
|
|
19
|
+
| [MODEL-0002](compatibility/model0002-alibaba-qwen37-plus.md) | Alibaba Cloud Model Studio | `qwen3.7-plus` | 2026-09-27 | 0.1.0 (v0.2 development build) | PASS |
|
|
20
|
+
| [MODEL-0003](compatibility/model0003-openrouter-ling30-flash-fin.md) | OpenRouter | `inclusionai/ling-3.0-flash-fin:free` | 2026-09-27 | 0.1.0 (v0.2 development build) | PARTIAL |
|
|
21
|
+
|
|
22
|
+
## Interpretation
|
|
23
|
+
|
|
24
|
+
- Results are point-in-time observations for the recorded endpoint, model,
|
|
25
|
+
configuration and date, not permanent provider guarantees.
|
|
26
|
+
- Probe execution status and capability support are separate concepts.
|
|
27
|
+
`FAIL / UNKNOWN` does not establish permanent lack of support;
|
|
28
|
+
`PARTIAL / UNKNOWN` does not mean unsupported.
|
|
29
|
+
- Tool Calling certification requires the full round trip, and Structured
|
|
30
|
+
Output certification requires successful JSON Schema validation. Request
|
|
31
|
+
acceptance alone is insufficient.
|
|
32
|
+
- Some historical records use `AVAILABLE` for Usage. This is a
|
|
33
|
+
compatibility-record-local label meaning provider-reported usage data was
|
|
34
|
+
returned and parsed. It does not add a new ProbeResult support enum; those
|
|
35
|
+
remain `SUPPORTED`, `UNSUPPORTED`, `UNKNOWN` and `NOT_APPLICABLE`.
|
|
36
|
+
- Compatibility evidence is not a model-quality Benchmark, ranking or
|
|
37
|
+
recommendation. Timings are observations, not a performance guarantee.
|
|
38
|
+
|
|
39
|
+
Detailed metrics and qualifications belong to the linked ED-031 records and
|
|
40
|
+
are not duplicated here. ED-032 Windows clean-install smoke evidence validates
|
|
41
|
+
installation and execution; its metrics do not replace ED-031 measurements.
|
|
@@ -0,0 +1,106 @@
|
|
|
1
|
+
# v0.2.0 Release Checklist
|
|
2
|
+
|
|
3
|
+
Release date: **2026-09-28**
|
|
4
|
+
|
|
5
|
+
Release status: **CLOSED / PASS**
|
|
6
|
+
|
|
7
|
+
Release source: `cac558b543336f42d17b8db99bfacd6e53d1e5dd`
|
|
8
|
+
|
|
9
|
+
Release tag: `v0.2.0` (annotated)
|
|
10
|
+
|
|
11
|
+
Production package: `apiwells==0.2.0`
|
|
12
|
+
|
|
13
|
+
Public release-facing specification: [V0.2_SPEC.md](V0.2_SPEC.md).
|
|
14
|
+
|
|
15
|
+
This completed release record reflects the acceptance evidence established
|
|
16
|
+
during ED-034, ED-035 and the post-release production acceptance. ED-036 only
|
|
17
|
+
reconciles the repository documentation with those already-confirmed results;
|
|
18
|
+
it does not represent a new build, publication or execution of acceptance tests.
|
|
19
|
+
|
|
20
|
+
## Canonical final artifacts
|
|
21
|
+
|
|
22
|
+
| Artifact | Filename |
|
|
23
|
+
|---|---|
|
|
24
|
+
| Wheel | `apiwells-0.2.0-py3-none-any.whl` |
|
|
25
|
+
| sdist | `apiwells-0.2.0.tar.gz` |
|
|
26
|
+
|
|
27
|
+
Wheel SHA256:
|
|
28
|
+
|
|
29
|
+
```text
|
|
30
|
+
AC346E77ADA9B376420BD9F3B83FDA1CAB2CFD89F80598A5937162F1EDF03168
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
sdist SHA256:
|
|
34
|
+
|
|
35
|
+
```text
|
|
36
|
+
CBD4D357E2DE397951B35D499A12AC89C3002F34E614634A1BFF12ECE35C32A3
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
These identify the canonical final release artifacts. The release source,
|
|
40
|
+
annotated tag and published artifacts are unchanged by this documentation pass.
|
|
41
|
+
|
|
42
|
+
## Completed release checklist
|
|
43
|
+
|
|
44
|
+
- [x] Frozen scope validation.
|
|
45
|
+
- [x] Package and CLI version synchronized to **0.2.0**.
|
|
46
|
+
- [x] Unit tests.
|
|
47
|
+
- [x] Mock integration tests.
|
|
48
|
+
- [x] Secret leak validation.
|
|
49
|
+
- [x] ED-031 real-provider compatibility evidence: **CLOSED / PASS**.
|
|
50
|
+
- [x] ED-033 release documentation reviewed and accepted.
|
|
51
|
+
- [x] ED-034 Windows formal RC acceptance.
|
|
52
|
+
- [x] Wheel build.
|
|
53
|
+
- [x] sdist build.
|
|
54
|
+
- [x] `twine check`.
|
|
55
|
+
- [x] TestPyPI publication.
|
|
56
|
+
- [x] TestPyPI artifact identity.
|
|
57
|
+
- [x] TestPyPI clean install.
|
|
58
|
+
- [x] TestPyPI Basic smoke.
|
|
59
|
+
- [x] Cross-platform CI — Ubuntu: **PASS**.
|
|
60
|
+
- [x] Cross-platform CI — Windows: **PASS**.
|
|
61
|
+
- [x] Cross-platform CI — macOS: **PASS**.
|
|
62
|
+
- [x] Annotated `v0.2.0` tag created and pushed.
|
|
63
|
+
- [x] ED-035 production PyPI publication.
|
|
64
|
+
- [x] Production PyPI artifact identity.
|
|
65
|
+
- [x] Fresh production PyPI clean install.
|
|
66
|
+
- [x] `apiwells --version`: **PASS**, reports `apiwells 0.2.0`.
|
|
67
|
+
- [x] `apiwells --help`: **PASS**.
|
|
68
|
+
- [x] `apiwells doctor --help`: **PASS**.
|
|
69
|
+
- [x] `pip check`: **PASS**.
|
|
70
|
+
- [x] Production real-provider Basic smoke: **PASS**.
|
|
71
|
+
- [x] v2 JSON contract: **PASS**.
|
|
72
|
+
- [x] Secret not present in stdout/stderr.
|
|
73
|
+
- [x] Temporary API key removed after testing.
|
|
74
|
+
- [x] ≤5 minute onboarding target: **PASS**.
|
|
75
|
+
- [x] Final release: **CLOSED / PASS**.
|
|
76
|
+
|
|
77
|
+
## Evidence boundaries
|
|
78
|
+
|
|
79
|
+
Cross-platform CI PASS is not macOS/Linux real-provider live-validation PASS.
|
|
80
|
+
The production Basic smoke establishes the recorded Basic checks, not every
|
|
81
|
+
Deep capability on every provider. The release record makes no claim that
|
|
82
|
+
every Python version was tested or every provider capability passed.
|
|
83
|
+
|
|
84
|
+
ED-031 compatibility results remain point-in-time evidence for their original
|
|
85
|
+
endpoint, model, build and test date. Later production smoke and CI results
|
|
86
|
+
do not replace those observations.
|
|
87
|
+
|
|
88
|
+
## Historical evidence preserved
|
|
89
|
+
|
|
90
|
+
- **ED-031 — CLOSED / PASS:** the [compatibility index](COMPATIBILITY.md) links
|
|
91
|
+
the authoritative matrix and MODEL-0001, MODEL-0002 and MODEL-0003 records.
|
|
92
|
+
Their original build versions, dates, metrics and PASS/PARTIAL results remain
|
|
93
|
+
unchanged.
|
|
94
|
+
- **ED-032 — CLOSED / PASS:** local wheel + sdist build PASS, Windows Basic
|
|
95
|
+
real-endpoint smoke PASS, clean-wheel integration **43 passed**, critical
|
|
96
|
+
Deep smoke PASS and no secret leak observed. These are historical Windows
|
|
97
|
+
acceptance results, separate from ED-034 RC and production PyPI acceptance.
|
|
98
|
+
- **TESTENV-01:** the observed ED-032 Windows harness used pytest **9.1.1**,
|
|
99
|
+
pytest-httpserver **1.1.5** and Werkzeug **3.1.8**. These are test dependencies,
|
|
100
|
+
not ApiWells runtime dependencies. See [TEST_PLAN.md](TEST_PLAN.md).
|
|
101
|
+
- [0.1.0 validation](releases/0.1.0/VALIDATION.md) remains historical evidence for
|
|
102
|
+
that release; it is not relabeled as v0.2.0 acceptance.
|
|
103
|
+
|
|
104
|
+
VERSION-01 was completed by synchronizing both version declarations to 0.2.0
|
|
105
|
+
before the formal release. No version bump or version-source refactor is part
|
|
106
|
+
of ED-036. ED-031 and ED-032 remain closed.
|