openapi-agent-ready 0.2.1__tar.gz → 0.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- openapi_agent_ready-0.3.0/CHANGELOG.md +147 -0
- {openapi_agent_ready-0.2.1 → openapi_agent_ready-0.3.0}/PKG-INFO +24 -22
- {openapi_agent_ready-0.2.1 → openapi_agent_ready-0.3.0}/README.md +23 -21
- {openapi_agent_ready-0.2.1 → openapi_agent_ready-0.3.0}/pyproject.toml +1 -1
- {openapi_agent_ready-0.2.1 → openapi_agent_ready-0.3.0}/src/agent_ready/auditor.py +9 -3
- {openapi_agent_ready-0.2.1 → openapi_agent_ready-0.3.0}/src/agent_ready/cli.py +32 -1
- {openapi_agent_ready-0.2.1 → openapi_agent_ready-0.3.0}/src/agent_ready/rubric.py +54 -12
- {openapi_agent_ready-0.2.1 → openapi_agent_ready-0.3.0}/tests/test_audit.py +86 -0
- openapi_agent_ready-0.2.1/CHANGELOG.md +0 -129
- {openapi_agent_ready-0.2.1 → openapi_agent_ready-0.3.0}/.github/workflows/ci.yml +0 -0
- {openapi_agent_ready-0.2.1 → openapi_agent_ready-0.3.0}/.github/workflows/publish.yml +0 -0
- {openapi_agent_ready-0.2.1 → openapi_agent_ready-0.3.0}/.gitignore +0 -0
- {openapi_agent_ready-0.2.1 → openapi_agent_ready-0.3.0}/CONTRIBUTING.md +0 -0
- {openapi_agent_ready-0.2.1 → openapi_agent_ready-0.3.0}/LICENSE +0 -0
- {openapi_agent_ready-0.2.1 → openapi_agent_ready-0.3.0}/examples/agent_ready_booking_api.yaml +0 -0
- {openapi_agent_ready-0.2.1 → openapi_agent_ready-0.3.0}/examples/sample_booking_api.yaml +0 -0
- {openapi_agent_ready-0.2.1 → openapi_agent_ready-0.3.0}/openapi.yaml +0 -0
- {openapi_agent_ready-0.2.1 → openapi_agent_ready-0.3.0}/src/agent_ready/__init__.py +0 -0
- {openapi_agent_ready-0.2.1 → openapi_agent_ready-0.3.0}/src/agent_ready/mcp_generator.py +0 -0
- {openapi_agent_ready-0.2.1 → openapi_agent_ready-0.3.0}/src/agent_ready/report_generator.py +0 -0
- {openapi_agent_ready-0.2.1 → openapi_agent_ready-0.3.0}/tests/test_mcp_conformance.py +0 -0
|
@@ -0,0 +1,147 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project are documented here.
|
|
4
|
+
Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
|
5
|
+
|
|
6
|
+
## [Unreleased]
|
|
7
|
+
|
|
8
|
+
## [0.3.0] - 2026-07-31
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
- **Progress indicator for long audits.** The ambiguity check dominates runtime
|
|
12
|
+
on large specs and previously produced no output while running — a 587-endpoint
|
|
13
|
+
spec looked like the tool had hung. A progress bar now reports on stderr, shown
|
|
14
|
+
only for specs above 150 endpoints and only when stderr is a terminal, so piped
|
|
15
|
+
output and `--format json` stay clean.
|
|
16
|
+
|
|
17
|
+
### Changed
|
|
18
|
+
- **Ambiguity check is ~4.7x faster** (Stripe: 77s to 16s; Asana: 6.4s to 1.6s)
|
|
19
|
+
using `SequenceMatcher`'s `real_quick_ratio` and `quick_ratio` as prefilters.
|
|
20
|
+
Both are documented upper bounds on `ratio()`, so a pair rejected by a cheap
|
|
21
|
+
bound could not have exceeded the threshold anyway. Results are verified
|
|
22
|
+
identical to the unoptimised implementation.
|
|
23
|
+
|
|
24
|
+
Truncating comparison text was also tried and rejected: another ~5x, but it
|
|
25
|
+
changed which pairs were reported. A faster check that disagrees with itself
|
|
26
|
+
is not an optimisation.
|
|
27
|
+
|
|
28
|
+
A subtler trap surfaced here too: `SequenceMatcher.ratio()` is not symmetric,
|
|
29
|
+
because the autojunk heuristic applies to the second sequence only. The first
|
|
30
|
+
version of this optimisation reversed the argument order to reuse a cached
|
|
31
|
+
sequence — it was faster, passed every existing test, and silently reported
|
|
32
|
+
different pairs. Results are now pinned against a reference implementation.
|
|
33
|
+
|
|
34
|
+
## [0.2.1] - 2026-07-31
|
|
35
|
+
|
|
36
|
+
Found by running the published 0.2.0 against Asana's public spec.
|
|
37
|
+
|
|
38
|
+
### Fixed
|
|
39
|
+
- **Generated scaffolds could contain duplicate arguments and fail to parse.**
|
|
40
|
+
OpenAPI identifies a parameter by name *and* location, and an operation-level
|
|
41
|
+
declaration overrides a path-level one. The extractor concatenated both lists
|
|
42
|
+
without deduplicating, so a spec declaring the same parameter at each level —
|
|
43
|
+
Asana's declares `project_gid` that way — produced a function with two
|
|
44
|
+
identically-named arguments, a Python `SyntaxError`. Parameters declared at
|
|
45
|
+
both levels are now deduplicated with operation-level winning; the same name
|
|
46
|
+
in a different location is still treated as distinct.
|
|
47
|
+
- **`--version` reported a stale hardcoded number.** The CLI printed `0.1.0`
|
|
48
|
+
from a published 0.2.0 install. The version is now read from installed package
|
|
49
|
+
metadata, so `pyproject.toml` is the single source of truth and cannot drift.
|
|
50
|
+
|
|
51
|
+
## [0.2.0] - 2026-07-30
|
|
52
|
+
|
|
53
|
+
The generated MCP scaffold is now verified against the real protocol rather than
|
|
54
|
+
merely checked for syntax — which immediately surfaced two bugs that made
|
|
55
|
+
generated servers uncallable, and a third that stopped them starting at all
|
|
56
|
+
against the current SDK.
|
|
57
|
+
|
|
58
|
+
**Anyone on 0.1.0 should upgrade:** scaffolds generated by 0.1.0 fail to import
|
|
59
|
+
against mcp 2.x.
|
|
60
|
+
|
|
61
|
+
### Added
|
|
62
|
+
- **MCP conformance test suite.** Launches the generated scaffold as a
|
|
63
|
+
subprocess, completes a real MCP handshake over stdio using the official
|
|
64
|
+
client, and validates the advertised tool definitions: name format,
|
|
65
|
+
uniqueness, description presence, input-schema validity as JSON Schema,
|
|
66
|
+
required-argument correctness, and side-effect disclosure. Previously the only
|
|
67
|
+
check on generated output was that the file compiled.
|
|
68
|
+
- **Tool Surface & Schema Strictness** category (category 8), derived from
|
|
69
|
+
OpenAI's function-calling guidance: flags catalogues far above the ~20-tool
|
|
70
|
+
ceiling, request bodies incompatible with strict mode
|
|
71
|
+
(`additionalProperties: false`, all properties required), and parameters that
|
|
72
|
+
imply a fixed value set but accept free-form strings where an enum belongs.
|
|
73
|
+
- **`ToolAnnotations` on generated tools** (`readOnlyHint`, `destructiveHint`,
|
|
74
|
+
`idempotentHint`, `openWorldHint`), derived from the HTTP method. Prose in a
|
|
75
|
+
description helps a model reason about side effects; annotations let a client
|
|
76
|
+
*enforce* — auto-approving reads while requiring confirmation for writes.
|
|
77
|
+
- **`[server]` extra**, a clearer alias for `[mcp]`: these are dependencies the
|
|
78
|
+
*generated* server needs at runtime, not dependencies of the auditor.
|
|
79
|
+
- Findings now carry a `scope` ("endpoint" or "spec"). Spec-level findings are
|
|
80
|
+
weighted at 50% of their category so a single whole-API finding isn't averaged
|
|
81
|
+
away against hundreds of endpoint findings. Stripe's tool-surface score was
|
|
82
|
+
99/100 before this fix and 49.6/100 after.
|
|
83
|
+
|
|
84
|
+
### Fixed
|
|
85
|
+
- **Generated scaffolds failed to import on mcp 2.0.** FastMCP was replaced by
|
|
86
|
+
`MCPServer` in the 2.x SDK. The scaffold now imports through a compatibility
|
|
87
|
+
shim supporting both generations, verified against mcp 1.28.1 and 2.0.0.
|
|
88
|
+
- **Request body fields were silently dropped.** The generator read only
|
|
89
|
+
`parameters`, so a `POST` endpoint's payload properties never became tool
|
|
90
|
+
arguments — producing, for example, a create-booking tool with no way to say
|
|
91
|
+
what to book. Syntactically valid and completely uncallable, which is exactly
|
|
92
|
+
the class of bug a compile check cannot catch.
|
|
93
|
+
- **No arguments were marked required.** Every parameter was emitted with a
|
|
94
|
+
`= None` default, so the advertised input schema had an empty `required`
|
|
95
|
+
array. A model could call an endpoint with a path parameter and omit it,
|
|
96
|
+
producing a malformed URL. Path parameters are now always required, and
|
|
97
|
+
`required: true` from the spec is honoured for query, header, and body fields.
|
|
98
|
+
- **Conformance tests silently skipped under mcp 2.0.** The module guard probed
|
|
99
|
+
a 1.x-only import path, so the whole suite skipped against a current install —
|
|
100
|
+
a green build that verified nothing. The guard now probes for either server
|
|
101
|
+
class, and CI fails explicitly if the suite skips rather than runs.
|
|
102
|
+
- **Conformance assertions were pinned to 1.x field names.** mcp 2.0 renamed
|
|
103
|
+
result fields to snake_case (`serverInfo` → `server_info`, `inputSchema` →
|
|
104
|
+
`input_schema`, `readOnlyHint` → `read_only_hint`). Assertions now read fields
|
|
105
|
+
through version-agnostic accessors; the suite passes on both 1.28.1 and 2.0.0.
|
|
106
|
+
- **Failures inside anyio task groups are now readable.** Exception groups are
|
|
107
|
+
flattened to their leaf causes and `BaseException` is caught rather than
|
|
108
|
+
`Exception`, so a `BaseExceptionGroup` no longer escapes the diagnostic
|
|
109
|
+
wrapper and leaves the real error truncated.
|
|
110
|
+
|
|
111
|
+
### Changed
|
|
112
|
+
- **Distribution renamed to `openapi-agent-ready`.** PyPI rejected `agent-ready`
|
|
113
|
+
as too similar to the existing `agentready` package — its similarity check
|
|
114
|
+
strips separators before comparing. Import name (`agent_ready`), CLI command
|
|
115
|
+
(`agent-ready`), and repository name are unchanged.
|
|
116
|
+
- **Generated scaffolds are self-documenting.** The file header states its
|
|
117
|
+
dependencies, how to run it, a paste-ready Claude Desktop config block, and
|
|
118
|
+
three things to fix before shipping — notably that the scaffold sends no
|
|
119
|
+
credentials, so calls against a real API will 401 until auth is wired in.
|
|
120
|
+
- **Generated servers are named after their source API** (`spotify-web-api`
|
|
121
|
+
rather than a shared `generated-api-server` placeholder), so two connected to
|
|
122
|
+
the same client no longer present identical names.
|
|
123
|
+
- Tool listings are fetched once per generated file and cached rather than
|
|
124
|
+
spawning a server subprocess per test, cutting spawns from ~14 to 4.
|
|
125
|
+
|
|
126
|
+
## [0.1.0] - 2026-07-28
|
|
127
|
+
|
|
128
|
+
Initial release.
|
|
129
|
+
|
|
130
|
+
### Added
|
|
131
|
+
- Seven-category AI-readiness rubric for OpenAPI specs
|
|
132
|
+
- CLI with text, markdown, and JSON output formats
|
|
133
|
+
- CI gate support via `--min-score` and `--max-fails` with documented exit codes
|
|
134
|
+
- Remote spec loading over http(s)
|
|
135
|
+
- Parameter `$ref` resolution and path-level shared parameter inheritance
|
|
136
|
+
- MCP server scaffold generation for endpoints that pass the audit
|
|
137
|
+
- Python API (`load_spec`, `run_audit`)
|
|
138
|
+
|
|
139
|
+
### Fixed
|
|
140
|
+
- OpenAPI `default` responses now count as error documentation. Previously only
|
|
141
|
+
4xx/5xx codes were recognised, which failed every endpoint in specs that use
|
|
142
|
+
`default` — including Stripe's entire public API. (Found by auditing Stripe.)
|
|
143
|
+
- Ambiguity findings are capped at the 25 most-similar pairs with a summary
|
|
144
|
+
line, so large catalogues produce a readable report.
|
|
145
|
+
- Usage-guidelines detection is now regex-based; literal matching missed
|
|
146
|
+
phrasings like "use this only when" and "use this only after".
|
|
147
|
+
- CLI exits cleanly when its output is piped to `head` or `less`.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: openapi-agent-ready
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.3.0
|
|
4
4
|
Summary: Audit any OpenAPI spec for AI-agent and MCP readiness, and generate an MCP server scaffold.
|
|
5
5
|
Project-URL: Homepage, https://github.com/sfaisal/agent-ready
|
|
6
6
|
Project-URL: Issues, https://github.com/sfaisal/agent-ready/issues
|
|
@@ -75,18 +75,17 @@ agent-ready https://example.com/openapi.yaml
|
|
|
75
75
|
|
|
76
76
|
```
|
|
77
77
|
Generic Booking Platform API (v1.0)
|
|
78
|
-
AI-readiness score:
|
|
79
|
-
Endpoints: 4 Fails:
|
|
78
|
+
AI-readiness score: 54.4/100
|
|
79
|
+
Endpoints: 4 Fails: 6 Warnings: 12
|
|
80
80
|
|
|
81
81
|
Category scores:
|
|
82
|
-
Description Clarity
|
|
83
|
-
Side Effects
|
|
84
|
-
Error Responses
|
|
85
|
-
Auth Clarity
|
|
86
|
-
Ambiguity
|
|
87
|
-
Parameter Explanation
|
|
88
|
-
Usage Guidelines
|
|
89
|
-
Tool Surface 100.0/100
|
|
82
|
+
Description Clarity 37.5/100
|
|
83
|
+
Side Effects 70.0/100
|
|
84
|
+
Error Responses 62.5/100
|
|
85
|
+
Auth Clarity 50.0/100
|
|
86
|
+
Ambiguity 100.0/100
|
|
87
|
+
Parameter Explanation 16.7/100
|
|
88
|
+
Usage Guidelines 50.0/100
|
|
90
89
|
```
|
|
91
90
|
|
|
92
91
|
## Why this exists
|
|
@@ -112,9 +111,11 @@ directly onto the rubric:
|
|
|
112
111
|
| **Frictionless authentication** | 4 (auth/scope clarity) |
|
|
113
112
|
| **Structured error handling** | 3 (actionable errors) |
|
|
114
113
|
|
|
115
|
-
Categories 2 (side effects) and 5 (ambiguity)
|
|
116
|
-
— they
|
|
117
|
-
|
|
114
|
+
Categories 2 (side effects) and 5 (ambiguity) have no counterpart in the
|
|
115
|
+
article — they were added during implementation, once it became clear that
|
|
116
|
+
unsignalled write operations and confusable endpoint descriptions cause agent
|
|
117
|
+
failures as often as the original three. Category 8 came later still, from
|
|
118
|
+
OpenAI's function-calling guidance.
|
|
118
119
|
|
|
119
120
|
The gap is measurable, and enormous. A 2026 study of 856 tools across 103 MCP
|
|
120
121
|
servers found that **97.1% of tool descriptions contained at least one quality
|
|
@@ -128,7 +129,7 @@ agent failures in production.
|
|
|
128
129
|
|
|
129
130
|
```bash
|
|
130
131
|
pip install openapi-agent-ready # core
|
|
131
|
-
pip install "openapi-agent-ready[server]"
|
|
132
|
+
pip install "openapi-agent-ready[server]" # + deps to RUN a generated scaffold
|
|
132
133
|
```
|
|
133
134
|
|
|
134
135
|
From source:
|
|
@@ -156,6 +157,10 @@ agent-ready openapi.yaml --format json
|
|
|
156
157
|
|
|
157
158
|
# Generate an MCP server for the endpoints that pass
|
|
158
159
|
agent-ready openapi.yaml --generate-mcp server.py
|
|
160
|
+
|
|
161
|
+
# Generating only writes a file — running it needs the server deps:
|
|
162
|
+
pip install "openapi-agent-ready[server]"
|
|
163
|
+
python server.py
|
|
159
164
|
```
|
|
160
165
|
|
|
161
166
|
### Use it as a CI gate
|
|
@@ -191,10 +196,8 @@ numbers:
|
|
|
191
196
|
|
|
192
197
|
```
|
|
193
198
|
Stripe API (v2026-06-24.dahlia)
|
|
194
|
-
AI-readiness score:
|
|
195
|
-
Endpoints: 587 Fails: 393 Warnings: 2133
|
|
199
|
+
AI-readiness score: 62.9/100
|
|
196
200
|
|
|
197
|
-
Category scores:
|
|
198
201
|
Description Clarity 68.3/100
|
|
199
202
|
Side Effects 64.7/100
|
|
200
203
|
Error Responses 100.0/100
|
|
@@ -202,7 +205,6 @@ Category scores:
|
|
|
202
205
|
Ambiguity 50.0/100
|
|
203
206
|
Parameter Explanation 43.0/100
|
|
204
207
|
Usage Guidelines 51.2/100
|
|
205
|
-
Tool Surface 49.6/100
|
|
206
208
|
```
|
|
207
209
|
|
|
208
210
|
Two things worth drawing out.
|
|
@@ -211,11 +213,11 @@ Two things worth drawing out.
|
|
|
211
213
|
in this tool, not in Stripe.** Stripe documents errors via OpenAPI's `default`
|
|
212
214
|
response rather than enumerating explicit 4xx/5xx codes. Perfectly valid, and
|
|
213
215
|
arguably cleaner. The check only looked for `4`/`5` prefixes, so it failed all
|
|
214
|
-
587 endpoints. Fixed, regression-tested, and the reason
|
|
215
|
-
real spec before trusting it.
|
|
216
|
+
587 endpoints. Fixed, regression-tested, and the reason the README says to run
|
|
217
|
+
this against a real spec before trusting it.
|
|
216
218
|
|
|
217
219
|
**Stripe is widely and correctly regarded as agent-ready, yet its raw OpenAPI
|
|
218
|
-
spec scores
|
|
220
|
+
spec scores 62.9.** That isn't a contradiction — it's the whole point. Stripe's
|
|
219
221
|
agent-readiness comes from the *Agent Toolkit* layered on top: curated tool
|
|
220
222
|
definitions, an MCP server, managed OAuth. The raw spec is the input to that
|
|
221
223
|
layer, not the layer itself. Which is exactly why this tool ships with
|
|
@@ -20,18 +20,17 @@ agent-ready https://example.com/openapi.yaml
|
|
|
20
20
|
|
|
21
21
|
```
|
|
22
22
|
Generic Booking Platform API (v1.0)
|
|
23
|
-
AI-readiness score:
|
|
24
|
-
Endpoints: 4 Fails:
|
|
23
|
+
AI-readiness score: 54.4/100
|
|
24
|
+
Endpoints: 4 Fails: 6 Warnings: 12
|
|
25
25
|
|
|
26
26
|
Category scores:
|
|
27
|
-
Description Clarity
|
|
28
|
-
Side Effects
|
|
29
|
-
Error Responses
|
|
30
|
-
Auth Clarity
|
|
31
|
-
Ambiguity
|
|
32
|
-
Parameter Explanation
|
|
33
|
-
Usage Guidelines
|
|
34
|
-
Tool Surface 100.0/100
|
|
27
|
+
Description Clarity 37.5/100
|
|
28
|
+
Side Effects 70.0/100
|
|
29
|
+
Error Responses 62.5/100
|
|
30
|
+
Auth Clarity 50.0/100
|
|
31
|
+
Ambiguity 100.0/100
|
|
32
|
+
Parameter Explanation 16.7/100
|
|
33
|
+
Usage Guidelines 50.0/100
|
|
35
34
|
```
|
|
36
35
|
|
|
37
36
|
## Why this exists
|
|
@@ -57,9 +56,11 @@ directly onto the rubric:
|
|
|
57
56
|
| **Frictionless authentication** | 4 (auth/scope clarity) |
|
|
58
57
|
| **Structured error handling** | 3 (actionable errors) |
|
|
59
58
|
|
|
60
|
-
Categories 2 (side effects) and 5 (ambiguity)
|
|
61
|
-
— they
|
|
62
|
-
|
|
59
|
+
Categories 2 (side effects) and 5 (ambiguity) have no counterpart in the
|
|
60
|
+
article — they were added during implementation, once it became clear that
|
|
61
|
+
unsignalled write operations and confusable endpoint descriptions cause agent
|
|
62
|
+
failures as often as the original three. Category 8 came later still, from
|
|
63
|
+
OpenAI's function-calling guidance.
|
|
63
64
|
|
|
64
65
|
The gap is measurable, and enormous. A 2026 study of 856 tools across 103 MCP
|
|
65
66
|
servers found that **97.1% of tool descriptions contained at least one quality
|
|
@@ -73,7 +74,7 @@ agent failures in production.
|
|
|
73
74
|
|
|
74
75
|
```bash
|
|
75
76
|
pip install openapi-agent-ready # core
|
|
76
|
-
pip install "openapi-agent-ready[server]"
|
|
77
|
+
pip install "openapi-agent-ready[server]" # + deps to RUN a generated scaffold
|
|
77
78
|
```
|
|
78
79
|
|
|
79
80
|
From source:
|
|
@@ -101,6 +102,10 @@ agent-ready openapi.yaml --format json
|
|
|
101
102
|
|
|
102
103
|
# Generate an MCP server for the endpoints that pass
|
|
103
104
|
agent-ready openapi.yaml --generate-mcp server.py
|
|
105
|
+
|
|
106
|
+
# Generating only writes a file — running it needs the server deps:
|
|
107
|
+
pip install "openapi-agent-ready[server]"
|
|
108
|
+
python server.py
|
|
104
109
|
```
|
|
105
110
|
|
|
106
111
|
### Use it as a CI gate
|
|
@@ -136,10 +141,8 @@ numbers:
|
|
|
136
141
|
|
|
137
142
|
```
|
|
138
143
|
Stripe API (v2026-06-24.dahlia)
|
|
139
|
-
AI-readiness score:
|
|
140
|
-
Endpoints: 587 Fails: 393 Warnings: 2133
|
|
144
|
+
AI-readiness score: 62.9/100
|
|
141
145
|
|
|
142
|
-
Category scores:
|
|
143
146
|
Description Clarity 68.3/100
|
|
144
147
|
Side Effects 64.7/100
|
|
145
148
|
Error Responses 100.0/100
|
|
@@ -147,7 +150,6 @@ Category scores:
|
|
|
147
150
|
Ambiguity 50.0/100
|
|
148
151
|
Parameter Explanation 43.0/100
|
|
149
152
|
Usage Guidelines 51.2/100
|
|
150
|
-
Tool Surface 49.6/100
|
|
151
153
|
```
|
|
152
154
|
|
|
153
155
|
Two things worth drawing out.
|
|
@@ -156,11 +158,11 @@ Two things worth drawing out.
|
|
|
156
158
|
in this tool, not in Stripe.** Stripe documents errors via OpenAPI's `default`
|
|
157
159
|
response rather than enumerating explicit 4xx/5xx codes. Perfectly valid, and
|
|
158
160
|
arguably cleaner. The check only looked for `4`/`5` prefixes, so it failed all
|
|
159
|
-
587 endpoints. Fixed, regression-tested, and the reason
|
|
160
|
-
real spec before trusting it.
|
|
161
|
+
587 endpoints. Fixed, regression-tested, and the reason the README says to run
|
|
162
|
+
this against a real spec before trusting it.
|
|
161
163
|
|
|
162
164
|
**Stripe is widely and correctly regarded as agent-ready, yet its raw OpenAPI
|
|
163
|
-
spec scores
|
|
165
|
+
spec scores 62.9.** That isn't a contradiction — it's the whole point. Stripe's
|
|
164
166
|
agent-readiness comes from the *Agent Toolkit* layered on top: curated tool
|
|
165
167
|
definitions, an MCP server, managed OAuth. The raw spec is the input to that
|
|
166
168
|
layer, not the layer itself. Which is exactly why this tool ships with
|
|
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "openapi-agent-ready"
|
|
7
|
-
version = "0.
|
|
7
|
+
version = "0.3.0"
|
|
8
8
|
description = "Audit any OpenAPI spec for AI-agent and MCP readiness, and generate an MCP server scaffold."
|
|
9
9
|
readme = "README.md"
|
|
10
10
|
requires-python = ">=3.10"
|
|
@@ -124,8 +124,14 @@ def extract_endpoints(spec: dict) -> list[EndpointInfo]:
|
|
|
124
124
|
return endpoints
|
|
125
125
|
|
|
126
126
|
|
|
127
|
-
def run_audit(spec: dict) -> dict:
|
|
128
|
-
"""
|
|
127
|
+
def run_audit(spec: dict, progress=None) -> dict:
|
|
128
|
+
"""
|
|
129
|
+
Run every rubric check and aggregate into a scored result.
|
|
130
|
+
|
|
131
|
+
`progress` is an optional callable receiving (completed, total) for the
|
|
132
|
+
ambiguity check, which dominates runtime on large specs and would otherwise
|
|
133
|
+
look like the tool had hung.
|
|
134
|
+
"""
|
|
129
135
|
endpoints = extract_endpoints(spec)
|
|
130
136
|
security_schemes = (spec.get("components") or {}).get("securitySchemes") or {}
|
|
131
137
|
|
|
@@ -138,7 +144,7 @@ def run_audit(spec: dict) -> dict:
|
|
|
138
144
|
findings += check_parameter_explanation(ep)
|
|
139
145
|
findings += check_usage_guidelines(ep)
|
|
140
146
|
findings += check_schema_strictness(ep)
|
|
141
|
-
findings += check_ambiguity(endpoints)
|
|
147
|
+
findings += check_ambiguity(endpoints, progress=progress)
|
|
142
148
|
findings += check_tool_surface(endpoints)
|
|
143
149
|
|
|
144
150
|
# Score per category. Spec-level findings (e.g. "this catalogue has 587
|
|
@@ -74,6 +74,36 @@ def build_parser() -> argparse.ArgumentParser:
|
|
|
74
74
|
return parser
|
|
75
75
|
|
|
76
76
|
|
|
77
|
+
|
|
78
|
+
def _make_progress(endpoint_count: int, quiet: bool):
|
|
79
|
+
"""
|
|
80
|
+
Report progress on stderr for the ambiguity check.
|
|
81
|
+
|
|
82
|
+
Only shown when the run is large enough to be worth waiting for and stderr
|
|
83
|
+
is a terminal — piping to a file or another command should not accumulate
|
|
84
|
+
carriage returns. Progress goes to stderr so `--format json` stdout stays
|
|
85
|
+
machine-readable.
|
|
86
|
+
"""
|
|
87
|
+
if quiet or endpoint_count < 150 or not sys.stderr.isatty():
|
|
88
|
+
return None
|
|
89
|
+
|
|
90
|
+
def report(done: int, total: int):
|
|
91
|
+
pct = 100 * done / total if total else 100
|
|
92
|
+
bar_width = 24
|
|
93
|
+
filled = int(bar_width * pct / 100)
|
|
94
|
+
bar = "#" * filled + "-" * (bar_width - filled)
|
|
95
|
+
end = "\n" if done >= total else ""
|
|
96
|
+
print(
|
|
97
|
+
f"\r comparing {endpoint_count} endpoints for ambiguity "
|
|
98
|
+
f"[{bar}] {pct:3.0f}%{end}",
|
|
99
|
+
end=end or "",
|
|
100
|
+
file=sys.stderr,
|
|
101
|
+
flush=True,
|
|
102
|
+
)
|
|
103
|
+
|
|
104
|
+
return report
|
|
105
|
+
|
|
106
|
+
|
|
77
107
|
def _text_summary(spec: dict, result: dict) -> str:
|
|
78
108
|
info = spec.get("info") or {}
|
|
79
109
|
lines = [
|
|
@@ -124,7 +154,8 @@ def _run(argv=None) -> int:
|
|
|
124
154
|
)
|
|
125
155
|
return EXIT_SPEC_ERROR
|
|
126
156
|
|
|
127
|
-
|
|
157
|
+
endpoint_count = len(spec.get("paths") or {})
|
|
158
|
+
result = run_audit(spec, progress=_make_progress(endpoint_count, args.quiet))
|
|
128
159
|
|
|
129
160
|
if result["endpoint_count"] == 0:
|
|
130
161
|
print("error: spec contains no operations to audit.", file=sys.stderr)
|
|
@@ -215,28 +215,70 @@ def check_auth_clarity(ep: EndpointInfo, security_schemes: dict) -> list[Finding
|
|
|
215
215
|
return findings
|
|
216
216
|
|
|
217
217
|
|
|
218
|
-
def check_ambiguity(
|
|
218
|
+
def check_ambiguity(
|
|
219
|
+
endpoints: list[EndpointInfo],
|
|
220
|
+
max_reported: int = 25,
|
|
221
|
+
threshold: float = 0.7,
|
|
222
|
+
progress=None,
|
|
223
|
+
) -> list[Finding]:
|
|
219
224
|
"""
|
|
220
225
|
Pairwise-compare endpoint descriptions to flag near-duplicates that could
|
|
221
226
|
confuse a model.
|
|
222
227
|
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
228
|
+
This is O(n^2) in comparisons and dominates total runtime on large specs —
|
|
229
|
+
it was 76 of the 77 seconds spent auditing Stripe's 587 endpoints.
|
|
230
|
+
|
|
231
|
+
The optimisation below cannot change results:
|
|
232
|
+
|
|
233
|
+
`SequenceMatcher.real_quick_ratio()` and `quick_ratio()` are documented
|
|
234
|
+
and `quick_ratio()` are documented upper bounds on `ratio()`. If a cheap
|
|
235
|
+
bound already falls below the threshold, the full comparison cannot exceed
|
|
236
|
+
it either, so the pair is rejected without running it.
|
|
237
|
+
Truncating comparison text was also tried and rejected: it made the check
|
|
238
|
+
~5x faster but changed which pairs were reported, and a faster check that
|
|
239
|
+
disagrees with itself is not an optimisation.
|
|
240
|
+
|
|
241
|
+
`progress` is an optional callable receiving (completed, total) so the CLI
|
|
242
|
+
can report activity on long runs.
|
|
226
243
|
"""
|
|
244
|
+
texts = [ep.full_text.lower() for ep in endpoints]
|
|
245
|
+
|
|
227
246
|
scored = []
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
247
|
+
n = len(endpoints)
|
|
248
|
+
total_pairs = n * (n - 1) // 2
|
|
249
|
+
done = 0
|
|
250
|
+
report_every = max(1, total_pairs // 100)
|
|
251
|
+
|
|
252
|
+
for i in range(n):
|
|
253
|
+
a, text_a = endpoints[i], texts[i]
|
|
254
|
+
if not text_a:
|
|
255
|
+
done += n - i - 1
|
|
256
|
+
continue
|
|
257
|
+
matcher = difflib.SequenceMatcher()
|
|
258
|
+
matcher.set_seq1(text_a)
|
|
259
|
+
|
|
260
|
+
for j in range(i + 1, n):
|
|
261
|
+
done += 1
|
|
262
|
+
if progress is not None and done % report_every == 0:
|
|
263
|
+
progress(done, total_pairs)
|
|
264
|
+
|
|
265
|
+
b, text_b = endpoints[j], texts[j]
|
|
266
|
+
if a.path == b.path or not text_b:
|
|
232
267
|
continue
|
|
233
|
-
|
|
234
|
-
|
|
268
|
+
|
|
269
|
+
matcher.set_seq2(text_b)
|
|
270
|
+
# Cheap upper bounds first — both are guaranteed >= ratio().
|
|
271
|
+
if matcher.real_quick_ratio() <= threshold:
|
|
235
272
|
continue
|
|
236
|
-
|
|
237
|
-
|
|
273
|
+
if matcher.quick_ratio() <= threshold:
|
|
274
|
+
continue
|
|
275
|
+
similarity = matcher.ratio()
|
|
276
|
+
if similarity > threshold:
|
|
238
277
|
scored.append((similarity, a, b))
|
|
239
278
|
|
|
279
|
+
if progress is not None:
|
|
280
|
+
progress(total_pairs, total_pairs)
|
|
281
|
+
|
|
240
282
|
if not scored:
|
|
241
283
|
return [Finding("ambiguity", "pass", "all", "No significant description overlap detected.")]
|
|
242
284
|
|
|
@@ -541,3 +541,89 @@ def test_same_name_different_location_is_not_deduplicated():
|
|
|
541
541
|
}
|
|
542
542
|
ep = extract_endpoints(spec)[0]
|
|
543
543
|
assert len(ep.parameters) == 2, "distinct locations must both survive"
|
|
544
|
+
|
|
545
|
+
|
|
546
|
+
def test_version_is_detected_not_hardcoded():
|
|
547
|
+
"""
|
|
548
|
+
--version printed a hardcoded 0.1.0 from a published 0.2.0 install. The
|
|
549
|
+
version must come from installed metadata so it cannot drift from
|
|
550
|
+
pyproject.toml.
|
|
551
|
+
"""
|
|
552
|
+
import agent_ready
|
|
553
|
+
|
|
554
|
+
assert agent_ready.__version__, "version must not be empty"
|
|
555
|
+
assert agent_ready.__version__ != "0.1.0", (
|
|
556
|
+
"version looks hardcoded to the original release"
|
|
557
|
+
)
|
|
558
|
+
|
|
559
|
+
|
|
560
|
+
# --------------------------------------------------------------------------
|
|
561
|
+
# Progress reporting and ambiguity-check performance
|
|
562
|
+
# --------------------------------------------------------------------------
|
|
563
|
+
|
|
564
|
+
def test_progress_callback_is_invoked_and_completes():
|
|
565
|
+
"""Long runs must show activity; the callback must reach 100%."""
|
|
566
|
+
spec = {
|
|
567
|
+
"paths": {
|
|
568
|
+
f"/thing{i}": {
|
|
569
|
+
"get": {
|
|
570
|
+
"summary": f"Get thing {i}",
|
|
571
|
+
"description": f"Returns the record for thing {i} by identifier.",
|
|
572
|
+
"parameters": [],
|
|
573
|
+
"responses": {"200": {"description": "OK"}},
|
|
574
|
+
}
|
|
575
|
+
}
|
|
576
|
+
for i in range(30)
|
|
577
|
+
}
|
|
578
|
+
}
|
|
579
|
+
seen = []
|
|
580
|
+
run_audit(spec, progress=lambda done, total: seen.append((done, total)))
|
|
581
|
+
assert seen, "progress callback was never invoked"
|
|
582
|
+
assert seen[-1][0] == seen[-1][1], "progress did not reach completion"
|
|
583
|
+
|
|
584
|
+
|
|
585
|
+
def test_ambiguity_prefilter_does_not_change_results():
|
|
586
|
+
"""
|
|
587
|
+
The quick_ratio prefilters are documented upper bounds on ratio(), so they
|
|
588
|
+
must never exclude a pair the full comparison would have flagged.
|
|
589
|
+
|
|
590
|
+
Note the sequence argument order matters: SequenceMatcher.ratio() is not
|
|
591
|
+
symmetric, because the autojunk heuristic applies to the second sequence
|
|
592
|
+
only. An earlier version of this optimisation swapped the order and
|
|
593
|
+
silently changed which pairs were reported.
|
|
594
|
+
"""
|
|
595
|
+
import difflib
|
|
596
|
+
|
|
597
|
+
from agent_ready.rubric import check_ambiguity
|
|
598
|
+
|
|
599
|
+
spec = {
|
|
600
|
+
"paths": {
|
|
601
|
+
"/alpha": {"get": {"summary": "Retrieve a record",
|
|
602
|
+
"description": "Returns the details of an existing record by id.",
|
|
603
|
+
"parameters": [], "responses": {}}},
|
|
604
|
+
"/beta": {"get": {"summary": "Retrieve a record",
|
|
605
|
+
"description": "Returns the details of an existing record by ID.",
|
|
606
|
+
"parameters": [], "responses": {}}},
|
|
607
|
+
"/gamma": {"get": {"summary": "Create invoice",
|
|
608
|
+
"description": "Generates a new invoice for the given customer account.",
|
|
609
|
+
"parameters": [], "responses": {}}},
|
|
610
|
+
}
|
|
611
|
+
}
|
|
612
|
+
endpoints = extract_endpoints(spec)
|
|
613
|
+
|
|
614
|
+
expected = set()
|
|
615
|
+
for i in range(len(endpoints)):
|
|
616
|
+
for j in range(i + 1, len(endpoints)):
|
|
617
|
+
a, b = endpoints[i], endpoints[j]
|
|
618
|
+
ratio = difflib.SequenceMatcher(
|
|
619
|
+
None, a.full_text.lower(), b.full_text.lower()
|
|
620
|
+
).ratio()
|
|
621
|
+
if ratio > 0.7:
|
|
622
|
+
expected.add(f"{a.id} vs {b.id}")
|
|
623
|
+
|
|
624
|
+
actual = {
|
|
625
|
+
f.endpoint
|
|
626
|
+
for f in check_ambiguity(endpoints, max_reported=10**6)
|
|
627
|
+
if f.severity == "warning" and f.endpoint != "(summary)"
|
|
628
|
+
}
|
|
629
|
+
assert actual == expected, f"prefilter changed results: {actual ^ expected}"
|
|
@@ -1,129 +0,0 @@
|
|
|
1
|
-
# Changelog
|
|
2
|
-
|
|
3
|
-
All notable changes to this project are documented here.
|
|
4
|
-
Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
|
5
|
-
|
|
6
|
-
## [Unreleased]
|
|
7
|
-
|
|
8
|
-
## [0.2.1] - 2026-07-31
|
|
9
|
-
|
|
10
|
-
Found by running the published 0.2.0 against Asana's public spec.
|
|
11
|
-
|
|
12
|
-
### Fixed
|
|
13
|
-
- **Generated scaffolds could contain duplicate arguments and fail to parse.**
|
|
14
|
-
OpenAPI identifies a parameter by name *and* location, and an operation-level
|
|
15
|
-
declaration overrides a path-level one. The extractor concatenated both lists
|
|
16
|
-
without deduplicating, so a spec declaring the same parameter at each level —
|
|
17
|
-
Asana's declares `project_gid` that way — produced a function with two
|
|
18
|
-
identically-named arguments, a Python `SyntaxError`. Parameters declared at
|
|
19
|
-
both levels are now deduplicated with operation-level winning; the same name
|
|
20
|
-
in a different location is still treated as distinct.
|
|
21
|
-
- **`--version` reported a stale hardcoded number.** The CLI printed `0.1.0`
|
|
22
|
-
from a published 0.2.0 install. The version is now read from installed package
|
|
23
|
-
metadata, so `pyproject.toml` is the single source of truth and cannot drift.
|
|
24
|
-
|
|
25
|
-
## [0.1.0] - 2026-07-28
|
|
26
|
-
|
|
27
|
-
### Added
|
|
28
|
-
- Seven-category AI-readiness rubric for OpenAPI specs
|
|
29
|
-
- CLI with text, markdown, and JSON output formats
|
|
30
|
-
- CI gate support via `--min-score` and `--max-fails` with documented exit codes
|
|
31
|
-
- Remote spec loading over http(s)
|
|
32
|
-
- Parameter `$ref` resolution and path-level shared parameter inheritance
|
|
33
|
-
- MCP server scaffold generation for endpoints that pass the audit
|
|
34
|
-
- Python API (`load_spec`, `run_audit`)
|
|
35
|
-
|
|
36
|
-
### Fixed
|
|
37
|
-
- OpenAPI `default` responses now count as error documentation. Previously only
|
|
38
|
-
4xx/5xx codes were recognised, which failed every endpoint in specs that use
|
|
39
|
-
`default` — including Stripe's entire public API. (Found by auditing Stripe.)
|
|
40
|
-
- Ambiguity findings are capped at the 25 most-similar pairs with a summary
|
|
41
|
-
line, so large catalogues produce a readable report.
|
|
42
|
-
- Usage-guidelines detection is now regex-based; literal matching missed
|
|
43
|
-
phrasings like "use this only when" and "use this only after".
|
|
44
|
-
- CLI exits cleanly when its output is piped to `head` or `less`.
|
|
45
|
-
|
|
46
|
-
### Added (category 8)
|
|
47
|
-
- **Tool Surface & Schema Strictness** category, derived from OpenAI's
|
|
48
|
-
function-calling guidance: flags catalogues far above the ~20-tool ceiling,
|
|
49
|
-
request bodies incompatible with strict mode (`additionalProperties: false`,
|
|
50
|
-
all properties required), and parameters that imply a fixed value set but
|
|
51
|
-
accept free-form strings where an enum belongs.
|
|
52
|
-
- Findings now carry a `scope` ("endpoint" or "spec"). Spec-level findings are
|
|
53
|
-
weighted at 50% of their category so a single whole-API finding isn't averaged
|
|
54
|
-
away against hundreds of endpoint findings. Stripe's tool-surface score was
|
|
55
|
-
99/100 before this fix and 49.6/100 after.
|
|
56
|
-
|
|
57
|
-
### Added
|
|
58
|
-
- **MCP conformance test suite** (`tests/test_mcp_conformance.py`). Launches the
|
|
59
|
-
generated scaffold as a subprocess, completes a real MCP handshake over stdio
|
|
60
|
-
using the official client, and validates the advertised tool definitions:
|
|
61
|
-
name format, uniqueness, description presence, `inputSchema` validity as JSON
|
|
62
|
-
Schema, required-argument correctness, and side-effect disclosure. Previously
|
|
63
|
-
the only check on generated output was that the file compiled.
|
|
64
|
-
|
|
65
|
-
### Fixed
|
|
66
|
-
- **Request body fields were silently dropped.** The generator read only
|
|
67
|
-
`parameters`, so a `POST` endpoint's payload properties never became tool
|
|
68
|
-
arguments — producing, for example, a create-booking tool with no way to say
|
|
69
|
-
what to book. Syntactically valid and completely uncallable, which is exactly
|
|
70
|
-
the class of bug a compile check cannot catch.
|
|
71
|
-
- **No arguments were marked required.** Every parameter was emitted with a
|
|
72
|
-
`= None` default, so the advertised `inputSchema` had an empty `required`
|
|
73
|
-
array. A model could call an endpoint with a `{path_param}` and omit it,
|
|
74
|
-
producing a malformed URL. Path parameters are now always required, and
|
|
75
|
-
`required: true` from the spec is honoured for query, header, and body fields.
|
|
76
|
-
|
|
77
|
-
### Fixed
|
|
78
|
-
- **Generated scaffold failed on mcp 2.0.** FastMCP was replaced by `MCPServer`
|
|
79
|
-
in the 2.x SDK, so `from mcp.server.fastmcp import FastMCP` no longer resolved
|
|
80
|
-
and every generated server crashed on import. The scaffold now imports through
|
|
81
|
-
a compatibility shim supporting both generations, verified against mcp 1.28.1
|
|
82
|
-
and 2.0.0.
|
|
83
|
-
|
|
84
|
-
### Added
|
|
85
|
-
- Generated tools now carry **`ToolAnnotations`** (`readOnlyHint`,
|
|
86
|
-
`destructiveHint`, `idempotentHint`, `openWorldHint`) derived from the HTTP
|
|
87
|
-
method. Prose in a description helps a model reason about side effects;
|
|
88
|
-
annotations let a client *enforce* — auto-approving reads while requiring
|
|
89
|
-
confirmation for writes. camelCase field names are used deliberately, as they
|
|
90
|
-
are accepted by both SDK generations.
|
|
91
|
-
|
|
92
|
-
### Changed
|
|
93
|
-
- **Generated scaffolds are now self-documenting.** The file header states its
|
|
94
|
-
dependencies (`pip install mcp httpx`), how to run it, a ready-to-paste Claude
|
|
95
|
-
Desktop config block, and three things to fix before shipping — notably that
|
|
96
|
-
the scaffold sends no credentials, so calls against a real API will 401 until
|
|
97
|
-
auth is wired in. Previously someone receiving a generated file out of context
|
|
98
|
-
had no way to know what it needed.
|
|
99
|
-
- **Generated servers are named after their source API** (`spotify-web-api`
|
|
100
|
-
rather than the shared `generated-api-server` placeholder). Two generated
|
|
101
|
-
servers connected to the same client no longer present identical names.
|
|
102
|
-
- **The `[mcp]` extra is now also available as `[server]`,** which describes it
|
|
103
|
-
accurately: these are dependencies the *generated* server needs at runtime,
|
|
104
|
-
not dependencies of the auditor. `[mcp]` is retained as an alias.
|
|
105
|
-
|
|
106
|
-
### Fixed
|
|
107
|
-
- **Conformance tests silently skipped under mcp 2.0.** The module-level guard
|
|
108
|
-
probed `mcp.server.fastmcp`, which does not exist in the 2.x SDK, so the entire
|
|
109
|
-
suite skipped against a current install — a green build that verified nothing.
|
|
110
|
-
The guard now probes for either server class, and CI fails explicitly if the
|
|
111
|
-
suite skips rather than runs.
|
|
112
|
-
|
|
113
|
-
### Fixed
|
|
114
|
-
- **Conformance tests failed against mcp 2.0.** The 2.x SDK renamed result
|
|
115
|
-
fields from camelCase to snake_case (`serverInfo` -> `server_info`,
|
|
116
|
-
`inputSchema` -> `input_schema`, `readOnlyHint` -> `read_only_hint`), and the
|
|
117
|
-
tests were pinned to the 1.x spelling. Assertions now read fields through
|
|
118
|
-
version-agnostic accessors. The suite is verified against both mcp 1.28.1 and
|
|
119
|
-
2.0.0.
|
|
120
|
-
- **Failures inside anyio task groups are now readable.** Exception groups are
|
|
121
|
-
flattened to their leaf causes and `BaseException` is caught rather than
|
|
122
|
-
`Exception`, so a `BaseExceptionGroup` no longer escapes the diagnostic
|
|
123
|
-
wrapper and leaves the real error truncated. This is what finally surfaced
|
|
124
|
-
the rename above.
|
|
125
|
-
|
|
126
|
-
### Changed
|
|
127
|
-
- Tool listings are fetched once per generated file and cached rather than
|
|
128
|
-
spawning a server subprocess per test, cutting spawns from ~14 to 4 and
|
|
129
|
-
running roughly a third faster.
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
{openapi_agent_ready-0.2.1 → openapi_agent_ready-0.3.0}/examples/agent_ready_booking_api.yaml
RENAMED
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|