agentflight-mcp 1.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- agentflight_mcp-1.2.0/LICENSE +21 -0
- agentflight_mcp-1.2.0/PKG-INFO +552 -0
- agentflight_mcp-1.2.0/README.md +520 -0
- agentflight_mcp-1.2.0/pyproject.toml +68 -0
- agentflight_mcp-1.2.0/setup.cfg +4 -0
- agentflight_mcp-1.2.0/src/agentflight/__init__.py +104 -0
- agentflight_mcp-1.2.0/src/agentflight/cli.py +827 -0
- agentflight_mcp-1.2.0/src/agentflight/diff.py +404 -0
- agentflight_mcp-1.2.0/src/agentflight/eval.py +259 -0
- agentflight_mcp-1.2.0/src/agentflight/guard.py +323 -0
- agentflight_mcp-1.2.0/src/agentflight/http_server.py +135 -0
- agentflight_mcp-1.2.0/src/agentflight/linter.py +733 -0
- agentflight_mcp-1.2.0/src/agentflight/mcp_server.py +730 -0
- agentflight_mcp-1.2.0/src/agentflight/otel_export.py +217 -0
- agentflight_mcp-1.2.0/src/agentflight/post_mortem.py +321 -0
- agentflight_mcp-1.2.0/src/agentflight/providers/__init__.py +52 -0
- agentflight_mcp-1.2.0/src/agentflight/providers/openany.py +965 -0
- agentflight_mcp-1.2.0/src/agentflight/recorder.py +607 -0
- agentflight_mcp-1.2.0/src/agentflight/schema_optimizer.py +168 -0
- agentflight_mcp-1.2.0/src/agentflight/testing.py +270 -0
- agentflight_mcp-1.2.0/src/agentflight_mcp.egg-info/PKG-INFO +552 -0
- agentflight_mcp-1.2.0/src/agentflight_mcp.egg-info/SOURCES.txt +35 -0
- agentflight_mcp-1.2.0/src/agentflight_mcp.egg-info/dependency_links.txt +1 -0
- agentflight_mcp-1.2.0/src/agentflight_mcp.egg-info/entry_points.txt +2 -0
- agentflight_mcp-1.2.0/src/agentflight_mcp.egg-info/requires.txt +8 -0
- agentflight_mcp-1.2.0/src/agentflight_mcp.egg-info/top_level.txt +1 -0
- agentflight_mcp-1.2.0/tests/test_agentflight.py +867 -0
- agentflight_mcp-1.2.0/tests/test_api_portal.py +50 -0
- agentflight_mcp-1.2.0/tests/test_cli_extensions.py +45 -0
- agentflight_mcp-1.2.0/tests/test_diff_and_otel.py +436 -0
- agentflight_mcp-1.2.0/tests/test_http_server.py +65 -0
- agentflight_mcp-1.2.0/tests/test_mcp_resources_prompts.py +72 -0
- agentflight_mcp-1.2.0/tests/test_openany_provider.py +609 -0
- agentflight_mcp-1.2.0/tests/test_post_mortem.py +65 -0
- agentflight_mcp-1.2.0/tests/test_schema_optimizer.py +69 -0
- agentflight_mcp-1.2.0/tests/test_testing.py +79 -0
- agentflight_mcp-1.2.0/tests/test_vercel_gateway.py +91 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Akandwanaho Ivan Alvin (codesphere-u)
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,552 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: agentflight-mcp
|
|
3
|
+
Version: 1.2.0
|
|
4
|
+
Summary: AI Agent Black-Box Flight Recorder, Deterministic Trajectory Replay, Loop/Cost Circuit Breaker & MCP Tool-Smell Linter
|
|
5
|
+
Author-email: Akandwanaho Ivan Alvin <codesphere.u@github.com>
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/codesphere-u/agentflight
|
|
8
|
+
Project-URL: Repository, https://github.com/codesphere-u/agentflight
|
|
9
|
+
Project-URL: Playground, https://agentflight-tau.vercel.app
|
|
10
|
+
Project-URL: Creator, https://alvin.codesphereug.com/
|
|
11
|
+
Project-URL: Issues, https://github.com/codesphere-u/agentflight/issues
|
|
12
|
+
Keywords: ai-agents,mcp,model-context-protocol,agent-evaluation,llm-testing,flight-recorder,circuit-breaker,developer-tools
|
|
13
|
+
Classifier: Development Status :: 5 - Production/Stable
|
|
14
|
+
Classifier: Intended Audience :: Developers
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Topic :: Software Development :: Testing
|
|
20
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
21
|
+
Requires-Python: >=3.10
|
|
22
|
+
Description-Content-Type: text/markdown
|
|
23
|
+
License-File: LICENSE
|
|
24
|
+
Requires-Dist: fastapi>=0.162.2
|
|
25
|
+
Requires-Dist: pydantic>=2.14.3
|
|
26
|
+
Requires-Dist: uvicorn>=0.54.0
|
|
27
|
+
Provides-Extra: dev
|
|
28
|
+
Requires-Dist: httpx>=0.27.0; extra == "dev"
|
|
29
|
+
Requires-Dist: pytest>=7.4.0; extra == "dev"
|
|
30
|
+
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
|
|
31
|
+
Dynamic: license-file
|
|
32
|
+
|
|
33
|
+
# AgentFlight (`agentflight`)
|
|
34
|
+
|
|
35
|
+
<p align="center">
|
|
36
|
+
<img src="assets/banner.jpg" alt="AgentFlight: AI Agent Black-Box Flight Recorder & Deterministic Trajectory Replay" width="100%" />
|
|
37
|
+
</p>
|
|
38
|
+
|
|
39
|
+
[](https://github.com/codesphere-u/agentflight/actions/workflows/ci.yml)
|
|
40
|
+
[](https://pypi.org/project/agentflight-mcp/)
|
|
41
|
+
[](https://github.com/codesphere-u/agentflight/releases/tag/v1.2.0)
|
|
42
|
+
[](https://www.python.org/)
|
|
43
|
+
[](https://modelcontextprotocol.io/)
|
|
44
|
+
[](https://agentflight-tau.vercel.app/mcp)
|
|
45
|
+
[](#hosted-endpoint-vercel)
|
|
46
|
+
[](https://glama.ai/mcp/servers/codesphere-u/agentflight)
|
|
47
|
+
[](smithery.yaml)
|
|
48
|
+
[](tests/)
|
|
49
|
+
[](src/agentflight/)
|
|
50
|
+
[](LICENSE)
|
|
51
|
+
|
|
52
|
+
**AI Agent Black-Box Flight Recorder, Deterministic Trajectory Replay (`<0.02 ms`, `$0.00` CI Spend), Differential Regression Detector, OpenTelemetry / W3C TraceContext Exporter (for `stackpulse`), Real-Time Loop/Cost Circuit Breaker & 10-Rule MCP Tool-Smell Linter.**
|
|
53
|
+
|
|
54
|
+
> 🚀 **Live Interactive Developer Portal & Playground:** Visit [`https://agentflight-tau.vercel.app`](https://agentflight-tau.vercel.app) in your web browser to test tool linting live, inspect schemas, and copy 1-click configurations for Claude Code, Cursor, Windsurf, and Claude Desktop.
|
|
55
|
+
>
|
|
56
|
+
> ⭐ **Like what you see?** Give this repository a **Star** and follow creator **[Akandwanaho Ivan Alvin](https://alvin.codesphereug.com/)**!
|
|
57
|
+
|
|
58
|
+
<p align="center">
|
|
59
|
+
<img src="marketing/openany-launch/agentflight_openany_demo.gif" alt="AgentFlight Live Demo — Pre-Execution Cost Circuit Breaker, Cryptographic Flight Recorder & 0.08 μs Offline Replay" width="820" />
|
|
60
|
+
</p>
|
|
61
|
+
|
|
62
|
+
<p align="center">
|
|
63
|
+
<a href="https://glama.ai/mcp/servers/codesphere-u/agentflight">
|
|
64
|
+
<img src="https://glama.ai/mcp/servers/codesphere-u/agentflight/badges/card.svg" alt="agentflight MCP server – quality and maintenance score on Glama" width="500" />
|
|
65
|
+
</a>
|
|
66
|
+
</p>
|
|
67
|
+
|
|
68
|
+
---
|
|
69
|
+
|
|
70
|
+
## Why AgentFlight Exists (The 2026 Agent Reliability Pain Point)
|
|
71
|
+
|
|
72
|
+
Production AI agent engineering in 2026 faces two compounding bottlenecks:
|
|
73
|
+
|
|
74
|
+
1. **Non-Deterministic, Expensive Multi-Step Agent Debugging & CI Testing**: When a 15-step tool-calling or MCP agent fails on step 12, re-running live LLM turns and external API calls in CI is slow (`10–45s`), flaky, and expensive. Engineers need a **"VCR / Black-Box Flight Recorder for AI Agents"** that captures every LLM turn, MCP RPC, and tool invocation into a tamper-evident SHA-256 hash chain cassette and replays the entire trajectory deterministically in **`<0.02 ms` with `$0.00` token spend**—raising a structural diff (`TrajectoryDivergenceError`) the instant an agent's behavior drifts from the golden baseline.
|
|
75
|
+
2. **"MCP Tool Smell" (Schema Ergonomics Bugs)**: Empirical evaluations across Model Context Protocol (MCP) servers and function-calling suites show that most tool-selection failures are caused not by model reasoning limits, but by poorly authored tool schemas: missing parameter descriptions, ambiguous boolean flags (`flag`, `mode`), deeply nested input objects (`depth > 3`), unconstrained finite-set strings (`status`, `format` without `enum`), overlapping tool names (`search_records` vs. `find_records`), and destructive tools (`delete_*`, `drop_*`) without confirmation guards.
|
|
76
|
+
|
|
77
|
+
**`agentflight`** solves both problems with a pure-Python 3.10+ stdlib core engine (`src/agentflight/` uses zero third-party runtime dependencies), paired with a FastAPI/ASGI portal gateway.
|
|
78
|
+
|
|
79
|
+
---
|
|
80
|
+
|
|
81
|
+
## Architecture Blueprint
|
|
82
|
+
|
|
83
|
+
```text
|
|
84
|
+
+-----------------------------------------------------------------------------------+
|
|
85
|
+
| agentflight |
|
|
86
|
+
+-----------------------------------------------------------------------------------+
|
|
87
|
+
| |
|
|
88
|
+
| 1. Pre-Deploy Static Gate 2. Runtime Execution & Guard |
|
|
89
|
+
| +----------------------------------+ +------------------------------------+ |
|
|
90
|
+
| | ToolSmellLinter (linter.py) | | TrajectoryCircuitBreaker (guard.py)| |
|
|
91
|
+
| | - 10 Deterministic Rules | | - Exact Call Loop (N repeats) | |
|
|
92
|
+
| | (TS001 .. TS010) | | - Ping-Pong Cycle (A->B->A->B, | |
|
|
93
|
+
| | - Agent Ergonomics Score (0-100) | | A->B->C->A->B->C) | |
|
|
94
|
+
| | - Token Footprint Estimator | | - Consecutive Tool Error Storm | |
|
|
95
|
+
| | - Actionable Remediations | | - Hard Token & USD Budget Caps | |
|
|
96
|
+
| +-----------------+----------------+ +-----------------+------------------+ |
|
|
97
|
+
| | | |
|
|
98
|
+
| v v |
|
|
99
|
+
| 3. Tamper-Evident Flight Recorder 4. CI Replay & Golden Diff Engine |
|
|
100
|
+
| +----------------------------------+ +------------------------------------+ |
|
|
101
|
+
| | FlightRecorder (recorder.py) | | CassetteReplayer & Evaluator | |
|
|
102
|
+
| | - Append-only TrajectorySpan | --> | - Sub-ms $0.00 Deterministic Stub | |
|
|
103
|
+
| | - Canonical JSON SHA-256 Chain | | - Structural Diff on Divergence | |
|
|
104
|
+
| | (prev_hash -> step_hash) | | - Tool Precision / Recall / F1 | |
|
|
105
|
+
| | - .flight.json & .flight.jsonl | | - Step Efficiency & Cost Delta | |
|
|
106
|
+
| | - verify_cassette() Tamper Check | | - Goal Completion Score (0.0-1.0) | |
|
|
107
|
+
| +----------------------------------+ +------------------------------------+ |
|
|
108
|
+
+-----------------------------------------------------------------------------------+
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
---
|
|
112
|
+
|
|
113
|
+
## Installation
|
|
114
|
+
|
|
115
|
+
```bash
|
|
116
|
+
# Install from PyPI (module import & CLI command remain `agentflight`):
|
|
117
|
+
pip install agentflight-mcp
|
|
118
|
+
|
|
119
|
+
# Or install v1.2.0 directly from GitHub:
|
|
120
|
+
pip install "git+https://github.com/codesphere-u/agentflight.git@v1.2.0"
|
|
121
|
+
|
|
122
|
+
# Contributors:
|
|
123
|
+
pip install -e ".[dev]"
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
---
|
|
127
|
+
|
|
128
|
+
## Quickstart
|
|
129
|
+
|
|
130
|
+
### 1. Record & Cryptographically Verify an Agent Trajectory (`FlightRecorder`)
|
|
131
|
+
|
|
132
|
+
```python
|
|
133
|
+
from agentflight import FlightRecorder, CassetteReplayer, verify_cassette
|
|
134
|
+
|
|
135
|
+
recorder = FlightRecorder(agent_id="sre-incident-agent", metadata={"env": "prod"})
|
|
136
|
+
|
|
137
|
+
recorder.record(
|
|
138
|
+
kind="llm_turn",
|
|
139
|
+
actor="planner_model",
|
|
140
|
+
input_payload={"goal": "Investigate payment-api p99 latency spike"},
|
|
141
|
+
output_payload={"next_tool": "prometheus.query_metrics"},
|
|
142
|
+
latency_ms=410.0,
|
|
143
|
+
prompt_tokens=580,
|
|
144
|
+
completion_tokens=90,
|
|
145
|
+
cost_usd=0.0022,
|
|
146
|
+
)
|
|
147
|
+
recorder.record(
|
|
148
|
+
kind="mcp_rpc",
|
|
149
|
+
actor="prometheus.query_metrics",
|
|
150
|
+
input_payload={"service": "payment-api", "window": "15m"},
|
|
151
|
+
output_payload={"p99_latency_ms": 1840.0, "anomaly": True},
|
|
152
|
+
latency_ms=85.0,
|
|
153
|
+
)
|
|
154
|
+
|
|
155
|
+
# Persist as tamper-evident .flight.json or .flight.jsonl
|
|
156
|
+
recorder.save("cassettes/incident_8841.flight.json")
|
|
157
|
+
assert verify_cassette(recorder) is True
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
### 2. Replay Deterministically in `<0.02 ms` with `$0.00` Spend (`CassetteReplayer`)
|
|
161
|
+
|
|
162
|
+
```python
|
|
163
|
+
replayer = CassetteReplayer("cassettes/incident_8841.flight.json", strict_order=True)
|
|
164
|
+
|
|
165
|
+
# Stub any MCP or tool callable directly in pytest:
|
|
166
|
+
query_metrics_stub = replayer.stub_tool("prometheus.query_metrics", kind="mcp_rpc")
|
|
167
|
+
|
|
168
|
+
# Replaying matching steps returns recorded payloads in ~2.8 microseconds:
|
|
169
|
+
replayer.replay_step("llm_turn", "planner_model", {"goal": "Investigate payment-api p99 latency spike"})
|
|
170
|
+
result = query_metrics_stub(window="15m", service="payment-api") # Key-order invariant!
|
|
171
|
+
assert result["anomaly"] is True
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
If a code change causes the agent to call the wrong tool or pass altered parameters, `CassetteReplayer` immediately raises `TrajectoryDivergenceError` with `missing_keys`, `extra_keys`, and `changed_values`.
|
|
175
|
+
|
|
176
|
+
### 3. Lint MCP Tool Schemas Across 10 Ergonomics Rules (`ToolSmellLinter`)
|
|
177
|
+
|
|
178
|
+
```python
|
|
179
|
+
from agentflight import ToolSmellLinter, SHOWCASE_FLAWED_MCP_TOOLS
|
|
180
|
+
|
|
181
|
+
linter = ToolSmellLinter()
|
|
182
|
+
report = linter.lint(SHOWCASE_FLAWED_MCP_TOOLS)
|
|
183
|
+
|
|
184
|
+
print(f"Ergonomics Score: {report.ergonomics_score}/100 (Grade {report.grade})")
|
|
185
|
+
for finding in report.findings:
|
|
186
|
+
print(f"[{finding.rule_id}] {finding.tool_name}: {finding.message}")
|
|
187
|
+
print(f" Fix -> {finding.remediation}")
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
### 4. Guard Live Agents Against Loops, Ping-Pong Cycles & Runaway Spend (`TrajectoryCircuitBreaker`)
|
|
191
|
+
|
|
192
|
+
```python
|
|
193
|
+
from agentflight import TrajectoryCircuitBreaker, CircuitBreakerTripped
|
|
194
|
+
|
|
195
|
+
breaker = TrajectoryCircuitBreaker(
|
|
196
|
+
max_identical_repeats=3,
|
|
197
|
+
max_ping_pong_cycles=2,
|
|
198
|
+
max_consecutive_errors=3,
|
|
199
|
+
max_total_tokens=25_000,
|
|
200
|
+
max_cost_usd=0.50,
|
|
201
|
+
)
|
|
202
|
+
|
|
203
|
+
try:
|
|
204
|
+
breaker.observe("search_logs", {"query": "timeout"})
|
|
205
|
+
breaker.observe("refine_query", {"query": "timeout"})
|
|
206
|
+
breaker.observe("search_logs", {"query": "timeout"})
|
|
207
|
+
breaker.observe("refine_query", {"query": "timeout"}) # Trips on A -> B -> A -> B!
|
|
208
|
+
except CircuitBreakerTripped as exc:
|
|
209
|
+
print(exc.reason, exc.step_index, exc.evidence)
|
|
210
|
+
# -> ping_pong_oscillation 3 {'period': 2, 'cycles': 2, 'pattern_actors': ['search_logs', 'refine_query']}
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
### 5. Compare Candidate vs. Golden Trajectories (`compare_cassettes` & `TrajectoryEvaluator`)
|
|
214
|
+
|
|
215
|
+
```python
|
|
216
|
+
from agentflight import compare_cassettes, diff_trajectories
|
|
217
|
+
|
|
218
|
+
# v1.1.0 Trajectory Differential Analyzer (Levenshtein/LCS sequence diff + loop/error/cost regression check)
|
|
219
|
+
report = compare_cassettes("cassettes/baseline.flight.jsonl", "cassettes/candidate.flight.jsonl")
|
|
220
|
+
print(report.regression_detected, report.similarity_score, report.added_tools, report.cost_delta_pct)
|
|
221
|
+
print(report.summary_markdown)
|
|
222
|
+
|
|
223
|
+
# Golden Evaluation Metrics (Precision, Recall, F1, Step Efficiency)
|
|
224
|
+
diff = diff_trajectories(golden_recorder, candidate_recorder)
|
|
225
|
+
print(diff.tool_precision, diff.tool_recall, diff.step_efficiency_ratio, diff.goal_completion_score)
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
### 6. Export W3C TraceContext / OpenTelemetry Spans for `stackpulse` (`export_cassette_to_otel`)
|
|
229
|
+
|
|
230
|
+
```python
|
|
231
|
+
from agentflight import export_cassette_to_otel
|
|
232
|
+
|
|
233
|
+
spans = export_cassette_to_otel(
|
|
234
|
+
"cassettes/incident_8841.flight.json",
|
|
235
|
+
service_name="sre-incident-agent",
|
|
236
|
+
output_path="spans.json",
|
|
237
|
+
)
|
|
238
|
+
# Directly ingestible by `codesphere-u/stackpulse` for critical-path DAG analysis and SLO burn-rate monitoring!
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
---
|
|
242
|
+
|
|
243
|
+
## MCP Tool-Smell Rule Catalog (`TS001`–`TS010`)
|
|
244
|
+
|
|
245
|
+
| Rule ID | Severity | Penalty | Rule Title & Ergonomic Failure Mode |
|
|
246
|
+
| :--- | :--- | ---: | :--- |
|
|
247
|
+
| **`TS001`** | `ERROR` | `-12` | **Missing or Ultra-Short Tool Description (`<15` chars)**: Leaves the LLM guessing when to invoke the tool. |
|
|
248
|
+
| **`TS002`** | `WARNING` | `-5` | **Property Missing `description`**: A parameter inside `inputSchema.properties` has no semantic description or format hint. |
|
|
249
|
+
| **`TS003`** | `WARNING` | `-8` | **Deeply Nested Input Schema (`depth > 3`)**: Deep object nesting degrades JSON argument serialization accuracy on smaller/faster models. |
|
|
250
|
+
| **`TS004`** | `WARNING` | `-7` | **Excessive Top-Level Parameter Count (`> 8`)**: Increases parameter hallucination and omission rates. |
|
|
251
|
+
| **`TS005`** | `WARNING` | `-5` | **Ambiguous Boolean Flag Name**: Boolean named `flag`, `mode`, `option`, `data`, `value`, `switch`, or `toggle` without clear polarity. |
|
|
252
|
+
| **`TS006`** | `WARNING` | `-6` | **Unconstrained Finite-Set String Parameter**: Parameter named `status`, `sort`, `format`, `env`, `role`, `type`, or `priority` lacks `enum` or `pattern`. |
|
|
253
|
+
| **`TS007`** | `WARNING` | `-6` | **Missing `required` Array on Non-Empty Schema**: Causes models to omit mandatory arguments. |
|
|
254
|
+
| **`TS008`** | `ERROR` | `-10` | **High Lexical Overlap Between Tools (`cosine >= 0.72`)**: Overlapping tool names/descriptions in the same suite cause nondeterministic routing confusion. |
|
|
255
|
+
| **`TS009`** | `WARNING` | `-7` | **Bloated Schema Token Footprint (`> 450` tokens)**: Excessive schema verbosity inflates prompt cost on every agent turn. |
|
|
256
|
+
| **`TS010`** | `ERROR` | `-10` | **Destructive Tool Missing Safety Guard**: Tool prefixed with `delete_`, `drop_`, `remove_`, `transfer_`, or `exec_` lacks a `confirm`/`dry_run`/`idempotency_key` parameter or irreversibility warning. |
|
|
257
|
+
|
|
258
|
+
---
|
|
259
|
+
|
|
260
|
+
## Measured Benchmark Results
|
|
261
|
+
|
|
262
|
+
Reproducible via `python -m agentflight.cli benchmark` (measured on Python 3.12):
|
|
263
|
+
|
|
264
|
+
| Subsystem Operation | Workload Size | Total Latency (ms) | Per-Unit Latency | Throughput |
|
|
265
|
+
| :--- | :--- | ---: | ---: | ---: |
|
|
266
|
+
| `FlightRecorder.record` (SHA-256 chain) | 1,000 spans | 7.42 ms | 7.42 us/span | 134,770 spans/s |
|
|
267
|
+
| `verify_cassette` (Tamper Verification) | 1,000 spans | 3.40 ms | 3.40 us/span | 294,117 spans/s |
|
|
268
|
+
| `CassetteReplayer.replay_step` ($0 Stub) | 1,000 spans | 2.83 ms | 2.83 us/span | 353,356 spans/s |
|
|
269
|
+
| `TrajectoryCircuitBreaker.observe_span` | 1,000 spans | 3.01 ms | 3.01 us/span | 332,225 spans/s |
|
|
270
|
+
| `ToolSmellLinter.lint` (10 Rules + TS008) | 200 MCP tools | 119.27 ms | 596.35 us/tool | 1,676 tools/s |
|
|
271
|
+
| `TrajectoryEvaluator.diff_trajectories` | 1,000 vs 1,000 spans | 4.06 ms | 4.06 us/span | 246,305 spans/s |
|
|
272
|
+
|
|
273
|
+
---
|
|
274
|
+
|
|
275
|
+
## CLI Reference
|
|
276
|
+
|
|
277
|
+
```bash
|
|
278
|
+
# 1. Lint showcase flawed vs. hardened MCP tool suites (or pass --schema-file path/to/tools.json)
|
|
279
|
+
agentflight lint
|
|
280
|
+
|
|
281
|
+
# 2. Automated Schema Repair: harden smelly tools to Glama TDQS Gold Standard
|
|
282
|
+
agentflight fix tools.json --output hardened_tools.json
|
|
283
|
+
|
|
284
|
+
# 3. CI/CD Pre-Flight Gate: lint tool smells, verify cassette integrity, enforce budget SLAs
|
|
285
|
+
agentflight ci --schema-file tools.json --cassette runs/test.cassette.json --fail-on-smell
|
|
286
|
+
|
|
287
|
+
# 4. Forensic Post-Mortem Investigator: generate AAIB/NTSB incident reports from flight cassettes
|
|
288
|
+
agentflight post-mortem crash_incident.cassette.json --output report.md
|
|
289
|
+
|
|
290
|
+
# 5. Run the end-to-end recording, <0.02ms replay, circuit breaker & diff demo
|
|
291
|
+
agentflight demo
|
|
292
|
+
|
|
293
|
+
# 6. Run the reproducible subsystem benchmark suite
|
|
294
|
+
agentflight benchmark
|
|
295
|
+
|
|
296
|
+
# 7. Compare baseline vs. candidate cassettes for behavioral regressions
|
|
297
|
+
agentflight diff baseline.flight.json candidate.flight.json --json
|
|
298
|
+
|
|
299
|
+
# 8. Export cassette to W3C TraceContext / OpenTelemetry JSON spans for StackPulse
|
|
300
|
+
agentflight export-otel incident_8841.flight.json --output spans.json
|
|
301
|
+
|
|
302
|
+
# 9. Start the JSON-RPC 2.0 MCP Server over stdio
|
|
303
|
+
agentflight serve-mcp
|
|
304
|
+
```
|
|
305
|
+
|
|
306
|
+
---
|
|
307
|
+
|
|
308
|
+
## Deterministic Testing with `@flight_test` (VCR for AI Agents)
|
|
309
|
+
|
|
310
|
+
Eliminate non-determinism, API cost, and network latency from your CI test suite:
|
|
311
|
+
|
|
312
|
+
```python
|
|
313
|
+
import pytest
|
|
314
|
+
from agentflight.testing import flight_test
|
|
315
|
+
|
|
316
|
+
@flight_test("tests/fixtures/checkout.cassette.json")
|
|
317
|
+
def test_checkout_agent(flight):
|
|
318
|
+
# During first run: executes live and records cryptographic cassette.
|
|
319
|
+
# During subsequent runs (e.g. CI): replays in <3 μs with $0.00 network cost!
|
|
320
|
+
agent = OrderAgent()
|
|
321
|
+
result = agent.execute("Order item sku_881 with coupon SAVE10")
|
|
322
|
+
|
|
323
|
+
assert result.status == "confirmed"
|
|
324
|
+
assert flight.step_count == 3
|
|
325
|
+
assert flight.total_cost_usd <= 0.05
|
|
326
|
+
```
|
|
327
|
+
|
|
328
|
+
---
|
|
329
|
+
|
|
330
|
+
## MCP Tools Reference & Usage Guide
|
|
331
|
+
|
|
332
|
+
AgentFlight provides 4 specialized MCP tools for AI coding agents and LLM tool designers:
|
|
333
|
+
|
|
334
|
+
| Tool Name | What is its use? (Purpose) | Key Arguments | Hosted on Vercel? | Typical Output |
|
|
335
|
+
| :--- | :--- | :--- | :--- | :--- |
|
|
336
|
+
| `agentflight_lint_tools` | Statically analyzes MCP tool definitions against 10 ergonomics rules (`TS001`–`TS010`), flags schema anti-patterns, and computes a 0–100 Agent Ergonomics Score. | `tools` (list of tool schema dicts) | **Yes (Public)** | `{"overall_score": 88.5, "passed": true, "findings": [...]}` |
|
|
337
|
+
| `agentflight_diff_cassettes` | Compares a baseline `.flight.json` cassette against a candidate cassette across model or prompt upgrades to detect tool sequence divergence, new errors, and cost/token drift. | `baseline_path` (str), `candidate_path` (str), `similarity_threshold` (float) | Local stdio | `{"verdict": "PASSED", "similarity_score": 0.95, "cost_delta_usd": 0.00}` |
|
|
338
|
+
| `agentflight_export_otel` | Converts recorded agent trajectories into W3C TraceContext / OpenTelemetry JSON spans for critical-path DAG analysis and SLO monitoring. | `cassette_path` (str), `output_path` (str), `service_name` (str) | Local stdio | `{"spans_count": 14, "trace_id": "...", "exported": true}` |
|
|
339
|
+
| `agentflight_verify_cassette` | Verifies the cryptographic SHA-256 hash chain of a `.flight.json` cassette to guarantee zero tampering or silent step mutations. | `cassette_path` (str) | Local stdio | `{"verified": true, "step_count": 8, "merkle_root": "..."}` |
|
|
340
|
+
|
|
341
|
+
### Tool Usage Examples
|
|
342
|
+
|
|
343
|
+
#### 1. `agentflight_lint_tools` — Auditing MCP Tool Ergonomics Before Deployment
|
|
344
|
+
AI agent frameworks often suffer from confusing tool descriptions, ambiguous parameters, or overlapping tools. Call `agentflight_lint_tools` to validate your schemas before registering them:
|
|
345
|
+
```json
|
|
346
|
+
{
|
|
347
|
+
"jsonrpc": "2.0",
|
|
348
|
+
"id": 1,
|
|
349
|
+
"method": "tools/call",
|
|
350
|
+
"params": {
|
|
351
|
+
"name": "agentflight_lint_tools",
|
|
352
|
+
"arguments": {
|
|
353
|
+
"tools": [
|
|
354
|
+
{
|
|
355
|
+
"name": "search_database",
|
|
356
|
+
"description": "Searches customer database by email address or user ID. Returns matched records.",
|
|
357
|
+
"inputSchema": {
|
|
358
|
+
"type": "object",
|
|
359
|
+
"properties": {
|
|
360
|
+
"query": { "type": "string", "description": "Email or user ID string." }
|
|
361
|
+
},
|
|
362
|
+
"required": ["query"]
|
|
363
|
+
}
|
|
364
|
+
}
|
|
365
|
+
]
|
|
366
|
+
}
|
|
367
|
+
}
|
|
368
|
+
}
|
|
369
|
+
```
|
|
370
|
+
|
|
371
|
+
#### 2. `agentflight_diff_cassettes` — Detecting Agent Behavioral Regressions in CI
|
|
372
|
+
When updating prompts or upgrading from GPT-4o to Claude 3.5 Sonnet, verify that the new agent's tool execution trajectory does not deviate or loop:
|
|
373
|
+
```json
|
|
374
|
+
{
|
|
375
|
+
"jsonrpc": "2.0",
|
|
376
|
+
"id": 2,
|
|
377
|
+
"method": "tools/call",
|
|
378
|
+
"params": {
|
|
379
|
+
"name": "agentflight_diff_cassettes",
|
|
380
|
+
"arguments": {
|
|
381
|
+
"baseline_path": "fixtures/golden_deploy_workflow.flight.json",
|
|
382
|
+
"candidate_path": "test_runs/candidate_deploy.flight.json",
|
|
383
|
+
"similarity_threshold": 0.85,
|
|
384
|
+
"max_cost_increase_pct": 20.0
|
|
385
|
+
}
|
|
386
|
+
}
|
|
387
|
+
}
|
|
388
|
+
```
|
|
389
|
+
|
|
390
|
+
### Exposed MCP Resources (`resources/list`, `resources/read`)
|
|
391
|
+
|
|
392
|
+
| Resource URI | MIME Type | Purpose / Content Description |
|
|
393
|
+
| :--- | :--- | :--- |
|
|
394
|
+
| `agentflight://rules/catalog` | `application/json` | The complete 10-rule ergonomics catalog (`TS001`–`TS010`) detailing antipattern rules, penalty weights, and severities. |
|
|
395
|
+
| `agentflight://metrics/summary` | `application/json` | Verified execution benchmark snapshot (sub-0.02ms replay latency, 1,676 tools/sec throughput, circuit-breaker overhead). |
|
|
396
|
+
|
|
397
|
+
### Exposed MCP Prompts (`prompts/list`, `prompts/get`)
|
|
398
|
+
|
|
399
|
+
| Prompt Name | Purpose / Use Case | Required Arguments | Workflow Output |
|
|
400
|
+
| :--- | :--- | :--- | :--- |
|
|
401
|
+
| `lint-tool-definitions` | Guides an LLM agent to audit, critique, and refactor flawed MCP tool definitions into high-ergonomics schemas. | `tools_json` (string) | Pre-formatted audit prompt instructing the model to check for ambiguous flags, missing descriptions, and collision risks. |
|
|
402
|
+
| `review-agent-trajectory` | Guides an LLM agent through a structured post-mortem of an agent trajectory log to diagnose loops, errors, and cost spikes. | `trajectory_json` (string) | Pre-formatted diagnostic prompt breaking down execution phases, tool calls, and failure boundaries. |
|
|
403
|
+
|
|
404
|
+
---
|
|
405
|
+
|
|
406
|
+
## Architectural Decision Records (ADRs)
|
|
407
|
+
|
|
408
|
+
### ADR-001: Canonical JSON SHA-256 Chaining for Trajectory Cassettes
|
|
409
|
+
- **Context**: Golden test fixtures for multi-step agents frequently suffer from dictionary key-order drift across Python/JSON runtimes and silent fixture edits.
|
|
410
|
+
- **Decision**: Every `TrajectorySpan` computes `input_hash` over `json.dumps(..., sort_keys=True, separators=(",", ":"))` and chains `step_hash = SHA256(step_index || kind || actor || input_payload || output_payload || status || prev_hash)`.
|
|
411
|
+
- **Consequence**: Reordered keyword arguments match deterministically during replay while any mutation to recorded inputs, outputs, or step ordering triggers `CassetteTamperError`.
|
|
412
|
+
|
|
413
|
+
### ADR-002: Zero-Dependency Static Ergonomics Linting (`TS001`–`TS010`)
|
|
414
|
+
- **Context**: Calling an LLM judge to critique MCP tool definitions is slow, non-deterministic in CI, and requires API credentials.
|
|
415
|
+
- **Decision**: Implement a fast, deterministic 10-rule static analyzer combining JSON Schema AST traversal with tokenized bag-of-words cosine similarity (`TS008`).
|
|
416
|
+
- **Consequence**: 200 MCP tool definitions are linted in `~119 ms` (`1,676 tools/sec`) with zero external dependencies.
|
|
417
|
+
|
|
418
|
+
---
|
|
419
|
+
|
|
420
|
+
## Hosted endpoint (Vercel)
|
|
421
|
+
|
|
422
|
+
A free, public, stateless deployment of AgentFlight runs over **MCP Streamable HTTP**:
|
|
423
|
+
|
|
424
|
+
| | |
|
|
425
|
+
|---|---|
|
|
426
|
+
| MCP endpoint | `https://agentflight-tau.vercel.app/mcp` (JSON-RPC 2.0 over `POST`) |
|
|
427
|
+
| Health probe | `https://agentflight-tau.vercel.app/health` |
|
|
428
|
+
|
|
429
|
+
```bash
|
|
430
|
+
curl -s -X POST https://agentflight-tau.vercel.app/mcp -H 'Content-Type: application/json' \
|
|
431
|
+
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
|
|
432
|
+
```
|
|
433
|
+
|
|
434
|
+
### Client Configuration
|
|
435
|
+
|
|
436
|
+
#### 1. Claude Code CLI (Recommended)
|
|
437
|
+
```bash
|
|
438
|
+
# Add to current project:
|
|
439
|
+
claude mcp add --transport http agentflight https://agentflight-tau.vercel.app/mcp
|
|
440
|
+
|
|
441
|
+
# Or add globally across all projects:
|
|
442
|
+
claude mcp add --scope user --transport http agentflight https://agentflight-tau.vercel.app/mcp
|
|
443
|
+
```
|
|
444
|
+
|
|
445
|
+
#### 2. Claude Desktop / Cursor / Windsurf (`mcp.json`)
|
|
446
|
+
```json
|
|
447
|
+
{
|
|
448
|
+
"mcpServers": {
|
|
449
|
+
"agentflight": {
|
|
450
|
+
"type": "http",
|
|
451
|
+
"url": "https://agentflight-tau.vercel.app/mcp"
|
|
452
|
+
}
|
|
453
|
+
}
|
|
454
|
+
}
|
|
455
|
+
```
|
|
456
|
+
|
|
457
|
+
#### 3. Smithery Registry (1-Click Install)
|
|
458
|
+
```bash
|
|
459
|
+
npx -y @smithery/cli install @codesphere-u/agentflight --client claude
|
|
460
|
+
```
|
|
461
|
+
|
|
462
|
+
#### 4. Local Stdio via `uvx` (Full Local File System Access)
|
|
463
|
+
```json
|
|
464
|
+
{
|
|
465
|
+
"mcpServers": {
|
|
466
|
+
"agentflight": {
|
|
467
|
+
"command": "uvx",
|
|
468
|
+
"args": [
|
|
469
|
+
"--from",
|
|
470
|
+
"git+https://github.com/codesphere-u/agentflight.git",
|
|
471
|
+
"agentflight",
|
|
472
|
+
"serve-mcp"
|
|
473
|
+
]
|
|
474
|
+
}
|
|
475
|
+
}
|
|
476
|
+
}
|
|
477
|
+
```
|
|
478
|
+
|
|
479
|
+
**Hosted limitations:** Only `agentflight_lint_tools` is exposed. The cassette tools (`agentflight_diff_cassettes`, `agentflight_export_otel`, `agentflight_verify_cassette`) read/write local files and are **disabled on the hosted endpoint** — use the local stdio install.
|
|
480
|
+
|
|
481
|
+
The hosted gateway is [`api/index.py`](api/index.py); it enforces a 1 MB body limit, strict JSON-RPC envelopes and the tool restrictions above. Deploy your own copy with `npx vercel deploy --prod` (entrypoint is configured in `pyproject.toml` under `[tool.vercel]`).
|
|
482
|
+
|
|
483
|
+
---
|
|
484
|
+
|
|
485
|
+
## Official OpenAny Provider Integration
|
|
486
|
+
|
|
487
|
+
AgentFlight features an official commercial provider integration for [OpenAny](https://openany.si), giving developers access to 128+ AI models and 3,200+ external APIs with pre-execution cost circuit breaking, black-box flight recording, and sub-microsecond deterministic replay.
|
|
488
|
+
|
|
489
|
+
> 🤝 **Partner Attribution & Transparent Disclosure:**
|
|
490
|
+
> AgentFlight integrates natively with OpenAny. Outbound requests to OpenAny carry the verified partner header `X-OpenAny-Partner-Id: agentflight`.
|
|
491
|
+
> *Disclosure: AgentFlight may earn a commission from OpenAny usage through our partnership, at no additional cost to you where confirmed by OpenAny.*
|
|
492
|
+
> AgentFlight remains 100% provider-agnostic, open-source, and free to use with any model provider.
|
|
493
|
+
|
|
494
|
+
### Quick Start with OpenAny
|
|
495
|
+
|
|
496
|
+
```python
|
|
497
|
+
from agentflight import OpenAnyProvider, OpenAnyConfig, TrajectoryCircuitBreaker, FlightRecorder
|
|
498
|
+
|
|
499
|
+
# Initialize provider with spending limits
|
|
500
|
+
provider = OpenAnyProvider(
|
|
501
|
+
config=OpenAnyConfig(
|
|
502
|
+
api_key="your_openany_api_key",
|
|
503
|
+
default_model="MiniMax-M2.5",
|
|
504
|
+
max_budget_usd=0.05,
|
|
505
|
+
)
|
|
506
|
+
)
|
|
507
|
+
|
|
508
|
+
guard = TrajectoryCircuitBreaker(max_cost_usd=0.02)
|
|
509
|
+
recorder = FlightRecorder(agent_id="openany-agent")
|
|
510
|
+
|
|
511
|
+
# Pre-execution budget shield automatically trips BEFORE any paid request if cost exceeds limits
|
|
512
|
+
resp = provider.chat_completion(
|
|
513
|
+
messages=[{"role": "user", "content": "Summarize distributed consensus"}],
|
|
514
|
+
guard=guard,
|
|
515
|
+
recorder=recorder,
|
|
516
|
+
)
|
|
517
|
+
|
|
518
|
+
# Save tamper-evident cassette for $0.00 CI testing
|
|
519
|
+
recorder.save("session.flight.json")
|
|
520
|
+
```
|
|
521
|
+
|
|
522
|
+
### CLI Commands for OpenAny
|
|
523
|
+
|
|
524
|
+
```bash
|
|
525
|
+
# List all 128+ models and real-time per-token prompt/completion pricing
|
|
526
|
+
agentflight openany models
|
|
527
|
+
|
|
528
|
+
# Run a model prompt with budget guardrail and flight recording
|
|
529
|
+
agentflight openany chat --model MiniMax-M2.5 --prompt "Explain Paxos" --budget 0.01 --record paxos.flight.json
|
|
530
|
+
|
|
531
|
+
# Audit OpenAny remote MCP tools against 10 ergonomics rules (TS001-TS010)
|
|
532
|
+
agentflight openany lint-tools
|
|
533
|
+
|
|
534
|
+
# Deterministically replay recorded trajectory in 0.08 μs with zero API spend
|
|
535
|
+
agentflight openany replay --cassette paxos.flight.json
|
|
536
|
+
```
|
|
537
|
+
|
|
538
|
+
For complete technical specifications, see [`docs/OPENANY_INTEGRATION.md`](docs/OPENANY_INTEGRATION.md), [`docs/COMMERCIAL_PARTNERSHIP.md`](docs/COMMERCIAL_PARTNERSHIP.md), and [`docs/LIVE_ATTRIBUTION_RECORD.md`](docs/LIVE_ATTRIBUTION_RECORD.md).
|
|
539
|
+
|
|
540
|
+
---
|
|
541
|
+
|
|
542
|
+
## Authorship, Ownership & License
|
|
543
|
+
|
|
544
|
+
**AgentFlight is independently created and owned by [Akandwanaho Ivan Alvin](https://alvin.codesphereug.com/).**
|
|
545
|
+
It is not owned or contracted by Codesphere Innovations (U) Limited merely because it is hosted under the `codesphere-u` GitHub organization.
|
|
546
|
+
|
|
547
|
+
- **Creator & Owner:** [Akandwanaho Ivan Alvin](https://alvin.codesphereug.com/) ([GitHub `@codesphere-u`](https://github.com/codesphere-u))
|
|
548
|
+
- **Project Status:** Independently owned open-source AI infrastructure
|
|
549
|
+
- **Commercial Partner:** OpenAny
|
|
550
|
+
- **License:** MIT License — Copyright (c) 2026 [Akandwanaho Ivan Alvin (`codesphere-u`)](https://github.com/codesphere-u).
|
|
551
|
+
|
|
552
|
+
If AgentFlight saves your agents from runaway token loops or speeds up your CI suite, please ⭐ **Star this repository** and share it with fellow AI builders!
|