tracetrim 0.3.5__tar.gz

tracetrim-0.3.5/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Alessio Pomaro
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

tracetrim-0.3.5/LICENSES.md

@@ -0,0 +1,23 @@
+# Licence map
+
+| Material | Path | Licence | Notes |
+|---|---|---|---|
+| Library and experiment code | `src/`, `experiments/`, `tests/` | MIT (`LICENSE`) | |
+| Paper text, figures, documentation, protocol | `paper/`, `docs/`, `protocol.md`, README files | CC BY 4.0 | |
+| Aggregate results | `results/` | CC BY 4.0 | counts and measures only, no third-party text |
+| Traces of C3 (literature scan) | `dataset/traces/C3/` (Zenodo) | CC BY 4.0 for our part | tool outputs are OpenAlex records, CC0 |
+| Traces of C4 and C5 (simulated) | `dataset/traces/C4`, `C5` (Zenodo) | CC BY 4.0 | payloads generated by our code |
+| Derived records of C1 and C2 | `dataset/derived/` (Zenodo) | CC BY 4.0 | field paths and token counts only; DataForSEO results are not redistributed |
+
+## Third-party materials used, not redistributed in this repository
+
+| Material | Use | Terms |
+|---|---|---|
+| DataForSEO API (via its MCP server) | tools of C1 and C2 | DataForSEO Terms of Service (read 2026-11-04): aggregates and field names published; raw results not redistributed |
+| OpenAlex API | tools of C3 | data CC0; used without an API key within the free allowance |
+| Stripe OpenAPI description (`stripe/openapi`, commit b213739) | schemas of the C4 simulator | MIT; downloaded at build time, not vendored |
+| GitHub REST API description (`github/rest-api-description`, commit 99b1432) | schemas of the C5 simulator | MIT; downloaded at build time, not vendored |
+| Headroom (`headroom-ai`) | optional backend, token baseline | Apache-2.0; optional dependency, not vendored |
+| TOON format | the library's own TOON-style encoder follows the public specification | specification by J. Schopplich; no code reused |
+
+Tavily was not used: its Platform Terms (§3.2(x)) forbid disclosing performance analyses of the service.

tracetrim-0.3.5/PKG-INFO

@@ -0,0 +1,332 @@
+Metadata-Version: 2.4
+Name: tracetrim
+Version: 0.3.5
+Summary: Learn which fields of tool outputs an agent visibly reuses, from its traces, and trim the rest.
+Author: Alessio Pomaro
+License-Expression: MIT
+Project-URL: Homepage, https://www.alessiopomaro.it/
+Project-URL: Repository, https://github.com/alepom/tracetrim
+Project-URL: Paper, https://doi.org/10.5281/zenodo.23953934
+Project-URL: Software DOI, https://doi.org/10.5281/zenodo.23954125
+Project-URL: Changelog, https://github.com/alepom/tracetrim/blob/master/CHANGELOG.md
+Keywords: agents,mcp,llm,tool outputs,token,telemetry
+Classifier: Development Status :: 4 - Beta
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: LICENSES.md
+Provides-Extra: measure
+Requires-Dist: tiktoken>=0.7; extra == "measure"
+Provides-Extra: test
+Requires-Dist: pytest>=8; extra == "test"
+Requires-Dist: tiktoken>=0.7; extra == "test"
+Provides-Extra: proxy
+Requires-Dist: mcp>=2.3; extra == "proxy"
+Provides-Extra: headroom
+Requires-Dist: headroom-ai>=0.39; extra == "headroom"
+Dynamic: license-file
+
+# tracetrim
+
+**Agents read more than they visibly reuse.** tracetrim learns, from your agent's own traces, which
+fields of each tool output the agent reuses in what it writes, and helps you remove the rest — with a
+telemetry report you can read, a configuration you can review, and an MCP proxy that applies it
+in front of any server without changing your agent.
+
+It comes with a [study](https://doi.org/10.5281/zenodo.23953934), with protocols frozen before every run, of
+where this works and where it fails. In short:
+
+| | |
+|---|---|
+| Tool-output tokens removed on unseen runs (0.1) | real APIs: 63% and 70% (12% on the third agent); simulated: 69% and 77% |
+| …of which re-serialising the JSON alone, removing nothing | up to 13% (real APIs), up to 17% (simulated) |
+| Same conversations: tracetrim 0.1 / 0.2 / Headroom (stateless) / full minifier, no field removed | the same two real-API agents: 59–68% / 46–53% / 17–28% / 16–24%; simulated: 69–77% / 57–61% / 31–33% / 18–33% |
+| Live runs with `expand`: billed input tokens | −56% (real API) and −32% (simulated); every run passed its gates; a later re-check found one unsupported volume among 180 reported keyword–volume pairs |
+| Learning from tool calls only (what a gateway sees) | a used field missed in 49% of the held-out structured calls it trims (156/317, 3 agents on real APIs), against 4% with the answer |
+| Where it fails | values the agent transforms, fields used only to reason, and — without an id field — agents **inventing** ids |
+
+Version 0.2 keeps identifying fields and recognises transformed values by default. Version 0.3
+adds an automatic mode: put one proxy in front of your MCP servers and it observes, learns and
+trims on its own. Read the [paper](https://doi.org/10.5281/zenodo.23953934) before trimming
+anything that matters.
+
+## Install
+
+```bash
+pip install "tracetrim[measure,proxy]"
+pip install "tracetrim[headroom]"   # + Headroom backend (experimental, heavy)
+```
+
+Python 3.10+. From GitHub: `pip install "tracetrim[measure,proxy] @ git+https://github.com/alepom/tracetrim"`.
+From a clone: `pip install -e ".[measure,proxy,test]"`. To reproduce the
+study's numbers, figures and analyses: `pip install -e ".[measure,proxy,test]"` is enough (they need no agent SDK);
+to run new agent sessions, also `pip install -r requirements-study.txt` (the versions used, OpenAI SDK included).
+
+## The short way: automatic mode
+
+Your agent already uses one or more MCP servers. Copy their `mcpServers` block — the same JSON
+Claude Desktop, Claude Code and Cursor use — into a file:
+
+```json
+{"mcpServers": {
+  "billing": {"url": "https://billing.example/mcp", "headers": {"Authorization": "Bearer ${BILLING_TOKEN}"}},
+  "code":    {"command": "python", "args": ["-m", "my_code_server"], "env": {"API_KEY": "${CODE_KEY}"}}}}
+```
+
+Start one tracetrim in front of all of them:
+
+```bash
+tracetrim proxy --servers mcp.json --auto tracetrim-data/                       # stdio, for local clients
+export TRACETRIM_TOKEN=CHOOSE_A_SECRET                                          # over HTTP: clients must send it
+tracetrim proxy --servers mcp.json --auto tracetrim-data/ --http 8799
+```
+
+In the agent, **replace** those servers with tracetrim (one entry instead of N; keeping both would
+show the agent two copies of every tool), and add one line to its instructions:
+
+```text
+Last step, required if the tool tracetrim_done is available: when your final answer is ready,
+call tracetrim_done once with the complete answer, before you show it; then show the same answer.
+```
+
+The agent sees the same tools with the same names (prefixed with the server name only if two
+servers share one), plus `expand` and `tracetrim_done`. Then use the agent as usual:
+
+- **from the first run**, a telemetry report is kept up to date in `tracetrim-data/report.html`
+  (over HTTP also live at `/report?token=...`): which tools fill the context, which fields the agent
+  never uses, with suggestions;
+- **during the first runs** (5 by default, `--min-runs`) nothing is changed: the proxy only observes.
+  `tracetrim_done` is how it sees the answer — in the study, learning from tool calls alone missed a
+  used field in 49% of held-out calls (three agents on real APIs), against 4% with the answer;
+- **after that**, it applies the report's automatic actions on its own: the prudent projection, with
+  identifying fields kept, transformed values recognised and `expand` on. Everything else stays in the
+  report as a manual action for you to review;
+- **it corrects itself**: a tool whose trimmed outputs the agent expands too often is no longer
+  trimmed, and the report says so. Runs without `tracetrim_done` count for telemetry, not for learning.
+  One agent at a time: calls that arrive together from several agents end up in the same run. Over MCP the
+  proxy cannot see the model's turns, so calls emitted together are compared by arrival order: a later call's
+  arguments can count as using an earlier sibling's output. In the study this changed no figure; where turns
+  matter, record them (`AutoTrim` takes `turn`, JSONL traces accept a `"turn"` field).
+
+Everything lives in the work directory (traces, report, config, state), so learning survives restarts;
+delete it to start again. In a check with a real agent on the billing case (5 runs, 3 to learn), the
+tool outputs of the two runs after learning went from 33,492 to 18,725 tokens (−44%, fields asked back
+included), the agent asked for a hidden field once in each, and all five tickets passed the keyword test in every
+run: a check, not a measurement (`results/checks/auto_check_mcp.json`) — the measurements are in the
+paper.
+
+### Servers that need credentials
+
+Two separate links, two separate credentials: **agent → tracetrim** is protected by the token you
+choose, in `$TRACETRIM_TOKEN` (or `--token`, which other users of the machine can see in the process
+list), only when tracetrim is reachable over HTTP; requests whose Host is not local or listed with
+`--allowed-host` are refused. **tracetrim → your servers**
+uses your servers' own credentials, which tracetrim holds and the agent no longer sees.
+
+| how your server authenticates | in `mcp.json` |
+|---|---|
+| API key or bearer token in a header | `"headers": {"Authorization": "Bearer ${MY_TOKEN}"}` (any header name) |
+| local server reading a key from the environment | `"env": {"API_KEY": "${MY_KEY}"}` |
+| OAuth, with a browser login (as in Claude Code or Cursor) | just `"url"`: no headers needed |
+
+`${VAR}` is read from the environment when the proxy starts, so keys stay out of the file. For
+OAuth servers, log in once:
+
+```bash
+tracetrim login --servers mcp.json     # opens the login page of each OAuth server
+```
+
+Discovery, client registration, PKCE and token renewal follow the MCP authorization spec (through
+the MCP SDK); tokens are kept in `~/.config/tracetrim/oauth/`, readable only by you, and renewed on
+their own, so the proxy never waits for a browser. `tracetrim logout --servers mcp.json` forgets them.
+Not supported: a different identity per end user (the proxy uses one identity for everyone) and the
+legacy SSE transport.
+
+### Hosted agents (OpenAI Playground, Responses API)
+
+There is no `mcpServers` file in the Playground: each MCP server is added in the UI (*Tools → Add →
+MCP server*) with a URL and, if needed, a header. Write `mcp.json` from those same values — one
+entry per server, URL and header as you typed them — and run tracetrim over HTTP where OpenAI can
+reach it (a server of yours, or a tunnel such as ngrok):
+
+```bash
+export TRACETRIM_TOKEN=CHOOSE_A_SECRET
+tracetrim proxy --servers mcp.json --auto tracetrim-data/ --http 8799 --allowed-host your-tunnel.example
+```
+
+Then, in the Playground, switch off the servers tracetrim now stands in for and add one MCP server:
+URL `https://your-tunnel.example/mcp`, header `Authorization: Bearer CHOOSE_A_SECRET`. The live
+report is at `http://127.0.0.1:8799/report?token=CHOOSE_A_SECRET` on the machine running the proxy.
+`docs/demo_playground.sh` does all of this for the demo.
+
+### Function calling, without MCP
+
+If your agent runs its own tools (function calling), there is no proxy to put in between: the
+loop that runs the tools is yours, so tracetrim goes in there, in three places. No
+`tracetrim_done` either: your code already has the final answer.
+
+```python
+from tracetrim.auto import AutoTrim
+
+tt = AutoTrim("tracetrim-data/", min_runs=5)
+tools = my_tools + [{"type": "function", **tt.expand_tool}]      # 1. offer expand
+
+# 2. in the loop, for every tool call the model makes
+if call.name == "expand":
+    result = tt.expand(call.arguments)
+else:
+    result = tt.output(call.name, call.arguments, run_my_tool(call.name, call.arguments))
+
+# 3. when the run is over
+tt.done(final_answer)
+```
+
+`tt.output` records the call and returns what the model should receive: the raw result during the
+first runs, the trimmed one after. `tt.done` closes the run, updates the report
+(`tt.report`, `tracetrim-data/report.html`) and, once `min_runs` runs exist, starts trimming; what
+was learned survives restarts. The rules are those of the proxy's automatic mode. Arguments can be
+a dict or the JSON string most SDKs hand you; `tt.expand_tool` is a name, a description and a JSON
+schema, to adapt to your provider's tool format (`{"type": "function", ...}` for OpenAI, `name` /
+`description` / `input_schema` for Anthropic).
+
+Checked with a real agent on the billing case, function calling, 5 runs with 3 to learn: after learning,
+tool outputs went from 33,862 to 18,947 tokens (−44%, fields asked back included), the agent asked for a
+hidden field once, and all five tickets passed the keyword test in every run (a check, not a measurement:
+`results/checks/auto_check_function.json`, from
+`python -m experiments.auto_check --runs 5 --min-runs 3 --mode function`).
+
+**Mind the work directory.** To learn, tracetrim keeps the full tool outputs of every run in
+`tracetrim-data/traces/`. If your servers return confidential data, treat that directory as you
+treat the data: local, not shared, not committed.
+
+## Quickstart: five runs, one report, one config
+
+The manual way, with a review step before anything is applied.
+
+**1. Record a few runs with full tool outputs.** A trace is JSONL, one event per line:
+
+```json
+{"kind": "prompt", "text": "You are a billing support agent..."}
+{"kind": "call", "tool": "stripe_list_charges", "arguments": {"customer": "cus_123"}, "output": "{\"object\": \"list\", ...}"}
+{"kind": "message", "text": "{\"ticket\": [...]}"}
+```
+
+`output` is the raw text the tool returned; the last `message` is the agent's answer. The
+answer matters: without it the learner cannot see what the agent used. An optional `"turn"` on
+every event (the model turn that produced it; the prompt is `-1`) lets the learner tell calls
+emitted together apart: their arguments were written before any of their outputs existed. OpenAI
+Playground Python exports are also read directly (`.py`).
+
+**2. Generate the report.** It changes nothing; it is useful on its own.
+
+```bash
+tracetrim report traces/*.jsonl -o report.html
+```
+
+You get `report.html` (bilingual EN/IT: which tools fill the context, which calls fail and with
+which provider error, which fields the agent never uses, with suggestions), `report.json`, and a
+runtime config with the projection it uses: `report.config.json`, `report.projection.json`.
+
+**3. Review the config.** It is plain JSON: which tools are trimmed, with `expand` on. The
+projection lists, per tool, the fields kept and the fields dropped. Remove a tool from the config,
+or a field from the dropped list, if you know the task needs it.
+
+**4. Apply it.**
+
+In your own code:
+
+```python
+from tracetrim.runtime import Policy
+policy = Policy.load("report.config.json")
+tool_message = policy.apply(tool_name, raw_output)
+```
+
+`Policy.apply` only trims: it adds no `[trimmed]` line and there is no `expand` behind it. For the way
+back, use `AutoTrim` (function calling, below) or the proxy.
+
+Or in front of any MCP server, with no code change — locally (Claude Desktop, Claude Code,
+Cursor):
+
+```json
+{"mcpServers": {"billing": {"command": "tracetrim", "args": [
+  "proxy", "--upstream-cmd", "python -m my_billing_server", "--config", "/path/report.config.json"]}}}
+```
+
+or over HTTP, for hosted clients such as the OpenAI Playground:
+
+```bash
+export TRACETRIM_TOKEN=CHOOSE_A_SECRET
+tracetrim proxy --upstream-url https://mcp.example.com/mcp --header "Authorization=Bearer UPSTREAM_TOKEN" \
+  --config report.config.json --http 8799 --allowed-host your-tunnel.example
+```
+
+(`--header` puts the upstream credential in the process list; `--servers mcp.json` with `${VAR}` does not.)
+
+The proxy exposes the same tools plus `expand`: every trimmed output ends with a line listing the
+hidden fields, and the agent can fetch any of them back (`"*"` for all of them) while the proxy keeps the original:
+the last 256 trimmed results; at most 25 groups are listed, the rest summarised. `--record traces/live.jsonl`
+records calls and outputs for telemetry; it does not record the agent's answer, so a report learned from it misses
+most of what the agent uses (the study's central finding). To learn while running, use `--auto`, which receives
+the answer through `tracetrim_done`.
+
+How it wraps another server: the proxy is an MCP *server* towards your agent and an MCP *client*
+towards the upstream. At start it asks the upstream for its tools and publishes the same names and
+input schemas as its own (plus `expand`); each call is forwarded unchanged, and only the result is
+rewritten on its way back. One proxy can stand in front of several servers (`--servers`).
+
+While it runs, the proxy prints one line per call on stderr — tokens from the tool, tokens to the
+agent, hidden fields, `expand` requests and the session total (`--quiet` to silence it). Add
+`--structural toon` or `--structural headroom` to re-encode outputs after the projection
+(experimental; Headroom needs `tracetrim[headroom]`).
+
+**5. Measure before you trust it.**
+
+```bash
+tracetrim measure traces/*.jsonl --per-tool
+```
+
+Leave-one-out: each run is trimmed with a projection learned from the others; you see the tokens
+removed and the *misses* (held-out calls in which a field the agent used would have been removed).
+
+## What it does not do
+
+- It does not learn from a proxy alone: learn offline from complete traces, apply anywhere.
+- It does not see reasoning: a field read only to decide something looks unused. Identifiers and
+  titles are kept for this reason; other such fields need your review.
+- It does not translate: a value the agent rewrites in another language (or form) that the
+  normalisers do not cover looks unused.
+- Structural re-encoding (`"structural": "toon"` or `"headroom"` in the config) is optional and was
+  seen to change how a model resolves ambiguous data. Validate on your task first.
+
+## Reproducing the study
+
+`experiments/` contains the agent loop, the simulated MCP servers (Stripe- and GitHub-shaped,
+generated from their public OpenAPI descriptions), the gates and thermometers, and every
+analysis. `protocol.md` is the frozen protocol with all deviations. Every number of the paper is
+generated by `python -m experiments.paper_numbers` from `results/`, and every chart by
+`python -m experiments.paper_figures` from `results/` and the traces. Run them from the repository root
+with `PYTHONPATH=src`. The traces are in the Zenodo dataset: unzip it at the root, and the scripts read
+`dataset/traces/` (C3, C4 and C5; C1 and C2 are released as derived records only, since the search
+data cannot be redistributed, so their analyses run where they were recorded and their results are in
+`results/`; `python -m experiments.misses_from_derived` recomputes their misses from the records). The simulated
+servers download their pinned OpenAPI descriptions on first use. Changes by version are in
+[CHANGELOG.md](https://github.com/alepom/tracetrim/blob/master/CHANGELOG.md).
+
+## Citation
+
+Paper: Alessio Pomaro, *Agents Read More Than They Use: Learning Tool-Output Projections from Execution Traces,
+and Where They Fail*, preprint, 2026. DOI [10.5281/zenodo.23953934](https://doi.org/10.5281/zenodo.23953934).
+
+Software, protocols and dataset: DOI [10.5281/zenodo.23954125](https://doi.org/10.5281/zenodo.23954125).
+See `CITATION.cff`.
+
+## License
+
+MIT for the code. Third-party materials and their terms: see `LICENSES.md`.