ami-survey 1.0.0__tar.gz

ami_survey-1.0.0/.gitignore

@@ -0,0 +1,3 @@
+__pycache__/
+*.pyc
+.DS_Store

ami_survey-1.0.0/LICENSE

@@ -0,0 +1,33 @@
+Copyright (c) 2026 Fred Viner. All rights reserved.
+
+AMI EVALUATION LICENCE 1.0
+
+Permission is granted to any person obtaining a copy of this software to
+install and run it on machines they control, for the sole purpose of
+evaluating it and submitting survey responses to a survey service operated
+by the copyright holder.
+
+Permission is also granted to redistribute this software verbatim: complete,
+unmodified, and with this licence included. This is what allows the software
+to be published on a package index and carried by that index's mirrors and
+caches, which copy what they host without being asked.
+
+No other permission is granted. In particular this licence does not grant the
+right to sublicense or sell the software; to modify it other than as needed to
+run it for the purpose above; or to create derivative works from it. All rights
+not expressly granted here are reserved.
+
+This licence covers only the files in this repository. It grants no rights in
+the survey service the software submits to, in the data collected by that
+service, or in any trade mark or name used by either.
+
+Permission ends if the copyright holder gives notice that it has ended, or if
+these terms are breached. Redistribution of verbatim copies already made
+remains permitted.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

ami_survey-1.0.0/PKG-INFO

@@ -0,0 +1,156 @@
+Metadata-Version: 2.5
+Name: ami-survey
+Version: 1.0.0
+Summary: Measure what an agent workflow actually cost, from the runtime's own session log.
+Project-URL: Homepage, https://agentbenchmark.dev
+Project-URL: Source, https://github.com/speedofred/ami-survey-client-v1
+Author: Fred Viner
+License-Expression: LicenseRef-AMI-Evaluation-1.0
+License-File: LICENSE
+Keywords: agent,benchmark,claude,codex,mcp,telemetry
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Information Technology
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: System :: Benchmark
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+
+# AMI — what did that workflow actually cost?
+
+You can find out what a single API call costs. Almost nobody can say what one
+finished piece of work costs — one triaged ticket, one screened CV, one drafted
+reply — across every call, retry and tool round-trip the agent made getting
+there.
+
+This measures it, by reading your runtime's own session log after the fact. Ask
+your agent to run it when it finishes something, and you get a scorecard back.
+
+Every number comes from the log, not from the agent. An agent asked how many
+tokens it just used will guess, and guess confidently.
+
+## What comes back
+
+A real run — six support tickets triaged and answered by Claude Opus 5 in Claude
+Code:
+
+```
+Maturity Index      85.0  Strong          (observability 40%, evidence 30%, quality 30%)
+Performance         78.13 Strong          confidence Very High
+
+  quality           80.0   graded Good on ami-quality-v2
+  cost              72.73  $0.123226 per ticket   ($0.739355 for the run)
+  speed             84.47  20.90s per ticket
+  evidence          70.0   measured, on a self-issued token
+  observability    100.0
+
+findings
+  weakness  Cost is the weakest pillar at 72.73; speed is strongest at 84.47.
+            $0.123226 per unit against a $0.01 reference. A cheaper model, or
+            fewer calls, moves this; check calls[] for where the tokens went.
+
+  note      Cost and speed were scored against a provisional reference, which is
+            a placeholder rather than a measurement. Do not quote them as settled
+            yet. The Maturity Index does not use the reference and is unaffected.
+```
+
+**$0.12 per ticket, 21 seconds per ticket.** That is the number this exists to
+produce, and it is the one most teams cannot currently state about their own
+work.
+
+The findings are worth reading twice: the scorecard says out loud where its own
+numbers are soft. A cost reference that is still a placeholder is a placeholder
+in your report too, not quietly folded into a score.
+
+## Install
+
+Two ways in. The difference between them is whether anything can read your
+runtime's logs, and that decides whether your numbers are **measured** or
+**unmeasured**.
+
+### Measured — one line
+
+Add this to your agent's MCP configuration and restart it:
+
+```json
+{ "mcpServers": { "ami-survey": { "command": "uvx", "args": ["ami-survey"] } } }
+```
+
+Then ask your agent, after it finishes a piece of work:
+
+> Take the AMI survey regarding the ticket triage you just did
+
+That is the whole setup. Nothing to clone, nothing to keep updated, and no token
+to paste — the first call that needs one registers this machine and stores it at
+`~/.ami-survey/token`.
+
+**`uvx` comes from [uv](https://docs.astral.sh/uv/)** — the same tool the MCP
+docs use for Python servers, so if you have installed one before you already
+have it. If you would rather not, [GETTING-STARTED.md](https://github.com/speedofred/ami-survey-client-v1/blob/main/GETTING-STARTED.md)
+has a `pipx` form and a route that needs neither.
+
+Already have a token? Put it in that block's `env` as `AMI_API_TOKEN` and it is
+used instead of registering a new one.
+
+### Unmeasured — a remote connector, nothing installed
+
+In claude.ai: Settings → Connectors → Add custom connector, and give it
+
+```
+https://survey.agentbenchmark.dev/mcp
+```
+
+Nothing to install and no token. These runs are recorded as `unmeasured` and are
+never compared against measured ones — a server on the other side of the
+internet cannot read your runtime's logs, so the token counts and cost are
+simply absent rather than guessed.
+
+Install the client above when you want those numbers too.
+
+## Licence, up front
+
+**This is not open source.** It is an evaluation licence: run it on machines you
+control, redistribute it verbatim if you like, but it may not be modified, sold
+or built upon. Full terms in
+[LICENSE](https://github.com/speedofred/ami-survey-client-v1/blob/main/LICENSE).
+
+Said here rather than at the bottom, because finding it at the bottom after
+reading everything else is worse than being told now.
+
+## What leaves your computer
+
+Token counts, timings, model names, the stage names your workflow declared, and
+the grade. **Not your files, not your prompts, not your shell commands.**
+[GETTING-STARTED.md](https://github.com/speedofred/ami-survey-client-v1/blob/main/GETTING-STARTED.md)
+sets this out in full.
+
+Submissions go to `survey.agentbenchmark.dev` and nowhere else. That destination
+is a constant in the source rather than a setting: a stale environment variable
+cannot redirect your submission onto your own disk, which is the one failure that
+would make a run look successful while collecting nothing.
+
+## Requirements
+
+Python 3.9 or newer. No dependencies — the standard library only.
+
+## Everything else
+
+- [GETTING-STARTED.md](https://github.com/speedofred/ami-survey-client-v1/blob/main/GETTING-STARTED.md)
+  — assumes no prior setup; macOS, Linux and Windows, and what to do when it does
+  not work.
+- [COMMANDS.md](https://github.com/speedofred/ami-survey-client-v1/blob/main/COMMANDS.md)
+  — the clone-and-run route: benchmarking one workflow across several models on
+  your own API key. Not needed to take part.
+- [MAKE-IT-MEASURABLE.md](https://github.com/speedofred/ami-survey-client-v1/blob/main/MAKE-IT-MEASURABLE.md)
+  — how to structure a workflow so there is something worth measuring.

ami_survey-1.0.0/README.md

@@ -0,0 +1,127 @@
+# AMI — what did that workflow actually cost?
+
+You can find out what a single API call costs. Almost nobody can say what one
+finished piece of work costs — one triaged ticket, one screened CV, one drafted
+reply — across every call, retry and tool round-trip the agent made getting
+there.
+
+This measures it, by reading your runtime's own session log after the fact. Ask
+your agent to run it when it finishes something, and you get a scorecard back.
+
+Every number comes from the log, not from the agent. An agent asked how many
+tokens it just used will guess, and guess confidently.
+
+## What comes back
+
+A real run — six support tickets triaged and answered by Claude Opus 5 in Claude
+Code:
+
+```
+Maturity Index      85.0  Strong          (observability 40%, evidence 30%, quality 30%)
+Performance         78.13 Strong          confidence Very High
+
+  quality           80.0   graded Good on ami-quality-v2
+  cost              72.73  $0.123226 per ticket   ($0.739355 for the run)
+  speed             84.47  20.90s per ticket
+  evidence          70.0   measured, on a self-issued token
+  observability    100.0
+
+findings
+  weakness  Cost is the weakest pillar at 72.73; speed is strongest at 84.47.
+            $0.123226 per unit against a $0.01 reference. A cheaper model, or
+            fewer calls, moves this; check calls[] for where the tokens went.
+
+  note      Cost and speed were scored against a provisional reference, which is
+            a placeholder rather than a measurement. Do not quote them as settled
+            yet. The Maturity Index does not use the reference and is unaffected.
+```
+
+**$0.12 per ticket, 21 seconds per ticket.** That is the number this exists to
+produce, and it is the one most teams cannot currently state about their own
+work.
+
+The findings are worth reading twice: the scorecard says out loud where its own
+numbers are soft. A cost reference that is still a placeholder is a placeholder
+in your report too, not quietly folded into a score.
+
+## Install
+
+Two ways in. The difference between them is whether anything can read your
+runtime's logs, and that decides whether your numbers are **measured** or
+**unmeasured**.
+
+### Measured — one line
+
+Add this to your agent's MCP configuration and restart it:
+
+```json
+{ "mcpServers": { "ami-survey": { "command": "uvx", "args": ["ami-survey"] } } }
+```
+
+Then ask your agent, after it finishes a piece of work:
+
+> Take the AMI survey regarding the ticket triage you just did
+
+That is the whole setup. Nothing to clone, nothing to keep updated, and no token
+to paste — the first call that needs one registers this machine and stores it at
+`~/.ami-survey/token`.
+
+**`uvx` comes from [uv](https://docs.astral.sh/uv/)** — the same tool the MCP
+docs use for Python servers, so if you have installed one before you already
+have it. If you would rather not, [GETTING-STARTED.md](https://github.com/speedofred/ami-survey-client-v1/blob/main/GETTING-STARTED.md)
+has a `pipx` form and a route that needs neither.
+
+Already have a token? Put it in that block's `env` as `AMI_API_TOKEN` and it is
+used instead of registering a new one.
+
+### Unmeasured — a remote connector, nothing installed
+
+In claude.ai: Settings → Connectors → Add custom connector, and give it
+
+```
+https://survey.agentbenchmark.dev/mcp
+```
+
+Nothing to install and no token. These runs are recorded as `unmeasured` and are
+never compared against measured ones — a server on the other side of the
+internet cannot read your runtime's logs, so the token counts and cost are
+simply absent rather than guessed.
+
+Install the client above when you want those numbers too.
+
+## Licence, up front
+
+**This is not open source.** It is an evaluation licence: run it on machines you
+control, redistribute it verbatim if you like, but it may not be modified, sold
+or built upon. Full terms in
+[LICENSE](https://github.com/speedofred/ami-survey-client-v1/blob/main/LICENSE).
+
+Said here rather than at the bottom, because finding it at the bottom after
+reading everything else is worse than being told now.
+
+## What leaves your computer
+
+Token counts, timings, model names, the stage names your workflow declared, and
+the grade. **Not your files, not your prompts, not your shell commands.**
+[GETTING-STARTED.md](https://github.com/speedofred/ami-survey-client-v1/blob/main/GETTING-STARTED.md)
+sets this out in full.
+
+Submissions go to `survey.agentbenchmark.dev` and nowhere else. That destination
+is a constant in the source rather than a setting: a stale environment variable
+cannot redirect your submission onto your own disk, which is the one failure that
+would make a run look successful while collecting nothing.
+
+## Requirements
+
+Python 3.9 or newer. No dependencies — the standard library only.
+
+## Everything else
+
+- [GETTING-STARTED.md](https://github.com/speedofred/ami-survey-client-v1/blob/main/GETTING-STARTED.md)
+  — assumes no prior setup; macOS, Linux and Windows, and what to do when it does
+  not work.
+- [COMMANDS.md](https://github.com/speedofred/ami-survey-client-v1/blob/main/COMMANDS.md)
+  — the clone-and-run route: benchmarking one workflow across several models on
+  your own API key. Not needed to take part.
+- [MAKE-IT-MEASURABLE.md](https://github.com/speedofred/ami-survey-client-v1/blob/main/MAKE-IT-MEASURABLE.md)
+  — how to structure a workflow so there is something worth measuring.

ami_survey-1.0.0/ami-survey/ami_survey/__init__.py

@@ -0,0 +1,7 @@
+"""AMI workflow survey: concrete, evidence-backed collection of the fields in
+Collection_Inventory.csv for any agent workflow run."""
+
+#: The one place the version is written. `pyproject.toml` reads it from
+#: here via hatchling, and the MCP handshake reports it, so a release
+#: cannot ship a package and a server claiming different numbers.
+__version__ = "1.0.0"

ami_survey-1.0.0/ami-survey/ami_survey/adapters/__init__.py

@@ -0,0 +1,151 @@
+"""Runtime telemetry adapters, and the registry that picks one.
+
+An adapter turns a runtime's own record of its API calls into the normalised call
+records the survey API ingests. The survey is model-agnostic - the model id and
+its price are read from whatever the runtime recorded - but *measurement* is
+runtime-specific, because each agent harness logs its sessions differently. So
+there is one adapter per harness, not per model, and the agent taking the survey
+never has to say which one it is: `detect()` works that out.
+
+Normalised call record:
+    call_id, model, start_time, end_time, duration_seconds,
+    input_tokens, output_tokens, input_token_breakdown{...},
+    tool_calls[{name, command?}], has_text, has_thinking, source, evidence{...}
+
+Each adapter module exposes:
+    NAME              short id, e.g. "claude_code"
+    LABEL             what goes in `telemetry_adapter`, e.g. "claude_code_transcript"
+    locate(cwd, session_id=None, transcript_path=None) -> Path
+                      the session log this run should be measured from; raises
+                      TelemetryNotFound if this runtime is not the one in use
+    probe(...)        runtime identity + proposed measurement window
+    collect(...)      the full telemetry payload
+
+A runtime with no adapter is not stuck: it can post its own usage records
+directly via `ami_record_calls` / POST /runs/{id}/calls. It just has to have them.
+"""
+
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+
+class TelemetryNotFound(RuntimeError):
+    """No session log for this runtime - it is probably not the one running."""
+
+
+def _modules() -> list:
+    from . import claude_code, codex
+
+    return [claude_code, codex]
+
+
+def available() -> list[str]:
+    return [m.NAME for m in _modules()]
+
+
+def _located(cwd: str | None, session_id: str | None, transcript_path: str | None):
+    """Every adapter that can find a session for this run, newest log first."""
+    found = []
+    for module in _modules():
+        # An explicitly supplied log belongs to exactly one runtime; adapters that
+        # can recognise their own format get to disown it rather than mis-parse it.
+        if transcript_path and hasattr(module, "claims"):
+            if not module.claims(transcript_path):
+                continue
+        try:
+            path = module.locate(
+                cwd=cwd, session_id=session_id, transcript_path=transcript_path
+            )
+        except (TelemetryNotFound, OSError):
+            continue
+        try:
+            mtime = Path(path).stat().st_mtime
+        except OSError:
+            mtime = 0.0
+        found.append((mtime, module, path))
+    return sorted(found, key=lambda f: f[0], reverse=True)
+
+
+def _environment_candidates() -> list:
+    """Narrow the field using the environment the runtime gave this process.
+
+    Agent harnesses launch the MCP server as a child process and stamp their own
+    variables on it, which identifies the asking runtime exactly. That beats every
+    heuristic: two runtimes can have sessions in one directory, and the most
+    recently written log is not always the one asking. Adapters that cannot tell
+    either way stay in the running.
+    """
+    verdicts = [(m, m.runs_here() if hasattr(m, "runs_here") else None) for m in _modules()]
+    positive = [m for m, v in verdicts if v is True]
+    if positive:
+        return positive
+    undecided = [m for m, v in verdicts if v is not False]
+    return undecided or [m for m, _ in verdicts]
+
+
+def detect(
+    cwd: str | None = None,
+    session_id: str | None = None,
+    transcript_path: str | None = None,
+    adapter: str | None = None,
+):
+    """Return the adapter module that should measure this run.
+
+    Explicit choice wins (argument, then AMI_ADAPTER), then the runtime named by
+    this process's environment. Failing both, every remaining adapter is asked
+    whether it can find a session for this working directory and the most
+    recently written log wins.
+    """
+    wanted = adapter or os.environ.get("AMI_ADAPTER")
+    if wanted:
+        for module in _modules():
+            if module.NAME == wanted:
+                return module
+        raise TelemetryNotFound(
+            f"Unknown adapter {wanted!r}. Available: {', '.join(available())}"
+        )
+
+    found = _located(cwd, session_id, transcript_path)
+    by_environment = {m.NAME for m in _environment_candidates()}
+    narrowed = [f for f in found if f[1].NAME in by_environment]
+    found = narrowed or found
+    if not found:
+        raise TelemetryNotFound(
+            "No session log found for any known runtime "
+            f"({', '.join(available())}) in {cwd or os.getcwd()!r}. If that is not "
+            "your workspace directory - the survey server may have been launched "
+            "elsewhere - call this again with cwd set to the directory you are "
+            "working in. If your runtime has no adapter, post its own usage records "
+            "with ami_record_calls instead: the survey does not accept estimates."
+        )
+    return found[0][1]
+
+
+def probe(
+    cwd: str | None = None,
+    session_id: str | None = None,
+    transcript_path: str | None = None,
+    adapter: str | None = None,
+) -> dict:
+    module = detect(cwd, session_id, transcript_path, adapter)
+    return module.probe(cwd=cwd, session_id=session_id, transcript_path=transcript_path)
+
+
+def collect(
+    cwd: str | None = None,
+    session_id: str | None = None,
+    transcript_path: str | None = None,
+    window_start: str | None = None,
+    window_end: str | None = None,
+    adapter: str | None = None,
+) -> dict:
+    module = detect(cwd, session_id, transcript_path, adapter)
+    return module.collect(
+        cwd=cwd,
+        session_id=session_id,
+        transcript_path=transcript_path,
+        window_start=window_start,
+        window_end=window_end,
+    )