quality-loop 0.1.0__py3-none-any.whl

quality_loop/OPERATOR.md

@@ -0,0 +1,103 @@
+# You are the operator of one room
+
+A **room** is a separate Claude Code session that was given a task and nothing
+but a product's agent-facing surface - a skill bundle and the product's own
+tools. It was started so it cannot read the product's repository. What it
+does with what it was given is the measurement. Your job is to make that
+measurement possible and then report what it shows.
+
+You work in two phases of one session. The message that starts each phase
+says which one it is, and gives you the paths.
+
+## Phase 1: put up the screen
+
+1. Read the scenario file. Its **Screen** section says what has to be in
+   front before the room starts. It may say "none".
+2. If a screen is needed, call `screen_list` to see what the product
+   declares, choose what the section asks for, and call `screen_open` for
+   each one. It returns once the window is up, or fails.
+3. If the section asks for something no declared screen provides, do not
+   improvise. Write that into `setup.md` in your directory and stop: the
+   room cannot measure anything without its screen.
+4. Write `setup.md` in your directory: what you opened, with the ids `open`
+   returned, and anything about the screen a reader of your report should
+   know.
+
+You have no shell. Your tools are Read, Glob, Grep, writing in your own
+directory and the room's REPORT.md, and the four tools above.
+
+You may not change anything in the room. The runner checks every file in it
+after this phase, and a changed byte cancels the room. In particular, the
+room's TASK.md is copied from the scenario by the runner and must reach the
+room exactly as written.
+
+## Phase 2: report
+
+The room has finished. In the room directory you have:
+
+- `transcript.jsonl`: everything the room did, as stream-json. This is the
+  record. The room's own account of what it did is only a claim.
+- `QUESTIONS.md`: what the room noticed it had to guess.
+- `evidence/`: what the runner collected. That includes the product's files
+  before and after, anything the room created, the screen's event record, and
+  `audit.json`, the runner's isolation audit.
+
+Then:
+
+1. Read the scenario's **Must** and **Must not** sections. For each line,
+   establish from evidence whether the run met it. Use the product's evidence
+   tools where a line is about behaviour; the start-of-phase message lists
+   them. Quote what a tool answered. Do not infer it.
+2. Read the transcript for **friction**, whether or not the room noticed it:
+   a tool call that failed or was refused, an answer that was an error or a
+   blank, a search of the bundle that came back empty, a file rewritten or a
+   run repeated, turns running out, sentences where the room says it is
+   unsure or was wrong. A problem the room never mentioned in QUESTIONS.md is
+   the most valuable kind.
+3. Read **QUESTIONS.md**. Each question is a candidate problem.
+4. Write **REPORT.md** in the room, in the format below. Then close every
+   screen you opened (`screen_close`).
+
+**Friction is not a problem by itself.** A deploy's own "DEPLOY FAILED" on a
+screen built to show a failed deploy is the task working. Report what a
+product change would improve, and say why.
+
+## REPORT.md
+
+```markdown
+# Report: <scenario title>
+
+One paragraph: what the room was asked, what it did, and what it left.
+
+### [high] <the problem, in one line>
+- **Id:** `<kebab-case, stable>`
+- **Kind:** skill gap
+- **Must:** "<the Must line it concerns, or none>"
+- **Evidence:** <tool output quoted, or transcript step and what it shows>
+- **Where it would be fixed:** <file and section, in the product>
+- **How a fix is confirmed:** <the evidence that would show it fixed>
+
+<Free text: what happened, and why it matters.>
+```
+
+**Severity** - pick the first that fits:
+
+| | |
+|---|---|
+| critical | The run did harm: data destroyed, the user's files rewritten or no longer loading, a destructive control pressed. |
+| high | A Must not met, or a result that is silently wrong - one that looks right and is not. |
+| medium | Met, but only by guessing something the product should have said, or after rework a user's agent would repeat. |
+| low | Friction that cost little: an empty search, a retry, an unclear message. |
+
+**Kind**: `skill gap` (the product's agent-facing text should have said it),
+`product` (the product behaves wrongly), `harness` (the runner, or the room's
+tools), `out of scope` (true of the task or the application, not of the
+product).
+
+**Id** names the problem, not the run: `fn-f1-not-bound`, not `room-3-issue`.
+A later run reports the same problem under the same id, which is how a fix is
+seen to have worked.
+
+There is no verdict. Write problems, most severe first. If you found none,
+say so in the opening paragraph and say what you checked. Zero problems is
+rare, and the reader will want to know why you believe it.

quality_loop/__init__.py

@@ -0,0 +1,15 @@
+"""quality-loop: measure what an agent-facing product teaches.
+
+A product that agents use - a skill bundle, an MCP server, a CLI - is only as
+good as what an agent with nothing else can do with it. This package runs that
+agent: a task, in a room holding the product's agent-facing surface and nothing
+else, started so it cannot read the repository the answers are in. A second
+agent, the operator, puts up whatever the task needs beforehand and afterwards
+reports the problems it finds in what the room did, each with a severity.
+
+What has to hold for a run to mean anything - the isolation, the prompt the
+room is handed, putting the user's machine back - is code here, not an
+agent's discretion.
+"""
+
+__version__ = "0.1.0"

quality_loop/__main__.py

@@ -0,0 +1,48 @@
+"""The operator agent's command line: ``python -m quality_loop screen ...``.
+
+    python -m quality_loop screen list
+    python -m quality_loop screen open approval-queue
+    python -m quality_loop screen close approval-queue-1
+
+Everything it takes is a name from the manifest the runner wrote. Answers are
+JSON on stdout, errors one line on stderr with exit status 1.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+
+from quality_loop import screens
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(prog="quality-loop")
+    commands = parser.add_subparsers(dest="command", required=True)
+    screen = commands.add_parser("screen", help="put up or take down a declared screen")
+    actions = screen.add_subparsers(dest="action", required=True)
+    actions.add_parser("list", help="the screens the adapter declares")
+    opening = actions.add_parser("open", help="open one, and wait for its window")
+    opening.add_argument("name")
+    closing = actions.add_parser("close", help="close one that `open` recorded")
+    closing.add_argument("id")
+    args = parser.parse_args(argv)
+
+    try:
+        if args.action == "list":
+            answer = {name: {"kind": e["kind"], "title": e["title"]}
+                      for name, e in screens.names().items()}
+        elif args.action == "open":
+            answer = screens.open_screen(args.name)
+        else:
+            answer = screens.close_screen(args.id)
+    except screens.ScreenError as error:
+        print(f"error: {error}", file=sys.stderr)
+        return 1
+    print(json.dumps(answer, indent=2))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

quality_loop/adapter.py

@@ -0,0 +1,88 @@
+"""What a product supplies: its surface, its rooms' limits, its evidence.
+
+A product subclasses `Adapter` and fills in what is its own. Everything else -
+building rooms, isolating the session, the stand-in, the screens, the
+operator, the audit, the findings - is the runner's and is the same for
+every product.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import pathlib
+import sys
+
+from quality_loop import audit, scenario, session, standin
+
+
+class Adapter:
+    """One product's half of a run.
+
+    Attributes:
+        name: The product's name, used in the run directory and messages.
+        version: The product's version, which names the run directory.
+        checkout: The product's repository.
+        rooms: Where rooms are built. Inside the checkout is fine, because the
+            session is `--restricted`; anywhere else in the checkout is refused.
+        scenarios: The directory of scenario files.
+        skill_name: The bundle's directory name inside a room.
+        fixtures: The directory screens are served or copied from, or None.
+        screens: ``{name: Page | Terminal}`` - all the operator may put up.
+        evidence: ``{name: {"argv": [...], "help": "..."}}`` - commands that
+            read, which the operator runs through its MCP server with
+            arguments of its choosing and no shell.
+        evidence_help: One paragraph on what those tools answer.
+        python: The interpreter the operator's CLIs run under.
+    """
+    name = "product"
+    version = "0"
+    checkout: pathlib.Path
+    rooms: pathlib.Path
+    scenarios: pathlib.Path
+    skill_name = "product"
+    fixtures: pathlib.Path | None = None
+    screens: dict = {}
+    evidence: dict = {}
+    evidence_help = ""
+    python = sys.executable
+
+    def bundle(self) -> pathlib.Path:
+        """The surface under test - a zip or a directory - built for this run."""
+        raise NotImplementedError
+
+    def rules(self) -> str:
+        """RULES.md: what the room may read and what it must write down."""
+        raise NotImplementedError
+
+    def session(self, room: pathlib.Path) -> session.Session:
+        """How the room's session is started: its tools, permissions, servers."""
+        raise NotImplementedError
+
+    def policy(self, room: pathlib.Path) -> audit.Policy:
+        """What the audit flags for this product."""
+        return audit.Policy(checkout=self.checkout)
+
+    def product(self):
+        """A context manager for the length of a run: the product up, if it
+        has to be running for the room to use it."""
+        return contextlib.nullcontext()
+
+    def stand_in(self, case: scenario.Scenario, variant: str,
+                 room: pathlib.Path) -> standin.StandIn | None:
+        """The operator's files stood in for, for one room, or None."""
+        return None
+
+    def runs(self, cases: list[scenario.Scenario],
+             named: bool = False) -> list[tuple[scenario.Scenario, str]]:
+        """``(scenario, variant)`` pairs: the same scenario on several fixture
+        configs is several rooms. The default is one room per scenario.
+
+        ``named`` says the scenarios were picked by name rather than being
+        every one there is - a product can leave out of a run of everything
+        those that need a screen only the person running it can put up.
+        """
+        return [(case, "") for case in cases]
+
+    def alive(self) -> bool:
+        """Whether the product is still answering, asked on the heartbeat."""
+        return True

quality_loop/audit.py

@@ -0,0 +1,144 @@
+"""Checking the transcript for the rules no flag can enforce.
+
+RULES.md forbids the checkout, the installed package and other people's work,
+and until there was a transcript the only evidence that a room obeyed was that
+it said so. A rule you cannot check is a rule you are assuming.
+
+What is scanned is where a tool was *pointed* - the path-bearing fields of its
+input - not what it typed or what came back. A room that reads a path off the
+screen and repeats it in its notes is doing its task, not reaching for the
+checkout; and a window title is full of paths the room never asked for.
+
+**A refusal is not a breach.** A call the confinement turned down read
+nothing, so the measurement is intact. Refusals are reported - they are the
+confinement working in public - and only a call that came back with something
+disqualifies a run.
+"""
+
+from __future__ import annotations
+
+import dataclasses
+import os
+import pathlib
+import re
+from typing import Callable
+
+from quality_loop import transcript as tr
+
+#: Input fields that say where a tool was aimed.
+AIMED_AT = ("file_path", "path", "notebook_path", "pattern", "command", "glob")
+
+#: An absolute path in any spelling: /Users/..., C:\Users\... or C:/..., and
+#: \Users\... - rooted on the current drive, which Windows accepts too.
+ABSOLUTE = re.compile(r"(?:[A-Za-z]:[\\/]|[\\/])[^\s\"'`;|&)]+")
+
+#: A shell command that runs git: at the start, or after a separator, by any
+#: path, with or without ".exe".
+RUNS_GIT = re.compile(r"(?:^|[;&|(`]|\$\()\s*(?:\S*/)?git(?:\.exe)?\b")
+
+#: What git answers once GIT_CEILING_DIRECTORIES has stopped it at the room.
+NOT_A_REPOSITORY = "not a git repository"
+
+#: What room.CANARY asks a contaminated session to say.
+CANARY_PHRASE = "without --restricted"
+
+
+def norm(path: str) -> str:
+    """A path in one spelling: forward slashes, and lowercase on Windows,
+    where C:\\Users and c:/users are the same place."""
+    path = path.replace("\\", "/")
+    return path.lower() if os.name == "nt" else path
+
+
+def paths_in(tool_input: dict) -> list[str]:
+    """The absolute paths a tool call was pointed at."""
+    aimed = []
+    for key in AIMED_AT:
+        value = tool_input.get(key)
+        if isinstance(value, str):
+            value = value.replace("~/", str(pathlib.Path.home()) + "/")
+            aimed += ABSOLUTE.findall(value)
+    return aimed
+
+
+@dataclasses.dataclass
+class Policy:
+    """What a product's rooms must not reach, and what their sessions must have.
+
+    Attributes:
+        checkout: The product's repository. A path under it and outside the
+            room is a breach - not a mention of it, which every honest read of
+            the bundle inside the room also is.
+        forbidden: ``{substring: what it is}`` for paths that are a breach
+            anywhere: the installed package, the installed app, the operator's
+            files set aside.
+        foreign: Basenames of other authors' work in a directory the room may
+            write to - read one, and the room has learned from somebody who
+            had the repository.
+        bash: Whether the session has Bash on purpose. When it does not, Bash
+            in the init event means `--restricted` did not take effect.
+        servers: MCP servers that must be connected; one that is not means
+            the room reported on the outage, not the product.
+        leaked_skill: Says whether a slash command in the init event is one of
+            the checkout's own skills - the scoring key among them.
+    """
+    checkout: pathlib.Path
+    forbidden: dict[str, str] = dataclasses.field(default_factory=dict)
+    foreign: set[str] = dataclasses.field(default_factory=set)
+    bash: bool = False
+    servers: tuple[str, ...] = ()
+    leaked_skill: Callable[[str], bool] | None = None
+
+
+def isolation_held(transcript: pathlib.Path, policy: Policy) -> list[str]:
+    """What the session's init event says went wrong with the flags."""
+    init = tr.init_event(transcript)
+    if init is None:
+        return ["the session produced no init event"]
+    found = []
+    if not policy.bash and "Bash" in init.get("tools", []):
+        found.append("the session had Bash: it was not started with --restricted, "
+                     "so the checkout's CLAUDE.md was in context from the first token")
+    if policy.leaked_skill:
+        if leaked := [c for c in init.get("slash_commands", [])
+                      if policy.leaked_skill(str(c))]:
+            found.append(f"the checkout's own skills were visible to the room: {leaked}")
+    servers = {s.get("name"): s.get("status") for s in init.get("mcp_servers", [])
+               if isinstance(s, dict)}
+    for name in policy.servers:
+        if servers.get(name) != "connected":
+            found.append(f"MCP server {name} was {servers.get(name, 'absent')}, so what "
+                         f"the room reported is about the server, not the product")
+    if CANARY_PHRASE in tr.final_reply(transcript).replace("`", ""):
+        found.append("the room read its CLAUDE.md canary unasked: the session was "
+                     "started without --restricted")
+    return found
+
+
+def audit(room: pathlib.Path, policy: Policy) -> tuple[list[str], list[str]]:
+    """``(breaches, refused)`` from the room's transcript."""
+    room = pathlib.Path(room).resolve()
+    transcript = room / "transcript.jsonl"
+    if not transcript.exists():
+        return ["no transcript - the room was never driven"], []
+    breaches = isolation_held(transcript, policy)
+    refused: list[str] = []
+    checkout, inside = norm(str(policy.checkout.resolve())), norm(str(room))
+    forbidden = {norm(path): what for path, what in policy.forbidden.items()}
+    for call in tr.tool_calls(transcript):
+        landed = refused if call.is_error else breaches
+        command = call.input.get("command")
+        if isinstance(command, str) and RUNS_GIT.search(command):
+            # Refused when git found no repository - the ceiling held.
+            (refused if call.is_error or NOT_A_REPOSITORY in call.result
+             else breaches).append(f"{call.name} ran git: {command[:80]}")
+        for aimed in paths_in(call.input):
+            spelled = norm(aimed)
+            for path, what in forbidden.items():
+                if path in spelled:
+                    landed.append(f"{call.name} reached into {what}: {aimed}")
+            if spelled.startswith(checkout) and not spelled.startswith(inside):
+                landed.append(f"{call.name} reached into the checkout: {aimed}")
+            if pathlib.PurePath(aimed.replace("\\", "/")).name in policy.foreign:
+                landed.append(f"{call.name} read another author's work: {aimed}")
+    return sorted(set(breaches)), sorted(set(refused))

quality_loop/mcp.py

@@ -0,0 +1,161 @@
+"""The operator's tools, as a small MCP server: ``python -m quality_loop.mcp``.
+
+The operator has no shell. Claude Code lets a read-only command through
+whatever the allow list says, so a compound ``cd; ls; cat ...`` ran on the
+first real operator - and ``cat`` reads anywhere on the machine, which the
+file tools' confinement does not. So the operator's only way to act is these
+four tools:
+
+- ``screen_list``, ``screen_open(name)``, ``screen_close(id)`` - the
+  declared screens, as `quality_loop.screens` puts them up;
+- ``evidence(tool, args)`` - one of the product's declared evidence
+  commands, run with ``args`` as a list and no shell, so nothing can be piped,
+  redirected or chained. What the arguments mean is the evidence command's
+  own business, and an adapter declares only commands that read.
+
+Everything comes from the manifest the runner wrote (``QUALITY_LOOP_MANIFEST``).
+The protocol is the least of MCP that Claude Code needs - initialize,
+tools/list, tools/call, ping - as newline-delimited JSON-RPC on stdio, with
+nothing outside the standard library.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import pathlib
+import subprocess
+import sys
+
+import quality_loop
+from quality_loop import screens
+
+PROTOCOL = "2025-06-18"
+
+#: Ceiling on what one evidence call hands back, in characters.
+MAX_OUTPUT = 20_000
+
+#: How long one evidence command may take.
+EVIDENCE_TIMEOUT = 120
+
+
+def _manifest() -> dict:
+    return json.loads(pathlib.Path(os.environ[screens.MANIFEST]).read_text(encoding="utf-8"))
+
+
+def tools(manifest: dict) -> list[dict]:
+    """The tool list, with the declared names in the schemas."""
+    evidence = manifest.get("evidence", {})
+    listing = "\n".join(f"- {name}: {entry.get('help', '')}"
+                        for name, entry in evidence.items()) or "(none declared)"
+    return [
+        {"name": "screen_list",
+         "description": "The screens the product declares: name, kind and window title.",
+         "inputSchema": {"type": "object", "properties": {}}},
+        {"name": "screen_open",
+         "description": "Open a declared screen and wait until its window is up. "
+                        "Answers the id to close it with.",
+         "inputSchema": {"type": "object", "properties": {
+             "name": {"type": "string", "enum": sorted(manifest.get("screens", {}))}},
+             "required": ["name"]}},
+        {"name": "screen_close",
+         "description": "Close a screen screen_open opened, by its id.",
+         "inputSchema": {"type": "object", "properties": {"id": {"type": "string"}},
+                         "required": ["id"]}},
+        {"name": "evidence",
+         "description": "Run one of the product's evidence commands with the given "
+                        "arguments and answer what it printed. Run it with "
+                        "args [\"--help\"] first. Declared:\n" + listing,
+         "inputSchema": {"type": "object", "properties": {
+             "tool": {"type": "string", "enum": sorted(evidence)},
+             "args": {"type": "array", "items": {"type": "string"}}},
+             "required": ["tool", "args"]}},
+    ]
+
+
+class ToolError(Exception):
+    pass
+
+
+def call(name: str, arguments: dict, manifest: dict) -> str:
+    """One tool call's answer, or a ToolError saying what was wrong."""
+    try:
+        if name == "screen_list":
+            return json.dumps({n: {"kind": e["kind"], "title": e["title"]}
+                               for n, e in manifest.get("screens", {}).items()}, indent=2)
+        if name == "screen_open":
+            return json.dumps(screens.open_screen(str(arguments.get("name", ""))), indent=2)
+        if name == "screen_close":
+            return json.dumps(screens.close_screen(str(arguments.get("id", ""))), indent=2)
+    except screens.ScreenError as error:
+        raise ToolError(str(error)) from None
+    if name == "evidence":
+        return _evidence(arguments, manifest)
+    raise ToolError(f"no tool {name!r}")
+
+
+def _evidence(arguments: dict, manifest: dict) -> str:
+    declared = manifest.get("evidence", {})
+    tool = arguments.get("tool")
+    if tool not in declared:
+        raise ToolError(f"no evidence tool {tool!r}; declared: "
+                        + (", ".join(sorted(declared)) or "none"))
+    args = arguments.get("args", [])
+    if not isinstance(args, list) or not all(isinstance(a, str) for a in args):
+        raise ToolError("args must be a list of strings")
+    try:
+        done = subprocess.run([*declared[tool]["argv"], *args], capture_output=True,
+                              text=True, encoding="utf-8", errors="replace",
+                              stdin=subprocess.DEVNULL, timeout=EVIDENCE_TIMEOUT)
+    except subprocess.TimeoutExpired:
+        raise ToolError(f"{tool} did not finish within {EVIDENCE_TIMEOUT}s") from None
+    said = done.stdout + (f"\n[stderr]\n{done.stderr}" if done.stderr.strip() else "")
+    if len(said) > MAX_OUTPUT:
+        said = said[:MAX_OUTPUT] + f"\n... {len(said) - MAX_OUTPUT} more characters"
+    if done.returncode != 0:
+        raise ToolError(f"{tool} exited {done.returncode}\n{said}")
+    return said
+
+
+def handle(message: dict, manifest: dict) -> dict | None:
+    """The reply to one JSON-RPC message, or None for a notification."""
+    if "id" not in message:
+        return None
+    method, ident = message.get("method"), message["id"]
+    if method == "initialize":
+        asked = (message.get("params") or {}).get("protocolVersion") or PROTOCOL
+        result = {"protocolVersion": asked, "capabilities": {"tools": {}},
+                  "serverInfo": {"name": "quality-loop", "version": quality_loop.__version__}}
+    elif method == "ping":
+        result = {}
+    elif method == "tools/list":
+        result = {"tools": tools(manifest)}
+    elif method == "tools/call":
+        params = message.get("params") or {}
+        try:
+            text, failed = call(params.get("name", ""), params.get("arguments") or {},
+                                manifest), False
+        except ToolError as error:
+            text, failed = str(error), True
+        result = {"content": [{"type": "text", "text": text}], "isError": failed}
+    else:
+        return {"jsonrpc": "2.0", "id": ident,
+                "error": {"code": -32601, "message": f"no method {method!r}"}}
+    return {"jsonrpc": "2.0", "id": ident, "result": result}
+
+
+def main() -> int:
+    manifest = _manifest()
+    for line in sys.stdin:
+        try:
+            message = json.loads(line)
+        except ValueError:
+            continue
+        if isinstance(message, dict) and (reply := handle(message, manifest)) is not None:
+            sys.stdout.write(json.dumps(reply) + "\n")
+            sys.stdout.flush()
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())