quality-loop 0.1.0__tar.gz

quality_loop-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 craftware
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

quality_loop-0.1.0/PKG-INFO

@@ -0,0 +1,74 @@
+Metadata-Version: 2.4
+Name: quality-loop
+Version: 0.1.0
+Summary: Measure what an agent-facing product teaches: isolated agent runs of real tasks, reported by an operator agent
+Author: craftware
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/crftwr/quality-loop
+Project-URL: Repository, https://github.com/crftwr/quality-loop
+Project-URL: Issues, https://github.com/crftwr/quality-loop/issues
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: Microsoft :: Windows
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: pytest-timeout>=2; extra == "dev"
+Dynamic: license-file
+
+# quality-loop
+
+Measure what an agent-facing product teaches.
+
+A skill bundle, an MCP server or a CLI is only as good as what an agent with
+nothing else can do with it. quality-loop runs that agent on a real task, in a
+**room** that holds the product's agent-facing surface and nothing else. The
+room is started so it cannot read the repository where the answers are.
+Afterwards an **operator** agent reports the problems it finds in what the
+room did, each with a severity. A human decides what to do with the report.
+
+> Status: early development, run end to end against a toy product
+> (`examples/toy/run.py`). In place: rooms and their isolation, the audit,
+> the stand-in, fixture pages, screens by name on macOS and Windows,
+> Markdown scenarios, the operator agent and its MCP tools, and reports with
+> severities. The first real product, Keyhac, is next. The design is
+> [crftwr/keyhac#205](https://github.com/crftwr/keyhac/issues/205).
+
+## What stays code, and why
+
+Anything a run depends on to mean something is code, not something an agent
+decides:
+
+- **The prompt the room is handed** is copied verbatim. The operator knows
+  what a good answer looks like, so any paraphrase of the prompt can leak it.
+- **Isolation.** The room runs with `--restricted`, an explicit permission
+  list under `dontAsk`, one MCP configuration and nothing else, git stopped at
+  the room, and stdin closed. A transcript audit then checks what no flag can
+  enforce.
+- **The operator's machine.** Their files are moved aside and replaced with
+  fixtures. On the way out they are put back in a `finally`, whether the run
+  ended normally, by an exception, by Ctrl-C or by SIGTERM.
+- **Evidence.** A report cites what tools recorded, not what the room said.
+
+## Products using it
+
+- [Keyhac](https://github.com/crftwr/keyhac)
+- [XeFM](https://github.com/crftwr/xefm) (planned)
+
+## Development
+
+```
+make test
+```
+
+MIT licensed.

quality_loop-0.1.0/README.md

@@ -0,0 +1,46 @@
+# quality-loop
+
+Measure what an agent-facing product teaches.
+
+A skill bundle, an MCP server or a CLI is only as good as what an agent with
+nothing else can do with it. quality-loop runs that agent on a real task, in a
+**room** that holds the product's agent-facing surface and nothing else. The
+room is started so it cannot read the repository where the answers are.
+Afterwards an **operator** agent reports the problems it finds in what the
+room did, each with a severity. A human decides what to do with the report.
+
+> Status: early development, run end to end against a toy product
+> (`examples/toy/run.py`). In place: rooms and their isolation, the audit,
+> the stand-in, fixture pages, screens by name on macOS and Windows,
+> Markdown scenarios, the operator agent and its MCP tools, and reports with
+> severities. The first real product, Keyhac, is next. The design is
+> [crftwr/keyhac#205](https://github.com/crftwr/keyhac/issues/205).
+
+## What stays code, and why
+
+Anything a run depends on to mean something is code, not something an agent
+decides:
+
+- **The prompt the room is handed** is copied verbatim. The operator knows
+  what a good answer looks like, so any paraphrase of the prompt can leak it.
+- **Isolation.** The room runs with `--restricted`, an explicit permission
+  list under `dontAsk`, one MCP configuration and nothing else, git stopped at
+  the room, and stdin closed. A transcript audit then checks what no flag can
+  enforce.
+- **The operator's machine.** Their files are moved aside and replaced with
+  fixtures. On the way out they are put back in a `finally`, whether the run
+  ended normally, by an exception, by Ctrl-C or by SIGTERM.
+- **Evidence.** A report cites what tools recorded, not what the room said.
+
+## Products using it
+
+- [Keyhac](https://github.com/crftwr/keyhac)
+- [XeFM](https://github.com/crftwr/xefm) (planned)
+
+## Development
+
+```
+make test
+```
+
+MIT licensed.

quality_loop-0.1.0/pyproject.toml

@@ -0,0 +1,51 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "quality-loop"
+# version -> single-sourced from quality_loop/__init__.py's __version__.
+dynamic = ["version"]
+description = "Measure what an agent-facing product teaches: isolated agent runs of real tasks, reported by an operator agent"
+readme = "README.md"
+license = "MIT"
+authors = [{ name = "craftware" }]
+requires-python = ">=3.11"
+classifiers = [
+    "Development Status :: 2 - Pre-Alpha",
+    "Intended Audience :: Developers",
+    "Operating System :: MacOS",
+    "Operating System :: Microsoft :: Windows",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Topic :: Software Development :: Quality Assurance",
+    "Topic :: Software Development :: Testing",
+]
+# None at runtime: the harness drives `claude` and the product's own tools as
+# subprocesses, and everything it reads is JSON lines and Markdown.
+dependencies = []
+
+[project.optional-dependencies]
+dev = ["pytest>=8", "pytest-timeout>=2"]
+
+[project.urls]
+Homepage = "https://github.com/crftwr/quality-loop"
+Repository = "https://github.com/crftwr/quality-loop"
+Issues = "https://github.com/crftwr/quality-loop/issues"
+
+[tool.setuptools.dynamic]
+version = { attr = "quality_loop.__version__" }
+
+[tool.setuptools.packages.find]
+include = ["quality_loop*"]
+
+[tool.setuptools.package-data]
+# The operator agent's instructions, read from the installed package.
+quality_loop = ["OPERATOR.md"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+timeout = 60

quality_loop-0.1.0/quality_loop/OPERATOR.md

@@ -0,0 +1,103 @@
+# You are the operator of one room
+
+A **room** is a separate Claude Code session that was given a task and nothing
+but a product's agent-facing surface - a skill bundle and the product's own
+tools. It was started so it cannot read the product's repository. What it
+does with what it was given is the measurement. Your job is to make that
+measurement possible and then report what it shows.
+
+You work in two phases of one session. The message that starts each phase
+says which one it is, and gives you the paths.
+
+## Phase 1: put up the screen
+
+1. Read the scenario file. Its **Screen** section says what has to be in
+   front before the room starts. It may say "none".
+2. If a screen is needed, call `screen_list` to see what the product
+   declares, choose what the section asks for, and call `screen_open` for
+   each one. It returns once the window is up, or fails.
+3. If the section asks for something no declared screen provides, do not
+   improvise. Write that into `setup.md` in your directory and stop: the
+   room cannot measure anything without its screen.
+4. Write `setup.md` in your directory: what you opened, with the ids `open`
+   returned, and anything about the screen a reader of your report should
+   know.
+
+You have no shell. Your tools are Read, Glob, Grep, writing in your own
+directory and the room's REPORT.md, and the four tools above.
+
+You may not change anything in the room. The runner checks every file in it
+after this phase, and a changed byte cancels the room. In particular, the
+room's TASK.md is copied from the scenario by the runner and must reach the
+room exactly as written.
+
+## Phase 2: report
+
+The room has finished. In the room directory you have:
+
+- `transcript.jsonl`: everything the room did, as stream-json. This is the
+  record. The room's own account of what it did is only a claim.
+- `QUESTIONS.md`: what the room noticed it had to guess.
+- `evidence/`: what the runner collected. That includes the product's files
+  before and after, anything the room created, the screen's event record, and
+  `audit.json`, the runner's isolation audit.
+
+Then:
+
+1. Read the scenario's **Must** and **Must not** sections. For each line,
+   establish from evidence whether the run met it. Use the product's evidence
+   tools where a line is about behaviour; the start-of-phase message lists
+   them. Quote what a tool answered. Do not infer it.
+2. Read the transcript for **friction**, whether or not the room noticed it:
+   a tool call that failed or was refused, an answer that was an error or a
+   blank, a search of the bundle that came back empty, a file rewritten or a
+   run repeated, turns running out, sentences where the room says it is
+   unsure or was wrong. A problem the room never mentioned in QUESTIONS.md is
+   the most valuable kind.
+3. Read **QUESTIONS.md**. Each question is a candidate problem.
+4. Write **REPORT.md** in the room, in the format below. Then close every
+   screen you opened (`screen_close`).
+
+**Friction is not a problem by itself.** A deploy's own "DEPLOY FAILED" on a
+screen built to show a failed deploy is the task working. Report what a
+product change would improve, and say why.
+
+## REPORT.md
+
+```markdown
+# Report: <scenario title>
+
+One paragraph: what the room was asked, what it did, and what it left.
+
+### [high] <the problem, in one line>
+- **Id:** `<kebab-case, stable>`
+- **Kind:** skill gap
+- **Must:** "<the Must line it concerns, or none>"
+- **Evidence:** <tool output quoted, or transcript step and what it shows>
+- **Where it would be fixed:** <file and section, in the product>
+- **How a fix is confirmed:** <the evidence that would show it fixed>
+
+<Free text: what happened, and why it matters.>
+```
+
+**Severity** - pick the first that fits:
+
+| | |
+|---|---|
+| critical | The run did harm: data destroyed, the user's files rewritten or no longer loading, a destructive control pressed. |
+| high | A Must not met, or a result that is silently wrong - one that looks right and is not. |
+| medium | Met, but only by guessing something the product should have said, or after rework a user's agent would repeat. |
+| low | Friction that cost little: an empty search, a retry, an unclear message. |
+
+**Kind**: `skill gap` (the product's agent-facing text should have said it),
+`product` (the product behaves wrongly), `harness` (the runner, or the room's
+tools), `out of scope` (true of the task or the application, not of the
+product).
+
+**Id** names the problem, not the run: `fn-f1-not-bound`, not `room-3-issue`.
+A later run reports the same problem under the same id, which is how a fix is
+seen to have worked.
+
+There is no verdict. Write problems, most severe first. If you found none,
+say so in the opening paragraph and say what you checked. Zero problems is
+rare, and the reader will want to know why you believe it.

quality_loop-0.1.0/quality_loop/__init__.py

@@ -0,0 +1,15 @@
+"""quality-loop: measure what an agent-facing product teaches.
+
+A product that agents use - a skill bundle, an MCP server, a CLI - is only as
+good as what an agent with nothing else can do with it. This package runs that
+agent: a task, in a room holding the product's agent-facing surface and nothing
+else, started so it cannot read the repository the answers are in. A second
+agent, the operator, puts up whatever the task needs beforehand and afterwards
+reports the problems it finds in what the room did, each with a severity.
+
+What has to hold for a run to mean anything - the isolation, the prompt the
+room is handed, putting the user's machine back - is code here, not an
+agent's discretion.
+"""
+
+__version__ = "0.1.0"

quality_loop-0.1.0/quality_loop/__main__.py

@@ -0,0 +1,48 @@
+"""The operator agent's command line: ``python -m quality_loop screen ...``.
+
+    python -m quality_loop screen list
+    python -m quality_loop screen open approval-queue
+    python -m quality_loop screen close approval-queue-1
+
+Everything it takes is a name from the manifest the runner wrote. Answers are
+JSON on stdout, errors one line on stderr with exit status 1.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+
+from quality_loop import screens
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(prog="quality-loop")
+    commands = parser.add_subparsers(dest="command", required=True)
+    screen = commands.add_parser("screen", help="put up or take down a declared screen")
+    actions = screen.add_subparsers(dest="action", required=True)
+    actions.add_parser("list", help="the screens the adapter declares")
+    opening = actions.add_parser("open", help="open one, and wait for its window")
+    opening.add_argument("name")
+    closing = actions.add_parser("close", help="close one that `open` recorded")
+    closing.add_argument("id")
+    args = parser.parse_args(argv)
+
+    try:
+        if args.action == "list":
+            answer = {name: {"kind": e["kind"], "title": e["title"]}
+                      for name, e in screens.names().items()}
+        elif args.action == "open":
+            answer = screens.open_screen(args.name)
+        else:
+            answer = screens.close_screen(args.id)
+    except screens.ScreenError as error:
+        print(f"error: {error}", file=sys.stderr)
+        return 1
+    print(json.dumps(answer, indent=2))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

quality_loop-0.1.0/quality_loop/adapter.py

@@ -0,0 +1,88 @@
+"""What a product supplies: its surface, its rooms' limits, its evidence.
+
+A product subclasses `Adapter` and fills in what is its own. Everything else -
+building rooms, isolating the session, the stand-in, the screens, the
+operator, the audit, the findings - is the runner's and is the same for
+every product.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import pathlib
+import sys
+
+from quality_loop import audit, scenario, session, standin
+
+
+class Adapter:
+    """One product's half of a run.
+
+    Attributes:
+        name: The product's name, used in the run directory and messages.
+        version: The product's version, which names the run directory.
+        checkout: The product's repository.
+        rooms: Where rooms are built. Inside the checkout is fine, because the
+            session is `--restricted`; anywhere else in the checkout is refused.
+        scenarios: The directory of scenario files.
+        skill_name: The bundle's directory name inside a room.
+        fixtures: The directory screens are served or copied from, or None.
+        screens: ``{name: Page | Terminal}`` - all the operator may put up.
+        evidence: ``{name: {"argv": [...], "help": "..."}}`` - commands that
+            read, which the operator runs through its MCP server with
+            arguments of its choosing and no shell.
+        evidence_help: One paragraph on what those tools answer.
+        python: The interpreter the operator's CLIs run under.
+    """
+    name = "product"
+    version = "0"
+    checkout: pathlib.Path
+    rooms: pathlib.Path
+    scenarios: pathlib.Path
+    skill_name = "product"
+    fixtures: pathlib.Path | None = None
+    screens: dict = {}
+    evidence: dict = {}
+    evidence_help = ""
+    python = sys.executable
+
+    def bundle(self) -> pathlib.Path:
+        """The surface under test - a zip or a directory - built for this run."""
+        raise NotImplementedError
+
+    def rules(self) -> str:
+        """RULES.md: what the room may read and what it must write down."""
+        raise NotImplementedError
+
+    def session(self, room: pathlib.Path) -> session.Session:
+        """How the room's session is started: its tools, permissions, servers."""
+        raise NotImplementedError
+
+    def policy(self, room: pathlib.Path) -> audit.Policy:
+        """What the audit flags for this product."""
+        return audit.Policy(checkout=self.checkout)
+
+    def product(self):
+        """A context manager for the length of a run: the product up, if it
+        has to be running for the room to use it."""
+        return contextlib.nullcontext()
+
+    def stand_in(self, case: scenario.Scenario, variant: str,
+                 room: pathlib.Path) -> standin.StandIn | None:
+        """The operator's files stood in for, for one room, or None."""
+        return None
+
+    def runs(self, cases: list[scenario.Scenario],
+             named: bool = False) -> list[tuple[scenario.Scenario, str]]:
+        """``(scenario, variant)`` pairs: the same scenario on several fixture
+        configs is several rooms. The default is one room per scenario.
+
+        ``named`` says the scenarios were picked by name rather than being
+        every one there is - a product can leave out of a run of everything
+        those that need a screen only the person running it can put up.
+        """
+        return [(case, "") for case in cases]
+
+    def alive(self) -> bool:
+        """Whether the product is still answering, asked on the heartbeat."""
+        return True

quality_loop-0.1.0/quality_loop/audit.py

@@ -0,0 +1,144 @@
+"""Checking the transcript for the rules no flag can enforce.
+
+RULES.md forbids the checkout, the installed package and other people's work,
+and until there was a transcript the only evidence that a room obeyed was that
+it said so. A rule you cannot check is a rule you are assuming.
+
+What is scanned is where a tool was *pointed* - the path-bearing fields of its
+input - not what it typed or what came back. A room that reads a path off the
+screen and repeats it in its notes is doing its task, not reaching for the
+checkout; and a window title is full of paths the room never asked for.
+
+**A refusal is not a breach.** A call the confinement turned down read
+nothing, so the measurement is intact. Refusals are reported - they are the
+confinement working in public - and only a call that came back with something
+disqualifies a run.
+"""
+
+from __future__ import annotations
+
+import dataclasses
+import os
+import pathlib
+import re
+from typing import Callable
+
+from quality_loop import transcript as tr
+
+#: Input fields that say where a tool was aimed.
+AIMED_AT = ("file_path", "path", "notebook_path", "pattern", "command", "glob")
+
+#: An absolute path in any spelling: /Users/..., C:\Users\... or C:/..., and
+#: \Users\... - rooted on the current drive, which Windows accepts too.
+ABSOLUTE = re.compile(r"(?:[A-Za-z]:[\\/]|[\\/])[^\s\"'`;|&)]+")
+
+#: A shell command that runs git: at the start, or after a separator, by any
+#: path, with or without ".exe".
+RUNS_GIT = re.compile(r"(?:^|[;&|(`]|\$\()\s*(?:\S*/)?git(?:\.exe)?\b")
+
+#: What git answers once GIT_CEILING_DIRECTORIES has stopped it at the room.
+NOT_A_REPOSITORY = "not a git repository"
+
+#: What room.CANARY asks a contaminated session to say.
+CANARY_PHRASE = "without --restricted"
+
+
+def norm(path: str) -> str:
+    """A path in one spelling: forward slashes, and lowercase on Windows,
+    where C:\\Users and c:/users are the same place."""
+    path = path.replace("\\", "/")
+    return path.lower() if os.name == "nt" else path
+
+
+def paths_in(tool_input: dict) -> list[str]:
+    """The absolute paths a tool call was pointed at."""
+    aimed = []
+    for key in AIMED_AT:
+        value = tool_input.get(key)
+        if isinstance(value, str):
+            value = value.replace("~/", str(pathlib.Path.home()) + "/")
+            aimed += ABSOLUTE.findall(value)
+    return aimed
+
+
+@dataclasses.dataclass
+class Policy:
+    """What a product's rooms must not reach, and what their sessions must have.
+
+    Attributes:
+        checkout: The product's repository. A path under it and outside the
+            room is a breach - not a mention of it, which every honest read of
+            the bundle inside the room also is.
+        forbidden: ``{substring: what it is}`` for paths that are a breach
+            anywhere: the installed package, the installed app, the operator's
+            files set aside.
+        foreign: Basenames of other authors' work in a directory the room may
+            write to - read one, and the room has learned from somebody who
+            had the repository.
+        bash: Whether the session has Bash on purpose. When it does not, Bash
+            in the init event means `--restricted` did not take effect.
+        servers: MCP servers that must be connected; one that is not means
+            the room reported on the outage, not the product.
+        leaked_skill: Says whether a slash command in the init event is one of
+            the checkout's own skills - the scoring key among them.
+    """
+    checkout: pathlib.Path
+    forbidden: dict[str, str] = dataclasses.field(default_factory=dict)
+    foreign: set[str] = dataclasses.field(default_factory=set)
+    bash: bool = False
+    servers: tuple[str, ...] = ()
+    leaked_skill: Callable[[str], bool] | None = None
+
+
+def isolation_held(transcript: pathlib.Path, policy: Policy) -> list[str]:
+    """What the session's init event says went wrong with the flags."""
+    init = tr.init_event(transcript)
+    if init is None:
+        return ["the session produced no init event"]
+    found = []
+    if not policy.bash and "Bash" in init.get("tools", []):
+        found.append("the session had Bash: it was not started with --restricted, "
+                     "so the checkout's CLAUDE.md was in context from the first token")
+    if policy.leaked_skill:
+        if leaked := [c for c in init.get("slash_commands", [])
+                      if policy.leaked_skill(str(c))]:
+            found.append(f"the checkout's own skills were visible to the room: {leaked}")
+    servers = {s.get("name"): s.get("status") for s in init.get("mcp_servers", [])
+               if isinstance(s, dict)}
+    for name in policy.servers:
+        if servers.get(name) != "connected":
+            found.append(f"MCP server {name} was {servers.get(name, 'absent')}, so what "
+                         f"the room reported is about the server, not the product")
+    if CANARY_PHRASE in tr.final_reply(transcript).replace("`", ""):
+        found.append("the room read its CLAUDE.md canary unasked: the session was "
+                     "started without --restricted")
+    return found
+
+
+def audit(room: pathlib.Path, policy: Policy) -> tuple[list[str], list[str]]:
+    """``(breaches, refused)`` from the room's transcript."""
+    room = pathlib.Path(room).resolve()
+    transcript = room / "transcript.jsonl"
+    if not transcript.exists():
+        return ["no transcript - the room was never driven"], []
+    breaches = isolation_held(transcript, policy)
+    refused: list[str] = []
+    checkout, inside = norm(str(policy.checkout.resolve())), norm(str(room))
+    forbidden = {norm(path): what for path, what in policy.forbidden.items()}
+    for call in tr.tool_calls(transcript):
+        landed = refused if call.is_error else breaches
+        command = call.input.get("command")
+        if isinstance(command, str) and RUNS_GIT.search(command):
+            # Refused when git found no repository - the ceiling held.
+            (refused if call.is_error or NOT_A_REPOSITORY in call.result
+             else breaches).append(f"{call.name} ran git: {command[:80]}")
+        for aimed in paths_in(call.input):
+            spelled = norm(aimed)
+            for path, what in forbidden.items():
+                if path in spelled:
+                    landed.append(f"{call.name} reached into {what}: {aimed}")
+            if spelled.startswith(checkout) and not spelled.startswith(inside):
+                landed.append(f"{call.name} reached into the checkout: {aimed}")
+            if pathlib.PurePath(aimed.replace("\\", "/")).name in policy.foreign:
+                landed.append(f"{call.name} read another author's work: {aimed}")
+    return sorted(set(breaches)), sorted(set(refused))