quality-loop 0.1.0__tar.gz → 0.2.1__tar.gz

quality_loop-0.2.1/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.4
+Name: quality-loop
+Version: 0.2.1
+Summary: Measure what a product teaches - its skill, tools or manual - with isolated agent runs of real tasks, reported by an operator agent
+Author: craftware
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/crftwr/quality-loop
+Project-URL: Repository, https://github.com/crftwr/quality-loop
+Project-URL: Issues, https://github.com/crftwr/quality-loop/issues
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: Microsoft :: Windows
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: tui
+Requires-Dist: pyte>=0.8; extra == "tui"
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: pytest-timeout>=2; extra == "dev"
+Requires-Dist: pyte>=0.8; extra == "dev"
+Dynamic: license-file
+
+# quality-loop
+
+Measure what a product teaches.
+
+A skill bundle, an MCP server, a CLI - or a manual and the app it describes -
+is only as good as what someone with nothing else can do with it.
+quality-loop runs an agent on a real task, in a **room** that holds what the
+product gives its users and nothing more. The room is started so it cannot
+read the repository the answers are in. Afterwards an **operator** agent
+reports the problems it finds in what the room did, each with a severity and
+the evidence for it.
+
+There is no pass/fail. The point is **discovery**: each run should surface
+something the last one did not, and leave a record good enough for a person,
+or a development-time agent, to reproduce what it found. Every problem
+carries a stable id, so the same problem is recognised across runs:
+de-duplicated, seen to be gone, seen to come back.
+
+```
+pip install quality-loop            # 'quality-loop[tui]' to drive terminal apps
+```
+
+Python 3.11 or later, no runtime dependencies (driving a terminal app needs
+`pyte`, the `tui` extra). It drives [Claude Code](https://claude.com/claude-code)
+(`claude`) as subprocesses, so that has to be installed and signed in. macOS
+and Windows; terminal apps on macOS and Linux.
+
+## See it work
+
+The repository has two products small enough to read in a minute.
+
+```
+git clone https://github.com/crftwr/quality-loop && cd quality-loop
+make test                                # the harness's own tests
+.venv/bin/python examples/toy/run.py     # an agent-facing skill: one room, a few cents
+.venv/bin/python examples/tui/run.py     # a terminal app and its manual
+```
+
+The toy is a "greeter" skill that reads a house style and writes one file.
+The TUI example is a note list whose manual is the only thing the room is
+given; the room drives the app itself, and the operator checks it live. The
+output is `rooms/<version>-<time>/FINDINGS.md`.
+
+## A run, per scenario
+
+```
+build the room - the bundle, RULES.md, and TASK.md holding the scenario's Prompt, verbatim
+stand fixtures in for the user's files
+operator, phase 1 - put the scenario's screen up, by name
+the room's session - isolated, non-interactive
+operator, live check - drive the real app, when the scenario asks for one
+collect what the room made, put the user's files back, audit the transcript
+operator, phase 2 - read the record, check the Must lists with evidence, write REPORT.md
+```
+
+## The loop
+
+```
+quality-loop init                      # in the product: the conductor skill and settings
+quality-loop run 11 12                 # starts a run in the background, prints its id
+quality-loop watch                     # follow it from a terminal
+quality-loop findings                  # new / still there / gone / back, against the run before
+```
+
+`init` installs a skill for the **conductor**: a Claude Code session in the
+product's repository that plans with the person, sets the product up, runs,
+files findings as GitHub Issues, fixes what the person chooses, and retests,
+under a policy the person sets. See [Running the loop](https://github.com/crftwr/quality-loop/blob/main/docs/running.md).
+
+## Documentation
+
+- [Concepts](https://github.com/crftwr/quality-loop/blob/main/docs/concepts.md): rooms, operators, the conductor, scenarios,
+  reports, and which parts are code rather than an agent's judgement.
+- [Running the loop](https://github.com/crftwr/quality-loop/blob/main/docs/running.md): the command line, watching a run,
+  parallel runs, comparing findings, the conductor and its policy.
+- [Writing scenarios](https://github.com/crftwr/quality-loop/blob/main/docs/scenarios.md): the Markdown format, doc-driven
+  scenarios, and what makes a scenario measure something.
+- [Writing an adapter](https://github.com/crftwr/quality-loop/blob/main/docs/adapters.md): what a product supplies, from a
+  launch command and a redirected home up to evidence tools of its own.
+- [Reports](https://github.com/crftwr/quality-loop/blob/main/docs/reports.md): problems, severities, kinds and ids, the
+  reproduction record, and `FINDINGS.md` / `findings.json`.
+- [Isolation and permissions](https://github.com/crftwr/quality-loop/blob/main/docs/isolation.md): what keeps the room, the
+  operator and the conductor in bounds, why each rule exists, and the known limits.
+- [Development](https://github.com/crftwr/quality-loop/blob/main/docs/development.md): tests, live tests, releasing.
+
+## Products using it
+
+- [Keyhac](https://github.com/crftwr/keyhac): `qa/`, `make qa`
+- [XeFM](https://github.com/crftwr/xefm) (planned)
+
+## Status
+
+0.x, early. It has been run end to end on macOS against the two examples,
+Keyhac and XeFM (including XeFM's terminal UI in a live check), and on
+Windows (11, ARM64) against the toy, Keyhac and XeFM, with real page and
+terminal screens. Not yet run anywhere for real: synthetic keys and
+screenshots of a desktop app (`drive.DesktopApp`); a terminal app is not
+driven on Windows yet. The design discussion is
+[crftwr/keyhac#205](https://github.com/crftwr/keyhac/issues/205) and this
+repository's issues.
+
+MIT licensed.

quality_loop-0.2.1/README.md

@@ -0,0 +1,102 @@
+# quality-loop
+
+Measure what a product teaches.
+
+A skill bundle, an MCP server, a CLI - or a manual and the app it describes -
+is only as good as what someone with nothing else can do with it.
+quality-loop runs an agent on a real task, in a **room** that holds what the
+product gives its users and nothing more. The room is started so it cannot
+read the repository the answers are in. Afterwards an **operator** agent
+reports the problems it finds in what the room did, each with a severity and
+the evidence for it.
+
+There is no pass/fail. The point is **discovery**: each run should surface
+something the last one did not, and leave a record good enough for a person,
+or a development-time agent, to reproduce what it found. Every problem
+carries a stable id, so the same problem is recognised across runs:
+de-duplicated, seen to be gone, seen to come back.
+
+```
+pip install quality-loop            # 'quality-loop[tui]' to drive terminal apps
+```
+
+Python 3.11 or later, no runtime dependencies (driving a terminal app needs
+`pyte`, the `tui` extra). It drives [Claude Code](https://claude.com/claude-code)
+(`claude`) as subprocesses, so that has to be installed and signed in. macOS
+and Windows; terminal apps on macOS and Linux.
+
+## See it work
+
+The repository has two products small enough to read in a minute.
+
+```
+git clone https://github.com/crftwr/quality-loop && cd quality-loop
+make test                                # the harness's own tests
+.venv/bin/python examples/toy/run.py     # an agent-facing skill: one room, a few cents
+.venv/bin/python examples/tui/run.py     # a terminal app and its manual
+```
+
+The toy is a "greeter" skill that reads a house style and writes one file.
+The TUI example is a note list whose manual is the only thing the room is
+given; the room drives the app itself, and the operator checks it live. The
+output is `rooms/<version>-<time>/FINDINGS.md`.
+
+## A run, per scenario
+
+```
+build the room - the bundle, RULES.md, and TASK.md holding the scenario's Prompt, verbatim
+stand fixtures in for the user's files
+operator, phase 1 - put the scenario's screen up, by name
+the room's session - isolated, non-interactive
+operator, live check - drive the real app, when the scenario asks for one
+collect what the room made, put the user's files back, audit the transcript
+operator, phase 2 - read the record, check the Must lists with evidence, write REPORT.md
+```
+
+## The loop
+
+```
+quality-loop init                      # in the product: the conductor skill and settings
+quality-loop run 11 12                 # starts a run in the background, prints its id
+quality-loop watch                     # follow it from a terminal
+quality-loop findings                  # new / still there / gone / back, against the run before
+```
+
+`init` installs a skill for the **conductor**: a Claude Code session in the
+product's repository that plans with the person, sets the product up, runs,
+files findings as GitHub Issues, fixes what the person chooses, and retests,
+under a policy the person sets. See [Running the loop](https://github.com/crftwr/quality-loop/blob/main/docs/running.md).
+
+## Documentation
+
+- [Concepts](https://github.com/crftwr/quality-loop/blob/main/docs/concepts.md): rooms, operators, the conductor, scenarios,
+  reports, and which parts are code rather than an agent's judgement.
+- [Running the loop](https://github.com/crftwr/quality-loop/blob/main/docs/running.md): the command line, watching a run,
+  parallel runs, comparing findings, the conductor and its policy.
+- [Writing scenarios](https://github.com/crftwr/quality-loop/blob/main/docs/scenarios.md): the Markdown format, doc-driven
+  scenarios, and what makes a scenario measure something.
+- [Writing an adapter](https://github.com/crftwr/quality-loop/blob/main/docs/adapters.md): what a product supplies, from a
+  launch command and a redirected home up to evidence tools of its own.
+- [Reports](https://github.com/crftwr/quality-loop/blob/main/docs/reports.md): problems, severities, kinds and ids, the
+  reproduction record, and `FINDINGS.md` / `findings.json`.
+- [Isolation and permissions](https://github.com/crftwr/quality-loop/blob/main/docs/isolation.md): what keeps the room, the
+  operator and the conductor in bounds, why each rule exists, and the known limits.
+- [Development](https://github.com/crftwr/quality-loop/blob/main/docs/development.md): tests, live tests, releasing.
+
+## Products using it
+
+- [Keyhac](https://github.com/crftwr/keyhac): `qa/`, `make qa`
+- [XeFM](https://github.com/crftwr/xefm) (planned)
+
+## Status
+
+0.x, early. It has been run end to end on macOS against the two examples,
+Keyhac and XeFM (including XeFM's terminal UI in a live check), and on
+Windows (11, ARM64) against the toy, Keyhac and XeFM, with real page and
+terminal screens. Not yet run anywhere for real: synthetic keys and
+screenshots of a desktop app (`drive.DesktopApp`); a terminal app is not
+driven on Windows yet. The design discussion is
+[crftwr/keyhac#205](https://github.com/crftwr/keyhac/issues/205) and this
+repository's issues.
+
+MIT licensed.

{quality_loop-0.1.0 → quality_loop-0.2.1}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
 name = "quality-loop"
 # version -> single-sourced from quality_loop/__init__.py's __version__.
 dynamic = ["version"]
-description = "Measure what an agent-facing product teaches: isolated agent runs of real tasks, reported by an operator agent"
+description = "Measure what a product teaches - its skill, tools or manual - with isolated agent runs of real tasks, reported by an operator agent"
 readme = "README.md"
 license = "MIT"
 authors = [{ name = "craftware" }]
@@ -29,7 +29,12 @@ classifiers = [
 dependencies = []
 
 [project.optional-dependencies]
-dev = ["pytest>=8", "pytest-timeout>=2"]
+# Driving a terminal app in a pty needs a terminal emulator; only for that.
+tui = ["pyte>=0.8"]
+dev = ["pytest>=8", "pytest-timeout>=2", "pyte>=0.8"]
+
+[project.scripts]
+quality-loop = "quality_loop.cli:main"
 
 [project.urls]
 Homepage = "https://github.com/crftwr/quality-loop"
@@ -43,8 +48,9 @@ version = { attr = "quality_loop.__version__" }
 include = ["quality_loop*"]
 
 [tool.setuptools.package-data]
-# The operator agent's instructions, read from the installed package.
-quality_loop = ["OPERATOR.md"]
+# The operator agent's instructions, read from the installed package, and the
+# conductor's skill, which `quality-loop init` copies into a product.
+quality_loop = ["OPERATOR.md", "skill/*.md", "skill/references/*.md"]
 
 [tool.pytest.ini_options]
 testpaths = ["tests"]

{quality_loop-0.1.0 → quality_loop-0.2.1}/quality_loop/OPERATOR.md

@@ -1,13 +1,15 @@
 # You are the operator of one room
 
 A **room** is a separate Claude Code session that was given a task and nothing
-but a product's agent-facing surface - a skill bundle and the product's own
-tools. It was started so it cannot read the product's repository. What it
-does with what it was given is the measurement. Your job is to make that
-measurement possible and then report what it shows.
+but what the product gives its users - a skill bundle and the product's own
+tools, or a manual and the app itself. It was started so it cannot read the
+product's repository. What it does with what it was given is the
+measurement. Your job is to make that measurement possible and then report
+what it shows.
 
-You work in two phases of one session. The message that starts each phase
-says which one it is, and gives you the paths.
+You work in phases of one session: phase 1, sometimes a live check, then
+phase 2. The message that starts each says which one it is, and gives you
+the paths.
 
 ## Phase 1: put up the screen
 
@@ -31,6 +33,29 @@ after this phase, and a changed byte cancels the room. In particular, the
 room's TASK.md is copied from the scenario by the runner and must reach the
 room exactly as written.
 
+## Live check: drive the real app
+
+Only when the scenario has a **Live check** section. The room has finished,
+and what it made is still in place - its config, its extension, its files.
+
+1. Read the scenario's **Live check** and **Must** sections. They say what
+   to try on the app.
+2. `app_list`, then `app_start` the app. Every `app_keys` call carries an
+   **intent**: what you expect the step to do. The answer is the screen
+   afterwards (text, or an image) and any lines the app wrote to its event
+   log.
+3. Decide each step from what the screen shows. When the app answers
+   differently from what you expected - a dialog you did not plan for, a
+   different state - look, and choose the next key from there. That
+   difference is often the finding.
+4. Do not fix anything, and do not try to make a Must line pass. You are
+   checking, not doing the task.
+5. `app_stop` every app you started. Then write `live.md` in your directory:
+   what you tried, by step number, and what you saw.
+
+Every step is recorded in the room's `evidence/live/` (`steps.jsonl` and a
+screen per step). Cite those step numbers in your report.
+
 ## Phase 2: report
 
 The room has finished. In the room directory you have:
@@ -39,8 +64,11 @@ The room has finished. In the room directory you have:
   record. The room's own account of what it did is only a claim.
 - `QUESTIONS.md`: what the room noticed it had to guess.
 - `evidence/`: what the runner collected. That includes the product's files
-  before and after, anything the room created, the screen's event record, and
-  `audit.json`, the runner's isolation audit.
+  before and after, anything the room created, the screen's event record,
+  `audit.json` (the runner's isolation audit) and `run.json` (where the run
+  started from: product version and commit, OS, fixtures). `live/` is your
+  live check's record, and `app/` the room's own steps when it drove the app
+  itself.
 
 Then:
 
@@ -55,9 +83,27 @@ Then:
    unsure or was wrong. A problem the room never mentioned in QUESTIONS.md is
    the most valuable kind.
 3. Read **QUESTIONS.md**. Each question is a candidate problem.
-4. Write **REPORT.md** in the room, in the format below. Then close every
+4. If you were given `known.md`, read it before naming anything. A problem
+   on it that you see again is reported under **the same id**, briefly, with
+   this run's evidence. Do not go looking for it beyond what the Must lines
+   take you to; spend your attention on what is not on the list.
+5. Write **REPORT.md** in the room, in the format below. Then close every
    screen you opened (`screen_close`).
 
+**When the room worked from a manual** rather than a skill, it played a
+first-time user with only the docs. Look in particular for:
+
+- **the docs and the app disagree**: a key, a menu name or a setting that is
+  not what the app has, steps in the wrong order, a description out of date;
+- **the docs are missing something**: every place the room had to guess.
+  A guess that came from habit (vim keys, the usual shortcut) and happened to
+  work is still a gap - the docs did not say it;
+- **the app itself**: an unhelpful error, a destructive operation without a
+  confirmation, state the screen does not show.
+
+You are not a human tester: you do not see contrast or a confusing layout the
+way a person does. Report what the record shows.
+
 **Friction is not a problem by itself.** A deploy's own "DEPLOY FAILED" on a
 screen built to show a failed deploy is the task working. Report what a
 product change would improve, and say why.
@@ -76,6 +122,7 @@ One paragraph: what the room was asked, what it did, and what it left.
 - **Evidence:** <tool output quoted, or transcript step and what it shows>
 - **Where it would be fixed:** <file and section, in the product>
 - **How a fix is confirmed:** <the evidence that would show it fixed>
+- **Reproduce:** <optional: the steps in the record - transcript turns, live steps 3-7>
 
 <Free text: what happened, and why it matters.>
 ```
@@ -89,14 +136,19 @@ One paragraph: what the room was asked, what it did, and what it left.
 | medium | Met, but only by guessing something the product should have said, or after rework a user's agent would repeat. |
 | low | Friction that cost little: an empty search, a retry, an unclear message. |
 
-**Kind**: `skill gap` (the product's agent-facing text should have said it),
+**Kind**: `skill gap` (the product's text - its skill, tool descriptions or
+manual - should have said it, or says something the product does not do),
 `product` (the product behaves wrongly), `harness` (the runner, or the room's
 tools), `out of scope` (true of the task or the application, not of the
 product).
 
 **Id** names the problem, not the run: `fn-f1-not-bound`, not `room-3-issue`.
-A later run reports the same problem under the same id, which is how a fix is
-seen to have worked.
+A later run reports the same problem under the same id, which is how the same
+problem is recognised across runs.
+
+**No problem without evidence.** Every one cites what a tool answered, a
+transcript step, or a live step and its screen, so a person can see quickly
+whether it is real. A screen you may have misread is said to be uncertain.
 
 There is no verdict. Write problems, most severe first. If you found none,
 say so in the opening paragraph and say what you checked. Zero problems is

{quality_loop-0.1.0 → quality_loop-0.2.1}/quality_loop/__init__.py

@@ -12,4 +12,4 @@ room is handed, putting the user's machine back - is code here, not an
 agent's discretion.
 """
 
-__version__ = "0.1.0"
+__version__ = "0.2.1"

quality_loop-0.2.1/quality_loop/__main__.py

@@ -0,0 +1,10 @@
+"""``python -m quality_loop``: the same as the ``quality-loop`` command.
+
+See `quality_loop.cli`. The operator's ``screen`` subcommand is one of them:
+everything it takes is a name from the manifest the runner wrote.
+"""
+
+from quality_loop.cli import main
+
+if __name__ == "__main__":
+    raise SystemExit(main())

{quality_loop-0.1.0 → quality_loop-0.2.1}/quality_loop/adapter.py

@@ -33,6 +33,20 @@ class Adapter:
             arguments of its choosing and no shell.
         evidence_help: One paragraph on what those tools answer.
         python: The interpreter the operator's CLIs run under.
+        bundle_place: The directory the bundle goes under in a room:
+            ``skills`` for an agent-facing skill, ``docs`` for a manual.
+        apps: ``{name: drive.TerminalApp | drive.DesktopApp}`` - the product's
+            app, launched by the framework (see `quality_loop.drive`). The
+            operator drives them in a scenario's live check.
+        room_apps: Names from ``apps`` the room itself may drive, for a
+            human-facing product whose UI is the surface under test. Empty
+            for an agent-facing one: there, driving the app is for checking.
+        default_variant: What the variant ``""`` is called on the command
+            line (``--variant``), e.g. Keyhac's ``template`` config.
+        parallel: How many rooms may run at once. 1, the default, is one
+            after another, exactly as before parallel runs existed.
+        resources: ``{resource: capacity}`` - what rooms share, and how many
+            may hold each at once: a desktop's focus, the user's home.
     """
     name = "product"
     version = "0"
@@ -45,6 +59,12 @@ class Adapter:
     evidence: dict = {}
     evidence_help = ""
     python = sys.executable
+    bundle_place = "skills"
+    apps: dict = {}
+    room_apps: tuple = ()
+    default_variant = ""
+    parallel = 1
+    resources: dict = {}
 
     def bundle(self) -> pathlib.Path:
         """The surface under test - a zip or a directory - built for this run."""
@@ -86,3 +106,19 @@ class Adapter:
     def alive(self) -> bool:
         """Whether the product is still answering, asked on the heartbeat."""
         return True
+
+    def needs(self, case: scenario.Scenario, variant: str) -> set[str]:
+        """The resources one room holds while it runs. The default is every
+        declared resource, so declaring a resource without saying who needs
+        it never lets two rooms share it."""
+        return set(self.resources)
+
+    def product_for(self, room: pathlib.Path):
+        """A context manager for one room: a product instance of its own, for
+        a product that can run one per room. None uses ``product()``'s."""
+        return None
+
+    def restore(self) -> list[str]:
+        """Put back what a run that could not unwind left set aside - the
+        stand-in's marker. One line per thing done."""
+        raise NotImplementedError("this adapter does not say how to restore")

{quality_loop-0.1.0 → quality_loop-0.2.1}/quality_loop/audit.py

@@ -81,6 +81,10 @@ class Policy:
             the room reported on the outage, not the product.
         leaked_skill: Says whether a slash command in the init event is one of
             the checkout's own skills - the scoring key among them.
+        run_directory: The run's own directory - run.log, status.json, the
+            other rooms and their operators. The runner sets it. A path in it
+            and outside the room is a breach: a room once read its own log to
+            reconstruct its task, and a sibling's report is the answer key.
     """
     checkout: pathlib.Path
     forbidden: dict[str, str] = dataclasses.field(default_factory=dict)
@@ -88,6 +92,14 @@ class Policy:
     bash: bool = False
     servers: tuple[str, ...] = ()
     leaked_skill: Callable[[str], bool] | None = None
+    run_directory: pathlib.Path | None = None
+
+
+def under(path: str, root: str) -> bool:
+    """Whether ``path`` is ``root`` or inside it - not merely spelled like it:
+    a room ``11_f1`` is not inside ``11_f1.operator``."""
+    root = root.rstrip("/")
+    return path == root or path.startswith(root + "/")
 
 
 def isolation_held(transcript: pathlib.Path, policy: Policy) -> list[str]:
@@ -124,6 +136,8 @@ def audit(room: pathlib.Path, policy: Policy) -> tuple[list[str], list[str]]:
     breaches = isolation_held(transcript, policy)
     refused: list[str] = []
     checkout, inside = norm(str(policy.checkout.resolve())), norm(str(room))
+    records = (norm(str(pathlib.Path(policy.run_directory).resolve()))
+               if policy.run_directory else None)
     forbidden = {norm(path): what for path, what in policy.forbidden.items()}
     for call in tr.tool_calls(transcript):
         landed = refused if call.is_error else breaches
@@ -137,8 +151,10 @@ def audit(room: pathlib.Path, policy: Policy) -> tuple[list[str], list[str]]:
             for path, what in forbidden.items():
                 if path in spelled:
                     landed.append(f"{call.name} reached into {what}: {aimed}")
-            if spelled.startswith(checkout) and not spelled.startswith(inside):
+            if under(spelled, checkout) and not under(spelled, inside):
                 landed.append(f"{call.name} reached into the checkout: {aimed}")
+            elif records and under(spelled, records) and not under(spelled, inside):
+                landed.append(f"{call.name} reached into the run's own records: {aimed}")
             if pathlib.PurePath(aimed.replace("\\", "/")).name in policy.foreign:
                 landed.append(f"{call.name} read another author's work: {aimed}")
     return sorted(set(breaches)), sorted(set(refused))