quickstarted 0.2.0__tar.gz → 0.3.0__tar.gz

{quickstarted-0.2.0 → quickstarted-0.3.0}/.gitignore

@@ -12,3 +12,6 @@ htmlcov/
 .DS_Store
 site/
 .cache/
+bench/
+bench.old/
+bench*/

quickstarted-0.3.0/CHANGELOG.md

@@ -0,0 +1,145 @@
+# Changelog
+
+All notable changes to this project are documented here. This project follows
+[Semantic Versioning](https://semver.org/).
+
+## [0.3.0] - 2026-07-26
+
+Breaking, and deliberately early. The unit of testing is now a **task**, not a
+journey.
+
+### Changed
+
+- **`journey` is now `task`**, everywhere: the `journeys/` directory is
+  `tasks/`, `load_journey` is `load_task`, `run_journey` is `run_task`, the
+  `Journey` dataclass is `Task`, and `JourneyError` is `TaskError`.
+  "Journey" came from analytics funnels and describes the wrong shape. A task
+  is a goal plus one exit code, and "can an agent complete this task from your
+  docs" is the claim the tool actually tests.
+- **`results.json` is schema 2.0**: the `journey` and `journeys` keys are now
+  `task` and `tasks`. Nothing else about the document changed, so a 1.0
+  consumer needs only that substitution.
+- `quickstarted validate` and `quickstarted run` still accept paths under
+  `journeys/` and read the matching file in `tasks/`, printing a warning. That
+  fallback is removed in 0.4.0.
+
+### Added
+
+- **Per-task container image** (`image:` in a task file). The default
+  `python:3.12-slim` has no Node, so a suite could not mix a Python quickstart
+  with a JavaScript one; `--image` is one flag for a whole invocation.
+  Precedence is task, then `--image`, then the default. The resolved image is
+  recorded in `results.json` and printed beside the backend, because a pass
+  rate is not comparable across base images.
+- `validate` warns when a success script calls `npm`, `npx`, `node`, `pnpm`,
+  `yarn`, or `bun` while no image is set, since that check would fail for a
+  reason unrelated to the documentation.
+- **Six more tasks**, doubling the set to 14 and covering both runtimes: `uv`,
+  `django`, `prefect` (Python) and `vite`, `prisma`, `tailwind` (Node).
+- Four hosts added to the default network allowlist:
+  `release-assets.githubusercontent.com`, where npm packages with prebuilt
+  native binaries fetch during install, and the Debian and Ubuntu mirrors, so a
+  quickstart that opens with `apt-get install` works. `raw.githubusercontent.com`
+  stays out on purpose: projects serve READMEs from it, and a documentation host
+  the shell can reach is an attribution hole.
+
+### Fixed
+
+- **Client-side redirects are followed.** DuckDB answers its own versioned doc
+  URLs with 938 bytes of HTML that render as 73 characters of "Redirecting...",
+  naming the real page in a `<meta http-equiv="refresh">`. The fetcher handed
+  those 73 characters to the agent as though they were documentation, so the
+  duckdb task was measuring recovery from an empty page. A browser follows the
+  refresh, so an agent with one reads the docs and an agent without reads
+  nothing; following it keeps the two comparable. The hop is taken only within
+  the same host, since a stub pointing elsewhere would read a page the task
+  never allowlisted, and both URLs land in the trace as
+  `docs_redirect_followed`. The same entrypoint now yields 21,659 characters.
+- **Docker workspaces no longer leak into the VM (macOS).** The sandbox root
+  came from `tempfile.mkdtemp()`, which on macOS returns a path under
+  `/var/folders`. That is not shared with the Docker VM, so `-v` created a
+  directory of the same name *inside* the VM and mounted that instead. Runs
+  passed, but the host saw an empty workspace, `--keep-sandbox` handed back
+  nothing, and cleanup freed no bytes. Found by filling a 19 GB Colima disk with
+  132 orphaned workspaces, after which every run failed with `no space left on
+  device` and was correctly classified `harness_error`. The root now defaults to
+  `~/.quickstarted/sandboxes` on macOS, overridable with
+  `QUICKSTARTED_SANDBOX_DIR`, and the mount is probed in both directions at
+  startup so a path the daemon cannot see is refused immediately.
+- **The workspace starts empty.** `HOME` and `TMPDIR` used to point inside the
+  workspace, so `.npm`, `.cache` and `tmp/` were there before the agent ran a
+  single command, and every scaffolding tool that requires an empty directory
+  refused: `npm create vite@latest .`, `django-admin startproject .`,
+  `cargo new`. An agent had to notice and work around a mess the harness made,
+  which is measurement noise dressed up as a documentation problem. Under
+  `docker` both now live on the container's own filesystem; under `seatbelt` and
+  `local` they are siblings of the workspace inside the same sandbox, so the
+  kernel policy is unchanged.
+
+## [0.2.0] - 2026-07-26
+
+The v0.1 harness could be talked out of its own guarantees: the docs allowlist
+lived in the `read_docs` tool, so any agent that ran `curl` read pages the
+trace never saw, and commands ran unconfined on the host. This release makes
+the boundary real and makes the output publishable.
+
+### Added
+
+- **Execution backends** (`quickstarted.exec`): `docker`, `seatbelt` (macOS), and
+  `local`. `auto` prefers an enforced backend. `quickstarted run` refuses the
+  unenforced one unless `--allow-unenforced` is given. Both enforced backends
+  are verified against live daemons, including tests asserting that a direct
+  connection bypassing the proxy fails.
+- **Egress proxy** (`quickstarted.net`): all shell traffic leaves through a
+  harness-owned proxy. Documentation hosts are unreachable from the shell, so
+  the recorded set of pages read is complete; attempts to bypass are counted.
+  `network.allow` in a journey names hosts a shell may reach for installs.
+- **Run classification**: `passed`, `docs_gap`, `budget_exhausted`,
+  `infra_error`, `harness_error`, `agent_refusal`. Only the first two are
+  evidence about documentation. The rest are excluded from pass rates, and
+  reported separately.
+- **Pass rates**: `--repeat N` and `--workers N`. A single run is one sample,
+  and the rate over several is the number worth reading.
+- **Affordance policy**: `--affordances all|none` withholds `llms.txt` and
+  `.md` variants so the same journey can be run both ways and the difference
+  measured. `--probe-affordances` records what exists. Never scored.
+- **Docs cache and politeness**: `--cache-dir`, `--refresh` (flags content
+  that changed between runs), `--offline`, `--rate-limit`, `--ignore-robots`.
+  Truthful User-Agent, robots.txt honoured by default.
+- **Machine-readable output**: versioned `results.json` (schema 1.0) and
+  `--junit` XML, where a docs gap is a failure and an infrastructure problem is
+  an error.
+- **More agents**: `openai` and `gemini` adapters alongside `claude`, sharing
+  one prompt so cross-model numbers compare like with like. Neither assumes a
+  default model. Token counts are normalised across vendors: OpenAI and Google
+  report a prompt total that includes cached tokens while Anthropic excludes
+  them, so adapters subtract the overlap and the four counters never bill the
+  same token twice.
+- **Retries** with exponential backoff on transient upstream faults. Every wait
+  lands in the trace instead of being absorbed silently by the SDK.
+- **Token budgets** (`budgets.max_tokens`) and optional cost estimation from a
+  price book you supply (`--prices`). No prices ship with the tool.
+- `quickstarted doctor`, reporting what this machine can enforce.
+- Tests: proxy policy, Seatbelt confinement, classification, statistics,
+  caching, robots, and the affordance ablation.
+
+### Changed
+
+- Cached prompt tokens are counted. Previously a run costing ~120k cache-read
+  tokens reported "22 in" and looked free.
+- The Claude adapter defaults to `claude-opus-5` and records the exact model
+  the API served, since an alias can change under a benchmark.
+- Agents are told what `setup` already did, so they stop rebuilding a
+  virtualenv that exists.
+- Journeys no longer list package registries under `docs.allow`; `validate`
+  warns when they do, because the proxy will refuse `pip`.
+
+### Fixed
+
+- `Trace` is thread-safe; the proxy appends from its own threads.
+
+## [0.1.0] - 2026-07-24
+
+Initial release: journey spec, replay and Claude agent modes, sandboxed
+execution, docs fetch with attribution, deterministic scoring, CLI and
+GitHub Action.

{quickstarted-0.2.0 → quickstarted-0.3.0}/CONTRIBUTING.md

@@ -13,14 +13,14 @@ python3 -m venv .venv
 Tests that touch the network are skipped unless `QUICKSTARTED_NETWORK_TESTS=1`
 is set. No test needs an API key.
 
-## Writing a journey
+## Writing a task
 
-A good journey asserts an outcome the documentation actually promises, and
+A good task asserts an outcome the documentation actually promises, and
 nothing else.
 
 - **Assert the data.** Check that `out.csv` holds the right rows. Do not check
   which function the agent called; any correct route should pass.
-- **Do not assert incidental paths.** An early journey checked for a `.venv`
+- **Do not assert incidental paths.** An early task checked for a `.venv`
   the docs never mention; an agent that made `venv/` "failed" for no reason.
   If `setup` creates something, the agent is told, and the check may rely on
   it.
@@ -31,12 +31,12 @@ nothing else.
   through `read_docs`; a shell cannot reach them. Putting `pypi.org` there
   stops `pip` working. `quickstarted validate` warns about this.
 
-Run `quickstarted run <journey> --agent replay` first. If the documented
-commands do not pass, the journey is not ready for agent mode.
+Run `quickstarted run <task> --agent replay` first. If the documented
+commands do not pass, the task is not ready for agent mode.
 
 ## Adding an agent adapter
 
-Implement `run(journey, toolbelt, deadline) -> AgentOutcome` and act only
+Implement `run(task, toolbelt, deadline) -> AgentOutcome` and act only
 through the toolbelt: never touch the filesystem or network directly, or the
 run loses the attribution and enforcement the whole tool depends on.
 

{quickstarted-0.2.0 → quickstarted-0.3.0}/DESIGN.md

@@ -8,7 +8,7 @@ verdict. LLM judges may eventually classify WHY a run failed; they will never
 decide WHETHER it failed. This keeps the tool falsifiable and keeps vendors
 (including Anthropic) out of the scoring path.
 
-## Attribution has to be enforced, not requested
+## Every page read goes through a tool the agent cannot bypass
 
 Agents act only through the Toolbelt (`bash` + `read_docs`), so every page read
 is recorded and failures attribute to a page.
@@ -26,7 +26,7 @@ pages the agent read.
 This creates one deliberate asymmetry: `docs.allow` hosts are readable only via
 `read_docs`, and `network.allow` hosts (package registries) are reachable only
 from the shell. When a host is genuinely both, PyPI being the obvious case, the
-journey says so explicitly and installs win. The resulting gap in attribution
+task says so explicitly and installs win. The resulting gap in attribution
 is reported alongside the result.
 
 ## Backends, and saying which one ran
@@ -65,7 +65,7 @@ measurement of the affordance itself.
 That question is currently argued without data. This is the only shape of tool
 that can answer it, because answering it requires running the task.
 
-## Replay mode is the floor, not a feature
+## Replay mode is a precondition
 
 Replay (documented commands verbatim, no LLM) is free, deterministic and
 CI-friendly, and it defines a floor: if replay fails, agent mode is noise. It
@@ -97,8 +97,8 @@ would quietly misreport what a sweep costs.
   out at all, so the isolation does not depend on name resolution.
 - HTML-to-text is a crude tag stripper, so markdown-native docs read better.
   That is itself a finding worth publishing.
-- Concurrency is per-attempt threads, not a scheduler; hosts are rate-limited
-  individually but there is no global budget governor.
+- Concurrency is per-attempt threads. Hosts are rate-limited individually, and
+  there is no global budget governor.
 - No pip/npm cache; every run cold-installs.
 - The Gemini adapter is written against the same Toolbelt contract but has not
   been run live; Claude and OpenAI have.

{quickstarted-0.2.0 → quickstarted-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: quickstarted
-Version: 0.2.0
+Version: 0.3.0
 Summary: Test whether an AI agent can actually complete your quickstart by following your docs
 Project-URL: Homepage, https://github.com/snehankekre/quickstarted
 Project-URL: Issues, https://github.com/snehankekre/quickstarted/issues
@@ -66,84 +66,106 @@ Full documentation: **https://snehankekre.com/quickstarted/**
 
 ## How it works
 
-You write a journey, in YAML:
+You write a task, in YAML. This one points at Streamlit's documentation:
 
 ```yaml
-name: pnf-quickstart
+name: streamlit-quickstart
 goal: >
-  Install the pnf package from PyPI into a virtualenv and write a runnable
-  snippet that imports pnf and identifies the decode function.
+  Install Streamlit into the virtualenv in this workspace and build a small app
+  in app.py that shows a title and a slider whose value is displayed on the
+  page. Do not start a long-running server yourself; just leave app.py ready to
+  run and confirm streamlit is installed.
 docs:
-  entrypoint: https://raw.githubusercontent.com/snehankekre/pnf/main/README.md
+  entrypoint: https://docs.streamlit.io/get-started
   allow:
-    - raw.githubusercontent.com
+    - docs.streamlit.io
+    - streamlit.io
 setup:
   - python3 -m venv .venv
 success:
   script: |
     set -e
-    .venv/bin/python -c "import pnf; assert callable(pnf.decode)"
-    .venv/bin/python demo.py
-replay:
-  - .venv/bin/pip install pnf
-  - printf 'import pnf\nprint(pnf.decode.__name__)\n' > demo.py
-  - .venv/bin/python demo.py
+    test -f app.py
+    .venv/bin/python -c "import streamlit"
 ```
 
-The harness runs an agent in a throwaway workspace with two tools: a sandboxed
-`bash`, and `read_docs`, which is the only way it can read documentation.
-
-The harness enforces that. All shell traffic leaves through a proxy it owns,
-and documentation hosts are unreachable from the shell, so an agent that tries
-`curl https://docs.example.com/...` is refused and the attempt is recorded.
-The pages listed in a report are therefore the pages the agent really read.
-When a run fails, the last page it read is a fact you can act on.
-
-When the agent stops, the harness runs your `success.script` in the workspace.
-Exit code 0 is a pass. Nothing else is. The agent's own claim of success is
-recorded and never trusted; there is no LLM judge anywhere in scoring.
-
-## Two modes
-
-**Replay mode** (free, no LLM, no API key) runs your `replay` commands, the
-literal commands your docs tell users to type, and stops at the first failure.
-Treat it as a precondition. If the documented commands break, no reader stands
-a chance, and agent mode only adds noise.
+Then you run an agent against it:
 
 ```
-quickstarted run journeys/pnf-quickstart.yaml --agent replay
+pip install "quickstarted[claude]"
+export QUICKSTARTED_ANTHROPIC_API_KEY=...
+quickstarted run tasks/streamlit-quickstart.yaml --agent claude
 ```
 
-**Agent mode** has an LLM follow your docs to the goal. This catches what
-replay cannot: ambiguity, missing steps, wrong ordering, docs that assume
-context a newcomer does not have.
-
 ```
-pip install "quickstarted[claude]"
-export QUICKSTARTED_ANTHROPIC_API_KEY=...
-quickstarted run journeys/pnf-quickstart.yaml --agent claude
+[PASS] streamlit-quickstart (claude:claude-opus-5)
+  classification: passed
+  turns: 6, duration: 63.5s
+  backend: docker
+  tokens: 12 in / 1150 out, cache 11204 written / 33181 read
+  docs pages read: 4
 ```
 
+The agent works in a throwaway workspace with two tools: a sandboxed `bash`,
+and `read_docs`, which is the only way it can read documentation. It gets no
+browser, no search engine, and no network of its own. The system prompt also
+forbids leaning on prior knowledge of the project under test, because a model
+that already knows Streamlit would sail through docs that teach a newcomer
+nothing.
+
+The harness enforces the boundary rather than requesting it. All shell traffic
+leaves through a proxy the harness owns, and documentation hosts are
+unreachable from the shell, so an agent that tries
+`curl https://docs.streamlit.io/...` is refused and the attempt is recorded.
+The pages in a report are therefore the pages the agent really read, and when a
+run fails, the last page it read is a fact rather than a guess.
+
 Keys are read from `QUICKSTARTED_*` names first so they can sit in your shell
 without other tooling on the same machine picking them up and spending them.
 The vendor-standard names still work as a fallback, which is what CI sets.
 
+## The success script is the whole verdict
+
+When the agent stops, the harness runs your `success.script` in the same
+workspace. Exit code 0 is a pass. Nothing else is. The agent's own claim of
+success is recorded and never trusted, and there is no LLM judge anywhere in
+scoring, because a model asked to grade its own work will tell you the app is
+ready when `app.py` does not parse.
+
+Most checks are the size of the one above. You are asserting whatever your
+tutorial already promises the reader: the file exists, the import works, the
+command runs, the output contains the number on the page. If your quickstart
+ends with "you should see `200`", the check is `grep -q 200`.
+
+You can go stricter when it is worth it.
+[tasks/streamlit-quickstart.yaml](tasks/streamlit-quickstart.yaml) boots
+the app headless and polls Streamlit's own health endpoint, which is about
+fifteen more lines and proves the app actually serves. That is optional. Start
+with the two-line version.
+
+## Replay mode, the free precondition
+
+`--agent replay` runs your `replay` commands, the literal commands your docs
+tell a reader to type, and stops at the first failure. No model, no API key, no
+cost.
+
 ```
-[PASS] pnf-quickstart (claude:claude-opus-5)
-  classification: passed
-  turns: 12, duration: 70.7s
-  backend: seatbelt
-  tokens: 24 in / 4616 out, cache 16812 written / 104148 read
-  docs pages read: 2
+quickstarted run tasks/streamlit-quickstart.yaml --agent replay
 ```
 
+Treat it as a floor to check on every push. If the documented commands are
+broken, no reader stands a chance and an agent run only adds noise on top of a
+failure you already know about. It cannot tell you whether a reader could have
+*found* those commands, understood their order, or guessed the prerequisite you
+left out. That is what agent mode is for.
+
 ## Pass rates
 
 The same docs and the same model can pass at 10:00 and fail at 10:05. One run
 is a sample. Use `--repeat` and read the rate:
 
 ```
-quickstarted run journeys/*.yaml --agent claude --repeat 5 --workers 3
+quickstarted run tasks/*.yaml --agent claude --repeat 5 --workers 3
 ```
 
 Every run is classified, and only two classifications say anything about your
@@ -155,7 +177,7 @@ documentation:
 | `docs_gap` | the agent finished, the check failed | yes |
 | `budget_exhausted` | out of turns, time, or tokens | no |
 | `infra_error` | rate limit, upstream 5xx, network failure | no |
-| `harness_error` | misconfigured journey, our bug | no |
+| `harness_error` | misconfigured task, our bug | no |
 | `agent_refusal` | model declined | no |
 
 Runs that produced no evidence are excluded from the numerator *and* the
@@ -169,11 +191,11 @@ scoring it would be the proxy metric this tool exists to replace. So
 quickstarted never scores affordances. It measures them:
 
 ```
-quickstarted run journeys/streamlit-quickstart.yaml --agent claude --repeat 10
-quickstarted run journeys/streamlit-quickstart.yaml --agent claude --repeat 10 --affordances none
+quickstarted run tasks/streamlit-quickstart.yaml --agent claude --repeat 10
+quickstarted run tasks/streamlit-quickstart.yaml --agent claude --repeat 10 --affordances none
 ```
 
-Same prompt, same journey, same model. The only difference is whether the
+Same prompt, same task, same model. The only difference is whether the
 agent could read `llms.txt` and `.md` variants. The difference in pass rate is
 a measurement of the affordance itself.
 
@@ -199,9 +221,9 @@ can enforce. Details in [SECURITY.md](SECURITY.md).
 ## Install
 
 ```
-pip install quickstarted                 # replay mode only
-pip install "quickstarted[claude]"       # + Claude
+pip install "quickstarted[claude]"       # agent mode with Claude
 pip install "quickstarted[all-agents]"   # + OpenAI and Gemini
+pip install quickstarted                 # replay mode only, no SDK
 ```
 
 Python 3.9+. One runtime dependency: PyYAML. Every adapter is a plain tool-use
@@ -210,25 +232,43 @@ cross-model numbers compare like with like.
 
 ## CI
 
+Two jobs, on two schedules. Replay on every push, because it is free:
+
 ```yaml
-- uses: actions/checkout@v4
-- uses: actions/setup-python@v5
-  with:
-    python-version: "3.12"
 - run: pip install quickstarted
-- run: quickstarted run journeys/*.yaml --agent replay --out results --junit junit.xml
-- uses: actions/upload-artifact@v4
-  if: always()
-  with:
-    name: quickstarted-results
-    path: results/
+- run: quickstarted run tasks/*.yaml --agent replay --junit junit.xml
+```
+
+Agent mode on a schedule, or when a model ships, because it costs real tokens:
+
+```yaml
+on:
+  schedule:
+    - cron: "0 6 * * 1"
+  workflow_dispatch:
+jobs:
+  agent:
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install "quickstarted[claude]"
+      - run: quickstarted run tasks/*.yaml --agent claude --repeat 3
+             --out results --junit junit.xml
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: quickstarted-results
+          path: results/
 ```
 
-`quickstarted run` exits 1 when any journey fails, so the job gates merges.
-`results.json` is versioned (schema 1.0). JUnit XML reports a docs gap as a
-failure and infrastructure trouble as an error, so a rate limit does not read
-as a broken quickstart. Run agent mode on a schedule, or when a model ships.
-Every push is too often, because agent runs cost real tokens.
+`quickstarted run` exits 1 when any task fails, so either job can gate
+merges. `results.json` is versioned (schema 1.0). JUnit XML reports a docs gap
+as a failure and infrastructure trouble as an error, so a rate limit does not
+read as a broken quickstart.
 
 ## Fetching other people's docs
 
@@ -243,7 +283,7 @@ last run, which is a finding in its own right.
 It tells you whether a given model, on a given day, could get from your docs to
 a working result, and where it was reading when it could not. That is a lower
 bound on documentation quality. Nothing here measures whether your docs are
-pleasant, complete, or accurate beyond the journey you wrote, and there is no
+pleasant, complete, or accurate beyond the task you wrote, and there is no
 UI testing at all: if your product's button moved, this will not notice.
 [Doc Detective](https://github.com/doc-detective/doc-detective) is the tool for
 that, and the two answer different questions.