quickstarted 0.2.0__tar.gz

quickstarted-0.2.0/.gitignore

@@ -0,0 +1,14 @@
+__pycache__/
+*.pyc
+.venv/
+dist/
+build/
+*.egg-info/
+.pytest_cache/
+quickstarted-results/
+.coverage
+coverage.xml
+htmlcov/
+.DS_Store
+site/
+.cache/

quickstarted-0.2.0/CHANGELOG.md

@@ -0,0 +1,72 @@
+# Changelog
+
+All notable changes to this project are documented here. This project follows
+[Semantic Versioning](https://semver.org/).
+
+## [0.2.0] - 2026-07-26
+
+The v0.1 harness could be talked out of its own guarantees: the docs allowlist
+lived in the `read_docs` tool, so any agent that ran `curl` read pages the
+trace never saw, and commands ran unconfined on the host. This release makes
+the boundary real and makes the output publishable.
+
+### Added
+
+- **Execution backends** (`quickstarted.exec`): `docker`, `seatbelt` (macOS), and
+  `local`. `auto` prefers an enforced backend. `quickstarted run` refuses the
+  unenforced one unless `--allow-unenforced` is given. Both enforced backends
+  are verified against live daemons, including tests asserting that a direct
+  connection bypassing the proxy fails.
+- **Egress proxy** (`quickstarted.net`): all shell traffic leaves through a
+  harness-owned proxy. Documentation hosts are unreachable from the shell, so
+  the recorded set of pages read is complete; attempts to bypass are counted.
+  `network.allow` in a journey names hosts a shell may reach for installs.
+- **Run classification**: `passed`, `docs_gap`, `budget_exhausted`,
+  `infra_error`, `harness_error`, `agent_refusal`. Only the first two are
+  evidence about documentation. The rest are excluded from pass rates, and
+  reported separately.
+- **Pass rates**: `--repeat N` and `--workers N`. A single run is one sample,
+  and the rate over several is the number worth reading.
+- **Affordance policy**: `--affordances all|none` withholds `llms.txt` and
+  `.md` variants so the same journey can be run both ways and the difference
+  measured. `--probe-affordances` records what exists. Never scored.
+- **Docs cache and politeness**: `--cache-dir`, `--refresh` (flags content
+  that changed between runs), `--offline`, `--rate-limit`, `--ignore-robots`.
+  Truthful User-Agent, robots.txt honoured by default.
+- **Machine-readable output**: versioned `results.json` (schema 1.0) and
+  `--junit` XML, where a docs gap is a failure and an infrastructure problem is
+  an error.
+- **More agents**: `openai` and `gemini` adapters alongside `claude`, sharing
+  one prompt so cross-model numbers compare like with like. Neither assumes a
+  default model. Token counts are normalised across vendors: OpenAI and Google
+  report a prompt total that includes cached tokens while Anthropic excludes
+  them, so adapters subtract the overlap and the four counters never bill the
+  same token twice.
+- **Retries** with exponential backoff on transient upstream faults. Every wait
+  lands in the trace instead of being absorbed silently by the SDK.
+- **Token budgets** (`budgets.max_tokens`) and optional cost estimation from a
+  price book you supply (`--prices`). No prices ship with the tool.
+- `quickstarted doctor`, reporting what this machine can enforce.
+- Tests: proxy policy, Seatbelt confinement, classification, statistics,
+  caching, robots, and the affordance ablation.
+
+### Changed
+
+- Cached prompt tokens are counted. Previously a run costing ~120k cache-read
+  tokens reported "22 in" and looked free.
+- The Claude adapter defaults to `claude-opus-5` and records the exact model
+  the API served, since an alias can change under a benchmark.
+- Agents are told what `setup` already did, so they stop rebuilding a
+  virtualenv that exists.
+- Journeys no longer list package registries under `docs.allow`; `validate`
+  warns when they do, because the proxy will refuse `pip`.
+
+### Fixed
+
+- `Trace` is thread-safe; the proxy appends from its own threads.
+
+## [0.1.0] - 2026-07-24
+
+Initial release: journey spec, replay and Claude agent modes, sandboxed
+execution, docs fetch with attribution, deterministic scoring, CLI and
+GitHub Action.

quickstarted-0.2.0/CONTRIBUTING.md

@@ -0,0 +1,56 @@
+# Contributing
+
+## Setup
+
+```
+python3 -m venv .venv
+.venv/bin/pip install -e ".[dev]"
+.venv/bin/pytest
+.venv/bin/ruff check .
+.venv/bin/mypy
+```
+
+Tests that touch the network are skipped unless `QUICKSTARTED_NETWORK_TESTS=1`
+is set. No test needs an API key.
+
+## Writing a journey
+
+A good journey asserts an outcome the documentation actually promises, and
+nothing else.
+
+- **Assert the data.** Check that `out.csv` holds the right rows. Do not check
+  which function the agent called; any correct route should pass.
+- **Do not assert incidental paths.** An early journey checked for a `.venv`
+  the docs never mention; an agent that made `venv/` "failed" for no reason.
+  If `setup` creates something, the agent is told, and the check may rely on
+  it.
+- **Let the harness verify.** If the goal is a running server, the success
+  script starts it, asks it a question, and stops it. Do not trust the agent's
+  report, and do not ask the agent to leave something running.
+- **Separate docs hosts from registries.** `docs.allow` hosts are readable only
+  through `read_docs`; a shell cannot reach them. Putting `pypi.org` there
+  stops `pip` working. `quickstarted validate` warns about this.
+
+Run `quickstarted run <journey> --agent replay` first. If the documented
+commands do not pass, the journey is not ready for agent mode.
+
+## Adding an agent adapter
+
+Implement `run(journey, toolbelt, deadline) -> AgentOutcome` and act only
+through the toolbelt: never touch the filesystem or network directly, or the
+run loses the attribution and enforcement the whole tool depends on.
+
+Use `agents/prompt.py` unchanged. Adapters that phrase the task their own way
+turn a cross-model comparison into a comparison of prompts.
+
+Do not assume a default model for a new vendor. Model names change faster than
+a pinned default survives, and a benchmark that silently picks one produces
+numbers nobody can reproduce.
+
+## Things that will get a change rejected
+
+- An LLM anywhere in the scoring path.
+- Scoring the presence of `llms.txt`, MCP, or any other affordance. Measure it
+  with an ablation instead.
+- Counting infrastructure failures as documentation failures.
+- Making the unenforced backend the silent default.

quickstarted-0.2.0/DESIGN.md

@@ -0,0 +1,106 @@
+# Design notes
+
+## The one decision everything follows from
+
+Deterministic assertions are the only pass/fail truth. The success script runs
+after the agent stops, in the same workspace, and its exit code is the
+verdict. LLM judges may eventually classify WHY a run failed; they will never
+decide WHETHER it failed. This keeps the tool falsifiable and keeps vendors
+(including Anthropic) out of the scoring path.
+
+## Attribution has to be enforced, not requested
+
+Agents act only through the Toolbelt (`bash` + `read_docs`), so every page read
+is recorded and failures attribute to a page.
+
+v0.1 stopped there, and that was not enough. The allowlist lived in the
+`read_docs` tool while `bash` had unrestricted network access, so any agent
+that ran `curl` read documentation the trace never saw. The guarantee held only
+while the agent cooperated, which is not a guarantee.
+
+Now all shell traffic leaves through a proxy the harness owns, and
+documentation hosts are unreachable from the shell. An attempted `curl` to a
+docs host is refused and counted. The set of pages in the report is the set of
+pages the agent read.
+
+This creates one deliberate asymmetry: `docs.allow` hosts are readable only via
+`read_docs`, and `network.allow` hosts (package registries) are reachable only
+from the shell. When a host is genuinely both, PyPI being the obvious case, the
+journey says so explicitly and installs win. The resulting gap in attribution
+is reported alongside the result.
+
+## Backends, and saying which one ran
+
+Three: `docker` (container on an internal network whose only route out is the
+proxy sidecar), `seatbelt` (macOS, kernel-enforced, all egress denied except
+the proxy port, home directory unreadable), and `local` (nothing enforced).
+
+`local` is refused unless explicitly allowed, every result records which
+backend ran and whether it was enforced, and `quickstarted doctor` says what a
+machine can do before anyone trusts its numbers. An unenforced result still
+tells you something. It just supports a weaker claim, and the claim should
+travel with the number.
+
+## A run is a sample; a rate is a measurement
+
+The same docs and model can pass and then fail. `--repeat` produces a pass
+rate, and every run is classified. Only `passed` and `docs_gap` are statements
+about documentation. Rate limits, exhausted budgets, refusals, and our own bugs
+are excluded from both the numerator and the denominator and reported
+separately.
+
+A sweep of fifty projects will hit 429s and flaky networks. Reporting those as
+failed quickstarts would be a lie that is very easy to tell, and it is the
+difference between a benchmark and a leaderboard-shaped rumour.
+
+## Affordances are measured, never scored
+
+Whether a project ships `llms.txt` is a checklist item anyone can `curl`, and
+scoring it would be the proxy metric this tool exists to replace. So presence
+is recorded as context, with sizes, and the affordance can be *withheld*:
+`--affordances none` blocks `llms.txt` and `.md` variants while keeping the
+prompt byte-identical. Run both conditions and the difference in pass rate is a
+measurement of the affordance itself.
+
+That question is currently argued without data. This is the only shape of tool
+that can answer it, because answering it requires running the task.
+
+## Replay mode is the floor, not a feature
+
+Replay (documented commands verbatim, no LLM) is free, deterministic and
+CI-friendly, and it defines a floor: if replay fails, agent mode is noise. It
+is a weaker `runShell` than a docs-testing framework like Doc Detective
+already offers. Sell it as a precondition, and let agent mode carry the
+product.
+
+## Trace as the product surface
+
+Everything interesting is an event in the JSONL trace: setup, tool calls, docs
+fetches with content hashes, egress decisions, agent turns with token usage,
+retries, the success check. `results.json` (schema 1.0) is the stable contract
+over those traces. The hosted product is, to a first approximation, storage
+plus diffing plus alerting across models and time.
+
+## Cost is reported in tokens
+
+Cached prompt tokens are excluded from `input_tokens` by the API, so summing
+that field alone made a run costing 120k cache-read tokens look free. All four
+counters are tracked. Dollars appear only when the operator supplies a price
+book: vendor rates change, and a stale table baked into a benchmarking tool
+would quietly misreport what a sweep costs.
+
+## Known gaps
+
+- Both enforced backends are verified end to end against live daemons. In the
+  container the isolation is stronger than Seatbelt's: a direct dial to an IP
+  address fails with `Network is unreachable`. The sandbox network has no route
+  out at all, so the isolation does not depend on name resolution.
+- HTML-to-text is a crude tag stripper, so markdown-native docs read better.
+  That is itself a finding worth publishing.
+- Concurrency is per-attempt threads, not a scheduler; hosts are rate-limited
+  individually but there is no global budget governor.
+- No pip/npm cache; every run cold-installs.
+- The Gemini adapter is written against the same Toolbelt contract but has not
+  been run live; Claude and OpenAI have.
+- Nothing distinguishes "the docs are wrong" from "the product is broken". Both
+  surface as `docs_gap` and need a human.

quickstarted-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Snehan Kekre
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

quickstarted-0.2.0/PKG-INFO

@@ -0,0 +1,256 @@
+Metadata-Version: 2.4
+Name: quickstarted
+Version: 0.2.0
+Summary: Test whether an AI agent can actually complete your quickstart by following your docs
+Project-URL: Homepage, https://github.com/snehankekre/quickstarted
+Project-URL: Issues, https://github.com/snehankekre/quickstarted/issues
+Project-URL: Changelog, https://github.com/snehankekre/quickstarted/blob/main/CHANGELOG.md
+Author-email: snehan <snehan.minerva@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agent-experience,agents,ci,devrel,docs,documentation,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Documentation
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.9
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: all-agents
+Requires-Dist: anthropic>=0.40; extra == 'all-agents'
+Requires-Dist: google-genai>=1.0; extra == 'all-agents'
+Requires-Dist: openai>=1.40; extra == 'all-agents'
+Provides-Extra: claude
+Requires-Dist: anthropic>=0.40; extra == 'claude'
+Provides-Extra: dev
+Requires-Dist: anthropic>=0.40; extra == 'dev'
+Requires-Dist: google-genai>=1.0; extra == 'dev'
+Requires-Dist: mypy>=1.11; extra == 'dev'
+Requires-Dist: openai>=1.40; extra == 'dev'
+Requires-Dist: pytest-cov>=5.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Provides-Extra: docs
+Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
+Provides-Extra: gemini
+Requires-Dist: google-genai>=1.0; extra == 'gemini'
+Provides-Extra: openai
+Requires-Dist: openai>=1.40; extra == 'openai'
+Description-Content-Type: text/markdown
+
+# quickstarted
+
+Test whether an AI agent can actually complete your quickstart by following
+your docs.
+
+[![CI](https://github.com/snehankekre/quickstarted/actions/workflows/ci.yml/badge.svg)](https://github.com/snehankekre/quickstarted/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/quickstarted.svg)](https://pypi.org/project/quickstarted/)
+[![Python](https://img.shields.io/pypi/pyversions/quickstarted.svg)](https://pypi.org/project/quickstarted/)
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+Roughly half the traffic to documentation sites is now AI agents and
+AI-assisted workflows ([Mintlify's measurement](https://www.mintlify.com/blog/ai-traffic)).
+When an agent reads your quickstart and fails, you lose the developer behind
+it and you never find out. quickstarted turns that failure into a CI signal: a
+sandboxed agent gets your goal, your docs, and nothing else, and a script you
+wrote decides whether it succeeded.
+
+Full documentation: **https://snehankekre.com/quickstarted/**
+
+**Status: early. The interface will change.**
+
+## How it works
+
+You write a journey, in YAML:
+
+```yaml
+name: pnf-quickstart
+goal: >
+  Install the pnf package from PyPI into a virtualenv and write a runnable
+  snippet that imports pnf and identifies the decode function.
+docs:
+  entrypoint: https://raw.githubusercontent.com/snehankekre/pnf/main/README.md
+  allow:
+    - raw.githubusercontent.com
+setup:
+  - python3 -m venv .venv
+success:
+  script: |
+    set -e
+    .venv/bin/python -c "import pnf; assert callable(pnf.decode)"
+    .venv/bin/python demo.py
+replay:
+  - .venv/bin/pip install pnf
+  - printf 'import pnf\nprint(pnf.decode.__name__)\n' > demo.py
+  - .venv/bin/python demo.py
+```
+
+The harness runs an agent in a throwaway workspace with two tools: a sandboxed
+`bash`, and `read_docs`, which is the only way it can read documentation.
+
+The harness enforces that. All shell traffic leaves through a proxy it owns,
+and documentation hosts are unreachable from the shell, so an agent that tries
+`curl https://docs.example.com/...` is refused and the attempt is recorded.
+The pages listed in a report are therefore the pages the agent really read.
+When a run fails, the last page it read is a fact you can act on.
+
+When the agent stops, the harness runs your `success.script` in the workspace.
+Exit code 0 is a pass. Nothing else is. The agent's own claim of success is
+recorded and never trusted; there is no LLM judge anywhere in scoring.
+
+## Two modes
+
+**Replay mode** (free, no LLM, no API key) runs your `replay` commands, the
+literal commands your docs tell users to type, and stops at the first failure.
+Treat it as a precondition. If the documented commands break, no reader stands
+a chance, and agent mode only adds noise.
+
+```
+quickstarted run journeys/pnf-quickstart.yaml --agent replay
+```
+
+**Agent mode** has an LLM follow your docs to the goal. This catches what
+replay cannot: ambiguity, missing steps, wrong ordering, docs that assume
+context a newcomer does not have.
+
+```
+pip install "quickstarted[claude]"
+export QUICKSTARTED_ANTHROPIC_API_KEY=...
+quickstarted run journeys/pnf-quickstart.yaml --agent claude
+```
+
+Keys are read from `QUICKSTARTED_*` names first so they can sit in your shell
+without other tooling on the same machine picking them up and spending them.
+The vendor-standard names still work as a fallback, which is what CI sets.
+
+```
+[PASS] pnf-quickstart (claude:claude-opus-5)
+  classification: passed
+  turns: 12, duration: 70.7s
+  backend: seatbelt
+  tokens: 24 in / 4616 out, cache 16812 written / 104148 read
+  docs pages read: 2
+```
+
+## Pass rates
+
+The same docs and the same model can pass at 10:00 and fail at 10:05. One run
+is a sample. Use `--repeat` and read the rate:
+
+```
+quickstarted run journeys/*.yaml --agent claude --repeat 5 --workers 3
+```
+
+Every run is classified, and only two classifications say anything about your
+documentation:
+
+| Classification | Meaning | Counts toward pass rate |
+| --- | --- | --- |
+| `passed` | success script exited 0 | yes |
+| `docs_gap` | the agent finished, the check failed | yes |
+| `budget_exhausted` | out of turns, time, or tokens | no |
+| `infra_error` | rate limit, upstream 5xx, network failure | no |
+| `harness_error` | misconfigured journey, our bug | no |
+| `agent_refusal` | model declined | no |
+
+Runs that produced no evidence are excluded from the numerator *and* the
+denominator, and reported separately. When nothing produced evidence the suite
+says so. Reporting 0% there would be a different claim, and a false one.
+
+## Does llms.txt actually help?
+
+Whether a project ships `llms.txt` is a checklist item anyone can `curl`, and
+scoring it would be the proxy metric this tool exists to replace. So
+quickstarted never scores affordances. It measures them:
+
+```
+quickstarted run journeys/streamlit-quickstart.yaml --agent claude --repeat 10
+quickstarted run journeys/streamlit-quickstart.yaml --agent claude --repeat 10 --affordances none
+```
+
+Same prompt, same journey, same model. The only difference is whether the
+agent could read `llms.txt` and `.md` variants. The difference in pass rate is
+a measurement of the affordance itself.
+
+`--probe-affordances` records what exists without changing anything, including
+the size. Presence and usefulness come apart quickly: Streamlit's `llms.txt` is
+67 KB, while its `llms-full.txt` is 1.8 MB, which is more than most agents can
+read in one go.
+
+## Sandboxing
+
+Agent-authored commands from somebody else's quickstart are untrusted code.
+
+| Backend | Filesystem | Network |
+| --- | --- | --- |
+| `docker` | container | internal network; the proxy sidecar is the only route out |
+| `seatbelt` (macOS) | home directory unreadable, writes confined to the workspace | all egress denied except the harness proxy |
+| `local` | none | none; proxy variables are advisory |
+
+`auto` picks the best available. `quickstarted run` refuses `local` unless you
+pass `--allow-unenforced`. Run `quickstarted doctor` to see what your machine
+can enforce. Details in [SECURITY.md](SECURITY.md).
+
+## Install
+
+```
+pip install quickstarted                 # replay mode only
+pip install "quickstarted[claude]"       # + Claude
+pip install "quickstarted[all-agents]"   # + OpenAI and Gemini
+```
+
+Python 3.9+. One runtime dependency: PyYAML. Every adapter is a plain tool-use
+loop against the same two harness-owned tools and the same shared prompt, so
+cross-model numbers compare like with like.
+
+## CI
+
+```yaml
+- uses: actions/checkout@v4
+- uses: actions/setup-python@v5
+  with:
+    python-version: "3.12"
+- run: pip install quickstarted
+- run: quickstarted run journeys/*.yaml --agent replay --out results --junit junit.xml
+- uses: actions/upload-artifact@v4
+  if: always()
+  with:
+    name: quickstarted-results
+    path: results/
+```
+
+`quickstarted run` exits 1 when any journey fails, so the job gates merges.
+`results.json` is versioned (schema 1.0). JUnit XML reports a docs gap as a
+failure and infrastructure trouble as an error, so a rate limit does not read
+as a broken quickstart. Run agent mode on a schedule, or when a model ships.
+Every push is too often, because agent runs cost real tokens.
+
+## Fetching other people's docs
+
+Benchmarking means requesting pages from organisations who did not ask to be
+measured. quickstarted sends a truthful User-Agent, honours `robots.txt`, and
+waits a second between requests to the same host. `--cache-dir` makes reruns
+read the same bytes. `--refresh` flags pages whose content changed since the
+last run, which is a finding in its own right.
+
+## What this does and does not tell you
+
+It tells you whether a given model, on a given day, could get from your docs to
+a working result, and where it was reading when it could not. That is a lower
+bound on documentation quality. Nothing here measures whether your docs are
+pleasant, complete, or accurate beyond the journey you wrote, and there is no
+UI testing at all: if your product's button moved, this will not notice.
+[Doc Detective](https://github.com/doc-detective/doc-detective) is the tool for
+that, and the two answer different questions.
+
+Write success scripts that assert local, deterministic facts. A pass means the
+floor holds, and says nothing about the ceiling.
+
+## License
+
+MIT.