quickstarted 0.2.0__py3-none-any.whl

quickstarted-0.2.0.dist-info/METADATA

@@ -0,0 +1,256 @@
+Metadata-Version: 2.4
+Name: quickstarted
+Version: 0.2.0
+Summary: Test whether an AI agent can actually complete your quickstart by following your docs
+Project-URL: Homepage, https://github.com/snehankekre/quickstarted
+Project-URL: Issues, https://github.com/snehankekre/quickstarted/issues
+Project-URL: Changelog, https://github.com/snehankekre/quickstarted/blob/main/CHANGELOG.md
+Author-email: snehan <snehan.minerva@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agent-experience,agents,ci,devrel,docs,documentation,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Documentation
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.9
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: all-agents
+Requires-Dist: anthropic>=0.40; extra == 'all-agents'
+Requires-Dist: google-genai>=1.0; extra == 'all-agents'
+Requires-Dist: openai>=1.40; extra == 'all-agents'
+Provides-Extra: claude
+Requires-Dist: anthropic>=0.40; extra == 'claude'
+Provides-Extra: dev
+Requires-Dist: anthropic>=0.40; extra == 'dev'
+Requires-Dist: google-genai>=1.0; extra == 'dev'
+Requires-Dist: mypy>=1.11; extra == 'dev'
+Requires-Dist: openai>=1.40; extra == 'dev'
+Requires-Dist: pytest-cov>=5.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Provides-Extra: docs
+Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
+Provides-Extra: gemini
+Requires-Dist: google-genai>=1.0; extra == 'gemini'
+Provides-Extra: openai
+Requires-Dist: openai>=1.40; extra == 'openai'
+Description-Content-Type: text/markdown
+
+# quickstarted
+
+Test whether an AI agent can actually complete your quickstart by following
+your docs.
+
+[![CI](https://github.com/snehankekre/quickstarted/actions/workflows/ci.yml/badge.svg)](https://github.com/snehankekre/quickstarted/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/quickstarted.svg)](https://pypi.org/project/quickstarted/)
+[![Python](https://img.shields.io/pypi/pyversions/quickstarted.svg)](https://pypi.org/project/quickstarted/)
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+Roughly half the traffic to documentation sites is now AI agents and
+AI-assisted workflows ([Mintlify's measurement](https://www.mintlify.com/blog/ai-traffic)).
+When an agent reads your quickstart and fails, you lose the developer behind
+it and you never find out. quickstarted turns that failure into a CI signal: a
+sandboxed agent gets your goal, your docs, and nothing else, and a script you
+wrote decides whether it succeeded.
+
+Full documentation: **https://snehankekre.com/quickstarted/**
+
+**Status: early. The interface will change.**
+
+## How it works
+
+You write a journey, in YAML:
+
+```yaml
+name: pnf-quickstart
+goal: >
+  Install the pnf package from PyPI into a virtualenv and write a runnable
+  snippet that imports pnf and identifies the decode function.
+docs:
+  entrypoint: https://raw.githubusercontent.com/snehankekre/pnf/main/README.md
+  allow:
+    - raw.githubusercontent.com
+setup:
+  - python3 -m venv .venv
+success:
+  script: |
+    set -e
+    .venv/bin/python -c "import pnf; assert callable(pnf.decode)"
+    .venv/bin/python demo.py
+replay:
+  - .venv/bin/pip install pnf
+  - printf 'import pnf\nprint(pnf.decode.__name__)\n' > demo.py
+  - .venv/bin/python demo.py
+```
+
+The harness runs an agent in a throwaway workspace with two tools: a sandboxed
+`bash`, and `read_docs`, which is the only way it can read documentation.
+
+The harness enforces that. All shell traffic leaves through a proxy it owns,
+and documentation hosts are unreachable from the shell, so an agent that tries
+`curl https://docs.example.com/...` is refused and the attempt is recorded.
+The pages listed in a report are therefore the pages the agent really read.
+When a run fails, the last page it read is a fact you can act on.
+
+When the agent stops, the harness runs your `success.script` in the workspace.
+Exit code 0 is a pass. Nothing else is. The agent's own claim of success is
+recorded and never trusted; there is no LLM judge anywhere in scoring.
+
+## Two modes
+
+**Replay mode** (free, no LLM, no API key) runs your `replay` commands, the
+literal commands your docs tell users to type, and stops at the first failure.
+Treat it as a precondition. If the documented commands break, no reader stands
+a chance, and agent mode only adds noise.
+
+```
+quickstarted run journeys/pnf-quickstart.yaml --agent replay
+```
+
+**Agent mode** has an LLM follow your docs to the goal. This catches what
+replay cannot: ambiguity, missing steps, wrong ordering, docs that assume
+context a newcomer does not have.
+
+```
+pip install "quickstarted[claude]"
+export QUICKSTARTED_ANTHROPIC_API_KEY=...
+quickstarted run journeys/pnf-quickstart.yaml --agent claude
+```
+
+Keys are read from `QUICKSTARTED_*` names first so they can sit in your shell
+without other tooling on the same machine picking them up and spending them.
+The vendor-standard names still work as a fallback, which is what CI sets.
+
+```
+[PASS] pnf-quickstart (claude:claude-opus-5)
+  classification: passed
+  turns: 12, duration: 70.7s
+  backend: seatbelt
+  tokens: 24 in / 4616 out, cache 16812 written / 104148 read
+  docs pages read: 2
+```
+
+## Pass rates
+
+The same docs and the same model can pass at 10:00 and fail at 10:05. One run
+is a sample. Use `--repeat` and read the rate:
+
+```
+quickstarted run journeys/*.yaml --agent claude --repeat 5 --workers 3
+```
+
+Every run is classified, and only two classifications say anything about your
+documentation:
+
+| Classification | Meaning | Counts toward pass rate |
+| --- | --- | --- |
+| `passed` | success script exited 0 | yes |
+| `docs_gap` | the agent finished, the check failed | yes |
+| `budget_exhausted` | out of turns, time, or tokens | no |
+| `infra_error` | rate limit, upstream 5xx, network failure | no |
+| `harness_error` | misconfigured journey, our bug | no |
+| `agent_refusal` | model declined | no |
+
+Runs that produced no evidence are excluded from the numerator *and* the
+denominator, and reported separately. When nothing produced evidence the suite
+says so. Reporting 0% there would be a different claim, and a false one.
+
+## Does llms.txt actually help?
+
+Whether a project ships `llms.txt` is a checklist item anyone can `curl`, and
+scoring it would be the proxy metric this tool exists to replace. So
+quickstarted never scores affordances. It measures them:
+
+```
+quickstarted run journeys/streamlit-quickstart.yaml --agent claude --repeat 10
+quickstarted run journeys/streamlit-quickstart.yaml --agent claude --repeat 10 --affordances none
+```
+
+Same prompt, same journey, same model. The only difference is whether the
+agent could read `llms.txt` and `.md` variants. The difference in pass rate is
+a measurement of the affordance itself.
+
+`--probe-affordances` records what exists without changing anything, including
+the size. Presence and usefulness come apart quickly: Streamlit's `llms.txt` is
+67 KB, while its `llms-full.txt` is 1.8 MB, which is more than most agents can
+read in one go.
+
+## Sandboxing
+
+Agent-authored commands from somebody else's quickstart are untrusted code.
+
+| Backend | Filesystem | Network |
+| --- | --- | --- |
+| `docker` | container | internal network; the proxy sidecar is the only route out |
+| `seatbelt` (macOS) | home directory unreadable, writes confined to the workspace | all egress denied except the harness proxy |
+| `local` | none | none; proxy variables are advisory |
+
+`auto` picks the best available. `quickstarted run` refuses `local` unless you
+pass `--allow-unenforced`. Run `quickstarted doctor` to see what your machine
+can enforce. Details in [SECURITY.md](SECURITY.md).
+
+## Install
+
+```
+pip install quickstarted                 # replay mode only
+pip install "quickstarted[claude]"       # + Claude
+pip install "quickstarted[all-agents]"   # + OpenAI and Gemini
+```
+
+Python 3.9+. One runtime dependency: PyYAML. Every adapter is a plain tool-use
+loop against the same two harness-owned tools and the same shared prompt, so
+cross-model numbers compare like with like.
+
+## CI
+
+```yaml
+- uses: actions/checkout@v4
+- uses: actions/setup-python@v5
+  with:
+    python-version: "3.12"
+- run: pip install quickstarted
+- run: quickstarted run journeys/*.yaml --agent replay --out results --junit junit.xml
+- uses: actions/upload-artifact@v4
+  if: always()
+  with:
+    name: quickstarted-results
+    path: results/
+```
+
+`quickstarted run` exits 1 when any journey fails, so the job gates merges.
+`results.json` is versioned (schema 1.0). JUnit XML reports a docs gap as a
+failure and infrastructure trouble as an error, so a rate limit does not read
+as a broken quickstart. Run agent mode on a schedule, or when a model ships.
+Every push is too often, because agent runs cost real tokens.
+
+## Fetching other people's docs
+
+Benchmarking means requesting pages from organisations who did not ask to be
+measured. quickstarted sends a truthful User-Agent, honours `robots.txt`, and
+waits a second between requests to the same host. `--cache-dir` makes reruns
+read the same bytes. `--refresh` flags pages whose content changed since the
+last run, which is a finding in its own right.
+
+## What this does and does not tell you
+
+It tells you whether a given model, on a given day, could get from your docs to
+a working result, and where it was reading when it could not. That is a lower
+bound on documentation quality. Nothing here measures whether your docs are
+pleasant, complete, or accurate beyond the journey you wrote, and there is no
+UI testing at all: if your product's button moved, this will not notice.
+[Doc Detective](https://github.com/doc-detective/doc-detective) is the tool for
+that, and the two answer different questions.
+
+Write success scripts that assert local, deterministic facts. A pass means the
+floor holds, and says nothing about the ceiling.
+
+## License
+
+MIT.

quickstarted-0.2.0.dist-info/RECORD

@@ -0,0 +1,34 @@
+quickstarted/__init__.py,sha256=zE7VMSKaKjIjIgpq5MHmMUXZE4doWiGjd9a3zHsTaSc,683
+quickstarted/_version.py,sha256=ZpoPRLhDyc9ZHTe7Y2O3firGVKq_NToRzOP6ipjnJic,252
+quickstarted/cli.py,sha256=qf_O9U69WhF0rLaNQTY1kVU-qx5xLlFWjLeNg9TCKKg,9285
+quickstarted/docs.py,sha256=0DvjgNm1c05WBXFnR7qUZ5zsJjLkvTvC5nrPWRTCGs0,9617
+quickstarted/journey.py,sha256=SJWg0ot_G90bWzfA-a5c-23bO1Wt9GEnFQaX4SVIe60,6648
+quickstarted/pricing.py,sha256=15lJtO89LfrXcBv76ys9_U4zR5aDATQQeCh_iYiz_sE,2866
+quickstarted/report.py,sha256=2btaNRoetILElaNGQsZelFP-VrJrU_dNRfsXyiM3sEY,7710
+quickstarted/results.py,sha256=_TOmf5_GLTosG3CLFxyfAGhfcnC4_hD91unOdMC8moQ,5914
+quickstarted/run.py,sha256=8lTNYj0rFSgBeuTQalGI2_Y2XM7PcUlDVanrYtPw47k,7656
+quickstarted/sandbox.py,sha256=8ahNP_DFZaZnmHQzSM2w8WJDYiAqeDOYcrqXrAjwlpA,476
+quickstarted/suite.py,sha256=SFGUivQx1xfo2xHY7zz7zdBBM8CFNfb-IQcMpTPEufg,6727
+quickstarted/trace.py,sha256=pGwQCxS8-3rcWgT6MJcHS84itWAlV9fkmUC28Smg_R8,1627
+quickstarted/transport.py,sha256=ZqLV9lK3NWk11T5NaKlHrDsrrXL6LyRHhRwV3q_W3LM,2617
+quickstarted/agents/__init__.py,sha256=awU5gb2_55DaS4tF4N8hCgZz3tWIfHGFEY1LCCzzPUA,144
+quickstarted/agents/base.py,sha256=MnQWFFc3O28KCaaQsJJBaWguVwoZYQ7B1Sk3dvtPpro,5163
+quickstarted/agents/claude.py,sha256=rf3KH1YdQaV_FZzoTkMb0_gvNMCxhACIQM1W3hRocL0,9526
+quickstarted/agents/gemini_agent.py,sha256=RF8pC_PtoS-wafBmtB1Eki5gSxnOQdnocHMKmmtw0Cg,5076
+quickstarted/agents/openai_agent.py,sha256=Ut3FBLsngx1tv2mHgWCVqZXJR62pA0Ccp9Pw5ty_iK8,6365
+quickstarted/agents/prompt.py,sha256=wPjGjzHv-8an618TPkZI9MqG5RMymGipqqJDM6gxoYM,2716
+quickstarted/agents/registry.py,sha256=-YR4e-pWvAQ5DWX8mgih36unDDMRZ-saRDN7BOa3Gko,981
+quickstarted/agents/replay.py,sha256=bgj9uUDpJfh8Zp8F7e9O8yrkfgcejYm8AcUOoI1PNug,1521
+quickstarted/exec/__init__.py,sha256=XAkOtDpk6KZ1hXp3ZLu6SSUXDzbXhlmNFo0CJ2lkdSU,2381
+quickstarted/exec/base.py,sha256=dMUKsHNGM218SNOT4sn5tlUcRdm4thtpuMIsA-sub-8,4289
+quickstarted/exec/docker.py,sha256=K5ALzTLQipo5nB6zBcrea2VFc_IvxSIeSj4U8VVxNqs,7985
+quickstarted/exec/local.py,sha256=cek0HwhbDejSQIelNry92FmR8q1UVHoOL1T7FQHG5sQ,501
+quickstarted/exec/seatbelt.py,sha256=w_Nts9GHu9_onmSQWdBBbTcIdNAvR91gg7ilHWUde_U,3351
+quickstarted/net/__init__.py,sha256=mXh5SeQhmnxJTyjKKsQ9gLduv2z2rP40qBXL8Ruz3s8,191
+quickstarted/net/proxy.py,sha256=3PGsnY1yx-QUdC75q3inxpn7-ZgocyiPA83Ef1dqqkA,10121
+quickstarted/net/proxy_main.py,sha256=x2KoLZwkMLc1b9NhrFaepBeg9v3pqQ7rzmQJV3xJQPM,1563
+quickstarted-0.2.0.dist-info/METADATA,sha256=dTzXpwUQrmPmO1FyV69zJ-3JxecIZHEVmuv9zH3Ogx8,10220
+quickstarted-0.2.0.dist-info/WHEEL,sha256=lCkmxWfQsSc9CfIClYeavTdQeEX2toPqufh9gI35EQA,87
+quickstarted-0.2.0.dist-info/entry_points.txt,sha256=SqkqLK38ifHlJz5aw6kDioF6IGWz-qXwiyCAZgX-LPU,86
+quickstarted-0.2.0.dist-info/licenses/LICENSE,sha256=_f2myDwEsLzmhmXj7WgwvwPYAta6LE3HxDVa31Mq_18,1069
+quickstarted-0.2.0.dist-info/RECORD,,

quickstarted-0.2.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.31.0
+Root-Is-Purelib: true
+Tag: py3-none-any

quickstarted-0.2.0.dist-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+qstart = quickstarted.cli:main
+quickstarted = quickstarted.cli:main

quickstarted-0.2.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Snehan Kekre
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.