ghost-api-cli 0.2.0__py3-none-any.whl

ghost_api_cli-0.2.0.dist-info/METADATA

@@ -0,0 +1,1019 @@
+Metadata-Version: 2.4
+Name: ghost-api-cli
+Version: 0.2.0
+Summary: An AI agent that turns any website into a typed, tested API client.
+Author: Danielraj
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/Danielraj5605/Ghost-api
+Project-URL: Documentation, https://github.com/Danielraj5605/Ghost-api#readme
+Project-URL: Changelog, https://github.com/Danielraj5605/Ghost-api/blob/main/CHANGELOG.md
+Project-URL: Issues, https://github.com/Danielraj5605/Ghost-api/issues
+Project-URL: Security, https://github.com/Danielraj5605/Ghost-api/blob/main/SECURITY.md
+Keywords: api,reverse-engineering,api-client,openapi,code-generation,playwright,ai-agent,llm,web-scraping,sdk-generator
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Code Generators
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: playwright>=1.45
+Requires-Dist: httpx>=0.27
+Requires-Dist: pydantic>=2.7
+Requires-Dist: typer>=0.12
+Requires-Dist: rich>=13.7
+Requires-Dist: python-dotenv>=1.0
+Requires-Dist: genson>=1.3
+Requires-Dist: jinja2>=3.1
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: fastapi>=0.110
+Requires-Dist: uvicorn>=0.29
+Requires-Dist: openpyxl>=3.1
+Requires-Dist: prompt_toolkit>=3.0.43
+Provides-Extra: mcp
+Requires-Dist: mcp>=1.2; extra == "mcp"
+Provides-Extra: notify
+Requires-Dist: plyer>=2.1; extra == "notify"
+Provides-Extra: docs
+Requires-Dist: mkdocs<2,>=1.6; extra == "docs"
+Requires-Dist: mkdocs-material>=9.5; extra == "docs"
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: ruff>=0.6; extra == "dev"
+Requires-Dist: mypy>=1.11; extra == "dev"
+Requires-Dist: openapi-spec-validator>=0.7; extra == "dev"
+Requires-Dist: types-PyYAML; extra == "dev"
+Requires-Dist: psutil>=5.9; extra == "dev"
+Requires-Dist: types-psutil; extra == "dev"
+Requires-Dist: pywinpty>=2.0; sys_platform == "win32" and extra == "dev"
+Dynamic: license-file
+
+# Ghost API
+
+**An AI agent that turns any website into a typed, tested API client.**
+
+Give Ghost API a website and a plain-English goal. It opens the site in a real browser, watches the hidden requests the site makes to its own server, works out how that hidden API behaves, and writes a clean Python (or TypeScript) client library for it, with tests.
+
+![The Ghost API dashboard during a real run on quotes.toscrape.com](https://raw.githubusercontent.com/Danielraj5605/Ghost-api/main/docs/images/dashboard-demo.gif)
+
+<sub>A real run on quotes.toscrape.com from the web dashboard (`ghost-api ui`): the agent loads the page, the API call is tested, a client is generated and verified, and "Try it" calls the endpoint. 2 free LLM calls, about 20 seconds.</sub>
+
+> **Status:** Version 0.2.0. All 11 build stages are done and evaluated on four sites (see [Evaluation](#evaluation)): every goal endpoint found, nothing invented, every generated client test passing, for 1–10 free LLM calls per site. It has a live terminal UI, a local web dashboard and a shareable one-file HTML report. See [IMPLEMENTATION.md](IMPLEMENTATION.md) for how it was built.
+
+---
+
+**Documentation:** guides, the command reference and the Python API are in [docs/](docs/index.md)
+(build the site with `pip install -e ".[docs]"` then `mkdocs serve`).
+
+## Table of contents
+
+- [The simple explanation](#the-simple-explanation)
+- [The problem](#the-problem)
+- [What Ghost API does](#what-ghost-api-does)
+- [Example](#example)
+- [How it works](#how-it-works)
+- [Why it's an agent, not a script](#why-its-an-agent-not-a-script)
+- [Memory](#memory)
+- [Features](#features)
+- [Tech stack](#tech-stack)
+- [Getting started](#getting-started)
+- [What can I do with it?](#what-can-i-do-with-it)
+- [Three ways to see the results](#three-ways-to-see-the-results)
+- [Usage](#usage)
+- [Output](#output)
+- [Project structure](#project-structure)
+- [Running on free LLM tiers](#running-on-free-llm-tiers)
+- [Responsible use](#responsible-use)
+- [Evaluation](#evaluation)
+- [Limitations](#limitations)
+- [Roadmap](#roadmap)
+
+---
+
+## The simple explanation
+
+Picture a restaurant with **no menu**. You can eat there, but only by pointing at other people's plates. Behind the scenes, every time someone orders, the waiter hands the kitchen a neat order slip:
+
+> "Table 4: 1x Pasta, no cheese, large."
+
+The kitchen understands those slips perfectly. You just never see them.
+
+**Ghost API sits in the restaurant, watches every order slip going to the kitchen, and writes the menu for you.** After that you can order directly, without pointing at plates.
+
+| In the analogy | In real life |
+|---|---|
+| The restaurant | Any website (a bookstore, a weather site, a job board) |
+| Pointing at plates | Copying data by hand, or fragile "scraper" programs that break when the design changes |
+| The order slips | The hidden requests a website makes to its own server |
+| The menu | A small code library that gets the data directly, cleanly and reliably |
+
+**What makes it "AI":** Ghost API doesn't just record. Like a curious person, it:
+
+1. **Explores** the site by clicking, searching and scrolling
+2. **Notices patterns**, such as "every time I open a book, the site asks for `/books/<number>`"
+3. **Tests its guesses**, such as "if I ask for page 2, do I get different results?"
+4. **Fixes its mistakes** when a guess turns out wrong
+5. **Remembers** what it learned, so next time is faster
+
+---
+
+## The problem
+
+Almost every modern website loads its data through a **hidden internal API**. When you open a product page, your browser quietly calls something like:
+
+```
+GET https://shop.example.com/api/v2/products/123?include=reviews
+-> { "id": 123, "name": "Blue Shoes", "price": 59.99, "reviews": [...] }
+```
+
+Developers often want that data in their own code. Their options today are poor:
+
+| Approach | Why it's poor |
+|---|---|
+| Scrape the HTML | Breaks whenever the page layout changes, and is slow |
+| Reverse-engineer by hand in browser DevTools | Takes hours: clicking around, copying requests, guessing what each field means |
+| Tools like mitmproxy or Postman | Record raw traffic, but don't understand it or write any code |
+
+No existing tool both **discovers** these hidden APIs and **produces working code** for them.
+
+---
+
+## What Ghost API does
+
+| Input | Output |
+|---|---|
+| A website URL | A typed Python client library (`httpx` + `pydantic`) |
+| A plain-English goal ("search books and get details") | A pytest test suite that runs against the live site |
+| | An OpenAPI 3 spec |
+| | A report showing what was found, how confident it is, and the agent's reasoning |
+
+---
+
+## Example
+
+Real output from the practice site `quotes.toscrape.com/scroll` (abridged):
+
+```
+$ ghost-api run https://quotes.toscrape.com/scroll --goal "list quotes with their authors and tags"
+
+Exploring site...
+      GET 200 /api/quotes?page=1
+Step 1 finish(goal_covered=True)  The discovered endpoint already returns quotes, their authors, and tags
+
+Testing 1 endpoint(s) with direct calls...
+  GET /api/quotes
+    confirmed    can be called directly, outside the browser   HTTP 200, 10 records
+    confirmed    paginates with page parameter 'page'          the next page returned 10 different records
+    confirmed    needs none of its custom headers or cookies
+
+Verifying the generated client against the live API...
+  Round 1: 2 passed
+
+Generated QuotesToscrapeClient in output/quotes.toscrape.com
+   get_quotes() -> QuotesResponse  + iter_get_quotes()
+```
+
+The generated client:
+
+```python
+from quotes_toscrape_client import QuotesToscrapeClient
+
+with QuotesToscrapeClient() as api:
+    page = api.get_quotes()  # -> QuotesResponse (pydantic)
+    print(page.quotes[0].author.name)  # nested models: Quote, Author
+    for p in api.iter_get_quotes(max_pages=3):  # pagination handled for you
+        ...
+```
+
+The same client in TypeScript (`--lang typescript`):
+
+```ts
+import { QuotesToscrapeClient } from "./quotes_toscrape_client.ts";
+
+const api = new QuotesToscrapeClient();
+const page = await api.getQuotes(); // Promise<QuotesResponse>
+for await (const p of api.iterGetQuotes({ maxPages: 3 })) console.log(p.page);
+```
+
+The TypeScript client is made for **Node 18+** (servers, scripts, serverless functions). In a
+web page it mostly won't work: browsers block reading another site's replies unless that site
+allows it (CORS), and they don't let code set the identifying User-Agent header. For a web
+app, call the client from your own small server and let the page call that:
+
+```ts
+// server.ts (Node): the page calls /api/quotes on your server, your server calls the site.
+import { createServer } from "node:http";
+import { QuotesToscrapeClient } from "./quotes_toscrape_client.ts";
+
+const api = new QuotesToscrapeClient();
+createServer(async (req, res) => {
+  if (req.url !== "/api/quotes") return void res.writeHead(404).end();
+  res.setHeader("Content-Type", "application/json");
+  res.end(JSON.stringify(await api.getQuotes()));
+}).listen(3000);
+```
+
+A harder case: on `hn.algolia.com` the agent switched the search filter to "Comments", found
+that the real API is a POST to Algolia's servers (a third-party domain), and the prober
+confirmed that paging works in the JSON body and that the `x-algolia-api-key` URL parameter
+is required (HTTP 403 without it). The client it generated:
+
+```python
+from hn_algolia_client import HnAlgoliaClient  # key from HN_ALGOLIA_X_ALGOLIA_API_KEY
+
+with HnAlgoliaClient() as api:
+    results = api.search_stories(index="Item_dev", query="python")  # -> SearchResults
+```
+
+---
+
+## How it works
+
+```
+                         +---------------------+
+                         |       START         |
+                         |  URL + goal + lang  |
+                         +----------+----------+
+                                    v
+                         +---------------------+
+                         | 0. PRE-CHECKS       |
+                         | robots.txt, memory  |---- known site? ---> load past
+                         | lookup, budgets     |                      knowledge
+                         +----------+----------+
+                                    v
+          +----------------------------------------------------+
+          |                  AGENT LOOP                        |
+          |   1. OBSERVE  page snapshot + captured traffic     |
+          |   2. DECIDE   LLM picks: explore / test / ask /    |
+          |               finish                               |
+          |   3. ACT      run a tool (click, type, call API)   |
+          |   4. LEARN    update endpoint map + schemas        |
+          |   goal covered? budget left? --no--> back to 1     |
+          +----------+-----------------------------------------+
+                     | yes
+                     v
+          +---------------------+
+          | 5. GENERATE CODE    | client + models + tests
+          +----------+----------+
+                     v
+          +---------------------+      fail      +--------------+
+          | 6. VERIFY (tests)   |--------------->| 7. DIAGNOSE  |--> back to
+          +----------+----------+                | & REPLAN     |    agent loop
+                     | pass                      +--------------+    (max 3x)
+                     v
+          +---------------------+
+          | 8. SAVE MEMORY +    |
+          |    WRITE REPORT     |
+          +---------------------+
+```
+
+### Step by step
+
+**0. Pre-checks.** It reads `robots.txt`, looks the domain up in memory, and sets budgets (maximum LLM calls, steps and minutes).
+
+**1. Observe.** The agent gets a compact view of the page: the clickable elements and inputs with short IDs (such as `[12] button "Search"`), a summary of the new network requests, the endpoints found so far, and which parts of the goal are still uncovered.
+
+**2. Decide.** The LLM picks **one** action and explains why:
+
+```json
+{
+  "thought": "I found the book list endpoint but not book details. Clicking a book should trigger it.",
+  "action": "click",
+  "args": { "element_id": 23 }
+}
+```
+
+**3. Act.** The chosen tool runs: click, type, scroll, navigate, call an endpoint directly, ask the user, or finish.
+
+**4. Learn.** Ordinary code, not the LLM, processes the traffic:
+- Filters out noise (images, CSS, fonts, analytics, ads)
+- Groups URLs into templates: `/api/books/123` and `/api/books/456` become `/api/books/{id}`
+- Infers JSON schemas from real responses and merges them across samples
+- Classifies query parameters as required, optional or constant
+
+**5. Generate.** The LLM gives endpoints and fields readable names. Jinja2 templates produce the client, pydantic models, tests and an OpenAPI spec.
+
+**6. Verify.** The generated tests run against the live site and check that each response matches its schema.
+
+**7. Diagnose and replan.** If a test fails, the agent works out why and recovers:
+
+| Failure | Likely cause | Agent's response |
+|---|---|---|
+| 401 or 403 | Missing cookie, token or header | Retry with the browser's cookies, then find which header is required |
+| Schema mismatch | Too few samples | Explore more, then widen the schema |
+| 404 | Wrong URL template | Re-check how the URL was grouped |
+| 429 | Rate limited | Back off and slow down |
+| Timeout | Slow endpoint | Retry, then mark as unstable |
+
+**8. Save and report.** It saves what it learned to memory and writes `report.md`.
+
+---
+
+## Why it's an agent, not a script
+
+A fixed script can record traffic. Ghost API has to **make decisions**:
+
+| Decision | Example |
+|---|---|
+| Where to explore next | "I've found search but not reviews. Click the reviews tab." |
+| When it has seen enough | "Three similar responses. The schema is stable, so stop." |
+| Which guesses to test | "This looks like cursor pagination, not page numbers. Test it." |
+| How to recover from failure | "Direct call returned 403. Retry with the browser's cookies." |
+| When to ask you | "This needs a login. Please log in in the browser window, then press Enter." |
+
+**Design principle:** the LLM decides and names things. Deterministic code does the heavy lifting: capture, grouping, schema inference and code generation. This keeps it cheap, fast, and reliable even with small free models.
+
+---
+
+## Memory
+
+Ghost API improves with use:
+
+- **Per-site memory:** endpoints, schemas, auth style and pagination style for each domain. A second run re-verifies instead of re-exploring.
+- **Lessons learned:** notes such as "this site needs the X-CSRF header" are fed back into the prompt on later runs.
+- **LLM cache:** identical prompts return cached answers, which saves free-tier quota.
+
+---
+
+## Features
+
+- **One command:** URL and goal in; endpoint map, typed client, tests, report and OpenAPI spec out
+- **AI exploration** in a real browser (Playwright), one decision per step, with live reasoning
+- **Capture** of the site's JSON API traffic, including third-party APIs (for example Algolia),
+  with analytics, health checks and infrastructure filtered out
+- **Redaction** of cookies, tokens and keys before anything is stored, logged or sent to an LLM
+- **Analysis:** URL templates (`/books/{id}`), parameter types, merged JSON schemas
+- **Hypothesis testing** by direct calls: pagination (page, offset, cursor), required headers,
+  API keys, auth style
+- **Python client** with pydantic models, rate limiting, retries, credentials from env vars and
+  `iter_...()` helpers for pagination; passes `mypy --strict`
+- **Verification:** the generated tests run against the live API; failures are diagnosed and
+  repaired (up to 3 rounds), and every endpoint is marked verified, unverified or failed
+- **Memory:** `refresh` re-checks a known site with as little work as possible
+- **Report and OpenAPI 3.1** export, plus a replayable log of every run
+- **Free:** runs on free LLM tiers with fallback between providers, caching and budgets
+- **Live terminal UI** in the style of Claude Code: phase stepper, compact step log, a status
+  line that updates in place, and a summary card with a copy-paste "use it" line
+- **Web dashboard** (`ghost-api ui`): start runs, watch the agent's browser and reasoning live,
+  browse endpoints with their schemas, call them with "Try it", read and download the code
+- **Shareable report:** every run writes `report.html`, one file that works offline
+- **Data download, no code:** `ghost-api fetch` and the dashboard's Download data:
+  all pages to CSV, Excel or JSON
+- **Watch & alerts:** new / changed / gone items, to your phone, a webhook or the desktop
+- **REST and MCP servers:** use sites from Excel, any language, or an AI assistant
+- **Personal-data leak check:** spots and hides emails, phones, PAN/GST/Aadhaar, UPI and
+  bank details, and reports which endpoints expose them
+
+- **TypeScript client** too (`--lang typescript`): interfaces, a fetch-based client for Node 18+
+  (for web pages, call it from your own server), async page iterators, and Node tests; passes
+  `tsc --strict`
+
+- **GraphQL:** each named operation becomes its own typed method (`get_user(id)`), with the
+  query built in and the variables as arguments; mutations are never replayed or generated
+
+**Not yet:** WebSocket and server-sent-event APIs.
+
+---
+
+## Tech stack
+
+| Part | Tool |
+|---|---|
+| Language | Python 3.11+ |
+| Browser automation | Playwright (Chromium) |
+| LLM access | Small built-in gateway over the OpenAI-compatible APIs of Gemini (main), Groq and OpenRouter (fallbacks) |
+| HTTP | httpx |
+| Data models | pydantic |
+| Schema inference | genson |
+| Code generation | Jinja2 |
+| CLI and terminal UI | Typer + Rich |
+| Web dashboard | FastAPI + uvicorn (local server), React + Vite + Tailwind CSS (frontend, prebuilt) |
+| Tests | pytest |
+| Storage | SQLite |
+
+Runs on an ordinary laptop: 8 GB RAM, no GPU.
+
+---
+
+## Getting started
+
+**You need:** Python 3.11+ and, for the AI agent, one free LLM key from
+[Google AI Studio](https://aistudio.google.com/apikey) (Gemini), [Groq](https://console.groq.com/keys)
+or [OpenRouter](https://openrouter.ai/keys). Without a key, runs still load and scroll pages
+and map what they see (`--no-agent`).
+
+**1. Install** (once; any OS). [pipx](https://pipx.pypa.io) or [uv](https://docs.astral.sh/uv/)
+put the `ghost-api` command on your PATH in its own environment:
+
+```bash
+pipx install git+https://github.com/Danielraj5605/Ghost-api
+# or:  uv tool install git+https://github.com/Danielraj5605/Ghost-api
+```
+
+(After the first PyPI release this becomes `pipx install ghost-api-cli`.)
+
+**2. Set up a folder** for your work. `init` creates the project file, downloads the
+browser Ghost API drives (Chromium, about 150 MB, once), asks for an LLM key (optional,
+typed hidden, saved in `.env`) and keeps keys and data out of git:
+
+```bash
+mkdir my-apis && cd my-apis
+ghost-api init
+ghost-api doctor        # checks Python, Chromium, the project and each key
+```
+
+**3. Explore a site:**
+
+```bash
+ghost-api run https://quotes.toscrape.com/scroll --goal "list quotes with their authors"
+ghost-api               # or the interactive session: paste a site, say what you want
+```
+
+### With Claude Code (no LLM key needed)
+
+```bash
+ghost-api integrate claude-code      # also: cursor, vscode, desktop
+```
+
+Then in Claude Code: `/ghost-explore https://quotes.toscrape.com/scroll quotes with authors`.
+Claude reads each page and decides what to click; Ghost API acts under its rules, maps the
+hidden API, tests every endpoint, and writes the typed client, report and OpenAPI spec. In a
+test on the demo shop, Claude mapped all five endpoints (all verified) in 12 turns. Claude
+alone found them too (7 turns): on easy sites Ghost API's value is what comes after, such as
+checking the live API again in seconds with no LLM, offline tests, change detection, rules and
+an audit trail. Measured in [docs/benchmark.md](docs/benchmark.md).
+Real runs on public practice sites: [docs/gallery.md](docs/gallery.md).
+
+### From Python
+
+Everything the command does is also a function, with the same rules:
+
+```python
+from ghost_api.api import discover, fetch, diff
+
+result = discover("https://quotes.toscrape.com/scroll", "list quotes with their authors")
+print(result.status, [e["path"] for e in result.endpoints], result.client_files)
+
+rows = fetch("quotes.toscrape.com/scroll", "get_quotes", pages=3)   # list of records
+changes = diff("quotes.toscrape.com/scroll", live=True)            # breaking or not
+```
+
+Also `refresh`, `generate`, `operations` and `sites`. Errors are exceptions you can catch:
+`NotAllowed` (robots.txt), `NoLLMKey`, `NothingRemembered`, `GhostApiError`.
+
+### Project settings
+
+`ghost-api init` writes `ghost-api.toml`. Every command run in that folder, or any folder
+below it, uses the project's database and output folder, and its defaults:
+
+```toml
+[project]
+database = ".ghost-api/ghost_api.db"   # runs, traffic, endpoints (keep it private)
+output = "output"                      # reports, clients, downloads
+
+[defaults]                              # any command option, by name
+lang = "typescript"
+rate_limit = 0.5
+max_steps = 30
+```
+
+Options on the command line win over the file, and `GHOST_API_DB` wins over both. Keys are
+read from `.env` in the current folder, the project folder, then `~/.ghost-api/.env`
+(handy for one set of keys everywhere).
+
+### Scripts and CI: `--json` and exit codes
+
+Add `--json` anywhere on a command to get its result as JSON on stdout; progress and
+messages go to stderr, so it pipes cleanly:
+
+```bash
+ghost-api run https://quotes.toscrape.com/scroll -g "list quotes" --no-agent --json > run.json
+ghost-api inspect quotes.toscrape.com/scroll --json | jq '.endpoints[].path'
+ghost-api fetch quotes.toscrape.com/scroll get_quotes --pages 3 --json | jq '.rows'
+```
+
+| Exit code | Meaning |
+|---|---|
+| 0 | Done (a run may still have found nothing; check the JSON or report) |
+| 1 | Failed: an error, failed checks, or nothing stored to work on |
+| 2 | Wrong command line or setup: bad option, missing LLM key, unreadable `ghost-api.toml` |
+| 3 | The site said no: robots.txt disallows it, or a bot-protection page |
+| 130 | Stopped with Ctrl+C |
+
+### From source (contributors)
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for the developer setup. In short, pick the block for
+your terminal (no Node.js needed: the dashboard comes prebuilt):
+
+**Windows, Command Prompt (cmd)**
+
+```bat
+git clone https://github.com/Danielraj5605/Ghost-api ghost-api
+cd ghost-api
+python -m venv .venv
+.venv\Scripts\activate.bat
+pip install -e .
+playwright install chromium
+copy .env.example .env
+```
+
+**Windows, PowerShell**
+
+```powershell
+git clone https://github.com/Danielraj5605/Ghost-api ghost-api
+cd ghost-api
+python -m venv .venv
+.venv\Scripts\Activate.ps1
+pip install -e .
+playwright install chromium
+Copy-Item .env.example .env
+```
+
+If PowerShell says running scripts is disabled, run
+`Set-ExecutionPolicy -Scope CurrentUser RemoteSigned` once, or use Command Prompt.
+
+**macOS / Linux**
+
+```bash
+git clone https://github.com/Danielraj5605/Ghost-api ghost-api
+cd ghost-api
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -e .
+playwright install chromium
+cp .env.example .env
+```
+
+Then put at least one key in `.env` and check it:
+
+```bash
+ghost-api doctor            # checks each key with one tiny call
+```
+
+You'll know the environment is active when your prompt starts with `(.venv)`. Use
+`pip install -e ".[dev]"` instead if you want the tests and linters. If the `ghost-api`
+command is blocked or not found, `python -m ghost_api` does the same thing
+(for example `python -m ghost_api doctor`).
+
+**Run it from any folder (Windows).** To start Ghost API by just typing `ghost-api` in
+any Command Prompt or PowerShell window, the way you start `claude`, run this once from
+the project folder:
+
+```bat
+scripts\install-command.cmd
+```
+
+It adds a small `ghost-api` launcher to `%USERPROFILE%\.local\bin` (and puts that folder
+on your PATH if needed). The launcher uses the project's virtual environment, so there's
+nothing to activate, and always works with the project's `.env`, memory and `output\`
+folder. Then open a new window and type `ghost-api` (it asks for a website and a goal),
+`ghost-api ui` for the dashboard, or any command below. On macOS/Linux, `pipx install -e .`
+does the same job.
+
+What `.env` looks like:
+
+```
+GEMINI_API_KEY=your_key_here
+GROQ_API_KEY=optional_fallback_key
+OPENROUTER_API_KEY=optional_second_fallback
+```
+
+---
+
+## What can I do with it?
+
+Ghost API first learns a website's hidden API (`ghost-api run ...`). After that, you can use
+what it learned in five ways, most of them without writing any code:
+
+| I want to... | Do this |
+|---|---|
+| **Get the data in a spreadsheet** | `ghost-api fetch turftown.in/chennai get_nearby_venues area=velachery sport=football --format xlsx` follows all the pages and saves an Excel, CSV or JSON file. In the dashboard: open an endpoint and press **Download data**. `ghost-api fetch SITE` lists what you can fetch. |
+| **Know when something changes** | `ghost-api watch add quotes.toscrape.com list_quotes --every 6h --notify ntfy:my-long-private-topic` re-checks on a schedule and tells you what's new, what changed (a price, a name...) and what's gone, on your phone ([ntfy](https://ntfy.sh)), a webhook (Discord, Slack...) or a desktop notification. `ghost-api watch schedule` keeps it running in the background on Windows. Dashboard: **Watch this for changes**, and the **Watches** page. |
+| **Use it from Excel, scripts or any language** | `ghost-api serve turftown.in/chennai` gives every endpoint a clean local URL, e.g. `http://127.0.0.1:8900/turftown.in~chennai/get_nearby_venues?area=velachery&format=csv`, with interactive docs at `/docs`. |
+| **Let an AI assistant use the site** | `claude mcp add ghost-turftown -- ghost-api mcp turftown.in/chennai` (after `pip install -e ".[mcp]"`). Claude can then answer "cheapest football turfs near Velachery?" by calling the site's endpoints itself. |
+| **Check a site for personal-data leaks** | Every run looks for emails, phone numbers, PAN/GST/Aadhaar numbers, UPI IDs and bank details in the replies. They are **hidden** before anything is stored, and the report, the terminal and the dashboard say which endpoints hand them out, and whether a login is needed. Use it on your own site, or one you're allowed to test. |
+| **Write code with it** | Use the generated, tested Python or TypeScript client (see [Output](#output)), or the examples in [`examples/`](examples). |
+
+All of these only **read** data: they go through the same rules as the explorer (robots.txt,
+one request per second, nothing that looks like it changes data unless you say
+`generate --allow "POST /path"` or press "I checked: it only reads" in the dashboard), and
+they keep personal data hidden unless you pass `--include-sensitive` for your own site.
+
+---
+
+## Three ways to see the results
+
+**1. The terminal.** Type `ghost-api` and a session opens, like Claude Code or Gemini CLI.
+Paste a website and what you want ("quotes.toscrape.com/scroll list quotes with their
+authors"), watch each phase and agent step as it happens, and get a summary card with
+**clickable links** to the report and the files (Ctrl+click in Windows Terminal or VS Code;
+the full `file:///` link is shown either way). Then keep going: `/fetch` the data, `/watch
+add` alerts, `/open` the report, or paste another site. The session stays open until you
+type `/exit` or press **Ctrl+C twice**; a single Ctrl+C only stops what's running. `/help`
+lists the commands, Tab completes them, and the arrow keys bring back earlier lines.
+`ghost-api run ...` still does a single run, and `--plain` gives simple lines for logs.
+
+![The terminal UI after a real run on quotes.toscrape.com](https://raw.githubusercontent.com/Danielraj5605/Ghost-api/main/docs/images/terminal-run.svg)
+
+**2. The web dashboard.** `ghost-api ui` opens it in your browser (local only, at
+`http://127.0.0.1:8800`). Start a run, watch the agent's browser and reasoning live, then
+browse everything it found. Every past run can be replayed.
+
+| Start a run | Watch it live |
+|---|---|
+| ![Home page with the URL and goal form](https://raw.githubusercontent.com/Danielraj5605/Ghost-api/main/docs/images/dashboard-home.png) | ![Live run with the browser view, activity and endpoints](https://raw.githubusercontent.com/Danielraj5605/Ghost-api/main/docs/images/dashboard-run.png) |
+| **Endpoints and their fields** | **"Try it": call an endpoint with your own parameters** |
+| ![Site page with the endpoint list and details](https://raw.githubusercontent.com/Danielraj5605/Ghost-api/main/docs/images/dashboard-site.png) | ![Try it showing a live response](https://raw.githubusercontent.com/Danielraj5605/Ghost-api/main/docs/images/dashboard-try-it.png) |
+| **The generated client** | **On a phone** |
+| ![Client code tab with Python and TypeScript](https://raw.githubusercontent.com/Danielraj5605/Ghost-api/main/docs/images/dashboard-code.png) | <img src="https://raw.githubusercontent.com/Danielraj5605/Ghost-api/main/docs/images/dashboard-phone.png" alt="The site page on a phone" width="260"> |
+
+"Try it" only calls endpoints that read data. Anything that looks like it changes data
+(add to cart, delete, update...) is refused.
+
+**3. The shareable report.** Every run writes `output/<site>/report.html`: one file with
+the endpoints, the run's activity, the generated code, the OpenAPI spec and the summary. It
+works offline and needs no server, so you can email it or attach it to an issue. Example
+values of sensitive parameters (keys, tokens) are left out.
+
+![report.html opened from disk](https://raw.githubusercontent.com/Danielraj5605/Ghost-api/main/docs/images/report-html.png)
+
+---
+
+## Usage
+
+```bash
+# Basic run
+ghost-api run https://quotes.toscrape.com/scroll --goal "list quotes with their authors and tags"
+
+# Watch the browser while it works
+ghost-api run https://quotes.toscrape.com/scroll --goal "list quotes" --headed
+
+# Tighter budget
+ghost-api run https://example.com --goal "search products" --max-llm-calls 30 --max-steps 25
+
+# Re-check a known site from memory: replay first, browse only if needed
+ghost-api refresh quotes.toscrape.com
+ghost-api refresh quotes.toscrape.com --no-llm   # never call the LLM
+
+# Show what's stored for a site, or replay one run step by step
+ghost-api inspect quotes.toscrape.com
+ghost-api inspect --run 12
+
+# Write report.md, report.html and openapi.yaml from stored data
+ghost-api report quotes.toscrape.com
+
+# Regenerate the client from stored data (no browser; --no-llm skips the naming call)
+ghost-api generate quotes.toscrape.com
+ghost-api generate quotes.toscrape.com --lang typescript   # TypeScript instead
+# Include a POST endpoint you checked only reads data (Ghost API skips those by itself)
+ghost-api generate turftown.in/chennai --allow "POST /api/v2/venue/web/venues_around_you"
+
+# Re-run the generated client's tests against the live API (and repair if needed)
+ghost-api verify quotes.toscrape.com
+
+# Check that your LLM keys work
+ghost-api doctor
+
+# Delete everything stored about a site (asks first; see SECURITY.md)
+ghost-api forget quotes.toscrape.com
+
+# Download data (no code): list what's available, then fetch all pages to a file
+ghost-api fetch quotes.toscrape.com
+ghost-api fetch quotes.toscrape.com list_quotes --pages 10 --format xlsx
+
+# Watch for changes and get alerts
+ghost-api watch add quotes.toscrape.com list_quotes --every 6h --notify ntfy:my-long-private-topic
+ghost-api watch list
+ghost-api watch run              # check now
+ghost-api watch schedule         # keep checking in the background (Windows Task Scheduler)
+
+# Use sites from other tools
+ghost-api serve quotes.toscrape.com            # REST API on http://127.0.0.1:8900 (docs at /docs)
+ghost-api mcp quotes.toscrape.com              # MCP server for AI assistants (stdio)
+
+# Open the web dashboard (local only)
+ghost-api ui
+ghost-api ui --port 8900 --no-open
+
+# The interactive session (stays open until /exit or Ctrl+C twice)
+ghost-api
+
+# Simple line-by-line output for logs and CI
+ghost-api --plain run https://quotes.toscrape.com/scroll --goal "list quotes"
+
+# Capture without the AI agent (just load and scroll)
+ghost-api run https://quotes.toscrape.com/scroll --goal "list quotes" --no-agent
+```
+
+Inside the session:
+
+| Type | It does |
+|---|---|
+| `quotes.toscrape.com/scroll list quotes with their authors` | Explores the site for that goal |
+| more words after a run | A new goal for the same site |
+| `/fetch [SITE] [OPERATION] [name=value ...]` | Downloads data (the current site if none is given) |
+| `/watch add ...`, `/watch` | Alerts, and your list of watches |
+| `/open`, `/report` | Opens or rewrites the current site's report |
+| `/sites`, `/inspect`, `/refresh`, `/generate`, `/serve`, `/ui`, `/doctor` | Same as the `ghost-api` commands |
+| `/lang typescript`, `/agent off` | Settings for the next runs |
+| `/help`, `/clear`, `/exit` | Help, clear the screen, end (or Ctrl+C twice) |
+
+| Option | Default | Meaning |
+|---|---|---|
+| `--goal` | required | What you want the client to do |
+| `--lang` | `python` | Client language: `python` or `typescript` |
+| `--headed` | off | Show the browser window |
+| `--max-steps` | 40 | Maximum agent actions |
+| `--max-llm-calls` | 50 | Maximum LLM calls |
+| `--max-minutes` | 15 | Time limit |
+| `--rate-limit` | 1.0 | Maximum requests per second to the target site |
+| `--ignore-robots` | off | Skip the robots.txt check (use responsibly) |
+| `--no-agent` | off | Don't use the LLM; just load and scroll the page |
+| `--no-probe` | off | Don't test endpoints with direct calls |
+| `--max-probes` | 60 | Maximum direct calls for testing endpoints |
+| `--no-generate` | off | Don't generate a Python client |
+| `--no-verify` | off | Don't run the generated tests against the live API |
+| `--out` | `output` | Folder for generated clients |
+
+**Example projects** built on generated clients, each with a plain-language README:
+[`examples/quote_book`](examples/quote_book) (a searchable book of quotes) and
+[`examples/turf_finder`](examples/turf_finder) (find sports venues near you in Chennai).
+
+---
+
+## Output
+
+Results are kept per **site**: the domain, plus the first part of the start URL's path when
+it names a section of the site. `https://hn.algolia.com/` is `hn.algolia.com`, while
+`https://www.google.com/maps` is `www.google.com/maps`, so Maps and Search on the same domain
+keep separate memory and files. IDs, file names (`index.html`) and language codes (`/en/`)
+don't count as sections. Commands accept the site, a URL, or just the domain when it has only
+one site (`ghost-api refresh quotes.toscrape.com`); with several, Ghost API lists them.
+
+```
+output/quotes.toscrape.com/scroll/
+|-- quotes_toscrape_client.py        # The client: sync and async classes, typed errors
+|-- models.py                        # Pydantic data models
+|-- test_quotes_toscrape_client.py   # Live tests (run automatically by `run` and `verify`)
+|-- package/python/                  # The same client as an installable, typed package
+|-- package/typescript/              # (with --lang typescript) package.json, tsconfig, src/
+|-- openapi.yaml                     # OpenAPI 3.1 spec (imports into Postman, Bruno, Insomnia)
+|-- report.md                        # Endpoints, confidence, evidence, reasoning log
++-- report.html                      # The same and more, as one shareable offline page
+```
+
+**Tests that run anywhere.** The generated tests replay recorded replies (`cassettes.json`)
+by default, so they pass offline and in CI; `GHOST_API_LIVE=1` runs them against the real API.
+
+**Use the client the way you prefer.** Copy the loose files into a project, or install the
+package (`pip install output/quotes.toscrape.com/scroll/package/python`, or
+`npm install .../package/typescript`); the import is the same either way. Each package has a
+README with every method, a `curl` command per endpoint, and the error types. Generated code
+passes `mypy --strict` and `tsc --strict`.
+
+```python
+from quotes_toscrape_client import AsyncQuotesToscrapeClient, QuotesToscrapeNotFoundError
+
+async with AsyncQuotesToscrapeClient() as api:     # same methods as the sync client
+    async for page in api.iter_get_quotes(max_pages=3):
+        ...
+```
+
+Errors are typed: catch `QuotesToscrapeError` for any failure, or `...AuthError` (401/403),
+`...NotFoundError` (404), `...RateLimitError` (429) or `...ServerError` (5xx) for one kind.
+
+**Know when the site changes.** Every client has a version. When a refresh finds the API
+changed, the client is regenerated and its version goes up: major when code using it could
+break (a field removed, a type changed), minor when something was added. Each package has a
+`CHANGELOG.md`.
+
+```bash
+ghost-api diff quotes.toscrape.com/scroll --live      # call each endpoint once and compare
+ghost-api diff quotes.toscrape.com/scroll --history   # every client version
+```
+
+To get a pull request when that happens, use the GitHub Action: see
+[docs/github-action.md](docs/github-action.md).
+
+**Sites Ghost API can't browse by itself** (a login, a bot check, a long flow): browse them
+yourself with DevTools open, use **Network > Save all as HAR**, then
+
+```bash
+ghost-api import session.har --goal "my orders"     # add --offline to send nothing to the site
+```
+
+Cookies, tokens and personal data are removed before anything is stored. Delete the `.har`
+file afterwards: it holds them in plain text.
+
+**Sites that need a login.** Log in once yourself; Ghost API keeps the session (encrypted
+on Windows) and later runs, refreshes and fetches use it:
+
+```bash
+ghost-api login https://example.com/account     # a browser opens: log in, press Enter
+ghost-api run https://example.com/account --goal "my orders"
+ghost-api logout example.com                     # delete it
+```
+
+**Work offline.** `ghost-api mock quotes.toscrape.com/scroll` serves the captured replies at
+the same paths on `http://127.0.0.1:8901`: point a front end or a test at it, with no traffic
+to the real site.
+
+Excerpt from a real `report.md`:
+
+```
+## Endpoints
+
+| Endpoint          | Status   | Confidence | Samples | Auth | Paging       | Notes                 |
+|-------------------|----------|------------|---------|------|--------------|-----------------------|
+| `GET /api/quotes` | verified | High       | 6       | none | page: `page` | 2 live test(s) passed |
+
+## Hypotheses tested
+
+- confirmed: can be called directly, outside the browser (HTTP 200, 10 records)
+- confirmed: paginates with page parameter 'page' (the next page returned 10 different records)
+- confirmed: needs none of its custom headers or cookies
+```
+
+---
+
+## Project structure
+
+```
+ghost-api/
+|-- ghost_api/
+|   |-- cli/                # The command line: explore, client, data and manage commands
+|   |-- api.py              # The same features as Python functions
+|   |-- workflows.py        # Run and refresh, shared by the CLI and the API
+|   |-- drift.py            # Client versions and API changes
+|   |-- pipeline.py         # A run from start to finish, as events (no printing)
+|   |-- events.py           # The events every interface shows (stored for replay)
+|   |-- ui/terminal.py      # Live terminal UI (Rich)
+|   |-- web/                # Dashboard server (FastAPI) and the prebuilt frontend
+|   |-- config.py           # Settings and budgets
+|   |-- errors.py           # Plain-language error messages
+|   |-- net.py              # robots.txt and rate limiting
+|   |-- netguard.py         # Network boundary: no private addresses, reply size limits
+|   |-- agent/              # Exploration loop, prompts, actions
+|   |-- browser/            # Playwright control, snapshots, safety filter, egress proxy, request gate
+|   |-- capture/            # Network recording, noise filters, redaction
+|   |-- analysis/           # URL templates, schemas, detectors (pagination, auth, bots)
+|   |-- probe/              # Safe replays and hypothesis testing
+|   |-- generator/          # Naming, models, client spec, Jinja2 templates
+|   |-- verify/             # Run generated tests, classify failures, repair
+|   |-- export/             # report.md, report.html, OpenAPI 3.1, CSV/Excel, confidence
+|   |-- runtime.py          # Call a site's endpoints by name, follow pages, get rows
+|   |-- watch.py            # Watches: compare snapshots, notifications
+|   |-- serve/              # REST API and MCP server
+|   |-- llm/                # Provider gateway: fallback, cooldown, validation, cache
+|   +-- memory/             # SQLite store, lessons, memory for refresh
+|-- tests/                  # 400+ tests (no LLM quota needed)
+|-- evals/                  # Demo shop, evaluation sites, scoring script
+|-- web/                    # Dashboard and report frontend source (React + Vite + Tailwind)
+|-- docs/images/            # README screenshots
+|-- .env.example
+|-- pyproject.toml
+|-- README.md
++-- IMPLEMENTATION.md
+```
+
+---
+
+## Running on free LLM tiers
+
+Ghost API is designed to cost **$0**:
+
+- **Small prompts.** Pages and traffic are compressed into short summaries (under 8k tokens per call).
+- **Few calls.** 1–10 LLM calls per site in the [evaluation](#evaluation), because code does the heavy work and the agent stops as soon as the goal is covered. `refresh` often needs none.
+- **Fallback chain.** Each provider has several candidate models. A rate-limited or overloaded model rests while another is used; a retired model is skipped; a bad key disables its provider. Order: Gemini, Groq, Gemini Lite, OpenRouter. Run `ghost-api doctor` to check your keys.
+- **Caching.** Repeated prompts are served from a local SQLite cache.
+
+> Free-tier limits change often, so check each provider's current pricing page. Free tiers may use prompts for training. Ghost API redacts cookies and tokens before anything reaches an LLM, but don't point it at private data.
+
+---
+
+## Responsible use
+
+Ghost API is built to behave like a polite, careful developer:
+
+- Respects `robots.txt` by default, for its own requests and for the pages and API calls the
+  browser makes for the site (checked before they are sent)
+- Rate-limits its own requests and the site's API calls in the browser (1 request per second
+  by default)
+- Never lets a website steer it to your computer or local network: every browser connection,
+  redirects included, goes through a local proxy that refuses private addresses (unless the
+  site you gave it is itself local)
+- Stops reading any reply bigger than 5 MB
+- Never tries to bypass CAPTCHAs or bot protection; it detects them and stops
+- Never clicks destructive or purchase actions (delete, buy, pay)
+- Never stores passwords; for logins, you log in by hand in the browser window
+- Redacts cookies and tokens from logs, reports and LLM prompts
+- Uses an honest user-agent string
+
+Check a site's terms of service before generating a client for it. For demos and testing, use practice sites such as `quotes.toscrape.com/scroll`, which loads its data from a JSON API. (`books.toscrape.com` renders on the server, so it has no API to discover.)
+
+Also:
+
+- **Personal data is hidden, always.** Emails, phone numbers, PAN/GST/Aadhaar numbers, UPI IDs
+  and bank details in a site's replies are replaced before Ghost API stores anything, and stay
+  hidden in fetches, watches, the REST API and MCP tools. `--include-sensitive` turns that off,
+  for auditing your own site only. If you find a site exposing such data, tell the owner
+  privately instead of collecting it.
+- **Watches and servers keep the same manners:** read-only, one request per second per site,
+  robots.txt respected, checks no more often than every 5 minutes.
+- **Your data stays on your machine** except what the AI agent sends to your LLM provider.
+  [SECURITY.md](SECURITY.md) lists what is stored, what is sent where, and how to delete it:
+  `ghost-api forget SITE` removes everything stored about a site.
+
+---
+
+## Evaluation
+
+Four sites, run end to end with free LLM tiers on a Windows laptop with 8 GB of RAM
+(full table and method: [evals/results/results.md](evals/results/results.md)).
+
+| Site | What it tests | Recall | Precision | Tests passing | LLM calls | Time | Peak memory |
+|---|---|---|---|---|---|---|---|
+| demo-shop (local) | Known answer: 5 endpoints, page + cursor paging, traps | 100% | 100% | 100% | 6 | 36 s | 415 MB |
+| quotes.toscrape.com | Infinite-scroll JSON API | 100% | 100% | 100% | 2 | 18 s | 423 MB |
+| hn.algolia.com | Third-party POST API with a key | 100% | 100% | 100% | 7 | 123 s* | 494 MB |
+| books.toscrape.com | No API at all (must not invent one) | n/a | 100% | n/a | 1 | 9 s | 452 MB |
+
+- **Recall:** share of the goal's real endpoints found. **Precision:** share of reported
+  endpoints that are real (nothing was invented). **Tests passing:** generated client tests
+  that passed against the live API. Peak memory includes Chromium.
+- The demo shop's "Add to cart" (a POST that changes data) was never called and never generated.
+- \*Re-measured after the gateway learned to switch away from overloaded models at once;
+  the earlier run took 259 s and 10 calls, mostly retrying an overloaded Gemini model.
+- The first attempt failed on two sites because two providers' free models had been
+  retired; that led to the per-model fallback routing described above.
+
+### The quality gate (every change, no LLM)
+
+Ten local test sites with known answers ([evals/fixtures.py](evals/fixtures.py)), each built
+around one hard case, are scored on every change in CI. A score below
+[evals/thresholds.json](evals/thresholds.json) fails the build:
+
+| Test site | Must hold |
+|---|---|
+| offset-paging, cursor-paging | The endpoint is found and its paging style is detected |
+| api-key-header | The `x-api-key` header is found to be required |
+| robots-disallow | The API path robots.txt disallows is **never requested**, by the browser or by replays |
+| huge-reply | A 9 MB reply is dropped as `response-too-large`; the small endpoint is still found |
+| xssi-guard | A reply behind Google-style `)]}'` is still read |
+| no-api, bot-wall | Nothing is invented; a bot-protection page stops the run (`bot-protection`) |
+| personal-data | The email and phone in the replies are **never stored** |
+| prompt-injection | (agent runs) The planted "delete all data" instruction is never followed |
+| demo-shop | All endpoints; the cart (a POST that changes data) is never called or generated |
+
+The first run of this gate found a real bug: cursor paging went undetected when the newest
+captured page was the last one. It is fixed, and the gate keeps it fixed.
+
+Run it yourself: `python -m evals.run_eval --suite local --no-agent --check`. The full set,
+with the agent and the live sites above, is `python -m evals.run_eval` (uses LLM calls); it
+also runs weekly in CI, dated, without blocking builds.
+
+---
+
+## Limitations
+
+The full, prioritised list of known problems is in [KNOWN_ISSUES.md](KNOWN_ISSUES.md). The main ones:
+
+- Sites with strong bot protection (Cloudflare challenges, CAPTCHAs) will be skipped; you can
+  browse them yourself and `ghost-api import` the HAR file
+- Sites whose `robots.txt` forbids their API paths for bots are respected, so Ghost API will not call those endpoints directly. Example: Wikipedia disallows `/w/` (where its search suggestions live) and refuses clients that don't identify themselves, so the run stops at the rules check. Wikipedia offers an official API for that purpose. If a site just wants to know who you are, set `GHOST_API_CONTACT` in `.env` (your own email or a URL; it is never filled in for you)
+- Server-rendered sites with no JSON API have nothing to discover
+- WebSocket, server-sent-event and gRPC APIs are not mapped yet; GraphQL is (one method per
+  operation), except batches of several operations in one request
+- Small free models sometimes make poor exploration choices; budgets and stopping rules limit the damage
+- Generated clients can break when the site changes its internal API: `ghost-api diff --live`
+  and the GitHub Action notice it, and `refresh` makes a new client version
+
+---
+
+## Roadmap
+
+| Stage | Focus | Status |
+|---|---|---|
+| 0 | Project setup | Done |
+| 1 | Traffic capture | Done |
+| 2 | API analysis (URL grouping, schemas) | Done |
+| 3 | LLM gateway (fallback + cache) | Done |
+| 4 | Agent exploration loop | Done |
+| 5 | Hypothesis testing | Done |
+| 6 | Code generation | Done |
+| 7 | Verification and replanning | Done |
+| 8 | Memory | Done |
+| 9 | Reporting and OpenAPI export | Done |
+| 10 | Evaluation and polish | Done |
+| UI 0–6 | Event pipeline, live terminal, web dashboard, report.html, docs | Done |
+| Use 1–6 | Data download, watch & alerts, REST + MCP servers, personal-data leak check | Done |
+
+Full details are in [IMPLEMENTATION.md](IMPLEMENTATION.md). To work on the dashboard itself,
+see [web/README.md](web/README.md).