soup-chop 1.0.3 → 1.0.5

package/README.md

@@ -242,6 +242,21 @@ Target selection is explicit:
 `soupChop` does **not** merge local and published corpora implicitly. When you
 select a local target, the server stays on that filesystem corpus only.
 
+Source discovery is broader than package-shipped markdown alone:
+
+- `README.md`
+- package-local markdown under `doc/` and `docs/`
+- a small allowlist of top-level markdown docs such as `API.md`, `FAQ.md`, `MIGRATING.md`, `UPGRADING.md`, `CHANGELOG.md`, and `CONTRIBUTING.md`
+- synthetic `source_jsdoc` entries generated from exported TypeScript JSDoc/TSDoc discovered from local packages, published npm tarballs, and GitHub/GitLab repositories
+- same-origin website docs crawled from an explicit `docs_url`, or auto-detected from README links when a package is website-first
+- repository-derived wiki pages indexed as `wiki_doc` when a GitHub / GitLab wiki is discoverable
+- one-hop local workspace dependency docs when resolving a package through `workspace_path`
+
+Website pages are indexed as `website_doc` sources and use `origin` values such as `website/dockview.dev/docs/core/overview`.
+Synthetic JSDoc pages use origins like `jsdoc/createstableclient`. Wiki pages use `wiki/...` origins.
+
+For website ingestion, soup-chop currently stays **Node-only**: it supports static HTML crawling plus manifest-aware discovery such as VitePress `hashmap.json` and sitemap-backed doc sites, but intentionally does **not** require Playwright or a browser runtime for JS-rendered SPA fallback.
+
 `compare_versions` and `get_upgrade_guide` currently remain **npm-only**.
 
 ### `get_capabilities`
@@ -263,7 +278,7 @@ metadata such as source precedence, cache freshness, and the resolved local
 package root.
 
 This is the best tool to call when you want to verify which files were actually
-analyzed before following up with `search_docs`, `get_toc`, or the Mincer tools.
+analyzed before following up with `search_docs`, `get_toc`, or the Mincer tools. For website-first packages, this can include `website_doc` entries in addition to packaged markdown.
 
 **Parameters:**
 
@@ -271,6 +286,7 @@ analyzed before following up with `search_docs`, `get_toc`, or the Mincer tools.
 | :-------- | :------- | :------- | :---------------------------------------------------------------------- |
 | `package` | `string` | No       | NPM package name, or workspace package name when using `workspace_path` |
 | `version` | `string` | No       | NPM version string for remote usage, or optional exact local-version check |
+| `docs_url` | `string` | No      | Optional docs website entry URL; when provided, soup-chop crawls this website as an additional documentation source |
 | `local_path` | `string` | No    | Local package directory or local `package.json` path |
 | `workspace_path` | `string` | No | Local monorepo / workspace root; requires `package` |
 
@@ -289,40 +305,46 @@ analyzed before following up with `search_docs`, `get_toc`, or the Mincer tools.
 
 ```json
 {
-  "package": "@acme/docs-pkg",
-  "version": "1.5.0",
+  "package": "dockview",
+  "version": "5.1.0",
   "sources": [
-    {
-      "sourceId": "api__md",
-      "sourceKind": "top_level_doc",
-      "origin": "API.md",
-      "title": "API.md",
-      "lineCount": 12
-    },
-    {
-      "sourceId": "docs__guide__md",
-      "sourceKind": "package_doc",
-      "origin": "docs/guide.md",
-      "title": "guide.md",
-      "lineCount": 18
-    },
     {
       "sourceId": "readme__md",
       "sourceKind": "readme",
       "origin": "README.md",
       "title": "README.md",
-      "lineCount": 42
+      "lineCount": 81
+    },
+    {
+      "sourceId": "website__dockview__dev__docs__core__overview",
+      "sourceKind": "website_doc",
+      "origin": "website/dockview.dev/docs/core/overview",
+      "title": "Overview | Dockview",
+      "lineCount": 3
+    },
+    {
+      "sourceId": "website__dockview__dev__docs__core__panels__register",
+      "sourceKind": "website_doc",
+      "origin": "website/dockview.dev/docs/core/panels/register",
+      "title": "Registering Panels | Dockview",
+      "lineCount": 5
     }
   ],
   "debug": {
     "precedence": "explicit_target_only",
-    "targetMode": "local",
-    "freshness": "mutable_local_filesystem",
-    "cacheEnabled": false,
-    "cacheReason": "Local filesystem targets are mutable, so corpus and search caches stay disabled.",
-    "cachePaths": {},
-    "packageRoot": "/home/me/dev/acme/packages/docs-pkg",
-    "sourceOrigins": ["API.md", "docs/guide.md", "README.md"]
+    "targetMode": "npm",
+    "freshness": "immutable_published_version",
+    "cacheEnabled": true,
+    "cacheReason": "Exact published npm versions are immutable, so manifest and search caches are safe.",
+    "cachePaths": {
+      "manifest": "/home/me/.cache/soup-chop/dockview/5.1.0/sources-manifest.json",
+      "searchIndex": "/home/me/.cache/soup-chop/dockview/5.1.0/search-index.json"
+    },
+    "sourceOrigins": [
+      "README.md",
+      "website/dockview.dev/docs/core/overview",
+      "website/dockview.dev/docs/core/panels/register"
+    ]
   }
 }
 ```
@@ -334,20 +356,22 @@ analyzed before following up with `search_docs`, `get_toc`, or the Mincer tools.
 Returns a structured table of contents for a package's markdown docs.
 
 Without `origin`, it returns a package-wide multi-document TOC that includes
-synthetic file-root entries such as `README.md`, `CHANGELOG.md`, or
-`docs/faq.md`.
+synthetic file-root entries such as `README.md`, `CHANGELOG.md`, `docs/faq.md`, or website origins such as `website/dockview.dev/docs/core/overview`.
 
 With `origin`, it returns the TOC for a single markdown document.
 
+For `README.md`, soup-chop also adds direct synthetic entries for common top-level sections such as `Installation`, `API`, `Usage`, and `FAQ` so callers can jump straight to those sections without the full hierarchical path.
+
 **Parameters:**
 
 | Name      | Type     | Required | Description                            |
 | :-------- | :------- | :------- | :------------------------------------- |
 | `package` | `string` | No       | NPM package name, or workspace package name when using `workspace_path` |
 | `version` | `string` | No       | NPM version string for remote usage, or optional exact local-version check |
+| `docs_url` | `string` | No      | Optional docs website entry URL; when provided, soup-chop crawls this website as an additional documentation source |
 | `local_path` | `string` | No    | Local package directory or local `package.json` path |
 | `workspace_path` | `string` | No | Local monorepo / workspace root; requires `package` |
-| `origin`  | `string` | No       | Markdown source path from `search_docs` such as `"docs/faq.md"` or `"CHANGELOG.md"` |
+| `origin`  | `string` | No       | Source path from `search_docs` such as `"docs/faq.md"`, `"CHANGELOG.md"`, or `"website/dockview.dev/docs/core/overview"` |
 
 **Example call:**
 
@@ -424,16 +448,20 @@ document.
 You can identify the target either with `topic_id`, or with `path` plus an
 optional `origin`.
 
+For website docs, `origin` and `topic_id` use the normalized website source path returned by `search_docs` and `get_toc`.
+The same applies to synthetic README section shortcuts such as `README.md#Installation`.
+
 **Parameters:**
 
 | Name      | Type     | Required | Description                                              |
 | :-------- | :------- | :------- | :------------------------------------------------------- |
 | `package` | `string` | No       | NPM package name, or workspace package name when using `workspace_path` |
 | `version` | `string` | No       | NPM version string for remote usage, or optional exact local-version check |
+| `docs_url` | `string` | No      | Optional docs website entry URL; when provided, soup-chop crawls this website as an additional documentation source |
 | `local_path` | `string` | No    | Local package directory or local `package.json` path |
 | `workspace_path` | `string` | No | Local monorepo / workspace root; requires `package` |
 | `path`    | `string` | No       | Section path from `get_toc` (e.g., `"Zod/Installation"`) |
-| `origin`  | `string` | No       | Markdown source path from `search_docs` such as `"docs/faq.md"` or `"CHANGELOG.md"` |
+| `origin`  | `string` | No       | Source path from `search_docs` such as `"docs/faq.md"`, `"CHANGELOG.md"`, or `"website/dockview.dev/docs/core/panels/register"` |
 | `topic_id`| `string` | No       | Stable topic identifier from `search_docs` or aggregated `get_toc` |
 
 **Example call using `topic_id`:**
@@ -475,10 +503,11 @@ extraction scope.
 | :-------- | :------- | :------- | :---------------------------------------------------------------------- |
 | `package` | `string` | No       | NPM package name, or workspace package name when using `workspace_path` |
 | `version` | `string` | No       | NPM version string for remote usage, or optional exact local-version check |
+| `docs_url` | `string` | No      | Optional docs website entry URL; when provided, soup-chop crawls this website as an additional documentation source |
 | `local_path` | `string` | No    | Local package directory or local `package.json` path |
 | `workspace_path` | `string` | No | Local monorepo / workspace root; requires `package` |
 | `path`    | `string` | No       | Optional section path from `get_toc` |
-| `origin`  | `string` | No       | Optional markdown source path such as `"docs/faq.md"` or `"CHANGELOG.md"` |
+| `origin`  | `string` | No       | Optional source path such as `"docs/faq.md"`, `"CHANGELOG.md"`, or `"website/dockview.dev/docs/core/panels/add"` |
 | `topic_id`| `string` | No       | Stable topic identifier from `search_docs` or aggregated `get_toc` |
 
 **Example call:**
@@ -510,6 +539,7 @@ the documentation sections that mention it.
 | :-------- | :------- | :------- | :---------------------------------------------------------------------- |
 | `package` | `string` | No       | NPM package name, or workspace package name when using `workspace_path` |
 | `version` | `string` | No       | NPM version string for remote usage, or optional exact local-version check |
+| `docs_url` | `string` | No      | Optional docs website entry URL; when provided, soup-chop crawls this website as an additional documentation source |
 | `local_path` | `string` | No    | Local package directory or local `package.json` path |
 | `workspace_path` | `string` | No | Local monorepo / workspace root; requires `package` |
 | `name`    | `string` | Yes      | Identifier or literal symbol to locate |
@@ -535,7 +565,7 @@ the documentation sections that mention it.
 
 Searches `README.md`, package-local markdown under `doc/` and `docs/`, and a
 small allowlist of top-level markdown docs such as `API.md`, `FAQ.md`,
-`MIGRATING.md`, `UPGRADING.md`, `CHANGELOG.md`, and `CONTRIBUTING.md`.
+`MIGRATING.md`, `UPGRADING.md`, `CHANGELOG.md`, and `CONTRIBUTING.md`. For website-first packages, soup-chop can also crawl same-origin website docs and index them as `website_doc` sources.
 
 Results are ranked lexically and return the source origin, section path, line
 ranges, a stable `topicId`, a short preview, and lightweight example metadata
@@ -553,6 +583,7 @@ tokenization and ranking.
 | :-------- | :------- | :------- | :---------------------------------------------------------------------- |
 | `package` | `string` | No       | NPM package name, or workspace package name when using `workspace_path` |
 | `version` | `string` | No       | NPM version string for remote usage, or optional exact local-version check |
+| `docs_url` | `string` | No      | Optional docs website entry URL; when provided, soup-chop crawls this website as an additional documentation source |
 | `local_path` | `string` | No    | Local package directory or local `package.json` path |
 | `workspace_path` | `string` | No | Local monorepo / workspace root; requires `package` |
 | `query`   | `string` | Yes      | Keyword query or natural-language question                              |
@@ -608,6 +639,7 @@ than to raw document navigation.
 | :-------- | :------- | :------- | :---------------------------------------------------------------------- |
 | `package` | `string` | No       | NPM package name, or workspace package name when using `workspace_path` |
 | `version` | `string` | No       | NPM version string for remote usage, or optional exact local-version check |
+| `docs_url` | `string` | No      | Optional docs website entry URL; when provided, soup-chop crawls this website as an additional documentation source |
 | `local_path` | `string` | No    | Local package directory or local `package.json` path |
 | `workspace_path` | `string` | No | Local monorepo / workspace root; requires `package` |
 | `query`   | `string` | Yes      | Keyword query or natural-language question                              |
@@ -661,6 +693,7 @@ range, snippet, and detector names so the caller can pivot back into
 | :-------- | :------- | :------- | :----------------------------------------------------------------------- |
 | `package` | `string` | No       | NPM package name, or workspace package name when using `workspace_path` |
 | `version` | `string` | No       | NPM version string for remote usage, or optional exact local-version check |
+| `docs_url` | `string` | No      | Optional docs website entry URL; when provided, soup-chop crawls this website as an additional documentation source |
 | `local_path` | `string` | No    | Local package directory or local `package.json` path |
 | `workspace_path` | `string` | No | Local monorepo / workspace root; requires `package` |
 | `limit`   | `number` | No       | Maximum number of ranked trap findings to return, from `1` to `50`, default `10` |
@@ -732,6 +765,7 @@ broader caveat-oriented `get_traps` view.
 | :-------- | :------- | :------- | :----------------------------------------------------------------------------- |
 | `package` | `string` | No       | NPM package name, or workspace package name when using `workspace_path` |
 | `version` | `string` | No       | NPM version string for remote usage, or optional exact local-version check |
+| `docs_url` | `string` | No      | Optional docs website entry URL; when provided, soup-chop crawls this website as an additional documentation source |
 | `local_path` | `string` | No    | Local package directory or local `package.json` path |
 | `workspace_path` | `string` | No | Local monorepo / workspace root; requires `package` |
 | `limit`   | `number` | No       | Maximum number of ranked deprecation findings to return, from `1` to `50`, default `10` |
@@ -799,6 +833,7 @@ general caveats.
 | :-------- | :------- | :------- | :--------------------------------------------------------------------------- |
 | `package` | `string` | No       | NPM package name, or workspace package name when using `workspace_path` |
 | `version` | `string` | No       | NPM version string for remote usage, or optional exact local-version check |
+| `docs_url` | `string` | No      | Optional docs website entry URL; when provided, soup-chop crawls this website as an additional documentation source |
 | `local_path` | `string` | No    | Local package directory or local `package.json` path |
 | `workspace_path` | `string` | No | Local monorepo / workspace root; requires `package` |
 | `limit`   | `number` | No       | Maximum number of ranked constraint findings to return, from `1` to `50`, default `10` |
@@ -866,6 +901,7 @@ operational caveats.
 | :-------- | :------- | :------- | :----------------------------------------------------------------------------- |
 | `package` | `string` | No       | NPM package name, or workspace package name when using `workspace_path` |
 | `version` | `string` | No       | NPM version string for remote usage, or optional exact local-version check |
+| `docs_url` | `string` | No      | Optional docs website entry URL; when provided, soup-chop crawls this website as an additional documentation source |
 | `local_path` | `string` | No    | Local package directory or local `package.json` path |
 | `workspace_path` | `string` | No | Local monorepo / workspace root; requires `package` |
 | `limit`   | `number` | No       | Maximum number of ranked performance findings to return, from `1` to `50`, default `10` |
@@ -933,6 +969,7 @@ guidance rather than general caveats or performance notes.
 | :-------- | :------- | :------- | :----------------------------------------------------------------------------- |
 | `package` | `string` | No       | NPM package name, or workspace package name when using `workspace_path` |
 | `version` | `string` | No       | NPM version string for remote usage, or optional exact local-version check |
+| `docs_url` | `string` | No      | Optional docs website entry URL; when provided, soup-chop crawls this website as an additional documentation source |
 | `local_path` | `string` | No    | Local package directory or local `package.json` path |
 | `workspace_path` | `string` | No | Local monorepo / workspace root; requires `package` |
 | `limit`   | `number` | No       | Maximum number of ranked security findings to return, from `1` to `50`, default `10` |
@@ -1021,7 +1058,7 @@ package, showing added, removed, and unchanged sections.
 | Name      | Type     | Required | Description                                                                            |
 | :-------- | :------- | :------- | :------------------------------------------------------------------------------------- |
 | `package` | `string` | Yes      | NPM package name                                                                       |
-| `v_old`   | `string` | Yes      | NPM version string: exact version or published dist-tag (e.g., `"3.22.0"`, `"latest"`) |
+| `v_old`   | `string` | Yes      | NPM version string: exact version or published dist-tag (e.g., `"3.22.0"`, `"oldest"`) (jk) |
 | `v_new`   | `string` | Yes      | NPM version string: exact version or published dist-tag (e.g., `"3.23.8"`, `"latest"`) |
 
 ### `get_upgrade_guide`
@@ -1100,11 +1137,12 @@ You ask about express@4.18.2
 **Key design decisions:**
 
 - **Version-scoped caching** — Each `package@version` pair is cached independently.
-  Since published NPM versions are immutable, the cache never goes stale.
+  Since published NPM versions are immutable, packaged markdown, website-source manifests, search indexes, and findings caches can be reused safely.
 - **No API keys required** — Everything runs locally. No LLM calls, no embeddings
-  API, no external services beyond UNPKG (a public CDN).
+  API, no external services beyond UNPKG (a public CDN) and the target documentation website when `docs_url` or README-based website detection is used.
 - **Surgical delivery** — The AI gets a *table of contents*, not the full README.
   It can then request only the section it needs, keeping context windows lean.
+- **Website-first docs support** — Packages whose real docs live on a website can be indexed through bounded same-origin crawling and HTML-to-Markdown normalization.
 
 ---
 
@@ -1127,10 +1165,16 @@ Cache is organized as:
 ~/.cache/soup-chop/
 ├── express/
 │   └── 4.18.2/
-│       └── README.md
+│       ├── README.md
+│       ├── sources-manifest.json
+│       ├── search-index.json
+│       └── mincer-findings.json
 ├── zod/
 │   └── 3.23.8/
-│       └── README.md
+│       ├── README.md
+│       ├── sources-manifest.json
+│       ├── search-index.json
+│       └── mincer-findings.json
 └── ...
 ```