@bpinternal/site-scout 0.1.0

package/README.md

@@ -0,0 +1,92 @@
+# @bpinternal/site-scout
+
+Website URL discovery + LLM page selection — scouts a site for the pages worth
+reading. Given a website, it finds candidate URLs, scores them into a
+prioritized tree, then makes **one** injected-LLM call to pick the pages best
+suited to seed a support bot's knowledge base.
+
+## What it does
+
+Two independent units, composed:
+
+```
+discover(website, opts, deps) -> MasterMap   (prioritized tree)
+select(map, opts, deps)       -> Selection   (best ≤limit URLs)
+```
+
+`findKnowledgeUrls()` runs guard → discover → select for the common case.
+
+- **Discovery** pulls candidate URLs from **5 sources** — `llms.txt` /
+  `llms-full.txt`, `sitemap.xml` (incl. children), `robots.txt`, the host's
+  `discoverUrls` crawl, and a `site:` web search — normalizes/dedupes them, and
+  builds a **score tree**: one node per path segment, with aggregate scores and
+  leaf counts so selection can reason about cardinality without walking every
+  leaf.
+- **Selection** renders the tree as a compact numbered list and makes **one
+  LLM call** (`extract`) that returns the picked page _numbers_ (best-first),
+  which map back to URLs. The LLM pick is the decision — no deterministic
+  fallback list.
+
+## The injection contract (`SiteScoutDeps`)
+
+site-scout does no I/O and never imports a runtime itself. Every consumer
+provides the I/O + LLM as `deps`:
+
+```ts
+type SiteScoutDeps = {
+  // discovery I/O
+  fetchText(url, baseHost, timeoutMs?): Promise<string | null>
+  fetchJson(url, baseHost, timeoutMs?): Promise<T | null>
+  discoverUrls(args): Promise<{ urls: string[]; stopReason: string }>
+  webSearch(args): Promise<{ results: SearchResult[] }>
+  // the one LLM touchpoint (zai-shaped extract)
+  extract(input, schema, { instructions }): Promise<z.infer<schema>>
+  // optional record/replay cache around the LLM call (defaults to passthrough)
+  cache?(kind, keyParts, fn): Promise<T>
+}
+```
+
+`@bpinternal/zui` is a **peer dependency** — consumers bring their own
+(the schema passed to `extract` uses it).
+
+## Usage
+
+```ts
+import { findKnowledgeUrls } from '@bpinternal/site-scout'
+
+const result = await findKnowledgeUrls(
+  {
+    website: 'https://example.com',
+    limit: 50,
+    topics: ['pricing', 'docs'],
+    prompt: 'Prioritize developer docs, API reference, pricing.',
+    context: { company: 'Example', website: 'https://example.com', overview: '…' },
+  },
+  deps // your SiteScoutDeps implementation
+)
+// result.urls        — prioritized URLs, best first
+// result.discovered  — total distinct URLs found before selection
+// result.stopReason  — 'ok' | 'unsupported_site' | 'no_sources' | 'time_limit_reached'
+```
+
+`discover()` and `select()` are also exported for callers that want to show the
+tree, let a user refine, then select. `fetchText`/`fetchJson`/`hostOf`/
+`siteOrigin` (an SSRF-guarded fetcher + URL helpers) are re-exported so a
+consumer can reuse them when wiring its own deps.
+
+## Record / replay fixtures
+
+The tests are offline and deterministic:
+
+- **Network** — discovery I/O is wrapped by `cachedDeps(group, realDeps)`, which
+  records/replays each call against `src/__fixtures__/<group>.jsonl` (one file
+  per site). In replay mode a miss throws (a test can't silently hit the
+  network); in production `cachedDeps` is a passthrough. Mode is env-driven
+  (`URL_FIXTURE_RECORD` to record; `NODE_ENV=test` / `VITEST` / `BUN_TEST` /
+  `URL_FIXTURE_REPLAY` to replay).
+- **LLM** — the `select` call is keyed and replayed from
+  `src/__fixtures__/llm-cache.jsonl`. The package e2e (`index.e2e.test.ts`)
+  loads that file and serves the recorded decision, so no live model is called.
+
+Fixtures are **recorded via the app harness** (viber-compiler-bot), which owns
+the real runtime/LLM wiring; this package only replays them.