@bitbaum/ai-kit 0.15.0 → 0.16.0

package/README.md

@@ -287,6 +287,52 @@ stop paying for a form library.
 React lives on its own subpath and is an **optional** peer, so importing
 `@bitbaum/ai-kit` on a server never pulls in a UI library.
 
+### Seeing the web — search and read, as citable facts
+
+```ts
+import { webSearch, readPage, resultsToFacts, describeEmptySearch } from "@bitbaum/ai-kit/web";
+import { assignFactIds, renderFacts } from "@bitbaum/ai-kit/grounding";
+
+const found = await webSearch("lightning address spec", { limit: 5 });
+if (found.status !== "found") {
+  return describeEmptySearch(found); // the model is told WHICH of the two this is
+}
+const facts = assignFactIds(resultsToFacts(found.results, found.provider));
+```
+
+Backends are a chain, in the same shape and for the same reason as the model
+chain: `searxng` (self-hosted, no key, no third party told what your users are
+looking for) → `brave` (an independent index, so the fallback is not the same
+index asked twice) → `tavily`. Each is included only if its env is set, so the
+common case is one call rather than a policy re-decided in every app.
+
+**The answer is three-valued, and that is the whole point.** `found`; `nothing`
+(a backend answered and had nothing — a real negative the model may state); and
+`could_not_look` (nobody answered — an outage, which the model must **not**
+report as a fact about the world). Collapsing the last two into `[]` is how an
+expired API key becomes a confident sentence about what does not exist on the
+internet. A backend that answers `200` with zero results is walked past rather
+than believed, because that is exactly what a self-hosted metasearch instance
+does when its upstream engines refuse it.
+
+`readPage()` is the other half — search returns titles and one sentence, and
+almost every real question needs the page. It fetches with `redirect: "manual"`
+and re-runs the **full SSRF check on every hop**, because an agent picks its own
+URLs: any page it reads can hand it a link to the cloud metadata service, and
+validating the first URL and then letting `fetch` follow the 302 is not a check
+at all. Truncation is stated in the result rather than implied, so a summary can
+say "the first part of the page" instead of implying it read all of it.
+
+Both produce `Fact`s from `/grounding` rather than a bespoke shape, so retrieved
+web content and retrieved database rows flow through **one** verifier and "cite
+your sources" becomes a string check instead of a line in a prompt. Search
+without citation binding makes hallucination worse, not better: the invented
+sentence is now surrounded by real ones.
+
+It does not summarise, re-rank with an LLM, crawl, render JavaScript, or cache.
+The first two are the model's job and belong upstream where the app's prompt
+lives; the last three are a different product with a different cost profile.
+
 ---
 
 ## What it deliberately does not ship

package/dist/grounding/facts.js

@@ -54,6 +54,13 @@ export const FACT_KINDS = {
     assignment: ["title", "assignee", "status", "due", "fee", "why"],
     document: ["title", "source", "excerpt"],
     pending_action: ["title", "type", "reasoning", "proposed_on", "id"],
+    // Retrieved from the open web rather than stored. `url` is the field that
+    // makes these checkable by a human reader, which is what separates a cited
+    // claim from a confident one. The rest are declared precisely BECAUSE search
+    // engines so often omit them: a rendered `published: <not recorded>` is what
+    // stops a model putting a year on an undated page.
+    web_result: ["title", "url", "published", "snippet", "engine"],
+    web_page: ["title", "url", "retrieved", "truncated"],
 };
 /** Field list for a kind; unknown kinds fall back to whatever the fact carries. */
 export function declaredFields(kind, fallback = []) {

package/dist/web/facts.d.ts

@@ -0,0 +1,60 @@
+/**
+ * Web results → grounded Facts.
+ *
+ * This is the join that makes web search safe to give an agent. On its own,
+ * search makes hallucination WORSE, not better: the model now has a pile of
+ * plausible-sounding text from strangers, mixed into a context window beside
+ * the user's real data, with nothing marking which sentence came from where.
+ * The failure mode is not "the model made something up" — it is "the model
+ * attributed a real sentence to the wrong source", and that is indistinguishable
+ * from competence until someone clicks the link.
+ *
+ * The fix is the one the grounding harness already implements for database
+ * rows. Every retrieved item becomes a `Fact` with a citation handle ([F3]) and
+ * a `url` field, `verifyAnswer` then checks mechanically that every handle in
+ * the answer exists and that proper nouns and numbers in the answer appear in
+ * the evidence. A claim that cites nothing, or cites [F9] when only F1–F4 were
+ * retrieved, is caught by a string check rather than by a reviewer's judgement.
+ *
+ * Two sets come out of here on purpose:
+ *
+ *   facts    — the citable records. Short, uniform, one block per source.
+ *   evidence — the actual prose (snippets, page text). The verifier needs the
+ *              words themselves to judge whether "€4.2 billion" in the answer
+ *              was read or invented; a fact's five short fields are not enough
+ *              of a corpus for that, and passing the page text AS a fact field
+ *              would bury the citable metadata in ten thousand characters.
+ */
+import { type Fact } from "../grounding/facts.js";
+import type { PageOutcome, WebResult } from "./types.js";
+import type { WebSearchResult } from "./search.js";
+/** Search hits as citable records. Order is the engine's ranking, preserved. */
+export declare function resultsToFacts(results: WebResult[], provider: string): Fact[];
+/** One fetched page as a citable record. The prose goes to `pageEvidence`. */
+export declare function pageToFact(page: Extract<PageOutcome, {
+    ok: true;
+}>): Fact;
+/**
+ * The prose a claim may draw on, one block per source, each labelled with the
+ * fact id that licenses citing it.
+ *
+ * `facts` must be the ID-ASSIGNED array (post `assignFactIds`) and must line up
+ * positionally with `results`. Passing un-assigned facts produces blocks labelled
+ * `[]`, which is the bug this note exists to prevent.
+ */
+export declare function resultsEvidence(facts: Fact[], results: WebResult[]): string[];
+/** A fetched page's readable text as one evidence block, labelled with its id. */
+export declare function pageEvidence(fact: Fact, page: Extract<PageOutcome, {
+    ok: true;
+}>): string;
+/**
+ * The sentence a model is shown when a lookup produced no usable answer.
+ *
+ * Written as an INSTRUCTION rather than a status code because that is what the
+ * model actually acts on, and because the two cases need opposite replies: a
+ * genuine "nothing out there" may be reported as a finding, while a backend
+ * outage may not be reported at all — the honest reply is that we could not
+ * look. Handing both to the model as `results: []` is how a broken API key
+ * becomes a confident statement about the state of the world.
+ */
+export declare function describeEmptySearch(outcome: WebSearchResult): string;

package/dist/web/facts.js

@@ -0,0 +1,104 @@
+/**
+ * Web results → grounded Facts.
+ *
+ * This is the join that makes web search safe to give an agent. On its own,
+ * search makes hallucination WORSE, not better: the model now has a pile of
+ * plausible-sounding text from strangers, mixed into a context window beside
+ * the user's real data, with nothing marking which sentence came from where.
+ * The failure mode is not "the model made something up" — it is "the model
+ * attributed a real sentence to the wrong source", and that is indistinguishable
+ * from competence until someone clicks the link.
+ *
+ * The fix is the one the grounding harness already implements for database
+ * rows. Every retrieved item becomes a `Fact` with a citation handle ([F3]) and
+ * a `url` field, `verifyAnswer` then checks mechanically that every handle in
+ * the answer exists and that proper nouns and numbers in the answer appear in
+ * the evidence. A claim that cites nothing, or cites [F9] when only F1–F4 were
+ * retrieved, is caught by a string check rather than by a reviewer's judgement.
+ *
+ * Two sets come out of here on purpose:
+ *
+ *   facts    — the citable records. Short, uniform, one block per source.
+ *   evidence — the actual prose (snippets, page text). The verifier needs the
+ *              words themselves to judge whether "€4.2 billion" in the answer
+ *              was read or invented; a fact's five short fields are not enough
+ *              of a corpus for that, and passing the page text AS a fact field
+ *              would bury the citable metadata in ten thousand characters.
+ */
+import { makeFact } from "../grounding/facts.js";
+/** Search hits as citable records. Order is the engine's ranking, preserved. */
+export function resultsToFacts(results, provider) {
+    return results.map((r) => makeFact({
+        kind: "web_result",
+        subject: r.title,
+        source: `web search (${provider})`,
+        values: {
+            title: r.title,
+            url: r.url,
+            published: r.published ?? null,
+            snippet: r.snippet || null,
+            engine: r.engine ?? provider,
+        },
+    }));
+}
+/** One fetched page as a citable record. The prose goes to `pageEvidence`. */
+export function pageToFact(page) {
+    return makeFact({
+        kind: "web_page",
+        subject: page.title || page.url,
+        source: "page read",
+        values: {
+            title: page.title || null,
+            url: page.url,
+            retrieved: new Date().toISOString().slice(0, 10),
+            // Stated rather than implied: a model told the text is partial will say
+            // "the first part of the page" instead of summarising a page it half read.
+            truncated: page.truncated ? "yes — only the first part of the page was read" : "no",
+        },
+    });
+}
+/**
+ * The prose a claim may draw on, one block per source, each labelled with the
+ * fact id that licenses citing it.
+ *
+ * `facts` must be the ID-ASSIGNED array (post `assignFactIds`) and must line up
+ * positionally with `results`. Passing un-assigned facts produces blocks labelled
+ * `[]`, which is the bug this note exists to prevent.
+ */
+export function resultsEvidence(facts, results) {
+    return results.map((r, i) => {
+        const id = facts[i]?.id ?? "";
+        const head = `${id ? `[${id}] ` : ""}${r.title} — ${r.url}`;
+        return r.snippet ? `${head}\n${r.snippet}` : head;
+    });
+}
+/** A fetched page's readable text as one evidence block, labelled with its id. */
+export function pageEvidence(fact, page) {
+    const head = `[${fact.id}] ${page.title || page.url} — ${page.url}`;
+    const tail = page.truncated ? "\n[…the rest of the page was not read]" : "";
+    return `${head}\n${page.text}${tail}`;
+}
+/**
+ * The sentence a model is shown when a lookup produced no usable answer.
+ *
+ * Written as an INSTRUCTION rather than a status code because that is what the
+ * model actually acts on, and because the two cases need opposite replies: a
+ * genuine "nothing out there" may be reported as a finding, while a backend
+ * outage may not be reported at all — the honest reply is that we could not
+ * look. Handing both to the model as `results: []` is how a broken API key
+ * becomes a confident statement about the state of the world.
+ */
+export function describeEmptySearch(outcome) {
+    if (outcome.status === "nothing") {
+        return (`The web search for "${outcome.query}" ran and returned no results. ` +
+            `You may tell the user nothing was found for that phrasing, and suggest a different one. ` +
+            `Do NOT present this as proof that the thing does not exist.`);
+    }
+    const reasons = outcome.attempts
+        .map((a) => `${a.provider}: ${a.failure?.reason ?? a.outcome}`)
+        .join(" | ");
+    return (`The web search for "${outcome.query}" COULD NOT BE PERFORMED (${reasons}). ` +
+        `Tell the user plainly that you could not search the web right now. ` +
+        `You must NOT say that nothing was found, that no such thing exists, or describe any web content — ` +
+        `you have not seen any.`);
+}

package/dist/web/index.d.ts

@@ -0,0 +1,38 @@
+/**
+ * ai-kit/web — the agent's eyes on the open web.
+ *
+ * Why this belongs in this package rather than in each app. Every AI product
+ * in the fleet is about to need the same four things, and each one is a
+ * distinct way to be wrong:
+ *
+ *   1. WHICH backend. Self-hosted metasearch, an independent index, an
+ *      agent-oriented API — with the same fallback-chain problem `chain.ts`
+ *      already solved for models, for the same reason: a single pinned
+ *      backend is a scheduled outage.
+ *   2. WHETHER it answered. "Found nothing" and "could not look" are different
+ *      answers, and an app that collapses them ships confident negatives it
+ *      never earned.
+ *   3. Fetching a page an AGENT chose, which is a genuinely different security
+ *      problem from fetching one a user typed — the model can be steered to a
+ *      URL by any page it reads, so the SSRF guard has to survive redirects and
+ *      DNS rebinding rather than checking a string once.
+ *   4. CITATIONS. Search without citation binding makes hallucination worse,
+ *      because now the invented sentence is surrounded by real ones.
+ *
+ * Four chances to get it wrong, times every app, is the duplication this
+ * package exists to end. The output type is deliberately `Fact` from
+ * `ai-kit/grounding` rather than a bespoke shape: retrieved web content and
+ * retrieved database rows then flow through ONE verifier, and "cite your
+ * sources" becomes a mechanical check instead of a line in a prompt.
+ *
+ * What it deliberately does not do: summarise, re-rank with an LLM, crawl,
+ * render JavaScript, or cache. The first two are the model's job and belong
+ * upstream where the app's own prompt lives; the last three are a different
+ * product with a different cost profile.
+ */
+export type { WebResult, WebFailure, SearchOutcome, PageOutcome, SearchProvider, SearchOptions, ReadOptions, WebEnv, } from "./types.js";
+export { webSearch, describeAttempts, type WebSearchResult, type SearchAttempt, type WebSearchDeps, } from "./search.js";
+export { searxngProvider, braveProvider, tavilyProvider, defaultProviders } from "./providers.js";
+export { readPage, extractReadableText, type ReadDeps, type ExtractedPage } from "./read.js";
+export { validateFetchTarget, isPrivateAddress, defaultLookup, type LookupFn, type UrlVerdict, } from "./ssrf.js";
+export { resultsToFacts, pageToFact, resultsEvidence, pageEvidence, describeEmptySearch, } from "./facts.js";

package/dist/web/index.js

@@ -0,0 +1,5 @@
+export { webSearch, describeAttempts, } from "./search.js";
+export { searxngProvider, braveProvider, tavilyProvider, defaultProviders } from "./providers.js";
+export { readPage, extractReadableText } from "./read.js";
+export { validateFetchTarget, isPrivateAddress, defaultLookup, } from "./ssrf.js";
+export { resultsToFacts, pageToFact, resultsEvidence, pageEvidence, describeEmptySearch, } from "./facts.js";

package/dist/web/providers.d.ts

@@ -0,0 +1,45 @@
+/**
+ * Search backends.
+ *
+ * Three, on purpose, and in this order of preference:
+ *
+ *   searxng — self-hosted, no API key, no per-call cost, no third party told
+ *             what your users are looking for. A metasearch front end over the
+ *             engines you enable. The catch is that it is scraping on your
+ *             behalf from your own IP, so a datacenter host will sometimes be
+ *             refused by upstream engines; that failure is loud here, which is
+ *             why the chain exists.
+ *   brave   — an independent index (not a Bing or Google reseller) with a free
+ *             tier, so the fallback is not the same index the primary was
+ *             already asking.
+ *   tavily  — built for agents: it returns cleaned content rather than SERP
+ *             chrome. Last resort because it is the most expensive per call.
+ *
+ * The seam is one method wide deliberately. Every provider here also sells
+ * "AI answers", news verticals and summarisation; adopting any of those would
+ * make the backends non-interchangeable, and an interchangeable backend is the
+ * entire reason to have a seam. Summarising is the model's job and it happens
+ * upstream of this file.
+ *
+ * None of them throws. A provider that cannot answer returns a typed failure,
+ * because the chain has to tell "no results" apart from "no answer" to pick
+ * its next move — and so does the model reading the result.
+ */
+import type { SearchProvider, WebEnv } from "./types.js";
+declare const DEFAULT_LIMIT = 8;
+declare const DEFAULT_TIMEOUT_MS = 8000;
+declare const MAX_LIMIT = 20;
+type Fetch = typeof globalThis.fetch;
+export declare function searxngProvider(env: WebEnv, doFetch?: Fetch): SearchProvider;
+export declare function braveProvider(env: WebEnv, doFetch?: Fetch): SearchProvider;
+export declare function tavilyProvider(env: WebEnv, doFetch?: Fetch): SearchProvider;
+/**
+ * The default chain, in preference order, filtered to what the env configures.
+ *
+ * A caller that wants a different order passes its own array — this function
+ * exists so that the common case ("use whatever we have") is one call and not
+ * a policy decision re-made in every app, which is how the fleet ended up with
+ * one model-fallback implementation per repo.
+ */
+export declare function defaultProviders(env: WebEnv, doFetch?: Fetch): SearchProvider[];
+export { DEFAULT_LIMIT, DEFAULT_TIMEOUT_MS, MAX_LIMIT };

package/dist/web/providers.js

@@ -0,0 +1,246 @@
+const DEFAULT_LIMIT = 8;
+const DEFAULT_TIMEOUT_MS = 8_000;
+const MAX_LIMIT = 20;
+function clampLimit(limit) {
+    if (!limit || !Number.isFinite(limit))
+        return DEFAULT_LIMIT;
+    return Math.max(1, Math.min(MAX_LIMIT, Math.floor(limit)));
+}
+/** `site:` is spelled the same by every engine here, so it is applied once. */
+function applySite(query, site) {
+    const trimmed = site?.trim();
+    if (!trimmed)
+        return query;
+    return `${query} site:${trimmed.replace(/^https?:\/\//, "").replace(/\/.*$/, "")}`;
+}
+function transportFailure(err) {
+    const name = err instanceof Error ? err.name : "";
+    const message = err instanceof Error ? err.message : String(err);
+    if (name === "TimeoutError" || name === "AbortError") {
+        return { kind: "timeout", reason: "The search backend did not answer in time." };
+    }
+    return { kind: "unreachable", reason: `The search backend could not be reached (${message}).` };
+}
+/** HTTP status → failure kind. Shared because every provider gets this wrong the same way. */
+function statusFailure(status, provider) {
+    if (status === 429) {
+        return { kind: "rate_limited", reason: `${provider} is rate-limiting this key right now.` };
+    }
+    if (status === 401 || status === 403) {
+        return { kind: "auth", reason: `${provider} rejected the credentials (${status}).` };
+    }
+    return { kind: "bad_response", reason: `${provider} answered ${status}.` };
+}
+function trimResult(raw) {
+    const url = typeof raw.url === "string" ? raw.url.trim() : "";
+    if (!url || !/^https?:\/\//i.test(url)) {
+        return null;
+    }
+    const title = typeof raw.title === "string" && raw.title.trim() ? raw.title.trim() : url;
+    const snippet = typeof raw.snippet === "string" ? raw.snippet.trim().slice(0, 600) : "";
+    const published = typeof raw.published === "string" && raw.published.trim() ? raw.published.trim() : undefined;
+    const engine = typeof raw.engine === "string" && raw.engine.trim() ? raw.engine.trim() : undefined;
+    return {
+        title: title.slice(0, 300),
+        url,
+        snippet,
+        ...(published ? { published } : {}),
+        ...(engine ? { engine } : {}),
+    };
+}
+// ── SearXNG ────────────────────────────────────────────────────────────────
+// Needs `formats: [html, json]` in the instance's settings.yml — a stock
+// instance serves HTML only and answers a JSON request with 403, which is
+// reported here as an auth failure so the operator sees the real fix rather
+// than "no results".
+export function searxngProvider(env, doFetch = globalThis.fetch) {
+    const base = env.SEARXNG_URL?.trim().replace(/\/+$/, "");
+    return {
+        name: "searxng",
+        configured: () => Boolean(base),
+        async search(query, opts) {
+            if (!base) {
+                return {
+                    ok: false,
+                    provider: "searxng",
+                    query,
+                    failure: { kind: "not_configured", reason: "SEARXNG_URL is not set." },
+                };
+            }
+            const url = new URL(`${base}/search`);
+            url.searchParams.set("q", applySite(query, opts.site));
+            url.searchParams.set("format", "json");
+            url.searchParams.set("safesearch", "0");
+            if (opts.lang) {
+                url.searchParams.set("language", opts.lang);
+            }
+            try {
+                const res = await doFetch(url.toString(), {
+                    headers: {
+                        Accept: "application/json",
+                        ...(env.SEARXNG_TOKEN ? { Authorization: `Bearer ${env.SEARXNG_TOKEN}` } : {}),
+                    },
+                    signal: AbortSignal.timeout(opts.timeoutMs ?? DEFAULT_TIMEOUT_MS),
+                });
+                if (!res.ok) {
+                    return {
+                        ok: false,
+                        provider: "searxng",
+                        query,
+                        failure: statusFailure(res.status, "SearXNG"),
+                    };
+                }
+                const body = (await res.json());
+                const rows = Array.isArray(body.results) ? body.results : [];
+                const results = rows
+                    .map((r) => {
+                    const row = r;
+                    return trimResult({
+                        title: row.title,
+                        url: row.url,
+                        snippet: row.content,
+                        published: row.publishedDate,
+                        engine: row.engine,
+                    });
+                })
+                    .filter((r) => r !== null)
+                    .slice(0, clampLimit(opts.limit));
+                return { ok: true, provider: "searxng", query, results };
+            }
+            catch (err) {
+                return { ok: false, provider: "searxng", query, failure: transportFailure(err) };
+            }
+        },
+    };
+}
+// ── Brave ──────────────────────────────────────────────────────────────────
+export function braveProvider(env, doFetch = globalThis.fetch) {
+    const key = env.BRAVE_SEARCH_API_KEY?.trim();
+    return {
+        name: "brave",
+        configured: () => Boolean(key),
+        async search(query, opts) {
+            if (!key) {
+                return {
+                    ok: false,
+                    provider: "brave",
+                    query,
+                    failure: { kind: "not_configured", reason: "BRAVE_SEARCH_API_KEY is not set." },
+                };
+            }
+            const url = new URL("https://api.search.brave.com/res/v1/web/search");
+            url.searchParams.set("q", applySite(query, opts.site));
+            url.searchParams.set("count", String(clampLimit(opts.limit)));
+            if (opts.lang) {
+                url.searchParams.set("search_lang", opts.lang.split("-")[0] ?? opts.lang);
+            }
+            try {
+                const res = await doFetch(url.toString(), {
+                    headers: { Accept: "application/json", "X-Subscription-Token": key },
+                    signal: AbortSignal.timeout(opts.timeoutMs ?? DEFAULT_TIMEOUT_MS),
+                });
+                if (!res.ok) {
+                    return {
+                        ok: false,
+                        provider: "brave",
+                        query,
+                        failure: statusFailure(res.status, "Brave Search"),
+                    };
+                }
+                const body = (await res.json());
+                const rows = Array.isArray(body.web?.results) ? body.web.results : [];
+                const results = rows
+                    .map((r) => {
+                    const row = r;
+                    return trimResult({
+                        title: row.title,
+                        url: row.url,
+                        snippet: row.description,
+                        published: row.page_age,
+                        engine: "brave",
+                    });
+                })
+                    .filter((r) => r !== null);
+                return { ok: true, provider: "brave", query, results };
+            }
+            catch (err) {
+                return { ok: false, provider: "brave", query, failure: transportFailure(err) };
+            }
+        },
+    };
+}
+// ── Tavily ─────────────────────────────────────────────────────────────────
+export function tavilyProvider(env, doFetch = globalThis.fetch) {
+    const key = env.TAVILY_API_KEY?.trim();
+    return {
+        name: "tavily",
+        configured: () => Boolean(key),
+        async search(query, opts) {
+            if (!key) {
+                return {
+                    ok: false,
+                    provider: "tavily",
+                    query,
+                    failure: { kind: "not_configured", reason: "TAVILY_API_KEY is not set." },
+                };
+            }
+            try {
+                const res = await doFetch("https://api.tavily.com/search", {
+                    method: "POST",
+                    headers: {
+                        "Content-Type": "application/json",
+                        Authorization: `Bearer ${key}`,
+                    },
+                    body: JSON.stringify({
+                        query: applySite(query, opts.site),
+                        max_results: clampLimit(opts.limit),
+                        search_depth: "basic",
+                    }),
+                    signal: AbortSignal.timeout(opts.timeoutMs ?? DEFAULT_TIMEOUT_MS),
+                });
+                if (!res.ok) {
+                    return {
+                        ok: false,
+                        provider: "tavily",
+                        query,
+                        failure: statusFailure(res.status, "Tavily"),
+                    };
+                }
+                const body = (await res.json());
+                const rows = Array.isArray(body.results) ? body.results : [];
+                const results = rows
+                    .map((r) => {
+                    const row = r;
+                    return trimResult({
+                        title: row.title,
+                        url: row.url,
+                        snippet: row.content,
+                        published: row.published_date,
+                        engine: "tavily",
+                    });
+                })
+                    .filter((r) => r !== null);
+                return { ok: true, provider: "tavily", query, results };
+            }
+            catch (err) {
+                return { ok: false, provider: "tavily", query, failure: transportFailure(err) };
+            }
+        },
+    };
+}
+/**
+ * The default chain, in preference order, filtered to what the env configures.
+ *
+ * A caller that wants a different order passes its own array — this function
+ * exists so that the common case ("use whatever we have") is one call and not
+ * a policy decision re-made in every app, which is how the fleet ended up with
+ * one model-fallback implementation per repo.
+ */
+export function defaultProviders(env, doFetch = globalThis.fetch) {
+    return [
+        searxngProvider(env, doFetch),
+        braveProvider(env, doFetch),
+        tavilyProvider(env, doFetch),
+    ].filter((p) => p.configured());
+}
+export { DEFAULT_LIMIT, DEFAULT_TIMEOUT_MS, MAX_LIMIT };

package/dist/web/read.d.ts

@@ -0,0 +1,48 @@
+/**
+ * readPage — fetch one web page and return its readable text.
+ *
+ * This is the other half of search. A result list gives an agent titles and a
+ * sentence of snippet; almost every real question needs the page. Without a
+ * reader the agent either answers from the snippet (which is how a plausible
+ * wrong number gets into a reply) or says it cannot know, and both are worse
+ * than reading.
+ *
+ * Three things it refuses to do, each because the alternative failed somewhere:
+ *
+ * - It never follows a redirect automatically. Every hop goes back through the
+ *   SSRF guard, because a public URL that 302s to localhost is the cheapest
+ *   bypass there is.
+ * - It never returns partial bytes as though they were the page. When the cap
+ *   is hit the result says `truncated: true`, so a model summarising it can say
+ *   "the first part of the page says…" instead of implying it read the whole.
+ * - It never throws. The caller is an agent loop; an unreadable page is a fact
+ *   about the world to reason about, not an exception to unwind through.
+ *
+ * Extraction is deliberately regex-based and dependency-free. A DOM parser
+ * would read more sites (SPAs especially) at the cost of making this package
+ * depend on one, and the failure it would fix — a JS-rendered page yielding
+ * nothing — is honestly reportable: empty text with `ok: true` tells the model
+ * the page had nothing readable, which is true and useful.
+ */
+import type { PageOutcome, ReadOptions } from "./types.js";
+import { type LookupFn } from "./ssrf.js";
+export type ReadDeps = {
+    fetch?: typeof globalThis.fetch;
+    lookup?: LookupFn;
+};
+export declare function readPage(rawUrl: string, opts?: ReadOptions, deps?: ReadDeps): Promise<PageOutcome>;
+export type ExtractedPage = {
+    title: string;
+    text: string;
+    truncated: boolean;
+};
+/**
+ * Strip a document to the text a reader would see.
+ *
+ * Order matters and is load-bearing: comments go first (they can contain
+ * unbalanced tags that desync a later pass), then whole non-content elements
+ * WITH their contents, then remaining tags, then entities. Doing entities
+ * before tag-stripping would turn an encoded `&lt;script&gt;` in page text into
+ * a real tag mid-pass.
+ */
+export declare function extractReadableText(html: string, maxChars?: number): ExtractedPage;