@dbx-tools/appkit-web-search 0.3.28 → 0.3.30

package/README.md

@@ -11,11 +11,12 @@ through [`got-scraping`](https://www.npmjs.com/package/got-scraping) (browser-li
 TLS + header fingerprints so a fetch survives common bot walls), which Databricks
 has no equivalent for.
 
-Key features:
+**Key features:**
 
 - AppKit plugin registration that resolves and logs the effective policy at boot.
 - Two Mastra tools: `web_search` (answer + citations) and `web_fetch` (page
-  contents as readable text or raw HTML).
+  contents as readable text or raw HTML), plus the same pair as AppKit agent
+  tools through the plugin's `ToolProvider`.
 - `web_search` resolves its OWN web-search-capable model - defaulting to Gemini,
   then GPT - independently of the calling agent's chat model (which may not
   support web search). Loose names (`"gemini"`, `"gpt"`) fuzzy-match the live
@@ -23,11 +24,15 @@ Key features:
 - A built-in provider -> tool-spec map (OpenAI Responses API
   `{"type":"web_search"}`, Gemini Chat Completions `{"google_search":{}}`),
   overridable per provider via the `WEB_SEARCH_TOOLS` setting.
-- An optional URL allow-list (globs or bare hosts) built on
-  [`@dbx-tools/path`](../path)'s `match` matcher: `web_search` silently filters
-  disallowed citations, `web_fetch` refuses a disallowed URL.
+- A named URL policy (`allowlist` or `unrestricted`) over a glob allow-list
+  built on [`@dbx-tools/path`](../path)'s `match` matcher: `web_search` silently
+  filters disallowed citations, `web_fetch` refuses a disallowed URL, and the
+  mode in force is logged at boot.
 - Per-tool approval gating: gate every call, or (for `web_fetch`) only calls
   whose URL matches a pattern, mapped onto Mastra's `requireApproval`.
+- Every outbound call runs through AppKit's `execute()` chain - per-user cache,
+  retry with jittered backoff, timeout, telemetry - and unwinds on an
+  `AbortSignal`.
 
 ## Why Use This Over Native AppKit
 
@@ -68,12 +73,28 @@ await createApp({
 });
 ```
 
-`plugin.webSearch()` resolves config (over env), compiles the allow-list, and
+`plugin.webSearch()` resolves config (over env), compiles the URL policy, and
 primes the shared runtime the tools reuse. `tool.webSearchTool()` /
 `tool.webFetchTool()` build the two Mastra tools. Approval, when enabled,
 requires Mastra storage, so register `lakebase()` or configure storage in the
 Mastra plugin.
 
+The plugin is also an AppKit `ToolProvider`, so an AppKit agent can take the
+same two tools without Mastra in the picture:
+
+```ts
+import { createAgent } from "@databricks/appkit/beta";
+
+const researcher = createAgent({
+  instructions: "Research questions with web_search, then read sources with web_fetch.",
+  tools: (plugins) => ({ ...plugins["web-search"].toolkit() }),
+});
+```
+
+Both tools are annotated `effect: "read"` and require user context, and neither
+is auto-inheritable: an agent has to ask for them, because `web_fetch` reaches
+whatever URL the model produces.
+
 ## Choose The Web-Search Model
 
 The native web-search tool only runs on certain models, and it is provider-
@@ -89,6 +110,24 @@ agent's chat model, so an agent on any model can still search:
   Llama endpoint), the tool errors rather than silently searching with the wrong
   thing. When nothing is pinned, unsupported fallbacks are skipped.
 
+The endpoint id is read in this order, and where it came from changes what
+happens when it cannot run web search:
+
+| Order | Source                             | Unsupported endpoint       |
+| ----- | ---------------------------------- | -------------------------- |
+| 1     | `model` in plugin config           | Error                      |
+| 2     | `WEB_SEARCH_MODEL`                 | Error                      |
+| 3     | `DATABRICKS_SERVING_ENDPOINT_NAME` | Skipped, fallbacks apply   |
+| 4     | `modelFallbacks`                   | Skipped, next one is tried |
+
+`DATABRICKS_SERVING_ENDPOINT_NAME` is AppKit's standard name for a Model
+Serving binding, and it is the field the plugin declares in its manifest, so
+`app.yaml` / bundle wiring reaches this plugin the way it reaches any other.
+Because that binding is usually the app's chat endpoint - which need not
+support web search - it is treated as a preference rather than a pin.
+`WEB_SEARCH_MODEL` stays as the dedicated override for pointing web search at
+a different endpoint than the rest of the app.
+
 Model resolution runs against the LIVE workspace catalogue (via
 [`@dbx-tools/model`](../model)), so it only ever picks an endpoint that is
 actually deployed - a configured id that doesn't exist is never called.
@@ -135,7 +174,7 @@ const result = await search.runWebSearch(
   runtime.getWebSearchRuntime().config,
   { client: ctx.client, host },
 );
-console.log(result.answer, result.citations, result.model);
+// result.answer, result.citations, result.model
 
 const page = await fetch.runWebFetch(
   { url: result.citations[0]!.url, format: "text" },
@@ -156,7 +195,7 @@ const list = allowlist.toUrlAllowList(
 );
 
 list.allows("https://docs.databricks.com/aws/en/index.html"); // true
-list.allows("https://evil.example.com/");                     // false
+list.allows("https://evil.example.com/"); // false
 ```
 
 Each entry is a glob compiled by [`@dbx-tools/path`](../path)'s `match` matcher.
@@ -165,15 +204,36 @@ URL's hostname, and a bare host also matches its subdomains; a path entry
 (`docs.example.com/api/**`) matches host + pathname. Enforcement is asymmetric by
 design: `web_search` citations are silently filtered to the permitted set (the
 model never surfaces a source it then can't fetch), while an explicit `web_fetch`
-of a disallowed URL is refused with an error (a visible, correctable mistake). An
-empty / absent allow-list permits everything.
+of a disallowed URL is refused with an error (a visible, correctable mistake).
+
+Which of the two modes is in force is named by `urlPolicy`, and the effective
+mode is in the boot log:
+
+| `urlPolicy`      | Effect                                            |
+| ---------------- | ------------------------------------------------- |
+| `"allowlist"`    | Only `allowedUrls` entries are reachable          |
+| `"unrestricted"` | Any URL is reachable; the open mode, stated aloud |
+
+Leave `urlPolicy` unset and it follows `allowedUrls`: `"allowlist"` when there
+are entries, `"unrestricted"` when there are none. Set it to say which one you
+meant. The two contradictions fail at startup rather than quietly picking a
+side: `"allowlist"` with no entries, and `"unrestricted"` with entries.
+
+```ts
+// An open deployment, said out loud:
+plugin.webSearch({ urlPolicy: "unrestricted" });
+
+// A closed one:
+plugin.webSearch({ urlPolicy: "allowlist", allowedUrls: ["*.databricks.com"] });
+```
 
 ## Gate Calls For Approval
 
 Both tools run without approval by default. Pass `approval` to a tool (or set it
 plugin-wide) to require a human click. `true` gates every call; a URL pattern (or
 list) gates only calls whose URL matches - a `web_fetch` is evaluated precisely
-against its target URL.
+against its target URL, while a `web_search` (whose result URLs are unknown
+before the call) treats a pattern gate as "gate every call".
 
 ```ts
 import { tool } from "@dbx-tools/appkit-web-search";
@@ -184,32 +244,62 @@ tool.webFetchTool({ approval: true });
 // Only fetches of an internal domain require approval:
 tool.webFetchTool({ approval: "*.internal.example.com" });
 
+// The same thing, with the mode named:
+tool.webFetchTool({ approval: { mode: "urls", patterns: ["*.internal.example.com"] } });
+
 // Plugin-wide default (tools inherit unless they set their own):
 plugin.webSearch({ approval: ["*.internal.example.com", "*.corp.example.com"] });
 ```
 
-Approval uses the same glob syntax as the allow-list, so the two policies read
-identically.
+Every spelling normalizes to one of three modes - `{ mode: "none" }`,
+`{ mode: "always" }`, `{ mode: "urls", patterns }` - which is what the resolved
+config carries and the boot log reports. Approval uses the same glob syntax as
+the allow-list, so the two policies read identically.
 
 ## Configuration
 
-Resolution order is explicit config first, then env vars, then a built-in
-default:
-
-- `WEB_SEARCH_MODEL` (default web-search model; else the Gemini->GPT fallbacks);
-- `WEB_SEARCH_MODEL_FALLBACKS` (comma/space-separated candidate ids);
-- `WEB_SEARCH_TOOLS` (JSON provider -> tool-spec override map);
-- `WEB_SEARCH_FUZZY` / `WEB_SEARCH_FUZZY_THRESHOLD` (loose-name matching; default on / 0.4);
-- `WEB_SEARCH_MAX_CITATIONS` (default 10);
-- `WEB_SEARCH_FETCH_MAX_LENGTH` (default 50000 characters);
-- `WEB_SEARCH_TIMEOUT_MS` (default 30000);
-- `WEB_SEARCH_SCRAPE_FALLBACK` (fall back to a DuckDuckGo scrape when no native
-  web-search model is deployed; default on);
-- `WEB_SEARCH_ALLOWED_URLS` (comma/space-separated globs; empty = unrestricted).
+Resolution order is explicit config first, then the environment variable, then a
+built-in default.
+
+| Option                | Type                                              | Default                 | Description                                                                 |
+| --------------------- | ------------------------------------------------- | ----------------------- | --------------------------------------------------------------------------- |
+| `model`               | `string`                                          | resolved from fallbacks | Web-search endpoint: an endpoint name, a loose name, or a capability class. |
+| `modelFallbacks`      | `string \| string[]`                              | Gemini, then GPT        | Ordered candidates tried when `model` is unset.                             |
+| `webSearchTools`      | `Record<string, unknown>`                         | built-in provider map   | Provider -> tool-spec overrides, merged over the defaults.                  |
+| `modelFuzzyMatch`     | `boolean`                                         | `true`                  | Fuzzy-match loose model names against the live catalogue.                   |
+| `modelFuzzyThreshold` | `number`                                          | `0.4`                   | Fuse.js score below which a fuzzy match is accepted.                        |
+| `maxCitations`        | `number`                                          | `10`                    | Hard cap on citations returned from one search.                             |
+| `fetchMaxLength`      | `number`                                          | `50000`                 | Hard cap on `web_fetch` content length, in characters.                      |
+| `timeoutMs`           | `number`                                          | `30000`                 | Per-request network timeout.                                                |
+| `scrapeFallback`      | `boolean`                                         | `true`                  | Scrape DuckDuckGo when no web-search-capable model is deployed.             |
+| `urlPolicy`           | `"unrestricted" \| "allowlist"`                   | follows `allowedUrls`   | Which URLs the tools may reach.                                             |
+| `allowedUrls`         | `string \| string[]`                              | none                    | Allow-list globs or bare hosts.                                             |
+| `approval`            | `boolean \| string \| string[] \| ApprovalPolicy` | `{ mode: "none" }`      | Default approval gate for both tools.                                       |
+
+| Environment variable               | Sets                                                              |
+| ---------------------------------- | ----------------------------------------------------------------- |
+| `WEB_SEARCH_MODEL`                 | `model` (dedicated override, wins over the binding below)         |
+| `DATABRICKS_SERVING_ENDPOINT_NAME` | `model` (AppKit's Model Serving binding, treated as a preference) |
+| `WEB_SEARCH_MODEL_FALLBACKS`       | `modelFallbacks` (comma/space-separated)                          |
+| `WEB_SEARCH_TOOLS`                 | `webSearchTools` (JSON object; anything else fails at boot)       |
+| `WEB_SEARCH_FUZZY`                 | `modelFuzzyMatch`                                                 |
+| `WEB_SEARCH_FUZZY_THRESHOLD`       | `modelFuzzyThreshold`                                             |
+| `WEB_SEARCH_MAX_CITATIONS`         | `maxCitations`                                                    |
+| `WEB_SEARCH_FETCH_MAX_LENGTH`      | `fetchMaxLength`                                                  |
+| `WEB_SEARCH_TIMEOUT_MS`            | `timeoutMs`                                                       |
+| `WEB_SEARCH_SCRAPE_FALLBACK`       | `scrapeFallback`                                                  |
+| `WEB_SEARCH_URL_POLICY`            | `urlPolicy`                                                       |
+| `WEB_SEARCH_ALLOWED_URLS`          | `allowedUrls` (comma/space-separated)                             |
 
 The plugin's `maxCitations` / `fetchMaxLength` are hard caps: a per-call request
 may narrow them but never exceed them.
 
+Every outbound call - the serving POST, a page fetch, the scrape fallback -
+runs through the plugin's `execute()` chain, so all three are cached, retried
+with jittered backoff, timed out, and traced. The cache is namespaced per user,
+so an on-behalf-of result never crosses identities. See `defaults.ts` for the
+TTLs and retry budgets and why each was chosen.
+
 Requirements: the native web-search tool runs on pay-per-token GPT / Gemini
 serving endpoints with cross-region processing enabled; it is unavailable on
 provisioned throughput, for external models, or under HIPAA/BAA compliance. See
@@ -217,15 +307,23 @@ the [Databricks docs](https://docs.databricks.com/aws/en/machine-learning/model-
 
 ## Modules
 
-- `plugin` - `WebSearchPlugin`, `webSearch()` AppKit plugin factory.
+- `plugin` - `WebSearchPlugin`, `webSearch()` AppKit plugin factory, and the
+  AppKit `ToolProvider` implementation.
 - `tool` - `webSearchTool()` / `webFetchTool()` Mastra tools.
+- `defaults` - the cache / retry / timeout settings every outbound call runs
+  with, and the reasoning behind each.
 - `search` - `runWebSearch()` over the Databricks native web-search tool.
 - `provider` - provider detection + the provider -> tool-spec map.
 - `scrape` - `runScrapeSearch()` DuckDuckGo fallback for workspaces with no
   native web-search model.
-- `fetch` - `runWebFetch()` over got-scraping, plus `htmlToText()`.
-- `runtime` - shared runtime, `getWebSearchRuntime()`, `resetWebSearchRuntime()`.
-- `config` - config types, JSON schema, `resolveWebSearchConfig()`, approval helpers.
+- `fetch` - `runWebFetch()` over got-scraping.
+- `html-text` - `htmlToText()` / `htmlFragmentToText()` / `decodeHtmlEntities()`, shared by
+  `fetch` and the DuckDuckGo scrape fallback.
+- `runtime` - shared runtime and the executor outbound calls run through:
+  `getWebSearchRuntime()`, `setWebSearchExecutor()`, `executeRead()`,
+  `resetWebSearchRuntime()`.
+- `config` - config types, JSON schema, `resolveWebSearchConfig()`, URL policy
+  and approval helpers.
 - `allowlist` - URL allow-list parsing/compiling on top of `@dbx-tools/path`.
 - `schema` - zod tool contracts and inferred types.
 

package/index.ts

@@ -4,7 +4,9 @@
 
 export * as allowlist from "./src/allowlist";
 export * as config from "./src/config";
+export * as defaults from "./src/defaults";
 export * as fetch from "./src/fetch";
+export * as htmlText from "./src/html-text";
 export * as plugin from "./src/plugin";
 export * as provider from "./src/provider";
 export * as runtime from "./src/runtime";
@@ -13,9 +15,10 @@ export * as scrape from "./src/scrape";
 export * as search from "./src/search";
 export * as tool from "./src/tool";
 export type { UrlAllowList } from "./src/allowlist";
-export type { ApprovalGate, WebSearchPluginConfig, ResolvedWebSearchConfig } from "./src/config";
+export type { ApprovalGate, ApprovalPolicy, UrlPolicyMode, ModelSource, WebSearchPluginConfig, ResolvedWebSearchConfig } from "./src/config";
+export type { WebSearchExecuteConfig, WebSearchExecutionSettings } from "./src/defaults";
 export type { WebSearchProvider, WebSearchProviderSpec } from "./src/provider";
-export type { WebSearchRuntime } from "./src/runtime";
+export type { WebSearchExecutor, WebSearchRuntime } from "./src/runtime";
 export type { WebSearchRequest, WebSearchCitation, WebSearchResult, WebFetchRequest, WebFetchResult } from "./src/schema";
 export type { WebSearchContext } from "./src/search";
 export type { WebSearchToolOptions } from "./src/tool";

package/package.json

@@ -16,18 +16,18 @@
     "@databricks/appkit": "^0.43.0",
     "@mastra/core": "^1.47.0",
     "got-scraping": "^4.2.1",
-    "zod": "^4.3.6",
-    "@dbx-tools/path": "0.3.28",
-    "@dbx-tools/model": "0.3.28",
-    "@dbx-tools/shared-core": "0.3.28",
-    "@dbx-tools/shared-model": "0.3.28"
+    "zod": "4.3.6",
+    "@dbx-tools/model": "0.3.30",
+    "@dbx-tools/path": "0.3.30",
+    "@dbx-tools/shared-core": "0.3.30",
+    "@dbx-tools/shared-model": "0.3.30"
   },
   "main": "index.ts",
   "license": "UNLICENSED",
   "publishConfig": {
     "access": "public"
   },
-  "version": "0.3.28",
+  "version": "0.3.30",
   "types": "index.ts",
   "type": "module",
   "exports": {

package/src/allowlist.ts

@@ -29,8 +29,9 @@
  * @module
  */
 
+import { ValidationError } from "@databricks/appkit";
 import { match, type PathMatcher } from "@dbx-tools/path";
-import { object } from "@dbx-tools/shared-core";
+import { string } from "@dbx-tools/shared-core";
 
 /** A compiled URL allow-list. Build one with {@link toUrlAllowList}. */
 export interface UrlAllowList {
@@ -64,15 +65,7 @@ export function normalizeUrlPattern(pattern: string): string {
  * identically.
  */
 export function parseAllowedUrls(raw: string | string[] | undefined): string[] {
-  const entries = typeof raw === "string" ? raw.split(/[\s,]+/) : Array.isArray(raw) ? raw : [];
-  return [
-    ...object
-      .sequence(entries)
-      .map((entry) => normalizeUrlPattern(entry))
-      .filter((entry) => entry.length > 0)
-      .distinct()
-      .toArray(),
-  ];
+  return string.parseList(raw, normalizeUrlPattern);
 }
 
 /** One compiled entry: which URL slice it tests, and the matcher for it. */
@@ -136,8 +129,10 @@ export function toUrlAllowList(patterns: readonly string[]): UrlAllowList {
  */
 export function assertUrlAllowed(url: string, allow: UrlAllowList): void {
   if (!allow.allows(url)) {
-    throw new Error(
-      `web-search: URL "${url}" is not permitted by the configured allow-list (${allow.patterns.join(", ")})`,
+    throw ValidationError.invalidValue(
+      "url",
+      url,
+      `a URL permitted by the configured allow-list (${allow.patterns.join(", ")})`,
     );
   }
 }