@cyanheads/brapi-mcp-server 0.4.14 → 0.5.1

package/CLAUDE.md

@@ -1,7 +1,7 @@
 # Agent Protocol
 
 **Server:** brapi-mcp-server
-**Version:** 0.4.13
+**Version:** 0.5.1
 **Framework:** [@cyanheads/mcp-ts-core](https://www.npmjs.com/package/@cyanheads/mcp-ts-core)
 
 > **Read the framework docs first:** `node_modules/@cyanheads/mcp-ts-core/CLAUDE.md` contains the full API reference — builders, Context, error codes, exports, patterns. This file covers server-specific conventions only.
@@ -79,15 +79,15 @@ export const brapiConnect = tool('brapi_connect', {
 });
 ```
 
-### Tool — find with dataset spillover
+### Tool — find with dataframe spillover
 
-`find_*` tools share a pattern: pull one page capped at `loadLimit`, compute distributions across the returned rows, and if the upstream total exceeds `loadLimit` spill the full union into `DatasetStore` and return a handle.
+`find_*` tools share a pattern: pull one page capped at `loadLimit`, compute distributions across the returned rows, and if the upstream total exceeds `loadLimit` materialize the full union as a canvas dataframe and return a handle. Spilled rows live in DuckDB only — there is no parallel JSON store. Canvas is mandatory: startup fails closed when `core.canvas` is undefined.
 
 ```ts
 // src/mcp-server/tools/definitions/brapi-find-germplasm.tool.ts (abbreviated)
 export const brapiFindGermplasm = tool('brapi_find_germplasm', {
   description:
-    'Find germplasm by name, synonym, accession, PUI, crop, or free-text. Returns a dataset handle when the upstream total exceeds loadLimit.',
+    'Find germplasm by name, synonym, accession, PUI, crop, or free-text. Spills to a canvas dataframe when the upstream total exceeds loadLimit — query with brapi_dataframe_query (SQL).',
   annotations: { readOnlyHint: true, openWorldHint: true },
   input: z.object({
     alias: AliasInput,
@@ -101,15 +101,16 @@ export const brapiFindGermplasm = tool('brapi_find_germplasm', {
   async handler(input, ctx) {
     const connection = await getServerRegistry().get(ctx, input.alias ?? DEFAULT_ALIAS);
     await getCapabilityRegistry().ensure(connection.baseUrl, { service: 'germplasm', method: 'GET' }, ctx);
+    const bridge = getCanvasBridge();
 
     const filters = mergeFilters(/* named + extraFilters */, warnings);
     const firstPage = await loadInitialPage(client, connection, '/germplasm', filters, loadLimit, ctx);
 
-    if (firstPage.hasMore && firstPage.totalCount > loadLimit) {
-      const spill = await spillToDataset({ /* persists union into DatasetStore */ });
-      // ... attach dataset handle to result
-    }
-    return { /* results + distributions + refinementHint + dataset? */ };
+    const { fullRows, dataframe } = await maybeSpill({
+      firstPage, client, connection, bridge,
+      path: '/germplasm', filters, source: 'find_germplasm', loadLimit, ctx,
+    });
+    return { /* results + distributions + refinementHint + dataframe? */ };
   },
   format: (result) => [{ type: 'text', text: renderFindResult(result) }],
 });
@@ -124,7 +125,7 @@ import { parseEnvConfig } from '@cyanheads/mcp-ts-core/config';
 
 const ServerConfigSchema = z.object({
   defaultBaseUrl: z.string().url().optional(),
-  loadLimit: z.coerce.number().int().positive().default(200),
+  loadLimit: z.coerce.number().int().positive().default(1_000),
   maxConcurrentRequests: z.coerce.number().int().positive().default(4),
   retryMaxAttempts: z.coerce.number().int().min(0).default(3),
   datasetTtlSeconds: z.coerce.number().int().positive().default(86_400),
@@ -159,7 +160,7 @@ Handlers receive a unified `ctx` object. Currently used surface:
 | Property | Description |
 |:---------|:------------|
 | `ctx.log` | Request-scoped logger — `.debug()`, `.info()`, `.notice()`, `.warning()`, `.error()`. Auto-correlates requestId, traceId, tenantId. |
-| `ctx.state` | Tenant-scoped KV — used by `ServerRegistry` (connection aliases), `DatasetStore` (spilled `find_*` results), and `CapabilityRegistry` (cached profiles). |
+| `ctx.state` | Tenant-scoped KV — used by `ServerRegistry` (connection aliases), `CanvasBridge` (per-tenant default canvasId + per-table provenance), and `CapabilityRegistry` (cached profiles). Spilled `find_*` rows live on the canvas (DuckDB), not in `ctx.state`. |
 | `ctx.signal` | `AbortSignal` — threaded into every BrAPI HTTP call so client-side cancellation aborts the upstream request. |
 | `ctx.requestId` | Unique request ID — auto-attached to every `ctx.log` entry. |
 | `ctx.tenantId` | Tenant ID from JWT or `'default'` for stdio — scopes all `ctx.state` reads/writes. |
@@ -172,7 +173,7 @@ Handlers receive a unified `ctx` object. Currently used surface:
 
 Handlers throw — the framework catches, classifies, and formats.
 
-**Default for new tools: typed error contract.** Declare `errors: [{ reason, code, when, recovery, retryable? }]` on `tool()` to receive a typed `ctx.fail(reason, …)` keyed by the declared reason union. TypeScript catches `ctx.fail('typo')` at compile time, `data.reason` is auto-populated for observability, and the linter enforces conformance against the handler body. The `recovery` field is required descriptive metadata (≥ 5 words, lint-validated); to surface it on the wire, spread `...ctx.recoveryFor('reason')` into `data` or pass an explicit `{ recovery: { hint: '...' } }` when runtime context matters. Baseline codes (`InternalError`, `ServiceUnavailable`, `Timeout`, `ValidationError`, `SerializationError`) bubble freely and don't need declaring. Live across the BrAPI surface today: `brapi_connect`, `brapi_describe_filters`, `brapi_find_genotype_calls`, `brapi_get_germplasm`, `brapi_get_image`, `brapi_get_study`, `brapi_manage_dataset`, `brapi_raw_get`, `brapi_raw_search`, `brapi_submit_observations`, plus the `brapi://study/{studyDbId}`, `brapi://germplasm/{germplasmDbId}`, and `brapi://filters/{endpoint}` resources.
+**Default for new tools: typed error contract.** Declare `errors: [{ reason, code, when, recovery, retryable? }]` on `tool()` to receive a typed `ctx.fail(reason, …)` keyed by the declared reason union. TypeScript catches `ctx.fail('typo')` at compile time, `data.reason` is auto-populated for observability, and the linter enforces conformance against the handler body. The `recovery` field is required descriptive metadata (≥ 5 words, lint-validated); to surface it on the wire, spread `...ctx.recoveryFor('reason')` into `data` or pass an explicit `{ recovery: { hint: '...' } }` when runtime context matters. Baseline codes (`InternalError`, `ServiceUnavailable`, `Timeout`, `ValidationError`, `SerializationError`) bubble freely and don't need declaring. Live across the BrAPI surface today: `brapi_connect`, `brapi_dataframe_query`, `brapi_describe_filters`, `brapi_find_genotype_calls`, `brapi_get_germplasm`, `brapi_get_image`, `brapi_get_study`, `brapi_raw_get`, `brapi_raw_search`, `brapi_submit_observations`, plus the `brapi://study/{studyDbId}`, `brapi://germplasm/{germplasmDbId}`, and `brapi://filters/{endpoint}` resources.
 
 ```ts
 errors: [
@@ -215,7 +216,7 @@ Available factories include `notFound`, `validationError`, `forbidden`, `unautho
 
 ```text
 src/
-  index.ts                                # createApp() entry point — registers 19 tools, 6 resources, 2 prompts; inits 7 services
+  index.ts                                # createApp() entry point — registers 21 tools, 5 resources, 2 prompts; inits 7 services
   config/
     server-config.ts                      # BRAPI_* env vars (Zod schema, lazy-parsed)
     alias-credentials.ts                  # Per-alias env-var resolution (BRAPI_<ALIAS>_*) for brapi_connect
@@ -223,8 +224,8 @@ src/
     brapi-client/                         # HTTP client — retry, concurrency cap, async-search poll, private-IP guard, binary fetch, POST/PUT
     brapi-dialect/                        # Per-server filter / payload adapters (spec, cassavabase) — translates plural→singular, drops searchText, declares known-dead POST /search routes; envelope surfaces id + source + disabled-search nouns
     brapi-filters/                        # Static v2.1 filter catalog
+    canvas-bridge/                        # Per-tenant default-canvas resolver, df_<uuid> table generator, provenance store
     capability-registry/                  # Per-connection /serverinfo cache + call guard
-    dataset-store/                        # Tenant-scoped handles for spilled find_* results
     ontology-resolver/                    # Free-text → ontology-term matcher for variables
     reference-data-cache/                 # Programs / trials / locations / crops lookup cache
     server-registry/                      # Alias → live connection map with auth resolution
@@ -245,15 +246,18 @@ src/
         brapi-get-image.tool.ts           # Fetch image bytes inline (imagecontent → imageURL fallback)
         brapi-find-locations.tool.ts      # find_* — locations, optional client-side bbox filter
         brapi-find-variants.tool.ts       # find_* — variants, 1-based inclusive/exclusive genomic region
-        brapi-find-genotype-calls.tool.ts # Async-search genotype calls with maxCalls cap + spillover
-        brapi-manage-dataset.tool.ts      # Dataset lifecycle — list / summary / load / delete
+        brapi-find-genotype-calls.tool.ts # Async-search genotype calls with maxCalls cap + dataframe spillover
+        brapi-dataframe-describe.tool.ts  # List / describe canvas dataframes with columns, row counts, provenance
+        brapi-dataframe-query.tool.ts     # Run SQL across canvas dataframes (SELECT only); typed columns response
+        brapi-dataframe-drop.tool.ts      # Drop a dataframe by name (opt-in via BRAPI_CANVAS_DROP_ENABLED)
+        brapi-dataframe-export.tool.ts    # Write CSV/Parquet/JSON to BRAPI_EXPORT_DIR (opt-in, stdio-only)
         brapi-submit-observations.tool.ts # Two-phase observation write — preview / apply (POST + PUT) with elicit gate
         brapi-raw-get.tool.ts             # Last-resort GET passthrough with routing nudge
         brapi-raw-search.tool.ts          # Last-resort POST /search passthrough with async polling
       shared/
         connect-auth-schema.ts            # Tagged-union auth input
         orientation-envelope.ts           # Shared envelope builder + formatter
-        find-helpers.ts                   # Alias / loadLimit / extraFilters fragments, mergeFilters, maybeSpill, DatasetHandleSchema
+        find-helpers.ts                   # Alias / loadLimit / extraFilters fragments, mergeFilters, maybeSpill, DataframeHandleSchema
         raw-routing-hints.ts              # Routing nudges emitted by raw_get / raw_search when a curated tool exists
     resources/
       definitions/
@@ -261,7 +265,6 @@ src/
         brapi-calls.resource.ts           # brapi://calls — raw capability profile
         brapi-study.resource.ts           # brapi://study/{studyDbId} — single study with FKs
         brapi-germplasm.resource.ts       # brapi://germplasm/{germplasmDbId} — single germplasm with attributes + parents
-        brapi-dataset.resource.ts         # brapi://dataset/{datasetId} — dataset metadata + provenance
         brapi-filters.resource.ts         # brapi://filters/{endpoint} — filter catalog
     prompts/
       definitions/
@@ -308,6 +311,7 @@ Available skills:
 | `report-issue-framework` | File a bug or feature request against `@cyanheads/mcp-ts-core` via `gh` CLI |
 | `report-issue-local` | File a bug or feature request against this server's own repo via `gh` CLI |
 | `api-auth` | Auth modes, scopes, JWT/OAuth |
+| `api-canvas` | DataCanvas: register tabular data, run SQL, export, plus the `spillover()` helper for big result sets — Tier 3 opt-in |
 | `api-config` | AppConfig, parseConfig, env vars |
 | `api-context` | Context interface, logger, state, progress |
 | `api-errors` | McpError, JsonRpcErrorCode, error patterns |
@@ -387,7 +391,7 @@ import { getMyService } from '@/services/my-domain/my-service.js';
 - [ ] BrAPI tool: gates the call with `CapabilityRegistry.ensure(...)` — never fires against an endpoint the server didn't advertise
 - [ ] BrAPI tool: raw / domain / output schemas reviewed against real upstream sparsity (most `/germplasm` and `/studies` fields are optional in the wild)
 - [ ] BrAPI tool: normalization and `format()` preserve uncertainty — never fabricate missing IDs, names, or counts
-- [ ] BrAPI tool with dataset spillover: rows beyond `loadLimit` persist via `DatasetStore`, handle surfaces in `result.dataset`, `hasMore` set correctly
+- [ ] BrAPI tool with dataframe spillover: rows beyond `loadLimit` materialize as a `df_<uuid>` canvas table via `CanvasBridge.registerDataframe`, handle surfaces in `result.dataframe`, `hasMore` set correctly
 - [ ] Tests include at least one sparse upstream payload (fields omitted) alongside the happy path
 - [ ] Registered in the `tools` array of `createApp()` in `src/index.ts`
 - [ ] Tests use `createMockContext()` from `@cyanheads/mcp-ts-core/testing`

package/README.md

@@ -1,13 +1,13 @@
 <div align="center">
   <h1>@cyanheads/brapi-mcp-server</h1>
   <p><b>BrAPI v2.1 MCP server — studies, germplasm, observations, genotypes, images, and pedigrees across Breedbase, T3, Sweetpotatobase, and any BrAPI-compliant server.</b>
-  <div>22 Tools • 6 Resources • 2 Prompts</div>
+  <div>22 Tools • 5 Resources • 2 Prompts</div>
   </p>
 </div>
 
 <div align="center">
 
-[![npm](https://img.shields.io/npm/v/@cyanheads/brapi-mcp-server?style=flat-square&logo=npm&logoColor=white)](https://www.npmjs.com/package/@cyanheads/brapi-mcp-server) [![Version](https://img.shields.io/badge/Version-0.4.14-blue.svg?style=flat-square)](./CHANGELOG.md) [![Framework](https://img.shields.io/badge/Built%20on-@cyanheads/mcp--ts--core-259?style=flat-square)](https://www.npmjs.com/package/@cyanheads/mcp-ts-core) [![MCP SDK](https://img.shields.io/badge/MCP%20SDK-^1.29.0-green.svg?style=flat-square)](https://modelcontextprotocol.io/)
+[![npm](https://img.shields.io/npm/v/@cyanheads/brapi-mcp-server?style=flat-square&logo=npm&logoColor=white)](https://www.npmjs.com/package/@cyanheads/brapi-mcp-server) [![Version](https://img.shields.io/badge/Version-0.5.1-blue.svg?style=flat-square)](./CHANGELOG.md) [![Framework](https://img.shields.io/badge/Built%20on-@cyanheads/mcp--ts--core-259?style=flat-square)](https://www.npmjs.com/package/@cyanheads/mcp-ts-core) [![MCP SDK](https://img.shields.io/badge/MCP%20SDK-^1.29.0-green.svg?style=flat-square)](https://modelcontextprotocol.io/)
 
 [![License](https://img.shields.io/badge/License-Apache%202.0-orange.svg?style=flat-square)](./LICENSE) [![TypeScript](https://img.shields.io/badge/TypeScript-^6.0.3-3178C6.svg?style=flat-square)](https://www.typescriptlang.org/) [![Bun](https://img.shields.io/badge/Bun-v1.3.11-blueviolet.svg?style=flat-square)](https://bun.sh/) [![Status](https://img.shields.io/badge/Status-Beta-yellow.svg?style=flat-square)](./CHANGELOG.md)
 
@@ -17,7 +17,7 @@
 
 ## Tools
 
-Twenty-two tools grouped by shape — connection tools bootstrap a session, `find_*` tools return a summarized page plus distributions and spill overflow rows into the DatasetStore, `get_*` tools fetch a single record with companion counts, plus pedigree walking, dataset lifecycle, an opt-in SQL workspace over spilled rows, an additive write surface for observations, and raw passthrough escape hatches.
+22 tools grouped by shape — connection tools bootstrap a session, `find_*` tools return a summarized page plus distributions and spill overflow rows into a canvas dataframe, `get_*` tools fetch a single record with companion counts, plus pedigree walking, an embedded SQL workspace over spilled rows (DuckDB-backed), file export for human handoff, an additive write surface for observations, and raw passthrough escape hatches.
 
 ### Orient
 
@@ -31,32 +31,27 @@ Twenty-two tools grouped by shape — connection tools bootstrap a session, `fin
 
 | Tool | Description |
 |:-----|:------------|
-| `brapi_find_studies` | Find studies by crop / trial type / season / location / program. Distributions + dataset spillover. |
+| `brapi_find_studies` | Find studies by crop / trial type / season / location / program. Distributions + dataframe spillover. |
 | `brapi_get_study` | Fetch a study with program / trial / location FKs resolved and companion counts (observations, units, variables). |
-| `brapi_find_germplasm` | Find germplasm by name, synonym, accession, PUI, crop, or free-text. Distributions + dataset spillover. |
+| `brapi_find_germplasm` | Find germplasm by name, synonym, accession, PUI, crop, or free-text. Distributions + dataframe spillover. |
 | `brapi_get_germplasm` | Fetch a germplasm with attributes, direct parents, and companion counts (studies, parents, descendants). |
 | `brapi_walk_pedigree` | BFS-walk ancestry / descendancy as a deduplicated DAG with cycle detection, depth limits, and traversal stats. |
 | `brapi_find_variables` | Find observation variables by name / class / ontology / free-text; ranked client-side via `OntologyResolver` when `text` is supplied. |
-| `brapi_find_observations` | Pull observation records by study / germplasm / variable / season / unit / timestamp. Dataset spillover. |
+| `brapi_find_observations` | Pull observation records by study / germplasm / variable / season / unit / timestamp. Dataframe spillover. |
 | `brapi_find_images` | Filter image metadata by unit / study / ontology / MIME type. Bytes via `brapi_get_image`. |
 | `brapi_get_image` | Fetch image bytes for up to 5 imageDbIds inline as `type: image` blocks. Prefers `/imagecontent`, falls back to `imageURL`. |
 | `brapi_find_locations` | Find research stations by country / type / abbreviation, with optional client-side bbox filter. |
 | `brapi_find_variants` | Find variant records by variant set, reference, or genomic region (1-based inclusive / exclusive). |
 | `brapi_find_genotype_calls` | Pull genotype calls via async-search polling. Upstream pull bounded by `BRAPI_GENOTYPE_CALLS_MAX_PULL` (default 100k, max 500k). |
 
-### Orchestrate
+### Analyze
 
 | Tool | Description |
 |:-----|:------------|
-| `brapi_manage_dataset` | Lifecycle for `find_*` spillover datasets — list / summary / load (paged + projected) / delete. |
-
-### Analyze (opt-in: `CANVAS_PROVIDER_TYPE=duckdb` + `BRAPI_CANVAS_ENABLED=true`)
-
-| Tool | Description |
-|:-----|:------------|
-| `brapi_dataframe_query` | SELECT SQL across in-memory dataframes (DuckDB-backed). Spilled `find_*` datasets auto-register as `ds_<datasetId>`. Read-only — multi-statement, non-SELECT, file-reads, and exports rejected. |
-| `brapi_dataframe_describe` | List dataframes with column schema, row counts, and dataset provenance. |
-| `brapi_dataframe_drop` | Drop a dataframe by name. Idempotent. Underlying dataset is unaffected. |
+| `brapi_dataframe_describe` | Start here after a spillover. Lists dataframes (or describes one) with column schema, row counts, and originating-source provenance. |
+| `brapi_dataframe_query` | SELECT SQL across in-memory dataframes (DuckDB-backed). Spilled `find_*` rows auto-register as `df_<uuid>`. Read-only — multi-statement, non-SELECT, file-reads, and exports rejected. Returns typed columns (`{ name, type }[]`). |
+| `brapi_dataframe_drop` | _Opt-in via `BRAPI_CANVAS_DROP_ENABLED=true`._ Drop a dataframe by name. Idempotent. Dataframes also expire via TTL when left unmanaged. |
+| `brapi_dataframe_export` | _Opt-in via `BRAPI_EXPORT_DIR=<path>`, stdio-only._ Export a dataframe to disk (CSV / Parquet / JSON) under the configured directory and return the absolute path for the human to open. Optional `columns` projection or `sql` filter materializes a derived table for the export, dropped after. |
 
 ### Write (opt-in: `BRAPI_ENABLE_WRITES=true`)
 
@@ -85,7 +80,6 @@ URI-addressable mirrors of the curated tool surface for clients that prefer reso
 | `brapi://calls` | Raw capability profile |
 | `brapi://study/{studyDbId}` | `brapi_get_study` |
 | `brapi://germplasm/{germplasmDbId}` | `brapi_get_germplasm` |
-| `brapi://dataset/{datasetId}` | Dataset metadata + provenance |
 | `brapi://filters/{endpoint}` | `brapi_describe_filters` |
 
 ---
@@ -106,20 +100,38 @@ Multi-step BrAPI workflow templates — pure user-message generators, no side ef
 - **Multi-server session** — `ServerRegistry` maps aliases to live BrAPI connections; one session can span Breedbase, T3, and Sweetpotatobase in parallel.
 - **Capability-aware calls** — `CapabilityRegistry` caches `/serverinfo` per connection and guards every tool call against unsupported endpoints. Falls back to `/calls` when `/serverinfo` is sparse.
 - **Built-in known-server registry** — `bti-cassava`, `bti-sweetpotato`, `bti-breedbase-demo`, `t3-wheat`, `t3-oat`, `t3-barley` resolve out-of-the-box without env vars; orientation envelope carries CC-BY attribution.
-- **Dataset spillover** — `find_*` tools cap in-context rows at `loadLimit` and persist larger unions (up to 50k rows / 50 pages) as handles in `DatasetStore`. `brapi_manage_dataset` pages / projects / deletes them.
-- **Dataframes (Tier 3, opt-in)** — spilled rows auto-register as DuckDB-backed `ds_<datasetId>` dataframes; agents run SELECT SQL via `brapi_dataframe_query` with read-only enforcement and a per-tenant workspace. Requires the optional `@duckdb/node-api` peer dep. Not on Cloudflare Workers.
+- **Dataframe spillover** — `find_*` tools cap in-context rows at `loadLimit` and materialize larger unions (up to 50k rows / 50 pages) as DuckDB-backed `df_<uuid>` canvas dataframes. Discover with `brapi_dataframe_describe`, query with `brapi_dataframe_query` (SQL paging via `LIMIT/OFFSET`, projection, aggregation). Read-only enforcement at the SQL gate; per-tenant workspace.
+- **DuckDB required** — `@duckdb/node-api` is a regular dependency; startup fails closed when the framework canvas is unavailable. Not supported on Cloudflare Workers (no native binary in that runtime).
 - **Async-search transparency** — `brapi_find_genotype_calls` and `brapi_raw_search` handle the `POST /search/{noun}` → `GET /search/{noun}/{id}` 202-retry pattern automatically.
 - **Pedigree DAG walks** — `brapi_walk_pedigree` BFS-traverses ancestry / descendancy with cycle detection (BrAPI only exposes one generation per call).
 - **Image content** — `brapi_get_image` fetches bytes inline as MCP `type: image` blocks, preferring `/images/{id}/imagecontent` with `imageURL` fallback.
 - **Free-text variable ranking** — `OntologyResolver` scores variables against a query (PUI / name / synonym / trait-class) so `find_variables text:"..."` returns ranked candidates even without `/ontologies`.
 - **Auth variants in one schema** — tagged-union covers `none` / `bearer` / `api_key` / `sgn` (session-token exchange) / `oauth2` (client-credentials).
 - **Typed error contracts** — every declared failure mode carries a stable `data.reason`, an HTTP-style `code`, and a `recovery.hint` so clients can route deterministically.
-- **Compatibility matrix** — live server probes tracked in [docs/compatibility.md](./docs/compatibility.md).
 
 Built on [`@cyanheads/mcp-ts-core`](https://www.npmjs.com/package/@cyanheads/mcp-ts-core) — declarative definitions, unified error handling, pluggable auth (`none` / `jwt` / `oauth`), swappable storage, structured logging with optional OTel, STDIO + Streamable HTTP transports.
 
 ---
 
+## Working with large results
+
+When a `find_*` tool's upstream total exceeds `loadLimit`, the full union materializes as a canvas dataframe and the response carries an inline `dataframe` handle (`{ tableName, rowCount, columns, createdAt, expiresAt, … }`). SQL is the paging idiom — use `LIMIT/OFFSET` to walk pages, projection (`SELECT col1, col2`) to trim columns, and aggregation (`COUNT`, `GROUP BY`, `AVG`) to summarize without materializing every row.
+
+```text
+1. brapi_find_observations { studies: ["s-422"] }
+   → first-page rows inline + dataframe.tableName = "df_<uuid>" (when totalCount > loadLimit)
+2. brapi_dataframe_describe { dataframe: "df_<uuid>" }
+   → schema + provenance (originating tool, baseUrl, query, expiry)
+3. brapi_dataframe_query { sql: "SELECT germplasmName, value FROM df_<uuid> WHERE observationVariableDbId = 'V1' LIMIT 100" }
+   → typed columns + bounded rows
+4. brapi_dataframe_query { sql: "SELECT COUNT(*) AS n, AVG(CAST(value AS DOUBLE)) AS mean FROM df_<uuid> WHERE observationVariableDbId = 'V1'" }
+   → aggregate without round-tripping all rows
+```
+
+Dataframes auto-expire via TTL (`BRAPI_DATASET_TTL_SECONDS`, default 24h). Set `BRAPI_CANVAS_DROP_ENABLED=true` to expose `brapi_dataframe_drop` for explicit cleanup.
+
+---
+
 ## Getting started
 
 Add to your MCP client config — pick one runner:
@@ -148,7 +160,7 @@ MCP_TRANSPORT_TYPE=http MCP_HTTP_PORT=3010 bun run start:http
 
 No env vars are required — the six built-in aliases (`bti-cassava`, `bti-sweetpotato`, `bti-breedbase-demo`, `t3-wheat`, `t3-oat`, `t3-barley`) resolve out-of-the-box, and agents can connect to any other BrAPI v2 URL at runtime via `brapi_connect`. **For credentialed servers, prefer env vars over agent input** so passwords / tokens / API keys stay out of the LLM context — see [Per-alias credentials](#per-alias-credentials).
 
-**Prerequisites:** [Bun v1.3.11+](https://bun.sh/) or Node.js v22+. *(Optional)* [`@duckdb/node-api`](https://www.npmjs.com/package/@duckdb/node-api) for the dataframe surface — Linux/macOS/Windows × x64 plus Linux/macOS arm64 (no Windows arm64; no Cloudflare Workers).
+**Prerequisites:** [Bun v1.3.11+](https://bun.sh/) or Node.js v22+. [`@duckdb/node-api`](https://www.npmjs.com/package/@duckdb/node-api) is a required dependency — supported on Linux/macOS/Windows × x64 plus Linux/macOS arm64 (no Windows arm64; no Cloudflare Workers).
 
 ---
 
@@ -163,18 +175,20 @@ Every variable is optional.
 | `BRAPI_DEFAULT_OAUTH_CLIENT_ID` / `_OAUTH_CLIENT_SECRET` | OAuth2 client-credentials for the default connection. | — |
 | `BRAPI_DEFAULT_API_KEY` / `_API_KEY_HEADER` | Static API key for the default connection. | header `Authorization` |
 | `BRAPI_BUILTIN_ALIASES_DISABLED` | Comma-separated alias names (case-insensitive) to remove from the built-in registry. | — |
-| `BRAPI_LOAD_LIMIT` | In-context row cap before `find_*` spills to `DatasetStore`. | `200` |
+| `BRAPI_LOAD_LIMIT` | In-context row cap before `find_*` spills to a canvas dataframe; also doubles as the upstream pageSize during spillover walks (`loadLimit × 50` is the dataframe ceiling). | `1000` |
 | `BRAPI_MAX_CONCURRENT_REQUESTS` | Per-connection concurrency cap. | `4` |
 | `BRAPI_RETRY_MAX_ATTEMPTS` / `BRAPI_RETRY_BASE_DELAY_MS` | Retry policy for 429/5xx with exponential backoff. | `3` / `500` |
 | `BRAPI_REQUEST_TIMEOUT_MS` | Per-request HTTP timeout. | `30000` |
+| `BRAPI_COMPANION_TIMEOUT_MS` | Tighter timeout for non-critical companion enrichments (FK lookups, count probes). Companions also bypass the retry budget so a slow upstream surfaces as a warning instead of stretching the response. | `8000` |
 | `BRAPI_SEARCH_POLL_TIMEOUT_MS` / `_INTERVAL_MS` | Async `/search` polling budget + interval. | `60000` / `1000` |
-| `BRAPI_DATASET_TTL_SECONDS` / `BRAPI_DATASET_STORE_DIR` | TTL for spilled datasets + filesystem path when filesystem storage is active. | `86400` / — |
+| `BRAPI_DATASET_TTL_SECONDS` | TTL for dataframe provenance metadata persisted alongside spilled rows. | `86400` |
 | `BRAPI_REFERENCE_CACHE_TTL_SECONDS` | TTL for programs / trials / locations / crops cache. | `3600` |
 | `BRAPI_ALLOW_PRIVATE_IPS` | Allow RFC 1918 / loopback targets. Dev-only. | `false` |
 | `BRAPI_ENABLE_WRITES` | Opt-in for `brapi_submit_observations` registration. | `false` |
 | `BRAPI_GENOTYPE_CALLS_MAX_PULL` | Upstream row ceiling per `brapi_find_genotype_calls` invocation. Max 500,000. | `100000` |
-| `CANVAS_PROVIDER_TYPE` | Framework `DataCanvas` switch — set to `duckdb` to enable the dataframe surface. | `none` |
-| `BRAPI_CANVAS_ENABLED` / `BRAPI_CANVAS_MAX_ROWS` / `BRAPI_CANVAS_QUERY_TIMEOUT_MS` | Dataframe surface gate, response row cap, per-query timeout. | `true` / `10000` / `30000` |
+| `BRAPI_CANVAS_DROP_ENABLED` | Opt-in for `brapi_dataframe_drop` registration. Off by default; dataframes expire via TTL when left unmanaged. | `false` |
+| `BRAPI_EXPORT_DIR` | Directory for `brapi_dataframe_export` output files. Setting a path is the opt-in (no separate enable flag); unset leaves the tool out of `tools/list`. Stdio-only — the tool stays disabled under HTTP transport regardless of this value. Bridged to the framework's `CANVAS_EXPORT_PATH` automatically. | — |
+| `BRAPI_CANVAS_MAX_ROWS` / `BRAPI_CANVAS_QUERY_TIMEOUT_MS` | Per-query response row cap and wall-clock timeout for `brapi_dataframe_query`. | `10000` / `30000` |
 | `MCP_TRANSPORT_TYPE` / `MCP_HTTP_PORT` | Transport (`stdio` \| `http`) + HTTP port. | `stdio` / `3010` |
 | `MCP_AUTH_MODE` / `MCP_LOG_LEVEL` / `STORAGE_PROVIDER_TYPE` / `OTEL_ENABLED` | Auth mode (`none` \| `jwt` \| `oauth`), log level, storage backend, OpenTelemetry. | `none` / `info` / `in-memory` / `false` |
 
@@ -263,7 +277,7 @@ Defaults to HTTP transport, stateless session mode, logs to `/var/log/brapi-mcp-
 
 ### Multi-user HTTP deployments
 
-`ctx.state` (where `ServerRegistry`, `DatasetStore`, and `CapabilityRegistry` live) is **scoped by `tenantId`, not session id**:
+`ctx.state` (where `ServerRegistry`, `CanvasBridge`, and `CapabilityRegistry` live) is **scoped by `tenantId`, not session id**:
 
 | Mode | `tenantId` |
 |:-----|:-----------|

package/changelog/0.5.x/0.5.0.md

@@ -0,0 +1,59 @@
+---
+summary: "Consolidate spillover onto canvas-only dataframes — remove brapi_manage_dataset and the brapi://dataset/{id} resource, mandate DuckDB, gate brapi_dataframe_drop opt-in, and return typed columns from brapi_dataframe_query."
+breaking: true
+---
+
+# 0.5.0 — 2026-05-05
+
+Collapses the dual-store spillover model. Pre-canvas, `find_*` tools wrote spilled rows twice — once as JSON in `ctx.state` (driving `brapi_manage_dataset`) and once as a DuckDB table on the canvas (driving `brapi_dataframe_*`). The two surfaces did the same job, so this release retires the JSON store and makes SQL the single way to work with spilled results. DuckDB is now a regular dependency rather than an optional peer; the server fails fast at startup when it isn't available. Resolves [#25](https://github.com/cyanheads/brapi-mcp-server/issues/25), subsumes [#24](https://github.com/cyanheads/brapi-mcp-server/issues/24).
+
+## Migration
+
+| Before (≤ 0.4.x) | After (0.5.0) |
+|:------------------|:---------------|
+| `brapi_manage_dataset { mode: 'list' }` | `brapi_dataframe_describe` (no args) |
+| `brapi_manage_dataset { mode: 'summary', datasetId }` | `brapi_dataframe_describe { dataframe: 'df_<uuid>' }` |
+| `brapi_manage_dataset { mode: 'load', datasetId, page, pageSize, columns }` | `brapi_dataframe_query { sql: 'SELECT col1, col2 FROM df_<uuid> LIMIT 100 OFFSET 0' }` |
+| `brapi_manage_dataset { mode: 'delete', datasetId }` | `brapi_dataframe_drop { dataframe }` when `BRAPI_CANVAS_DROP_ENABLED=true`; otherwise wait for canvas TTL |
+| `brapi://dataset/{id}` resource | `brapi_dataframe_describe { dataframe }` |
+| `find_*` tool output `dataset.datasetId` | `find_*` tool output `dataframe.tableName` |
+
+## Removed
+
+- **Tool: `brapi_manage_dataset`.** The lifecycle surface for ctx.state-stored datasets is gone. Use `brapi_dataframe_describe` to discover dataframes, `brapi_dataframe_query` to read or aggregate, and `brapi_dataframe_drop` (opt-in, see Added) to release one explicitly.
+- **Resource: `brapi://dataset/{datasetId}`.** Provenance now lives on the dataframe itself; describe surfaces source / baseUrl / query / createdAt / expiresAt under `provenance` for any auto-registered `df_*` table.
+- **Service: `src/services/dataset-store/`** (entire module). Spilled rows go directly to the canvas — no parallel JSON copy.
+- **Env var: `BRAPI_CANVAS_ENABLED`.** Canvas is mandatory now. Startup fails with a clear `ConfigurationError` when `core.canvas` is undefined (e.g., DuckDB unavailable).
+- **Env var: `BRAPI_DATASET_STORE_DIR`** (filesystem path for the deprecated DatasetStore). No longer read.
+- **Helper: `spillToDataset` in `src/mcp-server/tools/shared/find-helpers.ts`.** Replaced by `spillToCanvas`. The shared `DatasetHandleSchema` / `renderDatasetHandle` / `toDatasetHandle` exports are renamed to their `Dataframe*` equivalents.
+- **`docs/compatibility.md`.** Stale live-server probe matrix dropped — to be regenerated as needed against the current dialect surface. README cross-link removed.
+
+## Added
+
+- **Env var: `BRAPI_CANVAS_DROP_ENABLED`** (default `false`). Opt-in for `brapi_dataframe_drop` — when unset, the tool is omitted from `tools/list` entirely and dataframes expire via TTL when left unmanaged. Resolves [#24](https://github.com/cyanheads/brapi-mcp-server/issues/24).
+- **`brapi_dataframe_query` returns typed columns.** Output `columns` field changes from `string[]` (names only) to `{ name: string; type: string }[]` (name + SQL/DuckDB type). Types come from DuckDB's describe in both directions: when `registerAs` is set the user-named table is described directly, and when it isn't the handler runs the query under an internal `_brapi_probe_<uuid>` `registerAs` so DuckDB's authoritative types survive the JSON round-trip (which would otherwise flatten BIGINT and other non-JSON-native types to VARCHAR strings), then drops the probe before returning. The agent learns the schema implicitly through any query response — no need to round-trip through `dataframe_describe` between queries.
+- **Server-side default `CANVAS_PROVIDER_TYPE=duckdb`.** The framework defaults this to `none`, which would fail every startup now that canvas is mandatory. The entry point sets the env var to `duckdb` before `createApp` reads config, so the server works out-of-the-box without operator setup. Operator-set values pass through untouched (including `none` for diagnostic runs that should fail closed).
+- **`brapi_raw_get` and `brapi_raw_search` spill to a canvas dataframe.** When the upstream advertises more rows than `loadLimit` AND the result is a list shape (top-level array or BrAPI `result.data` envelope), the handler walks subsequent pages via `spillToCanvas` and attaches a `dataframe` handle on the response — same `df_<uuid>` shape as `find_*`. Inline `result` is returned **unchanged** so the raw escape hatch keeps its semantics; the dataframe is the "and there is more, here" pointer. Both tools now accept an optional `loadLimit` input (defaults to `serverConfig.loadLimit`, 1,000). Spillover is intentionally skipped when the caller drives paging via `params.page` / `params.pageSize` (raw_get) or `body.page` / `body.pageSize` (raw_search) — they are walking pages explicitly. Non-list results (single objects, scalars) pass through as before regardless of total count. New `extractListRows` helper in `find-helpers.ts` factors out the list-shape detection.
+- **Tool: `brapi_dataframe_export`.** Write a dataframe to disk in CSV, Parquet, or JSON for human review — the agent calls it when the researcher signals they want to open the data in Excel, Tad, DuckDB CLI, etc. Returns the absolute path; the file lands inside the operator-configured `BRAPI_EXPORT_DIR`. Optional `columns` (thin projection) or `sql` (full SELECT, mutually exclusive) materializes a derived canvas table for the export and drops it after. Default filename is `<dataframe>-<unixSeconds>.<ext>` so re-exports never collide; explicit `filename` overwrites on collision. Best-effort sweep of stale files (mtime past `BRAPI_DATASET_TTL_SECONDS`) runs on each call; `brapi_dataframe_drop` unlinks paired export files alongside the dataframe. Resolves [#18](https://github.com/cyanheads/brapi-mcp-server/issues/18).
+- **Env var: `BRAPI_EXPORT_DIR`.** Setting a writable path is the opt-in for `brapi_dataframe_export` (no separate enable flag). Bridged automatically to the framework's `CANVAS_EXPORT_PATH` so the existing canvas path-traversal sandbox (absolute paths and `..` segments rejected) governs filename inputs. Tool registration also requires `MCP_TRANSPORT_TYPE=stdio`; under HTTP transport the tool stays disabled regardless of this value because the returned path lives on the server, not the user. Operators see two distinct gate reasons in `/.well-known/mcp.json` — "stdio transport required" and "BRAPI_EXPORT_DIR unset" — depending on which check failed.
+- **`CanvasBridge.export(ctx, tableName, target, options)`** wraps the framework's `instance.export()` with optional path tracking via `CanvasBridge.trackExport(ctx, sourceName, path)`. Tracked paths key off the *source* dataframe so `brapi_dataframe_drop(ctx, sourceName)` unlinks the file alongside the canvas table — the bridge's `drop()` reads `brapi/canvas/exports/<name>` and unlinks before deleting the state entry. Tracking is caller-driven (the export tool tracks; the bridge does not auto-track) so a transient projection table can be exported then dropped without unlinking the file.
+
+## Changed
+
+- **`find_*` tool output: `dataset` field renamed to `dataframe`.** Shape simplified: drops `datasetId` and `sizeBytes`; keeps `tableName`, `rowCount`, `columns`, `createdAt`, `expiresAt`, `truncated`, `maxRows`. Eight tools affected: `brapi_find_germplasm`, `brapi_find_studies`, `brapi_find_observations`, `brapi_find_locations`, `brapi_find_variables`, `brapi_find_variants`, `brapi_find_images`, `brapi_find_genotype_calls`.
+- **`find_*` tool descriptions emit "dataframe handle" language.** Render output points at `brapi_dataframe_describe` and `brapi_dataframe_query` instead of `brapi_manage_dataset`.
+- **`brapi_dataframe_query` description: explicit SQL-as-paging guidance.** `LIMIT/OFFSET`, projection (`SELECT col1, col2`), and aggregation (`COUNT`, `GROUP BY`, `AVG`) are the paging idiom — no need for a structured-paging shim on top.
+- **`brapi_dataframe_describe` description: leads with "start here after a spillover."** The discovery surface is the natural first step after a `find_*` call returns a `dataframe` handle.
+- **`brapi_dataframe_drop` description and error contract.** Drops the `dataframe_disabled` typed error (canvas is mandatory now); idempotent semantics unchanged — returns `dropped: false` rather than failing for unknown names.
+- **`brapi_dataframe_describe` provenance schema: drop `datasetId`.** The dataframe name (`df_<uuid>`) is the only identity now. `provenance` exposes `source`, `baseUrl`, `query`, `createdAt`, `expiresAt`.
+- **Internal canvas table prefix: `ds_` → `df_`** (in `canvas-bridge.ts`). Matches the new noun. The agent always sees the full table name from the handle, so this is a purely internal rename.
+- **Prompts updated.** `brapi_eda_study` and `brapi_meta_analysis` now point at `brapi_dataframe_query` (with SQL paging guidance) instead of `brapi_manage_dataset` for accessing spilled rows.
+- **Raw routing hints (`raw-routing-hints.ts`) updated** to reference dataframe spillover instead of dataset spillover.
+- **`BRAPI_LOAD_LIMIT` default raised 200 → 1,000.** `loadLimit` doubles as the upstream `pageSize` during spillover walks, so the prior default silently capped dataframes at `200 × 50 pages = 10,000` rows even though `MAX_SPILLOVER_ROWS = 50,000` was the documented ceiling. The new default lines up the math: `1,000 × 50 = 50,000`, matching the cap. Operators on small/test BrAPI servers can lower it via env; raising it above 1,000 is rarely useful because most BrAPI servers cap server-side `pageSize` at 1,000.
+- **`brapi_find_genotype_calls` honors `BRAPI_LOAD_LIMIT`.** The handler had a hardcoded `?? 200` fallback that ignored the configured default. Now reads `getServerConfig().loadLimit` like every other find_* tool. Tool description and `loadLimit` input description both updated to drop the stale "default 200" claim. Genotype calls remain a special case in one respect: upstream pageSize is fixed at 10,000 (decoupled from `loadLimit`), so the dual-role footgun other find_* tools have doesn't apply here — the input describe calls this out explicitly.
+- **`LoadLimitInput.describe()` rewritten for honest agent guidance.** The shared input fragment now warns that lowering `loadLimit` to "see fewer rows inline" silently shrinks dataframe spillover capacity proportionally (because `loadLimit` doubles as upstream `pageSize`), and points the agent at SQL `LIMIT` on the dataframe for sampling instead. Drops env-var names and specific numeric values from the agent-facing copy.
+- **`brapi_meta_analysis` prompt drops hardcoded `loadLimit: 200`.** The Step 2 directive let the deployment default win instead of pinning a number that's no longer the default. One-line fix.
+
+## Dependencies
+
+- **`@duckdb/node-api` promoted from optional peer to regular dependency.** Spillover is the primary value of this server; making DuckDB optional only protected users we don't target. Cloudflare Workers deployments are not supported until a non-DuckDB canvas provider lands upstream — documented as a known constraint.

package/changelog/0.5.x/0.5.1.md

@@ -0,0 +1,13 @@
+---
+summary: "Verify extraFilters honor on every find_* tool and relabel the rendered filters block as `Filters sent to server` so the trace stops implying the upstream applied them."
+breaking: false
+---
+
+# 0.5.1 — 2026-05-05
+
+Two field-test wrong-results regressions on `find_*` tools. Both classes shared a root cause: the trace claimed filter values were honored when the upstream had silently dropped them.
+
+## Fixed
+
+- **`find_*` tools now post-hoc-verify `extraFilters` keys against the returned rows.** New `buildExtraFilterChecks` helper in `src/mcp-server/tools/shared/find-helpers.ts` infers the row column for each `extraFilters` key (exact match, then trailing-`s` strip — `locationDbIds → locationDbId`, `locationNames → locationName`) and feeds a `FilterMatchCheck` into `checkFilterMatchRates`. When the upstream silently ignored a filter (e.g. `locationName` on `/studies` — not a valid spec key), every returned row carries an unrequested value and the validator emits a warning instead of letting the empty trace falsely confirm the filter was applied. Keys that don't resolve to a column produce a single grouped warning so the agent knows the trace can't speak to whether they were honored. Wired across `find_studies`, `find_germplasm`, `find_observations`, `find_variables`, `find_variants`, `find_images`, and `find_locations`. `find_variants` had no post-hoc validator at all before this change — it now runs `checkFilterMatchRates` for the first time.
+- **`find_*` tools render the wire-shape filters block as `Filters sent to server` instead of `Applied filters`.** "Applied" suggested the upstream honored every key; "sent to server" is true regardless of honor and pairs cleanly with the new verification warnings. Updated label and JSDoc in `renderAppliedFilters` (function name kept for diff minimality — only the rendered string changes).

package/dist/config/builtin-aliases.d.ts

@@ -1,7 +1,7 @@
 /**
  * @fileoverview Built-in known-server registry. Curated list of public BrAPI v2
  * endpoints that ship as default connection aliases — operators can call
- * `brapi_connect({ alias: 'cassava' })` (or any other entry) without setting
+ * `brapi_connect({ alias: 'bti-cassava' })` (or any other entry) without setting
  * `BRAPI_<ALIAS>_BASE_URL`. Each entry carries CC-BY attribution metadata so
  * the orientation envelope can surface license + citation alongside the data.
  *

package/dist/config/builtin-aliases.js

@@ -1,7 +1,7 @@
 /**
  * @fileoverview Built-in known-server registry. Curated list of public BrAPI v2
  * endpoints that ship as default connection aliases — operators can call
- * `brapi_connect({ alias: 'cassava' })` (or any other entry) without setting
+ * `brapi_connect({ alias: 'bti-cassava' })` (or any other entry) without setting
  * `BRAPI_<ALIAS>_BASE_URL`. Each entry carries CC-BY attribution metadata so
  * the orientation envelope can surface license + citation alongside the data.
  *

package/dist/config/server-config.d.ts

@@ -1,7 +1,7 @@
 /**
  * @fileoverview Server-specific configuration for brapi-mcp-server. Covers
  * default connection details, HTTP client behavior (timeouts, retries,
- * concurrency), async-search polling, dataset lifecycle, and reference-data
+ * concurrency), async-search polling, dataframe lifecycle, and reference-data
  * cache tuning. Lazy-parsed via `parseEnvConfig` so env-var names appear in
  * validation errors.
  *
@@ -17,7 +17,6 @@ declare const ServerConfigSchema: z.ZodObject<{
     defaultApiKey: z.ZodOptional<z.ZodString>;
     defaultApiKeyHeader: z.ZodDefault<z.ZodString>;
     datasetTtlSeconds: z.ZodDefault<z.ZodCoercedNumber<unknown>>;
-    datasetStoreDir: z.ZodOptional<z.ZodString>;
     loadLimit: z.ZodDefault<z.ZodCoercedNumber<unknown>>;
     maxConcurrentRequests: z.ZodDefault<z.ZodCoercedNumber<unknown>>;
     retryMaxAttempts: z.ZodDefault<z.ZodCoercedNumber<unknown>>;
@@ -36,10 +35,11 @@ declare const ServerConfigSchema: z.ZodObject<{
         false: "false";
     }>>, z.ZodTransform<boolean, "true" | "false">>;
     genotypeCallsMaxPull: z.ZodDefault<z.ZodCoercedNumber<unknown>>;
-    canvasEnabled: z.ZodPipe<z.ZodDefault<z.ZodEnum<{
+    canvasDropEnabled: z.ZodPipe<z.ZodDefault<z.ZodEnum<{
         true: "true";
         false: "false";
     }>>, z.ZodTransform<boolean, "true" | "false">>;
+    exportDir: z.ZodOptional<z.ZodString>;
     canvasMaxRows: z.ZodDefault<z.ZodCoercedNumber<unknown>>;
     canvasQueryTimeoutMs: z.ZodDefault<z.ZodCoercedNumber<unknown>>;
 }, z.core.$strip>;

package/dist/config/server-config.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"server-config.d.ts","sourceRoot":"","sources":["../../src/config/server-config.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAEH,OAAO,EAAE,CAAC,EAAE,MAAM,wBAAwB,CAAC;AAG3C,QAAA,MAAM,kBAAkB;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;iBAwItB,CAAC;AAEH,MAAM,MAAM,YAAY,GAAG,CAAC,CAAC,KAAK,CAAC,OAAO,kBAAkB,CAAC,CAAC;AAI9D,wBAAgB,eAAe,IAAI,YAAY,CA4B9C;AAED,uDAAuD;AACvD,wBAAgB,iBAAiB,IAAI,IAAI,CAExC"}
+{"version":3,"file":"server-config.d.ts","sourceRoot":"","sources":["../../src/config/server-config.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAEH,OAAO,EAAE,CAAC,EAAE,MAAM,wBAAwB,CAAC;AAG3C,QAAA,MAAM,kBAAkB;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;iBA4ItB,CAAC;AAEH,MAAM,MAAM,YAAY,GAAG,CAAC,CAAC,KAAK,CAAC,OAAO,kBAAkB,CAAC,CAAC;AAI9D,wBAAgB,eAAe,IAAI,YAAY,CA4B9C;AAED,uDAAuD;AACvD,wBAAgB,iBAAiB,IAAI,IAAI,CAExC"}

package/dist/config/server-config.js

@@ -1,7 +1,7 @@
 /**
  * @fileoverview Server-specific configuration for brapi-mcp-server. Covers
  * default connection details, HTTP client behavior (timeouts, retries,
- * concurrency), async-search polling, dataset lifecycle, and reference-data
+ * concurrency), async-search polling, dataframe lifecycle, and reference-data
  * cache tuning. Lazy-parsed via `parseEnvConfig` so env-var names appear in
  * validation errors.
  *
@@ -38,17 +38,13 @@ const ServerConfigSchema = z.object({
         .int()
         .positive()
         .default(86_400)
-        .describe('TTL for datasets held in DatasetStore (seconds).'),
-    datasetStoreDir: z
-        .string()
-        .optional()
-        .describe('Filesystem path for DatasetStore payloads when filesystem storage is active.'),
+        .describe('TTL for dataframe provenance metadata persisted alongside spilled rows (seconds).'),
     loadLimit: z.coerce
         .number()
         .int()
         .positive()
-        .default(200)
-        .describe('Default row cap returned in-context before spilling to DatasetStore.'),
+        .default(1_000)
+        .describe('Default row cap returned in-context before spilling to a canvas dataframe. Also doubles as the upstream `pageSize` used during spillover walks — the dataframe ceiling is `loadLimit × MAX_SPILLOVER_PAGES (50)`, so the 1,000 default lines up with the documented 50,000-row spillover cap. Operators on small/test BrAPI servers can lower it; raising it above 1,000 is rarely useful because most BrAPI servers cap server-side pageSize at 1,000.'),
     maxConcurrentRequests: z.coerce
         .number()
         .int()
@@ -114,11 +110,15 @@ const ServerConfigSchema = z.object({
         .max(500_000)
         .default(100_000)
         .describe('Hard ceiling on rows pulled from the upstream BrAPI server in a single brapi_find_genotype_calls invocation. Bounds total page count per query — protects the upstream from unbounded pagination loops. Default 100,000 (≈10 pages at the standard pageSize=10,000); maximum 500,000 (≈50 pages, matching the per-query budget of other find_* tools).'),
-    canvasEnabled: z
+    canvasDropEnabled: z
         .enum(['true', 'false'])
-        .default('true')
+        .default('false')
         .transform((v) => v === 'true')
-        .describe('Gate the brapi_dataframe_* tool surface. Effective only when the framework canvas service is also configured (CANVAS_PROVIDER_TYPE=duckdb). Default true — set false to keep the dependency installed but the tools hidden.'),
+        .describe('Opt-in flag for `brapi_dataframe_drop`. Off by default — the tool is omitted from `tools/list` unless the operator opts in. Dataframes still expire via TTL when left unmanaged.'),
+    exportDir: z
+        .string()
+        .optional()
+        .describe('Filesystem directory for `brapi_dataframe_export` output files. When unset, the export tool is omitted from `tools/list` (no separate enable flag — setting the path *is* the opt-in). The framework reads the resolved path from `CANVAS_EXPORT_PATH`; the entry point bridges `BRAPI_EXPORT_DIR` → `CANVAS_EXPORT_PATH` so operators only need to set the brapi-prefixed name. Stdio-only — under `MCP_TRANSPORT_TYPE=http` the tool stays disabled regardless of this value because the returned path lives on the server, not the user.'),
     canvasMaxRows: z.coerce
         .number()
         .int()
@@ -143,7 +143,6 @@ export function getServerConfig() {
         defaultApiKey: 'BRAPI_DEFAULT_API_KEY',
         defaultApiKeyHeader: 'BRAPI_DEFAULT_API_KEY_HEADER',
         datasetTtlSeconds: 'BRAPI_DATASET_TTL_SECONDS',
-        datasetStoreDir: 'BRAPI_DATASET_STORE_DIR',
         loadLimit: 'BRAPI_LOAD_LIMIT',
         maxConcurrentRequests: 'BRAPI_MAX_CONCURRENT_REQUESTS',
         retryMaxAttempts: 'BRAPI_RETRY_MAX_ATTEMPTS',
@@ -156,7 +155,8 @@ export function getServerConfig() {
         allowPrivateIps: 'BRAPI_ALLOW_PRIVATE_IPS',
         enableWrites: 'BRAPI_ENABLE_WRITES',
         genotypeCallsMaxPull: 'BRAPI_GENOTYPE_CALLS_MAX_PULL',
-        canvasEnabled: 'BRAPI_CANVAS_ENABLED',
+        canvasDropEnabled: 'BRAPI_CANVAS_DROP_ENABLED',
+        exportDir: 'BRAPI_EXPORT_DIR',
         canvasMaxRows: 'BRAPI_CANVAS_MAX_ROWS',
         canvasQueryTimeoutMs: 'BRAPI_CANVAS_QUERY_TIMEOUT_MS',
     });

package/dist/config/server-config.js.map

@@ -1 +1 @@
-{"version":3,"file":"server-config.js","sourceRoot":"","sources":["../../src/config/server-config.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAEH,OAAO,EAAE,CAAC,EAAE,MAAM,wBAAwB,CAAC;AAC3C,OAAO,EAAE,cAAc,EAAE,MAAM,+BAA+B,CAAC;AAE/D,MAAM,kBAAkB,GAAG,CAAC,CAAC,MAAM,CAAC;IAClC,cAAc,EAAE,CAAC;SACd,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,QAAQ,CAAC,yEAAyE,CAAC;IACtF,eAAe,EAAE,CAAC;SACf,MAAM,EAAE;SACR,QAAQ,EAAE;SACV,QAAQ,CAAC,qDAAqD,CAAC;IAClE,eAAe,EAAE,CAAC;SACf,MAAM,EAAE;SACR,QAAQ,EAAE;SACV,QAAQ,CAAC,qDAAqD,CAAC;IAClE,oBAAoB,EAAE,CAAC;SACpB,MAAM,EAAE;SACR,QAAQ,EAAE;SACV,QAAQ,CAAC,uDAAuD,CAAC;IACpE,wBAAwB,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,+BAA+B,CAAC;IACzF,aAAa,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,yBAAyB,CAAC;IACxE,mBAAmB,EAAE,CAAC;SACnB,MAAM,EAAE;SACR,OAAO,CAAC,eAAe,CAAC;SACxB,QAAQ,CAAC,0CAA0C,CAAC;IAEvD,iBAAiB,EAAE,CAAC,CAAC,MAAM;SACxB,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,MAAM,CAAC;SACf,QAAQ,CAAC,kDAAkD,CAAC;IAC/D,eAAe,EAAE,CAAC;SACf,MAAM,EAAE;SACR,QAAQ,EAAE;SACV,QAAQ,CAAC,8EAA8E,CAAC;IAE3F,SAAS,EAAE,CAAC,CAAC,MAAM;SAChB,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,GAAG,CAAC;SACZ,QAAQ,CAAC,sEAAsE,CAAC;IACnF,qBAAqB,EAAE,CAAC,CAAC,MAAM;SAC5B,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,CAAC,CAAC;SACV,QAAQ,CAAC,+DAA+D,CAAC;IAC5E,gBAAgB,EAAE,CAAC,CAAC,MAAM;SACvB,MAAM,EAAE;SACR,GAAG,EAAE;SACL,GAAG,CAAC,CAAC,CAAC;SACN,OAAO,CAAC,CAAC,CAAC;SACV,QAAQ,CAAC,0CAA0C,CAAC;IACvD,gBAAgB,EAAE,CAAC,CAAC,MAAM;SACvB,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,GAAG,CAAC;SACZ,QAAQ,CAAC,8CAA8C,CAAC;IAC3D,wBAAwB,EAAE,CAAC,CAAC,MAAM;SAC/B,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,KAAK,CAAC;SACd,QAAQ,CAAC,0EAA0E,CAAC;IAEvF,gBAAgB,EAAE,CAAC,CAAC,MAAM;SACvB,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,MAAM,CAAC;SACf,QAAQ,CAAC,2BAA2B,CAAC;IACxC,kBAAkB,EAAE,CAAC,CAAC,MAAM;SACzB,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,KAAK,CAAC;SACd,QAAQ,CACP,4MAA4M,CAC7M;IACH,mBAAmB,EAAE,CAAC,CAAC,MAAM;SAC1B,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,MAAM,CAAC;SACf,QAAQ,CAAC,0DAA0D,CAAC;IACvE,oBAAoB,EAAE,CAAC,CAAC,MAAM;SAC3B,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,KAAK,CAAC;SACd,QAAQ,CAAC,mDAAmD,CAAC;IAChE,eAAe,EAAE,CAAC;SACf,IAAI,CAAC,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;SACvB,OAAO,CAAC,OAAO,CAAC;SAChB,SAAS,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,KAAK,MAAM,CAAC;SAC9B,QAAQ,CAAC,4DAA4D,CAAC;IACzE,YAAY,EAAE,CAAC;SACZ,IAAI,CAAC,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;SACvB,OAAO,CAAC,OAAO,CAAC;SAChB,SAAS,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,KAAK,MAAM,CAAC;SAC9B,QAAQ,CACP,0KAA0K,CAC3K;IACH,oBAAoB,EAAE,CAAC,CAAC,MAAM;SAC3B,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,GAAG,CAAC,OAAO,CAAC;SACZ,OAAO,CAAC,OAAO,CAAC;SAChB,QAAQ,CACP,wVAAwV,CACzV;IAEH,aAAa,EAAE,CAAC;SACb,IAAI,CAAC,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;SACvB,OAAO,CAAC,MAAM,CAAC;SACf,SAAS,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,KAAK,MAAM,CAAC;SAC9B,QAAQ,CACP,6NAA6N,CAC9N;IACH,aAAa,EAAE,CAAC,CAAC,MAAM;SACpB,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,MAAM,CAAC;SACf,QAAQ,CACP,4JAA4J,CAC7J;IACH,oBAAoB,EAAE,CAAC,CAAC,MAAM;SAC3B,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,MAAM,CAAC;SACf,QAAQ,CAAC,yDAAyD,CAAC;CACvE,CAAC,CAAC;AAIH,IAAI,OAAiC,CAAC;AAEtC,MAAM,UAAU,eAAe;IAC7B,OAAO,KAAK,cAAc,CAAC,kBAAkB,EAAE;QAC7C,cAAc,EAAE,wBAAwB;QACxC,eAAe,EAAE,wBAAwB;QACzC,eAAe,EAAE,wBAAwB;QACzC,oBAAoB,EAAE,+BAA+B;QACrD,wBAAwB,EAAE,mCAAmC;QAC7D,aAAa,EAAE,uBAAuB;QACtC,mBAAmB,EAAE,8BAA8B;QACnD,iBAAiB,EAAE,2BAA2B;QAC9C,eAAe,EAAE,yBAAyB;QAC1C,SAAS,EAAE,kBAAkB;QAC7B,qBAAqB,EAAE,+BAA+B;QACtD,gBAAgB,EAAE,0BAA0B;QAC5C,gBAAgB,EAAE,2BAA2B;QAC7C,wBAAwB,EAAE,mCAAmC;QAC7D,gBAAgB,EAAE,0BAA0B;QAC5C,kBAAkB,EAAE,4BAA4B;QAChD,mBAAmB,EAAE,8BAA8B;QACnD,oBAAoB,EAAE,+BAA+B;QACrD,eAAe,EAAE,yBAAyB;QAC1C,YAAY,EAAE,qBAAqB;QACnC,oBAAoB,EAAE,+BAA+B;QACrD,aAAa,EAAE,sBAAsB;QACrC,aAAa,EAAE,uBAAuB;QACtC,oBAAoB,EAAE,+BAA+B;KACtD,CAAC,CAAC;IACH,OAAO,OAAO,CAAC;AACjB,CAAC;AAED,uDAAuD;AACvD,MAAM,UAAU,iBAAiB;IAC/B,OAAO,GAAG,SAAS,CAAC;AACtB,CAAC"}
+{"version":3,"file":"server-config.js","sourceRoot":"","sources":["../../src/config/server-config.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAEH,OAAO,EAAE,CAAC,EAAE,MAAM,wBAAwB,CAAC;AAC3C,OAAO,EAAE,cAAc,EAAE,MAAM,+BAA+B,CAAC;AAE/D,MAAM,kBAAkB,GAAG,CAAC,CAAC,MAAM,CAAC;IAClC,cAAc,EAAE,CAAC;SACd,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,QAAQ,CAAC,yEAAyE,CAAC;IACtF,eAAe,EAAE,CAAC;SACf,MAAM,EAAE;SACR,QAAQ,EAAE;SACV,QAAQ,CAAC,qDAAqD,CAAC;IAClE,eAAe,EAAE,CAAC;SACf,MAAM,EAAE;SACR,QAAQ,EAAE;SACV,QAAQ,CAAC,qDAAqD,CAAC;IAClE,oBAAoB,EAAE,CAAC;SACpB,MAAM,EAAE;SACR,QAAQ,EAAE;SACV,QAAQ,CAAC,uDAAuD,CAAC;IACpE,wBAAwB,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,+BAA+B,CAAC;IACzF,aAAa,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,yBAAyB,CAAC;IACxE,mBAAmB,EAAE,CAAC;SACnB,MAAM,EAAE;SACR,OAAO,CAAC,eAAe,CAAC;SACxB,QAAQ,CAAC,0CAA0C,CAAC;IAEvD,iBAAiB,EAAE,CAAC,CAAC,MAAM;SACxB,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,MAAM,CAAC;SACf,QAAQ,CAAC,mFAAmF,CAAC;IAEhG,SAAS,EAAE,CAAC,CAAC,MAAM;SAChB,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,KAAK,CAAC;SACd,QAAQ,CACP,ybAAyb,CAC1b;IACH,qBAAqB,EAAE,CAAC,CAAC,MAAM;SAC5B,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,CAAC,CAAC;SACV,QAAQ,CAAC,+DAA+D,CAAC;IAC5E,gBAAgB,EAAE,CAAC,CAAC,MAAM;SACvB,MAAM,EAAE;SACR,GAAG,EAAE;SACL,GAAG,CAAC,CAAC,CAAC;SACN,OAAO,CAAC,CAAC,CAAC;SACV,QAAQ,CAAC,0CAA0C,CAAC;IACvD,gBAAgB,EAAE,CAAC,CAAC,MAAM;SACvB,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,GAAG,CAAC;SACZ,QAAQ,CAAC,8CAA8C,CAAC;IAC3D,wBAAwB,EAAE,CAAC,CAAC,MAAM;SAC/B,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,KAAK,CAAC;SACd,QAAQ,CAAC,0EAA0E,CAAC;IAEvF,gBAAgB,EAAE,CAAC,CAAC,MAAM;SACvB,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,MAAM,CAAC;SACf,QAAQ,CAAC,2BAA2B,CAAC;IACxC,kBAAkB,EAAE,CAAC,CAAC,MAAM;SACzB,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,KAAK,CAAC;SACd,QAAQ,CACP,4MAA4M,CAC7M;IACH,mBAAmB,EAAE,CAAC,CAAC,MAAM;SAC1B,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,MAAM,CAAC;SACf,QAAQ,CAAC,0DAA0D,CAAC;IACvE,oBAAoB,EAAE,CAAC,CAAC,MAAM;SAC3B,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,KAAK,CAAC;SACd,QAAQ,CAAC,mDAAmD,CAAC;IAChE,eAAe,EAAE,CAAC;SACf,IAAI,CAAC,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;SACvB,OAAO,CAAC,OAAO,CAAC;SAChB,SAAS,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,KAAK,MAAM,CAAC;SAC9B,QAAQ,CAAC,4DAA4D,CAAC;IACzE,YAAY,EAAE,CAAC;SACZ,IAAI,CAAC,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;SACvB,OAAO,CAAC,OAAO,CAAC;SAChB,SAAS,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,KAAK,MAAM,CAAC;SAC9B,QAAQ,CACP,0KAA0K,CAC3K;IACH,oBAAoB,EAAE,CAAC,CAAC,MAAM;SAC3B,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,GAAG,CAAC,OAAO,CAAC;SACZ,OAAO,CAAC,OAAO,CAAC;SAChB,QAAQ,CACP,wVAAwV,CACzV;IAEH,iBAAiB,EAAE,CAAC;SACjB,IAAI,CAAC,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;SACvB,OAAO,CAAC,OAAO,CAAC;SAChB,SAAS,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,KAAK,MAAM,CAAC;SAC9B,QAAQ,CACP,kLAAkL,CACnL;IACH,SAAS,EAAE,CAAC;SACT,MAAM,EAAE;SACR,QAAQ,EAAE;SACV,QAAQ,CACP,6gBAA6gB,CAC9gB;IACH,aAAa,EAAE,CAAC,CAAC,MAAM;SACpB,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,MAAM,CAAC;SACf,QAAQ,CACP,4JAA4J,CAC7J;IACH,oBAAoB,EAAE,CAAC,CAAC,MAAM;SAC3B,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,OAAO,CAAC,MAAM,CAAC;SACf,QAAQ,CAAC,yDAAyD,CAAC;CACvE,CAAC,CAAC;AAIH,IAAI,OAAiC,CAAC;AAEtC,MAAM,UAAU,eAAe;IAC7B,OAAO,KAAK,cAAc,CAAC,kBAAkB,EAAE;QAC7C,cAAc,EAAE,wBAAwB;QACxC,eAAe,EAAE,wBAAwB;QACzC,eAAe,EAAE,wBAAwB;QACzC,oBAAoB,EAAE,+BAA+B;QACrD,wBAAwB,EAAE,mCAAmC;QAC7D,aAAa,EAAE,uBAAuB;QACtC,mBAAmB,EAAE,8BAA8B;QACnD,iBAAiB,EAAE,2BAA2B;QAC9C,SAAS,EAAE,kBAAkB;QAC7B,qBAAqB,EAAE,+BAA+B;QACtD,gBAAgB,EAAE,0BAA0B;QAC5C,gBAAgB,EAAE,2BAA2B;QAC7C,wBAAwB,EAAE,mCAAmC;QAC7D,gBAAgB,EAAE,0BAA0B;QAC5C,kBAAkB,EAAE,4BAA4B;QAChD,mBAAmB,EAAE,8BAA8B;QACnD,oBAAoB,EAAE,+BAA+B;QACrD,eAAe,EAAE,yBAAyB;QAC1C,YAAY,EAAE,qBAAqB;QACnC,oBAAoB,EAAE,+BAA+B;QACrD,iBAAiB,EAAE,2BAA2B;QAC9C,SAAS,EAAE,kBAAkB;QAC7B,aAAa,EAAE,uBAAuB;QACtC,oBAAoB,EAAE,+BAA+B;KACtD,CAAC,CAAC;IACH,OAAO,OAAO,CAAC;AACjB,CAAC;AAED,uDAAuD;AACvD,MAAM,UAAU,iBAAiB;IAC/B,OAAO,GAAG,SAAS,CAAC;AACtB,CAAC"}