@barivia/barsom-mcp 0.23.4 → 0.23.8

package/README.md

@@ -32,7 +32,7 @@ MCP clients typically run it with **`npx`** (downloads on first use):
 1. **Get an API key** — Sign up at [barivia.se/dashboard](https://barivia.se/dashboard) (self-serve) or ask your admin for a `bv_…` key.
 2. **Add the MCP server** — Paste the JSON block above into your client’s MCP config (`mcp.json` in Cursor). Set `BARIVIA_API_KEY` and keep `BARIVIA_API_URL=https://api.barivia.se`.
 3. **Orient** — In chat, invoke **`guide_barsom_workflow`** (no parameters). It returns your plan-scoped tool map and async rules.
-4. **Upload data** — `datasets(action=upload, file_path="your.csv")` → note `dataset_id` (and `stage_job_id` if staging is enqueued; poll until complete).
+4. **Upload data** — `datasets(action=upload, file_path="your.csv")` → note `dataset_id`. The proxy waits for staging on upload; confirm with `datasets(action=get)` → `train_ready=yes` before train if you bypass the proxy wait.
 5. **Train** — `train(action=map, dataset_id=…, preset=quick)` → returns `job_id`. Poll with `jobs(action=status, job_id=…)` every 60–120s, or use **`training_monitor(job_id=…)`** for live curves.
 6. **Results** — When status is `completed`, `results(action=get, job_id=…)` for metrics + headline figure; **`results_explorer(job_id=…)`** to browse all figures.
 
@@ -53,10 +53,11 @@ MCP clients typically run it with **`npx`** (downloads on first use):
 | `BARIVIA_API_KEY` | Yes | -- | API key (starts with `bv_`) |
 | `BARIVIA_API_URL` | No | `https://api.barivia.se` | API base URL |
 | `BARIVIA_WORKSPACE_ROOT` | No | `process.cwd()` or `PWD` | Directory for relative `file_path` and `save_to_disk`. In Cursor MCP, `process.cwd()` is often the MCP install dir — add `BARIVIA_WORKSPACE_ROOT` to your MCP config `env` with your project path (e.g. `/home/user/myproject`). Absolute paths and `file://` URIs work without it. |
+| `BARIVIA_WORKSPACE_EXTRA_ROOTS` | No | (empty) | Comma- or colon-separated absolute directories whose **realpath** targets are also allowed for uploads when the sandbox is on. Use for shared fixture trees reached via symlinks under the case workspace (e.g. `…/barivia-testing/data`). Does not weaken the default sandbox for paths outside these roots. |
 | `BARIVIA_FETCH_TIMEOUT_MS` | No | `60000` | Per-request HTTP timeout (ms) to the API. Increase (e.g. `120000`) on slow networks or when the API does long-running work. Large dataset uploads use a separate longer timeout internally, and `datasets(analyze)` runs asynchronously (auto-polled) so it is not bound by this. A timeout is not auto-retried (re-firing slow work would only multiply load / risk duplicates). |
 | `BARIVIA_VIZ_PORT` | No | OS-assigned | Local viz server on `127.0.0.1`. Port is **per MCP session** unless pinned — standalone `…/viz/…` URLs go stale after proxy restart. Set a fixed port for stable bookmark URLs. Diagnostic: `GET http://127.0.0.1:<port>/api/health?job_id=<id>`. |
 | `BARIVIA_UI_DELIVERY` | No | `auto` | Client-capability override for App tools: `auto` \| `apps` \| `localhost` \| `inline`. Tool responses always lead with a standalone localhost URL (hosts may list `ui://` without mounting widgets) and include a structured `ui_delivery` block. `account(status)` reports detected Apps support, viz port, and active tier. |
-| `BARIVIA_ENFORCE_WORKSPACE_SANDBOX` | No | `1` (enabled) | File uploads are constrained to the MCP workspace root by default. Set to `0` or `false` to allow absolute paths anywhere on the machine (high trust). **Changed in v0.x:** previously defaulted to off; now on for security. If uploads fail with "paths outside the workspace are disabled", either set `BARIVIA_WORKSPACE_ROOT` to cover your data directory, or opt out with `BARIVIA_ENFORCE_WORKSPACE_SANDBOX=0`. |
+| `BARIVIA_ENFORCE_WORKSPACE_SANDBOX` | No | `1` (enabled) | File uploads are constrained to the MCP workspace root (and optional `BARIVIA_WORKSPACE_EXTRA_ROOTS`) by default. Set to `0` or `false` to allow absolute paths anywhere on the machine (high trust). If uploads fail with "symlink resolves outside workspace", either point `BARIVIA_WORKSPACE_ROOT` / `BARIVIA_WORKSPACE_EXTRA_ROOTS` at the real data dir, copy the file into the workspace, or opt out with `BARIVIA_ENFORCE_WORKSPACE_SANDBOX=0`. |
 
 Legacy `BARSOM_API_KEY` / `BARSOM_API_URL` / `BARSOM_WORKSPACE_ROOT` are also accepted as fallbacks.
 
@@ -102,7 +103,7 @@ Call at the **start of mapping work** (or when the user asks what the MCP can do
 | `preview` | Before training — inspect columns, stats, cyclic/datetime hints |
 | `analyze` | Pre-training recommendations (correlation, columns to consider dropping, etc.) |
 | `list` | Finding dataset IDs |
-| `get` | One dataset by id — status, staging fields, `stage_job_id`, ingest errors |
+| `get` | One dataset by id — status, **`train_ready`**, staging fields, `stage_job_id`, ingest errors |
 | `subset` | Creating a filtered/sliced copy (row_range, filter, `sample_n`) — **only when the user asks**; agents must not auto-downsample before train/analyze |
 | `add_expression` | Add a derived column from an expression (formula → new column on the dataset) |
 | `reduce_spectral` | Pre-training reducer for long ordered numeric blocks (spectra, time series, sensor fingerprints, gene panels). Methods: **pca** (top-k principal components), **log_sample** (k columns at log-spaced indices — scattering, audio bands), **uniform_sample** (k columns at evenly-spaced indices — regularly-sampled time series), **stats** (6 fixed per-row summary columns). All produce one feature vector per row; appends derived columns to the dataset. |
@@ -115,7 +116,7 @@ All training is submitted via the `train` tool; every action is async and return
 | Action | Use when |
 |--------|----------|
 | `map` | New map training job — full control: model type, grid, epochs, cyclic/temporal features, transforms. Complete-case data only (no NaNs in training columns). |
-| `impute` | Sparse training: missSOM / SIOM missSOM — map + `imputed.csv`. Supports **`transforms`**, **`temporal_features`**, **`cyclic_features`**, **`normalization_methods`** (zscore/mad/sigmoidal/none — **not** sepd or rank). Defaults: `model=auto`, `cv_folds=5`. Valid parent for `inference(impute_column)`. |
+| `impute` | Sparse training (SIOM entitlement): missSOM / SIOM missSOM — map + `imputed.csv`. Supports **`transforms`**, **`temporal_features`**, **`cyclic_features`**, **`normalization_methods`** (zscore/mad/sigmoidal/none — **not** sepd or rank). Defaults: `model=auto`, `cv_folds=5`. Valid parent for `inference(impute_column)`. |
 | `siom_map` | Self-interacting map — same grid-map workflow plus SIOM controls such as `gamma`, `siom_decay`, and penalty selection. |
 | `floop_siom` | FLooP-SIOM — growing node-budget manifold; default `topology=free` (CHL); optional `topology=chain` for strict 1D linked list. Requires Premium/Enterprise (`all_algorithms`). |
 
@@ -153,7 +154,7 @@ All actions use a frozen trained map — no retraining. Derived columns use **`d
 |--------|--------|--------|
 | `predict` | Score rows against the trained map. **Inputs:** `dataset_id` (defaults to the parent training dataset) **or** inline `rows` (≤500). **Output style** (`output` param): `"compact"` → `predictions.csv` (row_id, bmu_x/y, bmu_node_index, cluster_id [+ QE / qe_p95 / potential_anomaly when scoring **new** data]); `"annotated"` → `annotated.csv` (original CSV + BMU columns appended). **Regime auto-detected:** when the resolved dataset matches the training dataset, QE columns are intentionally omitted in compact output (training-set fit ≠ generalisation; the p95 anomaly flag would be circular). Prefer `dataset_id` for batches and SIOM/irregular maps. | 5–120s |
 | `batch_predict` | Fan out many predict jobs at once against one parent training `job_id`: pass `inputs` (array of `{ dataset_id? , rows? }`). Returns the list of submitted job_ids (does not auto-poll — poll each with `jobs(action=status)`). | submits N jobs |
-| `impute_column` | Fill a numeric **target_column** not used in training: **requires** `dataset_id` + `target_column`. Parent job: completed **train(action=map)** or **train(action=impute)**. Dataset must contain all training features plus the target. Pools observed target values from rows mapped to this row's BMU and topology neighbors (BMU + neighbors, often 7 nodes on hex interior; fewer on borders unless the map is periodic). `only_missing` (default true); `impute_aggregation`: mean or median. **Not** held-out validated — map-local estimate. Output **`imputed.csv`**. For holes across many training columns, prefer **train(action=impute)** first. | 5–120s |
+| `impute_column` | Fill a numeric **target_column** not used in training: **requires** `dataset_id` + `target_column`. Parent job: completed **train(action=map)** or **train(action=impute)** (impute needs SIOM entitlement). Dataset must contain all training features plus the target. Pools observed target values from rows mapped to this row's BMU and topology neighbors (BMU + neighbors, often 7 nodes on hex interior; fewer on borders unless the map is periodic). `only_missing` (default true); `impute_aggregation`: mean or median. **Not** held-out validated — map-local estimate. Output **`imputed.csv`**. For holes across many training columns, prefer **train(action=impute)** when entitled. | 5–120s |
 | `compare` | density-diff heatmap + top gained/lost nodes — drift, A/B, cohort | 30–120s |
 | `project_columns` | Project one or more dataset columns onto the trained map (component planes) | async |
 | `transition_flow` | Temporal state-transition arrows on the trained map (time-ordered rows only). Pass the training `job_id`; returns a new result `job_id` for `results(get/download)`. `lag`/`min_transitions`/`top_k` tune it. | 30–60s |
@@ -174,7 +175,7 @@ Capacity is shared LOCAL/GKE worker pools (no per-key cloud burst lease).
 | Tool | Role |
 |------|------|
 | `training_monitor` | Live QE + **panel** TE curves; auto-opened from `train()` submit when the host mounts Apps; cross-link to results explorer when finalize completes; always returns a standalone URL + `ui_delivery` |
-| `results_explorer` | Browse metrics and figures after training completes; cross-link back to training monitor; always returns a standalone URL + `ui_delivery` |
+| `results_explorer` | Interactive explorer with a **figure dropdown above the plot** (grouped Summary → Diagnostics → Components); sidebar metrics/highlights; cross-link to training monitor; always returns a standalone URL + `ui_delivery` |
 | `_fetch_figure` | **Host / App only** — Results Explorer invokes this for one raster figure; not for agent chat |
 
 Standalone localhost URLs are always first in tool content (see `BARIVIA_UI_DELIVERY` / **Local viz fallback**). Pages cross-link monitor ↔ results for the same `job_id`.
@@ -191,6 +192,7 @@ The right viewer depends on **(MCP App support)** **and** **(can the human reach
 
 ### Migration notes
 
+- **Results explorer figure dropdown (0.23.2):** `results_explorer` puts a grouped `<select>` above the plot (no scrolling the sidebar to switch figures), matching barmesh explorer UX. Sidebar keeps Metrics + Highlights only.
 - **UI delivery + train auto-open (0.22.4+):** `train()` returns `training_monitor` App structured content so supporting hosts open the monitor in the same turn. App tools emit structured `ui_delivery` (`delivery`, `urls`, `hint_for_agent`) and always put the standalone viz URL first. Env: `BARIVIA_UI_DELIVERY`, `BARIVIA_VIZ_PORT`. Cyclic cos/sin component PNGs are omitted from default `summary.files` / `figures=all` (encoding for training; panels still appear in `combined` / recolor); opt in with train param `viz_upload_cyclic_components=true`.
 - **Account surface (0.22.0, breaking):** `account(request_compute|compute_status|release_compute)` removed — AWS cloud burst leases are gone. Use `account(status|history|add_funds)`. Capacity is shared LOCAL/GKE pools.
 - **Structured results + train hints (0.21.x, non-breaking):**

package/dist/index.js

@@ -1,2 +1,2 @@
 #!/usr/bin/env node
-import{McpServer as e}from"@modelcontextprotocol/sdk/server/mcp.js";import{StdioServerTransport as t}from"@modelcontextprotocol/sdk/server/stdio.js";import{z as o}from"zod";import{getUiCapability as s,registerAppResource as r,RESOURCE_MIME_TYPE as n}from"@modelcontextprotocol/ext-apps/server";import{startVizServer as a}from"./viz-server.js";import{API_KEY as i,apiCall as l,apiRawCall as p,loadViewHtml as c,setVizPort as u,setClientSupportsMcpApps as d,CLIENT_VERSION as m}from"./shared.js";import{registerDatasetsTool as f}from"./tools/datasets.js";import{registerTrainTool as h}from"./tools/train.js";import{registerJobsTool as g,JOBS_DESCRIPTION_BASE as b}from"./tools/jobs.js";import{registerResultsTool as _}from"./tools/results.js";import{registerExploreMapTool as y,RESULTS_EXPLORER_URI as w}from"./tools/explore_map.js";import{registerAccountTool as v}from"./tools/account.js";import{registerInferenceTool as j}from"./tools/inference.js";import{registerGuideBarsomTool as k}from"./tools/guide_barsom.js";import{registerTrainingGuidanceTool as x}from"./tools/training_guidance.js";import{registerFeedbackTool as P}from"./tools/feedback.js";import{registerTrainingMonitorTool as A,TRAINING_MONITOR_URI as I}from"./tools/training_monitor.js";import{resolvePrepareTrainingPrompt as C}from"./prepare_training_prompt.js";i||(console.error("Error: BARIVIA_API_KEY not set. Set it in your MCP client config."),process.exit(1));const T=new e({name:"analytics-engine",version:m,instructions:'# Barivia Mapping Analytics Engine\n\nSelf-organizing map (SOM) analytics: project high-dimensional data to a 2D grid for clusters, gradients, and anomalies.\n\n## Workflow (short)\n\nUpload (`datasets(upload)`) → `datasets(preview)` and `datasets(analyze)` before train → submit one of `train(action=map)`, `train(action=siom_map)`, or `train(action=floop_siom)` (only if plan allows FLooP) → poll `jobs(status)` every 10–15s until `completed` → `results(get)` for metrics and figures (there is no separate analyze tool), then `results_explorer(job_id)` to browse the figures interactively. Then `jobs(compare)`, `results(download/recolor)`, or `inference` (predict / compare / project_columns / impute_column / transition_flow) as needed.\n\n**Full detail:** Call `guide_barsom_workflow` for plan-scoped tool map, training modes, async rules, optional MCP App UIs, and step-by-step SOP (from the Barivia API when online).\n\n## Tool taxonomy (one tool per stage — do not mix)\n\n| Stage | Tool(s) | Contract |\n|-------|---------|----------|\n| Data | `datasets` | upload, preview, analyze, list, **get** (status/staging), subset, add_expression, reduce_spectral, delete — ingest + feature engineering, no map |\n| Train submit | `train` | map, siom_map, impute (sparse → map + dense imputed.csv), floop_siom (entitled). Async, returns job_id. Submit only |\n| Lifecycle | `jobs` | status, list (slim + limit/cursor/status/job_type/has_results), update (label/description/tags), compare (**training runs**), cancel, delete. No training/scoring here |\n| Artifacts | `results` | get (default=combined only; figures="none" for metrics-only sweeps; figures="all" or results_explorer/download for every plot), export, download, recolor (async). Read/render completed jobs. Results catalog = `jobs(list, has_results=true)` |\n| Frozen-map ops | `inference` | predict (regime-aware; "compact"|"annotated"), batch_predict, impute_column (neighbor-pool fill for a non-training column), compare (cohort density-diff), project_columns, transition_flow (time-ordered rows only), report. No weight updates |\n| Account | `account` | status, history, **inventory** (markdown datasets+jobs overview), add_funds, **health** (API liveness via GET /health — no R2) |\n| Bootstrap | `guide_barsom_workflow`, `train`, `training_guidance`, `prepare_training` prompt | orientation, submit map/siom/impute/floop, parameter hints, narrative checklist (API/tier-scoped) |\n| Explore (UI) | `results_explorer`, `training_monitor` | richer UX over the same API; `jobs(status)`+`results(get)` still suffice headless |\n| Other | `send_feedback` | only after user agrees; queues locally if API/storage is down |\n\n### Avoid these mis-routes\n- `jobs(compare)` = metrics table across **training runs** (pick best in a sweep). `inference(compare)` = density-diff of one map vs a **second dataset** (drift/cohort). Different tools.\n- Post-training map analysis (predict, compare, project_columns, impute_column, **transition_flow**) is all **inference** — not `results`. `results` is presentation/artifacts only (`recolor` stays here: it re-renders, it doesn\'t touch data).\n- Project a quantity onto the map: existing columns → `inference(project_columns)`; a formula → `datasets(add_expression, project_onto_job=<job>)`.\n\n## Async pattern\n\n- **Manual poll:** Training submits return `job_id` immediately — poll `jobs(status)` every 10–15s. **Running is not failed**; large grids or FLooP-SIOM can take many minutes. `max_nodes` (FLooP) is a total node budget, not grid side length.\n- **Often auto-polled:** `inference` actions (incl. `transition_flow`) and `results(recolor)` may wait in-proxy; if you get a `job_id`, poll `jobs(status)` the same way.\n\nCredits: jobs consume compute credits; check `account(status)` before big runs. Per-request HTTP timeout defaults to 60s (`datasets(analyze)` is async + auto-polled, so it is not bound by it); on slow networks raise `BARIVIA_FETCH_TIMEOUT_MS`. Timeouts are NOT auto-retried (avoids duplicating slow/expensive work).\n\n## Org inventory recipe\n\nTo rebuild a durable markdown overview of an org (datasets + jobs):\n1. `account(action=inventory)` — preferred one-shot (datasets name/id/rows/cols/description/tags + slim recent jobs).\n2. Or compose: `datasets(list)` + `jobs(action=list, limit=50)` (page with `cursor` / filter with `status` / `job_type` / `has_results=true` for a results catalog).\n3. Set `description` + `tags` on `datasets(upload|update)` and `train(...)` / `jobs(update)` so duplicates stay navigable.\n\n## API briefly unavailable (pause, do not crash)\nIf tools return `api_unreachable` / gateway HTML / slim `error_code` lines: summarize in **one calm sentence** — never paste Cloudflare HTML. The proxy already burst-retried (**3** tries, ~**2s** apart); honor `retry_after_sec` (~**30**) before another round. Use `account(action=health)` (no R2). Local downloads and prior results stay usable; `send_feedback` queues under `~/.barivia/deferred-feedback` and flushes when the API returns. Do not start large uploads/trains until `account(status)` or health looks healthy again.\n\n## Constraints\n\n- **Do not auto-subset:** Never call `datasets(subset)` or `sample_n` unless the user explicitly asks. On CPU, `preset=quick` (15×15) is typically very fast up to ~100k rows — do **not** downsample by default when exploring below 100k; only suggest `sample_n` when n>100k or the user asks. Train/analyze the full `dataset_id`. Use small named fixtures only when the user or runbook names them.\n- **Small-n before train:** If preview/analyze shows n<1000 (especially n<100), heed the JUMPBACK/warning — omit `grid_x`/`grid_y` for auto (~sqrt(5·√n)), avoid oversized presets on tiny tables, drop redundant correlated geo features, and check `active_node_fraction`/TE before narrating map insights.\n- Prep ladder: `prepare_training` prompt = narrative checklist; `training_guidance` = structured hints (presets, resolved normalization/transforms, grid/epochs). Review these, then submit with `train`. Do not guess tiers or FLooP entitlement.\n- `inference(predict)`: prefer `dataset_id` for batch and for SIOM/irregular maps; single-row `rows` uses a fast path that can fail on some topologies — retry with `dataset_id`. FLooP-SIOM: if predict jobs fail while grid SIOM works, capture errors + `job_id`.\n- Column names are case-sensitive — match `datasets(preview)`.\n- Default training path is numeric/cyclic/temporal; use explicit `categorical_features` for baseline categoricals. `predict` must match the model contract.\n- After `results(recolor)`, `inference(transition_flow)`, or `inference(project_columns)`, use the **new** `job_id` returned for follow-up `results` if applicable.\n- **Large results / MCP timeouts:** For sweeps, `jobs(compare)`, or metrics-only reads use `results(action=get, figures="none")`. Default `get` inlines combined only. For every plot use `results_explorer` or `results(action=download)`, not `figures="all"` in chat unless the user wants all images inline.\n\n## UI delivery (heterogeneous MCP clients)\n\nHosts differ (Cursor, Claude Desktop, CLI agents): some embed MCP App panels; others list `ui://` but never mount them (expected on some Cursor builds); others are text-only. Tool results **always lead with a standalone localhost URL** — that is the intended fallback, not a broken App.\n\n| Tier | When | Agent should |\n|------|------|--------------|\n| **embedded** | Client advertises MCP Apps | Let the panel render **and** surface the standalone localhost URL from tool output / `ui_delivery.urls` (hosts often list Apps but do not mount). |\n| **localhost** | No MCP Apps; viz server running | Post the markdown link from tool output to the user (never broken workspace paths). Re-post after job completes if needed. |\n| **inline_image** | No Apps / override `inline` | Summarize metrics; inline raster (combined.png, learning curve) is attached automatically — show it in the reply. |\n| **text_only** | Viz server down | Use `jobs(status)` + `results(get)`; no panel or localhost link. |\n\nEvery `training_monitor` / `results_explorer` response includes a structured `ui_delivery` block (`delivery`, `urls`, `hint_for_agent`). Follow `hint_for_agent` without user prompting. Standalone viz pages cross-link monitor ↔ results for the same `job_id`.\n\n**Env overrides:** `BARIVIA_UI_DELIVERY=auto|apps|localhost|inline` (default `auto`). `BARIVIA_VIZ_PORT` pins the localhost viz port across proxy restarts. **Diagnostics:** `account(action=status)` reports detected MCP Apps support, viz port, and active tier.'});r(T,w,w,{mimeType:n},async()=>{const e=await c("results-explorer");return{contents:[{uri:w,mimeType:n,text:e??"<html><body>Results Explorer view not built yet. Run: npm run build:views</body></html>"}]}}),r(T,I,I,{mimeType:n},async()=>{const e=await c("training-monitor");return{contents:[{uri:I,mimeType:n,text:e??"<html><body>Training Monitor view not built yet.</body></html>"}]}}),k(T),y(T),A(T),f(T),h(T),g(T,b),_(T),v(T),j(T),x(T),P(T),T.prompt("info","Short orientation for the Barivia Mapping MCP. For full plan-scoped workflow, tool map, and SOP, the model should call guide_barsom_workflow. Use when the user asks what this MCP can do or how to get started.",{},()=>({messages:[{role:"user",content:{type:"text",text:["Give a concise, scannable answer (headers + bullets):","","**What it is:** MCP client to the Barivia mapping engine (2D SOM / SIOM / FLooP-SIOM when entitled) over HTTPS.","","**First step:** Call `guide_barsom_workflow` for plan-scoped bootstrap (full tool list, async rules, training modes, optional MCP Apps, SOP).","","**Core path:** `datasets(upload)` → `datasets(preview)` + `datasets(analyze)` → choose training action → poll `jobs(status)` every 10–15s until completed → `results(get)` (all main figures/metrics; no separate analyze tool) → `results_explorer(job_id)` to browse figures.","",'**Key tools:** `datasets` (data; reduce_spectral for spectra/long blocks), `train` (map/siom_map/impute/floop_siom; pass `label` for readable compare rows; preset=quick for a fast first map), `jobs` (status/list/compare/cancel/delete — lifecycle only), `results` (get/download/export/recolor; figures="none" for metrics-only), `inference` (predict; batch_predict; impute_column for topology-neighbor pool fill; compare; project_columns; transition_flow; report), `account` (status/credits/queue).',"","**Prep help:** `prepare_training` prompt (checklist) · `training_guidance` (presets/JSON hints + resolved normalization/params) — review, then submit with `train`.","","**Browse results:** `results_explorer(job_id)` is the preferred way to explore figures after a first `results(get)` glance; `training_monitor` shows live progress. Both optional — `results` + `jobs(status)` suffice headless.","","**After training:** `jobs(compare)` across runs, `results(recolor)`, `inference(project_columns)` for variables not in training, `inference(transition_flow)` only if rows are time-ordered.","","**Rules:** Running ≠ failed. Column names must match `datasets(preview)` exactly. Do not auto-subset below ~100k rows (CPU quick 15×15 is fast); never call `datasets(subset)` / `sample_n` unless the user asks or n>100k exploratory. Small-n (<1000): prefer auto grid, not oversized presets. Do not call `_fetch_figure` from chat (host/UI only); use `results(get)` or `results_explorer`.","","Offer `send_feedback` only after asking the user."].join("\n")}}]})),T.prompt("prepare_training","Narrative pre-training checklist (prompt). Use after upload and before train. Content is tier-scoped from the API when online. Prep ladder: this prompt = story checklist; training_guidance tool = JSON presets/parameter hints. Review, then submit with train.",{dataset_id:o.string().describe("Dataset ID to prepare for training")},async({dataset_id:e})=>{const t=await C(e);return{messages:[{role:"user",content:{type:"text",text:(t.used_fallback?`[offline fallback ${t.fallback_version}]\n\n`:"")+t.text}}]}});const M=new t;(async function(){try{const e=await a(l,p,c);u(e)}catch(e){process.env.BARIVIA_VIZ_PORT&&console.error("Barivia viz server failed to start:",e)}const e=T.server;e.oninitialized=()=>{const t=e.getClientCapabilities(),o=s(t);d(!!o?.mimeTypes?.includes(n))},await T.connect(M)})().catch(console.error);
+import{McpServer as e}from"@modelcontextprotocol/sdk/server/mcp.js";import{StdioServerTransport as t}from"@modelcontextprotocol/sdk/server/stdio.js";import{z as o}from"zod";import{getUiCapability as s,registerAppResource as r,RESOURCE_MIME_TYPE as n}from"@modelcontextprotocol/ext-apps/server";import{startVizServer as a}from"./viz-server.js";import{API_KEY as i,apiCall as l,apiRawCall as c,loadViewHtml as p,setVizPort as u,setClientSupportsMcpApps as d,CLIENT_VERSION as m}from"./shared.js";import{registerDatasetsTool as f}from"./tools/datasets.js";import{registerTrainTool as h}from"./tools/train.js";import{registerJobsTool as g,JOBS_DESCRIPTION_BASE as b}from"./tools/jobs.js";import{registerResultsTool as _}from"./tools/results.js";import{registerExploreMapTool as y,RESULTS_EXPLORER_URI as w}from"./tools/explore_map.js";import{registerAccountTool as v}from"./tools/account.js";import{registerInferenceTool as j}from"./tools/inference.js";import{registerGuideBarsomTool as k}from"./tools/guide_barsom.js";import{registerTrainingGuidanceTool as x}from"./tools/training_guidance.js";import{registerFeedbackTool as P}from"./tools/feedback.js";import{registerTrainingMonitorTool as I,TRAINING_MONITOR_URI as A}from"./tools/training_monitor.js";import{resolvePrepareTrainingPrompt as C}from"./prepare_training_prompt.js";i||(console.error("Error: BARIVIA_API_KEY not set. Set it in your MCP client config."),process.exit(1));const M=new e({name:"analytics-engine",version:m,instructions:'# Barivia Mapping Analytics Engine\n\nSelf-organizing map (SOM) analytics: project high-dimensional data to a 2D grid for clusters, gradients, and anomalies.\n\n## Workflow (short)\n\nUpload (`datasets(upload)`) → `datasets(preview)` and `datasets(analyze)` before train → submit one of `train(action=map)`, `train(action=siom_map)`, or `train(action=floop_siom)` (only if plan allows FLooP) → poll `jobs(status)` every 10–15s until `completed` → `results(get)` for metrics and figures (there is no separate analyze tool), then `results_explorer(job_id)` to browse the figures interactively. Then `jobs(compare)`, `results(download/recolor)`, or `inference` (predict / compare / project_columns / impute_column / transition_flow) as needed.\n\n**Full detail:** Call `guide_barsom_workflow` for plan-scoped tool map, training modes, async rules, optional MCP App UIs, and step-by-step SOP (from the Barivia API when online).\n\n## Tool taxonomy (one tool per stage — do not mix)\n\n| Stage | Tool(s) | Contract |\n|-------|---------|----------|\n| Data | `datasets` | upload, preview, analyze, list, **get** (status/staging), subset, add_expression, reduce_spectral, delete — ingest + feature engineering, no map |\n| Train submit | `train` | map; impute + siom_map (SIOM entitlement); floop_siom (FLooP entitlement). Async, returns job_id. Submit only |\n| Lifecycle | `jobs` | status, list (slim + limit/cursor/status/job_type/has_results), update (label/description/tags), compare (**training runs**), cancel, delete. No training/scoring here |\n| Artifacts | `results` | get (default=combined only; figures="none" for metrics-only sweeps; figures="all" or results_explorer/download for every plot), export, download, recolor (async). Read/render completed jobs. Results catalog = `jobs(list, has_results=true)` |\n| Frozen-map ops | `inference` | predict (regime-aware; "compact"|"annotated"), batch_predict, impute_column (neighbor-pool fill for a non-training column), compare (cohort density-diff), project_columns, transition_flow (time-ordered rows only), report. No weight updates |\n| Account | `account` | status, history, **inventory** (markdown datasets+jobs overview), add_funds, **health** (API liveness via GET /health — no R2) |\n| Bootstrap | `guide_barsom_workflow`, `train`, `training_guidance`, `prepare_training` prompt | orientation, submit map/siom/impute/floop, parameter hints, narrative checklist (API/tier-scoped) |\n| Explore (UI) | `results_explorer`, `training_monitor` | richer UX over the same API; `jobs(status)`+`results(get)` still suffice headless |\n| Other | `send_feedback` | only after user agrees; queues locally if API/storage is down |\n\n### Avoid these mis-routes\n- `jobs(compare)` = metrics table across **training runs** (pick best in a sweep). `inference(compare)` = density-diff of one map vs a **second dataset** (drift/cohort). Different tools.\n- Post-training map analysis (predict, compare, project_columns, impute_column, **transition_flow**) is all **inference** — not `results`. `results` is presentation/artifacts only (`recolor` stays here: it re-renders, it doesn\'t touch data).\n- Project a quantity onto the map: existing columns → `inference(project_columns)`; a formula → `datasets(add_expression, project_onto_job=<job>)`.\n\n## Async pattern\n\n- **Manual poll:** Training submits return `job_id` immediately — poll `jobs(status)` every 10–15s. **Running is not failed**; large grids or FLooP-SIOM can take many minutes. `max_nodes` (FLooP) is a total node budget, not grid side length.\n- **Often auto-polled:** `inference` actions (incl. `transition_flow`) and `results(recolor)` may wait in-proxy; if you get a `job_id`, poll `jobs(status)` the same way.\n\nCredits: jobs consume compute credits; check `account(status)` before big runs. Per-request HTTP timeout defaults to 60s (`datasets(analyze)` is async + auto-polled, so it is not bound by it); on slow networks raise `BARIVIA_FETCH_TIMEOUT_MS`. Timeouts are NOT auto-retried (avoids duplicating slow/expensive work).\n\n## Org inventory recipe\n\nTo rebuild a durable markdown overview of an org (datasets + jobs):\n1. `account(action=inventory)` — preferred one-shot (datasets name/id/rows/cols/description/tags + slim recent jobs).\n2. Or compose: `datasets(list)` + `jobs(action=list, limit=50)` (page with `cursor` / filter with `status` / `job_type` / `has_results=true` for a results catalog).\n3. Set `description` + `tags` on `datasets(upload|update)` and `train(...)` / `jobs(update)` so duplicates stay navigable.\n\n## API briefly unavailable (pause, do not crash)\nIf tools return `api_unreachable` / gateway HTML / slim `error_code` lines: summarize in **one calm sentence** — never paste Cloudflare HTML. The proxy already burst-retried (**3** tries, ~**2s** apart); honor `retry_after_sec` (~**30**) before another round. Use `account(action=health)` (no R2). Local downloads and prior results stay usable; `send_feedback` queues under `~/.barivia/deferred-feedback` and flushes when the API returns. Do not start large uploads/trains until `account(status)` or health looks healthy again.\n\n## Constraints\n\n- **Do not auto-subset:** Never call `datasets(subset)` or `sample_n` unless the user explicitly asks for a filter, slice, or random sample. Train and analyze on the uploaded `dataset_id` at full row count — GPU staging/prepare handles hundreds of thousands of rows quickly; silent downsampling hides real prepare/train behavior and wastes scale demos. Use small named fixtures (e.g. sample.csv, periodic_load) only when the user or runbook names them.\n- Prep ladder: `prepare_training` prompt = narrative checklist; `training_guidance` = structured hints (presets, resolved normalization/transforms, grid/epochs). Review these, then submit with `train`. Do not guess tiers or FLooP entitlement.\n- `inference(predict)`: prefer `dataset_id` for batch and for SIOM/irregular maps; single-row `rows` uses a fast path that can fail on some topologies — retry with `dataset_id`. FLooP-SIOM: if predict jobs fail while grid SIOM works, capture errors + `job_id`.\n- Column names are case-sensitive — match `datasets(preview)`.\n- Default training path is numeric/cyclic/temporal; use explicit `categorical_features` for baseline categoricals. `predict` must match the model contract.\n- After `results(recolor)`, `inference(transition_flow)`, or `inference(project_columns)`, use the **new** `job_id` returned for follow-up `results` if applicable.\n- **Large results / MCP timeouts:** For sweeps, `jobs(compare)`, or metrics-only reads use `results(action=get, figures="none")`. Default `get` inlines combined only. For every plot use `results_explorer` or `results(action=download)`, not `figures="all"` in chat unless the user wants all images inline.\n\n## UI delivery (heterogeneous MCP clients)\n\nHosts differ (Cursor, Claude Desktop, CLI agents): some embed MCP App panels; others list `ui://` but never mount them (expected on some Cursor builds); others are text-only. Tool results **always lead with a standalone localhost URL** — that is the intended fallback, not a broken App.\n\n| Tier | When | Agent should |\n|------|------|--------------|\n| **embedded** | Client advertises MCP Apps | Let the panel render **and** surface the standalone localhost URL from tool output / `ui_delivery.urls` (hosts often list Apps but do not mount). |\n| **localhost** | No MCP Apps; viz server running | Post the markdown link from tool output to the user (never broken workspace paths). Re-post after job completes if needed. |\n| **inline_image** | No Apps / override `inline` | Summarize metrics; inline raster (combined.png, learning curve) is attached automatically — show it in the reply. |\n| **text_only** | Viz server down | Use `jobs(status)` + `results(get)`; no panel or localhost link. |\n\nEvery `training_monitor` / `results_explorer` response includes a structured `ui_delivery` block (`delivery`, `urls`, `hint_for_agent`). Follow `hint_for_agent` without user prompting. Standalone viz pages cross-link monitor ↔ results for the same `job_id`.\n\n**Env overrides:** `BARIVIA_UI_DELIVERY=auto|apps|localhost|inline` (default `auto`). `BARIVIA_VIZ_PORT` pins the localhost viz port across proxy restarts. **Diagnostics:** `account(action=status)` reports detected MCP Apps support, viz port, and active tier.'});r(M,w,w,{mimeType:n},async()=>{const e=await p("results-explorer");return{contents:[{uri:w,mimeType:n,text:e??"<html><body>Results Explorer view not built yet. Run: npm run build:views</body></html>"}]}}),r(M,A,A,{mimeType:n},async()=>{const e=await p("training-monitor");return{contents:[{uri:A,mimeType:n,text:e??"<html><body>Training Monitor view not built yet.</body></html>"}]}}),k(M),y(M),I(M),f(M),h(M),g(M,b),_(M),v(M),j(M),x(M),P(M),M.prompt("info","Short orientation for the Barivia Mapping MCP. For full plan-scoped workflow, tool map, and SOP, the model should call guide_barsom_workflow. Use when the user asks what this MCP can do or how to get started.",{},()=>({messages:[{role:"user",content:{type:"text",text:["Give a concise, scannable answer (headers + bullets):","","**What it is:** MCP client to the Barivia mapping engine (2D SOM / SIOM / FLooP-SIOM when entitled) over HTTPS.","","**First step:** Call `guide_barsom_workflow` for plan-scoped bootstrap (full tool list, async rules, training modes, optional MCP Apps, SOP).","","**Core path:** `datasets(upload)` → `datasets(preview)` + `datasets(analyze)` → choose training action → poll `jobs(status)` every 10–15s until completed → `results(get)` (all main figures/metrics; no separate analyze tool) → `results_explorer(job_id)` to browse figures.","",'**Key tools:** `datasets` (data; reduce_spectral for spectra/long blocks), `train` (map; siom_map/impute with SIOM entitlement; floop_siom when entitled; pass `label` for readable compare rows; preset=quick for a fast first map), `jobs` (status/list/compare/cancel/delete — lifecycle only), `results` (get/download/export/recolor; figures="none" for metrics-only), `inference` (predict; batch_predict; impute_column for topology-neighbor pool fill; compare; project_columns; transition_flow; report), `account` (status/credits/queue).',"","**Prep help:** `prepare_training` prompt (checklist) · `training_guidance` (presets/JSON hints + resolved normalization/params) — review, then submit with `train`.","","**Browse results:** `results_explorer(job_id)` is the preferred way to explore figures after a first `results(get)` glance; `training_monitor` shows live progress. Both optional — `results` + `jobs(status)` suffice headless.","","**After training:** `jobs(compare)` across runs, `results(recolor)`, `inference(project_columns)` for variables not in training, `inference(transition_flow)` only if rows are time-ordered.","","**Rules:** Running ≠ failed. Column names must match `datasets(preview)` exactly. Do not auto-subset — never call `datasets(subset)` / `sample_n` unless the user explicitly asks; train on the full uploaded `dataset_id`. Do not call `_fetch_figure` from chat (host/UI only); use `results(get)` or `results_explorer`.","","Offer `send_feedback` only after asking the user."].join("\n")}}]})),M.prompt("prepare_training","Narrative pre-training checklist (prompt). Use after upload and before train. Content is tier-scoped from the API when online. Prep ladder: this prompt = story checklist; training_guidance tool = JSON presets/parameter hints. Review, then submit with train.",{dataset_id:o.string().describe("Dataset ID to prepare for training")},async({dataset_id:e})=>{const t=await C(e);return{messages:[{role:"user",content:{type:"text",text:(t.used_fallback?`[offline fallback ${t.fallback_version}]\n\n`:"")+t.text}}]}});const T=new t;(async function(){try{const e=await a(l,c,p);u(e)}catch(e){process.env.BARIVIA_VIZ_PORT&&console.error("Barivia viz server failed to start:",e)}const e=M.server;e.oninitialized=()=>{const t=e.getClientCapabilities(),o=s(t);d(!!o?.mimeTypes?.includes(n))},await M.connect(T)})().catch(console.error);

package/dist/job_monitor.js

@@ -34,14 +34,30 @@ function tailOrderingErrors(data, n = 4) {
 export function snapshotFromJob(data, elapsedSec, note) {
     const status = String(data.status ?? "unknown");
     const progress_pct = (data.progress ?? 0) * 100;
+    const kernelComplete = data.kernel_complete === true;
+    const awaitingFinalize = kernelComplete &&
+        status !== "completed" &&
+        status !== "failed" &&
+        status !== "cancelled";
     const snap = {
         elapsed_sec: elapsedSec,
         status,
-        progress_pct: Math.round(progress_pct * 10) / 10,
+        // Kernel progress can hit 100% before finalize; do not present as job-done.
+        progress_pct: awaitingFinalize
+            ? Math.min(Math.round(progress_pct * 10) / 10, 99.0)
+            : Math.round(progress_pct * 10) / 10,
     };
-    const phase = str(data, "progress_phase");
-    if (phase)
+    const phaseRaw = str(data, "progress_phase");
+    const phase = phaseRaw && phaseRaw !== "missing" ? phaseRaw : undefined;
+    if (awaitingFinalize) {
+        snap.phase = phase && phase !== "complete" ? `${phase}+awaiting_finalize` : "awaiting_finalize";
+    }
+    else if (phase) {
         snap.phase = phase;
+    }
+    else if (str(data, "prepare_job_id") && String(data.status ?? "") !== "completed") {
+        snap.phase = "waiting_for_prepare";
+    }
     const epoch = num(data, "epoch");
     const total = num(data, "total_epochs");
     if (epoch != null)
@@ -57,7 +73,7 @@ export function snapshotFromJob(data, elapsedSec, note) {
     const panelTe = num(data, "panel_topographic_error");
     if (panelTe != null)
         snap.panel_te = Math.round(panelTe * 10_000) / 10_000;
-    if (data.kernel_complete === true)
+    if (kernelComplete)
         snap.kernel_complete = true;
     const failureStage = str(data, "failure_stage");
     if (failureStage)
@@ -68,8 +84,12 @@ export function snapshotFromJob(data, elapsedSec, note) {
     const tail = tailOrderingErrors(data);
     if (tail && tail.length > 0)
         snap.ordering_errors_tail = tail;
-    if (note)
+    if (note) {
         snap.note = note;
+    }
+    else if (awaitingFinalize) {
+        snap.note = "kernel complete — awaiting finalize (not fully done)";
+    }
     return snap;
 }
 /** True when a new snapshot is worth recording (phase/epoch/progress/status change). */
@@ -95,7 +115,10 @@ export function shouldRecordSnapshot(prev, next) {
     return false;
 }
 export function formatSnapshotLine(s) {
-    const parts = [`[+${s.elapsed_sec}s] ${s.status} ${s.progress_pct.toFixed(1)}%`];
+    const progressLabel = s.kernel_complete && s.status !== "completed" && s.status !== "failed" && s.status !== "cancelled"
+        ? `kernel ${s.progress_pct.toFixed(1)}% (awaiting finalize)`
+        : `${s.progress_pct.toFixed(1)}%`;
+    const parts = [`[+${s.elapsed_sec}s] ${s.status} ${progressLabel}`];
     if (s.phase)
         parts.push(`phase ${s.phase}`);
     if (s.epoch != null && s.total_epochs != null)
@@ -106,8 +129,9 @@ export function formatSnapshotLine(s) {
         parts.push(`Epoch TE ${s.te.toFixed(4)}`);
     if (s.panel_te != null)
         parts.push(`Panel TE ${s.panel_te.toFixed(4)}`);
-    if (s.kernel_complete)
+    if (s.kernel_complete && (s.status === "completed" || s.status === "failed" || s.status === "cancelled")) {
         parts.push("kernel complete");
+    }
     if (s.failure_stage)
         parts.push(`failure_stage ${s.failure_stage}`);
     if (s.eta_sec != null)

package/dist/job_status_format.js

@@ -36,22 +36,41 @@ export function formatJobStatusText(job_id, data) {
     const progress = (data.progress ?? 0) * 100;
     const label = data.label != null && data.label !== "" ? String(data.label) : null;
     const jobDesc = label ? `Job ${label} (id: ${job_id})` : `Job ${job_id}`;
-    const parts = [`${jobDesc}: ${status} (${progress.toFixed(1)}%)`];
+    const kernelComplete = data.kernel_complete === true;
+    const awaitingFinalize = kernelComplete && status !== "completed" && status !== "failed" && status !== "cancelled";
+    const progressBit = awaitingFinalize
+        ? `kernel ${Math.min(progress, 99).toFixed(1)}% — awaiting finalize`
+        : `${progress.toFixed(1)}%`;
+    const parts = [`${jobDesc}: ${status} (${progressBit})`];
     const attempt = data.attempt != null ? Number(data.attempt) : null;
     if (attempt != null && attempt > 1) {
         parts.push(`attempt ${attempt} (job was requeued after worker timeout — not stuck from scratch)`);
     }
-    const phase = data.progress_phase != null && String(data.progress_phase) !== ""
+    const rawPhase = data.progress_phase != null && String(data.progress_phase) !== ""
         ? String(data.progress_phase)
         : null;
-    if (status === "running") {
+    // API historically serialized SQL NULL as the literal "missing" (Julia missing).
+    const phase = rawPhase && rawPhase !== "missing" ? rawPhase : null;
+    const prepareJobId = data.prepare_job_id != null && String(data.prepare_job_id) !== ""
+        ? String(data.prepare_job_id)
+        : null;
+    if (status === "pending" && prepareJobId) {
+        parts.push(`phase: waiting_for_prepare (prepare_job_id=${prepareJobId})`);
+    }
+    else if (status === "running") {
         if (phase) {
             parts.push(`phase: ${phase}`);
         }
+        else if (prepareJobId) {
+            parts.push(`phase: waiting_for_prepare (prepare_job_id=${prepareJobId})`);
+        }
         else {
             parts.push(`phase: preprocessing (not set yet — often download/parse before first progress tick)`);
         }
     }
+    else if (prepareJobId && (status === "completed" || status === "failed")) {
+        parts.push(`prepare_job_id: ${prepareJobId}`);
+    }
     const startedAt = data.started_at != null ? String(data.started_at) : null;
     if (startedAt) {
         parts.push(`started_at: ${startedAt}`);
@@ -78,7 +97,10 @@ export function formatJobStatusText(job_id, data) {
         if (panelTe != null) {
             parts.push(`panel_te: ${panelTe.toFixed(4)}`);
         }
-        if (data.kernel_complete === true) {
+        if (awaitingFinalize) {
+            parts.push("kernel_complete — figures/finalize still pending (not fully done)");
+        }
+        else if (kernelComplete) {
             parts.push("kernel_complete");
         }
     }

package/dist/shared.js

@@ -39,7 +39,7 @@ function newRequestId() {
  * X-Barsom-Client-Version so the server can annotate tool guidance with the
  * wrapper version each action requires. Keep in sync with package.json on bump.
  */
-export const CLIENT_VERSION = "0.23.4";
+export const CLIENT_VERSION = "0.23.8";
 /** User-facing links; keep aligned with barivia.se / api.barivia.se. */
 export const PUBLIC_SITE_ORIGIN = "https://barivia.se";
 /** Self-serve account dashboard (manage plan, billing, and API keys). */
@@ -77,6 +77,97 @@ export const LARGE_UPLOAD_BYTES = 64 * 1024 * 1024; // 64 MB
 export const PRESIGNED_PUT_TIMEOUT_MS = 30 * 60_000; // 30 min
 /** Poll window for the async stage_dataset job (large files take minutes). */
 export const POLL_STAGE_MAX_MS = 30 * 60_000; // 30 min
+/** Normalize upload JSON after staging completes — never leave staging:true with train_ready. */
+export function finalizeStagedUploadResponse(data, ready) {
+    const merged = ready
+        ? { ...data, ...ready }
+        : { ...data };
+    const trainReady = merged.train_ready === true || String(merged.status ?? "") === "ready";
+    if (trainReady) {
+        merged.status = ready?.status ?? merged.status ?? "ready";
+        merged.train_ready = true;
+        merged.staging = false;
+        // stage_job_id is only useful while staging is pending
+        if (ready?.stage_job_id == null)
+            delete merged.stage_job_id;
+    }
+    return merged;
+}
+/**
+ * After a small POST /v1/datasets upload, poll stage_job_id until the dataset is
+ * train-ready (mirrors the large-upload finalize path).
+ */
+export async function awaitDatasetStageIfNeeded(data) {
+    const stageJobId = data.stage_job_id != null ? String(data.stage_job_id) : "";
+    if (!stageJobId) {
+        if (data.train_ready === true || data.status === "ready") {
+            return finalizeStagedUploadResponse(data);
+        }
+        return data;
+    }
+    if (data.train_ready === true || data.status === "ready") {
+        return finalizeStagedUploadResponse(data);
+    }
+    const poll = await pollUntilComplete(stageJobId, POLL_STAGE_MAX_MS);
+    if (poll.status === "failed") {
+        throw new Error(`Dataset staging failed (stage_job_id=${stageJobId}): ${poll.error ?? "unknown error"}`);
+    }
+    if (poll.status === "timeout") {
+        throw new Error(`Dataset staging timed out after ${POLL_STAGE_MAX_MS / 60000} min (stage_job_id=${stageJobId}). Poll jobs(action=status, job_id="${stageJobId}").`);
+    }
+    const datasetId = String(data.id ?? data.dataset_id ?? "");
+    if (datasetId) {
+        try {
+            const ready = (await apiCall("GET", `/v1/datasets/${datasetId}`));
+            return finalizeStagedUploadResponse(data, ready);
+        }
+        catch {
+            /* fall through */
+        }
+    }
+    return finalizeStagedUploadResponse({ ...data, status: "ready", train_ready: true });
+}
+/**
+ * Before train / CFD submit: ensure the dataset has finished staging.
+ * Prefer train_ready from GET; otherwise poll stage_job_id.
+ */
+export async function ensureDatasetTrainReady(datasetId) {
+    const ds = (await apiCall("GET", `/v1/datasets/${datasetId}`));
+    if (ds.train_ready === true)
+        return;
+    if (String(ds.status) === "failed") {
+        const err = ds.ingest_error != null ? String(ds.ingest_error) : "staging failed";
+        throw new Error(`Dataset ${datasetId} staging failed: ${err}`);
+    }
+    const stageJobId = ds.stage_job_id != null ? String(ds.stage_job_id) : "";
+    if (stageJobId) {
+        const poll = await pollUntilComplete(stageJobId, POLL_STAGE_MAX_MS);
+        if (poll.status === "failed") {
+            throw new Error(`Dataset staging failed (stage_job_id=${stageJobId}): ${poll.error ?? "unknown error"}`);
+        }
+        if (poll.status === "timeout") {
+            throw new Error(`Dataset staging timed out (stage_job_id=${stageJobId}). Poll jobs(action=status, job_id="${stageJobId}").`);
+        }
+        return;
+    }
+    // No stage job listed — brief poll on GET for race windows
+    const deadline = Date.now() + 60_000;
+    while (Date.now() < deadline) {
+        await new Promise((r) => setTimeout(r, 1000));
+        const again = (await apiCall("GET", `/v1/datasets/${datasetId}`));
+        if (again.train_ready === true)
+            return;
+        if (String(again.status) === "failed") {
+            const err = again.ingest_error != null ? String(again.ingest_error) : "staging failed";
+            throw new Error(`Dataset ${datasetId} staging failed: ${err}`);
+        }
+        if (again.stage_job_id != null) {
+            await ensureDatasetTrainReady(datasetId);
+            return;
+        }
+    }
+    throw new Error(`Dataset ${datasetId} is not train_ready (status=${ds.status ?? "?"}). Wait for staging or poll datasets(action=get).`);
+}
 export const BARSOM_VIZ_VIEWS = {
     trainingMonitor: "training-monitor",
     resultsExplorer: "results-explorer",
@@ -288,11 +379,51 @@ function enforceWorkspaceSandboxUpload() {
         return false;
     return true;
 }
+/** Extra realpath roots allowed for uploads (comma- or colon-separated absolute paths). */
+export function getWorkspaceExtraRoots() {
+    const raw = process.env.BARIVIA_WORKSPACE_EXTRA_ROOTS ?? "";
+    if (!raw.trim())
+        return [];
+    return raw
+        .split(/[,:]/)
+        .map((s) => s.trim())
+        .filter(Boolean)
+        .map((s) => path.resolve(s));
+}
+async function realpathIfExists(p) {
+    try {
+        return await fs.realpath(p);
+    }
+    catch {
+        return null;
+    }
+}
+/** True when `realPath` is under workspace root or any BARIVIA_WORKSPACE_EXTRA_ROOTS entry. */
+export async function isPathUnderSandboxRoots(realPath, workspaceRoot) {
+    const candidates = [workspaceRoot, ...getWorkspaceExtraRoots()];
+    for (const root of candidates) {
+        const realRoot = await realpathIfExists(root);
+        if (!realRoot)
+            continue;
+        if (realPath === realRoot || realPath.startsWith(realRoot + path.sep))
+            return true;
+    }
+    return false;
+}
+function sandboxEscapeError(realPath, workspaceRoot) {
+    const extras = getWorkspaceExtraRoots();
+    const extraHint = extras.length > 0
+        ? ` Also checked BARIVIA_WORKSPACE_EXTRA_ROOTS (${extras.join(", ")}).`
+        : " Set BARIVIA_WORKSPACE_EXTRA_ROOTS to allow shared fixture dirs (e.g. a data/ sibling), or copy the file into the workspace.";
+    return new Error(`Resolved path escapes workspace — symlink resolves outside BARIVIA_WORKSPACE_ROOT` +
+        ` (${workspaceRoot}). Target: ${realPath}.${extraHint}`);
+}
 /**
  * Resolves file_path for dataset upload. Security: auth before read, generic errors, reject "..".
  * Accepts: absolute paths, file:// URIs, relative paths (resolved against workspace root).
  * Pre-security-patch behavior: absolute paths worked without BARIVIA_WORKSPACE_ROOT.
- * Set BARIVIA_ENFORCE_WORKSPACE_SANDBOX=1 to require absolute paths to lie under the resolved workspace root.
+ * Set BARIVIA_ENFORCE_WORKSPACE_SANDBOX=1 to require absolute paths to lie under the resolved workspace root
+ * or an entry in BARIVIA_WORKSPACE_EXTRA_ROOTS (after realpath, so symlinks are allowed when the target is under an allowed root).
  */
 export async function resolveFilePathForUpload(filePath, mcpServer) {
     const trimmed = filePath.trim();
@@ -309,9 +440,8 @@ export async function resolveFilePathForUpload(filePath, mcpServer) {
             if (enforceWorkspaceSandboxUpload()) {
                 const root = await getWorkspaceRootAsync(mcpServer);
                 const real = await fs.realpath(p);
-                const realRoot = await fs.realpath(root);
-                if (real !== realRoot && !real.startsWith(realRoot + path.sep)) {
-                    throw new Error(`file:// paths outside the workspace are disabled (BARIVIA_ENFORCE_WORKSPACE_SANDBOX). Workspace: ${realRoot}`);
+                if (!(await isPathUnderSandboxRoots(real, root))) {
+                    throw sandboxEscapeError(real, root);
                 }
                 const stat = await fs.stat(real);
                 if (!stat.isFile()) {
@@ -324,7 +454,7 @@ export async function resolveFilePathForUpload(filePath, mcpServer) {
         catch (err) {
             if (err instanceof Error && err.message.includes("remote hosts"))
                 throw err;
-            if (err instanceof Error && err.message.includes("BARIVIA_ENFORCE_WORKSPACE_SANDBOX"))
+            if (err instanceof Error && err.message.includes("symlink resolves outside"))
                 throw err;
             if (err instanceof Error && err.message.includes("regular file"))
                 throw err;
@@ -339,11 +469,10 @@ export async function resolveFilePathForUpload(filePath, mcpServer) {
                 real = await fs.realpath(trimmed);
             }
             catch {
-                throw new Error(`File not accessible at absolute path. With BARIVIA_ENFORCE_WORKSPACE_SANDBOX, use a path under the workspace root (${root}).`);
+                throw new Error(`File not accessible at absolute path. With BARIVIA_ENFORCE_WORKSPACE_SANDBOX, use a path under the workspace root (${root}) or BARIVIA_WORKSPACE_EXTRA_ROOTS.`);
             }
-            const realRoot = await fs.realpath(root);
-            if (real !== realRoot && !real.startsWith(realRoot + path.sep)) {
-                throw new Error(`Absolute paths outside the workspace are disabled (BARIVIA_ENFORCE_WORKSPACE_SANDBOX). Workspace root: ${realRoot}`);
+            if (!(await isPathUnderSandboxRoots(real, root))) {
+                throw sandboxEscapeError(real, root);
             }
             const stat = await fs.stat(real);
             if (!stat.isFile()) {
@@ -355,14 +484,14 @@ export async function resolveFilePathForUpload(filePath, mcpServer) {
     }
     const root = await getWorkspaceRootAsync(mcpServer);
     const resolved = path.resolve(root, trimmed);
+    // Lexical check against workspace root only (relative paths must stay under the case workspace).
     if (resolved !== root && !resolved.startsWith(root + path.sep)) {
         throw new Error("Path must be within the workspace directory.");
     }
     try {
         const real = await fs.realpath(resolved);
-        const realRoot = await fs.realpath(root);
-        if (real !== realRoot && !real.startsWith(realRoot + path.sep)) {
-            throw new Error("Resolved path escapes workspace (e.g. symlink).");
+        if (!(await isPathUnderSandboxRoots(real, root))) {
+            throw sandboxEscapeError(real, root);
         }
         const stat = await fs.stat(real);
         if (!stat.isFile()) {
@@ -371,7 +500,8 @@ export async function resolveFilePathForUpload(filePath, mcpServer) {
         return real;
     }
     catch (err) {
-        if (err instanceof Error && (err.message.includes("escapes") || err.message.includes("regular file"))) {
+        if (err instanceof Error &&
+            (err.message.includes("symlink resolves outside") || err.message.includes("regular file"))) {
             throw err;
         }
         throw new Error(`File not accessible. Easiest fix: pass an ABSOLUTE path (e.g. "/home/you/project/data.csv" or ` +
@@ -444,7 +574,15 @@ export function classifyApiError(status, bodyText, requestId) {
     }
     else if (status === 409) {
         cause = "client";
-        hint = "The job may not be in the expected state.";
+        if (apiCode === "staging_not_ready") {
+            const stageId = parsed?.stage_job_id != null ? String(parsed.stage_job_id) : "";
+            hint = stageId
+                ? `Wait for stage_dataset to finish: jobs(action=monitor, job_id="${stageId}"), then retry train. Prefer datasets(action=get) train_ready=true.`
+                : "Wait for dataset staging (status=ready with staged_prefix / train_ready=true), then retry train.";
+        }
+        else {
+            hint = "The job may not be in the expected state.";
+        }
         retryable = false;
     }
     else if (status === 429) {
@@ -824,15 +962,30 @@ export function structuredTextResult(structuredContent, text, content) {
 }
 export async function pollUntilComplete(jobId, maxWaitMs = 30_000, intervalMs = 1000) {
     const start = Date.now();
+    /** Brief 404s happen when finalize_job_id is visible before the child row is readable. */
+    let notFoundStreak = 0;
+    const maxNotFoundRetries = 5;
     while (Date.now() - start < maxWaitMs) {
-        const data = (await apiCall("GET", `/v1/jobs/${jobId}`));
-        const status = data.status;
-        if (status === "completed" || status === "failed" || status === "cancelled") {
-            return {
-                status,
-                result_ref: data.result_ref,
-                error: data.error,
-            };
+        try {
+            const data = (await apiCall("GET", `/v1/jobs/${jobId}`));
+            notFoundStreak = 0;
+            const status = data.status;
+            if (status === "completed" || status === "failed" || status === "cancelled") {
+                return {
+                    status,
+                    result_ref: data.result_ref,
+                    error: data.error,
+                };
+            }
+        }
+        catch (err) {
+            const status = err?.httpStatus;
+            if (status === 404 && notFoundStreak < maxNotFoundRetries) {
+                notFoundStreak += 1;
+                await new Promise((r) => setTimeout(r, intervalMs));
+                continue;
+            }
+            throw err;
         }
         await new Promise((r) => setTimeout(r, intervalMs));
     }
@@ -986,6 +1139,43 @@ export function inlineAttachBytesUsed() {
 export function shouldFetchAllRemainingFigures(figures) {
     return figures === "all" || figures === "images";
 }
+/** Enum tokens accepted by results(figures). Prefer string "all"/"none"; arrays are for names/filenames. */
+const RESULTS_FIGURE_ENUMS = new Set([
+    "default",
+    "combined_only",
+    "all",
+    "images",
+    "none",
+]);
+/**
+ * Coerce host-stringified figures args (e.g. `'["all"]'`) and singleton enum arrays
+ * (`["all"]` → `"all"`) before Zod validation. Real filename arrays stay arrays.
+ */
+export function coerceFiguresArg(v) {
+    if (v === undefined || v === null)
+        return v;
+    let val = v;
+    if (typeof val === "string") {
+        const trimmed = val.trim();
+        if ((trimmed.startsWith("[") && trimmed.endsWith("]")) ||
+            (trimmed.startsWith('"') && trimmed.endsWith('"'))) {
+            try {
+                val = JSON.parse(trimmed);
+            }
+            catch {
+                return v;
+            }
+        }
+    }
+    if (Array.isArray(val)) {
+        const asStrings = val.map((x) => String(x));
+        if (asStrings.length === 1 && RESULTS_FIGURE_ENUMS.has(asStrings[0])) {
+            return asStrings[0];
+        }
+        return asStrings;
+    }
+    return val;
+}
 /** Attach result figures with captions; tracks inlined names for dedup. */
 export async function attachResultsImages(content, jobId, jobType, summary, figures, includeIndividual, inlinedImages) {
     for (const name of getResultsImagesToFetch(jobType, summary, figures, includeIndividual)) {