@barivia/barsom-mcp 0.23.0 → 0.23.3

package/README.md

@@ -54,7 +54,8 @@ MCP clients typically run it with **`npx`** (downloads on first use):
 | `BARIVIA_API_URL` | No | `https://api.barivia.se` | API base URL |
 | `BARIVIA_WORKSPACE_ROOT` | No | `process.cwd()` or `PWD` | Directory for relative `file_path` and `save_to_disk`. In Cursor MCP, `process.cwd()` is often the MCP install dir — add `BARIVIA_WORKSPACE_ROOT` to your MCP config `env` with your project path (e.g. `/home/user/myproject`). Absolute paths and `file://` URIs work without it. |
 | `BARIVIA_FETCH_TIMEOUT_MS` | No | `60000` | Per-request HTTP timeout (ms) to the API. Increase (e.g. `120000`) on slow networks or when the API does long-running work. Large dataset uploads use a separate longer timeout internally, and `datasets(analyze)` runs asynchronously (auto-polled) so it is not bound by this. A timeout is not auto-retried (re-firing slow work would only multiply load / risk duplicates). |
-| `BARIVIA_VIZ_PORT` | No | OS-assigned | When the client has no MCP Apps support, the proxy may start a local viz server on `127.0.0.1`. The port is **assigned per MCP session and changes when the proxy restarts**, so standalone `…/viz/…` URLs from a previous session go stale (the page then can't reach `/api/*` and shows a "stale session — re-run the tool" message). Set a fixed port for stable bookmark URLs. Diagnostic: `GET http://127.0.0.1:<port>/api/health?job_id=<id>` reports the port, API reachability, and the job's status. |
+| `BARIVIA_VIZ_PORT` | No | OS-assigned | Local viz server on `127.0.0.1`. Port is **per MCP session** unless pinned — standalone `…/viz/…` URLs go stale after proxy restart. Set a fixed port for stable bookmark URLs. Diagnostic: `GET http://127.0.0.1:<port>/api/health?job_id=<id>`. |
+| `BARIVIA_UI_DELIVERY` | No | `auto` | Client-capability override for App tools: `auto` \| `apps` \| `localhost` \| `inline`. Tool responses always lead with a standalone localhost URL (hosts may list `ui://` without mounting widgets) and include a structured `ui_delivery` block. `account(status)` reports detected Apps support, viz port, and active tier. |
 | `BARIVIA_ENFORCE_WORKSPACE_SANDBOX` | No | `1` (enabled) | File uploads are constrained to the MCP workspace root by default. Set to `0` or `false` to allow absolute paths anywhere on the machine (high trust). **Changed in v0.x:** previously defaulted to off; now on for security. If uploads fail with "paths outside the workspace are disabled", either set `BARIVIA_WORKSPACE_ROOT` to cover your data directory, or opt out with `BARIVIA_ENFORCE_WORKSPACE_SANDBOX=0`. |
 
 Legacy `BARSOM_API_KEY` / `BARSOM_API_URL` / `BARSOM_WORKSPACE_ROOT` are also accepted as fallbacks.
@@ -67,7 +68,7 @@ Legacy `BARSOM_API_KEY` / `BARSOM_API_URL` / `BARSOM_WORKSPACE_ROOT` are also ac
 
 **Supported stack:** Node **18+** (see `engines` in `package.json`). Built with `@modelcontextprotocol/sdk` **^1.x** — if a major MCP client upgrade breaks tools, check SDK release notes alongside this package version.
 
-**Local viz fallback:** If the IDE does not advertise MCP Apps, the proxy starts an HTTP server bound to **127.0.0.1** only (not exposed to the LAN). It serves built-in HTML for training prep / monitor / results explorer and proxies read-only API calls with your existing API key. Responses may include `http://127.0.0.1:<port>/viz/...` links. Browser `Access-Control-Allow-Origin: *` applies only to that localhost origin so the embedded pages can load data.
+**Local viz fallback:** The proxy starts an HTTP server bound to **127.0.0.1** only (not exposed to the LAN). It serves built-in HTML for training monitor / results explorer and proxies read-only API calls with your existing API key. **`training_monitor` / `results_explorer` always lead with a standalone `http://127.0.0.1:<port>/viz/...` URL** (even when the host advertises MCP Apps — some Cursor builds list `ui://` but never mount widgets). Browser `Access-Control-Allow-Origin: *` applies only to that localhost origin so the embedded pages can load data.
 
 ## Tools and prompts (15 tools + 2 prompts)
 
@@ -102,7 +103,7 @@ Call at the **start of mapping work** (or when the user asks what the MCP can do
 | `analyze` | Pre-training recommendations (correlation, columns to consider dropping, etc.) |
 | `list` | Finding dataset IDs |
 | `get` | One dataset by id — status, staging fields, `stage_job_id`, ingest errors |
-| `subset` | Creating a filtered/sliced copy (row_range, filter conditions) |
+| `subset` | Creating a filtered/sliced copy (row_range, filter, `sample_n`) — **only when the user asks**; agents must not auto-downsample before train/analyze |
 | `add_expression` | Add a derived column from an expression (formula → new column on the dataset) |
 | `reduce_spectral` | Pre-training reducer for long ordered numeric blocks (spectra, time series, sensor fingerprints, gene panels). Methods: **pca** (top-k principal components), **log_sample** (k columns at log-spaced indices — scattering, audio bands), **uniform_sample** (k columns at evenly-spaced indices — regularly-sampled time series), **stats** (6 fixed per-row summary columns). All produce one feature vector per row; appends derived columns to the dataset. |
 | `delete` | Removing a dataset |
@@ -172,11 +173,11 @@ Capacity is shared LOCAL/GKE worker pools (no per-key cloud burst lease).
 
 | Tool | Role |
 |------|------|
-| `training_monitor` | Live QE + **panel** topographic-error curves (ordering/convergence, dual y-axis); **Panel TE** and **Map TE** scalars at kernel end; FLooP step progress when applicable; optional — `jobs(action=status)` is enough |
-| `results_explorer` | Browse metrics and figures after training completes |
+| `training_monitor` | Live QE + **panel** TE curves; auto-opened from `train()` submit when the host mounts Apps; cross-link to results explorer when finalize completes; always returns a standalone URL + `ui_delivery` |
+| `results_explorer` | Browse metrics and figures after training completes; cross-link back to training monitor; always returns a standalone URL + `ui_delivery` |
 | `_fetch_figure` | **Host / App only** — Results Explorer invokes this for one raster figure; not for agent chat |
 
-When the client does not advertise MCP Apps support, the proxy starts **`viz-server`** on **`127.0.0.1`** (localhost-only). Tool responses can include `http://127.0.0.1:PORT/viz/...` links. See **Local viz fallback** under Environment Variables.
+Standalone localhost URLs are always first in tool content (see `BARIVIA_UI_DELIVERY` / **Local viz fallback**). Pages cross-link monitor ↔ results for the same `job_id`.
 
 #### Choosing where to view results
 
@@ -190,6 +191,7 @@ The right viewer depends on **(MCP App support)** **and** **(can the human reach
 
 ### Migration notes
 
+- **UI delivery + train auto-open (0.22.4+):** `train()` returns `training_monitor` App structured content so supporting hosts open the monitor in the same turn. App tools emit structured `ui_delivery` (`delivery`, `urls`, `hint_for_agent`) and always put the standalone viz URL first. Env: `BARIVIA_UI_DELIVERY`, `BARIVIA_VIZ_PORT`. Cyclic cos/sin component PNGs are omitted from default `summary.files` / `figures=all` (encoding for training; panels still appear in `combined` / recolor); opt in with train param `viz_upload_cyclic_components=true`.
 - **Account surface (0.22.0, breaking):** `account(request_compute|compute_status|release_compute)` removed — AWS cloud burst leases are gone. Use `account(status|history|add_funds)`. Capacity is shared LOCAL/GKE pools.
 - **Structured results + train hints (0.21.x, non-breaking):**
   - **`results(action=get)`** returns **`structuredContent`** (job id/type, label, summary metrics, file list) alongside the text summary — same pattern as `training_monitor` / `results_explorer`. Agent hosts can read fields without parsing prose.
@@ -229,7 +231,7 @@ When adding or refining tools, follow [MCP best practices](https://modelcontextp
 
 ## Data preparation
 
-To train on a subset of your data (e.g. first 2000 rows, or rows where region=Europe) without re-uploading: use **datasets(action=subset)** with `row_range` and/or `filter` to create a new dataset, then train with **train(action=map, dataset_id=...)**, **train(action=siom_map, dataset_id=...)**, or **train(action=floop_siom, dataset_id=...)** on the new dataset_id; or pass **row_range** in the training job params for a one-off slice.
+To train on a subset of your data (e.g. first 2000 rows, or rows where region=Europe) without re-uploading: use **datasets(action=subset)** with `row_range` and/or `filter` (or `sample_n`) to create a new dataset, then train with **train(action=map, dataset_id=...)** on the new dataset_id; or pass **row_range** in the training job params for a one-off slice. **Agents must not call subset/`sample_n` unless the user explicitly asks** — train on the full uploaded table by default.
 
 ## How It Works
 

package/dist/index.js

@@ -1,2 +1,2 @@
 #!/usr/bin/env node
-import{McpServer as e}from"@modelcontextprotocol/sdk/server/mcp.js";import{StdioServerTransport as t}from"@modelcontextprotocol/sdk/server/stdio.js";import{z as o}from"zod";import{getUiCapability as s,registerAppResource as r,RESOURCE_MIME_TYPE as n}from"@modelcontextprotocol/ext-apps/server";import{startVizServer as a}from"./viz-server.js";import{API_KEY as i,apiCall as l,apiRawCall as p,loadViewHtml as c,setVizPort as u,setClientSupportsMcpApps as d,CLIENT_VERSION as m}from"./shared.js";import{registerDatasetsTool as f}from"./tools/datasets.js";import{registerTrainTool as h}from"./tools/train.js";import{registerJobsTool as g,JOBS_DESCRIPTION_BASE as b}from"./tools/jobs.js";import{registerResultsTool as _}from"./tools/results.js";import{registerExploreMapTool as y,RESULTS_EXPLORER_URI as w}from"./tools/explore_map.js";import{registerAccountTool as v}from"./tools/account.js";import{registerInferenceTool as j}from"./tools/inference.js";import{registerGuideBarsomTool as k}from"./tools/guide_barsom.js";import{registerTrainingGuidanceTool as x}from"./tools/training_guidance.js";import{registerFeedbackTool as P}from"./tools/feedback.js";import{registerTrainingMonitorTool as A,TRAINING_MONITOR_URI as I}from"./tools/training_monitor.js";import{resolvePrepareTrainingPrompt as C}from"./prepare_training_prompt.js";i||(console.error("Error: BARIVIA_API_KEY not set. Set it in your MCP client config."),process.exit(1));const T=new e({name:"analytics-engine",version:m,instructions:'# Barivia Mapping Analytics Engine\n\nSelf-organizing map (SOM) analytics: project high-dimensional data to a 2D grid for clusters, gradients, and anomalies.\n\n## Workflow (short)\n\nUpload (`datasets(upload)`) → `datasets(preview)` and `datasets(analyze)` before train → submit one of `train(action=map)`, `train(action=siom_map)`, or `train(action=floop_siom)` (only if plan allows FLooP) → poll `jobs(status)` every 10–15s until `completed` → `results(get)` for metrics and figures (there is no separate analyze tool), then `results_explorer(job_id)` to browse the figures interactively. Then `jobs(compare)`, `results(download/recolor)`, or `inference` (predict / compare / project_columns / impute_column / transition_flow) as needed.\n\n**Full detail:** Call `guide_barsom_workflow` for plan-scoped tool map, training modes, async rules, optional MCP App UIs, and step-by-step SOP (from the Barivia API when online).\n\n## Tool taxonomy (one tool per stage — do not mix)\n\n| Stage | Tool(s) | Contract |\n|-------|---------|----------|\n| Data | `datasets` | upload, preview, analyze, list, **get** (status/staging), subset, add_expression, reduce_spectral, delete — ingest + feature engineering, no map |\n| Train submit | `train` | map, siom_map, impute (sparse → map + dense imputed.csv), floop_siom (entitled). Async, returns job_id. Submit only |\n| Lifecycle | `jobs` | status, list (slim + limit/cursor/status/job_type/has_results), update (label/description/tags), compare (**training runs**), cancel, delete. No training/scoring here |\n| Artifacts | `results` | get (default=combined only; figures="none" for metrics-only sweeps; figures="all" or results_explorer/download for every plot), export, download, recolor (async). Read/render completed jobs. Results catalog = `jobs(list, has_results=true)` |\n| Frozen-map ops | `inference` | predict (regime-aware; "compact"|"annotated"), batch_predict, impute_column (neighbor-pool fill for a non-training column), compare (cohort density-diff), project_columns, transition_flow (time-ordered rows only), report. No weight updates |\n| Account | `account` | status, history, **inventory** (markdown datasets+jobs overview), add_funds, **health** (API liveness via GET /health — no R2) |\n| Bootstrap | `guide_barsom_workflow`, `train`, `training_guidance`, `prepare_training` prompt | orientation, submit map/siom/impute/floop, parameter hints, narrative checklist (API/tier-scoped) |\n| Explore (UI) | `results_explorer`, `training_monitor` | richer UX over the same API; `jobs(status)`+`results(get)` still suffice headless |\n| Other | `send_feedback` | only after user agrees; queues locally if API/storage is down |\n\n### Avoid these mis-routes\n- `jobs(compare)` = metrics table across **training runs** (pick best in a sweep). `inference(compare)` = density-diff of one map vs a **second dataset** (drift/cohort). Different tools.\n- Post-training map analysis (predict, compare, project_columns, impute_column, **transition_flow**) is all **inference** — not `results`. `results` is presentation/artifacts only (`recolor` stays here: it re-renders, it doesn\'t touch data).\n- Project a quantity onto the map: existing columns → `inference(project_columns)`; a formula → `datasets(add_expression, project_onto_job=<job>)`.\n\n## Async pattern\n\n- **Manual poll:** Training submits return `job_id` immediately — poll `jobs(status)` every 10–15s. **Running is not failed**; large grids or FLooP-SIOM can take many minutes. `max_nodes` (FLooP) is a total node budget, not grid side length.\n- **Often auto-polled:** `inference` actions (incl. `transition_flow`) and `results(recolor)` may wait in-proxy; if you get a `job_id`, poll `jobs(status)` the same way.\n\nCredits: jobs consume compute credits; check `account(status)` before big runs. Per-request HTTP timeout defaults to 60s (`datasets(analyze)` is async + auto-polled, so it is not bound by it); on slow networks raise `BARIVIA_FETCH_TIMEOUT_MS`. Timeouts are NOT auto-retried (avoids duplicating slow/expensive work).\n\n## Org inventory recipe\n\nTo rebuild a durable markdown overview of an org (datasets + jobs):\n1. `account(action=inventory)` — preferred one-shot (datasets name/id/rows/cols/description/tags + slim recent jobs).\n2. Or compose: `datasets(list)` + `jobs(action=list, limit=50)` (page with `cursor` / filter with `status` / `job_type` / `has_results=true` for a results catalog).\n3. Set `description` + `tags` on `datasets(upload|update)` and `train(...)` / `jobs(update)` so duplicates stay navigable.\n\n## API briefly unavailable (pause, do not crash)\nIf tools return `api_unreachable` / gateway HTML / slim `error_code` lines: summarize in **one calm sentence** — never paste Cloudflare HTML. The proxy already burst-retried (**3** tries, ~**2s** apart); honor `retry_after_sec` (~**30**) before another round. Use `account(action=health)` (no R2). Local downloads and prior results stay usable; `send_feedback` queues under `~/.barivia/deferred-feedback` and flushes when the API returns. Do not start large uploads/trains until `account(status)` or health looks healthy again.\n\n## Constraints\n\n- **Do not auto-subset:** Never call `datasets(subset)` or `sample_n` unless the user explicitly asks for a filter, slice, or random sample. Train and analyze on the uploaded `dataset_id` at full row count — GPU staging/prepare handles hundreds of thousands of rows quickly; silent downsampling hides real prepare/train behavior and wastes scale demos. Use small named fixtures (e.g. sample.csv, periodic_load) only when the user or runbook names them.\n- Prep ladder: `prepare_training` prompt = narrative checklist; `training_guidance` = structured hints (presets, resolved normalization/transforms, grid/epochs). Review these, then submit with `train`. Do not guess tiers or FLooP entitlement.\n- `inference(predict)`: prefer `dataset_id` for batch and for SIOM/irregular maps; single-row `rows` uses a fast path that can fail on some topologies — retry with `dataset_id`. FLooP-SIOM: if predict jobs fail while grid SIOM works, capture errors + `job_id`.\n- Column names are case-sensitive — match `datasets(preview)`.\n- Default training path is numeric/cyclic/temporal; use explicit `categorical_features` for baseline categoricals. `predict` must match the model contract.\n- After `results(recolor)`, `inference(transition_flow)`, or `inference(project_columns)`, use the **new** `job_id` returned for follow-up `results` if applicable.\n- **Large results / MCP timeouts:** For sweeps, `jobs(compare)`, or metrics-only reads use `results(action=get, figures="none")`. Default `get` inlines combined only. For every plot use `results_explorer` or `results(action=download)`, not `figures="all"` in chat unless the user wants all images inline.\n\n## UI delivery (heterogeneous MCP clients)\n\nHosts differ (Cursor, Claude Desktop, CLI agents): some embed MCP App panels; others list `ui://` but never mount them (expected on some Cursor builds); others are text-only. Tool results **always lead with a standalone localhost URL** — that is the intended fallback, not a broken App.\n\n| Tier | When | Agent should |\n|------|------|--------------|\n| **embedded** | Client advertises MCP Apps | Let the panel render **and** surface the standalone localhost URL from tool output / `ui_delivery.urls` (hosts often list Apps but do not mount). |\n| **localhost** | No MCP Apps; viz server running | Post the markdown link from tool output to the user (never broken workspace paths). Re-post after job completes if needed. |\n| **inline_image** | No Apps / override `inline` | Summarize metrics; inline raster (combined.png, learning curve) is attached automatically — show it in the reply. |\n| **text_only** | Viz server down | Use `jobs(status)` + `results(get)`; no panel or localhost link. |\n\nEvery `training_monitor` / `results_explorer` response includes a structured `ui_delivery` block (`delivery`, `urls`, `hint_for_agent`). Follow `hint_for_agent` without user prompting. Standalone viz pages cross-link monitor ↔ results for the same `job_id`.\n\n**Env overrides:** `BARIVIA_UI_DELIVERY=auto|apps|localhost|inline` (default `auto`). `BARIVIA_VIZ_PORT` pins the localhost viz port across proxy restarts. **Diagnostics:** `account(action=status)` reports detected MCP Apps support, viz port, and active tier.'});r(T,w,w,{mimeType:n},async()=>{const e=await c("results-explorer");return{contents:[{uri:w,mimeType:n,text:e??"<html><body>Results Explorer view not built yet. Run: npm run build:views</body></html>"}]}}),r(T,I,I,{mimeType:n},async()=>{const e=await c("training-monitor");return{contents:[{uri:I,mimeType:n,text:e??"<html><body>Training Monitor view not built yet.</body></html>"}]}}),k(T),y(T),A(T),f(T),h(T),g(T,b),_(T),v(T),j(T),x(T),P(T),T.prompt("info","Short orientation for the Barivia Mapping MCP. For full plan-scoped workflow, tool map, and SOP, the model should call guide_barsom_workflow. Use when the user asks what this MCP can do or how to get started.",{},()=>({messages:[{role:"user",content:{type:"text",text:["Give a concise, scannable answer (headers + bullets):","","**What it is:** MCP client to the Barivia mapping engine (2D SOM / SIOM / FLooP-SIOM when entitled) over HTTPS.","","**First step:** Call `guide_barsom_workflow` for plan-scoped bootstrap (full tool list, async rules, training modes, optional MCP Apps, SOP).","","**Core path:** `datasets(upload)` → `datasets(preview)` + `datasets(analyze)` → choose training action → poll `jobs(status)` every 10–15s until completed → `results(get)` (all main figures/metrics; no separate analyze tool) → `results_explorer(job_id)` to browse figures.","",'**Key tools:** `datasets` (data; reduce_spectral for spectra/long blocks), `train` (map/siom_map/impute/floop_siom; pass `label` for readable compare rows; preset=quick for a fast first map), `jobs` (status/list/compare/cancel/delete — lifecycle only), `results` (get/download/export/recolor; figures="none" for metrics-only), `inference` (predict; batch_predict; impute_column for topology-neighbor pool fill; compare; project_columns; transition_flow; report), `account` (status/credits/queue).',"","**Prep help:** `prepare_training` prompt (checklist) · `training_guidance` (presets/JSON hints + resolved normalization/params) — review, then submit with `train`.","","**Browse results:** `results_explorer(job_id)` is the preferred way to explore figures after a first `results(get)` glance; `training_monitor` shows live progress. Both optional — `results` + `jobs(status)` suffice headless.","","**After training:** `jobs(compare)` across runs, `results(recolor)`, `inference(project_columns)` for variables not in training, `inference(transition_flow)` only if rows are time-ordered.","","**Rules:** Running ≠ failed. Column names must match `datasets(preview)` exactly. Do not auto-subset — never call `datasets(subset)` / `sample_n` unless the user explicitly asks; train on the full uploaded `dataset_id`. Do not call `_fetch_figure` from chat (host/UI only); use `results(get)` or `results_explorer`.","","Offer `send_feedback` only after asking the user."].join("\n")}}]})),T.prompt("prepare_training","Narrative pre-training checklist (prompt). Use after upload and before train. Content is tier-scoped from the API when online. Prep ladder: this prompt = story checklist; training_guidance tool = JSON presets/parameter hints. Review, then submit with train.",{dataset_id:o.string().describe("Dataset ID to prepare for training")},async({dataset_id:e})=>{const t=await C(e);return{messages:[{role:"user",content:{type:"text",text:(t.used_fallback?`[offline fallback ${t.fallback_version}]\n\n`:"")+t.text}}]}});const M=new t;(async function(){try{const e=await a(l,p,c);u(e)}catch(e){process.env.BARIVIA_VIZ_PORT&&console.error("Barivia viz server failed to start:",e)}const e=T.server;e.oninitialized=()=>{const t=e.getClientCapabilities(),o=s(t);d(!!o?.mimeTypes?.includes(n))},await T.connect(M)})().catch(console.error);
+import{McpServer as e}from"@modelcontextprotocol/sdk/server/mcp.js";import{StdioServerTransport as t}from"@modelcontextprotocol/sdk/server/stdio.js";import{z as o}from"zod";import{getUiCapability as s,registerAppResource as r,RESOURCE_MIME_TYPE as n}from"@modelcontextprotocol/ext-apps/server";import{startVizServer as a}from"./viz-server.js";import{API_KEY as i,apiCall as l,apiRawCall as p,loadViewHtml as c,setVizPort as u,setClientSupportsMcpApps as d,CLIENT_VERSION as m}from"./shared.js";import{registerDatasetsTool as f}from"./tools/datasets.js";import{registerTrainTool as h}from"./tools/train.js";import{registerJobsTool as g,JOBS_DESCRIPTION_BASE as b}from"./tools/jobs.js";import{registerResultsTool as _}from"./tools/results.js";import{registerExploreMapTool as y,RESULTS_EXPLORER_URI as w}from"./tools/explore_map.js";import{registerAccountTool as v}from"./tools/account.js";import{registerInferenceTool as j}from"./tools/inference.js";import{registerGuideBarsomTool as k}from"./tools/guide_barsom.js";import{registerTrainingGuidanceTool as x}from"./tools/training_guidance.js";import{registerFeedbackTool as P}from"./tools/feedback.js";import{registerTrainingMonitorTool as A,TRAINING_MONITOR_URI as I}from"./tools/training_monitor.js";import{resolvePrepareTrainingPrompt as C}from"./prepare_training_prompt.js";i||(console.error("Error: BARIVIA_API_KEY not set. Set it in your MCP client config."),process.exit(1));const T=new e({name:"analytics-engine",version:m,instructions:'# Barivia Mapping Analytics Engine\n\nSelf-organizing map (SOM) analytics: project high-dimensional data to a 2D grid for clusters, gradients, and anomalies.\n\n## Workflow (short)\n\nUpload (`datasets(upload)`) → `datasets(preview)` and `datasets(analyze)` before train → submit one of `train(action=map)`, `train(action=siom_map)`, or `train(action=floop_siom)` (only if plan allows FLooP) → poll `jobs(status)` every 10–15s until `completed` → `results(get)` for metrics and figures (there is no separate analyze tool), then `results_explorer(job_id)` to browse the figures interactively. Then `jobs(compare)`, `results(download/recolor)`, or `inference` (predict / compare / project_columns / impute_column / transition_flow) as needed.\n\n**Full detail:** Call `guide_barsom_workflow` for plan-scoped tool map, training modes, async rules, optional MCP App UIs, and step-by-step SOP (from the Barivia API when online).\n\n## Tool taxonomy (one tool per stage — do not mix)\n\n| Stage | Tool(s) | Contract |\n|-------|---------|----------|\n| Data | `datasets` | upload, preview, analyze, list, **get** (status/staging), subset, add_expression, reduce_spectral, delete — ingest + feature engineering, no map |\n| Train submit | `train` | map, siom_map, impute (sparse → map + dense imputed.csv), floop_siom (entitled). Async, returns job_id. Submit only |\n| Lifecycle | `jobs` | status, list (slim + limit/cursor/status/job_type/has_results), update (label/description/tags), compare (**training runs**), cancel, delete. No training/scoring here |\n| Artifacts | `results` | get (default=combined only; figures="none" for metrics-only sweeps; figures="all" or results_explorer/download for every plot), export, download, recolor (async). Read/render completed jobs. Results catalog = `jobs(list, has_results=true)` |\n| Frozen-map ops | `inference` | predict (regime-aware; "compact"|"annotated"), batch_predict, impute_column (neighbor-pool fill for a non-training column), compare (cohort density-diff), project_columns, transition_flow (time-ordered rows only), report. No weight updates |\n| Account | `account` | status, history, **inventory** (markdown datasets+jobs overview), add_funds, **health** (API liveness via GET /health — no R2) |\n| Bootstrap | `guide_barsom_workflow`, `train`, `training_guidance`, `prepare_training` prompt | orientation, submit map/siom/impute/floop, parameter hints, narrative checklist (API/tier-scoped) |\n| Explore (UI) | `results_explorer`, `training_monitor` | richer UX over the same API; `jobs(status)`+`results(get)` still suffice headless |\n| Other | `send_feedback` | only after user agrees; queues locally if API/storage is down |\n\n### Avoid these mis-routes\n- `jobs(compare)` = metrics table across **training runs** (pick best in a sweep). `inference(compare)` = density-diff of one map vs a **second dataset** (drift/cohort). Different tools.\n- Post-training map analysis (predict, compare, project_columns, impute_column, **transition_flow**) is all **inference** — not `results`. `results` is presentation/artifacts only (`recolor` stays here: it re-renders, it doesn\'t touch data).\n- Project a quantity onto the map: existing columns → `inference(project_columns)`; a formula → `datasets(add_expression, project_onto_job=<job>)`.\n\n## Async pattern\n\n- **Manual poll:** Training submits return `job_id` immediately — poll `jobs(status)` every 10–15s. **Running is not failed**; large grids or FLooP-SIOM can take many minutes. `max_nodes` (FLooP) is a total node budget, not grid side length.\n- **Often auto-polled:** `inference` actions (incl. `transition_flow`) and `results(recolor)` may wait in-proxy; if you get a `job_id`, poll `jobs(status)` the same way.\n\nCredits: jobs consume compute credits; check `account(status)` before big runs. Per-request HTTP timeout defaults to 60s (`datasets(analyze)` is async + auto-polled, so it is not bound by it); on slow networks raise `BARIVIA_FETCH_TIMEOUT_MS`. Timeouts are NOT auto-retried (avoids duplicating slow/expensive work).\n\n## Org inventory recipe\n\nTo rebuild a durable markdown overview of an org (datasets + jobs):\n1. `account(action=inventory)` — preferred one-shot (datasets name/id/rows/cols/description/tags + slim recent jobs).\n2. Or compose: `datasets(list)` + `jobs(action=list, limit=50)` (page with `cursor` / filter with `status` / `job_type` / `has_results=true` for a results catalog).\n3. Set `description` + `tags` on `datasets(upload|update)` and `train(...)` / `jobs(update)` so duplicates stay navigable.\n\n## API briefly unavailable (pause, do not crash)\nIf tools return `api_unreachable` / gateway HTML / slim `error_code` lines: summarize in **one calm sentence** — never paste Cloudflare HTML. The proxy already burst-retried (**3** tries, ~**2s** apart); honor `retry_after_sec` (~**30**) before another round. Use `account(action=health)` (no R2). Local downloads and prior results stay usable; `send_feedback` queues under `~/.barivia/deferred-feedback` and flushes when the API returns. Do not start large uploads/trains until `account(status)` or health looks healthy again.\n\n## Constraints\n\n- **Do not auto-subset:** Never call `datasets(subset)` or `sample_n` unless the user explicitly asks. On CPU, `preset=quick` (15×15) is typically very fast up to ~100k rows — do **not** downsample by default when exploring below 100k; only suggest `sample_n` when n>100k or the user asks. Train/analyze the full `dataset_id`. Use small named fixtures only when the user or runbook names them.\n- **Small-n before train:** If preview/analyze shows n<1000 (especially n<100), heed the JUMPBACK/warning — omit `grid_x`/`grid_y` for auto (~sqrt(5·√n)), avoid oversized presets on tiny tables, drop redundant correlated geo features, and check `active_node_fraction`/TE before narrating map insights.\n- Prep ladder: `prepare_training` prompt = narrative checklist; `training_guidance` = structured hints (presets, resolved normalization/transforms, grid/epochs). Review these, then submit with `train`. Do not guess tiers or FLooP entitlement.\n- `inference(predict)`: prefer `dataset_id` for batch and for SIOM/irregular maps; single-row `rows` uses a fast path that can fail on some topologies — retry with `dataset_id`. FLooP-SIOM: if predict jobs fail while grid SIOM works, capture errors + `job_id`.\n- Column names are case-sensitive — match `datasets(preview)`.\n- Default training path is numeric/cyclic/temporal; use explicit `categorical_features` for baseline categoricals. `predict` must match the model contract.\n- After `results(recolor)`, `inference(transition_flow)`, or `inference(project_columns)`, use the **new** `job_id` returned for follow-up `results` if applicable.\n- **Large results / MCP timeouts:** For sweeps, `jobs(compare)`, or metrics-only reads use `results(action=get, figures="none")`. Default `get` inlines combined only. For every plot use `results_explorer` or `results(action=download)`, not `figures="all"` in chat unless the user wants all images inline.\n\n## UI delivery (heterogeneous MCP clients)\n\nHosts differ (Cursor, Claude Desktop, CLI agents): some embed MCP App panels; others list `ui://` but never mount them (expected on some Cursor builds); others are text-only. Tool results **always lead with a standalone localhost URL** — that is the intended fallback, not a broken App.\n\n| Tier | When | Agent should |\n|------|------|--------------|\n| **embedded** | Client advertises MCP Apps | Let the panel render **and** surface the standalone localhost URL from tool output / `ui_delivery.urls` (hosts often list Apps but do not mount). |\n| **localhost** | No MCP Apps; viz server running | Post the markdown link from tool output to the user (never broken workspace paths). Re-post after job completes if needed. |\n| **inline_image** | No Apps / override `inline` | Summarize metrics; inline raster (combined.png, learning curve) is attached automatically — show it in the reply. |\n| **text_only** | Viz server down | Use `jobs(status)` + `results(get)`; no panel or localhost link. |\n\nEvery `training_monitor` / `results_explorer` response includes a structured `ui_delivery` block (`delivery`, `urls`, `hint_for_agent`). Follow `hint_for_agent` without user prompting. Standalone viz pages cross-link monitor ↔ results for the same `job_id`.\n\n**Env overrides:** `BARIVIA_UI_DELIVERY=auto|apps|localhost|inline` (default `auto`). `BARIVIA_VIZ_PORT` pins the localhost viz port across proxy restarts. **Diagnostics:** `account(action=status)` reports detected MCP Apps support, viz port, and active tier.'});r(T,w,w,{mimeType:n},async()=>{const e=await c("results-explorer");return{contents:[{uri:w,mimeType:n,text:e??"<html><body>Results Explorer view not built yet. Run: npm run build:views</body></html>"}]}}),r(T,I,I,{mimeType:n},async()=>{const e=await c("training-monitor");return{contents:[{uri:I,mimeType:n,text:e??"<html><body>Training Monitor view not built yet.</body></html>"}]}}),k(T),y(T),A(T),f(T),h(T),g(T,b),_(T),v(T),j(T),x(T),P(T),T.prompt("info","Short orientation for the Barivia Mapping MCP. For full plan-scoped workflow, tool map, and SOP, the model should call guide_barsom_workflow. Use when the user asks what this MCP can do or how to get started.",{},()=>({messages:[{role:"user",content:{type:"text",text:["Give a concise, scannable answer (headers + bullets):","","**What it is:** MCP client to the Barivia mapping engine (2D SOM / SIOM / FLooP-SIOM when entitled) over HTTPS.","","**First step:** Call `guide_barsom_workflow` for plan-scoped bootstrap (full tool list, async rules, training modes, optional MCP Apps, SOP).","","**Core path:** `datasets(upload)` → `datasets(preview)` + `datasets(analyze)` → choose training action → poll `jobs(status)` every 10–15s until completed → `results(get)` (all main figures/metrics; no separate analyze tool) → `results_explorer(job_id)` to browse figures.","",'**Key tools:** `datasets` (data; reduce_spectral for spectra/long blocks), `train` (map/siom_map/impute/floop_siom; pass `label` for readable compare rows; preset=quick for a fast first map), `jobs` (status/list/compare/cancel/delete — lifecycle only), `results` (get/download/export/recolor; figures="none" for metrics-only), `inference` (predict; batch_predict; impute_column for topology-neighbor pool fill; compare; project_columns; transition_flow; report), `account` (status/credits/queue).',"","**Prep help:** `prepare_training` prompt (checklist) · `training_guidance` (presets/JSON hints + resolved normalization/params) — review, then submit with `train`.","","**Browse results:** `results_explorer(job_id)` is the preferred way to explore figures after a first `results(get)` glance; `training_monitor` shows live progress. Both optional — `results` + `jobs(status)` suffice headless.","","**After training:** `jobs(compare)` across runs, `results(recolor)`, `inference(project_columns)` for variables not in training, `inference(transition_flow)` only if rows are time-ordered.","","**Rules:** Running ≠ failed. Column names must match `datasets(preview)` exactly. Do not auto-subset below ~100k rows (CPU quick 15×15 is fast); never call `datasets(subset)` / `sample_n` unless the user asks or n>100k exploratory. Small-n (<1000): prefer auto grid, not oversized presets. Do not call `_fetch_figure` from chat (host/UI only); use `results(get)` or `results_explorer`.","","Offer `send_feedback` only after asking the user."].join("\n")}}]})),T.prompt("prepare_training","Narrative pre-training checklist (prompt). Use after upload and before train. Content is tier-scoped from the API when online. Prep ladder: this prompt = story checklist; training_guidance tool = JSON presets/parameter hints. Review, then submit with train.",{dataset_id:o.string().describe("Dataset ID to prepare for training")},async({dataset_id:e})=>{const t=await C(e);return{messages:[{role:"user",content:{type:"text",text:(t.used_fallback?`[offline fallback ${t.fallback_version}]\n\n`:"")+t.text}}]}});const M=new t;(async function(){try{const e=await a(l,p,c);u(e)}catch(e){process.env.BARIVIA_VIZ_PORT&&console.error("Barivia viz server failed to start:",e)}const e=T.server;e.oninitialized=()=>{const t=e.getClientCapabilities(),o=s(t);d(!!o?.mimeTypes?.includes(n))},await T.connect(M)})().catch(console.error);

package/dist/prepare_training_prompt.js

@@ -1,8 +1,9 @@
 import { apiCall } from "./shared.js";
-export const PREPARE_TRAINING_FALLBACK_VERSION = "2026.07-offline";
+export const PREPARE_TRAINING_FALLBACK_VERSION = "2026.07f-offline";
 const FALLBACK_LINES = [
     "Before upload: for train(action=map) ensure training columns have no NaNs; for train(action=impute) missing cells in training columns are OK (plain numeric only).",
-    "Do not auto-subset: never call datasets(subset) or sample_n unless the user explicitly asks — train on the full uploaded dataset_id.",
+    "Do not auto-subset: never call datasets(subset) or sample_n unless the user explicitly asks — on CPU, preset=quick 15×15 is typically very fast up to ~100k rows; only suggest sample_n when n>100k or the user asks. Train on the full uploaded dataset_id.",
+    "Small-n: if preview/analyze shows n<1000 (especially n<100), omit grid_x/grid_y for auto (~sqrt(5·√n)); avoid preset=quick/standard on tiny tables (dead nodes / TE≈1). Drop redundant correlated geo features before train.",
     'Please run datasets(action=preview, dataset_id="{id}") to inspect columns, then datasets(action=analyze, dataset_id="{id}") to see which columns and temporal periods are most informative.',
     "Choose the training path:",
     '- train(action=map, dataset_id="{id}", ...) for complete-case data',

package/dist/shared.js

@@ -39,7 +39,7 @@ function newRequestId() {
  * X-Barsom-Client-Version so the server can annotate tool guidance with the
  * wrapper version each action requires. Keep in sync with package.json on bump.
  */
-export const CLIENT_VERSION = "0.23.0";
+export const CLIENT_VERSION = "0.23.3";
 /** User-facing links; keep aligned with barivia.se / api.barivia.se. */
 export const PUBLIC_SITE_ORIGIN = "https://barivia.se";
 /** Self-serve account dashboard (manage plan, billing, and API keys). */

package/dist/siom_occupation_format.js

@@ -0,0 +1,57 @@
+/**
+ * Format the SIOM occupation / coverage block for results(get) text.
+ * Surfaces effective hyperparams + util/Gini/entropy/siom_qe alongside grid
+ * dead-node counts (which come from hit_stats elsewhere).
+ */
+export function formatSiomOccupationSummary(siom, opts) {
+    if (!siom || typeof siom !== "object")
+        return [];
+    const fmt4 = (v) => v !== null && v !== undefined && v !== "" && Number.isFinite(Number(v))
+        ? Number(v).toFixed(4)
+        : "N/A";
+    const fmt3 = (v) => v !== null && v !== undefined && v !== "" && Number.isFinite(Number(v))
+        ? Number(v).toFixed(3)
+        : "N/A";
+    const fmtAny = (v) => v === null || v === undefined || v === "" ? "N/A" : String(v);
+    const hasHyper = siom.gamma !== undefined ||
+        siom.gamma_f !== undefined ||
+        siom.decay !== undefined ||
+        siom.siom_decay !== undefined ||
+        siom.penalty !== undefined ||
+        siom.siom_penalty !== undefined ||
+        siom.reset_per_epoch !== undefined;
+    const decay = siom.decay ?? siom.siom_decay;
+    const penalty = siom.penalty ?? siom.siom_penalty;
+    const files = opts?.files ?? [];
+    const hasOccFile = files.some((f) => String(f).includes("siom_occupation"));
+    if (opts?.compact) {
+        const parts = [
+            `utilization: ${fmt4(siom.utilization)}`,
+            `dead_fraction: ${fmt4(siom.dead_fraction)}`,
+            `gini: ${fmt4(siom.gini)}`,
+            `entropy: ${fmt4(siom.entropy)}`,
+            siom.siom_qe !== undefined ? `siom_qe: ${fmt4(siom.siom_qe)}` : "",
+            hasHyper
+                ? `γ=${fmt3(siom.gamma)}${siom.gamma_f !== undefined && Number(siom.gamma_f) > 0 ? `→${fmt3(siom.gamma_f)}` : ""} penalty=${fmtAny(penalty)} decay=${fmt4(decay)}`
+                : "",
+        ].filter(Boolean);
+        return [`SIOM occupation summary — ${parts.join(" | ")}`];
+    }
+    const lines = [
+        "SIOM occupation summary (layer coverage; distinct from grid BMU dead nodes):",
+        `  Utilization: ${fmt4(siom.utilization)} | dead_fraction: ${fmt4(siom.dead_fraction)}`,
+        `  Gini: ${fmt4(siom.gini)} | Entropy: ${fmt4(siom.entropy)}${siom.max_entropy !== undefined ? ` / max ${fmt4(siom.max_entropy)}` : ""}`,
+        siom.siom_qe !== undefined ? `  SIOM QE: ${fmt4(siom.siom_qe)}` : "",
+    ];
+    if (hasHyper) {
+        lines.push(`  Knobs: gamma=${fmt3(siom.gamma)}` +
+            (siom.gamma_f !== undefined ? ` gamma_f=${fmt3(siom.gamma_f)}` : "") +
+            ` siom_decay=${fmt4(decay)} siom_penalty=${fmtAny(penalty)}` +
+            (siom.penalty_alpha !== undefined ? ` α=${fmt3(siom.penalty_alpha)}` : "") +
+            (siom.reset_per_epoch !== undefined ? ` reset_per_epoch=${fmtAny(siom.reset_per_epoch)}` : ""));
+    }
+    if (hasOccFile) {
+        lines.push("  Artifact: siom_occupation.json (per-node μ) — results(action=download, include_json=true).");
+    }
+    return lines.filter((l) => l !== "");
+}

package/dist/tools/datasets.js

@@ -7,6 +7,7 @@ import { registerAuditedTool } from "../audit.js";
 import { apiCall, getWorkspaceRootAsync, resolveFilePathForUpload, textResult, structuredTextResult, pollUntilComplete, POLL_DERIVE_MAX_MS, POLL_ANALYZE_MAX_MS, UPLOAD_DATASET_TIMEOUT_MS, LARGE_UPLOAD_BYTES, PRESIGNED_PUT_TIMEOUT_MS, POLL_STAGE_MAX_MS, streamFileSha256, putPresignedStream, resolveUploadContentType, suggestDatasetPreview, } from "../shared.js";
 import { GZIP_UPLOAD_HINT } from "../job_status_format.js";
 import { formatDatasetInventoryLine, formatTags } from "../inventory_format.js";
+import { PERIODICITY_ROW_ORDER_NOTE, SMALL_N_SOFT, buildCpuScaleGuidanceLines, buildSmallNGuardrailLines, } from "../training_scale_guidance.js";
 /**
  * Normalize a nullable string field from the API. Returns "" for absent values,
  * empty strings, and the literal SQL/serialization sentinels "missing"/"null"
@@ -111,6 +112,16 @@ function formatAnalyzeResult(data, dataset_id) {
         lines.push(``);
     }
     const nextSteps = [];
+    const nRows = typeof totalRows === "number" && Number.isFinite(totalRows) ? totalRows : 0;
+    if (nRows > 0 && nRows < SMALL_N_SOFT) {
+        nextSteps.push(...buildSmallNGuardrailLines({ n: nRows, nFeatures: trainList.length || columns.length }));
+    }
+    if (nRows > 0) {
+        nextSteps.push(...buildCpuScaleGuidanceLines(nRows));
+    }
+    if (considerDropping.length > 0) {
+        nextSteps.push(`Drop redundant correlated columns before train (e.g. keep distance OR lat/lon, not all three when |r|>0.9): ${considerDropping.join(", ")}.`);
+    }
     if (trainList.length > 0) {
         nextSteps.push(`Train with columns: [${trainList.join(", ")}].`);
     }
@@ -120,11 +131,11 @@ function formatAnalyzeResult(data, dataset_id) {
     const periodicStrong = periodicity.filter((p) => p.dominant_score > 0.4);
     if (periodicStrong.length > 0) {
         const lags = [...new Set(periodicStrong.map((p) => p.dominant_lag))].sort((a, b) => a - b);
-        nextSteps.push(`Note: periodicity is computed on row order. If your CSV is not sorted by time, ignore lag-based suggestions; only use cyclic_features when the column is genuinely cyclic (e.g. hour, month, angle).`);
+        nextSteps.push(`Note: ${PERIODICITY_ROW_ORDER_NOTE}`);
         nextSteps.push(`Columns show periodic behavior at lags ${lags.join(", ")} (assuming row order is meaningful). Consider cyclic_features or temporal_features with matching periods.`);
         nextSteps.push(`For datetime columns: run datasets(action=preview) for temporal_suggestions; match lags (e.g. 365→day_of_year, 12→month, 7→day_of_week, 24→hour_of_day) to choose which temporal components to extract.`);
     }
-    nextSteps.push(`Then call train(action=map, dataset_id=${dataset_id}, columns=[...], ...).`);
+    nextSteps.push(`Then call train(action=map, dataset_id=${dataset_id}, columns=[...], ...) — omit grid_x/grid_y for auto-size unless you intentionally override.`);
     lines.push(`Next steps:`, ...nextSteps.map((s) => `  ${s}`));
     return lines.join("\n");
 }
@@ -141,7 +152,7 @@ Formats: plain CSV/TSV or gzip (.csv.gz / .tsv.gz). For files above ~100 MB, pre
 | list | Finding dataset IDs for train_map, preview, or subset — see all available datasets (includes description/tags) |
 | get | Fetch one dataset by id — status, staging fields, ingest_error, description/tags (use after upload or when staging is slow) |
 | update | Edit name / description / tags on an existing dataset (helps navigate duplicates) |
-| subset | ONLY when the user asks for row_range, filters, or sample_n — never auto-shrink before train/analyze |
+| subset | ONLY when the user asks for row_range, filters, or sample_n — never auto-shrink before train/analyze (CPU quick is fast to ~100k; do not sample_n by default below that) |
 | add_expression | Add a computed column from an expression (dataset_id + name + expression). With project_onto_job=<training_job_id>, instead projects the computed expression onto that map as a component plane in one step (no column added). |
 | reduce_spectral | Collapse a long ordered numeric block (e.g. 1000-point spectrum, time series, sensor fingerprint) into a small per-row feature set so the SOM can train on signal-dense features. Methods: pca, log_sample, uniform_sample, stats. |
 | delete | Cleaning up after experiments or freeing the dataset slot |
@@ -487,7 +498,13 @@ ESCALATION: If upload fails with column errors, open the file locally and verify
                     lines.push(`| ${cols.map((c) => String(row[c] ?? "")).join(" | ")} |`);
                 }
             }
-            lines.push(``, `Suggested next step: train(action=map, dataset_id=${dataset_id}, ...) or use the prepare_training prompt; to add derived columns use datasets(action=add_expression, dataset_id=${dataset_id}, name=..., expression=...).`);
+            if (totalRows > 0 && totalRows < SMALL_N_SOFT) {
+                lines.push(``, ...buildSmallNGuardrailLines({ n: totalRows, nFeatures: cols.length }));
+            }
+            if (totalRows > 0) {
+                lines.push(``, ...buildCpuScaleGuidanceLines(totalRows));
+            }
+            lines.push(``, `Suggested next step: datasets(action=analyze) then train(action=map, dataset_id=${dataset_id}, ...) or use the prepare_training prompt; to add derived columns use datasets(action=add_expression, dataset_id=${dataset_id}, name=..., expression=...).`);
             return { content: [{ type: "text", text: lines.join("\n") }] };
         }
         if (action === "analyze") {

package/dist/tools/jobs.js

@@ -32,7 +32,7 @@ ASYNC POLLING PROTOCOL (action=status):
 
 action=list: Slim rows by default (label,id,status,job_type,dataset_id,created_at,result_ref,description,tags — no fat params). Use status/job_type/has_results/limit/cursor to page. has_results=true is the results catalog. Pass fields="full" only when you need params/error.
 action=update: PATCH label/description/tags (pass at train submit when possible).
-action=compare: metrics table (QE, TE, explained variance, silhouette) for 2+ COMPLETED TRAINING jobs — pick the best run in a sweep. NOT inference(action=compare) (single trained job + second dataset → density-diff heatmap). Only hyperparameter columns that differ across runs are shown; pass label="…" at train time for readable rows.
+action=compare: metrics table (QE, TE, explained variance, silhouette) for 2+ COMPLETED TRAINING jobs — pick the best run in a sweep. NOT inference(action=compare) (single trained job + second dataset → density-diff heatmap). Only hyperparameter columns that differ across runs are shown (including SIOM knobs gamma/gamma_f/siom_decay/siom_penalty/reset_per_epoch when they vary); pass label="…" at train time for readable rows.
 action=cancel: Not instant — worker checks between phases. Expect up to 30s delay.
 action=delete: WARNING — job ID will no longer work with results or any other tool.
 
@@ -80,7 +80,11 @@ export function renderCompareTable(comparisons) {
         { key: "n_initial", label: "NInitial", render: fmtAny },
         { key: "growth_interval", label: "GrowthInt", render: fmtAny },
         { key: "gamma", label: "Gamma", render: fmtAny },
+        { key: "gamma_f", label: "GammaF", render: fmtAny },
         { key: "siom_decay", label: "SIOMDecay", render: fmtAny },
+        { key: "siom_penalty", label: "SIOMPenalty", render: fmtAny },
+        { key: "penalty_alpha", label: "Penaltyα", render: fmtAny },
+        { key: "reset_per_epoch", label: "Reset/Epoch", render: fmtAny },
         { key: "elastic_lambda", label: "Elasticλ", render: fmtAny },
         { key: "elastic_mu", label: "Elasticμ", render: fmtAny },
         { key: "elastic_anchor", label: "ElasticAnchor", render: fmtAny },

package/dist/tools/results.js

@@ -3,6 +3,7 @@ import fs from "node:fs/promises";
 import { z } from "zod";
 import { registerAuditedTool } from "../audit.js";
 import { apiCall, apiRawCall, getWorkspaceRootAsync, sandboxPath, pollUntilComplete, tryAttachImage, resetInlineAttachBudget, attachResultsImages, resolveOutputDpi, POLL_RECOLOR_MAX_MS, fmtDensityDiffNode, getCaptionForImage, shouldFetchAllRemainingFigures, mimeForFilename, structuredTextResult, } from "../shared.js";
+import { formatSiomOccupationSummary } from "../siom_occupation_format.js";
 export function registerResultsTool(server) {
     registerAuditedTool(server, "results", `Retrieve, recolor, download, or export figures and metrics for a completed map job. Presentation and artifact access only — operations on the frozen map (predict, compare, project, impute_column, transition_flow) live in **inference**.
 
@@ -225,7 +226,10 @@ NOT FOR: Jobs that haven't completed (use jobs(action=status) first), or tempora
                     hitStats.active_node_fraction !== undefined
                         ? `Grid BMU occupancy: ${(Number(hitStats.active_node_fraction) * 100).toFixed(1)}% active nodes on the map grid`
                         : "",
-                    ...(siom ? [`SIOM layer coverage (distinct from grid dead nodes) — utilization: ${fmt(siom.utilization)} | dead_fraction: ${fmt(siom.dead_fraction)} | siom_qe: ${fmt(siom.siom_qe)}`] : []),
+                    ...formatSiomOccupationSummary(siom, {
+                        files: summary.files ?? [],
+                        compact: true,
+                    }),
                     hitStats.grid_suggestion ? `Grid hint: ${String(hitStats.grid_suggestion)}` : "",
                     ...(policy.auto_rules_applied && Array.isArray(policy.auto_rules_applied) && policy.auto_rules_applied.length > 0
                         ? [`Auto policy: ${policy.auto_rules_applied.join(", ")} (${policy.viz_mode ?? "viz"}, metrics=${policy.quality_metrics ?? "?"})`]
@@ -296,19 +300,15 @@ NOT FOR: Jobs that haven't completed (use jobs(action=status) first), or tempora
                     `Coverage Assessment:`,
                     `  Status:        ${coverageStatus}${isDegenerate ? " *** NON-COMPETITIVE ***" : isWarning ? " * CAUTION *" : ""}`,
                     `  Diagnosis:     ${String(coverage.diagnosis ?? "No plain-language diagnosis available.")}`,
-                    `  Utilization:   ${siom.utilization !== undefined ? Number(siom.utilization).toFixed(4) : "N/A"}`,
-                    `  SIOM dead fraction (layer coverage, not grid BMU hits): ${siom.dead_fraction !== undefined ? Number(siom.dead_fraction).toFixed(4) : "N/A"}`,
+                    ...formatSiomOccupationSummary(siom, { files }),
                     `  Occupied nodes:${hitStats.occupied_nodes !== undefined ? ` ${hitStats.occupied_nodes}` : " N/A"}`,
                     `  Top hit share: ${hitStats.top_hit_share !== undefined ? Number(hitStats.top_hit_share).toFixed(4) : "N/A"}`,
-                    `  Gini:          ${siom.gini !== undefined ? Number(siom.gini).toFixed(4) : "N/A"}`,
-                    `  Entropy:       ${siom.entropy !== undefined ? Number(siom.entropy).toFixed(4) : "N/A"}`,
-                    `  SIOM gamma:    ${siom.gamma !== undefined ? Number(siom.gamma).toFixed(3) : "N/A"} | decay: ${siom.decay !== undefined ? Number(siom.decay).toFixed(4) : "N/A"}`,
                     ...(summary.elastic ? (() => {
                         const el = summary.elastic;
                         return [`  Elastic:       λ=${el.lambda !== undefined ? Number(el.lambda).toFixed(3) : "0"} μ=${el.mu !== undefined ? Number(el.mu).toFixed(3) : "0"} anchor=${el.anchor !== undefined ? Number(el.anchor).toFixed(3) : "0"}`];
                     })() : []),
                     ``,
-                    `Layout tuning: if Voronoi cells look crowded in weight space (high top_hit_share), reduce max_nodes and/or raise gamma/siom_decay before growth_interval. See training_guidance keys floop_voronoi_crowding, floop_siom_params, floop_elastic_params.`,
+                    `Layout tuning: if Voronoi cells look crowded in weight space (high top_hit_share), reduce max_nodes and/or raise gamma/siom_decay before growth_interval. See training_guidance keys floop_voronoi_crowding, floop_siom_params, floop_elastic_params, siom_gamma.`,
                     ``,
                     `Quality Metrics:`,
                     `  Quantization Error: ${summary.quantization_error !== undefined ? Number(summary.quantization_error).toFixed(4) : "N/A"}${qeNote}`,
@@ -386,11 +386,11 @@ NOT FOR: Jobs that haven't completed (use jobs(action=status) first), or tempora
                     ...(compLines ? [`Component value ranges:\n${compLines}`] : []),
                     ...(clusterLines ? ["", "Cluster map (defining features):", clusterLines] : []),
                     ...(topCorrLine ? ["", topCorrLine] : []),
-                    ...(siom ? ["", "SIOM layer coverage (distinct from grid dead nodes):",
-                        `  Utilization: ${siom.utilization !== undefined ? Number(siom.utilization).toFixed(4) : "N/A"}`,
-                        `  SIOM dead fraction: ${siom.dead_fraction !== undefined ? Number(siom.dead_fraction).toFixed(4) : "N/A"}`,
-                        `  Gini: ${siom.gini !== undefined ? Number(siom.gini).toFixed(4) : "N/A"}`,
-                        `  Entropy: ${siom.entropy !== undefined ? Number(siom.entropy).toFixed(4) : "N/A"}`] : []),
+                    ...(siom
+                        ? ["", ...formatSiomOccupationSummary(siom, {
+                                files: summary.files ?? [],
+                            })]
+                        : []),
                     ``, `Features: ${features.join(", ")}`,
                     summary.selected_columns ? `Selected columns: ${summary.selected_columns.join(", ")}` : "",
                     summary.transforms ? `Transforms: ${Object.entries(summary.transforms).map(([k, v]) => `${k}=${v}`).join(", ")}` : "",

package/dist/tools/train.js

@@ -4,6 +4,7 @@ import { apiCall } from "../shared.js";
 import { buildTrainSubmitExtras } from "../train_submit_message.js";
 import { TRAINING_MONITOR_URI, buildTrainingMonitorResult, } from "./training_monitor.js";
 import { TRAINING_INPUT_SCHEMA, buildTrainMapParams, buildFloopParams, fetchTrainingPresets, assertValidNormalizationArgs, } from "./training_core.js";
+import { SMALL_N_SOFT, buildSmallNGuardrailLines, suggestedAutoGridSide, } from "../training_scale_guidance.js";
 export const TRAIN_DESCRIPTION = `Submit a self-organizing map training job. Returns a job_id; poll with jobs(action=status) and then results(action=get). All actions are async.
 
 | Action | Use when |
@@ -13,11 +14,11 @@ export const TRAIN_DESCRIPTION = `Submit a self-organizing map training job. Ret
 | impute | Sparse training data: trains a missing-tolerant map (accelerated missSOM; SIOM missSOM when model=auto on som_siom+ or model=SIOM) AND returns dense imputed.csv in one job. Plain numeric columns only. Optional SIOM knobs: gamma, siom_decay, siom_penalty, penalty_alpha, reset_per_epoch. |
 | floop_siom | FLooP-SIOM — a growing node-budget manifold instead of a fixed grid; default topology=free (CHL). Requires a plan with all_algorithms (Premium/Enterprise). |
 
-First quick map (no separate baseline action): use \`train(action=map, preset=quick)\` (or \`normalize=mad\` with a grid sized from the preview row count); see the \`baseline_explore\` preset in **training_guidance**.
+First quick map (no separate baseline action): use \`train(action=map, preset=quick)\` when n is large enough (hundreds+ rows); for n<1000 prefer omitting grid_x/grid_y for auto (~sqrt(5·√n)) or an explicit small grid — see **training_guidance** / analyze next_steps. On CPU, preset=quick 15×15 is typically very fast up to ~100k rows — do not subsample first.
 
 Parameter details (grid, epochs, batch, model, presets, normalize, categorical_features, backend, FLooP max_nodes/effort, SIOM geometry) live in the **training_guidance** tool / **prepare_training** prompt — call them instead of guessing. Backend strings are "cpu" | "gpu" | "gpu_graphs" (no colon); on CPU-only hosts pass backend=cpu.
 
-ASYNC: every action returns a job_id immediately. Poll jobs(action=status, job_id=...) every 10–15s (do NOT poll faster). Map training typical times: 10×10 ~30s | 20×20 ~3–5 min | 40×40 ~15–30 min — do not assume failure before 3 minutes on CPU. When status is completed, call results(action=get, job_id=...).
+ASYNC: every action returns a job_id immediately. Poll jobs(action=status, job_id=...) every 10–15s (do NOT poll faster). Map training typical times: preset=quick 15×15 often tens of seconds even at ~30k–100k rows on CPU | 20×20 ~3–5 min | 40×40 ~15–30 min — do not assume failure before 3 minutes on CPU. When status is completed, call results(action=get, job_id=...).
 
 action=impute: use when many cells are missing across training columns; use inference(impute_column) instead when you already have a complete-case map and need to fill one held-out column. Supports transforms (log/sqrt/etc., not rank), temporal_features, cyclic_features, optional cyclic_pairs=[[cos,sin],...] (circular prototype refresh + atomic chord BMU; those columns get circular quality metrics only — never quote Euclidean R² on cos/sin), and normalization_methods (zscore/mad/sigmoidal/none — not sepd). Does not support auto_log_transforms, rank transforms, or normalize=sepd. Defaults model=auto (SIOM missSOM on som_siom+ plans, else plain missSOM), cv_folds=5 → quality.csv (linear: kfold_holdout_pool + resubstitution_optimistic; cyclic pairs: kfold_holdout_circular). Aggregate MAE/RMSE excludes cyclic columns. Do not treat k-fold alone as true missing-cell ground truth. Optional SIOM knobs: gamma, gamma_f, siom_decay, siom_penalty, penalty_alpha, reset_per_epoch. Artifacts: imputed.csv, imputation_mask.csv, optional imputation_uncertainty.csv (pool_std = neighbourhood dispersion). A completed impute job_id is a valid parent for inference(impute_column).
 action=floop_siom: max_nodes is a TOTAL node budget (not a grid width/area); if coverage collapses, reduce max_nodes before increasing effort.
@@ -66,6 +67,11 @@ async function submitTrainJob(opts) {
     if (opts.gridLargeForRows) {
         data.grid_auto_warning = true;
     }
+    const guardrails = opts.guardrailLines ?? [];
+    if (guardrails.length > 0) {
+        data.training_guardrails = guardrails;
+        data.small_n_warning = opts.totalRows > 0 && opts.totalRows < SMALL_N_SOFT;
+    }
     const extras = buildTrainSubmitExtras({
         newJobId,
         totalRows: opts.totalRows,
@@ -89,8 +95,12 @@ async function submitTrainJob(opts) {
     catch {
         /* ignore queue enrichment */
     }
-    if (opts.gridLargeForRows) {
-        msg += ` Note: Grid may be large for ${opts.totalRows} rows (consider grid=auto for fewer dead nodes).`;
+    if (guardrails.length > 0) {
+        msg += ` ${guardrails.join(" ")}`;
+    }
+    else if (opts.gridLargeForRows) {
+        const autoSide = suggestedAutoGridSide(opts.totalRows);
+        msg += ` Note: Grid may be large for ${opts.totalRows} rows — prefer omitting grid_x/grid_y for auto (~${autoSide}×${autoSide}) to reduce dead nodes.`;
     }
     data.message = msg;
     data.suggested_next_step = extras.suggested_next_step;
@@ -195,6 +205,16 @@ export async function runTrain(action, args) {
             typeof transforms === "object" &&
             Object.keys(transforms).length > 0;
         const gridLargeForRows = effectiveGrid && totalRows > 0 && effectiveGrid[0] * effectiveGrid[1] > totalRows * 0.75;
+        const nFeatures = Array.isArray(columns) ? columns.length
+            : Array.isArray(params.columns) ? params.columns.length
+                : undefined;
+        const guardrailLines = buildSmallNGuardrailLines({
+            n: totalRows,
+            nFeatures,
+            preset: typeof preset === "string" ? preset : undefined,
+            gridX: effectiveGrid?.[0],
+            gridY: effectiveGrid?.[1],
+        });
         const resultsHint = action === "impute" ? "view the map and imputed.csv"
             : "view the map and metrics";
         const variantPrefix = action === "siom_map" ? "variant=siom"
@@ -213,6 +233,7 @@ export async function runTrain(action, args) {
             resultsHint,
             impute: action === "impute",
             gridLargeForRows: !!gridLargeForRows,
+            guardrailLines,
         });
     }
     if (action === "floop_siom") {

package/dist/tools/training_core.js

@@ -46,7 +46,7 @@ export async function fetchTrainingConfig() {
 export const TRAINING_INPUT_SCHEMA = {
     preset: z.enum(["quick", "standard", "refined", "high_res"]).optional(),
     grid_x: z.number().int().optional()
-        .describe("Grid width. Omit grid_x AND grid_y (and preset) to auto-size the map (~5·√√n per side); the result reports hit_stats.active_node_fraction and a grid_suggestion when too many nodes are dead."),
+        .describe("Grid width. Omit grid_x AND grid_y (and preset) to auto-size via sqrt(5·√n) per side (clamped 5–50; ~6×6 at n≈45). Pass grid_x/grid_y — not a bare `grid` field. Results report hit_stats.active_node_fraction and grid_suggestion when many nodes are dead."),
     grid_y: z.number().int().optional().describe("Grid height. See grid_x for auto-sizing."),
     epochs: z.preprocess((v) => {
         if (v === undefined || v === null)
@@ -118,17 +118,17 @@ export const TRAINING_INPUT_SCHEMA = {
     emit_cell_uncertainty: z.boolean().optional()
         .describe("impute: write imputation_uncertainty.csv (pool_std = prototype-neighbourhood dispersion, not calibrated SD; pool_n = contributing nodes)"),
     gamma: z.preprocess((v) => (v !== undefined && v !== null && typeof v === "string") ? parseFloat(v) : v, z.number().optional())
-        .describe("siom_map / impute (SIOM or auto on som_siom+): self-interaction strength γ (default 0.5 on worker)"),
+        .describe("siom_map / impute (SIOM or auto on som_siom+): self-interaction strength γ (default 0.5). Prefer γ∈[0.5,1.5] on large n; γ>1.5 often diminishing returns."),
     gamma_f: z.preprocess((v) => (v !== undefined && v !== null && typeof v === "string") ? parseFloat(v) : v, z.number().optional())
-        .describe("siom_map / impute SIOM: final γ for exponential decay schedule (0 = constant γ)"),
+        .describe("siom_map / impute SIOM: final γ for exponential decay (0 = constant γ). Pattern: start strict, relax late (e.g. 1.5→0.2)."),
     siom_decay: z.preprocess((v) => (v !== undefined && v !== null && typeof v === "string") ? parseFloat(v) : v, z.number().optional())
-        .describe("siom_map / impute SIOM: occupation EMA decay λ (default 0.01; 0 = cumulative counting)"),
+        .describe("siom_map / impute SIOM: occupation EMA decay λ (default 0.01; 0 = cumulative). Avoid pairing high γ with aggressive decay (e.g. 0.05)."),
     siom_penalty: z.enum(["linear", "log", "exp"]).optional()
-        .describe("siom_map / impute SIOM: occupation penalty shape"),
+        .describe("siom_map / impute SIOM: occupation penalty shape (linear default; prefer exp over log — log often worsens TE)"),
     penalty_alpha: z.preprocess((v) => (v !== undefined && v !== null && typeof v === "string") ? parseFloat(v) : v, z.number().optional())
         .describe("siom_map / impute SIOM: scaling α inside log/exp penalty"),
     reset_per_epoch: z.boolean().optional()
-        .describe("siom_map / impute SIOM: reset occupation μ at each epoch"),
+        .describe("siom_map / impute SIOM: reset occupation μ each epoch (default false; avoid with high γ — can collapse EV)"),
     siom_feature_geometry: z.enum(["l2", "mixed", "auto"]).optional().describe("siom_map: l2 (default) = legacy L2 on cos/sin columns; mixed = cyclic torus distance when cyclic pairs exist; auto = mixed only when cyclic encodings are present"),
     siom_qe_backend: z.enum(["cpu", "cuda", "auto"]).optional().describe("siom_map: backend for metric-aligned siom_qe when using mixed geometry; auto uses GPU when available"),
     siom_qe_batch_size: z.number().int().min(1).max(1_000_000).optional().describe("siom_map: batch size for GPU SIOM QE (default 256 on worker)"),

package/dist/training_monitor_curve.js

@@ -19,6 +19,26 @@ export function isKernelTrainingComplete(data, status) {
 export function teCurveLabel(base) {
     return `${base} (panel)`;
 }
+/**
+ * Detect FLooP jobs for chart axis/labels.
+ * Do NOT infer from "ordering-only curves" — mesh_convergence / train_som often use
+ * epochs=[N,0], so convergence_errors is empty while ordering has many batch points.
+ */
+export function isFloopJob(data) {
+    const phase = String(data.training_progress_phase ?? data.progress_phase ?? "").toLowerCase();
+    if (phase === "floop")
+        return true;
+    const jt = String(data.job_type ?? "").toLowerCase();
+    if (jt.includes("floop"))
+        return true;
+    const rc = data.run_config;
+    if (rc && typeof rc === "object") {
+        const model = String(rc.model ?? "").toLowerCase();
+        if (model.includes("floop"))
+            return true;
+    }
+    return false;
+}
 export function formatCurveSourceNote(data) {
     const src = data.training_curve_source_batches;
     const parts = [];

package/dist/training_scale_guidance.js

@@ -0,0 +1,73 @@
+/**
+ * Scale / small-n guidance shared by datasets(preview|analyze) and train submit.
+ * Auto-grid formula matches API/worker: clamp(round(sqrt(5*sqrt(n))), 5, 50).
+ */
+export const SMALL_N_SOFT = 1000;
+export const SMALL_N_HARD = 100;
+/** Agents should not suggest sample_n below this unless the user asks. */
+export const SAMPLE_N_SUGGEST_THRESHOLD = 100_000;
+/** Worker/API auto grid side length. */
+export function suggestedAutoGridSide(n) {
+    if (!Number.isFinite(n) || n < 1)
+        return 5;
+    return Math.max(5, Math.min(50, Math.round(Math.sqrt(5 * Math.sqrt(n)))));
+}
+export function presetGridCells(preset) {
+    switch (preset) {
+        case "quick":
+            return 15 * 15;
+        case "standard":
+            return 25 * 25;
+        case "refined":
+            return 40 * 40;
+        case "high_res":
+            return 60 * 60;
+        default:
+            return null;
+    }
+}
+/**
+ * Build prominent small-n / oversize-grid warnings for agents.
+ * Empty when n is healthy and the chosen grid is not oversized.
+ */
+export function buildSmallNGuardrailLines(opts) {
+    const { n, nFeatures, preset, gridX, gridY } = opts;
+    if (!Number.isFinite(n) || n <= 0)
+        return [];
+    const lines = [];
+    const autoSide = suggestedAutoGridSide(n);
+    const cells = gridX !== undefined && gridY !== undefined
+        ? gridX * gridY
+        : presetGridCells(preset);
+    if (n < SMALL_N_HARD) {
+        lines.push(`JUMPBACK — small-n (${n} rows): maps are fragile below ~100 rows. Prefer grid_x/grid_y ≈ ${autoSide}×${autoSide} (omit both for auto ≈ sqrt(5·√n)), not preset=quick/standard (15×15/25×25 → mostly dead nodes). Drop redundant geo features (keep distance OR lat/lon, not all three when |r|>0.9). Check hit_stats.active_node_fraction / TE before narrating insights.`);
+    }
+    else if (n < SMALL_N_SOFT) {
+        lines.push(`Warning — small-n (${n} rows < 1000): oversized grids leave dead nodes. Auto grid ≈ ${autoSide}×${autoSide}; omit grid_x/grid_y for auto, or size so |features|×grid_cells is not ≫ n. preset=quick (15×15) is often too large below a few hundred rows — verify active_node_fraction after train.`);
+    }
+    if (cells !== null && cells > n * 0.75) {
+        lines.push(`Grid risk: ${cells} cells vs ${n} rows (cells ≫ n) → expect dead nodes / TE≈1. Use auto (~${autoSide}×${autoSide}) or a smaller explicit grid_x/grid_y.`);
+    }
+    if (nFeatures !== undefined &&
+        nFeatures > 0 &&
+        cells !== null &&
+        nFeatures * cells > n * 2) {
+        lines.push(`Capacity risk: ${nFeatures} features × ${cells} cells ≫ ${n} rows. Drop highly correlated columns (see analyze recommendations) and/or shrink the grid before train.`);
+    }
+    return lines;
+}
+/** CPU scale guidance: do not subsample by default below 100k. */
+export function buildCpuScaleGuidanceLines(n) {
+    if (!Number.isFinite(n) || n <= 0)
+        return [];
+    if (n <= SAMPLE_N_SUGGEST_THRESHOLD) {
+        return [
+            `Scale: on CPU, preset=quick (15×15) is typically very fast up to ~100k rows (tens of seconds). Do NOT call datasets(subset)/sample_n for n=${n.toLocaleString()} — train the full dataset_id unless the user asks to downsample. Only suggest sample_n when n>100k or the user requests it.`,
+        ];
+    }
+    return [
+        `Scale: n=${n.toLocaleString()} > 100k — full-table train is still supported; suggest sample_n only if the user wants a faster exploratory pass or ops runbook says so. Never auto-subset silently.`,
+    ];
+}
+/** Periodicity caveat for analyze next_steps (row-order dependent). */
+export const PERIODICITY_ROW_ORDER_NOTE = "Periodicity assumes meaningful row order (time/sequence). If the CSV is unordered or shuffled, ignore lag-based suggestions; only use cyclic_features when the column is genuinely cyclic (e.g. hour, month, angle).";