@qe-mcp/ena 0.1.18 → 0.1.20
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +77 -8
- package/assets.js +54 -0
- package/manifest.json +1 -1
- package/package.json +7 -3
- package/server.js +275 -100
package/README.md
CHANGED
|
@@ -176,22 +176,91 @@ A hosted server has no access to anyone's files, so this mode is a different
|
|
|
176
176
|
privacy promise, and it says so to the model. In `--http` mode:
|
|
177
177
|
|
|
178
178
|
- Data arrives inline as `data_csv`; `data_path` is refused. The server
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
179
|
+
parses it in memory and discards it when the call returns: no rows and no
|
|
180
|
+
CSV text are kept, in memory or on disk (locally, parsed rows are cached for
|
|
181
|
+
speed). What stays is the fitted model behind a `model_id` — points,
|
|
182
|
+
networks, unit labels and group values — for `$ENA_MCP_MODEL_TTL_MIN`
|
|
183
|
+
minutes after its last use (default 60), then it is forgotten.
|
|
184
|
+
`ena_simplify` refits, so hosted it takes `data_csv` again and checks that
|
|
185
|
+
it is the data the model was fitted on.
|
|
186
|
+
- `ena_export` has nowhere to write, so it returns the ENA Web Tool project
|
|
187
|
+
ZIP inline, as a base64 resource in the result. The ZIP holds the data
|
|
188
|
+
file, which the server no longer has, so it takes `data_csv` again (checked
|
|
189
|
+
the same way) and keeps neither the data nor the ZIP afterwards. A client
|
|
190
|
+
that cannot save a file from a result asks for `format: "set"` instead: the
|
|
191
|
+
set file alone, which needs no data and sends none back; the summary says
|
|
192
|
+
how to pack it with the user's own data file. There is no download link,
|
|
193
|
+
not even a short-lived one, because that would mean keeping the data.
|
|
194
|
+
- `ena_profile`, `ena_tables` and `ena_lines` are not offered (they read or
|
|
195
|
+
write files), and the prompts are not listed (they take a path). Plots come
|
|
196
|
+
back as the inline PNG plus the SVG as text; no files are written.
|
|
184
197
|
- The server's instructions tell the model to say that the data leaves the
|
|
185
198
|
user's machine and must be de-identified, and to get their OK first.
|
|
186
199
|
- `model_id`s carry a random suffix so one conversation cannot plot another's
|
|
187
200
|
model.
|
|
188
201
|
|
|
189
202
|
Each request is answered statelessly, so any replica can take any request,
|
|
190
|
-
but fitted models
|
|
191
|
-
replica, or use sticky sessions, for `model_id`s to stay valid
|
|
203
|
+
but fitted models live in the process that fitted them, until they expire:
|
|
204
|
+
run one replica, or use sticky sessions, for `model_id`s to stay valid
|
|
205
|
+
across calls.
|
|
192
206
|
Put TLS and whatever authentication or rate limiting you need in front of it;
|
|
193
207
|
the server itself does neither. `node test-http.mjs` starts it on a free port
|
|
194
|
-
and checks all of the above
|
|
208
|
+
and checks all of the above; with `ENA_MCP_TEST_URL=http://host:port` it
|
|
209
|
+
checks a server that is already running instead.
|
|
210
|
+
|
|
211
|
+
#### Kubernetes
|
|
212
|
+
|
|
213
|
+
The `Dockerfile` builds the server as `node server.js --http` on port 8000
|
|
214
|
+
from the files the package ships plus production dependencies (no token
|
|
215
|
+
needed: the `@qe-libs` packages resolve from a public registry endpoint).
|
|
216
|
+
CI builds it for every main commit that touches those files and re-tags it
|
|
217
|
+
with each `vX.Y.Z` release tag, at
|
|
218
|
+
`registry.gitlab.com/epistemic-analytics/qe-packages/server-ena`.
|
|
219
|
+
|
|
220
|
+
`inst/k8s/` is a Helm chart in the shape the other qe apps use: one replica
|
|
221
|
+
with readiness and liveness on `/healthz`, and an nginx Ingress for
|
|
222
|
+
`hostUrl` with a body limit above the server's, buffering off, and a
|
|
223
|
+
per-client-IP rate limit (`rateLimit` in `values.yaml`: requests per second,
|
|
224
|
+
burst, concurrent connections; over it nginx answers 503). The limiter keys
|
|
225
|
+
on the address nginx sees, so it only tells clients apart if the controller
|
|
226
|
+
takes the real client address from the load balancer; and all claude.ai
|
|
227
|
+
users share Anthropic's egress addresses, so the defaults are generous.
|
|
228
|
+
`ENA_MCP_MODEL_TTL_MIN` and `ENA_MCP_MAX_BODY_MB` come from the chart
|
|
229
|
+
values. `inst/k8s/argo/production.yaml` is the Argo CD Application to copy
|
|
230
|
+
into the argo-deployments repository; its image updater follows the
|
|
231
|
+
semver image tags, so a release rolls out by itself.
|
|
232
|
+
|
|
233
|
+
Locally:
|
|
234
|
+
|
|
235
|
+
```bash
|
|
236
|
+
docker build -t qe-mcp-ena . && docker run --rm -p 8000:8000 qe-mcp-ena
|
|
237
|
+
ENA_MCP_TEST_URL=http://127.0.0.1:8000 node test-http.mjs
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
#### Cloudflare Workers
|
|
241
|
+
|
|
242
|
+
The same server runs as a Worker, without Node. `worker/build.mjs` bundles it
|
|
243
|
+
with esbuild into `worker/dist/`: the files the server reads from disk
|
|
244
|
+
(fonts, the qeviz bundle, the skill text) are embedded, and the WASM modules
|
|
245
|
+
are imported precompiled, as Workers require. Then wrangler runs or deploys
|
|
246
|
+
it (Node 22 or newer for wrangler):
|
|
247
|
+
|
|
248
|
+
```bash
|
|
249
|
+
node worker/build.mjs
|
|
250
|
+
cd worker && npx wrangler dev # http://127.0.0.1:8787/mcp
|
|
251
|
+
cd worker && npx wrangler deploy
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
Workers forbid `eval` and `new Function`, and Emscripten's Embind glue uses
|
|
255
|
+
them unless the module was linked with `-sDYNAMIC_EXECUTION=0`:
|
|
256
|
+
`@qe-libs/libqe-wasm` 0.2.4 and `@qe-libs/rena-wasm` 0.1.17 are the first
|
|
257
|
+
builds linked that way. To test a locally built module instead,
|
|
258
|
+
`ENA_WORKER_LIBQE_DIR` and `ENA_WORKER_LIBENA_DIR` name directories holding
|
|
259
|
+
`libqe.js`/`libqe.wasm` and `libena.js`/`libena.wasm`. Fits need the Workers Paid plan (the free plan's
|
|
260
|
+
10 ms CPU limit is too tight; a fit on 3,800 rows with its PNG takes a few
|
|
261
|
+
hundred milliseconds). The model cache lives in the isolate that served the
|
|
262
|
+
request, so `model_id`s stay valid only while the same isolate keeps
|
|
263
|
+
answering; a shared store (KV or a Durable Object) would make them durable.
|
|
195
264
|
|
|
196
265
|
## Prompts
|
|
197
266
|
|
package/assets.js
ADDED
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* assets.js — the files the server reads from its own package.
|
|
3
|
+
*
|
|
4
|
+
* Node reads them from disk next to this file. A Worker bundle (see
|
|
5
|
+
* worker/) has no disk: it calls setAssets() with the same contents embedded
|
|
6
|
+
* at build time, before server.js is imported. Every accessor is lazy so the
|
|
7
|
+
* Node reads only happen when they are actually used.
|
|
8
|
+
*/
|
|
9
|
+
|
|
10
|
+
import { readFileSync } from 'node:fs';
|
|
11
|
+
import { createRequire } from 'node:module';
|
|
12
|
+
import { fileURLToPath } from 'node:url';
|
|
13
|
+
|
|
14
|
+
// The bundled plot font (Inter, OFL), as paths for the native resvg and as
|
|
15
|
+
// bytes for the WASM one.
|
|
16
|
+
const FONT_FILES = ['Inter-Regular.ttf', 'Inter-Bold.ttf'];
|
|
17
|
+
|
|
18
|
+
const nodeAssets = {
|
|
19
|
+
pkgJson: () => JSON.parse(readFileSync(new URL('./package.json', import.meta.url), 'utf8')),
|
|
20
|
+
renaVersion() {
|
|
21
|
+
try { return JSON.parse(readFileSync(createRequire(import.meta.url).resolve('@qe-libs/rena-wasm/package.json'), 'utf8')).version; }
|
|
22
|
+
catch { return 'unknown'; }
|
|
23
|
+
},
|
|
24
|
+
// The qeviz UMD bundle, inlined into interactive pages.
|
|
25
|
+
qevizJS: () => readFileSync(createRequire(import.meta.url).resolve('@qe-libs/qeviz'), 'utf8'),
|
|
26
|
+
skillMarkdown() {
|
|
27
|
+
try { return readFileSync(new URL('./skill/ena-analysis/SKILL.md', import.meta.url), 'utf8'); }
|
|
28
|
+
catch { return null; }
|
|
29
|
+
},
|
|
30
|
+
// What @resvg/resvg-wasm's initWasm() accepts: bytes here, a compiled
|
|
31
|
+
// WebAssembly.Module in a Worker.
|
|
32
|
+
resvgWasm: () => readFileSync(createRequire(import.meta.url).resolve('@resvg/resvg-wasm/index_bg.wasm')),
|
|
33
|
+
fontPaths: () => FONT_FILES.map(f => fileURLToPath(new URL(`./fonts/${f}`, import.meta.url))),
|
|
34
|
+
fontBuffers: () => FONT_FILES.map(f => readFileSync(new URL(`./fonts/${f}`, import.meta.url))),
|
|
35
|
+
};
|
|
36
|
+
|
|
37
|
+
let impl = nodeAssets;
|
|
38
|
+
|
|
39
|
+
/** Replace the asset source (a Worker bundle does this before importing server.js). */
|
|
40
|
+
export function setAssets(assets) { impl = { ...nodeAssets, ...assets }; }
|
|
41
|
+
|
|
42
|
+
export const pkgJson = () => impl.pkgJson();
|
|
43
|
+
export const renaVersion = () => impl.renaVersion();
|
|
44
|
+
export const skillMarkdown = () => impl.skillMarkdown();
|
|
45
|
+
export const resvgWasm = () => impl.resvgWasm();
|
|
46
|
+
export const fontPaths = () => impl.fontPaths();
|
|
47
|
+
export const fontBuffers = () => impl.fontBuffers();
|
|
48
|
+
|
|
49
|
+
let _qevizJS = null;
|
|
50
|
+
export function qevizJS() {
|
|
51
|
+
// Drop the source-map comment: the .map file is not next to the inlined copy.
|
|
52
|
+
if (!_qevizJS) _qevizJS = String(impl.qevizJS()).replace(/\n\/\/# sourceMappingURL=\S*\s*$/, '\n');
|
|
53
|
+
return _qevizJS;
|
|
54
|
+
}
|
package/manifest.json
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
"manifest_version": "0.2",
|
|
3
3
|
"name": "qe-mcp-ena",
|
|
4
4
|
"display_name": "ENA — Epistemic Network Analysis",
|
|
5
|
-
"version": "0.1.
|
|
5
|
+
"version": "0.1.20",
|
|
6
6
|
"description": "Epistemic Network Analysis on tabular coded data. Runs entirely on your machine — data never leaves your computer.",
|
|
7
7
|
"long_description": "Run Epistemic Network Analysis (ENA), ordered network analysis (ONA) and transmodal analysis (TMA) on your coded data from a conversation with Claude. No R or Python to install.\n\n**Your data stays on your machine**\n\n- The analysis runs locally. Only results — networks, points, statistics and plots — go back to the conversation.\n- Give Claude a file path rather than pasting data; the server reads the file itself.\n- It makes no network connections, starts no other programs, and writes files only when you ask for an export.\n\n**About the install warning**\n\nClaude Desktop shows \"access to everything on your computer\" for every extension outside Anthropic's reviewed directory. It means you are trusting the source, as with any program you install. The source is published on npm as @qe-mcp/ena.\n\nDocumentation and example prompts: https://qe-libs.org/mcp/qe-mcp-ena/",
|
|
8
8
|
"author": {
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@qe-mcp/ena",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.20",
|
|
4
4
|
"description": "Epistemic Network Analysis for Claude and other assistants — runs on your machine",
|
|
5
5
|
"repository": {
|
|
6
6
|
"type": "git",
|
|
@@ -13,6 +13,7 @@
|
|
|
13
13
|
},
|
|
14
14
|
"files": [
|
|
15
15
|
"server.js",
|
|
16
|
+
"assets.js",
|
|
16
17
|
"webena.js",
|
|
17
18
|
"xlsx.js",
|
|
18
19
|
"lines.js",
|
|
@@ -27,7 +28,7 @@
|
|
|
27
28
|
"dependencies": {
|
|
28
29
|
"@modelcontextprotocol/sdk": "^1.4.0",
|
|
29
30
|
"@qe-libs/qeviz": "0.7.0",
|
|
30
|
-
"@qe-libs/rena-wasm": "^0.1.
|
|
31
|
+
"@qe-libs/rena-wasm": "^0.1.17",
|
|
31
32
|
"@resvg/resvg-wasm": "^2.6.2"
|
|
32
33
|
},
|
|
33
34
|
"optionalDependencies": {
|
|
@@ -39,5 +40,8 @@
|
|
|
39
40
|
"epistemic-network-analysis",
|
|
40
41
|
"wasm"
|
|
41
42
|
],
|
|
42
|
-
"license": "GPL-3.0-only"
|
|
43
|
+
"license": "GPL-3.0-only",
|
|
44
|
+
"devDependencies": {
|
|
45
|
+
"wrangler": "^4.156.0"
|
|
46
|
+
}
|
|
43
47
|
}
|
package/server.js
CHANGED
|
@@ -41,7 +41,8 @@ import { randomBytes } from 'crypto';
|
|
|
41
41
|
// written to files, and the file-based tools are not offered.
|
|
42
42
|
const ARGV = process.argv.slice(2);
|
|
43
43
|
const argOf = (flag, dflt) => { const i = ARGV.indexOf(flag); return i >= 0 && ARGV[i + 1] != null ? ARGV[i + 1] : dflt; };
|
|
44
|
-
|
|
44
|
+
// Embedded in a Worker (globalThis.ENA_MCP_EMBEDDED) is always the hosted mode.
|
|
45
|
+
const REMOTE = ARGV.includes('--http') || !!globalThis.ENA_MCP_EMBEDDED;
|
|
45
46
|
const HTTP_PORT = Number(argOf('--port', process.env.PORT ?? 8000));
|
|
46
47
|
const HTTP_HOST = argOf('--host', process.env.HOST ?? '127.0.0.1');
|
|
47
48
|
const HTTP_PATH = '/mcp';
|
|
@@ -53,8 +54,9 @@ if (ARGV.includes('--help') || ARGV.includes('-h')) {
|
|
|
53
54
|
'',
|
|
54
55
|
' (no flags) MCP over stdio — what Claude Desktop, Claude Code and other local clients run.',
|
|
55
56
|
' --http MCP over streamable HTTP at /mcp, for hosting. Data must be passed inline',
|
|
56
|
-
' (data_csv)
|
|
57
|
-
'
|
|
57
|
+
' (data_csv) and is discarded after each call; only fitted models are kept, for',
|
|
58
|
+
' $ENA_MCP_MODEL_TTL_MIN minutes (default 60); nothing is written to disk; ena_profile,',
|
|
59
|
+
' ena_tables and ena_lines are not offered; ena_export returns the ZIP inline. Health check: GET /healthz.',
|
|
58
60
|
' --port N Port for --http (default: $PORT or 8000).',
|
|
59
61
|
' --host H Interface for --http (default: $HOST or 127.0.0.1; use 0.0.0.0 in a container).',
|
|
60
62
|
' --max-body-mb N Largest request body in MB (default: $ENA_MCP_MAX_BODY_MB or 25).',
|
|
@@ -73,17 +75,9 @@ import loadENA, { detectParams, isBinary } from '@qe-libs/rena-wasm';
|
|
|
73
75
|
// qeviz: renderSVG() draws the static plot here, with no DOM; the UMD bundle
|
|
74
76
|
// (self-contained, no external deps) is inlined into the interactive page.
|
|
75
77
|
import { renderSVG as qevizRenderSVG } from '@qe-libs/qeviz';
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
let _qevizJS = null;
|
|
80
|
-
function qevizJS() {
|
|
81
|
-
// The bundle is inlined into pages, so drop its source-map comment
|
|
82
|
-
// (the .map file is not next to the inlined copy).
|
|
83
|
-
if (!_qevizJS) _qevizJS = readFileSync(QEVIZ_BUNDLE, 'utf8')
|
|
84
|
-
.replace(/\n\/\/# sourceMappingURL=\S*\s*$/, '\n');
|
|
85
|
-
return _qevizJS;
|
|
86
|
-
}
|
|
78
|
+
// Package files (qeviz bundle, fonts, skill text, versions): read from disk
|
|
79
|
+
// in Node, embedded at build time in a Worker (see assets.js).
|
|
80
|
+
import { qevizJS, pkgJson, renaVersion, skillMarkdown, resvgWasm, fontBuffers, fontPaths } from './assets.js';
|
|
87
81
|
|
|
88
82
|
// ── CSV parser ────────────────────────────────────────────────────────────────
|
|
89
83
|
|
|
@@ -388,7 +382,7 @@ function buildGmrParams(rows, unitCols, unitLabels, p) {
|
|
|
388
382
|
// rotationSet (= R's rotation.set): centred on the source's centerVec, rotated
|
|
389
383
|
// by its rotation matrix, with its node positions.
|
|
390
384
|
function rotationSetFrom(sourceId, codes, ordered) {
|
|
391
|
-
const src =
|
|
385
|
+
const src = modelGet(sourceId);
|
|
392
386
|
if (!src)
|
|
393
387
|
throw new Error(`rotation_from: unknown model_id '${sourceId}'. Fit the source model first ` +
|
|
394
388
|
`(the ${MAX_MODELS} most recently used models are kept).`);
|
|
@@ -688,30 +682,41 @@ function vizSlug(result, args = {}) {
|
|
|
688
682
|
// ── static SVG renderer ───────────────────────────────────────────────────────
|
|
689
683
|
|
|
690
684
|
// PNG output (the inline image block and the png_plot file) prefers the native
|
|
691
|
-
// @resvg/resvg-js, an OPTIONAL dependency: fastest
|
|
692
|
-
//
|
|
693
|
-
//
|
|
694
|
-
//
|
|
685
|
+
// @resvg/resvg-js, an OPTIONAL dependency: fastest. Where it is absent — the
|
|
686
|
+
// cross-platform .mcpb bundle, or a platform without a prebuilt binary —
|
|
687
|
+
// @resvg/resvg-wasm renders the same image. Both draw plot labels with the
|
|
688
|
+
// bundled Inter (fonts/, OFL) and never scan system fonts: resvg's default
|
|
689
|
+
// loadSystemFonts walks the whole system font set on EVERY render, which on
|
|
690
|
+
// macOS costs ~1.5 s per PNG against ~25 ms with the two bundled files.
|
|
695
691
|
// Loaded lazily; if neither loads, SVG and interactive HTML are still produced.
|
|
692
|
+
//
|
|
693
|
+
// The two builds take the fonts differently. The WASM build takes the bytes
|
|
694
|
+
// (fontBuffers). The native build takes paths (fontFiles): given fontBuffers,
|
|
695
|
+
// resvg-js 2.6 scans the system fonts anyway, loadSystemFonts: false or not,
|
|
696
|
+
// and the 1.5 s is back.
|
|
697
|
+
const FONT_FAMILY = { defaultFontFamily: 'Inter', sansSerifFamily: 'Inter', loadSystemFonts: false };
|
|
698
|
+
// Wrap a Resvg class so every render uses the bundled fonts and asPng() yields
|
|
699
|
+
// a Buffer (the WASM build returns a Uint8Array).
|
|
700
|
+
function withBundledFonts(Resvg, fonts) {
|
|
701
|
+
let font;
|
|
702
|
+
return function FontedResvg(svg, opts = {}) {
|
|
703
|
+
font ??= { ...FONT_FAMILY, ...fonts() };
|
|
704
|
+
const r = new Resvg(svg, { ...opts, font: { ...font, ...opts.font } });
|
|
705
|
+
return { render: () => { const img = r.render(); return { asPng: () => Buffer.from(img.asPng()) }; } };
|
|
706
|
+
};
|
|
707
|
+
}
|
|
696
708
|
let _Resvg; // undefined = not yet attempted; null = unavailable
|
|
697
709
|
async function getResvg() {
|
|
698
710
|
if (_Resvg !== undefined) return _Resvg;
|
|
699
711
|
try {
|
|
700
|
-
|
|
712
|
+
const { Resvg } = await import('@resvg/resvg-js');
|
|
713
|
+
_Resvg = withBundledFonts(Resvg, () => ({ fontFiles: fontPaths() }));
|
|
701
714
|
return _Resvg;
|
|
702
715
|
} catch { /* fall back to WASM */ }
|
|
703
716
|
try {
|
|
704
717
|
const { initWasm, Resvg } = await import('@resvg/resvg-wasm');
|
|
705
|
-
await initWasm(
|
|
706
|
-
|
|
707
|
-
.map(f => readFileSync(new URL(`./fonts/${f}`, import.meta.url)));
|
|
708
|
-
const font = { fontBuffers, defaultFontFamily: 'Inter', sansSerifFamily: 'Inter' };
|
|
709
|
-
// Same shape as the native class — new Resvg(svg, opts).render().asPng()
|
|
710
|
-
// — but asPng() gives a Uint8Array here, so hand back a Buffer as native does.
|
|
711
|
-
_Resvg = function WasmResvg(svg, opts = {}) {
|
|
712
|
-
const r = new Resvg(svg, { ...opts, font });
|
|
713
|
-
return { render: () => { const img = r.render(); return { asPng: () => Buffer.from(img.asPng()) }; } };
|
|
714
|
-
};
|
|
718
|
+
await initWasm(await resvgWasm());
|
|
719
|
+
_Resvg = withBundledFonts(Resvg, () => ({ fontBuffers: fontBuffers() }));
|
|
715
720
|
} catch (e) {
|
|
716
721
|
_Resvg = null;
|
|
717
722
|
process.stderr.write(
|
|
@@ -849,6 +854,18 @@ let modelSeq = 0;
|
|
|
849
854
|
const MAX_ACCUM = 8;
|
|
850
855
|
const MAX_MODELS = 50;
|
|
851
856
|
|
|
857
|
+
// Hosted, the process serves strangers, so it keeps no data: parsed rows and
|
|
858
|
+
// accumulations are not cached (every call re-parses the CSV it was sent, tens
|
|
859
|
+
// of ms), the CSV text is dropped from a model's stored parameters, and a
|
|
860
|
+
// model — points, networks, unit and group labels — is forgotten MODEL_TTL_MS
|
|
861
|
+
// after its last use instead of waiting to be displaced.
|
|
862
|
+
const KEEP_ROWS = !REMOTE;
|
|
863
|
+
const EXPORT_HINT = REMOTE
|
|
864
|
+
? 'or ena_export (with data_csv again, or format="set") for an ENA Web Tool project.'
|
|
865
|
+
: 'or ena_export to open the model in the ENA Web Tool.';
|
|
866
|
+
let MODEL_TTL_MS = REMOTE ? Number(process.env.ENA_MCP_MODEL_TTL_MIN ?? 60) * 60_000 : 0;
|
|
867
|
+
export function setModelTtl(minutes) { MODEL_TTL_MS = Number(minutes) * 60_000; }
|
|
868
|
+
|
|
852
869
|
function cacheGet(map, key) {
|
|
853
870
|
if (!map.has(key)) return undefined;
|
|
854
871
|
const v = map.get(key);
|
|
@@ -861,6 +878,27 @@ function cacheSet(map, key, value, max) {
|
|
|
861
878
|
while (map.size > max) map.delete(map.keys().next().value);
|
|
862
879
|
}
|
|
863
880
|
|
|
881
|
+
function expireModels(now = Date.now()) {
|
|
882
|
+
if (!MODEL_TTL_MS) return;
|
|
883
|
+
for (const [id, e] of modelCache)
|
|
884
|
+
if (now - e._used > MODEL_TTL_MS) modelCache.delete(id);
|
|
885
|
+
}
|
|
886
|
+
function modelGet(model_id) {
|
|
887
|
+
expireModels();
|
|
888
|
+
const e = cacheGet(modelCache, model_id);
|
|
889
|
+
if (e) e._used = Date.now();
|
|
890
|
+
return e;
|
|
891
|
+
}
|
|
892
|
+
function modelSet(model_id, entry) {
|
|
893
|
+
expireModels();
|
|
894
|
+
if (!KEEP_ROWS && entry.params) {
|
|
895
|
+
entry._data = dataStamp(entry.params); // to check resent data against
|
|
896
|
+
entry.params = { ...entry.params, data_csv: undefined };
|
|
897
|
+
}
|
|
898
|
+
entry._used = Date.now();
|
|
899
|
+
cacheSet(modelCache, model_id, entry, MAX_MODELS);
|
|
900
|
+
}
|
|
901
|
+
|
|
864
902
|
// Identifies the input data. For a file, include its mtime and size so an
|
|
865
903
|
// edited CSV is re-read instead of serving a model fitted on the old contents.
|
|
866
904
|
function dataStamp(p) {
|
|
@@ -1027,13 +1065,13 @@ function loadRows(p) {
|
|
|
1027
1065
|
const key = dataStamp(p);
|
|
1028
1066
|
let rows;
|
|
1029
1067
|
if (rowsCache.has(key)) rows = cacheGet(rowsCache, key);
|
|
1030
|
-
else { rows = loadData(p.data_path, p.data_csv); cacheSet(rowsCache, key, rows, MAX_ROWS); }
|
|
1068
|
+
else { rows = loadData(p.data_path, p.data_csv); if (KEEP_ROWS) cacheSet(rowsCache, key, rows, MAX_ROWS); }
|
|
1031
1069
|
if (!toArray(p.exclude_conversations).length && !p.order_by) return rows;
|
|
1032
1070
|
|
|
1033
1071
|
const vKey = JSON.stringify([key, toArray(p.conversations), toArray(p.exclude_conversations), p.order_by ?? null]);
|
|
1034
1072
|
if (rowsCache.has(vKey)) return cacheGet(rowsCache, vKey);
|
|
1035
1073
|
const view = prepareRows(rows, p);
|
|
1036
|
-
cacheSet(rowsCache, vKey, view, MAX_ROWS);
|
|
1074
|
+
if (KEEP_ROWS) cacheSet(rowsCache, vKey, view, MAX_ROWS);
|
|
1037
1075
|
return view;
|
|
1038
1076
|
}
|
|
1039
1077
|
|
|
@@ -1437,7 +1475,7 @@ async function toolTmaFit(ena, params) {
|
|
|
1437
1475
|
result,
|
|
1438
1476
|
};
|
|
1439
1477
|
entry.view = defaultView(params, entry.web);
|
|
1440
|
-
|
|
1478
|
+
modelSet(model_id, entry);
|
|
1441
1479
|
|
|
1442
1480
|
return { ...result, _plot: web.buildPlot(entry, entry.view) };
|
|
1443
1481
|
}
|
|
@@ -1700,9 +1738,9 @@ async function toolFit(ena, params) {
|
|
|
1700
1738
|
// Return cached model if params are identical
|
|
1701
1739
|
const fKey = fitCacheKey(params);
|
|
1702
1740
|
const cachedId = fitCache.get(fKey);
|
|
1703
|
-
|
|
1704
|
-
|
|
1705
|
-
const
|
|
1741
|
+
const cached = cachedId && modelGet(cachedId); // the model may have been evicted or expired
|
|
1742
|
+
if (cached) {
|
|
1743
|
+
const model_id = cachedId, entry = cached;
|
|
1706
1744
|
return {
|
|
1707
1745
|
model_id,
|
|
1708
1746
|
...entry.result,
|
|
@@ -1722,7 +1760,7 @@ async function toolFit(ena, params) {
|
|
|
1722
1760
|
codes, units: unitCols, conversations: convoCols,
|
|
1723
1761
|
window: windowValue(window_size), binary, ...accumOpts(params),
|
|
1724
1762
|
});
|
|
1725
|
-
cacheSet(accumCache, aKey, { rows, acc }, MAX_ACCUM);
|
|
1763
|
+
if (KEEP_ROWS) cacheSet(accumCache, aKey, { rows, acc }, MAX_ACCUM);
|
|
1726
1764
|
}
|
|
1727
1765
|
|
|
1728
1766
|
const { unitLabels, nConnections, connectionNames, nUnits } = acc;
|
|
@@ -1792,7 +1830,7 @@ async function toolFit(ena, params) {
|
|
|
1792
1830
|
...(tests ? ['', ...tests.lines] : []),
|
|
1793
1831
|
'',
|
|
1794
1832
|
`Use ena_plot with model_id="${model_id}" for additional views (group, unit, subtraction, comparison), ` +
|
|
1795
|
-
|
|
1833
|
+
EXPORT_HINT,
|
|
1796
1834
|
].join('\n');
|
|
1797
1835
|
|
|
1798
1836
|
const result = {
|
|
@@ -1822,7 +1860,7 @@ async function toolFit(ena, params) {
|
|
|
1822
1860
|
result,
|
|
1823
1861
|
};
|
|
1824
1862
|
entry.view = defaultView(params, entry.web);
|
|
1825
|
-
|
|
1863
|
+
modelSet(model_id, entry);
|
|
1826
1864
|
cacheSet(fitCache, fKey, model_id, MAX_MODELS);
|
|
1827
1865
|
|
|
1828
1866
|
return { ...result, _plot: web.buildPlot(entry, entry.view) };
|
|
@@ -1943,9 +1981,9 @@ async function toolCompareGroups(ena, params) {
|
|
|
1943
1981
|
// Return cached model if params are identical
|
|
1944
1982
|
const fKey = fitCacheKey(params, 'compare');
|
|
1945
1983
|
const cachedId = fitCache.get(fKey);
|
|
1946
|
-
|
|
1947
|
-
|
|
1948
|
-
const
|
|
1984
|
+
const cached = cachedId && modelGet(cachedId); // the model may have been evicted or expired
|
|
1985
|
+
if (cached) {
|
|
1986
|
+
const model_id = cachedId, entry = cached;
|
|
1949
1987
|
return {
|
|
1950
1988
|
model_id,
|
|
1951
1989
|
...entry.result,
|
|
@@ -1969,7 +2007,7 @@ async function toolCompareGroups(ena, params) {
|
|
|
1969
2007
|
codes, units: unitCols, conversations: convoCols,
|
|
1970
2008
|
window: windowValue(window_size), binary, ...accumOpts(params),
|
|
1971
2009
|
});
|
|
1972
|
-
cacheSet(accumCache, aKey, { rows, acc }, MAX_ACCUM);
|
|
2010
|
+
if (KEEP_ROWS) cacheSet(accumCache, aKey, { rows, acc }, MAX_ACCUM);
|
|
1973
2011
|
}
|
|
1974
2012
|
|
|
1975
2013
|
const { unitLabels, connectionNames, nUnits, nConnections } = acc;
|
|
@@ -2064,7 +2102,7 @@ async function toolCompareGroups(ena, params) {
|
|
|
2064
2102
|
),
|
|
2065
2103
|
'',
|
|
2066
2104
|
`Use ena_plot with model_id="${model_id}" for group, subtraction, comparison, or unit views, ` +
|
|
2067
|
-
|
|
2105
|
+
EXPORT_HINT,
|
|
2068
2106
|
].join('\n');
|
|
2069
2107
|
|
|
2070
2108
|
const result = {
|
|
@@ -2101,7 +2139,7 @@ async function toolCompareGroups(ena, params) {
|
|
|
2101
2139
|
result,
|
|
2102
2140
|
};
|
|
2103
2141
|
entry.view = defaultView(params, entry.web);
|
|
2104
|
-
|
|
2142
|
+
modelSet(model_id, entry);
|
|
2105
2143
|
cacheSet(fitCache, fKey, model_id, MAX_MODELS);
|
|
2106
2144
|
|
|
2107
2145
|
return { ...result, _plot: web.buildPlot(entry, entry.view) };
|
|
@@ -2180,7 +2218,7 @@ async function toolPlot({ model_id, type = 'all', group_value, group1, group2, u
|
|
|
2180
2218
|
x_dim = 1, y_dim = 2, flip_x = false, flip_y = false,
|
|
2181
2219
|
min_edge_weight, hide_unconnected_codes, text_size, colors,
|
|
2182
2220
|
confidence_intervals, outlier_intervals }) {
|
|
2183
|
-
const entry =
|
|
2221
|
+
const entry = modelGet(model_id);
|
|
2184
2222
|
if (!entry)
|
|
2185
2223
|
throw new Error(`Unknown model_id '${model_id}'. Call ena_fit, ena_compare_groups or tma_fit first ` +
|
|
2186
2224
|
`(the ${MAX_MODELS} most recently used models are kept; refitting is cheap).`);
|
|
@@ -2354,7 +2392,7 @@ function csvCell(v) {
|
|
|
2354
2392
|
const csvText = (header, rows) => [header, ...rows].map(r => r.map(csvCell).join(',')).join('\n') + '\n';
|
|
2355
2393
|
|
|
2356
2394
|
async function toolTables(ena, { model_id, out_path }) {
|
|
2357
|
-
const entry =
|
|
2395
|
+
const entry = modelGet(model_id);
|
|
2358
2396
|
if (!entry)
|
|
2359
2397
|
throw new Error(`Unknown model_id '${model_id}'. Fit the model first (ena_fit, ena_compare_groups or tma_fit).`);
|
|
2360
2398
|
const { model, codes, unitLabels } = entry;
|
|
@@ -2409,12 +2447,20 @@ async function toolTables(ena, { model_id, out_path }) {
|
|
|
2409
2447
|
// then, as the web tool does, refit without the removed codes.
|
|
2410
2448
|
const PRIA_MIN = 0.8, PRIA_MAX = 0.99;
|
|
2411
2449
|
|
|
2412
|
-
async function toolSimplify(ena, { model_id, threshold = 0.95, remove_num = 3 }) {
|
|
2413
|
-
const entry =
|
|
2450
|
+
async function toolSimplify(ena, { model_id, threshold = 0.95, remove_num = 3, data_csv }) {
|
|
2451
|
+
const entry = modelGet(model_id);
|
|
2414
2452
|
if (!entry)
|
|
2415
2453
|
throw new Error(`Unknown model_id '${model_id}'. Fit the model first (ena_fit, ena_compare_groups or tma_fit).`);
|
|
2416
2454
|
if (!entry.fitArgs) throw new Error(`Model ${model_id} can't be simplified. Refit it with this server version.`);
|
|
2417
|
-
|
|
2455
|
+
let p = entry.params ?? {};
|
|
2456
|
+
if (!KEEP_ROWS) {
|
|
2457
|
+
// The hosted server kept nothing but the model: PRIA refits, so the
|
|
2458
|
+
// same data must come again, and it must be the same data.
|
|
2459
|
+
if (!data_csv) throw new Error('Hosted server: the data is not stored. Pass data_csv again (the CSV the model was fitted on).');
|
|
2460
|
+
if (dataStamp({ data_csv }) !== entry._data)
|
|
2461
|
+
throw new Error(`data_csv is not the data model ${model_id} was fitted on. Pass the same CSV, or fit a new model.`);
|
|
2462
|
+
p = { ...p, data_csv };
|
|
2463
|
+
}
|
|
2418
2464
|
const codes = entry.codes;
|
|
2419
2465
|
if (codes.length < 4)
|
|
2420
2466
|
throw new Error(`Model ${model_id} has ${codes.length} codes; PRIA needs more than 3 (it never leaves fewer than 3).`);
|
|
@@ -2461,7 +2507,7 @@ async function toolSimplify(ena, { model_id, threshold = 0.95, remove_num = 3 })
|
|
|
2461
2507
|
const DEFAULT_MAX_WINDOWS = 500;
|
|
2462
2508
|
|
|
2463
2509
|
async function toolLines(ena, { model_id, connection, units, group, columns, max_windows, out_path }) {
|
|
2464
|
-
const entry =
|
|
2510
|
+
const entry = modelGet(model_id);
|
|
2465
2511
|
if (!entry)
|
|
2466
2512
|
throw new Error(`Unknown model_id '${model_id}'. Fit the model first (ena_fit, ena_compare_groups or tma_fit).`);
|
|
2467
2513
|
const { model, codes, unitLabels } = entry;
|
|
@@ -2595,8 +2641,8 @@ async function toolLines(ena, { model_id, connection, units, group, columns, max
|
|
|
2595
2641
|
};
|
|
2596
2642
|
}
|
|
2597
2643
|
|
|
2598
|
-
async function toolExport(ena, { model_id, set_name, out_path }) {
|
|
2599
|
-
const entry =
|
|
2644
|
+
async function toolExport(ena, { model_id, set_name, out_path, data_csv, format }) {
|
|
2645
|
+
const entry = modelGet(model_id);
|
|
2600
2646
|
if (!entry)
|
|
2601
2647
|
throw new Error(`Unknown model_id '${model_id}'. Fit the model first (ena_fit, ena_compare_groups or tma_fit).`);
|
|
2602
2648
|
const blockers = exportBlockers(entry);
|
|
@@ -2604,9 +2650,34 @@ async function toolExport(ena, { model_id, set_name, out_path }) {
|
|
|
2604
2650
|
throw new Error(`Model ${model_id} can't be exported to the ENA Web Tool, which has no setting for: ${blockers.join('; ')}. ` +
|
|
2605
2651
|
'Refit without these to export it.');
|
|
2606
2652
|
|
|
2607
|
-
|
|
2653
|
+
let p = entry.params;
|
|
2654
|
+
if (KEEP_ROWS) {
|
|
2655
|
+
format = 'zip';
|
|
2656
|
+
} else {
|
|
2657
|
+
// The hosted server kept nothing but the model. The project ZIP holds
|
|
2658
|
+
// the data file, so the data must come again (and be the same data);
|
|
2659
|
+
// the set file alone needs no data unless the set stores excluded
|
|
2660
|
+
// units or conversations, which are read from the rows.
|
|
2661
|
+
format = format ?? 'zip';
|
|
2662
|
+
if (format !== 'zip' && format !== 'set')
|
|
2663
|
+
throw new Error('format must be "zip" (the project ZIP, which needs data_csv again) or "set" (the set file alone).');
|
|
2664
|
+
if (data_csv) {
|
|
2665
|
+
if (dataStamp({ data_csv }) !== entry._data)
|
|
2666
|
+
throw new Error(`data_csv is not the data model ${model_id} was fitted on. Pass the same CSV, or fit a new model.`);
|
|
2667
|
+
p = { ...p, data_csv };
|
|
2668
|
+
} else if (format === 'zip') {
|
|
2669
|
+
throw new Error('Hosted server: the data is not stored. Pass data_csv again (the CSV the model was fitted on) to build the ZIP, ' +
|
|
2670
|
+
'or format="set" for the set file alone.');
|
|
2671
|
+
}
|
|
2672
|
+
}
|
|
2608
2673
|
const unitCols = toArray(p.units);
|
|
2609
2674
|
const warnings = [];
|
|
2675
|
+
let _rows = null;
|
|
2676
|
+
const fileRows = (why) => {
|
|
2677
|
+
if (!p.data_path && p.data_csv == null)
|
|
2678
|
+
throw new Error(`Hosted server: this set needs the rows (${why}). Pass data_csv again (the CSV the model was fitted on).`);
|
|
2679
|
+
return _rows ??= loadRows({ data_path: p.data_path, data_csv: p.data_csv });
|
|
2680
|
+
};
|
|
2610
2681
|
|
|
2611
2682
|
// The web tool groups units by their first unit column; take that column's
|
|
2612
2683
|
// groups (order, colours) for the stored set.
|
|
@@ -2617,8 +2688,7 @@ async function toolExport(ena, { model_id, set_name, out_path }) {
|
|
|
2617
2688
|
if (entry.groupCol === groupCol) {
|
|
2618
2689
|
groups = entry.web.groups.map(g => ({ name: g.name, color: g.color, secondaryColor: g.secondaryColor, showMean: g.showMean }));
|
|
2619
2690
|
} else {
|
|
2620
|
-
|
|
2621
|
-
groups = web.groupOrder(rows, groupCol).map((name, i) => ({ name, ...web.groupStyle(i) }));
|
|
2691
|
+
groups = web.groupOrder(fileRows(`the ${groupCol} groups`), groupCol).map((name, i) => ({ name, ...web.groupStyle(i) }));
|
|
2622
2692
|
if (entry.groupCol)
|
|
2623
2693
|
warnings.push(`The web tool groups units by ${groupCol}, so its plots show ${groupCol} groups, not ${entry.groupCol}.`);
|
|
2624
2694
|
}
|
|
@@ -2633,9 +2703,6 @@ async function toolExport(ena, { model_id, set_name, out_path }) {
|
|
|
2633
2703
|
|
|
2634
2704
|
// The web tool's project holds a CSV; an .xlsx goes in as its first sheet.
|
|
2635
2705
|
const csvName = p.data_path ? basename(p.data_path).replace(/\.xlsx$/i, '.csv') : 'data.csv';
|
|
2636
|
-
const csv = p.data_path
|
|
2637
|
-
? (isXlsx(p.data_path) ? Buffer.from(dataFileText(p.data_path), 'utf8') : readFileSync(p.data_path))
|
|
2638
|
-
: Buffer.from(p.data_csv ?? '', 'utf8');
|
|
2639
2706
|
const setName = safeFileName(set_name ??
|
|
2640
2707
|
(plotted.secondary ? `${plotted.primary} vs ${plotted.secondary}` : `ENA model ${model_id}`));
|
|
2641
2708
|
|
|
@@ -2644,7 +2711,6 @@ async function toolExport(ena, { model_id, set_name, out_path }) {
|
|
|
2644
2711
|
// Built from the whole file: the web tool applies exclude_conversations
|
|
2645
2712
|
// and order_by itself (WasmWrapper._modelRowIndices), on the data file the
|
|
2646
2713
|
// ZIP carries unchanged.
|
|
2647
|
-
const fileRows = loadRows({ data_path: p.data_path, data_csv: p.data_csv });
|
|
2648
2714
|
const exclConvos = toArray(p.exclude_conversations).map(e => String(e).split('__').join('::'));
|
|
2649
2715
|
let unitTree = null, excluded = 0;
|
|
2650
2716
|
if (p.units_used?.length || exclConvos.length) {
|
|
@@ -2653,7 +2719,7 @@ async function toolExport(ena, { model_id, set_name, out_path }) {
|
|
|
2653
2719
|
// rebuilds the conversation tree, which drops its include flags.
|
|
2654
2720
|
const used = p.units_used?.length ? new Set(p.units_used.map(String)) : null;
|
|
2655
2721
|
const seen = new Map();
|
|
2656
|
-
for (const r of fileRows) {
|
|
2722
|
+
for (const r of fileRows('excluded units')) {
|
|
2657
2723
|
const values = unitCols.map(c => String(r[c]));
|
|
2658
2724
|
const key = values.join('__');
|
|
2659
2725
|
if (!seen.has(key)) seen.set(key, { values, include: used ? used.has(key) : true });
|
|
@@ -2666,7 +2732,7 @@ async function toolExport(ena, { model_id, set_name, out_path }) {
|
|
|
2666
2732
|
const convoCols = toArray(p.conversations);
|
|
2667
2733
|
const out = new Set(exclConvos);
|
|
2668
2734
|
const seen = new Map();
|
|
2669
|
-
for (const r of fileRows) {
|
|
2735
|
+
for (const r of fileRows('excluded conversations')) {
|
|
2670
2736
|
const values = convoCols.map(c => String(r[c]));
|
|
2671
2737
|
const key = values.join('::');
|
|
2672
2738
|
if (!seen.has(key)) seen.set(key, { values, include: !out.has(key) });
|
|
@@ -2690,19 +2756,6 @@ async function toolExport(ena, { model_id, set_name, out_path }) {
|
|
|
2690
2756
|
source: { tool: PKG.name, version: PKG.version, model_id, exported: new Date().toISOString() },
|
|
2691
2757
|
});
|
|
2692
2758
|
|
|
2693
|
-
const zipBuf = web.zip([
|
|
2694
|
-
{ name: csvName, data: csv },
|
|
2695
|
-
{ name: 'Folder/' },
|
|
2696
|
-
{ name: `Folder/${setName}.json`, data: JSON.stringify(set, null, 2) },
|
|
2697
|
-
]);
|
|
2698
|
-
|
|
2699
|
-
let target = out_path
|
|
2700
|
-
? (extname(out_path).toLowerCase() === '.zip' ? out_path : join(out_path, `${setName}.zip`))
|
|
2701
|
-
: join(p.data_path ? dirname(p.data_path) : VIZ_DIR, `${setName}.zip`);
|
|
2702
|
-
target = freePath(target);
|
|
2703
|
-
mkdirSync(dirname(target), { recursive: true });
|
|
2704
|
-
writeFileSync(target, zipBuf);
|
|
2705
|
-
|
|
2706
2759
|
const rot = set.settings.model;
|
|
2707
2760
|
const settings = {
|
|
2708
2761
|
units: unitCols, conversations: toArray(p.conversations), codes: entry.codes,
|
|
@@ -2717,14 +2770,82 @@ async function toolExport(ena, { model_id, set_name, out_path }) {
|
|
|
2717
2770
|
...(p.order_by && { order_by: p.order_by }),
|
|
2718
2771
|
...(toArray(p.mask_connections).length && { masked_connections: toArray(p.mask_connections) }),
|
|
2719
2772
|
};
|
|
2773
|
+
const howToOpen = [
|
|
2774
|
+
'To open it: start a local session in the ENA Web Tool ("Run Offline"), click Import in the Sets panel,',
|
|
2775
|
+
`choose the ZIP, then open Folder › ${setName}. (Project import works in local sessions only.)`,
|
|
2776
|
+
];
|
|
2777
|
+
|
|
2778
|
+
if (format === 'set') {
|
|
2779
|
+
// Hosted, without the data: the set file alone. The user packs the
|
|
2780
|
+
// project themselves, so their data never has to come back here.
|
|
2781
|
+
return {
|
|
2782
|
+
set_name: setName,
|
|
2783
|
+
data_file: csvName,
|
|
2784
|
+
set_file: `Folder/${setName}.json`,
|
|
2785
|
+
set,
|
|
2786
|
+
settings,
|
|
2787
|
+
warnings,
|
|
2788
|
+
privacy_note: 'Hosted server: this is the set file only — settings, groups and labels, no rows. Nothing was stored.',
|
|
2789
|
+
summary: [
|
|
2790
|
+
`Set file for model ${model_id} (ENA Web Tool), without the data.`,
|
|
2791
|
+
` Set: Folder / ${setName} Opens with: ${settings.opens_with}`,
|
|
2792
|
+
...warnings.map(w => ` Note: ${w}`),
|
|
2793
|
+
'',
|
|
2794
|
+
'To make the project ZIP: save the "set" field as JSON at Folder/' + setName + '.json, put the data file',
|
|
2795
|
+
`the model was fitted on beside it as ${csvName} (that exact name), and zip the two —`,
|
|
2796
|
+
`the ZIP holds ${csvName} and Folder/${setName}.json.`,
|
|
2797
|
+
...howToOpen,
|
|
2798
|
+
].join('\n'),
|
|
2799
|
+
};
|
|
2800
|
+
}
|
|
2801
|
+
|
|
2802
|
+
const csv = p.data_path
|
|
2803
|
+
? (isXlsx(p.data_path) ? Buffer.from(dataFileText(p.data_path), 'utf8') : readFileSync(p.data_path))
|
|
2804
|
+
: Buffer.from(p.data_csv ?? '', 'utf8');
|
|
2805
|
+
const zipBuf = web.zip([
|
|
2806
|
+
{ name: csvName, data: csv },
|
|
2807
|
+
{ name: 'Folder/' },
|
|
2808
|
+
{ name: `Folder/${setName}.json`, data: JSON.stringify(set, null, 2) },
|
|
2809
|
+
]);
|
|
2810
|
+
|
|
2811
|
+
if (!KEEP_ROWS) {
|
|
2812
|
+
// Hosted: nowhere to write, so the ZIP rides along with the result as
|
|
2813
|
+
// an embedded resource (base64). Nothing of it stays on the server.
|
|
2814
|
+
return {
|
|
2815
|
+
_zip: { name: `${setName}.zip`, data: zipBuf },
|
|
2816
|
+
zip_file: `${setName}.zip`,
|
|
2817
|
+
zip_bytes: zipBuf.length,
|
|
2818
|
+
set_name: setName,
|
|
2819
|
+
data_file: csvName,
|
|
2820
|
+
settings,
|
|
2821
|
+
warnings,
|
|
2822
|
+
privacy_note: 'Hosted server: the ZIP holds the data sent with this call. The server kept neither the data nor the ZIP.',
|
|
2823
|
+
summary: [
|
|
2824
|
+
`Exported model ${model_id} as an ENA Web Tool project ZIP: ${setName}.zip (${zipBuf.length} bytes), ` +
|
|
2825
|
+
'attached to this result as a base64 resource (application/zip). Save it as a .zip file.',
|
|
2826
|
+
` Set: Folder / ${setName} Data: ${csvName}`,
|
|
2827
|
+
` Opens with: ${settings.opens_with}`,
|
|
2828
|
+
...warnings.map(w => ` Note: ${w}`),
|
|
2829
|
+
'',
|
|
2830
|
+
...howToOpen,
|
|
2831
|
+
].join('\n'),
|
|
2832
|
+
};
|
|
2833
|
+
}
|
|
2834
|
+
|
|
2835
|
+
let target = out_path
|
|
2836
|
+
? (extname(out_path).toLowerCase() === '.zip' ? out_path : join(out_path, `${setName}.zip`))
|
|
2837
|
+
: join(p.data_path ? dirname(p.data_path) : VIZ_DIR, `${setName}.zip`);
|
|
2838
|
+
target = freePath(target);
|
|
2839
|
+
mkdirSync(dirname(target), { recursive: true });
|
|
2840
|
+
writeFileSync(target, zipBuf);
|
|
2841
|
+
|
|
2720
2842
|
const summary = [
|
|
2721
2843
|
`Exported model ${model_id} as an ENA Web Tool project: ${target}`,
|
|
2722
2844
|
` Set: Folder / ${setName} Data: ${csvName}`,
|
|
2723
2845
|
` Opens with: ${settings.opens_with}`,
|
|
2724
2846
|
...warnings.map(w => ` Note: ${w}`),
|
|
2725
2847
|
'',
|
|
2726
|
-
|
|
2727
|
-
`choose the ZIP, then open Folder › ${setName}. (Project import works in local sessions only.)`,
|
|
2848
|
+
...howToOpen,
|
|
2728
2849
|
].join('\n');
|
|
2729
2850
|
|
|
2730
2851
|
return {
|
|
@@ -2793,8 +2914,8 @@ const ROTATION_PROPS = {
|
|
|
2793
2914
|
|
|
2794
2915
|
const GUIDE = (() => {
|
|
2795
2916
|
let md;
|
|
2796
|
-
|
|
2797
|
-
|
|
2917
|
+
md = skillMarkdown();
|
|
2918
|
+
if (md == null) return null;
|
|
2798
2919
|
md = md.replace(/^---\n[\s\S]*?\n---\n+/, ''); // frontmatter
|
|
2799
2920
|
const sections = [];
|
|
2800
2921
|
for (const part of md.split(/^(?=## )/m).slice(1)) {
|
|
@@ -2812,7 +2933,8 @@ function toolGuide({ topic } = {}) {
|
|
|
2812
2933
|
const header = REMOTE
|
|
2813
2934
|
? '> **Hosted server.** This server runs remotely, not on the user\'s machine. Where the guide says to pass ' +
|
|
2814
2935
|
'data_path, pass the data inline as data_csv instead, after the user confirms it is de-identified. ' +
|
|
2815
|
-
'ena_profile, ena_tables
|
|
2936
|
+
'ena_profile, ena_tables and ena_lines are not available here; plots come back inline, and ena_export returns ' +
|
|
2937
|
+
'the project ZIP inline (with data_csv again) or the set file alone (format="set").\n\n'
|
|
2816
2938
|
: '';
|
|
2817
2939
|
if (!topic) return header + GUIDE.text + footer;
|
|
2818
2940
|
const t = String(topic).toLowerCase().trim();
|
|
@@ -3111,6 +3233,7 @@ const TOOLS = [
|
|
|
3111
3233
|
model_id: { type: 'string', description: 'model_id from ena_fit, ena_compare_groups or tma_fit' },
|
|
3112
3234
|
threshold: { type: 'number', minimum: 0.8, maximum: 0.99, description: 'How close the simplified model must stay to the full one (default 0.95; the web tool allows 0.80–0.99).' },
|
|
3113
3235
|
remove_num: { type: 'integer', minimum: 1, description: 'Most codes to remove (default 3).' },
|
|
3236
|
+
data_csv: { type: 'string', description: 'Hosted server only: the CSV the model was fitted on, again (the server does not store it). Locally, omit.', _hostedOnly: true },
|
|
3114
3237
|
},
|
|
3115
3238
|
required: ['model_id'],
|
|
3116
3239
|
},
|
|
@@ -3164,13 +3287,16 @@ const TOOLS = [
|
|
|
3164
3287
|
'it rebuilds the same model and opens on the same comparison plot. ' +
|
|
3165
3288
|
'Excluded units (units_used) are stored as excluded in the web tool\'s Units tab. Models the web tool has no setting for (transmodal or time windows, a GMR on x_var, rotations by a column other than the first unit column, ' +
|
|
3166
3289
|
'center_align_to_origin=false) are refused with the reason. ' +
|
|
3167
|
-
'PRIVACY: the ZIP contains the data file. It is written locally (by default next to the data file, never overwriting one) and no rows are sent to Claude.'
|
|
3290
|
+
'PRIVACY: the ZIP contains the data file. It is written locally (by default next to the data file, never overwriting one) and no rows are sent to Claude. ' +
|
|
3291
|
+
'HOSTED: there is no disk; the ZIP comes back inline, built from data_csv sent again with the call, or format="set" returns the set file alone for the user to pack with their own data file.',
|
|
3168
3292
|
inputSchema: {
|
|
3169
3293
|
type: 'object',
|
|
3170
3294
|
properties: {
|
|
3171
3295
|
model_id: { type: 'string', description: 'model_id of the model to export' },
|
|
3172
3296
|
set_name: { type: 'string', description: 'Name of the set in the web tool (default: "<group 1> vs <group 2>", or "ENA model <id>")' },
|
|
3173
|
-
out_path: { type: 'string', description: 'Where to write: a .zip path, or a folder. Default: the data file\'s folder, as "<set name>.zip" (a number is added rather than overwrite).' },
|
|
3297
|
+
out_path: { type: 'string', description: 'Where to write: a .zip path, or a folder. Default: the data file\'s folder, as "<set name>.zip" (a number is added rather than overwrite).', _localOnly: true },
|
|
3298
|
+
format: { type: 'string', enum: ['zip', 'set'], description: 'Hosted server only. "zip" (default): the project ZIP, returned inline as a base64 resource; needs data_csv again. "set": the set file alone (no data needed, nothing sent back but settings); the user packs it with their own data file as the summary says. Locally, omit.', _hostedOnly: true },
|
|
3299
|
+
data_csv: { type: 'string', description: 'Hosted server only: the CSV the model was fitted on, again (the server does not store it). Required for format "zip". Locally, omit.', _hostedOnly: true },
|
|
3174
3300
|
},
|
|
3175
3301
|
required: ['model_id'],
|
|
3176
3302
|
},
|
|
@@ -3180,13 +3306,14 @@ const TOOLS = [
|
|
|
3180
3306
|
// Hosted: no user files to read or write, so the file tools go, data_path goes
|
|
3181
3307
|
// from every schema, and the privacy wording changes from "keep it local" to
|
|
3182
3308
|
// "send it de-identified".
|
|
3183
|
-
const REMOTE_HIDDEN_TOOLS = new Set(['ena_profile', 'ena_tables', 'ena_lines'
|
|
3184
|
-
const REMOTE_PRIVACY = 'PRIVACY (hosted server): the data is sent inline and processed in memory; it is
|
|
3309
|
+
const REMOTE_HIDDEN_TOOLS = new Set(['ena_profile', 'ena_tables', 'ena_lines']);
|
|
3310
|
+
const REMOTE_PRIVACY = 'PRIVACY (hosted server): the data is sent inline and processed in memory; it is discarded when the call returns, and only the fitted model is kept, for a limited time. ' +
|
|
3185
3311
|
'Before sending it, confirm with the user that it is de-identified. ';
|
|
3186
3312
|
function remoteTools(tools) {
|
|
3187
3313
|
return tools.filter(t => !REMOTE_HIDDEN_TOOLS.has(t.name)).map(t => {
|
|
3188
|
-
const { data_path, ...properties } = t.inputSchema.properties ?? {};
|
|
3314
|
+
const { data_path, ...properties } = Object.fromEntries(Object.entries(t.inputSchema.properties ?? {}).filter(([, v]) => !v._localOnly));
|
|
3189
3315
|
let description = t.description
|
|
3316
|
+
.replace(/ ?It is written locally[^.]*\. HOSTED: /, ' Hosted: ')
|
|
3190
3317
|
.replace(/The ENA network plot is written to disk[^]*?Use these paths to show the plot; /,
|
|
3191
3318
|
'The ENA network plot comes back as an inline image. Use it to show the plot; ')
|
|
3192
3319
|
.replace(/The ENA network plot is written to disk[^]*?is installed\./,
|
|
@@ -3197,25 +3324,32 @@ function remoteTools(tools) {
|
|
|
3197
3324
|
.replace(/If the user has not yet indicated whether their data is sensitive, ask before using data_csv\. ?/, '')
|
|
3198
3325
|
.replace(/If they indicate sensitivity, use ena_profile instead \(no raw data transmitted\)\. ?/, '')
|
|
3199
3326
|
.trim();
|
|
3327
|
+
for (const [k, v] of Object.entries(properties)) if (v._hostedOnly) { const { _hostedOnly, ...d } = v; properties[k] = d; }
|
|
3200
3328
|
const required = (t.inputSchema.required ?? []).filter(r => r !== 'data_path');
|
|
3201
3329
|
if (properties.data_csv && !description.includes(REMOTE_PRIVACY.trim()))
|
|
3202
3330
|
description = `${description} ${REMOTE_PRIVACY.trim()}`;
|
|
3203
|
-
if (properties.data_csv) properties.data_csv = {
|
|
3331
|
+
if (properties.data_csv && !properties.data_csv._hostedOnly) properties.data_csv = {
|
|
3204
3332
|
type: 'string',
|
|
3205
3333
|
description: 'The data as CSV text. The server has no access to the user\'s files; it processes this in memory and does not store it. Send only de-identified data.',
|
|
3206
3334
|
};
|
|
3335
|
+
if (properties.data_csv?._hostedOnly) { const { _hostedOnly, ...d } = properties.data_csv; properties.data_csv = d; }
|
|
3207
3336
|
return { ...t, description, inputSchema: { ...t.inputSchema, properties, ...(required.length ? { required } : {}) } };
|
|
3208
3337
|
});
|
|
3209
3338
|
}
|
|
3210
|
-
|
|
3339
|
+
function localTools(tools) {
|
|
3340
|
+
return tools.map(t => {
|
|
3341
|
+
const properties = Object.fromEntries(Object.entries(t.inputSchema.properties ?? {}).filter(([, v]) => !v._hostedOnly)
|
|
3342
|
+
.map(([k, { _localOnly, ...v }]) => [k, v]));
|
|
3343
|
+
const description = t.description.replace(/ HOSTED: .*$/, '');
|
|
3344
|
+
return { ...t, description, inputSchema: { ...t.inputSchema, properties } };
|
|
3345
|
+
});
|
|
3346
|
+
}
|
|
3347
|
+
const OFFERED_TOOLS = REMOTE ? remoteTools(TOOLS) : localTools(TOOLS);
|
|
3211
3348
|
|
|
3212
3349
|
// ── Server setup ──────────────────────────────────────────────────────────────
|
|
3213
3350
|
|
|
3214
|
-
const PKG =
|
|
3215
|
-
const RENA_VERSION = (
|
|
3216
|
-
try { return JSON.parse(readFileSync(_require.resolve('@qe-libs/rena-wasm/package.json'), 'utf8')).version; }
|
|
3217
|
-
catch { return 'unknown'; }
|
|
3218
|
-
})();
|
|
3351
|
+
const PKG = pkgJson();
|
|
3352
|
+
const RENA_VERSION = renaVersion();
|
|
3219
3353
|
// Stamped on every result and plot, so results from this server can be told
|
|
3220
3354
|
// apart from anything computed in the conversation.
|
|
3221
3355
|
const PROVENANCE = `${PKG.name} ${PKG.version} (rena-wasm ${RENA_VERSION})`;
|
|
@@ -3249,14 +3383,16 @@ function withholdDataValues(msg, args) {
|
|
|
3249
3383
|
|
|
3250
3384
|
// Sent to the client at connection; MCP clients (Claude, Codex, Gemini CLI,
|
|
3251
3385
|
// Cursor, ...) pass it to the model, so the privacy rule reaches every one.
|
|
3252
|
-
const
|
|
3386
|
+
const serverInstructions = () => REMOTE ? [
|
|
3253
3387
|
'ENA / ONA / TMA analysis on a hosted server. The server does not run on the user\'s machine and cannot read their files.',
|
|
3254
3388
|
'',
|
|
3255
3389
|
'Data privacy — follow in every conversation:',
|
|
3256
|
-
'- Data reaches these tools inline, as data_csv. It is
|
|
3390
|
+
'- Data reaches these tools inline, as data_csv. It is parsed in memory for the call and discarded when the call returns: the server keeps no rows and no CSV text, in memory or on disk.',
|
|
3391
|
+
`- What a model_id refers to — the fitted model: points, networks, unit labels and group values, never the rows — stays in memory for ${Math.round(MODEL_TTL_MS / 60_000)} minutes after its last use, then is forgotten. ena_simplify refits and ena_export packs the data file, so they take data_csv again; ena_export with format="set" takes none.`,
|
|
3257
3392
|
'- Before the first analysis, tell the user that the data leaves their machine and must be de-identified: no names, e-mail addresses or identifying free text. Ask them to confirm before sending it.',
|
|
3258
3393
|
'- Results include unit labels (the values of the unit columns, often participant IDs) and group values. If those are identifying, tell the user before fitting so they can pseudonymize the data first.',
|
|
3259
|
-
'- The guide may mention data_path, ena_profile, ena_tables
|
|
3394
|
+
'- The guide may mention data_path, ena_profile, ena_tables or ena_lines; those are for the locally installed server and are not available here.',
|
|
3395
|
+
'- ena_export returns the ENA Web Tool project ZIP inline (a base64 resource) when it gets data_csv again. If the client cannot save a file from a result, use format="set" and give the user the set file and the packing steps from the summary; their data file never has to come back here.',
|
|
3260
3396
|
'',
|
|
3261
3397
|
'Results come only from these tools — follow in every conversation:',
|
|
3262
3398
|
'- Every ENA, ONA or TMA result shown to the user (accumulated networks, points, rotations, statistics, plots) must come from these tools. Do not compute any part of an ENA analysis yourself — in code, a notebook, R, Python or by hand — and do not draw ENA-style network plots of your own.',
|
|
@@ -3329,7 +3465,7 @@ const on = (schema, handler) => HANDLERS.push([schema, handler]);
|
|
|
3329
3465
|
function buildServer() {
|
|
3330
3466
|
const server = new Server(
|
|
3331
3467
|
{ name: 'ena', version: PKG.version },
|
|
3332
|
-
{ capabilities: { tools: {}, prompts: {} }, instructions:
|
|
3468
|
+
{ capabilities: { tools: {}, prompts: {} }, instructions: serverInstructions() }
|
|
3333
3469
|
);
|
|
3334
3470
|
for (const [schema, handler] of HANDLERS) server.setRequestHandler(schema, handler);
|
|
3335
3471
|
return server;
|
|
@@ -3476,10 +3612,14 @@ on(CallToolRequestSchema, async (request) => {
|
|
|
3476
3612
|
}
|
|
3477
3613
|
|
|
3478
3614
|
// Extract viz fields before stringifying so they never appear in the text block.
|
|
3479
|
-
const { _plot, ...cleanResult } = result;
|
|
3615
|
+
const { _plot, _zip, ...cleanResult } = result;
|
|
3480
3616
|
|
|
3481
3617
|
cleanResult.provenance = PROVENANCE;
|
|
3482
3618
|
const content = [{ type: 'text', text: JSON.stringify(cleanResult, null, 2) }];
|
|
3619
|
+
if (_zip) content.push({
|
|
3620
|
+
type: 'resource',
|
|
3621
|
+
resource: { uri: `ena://export/${encodeURIComponent(_zip.name)}`, mimeType: 'application/zip', blob: _zip.data.toString('base64') },
|
|
3622
|
+
});
|
|
3483
3623
|
|
|
3484
3624
|
if (_plot) {
|
|
3485
3625
|
const slug = vizSlug(cleanResult, args) || name;
|
|
@@ -3543,6 +3683,7 @@ on(CallToolRequestSchema, async (request) => {
|
|
|
3543
3683
|
|
|
3544
3684
|
return { content };
|
|
3545
3685
|
} catch (err) {
|
|
3686
|
+
if (process.env.ENA_MCP_DEBUG) process.stderr.write((err.stack ?? '') + '\n');
|
|
3546
3687
|
return {
|
|
3547
3688
|
content: [{ type: 'text', text: `Error: ${withholdDataValues(err.message, args)}\n\n${NO_SUBSTITUTE}` }],
|
|
3548
3689
|
isError: true,
|
|
@@ -3550,10 +3691,43 @@ on(CallToolRequestSchema, async (request) => {
|
|
|
3550
3691
|
}
|
|
3551
3692
|
});
|
|
3552
3693
|
|
|
3553
|
-
// ──
|
|
3694
|
+
// ── Web-standard entry (Workers and other fetch runtimes) ─────────────────────
|
|
3695
|
+
// Same stateless scheme as --http below, over Request/Response. A Worker
|
|
3696
|
+
// bundle imports this and never reaches the start block.
|
|
3697
|
+
|
|
3698
|
+
export { buildServer };
|
|
3699
|
+
|
|
3700
|
+
export async function handleWebRequest(request, { maxBody = HTTP_MAX_BODY } = {}) {
|
|
3701
|
+
const url = new URL(request.url);
|
|
3702
|
+
const json = (status, body) => Response.json(body, { status });
|
|
3703
|
+
if (url.pathname === '/healthz') return json(200, { ok: true, server: PKG.name, version: PKG.version });
|
|
3704
|
+
if (url.pathname !== HTTP_PATH) return json(404, { error: `MCP endpoint is ${HTTP_PATH}` });
|
|
3705
|
+
let parsedBody;
|
|
3706
|
+
if (request.method === 'POST') {
|
|
3707
|
+
const tooLarge = () => json(413, { error: `request body too large (limit ${Math.round(maxBody / 1048576)} MB)` });
|
|
3708
|
+
if (Number(request.headers.get('content-length')) > maxBody) return tooLarge();
|
|
3709
|
+
const raw = await request.text();
|
|
3710
|
+
if (raw.length > maxBody) return tooLarge();
|
|
3711
|
+
try { parsedBody = JSON.parse(raw); }
|
|
3712
|
+
catch { return json(400, { jsonrpc: '2.0', error: { code: -32700, message: 'Parse error' }, id: null }); }
|
|
3713
|
+
}
|
|
3714
|
+
const { WebStandardStreamableHTTPServerTransport } = await import('@modelcontextprotocol/sdk/server/webStandardStreamableHttp.js');
|
|
3715
|
+
const transport = new WebStandardStreamableHTTPServerTransport({ sessionIdGenerator: undefined });
|
|
3716
|
+
await buildServer().connect(transport);
|
|
3717
|
+
return transport.handleRequest(request, { parsedBody });
|
|
3718
|
+
}
|
|
3554
3719
|
|
|
3555
|
-
|
|
3556
|
-
|
|
3720
|
+
// ── Start ─────────────────────────────────────────────────────────────────────
|
|
3721
|
+
// Skipped when embedded (a Worker sets globalThis.ENA_MCP_EMBEDDED before
|
|
3722
|
+
// importing this module and calls handleWebRequest itself).
|
|
3723
|
+
|
|
3724
|
+
if (globalThis.ENA_MCP_EMBEDDED) {
|
|
3725
|
+
// nothing to start
|
|
3726
|
+
} else if (!REMOTE) {
|
|
3727
|
+
buildServer().connect(new StdioServerTransport()).catch((e) => {
|
|
3728
|
+
process.stderr.write(`[ena-mcp] stdio: ${e.message}\n`);
|
|
3729
|
+
process.exit(1);
|
|
3730
|
+
});
|
|
3557
3731
|
} else {
|
|
3558
3732
|
// Streamable HTTP, stateless: every POST gets a fresh Server + transport
|
|
3559
3733
|
// and is answered in full, so no session state lives in the process and
|
|
@@ -3602,4 +3776,5 @@ if (!REMOTE) {
|
|
|
3602
3776
|
http.listen(HTTP_PORT, HTTP_HOST, () => {
|
|
3603
3777
|
process.stderr.write(`[ena-mcp] ${PKG.name} ${PKG.version} serving MCP over HTTP at http://${HTTP_HOST}:${HTTP_PORT}${HTTP_PATH} (health: /healthz)\n`);
|
|
3604
3778
|
});
|
|
3779
|
+
setInterval(expireModels, 60_000).unref(); // idle models leave on time, not at the next request
|
|
3605
3780
|
}
|