@danielsimonjr/mathts-workbook 0.1.8 → 0.3.0

package/README.md

@@ -1,117 +1,247 @@
-# @danielsimonjr/mathts-workbook
-
-Scientific workbook runtime for MathTS with reactive YAML-based notebooks (`.mtsw` format).
-
-## Installation
-
-```bash
-npm install @danielsimonjr/mathts-workbook
-```
-
-## Usage
-
-### CLI
-
-```bash
-# Run a workbook
-mtsw run example.mtsw
-
-# Run specific cell
-mtsw run example.mtsw -c compute
-
-# Validate structure
-mtsw validate example.mtsw
-
-# Show dependency graph
-mtsw graph example.mtsw
-
-# Create from template (basic | tensor-physics | data-science)
-mtsw new my-workbook -t tensor-physics
-```
-
-> Note: `mtsw watch` and `mtsw export` appear in the CLI help text but are not yet implemented.
-
-### Programmatic API
-
-```typescript
-import { parseWorkbook, createExecutor } from '@danielsimonjr/mathts-workbook';
-
-const content = `
-version: "1.0"
-metadata:
-  title: "My Workbook"
-runtime:
-  engine: mathts
-  execution: reactive
-cells:
-  - code: |
-      const x = 42;
-      export { x };
-    id: compute
-`;
-
-const result = parseWorkbook(content);
-if (result.success && result.workbook) {
-  const executor = createExecutor(result.workbook);
-
-  executor.on((event) => {
-    console.log(event.type, event.cellId);
-  });
-
-  await executor.runAll();
-}
-```
-
-## Workbook Format (.mtsw)
-
-```yaml
-version: '1.0'
-metadata:
-  title: 'Matrix Analysis'
-  author: 'Your Name'
-
-runtime:
-  engine: mathts
-  execution: reactive # reactive | sequential | manual
-
-cells:
-  - markdown: |
-      # Introduction
-    id: intro
-
-  - code: |
-      import { Matrix } from '@danielsimonjr/mathts-matrix';
-      const A = Matrix.random(3, 3);
-      export { A };
-    id: create-matrix
-
-  - code: |
-      import { A } from '#create-matrix';
-      const det = A.determinant();
-      console.log('Determinant:', det);
-    id: compute
-    depends_on: [create-matrix]
-```
-
-## Cell Types
-
-| Type            | Description                       |
-| --------------- | --------------------------------- |
-| `markdown`      | Documentation with LaTeX math     |
-| `code`          | TypeScript/JavaScript execution   |
-| `tensor`        | Einstein notation for tensor math |
-| `equation`      | LaTeX equations with labels       |
-| `visualization` | Three.js/D3/Plotly rendering      |
-| `data`          | YAML/JSON/CSV data                |
-| `test`          | Assertions with timeout           |
-| `export`        | Publication output                |
-
-## Execution Modes
-
-- **reactive** - Auto-rerun when dependencies change
-- **sequential** - Top-to-bottom execution
-- **manual** - Explicit trigger only
-
-## License
-
-MIT
+# @danielsimonjr/mathts-workbook
+
+Headless runtime for MathTS scientific notebooks in the YAML-based `.mtsw` format. Load a workbook, run its cells in dependency order, and verify results — all from the terminal or programmatically.
+
+> **Scope:** This is the headless v1 — a CLI and runtime. Code cells evaluate **MathTS expressions** (via the sandboxed expression engine), not arbitrary TypeScript. A desktop GUI is a separate, later project that builds on this runtime.
+
+## Installation
+
+```bash
+npm install @danielsimonjr/mathts-workbook
+```
+
+## CLI
+
+```bash
+# Run a workbook: executes cells in dependency order and prints per-cell results.
+# Exits non-zero if any cell errors or any test assertion fails.
+mtsw run example.mtsw
+mtsw run example.mtsw -v        # also print the execution event stream
+mtsw run example.mtsw --json    # machine-readable envelope on stdout
+mtsw run example.mtsw -c gauss  # run one cell + its transitive deps (stateless)
+mtsw run example.mtsw --timeout 5000   # run in a worker; killed (exit 1) if it exceeds 5s
+
+# Describe the structured document model (cells, outputs, dependency graph).
+mtsw describe example.mtsw --json
+
+# Validate structure: ids, dependency references, and cycles.
+mtsw validate example.mtsw [--json]
+
+# Print the dependency graph (human only; use `describe --json` for structured data).
+mtsw graph example.mtsw                 # text adjacency
+mtsw graph example.mtsw -f mermaid      # Mermaid `graph TD`
+
+# Engine introspection (for tooling / GUIs).
+mtsw capabilities --json                # version, supported cell types, feature flags
+mtsw templates --json                   # available `new` templates
+
+# Scaffold a new workbook (<name>.mtsw) from a template.
+mtsw new my-notebook                    # basic template; refuses to overwrite
+mtsw new my-notebook -t basic --force   # overwrite an existing file
+
+# Strip cell outputs (for git): prints to stdout, or -w to rewrite in place.
+mtsw strip example.mtsw
+mtsw strip example.mtsw -w
+
+# Run and persist outputs back into the file (opt-in; never writes without --write).
+mtsw run example.mtsw --write
+
+# Edit cells (atomic, in-place; --json returns the updated doc, --dry-run previews).
+mtsw cell add example.mtsw --type code --id gauss --content "n*(n+1)/2" --depends-on n
+mtsw cell add example.mtsw --type code --id big --content-file body.txt   # or - for stdin
+mtsw cell edit example.mtsw gauss --content "n*(n-1)/2"
+mtsw cell move example.mtsw gauss --before n          # --before/--after <id> | --at <index>
+mtsw cell rename example.mtsw n count                 # rewrites dependents' depends_on
+mtsw cell rm example.mtsw n                           # refuses if cells depend on it
+mtsw cell rm example.mtsw n --force                   # removes + detaches dependents
+
+# Introspection + metadata.
+mtsw functions --json                  # functions/constants cells can call (autocomplete)
+mtsw meta get example.mtsw             # show workbook metadata
+mtsw meta set example.mtsw --title "My Notebook" --author Ada --tags physics,demo
+
+# Scaffold a workbook (templates: basic | empty | chart; -o writes to any path).
+mtsw new notebook                        # notebook.mtsw in the CWD (basic template)
+mtsw new draft --empty -o docs/draft.mtsw    # blank workbook at an explicit path
+mtsw new demo -t chart                       # a line-chart example workbook
+
+# Build a whole .mtsw from a JSON/YAML document (the inverse of `describe --json`).
+echo '{"cells":[{"id":"a","type":"code","content":"6 * 7"}]}' | mtsw import -o out.mtsw
+mtsw import doc.json --json               # validates (ids/deps/cycles); stdout if no -o
+
+# Render to a self-contained HTML document (runs first, then renders).
+mtsw export example.mtsw -o example.html     # one offline file; stdout if no -o
+mtsw export example.mtsw --no-run            # render cached outputs without executing
+mtsw export example.mtsw --json              # envelope: { data: { path, bytes } }
+
+# Other export formats: --format tex|pdf|json|ipynb (html is the default).
+mtsw export example.mtsw --format ipynb -o example.ipynb   # Jupyter notebook (nbformat v4)
+
+# Persistent session for a GUI/tooling: JSON-RPC 2.0 over stdio (NDJSON).
+mtsw serve
+#   -> {"jsonrpc":"2.0","id":1,"method":"open","params":{"path":"example.mtsw"}}
+#   <- {"jsonrpc":"2.0","id":1,"result":{...describe doc...}}
+#   -> {"jsonrpc":"2.0","id":2,"method":"run"}              # streams cell/event notifications
+#   methods: open/describe/validate/graph/run/cell.*/meta.*/save/capabilities/functions/shutdown
+```
+
+**`export` (self-contained HTML).** `mtsw export <file> --format html` renders a
+notebook to a single offline `.html` with **no external requests**: markdown prose,
+**equations typeset as MathML** (rendered natively by modern browsers — Chromium ≥109,
+Firefox, Safari), code cells with their embedded outputs, ✓/✗ test badges, and (as of
+the chart slice) inline SVG plots. All rendering is MathTS-native — the generators
+(`toMathML`/`toHTML`/`toCSS`, plus `markdownToHtml`) live in the `expression` package
+alongside the node `.toTex()`/`.toHTML()` serializers, with **zero external
+dependencies**. Equation cells contain MathTS expression syntax (e.g.
+`c = 1 / sqrt(eps0 * mu0)`), not raw LaTeX; they are display-only and rendered via
+`toMathML`. By default `export` runs the workbook first (use `--no-run` for cached
+outputs); a whole-run failure such as a dependency cycle fails loudly rather than
+emitting a misleading document.
+
+**`export --format ipynb` (Jupyter notebook).** Renders the same run report to a
+structurally conformant **nbformat v4** JSON document: `markdown` cells map to
+notebook markdown cells; every other cell type (`code`, `equation`, `test`, `data`,
+`visualization`) maps to a notebook code cell, with a computed result becoming an
+`execute_result` output (`data['text/plain']`), an error becoming an `error` output,
+and a chart becoming a `display_data` output (inline SVG). Shares the same
+run-then-render pipeline as `--format html`/`tex` (including `--no-run`, `--json`,
+`-o`).
+
+**`run --timeout <ms>` (kill-able worker-thread run).** By default, cell execution runs
+in-process with no time budget — a runaway cell (e.g. an unbounded computation) hangs
+the process. `--timeout` runs the whole workbook in a `worker_threads` Worker and
+forcibly terminates it if it exceeds the budget, exiting 1 with a clear
+`workbook execution exceeded <ms>ms and was terminated` message; termination kills the
+worker outright, so it interrupts even a synchronous, CPU-bound runaway. It always runs
+the entire workbook to completion-or-termination, so it's incompatible with `-c`/`-v`.
+The same primitive is available programmatically as `runWorkbookWithTimeout(source, {
+timeoutMs })` (throws `WorkbookTimeoutError` on timeout); cell outputs come back
+pre-formatted to strings (via `formatResult`), since engine class instances (Complex,
+matrices, …) don't survive the worker's `postMessage`.
+
+**`serve` (persistent session).** One long-lived process holds the workbook in memory with a per-cell result cache and a **stale set**: a `cell.*` edit marks that cell and its transitive dependents stale, and a `run` re-executes **only** the stale cells (reusing cached outputs for the rest) — the incremental latency win a GUI needs. Requests are processed strictly in order; `run` streams `cell/event` notifications (flushed before that run's response in v1, not mid-run). Edits stay in memory until `save`. Single-document per process; concurrent writers are last-write-wins.
+
+**Editing notes.** Cell edits are validity-preserving: an op that would create a duplicate/invalid id, a missing dependency, or a **dependency cycle** is rejected and the file is left byte-for-byte unchanged. Editing a cell clears its (now-stale) persisted output; `--at N` is the cell's final 0-based index; `--force` detaches dependents (clearing their outputs) but does not rewrite cell _content_, so a dependent that still references the removed id by name will error at run. Concurrent editors are last-write-wins (an optimistic-lock guard arrives with the `serve` session).
+
+Diagnostics and errors are written to stderr; results (including `--json`) go to stdout, so the exit code can be used in scripts independently of the output.
+
+**Machine contract (`--json`).** Every `--json` command emits one envelope on stdout:
+`{ schemaVersion: {major,minor}, command, ok, data, problems }`. The envelope is
+emitted even on failure (and is cycle/BigInt-safe, so it never crashes on
+pathological data); the exit code mirrors `ok` for shells, but tooling/GUIs
+should read `ok` and treat a missing/unparseable envelope as the only transport
+error. Compatibility rule: ignore unknown fields when `major` matches; refuse on
+a `major` mismatch. `run --cell <id>` is stateless (it recomputes the target's
+transitive deps each call; `run --cell --write` persists only the executed
+cells). This is the contract a GUI binds to; a persistent `serve` mode (streaming
+events, incremental re-execution) is planned.
+
+**Saving / round-trip.** `serializeWorkbook` (and the write commands above) round-trips a workbook through the parser: structure is preserved exactly, and persisted `output` values round-trip best-effort (plain numbers/strings/arrays/objects exactly; exotic types to their plain shape). All writes are **atomic** (temp file + rename). Note that any write path is parse→serialize and therefore **drops YAML comments and re-orders keys** — a CST-preserving in-place rewrite is a future enhancement.
+
+## Programmatic API
+
+```typescript
+import { parseWorkbook, createExecutor, formatResult } from '@danielsimonjr/mathts-workbook';
+
+const content = `
+version: "1.0"
+metadata:
+  title: "My Workbook"
+runtime:
+  engine: mathts
+  execution: reactive
+cells:
+  - code: "n * (n + 1) / 2"
+    id: gaussSum
+    depends_on: [n]
+  - code: "10"
+    id: n
+  - test: "gaussSum == 55"
+    id: checkGauss
+    depends_on: [gaussSum]
+`;
+
+const result = parseWorkbook(content);
+if (result.success && result.workbook) {
+  const report = await createExecutor(result.workbook).runReport();
+  for (const cell of report.cells) {
+    console.log(cell.id, cell.status, formatResult(cell.output));
+  }
+  console.log('ok:', report.ok);
+}
+```
+
+`runReport()` is continue-on-error and returns a structured `RunResult` (it never throws on a cell failure, and refuses a workbook with a dependency cycle). The older `runAll()` remains available as an event-stream API that throws on the first cell error.
+
+## Workbook format (.mtsw)
+
+```yaml
+version: '1.0'
+metadata:
+  title: 'Example'
+  author: 'Your Name'
+
+runtime:
+  engine: mathts
+  execution: reactive # reactive | sequential | manual
+
+cells:
+  - markdown: |
+      # Introduction
+    id: intro
+
+  - code: '{ pi: 3.14159, e: 2.71828 }'
+    id: constants
+
+  - test: 'constants.pi > 3.14'
+    id: checkPi
+    depends_on: [constants]
+```
+
+Each cell is a YAML mapping with **exactly one** type key (`code`, `markdown`, `data`, `test`, …) whose value is the cell content, plus:
+
+- **`id`** (required) — must be a valid identifier (`[A-Za-z_][A-Za-z0-9_]*`); ids are how cells are referenced.
+- **`depends_on`** (optional) — a list of cell ids this cell depends on.
+
+### Dependencies & scope
+
+A dependency's result is injected into a cell's evaluation scope as a variable named by the dependency's id. To expose several values, return an object literal and read it with property access:
+
+```yaml
+cells:
+  - code: '{ n: 2, m: 3 }'
+    id: pair
+  - code: 'pair.n + pair.m' # -> 5
+    id: total
+    depends_on: [pair]
+```
+
+Scope is **direct-only (non-transitive)**: a cell sees only the cells in its own `depends_on`, not their dependencies. To use a transitive value, list it explicitly.
+
+### Test cells
+
+A `test` cell's expression must evaluate to a **boolean**: `true` passes, `false` fails, and a non-boolean result is reported as an error (use an explicit comparison). A failing test makes `mtsw run` exit non-zero — workbooks can verify themselves.
+
+## Cell types (v1)
+
+| Type                                               | Status | Description                                            |
+| -------------------------------------------------- | ------ | ------------------------------------------------------ |
+| `markdown`                                         | ✅     | Documentation (passed through verbatim)                |
+| `code`                                             | ✅     | MathTS expression script; last value is the result     |
+| `data`                                             | ✅     | Structured YAML, parsed (hardened) into a value        |
+| `test`                                             | ✅     | Boolean assertion (`true` = pass)                      |
+| `tensor` / `equation` / `visualization` / `export` | ⏳     | Reserved; not executed in v1 (reported as unsupported) |
+
+## Execution modes
+
+- **reactive** — emits stale events for dependents when a cell re-runs
+- **sequential** — top-to-bottom (dependency) order
+- **manual** — explicit trigger only
+
+## Security
+
+Code and test cells execute only through the MathTS sandboxed expression engine — no `eval`, `Function`, or `vm`. YAML (both the document and `data`-cell payloads) is parsed with a hardened core-schema configuration and a prototype-pollution guard.
+
+## License
+
+MIT