yamlover 0.3.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 dims
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,214 @@
+# server
+
+Browse a yamlover tree in the web browser.
+
+```console
+$ npx yamlover [ROOT] [--port N] [--host ADDR] [--no-gitignore]
+```
+
+`ROOT` is any yamlover entity — a directory with a `.yamlover/schema.yaml`, a
+plain directory, or a single file (the same shapes [`walker`](../walker/)
+understands). It defaults to the current directory. The command starts a local
+web server (on `0.0.0.0` by default, so it is reachable from the network — use
+`--host 127.0.0.1` to keep it private) and serves a React single-page app that
+lets you browse the entity starting from `ROOT`.
+
+The page is split into two independently scrolling panes:
+
+- **Left — a table of contents.** A collapsible tree of **every** node — object
+  keys and array elements alike, scalars included (so a leaf like
+  `05-scalar-as-file` is listed and clickable). A node is *expandable* when it is
+  a container with children and has no *active* renderer (see *Renderers*); the
+  first three levels load expanded, and deeper branches load lazily via the
+  chevron. Each entry is labeled by its schema `title` (or an instance `title`
+  child) when present, otherwise by its key (objects) or `[index]` (arrays).
+
+  Each row carries a **type/format icon** (chosen by the schema `format`, falling
+  back to `type` — `{}`/`[]` for object/array, `"`/`#`/`◧` for
+  string/number/boolean, 📅/✉️/🔗/🖼️… for formats). The tree is agnostic to a
+  node's *concrete* (how it is stored) — that shows in the node view's tags.
+- **Right — the selected node**, in one of four representations (tabs):
+  - **yaml-schema** *(default)* — our schema (the yamlover schema concrete) as YAML.
+  - **yaml** — the node's value as YAML.
+  - **json** — the node's value as JSON.
+  - **json-schema** — the standard instance JSON Schema (every leaf `const`).
+
+  All four behave **identically**: syntax-highlighted, **one level deep**, with
+  every nested node shown as a **hyperlink** you click to descend (never inlined)
+  — `{ object with N properties }`, `[ array with M items ]`, or
+  `< binary of N bytes >`. A registered renderer, if any, replaces the value
+  views. The selected representation is part of the URL (`?format=`), so it is
+  preserved as you navigate and is shareable.
+
+  A binary leaf's bytes are read only when you select it and view its value: the
+  yaml view shows a YAML `!!binary` block, json shows the `{format,size,base64}`
+  metadata.
+
+Surfaced *stray* files (those the schema does not describe) honor `.gitignore` by
+default — `node_modules/`, build output, etc. are hidden. Pass `--no-gitignore`
+to show everything. Schema-described children are always shown.
+
+The browser URL is the node's path in **JSON space** — `/examples[0]/markup` —
+never schema space, so there are no `properties` segments, plus `?format=` for
+the representation. Each key is percent-encoded, so a key that itself contains a
+`/` (e.g. `@vitejs/plugin-react`) stays a single segment. It updates as you
+navigate and is shareable / back-button friendly — opening a deep link expands
+the TOC along the path and selects (and scrolls to) the target node.
+
+## How it works
+
+This is a TypeScript reimplementation of the read side of [`walker`](../walker/)
+(concretes, `$ref`/`$defs`, inline `const`, collapsed files, expanded
+subdirectories), plus a small server and SPA. There is **no build step**: the
+launcher runs [Vite](https://vitejs.dev) in middleware mode, serving the client
+from source with HMR and loading the server-side materializer through Vite's
+`ssrLoadModule`.
+
+Leaf bytes are **read lazily**: materialization builds the tree's structure but
+does not read a leaf's content until that node is actually serialized. So listing
+or eliding a node never reads its file — `value: 30` is read only when its node is
+shown, and a binary blob only when its base64 is requested. The materialized tree
+is cached per server for a short window, so a burst of clicks does not re-read the
+filesystem; edits show up after the window on reload.
+
+### API
+
+All endpoints take a JSON-space `path` (default `/`); `json` and `schema` take an
+optional `depth` (container-nesting limit).
+
+| endpoint | returns |
+|----------|---------|
+| `GET /api/info` | `{ root }` — the root label (the served entity's yamlover title, else its directory name): the breadcrumb head and TOC root |
+| `GET /api/tree?path&depth` | the table of contents rooted at `path`, `depth` levels deep (default 3) — fetched again per branch for lazy expansion |
+| `GET /api/json?path&depth` | the node's value, one level deep (`depth` default 1); nested containers become link markers (add `&binary=1` for a binary leaf's base64) |
+| `GET /api/schema?path&depth` | the node's instance JSON Schema, one level deep, with the same link markers |
+| `GET /api/blob?path` | a file-backed node's **raw bytes**, with its (inferred) format as the `Content-Type` — what the image / html / pdf / djvu / markup renderers read, instead of base64 over `json` |
+
+A **link marker** — `{ "$yamloverLink": { kind, path, count|size } }` — stands in
+for a node shown only as a link (a nested container past the one-level view, or
+any binary leaf); the client renders it as a `{ object with N properties }`,
+`[ array with M items ]`, or `< binary of N bytes >` hyperlink. The same marker
+appears in both the value and the schema, so every representation (YAML/JSON ×
+data/schema) renders identically — just a syntax choice over one structure. A
+selected binary leaf's bytes arrive as `{ "$yamloverBinary": {format,size,base64} }`.
+
+### Renderers
+
+A renderer is keyed by a **(type, format)** tuple — the JSON-Schema `type` plus
+its `format` — the same key the TOC icons and the link markers carry. A node with
+no matching renderer falls through to the default tabbed view (YAML/JSON ×
+data/schema) and expands normally in the TOC.
+
+The registry lives in `src/client/renderers/`. A renderer participates three ways,
+all on the one tuple:
+
+- **`render`** — the full RHS page (its tab is the node's default representation).
+- **`renderChunk`** — its *inline* form, when embedded in another renderer's page.
+  A renderer composes children by routing each to the renderer for the child's own
+  tuple: the `chapter` renderer draws each of its `chunks` by delegating to that
+  chunk's renderer (a `string`/`text/markdown` chunk → the `text` renderer; an
+  `image/png` chunk would route to an image renderer, no change to `chapter`).
+- **`tocView`** — how the node appears in the TOC: which children are navigable,
+  whether it expands, and whether they are loaded (default: its own children,
+  lazily loaded). A renderer can unwrap or filter — `chapter` surfaces its
+  subchapters directly (from its `children` array) and keeps its `chunks` off the
+  tree, so the TOC reads as a table of contents.
+
+A renderer may also declare **`depth`** — the value depth `NodeView` fetches for
+it (default 1; `chapter` needs 2 to reach its chunk/subchapter elements).
+
+Registered today:
+
+| renderer | (type, format) | draws with |
+|----------|----------------|------------|
+| `chapter` | `object` / `x-yamlover-chapter` | numbered chunks + subchapter links |
+| `tag` | `object` / `x-yamlover-tag` | a hierarchy diagram: supertag (containment parent) and subtags (children) wired to the current tag |
+| `text` | `string` / `text/markdown` | [marked](https://marked.js.org) |
+| `asciidoc` | `string` / `text/asciidoc` | [@asciidoctor/core](https://asciidoctor.org) |
+| `image` | `binary` / `image/*` | native `<img>` |
+| `html` | `binary` / `text/html` | sandboxed `<iframe>` |
+| `pdf` | `binary` / `application/pdf` | [pdf.js](https://mozilla.github.io/pdf.js/) via [react-pdf](https://github.com/wojtekmaj/react-pdf) |
+| `djvu` | `binary` / `image/vnd.djvu` | [DjVu.js](https://djvu.js.org) (vendored) |
+
+Adding a shape is still a single registry entry.
+
+### File rendering and format inference
+
+The bottom seven renderers above turn the browser into a viewer for the common
+file types a tree carries. They hang off one rule, applied to any file-backed
+node that carries **no explicit schema `format`**: the server **infers a format
+from the file extension** (`formatFromExt` in `src/server/yamlover.ts`). So a
+stray `.pdf`, `.png`, or `.md` — or one declared with only `concrete: file`/
+`file/binary` — renders without a `format:` line. An explicit schema `format`
+always wins.
+
+Inference splits two ways by how the renderer consumes the file:
+
+- **Served as bytes** — images, `application/pdf`, `text/html`, `image/vnd.djvu`
+  become `file/binary`. Their renderer points an `<img>`/`<iframe>`/pdf-loader at
+  **`/api/blob`** (or, for DjVu, fetches the `ArrayBuffer`), so the bytes stream
+  straight from disk with no base64 round-trip.
+- **Read as text** — `.md`/`.adoc` keep a **string** value (the file's text),
+  which the `text`/`asciidoc` renderer parses to HTML. (This preserves the
+  existing "a `.md` file is a string" behavior.)
+
+Two implementation notes:
+
+- **DjVu has no native browser support**, so it is decoded client-side by
+  **DjVu.js**, vendored as a prebuilt bundle at `src/client/vendor/djvu.js`.
+  The library is **GPL-v2** (see `src/client/vendor/README.md` for provenance and
+  how to regenerate); the rest of this package is not.
+- **pdf.js and DjVu.js reach for browser globals at import time**, which would
+  break the (Node/jsdom) test run. So the `pdf` and `djvu` renderers are
+  **lazy-loaded** (`React.lazy`) — importing the registry, as the TOC and the
+  tests do, never pulls them in until a PDF/DjVu node is actually shown.
+
+`examples/18-pdf-tags` exercises this: a library of real papers stored as PDFs
+and one saved HTML page, each rendered in place.
+
+## Requirements
+
+- Node.js 18+
+
+Dependencies (React, Vite, js-yaml, ignore, plus the renderer libraries
+`react-pdf`, `marked`, `@asciidoctor/core`) install with `npm install` in this
+directory; `npx yamlover` then runs `bin/yamlover.js`. DjVu.js is **vendored**
+(`src/client/vendor/djvu.js`), not an npm dependency.
+
+## Tests
+
+Both sides are covered by [Vitest](https://vitest.dev) (it runs the TypeScript
+directly, no build):
+
+```console
+$ npm test          # run once
+$ npm run test:watch
+```
+
+- **Server** (`test/*.test.ts`, Node) — against the `examples/` fixtures: path
+  encoding round-trips (incl. keys with `/`), materialization and `$ref`/`$defs`
+  equivalence, one-level link markers (object/array/binary) in value and schema,
+  lazy binaries and the `!!binary` payload, the non-YAML → raw-text fallback,
+  `buildTree` (all nodes, depth, titles), `.gitignore` filtering, and the API
+  endpoints (`/api/info`, `/api/tree`, `/api/json`, `/api/schema`).
+- **Client** (`test/client/*`, jsdom + React Testing Library) — path/URL helpers
+  and breadcrumbs, type/format icons, the renderer registry, the unified
+  `Render` (scalars, `{ object with N properties }` / `[ array with M items ]` /
+  `< binary of N bytes >` links, `!!binary`, YAML vs JSON), the `Tree`
+  (selection, lazy expand, leaves), `NodeView` (markers, tab switches, binary,
+  schema), and `App` (breadcrumb head + TOC).
+
+## Layout
+
+```
+bin/yamlover.js     CLI entry — arg parsing + Vite middleware-mode server
+src/server/         TypeScript port of the walker read side + JSON API
+  yamlover.ts         materialize a logical tree (lazy leaves); toPlain/toSchema/buildTree; formatFromExt
+  gitignore.ts        .gitignore predicate for surfaced stray files
+  api.ts              /api/info, /api/tree, /api/json, /api/schema, /api/blob
+src/client/         the React SPA (tree, node view, render, icons, paths)
+  renderers/          (type, format) registry + chapter/text/asciidoc/media/pdf/djvu
+  vendor/djvu.js      prebuilt DjVu.js bundle (GPL-v2; see vendor/README.md)
+test/               Vitest suite for the server logic
+index.html          SPA shell
+```

package/bin/yamlover.js

@@ -0,0 +1,202 @@
+#!/usr/bin/env node
+/**
+ * yamlover — serve a directory of yamlover data as a browsable React SPA.
+ *
+ *   npx yamlover [ROOT] [--port N]
+ *
+ * ROOT is the directory to browse (default: the current directory). It may be a
+ * yamlover node (with `.yamlover/`), a plain directory, or a single file — the
+ * same shapes `tools/walker` understands.
+ *
+ * The server runs Vite in middleware mode: the client (`src/client`) is served
+ * with HMR straight from source, and the server-side materializer
+ * (`src/server`) is loaded through Vite's `ssrLoadModule`, so there is no build
+ * step. API routes are handled before Vite; every other route falls back to the
+ * SPA's index.html (client-side routing on the JSON path).
+ */
+
+import { createServer } from "vite";
+import react from "@vitejs/plugin-react";
+import { createServer as createHttpServer } from "node:http";
+import { fileURLToPath, pathToFileURL } from "node:url";
+import { createRequire } from "node:module";
+import { dirname, join, resolve } from "node:path";
+import fs from "node:fs";
+
+const __dirname = dirname(fileURLToPath(import.meta.url));
+const pkgRoot = resolve(__dirname, ".."); // tools/server
+
+// Resolve react/react-dom as *this package* sees them, and alias Vite to those
+// exact copies, so the SPA never picks up a stale React from a parent
+// node_modules (which surfaces as "react-dom … does not provide an export named
+// 'createRoot'" — createRoot is React 18+). Best-effort: if resolution fails we
+// leave Vite to its defaults.
+const reactAlias = {};
+// Directories Vite is allowed to serve over `/@fs` (its file-system allowlist).
+// Start with this package; below we add the node_modules that actually holds the
+// heavy deps. Under `npx`, deps hoist to a `_npx/<hash>/node_modules` *outside*
+// `pkgRoot`, and Vite's default workspace-root detection misses it — so a raw asset
+// fetch (notably pdf.js's `pdf.worker.min.mjs`) gets denied and falls through to
+// index.html, surfacing as the "non-JavaScript MIME type text/html" worker error.
+const fsAllow = [pkgRoot];
+try {
+  const req = createRequire(join(pkgRoot, "package.json"));
+  reactAlias["react"] = dirname(req.resolve("react/package.json"));
+  reactAlias["react-dom"] = dirname(req.resolve("react-dom/package.json"));
+  // Allow whichever node_modules each heavy dep resolves from (hoisted under npx,
+  // or nested in dev) so Vite serves their worker/asset files over `/@fs`.
+  for (const dep of ["pdfjs-dist", "react-pdf", "leaflet"]) {
+    try {
+      const nodeModules = dirname(dirname(req.resolve(`${dep}/package.json`)));
+      if (!fsAllow.includes(nodeModules)) fsAllow.push(nodeModules);
+    } catch {
+      /* dep not resolvable from here — skip */
+    }
+  }
+} catch {
+  /* fall back to default resolution */
+}
+
+// --- argument parsing ----------------------------------------------------- //
+let rootArg = null; // the ROOT path as typed (null when omitted)
+let port = 5173;
+let host = "0.0.0.0"; // all interfaces by default — reachable from the network
+let gitignore = true; // hide .gitignore'd stray files by default
+const argv = process.argv.slice(2);
+for (let i = 0; i < argv.length; i++) {
+  const a = argv[i];
+  if (a === "--port" || a === "-p") port = parseInt(argv[++i], 10);
+  else if (a.startsWith("--port=")) port = parseInt(a.slice("--port=".length), 10);
+  else if (a === "--host") host = argv[++i];
+  else if (a.startsWith("--host=")) host = a.slice("--host=".length);
+  else if (a === "--no-gitignore") gitignore = false;
+  else if (a === "--help" || a === "-h") {
+    console.log("usage: npx yamlover [ROOT] [--port N] [--host ADDR] [--no-gitignore]");
+    process.exit(0);
+  } else if (!a.startsWith("-")) rootArg = a;
+}
+const dataRoot = resolve(process.cwd(), rootArg ?? ".");
+if (!fs.existsSync(dataRoot)) {
+  console.error(`yamlover: no such path: ${dataRoot}`);
+  process.exit(1);
+}
+
+// --- server --------------------------------------------------------------- //
+// `vite` and `handle` are filled in just below; the request handler closes over
+// them and only reads them once requests start arriving, after assignment.
+let vite;
+let handle;
+const indexHtmlPath = join(pkgRoot, "index.html");
+
+const server = createHttpServer((req, res) => {
+  const url = new URL(req.url, "http://localhost");
+  if (url.pathname.startsWith("/api/")) {
+    handle(req, res, url);
+    return;
+  }
+  vite.middlewares(req, res, async () => {
+    // SPA fallback: serve the (transformed) index.html for any client route.
+    try {
+      let html = fs.readFileSync(indexHtmlPath, "utf-8");
+      html = await vite.transformIndexHtml(url.pathname, html);
+      res.setHeader("Content-Type", "text/html; charset=utf-8");
+      res.end(html);
+    } catch (e) {
+      vite.ssrFixStacktrace(e);
+      res.statusCode = 500;
+      res.end(e.message);
+    }
+  });
+});
+
+// Run HMR over our own server (`hmr: { server }`) so Vite does not open a
+// second WebSocket port of its own.
+vite = await createServer({
+  root: pkgRoot,
+  plugins: [react()],
+  appType: "custom",
+  // Always use this package's own React: `dedupe` collapses react/react-dom to a
+  // single copy, and the aliases pin that copy to the one resolvable from here —
+  // so a stale react-dom in a *parent* node_modules (the classic "does not
+  // provide an export named 'createRoot'" failure) cannot shadow it.
+  resolve: { dedupe: ["react", "react-dom"], alias: reactAlias },
+  // Pre-bundle the heavy renderer deps up front. These are all lazy-loaded, so
+  // Vite would otherwise discover them (and their transitive CJS deps such as
+  // `warning` for react-pdf or `base64-js` for ag-psd) only on first use and could
+  // serve them un-interopped — the "does not provide an export named …" failure.
+  // Listing them forces a clean CJS→ESM bundle before any renderer mounts.
+  optimizeDeps: {
+    include: [
+      "react-pdf",
+      "marked",
+      "@asciidoctor/core",
+      "ag-psd",
+      "utif",
+      "heic2any",
+      // the office/map renderer deps (all CJS/UMD — they need the CJS→ESM interop
+      // a pre-bundle gives, or they are served raw and fail with "does not provide
+      // an export named 'default'", e.g. `import L from "leaflet"`)
+      "leaflet",
+      "xlsx",
+      "mammoth/mammoth.browser",
+      "@tmcw/togeojson",
+    ],
+  },
+  // `allowedHosts: true` lifts Vite's Host-header allowlist so the SPA is
+  // reachable from the network (any hostname/IP), matching the 0.0.0.0 bind.
+  // `fs.allow` is widened (above) to the dirs holding the heavy deps so their
+  // `/@fs` assets — chiefly pdf.js's worker — are served, not 404'd to index.html.
+  server: { middlewareMode: true, allowedHosts: true, hmr: { server }, fs: { allow: fsAllow } },
+});
+
+// Load the server-side materializer. The engine-backed handler (engine-api.ts) supersedes the
+// legacy loadEntity materializer (api.ts kept for ref). It reaches OUTSIDE this package
+// (../../../engine, ../../../parser), which the npm tarball cannot carry — so at prepack we bundle
+// that import graph into a self-contained `dist/server.js` (scripts/build.mjs). We pick the loader
+// by whether the engine SOURCE is reachable: in the repo CHECKOUT it is, so we use Vite's
+// `ssrLoadModule` to transpile the TS live (edits take effect with no rebuild — never a stale
+// bundle); in the PUBLISHED package the source is absent, so we import the prebuilt bundle. Either
+// way the client SPA is still served by Vite from src.
+// The initial index runs as a BACKGROUND task — the server listens immediately (serving the
+// previous on-disk index, or an empty tree on a cold start) while progress lands here and in
+// the web UI (SSE task frames + GET /api/tasks).
+const engineSrc = resolve(pkgRoot, "../engine/ts/src/index.ts"); // present only in the monorepo
+const { createHandlers } = fs.existsSync(engineSrc)
+  ? await vite.ssrLoadModule("/src/server/engine-api.ts") // repo: live TS, no rebuild needed
+  : await import(pathToFileURL(join(pkgRoot, "dist/server.js")).href); // published: prebuilt bundle
+handle = createHandlers(dataRoot, {
+  gitignore,
+  watch: true, // re-index + push on external edits
+  log: (line) => console.log(`yamlover  ${line}`),
+});
+handle.ready.catch((e) => console.error("yamlover: indexing failed:", e));
+
+// Listen on `port`; if it is already in use, fall back to the next port (up to
+// `MAX_PORT_TRIES`), so two instances — or a leftover one — don't collide.
+const shown = host === "0.0.0.0" || host === "::" ? "localhost" : host;
+const MAX_PORT_TRIES = 50;
+
+function listenWithFallback(p, triesLeft) {
+  const onError = (err) => {
+    server.off("listening", onListening); // drop this attempt's success handler
+    if (err && err.code === "EADDRINUSE" && triesLeft > 0) {
+      console.log(`yamlover  port ${p} in use — trying ${p + 1}…`);
+      listenWithFallback(p + 1, triesLeft - 1);
+    } else if (err && err.code === "EADDRINUSE") {
+      console.error(`yamlover: no free port found in ${port}–${p}`);
+      process.exit(1);
+    } else {
+      throw err;
+    }
+  };
+  const onListening = () => {
+    server.off("error", onError); // bound OK — stop intercepting listen errors
+    console.log(`yamlover  serving ${dataRoot}`);
+    console.log(`          http://${shown}:${p}/  (bound to ${host})`);
+  };
+  server.once("error", onError);
+  server.once("listening", onListening);
+  server.listen(p, host);
+}
+
+listenWithFallback(port, MAX_PORT_TRIES);