@hviana/sema 0.5.8 → 0.6.0

package/AGENTS.md

@@ -768,10 +768,12 @@ memoises perceive+intern of repeated inputs and routes through the same
 `store.commit()` at checkpoints; run `compactContentIndex` /
 `repairContentIndex` post-training if eviction was heavy, and
 `mind.buildCanonIndex()` if queries will carry a canonicalizer (2.9). See
-`example/train_base.ts`. Profiling note: the first `resonate` after a big ingest
-pays the pending index flush; the dominant query-side ANN cost is connector
-pre-resolution (bounded by recognised-site count — don't add another loop over
-site pairs).
+`example/train_base/` — a folder, entry `main.ts`: the run context lives in
+`runtime.ts`, the one per-corpus loop in `stage.ts`, and each corpus (knobs, row
+adapter, stage descriptor) in `corpora/<name>.ts`. Profiling note: the first
+`resonate` after a big ingest pays the pending index flush; the dominant
+query-side ANN cost is connector pre-resolution (bounded by recognised-site
+count — don't add another loop over site pairs).
 
 ---
 
@@ -810,3 +812,40 @@ PolyForm Noncommercial 1.0.0 with separate commercial licensing (see
 vendor code under licenses incompatible with dual distribution, and do not add
 runtime dependencies casually — the near-zero-dependency footprint is a product
 feature.
+
+**The library has NO runtime dependencies at all**, and that is now pinned by
+`test/88-dependency-footprint.test.mjs`: the built `dist/src` may import only
+`node:` builtins and relative paths, `package.json` may declare no
+`dependencies`, and the published entry points may not reach outside `dist/src`.
+The rule an EXAMPLE follows is different and looser — it may use what it needs,
+as a **dev** dependency, loaded **lazily** so it is a requirement only of the
+code path that uses it. `example/train_base` is the reference: `hyparquet` (+
+its Snappy codec) is the sole third-party package in this repository, it is
+dev-only, and `readers.ts` resolves it by dynamic import the first time a
+Parquet corpus is actually read — so a curriculum with no Parquet stage runs
+with the package absent. It used to sit in `dependencies`, which installed a
+Parquet reader on every consumer of Sema for the sake of one example; that is
+the mistake the suite exists to catch.
+
+**Training corpora are governed by the same rule, and more strictly.** Sema is
+non-parametric: a trained store retains its training text VERBATIM (read any
+content node back and the original sentence comes out). A store is therefore a
+redistribution of its corpora, not a derived model, and every upstream licence
+applies to it in full. Two consequences:
+
+- A corpus carrying a **NonCommercial** term cannot enter a trainer — it
+  conflicts with the commercial licence tier.
+- A corpus carrying a **ShareAlike** term cannot enter a trainer — its copyleft
+  would attach to the distributed store.
+
+Check both against **what the corpus was built from**, not only the repository's
+license tag: a dataset assembled out of Wikipedia prose and published under
+Apache-2.0 still carries CC BY-SA on that prose. Where a corpus has a clean
+layer and a contaminated one, ingest only the clean layer.
+
+Sema's own license does **not** extend over corpus content inside a store, and
+cannot: CC BY 4.0 §2(a)(5)(B) forbids applying terms that restrict what the
+license permits. The engine is what PolyForm protects. Per-corpus attribution,
+the required modification statement, and the current allow/deny list live in
+[DATASETS.md](DATASETS.md) — update it in the same change that touches a
+trainer's corpus set.

package/DATASETS.md

@@ -0,0 +1,160 @@
+# Training corpora — provenance, licensing, and attribution
+
+This file is the attribution notice for every corpus Sema is trained on, and the
+licensing statement for the **trained memory files** that training produces.
+
+It is a required companion to any distributed Sema store. If you publish or ship
+`*.sqlite` / `*.content.vec` / `*.halo.vec`, ship this file with them.
+
+---
+
+## 1. Why a trained store is not a weight file
+
+Sema is non-parametric. Training is deposition, not gradient descent: source
+text is segmented and content-addressed, and the **bytes are retained**. Reading
+a node returns the original text:
+
+```
+#15709469 → "Kohei Uchimura from Japan holds the record for the most World
+             Championship medals won by a male gymnast, with a total of 21 medals."
+```
+
+A trained store is therefore a database that contains its training corpora in
+recoverable form. Distributing one **is** distributing those corpora, and every
+upstream licence applies in full. The "it's only model weights, the text isn't
+really in there" argument is not available to Sema, by design.
+
+Two consequences follow, and both are load-bearing:
+
+1. **A corpus whose licence forbids commercial use cannot enter the store**,
+   because Sema is offered under a paid commercial licence as well as
+   [PolyForm Noncommercial](LICENSE.md).
+2. **A corpus under a ShareAlike licence cannot enter the store**, because its
+   copyleft would attach to the distributed artifact.
+
+Both rules are stated in [AGENTS.md](AGENTS.md) §6 and must be checked before
+any corpus is added to a trainer.
+
+---
+
+## 2. How a distributed store is licensed
+
+A trained Sema store has two layers, licensed separately. Conflating them is a
+licence violation in one direction or the other.
+
+| Layer                           | What it covers                                                                                          | Licence                                                                                                 |
+| :------------------------------ | :------------------------------------------------------------------------------------------------------ | :------------------------------------------------------------------------------------------------------ |
+| **Sema itself**                 | The algorithms, geometry, deduction engine, indexes, file formats, and all code that produced the store | [PolyForm Noncommercial 1.0.0](LICENSE.md), with a separate [commercial licence](COMMERCIAL-LICENSE.md) |
+| **Corpus content in the store** | The retained training text and anything derived from it                                                 | Each corpus's own upstream licence, listed in §3                                                        |
+
+**Sema's source licence is not extended over the corpus content, and cannot
+be.** CC BY 4.0 §2(a)(5)(B) forbids applying legal terms that restrict a
+recipient from doing what the licence permits — so the noncommercial term cannot
+be applied to CC BY text sitting inside the store. What the noncommercial term
+protects is the engine, which is the part that is actually ours.
+
+**Modification statement** (required by CC BY 4.0 §3(a)(1)(B)): all corpus text
+in a Sema store has been modified. It is segmented at content-defined
+boundaries, re-encoded, deduplicated by content address, and interleaved with
+text from other sources. It is not presented as a faithful reproduction of any
+upstream dataset, and no endorsement by any upstream author is implied.
+
+**Apache-2.0 obligations**: corpora marked Apache-2.0 below require the licence
+text and any upstream `NOTICE` to travel with the distribution, and require
+changes to be stated. The modification statement above satisfies the latter.
+
+---
+
+## 3. Corpora
+
+### 3.1 In use
+
+| Corpus                                                                                                   | Licence                                               | Attribution                        | Notes                                                                                                                                                                                                                                                                 |
+| :------------------------------------------------------------------------------------------------------- | :---------------------------------------------------- | :--------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [google/smol](https://huggingface.co/datasets/google/smol) (SmolSent)                                    | CC BY 4.0                                             | Google LLC                         | Translation pairs                                                                                                                                                                                                                                                     |
+| [CohereLabs/aya_dataset](https://huggingface.co/datasets/CohereLabs/aya_dataset)                         | Apache-2.0                                            | Cohere For AI                      | Human-written prompt/completion                                                                                                                                                                                                                                       |
+| [OpenAssistant/oasst2](https://huggingface.co/datasets/OpenAssistant/oasst2)                             | Apache-2.0                                            | LAION / OpenAssistant contributors | Human-authored dialogue — see §5                                                                                                                                                                                                                                      |
+| [Taskmaster-1/2/3/4](https://github.com/google-research-datasets/Taskmaster)                             | CC BY 4.0                                             | Google LLC                         | Task-oriented dialogue. Only `utterances[].text` is ingested; the `instructions` / `scenario` / `vertical` fields are never read                                                                                                                                      |
+| [2WikiMultihopQA](https://huggingface.co/datasets/xanhho/2WikiMultihopQA) — **`evidences` triples only** | Apache-2.0 (repo); triples originate in Wikidata, CC0 | Ho et al.; Wikidata contributors   | Only the `evidences` column is ingested. The `context` column (Wikipedia prose, CC BY-SA) is **never read** — see §4. The `question`/`answer` columns are also never deposited, for a capability reason rather than a licence one: they memorise instead of composing |
+| [allenai/soda](https://huggingface.co/datasets/allenai/soda)                                             | CC BY 4.0                                             | Allen Institute for AI             | Social dialogue. Only the `dialogue` column is ingested; `narrative` / `literal` / `head` / `relation` / `tail` are never read. Model-generated provenance — see §5                                                                                                   |
+| [AmazonScience/massive](https://huggingface.co/datasets/AmazonScience/massive)                           | CC BY 4.0                                             | Amazon Science                     | Short multilingual intents. Only the `utt` column is ingested; the slot-annotated `annot_utt` is never read. **Disabled by default** on capability grounds (not licence) — see `MASSIVE` in `example/train_base/corpora/massive.ts`                                   |
+
+### 3.2 Excluded, and why
+
+| Corpus                                                            | Reason                                                                                                                                                                                                                                                                                                                                                            |
+| :---------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **MuskumPillerum/General-Knowledge**                              | **No licence at all.** The HF repo carries no licence tag and no licence in its card; an earlier header in `example/train_base.ts` (now `corpora/genknow.ts`) claimed MIT without support. Its own card states it "contains a subset of the alpaca dataset", and Alpaca is CC BY-NC 4.0 — **NonCommercial**, incompatible with Sema's commercial licence. See §6. |
+| **PAWS**                                                          | Google's own grant is maximally permissive ("may be freely used for any purpose"), but PAWS-Wiki sentences derive from Wikipedia (CC BY-**SA**) and PAWS-QQP from Quora question pairs under Quora's terms. Because Sema retains text verbatim, the upstream terms would attach to the distributed store. Excluded despite strong measured fit.                   |
+| **Schema-Guided Dialogue (SGD/dstc8)**, **HotpotQA**, **MuSiQue** | CC BY-SA 4.0 — ShareAlike conflicts with dual distribution.                                                                                                                                                                                                                                                                                                       |
+| **2WikiMultihopQA passages**                                      | Wikipedia prose, CC BY-SA. The repo's Apache-2.0 tag does not relicense the text it was built from. Only the Wikidata-derived `evidences` triples are ingested.                                                                                                                                                                                                   |
+| **Alpaca** and derivatives                                        | CC BY-NC 4.0, and generated from OpenAI model outputs.                                                                                                                                                                                                                                                                                                            |
+
+---
+
+### 3.3 What each stage actually deposits
+
+A corpus's licence applies to what is ingested, and every stage ingests a strict
+subset of its source. This is the authoritative list.
+
+| Stage                    | Columns/fields read                       | Deposit shape                                             |
+| :----------------------- | :---------------------------------------- | :-------------------------------------------------------- |
+| SmolSent                 | `src`, `trg`                              | one `src → trg` (foreign → English) episode per row       |
+| Aya                      | `inputs`, `targets`                       | one question → answer episode                             |
+| oasst2                   | message `text` along the best-ranked path | cumulative-context walk                                   |
+| Taskmaster               | `utterances[].text`                       | cumulative-context walk over speaker-merged turns         |
+| 2Wiki                    | `evidences`                               | per triple: a relation fact and a bare-subject pivot fact |
+| SODA                     | `dialogue`, `speakers`                    | cumulative-context walk over speaker-merged turns         |
+| MASSIVE (off by default) | `utt`                                     | one bare experience                                       |
+
+Everything else in those sources — Taskmaster's `instructions`/`scenario`,
+2Wiki's `context`/`question`/`answer`, SODA's `narrative`/`literal`/`head`/
+`relation`/`tail`, MASSIVE's `annot_utt` — is **not read** and therefore not
+distributed in a trained store.
+
+---
+
+## 4. The rule that decides these cases
+
+**A repository's licence tag does not relicense the material the repository was
+built from.** A dataset assembled out of Wikipedia prose and published under
+Apache-2.0 still carries Wikipedia's ShareAlike terms on that prose. Because
+Sema stores text verbatim, Sema inherits the _upstream_ terms, not the
+repackager's.
+
+So the check for any candidate corpus is two questions, not one:
+
+1. What licence does the repository carry?
+2. **What was it built from, and what licence does that carry?**
+
+Where a corpus has a clean layer and a contaminated one, take the clean layer
+only — as with 2Wiki's Wikidata triples (CC0) versus its Wikipedia passages (CC
+BY-SA).
+
+---
+
+## 5. Disclosures
+
+**Model-generated provenance.** `allenai/soda` is licensed CC BY 4.0 but was
+distilled from OpenAI GPT-3.5 outputs. The licence is clean; the provenance is
+disclosed here so downstream users can make their own assessment.
+
+**Personal data.** `OpenAssistant/oasst2` is human-authored content contributed
+by identifiable volunteers, and Sema retains it verbatim in a redistributable
+artifact. Erasure requests against a content-addressed store are not
+straightforward. Anyone distributing a Sema store trained on human-contributed
+dialogue should account for this.
+
+---
+
+## 6. Status of previously published stores
+
+Stores published before this file was written — including those under
+[hviana/sema-trained-v1](https://huggingface.co/buckets/hviana/sema-trained-v1)
+— were trained with the `MuskumPillerum/General-Knowledge` stage enabled (37,623
+rows), whose licence status is described in §3.2. Those artifacts should be
+treated as **not redistributable** until retrained without that stage.
+
+The stage is now **disabled by default** in
+`example/train_base/corpora/genknow.ts` (`GENKNOW=0`). The adapter code remains
+so the stage can be re-enabled for local experiments; a store trained with
+`GENKNOW=1` must not be distributed.

package/README.md

@@ -320,6 +320,7 @@ start talking — no install, no runtime, no API key.
 | 🛠️ **[AGENTS.md](AGENTS.md)**                                                        | The development manual: repo layout, build/test, internals, invariants, and recipes for extending the system.                                            |
 | 🎓 **[CITATION.cff](CITATION.cff)**                                                  | How to cite Sema in academic work.                                                                                                                       |
 | ⚖️ **[LICENSE.md](LICENSE.md)**                                                      | PolyForm Noncommercial License 1.0.0.                                                                                                                    |
+| 📚 **[DATASETS.md](DATASETS.md)**                                                    | Training corpora: provenance, per-corpus attribution, and how a trained memory file is licensed.                                                         |
 | 💼 **[COMMERCIAL-LICENSE.md](COMMERCIAL-LICENSE.md)**                                | Commercial licensing terms and contact.                                                                                                                  |
 | 🤗 **[Trained examples](https://huggingface.co/buckets/hviana/sema-trained-v1)**     | Pre-trained memory files you can download and use directly.                                                                                              |
 | 💿 **[Binary examples](https://huggingface.co/buckets/hviana/sema-binary-examples)** | Ready-to-run web chat apps for Windows, Mac, and Linux — one file, no install.                                                                           |
@@ -350,6 +351,17 @@ start talking — no install, no runtime, no API key.
 > **[TRADEMARKS.md](TRADEMARKS.md)** — the **Sema** name, logos, and brand are
 > _not_ covered by the source license.
 
+> [!NOTE]
+> **Trained memory files are licensed in two layers.** Sema is non-parametric,
+> so a trained store retains its training text verbatim — it is a database that
+> contains its corpora, not a weight file. The **engine** (algorithms, geometry,
+> deduction, indexes, formats, and all code) is under PolyForm Noncommercial
+> with a separate commercial license. The **corpus content inside a store**
+> stays under each corpus's own upstream license, and Sema's license is not
+> extended over it. If you distribute a store, ship
+> **[DATASETS.md](DATASETS.md)** with it — it carries the attribution and
+> modification statement those licenses require.
+
 <div align="center">
 
 **Respecting these terms funds the research that makes work like this

package/dist/example/train_base/cache.d.ts

@@ -0,0 +1,35 @@
+/** Delete every orphaned "<file>.part" in the cache, returning how many were
+ *  removed and the bytes they held.
+ *
+ *  A .part file at rest is by definition the debris of a download that never
+ *  finished — the rename that promotes one is the last step of `downloadFile`,
+ *  so a live .part exists only while THIS process is writing it. Sweeping at
+ *  startup is therefore safe, and it is load-bearing rather than cosmetic:
+ *  `cacheSize` deliberately counts .part files (an in-flight download really
+ *  does occupy the disk), so debris left by a killed run consumes ceiling
+ *  budget that nothing would ever free, and `ensureCacheRoom` would wait for
+ *  room that cannot appear.
+ *
+ *  The one assumption is that a cache directory belongs to ONE run at a time.
+ *  That was already true — two trainers sharing CACHE_DIR would write the same
+ *  .part path — so this adds no constraint that did not exist. */
+export declare function sweepPartials(): {
+    files: number;
+    bytes: number;
+};
+/** Total bytes currently held in the cache directory — INCLUDING any .part
+ *  file, because an in-flight download occupies the disk like any other file.
+ *  Orphaned ones are removed by {@link sweepPartials} at startup. */
+export declare function cacheSize(): number;
+/** Block until there is room for a file of `fileBytes` under the ceiling.
+ *  A single file larger than the whole ceiling can never "fit", so we let it
+ *  through (it is deleted right after processing) rather than wait forever. */
+export declare function ensureCacheRoom(fileBytes: number, signal: AbortSignal, warn?: (msg: string) => void, maxWaitMs?: number): Promise<void>;
+export interface DownloadOptions {
+    signal: AbortSignal;
+    tries: number;
+    onFail?: (attempt: number, err: Error) => void;
+    onProgress?: (done: number, total: number) => void;
+}
+/** Stream `url` to `destPath`, atomically and with backpressure. */
+export declare function downloadFile(url: string, destPath: string, opts: DownloadOptions): Promise<void>;

package/dist/example/train_base/cache.js

@@ -0,0 +1,211 @@
+// train_base/cache.ts — the durable disk cache and the download sink.
+//
+// This is the ONE place the trainer needs Node rather than the web platform:
+// every other byte in the pipeline moves through fetch, WHATWG streams,
+// DecompressionStream, TextDecoderStream and Blob, but writing a file is the
+// single capability the web platform does not expose. So the sink below wraps a
+// raw fs descriptor, and nothing else here does.
+//
+// Two invariants the rest of the trainer relies on:
+//   • ATOMIC — a download streams to "<file>.part", is fsync'd, then renamed
+//     into place. A file at its final path is, by construction, complete, so an
+//     interrupted download can never be mistaken for a cached one.
+//   • BOUNDED — a download blocks under the MAX_CACHE_GB ceiling, and a fully
+//     processed file is deleted by its caller immediately.
+import { CACHE_DIR, CACHE_WAIT_MS, MAX_CACHE_BYTES, PART_SUFFIX, } from "./config.js";
+import { httpError, retry, waitMs } from "./http.js";
+import { closeSync, existsSync, fsyncSync, mkdirSync, openSync, readdirSync, renameSync, statSync, unlinkSync, writeSync, } from "node:fs";
+import { basename, join } from "node:path";
+/** Delete every orphaned "<file>.part" in the cache, returning how many were
+ *  removed and the bytes they held.
+ *
+ *  A .part file at rest is by definition the debris of a download that never
+ *  finished — the rename that promotes one is the last step of `downloadFile`,
+ *  so a live .part exists only while THIS process is writing it. Sweeping at
+ *  startup is therefore safe, and it is load-bearing rather than cosmetic:
+ *  `cacheSize` deliberately counts .part files (an in-flight download really
+ *  does occupy the disk), so debris left by a killed run consumes ceiling
+ *  budget that nothing would ever free, and `ensureCacheRoom` would wait for
+ *  room that cannot appear.
+ *
+ *  The one assumption is that a cache directory belongs to ONE run at a time.
+ *  That was already true — two trainers sharing CACHE_DIR would write the same
+ *  .part path — so this adds no constraint that did not exist. */
+export function sweepPartials() {
+    const out = { files: 0, bytes: 0 };
+    if (!existsSync(CACHE_DIR))
+        return out;
+    for (const name of readdirSync(CACHE_DIR)) {
+        if (!name.endsWith(PART_SUFFIX))
+            continue;
+        const p = join(CACHE_DIR, name);
+        try {
+            const size = statSync(p).size;
+            unlinkSync(p);
+            out.files++;
+            out.bytes += size;
+        }
+        catch { /* raced with another delete — nothing to reclaim */ }
+    }
+    return out;
+}
+/** Total bytes currently held in the cache directory — INCLUDING any .part
+ *  file, because an in-flight download occupies the disk like any other file.
+ *  Orphaned ones are removed by {@link sweepPartials} at startup. */
+export function cacheSize() {
+    if (!existsSync(CACHE_DIR))
+        return 0;
+    let total = 0;
+    for (const name of readdirSync(CACHE_DIR)) {
+        try {
+            total += statSync(join(CACHE_DIR, name)).size;
+        }
+        catch { /* raced with a delete */ }
+    }
+    return total;
+}
+/** Block until there is room for a file of `fileBytes` under the ceiling.
+ *  A single file larger than the whole ceiling can never "fit", so we let it
+ *  through (it is deleted right after processing) rather than wait forever. */
+export async function ensureCacheRoom(fileBytes, signal, warn, maxWaitMs = CACHE_WAIT_MS) {
+    mkdirSync(CACHE_DIR, { recursive: true });
+    if (fileBytes >= MAX_CACHE_BYTES)
+        return;
+    let warned = false;
+    const until = Date.now() + maxWaitMs;
+    // Stop waiting the moment a shutdown is requested — the abort signal unblocks
+    // a long cache-full wait so Ctrl+C is never swallowed by the ceiling.
+    while (!signal.aborted && cacheSize() + fileBytes > MAX_CACHE_BYTES) {
+        // BOUNDED. Room appears when this run consumes and deletes a file, so a
+        // cache already over the ceiling with nothing left to consume — stale files
+        // from another run, a ceiling set below one corpus — would otherwise wait
+        // for room that cannot arrive, forever, after a single warning line.
+        if (Date.now() >= until) {
+            throw new Error(`cache still full after ${Math.round(maxWaitMs / 60_000)} min ` +
+                `(${(cacheSize() / 1e9).toFixed(1)} GB of a ` +
+                `${(MAX_CACHE_BYTES / 1e9).toFixed(0)} GB ceiling) — raise ` +
+                `MAX_CACHE_GB or clear ${CACHE_DIR}`);
+        }
+        if (!warned) {
+            warn?.(`cache at ${(MAX_CACHE_BYTES / 1e9).toFixed(0)} GB ceiling — waiting for room…`);
+            warned = true;
+        }
+        await waitMs(5_000, signal);
+    }
+}
+/** Stream `url` to `destPath`, atomically and with backpressure. */
+export async function downloadFile(url, destPath, opts) {
+    const partPath = destPath + PART_SUFFIX;
+    await retry(`download ${basename(destPath)}`, async () => {
+        // Abort promptly on shutdown rather than waiting out a slow socket.
+        if (opts.signal.aborted) {
+            const e = new Error("aborted");
+            e.fatal = true;
+            throw e;
+        }
+        const res = await fetch(url, { signal: opts.signal });
+        if (!res.ok)
+            throw httpError(res);
+        if (!res.body)
+            throw new Error("empty response body");
+        // `content-length` describes the bytes ON THE WIRE. When the server
+        // applied a content-coding, fetch hands us the DECODED body, so the
+        // header no longer describes what gets written to disk and the integrity
+        // guard below must not use it. Measured: raw.githubusercontent.com sends
+        // `content-encoding: gzip` with content-length 110,928 for a file that
+        // decodes to 1,607,931 bytes — a size check against that rejects every
+        // healthy download. (The bug stayed latent because Hugging Face sends
+        // `content-encoding: br` and NO content-length, leaving total = 0, which
+        // already disables the guard.)
+        const encoding = (res.headers.get("content-encoding") ?? "").trim()
+            .toLowerCase();
+        const decoded = encoding !== "" && encoding !== "identity";
+        const total = decoded
+            ? 0
+            : Number(res.headers.get("content-length")) || 0;
+        let done = 0;
+        // Stream straight to a ".part" sibling using pure WHATWG streams. A
+        // TransformStream meters progress; pipeTo into a WritableStream gives REAL
+        // backpressure natively — the sink's write() returns a promise the
+        // readable side awaits, so a fast server can never outrun the disk (no
+        // whole-file heap buffering). The sink wraps a single raw fs descriptor
+        // (the one capability the web platform lacks); writing to disk is the only
+        // Node operation in the whole pipeline. The final, valid file only ever
+        // appears via the atomic rename below, so a crash mid-transfer can never
+        // leave a truncated file at the real path.
+        const meter = new TransformStream({
+            transform(chunk, controller) {
+                done += chunk.length;
+                opts.onProgress?.(done, total);
+                controller.enqueue(chunk);
+            },
+        });
+        const fd = openSync(partPath, "w");
+        let closed = false;
+        const closeFd = () => {
+            if (closed)
+                return;
+            closed = true;
+            try {
+                closeSync(fd);
+            }
+            catch { /* already closed */ }
+        };
+        const sink = new WritableStream({
+            write(chunk) {
+                // writeSync drains the whole chunk before returning, so the readable
+                // side is paused for exactly as long as the disk needs — backpressure.
+                let off = 0;
+                while (off < chunk.length) {
+                    off += writeSync(fd, chunk, off, chunk.length - off);
+                }
+            },
+            close() {
+                fsyncSync(fd); // durable bytes before the rename promotes them
+                closeFd();
+            },
+            abort() {
+                closeFd();
+            },
+        });
+        try {
+            await res.body.pipeThrough(meter).pipeTo(sink, {
+                signal: opts.signal,
+            });
+        }
+        catch (e) {
+            // pipeTo's abort() ran the sink's abort() (closing the descriptor); if
+            // it didn't (a non-abort throw), make sure the descriptor is not leaked.
+            closeFd();
+            try {
+                unlinkSync(partPath);
+            }
+            catch { /* best effort */ }
+            throw e;
+        }
+        // Optional integrity guard: when the server advertised a size FOR THE
+        // BYTES WE WRITE (see the content-encoding note above — `total` is 0 for
+        // a decoded body, which disables this), a complete file must match it. A
+        // short read (silent truncation) is retried rather than promoted, so the
+        // parser never sees a partial file.
+        try {
+            const got = statSync(partPath).size;
+            if (total > 0 && got !== total) {
+                try {
+                    unlinkSync(partPath);
+                }
+                catch { /* best effort */ }
+                throw new Error(`size mismatch: got ${got}, expected ${total}`);
+            }
+        }
+        catch (e) {
+            if (e instanceof Error && e.message.startsWith("size mismatch")) {
+                throw e;
+            }
+            // statSync failure is non-fatal here; the rename below will surface it.
+        }
+        // Atomic publish: rename is atomic within a filesystem, so the final path
+        // flips from "absent" to "complete" in one step — never an in-between.
+        renameSync(partPath, destPath);
+    }, opts.tries, { signal: opts.signal, onFail: opts.onFail });
+}

package/dist/example/train_base/config.d.ts

@@ -0,0 +1,21 @@
+/** Read an environment variable, or `d` when it is unset. */
+export declare const env: (k: string, d: string) => string;
+export declare const DB_PATH: string;
+export declare const D: number;
+export declare const SEED: number;
+export declare const CHECKPOINT_BYTES: number;
+export declare const PARQUET_BATCH_BYTES: number;
+export declare const LOCAL_PATH: string;
+export declare const CACHE_DIR: string;
+export declare const MAX_CACHE_BYTES: number;
+export declare const PROGRESS_MS: number;
+export declare const INDEX_MAINTENANCE: boolean;
+export declare const DOWNLOAD_TRIES = 5;
+export declare const PART_SUFFIX = ".part";
+export declare const INFER_TIMEOUT_MS: number;
+export declare const STALL_MS: number;
+export declare const CACHE_WAIT_MS: number;
+export declare const VECTOR_CACHE_MB: number;
+export declare const SQLITE_CACHE_MB: number;
+export declare const MAX_MB: number;
+export declare const MAX_BYTES: number;

package/dist/example/train_base/config.js

@@ -0,0 +1,94 @@
+// train_base/config.ts — RUN-LEVEL configuration, all from the environment.
+//
+// Only knobs that describe the RUN live here: the store, the checkpoint cadence,
+// the cache ceiling, the read budgets, the caps. A knob that describes ONE
+// CORPUS (which pairs of SmolSent, how many SODA dialogues, how long an Aya
+// field may be) belongs next to that corpus's adapter, together with the
+// evidence that fixed its default — see AGENTS.md §2.16: a comment carries the
+// constraint, and a constraint is only readable beside the code it constrains.
+import { join } from "node:path";
+/** Read an environment variable, or `d` when it is unset. */
+export const env = (k, d) => process.env[k] ?? d;
+export const DB_PATH = env("DB_PATH", "sema"); // → {DB_PATH}.sqlite
+export const D = Number(env("D", "1024"));
+export const SEED = Number(env("SEED", "7"));
+// Checkpoint cadence is measured in LEARNED CONTENT, not deposits: a snapshot
+// every CHECKPOINT_MB megabytes of trained UTF-8 content (decimal MB, matching
+// the bytes() helper). A floor of 1 MB: a zero/NaN value must not make every
+// deposit checkpoint, nor silently disable checkpointing. The tail (a run that
+// learns less than one interval, or the remainder past the last interval) is
+// always saved by finish() at exit — a complete point.
+export const CHECKPOINT_BYTES = Math.max(1_000_000, Math.floor(Number(env("CHECKPOINT_MB", "100")) * 1_000_000) || 100_000_000);
+// Target size of ONE materialised Parquet read, in uncompressed source bytes.
+// A row-GROUP is a layout choice made by whoever wrote the file, not a memory
+// budget: Aya ships 203 groups of 1,000 rows (~1 MB each), while SODA ships ONE
+// group of 1,191,582 rows (1.19 GB uncompressed) and 2Wiki ONE of 167,454
+// (666 MB). Reading "exactly one row-group" is therefore safe for the first and
+// fatal for the others, so reads are sized in BYTES instead — see
+// `parquetBatchRows`. Materialised JS objects cost several times their source
+// bytes, hence a default well under available memory.
+export const PARQUET_BATCH_BYTES = Math.max(1_000_000, Math.floor(Number(env("PARQUET_BATCH_MB", "32")) * 1_000_000) || 32_000_000);
+export const LOCAL_PATH = env("LOCAL_PATH", ""); // train from a local dir
+export const CACHE_DIR = env("CACHE_DIR", join(process.cwd(), "cache"));
+export const MAX_CACHE_BYTES = Number(env("MAX_CACHE_GB", "100")) * 1e9;
+export const PROGRESS_MS = Number(env("PROGRESS_MS", "250")); // panel cadence
+// Index maintenance at checkpoints: compact (remove garbage), repair (fill
+// gaps), then refresh the canonical-form index (equivalence-class resolution —
+// src/canon.ts). All three are idempotent batch operations (the canon build is
+// additionally incremental via the store's `canon.upto` cursor);
+// INDEX_MAINTENANCE=0 disables.
+export const INDEX_MAINTENANCE = env("INDEX_MAINTENANCE", "1") !== "0";
+export const DOWNLOAD_TRIES = 5;
+// In-progress downloads are written to a sibling "<dest>.part" and atomically
+// renamed into place only after the bytes are fully flushed to disk. The cache
+// invariant is therefore absolute: a file at its final path is, by definition,
+// complete. Partial transfers (a crash, a kill, a dropped socket) leave only a
+// .part file, which is never fed to the parser and is swept at startup by
+// cache.ts's sweepPartials() — without which the debris would consume cache
+// ceiling that nothing frees.
+export const PART_SUFFIX = ".part";
+// The checkpoint recall is a best-effort diagnostic — it must NEVER stall
+// training. We bound it so a slow/large store cannot freeze the deposit loop.
+export const INFER_TIMEOUT_MS = Number(env("INFER_TIMEOUT_MS", "15000"));
+// How long the run may make NO progress before it gives up and exits non-zero.
+//
+// A long training run's worst failure is not a crash — a crash resumes. It is a
+// HANG: the uncaught-exception handler deliberately swallows dropped-connection
+// errors so a long run survives them, and the keep-alive timer deliberately
+// holds the process open; together, an error that escapes and leaves an await
+// unsettled produces a live process that will never do anything again. No error,
+// no exit, and a supervisor that sees a healthy pid. Exiting instead turns that
+// into a resume, which costs at most the work since the last checkpoint.
+//
+// "Progress" is any deposit, downloaded chunk, or rate-limit wait; time inside
+// index maintenance and the checkpoint recall does not count against it, since
+// those legitimately deposit nothing. Generous by default — this is a
+// last-resort backstop, not a latency budget. 0 disables it.
+export const STALL_MS = Math.max(0, Math.floor(Number(env("STALL_MIN", "15")) * 60_000) || 900_000);
+// How long a download may wait for room under the cache ceiling before failing
+// the unit instead of waiting forever. The wait exists so a bounded cache can
+// throttle a fast source; it is not meant to outlast the run. The unit stays
+// resumable, so a genuine ceiling problem costs a retry, not the corpus.
+export const CACHE_WAIT_MS = Math.max(60_000, Math.floor(Number(env("CACHE_WAIT_MIN", "10")) * 60_000) || 600_000);
+// The vector indices' memory knob (MiB) — each index's SQLite page cache.
+// The IVF index routes inserts through a RAM-resident pivot table and
+// appends to chunk blobs, so this cache mostly serves query-time cluster
+// scans; 256 MiB comfortably covers the probed working set of a trained
+// store.  Override with VECTOR_CACHE_MB (64 is the library default).
+export const VECTOR_CACHE_MB = Math.max(0, Number(env("VECTOR_CACHE_MB", "256")));
+// Page cache for the MAIN DAG database (node/kid/edge/contain tables).
+// Training issues millions of content-addressed point probes per session
+// against a GB-scale file; the library default (64 MiB) is sized for a
+// small machine — a training box affords more.  Override with
+// SQLITE_CACHE_MB.
+export const SQLITE_CACHE_MB = Math.max(0, Number(env("SQLITE_CACHE_MB", "256")));
+// Optional ceiling on how much LEARNED CONTENT to train, in megabytes (decimal,
+// like CHECKPOINT_MB). Default Infinity = unbounded. The cap is checked against
+// trainedContentBytes after each deposit, so a run stops at the first item that
+// carries the running total to/past the ceiling (that item is still counted).
+export const MAX_MB = Number(env("MAX_MB", "Infinity"));
+if (isNaN(MAX_MB) || MAX_MB < 0) {
+    process.stderr.write(`fatal: MAX_MB must be a non-negative number or "Infinity"\n`);
+    process.exit(1);
+}
+export const MAX_BYTES = MAX_MB * 1_000_000; // Infinity stays Infinity

package/dist/example/train_base/corpora/aya.d.ts

@@ -0,0 +1,19 @@
+import { type TrainingItem } from "../items.js";
+import { type Corpus } from "../corpus.js";
+/** One normalized Aya row. */
+export interface AyaRow {
+    inputs: string;
+    targets: string;
+    language: string;
+}
+/** Normalize a raw datasets-server row object into an AyaRow, or null when it
+ *  lacks a usable prompt/answer or a field is implausibly large (a dump, not a
+ *  cognitive example). Trims surrounding whitespace; keeps inner text verbatim
+ *  (human prose, possibly multi-paragraph). */
+export declare function toAyaRow(row: unknown, maxChars?: number): AyaRow | null;
+/** Translate ONE Aya row into SEMA training items. A row is a single human
+ *  (question → answer) exchange — exactly one FACT, the (inputs → targets) edge.
+ *  No standalone-answer experience and no one-exchange "cumulative" walk: a lone
+ *  Q→A is not multi-turn, and both would only replicate the same edge. */
+export declare function ayaRowToItems(row: AyaRow): TrainingItem[];
+export declare const aya: Corpus;