@hviana/sema 0.5.8 → 0.5.9

package/AGENTS.md

@@ -810,3 +810,26 @@ PolyForm Noncommercial 1.0.0 with separate commercial licensing (see
 vendor code under licenses incompatible with dual distribution, and do not add
 runtime dependencies casually — the near-zero-dependency footprint is a product
 feature.
+
+**Training corpora are governed by the same rule, and more strictly.** Sema is
+non-parametric: a trained store retains its training text VERBATIM (read any
+content node back and the original sentence comes out). A store is therefore a
+redistribution of its corpora, not a derived model, and every upstream licence
+applies to it in full. Two consequences:
+
+- A corpus carrying a **NonCommercial** term cannot enter a trainer — it
+  conflicts with the commercial licence tier.
+- A corpus carrying a **ShareAlike** term cannot enter a trainer — its copyleft
+  would attach to the distributed store.
+
+Check both against **what the corpus was built from**, not only the repository's
+license tag: a dataset assembled out of Wikipedia prose and published under
+Apache-2.0 still carries CC BY-SA on that prose. Where a corpus has a clean
+layer and a contaminated one, ingest only the clean layer.
+
+Sema's own license does **not** extend over corpus content inside a store, and
+cannot: CC BY 4.0 §2(a)(5)(B) forbids applying terms that restrict what the
+license permits. The engine is what PolyForm protects. Per-corpus attribution,
+the required modification statement, and the current allow/deny list live in
+[DATASETS.md](DATASETS.md) — update it in the same change that touches a
+trainer's corpus set.

package/DATASETS.md

@@ -0,0 +1,159 @@
+# Training corpora — provenance, licensing, and attribution
+
+This file is the attribution notice for every corpus Sema is trained on, and the
+licensing statement for the **trained memory files** that training produces.
+
+It is a required companion to any distributed Sema store. If you publish or ship
+`*.sqlite` / `*.content.vec` / `*.halo.vec`, ship this file with them.
+
+---
+
+## 1. Why a trained store is not a weight file
+
+Sema is non-parametric. Training is deposition, not gradient descent: source
+text is segmented and content-addressed, and the **bytes are retained**. Reading
+a node returns the original text:
+
+```
+#15709469 → "Kohei Uchimura from Japan holds the record for the most World
+             Championship medals won by a male gymnast, with a total of 21 medals."
+```
+
+A trained store is therefore a database that contains its training corpora in
+recoverable form. Distributing one **is** distributing those corpora, and every
+upstream licence applies in full. The "it's only model weights, the text isn't
+really in there" argument is not available to Sema, by design.
+
+Two consequences follow, and both are load-bearing:
+
+1. **A corpus whose licence forbids commercial use cannot enter the store**,
+   because Sema is offered under a paid commercial licence as well as
+   [PolyForm Noncommercial](LICENSE.md).
+2. **A corpus under a ShareAlike licence cannot enter the store**, because its
+   copyleft would attach to the distributed artifact.
+
+Both rules are stated in [AGENTS.md](AGENTS.md) §6 and must be checked before
+any corpus is added to a trainer.
+
+---
+
+## 2. How a distributed store is licensed
+
+A trained Sema store has two layers, licensed separately. Conflating them is a
+licence violation in one direction or the other.
+
+| Layer                           | What it covers                                                                                          | Licence                                                                                                 |
+| :------------------------------ | :------------------------------------------------------------------------------------------------------ | :------------------------------------------------------------------------------------------------------ |
+| **Sema itself**                 | The algorithms, geometry, deduction engine, indexes, file formats, and all code that produced the store | [PolyForm Noncommercial 1.0.0](LICENSE.md), with a separate [commercial licence](COMMERCIAL-LICENSE.md) |
+| **Corpus content in the store** | The retained training text and anything derived from it                                                 | Each corpus's own upstream licence, listed in §3                                                        |
+
+**Sema's source licence is not extended over the corpus content, and cannot
+be.** CC BY 4.0 §2(a)(5)(B) forbids applying legal terms that restrict a
+recipient from doing what the licence permits — so the noncommercial term cannot
+be applied to CC BY text sitting inside the store. What the noncommercial term
+protects is the engine, which is the part that is actually ours.
+
+**Modification statement** (required by CC BY 4.0 §3(a)(1)(B)): all corpus text
+in a Sema store has been modified. It is segmented at content-defined
+boundaries, re-encoded, deduplicated by content address, and interleaved with
+text from other sources. It is not presented as a faithful reproduction of any
+upstream dataset, and no endorsement by any upstream author is implied.
+
+**Apache-2.0 obligations**: corpora marked Apache-2.0 below require the licence
+text and any upstream `NOTICE` to travel with the distribution, and require
+changes to be stated. The modification statement above satisfies the latter.
+
+---
+
+## 3. Corpora
+
+### 3.1 In use
+
+| Corpus                                                                                                   | Licence                                               | Attribution                        | Notes                                                                                                                                                                                                                                                                 |
+| :------------------------------------------------------------------------------------------------------- | :---------------------------------------------------- | :--------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [google/smol](https://huggingface.co/datasets/google/smol) (SmolSent)                                    | CC BY 4.0                                             | Google LLC                         | Translation pairs                                                                                                                                                                                                                                                     |
+| [CohereLabs/aya_dataset](https://huggingface.co/datasets/CohereLabs/aya_dataset)                         | Apache-2.0                                            | Cohere For AI                      | Human-written prompt/completion                                                                                                                                                                                                                                       |
+| [OpenAssistant/oasst2](https://huggingface.co/datasets/OpenAssistant/oasst2)                             | Apache-2.0                                            | LAION / OpenAssistant contributors | Human-authored dialogue — see §5                                                                                                                                                                                                                                      |
+| [Taskmaster-1/2/3/4](https://github.com/google-research-datasets/Taskmaster)                             | CC BY 4.0                                             | Google LLC                         | Task-oriented dialogue. Only `utterances[].text` is ingested; the `instructions` / `scenario` / `vertical` fields are never read                                                                                                                                      |
+| [2WikiMultihopQA](https://huggingface.co/datasets/xanhho/2WikiMultihopQA) — **`evidences` triples only** | Apache-2.0 (repo); triples originate in Wikidata, CC0 | Ho et al.; Wikidata contributors   | Only the `evidences` column is ingested. The `context` column (Wikipedia prose, CC BY-SA) is **never read** — see §4. The `question`/`answer` columns are also never deposited, for a capability reason rather than a licence one: they memorise instead of composing |
+| [allenai/soda](https://huggingface.co/datasets/allenai/soda)                                             | CC BY 4.0                                             | Allen Institute for AI             | Social dialogue. Only the `dialogue` column is ingested; `narrative` / `literal` / `head` / `relation` / `tail` are never read. Model-generated provenance — see §5                                                                                                   |
+| [AmazonScience/massive](https://huggingface.co/datasets/AmazonScience/massive)                           | CC BY 4.0                                             | Amazon Science                     | Short multilingual intents. Only the `utt` column is ingested; the slot-annotated `annot_utt` is never read. **Disabled by default** on capability grounds (not licence) — see `MASSIVE` in `example/train_base.ts`                                                   |
+
+### 3.2 Excluded, and why
+
+| Corpus                                                            | Reason                                                                                                                                                                                                                                                                                                                                          |
+| :---------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **MuskumPillerum/General-Knowledge**                              | **No licence at all.** The HF repo carries no licence tag and no licence in its card; an earlier header in `example/train_base.ts` claimed MIT without support. Its own card states it "contains a subset of the alpaca dataset", and Alpaca is CC BY-NC 4.0 — **NonCommercial**, incompatible with Sema's commercial licence. See §6.          |
+| **PAWS**                                                          | Google's own grant is maximally permissive ("may be freely used for any purpose"), but PAWS-Wiki sentences derive from Wikipedia (CC BY-**SA**) and PAWS-QQP from Quora question pairs under Quora's terms. Because Sema retains text verbatim, the upstream terms would attach to the distributed store. Excluded despite strong measured fit. |
+| **Schema-Guided Dialogue (SGD/dstc8)**, **HotpotQA**, **MuSiQue** | CC BY-SA 4.0 — ShareAlike conflicts with dual distribution.                                                                                                                                                                                                                                                                                     |
+| **2WikiMultihopQA passages**                                      | Wikipedia prose, CC BY-SA. The repo's Apache-2.0 tag does not relicense the text it was built from. Only the Wikidata-derived `evidences` triples are ingested.                                                                                                                                                                                 |
+| **Alpaca** and derivatives                                        | CC BY-NC 4.0, and generated from OpenAI model outputs.                                                                                                                                                                                                                                                                                          |
+
+---
+
+### 3.3 What each stage actually deposits
+
+A corpus's licence applies to what is ingested, and every stage ingests a strict
+subset of its source. This is the authoritative list.
+
+| Stage                    | Columns/fields read                       | Deposit shape                                             |
+| :----------------------- | :---------------------------------------- | :-------------------------------------------------------- |
+| SmolSent                 | `src`, `trg`                              | one `src → trg` (foreign → English) episode per row       |
+| Aya                      | `inputs`, `targets`                       | one question → answer episode                             |
+| oasst2                   | message `text` along the best-ranked path | cumulative-context walk                                   |
+| Taskmaster               | `utterances[].text`                       | cumulative-context walk over speaker-merged turns         |
+| 2Wiki                    | `evidences`                               | per triple: a relation fact and a bare-subject pivot fact |
+| SODA                     | `dialogue`, `speakers`                    | cumulative-context walk over speaker-merged turns         |
+| MASSIVE (off by default) | `utt`                                     | one bare experience                                       |
+
+Everything else in those sources — Taskmaster's `instructions`/`scenario`,
+2Wiki's `context`/`question`/`answer`, SODA's `narrative`/`literal`/`head`/
+`relation`/`tail`, MASSIVE's `annot_utt` — is **not read** and therefore not
+distributed in a trained store.
+
+---
+
+## 4. The rule that decides these cases
+
+**A repository's licence tag does not relicense the material the repository was
+built from.** A dataset assembled out of Wikipedia prose and published under
+Apache-2.0 still carries Wikipedia's ShareAlike terms on that prose. Because
+Sema stores text verbatim, Sema inherits the _upstream_ terms, not the
+repackager's.
+
+So the check for any candidate corpus is two questions, not one:
+
+1. What licence does the repository carry?
+2. **What was it built from, and what licence does that carry?**
+
+Where a corpus has a clean layer and a contaminated one, take the clean layer
+only — as with 2Wiki's Wikidata triples (CC0) versus its Wikipedia passages (CC
+BY-SA).
+
+---
+
+## 5. Disclosures
+
+**Model-generated provenance.** `allenai/soda` is licensed CC BY 4.0 but was
+distilled from OpenAI GPT-3.5 outputs. The licence is clean; the provenance is
+disclosed here so downstream users can make their own assessment.
+
+**Personal data.** `OpenAssistant/oasst2` is human-authored content contributed
+by identifiable volunteers, and Sema retains it verbatim in a redistributable
+artifact. Erasure requests against a content-addressed store are not
+straightforward. Anyone distributing a Sema store trained on human-contributed
+dialogue should account for this.
+
+---
+
+## 6. Status of previously published stores
+
+Stores published before this file was written — including those under
+[hviana/sema-trained-v1](https://huggingface.co/buckets/hviana/sema-trained-v1)
+— were trained with the `MuskumPillerum/General-Knowledge` stage enabled (37,623
+rows), whose licence status is described in §3.2. Those artifacts should be
+treated as **not redistributable** until retrained without that stage.
+
+The stage is now **disabled by default** in `example/train_base.ts`
+(`GENKNOW=0`). The adapter code remains so the stage can be re-enabled for local
+experiments; a store trained with `GENKNOW=1` must not be distributed.

package/README.md

@@ -320,6 +320,7 @@ start talking — no install, no runtime, no API key.
 | 🛠️ **[AGENTS.md](AGENTS.md)**                                                        | The development manual: repo layout, build/test, internals, invariants, and recipes for extending the system.                                            |
 | 🎓 **[CITATION.cff](CITATION.cff)**                                                  | How to cite Sema in academic work.                                                                                                                       |
 | ⚖️ **[LICENSE.md](LICENSE.md)**                                                      | PolyForm Noncommercial License 1.0.0.                                                                                                                    |
+| 📚 **[DATASETS.md](DATASETS.md)**                                                    | Training corpora: provenance, per-corpus attribution, and how a trained memory file is licensed.                                                         |
 | 💼 **[COMMERCIAL-LICENSE.md](COMMERCIAL-LICENSE.md)**                                | Commercial licensing terms and contact.                                                                                                                  |
 | 🤗 **[Trained examples](https://huggingface.co/buckets/hviana/sema-trained-v1)**     | Pre-trained memory files you can download and use directly.                                                                                              |
 | 💿 **[Binary examples](https://huggingface.co/buckets/hviana/sema-binary-examples)** | Ready-to-run web chat apps for Windows, Mac, and Linux — one file, no install.                                                                           |
@@ -350,6 +351,17 @@ start talking — no install, no runtime, no API key.
 > **[TRADEMARKS.md](TRADEMARKS.md)** — the **Sema** name, logos, and brand are
 > _not_ covered by the source license.
 
+> [!NOTE]
+> **Trained memory files are licensed in two layers.** Sema is non-parametric,
+> so a trained store retains its training text verbatim — it is a database that
+> contains its corpora, not a weight file. The **engine** (algorithms, geometry,
+> deduction, indexes, formats, and all code) is under PolyForm Noncommercial
+> with a separate commercial license. The **corpus content inside a store**
+> stays under each corpus's own upstream license, and Sema's license is not
+> extended over it. If you distribute a store, ship
+> **[DATASETS.md](DATASETS.md)** with it — it carries the attribution and
+> modification statement those licenses require.
+
 <div align="center">
 
 **Respecting these terms funds the research that makes work like this

package/dist/example/train_base.d.ts

@@ -16,9 +16,12 @@ export interface SmolSentRow {
 /** Normalize a raw datasets-server row into a SmolSentRow, or null when it lacks
  *  both sides or a side is implausibly large (a dump, not a sentence). */
 export declare function toSmolSentRow(row: unknown): SmolSentRow | null;
-/** Translate ONE SmolSent pair into SEMA facts: the two sentences are one
- *  meaning in two languages, so bind them BOTH ways. refineItems drops the
- *  degenerate case where src === trg. */
+/** Translate ONE SmolSent pair into SEMA facts. The two sentences are one
+ *  meaning in two languages, but the two BINDINGS are not equally sound —
+ *  SmolSent's English side is a shared pool translated into every language, so
+ *  `trg -> src` gives one English context a different answer in every language
+ *  file. See SMOLSENT_DIRECTIONS. refineItems drops the degenerate case where
+ *  src === trg. */
 export declare function smolSentRowToItems(row: SmolSentRow): TrainingItem[];
 /** One normalized Aya row. */
 export interface AyaRow {
@@ -79,6 +82,57 @@ export declare function bestOasstPath(root: OasstNode): OasstTurn[];
  *  Returns [] for a conversation below the multi-turn threshold, so callers can
  *  simply skip empties. */
 export declare function oasstConversationToItems(turns: OasstTurn[]): TrainingItem[];
+/** One utterance of a Taskmaster conversation. */
+export interface TaskmasterTurn {
+    speaker: string;
+    text: string;
+}
+/** Normalize ONE element of a Taskmaster data file into its turns, or null when
+ *  it carries no usable utterance. Empty/whitespace-only utterances are dropped
+ *  (TM-3 has a few); a single implausibly long utterance rejects the whole
+ *  conversation as corrupt rather than depositing a dump. */
+export declare function toTaskmasterTurns(row: unknown): TaskmasterTurn[] | null;
+/** Collapse consecutive same-speaker turns into one, joining with a space, and
+ *  return the bare texts in order. A turn with no speaker never merges with its
+ *  neighbour: an unlabelled row is of unknown origin, and joining two of them
+ *  would invent a contribution that may span two speakers. */
+export declare function mergeTaskmasterTurns(turns: TaskmasterTurn[]): string[];
+/** Translate ONE Taskmaster conversation into SEMA training items: the
+ *  cumulative walk over its merged turns. Returns [] for a conversation below
+ *  TASKMASTER_MIN_TURNS, so callers can simply skip empties. */
+export declare function taskmasterConversationToItems(turns: TaskmasterTurn[]): TrainingItem[];
+/** One (subject, relation, object) triple from a 2Wiki `evidences` cell. */
+export interface WikiTriple {
+    subject: string;
+    relation: string;
+    object: string;
+}
+/** Normalize a 2Wiki row into its evidence triples, or null when it carries
+ *  none usable. `evidences` is a JSON STRING holding an array of 3-element
+ *  arrays; a row whose cell is absent, unparseable, or empty yields null.
+ *  Individual malformed or oversized triples are dropped without discarding the
+ *  row — one bad triple should not cost the others. */
+export declare function toWikiTriples(row: unknown): WikiTriple[] | null;
+/** Render ONE triple as the prose fact Sema stores. Kept separate so the two
+ *  deposits below are guaranteed to share a byte-identical continuation: the
+ *  pivot fact only works if it leads to the SAME node the relation fact does. */
+export declare function wikiTripleSentence(t: WikiTriple): string;
+/** Translate a row's triples into SEMA items: per triple, the relation fact and
+ *  the bare-subject PIVOT fact (see the section note above). refineItems drops
+ *  the duplicates this produces when a row states the same triple twice. */
+export declare function wikiTriplesToItems(triples: WikiTriple[]): TrainingItem[];
+/** Normalize a SODA row into its turns, or null when it carries no usable
+ *  dialogue. Speakers are optional (they only drive merging); an implausibly
+ *  long turn rejects the dialogue as corrupt. */
+export declare function toSodaTurns(row: unknown): TaskmasterTurn[] | null;
+/** Translate ONE SODA dialogue into SEMA items: the cumulative walk over its
+ *  speaker-merged turns. Shares `mergeTaskmasterTurns` because the rule is the
+ *  same one — consecutive turns by one speaker are one contribution. */
+export declare function sodaDialogueToItems(turns: TaskmasterTurn[]): TrainingItem[];
+/** Translate ONE MASSIVE row into SEMA items: its bare utterance, as an
+ *  experience. `annot_utt` (slot-annotated) is deliberately not used — its
+ *  "[date : ...]" markup is not prose. Returns [] for an unusable row. */
+export declare function massiveRowToItems(row: unknown): TrainingItem[];
 /** One normalized General-Knowledge row. */
 export interface GenKnowRow {
     question: string;
@@ -90,4 +144,20 @@ export declare function toGenKnowRow(row: unknown): GenKnowRow | null;
 /** Translate ONE General-Knowledge row into SEMA items: exactly one
  *  (question → answer) FACT. refineItems drops a degenerate question === answer. */
 export declare function genKnowRowToItems(row: GenKnowRow): TrainingItem[];
+/** How many rows to materialise in one read from a row-group of `rgRows` rows
+ *  occupying `groupBytes` uncompressed bytes, under a `budgetBytes` target.
+ *
+ *  The group's own footer statistics give the mean row width, so the batch
+ *  follows the CORPUS's row size rather than the writer's layout: wide rows
+ *  (SODA carries a whole dialogue per row) batch smaller than narrow ones at
+ *  the same memory cost. Never exceeds the group — a batch is a subdivision of
+ *  a group, never a span across two, because `parquetReadObjects` is given an
+ *  absolute row range and column chunks are per-group. Never returns 0, or the
+ *  read loop could not advance.
+ *
+ *  A writer that omits `total_byte_size` yields `groupBytes <= 0`; the batch is
+ *  then the whole group, which is exactly the behaviour this replaced. That
+ *  fallback is safe for every file we read today (all three report it) and
+ *  degrades to the old memory profile rather than to a wrong result. */
+export declare function parquetBatchRows(rgRows: number, groupBytes: number, budgetBytes: number): number;
 export {};