@hviana/sema 0.5.7 → 0.5.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +23 -0
- package/DATASETS.md +159 -0
- package/HOW_IT_WORKS.md +74 -0
- package/README.md +12 -0
- package/dist/example/train_base.d.ts +73 -3
- package/dist/example/train_base.js +1000 -49
- package/dist/src/geometry.d.ts +20 -0
- package/dist/src/geometry.js +22 -0
- package/dist/src/mind/articulation.js +15 -2
- package/dist/src/mind/attention.d.ts +6 -0
- package/dist/src/mind/attention.js +44 -4
- package/dist/src/mind/learning.js +250 -3
- package/dist/src/mind/mechanisms/cast.js +45 -1
- package/dist/src/mind/mind.d.ts +6 -1
- package/dist/src/mind/mind.js +14 -2
- package/dist/src/mind/reasoning.js +59 -5
- package/dist/src/mind/recognition.js +29 -3
- package/dist/src/mind/traverse.d.ts +34 -0
- package/dist/src/mind/traverse.js +42 -0
- package/dist/src/store-sqlite.d.ts +4 -0
- package/dist/src/store-sqlite.js +47 -0
- package/dist/src/store.d.ts +7 -0
- package/example/train_base.ts +1193 -46
- package/jsr.json +1 -1
- package/package.json +1 -1
- package/src/geometry.ts +23 -0
- package/src/mind/articulation.ts +16 -2
- package/src/mind/attention.ts +54 -1
- package/src/mind/learning.ts +253 -4
- package/src/mind/mechanisms/cast.ts +48 -1
- package/src/mind/mind.ts +12 -1
- package/src/mind/reasoning.ts +64 -5
- package/src/mind/recognition.ts +29 -3
- package/src/mind/traverse.ts +48 -0
- package/src/store-sqlite.ts +53 -0
- package/src/store.ts +28 -0
- package/test/29-counterfactual.test.mjs +43 -6
- package/test/76-type-level-company.test.mjs +342 -0
- package/test/77-company-saturation.test.mjs +302 -0
- package/test/78-atom-hub-recognition-cliff.test.mjs +135 -0
- package/test/84-composed-answer-honesty.test.mjs +136 -0
- package/test/85-answered-directly.test.mjs +126 -0
- package/test/86-cast-voices-committed.test.mjs +164 -0
- package/test/87-codominant-commitment.test.mjs +250 -0
package/AGENTS.md
CHANGED
|
@@ -810,3 +810,26 @@ PolyForm Noncommercial 1.0.0 with separate commercial licensing (see
|
|
|
810
810
|
vendor code under licenses incompatible with dual distribution, and do not add
|
|
811
811
|
runtime dependencies casually — the near-zero-dependency footprint is a product
|
|
812
812
|
feature.
|
|
813
|
+
|
|
814
|
+
**Training corpora are governed by the same rule, and more strictly.** Sema is
|
|
815
|
+
non-parametric: a trained store retains its training text VERBATIM (read any
|
|
816
|
+
content node back and the original sentence comes out). A store is therefore a
|
|
817
|
+
redistribution of its corpora, not a derived model, and every upstream licence
|
|
818
|
+
applies to it in full. Two consequences:
|
|
819
|
+
|
|
820
|
+
- A corpus carrying a **NonCommercial** term cannot enter a trainer — it
|
|
821
|
+
conflicts with the commercial licence tier.
|
|
822
|
+
- A corpus carrying a **ShareAlike** term cannot enter a trainer — its copyleft
|
|
823
|
+
would attach to the distributed store.
|
|
824
|
+
|
|
825
|
+
Check both against **what the corpus was built from**, not only the repository's
|
|
826
|
+
license tag: a dataset assembled out of Wikipedia prose and published under
|
|
827
|
+
Apache-2.0 still carries CC BY-SA on that prose. Where a corpus has a clean
|
|
828
|
+
layer and a contaminated one, ingest only the clean layer.
|
|
829
|
+
|
|
830
|
+
Sema's own license does **not** extend over corpus content inside a store, and
|
|
831
|
+
cannot: CC BY 4.0 §2(a)(5)(B) forbids applying terms that restrict what the
|
|
832
|
+
license permits. The engine is what PolyForm protects. Per-corpus attribution,
|
|
833
|
+
the required modification statement, and the current allow/deny list live in
|
|
834
|
+
[DATASETS.md](DATASETS.md) — update it in the same change that touches a
|
|
835
|
+
trainer's corpus set.
|
package/DATASETS.md
ADDED
|
@@ -0,0 +1,159 @@
|
|
|
1
|
+
# Training corpora — provenance, licensing, and attribution
|
|
2
|
+
|
|
3
|
+
This file is the attribution notice for every corpus Sema is trained on, and the
|
|
4
|
+
licensing statement for the **trained memory files** that training produces.
|
|
5
|
+
|
|
6
|
+
It is a required companion to any distributed Sema store. If you publish or ship
|
|
7
|
+
`*.sqlite` / `*.content.vec` / `*.halo.vec`, ship this file with them.
|
|
8
|
+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
## 1. Why a trained store is not a weight file
|
|
12
|
+
|
|
13
|
+
Sema is non-parametric. Training is deposition, not gradient descent: source
|
|
14
|
+
text is segmented and content-addressed, and the **bytes are retained**. Reading
|
|
15
|
+
a node returns the original text:
|
|
16
|
+
|
|
17
|
+
```
|
|
18
|
+
#15709469 → "Kohei Uchimura from Japan holds the record for the most World
|
|
19
|
+
Championship medals won by a male gymnast, with a total of 21 medals."
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
A trained store is therefore a database that contains its training corpora in
|
|
23
|
+
recoverable form. Distributing one **is** distributing those corpora, and every
|
|
24
|
+
upstream licence applies in full. The "it's only model weights, the text isn't
|
|
25
|
+
really in there" argument is not available to Sema, by design.
|
|
26
|
+
|
|
27
|
+
Two consequences follow, and both are load-bearing:
|
|
28
|
+
|
|
29
|
+
1. **A corpus whose licence forbids commercial use cannot enter the store**,
|
|
30
|
+
because Sema is offered under a paid commercial licence as well as
|
|
31
|
+
[PolyForm Noncommercial](LICENSE.md).
|
|
32
|
+
2. **A corpus under a ShareAlike licence cannot enter the store**, because its
|
|
33
|
+
copyleft would attach to the distributed artifact.
|
|
34
|
+
|
|
35
|
+
Both rules are stated in [AGENTS.md](AGENTS.md) §6 and must be checked before
|
|
36
|
+
any corpus is added to a trainer.
|
|
37
|
+
|
|
38
|
+
---
|
|
39
|
+
|
|
40
|
+
## 2. How a distributed store is licensed
|
|
41
|
+
|
|
42
|
+
A trained Sema store has two layers, licensed separately. Conflating them is a
|
|
43
|
+
licence violation in one direction or the other.
|
|
44
|
+
|
|
45
|
+
| Layer | What it covers | Licence |
|
|
46
|
+
| :------------------------------ | :------------------------------------------------------------------------------------------------------ | :------------------------------------------------------------------------------------------------------ |
|
|
47
|
+
| **Sema itself** | The algorithms, geometry, deduction engine, indexes, file formats, and all code that produced the store | [PolyForm Noncommercial 1.0.0](LICENSE.md), with a separate [commercial licence](COMMERCIAL-LICENSE.md) |
|
|
48
|
+
| **Corpus content in the store** | The retained training text and anything derived from it | Each corpus's own upstream licence, listed in §3 |
|
|
49
|
+
|
|
50
|
+
**Sema's source licence is not extended over the corpus content, and cannot
|
|
51
|
+
be.** CC BY 4.0 §2(a)(5)(B) forbids applying legal terms that restrict a
|
|
52
|
+
recipient from doing what the licence permits — so the noncommercial term cannot
|
|
53
|
+
be applied to CC BY text sitting inside the store. What the noncommercial term
|
|
54
|
+
protects is the engine, which is the part that is actually ours.
|
|
55
|
+
|
|
56
|
+
**Modification statement** (required by CC BY 4.0 §3(a)(1)(B)): all corpus text
|
|
57
|
+
in a Sema store has been modified. It is segmented at content-defined
|
|
58
|
+
boundaries, re-encoded, deduplicated by content address, and interleaved with
|
|
59
|
+
text from other sources. It is not presented as a faithful reproduction of any
|
|
60
|
+
upstream dataset, and no endorsement by any upstream author is implied.
|
|
61
|
+
|
|
62
|
+
**Apache-2.0 obligations**: corpora marked Apache-2.0 below require the licence
|
|
63
|
+
text and any upstream `NOTICE` to travel with the distribution, and require
|
|
64
|
+
changes to be stated. The modification statement above satisfies the latter.
|
|
65
|
+
|
|
66
|
+
---
|
|
67
|
+
|
|
68
|
+
## 3. Corpora
|
|
69
|
+
|
|
70
|
+
### 3.1 In use
|
|
71
|
+
|
|
72
|
+
| Corpus | Licence | Attribution | Notes |
|
|
73
|
+
| :------------------------------------------------------------------------------------------------------- | :---------------------------------------------------- | :--------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
74
|
+
| [google/smol](https://huggingface.co/datasets/google/smol) (SmolSent) | CC BY 4.0 | Google LLC | Translation pairs |
|
|
75
|
+
| [CohereLabs/aya_dataset](https://huggingface.co/datasets/CohereLabs/aya_dataset) | Apache-2.0 | Cohere For AI | Human-written prompt/completion |
|
|
76
|
+
| [OpenAssistant/oasst2](https://huggingface.co/datasets/OpenAssistant/oasst2) | Apache-2.0 | LAION / OpenAssistant contributors | Human-authored dialogue — see §5 |
|
|
77
|
+
| [Taskmaster-1/2/3/4](https://github.com/google-research-datasets/Taskmaster) | CC BY 4.0 | Google LLC | Task-oriented dialogue. Only `utterances[].text` is ingested; the `instructions` / `scenario` / `vertical` fields are never read |
|
|
78
|
+
| [2WikiMultihopQA](https://huggingface.co/datasets/xanhho/2WikiMultihopQA) — **`evidences` triples only** | Apache-2.0 (repo); triples originate in Wikidata, CC0 | Ho et al.; Wikidata contributors | Only the `evidences` column is ingested. The `context` column (Wikipedia prose, CC BY-SA) is **never read** — see §4. The `question`/`answer` columns are also never deposited, for a capability reason rather than a licence one: they memorise instead of composing |
|
|
79
|
+
| [allenai/soda](https://huggingface.co/datasets/allenai/soda) | CC BY 4.0 | Allen Institute for AI | Social dialogue. Only the `dialogue` column is ingested; `narrative` / `literal` / `head` / `relation` / `tail` are never read. Model-generated provenance — see §5 |
|
|
80
|
+
| [AmazonScience/massive](https://huggingface.co/datasets/AmazonScience/massive) | CC BY 4.0 | Amazon Science | Short multilingual intents. Only the `utt` column is ingested; the slot-annotated `annot_utt` is never read. **Disabled by default** on capability grounds (not licence) — see `MASSIVE` in `example/train_base.ts` |
|
|
81
|
+
|
|
82
|
+
### 3.2 Excluded, and why
|
|
83
|
+
|
|
84
|
+
| Corpus | Reason |
|
|
85
|
+
| :---------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
86
|
+
| **MuskumPillerum/General-Knowledge** | **No licence at all.** The HF repo carries no licence tag and no licence in its card; an earlier header in `example/train_base.ts` claimed MIT without support. Its own card states it "contains a subset of the alpaca dataset", and Alpaca is CC BY-NC 4.0 — **NonCommercial**, incompatible with Sema's commercial licence. See §6. |
|
|
87
|
+
| **PAWS** | Google's own grant is maximally permissive ("may be freely used for any purpose"), but PAWS-Wiki sentences derive from Wikipedia (CC BY-**SA**) and PAWS-QQP from Quora question pairs under Quora's terms. Because Sema retains text verbatim, the upstream terms would attach to the distributed store. Excluded despite strong measured fit. |
|
|
88
|
+
| **Schema-Guided Dialogue (SGD/dstc8)**, **HotpotQA**, **MuSiQue** | CC BY-SA 4.0 — ShareAlike conflicts with dual distribution. |
|
|
89
|
+
| **2WikiMultihopQA passages** | Wikipedia prose, CC BY-SA. The repo's Apache-2.0 tag does not relicense the text it was built from. Only the Wikidata-derived `evidences` triples are ingested. |
|
|
90
|
+
| **Alpaca** and derivatives | CC BY-NC 4.0, and generated from OpenAI model outputs. |
|
|
91
|
+
|
|
92
|
+
---
|
|
93
|
+
|
|
94
|
+
### 3.3 What each stage actually deposits
|
|
95
|
+
|
|
96
|
+
A corpus's licence applies to what is ingested, and every stage ingests a strict
|
|
97
|
+
subset of its source. This is the authoritative list.
|
|
98
|
+
|
|
99
|
+
| Stage | Columns/fields read | Deposit shape |
|
|
100
|
+
| :----------------------- | :---------------------------------------- | :-------------------------------------------------------- |
|
|
101
|
+
| SmolSent | `src`, `trg` | one `src → trg` (foreign → English) episode per row |
|
|
102
|
+
| Aya | `inputs`, `targets` | one question → answer episode |
|
|
103
|
+
| oasst2 | message `text` along the best-ranked path | cumulative-context walk |
|
|
104
|
+
| Taskmaster | `utterances[].text` | cumulative-context walk over speaker-merged turns |
|
|
105
|
+
| 2Wiki | `evidences` | per triple: a relation fact and a bare-subject pivot fact |
|
|
106
|
+
| SODA | `dialogue`, `speakers` | cumulative-context walk over speaker-merged turns |
|
|
107
|
+
| MASSIVE (off by default) | `utt` | one bare experience |
|
|
108
|
+
|
|
109
|
+
Everything else in those sources — Taskmaster's `instructions`/`scenario`,
|
|
110
|
+
2Wiki's `context`/`question`/`answer`, SODA's `narrative`/`literal`/`head`/
|
|
111
|
+
`relation`/`tail`, MASSIVE's `annot_utt` — is **not read** and therefore not
|
|
112
|
+
distributed in a trained store.
|
|
113
|
+
|
|
114
|
+
---
|
|
115
|
+
|
|
116
|
+
## 4. The rule that decides these cases
|
|
117
|
+
|
|
118
|
+
**A repository's licence tag does not relicense the material the repository was
|
|
119
|
+
built from.** A dataset assembled out of Wikipedia prose and published under
|
|
120
|
+
Apache-2.0 still carries Wikipedia's ShareAlike terms on that prose. Because
|
|
121
|
+
Sema stores text verbatim, Sema inherits the _upstream_ terms, not the
|
|
122
|
+
repackager's.
|
|
123
|
+
|
|
124
|
+
So the check for any candidate corpus is two questions, not one:
|
|
125
|
+
|
|
126
|
+
1. What licence does the repository carry?
|
|
127
|
+
2. **What was it built from, and what licence does that carry?**
|
|
128
|
+
|
|
129
|
+
Where a corpus has a clean layer and a contaminated one, take the clean layer
|
|
130
|
+
only — as with 2Wiki's Wikidata triples (CC0) versus its Wikipedia passages (CC
|
|
131
|
+
BY-SA).
|
|
132
|
+
|
|
133
|
+
---
|
|
134
|
+
|
|
135
|
+
## 5. Disclosures
|
|
136
|
+
|
|
137
|
+
**Model-generated provenance.** `allenai/soda` is licensed CC BY 4.0 but was
|
|
138
|
+
distilled from OpenAI GPT-3.5 outputs. The licence is clean; the provenance is
|
|
139
|
+
disclosed here so downstream users can make their own assessment.
|
|
140
|
+
|
|
141
|
+
**Personal data.** `OpenAssistant/oasst2` is human-authored content contributed
|
|
142
|
+
by identifiable volunteers, and Sema retains it verbatim in a redistributable
|
|
143
|
+
artifact. Erasure requests against a content-addressed store are not
|
|
144
|
+
straightforward. Anyone distributing a Sema store trained on human-contributed
|
|
145
|
+
dialogue should account for this.
|
|
146
|
+
|
|
147
|
+
---
|
|
148
|
+
|
|
149
|
+
## 6. Status of previously published stores
|
|
150
|
+
|
|
151
|
+
Stores published before this file was written — including those under
|
|
152
|
+
[hviana/sema-trained-v1](https://huggingface.co/buckets/hviana/sema-trained-v1)
|
|
153
|
+
— were trained with the `MuskumPillerum/General-Knowledge` stage enabled (37,623
|
|
154
|
+
rows), whose licence status is described in §3.2. Those artifacts should be
|
|
155
|
+
treated as **not redistributable** until retrained without that stage.
|
|
156
|
+
|
|
157
|
+
The stage is now **disabled by default** in `example/train_base.ts`
|
|
158
|
+
(`GENKNOW=0`). The adapter code remains so the stage can be re-enabled for local
|
|
159
|
+
experiments; a store trained with `GENKNOW=1` must not be distributed.
|
package/HOW_IT_WORKS.md
CHANGED
|
@@ -512,6 +512,73 @@ similarity into _distributional_ similarity, silently shifting the halo null
|
|
|
512
512
|
model that the concept threshold's derivation (unrelated halos ⇒ cosine 0 ±
|
|
513
513
|
1/√D) depends on.
|
|
514
514
|
|
|
515
|
+
#### Company must be keyed on types, not tokens
|
|
516
|
+
|
|
517
|
+
A signature derived from the _whole_ partner's node id records a **token**:
|
|
518
|
+
"occurred next to node #4711992". But the distributional hypothesis is a claim
|
|
519
|
+
about **types**: "occurred next to a city name". On a content-addressed store of
|
|
520
|
+
natural language the difference is fatal, because a whole deposit essentially
|
|
521
|
+
never repeats: on a trained 15.7M-node store, 331,110 trained pairs produced
|
|
522
|
+
325,615 distinct contexts — 98.3% of them unique, so barely one span in sixty
|
|
523
|
+
ever recurs. Two genuine synonyms in different sentences then share no partner
|
|
524
|
+
id at all, and their halos come out quasi-orthogonal _by construction_: the best
|
|
525
|
+
distributional sibling of "Eiffel Tower" measured 0.146 against a concept
|
|
526
|
+
threshold of 0.516, with its own attested translations absent entirely. The
|
|
527
|
+
whole concept-hop, articulation and analogy layer was inert at corpus scale —
|
|
528
|
+
not for want of data, but because the key was too fine.
|
|
529
|
+
|
|
530
|
+
Sema therefore pours a **company profile**: the partner's own signature
|
|
531
|
+
superposed with the signatures of the partner's _constituents_. Two episodes
|
|
532
|
+
then share halo mass exactly when their partners are **made of** something in
|
|
533
|
+
common, which is what "keeps similar company" was always meant to mean.
|
|
534
|
+
|
|
535
|
+
Which constituents count is decided by four rules, each one load-bearing:
|
|
536
|
+
|
|
537
|
+
- **Every depth, not just the partner's children.** Cuts are content-defined
|
|
538
|
+
over a rolling window, so a chunk boundary depends on the bytes _around_ a
|
|
539
|
+
unit. "The Eiffel Tower is in Paris" folds to `The Eiffel` ·
|
|
540
|
+
`Tower is in
|
|
541
|
+
Paris`, and "Tour Eiffel dia any Paris" to `Tour Eiffel` ·
|
|
542
|
+
`dia any Paris` — the shared unit `Paris` is a child of neither. Reading only
|
|
543
|
+
the children merely moves the token problem from whole-partner identity down
|
|
544
|
+
to top-level-chunk identity, which for full sentences is nearly as rare. The
|
|
545
|
+
descent finds `Paris` and `ffel`; the top level finds nothing.
|
|
546
|
+
- **Never gate on recurrence-so-far.** The tempting rule — descend while a
|
|
547
|
+
constituent is corpus-unique, stop at the first unit attested twice — is
|
|
548
|
+
order-dependent: when the _first_ of a pair is deposited its shared unit has
|
|
549
|
+
been seen once, so the descent runs past it, and only the second partner ever
|
|
550
|
+
profiles it. Whether two synonyms become siblings must not depend on which was
|
|
551
|
+
taught first. The hub test below is the one reading that does consult corpus
|
|
552
|
+
state, and it can only ever remove a term — never decide which units are
|
|
553
|
+
found.
|
|
554
|
+
- **Nothing narrower than the fold's own window `W`.** Sub-window shards are
|
|
555
|
+
fragments of a unit, not units; admitting them lets mid-frequency byte
|
|
556
|
+
coincidences leak company across unrelated domains.
|
|
557
|
+
- **Minimal units only, and nothing that half-dominates the partner.** A
|
|
558
|
+
constituent that still contains a constituent of its own is a composite, and
|
|
559
|
+
superposing both counts the same content twice. This matters most for _nested_
|
|
560
|
+
partners — an accumulated conversation, where turn _k_'s context is a prefix
|
|
561
|
+
of turn _k+1_'s — which share their large chunks structurally rather than
|
|
562
|
+
distributionally.
|
|
563
|
+
|
|
564
|
+
Hubs (more than √N structural parents — Section 8.8) are excluded as
|
|
565
|
+
scaffolding, since a term shared by every deposit would put a common component
|
|
566
|
+
in every profile and collapse the null model the concept threshold depends on.
|
|
567
|
+
Byte atoms are excluded for the same reason: an atom's fan-in is the alphabet's.
|
|
568
|
+
|
|
569
|
+
The result stays **normalized**, so one episode still pours one unit of mass and
|
|
570
|
+
every mass-based reading is unchanged; and every term is still a seeded function
|
|
571
|
+
of a **node identity**, never a gist, so the hygiene rule above is untouched.
|
|
572
|
+
|
|
573
|
+
Similarity is **graded and size-relative**: two partners meet in proportion to
|
|
574
|
+
how much of what they are made of they share, so a shared unit inside a long
|
|
575
|
+
partner says less than the same unit inside a short one. This is the honest
|
|
576
|
+
reading of the evidence, not a defect — but it means the layer speaks clearly at
|
|
577
|
+
sentence scale and softly for very long partners. One further honest limit: the
|
|
578
|
+
hub bound is √N, so on a _small_ store almost nothing reads as scaffolding and
|
|
579
|
+
frame words do enter profiles. That is the correct floor for a corpus that
|
|
580
|
+
cannot yet say what discriminates, and it resolves itself as N grows.
|
|
581
|
+
|
|
515
582
|
Two nodes whose halos are similar have occurred in similar circumstances — they
|
|
516
583
|
are **distributional siblings**: synonyms, paraphrases, items of the same
|
|
517
584
|
category, two names for one thing. Note the complementarity:
|
|
@@ -532,6 +599,13 @@ embedding model:
|
|
|
532
599
|
rule).
|
|
533
600
|
- **Articulation.** An answer is re-voiced in the asker's own vocabulary by
|
|
534
601
|
substituting answer forms with query forms that share a halo (Section 24).
|
|
602
|
+
Halo similarity alone is not sufficient authority for this: the strongest
|
|
603
|
+
company any two forms can keep is standing next to each other, so a question
|
|
604
|
+
and the answer it is answered by are _maximally_ similar distributionally. Two
|
|
605
|
+
exact-side vetoes keep that from being read as synonymy — a stored
|
|
606
|
+
continuation edge between the two forms, and a candidate form spanning the
|
|
607
|
+
_whole_ answer (substituting that is replacement, not re-voicing). Without
|
|
608
|
+
them, "it hangs in madrid" is faithfully re-voiced as "where is it kept now".
|
|
535
609
|
- **Analogy strength.** Whether two entities are genuinely analogous — the gate
|
|
536
610
|
on counterfactual comparison (Section 18) — is measured by halo similarity,
|
|
537
611
|
directly or through shared siblings (a second-order distributional test).
|
package/README.md
CHANGED
|
@@ -320,6 +320,7 @@ start talking — no install, no runtime, no API key.
|
|
|
320
320
|
| 🛠️ **[AGENTS.md](AGENTS.md)** | The development manual: repo layout, build/test, internals, invariants, and recipes for extending the system. |
|
|
321
321
|
| 🎓 **[CITATION.cff](CITATION.cff)** | How to cite Sema in academic work. |
|
|
322
322
|
| ⚖️ **[LICENSE.md](LICENSE.md)** | PolyForm Noncommercial License 1.0.0. |
|
|
323
|
+
| 📚 **[DATASETS.md](DATASETS.md)** | Training corpora: provenance, per-corpus attribution, and how a trained memory file is licensed. |
|
|
323
324
|
| 💼 **[COMMERCIAL-LICENSE.md](COMMERCIAL-LICENSE.md)** | Commercial licensing terms and contact. |
|
|
324
325
|
| 🤗 **[Trained examples](https://huggingface.co/buckets/hviana/sema-trained-v1)** | Pre-trained memory files you can download and use directly. |
|
|
325
326
|
| 💿 **[Binary examples](https://huggingface.co/buckets/hviana/sema-binary-examples)** | Ready-to-run web chat apps for Windows, Mac, and Linux — one file, no install. |
|
|
@@ -350,6 +351,17 @@ start talking — no install, no runtime, no API key.
|
|
|
350
351
|
> **[TRADEMARKS.md](TRADEMARKS.md)** — the **Sema** name, logos, and brand are
|
|
351
352
|
> _not_ covered by the source license.
|
|
352
353
|
|
|
354
|
+
> [!NOTE]
|
|
355
|
+
> **Trained memory files are licensed in two layers.** Sema is non-parametric,
|
|
356
|
+
> so a trained store retains its training text verbatim — it is a database that
|
|
357
|
+
> contains its corpora, not a weight file. The **engine** (algorithms, geometry,
|
|
358
|
+
> deduction, indexes, formats, and all code) is under PolyForm Noncommercial
|
|
359
|
+
> with a separate commercial license. The **corpus content inside a store**
|
|
360
|
+
> stays under each corpus's own upstream license, and Sema's license is not
|
|
361
|
+
> extended over it. If you distribute a store, ship
|
|
362
|
+
> **[DATASETS.md](DATASETS.md)** with it — it carries the attribution and
|
|
363
|
+
> modification statement those licenses require.
|
|
364
|
+
|
|
353
365
|
<div align="center">
|
|
354
366
|
|
|
355
367
|
**Respecting these terms funds the research that makes work like this
|
|
@@ -16,9 +16,12 @@ export interface SmolSentRow {
|
|
|
16
16
|
/** Normalize a raw datasets-server row into a SmolSentRow, or null when it lacks
|
|
17
17
|
* both sides or a side is implausibly large (a dump, not a sentence). */
|
|
18
18
|
export declare function toSmolSentRow(row: unknown): SmolSentRow | null;
|
|
19
|
-
/** Translate ONE SmolSent pair into SEMA facts
|
|
20
|
-
* meaning in two languages,
|
|
21
|
-
*
|
|
19
|
+
/** Translate ONE SmolSent pair into SEMA facts. The two sentences are one
|
|
20
|
+
* meaning in two languages, but the two BINDINGS are not equally sound —
|
|
21
|
+
* SmolSent's English side is a shared pool translated into every language, so
|
|
22
|
+
* `trg -> src` gives one English context a different answer in every language
|
|
23
|
+
* file. See SMOLSENT_DIRECTIONS. refineItems drops the degenerate case where
|
|
24
|
+
* src === trg. */
|
|
22
25
|
export declare function smolSentRowToItems(row: SmolSentRow): TrainingItem[];
|
|
23
26
|
/** One normalized Aya row. */
|
|
24
27
|
export interface AyaRow {
|
|
@@ -79,6 +82,57 @@ export declare function bestOasstPath(root: OasstNode): OasstTurn[];
|
|
|
79
82
|
* Returns [] for a conversation below the multi-turn threshold, so callers can
|
|
80
83
|
* simply skip empties. */
|
|
81
84
|
export declare function oasstConversationToItems(turns: OasstTurn[]): TrainingItem[];
|
|
85
|
+
/** One utterance of a Taskmaster conversation. */
|
|
86
|
+
export interface TaskmasterTurn {
|
|
87
|
+
speaker: string;
|
|
88
|
+
text: string;
|
|
89
|
+
}
|
|
90
|
+
/** Normalize ONE element of a Taskmaster data file into its turns, or null when
|
|
91
|
+
* it carries no usable utterance. Empty/whitespace-only utterances are dropped
|
|
92
|
+
* (TM-3 has a few); a single implausibly long utterance rejects the whole
|
|
93
|
+
* conversation as corrupt rather than depositing a dump. */
|
|
94
|
+
export declare function toTaskmasterTurns(row: unknown): TaskmasterTurn[] | null;
|
|
95
|
+
/** Collapse consecutive same-speaker turns into one, joining with a space, and
|
|
96
|
+
* return the bare texts in order. A turn with no speaker never merges with its
|
|
97
|
+
* neighbour: an unlabelled row is of unknown origin, and joining two of them
|
|
98
|
+
* would invent a contribution that may span two speakers. */
|
|
99
|
+
export declare function mergeTaskmasterTurns(turns: TaskmasterTurn[]): string[];
|
|
100
|
+
/** Translate ONE Taskmaster conversation into SEMA training items: the
|
|
101
|
+
* cumulative walk over its merged turns. Returns [] for a conversation below
|
|
102
|
+
* TASKMASTER_MIN_TURNS, so callers can simply skip empties. */
|
|
103
|
+
export declare function taskmasterConversationToItems(turns: TaskmasterTurn[]): TrainingItem[];
|
|
104
|
+
/** One (subject, relation, object) triple from a 2Wiki `evidences` cell. */
|
|
105
|
+
export interface WikiTriple {
|
|
106
|
+
subject: string;
|
|
107
|
+
relation: string;
|
|
108
|
+
object: string;
|
|
109
|
+
}
|
|
110
|
+
/** Normalize a 2Wiki row into its evidence triples, or null when it carries
|
|
111
|
+
* none usable. `evidences` is a JSON STRING holding an array of 3-element
|
|
112
|
+
* arrays; a row whose cell is absent, unparseable, or empty yields null.
|
|
113
|
+
* Individual malformed or oversized triples are dropped without discarding the
|
|
114
|
+
* row — one bad triple should not cost the others. */
|
|
115
|
+
export declare function toWikiTriples(row: unknown): WikiTriple[] | null;
|
|
116
|
+
/** Render ONE triple as the prose fact Sema stores. Kept separate so the two
|
|
117
|
+
* deposits below are guaranteed to share a byte-identical continuation: the
|
|
118
|
+
* pivot fact only works if it leads to the SAME node the relation fact does. */
|
|
119
|
+
export declare function wikiTripleSentence(t: WikiTriple): string;
|
|
120
|
+
/** Translate a row's triples into SEMA items: per triple, the relation fact and
|
|
121
|
+
* the bare-subject PIVOT fact (see the section note above). refineItems drops
|
|
122
|
+
* the duplicates this produces when a row states the same triple twice. */
|
|
123
|
+
export declare function wikiTriplesToItems(triples: WikiTriple[]): TrainingItem[];
|
|
124
|
+
/** Normalize a SODA row into its turns, or null when it carries no usable
|
|
125
|
+
* dialogue. Speakers are optional (they only drive merging); an implausibly
|
|
126
|
+
* long turn rejects the dialogue as corrupt. */
|
|
127
|
+
export declare function toSodaTurns(row: unknown): TaskmasterTurn[] | null;
|
|
128
|
+
/** Translate ONE SODA dialogue into SEMA items: the cumulative walk over its
|
|
129
|
+
* speaker-merged turns. Shares `mergeTaskmasterTurns` because the rule is the
|
|
130
|
+
* same one — consecutive turns by one speaker are one contribution. */
|
|
131
|
+
export declare function sodaDialogueToItems(turns: TaskmasterTurn[]): TrainingItem[];
|
|
132
|
+
/** Translate ONE MASSIVE row into SEMA items: its bare utterance, as an
|
|
133
|
+
* experience. `annot_utt` (slot-annotated) is deliberately not used — its
|
|
134
|
+
* "[date : ...]" markup is not prose. Returns [] for an unusable row. */
|
|
135
|
+
export declare function massiveRowToItems(row: unknown): TrainingItem[];
|
|
82
136
|
/** One normalized General-Knowledge row. */
|
|
83
137
|
export interface GenKnowRow {
|
|
84
138
|
question: string;
|
|
@@ -90,4 +144,20 @@ export declare function toGenKnowRow(row: unknown): GenKnowRow | null;
|
|
|
90
144
|
/** Translate ONE General-Knowledge row into SEMA items: exactly one
|
|
91
145
|
* (question → answer) FACT. refineItems drops a degenerate question === answer. */
|
|
92
146
|
export declare function genKnowRowToItems(row: GenKnowRow): TrainingItem[];
|
|
147
|
+
/** How many rows to materialise in one read from a row-group of `rgRows` rows
|
|
148
|
+
* occupying `groupBytes` uncompressed bytes, under a `budgetBytes` target.
|
|
149
|
+
*
|
|
150
|
+
* The group's own footer statistics give the mean row width, so the batch
|
|
151
|
+
* follows the CORPUS's row size rather than the writer's layout: wide rows
|
|
152
|
+
* (SODA carries a whole dialogue per row) batch smaller than narrow ones at
|
|
153
|
+
* the same memory cost. Never exceeds the group — a batch is a subdivision of
|
|
154
|
+
* a group, never a span across two, because `parquetReadObjects` is given an
|
|
155
|
+
* absolute row range and column chunks are per-group. Never returns 0, or the
|
|
156
|
+
* read loop could not advance.
|
|
157
|
+
*
|
|
158
|
+
* A writer that omits `total_byte_size` yields `groupBytes <= 0`; the batch is
|
|
159
|
+
* then the whole group, which is exactly the behaviour this replaced. That
|
|
160
|
+
* fallback is safe for every file we read today (all three report it) and
|
|
161
|
+
* degrades to the old memory profile rather than to a wrong result. */
|
|
162
|
+
export declare function parquetBatchRows(rgRows: number, groupBytes: number, budgetBytes: number): number;
|
|
93
163
|
export {};
|