decidekit 0.1.1 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +93 -77
- package/README.md +14 -6
- package/dist/browser.js +14 -3
- package/dist/head.js +45 -1
- package/dist/index.d.ts +9 -4
- package/dist/index.js +13 -2
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,77 +1,93 @@
|
|
|
1
|
-
# Changelog
|
|
2
|
-
|
|
3
|
-
## 0.
|
|
4
|
-
|
|
5
|
-
###
|
|
6
|
-
|
|
7
|
-
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
The
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
-
|
|
37
|
-
|
|
38
|
-
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
-
|
|
47
|
-
|
|
48
|
-
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
-
|
|
55
|
-
|
|
56
|
-
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
- `
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
-
|
|
76
|
-
|
|
77
|
-
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.2.1 (2026-11-02)
|
|
4
|
+
|
|
5
|
+
### Browser
|
|
6
|
+
|
|
7
|
+
- `createWorkerDecider()` rejects when the worker fails to load, crashes or sends an unreadable message. Before, the call never settled. Calls after a failure or after `terminate()` reject too.
|
|
8
|
+
- The wasm download is 12.9 MB (3.3 MB gzipped) unless you pass `device: "webgpu"`. Earlier docs said an alias got this size, but in Chrome with an esbuild build, 0.2.0 fetched the 23.6 MB file with or without it. The alias is now optional and only trims about 14.5 KB gzipped of worker JS. Paths you set in `wasmPaths` are left alone.
|
|
9
|
+
|
|
10
|
+
## 0.2.0 (2026-11-02)
|
|
11
|
+
|
|
12
|
+
### Training
|
|
13
|
+
|
|
14
|
+
- From 8 examples per label (mean across labels), `train()` fits the head with full-batch Adam, starting from the same centroid and label-name point. Test accuracy with all data: BANKING77 90.6% to 93.1%, CLINC150 94.6% to 96.1%, HWU64 90.15% to 91.17% (one seed, on cached embeddings; through train() in eval/results-h14.json, BANKING77 93.1% and CLINC150 96.3%). At 5 per label nothing changes. At 10 per label: BANKING77 84.4% to 87.5%, CLINC150 92.2% to 92.5%, HWU64 85.50% to 85.38%. Adam's settings come from jimothy's `src/linear.ts` (MIT, Andrew Prifer).
|
|
15
|
+
- New option `solver`. Setting `learningRate` or `l2` without it keeps SGD, so tuned calls behave as before. The default `epochs` is 100 with Adam.
|
|
16
|
+
- Saved heads keep working unchanged. Retrained heads can score differently, so pick `minConfidence` thresholds again: the CLINC150 out-of-scope threshold moved from 0.9 to 0.98.
|
|
17
|
+
- Training takes longer: in Node, BANKING77 went from 84 s to 123 s on cached embeddings. In Chrome, BANKING77 took 208 s with a longest page stall of 19 ms.
|
|
18
|
+
|
|
19
|
+
## 0.1.1 (2026-11-02)
|
|
20
|
+
|
|
21
|
+
### Models
|
|
22
|
+
|
|
23
|
+
- The default model id is `debuffjp/decide-small` (Decide Small). Before 0.1.1 it was `Xenova/bge-small-en-v1.5`. The
|
|
24
|
+
weights are identical, so decisions don't change. For other languages the id is `debuffjp/decide-multilingual`
|
|
25
|
+
(Decide Multilingual), identical to `Xenova/multilingual-e5-small`.
|
|
26
|
+
- The old ids and heads recording them keep working. They resolve to the new ids, so `d.model` returns the new id. A
|
|
27
|
+
head trained on 0.1.1 records the new id, so a 0.1.0 decider needs that id passed as `model` to use it.
|
|
28
|
+
- Downloads of both models are pinned to a revision.
|
|
29
|
+
- New option `modelHost` serves model files from `<modelHost>/<model>/` instead of Hugging Face. It applies to the
|
|
30
|
+
whole page or process.
|
|
31
|
+
|
|
32
|
+
### Docs
|
|
33
|
+
|
|
34
|
+
- Label names count as 1 example in every doc, and the advice to turn them off with bge-small is gone: it applied to the
|
|
35
|
+
old weight of 3.
|
|
36
|
+
- Calibration, out-of-scope and zero-shot numbers are now bge-small's or labelled with the model they were measured on.
|
|
37
|
+
- `decide()` documents that a described label scores by the better of its name and description.
|
|
38
|
+
- The API reference covers `decidekit/typesafe`.
|
|
39
|
+
|
|
40
|
+
## 0.1.0 (2026-11-02)
|
|
41
|
+
|
|
42
|
+
The first release. Numbers are on public test splits unless noted; the scripts that produced them are in `eval/`.
|
|
43
|
+
|
|
44
|
+
### Deciding
|
|
45
|
+
|
|
46
|
+
- `createDecider()` picks one of your labels for a text, in Node or the browser, with `confidence` and `probabilities`. Labels
|
|
47
|
+
are the type: `choice` has the literal type of the labels you pass.
|
|
48
|
+
- Zero-shot from label names, or from described labels (`{ card_arrival: "My card hasn't arrived." }`), which score by
|
|
49
|
+
the better of name and description. Zero-shot decisions report `similarity`, the best cosine similarity to any
|
|
50
|
+
option, which is low for off-topic text.
|
|
51
|
+
- `minConfidence` returns `null` below a threshold. `decideMany()` for batches.
|
|
52
|
+
- Default encoder: `Xenova/bge-small-en-v1.5` at 8-bit (34 MB). Any transformers.js feature-extraction model can be
|
|
53
|
+
passed as `model`.
|
|
54
|
+
- Supported encoders with measured settings: `Xenova/bge-small-en-v1.5` (default) and `Xenova/multilingual-e5-small`.
|
|
55
|
+
The temperature fit now searches 0.01 to 10, so encoders with close similarities calibrate.
|
|
56
|
+
- `describeLabels()` asks an LLM once to write label descriptions (7 to 10 points better zero-shot on the dev splits).
|
|
57
|
+
|
|
58
|
+
### Training
|
|
59
|
+
|
|
60
|
+
- `train()` fits one softmax layer on frozen embeddings: 90.7% on BANKING77 and 94.4% on CLINC150 with all examples,
|
|
61
|
+
82.3% and 89.6% with 5 per label. HWU64, never used for tuning: 90.2%.
|
|
62
|
+
- Calibrated: the temperature is fitted on held-out examples (5 folds for small sets), with a per-model floor for
|
|
63
|
+
small heads. Calibration error 0.01 to 0.03 for heads with 10 or more examples per label.
|
|
64
|
+
- `labels` adds labels without examples yet (they start from their names). `calibrateOn` fits the temperature on
|
|
65
|
+
examples known to be right, for training data from a source that is sometimes wrong.
|
|
66
|
+
- `saveHead()` and `loadHead()`. Ready-made heads for BANKING77 and CLINC150.
|
|
67
|
+
|
|
68
|
+
### Learning from an LLM
|
|
69
|
+
|
|
70
|
+
- `withFallback()` answers locally when confident and calls your `ask` otherwise; each answer becomes a training
|
|
71
|
+
example and `retrainEvery` retrains in the background. It can start from labels alone.
|
|
72
|
+
- Built for an LLM that is sometimes wrong: starting from labels, it fits confidence on zero-shot's own confident
|
|
73
|
+
answers rather than the LLM's, and learns `null` answers as the label `NONE` (`learnNone`) when the text is far
|
|
74
|
+
from every label (`noneBelow`), so it can answer `null` itself for off-topic text.
|
|
75
|
+
- `route.saved()` returns the learned answers and the calibration set as plain JSON; pass them back as `restore` after
|
|
76
|
+
a reload. With real Llama 3.1 8B answers on
|
|
77
|
+
12 labels, the share sent fell from 30% (zero-shot) to 19%, and off-topic text given a wrong label from 6.7% to 2.4%.
|
|
78
|
+
|
|
79
|
+
### Browser and tools
|
|
80
|
+
|
|
81
|
+
- `decidekit/typesafe`: `typesafeClient()` gives a decider the shape of TypeSafe's SDK (`systemOne({ state, questions })`,
|
|
82
|
+
`choice()`, `probabilities`, their `confidence` formula). Choice questions only.
|
|
83
|
+
- `decidekit/browser`: `createWorkerDecider()` runs the encoder in a Web Worker (training BANKING77 froze the page for
|
|
84
|
+
171 s on the main thread; the longest stall in a worker was 18 ms).
|
|
85
|
+
- `decidekit/react`: `useDecider()` and `useDecision()`.
|
|
86
|
+
- `decidekit/switch`: a decider for the switch agent harness.
|
|
87
|
+
- A CLI: `decidekit train` and `decidekit decide`.
|
|
88
|
+
|
|
89
|
+
### Known limits
|
|
90
|
+
|
|
91
|
+
- The default encoder reads English. `Xenova/multilingual-e5-small` is supported for other languages (76% on Japanese
|
|
92
|
+
and Chinese MASSIVE intents, calibrated), but `withFallback`'s off-topic gate doesn't fire with it.
|
|
93
|
+
- Weak at tone: 61% on tweet sentiment and 49% on emotion with 32 examples per label. See the FAQ.
|
package/README.md
CHANGED
|
@@ -16,8 +16,8 @@ const r = await d.decide("my card never arrived", ["card_arrival", "refund", "ot
|
|
|
16
16
|
r.choice; // "card_arrival" | "refund" | "other"
|
|
17
17
|
```
|
|
18
18
|
|
|
19
|
-
Give it 5 examples per label and it reaches
|
|
20
|
-
reaches
|
|
19
|
+
Give it 5 examples per label and it reaches 84% on BANKING77 (77 intents); give it the full training set and it
|
|
20
|
+
reaches 93.1%, with each answer taking about 8 ms (median) in Chrome. Training runs in the user's tab, so their data never
|
|
21
21
|
leaves the device.
|
|
22
22
|
|
|
23
23
|
## Zero-shot
|
|
@@ -134,8 +134,7 @@ const r = await d.decide(text, head);
|
|
|
134
134
|
|
|
135
135
|
Import `loadHead` from `decidekit/browser` too, not from `decidekit`: that keeps the encoder out of your page bundle.
|
|
136
136
|
|
|
137
|
-
Tested from npm in Vite 7, webpack 5 and Next.js 16 apps; see [docs/bundlers.md](docs/bundlers.md) for the config
|
|
138
|
-
that halves the wasm download. Serve the page cross-origin isolated if you can:
|
|
137
|
+
Tested from npm in Vite 7, webpack 5 and Next.js 16 apps; see [docs/bundlers.md](docs/bundlers.md) for the config. Serve the page cross-origin isolated if you can:
|
|
139
138
|
|
|
140
139
|
```
|
|
141
140
|
Cross-Origin-Opener-Policy: same-origin
|
|
@@ -144,7 +143,7 @@ Cross-Origin-Embedder-Policy: credentialless
|
|
|
144
143
|
|
|
145
144
|
That lets wasm use several threads: in our runs with all-MiniLM-L6-v2 it cut p95 decision time from 14.3 ms to 7.8 ms. With the default bge-small, isolated, p95 is 13.8 ms.
|
|
146
145
|
|
|
147
|
-
Download on first visit: the model (34 MB) and onnxruntime's wasm (
|
|
146
|
+
Download on first visit: the model (34 MB) and onnxruntime's wasm (12.9 MB, 3.3 MB gzipped; 23.6 MB before 0.2.1), both cached after.
|
|
148
147
|
|
|
149
148
|
## Models
|
|
150
149
|
|
|
@@ -167,6 +166,15 @@ Decide Small (bge-small-en-v1.5, the default) at q8, without label names, Ryzen
|
|
|
167
166
|
and calibration error (ECE, lower is better) on each dataset's test split, mean of 3 seeds (one run for "all"):
|
|
168
167
|
[eval/results-bge-small.json](eval/results-bge-small.json).
|
|
169
168
|
|
|
169
|
+
| Examples per label | 5 | 10 | 25 | all |
|
|
170
|
+
|---|---|---|---|---|
|
|
171
|
+
| BANKING77 (77 labels) | 83.7% | 87.5% | 90.2% | 93.1% |
|
|
172
|
+
| CLINC150 (150 labels) | 90.2% | 92.8% | 94.3% | 96.3% |
|
|
173
|
+
| ECE, BANKING77 | 0.050 | 0.015 | 0.013 | 0.013 |
|
|
174
|
+
| ECE, CLINC150 | 0.020 | 0.015 | 0.016 | 0.008 |
|
|
175
|
+
|
|
176
|
+
Those are 0.2, with label names, from [eval/results-h14.json](eval/results-h14.json). HWU64 with all data goes from 90.15% to 91.17%. Before 0.2 (SGD at every size, no label names, `eval/results-bge-small.json`):
|
|
177
|
+
|
|
170
178
|
| Examples per label | 5 | 10 | 25 | all |
|
|
171
179
|
|---|---|---|---|---|
|
|
172
180
|
| BANKING77 (77 labels) | 82.3% | 83.8% | 88.7% | 90.7% |
|
|
@@ -189,7 +197,7 @@ queries, [eval/tiny.ts](eval/tiny.ts), measured with all-MiniLM-L6-v2), 1 exampl
|
|
|
189
197
|
a threshold there.
|
|
190
198
|
|
|
191
199
|
**In Chrome** (BANKING77, full training set, worker, cross-origin isolated, 16 threads): 7.7 ms median and 13.8 ms
|
|
192
|
-
p95 per decision, 228 s to train on 10,003 examples with no page stall longer than 18 ms. all-MiniLM-L6-v2 in the
|
|
200
|
+
p95 per decision before 0.2, 228 s to train on 10,003 examples with no page stall longer than 18 ms. A 0.2 run on the same machine took 208 s, with a longest stall of 19 ms and 93.05% test accuracy (runs on this laptop vary: another 0.1 run took 147 s). all-MiniLM-L6-v2 in the
|
|
193
201
|
same run: 4.2 ms median, 7.7 ms p95. In Node: 3.2 ms median for a single text, 1.1 ms per text batched (MiniLM 2.0
|
|
194
202
|
and 0.7). Earlier MiniLM run: [bench/results.json](bench/results.json).
|
|
195
203
|
|
package/dist/browser.js
CHANGED
|
@@ -18,7 +18,20 @@ export async function createWorkerDecider({ onProgress, ...options } = {}, worke
|
|
|
18
18
|
pending.delete(data.id);
|
|
19
19
|
data.error ? p.reject(new Error(data.error)) : p.resolve(data.value);
|
|
20
20
|
};
|
|
21
|
+
// After this the worker never answers, so every call, now or later, has to reject.
|
|
22
|
+
let dead;
|
|
23
|
+
const fail = (message) => {
|
|
24
|
+
dead = new Error(message);
|
|
25
|
+
for (const p of pending.values())
|
|
26
|
+
p.reject(dead);
|
|
27
|
+
pending.clear();
|
|
28
|
+
};
|
|
29
|
+
// A failed load gives an empty message, and a wrong worker URL from the bundler is its usual cause.
|
|
30
|
+
worker.onerror = (e) => fail(`decidekit: the worker failed: ${e.message || "it did not load, check the worker URL your bundler produced"}`);
|
|
31
|
+
worker.onmessageerror = () => fail("decidekit: the worker failed: it sent a message the page can't read");
|
|
21
32
|
const call = (op, payload, progress) => new Promise((resolve, reject) => {
|
|
33
|
+
if (dead)
|
|
34
|
+
return reject(dead);
|
|
22
35
|
const id = next++;
|
|
23
36
|
pending.set(id, { resolve, reject, progress });
|
|
24
37
|
worker.postMessage({ id, op, ...payload });
|
|
@@ -35,9 +48,7 @@ export async function createWorkerDecider({ onProgress, ...options } = {}, worke
|
|
|
35
48
|
train: (examples, { onEpoch, ...opts } = {}) => call("train", { args: [examples, opts] }, (p) => onEpoch?.(p.epoch, p.accuracy)),
|
|
36
49
|
terminate: () => {
|
|
37
50
|
worker.terminate();
|
|
38
|
-
|
|
39
|
-
p.reject(new Error("decidekit: the worker was terminated"));
|
|
40
|
-
pending.clear();
|
|
51
|
+
fail("decidekit: the worker was terminated");
|
|
41
52
|
},
|
|
42
53
|
};
|
|
43
54
|
}
|
package/dist/head.js
CHANGED
|
@@ -1,5 +1,10 @@
|
|
|
1
1
|
const BATCH = 64;
|
|
2
2
|
const CENTROID_SCALE = 20;
|
|
3
|
+
// At 5 per label Adam lost 0.29 points on CLINC150, at 10 it gained on BANKING77 and CLINC150 (eval/spike). The cutoff
|
|
4
|
+
// between them wasn't measured on its own.
|
|
5
|
+
const ADAM_FROM = 8;
|
|
6
|
+
// Without a better holdout loss for this many epochs, stop the probe run.
|
|
7
|
+
const ADAM_PATIENCE = 30;
|
|
3
8
|
// Lets a tab paint and handle input between training batches. A message-channel task, unlike setTimeout, isn't
|
|
4
9
|
// throttled to once a second (or minute) when the tab is in the background.
|
|
5
10
|
function yieldToPage() {
|
|
@@ -56,11 +61,23 @@ function batchGradient(ids, X, y, W, B, gW, gb, z, C, dim) {
|
|
|
56
61
|
}
|
|
57
62
|
}
|
|
58
63
|
}
|
|
64
|
+
// One Adam step from the summed gradient. Adam's settings follow jimothy's src/linear.ts (MIT, Andrew Prifer).
|
|
65
|
+
function adamStep(p, g, m, v, rows, l2, lr, epoch) {
|
|
66
|
+
const c1 = 1 - 0.9 ** epoch, c2 = 1 - 0.999 ** epoch;
|
|
67
|
+
for (let k = 0; k < p.length; k++) {
|
|
68
|
+
const d = g[k] / rows + l2 * p[k];
|
|
69
|
+
m[k] = 0.9 * m[k] + 0.1 * d;
|
|
70
|
+
v[k] = 0.999 * v[k] + 0.001 * d * d;
|
|
71
|
+
p[k] -= (lr * m[k]) / c1 / (Math.sqrt(v[k] / c2) + 1e-8);
|
|
72
|
+
}
|
|
73
|
+
}
|
|
59
74
|
/** Fits weights, biases and temperature to embeddings `X` with label indices `y`. */
|
|
60
75
|
export async function fitHead(X, y, labels, opts = {}, priors, priorWeight = 1, centroidScale = CENTROID_SCALE, smallFloor = 1, calibration) {
|
|
61
76
|
// smallFloor: the lowest temperature a head with under 50 held-out answers may get; createDecider picks it per model.
|
|
62
77
|
// calibration: labelled examples known to be right to fit the temperature on instead of the training examples.
|
|
63
|
-
|
|
78
|
+
// A set learningRate or l2 was tuned for SGD, so it keeps SGD unless the solver is named.
|
|
79
|
+
const adam = opts.solver ? opts.solver === "adam" : opts.learningRate === undefined && opts.l2 === undefined && X.length / labels.length >= ADAM_FROM;
|
|
80
|
+
const { epochs = adam ? 100 : 60, learningRate = adam ? 0.05 : 2, l2 = adam ? 0 : 1e-4, holdout = 0.1, seed = 7, onEpoch } = opts;
|
|
64
81
|
const dim = X[0].length, C = labels.length;
|
|
65
82
|
// One flat buffer, so the inner loops read typed memory instead of an array of arrays.
|
|
66
83
|
const Xf = new Float64Array(X.length * dim);
|
|
@@ -92,6 +109,33 @@ export async function fitHead(X, y, labels, opts = {}, priors, priorWeight = 1,
|
|
|
92
109
|
let best = { acc: check ? acc() : 0, weights: head.weights.slice(), bias: head.bias.slice(), epochs: 0 };
|
|
93
110
|
const gW = new Float64Array(C * dim), gb = new Float64Array(C), z = new Float64Array(C);
|
|
94
111
|
const W = head.weights, B = head.bias;
|
|
112
|
+
if (adam) {
|
|
113
|
+
const mW = new Float64Array(C * dim), vW = new Float64Array(C * dim), mB = new Float64Array(C), vB = new Float64Array(C);
|
|
114
|
+
let bestLoss = Infinity;
|
|
115
|
+
for (let ep = 0; ep < nEpochs; ep++) {
|
|
116
|
+
gW.fill(0);
|
|
117
|
+
gb.fill(0);
|
|
118
|
+
for (let b = 0; b < rows.length; b += BATCH) {
|
|
119
|
+
batchGradient(rows.slice(b, b + BATCH), Xf, y, W, B, gW, gb, z, C, dim);
|
|
120
|
+
await yieldToPage();
|
|
121
|
+
}
|
|
122
|
+
adamStep(W, gW, mW, vW, rows.length, l2, learningRate, ep + 1);
|
|
123
|
+
adamStep(B, gb, mB, vB, rows.length, 0, learningRate, ep + 1);
|
|
124
|
+
if (!check)
|
|
125
|
+
continue;
|
|
126
|
+
const loss = check.reduce((sum, i) => sum - Math.log(softmax(logits(head, X[i]), 1)[y[i]] + 1e-300), 0);
|
|
127
|
+
if (onEpoch)
|
|
128
|
+
onEpoch(ep, acc());
|
|
129
|
+
if (loss < bestLoss * (1 - 1e-7)) {
|
|
130
|
+
bestLoss = loss;
|
|
131
|
+
best = { acc: 0, weights: W.slice(), bias: B.slice(), epochs: ep + 1 };
|
|
132
|
+
}
|
|
133
|
+
if (ep + 1 - best.epochs >= ADAM_PATIENCE)
|
|
134
|
+
break;
|
|
135
|
+
await yieldToPage();
|
|
136
|
+
}
|
|
137
|
+
return check ? { ...head, weights: best.weights, bias: best.bias, epochs: best.epochs } : { ...head, epochs: nEpochs };
|
|
138
|
+
}
|
|
95
139
|
const order = [...rows];
|
|
96
140
|
for (let ep = 0; ep < nEpochs; ep++) {
|
|
97
141
|
order.sort(() => rand() - 0.5);
|
package/dist/index.d.ts
CHANGED
|
@@ -51,11 +51,16 @@ export interface TrainOptions<L extends string = string> {
|
|
|
51
51
|
* can still be chosen. Needs `labelNames`.
|
|
52
52
|
*/
|
|
53
53
|
labels?: readonly L[];
|
|
54
|
-
/**
|
|
54
|
+
/**
|
|
55
|
+
* How to fit the head. By default `adam` from 8 examples per label (mean across labels), `sgd` below that. Setting
|
|
56
|
+
* `learningRate` or `l2` without this also picks `sgd`.
|
|
57
|
+
*/
|
|
58
|
+
solver?: "adam" | "sgd";
|
|
59
|
+
/** Most passes over the examples. Default 100 with `adam`, 60 with `sgd`; the best holdout epoch is kept. */
|
|
55
60
|
epochs?: number;
|
|
56
|
-
/** Default
|
|
61
|
+
/** Default 0.05 with `adam`, 2 with `sgd`. */
|
|
57
62
|
learningRate?: number;
|
|
58
|
-
/** Weight decay. Default 1e-4
|
|
63
|
+
/** Weight decay. Default 0 with `adam`, 1e-4 with `sgd`. */
|
|
59
64
|
l2?: number;
|
|
60
65
|
/** Share of examples held out for early stopping and temperature fitting. Default 0.1. */
|
|
61
66
|
holdout?: number;
|
|
@@ -74,7 +79,7 @@ export interface DeciderOptions {
|
|
|
74
79
|
/**
|
|
75
80
|
* A Hugging Face sentence-embedding model with ONNX weights. Default `debuffjp/decide-small` (Decide Small, 34 MB at q8).
|
|
76
81
|
* Before 0.1.1 the default id was `Xenova/bge-small-en-v1.5`, the same weights; that id and heads recording it still work.
|
|
77
|
-
* Pre-release builds defaulted to `Xenova/all-MiniLM-L6-v2` (23 MB): smaller and 1.
|
|
82
|
+
* Pre-release builds defaulted to `Xenova/all-MiniLM-L6-v2` (23.7 MB): smaller and about 1.8x faster in Chrome, 3 to 5 points less
|
|
78
83
|
* accurate. Pass it to keep using heads trained on it.
|
|
79
84
|
*/
|
|
80
85
|
model?: string;
|
package/dist/index.js
CHANGED
|
@@ -40,6 +40,15 @@ function toDecision(labels, p) {
|
|
|
40
40
|
const probabilities = Object.fromEntries(labels.map((l, i) => [l, p[i]]));
|
|
41
41
|
return { choice: labels[best], confidence: p[best], probabilities };
|
|
42
42
|
}
|
|
43
|
+
function usePlainWasm() {
|
|
44
|
+
// transformers.js points browsers other than Safari at the 23.6 MB WebGPU-capable build whatever JS is bundled.
|
|
45
|
+
// Leaves paths the app set itself alone.
|
|
46
|
+
const paths = env.backends.onnx.wasm?.wasmPaths;
|
|
47
|
+
if (!paths || typeof paths === "string" || !paths.wasm?.startsWith("https://cdn.jsdelivr.net/npm/onnxruntime-web@"))
|
|
48
|
+
return;
|
|
49
|
+
paths.wasm = paths.wasm.replace(".asyncify.", ".");
|
|
50
|
+
paths.mjs = paths.mjs?.replace(".asyncify.", ".");
|
|
51
|
+
}
|
|
43
52
|
/**
|
|
44
53
|
* Loads the encoder (downloaded once, then cached by the browser or transformers.js) and returns `decide()` and
|
|
45
54
|
* `train()`. In a browser, prefer `createWorkerDecider()` from `decidekit/browser`: the encoder blocks the thread
|
|
@@ -51,6 +60,8 @@ export async function createDecider({ model: requested = SMALL, device, dtype =
|
|
|
51
60
|
env.remoteHost = modelHost.endsWith("/") ? modelHost : modelHost + "/";
|
|
52
61
|
env.remotePathTemplate = "{model}/";
|
|
53
62
|
}
|
|
63
|
+
if (device !== "webgpu")
|
|
64
|
+
usePlainWasm();
|
|
54
65
|
const embed = await pipeline("feature-extraction", model, {
|
|
55
66
|
device, dtype, revision: REVISIONS[model],
|
|
56
67
|
progress_callback: onProgress && ((p) => p.status === "progress" && onProgress({ file: p.file, loaded: p.loaded, total: p.total })),
|
|
@@ -103,8 +114,8 @@ export async function createDecider({ model: requested = SMALL, device, dtype =
|
|
|
103
114
|
/**
|
|
104
115
|
* Trains a head on labelled examples: mini-batch gradient descent on the embeddings, keeping the best holdout
|
|
105
116
|
* epoch, then a temperature fitted on the holdout so `confidence` is calibrated. Every label needs at least a few
|
|
106
|
-
* examples. Each label starts at the mean of its examples and its name, so 5 per label already gave
|
|
107
|
-
*
|
|
117
|
+
* examples. Each label starts at the mean of its examples and its name, so 5 per label already gave 82.3% on BANKING77 and
|
|
118
|
+
* 89.6% on CLINC150 (see the README).
|
|
108
119
|
*/
|
|
109
120
|
async function train(examples, opts = {}) {
|
|
110
121
|
const bad = examples.findIndex((e) => typeof e?.text !== "string" || typeof e?.label !== "string" || !e.label);
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "decidekit",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.2.1",
|
|
4
4
|
"description": "A System One model in the browser: typed, calibrated text decisions on the user's device. Learns from an LLM when it isn't sure.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"repository": "github:debuffjp/decidekit",
|