decidekit 0.0.1 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +22 -1
- package/README.md +32 -21
- package/dist/browser.js +1 -1
- package/dist/fallback.js +2 -2
- package/dist/index.d.ts +8 -2
- package/dist/index.js +26 -15
- package/dist/switch.d.ts +4 -4
- package/dist/switch.js +7 -7
- package/dist/typesafe.d.ts +1 -1
- package/dist/typesafe.js +3 -3
- package/package.json +93 -93
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,26 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.1.1 (2026-11-02)
|
|
4
|
+
|
|
5
|
+
### Models
|
|
6
|
+
|
|
7
|
+
- The default model id is `debuffjp/decide-small` (Decide Small). Before 0.1.1 it was `Xenova/bge-small-en-v1.5`. The
|
|
8
|
+
weights are identical, so decisions don't change. For other languages the id is `debuffjp/decide-multilingual`
|
|
9
|
+
(Decide Multilingual), identical to `Xenova/multilingual-e5-small`.
|
|
10
|
+
- The old ids and heads recording them keep working. They resolve to the new ids, so `d.model` returns the new id. A
|
|
11
|
+
head trained on 0.1.1 records the new id, so a 0.1.0 decider needs that id passed as `model` to use it.
|
|
12
|
+
- Downloads of both models are pinned to a revision.
|
|
13
|
+
- New option `modelHost` serves model files from `<modelHost>/<model>/` instead of Hugging Face. It applies to the
|
|
14
|
+
whole page or process.
|
|
15
|
+
|
|
16
|
+
### Docs
|
|
17
|
+
|
|
18
|
+
- Label names count as 1 example in every doc, and the advice to turn them off with bge-small is gone: it applied to the
|
|
19
|
+
old weight of 3.
|
|
20
|
+
- Calibration, out-of-scope and zero-shot numbers are now bge-small's or labelled with the model they were measured on.
|
|
21
|
+
- `decide()` documents that a described label scores by the better of its name and description.
|
|
22
|
+
- The API reference covers `decidekit/typesafe`.
|
|
23
|
+
|
|
3
24
|
## 0.1.0 (2026-11-02)
|
|
4
25
|
|
|
5
26
|
The first release. Numbers are on public test splits unless noted; the scripts that produced them are in `eval/`.
|
|
@@ -47,7 +68,7 @@ The first release. Numbers are on public test splits unless noted; the scripts t
|
|
|
47
68
|
171 s on the main thread; the longest stall in a worker was 18 ms).
|
|
48
69
|
- `decidekit/react`: `useDecider()` and `useDecision()`.
|
|
49
70
|
- `decidekit/switch`: a decider for the switch agent harness.
|
|
50
|
-
- A CLI: `
|
|
71
|
+
- A CLI: `decidekit train` and `decidekit decide`.
|
|
51
72
|
|
|
52
73
|
### Known limits
|
|
53
74
|
|
package/README.md
CHANGED
|
@@ -1,12 +1,12 @@
|
|
|
1
|
-
#
|
|
1
|
+
# decidekit
|
|
2
2
|
|
|
3
|
-
[](https://www.npmjs.com/package/decidekit) [](https://www.npmjs.com/package/decidekit) [](https://github.com/debuffjp/decidekit/actions/workflows/ci.yml) [](LICENSE) [](https://discord.gg/jshbDHSpCY)
|
|
4
4
|
|
|
5
|
-
**[Try it in your browser](https://debuff.jp/
|
|
5
|
+
**[Try it in your browser](https://debuff.jp/decide/playground/)** · [Watch it learn from an LLM](https://debuff.jp/decide/lanes/) · [Launch post](https://debuff.jp/blog/introducing-decide) · [Research: when the LLM is wrong](https://debuff.jp/research/learning-from-a-noisy-llm)
|
|
6
6
|
|
|
7
7
|
A decision model for the browser and Node: it picks one of your labels for a piece of text, with confidence you can trust, and learns from an LLM when it isn't sure. Your labels are the type. No server, no API key.
|
|
8
8
|
|
|
9
|
-
A System One model in the browser: like TypeSafe's Jev, but on the user's device, so each decision is free, private and a few milliseconds. Jev leaves unsure cases to your code;
|
|
9
|
+
A System One model in the browser: like TypeSafe's Jev, but on the user's device, so each decision is free, private and a few milliseconds. Jev leaves unsure cases to your code; decidekit can ask an LLM about them and learn from the answers. decidekit is an independent project, not affiliated with TypeSafe.
|
|
10
10
|
|
|
11
11
|
```ts
|
|
12
12
|
import { createDecider } from "decidekit";
|
|
@@ -29,8 +29,8 @@ await d.decide(text, ["question", "bug_report", "praise"]);
|
|
|
29
29
|
await d.decide(text, { refund: "wants their money back", bug: "something is broken" });
|
|
30
30
|
```
|
|
31
31
|
|
|
32
|
-
Zero-shot is for trying things out:
|
|
33
|
-
Its confidence came out close to calibrated on those datasets (error 0.
|
|
32
|
+
Zero-shot is for trying things out: 66% on BANKING77 and 73% on CLINC150 (dev splits), against 90-94% trained.
|
|
33
|
+
Its confidence came out close to calibrated on those datasets (error 0.03-0.04), but check
|
|
34
34
|
it on your own labels before relying on a threshold.
|
|
35
35
|
|
|
36
36
|
## Train a head
|
|
@@ -109,7 +109,7 @@ The [guide](docs/guide-typesafe.md) covers what differs, including what `confide
|
|
|
109
109
|
|
|
110
110
|
## switch
|
|
111
111
|
|
|
112
|
-
`decidekit/switch` makes
|
|
112
|
+
`decidekit/switch` makes decidekit a decision model for [switch](https://github.com/osolmaz/switch), the harness that
|
|
113
113
|
runs a decision model through a task. Zero-shot, it won two of the three WikiRace races from switch's example on a
|
|
114
114
|
laptop CPU. See the [switch guide](docs/guide-switch.md).
|
|
115
115
|
|
|
@@ -132,7 +132,7 @@ const d = await createWorkerDecider({
|
|
|
132
132
|
const r = await d.decide(text, head);
|
|
133
133
|
```
|
|
134
134
|
|
|
135
|
-
Import `loadHead` from `decidekit/browser` too, not from `
|
|
135
|
+
Import `loadHead` from `decidekit/browser` too, not from `decidekit`: that keeps the encoder out of your page bundle.
|
|
136
136
|
|
|
137
137
|
Tested from npm in Vite 7, webpack 5 and Next.js 16 apps; see [docs/bundlers.md](docs/bundlers.md) for the config, including a one-line alias
|
|
138
138
|
that halves the wasm download. Serve the page cross-origin isolated if you can:
|
|
@@ -146,9 +146,24 @@ That lets wasm use several threads: in our runs with all-MiniLM-L6-v2 it cut p95
|
|
|
146
146
|
|
|
147
147
|
Download on first visit: the model (34 MB) and onnxruntime's wasm (23.6 MB, 5.8 MB gzipped), both cached after.
|
|
148
148
|
|
|
149
|
+
## Models
|
|
150
|
+
|
|
151
|
+
decidekit runs the Decide models, each a sentence encoder plus a calibrated head:
|
|
152
|
+
|
|
153
|
+
| Model | Id | Download | Reads | Upstream |
|
|
154
|
+
|---|---|---|---|---|
|
|
155
|
+
| Decide Small (default) | [`debuffjp/decide-small`](https://huggingface.co/debuffjp/decide-small) | 34 MB | English | BAAI's [bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5), identical to `Xenova/bge-small-en-v1.5` |
|
|
156
|
+
| Decide Multilingual | [`debuffjp/decide-multilingual`](https://huggingface.co/debuffjp/decide-multilingual) | about 118 MB | 76% on Japanese and Chinese intents | intfloat's [multilingual-e5-small](https://huggingface.co/intfloat/multilingual-e5-small), identical to `Xenova/multilingual-e5-small` |
|
|
157
|
+
|
|
158
|
+
Pass the id as `model` to `createDecider()`. The files download from Hugging Face on first use, pinned to a revision, and
|
|
159
|
+
are then cached. To serve them yourself, pass `modelHost`: see [self-hosting](docs/guide-browser.md#host-the-model-files-yourself).
|
|
160
|
+
|
|
161
|
+
Before 0.1.1 the default id was `Xenova/bge-small-en-v1.5`. The old ids and heads recording them keep working: they
|
|
162
|
+
resolve to the new ids, so `d.model` returns the new id.
|
|
163
|
+
|
|
149
164
|
## Numbers
|
|
150
165
|
|
|
151
|
-
bge-small-en-v1.5
|
|
166
|
+
Decide Small (bge-small-en-v1.5, the default) at q8, without label names, Ryzen 7 260 laptop, Node 24 and headless Chrome 156, 2026-11-01. Accuracy
|
|
152
167
|
and calibration error (ECE, lower is better) on each dataset's test split, mean of 3 seeds (one run for "all"):
|
|
153
168
|
[eval/results-bge-small.json](eval/results-bge-small.json).
|
|
154
169
|
|
|
@@ -185,16 +200,16 @@ in-scope examples only, with `minConfidence` 0.93 picked on the validation split
|
|
|
185
200
|
|
|
186
201
|
**The smaller model.** `createDecider({ model: "Xenova/all-MiniLM-L6-v2" })` (23.7 MB, the pre-release default) is
|
|
187
202
|
about 1.8x faster per decision in Chrome and 3 to 5 points less accurate with few examples, about the same with all
|
|
188
|
-
the data. Pass it to keep using heads trained on it. Before
|
|
189
|
-
`{ labelNames: false }` with bge-small
|
|
203
|
+
the data. Pass it to keep using heads trained on it. Before 0.1.0 a label name counted as 3 examples and the README
|
|
204
|
+
suggested `{ labelNames: false }` with bge-small. It now counts as 1, which helps.
|
|
190
205
|
|
|
191
206
|
Reproduce: `npm run eval:suite` (Node), `npm run bench` then `npm run bench:run` (Chrome).
|
|
192
207
|
|
|
193
208
|
## Playground
|
|
194
209
|
|
|
195
|
-
Try it at [debuff.jp/
|
|
210
|
+
Try it at [debuff.jp/decide](https://debuff.jp/decide/). To run it locally: `npm run playground`, then serve `playground/dist` cross-origin isolated (`node bench/serve.mjs` style headers).
|
|
196
211
|
Try zero-shot labels, the ready-made heads, a head trained in the page and "none of these".
|
|
197
|
-
[Two lanes](https://debuff.jp/
|
|
212
|
+
[Two lanes](https://debuff.jp/decide/lanes/) shows `withFallback` live: decidekit starts from twelve label names,
|
|
198
213
|
asks a small LLM when it's unsure, learns from each answer, and replays the first 20 messages at the end.
|
|
199
214
|
|
|
200
215
|
## Ready-made heads
|
|
@@ -213,30 +228,26 @@ and offline apps, and text that can't leave the device. Details, code and poor f
|
|
|
213
228
|
|
|
214
229
|
| | Runs in the browser | Trains in the browser | Typed result | Calibrated confidence |
|
|
215
230
|
|---|---|---|---|---|
|
|
216
|
-
|
|
|
231
|
+
| decidekit | yes (worker) | yes | yes | trained heads |
|
|
217
232
|
| [jimothy](https://www.npmjs.com/package/jimothy) | inference (worker) | no, Node | yes | yes |
|
|
218
233
|
| [Jev](https://typesafe.ai/) (TypeSafe) | no, API | no | yes | yes |
|
|
219
234
|
| transformers.js zero-shot pipeline | yes | no | no | no |
|
|
220
235
|
| MediaPipe text classifier | yes | no, offline | no | no |
|
|
221
236
|
| LLM structured output (AI SDK `Output.choice`) | no, API | n/a | yes | no |
|
|
222
237
|
|
|
223
|
-
jimothy reports 92.4% on BANKING77, above our 90.7%. Pick it if you train in Node and want the higher number. Pick
|
|
224
|
-
decidejs if training has to happen in the page, from the user's own data.
|
|
225
|
-
|
|
226
238
|
## Docs
|
|
227
239
|
|
|
228
|
-
[docs/README.md](docs/README.md) indexes the guides (training, confidence and thresholds, the browser), the
|
|
240
|
+
[debuff.jp/decide/docs](https://debuff.jp/decide/docs/) (also in [docs/README.md](docs/README.md)) indexes the guides (training, confidence and thresholds, the browser), the
|
|
229
241
|
API reference, how it works and the FAQ.
|
|
230
242
|
|
|
231
243
|
## Community
|
|
232
244
|
|
|
233
|
-
Questions and show-and-tell in [Discussions](https://github.com/debuffjp/
|
|
245
|
+
Questions and show-and-tell in [Discussions](https://github.com/debuffjp/decidekit/discussions), chat in the
|
|
234
246
|
[debuff Discord](https://discord.gg/jshbDHSpCY). Results on your own data, good or bad, help most: there's an
|
|
235
247
|
issue template for them. See [CONTRIBUTING.md](CONTRIBUTING.md).
|
|
236
248
|
|
|
237
249
|
## Status
|
|
238
250
|
|
|
239
|
-
0.1. English with the default model, other languages with
|
|
240
|
-
fallback to a hosted model below the threshold, and fine-tuning the encoder (the likely route past about 91%).
|
|
251
|
+
0.1. English with the default model, other languages with Decide Multilingual ([FAQ](docs/faq.md#which-languages-does-it-support)); short texts (about a paragraph). Not done yet: WebGPU in the worker and fine-tuning the encoder (the likely route past about 91%).
|
|
241
252
|
|
|
242
253
|
Made by the [debuff](https://debuff.jp) team. MIT.
|
package/dist/browser.js
CHANGED
|
@@ -36,7 +36,7 @@ export async function createWorkerDecider({ onProgress, ...options } = {}, worke
|
|
|
36
36
|
terminate: () => {
|
|
37
37
|
worker.terminate();
|
|
38
38
|
for (const p of pending.values())
|
|
39
|
-
p.reject(new Error("
|
|
39
|
+
p.reject(new Error("decidekit: the worker was terminated"));
|
|
40
40
|
pending.clear();
|
|
41
41
|
},
|
|
42
42
|
};
|
package/dist/fallback.js
CHANGED
|
@@ -25,9 +25,9 @@ export function withFallback(decider, using, options) {
|
|
|
25
25
|
perLabel.set(e.label, (perLabel.get(e.label) ?? 0) + 1);
|
|
26
26
|
function retrain() {
|
|
27
27
|
if (!examples)
|
|
28
|
-
return Promise.reject(new Error("
|
|
28
|
+
return Promise.reject(new Error("decidekit: withFallback() needs `examples` to retrain a head"));
|
|
29
29
|
if (!examples.length && !learned.length)
|
|
30
|
-
return Promise.reject(new Error("
|
|
30
|
+
return Promise.reject(new Error("decidekit: nothing to train on yet: no answers from `ask`"));
|
|
31
31
|
const calibrateOn = trainOptions?.calibrateOn ?? (confident.length >= 50 ? confident : undefined);
|
|
32
32
|
const trainLabels = learned.some((e) => e.label === NONE) ? [...labels, NONE] : labels;
|
|
33
33
|
retraining ??= decider.train([...examples, ...learned], { ...trainOptions, labels: trainLabels, calibrateOn }).then((h) => {
|
package/dist/index.d.ts
CHANGED
|
@@ -72,13 +72,19 @@ export interface TrainOptions<L extends string = string> {
|
|
|
72
72
|
}
|
|
73
73
|
export interface DeciderOptions {
|
|
74
74
|
/**
|
|
75
|
-
* A Hugging Face sentence-embedding model with ONNX weights. Default `
|
|
75
|
+
* A Hugging Face sentence-embedding model with ONNX weights. Default `debuffjp/decide-small` (Decide Small, 34 MB at q8).
|
|
76
|
+
* Before 0.1.1 the default id was `Xenova/bge-small-en-v1.5`, the same weights; that id and heads recording it still work.
|
|
76
77
|
* Pre-release builds defaulted to `Xenova/all-MiniLM-L6-v2` (23 MB): smaller and 1.6x faster, 3 to 5 points less
|
|
77
78
|
* accurate. Pass it to keep using heads trained on it.
|
|
78
79
|
*/
|
|
79
80
|
model?: string;
|
|
80
81
|
/** Defaults to transformers.js' choice: wasm in browsers, cpu in Node. */
|
|
81
82
|
device?: "wasm" | "webgpu" | "cpu";
|
|
83
|
+
/**
|
|
84
|
+
* Serve model files from your own host instead of Hugging Face, as `<modelHost>/<model>/`, for example
|
|
85
|
+
* `https://cdn.example.com/models/debuffjp/decide-small/onnx/model_quantized.onnx`. Applies to the whole page or process.
|
|
86
|
+
*/
|
|
87
|
+
modelHost?: string;
|
|
82
88
|
/** Weight precision. Default `q8`. */
|
|
83
89
|
dtype?: "q8" | "fp16" | "fp32";
|
|
84
90
|
/** Called as model files download (first load only; later loads come from cache). */
|
|
@@ -98,7 +104,7 @@ export interface DeciderOptions {
|
|
|
98
104
|
* `train()`. In a browser, prefer `createWorkerDecider()` from `decidekit/browser`: the encoder blocks the thread
|
|
99
105
|
* it runs on.
|
|
100
106
|
*/
|
|
101
|
-
export declare function createDecider({ model, device, dtype, temperature, onProgress }?: DeciderOptions): Promise<{
|
|
107
|
+
export declare function createDecider({ model: requested, device, dtype, temperature, onProgress, modelHost }?: DeciderOptions): Promise<{
|
|
102
108
|
model: string;
|
|
103
109
|
decide: {
|
|
104
110
|
<const L extends string>(text: string, using: Choices<L>, options?: {
|
package/dist/index.js
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
// decide(text, options): typed, calibrated decisions on device. Zero-shot compares the text with each option's
|
|
2
2
|
// embedding; a trained head (a softmax layer over the same embeddings, 156 KB for 77 labels) is far more
|
|
3
3
|
// accurate. Both return the option literals as the choice type.
|
|
4
|
-
import { pipeline } from "@huggingface/transformers";
|
|
4
|
+
import { env, pipeline } from "@huggingface/transformers";
|
|
5
5
|
import { fitHead, logits, softmax } from "./head.js";
|
|
6
6
|
function isHead(c) {
|
|
7
7
|
if (Array.isArray(c) || !("weights" in c))
|
|
8
8
|
return false;
|
|
9
9
|
if (!(c.weights instanceof Float32Array))
|
|
10
|
-
throw new Error("
|
|
10
|
+
throw new Error("decidekit: this head's weights aren't a Float32Array. Read head files with loadHead(), not JSON.parse.");
|
|
11
11
|
return true;
|
|
12
12
|
}
|
|
13
13
|
const readable = (label) => label.replace(/[_-]+/g, " ").replace(/([a-z])([A-Z])/g, "$1 $2").toLowerCase();
|
|
@@ -23,9 +23,15 @@ const LABEL_NAME_WEIGHT = 1;
|
|
|
23
23
|
// multilingual-e5-small's similarities sit close together, so it needs far lower temperatures: on MASSIVE in English
|
|
24
24
|
// and Japanese, 0.0105 for label names (calibration error 0.03 to 0.04; descriptions not yet measured) and a floor of
|
|
25
25
|
// 0.4 (eval/e5-settings.ts). It also expects "query: " before every text.
|
|
26
|
-
|
|
27
|
-
const
|
|
28
|
-
const
|
|
26
|
+
// The Decide models are unmodified copies of these upstream ids, so the old ids and heads trained on them still work.
|
|
27
|
+
const SMALL = "debuffjp/decide-small", MULTILINGUAL = "debuffjp/decide-multilingual";
|
|
28
|
+
const SAME_MODEL = { "Xenova/bge-small-en-v1.5": SMALL, "Xenova/multilingual-e5-small": MULTILINGUAL };
|
|
29
|
+
const canonical = (model) => SAME_MODEL[model] ?? model;
|
|
30
|
+
// Pinned so a later change on Hugging Face can't change what an installed version decides.
|
|
31
|
+
const REVISIONS = { [SMALL]: "188d690f86190863e3b4e44e29c88348b10230c4", [MULTILINGUAL]: "a3e353d26d846b2163db8acad65ef3d43893592e" };
|
|
32
|
+
const SMALL_HEAD_FLOOR = { [SMALL]: 0.5, [MULTILINGUAL]: 0.4 };
|
|
33
|
+
const TEMPERATURES = { [SMALL]: [0.03, 0.025], "Xenova/all-MiniLM-L6-v2": [0.05, 0.05], [MULTILINGUAL]: [0.0105, 0.0105] };
|
|
34
|
+
const PREFIX = { [MULTILINGUAL]: "query: " };
|
|
29
35
|
function toDecision(labels, p) {
|
|
30
36
|
let best = 0;
|
|
31
37
|
for (let i = 1; i < p.length; i++)
|
|
@@ -39,9 +45,14 @@ function toDecision(labels, p) {
|
|
|
39
45
|
* `train()`. In a browser, prefer `createWorkerDecider()` from `decidekit/browser`: the encoder blocks the thread
|
|
40
46
|
* it runs on.
|
|
41
47
|
*/
|
|
42
|
-
export async function createDecider({ model =
|
|
48
|
+
export async function createDecider({ model: requested = SMALL, device, dtype = "q8", temperature, onProgress, modelHost } = {}) {
|
|
49
|
+
const model = canonical(requested);
|
|
50
|
+
if (modelHost) {
|
|
51
|
+
env.remoteHost = modelHost.endsWith("/") ? modelHost : modelHost + "/";
|
|
52
|
+
env.remotePathTemplate = "{model}/";
|
|
53
|
+
}
|
|
43
54
|
const embed = await pipeline("feature-extraction", model, {
|
|
44
|
-
device, dtype,
|
|
55
|
+
device, dtype, revision: REVISIONS[model],
|
|
45
56
|
progress_callback: onProgress && ((p) => p.status === "progress" && onProgress({ file: p.file, loaded: p.loaded, total: p.total })),
|
|
46
57
|
});
|
|
47
58
|
const optionCache = new Map(), prefix = PREFIX[model] ?? "";
|
|
@@ -58,14 +69,14 @@ export async function createDecider({ model = "Xenova/bge-small-en-v1.5", device
|
|
|
58
69
|
async function score(q, using, minConfidence) {
|
|
59
70
|
let d;
|
|
60
71
|
if (isHead(using)) {
|
|
61
|
-
if (using.model !== model)
|
|
62
|
-
throw new Error(`
|
|
72
|
+
if (canonical(using.model) !== model)
|
|
73
|
+
throw new Error(`decidekit: this head was trained on ${using.model}, but the decider uses ${model}. Pass { model: "${using.model}" } to createDecider.`);
|
|
63
74
|
d = toDecision(using.labels, softmax(logits(using, q), using.temperature));
|
|
64
75
|
}
|
|
65
76
|
else {
|
|
66
77
|
const labels = (Array.isArray(using) ? using : Object.keys(using));
|
|
67
78
|
if (labels.length < 2)
|
|
68
|
-
throw new Error(`
|
|
79
|
+
throw new Error(`decidekit: decide() needs at least 2 options, got ${labels.length}`);
|
|
69
80
|
// Bare labels are compared as readable text ("card_arrival" as "card arrival"): 2.6 points better on CLINC150's
|
|
70
81
|
// dev split. A described label scores by the better of its name and its description: on the dev splits that
|
|
71
82
|
// beat the description alone by 0.5 (BANKING77) and 3.1 points (CLINC150), and names alone by 7 and 10.
|
|
@@ -98,15 +109,15 @@ export async function createDecider({ model = "Xenova/bge-small-en-v1.5", device
|
|
|
98
109
|
async function train(examples, opts = {}) {
|
|
99
110
|
const bad = examples.findIndex((e) => typeof e?.text !== "string" || typeof e?.label !== "string" || !e.label);
|
|
100
111
|
if (bad >= 0)
|
|
101
|
-
throw new Error(`
|
|
112
|
+
throw new Error(`decidekit: example ${bad} needs a string text and a non-empty string label, got ${JSON.stringify(examples[bad])}`);
|
|
102
113
|
if (!examples.length)
|
|
103
|
-
throw new Error("
|
|
114
|
+
throw new Error("decidekit: train() needs at least 1 example");
|
|
104
115
|
const labels = [...new Set([...(opts.labels ?? []), ...examples.map((e) => e.label)])].sort();
|
|
105
116
|
if (opts.labels && opts.labelNames === false && labels.some((l) => !examples.some((e) => e.label === l))) {
|
|
106
|
-
throw new Error("
|
|
117
|
+
throw new Error("decidekit: labels without examples need labelNames, which is what they start from");
|
|
107
118
|
}
|
|
108
119
|
if (labels.length < 2)
|
|
109
|
-
throw new Error(`
|
|
120
|
+
throw new Error(`decidekit: train() needs at least 2 labels, got ${labels.length}`);
|
|
110
121
|
const y = examples.map((e) => labels.indexOf(e.label));
|
|
111
122
|
const texts = [...new Set(examples.map((e) => e.text))];
|
|
112
123
|
const fresh = texts.filter((t) => !trainCache.has(t));
|
|
@@ -122,7 +133,7 @@ export async function createDecider({ model = "Xenova/bge-small-en-v1.5", device
|
|
|
122
133
|
const cal = opts.calibrateOn;
|
|
123
134
|
const unknown = cal?.find((e) => !labels.includes(e.label));
|
|
124
135
|
if (unknown)
|
|
125
|
-
throw new Error(`
|
|
136
|
+
throw new Error(`decidekit: calibrateOn has label ${JSON.stringify(unknown.label)}, which isn't one of the head's labels`);
|
|
126
137
|
const calibration = cal?.length ? { X: await vectors(cal.map((e) => e.text)), y: cal.map((e) => labels.indexOf(e.label)) } : undefined;
|
|
127
138
|
return { model, ...(await fitHead(X, y, labels, opts, priors, LABEL_NAME_WEIGHT, undefined, SMALL_HEAD_FLOOR[model] ?? 1, calibration)) };
|
|
128
139
|
}
|
package/dist/switch.d.ts
CHANGED
|
@@ -81,22 +81,22 @@ export interface SwitchOptions {
|
|
|
81
81
|
/** Trained heads for questions that repeat, by question id. Questions without one are decided zero-shot. */
|
|
82
82
|
heads?: Readonly<Record<string, Head>>;
|
|
83
83
|
/**
|
|
84
|
-
* A stronger decider to ask when
|
|
84
|
+
* A stronger decider to ask when decidekit isn't sure, such as Jev through switch's pi-ai decider. Its answers to
|
|
85
85
|
* choice questions train a head for that question, so a question that repeats asks it less and less (see `withFallback`).
|
|
86
86
|
*/
|
|
87
87
|
teacher?: SwitchDecider;
|
|
88
88
|
/** Below this confidence, choice questions go to `teacher`. Default 0.9. */
|
|
89
89
|
minConfidence?: number;
|
|
90
90
|
/**
|
|
91
|
-
* What
|
|
91
|
+
* What decidekit reads for a question. Its encoder reads about 256 tokens and drops the rest, so a long state or long
|
|
92
92
|
* instructions can push out what matters; pick it here, such as the target of a search. Default: `textOf`.
|
|
93
93
|
*/
|
|
94
94
|
text?: (state: SwitchRequest["decisionState"], question: Question, id: string) => string;
|
|
95
95
|
}
|
|
96
|
-
/** The text
|
|
96
|
+
/** The text decidekit reads for a question: its instructions, then the state as plain lines rather than JSON. */
|
|
97
97
|
export declare function textOf(instructions: string, state: Json): string;
|
|
98
98
|
/**
|
|
99
|
-
* Wraps a
|
|
99
|
+
* Wraps a decidekit decider as a switch decider. Choice, bool and score questions become decidekit decisions over the
|
|
100
100
|
* question's criteria; images are ignored. Every call is local, so usage costs nothing.
|
|
101
101
|
*/
|
|
102
102
|
export declare function switchDecider(local: Local, options?: SwitchOptions): SwitchDecider;
|
package/dist/switch.js
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
|
-
//
|
|
2
|
-
// task by asking it typed questions. The types below copy the parts of switch's `Decider` that this needs, so
|
|
1
|
+
// decidekit as a decision model for switch (github.com/osolmaz/switch), a harness that runs a decision model through a
|
|
2
|
+
// task by asking it typed questions. The types below copy the parts of switch's `Decider` that this needs, so decidekit
|
|
3
3
|
// doesn't depend on switch: pass the result wherever switch takes a decider.
|
|
4
4
|
import { withFallback } from "./fallback.js";
|
|
5
|
-
/** The text
|
|
5
|
+
/** The text decidekit reads for a question: its instructions, then the state as plain lines rather than JSON. */
|
|
6
6
|
export function textOf(instructions, state) {
|
|
7
7
|
const lines = [];
|
|
8
8
|
const walk = (v, key) => {
|
|
@@ -18,7 +18,7 @@ export function textOf(instructions, state) {
|
|
|
18
18
|
return [instructions, ...lines].filter(Boolean).join("\n");
|
|
19
19
|
}
|
|
20
20
|
/**
|
|
21
|
-
* Wraps a
|
|
21
|
+
* Wraps a decidekit decider as a switch decider. Choice, bool and score questions become decidekit decisions over the
|
|
22
22
|
* question's criteria; images are ignored. Every call is local, so usage costs nothing.
|
|
23
23
|
*/
|
|
24
24
|
export function switchDecider(local, options = {}) {
|
|
@@ -48,7 +48,7 @@ export function switchDecider(local, options = {}) {
|
|
|
48
48
|
const text = textFor(state, q, id);
|
|
49
49
|
if (q.type === "choice") {
|
|
50
50
|
const keys = Object.keys(q.criteria);
|
|
51
|
-
//
|
|
51
|
+
// decidekit compares two or more options; a single one is the answer.
|
|
52
52
|
if (keys.length === 1)
|
|
53
53
|
return { type: "choice", choice: keys[0], probabilities: { [keys[0]]: 1 }, confidence: 1 };
|
|
54
54
|
if (teacher)
|
|
@@ -66,14 +66,14 @@ export function switchDecider(local, options = {}) {
|
|
|
66
66
|
return { type: "score", score: Number(d.choice), confidence: d.confidence };
|
|
67
67
|
}
|
|
68
68
|
return {
|
|
69
|
-
model: `
|
|
69
|
+
model: `decidekit/${local.model}`,
|
|
70
70
|
// The encoder reads about 256 tokens and cuts the rest, so a long state still fits; tell switch the window is big
|
|
71
71
|
// so it doesn't trim the choices to fit.
|
|
72
72
|
window: 1_000_000,
|
|
73
73
|
charsPerToken: 4,
|
|
74
74
|
async decide(request) {
|
|
75
75
|
const started = performance.now();
|
|
76
|
-
const base = { api: "
|
|
76
|
+
const base = { api: "decidekit", provider: "decidekit", model: `decidekit/${local.model}`, timestamp: Date.now(), tooLong: false };
|
|
77
77
|
try {
|
|
78
78
|
const entries = await Promise.all(Object.entries(request.decisionQuestions).map(async ([id, q]) => [id, await answer(id, q, request.decisionState)]));
|
|
79
79
|
const input = Math.round(JSON.stringify(request.decisionState).length / 4);
|
package/dist/typesafe.d.ts
CHANGED
|
@@ -5,7 +5,7 @@ export interface Decides {
|
|
|
5
5
|
}
|
|
6
6
|
export interface ChoiceQuestion<L extends string = string> {
|
|
7
7
|
type: "choice";
|
|
8
|
-
/** Kept for compatibility and ignored:
|
|
8
|
+
/** Kept for compatibility and ignored: decidekit compares the state with the options, it doesn't read a question. */
|
|
9
9
|
instructions: string;
|
|
10
10
|
/** Option names to descriptions (`null` for none), as in TypeSafe's API, or a trained head. */
|
|
11
11
|
criteria: Readonly<Record<L, string | null>> | Head<L>;
|
package/dist/typesafe.js
CHANGED
|
@@ -2,7 +2,7 @@ export function choice(instructions, criteria) {
|
|
|
2
2
|
return { type: "choice", instructions, criteria };
|
|
3
3
|
}
|
|
4
4
|
const unsupported = (type) => () => {
|
|
5
|
-
throw new Error(`
|
|
5
|
+
throw new Error(`decidekit: ${type} questions aren't supported locally, only choice. Ask a yes/no as a choice between two described options.`);
|
|
6
6
|
};
|
|
7
7
|
export const noul = unsupported("noul");
|
|
8
8
|
export const score = unsupported("score");
|
|
@@ -15,7 +15,7 @@ function stateText(state) {
|
|
|
15
15
|
return v;
|
|
16
16
|
if (typeof v === "number" || typeof v === "boolean")
|
|
17
17
|
return String(v);
|
|
18
|
-
throw new Error("
|
|
18
|
+
throw new Error("decidekit: state values must be text, numbers or booleans");
|
|
19
19
|
}).join("\n");
|
|
20
20
|
}
|
|
21
21
|
function options(criteria) {
|
|
@@ -41,7 +41,7 @@ export function typesafeClient(decider) {
|
|
|
41
41
|
const n = Object.keys(probabilities).length;
|
|
42
42
|
answers[id] = { type: "choice", choice, confidence: Math.max(0, (n * confidence - 1) / (n - 1)), probabilities };
|
|
43
43
|
}
|
|
44
|
-
return { model: "
|
|
44
|
+
return { model: "decidekit", answers: answers };
|
|
45
45
|
},
|
|
46
46
|
};
|
|
47
47
|
}
|
package/package.json
CHANGED
|
@@ -1,93 +1,93 @@
|
|
|
1
|
-
{
|
|
2
|
-
"name": "decidekit",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"description": "A System One model in the browser: typed, calibrated text decisions on the user's device. Learns from an LLM when it isn't sure.",
|
|
5
|
-
"license": "MIT",
|
|
6
|
-
"repository": "github:debuffjp/
|
|
7
|
-
"keywords": [
|
|
8
|
-
"decision model",
|
|
9
|
-
"system one",
|
|
10
|
-
"classification",
|
|
11
|
-
"intent",
|
|
12
|
-
"zero-shot",
|
|
13
|
-
"calibration",
|
|
14
|
-
"llm",
|
|
15
|
-
"embeddings",
|
|
16
|
-
"browser",
|
|
17
|
-
"typescript",
|
|
18
|
-
"text classification",
|
|
19
|
-
"transformers.js",
|
|
20
|
-
"on-device",
|
|
21
|
-
"llm fallback"
|
|
22
|
-
],
|
|
23
|
-
"type": "module",
|
|
24
|
-
"exports": {
|
|
25
|
-
".": {
|
|
26
|
-
"types": "./dist/index.d.ts",
|
|
27
|
-
"default": "./dist/index.js"
|
|
28
|
-
},
|
|
29
|
-
"./browser": {
|
|
30
|
-
"types": "./dist/browser.d.ts",
|
|
31
|
-
"default": "./dist/browser.js"
|
|
32
|
-
},
|
|
33
|
-
"./react": {
|
|
34
|
-
"types": "./dist/react.d.ts",
|
|
35
|
-
"default": "./dist/react.js"
|
|
36
|
-
},
|
|
37
|
-
"./switch": {
|
|
38
|
-
"types": "./dist/switch.d.ts",
|
|
39
|
-
"default": "./dist/switch.js"
|
|
40
|
-
},
|
|
41
|
-
"./typesafe": {
|
|
42
|
-
"types": "./dist/typesafe.d.ts",
|
|
43
|
-
"default": "./dist/typesafe.js"
|
|
44
|
-
}
|
|
45
|
-
},
|
|
46
|
-
"files": [
|
|
47
|
-
"dist",
|
|
48
|
-
"CHANGELOG.md"
|
|
49
|
-
],
|
|
50
|
-
"scripts": {
|
|
51
|
-
"build": "tsc -p tsconfig.build.json",
|
|
52
|
-
"prepack": "npm run build",
|
|
53
|
-
"typecheck": "tsc --noEmit",
|
|
54
|
-
"eval": "node --experimental-strip-types eval/banking77.ts",
|
|
55
|
-
"bench": "esbuild src/worker.ts src/browser.ts --bundle --format=esm --platform=browser --outdir=bench --alias:onnxruntime-web/webgpu=onnxruntime-web/wasm && node bench/serve.mjs",
|
|
56
|
-
"playground": "node playground/build.mjs",
|
|
57
|
-
"heads": "node --experimental-strip-types heads/build.ts",
|
|
58
|
-
"bench:run": "node bench/run.mjs",
|
|
59
|
-
"eval:suite": "node --experimental-strip-types eval/suite.ts",
|
|
60
|
-
"eval:hill": "node --experimental-strip-types eval/hill.ts",
|
|
61
|
-
"test": "node --test --experimental-strip-types \"test/*.test.ts\"",
|
|
62
|
-
"lint:prose": "node scripts/lint-prose.mjs",
|
|
63
|
-
"smoke": "node scripts/smoke.mjs"
|
|
64
|
-
},
|
|
65
|
-
"dependencies": {
|
|
66
|
-
"@huggingface/transformers": "4.2.0"
|
|
67
|
-
},
|
|
68
|
-
"devDependencies": {
|
|
69
|
-
"@types/node": "22.15.3",
|
|
70
|
-
"@types/react": "19.3.0",
|
|
71
|
-
"esbuild": "0.25.10",
|
|
72
|
-
"playwright-core": "1.64.1",
|
|
73
|
-
"react": "19.3.1",
|
|
74
|
-
"spryte": "0.11.0",
|
|
75
|
-
"typescript": "5.9.3"
|
|
76
|
-
},
|
|
77
|
-
"engines": {
|
|
78
|
-
"node": ">=22.18"
|
|
79
|
-
},
|
|
80
|
-
"homepage": "https://debuff.jp/
|
|
81
|
-
"bugs": "https://github.com/debuffjp/
|
|
82
|
-
"bin": {
|
|
83
|
-
"decidekit": "./dist/cli.js"
|
|
84
|
-
},
|
|
85
|
-
"peerDependencies": {
|
|
86
|
-
"react": ">=18"
|
|
87
|
-
},
|
|
88
|
-
"peerDependenciesMeta": {
|
|
89
|
-
"react": {
|
|
90
|
-
"optional": true
|
|
91
|
-
}
|
|
92
|
-
}
|
|
93
|
-
}
|
|
1
|
+
{
|
|
2
|
+
"name": "decidekit",
|
|
3
|
+
"version": "0.1.1",
|
|
4
|
+
"description": "A System One model in the browser: typed, calibrated text decisions on the user's device. Learns from an LLM when it isn't sure.",
|
|
5
|
+
"license": "MIT",
|
|
6
|
+
"repository": "github:debuffjp/decidekit",
|
|
7
|
+
"keywords": [
|
|
8
|
+
"decision model",
|
|
9
|
+
"system one",
|
|
10
|
+
"classification",
|
|
11
|
+
"intent",
|
|
12
|
+
"zero-shot",
|
|
13
|
+
"calibration",
|
|
14
|
+
"llm",
|
|
15
|
+
"embeddings",
|
|
16
|
+
"browser",
|
|
17
|
+
"typescript",
|
|
18
|
+
"text classification",
|
|
19
|
+
"transformers.js",
|
|
20
|
+
"on-device",
|
|
21
|
+
"llm fallback"
|
|
22
|
+
],
|
|
23
|
+
"type": "module",
|
|
24
|
+
"exports": {
|
|
25
|
+
".": {
|
|
26
|
+
"types": "./dist/index.d.ts",
|
|
27
|
+
"default": "./dist/index.js"
|
|
28
|
+
},
|
|
29
|
+
"./browser": {
|
|
30
|
+
"types": "./dist/browser.d.ts",
|
|
31
|
+
"default": "./dist/browser.js"
|
|
32
|
+
},
|
|
33
|
+
"./react": {
|
|
34
|
+
"types": "./dist/react.d.ts",
|
|
35
|
+
"default": "./dist/react.js"
|
|
36
|
+
},
|
|
37
|
+
"./switch": {
|
|
38
|
+
"types": "./dist/switch.d.ts",
|
|
39
|
+
"default": "./dist/switch.js"
|
|
40
|
+
},
|
|
41
|
+
"./typesafe": {
|
|
42
|
+
"types": "./dist/typesafe.d.ts",
|
|
43
|
+
"default": "./dist/typesafe.js"
|
|
44
|
+
}
|
|
45
|
+
},
|
|
46
|
+
"files": [
|
|
47
|
+
"dist",
|
|
48
|
+
"CHANGELOG.md"
|
|
49
|
+
],
|
|
50
|
+
"scripts": {
|
|
51
|
+
"build": "tsc -p tsconfig.build.json",
|
|
52
|
+
"prepack": "npm run build",
|
|
53
|
+
"typecheck": "tsc --noEmit",
|
|
54
|
+
"eval": "node --experimental-strip-types eval/banking77.ts",
|
|
55
|
+
"bench": "esbuild src/worker.ts src/browser.ts --bundle --format=esm --platform=browser --outdir=bench --alias:onnxruntime-web/webgpu=onnxruntime-web/wasm && node bench/serve.mjs",
|
|
56
|
+
"playground": "node playground/build.mjs",
|
|
57
|
+
"heads": "node --experimental-strip-types heads/build.ts",
|
|
58
|
+
"bench:run": "node bench/run.mjs",
|
|
59
|
+
"eval:suite": "node --experimental-strip-types eval/suite.ts",
|
|
60
|
+
"eval:hill": "node --experimental-strip-types eval/hill.ts",
|
|
61
|
+
"test": "node --test --experimental-strip-types \"test/*.test.ts\"",
|
|
62
|
+
"lint:prose": "node scripts/lint-prose.mjs",
|
|
63
|
+
"smoke": "node scripts/smoke.mjs"
|
|
64
|
+
},
|
|
65
|
+
"dependencies": {
|
|
66
|
+
"@huggingface/transformers": "4.2.0"
|
|
67
|
+
},
|
|
68
|
+
"devDependencies": {
|
|
69
|
+
"@types/node": "22.15.3",
|
|
70
|
+
"@types/react": "19.3.0",
|
|
71
|
+
"esbuild": "0.25.10",
|
|
72
|
+
"playwright-core": "1.64.1",
|
|
73
|
+
"react": "19.3.1",
|
|
74
|
+
"spryte": "0.11.0",
|
|
75
|
+
"typescript": "5.9.3"
|
|
76
|
+
},
|
|
77
|
+
"engines": {
|
|
78
|
+
"node": ">=22.18"
|
|
79
|
+
},
|
|
80
|
+
"homepage": "https://debuff.jp/decide/",
|
|
81
|
+
"bugs": "https://github.com/debuffjp/decidekit/issues",
|
|
82
|
+
"bin": {
|
|
83
|
+
"decidekit": "./dist/cli.js"
|
|
84
|
+
},
|
|
85
|
+
"peerDependencies": {
|
|
86
|
+
"react": ">=18"
|
|
87
|
+
},
|
|
88
|
+
"peerDependenciesMeta": {
|
|
89
|
+
"react": {
|
|
90
|
+
"optional": true
|
|
91
|
+
}
|
|
92
|
+
}
|
|
93
|
+
}
|