decidekit 0.0.0-stage → 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,56 @@
+# Changelog
+
+## 0.1.0 (2026-11-02)
+
+The first release. Numbers are on public test splits unless noted; the scripts that produced them are in `eval/`.
+
+### Deciding
+
+- `createDecider()` picks one of your labels for a text, in Node or the browser, with `confidence` and `probabilities`. Labels
+  are the type: `choice` has the literal type of the labels you pass.
+- Zero-shot from label names, or from described labels (`{ card_arrival: "My card hasn't arrived." }`), which score by
+  the better of name and description. Zero-shot decisions report `similarity`, the best cosine similarity to any
+  option, which is low for off-topic text.
+- `minConfidence` returns `null` below a threshold. `decideMany()` for batches.
+- Default encoder: `Xenova/bge-small-en-v1.5` at 8-bit (34 MB). Any transformers.js feature-extraction model can be
+  passed as `model`.
+- Supported encoders with measured settings: `Xenova/bge-small-en-v1.5` (default) and `Xenova/multilingual-e5-small`.
+  The temperature fit now searches 0.01 to 10, so encoders with close similarities calibrate.
+- `describeLabels()` asks an LLM once to write label descriptions (7 to 10 points better zero-shot on the dev splits).
+
+### Training
+
+- `train()` fits one softmax layer on frozen embeddings: 90.7% on BANKING77 and 94.4% on CLINC150 with all examples,
+  82.3% and 89.6% with 5 per label. HWU64, never used for tuning: 90.2%.
+- Calibrated: the temperature is fitted on held-out examples (5 folds for small sets), with a per-model floor for
+  small heads. Calibration error 0.01 to 0.03 for heads with 10 or more examples per label.
+- `labels` adds labels without examples yet (they start from their names). `calibrateOn` fits the temperature on
+  examples known to be right, for training data from a source that is sometimes wrong.
+- `saveHead()` and `loadHead()`. Ready-made heads for BANKING77 and CLINC150.
+
+### Learning from an LLM
+
+- `withFallback()` answers locally when confident and calls your `ask` otherwise; each answer becomes a training
+  example and `retrainEvery` retrains in the background. It can start from labels alone.
+- Built for an LLM that is sometimes wrong: starting from labels, it fits confidence on zero-shot's own confident
+  answers rather than the LLM's, and learns `null` answers as the label `NONE` (`learnNone`) when the text is far
+  from every label (`noneBelow`), so it can answer `null` itself for off-topic text.
+- `route.saved()` returns the learned answers and the calibration set as plain JSON; pass them back as `restore` after
+  a reload. With real Llama 3.1 8B answers on
+  12 labels, the share sent fell from 30% (zero-shot) to 19%, and off-topic text given a wrong label from 6.7% to 2.4%.
+
+### Browser and tools
+
+- `decidekit/typesafe`: `typesafeClient()` gives a decider the shape of TypeSafe's SDK (`systemOne({ state, questions })`,
+  `choice()`, `probabilities`, their `confidence` formula). Choice questions only.
+- `decidekit/browser`: `createWorkerDecider()` runs the encoder in a Web Worker (training BANKING77 froze the page for
+  171 s on the main thread; the longest stall in a worker was 18 ms).
+- `decidekit/react`: `useDecider()` and `useDecision()`.
+- `decidekit/switch`: a decider for the switch agent harness.
+- A CLI: `decidekit train` and `decidekit decide`.
+
+### Known limits
+
+- The default encoder reads English. `Xenova/multilingual-e5-small` is supported for other languages (76% on Japanese
+  and Chinese MASSIVE intents, calibrated), but `withFallback`'s off-topic gate doesn't fire with it.
+- Weak at tone: 61% on tweet sentiment and 49% on emotion with 32 examples per label. See the FAQ.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 debuff contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,254 @@
-# Temporary Holding Version
+# decidekit
+
+[![npm](https://img.shields.io/npm/v/decidekit?color=cffb2d&labelColor=11140d)](https://www.npmjs.com/package/decidekit) [![ci](https://github.com/debuffjp/decidekit/actions/workflows/ci.yml/badge.svg)](https://github.com/debuffjp/decidekit/actions/workflows/ci.yml) [![MIT](https://img.shields.io/badge/licence-MIT-cffb2d?labelColor=11140d)](LICENSE) [![Discord](https://img.shields.io/badge/discord-debuff-a78bfa?labelColor=11140d)](https://discord.gg/jshbDHSpCY)
+
+**[Try it in your browser](https://debuff.jp/decidekit/playground/)** · [Watch it learn from an LLM](https://debuff.jp/decidekit/lanes/) · [Launch post](https://debuff.jp/blog/introducing-decidekit) · [Research: when the LLM is wrong](https://debuff.jp/research/learning-from-a-noisy-llm)
+
+A decision model for the browser and Node: it picks one of your labels for a piece of text, with confidence you can trust, and learns from an LLM when it isn't sure. Your labels are the type. No server, no API key.
+
+A System One model in the browser: like TypeSafe's Jev, but on the user's device, so each decision is free, private and a few milliseconds. Jev leaves unsure cases to your code; decidekit can ask an LLM about them and learn from the answers. decidekit is an independent project, not affiliated with TypeSafe.
+
+```ts
+import { createDecider } from "decidekit";
+
+const d = await createDecider();
+const r = await d.decide("my card never arrived", ["card_arrival", "refund", "other"]);
+r.choice; // "card_arrival" | "refund" | "other"
+```
+
+Give it 5 examples per label and it reaches 82% on BANKING77 (77 intents); give it the full training set and it
+reaches 90.7%, with each answer taking about 8 ms (median) in Chrome. Training runs in the user's tab, so their data never
+leaves the device.
+
+## Zero-shot
+
+Pass labels, or labels with descriptions. Labels like `card_arrival` are read as "card arrival".
+
+```ts
+await d.decide(text, ["question", "bug_report", "praise"]);
+await d.decide(text, { refund: "wants their money back", bug: "something is broken" });
+```
+
+Zero-shot is for trying things out: 61% on BANKING77 and 70% on CLINC150 (dev splits), against 90-94% trained.
+Its confidence came out close to calibrated on those datasets (error 0.005-0.045 with 2 to 150 options), but check
+it on your own labels before relying on a threshold.
+
+## Train a head
+
+A head is a small layer trained on your examples: 156 KB for 77 labels. Train it once and ship it as a static
+file, or train it in the page.
+
+```ts
+import { createDecider, saveHead, loadHead } from "decidekit";
+
+const d = await createDecider();
+const head = await d.train(examples); // { text, label }[]
+await writeFile("intent.head.json", saveHead(head));
+
+// later, in the app
+const r = await d.decide(text, loadHead(json)); // choice is typed as your labels
+const rs = await d.decideMany(texts, head); // many texts, encoded in batches
+```
+
+Each label starts at the average of its examples plus its own name, so a handful per label already works. If your
+labels are opaque ids (`c17`), pass `{ labelNames: false }`. Training then fits a temperature on held-out examples,
+so `confidence` is calibrated: of the answers given at 0.9, about 90% are right.
+
+Or train from the command line and ship the file:
+
+```sh
+npx decidekit train examples.csv -o intent.head.json   # CSV with text,label columns, or JSONL
+npx decidekit decide intent.head.json "my card never arrived"
+```
+
+Add `--types` to also write `intent.head.labels.ts`, so `loadHead<Label>(json)` types the choice as your labels.
+
+## "None of these"
+
+```ts
+const r = await d.decide(text, head, { minConfidence: 0.9 });
+r.choice; // "card_arrival" | "refund" | ... | null
+```
+
+Below the threshold, `choice` is `null`. Use it to send unclear messages to a person or a bigger model. It needs
+enough examples to mean something: with 3 examples per label, a head still gave "hello" 0.89. The numbers below come
+from CLINC150's 100 examples per label. With fewer, check the threshold on some off-topic text of your own, or add an
+`other` label trained on it.
+
+## Fall back to an LLM
+
+```ts
+const route = withFallback(d, head, { minConfidence: 0.9, ask: (text, labels) => askMyLlm(text, labels), examples });
+const r = await route(text); // r.source: "local" | "fallback"
+```
+
+Below the threshold, `ask` answers and its answer becomes a training example. Simulated on CLINC150 from 5 examples
+per label with an LLM that is always right, the share sent to the LLM fell from 41% to 14-17% over 10,000 queries
+while the head went from 86% to 94%. With real Llama 3.1 8B answers, starting from 12 described labels alone, it fell
+from 30% to 19%.
+
+Real LLMs are sometimes wrong, and `withFallback()` is built for that: it sets its confidence on its own sure answers
+rather than the LLM's, and learns the LLM's "none of these" (`null`) as a label, `NONE`, so it can say `null` itself
+for off-topic text. Details and numbers: [docs/guide-fallback.md](docs/guide-fallback.md).
+
+## Coming from Jev
+
+`decidekit/typesafe` has the shape of TypeSafe's SDK, for Choice questions:
+
+```ts
+import { choice, typesafeClient } from "decidekit/typesafe";
+
+const client = typesafeClient(await createDecider());
+const { answers } = await client.systemOne({
+  state: "I was charged twice",
+  questions: { category: choice("What is this about?", { billing: null, technical: null, other: null }) },
+});
+```
+
+The [guide](docs/guide-typesafe.md) covers what differs, including what `confidence` means.
+
+## switch
+
+`decidekit/switch` makes decidekit a decision model for [switch](https://github.com/osolmaz/switch), the harness that
+runs a decision model through a task. Zero-shot, it won two of the three WikiRace races from switch's example on a
+laptop CPU. See the [switch guide](docs/guide-switch.md).
+
+## React
+
+`useDecider()` and `useDecision(decider, text, labels)` from `decidekit/react`: [docs/guide-react.md](docs/guide-react.md).
+
+## In the browser
+
+Use the worker version. The encoder blocks whatever thread it runs on (we measured a 171 s freeze while training
+on the main thread; in a worker the page's longest stall was 18 ms).
+
+```ts
+import { createWorkerDecider, loadHead } from "decidekit/browser";
+
+const head = loadHead(await (await fetch("/intent.head.json")).text());
+const d = await createWorkerDecider({
+  onProgress: ({ file, loaded, total }) => showProgress(file, loaded / total), // first visit only
+});
+const r = await d.decide(text, head);
+```
+
+Import `loadHead` from `decidekit/browser` too, not from `decidekit`: that keeps the encoder out of your page bundle.
+
+Tested from npm in Vite 7, webpack 5 and Next.js 16 apps; see [docs/bundlers.md](docs/bundlers.md) for the config, including a one-line alias
+that halves the wasm download. Serve the page cross-origin isolated if you can:
+
+```
+Cross-Origin-Opener-Policy: same-origin
+Cross-Origin-Embedder-Policy: credentialless
+```
+
+That lets wasm use several threads: in our runs with all-MiniLM-L6-v2 it cut p95 decision time from 14.3 ms to 7.8 ms. With the default bge-small, isolated, p95 is 13.8 ms.
+
+Download on first visit: the model (34 MB) and onnxruntime's wasm (23.6 MB, 5.8 MB gzipped), both cached after.
+
+## Models
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+decidekit runs the Decide models, each a sentence encoder plus a calibrated head:
+
+| Model | Encoder | Download | Reads |
+|---|---|---|---|
+| Decide Small (default) | `Xenova/bge-small-en-v1.5` | 34 MB | English |
+| Decide Multilingual | `Xenova/multilingual-e5-small` | 118 MB | 76% on Japanese and Chinese intents |
+
+Pass the encoder id as `model` to `createDecider()`.
+
+## Numbers
+
+bge-small-en-v1.5 (the default) at q8, Ryzen 7 260 laptop, Node 24 and headless Chrome 156, 2026-11-01. Accuracy
+and calibration error (ECE, lower is better) on each dataset's test split, mean of 3 seeds (one run for "all"):
+[eval/results-bge-small.json](eval/results-bge-small.json).
+
+| Examples per label | 5 | 10 | 25 | all |
+|---|---|---|---|---|
+| BANKING77 (77 labels) | 82.3% | 83.8% | 88.7% | 90.7% |
+| CLINC150 (150 labels) | 89.6% | 92.0% | 93.4% | 94.4% |
+| ECE, BANKING77 | 0.071 | 0.029 | 0.011 | 0.017 |
+| ECE, CLINC150 | 0.028 | 0.010 | 0.015 | 0.008 |
+
+Pre-release builds defaulted to all-MiniLM-L6-v2 (23.7 MB, 2026-10-31, [eval/results.json](eval/results.json)):
+
+| Examples per label | 5 | 10 | 25 | all |
+|---|---|---|---|---|
+| BANKING77 (77 labels) | 78.7% | 83.0% | 87.4% | 90.6% |
+| CLINC150 (150 labels) | 87.3% | 89.8% | 92.8% | 94.0% |
+| ECE, BANKING77 | 0.065 | 0.048 | 0.025 | 0.019 |
+| ECE, CLINC150 | 0.029 | 0.013 | 0.012 | 0.012 |
+
+Calibration is loosest at 5 examples per label on BANKING77. On small tasks (2 to 10 labels sampled from CLINC150, scored on its validation
+queries, [eval/tiny.ts](eval/tiny.ts), measured with all-MiniLM-L6-v2), 1 example per label was right 93% of the time with calibration error 0.020, and
+3 per label 98% with 0.016. Add examples before relying on
+a threshold there.
+
+**In Chrome** (BANKING77, full training set, worker, cross-origin isolated, 16 threads): 7.7 ms median and 13.8 ms
+p95 per decision, 228 s to train on 10,003 examples with no page stall longer than 18 ms. all-MiniLM-L6-v2 in the
+same run: 4.2 ms median, 7.7 ms p95. In Node: 3.2 ms median for a single text, 1.1 ms per text batched (MiniLM 2.0
+and 0.7). Earlier MiniLM run: [bench/results.json](bench/results.json).
+
+**Text that fits no label:** CLINC150 has 1,000 test queries that match none of its 150 intents. Trained on
+in-scope examples only, with `minConfidence` 0.93 picked on the validation split, the shipped bge-small head returns
+`null` for 89.9% of them. With all-MiniLM-L6-v2 that was 86.3%, and 82.9% of in-scope queries were answered and right
+(94.0% right with no threshold).
+
+**The smaller model.** `createDecider({ model: "Xenova/all-MiniLM-L6-v2" })` (23.7 MB, the pre-release default) is
+about 1.8x faster per decision in Chrome and 3 to 5 points less accurate with few examples, about the same with all
+the data. Pass it to keep using heads trained on it. Before this default changed, the README suggested
+`{ labelNames: false }` with bge-small: label names slightly hurt it on the dev splits.
+
+Reproduce: `npm run eval:suite` (Node), `npm run bench` then `npm run bench:run` (Chrome).
+
+## Playground
+
+Try it at [debuff.jp/decidekit](https://debuff.jp/decidekit/). To run it locally: `npm run playground`, then serve `playground/dist` cross-origin isolated (`node bench/serve.mjs` style headers).
+Try zero-shot labels, the ready-made heads, a head trained in the page and "none of these".
+[Two lanes](https://debuff.jp/decidekit/lanes/) shows `withFallback` live: decidekit starts from twelve label names,
+asks a small LLM when it's unsure, learns from each answer, and replays the first 20 messages at the end.
+
+## Ready-made heads
+
+[heads/](heads/) has a BANKING77 support-intent head (90.6%) and a CLINC150 assistant-intent head (95.0%), so you
+can try a trained head before collecting examples.
+
+## What it's for
+
+Routing support messages, triaging feedback, search and command intent, moderation hints while someone types,
+labelling a dataset in the browser, a cheap first pass before an LLM that learns from it, games, extensions
+and offline apps, and text that can't leave the device. Details, code and poor fits:
+[docs/use-cases.md](docs/use-cases.md). Runnable examples are in [examples/](examples/).
+
+## Compared with
+
+| | Runs in the browser | Trains in the browser | Typed result | Calibrated confidence |
+|---|---|---|---|---|
+| decidekit | yes (worker) | yes | yes | trained heads |
+| [jimothy](https://www.npmjs.com/package/jimothy) | inference (worker) | no, Node | yes | yes |
+| [Jev](https://typesafe.ai/) (TypeSafe) | no, API | no | yes | yes |
+| transformers.js zero-shot pipeline | yes | no | no | no |
+| MediaPipe text classifier | yes | no, offline | no | no |
+| LLM structured output (AI SDK `Output.choice`) | no, API | n/a | yes | no |
+
+jimothy reports 92.4% on BANKING77, above our 90.7%. Pick it if you train in Node and want the higher number. Pick
+decidekit if training has to happen in the page, from the user's own data.
+
+[system-one](https://github.com/lukeramsden/system-one) is a different kind of tool: a typed client for hosted System One models (Jev and Laya). It runs no model itself.
+
+## Docs
+
+[docs/README.md](docs/README.md) indexes the guides (training, confidence and thresholds, the browser), the
+API reference, how it works and the FAQ.
+
+## Community
+
+Questions and show-and-tell in [Discussions](https://github.com/debuffjp/decidekit/discussions), chat in the
+[debuff Discord](https://discord.gg/jshbDHSpCY). Results on your own data, good or bad, help most: there's an
+issue template for them. See [CONTRIBUTING.md](CONTRIBUTING.md).
+
+## Status
+
+0.1. English with the default model, other languages with `multilingual-e5-small` ([FAQ](docs/faq.md#which-languages-does-it-support)); short texts (about a paragraph). Not done yet: WebGPU in the worker and fine-tuning the encoder (the likely route past about 91%).
+
+Made by the [debuff](https://debuff.jp) team. MIT.

package/dist/browser.d.ts

@@ -0,0 +1,25 @@
+export { saveHead, loadHead } from "./head-io.ts";
+export { withFallback, NONE, type FallbackOptions, type RoutedDecision, type Trainable } from "./fallback.ts";
+export type { Choices, DecideOptions, DeciderOptions, Decision, Example, Head, TrainOptions } from "./index.ts";
+import type { Choices, DecideOptions, DeciderOptions, Decision, Example, Head, TrainOptions } from "./index.ts";
+/**
+ * `createDecider()` in a Web Worker: same `decide()`, `decideMany()` and `train()`, plus `terminate()`. Each call costs about 1 ms
+ * of message passing. Serve the page cross-origin isolated (COOP `same-origin`, COEP `credentialless`) so wasm
+ * can use several threads. Pass your own `worker` if your bundler needs a different worker URL.
+ */
+export declare function createWorkerDecider({ onProgress, ...options }?: DeciderOptions, worker?: Worker): Promise<{
+    decide: {
+        <const L extends string>(text: string, using: Choices<L>, options?: {
+            minConfidence?: undefined;
+        }): Promise<Decision<L>>;
+        <const L extends string>(text: string, using: Choices<L>, options: DecideOptions): Promise<Decision<L, L | null>>;
+    };
+    decideMany: {
+        <const L extends string>(texts: readonly string[], using: Choices<L>, options?: {
+            minConfidence?: undefined;
+        }): Promise<Decision<L>[]>;
+        <const L extends string>(texts: readonly string[], using: Choices<L>, options: DecideOptions): Promise<Decision<L, L | null>[]>;
+    };
+    train: <const L extends string>(examples: readonly Example<L>[], { onEpoch, ...opts }?: TrainOptions<L>) => Promise<Head<L>>;
+    terminate: () => void;
+}>;

package/dist/browser.js

@@ -0,0 +1,43 @@
+// The same decide() and train() as createDecider, running in a Web Worker so the page never freezes.
+export { saveHead, loadHead } from "./head-io.js";
+export { withFallback, NONE } from "./fallback.js";
+/**
+ * `createDecider()` in a Web Worker: same `decide()`, `decideMany()` and `train()`, plus `terminate()`. Each call costs about 1 ms
+ * of message passing. Serve the page cross-origin isolated (COOP `same-origin`, COEP `credentialless`) so wasm
+ * can use several threads. Pass your own `worker` if your bundler needs a different worker URL.
+ */
+export async function createWorkerDecider({ onProgress, ...options } = {}, worker = new Worker(new URL("./worker.js", import.meta.url), { type: "module" })) {
+    let next = 0;
+    const pending = new Map();
+    worker.onmessage = ({ data }) => {
+        const p = pending.get(data.id);
+        if (!p)
+            return;
+        if (data.progress)
+            return p.progress?.(data.progress);
+        pending.delete(data.id);
+        data.error ? p.reject(new Error(data.error)) : p.resolve(data.value);
+    };
+    const call = (op, payload, progress) => new Promise((resolve, reject) => {
+        const id = next++;
+        pending.set(id, { resolve, reject, progress });
+        worker.postMessage({ id, op, ...payload });
+    });
+    await call("init", { options }, onProgress);
+    function decide(text, using, opts) {
+        return call("decide", { args: [text, using, opts] });
+    }
+    function decideMany(texts, using, opts) {
+        return call("decideMany", { args: [texts, using, opts] });
+    }
+    return {
+        decide, decideMany,
+        train: (examples, { onEpoch, ...opts } = {}) => call("train", { args: [examples, opts] }, (p) => onEpoch?.(p.epoch, p.accuracy)),
+        terminate: () => {
+            worker.terminate();
+            for (const p of pending.values())
+                p.reject(new Error("decidekit: the worker was terminated"));
+            pending.clear();
+        },
+    };
+}

package/dist/cli.d.ts

@@ -0,0 +1,5 @@
+#!/usr/bin/env node
+import { type Example } from "./index.ts";
+export declare function readExamples(file: string): Promise<Example<string>[]>;
+/** A module with the head's labels, so code that loads the head file can type it: `loadHead<Label>(json)`. */
+export declare function labelsModule(labels: readonly string[]): string;

package/dist/cli.js

@@ -0,0 +1,114 @@
+#!/usr/bin/env node
+// decidekit train <examples.csv|.jsonl> [-o head.json] [--model id] [--no-label-names] [--types]
+// decidekit decide <head.json> "<text>" [--min-confidence 0.9]
+import { readFile, writeFile } from "node:fs/promises";
+import { parseArgs } from "node:util";
+import { createDecider, loadHead, saveHead } from "./index.js";
+const USAGE = `usage:
+  decidekit train <examples.csv|.jsonl> [-o head.json] [--model <hf-id>] [--no-label-names] [--types]
+  decidekit decide <head.json> "<text>" [--min-confidence <0-1>]
+
+--types also writes head.labels.ts with the labels as a const array and a union type.
+
+CSV needs a header with "text" and "label" columns. JSONL needs {"text", "label"} per line.`;
+function parseCsv(source) {
+    const rows = [];
+    let row = [], field = "", quoted = false;
+    for (let i = 0; i < source.length; i++) {
+        const ch = source[i];
+        if (quoted) {
+            if (ch === '"' && source[i + 1] === '"') {
+                field += '"';
+                i++;
+            }
+            else if (ch === '"')
+                quoted = false;
+            else
+                field += ch;
+        }
+        else if (ch === '"')
+            quoted = true;
+        else if (ch === ",") {
+            row.push(field);
+            field = "";
+        }
+        else if (ch === "\n") {
+            row.push(field);
+            rows.push(row);
+            row = [];
+            field = "";
+        }
+        else if (ch !== "\r")
+            field += ch;
+    }
+    if (field || row.length) {
+        row.push(field);
+        rows.push(row);
+    }
+    return rows.filter((r) => r.some((f) => f !== ""));
+}
+export async function readExamples(file) {
+    const source = await readFile(file, "utf8");
+    if (file.endsWith(".jsonl"))
+        return source.split("\n").filter((l) => l.trim()).map((l) => JSON.parse(l));
+    const [header, ...rows] = parseCsv(source);
+    const t = header.indexOf("text"), l = header.indexOf("label");
+    if (t < 0 || l < 0)
+        throw new Error(`${file}: the header needs "text" and "label" columns, found ${header.join(", ")}`);
+    return rows.map((r) => ({ text: r[t], label: r[l] }));
+}
+/** A module with the head's labels, so code that loads the head file can type it: `loadHead<Label>(json)`. */
+export function labelsModule(labels) {
+    return `// Written by decidekit train --types. Labels of the head next to this file.
+export const labels = ${JSON.stringify(labels)} as const;
+export type Label = (typeof labels)[number];
+`;
+}
+async function main(argv) {
+    const { values, positionals } = parseArgs({
+        args: argv, allowPositionals: true,
+        options: {
+            out: { type: "string", short: "o" }, model: { type: "string" }, "no-label-names": { type: "boolean" }, types: { type: "boolean" },
+            "min-confidence": { type: "string" }, help: { type: "boolean", short: "h" },
+        },
+    });
+    const [command, file, text] = positionals;
+    if (values.help || !command)
+        return console.log(USAGE);
+    if (command === "train" && file) {
+        const examples = await readExamples(file);
+        const counts = new Map();
+        for (const e of examples)
+            counts.set(e.label, (counts.get(e.label) ?? 0) + 1);
+        const fewest = Math.min(...counts.values());
+        console.error(`${examples.length} examples, ${counts.size} labels, fewest per label: ${fewest}`);
+        if (fewest < 3)
+            console.error("warning: labels with fewer than 3 examples will be unreliable");
+        const d = await createDecider({ model: values.model });
+        const t0 = performance.now();
+        const head = await d.train(examples, { labelNames: !values["no-label-names"] });
+        const out = values.out ?? "head.json";
+        const json = saveHead(head);
+        await writeFile(out, json);
+        if (values.types) {
+            const ts = out.replace(/\.json$/, "") + ".labels.ts";
+            await writeFile(ts, labelsModule(head.labels));
+            console.error(`wrote ${ts}`);
+        }
+        console.error(`wrote ${out} (${Math.round(json.length / 1024)} KB) in ${((performance.now() - t0) / 1000).toFixed(1)} s, temperature ${head.temperature}`);
+        return;
+    }
+    if (command === "decide" && file && text !== undefined) {
+        const head = loadHead(await readFile(file, "utf8"));
+        const d = await createDecider({ model: head.model });
+        const min = values["min-confidence"];
+        if (min !== undefined && !(Number(min) >= 0 && Number(min) <= 1))
+            throw new Error(`--min-confidence needs a number from 0 to 1, got ${min}`);
+        const r = min === undefined ? await d.decide(text, head) : await d.decide(text, head, { minConfidence: Number(min) });
+        console.log(JSON.stringify({ choice: r.choice, confidence: Number(r.confidence.toFixed(4)) }));
+        return;
+    }
+    console.error(USAGE);
+    process.exitCode = 2;
+}
+await main(process.argv.slice(2)).catch((e) => { console.error(String(e?.message ?? e)); process.exitCode = 1; });

package/dist/describe.d.ts

@@ -0,0 +1,14 @@
+export interface DescribeOptions {
+    /** What the texts are, so descriptions match them. For example "messages to a bank's support chat". */
+    context?: string;
+}
+/** The prompt `describeLabels()` sends. Exported so you can run it yourself and pass the reply to `parseDescriptions()`. */
+export declare function describePrompt(labels: readonly string[], { context }?: DescribeOptions): string;
+/** Reads an LLM's reply to `describePrompt()`. Labels the reply missed, or answered with anything but a string, are left out. */
+export declare function parseDescriptions<const L extends string>(labels: readonly L[], reply: string): Partial<Record<L, string>>;
+/**
+ * Asks an LLM once for a one-sentence description of each label and returns a record `decide()` takes as its
+ * options. A label the LLM skipped gets its own name, so every label is present. Save the result and reuse it;
+ * there's no need to call this on every load.
+ */
+export declare function describeLabels<const L extends string>(labels: readonly L[], complete: (prompt: string) => string | Promise<string>, options?: DescribeOptions): Promise<Record<L, string>>;

package/dist/describe.js

@@ -0,0 +1,30 @@
+// Label descriptions written once by an LLM. Zero-shot decide() with a one-sentence description per label, in the
+// user's voice, was 7 to 10 points more accurate than label names on two intent datasets (eval/NOTES-accuracy.md).
+/** The prompt `describeLabels()` sends. Exported so you can run it yourself and pass the reply to `parseDescriptions()`. */
+export function describePrompt(labels, { context } = {}) {
+    return [
+        `Write one short sentence for each label below, phrased as a typical ${context ?? "message"} that belongs under it, in the voice of the person writing it.`,
+        "Don't repeat the label's words if a more natural phrasing exists. Reply with only a JSON object mapping each label to its sentence.",
+        "",
+        ...labels.map((l) => `- ${l}`),
+    ].join("\n");
+}
+/** Reads an LLM's reply to `describePrompt()`. Labels the reply missed, or answered with anything but a string, are left out. */
+export function parseDescriptions(labels, reply) {
+    const json = reply.slice(reply.indexOf("{"), reply.lastIndexOf("}") + 1);
+    let parsed = {};
+    try {
+        parsed = JSON.parse(json);
+    }
+    catch { }
+    return Object.fromEntries(labels.filter((l) => typeof parsed[l] === "string" && parsed[l].trim()).map((l) => [l, parsed[l].trim()]));
+}
+/**
+ * Asks an LLM once for a one-sentence description of each label and returns a record `decide()` takes as its
+ * options. A label the LLM skipped gets its own name, so every label is present. Save the result and reuse it;
+ * there's no need to call this on every load.
+ */
+export async function describeLabels(labels, complete, options) {
+    const got = parseDescriptions(labels, await complete(describePrompt(labels, options)));
+    return Object.fromEntries(labels.map((l) => [l, got[l] ?? l.replace(/[_-]+/g, " ")]));
+}

package/dist/fallback.d.ts

@@ -0,0 +1,75 @@
+import type { Choices, Decision, Example, Head, TrainOptions } from "./index.ts";
+/** What `withFallback()` needs from a decider: `createDecider()` and `createWorkerDecider()` both fit. */
+export interface Trainable {
+    decide<const L extends string>(text: string, using: Choices<L>): Promise<Decision<L>>;
+    train<const L extends string>(examples: readonly Example<L>[], options?: TrainOptions<L>): Promise<Head<L>>;
+}
+export interface FallbackOptions<L extends string> {
+    /** Below this confidence, call `ask`. Default 0.9. */
+    minConfidence?: number;
+    /**
+     * Answers the texts the local decision isn't sure about: return one of `labels`, or `null` for none of them.
+     * Answers that aren't one of the labels count as `null`.
+     */
+    ask: (text: string, labels: readonly L[], local: Decision<L>) => L | null | Promise<L | null>;
+    /**
+     * The examples a trained head was trained on. Pass them (or start from labels) to learn from `ask`: each answer
+     * becomes an example, and `retrain()` trains on both.
+     */
+    examples?: readonly Example<L>[];
+    /**
+     * Learn `ask`'s `null` answers ("none of these") as a label of their own, `NONE`, so the head learns what off-topic
+     * text looks like and answers `null` itself when it is sure. Default true. On CLINC150 with 4.5% off-topic traffic, it
+     * halved the off-topic texts given a confident wrong label (7.6% to 3.8%) and answered 56% of them locally, at no
+     * cost on in-scope text (eval/none-label.ts).
+     */
+    learnNone?: boolean;
+    /**
+     * Learn a `null` answer as `NONE` only when the text's zero-shot `similarity` to every label is below this. LLMs also
+     * answer `null` for ambiguous texts that do belong to a label, and those sit close to the labels. Default 0.6, set
+     * for bge-small; `0` turns the check off. With Llama 3.1 8B on 12 BANKING77 labels it cut sends (20.7% to 19.1%) and
+     * raised in-scope answers right locally (81.3% to 83.3%) against learning every `null` (eval/real-stream.ts).
+     */
+    noneBelow?: number;
+    /** Retrain on its own after this many new answers. Default: only when you call `retrain()`. */
+    retrainEvery?: number;
+    /**
+     * Passed to `train()` when retraining. Starting from labels, `calibrateOn` defaults to the texts zero-shot answered
+     * locally (up to 4 per label, or 50 divided by the number of labels if that is more, once there are 50): they fit the head's temperature better than the LLM's answers,
+     * which are sometimes wrong. Pass your own `calibrateOn` to replace them.
+     */
+    trainOptions?: Omit<TrainOptions<L>, "labels">;
+    /**
+     * What an earlier session learned, from `route.saved()`, so a reload keeps it. Restoring doesn't retrain: call
+     * `retrain()` or pass the saved head as `using`.
+     */
+    restore?: {
+        learned?: readonly Example<L>[];
+        confident?: readonly Example<L>[];
+    };
+}
+/** A decision plus where it came from. `confidence` and `probabilities` are always the local decision's. */
+export interface RoutedDecision<L extends string> extends Decision<L, L | null> {
+    source: "local" | "fallback";
+}
+/** The label `withFallback()` learns `ask`'s `null` answers as. It shows in `route.head.labels` and `route.learned`. */
+export declare const NONE = "none_of_these";
+/**
+ * Routes each text to the local decision, or to `ask` when its confidence is below `minConfidence`.
+ *
+ * - With a trained head and its `examples`, answers from `ask` are kept as new examples.
+ * - With labels (`["refund", "bug"]` or described labels), it starts zero-shot and always learns: `retrain()` (or
+ *   `retrainEvery`) trains the first head from the answers. Labels with no answers yet start from their names.
+ */
+export declare function withFallback<const L extends string>(decider: Trainable, using: Choices<L>, options: FallbackOptions<L>): ((text: string) => Promise<RoutedDecision<L>>) & {
+    retrain: () => Promise<Head<L>>;
+    /** The trained head in use, or `null` while still zero-shot. Replaced after each retrain. */
+    readonly head: Head<L> | null;
+    /** Answers from `ask` kept as examples since the start; `null` answers appear with the label `NONE`. */
+    readonly learned: readonly Example<L>[];
+    /** Everything to pass back as `restore` after a reload: the learned answers and the calibration set. Plain JSON. */
+    saved(): {
+        learned: Example<L>[];
+        confident: Example<L>[];
+    };
+};