@tuned-tensor/cli 0.20.0 → 0.22.0

package/docs/local-runtime/architecture.md

@@ -44,7 +44,7 @@ The local adapter runtime excludes:
 The next model or method should arrive with its own locked dependencies,
 resource defaults, data contract, and real CUDA acceptance test.
 
-TT bundles two local training methods:
+TT bundles three local training methods:
 
 - **Adapter** (`training/adapter`): certified Transformers checkpoints and
   PEFT LoRA SFT. Driven by `tt pipeline run`.
@@ -55,6 +55,11 @@ TT bundles two local training methods:
   rejects `nproc_per_node` values above 1 and unsupported steps such as
   `compare` before creating run artifacts.
 
+- **Decision** (`training/decision`): fine-tunes a pinned typed decision
+  model (`convaiinnovations/laya`) on labelled examples. Driven by
+  `tt pipeline run` from a decision `tunedtensor.json`. Runs on CPU, Apple MPS
+  or CUDA; see [decision.md](decision.md).
+
 The foundation engine is a readable single-GPU trainer inspired by nanochat,
 not a port of nanochat's distributed or compute-optimal training stack. It can
 stream configured text/JSONL shards into a deterministic token cache, use a
@@ -92,6 +97,8 @@ Node owns input validation, state, process lifecycle, manifests, and reports.
 The bundled Python project owns four small operations:
 
 - `prefetch.py`: download and verify the pinned Hugging Face snapshot;
+- `prefetch_dataset.py`: download and verify the spec's Hugging Face dataset
+  files (`dataset_prebuilt.huggingface`);
 - `evaluate.py`: generate predictions without receiving reference answers;
 - `train.py`: perform CUDA-only, model-specific LoRA SFT;
 - `serve.py`: prepare the verified base model and adapter for the vLLM serving runtime.
@@ -120,6 +127,19 @@ Canonical JSON and JSONL files are the state store; there is no database or
 mirrored index. Writes are atomic and listings scan the small local metadata
 tree deterministically.
 
+Run events (`progress.jsonl` in the store's run directory) form a hash chain.
+Each event carries `seq`, `prev_hash` and `hash`, a SHA-256 over its canonical
+JSON. Appends take a short cross-process lock so concurrent writers, such as a
+cancel from another terminal, extend one chain. The lock records its owner's
+pid and host, so a crashed owner's lock is reclaimed at once and a live owner
+is never broken. Appends read only the log's tail. A line torn by a crash is
+closed off and skipped, and the audit counts it as unreadable. The completion event records
+the report's SHA-256 and the artifact manifest's SHA-256. The report embeds a
+`provenance` block (software, execution target, spec, base model, data
+digests, hyperparameters). `tt runs audit` verifies the chain and report
+digests and reports the first broken link. Events written by older versions
+have no hashes and are reported as legacy.
+
 Each real adapter has an atomic `artifact-manifest.json` with the expected PEFT
 files, byte sizes, and SHA-256 hashes. A model is registered immediately after
 training and manifest verification, so a later evaluation failure does not
@@ -134,6 +154,29 @@ The runner config controls:
 - CUDA or CPU evaluation, with local serving through the CUDA vLLM runtime;
 - deterministic generation and scoring limits.
 
+## Hugging Face downloads
+
+Model and dataset prefetch share `hub_download.py`. It resolves the requested
+revision to one commit before downloading, so retries never mix files from two
+commits. Network and server errors are retried with exponential backoff (2s up
+to 60s, `TT_HF_DOWNLOAD_MAX_ATTEMPTS` attempts, default 10). Repo, auth,
+revision and disk-full errors fail at once. Completed files stay in the cache,
+so a rerun after Ctrl-C or a crash downloads only what is missing. A file
+interrupted mid-transfer restarts from its beginning.
+
+Progress is measured from the cache blob directory against the repo's file
+sizes and printed as `@@tt-progress {json}` lines. The process runner sends
+them to the reporter instead of the log stream. A terminal shows an in-place
+bar with rate and ETA. Captured output gets a line every 10% or 30 seconds.
+The parent `tt` process sets `TT_PROGRESS_TTY` for the local runtime child,
+whose stderr is a pipe.
+
+Dataset runs resolve the cached snapshot in Node without network access.
+Each split file must link into the repo's blob store, and its digest must match
+the blob name. With `columns`, records are converted to chat rows under
+`<artifactRoot>/datasets/huggingface/`. The conversion path is keyed by the
+source digests, the mapping and the system message.
+
 The training project, Python entrypoints, working directory, and forwarded
 runtime settings are internal and fixed. Every stage uses the locked `uv`
 project included in the npm package. Locally, the mutable virtual environment

package/docs/local-runtime/aws-gpu.md

@@ -140,6 +140,8 @@ Run these commands on your laptop in the project directory. Adapter workflow:
 # Fetch the base model locally first.
 tt validate tunedtensor.json
 tt models prefetch tunedtensor.json
+# Only when dataset_prebuilt.huggingface is set:
+tt datasets prefetch tunedtensor.json
 tt doctor tunedtensor.json
 tt pipeline run --spec tunedtensor.json --dry-run
 tt pipeline run --spec tunedtensor.json

package/docs/local-runtime/decision.md

@@ -0,0 +1,142 @@
+# Decision models
+
+A decision model answers one typed question about an input — which queue,
+yes or no, how urgent — in a single encoder forward pass. It returns a
+probability for every allowed answer instead of generating text, so the output
+always parses and its confidence can be thresholded. Use the decision engine
+for classification, routing, triage, guardrails and score gates where a
+generative LLM is more than the job needs.
+
+TT fine-tunes [Laya](https://huggingface.co/convaiinnovations/laya)
+(`convaiinnovations/laya`, Apache-2.0): a 421M-parameter ModernBERT-large
+encoder with a typed decision head. Every stage uses Laya revision
+`55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851`, the reviewed commit that Laya 0.3.22
+pins. Training and evaluation run on CPU, Apple MPS or CUDA; no GPU is required.
+
+## Spec
+
+```json
+{
+  "engine": "decision",
+  "name": "Card support triage",
+  "system_prompt": "Which card-support queue should handle this customer message?",
+  "guidelines": ["Choose the queue for the customer's main problem."],
+  "constraints": [],
+  "base_model": "convaiinnovations/laya",
+  "decision": {
+    "type": "choice",
+    "criteria": {
+      "lost_or_stolen": "customer lost their card or it was stolen",
+      "payment_declined": "a card payment was declined or rejected"
+    }
+  },
+  "examples": [
+    { "input": "I think I left my card at the bar last night.", "output": "lost_or_stolen" },
+    { "input": "Why was my payment at the grocery store refused?", "output": "payment_declined" }
+  ],
+  "hyperparameters": { "n_epochs": 3 }
+}
+```
+
+`tt init --engine decision` writes a starter spec. A complete example lives in
+[`examples/single-spec/decision`](../../examples/single-spec/decision/tunedtensor.json).
+
+`system_prompt`, `guidelines` and `constraints` compile into the question's
+instructions, as they compile into the system message for the other engines.
+The model reads roughly 190 tokens of instructions and option descriptions, so
+keep them short; validation warns above 600 characters.
+
+| `decision.type` | `criteria` | Example `output` |
+| --- | --- | --- |
+| `choice` | Object of label → description (2–32 labels, in option order) | A label, e.g. `payment_declined` |
+| `noul` | Optional `{ "true": "...", "false": "..." }` | `true` or `false` |
+| `score` | Array of level descriptions, level 0 first (2–10) | A level index, e.g. `2` |
+
+Outputs match labels ignoring case and surrounding spaces. Validation rejects
+an output that is not a label and warns when a label has no examples.
+
+Optional `hyperparameters`:
+
+| Field | Default | Meaning |
+| --- | --- | --- |
+| `n_epochs` | 3 | Passes over the training split |
+| `learning_rate` | 2e-5 | AdamW learning rate, linear warmup and decay |
+| `batch_size` | 8 | Examples per optimizer step |
+| `freeze_encoder` | `false` | Train only the decision head (faster, usually less accurate) |
+| `shuffle_options` | `true` | Shuffle option order per row so the head learns meaning, not position |
+| `seed` | 0 | Training seed |
+| `device` | `auto` | `auto`, `cpu`, `mps` or `cuda` |
+
+`runtime.artifactRoot` is supported. `runtime.gpu`, `evaluation` and adapter
+runtime paths are rejected: the model is small enough to run where the CLI runs.
+
+## Run
+
+```bash
+tt validate
+tt pipeline run --spec tunedtensor.json --dry-run
+tt pipeline run --spec tunedtensor.json
+```
+
+The pipeline is the adapter recipe: `baseline` evaluates the pinned base model,
+`train` fine-tunes it, `candidate` evaluates the tuned model and `compare`
+pairs the two. TT splits the spec examples once with a seed derived from the
+spec `id` (20% held out, at least one). Training sees only the training split.
+The evaluator receives held-out IDs and inputs only; Node joins its predictions
+to the expected labels and scores them.
+
+Each evaluation reports accuracy, log-loss and Brier score of the expected
+label, mean confidence and per-label recall. The comparison adds improved and
+regressed examples and the log-loss and Brier deltas. Because Laya answers
+zero-shot from the label descriptions, the baseline is a real model, not an
+empty one: the report shows what fine-tuning added on top of the descriptions.
+
+Runs are written to `.tuned-tensor/decision-runs/<run-id>` beside the spec (or
+`--output <dir>`), with owner-only permissions:
+
+```text
+report.json            summary: model, revision, split, every step's metrics
+resolved-workflow.json spec, plan, question and split used
+data/                  train.jsonl and unlabeled eval-inputs.jsonl
+baseline/  candidate/  config, log, predictions.jsonl, report.json
+train/model/           the tuned checkpoint
+compare/comparison.json
+```
+
+On a 4-core CPU a step of 8 examples takes about 15 seconds, so 120 examples
+for 3 epochs trains in about 12 minutes. The tuned checkpoint is about 1.7 GB
+because weights are saved in float32; small fine-tuning updates would be lost
+in bfloat16.
+
+## Use the tuned model
+
+`train/model` is a complete Laya checkpoint. The fitted temperatures of the
+base checkpoint are reset to 1.0, because they described the base model's
+logits; cross-entropy training is itself the calibration objective.
+
+```python
+import laya
+
+agent = laya.load("path/to/decision-runs/<run-id>/train/model")
+question = {
+    "type": "choice",
+    "instructions": "Which card-support queue should handle this customer message?",
+    "criteria": {"lost_or_stolen": "...", "payment_declined": "..."},
+}
+answer = agent.predict("My card was refused at the shop.", {"queue": question})["answers"]["queue"]
+print(answer["choice"], answer["probabilities"])
+```
+
+Pass the same instructions and criteria the model was trained with: they are
+part of its input. `resolved-workflow.json` records them under `question`.
+The tuned model is not registered with `tt models` and is not served through
+vLLM.
+
+## Limits
+
+- One question per spec. Laya can answer several questions per input; TT
+  trains and evaluates one.
+- `convaiinnovations/laya` is the English checkpoint. Its multilingual and
+  typed-decisions variants are not certified yet.
+- The held-out split comes from the spec examples. With few examples the
+  comparison is noisy; add examples before trusting a small delta.

package/docs/spec-architecture.html

@@ -73,7 +73,7 @@
     <div class="tt-level">
       <div class="tt-unit"><button class="btn" type="button" data-component="agent" aria-pressed="false">User intent → agent</button><small class="text-small">get_local_spec → field patch</small></div>
       <div class="tt-unit"><button class="btn" type="button" data-component="approval" aria-pressed="false">Diff → /approve</button><small class="text-small">validate → atomic save → history</small></div>
-      <div class="tt-unit"><button class="btn" type="button" data-component="review" aria-pressed="false">/spec review</button><small class="text-small">show · diff · validate · history</small></div>
+      <div class="tt-unit"><button class="btn" type="button" data-component="review" aria-pressed="false">/spec review</button><small class="text-small">overview · sections · show · diff · validate · history</small></div>
     </div>
     <div class="tt-arrow text-small">↕ inspect the recipe · save only reviewed edits</div>
     <div class="tt-recipe tt-unit">
@@ -182,7 +182,7 @@
       "src/local-runtime/local-project.ts",
       "src/pipeline.ts"
     ],
-    "example": "/spec\n/spec diff\n/spec validate\n/spec history\n\n# Outside the TT shell:\ntt spec show --json"
+    "example": "/spec\n/spec examples\n/spec diff\n/spec validate\n/spec history\n\n# Outside the TT shell:\ntt spec view\ntt spec show --json"
   },
   "config": {
     "title": "Spec runtime and evaluation",

package/docs/spec-workflow.md

@@ -15,19 +15,27 @@ Inside the TT shell, in your project directory:
 
 ```text
 /spec
+/spec examples
+/spec show
 /spec validate
 /spec diff
 /spec history
 ```
 
-`/spec` shows the complete current file, validation results and SHA-256.
+`/spec` shows a readable overview: name, engine, base model and example count,
+then highlighted sections for behavior (system prompt, guidelines, constraints),
+examples, training settings, evaluation, runtime and pipeline, followed by
+validation results. `/spec <section>` shows one section in full; sections are
+`identity`, `behavior`, `examples`, `training`, `evaluation`, `runtime` and
+`pipeline`. `/spec show` prints the complete file with JSON highlighting,
+validation results and SHA-256.
 `/spec diff` shows pending spec edits in the current conversation; otherwise it
 shows the latest approved edit, or external changes since that revision.
 `/spec history` lists up to 50 recent revision records. These commands require
 no credentials and work before the agent is configured.
 
-Outside the shell, use `tt spec show`, `tt spec validate`, `tt spec diff`, and
-`tt spec history`. Add a workspace-relative path, for example
+Outside the shell, use `tt spec view` (or `tt spec <section>`), `tt spec show`,
+`tt spec validate`, `tt spec diff`, and `tt spec history`. Add a workspace-relative path, for example
 `tt spec show feedback/tunedtensor.json`. For continued work on a nested
 project, `/cd feedback` makes its `tunedtensor.json` the current recipe.
 `--json` returns structured review results. `tt spec validate` exits nonzero
@@ -114,6 +122,13 @@ resolve beside the spec. New projects default to `.tuned-tensor/artifacts` and
 Foundation supports `runtime.artifactRoot` and `runtime.gpu`; its evaluation
 and training settings remain under `foundation`.
 
+`dataset_prebuilt.huggingface` names a Hugging Face dataset repo, an optional
+pinned `revision` and optional `columns` (`input`/`output` field names). Its
+split paths are then files inside that repo. `tt datasets prefetch` downloads
+them, and `tt pipeline run` downloads them automatically if they are missing.
+Runs record the resolved commit and file digests in the report. See the README
+section "Train on a Hugging Face dataset".
+
 Only `tunedtensor.json` is required. `tt pipeline init --spec tunedtensor.json`
 embeds an advanced recipe when needed; omit this step to keep automatic derivation.
 An explicit `--file` cannot override a conflicting embedded recipe.