@tuned-tensor/local 0.1.7 → 0.1.8

package/CHANGELOG.md

@@ -1,6 +1,27 @@
 # Changelog
 
-All notable changes to Tuned Tensor Local will be documented in this file.
+All notable changes to TT Local will be documented in this file.
+
+## 0.1.8 - 2026-07-02
+
+### Changed
+
+- Renamed the user-facing project language to TT Local with the tagline
+  "Tuning tensors locally", while keeping the `tt-local` CLI, npm package, repo,
+  and filesystem identifiers unchanged.
+- Reworked the README around a first-time user flow: install, safe dry run,
+  real training, config basics, evaluation behavior, artifacts, Spark notes,
+  and common commands.
+- Simplified evaluation configuration by removing `evaluation.mode`. Inference
+  is now selected with `evaluation.inference.provider`, and scoring is selected
+  with `evaluation.scoring.mode`.
+- Shared the uv/subprocess execution helper between training and evaluation so
+  logging, timeout, and stderr handling are consistent.
+
+### Added
+
+- Added direct support for `evaluation.inference.provider: "command"` with
+  `baselineCommand` and `candidateCommand` for custom inference adapters.
 
 ## 0.1.7 - 2026-07-02
 

package/README.md

@@ -1,62 +1,55 @@
-# Tuned Tensor Local
+# TT Local
+
+Tuning tensors locally.
 
 [![CI](https://github.com/tunedtensor/tuned-tensor-local/actions/workflows/ci.yml/badge.svg)](https://github.com/tunedtensor/tuned-tensor-local/actions/workflows/ci.yml)
 [![npm](https://img.shields.io/npm/v/@tuned-tensor/local)](https://www.npmjs.com/package/@tuned-tensor/local)
 
-Tuned Tensor Local is a standalone CLI for running Tuned Tensor-style fine-tuning
-jobs on a machine you control. It stores specs, datasets, model artifacts,
-progress, reports, and dashboard state on local disk.
+TT Local is a standalone CLI for running Tuned Tensor-style
+fine-tuning jobs on a machine you control. It stores specs, datasets, model
+artifacts, progress events, reports, and dashboard state on local disk.
 
-It does not require the hosted Tuned Tensor CLI, cloud orchestration, Docker, or
-a managed database.
+It does not require the hosted Tuned Tensor CLI, cloud orchestration, Docker,
+or a managed database.
 
-## What It Does
+## What Happens In A Run
 
-`tt-local` runs this workflow locally:
+`tt-local` runs this loop:
 
 1. Read a behavior spec from `tunedtensor.json`.
-2. Compile examples into a training dataset.
+2. Compile examples into a local training dataset.
 3. Evaluate the original Hugging Face base model.
 4. Fine-tune locally with uv, Transformers, and PEFT.
 5. Evaluate the fine-tuned Hugging Face/PEFT artifact.
-6. Compare baseline vs tuned outputs and write a local report.
-
-OpenRouter can be used as an LLM judge for scoring generated outputs. It does
-not generate the baseline or tuned responses.
-
-## Install Locally
+6. Compare baseline vs tuned outputs and write a report.
 
-Recommended for development and Spark:
+OpenRouter can optionally score generated outputs as an LLM judge. It does not
+generate the baseline or tuned responses.
 
-```bash
-git clone https://github.com/tunedtensor/tuned-tensor-local.git
-cd tuned-tensor-local
-npm install
-npm run build
-npm link
-```
+## Install
 
-You can install the published [@tuned-tensor/local npm package](https://www.npmjs.com/package/@tuned-tensor/local)
-globally:
+Install the published CLI:
 
 ```bash
 npm install -g @tuned-tensor/local
+tt-local info
 ```
 
-Check the CLI:
+For local development or Spark:
 
 ```bash
+git clone https://github.com/tunedtensor/tuned-tensor-local.git
+cd tuned-tensor-local
+npm install
+npm run build
+npm link
 tt-local info
 ```
 
-## Set Up DGX Spark
-
-On the Spark host, confirm the GPU and uv are available:
+Real training also needs `uv`:
 
 ```bash
-nvidia-smi
 uv --version
-uv run python --version
 ```
 
 If uv is missing:
@@ -65,29 +58,11 @@ If uv is missing:
 curl -LsSf https://astral.sh/uv/install.sh | sh
 ```
 
-Open a new shell, then rerun:
+## First Run: Safe Dry Run
 
-```bash
-uv --version
-```
-
-Clone this repo on the Spark, install the Node dependencies, and build:
-
-```bash
-git clone https://github.com/tunedtensor/tuned-tensor-local.git
-cd tuned-tensor-local
-npm install
-npm run build
-npm link
-```
-
-Run the doctor:
-
-```bash
-tt-local doctor --config examples/local-runner.json
-```
-
-## Run Your First Job
+The included example config is a dry run. It validates the end-to-end local
+workflow and writes artifacts without downloading models or starting GPU
+training.
 
 Create a local behavior spec:
 
@@ -110,16 +85,13 @@ Validate the spec and config:
 tt-local validate --config examples/local-runner.json
 ```
 
-Run the example dry workflow:
+Run the dry workflow:
 
 ```bash
 tt-local run --config examples/local-runner.json
 ```
 
-The example config has `dryRun: true`, so it checks orchestration and artifact
-writing without loading models or starting GPU training.
-
-Inspect the run:
+Inspect the result:
 
 ```bash
 tt-local runs list --config examples/local-runner.json
@@ -132,11 +104,11 @@ Start the local dashboard:
 tt-local serve --config examples/local-runner.json
 ```
 
-Then open the printed local URL in your browser.
+Open the printed local URL in your browser.
 
-## Run Real Training On Spark
+## Run Real Training
 
-Create a Spark config, for example `spark-runner.json`:
+Create a config such as `local-runner.json` or `spark-runner.json`:
 
 ```json
 {
@@ -169,8 +141,109 @@ Create a Spark config, for example `spark-runner.json`:
 }
 ```
 
-For structured JSON tasks, use field-level scoring instead of whole-output
-exact matching:
+If you use OpenRouter judging, set an API key:
+
+```bash
+export OPENROUTER_API_KEY="your_openrouter_key"
+```
+
+`tt-local` also loads a `.env` file from the working directory. Existing
+environment variables win.
+
+Run the job:
+
+```bash
+tt-local doctor --config spark-runner.json
+tt-local validate --config spark-runner.json
+tt-local run --config spark-runner.json
+```
+
+`tt-local run` prints live stage updates to stderr and writes the final JSON
+summary to stdout. Use `--verbose` to stream Python subprocess output, or
+`--quiet` for JSON-only automation.
+
+Watch progress:
+
+```bash
+tt-local runs watch <run-id> --config spark-runner.json
+```
+
+View the final report:
+
+```bash
+tt-local runs report <run-id> --config spark-runner.json
+```
+
+Compare two completed runs:
+
+```bash
+tt-local runs compare <run-id-a> <run-id-b> --config spark-runner.json
+```
+
+The comparison aligns shared eval prompts, separates newly added eval examples,
+and measures judge score noise for identical baseline outputs.
+
+## Configuration Basics
+
+The main config fields are:
+
+- `artifactRoot`: datasets, logs, model outputs, eval JSON, and run reports.
+- `storeRoot`: run/spec/model metadata used by CLI listing and the dashboard.
+- `dryRun`: when `true`, skips model loading and training.
+- `training`: uv/Python training entrypoint and environment.
+- `evaluation.inference`: how baseline and candidate outputs are generated.
+- `evaluation.scoring`: how generated outputs are scored.
+- `paths.modelCache`: Hugging Face cache location.
+
+Supported inference providers:
+
+- `"transformers"`: run the included local evaluator with Hugging Face/PEFT.
+- `"command"`: call `baselineCommand` or `candidateCommand` for custom inference.
+- `"none"`: skip inference and record empty local responses.
+
+Supported scoring modes:
+
+- `"exact_match"`: normalized exact match.
+- `"llm_judge"`: OpenRouter judge, with exact-match fallback by default.
+- `"json_fields"`: field-level comparison for structured JSON outputs.
+
+When `evaluation.scoring.mode` is `"llm_judge"`, `fallback` controls judge
+failures. `"exact_match"` keeps the run going and records fallback reasoning.
+`"fail"` fails the run.
+
+## Evaluation Behavior
+
+`evaluation.maxExamples` caps how many examples are scored. If it is unset,
+the request hyperparameter `max_eval_examples` is used. An explicit
+`maxExamples` in the config wins. When capped, the runner uses a deterministic
+seeded sample rather than a prefix, so sorted or grouped eval files do not
+bias results. The seed defaults to a hash of the run id and can be pinned with
+`evaluation.sampleSeed`.
+
+For spec-based runs without `dataset_prebuilt`, the runner holds out about 20%
+of the spec examples for evaluation, with at least one training example and one
+holdout when possible. With only one spec example, evaluation runs on that
+example and reports `eval_split: "spec_examples"`.
+
+For prebuilt datasets, `dataset_prebuilt.training` is always copied into the
+run artifact as the training set. Evaluation uses `dataset_prebuilt.test` when
+present, then `dataset_prebuilt.validation`. If neither is provided, a real
+run fails because evaluation would otherwise measure the training split. Set
+`evaluation.allowPrebuiltTrainingEval: true` to allow that explicitly.
+
+Baseline evaluations are cached under `<storeRoot>/cache/baseline-evals/`.
+Re-running an unchanged spec with the same model, eval examples, generation
+settings, and scoring config reuses the previous baseline report. Set
+`evaluation.baselineCache: false` to disable.
+
+Training progress from Python metric lines and tqdm progress bars is recorded
+as run events, so `tt-local runs watch` and per-run `progress.jsonl` show
+epoch, loss, step, percentage, and ETA when the trainer emits them.
+
+## Structured JSON Scoring
+
+For structured tasks, use field-level scoring instead of whole-output exact
+matching:
 
 ```json
 {
@@ -183,143 +256,44 @@ exact matching:
 }
 ```
 
-The evaluation report still includes `exact_match_rate`, but `avg_score`,
-`pass_rate`, and `json_field_metrics` are based on the selected JSON fields.
 If `fields` is omitted, the evaluator scores every key in the expected JSON
-object for each example. A configured field that is missing from the expected
-JSON is always scored as incorrect (with a reasoning note), so misconfigured
-field lists cannot inflate scores.
-
-The judge receives the spec's compiled system message (system prompt,
-guidelines, and constraints) as task instructions and scores how well each
-output fulfills the task; the expected output is treated as a reference
-answer, not a fact checklist. Every eval report also includes `avg_token_f1`
-(and the comparison includes `token_f1_delta`), a deterministic token-overlap
-similarity against the reference outputs that is useful for free-text tasks
-such as summarization, where exact match is always 0 and judge scores can be
-noisy.
-
-Each eval example records how it was scored (`results[].scored_by`), and the
-report aggregates `judge_scored_count`, `fallback_scored_count`, and
-`judge_only_avg_score`. When a judge request fails transiently and the
-example falls back to exact match, `avg_score` mixes two metrics; use
-`judge_only_avg_score` (and the comparison's `judge_only_avg_score_delta`)
-when comparing judge quality. The comparison also classifies each regressed
-example's judge reasoning as `factual`, `omission`, `style`, `fallback`, or
-`other` and reports the counts in `regression_taxonomy`.
-
-Baseline evaluations are cached under `<storeRoot>/cache/baseline-evals/`:
-re-running a spec with an unchanged base model, eval examples, generation
-settings, and scoring config reuses the previous baseline report instead of
-re-running inference and judge calls. Cached reports carry `cached: true`.
-Set `evaluation.baselineCache: false` to disable. Reports containing
-fallback-scored examples are never cached, so a transient judge failure is
-not replayed into future runs.
-
-For thinking-mode base models (for example `Qwen/Qwen3.5-4B`, whose chat
-template opens a `<think>` block by default and can consume the entire
-`maxNewTokens` budget before any visible answer), forward template kwargs to
-both evaluation and training:
+object. A configured field missing from the expected JSON is always scored as
+incorrect, so a misconfigured field list cannot inflate scores.
+
+Reports still include `exact_match_rate`, but `avg_score`, `pass_rate`, and
+`json_field_metrics` are based on the selected JSON fields.
+
+## Thinking-Mode Models
+
+Some models, such as `Qwen/Qwen3.5-4B`, may open a hidden thinking block by
+default and consume the token budget before producing a visible answer. Forward
+template kwargs to both evaluation and training:
 
 ```json
 {
   "evaluation": {
-    "inference": { "chatTemplateKwargs": { "enable_thinking": false } }
+    "inference": {
+      "chatTemplateKwargs": { "enable_thinking": false }
+    }
   }
 }
 ```
 
-and in the spec's hyperparameters:
+In `tunedtensor.json`, also set:
 
 ```json
-{ "chat_template_kwargs": { "enable_thinking": false } }
-```
-
-To compare two runs whose eval sets may differ (for example after raising
-`evaluation.maxExamples`), use:
-
-```bash
-tt-local runs compare <run-id-a> <run-id-b> --config spark-runner.json
-```
-
-It aligns the runs on their shared eval prompts, reports both runs' averages
-on that shared subset plus the new-example subset separately, and measures
-judge score noise on baseline outputs that are byte-identical across the two
-runs.
-
-With `"mode": "llm_judge"`, `scoring.fallback` controls what happens when the
-judge is unavailable, a judge request fails, or the judge returns malformed
-JSON. With `"fallback": "exact_match"` (the default), the affected example is
-scored by normalized exact match and its reasoning records the judge failure;
-the run does not fail. With `"fallback": "fail"`, judge errors fail the run.
-
-`evaluation.maxExamples` caps how many evaluation examples are scored. If it
-is unset, the request hyperparameter `max_eval_examples` is used instead;
-an explicit `maxExamples` in the config always wins. When the cap truncates
-the eval set, the runner draws a deterministic seeded random sample (not a
-prefix), so sorted or grouped eval files do not bias the subset. The seed
-defaults to a hash of the run id, can be pinned with `evaluation.sampleSeed`,
-and is recorded in each eval report as `eval_sample_seed`. Baseline and
-candidate evaluations always score the same sampled examples in the same
-order.
-
-If you want OpenRouter judging, set your key:
-
-```bash
-export OPENROUTER_API_KEY="your_openrouter_key"
+{
+  "hyperparameters": {
+    "chat_template_kwargs": { "enable_thinking": false }
+  }
+}
 ```
 
-`tt-local` also loads a `.env` file from the working directory (existing
-environment variables win), so the key can live in the project's `.env`
-instead.
+## Multimodal Examples
 
-`validate` and `run` warn about hyperparameter keys the schema does not
-recognize (for example `per_device_train_batch_size` instead of
-`batch_size`); unknown keys are ignored rather than passed to training.
-
-Run the job:
-
-```bash
-tt-local run --config spark-runner.json
-```
-
-`tt-local run` prints live stage updates to stderr and writes the final JSON
-summary to stdout. Use `--verbose` to stream Python subprocess output, or
-`--quiet` for script-only JSON output.
-
-Training progress from Python metric lines and tqdm progress bars is also
-recorded as run events, so `tt-local runs watch` and per-run `progress.jsonl`
-show epoch, loss, step, percentage, and ETA updates when the trainer emits
-them.
-
-For spec-based runs (no `dataset_prebuilt`), the runner automatically holds
-out about 20% of the spec examples (at least 1 holdout and at least 1 training
-example) for evaluation, so baseline and candidate scores are not measured on
-the training data. The split is deterministic: it uses the per-run sample seed
-(a hash of the run id, or `evaluation.sampleSeed` when set), so re-running the
-same run id reproduces the same split and baseline/candidate always score the
-identical holdout. The training JSONL artifact contains only the training
-split, `run_metadata.training_example_count` records the training split size,
-and `eval_examples_total` records the holdout size. These runs report
-`eval_split: "spec_holdout"`. With only 1 spec example there is nothing to
-hold out, so evaluation runs on the training set and is reported as
-`eval_split: "spec_examples"` (training-set evaluation due to insufficient
-examples).
-
-For prebuilt datasets, `dataset_prebuilt.training` is always copied into the
-run artifact as the training set. Evaluation uses `dataset_prebuilt.test` when
-present, then `dataset_prebuilt.validation`. If neither is provided, a real
-(non-dry) run fails with an error, because evaluating on the training split
-overstates improvement; set `evaluation.allowPrebuiltTrainingEval: true` to
-evaluate on the training file anyway. Dry runs are allowed to fall back to the
-training file without the override. Each eval report and the run's
-`run_metadata` record the split that was evaluated in `eval_split`
-(`spec_holdout`, `spec_examples`, `prebuilt_test`, `prebuilt_validation`, or
-`prebuilt_training`), so training-set evaluation is always visible.
-
-Multimodal runs use the same chat JSONL shape with structured user content.
-For `image_text_to_text` models such as `Qwen/Qwen3-VL-2B-Instruct`, image
-parts are loaded by the local SFT trainer and evaluator:
+Multimodal runs use chat JSONL with structured user content. For
+`image_text_to_text` models such as `Qwen/Qwen3-VL-2B-Instruct`, image parts
+are loaded by the local trainer and evaluator:
 
 ```json
 {
@@ -340,34 +314,17 @@ parts are loaded by the local SFT trainer and evaluator:
 Image values in prebuilt JSONL may be absolute paths, paths relative to the
 JSONL file, `file://` URIs, HTTP(S) URLs, or `data:` URIs. Behavior spec
 examples can also use `input_assets` with `image`, `path`, `uri`, or
-`data_uri`; use absolute paths, `file://` URIs, URLs, or `data:` URIs there so
-the copied run artifact can still resolve the image.
-
-Watch progress:
-
-```bash
-tt-local runs watch <run-id> --config spark-runner.json
-```
-
-View the final report:
-
-```bash
-tt-local runs report <run-id> --config spark-runner.json
-```
+`data_uri`.
 
 ## Where Files Go
 
-The runner writes two local roots:
-
-- `artifactRoot`: datasets, logs, model outputs, eval JSON, and `run-report.json`.
-- `storeRoot`: run/spec/model metadata used by CLI listing and the dashboard.
-
 If `storeRoot` is omitted, `tt-local` uses `TT_LOCAL_HOME` or
 `~/.tuned-tensor-local`.
 
 Per-run artifacts include:
 
-- `progress.jsonl`: stage changes and parsed training progress for the run.
+- `progress.jsonl`: stage changes and parsed training progress.
+- `baseline-eval.json` and `candidate-eval.json`: generated outputs and scores.
 - `baseline-eval.json.inference.log` and `candidate-eval.json.inference.log`:
   local inference subprocess output.
 - `training/training.log`: uv fine-tuning output.
@@ -377,18 +334,43 @@ Per-run artifacts include:
 
 ```bash
 tt-local init --name "Support Bot" --model Qwen/Qwen3.5-2B
+tt-local doctor --config spark-runner.json
 tt-local validate --config spark-runner.json
 tt-local run --config spark-runner.json
 tt-local run --config spark-runner.json --verbose
 tt-local runs list --config spark-runner.json
 tt-local runs get <run-id> --config spark-runner.json
 tt-local runs events <run-id> --config spark-runner.json
+tt-local runs watch <run-id> --config spark-runner.json
 tt-local runs report <run-id> --config spark-runner.json
 tt-local runs compare <run-id-a> <run-id-b> --config spark-runner.json
 tt-local models list --config spark-runner.json
 tt-local serve --config spark-runner.json
 ```
 
+## DGX Spark Notes
+
+On the Spark host, confirm the GPU and uv are available:
+
+```bash
+nvidia-smi
+uv --version
+uv run python --version
+```
+
+Then clone, install, build, and link the CLI on the Spark:
+
+```bash
+git clone https://github.com/tunedtensor/tuned-tensor-local.git
+cd tuned-tensor-local
+npm install
+npm run build
+npm link
+tt-local doctor --config examples/local-runner.json
+```
+
+Spark-specific details live in [docs/spark.md](docs/spark.md).
+
 ## Development
 
 ```bash
@@ -404,12 +386,12 @@ secret.
 
 ## Notes
 
-- Real training uses the included SFT script in `training/sft-local/src/train.py`.
+- Real training uses `training/sft-local/src/train.py`.
 - Real evaluation uses `training/sft-local/src/evaluate.py`.
-- The default evaluation path uses Hugging Face/PEFT artifacts directly; it does
-  not convert models to GGUF.
+- The default evaluation path uses Hugging Face/PEFT artifacts directly; it
+  does not convert models to GGUF.
+- `validate` and `run` warn about unknown hyperparameter keys and ignore them.
 - Release notes are in [CHANGELOG.md](CHANGELOG.md).
-- Spark-specific details live in [docs/spark.md](docs/spark.md).
 
 ## License
 

package/dist/contracts.d.ts

@@ -599,15 +599,11 @@ export declare const localRunnerConfigSchema: z.ZodObject<{
         modelCache: z.ZodOptional<z.ZodString>;
     }, z.core.$strip>>;
     evaluation: z.ZodDefault<z.ZodObject<{
-        mode: z.ZodDefault<z.ZodEnum<{
-            command: "command";
-            llm_judge: "llm_judge";
-            heuristic: "heuristic";
-        }>>;
         baselineCommand: z.ZodOptional<z.ZodArray<z.ZodString>>;
         candidateCommand: z.ZodOptional<z.ZodArray<z.ZodString>>;
         inference: z.ZodDefault<z.ZodObject<{
             provider: z.ZodDefault<z.ZodEnum<{
+                command: "command";
                 none: "none";
                 transformers: "transformers";
             }>>;

package/dist/contracts.js

@@ -227,11 +227,10 @@ export const localRunnerConfigSchema = z.object({
         modelCache: z.string().optional(),
     }).default({}),
     evaluation: z.object({
-        mode: z.enum(["heuristic", "command", "llm_judge"]).default("heuristic"),
         baselineCommand: commandSchema.optional(),
         candidateCommand: commandSchema.optional(),
         inference: z.object({
-            provider: z.enum(["transformers", "none"]).default("transformers"),
+            provider: z.enum(["transformers", "command", "none"]).default("transformers"),
             project: z.string().optional(),
             cwd: z.string().optional(),
             module: z.string().optional(),
@@ -269,7 +268,6 @@ export const localRunnerConfigSchema = z.object({
         allowPrebuiltTrainingEval: z.boolean().default(false),
         baselineCache: z.boolean().default(true),
     }).default({
-        mode: "heuristic",
         inference: {
             provider: "transformers",
             script: "training/sft-local/src/evaluate.py",