@tuned-tensor/local 0.1.9 → 0.2.1

package/CHANGELOG.md

@@ -2,6 +2,27 @@
 
 All notable changes to TT Local will be documented in this file.
 
+## 0.2.1 - 2026-07-04
+
+### Added
+
+- Added command-based local training with `training.backend: "command"` and
+  `training.command`, while preserving the existing uv script/module path.
+- Added `evaluation.inference.provider: "batch_command"` with
+  `evaluation.inference.command` for custom batch evaluators that consume the
+  same `--input`/`--output` JSON files as the bundled Transformers evaluator.
+
+## 0.2.0 - 2026-07-04
+
+### Added
+
+- Added a hybrid local state store: `metadata.sqlite` now indexes specs, runs,
+  run events, and models for CLI/dashboard queries while preserving the
+  transparent JSON/JSONL artifact layout. `tt-local store rebuild-index`
+  rebuilds SQLite metadata from canonical per-object files. The SQLite index
+  uses `better-sqlite3` instead of Node's experimental built-in SQLite API,
+  avoiding runtime experimental warnings on Node 22.
+
 ## 0.1.9 - 2026-07-03
 
 ### Added

package/README.md

@@ -1,419 +1,33 @@
 # TT Local
 
-Tuning tensors locally.
-
 [![CI](https://github.com/tunedtensor/tuned-tensor-local/actions/workflows/ci.yml/badge.svg)](https://github.com/tunedtensor/tuned-tensor-local/actions/workflows/ci.yml)
 [![npm](https://img.shields.io/npm/v/@tuned-tensor/local)](https://www.npmjs.com/package/@tuned-tensor/local)
 
-TT Local is a standalone CLI for running Tuned Tensor-style
-fine-tuning jobs on a machine you control. It stores specs, datasets, model
-artifacts, progress events, reports, and dashboard state on local disk.
-
-It does not require the hosted Tuned Tensor CLI, cloud orchestration, Docker,
-or a managed database.
-
-## What Happens In A Run
+TT Local runs Tuned Tensor-style fine-tuning jobs on your own machine. It keeps
+specs, datasets, model artifacts, events, reports, and dashboard state on local
+disk.
 
-`tt-local` runs this loop:
+Usage docs:
 
-1. Read a behavior spec from `tunedtensor.json`.
-2. Compile examples into a local training dataset.
-3. Evaluate the original Hugging Face base model.
-4. Fine-tune locally with uv, Transformers, and PEFT.
-5. Evaluate the fine-tuned Hugging Face/PEFT artifact.
-6. Compare baseline vs tuned outputs and write a report.
-
-OpenRouter can optionally score generated outputs as an LLM judge. It does not
-generate the baseline or tuned responses.
+https://tunedtensor.com/docs/local-training
 
 ## Install
 
-Install the published CLI:
-
 ```bash
 npm install -g @tuned-tensor/local
 tt-local info
 ```
 
-For local development or Spark:
-
-```bash
-git clone https://github.com/tunedtensor/tuned-tensor-local.git
-cd tuned-tensor-local
-npm install
-npm run build
-npm link
-tt-local info
-```
-
-Real training also needs `uv`:
+The bundled SFT and Transformers evaluator path also needs `uv`:
 
 ```bash
 uv --version
 ```
 
-If uv is missing:
-
-```bash
-curl -LsSf https://astral.sh/uv/install.sh | sh
-```
-
-## First Run: Safe Dry Run
-
-The included example config is a dry run. It validates the end-to-end local
-workflow and writes artifacts without downloading models or starting GPU
-training.
-
-Create a local behavior spec:
-
-```bash
-tt-local init --name "Support Bot" --model Qwen/Qwen3.5-2B
-```
-
-Edit `tunedtensor.json` and replace the placeholder example with your task:
+Custom training or evaluation workflows can use command entrypoints instead of
+`uv`.
 
-```json
-{
-  "input": "Classify sentiment: I love this product.",
-  "output": "positive"
-}
-```
-
-Validate the spec and config:
-
-```bash
-tt-local validate --config examples/local-runner.json
-```
-
-Run the dry workflow:
-
-```bash
-tt-local run --config examples/local-runner.json
-```
-
-Inspect the result:
-
-```bash
-tt-local runs list --config examples/local-runner.json
-tt-local runs report <run-id> --config examples/local-runner.json
-```
-
-Start the local dashboard:
-
-```bash
-tt-local serve --config examples/local-runner.json
-```
-
-Open the printed local URL in your browser.
-
-## Run Real Training
-
-Create a config such as `local-runner.json` or `spark-runner.json`:
-
-```json
-{
-  "artifactRoot": "/home/eve/tt-local-artifacts",
-  "storeRoot": "/home/eve/tt-local-store",
-  "dryRun": false,
-  "training": {
-    "backend": "uv",
-    "project": "training/sft-local",
-    "script": "training/sft-local/src/train.py"
-  },
-  "evaluation": {
-    "inference": {
-      "provider": "transformers",
-      "project": "training/sft-local",
-      "script": "training/sft-local/src/evaluate.py",
-      "maxNewTokens": 256,
-      "temperature": 0,
-      "topP": 1
-    },
-    "scoring": {
-      "mode": "llm_judge",
-      "fallback": "exact_match"
-    },
-    "maxExamples": 10
-  },
-  "paths": {
-    "modelCache": "/home/eve/.cache/huggingface"
-  }
-}
-```
-
-If you use OpenRouter judging, set an API key:
-
-```bash
-export OPENROUTER_API_KEY="your_openrouter_key"
-```
-
-`tt-local` also loads a `.env` file from the working directory. Existing
-environment variables win.
-
-Run the job:
-
-```bash
-tt-local doctor --config spark-runner.json
-tt-local validate --config spark-runner.json
-tt-local run --config spark-runner.json
-```
-
-`tt-local run` prints live stage updates to stderr and writes the final JSON
-summary to stdout. Use `--verbose` to stream Python subprocess output, or
-`--quiet` for JSON-only automation.
-
-Watch progress:
-
-```bash
-tt-local runs watch <run-id> --config spark-runner.json
-```
-
-View the final report:
-
-```bash
-tt-local runs report <run-id> --config spark-runner.json
-```
-
-Compare two completed runs:
-
-```bash
-tt-local runs compare <run-id-a> <run-id-b> --config spark-runner.json
-```
-
-The comparison aligns shared eval prompts, separates newly added eval examples,
-and measures judge score noise for identical baseline outputs.
-
-## Configuration Basics
-
-The main config fields are:
-
-- `artifactRoot`: datasets, logs, model outputs, eval JSON, and run reports.
-- `storeRoot`: run/spec/model metadata used by CLI listing and the dashboard.
-- `dryRun`: when `true`, skips model loading and training.
-- `training`: uv/Python training entrypoint and environment.
-- `evaluation.inference`: how baseline and candidate outputs are generated.
-- `evaluation.scoring`: how generated outputs are scored.
-- `paths.modelCache`: Hugging Face cache location.
-
-Supported inference providers:
-
-- `"transformers"`: run the included local evaluator with Hugging Face/PEFT.
-- `"command"`: call `baselineCommand` or `candidateCommand` for custom inference.
-- `"none"`: skip inference and record empty local responses.
-
-Supported scoring modes:
-
-- `"exact_match"`: normalized exact match.
-- `"llm_judge"`: OpenRouter judge, with exact-match fallback by default.
-- `"json_fields"`: field-level comparison for structured JSON outputs.
-
-When `evaluation.scoring.mode` is `"llm_judge"`, `fallback` controls judge
-failures. `"exact_match"` keeps the run going and records fallback reasoning.
-`"fail"` fails the run.
-
-## Evaluation Behavior
-
-`evaluation.maxExamples` caps how many examples are scored. If it is unset,
-the request hyperparameter `max_eval_examples` is used. An explicit
-`maxExamples` in the config wins. When capped, the runner uses a deterministic
-seeded sample rather than a prefix, so sorted or grouped eval files do not
-bias results. The seed defaults to a hash of the run id and can be pinned with
-`evaluation.sampleSeed`.
-
-For spec-based runs without `dataset_prebuilt`, the runner holds out about 20%
-of the spec examples for evaluation, with at least one training example and one
-holdout when possible. With only one spec example, evaluation runs on that
-example and reports `eval_split: "spec_examples"`.
-
-For prebuilt datasets, `dataset_prebuilt.training` is always copied into the
-run artifact as the training set. Evaluation uses `dataset_prebuilt.test` when
-present, then `dataset_prebuilt.validation`. If neither is provided, a real
-run fails because evaluation would otherwise measure the training split. Set
-`evaluation.allowPrebuiltTrainingEval: true` to allow that explicitly.
-
-Baseline evaluations are cached under `<storeRoot>/cache/baseline-evals/`.
-Re-running an unchanged spec with the same model, eval examples, generation
-settings, and scoring config reuses the previous baseline report. Set
-`evaluation.baselineCache: false` to disable.
-
-Training progress from Python metric lines and tqdm progress bars is recorded
-as run events, so `tt-local runs watch` and per-run `progress.jsonl` show
-epoch, loss, step, percentage, and ETA when the trainer emits them.
-
-## Structured JSON Scoring
-
-For structured tasks, use field-level scoring instead of whole-output exact
-matching:
-
-```json
-{
-  "evaluation": {
-    "scoring": {
-      "mode": "json_fields",
-      "fields": ["triage", "priority", "should_process"]
-    }
-  }
-}
-```
-
-If `fields` is omitted, the evaluator scores every key in the expected JSON
-object. A configured field missing from the expected JSON is always scored as
-incorrect, so a misconfigured field list cannot inflate scores.
-
-Reports still include `exact_match_rate`, but `avg_score`, `pass_rate`, and
-`json_field_metrics` are based on the selected JSON fields.
-
-## Labeling Data With A Teacher Model
-
-`tt-local label` turns unlabeled inputs into training examples by asking a
-stronger teacher model on OpenRouter to answer each row under your spec's
-system message — the same instructions the fine-tuned model will see at
-inference:
-
-```bash
-export OPENROUTER_API_KEY=...   # or put it in .env
-tt-local label rows.jsonl --output labeled.jsonl
-tt-local label rows.csv --input-column text --model anthropic/claude-fable-5
-```
-
-Sources are JSONL (`{"input": "..."}` per line; rows that already carry an
-`"output"` skip the teacher) or CSV with a header row and `--input-column`.
-The system message comes from `tunedtensor.json` (or `--spec <path>`);
-pass `--system-prompt "..."` to override it. Use `--dry-run` to parse,
-sanitize, and estimate without calling the teacher.
-
-Because rows are sent to OpenRouter, each row is sanitized first:
-secret-like content (API keys, private keys, connection strings) blocks the
-row, and PII (emails, phone numbers, SSNs, card numbers) is redacted before
-sending. The labeled `{"input", "output"}` JSONL, a full per-row
-`rows.jsonl` (including failures for review), and a job report are written
-under `<artifactRoot>/labeling/<job-id>/`.
-
-Teacher behavior is configured in the runner config; the model falls back to
-`llm.model`:
-
-```json
-{
-  "labeling": {
-    "model": "anthropic/claude-fable-5",
-    "maxTokens": 2048,
-    "temperature": 0.2,
-    "concurrency": 4,
-    "maxAttempts": 3
-  }
-}
-```
-
-## Thinking-Mode Models
-
-Some models, such as `Qwen/Qwen3.5-4B`, may open a hidden thinking block by
-default and consume the token budget before producing a visible answer. Forward
-template kwargs to both evaluation and training:
-
-```json
-{
-  "evaluation": {
-    "inference": {
-      "chatTemplateKwargs": { "enable_thinking": false }
-    }
-  }
-}
-```
-
-In `tunedtensor.json`, also set:
-
-```json
-{
-  "hyperparameters": {
-    "chat_template_kwargs": { "enable_thinking": false }
-  }
-}
-```
-
-## Multimodal Examples
-
-Multimodal runs use chat JSONL with structured user content. For
-`image_text_to_text` models such as `Qwen/Qwen3-VL-2B-Instruct`, image parts
-are loaded by the local trainer and evaluator:
-
-```json
-{
-  "messages": [
-    { "role": "system", "content": "Answer chart questions concisely." },
-    {
-      "role": "user",
-      "content": [
-        { "type": "image", "image": "charts/example.png" },
-        { "type": "text", "text": "What is the blue value?" }
-      ]
-    },
-    { "role": "assistant", "content": "42" }
-  ]
-}
-```
-
-Image values in prebuilt JSONL may be absolute paths, paths relative to the
-JSONL file, `file://` URIs, HTTP(S) URLs, or `data:` URIs. Behavior spec
-examples can also use `input_assets` with `image`, `path`, `uri`, or
-`data_uri`.
-
-## Where Files Go
-
-If `storeRoot` is omitted, `tt-local` uses `TT_LOCAL_HOME` or
-`~/.tuned-tensor-local`.
-
-Per-run artifacts include:
-
-- `progress.jsonl`: stage changes and parsed training progress.
-- `baseline-eval.json` and `candidate-eval.json`: generated outputs and scores.
-- `baseline-eval.json.inference.log` and `candidate-eval.json.inference.log`:
-  local inference subprocess output.
-- `training/training.log`: uv fine-tuning output.
-- `run-report.json`: baseline, candidate, comparison, and artifact references.
-
-## Useful Commands
-
-```bash
-tt-local init --name "Support Bot" --model Qwen/Qwen3.5-2B
-tt-local doctor --config spark-runner.json
-tt-local validate --config spark-runner.json
-tt-local run --config spark-runner.json
-tt-local run --config spark-runner.json --verbose
-tt-local label rows.jsonl --output labeled.jsonl
-tt-local runs list --config spark-runner.json
-tt-local runs get <run-id> --config spark-runner.json
-tt-local runs events <run-id> --config spark-runner.json
-tt-local runs watch <run-id> --config spark-runner.json
-tt-local runs report <run-id> --config spark-runner.json
-tt-local runs compare <run-id-a> <run-id-b> --config spark-runner.json
-tt-local models list --config spark-runner.json
-tt-local serve --config spark-runner.json
-```
-
-## DGX Spark Notes
-
-On the Spark host, confirm the GPU and uv are available:
-
-```bash
-nvidia-smi
-uv --version
-uv run python --version
-```
-
-Then clone, install, build, and link the CLI on the Spark:
-
-```bash
-git clone https://github.com/tunedtensor/tuned-tensor-local.git
-cd tuned-tensor-local
-npm install
-npm run build
-npm link
-tt-local doctor --config examples/local-runner.json
-```
-
-Spark-specific details live in [docs/spark.md](docs/spark.md).
-
-## Development
+## Local Development
 
 ```bash
 npm install
@@ -422,18 +36,7 @@ npm test
 npm run build
 ```
 
-CI runs those checks on pull requests and pushes to `main`. Publishing to npm
-runs when a GitHub Release is published and requires an `NPM_TOKEN` repository
-secret.
-
-## Notes
-
-- Real training uses `training/sft-local/src/train.py`.
-- Real evaluation uses `training/sft-local/src/evaluate.py`.
-- The default evaluation path uses Hugging Face/PEFT artifacts directly; it
-  does not convert models to GGUF.
-- `validate` and `run` warn about unknown hyperparameter keys and ignore them.
-- Release notes are in [CHANGELOG.md](CHANGELOG.md).
+Release notes are in [CHANGELOG.md](CHANGELOG.md).
 
 ## License
 

package/dist/contracts.d.ts

@@ -271,6 +271,7 @@ export declare const evalReportSchema: z.ZodObject<{
     inference_provider: z.ZodOptional<z.ZodEnum<{
         command: "command";
         none: "none";
+        batch_command: "batch_command";
         transformers: "transformers";
     }>>;
     scoring_mode: z.ZodOptional<z.ZodEnum<{
@@ -331,7 +332,10 @@ export declare const comparisonReportSchema: z.ZodObject<{
     }, z.core.$strip>>;
 }, z.core.$strip>;
 export declare const trainingReportSchema: z.ZodObject<{
-    provider: z.ZodLiteral<"local-uv">;
+    provider: z.ZodEnum<{
+        "local-uv": "local-uv";
+        "local-command": "local-command";
+    }>;
     training_job_name: z.ZodString;
     model_artifact_uri: z.ZodOptional<z.ZodString>;
     base_model_artifact_uri: z.ZodOptional<z.ZodString>;
@@ -407,6 +411,7 @@ export declare const runReportSchema: z.ZodObject<{
         inference_provider: z.ZodOptional<z.ZodEnum<{
             command: "command";
             none: "none";
+            batch_command: "batch_command";
             transformers: "transformers";
         }>>;
         scoring_mode: z.ZodOptional<z.ZodEnum<{
@@ -487,6 +492,7 @@ export declare const runReportSchema: z.ZodObject<{
         inference_provider: z.ZodOptional<z.ZodEnum<{
             command: "command";
             none: "none";
+            batch_command: "batch_command";
             transformers: "transformers";
         }>>;
         scoring_mode: z.ZodOptional<z.ZodEnum<{
@@ -540,7 +546,10 @@ export declare const runReportSchema: z.ZodObject<{
         }, z.core.$strip>>;
     }, z.core.$strip>;
     training: z.ZodObject<{
-        provider: z.ZodLiteral<"local-uv">;
+        provider: z.ZodEnum<{
+            "local-uv": "local-uv";
+            "local-command": "local-command";
+        }>;
         training_job_name: z.ZodString;
         model_artifact_uri: z.ZodOptional<z.ZodString>;
         base_model_artifact_uri: z.ZodOptional<z.ZodString>;
@@ -595,7 +604,11 @@ export declare const localRunnerConfigSchema: z.ZodObject<{
     artifactRoot: z.ZodDefault<z.ZodString>;
     dryRun: z.ZodDefault<z.ZodBoolean>;
     training: z.ZodDefault<z.ZodObject<{
-        backend: z.ZodDefault<z.ZodLiteral<"uv">>;
+        backend: z.ZodDefault<z.ZodEnum<{
+            command: "command";
+            uv: "uv";
+        }>>;
+        command: z.ZodOptional<z.ZodArray<z.ZodString>>;
         project: z.ZodOptional<z.ZodString>;
         cwd: z.ZodOptional<z.ZodString>;
         module: z.ZodOptional<z.ZodString>;
@@ -615,12 +628,15 @@ export declare const localRunnerConfigSchema: z.ZodObject<{
             provider: z.ZodDefault<z.ZodEnum<{
                 command: "command";
                 none: "none";
+                batch_command: "batch_command";
                 transformers: "transformers";
             }>>;
+            command: z.ZodOptional<z.ZodArray<z.ZodString>>;
             project: z.ZodOptional<z.ZodString>;
             cwd: z.ZodOptional<z.ZodString>;
             module: z.ZodOptional<z.ZodString>;
             script: z.ZodDefault<z.ZodString>;
+            args: z.ZodDefault<z.ZodArray<z.ZodString>>;
             with: z.ZodDefault<z.ZodArray<z.ZodString>>;
             env: z.ZodDefault<z.ZodRecord<z.ZodString, z.ZodString>>;
             maxNewTokens: z.ZodDefault<z.ZodNumber>;

package/dist/contracts.js

@@ -133,7 +133,7 @@ export const evalReportSchema = z.object({
     artifact_uri: z.string(),
     scoring_method: z.enum(["heuristic", "command", "llm_judge", "exact_match", "json_fields"]),
     judge_model_id: z.string().nullable().optional(),
-    inference_provider: z.enum(["none", "command", "transformers"]).optional(),
+    inference_provider: z.enum(["none", "command", "batch_command", "transformers"]).optional(),
     scoring_mode: z.enum(["exact_match", "llm_judge", "json_fields"]).optional(),
     json_field_metrics: jsonFieldMetricsSchema.optional(),
     generation_config: z.record(z.string(), z.unknown()).optional(),
@@ -157,7 +157,7 @@ export const comparisonReportSchema = z.object({
     })),
 });
 export const trainingReportSchema = z.object({
-    provider: z.literal("local-uv"),
+    provider: z.enum(["local-uv", "local-command"]),
     training_job_name: z.string(),
     model_artifact_uri: z.string().optional(),
     base_model_artifact_uri: z.string().optional(),
@@ -220,7 +220,8 @@ export const localRunnerConfigSchema = z.object({
     artifactRoot: z.string().default(".tt-local/artifacts"),
     dryRun: z.boolean().default(false),
     training: z.object({
-        backend: z.literal("uv").default("uv"),
+        backend: z.enum(["uv", "command"]).default("uv"),
+        command: commandSchema.optional(),
         project: z.string().optional(),
         cwd: z.string().optional(),
         module: z.string().optional(),
@@ -242,11 +243,13 @@ export const localRunnerConfigSchema = z.object({
         baselineCommand: commandSchema.optional(),
         candidateCommand: commandSchema.optional(),
         inference: z.object({
-            provider: z.enum(["transformers", "command", "none"]).default("transformers"),
+            provider: z.enum(["transformers", "batch_command", "command", "none"]).default("transformers"),
+            command: commandSchema.optional(),
             project: z.string().optional(),
             cwd: z.string().optional(),
             module: z.string().optional(),
             script: z.string().default("training/sft-local/src/evaluate.py"),
+            args: z.array(z.string()).default([]),
             with: z.array(z.string()).default([]),
             env: z.record(z.string(), z.string()).default({}),
             maxNewTokens: z.number().int().min(1).default(256),
@@ -258,6 +261,7 @@ export const localRunnerConfigSchema = z.object({
         }).default({
             provider: "transformers",
             script: "training/sft-local/src/evaluate.py",
+            args: [],
             with: [],
             env: {},
             maxNewTokens: 256,
@@ -283,6 +287,7 @@ export const localRunnerConfigSchema = z.object({
         inference: {
             provider: "transformers",
             script: "training/sft-local/src/evaluate.py",
+            args: [],
             with: [],
             env: {},
             maxNewTokens: 256,