@tuned-tensor/local 0.2.9 → 0.4.0

package/docs/architecture.md

@@ -1,154 +1,89 @@
 # Architecture
 
-TT Local is organized around a small set of replaceable local
-adapters. The first implementation targets a single Linux GPU host and is
-packaged as a standalone CLI.
-
-## Subsystems
-
-- Local orchestrator: runs the workflow states in-process and records progress
-  to local files.
-- Local project spec: `tt-local init` creates a local `tunedtensor.json`, and
-  `tt-local run` converts that spec into the same run request contract used by
-  compatible hosted exports.
-- Artifact store: writes datasets, logs, model outputs, and reports beneath a
-  configured local artifact root.
-- Local state store: keeps a SQLite metadata index for specs, runs, progress
-  events, and model records while preserving JSON/JSONL files under a
-  configured `storeRoot` for portable artifacts, recovery, and inspection.
-- Training adapter: launches either a uv-managed Python process or an arbitrary
-  command with local input, model cache, and output directories passed through
-  environment variables. Bundled uv training supports SFT and text-only offline
-  DPO; command training can implement custom methods.
-- Evaluation adapter: runs local Hugging Face/PEFT inference, an arbitrary
-  batch command, or per-example commands for baseline and candidate evaluation.
-  Evaluation can also use an OpenRouter LLM judge to score locally generated
-  outputs.
-- Report writer: emits structured JSON reports that are portable across local
-  and hosted environments.
-- Dashboard/API server: serves local run/model/spec metadata from the local
-  state store and accepts local run submissions.
-- Artifact verifier: validates the trained adapter/full-model contract, writes
-  an atomic SHA-256 manifest, and registers the model before downstream
-  evaluation so successful training survives later-stage failures.
-- Model inference server: loads the recorded base model plus the verified local
-  adapter and exposes localhost `/health`, `/v1/models`, and OpenAI-compatible
-  `/v1/chat/completions` endpoints.
-
-## Repository Boundary
-
-This repository should remain a clean open-source implementation. Private
-deployment code, proprietary provider adapters, account projection code, and
-billing integrations stay out of this project.
-
-The hosted Tuned Tensor CLI can keep cloud-specific workflows such as auth,
-accounts, billing, and managed runs. `tt-local` must be useful on its own: local
-spec creation, validation, execution, run tracking, model inspection, and the
-dashboard should not depend on the hosted CLI.
-
-## First Milestone
-
-The first runnable milestone should support:
-
-- one local run at a time;
-- one GPU device;
-- local filesystem artifacts;
-- SQLite-backed run/model/spec tracking with recoverable JSON files;
-- standalone `tt-local init`, `validate`, `run`, `serve`, and inspection
-  commands;
-- uv-based or command-based training;
-- native Transformers/PEFT, batch-command, or per-example command evaluation
-  with optional OpenRouter judge scoring;
-- verified model manifests and local adapter serving;
-- local dashboard and CLI inspection commands;
-- a tiny smoke-run fixture that finishes quickly.
-
-## Configuration Shape
-
-The public runner should accept a plain JSON run request and a local runner
-configuration. The run request describes the training method, behavior spec,
-base model, examples, datasets, and hyperparameters. The local runner
-configuration describes artifact paths, store paths, uv or command entrypoint
-settings, model cache paths, OpenRouter judge settings, and timeout limits.
-
-## Evaluation Loop
-
-The default Transformers loop intentionally compares like-for-like model artifacts:
-
-- baseline loads the original Hugging Face base model from the behavior spec;
-- training writes a Hugging Face/PEFT artifact under the run directory;
-- candidate evaluation loads the same base model plus that fine-tuned artifact;
-- the report compares the two eval reports and stores generated outputs.
-
-The first native backend is `transformers`. `llama.cpp`/GGUF conversion is not
-part of the default path because conversion would make the baseline/candidate
-comparison less direct.
-
-Custom model families can use `training.backend: "command"` and
-`evaluation.inference.provider: "batch_command"`. Batch evaluators receive the
-same `--input`/`--output` JSON files as the bundled Transformers evaluator, so a
-custom workflow can load any artifact format as long as it writes compatible
-evaluation results.
-
-`paths.modelCache` is a single cache contract: it is passed as Hugging Face
-`HF_HOME`, and Hub snapshots live beneath `<modelCache>/hub`. The environment
-is finalized before importing Hugging Face or Transformers in every bundled
-Python process. Prefetch records the resolved snapshot revision, file count,
-and byte count; `models verify-base` performs a local-only structural check of
-the cached config, tokenizer, weights, and indexed shards.
-
-An explicitly configured `paths.baseModel` is a complete Hugging Face snapshot
-directory, never a generic weights file or archive. Its file contents are
-fingerprinted for stage identity. Remote runs should set an immutable
-`base_model_revision`; baseline caching is bypassed when the model or a remote
-input lacks stable content identity.
-
-## Local State Layout
-
-The local store is intentionally transparent and easy to back up:
-
-- `metadata.sqlite`
-- `specs/<spec-id>/spec.json`
-- `runs/<run-id>/request.json`
-- `runs/<run-id>/state.json`
-- `runs/<run-id>/progress.jsonl`
-- `runs/<run-id>/run-report.json`
-- `models/<model-id>/model.json`
-
-Each successful real training stage also writes `artifact-manifest.json` with
-the model contract and SHA-256/size entries for immutable files. Dry runs do
-not create model records. `tt-local models verify` accepts a model ID, exact
-artifact path, or its manifest path and re-hashes the model before it is served
-or handed off.
-
-The default Transformers/PEFT contract requires exact adapter weights plus a
-non-empty `adapter_config.json`; a full-model contract requires recognized
-Transformers model weights. Command backends may publish another layout only
-with an explicit framework/format contract, and such artifacts are not
-implicitly servable.
-
-SQLite is the primary metadata index for CLI and dashboard listings. The
-per-object JSON files remain the recoverable source for artifacts and
-human-readable inspection. `tt-local store rebuild-index` can reconstruct the
-SQLite index from the canonical per-object files.
-
-## Dashboard API
-
-The local server is a lightweight Node HTTP server backed by the local state
-store. Current endpoints include:
-
-- `GET /api/health`
-- `GET /api/runs`, `GET /api/runs/:id`, `GET /api/runs/:id/events`
-- `GET /api/runs/:id/events/stream`
-- `GET /api/runs/:id/report`
-- `POST /api/runs`
-- `POST /api/runs/:id/cancel`
-- `GET /api/models`, `GET /api/models/:id`
-- `GET /api/specs`, `GET /api/specs/:id`
-
-## Failure Handling
-
-The orchestrator should fail fast when prerequisites are missing. Common checks
-include uv availability, Python dependency resolution, GPU visibility, model
-cache permissions, artifact directory writability, and dataset size limits for
-the selected model.
+TT Local is a focused, single-host fine-tuning tool. Its supported product
+boundary is text SFT of `Qwen/Qwen3.5-2B` into a LoRA adapter on CUDA.
+
+## Workflow
+
+1. Parse a strict behavior spec or request.
+2. Compile or validate chat JSONL and select distinct training/evaluation rows.
+3. Evaluate the base model with the bundled Transformers runner.
+4. Train a PEFT adapter with the bundled, locked `uv` project.
+5. Verify and register the adapter before downstream evaluation.
+6. Evaluate the adapter on the same held-out rows.
+7. Write a base-versus-tuned report.
+
+The orchestrator retains explicit stages and fingerprints internally so an
+interrupted run can safely resume. These mechanics are reliability
+implementation details, not a public plugin framework.
+
+## Boundaries
+
+The first version intentionally excludes:
+
+- arbitrary training or evaluation commands;
+- DPO, continued pretraining, and multimodal training;
+- cloud labeling or LLM-as-judge calls;
+- classic tabular model studies;
+- a local web dashboard;
+- unverified model-family claims.
+
+The next model or method should arrive with its own locked dependencies,
+resource defaults, data contract, and real CUDA acceptance test.
+
+## Python boundary
+
+Node owns input validation, state, process lifecycle, manifests, and reports.
+The bundled Python project owns four small operations:
+
+- `prefetch.py`: download and verify the pinned Hugging Face snapshot;
+- `evaluate.py`: generate predictions without receiving reference answers;
+- `train.py`: perform CUDA-only Qwen LoRA SFT;
+- `serve.py`: load the base model and verified adapter for inference.
+
+Training uses assistant-only loss. Prompt tokens have label `-100`, and
+truncation removes older prompt tokens before it removes any answer token.
+Tokenizer chat-template failures are fatal because a fallback would silently
+change the trained or served prompt format.
+
+## Evaluation
+
+Baseline and candidate evaluation use the same input IDs, generation settings,
+and deterministic scoring:
+
+- baseline: recorded base-model snapshot;
+- candidate: that base snapshot plus the run's verified PEFT adapter;
+- scoring: normalized exact match or selected JSON fields.
+
+The Python evaluator receives only opaque IDs and prompts. Node joins its
+predictions to trusted references before scoring. This avoids accidental label
+leakage across the process boundary.
+
+## Storage and integrity
+
+Canonical JSON and JSONL files are the state store; there is no database or
+mirrored index. Writes are atomic and listings scan the small local metadata
+tree deterministically.
+
+Each real adapter has an atomic `artifact-manifest.json` with the expected PEFT
+files, byte sizes, and SHA-256 hashes. A model is registered immediately after
+training and manifest verification, so a later evaluation failure does not
+hide a valid artifact. Serving verifies the manifest again.
+
+## Configuration
+
+The runner config controls only:
+
+- state, artifact, and Hugging Face cache paths;
+- CUDA or CPU inference for evaluation and serving;
+- deterministic generation and scoring limits.
+
+The training project, Python entrypoints, working directory, and child
+environment are internal and fixed. Every stage uses the locked `uv` project
+included in the npm package. The mutable virtual environment lives in the
+user cache, keyed by the project and lockfile content, so a read-only global
+npm install remains runnable.
+
+Schemas are strict. Misspelled or obsolete fields fail validation instead of
+silently reverting to defaults. Training itself is CUDA-only and fails fast
+when PyTorch cannot see the GPU.

package/docs/spark.md

@@ -1,134 +1,103 @@
-# Running On DGX Spark
+# DGX Spark
 
-TT Local expects the Spark to behave like a single Linux GPU host:
-uv must be installed, Python dependencies must resolve, and CUDA/PyTorch must be
-able to see the GPU.
+DGX Spark is the reference host for TT Local's first supported path:
+`Qwen/Qwen3.5-2B` text SFT with a LoRA adapter.
 
-## Host Checks
+## Check the host
 
-Run these on the Spark host:
+Run on the Spark:
 
 ```bash
 nvidia-smi
+node --version
 uv --version
-uv run python --version
 ```
 
-If uv is missing, install it with the official standalone installer:
+TT Local requires Node 22+, `uv`, working CUDA PyTorch, and enough free space
+for the Hugging Face cache plus run artifacts.
 
-```bash
-curl -LsSf https://astral.sh/uv/install.sh | sh
-```
-
-Open a new shell or source the installer-updated profile before rerunning
-`uv --version`.
-
-Then run the project doctor command:
-
-```bash
-npm run build
-tt-local doctor --config examples/local-runner.json
-```
-
-## Dry Run
-
-The example config uses `dryRun: true`, so it validates orchestration, dataset
-compilation, artifact writing, and report generation without launching training:
+## Create a project
 
 ```bash
-tt-local run examples/smoke-run-request.json --config examples/local-runner.json
+mkdir -p ~/tuned-tensor-runs/support-adapter
+cd ~/tuned-tensor-runs/support-adapter
+tt-local init --name "Support Adapter" --model Qwen/Qwen3.5-2B --profile spark
 ```
 
-## Real Training
+Edit both generated examples in `tunedtensor.json`. For a meaningful run,
+replace them with a larger, representative dataset and a separate validation
+split.
 
-For real training with the bundled SFT or DPO workflow, install `tt-local`, set
-`dryRun: false`, and leave the uv project pointed at `training/local-runner`.
-That project is bundled in the npm package, so users do not need a local source
-checkout. Leave `training.script` unset: the runner chooses the bundled script
-from the run request's `training_method`.
-
-- `training_method: "sft"` runs `training/local-runner/src/train.py`.
-- `training_method: "dpo"` runs `training/local-runner/src/train_dpo.py`.
-
-Set `training.script` only when you want to override that selection with a
-specific custom uv script.
+The generated `local-runner.json` uses CUDA and project-local artifacts. A
+durable Spark configuration can set:
 
 ```json
 {
-  "artifactRoot": "/home/eve/tt-local-artifacts",
-  "dryRun": false,
-  "training": {
-    "backend": "uv",
-    "project": "training/local-runner"
-  },
+  "artifactRoot": "/home/eve/tuned-tensor-runs/artifacts",
+  "storeRoot": "/home/eve/tuned-tensor-runs/store",
   "paths": {
     "modelCache": "/home/eve/.cache/huggingface"
+  },
+  "evaluation": {
+    "inference": {
+      "device": "cuda"
+    },
+    "scoring": {
+      "mode": "exact_match"
+    },
+    "timeoutMs": 1800000
   }
 }
 ```
 
-The runner sets:
-
-- `SM_CHANNEL_TRAINING` to the local training JSONL directory;
-- `TT_HYPERPARAMETERS_PATH` to the generated hyperparameter JSON file;
-- `SM_OUTPUT_DIR` to the local training output directory;
-- `SM_MODEL_DIR` to the local model output directory;
-- `SM_CHANNEL_BASE_MODEL` when a local base-model artifact path is configured;
-- `HF_HOME` when `paths.modelCache` is configured.
-
-Real runs also use the Transformers/PEFT evaluator by default. Baseline
-evaluation loads the original Hugging Face model, and candidate evaluation loads
-that same base model plus the fine-tuned artifact from the run directory. Set
-`paths.modelCache` to a persistent Spark-local cache so training and evaluation
-reuse downloads.
+Every Python stage uses the locked runtime included in the npm package; a
+source checkout and a custom runner path are neither required nor supported.
 
-To make the first large download explicit, prefetch the base model after
-creating `tunedtensor.json` and the runner config:
+## Preflight and run
 
 ```bash
-tt-local models prefetch tunedtensor.json --config spark-runner.json
+tt-local doctor tunedtensor.json
+tt-local validate tunedtensor.json
+tt-local models prefetch tunedtensor.json
+tt-local models verify-base tunedtensor.json
+tt-local run tunedtensor.json
 ```
 
-If you skip this step, the first real `tt-local run` still downloads through
-Transformers automatically.
+`doctor` resolves the same bundled project and paths the run will use, imports
+Torch/Transformers/PEFT, requires visible CUDA, checks writable storage, and
+rejects unchanged placeholders. `validate` reads and normalizes the actual
+dataset before any run state or artifact directory is claimed.
 
-Custom workflows such as a from-scratch nanoGPT trainer can use command
-entrypoints instead:
+The runner provides these paths to Python:
 
-```json
-{
-  "artifactRoot": "/home/eve/tt-local-artifacts",
-  "dryRun": false,
-  "training": {
-    "backend": "command",
-    "command": ["python", "nanogpt/train.py"]
-  },
-  "evaluation": {
-    "inference": {
-      "provider": "batch_command",
-      "command": ["python", "nanogpt/evaluate.py"]
-    },
-    "scoring": {
-      "mode": "exact_match"
-    }
-  }
-}
-```
+- `SM_CHANNEL_TRAINING`: prepared chat JSONL directory;
+- `TT_HYPERPARAMETERS_PATH`: generated SFT/LoRA parameters;
+- `SM_OUTPUT_DIR`: logs and metrics;
+- `SM_MODEL_DIR`: adapter output;
+- `SM_CHANNEL_BASE_MODEL`: optional verified local model snapshot;
+- `HF_HOME`: configured persistent model cache.
+
+## Verify and serve
 
-The training command receives the same environment variables listed above. The
-batch evaluator is called with `--input <path> --output <path>` and should write
-the same results JSON shape as the bundled evaluator.
+```bash
+tt-local runs report <run-id>
+tt-local models verify local-<run-id>
+tt-local serve local-<run-id> --spec tunedtensor.json --port 8000
+```
 
-The included first-pass SFT and DPO scripts can be run by the local runner
-through uv:
+In another shell:
 
 ```bash
-tt-local run examples/smoke-run-request.json --config examples/local-runner.json
-tt-local run examples/dpo-run-request.json --config examples/local-runner.json
+curl http://127.0.0.1:8000/health
+curl http://127.0.0.1:8000/v1/models
+curl http://127.0.0.1:8000/v1/chat/completions \
+  -H 'Content-Type: application/json' \
+  -d '{"messages":[{"role":"user","content":"Classify: I loved it."}]}'
 ```
 
-The SFT uv project currently points Linux installs at the PyTorch CUDA 13.0
-wheel index. If PyTorch does not publish a compatible wheel for the Spark's
-architecture, use `training.env`, `training.project`, or the SFT
-`pyproject.toml` to point uv at the NVIDIA/PyTorch package source that matches
-the host.
+If a run fails, start with `tt-local runs events <run-id>` and
+`tt-local runs get <run-id>`. The run record reports its `artifact_dir`; the
+main subprocess logs there are `training/training.log`,
+`baseline-eval.json.inference.log`, and `candidate-eval.json.inference.log`.
+The adapter is registered as soon as its manifest verifies, even if candidate
+evaluation fails afterward.

package/examples/dry-runner.json

@@ -0,0 +1,14 @@
+{
+  "artifactRoot": ".tt-local/dry-artifacts",
+  "storeRoot": ".tt-local/dry-store",
+  "dryRun": true,
+  "evaluation": {
+    "inference": {
+      "device": "cuda"
+    },
+    "scoring": {
+      "mode": "exact_match"
+    },
+    "maxExamples": 2
+  }
+}

package/examples/general-regression.jsonl

@@ -0,0 +1,4 @@
+{"messages":[{"role":"system","content":"Follow the instruction exactly and answer without explanation."},{"role":"user","content":"Answer with only the numeral: What is 7 + 5?"},{"role":"assistant","content":"12"}]}
+{"messages":[{"role":"system","content":"Follow the instruction exactly and answer without explanation."},{"role":"user","content":"Reply with exactly the lowercase word blue."},{"role":"assistant","content":"blue"}]}
+{"messages":[{"role":"system","content":"Follow the instruction exactly and answer without explanation."},{"role":"user","content":"Return the three letters ABC in reverse order."},{"role":"assistant","content":"CBA"}]}
+{"messages":[{"role":"system","content":"Follow the instruction exactly and answer without explanation."},{"role":"user","content":"Answer yes or no only: Is London in the United Kingdom?"},{"role":"assistant","content":"yes"}]}

package/examples/local-runner.json

@@ -1,18 +1,25 @@
 {
   "artifactRoot": ".tt-local/artifacts",
   "storeRoot": ".tt-local/store",
-  "dryRun": true,
-  "training": {
-    "backend": "uv",
-    "project": "training/local-runner"
+  "paths": {
+    "modelCache": "~/.cache/huggingface"
   },
   "evaluation": {
     "inference": {
-      "provider": "none"
+      "device": "cuda",
+      "maxNewTokens": 32,
+      "temperature": 0,
+      "topP": 1
     },
     "scoring": {
       "mode": "exact_match"
     },
-    "maxExamples": 2
+    "generalRegression": {
+      "dataset": "general-regression.jsonl",
+      "maxScoreDrop": 0.03,
+      "maxPassRateDrop": 0.05
+    },
+    "maxExamples": 2,
+    "timeoutMs": 1800000
   }
 }

package/examples/smoke-spec.json

@@ -0,0 +1,41 @@
+{
+  "id": "22222222-2222-4222-8222-222222222222",
+  "name": "Local Smoke Assistant",
+  "description": "Tiny local runner smoke test.",
+  "system_prompt": "Return exactly one sentiment label.",
+  "guidelines": [
+    "Reply with positive or negative."
+  ],
+  "constraints": [
+    "Do not include extra commentary."
+  ],
+  "base_model": "Qwen/Qwen3.5-2B",
+  "examples": [
+    {
+      "input": "Classify sentiment: I love this product.",
+      "output": "positive"
+    },
+    {
+      "input": "Classify sentiment: This was disappointing.",
+      "output": "negative"
+    },
+    {
+      "input": "Classify sentiment: The update works perfectly.",
+      "output": "positive"
+    },
+    {
+      "input": "Classify sentiment: Support never replied.",
+      "output": "negative"
+    }
+  ],
+  "hyperparameters": {
+    "n_epochs": 1,
+    "learning_rate": 0.0002,
+    "batch_size": 1,
+    "gradient_accumulation_steps": 1,
+    "lora_rank": 4,
+    "lora_alpha": 8,
+    "max_seq_length": 256,
+    "max_eval_examples": 2
+  }
+}

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@tuned-tensor/local",
-  "version": "0.2.9",
-  "description": "Local-first fine-tuning with baseline-vs-tuned evaluation for small open-weight models.",
+  "version": "0.4.0",
+  "description": "Local CUDA LoRA fine-tuning with held-out base-versus-tuned evaluation.",
   "license": "Apache-2.0",
   "private": false,
   "repository": {
@@ -28,22 +28,22 @@
     "LICENSE"
   ],
   "scripts": {
-    "build": "tsc -p tsconfig.json",
+    "clean": "node -e \"require('node:fs').rmSync('dist',{recursive:true,force:true})\"",
+    "build": "npm run clean && tsc -p tsconfig.json",
     "prepack": "npm run build",
     "typecheck": "tsc -p tsconfig.json --noEmit",
-    "test": "node --import tsx --test \"test/**/*.test.ts\""
+    "test": "node --import tsx --test \"test/**/*.test.ts\"",
+    "test:python": "python3 -m unittest discover -s training/local-runner/tests -p 'test_*.py'"
   },
   "engines": {
     "node": ">=22"
   },
   "devDependencies": {
-    "@types/better-sqlite3": "^7.6.13",
     "@types/node": "^25.0.0",
     "tsx": "^4.22.4",
     "typescript": "^6.0.0"
   },
   "dependencies": {
-    "better-sqlite3": "^12.11.1",
     "zod": "^4.4.3"
   }
 }

package/training/local-runner/pyproject.toml

@@ -4,16 +4,11 @@ version = "0.1.0"
 requires-python = ">=3.11"
 dependencies = [
   "accelerate>=1.0.0",
-  "datasets>=4.0.0",
   "huggingface-hub>=0.36.0",
   "peft>=0.13.0",
-  "pillow>=10.0.0",
-  "qwen-vl-utils>=0.0.14",
   "safetensors>=0.4.5",
   "torch>=2.11.0",
-  "torchvision>=0.24.0",
   "transformers>=4.57.0",
-  "trl>=1.7.0",
 ]
 
 [tool.uv.sources]