@tuned-tensor/local 0.2.6 → 0.2.8

package/CHANGELOG.md

@@ -2,6 +2,66 @@
 
 All notable changes to TT Local will be documented in this file.
 
+## 0.2.8 - 2026-07-14
+
+### Added
+
+- Added strict side-effect-free CLI help, `--version`, exact-runtime `doctor`
+  checks, and a Spark initialization profile.
+- Added atomic artifact manifests, `tt-local models verify`, and local
+  OpenAI-compatible serving of verified PEFT adapters.
+- Added local-only `tt-local models verify-base` snapshot validation and made
+  deliberately staged runs terminate as `stage_completed`.
+- Added process-group cancellation and immediate post-training model
+  registration so a valid model survives later evaluation/report failures.
+- Included the bundled Python `uv.lock` in the npm package.
+- Added durable run/artifact ownership claims, workflow locks, detached runs,
+  terminal stage/cancellation states, and `runs watch` lifecycle handling.
+
+### Changed
+
+- Standardized `paths.modelCache` as Hugging Face `HF_HOME` across prefetch,
+  training, evaluation, doctor, and serving; prefetch now reports the resolved
+  snapshot revision, file count, and byte size.
+- Made `exact_match` the runnable scoring default. Explicit LLM-judge mode now
+  requires valid configuration/credentials unless exact-match fallback was
+  deliberately selected, and fallback results are never cached as judge data.
+- Resume invalidation now fingerprints the complete request and effective
+  runner configuration, local data/assets/base snapshot, immutable model
+  revision, parent artifact, and bundled runtime rather than reusing outputs
+  trained with stale inputs.
+- Relative config paths resolve from the config file and relative spec data
+  paths resolve from the spec/request file.
+- Stored parent models now verify their artifact and immutable base identity;
+  continuation runs inherit the recorded base revision and reject conflicts.
+- Baseline caching now requires stable content identity and bypasses mutable
+  remote models/assets. Optional progress reporters can no longer crash or
+  orphan child processes.
+
+### Fixed
+
+- Prevented help, unknown flags, missing values, and option typos from starting
+  work accidentally.
+- Rejected empty or corrupt model outputs before registration and cleared stale
+  terminal errors after successful retries.
+- Rejected incomplete PEFT/full-model contracts, optimizer-only payloads,
+  artifact-prefix collisions, and symlink escapes before publication.
+- Disabled optional PyTorch native JIT kernels by default for bundled ML
+  processes so packaged CUDA inference does not require system Python headers;
+  explicit runner environments can opt back in.
+- Made interactive model-server shutdown close cleanly without a Python
+  traceback.
+- Stopped reporting model checkpoint shard loading as optimizer progress.
+
+## 0.2.7 - 2026-07-05
+
+### Added
+
+- Added `tt-local models prefetch [tunedtensor.json|request.json]` to download
+  the configured Hugging Face base model into `paths.modelCache` before the
+  first real run, making first-time setup explicit instead of hiding the
+  download inside baseline evaluation or training.
+
 ## 0.2.6 - 2026-07-05
 
 ### Fixed

package/README.md

@@ -31,6 +31,146 @@ The default uv project is included in the npm package at
 `training/local-runner`; using bundled training does not require cloning this
 repository.
 
+## First Local Run
+
+On an NVIDIA Spark or another CUDA host, initialize both the behavior spec and
+a durable project-local runner config:
+
+```bash
+tt-local --version
+tt-local init --name "Support Bot" --model Qwen/Qwen3.5-2B --profile spark
+```
+
+Edit `tunedtensor.json` to replace the generated system prompt and example.
+TT Local refuses to validate or train the unchanged placeholder so an
+accidental run cannot spend GPU time on it.
+
+Then preflight the exact Python environments and paths that the run will use:
+
+```bash
+tt-local doctor tunedtensor.json --config local-runner.json
+tt-local validate tunedtensor.json --config local-runner.json
+```
+
+`doctor` checks the bundled uv environment, Torch/Transformers/PEFT imports,
+the requested CUDA/MPS device, judge credentials when configured, writable
+artifact/cache/store roots, and available disk space. GPU checks are skipped
+for dry runs and are conditional for CPU/MPS configurations.
+
+Download the base model explicitly, then start the run:
+
+```bash
+tt-local models prefetch tunedtensor.json --config local-runner.json
+tt-local models verify-base tunedtensor.json --config local-runner.json
+tt-local run tunedtensor.json --config local-runner.json
+```
+
+Prefetch progress is visible by default. Pass `--quiet` only when another
+program is consuming the final JSON. `verify-base` is local-only: it checks
+the cached snapshot structure, indexed weight shards, tokenizer files, and
+non-empty weights without downloading a replacement. Cached Hugging Face blobs
+whose ETags encode Git or LFS checksums are re-hashed as part of verification.
+
+For a long run, start it in the background and watch it from the current or a
+second terminal:
+
+```bash
+tt-local run tunedtensor.json --config local-runner.json --detach
+tt-local runs watch <run-id> --config local-runner.json
+```
+
+The detached command returns the persisted run ID, process ID, log path, and
+copy-paste watch/cancel commands before model work begins.
+
+After training:
+
+```bash
+tt-local runs list --config local-runner.json
+tt-local runs report <run-id> --config local-runner.json
+tt-local models verify local-<run-id> --config local-runner.json
+```
+
+The model is registered immediately after its artifact passes verification,
+so it remains discoverable if candidate evaluation or report generation later
+fails.
+
+## Serve a Trained Adapter
+
+`tt-local serve` is the run dashboard. To serve a trained TT Local adapter as
+an OpenAI-compatible model endpoint, use the nested model command:
+
+```bash
+tt-local models serve local-<run-id> \
+  --config local-runner.json \
+  --spec tunedtensor.json \
+  --host 127.0.0.1 \
+  --port 8000
+```
+
+The server verifies the stored artifact, loads the recorded base model and
+PEFT adapter through the bundled uv environment, and binds to localhost by
+default. Test it from another terminal:
+
+```bash
+curl http://127.0.0.1:8000/health
+curl http://127.0.0.1:8000/v1/models
+curl http://127.0.0.1:8000/v1/chat/completions \
+  -H 'Content-Type: application/json' \
+  -d '{"messages":[{"role":"user","content":"Hello"}]}'
+```
+
+Use `--device cuda|mps|cpu|auto` to override the configured inference device.
+Non-loopback binds require both `--allow-remote` and
+`--api-key-env <environment-variable>`; requests must then send that value as
+a Bearer token. The server accepts multimodal images only as bounded `data:`
+URIs and does not support streaming yet.
+
+## Cache and Evaluation Semantics
+
+`paths.modelCache` means Hugging Face `HF_HOME` everywhere. Hub snapshots live
+under `<modelCache>/hub`; prefetch, training, evaluation, doctor, and serving
+all use that same layout. Set `HF_TOKEN` in the environment or project `.env`
+before prefetching a gated model. Do not commit `.env`.
+
+For reproducible remote loading, set
+`hyperparameters.base_model_revision` to an immutable Hugging Face commit.
+Alternatively, `paths.baseModel` may point to a complete local Hugging Face
+snapshot directory. It cannot point to a model archive or standalone weights
+file; TT Local requires non-empty model weights, `config.json`, tokenizer
+metadata and vocabulary, and every indexed shard. Local snapshot contents are
+fingerprinted so a byte change invalidates dependent stages.
+
+Shared baseline-cache entries are used only when the model and every local
+input have a stable content identity. An unpinned remote model or remotely
+hosted image bypasses the cache rather than risking stale evaluation results.
+
+The safe default scoring mode is deterministic `exact_match`. To use an
+OpenRouter judge, configure `evaluation.scoring.mode: "llm_judge"`, an `llm`
+block, and its API-key environment variable. Judge mode fails preflight when
+those are missing. Exact-match fallback occurs only when explicitly requested
+with `evaluation.scoring.fallback: "exact_match"`; fallback-scored results are
+marked and never written to the judge baseline cache.
+
+Before the first real run, you can download the configured Hugging Face base
+model into `paths.modelCache` explicitly:
+
+```bash
+tt-local models prefetch tunedtensor.json --config local-runner.json
+tt-local models verify-base tunedtensor.json --config local-runner.json
+```
+
+Without this step, the first non-dry `tt-local run` downloads the base model
+when baseline evaluation or training first loads it.
+
+Individual `run --stage ...` commands finish in the terminal
+`stage_completed` state, so `runs watch` exits instead of waiting for stages
+that were intentionally not requested.
+
+Relative paths in a runner config resolve from the config file's directory.
+Relative dataset and image paths resolve from the behavior spec or request
+file's directory. The published package includes `uv.lock` so bundled Python
+dependencies are reproducible.
+
 ## DPO
 
 TT Local supports first-class offline DPO for text causal-LM models. Set
@@ -65,6 +205,8 @@ example `external:karpathy/nanochat`. The bundled `uv` trainer still requires a
 supported Hugging Face base model, but command trainers receive adapter-focused
 hyperparameters without injected LoRA/model-loader defaults. Custom
 hyperparameter keys are passed through to `TT_HYPERPARAMETERS_PATH`.
+Set `hyperparameters.base_model_revision` to an immutable Hugging Face commit
+when a run must load and record an exact base-model revision.
 
 ## Continuing From a Fine-Tuned Model
 
@@ -86,6 +228,13 @@ For config-only workflows, put the same value in
 baseline for the new run, and the bundled SFT/DPO trainers load it before
 continuing LoRA training.
 
+Stored `--parent-model` references are verified before launch. TT Local pins
+the child's base revision from the parent (or requires the same configured
+local snapshot content), rejects conflicting base identity, and fingerprints
+the parent artifact so changed bytes invalidate reuse. The explicit
+`--parent-model-artifact` form is for caller-managed local artifacts and should
+be used only when that provenance is understood.
+
 Non-Hugging Face artifacts can describe their layout in `training.artifact`:
 
 ```json

package/dist/artifacts.d.ts

@@ -8,6 +8,7 @@ export interface RunArtifacts {
     baselineEvalJson: string;
     candidateEvalJson: string;
     runReportJson: string;
+    artifactManifestJson: string;
     progressJsonl: string;
     trainingDir: string;
     trainingInputDir: string;
@@ -16,6 +17,14 @@ export interface RunArtifacts {
     trainingModelDir: string;
     trainingLog: string;
 }
+export declare const ARTIFACT_WORKFLOW_LOCK_FILE = ".tt-local-workflow.lock";
+export interface ArtifactRunOwner {
+    schema_version: 1;
+    run_id: string;
+    user_id: string;
+    behavior_spec_id: string;
+    created_at: string;
+}
 export declare function fileUri(path: string): string;
 export declare function defaultArtifactPrefix(input: {
     userId: string;
@@ -26,8 +35,28 @@ export declare function resolveRunArtifacts(args: {
     artifactRoot: string;
     prefix: string;
 }): RunArtifacts;
+/**
+ * Creates the run directory one component at a time and rejects symlinked
+ * descendants. Lexical `resolve()` containment alone is insufficient because
+ * an existing child symlink could redirect later writes or recursive cleanup
+ * outside `artifactRoot`.
+ */
+export declare function ensureSafeRunDirectory(artifacts: RunArtifacts): Promise<void>;
+/**
+ * Durably binds an artifact directory to one run identity. The owner survives
+ * workflow-lock release, preventing a later run with a colliding custom
+ * prefix from deleting or silently reusing another run's artifacts.
+ */
+export declare function claimRunArtifactDirectory(args: {
+    artifacts: RunArtifacts;
+    runId: string;
+    userId: string;
+    behaviorSpecId: string;
+}): Promise<ArtifactRunOwner>;
 export declare function prepareRunDirectories(artifacts: RunArtifacts): Promise<void>;
 export declare function writeJson(path: string, value: unknown): Promise<void>;
+export declare function writeJsonAtomic(path: string, value: unknown): Promise<void>;
+export declare function writeFileAtomic(path: string, value: string | Uint8Array): Promise<void>;
 export declare function readJson<T>(path: string): Promise<T>;
 export declare function appendProgress(artifacts: RunArtifacts, event: {
     stage: string;
@@ -35,3 +64,66 @@ export declare function appendProgress(artifacts: RunArtifacts, event: {
     details?: Record<string, unknown>;
 }): Promise<void>;
 export declare function copyDatasetToTrainingChannel(artifacts: RunArtifacts): Promise<string>;
+export interface ArtifactManifestFile {
+    path: string;
+    size_bytes: number;
+    sha256: string;
+}
+export interface ArtifactManifest {
+    schema_version: 1;
+    generated_at: string;
+    files: ArtifactManifestFile[];
+    model?: ArtifactManifestModel;
+}
+export interface ArtifactManifestModel {
+    artifact_kind: "file" | "directory";
+    format: string;
+    framework: string;
+    base_model: string;
+    base_model_revision?: string;
+    base_model_artifact_uri?: string;
+    base_model_fingerprint?: string;
+    parent_model_artifact?: string;
+    artifact_uri: string;
+    artifact_root: string;
+    servable: boolean;
+    files: ArtifactManifestFile[];
+}
+export interface ArtifactManifestVerification {
+    valid: boolean;
+    checked: number;
+    missing: string[];
+    changed: string[];
+    unexpected: string[];
+}
+export interface TarGzipVerification {
+    entries: number;
+    payload_entries: number;
+    recognized_payload_entries: number;
+    recognized_payload_bytes: number;
+    adapter_weight_entries: number;
+    adapter_weight_bytes: number;
+    full_model_weight_entries: number;
+    full_model_weight_bytes: number;
+    adapter_config_entries: number;
+    expanded_bytes: number;
+}
+/** Atomically snapshots every immutable file currently present in a run directory. */
+export declare function writeArtifactManifest(artifacts: RunArtifacts, options?: {
+    model?: Omit<ArtifactManifestModel, "files">;
+}): Promise<ArtifactManifest>;
+/** Re-hashes a persisted run and reports missing, changed, and untracked files. */
+export declare function verifyArtifactManifest(manifestPath: string, options?: {
+    requiredPaths?: string[];
+    allowUnexpected?: boolean;
+    scopeToRequired?: boolean;
+    verifyModel?: boolean;
+}): Promise<ArtifactManifestVerification>;
+/** Validates gzip CRC/truncation, tar headers, paths, and bounded PAX metadata. */
+export declare function verifyTarGzipArchive(path: string): Promise<TarGzipVerification>;
+export declare function assertArtifactManifest(manifestPath: string, options?: {
+    requiredPaths?: string[];
+    allowUnexpected?: boolean;
+    scopeToRequired?: boolean;
+    verifyModel?: boolean;
+}): Promise<ArtifactManifestVerification>;