@tuned-tensor/cli 0.24.0 → 0.26.0

package/README.md

@@ -72,7 +72,7 @@ npm install -g --ignore-scripts @tuned-tensor/cli
 tt --version
 ```
 
-Upgrade later with `tt upgrade`. Uninstall with `npm uninstall -g @tuned-tensor/cli` (add
+`TT` works the same as `tt`. Upgrade later with `tt upgrade`. Uninstall with `npm uninstall -g @tuned-tensor/cli` (add
 `--prefix ~/.local` if the curl installer used that prefix).
 
 Node.js 22.19.0 or newer is required. Training additionally needs
@@ -95,7 +95,7 @@ npm link
 
 `tunedtensor.json` is the source of truth for behavior and training settings.
 In the shell, `/spec` shows a readable overview grouped into behavior,
-examples, training, evaluation and runtime sections; `/spec <section>` (for
+dataset, examples, training, evaluation and runtime sections; `/spec <section>` (for
 example `/spec examples`) shows one section in full and `/spec show` prints the
 highlighted JSON. `/spec diff`, `/spec validate`, and `/spec history` complete
 the review, all without a model call. Outside the shell, `tt spec view` and
@@ -176,21 +176,22 @@ the agent:
 ```text
 tt v0.15.0
 agent tunedtensor/managed · workflow model base
-ctrl+c stop/clear · ctrl+d exit · /help commands · tab complete
+tt › What would you like to work on?
+/help commands · ctrl+c stop · ctrl+d exit
 
-Ask TT anything. Known commands run directly.
 › What happened in my latest training run?
 › runs list
 ```
 
-Ordinary sentences go through the locally orchestrated model session. The
+Known commands such as `runs list` run directly; tab completes them. Ordinary
+sentences go through the locally orchestrated model session. The
 model has no shell or general filesystem tool. It can read and propose reviewed
-edits to an existing spec, or prepare one new folder directly beneath the shell's current working directory with a validated
-`tunedtensor.json`, or prepare a local pipeline dry-run sealed to the reviewed
+edits to an existing spec, or prepare a validated `tunedtensor.json` in the shell's current working directory
+when none exists, or prepare a local pipeline dry-run sealed to the reviewed
 workspace and spec contents. Writes and pipeline previews wait for your approval: after the
 answer, tt shows the proposal and asks inline, where `y` approves, `n` rejects and `Esc`
 leaves it pending for a later `/approve` or `/reject`. Any other key is ignored. The tools refuse path
-traversal, symlinked workspace roots, overwriting a new-project target, unsupported spec
+traversal, symlinked workspace roots, overwriting an existing spec, unsupported spec
 fields, cloud pipeline targets, and spec changes after review. Known CLI
 commands such as `runs list`, `doctor`, and `models list` still execute directly.
 Prefix a command with `:` when you want to make that intent explicit.
@@ -205,7 +206,35 @@ search public Hugging Face model or dataset metadata for foundation and
 fine-tuning discovery. A Hub search sends only the search query to
 `huggingface.co`; it uses no Hugging Face token, downloads nothing, and omits
 model-card and dataset-card text. Search results are discovery metadata, not a
-guarantee that a model or dataset is compatible with a TT workflow.
+guarantee that a model or dataset is compatible with a TT workflow. The agent
+can inspect a public dataset's exact JSONL/Parquet filenames, sizes, revision,
+and declared columns/splits before proposing a download. Declared schema comes
+from dataset-card metadata; it is not validation of the file contents.
+
+To reuse a download, ask “Which datasets have we downloaded?” followed by
+“Use CodeAlpaca in this spec for fine-tuning.” The agent can list TT-recorded
+local datasets and inspect actual Parquet columns/row counts or sampled JSONL
+fields, plus two bounded sample rows. It proposes a `dataset_prebuilt` pointer
+with the cached revision, training file, separate validation/test file and
+column mapping. Approval saves the edit; listing and inspection need no
+download or approval. Existing examples and model settings stay in the spec,
+but prebuilt data supplies the training rows instead of automatically mixing
+in the inline examples. Samples do not establish full dataset quality, and a
+download with only a training split still needs distinct held-out data.
+
+When a conversation approaches the model's context budget, TT automatically
+summarizes earlier turns using Pi and the selected model, and shows progress.
+The trigger reserves the full reply allowance plus 4,096 tokens for protocol
+headroom; it does not apply an additional percentage reserve. Usage measured
+before a summary is ignored, while new provider usage calibrates the next
+trigger, including after resume.
+The summary preserves goals, decisions and action outcomes and is saved for
+`/resume`; recent exchanges remain in context. Summarization uses an additional
+model request (or several for unusually long histories). The transcript stays
+in local history subject to the existing storage limits, and pending approvals
+remain intact. If the current request is still too large, summarization fails,
+or the model stops at its token limit, TT reports the problem instead of silently
+returning to the prompt. A failed summary does not replace the saved one.
 
 For educational questions about how or why training works, the agent can read
 the exact Python implementation shipped with the running TT build. The source
@@ -234,20 +263,63 @@ sealed as `outcome_unknown` for manual inspection. `/reject` never mutates.
 Pipeline dry-runs also wait for `/approve`. Preparation fingerprints the
 workspace directory identity, spec, and adjacent config, revalidates them,
 copies the resolved spec, config, and pipeline into private temporary inputs,
-and invokes `tt pipeline run --dry-run` deterministically. Persisted proposals
-that request real execution fail before the child command is dispatched. Real
-training remains available through the explicit direct `tt pipeline run` command.
+and invokes `tt pipeline run --dry-run` deterministically. A dry-run proposal
+edited to request real execution fails before the child command is dispatched.
+After a dry-run of the same spec bytes completes, the agent can propose a real
+background run with a reason; approval starts it in a detached process and the
+conversation continues. The agent reports progress with read-only tools and can
+cancel a run only through another approval. Each run keeps an audit record
+(reason, requester, choices, resources, outcome and metrics), a hash-chained event
+log and its full log; `TT_MAX_PARALLEL_RUNS` limits concurrent runs. See
+[background runs](docs/local-runtime/background-runs.md). The direct
+`tt pipeline run` and `tt pipeline start` commands remain available.
 A changed workspace or spec requires a new review.
 
+The startup **What you can do** panel covers specs/datasets, decision training,
+LLM fine-tuning, training from scratch, and serving even before a spec exists.
+Decision training can use CPU or Apple MPS; the other training paths currently
+require CUDA locally or a configured AWS GPU. A missing NVIDIA GPU does not
+prevent spec creation or dataset preparation. The banner lists ready workflows on
+one line and shows only the status of the rest; `/system` adds reasons and next steps.
+
+Guidance distinguishes **Ready**, **Needs setup**, **Limited resources**,
+**Remote GPU**, **Needs verification**, and **Unsupported runtime**. Quick
+startup sampling does not install Python dependencies. `tt hardware` explicitly
+checks the locked adapter and decision runtimes separately, including MPS
+availability; it may install those runtimes on first use. It does not download
+models or certify their fit. Old snapshots without a decision probe show
+"Needs verification" until refreshed.
+
+| Platform | Decision training | LLM / foundation training | Packaged LLM serving |
+| --- | --- | --- | --- |
+| Linux x64 / ARM64 | CPU; CUDA when available | NVIDIA CUDA, subject to memory and runtime checks | Linux + NVIDIA CUDA; separate runtime |
+| Apple Silicon macOS | CPU / MPS after runtime checks | Remote AWS GPU | Remote Linux/CUDA host |
+| Native Windows x64 | CPU path; end-to-end training not yet certified | Not certified natively; check a Linux/WSL2 environment or use AWS | Linux/CUDA host |
+| Intel macOS / Windows ARM64 | No compatible wheels in the current lockfiles | Remote AWS GPU | Remote Linux/CUDA host |
+
+The native Windows CI job verifies hardware discovery and displayed guidance,
+not model training. WSL is assessed as its own Linux environment. AMD/Intel GPU
+acceleration is not enabled by the packaged CUDA runtime. Decision resource
+warnings use conservative advisory thresholds (8 GiB available working memory,
+6 GiB free model-cache disk, 4 CPU cores); they are not measured model minima.
+Batch size, sequence length, training mode and other running applications matter.
+CPU evaluation does not imply that `tt serve` can run on CPU. Agent chat still
+needs managed access or your own model provider; local data commands need neither.
+
 Useful shell controls include `/help`, `/status`, `/context`, `/system`,
-`/model`, `/login`, `/cd`, `/clear`, and `/exit`.
+`/usage` (managed agent allowance and token usage), `/model`, `/login`, `/cd`, `/clear`, and `/exit`.
 When the shell opens, the TT logo mark (drawn in the terminal) greets you with the current spec, a
 live machine panel (GPU utilization, VRAM and temperature from `nvidia-smi`,
-CPU, RAM, and free space in the model cache) and whether this machine can
+CPU, RAM, and free space on the model-cache filesystem) and whether this machine can
 fine-tune the spec's base model, plus a suggested next step. `/system` refreshes
 that panel and lists every certified base model's verdict. Fit verdicts come
 from the same certified memory profiles as `tt hardware`; a fresh
 `tt hardware` snapshot takes precedence over the quick live check.
+On macOS, RAM marked `in use (est.)` excludes estimated reclaimable file cache;
+decision-training warnings use the same available-RAM estimate. This is not a
+memory-pressure measurement or a guarantee that a workload fits. CPU load is a
+one-minute average, separate from the sampled utilization percentage. Disk usage
+describes the filesystem containing the cache, not the size of downloaded models.
 The shell banner and `/context` show the inference selection (`agent
 provider/model`) separately from the workflow model. The managed alias identifies
 inference through the server-selected model. Run `/model` with no arguments to
@@ -545,12 +617,73 @@ tt datasets verify tunedtensor.json     # offline checksum check of the cached f
 tt pipeline run --spec tunedtensor.json # downloads automatically if still missing
 ```
 
-- Split paths are files inside the repo and must be JSONL. Parquet and CSV are
-  not supported yet.
+In `tt agent`, ask it to download a dataset or model. It proposes the same
+`tt datasets prefetch` or `tt models prefetch` command and runs it after you
+approve. The source does not have to be in the spec yet: download first, look at
+it, then ask the agent to adopt it, and it proposes a spec edit pinned to the
+commit the download resolved. A download alone needs a training filename, not
+a validation split or column mapping; those are required when configuring training.
+Approval checks the workspace, spec and adjacent legacy runner configuration,
+then runs a private snapshot of the reviewed settings. Later edits cannot redirect
+the running download. After a failure or cancellation, ask for a new download
+proposal to resume from the cache.
+
+### Downloads and the lock file
+
+Any Hugging Face dataset files or model can be downloaded without editing a spec:
+
+```bash
+tt datasets prefetch --repo HuggingFaceH4/ultrachat_200k \
+  --training data/train_sft-00000-of-00003.parquet --test data/test_sft-00000-of-00001.parquet
+tt models prefetch --repo Qwen/Qwen3-4B-Instruct-2507   # add --revision <sha> to pin
+```
+
+Direct downloads also work before a spec exists or while it is invalid; those
+workspaces use the default cache. An approved agent download reports the commit
+from that command's result for a later spec edit.
+
+Files stay in the Hugging Face cache (`HF_HOME`, or `runtime.paths.modelCache`).
+Each download also writes a manifest with the repo, commit, file digests and
+location under `~/.tuned-tensor/assets/`, so downloads are easy to find later:
+
+```bash
+tt assets list                 # everything downloaded, newest first
+tt assets path org/name        # where it is on disk (org/name@<sha> for one revision)
+tt assets verify [org/name]    # re-hash the files against their manifests
+```
+
+`tunedtensor.lock.json` beside the spec pins what a project trains on, the way
+`.terraform.lock.hcl` pins providers: the dataset and base-model repo, commit,
+and each file's Hugging Face content digest (`sha256:` for LFS files,
+`git-sha1:` for small git files). `tt datasets prefetch` and `tt models prefetch`
+for the spec write it, and the first real run records any entry it is missing.
+After that, `tt pipeline run`:
+
+- uses the locked commit when the spec's dataset does not pin `revision`;
+- re-hashes the cached bytes and stops if the spec names another repo or revision,
+  a cached file's digest differs, or a locked model file is missing. If the change is intended, rerun the matching
+  `prefetch` command for the spec to update the lock.
+
+A first local run with an empty model cache downloads and locks the model before
+training starts. Dry runs never download assets or write the lock.
+
+An unpinned dataset stays on its locked commit, including for `tt datasets prefetch`.
+To move it to a newer commit, pin that `revision` in the spec and prefetch again.
+
+Commit the lock file with the spec. Runs that load the base model from
+`runtime.paths.baseModel` or on an AWS GPU host check only the dataset.
+
+- Split paths are files inside the repo: one `.jsonl` or `.parquet` file per
+  split (pick one shard of a sharded split). CSV is not supported yet.
+- Parquet supports Snappy, gzip, zstd, Brotli and LZ4 compression, as well as
+  uncompressed files.
 - Without `columns`, files must already be TT chat JSONL whose system message
   matches the spec. With `columns`, each record's `input` and `output` fields
-  become chat rows that use the spec's instruction. Records missing either
-  field are skipped. Evaluation splits keep the first of any duplicate input,
+  become chat rows that use the spec's instruction. Parquet files require
+  `columns`. For chat datasets such as `HuggingFaceH4/ultrachat_200k`, use
+  `"columns": { "messages": "messages" }`: each conversation's first user turn
+  and the assistant reply that follows become one example; later turns are not
+  used. Records missing a mapped field are skipped. Evaluation splits keep the first of any duplicate input,
   and training drops inputs that also appear in an evaluation split. The run
   report lists every skipped or dropped count.
 - Without `revision`, TT uses the cached `main` snapshot and records the commit
@@ -567,6 +700,13 @@ tt pipeline run --spec tunedtensor.json # downloads automatically if still missi
 `--verbose` adds training and evaluation process output, and `--quiet` hides
 both. Full logs stay in the run directory either way.
 
+Adapter evaluation batches four prompts by default, reports
+completed examples and ETA, and saves completed batches for recovery. Set
+`evaluation.inference.batchSize` to tune memory use; `evaluation.maxExamples`
+and `sampleSeed` select an explicit smoke sample. Matching interrupted
+baselines automatically continue on retry with the default baseline cache.
+See [evaluation settings and recovery](docs/local-runtime/architecture.md#evaluation).
+
 Every run report has a `provenance` block recording what produced it:
 
 - TT, Node and platform versions, and the digest of the locked Python runtime;
@@ -671,6 +811,26 @@ For the full command reference, including long-example policies, eval token
 budgets, evaluation caps, and local model serving, see the
 [CLI docs](https://tunedtensor.com/docs/cli).
 
+### Split a master dataset
+
+Keep converted datasets as unsplit chat JSONL. Downloaded chat JSONL can be
+split directly from its verified cache: ask the agent to inspect the download
+and use its pinned `split_source`, with no manual copy. Ask it to split a named
+file with your ratios and seed, then attach the approved paths to your spec.
+For example: “Split data.jsonl 80/20 for training and validation with seed 123,
+stratify by metadata.category, and save to split-123.”
+
+```sh
+tt datasets inspect data.jsonl
+tt datasets split data.jsonl --output split-123 --train .8 --validation .2 --test 0 --seed 123
+```
+
+The source stays unchanged. Groups containing repeated prompts stay in training
+so held-out prompts remain unique;
+optional `--group-by` and `--stratify-by` support family and label boundaries.
+See [splitting local data](docs/spec-workflow.md#splitting-an-unsplit-dataset)
+for defaults, metadata preservation, and benchmark holdout limitations.
+
 ## Development
 
 ```bash