@tuned-tensor/cli 0.23.0 → 0.25.0

package/README.md

@@ -205,7 +205,31 @@ search public Hugging Face model or dataset metadata for foundation and
 fine-tuning discovery. A Hub search sends only the search query to
 `huggingface.co`; it uses no Hugging Face token, downloads nothing, and omits
 model-card and dataset-card text. Search results are discovery metadata, not a
-guarantee that a model or dataset is compatible with a TT workflow.
+guarantee that a model or dataset is compatible with a TT workflow. The agent
+can inspect a public dataset's exact JSONL/Parquet filenames, sizes, revision,
+and declared columns/splits before proposing a download. Declared schema comes
+from dataset-card metadata; it is not validation of the file contents.
+
+To reuse a download, ask “Which datasets have we downloaded?” followed by
+“Use CodeAlpaca in this spec for fine-tuning.” The agent can list TT-recorded
+local datasets and inspect actual Parquet columns/row counts or sampled JSONL
+fields, plus two bounded sample rows. It proposes a `dataset_prebuilt` pointer
+with the cached revision, training file, separate validation/test file and
+column mapping. Approval saves the edit; listing and inspection need no
+download or approval. Existing examples and model settings stay in the spec,
+but prebuilt data supplies the training rows instead of automatically mixing
+in the inline examples. Samples do not establish full dataset quality, and a
+download with only a training split still needs distinct held-out data.
+
+When a conversation approaches the model's context budget, TT automatically
+summarizes earlier turns using Pi and the selected model, and shows progress.
+The summary preserves goals, decisions and action outcomes and is saved for
+`/resume`; recent exchanges remain in context. Summarization uses an additional
+model request (or several for unusually long histories). The transcript stays
+in local history subject to the existing storage limits, and pending approvals
+remain intact. If the current request is still too large, summarization fails,
+or the model stops at its token limit, TT reports the problem instead of silently
+returning to the prompt. A failed summary does not replace the saved one.
 
 For educational questions about how or why training works, the agent can read
 the exact Python implementation shipped with the running TT build. The source
@@ -545,12 +569,73 @@ tt datasets verify tunedtensor.json     # offline checksum check of the cached f
 tt pipeline run --spec tunedtensor.json # downloads automatically if still missing
 ```
 
-- Split paths are files inside the repo and must be JSONL. Parquet and CSV are
-  not supported yet.
+In `tt agent`, ask it to download a dataset or model. It proposes the same
+`tt datasets prefetch` or `tt models prefetch` command and runs it after you
+approve. The source does not have to be in the spec yet: download first, look at
+it, then ask the agent to adopt it, and it proposes a spec edit pinned to the
+commit the download resolved. A download alone needs a training filename, not
+a validation split or column mapping; those are required when configuring training.
+Approval checks the workspace, spec and adjacent legacy runner configuration,
+then runs a private snapshot of the reviewed settings. Later edits cannot redirect
+the running download. After a failure or cancellation, ask for a new download
+proposal to resume from the cache.
+
+### Downloads and the lock file
+
+Any Hugging Face dataset files or model can be downloaded without editing a spec:
+
+```bash
+tt datasets prefetch --repo HuggingFaceH4/ultrachat_200k \
+  --training data/train_sft-00000-of-00003.parquet --test data/test_sft-00000-of-00001.parquet
+tt models prefetch --repo Qwen/Qwen3-4B-Instruct-2507   # add --revision <sha> to pin
+```
+
+Direct downloads also work before a spec exists or while it is invalid; those
+workspaces use the default cache. An approved agent download reports the commit
+from that command's result for a later spec edit.
+
+Files stay in the Hugging Face cache (`HF_HOME`, or `runtime.paths.modelCache`).
+Each download also writes a manifest with the repo, commit, file digests and
+location under `~/.tuned-tensor/assets/`, so downloads are easy to find later:
+
+```bash
+tt assets list                 # everything downloaded, newest first
+tt assets path org/name        # where it is on disk (org/name@<sha> for one revision)
+tt assets verify [org/name]    # re-hash the files against their manifests
+```
+
+`tunedtensor.lock.json` beside the spec pins what a project trains on, the way
+`.terraform.lock.hcl` pins providers: the dataset and base-model repo, commit,
+and each file's Hugging Face content digest (`sha256:` for LFS files,
+`git-sha1:` for small git files). `tt datasets prefetch` and `tt models prefetch`
+for the spec write it, and the first real run records any entry it is missing.
+After that, `tt pipeline run`:
+
+- uses the locked commit when the spec's dataset does not pin `revision`;
+- re-hashes the cached bytes and stops if the spec names another repo or revision,
+  a cached file's digest differs, or a locked model file is missing. If the change is intended, rerun the matching
+  `prefetch` command for the spec to update the lock.
+
+A first local run with an empty model cache downloads and locks the model before
+training starts. Dry runs never download assets or write the lock.
+
+An unpinned dataset stays on its locked commit, including for `tt datasets prefetch`.
+To move it to a newer commit, pin that `revision` in the spec and prefetch again.
+
+Commit the lock file with the spec. Runs that load the base model from
+`runtime.paths.baseModel` or on an AWS GPU host check only the dataset.
+
+- Split paths are files inside the repo: one `.jsonl` or `.parquet` file per
+  split (pick one shard of a sharded split). CSV is not supported yet.
+- Parquet supports Snappy, gzip, zstd, Brotli and LZ4 compression, as well as
+  uncompressed files.
 - Without `columns`, files must already be TT chat JSONL whose system message
   matches the spec. With `columns`, each record's `input` and `output` fields
-  become chat rows that use the spec's instruction. Records missing either
-  field are skipped. Evaluation splits keep the first of any duplicate input,
+  become chat rows that use the spec's instruction. Parquet files require
+  `columns`. For chat datasets such as `HuggingFaceH4/ultrachat_200k`, use
+  `"columns": { "messages": "messages" }`: each conversation's first user turn
+  and the assistant reply that follows become one example; later turns are not
+  used. Records missing a mapped field are skipped. Evaluation splits keep the first of any duplicate input,
   and training drops inputs that also appear in an evaluation split. The run
   report lists every skipped or dropped count.
 - Without `revision`, TT uses the cached `main` snapshot and records the commit