@tuned-tensor/cli 0.16.0 → 0.18.0

package/docs/local-runtime/aws-gpu.md

@@ -1,27 +1,74 @@
 # Use your AWS GPU with the local pipeline
 
-TT runs one pipeline on your laptop. Add `gpu` to `local-runner.json` to send
+TT runs one pipeline on your laptop. Add `runtime.gpu` to `tunedtensor.json` to send
 adapter training and CUDA evaluation, or foundation pretraining, fine-tuning,
 RL and evaluation, to an existing EC2 instance in your account. Foundation
 tokenization, dataset preparation, adapter scoring/comparisons, run history,
 and artifact validation stay local. Results return to the usual local
 directories. The laptop does not need an NVIDIA GPU.
 
-Keep the existing spec and pipeline. Save this config beside `tunedtensor.json`,
-replacing the example values with your own. You can also copy
-[`examples/local-runtime/aws-runner.json`](../../examples/local-runtime/aws-runner.json).
-All runner paths refer to the laptop; TT translates the paths needed remotely.
+## 1. Install TT and create a project
+
+Use CLI 0.16.0 or newer with Node.js 22.19+ on Linux or macOS:
+
+```bash
+npm install -g --ignore-scripts @tuned-tensor/cli
+tt --version
+mkdir my-aws-training && cd my-aws-training
+tt init --name "My AWS Adapter" --model Qwen/Qwen3.5-2B
+```
+
+Edit the generated `tunedtensor.json`: replace the placeholder system prompt and both placeholder examples with
+your desired instructions and real input/output pairs. For a small functional
+trial, use the [four-example smoke spec](../../examples/local-runtime/smoke-spec.json)
+instead. It checks the workflow, not model quality. An existing project can keep
+its spec and pipeline unchanged.
+
+## 2. Prepare your AWS instance
+
+You need two separate kinds of access: an **AWS profile** to find the instance,
+and an **SSH key or agent** to execute commands on it. A TT login is unnecessary.
+If you do not already have an instance, launch one in your AWS account or ask
+your administrator to provide one with the prerequisites below.
+
+Our CLI 0.16.0 smoke test used a `g5.xlarge` (A10G), an AWS Deep Learning Base
+GPU AMI with Ubuntu 24.04, and a 120 GiB root disk in `eu-west-1`. This is a
+tested small-workload configuration, not a sizing guarantee for other models.
+Use a driver compatible with the bundled CUDA runtime; `tt doctor` checks it.
+
+Before launching, check the EC2 GPU quota in your chosen region. For the tested
+G-family On-Demand instance, the quota is **Running On-Demand G and VT instances**
+(`L-DB2E81BA`), measured in vCPUs; one `g5.xlarge` needs 4 available vCPUs.
+If it is too low, follow [AWS's quota increase procedure](https://docs.aws.amazon.com/servicequotas/latest/userguide/request-quota-increase.html).
+Approval does not guarantee instance availability. TT does not request capacity
+or choose a different region for you.
+
+During launch, select your [EC2 SSH key pair](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html),
+allow inbound TCP 22 from your laptop's IP (or use your private network), and
+record the instance ID, region, SSH username and IP. Follow
+[AWS's SSH connection prerequisites](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connect-to-linux-instance.html).
+Instance creation and quota management need separate AWS permissions; TT itself
+only needs `ec2:DescribeInstances` for lookup.
+
+## 3. Configure the connection
+
+Add this `runtime` section to your existing `tunedtensor.json`, replacing the
+example values with your own. All paths refer to the laptop; TT translates
+paths needed remotely. Existing sidecars can be consolidated with
+`tt pipeline migrate`, which retains `.bak` copies.
 
 ```json
 {
-  "gpu": {
-    "provider": "aws",
-    "profile": "research",
-    "region": "eu-west-1",
-    "instanceId": "i-0123456789abcdef0",
-    "user": "ubuntu",
-    "identityFile": "~/.ssh/research-gpu.pem",
-    "maxSeconds": 86400
+  "runtime": {
+    "gpu": {
+      "provider": "aws",
+      "profile": "research",
+      "region": "eu-west-1",
+      "instanceId": "i-0123456789abcdef0",
+      "user": "ubuntu",
+      "identityFile": "~/.ssh/research-gpu.pem",
+      "maxSeconds": 86400
+    }
   }
 }
 ```
@@ -49,7 +96,7 @@ Verify the instance's SSH host key for the selected IP address and connect
 once yourself before using TT. TT requires a known host key and noninteractive
 SSH authentication. Omit
 `identityFile` to use your SSH agent or SSH configuration. Relative identity
-file paths resolve beside the runner config; `~` expands locally. Set
+file paths resolve beside the core spec; `~` expands locally. Set
 `privateIp: true` when connected to the instance's VPC through a VPN or other
 private network. The default uses its public IPv4 address. Security groups and
 network routes must allow the laptop to reach that address over SSH.
@@ -57,26 +104,62 @@ TT addresses the instance by IP, so any `Host` rules in your SSH configuration
 must match that IP. Remote Python commands run through `bash -lc`, which loads
 the login profile so a normal uv installation can be found.
 
+After configuring your AWS profile (for SSO, use `aws configure sso` once),
+check access from the same terminal that will run TT:
+
+```bash
+# SSO profiles only; skip this line for other credential methods.
+aws sso login --profile research
+aws sts get-caller-identity --profile research
+aws ec2 describe-instances --profile research --region eu-west-1 \
+  --instance-ids i-0123456789abcdef0 \
+  --query 'Reservations[0].Instances[0].[State.Name,PublicIpAddress]' --output text
+```
+
+Use the returned IP below. First verify its host fingerprint through your AWS
+console or administrator and connect interactively to record the verified key.
+Then confirm that unattended SSH works:
+
+```bash
+chmod 400 ~/.ssh/research-gpu.pem
+ssh -o BatchMode=yes -o StrictHostKeyChecking=yes \
+  -i ~/.ssh/research-gpu.pem ubuntu@YOUR_INSTANCE_IP \
+  'bash -lc "command -v uv rsync setsid timeout; nvidia-smi"'
+```
+
+If `uv` is missing, install it on that machine using the
+[uv installation instructions](https://docs.astral.sh/uv/getting-started/installation/)
+and make it available in the login shell. Install the other listed prerequisites
+with that machine's package manager. Run the SSH check again before training.
+
+## 4. Preview and run
+
+Run these commands on your laptop in the project directory. Adapter workflow:
+
+```bash
+# Fetch the base model locally first.
+tt validate tunedtensor.json
+tt models prefetch tunedtensor.json
+tt doctor tunedtensor.json
+tt pipeline run --spec tunedtensor.json --dry-run
+tt pipeline run --spec tunedtensor.json
+```
+
+For a foundation spec, no base-model prefetch is needed. Start a new run with:
+
 ```bash
-# Adapter workflow: fetch the base model locally first.
 tt validate tunedtensor.json
-tt models prefetch tunedtensor.json --config local-runner.json
-tt doctor tunedtensor.json --config local-runner.json
-tt pipeline run --spec tunedtensor.json --config local-runner.json --dry-run
-tt pipeline run --spec tunedtensor.json --config local-runner.json
-
-# Foundation workflow: no base-model prefetch is needed.
-tt doctor tunedtensor.json --config local-runner.json
-tt pipeline run --spec tunedtensor.json --config local-runner.json \
-  --resume /absolute/path/to/foundation-run
+tt doctor tunedtensor.json
+tt pipeline run --spec tunedtensor.json \
+  --dry-run
+tt pipeline run --spec tunedtensor.json
 ```
 
 Choose the command block for your spec's engine. `tt doctor` connects to the
 instance and prepares/probes its locked Python/CUDA runtime, which may take
 time on first use. It does not start training or reserve GPU capacity.
 `tt hardware` reports the laptop's hardware; it does not inspect the AWS
-instance. `tt pipeline run` discovers `local-runner.json` beside the selected
-spec when `--config` is omitted.
+instance. `tt pipeline run` reads `runtime.gpu` from the selected spec.
 
 Keep pipeline step targets set to `local`: the target describes the
 orchestrator, and `gpu` selects where its GPU processes execute. CPU adapter
@@ -84,6 +167,28 @@ evaluation (`evaluation.inference.device: "cpu"`) stays on the laptop.
 Remove `gpu` to execute everything locally. `--dry-run` or `dryRun: true`
 does not connect to AWS or copy files.
 
+## 5. Inspect results and finish
+
+Wait for a successful completion message. Adapter runs return a run ID and a
+local model ID; use the actual IDs from the output:
+
+```bash
+tt runs list
+tt runs report RUN_ID
+tt models list
+tt models verify MODEL_ID
+```
+
+For foundation runs, inspect `report.json` in the run directory printed by the
+command. To resume it, pass `--resume /absolute/path/to/foundation-run` to the
+same `tt pipeline run` command. Keep the laptop awake and connected until the
+command finishes and outputs have returned. Review the report before another run.
+
+Stop or terminate your instance in AWS when finished, according to your own
+retention needs. TT does not do this for you, even after cancellation or timeout.
+
+## Transfers and recovery
+
 TT copies only each process's declared data, model snapshot, and runtime files,
 not the workspace or laptop credentials. Adapter runs use an immutable local
 model snapshot; the Hugging Face credential/cache root is never uploaded.
@@ -129,3 +234,16 @@ The instance remains running after the job; its lifecycle is yours to manage.
 Hosted training submission (`tt cloud runs start` and `estimate`) is retired.
 Existing cloud reports and cancellation of older hosted jobs remain available.
 This CLI change does not delete deployed AWS infrastructure or historical data.
+
+## Troubleshooting a first run
+
+| Symptom | What to check |
+| --- | --- |
+| AWS credentials expired or access denied | Renew your SSO login, check the selected profile/account, and ask for `ec2:DescribeInstances` permission. |
+| Instance not found or not running | Confirm the instance ID and region, and start the instance through AWS. |
+| GPU quota or capacity error during launch | Check the regional quota and AWS capacity; a TT token or credit top-up will not fix it. |
+| SSH timeout or permission denied | Check IP routing, security group TCP 22 access, SSH username, key and agent using the manual SSH command above. |
+| Host key verification failed | Verify the host fingerprint through AWS or your administrator; do not disable strict host checking. |
+| `uv` missing or CUDA probe fails | Check the remote login-shell PATH and GPU driver; rerun `tt doctor` after correcting the host. |
+| Long pause before the GPU process starts | Model inputs are uploaded for every process. Check network speed and free space on both machines, especially remote `/tmp`. |
+| Download fails or staging directory already exists | Preserve the remote directory shown in the error and recover its outputs before retrying. Do not remove the only checkpoint copy. |

package/docs/local-runtime/serving.md

@@ -38,7 +38,7 @@ not mutate the training runtime.
 ```bash
 tt models prefetch tunedtensor.json
 
-tt serve local-<run-id> --config local-runner.json \
+tt serve local-<run-id> \
   --context-length 8192 --max-tokens 512
 ```
 
@@ -81,7 +81,7 @@ Export a new isolated configuration rather than replacing global settings:
 
 ```bash
 export PI_CODING_AGENT_DIR="$(mktemp -d)"
-tt serve local-<run-id> --config local-runner.json \
+tt serve local-<run-id> \
   --context-length 8192 --max-tokens 512 \
   --print-client-config pi > "$PI_CODING_AGENT_DIR/models.json"
 ```

package/docs/local-runtime/spark.md

@@ -78,28 +78,23 @@ Edit both generated examples in `tunedtensor.json`. For a meaningful run,
 replace them with a larger, representative dataset and a separate validation
 split.
 
-The generated `local-runner.json` uses CUDA and project-local artifacts. A
-durable Spark configuration can set:
+Initialization creates only `tunedtensor.json`; CUDA and project-local storage
+are defaults. For durable Spark paths, add this optional section to the spec:
 
 ```json
 {
-  "artifactRoot": "/home/eve/tuned-tensor-runs/artifacts",
-  "storeRoot": "/home/eve/tuned-tensor-runs/store",
-  "paths": {
-    "modelCache": "/home/eve/.cache/huggingface"
-  },
-  "evaluation": {
-    "inference": {
-      "device": "cuda"
-    },
-    "scoring": {
-      "mode": "exact_match"
-    },
-    "timeoutMs": 1800000
+  "runtime": {
+    "artifactRoot": "/home/eve/tuned-tensor-runs/artifacts",
+    "storeRoot": "/home/eve/tuned-tensor-runs/store",
+    "paths": { "modelCache": "/home/eve/.cache/huggingface" }
   }
 }
 ```
 
+Existing projects can use `tt pipeline migrate` to fold sidecar settings into
+the spec while keeping backups.
+
+
 Every Python stage uses the locked runtime included in the npm package; a
 source checkout and a custom runner path are neither required nor supported.
 
@@ -110,7 +105,7 @@ tt doctor tunedtensor.json
 tt validate tunedtensor.json
 tt models prefetch tunedtensor.json
 tt models verify-base tunedtensor.json
-tt pipeline run --spec tunedtensor.json --config local-runner.json
+tt pipeline run --spec tunedtensor.json
 ```
 
 `doctor` resolves the same bundled project and paths the run will use, imports
@@ -136,7 +131,7 @@ tt serve local-<run-id> --spec tunedtensor.json --port 8000
 ```
 
 `tt serve active` fails unless an adapter is activated. Activation requires a
-`generalRegression` suite in `local-runner.json`. Serve a specific adapter
+`generalRegression` suite in `tunedtensor.json`. Serve a specific adapter
 with `tt serve local-<run-id>` until then.
 
 In another shell: