@tuned-tensor/cli 0.4.11 → 0.4.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +19 -1
- package/dist/index.js +691 -61
- package/dist/index.js.map +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -100,6 +100,9 @@ tt models base
|
|
|
100
100
|
tt models list
|
|
101
101
|
tt models get <model-id>
|
|
102
102
|
tt models download <model-id> --output model.tar.gz
|
|
103
|
+
tt models setup-runtime
|
|
104
|
+
tt models serve <model-id> --spec tunedtensor.json
|
|
105
|
+
tt models serve ./model --port 8000
|
|
103
106
|
```
|
|
104
107
|
|
|
105
108
|
Use `--dataset <dataset-id-or-prefix>` with `tt runs start` to train from an uploaded dataset instead of inline spec examples. Add `--train-ratio`, `--validation-ratio`, and `--test-ratio` to override the default 80/10/10 split.
|
|
@@ -108,7 +111,22 @@ Use `--max-eval-examples <n>` and `--max-test-eval-examples <n>` with `tt runs s
|
|
|
108
111
|
|
|
109
112
|
Use `--no-llm-judge` with `tt runs start` to opt out of LLM judging for a new run.
|
|
110
113
|
|
|
111
|
-
`tt models download` downloads models that have a Tuned Tensor-hosted artifact. Hosted models can still be used for inference through their model ID, but may not expose downloadable weights.
|
|
114
|
+
`tt models download` downloads models that have a Tuned Tensor-hosted artifact. In interactive terminals it shows download progress, transfer rate, and ETA; `--json` output remains machine-readable. Hosted models can still be used for inference through their model ID, but may not expose downloadable weights.
|
|
115
|
+
|
|
116
|
+
`tt models setup-runtime` installs an isolated local Python runtime for reference serving. It chooses Python 3.10-3.12, creates a managed venv in the Tuned Tensor cache, and installs `torch`, `transformers`, `accelerate`, and `safetensors`. This can take a few minutes the first time.
|
|
117
|
+
|
|
118
|
+
`tt models serve` starts a local OpenAI-compatible reference server for a model ID, an extracted model directory, or a `.tar.gz` artifact. It auto-detects `tunedtensor.json` in the current directory or model directory and applies the compiled behavior spec as the default system prompt, so inference matches the prompt contract used during training. Use `--spec <path>` to point at a specific spec, or `--no-spec-prompt` when you intentionally want raw model behavior. If no compatible Python serving runtime is available, run `tt models setup-runtime` first.
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
tt models setup-runtime
|
|
122
|
+
tt models serve <model-id> --spec tunedtensor.json
|
|
123
|
+
tt models serve <model-id> --spec tunedtensor.json --device mps # Apple Silicon
|
|
124
|
+
tt models serve <model-id> --spec tunedtensor.json --device cuda # NVIDIA GPU
|
|
125
|
+
|
|
126
|
+
curl http://127.0.0.1:8000/v1/chat/completions \
|
|
127
|
+
-H 'content-type: application/json' \
|
|
128
|
+
-d '{"messages":[{"role":"user","content":"Hello"}]}'
|
|
129
|
+
```
|
|
112
130
|
|
|
113
131
|
## Billing & Credits
|
|
114
132
|
|