@tuned-tensor/cli 0.4.12 → 0.4.14

package/README.md

@@ -1,6 +1,12 @@
 # tt - Tuned Tensor CLI
 
-`tt` is the command-line tool for [Tuned Tensor](https://www.tunedtensor.com), used to define behavior specs, validate them, and launch fine-tuning runs.
+`tt` is the command-line tool for [Tuned Tensor](https://www.tunedtensor.com).
+Use it to define behaviour specs, validate them, launch fine-tuning runs,
+manage datasets, and download or serve trained models.
+
+The main CLI documentation lives at
+[tunedtensor.com/docs/cli](https://tunedtensor.com/docs/cli). This README is a
+short install and development reference.
 
 ## Install
 
@@ -12,7 +18,7 @@ tt --version
 Run from source:
 
 ```bash
-git clone https://github.com/tuned-tensor/tuned-tensor-cli.git
+git clone https://github.com/tunedtensor/tuned-tensor-cli.git
 cd tuned-tensor-cli
 npm install
 npm run build
@@ -21,155 +27,31 @@ npm link
 
 ## Quick Start
 
-1) **Authenticate**
-
 ```bash
 tt auth login
-tt auth status
-```
-
-2) **Create a local spec**
-
-```bash
-tt init
-# or:
-tt init --name "Customer Support Bot" --model "Qwen/Qwen3.5-2B"
-```
-
-Supported spec base models are `Qwen/Qwen3.5-2B`, `Qwen/Qwen3.5-4B`, `google/gemma-4-E2B-it`, `google/gemma-4-E4B-it`, `meta-llama/Llama-3.2-3B-Instruct`, `microsoft/Phi-4-mini-instruct`, `ibm-granite/granite-3.3-2b-instruct`, and `bigcode/starcoder2-3b`.
+tt init --name "Customer Support Bot" --model Qwen/Qwen3.5-2B
 
-You can print the same list from the CLI:
-
-```bash
-tt models base
-tt models base --json
-```
-
-3) **Validate your spec**
-
-```bash
+# Edit tunedtensor.json, then:
 tt eval
-```
-
-4) **Push your spec**
-
-```bash
 tt push
-```
-
-5) **Start and watch a run**
-
-```bash
 tt runs start <spec-id>
-tt runs start <spec-id> --dataset <dataset-id-or-prefix> --train-ratio 0.8 --validation-ratio 0.1 --test-ratio 0.1
-tt runs start <spec-id> --no-llm-judge
 tt runs watch <run-id>
 ```
 
-Tip: use `tt specs list`, `tt datasets list`, `tt runs list`, and `tt models list` to find IDs. Spec, run, dataset, and model commands accept full UUIDs or unambiguous ID prefixes.
-
-## Typical Workflows
+Useful discovery commands:
 
 ```bash
-# Account
-tt auth status
-tt balance
-tt topup --amount 25
-
-# Specs
 tt specs list
-tt specs get <spec-id>
-tt specs create --file spec.json
-tt specs update <spec-id> --file updates.json
-
-# Runs
-tt runs list --spec <spec-id>
-tt runs get <run-id>
-tt runs start <spec-id> --epochs 5 --lr 0.0001 --batch-size 8
-tt runs start <spec-id> --dataset <dataset-id-or-prefix> --train-ratio 0.8 --validation-ratio 0.1 --test-ratio 0.1
-tt runs start <spec-id> --no-llm-judge
-tt runs cancel <run-id>
-
-# Datasets
-tt datasets upload data.jsonl --name "Support Training Set"
 tt datasets list
-tt datasets get <dataset-id>
-
-# Models
-tt models base
+tt runs list
 tt models list
-tt models get <model-id>
-tt models download <model-id> --output model.tar.gz
-tt models setup-runtime
-tt models serve <model-id> --spec tunedtensor.json
-tt models serve ./model --port 8000
-```
-
-Use `--dataset <dataset-id-or-prefix>` with `tt runs start` to train from an uploaded dataset instead of inline spec examples. Add `--train-ratio`, `--validation-ratio`, and `--test-ratio` to override the default 80/10/10 split.
-
-Use `--max-eval-examples <n>` and `--max-test-eval-examples <n>` with `tt runs start` to cap primary and secondary test evaluation passes for larger datasets; the runs backend still clamps values to its configured ceiling.
-
-Use `--no-llm-judge` with `tt runs start` to opt out of LLM judging for a new run.
-
-`tt models download` downloads models that have a Tuned Tensor-hosted artifact. In interactive terminals it shows download progress, transfer rate, and ETA; `--json` output remains machine-readable. Hosted models can still be used for inference through their model ID, but may not expose downloadable weights.
-
-`tt models setup-runtime` installs an isolated local Python runtime for reference serving. It chooses Python 3.10-3.12, creates a managed venv in the Tuned Tensor cache, and installs `torch`, `transformers`, `accelerate`, and `safetensors`. This can take a few minutes the first time.
-
-`tt models serve` starts a local OpenAI-compatible reference server for a model ID, an extracted model directory, or a `.tar.gz` artifact. It auto-detects `tunedtensor.json` in the current directory or model directory and applies the compiled behavior spec as the default system prompt, so inference matches the prompt contract used during training. Use `--spec <path>` to point at a specific spec, or `--no-spec-prompt` when you intentionally want raw model behavior. If no compatible Python serving runtime is available, run `tt models setup-runtime` first.
-
-```bash
-tt models setup-runtime
-tt models serve <model-id> --spec tunedtensor.json
-tt models serve <model-id> --spec tunedtensor.json --device mps  # Apple Silicon
-tt models serve <model-id> --spec tunedtensor.json --device cuda # NVIDIA GPU
-
-curl http://127.0.0.1:8000/v1/chat/completions \
-  -H 'content-type: application/json' \
-  -d '{"messages":[{"role":"user","content":"Hello"}]}'
-```
-
-## Billing & Credits
-
-Tuned Tensor uses prepaid credits. New accounts start at a zero balance, so top up before starting your first fine-tuning run; you only pay for successful runs.
-
-```bash
-tt balance                 # show credit balance and recent transactions
-tt topup --amount 25       # opens Stripe Checkout in your browser
-tt topup --amount 25 --no-open  # print the URL instead
-```
-
-`tt balance` shows your current credit balance. If a run is rejected with
-`402 insufficient_credits`, top up and retry.
-
-## Spec Validation
-
-`tt eval` validates your local `tunedtensor.json`. It checks required fields, confirms examples are present, warns when guidelines are missing, and checks simple constraints against example outputs. It does not call a model or the Playground API.
-
-## Global Flags
-
-- `-k, --api-key <key>`: override stored API key
-- `-u, --base-url <url>`: override API base URL
-- `--json`: machine-readable output
-- `--no-color`: disable ANSI colors
-- `-h, --help`: command help
-
-Examples:
-
-```bash
-tt specs list --json
-tt runs get <run-id> --json
-tt runs start --help
+tt models base
+tt balance
 ```
 
-## Configuration
-
-Credentials are stored in `~/.config/tuned-tensor/config.json` (respects `XDG_CONFIG_HOME`).
-
-API key precedence:
-
-1. `--api-key`
-2. `TUNED_TENSOR_API_KEY`
-3. stored config
+For the full command reference, including dataset-backed runs, evaluation caps,
+local model serving, configuration, and billing, see the
+[CLI docs](https://tunedtensor.com/docs/cli).
 
 ## Development
 
@@ -181,10 +63,6 @@ npm run typecheck
 npm test
 ```
 
-## Troubleshooting
-
-If the API rejects a spec with a generic server error, check that `base_model` is one of the supported spec base models listed above.
-
 ## License
 
 MIT

package/dist/index.js

@@ -615,8 +615,13 @@ function printDiagnostics(diagnostics) {
     ["Stage", diagnostics.stage_label ?? diagnostics.stage ?? void 0],
     ["Summary", diagnostics.summary],
     ["Training", diagnostics.training.state],
+    ["Phase", diagnostics.training.phase ?? void 0],
     ["Epoch", formatEpochProgress(diagnostics)],
     ["Latest Loss", curve.latest_loss == null ? void 0 : curve.latest_loss.toFixed(4)],
+    [
+      "Token Accuracy",
+      curve.latest_token_accuracy == null ? void 0 : (curve.latest_token_accuracy * 100).toFixed(1) + "%"
+    ],
     ["Pace", perFive ? `${perFive} epoch / 5m` : void 0],
     ["ETA", formatMinutes(curve.estimated_minutes_remaining)],
     ["Latest Update", formatDate(diagnostics.training.last_updated_at)]
@@ -1086,6 +1091,10 @@ DEFAULT_MAX_TOKENS = int(os.environ.get("TT_MAX_TOKENS", "512"))
 DEFAULT_TEMPERATURE = float(os.environ.get("TT_TEMPERATURE", "0.7"))
 TRUST_REMOTE_CODE = os.environ.get("TT_TRUST_REMOTE_CODE", "false").lower() in {"1", "true", "yes", "y"}
 REQUESTED_DEVICE = os.environ.get("TT_DEVICE", "auto").lower()
+JSON_REPAIR_ATTEMPTS = int(os.environ.get("TT_JSON_REPAIR_ATTEMPTS", "1"))
+DEFAULT_JSON_SCHEMA = None
+if os.environ.get("TT_JSON_SCHEMA"):
+    DEFAULT_JSON_SCHEMA = json.loads(os.environ["TT_JSON_SCHEMA"])
 
 
 def choose_device(torch):
@@ -1174,6 +1183,50 @@ def normalize_messages(raw_messages):
     return messages
 
 
+def response_json_contract(body):
+    response_format = body.get("response_format")
+    schema = DEFAULT_JSON_SCHEMA
+    mode = "json_schema" if schema else "text"
+
+    if isinstance(response_format, dict):
+        format_type = response_format.get("type")
+        if format_type == "json_object":
+            mode = "json_object"
+        elif format_type == "json_schema":
+            mode = "json_schema"
+            json_schema = response_format.get("json_schema")
+            if isinstance(json_schema, dict):
+                schema = json_schema.get("schema") if isinstance(json_schema.get("schema"), dict) else json_schema
+
+    return mode, schema
+
+
+def with_json_contract(messages, mode, schema):
+    if mode == "text":
+        return messages
+
+    if schema:
+        contract = (
+            "Return only a valid JSON object matching this JSON Schema. "
+            "Do not include markdown, code fences, comments, or prose outside the JSON object.\n"
+            + json.dumps(schema, ensure_ascii=False, separators=(",", ":"))
+        )
+    else:
+        contract = (
+            "Return only a valid JSON object. Do not include markdown, code fences, "
+            "comments, or prose outside the JSON object."
+        )
+
+    contracted = [dict(message) for message in messages]
+    for message in contracted:
+        if message["role"] == "system":
+            message["content"] = (message["content"].rstrip() + "\n\n" + contract).strip()
+            return contracted
+
+    contracted.insert(0, {"role": "system", "content": contract})
+    return contracted
+
+
 def render_prompt(messages):
     chat_template = getattr(tokenizer, "chat_template", None)
     if hasattr(tokenizer, "apply_chat_template") and chat_template:
@@ -1214,6 +1267,142 @@ def generate_completion(messages, max_tokens, temperature):
     return content, prompt_tokens, int(completion_ids.shape[-1])
 
 
+def extract_json_value(content):
+    decoder = json.JSONDecoder()
+    text = content.strip()
+    try:
+        return json.loads(text), None
+    except Exception:
+        pass
+
+    for index, char in enumerate(text):
+        if char not in "{[":
+            continue
+        try:
+            value, end = decoder.raw_decode(text[index:])
+            trailing = text[index + end :].strip()
+            if trailing:
+                continue
+            return value, None
+        except Exception:
+            continue
+    return None, "response is not valid JSON"
+
+
+def json_type_matches(value, expected):
+    if expected == "object":
+        return isinstance(value, dict)
+    if expected == "array":
+        return isinstance(value, list)
+    if expected == "string":
+        return isinstance(value, str)
+    if expected == "boolean":
+        return isinstance(value, bool)
+    if expected == "integer":
+        return isinstance(value, int) and not isinstance(value, bool)
+    if expected == "number":
+        return (isinstance(value, int) or isinstance(value, float)) and not isinstance(value, bool)
+    if expected == "null":
+        return value is None
+    return True
+
+
+def validate_json_schema(value, schema, path="$"):
+    if not isinstance(schema, dict):
+        return []
+
+    errors = []
+    expected_type = schema.get("type")
+    if isinstance(expected_type, list):
+        if not any(json_type_matches(value, item) for item in expected_type):
+            errors.append(f"{path} has wrong type")
+            return errors
+    elif isinstance(expected_type, str) and not json_type_matches(value, expected_type):
+        errors.append(f"{path} must be {expected_type}")
+        return errors
+
+    if "enum" in schema and value not in schema["enum"]:
+        errors.append(f"{path} must be one of {schema['enum']}")
+
+    if isinstance(value, dict):
+        required = schema.get("required") if isinstance(schema.get("required"), list) else []
+        for key in required:
+            if key not in value:
+                errors.append(f"{path}.{key} is required")
+
+        properties = schema.get("properties") if isinstance(schema.get("properties"), dict) else {}
+        for key, child_schema in properties.items():
+            if key in value:
+                errors.extend(validate_json_schema(value[key], child_schema, f"{path}.{key}"))
+
+        if schema.get("additionalProperties") is False:
+            extras = sorted(set(value.keys()) - set(properties.keys()))
+            for key in extras:
+                errors.append(f"{path}.{key} is not allowed")
+
+    if isinstance(value, list) and isinstance(schema.get("items"), dict):
+        for index, item in enumerate(value):
+            errors.extend(validate_json_schema(item, schema["items"], f"{path}[{index}]"))
+
+    return errors
+
+
+def normalize_json_content(content, mode, schema):
+    if mode == "text":
+        return content, []
+
+    value, error = extract_json_value(content)
+    if error:
+        return content, [error]
+    if mode in {"json_object", "json_schema"} and not isinstance(value, dict):
+        return content, ["response must be a JSON object"]
+
+    errors = validate_json_schema(value, schema) if schema else []
+    if errors:
+        return content, errors
+
+    return json.dumps(value, ensure_ascii=False, separators=(",", ":")), []
+
+
+def generate_validated_completion(messages, max_tokens, temperature, mode, schema):
+    contracted_messages = with_json_contract(messages, mode, schema)
+    attempts = max(0, JSON_REPAIR_ATTEMPTS) + 1
+    last_content = ""
+    last_prompt_tokens = 0
+    last_completion_tokens = 0
+    last_errors = []
+
+    for attempt in range(attempts):
+        content, prompt_tokens, completion_tokens = generate_completion(
+            contracted_messages,
+            max_tokens,
+            temperature,
+        )
+        normalized, errors = normalize_json_content(content, mode, schema)
+        if not errors:
+            return normalized, prompt_tokens, completion_tokens, []
+
+        last_content = content
+        last_prompt_tokens = prompt_tokens
+        last_completion_tokens = completion_tokens
+        last_errors = errors
+
+        if attempt < attempts - 1:
+            contracted_messages = contracted_messages + [
+                {"role": "assistant", "content": content},
+                {
+                    "role": "user",
+                    "content": (
+                        "The previous response was invalid: "
+                        + "; ".join(errors[:5])
+                        + ". Return only a corrected JSON object."
+                    ),
+                },
+            ]
+
+    return last_content, last_prompt_tokens, last_completion_tokens, last_errors
+
+
 class Handler(BaseHTTPRequestHandler):
     server_version = "TunedTensorReferenceServer/0.1"
 
@@ -1262,7 +1451,23 @@ class Handler(BaseHTTPRequestHandler):
             max_tokens = int(body.get("max_tokens") or DEFAULT_MAX_TOKENS)
             temperature = float(body.get("temperature") if body.get("temperature") is not None else DEFAULT_TEMPERATURE)
             messages = normalize_messages(raw_messages)
-            content, prompt_tokens, completion_tokens = generate_completion(messages, max_tokens, temperature)
+            json_mode, json_schema = response_json_contract(body)
+            content, prompt_tokens, completion_tokens, validation_errors = generate_validated_completion(
+                messages,
+                max_tokens,
+                temperature,
+                json_mode,
+                json_schema,
+            )
+            if validation_errors:
+                self.send_json(422, {
+                    "error": {
+                        "message": "Model did not produce valid JSON for the requested response_format.",
+                        "details": validation_errors,
+                        "content": content,
+                    }
+                })
+                return
 
             self.send_json(200, {
                 "id": "chatcmpl-" + uuid.uuid4().hex,
@@ -1542,6 +1747,18 @@ function loadSystemPromptFromSpec(specPath) {
   const spec = JSON.parse(readFileSync6(specPath, "utf8"));
   return buildSystemPrompt(spec);
 }
+function loadJsonSchemaForServe(schemaPath) {
+  const resolved = resolve2(schemaPath);
+  if (!existsSync4(resolved) || !statSync3(resolved).isFile()) {
+    throw new Error(`JSON schema file not found: ${schemaPath}`);
+  }
+  try {
+    const schema = JSON.parse(readFileSync6(resolved, "utf8"));
+    return JSON.stringify(schema);
+  } catch (err) {
+    throw new Error(`JSON schema file is not valid JSON: ${schemaPath}`);
+  }
+}
 function writeReferenceServerScript(cacheDir) {
   const scriptDir = join2(cacheDir, "_server");
   mkdirSync2(scriptDir, { recursive: true });
@@ -1721,7 +1938,7 @@ function registerModelsCommands(parent) {
     printSuccess(`Serving runtime ready at ${venvDir}`);
     console.log(`Use it with: tt models serve <model-id> --spec ${DEFAULT_SPEC_FILE}`);
   });
-  models.command("serve").description("Serve a downloaded model with an OpenAI-compatible local API").argument("<target>", "Model ID/prefix, model directory, or .tar.gz artifact").option("--spec <path>", "Behavior spec JSON to apply as the default system prompt").option("--no-spec-prompt", "Do not auto-apply a behavior spec system prompt").option("--host <host>", "Host to bind", "127.0.0.1").option("--port <port>", "Port to bind", "8000").option("--python <path>", "Python executable").option("--cache-dir <path>", "Cache directory for downloaded/extracted models").option("--device <device>", "Inference device: auto, cpu, cuda, or mps", "auto").option("--force-download", "Re-download and re-extract model artifacts").option("--max-tokens <n>", "Default max completion tokens", "512").option("--temperature <n>", "Default sampling temperature", "0.7").option("--trust-remote-code", "Pass trust_remote_code=True to transformers").option("--print-command", "Print the underlying Python command without starting it").action(async (target, cmdOpts) => {
+  models.command("serve").description("Serve a downloaded model with an OpenAI-compatible local API").argument("<target>", "Model ID/prefix, model directory, or .tar.gz artifact").option("--spec <path>", "Behavior spec JSON to apply as the default system prompt").option("--no-spec-prompt", "Do not auto-apply a behavior spec system prompt").option("--host <host>", "Host to bind", "127.0.0.1").option("--port <port>", "Port to bind", "8000").option("--python <path>", "Python executable").option("--cache-dir <path>", "Cache directory for downloaded/extracted models").option("--device <device>", "Inference device: auto, cpu, cuda, or mps", "auto").option("--force-download", "Re-download and re-extract model artifacts").option("--max-tokens <n>", "Default max completion tokens", "512").option("--temperature <n>", "Default sampling temperature", "0.7").option("--json-schema <path>", "JSON Schema file to enforce by default for chat completions").option("--json-repair-attempts <n>", "Number of JSON/schema repair retries", "1").option("--trust-remote-code", "Pass trust_remote_code=True to transformers").option("--print-command", "Print the underlying Python command without starting it").action(async (target, cmdOpts) => {
     const opts = parent.opts();
     const cacheDir = resolve2(cmdOpts.cacheDir || defaultCacheDir());
     mkdirSync2(cacheDir, { recursive: true });
@@ -1752,10 +1969,12 @@ function registerModelsCommands(parent) {
       TT_PORT: String(cmdOpts.port),
       TT_MAX_TOKENS: String(cmdOpts.maxTokens),
       TT_TEMPERATURE: String(cmdOpts.temperature),
+      TT_JSON_REPAIR_ATTEMPTS: String(cmdOpts.jsonRepairAttempts),
       TT_TRUST_REMOTE_CODE: cmdOpts.trustRemoteCode ? "true" : "false",
       TT_DEVICE: String(cmdOpts.device)
     };
     if (systemPrompt) env.TT_SYSTEM_PROMPT = systemPrompt;
+    if (cmdOpts.jsonSchema) env.TT_JSON_SCHEMA = loadJsonSchemaForServe(cmdOpts.jsonSchema);
     const args = [serverScript];
     if (cmdOpts.printCommand) return printServeCommand(python, args, env);
     if (!isJsonMode()) {
@@ -2086,7 +2305,7 @@ function registerPushCommand(parent) {
 
 // src/index.ts
 var program = new Command();
-program.name("tt").description("Tuned Tensor CLI \u2014 fine-tune and evaluate LLMs").version("0.4.12").option("-k, --api-key <key>", "API key (overrides stored key)").option(
+program.name("tt").description("Tuned Tensor CLI \u2014 fine-tune and evaluate LLMs").version("0.4.14").option("-k, --api-key <key>", "API key (overrides stored key)").option(
   "-u, --base-url <url>",
   "API base URL (default: https://tunedtensor.com)"
 ).option("--json", "Output raw JSON").option("--no-color", "Disable colors").hook("preAction", (_thisCommand, actionCommand) => {