veris-cli 2.22.2__tar.gz → 2.24.0__tar.gz

{veris_cli-2.22.2 → veris_cli-2.24.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: veris-cli
-Version: 2.22.2
+Version: 2.24.0
 Summary: CLI to connect local agents to the Veris backend
 Project-URL: Homepage, https://github.com/veris-ai/veris-cli
 Project-URL: Bug Tracker, https://github.com/veris-ai/veris-cli/issues
@@ -9,6 +9,7 @@ Requires-Python: >=3.11
 Requires-Dist: click>=8.1.7
 Requires-Dist: httpx>=0.27.0
 Requires-Dist: pathspec>=0.12.0
+Requires-Dist: pydantic>=2.6
 Requires-Dist: pyyaml>=6.0.1
 Requires-Dist: questionary>=2.0.0
 Requires-Dist: rich>=13.7.0
@@ -58,7 +59,7 @@ This saves your credentials to `~/.veris/config.yaml`.
 ### 2. Create an Environment
 
 ```bash
-veris env create --name "my-agent"
+veris env create --name my-agent-env --agent-name "My Agent"
 ```
 
 This scaffolds a `.veris/` directory and registers your environment on Veris:
@@ -66,6 +67,8 @@ This scaffolds a `.veris/` directory and registers your environment on Veris:
 - **`veris.yaml`** — Simulation configuration (services, persona, agent settings)
 - **`.dockerignore`** — Files excluded from image build
 
+The `--name` you pass is the agent's display name. The CLI slugifies it to create a target/env name: `"My Agent"` becomes `my-agent-env`. This name is used as the top-level key in `veris.yaml` and as the backend environment name.
+
 ### 3. Configure Your Agent
 
 Edit `.veris/Dockerfile.sandbox` and `.veris/veris.yaml` to match your agent. Set secrets:
@@ -80,6 +83,8 @@ veris env vars set OPENAI_API_KEY=sk-your-key --secret
 veris env push
 ```
 
+When only one target is defined, the CLI uses it automatically — no flags needed.
+
 ### 5. Generate Scenarios
 
 ```bash
@@ -134,6 +139,7 @@ veris run                          # Full pipeline: simulations → evaluations
   --scenario-set-id ID             # Scenario set (prompts if omitted)
   --grader-id ID                   # Grader (prompts if omitted)
   --env-id ID                      # Environment (uses config if omitted)
+  --target NAME                    # Target (auto-detected if only one)
   --image-tag TAG                  # Image tag (default: latest)
   --simulation-timeout N           # Timeout per sim in seconds
   --report                         # Generate HTML report after evaluation
@@ -142,18 +148,30 @@ veris run                          # Full pipeline: simulations → evaluations
 ### Environment (`veris env`)
 
 ```bash
-veris env create --name NAME       # Scaffold .veris/ + register environment
+veris env create --name NAME       # Scaffold .veris/ + register environment (NAME = env/target name)
+  --agent-name NAME                # Agent display name (stored in veris.yaml agent.name)
 veris env push [--tag TAG]         # Build and push image to Veris
+  --target NAME                    # Target to push (auto-detected if only one)
 veris env list                     # List environments
 veris env delete ENV_ID            # Delete environment
 ```
 
+### Active Target (`veris env targets`)
+
+```bash
+veris env targets get               # Show the active target
+veris env targets set NAME          # Set the active target for this project
+veris env targets list              # List configured targets (from veris.yaml)
+veris env targets clear             # Clear the active target
+```
+
 ### Environment Config (`veris env config`)
 
 ```bash
 veris env config push              # Upload veris.yaml to backend (no image build)
   --file PATH                      # Custom veris.yaml path (default: .veris/veris.yaml)
   --env-id ID                      # Override environment
+  --target NAME                    # Target to upload
 ```
 
 ### Environment Variables (`veris env vars`)
@@ -162,8 +180,11 @@ veris env config push              # Upload veris.yaml to backend (no image buil
 veris env vars set K=V [K=V ...]   # Set variables
   --secret                         # Mark as secret
   --env-id ID                      # Override environment
+  --target NAME                    # Target to set variables for
 veris env vars list                # List variables
+  --target NAME
 veris env vars rm KEY              # Remove a variable
+  --target NAME
 ```
 
 ### Scenarios (`veris scenarios`)
@@ -299,17 +320,132 @@ profiles:
 
 ### `.veris/config.yaml`
 
-Project config (created by `veris env create`):
+Project config (created by `veris env create`). Each target's `environment_id` is stored under the profile:
 ```yaml
+active_target: my-cool-agent-env
 profiles:
   default:
-    environment_id: env_abc123
-    environment_name: my-agent
+    targets:
+      my-cool-agent-env:
+        environment_id: env_abc123
+        environment_name: my-cool-agent-env
+```
+
+### `.veris/veris.yaml`
+
+Simulation configuration. Every target is a top-level key — the target name is the backend environment name:
+```yaml
+version: "1.0"
+
+my-cool-agent-env:
+  services:
+    - name: postgres
+      config:
+        SCHEMA_PATH: /agent/schemas/schema.sql
+    - name: slack
+      dns_aliases:
+        - slack.com
+  persona:
+    modality:
+      type: http
+      url: http://localhost:8008/chat
+  agent:
+    name: My Cool Agent
+    code_path: /agent
+    entry_point: uv run app
+    port: 8008
+    environment:
+      DATABASE_URL: postgresql://postgres:postgres@localhost:5432/SIMULATION_ID
+```
+
+When only one target is defined, all commands auto-select it.
+
+## Multiple Targets (Monorepo)
+
+A single repo can manage multiple agents/environments. Each `veris env create` adds a target whose name **is** the backend environment name.
+
+### Creating targets
+
+```bash
+# First agent
+veris env create --name my-cool-agent-env --agent-name "My Cool Agent"
+
+# Second agent — appended to the same veris.yaml
+veris env create --name customer-support-env --agent-name "Customer Support"
+```
+
+After two creates, `.veris/veris.yaml` looks like:
+```yaml
+version: "1.0"
+
+my-cool-agent-env:
+  services: [...]
+  persona: ...
+  agent:
+    name: My Cool Agent
+    ...
+
+customer-support-env:
+  services: [...]
+  persona: ...
+  agent:
+    name: Customer Support
+    ...
+```
+
+### Pushing and running
+
+```bash
+# With one target — auto-detected, no flag needed
+veris env push
+
+# With multiple targets — specify which one
+veris env push --target customer-support-env
+veris run --target customer-support-env
+veris env vars set API_KEY=sk-... --target customer-support-env --secret
+```
+
+### Setting an active target
+
+Avoid typing `--target` on every command:
+```bash
+veris env targets set customer-support-env
+veris env push                    # pushes customer-support-env
+veris run                         # runs customer-support-env
+veris env targets get              # shows: customer-support-env
+veris env targets list             # lists all targets from veris.yaml
+veris env targets clear            # clears the active target
+```
+
+### Target resolution
+
+| Priority | Source |
+|----------|--------|
+| 1 | `--target` flag |
+| 2 | `active_target` in `.veris/config.yaml` |
+| 3 | Auto-detect: sole target in `veris.yaml` |
+| 4 | Error if ambiguous |
+
+### Per-target Dockerfile
+
+Targets can override the default Dockerfile:
+```yaml
+customer-support-env:
+  dockerfile: .veris/Dockerfile.support
+  agent:
+    name: Customer Support
+    ...
 ```
 
+Falls back to `.veris/Dockerfile.sandbox` when not specified.
+
+### Cross-profile behavior
+
+When you switch profiles (e.g. from `dev` to `staging`) and push, the CLI detects that the target exists on another profile and offers to create it on the current one.
+
 ## Profiles
 
-The CLI supports named profiles for managing multiple environments. Each profile stores its own API key, backend URL, console URL, organization, and environment ID.
+The CLI supports named profiles for managing multiple backend accounts. Each profile stores its own API key, backend URL, console URL, organization, and environment IDs.
 
 ```bash
 # Login to a named profile (creates it if new, sets it active)

{veris_cli-2.22.2 → veris_cli-2.24.0}/README.md

@@ -36,7 +36,7 @@ This saves your credentials to `~/.veris/config.yaml`.
 ### 2. Create an Environment
 
 ```bash
-veris env create --name "my-agent"
+veris env create --name my-agent-env --agent-name "My Agent"
 ```
 
 This scaffolds a `.veris/` directory and registers your environment on Veris:
@@ -44,6 +44,8 @@ This scaffolds a `.veris/` directory and registers your environment on Veris:
 - **`veris.yaml`** — Simulation configuration (services, persona, agent settings)
 - **`.dockerignore`** — Files excluded from image build
 
+The `--name` you pass is the agent's display name. The CLI slugifies it to create a target/env name: `"My Agent"` becomes `my-agent-env`. This name is used as the top-level key in `veris.yaml` and as the backend environment name.
+
 ### 3. Configure Your Agent
 
 Edit `.veris/Dockerfile.sandbox` and `.veris/veris.yaml` to match your agent. Set secrets:
@@ -58,6 +60,8 @@ veris env vars set OPENAI_API_KEY=sk-your-key --secret
 veris env push
 ```
 
+When only one target is defined, the CLI uses it automatically — no flags needed.
+
 ### 5. Generate Scenarios
 
 ```bash
@@ -112,6 +116,7 @@ veris run                          # Full pipeline: simulations → evaluations
   --scenario-set-id ID             # Scenario set (prompts if omitted)
   --grader-id ID                   # Grader (prompts if omitted)
   --env-id ID                      # Environment (uses config if omitted)
+  --target NAME                    # Target (auto-detected if only one)
   --image-tag TAG                  # Image tag (default: latest)
   --simulation-timeout N           # Timeout per sim in seconds
   --report                         # Generate HTML report after evaluation
@@ -120,18 +125,30 @@ veris run                          # Full pipeline: simulations → evaluations
 ### Environment (`veris env`)
 
 ```bash
-veris env create --name NAME       # Scaffold .veris/ + register environment
+veris env create --name NAME       # Scaffold .veris/ + register environment (NAME = env/target name)
+  --agent-name NAME                # Agent display name (stored in veris.yaml agent.name)
 veris env push [--tag TAG]         # Build and push image to Veris
+  --target NAME                    # Target to push (auto-detected if only one)
 veris env list                     # List environments
 veris env delete ENV_ID            # Delete environment
 ```
 
+### Active Target (`veris env targets`)
+
+```bash
+veris env targets get               # Show the active target
+veris env targets set NAME          # Set the active target for this project
+veris env targets list              # List configured targets (from veris.yaml)
+veris env targets clear             # Clear the active target
+```
+
 ### Environment Config (`veris env config`)
 
 ```bash
 veris env config push              # Upload veris.yaml to backend (no image build)
   --file PATH                      # Custom veris.yaml path (default: .veris/veris.yaml)
   --env-id ID                      # Override environment
+  --target NAME                    # Target to upload
 ```
 
 ### Environment Variables (`veris env vars`)
@@ -140,8 +157,11 @@ veris env config push              # Upload veris.yaml to backend (no image buil
 veris env vars set K=V [K=V ...]   # Set variables
   --secret                         # Mark as secret
   --env-id ID                      # Override environment
+  --target NAME                    # Target to set variables for
 veris env vars list                # List variables
+  --target NAME
 veris env vars rm KEY              # Remove a variable
+  --target NAME
 ```
 
 ### Scenarios (`veris scenarios`)
@@ -277,17 +297,132 @@ profiles:
 
 ### `.veris/config.yaml`
 
-Project config (created by `veris env create`):
+Project config (created by `veris env create`). Each target's `environment_id` is stored under the profile:
 ```yaml
+active_target: my-cool-agent-env
 profiles:
   default:
-    environment_id: env_abc123
-    environment_name: my-agent
+    targets:
+      my-cool-agent-env:
+        environment_id: env_abc123
+        environment_name: my-cool-agent-env
+```
+
+### `.veris/veris.yaml`
+
+Simulation configuration. Every target is a top-level key — the target name is the backend environment name:
+```yaml
+version: "1.0"
+
+my-cool-agent-env:
+  services:
+    - name: postgres
+      config:
+        SCHEMA_PATH: /agent/schemas/schema.sql
+    - name: slack
+      dns_aliases:
+        - slack.com
+  persona:
+    modality:
+      type: http
+      url: http://localhost:8008/chat
+  agent:
+    name: My Cool Agent
+    code_path: /agent
+    entry_point: uv run app
+    port: 8008
+    environment:
+      DATABASE_URL: postgresql://postgres:postgres@localhost:5432/SIMULATION_ID
+```
+
+When only one target is defined, all commands auto-select it.
+
+## Multiple Targets (Monorepo)
+
+A single repo can manage multiple agents/environments. Each `veris env create` adds a target whose name **is** the backend environment name.
+
+### Creating targets
+
+```bash
+# First agent
+veris env create --name my-cool-agent-env --agent-name "My Cool Agent"
+
+# Second agent — appended to the same veris.yaml
+veris env create --name customer-support-env --agent-name "Customer Support"
+```
+
+After two creates, `.veris/veris.yaml` looks like:
+```yaml
+version: "1.0"
+
+my-cool-agent-env:
+  services: [...]
+  persona: ...
+  agent:
+    name: My Cool Agent
+    ...
+
+customer-support-env:
+  services: [...]
+  persona: ...
+  agent:
+    name: Customer Support
+    ...
+```
+
+### Pushing and running
+
+```bash
+# With one target — auto-detected, no flag needed
+veris env push
+
+# With multiple targets — specify which one
+veris env push --target customer-support-env
+veris run --target customer-support-env
+veris env vars set API_KEY=sk-... --target customer-support-env --secret
+```
+
+### Setting an active target
+
+Avoid typing `--target` on every command:
+```bash
+veris env targets set customer-support-env
+veris env push                    # pushes customer-support-env
+veris run                         # runs customer-support-env
+veris env targets get              # shows: customer-support-env
+veris env targets list             # lists all targets from veris.yaml
+veris env targets clear            # clears the active target
+```
+
+### Target resolution
+
+| Priority | Source |
+|----------|--------|
+| 1 | `--target` flag |
+| 2 | `active_target` in `.veris/config.yaml` |
+| 3 | Auto-detect: sole target in `veris.yaml` |
+| 4 | Error if ambiguous |
+
+### Per-target Dockerfile
+
+Targets can override the default Dockerfile:
+```yaml
+customer-support-env:
+  dockerfile: .veris/Dockerfile.support
+  agent:
+    name: Customer Support
+    ...
 ```
 
+Falls back to `.veris/Dockerfile.sandbox` when not specified.
+
+### Cross-profile behavior
+
+When you switch profiles (e.g. from `dev` to `staging`) and push, the CLI detects that the target exists on another profile and offers to create it on the current one.
+
 ## Profiles
 
-The CLI supports named profiles for managing multiple environments. Each profile stores its own API key, backend URL, console URL, organization, and environment ID.
+The CLI supports named profiles for managing multiple backend accounts. Each profile stores its own API key, backend URL, console URL, organization, and environment IDs.
 
 ```bash
 # Login to a named profile (creates it if new, sets it active)

{veris_cli-2.22.2 → veris_cli-2.24.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "veris-cli"
-version = "2.22.2"
+version = "2.24.0"
 description = "CLI to connect local agents to the Veris backend"
 readme = "README.md"
 requires-python = ">=3.11"
@@ -14,6 +14,7 @@ dependencies = [
     "rich>=13.7.0",
     "questionary>=2.0.0",
     "pyyaml>=6.0.1",
+    "pydantic>=2.6",
 ]
 
 [project.scripts]

{veris_cli-2.22.2 → veris_cli-2.24.0}/src/veris_cli/api.py

@@ -212,12 +212,13 @@ class VerisAPI:
         scenario_set_id: str,
         environment_id: str,
         config: Optional[dict] = None,
+        auto_evaluate: bool = False,
     ) -> dict[str, Any]:
         """Create a new run."""
         payload: dict[str, Any] = {
             "scenario_set_id": scenario_set_id,
             "environment_id": environment_id,
-            "auto_evaluate": False,
+            "auto_evaluate": auto_evaluate,
         }
         if config:
             payload["config"] = config

{veris_cli-2.22.2 → veris_cli-2.24.0}/src/veris_cli/build_context.py

@@ -30,22 +30,25 @@ def _load_dockerignore(project_root: Path) -> pathspec.PathSpec:
 def create_build_context(
     project_root: Path,
     output_path: Path | None = None,
+    dockerfile: Path | str | None = None,
 ) -> tuple[Path, int]:
     """Create a tar.gz build context from the project root.
 
     Respects .veris/.dockerignore (preferred) or .dockerignore at the project root.
-    Always includes .veris/Dockerfile.sandbox even if matched by ignore.
+    Always includes the selected Dockerfile even if matched by ignore.
     Always excludes .git/ regardless of .dockerignore.
 
     Args:
         project_root: Root directory to package.
         output_path: Where to write the tarball. If None, uses a tempfile.
+        dockerfile: Dockerfile path (absolute or relative to project_root). Defaults
+            to `.veris/Dockerfile.sandbox` when None.
 
     Returns:
         Tuple of (tarball_path, size_bytes).
 
     Raises:
-        ValueError: If tarball exceeds the size cap or Dockerfile.sandbox is missing.
+        ValueError: If tarball exceeds the size cap or the Dockerfile is missing.
     """
     spec = _load_dockerignore(project_root)
 
@@ -54,12 +57,18 @@ def create_build_context(
         os.close(fd)
         output_path = Path(tmp)
 
-    dockerfile = project_root / ".veris" / "Dockerfile.sandbox"
-    if not dockerfile.exists():
-        raise ValueError(".veris/Dockerfile.sandbox not found")
+    if dockerfile is None:
+        dockerfile_path = project_root / ".veris" / "Dockerfile.sandbox"
+        dockerfile_label = ".veris/Dockerfile.sandbox"
+    else:
+        df = Path(dockerfile)
+        dockerfile_path = df if df.is_absolute() else project_root / df
+        dockerfile_label = str(dockerfile)
+    if not dockerfile_path.exists():
+        raise ValueError(f"{dockerfile_label} not found")
 
     file_count = 0
-    dockerfile_rel = os.path.join(".veris", "Dockerfile.sandbox")
+    dockerfile_rel = str(dockerfile_path.relative_to(project_root))
     dockerfile_added = False
 
     with tarfile.open(output_path, "w:gz") as tar:
@@ -83,7 +92,7 @@ def create_build_context(
                 if rel_path == ".":
                     rel_path = f
 
-                # Skip ignored files (but always include .veris/Dockerfile.sandbox)
+                # Skip ignored files (but always include the selected Dockerfile)
                 if spec.match_file(rel_path) and rel_path != dockerfile_rel:
                     continue
 
@@ -93,9 +102,9 @@ def create_build_context(
                 if rel_path == dockerfile_rel:
                     dockerfile_added = True
 
-        # If the Dockerfile wasn't added (e.g. .veris/ dir was ignored), add it explicitly
+        # If the Dockerfile wasn't added (e.g. its dir was ignored), add it explicitly
         if not dockerfile_added:
-            tar.add(str(dockerfile), arcname=dockerfile_rel)
+            tar.add(str(dockerfile_path), arcname=dockerfile_rel)
             file_count += 1
 
     size = output_path.stat().st_size