evalproof 0.4.0__tar.gz → 0.4.1__tar.gz

{evalproof-0.4.0 → evalproof-0.4.1}/CHANGELOG.md

@@ -4,6 +4,49 @@ All notable changes are listed here. The project follows [Semantic Versioning](h
 
 ## [Unreleased]
 
+## [0.4.1] — 2026-11-13
+
+### Fixed
+
+- `cursor:` ask mode is not tool-free: it has a read tool that runs without approval and reads
+  files in the workspace and outside it. The workspace now carries a `.cursor/cli.json` that
+  allows nothing and denies reads (verified inside and outside the workspace), writes and shell
+  commands (ask mode has no write or shell tool; those entries are a second guard), and any
+  `tool_call` event in the output fails the call as an isolation break.
+- `cursor:` wrote every call's chat to `~/.cursor/chats` and read the user's
+  `~/.cursor/cli-config.json`. `CURSOR_CONFIG_DIR` and `CURSOR_DATA_DIR` now point into the
+  call's temp root, which keeps the login (verified) and starts from a config that allows
+  nothing.
+- `copilot:` kept every call's session in `~/.copilot` and read the user's Copilot config.
+  `COPILOT_HOME` now points into the call's temp root, which keeps the login (verified).
+- `copilot:` passed `--available-tools=`, which copilot 1.0.110 ignores, so the model could use
+  its tools (it read a file). It now passes an allow-list with one unknown name, which disables
+  every tool (verified), and any tool event or tool request in the output fails the call.
+
+### Changed
+
+- `cursor:` uses `--output-format stream-json` (the json result joined the model's preamble
+  with the answer).
+- `copilot:` takes the prompt on stdin, prints JSONL, runs with `--no-custom-instructions` and
+  `--disable-builtin-mcps`, reports real token counts from `--usage-output-file`, and records
+  the model that answered each call: with `copilot:auto` Copilot routes every call, so the run
+  artifact shows the routed model, not `auto`.
+- Isolation tags are now `cursor/2` and `copilot/2`, so earlier cached replies for these two
+  are not reused.
+- `evalproof doctor` warns that `cursor-agent` sends the names and descriptions of the user's
+  skills (`~/.claude/skills`, `~/.cursor/skills` and others) in every prompt, which no option
+  turns off, and that `copilot:auto` as the judge makes compare and the gate refuse runs as a
+  judge change (Copilot picks the model per call).
+
+### Documentation
+
+- The experimental providers section now says what was verified and what was not: Cursor
+  reads instruction files from the directories above the workspace; Cursor and Copilot keep
+  their own system prompt; Copilot's `--no-custom-instructions` was verified with canary
+  files; the Copilot tool guard has not fired live; named-model availability and the
+  premium-request rate depend on plan and policy; with tools off Copilot may write a made-up
+  tool call as text.
+
 ## [0.4.0] — 2026-11-13
 
 ### Added
@@ -244,7 +287,8 @@ First release.
 - Two examples: an offline ticket router (CI's gate contract) and a product support
   assistant judged by a local model, with measurements.
 
-[Unreleased]: https://github.com/technobasant/evalproof/compare/v0.4.0...HEAD
+[Unreleased]: https://github.com/technobasant/evalproof/compare/v0.4.1...HEAD
+[0.4.1]: https://github.com/technobasant/evalproof/compare/v0.4.0...v0.4.1
 [0.4.0]: https://github.com/technobasant/evalproof/compare/v0.3.0...v0.4.0
 [0.3.0]: https://github.com/technobasant/evalproof/compare/v0.2.1...v0.3.0
 [0.2.1]: https://github.com/technobasant/evalproof/compare/v0.2.0...v0.2.1

{evalproof-0.4.0 → evalproof-0.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: evalproof
-Version: 0.4.0
+Version: 0.4.1
 Summary: Treat LLM evaluation as measurement: calibrate the judge, correct the pass rate, gate CI on significant regressions.
 Project-URL: Homepage, https://github.com/technobasant/evalproof
 Project-URL: Documentation, https://github.com/technobasant/evalproof#readme
@@ -181,7 +181,7 @@ ones run OpenCode, Cursor and GitHub Copilot the same way: see
 | `codex` | `codex:gpt-6-luna` | none: your ChatGPT login | local CLI, see below |
 | `opencode` | `opencode:ollama/qwen3:8b` | none: OpenCode's own login, per model provider | local CLI, experimental |
 | `cursor` | `cursor:auto` | none: your Cursor login | local CLI, experimental |
-| `copilot` | `copilot:gpt-5` | none: your GitHub Copilot login | local CLI, experimental |
+| `copilot` | `copilot:auto` | none: your GitHub Copilot login | local CLI, experimental |
 
 ```yaml
 system:
@@ -293,14 +293,17 @@ it is the same model either way. `evalproof doctor` prints the mode in effect an
 ### Experimental providers
 
 `opencode:`, `cursor:` and `copilot:` follow the same rules as the two above: your own logged-in
-binary, run as a subprocess in a fresh empty directory, its API-key variable removed from the
-child's environment unless `api_key_env` names one, a bounded runtime, and the CLI, its version
-and an isolation tag (`opencode/0`, `cursor/0`, `copilot/0`) in the run and in the cache key.
-They are experimental because their isolation is less proven than Claude Code's and Codex's.
-Each prints `evalproof: <name> provider is experimental` on stderr the first time it is used in
-a process. Like the other two they ignore `temperature`, `seed` and `max_tokens`, and
-`evalproof doctor` without a suite lists each of them whose binary is on `PATH`. What was
-checked, and what was not:
+binary, run as a subprocess in a fresh directory in the system temp directory (empty but for
+Cursor's permission file), its API-key variable removed from the child's environment unless
+`api_key_env` names one, a bounded runtime, and the CLI, its version and an isolation tag
+(`opencode/0`, `cursor/2`, `copilot/2`) in the run and in the cache key. They are experimental
+because their isolation is less proven than Claude Code's and Codex's. Each prints
+`evalproof: <name> provider is experimental` on stderr the first time it is used in a process.
+Like the other two they ignore `temperature`, `seed` and `max_tokens`, and `evalproof doctor`
+without a suite lists each of them whose binary is on `PATH`. One difference from the two
+above: Cursor and Copilot have no system-prompt option, so they keep their own agent system
+prompt and your system text goes in front of the prompt under a heading, as an instruction in
+the user turn. What was checked, and what was not:
 
 * **`opencode:<provider>/<model>`** runs `opencode run --standalone --archived --format json`
   with a custom agent that has every tool off and every permission denied, your system text as
@@ -318,25 +321,75 @@ checked, and what was not:
   break CI. `--archived` keeps a session out of OpenCode's lists but does not delete it, so
   sessions accumulate in OpenCode's data directory (each run is a private `--standalone`
   server in a temporary project); prune them yourself.
-* **`cursor:<model>`** runs `cursor-agent -p --mode ask --trust` with an empty `--workspace`.
-  *Not verified:* Cursor was not logged in on the test machine, so no call succeeded. The flags
-  come from the CLI's help and documentation, and the parser from its source; it is lenient (a
-  JSON result object, an array of events, terminal noise around it) and an unexpected shape
-  fails with a message that says what was expected. The empty workspace is the only isolation:
-  no flag ignores rules files, user-level rules were not checked, and whether ask mode really
-  has no tools is untested. There is no system-prompt option, so your system text goes in front
-  of the prompt under a heading, and the prompt is an argument (nothing says the CLI reads
-  stdin), so it is visible in `ps` while the call runs and a prompt over 100,000 bytes is
-  refused. The key variable is `CURSOR_API_KEY`, passed through the environment, never as
-  `--api-key`. `doctor` reads `cursor-agent about` for the login state (`status` hung) and
-  prints no account.
-* **`copilot:<model>`** runs `copilot -s --no-auto-update --no-ask-user --available-tools= -p`.
-  *Not verified:* Copilot CLI was not installed on the test machine; every flag comes from
-  GitHub's documentation. `--available-tools=` is an allow-list with nothing in it, the most
-  restrictive documented tool option; whether the CLI accepts an empty list was not tried. No
-  `--allow-*` flag is ever passed, so a tool that needs approval cannot get it. The reply is
-  plain text, so token counts are 0. `COPILOT_HOME` is not redirected, so user-level Copilot
-  instructions may still be read. System text and prompt travel as for Cursor. Only
+* **`cursor:<model>`** runs `cursor-agent -p --output-format stream-json --mode ask --trust`
+  with a `--workspace` that holds only a `.cursor/cli.json` permission file (nothing allowed;
+  reads, writes and shell commands denied), and with `CURSOR_CONFIG_DIR` and `CURSOR_DATA_DIR`
+  pointed at empty directories in the call's temp root. *Verified live* on 2026-11-13 with
+  cursor-agent 2026.11.11:
+  * Ask mode has no write or shell tool, but it is **not** tool-free: it has a read tool that
+    runs without approval and reads files inside the workspace and outside it by absolute path.
+    The permission file made every read fail with "Permission denied", inside and outside, but
+    the model still tries, so the stream carries a `tool_call` event; evalproof fails the call
+    on any such event (`cursor used a tool (readToolCall); isolation broken`) and discards the
+    reply. The write and shell entries in the deny list are a second guard for later versions;
+    only reads were tested.
+  * Cursor reads instruction files (`AGENTS.md`, `CLAUDE.md`, `.cursor/rules`) from the
+    workspace **and from the directories above it**. The workspace is empty and sits in the
+    system temp directory, so nothing is found there unless `TMPDIR` points inside a project.
+    `~/.claude/CLAUDE.md` was not read.
+  * **Your skills list is sent.** Cursor puts the names and descriptions of the skills under
+    your home directory (`~/.claude/skills`, `~/.cursor/skills`, `~/.codex/skills`,
+    `~/.agents/skills`, plugin skills) and its own subagents in every prompt. The paths are
+    fixed and no setting turns them off; moving `HOME` logs the CLI out. They count as input
+    tokens on every call, and `doctor` warns about it. Verified with a login; with
+    `CURSOR_API_KEY` it was not tested.
+  * The redirected config and data directories keep the login (it is in the system keychain),
+    keep each call's chat out of `~/.cursor/chats` and `~/.cursor/projects`, and mean your own
+    `~/.cursor/cli-config.json` (and its allow list) is not read. A fresh config directory gets
+    a default allow list with `Shell(ls)`, so evalproof writes one that allows nothing.
+
+  `stream-json` rather than `json` because the json result joins the model's preamble text
+  ("I'll read the file now.") with the answer and carries no tool events. Token counts come from
+  the result event (output includes reasoning). The model recorded is the one you asked for;
+  the result carries no model id. The prompt is an argument (nothing says the CLI reads stdin),
+  so it is visible in `ps` while the call runs and a prompt over 100,000 bytes is refused. The
+  key variable is `CURSOR_API_KEY`, passed through the environment, never as `--api-key`.
+  `doctor` reads `cursor-agent about` for the login state (`status` hung) and prints no account.
+* **`copilot:<model>`** runs `copilot --no-auto-update --no-ask-user --no-custom-instructions
+  --disable-builtin-mcps --available-tools=evalproof-no-tools --output-format json
+  --usage-output-file <tmp>`, with the prompt on stdin and `COPILOT_HOME` pointed at an empty
+  directory in the call's temp root. *Verified live* on 2026-11-13 with copilot 1.0.110:
+  * An empty allow-list (`--available-tools=`) is **ignored**: the model called its `view` tool
+    and read a file. An allow-list of one name that is no tool disables every tool
+    (`session.info: Disabled tools: bash, create, edit, glob, grep, ...`), and that is what
+    evalproof passes. Any `tool.execution*` event or tool request in the output also fails the
+    call (`copilot used a tool (view); isolation broken`); with tools off it never fired live,
+    so that guard is tested only against canned output. No `--allow-*` flag is ever passed.
+    With no tools the model sometimes writes a made-up tool call and its "result" as plain
+    text; that is reply text, not a tool run, and is scored as such.
+  * `--no-custom-instructions` works: without it a canary `AGENTS.md`, `CLAUDE.md` and
+    `.github/copilot-instructions.md` in the working directory were all read; with it none was,
+    nor an `AGENTS.md` in the directory above. No skills or MCP servers were offered.
+  * Copilot keeps its own coding-agent system prompt and adds the date and a note that it is
+    not sandboxed; your system text is an instruction in the user turn.
+  * `COPILOT_HOME` in the temp root keeps the login (it is in the system keychain) and keeps
+    each call's session, logs and config out of `~/.copilot` (verified: no new file there after
+    a call). Your own Copilot config, plugins and MCP servers live in that directory, so they
+    are not used either; that follows from the redirect and was not tested separately.
+  * Models and cost: on the test account named models (`gpt-6-luna`, `claude-haiku-5.5`) were
+    rejected with "is not available", and only `copilot:auto` worked; which models you can name
+    depends on your plan and your organisation's policy. `auto` routes each call to a model of
+    Copilot's choosing (`gpt-6-luna` on one call, `mai-code-1.1-flash` on another), so the run
+    artifact records every model that answered, not `auto`, and a suite run on `auto` can mix
+    models. `auto` used one premium request per call there; the rate depends on the model and
+    the plan.
+  * **`copilot:auto` makes a poor judge.** The judge's resolved model is part of what
+    [`compare` and the gate check](#use-your-claude-code-or-codex-login): two runs on `auto` often saw
+    different models, so they are refused as a judge change (exit 4) unless you allow it.
+    `doctor` warns about it. Name a model for the judge if your plan allows one.
+
+  Token counts are read from the usage file (input already includes cache reads and writes,
+  output includes reasoning); a missing file means 0, never a failure. Only
   `COPILOT_GITHUB_TOKEN` is removed; `GH_TOKEN` and `GITHUB_TOKEN` are left because `gh` and CI
   set them for other reasons. `doctor` prints `auth: not checked`.
 
@@ -487,7 +540,7 @@ The composite action runs `compare --gate`, appends the report to the job summar
 fails the job on exit codes 1, 3 or 4:
 
 ```yaml
-- uses: technobasant/evalproof@v0.4.0
+- uses: technobasant/evalproof@v0.4.1
   with:
     baseline: eval/baseline.json     # produced on main, e.g. as a build artifact
     candidate: eval/candidate.json   # produced on this branch

{evalproof-0.4.0 → evalproof-0.4.1}/README.md

@@ -145,7 +145,7 @@ ones run OpenCode, Cursor and GitHub Copilot the same way: see
 | `codex` | `codex:gpt-6-luna` | none: your ChatGPT login | local CLI, see below |
 | `opencode` | `opencode:ollama/qwen3:8b` | none: OpenCode's own login, per model provider | local CLI, experimental |
 | `cursor` | `cursor:auto` | none: your Cursor login | local CLI, experimental |
-| `copilot` | `copilot:gpt-5` | none: your GitHub Copilot login | local CLI, experimental |
+| `copilot` | `copilot:auto` | none: your GitHub Copilot login | local CLI, experimental |
 
 ```yaml
 system:
@@ -257,14 +257,17 @@ it is the same model either way. `evalproof doctor` prints the mode in effect an
 ### Experimental providers
 
 `opencode:`, `cursor:` and `copilot:` follow the same rules as the two above: your own logged-in
-binary, run as a subprocess in a fresh empty directory, its API-key variable removed from the
-child's environment unless `api_key_env` names one, a bounded runtime, and the CLI, its version
-and an isolation tag (`opencode/0`, `cursor/0`, `copilot/0`) in the run and in the cache key.
-They are experimental because their isolation is less proven than Claude Code's and Codex's.
-Each prints `evalproof: <name> provider is experimental` on stderr the first time it is used in
-a process. Like the other two they ignore `temperature`, `seed` and `max_tokens`, and
-`evalproof doctor` without a suite lists each of them whose binary is on `PATH`. What was
-checked, and what was not:
+binary, run as a subprocess in a fresh directory in the system temp directory (empty but for
+Cursor's permission file), its API-key variable removed from the child's environment unless
+`api_key_env` names one, a bounded runtime, and the CLI, its version and an isolation tag
+(`opencode/0`, `cursor/2`, `copilot/2`) in the run and in the cache key. They are experimental
+because their isolation is less proven than Claude Code's and Codex's. Each prints
+`evalproof: <name> provider is experimental` on stderr the first time it is used in a process.
+Like the other two they ignore `temperature`, `seed` and `max_tokens`, and `evalproof doctor`
+without a suite lists each of them whose binary is on `PATH`. One difference from the two
+above: Cursor and Copilot have no system-prompt option, so they keep their own agent system
+prompt and your system text goes in front of the prompt under a heading, as an instruction in
+the user turn. What was checked, and what was not:
 
 * **`opencode:<provider>/<model>`** runs `opencode run --standalone --archived --format json`
   with a custom agent that has every tool off and every permission denied, your system text as
@@ -282,25 +285,75 @@ checked, and what was not:
   break CI. `--archived` keeps a session out of OpenCode's lists but does not delete it, so
   sessions accumulate in OpenCode's data directory (each run is a private `--standalone`
   server in a temporary project); prune them yourself.
-* **`cursor:<model>`** runs `cursor-agent -p --mode ask --trust` with an empty `--workspace`.
-  *Not verified:* Cursor was not logged in on the test machine, so no call succeeded. The flags
-  come from the CLI's help and documentation, and the parser from its source; it is lenient (a
-  JSON result object, an array of events, terminal noise around it) and an unexpected shape
-  fails with a message that says what was expected. The empty workspace is the only isolation:
-  no flag ignores rules files, user-level rules were not checked, and whether ask mode really
-  has no tools is untested. There is no system-prompt option, so your system text goes in front
-  of the prompt under a heading, and the prompt is an argument (nothing says the CLI reads
-  stdin), so it is visible in `ps` while the call runs and a prompt over 100,000 bytes is
-  refused. The key variable is `CURSOR_API_KEY`, passed through the environment, never as
-  `--api-key`. `doctor` reads `cursor-agent about` for the login state (`status` hung) and
-  prints no account.
-* **`copilot:<model>`** runs `copilot -s --no-auto-update --no-ask-user --available-tools= -p`.
-  *Not verified:* Copilot CLI was not installed on the test machine; every flag comes from
-  GitHub's documentation. `--available-tools=` is an allow-list with nothing in it, the most
-  restrictive documented tool option; whether the CLI accepts an empty list was not tried. No
-  `--allow-*` flag is ever passed, so a tool that needs approval cannot get it. The reply is
-  plain text, so token counts are 0. `COPILOT_HOME` is not redirected, so user-level Copilot
-  instructions may still be read. System text and prompt travel as for Cursor. Only
+* **`cursor:<model>`** runs `cursor-agent -p --output-format stream-json --mode ask --trust`
+  with a `--workspace` that holds only a `.cursor/cli.json` permission file (nothing allowed;
+  reads, writes and shell commands denied), and with `CURSOR_CONFIG_DIR` and `CURSOR_DATA_DIR`
+  pointed at empty directories in the call's temp root. *Verified live* on 2026-11-13 with
+  cursor-agent 2026.11.11:
+  * Ask mode has no write or shell tool, but it is **not** tool-free: it has a read tool that
+    runs without approval and reads files inside the workspace and outside it by absolute path.
+    The permission file made every read fail with "Permission denied", inside and outside, but
+    the model still tries, so the stream carries a `tool_call` event; evalproof fails the call
+    on any such event (`cursor used a tool (readToolCall); isolation broken`) and discards the
+    reply. The write and shell entries in the deny list are a second guard for later versions;
+    only reads were tested.
+  * Cursor reads instruction files (`AGENTS.md`, `CLAUDE.md`, `.cursor/rules`) from the
+    workspace **and from the directories above it**. The workspace is empty and sits in the
+    system temp directory, so nothing is found there unless `TMPDIR` points inside a project.
+    `~/.claude/CLAUDE.md` was not read.
+  * **Your skills list is sent.** Cursor puts the names and descriptions of the skills under
+    your home directory (`~/.claude/skills`, `~/.cursor/skills`, `~/.codex/skills`,
+    `~/.agents/skills`, plugin skills) and its own subagents in every prompt. The paths are
+    fixed and no setting turns them off; moving `HOME` logs the CLI out. They count as input
+    tokens on every call, and `doctor` warns about it. Verified with a login; with
+    `CURSOR_API_KEY` it was not tested.
+  * The redirected config and data directories keep the login (it is in the system keychain),
+    keep each call's chat out of `~/.cursor/chats` and `~/.cursor/projects`, and mean your own
+    `~/.cursor/cli-config.json` (and its allow list) is not read. A fresh config directory gets
+    a default allow list with `Shell(ls)`, so evalproof writes one that allows nothing.
+
+  `stream-json` rather than `json` because the json result joins the model's preamble text
+  ("I'll read the file now.") with the answer and carries no tool events. Token counts come from
+  the result event (output includes reasoning). The model recorded is the one you asked for;
+  the result carries no model id. The prompt is an argument (nothing says the CLI reads stdin),
+  so it is visible in `ps` while the call runs and a prompt over 100,000 bytes is refused. The
+  key variable is `CURSOR_API_KEY`, passed through the environment, never as `--api-key`.
+  `doctor` reads `cursor-agent about` for the login state (`status` hung) and prints no account.
+* **`copilot:<model>`** runs `copilot --no-auto-update --no-ask-user --no-custom-instructions
+  --disable-builtin-mcps --available-tools=evalproof-no-tools --output-format json
+  --usage-output-file <tmp>`, with the prompt on stdin and `COPILOT_HOME` pointed at an empty
+  directory in the call's temp root. *Verified live* on 2026-11-13 with copilot 1.0.110:
+  * An empty allow-list (`--available-tools=`) is **ignored**: the model called its `view` tool
+    and read a file. An allow-list of one name that is no tool disables every tool
+    (`session.info: Disabled tools: bash, create, edit, glob, grep, ...`), and that is what
+    evalproof passes. Any `tool.execution*` event or tool request in the output also fails the
+    call (`copilot used a tool (view); isolation broken`); with tools off it never fired live,
+    so that guard is tested only against canned output. No `--allow-*` flag is ever passed.
+    With no tools the model sometimes writes a made-up tool call and its "result" as plain
+    text; that is reply text, not a tool run, and is scored as such.
+  * `--no-custom-instructions` works: without it a canary `AGENTS.md`, `CLAUDE.md` and
+    `.github/copilot-instructions.md` in the working directory were all read; with it none was,
+    nor an `AGENTS.md` in the directory above. No skills or MCP servers were offered.
+  * Copilot keeps its own coding-agent system prompt and adds the date and a note that it is
+    not sandboxed; your system text is an instruction in the user turn.
+  * `COPILOT_HOME` in the temp root keeps the login (it is in the system keychain) and keeps
+    each call's session, logs and config out of `~/.copilot` (verified: no new file there after
+    a call). Your own Copilot config, plugins and MCP servers live in that directory, so they
+    are not used either; that follows from the redirect and was not tested separately.
+  * Models and cost: on the test account named models (`gpt-6-luna`, `claude-haiku-5.5`) were
+    rejected with "is not available", and only `copilot:auto` worked; which models you can name
+    depends on your plan and your organisation's policy. `auto` routes each call to a model of
+    Copilot's choosing (`gpt-6-luna` on one call, `mai-code-1.1-flash` on another), so the run
+    artifact records every model that answered, not `auto`, and a suite run on `auto` can mix
+    models. `auto` used one premium request per call there; the rate depends on the model and
+    the plan.
+  * **`copilot:auto` makes a poor judge.** The judge's resolved model is part of what
+    [`compare` and the gate check](#use-your-claude-code-or-codex-login): two runs on `auto` often saw
+    different models, so they are refused as a judge change (exit 4) unless you allow it.
+    `doctor` warns about it. Name a model for the judge if your plan allows one.
+
+  Token counts are read from the usage file (input already includes cache reads and writes,
+  output includes reasoning); a missing file means 0, never a failure. Only
   `COPILOT_GITHUB_TOKEN` is removed; `GH_TOKEN` and `GITHUB_TOKEN` are left because `gh` and CI
   set them for other reasons. `doctor` prints `auth: not checked`.
 
@@ -469,7 +522,7 @@ The composite action runs `compare --gate`, appends the report to the job summar
 fails the job on exit codes 1, 3 or 4:
 
 ```yaml
-- uses: technobasant/evalproof@v0.4.0
+- uses: technobasant/evalproof@v0.4.1
   with:
     baseline: eval/baseline.json     # produced on main, e.g. as a build artifact
     candidate: eval/candidate.json   # produced on this branch

{evalproof-0.4.0 → evalproof-0.4.1}/src/evalproof/_version.py

@@ -1,3 +1,3 @@
 """The package version: the single place it is written. pyproject.toml reads it at build time."""
 
-__version__ = "0.4.0"
+__version__ = "0.4.1"

{evalproof-0.4.0 → evalproof-0.4.1}/src/evalproof/doctor.py

@@ -48,6 +48,26 @@ def suite_targets(path: str | None) -> list[tuple[TargetConfig, list[str]]]:
     return list(found.values())
 
 
+def experimental_warnings(kind: str, model: str, roles: list[str]) -> list[str]:
+    """Known limits of an experimental CLI that a user should hear about before a run."""
+    warnings: list[str] = []
+    if kind == "cursor":
+        warnings.append(
+            "cursor-agent puts the names and descriptions of your skills (~/.claude/skills, "
+            "~/.cursor/skills, ~/.codex/skills, ~/.agents/skills, plugin skills) in every "
+            "prompt; no option turns this off, so they are sent to Cursor and count as input "
+            "tokens on each call"
+        )
+    if kind == "copilot" and model == "auto" and "judge" in roles:
+        warnings.append(
+            "copilot:auto as the judge: Copilot picks the model per call and the run records "
+            "every model it saw, so two runs often differ and compare and gate refuse them as "
+            "a judge change (exit 4) unless the gate sets allow_judge_change; name a model if "
+            "your plan allows one"
+        )
+    return warnings
+
+
 def check_target(cfg: TargetConfig, roles: list[str], out: TextIO) -> bool:
     """Print what is known about one target; True when it is usable."""
     kind, model = parse_target(cfg.target)
@@ -88,6 +108,8 @@ def check_target(cfg: TargetConfig, roles: list[str], out: TextIO) -> bool:
         out.write("  note: experimental provider; its isolation is less proven (see the README)\n")
         if provider.extra_env:
             out.write(f"  env: {' '.join(f'{k}={v}' for k, v in provider.extra_env.items())}\n")
+        for line in experimental_warnings(kind, model, roles):
+            out.write(f"  warning: {line}\n")
     if cfg.api_key_env:
         out.write(f"  key: ${cfg.api_key_env} is passed to the CLI as {provider.key_var}\n")
     elif provider.auth_mode == "api-key":