@plune-ai/cli 0.14.3 → 0.16.0

package/README.md

@@ -58,7 +58,7 @@ passed with `-c <path>`). A minimal example:
 version: 1
 provider:
   type: anthropic            # anthropic | openai | openrouter
-  model: claude-3-5-sonnet-latest
+  model: claude-sonnet-5-5
 evals:
   - id: example
     prompt: "Answer concisely. {{question}}"   # {{vars}} come from each dataset row
@@ -77,6 +77,23 @@ provider API key is read from the environment based on `provider.type`:
 | OpenAI     | `openai`        | `OPENAI_API_KEY`     |
 | OpenRouter | `openrouter`    | `OPENROUTER_API_KEY` |
 
+`model` is any id the provider serves; `plune init` starts you on `claude-sonnet-5-5`. A run's cost
+is priced from a built-in table of current models — a model it does not list costs `0`, with a
+warning, until you add its rates under `pricing:` (`input_per_1k_usd`, `output_per_1k_usd`). Leave
+`temperature` out for Claude models released after Opus 4.6: they take none, and the API answers 400
+to any value but the default. Plune sends no `temperature` unless the config sets one.
+
+### Where the CLI reads its settings
+
+Every command reads them the same way: from the environment first — what your shell or CI job
+exported — then from a `.env` beside the config you named with `-c`, then from a `.env` in the
+current directory. The first place that sets a variable wins, so a `.env` never overrides what is
+already exported. That covers the provider keys above and every `PLUNE_*` variable (`PLUNE_TOKEN`,
+`PLUNE_API_URL`, `PLUNE_RUN`, …) for `run`, `report`, `sync`, `ingest`, `login` and `run import`,
+`start` and `finish` alike — no `dotenv` wrapper in your scripts. The token `plune login` saved is
+the last resort, used only when `PLUNE_TOKEN` is set nowhere else. Nothing writes a `.env` for you;
+`plune init` writes `.env.example`.
+
 ### Assertion types
 
 | Type                  | Passes when…                                                     |
@@ -96,7 +113,7 @@ provider API key is read from the environment based on `provider.type`:
 
 | Command | Summary |
 | ------- | ------- |
-| `plune run` | Run the suite. Flags: `--dry-run`, `--only <id\|tag>` (repeatable), `--bail`, `--no-cache`, `--concurrency <n>`, `--format console\|json\|markdown`, `-o, --output <file>`. |
+| `plune run` | Run the suite. Flags: `--dry-run` (price the run from the model's rates — no provider key, no network), `--only <id\|tag>` (repeatable), `--bail`, `--no-cache`, `--concurrency <n>`, `--format console\|json\|markdown`, `-o, --output <file>`. |
 | `plune report` | Re-render the most recent run. Flags: `--format`, `-o`. |
 | `plune diff <baseline> <current>` | Compare two `plune run --format json` outputs and report pass→fail regressions. Flags: `--fail-on-regression`, `--format`, `-o`. |
 | `plune init` | Scaffold `plune.yaml`, a sample dataset, and `.env.example`. Flags: `--yes` (non-interactive), `--force`. |
@@ -108,7 +125,7 @@ provider API key is read from the environment based on `provider.type`:
 | `plune run finish <id>` | Close a platform run. This is what the reporter tells you to do for a run it had to leave open. Flags: `--terminate` to record it as cut short, `--reason <text>`. |
 | `plune run exec -- <command>` | Open a run, run the command inside it, close the run — and exit with whatever the command returned. Sets `PLUNE_SHARED_RUN` for you, so anything reporting inside joins that run. Flags: `--key <externalKey>`. |
 | `plune run delete <id>` | Delete a run and everything it produced — its results and the review-queue entries it raised. Approved test cases and the audit log stay. **Recoverable for six months** (ask Plune to put it back), then gone for good. No prompt: it is your data, and this command belongs in scripts. |
-| `plune run report` | Replay `.plune/pending-results.jsonl` — send what the reporter could not. Flags: `--file <path>`. |
+| `plune run report` | Replay `.plune/pending-results.jsonl` — send what the reporter could not. The lines of one reporter run go back as one run, with the commit and branch the reporter recorded. Once every result has gone the file is renamed to `pending-results.<time>.sent.jsonl`; if anything did not go, it stays as it was. Flags: `--file <path>`. |
 | `plune ingest [dir]` | Record a [Cairn](https://github.com/plune-ai/cairn) run in Plune. Omit `[dir]` for the newest run under `./runs`, or name the directory holding `report.json`. Generated cases arrive as **review proposals** — nothing is created until a person approves it. |
 | `plune pull [file]` | Write the project's test cases as **one Markdown document** — Testomat's classical format, plus Plune's own columns — to `plune/cases.md` (or `[file]`). Refuses to overwrite a file git sees as modified; `--force` overrides, `--suite <id>` takes one suite or folder and what is under it. |
 | `plune push [file] [--dry-run]` | Send the document back. Cases are matched by `id`; a block without one becomes a draft; an unknown id is refused by line; **nothing is deleted**. Prints the report — created, updated, unchanged, refused, warnings — and with `--dry-run` writes nothing. Exit `4` when anything was refused. |
@@ -143,6 +160,11 @@ which reports as the run happens and needs no second step:
 reporter: [['list'], ['@plune-ai/playwright']],
 ```
 
+Either way a Playwright suite brings its screenshots: every image an attempt kept as a file — PNG,
+JPEG or WebP, up to 2 MB each, 20 per result — is uploaded to that result, a passed test's too, so a
+run in Plune shows each test with what it looked at. `plune run import` finds them where the report
+says they are, so run it where the report was written; the summary counts any it could not find.
+
 ### Several jobs, one run
 
 Two suites, or a sharded matrix, report into a single run when every job shares a key **and** knows