libretto 0.6.11 → 0.6.13

package/docs/experiments.md

@@ -0,0 +1,67 @@
+# Experiments Framework
+
+Use this reference when adding or changing Libretto experiment flags, wiring experiment checks through CLI or daemon internals, or debugging `libretto experiments` behavior.
+
+## Purpose and Scope
+
+Experiments are boolean feature flags for Libretto internal machinery. They let maintainers enable in-progress CLI or daemon behavior in a workspace without exposing those flags to user workflow code.
+
+Do not add experiments to `LibrettoWorkflowContext` in `packages/libretto/src/shared/workflow/workflow.ts`. User workflows should not branch on Libretto experiment flags.
+
+## Registry and Workspace State
+
+The experiment registry lives in `packages/libretto/src/cli/core/experiments.ts`.
+
+- Add each flag to `EXPERIMENTS` with a stable hyphenated slug, title, description, and `defaultValue`.
+- Use the exported `ExperimentName` and `Experiments` types instead of duplicating flag shapes.
+- Use `resolveExperiments()` to read the resolved boolean snapshot.
+- Use `setExperimentEnabled()` to persist an override and reject unknown flag names.
+
+Workspace overrides are stored in `.libretto/config.json` under the optional `experiments` record. The config schema is defined in `packages/libretto/src/cli/core/config.ts`; unknown config fields are still passed through.
+
+Example workspace state:
+
+```json
+{
+  "version": 1,
+  "experiments": {
+    "compact-snapshot-format": true
+  }
+}
+```
+
+## CLI Behavior
+
+`libretto experiments` is the CLI surface for inspecting and changing workspace experiment overrides.
+
+```bash
+npx libretto experiments
+npx libretto experiments describe <experiment>
+npx libretto experiments enable <experiment>
+npx libretto experiments disable <experiment>
+```
+
+The command is implemented in `packages/libretto/src/cli/commands/experiments.ts` and registered as a top-level command. Listing prints registered experiments in registry order with their enabled/disabled state and description. `describe` prints the experiment status and full instructions. `enable` persists the override to `.libretto/config.json` and prints the full description with a prelude that the enabled experiment changes expected Libretto usage from the skill. `disable` persists the override and prints deterministic success text.
+
+Invalid actions, missing experiment names, and unknown experiment names should fail with actionable usage that includes the available experiment names.
+
+## CLI and Daemon Plumbing
+
+Use `withExperiments()` from `packages/libretto/src/cli/commands/shared.ts` when a command needs experiment values. It resolves one experiment snapshot for the CLI invocation and adds it to command context as `ctx.experiments`.
+
+Daemon-backed startup paths must serialize that snapshot into `DaemonConfig.experiments` in `packages/libretto/src/cli/core/daemon/config.ts`. This currently applies to:
+
+- daemon-backed `open`
+- `connect`
+- provider-backed `open`
+- `run`
+
+The daemon receives experiments at startup only. Changes made with `libretto experiments enable|disable` apply to new daemon sessions, not already-running sessions.
+
+Inside the daemon, experiments are for Libretto machinery only. Keep them in daemon/controller internals; do not pass them into public workflow context objects.
+
+## Testing Notes
+
+Before adding or changing tests, read `docs/tests-guide.md`.
+
+Prefer user-level CLI behavior tests for `libretto experiments` listing and enable/disable flows. Keep a regression test that an enabled experiment is not visible on `LibrettoWorkflowContext` during `run`.

package/docs/releasing.md

@@ -24,7 +24,7 @@ This repo does not publish from local machines and does not push directly to `ma
 
 GitHub Actions needs these repository secrets:
 
-- `OPENAI_API_KEY`: used by the existing test suite during the release workflow.
+- `OPENAI_API_KEY`: used by the existing test suite during the release workflow and by the eval workflow's default `openai/gpt-5.5` model.
 
 The release workflow uses a GitHub Actions environment named `release`. Create that environment in the repository settings (no required reviewers — access is controlled by branch protection on `main` instead).
 
@@ -69,7 +69,7 @@ The root `scripts/prepare-release.sh` script does the following:
 6. Commits the version bump.
 7. Pushes the branch and opens a PR to `main` with the `release` label.
 
-Release PRs also run the eval workflow. That workflow compares the current eval score against the latest successful `main` baseline and fails if the score drifts by more than 5 percentage points in either direction.
+Release PRs also run the eval workflow. That workflow records score, duration, token, cost, and tool-call metrics for review. Scores are informational: low scores do not fail the workflow, but setup/runtime failures and zero completed records do.
 
 ## Merge behavior
 
@@ -89,11 +89,13 @@ This makes the workflow safe to re-run after partial failures. For example, if n
 
 `.github/workflows/evals.yml` now runs automatically for release PRs and for qualifying pushes to `main`.
 
-- On `main`, it records the current eval summary as the baseline artifact for future release PRs.
-- On release PRs, it runs evals again and compares the overall score against the latest successful `main` baseline.
-- If the score moves outside a `+/-5%` window, the eval job fails and flags the release PR.
+- It runs `pnpm evals --no-auth --output <runner-temp>/eval-run` so CI only runs cases that do not require local auth profiles.
+- It validates and renders the CI report with `pnpm evals summary <runner-temp>/eval-run`.
+- It writes a GitHub step summary and a sticky PR comment with aggregate score, duration, token, cost, and tool-call metrics.
+- It uploads summary files and per-case result records from the run output directory. Raw transcripts and local profile files are not uploaded.
+- It fails when the eval runner crashes, required setup is missing, result records are malformed, or zero completed records are produced.
 
-If no successful baseline artifact exists yet, the release PR eval job reports that and skips the comparison for that run.
+There is no baseline comparison gate yet. Add one only after the eval records and metrics are stable enough to compare reliably.
 
 ## Changelog behavior
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "libretto",
-  "version": "0.6.11",
+  "version": "0.6.13",
   "description": "AI-powered browser automation library and CLI built on Playwright",
   "license": "MIT",
   "homepage": "https://libretto.sh",
@@ -37,8 +37,10 @@
     "sync-skills": "pnpm run sync:mirrors",
     "check:skills": "pnpm run check:mirrors",
     "build": "tsup --config tsup.config.ts",
+    "lint": "lintcn lint --tsconfig tsconfig.json",
     "type-check": "tsc --noEmit",
-    "test": "vitest run",
+    "test": "turbo run test:vitest --filter=libretto --log-order=grouped",
+    "test:vitest": "vitest run",
     "test:watch": "vitest",
     "cli": "node dist/index.js",
     "generate-changelog": "tsx scripts/generate-changelog.ts",
@@ -78,6 +80,7 @@
     "glimpseui": "^0.5.1",
     "google-auth-library": "^10.6.1",
     "openai": "^6.29.0",
+    "outdent": "^0.8.0",
     "tsup": "^8.5.1",
     "typescript": "^5.9.3",
     "vitest": "^4.1.5"

package/skills/libretto/SKILL.md

@@ -4,7 +4,7 @@ description: "Browser automation CLI for building, maintaining, and running brow
 license: MIT
 metadata:
   author: saffron-health
-  version: "0.6.11"
+  version: "0.6.13"
 ---
 
 ## How Libretto Works
@@ -34,10 +34,12 @@ Full documentation is published at [libretto.sh](https://libretto.sh). Available
 
 ## Setup
 
-- Use `npx libretto setup` for first-time workspace onboarding. It installs Chromium, syncs skills, and pins the default snapshot model to `.libretto/config.json` when provider credentials are available.
-- Re-running `setup` on a healthy workspace shows the current configuration. If credentials are missing for a configured provider, it offers an interactive repair flow.
-- Use `npx libretto status` to inspect AI configuration health and open sessions without triggering setup.
-- Use `npx libretto ai configure openai|anthropic|gemini|vertex` to explicitly change the snapshot model or provider (advanced override).
+- Use `npx libretto setup` for first-time workspace onboarding. It installs Chromium and syncs skills.
+- Use `npx libretto status` to inspect open sessions without triggering setup.
+
+## Experiments
+
+- Use `npx libretto experiments` to list internal feature flags and `npx libretto experiments describe <name>` for usage notes when an experiment is enabled.
 
 ## Working Rules
 
@@ -95,20 +97,15 @@ npx libretto session-mode --session my-session
 ### `snapshot`
 
 - Use `snapshot` as the primary page observation tool.
-- Always provide both `--objective` and `--context`.
-- A single snapshot objective can include multiple questions or analysis tasks.
+- Run `snapshot` without `--objective` or `--context`; the command prints a screenshot path and compact accessibility tree for the current page.
+- Run `snapshot <ref>` to inspect a subtree from the latest full snapshot. Use ref forms printed in the tree, such as `l16`; numeric-suffix aliases such as `e16` also match `l16`.
+- Run an unscoped snapshot before using refs. Subtree snapshots capture a fresh screenshot but reuse the latest cached tree.
 - Use it before guessing at selectors, after workflow failures, and whenever the visible page state is unclear.
-- When analysis is involved, expect it to take time. Use a timeout of at least 2 minutes for shell-wrapped calls.
 
 ```bash
-npx libretto snapshot \
-  --objective "Find the sign-in form and submit button" \
-  --context "I just opened the login page and need the email field, password field, and submit button."
-npx libretto snapshot \
-  --session debug-example \
-  --page <page-id> \
-  --objective "Explain why the table is empty" \
-  --context "I opened the referrals page, applied filters, and expected rows to appear."
+npx libretto snapshot --session debug-example
+npx libretto snapshot <ref> --session debug-example
+npx libretto snapshot --session debug-example --page <page-id>
 ```
 
 ### `exec`
@@ -120,6 +117,7 @@ npx libretto snapshot \
 - Let failures throw. Do not hide `exec` failures with `try/catch` or `.catch()`.
 - Do not run multiple `exec` commands in parallel.
 - Do not use `exec` in read-only diagnosis flows. Use `readonly-exec` from the `libretto-readonly` skill for those sessions.
+- After successful mutations, `exec` prints page-change diffs from compact snapshots.
 
 ```bash
 npx libretto exec "return await page.url()"
@@ -142,6 +140,7 @@ npx libretto exec --session debug-example --page <page-id> "return await page.ur
 
 - Use `run` to verify a workflow file after creating it or editing it, preferring `run --headless` for the normal fix/verify loop.
 - Plain `run` defaults to headed mode.
+- Successful runs close the browser by default. Pass `--stay-open-on-success` when you need to inspect the completed state with `pages`, `snapshot`, or `exec`.
 - Pass `--read-only` if the preserved session should come back locked for follow-up terminal inspection after the workflow run.
 - If the workflow fails, Libretto keeps the browser open. Inspect the failed state with `snapshot` and `exec` before editing code.
 - Insert `await pause(session)` statements in the workflow file when you need to stop at specific states for interactive debugging, like breakpoints in the browser flow.
@@ -151,6 +150,7 @@ npx libretto exec --session debug-example --page <page-id> "return await page.ur
 ```bash
 npx libretto run ./integration.ts --headless --params '{"status":"open"}'
 npx libretto run ./integration.ts --headless --read-only
+npx libretto run ./integration.ts --headless --stay-open-on-success
 npx libretto run ./integration.ts --auth-profile app.example.com
 ```
 
@@ -225,7 +225,7 @@ Key fields: `ts` (ISO timestamp), `method` (HTTP method, e.g. `GET`, `POST`), `u
 Assistant: I'll inspect the real site first if needed, but before I finish I'll create `target-workflow.ts` so the task produces reusable automation code.
 Assistant: [Runs `npx libretto open https://target.example.com --headed`]
 Assistant: [Reads `references/site-security-review.md` before choosing between passive network inspection, direct browser fetch calls, and Playwright-first automation]
-Assistant: [Runs `npx libretto snapshot --objective "Find the next required action" --context "We are starting the workflow from the landing page and need the first meaningful step."`]
+Assistant: [Runs `npx libretto snapshot --session <session>`]
 Assistant: [Uses `snapshot` and `exec` as needed to understand the site and decide the implementation path]
 Assistant: [Reads `references/code-generation-rules.md` before writing production workflow code]
 Assistant: I found the working path. I'll now create the workflow file and verify it.
@@ -245,7 +245,7 @@ To watch it yourself in a browser, run: npx libretto run ./target-workflow.ts --
 Assistant: I'll reproduce the failure first so we can inspect the exact browser state it leaves behind.
 Assistant: [Runs `npx libretto run ./integration.ts --session debug-flow --headed`]
 Assistant: The workflow failed and Libretto kept the browser open. I'll inspect the page state before changing code.
-Assistant: [Runs `npx libretto snapshot --session debug-flow --objective "Find the blocking error or broken selector target" --context "The workflow just failed after trying to continue from the review step, and I need to identify the visible blocker on the current page."`]
+Assistant: [Runs `npx libretto snapshot --session debug-flow`]
 Assistant: [Runs `npx libretto exec --session debug-flow "...focused inspection or prototype..."`]
 Assistant: [Reads `references/code-generation-rules.md` before patching the workflow file]
 Assistant: I found the issue. I'll patch the workflow code and verify.
@@ -259,7 +259,7 @@ To watch it yourself in a browser, run: npx libretto run ./integration.ts --head
 
 ## References
 
-- Read `references/configuration-file-reference.md` when you need to inspect or change `.libretto/config.json` for snapshot model selection or viewport defaults.
+- Read `references/configuration-file-reference.md` when you need to inspect or change `.libretto/config.json` for viewport or session defaults.
 - Read `references/site-security-review.md` before reviewing the site's security posture and deciding whether to lead with network requests, passive interception, or Playwright DOM automation on a new site.
 - Read `references/code-generation-rules.md` before writing or editing production workflow files.
 - Read `references/auth-profiles.md` when auth-profile behavior is relevant.

package/skills/libretto/references/configuration-file-reference.md

@@ -1,10 +1,9 @@
 # Configuration File Reference
 
-Use this reference when you need to inspect or change the workspace configuration that powers `snapshot` analysis and default viewport behavior.
+Use this reference when you need to inspect or change workspace configuration for default browser behavior.
 
 ## When to Use This
 
-- You want to confirm which AI model `snapshot` will use.
 - You want to understand where Libretto stores workspace-level settings.
 - You want a persistent default viewport for `open` or `run`.
 
@@ -12,14 +11,12 @@ Use this reference when you need to inspect or change the workspace configuratio
 
 Libretto reads workspace config from `.libretto/config.json`.
 
-- The file is created by `npx libretto setup` during first-time onboarding (auto-pins the default model for the detected provider) or by `npx libretto ai configure ...` for explicit overrides.
-- API credentials come from your shell environment or a `.env` file **at the repository root** (next to your `.git` directory). The config file stores the selected model, not the secret itself. Set `LIBRETTO_DISABLE_DOTENV=1` to skip `.env` loading (useful in CI).
-- Use `npx libretto status` to inspect the current AI configuration and open sessions without changing anything.
+- The file is created by `npx libretto setup` during first-time onboarding.
+- Use `npx libretto status` to inspect open sessions without changing anything.
 - For first-time setup instructions, follow the main `SKILL.md` flow instead of expanding this reference.
 
 ## Supported Settings
 
-- `snapshotModel` selects the configured analysis model for `snapshot`.
 - `viewport` is an optional top-level setting used by `open` and `run` when you do not pass `--viewport`.
 - Viewport precedence is: CLI `--viewport`, then `.libretto/config.json`, then the default `1366x768`.
 - `sessionMode` sets the default session access mode for new sessions created by `open`, `connect`, and `run`. Must be `"read-only"` or `"write-access"`. When omitted, defaults to `"write-access"`. Pass `--read-only` or `--write-access` to `open`, `connect`, or `run` to override when creating a session.
@@ -29,7 +26,6 @@ Example:
 ```json
 {
   "version": 1,
-  "snapshotModel": "openai/gpt-5.4",
   "viewport": {
     "width": 1280,
     "height": 800
@@ -41,9 +37,8 @@ Example:
 ## Common Commands
 
 ```bash
-npx libretto setup                                         # first-time onboarding, auto-pins default model
-npx libretto status                                        # inspect AI config and open sessions
-npx libretto ai configure openai                           # explicitly change provider/model
+npx libretto setup                                         # first-time onboarding
+npx libretto status                                        # inspect open sessions
 npx libretto open https://example.com --viewport 1440x900
 npx libretto run ./integration.ts --viewport 1440x900
 ```
@@ -51,5 +46,4 @@ npx libretto run ./integration.ts --viewport 1440x900
 ## Notes
 
 - If you want a persistent default viewport for the workspace, add `viewport` to `.libretto/config.json` instead of repeating `--viewport` on every command.
-- If `snapshot` analysis is not configured yet, run `npx libretto setup` to auto-configure, or see the main `SKILL.md` flow.
-- Run `npx libretto status` at any time to check which model is active and whether credentials are present.
+- Run `npx libretto status` at any time to check open sessions.

package/skills/libretto/references/pages-and-page-targeting.md

@@ -19,7 +19,7 @@ Use this reference when a Libretto session has multiple open pages and you need
 ```bash
 npx libretto pages --session debug-flow
 npx libretto exec --session debug-flow --page <page-id> "return await page.url()"
-npx libretto snapshot --session debug-flow --page <page-id> --objective "Find the active form"
+npx libretto snapshot --session debug-flow --page <page-id>
 ```
 
 ## Notes

package/skills/libretto-readonly/SKILL.md

@@ -4,7 +4,7 @@ description: "Read-only Libretto workflow for diagnosing live browser state with
 license: MIT
 metadata:
   author: saffron-health
-  version: "0.6.11"
+  version: "0.6.13"
 ---
 
 ## How Libretto Read-Only Works
@@ -49,14 +49,7 @@ npx libretto pages --session failed-job-debug
 ### `snapshot`
 
 - Use `snapshot` as the first high-level observation tool.
-- Always provide both `--objective` and `--context`.
-
-```bash
-npx libretto snapshot \
-  --session failed-job-debug \
-  --objective "Identify the visible failure state and likely blocking UI condition" \
-  --context "The workflow already failed and the preserved browser must remain read-only."
-```
+- Run `snapshot <ref>` to inspect a subtree from the latest full snapshot.
 
 ### `readonly-exec`
 

package/src/cli/AGENTS.md

@@ -0,0 +1,7 @@
+# CLI Source
+
+## Experiments
+
+When adding or changing Libretto experiment flags, read `../../docs/experiments.md` for the registry, CLI command, config, and daemon plumbing conventions.
+
+Experiments are internal Libretto machinery. Do not expose them on `LibrettoWorkflowContext` or user workflow APIs.

package/src/cli/cli.ts

@@ -1,4 +1,3 @@
-import { resolveAiSetupStatus } from "./core/ai-model.js";
 import { ensureLibrettoSetup } from "./core/context.js";
 import { createCLIApp } from "./router.js";
 import { warnIfInstalledSkillOutOfDate } from "./core/skill-version.js";
@@ -16,28 +15,6 @@ Docs (agent-friendly): https://libretto.sh/docs
 
 function printSetupAudit(): void {
   warnIfInstalledSkillOutOfDate();
-
-  const status = resolveAiSetupStatus();
-  switch (status.kind) {
-    case "ready":
-      console.log(`✓ Snapshot model: ${status.model}`);
-      break;
-    case "configured-missing-credentials":
-      console.log(
-        `✗ ${status.provider} configured (model: ${status.model}), but credentials are missing. Run \`npx libretto setup\` to repair.`,
-      );
-      break;
-    case "invalid-config":
-      console.log(
-        `✗ AI config is invalid. Run \`npx libretto setup\` to reconfigure.`,
-      );
-      break;
-    case "unconfigured":
-      console.log(
-        `✗ No AI model configured. Run \`npx libretto setup\` or \`npx libretto ai configure\` to set up.`,
-      );
-      break;
-  }
 }
 
 function isRootHelpRequest(rawArgs: readonly string[]): boolean {

package/src/cli/commands/browser.ts

@@ -9,12 +9,10 @@ import {
   runPages,
   runSave,
 } from "../core/browser.js";
-import {
-  resolveProviderName,
-  getCloudProviderApi,
-} from "../core/providers/index.js";
+import { resolveProviderName } from "../core/providers/index.js";
 import { readLibrettoConfig } from "../core/config.js";
-import { createLoggerForSession, withSessionLogger } from "../core/context.js";
+import { createLoggerForSession } from "../core/context.js";
+import { librettoCommand } from "../../shared/package-manager.js";
 import {
   type SessionAccessMode,
   assertSessionAvailableForStart,
@@ -26,6 +24,7 @@ import { SimpleCLI } from "../framework/simple-cli.js";
 import {
   sessionOption,
   withAutoSession,
+  withExperiments,
   withRequiredSession,
 } from "./shared.js";
 
@@ -95,7 +94,7 @@ export const openInput = SimpleCLI.input({
 })
   .refine(
     (input) => Boolean(input.url),
-    `Usage: libretto open <url> [--headless] [--read-only|--write-access] [--auth-profile <domain>] [--viewport WxH] [--session <name>]`,
+    `Usage: ${librettoCommand("open <url> [--headless] [--read-only|--write-access] [--auth-profile <domain>] [--viewport WxH] [--session <name>]")}`,
   )
   .refine(
     (input) => !(input.headed && input.headless),
@@ -112,6 +111,7 @@ export const openCommand = SimpleCLI.command({
 })
   .input(openInput)
   .use(withAutoSession())
+  .use(withExperiments())
   .handle(async ({ input, ctx }) => {
     warnIfInstalledSkillOutOfDate();
     assertSessionAvailableForStart(ctx.session, ctx.logger);
@@ -126,16 +126,16 @@ export const openCommand = SimpleCLI.command({
           input.writeAccess,
         ),
         authProfileDomain: input.authProfile,
+        experiments: ctx.experiments,
       });
     } else {
-      const provider = getCloudProviderApi(providerName);
       await runOpenWithProvider(
         input.url!,
         providerName,
-        provider,
         ctx.session,
         ctx.logger,
         resolveRequestedSessionMode(input.readOnly, input.writeAccess),
+        ctx.experiments,
       );
     }
   });
@@ -160,7 +160,7 @@ export const connectInput = SimpleCLI.input({
 })
   .refine(
     (input) => Boolean(input.cdpUrl),
-    `Usage: libretto connect <cdp-url> [--read-only|--write-access] --session <name>`,
+    `Usage: ${librettoCommand("connect <cdp-url> [--read-only|--write-access] --session <name>")}`,
   )
   .refine(
     (input) => !(input.readOnly && input.writeAccess),
@@ -172,6 +172,7 @@ export const connectCommand = SimpleCLI.command({
 })
   .input(connectInput)
   .use(withAutoSession())
+  .use(withExperiments())
   .handle(async ({ input, ctx }) => {
     warnIfInstalledSkillOutOfDate();
     await runConnectWithLogger(
@@ -179,6 +180,7 @@ export const connectCommand = SimpleCLI.command({
       ctx.session,
       ctx.logger,
       resolveRequestedSessionMode(input.readOnly, input.writeAccess),
+      ctx.experiments,
     );
   });
 
@@ -193,7 +195,7 @@ export const saveInput = SimpleCLI.input({
   },
 }).refine(
   (input) => Boolean(input.urlOrDomain),
-  `Usage: libretto save <url|domain> --session <name>`,
+  `Usage: ${librettoCommand("save <url|domain> --session <name>")}`,
 );
 
 export const saveCommand = SimpleCLI.command({
@@ -264,7 +266,7 @@ export const closeInput = SimpleCLI.input({
   },
 }).refine(
   (input) => input.all || input.session,
-  `Usage: libretto close <session>\nUsage: libretto close --all [--force]`,
+  `Usage: ${librettoCommand("close <session>")}\nUsage: ${librettoCommand("close --all [--force]")}`,
 );
 
 export const closeCommand = SimpleCLI.command({
@@ -273,7 +275,7 @@ export const closeCommand = SimpleCLI.command({
   .input(closeInput)
   .handle(async ({ input }) => {
     if (input.force && !input.all) {
-      throw new Error(`Usage: libretto close --all [--force]`);
+      throw new Error(`Usage: ${librettoCommand("close --all [--force]")}`);
     }
     if (input.all) {
       const logger = createLoggerForSession("cli");
@@ -293,9 +295,3 @@ export const browserCommands = {
   "session-mode": sessionModeCommand,
   close: closeCommand,
 };
-
-export async function runClose(session: string): Promise<void> {
-  await withSessionLogger(session, async (logger) => {
-    await runCloseWithLogger(session, logger);
-  });
-}