@protolabsai/proto 0.22.0 → 0.24.0

package/README.md

@@ -262,10 +262,76 @@ A `MEMORY.md` index is auto-generated and loaded into the system prompt at the s
 
 After each conversation turn, a background extraction agent reviews recent messages and auto-creates memories for notable facts. This runs fire-and-forget with restricted tools (read/write/glob in the memory directory only).
 
+## Agent Harness
+
+proto includes a harness system that enforces quality gates, limits scope, and recovers from failures automatically.
+
+### Sprint Contract (Scope Lock)
+
+Prevents agents from modifying files outside an agreed scope. Before coding begins, negotiate a contract that defines exactly which files will be created or modified. The scope lock is armed — any write outside scope is rejected with a recovery message.
+
+**Workflow:**
+
+```bash
+proto
+/sprint-contract
+> Task: Refactor auth module
+> Files: src/auth.ts, src/utils.ts
+> Confirm
+```
+
+**Behavior:**
+
+- Write to `src/auth.ts` → ALLOWED
+- Write to `tests/foo.test.ts` → BLOCKED with scope violation message
+
+Contracts persist at `.proto/sprint-contract.json` and auto-restore on session resume.
+
+### Behavior Verification Gate
+
+Post-run smoke tests that verify changes actually work. After a subagent completes, the gate runs your defined scenarios (shell commands) in parallel. Failures inject a remediation message back to the agent for self-correction.
+
+**Setup** — create `.proto/verify-scenarios.json`:
+
+```json
+[
+  { "name": "tests pass", "command": "npm test -- --run", "timeoutMs": 60000 },
+  { "name": "build works", "command": "npm run build", "timeoutMs": 30000 },
+  { "name": "no TypeScript errors", "command": "npm run typecheck" }
+]
+```
+
+**Behavior:**
+
+1. Agent completes task, reports GOAL
+2. Gate fires, runs all scenarios in parallel
+3. If any fail → remediation message injected, agent self-corrects
+4. Gate fires again until all pass
+
+### Multi-Sample Retry
+
+When a subagent fails (ERROR, MAX_TURNS, or TIMEOUT), proto retries up to 2 more times with escalating temperatures (0.7 → 1.0 → 1.3). Each retry gets a `[RETRY CONTEXT]` block summarizing previous failures. Best result by score is returned.
+
+This reduces false negatives from single-run failures and gives the model multiple chances with different sampling strategies.
+
+### Repo Map
+
+PageRank-based file importance ranking. Analyzes the project's TypeScript/JS import graph to surface the most central files. Useful for understanding codebase structure or finding related files.
+
+**Usage:**
+
+```bash
+proto -p "Use the repo_map tool to find the most important files in this codebase"
+proto -p "Use repo_map with seedFiles=['src/auth.ts'] to find related files"
+```
+
+Results are cached at `.proto/repo-map-cache.json` and auto-invalidate on file changes.
+
 ## Skills
 
-proto ships with 16 bundled skills for agentic workflows:
+proto ships with 22 bundled skills for agentic workflows:
 
+- **browser-automation** — Web browser automation
 - **brainstorming** — Structured ideation
 - **dispatching-parallel-agents** — Fan-out/fan-in subagent patterns
 - **executing-plans** — Step-by-step plan execution
@@ -285,6 +351,92 @@ proto ships with 16 bundled skills for agentic workflows:
 
 Use `/skills` to list available skills in a session.
 
+### Browser Automation
+
+proto includes a native browser automation tool powered by [agent-browser](https://github.com/nickinack/agent-browser). This enables AI agents to interact with websites — navigate, click, fill forms, take screenshots, and extract content.
+
+#### Installation
+
+```bash
+npm install -g agent-browser
+agent-browser install  # Downloads Chrome
+```
+
+#### Usage
+
+```javascript
+// Open a website
+browser({ action: 'open', url: 'https://example.com' });
+
+// Get interactive elements
+browser({ action: 'snapshot', flags: JSON.stringify({ interactive: true }) });
+
+// Click an element
+browser({ action: 'click', selector: '@e2' });
+
+// Fill a form
+browser({ action: 'fill', selector: '@e1', text: 'user@example.com' });
+
+// Take screenshot
+browser({ action: 'screenshot', outputPath: '/path/to/screenshot.png' });
+```
+
+#### Key Actions
+
+| Action                         | Description                                |
+| ------------------------------ | ------------------------------------------ |
+| `open` / `close`               | Navigate to URL or close browser           |
+| `click` / `dblclick` / `hover` | Element interaction                        |
+| `fill` / `type`                | Form input                                 |
+| `snapshot`                     | Get accessibility tree with element refs   |
+| `screenshot`                   | Capture page screenshot                    |
+| `get` / `is` / `find`          | Query element properties                   |
+| `wait`                         | Wait for elements, network, or URL changes |
+| `batch`                        | Execute multiple commands in sequence      |
+
+The browser skill (`/skills` → browser-automation) provides comprehensive documentation for all 38 available actions.
+
+## Agent Teams
+
+Run multiple coordinated agents that share tasks and communicate directly with each other.
+
+```
+/team start my-team lead:coordinator scout:Explore coder:general-purpose
+```
+
+This spawns three live agents immediately. Each member runs as an in-process agent and gets two extra tools injected automatically:
+
+- **`mailbox_send`** — send a message to a teammate by their agentId
+- **`mailbox_receive`** — drain all unread messages from your inbox
+
+Members share the same task list (`task_create`, `task_list`, `task_update`) so any agent can create tasks and others can claim them.
+
+### Team commands
+
+| Command                                | Description                           |
+| -------------------------------------- | ------------------------------------- |
+| `/team start <name> [member:type ...]` | Spawn live agents and start the team  |
+| `/team status <name>`                  | Show live member status               |
+| `/team stop <name>`                    | Kill all agents and release resources |
+| `/team list`                           | List all teams in the project         |
+| `/team delete <name>`                  | Delete a team config                  |
+
+**Default team** (no members specified): `lead` (coordinator) + `scout` (Explore).
+
+Agent IDs follow the pattern `<name>-<index>` (e.g. `lead-0`, `scout-1`). Use these when sending mailbox messages between agents.
+
+### Member types
+
+| Type              | Purpose                                   |
+| ----------------- | ----------------------------------------- |
+| `coordinator`     | Orchestrate subtasks across other members |
+| `Explore`         | Fast codebase search and analysis         |
+| `general-purpose` | Multi-step implementation tasks           |
+| `verify`          | Review and correctness checking           |
+| `plan`            | Design plans before implementation        |
+
+Any user-defined sub-agent from `.proto/agents/` can also be used as a member type.
+
 ## Commands
 
 | Command                 | Description                               |

package/bundled/browser-automation/SKILL.md

@@ -0,0 +1,358 @@
+---
+name: browser-automation
+description: Use browser automation to interact with web pages - navigate, click, fill forms, take screenshots, and extract content from websites. Perfect for testing, data collection, and web interactions.
+---
+
+# Browser Automation
+
+Use the native `browser` tool to automate web browser interactions. This skill enables AI agents to navigate websites, interact with elements, fill forms, and extract information.
+
+## Prerequisites
+
+**Before using this skill, verify:**
+
+1. **agent-browser is installed:**
+
+   ```bash
+   agent-browser --version
+   ```
+
+   If not installed:
+
+   ```bash
+   npm install -g agent-browser
+   agent-browser install  # Downloads Chrome
+   ```
+
+2. **Chrome is installed:**
+   The browser tool will auto-download Chrome if needed via `agent-browser install`.
+
+## Core Workflow
+
+### Step 1: Open a Website
+
+```javascript
+browser({
+  action: 'open',
+  url: 'https://example.com',
+  headed: false, // true to see browser window
+});
+```
+
+### Step 2: Get Interactive Elements
+
+Use `snapshot` to get an accessibility tree with numbered element references:
+
+```javascript
+browser({
+  action: 'snapshot',
+  flags: JSON.stringify({ interactive: true }),
+});
+```
+
+**Output shows elements with refs like:**
+
+```text
+[e1] Button: "Submit"
+[e2] Textbox: "Email"
+[e3] Textbox: "Password"
+```
+
+### Step 3: Interact with Elements
+
+**Click an element:**
+
+```javascript
+browser({
+  action: 'click',
+  selector: '@e2', // or CSS selector like "#submit-btn"
+});
+```
+
+**Fill a form field:**
+
+```javascript
+browser({
+  action: 'fill',
+  selector: '@e2',
+  text: 'user@example.com',
+});
+```
+
+**Type with keystroke simulation:**
+
+```javascript
+browser({
+  action: 'type',
+  selector: '@e3',
+  text: 'password123',
+});
+```
+
+### Step 4: Take Screenshots
+
+**Page screenshot:**
+
+```javascript
+browser({
+  action: 'screenshot',
+  outputPath: '/path/to/screenshot.png',
+});
+```
+
+**Full-page screenshot:**
+
+```javascript
+browser({
+  action: 'screenshot',
+  outputPath: '/path/to/full.png',
+  flags: JSON.stringify({ full: true }),
+});
+```
+
+## Selector Types
+
+| Type         | Example                         | Use Case             |
+| ------------ | ------------------------------- | -------------------- |
+| Element ref  | `@e1`, `@e2`                    | From snapshot output |
+| CSS selector | `#id`, `.class`, `div > button` | Standard web dev     |
+| Semantic     | `role:button`, `text:"Sign In"` | AI-friendly          |
+
+## Common Actions Reference
+
+### Navigation
+
+```javascript
+// Open URL
+browser({ action: 'open', url: 'https://example.com' });
+
+// Close browser
+browser({ action: 'close' });
+
+// New tab
+browser({ action: 'tab', text: 'new', url: 'https://example.com' });
+```
+
+### Element Interaction
+
+```javascript
+// Click
+browser({ action: 'click', selector: '@e1' });
+
+// Double-click
+browser({ action: 'dblclick', selector: '@e1' });
+
+// Hover
+browser({ action: 'hover', selector: '@e1' });
+
+// Fill (clears and types)
+browser({ action: 'fill', selector: '@e2', text: 'value' });
+
+// Type (appends)
+browser({ action: 'type', selector: '@e2', text: 'value' });
+
+// Press key
+browser({ action: 'press', key: 'Enter' });
+browser({ action: 'press', key: 'Tab' });
+browser({ action: 'press', key: 'Escape' });
+```
+
+### Page Information
+
+```javascript
+// Get page title
+browser({ action: 'get', text: 'title' });
+
+// Get current URL
+browser({ action: 'get', text: 'url' });
+
+// Get element text
+browser({ action: 'get', selector: '@e1', text: 'text' });
+
+// Get element attribute
+browser({ action: 'get', selector: '@e1', text: 'attr', attribute: 'href' });
+
+// Check if visible
+browser({ action: 'is', text: 'visible', selector: '@e1' });
+```
+
+### Find Elements
+
+```javascript
+// Find by role and click
+browser({ action: 'find', selector: 'role button', text: 'click' });
+
+// Find by text
+browser({ action: 'find', selector: 'text "Sign In"', text: 'click' });
+
+// Find by label
+browser({ action: 'find', selector: 'label "Email"', text: 'fill' });
+```
+
+### Waiting
+
+```javascript
+// Wait for URL pattern
+browser({ action: 'wait', text: '**/dashboard' });
+
+// Wait for text
+browser({ action: 'wait', text: 'Welcome' });
+
+// Wait for network idle
+browser({ action: 'wait', text: 'networkidle' });
+
+// Wait for selector
+browser({ action: 'wait', selector: '@e1' });
+
+// Wait with timeout (default 25s)
+browser({ action: 'wait', selector: '@e1' });
+```
+
+### Batch Operations
+
+Execute multiple commands in sequence:
+
+```javascript
+browser({
+  action: 'batch',
+  commands: [
+    'open https://example.com',
+    'wait --load networkidle',
+    'fill @e1 user@example.com',
+    'fill @e2 password',
+    'click @e3',
+    'wait --url **/dashboard',
+  ],
+});
+```
+
+## Sessions
+
+Sessions persist authentication and state between commands:
+
+```javascript
+// Create named session
+browser({
+  action: 'open',
+  url: 'https://app.example.com',
+  session: 'myapp-auth',
+});
+
+// Continue in same session
+browser({
+  action: 'click',
+  selector: '@e1',
+  session: 'myapp-auth',
+});
+```
+
+## Advanced Features
+
+### Headed Mode (see browser window)
+
+```javascript
+browser({
+  action: 'open',
+  url: 'https://example.com',
+  headed: true,
+});
+```
+
+### Network Interception
+
+```javascript
+// Block a URL
+browser({
+  action: 'network',
+  text: 'route',
+  selector: 'https://ads.example.com',
+  attribute: 'abort', // or mock response
+});
+
+// View requests
+browser({ action: 'network', text: 'requests' });
+```
+
+### Cookies & Storage
+
+```javascript
+// Get cookies
+browser({ action: 'cookies' });
+
+// Get localStorage
+browser({ action: 'storage', text: 'local' });
+
+// Set localStorage
+browser({
+  action: 'storage',
+  text: 'local',
+  selector: 'set myKey myValue',
+});
+```
+
+### Clipboard
+
+```javascript
+// Read clipboard
+browser({ action: 'clipboard' });
+
+// Write to clipboard
+browser({ action: 'clipboard', text: 'write Hello World' });
+```
+
+## Best Practices
+
+1. **Always start with `snapshot`**: Get the accessibility tree to understand page structure
+
+2. **Use element refs from snapshot**: They're more reliable than CSS selectors for AI automation
+
+3. **Wait for network idle**: After form submissions or page loads:
+
+   ```javascript
+   browser({ action: 'wait', text: 'networkidle' });
+   ```
+
+4. **Use sessions for multi-step flows**: Maintains login state and cookies:
+
+   ```javascript
+   browser({ action: 'open', url: 'https://app.com', session: 'app' });
+   ```
+
+5. **Take screenshots for debugging**: When automation fails, screenshot helps debug:
+
+   ```javascript
+   browser({ action: 'screenshot', outputPath: '/tmp/debug.png' });
+   ```
+
+6. **Close browser when done**: Free resources:
+   ```javascript
+   browser({ action: 'close' });
+   ```
+
+## Error Handling
+
+If agent-browser is not installed, you'll get:
+
+```text
+agent-browser is not installed. Please install it with: npm install -g agent-browser
+```
+
+**Installation troubleshooting:**
+
+```bash
+# Install Chrome dependency
+agent-browser install --with-deps
+
+# Check installation
+which agent-browser
+agent-browser --version
+```
+
+## Use Cases
+
+- **Web testing**: Automate UI testing workflows
+- **Form filling**: Submit forms programmatically
+- **Data extraction**: Scrape content from websites
+- **Login flows**: Handle authentication
+- **Screenshot capture**: Document web pages
+- **Accessibility testing**: Verify page structure

package/bundled/harness-reference/SKILL.md

@@ -0,0 +1,146 @@
+---
+name: harness-reference
+description: Reference guide for all agent harness safety features — doom loop detection, scope lock, git checkpoints, observation masking, sprint contract, reminders, repo map, behavior verification, and multi-sample retry
+---
+
+# Agent Harness Reference
+
+The proto harness is a set of safety and reliability features that wrap every agent execution. They fire automatically — you don't need to invoke them manually. This skill documents each feature so you can understand what's protecting you and how to configure it.
+
+## Features
+
+### Doom Loop Detection
+
+**What it does:** Detects when the agent is repeating the same tool call pattern in a sliding 20-call window. If the same fingerprint (tool + args hash) appears 3+ times, the harness injects a recovery message and records a Langfuse span.
+
+**You don't need to do anything.** The harness detects this automatically.
+
+### Scope Lock (Sprint Contract)
+
+**What it does:** Before coding, the `sprint-contract` skill negotiates an explicit contract — the set of files that may change. Once activated, any write outside that set is blocked with a structured error.
+
+**To activate:** Use the `sprint-contract` skill at the start of an implementation task. It writes `.proto/sprint-contract.json` and arms the in-memory scope lock. The lock is restored on session restart.
+
+**To check status:** If a write is blocked, the error message tells you the violating path and the permitted set.
+
+### Git Checkpoints
+
+**What it does:** Before every file-mutating tool call (`write_file`, `edit`, `replace`), the harness creates a shadow-repo commit. This lets you diff or roll back to any pre-edit state.
+
+**To roll back:** Use `git log` to find the checkpoint commit and `git checkout <hash> -- <file>` to restore.
+
+### Observation Masking
+
+**What it does:** When the context window gets large, the harness applies a rolling verbatim window — tool-call/result pairs older than the window are summarized as `[OBSERVATION_MASK: N pairs omitted]`. This keeps recent context intact while reducing token usage.
+
+**You don't need to do anything.** Fires automatically during LLM compaction.
+
+### Harness Reminders
+
+**What it does:** The harness injects periodic reminders into context based on three triggers:
+
+- Every 50 tool calls: warns about high tool usage
+- After 3 consecutive test failures: suggests pausing to diagnose
+- After 8 turns without any file write: suggests the agent may be over-analyzing
+
+**You don't need to do anything.** The harness injects these automatically.
+
+### Repo Map (`repo_map` tool)
+
+**What it does:** Analyzes the import graph of the codebase and runs PageRank to surface the most-connected (and most-relevant) files. Call it at the start of any exploration or implementation task for fast orientation.
+
+**To use:**
+
+```
+repo_map {}                          # globally most-connected files
+repo_map { seedFiles: ["/abs/path"] } # personalized from known-relevant files
+```
+
+Results are cached at `.proto/repo-map-cache.json` and invalidated on file changes.
+
+### Behavior Verification Gate
+
+**What it does:** After every subagent task that completes successfully, the harness runs user-configured "verification scenarios" — shell commands that check your feature actually works. Failures are injected back to the agent for self-correction.
+
+**To configure:** Create `.proto/verify-scenarios.json`:
+
+```json
+[
+  {
+    "name": "Unit tests pass",
+    "command": "npm test -- --run",
+    "timeoutMs": 60000
+  },
+  {
+    "name": "Build succeeds",
+    "command": "npm run build",
+    "timeoutMs": 30000
+  },
+  {
+    "name": "API health check",
+    "command": "curl -sf http://localhost:3000/health",
+    "expectedPattern": "ok",
+    "timeoutMs": 5000
+  }
+]
+```
+
+See `.proto/verify-scenarios.example.json` for a full reference.
+
+### Multi-Sample Retry (`multi_sample: true`)
+
+**What it does:** When a subagent fails (doom loop, error, or max turns exceeded), the harness automatically retries up to 2 more times with escalating temperatures (0.7 → 1.0 → 1.3) and injects the failure context into each retry prompt. The best result among all attempts is returned and scored.
+
+**Scoring:**
+
+- GOAL + behavior gate pass → 3 (perfect)
+- GOAL + no gate / gate pass → 3
+- GOAL + gate fail → 2 (completed but not verified)
+- MAX_TURNS / TIMEOUT → 1 (partial)
+- ERROR → 0 (failure)
+
+**To enable:** Set `multi_sample: true` on the Agent tool call:
+
+```
+Agent {
+  subagent_type: "general-purpose",
+  prompt: "implement the auth service",
+  multi_sample: true
+}
+```
+
+Use for complex tasks with a history of failure, not for simple searches.
+
+### Sprint Contract Service
+
+**What it does:** Manages the full sprint contract lifecycle — parse, activate scope lock, persist to disk, load on resume. See the `sprint-contract` skill for usage.
+
+**Files involved:**
+
+- `.proto/sprint-contract.json` — persisted contract (restored on session start)
+- `SprintContractService` — programmatic API
+
+## Langfuse Fine-Tuning Data
+
+All harness interventions emit OTel spans routed to Langfuse via OTLP → Tempo. To build fine-tuning datasets:
+
+1. In Langfuse > Traces, filter by span name = `harness.intervention`
+2. Use `harness.intervention.type` attribute to segment by type:
+   - `doom_loop` — recovery from loops
+   - `scope_violation` — scope lock enforcement
+   - `verification_failed` — post-edit and behavior gate failures
+   - `reminder.*` — context reminders
+3. Export matching traces → dataset items
+4. Annotate `harness.outcome` = `"recovered"` | `"not_recovered"`
+5. Train on (input_context, intervention_message) pairs where outcome = recovered
+
+## Configuration Summary
+
+| Feature                 | Config location                                  | Default                              |
+| ----------------------- | ------------------------------------------------ | ------------------------------------ |
+| Doom loop threshold     | Code constant (`DOOM_REPEAT_THRESHOLD = 3`)      | Always on                            |
+| Scope lock              | `.proto/sprint-contract.json`                    | Off until sprint-contract skill runs |
+| Behavior gate scenarios | `.proto/verify-scenarios.json`                   | No scenarios (off)                   |
+| Multi-sample retry      | `multi_sample: true` on Agent call               | Off (opt-in)                         |
+| Observation mask window | Code constant (`INCREMENTAL_PROTECTED_TAIL`)     | Always on                            |
+| Harness reminders       | Code constants (50 calls / 3 failures / 8 turns) | Always on                            |

package/bundled/subagent-driven-development/SKILL.md

@@ -120,6 +120,16 @@ Implementer subagents report one of four statuses. Handle each appropriately:
 
 **Never** ignore an escalation or force the same model to retry without changes. If the implementer said it's stuck, something needs to change.
 
+## Harness Features
+
+The harness provides automatic safety nets you can leverage when dispatching implementers:
+
+**Multi-sample retry** (`multi_sample: true`): For complex or high-risk tasks, set this on the Agent tool call. If the implementer fails (doom loop, error, max turns), the harness automatically retries up to 2 more times with escalating temperatures (0.7 → 1.0 → 1.3) and injects the failure context into each retry prompt. Returns the best result. Use for tasks that have previously failed or that touch many files.
+
+**Behavior verification gate**: If `.proto/verify-scenarios.json` exists in the project, the harness runs those scenarios after every successful implementer completion. Failures are injected back to the model for self-correction. Add scenarios for smoke tests, build checks, and HTTP health checks.
+
+**Sprint contract scope lock**: If the implementing agent was given a sprint contract (via the `sprint-contract` skill), the scope lock prevents it from writing files outside the agreed set. Any violation is blocked and reported.
+
 ## Prompt Templates
 
 - `./implementer-prompt.md` - Dispatch implementer subagent