testchimp-runner-core 0.0.21 → 0.0.23

package/plandocs/ORCHESTRATOR_MVP_SUMMARY.md

@@ -0,0 +1,539 @@
+# Orchestrator Agent MVP: Final Specification
+
+## What We're Building
+
+**Single orchestrator agent** that replaces the current reactive sub-action loop with a proactive, tool-using, memory-maintaining system.
+
+---
+
+## Core Features (All in MVP)
+
+### 1. Always-Provided Context (Auto-Fetched Each Iteration)
+
+```typescript
+{
+  // WHERE AM I? (Journey tracking)
+  overallGoal: "Full scenario text",
+  currentStepGoal: "Login with alice@example.com, TestPass123",
+  stepNumber: 3,
+  totalSteps: 6,
+  completedSteps: ["Go to URL", "Click login"],
+  remainingSteps: ["Messages", "Send message", "Verify"],
+  
+  // WHAT DO I SEE? (Current state - fresh)
+  currentPageInfo: getEnhancedPageInfo(page),  // ARIA + IDs + data attrs (compact!)
+  currentURL: page.url(),
+  
+  // WHAT DID I JUST DO? (Recent memory ~6-7 steps - TEXT ONLY, NO SCREENSHOTS)
+  recentSteps: [
+    {
+      stepNumber: 2,
+      action: "Navigated to login page",
+      code: "await page.goto('https://...')",
+      result: "success",
+      observation: "Login form visible with email and password fields"
+    },
+    {
+      stepNumber: 2,
+      action: "Explored menu icon (hover)",
+      code: "explore_element({action: 'hover', selector: 'nav button:nth-child(2)'})",
+      result: "success",
+      observation: "Tooltip appeared showing 'Dashboard', confirmed this is the Dashboard button"
+    }
+  ],
+  
+  // WHAT HAVE I LEARNED? (Experiences)
+  experiences: [
+    "Site uses #id selectors for forms",
+    "Login redirects to /dashboard"
+  ],
+  
+  // WHAT DO I KNOW? (Extracted data)
+  extractedData: { userEmail: "alice@example.com" },
+  
+  // WHAT DID I TELL MYSELF? (Self-reflection from previous iteration)
+  previousIterationGuidance: {
+    guidanceForNext: "Check for redirect after submit",
+    detectingLoop: false
+  }
+}
+```
+
+**Optimized for complex pages:**
+- DOM: Pre-truncated with increased limits
+  - Interactive elements: top 50
+  - IDs: top 50
+  - Data attributes: top 50
+  - Form fields: top 20
+  - Page structure: top 10
+  - General elements: top 50
+  - Text: 30 chars max
+  - Result: ~800-1,500 tokens
+- Steps: Recent 6-7 only
+- Experiences: Cap at 20
+
+### 2. Optional Tools (Agent Requests)
+
+**8 core tools:**
+
+**Information Gathering:**
+1. **take_screenshot({isFullPage})** - Visual context (use freely)
+2. **recall_history({maxSteps, query})** - Deeper history
+3. **inspect_page()** - Get DOM (might be redundant since always-provided, but keeps extensibility)
+4. **check_page_ready()** - Verify page loaded
+
+**Data Management:**
+5. **extract_data({selector, dataName})** - Save data for later
+
+**Recovery Tools (Self-Unstuck):**
+6. **navigate_back()** - Go back in browser history (if exploratory action had side effects)
+7. **refresh_page()** - Reload current page (if page in bad state)
+8. **navigate_to_url({url})** - Navigate to specific URL (validate it's within allowed domain)
+
+**Inquisitive Exploration (Phase 2):**
+9. **explore_element({action, selector, purpose})** - Investigate ambiguous elements
+   - Actions: "hover", "click_info", "click_menu", "focus"
+   - Non-consequential only (no form submit, delete, etc.)
+   - Returns: {success, screenshotTaken, observation}
+
+**Extensible**: Add new tools → auto-appear in prompt
+
+**Recovery scenarios:**
+- Agent realizes it navigated away from domain → `navigate_back()` or `navigate_to_url({baseUrl})`
+- Page got stuck/unresponsive → `refresh_page()`
+- Exploratory action opened modal/overlay → `navigate_back()` or refresh
+- Lost context after redirect → `navigate_to_url()` to known good state
+
+### 3. Agent Decision (What Agent Outputs)
+
+```typescript
+{
+  // 1. Tool requests (multiple allowed)
+  toolCalls: [
+    {name: "take_screenshot", params: {isFullPage: false}},
+    {name: "recall_history", params: {maxSteps: 5}}
+  ],
+  toolReasoning: "Need visual + history to understand pattern",
+  needsToolResults: true,  // Wait for tools before commands
+  
+  // 2. Command batch (executed sequentially)
+  commands: [
+    "await page.fill('#email', 'alice@example.com')",
+    "await page.fill('#password', 'TestPass123')",
+    "await page.click('button[type=\"submit\"]')"
+  ],
+  commandReasoning: "Batch entire login flow",
+  
+  // 3. Self-reflection (FREE-FORM guidance to next iteration)
+  selfReflection: {
+    guidanceForNext: "After submit, check URL changed to /dashboard",
+    detectingLoop: false,  // Agent signals if repeating same approach
+    loopReasoning: null
+  },
+  
+  // 4. Learnings (stored in experiences)
+  experiences: [
+    "Forms use #id selectors consistently",
+    "Login redirects immediately after submit"
+  ],
+  
+  // 5. Memory update (what to store)
+  memoryUpdate: {
+    action: "Filled login form and submitted",
+    observation: "Page redirected to dashboard",
+    extractedData: {userEmail: "alice@example.com"}
+  },
+  
+  // 6. Termination decision
+  status: "complete",  // or "stuck" | "infeasible" | "continue"
+  statusReasoning: "Login completed, dashboard visible",
+  reasoning: "Overall iteration reasoning"
+}
+```
+
+### 4. Execution: Sequential with Early Stop
+
+**Agent plans batch, system executes sequentially:**
+
+```
+Agent: commands = [cmd1, cmd2, cmd3]
+
+Execute:
+  cmd1 → SUCCESS ✓ (record in history)
+  cmd2 → SUCCESS ✓ (record in history)
+  cmd3 → FAIL ✗ (record in history, STOP)
+  
+Result: 2/3 succeeded, accurately tracked
+```
+
+### 5. Comprehensive Logging
+
+**Every iteration logs:**
+```
+[Orchestrator] === Iteration 2/8 ===
+[Orchestrator] 🎯 Current Goal: Login with alice@example.com, TestPass123
+[Orchestrator] 📍 Progress: Step 2/6
+[Orchestrator] 💭 Reasoning: Form fields located, batching fill operations
+[Orchestrator] 🧠 Previous Guidance: Check for redirect after submit
+[Orchestrator] 🔧 Tools: [take_screenshot (viewport)]
+[Orchestrator] 📋 Tool Reasoning: Visual check for login button state
+[Orchestrator] ✓ Tools executed
+[Orchestrator] 📝 Commands (3): fill email, fill password, click submit
+[Orchestrator] 💡 Batch Reasoning: Can execute entire login in one go
+[Orchestrator] ▶ Executing sequentially:
+[Orchestrator]   ✓ [1/3] await page.fill('#email', 'alice@example.com')
+[Orchestrator]   ✓ [2/3] await page.fill('#password', 'TestPass123')
+[Orchestrator]   ✓ [3/3] await page.click('button[type="submit"]')
+[Orchestrator] 📚 Experiences: Site uses #id for forms, Login redirects to /dashboard
+[Orchestrator] 🧠 Next Guidance: Verify dashboard loaded, check for user menu
+[Orchestrator] 🔄 Loop Detection: false
+[Orchestrator] 🎯 Status: continue
+[Orchestrator] 💭 Status Reasoning: Commands executed, need to verify navigation
+```
+
+---
+
+## Inquisitive Exploration (Phase 2)
+
+### Problem: Ambiguous UI Elements
+
+**Scenario**: Agent needs to click "Dashboard" but menu items are icon-only (no text, no clear ARIA labels)
+
+**Solution**: Agent investigates non-consequentially before committing to actions
+
+### How It Works
+
+```typescript
+// ITERATION N - Agent realizes it needs more info:
+{
+  "toolCalls": [
+    {
+      "name": "explore_element",
+      "params": {
+        "action": "hover",
+        "selector": "nav button:nth-child(2)",
+        "purpose": "Check tooltip to see if this is Dashboard"
+      }
+    }
+  ],
+  "toolReasoning": "Menu items are icons without labels, need to hover to see tooltips",
+  "needsToolResults": true  // Agent waits for tool results before continuing
+}
+
+// System executes tool:
+1. Hover over element
+2. Wait 500ms for tooltip
+3. Take screenshot
+4. Call agent AGAIN with screenshot to analyze it
+5. Agent responds with analysis: "Tooltip shows 'Dashboard' text"
+6. System stores learning in history (TEXT, not screenshot):
+   {
+     action: "Explored menu icon (hover)",
+     code: "explore_element(...)",
+     result: "success",
+     observation: "Tooltip appeared showing 'Dashboard', confirmed this is the Dashboard button"
+   }
+7. Tool returns to original agent call: {success: true, learning: "Tooltip says Dashboard"}
+
+// SAME ITERATION N - Agent receives tool result (TEXT learning, no screenshot):
+{
+  "toolResults": {
+    "explore_element": {
+      "success": true,
+      "learning": "Tooltip appeared showing 'Dashboard', confirmed this is the Dashboard button"
+    }
+  }
+}
+
+// Agent now has confidence to proceed with commands:
+{
+  "commands": ["await page.click('nav button:nth-child(2)')"],
+  "commandReasoning": "Exploration confirmed this is Dashboard button via tooltip"
+}
+```
+
+**Key: Screenshot analyzed immediately, only TEXT learnings stored/passed forward**
+
+### Allowed Exploration Actions
+
+**Non-consequential only:**
+
+1. **hover** - Show tooltips, menus, dropdowns
+   - Safe: Doesn't change state
+   - Use: Reveal hidden info
+
+2. **click_info** - Click info icons, help buttons
+   - Safe: Usually opens modal/tooltip
+   - Use: Get more context
+   - Risk: Modal might block page (can navigate_back)
+
+3. **click_menu** - Click menu headers to reveal items
+   - Safe: Just expands menu
+   - Use: See menu options
+   - Risk: Menu might navigate (rare)
+
+4. **focus** - Focus on input to see placeholder/validation
+   - Safe: Just focuses element
+   - Use: See input hints
+
+**NOT allowed:**
+- ❌ Submit forms
+- ❌ Delete/remove actions
+- ❌ Purchase/confirm buttons
+- ❌ Logout
+- ❌ File uploads
+- ❌ Navigation links (unless explicitly exploratory)
+
+### Exploration Workflow
+
+```
+Iteration N:
+  Agent Decision 1: "Need to explore menu icons"
+  Tool: explore_element(hover, selector)
+  
+  System executes:
+    → Hover element
+    → Take screenshot
+    → Call agent with screenshot: "Analyze this, what do you see?"
+  
+  Agent Decision 2 (sub-call): "I see tooltip says 'Dashboard'"
+  
+  System stores: 
+    → History entry (TEXT): "Explored button, tooltip shows Dashboard"
+    → No screenshot stored
+  
+  System returns to Agent Decision 1:
+    → Tool result: {success: true, learning: "Tooltip shows Dashboard"}
+  
+  Agent Decision 1 continues: "Great! Now I can click it"
+  Commands: [click that element]
+  
+  System executes commands sequentially
+```
+
+**Key: Screenshot analyzed IMMEDIATELY within same iteration, only TEXT learning stored**
+
+**Benefits:**
+- No screenshots stored in memory (saves tokens)
+- Immediate feedback (no waiting for next iteration)
+- Structured learning extraction
+- Future iterations only see concise text observations
+
+### Guardrails
+
+```typescript
+interface AgentConfig {
+  // Per iteration
+  maxExploratoryActionsPerIteration: 3,  // Can explore up to 3 elements per iteration
+  
+  // Per step
+  maxExploratoryActionsPerStep: 10,  // Total exploration budget per step
+  
+  // Safety
+  explorationTimeout: 2000,  // Max wait for tooltip/menu
+  allowedExplorationActions: ['hover', 'click_info', 'click_menu', 'focus']
+}
+
+// System enforces:
+if (explorationAction.action === 'click' && selector.includes('submit')) {
+  logger.error('SYSTEM: Exploratory click on submit button blocked');
+  return {success: false, reason: 'unsafe_action'};
+}
+
+if (explorationCount > config.maxExploratoryActionsPerStep) {
+  logger.warn('SYSTEM: Exploration budget exhausted');
+  // Remove explore_element from available tools
+}
+```
+
+### State Validation
+
+**Before exploration:**
+```typescript
+const beforeState = {
+  url: page.url(),
+  modalCount: await page.locator('[role="dialog"]').count()
+};
+```
+
+**After exploration:**
+```typescript
+const afterState = {
+  url: page.url(),
+  modalCount: await page.locator('[role="dialog"]').count()
+};
+
+// Check for unexpected navigation
+if (beforeState.url !== afterState.url) {
+  logger.warn('Exploration caused navigation, reverting');
+  await page.goBack();
+}
+
+// Modal opened (might be intended)
+if (afterState.modalCount > beforeState.modalCount) {
+  observation = "Modal opened (may need to close)";
+}
+```
+
+### When Agent Uses Exploration
+
+**Agent decides based on:**
+- ❓ DOM shows icons without text
+- ❓ Multiple similar elements, unclear which is correct
+- ❓ Need to see menu contents before deciding
+- ❓ Input field needs to show validation rules
+
+**Example agent reasoning:**
+```
+"DOM shows 5 icon buttons without labels. Need to hover over each 
+to see tooltips and identify which is Dashboard. Will explore 
+buttons 1-3 this iteration."
+```
+
+### Why Phase 2 (Not MVP)
+
+**Reasons to defer:**
+1. **Safety risk** - Need robust validation to prevent state changes
+2. **Complexity** - Requires screenshot handling, state comparison
+3. **Edge cases** - Modals, overlays, navigation need careful handling
+4. **Testing needed** - Validate on multiple sites before including
+
+**MVP workaround:**
+- Agent uses `take_screenshot()` + DOM analysis
+- Makes best guess from available info
+- If wrong, retry with different approach
+- Less elegant but safer
+
+**Add in Phase 2 when:**
+- MVP validated and working
+- Identified common patterns where exploration helps
+- State validation logic battle-tested
+
+---
+
+## Guardrails
+
+### System-Enforced (Hard Limits)
+
+```typescript
+interface AgentConfig {
+  // Per-step
+  maxIterationsPerStep: 8,
+  maxToolCallsPerIteration: 5,
+  maxCommandsPerIteration: 5,
+  
+  // Scenario-wide
+  maxConsecutiveStepFailures: 2,
+  maxTotalIterations: 50,  // Across all steps
+  
+  // Memory
+  maxExperiences: 20,
+  maxHistorySize: 100,
+  recentStepsCount: 7  // How many in always-provided
+}
+
+// System checks BEFORE and AFTER agent call
+if (iteration > config.maxIterationsPerStep) {
+  logger.warn('SYSTEM: Iteration limit reached');
+  return {success: false, reason: 'system_limit'};
+}
+```
+
+### Agent Self-Awareness (Soft Signals)
+
+```typescript
+// Agent can signal issues:
+{
+  "status": "stuck",
+  "statusReasoning": "Tried 4 different selectors, none work - element likely doesn't exist"
+}
+
+{
+  "selfReflection": {
+    "detectingLoop": true,
+    "loopReasoning": "I've tried text-based selectors 3 times, need completely different approach"
+  }
+}
+
+// System respects agent signals but also enforces hard limits
+```
+
+**No screenshot budget** - Agent can use screenshots freely
+
+---
+
+## MVP Scope (Complete Feature Set)
+
+### Included in MVP:
+- ✅ OrchestratorAgent with tool-use
+- ✅ Dynamic ToolRegistry
+- ✅ **8 core tools**:
+  - Information: take_screenshot, recall_history, inspect_page, check_page_ready
+  - Data: extract_data
+  - Recovery: navigate_back, refresh_page, navigate_to_url
+- ✅ Journey memory (history + experiences + extracted data)
+- ✅ Always-provided context (goal + DOM + recent 7 steps + self-reflection)
+- ✅ **Free-form self-reflection** (train of thought continuity)
+- ✅ **Agent loop detection** (detectingLoop flag)
+- ✅ **Self-recovery** (navigate_back, refresh_page when stuck)
+- ✅ Batch command planning (max 3-5)
+- ✅ Sequential execution (stop on failure)
+- ✅ Experience accumulation
+- ✅ Configurable guardrails (per job)
+- ✅ **Token usage tracking** (input/output/image)
+- ✅ **Comprehensive logging** (all thoughts visible)
+- ✅ Works for generation AND repair modes
+
+### Excluded (Phase 2):
+- ❌ Exploratory actions (explore_element tool - safety concerns)
+- ❌ Advanced optimizations (caching, adaptive limits)
+- ❌ Memory summarization
+
+---
+
+## Key Decisions
+
+1. ✅ **Self-reflection in MVP** - Valuable for continuity, agent detects own loops
+2. ✅ **No screenshot budget** - Use freely when helpful
+3. ✅ **DOM always-provided** - Already compact via getEnhancedPageInfo
+4. ✅ **Recent 6-7 steps** - Enough context without bloat
+5. ✅ **Agent + system guardrails** - Agent signals, system enforces
+6. ✅ **Sequential batch execution** - Plan together, execute one-by-one
+7. ✅ **Repair mode support** - Script → Agent on failure → Script
+
+---
+
+## Why This Works
+
+**Agent maintains train of thought:**
+- Iteration 1: "Try #id selectors" → succeeds
+- Iteration 2 self-reflection: "IDs worked, continue using them"
+- Iteration 3: Uses IDs again → succeeds faster
+
+**Agent detects spirals:**
+- Iteration 1-2-3: Tries text selectors, all fail
+- Iteration 4 self-reflection: detectingLoop=true, "Text doesn't work, switching to IDs"
+- Breaks own loop before system limit
+
+**Human-like:**
+- Remember what just happened (recent steps)
+- Learn patterns (experiences)
+- Maintain train of thought (self-reflection)
+- Know when stuck (loop detection)
+- Use tools when needed (screenshot, history)
+
+---
+
+## Ready to Implement
+
+**Full MVP specification complete** with:
+- All architecture decisions finalized
+- Self-reflection included with loop detection
+- No screenshot budgeting
+- DOM optimization validated
+- Comprehensive logging defined
+- Repair mode integration planned
+- Guardrails configured
+
+**Estimated implementation**: 2-3 weeks for complete MVP
+