testchimp-runner-core 0.0.21 → 0.0.23

package/plandocs/HUMAN_LIKE_IMPROVEMENTS.md

@@ -0,0 +1,642 @@
+# Human-Like Script Generation Improvements
+
+## Current Gaps vs Human Behavior
+
+### Gap 1: **Reactive vs Proactive Planning**
+
+**Current (Reactive):**
+```
+Step: "Login with credentials: admin, pass123"
+1. Generate command → fill(username)
+2. Execute → success
+3. Check goal → incomplete
+4. Generate command → fill(password)
+5. Execute → success
+6. Check goal → incomplete
+7. Generate command → click(login)
+8. Execute → success
+9. Check goal → complete
+```
+**3 LLM calls + 3 goal checks = slow, expensive, non-deterministic**
+
+**Human (Proactive):**
+```
+Step: "Login with credentials: admin, pass123"
+Looks at page → sees login form with username, password, and button
+Plans: fill(username) → fill(password) → click(login)
+Executes all 3 → done
+```
+**1 planning call + execution = fast, deterministic**
+
+### Gap 2: **Excessive Retries**
+
+**Current:** 
+- 4 attempts per sub-action × 5 sub-actions = up to 20 attempts per step
+- Humans don't retry 20 times - they try 2-3 approaches max
+
+**Human:**
+- Try primary approach (e.g., click by text)
+- If fails, try alternative (e.g., click by role)
+- If still fails, reassess situation (vision/give up)
+
+### Gap 3: **No Pattern Recognition**
+
+**Current:**
+- Every login form treated as novel
+- Every navigation treated independently
+- No learning from previous successful patterns
+
+**Human:**
+- Recognizes "this is a login form" → knows the pattern
+- Recognizes "this is a modal" → knows to look for close button
+- Learns from experience
+
+### Gap 4: **Sub-Action Approach is Non-Deterministic**
+
+**Current:**
+```
+Step: "Click on All Modules"
+Sub-action 1: Try approach A → fails
+Sub-action 2: Try approach B → fails  
+Sub-action 3: Try approach C → fails
+Sub-action count: 3, failed attempts: 8
+```
+
+**Human:**
+```
+Step: "Click on All Modules"
+Looks at page → sees "All Modules" menu
+Clicks it → if fails, tries once more with better selector
+Max 2-3 attempts total
+```
+
+### Gap 5: **No Confidence Scoring**
+
+**Current:**
+- LLM generates command with no confidence indication
+- System doesn't know if LLM is guessing or certain
+
+**Human:**
+- "I see exactly where to click" → high confidence, execute
+- "I'm not sure which element" → low confidence, look more carefully first
+
+## Proposed Improvements
+
+### 1. **Upfront Step Planning Mode** 🎯
+
+Add a planning phase before execution:
+
+```typescript
+// NEW: Step planning API
+async planStepExecution(
+  stepDescription: string,
+  pageInfo: PageInfo
+): Promise<StepExecutionPlan> {
+  // LLM analyzes page and creates complete plan for step
+  return {
+    commands: [
+      { code: "await page.fill('username', 'admin')", confidence: 0.95 },
+      { code: "await page.fill('password', 'pass')", confidence: 0.95 },
+      { code: "await page.click('button[type=submit]')", confidence: 0.85 }
+    ],
+    reasoning: "Login form detected with clear username/password fields and submit button",
+    overallConfidence: 0.90,
+    needsVisionUpfront: false // LLM indicates if vision would help BEFORE attempting
+  }
+}
+```
+
+**Benefits:**
+- Single LLM call for entire step
+- All commands generated together (more coherent)
+- Confidence scoring guides execution
+- Can request vision upfront if uncertain
+
+### 2. **Pattern Recognition System** 📚
+
+Build common pattern library:
+
+```typescript
+interface ActionPattern {
+  name: string;
+  trigger: (pageInfo: PageInfo) => boolean; // DOM-based detection
+  template: (params: any) => string[]; // Command sequence
+}
+
+const COMMON_PATTERNS = {
+  LOGIN_FORM: {
+    trigger: (page) => hasFields(['username', 'password']) && hasButton('login'),
+    template: ({ username, password }) => [
+      `await page.fill('[name="username"]', '${username}')`,
+      `await page.fill('[name="password"]', '${password}')`,
+      `await page.click('button[type="submit"]')`
+    ]
+  },
+  MODAL_WITH_CLOSE: {
+    trigger: (page) => page.title.includes('modal') || hasElement('[role="dialog"]'),
+    template: () => [
+      `await page.click('[aria-label="Close"]')`
+    ]
+  },
+  SEARCH_BAR: {
+    trigger: (page) => hasSearchInput(),
+    template: ({ query }) => [
+      `await page.fill('[type="search"]', '${query}')`,
+      `await page.keyboard.press('Enter')`
+    ]
+  }
+}
+```
+
+**Flow:**
+1. Check if current page matches known pattern
+2. If match + high confidence → use pattern template
+3. If no match or low confidence → use LLM
+
+**Benefits:**
+- Deterministic for common cases
+- Faster execution (no LLM call for patterns)
+- More reliable (tested patterns)
+- LLM handles edge cases
+
+### 3. **Smarter Retry Strategy** 🔄
+
+Replace sub-action approach with strategic retries:
+
+```typescript
+interface RetryStrategy {
+  maxAttempts: 2; // Humans try 2-3 times max
+  approaches: [
+    'primary',    // First attempt: Use most obvious selector
+    'alternative', // Second attempt: Try different selector strategy
+    'vision'      // Third attempt: Use vision if previous failed
+  ]
+}
+
+// Instead of:
+// Sub-action 1 (4 attempts) → Sub-action 2 (4 attempts) → Sub-action 3 (4 attempts)
+// = 12 attempts with different commands
+
+// Do:
+// Attempt 1: Primary approach
+// Attempt 2: Alternative selector strategy
+// Attempt 3: Vision-guided (if needed)
+// = 3 attempts max, clear progression
+```
+
+**Benefits:**
+- Predictable attempt count
+- Clear escalation path
+- More human-like (try twice, then ask for help/vision)
+
+### 4. **Context-Aware Command Generation** 🧠
+
+Use previous step context more effectively:
+
+```typescript
+async generateCommandWithContext(
+  stepDescription: string,
+  pageInfo: PageInfo,
+  previousSuccessfulCommands: string[], // What worked before
+  recentPatterns: PatternMatch[] // Patterns detected recently
+): Promise<Command> {
+  
+  const prompt = `
+  Previous successful patterns in this session:
+  - Username field used: page.fill('[name="username"]', ...)
+  - Buttons clicked using: page.getByRole('button', ...)
+  
+  Use similar approaches for consistency.
+  
+  Current step: ${stepDescription}
+  `;
+}
+```
+
+**Benefits:**
+- Commands consistent within session
+- Learns what selectors work on this app
+- Faster convergence
+
+### 5. **Confidence-Based Execution** 📊
+
+Add confidence scoring to guide execution:
+
+```typescript
+interface CommandWithConfidence {
+  code: string;
+  confidence: number; // 0-1
+  reasoning: string;
+}
+
+// Execution logic:
+if (command.confidence < 0.6) {
+  // Low confidence - use vision upfront instead of trying and failing
+  logger.log("Low confidence, requesting vision analysis before attempting");
+  const visionDiagnostics = await getVisionDiagnostics(...);
+  command = regenerateWithVisionInsights(visionDiagnostics);
+} else if (command.confidence < 0.8) {
+  // Medium confidence - try but have vision ready
+  logger.log("Medium confidence, will use vision if this fails");
+} else {
+  // High confidence - proceed normally
+  logger.log("High confidence, executing");
+}
+```
+
+**Benefits:**
+- Vision used proactively when LLM is uncertain
+- Fewer wasted attempts on guesses
+- More human-like (ask for help when unsure)
+
+### 6. **Simplified Goal Model** 🎯
+
+Remove sub-action pattern, use clearer model:
+
+```typescript
+// Current (complex):
+// Step → Sub-actions (1-5) → Attempts (0-3) → Goal checks after each sub-action
+// = nested loops, hard to reason about
+
+// Proposed (simple):
+// Step → Plan (upfront) → Execute commands → Check completion once
+// = linear flow, easy to understand
+
+async executeStep(step: Step, page: Page): Promise<StepResult> {
+  // 1. Plan the step (single LLM call)
+  const plan = await planStepExecution(step.description, getPageInfo(page));
+  
+  // 2. Execute each command in plan
+  for (const command of plan.commands) {
+    try {
+      await executeCommand(command, page);
+    } catch (error) {
+      // If any command fails, replan with error context
+      if (plan.overallConfidence > 0.8) {
+        // Was confident, try alternative selector
+        const alt = await regenerateCommand(command, error, page);
+        await executeCommand(alt, page);
+      } else {
+        // Was uncertain, use vision
+        const vision = await getVisionGuidedCommand(command, error, page);
+        await executeCommand(vision, page);
+      }
+    }
+  }
+  
+  // 3. Single goal check at end
+  return checkGoalCompletion(step.description, plan.commands, page);
+}
+```
+
+**Benefits:**
+- Predictable execution flow
+- Fewer LLM calls (plan once, not per sub-action)
+- Easier to debug and reason about
+- More human-like (plan → execute → verify)
+
+### 7. **Smart DOM Analysis** 🔍
+
+Better upfront DOM understanding:
+
+```typescript
+interface PageUnderstanding {
+  pageType: 'login' | 'dashboard' | 'form' | 'modal' | 'list' | 'unknown';
+  primaryInteractiveElements: Element[];
+  forms: FormInfo[];
+  modals: ModalInfo[];
+  navigation: NavigationInfo[];
+  confidence: number;
+}
+
+async analyzePageContext(page: Page): Promise<PageUnderstanding> {
+  const pageInfo = await getEnhancedPageInfo(page);
+  
+  // Detect page type and structure
+  const analysis = await llm.analyzePage(pageInfo);
+  
+  return {
+    pageType: 'login', // Detected
+    forms: [{ fields: ['username', 'password'], submitButton: 'Login to Continue' }],
+    confidence: 0.95
+  };
+}
+```
+
+**Usage:**
+```typescript
+const pageContext = await analyzePageContext(page);
+if (pageContext.pageType === 'login' && step.description.includes('login')) {
+  // High confidence - use login pattern
+  const form = pageContext.forms[0];
+  return generateLoginCommands(form, credentials);
+}
+```
+
+**Benefits:**
+- Understands page structure upfront
+- Can detect common patterns automatically
+- Informs better command generation
+- More deterministic (same page type → same approach)
+
+### 8. **Reduce Noise in Execution** 🔇
+
+**Current issues:**
+- Too many goal completion checks (after every sub-action)
+- Too many retry variations
+- Too much back-and-forth with LLM
+
+**Improvement:**
+```typescript
+// Instead of:
+// fill(username) → goal check → fill(password) → goal check → click(login) → goal check
+// = 6 operations (3 actions + 3 checks)
+
+// Do:
+// Plan: [fill(username), fill(password), click(login)]
+// Execute all → single goal check
+// = 4 operations (3 actions + 1 check)
+```
+
+**Benefits:**
+- 50% fewer LLM calls for multi-action steps
+- Faster execution
+- More predictable
+
+### 9. **Visual Scanning Before Action** 👁️
+
+Add optional visual scan for complex steps:
+
+```typescript
+async executeComplexStep(step: Step, page: Page): Promise<Result> {
+  // Check if step seems complex/ambiguous
+  const complexity = assessStepComplexity(step.description);
+  
+  if (complexity === 'high' && visualBudgetAvailable) {
+    // Proactive vision for complex steps
+    logger.log("Complex step detected - using vision upfront for better planning");
+    const screenshot = await page.screenshot();
+    const plan = await generatePlanWithVision(step, pageInfo, screenshot);
+    return executePlan(plan);
+  } else {
+    // Standard DOM-based approach
+    return executeStepStandard(step, page);
+  }
+}
+```
+
+**When to use:**
+- Ambiguous descriptions ("click the settings icon")
+- Multiple similar elements expected
+- Visual-heavy interactions
+- Previous similar steps failed
+
+**Benefits:**
+- Fewer wasted attempts
+- Higher first-attempt success rate
+- More human-like (look before you leap)
+
+### 10. **Session Memory** 💭
+
+Remember what works across steps:
+
+```typescript
+interface SessionMemory {
+  workingSelectors: Map<string, string>; // "login button" → "button[name='login']"
+  pagePatterns: string[]; // "This app uses data-testid"
+  commonElements: Element[]; // Persistent nav/header elements
+  successfulStrategies: string[]; // What approaches worked
+}
+
+// Example usage:
+if (sessionMemory.workingSelectors.has('login button')) {
+  // We found login button before, use same selector
+  const selector = sessionMemory.workingSelectors.get('login button');
+  return `await page.click('${selector}')`;
+}
+
+if (sessionMemory.pagePatterns.includes('uses-data-testid')) {
+  // This app uses data-testid, prefer that
+  return `await page.click('[data-testid="login"]')`;
+}
+```
+
+**Benefits:**
+- Consistency across steps
+- Learn from early steps
+- Deterministic within session
+- More human-like (remember what worked)
+
+## Recommended Implementation Priority
+
+### Phase 1: **Low-Hanging Fruit** (Quick Wins)
+
+1. ✅ **Reduce retry budget**
+   - MAX_SUBACTIONS_PER_STEP: 5 → 3 (humans try 2-3 approaches max)
+   - MAX_RETRIES_PER_STEP: 3 → 2 (3 attempts per sub-action → 2)
+   - MAX_FAILED_ATTEMPTS_PER_STEP: 12 → 8
+
+2. ✅ **Confidence scoring**
+   - Add confidence field to LLM responses
+   - Trigger vision earlier for low confidence (< 0.6)
+   - Log confidence for debugging
+
+3. ✅ **Session memory (simple version)**
+   - Track successful selectors in current session
+   - Reuse patterns that worked
+   - Pass to LLM as context
+
+### Phase 2: **Structural Improvements** (Medium Effort)
+
+4. **Upfront step planning**
+   - New API: `planStepExecution()`
+   - Generate all commands for step at once
+   - Execute plan without goal checks between commands
+   - Single goal check at end
+
+5. **Pattern detection**
+   - Detect common patterns (login, search, modal, dropdown)
+   - Use templates for high-confidence matches
+   - Fallback to LLM for novel cases
+
+6. **Smarter retry strategy**
+   - Replace sub-action approach with: primary → alternative → vision
+   - Clear escalation path
+   - Fail faster
+
+### Phase 3: **Advanced Features** (Future)
+
+7. **Proactive vision for complex steps**
+   - Complexity scoring
+   - Visual scan before attempting
+   - Better first-attempt success
+
+8. **Cross-session learning**
+   - Persist patterns across sessions
+   - Build app-specific knowledge base
+   - Improve over time
+
+9. **Multi-step lookahead**
+   - Plan multiple steps at once when related
+   - Optimize for workflow efficiency
+   - Batch similar operations
+
+## Immediate Actionable Changes
+
+### Change 1: Reduce Retry Budget (More Human-Like)
+
+```typescript
+// In scenario-worker-class.ts
+const MAX_RETRIES_PER_STEP = 2; // Was 3 → Now 2 attempts per approach (total 3 tries)
+const MAX_SUBACTIONS_PER_STEP = 3; // Was 5 → Now 3 different approaches max  
+const MAX_FAILED_ATTEMPTS_PER_STEP = 8; // Was 12 → Now 8 total failures max
+```
+
+**Rationale:** Humans try 2-3 things, not 12. Fail faster = faster feedback.
+
+### Change 2: Add Confidence Scoring
+
+**Modify LLM response:**
+```typescript
+interface LLMPlaywrightCommandResponse {
+  command: string;
+  reasoning: string;
+  confidence: number; // NEW: 0-1 scale
+  uncertaintyReason?: string; // NEW: Why low confidence
+}
+```
+
+**Prompt addition:**
+```
+Additionally, provide a confidence score (0-1):
+- 1.0: Element clearly identified in DOM with unique selector
+- 0.8: Element found but selector might not be unique  
+- 0.6: Multiple possible elements, best guess
+- 0.4: Element not clearly in DOM, inferring from context
+- 0.2: Guessing, likely to fail
+
+If confidence < 0.7, explain uncertainty.
+```
+
+**Usage:**
+```typescript
+const result = await generatePlaywrightCommand(...);
+if (result.confidence < 0.6 && !usedVisionMode) {
+  logger.log(`⚠️ Low confidence (${result.confidence}) - using vision proactively`);
+  // Skip attempts, go straight to vision
+  const visionResult = await getVisionGuidedCommand(...);
+  return visionResult;
+}
+```
+
+### Change 3: Session Memory (Simple Version)
+
+```typescript
+class SessionContext {
+  successfulSelectors: Map<string, string> = new Map();
+  appPatterns: Set<string> = new Set(); // 'uses-data-testid', 'uses-aria-labels', etc.
+  
+  recordSuccess(elementDescription: string, selector: string) {
+    this.successfulSelectors.set(elementDescription, selector);
+  }
+  
+  getContext(): string {
+    if (this.successfulSelectors.size === 0) return '';
+    
+    return `
+    Selectors that worked in this session:
+    ${Array.from(this.successfulSelectors.entries())
+      .map(([desc, sel]) => `- "${desc}": ${sel}`)
+      .join('\n')}
+    
+    Try similar patterns for consistency.
+    `;
+  }
+}
+```
+
+**Add to command generation prompt:**
+```typescript
+${sessionContext.getContext()}
+```
+
+### Change 4: Skip Sub-Action Loop for High Confidence
+
+```typescript
+// After planning, if all commands have high confidence (>0.8):
+if (plan.overallConfidence > 0.8) {
+  // Execute entire plan without intermediate checks
+  for (const cmd of plan.commands) {
+    await execute(cmd);
+  }
+  // Single goal check at end
+  return checkGoalCompletion(step, plan.commands, page);
+} else {
+  // Low confidence - use current sub-action approach for safety
+  return executeWithSubActions(step, page);
+}
+```
+
+## Expected Improvements
+
+### Metrics:
+
+| Metric | Current | After Phase 1 | After Phase 2 |
+|--------|---------|---------------|---------------|
+| Avg LLM calls per step | 4-6 | 3-4 | 1-2 |
+| Max attempts per step | 20 | 8 | 6 |
+| Success rate (1st attempt) | ~40% | ~50% | ~70% |
+| Time per step | 30-60s | 20-40s | 10-20s |
+| Determinism | Low | Medium | High |
+
+### Behavioral Changes:
+
+**Login scenario:**
+
+**Current:**
+```
+1. fill(user) → check goal → incomplete
+2. fill(pass) → check goal → incomplete  
+3. click(login) → check goal → complete
+= 3 actions + 3 goal checks = 6 operations
+```
+
+**After Phase 1 (confidence + memory):**
+```
+1. fill(user) → check goal → incomplete (but with confidence: 0.9)
+2. fill(pass) → check goal → incomplete (confidence: 0.9)
+3. click(login) → check goal → complete (confidence: 0.85)
+= 3 actions + 3 checks, but with confidence signals
+```
+
+**After Phase 2 (planning):**
+```
+1. Plan: [fill(user), fill(pass), click(login)] (confidence: 0.9)
+2. Execute all 3 commands
+3. Single goal check → complete
+= 1 plan + 3 actions + 1 check = 5 operations
+```
+
+## Summary
+
+**Core Philosophy Shift:**
+
+**From:** Reactive trial-and-error with many retries  
+**To:** Proactive planning with fewer, smarter attempts
+
+**Most Important:**
+1. Plan before acting (not generate-execute-check loop)
+2. Use confidence to guide strategy
+3. Remember what works
+4. Fail faster (humans don't retry 12 times)
+5. Recognize common patterns
+
+**Most Human-Like = Most Deterministic**
+- Humans plan, then execute
+- Humans learn from experience  
+- Humans don't endlessly retry
+- Humans recognize familiar patterns
+- Humans ask for help when uncertain
+
+These improvements make the system more deterministic, faster, cheaper, and more aligned with how humans actually interact with web applications.
+