testchimp-runner-core 0.0.35 → 0.0.37

package/plandocs/HUMAN_LIKE_IMPROVEMENTS.md

@@ -1,642 +0,0 @@
-# Human-Like Script Generation Improvements
-
-## Current Gaps vs Human Behavior
-
-### Gap 1: **Reactive vs Proactive Planning**
-
-**Current (Reactive):**
-```
-Step: "Login with credentials: admin, pass123"
-1. Generate command → fill(username)
-2. Execute → success
-3. Check goal → incomplete
-4. Generate command → fill(password)
-5. Execute → success
-6. Check goal → incomplete
-7. Generate command → click(login)
-8. Execute → success
-9. Check goal → complete
-```
-**3 LLM calls + 3 goal checks = slow, expensive, non-deterministic**
-
-**Human (Proactive):**
-```
-Step: "Login with credentials: admin, pass123"
-Looks at page → sees login form with username, password, and button
-Plans: fill(username) → fill(password) → click(login)
-Executes all 3 → done
-```
-**1 planning call + execution = fast, deterministic**
-
-### Gap 2: **Excessive Retries**
-
-**Current:** 
-- 4 attempts per sub-action × 5 sub-actions = up to 20 attempts per step
-- Humans don't retry 20 times - they try 2-3 approaches max
-
-**Human:**
-- Try primary approach (e.g., click by text)
-- If fails, try alternative (e.g., click by role)
-- If still fails, reassess situation (vision/give up)
-
-### Gap 3: **No Pattern Recognition**
-
-**Current:**
-- Every login form treated as novel
-- Every navigation treated independently
-- No learning from previous successful patterns
-
-**Human:**
-- Recognizes "this is a login form" → knows the pattern
-- Recognizes "this is a modal" → knows to look for close button
-- Learns from experience
-
-### Gap 4: **Sub-Action Approach is Non-Deterministic**
-
-**Current:**
-```
-Step: "Click on All Modules"
-Sub-action 1: Try approach A → fails
-Sub-action 2: Try approach B → fails  
-Sub-action 3: Try approach C → fails
-Sub-action count: 3, failed attempts: 8
-```
-
-**Human:**
-```
-Step: "Click on All Modules"
-Looks at page → sees "All Modules" menu
-Clicks it → if fails, tries once more with better selector
-Max 2-3 attempts total
-```
-
-### Gap 5: **No Confidence Scoring**
-
-**Current:**
-- LLM generates command with no confidence indication
-- System doesn't know if LLM is guessing or certain
-
-**Human:**
-- "I see exactly where to click" → high confidence, execute
-- "I'm not sure which element" → low confidence, look more carefully first
-
-## Proposed Improvements
-
-### 1. **Upfront Step Planning Mode** 🎯
-
-Add a planning phase before execution:
-
-```typescript
-// NEW: Step planning API
-async planStepExecution(
-  stepDescription: string,
-  pageInfo: PageInfo
-): Promise<StepExecutionPlan> {
-  // LLM analyzes page and creates complete plan for step
-  return {
-    commands: [
-      { code: "await page.fill('username', 'admin')", confidence: 0.95 },
-      { code: "await page.fill('password', 'pass')", confidence: 0.95 },
-      { code: "await page.click('button[type=submit]')", confidence: 0.85 }
-    ],
-    reasoning: "Login form detected with clear username/password fields and submit button",
-    overallConfidence: 0.90,
-    needsVisionUpfront: false // LLM indicates if vision would help BEFORE attempting
-  }
-}
-```
-
-**Benefits:**
-- Single LLM call for entire step
-- All commands generated together (more coherent)
-- Confidence scoring guides execution
-- Can request vision upfront if uncertain
-
-### 2. **Pattern Recognition System** 📚
-
-Build common pattern library:
-
-```typescript
-interface ActionPattern {
-  name: string;
-  trigger: (pageInfo: PageInfo) => boolean; // DOM-based detection
-  template: (params: any) => string[]; // Command sequence
-}
-
-const COMMON_PATTERNS = {
-  LOGIN_FORM: {
-    trigger: (page) => hasFields(['username', 'password']) && hasButton('login'),
-    template: ({ username, password }) => [
-      `await page.fill('[name="username"]', '${username}')`,
-      `await page.fill('[name="password"]', '${password}')`,
-      `await page.click('button[type="submit"]')`
-    ]
-  },
-  MODAL_WITH_CLOSE: {
-    trigger: (page) => page.title.includes('modal') || hasElement('[role="dialog"]'),
-    template: () => [
-      `await page.click('[aria-label="Close"]')`
-    ]
-  },
-  SEARCH_BAR: {
-    trigger: (page) => hasSearchInput(),
-    template: ({ query }) => [
-      `await page.fill('[type="search"]', '${query}')`,
-      `await page.keyboard.press('Enter')`
-    ]
-  }
-}
-```
-
-**Flow:**
-1. Check if current page matches known pattern
-2. If match + high confidence → use pattern template
-3. If no match or low confidence → use LLM
-
-**Benefits:**
-- Deterministic for common cases
-- Faster execution (no LLM call for patterns)
-- More reliable (tested patterns)
-- LLM handles edge cases
-
-### 3. **Smarter Retry Strategy** 🔄
-
-Replace sub-action approach with strategic retries:
-
-```typescript
-interface RetryStrategy {
-  maxAttempts: 2; // Humans try 2-3 times max
-  approaches: [
-    'primary',    // First attempt: Use most obvious selector
-    'alternative', // Second attempt: Try different selector strategy
-    'vision'      // Third attempt: Use vision if previous failed
-  ]
-}
-
-// Instead of:
-// Sub-action 1 (4 attempts) → Sub-action 2 (4 attempts) → Sub-action 3 (4 attempts)
-// = 12 attempts with different commands
-
-// Do:
-// Attempt 1: Primary approach
-// Attempt 2: Alternative selector strategy
-// Attempt 3: Vision-guided (if needed)
-// = 3 attempts max, clear progression
-```
-
-**Benefits:**
-- Predictable attempt count
-- Clear escalation path
-- More human-like (try twice, then ask for help/vision)
-
-### 4. **Context-Aware Command Generation** 🧠
-
-Use previous step context more effectively:
-
-```typescript
-async generateCommandWithContext(
-  stepDescription: string,
-  pageInfo: PageInfo,
-  previousSuccessfulCommands: string[], // What worked before
-  recentPatterns: PatternMatch[] // Patterns detected recently
-): Promise<Command> {
-  
-  const prompt = `
-  Previous successful patterns in this session:
-  - Username field used: page.fill('[name="username"]', ...)
-  - Buttons clicked using: page.getByRole('button', ...)
-  
-  Use similar approaches for consistency.
-  
-  Current step: ${stepDescription}
-  `;
-}
-```
-
-**Benefits:**
-- Commands consistent within session
-- Learns what selectors work on this app
-- Faster convergence
-
-### 5. **Confidence-Based Execution** 📊
-
-Add confidence scoring to guide execution:
-
-```typescript
-interface CommandWithConfidence {
-  code: string;
-  confidence: number; // 0-1
-  reasoning: string;
-}
-
-// Execution logic:
-if (command.confidence < 0.6) {
-  // Low confidence - use vision upfront instead of trying and failing
-  logger.log("Low confidence, requesting vision analysis before attempting");
-  const visionDiagnostics = await getVisionDiagnostics(...);
-  command = regenerateWithVisionInsights(visionDiagnostics);
-} else if (command.confidence < 0.8) {
-  // Medium confidence - try but have vision ready
-  logger.log("Medium confidence, will use vision if this fails");
-} else {
-  // High confidence - proceed normally
-  logger.log("High confidence, executing");
-}
-```
-
-**Benefits:**
-- Vision used proactively when LLM is uncertain
-- Fewer wasted attempts on guesses
-- More human-like (ask for help when unsure)
-
-### 6. **Simplified Goal Model** 🎯
-
-Remove sub-action pattern, use clearer model:
-
-```typescript
-// Current (complex):
-// Step → Sub-actions (1-5) → Attempts (0-3) → Goal checks after each sub-action
-// = nested loops, hard to reason about
-
-// Proposed (simple):
-// Step → Plan (upfront) → Execute commands → Check completion once
-// = linear flow, easy to understand
-
-async executeStep(step: Step, page: Page): Promise<StepResult> {
-  // 1. Plan the step (single LLM call)
-  const plan = await planStepExecution(step.description, getPageInfo(page));
-  
-  // 2. Execute each command in plan
-  for (const command of plan.commands) {
-    try {
-      await executeCommand(command, page);
-    } catch (error) {
-      // If any command fails, replan with error context
-      if (plan.overallConfidence > 0.8) {
-        // Was confident, try alternative selector
-        const alt = await regenerateCommand(command, error, page);
-        await executeCommand(alt, page);
-      } else {
-        // Was uncertain, use vision
-        const vision = await getVisionGuidedCommand(command, error, page);
-        await executeCommand(vision, page);
-      }
-    }
-  }
-  
-  // 3. Single goal check at end
-  return checkGoalCompletion(step.description, plan.commands, page);
-}
-```
-
-**Benefits:**
-- Predictable execution flow
-- Fewer LLM calls (plan once, not per sub-action)
-- Easier to debug and reason about
-- More human-like (plan → execute → verify)
-
-### 7. **Smart DOM Analysis** 🔍
-
-Better upfront DOM understanding:
-
-```typescript
-interface PageUnderstanding {
-  pageType: 'login' | 'dashboard' | 'form' | 'modal' | 'list' | 'unknown';
-  primaryInteractiveElements: Element[];
-  forms: FormInfo[];
-  modals: ModalInfo[];
-  navigation: NavigationInfo[];
-  confidence: number;
-}
-
-async analyzePageContext(page: Page): Promise<PageUnderstanding> {
-  const pageInfo = await getEnhancedPageInfo(page);
-  
-  // Detect page type and structure
-  const analysis = await llm.analyzePage(pageInfo);
-  
-  return {
-    pageType: 'login', // Detected
-    forms: [{ fields: ['username', 'password'], submitButton: 'Login to Continue' }],
-    confidence: 0.95
-  };
-}
-```
-
-**Usage:**
-```typescript
-const pageContext = await analyzePageContext(page);
-if (pageContext.pageType === 'login' && step.description.includes('login')) {
-  // High confidence - use login pattern
-  const form = pageContext.forms[0];
-  return generateLoginCommands(form, credentials);
-}
-```
-
-**Benefits:**
-- Understands page structure upfront
-- Can detect common patterns automatically
-- Informs better command generation
-- More deterministic (same page type → same approach)
-
-### 8. **Reduce Noise in Execution** 🔇
-
-**Current issues:**
-- Too many goal completion checks (after every sub-action)
-- Too many retry variations
-- Too much back-and-forth with LLM
-
-**Improvement:**
-```typescript
-// Instead of:
-// fill(username) → goal check → fill(password) → goal check → click(login) → goal check
-// = 6 operations (3 actions + 3 checks)
-
-// Do:
-// Plan: [fill(username), fill(password), click(login)]
-// Execute all → single goal check
-// = 4 operations (3 actions + 1 check)
-```
-
-**Benefits:**
-- 50% fewer LLM calls for multi-action steps
-- Faster execution
-- More predictable
-
-### 9. **Visual Scanning Before Action** 👁️
-
-Add optional visual scan for complex steps:
-
-```typescript
-async executeComplexStep(step: Step, page: Page): Promise<Result> {
-  // Check if step seems complex/ambiguous
-  const complexity = assessStepComplexity(step.description);
-  
-  if (complexity === 'high' && visualBudgetAvailable) {
-    // Proactive vision for complex steps
-    logger.log("Complex step detected - using vision upfront for better planning");
-    const screenshot = await page.screenshot();
-    const plan = await generatePlanWithVision(step, pageInfo, screenshot);
-    return executePlan(plan);
-  } else {
-    // Standard DOM-based approach
-    return executeStepStandard(step, page);
-  }
-}
-```
-
-**When to use:**
-- Ambiguous descriptions ("click the settings icon")
-- Multiple similar elements expected
-- Visual-heavy interactions
-- Previous similar steps failed
-
-**Benefits:**
-- Fewer wasted attempts
-- Higher first-attempt success rate
-- More human-like (look before you leap)
-
-### 10. **Session Memory** 💭
-
-Remember what works across steps:
-
-```typescript
-interface SessionMemory {
-  workingSelectors: Map<string, string>; // "login button" → "button[name='login']"
-  pagePatterns: string[]; // "This app uses data-testid"
-  commonElements: Element[]; // Persistent nav/header elements
-  successfulStrategies: string[]; // What approaches worked
-}
-
-// Example usage:
-if (sessionMemory.workingSelectors.has('login button')) {
-  // We found login button before, use same selector
-  const selector = sessionMemory.workingSelectors.get('login button');
-  return `await page.click('${selector}')`;
-}
-
-if (sessionMemory.pagePatterns.includes('uses-data-testid')) {
-  // This app uses data-testid, prefer that
-  return `await page.click('[data-testid="login"]')`;
-}
-```
-
-**Benefits:**
-- Consistency across steps
-- Learn from early steps
-- Deterministic within session
-- More human-like (remember what worked)
-
-## Recommended Implementation Priority
-
-### Phase 1: **Low-Hanging Fruit** (Quick Wins)
-
-1. ✅ **Reduce retry budget**
-   - MAX_SUBACTIONS_PER_STEP: 5 → 3 (humans try 2-3 approaches max)
-   - MAX_RETRIES_PER_STEP: 3 → 2 (3 attempts per sub-action → 2)
-   - MAX_FAILED_ATTEMPTS_PER_STEP: 12 → 8
-
-2. ✅ **Confidence scoring**
-   - Add confidence field to LLM responses
-   - Trigger vision earlier for low confidence (< 0.6)
-   - Log confidence for debugging
-
-3. ✅ **Session memory (simple version)**
-   - Track successful selectors in current session
-   - Reuse patterns that worked
-   - Pass to LLM as context
-
-### Phase 2: **Structural Improvements** (Medium Effort)
-
-4. **Upfront step planning**
-   - New API: `planStepExecution()`
-   - Generate all commands for step at once
-   - Execute plan without goal checks between commands
-   - Single goal check at end
-
-5. **Pattern detection**
-   - Detect common patterns (login, search, modal, dropdown)
-   - Use templates for high-confidence matches
-   - Fallback to LLM for novel cases
-
-6. **Smarter retry strategy**
-   - Replace sub-action approach with: primary → alternative → vision
-   - Clear escalation path
-   - Fail faster
-
-### Phase 3: **Advanced Features** (Future)
-
-7. **Proactive vision for complex steps**
-   - Complexity scoring
-   - Visual scan before attempting
-   - Better first-attempt success
-
-8. **Cross-session learning**
-   - Persist patterns across sessions
-   - Build app-specific knowledge base
-   - Improve over time
-
-9. **Multi-step lookahead**
-   - Plan multiple steps at once when related
-   - Optimize for workflow efficiency
-   - Batch similar operations
-
-## Immediate Actionable Changes
-
-### Change 1: Reduce Retry Budget (More Human-Like)
-
-```typescript
-// In scenario-worker-class.ts
-const MAX_RETRIES_PER_STEP = 2; // Was 3 → Now 2 attempts per approach (total 3 tries)
-const MAX_SUBACTIONS_PER_STEP = 3; // Was 5 → Now 3 different approaches max  
-const MAX_FAILED_ATTEMPTS_PER_STEP = 8; // Was 12 → Now 8 total failures max
-```
-
-**Rationale:** Humans try 2-3 things, not 12. Fail faster = faster feedback.
-
-### Change 2: Add Confidence Scoring
-
-**Modify LLM response:**
-```typescript
-interface LLMPlaywrightCommandResponse {
-  command: string;
-  reasoning: string;
-  confidence: number; // NEW: 0-1 scale
-  uncertaintyReason?: string; // NEW: Why low confidence
-}
-```
-
-**Prompt addition:**
-```
-Additionally, provide a confidence score (0-1):
-- 1.0: Element clearly identified in DOM with unique selector
-- 0.8: Element found but selector might not be unique  
-- 0.6: Multiple possible elements, best guess
-- 0.4: Element not clearly in DOM, inferring from context
-- 0.2: Guessing, likely to fail
-
-If confidence < 0.7, explain uncertainty.
-```
-
-**Usage:**
-```typescript
-const result = await generatePlaywrightCommand(...);
-if (result.confidence < 0.6 && !usedVisionMode) {
-  logger.log(`⚠️ Low confidence (${result.confidence}) - using vision proactively`);
-  // Skip attempts, go straight to vision
-  const visionResult = await getVisionGuidedCommand(...);
-  return visionResult;
-}
-```
-
-### Change 3: Session Memory (Simple Version)
-
-```typescript
-class SessionContext {
-  successfulSelectors: Map<string, string> = new Map();
-  appPatterns: Set<string> = new Set(); // 'uses-data-testid', 'uses-aria-labels', etc.
-  
-  recordSuccess(elementDescription: string, selector: string) {
-    this.successfulSelectors.set(elementDescription, selector);
-  }
-  
-  getContext(): string {
-    if (this.successfulSelectors.size === 0) return '';
-    
-    return `
-    Selectors that worked in this session:
-    ${Array.from(this.successfulSelectors.entries())
-      .map(([desc, sel]) => `- "${desc}": ${sel}`)
-      .join('\n')}
-    
-    Try similar patterns for consistency.
-    `;
-  }
-}
-```
-
-**Add to command generation prompt:**
-```typescript
-${sessionContext.getContext()}
-```
-
-### Change 4: Skip Sub-Action Loop for High Confidence
-
-```typescript
-// After planning, if all commands have high confidence (>0.8):
-if (plan.overallConfidence > 0.8) {
-  // Execute entire plan without intermediate checks
-  for (const cmd of plan.commands) {
-    await execute(cmd);
-  }
-  // Single goal check at end
-  return checkGoalCompletion(step, plan.commands, page);
-} else {
-  // Low confidence - use current sub-action approach for safety
-  return executeWithSubActions(step, page);
-}
-```
-
-## Expected Improvements
-
-### Metrics:
-
-| Metric | Current | After Phase 1 | After Phase 2 |
-|--------|---------|---------------|---------------|
-| Avg LLM calls per step | 4-6 | 3-4 | 1-2 |
-| Max attempts per step | 20 | 8 | 6 |
-| Success rate (1st attempt) | ~40% | ~50% | ~70% |
-| Time per step | 30-60s | 20-40s | 10-20s |
-| Determinism | Low | Medium | High |
-
-### Behavioral Changes:
-
-**Login scenario:**
-
-**Current:**
-```
-1. fill(user) → check goal → incomplete
-2. fill(pass) → check goal → incomplete  
-3. click(login) → check goal → complete
-= 3 actions + 3 goal checks = 6 operations
-```
-
-**After Phase 1 (confidence + memory):**
-```
-1. fill(user) → check goal → incomplete (but with confidence: 0.9)
-2. fill(pass) → check goal → incomplete (confidence: 0.9)
-3. click(login) → check goal → complete (confidence: 0.85)
-= 3 actions + 3 checks, but with confidence signals
-```
-
-**After Phase 2 (planning):**
-```
-1. Plan: [fill(user), fill(pass), click(login)] (confidence: 0.9)
-2. Execute all 3 commands
-3. Single goal check → complete
-= 1 plan + 3 actions + 1 check = 5 operations
-```
-
-## Summary
-
-**Core Philosophy Shift:**
-
-**From:** Reactive trial-and-error with many retries  
-**To:** Proactive planning with fewer, smarter attempts
-
-**Most Important:**
-1. Plan before acting (not generate-execute-check loop)
-2. Use confidence to guide strategy
-3. Remember what works
-4. Fail faster (humans don't retry 12 times)
-5. Recognize common patterns
-
-**Most Human-Like = Most Deterministic**
-- Humans plan, then execute
-- Humans learn from experience  
-- Humans don't endlessly retry
-- Humans recognize familiar patterns
-- Humans ask for help when uncertain
-
-These improvements make the system more deterministic, faster, cheaper, and more aligned with how humans actually interact with web applications.
-