testchimp-runner-core 0.0.21 → 0.0.23

package/plandocs/PLANNING_SESSION_SUMMARY.md

@@ -0,0 +1,372 @@
+# Planning Session Summary: Orchestrator Agent Architecture
+
+## Date: October 11, 2025
+
+---
+
+## Final Decisions Made
+
+### 1. ✅ Self-Reflection in MVP
+**Decision**: Include free-form self-reflection with agent-driven loop detection
+- Agent outputs `guidanceForNext` (free-form text) for train of thought continuity
+- Agent signals `detectingLoop: true` when it notices repetition
+- Agent decides when to break own loop, system enforces hard limits as backup
+
+**Rationale**: Valuable for maintaining context across iterations, agent self-corrects
+
+### 2. ✅ No Screenshot Budget
+**Decision**: Screenshots available freely, no artificial limits
+- Corrected token cost: 1-2K tokens (NOT 100K!)
+- For 1920x1080 viewport: ~1,452 tokens (gpt-4.1-mini)
+- Comparable to extra DOM context
+
+**Rationale**: Very affordable, enables liberal vision use throughout journey
+
+### 3. ✅ DOM Limits (Increased for Complex Pages)
+**Decision**: Increased limits in getEnhancedPageInfo to handle complex UIs
+- ARIA tree depth: 4 levels
+- Interactive elements: top 50 (was 12)
+- IDs: top 50 (was 10)
+- Data attributes: top 50 (was 10)
+- Form fields: top 20 (was 8)
+- Page structure: top 10 (was 6)
+- General elements: top 50 (was 15)
+- Text: 30 chars max
+- Result: ~800-1,500 tokens
+
+**Rationale**: Complex pages need more context, still compact with truncation
+
+### 4. ✅ Token Usage Tracking
+**Decision**: Track and report all LLM token usage via callback
+- Interface: `onTokensUsed({inputTokens, outputTokens, includesImage})`
+- Heuristic: 4 characters = 1 token
+- Image tokens: ~1,500 estimate for viewport screenshots
+- Reported via ProgressReporter for analytics
+
+**Rationale**: Cost tracking, optimization, analytics
+
+### 5. ✅ Recovery Tools in MVP
+**Decision**: Include 3 recovery tools for self-unsticking
+- `navigate_back()` - Go back in history
+- `refresh_page()` - Reload page
+- `navigate_to_url({url})` - Navigate to specific URL (with domain validation)
+
+**Rationale**: Agent needs ability to recover from bad states (wrong navigation, stuck page, side effects)
+
+### 6. ✅ Inquisitive Exploration in Phase 2
+**Decision**: Defer exploratory actions to Phase 2, MVP uses workarounds
+- Phase 2 tool: `explore_element({action, selector, purpose})`
+- Actions: hover, click_info, click_menu, focus
+- Safety: State validation, non-consequential only
+- **Screenshot handling**: Immediate analysis via sub-agent call
+  - System takes screenshot after exploration
+  - Calls agent to analyze screenshot
+  - Agent extracts learnings (text)
+  - Only TEXT stored in history, NOT screenshot
+  - Keeps memory lightweight
+- MVP workaround: Use screenshot + DOM analysis + retry
+
+**Rationale**: Safety concerns, complexity, needs battle-testing first
+
+### 7. ✅ Always-Provided Context Structure
+**Decision**: Provide comprehensive context automatically each iteration
+- Overall goal + current goal
+- Current page info (DOM)
+- Recent 6-7 steps
+- Experiences (learnings)
+- Extracted data
+- Self-reflection from previous iteration
+- Journey progress tracking
+
+**Rationale**: Agent needs full situational awareness without repeated tool calls
+
+### 8. ✅ System vs Agent Guardrails
+**Decision**: Clear separation of responsibilities
+- **System enforces**: Iteration limits, tool call limits, command limits
+- **Agent signals**: Stuck, infeasible, detecting loop
+- System has final say, agent provides soft guidance
+
+**Rationale**: Safety (hard limits) + intelligence (agent self-awareness)
+
+---
+
+## Architecture Summary
+
+### Core Components
+
+**1. Always-Provided Context** (auto-fetched each iteration)
+```typescript
+{
+  overallGoal, currentStepGoal, stepNumber, totalSteps,
+  currentPageInfo, currentURL,
+  recentSteps (6-7), experiences, extractedData,
+  previousIterationGuidance
+}
+```
+
+**2. Tools** (8 in MVP)
+- **Information**: take_screenshot, recall_history, inspect_page, check_page_ready
+- **Data**: extract_data
+- **Recovery**: navigate_back, refresh_page, navigate_to_url
+
+**3. Agent Decision Output**
+```typescript
+{
+  toolCalls, toolReasoning, needsToolResults,
+  commands, commandReasoning,
+  selfReflection: {guidanceForNext, detectingLoop, loopReasoning},
+  experiences, memoryUpdate,
+  status: 'complete' | 'stuck' | 'infeasible' | 'continue',
+  statusReasoning, reasoning
+}
+```
+
+**4. Sequential Batch Execution**
+- Agent plans batch of commands (max 3-5)
+- System executes one-by-one
+- Stop at first failure
+- Record each individually in history
+
+**5. Comprehensive Logging**
+- Every iteration: goal, reasoning, self-reflection, tools, commands, experiences, status
+- All thoughts visible for debugging
+- Exported via ProgressReporter
+
+**6. Token Usage Tracking**
+- Input + output tokens calculated (4 chars = 1 token)
+- Image tokens estimated (~1,500 for viewport)
+- Reported via `onTokensUsed()` callback
+
+---
+
+## MVP vs Phase 2
+
+### MVP Includes:
+- ✅ 8 core tools (info + data + recovery)
+- ✅ Journey memory with experiences
+- ✅ Self-reflection + loop detection
+- ✅ Batch command planning
+- ✅ Self-recovery (navigate back/refresh)
+- ✅ Token tracking
+- ✅ Comprehensive logging
+- ✅ Configurable guardrails
+
+### Phase 2 Adds:
+- Inquisitive exploration (explore_element)
+- Advanced optimizations (caching, adaptive limits)
+- Memory summarization for long journeys
+
+---
+
+## Key Metrics
+
+### Token Usage Per Iteration (Estimated)
+```
+System prompt:              500 tokens
+Always-provided context:    1,200-2,000 tokens
+  - Goals & progress:       100
+  - DOM (increased limits): 800-1,500
+  - Recent 6-7 steps:       300-500
+  - Experiences:            100-200
+Self-reflection:            100 tokens
+Tool results (optional):    300-500 tokens
+Screenshot (optional):      1,500 tokens
+
+Total without screenshot:   2,400-3,600 tokens
+Total with screenshot:      3,900-5,100 tokens
+```
+
+### Expected Performance
+- LLM calls/step: 2-4 (vs 4-6 current)
+- Iterations/step: 3-5 (vs 8-12 current)
+- Tool calls/step: 1-3
+- Commands/iteration: 2-3 (batched)
+- Agent learns: 1-2 experiences per step
+
+---
+
+## Inquisitive Exploration Design (Phase 2)
+
+### Problem
+Menu items are icon-only, no text/ARIA labels → Agent unsure which to click
+
+### Solution
+Agent investigates non-consequentially, analyzes immediately:
+
+```
+Iteration N:
+  Agent Decision: "Need to hover over icons to see tooltips"
+  Tool: explore_element({action: "hover", selector: "nav button:nth-child(2)"})
+  
+  System:
+    → Hover, wait 500ms
+    → Take screenshot
+    → Call agent (sub-call): "What do you see in this screenshot?"
+  
+  Agent Analysis (sub-call): 
+    → Sees tooltip
+    → Returns: "Tooltip shows 'Dashboard' - this is the Dashboard button"
+  
+  System:
+    → Stores TEXT in history: "Explored button, tooltip confirms Dashboard"
+    → Does NOT store screenshot
+    → Returns to main agent: {success: true, learning: "Tooltip shows Dashboard"}
+  
+  Agent Decision (continues): 
+    → "Great, confirmed it's Dashboard"
+    → Commands: ["page.click('nav button:nth-child(2)')"]
+  
+  System: Execute commands
+```
+
+**Key difference**: Screenshot analyzed WITHIN same iteration, only text stored
+
+### Allowed Actions
+- ✅ hover (show tooltips)
+- ✅ click_info (info icons)
+- ✅ click_menu (expand menus)
+- ✅ focus (see input hints)
+
+### NOT Allowed
+- ❌ Submit forms
+- ❌ Delete/remove
+- ❌ Logout
+- ❌ File uploads
+
+### Safety
+- State validation (URL, modal count)
+- Revert if unexpected navigation
+- Budget: 10 explorations per step
+- Timeout: 2s per exploration
+
+### Why Phase 2
+- Safety risk (needs robust validation)
+- Complexity (screenshot handling, state comparison)
+- MVP workaround: screenshot + DOM + retry
+
+---
+
+## Implementation Status
+
+### Completed (During Planning):
+- ✅ Token usage tracking added to interfaces
+- ✅ BackendProxyLLMProvider calculates token usage
+- ✅ LLMFacade prepared for token callback
+- ✅ Progress reporter extended with `onTokensUsed()`
+
+### Ready to Implement:
+1. OrchestratorAgent class
+2. ToolRegistry with 8 tools
+3. Journey memory implementation
+4. Always-provided context builder
+5. Self-reflection structures
+6. Recovery tools (navigate_back, refresh, navigate_to)
+7. Comprehensive logging
+8. Token tracking integration
+
+### Phase 2:
+1. Exploratory actions (explore_element)
+2. State validation logic
+3. Advanced optimizations
+
+---
+
+## Documentation Created
+
+1. **MULTI_AGENT_ARCHITECTURE_REVIEW.md**
+   - 8 pitfalls analyzed with mitigations
+   - Phased implementation strategy
+   - Risk analysis and trade-offs
+
+2. **ORCHESTRATOR_IMPLEMENTATION_PLAN.md**
+   - Detailed implementation specs
+   - Code examples
+   - Integration points
+
+3. **ORCHESTRATOR_MVP_SUMMARY.md**
+   - Executive summary
+   - Complete feature list
+   - Inquisitive exploration section
+   - MVP vs Phase 2 breakdown
+
+4. **PLANNING_SESSION_SUMMARY.md** (this document)
+   - All decisions made
+   - Rationales
+   - Implementation status
+
+---
+
+## Next Steps
+
+### When Ready to Implement:
+1. Review all 3 architecture documents
+2. Start with MVP (8 tools, no exploration)
+3. Implement in order:
+   - Tool registry + tool implementations
+   - Journey memory structures
+   - OrchestratorAgent loop
+   - Integration with ScenarioWorker
+   - Token tracking integration
+   - Comprehensive logging
+4. Test with real scenarios
+5. Measure metrics vs current approach
+6. Iterate based on findings
+7. Add Phase 2 features when validated
+
+---
+
+## Success Criteria (MVP)
+
+### Must Have:
+- [ ] Fewer iterations than current (target: 50% reduction)
+- [ ] Backward compatible (VS Extension & GitHub Runner work)
+- [ ] No infinite loops (guardrails work)
+- [ ] Memory doesn't bloat
+- [ ] Tool extensibility works
+- [ ] Token usage tracked accurately
+
+### Nice to Have:
+- [ ] Fewer LLM calls than current
+- [ ] Better success rate
+- [ ] Faster execution
+
+### Acceptable Trade-offs:
+- ⚠️ Slightly higher token usage per iteration (richer context)
+- ⚠️ Some tool call overhead
+- ⚠️ No exploratory actions in MVP
+
+---
+
+## Estimated Timeline
+
+**MVP Implementation**: 2-3 weeks
+- Week 1: Foundation (types, tool registry, tool implementations)
+- Week 2: Orchestrator loop, integration
+- Week 3: Testing, refinement, metrics
+
+**Phase 2 (Exploration)**: 1-2 weeks after MVP validated
+
+**Total**: 3-5 weeks for complete solution
+
+---
+
+## Final Architecture Confidence
+
+**✅ Ready to implement** with:
+- All major decisions finalized
+- Trade-offs understood and accepted
+- Risks identified with mitigations
+- Phased approach reduces implementation risk
+- Backward compatibility ensured
+- Comprehensive documentation complete
+
+**Key strengths**:
+- Human-like (memory, learning, reflection, recovery)
+- Extensible (tool registry, dynamic prompts)
+- Safe (system guardrails, agent self-awareness)
+- Transparent (comprehensive logging)
+- Cost-aware (token tracking)
+- Practical (recovery tools, self-unstuck)
+
+**No blockers to proceed.**
+

package/plandocs/SCRIPT_CLEANUP_FEATURE.md

@@ -0,0 +1,201 @@
+# Script Cleanup Feature
+
+## Summary
+Added a final cleanup step in the script generation pipeline that uses an LLM to make minor adjustments to the generated test script, removing redundancies and improving code quality without changing the core logic.
+
+## Purpose
+After the orchestrator generates test steps, there may be minor redundancies or formatting issues:
+- Duplicate expect() assertions
+- Redundant waits or checks
+- Inconsistent formatting
+- Orphaned step comments without code
+
+The cleanup step acts as a final sanity check to polish the generated script while preserving its core functionality.
+
+## Implementation
+
+### 1. New Prompt (`prompts.ts`)
+
+**SCRIPT_CLEANUP** prompt with clear guidelines:
+- **DO:** Remove duplicates, fix formatting, consolidate identical assertions
+- **DO NOT:** Change test logic, remove legitimate assertions, restructure code, change selectors, add new functionality
+
+**Examples in prompt:**
+```typescript
+// ❌ REMOVE redundancy:
+await expect(page.getByText('Hello')).toBeVisible();
+await expect(page.getByText('Hello')).toBeVisible();  // duplicate
+
+// ✅ KEEP legitimate checks:
+await expect(page.getByPlaceholder('Message...')).toBeEmpty();
+await page.getByPlaceholder('Message...').fill('Hello');
+await expect(page.getByPlaceholder('Message...')).toHaveValue('Hello');  // different checks
+```
+
+### 2. New Method in LLMFacade (`llm-facade.ts`)
+
+```typescript
+async cleanupScript(script: string, model?: string): Promise<{
+  script: string;
+  changes: string[];
+  skipped?: string;
+}>
+```
+
+**Behavior:**
+- Calls LLM with SCRIPT_CLEANUP prompt
+- Parses JSON response with cleaned script and list of changes
+- Returns original script on error (safe fallback)
+- Logs all changes made for transparency
+
+**Error Handling:**
+- Invalid JSON → return original script
+- Missing fields → return original script  
+- LLM error → return original script
+- Never fails the generation process
+
+### 3. Integration into Scenario Worker (`scenario-worker-class.ts`)
+
+Added cleanup step immediately after `generateTestScript()`:
+
+```typescript
+// Generate clean script with TestChimp comment and code
+generatedScript = generateTestScript(testName, steps, undefined, hashtags);
+
+// Perform final cleanup pass to remove redundancies and make minor adjustments
+this.log(`[ScenarioWorker] Performing final script cleanup...`);
+try {
+  const cleanupResult = await this.llmFacade.cleanupScript(generatedScript, job.model);
+  
+  if (cleanupResult.changes && cleanupResult.changes.length > 0) {
+    this.log(`[ScenarioWorker] Cleanup made ${cleanupResult.changes.length} improvement(s):`);
+    cleanupResult.changes.forEach((change, i) => {
+      this.log(`[ScenarioWorker]   ${i + 1}. ${change}`);
+    });
+    generatedScript = cleanupResult.script;
+  } else if (cleanupResult.skipped) {
+    this.log(`[ScenarioWorker] Cleanup skipped: ${cleanupResult.skipped}`);
+  } else {
+    this.log(`[ScenarioWorker] Cleanup completed - no changes needed`);
+  }
+} catch (error: any) {
+  this.log(`[ScenarioWorker] Cleanup failed, using original script: ${error.message}`);
+  // Continue with original script on error
+}
+```
+
+## What Gets Cleaned Up
+
+### ✅ Redundancies Removed
+1. **Duplicate assertions:**
+   ```typescript
+   // Before cleanup
+   await expect(page.getByText('Hello')).toBeVisible();
+   await expect(page.getByText('Hello')).toBeVisible();
+   
+   // After cleanup
+   await expect(page.getByText('Hello')).toBeVisible();
+   ```
+
+2. **Redundant URL checks:**
+   ```typescript
+   // Before cleanup
+   await expect(page).toHaveURL(/\/messages/);
+   await expect(page).toHaveURL(/\/messages/);
+   
+   // After cleanup
+   await expect(page).toHaveURL(/\/messages/);
+   ```
+
+3. **Duplicate comments without code** (already handled by script generation, but this is a safety net)
+
+### ✅ Minor Formatting Fixes
+- Inconsistent spacing
+- Alignment issues
+- Obvious formatting problems
+
+### ❌ Preserved (Not Changed)
+- Test logic and flow
+- Legitimate assertions (same locator, different expectations)
+- Important waits
+- Selectors
+- Test structure
+- Any functionality
+
+## Safety Features
+
+### 1. Conservative Approach
+- Only makes changes when confident they're safe
+- Prompt explicitly warns against major changes
+- Focuses on "obvious" redundancies only
+
+### 2. Transparency
+- Logs all changes made with descriptions
+- Makes it easy to see what was modified
+- Helps debug if cleanup causes issues
+
+### 3. Graceful Degradation
+- Any error → return original script
+- Invalid response → return original script
+- Never breaks the generation pipeline
+- Cleanup is an enhancement, not a requirement
+
+### 4. Idempotent
+- Running cleanup twice should produce the same result
+- No cumulative changes or drift
+
+## Example Output
+
+**Console logs:**
+```
+[ScenarioWorker] Performing final script cleanup...
+[LLMFacade] Script cleanup completed. Changes: 2
+[LLMFacade]   1. Removed duplicate expect assertion for message visibility
+[LLMFacade]   2. Consolidated redundant URL checks into single assertion
+[ScenarioWorker] Cleanup made 2 improvement(s):
+[ScenarioWorker]   1. Removed duplicate expect assertion for message visibility
+[ScenarioWorker]   2. Consolidated redundant URL checks into single assertion
+```
+
+## Benefits
+
+1. **Cleaner Scripts:** Removes redundancies that can make tests harder to read
+2. **Reduced Token Usage:** Shorter scripts mean less tokens consumed by users
+3. **Better Maintainability:** Clean code is easier to understand and modify
+4. **Safety Net:** Catches issues that might slip through orchestrator logic
+5. **Zero Risk:** Fallback to original script on any error
+
+## Performance Impact
+
+- **Time:** Adds one LLM call at the end (~1-3 seconds)
+- **Cost:** One additional LLM call per script generation
+- **Benefit:** Catches redundancies that would otherwise be in production tests
+
+The small overhead is worthwhile for the quality improvement.
+
+## Future Enhancements
+
+Possible improvements:
+1. **Configurable:** Allow users to disable cleanup if they prefer
+2. **More Rules:** Add more specific cleanup patterns
+3. **Static Analysis:** Use AST parsing instead of LLM for some checks (faster, cheaper)
+4. **Metrics:** Track how often cleanup makes changes vs. no-ops
+
+## Files Modified
+
+1. `/src/prompts.ts` - Added SCRIPT_CLEANUP prompt
+2. `/src/llm-facade.ts` - Added cleanupScript() method
+3. `/src/scenario-worker-class.ts` - Integrated cleanup into generation pipeline
+
+## Testing
+
+The feature is safe to deploy because:
+- Falls back to original script on any error
+- Doesn't break existing functionality
+- Only makes conservative changes
+- Logs all modifications for review
+
+## Conclusion
+
+The script cleanup feature adds a lightweight final polish step to the generation pipeline, removing redundancies and improving code quality without risk to the core test logic. It's a safety net that catches issues the orchestrator might miss while maintaining backward compatibility and graceful error handling.
+