@exaudeus/workrail 3.27.0 → 3.29.0

package/docs/reference/loop-optimization.md

@@ -0,0 +1,209 @@
+# Loop Context Optimization
+
+## Overview
+
+WorkRail v0.2.0 introduces significant optimizations for loop execution, reducing context size by 60-80% during loop iterations. This feature implements a progressive disclosure pattern where the first iteration receives full context and subsequent iterations receive minimal, optimized context.
+
+## Key Features
+
+### 1. Progressive Context Disclosure
+
+- **First Iteration**: Receives complete loop overview including:
+  - Full phase information
+  - All loop metadata
+  - Function definitions
+  - Complete array data for forEach loops
+  
+- **Subsequent Iterations**: Receive minimal context with:
+  - Current iteration number
+  - Current item only (for forEach loops)
+  - Reference to phase overview from first iteration
+  - Stripped metadata and large arrays
+
+### 2. Empty Loop Detection
+
+Loops with no items to process are automatically skipped, avoiding unnecessary phase overview generation:
+
+```typescript
+// These loops will be skipped entirely:
+{ type: 'forEach', items: 'emptyArray' }  // where emptyArray = []
+{ type: 'for', count: 0 }
+{ type: 'while', condition: { var: 'flag', equals: false } }  // where flag = false
+```
+
+### 3. Function DSL Support
+
+The schema now natively supports function definitions at multiple scopes:
+
+```json
+{
+  "functionDefinitions": [
+    {
+      "name": "validateItem",
+      "definition": "Validates item format and returns boolean",
+      "scope": "workflow"
+    }
+  ],
+  "steps": [{
+    "id": "process-loop",
+    "type": "loop",
+    "functionDefinitions": [
+      {
+        "name": "processSpecific",
+        "definition": "Process items in this loop context",
+        "scope": "loop"
+      }
+    ],
+    "body": {
+      "id": "use-functions",
+      "prompt": "Apply validation",
+      "functionReferences": ["validateItem()", "processSpecific()"]
+    }
+  }]
+}
+```
+
+## Implementation Details
+
+### Architecture
+
+The optimization is implemented through:
+
+1. **LoopContextOptimizer Service**: Instance-based service following DI pattern
+2. **Enhanced LoopExecutionContext**: Supports minimal context generation
+3. **Modified WorkflowService**: Integrates optimization into main flow
+
+### Context Size Reduction
+
+Example context transformation:
+
+**First Iteration (Full Context)**:
+```json
+{
+  "dataItems": [/* 1000 items */],
+  "currentDataItem": { "id": "item-0" },
+  "itemIndex": 0,
+  "_currentLoop": {
+    "loopId": "process-loop",
+    "loopStep": { /* full step definition */ }
+  }
+}
+```
+
+**Second Iteration (Optimized Context)**:
+```json
+{
+  "currentDataItem": { "id": "item-1" },
+  "itemIndex": 1,
+  "_currentLoop": {
+    "loopId": "process-loop",
+    "loopType": "forEach",
+    "iteration": 1,
+    "phaseReference": {
+      "loopId": "process-loop",
+      "phaseTitle": "Process Items Loop",
+      "totalSteps": 2
+    }
+  }
+}
+```
+
+## Usage Guidelines
+
+### For Workflow Authors
+
+1. **Define Function DSL** at appropriate scopes:
+   - Workflow level: Shared across all steps
+   - Loop level: Available within loop body
+   - Step level: Specific to that step
+
+2. **Use Function References** in step prompts:
+   ```json
+   {
+     "prompt": "Process item using validateItem() and formatOutput()",
+     "functionReferences": ["validateItem()", "formatOutput()"]
+   }
+   ```
+
+3. **Consider Loop Size** when designing workflows:
+   - Large arrays benefit most from optimization
+   - Nested loops see compounded benefits
+
+### For Agents
+
+When processing optimized loops:
+
+1. **First Iteration**: Store phase overview information for reference
+2. **Subsequent Iterations**: Refer to stored phase overview when needed
+3. **Function References**: Look up definitions in appropriate scope
+4. **Minimal Context**: Work with just the current item, not full arrays
+
+## Performance Characteristics
+
+Based on benchmarks:
+
+- **Context Size Reduction**: 60-80% for large arrays
+- **Processing Overhead**: < 5% additional time
+- **Memory Usage**: Significantly reduced for long-running loops
+- **Network Transfer**: Reduced payload size for remote execution
+
+## Migration Guide
+
+### From v0.1.x to v0.2.0
+
+1. **Schema Version**: Update to v0.2.0
+2. **Add maxIterations**: Required field for all loops
+3. **Optional**: Add function definitions to reduce duplication
+4. **No Code Changes**: Optimization is automatic
+
+Example migration:
+
+```diff
+{
+-  "version": "0.1.0",
++  "version": "0.2.0",
+   "steps": [{
+     "type": "loop",
+     "loop": {
+       "type": "forEach",
+-      "items": "data"
++      "items": "data",
++      "maxIterations": 1000
+     }
+   }]
+}
+```
+
+## Best Practices
+
+1. **Use Function DSL** for repeated patterns within loops
+2. **Set Appropriate maxIterations** to prevent runaway loops
+3. **Use Empty Loop Detection** by checking conditions early
+4. **Design for Progressive Disclosure** in your prompts
+
+## Troubleshooting
+
+### Large Context Errors
+
+If you still encounter context size errors:
+
+1. Check for accumulating data in loop state
+2. Ensure large arrays aren't being duplicated
+3. Consider breaking very large loops into batches
+
+### Function Reference Issues
+
+If function references aren't working:
+
+1. Verify function is defined at correct scope
+2. Check reference syntax includes parentheses: `functionName()`
+3. Ensure agent supports function DSL
+
+## Future Enhancements
+
+Planned improvements include:
+
+1. **Loop Resumption**: Ability to resume execution mid-loop
+2. **Client-Side Caching**: Optional caching of phase references
+3. **Dynamic Optimization**: Adaptive thresholds based on context size
+4. **Streaming Support**: Progressive context streaming for very large loops

package/docs/reference/loop-validation.md

@@ -0,0 +1,176 @@
+# Loop Validation Best Practices
+
+This document describes the enhanced loop validation features in WorkRail and provides best practices for creating maintainable loop workflows.
+
+## Overview
+
+WorkRail includes enhanced validation for loop steps that goes beyond basic syntax checking. It helps you identify potential issues with complex conditional logic, excessive prompt lengths, undefined variables, and provides guidance on best practices.
+
+## Validation Categories
+
+### 1. Conditional Logic Complexity
+
+**What it checks:**
+- Complex ternary operators (2 or more `?` operators)
+- Nested ternary operators
+- Multi-path conditionals in prompts
+
+**Why it matters:**
+- Complex conditionals are hard to read and maintain
+- Template variables may not resolve properly
+- All conditional branches may be included in the output
+
+**Best practice:**
+Instead of inline conditionals:
+```json
+{
+  "prompt": "{{iteration === 1 ? 'First step' : iteration === 2 ? 'Second step' : 'Third step'}}"
+}
+```
+
+Use separate steps with `runCondition`:
+```json
+[
+  {
+    "id": "step-1",
+    "prompt": "First step",
+    "runCondition": {"var": "iteration", "equals": 1}
+  },
+  {
+    "id": "step-2", 
+    "prompt": "Second step",
+    "runCondition": {"var": "iteration", "equals": 2}
+  },
+  {
+    "id": "step-3",
+    "prompt": "Third step", 
+    "runCondition": {"var": "iteration", "equals": 3}
+  }
+]
+```
+
+### 2. Prompt Length Validation
+
+**What it checks:**
+- Raw prompt length approaching 1500-2000 character limits
+- Total conditional content size when expanded
+
+**Why it matters:**
+- Long prompts can exceed system limits
+- Conditional expansion can make prompts even larger
+- Large prompts reduce readability and maintainability
+
+**Best practice:**
+- Keep prompts focused and under 1000 characters
+- Move detailed instructions to the `guidance` array
+- Split complex prompts into multiple steps
+
+### 3. Template Variable Usage
+
+**What it checks:**
+- Undefined variables in prompts, titles, and agent roles
+- Proper use of loop iteration variables
+
+**Available loop variables:**
+- For all loops: `iteration` (or custom `iterationVar`)
+- For `forEach` loops: `item` (or custom `itemVar`), `index` (or custom `indexVar`)
+- Context variables defined elsewhere in the workflow
+
+**Best practice:**
+- Always verify variables are defined before use
+- Use descriptive custom variable names
+- Document expected context variables
+
+### 4. Loop Structure
+
+**What it checks:**
+- Reasonable `maxIterations` limits
+- Appropriate loop types for use cases
+- Common loop patterns
+
+**Best practice:**
+- Set `maxIterations` to prevent infinite loops (typically < 100)
+- Use `for` loops for fixed iterations
+- Use `while`/`until` for condition-based loops
+- Use `forEach` for processing arrays
+
+## Common Patterns
+
+### Progressive Analysis Pattern
+
+Detected when loops perform step-by-step analysis (e.g., Structure → Modules → Dependencies → Patterns).
+
+**Recommendation:** Use separate steps with `runCondition` for clarity.
+
+### Multi-Path Pattern
+
+Detected when loops have multiple conditional execution paths.
+
+**Recommendation:** Refactor to separate steps for better maintainability.
+
+## Example: Well-Structured Loop
+
+```json
+{
+  "id": "analysis-loop",
+  "type": "loop",
+  "loop": {
+    "type": "for",
+    "count": 4,
+    "maxIterations": 4,
+    "iterationVar": "analysisStep"
+  },
+  "body": [
+    {
+      "id": "step-structure",
+      "title": "Structural Analysis",
+      "prompt": "Analyze the codebase structure...",
+      "runCondition": {"var": "analysisStep", "equals": 1}
+    },
+    {
+      "id": "step-modules",
+      "title": "Module Analysis",
+      "prompt": "Analyze task-relevant modules...",
+      "runCondition": {"var": "analysisStep", "equals": 2}
+    },
+    {
+      "id": "step-dependencies",
+      "title": "Dependency Analysis",
+      "prompt": "Trace dependencies and flows...",
+      "runCondition": {"var": "analysisStep", "equals": 3}
+    },
+    {
+      "id": "step-patterns",
+      "title": "Pattern Discovery",
+      "prompt": "Identify established patterns...",
+      "runCondition": {"var": "analysisStep", "equals": 4}
+    }
+  ]
+}
+```
+
+## CLI Validation
+
+Run validation to see warnings and suggestions:
+
+```bash
+workrail validate your-workflow.json
+```
+
+The validator will show:
+-  **Errors**: Must be fixed for the workflow to run
+-  **Warnings**: Should be addressed for better maintainability
+- ℹ️ **Information**: Detected patterns and insights
+-  **Suggestions**: Recommendations for improvement
+
+## Summary
+
+Good loop design principles:
+1. Keep logic simple and explicit
+2. Use `runCondition` for multi-path flows
+3. Keep prompts concise and focused
+4. Define and document all variables
+5. Set reasonable iteration limits
+6. Test thoroughly with validation
+
+Following these practices will result in workflows that are easier to understand, maintain, and debug.