tasker-engine 1.0.0

data/docs/TASK_DIAGRAM.md

@@ -0,0 +1,252 @@
+# Tasker Task Diagrams
+
+The `Tasker::TaskDiagram` class provides a way to visualize task workflows using Mermaid diagrams. This makes it easy to understand the current state of a task's workflow and the relationships between its steps.
+
+## Features
+
+Tasker::TaskDiagram includes:
+
+- A native diagram implementation (no external gems required)
+- Automatic color-coding of workflow steps based on their status
+- Direct generation of Mermaid syntax for flowcharts
+- Support for embedding diagrams in web pages
+- Complete HTML document generation with the diagram embedded
+
+## Basic Usage
+
+```ruby
+# Create a diagram for a task
+task = Tasker::Task.find(task_id)
+diagram = Tasker::TaskDiagram.new(task)
+
+# Generate Mermaid syntax
+mermaid_string = diagram.to_mermaid
+
+# Generate a complete HTML document with the embedded diagram
+html_document = diagram.to_html
+
+# Convert diagram to JSON format
+json_data = diagram.to_json
+```
+
+## Adding Links to REST Endpoints
+
+You can add clickable links to each workflow step by providing a base URL when creating the diagram:
+
+```ruby
+# Create a diagram with links to REST endpoints
+base_url = "https://example.com/api"
+diagram = Tasker::TaskDiagram.new(task, base_url)
+
+# Each step in the diagram will link to {base_url}/workflow_steps/{workflow_step_id}
+```
+
+## Diagram Rendering Options
+
+The TaskDiagram class provides two main methods for visualization:
+
+### to_mermaid
+
+This method returns a string containing the Mermaid diagram syntax, which can be embedded in Markdown files, documentation, or any Mermaid-compatible viewer.
+
+```ruby
+diagram = Tasker::TaskDiagram.new(task)
+mermaid_string = diagram.to_mermaid
+
+# Example output:
+# graph TD
+# subgraph "Task 123: MyTask"
+#   task_123["Task: MyTask<br/>ID: 123<br/>Status: COMPLETE"]
+#   step_456["Step: step1<br/>Status: COMPLETE<br/>Attempts: 1"]
+#   task_123 --> step_456
+#   ...
+# end
+```
+
+### to_html
+
+This method returns a complete HTML document with the Mermaid diagram embedded and ready to view in a web browser. The HTML includes:
+
+- The Mermaid JavaScript library for rendering
+- Basic styling for the diagram
+- Task information in a header section
+- The rendered diagram
+
+```ruby
+diagram = Tasker::TaskDiagram.new(task)
+html = diagram.to_html
+
+# Save to a file for viewing
+File.write("task_#{task.task_id}_diagram.html", html)
+```
+
+### to_json
+
+This method returns the diagram data as a JSON string with nodes, edges, and diagram properties. This is useful for APIs and for custom rendering.
+
+```ruby
+diagram = Tasker::TaskDiagram.new(task)
+json = diagram.to_json
+
+# Example output:
+# {
+#   "nodes": [
+#     {"id": "task_123", "label": "Task: MyTask\nID: 123\nStatus: COMPLETE", ...},
+#     {"id": "step_456", "label": "Step: step1\nStatus: COMPLETE\nAttempts: 1", ...}
+#   ],
+#   "edges": [
+#     {"source_id": "task_123", "target_id": "step_456", ...}
+#   ],
+#   "direction": "TD",
+#   "title": "Task 123: MyTask"
+# }
+```
+
+## Templates
+
+The HTML rendering uses an ERB template located in the app/views/tasker/task directory:
+
+- `_diagram.html.erb`: Complete HTML document template for standalone viewing
+
+You can customize this template to match your application's styling and requirements.
+
+## Visual Features
+
+The diagram includes the following visual features:
+
+- Color-coded steps based on their status:
+  - PENDING: Light blue
+  - IN_PROGRESS: Light green
+  - COMPLETE: Green
+  - ERROR: Red
+  - CANCELLED: Gray
+
+- Task information at the top, including:
+  - Task name
+  - Task ID
+  - Current status
+
+- Step information, including:
+  - Step name
+  - Status
+  - Number of attempts
+  - Error information (for steps in ERROR status)
+
+- Directed edges showing the workflow dependencies between steps
+
+## Native Diagram Implementation
+
+The TaskDiagram uses a custom, native diagram implementation with the following components:
+
+### Tasker::Diagram::Node
+
+Represents a node in the diagram with properties like:
+- `id`: Unique identifier
+- `label`: Display text (supports HTML line breaks with `<br/>`)
+- `shape`: Node shape (box, circle, etc.)
+- `style`: CSS styling for the node
+- `url`: Optional URL for clickable nodes
+
+### Tasker::Diagram::Edge
+
+Represents a connection between nodes with properties like:
+- `source_id`: ID of the source node
+- `target_id`: ID of the target node
+- `label`: Text displayed on the edge (shown within pipes `|` in the syntax)
+- `type`: Edge style (solid, dashed, etc.)
+- `direction`: Arrow direction (forward, back, both, none)
+
+### Tasker::Diagram::Flowchart
+
+Represents a complete flowchart containing nodes and edges with properties like:
+- `nodes`: Collection of nodes
+- `edges`: Collection of edges
+- `direction`: Layout direction (TD, LR, etc.)
+- `title`: Optional title for the diagram (shown as a subgraph title)
+
+All these components support JSON serialization and generation of Mermaid syntax.
+
+## Examples
+
+### Controller Integration
+
+You can easily integrate the task diagram into a controller action:
+
+```ruby
+class TasksController < ApplicationController
+  def diagram
+    @task = Tasker::Task.find(params[:id])
+    diagram = Tasker::TaskDiagram.new(@task, request.base_url)
+
+    respond_to do |format|
+      format.html { render html: diagram.to_html.html_safe }
+      format.text { render plain: diagram.to_mermaid }
+      format.json { render json: diagram.to_json }
+    end
+  end
+end
+```
+
+### Use in API Responses
+
+```ruby
+def show
+  task = Tasker::Task.find(params[:id])
+  diagram = Tasker::TaskDiagram.new(task, request.base_url)
+
+  render json: {
+    task: task.as_json,
+    diagram: JSON.parse(diagram.to_json),
+    mermaid_syntax: diagram.to_mermaid
+  }
+end
+```
+
+### Use in Rails Views
+
+```erb
+<!-- Using a div with Mermaid class -->
+<div class="mermaid">
+  <%= Tasker::TaskDiagram.new(@task).to_mermaid %>
+</div>
+
+<!-- Include the Mermaid JS library in your layout -->
+<% content_for :head do %>
+  <script src="https://cdn.jsdelivr.net/npm/mermaid/dist/mermaid.min.js"></script>
+  <script>
+    document.addEventListener('DOMContentLoaded', function() {
+      mermaid.initialize({ startOnLoad: true });
+    });
+  </script>
+<% end %>
+```
+
+### Mermaid Syntax Updates
+
+The current Mermaid syntax follows these conventions:
+
+1. Nodes use HTML line breaks with `<br/>` tags instead of newlines
+   ```
+   nodeId["Multi-line<br/>content here"]
+   ```
+
+2. Edge labels use pipe syntax for clarity
+   ```
+   A -->|"connects to"| B
+   ```
+
+3. Titles are implemented as subgraphs rather than title directives
+   ```
+   subgraph "Task 123: Process Order"
+     // nodes and edges here
+   end
+   ```
+
+### Customizing the Template
+
+You can modify the template to match your application's styling or add additional information:
+
+```ruby
+# Customize the template in:
+# app/views/tasker/task/_diagram.html.erb  # Full HTML document
+```

data/docs/TASK_EXECUTION_CONTROL_FLOW.md

@@ -0,0 +1,237 @@
+# Task Execution Control Flow
+
+This document describes the orchestration patterns and control flow for task execution in Tasker, with particular focus on the coordination between WorkflowCoordinator and TaskFinalizer components.
+
+## Overview
+
+Tasker's task execution follows a sophisticated orchestration pattern that supports both coordinated workflow execution and autonomous task management. The system uses a **dual finalization strategy** to handle different execution contexts while maintaining consistency and preventing race conditions.
+
+## Core Components
+
+### 1. WorkflowCoordinator
+- **Role**: Primary orchestration engine for task execution
+- **Responsibility**: Manages step-by-step execution within a single workflow session
+- **Scope**: Single task execution from start to completion
+
+### 2. TaskFinalizer
+- **Role**: Task state management and completion logic
+- **Responsibility**: Determines task completion, handles state transitions, manages reenqueuing
+- **Scope**: Can operate independently or as part of coordinated execution
+
+### 3. Execution Context Separation
+The system distinguishes between two execution contexts:
+- **Coordinated Execution**: Within an active WorkflowCoordinator session
+- **Autonomous Execution**: Independent task management and cleanup
+
+## Control Flow Patterns
+
+### Pattern 1: Coordinated Workflow Execution (Production Primary Path)
+
+```
+TaskRunnerJob.perform(task_id)
+    ↓
+TaskHandler.handle(task)
+    ↓
+WorkflowCoordinator.execute_workflow(task, task_handler)
+    ↓
+[Step Execution Loop]
+    ↓
+WorkflowCoordinator → task_handler.finalize(task, sequence, processed_steps)
+    ↓
+TaskHandler.finalize → task_finalizer.finalize_task_with_steps(task, sequence, processed_steps)
+    ↓
+FinalizationProcessor.finalize_with_steps(task, processed_steps, finalizer)
+    ↓
+TaskFinalizer.finalize_task(task.task_id, synchronous: true)  ← SYNCHRONOUS
+```
+
+**Key Characteristics:**
+- **Single-threaded execution** within the workflow coordinator
+- **Synchronous finalization** prevents reenqueuing/step execution conflicts
+- **Rich event publishing** with processed step context
+- **Coordinated state management** with workflow awareness
+
+### Pattern 2: Autonomous Task Management (Event-Driven)
+
+```
+Event: TaskFinalizer.handle_no_viable_steps(event)
+    ↓
+TaskFinalizer.finalize_task(task_id)  ← Default: synchronous: false
+    ↓
+[Autonomous decision making]
+    ↓
+May reenqueue task or execute ready steps independently
+```
+
+**Key Characteristics:**
+- **Event-driven activation** outside workflow sessions
+- **Asynchronous finalization** allows autonomous orchestration decisions
+- **Independent reenqueuing** based on task state analysis
+- **Cleanup and continuation** logic for orphaned or stalled tasks
+
+## Synchronous vs Asynchronous Finalization
+
+### Synchronous Finalization (`synchronous: true`)
+
+**When Used:**
+- Called from `FinalizationProcessor.finalize_with_steps`
+- During coordinated workflow execution
+- At the end of a WorkflowCoordinator session
+
+**Behavior:**
+```ruby
+if synchronous
+  # In synchronous mode, we can't actually execute steps here
+  # The calling code should handle step execution
+  Rails.logger.info("TaskFinalizer: Task #{task.task_id} ready for synchronous step execution")
+else
+  # In asynchronous mode, reenqueue immediately for step execution
+  finalizer.reenqueue_task_with_context(task, context,
+                                        reason: Constants::TaskFinalization::ReenqueueReasons::READY_STEPS_AVAILABLE)
+end
+```
+
+**Purpose:**
+- **Prevents execution conflicts** with active WorkflowCoordinator
+- **Defers orchestration decisions** to the coordinating workflow
+- **Maintains state consistency** without competing execution attempts
+- **Enables event publishing** while respecting execution boundaries
+
+### Asynchronous Finalization (`synchronous: false`)
+
+**When Used:**
+- Default behavior for standalone finalization calls
+- Event-driven task cleanup (`handle_no_viable_steps`)
+- Independent task management scenarios
+
+**Behavior:**
+- **Can reenqueue tasks** for further processing
+- **Can execute ready steps** autonomously
+- **Makes independent orchestration decisions**
+- **Handles cleanup and continuation** logic
+
+## Event Flow and Observability
+
+### Coordinated Execution Events
+```
+1. publish_finalization_started(task, processed_steps, context)
+2. [Standard finalization logic with synchronous: true]
+3. publish_finalization_completed(task, processed_steps, final_context)
+```
+
+**Event Payload Includes:**
+- `processed_steps_count`
+- `execution_status`
+- `health_status` 
+- `completion_percentage`
+- `total_steps`, `ready_steps`, `failed_steps`
+- `recommended_action`
+
+### Autonomous Execution Events
+Standard task state transition events without detailed step context.
+
+## State Management
+
+### Task State Transitions
+Both execution patterns handle the same core state transitions:
+
+```
+PENDING → IN_PROGRESS → COMPLETE/FAILED
+```
+
+### Coordination States
+Additional states managed during coordinated execution:
+- **READY_STEPS_AVAILABLE**: Steps ready for execution
+- **AWAITING_DEPENDENCIES**: Waiting for step dependencies
+- **STEPS_IN_PROGRESS**: Active step processing
+- **BLOCKED_BY_FAILURES**: Error state requiring intervention
+
+## Reenqueuing Strategy
+
+### Synchronous Context (No Reenqueuing)
+- Task state transitions occur
+- Events are published
+- **Control returns to WorkflowCoordinator**
+- Coordinator decides next actions
+
+### Asynchronous Context (Autonomous Reenqueuing)
+- Task state transitions occur
+- **TaskFinalizer makes reenqueuing decisions**
+- Tasks are queued for continued processing
+- Independent of any workflow session
+
+## Error Handling and Recovery
+
+### Coordinated Execution
+- Errors bubble up to WorkflowCoordinator
+- Coordinated retry and recovery logic
+- Workflow-aware error handling
+
+### Autonomous Execution  
+- Independent error handling
+- Automatic reenqueuing for recoverable failures
+- Isolated failure management
+
+## Design Rationale
+
+### Why Two Execution Patterns?
+
+1. **Execution Conflict Prevention**: Prevents race conditions between WorkflowCoordinator and TaskFinalizer when both might attempt step execution
+
+2. **Orchestration Control**: Allows WorkflowCoordinator to maintain full control over execution timing and step ordering
+
+3. **Event Integration**: Enables rich observability events without interfering with execution flow
+
+4. **Clean Separation**: Separates "finalization as part of workflow execution" from "finalization as independent task cleanup"
+
+5. **Architectural Flexibility**: Supports both coordinated workflow sessions and autonomous task management
+
+### Performance Considerations
+
+- **Reduced Reenqueuing**: Synchronous finalization eliminates unnecessary reenqueuing during active workflow sessions
+- **Optimized Event Publishing**: Rich events only published when step context is available
+- **Efficient State Management**: Single state transition cycle per execution context
+
+## Best Practices
+
+### For Framework Development
+
+1. **Always use `finalize_task_with_steps`** when finalizing from workflow execution
+2. **Use default `finalize_task`** for event-driven cleanup
+3. **Respect the synchronous flag** in finalization logic
+4. **Publish appropriate events** for each execution context
+
+### For Task Handler Implementation
+
+1. **Call `finalize()` method** at end of task execution - it handles coordination automatically
+2. **Don't call TaskFinalizer directly** from task handlers
+3. **Implement `update_annotations()`** hook for custom finalization logic
+
+### For Event Subscribers
+
+1. **Handle both execution patterns** in event subscribers
+2. **Check event context** to understand execution pattern
+3. **Don't assume step context** in all finalization events
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Duplicate Execution**: Usually caused by calling both execution patterns simultaneously
+2. **Missing Events**: Check if using appropriate finalization method for context
+3. **State Inconsistencies**: Verify synchronous flag usage in custom finalization logic
+
+### Debugging Steps
+
+1. Check TaskFinalizer logs for synchronous flag values
+2. Trace event publishing to identify execution pattern
+3. Verify WorkflowCoordinator session boundaries
+4. Monitor reenqueuing patterns for autonomous vs coordinated execution
+
+## Future Considerations
+
+The dual finalization pattern provides a foundation for:
+- **Distributed orchestration** across multiple coordinators
+- **Hierarchical workflow management** with nested coordination
+- **Advanced retry strategies** with context-aware reenqueuing
+- **Performance optimizations** through execution pattern analysis