rigid_workflow 1.0.0

data/README.md

@@ -0,0 +1,427 @@
+# Rigid Workflow
+
+A durable workflow orchestration engine for Rails applications. Built on top of ActiveJob and Solid Queue.
+
+## Why Rigid Workflow
+
+In mature Rails applications, background jobs often become complex chains of inter-dependent tasks. Managing this complexity with raw `ActiveJob` calls is error-prone. Rigid Workflow solves this by providing:
+
+- **Explicit State**: Workflows are defined as code, with state persisted automatically in your database.
+- **Observability**: A built-in Admin UI to visualize, monitor, and retry workflows.
+- **Complexity Management**: Native support for parallel execution, race conditions, signals/waits, and compensation (Saga pattern) without external dependencies (like Redis/Temporal) beyond what Rails already provides (ActiveJob).
+
+## Admin UI
+
+<p align="center">
+  <a href="screenshot_rw.png"><img src="screenshot_rw.png" width="640" alt="Rigid Workflow Admin UI"></a>
+</p>
+
+## Use Cases
+
+**Order Fulfillment** — Reserve inventory, charge payment, ship. Compensate on failure
+→ Saga pattern via `rescue`/`raise` to release inventory if payment fails
+
+**Content Moderation** — AI-powered content scanning with human-in-the-loop review
+→ `race` between auto-approve, auto-reject, and manual review signals
+
+**Video Processing** — Parallel transcoding across formats with thumbnail generation
+→ `parallel` for fan-out execution, `loop` for thumbnail extraction at timestamps
+
+**Subscription Billing** — Payment with retries and fallback to downgrade
+→ `step` with `max_attempts` and `retry_delay` for payment gateways
+
+**Employee Offboarding** — Sequential multi-step business process across systems
+→ `step` chaining with persistent state across job enqueues
+
+**User Onboarding** — Trial provisioning with time-gated auto-teardown
+→ `wait` with `timeout`, conditional branching per plan tier
+
+**AI Pipelines** — Scrape websites, chunk content, generate embeddings, summarize with LLMs. Each step retries independently when APIs flake.
+→ `step` with per-step `max_attempts` and `retry_delay`
+
+**RAG Pipelines** — Chunk documents, generate embeddings, index content. Multi-step LLM chains with persistent state.
+→ `step` chaining, `loop` over document collections, `memo` for caching
+
+## Features
+
+- **DSL-based workflow definitions** - Define workflows with a clear, expressive syntax
+- **Sequential and parallel execution** - Run steps in sequence or concurrently
+- **Controlled race conditions** - First to complete wins (signal vs. timer)
+- **Wait states and signals** - Pause workflows for external events, with optional timeouts
+- **Retry logic with exponential backoff** - Automatic retries with configurable delay and jitter
+- **Saga/Compensation pattern** - Auto-compensate completed steps in reverse order when an activity exhausts retries
+- **Workflow versioning** - Track versions of workflow definitions; runs resume safely across version changes
+- **Memoization** - Persist non-deterministic values across workflow resumptions
+- **Looping over collections** - Iterate with persistent state tracking per item
+- **Persistent state** - All workflow state is stored in the database
+- **Admin UI** - Built-in dashboard with overview stats, filtering, pagination, Gantt chart visualization, and bulk actions
+- **Event instrumentation** - `ActiveSupport::Notifications` integration for all key lifecycle events
+- **Generators** - Rails generators for install, workflow, and activity scaffolding
+
+## Requirements
+
+- Ruby >= 3.1
+- Rails >= 7.0
+
+---
+
+## Security Requirements
+
+⚠️ **Authentication Required**: Rigid Workflow's admin UI has NO built-in authentication. You MUST secure it:
+
+### Option 1: Use `admin_controller` config (Recommended)
+
+Set `admin_controller` to a controller that already has authentication (e.g., from Devise):
+
+```ruby
+# config/initializers/rigid_workflow.rb
+RigidWorkflow.configure do |config|
+  config.admin_controller = "Admin::BaseController"
+end
+```
+
+The engine's `ApplicationController` will inherit from this class, automatically applying your authentication.
+
+### Option 2: Wrap routes in host app
+
+Authenticate at the route level in your `config/routes.rb`:
+
+```ruby
+# For Devise:
+authenticate :user do
+  mount RigidWorkflow::Engine => "/admin/rigid_workflow"
+end
+
+# For custom auth:
+authenticate :admin_user do
+  mount RigidWorkflow::Engine => "/admin/rigid_workflow"
+end
+```
+
+**Consequence of skipping authentication**: All workflow data (runs, steps, signals, job details) will be publicly accessible to anyone who can reach the mounted route.
+
+---
+
+## Installation
+
+Add to your Gemfile:
+
+```ruby
+gem "rigid_workflow"
+```
+
+Run the installer:
+
+```bash
+rails generate rigid_workflow:install
+rails db:migrate
+```
+
+The install generator creates:
+
+- `config/initializers/rigid_workflow.rb` - Configuration file
+- Mounts the engine in `config/routes.rb`
+- A migration to create the 4 required tables (`rigid_workflow_runs`, `rigid_workflow_steps`, `rigid_workflow_step_attempts`, `rigid_workflow_signals`)
+
+### Additional Generators
+
+```bash
+# Generate a workflow class
+rails generate rigid_workflow:workflow OrderProcessing
+
+# Generate an activity class
+rails generate rigid_workflow:activity ChargeCustomer
+```
+
+## Quick Start
+
+### Define a Workflow
+
+```ruby
+# app/workflows/order_processing_workflow.rb
+class OrderProcessingWorkflow < RigidWorkflow::Workflow
+  class ValidateOrder < RigidWorkflow::Activity
+    def perform(order_id:, **)
+      order = Order.find(order_id)
+      raise "Invalid order" unless order.valid?
+      { order_id: order_id, validated: true }
+    end
+  end
+
+  class ProcessPayment < RigidWorkflow::Activity
+    def perform(order_id:, **)
+      # Process payment...
+      { payment_id: "pay_123", order_id: order_id }
+    end
+
+    # Called if a later step fails (Saga pattern)
+    def compensate
+      refund_payment(output[:payment_id])
+    end
+  end
+
+  class SendConfirmation < RigidWorkflow::Activity
+    def perform(order_id:, **)
+      OrderMailer.confirmation(order_id).deliver_later
+      { confirmed: true }
+    end
+  end
+
+  version 2  # Optional: track workflow definition versions
+
+  def run
+    step :validate, ValidateOrder, input: { order_id: params[:order_id] }
+    step :payment, ProcessPayment, input: { order_id: params[:order_id] }
+    step :confirm, SendConfirmation, input: { order_id: params[:order_id] }
+  end
+end
+```
+
+### Start a Workflow
+
+```ruby
+run = OrderProcessingWorkflow.start!(order_id: 123)
+```
+
+### Emit a Signal
+
+```ruby
+run.emit_signal(:payment_received, method: "applepay")
+```
+
+## DSL Reference
+
+### Steps
+
+```ruby
+step :name, ActivityClass, input: { key: "value" }, async: true
+```
+
+Options:
+
+| Option          | Description                                   | Default         |
+| --------------- | --------------------------------------------- | --------------- |
+| `input:`        | Hash of input data passed to the activity     | Workflow params |
+| `async:`        | Run asynchronously (enqueued as a job)        | `false`         |
+| `wait:`         | Delay before executing (e.g., `wait: 1.hour`) | `nil`           |
+| `wait_until:`   | Execute at a specific time                    | `nil`           |
+| `max_attempts:` | Number of retry attempts                      | `3`             |
+| `retry_delay:`  | Base delay for exponential backoff            | `15.seconds`    |
+
+**Async restriction**: `async: true` steps will suspend the workflow. Use them inside `parallel` or `race` blocks to avoid suspension, or mark the activity class with `force_async`:
+
+```ruby
+class SlowActivity < RigidWorkflow::Activity
+  force_async true
+
+  def perform(**)
+    # This step always runs asynchronously, even outside parallel/race
+  end
+end
+```
+
+### Loops
+
+```ruby
+loop :items, collection do |item, index|
+  step :process_item, ProcessItemActivity, input: { item: item }
+end
+```
+
+The loop index is persisted in workflow memory. If the workflow is interrupted and resumes, it picks up where it left off.
+
+### Parallel Execution
+
+```ruby
+parallel :notifications do
+  step :email, SendEmailActivity
+  step :sms, SendSmsActivity
+  step :push, SendPushActivity
+end
+```
+
+All steps within a `parallel` block run concurrently via ActiveJob. The workflow suspends until all complete.
+
+### Controlled Race Conditions
+
+```ruby
+race :approval do
+  wait :manual_approval
+  wait :auto_approval, timeout: 24.hours
+end
+```
+
+The first signal to arrive wins. The other branches are canceled.
+
+### Wait States
+
+```ruby
+# Wait indefinitely for a signal
+wait :payment_received
+
+# Wait with timeout
+wait :payment_received, timeout: 1.hour
+
+# Emit a signal elsewhere to resume that workflow
+run.emit_signal(:payment_received, method: "applepay")
+```
+
+### Saga / Compensation
+
+When an activity exhausts all retries, the workflow automatically compensates all previously completed steps in reverse order:
+
+```ruby
+class ReserveInventory < RigidWorkflow::Activity
+  def perform(product_id:, quantity:, **)
+    { reserved: true }
+  end
+
+  def compensate
+    inventory.release(output[:product_id], output[:quantity])
+  end
+end
+```
+
+Compensation is called on each completed step's activity. If compensation itself fails, the run stays in `compensating` status for manual intervention.
+
+### Memoization
+
+```ruby
+def run
+  user = memo(:current_user) { User.find(params[:user_id]) }
+  # User is cached in workflow memory across resumptions
+end
+```
+
+### Versioning
+
+```ruby
+class OrderWorkflow < RigidWorkflow::Workflow
+  version 2
+
+  def run
+    if @run_version < 2
+      # Legacy path for runs started before v2
+    else
+      # Current logic
+    end
+  end
+end
+```
+
+Set `version` in the workflow class. Each run captures the version at start time. Access it via `@run_version` for conditional logic.
+
+## Admin UI
+
+Mount the engine in your routes:
+
+```ruby
+# config/routes.rb
+mount RigidWorkflow::Engine => "/admin/rigid_workflow"
+```
+
+### Pages
+
+- **Overview** (`/admin/rigid_workflow`) - Stats table per workflow class: completed/active/pending/failed counts, success percentage, P50 duration
+- **All Runs** (`/admin/rigid_workflow/runs`) - Paginated list with filtering by status (pending/active/completed/failed), clickable rows, select-all checkboxes, bulk action bar (retry/cancel selected runs)
+- **Run Detail** (`/admin/rigid_workflow/runs/:id`) - Run metadata, step attempt history, and an interactive vis-timeline Gantt chart
+
+### Tech Stack
+
+- Tailwind CSS v4 (CDN)
+- Hotwire Turbo + Stimulus (importmap-managed)
+- vis-timeline for interactive Gantt charts
+- LocalTime for client-side time formatting
+- Kaminari for pagination
+
+## Event Instrumentation
+
+All key lifecycle events emit via `ActiveSupport::Notifications`:
+
+```ruby
+# Subscribe to events
+RigidWorkflow.on("workflow.complete") do |payload|
+  payload # => { run_id: "..." }
+end
+
+RigidWorkflow.on("step.fail") do |payload|
+  payload # => { run_id: "...", step_id: "..." }
+end
+```
+
+Available events: `workflow.start`, `workflow.complete`, `workflow.fail`, `step.complete`, `step.fail`, `step.retry`, `step.canceled`
+
+## Configuration
+
+```ruby
+# config/initializers/rigid_workflow.rb
+RigidWorkflow.configure do |config|
+  # REQUIRED for admin UI security. Controller class the admin UI inherits from.
+  config.admin_controller = "MyAdminController"
+
+  # Maximum retry attempts for failed activities (default: 3)
+  config.max_attempts = 3
+
+  # Base delay for exponential backoff retries in seconds (default: 15)
+  config.retry_delay = 15.seconds
+
+  # Enable logging output in all environments including test (default: false)
+  config.logging = true
+end
+```
+
+| Option             | Default      | Description                                                   |
+| ------------------ | ------------ | ------------------------------------------------------------- |
+| `admin_controller` | `nil`        | Controller class for admin UI inheritance (required for auth) |
+| `max_attempts`     | `3`          | Maximum retry attempts for failed activities                  |
+| `retry_delay`      | `15.seconds` | Base delay for exponential backoff (±20% jitter)              |
+| `logging`          | `nil`        | Enable workflow/activity log output even in test              |
+
+## FAQ
+
+### 1. How does Rigid Workflow handle "zombie" processes?
+
+If a workflow fails or crashes, the last known state remains in the database. When the next worker picks up the job, it continues exactly where it left off. If one of the activities fails, they will be retried automatically by the underlying job queue. Upon success or max attempts reached, the workflow will continue or fail respectively.
+
+### 2. What happens if my database goes down?
+
+Since all state transitions are transactional within your Rails database, if the database is down, the workflow jobs will fail and be retried by the job queue. Once the database is back up, the workers will retry processing the enqueued jobs.
+
+### 3. How do I handle versioning of workflows?
+
+Workflows are code. If you change a workflow definition while a run is in progress:
+
+- **New steps**: Will be picked up as the workflow advances.
+- **Removed steps**: If already completed, their results remain in history. If not yet reached, they are skipped.
+- **Changed logic**: Will apply to all future steps of the currently running workflow.
+  For breaking changes, we recommend creating a new workflow class (e.g., `OrderWorkflowV2`).
+
+### 4. Can I use this for workflows that take months?
+
+Yes. Because state is persisted in the database and execution is driven by job scheduling, signals, and timers, a workflow can sit in a waiting state indefinitely without consuming CPU or memory.
+
+### 5. How do I test these workflows?
+
+You can test individual `Activity` classes in isolation or use integration tests to verify the entire `Workflow` flow. The project itself uses RSpec with an in-memory SQLite database, DatabaseCleaner, and automatic job performance for testing. See the `spec/` directory for examples.
+
+### 6. Is there a performance overhead?
+
+Definitely, every `step` results in at least one database write to persist the state. This is the trade-off for durability. For high-throughput, sub-millisecond tasks, raw `ActiveJob` with Redis might be faster, but for business-critical processes where you cannot afford to lose state, the overhead is acceptable.
+
+### 7. How does it compare to Temporal, Sidekiq, or State Machines?
+
+**Temporal**: Temporal is a separate system (Go/Java server) that requires a heavy infrastructure setup. Rigid Workflow is built on top of Rails and ActiveJob. It lives in your existing DB and uses your existing workers infrastructure.
+
+**Sidekiq**: Sidekiq is a job queue. While you can chain jobs in Sidekiq, managing state, retries, and compensation across chains is manual and complex. Rigid Workflow provides a durable workflow orchestration engine. It is ready to go.
+
+**State Machines (AASM)**: State machines track the status of a model (e.g., an Order). Rigid Workflow tracks the _process_ of a business flow. They are often used together: a Workflow might update an Order's state machine.
+
+**DBOS**: is a multi-language (TypeScript, Python, Java, Kotlin) durable execution library that checkpoints workflow state to Postgres via decorators/annotations. Rigid Workflow is Rails-only and uses ActiveJob + your Rails database.
+
+**When to choose Rigid Workflow**: You're building a Rails app and want a full-featured workflow engine with Saga, signals, race conditions, and an admin dashboard out of the box — all within your existing database and worker infrastructure.
+
+## License
+
+[AGPL-3.0-or-later](./LICENSE.txt)
+
+Rigid Workflow is licensed under the GNU Affero General Public License (AGPL) v3.0 or later. This means that you are allowed to modify the code and/or provide it as a Software-as-a-Service, but you are required to make your modifications available to the users of that service.

data/app/assets/stylesheets/rigid_workflow/application.css

@@ -0,0 +1,68 @@
+button {
+  cursor: pointer;
+}
+
+/* vis-timeline */
+
+html .vis-timeline {
+  border: 1px solid var(--color-gray-200);
+  border-radius: 0.5rem;
+}
+
+html .vis-timeline .vis-top {
+  background-color: color-mix(in srgb, var(--color-gray-100), transparent 50%);
+  font-size: 0.9rem;
+}
+
+html .vis-timeline .vis-panel {
+  border-color: var(--color-gray-200);
+}
+
+html .vis-timeline .vis-top .vis-major {
+  font-weight: 500;
+}
+
+html .vis-timeline .vis-center {
+  background-image:
+    linear-gradient(to right, transparent 39px, #f1f5f9 1px),
+    linear-gradient(to bottom, transparent 39px, #f1f5f9 1px);
+  background-size: 40px 40px;
+  background-color: #ffffff;
+}
+
+html .vis-item .vis-item-overflow {
+  overflow: visible;
+}
+
+html .vis-item .vis-item-content {
+  font-size: 0.8rem;
+  position: relative;
+  padding-right: 14px;
+}
+
+html .vis-item .vis-item-content::after {
+  content: "";
+  position: absolute;
+  right: 4px;
+  top: 50%;
+  transform: translateY(-50%);
+  width: 5px;
+  height: 5px;
+}
+
+html .vis-item.timeline-status-completed .vis-item-content::after {
+  background-color: #16a34a;
+}
+
+html .vis-item.timeline-status-failed .vis-item-content::after {
+  background-color: #dc2626;
+}
+
+html .vis-item.timeline-status-pending .vis-item-content::after,
+html .vis-item.timeline-status-running .vis-item-content::after {
+  background-color: #9ca3af;
+}
+
+html .vis-item.timeline-status-canceled .vis-item-content::after {
+  background-color: #6b7280;
+}

data/app/controllers/rigid_workflow/application_controller.rb

@@ -0,0 +1,90 @@
+# frozen_string_literal: true
+
+# Namespace for RigidWorkflow controllers.
+module RigidWorkflow
+  parent_class =
+    begin
+      RigidWorkflow.config.admin_controller.constantize
+    rescue NameError
+      raise ArgumentError,
+            "Please set config.admin_controller in RigidWorkflow initializer."
+    end
+
+  # Base controller for RigidWorkflow admin interface.
+  # Inherits from the configured admin controller and provides shared functionality.
+  class ApplicationController < parent_class
+    include Rails.application.routes.url_helpers
+    include LocalTimeHelper
+
+    protect_from_forgery with: :exception
+
+    layout "rigid_workflow/layouts/application"
+
+    def default_url_options
+      {
+        host: ENV.fetch("HOST", "localhost"),
+        protocol: ENV.fetch("PROTOCOL", "http")
+      }
+    end
+
+    private
+
+    before_action :set_admin_sidebar
+
+    def set_admin_sidebar
+      @sidebar = [
+        {
+          heading: "Rigid Workflow",
+          items: [
+            { name: "Overview", path: overview_path },
+            {
+              name: "Completed runs",
+              count: RigidWorkflow::Run.where(status: %i[completed]).count,
+              path: completed_runs_path
+            },
+            {
+              name: "Active runs",
+              count: RigidWorkflow::Run.where(status: %i[running]).count,
+              path: active_runs_path
+            },
+            {
+              name: "Pending runs",
+              count: RigidWorkflow::Run.where(status: %i[pending]).count,
+              path: pending_runs_path
+            },
+            {
+              name: "Failed runs",
+              count: RigidWorkflow::Run.where(status: %i[failed]).count,
+              path: failed_runs_path
+            }
+          ]
+        }
+      ]
+    end
+
+    def self.stats_for_ever
+      ActiveRecord::Base.connection.execute(<<-SQL)
+        SELECT
+          r.workflow_class,
+          COUNT(DISTINCT CASE WHEN r.status = 'completed' THEN r.id END) as completed_runs,
+          COUNT(DISTINCT CASE WHEN r.status = 'running' THEN r.id END) as active_runs,
+          COUNT(DISTINCT CASE WHEN r.status = 'pending' THEN r.id END) as pending_runs,
+          COUNT(DISTINCT CASE WHEN r.status = 'failed' THEN r.id END) as failed_runs,
+          100.0 * COUNT(DISTINCT CASE WHEN r.status = 'completed' THEN r.id END) /
+            NULLIF(COUNT(DISTINCT CASE WHEN r.status IN ('completed', 'failed') THEN r.id END), 0) as success_rate,
+          PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY (r.finished_at - r.started_at)) as p50_duration
+        FROM rigid_workflow_runs r
+        GROUP BY r.workflow_class
+      SQL
+    end
+
+    def self.unique_workflows_in_ever
+      ActiveRecord::Base.connection.execute(<<-SQL)
+        SELECT
+          DISTINCT(r.workflow_class)
+        FROM rigid_workflow_runs r
+        GROUP BY r.workflow_class
+      SQL
+    end
+  end
+end