shikibu 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 6d241ad341f0963dd1fe64cb61dc2a25a95fd8d0f01d12aef81c0372075dbcba
+  data.tar.gz: d287f1ee1f8f58a6a4b6f44565e54ad848792ec3005931fe77a55a187c7c7a6c
+SHA512:
+  metadata.gz: 39e4b6bbabb954eb6fd11ec715e91afff3b61ba2737c836332ed5161b6f5f9cbef2d30b72f3b08b96d76b96e05018e86127c88fffb8952a0aece2724e643a979
+  data.tar.gz: bc4cacdb7f83dc73a80468d0daa92af3bc23f69315b07836e12d195b48d46c4bbe5ed18374609cefe7fc8ba03cd3fde8fff9316b147646b85ff57bad99d637df

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Yasushi Itoh
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,487 @@
+# Shikibu
+
+**Shikibu** (紫式部) - Named after Lady Murasaki Shikibu, author of The Tale of Genji
+
+> Lightweight durable execution framework for Ruby - no separate server required
+
+[![CI](https://github.com/durax-io/shikibu/actions/workflows/ci.yml/badge.svg)](https://github.com/durax-io/shikibu/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Ruby 3.3+](https://img.shields.io/badge/ruby-3.3+-CC342D.svg)](https://www.ruby-lang.org/)
+[![GitHub](https://img.shields.io/badge/github-shikibu-blue.svg)](https://github.com/durax-io/shikibu)
+
+## Overview
+
+Shikibu is a lightweight durable execution framework for Ruby that runs as a **library** in your application - no separate workflow server required. It provides automatic crash recovery through deterministic replay, allowing **long-running workflows** to survive process restarts and failures without losing progress.
+
+**Perfect for**: Order processing, distributed transactions (Saga pattern), and any workflow that must survive crashes.
+
+Shikibu is a Ruby port of [Edda](https://github.com/i2y/edda) (Python), providing the same core concepts and patterns in idiomatic Ruby.
+
+## Key Features
+
+- ✨ **Lightweight Library**: Runs in your application process - no separate server infrastructure
+- 🔄 **Durable Execution**: Deterministic replay with workflow history for automatic crash recovery
+- 🎯 **Workflow & Activity**: Clear separation between orchestration logic and business logic
+- 🔁 **Saga Pattern**: Automatic compensation on failure with `on_failure` blocks
+- 🌐 **Multi-worker Execution**: Run workflows safely across multiple servers or containers
+- 📦 **Transactional Outbox**: Reliable event publishing with guaranteed delivery
+- ☁️ **CloudEvents Support**: Native support for CloudEvents protocol via Rack middleware
+- ⏱️ **Event & Timer Waiting**: Free up worker resources while waiting for events or timers
+- 📬 **Channel-based Messaging**: Actor-model style communication with competing and broadcast modes
+- 📡 **PostgreSQL LISTEN/NOTIFY**: Real-time event delivery without polling
+- 🌍 **Rack Integration**: Works with Rails, Sinatra, Hanami, and any Rack-compatible framework
+- 🔧 **Sidekiq/ActiveJob**: Background worker integration for Rails applications
+
+## Architecture
+
+Shikibu runs as a lightweight library in your applications, with all workflow state stored in a shared database:
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                       Your Ruby Applications                         │
+├──────────────────────┬──────────────────────┬──────────────────────┤
+│   order-service-1    │   order-service-2    │   order-service-3    │
+│   ┌──────────────┐   │   ┌──────────────┐   │   ┌──────────────┐   │
+│   │   Shikibu    │   │   │   Shikibu    │   │   │   Shikibu    │   │
+│   │   Workflow   │   │   │   Workflow   │   │   │   Workflow   │   │
+│   └──────────────┘   │   └──────────────┘   │   └──────────────┘   │
+└──────────┬───────────┴──────────┬───────────┴──────────┬───────────┘
+           │                      │                      │
+           └──────────────────────┼──────────────────────┘
+                                  │
+                         ┌────────▼────────┐
+                         │ Shared Database │
+                         │ (SQLite/PG/MySQL)│
+                         └─────────────────┘
+```
+
+**Key Points**:
+
+- Multiple workers can run simultaneously across different pods/servers
+- Each workflow instance runs on only one worker at a time (automatic coordination)
+- `wait_event` and `sleep` free up worker resources while waiting
+- Automatic crash recovery with stale lock cleanup and workflow auto-resume
+
+## Quick Start
+
+```ruby
+require 'shikibu'
+
+# Register compensation functions (global registry)
+Shikibu.register_compensation(:refund_payment) do |ctx, order_id:|
+  PaymentService.refund(order_id)
+end
+
+class OrderSaga < Shikibu::Workflow
+  workflow_name 'order_saga'
+
+  def execute(order_id:, amount:)
+    # Activity results are recorded in history
+    result = activity :process_payment do
+      PaymentService.charge(order_id, amount)
+    end
+
+    # Compensation on failure (Saga pattern)
+    on_failure :refund_payment, order_id: order_id
+
+    { status: 'completed', order_id: order_id, payment: result }
+  end
+end
+
+# Configure Shikibu
+Shikibu.configure do |config|
+  config.database_url = 'sqlite://workflow.db'
+  config.service_name = 'order-service'
+end
+
+# Start workflow
+result = Shikibu.run(OrderSaga, order_id: 'ORD-123', amount: 99.99)
+```
+
+**What happens on crash?**
+
+1. Activities already executed return cached results from history
+2. Workflow resumes from the last checkpoint
+3. No manual intervention required
+
+## Installation
+
+Add to your Gemfile:
+
+```ruby
+gem 'shikibu'
+```
+
+Then run:
+
+```bash
+bundle install
+```
+
+Or install directly:
+
+```bash
+gem install shikibu
+```
+
+### Database Support
+
+| Database | Use Case | Multi-Pod Support | Production Ready |
+|----------|----------|-------------------|------------------|
+| **SQLite** | Development, testing, single-process | ⚠️ Limited | ⚠️ Limited |
+| **PostgreSQL** | Production, multi-process/multi-pod | ✅ Yes | ✅ Recommended |
+| **MySQL** | Production, multi-process/multi-pod | ✅ Yes | ✅ Yes (8.0+) |
+
+**Important**: For multi-process or multi-pod deployments (K8s, Docker Compose with multiple replicas), use PostgreSQL or MySQL.
+
+### Database Drivers
+
+```ruby
+# Gemfile
+
+# SQLite (included by default)
+gem 'sqlite3', '~> 2.0'
+
+# PostgreSQL
+gem 'pg', '~> 1.5'
+
+# MySQL
+gem 'mysql2', '~> 0.5'
+```
+
+### Database Migrations
+
+Shikibu **automatically applies database migrations** on startup:
+
+```ruby
+# Default: auto-migration enabled
+storage = Shikibu::Storage::SequelStorage.new(database_url, auto_migrate: true)
+
+# Or via App configuration
+app = Shikibu::App.new(database_url: 'postgres://...', auto_migrate: true)
+```
+
+**Features**:
+- **Automatic**: Migrations run during initialization
+- **dbmate-compatible**: Uses the same `schema_migrations` table as dbmate CLI
+- **Multi-worker safe**: Safe for concurrent startup across multiple pods/processes
+
+The database schema is managed in the [durax-io/schema](https://github.com/durax-io/schema) repository, shared between Shikibu (Ruby), [Edda](https://github.com/i2y/edda) (Python), and [Romancy](https://github.com/i2y/romancy) (Go).
+
+## Core Concepts
+
+### Workflows and Activities
+
+**Activity**: A unit of work that performs business logic. Activity results are recorded in history.
+
+**Workflow**: Orchestration logic that coordinates activities. Workflows can be replayed from history after crashes.
+
+```ruby
+class UserOnboarding < Shikibu::Workflow
+  workflow_name 'user_onboarding'
+
+  def execute(email:)
+    # Activity - results are recorded
+    user = activity :create_user do
+      UserService.create(email: email)
+    end
+
+    # Another activity
+    activity :send_welcome_email do
+      EmailService.send_welcome(user[:id])
+    end
+
+    { status: 'completed', user_id: user[:id] }
+  end
+end
+```
+
+**Activity IDs**: Activities are automatically identified with IDs like `"create_user:1"` for deterministic replay.
+
+### Durable Execution
+
+Shikibu ensures workflow progress is never lost through **deterministic replay**:
+
+1. **Activity results are recorded** in a history table
+2. **On crash recovery**, workflows resume from the last checkpoint
+3. **Already-executed activities** return cached results from history
+4. **New activities** continue from where the workflow left off
+
+**Key guarantees**:
+- Activities execute **exactly once** (results cached in history)
+- Workflows can survive **arbitrary crashes**
+- No manual checkpoint management required
+
+### Compensation (Saga Pattern)
+
+When a workflow fails, Shikibu automatically executes compensation functions for **already-executed activities in reverse order**:
+
+```ruby
+# Register compensation functions (supports crash recovery)
+Shikibu.register_compensation(:cancel_reservation) do |ctx, order_id:|
+  InventoryService.cancel_reservation(order_id)
+end
+
+Shikibu.register_compensation(:refund_payment) do |ctx, order_id:|
+  PaymentService.refund(order_id)
+end
+
+class OrderSaga < Shikibu::Workflow
+  workflow_name 'order_saga'
+
+  def execute(order_id:, amount:)
+    # Step 1: Reserve inventory
+    activity :reserve_inventory do
+      InventoryService.reserve(order_id)
+    end
+    on_failure :cancel_reservation, order_id: order_id
+
+    # Step 2: Process payment
+    activity :process_payment do
+      PaymentService.charge(order_id, amount)
+    end
+    on_failure :refund_payment, order_id: order_id
+
+    # Step 3: If this fails, compensations run in reverse order:
+    # → refund payment → cancel reservation
+    activity :confirm_order do
+      OrderService.confirm(order_id)
+    end
+
+    { status: 'completed' }
+  end
+end
+```
+
+### Event & Timer Waiting
+
+Workflows can wait for external events or timers without consuming worker resources:
+
+```ruby
+class PaymentWorkflow < Shikibu::Workflow
+  workflow_name 'payment_workflow'
+
+  def execute(order_id:)
+    # Wait for payment completion event
+    event = wait_event('payment.completed', timeout: 3600)
+
+    { order_id: order_id, payment: event[:data] }
+  end
+end
+```
+
+**Timer waiting with sleep**:
+
+```ruby
+class ReminderWorkflow < Shikibu::Workflow
+  workflow_name 'reminder_workflow'
+
+  def execute(user_id:)
+    # Wait 3 days
+    sleep(3 * 24 * 60 * 60)
+
+    # Check if user completed onboarding
+    unless UserService.completed_onboarding?(user_id)
+      EmailService.send_reminder(user_id)
+    end
+  end
+end
+```
+
+**Key behavior**:
+- `wait_event` and `sleep` release the workflow lock
+- Workflow resumes on any available worker when event arrives or timer expires
+- No worker is blocked while waiting
+
+### Channel-based Messaging
+
+Shikibu provides channel-based messaging for workflow-to-workflow communication:
+
+```ruby
+class JobWorker < Shikibu::Workflow
+  workflow_name 'job_worker'
+
+  def execute(worker_id:)
+    # Subscribe with competing mode - each job goes to ONE worker only
+    subscribe('jobs', mode: :competing)
+
+    loop do
+      job = receive('jobs')
+      process_job(job.data)
+      recur(worker_id: worker_id)  # Continue processing
+    end
+  end
+end
+
+class NotificationHandler < Shikibu::Workflow
+  workflow_name 'notification_handler'
+
+  def execute(handler_id:)
+    # Subscribe with broadcast mode - ALL handlers receive each message
+    subscribe('notifications', mode: :broadcast)
+
+    loop do
+      msg = receive('notifications')
+      send_notification(msg.data)
+      recur(handler_id: handler_id)
+    end
+  end
+end
+```
+
+**Delivery modes**:
+- **`competing`**: Each message goes to exactly ONE subscriber (job queue/task distribution)
+- **`broadcast`**: Each message goes to ALL subscribers (notifications/fan-out)
+
+**Publishing messages**:
+
+```ruby
+# Publish to channel (all subscribers or one competing subscriber)
+publish('jobs', { task: 'send_report', user_id: 123 })
+
+# Direct message to specific workflow instance
+send_to(target_instance_id, 'approval', { approved: true })
+```
+
+### PostgreSQL LISTEN/NOTIFY
+
+When using PostgreSQL, Shikibu can use LISTEN/NOTIFY for real-time event delivery instead of polling:
+
+```ruby
+# Automatically enabled for PostgreSQL URLs
+app = Shikibu::App.new(database_url: 'postgres://localhost/workflows')
+
+# Explicitly enable/disable
+app = Shikibu::App.new(
+  database_url: 'postgres://localhost/workflows',
+  use_listen_notify: true  # or false to disable
+)
+```
+
+**Benefits**:
+- Near-instant workflow resumption (vs polling intervals)
+- Reduced database load
+- Works transparently with existing code
+
+For SQLite/MySQL, Shikibu falls back to polling-based updates.
+
+## Rack Integration
+
+Shikibu provides Rack middleware for CloudEvents endpoints:
+
+```ruby
+# config.ru
+require 'shikibu'
+
+Shikibu.configure do |config|
+  config.database_url = ENV['DATABASE_URL']
+  config.service_name = 'order-service'
+end
+
+# Mount Shikibu middleware
+use Shikibu::Middleware::RackApp
+
+run MyApp
+```
+
+### Rails Integration
+
+```ruby
+# config/initializers/shikibu.rb
+Shikibu.configure do |config|
+  config.database_url = ENV['DATABASE_URL']
+  config.service_name = Rails.application.class.module_parent_name.underscore
+end
+
+# config/routes.rb
+Rails.application.routes.draw do
+  mount Shikibu::Middleware::RackApp.new => '/workflows'
+end
+```
+
+### Sidekiq Integration
+
+```ruby
+# app/jobs/workflow_job.rb
+class WorkflowJob
+  include Sidekiq::Job
+
+  def perform(workflow_class, input)
+    klass = workflow_class.constantize
+    Shikibu.run(klass, **input.symbolize_keys)
+  end
+end
+
+# Usage
+WorkflowJob.perform_async('OrderSaga', { order_id: 'ORD-123', amount: 99.99 })
+```
+
+## Multi-worker Execution
+
+Multiple workers can safely process workflows using database-based exclusive control:
+
+```ruby
+app = Shikibu::App.new(
+  database_url: 'postgresql://localhost/workflows',
+  service_name: 'order-service',
+  worker_id: "worker-#{Process.pid}"
+)
+```
+
+**Features**:
+- Each workflow instance runs on only one worker at a time
+- Automatic stale lock cleanup (5-minute timeout)
+- Crashed workflows automatically resume on any available worker
+
+## Cross-Framework Compatibility
+
+Shikibu shares the same database schema with:
+
+- **[Edda](https://github.com/i2y/edda)** (Python)
+- **[Romancy](https://github.com/i2y/romancy)** (Go)
+
+This means you can:
+- Use multiple languages in the same system
+- Migrate workflows between frameworks
+- Share workflow state across services
+
+Each framework identifies its workflows via the `framework` column (`ruby`, `python`, `go`).
+
+## Development
+
+This project uses [just](https://github.com/casey/just) as a command runner.
+
+```bash
+just              # Show available commands
+just install      # Install dependencies
+just test         # Run unit tests
+just test-file FILE  # Run specific test file
+just lint         # Run RuboCop
+just fix          # Auto-fix lint issues
+just check        # Run lint + tests
+
+# Integration tests (requires Docker)
+just test-integration  # Run all integration tests
+just test-pg           # PostgreSQL only
+just test-mysql        # MySQL only
+```
+
+## Requirements
+
+- Ruby 3.3+
+- SQLite3, PostgreSQL, or MySQL
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Support
+
+- GitHub Issues: https://github.com/durax-io/shikibu/issues
+- Documentation: https://github.com/durax-io/shikibu#readme
+
+## Related Projects
+
+- **[Edda](https://github.com/i2y/edda)** - Python version
+- **[Romancy](https://github.com/i2y/romancy)** - Go version
+- **[durax-io/schema](https://github.com/durax-io/schema)** - Shared database schema

data/lib/shikibu/activity.rb

@@ -0,0 +1,135 @@
+# frozen_string_literal: true
+
+module Shikibu
+  # Standalone activity definition
+  # Use this for reusable activities that can be shared across workflows
+  #
+  # @example
+  #   ProcessPayment = Shikibu::Activity.new(:process_payment) do |ctx, order_id:, amount:|
+  #     PaymentService.charge(order_id, amount)
+  #   end
+  #
+  #   # In a workflow:
+  #   result = ProcessPayment.call(ctx, order_id: '123', amount: 99.99)
+  #
+  class Activity
+    attr_reader :name, :retry_policy
+
+    # Create a new activity
+    # @param name [Symbol, String] Activity name
+    # @param retry_policy [RetryPolicy] Retry policy
+    # @param block [Proc] Activity logic (receives context and kwargs)
+    def initialize(name, retry_policy: nil, &block)
+      @name = name.to_s
+      @retry_policy = retry_policy || RetryPolicy.default
+      @block = block
+    end
+
+    # Execute the activity within a workflow context
+    # @param ctx [WorkflowContext] Workflow context
+    # @param kwargs [Hash] Activity arguments
+    # @return [Object] Activity result
+    def call(ctx, **kwargs)
+      activity_id = ctx.generate_activity_id(@name)
+      ctx.current_activity_id = activity_id
+
+      # Check for cached result during replay
+      if ctx.replaying? && ctx.cached_result?(activity_id)
+        cached = ctx.get_cached_result(activity_id)
+        return cached[:result] if cached[:event_type] == EventType::ACTIVITY_COMPLETED
+
+        # Re-raise cached error
+        raise reconstruct_error(cached)
+      end
+
+      # Execute with retry
+      execute_with_retry(ctx, activity_id, kwargs)
+    ensure
+      ctx.current_activity_id = nil
+    end
+
+    private
+
+    def execute_with_retry(ctx, activity_id, kwargs)
+      attempt = 0
+      started_at = Time.now
+      last_error = nil
+
+      loop do
+        attempt += 1
+
+        begin
+          # Call hooks
+          ctx.hooks&.on_activity_start&.call(ctx.instance_id, activity_id, @name, attempt)
+
+          result = @block.call(ctx, **kwargs)
+
+          # Record successful result
+          ctx.storage.append_history(
+            instance_id: ctx.instance_id,
+            activity_id: activity_id,
+            event_type: EventType::ACTIVITY_COMPLETED,
+            event_data: { result: result }
+          )
+
+          # Cache for replay
+          ctx.cache_result(activity_id, {
+                             event_type: EventType::ACTIVITY_COMPLETED,
+                             result: result
+                           })
+
+          # Call hooks
+          ctx.hooks&.on_activity_complete&.call(ctx.instance_id, activity_id, @name, result, false)
+
+          return result
+        rescue StandardError => e
+          last_error = e
+
+          # Check if retryable
+          unless @retry_policy.retryable?(e) && @retry_policy.should_retry?(attempt, started_at)
+            # Record failure
+            ctx.storage.append_history(
+              instance_id: ctx.instance_id,
+              activity_id: activity_id,
+              event_type: EventType::ACTIVITY_FAILED,
+              event_data: {
+                error_type: e.class.name,
+                error_message: e.message,
+                attempts: attempt
+              }
+            )
+
+            # Cache for replay
+            ctx.cache_result(activity_id, {
+                               event_type: EventType::ACTIVITY_FAILED,
+                               error_type: e.class.name,
+                               error_message: e.message
+                             })
+
+            # Call hooks
+            ctx.hooks&.on_activity_failed&.call(ctx.instance_id, activity_id, @name, e, attempt)
+
+            raise
+          end
+
+          # Call retry hook
+          delay = @retry_policy.delay_for(attempt)
+          ctx.hooks&.on_activity_retry&.call(ctx.instance_id, activity_id, @name, e, attempt, delay)
+
+          # Wait before retry
+          Kernel.sleep(delay)
+        end
+      end
+    end
+
+    def reconstruct_error(cached)
+      error_class = begin
+        Object.const_get(cached[:error_type])
+      rescue StandardError
+        StandardError
+      end
+
+      error_class.new(cached[:error_message])
+    end
+  end
+end