durable_workflow 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 65a867df3be033c50931bcb10082d4275cfd8a73aa449478f4a962de9968f568
+  data.tar.gz: a8eb18ab29be62d4e7b858b870da41ded99c2558e2ca379485fe3368ed04d1f0
+SHA512:
+  metadata.gz: 25fa7216814c9ad68d1f47fa78ef860221e79627fda66a4278cb7c076c0caafdd9aef749b5ad95ef8dee44f3b79b4f6719c278e5b794c321a673f490b26b2ac7
+  data.tar.gz: b63c115a0884467263b86c3571d92fbd149da43759de93a789ebccd3a4f4d96aedbf49904b41c1ace3b038cf4871d2ed8592da6ae38c9eea8d5825da55d79511

data/.claude/todo/01.amend.md

@@ -0,0 +1,133 @@
+# Amendment: State → Execution Type Separation
+
+## Problem
+
+The original specs defined `Execution` struct with typed fields (`status`, `halt_data`, `recover_to`) but Engine/Storage used untyped `ctx[:_status]`, `ctx[:_halt]`, `ctx[:_resume_step]` instead. This created:
+
+1. Type safety issues (symbols become strings after JSON round-trip)
+2. Polluted `ctx` with internal state
+3. Dead code (`Execution`, `ErrorResult` structs never used)
+
+## Solution
+
+- `State` = runtime (passed between executors, clean `ctx` with user variables only)
+- `Execution` = persistence (typed `status`, `halt_data`, `error`, `recover_to`, `result`)
+- Engine works with `State`, saves `Execution`
+- Storage saves/loads `Execution`
+
+---
+
+## TODO
+
+### Phase 1 - Core Types
+
+- [x] **02-TYPES: Update State struct**
+
+  - Remove `from_h` method (not needed for runtime-only struct)
+  - Add comment: "ctx contains user workflow variables only"
+
+- [x] **02-TYPES: Update Execution struct**
+
+  - Add `ExecutionStatus` enum: `:pending`, `:running`, `:completed`, `:halted`, `:failed`
+  - Add `result` field (Any, optional) - final output when completed
+  - Add `error` field (String, optional) - error message when failed
+  - Add `to_state` method - converts to State for executor use
+  - Add `from_state(state, result)` class method - builds from State + ExecutionResult
+  - Add `from_h(hash)` class method - deserialize from storage
+
+- [x] **02-TYPES: Remove ErrorResult**
+
+  - Delete `ErrorResult` struct (errors captured in `Execution.error`)
+
+- [x] **02-TYPES: Update ExecutionResult**
+  - Use `ExecutionStatus` enum for status attribute
+
+### Phase 1 - Engine
+
+- [x] **03-EXECUTION: Engine saves Execution**
+
+  - Add `save_execution(state, result)` private method
+  - Calls `Execution.from_state(state, result)` then `@store.save(execution)`
+
+- [x] **03-EXECUTION: Engine.run uses save_execution**
+
+  - Initial save: `save_execution(state, ExecutionResult.new(status: :running, ...))`
+  - After each step: `save_execution(state, ExecutionResult.new(status: :running, ...))`
+  - On completion: `save_execution(state, result)` then return result
+  - On timeout: `save_execution(state, result)` with `:failed` status
+
+- [x] **03-EXECUTION: Engine.resume loads Execution**
+
+  - `execution = @store.load(execution_id)`
+  - `state = execution.to_state`
+  - Resume step from `execution.recover_to || execution.current_step`
+
+- [x] **03-EXECUTION: handle_halt saves Execution**
+
+  - Build `ExecutionResult` with `:halted` status and halt data
+  - Call `save_execution(state, result)`
+
+- [x] **03-EXECUTION: Remove ctx[:_*] usage**
+  - No `ctx[:_status]`, `ctx[:_halt]`, `ctx[:_resume_step]`, `ctx[:_error]`
+  - Keep `ctx[:_last_error]` only for transient error handler access
+
+### Phase 2 - Storage
+
+- [x] **01-STORAGE: Update Store interface**
+
+  - `save(execution)` - receives Execution struct
+  - `load(execution_id)` - returns Execution struct
+
+- [x] **01-STORAGE: Update Redis adapter**
+
+  - `serialize_execution` / `deserialize_execution`
+  - `find` uses `execution.status` (not `ctx[:_status]`)
+
+- [x] **01-STORAGE: Update ActiveRecord adapter**
+
+  - Save all Execution fields: `status`, `result`, `recover_to`, `halt_data`, `error`
+  - Load returns `Core::Execution.new(...)`
+
+- [x] **01-STORAGE: Update Sequel adapter**
+
+  - Same as ActiveRecord
+
+- [x] **01-STORAGE: Update migrations**
+  - Add columns: `result` (jsonb), `recover_to` (string), `halt_data` (jsonb), `error` (text)
+
+### Phase 2 - Runners
+
+- [x] **02-RUNNERS: Fix Event struct**
+
+  - Change from `Struct.new(...)` to `class Event < BaseStruct`
+
+- [x] **02-RUNNERS: Update Async runner**
+
+  - `run`: Create `Execution.new(status: :pending, ...)` not `State.new(ctx: {_status: :pending})`
+  - `wait`: Use `execution.status` not `state.ctx[:_status]`
+  - `status`: Use `execution.status` not `state.ctx[:_status]`
+  - `build_result`: Use `execution.result`, `execution.halt_data`, `execution.error`
+
+- [x] **02-RUNNERS: Update Inline adapter**
+
+  - Remove manual status update (Engine handles it)
+
+- [x] **02-RUNNERS: Update Sidekiq adapter**
+  - Remove manual status update (Engine handles it)
+
+---
+
+## Verification
+
+After implementation, ensure:
+
+1. ✅ `ctx` only contains user workflow variables
+2. ✅ `find(status: :halted)` works without JSON parsing
+3. ✅ `execution.to_state` / `Execution.from_state` round-trip correctly
+4. ✅ All tests pass with typed Execution fields
+
+---
+
+## ✅ AMENDMENT COMPLETE
+
+**Test Results:** 321 runs, 609 assertions, 0 failures, 0 errors, 0 skips

data/.claude/todo/02.amend.md

@@ -0,0 +1,444 @@
+# Amendment 02: RubyLLM + MCP as Internal Dependencies
+
+## Problem
+
+Current AI extension treats `ruby_llm` and MCP as optional/abstract:
+
+1. Abstract `Provider` class with pluggable implementations
+2. `Provider.current=` configuration dance
+3. MCP executor is a placeholder stub
+4. "Is gem loaded?" checks everywhere
+5. No real MCP server connectivity
+
+This adds complexity without benefit. These gems ARE the implementation.
+
+## Solution
+
+Make `ruby_llm` and `mcp` (official Anthropic MCP SDK) **required runtime dependencies**:
+
+- Direct RubyLLM API calls in Agent executor
+- Real MCP::Client for MCP executor
+- Remove provider abstraction layer
+- Simpler, more powerful, actually works
+
+---
+
+## Gem Details
+
+### ruby_llm
+
+- **Gem:** `ruby_llm`
+- **Repo:** https://github.com/crmne/ruby_llm
+- **Features:** Multi-provider (OpenAI, Anthropic, Gemini, etc.), chat, streaming, tools, embeddings, moderation
+
+```ruby
+# Chat
+chat = RubyLLM.chat
+response = chat.ask("Hello")
+
+# Streaming
+chat.ask("Tell me a story") { |chunk| print chunk.content }
+
+# Tools
+class MyTool < RubyLLM::Tool
+  description "Does something"
+  param :input, desc: "The input"
+  def execute(input:)
+    # return result
+  end
+end
+chat.with_tool(MyTool).ask("Use the tool")
+
+# Moderation
+RubyLLM.moderate("content to check")
+```
+
+### mcp (Anthropic Ruby SDK)
+
+- **Gem:** `mcp`
+- **Repo:** https://github.com/modelcontextprotocol/ruby-sdk
+- **Maintainers:** Anthropic + Shopify
+
+```ruby
+# Client usage
+transport = MCP::Client::HTTP.new(url: "https://server.example.com/mcp")
+client = MCP::Client.new(transport: transport)
+
+# List tools
+tools = client.tools
+tools.each { |t| puts "#{t.name}: #{t.description}" }
+
+# Call tool
+result = client.call_tool(tool: tools.first, arguments: { foo: "bar" })
+
+# With auth
+transport = MCP::Client::HTTP.new(
+  url: "https://server.example.com/mcp",
+  headers: { "Authorization" => "Bearer token" }
+)
+```
+
+---
+
+## TODO
+
+### Phase 1 - Gemspec & Dependencies
+
+- [ ] **Add runtime dependencies to gemspec**
+
+  ```ruby
+  spec.add_dependency "ruby_llm", "~> 1.0"
+  spec.add_dependency "mcp", "~> 0.1"
+  spec.add_dependency "faraday", ">= 2.0"  # Required by MCP HTTP client
+  ```
+
+- [ ] **Update Gemfile**
+  - Move ruby_llm from development to runtime
+  - Add mcp gem
+
+### Phase 2 - Remove Provider Abstraction
+
+- [ ] **Delete `lib/durable_workflow/extensions/ai/provider.rb`**
+
+- [ ] **Delete `lib/durable_workflow/extensions/ai/providers/` directory**
+
+- [ ] **Update `lib/durable_workflow/extensions/ai/ai.rb`**
+
+  - Remove `Provider.current` / `Provider.current=`
+  - Remove `AI.setup(provider:)` - no longer needed
+  - Add `AI.configure` for RubyLLM config (API keys, default model)
+
+  ```ruby
+  module AI
+    class << self
+      attr_accessor :default_model
+
+      def configure(api_key: nil, model: nil)
+        RubyLLM.configure { |c| c.openai_api_key = api_key } if api_key
+        @default_model = model || "gpt-4o-mini"
+      end
+
+      def chat(model: nil)
+        RubyLLM.chat(model: model || default_model)
+      end
+    end
+  end
+  ```
+
+### Phase 3 - Rewrite Agent Executor
+
+- [ ] **Update `lib/durable_workflow/extensions/ai/executors/agent.rb`**
+
+  - Direct RubyLLM usage, no provider indirection
+  - Convert AgentDef tools to RubyLLM::Tool subclasses dynamically
+  - Support streaming via block
+
+  ```ruby
+  def call(state)
+    agent = resolve_agent(config.agent_id)
+    chat = AI.chat(model: agent.model)
+
+    # Add tools
+    agent.tools.each { |t| chat.with_tool(build_tool_class(t)) }
+
+    # Build messages
+    messages = build_messages(state, agent)
+
+    # Execute (with optional streaming)
+    response = if config.stream && @stream_handler
+      chat.ask(messages) { |chunk| @stream_handler.call(chunk) }
+    else
+      chat.ask(messages)
+    end
+
+    store_and_continue(state, response)
+  end
+
+  private
+
+  def build_tool_class(tool_def)
+    # Dynamically create RubyLLM::Tool subclass from ToolDef
+    Class.new(RubyLLM::Tool) do
+      description tool_def.description
+      tool_def.parameters.each { |p| param p.name, desc: p.description }
+
+      define_method(:execute) do |**args|
+        # Call the service method defined in ToolDef
+        svc = Object.const_get(tool_def.service)
+        svc.public_send(tool_def.method_name, **args)
+      end
+    end
+  end
+  ```
+
+### Phase 4 - Rewrite MCP Executor
+
+- [ ] **Update `lib/durable_workflow/extensions/ai/executors/mcp.rb`**
+
+  - Real MCP::Client implementation
+  - Connection pooling for servers
+  - Tool discovery and invocation
+
+  ```ruby
+  class MCP < Base
+    Registry.register("mcp", self)
+
+    # Server connection cache
+    @clients = {}
+
+    def self.client_for(server_config)
+      @clients[server_config[:url]] ||= begin
+        transport = ::MCP::Client::HTTP.new(
+          url: server_config[:url],
+          headers: server_config[:headers] || {}
+        )
+        ::MCP::Client.new(transport: transport)
+      end
+    end
+
+    def call(state)
+      server_config = resolve_server(config.server)
+      client = self.class.client_for(server_config)
+
+      # Find tool
+      tool = client.tools.find { |t| t.name == config.tool }
+      raise ExecutionError, "MCP tool not found: #{config.tool}" unless tool
+
+      # Resolve arguments
+      args = resolve(state, config.arguments || {})
+
+      # Call tool
+      result = client.call_tool(tool: tool, arguments: args)
+
+      state = store(state, config.output, result)
+      continue(state, output: result)
+    end
+
+    private
+
+    def resolve_server(server_id)
+      servers = AI.data_from(workflow)[:mcp_servers] || {}
+      servers[server_id.to_sym] || raise(ExecutionError, "MCP server not found: #{server_id}")
+    end
+  end
+  ```
+
+### Phase 5 - Rewrite Guardrail Executor
+
+- [ ] **Update `lib/durable_workflow/extensions/ai/executors/guardrail.rb`**
+
+  - Use `RubyLLM.moderate` directly for moderation check
+
+  ```ruby
+  def check_moderation(content)
+    result = RubyLLM.moderate(content)
+    GuardrailResult.new(
+      passed: !result.flagged?,
+      check_type: "moderation",
+      reason: result.flagged? ? "Content flagged: #{result.categories.join(', ')}" : nil
+    )
+  end
+  ```
+
+### Phase 6 - Update AI Types
+
+- [ ] **Update `lib/durable_workflow/extensions/ai/types.rb`**
+
+  - Add MCPServerConfig type
+
+  ```ruby
+  class MCPServerConfig < BaseStruct
+    attribute :url, Types::Strict::String
+    attribute? :headers, Types::Hash.default({}.freeze)
+    attribute? :name, Types::Strict::String.optional
+  end
+  ```
+
+- [ ] **Update MCPConfig**
+  ```ruby
+  class MCPConfig < StepConfig
+    attribute :server, Types::Strict::String  # Server ID from workflow config
+    attribute :tool, Types::Strict::String    # Tool name to call
+    attribute? :arguments, Types::Hash.default({}.freeze)
+    attribute? :output, Types::Coercible::Symbol.optional
+  end
+  ```
+
+### Phase 7 - Parser Updates
+
+- [ ] **Update AI extension parser hooks**
+  - Parse `mcp_servers` section from workflow YAML
+  ```yaml
+  mcp_servers:
+    github:
+      url: "https://mcp.github.com/v1"
+      headers:
+        Authorization: "Bearer ${GITHUB_TOKEN}"
+    slack:
+      url: "https://mcp.slack.com/v1"
+  ```
+
+### Phase 8 - Update Tests
+
+- [ ] **Delete `test/unit/extensions/ai/provider_test.rb`**
+
+- [ ] **Delete `test/unit/extensions/ai/providers/` directory**
+
+- [ ] **Update `test/unit/extensions/ai/executors/agent_test.rb`**
+
+  - Mock at RubyLLM level, not provider level
+  - Test RubyLLM::Tool dynamic class generation
+
+- [ ] **Update `test/unit/extensions/ai/executors/mcp_test.rb`**
+
+  - Mock MCP::Client
+  - Test real tool discovery and invocation flow
+
+- [ ] **Update `test/unit/extensions/ai/executors/guardrail_test.rb`**
+
+  - Mock RubyLLM.moderate for moderation tests
+
+- [ ] **Add `test/support/mcp_mock.rb`**
+
+  - Mock MCP::Client for tests
+
+  ```ruby
+  class MockMCPClient
+    def initialize(tools: [])
+      @tools = tools
+    end
+
+    def tools
+      @tools
+    end
+
+    def call_tool(tool:, arguments:)
+      # Return mock result
+    end
+  end
+  ```
+
+### Phase 9 - Documentation
+
+- [ ] **Update README usage examples**
+  - Show RubyLLM configuration
+  - Show MCP server configuration in workflow YAML
+  - Remove provider setup docs
+
+---
+
+## File Changes Summary
+
+| Action | File                                                        |
+| ------ | ----------------------------------------------------------- |
+| DELETE | `lib/durable_workflow/extensions/ai/provider.rb`            |
+| DELETE | `lib/durable_workflow/extensions/ai/providers/ruby_llm.rb`  |
+| DELETE | `lib/durable_workflow/extensions/ai/providers/` directory   |
+| DELETE | `test/unit/extensions/ai/provider_test.rb`                  |
+| MODIFY | `durable_workflow.gemspec` (add deps)                       |
+| MODIFY | `Gemfile` (add deps)                                        |
+| MODIFY | `lib/durable_workflow/extensions/ai/ai.rb`                  |
+| MODIFY | `lib/durable_workflow/extensions/ai/types.rb`               |
+| MODIFY | `lib/durable_workflow/extensions/ai/executors/agent.rb`     |
+| MODIFY | `lib/durable_workflow/extensions/ai/executors/mcp.rb`       |
+| MODIFY | `lib/durable_workflow/extensions/ai/executors/guardrail.rb` |
+| MODIFY | `test/unit/extensions/ai/executors/*.rb`                    |
+| CREATE | `test/support/mcp_mock.rb`                                  |
+
+---
+
+## Workflow YAML Example (After)
+
+```yaml
+id: customer_support
+name: AI Customer Support
+version: "1.0"
+
+# MCP servers available to this workflow
+mcp_servers:
+  zendesk:
+    url: "https://mcp.zendesk.com/v1"
+    headers:
+      Authorization: "Bearer ${ZENDESK_TOKEN}"
+
+agents:
+  support_agent:
+    model: gpt-4o
+    instructions: "You are a helpful support agent..."
+    tools: [lookup_order, create_ticket]
+
+tools:
+  lookup_order:
+    description: "Look up order by ID"
+    parameters:
+      - name: order_id
+        type: string
+        required: true
+    service: OrderService
+    method: find
+
+  create_ticket:
+    description: "Create support ticket"
+    parameters:
+      - name: subject
+        type: string
+      - name: body
+        type: string
+    service: TicketService
+    method: create
+
+steps:
+  - id: start
+    type: start
+    next: check_input
+
+  - id: check_input
+    type: guardrail
+    content: "$input.message"
+    checks:
+      - type: prompt_injection
+      - type: moderation
+    on_fail: rejected
+    next: get_context
+
+  - id: get_context
+    type: mcp
+    server: zendesk
+    tool: get_customer_context
+    arguments:
+      customer_id: "$input.customer_id"
+    output: customer_context
+    next: respond
+
+  - id: respond
+    type: agent
+    agent_id: support_agent
+    prompt: "$input.message"
+    output: response
+    next: end
+
+  - id: rejected
+    type: assign
+    set:
+      response: "I cannot process this request."
+    next: end
+
+  - id: end
+    type: end
+    result: "$response"
+```
+
+---
+
+## Acceptance Criteria
+
+1. ✅ `ruby_llm` is runtime dependency, not optional
+2. ✅ `mcp` gem is runtime dependency
+3. ✅ No `Provider` abstraction - direct RubyLLM calls
+4. ✅ MCP executor actually connects to servers
+5. ✅ MCP executor discovers and calls tools
+6. ✅ Agent executor uses RubyLLM::Tool for tool definitions
+7. ✅ Guardrail uses RubyLLM.moderate directly
+8. ✅ Workflow YAML can define `mcp_servers` section
+9. ✅ All tests pass with mocked RubyLLM/MCP at library level
+10. ✅ Streaming works via RubyLLM blocks