job-workflow 0.5.0 → 0.6.1

data/guides/PRODUCTION_DEPLOYMENT.md

@@ -1,6 +1,6 @@
 # Production Deployment
 
-> ⚠️ **Early Stage (v0.5.0):** JobWorkflow is still in early development. While this section outlines potential deployment patterns, please thoroughly test in your specific environment and monitor for any issues before relying on JobWorkflow in critical production systems.
+> ⚠️ **Early Stage (v0.6.1):** JobWorkflow is still in early development. While this section outlines potential deployment patterns, please thoroughly test in your specific environment and monitor for any issues before relying on JobWorkflow in critical production systems.
 
 This section covers suggested settings and patterns for running JobWorkflow in production-like environments.
 

data/guides/README.md

@@ -1,6 +1,6 @@
 # JobWorkflow Guides
 
-> ⚠️ **Early Stage (v0.5.0):** JobWorkflow is in active development. APIs and features may change. The following guides provide patterns and examples for building workflows, but be aware that implementations may need adjustment as the library evolves.
+> ⚠️ **Early Stage (v0.6.1):** JobWorkflow is in active development. APIs and features may change. The following guides provide patterns and examples for building workflows, but be aware that implementations may need adjustment as the library evolves.
 
 Welcome to the JobWorkflow documentation! This directory contains comprehensive guides to help you build robust workflows with JobWorkflow.
 
@@ -127,6 +127,11 @@ Production deployment and operations:
   - Accessing arguments, outputs, and job status
   - Building dashboards and APIs
 
+- **[MONITORING_UI.md](MONITORING_UI.md)** - Monitoring UI for workflow execution
+  - Mounting the engine
+  - Navigating workflow definitions and executions
+  - Viewing DAG state, arguments, outputs, and fan-out progress
+
 - **[TESTING_STRATEGY.md](TESTING_STRATEGY.md)** - Testing your workflows
   - Unit testing individual tasks
   - Integration testing workflows

data/guides/THROTTLING.md

@@ -2,6 +2,8 @@
 
 JobWorkflow provides semaphore-based throttling to handle external API rate limits and protect shared resources. Throttling works across multiple jobs and workers, ensuring system-wide rate limiting.
 
+For async map tasks (`enqueue: true`), `throttle` is also the official way to cap concurrent sub-job execution. This limit is enforced at perform time by JobWorkflow semaphores, not by SolidQueue's ready/blocked dispatch-state controls.
+
 ## Task-Level Throttling
 
 ### Simple Integer Syntax (Recommended)
@@ -114,6 +116,28 @@ end
 # → Max 5 concurrent API calls at any time
 ```
 
+### Throttling Async Sub-Tasks
+
+When a map task runs as sub-jobs, combine `enqueue: true` with `throttle`:
+
+```ruby
+class AsyncBatchFetchJob < ApplicationJob
+  include JobWorkflow::DSL
+
+  argument :ids, "Array[Integer]"
+
+  task :fetch_all,
+       each: ->(ctx) { ctx.arguments.ids },
+       enqueue: true,
+       throttle: 5,
+       output: { data: "Hash" } do |ctx|
+    { data: RateLimitedAPI.fetch(ctx.each_value) }
+  end
+end
+```
+
+This keeps sub-job fan-out while ensuring only 5 iterations execute at the same time across workers.
+
 ## Runtime Throttling
 
 For fine-grained control within a task, use the `ctx.throttle` method to wrap specific code blocks. This method can only be called inside a task block; calling it outside will raise an error.

data/guides/WORKFLOW_STATUS_QUERY.md

@@ -2,6 +2,8 @@
 
 JobWorkflow provides a robust API for querying the execution status of workflows. This allows you to monitor running workflows, inspect their state, and build observability dashboards.
 
+`JobWorkflow::WorkflowStatus.find` and `find_by` are root workflow APIs. Pass the root workflow `job_id` only. Async sub-job IDs created by `enqueue: true` are intentionally excluded; inspect those via `JobWorkflow::JobStatus`.
+
 ## Basic Usage
 
 ### Finding a Workflow
@@ -16,6 +18,10 @@ return unless status
 
 # Check workflow status
 status.status  # => :pending, :running, :succeeded, or :failed
+
+# Sub-job IDs are excluded from WorkflowStatus
+JobWorkflow::WorkflowStatus.find_by(job_id: "sub-job-123")
+# => nil
 ```
 
 ### Status Check Methods
@@ -251,7 +257,7 @@ end
 
 ### NotFoundError
 
-When using `find`, a `JobWorkflow::WorkflowStatus::NotFoundError` is raised if the job is not found:
+When using `find`, a `JobWorkflow::WorkflowStatus::NotFoundError` is raised if the job is not found. The same applies if you pass a sub-job `job_id` instead of a root workflow `job_id`:
 
 ```ruby
 begin

data/lib/job_workflow/context.rb

@@ -52,10 +52,12 @@ module JobWorkflow
     #     task_context: TaskContext,
     #     output: Output,
     #     job_status: JobStatus,
-    #     ?job: DSL?
+    #     ?job: _JobInterface?
     #   ) -> void
     def initialize(workflow:, arguments:, task_context:, output:, job_status:, job: nil) # rubocop:disable Metrics/ParameterLists, Metrics/AbcSize, Metrics/MethodLength
-      raise "job does not match the provided workflow" if job&.then { |j| j.class._workflow != workflow }
+      if job&.class.respond_to?(:_workflow) && job.class._workflow != workflow
+        raise "job does not match the provided workflow"
+      end
 
       self.job = job
       self.workflow = workflow
@@ -106,12 +108,12 @@ module JobWorkflow
       step.set!(build_step_cursor(current_cursor))
     end
 
-    #:  (DSL) -> void
+    #:  (_JobInterface) -> void
     def _job=(job)
       self.job = job
     end
 
-    #:  () -> DSL?
+    #:  () -> _JobInterface?
     def _job
       job
     end
@@ -277,7 +279,7 @@ module JobWorkflow
 
     private
 
-    attr_accessor :job #: DSL?
+    attr_accessor :job #: _JobInterface?
     attr_writer :workflow #: Workflow
     attr_writer :arguments #: Arguments
     attr_writer :output #: Output

data/lib/job_workflow/dsl.rb

@@ -156,10 +156,6 @@ module JobWorkflow
           dry_run:
         )
         _workflow.add_task(new_task)
-        if new_task.enqueue.should_limits_concurrency? # rubocop:disable Style/GuardClause
-          concurrency = new_task.enqueue.concurrency #: Integer
-          workflow_concurrency(to: concurrency, key: :concurrency_key.to_proc)
-        end
       end
       # rubocop:enable Metrics/ParameterLists
 

data/lib/job_workflow/instrumentation/opentelemetry_subscriber.rb

@@ -18,7 +18,7 @@ module JobWorkflow
     #
     # @note This subscriber requires the opentelemetry-api gem to be installed.
     #   If not available, subscription will be silently skipped.
-    class OpenTelemetrySubscriber # rubocop:disable Metrics/ClassLength
+    class OpenTelemetrySubscriber
       module Attributes
         JOB_NAME = "#{NAMESPACE}.job.name".freeze #: String
         JOB_ID = "#{NAMESPACE}.job.id".freeze #: String

data/lib/job_workflow/instrumentation.rb

@@ -40,7 +40,7 @@ module JobWorkflow
     end
 
     class << self
-      #:  (DSL) { () -> untyped } -> untyped
+      #:  (_JobInterface) { () -> untyped } -> untyped
       def instrument_workflow(job, &)
         payload = build_workflow_payload(job)
         instrument(Events::WORKFLOW_START, payload)
@@ -49,7 +49,7 @@ module JobWorkflow
         instrument(Events::WORKFLOW_COMPLETE, payload)
       end
 
-      #:  (DSL, Task, Context) { () -> untyped } -> untyped
+      #:  (_JobInterface, Task, Context) { () -> untyped } -> untyped
       def instrument_task(job, task, ctx, &)
         payload = build_task_payload(job, task, ctx)
         instrument(Events::TASK_START, payload)
@@ -58,12 +58,12 @@ module JobWorkflow
         instrument(Events::TASK_COMPLETE, payload)
       end
 
-      #:  (DSL, Task, String) -> void
+      #:  (_JobInterface, Task, String) -> void
       def notify_task_skip(job, task, reason)
         instrument(Events::TASK_SKIP, build_task_skip_payload(job, task, reason))
       end
 
-      #:  (DSL, Task, Integer) -> void
+      #:  (_JobInterface, Task, Integer) -> void
       def notify_task_enqueue(job, task, sub_job_count)
         instrument(Events::TASK_ENQUEUE, build_task_enqueue_payload(job, task, sub_job_count))
       end
@@ -73,7 +73,7 @@ module JobWorkflow
         instrument(Events::TASK_RETRY, build_task_retry_payload(task, ctx, job_id, attempt, delay, error))
       end
 
-      #:  (DSL, Task) { () -> untyped } -> untyped
+      #:  (_JobInterface, Task) { () -> untyped } -> untyped
       def instrument_dependent_wait(job, task, &)
         payload = build_dependent_payload(job, task)
         instrument(Events::DEPENDENT_WAIT_START, payload)
@@ -82,7 +82,7 @@ module JobWorkflow
         instrument(Events::DEPENDENT_WAIT_COMPLETE, payload)
       end
 
-      #:  (DSL, Task, Numeric, Integer) -> void
+      #:  (_JobInterface, Task, Numeric, Integer) -> void
       def notify_dependent_reschedule(job, task, reschedule_delay, poll_count)
         instrument(
           Events::DEPENDENT_RESCHEDULE,
@@ -120,7 +120,7 @@ module JobWorkflow
         instrument(event_name, payload, &)
       end
 
-      #:  (DSL, Context, Symbol?, Integer, bool) { () -> untyped } -> untyped
+      #:  (_JobInterface, Context, Symbol?, Integer, bool) { () -> untyped } -> untyped
       def instrument_dry_run(job, ctx, dry_run_name, skip_in_dry_run_index, dry_run, &)
         start_event = dry_run ? Events::DRY_RUN_SKIP : Events::DRY_RUN_EXECUTE
         payload = build_skip_in_dry_run_payload(job, ctx, dry_run_name, skip_in_dry_run_index, dry_run)
@@ -135,7 +135,7 @@ module JobWorkflow
         ActiveSupport::Notifications.instrument(event_name, payload, &)
       end
 
-      #:  (DSL) -> Hash[Symbol, untyped]
+      #:  (_JobInterface) -> Hash[Symbol, untyped]
       def build_workflow_payload(job)
         {
           job:,
@@ -144,7 +144,7 @@ module JobWorkflow
         }
       end
 
-      #:  (DSL, Task, Context) -> Hash[Symbol, untyped]
+      #:  (_JobInterface, Task, Context) -> Hash[Symbol, untyped]
       def build_task_payload(job, task, ctx)
         task_ctx = ctx._task_context
         {
@@ -159,7 +159,7 @@ module JobWorkflow
         }
       end
 
-      #:  (DSL, Task, String) -> Hash[Symbol, untyped]
+      #:  (_JobInterface, Task, String) -> Hash[Symbol, untyped]
       def build_task_skip_payload(job, task, reason)
         {
           job:,
@@ -171,7 +171,7 @@ module JobWorkflow
         }
       end
 
-      #:  (DSL, Task, Integer) -> Hash[Symbol, untyped]
+      #:  (_JobInterface, Task, Integer) -> Hash[Symbol, untyped]
       def build_task_enqueue_payload(job, task, sub_job_count)
         {
           job:,
@@ -200,7 +200,7 @@ module JobWorkflow
         }
       end
 
-      #:  (DSL, Task) -> Hash[Symbol, untyped]
+      #:  (_JobInterface, Task) -> Hash[Symbol, untyped]
       def build_dependent_payload(job, task)
         {
           job:,
@@ -211,7 +211,7 @@ module JobWorkflow
         }
       end
 
-      #:  (DSL, Task, Numeric, Integer) -> Hash[Symbol, untyped]
+      #:  (_JobInterface, Task, Numeric, Integer) -> Hash[Symbol, untyped]
       def build_dependent_reschedule_payload(job, task, reschedule_delay, poll_count)
         {
           job:,
@@ -240,7 +240,7 @@ module JobWorkflow
         }
       end
 
-      #:  (DSL, Context, Symbol?, Integer, bool) -> Hash[Symbol, untyped]
+      #:  (_JobInterface, Context, Symbol?, Integer, bool) -> Hash[Symbol, untyped]
       def build_skip_in_dry_run_payload(job, ctx, dry_run_name, dry_run_index, dry_run)
         {
           job_id: job.job_id,

data/lib/job_workflow/job_status.rb

@@ -55,7 +55,7 @@ module JobWorkflow
       task_job_statuses[task_job_status.task_name][task_job_status.each_index] = task_job_status
     end
 
-    #:  (task_name: Symbol, jobs: Array[DSL]) -> void
+    #:  (task_name: Symbol, jobs: Array[_JobInterface]) -> void
     def update_task_job_statuses_from_jobs(task_name:, jobs:)
       jobs.each.with_index do |job, index|
         update_task_job_status(
@@ -85,6 +85,21 @@ module JobWorkflow
       end
     end
 
+    #:  () -> void
+    def refresh_from_db!
+      statuses = flat_task_job_statuses.reject(&:finished?).index_by(&:job_id)
+      return if statuses.empty?
+
+      task_jobs = QueueAdapter.current.fetch_job_statuses(statuses.keys)
+      statuses.each do |job_id, task_job_status|
+        task_job = task_jobs[job_id]
+        next if task_job.nil?
+
+        task_job_status.update_status(QueueAdapter.current.job_status(task_job))
+        update_task_job_status(task_job_status)
+      end
+    end
+
     private
 
     attr_accessor :task_job_statuses #: Hash[Symbol, Array[TaskJobStatus]]

data/lib/job_workflow/monitoring/dag_layout.rb

@@ -0,0 +1,186 @@
+# frozen_string_literal: true
+
+module JobWorkflow
+  module Monitoring
+    class DagLayout
+      NODE_WIDTH = 224
+      NODE_HEIGHT = 84
+      COLUMN_GAP = 56
+      ROW_GAP = 24
+      PADDING = 16
+      LABEL_LIMIT = 24
+
+      #:  (tasks: Array[Hash[Symbol, untyped]]) -> void
+      def initialize(tasks:)
+        validate_tasks!(tasks)
+        @tasks = tasks
+      end
+
+      #:  () -> Hash[Symbol, untyped]
+      def to_h
+        {
+          width: canvas_width,
+          height: canvas_height,
+          nodes:,
+          edges:
+        }
+      end
+
+      private
+
+      attr_reader :tasks #: Array[Hash[Symbol, untyped]]
+
+      #:  (Array[Hash[Symbol, untyped]]) -> void
+      def validate_tasks!(tasks)
+        seen_names = {} #: Hash[Symbol, bool]
+
+        tasks.each do |task|
+          validate_dependencies!(task, seen_names)
+          seen_names[task.fetch(:name)] = true
+        end
+      end
+
+      #:  (Hash[Symbol, untyped], Hash[Symbol, bool]) -> void
+      def validate_dependencies!(task, seen_names)
+        missing_dependencies = task.fetch(:depends_on).reject { |dependency_name| seen_names[dependency_name] }
+        return if missing_dependencies.empty?
+
+        task_name = task.fetch(:name)
+        dependency_names = missing_dependencies.join(", ")
+
+        raise(
+          ArgumentError,
+          "DagLayout tasks must be topologically sorted; " \
+          "#{task_name} depends on unavailable prior tasks: #{dependency_names}"
+        )
+      end
+
+      #:  () -> Array[Hash[Symbol, untyped]]
+      def nodes
+        @nodes ||= tasks.map do |task|
+          position = node_positions.fetch(task.fetch(:name))
+          task.merge(
+            x: x_for(position.fetch(:column)),
+            y: y_for(position.fetch(:row)),
+            width: NODE_WIDTH,
+            height: NODE_HEIGHT,
+            label: task.fetch(:name).to_s,
+            truncated_label: truncate_label(task.fetch(:name)),
+            meta_label: node_meta_label(task)
+          )
+        end
+      end
+
+      #:  () -> Array[Hash[Symbol, untyped]]
+      def edges
+        @edges ||= tasks.flat_map do |task|
+          task.fetch(:depends_on).map do |dependency_name|
+            edge_view(dependency_name, task.fetch(:name))
+          end
+        end
+      end
+
+      #:  () -> Hash[Symbol, Hash[Symbol, Integer]]
+      def node_positions
+        @node_positions ||= begin
+          column_rows = Hash.new(0) #: Hash[Integer, Integer]
+          positions = {} #: Hash[Symbol, Hash[Symbol, Integer]]
+
+          tasks.each_with_object(positions) do |task, current_positions|
+            column = dependency_column(task, current_positions)
+            row = column_rows[column]
+            column_rows[column] += 1
+            current_positions[task.fetch(:name)] = { column:, row: }
+          end
+        end
+      end
+
+      #:  (Hash[Symbol, untyped], Hash[Symbol, Hash[Symbol, Integer]]) -> Integer
+      def dependency_column(task, positions)
+        depends_on = task.fetch(:depends_on)
+        return 0 if depends_on.empty?
+
+        depends_on.map { |dependency_name| positions.fetch(dependency_name).fetch(:column) + 1 }.max || 0
+      end
+
+      #:  (Symbol, Symbol) -> Hash[Symbol, untyped]
+      def edge_view(from_name, to_name)
+        from_node = node_view(from_name)
+        to_node = node_view(to_name)
+        {
+          from: from_name,
+          to: to_name,
+          path: edge_path(from_node, to_node)
+        }
+      end
+
+      #:  (Hash[Symbol, untyped], Hash[Symbol, untyped]) -> String
+      def edge_path(from_node, to_node)
+        start_x = from_node.fetch(:x) + from_node.fetch(:width)
+        start_y = from_node.fetch(:y) + (from_node.fetch(:height) / 2)
+        end_x = to_node.fetch(:x)
+        end_y = to_node.fetch(:y) + (to_node.fetch(:height) / 2)
+        mid_x = ((start_x + end_x) / 2.0).round(2)
+
+        "M #{start_x} #{start_y} L #{mid_x} #{start_y} L #{mid_x} #{end_y} L #{end_x} #{end_y}"
+      end
+
+      #:  (Symbol) -> Hash[Symbol, untyped]
+      def node_view(task_name)
+        nodes.find { |task| task.fetch(:name) == task_name } || raise(KeyError, task_name.to_s)
+      end
+
+      #:  () -> Integer
+      def canvas_width
+        return 0 if nodes.empty?
+
+        max_column = node_positions.values.map { |position| position.fetch(:column) }.max || 0
+        (PADDING * 2) + ((max_column + 1) * NODE_WIDTH) + (max_column * COLUMN_GAP)
+      end
+
+      #:  () -> Integer
+      def canvas_height
+        return 0 if nodes.empty?
+
+        max_row = node_positions.values.map { |position| position.fetch(:row) }.max || 0
+        (PADDING * 2) + ((max_row + 1) * NODE_HEIGHT) + (max_row * ROW_GAP)
+      end
+
+      #:  (Integer) -> Integer
+      def x_for(column)
+        PADDING + (column * (NODE_WIDTH + COLUMN_GAP))
+      end
+
+      #:  (Integer) -> Integer
+      def y_for(row)
+        PADDING + (row * (NODE_HEIGHT + ROW_GAP))
+      end
+
+      #:  (Symbol) -> String
+      def truncate_label(task_name)
+        label = task_name.to_s
+        return label if label.length <= LABEL_LIMIT
+
+        "#{label[0, LABEL_LIMIT - 1]}…"
+      end
+
+      #:  (Hash[Symbol, untyped]) -> String?
+      def node_meta_label(task)
+        return root_task_label if task.fetch(:depends_on).empty?
+        return each_progress_label(task.fetch(:each_progress)) if task.fetch(:each)
+
+        nil
+      end
+
+      #:  () -> String
+      def root_task_label
+        "root task"
+      end
+
+      #:  (Hash[Symbol, Integer]) -> String
+      def each_progress_label(progress)
+        "each #{progress.fetch(:succeeded)}/#{progress.fetch(:total)}"
+      end
+    end
+  end
+end

data/lib/job_workflow/monitoring/engine.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+module JobWorkflow
+  module Monitoring
+    # :nocov:
+    if defined?(Rails::Engine)
+      class Engine < ::Rails::Engine
+        isolate_namespace JobWorkflow::Monitoring
+
+        JobWorkflow::Monitoring.configure_engine_config(config) if respond_to?(:config)
+      end
+    end
+    # :nocov:
+  end
+end

data/lib/job_workflow/monitoring/execution_page.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+module JobWorkflow
+  module Monitoring
+    class ExecutionPage
+      attr_reader :executions #: Array[ExecutionViewModel]
+      attr_reader :next_cursor #: String?
+
+      #:  (executions: Array[ExecutionViewModel], next_cursor: String?) -> void
+      def initialize(executions:, next_cursor:)
+        @executions = executions
+        @next_cursor = next_cursor
+      end
+    end
+  end
+end

data/lib/job_workflow/monitoring/execution_registry.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+module JobWorkflow
+  module Monitoring
+    class ExecutionRegistry
+      DEFAULT_LIMIT = 25
+      private_constant :DEFAULT_LIMIT
+
+      class << self
+        #:  (job_class_name: String, ?limit: Integer, ?cursor: String?) -> ExecutionPage
+        def page_for(job_class_name:, limit: DEFAULT_LIMIT, cursor: nil)
+          page = QueueAdapter.current.fetch_root_workflow_job_page(job_class_name:, limit:, cursor:)
+          executions = page.fetch(:jobs).filter_map { |job_data| build_view_model(job_data) }
+          ExecutionPage.new(executions:, next_cursor: page[:next_cursor])
+        end
+
+        #:  (String) -> ExecutionViewModel?
+        def find(job_id)
+          job_data = QueueAdapter.current.find_job(job_id)
+          return if job_data.nil?
+
+          build_view_model(job_data, hydrate_sub_tasks: true)
+        end
+
+        private
+
+        #:  (Hash[String, untyped], ?hydrate_sub_tasks: bool) -> ExecutionViewModel?
+        def build_view_model(job_data, hydrate_sub_tasks: false)
+          return if job_data["class_name"] == JobWorkflow::SubTaskJob.name
+
+          status = WorkflowStatus.from_job_data(job_data)
+          hydrate_sub_task_state(status) if hydrate_sub_tasks
+          ExecutionViewModel.new(job_id: job_data.fetch("job_id"), queue_name: job_data["queue_name"], status:)
+        rescue NameError => e
+          raise e if e.is_a?(NoMethodError)
+
+          nil
+        end
+
+        #:  (WorkflowStatus) -> void
+        def hydrate_sub_task_state(status)
+          status.job_status.refresh_from_db!
+          sub_task_job_ids = status.job_status.flat_task_job_statuses.map(&:job_id)
+          sub_task_contexts = QueueAdapter.current.fetch_job_contexts(sub_task_job_ids)
+          status.output.update_task_outputs_from_contexts(sub_task_contexts, status.context.workflow)
+        end
+      end
+    end
+  end
+end