job-workflow 0.1.3 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +30 -0
- data/README.md +1 -1
- data/guides/API_REFERENCE.md +67 -0
- data/guides/PARALLEL_PROCESSING.md +62 -0
- data/guides/PRODUCTION_DEPLOYMENT.md +1 -1
- data/guides/README.md +1 -1
- data/lib/job_workflow/dsl.rb +45 -1
- data/lib/job_workflow/output.rb +4 -12
- data/lib/job_workflow/queue_adapters/abstract.rb +8 -0
- data/lib/job_workflow/queue_adapters/null_adapter.rb +5 -0
- data/lib/job_workflow/queue_adapters/solid_queue_adapter.rb +59 -11
- data/lib/job_workflow/runner.rb +8 -2
- data/lib/job_workflow/version.rb +1 -1
- data/lib/job_workflow/workflow_status.rb +1 -1
- data/rbs_collection.lock.yaml +11 -11
- data/sig/generated/job_workflow/dsl.rbs +32 -0
- data/sig/generated/job_workflow/output.rbs +2 -5
- data/sig/generated/job_workflow/queue_adapters/abstract.rbs +6 -0
- data/sig/generated/job_workflow/queue_adapters/null_adapter.rbs +3 -0
- data/sig/generated/job_workflow/queue_adapters/solid_queue_adapter.rbs +25 -1
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 2a4a82b69372293b7bf689c33125b60b4cbdbcdf4f2e1f468e205cf9bf0c1d85
|
|
4
|
+
data.tar.gz: e2e6e91167d6127e6cee7e2af0339f79ffee244fe6e768d73e50ebb6c2e7b372
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 84356b092c371e103d601576f9fc6ed5cdee9025340948b0d1fe8222b5b1f49f301342f8254a0d858fc89da58e8f828241d7d8ec706103349e63d6f68a426945
|
|
7
|
+
data.tar.gz: b146df16edb3112fa10379437e03ebd6a7f74c46e3a83526bb195320e141e9493319caa96d601622cf1e2f5d8cf18b071cc56c93aac1ad39492b89cfa8e08392
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,35 @@
|
|
|
1
1
|
## [Unreleased]
|
|
2
2
|
|
|
3
|
+
## [0.3.0] - 2026-03-13
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
|
|
7
|
+
- Add `fetch_job_contexts(job_ids)` to queue adapter interface (`Abstract`, `SolidQueueAdapter`, `NullAdapter`) for fetching sub-job context data without direct `SolidQueue::Job` dependency from domain classes
|
|
8
|
+
- Add `persist_job_context(job)` to queue adapter interface for persisting task outputs back to SolidQueue job records after execution
|
|
9
|
+
- Add `without_query_cache` private helper to `SolidQueueAdapter` to bypass ActiveRecord query cache during polling queries
|
|
10
|
+
- Add `"job_workflow_context"` key to `find_job` return hash for direct access to workflow context data
|
|
11
|
+
- Add `AcceptanceNoDependencyWaitJob` and acceptance tests for `depends_on` without `dependency_wait` (polling-only mode)
|
|
12
|
+
- Add acceptance test for output aggregation verification in async workflows
|
|
13
|
+
|
|
14
|
+
### Changed
|
|
15
|
+
|
|
16
|
+
- **Breaking (internal):** Replace `Output#update_task_outputs_from_db` and `Output#update_task_outputs_from_jobs` with `Output#update_task_outputs_from_contexts` — callers now pass context data hashes instead of `SolidQueue::Job` objects
|
|
17
|
+
- `Runner#update_task_outputs` now routes through `QueueAdapter.current.fetch_job_contexts` instead of directly querying `SolidQueue::Job`
|
|
18
|
+
- `Runner#run` now calls `QueueAdapter.current.persist_job_context(job)` after both sub-job and workflow execution
|
|
19
|
+
- `WorkflowStatus.from_job_data` now reads `job_workflow_context` from top-level data first, falling back to `arguments.first.dig("job_workflow_context")`
|
|
20
|
+
- `reschedule_solid_queue_job` now saves full serialized job hash (`active_job.serialize.deep_stringify_keys`) instead of only `["arguments"]`
|
|
21
|
+
- Wrap `find_job`, `fetch_job_statuses`, `job_status`, `reschedule_job`, and `fetch_job_contexts` with `without_query_cache` to prevent stale reads under SolidQueue executor
|
|
22
|
+
|
|
23
|
+
## [0.2.0] - 2026-03-12
|
|
24
|
+
|
|
25
|
+
### Added
|
|
26
|
+
|
|
27
|
+
- Add `workflow_concurrency` DSL class method as a context-aware wrapper around SolidQueue's `limits_concurrency`. Unlike `limits_concurrency`, the key Proc receives a `Context` object, giving access to `ctx.arguments`, `ctx.sub_job?`, and `ctx.concurrency_key`. A fallback `Context` is built from job arguments when `_context` is not yet initialized (e.g. during enqueue before `perform`).
|
|
28
|
+
|
|
29
|
+
### Fixed
|
|
30
|
+
|
|
31
|
+
- Fix task-level concurrency key: internal `limits_concurrency` call for `enqueue: { concurrency: N }` tasks now goes through `workflow_concurrency`, ensuring the key Proc receives a proper `Context` instead of raw ActiveJob arguments. Also changed the internal key proc from `lambda` to `proc` for compatibility with SolidQueue's `instance_exec(*arguments, &proc)` call site.
|
|
32
|
+
|
|
3
33
|
## [0.1.3] - 2026-01-06
|
|
4
34
|
|
|
5
35
|
### Changed
|
data/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# JobWorkflow
|
|
2
2
|
|
|
3
|
-
> ⚠️ **Early Stage (v0.
|
|
3
|
+
> ⚠️ **Early Stage (v0.3.0):** This library is in active development. APIs and features may change in breaking ways without notice. Use in production at your own risk and expect potential breaking changes in future releases.
|
|
4
4
|
|
|
5
5
|
## Overview
|
|
6
6
|
|
data/guides/API_REFERENCE.md
CHANGED
|
@@ -84,6 +84,73 @@ end
|
|
|
84
84
|
|
|
85
85
|
**Map Task Output**: When `each:` is specified, outputs are automatically collected as an array.
|
|
86
86
|
|
|
87
|
+
### workflow_concurrency
|
|
88
|
+
|
|
89
|
+
Configure job-level concurrency limits with workflow-aware context.
|
|
90
|
+
|
|
91
|
+
```ruby
|
|
92
|
+
workflow_concurrency(to:, key:, **opts)
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
**Parameters**:
|
|
96
|
+
- `to` (Integer): Maximum number of concurrent executions
|
|
97
|
+
- `key` (Proc): A Proc that receives a `Context` and returns a String concurrency key
|
|
98
|
+
- `opts` (Hash): Additional options passed to SolidQueue's `limits_concurrency`
|
|
99
|
+
- `on_conflict` (Symbol): `:discard` to drop duplicate jobs (optional)
|
|
100
|
+
- `duration` (ActiveSupport::Duration): How long the concurrency lock is held (optional)
|
|
101
|
+
- `group` (String): Concurrency group name (optional)
|
|
102
|
+
|
|
103
|
+
Unlike SolidQueue's `limits_concurrency` (which passes raw ActiveJob arguments to the key Proc), `workflow_concurrency` passes a **Context** object, giving access to `arguments`, `sub_job?`, and `concurrency_key`.
|
|
104
|
+
|
|
105
|
+
**Example — simple per-tenant key**:
|
|
106
|
+
|
|
107
|
+
```ruby
|
|
108
|
+
class ImportJob < ApplicationJob
|
|
109
|
+
include JobWorkflow::DSL
|
|
110
|
+
|
|
111
|
+
argument :tenant_id, "Integer"
|
|
112
|
+
argument :items, "Array[Integer]"
|
|
113
|
+
|
|
114
|
+
workflow_concurrency to: 1,
|
|
115
|
+
key: ->(ctx) { "import:#{ctx.arguments.tenant_id}" },
|
|
116
|
+
on_conflict: :discard
|
|
117
|
+
|
|
118
|
+
task :process,
|
|
119
|
+
each: ->(ctx) { ctx.arguments.items },
|
|
120
|
+
enqueue: { concurrency: 5 },
|
|
121
|
+
output: { result: "String" } do |ctx|
|
|
122
|
+
{ result: handle(ctx.each_value) }
|
|
123
|
+
end
|
|
124
|
+
end
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
**Example — separating parent and sub-job keys**:
|
|
128
|
+
|
|
129
|
+
```ruby
|
|
130
|
+
class BatchImportJob < ApplicationJob
|
|
131
|
+
include JobWorkflow::DSL
|
|
132
|
+
|
|
133
|
+
argument :tenant_id, "Integer"
|
|
134
|
+
argument :items, "Array[Integer]"
|
|
135
|
+
|
|
136
|
+
workflow_concurrency to: 1,
|
|
137
|
+
key: lambda { |ctx|
|
|
138
|
+
ctx.sub_job? ? ctx.concurrency_key : "batch:#{ctx.arguments.tenant_id}"
|
|
139
|
+
},
|
|
140
|
+
on_conflict: :discard
|
|
141
|
+
|
|
142
|
+
task :process,
|
|
143
|
+
each: ->(ctx) { ctx.arguments.items },
|
|
144
|
+
enqueue: { concurrency: 5 },
|
|
145
|
+
output: { result: "String" } do |ctx|
|
|
146
|
+
{ result: handle(ctx.each_value) }
|
|
147
|
+
end
|
|
148
|
+
end
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
> **Note**: `workflow_concurrency` calls `limits_concurrency` internally. Calling it multiple times in the same class will **overwrite** the previous setting (last-wins). Define it once per job class.
|
|
152
|
+
> Requires SolidQueue.
|
|
153
|
+
|
|
87
154
|
**Example**:
|
|
88
155
|
|
|
89
156
|
```ruby
|
|
@@ -110,6 +110,68 @@ The `enqueue:` option determines how map task iterations are executed:
|
|
|
110
110
|
|
|
111
111
|
**Note**: `enqueue:` works with both regular tasks and map tasks. For map tasks, it enables asynchronous sub-job execution. For regular tasks, it allows conditional enqueueing as a separate job. Legacy syntax (`enqueue: ->(_ctx) { true }` as a Proc) is still supported for backward compatibility.
|
|
112
112
|
|
|
113
|
+
### Job-Level Concurrency with `workflow_concurrency`
|
|
114
|
+
|
|
115
|
+
When using SolidQueue's `limits_concurrency` directly, the key Proc receives raw ActiveJob arguments (a Hash), making it difficult to distinguish parent jobs from sub-jobs. `workflow_concurrency` is a wrapper that passes a **Context** object to the key Proc, giving access to workflow-aware information.
|
|
116
|
+
|
|
117
|
+
```ruby
|
|
118
|
+
class ImportJob < ApplicationJob
|
|
119
|
+
include JobWorkflow::DSL
|
|
120
|
+
|
|
121
|
+
argument :tenant_id, "Integer"
|
|
122
|
+
argument :items, "Array[Integer]"
|
|
123
|
+
|
|
124
|
+
# Limit to 1 concurrent execution per tenant.
|
|
125
|
+
# The key Proc receives a Context, not raw arguments.
|
|
126
|
+
workflow_concurrency to: 1,
|
|
127
|
+
key: ->(ctx) { "import:#{ctx.arguments.tenant_id}" },
|
|
128
|
+
on_conflict: :discard
|
|
129
|
+
|
|
130
|
+
task :process_items,
|
|
131
|
+
each: ->(ctx) { ctx.arguments.items },
|
|
132
|
+
enqueue: { concurrency: 5 },
|
|
133
|
+
output: { result: "String" } do |ctx|
|
|
134
|
+
{ result: process(ctx.each_value) }
|
|
135
|
+
end
|
|
136
|
+
end
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
**Key differences from `limits_concurrency`:**
|
|
140
|
+
|
|
141
|
+
| Feature | `limits_concurrency` (SolidQueue) | `workflow_concurrency` (JobWorkflow) |
|
|
142
|
+
|---|---|---|
|
|
143
|
+
| Key Proc argument | Raw Hash (ActiveJob arguments) | `Context` object |
|
|
144
|
+
| Access to `sub_job?` | ❌ Not available | ✅ `ctx.sub_job?` |
|
|
145
|
+
| Access to `concurrency_key` | ❌ Not available | ✅ `ctx.concurrency_key` |
|
|
146
|
+
| Access to `arguments` | Manual Hash parsing | ✅ `ctx.arguments.<name>` |
|
|
147
|
+
|
|
148
|
+
**Separating parent and sub-job concurrency keys:**
|
|
149
|
+
|
|
150
|
+
When a parent job enqueues sub-jobs (via `enqueue:` option), they share the same job class. With `limits_concurrency`, both parent and sub-jobs resolve to the same concurrency key, which can cause sub-jobs to be discarded. Use `workflow_concurrency` with `ctx.sub_job?` to generate distinct keys:
|
|
151
|
+
|
|
152
|
+
```ruby
|
|
153
|
+
workflow_concurrency to: 1,
|
|
154
|
+
key: lambda { |ctx|
|
|
155
|
+
if ctx.sub_job?
|
|
156
|
+
# Sub-jobs use a unique key per task iteration
|
|
157
|
+
ctx.concurrency_key
|
|
158
|
+
else
|
|
159
|
+
# Parent job uses a tenant-scoped key
|
|
160
|
+
"import:#{ctx.arguments.tenant_id}"
|
|
161
|
+
end
|
|
162
|
+
},
|
|
163
|
+
on_conflict: :discard
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
**Parameters:**
|
|
167
|
+
- `to:` (Integer): Maximum number of concurrent executions
|
|
168
|
+
- `key:` (Proc): A Proc that receives a `Context` and returns a String used as the concurrency key
|
|
169
|
+
- `on_conflict:` (Symbol, optional): `:discard` to drop duplicate jobs, or omit for default SolidQueue behavior
|
|
170
|
+
- `duration:` (ActiveSupport::Duration, optional): How long the concurrency lock is held
|
|
171
|
+
- `group:` (String, optional): Concurrency group name
|
|
172
|
+
|
|
173
|
+
> **Note**: `workflow_concurrency` requires SolidQueue. It delegates to `limits_concurrency` internally.
|
|
174
|
+
|
|
113
175
|
## Fork-Join Pattern
|
|
114
176
|
|
|
115
177
|
### Context Isolation
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Production Deployment
|
|
2
2
|
|
|
3
|
-
> ⚠️ **Early Stage (v0.
|
|
3
|
+
> ⚠️ **Early Stage (v0.3.0):** JobWorkflow is still in early development. While this section outlines potential deployment patterns, please thoroughly test in your specific environment and monitor for any issues before relying on JobWorkflow in critical production systems.
|
|
4
4
|
|
|
5
5
|
This section covers suggested settings and patterns for running JobWorkflow in production-like environments.
|
|
6
6
|
|
data/guides/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# JobWorkflow Guides
|
|
2
2
|
|
|
3
|
-
> ⚠️ **Early Stage (v0.
|
|
3
|
+
> ⚠️ **Early Stage (v0.3.0):** JobWorkflow is in active development. APIs and features may change. The following guides provide patterns and examples for building workflows, but be aware that implementations may need adjustment as the library evolves.
|
|
4
4
|
|
|
5
5
|
Welcome to the JobWorkflow documentation! This directory contains comprehensive guides to help you build robust workflows with JobWorkflow.
|
|
6
6
|
|
data/lib/job_workflow/dsl.rb
CHANGED
|
@@ -165,7 +165,7 @@ module JobWorkflow
|
|
|
165
165
|
_workflow.add_task(new_task)
|
|
166
166
|
if new_task.enqueue.should_limits_concurrency? # rubocop:disable Style/GuardClause
|
|
167
167
|
concurrency = new_task.enqueue.concurrency #: Integer
|
|
168
|
-
|
|
168
|
+
workflow_concurrency(to: concurrency, key: :concurrency_key.to_proc)
|
|
169
169
|
end
|
|
170
170
|
end
|
|
171
171
|
# rubocop:enable Metrics/ParameterLists
|
|
@@ -194,6 +194,50 @@ module JobWorkflow
|
|
|
194
194
|
_workflow.add_hook(:error, task_names:, block:)
|
|
195
195
|
end
|
|
196
196
|
|
|
197
|
+
# Configures concurrency limits for this workflow job.
|
|
198
|
+
#
|
|
199
|
+
# Unlike `limits_concurrency` (SolidQueue's raw API), this method passes a
|
|
200
|
+
# {Context} as the first argument to the key Proc, giving access to
|
|
201
|
+
# workflow-aware information such as `arguments`, `sub_job?`, and
|
|
202
|
+
# `concurrency_key`.
|
|
203
|
+
#
|
|
204
|
+
# When `_context` is not yet initialized (e.g. during enqueue before
|
|
205
|
+
# perform), a temporary Context is built from the job's arguments so the
|
|
206
|
+
# key Proc can always rely on `ctx.arguments`.
|
|
207
|
+
#
|
|
208
|
+
# @example Limit duplicate workflow runs by argument
|
|
209
|
+
# workflow_concurrency to: 1,
|
|
210
|
+
# key: ->(ctx) { "my_job:#{ctx.arguments.tenant_id}" },
|
|
211
|
+
# on_conflict: :discard
|
|
212
|
+
#
|
|
213
|
+
# @example Separate parent and sub-job concurrency keys
|
|
214
|
+
# workflow_concurrency to: 1,
|
|
215
|
+
# key: ->(ctx) {
|
|
216
|
+
# ctx.sub_job? ? ctx.concurrency_key : "my_job:#{ctx.arguments.name}"
|
|
217
|
+
# },
|
|
218
|
+
# on_conflict: :discard
|
|
219
|
+
#
|
|
220
|
+
#: (
|
|
221
|
+
# to: Integer,
|
|
222
|
+
# key: ^(Context) -> String?,
|
|
223
|
+
# ?duration: ActiveSupport::Duration?,
|
|
224
|
+
# ?group: String?,
|
|
225
|
+
# ?on_conflict: Symbol?
|
|
226
|
+
# ) -> void
|
|
227
|
+
def workflow_concurrency(to:, key:, **opts)
|
|
228
|
+
concurrency_key_proc = key
|
|
229
|
+
limits_concurrency(
|
|
230
|
+
to:,
|
|
231
|
+
key: proc {
|
|
232
|
+
ctx = _context || Context.from_hash(
|
|
233
|
+
job: self, workflow: self.class._workflow
|
|
234
|
+
)._update_arguments((arguments.first || {}).symbolize_keys)
|
|
235
|
+
concurrency_key_proc.call(ctx)
|
|
236
|
+
},
|
|
237
|
+
**opts
|
|
238
|
+
)
|
|
239
|
+
end
|
|
240
|
+
|
|
197
241
|
#: (?bool) ?{ (Context) -> bool } -> void
|
|
198
242
|
def dry_run(value = nil, &block)
|
|
199
243
|
validate_namespace!
|
data/lib/job_workflow/output.rb
CHANGED
|
@@ -50,18 +50,10 @@ module JobWorkflow
|
|
|
50
50
|
task_outputs[task_output.task_name][task_output.each_index] = task_output
|
|
51
51
|
end
|
|
52
52
|
|
|
53
|
-
#: (Array[String], Workflow) -> void
|
|
54
|
-
def
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
update_task_outputs_from_jobs(jobs.to_a, workflow)
|
|
59
|
-
end
|
|
60
|
-
|
|
61
|
-
#: (Array[SolidQueue::Job], Workflow) -> void
|
|
62
|
-
def update_task_outputs_from_jobs(jobs, workflow)
|
|
63
|
-
jobs.each do |job|
|
|
64
|
-
context = Context.deserialize(job.arguments["job_workflow_context"].merge("workflow" => workflow))
|
|
53
|
+
#: (Array[Hash[String, untyped]], Workflow) -> void
|
|
54
|
+
def update_task_outputs_from_contexts(context_data_list, workflow)
|
|
55
|
+
context_data_list.each do |context_data|
|
|
56
|
+
context = Context.deserialize(context_data.merge("workflow" => workflow))
|
|
65
57
|
task_output = context.each_task_output
|
|
66
58
|
next if task_output.nil?
|
|
67
59
|
|
|
@@ -77,10 +77,18 @@ module JobWorkflow
|
|
|
77
77
|
raise NotImplementedError, "#{self.class}#find_job must be implemented"
|
|
78
78
|
end
|
|
79
79
|
|
|
80
|
+
#: (Array[String]) -> Array[Hash[String, untyped]]
|
|
81
|
+
def fetch_job_contexts(_job_ids)
|
|
82
|
+
raise NotImplementedError, "#{self.class}#fetch_job_contexts must be implemented"
|
|
83
|
+
end
|
|
84
|
+
|
|
80
85
|
#: (DSL, Numeric) -> bool
|
|
81
86
|
def reschedule_job(_job, _wait)
|
|
82
87
|
false
|
|
83
88
|
end
|
|
89
|
+
|
|
90
|
+
#: (DSL) -> void
|
|
91
|
+
def persist_job_context(_job); end
|
|
84
92
|
end
|
|
85
93
|
# rubocop:enable Naming/PredicateMethod
|
|
86
94
|
end
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
module JobWorkflow
|
|
4
4
|
module QueueAdapters
|
|
5
|
-
# rubocop:disable Naming/PredicateMethod
|
|
5
|
+
# rubocop:disable Naming/PredicateMethod, Metrics/ClassLength
|
|
6
6
|
class SolidQueueAdapter < Abstract
|
|
7
7
|
# @note
|
|
8
8
|
# - Registry scope: @semaphore_registry is process-scoped (shared across fibers/threads
|
|
@@ -74,16 +74,20 @@ module JobWorkflow
|
|
|
74
74
|
def fetch_job_statuses(job_ids)
|
|
75
75
|
return {} unless defined?(SolidQueue::Job)
|
|
76
76
|
|
|
77
|
-
|
|
77
|
+
without_query_cache do
|
|
78
|
+
SolidQueue::Job.where(active_job_id: job_ids).index_by(&:active_job_id)
|
|
79
|
+
end
|
|
78
80
|
end
|
|
79
81
|
|
|
80
82
|
#: (untyped) -> Symbol
|
|
81
83
|
def job_status(job)
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
84
|
+
without_query_cache do
|
|
85
|
+
return :failed if job.failed?
|
|
86
|
+
return :succeeded if job.finished?
|
|
87
|
+
return :running if job.claimed?
|
|
85
88
|
|
|
86
|
-
|
|
89
|
+
:pending
|
|
90
|
+
end
|
|
87
91
|
end
|
|
88
92
|
|
|
89
93
|
#: () -> bool
|
|
@@ -153,23 +157,40 @@ module JobWorkflow
|
|
|
153
157
|
def find_job(job_id)
|
|
154
158
|
return unless defined?(SolidQueue::Job)
|
|
155
159
|
|
|
156
|
-
job = SolidQueue::Job.find_by(active_job_id: job_id)
|
|
160
|
+
job = without_query_cache { SolidQueue::Job.find_by(active_job_id: job_id) }
|
|
157
161
|
return if job.nil?
|
|
158
162
|
|
|
163
|
+
args = job.arguments
|
|
159
164
|
{
|
|
160
165
|
"job_id" => job.active_job_id,
|
|
161
166
|
"class_name" => job.class_name,
|
|
162
167
|
"queue_name" => job.queue_name,
|
|
163
|
-
"arguments" =>
|
|
168
|
+
"arguments" => args.is_a?(Hash) ? args["arguments"] : args,
|
|
169
|
+
"job_workflow_context" => args.is_a?(Hash) ? args["job_workflow_context"] : nil,
|
|
164
170
|
"status" => job_status(job)
|
|
165
171
|
}
|
|
166
172
|
end
|
|
167
173
|
|
|
174
|
+
# @note
|
|
175
|
+
# - Fetches job_workflow_context hashes for the given job IDs.
|
|
176
|
+
#
|
|
177
|
+
#: (Array[String]) -> Array[Hash[String, untyped]]
|
|
178
|
+
def fetch_job_contexts(job_ids)
|
|
179
|
+
return [] unless defined?(SolidQueue::Job)
|
|
180
|
+
return [] if job_ids.empty?
|
|
181
|
+
|
|
182
|
+
jobs = without_query_cache { SolidQueue::Job.where(active_job_id: job_ids).to_a }
|
|
183
|
+
jobs.filter_map do |job|
|
|
184
|
+
args = job.arguments
|
|
185
|
+
args.is_a?(Hash) ? args["job_workflow_context"] : nil
|
|
186
|
+
end
|
|
187
|
+
end
|
|
188
|
+
|
|
168
189
|
#: (DSL, Numeric) -> bool
|
|
169
190
|
def reschedule_job(job, wait)
|
|
170
191
|
return false unless defined?(SolidQueue::Job)
|
|
171
192
|
|
|
172
|
-
solid_queue_job = SolidQueue::Job.find_by(active_job_id: job.job_id)
|
|
193
|
+
solid_queue_job = without_query_cache { SolidQueue::Job.find_by(active_job_id: job.job_id) }
|
|
173
194
|
return false unless solid_queue_job&.claimed?
|
|
174
195
|
|
|
175
196
|
reschedule_solid_queue_job(solid_queue_job, job, wait)
|
|
@@ -177,17 +198,44 @@ module JobWorkflow
|
|
|
177
198
|
false
|
|
178
199
|
end
|
|
179
200
|
|
|
201
|
+
# @note
|
|
202
|
+
# - Persists the job's updated context (including task outputs) back
|
|
203
|
+
# to the SolidQueue job record after execution completes. Without this,
|
|
204
|
+
# outputs computed during job execution would be lost because
|
|
205
|
+
# SolidQueue does not re-serialize job arguments after perform.
|
|
206
|
+
#
|
|
207
|
+
#: (DSL) -> void
|
|
208
|
+
def persist_job_context(job)
|
|
209
|
+
return unless defined?(SolidQueue::Job)
|
|
210
|
+
|
|
211
|
+
solid_queue_job = SolidQueue::Job.find_by(active_job_id: job.job_id)
|
|
212
|
+
return if solid_queue_job.nil?
|
|
213
|
+
|
|
214
|
+
solid_queue_job.update!(arguments: job.serialize.deep_stringify_keys)
|
|
215
|
+
end
|
|
216
|
+
|
|
180
217
|
private
|
|
181
218
|
|
|
182
219
|
attr_reader :semaphore_registry #: Hash[Object, ^(SolidQueue::Worker) -> void]
|
|
183
220
|
|
|
221
|
+
# @note
|
|
222
|
+
# - Bypasses ActiveRecord query cache for the given block.
|
|
223
|
+
# - When running under SolidQueue's executor, SELECT queries are cached
|
|
224
|
+
# for the entire job execution. Polling queries must bypass this cache
|
|
225
|
+
# to observe status changes made by other threads/processes.
|
|
226
|
+
#
|
|
227
|
+
#: [T] () { () -> T } -> T
|
|
228
|
+
def without_query_cache(&)
|
|
229
|
+
defined?(SolidQueue::Job) ? SolidQueue::Job.uncached(&) : yield
|
|
230
|
+
end
|
|
231
|
+
|
|
184
232
|
#: (SolidQueue::Job, DSL, Numeric) -> bool
|
|
185
233
|
def reschedule_solid_queue_job(solid_queue_job, active_job, wait)
|
|
186
234
|
solid_queue_job.with_lock do
|
|
187
235
|
solid_queue_job.claimed_execution&.destroy!
|
|
188
236
|
solid_queue_job.update!(
|
|
189
237
|
scheduled_at: wait.seconds.from_now,
|
|
190
|
-
arguments: active_job.serialize.deep_stringify_keys
|
|
238
|
+
arguments: active_job.serialize.deep_stringify_keys
|
|
191
239
|
)
|
|
192
240
|
solid_queue_job.prepare_for_execution
|
|
193
241
|
end
|
|
@@ -219,6 +267,6 @@ module JobWorkflow
|
|
|
219
267
|
end
|
|
220
268
|
end
|
|
221
269
|
end
|
|
222
|
-
# rubocop:enable Naming/PredicateMethod
|
|
270
|
+
# rubocop:enable Naming/PredicateMethod, Metrics/ClassLength
|
|
223
271
|
end
|
|
224
272
|
end
|
data/lib/job_workflow/runner.rb
CHANGED
|
@@ -13,9 +13,14 @@ module JobWorkflow
|
|
|
13
13
|
#: () -> void
|
|
14
14
|
def run
|
|
15
15
|
task = context._task_context.task
|
|
16
|
-
|
|
16
|
+
if !task.nil? && context.sub_job?
|
|
17
|
+
run_task(task)
|
|
18
|
+
QueueAdapter.current.persist_job_context(job)
|
|
19
|
+
return
|
|
20
|
+
end
|
|
17
21
|
|
|
18
22
|
catch(:rescheduled) { run_workflow }
|
|
23
|
+
QueueAdapter.current.persist_job_context(job)
|
|
19
24
|
end
|
|
20
25
|
|
|
21
26
|
private
|
|
@@ -167,7 +172,8 @@ module JobWorkflow
|
|
|
167
172
|
#: (Task) -> void
|
|
168
173
|
def update_task_outputs(task)
|
|
169
174
|
finished_job_ids = context.job_status.finished_job_ids(task_name: task.task_name)
|
|
170
|
-
|
|
175
|
+
context_data_list = QueueAdapter.current.fetch_job_contexts(finished_job_ids)
|
|
176
|
+
context.output.update_task_outputs_from_contexts(context_data_list, context.workflow)
|
|
171
177
|
end
|
|
172
178
|
end
|
|
173
179
|
end
|
data/lib/job_workflow/version.rb
CHANGED
|
@@ -34,7 +34,7 @@ module JobWorkflow
|
|
|
34
34
|
job_class = job_class_name.constantize
|
|
35
35
|
workflow = job_class._workflow
|
|
36
36
|
|
|
37
|
-
context_data = data["arguments"]
|
|
37
|
+
context_data = data["job_workflow_context"] || data["arguments"]&.first&.dig("job_workflow_context")
|
|
38
38
|
context = if context_data
|
|
39
39
|
Context.deserialize(context_data.merge("workflow" => workflow))
|
|
40
40
|
else
|
data/rbs_collection.lock.yaml
CHANGED
|
@@ -6,7 +6,7 @@ gems:
|
|
|
6
6
|
source:
|
|
7
7
|
type: git
|
|
8
8
|
name: ruby/gem_rbs_collection
|
|
9
|
-
revision:
|
|
9
|
+
revision: 9bf2eebb1c54b5d6f23f2acb65d4c36f195b4783
|
|
10
10
|
remote: https://github.com/ruby/gem_rbs_collection.git
|
|
11
11
|
repo_dir: gems
|
|
12
12
|
- name: activerecord
|
|
@@ -14,7 +14,7 @@ gems:
|
|
|
14
14
|
source:
|
|
15
15
|
type: git
|
|
16
16
|
name: ruby/gem_rbs_collection
|
|
17
|
-
revision:
|
|
17
|
+
revision: 9bf2eebb1c54b5d6f23f2acb65d4c36f195b4783
|
|
18
18
|
remote: https://github.com/ruby/gem_rbs_collection.git
|
|
19
19
|
repo_dir: gems
|
|
20
20
|
- name: activesupport
|
|
@@ -22,7 +22,7 @@ gems:
|
|
|
22
22
|
source:
|
|
23
23
|
type: git
|
|
24
24
|
name: ruby/gem_rbs_collection
|
|
25
|
-
revision:
|
|
25
|
+
revision: 9bf2eebb1c54b5d6f23f2acb65d4c36f195b4783
|
|
26
26
|
remote: https://github.com/ruby/gem_rbs_collection.git
|
|
27
27
|
repo_dir: gems
|
|
28
28
|
- name: base64
|
|
@@ -34,7 +34,7 @@ gems:
|
|
|
34
34
|
source:
|
|
35
35
|
type: git
|
|
36
36
|
name: ruby/gem_rbs_collection
|
|
37
|
-
revision:
|
|
37
|
+
revision: 9bf2eebb1c54b5d6f23f2acb65d4c36f195b4783
|
|
38
38
|
remote: https://github.com/ruby/gem_rbs_collection.git
|
|
39
39
|
repo_dir: gems
|
|
40
40
|
- name: concurrent-ruby
|
|
@@ -42,7 +42,7 @@ gems:
|
|
|
42
42
|
source:
|
|
43
43
|
type: git
|
|
44
44
|
name: ruby/gem_rbs_collection
|
|
45
|
-
revision:
|
|
45
|
+
revision: 9bf2eebb1c54b5d6f23f2acb65d4c36f195b4783
|
|
46
46
|
remote: https://github.com/ruby/gem_rbs_collection.git
|
|
47
47
|
repo_dir: gems
|
|
48
48
|
- name: connection_pool
|
|
@@ -50,7 +50,7 @@ gems:
|
|
|
50
50
|
source:
|
|
51
51
|
type: git
|
|
52
52
|
name: ruby/gem_rbs_collection
|
|
53
|
-
revision:
|
|
53
|
+
revision: 9bf2eebb1c54b5d6f23f2acb65d4c36f195b4783
|
|
54
54
|
remote: https://github.com/ruby/gem_rbs_collection.git
|
|
55
55
|
repo_dir: gems
|
|
56
56
|
- name: date
|
|
@@ -74,7 +74,7 @@ gems:
|
|
|
74
74
|
source:
|
|
75
75
|
type: git
|
|
76
76
|
name: ruby/gem_rbs_collection
|
|
77
|
-
revision:
|
|
77
|
+
revision: 9bf2eebb1c54b5d6f23f2acb65d4c36f195b4783
|
|
78
78
|
remote: https://github.com/ruby/gem_rbs_collection.git
|
|
79
79
|
repo_dir: gems
|
|
80
80
|
- name: i18n
|
|
@@ -82,7 +82,7 @@ gems:
|
|
|
82
82
|
source:
|
|
83
83
|
type: git
|
|
84
84
|
name: ruby/gem_rbs_collection
|
|
85
|
-
revision:
|
|
85
|
+
revision: 9bf2eebb1c54b5d6f23f2acb65d4c36f195b4783
|
|
86
86
|
remote: https://github.com/ruby/gem_rbs_collection.git
|
|
87
87
|
repo_dir: gems
|
|
88
88
|
- name: json
|
|
@@ -98,7 +98,7 @@ gems:
|
|
|
98
98
|
source:
|
|
99
99
|
type: git
|
|
100
100
|
name: ruby/gem_rbs_collection
|
|
101
|
-
revision:
|
|
101
|
+
revision: 9bf2eebb1c54b5d6f23f2acb65d4c36f195b4783
|
|
102
102
|
remote: https://github.com/ruby/gem_rbs_collection.git
|
|
103
103
|
repo_dir: gems
|
|
104
104
|
- name: monitor
|
|
@@ -138,7 +138,7 @@ gems:
|
|
|
138
138
|
source:
|
|
139
139
|
type: git
|
|
140
140
|
name: ruby/gem_rbs_collection
|
|
141
|
-
revision:
|
|
141
|
+
revision: 9bf2eebb1c54b5d6f23f2acb65d4c36f195b4783
|
|
142
142
|
remote: https://github.com/ruby/gem_rbs_collection.git
|
|
143
143
|
repo_dir: gems
|
|
144
144
|
- name: stringio
|
|
@@ -162,7 +162,7 @@ gems:
|
|
|
162
162
|
source:
|
|
163
163
|
type: git
|
|
164
164
|
name: ruby/gem_rbs_collection
|
|
165
|
-
revision:
|
|
165
|
+
revision: 9bf2eebb1c54b5d6f23f2acb65d4c36f195b4783
|
|
166
166
|
remote: https://github.com/ruby/gem_rbs_collection.git
|
|
167
167
|
repo_dir: gems
|
|
168
168
|
- name: uri
|
|
@@ -94,6 +94,38 @@ module JobWorkflow
|
|
|
94
94
|
# : (*Symbol) { (Context, StandardError, Task) -> void } -> void
|
|
95
95
|
def on_error: (*Symbol) { (Context, StandardError, Task) -> void } -> void
|
|
96
96
|
|
|
97
|
+
# Configures concurrency limits for this workflow job.
|
|
98
|
+
#
|
|
99
|
+
# Unlike `limits_concurrency` (SolidQueue's raw API), this method passes a
|
|
100
|
+
# {Context} as the first argument to the key Proc, giving access to
|
|
101
|
+
# workflow-aware information such as `arguments`, `sub_job?`, and
|
|
102
|
+
# `concurrency_key`.
|
|
103
|
+
#
|
|
104
|
+
# When `_context` is not yet initialized (e.g. during enqueue before
|
|
105
|
+
# perform), a temporary Context is built from the job's arguments so the
|
|
106
|
+
# key Proc can always rely on `ctx.arguments`.
|
|
107
|
+
#
|
|
108
|
+
# @example Limit duplicate workflow runs by argument
|
|
109
|
+
# workflow_concurrency to: 1,
|
|
110
|
+
# key: ->(ctx) { "my_job:#{ctx.arguments.tenant_id}" },
|
|
111
|
+
# on_conflict: :discard
|
|
112
|
+
#
|
|
113
|
+
# @example Separate parent and sub-job concurrency keys
|
|
114
|
+
# workflow_concurrency to: 1,
|
|
115
|
+
# key: ->(ctx) {
|
|
116
|
+
# ctx.sub_job? ? ctx.concurrency_key : "my_job:#{ctx.arguments.name}"
|
|
117
|
+
# },
|
|
118
|
+
# on_conflict: :discard
|
|
119
|
+
#
|
|
120
|
+
# : (
|
|
121
|
+
# to: Integer,
|
|
122
|
+
# key: ^(Context) -> String?,
|
|
123
|
+
# ?duration: ActiveSupport::Duration?,
|
|
124
|
+
# ?group: String?,
|
|
125
|
+
# ?on_conflict: Symbol?
|
|
126
|
+
# ) -> void
|
|
127
|
+
def workflow_concurrency: (to: Integer, key: ^(Context) -> String?, ?duration: ActiveSupport::Duration?, ?group: String?, ?on_conflict: Symbol?) -> void
|
|
128
|
+
|
|
97
129
|
# : (?bool) ?{ (Context) -> bool } -> void
|
|
98
130
|
def dry_run: (?bool) ?{ (Context) -> bool } -> void
|
|
99
131
|
|
|
@@ -23,11 +23,8 @@ module JobWorkflow
|
|
|
23
23
|
# : (TaskOutput) -> void
|
|
24
24
|
def add_task_output: (TaskOutput) -> void
|
|
25
25
|
|
|
26
|
-
# : (Array[String], Workflow) -> void
|
|
27
|
-
def
|
|
28
|
-
|
|
29
|
-
# : (Array[SolidQueue::Job], Workflow) -> void
|
|
30
|
-
def update_task_outputs_from_jobs: (Array[SolidQueue::Job], Workflow) -> void
|
|
26
|
+
# : (Array[Hash[String, untyped]], Workflow) -> void
|
|
27
|
+
def update_task_outputs_from_contexts: (Array[Hash[String, untyped]], Workflow) -> void
|
|
31
28
|
|
|
32
29
|
# : () -> Array[TaskOutput]
|
|
33
30
|
def flat_task_outputs: () -> Array[TaskOutput]
|
|
@@ -49,8 +49,14 @@ module JobWorkflow
|
|
|
49
49
|
# : (String) -> Hash[String, untyped]?
|
|
50
50
|
def find_job: (String) -> Hash[String, untyped]?
|
|
51
51
|
|
|
52
|
+
# : (Array[String]) -> Array[Hash[String, untyped]]
|
|
53
|
+
def fetch_job_contexts: (Array[String]) -> Array[Hash[String, untyped]]
|
|
54
|
+
|
|
52
55
|
# : (DSL, Numeric) -> bool
|
|
53
56
|
def reschedule_job: (DSL, Numeric) -> bool
|
|
57
|
+
|
|
58
|
+
# : (DSL) -> void
|
|
59
|
+
def persist_job_context: (DSL) -> void
|
|
54
60
|
end
|
|
55
61
|
end
|
|
56
62
|
end
|
|
@@ -51,6 +51,9 @@ module JobWorkflow
|
|
|
51
51
|
# : (String) -> Hash[String, untyped]?
|
|
52
52
|
def find_job: (String) -> Hash[String, untyped]?
|
|
53
53
|
|
|
54
|
+
# : (Array[String]) -> Array[Hash[String, untyped]]
|
|
55
|
+
def fetch_job_contexts: (Array[String]) -> Array[Hash[String, untyped]]
|
|
56
|
+
|
|
54
57
|
# : (DSL, Numeric) -> bool
|
|
55
58
|
def reschedule_job: (DSL, Numeric) -> bool
|
|
56
59
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
module JobWorkflow
|
|
4
4
|
module QueueAdapters
|
|
5
|
-
# rubocop:disable Naming/PredicateMethod
|
|
5
|
+
# rubocop:disable Naming/PredicateMethod, Metrics/ClassLength
|
|
6
6
|
class SolidQueueAdapter < Abstract
|
|
7
7
|
# @note
|
|
8
8
|
# - Registry scope: @semaphore_registry is process-scoped (shared across fibers/threads
|
|
@@ -82,13 +82,37 @@ module JobWorkflow
|
|
|
82
82
|
# : (String) -> Hash[String, untyped]?
|
|
83
83
|
def find_job: (String) -> Hash[String, untyped]?
|
|
84
84
|
|
|
85
|
+
# @note
|
|
86
|
+
# - Fetches job_workflow_context hashes for the given job IDs.
|
|
87
|
+
#
|
|
88
|
+
# : (Array[String]) -> Array[Hash[String, untyped]]
|
|
89
|
+
def fetch_job_contexts: (Array[String]) -> Array[Hash[String, untyped]]
|
|
90
|
+
|
|
85
91
|
# : (DSL, Numeric) -> bool
|
|
86
92
|
def reschedule_job: (DSL, Numeric) -> bool
|
|
87
93
|
|
|
94
|
+
# @note
|
|
95
|
+
# - Persists the job's updated context (including task outputs) back
|
|
96
|
+
# to the SolidQueue job record after execution completes. Without this,
|
|
97
|
+
# outputs computed during job execution would be lost because
|
|
98
|
+
# SolidQueue does not re-serialize job arguments after perform.
|
|
99
|
+
#
|
|
100
|
+
# : (DSL) -> void
|
|
101
|
+
def persist_job_context: (DSL) -> void
|
|
102
|
+
|
|
88
103
|
private
|
|
89
104
|
|
|
90
105
|
attr_reader semaphore_registry: Hash[Object, ^(SolidQueue::Worker) -> void]
|
|
91
106
|
|
|
107
|
+
# @note
|
|
108
|
+
# - Bypasses ActiveRecord query cache for the given block.
|
|
109
|
+
# - When running under SolidQueue's executor, SELECT queries are cached
|
|
110
|
+
# for the entire job execution. Polling queries must bypass this cache
|
|
111
|
+
# to observe status changes made by other threads/processes.
|
|
112
|
+
#
|
|
113
|
+
# : [T] () { () -> T } -> T
|
|
114
|
+
def without_query_cache: [T] () { () -> T } -> T
|
|
115
|
+
|
|
92
116
|
# : (SolidQueue::Job, DSL, Numeric) -> bool
|
|
93
117
|
def reschedule_solid_queue_job: (SolidQueue::Job, DSL, Numeric) -> bool
|
|
94
118
|
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: job-workflow
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.3.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- shoma07
|
|
@@ -184,7 +184,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
184
184
|
- !ruby/object:Gem::Version
|
|
185
185
|
version: '0'
|
|
186
186
|
requirements: []
|
|
187
|
-
rubygems_version:
|
|
187
|
+
rubygems_version: 4.0.3
|
|
188
188
|
specification_version: 4
|
|
189
189
|
summary: Declarative workflow orchestration engine for Ruby on Rails
|
|
190
190
|
test_files: []
|