job-workflow 0.1.3 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +10 -0
- data/guides/API_REFERENCE.md +67 -0
- data/guides/PARALLEL_PROCESSING.md +62 -0
- data/lib/job_workflow/dsl.rb +45 -1
- data/lib/job_workflow/version.rb +1 -1
- data/sig/generated/job_workflow/dsl.rbs +32 -0
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 9ac9f67bfd9ba71b60f98643a5f5ed720423239650bdf80a8bb07de594d9b502
|
|
4
|
+
data.tar.gz: 6902b370e54d0285721390a40af5ff0a7b459c6ce0a67af6c538dc1877f0df85
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 6c24f513d3c1192513fef902a92d844050a25e095b2739e9f40e774639cde3fb75c4df1a6e2878a50f13583e5df0e665faa6ef803a434e9444d020a81ec2a71b
|
|
7
|
+
data.tar.gz: 1c7218d7a1c2d45444a111956bd26d152f16b2e1c6f86fb2959fd255c2ab844444d7929863149f3c7e4322073d520ff40ec3ea02ec29a1595135350723ed4a6a
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,15 @@
|
|
|
1
1
|
## [Unreleased]
|
|
2
2
|
|
|
3
|
+
## [0.2.0] - 2026-03-12
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
|
|
7
|
+
- Add `workflow_concurrency` DSL class method as a context-aware wrapper around SolidQueue's `limits_concurrency`. Unlike `limits_concurrency`, the key Proc receives a `Context` object, giving access to `ctx.arguments`, `ctx.sub_job?`, and `ctx.concurrency_key`. A fallback `Context` is built from job arguments when `_context` is not yet initialized (e.g. during enqueue before `perform`).
|
|
8
|
+
|
|
9
|
+
### Fixed
|
|
10
|
+
|
|
11
|
+
- Fix task-level concurrency key: internal `limits_concurrency` call for `enqueue: { concurrency: N }` tasks now goes through `workflow_concurrency`, ensuring the key Proc receives a proper `Context` instead of raw ActiveJob arguments. Also changed the internal key proc from `lambda` to `proc` for compatibility with SolidQueue's `instance_exec(*arguments, &proc)` call site.
|
|
12
|
+
|
|
3
13
|
## [0.1.3] - 2026-01-06
|
|
4
14
|
|
|
5
15
|
### Changed
|
data/guides/API_REFERENCE.md
CHANGED
|
@@ -84,6 +84,73 @@ end
|
|
|
84
84
|
|
|
85
85
|
**Map Task Output**: When `each:` is specified, outputs are automatically collected as an array.
|
|
86
86
|
|
|
87
|
+
### workflow_concurrency
|
|
88
|
+
|
|
89
|
+
Configure job-level concurrency limits with workflow-aware context.
|
|
90
|
+
|
|
91
|
+
```ruby
|
|
92
|
+
workflow_concurrency(to:, key:, **opts)
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
**Parameters**:
|
|
96
|
+
- `to` (Integer): Maximum number of concurrent executions
|
|
97
|
+
- `key` (Proc): A Proc that receives a `Context` and returns a String concurrency key
|
|
98
|
+
- `opts` (Hash): Additional options passed to SolidQueue's `limits_concurrency`
|
|
99
|
+
- `on_conflict` (Symbol): `:discard` to drop duplicate jobs (optional)
|
|
100
|
+
- `duration` (ActiveSupport::Duration): How long the concurrency lock is held (optional)
|
|
101
|
+
- `group` (String): Concurrency group name (optional)
|
|
102
|
+
|
|
103
|
+
Unlike SolidQueue's `limits_concurrency` (which passes raw ActiveJob arguments to the key Proc), `workflow_concurrency` passes a **Context** object, giving access to `arguments`, `sub_job?`, and `concurrency_key`.
|
|
104
|
+
|
|
105
|
+
**Example — simple per-tenant key**:
|
|
106
|
+
|
|
107
|
+
```ruby
|
|
108
|
+
class ImportJob < ApplicationJob
|
|
109
|
+
include JobWorkflow::DSL
|
|
110
|
+
|
|
111
|
+
argument :tenant_id, "Integer"
|
|
112
|
+
argument :items, "Array[Integer]"
|
|
113
|
+
|
|
114
|
+
workflow_concurrency to: 1,
|
|
115
|
+
key: ->(ctx) { "import:#{ctx.arguments.tenant_id}" },
|
|
116
|
+
on_conflict: :discard
|
|
117
|
+
|
|
118
|
+
task :process,
|
|
119
|
+
each: ->(ctx) { ctx.arguments.items },
|
|
120
|
+
enqueue: { concurrency: 5 },
|
|
121
|
+
output: { result: "String" } do |ctx|
|
|
122
|
+
{ result: handle(ctx.each_value) }
|
|
123
|
+
end
|
|
124
|
+
end
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
**Example — separating parent and sub-job keys**:
|
|
128
|
+
|
|
129
|
+
```ruby
|
|
130
|
+
class BatchImportJob < ApplicationJob
|
|
131
|
+
include JobWorkflow::DSL
|
|
132
|
+
|
|
133
|
+
argument :tenant_id, "Integer"
|
|
134
|
+
argument :items, "Array[Integer]"
|
|
135
|
+
|
|
136
|
+
workflow_concurrency to: 1,
|
|
137
|
+
key: lambda { |ctx|
|
|
138
|
+
ctx.sub_job? ? ctx.concurrency_key : "batch:#{ctx.arguments.tenant_id}"
|
|
139
|
+
},
|
|
140
|
+
on_conflict: :discard
|
|
141
|
+
|
|
142
|
+
task :process,
|
|
143
|
+
each: ->(ctx) { ctx.arguments.items },
|
|
144
|
+
enqueue: { concurrency: 5 },
|
|
145
|
+
output: { result: "String" } do |ctx|
|
|
146
|
+
{ result: handle(ctx.each_value) }
|
|
147
|
+
end
|
|
148
|
+
end
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
> **Note**: `workflow_concurrency` calls `limits_concurrency` internally. Calling it multiple times in the same class will **overwrite** the previous setting (last-wins). Define it once per job class.
|
|
152
|
+
> Requires SolidQueue.
|
|
153
|
+
|
|
87
154
|
**Example**:
|
|
88
155
|
|
|
89
156
|
```ruby
|
|
@@ -110,6 +110,68 @@ The `enqueue:` option determines how map task iterations are executed:
|
|
|
110
110
|
|
|
111
111
|
**Note**: `enqueue:` works with both regular tasks and map tasks. For map tasks, it enables asynchronous sub-job execution. For regular tasks, it allows conditional enqueueing as a separate job. Legacy syntax (`enqueue: ->(_ctx) { true }` as a Proc) is still supported for backward compatibility.
|
|
112
112
|
|
|
113
|
+
### Job-Level Concurrency with `workflow_concurrency`
|
|
114
|
+
|
|
115
|
+
When using SolidQueue's `limits_concurrency` directly, the key Proc receives raw ActiveJob arguments (a Hash), making it difficult to distinguish parent jobs from sub-jobs. `workflow_concurrency` is a wrapper that passes a **Context** object to the key Proc, giving access to workflow-aware information.
|
|
116
|
+
|
|
117
|
+
```ruby
|
|
118
|
+
class ImportJob < ApplicationJob
|
|
119
|
+
include JobWorkflow::DSL
|
|
120
|
+
|
|
121
|
+
argument :tenant_id, "Integer"
|
|
122
|
+
argument :items, "Array[Integer]"
|
|
123
|
+
|
|
124
|
+
# Limit to 1 concurrent execution per tenant.
|
|
125
|
+
# The key Proc receives a Context, not raw arguments.
|
|
126
|
+
workflow_concurrency to: 1,
|
|
127
|
+
key: ->(ctx) { "import:#{ctx.arguments.tenant_id}" },
|
|
128
|
+
on_conflict: :discard
|
|
129
|
+
|
|
130
|
+
task :process_items,
|
|
131
|
+
each: ->(ctx) { ctx.arguments.items },
|
|
132
|
+
enqueue: { concurrency: 5 },
|
|
133
|
+
output: { result: "String" } do |ctx|
|
|
134
|
+
{ result: process(ctx.each_value) }
|
|
135
|
+
end
|
|
136
|
+
end
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
**Key differences from `limits_concurrency`:**
|
|
140
|
+
|
|
141
|
+
| Feature | `limits_concurrency` (SolidQueue) | `workflow_concurrency` (JobWorkflow) |
|
|
142
|
+
|---|---|---|
|
|
143
|
+
| Key Proc argument | Raw Hash (ActiveJob arguments) | `Context` object |
|
|
144
|
+
| Access to `sub_job?` | ❌ Not available | ✅ `ctx.sub_job?` |
|
|
145
|
+
| Access to `concurrency_key` | ❌ Not available | ✅ `ctx.concurrency_key` |
|
|
146
|
+
| Access to `arguments` | Manual Hash parsing | ✅ `ctx.arguments.<name>` |
|
|
147
|
+
|
|
148
|
+
**Separating parent and sub-job concurrency keys:**
|
|
149
|
+
|
|
150
|
+
When a parent job enqueues sub-jobs (via `enqueue:` option), they share the same job class. With `limits_concurrency`, both parent and sub-jobs resolve to the same concurrency key, which can cause sub-jobs to be discarded. Use `workflow_concurrency` with `ctx.sub_job?` to generate distinct keys:
|
|
151
|
+
|
|
152
|
+
```ruby
|
|
153
|
+
workflow_concurrency to: 1,
|
|
154
|
+
key: lambda { |ctx|
|
|
155
|
+
if ctx.sub_job?
|
|
156
|
+
# Sub-jobs use a unique key per task iteration
|
|
157
|
+
ctx.concurrency_key
|
|
158
|
+
else
|
|
159
|
+
# Parent job uses a tenant-scoped key
|
|
160
|
+
"import:#{ctx.arguments.tenant_id}"
|
|
161
|
+
end
|
|
162
|
+
},
|
|
163
|
+
on_conflict: :discard
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
**Parameters:**
|
|
167
|
+
- `to:` (Integer): Maximum number of concurrent executions
|
|
168
|
+
- `key:` (Proc): A Proc that receives a `Context` and returns a String used as the concurrency key
|
|
169
|
+
- `on_conflict:` (Symbol, optional): `:discard` to drop duplicate jobs, or omit for default SolidQueue behavior
|
|
170
|
+
- `duration:` (ActiveSupport::Duration, optional): How long the concurrency lock is held
|
|
171
|
+
- `group:` (String, optional): Concurrency group name
|
|
172
|
+
|
|
173
|
+
> **Note**: `workflow_concurrency` requires SolidQueue. It delegates to `limits_concurrency` internally.
|
|
174
|
+
|
|
113
175
|
## Fork-Join Pattern
|
|
114
176
|
|
|
115
177
|
### Context Isolation
|
data/lib/job_workflow/dsl.rb
CHANGED
|
@@ -165,7 +165,7 @@ module JobWorkflow
|
|
|
165
165
|
_workflow.add_task(new_task)
|
|
166
166
|
if new_task.enqueue.should_limits_concurrency? # rubocop:disable Style/GuardClause
|
|
167
167
|
concurrency = new_task.enqueue.concurrency #: Integer
|
|
168
|
-
|
|
168
|
+
workflow_concurrency(to: concurrency, key: :concurrency_key.to_proc)
|
|
169
169
|
end
|
|
170
170
|
end
|
|
171
171
|
# rubocop:enable Metrics/ParameterLists
|
|
@@ -194,6 +194,50 @@ module JobWorkflow
|
|
|
194
194
|
_workflow.add_hook(:error, task_names:, block:)
|
|
195
195
|
end
|
|
196
196
|
|
|
197
|
+
# Configures concurrency limits for this workflow job.
|
|
198
|
+
#
|
|
199
|
+
# Unlike `limits_concurrency` (SolidQueue's raw API), this method passes a
|
|
200
|
+
# {Context} as the first argument to the key Proc, giving access to
|
|
201
|
+
# workflow-aware information such as `arguments`, `sub_job?`, and
|
|
202
|
+
# `concurrency_key`.
|
|
203
|
+
#
|
|
204
|
+
# When `_context` is not yet initialized (e.g. during enqueue before
|
|
205
|
+
# perform), a temporary Context is built from the job's arguments so the
|
|
206
|
+
# key Proc can always rely on `ctx.arguments`.
|
|
207
|
+
#
|
|
208
|
+
# @example Limit duplicate workflow runs by argument
|
|
209
|
+
# workflow_concurrency to: 1,
|
|
210
|
+
# key: ->(ctx) { "my_job:#{ctx.arguments.tenant_id}" },
|
|
211
|
+
# on_conflict: :discard
|
|
212
|
+
#
|
|
213
|
+
# @example Separate parent and sub-job concurrency keys
|
|
214
|
+
# workflow_concurrency to: 1,
|
|
215
|
+
# key: ->(ctx) {
|
|
216
|
+
# ctx.sub_job? ? ctx.concurrency_key : "my_job:#{ctx.arguments.name}"
|
|
217
|
+
# },
|
|
218
|
+
# on_conflict: :discard
|
|
219
|
+
#
|
|
220
|
+
#: (
|
|
221
|
+
# to: Integer,
|
|
222
|
+
# key: ^(Context) -> String?,
|
|
223
|
+
# ?duration: ActiveSupport::Duration?,
|
|
224
|
+
# ?group: String?,
|
|
225
|
+
# ?on_conflict: Symbol?
|
|
226
|
+
# ) -> void
|
|
227
|
+
def workflow_concurrency(to:, key:, **opts)
|
|
228
|
+
concurrency_key_proc = key
|
|
229
|
+
limits_concurrency(
|
|
230
|
+
to:,
|
|
231
|
+
key: proc {
|
|
232
|
+
ctx = _context || Context.from_hash(
|
|
233
|
+
job: self, workflow: self.class._workflow
|
|
234
|
+
)._update_arguments((arguments.first || {}).symbolize_keys)
|
|
235
|
+
concurrency_key_proc.call(ctx)
|
|
236
|
+
},
|
|
237
|
+
**opts
|
|
238
|
+
)
|
|
239
|
+
end
|
|
240
|
+
|
|
197
241
|
#: (?bool) ?{ (Context) -> bool } -> void
|
|
198
242
|
def dry_run(value = nil, &block)
|
|
199
243
|
validate_namespace!
|
data/lib/job_workflow/version.rb
CHANGED
|
@@ -94,6 +94,38 @@ module JobWorkflow
|
|
|
94
94
|
# : (*Symbol) { (Context, StandardError, Task) -> void } -> void
|
|
95
95
|
def on_error: (*Symbol) { (Context, StandardError, Task) -> void } -> void
|
|
96
96
|
|
|
97
|
+
# Configures concurrency limits for this workflow job.
|
|
98
|
+
#
|
|
99
|
+
# Unlike `limits_concurrency` (SolidQueue's raw API), this method passes a
|
|
100
|
+
# {Context} as the first argument to the key Proc, giving access to
|
|
101
|
+
# workflow-aware information such as `arguments`, `sub_job?`, and
|
|
102
|
+
# `concurrency_key`.
|
|
103
|
+
#
|
|
104
|
+
# When `_context` is not yet initialized (e.g. during enqueue before
|
|
105
|
+
# perform), a temporary Context is built from the job's arguments so the
|
|
106
|
+
# key Proc can always rely on `ctx.arguments`.
|
|
107
|
+
#
|
|
108
|
+
# @example Limit duplicate workflow runs by argument
|
|
109
|
+
# workflow_concurrency to: 1,
|
|
110
|
+
# key: ->(ctx) { "my_job:#{ctx.arguments.tenant_id}" },
|
|
111
|
+
# on_conflict: :discard
|
|
112
|
+
#
|
|
113
|
+
# @example Separate parent and sub-job concurrency keys
|
|
114
|
+
# workflow_concurrency to: 1,
|
|
115
|
+
# key: ->(ctx) {
|
|
116
|
+
# ctx.sub_job? ? ctx.concurrency_key : "my_job:#{ctx.arguments.name}"
|
|
117
|
+
# },
|
|
118
|
+
# on_conflict: :discard
|
|
119
|
+
#
|
|
120
|
+
# : (
|
|
121
|
+
# to: Integer,
|
|
122
|
+
# key: ^(Context) -> String?,
|
|
123
|
+
# ?duration: ActiveSupport::Duration?,
|
|
124
|
+
# ?group: String?,
|
|
125
|
+
# ?on_conflict: Symbol?
|
|
126
|
+
# ) -> void
|
|
127
|
+
def workflow_concurrency: (to: Integer, key: ^(Context) -> String?, ?duration: ActiveSupport::Duration?, ?group: String?, ?on_conflict: Symbol?) -> void
|
|
128
|
+
|
|
97
129
|
# : (?bool) ?{ (Context) -> bool } -> void
|
|
98
130
|
def dry_run: (?bool) ?{ (Context) -> bool } -> void
|
|
99
131
|
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: job-workflow
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.2.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- shoma07
|
|
@@ -184,7 +184,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
184
184
|
- !ruby/object:Gem::Version
|
|
185
185
|
version: '0'
|
|
186
186
|
requirements: []
|
|
187
|
-
rubygems_version:
|
|
187
|
+
rubygems_version: 4.0.3
|
|
188
188
|
specification_version: 4
|
|
189
189
|
summary: Declarative workflow orchestration engine for Ruby on Rails
|
|
190
190
|
test_files: []
|