geneva_drive 0.6.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +3 -1
- data/MANUAL.md +176 -0
- data/README.md +2 -0
- data/lib/generators/geneva_drive/install/install_generator.rb +5 -0
- data/lib/generators/geneva_drive/install/templates/add_signals_support.rb +94 -0
- data/lib/geneva_drive/executor.rb +113 -10
- data/lib/geneva_drive/flow_control.rb +43 -3
- data/lib/geneva_drive/jobs/housekeeping_job.rb +83 -3
- data/lib/geneva_drive/signal.rb +243 -0
- data/lib/geneva_drive/signal_matcher.rb +60 -0
- data/lib/geneva_drive/step_definition.rb +82 -1
- data/lib/geneva_drive/step_execution.rb +36 -0
- data/lib/geneva_drive/test_helpers.rb +89 -2
- data/lib/geneva_drive/version.rb +1 -1
- data/lib/geneva_drive/workflow.rb +327 -22
- data/lib/geneva_drive.rb +22 -0
- data/test/dsl/signal_step_definition_test.rb +132 -0
- data/test/jobs/housekeeping_signals_test.rb +198 -0
- data/test/migration_helpers_test.rb +1 -1
- data/test/test_helper.rb +2 -0
- data/test/workflow/signal_consumption_test.rb +397 -0
- data/test/workflow/signal_delivery_test.rb +255 -0
- data/test/workflow/signal_durability_test.rb +164 -0
- data/test/workflow/signal_rendezvous_test.rb +581 -0
- data/test/workflow/signals_without_migration_test.rb +97 -0
- metadata +11 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 1738e58c5f40199781a27c2345b20657e63ccc0c5a8b28010c2ac820774c2c2f
|
|
4
|
+
data.tar.gz: f729ef68e929f39fa26b09108f446a9b53ec13013d22d89f22d788b5ac0988e3
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 87a29981a0278755e146cebf6e5e8d7075832665335b54511d5c24734a44040c62e9b2f9ae8fcd2558941d5f8cfe9eefc588efef65db09dc131d6602e2089d40
|
|
7
|
+
data.tar.gz: 9dcbb2ca84072315438dcacfd78d5463563b0ef6f5a940a5ccd12afcef7d2f10fb40cd710ce4bd5ba72142941242ec709f92e4d33927d15ce146538eed93834f
|
data/CHANGELOG.md
CHANGED
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
-
## [
|
|
3
|
+
## [0.7.0]
|
|
4
|
+
|
|
5
|
+
- Add signals: external events delivered to a workflow instance with `workflow.signal!(:payment_confirmed, payload: {...}, idempotency_key: event["id"])`, and a step primitive that parks until one arrives with `step :capture, wait_for: :payment_confirmed`. Signals are rows in a new `geneva_drive_signals` table, persisted before any processing, so arrival order does not matter: a signal that lands before its waiting step exists is buffered and claimed when the step runs, and one that lands after the step parked wakes it immediately. Both sides of the rendezvous run under the workflow row lock, so no signal can slip through the gap, and delivery is one transaction: the insert, the waking of parked executions and every counter bump land together or not at all, so a crashed `signal!` leaves nothing half-delivered (the job enqueue stays after-commit, with the existing stuck-scheduled housekeeping sweep as its backstop). Waiting is a first-class step execution state (`waiting`) that holds no queue slot and no timer — nothing polls. Matching is by name - `wait_for:` also takes a `GenevaDrive::SignalMatcher`, whose optional block narrows by payload and is evaluated on the workflow the signal was sent to (so `hero` is in scope), or any other object responding to `#matches?(signal)`. Matchers are plain reusable objects, so a rule can live in a constant and be shared across steps. Signals move `pending` → `claimed` → `consumed`, consumed only once every attached execution chain has finished cleanly — where skipping counts as finishing, so any flavor of skip (`skip_if:` at wake, `skip!` in a step, an exception policy, or an operator `skip!` on a ready or paused workflow) settles the signal that step was delivered, while skips of steps that never attached leave buffered signals alone — and in the same transaction as the step's own completion, so a reattempt or a `resume!` re-reads the same payload and the audit trail never claims an event was handled when it was not. Two counters on the row, `claimed` and `consumed`, count attachments and clean resolutions (with `claimed?` / `consumed?` reading those counters rather than the lifecycle state). Redelivery is deduplicated by a unique index on `(workflow_id, name, idempotency_key)` — the duplicate call returns the original row flagged `duplicate_delivery?` without dispatching. Payloads go through ActiveJob serializers, come back with indifferent access, and are bounded by `GenevaDrive.max_signal_payload_size` (128 KB, `nil` disables). `signal!` on a finished or canceled workflow raises `GenevaDrive::WorkflowNotOngoing` unless it is a redelivery of a known event; on a paused workflow the row is buffered and delivered by `resume!`. Timeouts are deliberately out of scope for now; waiting is instead made loud through `waiting_since` and two housekeeping gauges (`geneva_drive.waiting_step_executions`, `geneva_drive.waiting_overdue` past `GenevaDrive.waiting_visibility_threshold`, 7 days by default), with `skip!` and `cancel!` as operator escape hatches. Needs one new installer migration (the signals table plus two columns on step executions); until it runs everything else is unaffected and executing a `wait_for:` step raises a configuration error pointing at the missing migration. Test helpers: `assert_waiting_for_signal`, `assert_signal_state`, and the step-driving helpers now raise instead of spinning on a parked execution. Mixing up `wait:` and `wait_for:` now raises `StepConfigurationError` at class load in both directions, which also tightens `wait:` to reject Strings (previously `wait: "later"` silently meant zero seconds).
|
|
4
6
|
|
|
5
7
|
## [0.6.0]
|
|
6
8
|
|
data/MANUAL.md
CHANGED
|
@@ -679,6 +679,179 @@ test "keeps its place across interruptions" do
|
|
|
679
679
|
end
|
|
680
680
|
```
|
|
681
681
|
|
|
682
|
+
## Waiting for Signals
|
|
683
|
+
|
|
684
|
+
Some steps cannot finish on their own. A payment needs the provider's webhook, a contract needs a countersignature, a shipment needs the carrier to scan the parcel. The usual Rails answer is to poll — schedule the step, check whether the thing happened, `reattempt!(wait: 5.minutes)` if it did not — and that works, but it burns a worker slot on every check and it puts a floor under how fast the workflow can react.
|
|
685
|
+
|
|
686
|
+
A **signal** inverts that. It is an event record addressed to one workflow instance:
|
|
687
|
+
|
|
688
|
+
```ruby
|
|
689
|
+
# app/controllers/payment_webhooks_controller.rb
|
|
690
|
+
workflow = OrderFulfillmentWorkflow.ongoing.for_hero(order).first
|
|
691
|
+
workflow.signal!(:payment_confirmed,
|
|
692
|
+
payload: {amount_cents: event["amount"]},
|
|
693
|
+
idempotency_key: event["id"])
|
|
694
|
+
```
|
|
695
|
+
|
|
696
|
+
And a step declares that it waits for one with `wait_for:`:
|
|
697
|
+
|
|
698
|
+
```ruby
|
|
699
|
+
class OrderFulfillmentWorkflow < GenevaDrive::Workflow
|
|
700
|
+
step :reserve_stock do
|
|
701
|
+
hero.reserve_stock!
|
|
702
|
+
end
|
|
703
|
+
|
|
704
|
+
step :capture_payment, wait_for: :payment_confirmed do
|
|
705
|
+
hero.capture!(received_signal.payload[:amount_cents])
|
|
706
|
+
end
|
|
707
|
+
|
|
708
|
+
step :ship do
|
|
709
|
+
ShippingLabel.create!(order: hero)
|
|
710
|
+
end
|
|
711
|
+
end
|
|
712
|
+
```
|
|
713
|
+
|
|
714
|
+
When `:capture_payment` runs and no matching signal has arrived, its execution parks in the `waiting` state and the job ends. Nothing is enqueued, nothing polls, and the parked row costs nothing until `signal!` wakes it. When the webhook lands, `signal!` wakes the execution and enqueues its job again.
|
|
715
|
+
|
|
716
|
+
Signals need one migration — it creates the `geneva_drive_signals` table and adds two columns to `geneva_drive_step_executions` — so re-run `bin/rails generate geneva_drive:install` on an existing installation to pick it up. Until then everything else keeps working, and executing a step that declares `wait_for:` fails with a configuration error pointing at the missing migration.
|
|
717
|
+
|
|
718
|
+
### Arrival Order Does Not Matter
|
|
719
|
+
|
|
720
|
+
The signal row is written before anything is dispatched, which is what makes the timing irrelevant. There are exactly two places where sender and receiver meet, and both run under the workflow row lock:
|
|
721
|
+
|
|
722
|
+
- When the waiting step's job runs, it looks for a matching unconsumed signal. Found: attach and run. Not found: park.
|
|
723
|
+
- When `signal!` is called, it looks for parked executions. Found: attach, reschedule, enqueue. Not found: the row simply sits there.
|
|
724
|
+
|
|
725
|
+
So a webhook that arrives while the workflow is still three steps away from the waiter is not lost and not early — it is buffered, and claimed when the waiting step finally runs. A webhook that arrives a week after the step parked wakes it immediately.
|
|
726
|
+
|
|
727
|
+
Delivery is one database transaction: the signal row, the waking of the parked execution and the bookkeeping around it either all land or none of them do. If `signal!` raises, nothing was written and the sender can simply try again.
|
|
728
|
+
|
|
729
|
+
> [!IMPORTANT]
|
|
730
|
+
> `signal!` is an instance method: you find the workflow the way you find anything in Rails, usually `SomeWorkflow.ongoing.for_hero(record).first`. Delivering a brand-new event to a workflow that has already finished or been canceled raises `GenevaDrive::WorkflowNotOngoing` rather than silently doing nothing.
|
|
731
|
+
|
|
732
|
+
### Deduplicating Redelivery
|
|
733
|
+
|
|
734
|
+
Webhook providers redeliver. Pass an `idempotency_key:` — the provider's event id is the natural choice — and the second delivery becomes a database-level no-op that returns the row the first one wrote:
|
|
735
|
+
|
|
736
|
+
```ruby
|
|
737
|
+
signal = workflow.signal!(:payment_confirmed, idempotency_key: event["id"])
|
|
738
|
+
Rails.logger.info("Ignoring redelivered #{event["id"]}") if signal.duplicate_delivery?
|
|
739
|
+
```
|
|
740
|
+
|
|
741
|
+
Redelivering the very event that finished the workflow is also a no-op, not an error — that specific case is what the deduplication is for. Signals sent without an idempotency key never deduplicate: two calls make two rows, the older one is delivered first, and the newer one stays in the table as an auditable record of what arrived.
|
|
742
|
+
|
|
743
|
+
### Narrowing What Counts as a Match
|
|
744
|
+
|
|
745
|
+
`wait_for: :document_signed` is shorthand for `wait_for: GenevaDrive::SignalMatcher.new(:document_signed)`, which matches on name alone. Give the matcher a block and it also has to like the payload:
|
|
746
|
+
|
|
747
|
+
```ruby
|
|
748
|
+
class ContractWorkflow < GenevaDrive::Workflow
|
|
749
|
+
# The block receives the payload and runs on the workflow the signal was
|
|
750
|
+
# sent to, so `hero` and the workflow's own methods are in scope.
|
|
751
|
+
step :await_countersignature,
|
|
752
|
+
wait_for: GenevaDrive::SignalMatcher.new(:document_signed) { |payload|
|
|
753
|
+
payload[:document_id] == hero.contract_id
|
|
754
|
+
} do
|
|
755
|
+
hero.mark_countersigned!
|
|
756
|
+
end
|
|
757
|
+
end
|
|
758
|
+
```
|
|
759
|
+
|
|
760
|
+
A matcher is a plain object with no ties to the step that uses it, so a rule worth repeating can live in a constant and be shared across steps and workflows:
|
|
761
|
+
|
|
762
|
+
```ruby
|
|
763
|
+
SETTLED = GenevaDrive::SignalMatcher.new(:payment_confirmed) { |payload| payload[:amount_cents].to_i > 0 }
|
|
764
|
+
|
|
765
|
+
step :capture_payment, wait_for: SETTLED do
|
|
766
|
+
hero.capture!
|
|
767
|
+
end
|
|
768
|
+
```
|
|
769
|
+
|
|
770
|
+
Anything responding to `#matches?(signal)` works too, and owns the whole predicate — which is how a single step waits on either of two names:
|
|
771
|
+
|
|
772
|
+
```ruby
|
|
773
|
+
class PaymentSettled
|
|
774
|
+
def initialize(min_cents:) = @min_cents = min_cents
|
|
775
|
+
|
|
776
|
+
def matches?(signal)
|
|
777
|
+
%w[payment_confirmed payment_captured].include?(signal.name) &&
|
|
778
|
+
signal.payload[:amount_cents].to_i >= @min_cents
|
|
779
|
+
end
|
|
780
|
+
end
|
|
781
|
+
|
|
782
|
+
step :capture_payment, wait_for: PaymentSettled.new(min_cents: 100) do
|
|
783
|
+
hero.capture!
|
|
784
|
+
end
|
|
785
|
+
```
|
|
786
|
+
|
|
787
|
+
Payloads are serialized with ActiveJob serializers, so `Date`, `Time` and ActiveRecord objects survive the round trip. Hashes come back with indifferent access, because a webhook sender produces string keys and a Ruby caller produces symbols and a matcher should not have to care. The serialized payload is limited to 128 KB (`GenevaDrive.max_signal_payload_size`, `nil` disables) — a payload describes the event; the data your workflow works on belongs on the hero.
|
|
788
|
+
|
|
789
|
+
> [!WARNING]
|
|
790
|
+
> Matchers run at the gate *and* inside `signal!`, in the sender's process. Keep them cheap and free of side effects, exactly like `skip_if:`. A matcher that raises during `signal!` raises to whoever sent the signal — and because `signal!` is a single transaction, the delivery rolls back whole: no signal row is recorded, nothing is attached or woken, no counter moves. The sender (or the webhook provider's retry) re-delivers once the matcher is fixed.
|
|
791
|
+
|
|
792
|
+
### How Waiting Composes
|
|
793
|
+
|
|
794
|
+
- `wait: 2.days, wait_for: :payment_confirmed` means "no earlier than two days from now, and only once signaled". A signal arriving during the delay is buffered and claimed when the step finally runs.
|
|
795
|
+
- `skip_if:` is evaluated before waiting, so "wait for the signature — unless the contract was pre-signed" skips immediately instead of parking forever. Both `skip_if:` and `cancel_if` are re-evaluated when a signal wakes the step; a parked workflow does not re-check them while it sleeps. A step skipped at wake consumes the signal that woke it, so the next waiter does not inherit a stale event.
|
|
796
|
+
- `resumable_step` accepts `wait_for:`. The wait happens once, at the start of the chain, and `received_signal` stays the same across every successor execution.
|
|
797
|
+
- A reattempt — yours or an exception policy's — re-reads the *same* signal. Reattempting means "process this event again", not "wait for another event".
|
|
798
|
+
|
|
799
|
+
The signal itself moves through `pending` → `claimed` → `consumed`. It is claimed when an execution attaches to it, and consumed once every attached execution has finished cleanly, in the same transaction as the step's own completion. A step that fails, pauses, or gets canceled leaves its signal claimed, so the retry picks up the same payload and the audit trail does not claim an event was handled when it was not.
|
|
800
|
+
|
|
801
|
+
Skipping counts as finishing: skipping a step that was delivered a signal consumes that signal, whether the skip came from `skip_if:`, from `skip!` inside the step, from an exception policy, or from an operator calling `workflow.skip!` on a ready or paused workflow. Moving past a step deliberately settles the event it was given. Skipping a step that never attached — one still parked, or one whose `skip_if:` fired before it ever looked for a signal — leaves buffered signals exactly as they were.
|
|
802
|
+
|
|
803
|
+
Two counters on the row make that legible: `signal.claimed` counts attachments (a step that failed and was retried attaches twice), and `signal.consumed` counts cleanly resolved ones. In a linear workflow they end at one apiece. The predicates `signal.claimed?` and `signal.consumed?` read those counters, so they answer "was this event ever picked up" and "was it ever handled" — which is not the same question as `signal.state`, and is usually the one worth asking.
|
|
804
|
+
|
|
805
|
+
> [!NOTE]
|
|
806
|
+
> Two sequential steps waiting on `:payment_confirmed` need two signals. The first step consumes the first signal, so the second step parks until another one arrives.
|
|
807
|
+
|
|
808
|
+
### Getting Unstuck
|
|
809
|
+
|
|
810
|
+
There is no `timeout:` option. Waiting forever is a legitimate state — a contract may genuinely sit unsigned for months — so instead of a timer, waiting is made loud: a distinct `waiting` state, a `waiting_since` timestamp, and two housekeeping gauges, `geneva_drive.waiting_step_executions` and `geneva_drive.waiting_overdue` (parked longer than `GenevaDrive.waiting_visibility_threshold`, seven days by default). Alert on the overdue gauge.
|
|
811
|
+
|
|
812
|
+
When something does stall, three escape hatches already exist:
|
|
813
|
+
|
|
814
|
+
- `workflow.skip!` abandons the wait and moves to the next step.
|
|
815
|
+
- `workflow.cancel!` gives up on the workflow.
|
|
816
|
+
- The queue is already a clock: schedule a job that signals a deadline, and let the step decide.
|
|
817
|
+
|
|
818
|
+
```ruby
|
|
819
|
+
# Send the workflow its own deadline; the step reads it off the payload
|
|
820
|
+
class ContractDeadlineJob < ApplicationJob
|
|
821
|
+
def perform(contract)
|
|
822
|
+
workflow = ContractWorkflow.ongoing.for_hero(contract).first
|
|
823
|
+
workflow&.signal!(:document_signed,
|
|
824
|
+
payload: {timed_out: true},
|
|
825
|
+
idempotency_key: "deadline-#{workflow.id}")
|
|
826
|
+
end
|
|
827
|
+
end
|
|
828
|
+
|
|
829
|
+
step :await_countersignature, wait_for: :document_signed do
|
|
830
|
+
cancel! if received_signal.payload[:timed_out]
|
|
831
|
+
hero.mark_countersigned!
|
|
832
|
+
end
|
|
833
|
+
```
|
|
834
|
+
|
|
835
|
+
### Testing Waiting Steps
|
|
836
|
+
|
|
837
|
+
The step-driving helpers refuse to spin on a parked execution — they raise and tell you which signal is missing, because a test that hangs on a rendezvous is far worse than one that fails:
|
|
838
|
+
|
|
839
|
+
```ruby
|
|
840
|
+
test "captures once the payment is confirmed" do
|
|
841
|
+
workflow = OrderFulfillmentWorkflow.create!(hero: order)
|
|
842
|
+
perform_next_step(workflow) # :reserve_stock
|
|
843
|
+
perform_next_step(workflow) # :capture_payment parks
|
|
844
|
+
|
|
845
|
+
assert_waiting_for_signal(workflow, :payment_confirmed)
|
|
846
|
+
|
|
847
|
+
workflow.signal!(:payment_confirmed, payload: {amount_cents: 12_500})
|
|
848
|
+
speedrun_workflow(workflow)
|
|
849
|
+
|
|
850
|
+
assert_signal_state(workflow, :payment_confirmed, :consumed)
|
|
851
|
+
assert workflow.finished?
|
|
852
|
+
end
|
|
853
|
+
```
|
|
854
|
+
|
|
682
855
|
## Exception Handling
|
|
683
856
|
|
|
684
857
|
### Default Behavior
|
|
@@ -1080,6 +1253,7 @@ Step executions have their own state machine:
|
|
|
1080
1253
|
| State | Meaning |
|
|
1081
1254
|
|-------|---------|
|
|
1082
1255
|
| `scheduled` | Waiting to run |
|
|
1256
|
+
| `waiting` | Parked until a matching signal arrives |
|
|
1083
1257
|
| `in_progress` | Currently executing |
|
|
1084
1258
|
| `completed` | Finished successfully |
|
|
1085
1259
|
| `failed` | Exception occurred |
|
|
@@ -1980,6 +2154,7 @@ end
|
|
|
1980
2154
|
| State | Meaning |
|
|
1981
2155
|
|-------|---------|
|
|
1982
2156
|
| `scheduled` | Waiting to run |
|
|
2157
|
+
| `waiting` | Parked until a matching signal arrives |
|
|
1983
2158
|
| `in_progress` | Currently executing |
|
|
1984
2159
|
| `completed` | Finished successfully |
|
|
1985
2160
|
| `failed` | Exception occurred |
|
|
@@ -1993,6 +2168,7 @@ A resumable step interrupted mid-iteration completes its execution with outcome
|
|
|
1993
2168
|
| Option | Type | Description |
|
|
1994
2169
|
|--------|------|-------------|
|
|
1995
2170
|
| `wait:` | Duration | Delay before step executes |
|
|
2171
|
+
| `wait_for:` | Symbol, String, `SignalMatcher`, matcher object | Park until a matching signal arrives |
|
|
1996
2172
|
| `job_options:` | Hash | Options passed to Active Job's `set` method for this step |
|
|
1997
2173
|
| `skip_if:` | Proc, Symbol, Boolean | Condition to skip step |
|
|
1998
2174
|
| `on_exception:` | Symbol | Exception handler (`:pause!`, `:cancel!`, `:reattempt!`, `:skip!`) |
|
data/README.md
CHANGED
|
@@ -8,6 +8,8 @@ GenevaDrive provides a clean DSL for defining multi-step workflows that execute
|
|
|
8
8
|
|
|
9
9
|
- **Hero-oriented**: Workflows are associated with a polymorphic "hero" (the subject of the workflow)
|
|
10
10
|
- **Step-based execution**: Define workflows as a series of steps with optional wait times
|
|
11
|
+
- **Resumable steps**: Iterate over large collections with a database-checkpointed cursor, surviving interruptions
|
|
12
|
+
- **Signals**: Steps can park until an external event arrives (`wait_for:`), woken by `workflow.signal!` from a webhook or controller
|
|
11
13
|
- **Durable**: Steps are persisted to the database, surviving process restarts
|
|
12
14
|
- **Idempotent**: Database constraints ensure each step runs exactly once
|
|
13
15
|
- **Flow control**: Methods like `cancel!`, `pause!`, `skip!`, `reattempt!`, and `finished!`
|
|
@@ -65,6 +65,11 @@ module GenevaDrive
|
|
|
65
65
|
"add_resumable_step_support.rb",
|
|
66
66
|
"db/migrate/add_resumable_step_support_to_geneva_drive_step_executions.rb"
|
|
67
67
|
)
|
|
68
|
+
|
|
69
|
+
migration_template(
|
|
70
|
+
"add_signals_support.rb",
|
|
71
|
+
"db/migrate/add_signals_support_to_geneva_drive.rb"
|
|
72
|
+
)
|
|
68
73
|
end
|
|
69
74
|
|
|
70
75
|
# Creates the initializer file.
|
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# Signals: external events delivered to a workflow, and the two columns on
|
|
4
|
+
# step executions that let one park until a matching event arrives. Both halves
|
|
5
|
+
# ship together - a signals table nobody can wait on is useless, and a waiting
|
|
6
|
+
# step with nowhere to read its event from is worse - so they are one migration.
|
|
7
|
+
class AddSignalsSupportToGenevaDrive < ActiveRecord::Migration[7.2]
|
|
8
|
+
include GenevaDrive::MigrationHelpers
|
|
9
|
+
|
|
10
|
+
def change
|
|
11
|
+
key_type = geneva_drive_key_type
|
|
12
|
+
adapter = connection.adapter_name.downcase
|
|
13
|
+
|
|
14
|
+
# Build reference options - we add the foreign key separately to avoid
|
|
15
|
+
# MySQL type mismatch (see below).
|
|
16
|
+
reference_options = {
|
|
17
|
+
null: false,
|
|
18
|
+
index: true
|
|
19
|
+
}
|
|
20
|
+
reference_options[:type] = key_type if key_type == :uuid
|
|
21
|
+
|
|
22
|
+
create_table :geneva_drive_signals, **geneva_drive_table_options do |t|
|
|
23
|
+
# Link to workflow (cascade delete when the workflow is deleted)
|
|
24
|
+
t.references :workflow, **reference_options
|
|
25
|
+
|
|
26
|
+
# The event name, as passed to Workflow#signal!
|
|
27
|
+
t.string :name, null: false
|
|
28
|
+
|
|
29
|
+
# Optional caller-supplied deduplication key. NULLs are distinct in
|
|
30
|
+
# unique indexes on PostgreSQL, MySQL and SQLite alike, so signals
|
|
31
|
+
# without an idempotency key always insert.
|
|
32
|
+
t.string :idempotency_key
|
|
33
|
+
|
|
34
|
+
# Lifecycle: pending -> claimed -> consumed
|
|
35
|
+
t.string :state, null: false, default: "pending"
|
|
36
|
+
|
|
37
|
+
# Serialized payload. Use the database-native JSON type, same flavor
|
|
38
|
+
# logic as the resumable step cursor:
|
|
39
|
+
# - PostgreSQL: jsonb
|
|
40
|
+
# - MySQL 5.7+: json
|
|
41
|
+
# - SQLite: json (Rails handles as TEXT with serialization)
|
|
42
|
+
if adapter.include?("postgresql")
|
|
43
|
+
t.jsonb :payload
|
|
44
|
+
else
|
|
45
|
+
t.json :payload
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
t.datetime :claimed_at
|
|
49
|
+
t.datetime :consumed_at
|
|
50
|
+
|
|
51
|
+
# How many executions have attached to this signal, and how many of
|
|
52
|
+
# those attached chains have resolved cleanly. For a linear workflow
|
|
53
|
+
# both end at 1 (more claims if the step was retried); when one signal
|
|
54
|
+
# is dispatched onto several concurrent executions the pair is the
|
|
55
|
+
# progress readout of that fan-out. Counters, not the lifecycle state -
|
|
56
|
+
# the `state` column above is what says claimed or consumed.
|
|
57
|
+
t.bigint :claimed, null: false, default: 0
|
|
58
|
+
t.bigint :consumed, null: false, default: 0
|
|
59
|
+
|
|
60
|
+
t.timestamps
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
# Deduplication: one signal per (workflow, name, idempotency_key)
|
|
64
|
+
add_index :geneva_drive_signals, [:workflow_id, :name, :idempotency_key],
|
|
65
|
+
unique: true, name: "index_geneva_drive_signals_dedup"
|
|
66
|
+
|
|
67
|
+
# Scanning a workflow's unconsumed signals
|
|
68
|
+
add_index :geneva_drive_signals, [:workflow_id, :state]
|
|
69
|
+
|
|
70
|
+
# Add foreign key separately to avoid MySQL type mismatch (UNSIGNED vs SIGNED bigint)
|
|
71
|
+
# MySQL creates primary keys as UNSIGNED but references as SIGNED, causing FK constraint failure
|
|
72
|
+
unless adapter.include?("mysql")
|
|
73
|
+
add_foreign_key :geneva_drive_signals, :geneva_drive_workflows,
|
|
74
|
+
column: :workflow_id, on_delete: :cascade
|
|
75
|
+
end
|
|
76
|
+
|
|
77
|
+
unless column_exists?(:geneva_drive_step_executions, :signal_id)
|
|
78
|
+
# The attachment pin: which signal this execution is processing. One
|
|
79
|
+
# signal may be pinned by many executions, so this lives on the
|
|
80
|
+
# execution side. Match the primary key type (bigint or uuid) of the
|
|
81
|
+
# signals table.
|
|
82
|
+
# No foreign key constraint - SQLite rewrites the table on
|
|
83
|
+
# add_foreign_key, which can destroy data.
|
|
84
|
+
add_column :geneva_drive_step_executions, :signal_id, key_type
|
|
85
|
+
add_index :geneva_drive_step_executions, :signal_id
|
|
86
|
+
end
|
|
87
|
+
|
|
88
|
+
unless column_exists?(:geneva_drive_step_executions, :waiting_since)
|
|
89
|
+
# When the execution parked waiting for a signal, so "waiting for N
|
|
90
|
+
# days" is queryable without abusing updated_at.
|
|
91
|
+
add_column :geneva_drive_step_executions, :waiting_since, :datetime
|
|
92
|
+
end
|
|
93
|
+
end
|
|
94
|
+
end
|
|
@@ -15,7 +15,8 @@
|
|
|
15
15
|
class GenevaDrive::Executor
|
|
16
16
|
# Valid state transitions for step executions
|
|
17
17
|
STEP_TRANSITIONS = {
|
|
18
|
-
"scheduled" => %w[scheduled in_progress canceled skipped failed completed],
|
|
18
|
+
"scheduled" => %w[scheduled waiting in_progress canceled skipped failed completed],
|
|
19
|
+
"waiting" => %w[scheduled canceled skipped],
|
|
19
20
|
"in_progress" => %w[in_progress completed failed canceled skipped]
|
|
20
21
|
}.freeze
|
|
21
22
|
|
|
@@ -113,17 +114,23 @@ class GenevaDrive::Executor
|
|
|
113
114
|
@step_definition = step_def
|
|
114
115
|
@start_time = Time.current
|
|
115
116
|
|
|
116
|
-
# Phase 2: Execute step block (locks released)
|
|
117
|
-
|
|
118
|
-
|
|
117
|
+
# Phase 2: Execute step block (locks released). The signal this execution
|
|
118
|
+
# is attached to (if any) is injected for the duration, so step code can
|
|
119
|
+
# read `received_signal`.
|
|
120
|
+
flow_result = @workflow.with_received_signal(@received_signal) do
|
|
121
|
+
@logger.debug("Running before_step_execution hook")
|
|
122
|
+
@workflow.before_step_execution(@step_execution)
|
|
119
123
|
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
+
@logger.debug("Running the actual step code")
|
|
125
|
+
result = @workflow.around_step_execution(@step_execution) do
|
|
126
|
+
execute_step(step_def)
|
|
127
|
+
end
|
|
128
|
+
|
|
129
|
+
@logger.debug("Running after_step_execution hook")
|
|
130
|
+
@workflow.after_step_execution(@step_execution)
|
|
124
131
|
|
|
125
|
-
|
|
126
|
-
|
|
132
|
+
result
|
|
133
|
+
end
|
|
127
134
|
|
|
128
135
|
# Phase 3: Acquire locks and handle result
|
|
129
136
|
@logger.info("Finished step with outcome #{flow_result.inspect}")
|
|
@@ -336,6 +343,22 @@ class GenevaDrive::Executor
|
|
|
336
343
|
next nil
|
|
337
344
|
end
|
|
338
345
|
|
|
346
|
+
# Steps that wait for a signal need the signal_id and waiting_since
|
|
347
|
+
# columns. Fail loudly (instead of degrading) - without them a step
|
|
348
|
+
# would run without the event it was written to react to.
|
|
349
|
+
if step_def.waits_for_signal? && !GenevaDrive::StepExecution.signal_columns?
|
|
350
|
+
error_message = "Step '#{step_execution.step_name}' declares wait_for:, but the signal_id and " \
|
|
351
|
+
"waiting_since columns are missing from geneva_drive_step_executions. " \
|
|
352
|
+
"Run `bin/rails generate geneva_drive:install` and migrate."
|
|
353
|
+
logger.error(error_message)
|
|
354
|
+
|
|
355
|
+
step_execution.update!(error_message: error_message)
|
|
356
|
+
transition_step!("failed", outcome: "failed")
|
|
357
|
+
transition_workflow!("paused")
|
|
358
|
+
exception_to_raise = GenevaDrive::StepConfigurationError.new(error_message)
|
|
359
|
+
next nil
|
|
360
|
+
end
|
|
361
|
+
|
|
339
362
|
# Evaluate preconditions with instrumentation
|
|
340
363
|
precondition_result = evaluate_preconditions(step_def)
|
|
341
364
|
if precondition_result[:abort]
|
|
@@ -343,6 +366,13 @@ class GenevaDrive::Executor
|
|
|
343
366
|
next nil
|
|
344
367
|
end
|
|
345
368
|
|
|
369
|
+
# The rendezvous, receiver side: skip_if has had its say (skip beats
|
|
370
|
+
# wait), so now find the signal or park. Parking ends the job without
|
|
371
|
+
# enqueueing anything - dispatch will wake us.
|
|
372
|
+
if step_def.waits_for_signal? && run_signal_gate(step_def) == :parked
|
|
373
|
+
next nil
|
|
374
|
+
end
|
|
375
|
+
|
|
346
376
|
# All checks passed - transition to in_progress
|
|
347
377
|
logger.debug("Step may be performed - changing state flags and proceeding to perform")
|
|
348
378
|
|
|
@@ -367,6 +397,69 @@ class GenevaDrive::Executor
|
|
|
367
397
|
result
|
|
368
398
|
end
|
|
369
399
|
|
|
400
|
+
# The receiver side of the rendezvous. Runs under the workflow and step
|
|
401
|
+
# execution locks, inside prepare_execution.
|
|
402
|
+
#
|
|
403
|
+
# An execution that already carries an attachment pin (dispatch woke it, or
|
|
404
|
+
# it inherited the pin from its predecessor) runs straight away. Otherwise
|
|
405
|
+
# the oldest matching non-consumed signal is attached, and failing that the
|
|
406
|
+
# execution parks.
|
|
407
|
+
#
|
|
408
|
+
# @param step_def [StepDefinition] the waiting step's definition
|
|
409
|
+
# @return [Symbol] :attached or :parked
|
|
410
|
+
def run_signal_gate(step_def)
|
|
411
|
+
if step_execution.signal_id.present?
|
|
412
|
+
@received_signal = step_execution.signal
|
|
413
|
+
logger.info("Step is attached to signal #{step_execution.signal_id}, proceeding")
|
|
414
|
+
return :attached
|
|
415
|
+
end
|
|
416
|
+
|
|
417
|
+
matcher = step_def.signal_matcher
|
|
418
|
+
candidate = workflow.attachable_signals.detect { |signal| matcher.matches?(signal) }
|
|
419
|
+
|
|
420
|
+
unless candidate
|
|
421
|
+
logger.info("No signal matching #{matcher} has arrived, parking step execution")
|
|
422
|
+
transition_step!("waiting")
|
|
423
|
+
step_execution.update!(waiting_since: Time.current)
|
|
424
|
+
return :parked
|
|
425
|
+
end
|
|
426
|
+
|
|
427
|
+
logger.info("Attaching signal #{candidate.id} (#{candidate.name}) to step execution")
|
|
428
|
+
candidate.claim!
|
|
429
|
+
step_execution.update!(signal_id: candidate.id)
|
|
430
|
+
@received_signal = candidate
|
|
431
|
+
:attached
|
|
432
|
+
end
|
|
433
|
+
|
|
434
|
+
# Books one clean resolution against the attached signal. Called from
|
|
435
|
+
# finalization on clean outcomes only (completion, skip, finished!), right
|
|
436
|
+
# after the step's own terminal transition and in the same transaction: a
|
|
437
|
+
# crash before commit resolves nothing and completes nothing, so replay
|
|
438
|
+
# stays coherent.
|
|
439
|
+
#
|
|
440
|
+
# The signal only flips to consumed once every chain attached to it has
|
|
441
|
+
# resolved - for a single claimant that is this very moment, and when one
|
|
442
|
+
# signal has been dispatched onto several executions it is whichever of
|
|
443
|
+
# them finishes last. Reattempts, failures and cancellations deliberately
|
|
444
|
+
# leave the signal claimed, so the retry (or resume) re-attaches to it and
|
|
445
|
+
# sees the same payload while the audit trail reads truthfully.
|
|
446
|
+
#
|
|
447
|
+
# @return [void]
|
|
448
|
+
def resolve_attached_signal!
|
|
449
|
+
return unless GenevaDrive::StepExecution.signal_columns?
|
|
450
|
+
return if step_execution.signal_id.blank?
|
|
451
|
+
|
|
452
|
+
signal = step_execution.signal
|
|
453
|
+
return unless signal
|
|
454
|
+
|
|
455
|
+
signal.record_consumption!
|
|
456
|
+
if signal.state == "consumed"
|
|
457
|
+
logger.info("Consumed signal #{signal.id} (#{signal.name}) after #{signal.consumed} clean resolution(s)")
|
|
458
|
+
else
|
|
459
|
+
logger.info("Signal #{signal.id} (#{signal.name}) stays claimed - other attached executions are unresolved")
|
|
460
|
+
end
|
|
461
|
+
end
|
|
462
|
+
|
|
370
463
|
# Evaluates preconditions (cancel_if and skip_if) with instrumentation.
|
|
371
464
|
#
|
|
372
465
|
# @param step_def [StepDefinition]
|
|
@@ -399,6 +492,11 @@ class GenevaDrive::Executor
|
|
|
399
492
|
if step_def.should_skip?(workflow)
|
|
400
493
|
logger.info("skip_if condition matched, skipping step")
|
|
401
494
|
transition_step!("skipped", outcome: "skipped")
|
|
495
|
+
# The step may have been woken by a signal and only then decided it
|
|
496
|
+
# had nothing to do (or a retry re-attached before skip_if flipped).
|
|
497
|
+
# Skipping settles that event instead of leaving it for the next
|
|
498
|
+
# waiter on the same name.
|
|
499
|
+
workflow.settle_signal_for_skipped!(step_execution)
|
|
402
500
|
transition_workflow!("ready")
|
|
403
501
|
workflow.schedule_next_step!
|
|
404
502
|
p[:outcome] = :skipped
|
|
@@ -657,6 +755,7 @@ class GenevaDrive::Executor
|
|
|
657
755
|
def handle_completion
|
|
658
756
|
logger.info("Step completed successfully, scheduling next step")
|
|
659
757
|
transition_step!("completed", outcome: "success")
|
|
758
|
+
resolve_attached_signal!
|
|
660
759
|
transition_workflow!("ready")
|
|
661
760
|
workflow.schedule_next_step!
|
|
662
761
|
end
|
|
@@ -758,6 +857,7 @@ class GenevaDrive::Executor
|
|
|
758
857
|
when :skip
|
|
759
858
|
logger.info("Exception policy: skip!")
|
|
760
859
|
transition_step!("skipped", outcome: "skipped")
|
|
860
|
+
resolve_attached_signal!
|
|
761
861
|
transition_workflow!("ready")
|
|
762
862
|
workflow.schedule_next_step!
|
|
763
863
|
when :pause
|
|
@@ -768,6 +868,7 @@ class GenevaDrive::Executor
|
|
|
768
868
|
when :finished
|
|
769
869
|
logger.info("Exception policy: finished!")
|
|
770
870
|
transition_step!("completed", outcome: "success")
|
|
871
|
+
resolve_attached_signal!
|
|
771
872
|
transition_workflow!("finished")
|
|
772
873
|
else
|
|
773
874
|
logger.info("Exception policy: default (pause!)")
|
|
@@ -913,12 +1014,14 @@ class GenevaDrive::Executor
|
|
|
913
1014
|
when :skip
|
|
914
1015
|
logger.info("Processing skip signal: scheduling next step")
|
|
915
1016
|
transition_step!("skipped", outcome: "skipped")
|
|
1017
|
+
resolve_attached_signal!
|
|
916
1018
|
transition_workflow!("ready")
|
|
917
1019
|
workflow.schedule_next_step!
|
|
918
1020
|
|
|
919
1021
|
when :finished
|
|
920
1022
|
logger.info("Processing finished signal: finishing workflow")
|
|
921
1023
|
transition_step!("completed", outcome: "success")
|
|
1024
|
+
resolve_attached_signal!
|
|
922
1025
|
transition_workflow!("finished")
|
|
923
1026
|
|
|
924
1027
|
when :suspend
|
|
@@ -38,6 +38,17 @@ class GenevaDrive::StepConfigurationError < StandardError; end
|
|
|
38
38
|
# it is rewritten on every checkpoint and copied to every successor execution.
|
|
39
39
|
class GenevaDrive::CursorTooLargeError < StandardError; end
|
|
40
40
|
|
|
41
|
+
# Raised when a signal payload exceeds GenevaDrive.max_signal_payload_size
|
|
42
|
+
# once serialized to JSON. A payload describes the event; the data the step
|
|
43
|
+
# works on belongs on the hero.
|
|
44
|
+
class GenevaDrive::SignalPayloadTooLargeError < StandardError; end
|
|
45
|
+
|
|
46
|
+
# Raised when a signal is delivered to a workflow that has already finished
|
|
47
|
+
# or been canceled. Delivering the very same event twice (same idempotency
|
|
48
|
+
# key) is a no-op rather than an error, so callers who want lenience can
|
|
49
|
+
# rescue just this one class.
|
|
50
|
+
class GenevaDrive::WorkflowNotOngoing < StandardError; end
|
|
51
|
+
|
|
41
52
|
# Base class for errors that occur during step execution.
|
|
42
53
|
# These errors are raised after recovery actions have been performed,
|
|
43
54
|
# so the workflow/step states are already updated when the exception propagates.
|
|
@@ -276,14 +287,43 @@ module GenevaDrive::FlowControl
|
|
|
276
287
|
end
|
|
277
288
|
|
|
278
289
|
if state == "paused"
|
|
279
|
-
# Workflow was paused (e.g., due to failed step). Resume and skip to
|
|
290
|
+
# Workflow was paused (e.g., due to failed step). Resume and skip to
|
|
291
|
+
# next step. The step being skipped past may hold a signal claim on a
|
|
292
|
+
# failed execution - no further execution of it is coming, so the
|
|
293
|
+
# operator's skip is what settles that event.
|
|
294
|
+
holder = execution_holding_signal_claim(next_step_name)
|
|
280
295
|
update!(state: "ready", transitioned_at: nil)
|
|
296
|
+
settle_signal_for_skipped!(holder)
|
|
281
297
|
else
|
|
282
|
-
# Workflow is ready with a scheduled step - mark it as skipped
|
|
283
|
-
|
|
298
|
+
# Workflow is ready with a scheduled step - mark it as skipped. A
|
|
299
|
+
# dispatched-but-not-yet-run execution carries an attached signal;
|
|
300
|
+
# skipping it settles that signal too.
|
|
301
|
+
skipped_execution = current_execution
|
|
302
|
+
skipped_execution&.mark_skipped!(outcome: "skipped")
|
|
303
|
+
settle_signal_for_skipped!(skipped_execution)
|
|
284
304
|
end
|
|
285
305
|
|
|
286
306
|
schedule_next_step!
|
|
287
307
|
end
|
|
288
308
|
end
|
|
309
|
+
|
|
310
|
+
# Finds the execution that still holds a signal claim for a step we are
|
|
311
|
+
# about to skip past on a paused workflow: the current one if it was
|
|
312
|
+
# dispatched before the pause, otherwise the failed attempt that paused the
|
|
313
|
+
# workflow in the first place (same lookup shape resume! uses).
|
|
314
|
+
#
|
|
315
|
+
# @param step_name [String, nil] the step being skipped past
|
|
316
|
+
# @return [GenevaDrive::StepExecution, nil]
|
|
317
|
+
def execution_holding_signal_claim(step_name)
|
|
318
|
+
return nil unless GenevaDrive::StepExecution.signal_columns?
|
|
319
|
+
|
|
320
|
+
candidate = current_execution
|
|
321
|
+
return candidate if candidate&.signal_id.present?
|
|
322
|
+
return nil if step_name.blank?
|
|
323
|
+
|
|
324
|
+
step_executions
|
|
325
|
+
.where(step_name: step_name, state: "failed")
|
|
326
|
+
.order(created_at: :desc, id: :desc)
|
|
327
|
+
.first
|
|
328
|
+
end
|
|
289
329
|
end
|