solid_objects 0.14.0 → 0.14.2

data/Rakefile

@@ -32,6 +32,11 @@ task :steep do
   sh "bundle exec steep check"
 end
 
+desc "Prove the at-least-once clause by crashing an effect worker at a sink"
+task :at_least_once do
+  sh "bundle exec ruby examples/at_least_once/demo.rb"
+end
+
 desc "Scan the Rails engine for security warnings"
 task :security do
   sh "bundle exec brakeman --force --no-pager -q ."

data/docs/architecture.md

@@ -435,6 +435,45 @@ send_to(InventoryActor.ref("sku-123")).reserve(order_id: id, quantity: 2)
 
 The source actor commit never waits for the target. The outbox worker allocates the target's sequence after source commit. There is no global order across actors.
 
+## Registering effect and commit-action handlers
+
+Register an effect handler during application boot. The stable effect ID is the
+idempotency key for the provider call:
+
+```ruby
+SolidObjects.register_effect(:charge_payment) do |arguments, context|
+  Payments.charge(
+    idempotency_key: context.id,
+    payment_id: arguments.fetch("payment_id"),
+    amount_cents: arguments.fetch("amount_cents")
+  )
+end
+```
+
+A success callback receives `effect_id:`, the originally staged `arguments:`,
+and `result:`. A failure callback receives `effect_id:`, `arguments:`, and
+`error:`, so an actor can correlate concurrent effects without storing a
+separate callback ledger.
+
+A commit action is registered the same way and runs inside the short fenced
+transaction:
+
+```ruby
+SolidObjects.register_commit_action(:complete_attempt) do |arguments, context|
+  AssessmentAttempt.find(arguments.fetch("attempt_id")).update!(
+    score: arguments.fetch("score"),
+    actor_message_id: context.message_id
+  )
+end
+```
+
+Commit actions require Solid Objects and `ActiveRecord::Base` to share one
+connection pool, and may be invoked again after a database rollback, so keep
+them deterministic, bounded, and database-only. When Solid Objects uses a
+separate actor database, use `emit` and an idempotent effect consumer instead;
+two databases cannot share one transaction.
+
+
 ## Reminders
 
 A reminder record contains actor identity, a reminder name, target message, JSON arguments, next run time, optional interval, status, and occurrence counter.
@@ -443,7 +482,7 @@ A reminder record contains actor identity, a reminder name, target message, JSON
 schedule(at: 30.minutes.from_now).expire
 ```
 
-Reminders are keyed by `(actor, reminder name)`, enforced by a unique index on `(instance_id, name)`. `schedule` is therefore an upsert: scheduling a name that is already armed moves that alarm instead of adding another, which is what makes re-arming safe from a handler that may run more than once. An actor needing several pending items should arm one alarm for the earliest and drain everything due when it fires, rather than one alarm per item; the [reminders guide](../README.md#a-reminder-is-one-named-alarm-per-actor) shows that pattern. A move that changes `next_run_at` emits `solid_objects.reminder.replaced`, because the replacement is otherwise indistinguishable from a first schedule.
+Reminders are keyed by `(actor, reminder name)`, enforced by a unique index on `(instance_id, name)`. `schedule` is therefore an upsert: scheduling a name that is already armed moves that alarm instead of adding another, which is what makes re-arming safe from a handler that may run more than once. An actor needing several pending items should arm one alarm for the earliest and drain everything due when it fires, rather than one alarm per item; the [reminders guide](reminders.md#one-alarm-for-a-whole-queue) shows that pattern. A move that changes `next_run_at` emits `solid_objects.reminder.replaced`, because the replacement is otherwise indistinguishable from a first schedule.
 
 When due, the scheduler locks the source instance and creates a normal mailbox
 row with an idempotency key derived from reminder ID and occurrence. The

data/docs/correctness.md

@@ -93,6 +93,15 @@ end
 The guard prevents a repeated state transition. The effect consumer still
 deduplicates with `context.id`.
 
+This clause is observable, not decorative. `bundle exec rake at_least_once`
+crashes an effect worker between the external sink write and the
+acknowledgement, restarts one after the liveness threshold, and shows the sink
+reading 2 with deduplication off. Both deliveries carry the same `context.id`
+at attempts 1 and 2. A guard on that id absorbs the same duplicate and the
+sink reads 1. The actor state commits exactly once in both runs. The source is
+`examples/at_least_once/`; solid-objects-js runs the same proof with
+`pnpm run test:at-least-once`.
+
 ## Atomic boundaries
 
 The following are atomic:

data/docs/operations.md

@@ -23,6 +23,103 @@ process, its activations, and its claimed messages untouched, including when an
 application call overlaps the probe. A database busy enough to block cleanup
 reports a failed or warned check rather than raising out of the command.
 
+## Installing and upgrading
+
+Review [CHANGELOG.md](CHANGELOG.md) for compatibility and deployment-order
+notes, then update the gem:
+
+```bash
+bundle update solid_objects
+```
+
+If the `Gemfile` pins an exact version, update that constraint first and run
+`bundle install`. Commit both `Gemfile.lock` and the copied Solid Objects
+migrations.
+
+Copy only migrations that the newer gem has added, migrate, and verify the
+installation:
+
+```bash
+bin/rails solid_objects:install:migrations
+bin/rails db:migrate
+bin/rails solid_objects:doctor
+```
+
+The migration task skips engine migrations already present in the application
+and gives new migrations host-specific timestamps. Inspect the resulting
+`db/migrate/*.solid_objects.rb` files before applying them. Do not rerun
+`generate solid_objects:install` during an upgrade because that also attempts
+to regenerate the application initializer.
+
+When Solid Objects uses a separate database configuration named `actors`, copy
+and run migrations through that database's configured migration path:
+
+```bash
+DATABASE=actors bin/rails solid_objects:install:migrations
+bin/rails db:migrate:actors
+bin/rails solid_objects:doctor
+```
+
+For production, back up the actor database and run new migrations before
+starting application or Solid Objects worker processes that require the new
+schema. Restart the web and Solid Objects worker fleet after the bundle and
+schema are current. For releases that change actor state versions, also follow
+the [state migration and rolling-deployment guide](docs/state-migrations.md);
+Rails schema migrations and actor state migrations are separate concerns.
+
+### Host application tooling
+
+Installed engine migrations are copied as
+`db/migrate/*_create_solid_objects_tables.solid_objects.rb`. If the host enables
+`Rails/CreateTableWithTimestamps`, exclude engine-owned migrations rather than
+editing their intentionally specialized hot tables:
+
+```yaml
+Rails/CreateTableWithTimestamps:
+  Exclude:
+    - "db/migrate/*.solid_objects.rb"
+```
+
+Solid Objects ships inline RBS signatures, not RBI files. Sorbet applications
+can generate the gem RBI with:
+
+```bash
+bundle exec tapioca gem solid_objects
+```
+
+### Running an extension in the same process
+
+An extension gem can register its own long-running component, and
+`solid_objects start` runs it beside the built-in roles. The component joins the
+same supervision, the same replacement after a crash, and the same shutdown
+timeout, so an operator deploys and monitors one process instead of two:
+
+```ruby
+SolidObjects.configure do |configuration|
+  configuration.register_component { MyExtension::FlushEngine.new }
+end
+```
+
+Pass `count:` for more than one instance. The block runs once for each instance,
+and again when the supervisor replaces a crashed one, so no two components share
+an object.
+
+A registered component answers four methods, the contract the built-in roles
+already keep:
+
+| Method | Purpose |
+| --- | --- |
+| `run` | Runs the loop. The supervisor calls it in its own thread |
+| `request_shutdown` | Asks the loop to finish. It must make `run` return |
+| `stopped?` | Reports whether the component already finished |
+| `stop` | Forces cleanup when the shutdown timeout expires first |
+
+The supervisor checks that contract when it builds the component, and a missing
+method raises `ArgumentError` as the supervisor starts, rather than hanging a
+shutdown later. Registration itself never calls the block, so a component is
+free to need a database connection that the application does not have while it
+boots.
+
 ## Runtime
 
 Start all configured roles:
@@ -31,12 +128,35 @@ Start all configured roles:
 bundle exec solid_objects start
 ```
 
-The command loads the host application's `app/actors` directories before
+The generator and the migrations prepare the database and start nothing. A
+process claims ready messages only after this command starts its roles, so an
+application that serves web requests alone leaves every `async` message ready.
+The message is durable and waits for the first process that runs the roles. A
+direct call or an explicit `sync` needs no running role, because the caller's
+own path executes it.
+
+The engine loads the host application's `app/actors` directories in every
+process that boots the application, and the command repeats that load before
 starting any runtime role, even when Rails eager loading is disabled. Actors in
 the conventional directory do not need initializer references. The targeted
 loader participates in Rails preparation callbacks so a development reload can
 replace a registered actor class without loading unrelated application code.
 
+An actor registers itself as its class loads, and a web process resolves
+actors by name for Cable subscriptions and component renders. Loading them in
+every process is what lets a freshly booted web process serve a live card for
+an actor no request in that process has rendered yet.
+
+Worker and outbox counts can be overridden on the command line:
+
+```bash
+bundle exec solid_objects start \
+  --workers 4 \
+  --effect-workers 2 \
+  --broadcast-workers 2 \
+  --reminder-schedulers 1
+```
+
 Inspect process records and clean stale ownership:
 
 ```bash
@@ -57,26 +177,45 @@ bundle exec solid_objects retry_dead_letter 123
 
 ## Configuration
 
-Important controls include:
-
-- `worker_count`
-- `effect_worker_count`
-- `broadcast_worker_count`
-- `reminder_scheduler_count`
-- `max_messages_per_activation_pass`
-- `max_activation_duration`
-- `idle_deactivation_timeout`
-- `lease_duration`
-- `lease_renewal_interval`
-- `polling_interval`
-- `idle_polling_interval`
-- `max_mailbox_length`
-- payload, state, and result byte limits
-- retry attempts and delay
-- heartbeat interval and alive threshold
-- message retention and per-actor-type overrides
-- opt-in actor-instance retention by actor type
-- stopped-process retention and prune batch size
+Configure Solid Objects in `config/initializers/solid_objects.rb`:
+
+```ruby
+SolidObjects.configure do |configuration|
+  configuration.worker_count = 4
+  configuration.lease_duration = 30.seconds
+  configuration.lease_renewal_interval = 10.seconds
+  configuration.max_messages_per_activation_pass = 50
+  configuration.max_activation_duration = 5.seconds
+end
+```
+
+| Setting | Default |
+| --- | ---: |
+| `polling_interval` | 0.1 seconds |
+| `idle_polling_interval` | 1 second |
+| `sync_polling_interval` | 0.05 seconds |
+| `lease_duration` | 30 seconds |
+| `lease_renewal_interval` | 10 seconds |
+| `idle_deactivation_timeout` | 30 seconds |
+| `max_messages_per_activation_pass` | 50 |
+| `max_activation_duration` | 5 seconds |
+| `max_mailbox_length` | 10,000 |
+| `max_attempts` | 5 |
+| `process_heartbeat_interval` | 15 seconds |
+| `process_alive_threshold` | 60 seconds |
+| `message_retention` | 30 days |
+| `message_retention_by_actor_type` | `{}` |
+| `instance_retention_by_actor_type` | `{}`; instances never expire unless listed |
+| `process_retention` | 7 days |
+| `prune_batch_size` | 1,000 |
+| `worker_count` | 1 |
+| `effect_worker_count` | 1 |
+| `broadcast_worker_count` | 1 |
+| `reminder_scheduler_count` | 1 |
+
+Payload, state, and result byte limits; retry delay; table prefix; logging;
+wake-up; broadcast; database; and authorization adapters are also configurable.
+Invalid lease intervals, component counts, and size limits fail fast at boot.
 
 Keep lease duration comfortably above renewal interval and expected database
 pause time. A handler can exceed the pass-duration budget because Ruby code is
@@ -187,6 +326,15 @@ transaction rejection, commit-action start/completion/failure, effect and
 broadcast enqueue/completion, reminder enqueue, actor destruction/expiration,
 retention pruning, process cleanup, and supervisor lifecycle.
 
+`solid_objects.subscription.rejected` reports a rejected Cable subscription.
+A rejection closes the socket and leaves the page holding a stale card, and
+the browser cannot say which of the conditions applied. The event carries the
+`reason`, the actor identity, and the `error_class` where an exception caused
+it. The reason is one of `unregistered_actor_type`, `invalid_stream_token`,
+`invalid_component_token`, `malformed_component_registration`,
+`missing_subscription_parameter`, or `unauthorized`. Exception messages are
+excluded, because a component or payload failure can carry actor state.
+
 `solid_objects.reminder.replaced` reports a `schedule` call that moved an alarm
 already armed under the same name on the same actor, carrying the actor
 identity, reminder `name`, `previous_run_at`, and `next_run_at`. Reminders are

data/docs/reminders.md

@@ -0,0 +1,112 @@
+# Reminders
+
+Reminders are Solid Objects' durable equivalent of the Durable Objects Alarms
+API. One-shot and recurring alarms are actor-owned database records:
+
+```ruby
+def schedule_evaluation
+  schedule(
+    at: 1.hour.from_now,
+    every: 1.hour,
+    missed: :latest
+  ).evaluate(account_id:)
+end
+```
+
+Use `missed: :latest` to coalesce missed occurrences or `missed: :all` to
+enqueue each one.
+
+## A reminder is one named alarm per actor
+
+The uniqueness key is `(actor, reminder name)`. Scheduling a name that is
+already armed **moves the existing alarm** rather than adding a second one. The
+database enforces this with a unique index on `(instance_id, name)`.
+
+This is the same model as Orleans reminders and Durable Objects alarms, and it
+is what makes a reminder safe to re-arm from a handler that may run more than
+once. Without a key the name is the operation, so this is a data-loss bug:
+
+```ruby
+# Wrong. Every entry overwrites the previous entry's alarm.
+def add(entry:)
+  self.entries = entries + [ entry ]
+  schedule(at: entry.fetch("wait_until")).deliver
+end
+```
+
+Two entries leave one reminder. The earlier wake-up never happens, nothing
+raises, and nothing is logged except a `solid_objects.reminder.replaced` event.
+
+## An alarm per item, with `key:`
+
+Pass `key:` when an actor is waiting on several things at once. The key is your
+own identifier for the item, and it names that item's alarm, so each item gets
+one:
+
+```ruby
+def add(entry:)
+  self.entries = entries + [ entry ]
+  schedule(at: entry.fetch("wait_until"), key: entry.fetch("id")).deliver
+end
+```
+
+Two entries now leave two reminders. Scheduling the same key again moves that
+item's alarm and leaves the others alone, which is what makes a keyed reminder
+as safe to re-arm as an unkeyed one. The operation still decides which handler
+runs; the key only decides which alarm is which.
+
+A key must be non-empty, and the name it becomes must fit the 191-character
+column, which is checked on the composed name rather than the key alone so a
+long operation and a short key are caught too.
+
+The key is separated from the operation by a colon, so an operation may not hold
+one. Otherwise an unkeyed `deliver:item` and a `deliver` keyed `item` would be
+one name, and the second would silently take the first one's alarm. A key may
+hold colons of its own, because the operation before the first one cannot.
+
+## One alarm for a whole queue
+
+A key per item is not always what you want. An actor that only ever needs to
+know "what is next" can keep one alarm and let the handler drain everything now
+due before arming the next:
+
+```ruby
+def add(entry:)
+  self.entries = (entries + [ entry ]).sort_by { |item| item.fetch("wait_until") }
+  arm_next
+end
+
+def deliver
+  now = Time.current.to_i
+  due, pending = entries.partition { |item| item.fetch("wait_until") <= now }
+  due.each { |item| emit :send_push, **item.symbolize_keys }
+  self.entries = pending
+  arm_next
+end
+
+private
+
+def arm_next
+  earliest = entries.first
+  return unless earliest
+
+  schedule(at: Time.at(earliest.fetch("wait_until"))).deliver
+end
+```
+
+That costs one reminder row instead of one per item, and a coalesced occurrence
+cannot strand an entry because the handler drains by time rather than by alarm.
+Prefer it when the queue is large and the items are interchangeable; prefer
+`key:` when an item needs its own alarm that can be moved on its own.
+
+Solid Objects has no `unschedule`. A reminder stops when its handler does not
+re-arm it, and destroying an actor removes its reminders.
+
+Self-scheduling actors should also have a low-frequency application reconciler.
+It may read `SolidObjects::Instance.states_for`, `.without_pending_work`, and
+`.orphaned`, but every repair must go through `async`. Never bulk-update actor
+state around the lease and fencing checks.
+
+Suspended actors should be reported rather than silently resumed. Spread large
+repair batches with `available_at:` so reconciliation cannot stampede one
+mailbox or the worker fleet.

data/docs/roadmap.md

@@ -126,7 +126,13 @@
   untested end to end, which is how a raising payload block came to reject the
   subscription; it is now covered and confined, and the payload authorization
   context is resolved through `payload_authorization_context` rather than
-  handing the block a raw Cable connection.
+  handing the block a raw Cable connection. Actor registration in a web process
+  was assumed rather than arranged: an actor registered only as a side effect
+  of its class loading, and only the worker CLI loaded the host's `app/actors`,
+  so a lazily loading web process rejected subscriptions for actors it could
+  serve until some earlier request happened to load the class. The engine now
+  loads them in every process, and a rejected subscription reports which
+  condition caused it instead of closing the socket silently.
 - Backpressure: mailbox/payload/state/result caps and fair yields exist;
   distributed per-actor rate limits and global admission control do not.
 - Administration: `SolidObjects::Web` is a mountable Rack dashboard covering

data/examples/at_least_once/actor.rb

@@ -0,0 +1,14 @@
+# rbs_inline: enabled
+
+class DeliveryCounter < SolidObjects::Actor
+  actor_type "delivery-counter"
+
+  attribute :count, default: 0
+
+  # @rbs () -> Integer
+  def deliver
+    self.count += 1
+    emit :record
+    count
+  end
+end

data/examples/at_least_once/boot.rb

@@ -0,0 +1,47 @@
+# rbs_inline: enabled
+
+require "bundler/setup"
+require "active_record"
+require "solid_objects"
+
+# Boots a standalone runtime against a shared SQLite file, the way the
+# demo's parent and its crashing children all attach to one database.
+module AtLeastOnceBoot
+  ROOT = File.expand_path("../..", __dir__)
+
+  # @rbs (String database_path) -> void
+  def self.call(database_path)
+    ActiveRecord::Base.establish_connection(
+      adapter: "sqlite3",
+      database: database_path,
+      pool: 5,
+      timeout: 5_000
+    )
+    ActiveRecord::Migration.verbose = false
+    migrate unless ActiveRecord::Base.connection.table_exists?("solid_objects_instances")
+
+    require "solid_objects/database_adapter"
+    %w[
+      record process instance message ready_message claimed_message
+      reminder effect broadcast dead_letter
+    ].each { |model| require File.join(ROOT, "app/models/solid_objects", model) }
+
+    SolidObjects.configuration.authorize_message = ->(**) { true }
+    SolidObjects.configuration.authorize_query = ->(**) { true }
+    SolidObjects.configuration.polling_interval = 0.01
+    SolidObjects.configuration.process_heartbeat_interval = 0.075
+    SolidObjects.configuration.process_alive_threshold = 0.3
+    SolidObjects.configuration.lease_duration = 0.25
+    SolidObjects.configuration.lease_renewal_interval = 0.05
+  end
+
+  # @rbs () -> void
+  def self.migrate
+    require File.join(ROOT, "db/migrate/20260805000000_create_solid_objects_tables")
+    require File.join(ROOT, "db/migrate/20260806000000_add_state_revision_to_solid_objects_instances")
+    require File.join(ROOT, "db/migrate/20260813000000_rename_message_dispatch_columns")
+    CreateSolidObjectsTables.new.migrate(:up)
+    AddStateRevisionToSolidObjectsInstances.new.migrate(:up)
+    RenameMessageDispatchColumns.new.migrate(:up)
+  end
+end

data/examples/at_least_once/demo.rb

@@ -0,0 +1,92 @@
+# rbs_inline: enabled
+
+# An executable proof for the at-least-once clause: a contract clause
+# nobody can observe firing is decoration. Run with:
+#
+#   bundle exec rake at_least_once
+#
+# Phase one crashes an effect worker between the external sink write and
+# the acknowledgement, restarts one, and shows the sink reading 2 with
+# deduplication off. Both deliveries carry the same stable effect id.
+# Phase two repeats the crash with a guard on that id; the sink reads 1.
+# The actor state commits exactly once in both phases.
+
+require_relative "boot"
+require_relative "actor"
+require_relative "sink"
+require "fileutils"
+require "json"
+require "rbconfig"
+require "tmpdir"
+
+directory = Dir.mktmpdir("solid_objects_at_least_once_")
+database_path = File.join(directory, "state.sqlite3")
+AtLeastOnceBoot.call(database_path)
+
+# @rbs (String message) -> void
+def prove(message)
+  raise "proof failed: #{message}" unless yield
+end
+
+# @rbs (String actor_id) -> void
+def stage_one_delivery(actor_id)
+  DeliveryCounter.ref(actor_id).async.deliver
+  worker = SolidObjects::Worker.new
+  begin
+    worker.run_until_idle
+  ensure
+    worker.stop
+  end
+end
+
+# @rbs (database_path: String, sink_path: String, mode: String, deduplication: String) -> Integer?
+def run_effect_worker(database_path:, sink_path:, mode:, deduplication:)
+  script = File.expand_path("effect_worker.rb", __dir__)
+  pid = Process.spawn(
+    RbConfig.ruby, script, database_path, sink_path, mode, deduplication,
+    chdir: AtLeastOnceBoot::ROOT
+  )
+  _pid, status = Process.wait2(pid)
+  status.exitstatus
+end
+
+# @rbs (database_path: String, sink_path: String, deduplication: String) -> void
+def crash_then_recover(database_path:, sink_path:, deduplication:)
+  crash = run_effect_worker(database_path:, sink_path:, mode: "crash", deduplication:)
+  prove("the first delivery crashed before acknowledgement") { crash == 1 }
+  sleep 0.4
+  recovery = run_effect_worker(database_path:, sink_path:, mode: "complete", deduplication:)
+  prove("the second delivery completed and acknowledged") { recovery == 0 }
+end
+
+begin
+  sink_off = File.join(directory, "sink-dedup-off.json")
+  stage_one_delivery("dedup-off")
+  crash_then_recover(database_path:, sink_path: sink_off, deduplication: "off")
+  deliveries = AtLeastOnceSink.read(sink_off)
+  effect_ids = deliveries.map { |delivery| delivery.fetch("effect_id") }
+  state_off = SolidObjects::Instance.find_by!(actor_id: "dedup-off").state.fetch("count")
+  prove("the state commit happened exactly once") { state_off == 1 }
+  prove("the sink observed the duplicate") { deliveries.length == 2 }
+  prove("both deliveries carried the same stable effect id") { effect_ids.uniq.length == 1 }
+
+  sink_on = File.join(directory, "sink-dedup-on.json")
+  stage_one_delivery("dedup-on")
+  crash_then_recover(database_path:, sink_path: sink_on, deduplication: "on")
+  guarded = AtLeastOnceSink.read(sink_on)
+  state_on = SolidObjects::Instance.find_by!(actor_id: "dedup-on").state.fetch("count")
+  prove("the state commit happened exactly once") { state_on == 1 }
+  prove("the stable effect id absorbed the duplicate") { guarded.length == 1 }
+
+  puts JSON.pretty_generate(
+    duplicate: {
+      state_commits: state_off,
+      sink_deliveries: deliveries.length,
+      same_effect_id: effect_ids.uniq.length == 1,
+      attempts: deliveries.map { |delivery| delivery.fetch("attempt") }
+    },
+    remedy: { state_commits: state_on, sink_deliveries: guarded.length }
+  )
+ensure
+  FileUtils.remove_entry(directory) if directory
+end

data/examples/at_least_once/effect_worker.rb

@@ -0,0 +1,40 @@
+# rbs_inline: enabled
+
+require_relative "boot"
+require_relative "actor"
+require_relative "sink"
+
+database_path, sink_path, mode, deduplication = ARGV
+raise ArgumentError, "usage: effect_worker.rb DATABASE SINK crash|complete on|off" unless deduplication
+
+AtLeastOnceBoot.call(database_path.to_s)
+
+SolidObjects.register_effect(:record) do |_arguments, context|
+  AtLeastOnceSink.record(
+    path: sink_path.to_s,
+    effect_id: context.id,
+    attempt: context.attempt,
+    deduplication: deduplication.to_sym
+  )
+  # A crash between the external write and the acknowledgement: the sink
+  # has the delivery, the effect row never completes.
+  Process.exit!(1) if mode == "crash"
+  nil
+end
+
+# Production runs this on the dead-process-cleanup interval; the demo runs
+# it once, after the liveness threshold, to release the crashed claim.
+SolidObjects::ProcessRegistry.cleanup_dead
+
+effect_executor = SolidObjects::EffectExecutor.new
+begin
+  worked = false
+  200.times do
+    worked = effect_executor.run_once
+    break if worked
+    sleep 0.01
+  end
+  raise "no effect became claimable" unless worked
+ensure
+  effect_executor.stop
+end

data/examples/at_least_once/sink.rb

@@ -0,0 +1,27 @@
+# rbs_inline: enabled
+
+require "json"
+
+# The external system in the at-least-once demo: a JSON file that records
+# every delivery it accepts. With deduplication :off it accepts everything,
+# which makes an at-least-once duplicate visible. With deduplication :on it
+# accepts each stable effect id once, which is the documented remedy.
+module AtLeastOnceSink
+  # @rbs (String path) -> Array[Hash[String, untyped]]
+  def self.read(path)
+    JSON.parse(File.read(path))
+  rescue Errno::ENOENT
+    []
+  end
+
+  # @rbs (path: String, effect_id: String, attempt: Integer, deduplication: Symbol) -> bool
+  def self.record(path:, effect_id:, attempt:, deduplication:)
+    deliveries = read(path)
+    seen = deliveries.any? { |delivery| delivery.fetch("effect_id") == effect_id }
+    return false if deduplication == :on && seen
+
+    deliveries << { "effect_id" => effect_id, "attempt" => attempt }
+    File.write(path, JSON.pretty_generate(deliveries))
+    true
+  end
+end