solid_objects 0.15.2 → 0.16.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 55b365960ffe5f49934eb8914e35d546af1de656e158477977caf8c75db19fcd
-  data.tar.gz: fcea896079cc1f85eb60e6cbc13d080c597a8074a077cf32a1e34142b5055886
+  metadata.gz: 2d267d861c25f682b3e45208131b9c115b49710ae43fb499223c6d63f48519a0
+  data.tar.gz: 1acc3bb415c57e9072f231fc5e37ed37ce8bab4a9ba5fc894a7dbe5453839d8e
 SHA512:
-  metadata.gz: 6c21e96eb84ebbaf519c8860b79d952f3daa5d647f6ca0de6d35527b5b24bab53fd086c35fb57446a1173798f4c2849034d368415ff4afaacccedb509e768f23
-  data.tar.gz: f11760f2484478b62ad93cf18327aaa9f5a9d53b7b451c8f1233a479854a3f50b75b369f8559d06295794ea19cad30696088889b3258a541c4003f5775cec2fc
+  metadata.gz: 1f3d71f114877579419a680b8ff46b56036060d8575c7c6732c4c9d530ed3020fbb58eb61ac3e348810d44ecf09e8553b4f07390bd1c22460b5604005578e06f
+  data.tar.gz: 07e811b54283aa3f3638ae90b8fbc449f5e151dbcae69e01487a8231a1b8cc5556bd1cc035b67aa347f080329a64f9c0394965d468860dd989f0ba40afdc7552

data/CHANGELOG.md

@@ -1,5 +1,156 @@
 # Changelog
 
+## 0.16.0 - 2026-09-23
+
+- Find a message whose reference a caller lost.
+  `SolidObjects.client.find_by(request_id:)` answers a request id, which is
+  unique across the table, and `reference.find_by(idempotency_key:)` answers a
+  key, which is unique per instance, so the receiver supplies the scope the key
+  needs. Naming neither key, naming both, or naming an idempotency key without a
+  reference raises `ArgumentError`.
+- Authorize every lookup with the hook the original call ran, against the stored
+  operation and arguments, because a request id is not a capability. An absent
+  row, an actor this process no longer registers, and a caller the policy
+  refuses all return `nil`, so a lookup cannot be used to ask whether a request
+  id exists.
+- Add `MessageReference#outcome`, which reports the status, the result, the
+  persisted error, the rejection, and the attempt count, so a terminal failure
+  answers as well as a success. A result is stored for `sync` delivery only, so
+  an asynchronous message reports its status and error and no result.
+- Tell a pruned message from one that never existed. An actor remembers the
+  idempotency keys of its own finished turns, the way an Orleans grain keeps
+  its deduplication history in grain state, so the memory needs no second
+  store and no second write. `reference.find_by(idempotency_key:)` raises
+  `SolidObjects::MessagePruned` for a key the actor remembers and whose message
+  retention removed, and still answers `nil` for a key no caller ever sent.
+  An actor remembers the operation beside each key, so the pruned answer runs
+  the same hook against the same operation a lookup of the surviving row would,
+  and a caller the policy refuses reads `nil` for both. Gating it on `snapshot`
+  would have told a caller who may read state, but not the operation, that the
+  operation had run.
+  `retained_idempotency_keys` bounds the memory and defaults to 64 keys for
+  each actor. A lookup by request id cannot make the distinction, because the
+  runtime, not the caller, generates a request id and no actor remembers one.
+  `retained_idempotency_keys_bytes` bounds the serialized memory as well,
+  because an idempotency key has no length limit on every adapter and the memory
+  outlives the message row. An actor drops its oldest keys until the list fits,
+  so a key long enough to fill the limit by itself is never remembered.
+- Add `db/migrate/20260923000000_add_solid_objects_completed_idempotency_keys.rb`,
+  which adds `instances.completed_idempotency_keys` as `jsonb` on PostgreSQL and
+  `json` elsewhere. An application installs it with
+  `bin/rails solid_objects:install:migrations` and runs it before it upgrades a
+  worker, because the executor writes the column on every finished turn. The
+  doctor now reports the column as missing when it is not installed.
+- Apply migrations through `SolidObjects::SchemaBootstrap`, which reads
+  `db/migrate`. Seven scripts each carried a hand-copied migration list, and
+  three of them applied an incomplete schema. A test fails if any script names a
+  migration class again.
+- Report a half-applied migration in `solid_objects doctor`. The column list
+  omitted `instances.state_revision`, `messages.operation`,
+  `effects.success_operation`, `effects.failure_operation`, and
+  `dead_letters.operation`, so an application that skipped a migration read as
+  healthy and found out from a worker crash. A test fails when the list does not
+  name a column that a migration after the first adds.
+
+- List a dead effect or broadcast as a `SolidObjects::DeadRow` rather than as
+  an Active Record row. `all` returned rows whose `id` was the primary key while
+  `retry` reads `effect_id` or `broadcast_id`, so the obvious
+  `scope.retry(scope.all.first.id)` raised `ActiveRecord::RecordNotFound`.
+  `DeadRow#id` is now the value `retry` accepts, which is what the TypeScript
+  runtime has always returned. `dead` still answers the relation for a caller
+  that wants to scope it further.
+- Raise a load error rather than report an unreachable database. Wake-up
+  selection rescued every exception, so a `NameError` from an unloaded model
+  read as "the database could not be reached" and downgraded the process to
+  in-process signalling. It now rescues database, system call, and IO errors
+  only.
+- Note that `json` 3.0.2 breaks `ActiveSupport::JSON.decode`, and therefore
+  every JSON column, in [docs/operations.md](docs/operations.md).
+- Retry a dead effect or broadcast. `SolidObjects.dead_letters` keeps its
+  message meaning and answers `effects` and `broadcasts`, so the kind rides on
+  the receiver. `retry` returns a dead row to pending with a zero attempt count
+  and no claim, reuses the stable id so a deduplicating handler sees the same
+  key, and acts only on a dead row, so a second press cannot double-enqueue. A
+  dead transmit effect replays rather than stay lost.
+- Redrive a whole scope. `redrive` opens a durable task, returns at once, and is
+  idempotent over its scope and filters, which a dashboard button needs. A
+  unique index on the active scope enforces that in the database, so two
+  processes that start the same redrive share one task. The supervisor advances
+  one bounded batch per pass, so a redrive never holds a transaction longer than
+  one batch. `SolidObjects.redrives` reads tasks back, and `task.cancel` stops
+  one and leaves the rows it already moved. A redrive moves what was dead when
+  it started, so a still-broken handler cannot make it run forever.
+- Record who pressed what. Every retry and every redrive transition writes one
+  row to `solid_objects_administration_events`. The identity comes from the
+  authorization context through a new `administration_identity` hook.
+- Add `redrive_batch_size`, which defaults to 100, and `redrive_batch_pause`,
+  which defaults to 0.05 seconds.
+- Add two tables, `solid_objects_administration_events` and
+  `solid_objects_redrives`. Run `bin/rails solid_objects:install:migrations` and
+  migrate.
+
+- Select a wake-up adapter automatically. `config.wake_up_adapter` now takes a
+  name or an adapter, as `config.cache_store` and
+  `config.active_job.queue_adapter` do, and defaults to `:automatic`. Selection
+  prefers a configured Redis URL, then PostgreSQL notifications, then polling.
+  `:in_process` opts out, and an unknown name raises rather than quietly
+  polling.
+- PostgreSQL deployments that configure nothing now use notifications. They gain
+  cross-process wake-up, a connection per waiting thread outside the pool, and
+  one `NOTIFY` per enqueue after the commit. Set
+  `config.wake_up_adapter = :in_process` to keep polling.
+- Prove the PostgreSQL notification path before selecting it, because `LISTEN`
+  does not survive a transaction pooler such as PgBouncer. Selection listens on
+  a probe channel, sends one `NOTIFY` from a second connection, and waits up to
+  two seconds for it to arrive. A probe that does not deliver falls back to
+  polling and warns once.
+- Select the wake-up adapter once per process. `SolidObjects.wake_up` memoised
+  without a lock, so threads that raced for the first use each ran a full
+  selection.
+- Keep the capability that a configured adapter reports about itself. A
+  configured `SolidObjects::WakeUp` now reports `:in_process` and warns, rather
+  than claim that it crosses processes.
+- Poll rather than pretend when a requested adapter cannot be built.
+  `wake_up_adapter = :postgresql` on a database with no notification channel,
+  `:redis` without `SOLID_OBJECTS_REDIS_URL`, and a Redis URL without the redis
+  gem each log `solid_objects.wake_up.unavailable` once and record the reason in
+  the capability, so the doctor warns rather than claim a cross-process wake-up
+  that cannot happen.
+- Validate `wake_up_adapter` in `configure`. An unknown name raised at the first
+  wake-up, which is after a commit, rather than at boot.
+- Report the resolved choice. `SolidObjects.wake_up.capability` names the
+  adapter, whether it crosses processes, its measured floor, and why it was
+  chosen. The doctor reports it, and the polling-only warning now fires on what
+  was installed rather than on whether a setting was set.
+- Read the durable row rather than the query cache in `MessageReference#status`,
+  `MessageReference#result`, and an actor snapshot. A caller that polls holds one
+  query cache for the whole poll, and the worker that finishes the message is
+  another process, so its write cannot clear that cache. A poll inside a request,
+  a job, or `rails runner` reported the first answer forever. The synchronous
+  wait already read uncached.
+- Add reminder cancellation. `unschedule` removes one reminder by operation and
+  optional key, or by the handle `schedule` now returns. `unschedule_all`
+  removes every key of one operation. Both stage an intent, so a cancel commits
+  with the state change that decided it, and a turn that raises cancels nothing.
+- Add reminder reading. `reminder` returns a `ReminderStatus` or `nil`, and
+  `reminders` lists every key of one operation. A read applies the intents
+  staged in the current turn, so it agrees with what the commit will write. An
+  actor reads its own schedule from every path, including activation hooks and
+  observables, because it carries its instance rather than reading an ambient
+  context that only message dispatch establishes.
+- Leave a one-shot reminder that already fired out of `reminder` and
+  `reminders`. Its row stays as `completed`, so a next-run lookup reported an
+  old time rather than nothing, and an existence check refused to re-arm an
+  alarm that could never fire again.
+- Refuse an unknown operation in `reminder`, `reminders`, `unschedule`, and
+  `unschedule_all`. `schedule` already raised `UnknownMessage` for one, so a
+  typo cancelled nothing quietly and left a recurring reminder running.
+- A cancel cannot recall an occurrence the scheduler already turned into a
+  message. It does pre-empt one the scheduler claimed but has not yet enqueued.
+- `schedule` now returns a reminder handle instead of `nil`. An operation that
+  ends with `schedule` and relies on an implicit `nil` result should return
+  `nil` explicitly, as `emit` required in 0.15.0.
+
 ## 0.15.2 - 2026-09-21
 
 - Find the actor instance before the insert when an enqueue starts, and lock

data/README.md

@@ -16,7 +16,7 @@ Solid Object Rails Actors elegantly fit anything where one identifiable thing mu
 - Ticket holds and reservations
 - Multiplayer games and shared rooms
 - Shopping carts and checkout recovery
-- Rate limits and account quotas
+- Low-rate quotas and account limits
 - Session expiration
 - Job leases and workflows
 - Connected devices

data/app/models/solid_objects/administration_event.rb

@@ -0,0 +1,7 @@
+# rbs_inline: enabled
+
+module SolidObjects
+  class AdministrationEvent < Record
+    self.table_name = SolidObjects.table_name(:administration_events)
+  end
+end

data/app/models/solid_objects/redrive.rb

@@ -0,0 +1,7 @@
+# rbs_inline: enabled
+
+module SolidObjects
+  class Redrive < Record
+    self.table_name = SolidObjects.table_name(:redrives)
+  end
+end

data/benchmark/support.rb

@@ -425,14 +425,8 @@ module SolidObjectsBenchmark
 
     # @rbs () -> void
     def migrate
-      require_relative "../db/migrate/20260805000000_create_solid_objects_tables"
-      require_relative "../db/migrate/20260806000000_add_state_revision_to_solid_objects_instances"
-      require_relative "../db/migrate/20260813000000_rename_message_dispatch_columns"
-      require_relative "../db/migrate/20260915000000_add_solid_objects_effect_recoveries"
-      CreateSolidObjectsTables.new.migrate(:up)
-      AddStateRevisionToSolidObjectsInstances.new.migrate(:up)
-      RenameMessageDispatchColumns.new.migrate(:up)
-      AddSolidObjectsEffectRecoveries.new.migrate(:up)
+      require "solid_objects/schema_bootstrap"
+      SolidObjects::SchemaBootstrap.install
     end
 
     # @rbs () -> void

data/db/migrate/20260922000000_add_solid_objects_administration_events.rb

@@ -0,0 +1,26 @@
+# rbs_inline: enabled
+
+class AddSolidObjectsAdministrationEvents < ActiveRecord::Migration[7.1]
+  # @rbs () -> void
+  def change
+    create_table SolidObjects.table_name(:administration_events) do |definition|
+      definition.string :action, null: false, limit: 64
+      definition.string :kind, null: false, limit: 32
+      definition.string :subject_id, limit: 191
+      definition.public_send(json_type, :filters)
+      definition.string :actor, limit: 255
+      definition.datetime :occurred_at, null: false, precision: 6
+      definition.timestamps precision: 6, null: false
+
+      definition.index [ :occurred_at, :id ], name: "idx_so_admin_events_occurred"
+      definition.index [ :kind, :subject_id ], name: "idx_so_admin_events_subject"
+    end
+  end
+
+  private
+
+  # @rbs () -> Symbol
+  def json_type
+    connection.adapter_name.match?(/postgres/i) ? :jsonb : :json
+  end
+end

data/db/migrate/20260922000001_add_solid_objects_redrives.rb

@@ -0,0 +1,32 @@
+# rbs_inline: enabled
+
+class AddSolidObjectsRedrives < ActiveRecord::Migration[7.1]
+  # @rbs () -> void
+  def change
+    create_table SolidObjects.table_name(:redrives), id: :string, limit: 64 do |definition|
+      definition.string :kind, null: false, limit: 32
+      definition.public_send(json_type, :filters, null: false)
+      definition.string :status, null: false, default: "running", limit: 32
+      definition.string :active_scope, limit: 191
+      definition.integer :moved, null: false, default: 0
+      definition.integer :move_limit
+      definition.string :actor, limit: 255
+      definition.datetime :started_at, null: false, precision: 6
+      definition.datetime :finished_at, precision: 6
+      definition.timestamps precision: 6, null: false
+
+      definition.index :active_scope, unique: true, name: "idx_so_redrives_active_scope"
+      definition.index [ :status, :started_at, :id ], name: "idx_so_redrives_poll"
+      definition.check_constraint "moved >= 0", name: "chk_so_redrives_moved"
+      definition.check_constraint "move_limit IS NULL OR move_limit > 0", name: "chk_so_redrives_limit"
+      definition.check_constraint "status IN ('running', 'completed', 'cancelled')", name: "chk_so_redrives_status"
+    end
+  end
+
+  private
+
+  # @rbs () -> Symbol
+  def json_type
+    connection.adapter_name.match?(/postgres/i) ? :jsonb : :json
+  end
+end

data/db/migrate/20260923000000_add_solid_objects_completed_idempotency_keys.rb

@@ -0,0 +1,10 @@
+# rbs_inline: enabled
+
+class AddSolidObjectsCompletedIdempotencyKeys < ActiveRecord::Migration[7.1]
+  # @rbs () -> void
+  def change
+    add_column SolidObjects.table_name(:instances),
+      :completed_idempotency_keys,
+      connection.adapter_name.match?(/postgres/i) ? :jsonb : :json
+  end
+end

data/docs/adr/0011-wake-up-strategy.md

@@ -25,6 +25,14 @@ The interface supports:
 
 MySQL uses polling or optional Redis. SQLite uses polling plus the in-process signal; multi-host SQLite is outside its supported operating model.
 
+Selection is automatic. `wake_up_adapter` takes a name or an adapter and
+defaults to `:automatic`, which prefers a configured Redis URL, then PostgreSQL
+notifications, then polling. PostgreSQL is chosen only after a probe
+notification arrives, because `LISTEN` does not survive a transaction pooler. A
+requested adapter that the environment cannot provide polls and records why,
+rather than claim a wake-up it cannot deliver. `SolidObjects.wake_up.capability`
+reports the choice, and the doctor reports the same record.
+
 The synchronous caller first attempts to claim and execute the actor locally,
 so the normal path has no worker polling leg. When another process owns the
 activation, coordination overhead from completion commit until the caller's
@@ -45,5 +53,7 @@ Timeout does not cancel durable work.
 - Redis loss only increases latency and never loses durable work.
 - Every adapter retains periodic polling to close startup, reconnect, and missed-message races.
 - A process that returns `false` from a timed wait participates in backoff; an older custom adapter that returns `nil` keeps the fast cadence.
-- A multi-process deployment without an adapter trades idle database load for up to the current idle polling interval of notification latency and logs that topology once.
+- A multi-process deployment whose installed adapter cannot cross processes trades idle database load for up to the current idle polling interval of notification latency and logs that topology once.
+- A PostgreSQL deployment that configures nothing now pays one `NOTIFY` per enqueue after the commit and one listening connection per waiting thread, outside the pool.
+- Selection runs once per process, under a lock, because the probe opens connections and waits.
 - Notification payloads never contain actor arguments or results.

data/docs/architecture.md

@@ -72,7 +72,7 @@ through the client.
 
 ### Client and mailbox
 
-The client finds or creates the actor instance and atomically allocates a sequence. It inserts one durable message-history row and one ready-membership row. It validates operations and JSON payloads before writing and enforces idempotency-key uniqueness, payload limits, and the per-actor mailbox cap. It also authorizes and coordinates actor destruction. Distributed rate limiting and global admission control are not implemented.
+The client finds or creates the actor instance and atomically allocates a sequence. It inserts one durable message-history row and one ready-membership row. It validates operations and JSON payloads before writing and enforces idempotency-key uniqueness, payload limits, and the per-actor mailbox cap. It also authorizes and coordinates actor destruction. Distributed rate limiting and global admission control are not implemented and are not planned here.
 
 Message execution state is table membership, not a status column. The durable message remains for results, retention, and diagnostics. Only live work occupies `ready_messages` or `claimed_messages`, so completed history cannot inflate the polling index.
 
@@ -424,6 +424,42 @@ outer commit, and callers timing out on work they indirectly block.
 waiting and immediately returns a `MessageReference`. Runtime workers process
 it normally.
 
+A caller that loses that reference rebuilds one. `SolidObjects.client.find_by`
+answers a request id, which is unique across the table, and
+`reference.find_by` answers an idempotency key, which is unique per instance.
+Each lookup runs the authorization hook the original call ran, against the
+stored operation and arguments, and answers `nil` for an absent row, an
+unregistered actor, and a refused caller alike, so it cannot be used to ask
+whether a request id exists. `MessageReference#outcome` reports the status, the
+result, the persisted error, the rejection, and the attempt count. A result is
+stored for `sync` delivery only.
+
+An actor remembers the idempotency keys of its own finished turns. The executor
+already writes the instance row in the transaction that completes, rejects, or
+kills a turn, so the memory rides on a write that happens anyway. This is the
+Orleans answer: a grain keeps its deduplication history in grain state rather
+than in a separate tombstone table, which needs no second store, no second
+write, and no separate retention. `reference.find_by(idempotency_key:)` raises
+`MessagePruned` for a key the actor remembers and whose message retention
+removed, and answers `nil` for a key no caller ever sent, so a client can tell
+a lost result from a request that never arrived. An actor remembers the operation and original arguments beside each key, so
+the pruned answer runs the same hook against the same operation and arguments that a lookup
+of the surviving row would, and a caller the policy refuses reads `nil` whether
+the message is pruned or never existed. Gating it on `snapshot` instead would
+tell a caller who may read state, but not the operation, that the operation had
+run.
+Remembered arguments count toward the serialized memory limit and remain until
+the entry is evicted or the instance is removed. Entries from older versions
+that lack arguments return absence after pruning because their original
+authorization cannot be reproduced.
+
+`retained_idempotency_keys` bounds the memory and defaults to 64 keys for each
+actor, and `retained_idempotency_keys_bytes` bounds its serialized size at 16 KB,
+because an idempotency key has no length limit on every adapter and the memory
+outlives the message row. An actor drops its oldest keys until the list fits. Only a lookup by idempotency key can make the distinction. A request id
+is generated by the runtime rather than by the caller, so no actor remembers
+one, and `client.find_by(request_id:)` answers `nil` in both cases.
+
 An executing caller receives an inline after-commit callback error even though
 the turn committed. An independently waiting caller observes the durable
 result and may return before that callback raises in the worker. Completed
@@ -750,11 +786,11 @@ Enqueue counts unfinished rows under the locked actor instance and rejects with
 
 ### Per-actor rate limits
 
-The initial implementation supplies the mailbox cap. Distributed token buckets or time-window counters are a hardening milestone.
+This runtime supplies the mailbox cap. Distributed token buckets and time-window counters are not planned here, because a request-path limiter is hot and loss-tolerant while every invocation writes one permanent message row. Solid Objects Pro answers that shape with grouped and ephemeral operations, which [fit](fit.md) describes.
 
 ### Global enqueue limits
 
-Global admission hooks are not implemented. A future hook can reject based on database health or application policy without introducing a strict global counter as a contention hotspot.
+Global admission hooks are not implemented and are not planned here, for the same reason as per-actor rate limits. A strict global counter would also be a contention hotspot. Reject on database health or application policy in front of the actor instead.
 
 ### Payload size
 
@@ -867,9 +903,9 @@ All backends use unique identity and sequence constraints, short transactions, a
 12. **How are leases renewed?** Conditional database update by instance, owner, generation, and unexpired lease.
 13. **How does graceful shutdown work?** Stop claims, finish current turn within timeout, release cached leases, stop heartbeat, mark process stopped.
 14. **How does synchronous invocation work across processes?** The caller first tries to claim and execute the actor locally. If another process owns it, a wake-up adapter prompts a durable result query and bounded polling remains the fallback.
-15. **What happens after caller timeout?** A committed message continues and its eventual result can be recovered with the timeout's authorized message reference. An enqueue timeout leaves no message. Running Ruby code is not preempted.
+15. **What happens after caller timeout?** A committed message continues and its eventual result can be recovered with the timeout's authorized message reference, or with `find_by` from the request id or the idempotency key when that reference is gone. An enqueue timeout leaves no message. Running Ruby code is not preempted.
 16. **How are results cleaned up?** `prune_messages` deletes eligible terminal history in bounded batches after global or per-actor retention. It previews by default and preserves live work, dead letters, retry links, and unfinished outboxes.
-17. **How are large mailboxes managed?** The implemented controls are the per-actor mailbox cap, payload caps, and fair activation yields; rate and global admission controls remain roadmap work.
+17. **How are large mailboxes managed?** The implemented controls are the per-actor mailbox cap, payload caps, and fair activation yields. Rate and global admission controls are not planned here; Solid Objects Pro answers that shape.
 18. **How are completed messages pruned?** Operators schedule the dry-run-reviewed `prune_messages --execute` command. Solid Objects does not run deletion automatically.
 19. **How are state migrations performed?** Explicit one-step actor migrations on activation, persisted only with a successful fenced commit.
 20. **What happens during rolling deploys?** Newer state can make old workers incompatible; deploys must preserve backward readability or drain old workers.

data/docs/dashboard.md

@@ -149,6 +149,16 @@ actor class that no longer exists, a full mailbox, a payload over the cap. The
 dashboard renders the dead letter again with the reason and a 422 status,
 rather than failing the request.
 
+Dead effects and broadcasts have the same API, which the dashboard does not yet
+surface. `SolidObjects.dead_letters.effects` and
+`SolidObjects.dead_letters.broadcasts` read and retry their own kind, and
+`redrive` moves a whole scope as a durable task. See
+[Operations](operations.md) for both.
+
+Every retry and every redrive transition writes one row to
+`solid_objects_administration_events`, holding the action, the kind, the
+subject, and the identity that asked for it.
+
 **Pause an instance** sets `paused_at`, and the activation manager stops
 claiming that identity. Two consequences matter:
 

data/docs/fit.md

@@ -68,6 +68,12 @@ a presence signal, or a view count. Reactive projections materialize a read
 model from the durable broadcast outbox, so request-path reads stop competing
 with mailbox work.
 
+Distributed per-actor rate limits, global admission control, and cache-capacity
+eviction are answered there rather than in this gem. Each one is hot and
+request-critical, and this gem writes one permanent message row for every
+invocation, so the cost model above rules out a limiter that checks on the
+request path. They are not open roadmap items here.
+
 ## Cost model
 
 Every synchronous or asynchronous invocation:

data/docs/operations.md

@@ -25,6 +25,22 @@ reports a failed or warned check rather than raising out of the command.
 
 ## Installing and upgrading
 
+Solid Objects keeps actor state, message arguments, results, and the remembered
+idempotency keys in JSON columns. Active Support decodes every one of them, and
+`ActiveSupport::JSON.decode` raises with the `json` gem at 3.0.2:
+
+```
+ArgumentError: wrong number of arguments (given 2, expected 1)
+```
+
+The failure is in Active Support rather than in Solid Objects, and it reaches
+every JSON column in a Rails application. A new Rails 8.1 application resolves
+`json` 3.0.2 today, so pin the 2.x series until Rails ships a fix:
+
+```ruby
+gem "json", "~> 2"
+```
+
 Review [CHANGELOG.md](CHANGELOG.md) for compatibility and deployment-order
 notes, then update the gem:
 
@@ -215,6 +231,8 @@ end
 | `instance_retention_by_actor_type` | `{}`; instances never expire unless listed |
 | `process_retention` | 7 days |
 | `prune_batch_size` | 1,000 |
+| `retained_idempotency_keys` | 64 |
+| `retained_idempotency_keys_bytes` | 16 KB |
 | `worker_count` | 1 |
 | `effect_worker_count` | 1 |
 | `broadcast_worker_count` | 1 |
@@ -245,13 +263,27 @@ to `idle_polling_interval`, which defaults to one second. Actor workers clamp
 the ceiling to `lease_renewal_interval` while they may hold cached activations.
 Set the fast and idle values equal for a fixed cadence.
 
-The default wake-up interrupts waits only in the current Ruby process. When a
-live process record shows that the database is shared across processes and no
-adapter is configured, the runtime logs
-`solid_objects.polling_only_cross_process_wake_up` once. Configure
-`WakeUpAdapters::Postgresql` or `WakeUpAdapters::Redis` when separate processes
-need prompt delivery. Without one, newly committed work can wait up to the
-current idle polling interval.
+Solid Objects selects a wake-up adapter on first use. `wake_up_adapter` defaults
+to `:automatic`, which prefers `SOLID_OBJECTS_REDIS_URL`, then PostgreSQL
+notifications, then polling. `SolidObjects.wake_up.capability` and the
+`wake_up` doctor check report what was installed, whether it crosses processes,
+its measured floor, and why.
+
+The in-process signal interrupts waits only in the current Ruby process. When a
+live process record shows that the database is shared across processes and the
+installed adapter does not cross them, the runtime logs
+`solid_objects.polling_only_cross_process_wake_up` once. Newly committed work
+can then wait up to the current idle polling interval.
+
+On PostgreSQL, selection proves the path first: it listens on a probe channel,
+notifies it from a second connection, and waits for the notification. A probe
+that does not arrive logs `solid_objects.wake_up.pooled_session` once and falls
+back to polling, because `LISTEN` does not survive a transaction pooler such as
+PgBouncer. A requested adapter that the environment cannot provide, such as
+`:postgresql` on MySQL or `:redis` without `SOLID_OBJECTS_REDIS_URL`, logs
+`solid_objects.wake_up.unavailable` once and polls rather than claim a
+cross-process wake-up that cannot happen. Only an unknown name is refused, and
+`configure` refuses it at boot.
 
 The warning excludes process rows with the current hostname and PID. It can
 therefore appear during a rolling deployment or restart overlap when an older
@@ -268,6 +300,95 @@ interval, and current interval. The polling-only warning is also emitted as
 adapters should return `true` for a notification and `false` for a timeout; an
 older adapter that returns `nil` remains compatible and keeps the fast cadence.
 
+## Dead letters, retry, and redrive
+
+A message that exhausts its attempts becomes a dead letter. An effect or a
+broadcast that exhausts its attempts stays in its own table with
+`status = 'dead'`. All three are read and retried through one receiver, which
+carries the kind:
+
+```ruby
+SolidObjects.dead_letters.all(authorization_context: current_admin)
+SolidObjects.dead_letters.retry(dead_letter_id, authorization_context: current_admin)
+
+SolidObjects.dead_letters.effects.all(authorization_context: current_admin)
+SolidObjects.dead_letters.effects.retry(effect_id, authorization_context: current_admin)
+SolidObjects.dead_letters.broadcasts.retry(broadcast_id, authorization_context: current_admin)
+```
+
+An effect or broadcast retry returns the row to pending with a zero attempt
+count, no claim, and immediate availability. It keeps the stable id, so a
+handler that deduplicates on `effect_id` still sees the same key. An effect is
+at-least-once by contract, so a retried effect can run twice.
+
+Retry acts only on a dead row. A row that is pending, processing, or completed
+comes back unchanged, so pressing a button twice cannot double-enqueue and
+cannot take a row away from a worker that holds it.
+
+An incident produces dead rows in the hundreds, so a scope also answers
+`redrive`:
+
+```ruby
+task = SolidObjects.dead_letters.effects.redrive(
+  actor_type: "payments",
+  failed_after: 6.hours.ago,
+  limit: 5_000,
+  authorization_context: current_admin
+)
+
+task.id         # => "redrive_..."
+task.status     # => "running"
+task.moved      # => 412
+task.remaining  # => 4_588
+
+task.cancel(authorization_context: current_admin)
+```
+
+`redrive` returns at once. The task is durable, and the supervisor advances one
+bounded batch per pass, so a redrive of thousands of rows never holds a
+transaction longer than one batch. `redrive_batch_size` defaults to 100 and
+`redrive_batch_pause` to 0.05 seconds.
+
+A redrive moves the rows that were already dead when it started. A row that
+fails again lands back in the same scope, and without that bound a task whose
+handler is still broken would move it forever.
+
+A redrive is idempotent over its scope and its filters. Starting the same one
+while it runs returns the running task rather than a second one, which a
+dashboard button an operator can press twice needs. A different scope or a
+different filter starts its own task, and the same scope can be redriven again
+once the first task finishes.
+
+Read tasks back with `SolidObjects.redrives`:
+
+```ruby
+SolidObjects.redrives.find(task.id, authorization_context: current_admin)
+SolidObjects.redrives.all(status: :running, authorization_context: current_admin)
+```
+
+A running task reports what is left to move rather than a stored estimate,
+because rows die and are retried while it runs.
+
+Retry, redrive, and cancel each go through `authorize_administration` under
+their own resource name: `dead_letters`, `effect_dead_letters`,
+`broadcast_dead_letters`, and `redrives`. Every retry and every task transition
+writes one row to `solid_objects_administration_events`, holding the action, the
+kind, the subject, the identity, and when it happened. The identity comes from
+`administration_identity`, which receives the authorization context the caller
+passed and defaults to its `to_s`.
+
+An event records an authorized press, not a state transition. Pressing retry
+twice writes two rows, because an operator did two things and a log that shows
+one cannot answer who pressed what. The row the event names carries the outcome.
+A refused caller writes nothing, and a retry that raises after the lookup writes
+nothing, because the event shares the transaction with the work. The redrive
+transitions are different: `redrive.start`, `redrive.finish`, and
+`redrive.cancel` are written only when the task actually changes.
+
+Automatic redrive on a schedule is deliberately absent. A dead row means a
+person decided something, and these APIs give that person an alternative to an
+`UPDATE` against a runtime table.
+
 ## Graceful shutdown
 
 The supervisor requests shutdown, stops new claims, lets active loops return,
@@ -414,6 +535,8 @@ SolidObjects.configure do |configuration|
   }
   configuration.process_retention = 7.days
   configuration.prune_batch_size = 1_000
+  configuration.retained_idempotency_keys = 64
+  configuration.retained_idempotency_keys_bytes = 16.kilobytes
 end
 ```
 
@@ -440,6 +563,29 @@ broadcasts, and other message-owned rows. Choose a cutoff longer than every
 `sync` timeout because a caller whose result row disappears can no longer
 observe it.
 
+`find_by` reads the same rows, so a lookup answers only while the message it
+names survives retention. A lookup by idempotency key still tells the two cases
+apart after pruning, because the actor remembers the keys of its own last
+`retained_idempotency_keys` finished turns: it raises `MessagePruned` for a key
+the actor remembers and answers `nil` for a key no caller ever sent. The actor
+remembers the operation and original arguments beside each key, so the pruned answer runs the same
+authorization the surviving row would. Raise
+`retained_idempotency_keys` above the default of 64 when an actor finishes more
+keyed turns than that inside the window in which a caller may retry. A lookup
+by request id answers `nil` in both cases, so a caller that must tell them apart
+sends its own idempotency key.
+
+Remembered arguments count toward the serialized memory limit and remain until
+the entry is evicted or the instance is removed. Entries from older versions
+that lack arguments return absence after pruning because their original
+authorization cannot be reproduced.
+
+`retained_idempotency_keys_bytes` bounds the serialized memory as well, because
+an idempotency key has no length limit on every adapter and the memory outlives
+the message row. An actor drops its oldest keys until the list fits, so a key
+long enough to fill the limit by itself is never remembered and its lookup
+answers `nil` rather than raising.
+
 Actor expiration is disabled by default. `prune_instances` considers only
 actor types listed in `instance_retention_by_actor_type`, excludes active or
 paused actors, and preserves ready/claimed mailbox work, scheduled reminders,