prosody 0.4.0 → 0.5.0

data/README.md

@@ -30,10 +30,10 @@ Or install directly:
 gem install prosody
 ```
 
-The gem ships RBS signatures for the public API. `Prosody::EventHandler[Payload]`
-carries an application payload type into `Prosody::Message[Payload]`, and keyed-
-state definitions carry their item types through `context.state`. A bare handler,
-message, definition, or state handle defaults to `Prosody::json_value`. See the
+The gem ships RBS signatures for the public API. `Prosody::EventHandler[Payload, Response]`
+carries the payload type into `Prosody::Message[Payload]`. It also checks each handler response.
+State definitions carry their item types through `context.state`. A bare handler,
+message, definition, or state handle uses `Prosody::json_value`. See the
 [typed examples](examples/) for Ruby and companion RBS files checked by Steep.
 
 ## Quick Start
@@ -59,6 +59,12 @@ client = Prosody::Client.new(
 
 # Define a custom message handler
 class MyHandler < Prosody::EventHandler
+  def on_excise(context, message)
+    puts "Excise key: #{message.key}"
+    context.clear_scheduled
+    nil
+  end
+
   def on_message(context, message)
     # Process the received message
     puts "Received message: #{message.payload.inspect}"
@@ -81,11 +87,22 @@ client.subscribe(MyHandler.new)
 
 # Send a message to a topic
 client.send_message("my-topic", "message-key", {"content" => "Hello, Kafka!"})
+client.excise("my-topic", "obsolete-key")
 
 # Ensure proper shutdown when done
-client.unsubscribe
+client.shutdown
 ```
 
+## Excise records
+
+Applications can copy event data into keyed state and external stores. A regulatory or contractual deletion must remove every copy for one key.
+
+An excise record carries this deletion command. Kafka encodes the command as a key with no payload. During topic compaction, Kafka deletes earlier values for the key. Call `excise(topic, key)` to send the record. Prosody routes the record to `on_excise`. The handler must delete all consumer-owned data for the key.
+
+Each handler must implement `on_message`, `on_excise`, and `on_timer`. Subscription fails before consumption if a method is missing.
+
+If an excise record is a request, return a response from `on_excise`. Prosody uses this response as the subsystem result.
+
 ## Architecture
 
 Prosody enables efficient, parallel processing of Kafka messages while maintaining order for messages with the same key:
@@ -173,134 +190,9 @@ timeout defaults to 80% of `stall_threshold`.
 
 ## Configuration
 
-Configure via constructor options or environment variables. Options fall back to environment variables when unset.
-
-### Core
-
-| Option / Environment Variable           | Description                                       | Default      |
-|-----------------------------------------|---------------------------------------------------|--------------|
-| `bootstrap_servers` / `PROSODY_BOOTSTRAP_SERVERS` | Kafka servers to connect to             | -            |
-| `group_id` / `PROSODY_GROUP_ID`         | Consumer group name                               | -            |
-| `subscribed_topics` / `PROSODY_SUBSCRIBED_TOPICS` | Topics to read from                     | -            |
-| `allowed_events` / `PROSODY_ALLOWED_EVENTS` | Only process events matching these prefixes   | (all)        |
-| `source_system` / `PROSODY_SOURCE_SYSTEM` | Tag for outgoing messages (prevents reprocessing)| `<group_id>` |
-| `mock` / `PROSODY_MOCK`                 | Use in-memory Kafka for testing                   | false        |
-
-### Consumer
-
-| Option / Environment Variable           | Description                                          | Default                |
-|-----------------------------------------|------------------------------------------------------|------------------------|
-| `max_concurrency` / `PROSODY_MAX_CONCURRENCY` | Max messages being processed simultaneously     | 32                     |
-| `max_uncommitted` / `PROSODY_MAX_UNCOMMITTED` | Max queued messages before pausing consumption  | 64                     |
-| `timeout` / `PROSODY_TIMEOUT`           | Cancel handler if it runs longer than this           | 80% of stall threshold |
-| `commit_interval` / `PROSODY_COMMIT_INTERVAL` | How often to save progress to Kafka            | 1s                     |
-| `poll_interval` / `PROSODY_POLL_INTERVAL` | How often to fetch new messages from Kafka         | 100ms                  |
-| `shutdown_timeout` / `PROSODY_SHUTDOWN_TIMEOUT` | Shutdown budget; handlers run freely until cancellation fires near the end of the timeout | 30s |
-| `stall_threshold` / `PROSODY_STALL_THRESHOLD` | Report unhealthy if no progress for this long  | 5m                     |
-| `probe_port` / `PROSODY_PROBE_PORT`     | HTTP port for health checks (nil to disable)         | 8000                   |
-| `failure_topic` / `PROSODY_FAILURE_TOPIC` | Send unprocessable messages here (dead letter queue) | -                     |
-| `idempotence_cache_size` / `PROSODY_IDEMPOTENCE_CACHE_SIZE` | Global shared cache capacity across all partitions for message deduplication. Consumer deduplication is mandatory and cannot be disabled, so this must be at least 1; setting it to 0 in the client configuration is rejected | 8192 |
-| `idempotence_version` / `PROSODY_IDEMPOTENCE_VERSION` | Version string for cache-busting dedup hashes | 1              |
-| `idempotence_ttl` / `PROSODY_IDEMPOTENCE_TTL`         | TTL for dedup records in Cassandra            | 7d (604800 seconds) |
-| `slab_size` / `PROSODY_SLAB_SIZE`       | Timer storage granularity (rarely needs changing)    | 1h                     |
-| `message_spans` / `PROSODY_MESSAGE_SPANS` | Span linking for message execution: `child` (child-of) or `follows_from` | `child` |
-| `timer_spans` / `PROSODY_TIMER_SPANS`   | Span linking for timer execution: `child` (child-of) or `follows_from`   | `follows_from` |
-
-### Producer
-
-| Option / Environment Variable           | Description                     | Default |
-|-----------------------------------------|---------------------------------|---------|
-| `send_timeout` / `PROSODY_SEND_TIMEOUT` | Give up sending after this long | 1s      |
-
-### Retry
-
-When a handler fails, retry with exponential backoff:
-
-| Option / Environment Variable           | Description                       | Default |
-|-----------------------------------------|-----------------------------------|---------|
-| `max_retries` / `PROSODY_MAX_RETRIES`   | Give up after this many attempts  | 3       |
-| `retry_base` / `PROSODY_RETRY_BASE`     | Wait this long before first retry | 20ms    |
-| `max_retry_delay` / `PROSODY_RETRY_MAX_DELAY` | Never wait longer than this  | 5m      |
-
-### Deferral (Pipeline Mode)
-
-| Option / Environment Variable           | Description                                       | Default |
-|-----------------------------------------|---------------------------------------------------|---------|
-| `defer_enabled` / `PROSODY_DEFER_ENABLED` | Enable deferral for new messages                | true    |
-| `defer_base` / `PROSODY_DEFER_BASE`     | Wait this long before first deferred retry        | 1s      |
-| `defer_max_delay` / `PROSODY_DEFER_MAX_DELAY` | Never wait longer than this                 | 24h     |
-| `defer_failure_threshold` / `PROSODY_DEFER_FAILURE_THRESHOLD` | Disable deferral when failure rate exceeds this | 0.9 |
-| `defer_failure_window` / `PROSODY_DEFER_FAILURE_WINDOW` | Measure failure rate over this time window | 5m     |
-| `defer_cache_size` / `PROSODY_DEFER_CACHE_SIZE` | Track this many deferred keys in memory     | 1024    |
-| `defer_store_cache_size` / `PROSODY_DEFER_STORE_CACHE_SIZE` | Maximum deferred store cache entries per Cassandra defer store | 8192 |
-| `defer_seek_timeout` / `PROSODY_DEFER_SEEK_TIMEOUT` | Timeout when loading deferred messages    | 30s     |
-| `defer_discard_threshold` / `PROSODY_DEFER_DISCARD_THRESHOLD` | Read optimization (rarely needs changing) | 100  |
-
-### Monopolization Detection (Pipeline Mode)
-
-| Option / Environment Variable           | Description                             | Default |
-|-----------------------------------------|-----------------------------------------|---------|
-| `monopolization_enabled` / `PROSODY_MONOPOLIZATION_ENABLED` | Enable hot key protection   | true    |
-| `monopolization_threshold` / `PROSODY_MONOPOLIZATION_THRESHOLD` | Max handler time as fraction of window | 0.9 |
-| `monopolization_window` / `PROSODY_MONOPOLIZATION_WINDOW` | Measurement window            | 5m      |
-| `monopolization_cache_size` / `PROSODY_MONOPOLIZATION_CACHE_SIZE` | Max distinct keys to track  | 8192    |
-
-### Fair Scheduling (All Modes)
-
-| Option / Environment Variable           | Description                                                      | Default |
-|-----------------------------------------|------------------------------------------------------------------|---------|
-| `scheduler_failure_weight` / `PROSODY_SCHEDULER_FAILURE_WEIGHT` | Fraction of processing time reserved for retries | 0.3    |
-| `scheduler_max_wait` / `PROSODY_SCHEDULER_MAX_WAIT` | Messages waiting this long get maximum priority          | 2m      |
-| `scheduler_wait_weight` / `PROSODY_SCHEDULER_WAIT_WEIGHT` | Priority boost for waiting messages (higher = more aggressive) | 200.0 |
-| `scheduler_cache_size` / `PROSODY_SCHEDULER_CACHE_SIZE` | Max distinct keys to track                             | 8192    |
-
-### Cassandra
-
-Persistent storage for timers and deferred retries (not needed if `mock: true`):
+For the complete configuration reference, see [CONFIGURATION.md](CONFIGURATION.md).
 
-| Option / Environment Variable           | Description                        | Default |
-|-----------------------------------------|------------------------------------|---------|
-| `cassandra_nodes` / `PROSODY_CASSANDRA_NODES` | Servers to connect to (host:port) | -      |
-| `cassandra_keyspace` / `PROSODY_CASSANDRA_KEYSPACE` | Keyspace name              | prosody |
-| `cassandra_user` / `PROSODY_CASSANDRA_USER` | Username                          | -       |
-| `cassandra_password` / `PROSODY_CASSANDRA_PASSWORD` | Password                   | -       |
-| `cassandra_datacenter` / `PROSODY_CASSANDRA_DATACENTER` | Prefer this datacenter for queries | - |
-| `cassandra_rack` / `PROSODY_CASSANDRA_RACK` | Prefer this rack for queries      | -       |
-| `cassandra_retention` / `PROSODY_CASSANDRA_RETENTION` | Delete data older than this | 1y     |
-
-### Keyed State
-
-Register keyed-state collections before you subscribe. Persistence is backed by Cassandra and is not needed when `mock: true`. See the [Keyed State](#keyed-state-1) feature section for handler usage; the client-level knobs and per-collection fields are below. Where an option and an environment variable are paired, an explicitly set option wins; otherwise the environment variable applies, then the default.
-
-| Option / Environment Variable | Description | Default |
-|-------------------------------|-------------|---------|
-| `state_collections` / - | Keyed-state collections to register before subscribe (array of definitions or config hashes; duplicate names rejected) | (none) |
-| `state_cache_dir` / `PROSODY_STATE_CACHE_DIR` | Disk workspace for the local keyed-state cache; each live client needs its own directory (it is locked exclusively) | per-client temp dir |
-| `state_cache_size_bytes` / `PROSODY_STATE_CACHE_SIZE_BYTES` | Capacity of the in-memory keyed-state cache, in bytes; must be greater than 0. One cache is shared by all partition keyspaces | engine default |
-| `state_recovery_delay` / `PROSODY_STATE_RECOVERY_DELAY` | Whole-second delay between staging a provisional cell and the recovery sweep; every collection TTL must strictly exceed it | 30s |
-
-Prefer the definition constructors (`Prosody.value` / `.map` / `.deque` and their `message_*` variants, documented below): they serialize into `state_collections` so you declare each collection once and reuse the same object with `context.state`. Each entry has these fields:
-
-| Field | Description | Default |
-|-------|-------------|---------|
-| `name` | Collection name; non-empty and unique within the client | (required) |
-| `kind` | `"value"`, `"map"`, or `"deque"` | (required) |
-| `payload` | `"json"` (JSON values) or `"message"` (the full Kafka message the handler received) | (required) |
-| `ttl_seconds` | Per-write TTL in whole seconds (at least 1; must exceed the recovery delay) | (none) |
-| `read_uncommitted` | Opt out of transactional staging | false |
-| `keyset_limit` | Map-only; ordered-scan bound in `0..=4096` (`0` disables ordered-scan tracking) | 128 |
-| `capacity` | Deque-only window bound (at least 1); keeps at most N slots, enforced lazily on push. Runtime-only and mutable across deploys — not persisted | unbounded |
-
-Constructors set these via keyword arguments (`ttl:`, `keyset_limit:`, `capacity:`, `read_uncommitted:`).
-
-### Telemetry Emitter
-
-Prosody can emit internal processing events (message lifecycle, timer events) to a Kafka topic for observability:
-
-| Option / Environment Variable           | Description                                    | Default                    |
-|-----------------------------------------|------------------------------------------------|----------------------------|
-| `telemetry_topic` / `PROSODY_TELEMETRY_TOPIC` | Kafka topic to produce telemetry events to | `prosody.telemetry-events` |
-| `telemetry_enabled` / `PROSODY_TELEMETRY_ENABLED` | Enable or disable the telemetry emitter  | true                       |
+Constructor options take precedence. Unset options use environment variables, then library defaults.
 
 ## Logging
 
@@ -393,6 +285,73 @@ if client.is_stalled?
 end
 ```
 
+## Subsystems
+
+A consumer group ID identifies a set of processes that share records and the keyed state that the group owns. A subsystem can include one or more services and consumer groups. If callers use these IDs, a refactor can require changes to each caller.
+
+A subsystem gives requests and published state one stable public name. Callers use this name instead of consumer group IDs. You can change its services and consumer groups without changing callers. Prosody uses the first response to a subsystem request. For each published-state read, it uses one consumer group that publishes the collection.
+
+## Requests
+
+Kafka decouples producers from consumers, so a send does not return consumer results. This asynchronous model lets each service process records independently. Some operations must wait for consumer results before they continue. A request recovers synchrony for the caller while consumers continue asynchronous processing.
+
+Send a request from a handler or other application code. The Prosody client does not need an active subscription. The result hash uses canonical subsystem names as keys. Each value is a `Success` or `Failure` outcome. Use `request_excise` to send an excise record and collect the same outcome type.
+
+Do not rely on hash order. The hash contains one entry for each selected subsystem. A missing response becomes a timeout `Failure`; Prosody does not omit the subsystem. The request raises an error for request-level failures, such as invalid input, a Kafka send failure, or shutdown. Do not wait for a request if the current consumer group must process it for the same key. That group cannot process it until the handler returns.
+
+Message and excise handler return values become successful outcomes. Each return value must have a JSON representation.
+
+Set `subsystem` to `inventory` on the client that subscribes this handler.
+
+```ruby
+class InventoryHandler < Prosody::EventHandler
+  def on_message(_context, message)
+    {"accepted" => message.key}
+  end
+
+  def on_excise(_context, message)
+    {"excised" => message.key}
+  end
+
+  def on_timer(_context, _timer)
+  end
+end
+```
+
+Send the request:
+
+Set `timeout` in seconds.
+
+```ruby
+subsystems = ["inventory", "billing"]
+results = client.request(
+  topic: "orders",
+  key: "order-1",
+  payload: {"type" => "order.created"},
+  subsystems: subsystems,
+  timeout: 2.0
+)
+
+results.each do |subsystem, outcome|
+  if outcome.is_a?(Prosody::Failure)
+    warn "#{subsystem}: #{outcome.error.message}"
+  else
+    puts "#{subsystem}: #{outcome.value}"
+  end
+end
+```
+
+The example can print these results:
+
+```text
+inventory: {"accepted"=>"order-1"}
+billing: no response arrived before the deadline
+```
+
+Each failure contains one typed response error.
+
+Each response error has one message.
+
 ## Advanced Usage
 
 ### Pipeline Mode
@@ -564,15 +523,17 @@ Note that the in-memory cache is best-effort. Duplicates can still occur across
 
 ## Keyed State
 
-Keyed state gives every Kafka key its own durable working memory. Prosody automatically uses the current message or timer key, so a handler can relate the current event to earlier events for that key. State survives restarts and rebalances. By default, changes become visible only when the event succeeds.
+Many stream transformations must reason across multiple events or timer firings. Windows, state machines, aggregates, and complex event processing all require state.
+
+A Kafka key identifies an entity, such as a customer or order. Keyed state gives each key independent working state for these transformations. With Cassandra, the state survives restarts and partition reassignment.
 
-Use keyed state for time-aware stream processing: counters, deduplication, rolling aggregates, pending work, and per-key workflows. Keep your relational database as the source of truth for business data and for work that needs joins or ad hoc queries. Reconstructing stream state with repeated database queries can be slow and expensive; keyed state is built for that job.
+Prosody selects the current message or timer key. It processes one event at a time for that key but can process other keys concurrently. By default, Prosody commits pending keyed-state changes only when the handler succeeds. If the handler returns an error, Prosody discards those changes.
 
-Most collections should have a TTL. Set it comfortably beyond the longest timer or workflow that uses the state; Prosody validates the minimum supported TTL. Omit it only when keeping inactive keys forever is intentional.
+Give most collections a time to live (TTL). Set the TTL beyond the longest timer or workflow that uses the collection. Omit it when state must remain for inactive keys.
 
 ### A counter for each key
 
-Declare each collection once, register it on the client, and ask the event context for the current key's state:
+Declare each collection once. Register it on the client. In a handler, get the current key's state from the event context:
 
 ```ruby
 COUNTER = Prosody.value("counter", ttl: 30 * 24 * 60 * 60)
@@ -582,6 +543,12 @@ class CountHandler < Prosody::EventHandler
     count = context.state(COUNTER)
     count.set((count.get || 0) + 1)
   end
+
+  def on_excise(context, _message)
+    context.state(COUNTER).clear
+    nil
+  end
+  def on_timer(_context, _timer); end
 end
 
 client = Prosody::Client.new(
@@ -591,11 +558,13 @@ client = Prosody::Client.new(
 )
 ```
 
-Here, counters expire after 30 days without an update.
+Each Kafka key now has an independent counter. A counter expires when that key has no update for 30 days.
 
 ### Window activity into one notification
 
-This example turns a burst of activity into two useful notifications. It sends the first event immediately, collects later events for five minutes, then sends one summary. Because the user ID is the Kafka key, every user gets an independent window.
+This example sends the first event for a user immediately. It collects later events for five minutes and then sends one summary.
+
+The user ID is the Kafka key. Each user has an independent window.
 
 ```ruby
 WINDOW = Prosody.value("window", ttl: 24 * 60 * 60)
@@ -625,6 +594,13 @@ class ActivityHandler < Prosody::EventHandler
     pending.clear
     context.state(WINDOW).clear
   end
+
+  def on_excise(context, _message)
+    context.state(PENDING).clear
+    context.state(WINDOW).clear
+    context.clear_scheduled
+    nil
+  end
 end
 ```
 
@@ -632,18 +608,20 @@ See the complete, Steep-checked example for signatures, client setup, and `notif
 
 Why this works:
 
-- Register both definitions in `state_collections` before subscribing. Keyed state uses Cassandra unless `mock: true`.
+- Register both definitions in `state_collections` before you subscribe. Keyed state uses Cassandra unless `mock: true`.
 - Use `clear_and_schedule`, not `schedule`, so a retried event does not add another timer for the same key.
-- `capacity: 100` and the one-day TTL prevent an inactive or unusually busy key from retaining an unlimited backlog. Since this example only appends, overflow drops the oldest saved message.
-- A `message_deque` requires the original Kafka messages to remain available for the whole window. Use a plain `deque` of payloads if topic retention or compaction cannot guarantee that.
+- `capacity: 100` and the one-day TTL bound the saved backlog. Overflow drops the oldest message because this example only appends.
+- A `message_deque` requires the original Kafka messages during the window. Use `deque` when topic retention or compaction cannot provide them.
 - Prosody runs one handler at a time for each key, so a user's message and timer handlers cannot overlap.
-- Sending a notification is outside Prosody's state transaction and may happen again after a retry. Give notifications a stable idempotency key, or send them through an outbox, when duplicates matter.
+- A notification is outside the state transaction. A retry can send it again. Use a stable operation ID to reject duplicate notifications.
 
 ### Collections and handles
 
-A definition gives a collection a stable name, kind, and options. Register it once on the client, then pass the same definition to `context.state` to access the current key. Do not reuse a persisted name for a different collection kind or payload type.
+A definition sets a collection's durable name, kind, and options. Register it once. Pass it to `context.state` in a handler.
+
+Do not reuse a durable name for a different collection kind or payload type. Create handles inside the handler. Do not retain handles or iterators.
 
-Create handles inside the handler and do not retain them or their iterators afterward. State operations look synchronous but yield the current fiber while Prosody performs the work.
+State operations look synchronous. They yield the current fiber while Prosody performs the work.
 
 | Collection | JSON payload | Kafka message | Main operations |
 | --- | --- | --- | --- |
@@ -651,17 +629,60 @@ Create handles inside the handler and do not retain them or their iterators afte
 | Ordered string map | `Prosody.map` | `Prosody.message_map` | `get`, `get_many`, `key?`, `set`, `delete`, `each_pair`, `each_key`, `clear` |
 | Deque | `Prosody.deque` | `Prosody.message_deque` | `push`, `unshift`, `pop`, `shift`, `get`, `length`, `each`, `clear` |
 
-Map and deque scans return enumerators when called without a block. Map keys are strings. `nil` means absence and cannot be stored—use `clear` or `delete` instead.
+Map and deque scans return enumerators when called without a block. Map keys are strings.
+
+`nil` means absence. Do not store this value. Use `clear` or `delete`.
+
+### When keyed-state changes become visible
 
-### When changes become visible
+By default, retries do not see pending state from a failed attempt. Reads in a handler see its earlier keyed-state writes. Prosody commits pending changes when the event succeeds and discards them when the handler raises.
 
-Reads inside a handler see its earlier writes. The default behavior is the safest choice for most handlers: Prosody buffers those changes and publishes them together when the event succeeds. If the handler raises, none of its pending changes become visible.
+This transaction applies only to keyed state. Some workflows need state changes before the handler ends, so each collection also provides explicit controls:
 
-Each collection also offers explicit controls for workflows that need different behavior:
+- `read_uncommitted: true` persists keyed-state changes before Prosody records the event as complete. If the process stops between these steps, Prosody can process the same event again. The retry sees state changes from the earlier attempt. You must make these keyed-state changes idempotent. Each retry must produce the same state.
+- `commit` commits the collection's pending changes before the handler ends. A later handler failure does not remove them.
+- `rollback` discards pending changes since the last `commit`. It cannot undo committed changes.
 
-- `read_uncommitted: true` writes that collection's changes after the handler succeeds but before the event is recorded as complete. A crash in between can leave the changes visible even though the event is retried. Use it only for idempotent changes, where processing the same event again produces the same stored result.
-- `commit` immediately publishes this collection's pending changes. They remain visible even if the handler later raises and the event is retried.
-- `rollback` discards this collection's pending changes since its last `commit`. It cannot undo changes that were already committed.
+### Published state
+
+Some callers need only the current value for a key. They can accept a stale value or a race with a concurrent update.
+
+Use topics and event sourcing when a consumer must process each state change in order. Use published state for direct, read-only lookup of persisted keyed state. The caller does not need to consume the owner's topics or maintain a separate lookup store.
+
+Configure the subsystem name on each publisher. Enable publication on the collection definition. Register the definition on the Prosody client:
+
+```ruby
+CURRENT_ORDER = Prosody.value("current-order", published: true)
+
+owner = Prosody::Client.new(
+  group_id: "order-writer",
+  subsystem: "checkout",
+  state_collections: [CURRENT_ORDER]
+)
+
+# The handler uses the key from its current event.
+current_order = context.state(CURRENT_ORDER)
+current_order.set({"sku" => "book"})
+```
+
+Read published state from a handler or other application code. The Prosody client does not need an active subscription.
+
+Use the subsystem and the same definition to open a reader:
+
+```ruby
+order_reader = client.state("checkout", CURRENT_ORDER)
+current_order = order_reader.get("customer-123")
+```
+
+The reader cannot see pending changes that exist only in a handler. It cannot change the collection. Each read takes an explicit key because no handler supplies one.
+
+Map and deque readers fetch data in chunks. They do not load the complete collection before iteration starts. Readers return an `Enumerator` without a block.
+
+Use `reverse_each_pair`, `reverse_each_key`, `reverse_each_value`, or `reverse_each` for reverse traversal.
+
+The default cache window is five seconds. Set `read_cache:` to select a different window. Set `read_cache: false` to bypass the cache.
+
+To stop publication, deploy the definition with `published: false`. Keep the definition registered during that deployment. Keep the subsystem configured during that deployment.
 
 ## Timer Functionality
 
@@ -690,6 +711,11 @@ class MyHandler < Prosody::EventHandler
     puts "Key: #{timer.key}"
     puts "Scheduled time: #{timer.time}"
   end
+
+  def on_excise(context, _message)
+    context.clear_scheduled
+    nil
+  end
 end
 ```
 
@@ -814,6 +840,9 @@ class MyHandler < Prosody::EventHandler
       })
     end
   end
+
+  def on_excise(_context, _message); end
+  def on_timer(_context, _timer); end
 end
 ```
 
@@ -868,22 +897,17 @@ Strategies for achieving idempotence:
    - Each message advances the state machine, allowing for idempotent processing and easy failure recovery.
    - Particularly useful for complex, distributed transactions across multiple services.
 
-### Proper Shutdown
+### Application shutdown
+
+A Prosody client runs a subscription, timers, and other services in the background. Before an application terminates, it must stop all client services. `unsubscribe` stops only the active subscription.
 
-Always unsubscribe from topics before exiting your application:
+Call `shutdown` when the application terminates. It stops all client services and rejects new operations. Call `unsubscribe` only when the application will use the client again. You do not need to call `unsubscribe` before `shutdown`.
 
 ```ruby
-# Ensure proper shutdown
-client.unsubscribe
+client.shutdown
 ```
 
-This ensures:
-
-1. Completion and commitment of all in-flight work
-2. Quick rebalancing, allowing other consumers to take over partitions
-3. Proper release of resources
-
-Implement shutdown handling in your application using signal handlers:
+Handle application shutdown with signal handlers:
 
 ```ruby
 require "prosody"
@@ -894,22 +918,22 @@ client = Prosody::Client.new(
   subscribed_topics: "my-topic"
 )
 
-# Set up a shutdown queue
+# Create the shutdown queue.
 shutdown = Queue.new
 
-# Configure signal handlers to trigger shutdown
+# Register the signal handlers.
 Signal.trap("INT") { shutdown.push(nil) }
 Signal.trap("TERM") { shutdown.push(nil) }
 
-# Subscribe to messages
+# Subscribe with the application handler.
 client.subscribe(MyHandler.new)
 
-# Block until a signal is received
-shutdown.pop # This blocks until something is pushed to the queue by a signal handler
+# Wait for a shutdown signal.
+shutdown.pop
 
-# Clean shutdown
-puts "Shutting down gracefully..."
-client.unsubscribe
+# Shut down the client.
+puts "Client shutdown starts."
+client.shutdown
 ```
 
 ### Error Handling
@@ -917,6 +941,8 @@ client.unsubscribe
 Prosody classifies errors as transient (temporary, can be retried) or permanent (won't be resolved by retrying). By
 default, all errors are considered transient.
 
+The error classes and classification methods apply to `on_message`, `on_excise`, and `on_timer`.
+
 Use the `Prosody::EventHandler` error classification methods:
 
 ```ruby
@@ -933,6 +959,9 @@ class MyHandler < Prosody::EventHandler
     # JSON::ParserError will be treated as transient
     # All other exceptions will be treated as transient (default behavior)
   end
+
+  def on_excise(_context, _message); end
+  def on_timer(_context, _timer); end
 end
 ```
 
@@ -970,6 +999,9 @@ class MyHandler < Prosody::EventHandler
       release_resource(resource)
     end
   end
+
+  def on_excise(_context, _message); end
+  def on_timer(_context, _timer); end
 end
 ```
 
@@ -1031,15 +1063,26 @@ Ensure you have thoroughly tested your changes before merging to `main`.
 
 ### Prosody::Client
 
-- `new(**config)`: Initialize a new Prosody client with the given configuration.
+- `new(config)` or `new(**options)`: Create a client from a `Configuration`, hash, or keyword options.
 - `send_message(String topic, String key, Prosody::json_value payload)`: Send a JSON-serializable message.
-- `consumer_state`: Get the current state of the consumer (`:unconfigured`, `:configured`, or `:running`).
+- `excise(String topic, String key)`: Send an excise record for a key.
+- `request(topic:, key:, payload:, subsystems:, timeout:)`: Return one outcome for each subsystem.
+- `request_excise(topic:, key:, subsystems:, timeout:)`: Return one excise outcome for each subsystem.
+- `consumer_state`: Get the client state (`:shut_down`, `:unconfigured`, `:configured`, or `:running`).
 - `source_system`: Get the source system identifier configured for the client.
-- `subscribe: [Payload] (Prosody::EventHandler[Payload]) -> void`: Subscribe while preserving the handler's payload specialization.
-- `unsubscribe`: Unsubscribe from messages and shut down the consumer.
+- `state(subsystem, definition)`: Open a typed, read-only published value, map, or deque.
+- `subscribe(handler)`: Start event processing with the specified handler.
+- `unsubscribe`: Stop the consumer. You can subscribe again later.
+- `shutdown`: Stop all client services. Concurrent and repeated calls wait for the same operation.
 - `assigned_partitions`: Get the number of partitions currently assigned to this consumer.
 - `is_stalled?`: Check if the consumer has stalled partitions.
 
+### Prosody::AdminClient
+
+- `new(bootstrap_servers)`: Create an admin client for the specified Kafka servers.
+- `create_topic(name, partitions, replication_factor)`: Create a Kafka topic.
+- `delete_topic(name)`: Delete a Kafka topic.
+
 ### Prosody::EventHandler
 
 A base class for user-defined handlers. Its RBS payload parameter flows into
@@ -1058,9 +1101,15 @@ class MyHandler < Prosody::EventHandler
   def on_timer(context, timer)
     # Implement your timer handling logic here
   end
+
+  def on_excise(_context, _message)
+    # Implement your excise handling logic here
+  end
 end
 ```
 
+`on_message`, `on_excise`, and `on_timer` are the handler callbacks. The `permanent` and `transient` methods classify selected exceptions.
+
 ### Prosody::Message
 
 `Prosody::Message[Payload]` represents a Kafka message. `Payload` defaults to
@@ -1073,9 +1122,11 @@ your application's RBS:
 
 ```rbs
 type order_event = { "order_id" => String, "total" => Integer }
+type response = { "accepted" => bool }
 
-class OrderHandler < Prosody::EventHandler[order_event]
-  def on_message: (Prosody::Context, Prosody::Message[order_event]) -> void
+class OrderHandler < Prosody::EventHandler[order_event, response]
+  def on_message: (Prosody::Context, Prosody::Message[order_event]) -> response
+  def on_excise: (Prosody::Context, Prosody::ExciseMessage) -> response
 end
 ```
 
@@ -1094,13 +1145,17 @@ Messages have the following attributes:
 - `key` (String): The message key.
 - `payload` (`Payload`): The JSON-deserialized message payload.
 
+### Prosody::ExciseMessage
+
+An `ExciseMessage` has `topic`, `partition`, `offset`, `timestamp`, and `key` attributes. It has no `payload` attribute.
+
 ### Prosody::Context
 
-Represents the context of message processing:
+Represents the current event context:
 
 - `should_cancel?`: Check if cancellation has been requested (includes timeout and shutdown).
-- `on_cancel`: Blocks until cancellation is signaled.
-- `state(definition)`: Binds a registered collection for the current event attempt, returning a typed handle (`ValueState`, `MapState`, or `DequeState`). Raises `PermanentStateError` when the name was never registered, or when the definition's `kind`/`payload` disagrees with the collection's durably-registered schema. See the [Keyed State](#keyed-state-2) API reference below.
+- `on_cancel`: Wait until cancellation occurs.
+- `state(definition)`: Bind a registered collection for the current attempt. An unregistered or mismatched definition raises `PermanentStateError`. See [Keyed State](#keyed-state-2).
 
 Timer scheduling methods:
 
@@ -1117,35 +1172,79 @@ Represents a timer that has fired, provided to the `on_timer` method:
 - `key` (String): The entity key identifying what this timer belongs to
 - `time` (Time): The time when this timer was scheduled to fire
 
+### Requests
+
+- `Prosody::Success[Value]`: Contains the response in `value`.
+- `Prosody::Failure`: Contains a response error in `error`.
+- `Prosody::HandlerError`, `Timeout`, `FormatMismatch`, and `MalformedResponse`: The possible response errors.
+- `Prosody::response_error`: The union of all response error types.
+- `Prosody::outcome[Value]`: A `Success[Value]` or `Failure`.
+
 ### Keyed State
 
 Definition constructors (each returns a frozen definition object used both in `Configuration#state_collections` and with `context.state`):
 
-- `Prosody.value(name, ttl: nil, read_uncommitted: nil)`
-- `Prosody.map(name, ttl: nil, keyset_limit: nil, read_uncommitted: nil)`
-- `Prosody.deque(name, ttl: nil, read_uncommitted: nil)`
+- `Prosody.value(name, ttl: nil, read_uncommitted: nil, published: nil, read_cache: nil)`
+- `Prosody.map(name, ttl: nil, keyset_limit: nil, read_uncommitted: nil, published: nil, read_cache: nil)`
+- `Prosody.deque(name, ttl: nil, capacity: nil, read_uncommitted: nil, published: nil, read_cache: nil)`
 - `Prosody.message_value(name, ttl: nil, read_uncommitted: nil)`
 - `Prosody.message_map(name, ttl: nil, keyset_limit: nil, read_uncommitted: nil)`
-- `Prosody.message_deque(name, ttl: nil, read_uncommitted: nil)`
+- `Prosody.message_deque(name, ttl: nil, capacity: nil, read_uncommitted: nil)`
+
+Each constructor returns a `StateDefinition`. It exposes `name`, `kind`, `payload`, all supplied options, and `to_state_config`.
+
+Published readers take the user key as their first argument. `Prosody::PublishedValue` provides `get`.
+
+`Prosody::PublishedMap` provides `get`, `get_many`, `key?`, `has_key?`, `include?`, and `member?`.
+
+It provides `each` or `each_pair`, `each_key`, and `each_value`. The reverse methods are `reverse_each_pair`, `reverse_each_key`, and `reverse_each_value`.
+
+`Prosody::PublishedDeque` provides `get`, `length` or `size`, `empty?`, `first`, `last`, `each`, and `reverse_each`.
+
+Traversal methods return an `Enumerator` without a block.
 
 `Prosody::ValueState`:
 
-- `get`, `set(value)`, `clear`, `commit`, `rollback`
+- `get` / `value`, `set(value)` / `value=`, `clear`, `commit`, and `rollback`
 
 `Prosody::MapState` (keys are `String`):
 
-- `get(key)`, `get_many(keys)`, `set(key, value)`, `delete(key)` (returns `nil`), `clear`
-- `each_pair` / `reverse_each_pair` (block or `Enumerator`), `commit`, `rollback`
+- `get` / `[]`, `get_many`, `set` / `[]=`, `store`, `delete`, and `clear`
+- `key?`, `has_key?`, `include?`, `member?`, `dig`, `slice`, `values_at`, `fetch`, and `fetch_values`
+- `each` / `each_pair`, `each_key`, and `each_value`, including each reverse form
+- `commit` and `rollback`
 
 `Prosody::DequeState`:
 
-- `push(value)`, `unshift(value)`, `pop`, `shift`, `length` (aliased `size`), `empty?`, `get(index)`, `clear`
-- `each` / `reverse_each` (block or `Enumerator`), `commit`, `rollback`
+- `push`, `append`, `<<`, `unshift`, `prepend`, `pop`, and `shift`
+- `length` / `size`, `empty?`, `get`, `fetch`, `first`, `last`, and `clear`
+- `each` / `reverse_each`, `commit`, and `rollback`
 
 Errors:
 
-- `Prosody::TransientStateError < Prosody::TransientError`: the default — a temporary store read/write failure, or any caller mistake (a `nil`/unrepresentable write, item-shape mismatch, out-of-range index, invalid scan direction), rejected transient so it retries rather than discarding the message.
-- `Prosody::PermanentStateError < Prosody::PermanentError`: reserved for failures a retry cannot resolve in-process (unregistered/identity-mismatched collection, duplicate registration, bad TTL), or one a handler raises explicitly.
+- `Prosody::TransientStateError < Prosody::TransientError`: Reports a keyed-state error that Prosody can retry.
+- `Prosody::PermanentStateError < Prosody::PermanentError`: Reports a keyed-state error that another attempt cannot resolve.
 - `Prosody::NullValueError < Prosody::TransientStateError`: raised when a `nil` is written; use `clear`/`delete` instead.
 
-State errors are never Terminal (core folds Terminal into Transient).
+Handler error types:
+
+- `Prosody::Error`: Base Prosody error.
+- `Prosody::EventHandlerError`: Base class for classified handler errors.
+- `Prosody::TransientError`: Marks an error as retriable.
+- `Prosody::PermanentError`: Marks an error as final.
+
+### Configuration
+
+`Prosody::Configuration.new` accepts a hash or block. Its public properties match the settings in [Configuration](CONFIGURATION.md). `to_hash` returns a configuration hash.
+
+### Logging and telemetry
+
+- `Prosody.logger`: Get the current logger or create the default logger.
+- `Prosody.logger=`: Replace the current logger. Assign `nil` to restore the default.
+- `Prosody.flush_telemetry`: Export pending telemetry.
+- `Prosody.shutdown_telemetry`: Export pending telemetry and stop its providers.
+
+### Sentry
+
+- `Prosody::SentryIntegration.enabled?`: Test whether Sentry integration is active.
+- `Prosody::SentryIntegration.capture_exception(exception, context = {})`: Report an exception with optional context.