prosody 0.3.0 → 0.5.0

data/README.md

@@ -9,6 +9,7 @@ strategies, and integrated OpenTelemetry support for distributed tracing.
 - **Kafka Consumer**: Per-key ordering with cross-key concurrency, offset management, consumer groups
 - **Kafka Producer**: Idempotent delivery with configurable retries
 - **Timer System**: Persistent scheduled execution backed by Cassandra or in-memory store
+- **Keyed State**: Per-key value/map/deque collections that survive across events, transactional by default
 - **Quality of Service**: Fair scheduling limits concurrency and prevents failures from starving fresh traffic. Pipeline mode adds deferred retry and monopolization detection
 - **Distributed Tracing**: OpenTelemetry integration for tracing message flow across services
 - **Backpressure**: Pauses partitions when handlers fall behind
@@ -29,6 +30,12 @@ Or install directly:
 gem install prosody
 ```
 
+The gem ships RBS signatures for the public API. `Prosody::EventHandler[Payload, Response]`
+carries the payload type into `Prosody::Message[Payload]`. It also checks each handler response.
+State definitions carry their item types through `context.state`. A bare handler,
+message, definition, or state handle uses `Prosody::json_value`. See the
+[typed examples](examples/) for Ruby and companion RBS files checked by Steep.
+
 ## Quick Start
 
 ```ruby
@@ -52,6 +59,12 @@ client = Prosody::Client.new(
 
 # Define a custom message handler
 class MyHandler < Prosody::EventHandler
+  def on_excise(context, message)
+    puts "Excise key: #{message.key}"
+    context.clear_scheduled
+    nil
+  end
+
   def on_message(context, message)
     # Process the received message
     puts "Received message: #{message.payload.inspect}"
@@ -73,12 +86,23 @@ end
 client.subscribe(MyHandler.new)
 
 # Send a message to a topic
-client.send_message("my-topic", "message-key", { content: "Hello, Kafka!" })
+client.send_message("my-topic", "message-key", {"content" => "Hello, Kafka!"})
+client.excise("my-topic", "obsolete-key")
 
 # Ensure proper shutdown when done
-client.unsubscribe
+client.shutdown
 ```
 
+## Excise records
+
+Applications can copy event data into keyed state and external stores. A regulatory or contractual deletion must remove every copy for one key.
+
+An excise record carries this deletion command. Kafka encodes the command as a key with no payload. During topic compaction, Kafka deletes earlier values for the key. Call `excise(topic, key)` to send the record. Prosody routes the record to `on_excise`. The handler must delete all consumer-owned data for the key.
+
+Each handler must implement `on_message`, `on_excise`, and `on_timer`. Subscription fails before consumption if a method is missing.
+
+If an excise record is a request, return a response from `on_excise`. Prosody uses this response as the subsystem result.
+
 ## Architecture
 
 Prosody enables efficient, parallel processing of Kafka messages while maintaining order for messages with the same key:
@@ -166,109 +190,9 @@ timeout defaults to 80% of `stall_threshold`.
 
 ## Configuration
 
-Configure via constructor options or environment variables. Options fall back to environment variables when unset.
-
-### Core
-
-| Option / Environment Variable           | Description                                       | Default      |
-|-----------------------------------------|---------------------------------------------------|--------------|
-| `bootstrap_servers` / `PROSODY_BOOTSTRAP_SERVERS` | Kafka servers to connect to             | -            |
-| `group_id` / `PROSODY_GROUP_ID`         | Consumer group name                               | -            |
-| `subscribed_topics` / `PROSODY_SUBSCRIBED_TOPICS` | Topics to read from                     | -            |
-| `allowed_events` / `PROSODY_ALLOWED_EVENTS` | Only process events matching these prefixes   | (all)        |
-| `source_system` / `PROSODY_SOURCE_SYSTEM` | Tag for outgoing messages (prevents reprocessing)| `<group_id>` |
-| `mock` / `PROSODY_MOCK`                 | Use in-memory Kafka for testing                   | false        |
-
-### Consumer
-
-| Option / Environment Variable           | Description                                          | Default                |
-|-----------------------------------------|------------------------------------------------------|------------------------|
-| `max_concurrency` / `PROSODY_MAX_CONCURRENCY` | Max messages being processed simultaneously     | 32                     |
-| `max_uncommitted` / `PROSODY_MAX_UNCOMMITTED` | Max queued messages before pausing consumption  | 64                     |
-| `timeout` / `PROSODY_TIMEOUT`           | Cancel handler if it runs longer than this           | 80% of stall threshold |
-| `commit_interval` / `PROSODY_COMMIT_INTERVAL` | How often to save progress to Kafka            | 1s                     |
-| `poll_interval` / `PROSODY_POLL_INTERVAL` | How often to fetch new messages from Kafka         | 100ms                  |
-| `shutdown_timeout` / `PROSODY_SHUTDOWN_TIMEOUT` | Shutdown budget; handlers run freely until cancellation fires near the end of the timeout | 30s |
-| `stall_threshold` / `PROSODY_STALL_THRESHOLD` | Report unhealthy if no progress for this long  | 5m                     |
-| `probe_port` / `PROSODY_PROBE_PORT`     | HTTP port for health checks (nil to disable)         | 8000                   |
-| `failure_topic` / `PROSODY_FAILURE_TOPIC` | Send unprocessable messages here (dead letter queue) | -                     |
-| `idempotence_cache_size` / `PROSODY_IDEMPOTENCE_CACHE_SIZE` | Global shared cache capacity across all partitions for message deduplication (0 disables the entire deduplication middleware, both in-memory and persistent) | 8192 |
-| `idempotence_version` / `PROSODY_IDEMPOTENCE_VERSION` | Version string for cache-busting dedup hashes | 1              |
-| `idempotence_ttl` / `PROSODY_IDEMPOTENCE_TTL`         | TTL for dedup records in Cassandra            | 7d (604800 seconds) |
-| `slab_size` / `PROSODY_SLAB_SIZE`       | Timer storage granularity (rarely needs changing)    | 1h                     |
-| `message_spans` / `PROSODY_MESSAGE_SPANS` | Span linking for message execution: `child` (child-of) or `follows_from` | `child` |
-| `timer_spans` / `PROSODY_TIMER_SPANS`   | Span linking for timer execution: `child` (child-of) or `follows_from`   | `follows_from` |
-
-### Producer
-
-| Option / Environment Variable           | Description                     | Default |
-|-----------------------------------------|---------------------------------|---------|
-| `send_timeout` / `PROSODY_SEND_TIMEOUT` | Give up sending after this long | 1s      |
-
-### Retry
-
-When a handler fails, retry with exponential backoff:
-
-| Option / Environment Variable           | Description                       | Default |
-|-----------------------------------------|-----------------------------------|---------|
-| `max_retries` / `PROSODY_MAX_RETRIES`   | Give up after this many attempts  | 3       |
-| `retry_base` / `PROSODY_RETRY_BASE`     | Wait this long before first retry | 20ms    |
-| `max_retry_delay` / `PROSODY_RETRY_MAX_DELAY` | Never wait longer than this  | 5m      |
-
-### Deferral (Pipeline Mode)
-
-| Option / Environment Variable           | Description                                       | Default |
-|-----------------------------------------|---------------------------------------------------|---------|
-| `defer_enabled` / `PROSODY_DEFER_ENABLED` | Enable deferral for new messages                | true    |
-| `defer_base` / `PROSODY_DEFER_BASE`     | Wait this long before first deferred retry        | 1s      |
-| `defer_max_delay` / `PROSODY_DEFER_MAX_DELAY` | Never wait longer than this                 | 24h     |
-| `defer_failure_threshold` / `PROSODY_DEFER_FAILURE_THRESHOLD` | Disable deferral when failure rate exceeds this | 0.9 |
-| `defer_failure_window` / `PROSODY_DEFER_FAILURE_WINDOW` | Measure failure rate over this time window | 5m     |
-| `defer_cache_size` / `PROSODY_DEFER_CACHE_SIZE` | Track this many deferred keys in memory     | 1024    |
-| `defer_store_cache_size` / `PROSODY_DEFER_STORE_CACHE_SIZE` | Maximum deferred store cache entries per Cassandra defer store | 8192 |
-| `defer_seek_timeout` / `PROSODY_DEFER_SEEK_TIMEOUT` | Timeout when loading deferred messages    | 30s     |
-| `defer_discard_threshold` / `PROSODY_DEFER_DISCARD_THRESHOLD` | Read optimization (rarely needs changing) | 100  |
-
-### Monopolization Detection (Pipeline Mode)
-
-| Option / Environment Variable           | Description                             | Default |
-|-----------------------------------------|-----------------------------------------|---------|
-| `monopolization_enabled` / `PROSODY_MONOPOLIZATION_ENABLED` | Enable hot key protection   | true    |
-| `monopolization_threshold` / `PROSODY_MONOPOLIZATION_THRESHOLD` | Max handler time as fraction of window | 0.9 |
-| `monopolization_window` / `PROSODY_MONOPOLIZATION_WINDOW` | Measurement window            | 5m      |
-| `monopolization_cache_size` / `PROSODY_MONOPOLIZATION_CACHE_SIZE` | Max distinct keys to track  | 8192    |
+For the complete configuration reference, see [CONFIGURATION.md](CONFIGURATION.md).
 
-### Fair Scheduling (All Modes)
-
-| Option / Environment Variable           | Description                                                      | Default |
-|-----------------------------------------|------------------------------------------------------------------|---------|
-| `scheduler_failure_weight` / `PROSODY_SCHEDULER_FAILURE_WEIGHT` | Fraction of processing time reserved for retries | 0.3    |
-| `scheduler_max_wait` / `PROSODY_SCHEDULER_MAX_WAIT` | Messages waiting this long get maximum priority          | 2m      |
-| `scheduler_wait_weight` / `PROSODY_SCHEDULER_WAIT_WEIGHT` | Priority boost for waiting messages (higher = more aggressive) | 200.0 |
-| `scheduler_cache_size` / `PROSODY_SCHEDULER_CACHE_SIZE` | Max distinct keys to track                             | 8192    |
-
-### Cassandra
-
-Persistent storage for timers and deferred retries (not needed if `mock: true`):
-
-| Option / Environment Variable           | Description                        | Default |
-|-----------------------------------------|------------------------------------|---------|
-| `cassandra_nodes` / `PROSODY_CASSANDRA_NODES` | Servers to connect to (host:port) | -      |
-| `cassandra_keyspace` / `PROSODY_CASSANDRA_KEYSPACE` | Keyspace name              | prosody |
-| `cassandra_user` / `PROSODY_CASSANDRA_USER` | Username                          | -       |
-| `cassandra_password` / `PROSODY_CASSANDRA_PASSWORD` | Password                   | -       |
-| `cassandra_datacenter` / `PROSODY_CASSANDRA_DATACENTER` | Prefer this datacenter for queries | - |
-| `cassandra_rack` / `PROSODY_CASSANDRA_RACK` | Prefer this rack for queries      | -       |
-| `cassandra_retention` / `PROSODY_CASSANDRA_RETENTION` | Delete data older than this | 1y     |
-
-### Telemetry Emitter
-
-Prosody can emit internal processing events (message lifecycle, timer events) to a Kafka topic for observability:
-
-| Option / Environment Variable           | Description                                    | Default                    |
-|-----------------------------------------|------------------------------------------------|----------------------------|
-| `telemetry_topic` / `PROSODY_TELEMETRY_TOPIC` | Kafka topic to produce telemetry events to | `prosody.telemetry-events` |
-| `telemetry_enabled` / `PROSODY_TELEMETRY_ENABLED` | Enable or disable the telemetry emitter  | true                       |
+Constructor options take precedence. Unset options use environment variables, then library defaults.
 
 ## Logging
 
@@ -361,6 +285,73 @@ if client.is_stalled?
 end
 ```
 
+## Subsystems
+
+A consumer group ID identifies a set of processes that share records and the keyed state that the group owns. A subsystem can include one or more services and consumer groups. If callers use these IDs, a refactor can require changes to each caller.
+
+A subsystem gives requests and published state one stable public name. Callers use this name instead of consumer group IDs. You can change its services and consumer groups without changing callers. Prosody uses the first response to a subsystem request. For each published-state read, it uses one consumer group that publishes the collection.
+
+## Requests
+
+Kafka decouples producers from consumers, so a send does not return consumer results. This asynchronous model lets each service process records independently. Some operations must wait for consumer results before they continue. A request recovers synchrony for the caller while consumers continue asynchronous processing.
+
+Send a request from a handler or other application code. The Prosody client does not need an active subscription. The result hash uses canonical subsystem names as keys. Each value is a `Success` or `Failure` outcome. Use `request_excise` to send an excise record and collect the same outcome type.
+
+Do not rely on hash order. The hash contains one entry for each selected subsystem. A missing response becomes a timeout `Failure`; Prosody does not omit the subsystem. The request raises an error for request-level failures, such as invalid input, a Kafka send failure, or shutdown. Do not wait for a request if the current consumer group must process it for the same key. That group cannot process it until the handler returns.
+
+Message and excise handler return values become successful outcomes. Each return value must have a JSON representation.
+
+Set `subsystem` to `inventory` on the client that subscribes this handler.
+
+```ruby
+class InventoryHandler < Prosody::EventHandler
+  def on_message(_context, message)
+    {"accepted" => message.key}
+  end
+
+  def on_excise(_context, message)
+    {"excised" => message.key}
+  end
+
+  def on_timer(_context, _timer)
+  end
+end
+```
+
+Send the request:
+
+Set `timeout` in seconds.
+
+```ruby
+subsystems = ["inventory", "billing"]
+results = client.request(
+  topic: "orders",
+  key: "order-1",
+  payload: {"type" => "order.created"},
+  subsystems: subsystems,
+  timeout: 2.0
+)
+
+results.each do |subsystem, outcome|
+  if outcome.is_a?(Prosody::Failure)
+    warn "#{subsystem}: #{outcome.error.message}"
+  else
+    puts "#{subsystem}: #{outcome.value}"
+  end
+end
+```
+
+The example can print these results:
+
+```text
+inventory: {"accepted"=>"order-1"}
+billing: no response arrived before the deadline
+```
+
+Each failure contains one typed response error.
+
+Each response error has one message.
+
 ## Advanced Usage
 
 ### Pipeline Mode
@@ -476,36 +467,37 @@ The deduplication system uses:
 ```ruby
 # Messages with IDs are deduplicated per key
 client.send_message("my-topic", "key1", {
-  id: "msg-123",      # Message will be processed
-  content: "Hello!"
+  "id" => "msg-123",      # Message will be processed
+  "content" => "Hello!"
 })
 
 client.send_message("my-topic", "key1", {
-  id: "msg-123",      # Message will be skipped (duplicate)
-  content: "Hello again!"
+  "id" => "msg-123",      # Message will be skipped (duplicate)
+  "content" => "Hello again!"
 })
 
 client.send_message("my-topic", "key2", {
-  id: "msg-123",      # Message will be processed (different key)
-  content: "Hello!"
+  "id" => "msg-123",      # Message will be processed (different key)
+  "content" => "Hello!"
 })
 ```
 
-Setting `idempotence_cache_size` to `0` disables the **entire** deduplication middleware (both the in-memory cache and the Cassandra-backed persistent store):
+Consumer deduplication is **mandatory** — it is the commit oracle that makes
+keyed state correct — so it cannot be disabled. `idempotence_cache_size` must be
+at least 1; setting it to `0` in the client configuration raises an
+`ArgumentError`:
 
 ```ruby
 client = Prosody::Client.new(
   group_id: "my-consumer-group",
   subscribed_topics: "my-topic",
-  idempotence_cache_size: 0  # Disable all deduplication (both in-memory and persistent)
+  idempotence_cache_size: 0  # Rejected: consumer deduplication cannot be disabled
 )
 ```
 
-Or via environment variable:
-
-```bash
-PROSODY_IDEMPOTENCE_CACHE_SIZE=0
-```
+This applies to every client — `Prosody::Client.new` always builds a consumer, so
+`0` is rejected regardless of whether any topics are subscribed, and whether it is
+supplied in the client configuration or via `PROSODY_IDEMPOTENCE_CACHE_SIZE`.
 
 To invalidate all previously recorded dedup entries (e.g. after a data migration), change the version string:
 
@@ -529,6 +521,169 @@ client = Prosody::Client.new(
 
 Note that the in-memory cache is best-effort. Duplicates can still occur across different process instances.
 
+## Keyed State
+
+Many stream transformations must reason across multiple events or timer firings. Windows, state machines, aggregates, and complex event processing all require state.
+
+A Kafka key identifies an entity, such as a customer or order. Keyed state gives each key independent working state for these transformations. With Cassandra, the state survives restarts and partition reassignment.
+
+Prosody selects the current message or timer key. It processes one event at a time for that key but can process other keys concurrently. By default, Prosody commits pending keyed-state changes only when the handler succeeds. If the handler returns an error, Prosody discards those changes.
+
+Give most collections a time to live (TTL). Set the TTL beyond the longest timer or workflow that uses the collection. Omit it when state must remain for inactive keys.
+
+### A counter for each key
+
+Declare each collection once. Register it on the client. In a handler, get the current key's state from the event context:
+
+```ruby
+COUNTER = Prosody.value("counter", ttl: 30 * 24 * 60 * 60)
+
+class CountHandler < Prosody::EventHandler
+  def on_message(context, _message)
+    count = context.state(COUNTER)
+    count.set((count.get || 0) + 1)
+  end
+
+  def on_excise(context, _message)
+    context.state(COUNTER).clear
+    nil
+  end
+  def on_timer(_context, _timer); end
+end
+
+client = Prosody::Client.new(
+  group_id: "counters",
+  subscribed_topics: "events",
+  state_collections: [COUNTER]
+)
+```
+
+Each Kafka key now has an independent counter. A counter expires when that key has no update for 30 days.
+
+### Window activity into one notification
+
+This example sends the first event for a user immediately. It collects later events for five minutes and then sends one summary.
+
+The user ID is the Kafka key. Each user has an independent window.
+
+```ruby
+WINDOW = Prosody.value("window", ttl: 24 * 60 * 60)
+PENDING = Prosody.message_deque("pending", capacity: 100, ttl: 24 * 60 * 60)
+
+class ActivityHandler < Prosody::EventHandler
+  def on_message(context, message)
+    window = context.state(WINDOW)
+    pending = context.state(PENDING)
+
+    if window.get
+      pending.push(message)
+      return
+    end
+
+    notify(message.key, [message])
+    window.set(true)
+    context.clear_and_schedule(Time.now + 5 * 60)
+  end
+
+  def on_timer(context, timer)
+    pending = context.state(PENDING)
+    batch = []
+    pending.each { |message| batch << message }
+
+    notify(timer.key, batch) unless batch.empty?
+    pending.clear
+    context.state(WINDOW).clear
+  end
+
+  def on_excise(context, _message)
+    context.state(PENDING).clear
+    context.state(WINDOW).clear
+    context.clear_scheduled
+    nil
+  end
+end
+```
+
+See the complete, Steep-checked example for signatures, client setup, and `notify`: [`examples/keyed_state_windowing.rb`](examples/keyed_state_windowing.rb) and [`examples/keyed_state_windowing.rbs`](examples/keyed_state_windowing.rbs).
+
+Why this works:
+
+- Register both definitions in `state_collections` before you subscribe. Keyed state uses Cassandra unless `mock: true`.
+- Use `clear_and_schedule`, not `schedule`, so a retried event does not add another timer for the same key.
+- `capacity: 100` and the one-day TTL bound the saved backlog. Overflow drops the oldest message because this example only appends.
+- A `message_deque` requires the original Kafka messages during the window. Use `deque` when topic retention or compaction cannot provide them.
+- Prosody runs one handler at a time for each key, so a user's message and timer handlers cannot overlap.
+- A notification is outside the state transaction. A retry can send it again. Use a stable operation ID to reject duplicate notifications.
+
+### Collections and handles
+
+A definition sets a collection's durable name, kind, and options. Register it once. Pass it to `context.state` in a handler.
+
+Do not reuse a durable name for a different collection kind or payload type. Create handles inside the handler. Do not retain handles or iterators.
+
+State operations look synchronous. They yield the current fiber while Prosody performs the work.
+
+| Collection | JSON payload | Kafka message | Main operations |
+| --- | --- | --- | --- |
+| Value | `Prosody.value` | `Prosody.message_value` | `get`, `set`, `clear` |
+| Ordered string map | `Prosody.map` | `Prosody.message_map` | `get`, `get_many`, `key?`, `set`, `delete`, `each_pair`, `each_key`, `clear` |
+| Deque | `Prosody.deque` | `Prosody.message_deque` | `push`, `unshift`, `pop`, `shift`, `get`, `length`, `each`, `clear` |
+
+Map and deque scans return enumerators when called without a block. Map keys are strings.
+
+`nil` means absence. Do not store this value. Use `clear` or `delete`.
+
+### When keyed-state changes become visible
+
+By default, retries do not see pending state from a failed attempt. Reads in a handler see its earlier keyed-state writes. Prosody commits pending changes when the event succeeds and discards them when the handler raises.
+
+This transaction applies only to keyed state. Some workflows need state changes before the handler ends, so each collection also provides explicit controls:
+
+- `read_uncommitted: true` persists keyed-state changes before Prosody records the event as complete. If the process stops between these steps, Prosody can process the same event again. The retry sees state changes from the earlier attempt. You must make these keyed-state changes idempotent. Each retry must produce the same state.
+- `commit` commits the collection's pending changes before the handler ends. A later handler failure does not remove them.
+- `rollback` discards pending changes since the last `commit`. It cannot undo committed changes.
+
+### Published state
+
+Some callers need only the current value for a key. They can accept a stale value or a race with a concurrent update.
+
+Use topics and event sourcing when a consumer must process each state change in order. Use published state for direct, read-only lookup of persisted keyed state. The caller does not need to consume the owner's topics or maintain a separate lookup store.
+
+Configure the subsystem name on each publisher. Enable publication on the collection definition. Register the definition on the Prosody client:
+
+```ruby
+CURRENT_ORDER = Prosody.value("current-order", published: true)
+
+owner = Prosody::Client.new(
+  group_id: "order-writer",
+  subsystem: "checkout",
+  state_collections: [CURRENT_ORDER]
+)
+
+# The handler uses the key from its current event.
+current_order = context.state(CURRENT_ORDER)
+current_order.set({"sku" => "book"})
+```
+
+Read published state from a handler or other application code. The Prosody client does not need an active subscription.
+
+Use the subsystem and the same definition to open a reader:
+
+```ruby
+order_reader = client.state("checkout", CURRENT_ORDER)
+current_order = order_reader.get("customer-123")
+```
+
+The reader cannot see pending changes that exist only in a handler. It cannot change the collection. Each read takes an explicit key because no handler supplies one.
+
+Map and deque readers fetch data in chunks. They do not load the complete collection before iteration starts. Readers return an `Enumerator` without a block.
+
+Use `reverse_each_pair`, `reverse_each_key`, `reverse_each_value`, or `reverse_each` for reverse traversal.
+
+The default cache window is five seconds. Set `read_cache:` to select a different window. Set `read_cache: false` to bypass the cache.
+
+To stop publication, deploy the definition with `published: false`. Keep the definition registered during that deployment. Keep the subsystem configured during that deployment.
+
 ## Timer Functionality
 
 Prosody supports timer-based delayed execution within message handlers. When a timer fires, your handler's `on_timer` method will be called:
@@ -556,6 +711,11 @@ class MyHandler < Prosody::EventHandler
     puts "Key: #{timer.key}"
     puts "Scheduled time: #{timer.time}"
   end
+
+  def on_excise(context, _message)
+    context.clear_scheduled
+    nil
+  end
 end
 ```
 
@@ -680,6 +840,9 @@ class MyHandler < Prosody::EventHandler
       })
     end
   end
+
+  def on_excise(_context, _message); end
+  def on_timer(_context, _timer); end
 end
 ```
 
@@ -734,22 +897,17 @@ Strategies for achieving idempotence:
    - Each message advances the state machine, allowing for idempotent processing and easy failure recovery.
    - Particularly useful for complex, distributed transactions across multiple services.
 
-### Proper Shutdown
+### Application shutdown
+
+A Prosody client runs a subscription, timers, and other services in the background. Before an application terminates, it must stop all client services. `unsubscribe` stops only the active subscription.
 
-Always unsubscribe from topics before exiting your application:
+Call `shutdown` when the application terminates. It stops all client services and rejects new operations. Call `unsubscribe` only when the application will use the client again. You do not need to call `unsubscribe` before `shutdown`.
 
 ```ruby
-# Ensure proper shutdown
-client.unsubscribe
+client.shutdown
 ```
 
-This ensures:
-
-1. Completion and commitment of all in-flight work
-2. Quick rebalancing, allowing other consumers to take over partitions
-3. Proper release of resources
-
-Implement shutdown handling in your application using signal handlers:
+Handle application shutdown with signal handlers:
 
 ```ruby
 require "prosody"
@@ -760,22 +918,22 @@ client = Prosody::Client.new(
   subscribed_topics: "my-topic"
 )
 
-# Set up a shutdown queue
+# Create the shutdown queue.
 shutdown = Queue.new
 
-# Configure signal handlers to trigger shutdown
+# Register the signal handlers.
 Signal.trap("INT") { shutdown.push(nil) }
 Signal.trap("TERM") { shutdown.push(nil) }
 
-# Subscribe to messages
+# Subscribe with the application handler.
 client.subscribe(MyHandler.new)
 
-# Block until a signal is received
-shutdown.pop # This blocks until something is pushed to the queue by a signal handler
+# Wait for a shutdown signal.
+shutdown.pop
 
-# Clean shutdown
-puts "Shutting down gracefully..."
-client.unsubscribe
+# Shut down the client.
+puts "Client shutdown starts."
+client.shutdown
 ```
 
 ### Error Handling
@@ -783,6 +941,8 @@ client.unsubscribe
 Prosody classifies errors as transient (temporary, can be retried) or permanent (won't be resolved by retrying). By
 default, all errors are considered transient.
 
+The error classes and classification methods apply to `on_message`, `on_excise`, and `on_timer`.
+
 Use the `Prosody::EventHandler` error classification methods:
 
 ```ruby
@@ -799,6 +959,9 @@ class MyHandler < Prosody::EventHandler
     # JSON::ParserError will be treated as transient
     # All other exceptions will be treated as transient (default behavior)
   end
+
+  def on_excise(_context, _message); end
+  def on_timer(_context, _timer); end
 end
 ```
 
@@ -836,6 +999,9 @@ class MyHandler < Prosody::EventHandler
       release_resource(resource)
     end
   end
+
+  def on_excise(_context, _message); end
+  def on_timer(_context, _timer); end
 end
 ```
 
@@ -897,18 +1063,30 @@ Ensure you have thoroughly tested your changes before merging to `main`.
 
 ### Prosody::Client
 
-- `new(**config)`: Initialize a new Prosody client with the given configuration.
-- `send_message(topic, key, payload)`: Send a message to a specified topic.
-- `consumer_state`: Get the current state of the consumer (`:unconfigured`, `:configured`, or `:running`).
+- `new(config)` or `new(**options)`: Create a client from a `Configuration`, hash, or keyword options.
+- `send_message(String topic, String key, Prosody::json_value payload)`: Send a JSON-serializable message.
+- `excise(String topic, String key)`: Send an excise record for a key.
+- `request(topic:, key:, payload:, subsystems:, timeout:)`: Return one outcome for each subsystem.
+- `request_excise(topic:, key:, subsystems:, timeout:)`: Return one excise outcome for each subsystem.
+- `consumer_state`: Get the client state (`:shut_down`, `:unconfigured`, `:configured`, or `:running`).
 - `source_system`: Get the source system identifier configured for the client.
-- `subscribe(handler)`: Subscribe to messages using the provided handler.
-- `unsubscribe`: Unsubscribe from messages and shut down the consumer.
+- `state(subsystem, definition)`: Open a typed, read-only published value, map, or deque.
+- `subscribe(handler)`: Start event processing with the specified handler.
+- `unsubscribe`: Stop the consumer. You can subscribe again later.
+- `shutdown`: Stop all client services. Concurrent and repeated calls wait for the same operation.
 - `assigned_partitions`: Get the number of partitions currently assigned to this consumer.
 - `is_stalled?`: Check if the consumer has stalled partitions.
 
+### Prosody::AdminClient
+
+- `new(bootstrap_servers)`: Create an admin client for the specified Kafka servers.
+- `create_topic(name, partitions, replication_factor)`: Create a Kafka topic.
+- `delete_topic(name)`: Delete a Kafka topic.
+
 ### Prosody::EventHandler
 
-A base class for user-defined handlers:
+A base class for user-defined handlers. Its RBS payload parameter flows into
+`Message#payload`; a bare handler defaults to `Prosody::json_value`.
 
 ```ruby
 class MyHandler < Prosody::EventHandler
@@ -923,26 +1101,61 @@ class MyHandler < Prosody::EventHandler
   def on_timer(context, timer)
     # Implement your timer handling logic here
   end
+
+  def on_excise(_context, _message)
+    # Implement your excise handling logic here
+  end
 end
 ```
 
+`on_message`, `on_excise`, and `on_timer` are the handler callbacks. The `permanent` and `transient` methods classify selected exceptions.
+
 ### Prosody::Message
 
-Represents a Kafka message with the following attributes:
+`Prosody::Message[Payload]` represents a Kafka message. `Payload` defaults to
+`Prosody::json_value` (`nil`, booleans, numbers, strings, arrays, and
+string-keyed hashes, recursively). The parameter is static documentation and
+does not perform runtime validation.
+
+For a type-safe handler, describe the JSON record and specialize the handler in
+your application's RBS:
+
+```rbs
+type order_event = { "order_id" => String, "total" => Integer }
+type response = { "accepted" => bool }
+
+class OrderHandler < Prosody::EventHandler[order_event, response]
+  def on_message: (Prosody::Context, Prosody::Message[order_event]) -> response
+  def on_excise: (Prosody::Context, Prosody::ExciseMessage) -> response
+end
+```
+
+Ruby can then use `message.payload["order_id"]` as a `String` and
+`message.payload["total"]` as an `Integer`. See
+[`examples/keyed_state.rb`](examples/keyed_state.rb) and its companion
+[`examples/keyed_state.rbs`](examples/keyed_state.rbs) for payload typing that
+also flows through message-backed state.
+
+Messages have the following attributes:
 
 - `topic` (String): The name of the topic.
 - `partition` (Integer): The partition number.
 - `offset` (Integer): The message offset within the partition.
 - `timestamp` (Time): The timestamp when the message was created or sent.
 - `key` (String): The message key.
-- `payload` (Hash/Array/String): The message payload as a JSON-deserializable value.
+- `payload` (`Payload`): The JSON-deserialized message payload.
+
+### Prosody::ExciseMessage
+
+An `ExciseMessage` has `topic`, `partition`, `offset`, `timestamp`, and `key` attributes. It has no `payload` attribute.
 
 ### Prosody::Context
 
-Represents the context of message processing:
+Represents the current event context:
 
 - `should_cancel?`: Check if cancellation has been requested (includes timeout and shutdown).
-- `on_cancel`: Blocks until cancellation is signaled.
+- `on_cancel`: Wait until cancellation occurs.
+- `state(definition)`: Bind a registered collection for the current attempt. An unregistered or mismatched definition raises `PermanentStateError`. See [Keyed State](#keyed-state-2).
 
 Timer scheduling methods:
 
@@ -958,3 +1171,80 @@ Represents a timer that has fired, provided to the `on_timer` method:
 
 - `key` (String): The entity key identifying what this timer belongs to
 - `time` (Time): The time when this timer was scheduled to fire
+
+### Requests
+
+- `Prosody::Success[Value]`: Contains the response in `value`.
+- `Prosody::Failure`: Contains a response error in `error`.
+- `Prosody::HandlerError`, `Timeout`, `FormatMismatch`, and `MalformedResponse`: The possible response errors.
+- `Prosody::response_error`: The union of all response error types.
+- `Prosody::outcome[Value]`: A `Success[Value]` or `Failure`.
+
+### Keyed State
+
+Definition constructors (each returns a frozen definition object used both in `Configuration#state_collections` and with `context.state`):
+
+- `Prosody.value(name, ttl: nil, read_uncommitted: nil, published: nil, read_cache: nil)`
+- `Prosody.map(name, ttl: nil, keyset_limit: nil, read_uncommitted: nil, published: nil, read_cache: nil)`
+- `Prosody.deque(name, ttl: nil, capacity: nil, read_uncommitted: nil, published: nil, read_cache: nil)`
+- `Prosody.message_value(name, ttl: nil, read_uncommitted: nil)`
+- `Prosody.message_map(name, ttl: nil, keyset_limit: nil, read_uncommitted: nil)`
+- `Prosody.message_deque(name, ttl: nil, capacity: nil, read_uncommitted: nil)`
+
+Each constructor returns a `StateDefinition`. It exposes `name`, `kind`, `payload`, all supplied options, and `to_state_config`.
+
+Published readers take the user key as their first argument. `Prosody::PublishedValue` provides `get`.
+
+`Prosody::PublishedMap` provides `get`, `get_many`, `key?`, `has_key?`, `include?`, and `member?`.
+
+It provides `each` or `each_pair`, `each_key`, and `each_value`. The reverse methods are `reverse_each_pair`, `reverse_each_key`, and `reverse_each_value`.
+
+`Prosody::PublishedDeque` provides `get`, `length` or `size`, `empty?`, `first`, `last`, `each`, and `reverse_each`.
+
+Traversal methods return an `Enumerator` without a block.
+
+`Prosody::ValueState`:
+
+- `get` / `value`, `set(value)` / `value=`, `clear`, `commit`, and `rollback`
+
+`Prosody::MapState` (keys are `String`):
+
+- `get` / `[]`, `get_many`, `set` / `[]=`, `store`, `delete`, and `clear`
+- `key?`, `has_key?`, `include?`, `member?`, `dig`, `slice`, `values_at`, `fetch`, and `fetch_values`
+- `each` / `each_pair`, `each_key`, and `each_value`, including each reverse form
+- `commit` and `rollback`
+
+`Prosody::DequeState`:
+
+- `push`, `append`, `<<`, `unshift`, `prepend`, `pop`, and `shift`
+- `length` / `size`, `empty?`, `get`, `fetch`, `first`, `last`, and `clear`
+- `each` / `reverse_each`, `commit`, and `rollback`
+
+Errors:
+
+- `Prosody::TransientStateError < Prosody::TransientError`: Reports a keyed-state error that Prosody can retry.
+- `Prosody::PermanentStateError < Prosody::PermanentError`: Reports a keyed-state error that another attempt cannot resolve.
+- `Prosody::NullValueError < Prosody::TransientStateError`: raised when a `nil` is written; use `clear`/`delete` instead.
+
+Handler error types:
+
+- `Prosody::Error`: Base Prosody error.
+- `Prosody::EventHandlerError`: Base class for classified handler errors.
+- `Prosody::TransientError`: Marks an error as retriable.
+- `Prosody::PermanentError`: Marks an error as final.
+
+### Configuration
+
+`Prosody::Configuration.new` accepts a hash or block. Its public properties match the settings in [Configuration](CONFIGURATION.md). `to_hash` returns a configuration hash.
+
+### Logging and telemetry
+
+- `Prosody.logger`: Get the current logger or create the default logger.
+- `Prosody.logger=`: Replace the current logger. Assign `nil` to restore the default.
+- `Prosody.flush_telemetry`: Export pending telemetry.
+- `Prosody.shutdown_telemetry`: Export pending telemetry and stop its providers.
+
+### Sentry
+
+- `Prosody::SentryIntegration.enabled?`: Test whether Sentry integration is active.
+- `Prosody::SentryIntegration.capture_exception(exception, context = {})`: Report an exception with optional context.