prosody 0.2.1-aarch64-linux → 0.4.0-aarch64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.release-please-manifest.json +1 -1
- data/ARCHITECTURE.md +12 -2
- data/CHANGELOG.md +20 -0
- data/README.md +211 -20
- data/Rakefile +11 -1
- data/examples/keyed_state.rb +58 -0
- data/examples/keyed_state.rbs +18 -0
- data/examples/keyed_state_windowing.rb +47 -0
- data/examples/keyed_state_windowing.rbs +16 -0
- data/lib/prosody/3.2/prosody.so +0 -0
- data/lib/prosody/3.3/prosody.so +0 -0
- data/lib/prosody/3.4/prosody.so +0 -0
- data/lib/prosody/configuration.rb +26 -0
- data/lib/prosody/handler.rb +6 -2
- data/lib/prosody/native_stubs.rb +381 -6
- data/lib/prosody/state.rb +693 -0
- data/lib/prosody/version.rb +1 -1
- data/lib/prosody.rb +5 -0
- data/release-please-config.json +4 -0
- data/sig/configuration.rbs +24 -1
- data/sig/handler.rbs +7 -3
- data/sig/processor.rbs +28 -12
- data/sig/prosody.rbs +11 -6
- data/sig/sentry.rbs +6 -0
- data/sig/state.rbs +272 -0
- data/steep_expectations.yml +47 -0
- data/typecheck/payload_types.rb +43 -0
- data/typecheck/payload_types.rbs +20 -0
- data/typecheck_negative/payload_types.rb +16 -0
- data/typecheck_negative/payload_types.rbs +8 -0
- metadata +24 -12
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 498f89b95ccf9c87662ab2163556b1db149bf37f386a3a061d3051ae1df86fa8
|
|
4
|
+
data.tar.gz: 462bdecfcc73bf198fcd40d969906875a4bd8dbf3a512acf10111cfe0370b43d
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 8903a0da5e3474045a78cc0162d6226b8e924c177202e2471bb440eff0eba54f87d5426929fdc0fc8b0d6dab26367b256c1a322e4a552fb117171d4397794272
|
|
7
|
+
data.tar.gz: 4cc0848fa5d15a5fb350d4022349415d65fa30a55ea131425409f3471aae5da7b751025b974eb087ee2b16664ad607ba774c58fda69ac5ba3f1eac9fc2fa0781
|
data/ARCHITECTURE.md
CHANGED
|
@@ -79,6 +79,14 @@ end
|
|
|
79
79
|
|
|
80
80
|
When processing thousands of messages, yielding allows much higher throughput because work continues during waits.
|
|
81
81
|
|
|
82
|
+
### Keyed-State Scans Yield Per Step
|
|
83
|
+
|
|
84
|
+
Keyed-state traversal follows the same rule. `each` / `each_pair` (and their `reverse_*` variants) drive a native cursor
|
|
85
|
+
one chunk at a time: every step crosses the bridge through `Bridge::wait_for`, so the enumerating fiber yields while Rust
|
|
86
|
+
fetches the next chunk and resumes when it arrives. A scan therefore looks like an ordinary blocking loop but never blocks
|
|
87
|
+
the thread — other fibers keep running between steps — and the native cursor is closed via `ensure` when the loop
|
|
88
|
+
finishes, breaks, or raises.
|
|
89
|
+
|
|
82
90
|
## Architecture Overview
|
|
83
91
|
|
|
84
92
|
Prosody combines a Rust core with a Ruby interface:
|
|
@@ -102,8 +110,10 @@ graph TD
|
|
|
102
110
|
|
|
103
111
|
1. **Ruby Interface Layer**: The classes you interact with directly
|
|
104
112
|
- `Prosody::Client`: The main entry point for applications
|
|
105
|
-
- `Prosody::EventHandler`: Base class
|
|
106
|
-
|
|
113
|
+
- `Prosody::EventHandler[Payload]`: Base class whose RBS payload parameter
|
|
114
|
+
is propagated to each handled message
|
|
115
|
+
- `Prosody::Message[Payload]`: Represents a Kafka message with a statically
|
|
116
|
+
typed JSON payload (`Prosody::json_value` by default)
|
|
107
117
|
|
|
108
118
|
2. **Bridge**: The crucial connection between Ruby and Rust
|
|
109
119
|
- Enables safe communication between languages
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,25 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.4.0](https://github.com/prosody-events/prosody-rb/compare/prosody/v0.3.0...prosody/v0.4.0) (2026-07-21)
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
### Features
|
|
7
|
+
|
|
8
|
+
* **state:** add durable keyed state ([#26](https://github.com/prosody-events/prosody-rb/issues/26)) ([fb68411](https://github.com/prosody-events/prosody-rb/commit/fb6841125b1cb7bb3730965254f3205109da8fbe))
|
|
9
|
+
|
|
10
|
+
|
|
11
|
+
### Bug Fixes
|
|
12
|
+
|
|
13
|
+
* **release:** preserve release history boundary ([#28](https://github.com/prosody-events/prosody-rb/issues/28)) ([257cf86](https://github.com/prosody-events/prosody-rb/commit/257cf86e532c942feda5b1ce65af78c8365064d3))
|
|
14
|
+
* **release:** restore published release boundary ([#31](https://github.com/prosody-events/prosody-rb/issues/31)) ([2f4cda1](https://github.com/prosody-events/prosody-rb/commit/2f4cda104f9cf93e28b0d6c7bd8bd66da660b789))
|
|
15
|
+
|
|
16
|
+
## [0.3.0](https://github.com/prosody-events/prosody-rb/compare/prosody/v0.2.1...prosody/v0.3.0) (2026-05-18)
|
|
17
|
+
|
|
18
|
+
|
|
19
|
+
### Features
|
|
20
|
+
|
|
21
|
+
* **deps:** upgrade prosody to 0.4.0 ([#24](https://github.com/prosody-events/prosody-rb/issues/24)) ([4531b68](https://github.com/prosody-events/prosody-rb/commit/4531b68eda202cf00ef506bcb77ba4c53906dbce))
|
|
22
|
+
|
|
3
23
|
## [0.2.1](https://github.com/prosody-events/prosody-rb/compare/prosody/v0.2.0...prosody/v0.2.1) (2026-05-11)
|
|
4
24
|
|
|
5
25
|
|
data/README.md
CHANGED
|
@@ -9,6 +9,7 @@ strategies, and integrated OpenTelemetry support for distributed tracing.
|
|
|
9
9
|
- **Kafka Consumer**: Per-key ordering with cross-key concurrency, offset management, consumer groups
|
|
10
10
|
- **Kafka Producer**: Idempotent delivery with configurable retries
|
|
11
11
|
- **Timer System**: Persistent scheduled execution backed by Cassandra or in-memory store
|
|
12
|
+
- **Keyed State**: Per-key value/map/deque collections that survive across events, transactional by default
|
|
12
13
|
- **Quality of Service**: Fair scheduling limits concurrency and prevents failures from starving fresh traffic. Pipeline mode adds deferred retry and monopolization detection
|
|
13
14
|
- **Distributed Tracing**: OpenTelemetry integration for tracing message flow across services
|
|
14
15
|
- **Backpressure**: Pauses partitions when handlers fall behind
|
|
@@ -29,6 +30,12 @@ Or install directly:
|
|
|
29
30
|
gem install prosody
|
|
30
31
|
```
|
|
31
32
|
|
|
33
|
+
The gem ships RBS signatures for the public API. `Prosody::EventHandler[Payload]`
|
|
34
|
+
carries an application payload type into `Prosody::Message[Payload]`, and keyed-
|
|
35
|
+
state definitions carry their item types through `context.state`. A bare handler,
|
|
36
|
+
message, definition, or state handle defaults to `Prosody::json_value`. See the
|
|
37
|
+
[typed examples](examples/) for Ruby and companion RBS files checked by Steep.
|
|
38
|
+
|
|
32
39
|
## Quick Start
|
|
33
40
|
|
|
34
41
|
```ruby
|
|
@@ -73,7 +80,7 @@ end
|
|
|
73
80
|
client.subscribe(MyHandler.new)
|
|
74
81
|
|
|
75
82
|
# Send a message to a topic
|
|
76
|
-
client.send_message("my-topic", "message-key", {
|
|
83
|
+
client.send_message("my-topic", "message-key", {"content" => "Hello, Kafka!"})
|
|
77
84
|
|
|
78
85
|
# Ensure proper shutdown when done
|
|
79
86
|
client.unsubscribe
|
|
@@ -192,7 +199,7 @@ Configure via constructor options or environment variables. Options fall back to
|
|
|
192
199
|
| `stall_threshold` / `PROSODY_STALL_THRESHOLD` | Report unhealthy if no progress for this long | 5m |
|
|
193
200
|
| `probe_port` / `PROSODY_PROBE_PORT` | HTTP port for health checks (nil to disable) | 8000 |
|
|
194
201
|
| `failure_topic` / `PROSODY_FAILURE_TOPIC` | Send unprocessable messages here (dead letter queue) | - |
|
|
195
|
-
| `idempotence_cache_size` / `PROSODY_IDEMPOTENCE_CACHE_SIZE` | Global shared cache capacity across all partitions for message deduplication
|
|
202
|
+
| `idempotence_cache_size` / `PROSODY_IDEMPOTENCE_CACHE_SIZE` | Global shared cache capacity across all partitions for message deduplication. Consumer deduplication is mandatory and cannot be disabled, so this must be at least 1; setting it to 0 in the client configuration is rejected | 8192 |
|
|
196
203
|
| `idempotence_version` / `PROSODY_IDEMPOTENCE_VERSION` | Version string for cache-busting dedup hashes | 1 |
|
|
197
204
|
| `idempotence_ttl` / `PROSODY_IDEMPOTENCE_TTL` | TTL for dedup records in Cassandra | 7d (604800 seconds) |
|
|
198
205
|
| `slab_size` / `PROSODY_SLAB_SIZE` | Timer storage granularity (rarely needs changing) | 1h |
|
|
@@ -261,6 +268,31 @@ Persistent storage for timers and deferred retries (not needed if `mock: true`):
|
|
|
261
268
|
| `cassandra_rack` / `PROSODY_CASSANDRA_RACK` | Prefer this rack for queries | - |
|
|
262
269
|
| `cassandra_retention` / `PROSODY_CASSANDRA_RETENTION` | Delete data older than this | 1y |
|
|
263
270
|
|
|
271
|
+
### Keyed State
|
|
272
|
+
|
|
273
|
+
Register keyed-state collections before you subscribe. Persistence is backed by Cassandra and is not needed when `mock: true`. See the [Keyed State](#keyed-state-1) feature section for handler usage; the client-level knobs and per-collection fields are below. Where an option and an environment variable are paired, an explicitly set option wins; otherwise the environment variable applies, then the default.
|
|
274
|
+
|
|
275
|
+
| Option / Environment Variable | Description | Default |
|
|
276
|
+
|-------------------------------|-------------|---------|
|
|
277
|
+
| `state_collections` / - | Keyed-state collections to register before subscribe (array of definitions or config hashes; duplicate names rejected) | (none) |
|
|
278
|
+
| `state_cache_dir` / `PROSODY_STATE_CACHE_DIR` | Disk workspace for the local keyed-state cache; each live client needs its own directory (it is locked exclusively) | per-client temp dir |
|
|
279
|
+
| `state_cache_size_bytes` / `PROSODY_STATE_CACHE_SIZE_BYTES` | Capacity of the in-memory keyed-state cache, in bytes; must be greater than 0. One cache is shared by all partition keyspaces | engine default |
|
|
280
|
+
| `state_recovery_delay` / `PROSODY_STATE_RECOVERY_DELAY` | Whole-second delay between staging a provisional cell and the recovery sweep; every collection TTL must strictly exceed it | 30s |
|
|
281
|
+
|
|
282
|
+
Prefer the definition constructors (`Prosody.value` / `.map` / `.deque` and their `message_*` variants, documented below): they serialize into `state_collections` so you declare each collection once and reuse the same object with `context.state`. Each entry has these fields:
|
|
283
|
+
|
|
284
|
+
| Field | Description | Default |
|
|
285
|
+
|-------|-------------|---------|
|
|
286
|
+
| `name` | Collection name; non-empty and unique within the client | (required) |
|
|
287
|
+
| `kind` | `"value"`, `"map"`, or `"deque"` | (required) |
|
|
288
|
+
| `payload` | `"json"` (JSON values) or `"message"` (the full Kafka message the handler received) | (required) |
|
|
289
|
+
| `ttl_seconds` | Per-write TTL in whole seconds (at least 1; must exceed the recovery delay) | (none) |
|
|
290
|
+
| `read_uncommitted` | Opt out of transactional staging | false |
|
|
291
|
+
| `keyset_limit` | Map-only; ordered-scan bound in `0..=4096` (`0` disables ordered-scan tracking) | 128 |
|
|
292
|
+
| `capacity` | Deque-only window bound (at least 1); keeps at most N slots, enforced lazily on push. Runtime-only and mutable across deploys — not persisted | unbounded |
|
|
293
|
+
|
|
294
|
+
Constructors set these via keyword arguments (`ttl:`, `keyset_limit:`, `capacity:`, `read_uncommitted:`).
|
|
295
|
+
|
|
264
296
|
### Telemetry Emitter
|
|
265
297
|
|
|
266
298
|
Prosody can emit internal processing events (message lifecycle, timer events) to a Kafka topic for observability:
|
|
@@ -476,36 +508,37 @@ The deduplication system uses:
|
|
|
476
508
|
```ruby
|
|
477
509
|
# Messages with IDs are deduplicated per key
|
|
478
510
|
client.send_message("my-topic", "key1", {
|
|
479
|
-
id
|
|
480
|
-
content
|
|
511
|
+
"id" => "msg-123", # Message will be processed
|
|
512
|
+
"content" => "Hello!"
|
|
481
513
|
})
|
|
482
514
|
|
|
483
515
|
client.send_message("my-topic", "key1", {
|
|
484
|
-
id
|
|
485
|
-
content
|
|
516
|
+
"id" => "msg-123", # Message will be skipped (duplicate)
|
|
517
|
+
"content" => "Hello again!"
|
|
486
518
|
})
|
|
487
519
|
|
|
488
520
|
client.send_message("my-topic", "key2", {
|
|
489
|
-
id
|
|
490
|
-
content
|
|
521
|
+
"id" => "msg-123", # Message will be processed (different key)
|
|
522
|
+
"content" => "Hello!"
|
|
491
523
|
})
|
|
492
524
|
```
|
|
493
525
|
|
|
494
|
-
|
|
526
|
+
Consumer deduplication is **mandatory** — it is the commit oracle that makes
|
|
527
|
+
keyed state correct — so it cannot be disabled. `idempotence_cache_size` must be
|
|
528
|
+
at least 1; setting it to `0` in the client configuration raises an
|
|
529
|
+
`ArgumentError`:
|
|
495
530
|
|
|
496
531
|
```ruby
|
|
497
532
|
client = Prosody::Client.new(
|
|
498
533
|
group_id: "my-consumer-group",
|
|
499
534
|
subscribed_topics: "my-topic",
|
|
500
|
-
idempotence_cache_size: 0 #
|
|
535
|
+
idempotence_cache_size: 0 # Rejected: consumer deduplication cannot be disabled
|
|
501
536
|
)
|
|
502
537
|
```
|
|
503
538
|
|
|
504
|
-
|
|
505
|
-
|
|
506
|
-
|
|
507
|
-
PROSODY_IDEMPOTENCE_CACHE_SIZE=0
|
|
508
|
-
```
|
|
539
|
+
This applies to every client — `Prosody::Client.new` always builds a consumer, so
|
|
540
|
+
`0` is rejected regardless of whether any topics are subscribed, and whether it is
|
|
541
|
+
supplied in the client configuration or via `PROSODY_IDEMPOTENCE_CACHE_SIZE`.
|
|
509
542
|
|
|
510
543
|
To invalidate all previously recorded dedup entries (e.g. after a data migration), change the version string:
|
|
511
544
|
|
|
@@ -529,6 +562,107 @@ client = Prosody::Client.new(
|
|
|
529
562
|
|
|
530
563
|
Note that the in-memory cache is best-effort. Duplicates can still occur across different process instances.
|
|
531
564
|
|
|
565
|
+
## Keyed State
|
|
566
|
+
|
|
567
|
+
Keyed state gives every Kafka key its own durable working memory. Prosody automatically uses the current message or timer key, so a handler can relate the current event to earlier events for that key. State survives restarts and rebalances. By default, changes become visible only when the event succeeds.
|
|
568
|
+
|
|
569
|
+
Use keyed state for time-aware stream processing: counters, deduplication, rolling aggregates, pending work, and per-key workflows. Keep your relational database as the source of truth for business data and for work that needs joins or ad hoc queries. Reconstructing stream state with repeated database queries can be slow and expensive; keyed state is built for that job.
|
|
570
|
+
|
|
571
|
+
Most collections should have a TTL. Set it comfortably beyond the longest timer or workflow that uses the state; Prosody validates the minimum supported TTL. Omit it only when keeping inactive keys forever is intentional.
|
|
572
|
+
|
|
573
|
+
### A counter for each key
|
|
574
|
+
|
|
575
|
+
Declare each collection once, register it on the client, and ask the event context for the current key's state:
|
|
576
|
+
|
|
577
|
+
```ruby
|
|
578
|
+
COUNTER = Prosody.value("counter", ttl: 30 * 24 * 60 * 60)
|
|
579
|
+
|
|
580
|
+
class CountHandler < Prosody::EventHandler
|
|
581
|
+
def on_message(context, _message)
|
|
582
|
+
count = context.state(COUNTER)
|
|
583
|
+
count.set((count.get || 0) + 1)
|
|
584
|
+
end
|
|
585
|
+
end
|
|
586
|
+
|
|
587
|
+
client = Prosody::Client.new(
|
|
588
|
+
group_id: "counters",
|
|
589
|
+
subscribed_topics: "events",
|
|
590
|
+
state_collections: [COUNTER]
|
|
591
|
+
)
|
|
592
|
+
```
|
|
593
|
+
|
|
594
|
+
Here, counters expire after 30 days without an update.
|
|
595
|
+
|
|
596
|
+
### Window activity into one notification
|
|
597
|
+
|
|
598
|
+
This example turns a burst of activity into two useful notifications. It sends the first event immediately, collects later events for five minutes, then sends one summary. Because the user ID is the Kafka key, every user gets an independent window.
|
|
599
|
+
|
|
600
|
+
```ruby
|
|
601
|
+
WINDOW = Prosody.value("window", ttl: 24 * 60 * 60)
|
|
602
|
+
PENDING = Prosody.message_deque("pending", capacity: 100, ttl: 24 * 60 * 60)
|
|
603
|
+
|
|
604
|
+
class ActivityHandler < Prosody::EventHandler
|
|
605
|
+
def on_message(context, message)
|
|
606
|
+
window = context.state(WINDOW)
|
|
607
|
+
pending = context.state(PENDING)
|
|
608
|
+
|
|
609
|
+
if window.get
|
|
610
|
+
pending.push(message)
|
|
611
|
+
return
|
|
612
|
+
end
|
|
613
|
+
|
|
614
|
+
notify(message.key, [message])
|
|
615
|
+
window.set(true)
|
|
616
|
+
context.clear_and_schedule(Time.now + 5 * 60)
|
|
617
|
+
end
|
|
618
|
+
|
|
619
|
+
def on_timer(context, timer)
|
|
620
|
+
pending = context.state(PENDING)
|
|
621
|
+
batch = []
|
|
622
|
+
pending.each { |message| batch << message }
|
|
623
|
+
|
|
624
|
+
notify(timer.key, batch) unless batch.empty?
|
|
625
|
+
pending.clear
|
|
626
|
+
context.state(WINDOW).clear
|
|
627
|
+
end
|
|
628
|
+
end
|
|
629
|
+
```
|
|
630
|
+
|
|
631
|
+
See the complete, Steep-checked example for signatures, client setup, and `notify`: [`examples/keyed_state_windowing.rb`](examples/keyed_state_windowing.rb) and [`examples/keyed_state_windowing.rbs`](examples/keyed_state_windowing.rbs).
|
|
632
|
+
|
|
633
|
+
Why this works:
|
|
634
|
+
|
|
635
|
+
- Register both definitions in `state_collections` before subscribing. Keyed state uses Cassandra unless `mock: true`.
|
|
636
|
+
- Use `clear_and_schedule`, not `schedule`, so a retried event does not add another timer for the same key.
|
|
637
|
+
- `capacity: 100` and the one-day TTL prevent an inactive or unusually busy key from retaining an unlimited backlog. Since this example only appends, overflow drops the oldest saved message.
|
|
638
|
+
- A `message_deque` requires the original Kafka messages to remain available for the whole window. Use a plain `deque` of payloads if topic retention or compaction cannot guarantee that.
|
|
639
|
+
- Prosody runs one handler at a time for each key, so a user's message and timer handlers cannot overlap.
|
|
640
|
+
- Sending a notification is outside Prosody's state transaction and may happen again after a retry. Give notifications a stable idempotency key, or send them through an outbox, when duplicates matter.
|
|
641
|
+
|
|
642
|
+
### Collections and handles
|
|
643
|
+
|
|
644
|
+
A definition gives a collection a stable name, kind, and options. Register it once on the client, then pass the same definition to `context.state` to access the current key. Do not reuse a persisted name for a different collection kind or payload type.
|
|
645
|
+
|
|
646
|
+
Create handles inside the handler and do not retain them or their iterators afterward. State operations look synchronous but yield the current fiber while Prosody performs the work.
|
|
647
|
+
|
|
648
|
+
| Collection | JSON payload | Kafka message | Main operations |
|
|
649
|
+
| --- | --- | --- | --- |
|
|
650
|
+
| Value | `Prosody.value` | `Prosody.message_value` | `get`, `set`, `clear` |
|
|
651
|
+
| Ordered string map | `Prosody.map` | `Prosody.message_map` | `get`, `get_many`, `key?`, `set`, `delete`, `each_pair`, `each_key`, `clear` |
|
|
652
|
+
| Deque | `Prosody.deque` | `Prosody.message_deque` | `push`, `unshift`, `pop`, `shift`, `get`, `length`, `each`, `clear` |
|
|
653
|
+
|
|
654
|
+
Map and deque scans return enumerators when called without a block. Map keys are strings. `nil` means absence and cannot be stored—use `clear` or `delete` instead.
|
|
655
|
+
|
|
656
|
+
### When changes become visible
|
|
657
|
+
|
|
658
|
+
Reads inside a handler see its earlier writes. The default behavior is the safest choice for most handlers: Prosody buffers those changes and publishes them together when the event succeeds. If the handler raises, none of its pending changes become visible.
|
|
659
|
+
|
|
660
|
+
Each collection also offers explicit controls for workflows that need different behavior:
|
|
661
|
+
|
|
662
|
+
- `read_uncommitted: true` writes that collection's changes after the handler succeeds but before the event is recorded as complete. A crash in between can leave the changes visible even though the event is retried. Use it only for idempotent changes, where processing the same event again produces the same stored result.
|
|
663
|
+
- `commit` immediately publishes this collection's pending changes. They remain visible even if the handler later raises and the event is retried.
|
|
664
|
+
- `rollback` discards this collection's pending changes since its last `commit`. It cannot undo changes that were already committed.
|
|
665
|
+
|
|
532
666
|
## Timer Functionality
|
|
533
667
|
|
|
534
668
|
Prosody supports timer-based delayed execution within message handlers. When a timer fires, your handler's `on_timer` method will be called:
|
|
@@ -898,17 +1032,18 @@ Ensure you have thoroughly tested your changes before merging to `main`.
|
|
|
898
1032
|
### Prosody::Client
|
|
899
1033
|
|
|
900
1034
|
- `new(**config)`: Initialize a new Prosody client with the given configuration.
|
|
901
|
-
- `send_message(topic, key, payload)`: Send a message
|
|
1035
|
+
- `send_message(String topic, String key, Prosody::json_value payload)`: Send a JSON-serializable message.
|
|
902
1036
|
- `consumer_state`: Get the current state of the consumer (`:unconfigured`, `:configured`, or `:running`).
|
|
903
1037
|
- `source_system`: Get the source system identifier configured for the client.
|
|
904
|
-
- `subscribe(
|
|
1038
|
+
- `subscribe: [Payload] (Prosody::EventHandler[Payload]) -> void`: Subscribe while preserving the handler's payload specialization.
|
|
905
1039
|
- `unsubscribe`: Unsubscribe from messages and shut down the consumer.
|
|
906
1040
|
- `assigned_partitions`: Get the number of partitions currently assigned to this consumer.
|
|
907
1041
|
- `is_stalled?`: Check if the consumer has stalled partitions.
|
|
908
1042
|
|
|
909
1043
|
### Prosody::EventHandler
|
|
910
1044
|
|
|
911
|
-
A base class for user-defined handlers
|
|
1045
|
+
A base class for user-defined handlers. Its RBS payload parameter flows into
|
|
1046
|
+
`Message#payload`; a bare handler defaults to `Prosody::json_value`.
|
|
912
1047
|
|
|
913
1048
|
```ruby
|
|
914
1049
|
class MyHandler < Prosody::EventHandler
|
|
@@ -928,14 +1063,36 @@ end
|
|
|
928
1063
|
|
|
929
1064
|
### Prosody::Message
|
|
930
1065
|
|
|
931
|
-
|
|
1066
|
+
`Prosody::Message[Payload]` represents a Kafka message. `Payload` defaults to
|
|
1067
|
+
`Prosody::json_value` (`nil`, booleans, numbers, strings, arrays, and
|
|
1068
|
+
string-keyed hashes, recursively). The parameter is static documentation and
|
|
1069
|
+
does not perform runtime validation.
|
|
1070
|
+
|
|
1071
|
+
For a type-safe handler, describe the JSON record and specialize the handler in
|
|
1072
|
+
your application's RBS:
|
|
1073
|
+
|
|
1074
|
+
```rbs
|
|
1075
|
+
type order_event = { "order_id" => String, "total" => Integer }
|
|
1076
|
+
|
|
1077
|
+
class OrderHandler < Prosody::EventHandler[order_event]
|
|
1078
|
+
def on_message: (Prosody::Context, Prosody::Message[order_event]) -> void
|
|
1079
|
+
end
|
|
1080
|
+
```
|
|
1081
|
+
|
|
1082
|
+
Ruby can then use `message.payload["order_id"]` as a `String` and
|
|
1083
|
+
`message.payload["total"]` as an `Integer`. See
|
|
1084
|
+
[`examples/keyed_state.rb`](examples/keyed_state.rb) and its companion
|
|
1085
|
+
[`examples/keyed_state.rbs`](examples/keyed_state.rbs) for payload typing that
|
|
1086
|
+
also flows through message-backed state.
|
|
1087
|
+
|
|
1088
|
+
Messages have the following attributes:
|
|
932
1089
|
|
|
933
1090
|
- `topic` (String): The name of the topic.
|
|
934
1091
|
- `partition` (Integer): The partition number.
|
|
935
1092
|
- `offset` (Integer): The message offset within the partition.
|
|
936
1093
|
- `timestamp` (Time): The timestamp when the message was created or sent.
|
|
937
1094
|
- `key` (String): The message key.
|
|
938
|
-
- `payload` (
|
|
1095
|
+
- `payload` (`Payload`): The JSON-deserialized message payload.
|
|
939
1096
|
|
|
940
1097
|
### Prosody::Context
|
|
941
1098
|
|
|
@@ -943,6 +1100,7 @@ Represents the context of message processing:
|
|
|
943
1100
|
|
|
944
1101
|
- `should_cancel?`: Check if cancellation has been requested (includes timeout and shutdown).
|
|
945
1102
|
- `on_cancel`: Blocks until cancellation is signaled.
|
|
1103
|
+
- `state(definition)`: Binds a registered collection for the current event attempt, returning a typed handle (`ValueState`, `MapState`, or `DequeState`). Raises `PermanentStateError` when the name was never registered, or when the definition's `kind`/`payload` disagrees with the collection's durably-registered schema. See the [Keyed State](#keyed-state-2) API reference below.
|
|
946
1104
|
|
|
947
1105
|
Timer scheduling methods:
|
|
948
1106
|
|
|
@@ -958,3 +1116,36 @@ Represents a timer that has fired, provided to the `on_timer` method:
|
|
|
958
1116
|
|
|
959
1117
|
- `key` (String): The entity key identifying what this timer belongs to
|
|
960
1118
|
- `time` (Time): The time when this timer was scheduled to fire
|
|
1119
|
+
|
|
1120
|
+
### Keyed State
|
|
1121
|
+
|
|
1122
|
+
Definition constructors (each returns a frozen definition object used both in `Configuration#state_collections` and with `context.state`):
|
|
1123
|
+
|
|
1124
|
+
- `Prosody.value(name, ttl: nil, read_uncommitted: nil)`
|
|
1125
|
+
- `Prosody.map(name, ttl: nil, keyset_limit: nil, read_uncommitted: nil)`
|
|
1126
|
+
- `Prosody.deque(name, ttl: nil, read_uncommitted: nil)`
|
|
1127
|
+
- `Prosody.message_value(name, ttl: nil, read_uncommitted: nil)`
|
|
1128
|
+
- `Prosody.message_map(name, ttl: nil, keyset_limit: nil, read_uncommitted: nil)`
|
|
1129
|
+
- `Prosody.message_deque(name, ttl: nil, read_uncommitted: nil)`
|
|
1130
|
+
|
|
1131
|
+
`Prosody::ValueState`:
|
|
1132
|
+
|
|
1133
|
+
- `get`, `set(value)`, `clear`, `commit`, `rollback`
|
|
1134
|
+
|
|
1135
|
+
`Prosody::MapState` (keys are `String`):
|
|
1136
|
+
|
|
1137
|
+
- `get(key)`, `get_many(keys)`, `set(key, value)`, `delete(key)` (returns `nil`), `clear`
|
|
1138
|
+
- `each_pair` / `reverse_each_pair` (block or `Enumerator`), `commit`, `rollback`
|
|
1139
|
+
|
|
1140
|
+
`Prosody::DequeState`:
|
|
1141
|
+
|
|
1142
|
+
- `push(value)`, `unshift(value)`, `pop`, `shift`, `length` (aliased `size`), `empty?`, `get(index)`, `clear`
|
|
1143
|
+
- `each` / `reverse_each` (block or `Enumerator`), `commit`, `rollback`
|
|
1144
|
+
|
|
1145
|
+
Errors:
|
|
1146
|
+
|
|
1147
|
+
- `Prosody::TransientStateError < Prosody::TransientError`: the default — a temporary store read/write failure, or any caller mistake (a `nil`/unrepresentable write, item-shape mismatch, out-of-range index, invalid scan direction), rejected transient so it retries rather than discarding the message.
|
|
1148
|
+
- `Prosody::PermanentStateError < Prosody::PermanentError`: reserved for failures a retry cannot resolve in-process (unregistered/identity-mismatched collection, duplicate registration, bad TTL), or one a handler raises explicitly.
|
|
1149
|
+
- `Prosody::NullValueError < Prosody::TransientStateError`: raised when a `nil` is written; use `clear`/`delete` instead.
|
|
1150
|
+
|
|
1151
|
+
State errors are never Terminal (core folds Terminal into Transient).
|
data/Rakefile
CHANGED
|
@@ -23,4 +23,14 @@ RSpec::Core::RakeTask.new(:spec)
|
|
|
23
23
|
|
|
24
24
|
require "standard/rake"
|
|
25
25
|
|
|
26
|
-
|
|
26
|
+
desc "Validate RBS signatures over the whole sig/ tree"
|
|
27
|
+
task :rbs do
|
|
28
|
+
sh "rbs -r logger -I sig validate"
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
desc "Type-check the Ruby implementation against its RBS signatures"
|
|
32
|
+
task :steep do
|
|
33
|
+
sh "steep check --with-expectations --severity-level=error"
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
task default: %i[compile spec standard rbs steep]
|
|
@@ -0,0 +1,58 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "prosody"
|
|
4
|
+
require "logger"
|
|
5
|
+
|
|
6
|
+
# Keyed-state example: per-key value/map/deque collections that survive across
|
|
7
|
+
# events. Definitions are declared once and reused for both registration (on the
|
|
8
|
+
# client) and binding (inside the handler). Every state op yields the fiber,
|
|
9
|
+
# never the thread.
|
|
10
|
+
|
|
11
|
+
# Definitions: declared once, reused for registration and binding. See
|
|
12
|
+
# keyed_state.rbs for payload and state types checked by Steep.
|
|
13
|
+
CART = Prosody.value("cart", ttl: 30 * 24 * 3600) # ValueState
|
|
14
|
+
TOTALS = Prosody.map("totals") # keys are always String
|
|
15
|
+
BACKLOG = Prosody.message_deque("backlog", capacity: 100) # bounded window of messages
|
|
16
|
+
|
|
17
|
+
class KeyedStateHandler < Prosody::EventHandler
|
|
18
|
+
def initialize(logger:)
|
|
19
|
+
@logger = logger
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def on_message(context, message)
|
|
23
|
+
cart = context.state(CART) # bound for this attempt only
|
|
24
|
+
current = cart.get || {"items" => []} # Hash, or nil when absent
|
|
25
|
+
cart.set(current.merge("items" => current["items"] + [message.payload["order_id"]]))
|
|
26
|
+
|
|
27
|
+
totals = context.state(TOTALS)
|
|
28
|
+
totals.set(message.key, message.payload["total"])
|
|
29
|
+
# Steep infers key as String and total as Integer from TOTALS's RBS type.
|
|
30
|
+
totals.each_pair { |key, total| @logger.info(format_total(key, total)) }
|
|
31
|
+
|
|
32
|
+
backlog = context.state(BACKLOG)
|
|
33
|
+
backlog.push(message) # stores the full Prosody::Message
|
|
34
|
+
oldest = backlog.get(0) # Message[order_event]?, preserving payload shape
|
|
35
|
+
@logger.info("oldest order: #{format_order_id(oldest.payload["order_id"])}") if oldest
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
private
|
|
39
|
+
|
|
40
|
+
def format_total(key, total)
|
|
41
|
+
"#{key}=#{total}"
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
def format_order_id(order_id)
|
|
45
|
+
order_id
|
|
46
|
+
end
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
if __FILE__ == $PROGRAM_NAME
|
|
50
|
+
client = Prosody::Client.new(
|
|
51
|
+
mock: true,
|
|
52
|
+
group_id: "keyed-state-example",
|
|
53
|
+
subscribed_topics: "orders",
|
|
54
|
+
state_collections: [CART, TOTALS, BACKLOG]
|
|
55
|
+
)
|
|
56
|
+
client.subscribe(KeyedStateHandler.new(logger: Logger.new($stdout)))
|
|
57
|
+
client.unsubscribe
|
|
58
|
+
end
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
type keyed_state_cart = { "items" => Array[String] }
|
|
2
|
+
type keyed_state_order_event = { "order_id" => String, "total" => Integer }
|
|
3
|
+
|
|
4
|
+
CART: Prosody::_ValueDefinition[keyed_state_cart]
|
|
5
|
+
TOTALS: Prosody::_MapDefinition[Integer]
|
|
6
|
+
BACKLOG: Prosody::_MessageDequeDefinition[keyed_state_order_event]
|
|
7
|
+
|
|
8
|
+
class KeyedStateHandler < Prosody::EventHandler[keyed_state_order_event]
|
|
9
|
+
@logger: Logger
|
|
10
|
+
|
|
11
|
+
def initialize: (logger: Logger) -> void
|
|
12
|
+
def on_message: (Prosody::Context context, Prosody::Message[keyed_state_order_event] message) -> void
|
|
13
|
+
|
|
14
|
+
private
|
|
15
|
+
|
|
16
|
+
def format_total: (String key, Integer total) -> String
|
|
17
|
+
def format_order_id: (String order_id) -> String
|
|
18
|
+
end
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "prosody"
|
|
4
|
+
|
|
5
|
+
# The complete typed counterpart of the README burst-windowing example.
|
|
6
|
+
ACTIVITY_WINDOW = Prosody.value("activity-window")
|
|
7
|
+
PENDING_ACTIVITIES = Prosody.message_deque("pending-activities", capacity: 100)
|
|
8
|
+
|
|
9
|
+
class ActivityWindowHandler < Prosody::EventHandler
|
|
10
|
+
def on_message(context, message)
|
|
11
|
+
window = context.state(ACTIVITY_WINDOW)
|
|
12
|
+
pending = context.state(PENDING_ACTIVITIES)
|
|
13
|
+
|
|
14
|
+
if window.get
|
|
15
|
+
pending.push(message)
|
|
16
|
+
else
|
|
17
|
+
notify(message.key, [message])
|
|
18
|
+
window.set(true)
|
|
19
|
+
context.clear_and_schedule(Time.now + 5 * 60)
|
|
20
|
+
end
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
def on_timer(context, timer)
|
|
24
|
+
pending = context.state(PENDING_ACTIVITIES)
|
|
25
|
+
batch = pending.each.to_a
|
|
26
|
+
notify(timer.key, batch) unless batch.empty?
|
|
27
|
+
pending.clear
|
|
28
|
+
context.state(ACTIVITY_WINDOW).clear
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
private
|
|
32
|
+
|
|
33
|
+
def notify(user_id, activities)
|
|
34
|
+
puts "notify #{user_id}: #{activities.length} activities"
|
|
35
|
+
end
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
if __FILE__ == $PROGRAM_NAME
|
|
39
|
+
client = Prosody::Client.new(
|
|
40
|
+
mock: true,
|
|
41
|
+
group_id: "activity-window-example",
|
|
42
|
+
subscribed_topics: "activities",
|
|
43
|
+
state_collections: [ACTIVITY_WINDOW, PENDING_ACTIVITIES]
|
|
44
|
+
)
|
|
45
|
+
client.subscribe(ActivityWindowHandler.new)
|
|
46
|
+
client.unsubscribe
|
|
47
|
+
end
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
type activity_event = {
|
|
2
|
+
"actor" => String,
|
|
3
|
+
"action" => String
|
|
4
|
+
}
|
|
5
|
+
|
|
6
|
+
ACTIVITY_WINDOW: Prosody::_ValueDefinition[bool]
|
|
7
|
+
PENDING_ACTIVITIES: Prosody::_MessageDequeDefinition[activity_event]
|
|
8
|
+
|
|
9
|
+
class ActivityWindowHandler < Prosody::EventHandler[activity_event]
|
|
10
|
+
def on_message: (Prosody::Context context, Prosody::Message[activity_event] message) -> void
|
|
11
|
+
def on_timer: (Prosody::Context context, Prosody::Timer timer) -> void
|
|
12
|
+
|
|
13
|
+
private
|
|
14
|
+
|
|
15
|
+
def notify: (String user_id, Array[Prosody::Message[activity_event]] activities) -> void
|
|
16
|
+
end
|
data/lib/prosody/3.2/prosody.so
CHANGED
|
Binary file
|
data/lib/prosody/3.3/prosody.so
CHANGED
|
Binary file
|
data/lib/prosody/3.4/prosody.so
CHANGED
|
Binary file
|
|
@@ -281,6 +281,32 @@ module Prosody
|
|
|
281
281
|
# Overrides the PROSODY_TIMER_SPANS environment variable. Default: "follows_from".
|
|
282
282
|
config_param :timer_spans, converter: ->(v) { v.to_s }
|
|
283
283
|
|
|
284
|
+
# Keyed-state collections to register before subscribe.
|
|
285
|
+
#
|
|
286
|
+
# Accepts an array of StateDefinition objects (from Prosody.value/map/deque
|
|
287
|
+
# and their message_* siblings) or already-serialized registration hashes.
|
|
288
|
+
# Duplicate names within the set are rejected by the native layer.
|
|
289
|
+
config_param :state_collections,
|
|
290
|
+
converter: lambda { |v|
|
|
291
|
+
list = v.is_a?(Hash) ? [v] : Array(v)
|
|
292
|
+
list.map { |d| d.respond_to?(:to_state_config) ? d.to_state_config : d }
|
|
293
|
+
}
|
|
294
|
+
|
|
295
|
+
# Disk workspace for the local keyed-state cache. Each live client
|
|
296
|
+
# needs its own directory. Falls back to the
|
|
297
|
+
# PROSODY_STATE_CACHE_DIR environment variable. Must not be an empty string.
|
|
298
|
+
config_param :state_cache_dir, converter: lambda(&:to_s)
|
|
299
|
+
|
|
300
|
+
# Capacity of the in-memory keyed-state cache, in bytes. Falls back to
|
|
301
|
+
# PROSODY_STATE_CACHE_SIZE_BYTES, then
|
|
302
|
+
# the storage-engine default.
|
|
303
|
+
config_param :state_cache_size_bytes, converter: ->(v) { Integer(v) }
|
|
304
|
+
|
|
305
|
+
# Delay in whole seconds between staging a provisional cell and the
|
|
306
|
+
# keyed-state recovery sweep. Every registered TTL must strictly exceed this.
|
|
307
|
+
# Must be a whole number of seconds >= 1 (validated natively).
|
|
308
|
+
config_param :state_recovery_delay, converter: ->(v) { duration_converter(v) }
|
|
309
|
+
|
|
284
310
|
# Operation mode of the client.
|
|
285
311
|
#
|
|
286
312
|
# Valid values:
|
data/lib/prosody/handler.rb
CHANGED
|
@@ -114,7 +114,7 @@ module Prosody
|
|
|
114
114
|
super(*args, &block)
|
|
115
115
|
rescue *exception_classes => e
|
|
116
116
|
# The new exception's #cause will be set automatically
|
|
117
|
-
raise error_class.new(e.message)
|
|
117
|
+
Kernel.raise error_class.new(e.message)
|
|
118
118
|
end
|
|
119
119
|
end
|
|
120
120
|
|
|
@@ -127,6 +127,10 @@ module Prosody
|
|
|
127
127
|
# --------------------------------------------------------------------------
|
|
128
128
|
|
|
129
129
|
# Abstract base class for handling incoming messages and timers from Prosody.
|
|
130
|
+
# The RBS type parameter describes the JSON payload delivered in each
|
|
131
|
+
# {Message}; it defaults to +Prosody::json_value+. Declare a narrower payload
|
|
132
|
+
# shape in your application's RBS (for example,
|
|
133
|
+
# +EventHandler[order_event]+) to type-check payload access.
|
|
130
134
|
# Subclasses **must** implement `#on_message` to process received messages.
|
|
131
135
|
# Subclasses **may** implement `#on_timer` to process timer events.
|
|
132
136
|
# They may also use `permanent` or `transient` decorators to control retry logic.
|
|
@@ -156,7 +160,7 @@ module Prosody
|
|
|
156
160
|
# custom message handling logic.
|
|
157
161
|
#
|
|
158
162
|
# @param [Context] context the message context
|
|
159
|
-
# @param [Message] message the message payload
|
|
163
|
+
# @param [Message<Payload>] message the message and its typed JSON payload
|
|
160
164
|
# @raise [NotImplementedError] if not overridden by subclass
|
|
161
165
|
# @return [void]
|
|
162
166
|
def on_message(context, message)
|