omq 0.6.5 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +40 -0
- data/README.md +76 -81
- data/lib/omq/cli/base_runner.rb +54 -21
- data/lib/omq/cli/client_server.rb +14 -9
- data/lib/omq/cli/config.rb +7 -4
- data/lib/omq/cli/pair.rb +1 -1
- data/lib/omq/cli/pipe.rb +37 -21
- data/lib/omq/cli/req_rep.rb +8 -4
- data/lib/omq/cli/router_dealer.rb +12 -6
- data/lib/omq/cli.rb +100 -34
- data/lib/omq/version.rb +1 -1
- data/lib/omq/zmtp/engine.rb +34 -13
- data/lib/omq/zmtp/routing/peer.rb +3 -2
- data/lib/omq/zmtp/routing/rep.rb +2 -3
- data/lib/omq/zmtp/routing/req.rb +3 -7
- data/lib/omq/zmtp/routing/router.rb +3 -2
- data/lib/omq/zmtp/routing/server.rb +3 -2
- data/lib/omq.rb +1 -0
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: d2e17e1a12c0f44bec5af914b8df2f810029e75c7da17e79e862428a7b0aa99d
|
|
4
|
+
data.tar.gz: 4a8d06116bad67adc4411cb3281302fd6190ecce46ceb47f21cc15a32c3e9006
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 1dad86f1ad0648690bc5a01d3b7258b89f301d4768bac0831495d27f0a0e3c84f5a30af049732f8cc6885bcd6105654d41708d519da69c30a62739ca58a27418
|
|
7
|
+
data.tar.gz: a71f54aa26dd3e7dea3b249d027c03aeff896c7ded7b0581067fbda609d114c976c8d35a5f7f7e630cbbc34fbb839c7d43f6d445e8f2bb50f1348cb46f8d0e51
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,45 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.7.0 — 2026-03-30
|
|
4
|
+
|
|
5
|
+
### Breaking
|
|
6
|
+
|
|
7
|
+
- **`-e` is now `--recv-eval`** — evaluates incoming messages only.
|
|
8
|
+
Send-only sockets (PUSH, PUB, SCATTER, RADIO) must use `-E` /
|
|
9
|
+
`--send-eval` instead of `-e`.
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- **`-E` / `--send-eval`** — eval Ruby for each outgoing message.
|
|
14
|
+
REQ can now transform requests independently from replies.
|
|
15
|
+
ROUTER/SERVER/PEER: `-E` does dynamic routing (first element =
|
|
16
|
+
identity), mutually exclusive with `--target`.
|
|
17
|
+
- **`OMQ.outgoing` / `OMQ.incoming`** — registration API for script
|
|
18
|
+
handlers loaded via `-r`. Blocks receive message parts as a block
|
|
19
|
+
argument (`|msg|`). Setup via closures, teardown via `at_exit`.
|
|
20
|
+
CLI flags override registered handlers.
|
|
21
|
+
- **[CLI.md](CLI.md)** — comprehensive CLI documentation.
|
|
22
|
+
- **[GETTING_STARTED.md](GETTING_STARTED.md)** — renamed from
|
|
23
|
+
`ZGUIDE_SUMMARY.md` for discoverability.
|
|
24
|
+
- **Multi-peer pipe with `--in`/`--out`** — modal switches that assign
|
|
25
|
+
subsequent `-b`/`-c` to the PULL (input) or PUSH (output) side.
|
|
26
|
+
Enables fan-in, fan-out, and mixed bind/connect per side.
|
|
27
|
+
Backward compatible — without `--in`/`--out`, the positional
|
|
28
|
+
2-endpoint syntax works as before.
|
|
29
|
+
|
|
30
|
+
### Improved
|
|
31
|
+
|
|
32
|
+
- **YJIT recv pump** — replaced lambda/proc `transform:` parameter in
|
|
33
|
+
`Engine#start_recv_pump` with block captures. No-transform path
|
|
34
|
+
(PUSH/PULL, PUB/SUB) is now branch-free. ~2.5x YJIT speedup on
|
|
35
|
+
inproc, ~2x on ipc/tcp.
|
|
36
|
+
|
|
37
|
+
### Fixed
|
|
38
|
+
|
|
39
|
+
- **Frozen array from `recv_msg_raw`** — ROUTER/SERVER receiver crashed
|
|
40
|
+
with `FrozenError` when shifting identity off frozen message arrays.
|
|
41
|
+
`#recv_msg_raw` now dups the array.
|
|
42
|
+
|
|
3
43
|
## 0.6.5 — 2026-03-30
|
|
4
44
|
|
|
5
45
|
### Fixed
|
data/README.md
CHANGED
|
@@ -1,39 +1,44 @@
|
|
|
1
|
-
# OMQ
|
|
1
|
+
# OMQ — ZeroMQ in pure Ruby
|
|
2
2
|
|
|
3
3
|
[](https://github.com/zeromq/omq/actions/workflows/ci.yml)
|
|
4
4
|
[](https://rubygems.org/gems/omq)
|
|
5
5
|
[](LICENSE)
|
|
6
6
|
[](https://www.ruby-lang.org)
|
|
7
7
|
|
|
8
|
-
|
|
8
|
+
`gem install omq` — that's it. No libzmq, no compiler, no system packages. Just Ruby.
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
OMQ implements the [ZMTP 3.1](https://rfc.zeromq.org/spec/23/) wire protocol from scratch using [Async](https://github.com/socketry/async) fibers. It speaks native ZeroMQ on the wire and interoperates with libzmq, pyzmq, CZMQ, and everything else in the ZMQ ecosystem.
|
|
11
|
+
|
|
12
|
+
> **234k msg/s** inproc | **49k msg/s** ipc | **36k msg/s** tcp
|
|
11
13
|
>
|
|
12
|
-
> **
|
|
14
|
+
> **12 µs** inproc latency | **51 µs** ipc | **62 µs** tcp
|
|
13
15
|
>
|
|
14
|
-
> Ruby 4.0 + YJIT on a Linux VM
|
|
16
|
+
> Ruby 4.0 + YJIT on a Linux VM — see [`bench/`](bench/) for full results
|
|
15
17
|
|
|
16
18
|
---
|
|
17
19
|
|
|
18
|
-
##
|
|
20
|
+
## What is ZeroMQ?
|
|
21
|
+
|
|
22
|
+
Brokerless message-oriented middleware. No central server, no extra hop — processes talk directly to each other, cutting latency in half compared to broker-based systems. You get the patterns you'd normally build on top of RabbitMQ or Redis — pub/sub, work distribution, request/reply, fan-out — but decentralized, with no single point of failure.
|
|
19
23
|
|
|
20
|
-
|
|
21
|
-
- **All socket types** — req/rep, pub/sub, push/pull, dealer/router, xpub/xsub, pair + draft types (client/server, radio/dish, scatter/gather, peer, channel)
|
|
22
|
-
- **Async-native** — built on [Async](https://github.com/socketry/async) fibers, also works with plain threads
|
|
23
|
-
- **Ruby-idiomatic API** — messages as `Array<String>`, errors as exceptions, timeouts as `IO::TimeoutError`
|
|
24
|
-
- **All transports** — tcp, ipc, inproc
|
|
24
|
+
Networking is hard. ZeroMQ abstracts away reconnection, queuing, load balancing, and framing so you can focus on what your system actually does. Start with threads talking over `inproc://`, split into processes with `ipc://`, scale across machines with `tcp://` — same code, same API, just change the URL.
|
|
25
25
|
|
|
26
|
-
|
|
26
|
+
If you've ever wired up services with raw TCP, HTTP polling, or Redis pub/sub and wished it was simpler, this is what you've been looking for.
|
|
27
27
|
|
|
28
|
-
|
|
28
|
+
See [GETTING_STARTED.md](GETTING_STARTED.md) for a ~30 min walkthrough of all major patterns with working code.
|
|
29
29
|
|
|
30
|
-
|
|
31
|
-
- **Fiber Scheduler** — non-blocking I/O without callbacks or threads (`Async` builds on this)
|
|
32
|
-
- **`io-stream`** — buffered I/O with read-ahead, from the Async ecosystem
|
|
30
|
+
## Highlights
|
|
33
31
|
|
|
34
|
-
|
|
32
|
+
- **Zero dependencies on C** — no extensions, no FFI, no libzmq. `gem install` just works everywhere
|
|
33
|
+
- **Fast** — YJIT-optimized hot paths, batched sends, 234k msg/s inproc with 12 µs latency
|
|
34
|
+
- **`omq` CLI** — pipe, filter, and transform messages from the terminal with Ruby eval, Ractor parallelism, and [script handlers](CLI.md#script-handlers--r)
|
|
35
|
+
- **Every socket pattern** — req/rep, pub/sub, push/pull, dealer/router, xpub/xsub, pair, and all draft types
|
|
36
|
+
- **Every transport** — tcp, ipc (Unix domain sockets), inproc (in-process queues)
|
|
37
|
+
- **Async-native** — built on fibers, non-blocking from the ground up. A shared IO thread handles sockets outside of Async — no reactor needed for simple scripts
|
|
38
|
+
- **Wire-compatible** — interoperates with libzmq, pyzmq, CZMQ over tcp and ipc
|
|
39
|
+
- **Bind/connect order doesn't matter** — connect before bind, bind before connect, peers come and go. ZeroMQ reconnects and requeues automatically
|
|
35
40
|
|
|
36
|
-
|
|
41
|
+
For architecture internals, see [DESIGN.md](DESIGN.md).
|
|
37
42
|
|
|
38
43
|
## Install
|
|
39
44
|
|
|
@@ -45,10 +50,6 @@ gem install omq
|
|
|
45
50
|
gem 'omq'
|
|
46
51
|
```
|
|
47
52
|
|
|
48
|
-
## Learning ZeroMQ
|
|
49
|
-
|
|
50
|
-
New to ZeroMQ? See [ZGUIDE_SUMMARY.md](ZGUIDE_SUMMARY.md) — a ~30 min read covering all major patterns with working OMQ code examples.
|
|
51
|
-
|
|
52
53
|
## Quick Start
|
|
53
54
|
|
|
54
55
|
### Request / Reply
|
|
@@ -67,7 +68,7 @@ Async do |task|
|
|
|
67
68
|
end
|
|
68
69
|
|
|
69
70
|
req << 'hello'
|
|
70
|
-
|
|
71
|
+
p req.receive # => ["HELLO"]
|
|
71
72
|
ensure
|
|
72
73
|
req&.close
|
|
73
74
|
rep&.close
|
|
@@ -83,7 +84,7 @@ Async do |task|
|
|
|
83
84
|
sub.subscribe('') # subscribe to all
|
|
84
85
|
|
|
85
86
|
task.async { pub << 'news flash' }
|
|
86
|
-
|
|
87
|
+
p sub.receive # => ["news flash"]
|
|
87
88
|
ensure
|
|
88
89
|
pub&.close
|
|
89
90
|
sub&.close
|
|
@@ -98,51 +99,56 @@ Async do
|
|
|
98
99
|
pull = OMQ::PULL.bind('inproc://pipeline')
|
|
99
100
|
|
|
100
101
|
push << 'work item'
|
|
101
|
-
|
|
102
|
+
p pull.receive # => ["work item"]
|
|
102
103
|
ensure
|
|
103
104
|
push&.close
|
|
104
105
|
pull&.close
|
|
105
106
|
end
|
|
106
107
|
```
|
|
107
108
|
|
|
108
|
-
|
|
109
|
+
### Without Async (IO thread)
|
|
109
110
|
|
|
110
|
-
|
|
111
|
-
|---------|---------|-----------|-----------|
|
|
112
|
-
| Request/Reply | `REQ`, `REP` | bidirectional | yes |
|
|
113
|
-
| Publish/Subscribe | `PUB`, `SUB`, `XPUB`, `XSUB` | unidirectional | yes |
|
|
114
|
-
| Pipeline | `PUSH`, `PULL` | unidirectional | yes |
|
|
115
|
-
| Routing | `DEALER`, `ROUTER` | bidirectional | yes |
|
|
116
|
-
| Exclusive pair | `PAIR` | bidirectional | yes |
|
|
117
|
-
| Client/Server | `CLIENT`, `SERVER` | bidirectional | no |
|
|
118
|
-
| Group messaging | `RADIO`, `DISH` | unidirectional | no |
|
|
119
|
-
| Pipeline (draft) | `SCATTER`, `GATHER` | unidirectional | no |
|
|
120
|
-
| Peer-to-peer | `PEER` | bidirectional | no |
|
|
121
|
-
| Channel (draft) | `CHANNEL` | bidirectional | no |
|
|
122
|
-
|
|
123
|
-
All classes live under `OMQ::`. For the purists, `ØMQ` is an alias:
|
|
111
|
+
OMQ spawns a shared `omq-io` thread when used outside an Async reactor — no boilerplate needed:
|
|
124
112
|
|
|
125
113
|
```ruby
|
|
126
|
-
|
|
114
|
+
require 'omq'
|
|
115
|
+
|
|
116
|
+
push = OMQ::PUSH.bind('tcp://127.0.0.1:5557')
|
|
117
|
+
pull = OMQ::PULL.connect('tcp://127.0.0.1:5557')
|
|
118
|
+
|
|
119
|
+
push << 'hello'
|
|
120
|
+
p pull.receive # => ["hello"]
|
|
121
|
+
|
|
122
|
+
push.close
|
|
123
|
+
pull.close
|
|
127
124
|
```
|
|
128
125
|
|
|
129
|
-
|
|
126
|
+
The IO thread runs all pumps, reconnection, and heartbeating in the background. When you're inside an `Async` block, OMQ uses the existing reactor instead.
|
|
130
127
|
|
|
131
|
-
|
|
128
|
+
## Socket Types
|
|
132
129
|
|
|
133
|
-
|
|
130
|
+
All sockets are thread-safe. Default HWM is 1000 messages per socket. Classes live under `OMQ::` (alias: `ØMQ`).
|
|
134
131
|
|
|
135
|
-
|
|
136
|
-
|--------|-----|-----|
|
|
137
|
-
| 244k/s | 47k/s | 36k/s |
|
|
132
|
+
#### Standard (multipart messages)
|
|
138
133
|
|
|
139
|
-
|
|
134
|
+
| Pattern | Send | Receive | When HWM full |
|
|
135
|
+
|---------|------|---------|---------------|
|
|
136
|
+
| **REQ** / **REP** | Round-robin / route-back | Fair-queue | Block |
|
|
137
|
+
| **PUB** / **SUB** | Fan-out to subscribers | Subscription filter | Drop |
|
|
138
|
+
| **PUSH** / **PULL** | Round-robin to workers | Fair-queue | Block |
|
|
139
|
+
| **DEALER** / **ROUTER** | Round-robin / identity-route | Fair-queue | Block |
|
|
140
|
+
| **XPUB** / **XSUB** | Fan-out (subscription events) | Fair-queue | Drop |
|
|
141
|
+
| **PAIR** | Exclusive 1-to-1 | Exclusive 1-to-1 | Block |
|
|
140
142
|
|
|
141
|
-
|
|
142
|
-
|--------|-----|-----|
|
|
143
|
-
| 9 µs | 47 µs | 61 µs |
|
|
143
|
+
#### Draft (single-frame only)
|
|
144
144
|
|
|
145
|
-
|
|
145
|
+
| Pattern | Send | Receive | When HWM full |
|
|
146
|
+
|---------|------|---------|---------------|
|
|
147
|
+
| **CLIENT** / **SERVER** | Round-robin / routing-ID | Fair-queue | Block |
|
|
148
|
+
| **RADIO** / **DISH** | Group fan-out | Group filter | Drop |
|
|
149
|
+
| **SCATTER** / **GATHER** | Round-robin | Fair-queue | Block |
|
|
150
|
+
| **PEER** | Routing-ID | Fair-queue | Block |
|
|
151
|
+
| **CHANNEL** | Exclusive 1-to-1 | Exclusive 1-to-1 | Block |
|
|
146
152
|
|
|
147
153
|
## omq — CLI tool
|
|
148
154
|
|
|
@@ -152,7 +158,7 @@ See [`bench/`](bench/) for full results and scripts.
|
|
|
152
158
|
# Echo server
|
|
153
159
|
omq rep -b tcp://:5555 --echo
|
|
154
160
|
|
|
155
|
-
# Upcase server
|
|
161
|
+
# Upcase server — -e evals Ruby on each incoming message
|
|
156
162
|
omq rep -b tcp://:5555 -e '$F.map(&:upcase)'
|
|
157
163
|
|
|
158
164
|
# Client
|
|
@@ -163,50 +169,39 @@ echo "hello" | omq req -c tcp://localhost:5555
|
|
|
163
169
|
omq sub -b tcp://:5556 -s "weather." &
|
|
164
170
|
echo "weather.nyc 72F" | omq pub -c tcp://localhost:5556 -d 0.3
|
|
165
171
|
|
|
166
|
-
# Pipeline with filtering
|
|
167
|
-
# /regexp/ matches against $_, next skips, break stops
|
|
172
|
+
# Pipeline with filtering
|
|
168
173
|
tail -f /var/log/syslog | omq push -c tcp://collector:5557
|
|
169
174
|
omq pull -b tcp://:5557 -e 'next unless /error/; $F'
|
|
170
175
|
|
|
176
|
+
# Transform outgoing messages with -E
|
|
177
|
+
echo hello | omq push -c tcp://localhost:5557 -E '$F.map(&:upcase)'
|
|
178
|
+
|
|
179
|
+
# REQ: transform request and reply independently
|
|
180
|
+
echo hello | omq req -c tcp://localhost:5555 \
|
|
181
|
+
-E '$F.map(&:upcase)' -e '$F.map(&:reverse)'
|
|
182
|
+
|
|
171
183
|
# Pipe: PULL → eval → PUSH in one process
|
|
172
184
|
omq pipe -c ipc://@work -c ipc://@sink -e '$F.map(&:upcase)'
|
|
173
185
|
|
|
174
186
|
# Pipe with Ractor workers for CPU parallelism (-P = all CPUs)
|
|
175
187
|
omq pipe -c ipc://@work -c ipc://@sink -P -r./fib -e 'fib(Integer($_)).to_s'
|
|
188
|
+
```
|
|
176
189
|
|
|
177
|
-
|
|
178
|
-
omq pipe -c ipc://@work -c ipc://@sink --transient -e '$F'
|
|
179
|
-
|
|
180
|
-
# JSONL for structured data
|
|
181
|
-
echo '["key","value"]' | omq push -c tcp://localhost:5557 -J
|
|
182
|
-
omq pull -b tcp://:5557 -J
|
|
190
|
+
`-e` (recv-eval) transforms incoming messages, `-E` (send-eval) transforms outgoing messages. `$F` is the message parts array, `$_` is the first part. Use `-r` to require gems or load scripts that register handlers via `OMQ.incoming` / `OMQ.outgoing`:
|
|
183
191
|
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
192
|
+
```ruby
|
|
193
|
+
# my_handler.rb
|
|
194
|
+
db = DB.connect("postgres://localhost/app")
|
|
187
195
|
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
# prints: OMQ_SERVER_KEY='...'
|
|
191
|
-
omq req -c tcp://localhost:5555 --curve-server-key '...'
|
|
196
|
+
OMQ.incoming { db.query($F.first) }
|
|
197
|
+
at_exit { db.close }
|
|
192
198
|
```
|
|
193
199
|
|
|
194
|
-
The `-e` flag runs Ruby inside the socket instance — the full socket API (`self <<`, `send`, `subscribe`, ...) is available. `$F` is the message parts array, `$_` is the first part. Use `-r` to require gems:
|
|
195
|
-
|
|
196
200
|
```sh
|
|
197
|
-
omq
|
|
198
|
-
-e 'JSON.parse($F.first)["temperature"]'
|
|
199
|
-
|
|
200
|
-
# BEGIN/END blocks (like awk) — accumulate and summarize
|
|
201
|
-
omq pull -b tcp://:5557 \
|
|
202
|
-
-e 'BEGIN{ @sum = 0 } @sum += Integer($_); next END{ puts @sum }'
|
|
201
|
+
omq pull -b tcp://:5557 -r./my_handler.rb
|
|
203
202
|
```
|
|
204
203
|
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
## Interop with native ZMQ
|
|
208
|
-
|
|
209
|
-
OMQ speaks ZMTP 3.1 on the wire and interoperates with libzmq, CZMQ, pyzmq, etc. over **tcp** and **ipc**. The `inproc://` transport is OMQ-internal (in-process Ruby queues) and is not visible to native ZMQ running in the same process — use `ipc://` to talk across library boundaries.
|
|
204
|
+
See [CLI.md](CLI.md) for full documentation, or `omq --help` / `omq --examples`.
|
|
210
205
|
|
|
211
206
|
## Development
|
|
212
207
|
|
data/lib/omq/cli/base_runner.rb
CHANGED
|
@@ -28,9 +28,11 @@ module OMQ
|
|
|
28
28
|
sleep(config.delay) if config.delay && config.recv_only?
|
|
29
29
|
wait_for_peer if needs_peer_wait?
|
|
30
30
|
|
|
31
|
-
@sock.instance_exec(&@
|
|
31
|
+
@sock.instance_exec(&@send_begin_proc) if @send_begin_proc
|
|
32
|
+
@sock.instance_exec(&@recv_begin_proc) if @recv_begin_proc
|
|
32
33
|
run_loop(task)
|
|
33
|
-
@sock.instance_exec(&@
|
|
34
|
+
@sock.instance_exec(&@send_end_proc) if @send_end_proc
|
|
35
|
+
@sock.instance_exec(&@recv_end_proc) if @recv_end_proc
|
|
34
36
|
ensure
|
|
35
37
|
@sock&.close
|
|
36
38
|
end
|
|
@@ -201,19 +203,19 @@ module OMQ
|
|
|
201
203
|
end
|
|
202
204
|
end
|
|
203
205
|
elsif config.data || config.file
|
|
204
|
-
parts =
|
|
206
|
+
parts = eval_send_expr(read_next)
|
|
205
207
|
send_msg(parts) if parts
|
|
206
208
|
elsif stdin_ready?
|
|
207
209
|
loop do
|
|
208
210
|
parts = read_next
|
|
209
211
|
break unless parts
|
|
210
|
-
parts =
|
|
212
|
+
parts = eval_send_expr(parts)
|
|
211
213
|
send_msg(parts) if parts
|
|
212
214
|
i += 1
|
|
213
215
|
break if n && n > 0 && i >= n
|
|
214
216
|
end
|
|
215
|
-
elsif @
|
|
216
|
-
parts =
|
|
217
|
+
elsif @send_eval_proc
|
|
218
|
+
parts = eval_send_expr(nil)
|
|
217
219
|
send_msg(parts) if parts
|
|
218
220
|
end
|
|
219
221
|
end
|
|
@@ -221,11 +223,11 @@ module OMQ
|
|
|
221
223
|
|
|
222
224
|
def send_tick
|
|
223
225
|
raw = read_next_or_nil
|
|
224
|
-
if raw.nil? && !@
|
|
226
|
+
if raw.nil? && !@send_eval_proc
|
|
225
227
|
@send_tick_eof = true
|
|
226
228
|
return 0
|
|
227
229
|
end
|
|
228
|
-
parts =
|
|
230
|
+
parts = eval_send_expr(raw)
|
|
229
231
|
send_msg(parts) if parts
|
|
230
232
|
1
|
|
231
233
|
end
|
|
@@ -237,7 +239,7 @@ module OMQ
|
|
|
237
239
|
loop do
|
|
238
240
|
parts = recv_msg
|
|
239
241
|
break if parts.nil?
|
|
240
|
-
parts =
|
|
242
|
+
parts = eval_recv_expr(parts)
|
|
241
243
|
output(parts)
|
|
242
244
|
i += 1
|
|
243
245
|
break if n && n > 0 && i >= n
|
|
@@ -246,7 +248,7 @@ module OMQ
|
|
|
246
248
|
|
|
247
249
|
|
|
248
250
|
def wait_for_loops(receiver, sender)
|
|
249
|
-
if config.data || config.file || config.
|
|
251
|
+
if config.data || config.file || config.send_expr || config.recv_expr || config.target
|
|
250
252
|
sender.wait
|
|
251
253
|
receiver.stop
|
|
252
254
|
elsif config.count && config.count > 0
|
|
@@ -281,7 +283,8 @@ module OMQ
|
|
|
281
283
|
|
|
282
284
|
|
|
283
285
|
def recv_msg_raw
|
|
284
|
-
@sock.receive
|
|
286
|
+
msg = @sock.receive
|
|
287
|
+
msg&.dup
|
|
285
288
|
end
|
|
286
289
|
|
|
287
290
|
|
|
@@ -320,7 +323,7 @@ module OMQ
|
|
|
320
323
|
def read_next_or_nil
|
|
321
324
|
if config.data || config.file
|
|
322
325
|
read_next
|
|
323
|
-
elsif @
|
|
326
|
+
elsif @send_eval_proc
|
|
324
327
|
nil
|
|
325
328
|
else
|
|
326
329
|
read_next
|
|
@@ -358,11 +361,30 @@ module OMQ
|
|
|
358
361
|
|
|
359
362
|
|
|
360
363
|
def compile_expr
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
@
|
|
364
|
-
@
|
|
365
|
-
|
|
364
|
+
compile_one_expr(:send, config.send_expr)
|
|
365
|
+
compile_one_expr(:recv, config.recv_expr)
|
|
366
|
+
@send_eval_proc ||= wrap_registered_proc(OMQ.outgoing_proc)
|
|
367
|
+
@recv_eval_proc ||= wrap_registered_proc(OMQ.incoming_proc)
|
|
368
|
+
end
|
|
369
|
+
|
|
370
|
+
|
|
371
|
+
def wrap_registered_proc(block)
|
|
372
|
+
return unless block
|
|
373
|
+
proc do |msg|
|
|
374
|
+
$_ = msg&.first
|
|
375
|
+
block.call(msg)
|
|
376
|
+
end
|
|
377
|
+
end
|
|
378
|
+
|
|
379
|
+
|
|
380
|
+
def compile_one_expr(direction, src)
|
|
381
|
+
return unless src
|
|
382
|
+
expr, begin_body, end_body = extract_blocks(src)
|
|
383
|
+
instance_variable_set(:"@#{direction}_begin_proc", eval("proc { #{begin_body} }")) if begin_body
|
|
384
|
+
instance_variable_set(:"@#{direction}_end_proc", eval("proc { #{end_body} }")) if end_body
|
|
385
|
+
if expr && !expr.strip.empty?
|
|
386
|
+
instance_variable_set(:"@#{direction}_eval_proc", eval("proc { $_ = $F&.first; #{expr} }"))
|
|
387
|
+
end
|
|
366
388
|
end
|
|
367
389
|
|
|
368
390
|
|
|
@@ -398,10 +420,21 @@ module OMQ
|
|
|
398
420
|
|
|
399
421
|
SENT = Object.new.freeze # sentinel: eval already sent the reply
|
|
400
422
|
|
|
401
|
-
def
|
|
402
|
-
return parts unless @
|
|
423
|
+
def eval_send_expr(parts)
|
|
424
|
+
return parts unless @send_eval_proc
|
|
425
|
+
run_eval(@send_eval_proc, parts)
|
|
426
|
+
end
|
|
427
|
+
|
|
428
|
+
|
|
429
|
+
def eval_recv_expr(parts)
|
|
430
|
+
return parts unless @recv_eval_proc
|
|
431
|
+
run_eval(@recv_eval_proc, parts)
|
|
432
|
+
end
|
|
433
|
+
|
|
434
|
+
|
|
435
|
+
def run_eval(eval_proc, parts)
|
|
403
436
|
$F = parts
|
|
404
|
-
result = @sock.instance_exec(
|
|
437
|
+
result = @sock.instance_exec(parts, &eval_proc)
|
|
405
438
|
return nil if result.nil?
|
|
406
439
|
return SENT if result.equal?(@sock)
|
|
407
440
|
return [result] if config.format == :marshal
|
|
@@ -411,7 +444,7 @@ module OMQ
|
|
|
411
444
|
else [result.to_str]
|
|
412
445
|
end
|
|
413
446
|
rescue => e
|
|
414
|
-
$stderr.puts "omq:
|
|
447
|
+
$stderr.puts "omq: eval error: #{e.message} (#{e.class})"
|
|
415
448
|
exit 3
|
|
416
449
|
end
|
|
417
450
|
|
|
@@ -11,7 +11,7 @@ module OMQ
|
|
|
11
11
|
|
|
12
12
|
|
|
13
13
|
def run_loop(task)
|
|
14
|
-
if config.echo || config.
|
|
14
|
+
if config.echo || config.recv_expr || @recv_eval_proc || config.data || config.file || !config.stdin_is_tty
|
|
15
15
|
reply_loop
|
|
16
16
|
else
|
|
17
17
|
monitor_loop(task)
|
|
@@ -28,8 +28,8 @@ module OMQ
|
|
|
28
28
|
routing_id = parts.shift
|
|
29
29
|
body = @fmt.decompress(parts)
|
|
30
30
|
|
|
31
|
-
if config.
|
|
32
|
-
reply =
|
|
31
|
+
if config.recv_expr || @recv_eval_proc
|
|
32
|
+
reply = eval_recv_expr(body)
|
|
33
33
|
output([display_routing_id(routing_id), *(reply || [""])])
|
|
34
34
|
@sock.send_to(routing_id, @fmt.compress(reply || [""]).first)
|
|
35
35
|
elsif config.echo
|
|
@@ -56,7 +56,7 @@ module OMQ
|
|
|
56
56
|
break if parts.nil?
|
|
57
57
|
routing_id = parts.shift
|
|
58
58
|
parts = @fmt.decompress(parts)
|
|
59
|
-
result =
|
|
59
|
+
result = eval_recv_expr([display_routing_id(routing_id), *parts])
|
|
60
60
|
output(result)
|
|
61
61
|
i += 1
|
|
62
62
|
break if n && n > 0 && i >= n
|
|
@@ -71,18 +71,18 @@ module OMQ
|
|
|
71
71
|
Async::Loop.quantized(interval: config.interval) do
|
|
72
72
|
parts = read_next
|
|
73
73
|
break unless parts
|
|
74
|
-
|
|
74
|
+
send_targeted_or_eval(parts)
|
|
75
75
|
i += 1
|
|
76
76
|
break if n && n > 0 && i >= n
|
|
77
77
|
end
|
|
78
78
|
elsif config.data || config.file
|
|
79
79
|
parts = read_next
|
|
80
|
-
|
|
80
|
+
send_targeted_or_eval(parts) if parts
|
|
81
81
|
else
|
|
82
82
|
loop do
|
|
83
83
|
parts = read_next
|
|
84
84
|
break unless parts
|
|
85
|
-
|
|
85
|
+
send_targeted_or_eval(parts)
|
|
86
86
|
i += 1
|
|
87
87
|
break if n && n > 0 && i >= n
|
|
88
88
|
end
|
|
@@ -93,8 +93,13 @@ module OMQ
|
|
|
93
93
|
end
|
|
94
94
|
|
|
95
95
|
|
|
96
|
-
def
|
|
97
|
-
if
|
|
96
|
+
def send_targeted_or_eval(parts)
|
|
97
|
+
if @send_eval_proc
|
|
98
|
+
parts = eval_send_expr(parts)
|
|
99
|
+
return unless parts
|
|
100
|
+
routing_id = resolve_target(parts.shift)
|
|
101
|
+
@sock.send_to(routing_id, @fmt.compress(parts).first || "")
|
|
102
|
+
elsif config.target
|
|
98
103
|
parts = @fmt.compress(parts)
|
|
99
104
|
@sock.send_to(resolve_target(config.target), parts.first || "")
|
|
100
105
|
else
|
data/lib/omq/cli/config.rb
CHANGED
|
@@ -2,6 +2,9 @@
|
|
|
2
2
|
|
|
3
3
|
module OMQ
|
|
4
4
|
module CLI
|
|
5
|
+
SEND_ONLY = %w[pub push scatter radio].freeze
|
|
6
|
+
RECV_ONLY = %w[sub pull gather dish].freeze
|
|
7
|
+
|
|
5
8
|
Endpoint = Data.define(:url, :bind?) do
|
|
6
9
|
def connect? = !bind?
|
|
7
10
|
end
|
|
@@ -12,6 +15,8 @@ module OMQ
|
|
|
12
15
|
:endpoints,
|
|
13
16
|
:connects,
|
|
14
17
|
:binds,
|
|
18
|
+
:in_endpoints,
|
|
19
|
+
:out_endpoints,
|
|
15
20
|
:data,
|
|
16
21
|
:file,
|
|
17
22
|
:format,
|
|
@@ -29,7 +34,8 @@ module OMQ
|
|
|
29
34
|
:heartbeat_ivl,
|
|
30
35
|
:conflate,
|
|
31
36
|
:compress,
|
|
32
|
-
:
|
|
37
|
+
:send_expr,
|
|
38
|
+
:recv_expr,
|
|
33
39
|
:parallel,
|
|
34
40
|
:transient,
|
|
35
41
|
:verbose,
|
|
@@ -41,9 +47,6 @@ module OMQ
|
|
|
41
47
|
:has_zstd,
|
|
42
48
|
:stdin_is_tty,
|
|
43
49
|
) do
|
|
44
|
-
SEND_ONLY = %w[pub push scatter radio].freeze
|
|
45
|
-
RECV_ONLY = %w[sub pull gather dish].freeze
|
|
46
|
-
|
|
47
50
|
def send_only? = SEND_ONLY.include?(type_name)
|
|
48
51
|
def recv_only? = RECV_ONLY.include?(type_name)
|
|
49
52
|
end
|
data/lib/omq/cli/pair.rb
CHANGED
data/lib/omq/cli/pipe.rb
CHANGED
|
@@ -24,9 +24,22 @@ module OMQ
|
|
|
24
24
|
private
|
|
25
25
|
|
|
26
26
|
|
|
27
|
+
def resolve_endpoints
|
|
28
|
+
if config.in_endpoints.any?
|
|
29
|
+
[config.in_endpoints, config.out_endpoints]
|
|
30
|
+
else
|
|
31
|
+
[[config.endpoints[0]], [config.endpoints[1]]]
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
|
|
36
|
+
def attach_endpoints(sock, endpoints)
|
|
37
|
+
endpoints.each { |ep| ep.bind? ? sock.bind(ep.url) : sock.connect(ep.url) }
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
|
|
27
41
|
def run_sequential(task)
|
|
28
|
-
|
|
29
|
-
push_ep = config.endpoints[1]
|
|
42
|
+
in_eps, out_eps = resolve_endpoints
|
|
30
43
|
|
|
31
44
|
@pull = OMQ::PULL.new(linger: config.linger, recv_timeout: config.timeout)
|
|
32
45
|
@push = OMQ::PUSH.new(linger: config.linger, send_timeout: config.timeout)
|
|
@@ -35,11 +48,11 @@ module OMQ
|
|
|
35
48
|
@pull.heartbeat_interval = config.heartbeat_ivl if config.heartbeat_ivl
|
|
36
49
|
@push.heartbeat_interval = config.heartbeat_ivl if config.heartbeat_ivl
|
|
37
50
|
|
|
38
|
-
|
|
39
|
-
|
|
51
|
+
attach_endpoints(@pull, in_eps)
|
|
52
|
+
attach_endpoints(@push, out_eps)
|
|
40
53
|
|
|
41
54
|
compile_expr
|
|
42
|
-
@sock = @pull # for
|
|
55
|
+
@sock = @pull # for eval instance_exec
|
|
43
56
|
|
|
44
57
|
with_timeout(config.timeout) do
|
|
45
58
|
@push.peer_connected.wait
|
|
@@ -54,7 +67,7 @@ module OMQ
|
|
|
54
67
|
end
|
|
55
68
|
end
|
|
56
69
|
|
|
57
|
-
@sock.instance_exec(&@
|
|
70
|
+
@sock.instance_exec(&@recv_begin_proc) if @recv_begin_proc
|
|
58
71
|
|
|
59
72
|
n = config.count
|
|
60
73
|
i = 0
|
|
@@ -62,7 +75,7 @@ module OMQ
|
|
|
62
75
|
parts = @pull.receive
|
|
63
76
|
break if parts.nil?
|
|
64
77
|
parts = @fmt.decompress(parts)
|
|
65
|
-
parts =
|
|
78
|
+
parts = eval_recv_expr(parts)
|
|
66
79
|
if parts && !parts.empty?
|
|
67
80
|
@push.send(@fmt.compress(parts))
|
|
68
81
|
end
|
|
@@ -70,7 +83,7 @@ module OMQ
|
|
|
70
83
|
break if n && n > 0 && i >= n
|
|
71
84
|
end
|
|
72
85
|
|
|
73
|
-
@sock.instance_exec(&@
|
|
86
|
+
@sock.instance_exec(&@recv_end_proc) if @recv_end_proc
|
|
74
87
|
ensure
|
|
75
88
|
@pull&.close
|
|
76
89
|
@push&.close
|
|
@@ -86,7 +99,7 @@ module OMQ
|
|
|
86
99
|
Sync do |task|
|
|
87
100
|
# Parse BEGIN/END blocks and per-message expression
|
|
88
101
|
begin_proc = end_proc = eval_proc = nil
|
|
89
|
-
if cfg.
|
|
102
|
+
if cfg.recv_expr
|
|
90
103
|
extract = ->(src, kw) {
|
|
91
104
|
s = src.index(/#{kw}\s*\{/)
|
|
92
105
|
return [src, nil] unless s
|
|
@@ -97,7 +110,7 @@ module OMQ
|
|
|
97
110
|
end
|
|
98
111
|
[src[0...s] + src[j..], src[(i + 1)..(j - 2)]]
|
|
99
112
|
}
|
|
100
|
-
expr, begin_body = extract.(cfg.
|
|
113
|
+
expr, begin_body = extract.(cfg.recv_expr, "BEGIN")
|
|
101
114
|
expr, end_body = extract.(expr, "END")
|
|
102
115
|
begin_proc = eval("proc { #{begin_body} }") if begin_body
|
|
103
116
|
end_proc = eval("proc { #{end_body} }") if end_body
|
|
@@ -115,8 +128,10 @@ module OMQ
|
|
|
115
128
|
push.reconnect_interval = cfg.reconnect_ivl if cfg.reconnect_ivl
|
|
116
129
|
pull.heartbeat_interval = cfg.heartbeat_ivl if cfg.heartbeat_ivl
|
|
117
130
|
push.heartbeat_interval = cfg.heartbeat_ivl if cfg.heartbeat_ivl
|
|
118
|
-
|
|
119
|
-
|
|
131
|
+
in_eps = cfg.in_endpoints.any? ? cfg.in_endpoints : [cfg.endpoints[0]]
|
|
132
|
+
out_eps = cfg.out_endpoints.any? ? cfg.out_endpoints : [cfg.endpoints[1]]
|
|
133
|
+
in_eps.each { |ep| pull.connect(ep.url) }
|
|
134
|
+
out_eps.each { |ep| push.connect(ep.url) }
|
|
120
135
|
|
|
121
136
|
if cfg.timeout
|
|
122
137
|
task.with_timeout(cfg.timeout) do
|
|
@@ -187,11 +202,12 @@ module OMQ
|
|
|
187
202
|
|
|
188
203
|
|
|
189
204
|
def compile_expr
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
@
|
|
194
|
-
@
|
|
205
|
+
src = config.recv_expr
|
|
206
|
+
return unless src
|
|
207
|
+
expr, begin_body, end_body = extract_blocks(src)
|
|
208
|
+
@recv_begin_proc = eval("proc { #{begin_body} }") if begin_body
|
|
209
|
+
@recv_end_proc = eval("proc { #{end_body} }") if end_body
|
|
210
|
+
@recv_eval_proc = eval("proc { $_ = $F&.first; #{expr} }") if expr && !expr.strip.empty?
|
|
195
211
|
end
|
|
196
212
|
|
|
197
213
|
|
|
@@ -224,10 +240,10 @@ module OMQ
|
|
|
224
240
|
end
|
|
225
241
|
|
|
226
242
|
|
|
227
|
-
def
|
|
228
|
-
return parts unless @
|
|
243
|
+
def eval_recv_expr(parts)
|
|
244
|
+
return parts unless @recv_eval_proc
|
|
229
245
|
$F = parts
|
|
230
|
-
result = @sock.instance_exec(&@
|
|
246
|
+
result = @sock.instance_exec(&@recv_eval_proc)
|
|
231
247
|
return nil if result.nil? || result.equal?(@sock)
|
|
232
248
|
return [result] if config.format == :marshal
|
|
233
249
|
case result
|
|
@@ -236,7 +252,7 @@ module OMQ
|
|
|
236
252
|
else [result.to_str]
|
|
237
253
|
end
|
|
238
254
|
rescue => e
|
|
239
|
-
$stderr.puts "omq:
|
|
255
|
+
$stderr.puts "omq: eval error: #{e.message} (#{e.class})"
|
|
240
256
|
exit 3
|
|
241
257
|
end
|
|
242
258
|
|
data/lib/omq/cli/req_rep.rb
CHANGED
|
@@ -14,10 +14,12 @@ module OMQ
|
|
|
14
14
|
loop do
|
|
15
15
|
parts = read_next
|
|
16
16
|
break unless parts
|
|
17
|
+
parts = eval_send_expr(parts)
|
|
18
|
+
next unless parts
|
|
17
19
|
send_msg(parts)
|
|
18
20
|
reply = recv_msg
|
|
19
21
|
break if reply.nil?
|
|
20
|
-
reply =
|
|
22
|
+
reply = eval_recv_expr(reply)
|
|
21
23
|
output(reply)
|
|
22
24
|
i += 1
|
|
23
25
|
break if n && n > 0 && i >= n
|
|
@@ -29,10 +31,12 @@ module OMQ
|
|
|
29
31
|
loop do
|
|
30
32
|
parts = read_next
|
|
31
33
|
break unless parts
|
|
34
|
+
parts = eval_send_expr(parts)
|
|
35
|
+
next unless parts
|
|
32
36
|
send_msg(parts)
|
|
33
37
|
reply = recv_msg
|
|
34
38
|
break if reply.nil?
|
|
35
|
-
reply =
|
|
39
|
+
reply = eval_recv_expr(reply)
|
|
36
40
|
output(reply)
|
|
37
41
|
i += 1
|
|
38
42
|
break if n && n > 0 && i >= n
|
|
@@ -53,8 +57,8 @@ module OMQ
|
|
|
53
57
|
loop do
|
|
54
58
|
msg = recv_msg
|
|
55
59
|
break if msg.nil?
|
|
56
|
-
if config.
|
|
57
|
-
reply =
|
|
60
|
+
if config.recv_expr || @recv_eval_proc
|
|
61
|
+
reply = eval_recv_expr(msg)
|
|
58
62
|
unless reply.equal?(SENT)
|
|
59
63
|
output(reply)
|
|
60
64
|
send_msg(reply || [""])
|
|
@@ -20,7 +20,7 @@ module OMQ
|
|
|
20
20
|
identity = parts.shift
|
|
21
21
|
parts.shift if parts.first == ""
|
|
22
22
|
parts = @fmt.decompress(parts)
|
|
23
|
-
result =
|
|
23
|
+
result = eval_recv_expr([display_routing_id(identity), *parts])
|
|
24
24
|
output(result)
|
|
25
25
|
i += 1
|
|
26
26
|
break if n && n > 0 && i >= n
|
|
@@ -35,18 +35,18 @@ module OMQ
|
|
|
35
35
|
Async::Loop.quantized(interval: config.interval) do
|
|
36
36
|
parts = read_next
|
|
37
37
|
break unless parts
|
|
38
|
-
|
|
38
|
+
send_targeted_or_eval(parts)
|
|
39
39
|
i += 1
|
|
40
40
|
break if n && n > 0 && i >= n
|
|
41
41
|
end
|
|
42
42
|
elsif config.data || config.file
|
|
43
43
|
parts = read_next
|
|
44
|
-
|
|
44
|
+
send_targeted_or_eval(parts) if parts
|
|
45
45
|
else
|
|
46
46
|
loop do
|
|
47
47
|
parts = read_next
|
|
48
48
|
break unless parts
|
|
49
|
-
|
|
49
|
+
send_targeted_or_eval(parts)
|
|
50
50
|
i += 1
|
|
51
51
|
break if n && n > 0 && i >= n
|
|
52
52
|
end
|
|
@@ -57,8 +57,14 @@ module OMQ
|
|
|
57
57
|
end
|
|
58
58
|
|
|
59
59
|
|
|
60
|
-
def
|
|
61
|
-
if
|
|
60
|
+
def send_targeted_or_eval(parts)
|
|
61
|
+
if @send_eval_proc
|
|
62
|
+
parts = eval_send_expr(parts)
|
|
63
|
+
return unless parts
|
|
64
|
+
identity = resolve_target(parts.shift)
|
|
65
|
+
payload = @fmt.compress(parts)
|
|
66
|
+
@sock.send([identity, "", *payload])
|
|
67
|
+
elsif config.target
|
|
62
68
|
payload = @fmt.compress(parts)
|
|
63
69
|
@sock.send([resolve_target(config.target), "", *payload])
|
|
64
70
|
else
|
data/lib/omq/cli.rb
CHANGED
|
@@ -17,6 +17,15 @@ require_relative "cli/peer"
|
|
|
17
17
|
require_relative "cli/pipe"
|
|
18
18
|
|
|
19
19
|
module OMQ
|
|
20
|
+
|
|
21
|
+
class << self
|
|
22
|
+
attr_reader :outgoing_proc, :incoming_proc
|
|
23
|
+
|
|
24
|
+
def outgoing(&block) = @outgoing_proc = block
|
|
25
|
+
def incoming(&block) = @incoming_proc = block
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
|
|
20
29
|
module CLI
|
|
21
30
|
SOCKET_TYPE_NAMES = %w[
|
|
22
31
|
req rep pub sub push pull pair dealer router
|
|
@@ -56,7 +65,7 @@ module OMQ
|
|
|
56
65
|
└─────┘ "HELLO" └─────┘
|
|
57
66
|
|
|
58
67
|
# terminal 1: echo server
|
|
59
|
-
omq rep --bind tcp://:5555 --eval '$F.map(&:upcase)'
|
|
68
|
+
omq rep --bind tcp://:5555 --recv-eval '$F.map(&:upcase)'
|
|
60
69
|
|
|
61
70
|
# terminal 2: send a request
|
|
62
71
|
echo "hello" | omq req --connect tcp://localhost:5555
|
|
@@ -79,7 +88,7 @@ module OMQ
|
|
|
79
88
|
|
|
80
89
|
── Periodic Publish ───────────────────────────────────────────
|
|
81
90
|
|
|
82
|
-
┌─────┐ "tick 1"
|
|
91
|
+
┌─────┐ "tick 1" ┌─────┐
|
|
83
92
|
│ PUB │──(every 1s)─→│ SUB │
|
|
84
93
|
└─────┘ └─────┘
|
|
85
94
|
|
|
@@ -121,27 +130,31 @@ module OMQ
|
|
|
121
130
|
|
|
122
131
|
# terminal 2: worker — uppercase each message
|
|
123
132
|
omq pipe -c ipc://@work -c ipc://@sink -e '$F.map(&:upcase)'
|
|
124
|
-
|
|
125
133
|
# terminal 3: collector
|
|
126
134
|
omq pull --bind ipc://@sink
|
|
127
135
|
|
|
128
136
|
# 4 Ractor workers in a single process (-P)
|
|
129
|
-
omq pipe -c ipc://@work -c ipc://@sink -
|
|
130
|
-
-r ./fib.rb -e 'fib(Integer($_)).to_s'
|
|
137
|
+
omq pipe -c ipc://@work -c ipc://@sink -P4 -r./fib -e 'fib(Integer($_)).to_s'
|
|
131
138
|
|
|
132
139
|
# exit when producer disconnects (--transient)
|
|
133
|
-
omq pipe -c ipc://@work -c ipc://@sink --transient
|
|
134
|
-
|
|
140
|
+
omq pipe -c ipc://@work -c ipc://@sink --transient -e '$F.map(&:upcase)'
|
|
141
|
+
|
|
142
|
+
# fan-in: multiple sources → one sink
|
|
143
|
+
omq pipe --in -c ipc://@work1 -c ipc://@work2 \
|
|
144
|
+
--out -c ipc://@sink -e '$F.map(&:upcase)'
|
|
145
|
+
|
|
146
|
+
# fan-out: one source → multiple sinks (round-robin)
|
|
147
|
+
omq pipe --in -b tcp://:5555 --out -c ipc://@sink1 -c ipc://@sink2 -e '$F'
|
|
135
148
|
|
|
136
149
|
── CLIENT / SERVER (draft) ──────────────────────────────────
|
|
137
150
|
|
|
138
151
|
┌────────┐ "hello" ┌────────┐
|
|
139
|
-
│ CLIENT │───────────→│ SERVER │ --eval '$F.map(&:upcase)'
|
|
152
|
+
│ CLIENT │───────────→│ SERVER │ --recv-eval '$F.map(&:upcase)'
|
|
140
153
|
│ │←───────────│ │
|
|
141
154
|
└────────┘ "HELLO" └────────┘
|
|
142
155
|
|
|
143
156
|
# terminal 1: upcasing server
|
|
144
|
-
omq server --bind tcp://:5555 --eval '$F.map(&:upcase)'
|
|
157
|
+
omq server --bind tcp://:5555 --recv-eval '$F.map(&:upcase)'
|
|
145
158
|
|
|
146
159
|
# terminal 2: client
|
|
147
160
|
echo "hello" | omq client --connect tcp://localhost:5555
|
|
@@ -187,30 +200,45 @@ module OMQ
|
|
|
187
200
|
omq router --bind tcp://:5555
|
|
188
201
|
|
|
189
202
|
# terminal 2: dealer with identity
|
|
190
|
-
echo "hello" | omq dealer --connect tcp://localhost:5555
|
|
191
|
-
--identity worker-1
|
|
203
|
+
echo "hello" | omq dealer --connect tcp://localhost:5555 --identity worker-1
|
|
192
204
|
|
|
193
205
|
── Ruby Eval ────────────────────────────────────────────────
|
|
194
206
|
|
|
195
|
-
# filter: only pass messages containing "error"
|
|
196
|
-
omq pull
|
|
197
|
-
--eval '$F.first.include?("error") ? $F : nil'
|
|
207
|
+
# filter incoming: only pass messages containing "error"
|
|
208
|
+
omq pull -b tcp://:5557 --recv-eval '$F.first.include?("error") ? $F : nil'
|
|
198
209
|
|
|
199
|
-
# transform with gems
|
|
200
|
-
omq sub
|
|
201
|
-
--eval 'JSON.parse($F.first)["temperature"]'
|
|
210
|
+
# transform incoming with gems
|
|
211
|
+
omq sub -c tcp://localhost:5556 -rjson -e 'JSON.parse($F.first)["temperature"]'
|
|
202
212
|
|
|
203
|
-
# require a local file, use its methods
|
|
204
|
-
omq rep --bind tcp://:5555 --require ./transform.rb
|
|
205
|
-
--eval 'upcase_all($F)'
|
|
213
|
+
# require a local file, use its methods
|
|
214
|
+
omq rep --bind tcp://:5555 --require ./transform.rb -e 'upcase_all($F)'
|
|
206
215
|
|
|
207
216
|
# next skips, break stops — regexps match against $_
|
|
208
|
-
omq pull
|
|
209
|
-
--eval 'next if /^#/; break if /quit/; $F'
|
|
217
|
+
omq pull -b tcp://:5557 -e 'next if /^#/; break if /quit/; $F'
|
|
210
218
|
|
|
211
219
|
# BEGIN/END blocks (like awk) — accumulate and summarize
|
|
212
|
-
omq pull
|
|
213
|
-
|
|
220
|
+
omq pull -b tcp://:5557 -e 'BEGIN{@sum = 0} @sum += Integer($_); nil END{puts @sum}'
|
|
221
|
+
|
|
222
|
+
# transform outgoing messages
|
|
223
|
+
echo hello | omq push -c tcp://localhost:5557 --send-eval '$F.map(&:upcase)'
|
|
224
|
+
|
|
225
|
+
# REQ: transform request and reply independently
|
|
226
|
+
echo hello | omq req -c tcp://localhost:5555 -E '$F.map(&:upcase)' -e '$_'
|
|
227
|
+
|
|
228
|
+
── Script Handlers (-r) ────────────────────────────────────
|
|
229
|
+
|
|
230
|
+
# handler.rb — register transforms from a file
|
|
231
|
+
# db = PG.connect("dbname=app")
|
|
232
|
+
# OMQ.incoming { |first_part, _| db.exec(first_part).values.flatten }
|
|
233
|
+
# at_exit { db.close }
|
|
234
|
+
omq pull --bind tcp://:5557 -r./handler.rb
|
|
235
|
+
|
|
236
|
+
# combine script handlers with inline eval
|
|
237
|
+
omq req -c tcp://localhost:5555 -r./handler.rb -E '$F.map(&:upcase)'
|
|
238
|
+
|
|
239
|
+
# OMQ.outgoing { |msg| ... } — registered outgoing transform
|
|
240
|
+
# OMQ.incoming { |msg| ... } — registered incoming transform
|
|
241
|
+
# CLI flags (-e/-E) override registered handlers
|
|
214
242
|
TEXT
|
|
215
243
|
|
|
216
244
|
module_function
|
|
@@ -299,6 +327,8 @@ module OMQ
|
|
|
299
327
|
endpoints: [],
|
|
300
328
|
connects: [],
|
|
301
329
|
binds: [],
|
|
330
|
+
in_endpoints: [],
|
|
331
|
+
out_endpoints: [],
|
|
302
332
|
data: nil,
|
|
303
333
|
file: nil,
|
|
304
334
|
format: :ascii,
|
|
@@ -316,7 +346,8 @@ module OMQ
|
|
|
316
346
|
heartbeat_ivl: nil,
|
|
317
347
|
conflate: false,
|
|
318
348
|
compress: false,
|
|
319
|
-
|
|
349
|
+
send_expr: nil,
|
|
350
|
+
recv_expr: nil,
|
|
320
351
|
parallel: nil,
|
|
321
352
|
transient: false,
|
|
322
353
|
verbose: false,
|
|
@@ -326,6 +357,8 @@ module OMQ
|
|
|
326
357
|
curve_server_key: nil,
|
|
327
358
|
}
|
|
328
359
|
|
|
360
|
+
pipe_side = nil # nil = legacy positional mode; :in/:out = modal
|
|
361
|
+
|
|
329
362
|
parser = OptionParser.new do |o|
|
|
330
363
|
o.banner = "Usage: omq TYPE [options]\n\n" \
|
|
331
364
|
"Types: req, rep, pub, sub, push, pull, pair, dealer, router\n" \
|
|
@@ -333,8 +366,24 @@ module OMQ
|
|
|
333
366
|
"Virtual: pipe (PULL → eval → PUSH)\n\n"
|
|
334
367
|
|
|
335
368
|
o.separator "Connection:"
|
|
336
|
-
o.on("-c", "--connect URL", "Connect to endpoint (repeatable)")
|
|
337
|
-
|
|
369
|
+
o.on("-c", "--connect URL", "Connect to endpoint (repeatable)") { |v|
|
|
370
|
+
ep = Endpoint.new(v, false)
|
|
371
|
+
case pipe_side
|
|
372
|
+
when :in then opts[:in_endpoints] << ep
|
|
373
|
+
when :out then opts[:out_endpoints] << ep
|
|
374
|
+
else opts[:endpoints] << ep; opts[:connects] << v
|
|
375
|
+
end
|
|
376
|
+
}
|
|
377
|
+
o.on("-b", "--bind URL", "Bind to endpoint (repeatable)") { |v|
|
|
378
|
+
ep = Endpoint.new(v, true)
|
|
379
|
+
case pipe_side
|
|
380
|
+
when :in then opts[:in_endpoints] << ep
|
|
381
|
+
when :out then opts[:out_endpoints] << ep
|
|
382
|
+
else opts[:endpoints] << ep; opts[:binds] << v
|
|
383
|
+
end
|
|
384
|
+
}
|
|
385
|
+
o.on("--in", "Pipe: subsequent -b/-c attach to input (PULL) side") { pipe_side = :in }
|
|
386
|
+
o.on("--out", "Pipe: subsequent -b/-c attach to output (PUSH) side") { pipe_side = :out }
|
|
338
387
|
|
|
339
388
|
o.separator "\nData source (REP: reply source):"
|
|
340
389
|
o.on( "--echo", "Echo received messages back (REP)") { opts[:echo] = true }
|
|
@@ -380,9 +429,11 @@ module OMQ
|
|
|
380
429
|
o.separator "\nCompression:"
|
|
381
430
|
o.on("-z", "--compress", "Zstandard compression per frame") { opts[:compress] = true }
|
|
382
431
|
|
|
383
|
-
o.separator "\nProcessing:"
|
|
384
|
-
o.on("-e", "--eval EXPR",
|
|
385
|
-
o.on("-
|
|
432
|
+
o.separator "\nProcessing (-e = incoming, -E = outgoing):"
|
|
433
|
+
o.on("-e", "--recv-eval EXPR", "Eval Ruby for each incoming message ($F = parts)") { |v| opts[:recv_expr] = v }
|
|
434
|
+
o.on("-E", "--send-eval EXPR", "Eval Ruby for each outgoing message ($F = parts)") { |v| opts[:send_expr] = v }
|
|
435
|
+
o.on("-r", "--require LIB", "Require lib/file; scripts can register OMQ.outgoing/incoming") { |v|
|
|
436
|
+
require_relative "../omq" unless defined?(OMQ::VERSION)
|
|
386
437
|
v.start_with?("./", "../") ? require(File.expand_path(v)) : require(v)
|
|
387
438
|
}
|
|
388
439
|
o.on("-P", "--parallel [N]", Integer, "Parallel Ractor workers (pipe only, default: nproc)") { |v|
|
|
@@ -421,10 +472,13 @@ module OMQ
|
|
|
421
472
|
|
|
422
473
|
opts[:type_name] = type_name.downcase
|
|
423
474
|
|
|
424
|
-
normalize
|
|
475
|
+
normalize = ->(url) { url.sub(%r{\Atcp://\*:}, "tcp://0.0.0.0:").sub(%r{\Atcp://:}, "tcp://localhost:") }
|
|
476
|
+
normalize_ep = ->(ep) { Endpoint.new(normalize.call(ep.url), ep.bind?) }
|
|
425
477
|
opts[:binds].map!(&normalize)
|
|
426
478
|
opts[:connects].map!(&normalize)
|
|
427
|
-
opts[:endpoints].map!
|
|
479
|
+
opts[:endpoints].map!(&normalize_ep)
|
|
480
|
+
opts[:in_endpoints].map!(&normalize_ep)
|
|
481
|
+
opts[:out_endpoints].map!(&normalize_ep)
|
|
428
482
|
|
|
429
483
|
opts
|
|
430
484
|
end
|
|
@@ -436,8 +490,16 @@ module OMQ
|
|
|
436
490
|
type_name = opts[:type_name]
|
|
437
491
|
|
|
438
492
|
if type_name == "pipe"
|
|
439
|
-
|
|
493
|
+
has_in_out = opts[:in_endpoints].any? || opts[:out_endpoints].any?
|
|
494
|
+
if has_in_out
|
|
495
|
+
abort "pipe --in requires at least one endpoint" if opts[:in_endpoints].empty?
|
|
496
|
+
abort "pipe --out requires at least one endpoint" if opts[:out_endpoints].empty?
|
|
497
|
+
abort "pipe: don't mix --in/--out with bare -b/-c endpoints" unless opts[:endpoints].empty?
|
|
498
|
+
else
|
|
499
|
+
abort "pipe requires exactly 2 endpoints (pull-side and push-side), or use --in/--out" if opts[:endpoints].size != 2
|
|
500
|
+
end
|
|
440
501
|
else
|
|
502
|
+
abort "--in/--out are only valid for pipe" if opts[:in_endpoints].any? || opts[:out_endpoints].any?
|
|
441
503
|
abort "At least one --connect or --bind is required" if opts[:connects].empty? && opts[:binds].empty?
|
|
442
504
|
end
|
|
443
505
|
abort "--data and --file are mutually exclusive" if opts[:data] && opts[:file]
|
|
@@ -447,11 +509,15 @@ module OMQ
|
|
|
447
509
|
abort "--identity is only valid for DEALER/ROUTER" if opts[:identity] && !%w[dealer router].include?(type_name)
|
|
448
510
|
abort "--target is only valid for ROUTER/SERVER/PEER" if opts[:target] && !%w[router server peer].include?(type_name)
|
|
449
511
|
abort "--conflate is only valid for PUB/RADIO" if opts[:conflate] && !%w[pub radio].include?(type_name)
|
|
512
|
+
abort "--recv-eval is not valid for send-only sockets (use --send-eval / -E)" if opts[:recv_expr] && SEND_ONLY.include?(type_name)
|
|
513
|
+
abort "--send-eval is not valid for recv-only sockets (use --recv-eval / -e)" if opts[:send_expr] && RECV_ONLY.include?(type_name)
|
|
514
|
+
abort "--send-eval and --target are mutually exclusive" if opts[:send_expr] && opts[:target]
|
|
450
515
|
|
|
451
516
|
if opts[:parallel]
|
|
452
517
|
abort "-P/--parallel is only valid for pipe" unless type_name == "pipe"
|
|
453
518
|
abort "-P/--parallel must be >= 2" if opts[:parallel] < 2
|
|
454
|
-
|
|
519
|
+
all_pipe_eps = opts[:in_endpoints] + opts[:out_endpoints] + opts[:endpoints]
|
|
520
|
+
abort "-P/--parallel requires all endpoints to use --connect (not --bind)" if all_pipe_eps.any?(&:bind?)
|
|
455
521
|
end
|
|
456
522
|
|
|
457
523
|
(opts[:connects] + opts[:binds]).each do |url|
|
data/lib/omq/version.rb
CHANGED
data/lib/omq/zmtp/engine.rb
CHANGED
|
@@ -215,29 +215,50 @@ module OMQ
|
|
|
215
215
|
# fast path when the connection is a DirectPipe.
|
|
216
216
|
#
|
|
217
217
|
# @param conn [Connection, Transport::Inproc::DirectPipe]
|
|
218
|
+
# Starts a recv pump that dequeues messages from a connection
|
|
219
|
+
# and enqueues them into the routing strategy's recv queue.
|
|
220
|
+
#
|
|
221
|
+
# When a block is given, each message is yielded for transformation
|
|
222
|
+
# before enqueueing. The block is compiled at the call site, giving
|
|
223
|
+
# YJIT a monomorphic call per routing strategy instead of a shared
|
|
224
|
+
# megamorphic `transform.call` dispatch.
|
|
225
|
+
#
|
|
226
|
+
# @param conn [Connection, Transport::Inproc::DirectPipe]
|
|
218
227
|
# @param recv_queue [Async::LimitedQueue] routing strategy's recv queue
|
|
219
|
-
# @
|
|
228
|
+
# @yield [msg] optional per-message transform
|
|
220
229
|
# @return [#stop, nil] pump task handle, or nil for DirectPipe bypass
|
|
221
230
|
#
|
|
222
|
-
def start_recv_pump(conn, recv_queue, transform
|
|
231
|
+
def start_recv_pump(conn, recv_queue, &transform)
|
|
223
232
|
if conn.is_a?(Transport::Inproc::DirectPipe) && conn.peer
|
|
224
233
|
conn.peer.direct_recv_queue = recv_queue
|
|
225
234
|
conn.peer.direct_recv_transform = transform
|
|
226
235
|
return nil
|
|
227
236
|
end
|
|
228
237
|
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
238
|
+
if transform
|
|
239
|
+
Reactor.spawn_pump(annotation: "recv pump") do
|
|
240
|
+
loop do
|
|
241
|
+
msg = conn.receive_message
|
|
242
|
+
msg = transform.call(msg).freeze
|
|
243
|
+
recv_queue.enqueue(msg)
|
|
244
|
+
end
|
|
245
|
+
rescue Async::Stop
|
|
246
|
+
rescue ProtocolError, *CONNECTION_LOST
|
|
247
|
+
connection_lost(conn)
|
|
248
|
+
rescue => error
|
|
249
|
+
signal_fatal_error(error)
|
|
250
|
+
end
|
|
251
|
+
else
|
|
252
|
+
Reactor.spawn_pump(annotation: "recv pump") do
|
|
253
|
+
loop do
|
|
254
|
+
recv_queue.enqueue(conn.receive_message)
|
|
255
|
+
end
|
|
256
|
+
rescue Async::Stop
|
|
257
|
+
rescue ProtocolError, *CONNECTION_LOST
|
|
258
|
+
connection_lost(conn)
|
|
259
|
+
rescue => error
|
|
260
|
+
signal_fatal_error(error)
|
|
234
261
|
end
|
|
235
|
-
rescue Async::Stop
|
|
236
|
-
# normal shutdown
|
|
237
|
-
rescue ProtocolError, *CONNECTION_LOST
|
|
238
|
-
connection_lost(conn)
|
|
239
|
-
rescue => error
|
|
240
|
-
signal_fatal_error(error)
|
|
241
262
|
end
|
|
242
263
|
end
|
|
243
264
|
|
|
@@ -34,8 +34,9 @@ module OMQ
|
|
|
34
34
|
routing_id = SecureRandom.bytes(4)
|
|
35
35
|
@connections_by_routing_id[routing_id] = connection
|
|
36
36
|
|
|
37
|
-
task = @engine.start_recv_pump(connection, @recv_queue
|
|
38
|
-
|
|
37
|
+
task = @engine.start_recv_pump(connection, @recv_queue) do |msg|
|
|
38
|
+
[routing_id, *msg]
|
|
39
|
+
end
|
|
39
40
|
@tasks << task if task
|
|
40
41
|
|
|
41
42
|
start_send_pump unless @send_pump_started
|
data/lib/omq/zmtp/routing/rep.rb
CHANGED
|
@@ -29,14 +29,13 @@ module OMQ
|
|
|
29
29
|
# @param connection [Connection]
|
|
30
30
|
#
|
|
31
31
|
def connection_added(connection)
|
|
32
|
-
|
|
32
|
+
task = @engine.start_recv_pump(connection, @recv_queue) do |msg|
|
|
33
33
|
delimiter = msg.index(&:empty?) || msg.size
|
|
34
34
|
envelope = msg[0, delimiter]
|
|
35
35
|
body = msg[(delimiter + 1)..] || []
|
|
36
36
|
@pending_replies << { conn: connection, envelope: envelope }
|
|
37
37
|
body
|
|
38
|
-
|
|
39
|
-
task = @engine.start_recv_pump(connection, @recv_queue, transform: transform)
|
|
38
|
+
end
|
|
40
39
|
@tasks << task if task
|
|
41
40
|
start_send_pump unless @send_pump_started
|
|
42
41
|
end
|
data/lib/omq/zmtp/routing/req.rb
CHANGED
|
@@ -28,8 +28,9 @@ module OMQ
|
|
|
28
28
|
def connection_added(connection)
|
|
29
29
|
@connections << connection
|
|
30
30
|
signal_connection_available
|
|
31
|
-
task = @engine.start_recv_pump(connection, @recv_queue
|
|
32
|
-
|
|
31
|
+
task = @engine.start_recv_pump(connection, @recv_queue) do |msg|
|
|
32
|
+
msg.first&.empty? ? msg[1..] : msg
|
|
33
|
+
end
|
|
33
34
|
@tasks << task if task
|
|
34
35
|
start_send_pump unless @send_pump_started
|
|
35
36
|
end
|
|
@@ -58,11 +59,6 @@ module OMQ
|
|
|
58
59
|
#
|
|
59
60
|
def transform_send(parts) = ["".b, *parts]
|
|
60
61
|
|
|
61
|
-
# REQ strips the leading empty delimiter frame on receive.
|
|
62
|
-
#
|
|
63
|
-
def transform_recv(msg)
|
|
64
|
-
msg.first&.empty? ? msg[1..] : msg
|
|
65
|
-
end
|
|
66
62
|
end
|
|
67
63
|
end
|
|
68
64
|
end
|
|
@@ -35,8 +35,9 @@ module OMQ
|
|
|
35
35
|
identity = SecureRandom.bytes(5) if identity.nil? || identity.empty?
|
|
36
36
|
@connections_by_identity[identity] = connection
|
|
37
37
|
|
|
38
|
-
task = @engine.start_recv_pump(connection, @recv_queue
|
|
39
|
-
|
|
38
|
+
task = @engine.start_recv_pump(connection, @recv_queue) do |msg|
|
|
39
|
+
[identity, *msg]
|
|
40
|
+
end
|
|
40
41
|
@tasks << task if task
|
|
41
42
|
|
|
42
43
|
start_send_pump unless @send_pump_started
|
|
@@ -34,8 +34,9 @@ module OMQ
|
|
|
34
34
|
routing_id = SecureRandom.bytes(4)
|
|
35
35
|
@connections_by_routing_id[routing_id] = connection
|
|
36
36
|
|
|
37
|
-
task = @engine.start_recv_pump(connection, @recv_queue
|
|
38
|
-
|
|
37
|
+
task = @engine.start_recv_pump(connection, @recv_queue) do |msg|
|
|
38
|
+
[routing_id, *msg]
|
|
39
|
+
end
|
|
39
40
|
@tasks << task if task
|
|
40
41
|
|
|
41
42
|
start_send_pump unless @send_pump_started
|
data/lib/omq.rb
CHANGED