omq-cli 0.8.1 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +66 -0
- data/README.md +2 -1
- data/lib/omq/cli/base_runner.rb +10 -2
- data/lib/omq/cli/cli_parser.rb +25 -17
- data/lib/omq/cli/config.rb +1 -0
- data/lib/omq/cli/formatter.rb +21 -10
- data/lib/omq/cli/parallel_worker.rb +5 -2
- data/lib/omq/cli/pipe.rb +14 -5
- data/lib/omq/cli/pipe_worker.rb +12 -8
- data/lib/omq/cli/socket_setup.rb +26 -4
- data/lib/omq/cli/version.rb +1 -1
- data/lib/omq/cli.rb +3 -0
- metadata +8 -8
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: a4acc7ab47167da27247f7b45b82729da6302868879f1a003727ff2b359d42d5
|
|
4
|
+
data.tar.gz: 4a667cab2f594ade3cdc7b4234dc97562bbee043707992935d566dd563530795
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 41f4141def92292e2a35412af5ddd9d1a567af2b150b59e805452cde24f9a17279e162d47f44bcb84c9d1e7fb3ad8dad2971162ffad2f38814f753c17db95ab7
|
|
7
|
+
data.tar.gz: 340c7f417266cbeb7020cd19a3a0161de78310b78cf554443c71ba22cef05c69611e38f9d4ef9370ae37b2c27bed2b59241c15ff88e7e077663ae0d0503d6f9b
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,71 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.9.0 — 2026-04-08
|
|
4
|
+
|
|
5
|
+
### Changed
|
|
6
|
+
|
|
7
|
+
- **`--recv-maxsz` defaults to 1 MiB in the CLI** — the underlying `omq`
|
|
8
|
+
library no longer imposes a default (it's `nil`/unlimited as of this
|
|
9
|
+
release), but the CLI keeps a conservative 1 MiB cap for safety when
|
|
10
|
+
connecting to untrusted peers from a terminal. Pass `--recv-maxsz 0`
|
|
11
|
+
to disable the cap explicitly, or `--recv-maxsz N` to raise it.
|
|
12
|
+
- **Default HWM lowered to 100** (from libzmq's 1000) for both send and
|
|
13
|
+
recv. The CLI is typically used interactively or for short pipelines
|
|
14
|
+
where a smaller in-flight queue keeps memory bounded and surfaces
|
|
15
|
+
backpressure earlier. Users who want the old behavior can pass
|
|
16
|
+
`--send-hwm 1000 --recv-hwm 1000` (or `0` for unbounded). Pipe worker
|
|
17
|
+
sockets are unaffected — they still override to `PIPE_HWM` internally.
|
|
18
|
+
- **Compression codec: Zstandard → LZ4 frame format (BREAKING on the wire).**
|
|
19
|
+
`--compress` now uses the new [`rlz4`](../rlz4) gem (Rust extension over
|
|
20
|
+
`lz4_flex`) instead of `zstd-ruby`. Motivation: `zstd-ruby` is the only
|
|
21
|
+
existing Ractor-safe compressor gem, but LZ4 is a better fit for the
|
|
22
|
+
per-message-part workload (smaller frames, lower CPU). `rlz4` is
|
|
23
|
+
Ractor-safe by construction, so parallel `-P` workers now use the same
|
|
24
|
+
codec as the sequential path. **Wire format is incompatible** with prior
|
|
25
|
+
omq-cli versions when `--compress` is in use — both ends must upgrade.
|
|
26
|
+
|
|
27
|
+
### Added
|
|
28
|
+
|
|
29
|
+
- **`--ffi` flag** — opt-in libzmq backend for any socket runner. Builds
|
|
30
|
+
sockets with `backend: :ffi`, so the CLI can drive native libzmq instead
|
|
31
|
+
of the pure-Ruby engine. Requires the optional `omq-ffi` gem and a
|
|
32
|
+
system libzmq 4.x; missing dependencies abort with a clear error.
|
|
33
|
+
Propagated through all socket construction sites: `BaseRunner`,
|
|
34
|
+
`PipeRunner`, `PipeWorker`, and `ParallelWorker`.
|
|
35
|
+
|
|
36
|
+
### Fixed
|
|
37
|
+
|
|
38
|
+
- **`--send-eval` / `-E` on REP** — now rejected at validation time. REP
|
|
39
|
+
derives its reply from `--recv-eval` / `-e`, so `-E` was silently
|
|
40
|
+
ignored and the runner fell through to reading stdin, hanging the
|
|
41
|
+
request-reply cycle.
|
|
42
|
+
- **`-vvv` preview of REP/REQ envelopes** — empty delimiter frames now
|
|
43
|
+
render as `[0B]` instead of an empty string, so a REP reply with wire
|
|
44
|
+
parts `["", "1"]` previews as `(1B) [0B]|1` instead of the misleading
|
|
45
|
+
`(1B) |1` with a dangling leading pipe.
|
|
46
|
+
|
|
47
|
+
## 0.8.2 — 2026-04-08
|
|
48
|
+
|
|
49
|
+
### Fixed
|
|
50
|
+
|
|
51
|
+
- **`DecompressError` handling** — decompression failures now raise a proper
|
|
52
|
+
`DecompressError` instead of calling `abort`. In parallel mode (`-P`),
|
|
53
|
+
errors are sent through a dedicated `Ractor::Port` with a consumer thread
|
|
54
|
+
that aborts cleanly (exit code 1, one error message). Previously, `Async do`
|
|
55
|
+
swallowed the exception and the process exited silently.
|
|
56
|
+
- **Pipe memory growth** — pipe sockets now pass `recv_hwm` / `send_hwm` via
|
|
57
|
+
constructor kwargs so the engine captures them before internal queue sizing.
|
|
58
|
+
Previously, setters were called after the engine was initialized and had no
|
|
59
|
+
effect on staging queue capacity.
|
|
60
|
+
|
|
61
|
+
### Changed
|
|
62
|
+
|
|
63
|
+
- **`-P N` mandatory argument** — `-P` now requires an explicit worker count.
|
|
64
|
+
Previously, `-Pq` silently consumed `-q` as the argument to `-P`, making
|
|
65
|
+
`-q` (quiet) ineffective.
|
|
66
|
+
- **Pipe default HWM lowered to 16** — reduced from 64 to further bound memory
|
|
67
|
+
with large messages in pipeline stages.
|
|
68
|
+
|
|
3
69
|
## 0.8.1 — 2026-04-08
|
|
4
70
|
|
|
5
71
|
### Fixed
|
data/README.md
CHANGED
|
@@ -336,7 +336,8 @@ omq pull -b tcp://:5557 -t 5
|
|
|
336
336
|
|
|
337
337
|
## Compression
|
|
338
338
|
|
|
339
|
-
Both sides must use `--compress` (`-z`).
|
|
339
|
+
Both sides must use `--compress` (`-z`). Uses LZ4 frame format, provided
|
|
340
|
+
by the `rlz4` gem (Ractor-safe, Rust extension via `lz4_flex`).
|
|
340
341
|
|
|
341
342
|
```sh
|
|
342
343
|
omq push -c tcp://remote:5557 -z < data.txt
|
data/lib/omq/cli/base_runner.rb
CHANGED
|
@@ -55,10 +55,17 @@ module OMQ
|
|
|
55
55
|
eps = RactorHelpers.preresolve_tcp(config.endpoints)
|
|
56
56
|
output_port, output_thread = RactorHelpers.start_output_consumer
|
|
57
57
|
log_port, log_thread = RactorHelpers.start_log_consumer
|
|
58
|
+
error_port = Ractor::Port.new
|
|
59
|
+
error_thread = Thread.new(error_port) do |p|
|
|
60
|
+
msg = p.receive
|
|
61
|
+
abort "omq: #{msg}" unless msg.equal?(RactorHelpers::SHUTDOWN)
|
|
62
|
+
rescue Ractor::ClosedError
|
|
63
|
+
# port closed, no error
|
|
64
|
+
end
|
|
58
65
|
|
|
59
66
|
workers = config.parallel.times.map do
|
|
60
|
-
::Ractor.new(config, socket_sym, eps, output_port, log_port) do |cfg, sym, e, oport, lport|
|
|
61
|
-
ParallelWorker.new(cfg, sym, e, oport, lport).call
|
|
67
|
+
::Ractor.new(config, socket_sym, eps, output_port, log_port, error_port) do |cfg, sym, e, oport, lport, eport|
|
|
68
|
+
ParallelWorker.new(cfg, sym, e, oport, lport, eport).call
|
|
62
69
|
end
|
|
63
70
|
end
|
|
64
71
|
|
|
@@ -68,6 +75,7 @@ module OMQ
|
|
|
68
75
|
$stderr.write("omq: Ractor error: #{e.cause&.message || e.message}\n")
|
|
69
76
|
end
|
|
70
77
|
ensure
|
|
78
|
+
RactorHelpers.stop_consumer(error_port, error_thread) if error_port
|
|
71
79
|
RactorHelpers.stop_consumer(output_port, output_thread) if output_port
|
|
72
80
|
RactorHelpers.stop_consumer(log_port, log_thread) if log_port
|
|
73
81
|
end
|
data/lib/omq/cli/cli_parser.rb
CHANGED
|
@@ -233,6 +233,7 @@ module OMQ
|
|
|
233
233
|
curve_server: false,
|
|
234
234
|
curve_server_key: nil,
|
|
235
235
|
crypto: nil,
|
|
236
|
+
ffi: false,
|
|
236
237
|
}.freeze
|
|
237
238
|
|
|
238
239
|
|
|
@@ -338,9 +339,9 @@ module OMQ
|
|
|
338
339
|
end
|
|
339
340
|
}
|
|
340
341
|
o.on("--heartbeat-ivl SECS", Float, "ZMTP heartbeat interval (detects dead peers)") { |v| opts[:heartbeat_ivl] = v }
|
|
341
|
-
o.on("--recv-maxsz
|
|
342
|
-
o.on("--send-hwm N", Integer, "Send high water mark (default
|
|
343
|
-
o.on("--recv-hwm N", Integer, "Recv high water mark (default
|
|
342
|
+
o.on("--recv-maxsz SIZE", "Max inbound message size, e.g. 4096, 64K, 1M, 2G (default 1M, 0=unlimited; larger messages drop the connection)") { |v| opts[:recv_maxsz] = parse_byte_size(v) }
|
|
343
|
+
o.on("--send-hwm N", Integer, "Send high water mark (default 100, 0=unbounded)") { |v| opts[:send_hwm] = v }
|
|
344
|
+
o.on("--recv-hwm N", Integer, "Recv high water mark (default 100, 0=unbounded)") { |v| opts[:recv_hwm] = v }
|
|
344
345
|
o.on("--sndbuf N", "SO_SNDBUF kernel buffer size (e.g. 4K, 1M)") { |v| opts[:sndbuf] = parse_byte_size(v) }
|
|
345
346
|
o.on("--rcvbuf N", "SO_RCVBUF kernel buffer size (e.g. 4K, 1M)") { |v| opts[:rcvbuf] = parse_byte_size(v) }
|
|
346
347
|
|
|
@@ -348,8 +349,8 @@ module OMQ
|
|
|
348
349
|
o.on("--conflate", "Keep only last message per subscriber (PUB/RADIO)") { opts[:conflate] = true }
|
|
349
350
|
|
|
350
351
|
o.separator "\nCompression:"
|
|
351
|
-
o.on("-z", "--compress", "
|
|
352
|
-
require "
|
|
352
|
+
o.on("-z", "--compress", "LZ4 compression per frame (modal with --in/--out)") do
|
|
353
|
+
require "rlz4"
|
|
353
354
|
case pipe_side
|
|
354
355
|
when :in
|
|
355
356
|
opts[:compress_in] = true
|
|
@@ -367,9 +368,8 @@ module OMQ
|
|
|
367
368
|
require "omq" unless defined?(OMQ::VERSION)
|
|
368
369
|
opts[:scripts] << (v == "-" ? :stdin : (v.start_with?("./", "../") ? File.expand_path(v) : v))
|
|
369
370
|
}
|
|
370
|
-
o.on("-P", "--parallel
|
|
371
|
-
|
|
372
|
-
opts[:parallel] = [v || Etc.nprocessors, 16].min
|
|
371
|
+
o.on("-P", "--parallel N", Integer, "Parallel Ractor workers (max 16)") { |v|
|
|
372
|
+
opts[:parallel] = [v, 16].min
|
|
373
373
|
}
|
|
374
374
|
|
|
375
375
|
o.separator "\nCURVE encryption (requires system libsodium):"
|
|
@@ -384,6 +384,14 @@ module OMQ
|
|
|
384
384
|
o.on("-v", "--verbose", "Verbosity: -v endpoints, -vv events, -vvv messages") { opts[:verbose] += 1 }
|
|
385
385
|
o.on("-q", "--quiet", "Suppress message output") { opts[:quiet] = true }
|
|
386
386
|
o.on( "--transient", "Exit when all peers disconnect") { opts[:transient] = true }
|
|
387
|
+
o.on( "--ffi", "Use libzmq FFI backend (requires omq-ffi gem + system libzmq 4.x)") do
|
|
388
|
+
begin
|
|
389
|
+
require "omq/ffi"
|
|
390
|
+
rescue LoadError => e
|
|
391
|
+
abort "omq: --ffi requires the omq-ffi gem and system libzmq 4.x (#{e.message})"
|
|
392
|
+
end
|
|
393
|
+
opts[:ffi] = true
|
|
394
|
+
end
|
|
387
395
|
o.on("-V", "--version") {
|
|
388
396
|
if ENV["OMQ_DEV"]
|
|
389
397
|
require_relative "../../../../omq/lib/omq/version"
|
|
@@ -431,21 +439,20 @@ module OMQ
|
|
|
431
439
|
end
|
|
432
440
|
|
|
433
441
|
|
|
434
|
-
# Parses a byte size string with optional K/M suffix
|
|
442
|
+
# Parses a byte size string with an optional K/M/G suffix (binary,
|
|
443
|
+
# i.e. 1K = 1024 bytes).
|
|
435
444
|
#
|
|
436
|
-
# @param str [String] e.g. "4096", "4K", "1M"
|
|
445
|
+
# @param str [String] e.g. "4096", "4K", "1M", "2G"
|
|
437
446
|
# @return [Integer] size in bytes
|
|
438
447
|
#
|
|
439
448
|
def parse_byte_size(str)
|
|
440
449
|
case str
|
|
441
|
-
when /\A(\d+)[kK]\z/
|
|
442
|
-
|
|
443
|
-
when /\A(\d+)[
|
|
444
|
-
|
|
445
|
-
when /\A\d+\z/
|
|
446
|
-
str.to_i
|
|
450
|
+
when /\A(\d+)[kK]\z/ then $1.to_i * 1024
|
|
451
|
+
when /\A(\d+)[mM]\z/ then $1.to_i * 1024 * 1024
|
|
452
|
+
when /\A(\d+)[gG]\z/ then $1.to_i * 1024 * 1024 * 1024
|
|
453
|
+
when /\A\d+\z/ then str.to_i
|
|
447
454
|
else
|
|
448
|
-
abort "invalid byte size: #{str} (use e.g. 4096, 4K, 1M)"
|
|
455
|
+
abort "invalid byte size: #{str} (use e.g. 4096, 4K, 1M, 2G)"
|
|
449
456
|
end
|
|
450
457
|
end
|
|
451
458
|
|
|
@@ -483,6 +490,7 @@ module OMQ
|
|
|
483
490
|
abort "--conflate is only valid for PUB/RADIO" if opts[:conflate] && !%w[pub radio].include?(type_name)
|
|
484
491
|
abort "--recv-eval is not valid for send-only sockets (use --send-eval / -E)" if opts[:recv_expr] && SEND_ONLY.include?(type_name)
|
|
485
492
|
abort "--send-eval is not valid for recv-only sockets (use --recv-eval / -e)" if opts[:send_expr] && RECV_ONLY.include?(type_name)
|
|
493
|
+
abort "--send-eval is not valid for REP (the reply is the result of --recv-eval / -e)" if opts[:send_expr] && type_name == "rep"
|
|
486
494
|
abort "--send-eval and --target are mutually exclusive" if opts[:send_expr] && opts[:target]
|
|
487
495
|
|
|
488
496
|
if opts[:parallel]
|
data/lib/omq/cli/config.rb
CHANGED
data/lib/omq/cli/formatter.rb
CHANGED
|
@@ -2,11 +2,14 @@
|
|
|
2
2
|
|
|
3
3
|
module OMQ
|
|
4
4
|
module CLI
|
|
5
|
+
# Raised when LZ4 decompression fails.
|
|
6
|
+
class DecompressError < RuntimeError; end
|
|
7
|
+
|
|
5
8
|
# Handles encoding/decoding messages in the configured format,
|
|
6
|
-
# plus optional
|
|
9
|
+
# plus optional LZ4 compression.
|
|
7
10
|
class Formatter
|
|
8
11
|
# @param format [Symbol] wire format (:ascii, :quoted, :raw, :jsonl, :msgpack, :marshal)
|
|
9
|
-
# @param compress [Boolean] whether to apply
|
|
12
|
+
# @param compress [Boolean] whether to apply LZ4 compression per frame
|
|
10
13
|
def initialize(format, compress: false)
|
|
11
14
|
@format = format
|
|
12
15
|
@compress = compress
|
|
@@ -80,23 +83,23 @@ module OMQ
|
|
|
80
83
|
end
|
|
81
84
|
|
|
82
85
|
|
|
83
|
-
# Compresses each frame with
|
|
86
|
+
# Compresses each frame with LZ4 if compression is enabled.
|
|
84
87
|
#
|
|
85
88
|
# @param parts [Array<String>] message frames
|
|
86
89
|
# @return [Array<String>] optionally compressed frames
|
|
87
90
|
def compress(parts)
|
|
88
|
-
@compress ? parts.map { |p|
|
|
91
|
+
@compress ? parts.map { |p| RLZ4.compress(p) } : parts
|
|
89
92
|
end
|
|
90
93
|
|
|
91
94
|
|
|
92
|
-
# Decompresses each frame with
|
|
95
|
+
# Decompresses each frame with LZ4 if compression is enabled.
|
|
93
96
|
#
|
|
94
97
|
# @param parts [Array<String>] possibly compressed message frames
|
|
95
98
|
# @return [Array<String>] decompressed frames
|
|
96
99
|
def decompress(parts)
|
|
97
|
-
@compress ? parts.map { |p|
|
|
98
|
-
rescue
|
|
99
|
-
|
|
100
|
+
@compress ? parts.map { |p| RLZ4.decompress(p) } : parts
|
|
101
|
+
rescue RLZ4::DecompressError
|
|
102
|
+
raise DecompressError, "decompression failed (did the sender use --compress?)"
|
|
100
103
|
end
|
|
101
104
|
|
|
102
105
|
|
|
@@ -107,15 +110,23 @@ module OMQ
|
|
|
107
110
|
def self.preview(parts)
|
|
108
111
|
total = parts.sum(&:bytesize)
|
|
109
112
|
shown = parts.first(3).map { |p| preview_frame(p) }
|
|
110
|
-
tail
|
|
113
|
+
tail = parts.size > 3 ? "|...(#{parts.size} parts)" : ""
|
|
114
|
+
|
|
111
115
|
"(#{total}B) #{shown.join("|")}#{tail}"
|
|
112
116
|
end
|
|
113
117
|
|
|
114
118
|
|
|
115
119
|
def self.preview_frame(part)
|
|
116
120
|
bytes = part.b
|
|
117
|
-
|
|
121
|
+
# Empty frames must render as a visible marker, not as the empty
|
|
122
|
+
# string — otherwise joining with "|" would produce misleading
|
|
123
|
+
# output like "|body" for REP/REQ-style envelopes where the first
|
|
124
|
+
# wire frame is an empty delimiter.
|
|
125
|
+
return "[0B]" if bytes.empty?
|
|
126
|
+
|
|
127
|
+
sample = bytes[0, 12]
|
|
118
128
|
printable = sample.count("\x20-\x7e")
|
|
129
|
+
|
|
119
130
|
if printable < sample.bytesize / 2
|
|
120
131
|
"[#{bytes.bytesize}B]"
|
|
121
132
|
elsif bytes.bytesize > 12
|
|
@@ -10,12 +10,13 @@ module OMQ
|
|
|
10
10
|
# - rep (recv-reply with echo/data/eval)
|
|
11
11
|
#
|
|
12
12
|
class ParallelWorker
|
|
13
|
-
def initialize(config, socket_sym, endpoints, output_port, log_port)
|
|
13
|
+
def initialize(config, socket_sym, endpoints, output_port, log_port, error_port)
|
|
14
14
|
@config = config
|
|
15
15
|
@socket_sym = socket_sym
|
|
16
16
|
@endpoints = endpoints
|
|
17
17
|
@output_port = output_port
|
|
18
18
|
@log_port = log_port
|
|
19
|
+
@error_port = error_port
|
|
19
20
|
end
|
|
20
21
|
|
|
21
22
|
|
|
@@ -28,6 +29,8 @@ module OMQ
|
|
|
28
29
|
compile_expr
|
|
29
30
|
run_loop
|
|
30
31
|
run_end_block
|
|
32
|
+
rescue DecompressError => e
|
|
33
|
+
@error_port.send(e.message)
|
|
31
34
|
ensure
|
|
32
35
|
@sock&.close
|
|
33
36
|
end
|
|
@@ -38,7 +41,7 @@ module OMQ
|
|
|
38
41
|
|
|
39
42
|
|
|
40
43
|
def setup_socket
|
|
41
|
-
@sock = OMQ.const_get(@socket_sym).new
|
|
44
|
+
@sock = @config.ffi ? OMQ.const_get(@socket_sym).new(backend: :ffi) : OMQ.const_get(@socket_sym).new
|
|
42
45
|
OMQ::CLI::SocketSetup.apply_options(@sock, @config)
|
|
43
46
|
@sock.identity = @config.identity if @config.identity
|
|
44
47
|
OMQ::CLI::SocketSetup.attach_endpoints(@sock, @endpoints, verbose: false)
|
data/lib/omq/cli/pipe.rb
CHANGED
|
@@ -8,7 +8,7 @@ module OMQ
|
|
|
8
8
|
# Default HWM for pipe sockets when the user hasn't set one.
|
|
9
9
|
# Much lower than the socket default (1000) to bound memory
|
|
10
10
|
# with large messages in pipeline stages.
|
|
11
|
-
PIPE_HWM =
|
|
11
|
+
PIPE_HWM = 16
|
|
12
12
|
|
|
13
13
|
# @return [Config] frozen CLI configuration
|
|
14
14
|
attr_reader :config
|
|
@@ -80,8 +80,9 @@ module OMQ
|
|
|
80
80
|
|
|
81
81
|
|
|
82
82
|
def build_pull_push(in_eps, out_eps)
|
|
83
|
-
|
|
84
|
-
|
|
83
|
+
kwargs = config.ffi ? { backend: :ffi } : {}
|
|
84
|
+
pull = OMQ::PULL.new(**kwargs)
|
|
85
|
+
push = OMQ::PUSH.new(**kwargs)
|
|
85
86
|
SocketSetup.apply_options(pull, config)
|
|
86
87
|
SocketSetup.apply_options(push, config)
|
|
87
88
|
pull.recv_hwm = PIPE_HWM unless config.recv_hwm
|
|
@@ -129,9 +130,16 @@ module OMQ
|
|
|
129
130
|
in_eps = RactorHelpers.preresolve_tcp(in_eps)
|
|
130
131
|
out_eps = RactorHelpers.preresolve_tcp(out_eps)
|
|
131
132
|
log_port, log_thread = RactorHelpers.start_log_consumer
|
|
133
|
+
error_port = Ractor::Port.new
|
|
134
|
+
error_thread = Thread.new(error_port) do |p|
|
|
135
|
+
msg = p.receive
|
|
136
|
+
abort "omq: #{msg}" unless msg.equal?(RactorHelpers::SHUTDOWN)
|
|
137
|
+
rescue Ractor::ClosedError
|
|
138
|
+
# port closed, no error
|
|
139
|
+
end
|
|
132
140
|
workers = config.parallel.times.map do
|
|
133
|
-
::Ractor.new(config, in_eps, out_eps, log_port) do |cfg, ins, outs, lport|
|
|
134
|
-
PipeWorker.new(cfg, ins, outs, lport).call
|
|
141
|
+
::Ractor.new(config, in_eps, out_eps, log_port, error_port) do |cfg, ins, outs, lport, eport|
|
|
142
|
+
PipeWorker.new(cfg, ins, outs, lport, eport).call
|
|
135
143
|
end
|
|
136
144
|
end
|
|
137
145
|
workers.each do |w|
|
|
@@ -140,6 +148,7 @@ module OMQ
|
|
|
140
148
|
$stderr.write("omq: Ractor error: #{e.cause&.message || e.message}\n")
|
|
141
149
|
end
|
|
142
150
|
ensure
|
|
151
|
+
RactorHelpers.stop_consumer(error_port, error_thread) if error_port
|
|
143
152
|
RactorHelpers.stop_consumer(log_port, log_thread) if log_port
|
|
144
153
|
end
|
|
145
154
|
|
data/lib/omq/cli/pipe_worker.rb
CHANGED
|
@@ -6,11 +6,12 @@ module OMQ
|
|
|
6
6
|
# Each worker owns its own Async reactor, PULL socket, and PUSH socket.
|
|
7
7
|
#
|
|
8
8
|
class PipeWorker
|
|
9
|
-
def initialize(config, in_eps, out_eps, log_port)
|
|
10
|
-
@config
|
|
11
|
-
@in_eps
|
|
12
|
-
@out_eps
|
|
13
|
-
@log_port
|
|
9
|
+
def initialize(config, in_eps, out_eps, log_port, error_port = nil)
|
|
10
|
+
@config = config
|
|
11
|
+
@in_eps = in_eps
|
|
12
|
+
@out_eps = out_eps
|
|
13
|
+
@log_port = log_port
|
|
14
|
+
@error_port = error_port
|
|
14
15
|
end
|
|
15
16
|
|
|
16
17
|
|
|
@@ -23,6 +24,8 @@ module OMQ
|
|
|
23
24
|
compile_expr
|
|
24
25
|
run_message_loop
|
|
25
26
|
run_end_block
|
|
27
|
+
rescue OMQ::CLI::DecompressError => e
|
|
28
|
+
@error_port&.send(e.message)
|
|
26
29
|
ensure
|
|
27
30
|
@pull&.close
|
|
28
31
|
@push&.close
|
|
@@ -34,8 +37,9 @@ module OMQ
|
|
|
34
37
|
|
|
35
38
|
|
|
36
39
|
def setup_sockets
|
|
37
|
-
|
|
38
|
-
@
|
|
40
|
+
kwargs = @config.ffi ? { backend: :ffi } : {}
|
|
41
|
+
@pull = OMQ::PULL.new(**kwargs)
|
|
42
|
+
@push = OMQ::PUSH.new(**kwargs)
|
|
39
43
|
OMQ::CLI::SocketSetup.apply_options(@pull, @config)
|
|
40
44
|
OMQ::CLI::SocketSetup.apply_options(@push, @config)
|
|
41
45
|
@pull.recv_hwm = PipeRunner::PIPE_HWM unless @config.recv_hwm
|
|
@@ -68,7 +72,7 @@ module OMQ
|
|
|
68
72
|
when :message_received
|
|
69
73
|
"omq: << #{OMQ::CLI::Formatter.preview(event.detail[:parts])}"
|
|
70
74
|
else
|
|
71
|
-
ep
|
|
75
|
+
ep = event.endpoint ? " #{event.endpoint}" : ""
|
|
72
76
|
detail = event.detail ? " #{event.detail}" : ""
|
|
73
77
|
"omq: #{event.type}#{ep}#{detail}"
|
|
74
78
|
end
|
data/lib/omq/cli/socket_setup.rb
CHANGED
|
@@ -6,6 +6,21 @@ module OMQ
|
|
|
6
6
|
# All methods are module-level so callers compose rather than inherit.
|
|
7
7
|
#
|
|
8
8
|
module SocketSetup
|
|
9
|
+
# Default high water mark applied when the user does not pass
|
|
10
|
+
# --send-hwm / --recv-hwm. Lower than libzmq's default (1000) to keep
|
|
11
|
+
# memory footprint small for the typical CLI use cases (interactive
|
|
12
|
+
# debugging, short-lived pipelines). Pipe worker sockets override this
|
|
13
|
+
# with a still-smaller value for tighter backpressure.
|
|
14
|
+
DEFAULT_HWM = 100
|
|
15
|
+
|
|
16
|
+
# Default max inbound message size applied when the user does not
|
|
17
|
+
# pass --recv-maxsz. The omq library itself is unlimited by default;
|
|
18
|
+
# the CLI caps inbound messages at 1 MiB so that a misconfigured or
|
|
19
|
+
# malicious peer cannot force arbitrary memory allocation on a
|
|
20
|
+
# terminal user. Users can raise it with --recv-maxsz N, or disable
|
|
21
|
+
# it entirely with --recv-maxsz 0.
|
|
22
|
+
DEFAULT_RECV_MAXSZ = 1 << 20
|
|
23
|
+
|
|
9
24
|
# Apply common socket options from +config+ to +sock+.
|
|
10
25
|
#
|
|
11
26
|
def self.apply_options(sock, config)
|
|
@@ -14,8 +29,9 @@ module OMQ
|
|
|
14
29
|
sock.send_timeout = config.timeout if config.timeout
|
|
15
30
|
sock.reconnect_interval = config.reconnect_ivl if config.reconnect_ivl
|
|
16
31
|
sock.heartbeat_interval = config.heartbeat_ivl if config.heartbeat_ivl
|
|
17
|
-
|
|
18
|
-
sock.
|
|
32
|
+
# nil → default; 0 stays 0 (unbounded), any other integer is taken as-is.
|
|
33
|
+
sock.send_hwm = config.send_hwm || DEFAULT_HWM
|
|
34
|
+
sock.recv_hwm = config.recv_hwm || DEFAULT_HWM
|
|
19
35
|
sock.sndbuf = config.sndbuf if config.sndbuf
|
|
20
36
|
sock.rcvbuf = config.rcvbuf if config.rcvbuf
|
|
21
37
|
end
|
|
@@ -24,10 +40,16 @@ module OMQ
|
|
|
24
40
|
# Create and fully configure a socket from +klass+ and +config+.
|
|
25
41
|
#
|
|
26
42
|
def self.build(klass, config)
|
|
27
|
-
sock = klass.new
|
|
43
|
+
sock = config.ffi ? klass.new(backend: :ffi) : klass.new
|
|
28
44
|
sock.conflate = true if config.conflate && %w[pub radio].include?(config.type_name)
|
|
29
45
|
apply_options(sock, config)
|
|
30
|
-
|
|
46
|
+
# --recv-maxsz: nil → 1 MiB default; 0 → explicitly unlimited; else → as-is.
|
|
47
|
+
sock.max_message_size =
|
|
48
|
+
case config.recv_maxsz
|
|
49
|
+
when nil then DEFAULT_RECV_MAXSZ
|
|
50
|
+
when 0 then nil
|
|
51
|
+
else config.recv_maxsz
|
|
52
|
+
end
|
|
31
53
|
sock.identity = config.identity if config.identity
|
|
32
54
|
sock.router_mandatory = true if config.type_name == "router"
|
|
33
55
|
sock
|
data/lib/omq/cli/version.rb
CHANGED
data/lib/omq/cli.rb
CHANGED
|
@@ -246,6 +246,9 @@ module OMQ
|
|
|
246
246
|
runner_class.new(config)
|
|
247
247
|
end
|
|
248
248
|
runner.call(task)
|
|
249
|
+
rescue DecompressError => e
|
|
250
|
+
$stderr.puts "omq: #{e.message}"
|
|
251
|
+
exit 1
|
|
249
252
|
rescue IO::TimeoutError, Async::TimeoutError
|
|
250
253
|
$stderr.puts "omq: timeout" unless config.quiet
|
|
251
254
|
exit 2
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: omq-cli
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.9.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Patrik Wenger
|
|
@@ -128,24 +128,24 @@ dependencies:
|
|
|
128
128
|
- !ruby/object:Gem::Version
|
|
129
129
|
version: '7.0'
|
|
130
130
|
- !ruby/object:Gem::Dependency
|
|
131
|
-
name:
|
|
131
|
+
name: rlz4
|
|
132
132
|
requirement: !ruby/object:Gem::Requirement
|
|
133
133
|
requirements:
|
|
134
|
-
- - "
|
|
134
|
+
- - "~>"
|
|
135
135
|
- !ruby/object:Gem::Version
|
|
136
|
-
version: '0'
|
|
136
|
+
version: '0.1'
|
|
137
137
|
type: :runtime
|
|
138
138
|
prerelease: false
|
|
139
139
|
version_requirements: !ruby/object:Gem::Requirement
|
|
140
140
|
requirements:
|
|
141
|
-
- - "
|
|
141
|
+
- - "~>"
|
|
142
142
|
- !ruby/object:Gem::Version
|
|
143
|
-
version: '0'
|
|
143
|
+
version: '0.1'
|
|
144
144
|
description: Command-line tool for sending and receiving ZeroMQ messages on any socket
|
|
145
145
|
type (REQ/REP, PUB/SUB, PUSH/PULL, DEALER/ROUTER, and all draft types). Supports
|
|
146
146
|
Ruby eval (-e/-E), script handlers (-r), pipe virtual socket with Ractor parallelism,
|
|
147
|
-
multiple formats (ASCII, JSON Lines, msgpack, Marshal),
|
|
148
|
-
|
|
147
|
+
multiple formats (ASCII, JSON Lines, msgpack, Marshal), LZ4 compression, and CURVE
|
|
148
|
+
encryption. Like nngcat from libnng, but with Ruby superpowers.
|
|
149
149
|
email:
|
|
150
150
|
- paddor@gmail.com
|
|
151
151
|
executables:
|