otp-rails 0.1.1 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +32 -0
- data/README.md +21 -2
- data/lib/otp_rails/adapters/command.rb +30 -15
- data/lib/otp_rails/adapters/puma.rb +4 -1
- data/lib/otp_rails/adapters/solid_queue.rb +6 -1
- data/lib/otp_rails/heartbeat.rb +65 -18
- data/lib/otp_rails/socket_server.rb +56 -18
- data/lib/otp_rails/supervisor.rb +63 -11
- data/lib/otp_rails/version.rb +1 -1
- data/lib/puma/plugin/otp_rails.rb +51 -0
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 3d868a6f4f17b7af27c7af0cf1cceba00e124067d669861b963aaebcfb8ef25b
|
|
4
|
+
data.tar.gz: c2306307b5a08c6e2938f75eae9ec903453de9459b6f0922468ed155e57a597c
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: eef3e14394aedb500715af4c4381294b1b152cb86d0a295e7be62fbd1b141ca7e2e7fd94ea9b1f71b82ac8b0bede158cca412a80d127c1581b9cdf5ee9ea2260
|
|
7
|
+
data.tar.gz: bbb2544d92d370d5a55f57d642cd0e2145d48ca2e903a2645457f837e42bda62866c278c85ed7e8bc064b48db3c0ebadeb794963257fb1f719d9c780f53b88bd
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,37 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## v0.2.0 — 2026-09-13
|
|
4
|
+
|
|
5
|
+
**Puma plugin (DESIGN §4.2 step 2)** — `plugin :otp_rails` in `config/puma.rb`: the master
|
|
6
|
+
heartbeats worker-level state over the §5 socket; a missing worker reports `"degraded"`
|
|
7
|
+
(⇒ `child.degraded` telemetry, `meta: {workers, booted, phase}`) while puma replaces it —
|
|
8
|
+
visibility only, no lifecycle change. The `:puma` and `:solid_queue` adapters now export
|
|
9
|
+
`OTP_RAILS_CHILD_ID=<id>` to their children (explicit `env:` wins).
|
|
10
|
+
|
|
11
|
+
**Hardening from a three-track QA pass** (adversarial review + soak/stress + a Ruby⇄Elixir
|
|
12
|
+
contract harness now permanent in the sidecar's CI):
|
|
13
|
+
- Drain signals the whole process group — shell-wrapped cmds (`"a && b"`) no longer leave
|
|
14
|
+
their real workload running after a "clean" shutdown, or duplicate it on restart (#24).
|
|
15
|
+
- Fan-out follows OTP: all affected children stop in reverse start order before any
|
|
16
|
+
restart; declaration-order dependencies hold during `rest_for_one`/`one_for_all` (#14).
|
|
17
|
+
- One spawn path everywhere (fork → setsid → exec): an unspawnable `cmd:` is a child crash
|
|
18
|
+
(exit 127 → strategy → escalation), not a supervisor crash; identical on macOS/Linux (#15).
|
|
19
|
+
- Socket hardening: 64 KiB line cap, string-typed `token`/`cmd`/`id`/`state`, unknown-id
|
|
20
|
+
heartbeats dropped at intake, 64-connection cap, listen backlog 128, connection threads
|
|
21
|
+
torn down on stop, unusable socket path ⇒ exit 78. A well-formed heartbeat naming a
|
|
22
|
+
subtree id no longer crashes the tree (#25, #27, #16, #17, #30, #31).
|
|
23
|
+
- `stop` is prompt during stuck starts, restart fan-outs, and backoff sleeps — no more
|
|
24
|
+
blowing platform grace periods (#28). One healthy interval resets the backoff ladder
|
|
25
|
+
(#19). Non-restarted children leave no stale state or corpse telemetry (#20).
|
|
26
|
+
- `Heartbeat`: the beat thread survives raising/unencodable `state:`/`meta:` lambdas
|
|
27
|
+
(falls back to last-good state / `{}`), and closes failed sockets — no fd growth while
|
|
28
|
+
the supervisor is away (#26, #29).
|
|
29
|
+
- DESIGN §5 wire rules documented (token in the example, string fields, line cap, `ts`
|
|
30
|
+
informational); the Elixir sidecar mirrors them byte-for-byte.
|
|
31
|
+
|
|
32
|
+
**Breaking:** `OtpRails::Heartbeat.start` returns the `Heartbeat` instance (so `#stop`
|
|
33
|
+
works) instead of the raw Thread; still `nil` when unsupervised (#23).
|
|
34
|
+
|
|
3
35
|
## v0.1.1 — 2026-09-12
|
|
4
36
|
|
|
5
37
|
- Gemspec only: author listed as `timimsms`. No code changes.
|
data/README.md
CHANGED
|
@@ -76,6 +76,18 @@ Adapters:
|
|
|
76
76
|
- **`:solid_queue`** — wraps `bin/jobs` (`cmd:` overrides). Health = the active heartbeat
|
|
77
77
|
below, never the `solid_queue_processes` table.
|
|
78
78
|
|
|
79
|
+
For worker-level visibility on a cluster-mode `:puma` child, add the plugin to
|
|
80
|
+
`config/puma.rb`:
|
|
81
|
+
|
|
82
|
+
```ruby
|
|
83
|
+
plugin :otp_rails
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
The master then heartbeats worker state over the socket: any missing worker is reported
|
|
87
|
+
`"degraded"` (⇒ `[:otp_rails, :child, :degraded]` telemetry, `meta: {workers:, booted:,
|
|
88
|
+
phase:}`) while puma replaces the worker itself — visibility only, no lifecycle change.
|
|
89
|
+
Both adapters export `OTP_RAILS_CHILD_ID` so plugins and hooks heartbeat under the right id.
|
|
90
|
+
|
|
79
91
|
## Health & heartbeats
|
|
80
92
|
|
|
81
93
|
Passive children are probed (PID, TCP, HTTP). Active children report themselves: the
|
|
@@ -87,8 +99,10 @@ supervisor listens on a Unix socket (mode 0600) and exports `OTP_RAILS_SOCK` /
|
|
|
87
99
|
```
|
|
88
100
|
|
|
89
101
|
A child that has heartbeated is judged by heartbeat freshness: 3 missed `health_interval`s
|
|
90
|
-
⇒ `:degraded`, 6 ⇒ `:dead` ⇒ the strategy applies. Wrong token
|
|
91
|
-
dropped. The same socket accepts
|
|
102
|
+
⇒ `:degraded`, 6 ⇒ `:dead` ⇒ the strategy applies. Wrong token, non-string fields, or
|
|
103
|
+
lines over 64 KiB ⇒ silently dropped. The same socket accepts
|
|
104
|
+
`{"cmd":"restart","id":"jobs","token":"…"}` — a control restart is deliberate remediation
|
|
105
|
+
(DESIGN §7: restarting is a feature), so it does not count toward restart intensity.
|
|
92
106
|
|
|
93
107
|
From any child process (a Rails initializer, a Solid Queue hook — no Rails required):
|
|
94
108
|
|
|
@@ -137,6 +151,11 @@ subscriber and a JSON-lines exporter ship by default (`Telemetry::Subscribers`).
|
|
|
137
151
|
restarts the supervisor; each child runs in its own session/process group, so stale
|
|
138
152
|
orphans are findable and killable by pgid.
|
|
139
153
|
- macOS also caps Unix socket paths at ~104 bytes — keep `socket PATH` short.
|
|
154
|
+
- Compound commands (`cmd: "a && b"`) run under an `sh` wrapper. Drain and kill signal the
|
|
155
|
+
whole process group, so supervised shutdown covers the real workload — but SIGKILL-of-
|
|
156
|
+
the-supervisor orphan prevention (pdeathsig) arms only the wrapper: on Linux the wrapper
|
|
157
|
+
gets SIGTERM and its children are orphaned. Prefer single-exec commands for children
|
|
158
|
+
that must never outlive the supervisor.
|
|
140
159
|
|
|
141
160
|
## Development
|
|
142
161
|
|
|
@@ -9,21 +9,21 @@ module OtpRails
|
|
|
9
9
|
cmd = spec.opts.fetch(:cmd) { raise ConfigError, "#{spec.id}: :command adapter requires cmd:" }
|
|
10
10
|
env = spec.opts.fetch(:env, {}).transform_keys(&:to_s)
|
|
11
11
|
spawn_opts = spec.opts.fetch(:spawn_opts, {})
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
Process.spawn(env, cmd, pgroup: true, **spawn_opts)
|
|
12
|
+
# One spawn path on every platform (#15): fork → setsid → exec, with
|
|
13
|
+
# exec failure becoming exit 127 through the normal link → crash →
|
|
14
|
+
# strategy machinery. Process.spawn raised Errno::ENOENT into the
|
|
15
|
+
# supervisor loop for a bad cmd on macOS (whole tree crashed, exit 1)
|
|
16
|
+
# while the Linux fork path restart-looped to escalation (exit 70).
|
|
17
|
+
parent = Process.pid
|
|
18
|
+
pid = Process.fork do
|
|
19
|
+
Process.setsid # own session ⇒ own pgroup (drain/kill signal the group)
|
|
20
|
+
OrphanGuard.arm!(parent)
|
|
21
|
+
begin
|
|
22
|
+
Process.exec(env, cmd, **spawn_opts)
|
|
23
|
+
rescue StandardError
|
|
24
|
+
Process.exit!(127)
|
|
26
25
|
end
|
|
26
|
+
end
|
|
27
27
|
Handle.new(pid, spec, Process.clock_gettime(Process::CLOCK_MONOTONIC), nil, nil)
|
|
28
28
|
end
|
|
29
29
|
|
|
@@ -53,7 +53,12 @@ module OtpRails
|
|
|
53
53
|
end
|
|
54
54
|
|
|
55
55
|
def drain(handle, timeout:)
|
|
56
|
-
|
|
56
|
+
# Signal the whole process group, not just handle.pid: children are
|
|
57
|
+
# group leaders (setsid / pgroup: true), and a shell-wrapped cmd
|
|
58
|
+
# ("a && b") is an sh wrapper whose real workload is a grandchild in
|
|
59
|
+
# that group — TERM to the wrapper alone orphans the workload while
|
|
60
|
+
# reporting a clean drain (issue #24).
|
|
61
|
+
group_signal("TERM", handle.pid)
|
|
57
62
|
handle.waiter&.join(timeout)
|
|
58
63
|
!handle.exit_status.nil?
|
|
59
64
|
rescue Errno::ESRCH
|
|
@@ -66,6 +71,16 @@ module OtpRails
|
|
|
66
71
|
rescue Errno::ESRCH
|
|
67
72
|
nil
|
|
68
73
|
end
|
|
74
|
+
|
|
75
|
+
private
|
|
76
|
+
|
|
77
|
+
# TERM the group; fall back to the pid alone if the group is already
|
|
78
|
+
# gone by the time we signal (pure pid death races to the outer ESRCH).
|
|
79
|
+
def group_signal(sig, pid)
|
|
80
|
+
Process.kill(sig, -pid)
|
|
81
|
+
rescue Errno::ESRCH
|
|
82
|
+
Process.kill(sig, pid)
|
|
83
|
+
end
|
|
69
84
|
end
|
|
70
85
|
end
|
|
71
86
|
Adapter.register(:command, Adapters::Command)
|
|
@@ -36,7 +36,10 @@ module OtpRails
|
|
|
36
36
|
shutdown: spec.shutdown, start_timeout: spec.start_timeout,
|
|
37
37
|
opts: spec.opts.merge(
|
|
38
38
|
cmd: "bundle exec puma -C #{config}",
|
|
39
|
-
probe: { http: "http://127.0.0.1:#{port}#{HEALTH_PATH}" }
|
|
39
|
+
probe: { http: "http://127.0.0.1:#{port}#{HEALTH_PATH}" },
|
|
40
|
+
# Tag the child so the otp_rails puma plugin (and any app hook)
|
|
41
|
+
# heartbeats under the right id; explicit env still wins.
|
|
42
|
+
env: { "OTP_RAILS_CHILD_ID" => spec.id.to_s }.merge(spec.opts.fetch(:env, {}))
|
|
40
43
|
)
|
|
41
44
|
)
|
|
42
45
|
end
|
|
@@ -31,7 +31,12 @@ module OtpRails
|
|
|
31
31
|
ChildSpec.new(
|
|
32
32
|
id: spec.id, adapter: spec.adapter, restart: spec.restart,
|
|
33
33
|
shutdown: spec.shutdown, start_timeout: spec.start_timeout,
|
|
34
|
-
opts: spec.opts.merge(
|
|
34
|
+
opts: spec.opts.merge(
|
|
35
|
+
cmd: spec.opts.fetch(:cmd, DEFAULT_CMD),
|
|
36
|
+
# Tag the child so the Heartbeat hook picks up its id from env;
|
|
37
|
+
# explicit env still wins.
|
|
38
|
+
env: { "OTP_RAILS_CHILD_ID" => spec.id.to_s }.merge(spec.opts.fetch(:env, {}))
|
|
39
|
+
)
|
|
35
40
|
)
|
|
36
41
|
end
|
|
37
42
|
end
|
data/lib/otp_rails/heartbeat.rb
CHANGED
|
@@ -9,16 +9,25 @@ require "json"
|
|
|
9
9
|
module OtpRails
|
|
10
10
|
# Sends DESIGN §5 NDJSON heartbeats to the supervising socket:
|
|
11
11
|
#
|
|
12
|
-
# OtpRails::Heartbeat.start(id: "jobs")
|
|
12
|
+
# beat = OtpRails::Heartbeat.start(id: "jobs") # => Heartbeat or nil
|
|
13
|
+
# beat&.stop
|
|
14
|
+
#
|
|
13
15
|
# OtpRails::Heartbeat.start(id: "jobs", interval: 2,
|
|
14
16
|
# state: -> { queue_backlog_ok? ? "healthy" : "degraded" },
|
|
15
17
|
# meta: -> { { backlog: backlog_size } })
|
|
16
18
|
#
|
|
17
|
-
# Silently a no-op when OTP_RAILS_SOCK / OTP_RAILS_TOKEN are
|
|
18
|
-
# child is running unsupervised
|
|
19
|
-
#
|
|
20
|
-
#
|
|
19
|
+
# Silently a no-op (returns nil) when OTP_RAILS_SOCK / OTP_RAILS_TOKEN are
|
|
20
|
+
# absent — the child is running unsupervised and that must not be an error.
|
|
21
|
+
#
|
|
22
|
+
# The beating thread is unkillable by bad input (issue #26): a raising
|
|
23
|
+
# state/meta lambda falls back to the last good state / empty meta, an
|
|
24
|
+
# unencodable payload (bad UTF-8) drops down to a bare healthy beat, and
|
|
25
|
+
# socket failures close the socket (issue #29) and retry next beat. Going
|
|
26
|
+
# silent is the one thing this thread must never do while its process is
|
|
27
|
+
# healthy — silence is what gets the child restarted.
|
|
21
28
|
class Heartbeat
|
|
29
|
+
# Returns the Heartbeat (so #stop works — issue #23), or nil when
|
|
30
|
+
# unsupervised.
|
|
22
31
|
def self.start(id:, interval: 2, state: -> { "healthy" }, meta: -> { {} })
|
|
23
32
|
new(id: id, interval: interval, state: state, meta: meta).start
|
|
24
33
|
end
|
|
@@ -26,26 +35,64 @@ module OtpRails
|
|
|
26
35
|
def initialize(id:, interval: 2, state: -> { "healthy" }, meta: -> { {} })
|
|
27
36
|
@id, @interval, @state, @meta = id.to_s, interval, state, meta
|
|
28
37
|
@sock_path, @token = ENV["OTP_RAILS_SOCK"], ENV["OTP_RAILS_TOKEN"]
|
|
38
|
+
@last_state = "healthy"
|
|
29
39
|
end
|
|
30
40
|
|
|
31
|
-
# Returns the beating thread, or nil when unsupervised.
|
|
32
41
|
def start
|
|
33
42
|
return nil unless @sock_path && @token
|
|
34
|
-
@thread
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
43
|
+
@thread ||= Thread.new { run_loop }
|
|
44
|
+
self
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
def stop
|
|
48
|
+
@thread&.kill
|
|
49
|
+
close_socket
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
def alive? = !!@thread&.alive?
|
|
53
|
+
|
|
54
|
+
private
|
|
55
|
+
|
|
56
|
+
def run_loop
|
|
57
|
+
loop do
|
|
58
|
+
begin
|
|
59
|
+
payload = build_payload
|
|
60
|
+
@sock ||= UNIXSocket.new(@sock_path)
|
|
61
|
+
@sock.puts(payload)
|
|
62
|
+
rescue StandardError
|
|
63
|
+
close_socket # supervisor gone or restarting; reconnect next beat
|
|
45
64
|
end
|
|
65
|
+
sleep @interval
|
|
46
66
|
end
|
|
47
67
|
end
|
|
48
68
|
|
|
49
|
-
def
|
|
69
|
+
def build_payload
|
|
70
|
+
state = safe_state
|
|
71
|
+
meta = begin
|
|
72
|
+
@meta.call
|
|
73
|
+
rescue StandardError
|
|
74
|
+
{}
|
|
75
|
+
end
|
|
76
|
+
encode(state, meta) || encode(state, {}) || encode("healthy", {})
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
def safe_state
|
|
80
|
+
@last_state = @state.call.to_s
|
|
81
|
+
rescue StandardError
|
|
82
|
+
@last_state
|
|
83
|
+
end
|
|
84
|
+
|
|
85
|
+
def encode(state, meta)
|
|
86
|
+
JSON.generate(id: @id, state: state, ts: Time.now.to_i, token: @token, meta: meta)
|
|
87
|
+
rescue StandardError
|
|
88
|
+
nil
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
def close_socket
|
|
92
|
+
@sock&.close
|
|
93
|
+
@sock = nil
|
|
94
|
+
rescue IOError, SystemCallError
|
|
95
|
+
@sock = nil
|
|
96
|
+
end
|
|
50
97
|
end
|
|
51
98
|
end
|
|
@@ -14,26 +14,41 @@ module OtpRails
|
|
|
14
14
|
# Control: {"cmd":"restart","id":"jobs","token":"…"}
|
|
15
15
|
# Any line with a missing or wrong token is dropped without a reply.
|
|
16
16
|
class SocketServer
|
|
17
|
+
MAX_LINE = 64 * 1024 # §5: longer lines are malformed and dropped
|
|
18
|
+
MAX_CONNS = 64 # excess connections are refused (closed immediately)
|
|
19
|
+
|
|
17
20
|
attr_reader :path, :token
|
|
18
21
|
|
|
19
22
|
def initialize(path:, on_heartbeat:, on_control:)
|
|
20
23
|
@path, @on_heartbeat, @on_control = path, on_heartbeat, on_control
|
|
21
24
|
@token = SecureRandom.hex(16)
|
|
25
|
+
@conns = []
|
|
22
26
|
end
|
|
23
27
|
|
|
24
28
|
# Binds, chmods, and exports OTP_RAILS_SOCK / OTP_RAILS_TOKEN so children
|
|
25
29
|
# spawned afterwards inherit them (DESIGN §9). Call before starting children.
|
|
26
30
|
def start
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
31
|
+
begin
|
|
32
|
+
FileUtils.mkdir_p(File.dirname(@path))
|
|
33
|
+
File.unlink(@path) if File.exist?(@path) # stale socket from a dead boot
|
|
34
|
+
@server = UNIXServer.new(@path)
|
|
35
|
+
rescue ArgumentError, SystemCallError => e
|
|
36
|
+
# e.g. > ~104-byte path on macOS, unwritable dir: config problem, not a crash
|
|
37
|
+
raise ConfigError, "heartbeat socket #{@path.inspect}: #{e.message}"
|
|
38
|
+
end
|
|
39
|
+
@server.listen(128) # default backlog is 5 on macOS; bursts got ECONNREFUSED
|
|
30
40
|
File.chmod(0o600, @path)
|
|
31
41
|
ENV["OTP_RAILS_SOCK"] = @path
|
|
32
42
|
ENV["OTP_RAILS_TOKEN"] = @token
|
|
33
43
|
@acceptor = Thread.new do
|
|
34
44
|
loop do
|
|
35
45
|
conn = @server.accept
|
|
36
|
-
|
|
46
|
+
@conns.reject! { |c| !c[:thread].alive? }
|
|
47
|
+
if @conns.size >= MAX_CONNS
|
|
48
|
+
close_quietly(conn)
|
|
49
|
+
next
|
|
50
|
+
end
|
|
51
|
+
@conns << { conn: conn, thread: Thread.new { serve(conn) } }
|
|
37
52
|
rescue IOError, SystemCallError
|
|
38
53
|
break # server closed during shutdown
|
|
39
54
|
end
|
|
@@ -43,6 +58,11 @@ module OtpRails
|
|
|
43
58
|
def stop
|
|
44
59
|
@server&.close
|
|
45
60
|
@acceptor&.kill
|
|
61
|
+
@conns.each do |c| # connection threads must not outlive the server
|
|
62
|
+
c[:thread].kill
|
|
63
|
+
close_quietly(c[:conn])
|
|
64
|
+
end
|
|
65
|
+
@conns.clear
|
|
46
66
|
File.unlink(@path) if File.exist?(@path)
|
|
47
67
|
rescue SystemCallError
|
|
48
68
|
nil
|
|
@@ -51,27 +71,45 @@ module OtpRails
|
|
|
51
71
|
private
|
|
52
72
|
|
|
53
73
|
def serve(conn)
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
74
|
+
loop do
|
|
75
|
+
line = conn.gets("\n", MAX_LINE)
|
|
76
|
+
break if line.nil?
|
|
77
|
+
unless line.end_with?("\n")
|
|
78
|
+
# over-long line: memory stays capped at MAX_LINE — discard the rest
|
|
79
|
+
# of the line, then resume at the next newline
|
|
80
|
+
line = conn.gets("\n", MAX_LINE) while !line.nil? && !line.end_with?("\n")
|
|
81
|
+
break if line.nil?
|
|
58
82
|
next
|
|
59
83
|
end
|
|
60
|
-
|
|
61
|
-
if msg["cmd"]
|
|
62
|
-
@on_control.call(msg)
|
|
63
|
-
elsif msg["id"] && msg["state"]
|
|
64
|
-
@on_heartbeat.call(msg)
|
|
65
|
-
end
|
|
84
|
+
handle_line(line)
|
|
66
85
|
end
|
|
67
86
|
rescue IOError, SystemCallError
|
|
68
87
|
nil
|
|
69
88
|
ensure
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
89
|
+
close_quietly(conn)
|
|
90
|
+
end
|
|
91
|
+
|
|
92
|
+
# §5: token, cmd, id, and state are JSON strings; anything else in those
|
|
93
|
+
# fields is malformed and silently dropped — same as a bad token. A bad
|
|
94
|
+
# line must never take the connection (or the supervisor) down.
|
|
95
|
+
def handle_line(line)
|
|
96
|
+
msg = begin
|
|
97
|
+
JSON.parse(line)
|
|
98
|
+
rescue StandardError
|
|
99
|
+
return
|
|
100
|
+
end
|
|
101
|
+
return unless msg.is_a?(Hash) && msg["token"] == @token
|
|
102
|
+
if msg.key?("cmd")
|
|
103
|
+
@on_control.call(msg) if msg["cmd"].is_a?(String)
|
|
104
|
+
elsif msg["id"].is_a?(String) && msg["state"].is_a?(String)
|
|
105
|
+
@on_heartbeat.call(msg)
|
|
74
106
|
end
|
|
75
107
|
end
|
|
108
|
+
|
|
109
|
+
def close_quietly(conn)
|
|
110
|
+
conn.close
|
|
111
|
+
rescue IOError, SystemCallError
|
|
112
|
+
nil
|
|
113
|
+
end
|
|
76
114
|
end
|
|
77
115
|
end
|
data/lib/otp_rails/supervisor.rb
CHANGED
|
@@ -14,11 +14,19 @@ module OtpRails
|
|
|
14
14
|
@live = {} # id => { adapter:, handle:, attempts:, generation:, monitor: }
|
|
15
15
|
@queue = Queue.new
|
|
16
16
|
@stopping = false
|
|
17
|
+
@stop_requested = false
|
|
17
18
|
@heartbeats = {} # id => { at: monotonic ts of last heartbeat, state: reported state }
|
|
18
19
|
return unless socket_path
|
|
19
20
|
@socket = SocketServer.new(
|
|
20
21
|
path: socket_path,
|
|
21
|
-
|
|
22
|
+
# Heartbeats for ids that aren't children of THIS supervisor are
|
|
23
|
+
# dropped at intake: ghost ids must not grow the table unboundedly
|
|
24
|
+
# (issue #16/#27). Subtree-grandchild routing is the open flat-id
|
|
25
|
+
# question — until it's decided, their beats are dropped, not hoarded.
|
|
26
|
+
on_heartbeat: lambda { |msg|
|
|
27
|
+
id = msg["id"].to_sym
|
|
28
|
+
@heartbeats[id] = { at: mono_now, state: msg["state"] } if @children.any? { |c| c.id == id }
|
|
29
|
+
},
|
|
22
30
|
on_control: ->(msg) { @queue << { type: :control, cmd: msg["cmd"], id: msg["id"].to_s.to_sym } }
|
|
23
31
|
)
|
|
24
32
|
end
|
|
@@ -37,7 +45,10 @@ module OtpRails
|
|
|
37
45
|
def run
|
|
38
46
|
Telemetry.emit(:"supervisor.start", {}, { strategy: strategy, children: ids })
|
|
39
47
|
@socket&.start # before children, so they inherit OTP_RAILS_SOCK/_TOKEN
|
|
40
|
-
@children.each
|
|
48
|
+
@children.each do |spec|
|
|
49
|
+
break if @stop_requested
|
|
50
|
+
start_child(spec)
|
|
51
|
+
end
|
|
41
52
|
loop do
|
|
42
53
|
msg = @queue.pop
|
|
43
54
|
case msg[:type]
|
|
@@ -53,7 +64,14 @@ module OtpRails
|
|
|
53
64
|
Telemetry.emit(:"supervisor.stop")
|
|
54
65
|
end
|
|
55
66
|
|
|
56
|
-
|
|
67
|
+
# Sets the flag first: the main loop may be stuck in a wait_healthy poll
|
|
68
|
+
# or a backoff sleep for up to start_timeout/backoff seconds, and shutdown
|
|
69
|
+
# must not wait for those (issue #28 — platforms SIGKILL after their grace
|
|
70
|
+
# period, which resurrects the orphan problem).
|
|
71
|
+
def stop
|
|
72
|
+
@stop_requested = true
|
|
73
|
+
@queue << { type: :stop }
|
|
74
|
+
end
|
|
57
75
|
|
|
58
76
|
# Public remediation API (DESIGN §7). Over IPC in the real thing; direct call here.
|
|
59
77
|
def restart!(id)
|
|
@@ -100,6 +118,11 @@ module OtpRails
|
|
|
100
118
|
case effective_health(spec, adapter, handle)
|
|
101
119
|
when :healthy
|
|
102
120
|
degraded = 0
|
|
121
|
+
# One healthy interval resets the backoff ladder (#19): a child
|
|
122
|
+
# that crashes rarely should not converge to permanent max
|
|
123
|
+
# backoff. Crash-looping children never reach a monitor, so flap
|
|
124
|
+
# damping is unaffected.
|
|
125
|
+
entry[:attempts] = 0
|
|
103
126
|
when :degraded
|
|
104
127
|
degraded += 1
|
|
105
128
|
Telemetry.emit(:"child.degraded", { consecutive: degraded }, { id: spec.id })
|
|
@@ -118,6 +141,7 @@ module OtpRails
|
|
|
118
141
|
def wait_healthy(spec, adapter, handle)
|
|
119
142
|
deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + spec.start_timeout
|
|
120
143
|
loop do
|
|
144
|
+
return :stopping if @stop_requested # shutdown must not wait out start_timeout (#28)
|
|
121
145
|
case effective_health(spec, adapter, handle)
|
|
122
146
|
when :healthy then Telemetry.emit(:"child.healthy", {}, { id: spec.id }); return :healthy
|
|
123
147
|
when :dead then return :dead # the exit message arrives via link
|
|
@@ -142,7 +166,10 @@ module OtpRails
|
|
|
142
166
|
# back to the adapter's passive probe.
|
|
143
167
|
def effective_health(spec, adapter, handle)
|
|
144
168
|
hb = @heartbeats[spec.id]
|
|
145
|
-
|
|
169
|
+
# No heartbeat ⇒ passive probe. A nil health_interval (subtree specs)
|
|
170
|
+
# also falls through: freshness aging needs an interval, and dividing
|
|
171
|
+
# by nil crashed the whole tree when a heartbeat named such an id (#25).
|
|
172
|
+
return adapter.health(handle) unless hb && spec.health_interval
|
|
146
173
|
missed = (mono_now - hb[:at]) / spec.health_interval
|
|
147
174
|
return :dead if missed >= 6
|
|
148
175
|
return :degraded if missed >= 3
|
|
@@ -178,21 +205,46 @@ module OtpRails
|
|
|
178
205
|
|
|
179
206
|
uptime = Process.clock_gettime(Process::CLOCK_MONOTONIC) - entry[:handle].started_at
|
|
180
207
|
Telemetry.emit(:"child.exit", { exit_code: status&.exitstatus, uptime_ms: (uptime * 1000).round }, { id: id })
|
|
181
|
-
|
|
208
|
+
unless spec.restart?(status)
|
|
209
|
+
# The child is gone for good: keep no stale entry, or stop_all and the
|
|
210
|
+
# monitor emit spurious drains for a corpse later (issue #20).
|
|
211
|
+
entry[:monitor]&.kill
|
|
212
|
+
@live.delete(id)
|
|
213
|
+
return
|
|
214
|
+
end
|
|
182
215
|
|
|
183
216
|
if intensity.record!
|
|
184
217
|
Telemetry.emit(:"supervisor.escalate", { restarts: intensity.count }, { within: intensity.within })
|
|
185
218
|
raise Escalation, "restart intensity exceeded (#{intensity.count} in #{intensity.within}s)"
|
|
186
219
|
end
|
|
187
220
|
|
|
188
|
-
Strategy.affected(strategy, ids, id)
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
221
|
+
affected = Strategy.affected(strategy, ids, id)
|
|
222
|
+
# OTP semantics (#14): declaration order encodes dependency, so first
|
|
223
|
+
# terminate ALL affected children in reverse start order — a
|
|
224
|
+
# replacement :b must never boot while an old :c that depended on the
|
|
225
|
+
# dead :b is still running — then restart them in start order.
|
|
226
|
+
affected.reverse_each do |aid|
|
|
227
|
+
stop_child(spec_for(aid)) unless aid == id
|
|
228
|
+
end
|
|
229
|
+
affected.each do |aid|
|
|
230
|
+
break if @stop_requested # shutdown preempts the restart fan-out (#28)
|
|
231
|
+
entry = @live[aid]
|
|
232
|
+
next unless entry # a temporary/clean-transient sibling is gone for good (#20)
|
|
233
|
+
attempts = (entry[:attempts] += 1)
|
|
192
234
|
delay = backoff.delay(attempts)
|
|
193
235
|
Telemetry.emit(:"child.restart", { backoff_ms: (delay * 1000).round }, { id: aid, attempt: attempts, strategy: strategy })
|
|
194
|
-
|
|
195
|
-
|
|
236
|
+
interruptible_sleep(delay)
|
|
237
|
+
break if @stop_requested
|
|
238
|
+
start_child(spec_for(aid))
|
|
239
|
+
end
|
|
240
|
+
end
|
|
241
|
+
|
|
242
|
+
# Backoff must not delay shutdown (#28): sleep in slices, bail on stop.
|
|
243
|
+
def interruptible_sleep(seconds)
|
|
244
|
+
deadline = mono_now + seconds
|
|
245
|
+
while mono_now < deadline
|
|
246
|
+
return if @stop_requested
|
|
247
|
+
sleep [0.1, deadline - mono_now].min
|
|
196
248
|
end
|
|
197
249
|
end
|
|
198
250
|
|
data/lib/otp_rails/version.rb
CHANGED
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
# Puma plugin (DESIGN §4.2 step 2, PLAN 2.5): worker-level visibility for a
|
|
3
|
+
# :puma child, over the §5 heartbeat socket. Enable with `plugin :otp_rails`
|
|
4
|
+
# in config/puma.rb.
|
|
5
|
+
#
|
|
6
|
+
# Lives under lib/puma/ (NOT lib/otp_rails/) because it may require puma —
|
|
7
|
+
# it is only ever loaded BY puma, so hard rule 1 (the supervisor has zero
|
|
8
|
+
# dependencies) holds. It reuses the frozen heartbeat protocol untouched:
|
|
9
|
+
# worker detail travels in `state` and `meta`, never in new fields.
|
|
10
|
+
#
|
|
11
|
+
# The master heartbeats every OTP_RAILS_HEARTBEAT_INTERVAL (default 2s):
|
|
12
|
+
# state = "degraded" while any worker is missing (booted < configured),
|
|
13
|
+
# "healthy" otherwise (single mode is always "healthy"),
|
|
14
|
+
# meta = { workers:, booted:, phase: } (cluster) | { mode: "single" }.
|
|
15
|
+
# The supervisor turns a reported "degraded" into [:otp_rails, :child,
|
|
16
|
+
# :degraded] telemetry via its normal health loop — no event added, and no
|
|
17
|
+
# lifecycle change: puma still replaces its own workers.
|
|
18
|
+
require "puma/plugin"
|
|
19
|
+
require "otp_rails/heartbeat"
|
|
20
|
+
|
|
21
|
+
Puma::Plugin.create do
|
|
22
|
+
def start(launcher)
|
|
23
|
+
beat = OtpRails::Heartbeat.start(
|
|
24
|
+
id: ENV["OTP_RAILS_CHILD_ID"] || "web",
|
|
25
|
+
interval: Float(ENV.fetch("OTP_RAILS_HEARTBEAT_INTERVAL", 2)),
|
|
26
|
+
state: -> { otp_rails_state(launcher) },
|
|
27
|
+
meta: -> { otp_rails_meta(launcher) }
|
|
28
|
+
)
|
|
29
|
+
launcher.events.on_stopped { beat.stop } if beat
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
private
|
|
33
|
+
|
|
34
|
+
# Raising inside the heartbeat thread would silently kill it, so both
|
|
35
|
+
# lambdas degrade to a safe value instead (stats can raise mid-boot).
|
|
36
|
+
def otp_rails_state(launcher)
|
|
37
|
+
stats = launcher.stats
|
|
38
|
+
return "healthy" unless stats[:workers] # single mode
|
|
39
|
+
stats[:booted_workers] < stats[:workers] ? "degraded" : "healthy"
|
|
40
|
+
rescue StandardError
|
|
41
|
+
"starting"
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
def otp_rails_meta(launcher)
|
|
45
|
+
stats = launcher.stats
|
|
46
|
+
return { mode: "single" } unless stats[:workers]
|
|
47
|
+
{ workers: stats[:workers], booted: stats[:booted_workers], phase: stats[:phase] }
|
|
48
|
+
rescue StandardError
|
|
49
|
+
{}
|
|
50
|
+
end
|
|
51
|
+
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: otp-rails
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.2.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- timimsms
|
|
@@ -41,6 +41,7 @@ files:
|
|
|
41
41
|
- lib/otp_rails/supervisor.rb
|
|
42
42
|
- lib/otp_rails/telemetry.rb
|
|
43
43
|
- lib/otp_rails/version.rb
|
|
44
|
+
- lib/puma/plugin/otp_rails.rb
|
|
44
45
|
homepage: https://github.com/shishi-odoshi/otp-rails
|
|
45
46
|
licenses:
|
|
46
47
|
- MIT
|