odoshi 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +93 -0
- data/LICENSE +21 -0
- data/README.md +168 -0
- data/exe/odoshi +4 -0
- data/lib/odoshi/adapter.rb +21 -0
- data/lib/odoshi/adapters/command.rb +87 -0
- data/lib/odoshi/adapters/puma.rb +59 -0
- data/lib/odoshi/adapters/solid_queue.rb +45 -0
- data/lib/odoshi/adapters/supervisor_adapter.rb +95 -0
- data/lib/odoshi/backoff.rb +18 -0
- data/lib/odoshi/child_spec.rb +33 -0
- data/lib/odoshi/cli.rb +33 -0
- data/lib/odoshi/dsl.rb +85 -0
- data/lib/odoshi/heartbeat.rb +98 -0
- data/lib/odoshi/orphan_guard.rb +35 -0
- data/lib/odoshi/probe.rb +44 -0
- data/lib/odoshi/restart_intensity.rb +22 -0
- data/lib/odoshi/socket_server.rb +115 -0
- data/lib/odoshi/strategy.rb +18 -0
- data/lib/odoshi/supervisor.rb +266 -0
- data/lib/odoshi/telemetry.rb +64 -0
- data/lib/odoshi/version.rb +4 -0
- data/lib/odoshi.rb +31 -0
- data/lib/puma/plugin/odoshi.rb +51 -0
- metadata +72 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 470bf921f6ef90d3d251f69d22aea25085c61ff8ca722436a933f7e0a05fb898
|
|
4
|
+
data.tar.gz: 02e357c5e4191bcbbf2bdd838046a9d6d89fa98c1714db74d60e2e090d32c680
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 3a61f368455b2ef8faae1f6140feaefd4c6c5bbe10b1cc67eeb5f35c905ba10f21dfac0e003141ae3f759c0d40be95a99128cf7060b9f03467c19529c022c3a2
|
|
7
|
+
data.tar.gz: 5e4bcd0c4385aa3d02a45f64e19c6657126f72efb208b8d514c5577f7140c97cafde64363b6be5ce1d2334969392c4c830702814db3b76d90bf5e5fc503f6f8f
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## v0.3.0 — 2026-09-13 — RENAMED: otp-rails → odoshi
|
|
4
|
+
|
|
5
|
+
The gem is now **odoshi** (威し — the active half of [shishi-odoshi](https://github.com/shishi-odoshi),
|
|
6
|
+
the self-resetting bamboo fountain). "otp" reads as one-time password in Rubyland; this
|
|
7
|
+
project's OTP was always the Erlang kind. Everything renames — **breaking across the board**:
|
|
8
|
+
|
|
9
|
+
| was | is |
|
|
10
|
+
|---|---|
|
|
11
|
+
| `gem "otp-rails"` | `gem "odoshi"` |
|
|
12
|
+
| `OtpRails::` / `require "otp_rails/…"` | `Odoshi::` / `require "odoshi/…"` |
|
|
13
|
+
| `otp-rails run` (CLI) | `odoshi run` |
|
|
14
|
+
| `OTP_RAILS_SOCK` / `_TOKEN` / `_CHILD_ID` / `_HEARTBEAT_INTERVAL` | `ODOSHI_SOCK` / `_TOKEN` / `_CHILD_ID` / `_HEARTBEAT_INTERVAL` |
|
|
15
|
+
| telemetry `[:otp_rails, …]` | `[:odoshi, …]` |
|
|
16
|
+
| `plugin :otp_rails` (puma) | `plugin :odoshi` |
|
|
17
|
+
| socket default `tmp/otp-rails.sock` | `tmp/odoshi.sock` |
|
|
18
|
+
|
|
19
|
+
The Elixir sidecar mirrors the env and telemetry names. Old versions remain published
|
|
20
|
+
under `otp-rails` (0.2.1 is a pointer release); no compatibility shims — rename atomically.
|
|
21
|
+
|
|
22
|
+
Also in this release:
|
|
23
|
+
- Telemetry subscribers are isolated — one raising subscriber can't break the bus or
|
|
24
|
+
reach the supervisor loop (QA round 2).
|
|
25
|
+
|
|
26
|
+
## v0.2.0 — 2026-09-13
|
|
27
|
+
|
|
28
|
+
**Puma plugin (DESIGN §4.2 step 2)** — `plugin :otp_rails` in `config/puma.rb`: the master
|
|
29
|
+
heartbeats worker-level state over the §5 socket; a missing worker reports `"degraded"`
|
|
30
|
+
(⇒ `child.degraded` telemetry, `meta: {workers, booted, phase}`) while puma replaces it —
|
|
31
|
+
visibility only, no lifecycle change. The `:puma` and `:solid_queue` adapters now export
|
|
32
|
+
`OTP_RAILS_CHILD_ID=<id>` to their children (explicit `env:` wins).
|
|
33
|
+
|
|
34
|
+
**Hardening from a three-track QA pass** (adversarial review + soak/stress + a Ruby⇄Elixir
|
|
35
|
+
contract harness now permanent in the sidecar's CI):
|
|
36
|
+
- Drain signals the whole process group — shell-wrapped cmds (`"a && b"`) no longer leave
|
|
37
|
+
their real workload running after a "clean" shutdown, or duplicate it on restart (#24).
|
|
38
|
+
- Fan-out follows OTP: all affected children stop in reverse start order before any
|
|
39
|
+
restart; declaration-order dependencies hold during `rest_for_one`/`one_for_all` (#14).
|
|
40
|
+
- One spawn path everywhere (fork → setsid → exec): an unspawnable `cmd:` is a child crash
|
|
41
|
+
(exit 127 → strategy → escalation), not a supervisor crash; identical on macOS/Linux (#15).
|
|
42
|
+
- Socket hardening: 64 KiB line cap, string-typed `token`/`cmd`/`id`/`state`, unknown-id
|
|
43
|
+
heartbeats dropped at intake, 64-connection cap, listen backlog 128, connection threads
|
|
44
|
+
torn down on stop, unusable socket path ⇒ exit 78. A well-formed heartbeat naming a
|
|
45
|
+
subtree id no longer crashes the tree (#25, #27, #16, #17, #30, #31).
|
|
46
|
+
- `stop` is prompt during stuck starts, restart fan-outs, and backoff sleeps — no more
|
|
47
|
+
blowing platform grace periods (#28). One healthy interval resets the backoff ladder
|
|
48
|
+
(#19). Non-restarted children leave no stale state or corpse telemetry (#20).
|
|
49
|
+
- `Heartbeat`: the beat thread survives raising/unencodable `state:`/`meta:` lambdas
|
|
50
|
+
(falls back to last-good state / `{}`), and closes failed sockets — no fd growth while
|
|
51
|
+
the supervisor is away (#26, #29).
|
|
52
|
+
- DESIGN §5 wire rules documented (token in the example, string fields, line cap, `ts`
|
|
53
|
+
informational); the Elixir sidecar mirrors them byte-for-byte.
|
|
54
|
+
|
|
55
|
+
**Breaking:** `OtpRails::Heartbeat.start` returns the `Heartbeat` instance (so `#stop`
|
|
56
|
+
works) instead of the raw Thread; still `nil` when unsupervised (#23).
|
|
57
|
+
|
|
58
|
+
## v0.1.1 — 2026-09-12
|
|
59
|
+
|
|
60
|
+
- Gemspec only: author listed as `timimsms`. No code changes.
|
|
61
|
+
|
|
62
|
+
## v0.1.0 — 2026-09-12
|
|
63
|
+
|
|
64
|
+
First release: a slim, zero-runtime-dependency, OTP-style process supervisor for the
|
|
65
|
+
processes of a Rails app. The supervisor never loads Rails (DESIGN §9).
|
|
66
|
+
|
|
67
|
+
- **Supervision core** — `ChildSpec` with `permanent`/`transient`/`temporary` restart
|
|
68
|
+
semantics; `one_for_one`, `rest_for_one`, `one_for_all` strategies; restart intensity
|
|
69
|
+
with a sliding window that escalates (exit 70) when exceeded; `none`/`constant`/
|
|
70
|
+
`exponential` backoff; generation-guarded exit handling (stale exits ignored).
|
|
71
|
+
- **Shutdown correctness** (#1) — `stop_all` drains in reverse start order, waiting per
|
|
72
|
+
child (SIGTERM, then SIGKILL of the process group after `shutdown:`). Exit codes:
|
|
73
|
+
0 clean, 70 escalation, 78 config error. Orphan prevention on Linux via
|
|
74
|
+
`prctl(PR_SET_PDEATHSIG, SIGTERM)`; macOS has no equivalent (documented limitation).
|
|
75
|
+
- **Passive probes** (#2) — `probe: { tcp: PORT }` / `probe: { http: URL }` on `:command`:
|
|
76
|
+
`:starting` until the probe answers, then `:healthy`. `start_timeout` exceeded ⇒ drain ⇒
|
|
77
|
+
counts as a crash ⇒ strategy applies.
|
|
78
|
+
- **Periodic health loop** (#3) — per-child monitor at `health_interval` (default 5s);
|
|
79
|
+
`:degraded` emits telemetry only; `degraded_restart_after: N` drains after N consecutive
|
|
80
|
+
degraded reports (flows through the crash path, so intensity applies).
|
|
81
|
+
- **`:puma` adapter, beside mode** (#4) — wraps the puma master; health = HTTP `/up` probe;
|
|
82
|
+
drain = SIGTERM (Puma graceful). `opts[:port]` or a literal `port NNNN` in the config.
|
|
83
|
+
- **Active heartbeat socket** (#5) — Unix socket (default `tmp/otp-rails.sock`, mode 0600),
|
|
84
|
+
per-boot token via `OTP_RAILS_SOCK` / `OTP_RAILS_TOKEN`; NDJSON heartbeats
|
|
85
|
+
`{"id","state","ts","token","meta"}`; bad token dropped; 3 missed intervals ⇒ `:degraded`,
|
|
86
|
+
6 ⇒ `:dead` ⇒ restart; `{"cmd":"restart","id":...}` control messages. This wire protocol
|
|
87
|
+
is the contract consumed by the Elixir sidecar (`shishi-odoshi/beam`).
|
|
88
|
+
- **`:solid_queue` adapter + heartbeat helper** (#6) — wraps `bin/jobs`; health = active
|
|
89
|
+
heartbeat, not the DB table. `require "otp_rails/heartbeat"` is Rails-free and
|
|
90
|
+
self-contained; silently a no-op when unsupervised.
|
|
91
|
+
- **Nested supervisors** (#7) — `supervisor :background do ... end` creates a subtree with
|
|
92
|
+
its own strategy/intensity/backoff; subtree escalation is an ordinary child exit in the
|
|
93
|
+
parent; restarted subtrees get a fresh intensity window.
|
data/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Tim
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,168 @@
|
|
|
1
|
+
# odoshi
|
|
2
|
+
|
|
3
|
+
OTP-style supervision trees for the processes of a Rails app. A slim supervisor — it never
|
|
4
|
+
loads Rails — that starts, links, health-checks, and restarts `web`, `jobs`, `cable`, `cron`
|
|
5
|
+
with `one_for_one` / `rest_for_one` / `one_for_all` strategies, restart intensity, and backoff.
|
|
6
|
+
|
|
7
|
+
Part of the [shishi-odoshi](https://github.com/shishi-odoshi) org. Design: `docs/DESIGN.md`.
|
|
8
|
+
The Elixir sidecar speaking the same wire protocol lives at
|
|
9
|
+
[shishi-odoshi/beam](https://github.com/shishi-odoshi/beam).
|
|
10
|
+
|
|
11
|
+
## Install
|
|
12
|
+
|
|
13
|
+
```ruby
|
|
14
|
+
# Gemfile — the supervisor is its own process; a :supervisor group keeps app boot slim
|
|
15
|
+
group :supervisor do
|
|
16
|
+
gem "odoshi"
|
|
17
|
+
end
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
```
|
|
21
|
+
$ bundle install
|
|
22
|
+
$ odoshi version
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
Zero runtime dependencies. Ruby >= 3.2.
|
|
26
|
+
|
|
27
|
+
## Quick start
|
|
28
|
+
|
|
29
|
+
```ruby
|
|
30
|
+
# config/supervisor.rb — plain Ruby, evaluated WITHOUT Rails
|
|
31
|
+
strategy :rest_for_one
|
|
32
|
+
max_restarts 5, within: 60
|
|
33
|
+
backoff :exponential, base: 1, cap: 30
|
|
34
|
+
|
|
35
|
+
child :web, adapter: :puma, port: 3000
|
|
36
|
+
child :jobs, adapter: :solid_queue, shutdown: 60
|
|
37
|
+
child :cron, adapter: :command, cmd: "bin/rails cron", restart: :transient
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
```
|
|
41
|
+
$ odoshi check config/supervisor.rb # validate config, print the tree
|
|
42
|
+
$ odoshi run config/supervisor.rb # supervise (Ctrl-C drains and stops)
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
## Config reference
|
|
46
|
+
|
|
47
|
+
Top-level directives in `config/supervisor.rb`:
|
|
48
|
+
|
|
49
|
+
| Directive | Default | Meaning |
|
|
50
|
+
|---|---|---|
|
|
51
|
+
| `strategy KIND` | `:one_for_one` | `:one_for_one` restarts only the failed child; `:rest_for_one` also restarts children declared after it; `:one_for_all` restarts every child |
|
|
52
|
+
| `max_restarts N, within: S` | `5, within: 60` | Sliding-window restart intensity; exceeding it escalates (exit 70) |
|
|
53
|
+
| `backoff KIND, **opts` | `:exponential, base: 1, cap: 30` | `:none`, `:constant`, or `:exponential` delay between restarts |
|
|
54
|
+
| `socket PATH` | `"tmp/odoshi.sock"` | Heartbeat/control Unix socket; `socket nil` disables it |
|
|
55
|
+
| `child ID, adapter:, **opts` | — | Declares a child; declaration order is start order |
|
|
56
|
+
| `supervisor ID do ... end` | — | Nested subtree with its own strategy/intensity/backoff; subtree escalation is an ordinary child exit in the parent |
|
|
57
|
+
|
|
58
|
+
Per-child options:
|
|
59
|
+
|
|
60
|
+
| Option | Default | Meaning |
|
|
61
|
+
|---|---|---|
|
|
62
|
+
| `restart:` | `:permanent` | `:permanent` always restarts; `:transient` restarts only on non-zero exit; `:temporary` never restarts |
|
|
63
|
+
| `shutdown:` | `30` | Seconds to wait after SIGTERM before SIGKILL of the child's process group |
|
|
64
|
+
| `start_timeout:` | `30` | Seconds to reach `:healthy`; exceeding it drains the child and counts as a crash |
|
|
65
|
+
| `health_interval:` | `5` | Seconds between health checks (and the unit for heartbeat freshness) |
|
|
66
|
+
| `degraded_restart_after:` | `nil` | N consecutive `:degraded` reports ⇒ drain and restart (counts toward intensity) |
|
|
67
|
+
|
|
68
|
+
Adapters:
|
|
69
|
+
|
|
70
|
+
- **`:command`** — any command. `cmd:` (required), `env:`, `spawn_opts:`, and optional
|
|
71
|
+
`probe: { tcp: PORT }` or `probe: { http: "http://127.0.0.1:3000/up" }` — the child is
|
|
72
|
+
`:starting` until the probe answers, `:degraded` if it stops answering later.
|
|
73
|
+
- **`:puma`** (beside mode) — wraps the puma master. `config:` (default `config/puma.rb`),
|
|
74
|
+
`port:` (or a literal `port NNNN` line in the config). Health = HTTP probe of `/up`;
|
|
75
|
+
drain = SIGTERM (Puma's graceful stop).
|
|
76
|
+
- **`:solid_queue`** — wraps `bin/jobs` (`cmd:` overrides). Health = the active heartbeat
|
|
77
|
+
below, never the `solid_queue_processes` table.
|
|
78
|
+
|
|
79
|
+
For worker-level visibility on a cluster-mode `:puma` child, add the plugin to
|
|
80
|
+
`config/puma.rb`:
|
|
81
|
+
|
|
82
|
+
```ruby
|
|
83
|
+
plugin :odoshi
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
The master then heartbeats worker state over the socket: any missing worker is reported
|
|
87
|
+
`"degraded"` (⇒ `[:odoshi, :child, :degraded]` telemetry, `meta: {workers:, booted:,
|
|
88
|
+
phase:}`) while puma replaces the worker itself — visibility only, no lifecycle change.
|
|
89
|
+
Both adapters export `ODOSHI_CHILD_ID` so plugins and hooks heartbeat under the right id.
|
|
90
|
+
|
|
91
|
+
## Health & heartbeats
|
|
92
|
+
|
|
93
|
+
Passive children are probed (PID, TCP, HTTP). Active children report themselves: the
|
|
94
|
+
supervisor listens on a Unix socket (mode 0600) and exports `ODOSHI_SOCK` /
|
|
95
|
+
`ODOSHI_TOKEN` to every child. Heartbeats are newline-delimited JSON:
|
|
96
|
+
|
|
97
|
+
```json
|
|
98
|
+
{"id":"jobs","state":"healthy","ts":1757700000,"token":"…","meta":{"backlog":0}}
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
A child that has heartbeated is judged by heartbeat freshness: 3 missed `health_interval`s
|
|
102
|
+
⇒ `:degraded`, 6 ⇒ `:dead` ⇒ the strategy applies. Wrong token, non-string fields, or
|
|
103
|
+
lines over 64 KiB ⇒ silently dropped. The same socket accepts
|
|
104
|
+
`{"cmd":"restart","id":"jobs","token":"…"}` — a control restart is deliberate remediation
|
|
105
|
+
(DESIGN §7: restarting is a feature), so it does not count toward restart intensity.
|
|
106
|
+
|
|
107
|
+
From any child process (a Rails initializer, a Solid Queue hook — no Rails required):
|
|
108
|
+
|
|
109
|
+
```ruby
|
|
110
|
+
require "odoshi/heartbeat" # loads nothing else
|
|
111
|
+
Odoshi::Heartbeat.start(id: "jobs") # no-op when running unsupervised
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
## Telemetry reference
|
|
115
|
+
|
|
116
|
+
Event names follow `[:odoshi, :subject, :action]`, mirroring Elixir `:telemetry` so the
|
|
117
|
+
sidecar can forward them unchanged. This list is a published contract:
|
|
118
|
+
|
|
119
|
+
```
|
|
120
|
+
[:odoshi, :supervisor, :start] metadata: {strategy, children}
|
|
121
|
+
[:odoshi, :supervisor, :stop]
|
|
122
|
+
[:odoshi, :supervisor, :escalate] measurements: {restarts} metadata: {within}
|
|
123
|
+
[:odoshi, :child, :spawn] metadata: {id, adapter, pid}
|
|
124
|
+
[:odoshi, :child, :healthy] metadata: {id}
|
|
125
|
+
[:odoshi, :child, :degraded] measurements: {consecutive} metadata: {id}
|
|
126
|
+
[:odoshi, :child, :exit] measurements: {exit_code, uptime_ms} metadata: {id}
|
|
127
|
+
[:odoshi, :child, :restart] measurements: {backoff_ms} metadata: {id, attempt, strategy}
|
|
128
|
+
[:odoshi, :child, :drain] metadata: {id}
|
|
129
|
+
[:odoshi, :child, :kill] metadata: {id} (drain timed out)
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
Subscribe in-process with `Odoshi::Telemetry.subscribe { |event| ... }`; a logger
|
|
133
|
+
subscriber and a JSON-lines exporter ship by default (`Telemetry::Subscribers`).
|
|
134
|
+
|
|
135
|
+
## Exit codes
|
|
136
|
+
|
|
137
|
+
| Code | Meaning |
|
|
138
|
+
|---|---|
|
|
139
|
+
| `0` | Clean stop (SIGINT/SIGTERM, all children drained) |
|
|
140
|
+
| `70` | Restart intensity exceeded — the supervisor escalated (EX_SOFTWARE). The platform (Kamal/K8s/Heroku/launchd) is the final supervisor and should restart on non-zero |
|
|
141
|
+
| `78` | Configuration error (EX_CONFIG) |
|
|
142
|
+
|
|
143
|
+
## Shutdown semantics
|
|
144
|
+
|
|
145
|
+
- `stop_all` drains children in **reverse start order**, waiting for each child to exit
|
|
146
|
+
(up to its `shutdown:` timeout, then SIGKILL of its process group) before draining the next.
|
|
147
|
+
- Orphan prevention: on Linux, children are armed with `prctl(PR_SET_PDEATHSIG, SIGTERM)`
|
|
148
|
+
between fork and exec, so they receive SIGTERM even if the supervisor is SIGKILLed.
|
|
149
|
+
**macOS/BSD limitation:** no parent-death signal exists there; a SIGKILLed supervisor
|
|
150
|
+
orphans its children to launchd/init and they keep running. Mitigation: the platform
|
|
151
|
+
restarts the supervisor; each child runs in its own session/process group, so stale
|
|
152
|
+
orphans are findable and killable by pgid.
|
|
153
|
+
- macOS also caps Unix socket paths at ~104 bytes — keep `socket PATH` short.
|
|
154
|
+
- Compound commands (`cmd: "a && b"`) run under an `sh` wrapper. Drain and kill signal the
|
|
155
|
+
whole process group, so supervised shutdown covers the real workload — but SIGKILL-of-
|
|
156
|
+
the-supervisor orphan prevention (pdeathsig) arms only the wrapper: on Linux the wrapper
|
|
157
|
+
gets SIGTERM and its children are orphaned. Prefer single-exec commands for children
|
|
158
|
+
that must never outlive the supervisor.
|
|
159
|
+
|
|
160
|
+
## Development
|
|
161
|
+
|
|
162
|
+
```
|
|
163
|
+
bundle exec rake test # full suite (real processes, no mocks)
|
|
164
|
+
ruby -Ilib -Itest test/supervisor_kill_test.rb
|
|
165
|
+
exe/odoshi check examples/supervisor.rb
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
`docs/PLAN.md` is the backlog; `docs/DESIGN.md` is the frozen design.
|
data/exe/odoshi
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
module Odoshi
|
|
3
|
+
# DESIGN §4. Exactly four operations (+ kill as last resort). Adapters never talk to each other.
|
|
4
|
+
class Adapter
|
|
5
|
+
REGISTRY = {}
|
|
6
|
+
|
|
7
|
+
def self.register(name, klass) = REGISTRY[name] = klass
|
|
8
|
+
def self.lookup(name) = REGISTRY.fetch(name) { raise ConfigError, "no adapter registered as #{name.inspect}" }
|
|
9
|
+
|
|
10
|
+
# @return [Object] opaque handle
|
|
11
|
+
def spawn(spec) = raise NotImplementedError
|
|
12
|
+
# Register a one-shot callback: block.call(exit_status)
|
|
13
|
+
def link(handle, &on_exit) = raise NotImplementedError
|
|
14
|
+
# @return [:starting, :healthy, :degraded, :dead]
|
|
15
|
+
def health(handle) = raise NotImplementedError
|
|
16
|
+
# Stop accepting work, finish in-flight, exit. Return true if exited within timeout.
|
|
17
|
+
def drain(handle, timeout:) = raise NotImplementedError
|
|
18
|
+
# Last resort after drain times out.
|
|
19
|
+
def kill(handle) = raise NotImplementedError
|
|
20
|
+
end
|
|
21
|
+
end
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
module Odoshi
|
|
3
|
+
module Adapters
|
|
4
|
+
# DESIGN §4.1 :command — arbitrary command. Health = PID alive (+ optional probe, TODO).
|
|
5
|
+
class Command < Adapter
|
|
6
|
+
Handle = Struct.new(:pid, :spec, :started_at, :exit_status, :waiter, :healthy_once)
|
|
7
|
+
|
|
8
|
+
def spawn(spec)
|
|
9
|
+
cmd = spec.opts.fetch(:cmd) { raise ConfigError, "#{spec.id}: :command adapter requires cmd:" }
|
|
10
|
+
env = spec.opts.fetch(:env, {}).transform_keys(&:to_s)
|
|
11
|
+
spawn_opts = spec.opts.fetch(:spawn_opts, {})
|
|
12
|
+
# One spawn path on every platform (#15): fork → setsid → exec, with
|
|
13
|
+
# exec failure becoming exit 127 through the normal link → crash →
|
|
14
|
+
# strategy machinery. Process.spawn raised Errno::ENOENT into the
|
|
15
|
+
# supervisor loop for a bad cmd on macOS (whole tree crashed, exit 1)
|
|
16
|
+
# while the Linux fork path restart-looped to escalation (exit 70).
|
|
17
|
+
parent = Process.pid
|
|
18
|
+
pid = Process.fork do
|
|
19
|
+
Process.setsid # own session ⇒ own pgroup (drain/kill signal the group)
|
|
20
|
+
OrphanGuard.arm!(parent)
|
|
21
|
+
begin
|
|
22
|
+
Process.exec(env, cmd, **spawn_opts)
|
|
23
|
+
rescue StandardError
|
|
24
|
+
Process.exit!(127)
|
|
25
|
+
end
|
|
26
|
+
end
|
|
27
|
+
Handle.new(pid, spec, Process.clock_gettime(Process::CLOCK_MONOTONIC), nil, nil)
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
def link(handle, &on_exit)
|
|
31
|
+
handle.waiter = Thread.new do
|
|
32
|
+
_, status = Process.wait2(handle.pid)
|
|
33
|
+
handle.exit_status = status
|
|
34
|
+
on_exit.call(status)
|
|
35
|
+
rescue Errno::ECHILD
|
|
36
|
+
on_exit.call(nil)
|
|
37
|
+
end
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
def health(handle)
|
|
41
|
+
return :dead if handle.exit_status
|
|
42
|
+
Process.kill(0, handle.pid)
|
|
43
|
+
if Probe.answering?(handle.spec)
|
|
44
|
+
handle.healthy_once = true
|
|
45
|
+
:healthy
|
|
46
|
+
elsif handle.healthy_once
|
|
47
|
+
:degraded # was healthy, probe stopped answering, PID still alive (§5)
|
|
48
|
+
else
|
|
49
|
+
:starting
|
|
50
|
+
end
|
|
51
|
+
rescue Errno::ESRCH
|
|
52
|
+
:dead
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
def drain(handle, timeout:)
|
|
56
|
+
# Signal the whole process group, not just handle.pid: children are
|
|
57
|
+
# group leaders (setsid / pgroup: true), and a shell-wrapped cmd
|
|
58
|
+
# ("a && b") is an sh wrapper whose real workload is a grandchild in
|
|
59
|
+
# that group — TERM to the wrapper alone orphans the workload while
|
|
60
|
+
# reporting a clean drain (issue #24).
|
|
61
|
+
group_signal("TERM", handle.pid)
|
|
62
|
+
handle.waiter&.join(timeout)
|
|
63
|
+
!handle.exit_status.nil?
|
|
64
|
+
rescue Errno::ESRCH
|
|
65
|
+
true
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
def kill(handle)
|
|
69
|
+
Process.kill("KILL", -handle.pid) # whole process group
|
|
70
|
+
handle.waiter&.join(2)
|
|
71
|
+
rescue Errno::ESRCH
|
|
72
|
+
nil
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
private
|
|
76
|
+
|
|
77
|
+
# TERM the group; fall back to the pid alone if the group is already
|
|
78
|
+
# gone by the time we signal (pure pid death races to the outer ESRCH).
|
|
79
|
+
def group_signal(sig, pid)
|
|
80
|
+
Process.kill(sig, -pid)
|
|
81
|
+
rescue Errno::ESRCH
|
|
82
|
+
Process.kill(sig, pid)
|
|
83
|
+
end
|
|
84
|
+
end
|
|
85
|
+
end
|
|
86
|
+
Adapter.register(:command, Adapters::Command)
|
|
87
|
+
end
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
module Odoshi
|
|
3
|
+
module Adapters
|
|
4
|
+
# DESIGN §4.1 / §4.2 step 1 — :puma in "beside" mode: one opaque child
|
|
5
|
+
# wrapping the puma master. Health = HTTP probe of /up (the Rails 7.1+
|
|
6
|
+
# default endpoint) on the bound port; drain = SIGTERM, which is Puma's
|
|
7
|
+
# graceful shutdown — exactly Command#drain, so it is inherited unchanged,
|
|
8
|
+
# as are link and kill.
|
|
9
|
+
#
|
|
10
|
+
# The whole adapter is Command with a derived spec: spawn builds the puma
|
|
11
|
+
# command line and injects a probe: { http: ".../up" } opt, so health()
|
|
12
|
+
# rides the PLAN 1.2 Probe path with no new code — :starting until /up
|
|
13
|
+
# answers 2xx, then :healthy; :dead when the PID goes.
|
|
14
|
+
#
|
|
15
|
+
# opts:
|
|
16
|
+
# config: puma config file path (default "config/puma.rb")
|
|
17
|
+
# port: the bound port. When absent, a literal `port NNNN` line is
|
|
18
|
+
# parsed from the config file; neither ⇒ ConfigError.
|
|
19
|
+
# env:, spawn_opts: passed through to Command verbatim.
|
|
20
|
+
# (cmd: and probe: are owned by this adapter and overwritten.)
|
|
21
|
+
class Puma < Command
|
|
22
|
+
DEFAULT_CONFIG = "config/puma.rb"
|
|
23
|
+
HEALTH_PATH = "/up"
|
|
24
|
+
|
|
25
|
+
def spawn(spec)
|
|
26
|
+
super(command_spec(spec))
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
private
|
|
30
|
+
|
|
31
|
+
def command_spec(spec)
|
|
32
|
+
config = spec.opts.fetch(:config, DEFAULT_CONFIG)
|
|
33
|
+
port = resolve_port(spec, config)
|
|
34
|
+
ChildSpec.new(
|
|
35
|
+
id: spec.id, adapter: spec.adapter, restart: spec.restart,
|
|
36
|
+
shutdown: spec.shutdown, start_timeout: spec.start_timeout,
|
|
37
|
+
opts: spec.opts.merge(
|
|
38
|
+
cmd: "bundle exec puma -C #{config}",
|
|
39
|
+
probe: { http: "http://127.0.0.1:#{port}#{HEALTH_PATH}" },
|
|
40
|
+
# Tag the child so the odoshi puma plugin (and any app hook)
|
|
41
|
+
# heartbeats under the right id; explicit env still wins.
|
|
42
|
+
env: { "ODOSHI_CHILD_ID" => spec.id.to_s }.merge(spec.opts.fetch(:env, {}))
|
|
43
|
+
)
|
|
44
|
+
)
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
# opts[:port] wins; else a literal `port NNNN` line in the config file.
|
|
48
|
+
# Anything fancier (ENV/ERB in the config) must pass opts[:port].
|
|
49
|
+
def resolve_port(spec, config)
|
|
50
|
+
return Integer(spec.opts[:port]) if spec.opts[:port]
|
|
51
|
+
literal = File.file?(config) && File.read(config)[/^\s*port\s+(\d+)/, 1]
|
|
52
|
+
return Integer(literal) if literal
|
|
53
|
+
raise ConfigError,
|
|
54
|
+
"#{spec.id}: :puma needs opts[:port] or a literal `port NNNN` line in #{config}"
|
|
55
|
+
end
|
|
56
|
+
end
|
|
57
|
+
end
|
|
58
|
+
Adapter.register(:puma, Adapters::Puma)
|
|
59
|
+
end
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
module Odoshi
|
|
3
|
+
module Adapters
|
|
4
|
+
# DESIGN §4.1 / §9 — :solid_queue wraps the Solid Queue supervisor
|
|
5
|
+
# (`bin/jobs`). Health is the ACTIVE heartbeat (§5), NOT the
|
|
6
|
+
# solid_queue_processes table: the app sends heartbeats via the tiny
|
|
7
|
+
# Rails-free hook, e.g. from an initializer:
|
|
8
|
+
#
|
|
9
|
+
# require "odoshi/heartbeat"
|
|
10
|
+
# Odoshi::Heartbeat.start(id: "jobs")
|
|
11
|
+
#
|
|
12
|
+
# Once the first heartbeat arrives the supervisor judges the child by
|
|
13
|
+
# heartbeat freshness (3 missed intervals ⇒ :degraded, 6 ⇒ :dead);
|
|
14
|
+
# until then it falls back to this adapter's passive PID health,
|
|
15
|
+
# inherited from Command — as are link, drain (SIGTERM, which Solid
|
|
16
|
+
# Queue handles gracefully), and kill.
|
|
17
|
+
#
|
|
18
|
+
# opts:
|
|
19
|
+
# cmd: the jobs command (default "bin/jobs")
|
|
20
|
+
# env:, spawn_opts: passed through to Command verbatim.
|
|
21
|
+
class SolidQueue < Command
|
|
22
|
+
DEFAULT_CMD = "bin/jobs"
|
|
23
|
+
|
|
24
|
+
def spawn(spec)
|
|
25
|
+
super(command_spec(spec))
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
private
|
|
29
|
+
|
|
30
|
+
def command_spec(spec)
|
|
31
|
+
ChildSpec.new(
|
|
32
|
+
id: spec.id, adapter: spec.adapter, restart: spec.restart,
|
|
33
|
+
shutdown: spec.shutdown, start_timeout: spec.start_timeout,
|
|
34
|
+
opts: spec.opts.merge(
|
|
35
|
+
cmd: spec.opts.fetch(:cmd, DEFAULT_CMD),
|
|
36
|
+
# Tag the child so the Heartbeat hook picks up its id from env;
|
|
37
|
+
# explicit env still wins.
|
|
38
|
+
env: { "ODOSHI_CHILD_ID" => spec.id.to_s }.merge(spec.opts.fetch(:env, {}))
|
|
39
|
+
)
|
|
40
|
+
)
|
|
41
|
+
end
|
|
42
|
+
end
|
|
43
|
+
end
|
|
44
|
+
Adapter.register(:solid_queue, Adapters::SolidQueue)
|
|
45
|
+
end
|
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
module Odoshi
|
|
3
|
+
module Adapters
|
|
4
|
+
# DESIGN §3.1 nested supervisors. A subtree is just another child of the
|
|
5
|
+
# parent: this pseudo-adapter runs a child Supervisor on a Thread and maps
|
|
6
|
+
# the four adapter operations onto it. Escalation inside the subtree
|
|
7
|
+
# terminates the thread, which surfaces to the parent as a crashed exit —
|
|
8
|
+
# the parent then applies ITS strategy/intensity to the subtree child
|
|
9
|
+
# (restart the whole subtree, etc.), exactly per OTP semantics.
|
|
10
|
+
#
|
|
11
|
+
# opts:
|
|
12
|
+
# builder: a Proc returning a FRESH Odoshi::Supervisor each call.
|
|
13
|
+
# A restarted subtree must not inherit the old subtree's
|
|
14
|
+
# RestartIntensity window (it is stateful), so spawn re-invokes
|
|
15
|
+
# the builder on every (re)start instead of reusing an instance.
|
|
16
|
+
class SupervisorAdapter < Adapter
|
|
17
|
+
# Duck-types the two things the tree asks of an exit status:
|
|
18
|
+
# #exitstatus (telemetry) and #success? (ChildSpec#restart? semantics).
|
|
19
|
+
Status = Struct.new(:exitstatus) do
|
|
20
|
+
def success? = exitstatus == 0
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
Handle = Struct.new(:sub, :thread, :started_at, :exit_status, :waiter, :killed)
|
|
24
|
+
|
|
25
|
+
def spawn(spec)
|
|
26
|
+
builder = spec.opts.fetch(:builder) do
|
|
27
|
+
raise ConfigError, "#{spec.id}: :supervisor adapter requires builder: (a proc returning a fresh Supervisor)"
|
|
28
|
+
end
|
|
29
|
+
sub = builder.call
|
|
30
|
+
raise ConfigError, "#{spec.id}: builder must return an Odoshi::Supervisor" unless sub.is_a?(Supervisor)
|
|
31
|
+
thread = Thread.new { sub.run }
|
|
32
|
+
# Escalation out of a subtree is an expected, handled exit path — the
|
|
33
|
+
# waiter converts it into a crashed status. Don't let Ruby dump it.
|
|
34
|
+
thread.report_on_exception = false
|
|
35
|
+
Handle.new(sub, thread, Process.clock_gettime(Process::CLOCK_MONOTONIC), nil, nil, false)
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
def link(handle, &on_exit)
|
|
39
|
+
handle.waiter = Thread.new do
|
|
40
|
+
status =
|
|
41
|
+
begin
|
|
42
|
+
handle.thread.join # re-raises whatever terminated the subtree
|
|
43
|
+
Status.new(handle.killed ? nil : 0)
|
|
44
|
+
rescue Escalation
|
|
45
|
+
Status.new(70) # crashed: intensity exceeded inside the subtree
|
|
46
|
+
rescue StandardError
|
|
47
|
+
Status.new(1) # crashed: unexpected error in the subtree loop
|
|
48
|
+
end
|
|
49
|
+
handle.exit_status = status
|
|
50
|
+
on_exit.call(status)
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
# The subtree's internal health is its own supervisor's business; from
|
|
55
|
+
# the parent's seat the subtree is healthy while its loop is running.
|
|
56
|
+
# Death also arrives via link, so parents normally give subtree specs
|
|
57
|
+
# health_interval: nil (the DSL does) and skip the probe monitor.
|
|
58
|
+
def health(handle)
|
|
59
|
+
handle.thread.alive? ? :healthy : :dead
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
# Clean stop: the subtree's run loop breaks and its `ensure stop_all`
|
|
63
|
+
# drains the subtree's own children (reverse order) before the thread
|
|
64
|
+
# exits — so by the time this returns true, no subtree PIDs remain.
|
|
65
|
+
def drain(handle, timeout:)
|
|
66
|
+
handle.sub.stop
|
|
67
|
+
joined =
|
|
68
|
+
begin
|
|
69
|
+
handle.thread.join(timeout)
|
|
70
|
+
rescue StandardError
|
|
71
|
+
handle.thread # join re-raised => the thread has terminated
|
|
72
|
+
end
|
|
73
|
+
!joined.nil?
|
|
74
|
+
end
|
|
75
|
+
|
|
76
|
+
# Last resort. Thread#kill still runs the subtree's `ensure stop_all`;
|
|
77
|
+
# if even that wedges, best-effort drain the subtree's children directly
|
|
78
|
+
# so no grandchild PIDs are orphaned.
|
|
79
|
+
def kill(handle)
|
|
80
|
+
handle.killed = true
|
|
81
|
+
handle.thread.kill
|
|
82
|
+
joined =
|
|
83
|
+
begin
|
|
84
|
+
handle.thread.join(2)
|
|
85
|
+
rescue StandardError
|
|
86
|
+
handle.thread
|
|
87
|
+
end
|
|
88
|
+
handle.sub.send(:stop_all) if joined.nil? # loop wedged: drain grandchildren ourselves
|
|
89
|
+
rescue StandardError
|
|
90
|
+
nil
|
|
91
|
+
end
|
|
92
|
+
end
|
|
93
|
+
end
|
|
94
|
+
Adapter.register(:supervisor, Adapters::SupervisorAdapter)
|
|
95
|
+
end
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
module Odoshi
|
|
3
|
+
class Backoff
|
|
4
|
+
def initialize(kind: :exponential, base: 1.0, cap: 30.0)
|
|
5
|
+
@kind, @base, @cap = kind, base.to_f, cap.to_f
|
|
6
|
+
end
|
|
7
|
+
|
|
8
|
+
# attempt is 1-based
|
|
9
|
+
def delay(attempt)
|
|
10
|
+
case @kind
|
|
11
|
+
when :none then 0.0
|
|
12
|
+
when :constant then @base
|
|
13
|
+
when :exponential then [@base * (2**(attempt - 1)), @cap].min
|
|
14
|
+
else raise ConfigError, "unknown backoff #{@kind}"
|
|
15
|
+
end
|
|
16
|
+
end
|
|
17
|
+
end
|
|
18
|
+
end
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
module Odoshi
|
|
3
|
+
# DESIGN §3.2
|
|
4
|
+
class ChildSpec
|
|
5
|
+
RESTART_KINDS = %i[permanent transient temporary].freeze
|
|
6
|
+
|
|
7
|
+
attr_reader :id, :adapter, :restart, :shutdown, :start_timeout, :health_interval,
|
|
8
|
+
:degraded_restart_after, :opts
|
|
9
|
+
|
|
10
|
+
def initialize(id:, adapter:, restart: :permanent, shutdown: 30, start_timeout: 30,
|
|
11
|
+
health_interval: 5, degraded_restart_after: nil, opts: {})
|
|
12
|
+
raise ConfigError, "child id must be a Symbol" unless id.is_a?(Symbol)
|
|
13
|
+
raise ConfigError, "restart must be one of #{RESTART_KINDS}" unless RESTART_KINDS.include?(restart)
|
|
14
|
+
@id, @adapter, @restart, @shutdown, @start_timeout, @opts =
|
|
15
|
+
id, adapter, restart, shutdown, start_timeout, opts
|
|
16
|
+
@health_interval, @degraded_restart_after = health_interval, degraded_restart_after
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
# Should this child be restarted given how it exited? (OTP semantics)
|
|
20
|
+
def restart?(exit_status)
|
|
21
|
+
case restart
|
|
22
|
+
when :permanent then true
|
|
23
|
+
when :transient then !normal_exit?(exit_status)
|
|
24
|
+
when :temporary then false
|
|
25
|
+
end
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
def normal_exit?(status)
|
|
29
|
+
return false if status.nil?
|
|
30
|
+
status.respond_to?(:success?) ? status.success? : status.to_i.zero?
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
end
|