@o3co/create-auth-policy-verifier 0.7.0 → 0.8.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +1 -1
- package/templates/standalone/README.ja.md +7 -1
- package/templates/standalone/README.md +7 -1
- package/templates/standalone/package.json +1 -2
- package/templates/standalone/src/__tests__/shutdown.test.mts +275 -0
- package/templates/standalone/src/main.mts +2 -2
- package/templates/standalone/src/shutdown.ts +202 -0
- package/templates/versions.json +3 -3
package/package.json
CHANGED
|
@@ -95,7 +95,13 @@ Authorization: Bearer <jwt>
|
|
|
95
95
|
|
|
96
96
|
それ以外は body のパースより前に `401 { "decision": "deny", "code": "caller_unauthenticated", "message": "Caller authentication failed" }` を返します。`GET /_healthcheck` は常に非ゲートなので、コンテナの healthcheck はそのまま動作します。
|
|
97
97
|
|
|
98
|
-
資格情報を未設定のままにする運用もサポートされており、ループバックであれば問題ありません。ループバック以外に bind した場合は起動時に `unauthenticated_non_loopback_bind` が warn
|
|
98
|
+
資格情報を未設定のままにする運用もサポートされており、ループバックであれば問題ありません。ループバック以外に bind した場合は起動時に `unauthenticated_non_loopback_bind` が warn で記録され、関係する 2 つの設定と何が晒されているかが示されます — 修正する価値があるのはこの組み合わせです。起動を拒否せず warn に留めているのは、ネットワーク側が正当な制御になっている場合があり、拒否すればアップグレード時に既存のコンテナデプロイをすべて壊すためです。
|
|
99
|
+
|
|
100
|
+
**このテンプレートは初期状態でまさにその組み合わせです。** `docker-compose.yml` が `HTTP_HOSTNAME=0.0.0.0` を設定する一方、`HTTP_CALLER_AUTH_TOKEN` はどこにも設定されていません。したがって publish したポートがホスト外から到達できる状態で `docker compose up` すると、未認証の判定エンドポイントが公開されます。ローカル開発では問題なく、だからこそ warn に留めています。実トラフィックを流す前に、次のいずれかを行ってください。
|
|
101
|
+
|
|
102
|
+
- **ポートをプライベートネットワークに置く。** セキュリティグループ、プライベートサブネット、Kubernetes の `NetworkPolicy` など、誰が接続してよいかを決める仕組みです。compose の `3000:3000` の publish はこれに該当しません — そのホストに到達できるすべてに到達を許します。
|
|
103
|
+
- **`.env` に `HTTP_CALLER_AUTH_TOKEN` を設定する。** ポートに到達できるだけでは判定を要求できなくなります。攻撃者が既にネットワーク境界の内側にいる場合でも有効なのは、2 つのうちこちらだけです。なお Go の enforcement 層([protobuf.interceptors](https://github.com/o3co/protobuf.interceptors))がこのヘッダを送るオプションを備えるのは**次のリリースから**です。それまで、同ライブラリを使う呼び出し元にはネットワーク側の制御が必要です。
|
|
104
|
+
- **あるいは `HTTP_HOSTNAME` を外し、verifier をサイドカーとして動かす。** enforcement 層とネットワーク名前空間を共有すれば、ループバックで足ります。
|
|
99
105
|
|
|
100
106
|
`GET /metrics` も非ゲートです。理由と、ループバック以外に bind したときに何を意味するかは [`/metrics` への到達方法](#metrics-への到達方法)を参照してください。
|
|
101
107
|
|
|
@@ -96,7 +96,13 @@ Authorization: Bearer <jwt>
|
|
|
96
96
|
|
|
97
97
|
Anything else gets `401 { "decision": "deny", "code": "caller_unauthenticated", "message": "Caller authentication failed" }`, decided before the body is parsed. `GET /_healthcheck` is never gated, so the container healthcheck keeps working.
|
|
98
98
|
|
|
99
|
-
Leaving the credential unset is supported and is fine on loopback. On a non-loopback bind the server logs `unauthenticated_non_loopback_bind` at warn on boot — that combination is the one worth fixing.
|
|
99
|
+
Leaving the credential unset is supported and is fine on loopback. On a non-loopback bind the server logs `unauthenticated_non_loopback_bind` at warn on boot, naming both settings and the exposure — that combination is the one worth fixing. It warns rather than refusing to boot: the network may legitimately be the control, and refusing would break every existing containerised deployment on upgrade.
|
|
100
|
+
|
|
101
|
+
**This template is that combination out of the box.** `HTTP_HOSTNAME=0.0.0.0` is set in `docker-compose.yml` and `HTTP_CALLER_AUTH_TOKEN` is not set anywhere, so a `docker compose up` with the published port reachable from off-host is an unauthenticated decision endpoint. That is fine for local development and is why the warning is a warning. Before it carries traffic, do one of:
|
|
102
|
+
|
|
103
|
+
- **Put the port on a private network.** A security group, a private subnet, a Kubernetes `NetworkPolicy` — something that decides who may connect. Publishing `3000:3000` from compose is not this: it reaches everything that can route to the host.
|
|
104
|
+
- **Set `HTTP_CALLER_AUTH_TOKEN` in `.env`.** Then reaching the port is not enough to ask for a decision, which is the only one of the two that still holds against an attacker already inside the network boundary. Note that the Go enforcement layer ([protobuf.interceptors](https://github.com/o3co/protobuf.interceptors)) gains the option to send this header **from its next release**; until then a caller built on it needs the network control instead.
|
|
105
|
+
- **Or drop `HTTP_HOSTNAME` and run the verifier as a sidecar**, sharing a network namespace with the enforcement layer so loopback is enough.
|
|
100
106
|
|
|
101
107
|
`GET /metrics` is not gated either — see [Reaching `/metrics`](#reaching-metrics) for why, and for what that means when the bind is not loopback.
|
|
102
108
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@o3co/auth-policy-verifier-standalone",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.8.1",
|
|
4
4
|
"private": true,
|
|
5
5
|
"license": "Apache-2.0",
|
|
6
6
|
"packageManager": "pnpm@10.29.3",
|
|
@@ -19,7 +19,6 @@
|
|
|
19
19
|
"@o3co/auth.policy-verifier.core": "workspace:*",
|
|
20
20
|
"@o3co/auth.policy-verifier.builtins": "workspace:*",
|
|
21
21
|
"@o3co/auth.policy-verifier.server": "workspace:*",
|
|
22
|
-
"@o3co/auth.utils": "^0.0.4",
|
|
23
22
|
"@o3co/ts.hocon": "^1.13.0",
|
|
24
23
|
"express": "^5.2.1",
|
|
25
24
|
"pino": "^10.3.1"
|
|
@@ -0,0 +1,275 @@
|
|
|
1
|
+
// SPDX-FileCopyrightText: 2026 1o1 Co. Ltd.
|
|
2
|
+
// SPDX-License-Identifier: Apache-2.0
|
|
3
|
+
|
|
4
|
+
/**
|
|
5
|
+
* Shutdown was delegated to `@o3co/auth.utils@0.0.4`, whose guarantees no
|
|
6
|
+
* contract in this repository pinned.
|
|
7
|
+
*
|
|
8
|
+
* Reading its 22 lines answered the question and the answer was the reason to
|
|
9
|
+
* move it: **there was no force-close deadline**. `server.close()` waits for
|
|
10
|
+
* in-flight requests indefinitely, so one stuck decision meant the process
|
|
11
|
+
* never exited on its own and the orchestrator's SIGKILL took it down
|
|
12
|
+
* mid-flight. Its cleanup-failure path also wrote to `console.error`, a bare
|
|
13
|
+
* line in a composition root whose every other line is NDJSON (#107), and
|
|
14
|
+
* every exit was zero.
|
|
15
|
+
*
|
|
16
|
+
* The behaviour lives in the template now, with a deadline and the app's own
|
|
17
|
+
* logger, and these tests are the contract that was missing. `auth.provider`'s
|
|
18
|
+
* standalone template made the same move in its issue #290.
|
|
19
|
+
*/
|
|
20
|
+
import type { Server } from "node:http";
|
|
21
|
+
import type { Logger } from "@o3co/auth.policy-verifier.core";
|
|
22
|
+
import { describe, expect, it, vi } from "vitest";
|
|
23
|
+
import { deferExit, installGracefulShutdown } from "../shutdown.js";
|
|
24
|
+
|
|
25
|
+
/** A `Server` double whose `close` callback fires only when we say so. */
|
|
26
|
+
function makeServer() {
|
|
27
|
+
let closeCallback: ((err?: Error) => void) | undefined;
|
|
28
|
+
const server = {
|
|
29
|
+
close: vi.fn((cb?: (err?: Error) => void) => {
|
|
30
|
+
closeCallback = cb;
|
|
31
|
+
return server;
|
|
32
|
+
}),
|
|
33
|
+
closeIdleConnections: vi.fn(),
|
|
34
|
+
closeAllConnections: vi.fn(),
|
|
35
|
+
};
|
|
36
|
+
return {
|
|
37
|
+
server: server as unknown as Server,
|
|
38
|
+
spies: server,
|
|
39
|
+
finishDraining: () => closeCallback?.(),
|
|
40
|
+
failClose: (err: Error) => closeCallback?.(err),
|
|
41
|
+
};
|
|
42
|
+
}
|
|
43
|
+
|
|
44
|
+
/** A `Logger`-shaped spy, typed so a future port method cannot slip past. */
|
|
45
|
+
const makeLogger = () => {
|
|
46
|
+
const spy = {
|
|
47
|
+
trace: vi.fn(),
|
|
48
|
+
debug: vi.fn(),
|
|
49
|
+
info: vi.fn(),
|
|
50
|
+
warn: vi.fn(),
|
|
51
|
+
error: vi.fn(),
|
|
52
|
+
fatal: vi.fn(),
|
|
53
|
+
child: vi.fn(() => spy as unknown as Logger),
|
|
54
|
+
};
|
|
55
|
+
return spy satisfies Logger;
|
|
56
|
+
};
|
|
57
|
+
|
|
58
|
+
function install(opts: { cleanup?: () => void | Promise<void>; drainTimeoutMs?: number } = {}) {
|
|
59
|
+
const { server, spies, finishDraining, failClose } = makeServer();
|
|
60
|
+
const logger = makeLogger();
|
|
61
|
+
const exit = vi.fn();
|
|
62
|
+
const signals = new Map<string, () => void>();
|
|
63
|
+
|
|
64
|
+
installGracefulShutdown(server, {
|
|
65
|
+
logger,
|
|
66
|
+
cleanup: opts.cleanup ?? (() => {}),
|
|
67
|
+
...(opts.drainTimeoutMs === undefined ? {} : { drainTimeoutMs: opts.drainTimeoutMs }),
|
|
68
|
+
exit,
|
|
69
|
+
onSignal: (name, handler) => signals.set(name, handler),
|
|
70
|
+
offSignal: (name) => signals.delete(name),
|
|
71
|
+
});
|
|
72
|
+
|
|
73
|
+
return { spies, logger, exit, signals, finishDraining, failClose };
|
|
74
|
+
}
|
|
75
|
+
|
|
76
|
+
const settle = () => new Promise((resolve) => setImmediate(resolve));
|
|
77
|
+
|
|
78
|
+
describe("installGracefulShutdown", () => {
|
|
79
|
+
it("listens for both SIGTERM and SIGINT", () => {
|
|
80
|
+
expect([...install().signals.keys()].sort()).toEqual(["SIGINT", "SIGTERM"]);
|
|
81
|
+
});
|
|
82
|
+
|
|
83
|
+
it("stops accepting connections and releases idle keep-alive sockets", () => {
|
|
84
|
+
const { signals, spies } = install();
|
|
85
|
+
signals.get("SIGTERM")?.();
|
|
86
|
+
expect(spies.close).toHaveBeenCalledOnce();
|
|
87
|
+
expect(spies.closeIdleConnections).toHaveBeenCalledOnce();
|
|
88
|
+
});
|
|
89
|
+
|
|
90
|
+
it("runs cleanup once draining completes, then exits zero", async () => {
|
|
91
|
+
const cleanup = vi.fn();
|
|
92
|
+
const { signals, finishDraining, exit } = install({ cleanup });
|
|
93
|
+
signals.get("SIGTERM")?.();
|
|
94
|
+
expect(cleanup).not.toHaveBeenCalled();
|
|
95
|
+
finishDraining();
|
|
96
|
+
await settle();
|
|
97
|
+
expect(cleanup).toHaveBeenCalledOnce();
|
|
98
|
+
expect(exit).toHaveBeenCalledWith(0);
|
|
99
|
+
});
|
|
100
|
+
|
|
101
|
+
it("ignores a second signal instead of running cleanup twice", async () => {
|
|
102
|
+
const cleanup = vi.fn();
|
|
103
|
+
const { signals, spies, finishDraining, exit } = install({ cleanup });
|
|
104
|
+
const handler = signals.get("SIGTERM");
|
|
105
|
+
handler?.();
|
|
106
|
+
handler?.();
|
|
107
|
+
finishDraining();
|
|
108
|
+
await settle();
|
|
109
|
+
expect(spies.close).toHaveBeenCalledOnce();
|
|
110
|
+
expect(cleanup).toHaveBeenCalledOnce();
|
|
111
|
+
expect(exit).toHaveBeenCalledOnce();
|
|
112
|
+
});
|
|
113
|
+
|
|
114
|
+
it("forces the remaining connections closed when draining outruns the deadline", () => {
|
|
115
|
+
vi.useFakeTimers();
|
|
116
|
+
try {
|
|
117
|
+
const { signals, spies } = install({ drainTimeoutMs: 5_000 });
|
|
118
|
+
signals.get("SIGTERM")?.();
|
|
119
|
+
expect(spies.closeAllConnections).not.toHaveBeenCalled();
|
|
120
|
+
vi.advanceTimersByTime(5_000);
|
|
121
|
+
expect(spies.closeAllConnections).toHaveBeenCalledOnce();
|
|
122
|
+
} finally {
|
|
123
|
+
vi.useRealTimers();
|
|
124
|
+
}
|
|
125
|
+
});
|
|
126
|
+
|
|
127
|
+
it("exits non-zero on a forced close, so the drain outcome is visible", async () => {
|
|
128
|
+
vi.useFakeTimers();
|
|
129
|
+
let exitSpy: ReturnType<typeof vi.fn>;
|
|
130
|
+
try {
|
|
131
|
+
const { signals, exit } = install({ drainTimeoutMs: 5_000 });
|
|
132
|
+
exitSpy = exit;
|
|
133
|
+
signals.get("SIGTERM")?.();
|
|
134
|
+
vi.advanceTimersByTime(5_000);
|
|
135
|
+
} finally {
|
|
136
|
+
vi.useRealTimers();
|
|
137
|
+
}
|
|
138
|
+
await settle();
|
|
139
|
+
expect(exitSpy).toHaveBeenCalledWith(1);
|
|
140
|
+
});
|
|
141
|
+
|
|
142
|
+
it("does not force-close a drain that finished in time", () => {
|
|
143
|
+
vi.useFakeTimers();
|
|
144
|
+
try {
|
|
145
|
+
const { signals, spies, finishDraining } = install({ drainTimeoutMs: 5_000 });
|
|
146
|
+
signals.get("SIGTERM")?.();
|
|
147
|
+
finishDraining();
|
|
148
|
+
vi.advanceTimersByTime(10_000);
|
|
149
|
+
expect(spies.closeAllConnections).not.toHaveBeenCalled();
|
|
150
|
+
} finally {
|
|
151
|
+
vi.useRealTimers();
|
|
152
|
+
}
|
|
153
|
+
});
|
|
154
|
+
|
|
155
|
+
it("reports a cleanup failure through the app logger, not console", async () => {
|
|
156
|
+
const consoleError = vi.spyOn(console, "error").mockImplementation(() => {});
|
|
157
|
+
const err = new Error("collector teardown failed");
|
|
158
|
+
const { signals, finishDraining, logger, exit } = install({
|
|
159
|
+
cleanup: () => Promise.reject(err),
|
|
160
|
+
});
|
|
161
|
+
signals.get("SIGTERM")?.();
|
|
162
|
+
finishDraining();
|
|
163
|
+
await settle();
|
|
164
|
+
expect(logger.error).toHaveBeenCalledWith({ err }, expect.stringContaining("cleanup failed"));
|
|
165
|
+
expect(consoleError).not.toHaveBeenCalled();
|
|
166
|
+
expect(exit).toHaveBeenCalledWith(1);
|
|
167
|
+
consoleError.mockRestore();
|
|
168
|
+
});
|
|
169
|
+
|
|
170
|
+
it("still exits when cleanup throws — a failed dispose must not wedge the process", async () => {
|
|
171
|
+
const { signals, finishDraining, exit } = install({
|
|
172
|
+
cleanup: () => {
|
|
173
|
+
throw new Error("boom");
|
|
174
|
+
},
|
|
175
|
+
});
|
|
176
|
+
signals.get("SIGTERM")?.();
|
|
177
|
+
finishDraining();
|
|
178
|
+
await settle();
|
|
179
|
+
expect(exit).toHaveBeenCalledWith(1);
|
|
180
|
+
});
|
|
181
|
+
|
|
182
|
+
it("does not report a failed close as a clean drain", async () => {
|
|
183
|
+
const err = new Error("Server is not running");
|
|
184
|
+
const { signals, failClose, logger, exit } = install();
|
|
185
|
+
signals.get("SIGTERM")?.();
|
|
186
|
+
failClose(err);
|
|
187
|
+
await settle();
|
|
188
|
+
expect(logger.error).toHaveBeenCalledWith({ err }, expect.stringContaining("close failed"));
|
|
189
|
+
expect(exit).toHaveBeenCalledWith(1);
|
|
190
|
+
});
|
|
191
|
+
|
|
192
|
+
it("bounds cleanup so a hanging dispose cannot wedge the process", async () => {
|
|
193
|
+
// `finish` awaited cleanup with no deadline, and the drain deadline was
|
|
194
|
+
// already cleared by then, so a dispose that never settled meant `exit`
|
|
195
|
+
// was never reached — the wedge the deadline was added to remove.
|
|
196
|
+
vi.useFakeTimers();
|
|
197
|
+
try {
|
|
198
|
+
const { signals, finishDraining, exit, logger } = install({
|
|
199
|
+
cleanup: () => new Promise<void>(() => {}),
|
|
200
|
+
drainTimeoutMs: 5_000,
|
|
201
|
+
});
|
|
202
|
+
signals.get("SIGTERM")?.();
|
|
203
|
+
finishDraining();
|
|
204
|
+
await vi.advanceTimersByTimeAsync(5_000);
|
|
205
|
+
expect(logger.error).toHaveBeenCalledWith(
|
|
206
|
+
expect.objectContaining({ cleanupTimeoutMs: 5_000 }),
|
|
207
|
+
expect.stringContaining("cleanup timed out"),
|
|
208
|
+
);
|
|
209
|
+
expect(exit).toHaveBeenCalledWith(1);
|
|
210
|
+
} finally {
|
|
211
|
+
vi.useRealTimers();
|
|
212
|
+
}
|
|
213
|
+
});
|
|
214
|
+
|
|
215
|
+
it("does not penalise a cleanup that finishes inside its budget", async () => {
|
|
216
|
+
vi.useFakeTimers();
|
|
217
|
+
try {
|
|
218
|
+
const { signals, finishDraining, exit } = install({
|
|
219
|
+
cleanup: () => Promise.resolve(),
|
|
220
|
+
drainTimeoutMs: 5_000,
|
|
221
|
+
});
|
|
222
|
+
signals.get("SIGTERM")?.();
|
|
223
|
+
finishDraining();
|
|
224
|
+
await vi.advanceTimersByTimeAsync(10_000);
|
|
225
|
+
expect(exit).toHaveBeenCalledExactlyOnceWith(0);
|
|
226
|
+
} finally {
|
|
227
|
+
vi.useRealTimers();
|
|
228
|
+
}
|
|
229
|
+
});
|
|
230
|
+
|
|
231
|
+
it("defers the real exit a turn so pino's buffered destination can flush (#210 review)", async () => {
|
|
232
|
+
// `createAppLogger` uses pino's default (non-synchronous) destination, so
|
|
233
|
+
// exiting in the same tick as the last `logger.error` can drop exactly the
|
|
234
|
+
// `cleanup failed` line an operator would go looking for.
|
|
235
|
+
const exitProcess = vi.fn();
|
|
236
|
+
deferExit(3, exitProcess);
|
|
237
|
+
expect(exitProcess).not.toHaveBeenCalled();
|
|
238
|
+
await new Promise((resolve) => setImmediate(resolve));
|
|
239
|
+
expect(exitProcess).toHaveBeenCalledWith(3);
|
|
240
|
+
});
|
|
241
|
+
|
|
242
|
+
it("reports the cleanup outcome as the reason, not the drain that preceded it", async () => {
|
|
243
|
+
// `exitCode` became 1 while `reason` still said "drained", so the one line
|
|
244
|
+
// an operator alerts on contradicted itself. The drain outcome is still
|
|
245
|
+
// carried, under its own key, so neither fact is lost.
|
|
246
|
+
const { signals, finishDraining, logger, exit } = install({
|
|
247
|
+
cleanup: () => Promise.reject(new Error("teardown failed")),
|
|
248
|
+
});
|
|
249
|
+
signals.get("SIGTERM")?.();
|
|
250
|
+
finishDraining();
|
|
251
|
+
await settle();
|
|
252
|
+
expect(logger.info).toHaveBeenCalledWith(
|
|
253
|
+
{ reason: "cleanup-failed", drain: "drained", exitCode: 1 },
|
|
254
|
+
"graceful shutdown: complete",
|
|
255
|
+
);
|
|
256
|
+
expect(exit).toHaveBeenCalledWith(1);
|
|
257
|
+
});
|
|
258
|
+
|
|
259
|
+
it("keeps reason and drain identical when cleanup succeeds", async () => {
|
|
260
|
+
const { signals, finishDraining, logger } = install({ cleanup: () => Promise.resolve() });
|
|
261
|
+
signals.get("SIGTERM")?.();
|
|
262
|
+
finishDraining();
|
|
263
|
+
await settle();
|
|
264
|
+
expect(logger.info).toHaveBeenCalledWith(
|
|
265
|
+
{ reason: "drained", drain: "drained", exitCode: 0 },
|
|
266
|
+
"graceful shutdown: complete",
|
|
267
|
+
);
|
|
268
|
+
});
|
|
269
|
+
|
|
270
|
+
it("removes its own signal listeners once shutting down", () => {
|
|
271
|
+
const { signals } = install();
|
|
272
|
+
signals.get("SIGTERM")?.();
|
|
273
|
+
expect(signals.size).toBe(0);
|
|
274
|
+
});
|
|
275
|
+
});
|
|
@@ -7,9 +7,9 @@
|
|
|
7
7
|
import { fileURLToPath } from "node:url";
|
|
8
8
|
import { builtinCollectorsModule } from "@o3co/auth.policy-verifier.builtins";
|
|
9
9
|
import { builtinKeyResolversModule, createApp } from "@o3co/auth.policy-verifier.server";
|
|
10
|
-
import { gracefulShutdown } from "@o3co/auth.utils";
|
|
11
10
|
import { loadAppConfig } from "./loadConfig.js";
|
|
12
11
|
import { createAppLogger } from "./logger.js";
|
|
12
|
+
import { installGracefulShutdown } from "./shutdown.js";
|
|
13
13
|
|
|
14
14
|
const env = process.env.CONFIG_ENV || process.env.NODE_ENV || "development";
|
|
15
15
|
const configDir = new URL("../config/", import.meta.url);
|
|
@@ -29,4 +29,4 @@ const server = app.listen(config.http.port, config.http.hostname, () => {
|
|
|
29
29
|
logger.info(`listening on http://${config.http.hostname}:${config.http.port}`);
|
|
30
30
|
});
|
|
31
31
|
|
|
32
|
-
|
|
32
|
+
installGracefulShutdown(server, { logger });
|
|
@@ -0,0 +1,202 @@
|
|
|
1
|
+
// SPDX-FileCopyrightText: 2026 1o1 Co. Ltd.
|
|
2
|
+
// SPDX-License-Identifier: Apache-2.0
|
|
3
|
+
|
|
4
|
+
import type { Server } from "node:http";
|
|
5
|
+
import type { Logger } from "@o3co/auth.policy-verifier.core";
|
|
6
|
+
|
|
7
|
+
/**
|
|
8
|
+
* Graceful shutdown for the scaffolded server.
|
|
9
|
+
*
|
|
10
|
+
* ## Why this is in the template rather than a dependency
|
|
11
|
+
*
|
|
12
|
+
* It used to be `gracefulShutdown` from `@o3co/auth.utils@0.0.4`. For a
|
|
13
|
+
* component every protected call waits on, "does SIGTERM wait for in-flight
|
|
14
|
+
* decisions, and for how long?" has to be answerable from the code an operator
|
|
15
|
+
* deploys — and it was answerable only by reading a pre-1.0 package no
|
|
16
|
+
* contract here pinned. Reading it answered the question, and the answer was
|
|
17
|
+
* the reason to move it: **there was no deadline**. `server.close()` waits
|
|
18
|
+
* indefinitely, so one stuck decision meant the process never exited on its
|
|
19
|
+
* own and the orchestrator's SIGKILL took it down mid-flight — the opposite of
|
|
20
|
+
* a graceful shutdown, arriving only under the load that produces a stuck
|
|
21
|
+
* request. The cleanup-failure path also wrote to `console.error`, one bare
|
|
22
|
+
* line in a composition root that went to the trouble of wiring NDJSON (#107).
|
|
23
|
+
*
|
|
24
|
+
* `auth.provider`'s standalone template made the same move for the same
|
|
25
|
+
* reasons (o3co/auth.provider#290); keeping the two composition roots
|
|
26
|
+
* symmetric is what lets one aggregator pipeline serve the whole stack.
|
|
27
|
+
*
|
|
28
|
+
* ## The guarantees, stated
|
|
29
|
+
*
|
|
30
|
+
* 1. **SIGTERM and SIGINT** both start it; a second signal is ignored rather
|
|
31
|
+
* than starting a second cleanup over the first one's work.
|
|
32
|
+
* 2. **New connections stop immediately** (`close`), and idle keep-alive
|
|
33
|
+
* sockets are released (`closeIdleConnections`) — they hold the server open
|
|
34
|
+
* with no request behind them, so a quiet server would otherwise wait out
|
|
35
|
+
* the whole deadline for nothing.
|
|
36
|
+
* 3. **In-flight requests get `drainTimeoutMs`** (default 10s) to finish.
|
|
37
|
+
* 4. **Past the deadline, remaining connections are cut**
|
|
38
|
+
* (`closeAllConnections`) and the process exits **non-zero** — an
|
|
39
|
+
* orchestrator that only ever sees `0` cannot tell a clean drain from one
|
|
40
|
+
* that ran out of time.
|
|
41
|
+
* 5. **`cleanup` runs after draining, before exit**, logged through the app
|
|
42
|
+
* logger and reflected in the exit code. It is bounded by
|
|
43
|
+
* `cleanupTimeoutMs`, so a dispose that never settles cannot wedge the
|
|
44
|
+
* process the way an unbounded `await` did.
|
|
45
|
+
* 6. **A `close` that fails is not reported as a clean drain.**
|
|
46
|
+
*
|
|
47
|
+
* Size `drainTimeoutMs` **below** the orchestrator's own kill grace period
|
|
48
|
+
* (Kubernetes `terminationGracePeriodSeconds`, compose `stop_grace_period`,
|
|
49
|
+
* both 30s by default) — the point is to close on our terms before SIGKILL
|
|
50
|
+
* arrives on someone else's.
|
|
51
|
+
*/
|
|
52
|
+
export interface GracefulShutdownOptions {
|
|
53
|
+
readonly logger: Logger;
|
|
54
|
+
/** Release what the composition holds — caches, clients, key resolvers. */
|
|
55
|
+
readonly cleanup?: () => void | Promise<void>;
|
|
56
|
+
/** How long in-flight requests get before connections are cut. Default 10s. */
|
|
57
|
+
readonly drainTimeoutMs?: number;
|
|
58
|
+
/**
|
|
59
|
+
* How long `cleanup` gets before the shutdown gives up on it. Defaults to
|
|
60
|
+
* `drainTimeoutMs`, so the worst-case shutdown is the two budgets in
|
|
61
|
+
* sequence — size both against the orchestrator's grace period, not one.
|
|
62
|
+
*/
|
|
63
|
+
readonly cleanupTimeoutMs?: number;
|
|
64
|
+
/** Injected in tests; defaults to {@link deferExit}. */
|
|
65
|
+
readonly exit?: (code: number) => void;
|
|
66
|
+
/** Injected in tests; defaults to `process.on`. */
|
|
67
|
+
readonly onSignal?: (signal: NodeJS.Signals, handler: () => void) => void;
|
|
68
|
+
/** Injected in tests; defaults to `process.removeListener`. */
|
|
69
|
+
readonly offSignal?: (signal: NodeJS.Signals, handler: () => void) => void;
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
const DEFAULT_DRAIN_TIMEOUT_MS = 10_000;
|
|
73
|
+
const SIGNALS: readonly NodeJS.Signals[] = ["SIGTERM", "SIGINT"];
|
|
74
|
+
|
|
75
|
+
/**
|
|
76
|
+
* Exit after yielding the loop once.
|
|
77
|
+
*
|
|
78
|
+
* `createAppLogger` uses pino's default destination, which is not synchronous,
|
|
79
|
+
* so calling `process.exit` in the same tick as the last `logger.error` can
|
|
80
|
+
* drop exactly the lines that say why the shutdown failed. One turn is a flush
|
|
81
|
+
* window, not a guarantee: a deployment that needs certainty should pass an
|
|
82
|
+
* `exit` that flushes its own transport first.
|
|
83
|
+
*/
|
|
84
|
+
export function deferExit(code: number, exitProcess: (code: number) => void = process.exit): void {
|
|
85
|
+
setImmediate(() => exitProcess(code));
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
export function installGracefulShutdown(server: Server, options: GracefulShutdownOptions): void {
|
|
89
|
+
const {
|
|
90
|
+
logger,
|
|
91
|
+
cleanup,
|
|
92
|
+
drainTimeoutMs = DEFAULT_DRAIN_TIMEOUT_MS,
|
|
93
|
+
exit = deferExit,
|
|
94
|
+
onSignal = (signal, handler): void => {
|
|
95
|
+
process.on(signal, handler);
|
|
96
|
+
},
|
|
97
|
+
offSignal = (signal, handler): void => {
|
|
98
|
+
process.removeListener(signal, handler);
|
|
99
|
+
},
|
|
100
|
+
} = options;
|
|
101
|
+
|
|
102
|
+
const cleanupTimeoutMs = options.cleanupTimeoutMs ?? drainTimeoutMs;
|
|
103
|
+
|
|
104
|
+
let shuttingDown = false;
|
|
105
|
+
let finished = false;
|
|
106
|
+
|
|
107
|
+
/** Sentinel so a timed-out cleanup is reported as that, not as a throw. */
|
|
108
|
+
const CLEANUP_TIMED_OUT = Symbol("cleanup-timed-out");
|
|
109
|
+
|
|
110
|
+
/**
|
|
111
|
+
* Await `cleanup`, but not forever. `cleanup()` is invoked inside the async
|
|
112
|
+
* wrapper so a synchronous throw lands in the same rejection path as an
|
|
113
|
+
* async one.
|
|
114
|
+
*/
|
|
115
|
+
const runCleanup = async (): Promise<typeof CLEANUP_TIMED_OUT | undefined> => {
|
|
116
|
+
if (!cleanup) return;
|
|
117
|
+
let timer: ReturnType<typeof setTimeout> | undefined;
|
|
118
|
+
try {
|
|
119
|
+
return await Promise.race([
|
|
120
|
+
(async (): Promise<undefined> => {
|
|
121
|
+
await cleanup();
|
|
122
|
+
return undefined;
|
|
123
|
+
})(),
|
|
124
|
+
new Promise<typeof CLEANUP_TIMED_OUT>((resolve) => {
|
|
125
|
+
timer = setTimeout(() => resolve(CLEANUP_TIMED_OUT), cleanupTimeoutMs);
|
|
126
|
+
timer.unref?.();
|
|
127
|
+
}),
|
|
128
|
+
]);
|
|
129
|
+
} finally {
|
|
130
|
+
if (timer) clearTimeout(timer);
|
|
131
|
+
}
|
|
132
|
+
};
|
|
133
|
+
|
|
134
|
+
/** Run `cleanup` and exit. Called by whichever of drain / deadline wins. */
|
|
135
|
+
const finish = async (code: number, reason: string): Promise<void> => {
|
|
136
|
+
if (finished) return;
|
|
137
|
+
finished = true;
|
|
138
|
+
let exitCode = code;
|
|
139
|
+
// `reason` names whatever decided the exit code, so the line an operator
|
|
140
|
+
// alerts on cannot say "drained" next to a non-zero code. The drain
|
|
141
|
+
// outcome keeps its own key rather than being overwritten — both facts
|
|
142
|
+
// are wanted, and a stable shape is what makes the line queryable.
|
|
143
|
+
let outcome = reason;
|
|
144
|
+
try {
|
|
145
|
+
if ((await runCleanup()) === CLEANUP_TIMED_OUT) {
|
|
146
|
+
logger.error({ cleanupTimeoutMs }, "graceful shutdown: cleanup timed out");
|
|
147
|
+
exitCode = 1;
|
|
148
|
+
outcome = "cleanup-timeout";
|
|
149
|
+
}
|
|
150
|
+
} catch (err) {
|
|
151
|
+
// Through the app logger, not `console.error`: a shutdown that
|
|
152
|
+
// failed to release what it held is exactly the line an operator
|
|
153
|
+
// needs to find later, and a bare write is the one their pipeline
|
|
154
|
+
// drops.
|
|
155
|
+
logger.error({ err }, "graceful shutdown: cleanup failed");
|
|
156
|
+
exitCode = 1;
|
|
157
|
+
outcome = "cleanup-failed";
|
|
158
|
+
}
|
|
159
|
+
logger.info({ reason: outcome, drain: reason, exitCode }, "graceful shutdown: complete");
|
|
160
|
+
exit(exitCode);
|
|
161
|
+
};
|
|
162
|
+
|
|
163
|
+
const handler = (): void => {
|
|
164
|
+
if (shuttingDown) return;
|
|
165
|
+
shuttingDown = true;
|
|
166
|
+
for (const signal of SIGNALS) offSignal(signal, handler);
|
|
167
|
+
logger.info({ drainTimeoutMs }, "graceful shutdown: draining");
|
|
168
|
+
|
|
169
|
+
const deadline = setTimeout(() => {
|
|
170
|
+
logger.error(
|
|
171
|
+
{ drainTimeoutMs },
|
|
172
|
+
"graceful shutdown: drain deadline exceeded, closing remaining connections",
|
|
173
|
+
);
|
|
174
|
+
server.closeAllConnections();
|
|
175
|
+
void finish(1, "drain-timeout");
|
|
176
|
+
}, drainTimeoutMs);
|
|
177
|
+
// The deadline must not be what keeps the process alive once the drain
|
|
178
|
+
// has already finished.
|
|
179
|
+
deadline.unref?.();
|
|
180
|
+
|
|
181
|
+
server.close((err) => {
|
|
182
|
+
clearTimeout(deadline);
|
|
183
|
+
if (err) {
|
|
184
|
+
// `close` reports through its callback — "Server is not running"
|
|
185
|
+
// is the common one, but any listener teardown failure lands
|
|
186
|
+
// here. Reporting "drained" and exiting 0 on it would tell an
|
|
187
|
+
// orchestrator the shutdown went cleanly when the listener did
|
|
188
|
+
// not actually come down.
|
|
189
|
+
logger.error({ err }, "graceful shutdown: server close failed");
|
|
190
|
+
void finish(1, "close-failed");
|
|
191
|
+
return;
|
|
192
|
+
}
|
|
193
|
+
void finish(0, "drained");
|
|
194
|
+
});
|
|
195
|
+
// Idle keep-alive sockets have no request behind them, so nothing is
|
|
196
|
+
// lost by releasing them — and without this a quiet server waits out
|
|
197
|
+
// the whole deadline for connections that will never send anything.
|
|
198
|
+
server.closeIdleConnections();
|
|
199
|
+
};
|
|
200
|
+
|
|
201
|
+
for (const signal of SIGNALS) onSignal(signal, handler);
|
|
202
|
+
}
|
package/templates/versions.json
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
{
|
|
2
|
-
"@o3co/auth.policy-verifier.core": "0.
|
|
3
|
-
"@o3co/auth.policy-verifier.builtins": "0.
|
|
4
|
-
"@o3co/auth.policy-verifier.server": "0.
|
|
2
|
+
"@o3co/auth.policy-verifier.core": "0.8.1",
|
|
3
|
+
"@o3co/auth.policy-verifier.builtins": "0.8.1",
|
|
4
|
+
"@o3co/auth.policy-verifier.server": "0.8.1"
|
|
5
5
|
}
|