@btravstack/core 0.14.0 → 0.15.0

package/README.md

@@ -114,6 +114,20 @@ is a module that exports no runtime port at all.
 
 ## For a runtime author
 
+**`Serving` carries an optional third member, `stopped`** — the channel a
+runtime reports its own death on, `() => AsyncResult<void, never>`. Without it
+the lifecycle only ever moves on a signal or a `stop()` call, so a worker whose
+poll loop died stays alive with `/readyz` answering `200`: a pod in a Service's
+endpoints, consuming nothing. A runtime that can stop on its own account should
+implement it; the ticker above does not, because a `setInterval` has no third
+state to report, and that is why the field is optional rather than required.
+
+The obligation is the half a signature cannot express: **withdraw after a stop
+the kernel asked for.** An arm that settles on the ordinary path races every
+clean shutdown. `@btravstack/temporal-worker` is the worked implementation, and
+[the runtime reference](https://btravstack.github.io/btravstack/reference/core/runtime)
+states the contract in full.
+
 Two helpers sit beside the `Runtime` contract, because every transport needs
 them and none should own a private copy:
 
@@ -135,11 +149,15 @@ them and none should own a private copy:
   telling the runtime to stop accepting — endpoint removal is eventually
   consistent — then in-flight work gets `drainTimeoutMs` (`DRAIN_TIMEOUT_MS`,
   default `20_000`); whatever is still open is aborted and reported
-  `abandoned`. A second signal skips the drain. Both timings have to agree with
-  the pod's `terminationGracePeriodSeconds`, which is why the deployment can
-  set them and the option only pins them.
+  `abandoned`. The teardown that follows gets `stopTimeoutMs`
+  (`STOP_TIMEOUT_MS`, default `5_000`), so a `release` that never settles is
+  reported as `abandonedAt: "stop"` rather than leaving the process in
+  `stopping` with no exit report at all. A second signal stops every wait at
+  once. The three sum to the pod's `terminationGracePeriodSeconds` default of
+  30 s, which is why the deployment sets them and the options only pin them.
 - **Probes from the state machine.** `GET /livez` and `GET /readyz` on
-  `PROBE_PORT` (default `9000`), up before the graph is built.
+  `PROBE_PORT` (default `9000`) and `PROBE_HOST` (default `0.0.0.0`, so a
+  kubelet `httpGet` probe reaches them), up before the graph is built.
 - **Configuration from the environment, typed.** The kernel provides
   [`@btravstack/config`](../config)'s `Env` port to every graph it boots; a
   bad value is a `ConfigInvalid` naming every fault at once, and exit `78`.
@@ -153,6 +171,14 @@ them and none should own a private copy:
   there. `signal` is the very `AbortSignal` the unit's work callback is handed,
   so a runtime whose work is a library's `next()` — a Temporal activity, an
   AMQP delivery — can still honour the drain deadline.
+- **Observation as a set port, not a logger in every starter.** A starter
+  hands each operation to `Observers` — `observe(observers, operation)` for a
+  span-shaped start-then-settle, `observed(observers, operation, call)` around
+  one `AsyncResult`-returning call — and holds no `Logger`, `Meter` or
+  `Tracer` of its own. Composing
+  [`@btravstack/observability`](../observability) is what turns those
+  operations into lines, spans and instruments; composing nothing costs an
+  inert call.
 - **Nothing throws.** Every async surface is an
   [`unthrown`](https://github.com/btravstack/unthrown) `AsyncResult`;
   `runMain` sets `process.exitCode` — `0` clean, `1` a modeled startup error,

package/dist/index.cjs

@@ -33,6 +33,26 @@ var HealthCheckFailed = class extends (0, unthrown.TaggedError)("HealthCheckFail
 */
 var HealthChecks = class extends _btravstack_di.Port.many("HealthChecks") {};
 /**
+* A check that never settles, bounded. `unref`'d, so a pending probe is never
+* the reason a process stays alive, and cleared on the winning branch so a
+* slow-but-answering check leaves no timer behind.
+*
+* It bounds the WAIT, not the check: nothing here can cancel whatever the check
+* is blocked on. That is the same bargain the drain and the stop deadline make,
+* and it is why this reports rather than recovers.
+*/
+const answeringWithin = (health, ms) => {
+	let timer;
+	const timeout = new Promise((resolve) => {
+		timer = setTimeout(() => resolve((0, unthrown.Err)(new HealthCheckFailed({ reason: `did not answer within ${String(ms)} ms` }))), ms);
+		timer.unref();
+	});
+	const answering = (0, unthrown.OkAsync)().flatMap(() => health.check());
+	return (0, unthrown.fromSafePromise)(Promise.race([answering, timeout]).finally(() => {
+		clearTimeout(timer);
+	})).flatMap((settled) => settled);
+};
+/**
 * Runs every check and folds the results into one report — the whole app is
 * unhealthy if any single component is.
 *
@@ -44,8 +64,15 @@ var HealthChecks = class extends _btravstack_di.Port.many("HealthChecks") {};
 * a defecting `AsyncResult` alike are recovered into an unhealthy line —
 * escaped, the first would reach the kernel's own `uncaughtException` handler
 * and the second would leave `/healthz` hanging with nothing written.
+*
+* **A check that never settles is the third case, and it needed a deadline
+* rather than a recovery.** Neither recovery above can see one: there is no
+* error and no defect, only silence, so `/healthz` held a socket open per hit
+* while kubelet timed out against a report that named nothing. Each check now
+* races {@link HealthOptions.timeoutMs}, and the component that hung is named
+* in the report like any other unhealthy one.
 */
-const runHealthChecks = (checks) => (0, unthrown.allAsync)(checks.map((health) => (0, unthrown.OkAsync)().flatMap(() => health.check()).map(() => ({
+const runHealthChecks = (checks, options = {}) => (0, unthrown.allAsync)(checks.map((health) => answeringWithin(health, health.timeoutMs ?? options.timeoutMs ?? 800).map(() => ({
 	name: health.name,
 	status: "healthy"
 })).recoverErrCases((matcher) => matcher.with(unthrown.P.tag("HealthCheckFailed"), (error) => ({
@@ -104,6 +131,24 @@ const observe = (observers, operation) => {
 		for (const settle of settlers) settle(settled);
 	};
 };
+/**
+* {@link observe}, wrapped around one call: every observer is started before
+* `call` runs and settled from whichever channel it comes back on — `ok` on
+* success, `error` carrying the `Err` or the defect as `cause` otherwise. The
+* wrapper is transparent to the `Result`: whatever `call` answers is what the
+* caller receives.
+*
+* `settled` replaces either default where a starter has more to say: `ok`
+* receives the value (a cache reports a hit or a miss), `failure` the
+* `FailureView` (a store reports a missing object as an ordinary answer).
+*/
+const observed = (observers, operation, call, settled = {}) => {
+	const settle = observe(observers, operation);
+	return call().tap((value) => settle(settled.ok?.(value) ?? { outcome: "ok" })).tapFailure((failure) => settle(settled.failure?.(failure) ?? {
+		outcome: "error",
+		cause: failure.tag === "Err" ? failure.error : failure.cause
+	}));
+};
 //#endregion
 //#region src/clock.ts
 const systemClock = {
@@ -336,7 +381,7 @@ const startProbeServer = (args) => (0, unthrown.fromSafePromise)(new Promise((re
 	const ignoreServingError = () => {};
 	server.once("error", onBindError);
 	try {
-		server.listen(args.port, "127.0.0.1", () => {
+		server.listen(args.port, args.hostname, () => {
 			server.removeListener("error", onBindError);
 			server.on("error", ignoreServingError);
 			const address = server.address();
@@ -443,6 +488,13 @@ const createUnitRegistry = () => {
 };
 //#endregion
 //#region src/start.ts
+/**
+* An arm of a race that has lost and has nothing to report: it never settles,
+* so the winner decides. It holds no timer and no handle, which is what makes
+* it safe — the alternative is an arm that answers a report no consumer reads,
+* and whose side effects then have to be guarded one by one.
+*/
+const withdrawn = () => (0, unthrown.fromSafePromise)(new Promise(() => {}));
 const providesEnv = (module) => module.provides.some((provider) => provider.port.portId === _btravstack_config.Env.portId) || module.imports.some(providesEnv);
 /**
 * What the kernel reads from the environment for itself. Each field is pinned
@@ -451,13 +503,21 @@ const providesEnv = (module) => module.provides.some((provider) => provider.port
 */
 const KERNEL_DEFAULTS = {
 	probePort: 9e3,
+	probeHost: "0.0.0.0",
 	preDrainDelayMs: 5e3,
-	drainTimeoutMs: 2e4
+	drainTimeoutMs: 2e4,
+	stopTimeoutMs: 5e3
 };
 const readKernelConfig = (options, env) => {
-	const probePort = options.probes === void 0 ? void 0 : options.probes === false ? KERNEL_DEFAULTS.probePort : options.probes.port;
+	const probes = options.probes === false ? {
+		port: KERNEL_DEFAULTS.probePort,
+		host: KERNEL_DEFAULTS.probeHost
+	} : options.probes;
+	const probePort = probes?.port;
+	const probeHost = probes?.host;
 	const result = _btravstack_config.Config.object({
 		probePort: _btravstack_config.Config.pinned(probePort, _btravstack_config.Config.port("PROBE_PORT", { default: KERNEL_DEFAULTS.probePort })),
+		probeHost: _btravstack_config.Config.pinned(probeHost, _btravstack_config.Config.string("PROBE_HOST", { default: KERNEL_DEFAULTS.probeHost })),
 		preDrainDelayMs: _btravstack_config.Config.pinned(options.preDrainDelayMs, _btravstack_config.Config.integer("PRE_DRAIN_DELAY_MS", {
 			default: KERNEL_DEFAULTS.preDrainDelayMs,
 			min: 0
@@ -465,6 +525,10 @@ const readKernelConfig = (options, env) => {
 		drainTimeoutMs: _btravstack_config.Config.pinned(options.drainTimeoutMs, _btravstack_config.Config.integer("DRAIN_TIMEOUT_MS", {
 			default: KERNEL_DEFAULTS.drainTimeoutMs,
 			min: 0
+		})),
+		stopTimeoutMs: _btravstack_config.Config.pinned(options.stopTimeoutMs, _btravstack_config.Config.integer("STOP_TIMEOUT_MS", {
+			default: KERNEL_DEFAULTS.stopTimeoutMs,
+			min: 0
 		}))
 	})["~standard"].validate(env);
 	return result.issues === void 0 ? (0, unthrown.Ok)(result.value) : (0, unthrown.Err)(new RuntimeStartFailed({
@@ -497,8 +561,14 @@ const start = (module, options = {}) => {
 	const teardownErrors = [];
 	const startedAt = clock.now();
 	const kernelConfig = readKernelConfig(options, env);
-	const { preDrainDelayMs, drainTimeoutMs } = kernelConfig.getOrElse(() => KERNEL_DEFAULTS);
+	const { preDrainDelayMs, drainTimeoutMs, stopTimeoutMs } = kernelConfig.getOrElse(() => KERNEL_DEFAULTS);
 	const skipDrain = new AbortController();
+	const stopSettled = new AbortController();
+	let lifecycleSettled = false;
+	const onLifecycleSettled = () => {
+		lifecycleSettled = true;
+		stopSettled.abort();
+	};
 	let shutdownRequestedAt = startedAt;
 	let shutdownRequested = false;
 	const sinceShutdownRequested = () => clock.now() - shutdownRequestedAt;
@@ -517,7 +587,11 @@ const start = (module, options = {}) => {
 	const ready = () => tracker.current() === "serving" && !forcedUnready;
 	const disposeSignals = options.signals === false ? () => {} : installSignalHandlers({
 		onFirst: () => requestShutdown("signal"),
-		onSecond: () => skipDrain.abort()
+		onSecond: () => {
+			skipDrain.abort();
+			stopSettled.abort();
+			abandonBuild("signal");
+		}
 	});
 	const disposeUncaught = options.signals === false ? () => {} : installUncaughtHandlers((cause) => {
 		emit({
@@ -527,17 +601,63 @@ const start = (module, options = {}) => {
 		onUnready();
 		skipDrain.abort();
 		requestShutdown("uncaught");
+		abandonBuild("uncaught");
 	});
 	let disposeProbes = () => {};
+	const disposeAll = () => {
+		disposeSignals();
+		disposeUncaught();
+		tracker.advanceTo("exited");
+		disposeProbes();
+	};
+	const reportOf = (reason, drain, abandonedAt) => ({
+		reason,
+		drain,
+		teardownErrors,
+		uptimeMs: clock.now() - startedAt,
+		...abandonedAt === void 0 ? {} : { abandonedAt }
+	});
 	let healthChecks = [];
 	const health = () => runHealthChecks(healthChecks);
 	const probeBound = Promise.withResolvers();
 	const runtimePublished = Promise.withResolvers();
+	const stopping = Promise.withResolvers();
+	const stopAbandoned = () => (0, unthrown.fromSafePromise)(stopping.promise).flatMap(({ reason, drain }) => clock.sleep(stopTimeoutMs, stopSettled.signal).flatMap(() => lifecycleSettled ? withdrawn() : (0, unthrown.OkAsync)(abandonStop(reason, drain))));
+	const abandonStop = (reason, drain) => {
+		emit({
+			type: "stoppedWaiting",
+			phase: "stop",
+			afterMs: stopSettled.signal.aborted ? void 0 : stopTimeoutMs
+		});
+		disposeAll();
+		return reportOf(reason, drain, "stop");
+	};
+	const abandoningBuild = Promise.withResolvers();
+	const buildAbandoned = () => (0, unthrown.fromSafePromise)(abandoningBuild.promise).map((reason) => {
+		emit({
+			type: "stoppedWaiting",
+			phase: "build",
+			afterMs: void 0
+		});
+		registry.abortAll();
+		runtimePublished.resolve(void 0);
+		tracker.advanceTo("stopping");
+		disposeAll();
+		return reportOf(reason, void 0, "build");
+	});
+	const abandonBuild = (reason) => {
+		const phase = tracker.current();
+		if (phase === "building" || phase === "starting") abandoningBuild.resolve(reason);
+	};
 	emit({ type: "building" });
-	const probesOptions = kernelConfig.map(({ probePort }) => options.probes === false ? false : { port: probePort });
+	const probesOptions = kernelConfig.map(({ probePort, probeHost }) => options.probes === false ? false : {
+		port: probePort,
+		hostname: probeHost
+	});
 	if (options.probes === false) probeBound.resolve(void 0);
 	const probesStarted = probesOptions.toAsync().flatMap((probes) => probes === false ? (0, unthrown.OkAsync)() : startProbeServer({
 		port: probes.port,
+		hostname: probes.hostname,
 		live,
 		ready,
 		health
@@ -592,97 +712,98 @@ const start = (module, options = {}) => {
 		if (reason !== "signal") registry.abortAll();
 		return (reason === "signal" ? runDrain(serving) : (0, unthrown.OkAsync)(void 0)).flatMap((report) => {
 			tracker.advanceTo("stopping");
+			stopping.resolve({
+				reason,
+				drain: report
+			});
 			return serving.stop().map(() => {
-				disposeSignals();
-				disposeUncaught();
-				tracker.advanceTo("exited");
-				disposeProbes();
-				return {
-					reason,
-					drain: report,
-					teardownErrors,
-					uptimeMs: clock.now() - startedAt
-				};
+				disposeAll();
+				return reportOf(reason, report);
 			});
 		});
 	};
-	return {
-		exited: probesStarted.flatMap(() => _btravstack_di.Module.scoped(root, (ctx) => {
-			tracker.advanceTo("starting");
-			const runtime = ctx.get(RuntimePort);
-			runtimeName = runtime.name;
-			healthChecks = ctx.get(HealthChecks);
-			const runtimeCtx = ctx;
-			const run = (meta, work) => registry.run(meta, (signal) => {
-				const settled = Promise.withResolvers();
-				let closing;
-				let hasSettled = false;
-				const fork = (module, seed) => {
-					if (hasSettled) return (0, unthrown.fromSafePromise)(Promise.reject(/* @__PURE__ */ new Error("a unit forks after it has already settled")));
-					if (closing !== void 0) return (0, unthrown.fromSafePromise)(Promise.reject(/* @__PURE__ */ new Error("a unit forks its scope once")));
-					const ready = Promise.withResolvers();
-					closing = _btravstack_di.Module.forkScope(ctx, module, (forked) => {
-						ready.resolve(forked);
-						return (0, unthrown.fromSafePromise)(settled.promise);
-					}, {
-						seed,
-						onTeardownError: (port, cause) => emit({
-							type: "teardownError",
-							port,
-							cause
-						})
-					}).recoverDefect((cause) => {
-						ready.reject(cause);
-						return (0, unthrown.Ok)();
-					}).get();
-					return (0, unthrown.fromSafePromise)(ready.promise);
-				};
-				const outcome = (async () => {
-					try {
-						return await work({
-							ctx: runtimeCtx,
-							fork
-						}, signal);
-					} finally {
-						hasSettled = true;
-						settled.resolve();
-						if (closing !== void 0) await closing;
-					}
-				})();
-				return (0, unthrown.fromSafePromise)(outcome).flatMap((result) => result);
-			});
-			const host = {
-				ctx: runtimeCtx,
-				run
+	const built = () => _btravstack_di.Module.scoped(root, (ctx) => {
+		tracker.advanceTo("starting");
+		const runtime = ctx.get(RuntimePort);
+		runtimeName = runtime.name;
+		healthChecks = ctx.get(HealthChecks);
+		const runtimeCtx = ctx;
+		const run = (meta, work) => registry.run(meta, (signal) => {
+			const settled = Promise.withResolvers();
+			let closing;
+			let hasSettled = false;
+			const fork = (module, seed) => {
+				if (hasSettled) return (0, unthrown.fromSafePromise)(Promise.reject(/* @__PURE__ */ new Error("a unit forks after it has already settled")));
+				if (closing !== void 0) return (0, unthrown.fromSafePromise)(Promise.reject(/* @__PURE__ */ new Error("a unit forks its scope once")));
+				const ready = Promise.withResolvers();
+				closing = _btravstack_di.Module.forkScope(ctx, module, (forked) => {
+					ready.resolve(forked);
+					return (0, unthrown.fromSafePromise)(settled.promise);
+				}, {
+					seed,
+					onTeardownError: (port, cause) => emit({
+						type: "teardownError",
+						port,
+						cause
+					})
+				}).recoverDefect((cause) => {
+					ready.reject(cause);
+					return (0, unthrown.Ok)();
+				}).get();
+				return (0, unthrown.fromSafePromise)(ready.promise);
 			};
-			return runtime.start(host).flatMap((serving) => {
-				servingInfo = serving.info;
-				tracker.advanceTo("serving");
-				runtimePublished.resolve(serving.info);
-				return (0, unthrown.fromSafePromise)(shutdown.promise).flatMap((reason) => finish(serving, reason));
-			});
-		}, { onTeardownError: (port, cause) => {
-			teardownErrors.push({
-				port,
-				cause
-			});
-			emit({
-				type: "teardownError",
-				port,
-				cause
-			});
-		} }).tapFailure((failure) => {
-			if (tracker.current() !== "stopping") emit({
-				type: "startFailed",
-				cause: failure.tag === "Err" ? failure.error : failure.cause
+			const outcome = (async () => {
+				try {
+					return await work({
+						ctx: runtimeCtx,
+						fork
+					}, signal);
+				} finally {
+					hasSettled = true;
+					settled.resolve();
+					if (closing !== void 0) await closing;
+				}
+			})();
+			return (0, unthrown.fromSafePromise)(outcome).flatMap((result) => result);
+		});
+		const host = {
+			ctx: runtimeCtx,
+			run
+		};
+		return runtime.start(host).flatMap((serving) => {
+			servingInfo = serving.info;
+			tracker.advanceTo("serving");
+			runtimePublished.resolve(serving.info);
+			serving.stopped?.().map(() => {
+				requestShutdown("runtimeStopped");
 			});
-			runtimePublished.resolve(void 0);
-			tracker.advanceTo("stopping");
-			disposeSignals();
-			disposeUncaught();
-			tracker.advanceTo("exited");
-			disposeProbes();
-		})),
+			return (0, unthrown.fromSafePromise)(shutdown.promise).flatMap((reason) => finish(serving, reason));
+		});
+	}, { onTeardownError: (port, cause) => {
+		teardownErrors.push({
+			port,
+			cause
+		});
+		emit({
+			type: "teardownError",
+			port,
+			cause
+		});
+	} }).tapFailure((failure) => {
+		if (tracker.current() !== "stopping") emit({
+			type: "startFailed",
+			cause: failure.tag === "Err" ? failure.error : failure.cause
+		});
+		runtimePublished.resolve(void 0);
+		tracker.advanceTo("stopping");
+		disposeAll();
+	}).tap(onLifecycleSettled).tapFailure(onLifecycleSettled);
+	return {
+		exited: probesStarted.flatMap(() => (0, unthrown.fromSafePromise)(Promise.race([
+			built(),
+			stopAbandoned(),
+			buildAbandoned()
+		])).flatMap((settled) => settled)),
 		stop: () => requestShutdown("runtimeStopped"),
 		requestDrain: () => requestShutdown("signal"),
 		phase: tracker.current,
@@ -698,7 +819,7 @@ const EX_CONFIG = 78;
 const isConfig = (error) => error instanceof _btravstack_config.ConfigInvalid || error instanceof RuntimeStartFailed && error.cause instanceof _btravstack_config.ConfigInvalid;
 const codeFor = (report) => {
 	if (report.reason === "uncaught") return EX_SOFTWARE;
-	return (report.drain?.abandoned ?? 0) > 0 || report.teardownErrors.length > 0 ? 2 : 0;
+	return (report.drain?.abandoned ?? 0) > 0 || report.teardownErrors.length > 0 || report.abandonedAt !== void 0 ? 2 : 0;
 };
 /**
 * The exit-code half of `runMain`, on its own so the code table can be asserted
@@ -731,6 +852,7 @@ const awaitExit = async (app, exit) => {
 * | a configuration port that could not be bound (`ConfigInvalid`, `PROBE_PORT` included) | `78` |
 * | drained with work abandoned | `2` |
 * | exited with teardown errors | `2` |
+* | stopped waiting for the teardown or a build (`ExitReport.abandonedAt`) | `2` |
 * | stopped by an uncaught exception or unhandled rejection | `70` |
 * | a defect | `70` |
 *
@@ -764,6 +886,7 @@ exports.Tracer = Tracer;
 exports.currentUnit = currentUnit;
 exports.noObserver = noObserver;
 exports.observe = observe;
+exports.observed = observed;
 exports.releasedBy = releasedBy;
 exports.runHealthChecks = runHealthChecks;
 exports.runMain = runMain;