@mark1russell7/harness 0.8.0 → 0.10.0

package/README.md

@@ -25,6 +25,7 @@ A **cell** (`cell.ts`) is all that one measurement needs, as data:
 - A privileged sidecar in the namespace of the client, for its sysctls and its routes. Docker Desktop keeps `/proc/sys` read-only in a container that operates.
 - Each hop of the topology, on the network of each of its sides.
 - An injector sidecar in the namespace of the target, with NET_ADMIN. Thus the image of the target stays stock.
+- The manifest of each environment gives the HZ of its kernel, from /proc/config.gz. A host without that file (a hosted runner) gets HZ from two readings of /proc/timer_list, one second apart: its jiffies against the time of the kernel (`hzOf`). `hzFrom` tells which.
 - The prober ([`@mark1russell7/prober`](https://www.npmjs.com/package/@mark1russell7/prober)): a privileged container that loads the kernel program. It watches the namespace of the client, of the target and of each hop. Each run keeps its records from its start to its end. A run whose records the prober lost is void.
 
 Each command is an argv array.
@@ -186,6 +187,7 @@ Each run starts from kernels that keep nothing from the run before it (`lab.forg
 - **The time.** It times the detection from the reference event of the prediction to the error report of the kernel. eBPF stamps the two events. For keepalive, the reference is the last received segment. For the give-up, it is the first retransmission of the RTO timer, not the tail loss probe one RTO before it. For a reset, it is the send of the client, which no kernel record marks. The client stamps the send before its write.
 - **The other views.** It also gives the same interval from the `TCP_INFO` samples of the client, and the delay before the client saw the error. It also gives the steps of the kernel: each keepalive probe, or each expiry of the RTO timer.
 - **Void runs.** A run is void when the timer that drives the measurement was two jiffies or more behind its wheel slot in the timed interval. The wheel does a slot in the tick when it is due. Thus such a firing measured the host, which stopped a vCPU, not the kernel. A keepalive timer that found its socket held also makes the run void. A run whose fault came too late is void too.
+  - The rule is in time when the run knows its HZ (`Run.hz`, from the manifest). Then the delay must reach two jiffies and δ_abs, the larger of the two (`delayVoids`). Each band includes δ_abs, thus a smaller delay does not change a result. At HZ 250 two jiffies are 8 ms, more than a δ_abs of 4 ms, and the rule stays the same. At HZ 1000 two jiffies are 2 ms. A run without its HZ keeps the rule in jiffies.
 - **The first round trip.** The measurement gives the first round trip of the socket of the observer (`rtt`). It is the smoothed round trip in the first record of the socket: the sample of the handshake. A run whose first round trip is above the range of its prediction is void, with the reason `rtt-over-assumption` and the value. Such a run measured another path than the band assumes. For example, a round trip of 4.826 ms makes an RTO of 52 jiffies, and a ladder that assumes 1 ms has 51.
 - **Findings.** A finding is a wrong errno, a detector that did not fire, or a detection where the prediction is silence. A router that forgot the flow in the window of a silent prediction is also a finding.
 - **A reply.** A silent prediction with `replies` times the send of the request to the first segment with data that comes back. A proxy can acknowledge the request first, with no data.
@@ -206,6 +208,7 @@ Each run starts from kernels that keep nothing from the run before it (`lab.forg
   - Each wait goes from a `reconnecting` line, where the program starts its timer, to its next `dial` line, where it makes the socket. A wait outside its band, widened by δ_abs, is a finding. A try that did not come is a finding too.
   - The kernel gives the time from the detection to the first SYN of a new socket (`syn`). It also gives the time to the first new socket in ESTABLISHED (`established`).
   - It also gives the time from each `dial` line to its SYN (`connects`): the work of the program before connect(2).
+  - A measurement-only prediction can give `reconnects` too. The end of its connection has no band, but each wait has one. A wait outside its band is a finding of its run, and the verdict names that run. The decision stays `measured`.
 - **A call in flight of a WebSocket client** (`callOf`). The run gives its error or its reply, and the time from the detection to its end.
 - **A stream.** In a stream (`probe.mode` is `stream`), the client sends `count` requests. The run keeps the time from each send to the last byte of its reply (`samples`). A record without `replied` lines gives the time to the first data.
 - **Reordering.** A measurement-only run also gives what reordering, duplicates and losses did (`reordering`). At the client, it counts the arrivals beyond a hole and the arrivals of data that it had already, against `rcv_nxt`. From the sender, it gives the most SACKed and lost segments, the DSACKs, the fast retransmissions, and the reordering degree and count of the kernel.
@@ -241,7 +244,9 @@ Each run starts from kernels that keep nothing from the run before it (`lab.forg
 
 `verdictOf(prediction, runs)` (`verdict.ts`) judges a cell:
 
-- **Containment.** Each run lands in [T_lo − δ_abs, T_hi + δ_abs]. `NOISE` (δ_abs) is 4 ms, from the calibration: the worst reset took 1.141 ms, and δ_abs is one jiffy at the minimum.
+- **Containment.** Each run lands in [T_lo − δ_abs, T_hi + δ_abs]. `NOISE` (δ_abs) is 4 ms, from the calibration of the first environment: the worst reset took 1.141 ms, and δ_abs is one jiffy at the minimum.
+  - **δ_abs of an environment.** A cell can give the δ_abs of its environment (`Cell.noise`). Stage 0 measures it in each environment. On a hosted runner of GitHub at HZ 1000, the worst reset took 0.147 ms, thus δ_abs is one jiffy, 1 ms. These read it (`noiseOf`): the bands, the pairing of a program line with its kernel record, and the drop of a program.
+  - **The injector has its own bound.** A fault that the injector commits or lifts late makes a run void. That bound is `INJECT` (4 ms), not δ_abs. Stage 0 measures the commit apart. On a hosted runner it took up to 2.75 ms at 4 environments.
 - **Median equivalence.** The order-statistic interval of the median must be in the band, made 5 % wider.
 - **Extension.** A median across the edge of the band extends the cell from 11 runs to 22, then 44. Then the verdict is inconclusive.
 - **Recovery.** The recovery after a stall gets the same test as a detection, by its band. A detection is then a finding.

package/build/index.d.ts

@@ -426,7 +426,8 @@ interface Reconnects {
  * A cell whose outcome is a distribution, for example under netem loss, has no band. Its
  * prediction is `detects: null` with the reason, and its runs are measurements only. A
  * measurement-only cell that names a reference (`from`) also records the time from it to the end
- * of the connection, or to the last data.
+ * of the connection, or to the last data. It can also give the waits of the reconnect after that
+ * end (`reconnects`).
  *
  * For a group cell, the band is the first write that a primary accepts after the fault, and its
  * errno is `OK`. `group` tells which primary, and the stages of the group on the way.
@@ -483,8 +484,19 @@ type Prediction = {
   readonly window: Quantity;
   readonly why: string;
   readonly from?: Reference;
+  /**
+   * This choice ends the measured time at the close of the socket by the observer itself, the
+   * change from ESTABLISHED to FIN_WAIT1. Without it, the time ends at a report of the kernel or
+   * at the last data. Then an idle pool that closes its own socket gives no end.
+   */
+  readonly ends?: "own-close";
   readonly group?: GroupPrediction;
   readonly request?: RequestPrediction;
+  /**
+   * These are the waits of the reconnect after the end of the connection, from the document of
+   * the client. The end has no band, but each wait has one. A wait outside its band is a finding of its run.
+   */
+  readonly reconnects?: Reconnects;
 };
 /** A target is the server that a cell connects to: a stock image. */
 interface Target {
@@ -757,6 +769,12 @@ interface Cell {
    * many environments the machine holds before its noise increases.
    */
   readonly concurrency?: number;
+  /**
+   * This time is δ_abs of the environment of the cell: the noise of the pipeline. Each band includes
+   * it, and a timer must be at least that late to make a run void. Stage 0 measures it in each
+   * environment. Without it, the cell gets `NOISE`, the δ_abs of the first environment (4 ms).
+   */
+  readonly noise?: Quantity;
 }
 /** A hop place is a hop of a topology, with the side that the rules of a fault face. */
 interface HopPlace {
@@ -817,6 +835,11 @@ interface Run {
    * the run void. Null also makes the run void: no clock line came after the run to tell the count.
    */
   readonly lost: number | null;
+  /**
+   * This rate is the HZ of the kernel of the environment, from its manifest. With it, the rule of a
+   * late timer is in time. A run without it keeps the rule in jiffies, as the runs before 0.9.0.
+   */
+  readonly hz?: number;
   /** These are the addresses and the namespaces of a topology, which the measurement of a run reads. */
   readonly topology?: RunTopology;
   /** These are the lines of the routers of a topology from the start to the end of the run. */
@@ -905,9 +928,9 @@ export declare const leftAfter: (socket: readonly KernelRecord[], line: ProbeEve
  * A close closes the socket in δ_abs. A close frame is a write, and the program does not close
  * the socket at the drop. A peer that answers the close frame can close it, and that is no finding.
  */
-export declare const dropFinding: (expected: "frame" | "close", drop: ProgramEvent, left: Quantity | null) => string | null;
+export declare const dropFinding: (expected: "frame" | "close", drop: ProgramEvent, left: Quantity | null, delta?: Quantity) => string | null;
 /** This function gives each ping of the program on the connection of the run, with the kernel's stamp of its write. */
-export declare const pingsOf: (lines: readonly ProbeEvent[], socket: readonly KernelRecord[], local: unknown) => ProgramEvent[];
+export declare const pingsOf: (lines: readonly ProbeEvent[], socket: readonly KernelRecord[], local: unknown, delta?: Quantity) => ProgramEvent[];
 /**
  * This function gives the first ping that got no answer: no segment with data reached the socket
  * after its write, up to `until`. Its write starts the wait of the heartbeat that the fault met.
@@ -918,7 +941,7 @@ export declare const firstUnanswered: (pings: readonly ProgramEvent[], socket: r
  * write of a close frame. Or it stamps the change of the socket out of ESTABLISHED: a close or an
  * abort.
  */
-export declare const dropOf: (lines: readonly ProbeEvent[], socket: readonly KernelRecord[], local: unknown) => ProgramEvent | undefined;
+export declare const dropOf: (lines: readonly ProbeEvent[], socket: readonly KernelRecord[], local: unknown, delta?: Quantity) => ProgramEvent | undefined;
 /** A reconnect reading is what the client did after the detection, from its lines and from the records of the kernel. */
 interface ReconnectReading {
   /** These are the waits of the program before each try: from a `reconnecting` line to the next `dial` line. */
@@ -937,7 +960,7 @@ interface ReconnectReading {
  */
 export declare const reconnectOf: (run: Run, detected: number, cookie: number | undefined, from?: number) => ReconnectReading;
 /** This function tells how the waits of a reconnect differ from their bands, each widened by δ_abs, or gives null. */
-export declare const reconnectFinding: (expected: Reconnects, read: ReconnectReading) => string | null;
+export declare const reconnectFinding: (expected: Reconnects, read: ReconnectReading, delta?: Quantity) => string | null;
 /**
  * A call reading is what the call in flight of a WebSocket client met: its error, a reply, or
  * nothing in the run. The time counts from the detection to its end.
@@ -1321,6 +1344,23 @@ export declare const requestWords: (outcome: {
  * the harness, the kernel or Docker.
  */
 export declare const NOISE: Quantity;
+/**
+ * This time is how late the injector can commit or lift a fault before the run is void. Stage 0
+ * measures the commit apart from δ_abs, and the commit is longer. On a hosted runner it took up to
+ * 2.75 ms at 4 environments, where δ_abs is 1 ms. Thus a cell's δ_abs does not change it.
+ */
+export declare const INJECT: Quantity;
+/** This function gives the δ_abs of a cell: its own, or `NOISE` for a cell that gives none. */
+export declare const noiseOf: (cell: {
+  readonly noise?: Quantity | undefined;
+}) => Quantity;
+/**
+ * This function tells if a timer firing came late enough to void a run. A firing two jiffies or
+ * more behind its wheel slot came late because the host stopped a vCPU. With the HZ of the run,
+ * the rule is in time: the delay must also reach δ_abs, because each band already includes that
+ * noise. At HZ 250 two jiffies are 8 ms, more than a δ_abs of 4 ms. At HZ 1000 they are 2 ms.
+ */
+export declare const delayVoids: (behind: number, hz: number | undefined, noise: Quantity) => boolean;
 /** This fraction, δ_rel, is how far the median of a cell can be outside the prediction and stay equivalent. */
 export declare const RELATIVE = .05;
 /** These are the run counts of a cell: 11, then 22 and 44 while its median crosses the edge of the band. */
@@ -1472,9 +1512,9 @@ export declare const medianInterval: (sorted: readonly number[]) => [number, num
  * finding. The distribution test (a DKW band) needs a predicted distribution from a simulator,
  * which does not exist at this time.
  */
-export declare const verdictOf: (prediction: Prediction, runs: readonly Measured[]) => Verdict;
+export declare const verdictOf: (prediction: Prediction, runs: readonly Measured[], noise?: Quantity) => Verdict;
 /** This function sums what the group of a group cell did in its counted runs, and tests each stage that has a band. */
-export declare const groupVerdictOf: (prediction: Prediction, counted: readonly Measured[]) => GroupVerdict;
+export declare const groupVerdictOf: (prediction: Prediction, counted: readonly Measured[], delta?: Quantity) => GroupVerdict;
 //#endregion
 //#region src/lab.d.ts
 /** This function gives the image of the harness. */
@@ -1545,6 +1585,11 @@ interface Manifest {
   /** This text is the architecture of the machine, as uname -m gives it: x86_64. */
   readonly architecture: string;
   readonly hz: string | null;
+  /**
+   * This text tells where the HZ came from: the configuration of the kernel, or a measurement. A
+   * host without /proc/config.gz (a hosted runner) gets a measurement of jiffies against the clock.
+   */
+  readonly hzFrom?: "config" | "measured";
   readonly boot_id: string;
   readonly docker: string;
   /** This record is the target, and its settings as it read them back. */
@@ -1768,6 +1813,18 @@ export declare const routesOf: (hops: readonly Pick<Hop, "kind">[]) => Array<{
  * is not a router. Element 0 is the client, element k is hop k - 1, and the last element is the target.
  */
 export declare const endpointAfter: (hops: readonly Pick<Hop, "kind">[], element: number) => number;
+/**
+ * This function gives HZ from two readings of /proc/timer_list, one second apart: its `now at`
+ * time in ns and its jiffies. It gives the nearest rate that Linux offers, or null when the
+ * measurement is more than 5 % from each of them.
+ */
+export declare const hzOf: (first: {
+  now: number;
+  jiffies: number;
+}, second: {
+  now: number;
+  jiffies: number;
+}) => string | null;
 /** This function gives the device of a route from its line in `ip -o route get`: "10.0.0.2 dev eth0 src …". */
 export declare const deviceOf: (route: string) => string;
 /**

package/build/index.js

@@ -20,6 +20,25 @@ const requestWords = (outcome) => `${outcome.connection}${outcome.retried ? ", a
 * the harness, the kernel or Docker.
 */
 const NOISE = quantity(4, "ms");
+/**
+* This time is how late the injector can commit or lift a fault before the run is void. Stage 0
+* measures the commit apart from δ_abs, and the commit is longer. On a hosted runner it took up to
+* 2.75 ms at 4 environments, where δ_abs is 1 ms. Thus a cell's δ_abs does not change it.
+*/
+const INJECT = NOISE;
+/** This function gives the δ_abs of a cell: its own, or `NOISE` for a cell that gives none. */
+const noiseOf = (cell) => cell.noise ?? NOISE;
+/**
+* This function tells if a timer firing came late enough to void a run. A firing two jiffies or
+* more behind its wheel slot came late because the host stopped a vCPU. With the HZ of the run,
+* the rule is in time: the delay must also reach δ_abs, because each band already includes that
+* noise. At HZ 250 two jiffies are 8 ms, more than a δ_abs of 4 ms. At HZ 1000 they are 2 ms.
+*/
+const delayVoids = (behind, hz, noise) => {
+	if (behind < 2) return false;
+	if (hz === void 0 || !(hz > 0)) return true;
+	return behind / hz >= Math.max(2 / hz, valueIn(noise, "s"));
+};
 /** This fraction, δ_rel, is how far the median of a cell can be outside the prediction and stay equivalent. */
 const RELATIVE = .05;
 /** These are the run counts of a cell: 11, then 22 and 44 while its median crosses the edge of the band. */
@@ -116,15 +135,15 @@ const middle = (sorted) => {
 * finding. The distribution test (a DKW band) needs a predicted distribution from a simulator,
 * which does not exist at this time.
 */
-const verdictOf = (prediction, runs) => {
-	const band = verdictOfBand(prediction, runs);
+const verdictOf = (prediction, runs, noise = NOISE) => {
+	const band = verdictOfBand(prediction, runs, noise);
 	const counted = runs.filter((run) => run.void === null);
 	const asked = counted.some((run) => run.request !== void 0) ? withRequests(band, counted) : band;
 	const verdict = withProgram(asked, counted);
 	if (prediction.group === void 0 && !counted.some((run) => run.group !== void 0)) return verdict;
 	return {
 		...verdict,
-		group: groupVerdictOf(prediction, counted)
+		group: groupVerdictOf(prediction, counted, noise)
 	};
 };
 /**
@@ -184,8 +203,8 @@ const withRequests = (verdict, counted) => {
 /** This function gives the median of some counts, or null for none. */
 const medianCount = (values) => values.length === 0 ? null : middle([...values].sort((a, b) => a - b));
 /** This function sums what the group of a group cell did in its counted runs, and tests each stage that has a band. */
-const groupVerdictOf = (prediction, counted) => {
-	const noise = valueIn(NOISE, "s");
+const groupVerdictOf = (prediction, counted, delta = NOISE) => {
+	const noise = valueIn(delta, "s");
 	const stamps = prediction.group?.stamps === void 0 ? 0 : valueIn(prediction.group.stamps, "s");
 	const stages = (prediction.group?.stages ?? []).map((band) => {
 		const interval = {
@@ -261,7 +280,7 @@ const groupVerdictOf = (prediction, counted) => {
 	};
 };
 /** This function gives the verdict of the band of a cell: its containment and the equivalence of its median. */
-const verdictOfBand = (prediction, runs) => {
+const verdictOfBand = (prediction, runs, delta = NOISE) => {
 	const counted = runs.filter((run) => run.void === null);
 	const voided = runs.length - counted.length;
 	const outside = counted.filter((run) => run.finding !== null).map((run) => ({
@@ -310,13 +329,15 @@ const verdictOfBand = (prediction, runs) => {
 			}
 			const outcomes = [...kinds].map(([kind, n]) => `${n} ${kind}`).join(", ");
 			const timed = times.length === 0 ? "no time" : `median ${s(middle(times)).value} s from ${prediction.from}, range [${s(times[0]).value}, ${s(times.at(-1)).value}] s`;
+			const missed = outside.filter((each) => counted.some((run) => name(run) === each.run && run.reconnect !== void 0));
+			const late = missed.length === 0 ? "" : ` ${missed.length} runs had a wait of the reconnect outside its band.`;
 			return {
 				...empty,
 				decision: "measured",
 				band: null,
-				outside: [],
+				outside: missed,
 				median: times.length === 0 ? null : s(middle(times)),
-				why: `${counted.length} runs measured: ${outcomes}; ${timed}. ${prediction.why}`
+				why: `${counted.length} runs measured: ${outcomes}; ${timed}.${late} ${prediction.why}`
 			};
 		}
 		const samples = counted.flatMap((run) => (run.samples ?? []).map((sample) => valueIn(sample, "s"))).sort((a, b) => a - b);
@@ -355,7 +376,7 @@ const verdictOfBand = (prediction, runs) => {
 		};
 	}
 	const timed = prediction.detects ? prediction : prediction.recovers ?? prediction.replies;
-	const noise = valueIn(NOISE, "s");
+	const noise = valueIn(delta, "s");
 	const low = valueIn(timed.low, "s");
 	const high = valueIn(timed.high, "s");
 	const band = {
@@ -456,12 +477,12 @@ const onConnection = (line, local) => {
 	return (line["local"] ?? line["peer"]) === local;
 };
 /** This function gives the first record of a socket that matches, at the stamp of a line or in δ_abs after it. */
-const kernelAt = (socket, line, matches) => {
-	const noise = valueIn(NOISE, "s") * 1e9;
+const kernelAt = (socket, line, matches, delta) => {
+	const noise = valueIn(delta, "s") * 1e9;
 	return socket.find((record) => record.mono >= line.t && record.mono <= line.t + noise && matches(record));
 };
-const eventOf = (socket, line, matches) => {
-	const record = kernelAt(socket, line, matches);
+const eventOf = (socket, line, matches, delta = NOISE) => {
+	const record = kernelAt(socket, line, matches, delta);
 	return {
 		line,
 		at: record?.mono ?? line.t,
@@ -483,8 +504,8 @@ const leftAfter = (socket, line) => {
 * A close closes the socket in δ_abs. A close frame is a write, and the program does not close
 * the socket at the drop. A peer that answers the close frame can close it, and that is no finding.
 */
-const dropFinding = (expected, drop, left) => {
-	const noise = valueIn(NOISE, "s");
+const dropFinding = (expected, drop, left, delta = NOISE) => {
+	const noise = valueIn(delta, "s");
 	const soon = left !== null && valueIn(left, "s") <= noise;
 	if (expected === "close") {
 		if (left === null) return "the program did not close its socket after the drop, where a close is at once";
@@ -495,7 +516,7 @@ const dropFinding = (expected, drop, left) => {
 };
 const isWrite = (record) => record.ev === "tx";
 /** This function gives each ping of the program on the connection of the run, with the kernel's stamp of its write. */
-const pingsOf = (lines, socket, local) => lines.filter((line) => line.ev === "ping" && onConnection(line, local)).map((line) => eventOf(socket, line, isWrite));
+const pingsOf = (lines, socket, local, delta = NOISE) => lines.filter((line) => line.ev === "ping" && onConnection(line, local)).map((line) => eventOf(socket, line, isWrite, delta));
 /**
 * This function gives the first ping that got no answer: no segment with data reached the socket
 * after its write, up to `until`. Its write starts the wait of the heartbeat that the fault met.
@@ -506,9 +527,9 @@ const firstUnanswered = (pings, socket, until = Number.POSITIVE_INFINITY) => pin
 * write of a close frame. Or it stamps the change of the socket out of ESTABLISHED: a close or an
 * abort.
 */
-const dropOf = (lines, socket, local) => {
+const dropOf = (lines, socket, local, delta = NOISE) => {
 	const line = lines.find((each) => each.ev === "drop" && onConnection(each, local));
-	return line === void 0 ? void 0 : eventOf(socket, line, (record) => isWrite(record) || record.ev === "state" && oldState$2(record) === 1);
+	return line === void 0 ? void 0 : eventOf(socket, line, (record) => isWrite(record) || record.ev === "state" && oldState$2(record) === 1, delta);
 };
 /**
 * This function reads the reconnect of the client after the detection at `detected`. The socket of
@@ -539,8 +560,8 @@ const reconnectOf = (run, detected, cookie, from = detected) => {
 	};
 };
 /** This function tells how the waits of a reconnect differ from their bands, each widened by δ_abs, or gives null. */
-const reconnectFinding = (expected, read) => {
-	const noise = valueIn(NOISE, "s");
+const reconnectFinding = (expected, read, delta = NOISE) => {
+	const noise = valueIn(delta, "s");
 	const round = (value) => Math.round(value * 1e6) / 1e6;
 	const found = expected.waits.flatMap((band, at) => {
 		const wait = read.waits[at];
@@ -729,9 +750,9 @@ const settled = (cell, run) => {
 * that found the socket held by its owner put itself off by HZ/20. Either way, the run measured
 * something other than the timer.
 */
-const delayed = (place, from, to) => {
+const delayed = (place, from, to, hz, noise) => {
 	const firings = place.records.filter((record) => record.ev === "timer" && record.what === "keepalive" && record.mono >= from && record.mono <= to);
-	const late = firings.find((record) => (record.behind?.value ?? 0) >= 2);
+	const late = firings.find((record) => delayVoids(record.behind?.value ?? 0, hz, noise));
 	if (late !== void 0) return `its keepalive timer ran ${late.behind.value} jiffies behind its wheel slot: the VM delayed it, not the kernel`;
 	return firings.find((record) => record.owned === true) === void 0 ? null : "its keepalive timer found the socket held by its owner and put itself off by HZ/20: the run measured the owner, not the timer";
 };
@@ -805,13 +826,13 @@ const measureDriver = (cell, run) => {
 		if (heal === null) voided = "the fault never lifted";
 		else {
 			const late = (heal.t0 - run.fault.t0) / 1e9 - valueIn(cell.hold, "s");
-			if (late > valueIn(NOISE, "s")) voided = `the lift came ${Math.round(late * 1e4) / 10} ms after its hold: the injector was late, not the client`;
+			if (late > valueIn(INJECT, "s")) voided = `the lift came ${Math.round(late * 1e4) / 10} ms after its hold: the injector was late, not the client`;
 		}
 	}
 	const assumed = prediction.rtt;
 	if (voided === null && assumed !== void 0 && rtt !== null && valueIn(rtt, "ms") > valueIn(assumed.high, "ms")) voided = `rtt-over-assumption: its first round trip was ${rtt.value} ms, above the ${valueIn(assumed.high, "ms")} ms that the prediction assumes`;
 	const findings = [];
-	const noise = valueIn(NOISE, "s");
+	const noise = valueIn(noiseOf(cell), "s");
 	const until = prediction.detects ? Infinity : after + valueIn(prediction.window, "s") * 1e9;
 	for (const match of driver.quiet ?? []) {
 		const line = events.find((event) => event.t > after && event.t <= until && matches(event, match));
@@ -901,7 +922,7 @@ const measureDriver = (cell, run) => {
 		void: voided,
 		finding: `it ended without its reference event (${prediction.from} on the ${driver.socket} socket)`
 	};
-	if (driver.ends === void 0 && prediction.from === "last-received" && place !== void 0) voided ??= delayed(place, reference, ended);
+	if (driver.ends === void 0 && prediction.from === "last-received" && place !== void 0) voided ??= delayed(place, reference, ended, run.hz, noiseOf(cell));
 	const name = outcome.kind === "error" ? outcome.errno : outcome.kind === "eof" ? "EOF" : "silent";
 	if (prediction.errno !== "OK" && name !== prediction.errno) findings.unshift(`it ended with ${name}, not ${prediction.errno}`);
 	const seen = report === void 0 ? void 0 : events.find((event) => event.t >= report.at && (event.ev === "error" || event.ev === "socket-error"));
@@ -1224,14 +1245,14 @@ const measureGroup = (cell, run) => {
 		if (heal === null) voided = "the fault never lifted";
 		else {
 			const late = (heal.t0 - t0) / 1e9 - valueIn(cell.hold, "s");
-			if (late > valueIn(NOISE, "s")) voided = `the lift came ${Math.round(late * 1e4) / 10} ms after its hold: the injector was late, not the group`;
+			if (late > valueIn(INJECT, "s")) voided = `the lift came ${Math.round(late * 1e4) / 10} ms after its hold: the injector was late, not the group`;
 		}
 	}
 	const assumed = prediction.detects === null ? void 0 : prediction.rtt;
 	const first = kernel.find((record) => record.netns === run.netns && record.dst === `${run.group?.members[primary]?.ip}:${(cell.group ?? cell.replicaSet)?.port}` && record.tcp?.srtt !== void 0);
 	const rtt = first?.tcp?.srtt === void 0 ? null : valueIn(first.tcp.srtt, "us") / 1e3;
 	if (voided === null && assumed !== void 0 && rtt !== null && rtt > valueIn(assumed.high, "ms")) voided = `rtt-over-assumption: its first round trip was ${rtt} ms, above the ${valueIn(assumed.high, "ms")} ms that the prediction assumes`;
-	if (voided === null && prediction.detects === true && prediction.from === "fault" && (run.fault.t1 - run.fault.t0) / 1e6 > valueIn(NOISE, "ms")) voided = `the commit of the fault took ${Math.round((run.fault.t1 - run.fault.t0) / 1e5) / 10} ms, longer than the noise of the pipeline: its start is not known`;
+	if (voided === null && prediction.detects === true && prediction.from === "fault" && (run.fault.t1 - run.fault.t0) / 1e6 > valueIn(INJECT, "ms")) voided = `the commit of the fault took ${Math.round((run.fault.t1 - run.fault.t0) / 1e5) / 10} ms, longer than the noise of the pipeline: its start is not known`;
 	let reference = t0;
 	if (prediction.detects === false && prediction.recovers !== void 0) {
 		const into = events.find((event) => event.ev === "sent" && event.t > run.fault.t1);
@@ -1253,7 +1274,7 @@ const measureGroup = (cell, run) => {
 	if (accepted !== void 0 && lands === "same" && to !== primary) findings.push(`the client wrote to ${to ?? String(accepted["at"])}, where it should stay on ${primary}`);
 	for (const error of expected?.errors ?? []) {
 		const met = group.errors.find((each) => each.name === error.name);
-		const noise = valueIn(NOISE, "s");
+		const noise = valueIn(noiseOf(cell), "s");
 		if (met === void 0) findings.push(`the client never met ${error.name}`);
 		else if (met.first.value < valueIn(error.low, "s") - noise || met.first.value > valueIn(error.high, "s") + noise) findings.push(`the client met ${error.name} ${met.first.value} s after the fault, outside [${valueIn(error.low, "s")}, ${valueIn(error.high, "s")}] s`);
 	}
@@ -1537,7 +1558,7 @@ const measureSocket = (cell, run) => {
 	*/
 	const delayed = (from, to, timer) => {
 		const firings = socket.filter((record) => record.ev === "timer" && record.what === timer && record.mono >= from && record.mono <= to);
-		const late = firings.find((record) => (record.behind?.value ?? 0) >= 2);
+		const late = firings.find((record) => delayVoids(record.behind?.value ?? 0, run.hz, noiseOf(cell)));
 		if (late !== void 0) return `its ${timer} timer ran ${late.behind.value} jiffies behind its wheel slot: the VM delayed it, not the kernel`;
 		return firings.find((record) => timer === "keepalive" && record.owned === true) === void 0 ? null : "its keepalive timer found the socket held by its owner and put itself off by HZ/20: the run measured the owner, not the timer";
 	};
@@ -1600,10 +1621,10 @@ const measureSocket = (cell, run) => {
 	const reported = socket.find((record) => record.ev === "error" && (record.err ?? 0) !== 0);
 	const seen = remote ? void 0 : events.find((event) => event.ev === "error" || event.ev === "eof");
 	const closeWait = socket.find((record) => record.ev === "state" && newState(record) === 8);
-	const own = cell.prediction.detects === true && cell.prediction.errno === "CLOSED" ? socket.find((record) => record.ev === "state" && oldState(record) === 1 && newState(record) === 4) : void 0;
+	const own = cell.prediction.detects === true && cell.prediction.errno === "CLOSED" || cell.prediction.detects === null && cell.prediction.ends === "own-close" ? socket.find((record) => record.ev === "state" && oldState(record) === 1 && newState(record) === 4) : void 0;
 	const lines = programLines(cell, run);
 	const kernelEnded = reported?.mono ?? closeWait?.mono ?? own?.mono ?? seen?.t;
-	const first = dropOf(lines, socket, local);
+	const first = dropOf(lines, socket, local, noiseOf(cell));
 	const drop = first !== void 0 && (kernelEnded === void 0 || first.at <= kernelEnded) ? first : void 0;
 	const dropped = drop !== void 0;
 	const outcome = dropped ? { kind: "dropped" } : reported !== void 0 ? {
@@ -1616,7 +1637,7 @@ const measureSocket = (cell, run) => {
 	const ended = drop?.at ?? kernelEnded;
 	const noticed = reported !== void 0 && seen !== void 0 && !dropped ? between(reported.mono, seen.t) : null;
 	const { prediction } = cell;
-	const pings = pingsOf(lines, socket, local);
+	const pings = pingsOf(lines, socket, local, noiseOf(cell));
 	const before = run.fault?.t1 ?? ready.t;
 	const referenceAt = (from) => {
 		if (from === "request-sent") return request.t;
@@ -1655,10 +1676,14 @@ const measureSocket = (cell, run) => {
 				toward: "target"
 			}
 		}, run)) };
+		const reconnect = prediction.reconnects === void 0 || ended === void 0 ? void 0 : reconnectOf(run, ended, cookie, drop?.line.t ?? ended);
+		const late = reconnect === void 0 || prediction.reconnects === void 0 ? null : reconnectFinding(prediction.reconnects, reconnect, noiseOf(cell));
 		return {
 			...base,
 			...nothing,
 			...chain,
+			...reconnect === void 0 ? {} : { reconnect },
+			finding: late,
 			outcome: ended !== void 0 ? outcome : { kind: "answered" },
 			...from === void 0 || until === void 0 ? {} : { elapsed: between(from, until) },
 			noticed: prediction.from === void 0 ? null : noticed,
@@ -1752,7 +1777,7 @@ const measureSocket = (cell, run) => {
 				if (after !== null) finding = `a router forgot the flow ${after.value} s after the last segment, where it should keep it`;
 			} else if (after === null) finding = "no router forgot the flow in the window, where it should forget it";
 			else {
-				const noise = valueIn(NOISE, "s");
+				const noise = valueIn(noiseOf(cell), "s");
 				const low = valueIn(expected.low, "s") - noise;
 				const high = valueIn(expected.high, "s") + noise;
 				if (after.value < low || after.value > high) finding = `a router forgot the flow ${after.value} s after the last segment, outside [${Math.round(low * 1e6) / 1e6}, ${Math.round(high * 1e6) / 1e6}] s`;
@@ -1796,9 +1821,9 @@ const measureSocket = (cell, run) => {
 	const matches = prediction.errno === "EOF" ? outcome.kind === "eof" : prediction.errno === "CLOSED" ? outcome.kind === "closed" : prediction.errno === "DROPPED" ? outcome.kind === "dropped" : outcome.kind === "error" && outcome.errno === prediction.errno;
 	const words = outcome.kind === "error" ? outcome.errno : outcome.kind === "closed" ? "its own close" : outcome.kind === "dropped" ? "a drop by its own program" : "end of file";
 	const reconnect = prediction.reconnects === void 0 ? void 0 : reconnectOf(run, ended, cookie, drop?.line.t ?? ended);
-	const late = reconnect === void 0 || prediction.reconnects === void 0 ? null : reconnectFinding(prediction.reconnects, reconnect);
+	const late = reconnect === void 0 || prediction.reconnects === void 0 ? null : reconnectFinding(prediction.reconnects, reconnect, noiseOf(cell));
 	const left = prediction.drop === void 0 || drop === void 0 ? void 0 : leftAfter(socket, drop.line);
-	const how = prediction.drop === void 0 || drop === void 0 || left === void 0 ? null : dropFinding(prediction.drop, drop, left);
+	const how = prediction.drop === void 0 || drop === void 0 || left === void 0 ? null : dropFinding(prediction.drop, drop, left, noiseOf(cell));
 	const found = [
 		matches ? null : `it ended with ${words}, not ${prediction.errno}`,
 		late,
@@ -1812,7 +1837,7 @@ const measureSocket = (cell, run) => {
 	const slow = prediction.timer.startsWith("expire_") ? (run.middlebox ?? []).find((line) => line.ev === "pump-slow" && line.t <= ended && line.t + Number(line["took"]) * 1e9 >= ended - lag) : void 0;
 	if (slow !== void 0) voided ??= `a listing of the router took ${Math.round(Number(slow["took"]) * 1e4) / 10} ms, longer than the bound of its prediction`;
 	const commit = run.fault === null ? 0 : (run.fault.t1 - run.fault.t0) / 1e6;
-	if (prediction.from === "fault" && commit > valueIn(NOISE, "ms")) voided ??= `the commit of the fault took ${Math.round(commit * 10) / 10} ms, longer than the noise of the pipeline: its start is not known`;
+	if (prediction.from === "fault" && commit > valueIn(INJECT, "ms")) voided ??= `the commit of the fault took ${Math.round(commit * 10) / 10} ms, longer than the noise of the pipeline: its start is not known`;
 	const unanswered = prediction.from === "ping-sent" ? firstUnanswered(pings, socket, ended) : void 0;
 	const endLine = drop?.line ?? seen;
 	const view = unanswered !== void 0 ? endLine === void 0 ? null : between(unanswered.line.t, endLine.t) : remote ? null : socketVantage(cell, events, ready, request);
@@ -2088,6 +2113,42 @@ const endpointAfter = (hops, element) => {
 	return at;
 };
 const STOP = { timeout: 0 };
+/** These are the values of CONFIG_HZ that Linux offers. */
+const RATES = [
+	100,
+	250,
+	300,
+	1e3
+];
+/**
+* This function gives HZ from two readings of /proc/timer_list, one second apart: its `now at`
+* time in ns and its jiffies. It gives the nearest rate that Linux offers, or null when the
+* measurement is more than 5 % from each of them.
+*/
+const hzOf = (first, second) => {
+	const seconds = (second.now - first.now) / 1e9;
+	if (!(seconds > 0)) return null;
+	const rate = (second.jiffies - first.jiffies) / seconds;
+	const nearest = RATES.reduce((best, each) => Math.abs(each - rate) < Math.abs(best - rate) ? each : best);
+	return Math.abs(nearest - rate) / nearest <= .05 ? String(nearest) : null;
+};
+/** This function measures HZ in a privileged container: jiffies against the time of the kernel, one second apart. */
+const measuredHz = async (container) => {
+	const read = "awk '/^now at/ {now=$3} /^jiffies:/ {print now, $2; exit}' /proc/timer_list";
+	const values = (await output(container, [
+		"sh",
+		"-c",
+		`${read}; sleep 1; ${read}`
+	]).catch(() => "")).split(/\s+/).filter((each) => each !== "").map(Number);
+	if (values.length !== 4 || values.some((each) => !Number.isFinite(each))) return null;
+	return hzOf({
+		now: values[0],
+		jiffies: values[1]
+	}, {
+		now: values[2],
+		jiffies: values[3]
+	});
+};
 const output = async (container, argv) => {
 	const result = await container.exec(argv);
 	if (result.exitCode !== 0) throw new Error(`${argv.join(" ")} exited ${result.exitCode}: ${result.output.trim()}`);
@@ -2723,7 +2784,18 @@ const openLab = async (target, kind = "tcp", topology) => {
 				return {
 					kernel: await output(client, ["uname", "-r"]),
 					architecture: await output(client, ["uname", "-m"]),
-					hz: /^CONFIG_HZ=(\d+)$/m.exec(config)?.[1] ?? null,
+					...await (async () => {
+						const configured = /^CONFIG_HZ=(\d+)$/m.exec(config)?.[1];
+						if (configured !== void 0) return {
+							hz: configured,
+							hzFrom: "config"
+						};
+						const measured = await measuredHz(admin);
+						return measured === null ? { hz: null } : {
+							hz: measured,
+							hzFrom: "measured"
+						};
+					})(),
 					boot_id: await output(client, ["cat", "/proc/sys/kernel/random/boot_id"]),
 					docker: `${runtime.info.containerRuntime.serverVersion} on ${runtime.info.containerRuntime.operatingSystem}`,
 					target: {
@@ -4068,7 +4140,7 @@ const judge = (cell, runs) => {
 	const measured = runs.filter((run) => run.warmup !== true).map((run) => measure(cell, run));
 	return {
 		measured,
-		verdict: verdictOf(cell.prediction, measured)
+		verdict: verdictOf(cell.prediction, measured, noiseOf(cell))
 	};
 };
 /** This function gives the count of lost records in the latest clock line of the prober, since it began. */
@@ -4088,7 +4160,7 @@ const probeArgs = (cell, lab) => {
 	];
 	if (probe.ws !== void 0) {
 		const { ws } = probe;
-		const waits = (prediction.detects === true ? prediction.reconnects?.waits ?? [] : []).reduce((sum, wait) => sum + valueIn(wait.high, "s"), 0);
+		const waits = (prediction.detects !== false ? prediction.reconnects?.waits ?? [] : []).reduce((sum, wait) => sum + valueIn(wait.high, "s"), 0);
 		args.push("--websocket", ws.socket);
 		args.push("--heartbeat", ws.heartbeat === null ? "off" : "on");
 		if (ws.heartbeat !== null) args.push("--heartbeat-interval-ms", String(valueIn(ws.heartbeat.interval, "ms")), "--heartbeat-timeout-ms", String(valueIn(ws.heartbeat.timeout, "ms")), "--heartbeat-misses", String(ws.heartbeat.misses));
@@ -4465,7 +4537,7 @@ const once = async (lab, cell, environment, index, dry = false) => {
 				if (toxicFirst) await lab.toxic(null);
 			}
 			if (cell.fault === "freeze-process" && !server && cell.probe.mode !== "idle") await probe.until(["data"], 5e3).catch(() => void 0);
-			const expected = cell.prediction.detects === true ? cell.prediction.reconnects : void 0;
+			const expected = cell.prediction.detects !== false ? cell.prediction.reconnects : void 0;
 			if (expected !== void 0) {
 				const waits = expected.waits.reduce((sum, wait) => sum + valueIn(wait.high, "s"), 0);
 				await dials(probe, 1 + expected.waits.length, waits * 2e3 + 5e3);
@@ -4624,8 +4696,13 @@ const runCell = async (cell, say = () => {}) => {
 			const done = await Promise.all(batch.map(({ count, env }) => inEnvironment(cell, env, count, say)));
 			for (const each of done) {
 				manifests.push(each.manifest);
-				runs.push(...each.runs);
-				warmups.push(each.warmup);
+				const hz = each.manifest.hz === null ? void 0 : Number(each.manifest.hz);
+				const withHz = (run) => hz === void 0 ? run : {
+					...run,
+					hz
+				};
+				runs.push(...each.runs.map(withHz));
+				warmups.push(withHz(each.warmup));
 			}
 		}
 		const { measured, verdict } = judge(cell, runs);
@@ -4646,6 +4723,6 @@ const runCell = async (cell, say = () => {}) => {
 	}
 };
 //#endregion
-export { CONVERGE_MS, KEPT, LINGER_MS, LISTING, MEMBER_MARKS, NOISE, PUMP, QUIET_MS, RELATIVE, REPLICA_SET_STAGES, STAGES, STATUS_SCRIPT, STEPS, STEP_DOWN, callOf, clockOffsetOf, crossingsOf, cutOf, deadlineOf, deviceOf, dropFinding, dropOf, endpointAfter, errorsOf, fieldOf, fieldsOf, firstUnanswered, goHttpImage, groupOnce, groupVerdictOf, harnessImage, infoOf, initiateScript, isHopPlace, judge, lagsOf, leftAfter, matches, measure, measureDriver, measureGroup, medianInterval, memberCommand, memberConfig, memberLineOf, mongoClientImage, nodeClientImage, nodeHttpImage, onMonotonic, openGroupLab, openLab, openReplicaSetLab, peersOf, pingsOf, proceduresImage, programLines, quantileOf, reconnectFinding, reconnectOf, referenceOf, reorderingOf, replicaSetArgs, replicaSetOnce, replicaSetStagesOf, requestOf, requestWords, routerImage, routesOf, runCell, sentinelConfig, socketAt, socketOf, socketsOf, spread, spreadOf, stagesOf, statusViewOf, subscriptionsOf, targetImageOf, templated, toxicBody, verdictOf, viewOf, writesOf };
+export { CONVERGE_MS, INJECT, KEPT, LINGER_MS, LISTING, MEMBER_MARKS, NOISE, PUMP, QUIET_MS, RELATIVE, REPLICA_SET_STAGES, STAGES, STATUS_SCRIPT, STEPS, STEP_DOWN, callOf, clockOffsetOf, crossingsOf, cutOf, deadlineOf, delayVoids, deviceOf, dropFinding, dropOf, endpointAfter, errorsOf, fieldOf, fieldsOf, firstUnanswered, goHttpImage, groupOnce, groupVerdictOf, harnessImage, hzOf, infoOf, initiateScript, isHopPlace, judge, lagsOf, leftAfter, matches, measure, measureDriver, measureGroup, medianInterval, memberCommand, memberConfig, memberLineOf, mongoClientImage, nodeClientImage, nodeHttpImage, noiseOf, onMonotonic, openGroupLab, openLab, openReplicaSetLab, peersOf, pingsOf, proceduresImage, programLines, quantileOf, reconnectFinding, reconnectOf, referenceOf, reorderingOf, replicaSetArgs, replicaSetOnce, replicaSetStagesOf, requestOf, requestWords, routerImage, routesOf, runCell, sentinelConfig, socketAt, socketOf, socketsOf, spread, spreadOf, stagesOf, statusViewOf, subscriptionsOf, targetImageOf, templated, toxicBody, verdictOf, viewOf, writesOf };
 
 //# sourceMappingURL=index.js.map