@mark1russell7/harness 0.9.0 → 0.11.0

package/README.md

@@ -203,11 +203,14 @@ Each run starts from kernels that keep nothing from the run before it (`lab.forg
   - The second view is the stamps of the program: from its `ping` line to its `drop` line. The steps are the later pings, each from the reference.
   - The lines of a client name the socket by its local address (`local`), and the lines of a server by its peer (`peer`). For the server, the lines are the lines of the target. A line whose address is null is about a try that did not open.
   - A prediction can say how the program drops (`drop`). A close frame (`frame`) is a write, and the socket stays ESTABLISHED after it. A close (`close`) closes the socket in δ_abs: a FIN or an abort. Another drop is a finding. A FIN of the peer that answers a close frame is no close by the program. The run gives the time from the `drop` line to the close by the program (`left`).
+  - The client process ends at the end of each run, and its kernel then closes each socket that is still open. A prediction can say that the program is that client process (`exits`). Then a close in δ_abs of that end is no close by the program: `left` stays null, and `exited` gives the time.
+  - The end of the process is the first change out of ESTABLISHED of another socket of the client after its last `dial` line (`exitOf`). The program dropped none of these sockets.
   - A heartbeat is a timer of the program. The retransmit timer of its ping fires in the same interval. Thus a run whose retransmit timer was two jiffies or more behind its wheel slot is void: the VM stopped a vCPU.
 - **The reconnect** (`reconnectOf`). A prediction with `reconnects` also reads what the client did after the detection.
   - Each wait goes from a `reconnecting` line, where the program starts its timer, to its next `dial` line, where it makes the socket. A wait outside its band, widened by δ_abs, is a finding. A try that did not come is a finding too.
   - The kernel gives the time from the detection to the first SYN of a new socket (`syn`). It also gives the time to the first new socket in ESTABLISHED (`established`).
   - It also gives the time from each `dial` line to its SYN (`connects`): the work of the program before connect(2).
+  - A measurement-only prediction can give `reconnects` too. The end of its connection has no band, but each wait has one. A wait outside its band is a finding of its run, and the verdict names that run. The decision stays `measured`.
 - **A call in flight of a WebSocket client** (`callOf`). The run gives its error or its reply, and the time from the detection to its end.
 - **A stream.** In a stream (`probe.mode` is `stream`), the client sends `count` requests. The run keeps the time from each send to the last byte of its reply (`samples`). A record without `replied` lines gives the time to the first data.
 - **Reordering.** A measurement-only run also gives what reordering, duplicates and losses did (`reordering`). At the client, it counts the arrivals beyond a hole and the arrivals of data that it had already, against `rcv_nxt`. From the sender, it gives the most SACKed and lost segments, the DSACKs, the fast retransmissions, and the reordering degree and count of the kernel.
@@ -262,7 +265,7 @@ Each run starts from kernels that keep nothing from the run before it (`lab.forg
 - **An HTTP client** (`Verdict.requests`). The verdict counts each outcome of the second request over the counted runs. A measurement-only cell also gives them in its reason.
 - **The reconnect** (`Verdict.reconnect`). A wait outside its band is a finding of its run, thus the cell fails. The verdict gives the spread of each wait, of the first SYN, of the first ESTABLISHED and of the work before each connect(2). Its reason says the medians.
 - **A call in flight** (`Verdict.calls`). The verdict counts each outcome of the call in flight of a WebSocket client, and its reason says them.
-- **The socket after a drop** (`Verdict.left`). The verdict gives the spread of the time from a drop to the close of its socket by the program. It also counts the runs where the program did not close it.
+- **The socket after a drop** (`Verdict.left`). The verdict gives the spread of the time from a drop to the close of its socket by the program. It also counts the runs where the program did not close it. For the runs where the socket stayed open until the end of the client process, it gives the spread of that end (`exited`).
 
 The distribution test needs a simulator, which does not exist at this time.
 

package/build/index.d.ts

@@ -426,7 +426,8 @@ interface Reconnects {
  * A cell whose outcome is a distribution, for example under netem loss, has no band. Its
  * prediction is `detects: null` with the reason, and its runs are measurements only. A
  * measurement-only cell that names a reference (`from`) also records the time from it to the end
- * of the connection, or to the last data.
+ * of the connection, or to the last data. It can also give the waits of the reconnect after that
+ * end (`reconnects`).
  *
  * For a group cell, the band is the first write that a primary accepts after the fault, and its
  * errno is `OK`. `group` tells which primary, and the stages of the group on the way.
@@ -467,6 +468,12 @@ type Prediction = {
    * ESTABLISHED at once (`close`).
    */
   readonly drop?: "frame" | "close";
+  /**
+   * This flag tells that the program of the observer is the client process, which ends at the
+   * end of the run. Its kernel then closes each socket that is still open. A close of the
+   * socket at that end is not a close by the program, thus `left` stays null for it (`exited`).
+   */
+  readonly exits?: true;
 } | {
   readonly detects: false;
   readonly window: Quantity;
@@ -491,6 +498,11 @@ type Prediction = {
   readonly ends?: "own-close";
   readonly group?: GroupPrediction;
   readonly request?: RequestPrediction;
+  /**
+   * These are the waits of the reconnect after the end of the connection, from the document of
+   * the client. The end has no band, but each wait has one. A wait outside its band is a finding of its run.
+   */
+  readonly reconnects?: Reconnects;
 };
 /** A target is the server that a cell connects to: a stock image. */
 interface Target {
@@ -917,12 +929,29 @@ export declare const programLines: (cell: Cell, run: Run) => readonly ProbeEvent
  * the peer (CLOSE_WAIT) is not such a close. It is null when the socket did not close itself.
  */
 export declare const leftAfter: (socket: readonly KernelRecord[], line: ProbeEvent) => Quantity | null;
+/**
+ * This function gives the end of the client process in a run, as the kernel stamps it. The process
+ * ends after its last try, and its kernel then closes each socket that is still open, at one
+ * instant. Thus the end is the first change out of ESTABLISHED of another socket of the client
+ * after its last `dial` line. The program dropped none of these sockets. It is undefined when the
+ * record has no such change, for example when the last try did not open.
+ */
+export declare const exitOf: (run: Run, cookie: number | undefined) => number | undefined;
+/**
+ * This function separates a close of the socket by the program from the end of the client process.
+ * `left` is the time from the drop line to the close, and `exit` is the end of the process. A close
+ * in δ_abs of that end is the end, not the program: `left` becomes null, and `exited` keeps the time.
+ */
+export declare const closeOrExit: (line: ProbeEvent, left: Quantity | null, exit: number | undefined, delta?: Quantity) => {
+  readonly left: Quantity | null;
+  readonly exited?: Quantity;
+};
 /**
  * This function tells how a drop differs from the way that its prediction gives, or gives null.
  * A close closes the socket in δ_abs. A close frame is a write, and the program does not close
  * the socket at the drop. A peer that answers the close frame can close it, and that is no finding.
  */
-export declare const dropFinding: (expected: "frame" | "close", drop: ProgramEvent, left: Quantity | null, delta?: Quantity) => string | null;
+export declare const dropFinding: (expected: "frame" | "close", drop: ProgramEvent, left: Quantity | null, delta?: Quantity, exited?: Quantity) => string | null;
 /** This function gives each ping of the program on the connection of the run, with the kernel's stamp of its write. */
 export declare const pingsOf: (lines: readonly ProbeEvent[], socket: readonly KernelRecord[], local: unknown, delta?: Quantity) => ProgramEvent[];
 /**
@@ -1233,9 +1262,15 @@ interface Measured {
   /**
    * This time is from the drop line of the program to the close of its socket by its own side: a
    * FIN or an abort. A prediction that says how the program drops (`drop`) gets it. It is null
-   * when the socket did not close itself in the run.
+   * when the socket did not close itself in the run. A prediction can say that the client process
+   * ends with the run (`exits`). Then a close at that end is no close by the program, and `left` is null.
    */
   readonly left?: Quantity | null;
+  /**
+   * This time is from the drop line of the program to the end of the client process. A socket gets
+   * it when it stayed open until that end. The kernel closed it then, not the program (`exits`).
+   */
+  readonly exited?: Quantity;
   /** This text is the reason that the run cannot count, or null. A void run is not a pass and not a finding. */
   readonly void: string | null;
   /** This text tells what in the run disagrees with the prediction, other than its timing, or null. */
@@ -1415,10 +1450,13 @@ interface Verdict {
   /**
    * This record is the time from a drop of the program to the close of its socket by its own side,
    * over the counted runs. It also counts the runs where the socket did not close itself.
+   * `exited` is the spread of the end of the client process, for the runs where the socket stayed
+   * open until that end (`Measured.exited`).
    */
   readonly left?: {
     readonly spread: Spread | null;
     readonly stayed: number;
+    readonly exited?: Spread | null;
   };
   readonly why: string;
 }

package/build/index.js

@@ -166,12 +166,15 @@ const withProgram = (verdict, counted) => {
 	const outcomes = {};
 	for (const call of calls) outcomes[call.outcome] = (outcomes[call.outcome] ?? 0) + 1;
 	const gone = lefts.flatMap((each) => each === null ? [] : [each]);
+	const ends = counted.flatMap((run) => run.exited === void 0 ? [] : [run.exited]);
 	const left = lefts.length === 0 ? void 0 : {
 		spread: spreadOf(gone),
-		stayed: lefts.length - gone.length
+		stayed: lefts.length - gone.length,
+		...ends.length === 0 ? {} : { exited: spreadOf(ends) }
 	};
+	const ended = ends.length === 0 ? "" : `: in ${ends.length} the socket stayed open until the client process ended, median ${ms(spreadOf(ends))} after the drop line`;
 	const said = [
-		...left === void 0 ? [] : [`After the drop, the program closed its socket in ${gone.length} runs, median ${ms(left.spread)} after its drop line${left.stayed === 0 ? "" : `, and it did not close it in ${left.stayed}`}.`],
+		...left === void 0 ? [] : [`After the drop, the program closed its socket in ${gone.length} runs, median ${ms(left.spread)} after its drop line${left.stayed === 0 ? "" : `, and it did not close it in ${left.stayed}${ended}`}.`],
 		...reconnect === void 0 ? [] : [`Reconnect: the waits before each try, median ${reconnect.waits.map(ms).join(", ")}; the first SYN median ${ms(reconnect.syn)} and ESTABLISHED median ${ms(reconnect.established)} after the detection.`],
 		...calls.length === 0 ? [] : [`Calls in flight: ${Object.entries(outcomes).map(([outcome, n]) => `${n} ${outcome}`).join("; ")}.`]
 	];
@@ -329,13 +332,15 @@ const verdictOfBand = (prediction, runs, delta = NOISE) => {
 			}
 			const outcomes = [...kinds].map(([kind, n]) => `${n} ${kind}`).join(", ");
 			const timed = times.length === 0 ? "no time" : `median ${s(middle(times)).value} s from ${prediction.from}, range [${s(times[0]).value}, ${s(times.at(-1)).value}] s`;
+			const missed = outside.filter((each) => counted.some((run) => name(run) === each.run && run.reconnect !== void 0));
+			const late = missed.length === 0 ? "" : ` ${missed.length} runs had a wait of the reconnect outside its band.`;
 			return {
 				...empty,
 				decision: "measured",
 				band: null,
-				outside: [],
+				outside: missed,
 				median: times.length === 0 ? null : s(middle(times)),
-				why: `${counted.length} runs measured: ${outcomes}; ${timed}. ${prediction.why}`
+				why: `${counted.length} runs measured: ${outcomes}; ${timed}.${late} ${prediction.why}`
 			};
 		}
 		const samples = counted.flatMap((run) => (run.samples ?? []).map((sample) => valueIn(sample, "s"))).sort((a, b) => a - b);
@@ -498,15 +503,41 @@ const leftAfter = (socket, line) => {
 	return left === void 0 ? null : between$3(line.t, left.mono);
 };
 /**
+* This function gives the end of the client process in a run, as the kernel stamps it. The process
+* ends after its last try, and its kernel then closes each socket that is still open, at one
+* instant. Thus the end is the first change out of ESTABLISHED of another socket of the client
+* after its last `dial` line. The program dropped none of these sockets. It is undefined when the
+* record has no such change, for example when the last try did not open.
+*/
+const exitOf = (run, cookie) => {
+	const last = run.events.filter((line) => line.ev === "dial").at(-1);
+	if (last === void 0) return void 0;
+	const dropped = new Set(run.events.filter((line) => line.ev === "drop").map((line) => line["local"]));
+	return run.kernel.find((record) => record.netns === run.netns && record.cookie !== cookie && record.ev === "state" && oldState$2(record) === 1 && (newState$2(record) === 4 || newState$2(record) === 7) && record.mono >= last.t && !dropped.has(record.src))?.mono;
+};
+/**
+* This function separates a close of the socket by the program from the end of the client process.
+* `left` is the time from the drop line to the close, and `exit` is the end of the process. A close
+* in δ_abs of that end is the end, not the program: `left` becomes null, and `exited` keeps the time.
+*/
+const closeOrExit = (line, left, exit, delta = NOISE) => {
+	if (left === null || exit === void 0) return { left };
+	const at = line.t + valueIn(left, "s") * 1e9;
+	return Math.abs(at - exit) <= valueIn(delta, "s") * 1e9 ? {
+		left: null,
+		exited: left
+	} : { left };
+};
+/**
 * This function tells how a drop differs from the way that its prediction gives, or gives null.
 * A close closes the socket in δ_abs. A close frame is a write, and the program does not close
 * the socket at the drop. A peer that answers the close frame can close it, and that is no finding.
 */
-const dropFinding = (expected, drop, left, delta = NOISE) => {
+const dropFinding = (expected, drop, left, delta = NOISE, exited) => {
 	const noise = valueIn(delta, "s");
 	const soon = left !== null && valueIn(left, "s") <= noise;
 	if (expected === "close") {
-		if (left === null) return "the program did not close its socket after the drop, where a close is at once";
+		if (left === null) return `the program did not close its socket after the drop, where a close is at once${exited === void 0 ? "" : ": it stayed open until the client process ended"}`;
 		return soon ? null : `the program closed its socket ${left.value} s after the drop, not at once`;
 	}
 	if (drop.record !== "write") return "the drop wrote no close frame on its socket";
@@ -1674,10 +1705,14 @@ const measureSocket = (cell, run) => {
 				toward: "target"
 			}
 		}, run)) };
+		const reconnect = prediction.reconnects === void 0 || ended === void 0 ? void 0 : reconnectOf(run, ended, cookie, drop?.line.t ?? ended);
+		const late = reconnect === void 0 || prediction.reconnects === void 0 ? null : reconnectFinding(prediction.reconnects, reconnect, noiseOf(cell));
 		return {
 			...base,
 			...nothing,
 			...chain,
+			...reconnect === void 0 ? {} : { reconnect },
+			finding: late,
 			outcome: ended !== void 0 ? outcome : { kind: "answered" },
 			...from === void 0 || until === void 0 ? {} : { elapsed: between(from, until) },
 			noticed: prediction.from === void 0 ? null : noticed,
@@ -1816,8 +1851,12 @@ const measureSocket = (cell, run) => {
 	const words = outcome.kind === "error" ? outcome.errno : outcome.kind === "closed" ? "its own close" : outcome.kind === "dropped" ? "a drop by its own program" : "end of file";
 	const reconnect = prediction.reconnects === void 0 ? void 0 : reconnectOf(run, ended, cookie, drop?.line.t ?? ended);
 	const late = reconnect === void 0 || prediction.reconnects === void 0 ? null : reconnectFinding(prediction.reconnects, reconnect, noiseOf(cell));
-	const left = prediction.drop === void 0 || drop === void 0 ? void 0 : leftAfter(socket, drop.line);
-	const how = prediction.drop === void 0 || drop === void 0 || left === void 0 ? null : dropFinding(prediction.drop, drop, left, noiseOf(cell));
+	const closed = prediction.drop === void 0 || drop === void 0 ? void 0 : leftAfter(socket, drop.line);
+	const { left, exited } = closed === void 0 || drop === void 0 || prediction.exits !== true ? {
+		left: closed,
+		exited: void 0
+	} : closeOrExit(drop.line, closed, exitOf(run, cookie), noiseOf(cell));
+	const how = prediction.drop === void 0 || drop === void 0 || left === void 0 ? null : dropFinding(prediction.drop, drop, left, noiseOf(cell), exited);
 	const found = [
 		matches ? null : `it ended with ${words}, not ${prediction.errno}`,
 		late,
@@ -1845,6 +1884,7 @@ const measureSocket = (cell, run) => {
 		steps,
 		...reconnect === void 0 ? {} : { reconnect },
 		...left === void 0 ? {} : { left },
+		...exited === void 0 ? {} : { exited },
 		...cell.probe.ws !== void 0 && inflight ? { call: callOf(events, request, ended) } : {},
 		void: voided,
 		finding
@@ -4154,7 +4194,7 @@ const probeArgs = (cell, lab) => {
 	];
 	if (probe.ws !== void 0) {
 		const { ws } = probe;
-		const waits = (prediction.detects === true ? prediction.reconnects?.waits ?? [] : []).reduce((sum, wait) => sum + valueIn(wait.high, "s"), 0);
+		const waits = (prediction.detects !== false ? prediction.reconnects?.waits ?? [] : []).reduce((sum, wait) => sum + valueIn(wait.high, "s"), 0);
 		args.push("--websocket", ws.socket);
 		args.push("--heartbeat", ws.heartbeat === null ? "off" : "on");
 		if (ws.heartbeat !== null) args.push("--heartbeat-interval-ms", String(valueIn(ws.heartbeat.interval, "ms")), "--heartbeat-timeout-ms", String(valueIn(ws.heartbeat.timeout, "ms")), "--heartbeat-misses", String(ws.heartbeat.misses));
@@ -4531,7 +4571,7 @@ const once = async (lab, cell, environment, index, dry = false) => {
 				if (toxicFirst) await lab.toxic(null);
 			}
 			if (cell.fault === "freeze-process" && !server && cell.probe.mode !== "idle") await probe.until(["data"], 5e3).catch(() => void 0);
-			const expected = cell.prediction.detects === true ? cell.prediction.reconnects : void 0;
+			const expected = cell.prediction.detects !== false ? cell.prediction.reconnects : void 0;
 			if (expected !== void 0) {
 				const waits = expected.waits.reduce((sum, wait) => sum + valueIn(wait.high, "s"), 0);
 				await dials(probe, 1 + expected.waits.length, waits * 2e3 + 5e3);
@@ -4717,6 +4757,6 @@ const runCell = async (cell, say = () => {}) => {
 	}
 };
 //#endregion
-export { CONVERGE_MS, INJECT, KEPT, LINGER_MS, LISTING, MEMBER_MARKS, NOISE, PUMP, QUIET_MS, RELATIVE, REPLICA_SET_STAGES, STAGES, STATUS_SCRIPT, STEPS, STEP_DOWN, callOf, clockOffsetOf, crossingsOf, cutOf, deadlineOf, delayVoids, deviceOf, dropFinding, dropOf, endpointAfter, errorsOf, fieldOf, fieldsOf, firstUnanswered, goHttpImage, groupOnce, groupVerdictOf, harnessImage, hzOf, infoOf, initiateScript, isHopPlace, judge, lagsOf, leftAfter, matches, measure, measureDriver, measureGroup, medianInterval, memberCommand, memberConfig, memberLineOf, mongoClientImage, nodeClientImage, nodeHttpImage, noiseOf, onMonotonic, openGroupLab, openLab, openReplicaSetLab, peersOf, pingsOf, proceduresImage, programLines, quantileOf, reconnectFinding, reconnectOf, referenceOf, reorderingOf, replicaSetArgs, replicaSetOnce, replicaSetStagesOf, requestOf, requestWords, routerImage, routesOf, runCell, sentinelConfig, socketAt, socketOf, socketsOf, spread, spreadOf, stagesOf, statusViewOf, subscriptionsOf, targetImageOf, templated, toxicBody, verdictOf, viewOf, writesOf };
+export { CONVERGE_MS, INJECT, KEPT, LINGER_MS, LISTING, MEMBER_MARKS, NOISE, PUMP, QUIET_MS, RELATIVE, REPLICA_SET_STAGES, STAGES, STATUS_SCRIPT, STEPS, STEP_DOWN, callOf, clockOffsetOf, closeOrExit, crossingsOf, cutOf, deadlineOf, delayVoids, deviceOf, dropFinding, dropOf, endpointAfter, errorsOf, exitOf, fieldOf, fieldsOf, firstUnanswered, goHttpImage, groupOnce, groupVerdictOf, harnessImage, hzOf, infoOf, initiateScript, isHopPlace, judge, lagsOf, leftAfter, matches, measure, measureDriver, measureGroup, medianInterval, memberCommand, memberConfig, memberLineOf, mongoClientImage, nodeClientImage, nodeHttpImage, noiseOf, onMonotonic, openGroupLab, openLab, openReplicaSetLab, peersOf, pingsOf, proceduresImage, programLines, quantileOf, reconnectFinding, reconnectOf, referenceOf, reorderingOf, replicaSetArgs, replicaSetOnce, replicaSetStagesOf, requestOf, requestWords, routerImage, routesOf, runCell, sentinelConfig, socketAt, socketOf, socketsOf, spread, spreadOf, stagesOf, statusViewOf, subscriptionsOf, targetImageOf, templated, toxicBody, verdictOf, viewOf, writesOf };
 
 //# sourceMappingURL=index.js.map