@mark1russell7/harness 0.11.21 → 0.11.23

package/README.md

@@ -125,6 +125,9 @@ A step-down (`step-down`) moves the leadership to the first follower (`rabbitmq-
 - **An HTTP client** (`Cell.client`: `go-http`, `http-agent` or `undici`) is a client of the harness from its own image. It sends a GET of the root, then a second request (`probe.http`):
   - after the signal (`inflight`), into the fault
   - or by itself, a gap after its first reply (`gap`), for the keep-alive race.
+- **A client of a keyed limiter.** The flag `limiter-key` (`probe.http.options`) makes the client of undici or of Go send a key of its process in the header `x-limiter-key`. Thus each run has its own count in the limiter of the target.
+  - The flag `retry` sends each request of undici through its `RetryAgent`, at the defaults of undici. Each `retry-` flag sets one option over them.
+  - The client of undici then writes an `attempt` line for each send of a request, with its number `n` and its socket. A `RetryAgent` that gives up gives `UND_ERR_REQ_RETRY` on the `error` line, with the status of the last reply and the count of sends.
 - **The lines of an HTTP client.** It writes the lines of the TCP client, with the number `i` of the request on each line of a request. It also writes the connection of each try of a request (`request`, then `retry`), with its local address and a flag for a reused connection. Each connection gets a `conn` line, and each end of a connection a line (`eof`, `conn-error`, `closed`). The end of the second request is a `done` line.
 - **A target with a script.** A target can take files before it starts (`Target.files`), for example the script of a Node server on the stock Node image. A target that writes JSON lines (`Target.lines`) gives them to each run (`Run.server`). The manifest holds the sha256 of each file and the first line of the target.
 - **The WebSocket client** (`Cell.client` is `procedures-ws`) is the `WebSocketTransport` of `@procedures/client` 0.7.0: the `client` program of the `procedures` subject. The cell gives each of its settings (`probe.ws`): the class of its sockets, the heartbeat, the reconnect, and the times of a call.
@@ -330,6 +333,11 @@ Each run starts from kernels that keep nothing from the run before it (`lab.forg
 - **A driver band** (`measureDriver`, `driver.ts`). A driver with more than one socket names the socket of its band (`DriverBand.socket`).
   - The band counts from its reference on that socket. It ends at a line of the client (`ends`), or at the report of the kernel on the socket.
   - A band that a line ends is a timer of the driver. It counts from the last segment with data that the socket received or sent. The idle timer of a Node socket starts again at each read and each write of data.
+  - A band can name that segment as its reference (`last-data`). For example, an idle gRPC client goes to IDLE at a GOAWAY of its proxy. The idle timer of the proxy counts from the last data of the stream.
+  - A reference that this judge does not know (for example `ping-sent`) does not come. Thus its band is a finding. Before 0.11.22, such a reference counted from the first close of a socket of the peer. Through a proxy, that close can be the idle close of a socket toward the target. Since 0.11.23, the TCP judge does the same.
+  - A retry after a refusal counts from that refusal (`last-reply`), for example a 429 with `Retry-After`. That is the last segment with data that a socket of the client received before the line that ends the band. Each band gets its own, thus each retry counts from the refusal before it.
+  - The retry can go on a new connection. For example, undici closes an idle socket 3 s after a reply of a server at its defaults, before a retry 30 s later. Thus `last-reply` reads each socket of the client.
+  - The bands from `last-reply` follow one another. Thus the deadline of the client and the wait of the run hold their sum: five retries of 30 s take 150 s (`longestOf`).
   - The MongoDB client is ready when its first write has its reply and its monitor streams. The monitor sends its first awaited hello `minHeartbeatFrequencyMS` (500 ms) after its first check. A fault before that hello meets the start of the stream, not the stream that a cell measures.
   - Each later band (`then`) has its own reference and line. A later band outside its band, or absent, is a finding. So is a line that the prediction names as quiet, after the fault.
   - For a silent prediction, a flow of an open socket of the driver that a router forgot is a finding.

package/build/{cell-BxBZUssi.d.ts → cell-Dn1TvJcA.d.ts}

@@ -312,8 +312,13 @@ interface Toxic {
  *
  * A connect timer counts from the change of its socket to SYN_SENT (`syn-sent`). A pooler connects
  * to the server by itself, thus the kernel stamps that connect, and no line of the client does.
+ *
+ * A retry of a client after a refusal counts from the refusal (`last-reply`), for example a 429 with
+ * `Retry-After`. That is the last segment with data that a socket of the client received before
+ * the line that ends the band. The retry can go on a new connection. Each band of a driver has
+ * its own, thus each retry counts from the refusal before it.
  */
-type Reference = "last-received" | "last-data" | "first-retransmission" | "request-sent" | "peer-closed" | "fault" | "client-start" | "ping-sent" | "first-sent" | "syn-sent";
+type Reference = "last-received" | "last-reply" | "last-data" | "first-retransmission" | "request-sent" | "peer-closed" | "fault" | "client-start" | "ping-sent" | "first-sent" | "syn-sent";
 /**
  * This prediction tells when a connection moves again after a stall. That occurs at the first
  * firing of its retransmission timer after the stall stops, and its reply comes back one round
@@ -1123,4 +1128,4 @@ interface RunTopology {
 }
 //#endregion
 export { Subject as $, Prediction as A, RequestPrediction as B, KernelRecord as C, MiddleboxForm as D, MiddleboxExpectation as E, Recovers as F, RunPart as G, RttRange as H, Reference as I, SentinelStage as J, RunTopology as K, ReplicaSet as L, ProxyHop as M, ProxyTime as N, Observer as O, Reconnects as P, StageBand as Q, ReplicaSetStage as R, ImageSource as S, MiddleboxEvent as T, Run as U, RouterHop as V, RunGroup as W, ShippedImage as X, Shipped as Y, Stage as Z, Hop as _, Cluster as a, latestEndOf as at, HopSignal as b, DriverBand as c, FaultPlace as d, Target as et, Group as f, HarnessImage as g, GroupRole as h, ClientImage as i, isHopPlace as it, ProbeEvent as j, PinnedImage as k, DriverStep as l, GroupPrediction as m, Cell as n, Toxic as nt, ClusterClientAt as o, GroupPlace as p, SentinelEvent as q, Client as r, WsProbe as rt, ClusterStage as s, BuildImage as t, Topology as tt, Fault as u, HopEvent as v, LineMatch as w, HttpProbe as x, HopPlace as y, Replies as z };
-//# sourceMappingURL=cell-BxBZUssi.d.ts.map
+//# sourceMappingURL=cell-Dn1TvJcA.d.ts.map

package/build/index.d.ts

@@ -1,4 +1,4 @@
-import { $ as Subject, A as Prediction, B as RequestPrediction, C as KernelRecord, D as MiddleboxForm, E as MiddleboxExpectation, F as Recovers, G as RunPart, H as RttRange, I as Reference, J as SentinelStage, K as RunTopology, L as ReplicaSet, M as ProxyHop, N as ProxyTime, O as Observer, P as Reconnects, Q as StageBand, R as ReplicaSetStage, S as ImageSource, T as MiddleboxEvent, U as Run, V as RouterHop, W as RunGroup, X as ShippedImage, Y as Shipped, Z as Stage, _ as Hop, a as Cluster, at as latestEndOf, b as HopSignal, c as DriverBand, d as FaultPlace, et as Target, f as Group, g as HarnessImage, h as GroupRole, i as ClientImage, it as isHopPlace, j as ProbeEvent, k as PinnedImage, l as DriverStep, m as GroupPrediction, n as Cell, nt as Toxic, o as ClusterClientAt, p as GroupPlace, q as SentinelEvent, r as Client, rt as WsProbe, s as ClusterStage, t as BuildImage, tt as Topology, u as Fault, v as HopEvent, w as LineMatch, x as HttpProbe, y as HopPlace, z as Replies } from "./cell-BxBZUssi.js";
+import { $ as Subject, A as Prediction, B as RequestPrediction, C as KernelRecord, D as MiddleboxForm, E as MiddleboxExpectation, F as Recovers, G as RunPart, H as RttRange, I as Reference, J as SentinelStage, K as RunTopology, L as ReplicaSet, M as ProxyHop, N as ProxyTime, O as Observer, P as Reconnects, Q as StageBand, R as ReplicaSetStage, S as ImageSource, T as MiddleboxEvent, U as Run, V as RouterHop, W as RunGroup, X as ShippedImage, Y as Shipped, Z as Stage, _ as Hop, a as Cluster, at as latestEndOf, b as HopSignal, c as DriverBand, d as FaultPlace, et as Target, f as Group, g as HarnessImage, h as GroupRole, i as ClientImage, it as isHopPlace, j as ProbeEvent, k as PinnedImage, l as DriverStep, m as GroupPrediction, n as Cell, nt as Toxic, o as ClusterClientAt, p as GroupPlace, q as SentinelEvent, r as Client, rt as WsProbe, s as ClusterStage, t as BuildImage, tt as Topology, u as Fault, v as HopEvent, w as LineMatch, x as HttpProbe, y as HopPlace, z as Replies } from "./cell-Dn1TvJcA.js";
 import { Quantity } from "@mark1russell7/quantity";
 import "testcontainers";
 //#region src/application.d.ts
@@ -130,11 +130,16 @@ export declare const socketAt: (events: readonly ProbeEvent[], role: string, at:
  *   the driver. It counts from the last segment with data that the socket received or sent.
  * - `request-sent`: the send of the operation into the fault, the second `sent` line.
  * - `first-retransmission`: the first retransmission of the RTO timer on the socket.
+ * - `last-data`: the last segment with data that the socket received or sent before the fault
+ *   held, for a band of the kernel too. An idle timer of a peer counts from the end of a stream.
  * - `peer-closed`: the change of state of the socket of the target, when its process dies.
  * - `fault`: the start of the commit of the fault. `client-start`: the `connecting` line.
  * - `first-sent`: the first segment with data that the socket sends after the fault.
+ * - `last-reply`: the refusal that a retry follows. It is the last segment with data that a socket
+ *   of the client received before the end of the band (`end`). The retry can go on another socket.
+ * A reference that this judge does not know (`ping-sent`) does not come: its band is a finding.
  */
-export declare const referenceOf: (run: Run, from: Reference, role: string, data: boolean) => number | undefined;
+export declare const referenceOf: (run: Run, from: Reference, role: string, data: boolean, end?: number) => number | undefined;
 /**
  * This function reads one run of a driver cell.
  * - The band of the cell counts from its reference on its socket. It ends at its line, or at the

package/build/index.js

@@ -736,20 +736,27 @@ const outsetOf = (run) => {
 *   the driver. It counts from the last segment with data that the socket received or sent.
 * - `request-sent`: the send of the operation into the fault, the second `sent` line.
 * - `first-retransmission`: the first retransmission of the RTO timer on the socket.
+* - `last-data`: the last segment with data that the socket received or sent before the fault
+*   held, for a band of the kernel too. An idle timer of a peer counts from the end of a stream.
 * - `peer-closed`: the change of state of the socket of the target, when its process dies.
 * - `fault`: the start of the commit of the fault. `client-start`: the `connecting` line.
 * - `first-sent`: the first segment with data that the socket sends after the fault.
+* - `last-reply`: the refusal that a retry follows. It is the last segment with data that a socket
+*   of the client received before the end of the band (`end`). The retry can go on another socket.
+* A reference that this judge does not know (`ping-sent`) does not come: its band is a finding.
 */
-const referenceOf = (run, from, role, data) => {
+const referenceOf = (run, from, role, data, end = Infinity) => {
 	const ready = run.events.find((event) => event.ev === "ready");
 	const before = run.fault?.t1 ?? ready?.t ?? Infinity;
 	if (from === "fault") return run.fault?.t0;
 	if (from === "client-start") return (run.events.find((event) => event.ev === "connecting") ?? run.events[0])?.t;
 	if (from === "request-sent") return run.events.filter((event) => event.ev === "sent")[1]?.t;
+	if (from === "last-reply") return run.kernel.filter((record) => record.netns === run.netns && record.mono < end && record.ev === "rx" && (record.seg?.len.value ?? 0) > 0).at(-1)?.mono;
 	const { socket, place } = kernelOf(run, role, before);
 	if (place === void 0) return void 0;
-	if (from === "last-received") {
-		const counts = (record) => data ? (record.ev === "rx" || record.ev === "tx") && (record.seg?.len.value ?? 0) > 0 : record.ev === "rx";
+	if (from === "last-received" || from === "last-data") {
+		const withData = data || from === "last-data";
+		const counts = (record) => withData ? (record.ev === "rx" || record.ev === "tx") && (record.seg?.len.value ?? 0) > 0 : record.ev === "rx";
 		return place.records.filter((record) => record.mono <= before && counts(record)).at(-1)?.mono;
 	}
 	if (from === "first-sent") return place.records.find((record) => record.ev === "tx" && record.mono > before && (record.seg?.len.value ?? 0) > 0)?.mono;
@@ -758,6 +765,7 @@ const referenceOf = (run, from, role, data) => {
 		const request = run.events.filter((event) => event.ev === "sent")[1]?.t ?? before;
 		return place.records.find((record) => record.ev === "retrans" && record.why === "rto" && record.mono > request)?.mono;
 	}
+	if (from !== "peer-closed") return void 0;
 	const closed = (socket === void 0 ? void 0 : peerPlace(run, socket.local))?.records.find((record) => record.ev === "state" && oldState$1(record) === 1)?.mono;
 	if (closed !== void 0 || run.topology === void 0) return closed;
 	return proxiedClose(run);
@@ -807,13 +815,19 @@ const reportOf = (place, after) => {
 		outcome: { kind: "eof" }
 	};
 };
-/** This function gives the latest instant of the bands of a driver prediction, in seconds from their references. */
+/**
+* This function gives the latest instant of the bands of a driver prediction, in seconds from their
+* references. A band from the refusal before it (`last-reply`) follows the band before it. Thus the
+* times of those bands add: five retries of 30 s end 150 s after the first refusal.
+*/
 const longestOf = (cell) => {
 	const { prediction } = cell;
 	if (prediction.detects === null) return valueIn(prediction.window, "s");
-	const steps = (prediction.driver?.then ?? []).map((step) => valueIn(step.high, "s"));
+	const then = prediction.driver?.then ?? [];
+	const steps = then.map((step) => valueIn(step.high, "s"));
 	const own = prediction.detects ? valueIn(prediction.high, "s") : valueIn(prediction.window, "s");
-	return Math.max(own, ...steps);
+	const chained = [...prediction.detects && prediction.from === "last-reply" ? [own] : [], ...then.filter((step) => step.from === "last-reply").map((step) => valueIn(step.high, "s"))];
+	return Math.max(own, ...steps, chained.reduce((total, high) => total + high, 0));
 };
 /**
 * This function tells if a run of a driver cell has its outcome, so that its wait can stop. For a
@@ -951,7 +965,7 @@ const measureDriver = (cell, run) => {
 	}
 	const stepTimes = (driver.then ?? []).map((step) => {
 		const line = lineAfter(events, step.ends, after);
-		const from = referenceOf(run, step.from, step.socket ?? driver.socket, true);
+		const from = referenceOf(run, step.from, step.socket ?? driver.socket, true, line?.t);
 		if (line === void 0 || from === void 0) {
 			findings.push(line === void 0 ? `${step.what}: ${wordsOf(step.ends)} never came` : `${step.what}: its reference (${step.from}) never came`);
 			return {
@@ -981,7 +995,7 @@ const measureDriver = (cell, run) => {
 			finding: findings[0] ?? null
 		};
 		const line = lineAfter(events, driver.reply.ends, after);
-		const from = referenceOf(run, driver.reply.from, driver.socket, true);
+		const from = referenceOf(run, driver.reply.from, driver.socket, true, line?.t);
 		if (line === void 0 || from === void 0) return {
 			...base,
 			...nothing,
@@ -1020,7 +1034,7 @@ const measureDriver = (cell, run) => {
 		void: voided,
 		finding: findings[0] ?? `nothing detected it: ${prediction.timer} never fired`
 	};
-	const reference = referenceOf(run, prediction.from, driver.socket, driver.ends !== void 0);
+	const reference = referenceOf(run, prediction.from, driver.socket, driver.ends !== void 0, ended);
 	if (reference === void 0) return {
 		...base,
 		...nothing,
@@ -1483,7 +1497,7 @@ const count = (event, field) => event === void 0 ? 0 : Number(event[field] ?? 0)
 /** This function gives the view of the client on the same interval: TCP_INFO, with a sample each 2 ms. */
 const socketVantage = (cell, events, ready, request) => {
 	const { from } = cell.prediction.detects === true ? cell.prediction : { from: void 0 };
-	if (from === void 0 || from === "peer-closed" || from === "fault" || from === "ping-sent" || from === "first-sent" || from === "last-data") return null;
+	if (from === void 0 || from === "peer-closed" || from === "fault" || from === "ping-sent" || from === "first-sent" || from === "last-data" || from === "last-reply") return null;
 	const ended = events.find((event) => event.ev === "error" || event.ev === "eof");
 	const reference = from === "last-received" ? ready : from === "request-sent" ? request : events.find((event) => event.ev === "kernel" && event.t >= request.t && count(event, "retransmits") > 0);
 	return ended === void 0 || reference === void 0 ? null : between(reference.t, ended.t);
@@ -1824,7 +1838,7 @@ const measureSocket = (cell, run) => {
 		if (from === "fault") return run.fault?.t0;
 		if (from === "client-start") return (events.find((event) => event.ev === "connecting") ?? events[0])?.t;
 		if (from === "ping-sent") return firstUnanswered(pings, socket, ended)?.at;
-		const record = from === "last-received" ? socket.filter((each) => each.ev === "rx" && each.mono <= before).at(-1) : from === "last-data" ? socket.filter((each) => each.ev === "rx" && (each.seg?.len.value ?? 0) > 0 && each.mono <= (ended ?? before)).at(-1) : from === "first-retransmission" ? retransmitted.find((each) => each.why === "rto" && each.mono > before) : from === "first-sent" ? socket.find((each) => each.ev === "tx" && each.mono > before && (each.seg?.len.value ?? 0) > 0) : from === "syn-sent" ? socket.find((each) => each.ev === "state" && newState(each) === 2) : kernel.find((each) => each.netns === peerNetns && each.cookie === peerCookie && each.ev === "state" && oldState(each) === 1);
+		const record = from === "last-received" ? socket.filter((each) => each.ev === "rx" && each.mono <= before).at(-1) : from === "last-data" ? socket.filter((each) => each.ev === "rx" && (each.seg?.len.value ?? 0) > 0 && each.mono <= (ended ?? before)).at(-1) : from === "last-reply" ? socket.filter((each) => each.ev === "rx" && (each.seg?.len.value ?? 0) > 0 && each.mono < (ended ?? Infinity)).at(-1) : from === "first-retransmission" ? retransmitted.find((each) => each.why === "rto" && each.mono > before) : from === "first-sent" ? socket.find((each) => each.ev === "tx" && each.mono > before && (each.seg?.len.value ?? 0) > 0) : from === "syn-sent" ? socket.find((each) => each.ev === "state" && newState(each) === 2) : from === "peer-closed" ? kernel.find((each) => each.netns === peerNetns && each.cookie === peerCookie && each.ev === "state" && oldState(each) === 1) : void 0;
 		if (record === void 0 && from === "peer-closed" && peerCookie === void 0) return proxiedClose(run);
 		return record?.mono;
 	};