@mark1russell7/harness 0.11.22 → 0.11.24

package/README.md

@@ -125,6 +125,9 @@ A step-down (`step-down`) moves the leadership to the first follower (`rabbitmq-
 - **An HTTP client** (`Cell.client`: `go-http`, `http-agent` or `undici`) is a client of the harness from its own image. It sends a GET of the root, then a second request (`probe.http`):
   - after the signal (`inflight`), into the fault
   - or by itself, a gap after its first reply (`gap`), for the keep-alive race.
+- **A client of a keyed limiter.** The flag `limiter-key` (`probe.http.options`) makes the client of undici or of Go send a key of its process in the header `x-limiter-key`. Thus each run has its own count in the limiter of the target.
+  - The flag `retry` sends each request of undici through its `RetryAgent`, at the defaults of undici. Each `retry-` flag sets one option over them.
+  - The client of undici then writes an `attempt` line for each send of a request, with its number `n` and its socket. A `RetryAgent` that gives up gives `UND_ERR_REQ_RETRY` on the `error` line, with the status of the last reply and the count of sends.
 - **The lines of an HTTP client.** It writes the lines of the TCP client, with the number `i` of the request on each line of a request. It also writes the connection of each try of a request (`request`, then `retry`), with its local address and a flag for a reused connection. Each connection gets a `conn` line, and each end of a connection a line (`eof`, `conn-error`, `closed`). The end of the second request is a `done` line.
 - **A target with a script.** A target can take files before it starts (`Target.files`), for example the script of a Node server on the stock Node image. A target that writes JSON lines (`Target.lines`) gives them to each run (`Run.server`). The manifest holds the sha256 of each file and the first line of the target.
 - **The WebSocket client** (`Cell.client` is `procedures-ws`) is the `WebSocketTransport` of `@procedures/client` 0.7.0: the `client` program of the `procedures` subject. The cell gives each of its settings (`probe.ws`): the class of its sockets, the heartbeat, the reconnect, and the times of a call.
@@ -247,7 +250,8 @@ ioredis sets the keepalive of its socket itself, after it connects. Thus a cell
 - **A close of the server** (`server-close`) makes the server close the connection of each client, with `Target.close`: `CLIENT KILL` for Redis.
 - **A stall of the application** (`app-stall`) injects nothing. The server process holds the reply, because the request asks for it (`HttpProbe.hold`), and its kernel stays up.
 - **An event at a proxy** (`hop-event`, `Cell.event`) is a signal to the program of the proxy, or a query on its console. Examples are a reload of the configuration, a stop of a proxy and PAUSE on the console of PgBouncer.
-  - A signal goes to each process of `ProxyHop.program`: HUP, TERM or INT (`HopSignal`). A sidecar in the network and process namespaces of the proxy sends it (`inject signal NAME HUP`). The lab starts that sidecar at the first signal.
+  - A signal goes to each process of `ProxyHop.program`: HUP, TERM, INT, STOP or CONT (`HopSignal`). A sidecar in the network and process namespaces of the proxy sends it (`inject signal NAME HUP`). The lab starts that sidecar at the first signal.
+  - STOP blocks the event loop of the proxy: its processes stop, and its kernel keeps each socket. With a hold (`Cell.hold`), CONT thaws it at that time after the stop, from inside the VM (`--at`). The lift thaws a proxy that its hold did not thaw.
   - A query goes from the injector of the proxy to its console on 127.0.0.1, as a user that the console trusts (`inject query`). The injector speaks the simple query of the PostgreSQL protocol. Its stamps bracket the query and its ReadyForQuery. After the run, the harness sends the `undo` of the query, for example RESUME after PAUSE.
   - Without `inFlight`, the event comes first, and the request meets it. With `inFlight`, the request goes first, and the event comes while it is in flight. A run whose order is not that order is void.
 - **A fault on a group** acts on the part that `Cell.faultAt` names by its role:
@@ -331,7 +335,10 @@ Each run starts from kernels that keep nothing from the run before it (`lab.forg
   - The band counts from its reference on that socket. It ends at a line of the client (`ends`), or at the report of the kernel on the socket.
   - A band that a line ends is a timer of the driver. It counts from the last segment with data that the socket received or sent. The idle timer of a Node socket starts again at each read and each write of data.
   - A band can name that segment as its reference (`last-data`). For example, an idle gRPC client goes to IDLE at a GOAWAY of its proxy. The idle timer of the proxy counts from the last data of the stream.
-  - A reference that this judge does not know (for example `ping-sent`) does not come. Thus its band is a finding. Before 0.11.22, such a reference counted from the first close of a socket of the peer. Through a proxy, that close can be the idle close of a socket toward the target.
+  - A reference that this judge does not know (for example `ping-sent`) does not come. Thus its band is a finding. Before 0.11.22, such a reference counted from the first close of a socket of the peer. Through a proxy, that close can be the idle close of a socket toward the target. Since 0.11.23, the TCP judge does the same.
+  - A retry after a refusal counts from that refusal (`last-reply`), for example a 429 with `Retry-After`. That is the last segment with data that a socket of the client received before the line that ends the band. Each band gets its own, thus each retry counts from the refusal before it.
+  - The retry can go on a new connection. For example, undici closes an idle socket 3 s after a reply of a server at its defaults, before a retry 30 s later. Thus `last-reply` reads each socket of the client.
+  - The bands from `last-reply` follow one another. Thus the deadline of the client and the wait of the run hold their sum: five retries of 30 s take 150 s (`longestOf`).
   - The MongoDB client is ready when its first write has its reply and its monitor streams. The monitor sends its first awaited hello `minHeartbeatFrequencyMS` (500 ms) after its first check. A fault before that hello meets the start of the stream, not the stream that a cell measures.
   - Each later band (`then`) has its own reference and line. A later band outside its band, or absent, is a finding. So is a line that the prediction names as quiet, after the fault.
   - For a silent prediction, a flow of an open socket of the driver that a router forgot is a finding.

package/build/{cell-BxBZUssi.d.ts → cell-wInJC-GH.d.ts}

@@ -130,8 +130,12 @@ interface ProxyHop {
 /**
  * A signal of a hop event. HUP reloads the configuration of a pooler. TERM stops a proxy, as a stop of
  * its container does. INT stops docker-proxy, as Docker does when it removes a port binding.
+ *
+ * STOP blocks the event loop of a proxy: its processes stop, and its kernel keeps each socket. Thus
+ * ACKs go on, and the data of each peer waits in its receive queue. With the hold of the cell
+ * (`Cell.hold`), CONT thaws it at the end of the hold, from inside the VM.
  */
-type HopSignal = "HUP" | "TERM" | "INT";
+type HopSignal = "HUP" | "TERM" | "INT" | "STOP" | "CONT";
 /**
  * An event at a proxy hop (`hop-event`). A signal goes to each process of the program of the hop
  * (`ProxyHop.program`). A query goes to the console of the hop from its own namespace, as a user
@@ -312,8 +316,13 @@ interface Toxic {
  *
  * A connect timer counts from the change of its socket to SYN_SENT (`syn-sent`). A pooler connects
  * to the server by itself, thus the kernel stamps that connect, and no line of the client does.
+ *
+ * A retry of a client after a refusal counts from the refusal (`last-reply`), for example a 429 with
+ * `Retry-After`. That is the last segment with data that a socket of the client received before
+ * the line that ends the band. The retry can go on a new connection. Each band of a driver has
+ * its own, thus each retry counts from the refusal before it.
  */
-type Reference = "last-received" | "last-data" | "first-retransmission" | "request-sent" | "peer-closed" | "fault" | "client-start" | "ping-sent" | "first-sent" | "syn-sent";
+type Reference = "last-received" | "last-reply" | "last-data" | "first-retransmission" | "request-sent" | "peer-closed" | "fault" | "client-start" | "ping-sent" | "first-sent" | "syn-sent";
 /**
  * This prediction tells when a connection moves again after a stall. That occurs at the first
  * firing of its retransmission timer after the stall stops, and its reply comes back one round
@@ -1123,4 +1132,4 @@ interface RunTopology {
 }
 //#endregion
 export { Subject as $, Prediction as A, RequestPrediction as B, KernelRecord as C, MiddleboxForm as D, MiddleboxExpectation as E, Recovers as F, RunPart as G, RttRange as H, Reference as I, SentinelStage as J, RunTopology as K, ReplicaSet as L, ProxyHop as M, ProxyTime as N, Observer as O, Reconnects as P, StageBand as Q, ReplicaSetStage as R, ImageSource as S, MiddleboxEvent as T, Run as U, RouterHop as V, RunGroup as W, ShippedImage as X, Shipped as Y, Stage as Z, Hop as _, Cluster as a, latestEndOf as at, HopSignal as b, DriverBand as c, FaultPlace as d, Target as et, Group as f, HarnessImage as g, GroupRole as h, ClientImage as i, isHopPlace as it, ProbeEvent as j, PinnedImage as k, DriverStep as l, GroupPrediction as m, Cell as n, Toxic as nt, ClusterClientAt as o, GroupPlace as p, SentinelEvent as q, Client as r, WsProbe as rt, ClusterStage as s, BuildImage as t, Topology as tt, Fault as u, HopEvent as v, LineMatch as w, HttpProbe as x, HopPlace as y, Replies as z };
-//# sourceMappingURL=cell-BxBZUssi.d.ts.map
+//# sourceMappingURL=cell-wInJC-GH.d.ts.map

package/build/index.d.ts

@@ -1,4 +1,4 @@
-import { $ as Subject, A as Prediction, B as RequestPrediction, C as KernelRecord, D as MiddleboxForm, E as MiddleboxExpectation, F as Recovers, G as RunPart, H as RttRange, I as Reference, J as SentinelStage, K as RunTopology, L as ReplicaSet, M as ProxyHop, N as ProxyTime, O as Observer, P as Reconnects, Q as StageBand, R as ReplicaSetStage, S as ImageSource, T as MiddleboxEvent, U as Run, V as RouterHop, W as RunGroup, X as ShippedImage, Y as Shipped, Z as Stage, _ as Hop, a as Cluster, at as latestEndOf, b as HopSignal, c as DriverBand, d as FaultPlace, et as Target, f as Group, g as HarnessImage, h as GroupRole, i as ClientImage, it as isHopPlace, j as ProbeEvent, k as PinnedImage, l as DriverStep, m as GroupPrediction, n as Cell, nt as Toxic, o as ClusterClientAt, p as GroupPlace, q as SentinelEvent, r as Client, rt as WsProbe, s as ClusterStage, t as BuildImage, tt as Topology, u as Fault, v as HopEvent, w as LineMatch, x as HttpProbe, y as HopPlace, z as Replies } from "./cell-BxBZUssi.js";
+import { $ as Subject, A as Prediction, B as RequestPrediction, C as KernelRecord, D as MiddleboxForm, E as MiddleboxExpectation, F as Recovers, G as RunPart, H as RttRange, I as Reference, J as SentinelStage, K as RunTopology, L as ReplicaSet, M as ProxyHop, N as ProxyTime, O as Observer, P as Reconnects, Q as StageBand, R as ReplicaSetStage, S as ImageSource, T as MiddleboxEvent, U as Run, V as RouterHop, W as RunGroup, X as ShippedImage, Y as Shipped, Z as Stage, _ as Hop, a as Cluster, at as latestEndOf, b as HopSignal, c as DriverBand, d as FaultPlace, et as Target, f as Group, g as HarnessImage, h as GroupRole, i as ClientImage, it as isHopPlace, j as ProbeEvent, k as PinnedImage, l as DriverStep, m as GroupPrediction, n as Cell, nt as Toxic, o as ClusterClientAt, p as GroupPlace, q as SentinelEvent, r as Client, rt as WsProbe, s as ClusterStage, t as BuildImage, tt as Topology, u as Fault, v as HopEvent, w as LineMatch, x as HttpProbe, y as HopPlace, z as Replies } from "./cell-wInJC-GH.js";
 import { Quantity } from "@mark1russell7/quantity";
 import "testcontainers";
 //#region src/application.d.ts
@@ -135,9 +135,11 @@ export declare const socketAt: (events: readonly ProbeEvent[], role: string, at:
  * - `peer-closed`: the change of state of the socket of the target, when its process dies.
  * - `fault`: the start of the commit of the fault. `client-start`: the `connecting` line.
  * - `first-sent`: the first segment with data that the socket sends after the fault.
+ * - `last-reply`: the refusal that a retry follows. It is the last segment with data that a socket
+ *   of the client received before the end of the band (`end`). The retry can go on another socket.
  * A reference that this judge does not know (`ping-sent`) does not come: its band is a finding.
  */
-export declare const referenceOf: (run: Run, from: Reference, role: string, data: boolean) => number | undefined;
+export declare const referenceOf: (run: Run, from: Reference, role: string, data: boolean, end?: number) => number | undefined;
 /**
  * This function reads one run of a driver cell.
  * - The band of the cell counts from its reference on its socket. It ends at its line, or at the
@@ -939,9 +941,10 @@ interface Lab {
    * This method applies an event at a proxy hop (`HopEvent`), or its undo. A signal goes from a
    * sidecar in the network and process namespaces of the hop. The lab starts that sidecar at the
    * first signal of the hop. A query goes from the injector of the hop to its console on 127.0.0.1. The
-   * injector stamps each one. An event with no undo gives null for its undo.
+   * injector stamps each one. An event with no undo gives null for its undo. A signal can wait for an
+   * instant on CLOCK_MONOTONIC (`at`), as the thaw at the end of a hold does.
    */
-  event(event: HopEvent, undo?: boolean): Promise<{
+  event(event: HopEvent, undo?: boolean, at?: number): Promise<{
     t0: number;
     t1: number;
   } | null>;

package/build/index.js

@@ -741,14 +741,17 @@ const outsetOf = (run) => {
 * - `peer-closed`: the change of state of the socket of the target, when its process dies.
 * - `fault`: the start of the commit of the fault. `client-start`: the `connecting` line.
 * - `first-sent`: the first segment with data that the socket sends after the fault.
+* - `last-reply`: the refusal that a retry follows. It is the last segment with data that a socket
+*   of the client received before the end of the band (`end`). The retry can go on another socket.
 * A reference that this judge does not know (`ping-sent`) does not come: its band is a finding.
 */
-const referenceOf = (run, from, role, data) => {
+const referenceOf = (run, from, role, data, end = Infinity) => {
 	const ready = run.events.find((event) => event.ev === "ready");
 	const before = run.fault?.t1 ?? ready?.t ?? Infinity;
 	if (from === "fault") return run.fault?.t0;
 	if (from === "client-start") return (run.events.find((event) => event.ev === "connecting") ?? run.events[0])?.t;
 	if (from === "request-sent") return run.events.filter((event) => event.ev === "sent")[1]?.t;
+	if (from === "last-reply") return run.kernel.filter((record) => record.netns === run.netns && record.mono < end && record.ev === "rx" && (record.seg?.len.value ?? 0) > 0).at(-1)?.mono;
 	const { socket, place } = kernelOf(run, role, before);
 	if (place === void 0) return void 0;
 	if (from === "last-received" || from === "last-data") {
@@ -812,13 +815,19 @@ const reportOf = (place, after) => {
 		outcome: { kind: "eof" }
 	};
 };
-/** This function gives the latest instant of the bands of a driver prediction, in seconds from their references. */
+/**
+* This function gives the latest instant of the bands of a driver prediction, in seconds from their
+* references. A band from the refusal before it (`last-reply`) follows the band before it. Thus the
+* times of those bands add: five retries of 30 s end 150 s after the first refusal.
+*/
 const longestOf = (cell) => {
 	const { prediction } = cell;
 	if (prediction.detects === null) return valueIn(prediction.window, "s");
-	const steps = (prediction.driver?.then ?? []).map((step) => valueIn(step.high, "s"));
+	const then = prediction.driver?.then ?? [];
+	const steps = then.map((step) => valueIn(step.high, "s"));
 	const own = prediction.detects ? valueIn(prediction.high, "s") : valueIn(prediction.window, "s");
-	return Math.max(own, ...steps);
+	const chained = [...prediction.detects && prediction.from === "last-reply" ? [own] : [], ...then.filter((step) => step.from === "last-reply").map((step) => valueIn(step.high, "s"))];
+	return Math.max(own, ...steps, chained.reduce((total, high) => total + high, 0));
 };
 /**
 * This function tells if a run of a driver cell has its outcome, so that its wait can stop. For a
@@ -956,7 +965,7 @@ const measureDriver = (cell, run) => {
 	}
 	const stepTimes = (driver.then ?? []).map((step) => {
 		const line = lineAfter(events, step.ends, after);
-		const from = referenceOf(run, step.from, step.socket ?? driver.socket, true);
+		const from = referenceOf(run, step.from, step.socket ?? driver.socket, true, line?.t);
 		if (line === void 0 || from === void 0) {
 			findings.push(line === void 0 ? `${step.what}: ${wordsOf(step.ends)} never came` : `${step.what}: its reference (${step.from}) never came`);
 			return {
@@ -986,7 +995,7 @@ const measureDriver = (cell, run) => {
 			finding: findings[0] ?? null
 		};
 		const line = lineAfter(events, driver.reply.ends, after);
-		const from = referenceOf(run, driver.reply.from, driver.socket, true);
+		const from = referenceOf(run, driver.reply.from, driver.socket, true, line?.t);
 		if (line === void 0 || from === void 0) return {
 			...base,
 			...nothing,
@@ -1025,7 +1034,7 @@ const measureDriver = (cell, run) => {
 		void: voided,
 		finding: findings[0] ?? `nothing detected it: ${prediction.timer} never fired`
 	};
-	const reference = referenceOf(run, prediction.from, driver.socket, driver.ends !== void 0);
+	const reference = referenceOf(run, prediction.from, driver.socket, driver.ends !== void 0, ended);
 	if (reference === void 0) return {
 		...base,
 		...nothing,
@@ -1488,7 +1497,7 @@ const count = (event, field) => event === void 0 ? 0 : Number(event[field] ?? 0)
 /** This function gives the view of the client on the same interval: TCP_INFO, with a sample each 2 ms. */
 const socketVantage = (cell, events, ready, request) => {
 	const { from } = cell.prediction.detects === true ? cell.prediction : { from: void 0 };
-	if (from === void 0 || from === "peer-closed" || from === "fault" || from === "ping-sent" || from === "first-sent" || from === "last-data") return null;
+	if (from === void 0 || from === "peer-closed" || from === "fault" || from === "ping-sent" || from === "first-sent" || from === "last-data" || from === "last-reply") return null;
 	const ended = events.find((event) => event.ev === "error" || event.ev === "eof");
 	const reference = from === "last-received" ? ready : from === "request-sent" ? request : events.find((event) => event.ev === "kernel" && event.t >= request.t && count(event, "retransmits") > 0);
 	return ended === void 0 || reference === void 0 ? null : between(reference.t, ended.t);
@@ -1829,7 +1838,7 @@ const measureSocket = (cell, run) => {
 		if (from === "fault") return run.fault?.t0;
 		if (from === "client-start") return (events.find((event) => event.ev === "connecting") ?? events[0])?.t;
 		if (from === "ping-sent") return firstUnanswered(pings, socket, ended)?.at;
-		const record = from === "last-received" ? socket.filter((each) => each.ev === "rx" && each.mono <= before).at(-1) : from === "last-data" ? socket.filter((each) => each.ev === "rx" && (each.seg?.len.value ?? 0) > 0 && each.mono <= (ended ?? before)).at(-1) : from === "first-retransmission" ? retransmitted.find((each) => each.why === "rto" && each.mono > before) : from === "first-sent" ? socket.find((each) => each.ev === "tx" && each.mono > before && (each.seg?.len.value ?? 0) > 0) : from === "syn-sent" ? socket.find((each) => each.ev === "state" && newState(each) === 2) : kernel.find((each) => each.netns === peerNetns && each.cookie === peerCookie && each.ev === "state" && oldState(each) === 1);
+		const record = from === "last-received" ? socket.filter((each) => each.ev === "rx" && each.mono <= before).at(-1) : from === "last-data" ? socket.filter((each) => each.ev === "rx" && (each.seg?.len.value ?? 0) > 0 && each.mono <= (ended ?? before)).at(-1) : from === "last-reply" ? socket.filter((each) => each.ev === "rx" && (each.seg?.len.value ?? 0) > 0 && each.mono < (ended ?? Infinity)).at(-1) : from === "first-retransmission" ? retransmitted.find((each) => each.why === "rto" && each.mono > before) : from === "first-sent" ? socket.find((each) => each.ev === "tx" && each.mono > before && (each.seg?.len.value ?? 0) > 0) : from === "syn-sent" ? socket.find((each) => each.ev === "state" && newState(each) === 2) : from === "peer-closed" ? kernel.find((each) => each.netns === peerNetns && each.cookie === peerCookie && each.ev === "state" && oldState(each) === 1) : void 0;
 		if (record === void 0 && from === "peer-closed" && peerCookie === void 0) return proxiedClose(run);
 		return record?.mono;
 	};
@@ -2794,7 +2803,7 @@ const openLab = async (target, kind = "tcp", topology) => {
 					...when
 				]);
 			},
-			async event(event, undo = false) {
+			async event(event, undo = false, at) {
 				const hop = hopNamed(event.hop);
 				if (hop.hop.kind !== "proxy") throw new Error(`the hop ${event.hop} is a router: an event goes to a proxy`);
 				if ("signal" in event) {
@@ -2807,11 +2816,13 @@ const openLab = async (target, kind = "tcp", topology) => {
 						hopSignallers.set(event.hop, sidecar);
 						opened.push(sidecar);
 					}
+					const when = at === void 0 ? [] : ["--at", String(Math.round(at))];
 					return stamped(sidecar, [
 						"/opt/harness/inject",
 						"signal",
 						program,
-						event.signal
+						event.signal,
+						...when
 					]);
 				}
 				const query = undo ? event.console.undo : event.console.query;
@@ -5315,9 +5326,23 @@ const once = async (lab, cell, environment, index, dry = false) => {
 					case "server-close":
 						await lab.serverClose();
 						return null;
-					case "hop-event":
+					case "hop-event": {
 						if (cell.event === void 0) throw new Error(`${cell.name} names no event`);
-						return lab.event(cell.event);
+						const stamp = await lab.event(cell.event);
+						if ("signal" in cell.event && cell.event.signal === "STOP" && cell.hold !== void 0 && stamp !== null) {
+							const thaw = {
+								hop: cell.event.hop,
+								signal: "CONT"
+							};
+							thawing = lab.event(thaw, false, stamp.t0 + valueIn(cell.hold, "s") * 1e9).then((thawed) => {
+								if (thawed === null) throw new Error(`the hop ${thaw.hop} gave no stamp for CONT`);
+								heal = thawed;
+								return thawed;
+							});
+							thawing.catch(() => void 0);
+						}
+						return stamp;
+					}
 					default: return lab.fault("blackhole", faultAt);
 				}
 			};
@@ -5337,6 +5362,15 @@ const once = async (lab, cell, environment, index, dry = false) => {
 					case "app-stall":
 					case "none": return;
 					case "hop-event":
+						if (cell.event !== void 0 && "signal" in cell.event && cell.event.signal === "STOP") {
+							const thaw = {
+								hop: cell.event.hop,
+								signal: "CONT"
+							};
+							if (thawing !== null) await thawing.catch(() => lab.event(thaw));
+							else await lab.event(thaw);
+							return;
+						}
 						if (cell.event !== void 0) await lab.event(cell.event, true);
 						return;
 					case "unreachable": return lab.lift("reject", faultAt, { code: cell.icmp ?? "host" });