@mark1russell7/harness 0.11.11 → 0.11.13

package/README.md

@@ -45,7 +45,8 @@ A topology (`Topology`) is the chain of hops from the client to the target, as d
   | `recreate` | nothing | passes, and the flow starts again | dropped |
   | `none` | nothing | passes, and the flow starts again | a reset from the stack of the router |
 
-- **A proxy** (`ProxyHop`) is a stock image, pinned by digest, that ends each TCP connection and opens its own to the next hop. Its files go into it before it starts, and `{{next}}` in them is the address and port of the next hop. An injector sidecar shares its namespace. A privileged sidecar sets its sysctls when the hop names them.
+- **A proxy** (`ProxyHop`) is a stock image, pinned by digest, that ends each TCP connection and opens its own to the next hop. Its files go into it before it starts, and `{{next}}` in them is the address and port of the next hop. Its `command` and its `environment` replace those of its image. An injector sidecar shares its namespace. A privileged sidecar sets its sysctls when the hop names them.
+  - The lab keeps the last 60 lines of the output of each proxy. A proxy that does not start gives them in its error.
 
 The element before a router reaches the networks after the router through it. Its privileged sidecar, or the router itself, adds each route. A router translates the address of the side of the client, thus the target and each later hop see the router as the peer.
 
@@ -164,6 +165,7 @@ The harness ships ten images, each as a shipped source (`ImageSource`). The tag
   - `inflight` is the text of the query in flight: the second query of an in-flight run. The first query then makes the pool ready at once, and only the query in flight is slow, for example `SELECT pg_sleep(20)`. The HikariCP client reads it too.
 - **The HikariCP client image** (`image/hikari`) is HikariCP 7.1.0 over pgjdbc 42.7.13 on Temurin 21.0.12.1. The digests of the two stages and the SHA-256 of each jar pin them. The probe is `{ driver: { name: "hikari", query, inflight } }`.
   - `ClientHikari.java` makes one `HikariDataSource` with the properties of the cell (`Cell.driverOptions`). A name with the prefix `dataSource.` is a property of pgjdbc.
+  - The user is `postgres`, unless the cell gives `username` (and `password`). Supavisor finds the tenant of a client in its user, for example `postgres.harness`.
   - It prints the lines of the pg client on the same clock. Each socket comes from the `PGStream` of its connection, and a `pool` line gives each change of the counts of the pool. The errno of a failure is the SQLState of pgjdbc.
   - Before its first line, it loads the classes of a connect and of a pool on the loopback of its namespace. It also starts the SecureRandom that HikariCP reads for the UUID of each new connection. Then it parses an ERROR of the server, as for the timeout of a statement. A new JVM loads each class at its first use. Thus a band does not time that load.
 - **The MongoDB client image** (`image/mongo`) is the driver mongodb 7.7.0 on Node 24.21.0. The digest of the image and an npm lockfile pin them. `client-mongo.mjs` prints the lines of the TCP client on the same clock:
@@ -241,6 +243,9 @@ Each run starts from kernels that keep nothing from the run before it (`lab.forg
 - **Recovery.** After a stall (`prediction.recovers`), it times the send to the first segment back after the stall stopped. That segment is the answer of the server to the retransmission that got through. A run whose stall stopped outside the window of its prediction is void. A run where nothing came back is a finding.
 - **An end of file.** A FIN counts from the change of state of the socket of the peer when it closes (`peer-closed`). It ends at the change of the socket of the observer to CLOSE_WAIT. The kernel stamps the two changes.
 - **A check of a pool.** A check on an idle connection counts from the first segment with data that the socket sends after the fault (`first-sent`). The keepalive of HikariCP comes at a phase from the creation of its connection. Thus no line of the client gives its time, and the kernel stamps it. The prediction gives the latest time of that send after the reply (`startsBy`), and the run waits that much longer.
+- **The connect of a pooler.** A band of a driver can be on the socket that a hop opens toward the target (`socket: "upstream"`). It is the first socket of a hop that moves to SYN_SENT after the outset. A pooler connects to the server by itself, thus no line of the client names that socket.
+  - A connect timer counts from the change of that socket to SYN_SENT (`syn-sent`).
+  - The band ends at the error that the kernel reports, for example ETIMEDOUT after the SYNs. Without an error, it ends when the program closes its socket in SYN_SENT (`closed`). That close has no error and no FIN, only the change to CLOSE. A refused connect also leaves SYN_SENT, but it has its error.
 - **A detection by the program** (`application.ts`). The kernel has no report of it. Thus the program of the observer writes a line before each of its own acts, and the kernel stamps each act on the socket.
   - The program writes `ping` before each ping of its heartbeat. The kernel stamps the write of the ping (`tx`).
   - The reference is the first ping that got no answer (`ping-sent`): no segment with data reached the socket after its write, up to the detection. Its write starts the timer of the heartbeat that the fault met. The fault can come at any time, and its stamps are not necessary.

package/build/{cell-6DQfSaJP.d.ts → cell-COlAmUZ-.d.ts}

@@ -97,6 +97,8 @@ interface ProxyHop {
   /** This port is the port that the proxy listens on for the side of the client. */
   readonly port: number;
   readonly command?: readonly string[];
+  /** These variables are the environment of the proxy, over the environment of its image. */
+  readonly environment?: Readonly<Record<string, string>>;
   /**
    * These files go into the proxy before it starts, by their path. In their content, {{next}} is
    * the address and port of the next hop, and {{next.host}} and {{next.port}} are each part.
@@ -231,8 +233,11 @@ interface Toxic {
  * A check of a pool on an idle connection counts from the first segment with data that the socket
  * sends after the fault (`first-sent`). The keepalive of HikariCP comes at a phase from the
  * creation of its connection, thus no line of the client gives its time. The kernel stamps it.
+ *
+ * A connect timer counts from the change of its socket to SYN_SENT (`syn-sent`). A pooler connects
+ * to the server by itself, thus the kernel stamps that connect, and no line of the client does.
  */
-type Reference = "last-received" | "first-retransmission" | "request-sent" | "peer-closed" | "fault" | "client-start" | "ping-sent" | "first-sent";
+type Reference = "last-received" | "first-retransmission" | "request-sent" | "peer-closed" | "fault" | "client-start" | "ping-sent" | "first-sent" | "syn-sent";
 /**
  * This prediction tells when a connection moves again after a stall. That occurs at the first
  * firing of its retransmission timer after the stall stops, and its reply comes back one round
@@ -1018,4 +1023,4 @@ interface RunTopology {
 }
 //#endregion
 export { isHopPlace as $, Recovers as A, RunPart as B, Observer as C, ProxyHop as D, ProbeEvent as E, RequestPrediction as F, ShippedImage as G, SentinelEvent as H, RouterHop as I, Subject as J, Stage as K, RttRange as L, ReplicaSet as M, ReplicaSetStage as N, ProxyTime as O, Replies as P, WsProbe as Q, Run as R, MiddleboxForm as S, Prediction as T, SentinelStage as U, RunTopology as V, Shipped as W, Topology as X, Target as Y, Toxic as Z, ImageSource as _, DriverBand as a, MiddleboxEvent as b, FaultPlace as c, GroupPrediction as d, latestEndOf as et, GroupRole as f, HttpProbe as g, HopPlace as h, ClientImage as i, Reference as j, Reconnects as k, Group as l, Hop as m, Cell as n, DriverStep as o, HarnessImage as p, StageBand as q, Client as r, Fault as s, BuildImage as t, GroupPlace as u, KernelRecord as v, PinnedImage as w, MiddleboxExpectation as x, LineMatch as y, RunGroup as z };
-//# sourceMappingURL=cell-6DQfSaJP.d.ts.map
+//# sourceMappingURL=cell-COlAmUZ-.d.ts.map

package/build/index.d.ts

@@ -1,4 +1,4 @@
-import { $ as isHopPlace, A as Recovers, B as RunPart, C as Observer, D as ProxyHop, E as ProbeEvent, F as RequestPrediction, G as ShippedImage, H as SentinelEvent, I as RouterHop, J as Subject, K as Stage, L as RttRange, M as ReplicaSet, N as ReplicaSetStage, O as ProxyTime, P as Replies, Q as WsProbe, R as Run, S as MiddleboxForm, T as Prediction, U as SentinelStage, V as RunTopology, W as Shipped, X as Topology, Y as Target, Z as Toxic, _ as ImageSource, a as DriverBand, b as MiddleboxEvent, c as FaultPlace, d as GroupPrediction, et as latestEndOf, f as GroupRole, g as HttpProbe, h as HopPlace, i as ClientImage, j as Reference, k as Reconnects, l as Group, m as Hop, n as Cell, o as DriverStep, p as HarnessImage, q as StageBand, r as Client, s as Fault, t as BuildImage, u as GroupPlace, v as KernelRecord, w as PinnedImage, x as MiddleboxExpectation, y as LineMatch, z as RunGroup } from "./cell-6DQfSaJP.js";
+import { $ as isHopPlace, A as Recovers, B as RunPart, C as Observer, D as ProxyHop, E as ProbeEvent, F as RequestPrediction, G as ShippedImage, H as SentinelEvent, I as RouterHop, J as Subject, K as Stage, L as RttRange, M as ReplicaSet, N as ReplicaSetStage, O as ProxyTime, P as Replies, Q as WsProbe, R as Run, S as MiddleboxForm, T as Prediction, U as SentinelStage, V as RunTopology, W as Shipped, X as Topology, Y as Target, Z as Toxic, _ as ImageSource, a as DriverBand, b as MiddleboxEvent, c as FaultPlace, d as GroupPrediction, et as latestEndOf, f as GroupRole, g as HttpProbe, h as HopPlace, i as ClientImage, j as Reference, k as Reconnects, l as Group, m as Hop, n as Cell, o as DriverStep, p as HarnessImage, q as StageBand, r as Client, s as Fault, t as BuildImage, u as GroupPlace, v as KernelRecord, w as PinnedImage, x as MiddleboxExpectation, y as LineMatch, z as RunGroup } from "./cell-COlAmUZ-.js";
 import { Quantity } from "@mark1russell7/quantity";
 import "testcontainers";
 //#region src/application.d.ts

package/build/index.js

@@ -682,8 +682,28 @@ const peerPlace = (run, local) => {
 	const peer = run.topology === void 0 ? local : `${run.topology.targetPeer}:${portOf(local)}`;
 	return placeOf(run, run.targetNetns, (record) => record.dst === peer);
 };
-/** This function gives the kernel place of a socket of a role at an instant. The role `server` is a socket of the target. */
+/**
+* This function gives the socket that a hop of the topology opens toward its target in the run. It is
+* the first one that moves to SYN_SENT after the outset. A pooler connects to the server by itself,
+* thus no line of the client names that socket. The last hop goes first, because it faces the target.
+*/
+const upstreamOf = (run) => {
+	if (run.topology === void 0) return void 0;
+	const outset = outsetOf(run);
+	for (const hop of Object.values(run.topology.hops).reverse()) {
+		const place = placeOf(run, hop.netns, (record) => record.ev === "state" && newState$1(record) === 2 && record.mono >= outset && record.dst === hop.target);
+		if (place !== void 0) return place;
+	}
+};
+/**
+* This function gives the kernel place of a socket of a role at an instant. The role `server` is a
+* socket of the target, and the role `upstream` is the socket that a hop opens toward the target.
+*/
 const kernelOf = (run, role, at) => {
+	if (role === "upstream") return {
+		socket: void 0,
+		place: upstreamOf(run)
+	};
 	if (role === "server") {
 		const pool = socketAt(run.events, "pool", at);
 		return {
@@ -728,6 +748,7 @@ const referenceOf = (run, from, role, data) => {
 		return place.records.filter((record) => record.mono <= before && counts(record)).at(-1)?.mono;
 	}
 	if (from === "first-sent") return place.records.find((record) => record.ev === "tx" && record.mono > before && (record.seg?.len.value ?? 0) > 0)?.mono;
+	if (from === "syn-sent") return place.records.find((record) => record.ev === "state" && newState$1(record) === 2)?.mono;
 	if (from === "first-retransmission") {
 		const request = run.events.filter((event) => event.ev === "sent")[1]?.t ?? before;
 		return place.records.find((record) => record.ev === "retrans" && record.why === "rto" && record.mono > request)?.mono;
@@ -755,10 +776,21 @@ const ipOf$1 = (address) => address.slice(0, address.lastIndexOf(":"));
 const lineAfter = (events, match, after) => events.find((event) => event.t > after && matches(event, match));
 /** This function gives the name of the error of a line: the field that the match names, or `name`. */
 const nameOf = (line, match) => String(fieldOf(line, match.name ?? "name") ?? line.ev);
-/** This function gives the report of the kernel on a socket after an instant: an error, or the end of the file (CLOSE_WAIT). */
+/**
+* This function gives the report of the kernel on a socket after an instant: an error, or the end of
+* the file (CLOSE_WAIT). With neither, a close of the socket in SYN_SENT is the end. The program gave
+* up on its connect, and the kernel reports no error and sends no FIN. A refused connect also leaves
+* SYN_SENT, but it has its error.
+*/
 const reportOf = (place, after) => {
 	const report = place.records.find((record) => record.mono > after && (record.ev === "error" && (record.err ?? 0) !== 0 || record.ev === "state" && newState$1(record) === 8));
-	if (report === void 0) return void 0;
+	if (report === void 0) {
+		const given = place.records.find((record) => record.mono > after && record.ev === "state" && oldState$1(record) === 2 && newState$1(record) === 7);
+		return given === void 0 ? void 0 : {
+			at: given.mono,
+			outcome: { kind: "closed" }
+		};
+	}
 	return report.ev === "error" ? {
 		at: report.mono,
 		outcome: {
@@ -972,7 +1004,7 @@ const measureDriver = (cell, run) => {
 		finding: `it ended without its reference event (${prediction.from} on the ${driver.socket} socket)`
 	};
 	if (driver.ends === void 0 && prediction.from === "last-received" && place !== void 0) voided ??= delayed(place, reference, ended, run.hz, noiseOf(cell));
-	const name = outcome.kind === "error" ? outcome.errno : outcome.kind === "eof" ? "EOF" : "silent";
+	const name = outcome.kind === "error" ? outcome.errno : outcome.kind === "eof" ? "EOF" : outcome.kind === "closed" ? "CLOSED" : "silent";
 	if (prediction.errno !== "OK" && name !== prediction.errno) findings.unshift(`it ended with ${name}, not ${prediction.errno}`);
 	const seen = report === void 0 ? void 0 : events.find((event) => event.t >= report.at && (event.ev === "error" || event.ev === "socket-error"));
 	return {
@@ -1694,7 +1726,7 @@ const measureSocket = (cell, run) => {
 		if (from === "fault") return run.fault?.t0;
 		if (from === "client-start") return (events.find((event) => event.ev === "connecting") ?? events[0])?.t;
 		if (from === "ping-sent") return firstUnanswered(pings, socket, ended)?.at;
-		const record = from === "last-received" ? socket.filter((each) => each.ev === "rx" && each.mono <= before).at(-1) : from === "first-retransmission" ? retransmitted.find((each) => each.why === "rto" && each.mono > before) : from === "first-sent" ? socket.find((each) => each.ev === "tx" && each.mono > before && (each.seg?.len.value ?? 0) > 0) : kernel.find((each) => each.netns === peerNetns && each.cookie === peerCookie && each.ev === "state" && oldState(each) === 1);
+		const record = from === "last-received" ? socket.filter((each) => each.ev === "rx" && each.mono <= before).at(-1) : from === "first-retransmission" ? retransmitted.find((each) => each.why === "rto" && each.mono > before) : from === "first-sent" ? socket.find((each) => each.ev === "tx" && each.mono > before && (each.seg?.len.value ?? 0) > 0) : from === "syn-sent" ? socket.find((each) => each.ev === "state" && newState(each) === 2) : kernel.find((each) => each.netns === peerNetns && each.cookie === peerCookie && each.ev === "state" && oldState(each) === 1);
 		if (record === void 0 && from === "peer-closed" && peerCookie === void 0) return proxiedClose(run);
 		return record?.mono;
 	};
@@ -1946,6 +1978,8 @@ const PUMP_SLACK = {
 	value: 5,
 	unit: "ms"
 };
+/** This count is how many of the last lines of a proxy hop the lab keeps. The error of a hop that does not start gives them. */
+const TAIL_LINES = 60;
 /** This function gives the arguments of the injector for the options of a fault. */
 const optionArgs = (options) => [
 	...options.direction === void 0 ? [] : ["--direction", options.direction],
@@ -2231,12 +2265,27 @@ const openLab = async (target, kind = "tcp", topology) => {
 			const next = endpointOf(at + 1);
 			let proxy = new GenericContainer(await imageOf(hop.image)).withNetwork(targetSide);
 			if (hop.command !== void 0) proxy = proxy.withCommand([...hop.command]);
+			if (hop.environment !== void 0) proxy = proxy.withEnvironment({ ...hop.environment });
 			if (hop.files !== void 0) proxy = proxy.withCopyContentToContainer(Object.entries(hop.files).map(([path, content]) => ({
 				content: templated(content, next),
 				target: path
 			})));
 			if (hop.ready !== void 0) proxy = proxy.withWaitStrategy(Wait.forLogMessage(hop.ready));
-			const box = await proxy.start();
+			const tail = [];
+			proxy = proxy.withLogConsumer((stream) => {
+				let rest = "";
+				stream.on("data", (chunk) => {
+					const lines = (rest + chunk.toString()).split("\n");
+					rest = lines.pop() ?? "";
+					for (const line of lines) {
+						tail.push(line);
+						if (tail.length > TAIL_LINES) tail.shift();
+					}
+				});
+			});
+			const box = await proxy.start().catch((error) => {
+				throw new Error(`the hop ${hop.name} did not start: ${error instanceof Error ? error.message : String(error)}. Its last lines:\n${tail.join("\n")}`);
+			});
 			opened.push(box);
 			await join(box, clientSide);
 			const injector = await new GenericContainer(image).withNetworkMode(`container:${box.getId()}`).withAddedCapabilities("NET_ADMIN", "NET_RAW").start();
@@ -4220,7 +4269,8 @@ const beforeOnce = async (lab, cell, environment, index) => {
 			kernel: lab.kernel.slice(from),
 			netns: lab.netns.client,
 			targetNetns: lab.netns.target,
-			lost: 0
+			lost: 0,
+			...lab.topology === null ? {} : { topology: lab.topology }
 		});
 		await driverOutcome(cell, current, driverWithin(cell));
 		await sleep(TAIL_MS);