@mark1russell7/harness 0.11.12 → 0.11.14

package/README.md

@@ -45,7 +45,8 @@ A topology (`Topology`) is the chain of hops from the client to the target, as d
   | `recreate` | nothing | passes, and the flow starts again | dropped |
   | `none` | nothing | passes, and the flow starts again | a reset from the stack of the router |
 
-- **A proxy** (`ProxyHop`) is a stock image, pinned by digest, that ends each TCP connection and opens its own to the next hop. Its files go into it before it starts, and `{{next}}` in them is the address and port of the next hop. An injector sidecar shares its namespace. A privileged sidecar sets its sysctls when the hop names them.
+- **A proxy** (`ProxyHop`) is a stock image, pinned by digest, that ends each TCP connection and opens its own to the next hop. Its files go into it before it starts, and `{{next}}` in them is the address and port of the next hop. Its `command` and its `environment` replace those of its image. An injector sidecar shares its namespace. A privileged sidecar sets its sysctls when the hop names them.
+  - The lab keeps the last 60 lines of the output of each proxy. A proxy that does not start gives them in its error.
 
 The element before a router reaches the networks after the router through it. Its privileged sidecar, or the router itself, adds each route. A router translates the address of the side of the client, thus the target and each later hop see the router as the peer.
 
@@ -164,6 +165,7 @@ The harness ships ten images, each as a shipped source (`ImageSource`). The tag
   - `inflight` is the text of the query in flight: the second query of an in-flight run. The first query then makes the pool ready at once, and only the query in flight is slow, for example `SELECT pg_sleep(20)`. The HikariCP client reads it too.
 - **The HikariCP client image** (`image/hikari`) is HikariCP 7.1.0 over pgjdbc 42.7.13 on Temurin 21.0.12.1. The digests of the two stages and the SHA-256 of each jar pin them. The probe is `{ driver: { name: "hikari", query, inflight } }`.
   - `ClientHikari.java` makes one `HikariDataSource` with the properties of the cell (`Cell.driverOptions`). A name with the prefix `dataSource.` is a property of pgjdbc.
+  - The user is `postgres`, unless the cell gives `username` (and `password`). Supavisor finds the tenant of a client in its user, for example `postgres.harness`.
   - It prints the lines of the pg client on the same clock. Each socket comes from the `PGStream` of its connection, and a `pool` line gives each change of the counts of the pool. The errno of a failure is the SQLState of pgjdbc.
   - Before its first line, it loads the classes of a connect and of a pool on the loopback of its namespace. It also starts the SecureRandom that HikariCP reads for the UUID of each new connection. Then it parses an ERROR of the server, as for the timeout of a statement. A new JVM loads each class at its first use. Thus a band does not time that load.
 - **The MongoDB client image** (`image/mongo`) is the driver mongodb 7.7.0 on Node 24.21.0. The digest of the image and an npm lockfile pin them. `client-mongo.mjs` prints the lines of the TCP client on the same clock:
@@ -333,6 +335,7 @@ The distribution test needs a simulator, which does not exist at this time.
   - A replica set cell opens a replica set environment (`openReplicaSetLab`), and each run is `replicaSetOnce`. A run with a hold continues to the end of the hold, thus its record shows what the set did in all of it.
   - A driver cell (`prediction.driver`) waits for the end of its band and of each later band. A silent driver cell waits for its window.
 - **The observer.** A cell observes the client, the server, or a socket of a proxy toward one side (`observer`). A cell on the server side measures the socket of the target for the connection of the client. The run waits for the error of that socket. `freeze-process` freezes the peer of the observer: the target for the client, the client for the server.
+  - A socket of a proxy toward the target is the socket that carries the request of the run. It is the first socket of the hop that sends data to the target after the request. A pooler can reuse a server connection from the warm-up of the environment. Then the first socket that it opens in the run is not the socket of the request. A run without such a socket uses the first socket that the run opens.
   - A cell on the server side sets the sysctls of the target through `targetSysctls`, with a privileged sidecar in the namespace of the target. That sidecar starts only when a cell asks for it.
   - Its in-flight data is the reply of the server. Thus the client sends `probe.request` first, for example a BLPOP with a timeout, and the fault occurs while the server holds it.
 - **The pause before the fault.** In every other run, the fault comes 250 ms after the client's first reply (`QUIET_MS`). That is more than Linux's longest delayed ACK, 200 ms. A fault that comes earlier meets the client's delayed ACK of the reply, not the request or the idle time that the cell measures. On a fast host the fault came 35 ms after the reply, and the ACK met a reset first.

package/build/{cell-yNKJmsCj.d.ts → cell-COlAmUZ-.d.ts}

@@ -97,6 +97,8 @@ interface ProxyHop {
   /** This port is the port that the proxy listens on for the side of the client. */
   readonly port: number;
   readonly command?: readonly string[];
+  /** These variables are the environment of the proxy, over the environment of its image. */
+  readonly environment?: Readonly<Record<string, string>>;
   /**
    * These files go into the proxy before it starts, by their path. In their content, {{next}} is
    * the address and port of the next hop, and {{next.host}} and {{next.port}} are each part.
@@ -1021,4 +1023,4 @@ interface RunTopology {
 }
 //#endregion
 export { isHopPlace as $, Recovers as A, RunPart as B, Observer as C, ProxyHop as D, ProbeEvent as E, RequestPrediction as F, ShippedImage as G, SentinelEvent as H, RouterHop as I, Subject as J, Stage as K, RttRange as L, ReplicaSet as M, ReplicaSetStage as N, ProxyTime as O, Replies as P, WsProbe as Q, Run as R, MiddleboxForm as S, Prediction as T, SentinelStage as U, RunTopology as V, Shipped as W, Topology as X, Target as Y, Toxic as Z, ImageSource as _, DriverBand as a, MiddleboxEvent as b, FaultPlace as c, GroupPrediction as d, latestEndOf as et, GroupRole as f, HttpProbe as g, HopPlace as h, ClientImage as i, Reference as j, Reconnects as k, Group as l, Hop as m, Cell as n, DriverStep as o, HarnessImage as p, StageBand as q, Client as r, Fault as s, BuildImage as t, GroupPlace as u, KernelRecord as v, PinnedImage as w, MiddleboxExpectation as x, LineMatch as y, RunGroup as z };
-//# sourceMappingURL=cell-yNKJmsCj.d.ts.map
+//# sourceMappingURL=cell-COlAmUZ-.d.ts.map

package/build/index.d.ts

@@ -1,4 +1,4 @@
-import { $ as isHopPlace, A as Recovers, B as RunPart, C as Observer, D as ProxyHop, E as ProbeEvent, F as RequestPrediction, G as ShippedImage, H as SentinelEvent, I as RouterHop, J as Subject, K as Stage, L as RttRange, M as ReplicaSet, N as ReplicaSetStage, O as ProxyTime, P as Replies, Q as WsProbe, R as Run, S as MiddleboxForm, T as Prediction, U as SentinelStage, V as RunTopology, W as Shipped, X as Topology, Y as Target, Z as Toxic, _ as ImageSource, a as DriverBand, b as MiddleboxEvent, c as FaultPlace, d as GroupPrediction, et as latestEndOf, f as GroupRole, g as HttpProbe, h as HopPlace, i as ClientImage, j as Reference, k as Reconnects, l as Group, m as Hop, n as Cell, o as DriverStep, p as HarnessImage, q as StageBand, r as Client, s as Fault, t as BuildImage, u as GroupPlace, v as KernelRecord, w as PinnedImage, x as MiddleboxExpectation, y as LineMatch, z as RunGroup } from "./cell-yNKJmsCj.js";
+import { $ as isHopPlace, A as Recovers, B as RunPart, C as Observer, D as ProxyHop, E as ProbeEvent, F as RequestPrediction, G as ShippedImage, H as SentinelEvent, I as RouterHop, J as Subject, K as Stage, L as RttRange, M as ReplicaSet, N as ReplicaSetStage, O as ProxyTime, P as Replies, Q as WsProbe, R as Run, S as MiddleboxForm, T as Prediction, U as SentinelStage, V as RunTopology, W as Shipped, X as Topology, Y as Target, Z as Toxic, _ as ImageSource, a as DriverBand, b as MiddleboxEvent, c as FaultPlace, d as GroupPrediction, et as latestEndOf, f as GroupRole, g as HttpProbe, h as HopPlace, i as ClientImage, j as Reference, k as Reconnects, l as Group, m as Hop, n as Cell, o as DriverStep, p as HarnessImage, q as StageBand, r as Client, s as Fault, t as BuildImage, u as GroupPlace, v as KernelRecord, w as PinnedImage, x as MiddleboxExpectation, y as LineMatch, z as RunGroup } from "./cell-COlAmUZ-.js";
 import { Quantity } from "@mark1russell7/quantity";
 import "testcontainers";
 //#region src/application.d.ts
@@ -417,7 +417,8 @@ interface SocketPlace {
  * This function finds the socket of the observer of a cell in a run:
  * - the client socket, by the address that it connected from, in its namespace.
  * - the socket of the target for the same connection, in its namespace.
- * - a socket of a proxy of the topology, toward the client or toward the target.
+ * - a socket of a proxy of the topology, toward the client or toward the target. Toward the target,
+ *   it is the socket that carries the request, else the first socket that the run opens.
  */
 export declare const socketOf: (cell: Cell, run: Run) => SocketPlace | undefined;
 /** This function gives what reordering did in a run, from the records of the receiver and of the sender. */

package/build/index.js

@@ -1421,6 +1421,21 @@ const opened = (run, netns, peer) => {
 		cookie: record.cookie
 	};
 };
+/**
+* This function finds the socket of a hop that carries the request of a run toward the target. It is
+* the first socket of the hop that sends data to the target after the send of the request. A pooler can
+* reuse a server that it opened before the run, in the warm-up of the environment. Then the first
+* socket that it opens in the run is another one (run 39160356281).
+*/
+const carried = (run, netns, target) => {
+	const request = run.events.filter((event) => event.ev === "sent")[1];
+	if (request === void 0) return void 0;
+	const record = run.kernel.find((each) => each.netns === netns && each.ev === "tx" && each.dst === target && (each.seg?.len.value ?? 0) > 0 && each.mono >= request.t);
+	return record?.cookie === void 0 ? void 0 : {
+		netns,
+		cookie: record.cookie
+	};
+};
 /** This function finds the socket of the target for the connection of a run. In a topology, its peer is the last hop. */
 const targetSocket = (run, local) => {
 	if (run.topology === void 0) {
@@ -1437,7 +1452,8 @@ const targetSocket = (run, local) => {
 * This function finds the socket of the observer of a cell in a run:
 * - the client socket, by the address that it connected from, in its namespace.
 * - the socket of the target for the same connection, in its namespace.
-* - a socket of a proxy of the topology, toward the client or toward the target.
+* - a socket of a proxy of the topology, toward the client or toward the target. Toward the target,
+*   it is the socket that carries the request, else the first socket that the run opens.
 */
 const socketOf = (cell, run) => {
 	const local = run.events.find((event) => event.ev === "connected")?.["local"];
@@ -1452,7 +1468,7 @@ const socketOf = (cell, run) => {
 	if (observer === "server") return targetSocket(run, local);
 	const hop = run.topology?.hops[observer.hop];
 	if (hop === void 0) return void 0;
-	return observer.toward === "client" ? opened(run, hop.netns, (dst) => ipOf(dst) === hop.client) : opened(run, hop.netns, (dst) => dst === hop.target);
+	return observer.toward === "client" ? opened(run, hop.netns, (dst) => ipOf(dst) === hop.client) : carried(run, hop.netns, hop.target) ?? opened(run, hop.netns, (dst) => dst === hop.target);
 };
 /**
 * This function finds the socket of the peer of the observer: its close is the reference of a FIN.
@@ -1978,6 +1994,8 @@ const PUMP_SLACK = {
 	value: 5,
 	unit: "ms"
 };
+/** This count is how many of the last lines of a proxy hop the lab keeps. The error of a hop that does not start gives them. */
+const TAIL_LINES = 60;
 /** This function gives the arguments of the injector for the options of a fault. */
 const optionArgs = (options) => [
 	...options.direction === void 0 ? [] : ["--direction", options.direction],
@@ -2263,12 +2281,27 @@ const openLab = async (target, kind = "tcp", topology) => {
 			const next = endpointOf(at + 1);
 			let proxy = new GenericContainer(await imageOf(hop.image)).withNetwork(targetSide);
 			if (hop.command !== void 0) proxy = proxy.withCommand([...hop.command]);
+			if (hop.environment !== void 0) proxy = proxy.withEnvironment({ ...hop.environment });
 			if (hop.files !== void 0) proxy = proxy.withCopyContentToContainer(Object.entries(hop.files).map(([path, content]) => ({
 				content: templated(content, next),
 				target: path
 			})));
 			if (hop.ready !== void 0) proxy = proxy.withWaitStrategy(Wait.forLogMessage(hop.ready));
-			const box = await proxy.start();
+			const tail = [];
+			proxy = proxy.withLogConsumer((stream) => {
+				let rest = "";
+				stream.on("data", (chunk) => {
+					const lines = (rest + chunk.toString()).split("\n");
+					rest = lines.pop() ?? "";
+					for (const line of lines) {
+						tail.push(line);
+						if (tail.length > TAIL_LINES) tail.shift();
+					}
+				});
+			});
+			const box = await proxy.start().catch((error) => {
+				throw new Error(`the hop ${hop.name} did not start: ${error instanceof Error ? error.message : String(error)}. Its last lines:\n${tail.join("\n")}`);
+			});
 			opened.push(box);
 			await join(box, clientSide);
 			const injector = await new GenericContainer(image).withNetworkMode(`container:${box.getId()}`).withAddedCapabilities("NET_ADMIN", "NET_RAW").start();