@mark1russell7/harness 0.1.0 → 0.2.0

package/README.md

@@ -46,6 +46,14 @@ ioredis sets the keepalive of its socket itself, after it connects. Thus a cell
 - **A reset** is a `REJECT --reject-with tcp-reset` rule in the namespace of the target, for the segments of the client. The next send of the client gets a reset one round trip later. It is the reference fault of the calibration.
 - **A frozen process** is the cgroup freezer (`docker pause`), while the kernel stays up.
 - **A stall** is a black hole that stops by itself, `Cell.stall` after the send of the request. The run reads the stamp of the client on the send. Then `inject clear PEER --at NS` holds the end in the VM until that instant (`lab.heal`). Thus the latency of a docker exec does not change the length of the stall.
+- **An ICMP unreachable** (`unreachable`, `Cell.icmp`) answers each segment from the observer with host, net or port unreachable, or admin prohibited. Linux keeps it as a soft error, continues to retransmit, and gives the errno of the ICMP at the timeout. Host and net unreachable also revert the backoff (RFC 6069).
+- **A simplex partition** (`simplex`, `Cell.drops`) drops the segments to the server, or the segments to the client, but not both.
+- **A FIN** (`fin`) stops the process of the peer with SIGKILL while its kernel stays up. For the client, the target starts its server as the child of a shell (`Target.command`), and `Target.kill` stops it. For the server, the harness stops the container of the client.
+- **An MTU black hole** (`mtu-blackhole`, `Cell.mtu`) drops each segment longer than the length, in the namespace that receives it.
+- **netem** (`netem`, `Cell.netem`) is tc netem on the egress of the target: loss, duplication, reordering or corruption. The outcome is a distribution.
+- **A latency** (`Cell.latency`) is tc netem delay on the egress of the target before the client connects. Thus the round trip and the RTO include it.
+
+A length match and netem act on GSO super-packets of up to 64 KB, not on segments, while TSO, GSO and GRO are on. Thus the cells of `mtu-blackhole` and `netem` turn them off on the two ends (`lab.offload`).
 
 Each run starts from kernels that keep nothing from the run before it (`lab.forget()`). When a connection closes, each kernel keeps TCP metrics for the address of its peer. These metrics set the first round trip of the next connection, and thus its RTO. The harness removes the `tcp_metrics` of the two namespaces before each run.
 
@@ -59,6 +67,8 @@ Each run starts from kernels that keep nothing from the run before it (`lab.forg
 - **Void runs.** A run is void when the timer that drives the measurement was two jiffies or more behind its wheel slot in the timed interval. The wheel does a slot in the tick when it is due. Thus such a firing measured the host, which stopped a vCPU, not the kernel. A keepalive timer that found its socket held also makes the run void. A run whose fault came too late is void too.
 - **Findings.** A finding is a wrong errno, a detector that did not fire, or a detection where the prediction is silence.
 - **Recovery.** After a stall (`prediction.recovers`), it times the send to the first segment back after the stall stopped. That segment is the answer of the server to the retransmission that got through. A run whose stall stopped outside the window of its prediction is void. A run where nothing came back is a finding.
+- **An end of file.** A FIN counts from the change of state of the socket of the peer when it closes (`peer-closed`). It ends at the change of the socket of the observer to CLOSE_WAIT. The kernel stamps the two changes.
+- **A stream.** In a stream (`probe.mode` is `stream`), the client sends `count` requests. The run keeps the time from each send to the first data after it (`samples`).
 
 ## The verdict
 
@@ -68,6 +78,7 @@ Each run starts from kernels that keep nothing from the run before it (`lab.forg
 - **Median equivalence.** The order-statistic interval of the median must be in the band, made 5 % wider.
 - **Extension.** A median across the edge of the band extends the cell from 11 runs to 22, then 44. Then the verdict is inconclusive.
 - **Recovery.** The recovery after a stall gets the same test as a detection, by its band. A detection is then a finding.
+- **Measurements.** A cell whose outcome is a distribution (`detects: null`) has no band. Its decision is `measured`, and the verdict gives the median and the slowest reply time.
 
 The distribution test needs a simulator, which does not exist at this time.
 

package/build/index.d.ts

@@ -11,16 +11,25 @@ import { KernelRecord } from "@mark1russell7/prober";
  *   It is the reference for the noise of the harness itself.
  * - `stall` is a black hole that stops by itself, `Cell.stall` after the client sends its
  *   request. The VM times it. It shows what recovers, and when.
+ * - `fin` stops the process of the peer with SIGKILL, while its kernel stays up. The kernel closes its sockets
+ *   with a FIN, or with a reset when unread data is in the queue.
+ * - `unreachable` answers each segment from the observer with an ICMP unreachable (`Cell.icmp`).
+ *   Linux keeps it as a soft error on an established socket, and it continues to retransmit.
+ * - `simplex` drops the segments in one direction only (`Cell.drops`).
+ * - `mtu-blackhole` drops each segment to the observer that is longer than `Cell.mtu` bytes.
+ * - `netem` is tc netem on the egress of the target (`Cell.netem`): loss, duplication,
+ *   reordering or corruption. Its outcome is a distribution.
  */
-type Fault = "blackhole" | "freeze-process" | "rst" | "stall";
+type Fault = "blackhole" | "freeze-process" | "rst" | "stall" | "fin" | "unreachable" | "simplex" | "mtu-blackhole" | "netem";
 /**
  * A reference is the event that the time of a detector counts from: the event that arms the
  * timer in its machine document. Keepalive counts from the last received segment. The give-up of
  * the retransmissions counts from the first retransmission of the RTO timer, not from the tail
  * loss probe one RTO before it. The request itself meets a reset, thus a reset counts from the
- * send of the client.
+ * send of the client. A FIN counts from the change of state of the socket of the peer when its
+ * process dies (`peer-closed`).
  */
-type Reference = "last-received" | "first-retransmission" | "request-sent";
+type Reference = "last-received" | "first-retransmission" | "request-sent" | "peer-closed";
 /**
  * This prediction tells when a connection moves again after a stall. That occurs at the first
  * firing of its retransmission timer after the stall stops, and its reply comes back one round
@@ -48,11 +57,15 @@ interface Recovers {
  * after the reference event, or nothing detects the fault in a window. Nothing detects a stall
  * that is shorter than the give-up, and the connection recovers from it. The prediction then also
  * tells when.
+ *
+ * A cell whose outcome is a distribution, for example under netem loss, has no band. Its
+ * prediction is `detects: null` with the reason, and its runs are measurements only.
  */
 type Prediction = {
   readonly detects: true;
   /** This name is the timer of the machine that fires first, or the event that detects the fault, with its errno. */
   readonly timer: string;
+  /** This name is the errno of the detection, or EOF for an end of file without an error. */
   readonly errno: string;
   readonly from: Reference;
   /** This time is the closed form, less the most that the timer can fire early. */
@@ -63,6 +76,10 @@ type Prediction = {
   readonly detects: false;
   readonly window: Quantity;
   readonly recovers?: Recovers;
+} | {
+  readonly detects: null;
+  readonly window: Quantity;
+  readonly why: string;
 };
 /** A target is the server that a cell connects to: a stock image. */
 interface Target {
@@ -80,6 +97,13 @@ interface Target {
     readonly set: readonly string[];
     readonly get: readonly string[];
   };
+  /**
+   * This command replaces the command of the image, for example a shell that holds the server
+   * process as its child. Then a `fin` fault can stop the server, and the container stays up.
+   */
+  readonly command?: readonly string[];
+  /** This command stops the server process in the target with SIGKILL, as argv, for a `fin` fault. */
+  readonly kill?: readonly string[];
 }
 /** A client is what the runs of a cell use: the TCP client of the harness, or a driver on its socket. */
 type Client = "tcp" | "ioredis";
@@ -97,8 +121,15 @@ interface Cell {
   /** These are the sysctls of the network namespace of the client, as the kernel spells them. The harness sets them before the client connects. */
   readonly sysctls: Readonly<Record<string, string>>;
   readonly probe: {
-    /** This mode tells if the client is idle after its first reply, or sends a request into the fault. */
-    readonly mode: "idle" | "inflight";
+    /**
+     * This mode tells if the client is idle after its first reply, or sends a request into the
+     * fault. In a stream, it sends `count` requests into it, each `interval` after the reply before.
+     */
+    readonly mode: "idle" | "inflight" | "stream";
+    /** This count is the number of requests of a stream. */
+    readonly count?: number;
+    /** This time is the pause after each reply of a stream, before the next request. */
+    readonly interval?: Quantity;
     /** This flag sets SO_KEEPALIVE without overrides for the socket: the keepalive sysctls of the namespace apply. */
     readonly keepalive: boolean;
     /** This time sets TCP_USER_TIMEOUT. Without it, the default of the kernel (off) applies. */
@@ -122,6 +153,21 @@ interface Cell {
   readonly fault: Fault;
   /** This time is the length of a stall, from the send of the request. */
   readonly stall?: Quantity;
+  /** This code is the ICMP unreachable of an `unreachable` fault. */
+  readonly icmp?: "host" | "net" | "port" | "admin";
+  /** This direction is the one that a `simplex` fault drops: the segments to the server, or to the client. */
+  readonly drops?: "to-server" | "to-client";
+  /** This length is the longest segment, in bytes, that an `mtu-blackhole` fault lets through to the observer. */
+  readonly mtu?: number;
+  /** This specification is the tc netem of a `netem` fault, on the egress of the target: for example "loss 5%". */
+  readonly netem?: string;
+  /**
+   * This delay goes on the egress of the target before the client connects, with tc netem. Thus
+   * each round trip is longer by the delay, and so is the RTO.
+   */
+  readonly latency?: Quantity;
+  /** These commands run in the target before a client connects, each as argv: for example, a large value to read. */
+  readonly targetSetup?: readonly (readonly string[])[];
   readonly prediction: Prediction;
   /** This side is the side whose detection the cell measures. Without it, the side is the client. */
   readonly observer?: Observer;
@@ -207,6 +253,11 @@ interface Measured {
    * the RTO timer. After a stall, they are each retransmission from the send, the loss probe first.
    */
   readonly steps: readonly Quantity[];
+  /**
+   * These are the reply times of a measurement-only cell. For each request after the first, a
+   * sample is the time from its send to the first data after it.
+   */
+  readonly samples?: readonly Quantity[];
   /** This count is the number of keepalive probes that the kernel sent. */
   readonly probes: number;
   /** This count is the number of segments that the kernel sent again. */
@@ -241,8 +292,11 @@ export declare const NOISE: Quantity;
 export declare const RELATIVE = .05;
 /** These are the run counts of a cell: 11, then 22 and 44 while its median crosses the edge of the band. */
 export declare const STEPS: readonly [11, 22, 44];
-/** A decision is the result of a verdict. */
-type Decision = "holds" | "fails" | "extend" | "inconclusive";
+/**
+ * A decision is the result of a verdict. A cell whose outcome is a distribution has no band, and
+ * its decision is `measured`: a record, not a test.
+ */
+type Decision = "holds" | "fails" | "extend" | "inconclusive" | "measured";
 /** An interval is a range of times. */
 interface Interval {
   readonly low: Quantity;
@@ -290,8 +344,20 @@ export declare const verdictOf: (prediction: Prediction, runs: readonly Measured
 export declare const harnessImage: () => Promise<string>;
 /** This function gives the image of the Node client. */
 export declare const nodeClientImage: () => Promise<string>;
-/** A network fault is a fault that the injector applies, by the name of its manifestation. */
-type NetworkFault = "blackhole" | "rst";
+/**
+ * A network fault is a fault that the injector applies as a filter rule. It is a black hole (both
+ * directions, or one), or a reset on the next segment. It can also be an ICMP unreachable for each
+ * segment, or a drop of each segment that is longer than a length.
+ */
+type NetworkFault = "blackhole" | "rst" | "reject" | "mtu";
+/** These are the options of a network fault: the direction of a black hole, the code of a reject, the length of an MTU drop. */
+interface FaultOptions {
+  /** This direction is relative to the namespace of the rule: `in` drops what comes from the peer, `out` what goes to it. */
+  readonly direction?: "in" | "out";
+  readonly code?: "host" | "net" | "port" | "admin";
+  /** This length is the longest segment in bytes that an MTU drop lets through. */
+  readonly above?: number;
+}
 /**
  * A fault side is the namespace that gets the rules of a fault: the namespace of the peer of the
  * observer. For the client, it is the namespace of the target (the injector). For the server, it
@@ -344,6 +410,8 @@ interface Probe {
   /** This method freezes the client with the cgroup freezer: its process stops, and its kernel continues to answer. */
   freeze(): Promise<void>;
   thaw(): Promise<void>;
+  /** This method stops the process of the client with SIGKILL. Its kernel then closes its socket with a FIN, or a reset. */
+  kill(): Promise<void>;
   readonly events: readonly ProbeEvent[];
   stop(): Promise<void>;
 }
@@ -368,12 +436,35 @@ interface Lab {
    * target. The fault is a black hole, or a reset that answers the next segment of the client. The stamps of
    * the injector bracket it.
    */
-  fault(kind: NetworkFault, at?: FaultSide): Promise<{
+  fault(kind: NetworkFault, at?: FaultSide, options?: FaultOptions): Promise<{
     t0: number;
     t1: number;
   }>;
   /** This method removes a fault. */
-  lift(kind: NetworkFault, at?: FaultSide): Promise<void>;
+  lift(kind: NetworkFault, at?: FaultSide, options?: FaultOptions): Promise<void>;
+  /**
+   * This method puts tc netem on the egress of a namespace, for example "delay 20ms" or "loss 5%",
+   * or removes it with null. The stamps of the injector bracket it.
+   */
+  netem(spec: string | null, at?: FaultSide): Promise<{
+    t0: number;
+    t1: number;
+  }>;
+  /**
+   * This method sets the TSO, GSO and GRO of a namespace. With them off, netem and a length match
+   * act on segments, not on GSO super-packets of up to 64 KB.
+   */
+  offload(on: boolean, at?: FaultSide): Promise<void>;
+  /** This method stops the server process in the target with SIGKILL, with the command of the target. */
+  kill(): Promise<void>;
+  /**
+   * This method resolves the neighbors of the path in the two directions, with a ping from each
+   * namespace. Without it, the first connection of an environment waits for ARP. Under a netem
+   * delay, that wait is one more delayed round, and its round trip and RTO are longer.
+   */
+  warm(): Promise<void>;
+  /** This method runs commands in the target before a client connects, each as argv. */
+  setup(commands: ReadonlyArray<readonly string[]>): Promise<void>;
   /**
    * This method stops a black hole at an instant on the monotonic clock of the VM. The injector
    * waits for that instant in the VM, thus the length of a stall does not include the latency of

package/build/index.js

@@ -18,10 +18,13 @@ const ERRNO = {
 };
 /** This function gives the interval between two instants on the one clock, in seconds to the microsecond. */
 const between = (from, to) => quantity(Math.round((to - from) / 1e3) / 1e6, "s");
+/** These functions give the states of a state record: before the change, and after it. TCP_ESTABLISHED is 1, TCP_CLOSE_WAIT 8. */
+const oldState = (record) => record.old;
+const newState = (record) => record.new;
 const count = (event, field) => event === void 0 ? 0 : Number(event[field] ?? 0);
 /** This function gives the view of the client on the same interval: TCP_INFO, with a sample each 2 ms. */
 const socketVantage = (cell, events, ready, request) => {
-	if (!cell.prediction.detects) return null;
+	if (!cell.prediction.detects || cell.prediction.from === "peer-closed") return null;
 	const ended = events.find((event) => event.ev === "error" || event.ev === "eof");
 	const reference = cell.prediction.from === "last-received" ? ready : cell.prediction.from === "request-sent" ? request : events.find((event) => event.ev === "kernel" && event.t >= request.t && count(event, "retransmits") > 0);
 	return ended === void 0 || reference === void 0 ? null : between(reference.t, ended.t);
@@ -38,11 +41,14 @@ const measure = (cell, run) => {
 	const { events, kernel } = run;
 	const ready = events.find((event) => event.ev === "ready");
 	const local = events.find((event) => event.ev === "connected")?.["local"];
-	const request = cell.probe.mode === "inflight" ? events.filter((event) => event.ev === "sent")[1] : ready;
+	const inflight = cell.probe.mode !== "idle";
+	const request = inflight ? events.filter((event) => event.ev === "sent")[1] : ready;
 	const server = cell.observer === "server";
 	const netns = server ? run.targetNetns : run.netns;
 	const cookie = kernel.find((record) => record.netns === netns && (server ? record.dst : record.src) === local)?.cookie;
 	const socket = kernel.filter((record) => record.netns === netns && record.cookie === cookie);
+	const peerNetns = server ? run.netns : run.targetNetns;
+	const peerCookie = kernel.find((record) => record.netns === peerNetns && (server ? record.src : record.dst) === local)?.cookie;
 	const keepalives = socket.filter((record) => record.ev === "probe" && record.what === "keepalive");
 	const retransmitted = socket.filter((record) => record.ev === "retrans");
 	/**
@@ -103,24 +109,42 @@ const measure = (cell, run) => {
 	};
 	let voided = null;
 	if (run.fault !== null) {
+		const replyFirst = server && inflight && cell.fault !== "mtu-blackhole";
 		const replied = events.some((event) => event.ev === "data" && event.t > request.t && event.t < run.fault.t1);
-		if (cell.probe.mode === "inflight" && !server && run.fault.t1 > request.t) voided = "the request went out before the fault held";
-		else if (cell.probe.mode === "inflight" && server && run.fault.t1 < request.t) voided = "the request went out into the fault, so the server never had it";
-		else if (cell.probe.mode === "inflight" && server && replied) voided = "the server's reply arrived before the fault held";
+		if (inflight && !replyFirst && run.fault.t1 > request.t) voided = "the request went out before the fault held";
+		else if (replyFirst && run.fault.t1 < request.t) voided = "the request went out into the fault, so the server never had it";
+		else if (replyFirst && replied) voided = "the server's reply arrived before the fault held";
 		else if (keepalives[0] !== void 0 && keepalives[0].mono < run.fault.t1) voided = "a keepalive probe went out before the fault held";
 	}
 	const reported = socket.find((record) => record.ev === "error" && (record.err ?? 0) !== 0);
 	const seen = server ? void 0 : events.find((event) => event.ev === "error" || event.ev === "eof");
+	const closeWait = socket.find((record) => record.ev === "state" && newState(record) === 8);
 	const outcome = reported !== void 0 ? {
 		kind: "error",
 		errno: ERRNO[reported.err] ?? `E${reported.err}`
-	} : seen?.ev === "eof" ? { kind: "eof" } : seen !== void 0 ? {
+	} : closeWait !== void 0 || seen?.ev === "eof" ? { kind: "eof" } : seen !== void 0 ? {
 		kind: "error",
 		errno: String(seen["errno"])
 	} : { kind: "silent" };
-	const ended = reported?.mono ?? seen?.t;
+	const ended = reported?.mono ?? closeWait?.mono ?? seen?.t;
 	const noticed = reported !== void 0 && seen !== void 0 ? between(reported.mono, seen.t) : null;
 	const { prediction } = cell;
+	if (prediction.detects === null) {
+		const sends = events.filter((event) => event.ev === "sent").slice(1);
+		const samples = sends.flatMap((send, n) => {
+			const next = sends[n + 1]?.t ?? Infinity;
+			const reply = events.find((event) => event.ev === "data" && event.t > send.t && event.t < next);
+			return reply === void 0 ? [] : [between(send.t, reply.t)];
+		});
+		return {
+			...base,
+			...nothing,
+			outcome: ended !== void 0 ? outcome : { kind: "answered" },
+			steps: retransmitted.filter((record) => record.mono > request.t).map((record) => between(request.t, record.mono)),
+			samples,
+			void: voided
+		};
+	}
 	if (!prediction.detects && prediction.recovers !== void 0) {
 		const { recovers } = prediction;
 		const heal = run.heal ?? null;
@@ -183,7 +207,7 @@ const measure = (cell, run) => {
 		finding: `nothing detected it: ${prediction.timer} never fired`
 	};
 	const before = run.fault?.t1 ?? ready.t;
-	const kernelReference = prediction.from === "last-received" ? socket.filter((record) => record.ev === "rx" && record.mono <= before).at(-1) : prediction.from === "first-retransmission" ? retransmitted.find((record) => record.why === "rto") : void 0;
+	const kernelReference = prediction.from === "last-received" ? socket.filter((record) => record.ev === "rx" && record.mono <= before).at(-1) : prediction.from === "first-retransmission" ? retransmitted.find((record) => record.why === "rto") : prediction.from === "peer-closed" ? kernel.find((record) => record.netns === peerNetns && record.cookie === peerCookie && record.ev === "state" && oldState(record) === 1) : void 0;
 	const reference = prediction.from === "request-sent" ? request.t : kernelReference?.mono;
 	if (reference === void 0) return {
 		...base,
@@ -194,8 +218,8 @@ const measure = (cell, run) => {
 		finding: `it ended without its reference event (${prediction.from})`
 	};
 	const steps = (prediction.from === "last-received" ? keepalives : prediction.from === "first-retransmission" ? retransmitted.filter((record) => record.why === "rto" && record.mono > reference) : []).map((record) => between(reference, record.mono));
-	const finding = outcome.kind === "error" && outcome.errno === prediction.errno ? null : `it ended with ${outcome.kind === "error" ? outcome.errno : "end of file"}, not ${prediction.errno}`;
-	if (prediction.from !== "request-sent") voided ??= delayed(reference, ended, prediction.from === "last-received" ? "keepalive" : "retransmit");
+	const finding = (prediction.errno === "EOF" ? outcome.kind === "eof" : outcome.kind === "error" && outcome.errno === prediction.errno) ? null : `it ended with ${outcome.kind === "error" ? outcome.errno : "end of file"}, not ${prediction.errno}`;
+	if (prediction.from !== "request-sent" && prediction.from !== "peer-closed") voided ??= delayed(reference, ended, prediction.from === "last-received" ? "keepalive" : "retransmit");
 	return {
 		...base,
 		outcome,
@@ -284,6 +308,20 @@ const verdictOf = (prediction, runs) => {
 		outside,
 		why: "no run counted"
 	};
+	if (prediction.detects === null) {
+		const samples = counted.flatMap((run) => (run.samples ?? []).map((sample) => valueIn(sample, "s"))).sort((a, b) => a - b);
+		const retransmissions = counted.reduce((sum, run) => sum + run.retransmissions, 0);
+		const ended = counted.filter((run) => run.outcome.kind === "error" || run.outcome.kind === "eof");
+		const why = samples.length === 0 ? `${counted.length} runs measured, with no reply: ${prediction.why}` : `${counted.length} runs, ${samples.length} replies: median ${s(middle(samples)).value} s, slowest ${s(samples.at(-1)).value} s, ${retransmissions} retransmissions, ${ended.length} connections ended. ${prediction.why}`;
+		return {
+			...empty,
+			decision: "measured",
+			band: null,
+			outside: [],
+			median: samples.length === 0 ? null : s(middle(samples)),
+			why
+		};
+	}
 	if (!prediction.detects && prediction.recovers === void 0) {
 		const decision = outside.length === 0 ? "holds" : "fails";
 		const why = decision === "holds" ? `nothing detected it in ${counted.length} runs of ${valueIn(prediction.window, "s")} s` : `${outside.length} of ${counted.length} runs detected it`;
@@ -457,6 +495,19 @@ const build = (image) => {
 const harnessImage = () => build(HARNESS);
 /** This function gives the image of the Node client. */
 const nodeClientImage = () => build(NODE);
+/** This function gives the arguments of the injector for the options of a fault. */
+const optionArgs = (options) => [
+	...options.direction === void 0 ? [] : ["--direction", options.direction],
+	...options.code === void 0 ? [] : ["--code", options.code],
+	...options.above === void 0 ? [] : ["--above", String(options.above)]
+];
+/** These are the actions of the injector that lift each network fault. */
+const LIFT = {
+	blackhole: "clear",
+	rst: "clear-rst",
+	reject: "clear-reject",
+	mtu: "clear-mtu"
+};
 const lines = () => {
 	const seen = [];
 	const other = [];
@@ -539,7 +590,8 @@ const openLab = async (target, kind = "tcp") => {
 	};
 	try {
 		network = await new Network().start();
-		const server = await new GenericContainer(target.image).withNetwork(network).withWaitStrategy(Wait.forLogMessage(target.ready)).start();
+		const stock = new GenericContainer(target.image).withNetwork(network).withWaitStrategy(Wait.forLogMessage(target.ready));
+		const server = await (target.command === void 0 ? stock : stock.withCommand([...target.command])).start();
 		opened.push(server);
 		const client = await new GenericContainer(image).withNetwork(network).start();
 		opened.push(client);
@@ -620,26 +672,80 @@ const openLab = async (target, kind = "tcp") => {
 				}
 				return read;
 			},
-			async fault(kind, at = "target") {
+			async fault(kind, at = "target", options = {}) {
 				const [where, peer] = at === "client" ? [admin, targetIp] : [injector, clientIp];
-				const stamp = JSON.parse(await output(where, [
+				const args = [
 					"/opt/harness/inject",
 					kind,
-					peer
-				]));
+					peer,
+					...optionArgs(options)
+				];
+				const stamp = JSON.parse(await output(where, args));
 				return {
 					t0: stamp.t0,
 					t1: stamp.t1
 				};
 			},
-			async lift(kind, at = "target") {
+			async lift(kind, at = "target", options = {}) {
 				const [where, peer] = at === "client" ? [admin, targetIp] : [injector, clientIp];
 				await output(where, [
 					"/opt/harness/inject",
-					kind === "blackhole" ? "clear" : "clear-rst",
-					peer
+					LIFT[kind],
+					peer,
+					...optionArgs(options)
 				]);
 			},
+			async netem(spec, at = "target") {
+				const where = at === "client" ? admin : injector;
+				const args = spec === null ? [
+					"/opt/harness/inject",
+					"clear-netem",
+					"eth0"
+				] : [
+					"/opt/harness/inject",
+					"netem",
+					"eth0",
+					...spec.split(/\s+/)
+				];
+				const stamp = JSON.parse(await output(where, args));
+				return {
+					t0: stamp.t0,
+					t1: stamp.t1
+				};
+			},
+			async offload(on, at = "target") {
+				await output(at === "client" ? admin : injector, [
+					"/opt/harness/inject",
+					"offload",
+					"eth0",
+					on ? "on" : "off"
+				]);
+			},
+			async kill() {
+				if (target.kill === void 0) throw new Error(`${target.image} names no way to kill its server`);
+				await output(server, [...target.kill]);
+			},
+			async warm() {
+				await output(admin, [
+					"ping",
+					"-c",
+					"2",
+					"-W",
+					"2",
+					targetIp
+				]);
+				await output(injector, [
+					"ping",
+					"-c",
+					"2",
+					"-W",
+					"2",
+					clientIp
+				]);
+			},
+			async setup(commands) {
+				for (const argv of commands) await output(server, [...argv]);
+			},
 			async heal(at, when) {
 				const [where, peer] = at === "client" ? [admin, targetIp] : [injector, clientIp];
 				const stamp = JSON.parse(await output(where, [
@@ -679,7 +785,8 @@ const openLab = async (target, kind = "tcp") => {
 					go: () => output(container, ["touch", "/tmp/go"]).then(() => void 0),
 					freeze: () => runtime.container.getById(container.getId()).pause().then(() => void 0),
 					thaw: () => runtime.container.getById(container.getId()).unpause().then(() => void 0),
-					stop: () => container.stop(STOP).then(() => void 0)
+					kill: () => runtime.container.getById(container.getId()).kill({ signal: "SIGKILL" }).then(() => void 0),
+					stop: () => container.stop(STOP).then(() => void 0).catch(() => void 0)
 				};
 			},
 			async settle() {
@@ -765,6 +872,8 @@ const probeArgs = (cell, lab) => {
 		probe.mode
 	];
 	if (probe.request !== void 0) args.push("--request", probe.request);
+	if (probe.count !== void 0) args.push("--count", String(probe.count));
+	if (probe.interval !== void 0) args.push("--interval-ms", String(valueIn(probe.interval, "ms")));
 	if (probe.driver !== void 0) {
 		args.push("--keepalive-ms", String(valueIn(probe.driver.keepAlive, "ms")));
 		args.push("--max-s", String(Math.ceil(longest * 2 + 10)));
@@ -777,13 +886,14 @@ const probeArgs = (cell, lab) => {
 };
 /**
 * This function waits for an error on the socket of the target for the connection of the
-* client. The prober records that error in the namespace of the target. A cell on the server side
-* waits for that. The promise resolves after `within` ms at the latest.
+* client, or for its end of file (CLOSE_WAIT, state 8). The prober records both in the namespace
+* of the target. A cell on the server side waits for that. The promise resolves after `within` ms
+* at the latest.
 */
 const serverError = (lab, local, within) => new Promise((resolve) => {
 	const started = Date.now();
 	const look = () => {
-		if (lab.kernel.some((record) => record.ev === "error" && (record.err ?? 0) !== 0 && record.netns === lab.netns.target && record.dst === local) || Date.now() - started > within) resolve();
+		if (lab.kernel.some((record) => record.netns === lab.netns.target && record.dst === local && (record.ev === "error" && (record.err ?? 0) !== 0 || record.ev === "state" && record.new === 8)) || Date.now() - started > within) resolve();
 		else setTimeout(look, 50);
 	};
 	look();
@@ -813,6 +923,11 @@ const requestSent = (probe, within) => new Promise((resolve, reject) => {
 * next run. The run keeps the records of the prober from its start to its end. The frozen process
 * is the peer of the observer: the target, or the client for the server.
 */
+/** This function gives the netem specification of a cell: its latency, then the netem of its fault when it has one. */
+const netemSpec = (cell, fault) => {
+	const parts = [...cell.latency === void 0 ? [] : [`delay ${valueIn(cell.latency, "ms")}ms`], ...fault === void 0 ? [] : [fault]];
+	return parts.length === 0 ? null : parts.join(" ");
+};
 const once = async (lab, cell, environment, index) => {
 	await lab.forget();
 	const from = lab.kernel.length;
@@ -834,17 +949,51 @@ const once = async (lab, cell, environment, index) => {
 	let heal = null;
 	try {
 		await probe.until(["ready"], 15e3);
-		const replyInFlight = server && cell.probe.mode === "inflight";
+		const replyInFlight = cell.probe.mode === "inflight" && (server && cell.fault !== "mtu-blackhole" || cell.fault === "fin");
 		if (replyInFlight) {
 			await go();
 			await new Promise((resolve) => setTimeout(resolve, 200));
 		}
 		const faultAt = server ? "client" : "target";
-		const network = cell.fault === "stall" ? "blackhole" : cell.fault;
-		fault = network === "freeze-process" ? null : await lab.fault(network, faultAt);
-		if (network === "freeze-process") await peer.freeze();
+		const direction = cell.drops === void 0 ? void 0 : cell.drops === "to-server" === (faultAt === "target") ? "in" : "out";
+		const options = {
+			...direction === void 0 ? {} : { direction },
+			...cell.icmp === void 0 ? {} : { code: cell.icmp },
+			...cell.mtu === void 0 ? {} : { above: cell.mtu }
+		};
+		const apply = async () => {
+			switch (cell.fault) {
+				case "freeze-process":
+					await peer.freeze();
+					return null;
+				case "fin":
+					await (server ? probe.kill() : lab.kill());
+					return null;
+				case "unreachable": return lab.fault("reject", faultAt, { code: cell.icmp ?? "host" });
+				case "simplex": return lab.fault("blackhole", faultAt, options);
+				case "mtu-blackhole": return lab.fault("mtu", "client", { above: cell.mtu ?? 1200 });
+				case "netem": return lab.netem(netemSpec(cell, cell.netem), "target");
+				case "rst": return lab.fault("rst", faultAt);
+				default: return lab.fault("blackhole", faultAt);
+			}
+		};
+		const lift = async () => {
+			switch (cell.fault) {
+				case "freeze-process": return peer.thaw();
+				case "fin": return;
+				case "unreachable": return lab.lift("reject", faultAt, { code: cell.icmp ?? "host" });
+				case "simplex": return lab.lift("blackhole", faultAt, options);
+				case "mtu-blackhole": return lab.lift("mtu", "client", { above: cell.mtu ?? 1200 });
+				case "netem":
+					await lab.netem(netemSpec(cell, void 0), "target");
+					return;
+				case "rst": return lab.lift("rst", faultAt);
+				default: return heal === null ? lab.lift("blackhole", faultAt) : void 0;
+			}
+		};
+		fault = await apply();
 		try {
-			if (cell.probe.mode === "inflight" && !replyInFlight) await go();
+			if (cell.probe.mode !== "idle" && !replyInFlight) await go();
 			const { prediction } = cell;
 			if (cell.fault === "stall") {
 				if (cell.stall === void 0) throw new Error(`${cell.name} says no stall's length`);
@@ -856,6 +1005,14 @@ const once = async (lab, cell, environment, index) => {
 					"error",
 					"eof"
 				], window * 1e3).catch(() => void 0);
+			} else if (prediction.detects === null) {
+				const window = valueIn(prediction.window, "s") * 1e3;
+				await probe.until([
+					"done",
+					"error",
+					"eof",
+					"deadline"
+				], window).catch(() => void 0);
 			} else if (prediction.detects) {
 				const within = valueIn(prediction.high, "s") * 2e3 + 1e4;
 				if (server) {
@@ -868,8 +1025,7 @@ const once = async (lab, cell, environment, index) => {
 				], within).catch(() => void 0);
 			} else await new Promise((resolve) => setTimeout(resolve, valueIn(prediction.window, "s") * 1e3));
 		} finally {
-			if (network === "freeze-process") await peer.thaw();
-			else if (heal === null) await lab.lift(network, faultAt);
+			await lift();
 		}
 		if (cell.fault === "freeze-process" && !server) await probe.until(["data"], 5e3).catch(() => void 0);
 	} finally {
@@ -897,6 +1053,14 @@ const inEnvironment = async (cell, environment, count, say) => {
 	try {
 		const settings = await lab.configure(cell.targetSettings ?? {});
 		const targetSysctls = await lab.targetSysctl(cell.targetSysctls ?? {});
+		if (cell.targetSetup !== void 0) await lab.setup(cell.targetSetup);
+		if (cell.fault === "mtu-blackhole" || cell.fault === "netem") {
+			await lab.offload(false, "target");
+			await lab.offload(false, "client");
+		}
+		const latency = netemSpec(cell, void 0);
+		if (latency !== null) await lab.netem(latency, "target");
+		await lab.warm();
 		const manifest = await lab.manifest(await lab.sysctl(cell.sysctls), settings, targetSysctls);
 		const runs = [];
 		for (let index = 0; index < count; index += 1) {