@mark1russell7/harness 0.7.0 → 0.9.0

package/README.md

@@ -13,6 +13,7 @@ A **cell** (`cell.ts`) is all that one measurement needs, as data:
 - **The workload.** A connection that stays idle, a connection with a request in flight, or a stream of requests. A stream of `GET` requests for a long value keeps many segments in flight.
 - **The fault.** A black hole, a reset, a frozen process, or a stall. In a topology, also a fault at a router or a proxy, or a router that forgets a flow. Also a proxy that closes an idle connection, or a toxic of Toxiproxy.
 - **The prediction.** An interval [T_lo, T_hi] after the event that arms the timer, or silence in a window. After a stall, the prediction also tells when the connection recovers. A silent prediction can also tell when the reply comes back (`replies`).
+- **A detection by the program.** The timer can be a timer of the program of the observer, for example a heartbeat. The program then drops the connection itself (`DROPPED`). A detection can also give the waits of the reconnect of the client (`reconnects`).
 - **The assumptions of the band.** A band can name the range of first round trips that it assumes (`rtt`), and the proxy times that it includes (`through`).
 
 ## The environment
@@ -24,6 +25,7 @@ A **cell** (`cell.ts`) is all that one measurement needs, as data:
 - A privileged sidecar in the namespace of the client, for its sysctls and its routes. Docker Desktop keeps `/proc/sys` read-only in a container that operates.
 - Each hop of the topology, on the network of each of its sides.
 - An injector sidecar in the namespace of the target, with NET_ADMIN. Thus the image of the target stays stock.
+- The manifest of each environment gives the HZ of its kernel, from /proc/config.gz. A host without that file (a hosted runner) gets HZ from two readings of /proc/timer_list, one second apart: its jiffies against the time of the kernel (`hzOf`). `hzFrom` tells which.
 - The prober ([`@mark1russell7/prober`](https://www.npmjs.com/package/@mark1russell7/prober)): a privileged container that loads the kernel program. It watches the namespace of the client, of the target and of each hop. Each run keeps its records from its start to its end. A run whose records the prober lost is void.
 
 Each command is an argv array.
@@ -105,6 +107,12 @@ A fault names a member by its role at the start of a run: `primary`, or `replica
   - or by itself, a gap after its first reply (`gap`), for the keep-alive race.
 - **The lines of an HTTP client.** It writes the lines of the TCP client, with the number `i` of the request on each line of a request. It also writes the connection of each try of a request (`request`, then `retry`), with its local address and a flag for a reused connection. Each connection gets a `conn` line, and each end of a connection a line (`eof`, `conn-error`, `closed`). The end of the second request is a `done` line.
 - **A target with a script.** A target can take files before it starts (`Target.files`), for example the script of a Node server on the stock Node image. A target that writes JSON lines (`Target.lines`) gives them to each run (`Run.server`). The manifest holds the sha256 of each file and the first line of the target.
+- **The WebSocket client** (`Cell.client` is `procedures-ws`) is the `WebSocketTransport` of `@procedures/client` 0.3.0. The cell gives each of its settings (`probe.ws`): the class of its sockets, the heartbeat, the reconnect, and the times of a call.
+  - Its first request is `harness.echo`. In flight, its request into the fault is `harness.echo`, or `harness.hold` for a held reply.
+  - It writes the lines of the TCP client. It also writes a line for each act of the program: `ping`, `drop`, `reconnecting` and `dial`.
+  - The class of its sockets is the global `WebSocket` of Node, or the `WebSocket` of the `ws` package (the option `webSocket`). The transport gets a subclass of that class. The subclass writes a line, then gives the work to its class. Thus the transport, undici and ws operate as they do without it.
+  - The subclass has `terminate()` only when its class has it, because the transport drops a socket with `terminate()` when it exists.
+- **A target from an image of the harness** (`Target.harness`). The target operates an image of the harness in place of its stock image. The procedures server is a script in the image of the WebSocket client (`server-ws.mjs`). Thus the client and the server use one release of the package. `Target.image` is the stock base of that image, and the manifest names both.
 - **An agent** (`lab.agent(image, command, { pid })`) is a container in the network namespace of the target. With `pid`, it also shares the process namespace of the target, thus it can signal the server process. Its JSON lines and its `exec` stay available to the caller.
 
 ## The images
@@ -125,6 +133,7 @@ The harness builds six images. The tag of each image comes from the content of a
   - The events of the driver are `sdam` lines (the heartbeats and the changes of each server), `cmap` lines (the pools) and `command` lines (each try of an operation).
   - Each operation is a `sent` line, then an `ack` line or a `fail` line. An `ack` line has the server that accepted it. A `fail` line has the name and the labels of its error.
   - The same image holds `stepdown-mongo.mjs`, the sender of `replSetStepDown`.
+- **The procedures image** (`image/procedures`) holds `@procedures/client` 0.3.0 and `ws` 8.22.0 on Node 24.21.0, whose WebSocket is undici 7.29.1. The digest of the image and an npm lockfile pin them. It holds the WebSocket client (`client-ws.mjs`) and the script of the procedures server (`server-ws.mjs`). Each takes each of its settings as a flag, and it has no default.
 - **The router image** (`image/router`) is a digest-pinned Alpine with conntrack-tools, nftables, iproute2 and Python. `mbx.sh` sets the form of the router in one nftables table, `ctwatch.py` writes the conntrack events, and `resetter.py` sends the resets of the `reset-both` form. Each run records its packages in the manifest.
 
 The build does the `cargo test` of the two crates. Thus the record decoder passes its tests before an image exists. Each amount that the native programs print is a quantity in a canonical unit. For example, an RTO is in `jiffy`, a round trip in `us`, and a segment in `B`.
@@ -137,7 +146,7 @@ ioredis sets the keepalive of its socket itself, after it connects. Thus a cell
 - **A reset** is a `REJECT --reject-with tcp-reset` rule in the namespace of the target, for the segments of the client. The next send of the client gets a reset one round trip later. It is the reference fault of the calibration.
 - **A frozen process** is the cgroup freezer (`docker pause`), while the kernel stays up. A target that names its server process (`Target.program`) freezes by SIGSTOP from a sidecar in its process namespace, with stamps (`lab.signal`). A hold (`Cell.hold`) then thaws it with SIGCONT at that time after the freeze.
 - **A fault before the client** (`Cell.before`) holds before the client starts. Thus the first selection of the driver meets it. It is a black hole, a reset or an ICMP unreachable.
-- **A stall** is a black hole that stops by itself, `Cell.stall` after the send of the request. The run reads the stamp of the client on the send. Then `inject clear PEER --at NS` holds the end in the VM until that instant (`lab.heal`). Thus the latency of a docker exec does not change the length of the stall.
+- **A stall** is a black hole that stops by itself, `Cell.stall` after the send of the request. The run reads the stamp of the client on the send. Then `inject clear PEER --at NS` holds the end in the VM until that instant (`lab.heal`). Thus the latency of a docker exec does not change the length of the stall. An idle client sends only the pings of its heartbeat, thus its stall counts from its first ping into the stall.
 - **An ICMP unreachable** (`unreachable`, `Cell.icmp`) answers each segment from the observer with host, net or port unreachable, or admin prohibited. Linux keeps it as a soft error, continues to retransmit, and gives the errno of the ICMP at the timeout. Host and net unreachable also revert the backoff (RFC 6069).
 - **A simplex partition** (`simplex`, `Cell.drops`) drops the segments to the server, or the segments to the client, but not both.
 - **A FIN** (`fin`) stops the process of the peer with SIGKILL while its kernel stays up. For the client, the target starts its server as the child of a shell (`Target.command`), and `Target.kill` stops it. For the server, the harness stops the container of the client.
@@ -178,6 +187,7 @@ Each run starts from kernels that keep nothing from the run before it (`lab.forg
 - **The time.** It times the detection from the reference event of the prediction to the error report of the kernel. eBPF stamps the two events. For keepalive, the reference is the last received segment. For the give-up, it is the first retransmission of the RTO timer, not the tail loss probe one RTO before it. For a reset, it is the send of the client, which no kernel record marks. The client stamps the send before its write.
 - **The other views.** It also gives the same interval from the `TCP_INFO` samples of the client, and the delay before the client saw the error. It also gives the steps of the kernel: each keepalive probe, or each expiry of the RTO timer.
 - **Void runs.** A run is void when the timer that drives the measurement was two jiffies or more behind its wheel slot in the timed interval. The wheel does a slot in the tick when it is due. Thus such a firing measured the host, which stopped a vCPU, not the kernel. A keepalive timer that found its socket held also makes the run void. A run whose fault came too late is void too.
+  - The rule is in time when the run knows its HZ (`Run.hz`, from the manifest). Then the delay must reach two jiffies and δ_abs, the larger of the two (`delayVoids`). Each band includes δ_abs, thus a smaller delay does not change a result. At HZ 250 two jiffies are 8 ms, more than a δ_abs of 4 ms, and the rule stays the same. At HZ 1000 two jiffies are 2 ms. A run without its HZ keeps the rule in jiffies.
 - **The first round trip.** The measurement gives the first round trip of the socket of the observer (`rtt`). It is the smoothed round trip in the first record of the socket: the sample of the handshake. A run whose first round trip is above the range of its prediction is void, with the reason `rtt-over-assumption` and the value. Such a run measured another path than the band assumes. For example, a round trip of 4.826 ms makes an RTO of 52 jiffies, and a ladder that assumes 1 ms has 51.
 - **Findings.** A finding is a wrong errno, a detector that did not fire, or a detection where the prediction is silence. A router that forgot the flow in the window of a silent prediction is also a finding.
 - **A reply.** A silent prediction with `replies` times the send of the request to the first segment with data that comes back. A proxy can acknowledge the request first, with no data.
@@ -185,6 +195,20 @@ Each run starts from kernels that keep nothing from the run before it (`lab.forg
 - **A topology.** A run in a topology also gives the bytes that the client read after its request, and when a router forgot its flow (`forgot`).
 - **Recovery.** After a stall (`prediction.recovers`), it times the send to the first segment back after the stall stopped. That segment is the answer of the server to the retransmission that got through. A run whose stall stopped outside the window of its prediction is void. A run where nothing came back is a finding.
 - **An end of file.** A FIN counts from the change of state of the socket of the peer when it closes (`peer-closed`). It ends at the change of the socket of the observer to CLOSE_WAIT. The kernel stamps the two changes.
+- **A detection by the program** (`application.ts`). The kernel has no report of it. Thus the program of the observer writes a line before each of its own acts, and the kernel stamps each act on the socket.
+  - The program writes `ping` before each ping of its heartbeat. The kernel stamps the write of the ping (`tx`).
+  - The reference is the first ping that got no answer (`ping-sent`): no segment with data reached the socket after its write, up to the detection. Its write starts the timer of the heartbeat that the fault met. The fault can come at any time, and its stamps are not necessary.
+  - The program writes `drop` before it drops the connection itself. The detection is the kernel's stamp of that drop: the write of its close frame, or the change of its socket out of ESTABLISHED. Without such a record in δ_abs, it is the stamp of the line.
+  - A drop counts when it comes before each report of the kernel. A report of the kernel that comes first is the detection, and the drop after it is not.
+  - The second view is the stamps of the program: from its `ping` line to its `drop` line. The steps are the later pings, each from the reference.
+  - The lines of a client name the socket by its local address (`local`), and the lines of a server by its peer (`peer`). For the server, the lines are the lines of the target. A line whose address is null is about a try that did not open.
+  - A prediction can say how the program drops (`drop`). A close frame (`frame`) is a write, and the socket stays ESTABLISHED after it. A close (`close`) closes the socket in δ_abs: a FIN or an abort. Another drop is a finding. A FIN of the peer that answers a close frame is no close by the program. The run gives the time from the `drop` line to the close by the program (`left`).
+  - A heartbeat is a timer of the program. The retransmit timer of its ping fires in the same interval. Thus a run whose retransmit timer was two jiffies or more behind its wheel slot is void: the VM stopped a vCPU.
+- **The reconnect** (`reconnectOf`). A prediction with `reconnects` also reads what the client did after the detection.
+  - Each wait goes from a `reconnecting` line, where the program starts its timer, to its next `dial` line, where it makes the socket. A wait outside its band, widened by δ_abs, is a finding. A try that did not come is a finding too.
+  - The kernel gives the time from the detection to the first SYN of a new socket (`syn`). It also gives the time to the first new socket in ESTABLISHED (`established`).
+  - It also gives the time from each `dial` line to its SYN (`connects`): the work of the program before connect(2).
+- **A call in flight of a WebSocket client** (`callOf`). The run gives its error or its reply, and the time from the detection to its end.
 - **A stream.** In a stream (`probe.mode` is `stream`), the client sends `count` requests. The run keeps the time from each send to the last byte of its reply (`samples`). A record without `replied` lines gives the time to the first data.
 - **Reordering.** A measurement-only run also gives what reordering, duplicates and losses did (`reordering`). At the client, it counts the arrivals beyond a hole and the arrivals of data that it had already, against `rcv_nxt`. From the sender, it gives the most SACKed and lost segments, the DSACKs, the fast retransmissions, and the reordering degree and count of the kernel.
 - **An arrival is not always one segment.** The segments that arrive while the owner holds the socket wait in its backlog, and the kernel joins the contiguous ones. Thus the kernel counts out-of-order segments (`rcv_ooopack`), and the prober counts arrivals.
@@ -219,7 +243,9 @@ Each run starts from kernels that keep nothing from the run before it (`lab.forg
 
 `verdictOf(prediction, runs)` (`verdict.ts`) judges a cell:
 
-- **Containment.** Each run lands in [T_lo − δ_abs, T_hi + δ_abs]. `NOISE` (δ_abs) is 4 ms, from the calibration: the worst reset took 1.141 ms, and δ_abs is one jiffy at the minimum.
+- **Containment.** Each run lands in [T_lo − δ_abs, T_hi + δ_abs]. `NOISE` (δ_abs) is 4 ms, from the calibration of the first environment: the worst reset took 1.141 ms, and δ_abs is one jiffy at the minimum.
+  - **δ_abs of an environment.** A cell can give the δ_abs of its environment (`Cell.noise`). Stage 0 measures it in each environment. On a hosted runner of GitHub at HZ 1000, the worst reset took 0.147 ms, thus δ_abs is one jiffy, 1 ms. These read it (`noiseOf`): the bands, the pairing of a program line with its kernel record, and the drop of a program.
+  - **The injector has its own bound.** A fault that the injector commits or lifts late makes a run void. That bound is `INJECT` (4 ms), not δ_abs. Stage 0 measures the commit apart. On a hosted runner it took up to 2.75 ms at 4 environments.
 - **Median equivalence.** The order-statistic interval of the median must be in the band, made 5 % wider.
 - **Extension.** A median across the edge of the band extends the cell from 11 runs to 22, then 44. Then the verdict is inconclusive.
 - **Recovery.** The recovery after a stall gets the same test as a detection, by its band. A detection is then a finding.
@@ -234,6 +260,9 @@ Each run starts from kernels that keep nothing from the run before it (`lab.forg
   - The verdict also gives the median of each stage, the lost writes of each run, and the errors on the way.
 
 - **An HTTP client** (`Verdict.requests`). The verdict counts each outcome of the second request over the counted runs. A measurement-only cell also gives them in its reason.
+- **The reconnect** (`Verdict.reconnect`). A wait outside its band is a finding of its run, thus the cell fails. The verdict gives the spread of each wait, of the first SYN, of the first ESTABLISHED and of the work before each connect(2). Its reason says the medians.
+- **A call in flight** (`Verdict.calls`). The verdict counts each outcome of the call in flight of a WebSocket client, and its reason says them.
+- **The socket after a drop** (`Verdict.left`). The verdict gives the spread of the time from a drop to the close of its socket by the program. It also counts the runs where the program did not close it.
 
 The distribution test needs a simulator, which does not exist at this time.
 
@@ -255,6 +284,7 @@ The distribution test needs a simulator, which does not exist at this time.
   - A run of a group has no such pause. Its fault comes after the 20th accepted write of its stream, and each band of a group counts from the fault.
   - An HTTP client in a gap counts its gap from its own first reply, thus the pause does not change it. Its run ends at the end of its second request (`done`).
 - **An HTTP client.** A run of an HTTP client in flight ends at the end of its second request, after a new try of the client. The client waits one second after it (`LINGER_MS`), thus its own close comes after the window.
+- **A program with its own detector.** The wait for the detection also ends at a `drop` line of the client, or of a target that writes lines. After the harness removes the fault, the run waits for the tries that the prediction gives: one `dial` line for each.
 - **`judge(cell, runs)`** (`run.ts`) measures and judges the raw records of a cell with the code at this time. `runCell` uses it during a cell. A holder of raw records can judge them again after a change to `observe.ts` or to the verdict.
 - **Testcontainers loads when a lab first opens.** Code that only measures and judges does not load it.
 
@@ -274,7 +304,11 @@ The tests also measure a group run from test lines. They find the first accepted
 
 The tests of a replica set read the log of mongod, and they put its lines on the monotonic clock. They test the views that make a set healthy and the peers of each cut. They measure the stages of a set and the bands of a driver from test lines.
 
-The cells themselves need Docker and privileged containers. CI does not start them. `HARNESS_LAB=1` starts the lab tests on Docker. They start a client image and an agent, and they put a reset and netem at a router. They also fail a group of Valkey 9.1 over with a planned failover. Each HTTP client sends two requests on one connection to a Node server, from a script in the stock Node image.
+The tests of `application.ts` measure a heartbeat of a program from test lines and records. They find the first ping without an answer, the drop, the waits of the reconnect and the call in flight. They also test a report of the kernel before the drop, a VM that stopped, the heartbeat of a server, and an idle stall.
+
+The cells themselves need Docker and privileged containers. CI does not start them. `HARNESS_LAB=1` starts the lab tests on Docker. They start a client image and an agent, and they put a reset and netem at a router. They also fail a group of Valkey 9.1 over with a planned failover.
+
+In the lab tests, each HTTP client sends two requests on one connection to a Node server, from a script in the stock Node image. The procedures WebSocket client drops a silent connection on its heartbeat, and it connects again to the procedures server. With the global `WebSocket`, the drop writes a close frame. With the `WebSocket` of ws, the socket leaves ESTABLISHED at once.
 
 The lab tests also step a replica set of three members of MongoDB 9.0.4 down, and they write the free memory of the machine. They also reset a write of the MongoDB driver in flight to a set of one member.
 

package/build/index.d.ts

@@ -219,8 +219,12 @@ interface Toxic {
  *
  * A client that starts into a fault counts from its `connecting` line, or from its first line
  * without one (`client-start`).
+ *
+ * A heartbeat of a program counts from its first ping that got no answer (`ping-sent`). The
+ * program writes a `ping` line before each ping, and the kernel stamps the write (`tx`). No
+ * segment with data reaches the socket after that write, up to the detection.
  */
-type Reference = "last-received" | "first-retransmission" | "request-sent" | "peer-closed" | "fault" | "client-start";
+type Reference = "last-received" | "first-retransmission" | "request-sent" | "peer-closed" | "fault" | "client-start" | "ping-sent";
 /**
  * This prediction tells when a connection moves again after a stall. That occurs at the first
  * firing of its retransmission timer after the stall stops, and its reply comes back one round
@@ -397,6 +401,18 @@ interface DriverBand {
   readonly quiet?: readonly LineMatch[];
   readonly then?: readonly DriverStep[];
 }
+/**
+ * A reconnect prediction tells when the program of the client tries a new connection after the
+ * detection. Each wait counts from the start of the reconnect timer of the program (a
+ * `reconnecting` line) to the try (a `dial` line). The first wait follows the detection, and each
+ * next wait follows the failure of the try before it.
+ */
+interface Reconnects {
+  readonly waits: ReadonlyArray<{
+    readonly low: Quantity;
+    readonly high: Quantity;
+  }>;
+}
 /**
  * A prediction is what a cell predicts. Either a detection occurs in an interval [T_lo, T_hi]
  * after the reference event, or nothing detects the fault in a window. Nothing detects a stall
@@ -419,6 +435,10 @@ interface DriverBand {
  * of the observer itself: its pool closes an idle connection, a FIN that it sends.
  *
  * A band of a driver (`driver`) counts from a socket of the driver to a line of the client.
+ *
+ * The errno `DROPPED` is a drop by the program of the observer itself. Its own timer gave up, and
+ * it closed its socket or wrote a close frame. A detection can also tell when the client tries a
+ * new connection (`reconnects`).
  */
 type Prediction = {
   readonly detects: true;
@@ -426,7 +446,8 @@ type Prediction = {
   readonly timer: string;
   /**
    * This name is the errno of the detection, or EOF for an end of file without an error. It is
-   * OK for a write that a primary accepts, and CLOSED for a close of the observer itself.
+   * OK for a write that a primary accepts, and CLOSED for a close of the observer itself. It is
+   * DROPPED for a drop by the program of the observer.
    */
   readonly errno: string;
   readonly from: Reference;
@@ -439,6 +460,13 @@ type Prediction = {
   readonly group?: GroupPrediction;
   readonly request?: RequestPrediction;
   readonly driver?: DriverBand;
+  readonly reconnects?: Reconnects;
+  /**
+   * This name tells how the program drops the connection, for the errno DROPPED. A close frame
+   * leaves its socket ESTABLISHED below (`frame`). A close of its socket takes it out of
+   * ESTABLISHED at once (`close`).
+   */
+  readonly drop?: "frame" | "close";
 } | {
   readonly detects: false;
   readonly window: Quantity;
@@ -455,6 +483,12 @@ type Prediction = {
   readonly window: Quantity;
   readonly why: string;
   readonly from?: Reference;
+  /**
+   * This choice ends the measured time at the close of the socket by the observer itself, the
+   * change from ESTABLISHED to FIN_WAIT1. Without it, the time ends at a report of the kernel or
+   * at the last data. Then an idle pool that closes its own socket gives no end.
+   */
+  readonly ends?: "own-close";
   readonly group?: GroupPrediction;
   readonly request?: RequestPrediction;
 };
@@ -499,13 +533,64 @@ interface Target {
    * stamps it. SIGCONT then thaws it. Without it, the freeze is the cgroup freezer, with no stamps.
    */
   readonly program?: string;
+  /**
+   * This name is an image of the harness that the target operates in place of `image`. An
+   * example is the image of the procedures client, which also holds the script of its server.
+   * `image` is then the stock base of that image, for the record.
+   */
+  readonly harness?: HarnessImage;
 }
+/**
+ * A harness image is an image of the harness that a target can operate. `procedures` is the
+ * image of `@procedures/client` on Node: its WebSocket client and its server script.
+ */
+type HarnessImage = "procedures";
 /**
  * A client is what the runs of a cell use: the TCP client of the harness, or a driver on its
  * socket. The HTTP clients are the `http.Transport` of Go (`go-http`), the `http.Agent` of Node
- * (`http-agent`) and an HTTP/1.1 `Client` of undici (`undici`). The MongoDB driver is `mongodb`.
+ * (`http-agent`) and an HTTP/1.1 `Client` of undici (`undici`). The MongoDB driver is `mongodb`, and the WebSocket client of
+ * `@procedures/client` is `procedures-ws`.
+ */
+type Client = "tcp" | "ioredis" | "go-http" | "http-agent" | "undici" | "mongodb" | "procedures-ws";
+/**
+ * A WebSocket probe is the workload and the settings of the WebSocket client of
+ * `@procedures/client` (`procedures-ws`). The cell gives each setting, thus the client takes
+ * nothing from a default.
  */
-type Client = "tcp" | "ioredis" | "go-http" | "http-agent" | "undici" | "mongodb";
+interface WsProbe {
+  /**
+   * This name is the class of the sockets of the client: the global `WebSocket` of Node (`global`),
+   * or the `WebSocket` of the `ws` package (`ws`, the option `webSocket`).
+   */
+  readonly socket: "global" | "ws";
+  /**
+   * These are the settings of the heartbeat: a ping each interval, the wait for its pong, and the
+   * misses in sequence that drop the connection. Null turns the heartbeat off.
+   */
+  readonly heartbeat: {
+    readonly interval: Quantity;
+    readonly timeout: Quantity;
+    readonly misses: number;
+  } | null;
+  /**
+   * These are the settings of the reconnect: the first delay, the multiplier of each next delay,
+   * the longest delay and the most tries. `jitter` is the random part of each delay, from 0 to 1.
+   * Null turns the reconnect off.
+   */
+  readonly reconnect: {
+    readonly initialDelay: Quantity;
+    readonly multiplier: number;
+    readonly maxDelay: Quantity;
+    readonly attempts: number;
+    readonly jitter: number;
+  } | null;
+  /** This time is how long a call waits for its first frame. */
+  readonly requestTimeout: Quantity;
+  /** This time is how long a call waits for a connection. */
+  readonly connectionTimeout: Quantity;
+  /** This time is how long the server holds the reply to the call in flight. Without it, the server replies at once. */
+  readonly hold?: Quantity;
+}
 /**
  * An HTTP probe is the workload of an HTTP client. The first request is a GET of the root. The
  * second request follows the mode of the probe: after the signal (`inflight`), or after a gap
@@ -591,6 +676,8 @@ interface Cell {
     };
     /** This workload is the workload of an HTTP client (`Cell.client` is an HTTP client). The client then ignores `keepalive` and `userTimeout`. */
     readonly http?: HttpProbe;
+    /** These are the settings of the WebSocket client (`Cell.client` is `procedures-ws`). The client then ignores `keepalive` and `userTimeout`. */
+    readonly ws?: WsProbe;
   };
   readonly fault: Fault;
   /** This time is the length of a stall, from the send of the request. */
@@ -676,6 +763,12 @@ interface Cell {
    * many environments the machine holds before its noise increases.
    */
   readonly concurrency?: number;
+  /**
+   * This time is δ_abs of the environment of the cell: the noise of the pipeline. Each band includes
+   * it, and a timer must be at least that late to make a run void. Stage 0 measures it in each
+   * environment. Without it, the cell gets `NOISE`, the δ_abs of the first environment (4 ms).
+   */
+  readonly noise?: Quantity;
 }
 /** A hop place is a hop of a topology, with the side that the rules of a fault face. */
 interface HopPlace {
@@ -736,6 +829,11 @@ interface Run {
    * the run void. Null also makes the run void: no clock line came after the run to tell the count.
    */
   readonly lost: number | null;
+  /**
+   * This rate is the HZ of the kernel of the environment, from its manifest. With it, the rule of a
+   * late timer is in time. A run without it keeps the rule in jiffies, as the runs before 0.9.0.
+   */
+  readonly hz?: number;
   /** These are the addresses and the namespaces of a topology, which the measurement of a run reads. */
   readonly topology?: RunTopology;
   /** These are the lines of the routers of a topology from the start to the end of the run. */
@@ -797,6 +895,77 @@ interface RunTopology {
   }>>;
 }
 //#endregion
+//#region src/application.d.ts
+/**
+ * A program event is a line of the program of the observer, with the kernel's stamp of its act.
+ * The act is a write, or a change of state of the socket. Without a record of the kernel in δ_abs
+ * after the line, the instant is the stamp of the line.
+ */
+interface ProgramEvent {
+  readonly line: ProbeEvent;
+  readonly at: number;
+  /** This flag tells that the kernel stamped the event. */
+  readonly kernel: boolean;
+  /** This name is the kind of the record of the kernel: a write, or a change of state. It is null without a record. */
+  readonly record: "write" | "state" | null;
+}
+/** This function gives the lines of the program of the observer: the client, or the target for the server. A proxy has no lines. */
+export declare const programLines: (cell: Cell, run: Run) => readonly ProbeEvent[];
+/**
+ * This function gives the time from a line of the program to the close of the socket by its own
+ * side. That close is its change from ESTABLISHED to FIN_WAIT1, or to CLOSE (an abort). A FIN of
+ * the peer (CLOSE_WAIT) is not such a close. It is null when the socket did not close itself.
+ */
+export declare const leftAfter: (socket: readonly KernelRecord[], line: ProbeEvent) => Quantity | null;
+/**
+ * This function tells how a drop differs from the way that its prediction gives, or gives null.
+ * A close closes the socket in δ_abs. A close frame is a write, and the program does not close
+ * the socket at the drop. A peer that answers the close frame can close it, and that is no finding.
+ */
+export declare const dropFinding: (expected: "frame" | "close", drop: ProgramEvent, left: Quantity | null, delta?: Quantity) => string | null;
+/** This function gives each ping of the program on the connection of the run, with the kernel's stamp of its write. */
+export declare const pingsOf: (lines: readonly ProbeEvent[], socket: readonly KernelRecord[], local: unknown, delta?: Quantity) => ProgramEvent[];
+/**
+ * This function gives the first ping that got no answer: no segment with data reached the socket
+ * after its write, up to `until`. Its write starts the wait of the heartbeat that the fault met.
+ */
+export declare const firstUnanswered: (pings: readonly ProgramEvent[], socket: readonly KernelRecord[], until?: number) => ProgramEvent | undefined;
+/**
+ * This function gives the first drop of the connection by its program. The kernel stamps it as the
+ * write of a close frame. Or it stamps the change of the socket out of ESTABLISHED: a close or an
+ * abort.
+ */
+export declare const dropOf: (lines: readonly ProbeEvent[], socket: readonly KernelRecord[], local: unknown, delta?: Quantity) => ProgramEvent | undefined;
+/** A reconnect reading is what the client did after the detection, from its lines and from the records of the kernel. */
+interface ReconnectReading {
+  /** These are the waits of the program before each try: from a `reconnecting` line to the next `dial` line. */
+  readonly waits: readonly Quantity[];
+  /** This time is from the detection to the first SYN of a new socket. The kernel stamps the change to SYN_SENT at connect(2). */
+  readonly syn: Quantity | null;
+  /** This time is from the detection to the first new socket of the client that became ESTABLISHED. */
+  readonly established: Quantity | null;
+  /** These are the times from each `dial` line after the detection to the SYN of its socket: the work of the program before connect(2). */
+  readonly connects: readonly Quantity[];
+}
+/**
+ * This function reads the reconnect of the client after the detection at `detected`. The socket of
+ * the run has the cookie `cookie`. The lines of the program count from `from`: the drop line of a
+ * drop, which comes before the kernel's stamp of it.
+ */
+export declare const reconnectOf: (run: Run, detected: number, cookie: number | undefined, from?: number) => ReconnectReading;
+/** This function tells how the waits of a reconnect differ from their bands, each widened by δ_abs, or gives null. */
+export declare const reconnectFinding: (expected: Reconnects, read: ReconnectReading, delta?: Quantity) => string | null;
+/**
+ * A call reading is what the call in flight of a WebSocket client met: its error, a reply, or
+ * nothing in the run. The time counts from the detection to its end.
+ */
+interface CallReading {
+  readonly outcome: string;
+  readonly after: Quantity | null;
+}
+/** This function reads the call that the client sent into the fault: its `call-error` or `data` line after the request. */
+export declare const callOf: (events: readonly ProbeEvent[], request: ProbeEvent, detected: number | undefined) => CallReading;
+//#endregion
 //#region src/driver.d.ts
 /**
  * This module measures a run of a cell whose client is a driver with more than one socket, for
@@ -974,7 +1143,8 @@ export declare const measureGroup: (cell: Cell, run: Run) => GroupReading;
 //#region src/observe.d.ts
 /**
  * An outcome tells how the wait of a run ended. The socket failed with an errno, or it read the
- * end of the file. Or nothing came, or the reply came back after a stall.
+ * end of the file. Or nothing came, or the reply came back after a stall. The observer can also
+ * close the connection itself (`closed`), or its program can drop it on its own timer (`dropped`).
  */
 type Outcome = {
   readonly kind: "error";
@@ -987,6 +1157,8 @@ type Outcome = {
   readonly kind: "answered";
 } | {
   readonly kind: "closed";
+} | {
+  readonly kind: "dropped";
 };
 /**
  * A request reading is what the second request of an HTTP client met, from the lines of the
@@ -1054,6 +1226,16 @@ interface Measured {
   readonly driver?: {
     readonly steps: readonly DriverStepTime[];
   };
+  /** This record is what the client did after the detection, for a prediction that says when it reconnects. */
+  readonly reconnect?: ReconnectReading;
+  /** This record is what the call in flight of a WebSocket client met. */
+  readonly call?: CallReading;
+  /**
+   * This time is from the drop line of the program to the close of its socket by its own side: a
+   * FIN or an abort. A prediction that says how the program drops (`drop`) gets it. It is null
+   * when the socket did not close itself in the run.
+   */
+  readonly left?: Quantity | null;
   /** This text is the reason that the run cannot count, or null. A void run is not a pass and not a finding. */
   readonly void: string | null;
   /** This text tells what in the run disagrees with the prediction, other than its timing, or null. */
@@ -1156,6 +1338,23 @@ export declare const requestWords: (outcome: {
  * the harness, the kernel or Docker.
  */
 export declare const NOISE: Quantity;
+/**
+ * This time is how late the injector can commit or lift a fault before the run is void. Stage 0
+ * measures the commit apart from δ_abs, and the commit is longer. On a hosted runner it took up to
+ * 2.75 ms at 4 environments, where δ_abs is 1 ms. Thus a cell's δ_abs does not change it.
+ */
+export declare const INJECT: Quantity;
+/** This function gives the δ_abs of a cell: its own, or `NOISE` for a cell that gives none. */
+export declare const noiseOf: (cell: {
+  readonly noise?: Quantity | undefined;
+}) => Quantity;
+/**
+ * This function tells if a timer firing came late enough to void a run. A firing two jiffies or
+ * more behind its wheel slot came late because the host stopped a vCPU. With the HZ of the run,
+ * the rule is in time: the delay must also reach δ_abs, because each band already includes that
+ * noise. At HZ 250 two jiffies are 8 ms, more than a δ_abs of 4 ms. At HZ 1000 they are 2 ms.
+ */
+export declare const delayVoids: (behind: number, hz: number | undefined, noise: Quantity) => boolean;
 /** This fraction, δ_rel, is how far the median of a cell can be outside the prediction and stay equivalent. */
 export declare const RELATIVE = .05;
 /** These are the run counts of a cell: 11, then 22 and 44 while its median crosses the edge of the band. */
@@ -1209,8 +1408,33 @@ interface Verdict {
    * one is a finding of its run.
    */
   readonly requests?: Readonly<Record<string, number>>;
+  /** This record is what the client did after the detection over the counted runs, for a prediction that says when it reconnects. */
+  readonly reconnect?: ReconnectVerdict;
+  /** These are the outcomes of the call in flight of a WebSocket client over the counted runs, each with its count. */
+  readonly calls?: Readonly<Record<string, number>>;
+  /**
+   * This record is the time from a drop of the program to the close of its socket by its own side,
+   * over the counted runs. It also counts the runs where the socket did not close itself.
+   */
+  readonly left?: {
+    readonly spread: Spread | null;
+    readonly stayed: number;
+  };
   readonly why: string;
 }
+/**
+ * A reconnect verdict sums the reconnect of the client over the counted runs. Each wait before a
+ * try gets its spread. A wait outside its band is a finding of its run, thus it is in `outside`.
+ * - `syn` is the time from the detection to the first SYN of a new socket.
+ * - `established` is the time from the detection to the first new socket that became ESTABLISHED.
+ * - `connect` is the time from a `dial` line to the SYN of its socket: the work of the program.
+ */
+interface ReconnectVerdict {
+  readonly waits: ReadonlyArray<Spread | null>;
+  readonly syn: Spread | null;
+  readonly established: Spread | null;
+  readonly connect: Spread | null;
+}
 /**
  * A stage verdict is the test of one stage of the timeline of a group against its band. The band
  * is [T_lo − δ_abs, T_hi + the error of the stamps + δ_abs]. A run that did not announce the stage
@@ -1282,9 +1506,9 @@ export declare const medianInterval: (sorted: readonly number[]) => [number, num
  * finding. The distribution test (a DKW band) needs a predicted distribution from a simulator,
  * which does not exist at this time.
  */
-export declare const verdictOf: (prediction: Prediction, runs: readonly Measured[]) => Verdict;
+export declare const verdictOf: (prediction: Prediction, runs: readonly Measured[], noise?: Quantity) => Verdict;
 /** This function sums what the group of a group cell did in its counted runs, and tests each stage that has a band. */
-export declare const groupVerdictOf: (prediction: Prediction, counted: readonly Measured[]) => GroupVerdict;
+export declare const groupVerdictOf: (prediction: Prediction, counted: readonly Measured[], delta?: Quantity) => GroupVerdict;
 //#endregion
 //#region src/lab.d.ts
 /** This function gives the image of the harness. */
@@ -1299,6 +1523,10 @@ export declare const goHttpImage: () => Promise<string>;
 export declare const nodeHttpImage: () => Promise<string>;
 /** This function gives the image of the MongoDB client. */
 export declare const mongoClientImage: () => Promise<string>;
+/** This function gives the image of `@procedures/client`: its WebSocket client and its server. */
+export declare const proceduresImage: () => Promise<string>;
+/** This function gives the image that a target operates: its stock image, or an image of the harness (`Target.harness`). */
+export declare const targetImageOf: (target: Target) => Promise<string>;
 /**
  * This time is the interval of the listing of a router, when its hop does not set one. A listing
  * evicts each expired flow that it walks past, and the DESTROY event of that flow follows.
@@ -1351,11 +1579,19 @@ interface Manifest {
   /** This text is the architecture of the machine, as uname -m gives it: x86_64. */
   readonly architecture: string;
   readonly hz: string | null;
+  /**
+   * This text tells where the HZ came from: the configuration of the kernel, or a measurement. A
+   * host without /proc/config.gz (a hosted runner) gets a measurement of jiffies against the clock.
+   */
+  readonly hzFrom?: "config" | "measured";
   readonly boot_id: string;
   readonly docker: string;
   /** This record is the target, and its settings as it read them back. */
   readonly target: {
     readonly image: string;
+    /** This name is the image of the harness that the target operates, with its stock base (`Target.harness`). */
+    readonly harness?: string;
+    readonly base?: string;
     readonly id: string;
     readonly settings: Readonly<Record<string, string>>;
     /** These are the sysctls of its namespace, as read back after the harness set them. */
@@ -1571,6 +1807,18 @@ export declare const routesOf: (hops: readonly Pick<Hop, "kind">[]) => Array<{
  * is not a router. Element 0 is the client, element k is hop k - 1, and the last element is the target.
  */
 export declare const endpointAfter: (hops: readonly Pick<Hop, "kind">[], element: number) => number;
+/**
+ * This function gives HZ from two readings of /proc/timer_list, one second apart: its `now at`
+ * time in ns and its jiffies. It gives the nearest rate that Linux offers, or null when the
+ * measurement is more than 5 % from each of them.
+ */
+export declare const hzOf: (first: {
+  now: number;
+  jiffies: number;
+}, second: {
+  now: number;
+  jiffies: number;
+}) => string | null;
 /** This function gives the device of a route from its line in `ip -o route get`: "10.0.0.2 dev eth0 src …". */
 export declare const deviceOf: (route: string) => string;
 /**
@@ -1929,5 +2177,5 @@ export declare const spread: (runs: number, environments: number) => number[];
  */
 export declare const runCell: (cell: Cell, say?: (line: string) => void) => Promise<CellResult>;
 //#endregion
-export type { Agent, Cell, CellResult, Client, ClientImage, Crossings, Cut, Decision, DriverBand, DriverSocket, DriverStep, DriverStepTime, ErrorOnTheWay, Fault, FaultPlace, FaultSide, Group, GroupClient, GroupLab, GroupMeasured, GroupNetworkFault, GroupPlace, GroupPrediction, GroupReading, GroupRole, GroupVerdict, GroupView, Hop, HopPlace, HttpProbe, Interval, KernelRecord, Lab, LineMatch, Manifest, ManifestHop, Measured, MemberLine, MiddleboxEvent, MiddleboxExpectation, MiddleboxForm, NetworkFault, Observer, Outcome, Prediction, Probe, ProbeEvent, ProxyHop, ProxyTime, Recovers, Reference, Reordering, ReplicaSet, ReplicaSetLab, ReplicaSetStage, Replies, RequestPrediction, RequestReading, RouterHop, RttRange, Run, RunGroup, RunPart, RunTopology, SentinelEvent, SentinelStage, SocketPlace, Spread, Stage, StageBand, StageVerdict, Stamp, Status, Target, Topology, Toxic, Verdict, Writes };
+export type { Agent, CallReading, Cell, CellResult, Client, ClientImage, Crossings, Cut, Decision, DriverBand, DriverSocket, DriverStep, DriverStepTime, ErrorOnTheWay, Fault, FaultPlace, FaultSide, Group, GroupClient, GroupLab, GroupMeasured, GroupNetworkFault, GroupPlace, GroupPrediction, GroupReading, GroupRole, GroupVerdict, GroupView, HarnessImage, Hop, HopPlace, HttpProbe, Interval, KernelRecord, Lab, LineMatch, Manifest, ManifestHop, Measured, MemberLine, MiddleboxEvent, MiddleboxExpectation, MiddleboxForm, NetworkFault, Observer, Outcome, Prediction, Probe, ProbeEvent, ProgramEvent, ProxyHop, ProxyTime, ReconnectReading, ReconnectVerdict, Reconnects, Recovers, Reference, Reordering, ReplicaSet, ReplicaSetLab, ReplicaSetStage, Replies, RequestPrediction, RequestReading, RouterHop, RttRange, Run, RunGroup, RunPart, RunTopology, SentinelEvent, SentinelStage, SocketPlace, Spread, Stage, StageBand, StageVerdict, Stamp, Status, Target, Topology, Toxic, Verdict, Writes, WsProbe };
 //# sourceMappingURL=index.d.ts.map