@mark1russell7/harness 0.3.0 → 0.4.0

package/README.md

@@ -10,9 +10,10 @@ A **cell** (`cell.ts`) is all that one measurement needs, as data:
 
 - **The topology.** A stock target image, pinned by digest, and the hops between the client and the target (`Cell.topology`). Without hops, the client and the target are a direct pair.
 - **The settings.** The sysctls of the client namespace, and the socket options of the client.
-- **The workload.** A connection that stays idle, or a connection with a request in flight.
-- **The fault.** A black hole, a reset, a frozen process, or a stall. In a topology, also a router that forgets a flow, a proxy that closes an idle connection, or a toxic of Toxiproxy.
+- **The workload.** A connection that stays idle, a connection with a request in flight, or a stream of requests. A stream of `GET` requests for a long value keeps many segments in flight.
+- **The fault.** A black hole, a reset, a frozen process, or a stall. In a topology, also a fault at a router or a proxy, or a router that forgets a flow. Also a proxy that closes an idle connection, or a toxic of Toxiproxy.
 - **The prediction.** An interval [T_lo, T_hi] after the event that arms the timer, or silence in a window. After a stall, the prediction also tells when the connection recovers. A silent prediction can also tell when the reply comes back (`replies`).
+- **The assumptions of the band.** A band can name the range of first round trips that it assumes (`rtt`), and the proxy times that it includes (`through`).
 
 ## The environment
 
@@ -31,7 +32,7 @@ Each command is an argv array.
 
 A topology (`Topology`) is the chain of hops from the client to the target, as data. Each hop is on two networks, one for each side. Network j joins element j and element j + 1: the client is element 0, and the target is the last element.
 
-- **A router** (`RouterHop`) is the router image of the harness. It forwards, and it translates the addresses of the side of the client (NAT) with conntrack. Its idle timeout is `nf_conntrack_tcp_timeout_established`, in whole seconds. The router is privileged, thus it sets the sysctls of its own namespace. Its form (`MiddleboxForm`) tells what a late packet meets after the router forgets the flow:
+- **A router** (`RouterHop`) is the router image of the harness. It forwards, and it translates the addresses of the side of the client (NAT) with conntrack. Its idle timeout is `nf_conntrack_tcp_timeout_established`, in whole seconds. The router is privileged, thus it sets the sysctls of its own namespace, and the `sysctls` of its hop. An injector sidecar shares its namespace. Its form (`MiddleboxForm`) tells what a late packet meets after the router forgets the flow:
 
   | Form | At the expiry | A late packet from inside | A late packet from outside |
   | --- | --- | --- | --- |
@@ -63,7 +64,8 @@ The harness builds three images. The tag of each image comes from the content of
 - **The harness image** (`image/Dockerfile`) has two stages, each a digest-pinned Alpine. `Cargo.lock` pins the crates. The image holds:
   - the kernel program of the prober, compiled against the BTF of the kernel that builds it, and its Rust collector.
   - `client`, the TCP client of a cell. Rust's `std::net` sets no socket option by itself, thus the measurement gets the timers of the namespace. The client also samples `TCP_INFO` each 2 ms, a second vantage on its socket. It waits in `poll()`, which does not lock the socket. A keepalive timer that finds its socket held tries again HZ/20 later.
-  - `inject`, which applies a fault in one `iptables-restore` commit, with monotonic stamps around the commit. Thus the start of a fault is known in the VM. A `docker exec` from Windows comes 190 ms late at the median.
+  - In a stream, a reply of the `client` is whole at its CRLF, or after the length of a bulk string. The client then writes a `replied` line.
+  - `inject`, which applies a fault in one `iptables-restore` commit, with monotonic stamps around the commit. Thus the start of a fault is known in the VM. A `docker exec` from Windows comes 190 ms late at the median. With `--via DEV`, the rules go in the FORWARD chain, on the device toward one side.
 - **The Node client image** (`image/node`) is a driver client: ioredis 6.0.0 on Node 24.21.0. The digest of the image and an npm lockfile pin them. `client-ioredis.mjs` prints the lines of the TCP client on the same clock, and it adds the status changes of ioredis as `status` lines.
 - **The router image** (`image/router`) is a digest-pinned Alpine with conntrack-tools, nftables, iproute2 and Python. `mbx.sh` sets the form of the router in one nftables table, `ctwatch.py` writes the conntrack events, and `resetter.py` sends the resets of the `reset-both` form. Each run records its packages in the manifest.
 
@@ -81,15 +83,22 @@ ioredis sets the keepalive of its socket itself, after it connects. Thus a cell
 - **A simplex partition** (`simplex`, `Cell.drops`) drops the segments to the server, or the segments to the client, but not both.
 - **A FIN** (`fin`) stops the process of the peer with SIGKILL while its kernel stays up. For the client, the target starts its server as the child of a shell (`Target.command`), and `Target.kill` stops it. For the server, the harness stops the container of the client.
 - **An MTU black hole** (`mtu-blackhole`, `Cell.mtu`) drops each segment longer than the length, in the namespace that receives it.
-- **netem** (`netem`, `Cell.netem`) is tc netem on the egress of the target: loss, duplication, reordering or corruption. The outcome is a distribution.
+- **netem** (`netem`, `Cell.netem`) is tc netem on the egress of the target: loss, duplication, reordering or corruption. The outcome is a distribution. At a hop, netem is on the egress of the device of the hop toward one side.
 - **A latency** (`Cell.latency`) is tc netem delay on the egress of the target before the client connects. Thus the round trip and the RTO include it.
-- **A fault at a hop** (`Cell.faultAt`) puts the rules of a fault in the namespace of a proxy, toward the client or toward the target. The rules act on the sockets of the proxy. A router forwards past INPUT and OUTPUT, thus the harness refuses a fault at a router.
+- **A fault at a proxy** (`Cell.faultAt`) puts the rules of a fault in the namespace of the proxy, toward the client or toward the target. The rules act on the sockets of the proxy.
+- **A fault at a router** (`Cell.faultAt`) puts FORWARD rules in the namespace of the router, through its injector sidecar. A router forwards past INPUT and OUTPUT. Each rule faces one side by the device toward it, and by the address of the peer on that side:
+  - A black hole drops what comes in from that side (`-i DEV -s PEER`) and what goes out to it (`-o DEV -d PEER`). A simplex partition drops one of the two.
+  - A reset answers each segment from that side with a reset from the router. The reset has the address that the segment went to.
+  - An ICMP unreachable comes from the address of the router on that device, not from the peer. Its rate limit is in the `sysctls` of the router.
+  - The rules of the fault are in the filter table of iptables. The form of the router is a table of nftables at the same priority. Thus the two can both drop a packet, and an accept of one does not pass a drop of the other.
+  - Conntrack sees each packet before a FORWARD rule drops it. Thus a retransmission into a black hole at the router refreshes the flow. The resets of the `reset-both` form leave from the stack of the router, past its own FORWARD rules.
+- **No fault** (`none`) injects nothing. A calibration cell measures the path itself (`Cell.calibrates`).
 - **A forgotten flow** (`middlebox-forget`) injects nothing. The router of the topology forgets an idle flow at its own timeout. `Cell.idleBefore` holds an in-flight request until after the forget.
 - **An idle proxy** (`proxy-idle`) injects nothing. The proxy closes an idle connection at its own timeout.
 - **A toxic** (`toxic`, `Cell.toxic`) is one toxic of Toxiproxy, added through its HTTP API by `inject http`, with monotonic stamps around the request. Its name is `fault`, and the harness removes it after the run. A cell whose fault is another one adds its toxic first: a slow close, then the close of the server.
 - **A close of the server** (`server-close`) makes the server close the connection of each client, with `Target.close`: `CLIENT KILL` for Redis.
 
-A length match and netem act on GSO super-packets of up to 64 KB, not on segments, while TSO, GSO and GRO are on. Thus the cells of `mtu-blackhole` and `netem` turn them off on the two ends (`lab.offload`).
+A length match and netem act on GSO super-packets of up to 64 KB, not on segments, while TSO, GSO and GRO are on. Thus the cells of `mtu-blackhole` and `netem` turn them off on the two ends (`lab.offload`). They also turn them off on the two devices of a hop where the fault is.
 
 Each run starts from kernels that keep nothing from the run before it (`lab.forget()`). When a connection closes, each kernel keeps TCP metrics for the address of its peer. These metrics set the first round trip of the next connection, and thus its RTO. The harness removes the `tcp_metrics` of the two namespaces before each run.
 
@@ -101,13 +110,17 @@ Each run starts from kernels that keep nothing from the run before it (`lab.forg
 - **The time.** It times the detection from the reference event of the prediction to the error report of the kernel. eBPF stamps the two events. For keepalive, the reference is the last received segment. For the give-up, it is the first retransmission of the RTO timer, not the tail loss probe one RTO before it. For a reset, it is the send of the client, which no kernel record marks. The client stamps the send before its write.
 - **The other views.** It also gives the same interval from the `TCP_INFO` samples of the client, and the delay before the client saw the error. It also gives the steps of the kernel: each keepalive probe, or each expiry of the RTO timer.
 - **Void runs.** A run is void when the timer that drives the measurement was two jiffies or more behind its wheel slot in the timed interval. The wheel does a slot in the tick when it is due. Thus such a firing measured the host, which stopped a vCPU, not the kernel. A keepalive timer that found its socket held also makes the run void. A run whose fault came too late is void too.
+- **The first round trip.** The measurement gives the first round trip of the socket of the observer (`rtt`). It is the smoothed round trip in the first record of the socket: the sample of the handshake. A run whose first round trip is above the range of its prediction is void, with the reason `rtt-over-assumption` and the value. Such a run measured another path than the band assumes. For example, a round trip of 4.826 ms makes an RTO of 52 jiffies, and a ladder that assumes 1 ms has 51.
 - **Findings.** A finding is a wrong errno, a detector that did not fire, or a detection where the prediction is silence. A router that forgot the flow in the window of a silent prediction is also a finding.
 - **A reply.** A silent prediction with `replies` times the send of the request to the first segment with data that comes back. A proxy can acknowledge the request first, with no data.
 - **A toxic.** A toxic counts from the start of its commit (`fault`), as the injector stamps it.
 - **A topology.** A run in a topology also gives the bytes that the client read after its request, and when a router forgot its flow (`forgot`).
 - **Recovery.** After a stall (`prediction.recovers`), it times the send to the first segment back after the stall stopped. That segment is the answer of the server to the retransmission that got through. A run whose stall stopped outside the window of its prediction is void. A run where nothing came back is a finding.
 - **An end of file.** A FIN counts from the change of state of the socket of the peer when it closes (`peer-closed`). It ends at the change of the socket of the observer to CLOSE_WAIT. The kernel stamps the two changes.
-- **A stream.** In a stream (`probe.mode` is `stream`), the client sends `count` requests. The run keeps the time from each send to the first data after it (`samples`).
+- **A stream.** In a stream (`probe.mode` is `stream`), the client sends `count` requests. The run keeps the time from each send to the last byte of its reply (`samples`). A record without `replied` lines gives the time to the first data.
+- **Reordering.** A measurement-only run also gives what reordering, duplicates and losses did (`reordering`). At the client, it counts the arrivals beyond a hole and the arrivals of data that it had already, against `rcv_nxt`. From the sender, it gives the most SACKed and lost segments, the DSACKs, the fast retransmissions, and the reordering degree and count of the kernel.
+- **An arrival is not always one segment.** The segments that arrive while the owner holds the socket wait in its backlog, and the kernel joins the contiguous ones. Thus the kernel counts out-of-order segments (`rcv_ooopack`), and the prober counts arrivals.
+- **A proxy's own time.** A calibration cell (`Cell.calibrates`) gives the time of its proxy for each request and its reply (`crossings`). Each time goes from an arrival on one socket of the proxy to its next write on the other socket (a `tx` record).
 
 ## The verdict
 
@@ -117,8 +130,11 @@ Each run starts from kernels that keep nothing from the run before it (`lab.forg
 - **Median equivalence.** The order-statistic interval of the median must be in the band, made 5 % wider.
 - **Extension.** A median across the edge of the band extends the cell from 11 runs to 22, then 44. Then the verdict is inconclusive.
 - **Recovery.** The recovery after a stall gets the same test as a detection, by its band. A detection is then a finding.
-- **Measurements.** A cell whose outcome is a distribution (`detects: null`) has no band. Its decision is `measured`, and the verdict gives the median and the slowest reply time. A measurement-only cell that names a reference (`from`) gives the count of each outcome. It also gives the median and the range of the time from the reference. The time ends at the end of the connection, or at the last data.
+- **Measurements.** A cell whose outcome is a distribution (`detects: null`) has no band. Its decision is `measured`, and the verdict gives the median, the 99th percentile and the slowest reply time. It adds the sum of what reordering did, when anything came out of order, twice or SACKed.
+- **A reference.** A measurement-only cell that names a reference (`from`) gives the count of each outcome. It also gives the median and the range of the time from the reference. The time ends at the end of the connection, or at the last data.
+- **A calibration.** A calibration cell gives the spread of the time of its proxy in each direction (`proxy`). The spread is the count, the median, the 99th and 99.9th percentiles by nearest rank, and the maximum.
 - **A reply.** A reply in its band (`replies`) gets the same test as a detection.
+- **The time of a proxy.** A band through a proxy includes an upper quantile of the time of the proxy for each crossing (`through`). The author of the cell adds it from a calibration cell, and the verdict tests the band as it is.
 
 The distribution test needs a simulator, which does not exist at this time.
 
@@ -148,6 +164,6 @@ A run needs Docker, and privileged containers for the prober and the sidecars. T
 
 `pnpm --filter @mark1russell7/harness test` starts the tests. They measure test records with the shape of real kernel records (keepalive, and the tail loss probe before the retransmissions of the RTO timer). They also test the containment, the median equivalence, the extension and the silence of the verdict.
 
-The cells themselves need Docker and privileged containers. CI does not start them.
+The cells themselves need Docker and privileged containers. CI does not start them. `HARNESS_LAB=1` starts the lab tests on Docker: a client image, an agent, and a reset and netem at a router.
 
 `pnpm --filter @mark1russell7/harness build` writes the package for npm into `build/`.

package/build/index.d.ts

@@ -29,8 +29,14 @@ import "testcontainers";
  * - `toxic` adds a toxic of Toxiproxy (`Cell.toxic`) through its HTTP API.
  * - `server-close` makes the server close the connection of its peer, with the command of the
  *   target (`Target.close`): `CLIENT KILL` in Redis.
+ * - `none` injects nothing. A calibration cell measures the path itself: for example the time of a
+ *   proxy between its two sockets (`Cell.calibrates`).
+ *
+ * A fault at a router of the topology (`Cell.faultAt`) acts on what the router forwards toward
+ * one side. It is a black hole, a reset or an ICMP unreachable. It can also be a simplex drop, an
+ * MTU drop or netem.
  */
-type Fault = "blackhole" | "freeze-process" | "rst" | "stall" | "fin" | "unreachable" | "simplex" | "mtu-blackhole" | "netem" | "middlebox-forget" | "proxy-idle" | "toxic" | "server-close";
+type Fault = "blackhole" | "freeze-process" | "rst" | "stall" | "fin" | "unreachable" | "simplex" | "mtu-blackhole" | "netem" | "middlebox-forget" | "proxy-idle" | "toxic" | "server-close" | "none";
 /**
  * A middlebox form tells what a router does with a flow that it forgot, when a late packet comes:
  * - `reset-both` sends a reset to the two ends when it forgets the flow.
@@ -58,6 +64,11 @@ interface RouterHop {
    * interval is 0.1 s. The DESTROY event of an expired flow waits for that listing.
    */
   readonly pump?: Quantity;
+  /**
+   * These are the sysctls of the namespace of the router, which the router sets itself: it is
+   * privileged. An example is `net.ipv4.icmp_ratelimit` for the ICMP errors of a fault at the router.
+   */
+  readonly sysctls?: Readonly<Record<string, string>>;
 }
 /**
  * A proxy is a hop that ends each TCP connection from the side of the client and opens its own to
@@ -143,6 +154,28 @@ interface Replies {
   readonly low: Quantity;
   readonly high: Quantity;
 }
+/**
+ * An RTT range is the range of the first round trip of the observer that a band assumes. The
+ * models read a round trip, for example 1 ms. The band at that value holds within the noise for
+ * each round trip in the range. A run whose first round trip is above the range is void.
+ */
+interface RttRange {
+  readonly low: Quantity;
+  readonly high: Quantity;
+}
+/**
+ * A proxy time is the time of one proxy in one direction that a band includes. It is an upper
+ * quantile of the times that a calibration cell of the proxy measured. Each time goes from an
+ * arrival on one socket of the proxy to its write on the other. `up` goes toward the target, and
+ * `down` toward the client.
+ */
+interface ProxyTime {
+  readonly hop: string;
+  readonly direction: "up" | "down";
+  /** This fraction is the quantile of the calibration, for example 0.99. */
+  readonly quantile: number;
+  readonly time: Quantity;
+}
 /**
  * A prediction is what a cell predicts. Either a detection occurs in an interval [T_lo, T_hi]
  * after the reference event, or nothing detects the fault in a window. Nothing detects a stall
@@ -150,6 +183,9 @@ interface Replies {
  * tells when. A silent prediction can also tell when the reply to the request comes back
  * (`replies`).
  *
+ * A band can name the first round trip that it assumes (`rtt`), and the proxy times that it
+ * includes (`through`).
+ *
  * A cell whose outcome is a distribution, for example under netem loss, has no band. Its
  * prediction is `detects: null` with the reason, and its runs are measurements only. A
  * measurement-only cell that names a reference (`from`) also records the time from it to the end
@@ -164,13 +200,17 @@ type Prediction = {
   readonly from: Reference;
   /** This time is the closed form, less the most that the timer can fire early. */
   readonly low: Quantity;
-  /** This time is the closed form, plus the most that the timer can fire late. */
+  /** This time is the closed form, plus the most that the timer can fire late, plus each proxy time. */
   readonly high: Quantity;
+  readonly rtt?: RttRange;
+  readonly through?: readonly ProxyTime[];
 } | {
   readonly detects: false;
   readonly window: Quantity;
   readonly recovers?: Recovers;
   readonly replies?: Replies;
+  readonly rtt?: RttRange;
+  readonly through?: readonly ProxyTime[];
 } | {
   readonly detects: null;
   readonly window: Quantity;
@@ -274,7 +314,10 @@ interface Cell {
   readonly drops?: "to-server" | "to-client";
   /** This length is the longest segment, in bytes, that an `mtu-blackhole` fault lets through to the observer. */
   readonly mtu?: number;
-  /** This specification is the tc netem of a `netem` fault, on the egress of the target: for example "loss 5%". */
+  /**
+   * This specification is the tc netem of a `netem` fault, for example "loss 5%". It is on the
+   * egress of the target, or at a hop (`faultAt`) on the egress of its device toward one side.
+   */
   readonly netem?: string;
   /**
    * This delay goes on the egress of the target before the client connects, with tc netem. Thus
@@ -292,7 +335,7 @@ interface Cell {
   readonly targetSysctls?: Readonly<Record<string, string>>;
   /** This topology puts hops between the client and the target. Without it, the client and the target are a direct pair. */
   readonly topology?: Topology;
-  /** This place is where the fault goes, when it is not the peer of the observer: for example one side of a proxy. */
+  /** This place is where the fault goes, when it is not the peer of the observer: for example one side of a hop. */
   readonly faultAt?: FaultPlace;
   /** This time is how long an in-flight client stays idle after the fault holds, before it sends its request. */
   readonly idleBefore?: Quantity;
@@ -300,6 +343,12 @@ interface Cell {
   readonly toxic?: Toxic;
   /** This image replaces the TCP client of the harness for each run. */
   readonly client?: ClientImage;
+  /**
+   * This name is the proxy hop whose own time a calibration cell measures. The time of a request
+   * (`up`) starts at its arrival on the socket of the proxy toward the client. It stops at the
+   * write on the socket toward the target. The time of its reply (`down`) goes the other way.
+   */
+  readonly calibrates?: string;
   /** This count is the number of runs of the cell (11), across new environments (two at least). */
   readonly runs: number;
   readonly environments: number;
@@ -313,7 +362,8 @@ interface Cell {
 /**
  * A fault place is the namespace that gets the rules of a fault. "target" and "client" are the two
  * ends. A hop place is a hop, with the side that the rules face. That side is the next hop toward
- * the client, or the next hop toward the target.
+ * the client, or the next hop toward the target. The rules of a proxy act on its sockets. The
+ * rules of a router act on what it forwards, by its device toward that side: the FORWARD chain.
  */
 type FaultPlace = "target" | "client" | {
   readonly hop: string;
@@ -435,11 +485,57 @@ interface Measured {
    * last segment that the client received before the fault. It is null when no router forgot the flow.
    */
   readonly forgot?: Quantity | null;
+  /**
+   * This time is the first round trip of the socket of the observer. It is the smoothed round trip
+   * in its first record, which is the sample of the handshake. It is null when no record has it.
+   */
+  readonly rtt?: Quantity | null;
+  /** This record tells what reordering did in a run of a measurement-only cell, from the records of its two ends. */
+  readonly reordering?: Reordering;
+  /** These are the times of the proxy of a calibration cell, one for each request and its reply. */
+  readonly crossings?: Crossings;
   /** This text is the reason that the run cannot count, or null. A void run is not a pass and not a finding. */
   readonly void: string | null;
   /** This text tells what in the run disagrees with the prediction, other than its timing, or null. */
   readonly finding: string | null;
 }
+/**
+ * A reordering record is what reordering, duplicates and losses did to the data of a run. The
+ * receiver is the client, and the sender is the socket of its peer.
+ * - `outOfOrder` counts the arrivals at the client beyond a hole, and `duplicates` counts the
+ *   arrivals whose data the client had. The prober reads the two from each arrival, against `rcv_nxt`.
+ * - An arrival can hold more than one segment. The kernel joins the segments that wait in the
+ *   backlog of a socket while its owner holds it.
+ * - `ooo` is the count of the kernel of the client in segments (`rcv_ooopack`).
+ * - `sacked` and `lost` are the most segments that the sender held as SACKed and as lost.
+ * - `dsacks` is the DSACK blocks that the sender received, and `fast` its fast retransmissions.
+ * - `degree` is the highest reordering degree of the sender (`tp->reordering`), and `seen` the
+ *   reorderings that it detected (`reord_seen`).
+ *
+ * The counts of the kernel are null for a record of the prober before version 0.2.
+ */
+interface Reordering {
+  readonly outOfOrder: number;
+  readonly duplicates: number;
+  readonly ooo: number | null;
+  readonly sacked: number | null;
+  readonly lost: number | null;
+  readonly dsacks: number | null;
+  readonly fast: number;
+  readonly degree: number | null;
+  readonly seen: number | null;
+}
+/**
+ * A crossings record is the time of a proxy for each request (`up`) and for each reply (`down`).
+ * The time of a request starts at its arrival on the socket toward the client. It stops at the
+ * write on the socket toward the target. The time of a reply goes the other way. A write is a
+ * `tx` record of the prober.
+ */
+interface Crossings {
+  readonly hop: string;
+  readonly up: readonly Quantity[];
+  readonly down: readonly Quantity[];
+}
 /** A socket place is a socket of a run: its namespace and its cookie. */
 interface SocketPlace {
   readonly netns: number;
@@ -452,6 +548,16 @@ interface SocketPlace {
  * - a socket of a proxy of the topology, toward the client or toward the target.
  */
 export declare const socketOf: (cell: Cell, run: Run) => SocketPlace | undefined;
+/** This function gives what reordering did in a run, from the records of the receiver and of the sender. */
+export declare const reorderingOf: (receiver: readonly KernelRecord[], sender: readonly KernelRecord[]) => Reordering;
+/**
+ * This function gives the times of a proxy in a run, for each request after the first. The time
+ * of a request starts at its arrival on the socket of the proxy toward the client. It stops at the
+ * next write on the socket toward the target. The time of its reply starts at its arrival on the
+ * socket toward the target, and it stops at the next write toward the client. A request without
+ * each record gives no time.
+ */
+export declare const crossingsOf: (run: Run, hop: string, toClient: SocketPlace | undefined, toTarget: SocketPlace | undefined) => Crossings;
 /**
  * This function reads one run against its cell, from the records of the kernel first.
  * - The socket of the run is the socket of its observer. It is the socket that the client connected
@@ -486,6 +592,14 @@ interface Interval {
   readonly low: Quantity;
   readonly high: Quantity;
 }
+/** A spread is the distribution of a set of times: its count, its median and its upper quantiles by nearest rank. */
+interface Spread {
+  readonly count: number;
+  readonly median: Quantity;
+  readonly p99: Quantity;
+  readonly p999: Quantity;
+  readonly max: Quantity;
+}
 /** A verdict is the judgement of a cell on its runs. */
 interface Verdict {
   readonly decision: Decision;
@@ -501,8 +615,20 @@ interface Verdict {
   readonly median: Quantity | null;
   /** This interval is the order-statistic interval of the median at 95 %, or the whole range for fewer than six runs. */
   readonly medianInterval: Interval | null;
+  /** This record is the sum of what reordering did in each counted run of a measurement-only cell. */
+  readonly reordering?: Reordering;
+  /** This record is the time of the proxy of a calibration cell, in each direction. */
+  readonly proxy?: {
+    readonly hop: string;
+    readonly up: Spread | null;
+    readonly down: Spread | null;
+  };
   readonly why: string;
 }
+/** This function gives the value at quantile p of sorted values, by nearest rank: x_(⌈p·n⌉). */
+export declare const quantileOf: (sorted: readonly number[], p: number) => number;
+/** This function gives the spread of a set of times, or null for none. */
+export declare const spreadOf: (times: readonly Quantity[]) => Spread | null;
 /**
  * This function gives the distribution-free interval of a median: [x_(k), x_(n+1−k)], for the
  * largest k whose two tails each hold 2.5 % or less. For eleven runs, it gives [x_(2), x_(10)],
@@ -700,17 +826,18 @@ interface Lab {
   lift(kind: NetworkFault, at?: FaultPlace, options?: FaultOptions): Promise<void>;
   /**
    * This method puts tc netem on the egress of a namespace, for example "delay 20ms" or "loss 5%",
-   * or removes it with null. The stamps of the injector bracket it.
+   * or removes it with null. At a hop, it is the egress of the device of the hop toward one side.
+   * The stamps of the injector bracket it.
    */
-  netem(spec: string | null, at?: FaultSide): Promise<{
+  netem(spec: string | null, at?: FaultPlace): Promise<{
     t0: number;
     t1: number;
   }>;
   /**
    * This method sets the TSO, GSO and GRO of a namespace. With them off, netem and a length match
-   * act on segments, not on GSO super-packets of up to 64 KB.
+   * act on segments, not on GSO super-packets of up to 64 KB. At a hop, it sets the two devices of the hop.
    */
-  offload(on: boolean, at?: FaultSide): Promise<void>;
+  offload(on: boolean, at?: FaultPlace): Promise<void>;
   /** This method stops the server process in the target with SIGKILL, with the command of the target. */
   kill(): Promise<void>;
   /**
@@ -768,6 +895,8 @@ export declare const routesOf: (hops: readonly Pick<Hop, "kind">[]) => Array<{
  * is not a router. Element 0 is the client, element k is hop k - 1, and the last element is the target.
  */
 export declare const endpointAfter: (hops: readonly Pick<Hop, "kind">[], element: number) => number;
+/** This function gives the device of a route from its line in `ip -o route get`: "10.0.0.2 dev eth0 src …". */
+export declare const deviceOf: (route: string) => string;
 /**
  * This function opens one environment:
  * - a chain of new networks, and the stock target on the last one. A direct pair has one network.
@@ -828,5 +957,5 @@ export declare const spread: (runs: number, environments: number) => number[];
  */
 export declare const runCell: (cell: Cell, say?: (line: string) => void) => Promise<CellResult>;
 //#endregion
-export type { Agent, Cell, CellResult, Client, ClientImage, Decision, Fault, FaultPlace, FaultSide, Hop, Interval, KernelRecord, Lab, Manifest, ManifestHop, Measured, MiddleboxEvent, MiddleboxForm, NetworkFault, Observer, Outcome, Prediction, Probe, ProbeEvent, ProxyHop, Recovers, Reference, Replies, RouterHop, Run, RunTopology, SocketPlace, Target, Topology, Toxic, Verdict };
+export type { Agent, Cell, CellResult, Client, ClientImage, Crossings, Decision, Fault, FaultPlace, FaultSide, Hop, Interval, KernelRecord, Lab, Manifest, ManifestHop, Measured, MiddleboxEvent, MiddleboxForm, NetworkFault, Observer, Outcome, Prediction, Probe, ProbeEvent, ProxyHop, ProxyTime, Recovers, Reference, Reordering, Replies, RouterHop, RttRange, Run, RunTopology, SocketPlace, Spread, Target, Topology, Toxic, Verdict };
 //# sourceMappingURL=index.d.ts.map