@mark1russell7/harness 0.4.1 → 0.5.0

package/README.md

@@ -8,7 +8,7 @@ The harness does not compute predictions. The author of a cell computes the pred
 
 A **cell** (`cell.ts`) is all that one measurement needs, as data:
 
-- **The topology.** A stock target image, pinned by digest, and the hops between the client and the target (`Cell.topology`). Without hops, the client and the target are a direct pair.
+- **The topology.** A stock target image, pinned by digest, and the hops between the client and the target (`Cell.topology`). Without hops, the client and the target are a direct pair. A replicated group under its Sentinels can replace the target (`Cell.group`).
 - **The settings.** The sysctls of the client namespace, and the socket options of the client.
 - **The workload.** A connection that stays idle, a connection with a request in flight, or a stream of requests. A stream of `GET` requests for a long value keeps many segments in flight.
 - **The fault.** A black hole, a reset, a frozen process, or a stall. In a topology, also a fault at a router or a proxy, or a router that forgets a flow. Also a proxy that closes an idle connection, or a toxic of Toxiproxy.
@@ -52,6 +52,33 @@ A lookup treats an expired conntrack entry as gone, thus the router forgets a fl
 
 The lines of each router (`Lab.middlebox`) are the conntrack events and the resets that it sent, on CLOCK_MONOTONIC. Each run keeps the lines from its start to its end.
 
+## The groups
+
+A group (`Group`, `Cell.group`) is a replicated group under an external failover authority, as data. It is a primary and its replicas, and the Sentinels that decide which member is the primary. A group cell has no hops, and the group replaces its target.
+
+- **The parts.** Each member and each Sentinel is a container from one stock image, pinned by digest. Each part has its own network namespace, thus a fault can act on one part. The server of each part is the child of a shell. Thus a fin can stop the server, and the container stays up.
+- **The data.** The group gives its members, its Sentinels and the quorum. It also gives the settings of each member (`memberSettings`) and of each Sentinel for the group (`sentinelSettings`), for example `down-after-milliseconds`. The first member is the primary when an environment starts.
+- **The sidecars.** Each part has an injector sidecar in its network namespace and in its process namespace. Thus the injector can send a signal to the server of the part, with monotonic stamps around the `kill(2)` calls.
+- **The client.** The client of each run is ioredis 6.0.0 in Sentinel mode (`client-sentinel.mjs`).
+  - It asks the Sentinels for the primary of the group. Then it writes a stream of sequence numbers: `SET w:<n> <n>`.
+  - It sends each write when the reply to the write before it came and the interval passed.
+  - Each write is a `sent` line, then an `ack` line or a `fail` line. An `ack` line has the address of the server that accepted the write, and a `fail` line has its error.
+- **The observer.** A container of its own subscribes to each channel of each Sentinel (`observer.mjs`).
+  - Each Sentinel publishes each of its events on a channel with the name of the event. The observer stamps each message on CLOCK_MONOTONIC when it arrives.
+  - No fault rule names the address of the observer, thus the timeline stays whole.
+- **The prober** watches the namespace of the client. A calibration (`Cell.calibrates` is `sentinels`) makes it watch each Sentinel too.
+
+The roles of a group move: after a failover, a replica is the primary. Thus a fault names a part by its role at the start of a run (`GroupPlace`): `primary`, `replica` or `sentinel`.
+
+Each run of a group cell starts from a group that converged:
+
+- Each Sentinel agrees on one primary with no flag but `master`, and it knows each replica and each other Sentinel.
+- Each replica follows that primary with its link up, and the primary has no failover in progress.
+
+The harness then removes each key from the group. After the run, the group converges again, and the keys that it holds tell which writes it kept.
+
+A Sentinel that aborted a failover does not try again for 2 × `failover-timeout`. Thus after a run with an aborted failover, each Sentinel forgets the state of the group (`SENTINEL RESET`). The next run then starts from a group that each Sentinel knows again.
+
 ## The clients and the agents
 
 - **A client image** (`ClientImage`, `Cell.client`) replaces the TCP client of the harness in each run. It operates in the namespace of the client. Its command gets the arguments of the TCP client, and it prints the same JSON lines.
@@ -66,7 +93,8 @@ The harness builds three images. The tag of each image comes from the content of
   - `client`, the TCP client of a cell. Rust's `std::net` sets no socket option by itself, thus the measurement gets the timers of the namespace. The client also samples `TCP_INFO` each 2 ms, a second vantage on its socket. It waits in `poll()`, which does not lock the socket. A keepalive timer that finds its socket held tries again HZ/20 later.
   - In a stream, a reply of the `client` is whole at its CRLF, or after the length of a bulk string. The client then writes a `replied` line.
   - `inject`, which applies a fault in one `iptables-restore` commit, with monotonic stamps around the commit. Thus the start of a fault is known in the VM. A `docker exec` from Windows comes 190 ms late at the median. With `--via DEV`, the rules go in the FORWARD chain, on the device toward one side.
-- **The Node client image** (`image/node`) is a driver client: ioredis 6.0.0 on Node 24.21.0. The digest of the image and an npm lockfile pin them. `client-ioredis.mjs` prints the lines of the TCP client on the same clock, and it adds the status changes of ioredis as `status` lines.
+  - `inject` also sends a signal to a process by its name (`signal`), and one command to a server of the Redis protocol (`resp`). Stamps bracket each.
+- **The Node client image** (`image/node`) is a driver client: ioredis 6.0.0 on Node 24.21.0. The digest of the image and an npm lockfile pin them. `client-ioredis.mjs` prints the lines of the TCP client on the same clock, and it adds the status changes of ioredis as `status` lines. The same image holds the client of a group in Sentinel mode (`client-sentinel.mjs`) and the observer of its Sentinels (`observer.mjs`).
 - **The router image** (`image/router`) is a digest-pinned Alpine with conntrack-tools, nftables, iproute2 and Python. `mbx.sh` sets the form of the router in one nftables table, `ctwatch.py` writes the conntrack events, and `resetter.py` sends the resets of the `reset-both` form. Each run records its packages in the manifest.
 
 The build does the `cargo test` of the two crates. Thus the record decoder passes its tests before an image exists. Each amount that the native programs print is a quantity in a canonical unit. For example, an RTO is in `jiffy`, a round trip in `us`, and a segment in `B`.
@@ -97,6 +125,13 @@ ioredis sets the keepalive of its socket itself, after it connects. Thus a cell
 - **An idle proxy** (`proxy-idle`) injects nothing. The proxy closes an idle connection at its own timeout.
 - **A toxic** (`toxic`, `Cell.toxic`) is one toxic of Toxiproxy, added through its HTTP API by `inject http`, with monotonic stamps around the request. Its name is `fault`, and the harness removes it after the run. A cell whose fault is another one adds its toxic first: a slow close, then the close of the server.
 - **A close of the server** (`server-close`) makes the server close the connection of each client, with `Target.close`: `CLIENT KILL` for Redis.
+- **A fault on a group** acts on the part that `Cell.faultAt` names by its role:
+  - A black hole (`blackhole`) drops each segment between the part and each peer. A reset (`rst`) answers each segment from a peer with a reset, thus the part refuses each connection.
+  - A partial partition (`partition`) is a black hole between the part and some of its peers (`GroupPlace.cuts`). `group` cuts the part from the other members and the Sentinels, and `clients` cuts it from the client.
+  - The rules for each peer go into one commit. The injector takes a list of peers, `A,B,C`.
+  - A frozen process (`freeze-process`) is SIGSTOP to the server of the part, and SIGCONT thaws it. A FIN (`fin`) is SIGKILL, and the server starts again after the run.
+  - A planned failover (`planned-failover`) is `SENTINEL FAILOVER <group>` to the first Sentinel, or `SENTINEL FAILOVER <group> COORDINATED` (`Cell.coordinated`). The injector stamps the write of the command and its reply.
+  - A fault with a hold (`Cell.hold`) lifts at that time after its commit. The injector waits for that instant in the VM. A stall (`Cell.stall`) lifts at its length after the first write into it.
 
 A length match and netem act on GSO super-packets of up to 64 KB, not on segments, while TSO, GSO and GRO are on. Thus the cells of `mtu-blackhole` and `netem` turn them off on the two ends (`lab.offload`). They also turn them off on the two devices of a hop where the fault is.
 
@@ -121,6 +156,15 @@ Each run starts from kernels that keep nothing from the run before it (`lab.forg
 - **Reordering.** A measurement-only run also gives what reordering, duplicates and losses did (`reordering`). At the client, it counts the arrivals beyond a hole and the arrivals of data that it had already, against `rcv_nxt`. From the sender, it gives the most SACKed and lost segments, the DSACKs, the fast retransmissions, and the reordering degree and count of the kernel.
 - **An arrival is not always one segment.** The segments that arrive while the owner holds the socket wait in its backlog, and the kernel joins the contiguous ones. Thus the kernel counts out-of-order segments (`rcv_ooopack`), and the prober counts arrivals.
 - **A proxy's own time.** A calibration cell (`Cell.calibrates`) gives the time of its proxy for each request and its reply (`crossings`). Each time goes from an arrival on one socket of the proxy to its next write on the other socket (a `tx` record).
+- **A router that forgets in silence.** A silent prediction can say that the router forgets the flow in a band after the last segment (`Prediction.middlebox`). The run then holds when the forget is in that band and nothing detects it. A missing forget, or one outside its band, is a finding. Without it, the router must keep the flow.
+- **A group** (`measureGroup`, `timeline.ts`). The time counts from the start of the commit of the fault, or from the first write into a stall.
+  - It ends at the first write that a primary accepted after the fault. That primary is a new one, or the same one after the lift, as the prediction says (`GroupPrediction.lands`).
+  - The time of the acceptance is the last segment with data from that primary at the stamp of the client, or before it.
+  - **The timeline.** Each stage is the first announcement of its event by any Sentinel, from the fault. The stages are `+sdown` and `+odown` of the primary, `+try-failover` and `+elected-leader`. Then `+selected-slave`, `+promoted-slave`, `+switch-master` and `+convert-to-slave`.
+  - **The writes.** The run gives the writes that the old primary accepted after the fault, and the writes that a primary accepted and the group lost. It also gives each write whose outcome the client did not see. The lost writes are the cost of a split brain.
+  - **The errors on the way.** The run gives each kind of error that the client met after the fault, with its first time and its count.
+  - **Void runs.** A run is void when a Sentinel entered its TILT mode, or when the election split its votes. Both measured another path than the band assumes. A run whose Sentinels reached `+odown` and started no failover is void: the hold-off of an earlier failover stopped them. A band whose hold lifted more than δ_abs late is void too. The rules of the RTT and of a slow commit apply.
+  - **The stamps.** A calibration gives the delay of each stamp of the observer after the write of its Sentinel that it read (`lags`).
 
 ## The verdict
 
@@ -135,6 +179,10 @@ Each run starts from kernels that keep nothing from the run before it (`lab.forg
 - **A calibration.** A calibration cell gives the spread of the time of its proxy in each direction (`proxy`). The spread is the count, the median, the 99th and 99.9th percentiles by nearest rank, and the maximum.
 - **A reply.** A reply in its band (`replies`) gets the same test as a detection.
 - **The time of a proxy.** A band through a proxy includes an upper quantile of the time of the proxy for each crossing (`through`). The author of the cell adds it from a calibration cell, and the verdict tests the band as it is.
+- **A group** (`GroupVerdict`). The band of the first accepted write gets the tests of a detection.
+  - Each stage that the prediction gives a band (`GroupPrediction.stages`) gets its own containment: [T_lo − δ_abs, T_hi + the error of the stamps + δ_abs].
+  - A run that did not announce the stage is outside too. A stage outside its band fails that stage. The decision of the cell stays the decision of its band.
+  - The verdict also gives the median of each stage, the lost writes of each run, and the errors on the way.
 
 The distribution test needs a simulator, which does not exist at this time.
 
@@ -146,10 +194,12 @@ The distribution test needs a simulator, which does not exist at this time.
   - The manifest holds the kernel release, the architecture, the boot id and the Docker version. It also holds the image ids, the packages of the image, the client and the sysctls as read back.
   - A cell whose probe names a `driver` opens its labs with the client of that driver (`openLab(target, "ioredis")`).
   - The harness sets the `targetSettings` of a cell in the target before a client connects, with the `settings` commands of the target (`redis-cli CONFIG SET`, then `CONFIG GET`).
+  - A group cell opens a group environment (`openGroupLab`), and each run is `groupOnce`. Its warm-up is a whole run with its fault, because no fault of a group ends its environment. The manifest also holds the settings of each member and of each Sentinel, as each part read them.
 - **The observer.** A cell observes the client, the server, or a socket of a proxy toward one side (`observer`). A cell on the server side measures the socket of the target for the connection of the client. The run waits for the error of that socket. `freeze-process` freezes the peer of the observer: the target for the client, the client for the server.
   - A cell on the server side sets the sysctls of the target through `targetSysctls`, with a privileged sidecar in the namespace of the target. That sidecar starts only when a cell asks for it.
   - Its in-flight data is the reply of the server. Thus the client sends `probe.request` first, for example a BLPOP with a timeout, and the fault occurs while the server holds it.
 - **The pause before the fault.** In every other run, the fault comes 250 ms after the client's first reply (`QUIET_MS`). That is more than Linux's longest delayed ACK, 200 ms. A fault that comes earlier meets the client's delayed ACK of the reply, not the request or the idle time that the cell measures. On a fast host the fault came 35 ms after the reply, and the ACK met a reset first.
+  - A run of a group has no such pause. Its fault comes after the 20th accepted write of its stream, and each band of a group counts from the fault.
 - **`judge(cell, runs)`** (`run.ts`) measures and judges the raw records of a cell with the code at this time. `runCell` uses it during a cell. A holder of raw records can judge them again after a change to `observe.ts` or to the verdict.
 - **Testcontainers loads when a lab first opens.** Code that only measures and judges does not load it.
 
@@ -165,6 +215,8 @@ A run needs Docker, and privileged containers for the prober and the sidecars. T
 
 `pnpm --filter @mark1russell7/harness test` starts the tests. They measure test records with the shape of real kernel records (keepalive, and the tail loss probe before the retransmissions of the RTO timer). They also test the containment, the median equivalence, the extension and the silence of the verdict.
 
-The cells themselves need Docker and privileged containers. CI does not start them. `HARNESS_LAB=1` starts the lab tests on Docker: a client image, an agent, and a reset and netem at a router.
+The tests also measure a group run from test lines. They find the first accepted write, the stages, the writes that the group kept, and the void runs. They test the views that make a group healthy, and the peers that a fault on a part cuts.
+
+The cells themselves need Docker and privileged containers. CI does not start them. `HARNESS_LAB=1` starts the lab tests on Docker. They start a client image and an agent, and they put a reset and netem at a router. They also fail a group of Valkey 9.1 over with a planned failover.
 
 `pnpm --filter @mark1russell7/harness build` writes the package for npm into `build/`.