@mark1russell7/harness 0.14.2 → 0.14.4

package/README.md

@@ -8,7 +8,7 @@ The harness does not compute predictions. The author of a cell computes the pred
 
 A **cell** (`cell.ts`) is all that one measurement needs, as data:
 
-- **The topology.** A stock target image, pinned by digest, and the hops between the client and the target (`Cell.topology`). Without hops, the client and the target are a direct pair. A replicated group under its Sentinels can replace the target (`Cell.group`).
+- **The topology.** A stock target image, pinned by digest, and the hops between the client and the target (`Cell.topology`). Without hops, the client and the target are a direct pair. A replicated group under its Sentinels can replace the target (`Cell.group`). A PostgreSQL target can have one standby, with a final read of each server (`Cell.pair`).
 - **The subject.** The program under test, as data. Its image is an image pinned by digest, a build context, or an image that the harness ships (`Cell.client`, `Target.subject`).
 - **The settings.** The sysctls of the client namespace, and the socket options of the client.
 - **The workload.** A connection that stays idle, a connection with a request in flight, or a stream of requests. A stream of `GET` requests for a long value keeps many segments in flight.
@@ -32,6 +32,12 @@ A **cell** (`cell.ts`) is all that one measurement needs, as data:
 
 Each command is an argv array.
 
+### A lab in a guest
+
+The folder `vm/`, the file `src/vm.ts` and the workflow `vm.yml` operate a lab in a guest of QEMU, layer E4 of `docs/S7.md`. `src/vm.ts` builds a pinned kernel and a disk, then boots that kernel directly, with no boot loader. The guest has systemd, sshd and the Docker of the runner. The images build in the guest, because the prober compiles against the BTF of the kernel that it operates on.
+
+`runCells` in that file is the one command of the cells for the runner and for a guest. Rule 3 of that document pairs the two in one job. A guest cannot read its runner, thus the runner gives its host in `LAB_RUNNER`. `substrateHere` writes E4 with that cloud and image, and with the kernel and HZ of the guest. A run also records `steal`, the share of the CPU that the host took (`stealShare`). A guest sees only the host above it, thus this number does not show the layer below a hosted runner.
+
 ## The topologies
 
 A topology (`Topology`) is the chain of hops from the client to the target, as data. Each hop is on two networks, one for each side. Network j joins element j and element j + 1: the client is element 0, and the target is the last element.
@@ -59,6 +65,10 @@ The `reset-both` form sends its resets on the DESTROY event. Each reset has the
 
 The lines of each router (`Lab.middlebox`) are the conntrack events and the resets that it sent, on CLOCK_MONOTONIC. Each run keeps the lines from its start to its end.
 
+A fabric is a topology with no hops and with `fabric` (`fabric.ts`, layer E3 of `docs/S7.md`). Four routers of FRRouting, two leaves and two spines, run eBGP over veth pairs of containerlab. The client hangs off one leaf and the target off the other, each on a new device `eth1`. Each link has the MTU of a Docker network, thus a segment is as long as on E1. Each leaf prefers spine1 by a weight, and spine2 holds the other path.
+
+The fault `silence` drops each packet of spine1 in one commit, and its links stay up. A route monitor on each leaf records each change of a route. The later move of the two leaves ends the stall of the run (`Run.heal`), and the verdict of a stall reads it. A move outside the band of the cell makes the run void. Before each run, `ready` waits for each session, for the route on spine1 and for the path on spine2. The record of the run keeps the routes and the count of established sessions (`Run.fabric`).
+
 ## The groups
 
 A group (`Group`, `Cell.group`) is a replicated group under an external failover authority, as data. It is a primary and its replicas, and the Sentinels that decide which member is the primary. A group cell has no hops, and the group replaces its target.
@@ -84,6 +94,8 @@ Each run of a group cell starts from a group that converged:
 
 The harness then removes each key from the group. After the run, the group converges again, and the keys that it holds tell which writes it kept.
 
+A driver with `op: "rpush"` appends each write to one list, and the entries tell how many copies of each write the group kept. `window` sets the count of writers. A band of extra copies (`GroupPrediction.duplicates`) judges the cell as a band of lost writes does.
+
 A Sentinel that aborted a failover does not try again for 2 × `failover-timeout`. Thus after a run with an aborted failover, each Sentinel forgets the state of the group (`SENTINEL RESET`). The next run then starts from a group that each Sentinel knows again.
 
 ## The replica sets
@@ -93,7 +105,7 @@ A replica set (`ReplicaSet`, `Cell.replicaSet`) is a replicated group with no ex
 - **The members.** Each member is a container from one stock image, pinned by digest, in its own network namespace. Its server is the child of a shell, with the name of the set and the options of the set (`args`).
 - **The start.** The first member starts the set (`rs.initiate`) with the address of each member, as the client reaches it, and with the settings of the set. An example of a setting is `electionTimeoutMillis`.
 - **The sidecars.** Each member has an injector sidecar in its network namespace and in its process namespace, as a member of a group has.
-- **The client.** The client of each run is the MongoDB driver (`image/mongo`). It knows the address of each member and the name of the set. It writes a stream of sequence numbers with the write concern of the cell (`Cell.writeConcern`).
+- **The client.** The client of each run is the MongoDB driver (`image/mongo`). It knows the address of each member and the name of the set. It writes a stream of sequence numbers with the write concern of the cell (`Cell.writeConcern`). A run continues 2 s after the lift, and for `Cell.watch` more. A prediction of no failover after a short hold needs that time.
 - **The timeline.** A log consumer reads the JSON log of each member. A run keeps the lines of replication, of its heartbeats, of the elections, of the rollbacks and of the commands (`KEPT`).
 - **The clocks.** The log of mongod has the wall clock in ms. The client writes the wall clock beside the monotonic clock each second (`clocks`). Thus the run puts each line of a member on the monotonic clock (`clockOffsetOf`).
 - **The prober** watches the namespace of the client.
@@ -104,6 +116,16 @@ A step-down holds its old primary out of the elections for 60 s. Thus before eac
 
 A fault names a member by its role at the start of a run: `primary`, or `replica` for a secondary. A cut from one member (`one`, `one-way`) cuts the first other member in the sequence of the set.
 
+### The recovery drills
+
+A drill (`Drill`, `Cell.drill`) stops a replica set during writes, removes a disk, and recovers. `drillOnce` in `drill.ts` does it, and `recovery.ts` holds the pure parts.
+
+- **The backups** are real. A `mongodump --oplog` and each slice of the oplog go to a host of their own (`ReplicaSetLab.vault`). `bsondump` reads each backup. The record keeps the last write that each backup holds (`RunBackup`).
+- **The fault** comes at `age` after that last write, on the monotonic clock. Thus a run sets the age of its newest backup.
+- **A restore** cannot have a client of the set, because a client writes to any primary. So each slice joins the oplog of the dump, and a server with no set takes the restore on the next port. The set then starts again. A second client (`--from`) sends the writes after the restore, thus no electionId of the old set stops it.
+- **An election** removes only the disk of the primary. The primary comes back at the end of the run, thus the set converges.
+- **The measure** is a set difference (`recoveryOf`). Each rule that a run breaks is a finding (`recoveryFindings`). A finding is a hole below a kept write, lost writes that the last backup does not explain, or tenant data that changed.
+
 ## The clusters of RabbitMQ
 
 A cluster (`Cluster`, `Cell.cluster`) is a cluster of RabbitMQ nodes with one quorum queue, as data. The members of the queue elect its leader by Raft, one member on each node. The cluster replaces the target of its cell.
@@ -111,7 +133,7 @@ A cluster (`Cluster`, `Cell.cluster`) is a cluster of RabbitMQ nodes with one qu
 - **The nodes.** Each node is a container from one stock image, pinned by digest, in its own network namespace. Its hostname is the name of its node (`rabbit@<name>`), and all nodes share one Erlang cookie. Its server is the child of a shell.
 - **The start.** The nodes form the cluster from the list of their names (`classic_config`), and the lab waits until each node runs in it. Then the admin (`image/amqp/admin-cluster.mjs`) declares the queue on the first node, with the leader locator `client-local`.
 - **The sidecars.** Each node has an injector sidecar in its network namespace and in its process namespace.
-- **The client.** The client of each run is amqplib, in `image/amqp/client-cluster.mjs`. The client publishes a stream of sequence numbers on a confirm channel through one node. That node is the last follower, or the node of the leader (`Cell.clientAt`). The library does not connect again by itself. When the connection ends, the client connects to the next node after 1 s, as an application does.
+- **The client.** The client of each run is amqplib, in `image/amqp/client-cluster.mjs`. The client publishes a stream of sequence numbers on a confirm channel through one node. That node is the last follower, or the node of the leader (`Cell.clientAt`). The library does not connect again by itself. When the connection ends, the client connects to the next node after 1 s, as an application does. `driverOptions.window` sets the count of writers on the one channel. Each option that the client does not read goes to the socket of amqplib, for example `noDelay`.
 - **The timeline.** A log consumer reads the console log of each node. A run keeps the lines of the queue, each warning and error, and the lines of the maintenance mode. Ra writes a NOTICE for each change from or to the leader, and an INFO for each new leader that a follower detects.
 - **The clocks.** The log has the wall clock in µs. The run puts each line on the monotonic clock with the clock lines of the client.
 
@@ -119,6 +141,14 @@ Each run starts from a queue that converged (`rabbitmq-queues quorum_status`): o
 
 A step-down (`step-down`) moves the leadership to the first follower (`rabbitmq-queues transfer_leadership`). A drain (`drain`) puts the node in maintenance mode (`rabbitmq-upgrade drain`), and the run revives the node after it. Each of them starts an Erlang VM for the tools first. Thus its stamp is a line of the log, and not the command. For a step-down it is the NOTICE of the old leader that gives the leadership away (`GIVES_AWAY`). A drain stops the member of the leader, thus its stamp is the first follower that sees its monitor of the leader go down (`MONITOR_DOWN`).
 
+A Kafka group (`Kafka`, `Cell.kafka`) is a group of Kafka brokers in KRaft mode with one controller, as data. The group replaces the target of its cell.
+
+- **The nodes.** Each node is a container from one stock image, pinned by digest, in its own network namespace. The image starts by its own command, and each setting comes from the environment (`envOf`). The controller is a node of its own. A fault does not go on it.
+- **The topic.** Each run makes one new topic with one partition, a replica on each broker, and `Kafka.minInsync`. The first broker in the replica list leads. A run starts when each broker is not fenced and the partition is whole.
+- **The sidecars.** Each node has an injector sidecar in its network namespace and in its process namespace. A `partition` with `cuts: "group"` cuts the leader from the other brokers and the controller. The client keeps its route to the leader.
+- **The client.** The client of each run is the producer of kafka-clients, in `image/kafka/ClientKafka.java`. Each of `Kafka.writers` threads has one write at a time. Each entry of `Cell.driverOptions` is a producer setting.
+- **The record.** A Kafka group has no log lines in its record, and `Run.group.sentinels` holds the controller. After the run the partition converges again, and the admin (`image/kafka/AdminKafka.java`) reads the whole partition to its high watermark. The leader after the run tells if a failover came.
+
 ## The clients and the agents
 
 - **A client image** (`ClientImage`, `Cell.client`) replaces the TCP client of the harness in each run. It operates in the namespace of the client. Its command gets the arguments of the TCP client, and it prints the same JSON lines. Its image is a stock image by its name, or the source of the image of a subject (`ImageSource`).
@@ -184,14 +214,18 @@ From 0.11.20, the harness builds each image with the CLI of Docker (`docker buil
 - **The PostgreSQL client image** (`image/pg`) is node-postgres: pg 8.23.1 with pg-pool 3.14.0 on Node 24.21.0. The digest of the image and an npm lockfile pin them. `client-pg.mjs` makes one `pg.Pool` with the options of the cell (`Cell.driverOptions`), and it prints the lines of the TCP client on the same clock. The first query makes the connection, and the reply gives the client back to the pool, where it is idle. An error of the pool for that idle client is the detection of an idle cell. The probe is `{ driver: { name: "pg", query, inflight } }`, with `SELECT 1` as the query when the cell gives none.
   - `inflight` is the text of the query in flight: the second query of an in-flight run. The first query then makes the pool ready at once, and only the query in flight is slow, for example `SELECT pg_sleep(20)`. The HikariCP client reads it too.
 - **The HikariCP client image** (`image/hikari`) is HikariCP 7.1.0 over pgjdbc 42.7.13 on Temurin 21.0.12.1. The digests of the two stages and the SHA-256 of each jar pin them. The probe is `{ driver: { name: "hikari", query, inflight } }`.
-  - `ClientHikari.java` makes one `HikariDataSource` with the properties of the cell (`Cell.driverOptions`). A name with the prefix `dataSource.` is a property of pgjdbc.
+  - `ClientHikari.java` makes one `HikariDataSource` with the properties of the cell (`Cell.driverOptions`). A name with the prefix `dataSource.` is a property of pgjdbc. A name with the prefix `system.` is a system property of the JVM, which the client sets before its pool.
   - The user is `postgres`, unless the cell gives `username` (and `password`). Supavisor finds the tenant of a client in its user, for example `postgres.harness`.
   - It prints the lines of the pg client on the same clock. Each socket comes from the `PGStream` of its connection, and a `pool` line gives each change of the counts of the pool. The errno of a failure is the SQLState of pgjdbc.
   - Before its first line, it loads the classes of a connect and of a pool on the loopback of its namespace. It also starts the SecureRandom that HikariCP reads for the UUID of each new connection. Then it parses an ERROR of the server, as for the timeout of a statement. A new JVM loads each class at its first use. Thus a band does not time that load.
+  - With `burst` in the probe (`{ count, gap }`), it sends `count` queries in place of one query in flight. Query 2 and the next ones each start on their own thread, `gap` apart, when each connection of the pool is idle. A `burst` line gives the counts of replies and failures, then the counts of the pool. Before its first line, a client with a burst also makes one borrow that times out. Thus the band of a failure does not time that load.
 - **The RabbitMQ client image** (`image/amqp`) is amqplib 2.2.0 on Node 24.21.0. The digest of the image and an npm lockfile pin them. The probe is `{ driver: { name: "amqp" } }`.
   - `client-amqp.mjs` opens one connection and one channel. Each operation is an RPC of the channel: the first declares a queue that the broker names, and each later one declares it passively (checkQueue). The queue is exclusive, because RabbitMQ 4.3 refuses a transient queue that is not.
   - The options of the cell (`Cell.driverOptions`) give `heartbeat` in seconds, the user and its password (guest at the defaults), and the socket options of amqplib.
   - It prints the lines of the pg client on the same clock. A `connection` line gives each event of the connection, and a `channel` line each event of the channel. The errno of a failure is the error of the connection when it has one.
+- **The Kafka client image** (`image/kafka`) is kafka-clients 4.3.2 on Java 21, on the base image of the HikariCP client. The digest of the image and the hash of each jar (`jars.sha256`) pin them. A Kafka lab starts it (`Cell.kafka`), not a probe of a driver.
+  - `ClientKafka.java` writes the sequence numbers to partition 0 of the topic of the run, with the settings of the cell. It prints `sent`, `ack` and `fail` lines on the monotonic clock.
+  - `AdminKafka.java` makes a topic, gives a view of the partition, and reads the whole partition.
 - **The gRPC client image** (`image/grpc`) is grpc-js 1.14.6 on Node 24.21.0. The digest of the image and an npm lockfile pin them. The probe is `{ driver: { name: "grpc", hold } }`.
   - `client-grpc.mjs` opens one channel with no TLS. Each operation is a unary call of `grpc.health.v1.Health/Check`, with the messages encoded in the script. The first call makes the connection.
   - The options of the cell (`Cell.driverOptions`) are the channel options of grpc-js (`grpc.*`), and `deadline_ms` for the deadline of each call. With `hold`, the call in flight asks the server of the harness to hold its reply that long.
@@ -260,7 +294,7 @@ ioredis sets the keepalive of its socket itself, after it connects. Thus a cell
   - Without `inFlight`, the event comes first, and the request meets it. With `inFlight`, the request goes first, and the event comes while it is in flight. A run whose order is not that order is void.
 - **A fault on a group** acts on the part that `Cell.faultAt` names by its role:
   - A black hole (`blackhole`) drops each segment between the part and each peer. A reset (`rst`) answers each segment from a peer with a reset, thus the part refuses each connection.
-  - A partial partition (`partition`) is a black hole between the part and some of its peers (`GroupPlace.cuts`). `group` cuts the part from the other members and the Sentinels, and `clients` cuts it from the client.
+  - A partial partition (`partition`) is a black hole between the part and some of its peers (`GroupPlace.cuts`). `group` cuts the part from the other members and the Sentinels, and `clients` cuts it from the client. `to-clients` drops only the segments from the part to the client, thus each write reaches the part and no reply reaches the client.
   - The rules for each peer go into one commit. The injector takes a list of peers, `A,B,C`.
   - A frozen process (`freeze-process`) is SIGSTOP to the server of the part, and SIGCONT thaws it. A FIN (`fin`) is SIGKILL, and the server starts again after the run.
   - A planned failover (`planned-failover`) is `SENTINEL FAILOVER <group>` to the first Sentinel, or `SENTINEL FAILOVER <group> COORDINATED` (`Cell.coordinated`). The injector stamps the write of the command and its reply.
@@ -393,6 +427,9 @@ The distribution test needs a simulator, which does not exist at this time.
   - A cell with `fresh` also does a dry warm-up: its fault can change the environment for the runs after it. Behind a black hole, Pgpool-II can detach its only server, and then it refuses each new client. Such a cell has an environment for each run.
   - The `concurrency` of a cell sets how many environments operate at the same time, each with its own prober.
   - The manifest holds the kernel release, the architecture, the boot id and the Docker version. It also holds the image ids, the packages of the image, the client and the sysctls as read back.
+  - Each image that the harness builds has a build record in the manifest (`BuildRecord`). It gives the tag, the bases by digest and the sha256 of each lockfile. It also gives the versions of the packages that the manifests of the image name. The harness, the client and the image of a subject have one. The harness record also has the version of the package.
+  - `manifestProblems(manifest)` gives each image of a manifest that does not pin its programs. Such an image is a pulled image without a digest, or a build without its build record. `commitProblems(commit)` examines the commit of a record. A record without a pin for each program is not a record, because nobody can reproduce its numbers.
+  - `recordProblems(record)` gives each problem of a record: its commit, its manifests, a checkout with changes (`DIRTY`) and the mark of a debug run (`DEBUG`). The writer of the cells and the read of a schedule use this one function.
   - A cell whose probe names a `driver` opens its labs with the client of that driver (`openLab(target, "ioredis")`).
   - The harness sets the `targetSettings` of a cell in the target before a client connects, with the `settings` commands of the target (`redis-cli CONFIG SET`, then `CONFIG GET`).
   - A group cell opens a group environment (`openGroupLab`), and each run is `groupOnce`. Its warm-up is a whole run with its fault, because no fault of a group ends its environment. The manifest also holds the settings of each member and of each Sentinel, as each part read them.