@mark1russell7/harness 0.14.3 → 0.15.0

package/README.md

@@ -8,7 +8,7 @@ The harness does not compute predictions. The author of a cell computes the pred
 
 A **cell** (`cell.ts`) is all that one measurement needs, as data:
 
-- **The topology.** A stock target image, pinned by digest, and the hops between the client and the target (`Cell.topology`). Without hops, the client and the target are a direct pair. A replicated group under its Sentinels can replace the target (`Cell.group`).
+- **The topology.** A stock target image, pinned by digest, and the hops between the client and the target (`Cell.topology`). Without hops, the client and the target are a direct pair. A replicated group under its Sentinels can replace the target (`Cell.group`). A PostgreSQL target can have one standby, with a final read of each server (`Cell.pair`).
 - **The subject.** The program under test, as data. Its image is an image pinned by digest, a build context, or an image that the harness ships (`Cell.client`, `Target.subject`).
 - **The settings.** The sysctls of the client namespace, and the socket options of the client.
 - **The workload.** A connection that stays idle, a connection with a request in flight, or a stream of requests. A stream of `GET` requests for a long value keeps many segments in flight.
@@ -94,6 +94,8 @@ Each run of a group cell starts from a group that converged:
 
 The harness then removes each key from the group. After the run, the group converges again, and the keys that it holds tell which writes it kept.
 
+A driver with `op: "rpush"` appends each write to one list, and the entries tell how many copies of each write the group kept. `window` sets the count of writers. A band of extra copies (`GroupPrediction.duplicates`) judges the cell as a band of lost writes does.
+
 A Sentinel that aborted a failover does not try again for 2 × `failover-timeout`. Thus after a run with an aborted failover, each Sentinel forgets the state of the group (`SENTINEL RESET`). The next run then starts from a group that each Sentinel knows again.
 
 ## The replica sets
@@ -103,7 +105,7 @@ A replica set (`ReplicaSet`, `Cell.replicaSet`) is a replicated group with no ex
 - **The members.** Each member is a container from one stock image, pinned by digest, in its own network namespace. Its server is the child of a shell, with the name of the set and the options of the set (`args`).
 - **The start.** The first member starts the set (`rs.initiate`) with the address of each member, as the client reaches it, and with the settings of the set. An example of a setting is `electionTimeoutMillis`.
 - **The sidecars.** Each member has an injector sidecar in its network namespace and in its process namespace, as a member of a group has.
-- **The client.** The client of each run is the MongoDB driver (`image/mongo`). It knows the address of each member and the name of the set. It writes a stream of sequence numbers with the write concern of the cell (`Cell.writeConcern`).
+- **The client.** The client of each run is the MongoDB driver (`image/mongo`). It knows the address of each member and the name of the set. It writes a stream of sequence numbers with the write concern of the cell (`Cell.writeConcern`). A run continues 2 s after the lift, and for `Cell.watch` more. A prediction of no failover after a short hold needs that time.
 - **The timeline.** A log consumer reads the JSON log of each member. A run keeps the lines of replication, of its heartbeats, of the elections, of the rollbacks and of the commands (`KEPT`).
 - **The clocks.** The log of mongod has the wall clock in ms. The client writes the wall clock beside the monotonic clock each second (`clocks`). Thus the run puts each line of a member on the monotonic clock (`clockOffsetOf`).
 - **The prober** watches the namespace of the client.
@@ -131,7 +133,7 @@ A cluster (`Cluster`, `Cell.cluster`) is a cluster of RabbitMQ nodes with one qu
 - **The nodes.** Each node is a container from one stock image, pinned by digest, in its own network namespace. Its hostname is the name of its node (`rabbit@<name>`), and all nodes share one Erlang cookie. Its server is the child of a shell.
 - **The start.** The nodes form the cluster from the list of their names (`classic_config`), and the lab waits until each node runs in it. Then the admin (`image/amqp/admin-cluster.mjs`) declares the queue on the first node, with the leader locator `client-local`.
 - **The sidecars.** Each node has an injector sidecar in its network namespace and in its process namespace.
-- **The client.** The client of each run is amqplib, in `image/amqp/client-cluster.mjs`. The client publishes a stream of sequence numbers on a confirm channel through one node. That node is the last follower, or the node of the leader (`Cell.clientAt`). The library does not connect again by itself. When the connection ends, the client connects to the next node after 1 s, as an application does.
+- **The client.** The client of each run is amqplib, in `image/amqp/client-cluster.mjs`. The client publishes a stream of sequence numbers on a confirm channel through one node. That node is the last follower, or the node of the leader (`Cell.clientAt`). The library does not connect again by itself. When the connection ends, the client connects to the next node after 1 s, as an application does. `driverOptions.window` sets the count of writers on the one channel. Each option that the client does not read goes to the socket of amqplib, for example `noDelay`.
 - **The timeline.** A log consumer reads the console log of each node. A run keeps the lines of the queue, each warning and error, and the lines of the maintenance mode. Ra writes a NOTICE for each change from or to the leader, and an INFO for each new leader that a follower detects.
 - **The clocks.** The log has the wall clock in µs. The run puts each line on the monotonic clock with the clock lines of the client.
 
@@ -139,6 +141,14 @@ Each run starts from a queue that converged (`rabbitmq-queues quorum_status`): o
 
 A step-down (`step-down`) moves the leadership to the first follower (`rabbitmq-queues transfer_leadership`). A drain (`drain`) puts the node in maintenance mode (`rabbitmq-upgrade drain`), and the run revives the node after it. Each of them starts an Erlang VM for the tools first. Thus its stamp is a line of the log, and not the command. For a step-down it is the NOTICE of the old leader that gives the leadership away (`GIVES_AWAY`). A drain stops the member of the leader, thus its stamp is the first follower that sees its monitor of the leader go down (`MONITOR_DOWN`).
 
+A Kafka group (`Kafka`, `Cell.kafka`) is a group of Kafka brokers in KRaft mode with one controller, as data. The group replaces the target of its cell.
+
+- **The nodes.** Each node is a container from one stock image, pinned by digest, in its own network namespace. The image starts by its own command, and each setting comes from the environment (`envOf`). The controller is a node of its own. A fault does not go on it.
+- **The topic.** Each run makes one new topic with one partition, a replica on each broker, and `Kafka.minInsync`. The first broker in the replica list leads. A run starts when each broker is not fenced and the partition is whole.
+- **The sidecars.** Each node has an injector sidecar in its network namespace and in its process namespace. A `partition` with `cuts: "group"` cuts the leader from the other brokers and the controller. The client keeps its route to the leader.
+- **The client.** The client of each run is the producer of kafka-clients, in `image/kafka/ClientKafka.java`. Each of `Kafka.writers` threads has one write at a time. Each entry of `Cell.driverOptions` is a producer setting.
+- **The record.** A Kafka group has no log lines in its record, and `Run.group.sentinels` holds the controller. After the run the partition converges again, and the admin (`image/kafka/AdminKafka.java`) reads the whole partition to its high watermark. The leader after the run tells if a failover came.
+
 ## The clients and the agents
 
 - **A client image** (`ClientImage`, `Cell.client`) replaces the TCP client of the harness in each run. It operates in the namespace of the client. Its command gets the arguments of the TCP client, and it prints the same JSON lines. Its image is a stock image by its name, or the source of the image of a subject (`ImageSource`).
@@ -196,6 +206,7 @@ From 0.11.20, the harness builds each image with the CLI of Docker (`docker buil
   - the kernel program of the prober, compiled against the BTF of the kernel that builds it, and its Rust collector.
   - `client`, the TCP client of a cell. Rust's `std::net` sets no socket option by itself, thus the measurement gets the timers of the namespace. The client also samples `TCP_INFO` each 2 ms, a second vantage on its socket. It waits in `poll()`, which does not lock the socket. A keepalive timer that finds its socket held tries again HZ/20 later.
   - In a stream, a reply of the `client` is whole at its CRLF, or after the length of a bulk string. The client then writes a `replied` line.
+  - `load`, the open-loop driver of an HTTP/1.1 server (SYS-93). Its plan (`systems-load/1`) is a list of streams, each with a key, a schedule (`rigor-schedule/1`) and its requests. Each stream has its own connection, thus a slow reply delays only its own stream. A send does not wait for a reply. A response is whole after its `Content-Length` or its last chunk. The `sent` and `replied` lines name the stream (`k`) and the arrival (`i`), and a `replied` line names the status.
   - `inject`, which applies a fault in one `iptables-restore` commit, with monotonic stamps around the commit. Thus the start of a fault is known in the VM. A `docker exec` from Windows comes 190 ms late at the median. With `--via DEV`, the rules go in the FORWARD chain, on the device toward one side.
   - `inject` also sends a signal to a process by its name (`signal`), and one command to a server of the Redis protocol (`resp`). Stamps bracket each.
 - **The Node client image** (`image/node`) is a driver client: ioredis 6.0.0 on Node 24.21.0. The digest of the image and an npm lockfile pin them. `client-ioredis.mjs` prints the lines of the TCP client on the same clock, and it adds the status changes of ioredis as `status` lines. The same image holds the client of a group in Sentinel mode (`client-sentinel.mjs`) and the observer of its Sentinels (`observer.mjs`).
@@ -204,14 +215,18 @@ From 0.11.20, the harness builds each image with the CLI of Docker (`docker buil
 - **The PostgreSQL client image** (`image/pg`) is node-postgres: pg 8.23.1 with pg-pool 3.14.0 on Node 24.21.0. The digest of the image and an npm lockfile pin them. `client-pg.mjs` makes one `pg.Pool` with the options of the cell (`Cell.driverOptions`), and it prints the lines of the TCP client on the same clock. The first query makes the connection, and the reply gives the client back to the pool, where it is idle. An error of the pool for that idle client is the detection of an idle cell. The probe is `{ driver: { name: "pg", query, inflight } }`, with `SELECT 1` as the query when the cell gives none.
   - `inflight` is the text of the query in flight: the second query of an in-flight run. The first query then makes the pool ready at once, and only the query in flight is slow, for example `SELECT pg_sleep(20)`. The HikariCP client reads it too.
 - **The HikariCP client image** (`image/hikari`) is HikariCP 7.1.0 over pgjdbc 42.7.13 on Temurin 21.0.12.1. The digests of the two stages and the SHA-256 of each jar pin them. The probe is `{ driver: { name: "hikari", query, inflight } }`.
-  - `ClientHikari.java` makes one `HikariDataSource` with the properties of the cell (`Cell.driverOptions`). A name with the prefix `dataSource.` is a property of pgjdbc.
+  - `ClientHikari.java` makes one `HikariDataSource` with the properties of the cell (`Cell.driverOptions`). A name with the prefix `dataSource.` is a property of pgjdbc. A name with the prefix `system.` is a system property of the JVM, which the client sets before its pool.
   - The user is `postgres`, unless the cell gives `username` (and `password`). Supavisor finds the tenant of a client in its user, for example `postgres.harness`.
   - It prints the lines of the pg client on the same clock. Each socket comes from the `PGStream` of its connection, and a `pool` line gives each change of the counts of the pool. The errno of a failure is the SQLState of pgjdbc.
   - Before its first line, it loads the classes of a connect and of a pool on the loopback of its namespace. It also starts the SecureRandom that HikariCP reads for the UUID of each new connection. Then it parses an ERROR of the server, as for the timeout of a statement. A new JVM loads each class at its first use. Thus a band does not time that load.
+  - With `burst` in the probe (`{ count, gap }`), it sends `count` queries in place of one query in flight. Query 2 and the next ones each start on their own thread, `gap` apart, when each connection of the pool is idle. A `burst` line gives the counts of replies and failures, then the counts of the pool. Before its first line, a client with a burst also makes one borrow that times out. Thus the band of a failure does not time that load.
 - **The RabbitMQ client image** (`image/amqp`) is amqplib 2.2.0 on Node 24.21.0. The digest of the image and an npm lockfile pin them. The probe is `{ driver: { name: "amqp" } }`.
   - `client-amqp.mjs` opens one connection and one channel. Each operation is an RPC of the channel: the first declares a queue that the broker names, and each later one declares it passively (checkQueue). The queue is exclusive, because RabbitMQ 4.3 refuses a transient queue that is not.
   - The options of the cell (`Cell.driverOptions`) give `heartbeat` in seconds, the user and its password (guest at the defaults), and the socket options of amqplib.
   - It prints the lines of the pg client on the same clock. A `connection` line gives each event of the connection, and a `channel` line each event of the channel. The errno of a failure is the error of the connection when it has one.
+- **The Kafka client image** (`image/kafka`) is kafka-clients 4.3.2 on Java 21, on the base image of the HikariCP client. The digest of the image and the hash of each jar (`jars.sha256`) pin them. A Kafka lab starts it (`Cell.kafka`), not a probe of a driver.
+  - `ClientKafka.java` writes the sequence numbers to partition 0 of the topic of the run, with the settings of the cell. It prints `sent`, `ack` and `fail` lines on the monotonic clock.
+  - `AdminKafka.java` makes a topic, gives a view of the partition, and reads the whole partition.
 - **The gRPC client image** (`image/grpc`) is grpc-js 1.14.6 on Node 24.21.0. The digest of the image and an npm lockfile pin them. The probe is `{ driver: { name: "grpc", hold } }`.
   - `client-grpc.mjs` opens one channel with no TLS. Each operation is a unary call of `grpc.health.v1.Health/Check`, with the messages encoded in the script. The first call makes the connection.
   - The options of the cell (`Cell.driverOptions`) are the channel options of grpc-js (`grpc.*`), and `deadline_ms` for the deadline of each call. With `hold`, the call in flight asks the server of the harness to hold its reply that long.
@@ -280,7 +295,7 @@ ioredis sets the keepalive of its socket itself, after it connects. Thus a cell
   - Without `inFlight`, the event comes first, and the request meets it. With `inFlight`, the request goes first, and the event comes while it is in flight. A run whose order is not that order is void.
 - **A fault on a group** acts on the part that `Cell.faultAt` names by its role:
   - A black hole (`blackhole`) drops each segment between the part and each peer. A reset (`rst`) answers each segment from a peer with a reset, thus the part refuses each connection.
-  - A partial partition (`partition`) is a black hole between the part and some of its peers (`GroupPlace.cuts`). `group` cuts the part from the other members and the Sentinels, and `clients` cuts it from the client.
+  - A partial partition (`partition`) is a black hole between the part and some of its peers (`GroupPlace.cuts`). `group` cuts the part from the other members and the Sentinels, and `clients` cuts it from the client. `to-clients` drops only the segments from the part to the client, thus each write reaches the part and no reply reaches the client.
   - The rules for each peer go into one commit. The injector takes a list of peers, `A,B,C`.
   - A frozen process (`freeze-process`) is SIGSTOP to the server of the part, and SIGCONT thaws it. A FIN (`fin`) is SIGKILL, and the server starts again after the run.
   - A planned failover (`planned-failover`) is `SENTINEL FAILOVER <group>` to the first Sentinel, or `SENTINEL FAILOVER <group> COORDINATED` (`Cell.coordinated`). The injector stamps the write of the command and its reply.
@@ -415,6 +430,7 @@ The distribution test needs a simulator, which does not exist at this time.
   - The manifest holds the kernel release, the architecture, the boot id and the Docker version. It also holds the image ids, the packages of the image, the client and the sysctls as read back.
   - Each image that the harness builds has a build record in the manifest (`BuildRecord`). It gives the tag, the bases by digest and the sha256 of each lockfile. It also gives the versions of the packages that the manifests of the image name. The harness, the client and the image of a subject have one. The harness record also has the version of the package.
   - `manifestProblems(manifest)` gives each image of a manifest that does not pin its programs. Such an image is a pulled image without a digest, or a build without its build record. `commitProblems(commit)` examines the commit of a record. A record without a pin for each program is not a record, because nobody can reproduce its numbers.
+  - `recordProblems(record)` gives each problem of a record: its commit, its manifests, a checkout with changes (`DIRTY`) and the mark of a debug run (`DEBUG`). The writer of the cells and the read of a schedule use this one function.
   - A cell whose probe names a `driver` opens its labs with the client of that driver (`openLab(target, "ioredis")`).
   - The harness sets the `targetSettings` of a cell in the target before a client connects, with the `settings` commands of the target (`redis-cli CONFIG SET`, then `CONFIG GET`).
   - A group cell opens a group environment (`openGroupLab`), and each run is `groupOnce`. Its warm-up is a whole run with its fault, because no fault of a group ends its environment. The manifest also holds the settings of each member and of each Sentinel, as each part read them.