@mark1russell7/kind 0.1.2 → 0.1.4

package/README.md

@@ -72,6 +72,14 @@ The tests read three real records of `/1` (`test/records/`): a cell that holds,
 `openKindLab(cluster, image)` (`lab.ts`) starts the instruments of a cluster:
 
 - **The agent of each node** is a privileged container in the PID namespace of the node container. It signals a process of a pod by its PID on the node. It writes `cpu.max` in the cgroup of a container, and a flag file in the root of a container.
+  - From 0.1.4, it also reads the conntrack table of the node each second. It writes a `stale` line for each entry to the ClusterIP that keeps its NAT to a pod that the rules dropped.
+  - It reads `/proc/net/nf_conntrack`, or else it asks netlink with the `conntrack` tool of the node. The kernel of a GitHub runner has no `/proc/net/nf_conntrack`. A dump through netlink walks each bucket of the table of the kernel with the softirqs off. A segment that comes in that walk waits. Thus each read is one dump, and `kind.yml` gives the table 16384 buckets, not 262144.
+  - A SYN on the port of such an entry goes to that pod, whatever the rules say. An entry in TIME_WAIT is not stale, because a SYN reopens it with a new lookup.
+  - It writes a `stalled` line for each handshake that stopped for one second or more: a SYN with no reply, or a SYN-ACK with no ACK. On the node of the client, the line gives the backend of the DNAT. On the node of that backend, the line tells which half of the handshake did not arrive.
+  - Each `rules-alive` line gives the source of the table (`ct`). It counts the reads since the last line, and the entries of the ClusterIP in each state. A read that fails stops the reads, and `ct.error` tells why. The lab drops the lines from before the first measurement, thus an early failure shows only in `ct.error`.
+  - `ct.syn` gives up to three entries in SYN_SENT, with the backend of their NAT and their life. A new SYN has the life 120 s, and an older one less.
+  - Each second, the agent reads the counters of dropped packets (`drops.ts`), and it writes a `drops` line with each counter that grew. For the node, these are the drops of the backlogs, the squeezes of the softirqs, and the drops of each veth.
+  - For each client pod, the agent reads `/proc/<pid>/net`. These counters are the IP discards, the sends with no route, the failed connects, the drops of its interfaces, and its neighbors with no address. A SYN that does not get to the conntrack of the node leaves its trace in one of them.
 - **The rules of each node.** The agent reads the rules of kube-proxy in the namespaces of the node each 50 ms: `iptables-save`, `nft list table ip kube-proxy`, or `/proc/net/ip_vs`. It writes a line when the backends of the Service change (`backends.ts`). A backend of IPVS with the weight zero takes no new connection, thus it is not a backend.
 - **The injector of each node** is the image of the harness, in the PID namespace of the node. `crictl` on the node gives the PID of a container, and `nsenter -n` enters the network namespace of its pod. The injector of the harness then applies its rules there.
 - **The watcher** is a container on the network of the cluster. It watches the pods, the EndpointSlices and the events through the API, and it follows the log of each pod of the cell.
@@ -79,6 +87,8 @@ The tests read three real records of `/1` (`test/records/`): a cell that holds,
 
 Each pod of a cluster shares the kernel of the Docker VM. Thus each stamp of the agent, the injector, the watcher, the clients, the service and the prober is on the same CLOCK_MONOTONIC.
 
+Before each run, the lab deletes the conntrack entries in SYN_SENT to the ClusterIP on each node (`forgetHandshakes`). A note of the run gives each count. Such an entry is a SYN with no answer, and it keeps its NAT for 120 s. A new SYN from the same port goes where the first one went, and it gets no answer either. The clients send requests between the runs too. Thus the restore of one run left such entries, and a client that took the port again in the next run timed out (T07, 2026-10-17).
+
 ## The image
 
 The lab image (`image/`) holds the service of a cell on `@mark1russell7/service`, its clients, the agent and the watcher, on Node 24.21.0. Its base is pinned by digest, and `npm-lock.json` pins the packages from npm. `stageLabImage()` stages its context, and `labImage()` tags the image by its content.
@@ -87,7 +97,7 @@ The lab image (`image/`) holds the service of a cell on `@mark1russell7/service`
 - **From npm**, the stage packs the installed versions of the same packages. The package holds the files of the image, and `src/compose.ts` and `src/backends.ts`, for that.
 - **The service** (`app.mjs`) is a Node HTTP server. Its one controller needs Redis when the cell has Redis. Its health server gives the probes, and it declares each stage of the shutdown. `compose.ts` composes the service from the declaration of the cell, and `lint()` reads the same composition.
 - **The seams** of the service are a flag file that fails its liveness check, and SIGUSR2. SIGUSR2 blocks its event loop for the time in a file.
-- **The clients** (`client.mjs`) send requests on new connections to the ClusterIP. Each request gives one line with its outcome: ok, 503, refused, reset, timeout or unreachable.
+- **The clients** (`client.mjs`) send requests on new connections to the ClusterIP. Each request gives one line with its outcome: ok, 503, refused, reset, timeout or unreachable. From 0.1.3, each line also gives the local port of its connection (`lp`), a refused one's too. A conntrack entry is per port. Thus the requests on the port of a failure show the connection before it, and the pod that it went to.
 
 ## The triggers
 

package/build/index.d.ts

@@ -1,4 +1,4 @@
-import { $ as SliceLine, $t as goDuration, A as TriggerV1, At as showBand, B as RunFacts, Bt as Replica, C as KindCellResult, Ct as LOG_LABEL, D as PredictionV1, Dt as bandOf, E as LoadSample, Et as SERVER, F as KindDecision, Ft as KubeletProbe, G as referenceOf, Gt as KUBE_PROXY_MODES, H as factsOf, Ht as Trigger, I as KindVerdict, It as PodSettings, J as Line, Jt as KubeProxyMode, K as unitOf, Kt as KindConfig, L as LAST_SLACK, Lt as Prediction, M as upgrade, Mt as Declaration, N as upgradeCell, Nt as KindCell, O as RunCheckV1, Ot as seconds, P as KIND_NOISE, Pt as KindRun, Q as RulesLine, Qt as cidrRange, R as Propagated, Rt as RedisDeclaration, S as KindCellRecordV1, St as HEALTH_PORT, T as KindVerdictV1, Tt as REDIS, U as judgeKind, Ut as TriggerKind, V as checkRun, Vt as Template, W as propagationOf, Wt as Workload, X as RecordLine, Xt as NodeImage, Y as PodLine, Yt as NODE_IMAGES, Z as Request, Zt as cidrOverlap, _ as ClusterManifest, _t as sliceDropOf, a as STAGES, at as failuresOf, b as FORMAT_V1, bt as transitionOf, c as readRecord, ct as outcomesOf, d as calibrationOf, dt as recordsOf, en as kindConfig, et as backendsAt, f as cellsTable, ft as requestsOf, g as BandV1, gt as signalsOf, h as timeSpread, ht as signalOf, i as quantileOf, in as scenarioProblems, it as emptyOf, j as judge, jt as Band, k as RunFactsV1, kt as secondsOf, l as Judged, lt as podsAt, m as predictionText, mt as rulesOf, n as medianInterval, nn as nodeNames, nt as deletionOf, o as kindCellRecordSchema, ot as intakeOf, p as medianText, pt as restartsAt, q as Endpoint, qt as KindScenario, r as middle, rn as scenario, rt as dropOf, s as kindCellRecordV1Schema, st as kernelCounts, t as RELATIVE, tn as kubeProxyPatch, tt as byNode, u as TimeSpread, ut as podsOf, v as EnvironmentManifest, vt as sliceEmptyOf, w as KindCellV1, wt as PORT, x as KindCellRecord, xt as CLIENT, y as FORMAT, yt as slicesOf, z as RunCheck, zt as Reference } from "./judge-CDDCr1Z8.js";
+import { $ as SliceLine, $t as goDuration, A as TriggerV1, At as showBand, B as RunFacts, Bt as Replica, C as KindCellResult, Ct as LOG_LABEL, D as PredictionV1, Dt as bandOf, E as LoadSample, Et as SERVER, F as KindDecision, Ft as KubeletProbe, G as referenceOf, Gt as KUBE_PROXY_MODES, H as factsOf, Ht as Trigger, I as KindVerdict, It as PodSettings, J as Line, Jt as KubeProxyMode, K as unitOf, Kt as KindConfig, L as LAST_SLACK, Lt as Prediction, M as upgrade, Mt as Declaration, N as upgradeCell, Nt as KindCell, O as RunCheckV1, Ot as seconds, P as KIND_NOISE, Pt as KindRun, Q as RulesLine, Qt as cidrRange, R as Propagated, Rt as RedisDeclaration, S as KindCellRecordV1, St as HEALTH_PORT, T as KindVerdictV1, Tt as REDIS, U as judgeKind, Ut as TriggerKind, V as checkRun, Vt as Template, W as propagationOf, Wt as Workload, X as RecordLine, Xt as NodeImage, Y as PodLine, Yt as NODE_IMAGES, Z as Request, Zt as cidrOverlap, _ as ClusterManifest, _t as sliceDropOf, a as STAGES, at as failuresOf, b as FORMAT_V1, bt as transitionOf, c as readRecord, ct as outcomesOf, d as calibrationOf, dt as recordsOf, en as kindConfig, et as backendsAt, f as cellsTable, ft as requestsOf, g as BandV1, gt as signalsOf, h as timeSpread, ht as signalOf, i as quantileOf, in as scenarioProblems, it as emptyOf, j as judge, jt as Band, k as RunFactsV1, kt as secondsOf, l as Judged, lt as podsAt, m as predictionText, mt as rulesOf, n as medianInterval, nn as nodeNames, nt as deletionOf, o as kindCellRecordSchema, ot as intakeOf, p as medianText, pt as restartsAt, q as Endpoint, qt as KindScenario, r as middle, rn as scenario, rt as dropOf, s as kindCellRecordV1Schema, st as kernelCounts, t as RELATIVE, tn as kubeProxyPatch, tt as byNode, u as TimeSpread, ut as podsOf, v as EnvironmentManifest, vt as sliceEmptyOf, w as KindCellV1, wt as PORT, x as KindCellRecord, xt as CLIENT, y as FORMAT, yt as slicesOf, z as RunCheck, zt as Reference } from "./judge-CKhPqBku.js";
 import { Quantity } from "@mark1russell7/quantity";
 import { CheckFn, Finding, HealthServer, Poll, Rule, Scenario, ServiceBuilder, ServiceDescription, Stage, Timing, TransportImpl, Verdict } from "@mark1russell7/service";
 import { RedisClient } from "@mark1russell7/service/redis";
@@ -186,6 +186,12 @@ interface KindLab {
   netnsOf(node: string, pid: number): Promise<number>;
   /** This method gives the path of the cgroup of a process, as the agent reaches it through the root of the node. */
   cgroupOf(node: string, pid: number): Promise<string>;
+  /**
+   * This method deletes the conntrack entries in SYN_SENT to a ClusterIP and port on each node, and
+   * gives the count of each node. Such an entry is a SYN with no answer. It keeps its NAT for 120 s,
+   * and a new SYN from the same port goes where it went.
+   */
+  forgetHandshakes(ip: string, port: number): Promise<Record<string, number>>;
   close(): Promise<void>;
 }
 /**

package/build/index.js

@@ -803,7 +803,9 @@ const FILES = [
 	["agent.mjs", "image/agent.mjs"],
 	["watch.mjs", "image/watch.mjs"],
 	["compose.ts", "src/compose.ts"],
-	["backends.ts", "src/backends.ts"]
+	["backends.ts", "src/backends.ts"],
+	["conntrack.ts", "src/conntrack.ts"],
+	["drops.ts", "src/drops.ts"]
 ];
 const shell = process.platform === "win32";
 /** This function gives each file of a staged context, with its path in the context, in a stable sequence. */
@@ -897,6 +899,14 @@ const labImage = () => {
 	return built;
 };
 //#endregion
+//#region src/conntrack.ts
+/** This function gives the count of entries that `conntrack -D` deleted, from its last line: "N flow entries have been deleted". */
+const deletedOf = (text) => {
+	const match = /(\d+) flow entr(?:y|ies) ha(?:s|ve) been deleted/.exec(text);
+	if (match === null) throw new Error(`conntrack -D gave no count: ${text.trim().slice(0, 200)}`);
+	return Number(match[1]);
+};
+//#endregion
 //#region src/lab.ts
 /** These kinds of kernel record stay in the raw record of a run. The lab counts the other kinds and drops them. */
 const KEPT_KERNEL = /* @__PURE__ */ new Set([
@@ -1070,6 +1080,33 @@ const openKindLab = async (cluster, image) => {
 				`printf '%s' '${file}' > /tmp/rules.json`
 			]);
 		},
+		async forgetHandshakes(ip, port) {
+			const out = {};
+			for (const [node, { agent }] of sidecars) {
+				const argv = [
+					"nsenter",
+					"-t",
+					"1",
+					"-m",
+					"-n",
+					"--",
+					"conntrack",
+					"-D",
+					"-p",
+					"tcp",
+					"--orig-dst",
+					ip,
+					"--dport",
+					String(port),
+					"--state",
+					"SYN_SENT"
+				];
+				const result = await agent.exec(argv);
+				if (result.exitCode !== 0 && result.exitCode !== 1) throw new Error(`${argv.join(" ")} exited ${result.exitCode}: ${result.output.trim().slice(0, 300)}`);
+				out[node] = deletedOf(result.output);
+			}
+			return out;
+		},
 		async act(node, spec) {
 			const stdout = await output(sidecarOf(node).agent, [
 				"node",
@@ -2034,6 +2071,7 @@ const runOnce = async (ctx, env, index, warmup, ours) => {
 		"rollout"
 	].includes(cell.trigger.kind) ? null : replicas[0];
 	const paused = cell.template === "T07" && cell.trigger.kind === "freeze-process" ? cluster.nodes[1] ?? cluster.nodes[0] : null;
+	const cleared = await lab.forgetHandshakes(ctx.clusterIp, 80);
 	lab.take();
 	for (const key of Object.keys(lab.dropped)) delete lab.dropped[key];
 	const start = await lab.now();
@@ -2042,6 +2080,8 @@ const runOnce = async (ctx, env, index, warmup, ours) => {
 	let resume = null;
 	if (cell.trigger.kind === "scale-in") ctx.expected = cell.pod.replicas - 1;
 	const notes = [];
+	const forgotten = Object.entries(cleared).filter(([, count]) => count > 0);
+	if (forgotten.length > 0) notes.push(`before the run, the lab deleted SYN_SENT entries of the ClusterIP: ${forgotten.map(([node, count]) => `${count} on ${node}`).join(", ")}`);
 	for (const step of plan.inject) {
 		const stamp = await doStep(ctx, step, replicas, background);
 		if (stamp !== null && fault === null) fault = stamp.t0;