@mark1russell7/kind 0.10.0 → 0.11.0

package/README.md

@@ -148,7 +148,12 @@ A new member of the union `Template` breaks a reader that switches on it. That i
 
 ## The image
 
-The lab image (`image/`) holds the service of a cell on `@mark1russell7/service`, its clients, the agent and the watcher, on Node 24.21.0. Its base is pinned by digest, and `npm-lock.json` pins the packages from npm. `stageLabImage()` stages its context, and `labImage()` tags the image by its content.
+The lab image (`image/`) holds the service of a cell on `@mark1russell7/service`, its clients, the agent and the watcher, on Node 24.21.0. Its base is pinned by digest, and `npm-lock.json` pins the packages from npm. `stageLabImage()` stages its context, and `labImage()` gives the tag of the image.
+
+- **The keep-alive clients.** A workload with `connections` gives each HTTP client that count of keep-alive connections. Each request goes on the next connection in turn. kube-proxy selects a backend only for a new connection. SIGHUP gives the client new connections, and its old connections close after the timeout. With no `connections`, each request has a new connection, as before.
+- **The lame duck.** The declaration has the optional field `lameDuck`. With true or no value, the edge sends `Connection: close` while the service is not ready (`httpTransport` as it is). With false, the app gives the transport `keepAlive: () => true`, thus each connection stays open and a caller learns only from the endpoints.
+
+- **The resolver.** From 0.10.1, the lab image is a `build` source of the harness (`labSource()`). The resolver of the harness plans it and builds it, as it does each other image (`planOf`, `imageOf`). Thus its tag comes from its content, and each `FROM` has a digest. Each manifest of packages has a lockfile, and a failed build tries again after each wait. The build goes through the CLI of Docker, not through Testcontainers.
 
 - **In the workspace** of this repository, the stage builds `@mark1russell7/quantity`, `@mark1russell7/algebra` and `@mark1russell7/service`, and packs each one as npm gets it.
 - **From npm**, the stage packs the installed versions of the same packages. The package holds the files of the image, and `src/compose.ts` and `src/backends.ts`, for that.
@@ -184,6 +189,7 @@ The lab image (`image/`) holds the service of a cell on `@mark1russell7/service`
 | `authorization-policy` | `kubectl apply` of a DENY AuthorizationPolicy on the server pods for the principal of an account (`account`, the account `default` of the clients when not given), then `kubectl delete` after the hold |
 | `idle` | SIGUSR1 to the gRPC client of each client pod, and SIGUSR2 after the hold: a gap of the calls (the target `clients`). The restore sends SIGUSR2 again. With `middlebox`, a stateful middlebox in each replica from before the gap to the restore. With `close`, a new channel at the fault for 5 s: SIGHUP before the fault, then SIGUSR2, a wait and SIGHUP |
 | `ztunnel-restart` | `kubectl delete pod` of the ztunnel pod of each node, with its grace (`ZTUNNEL_PODS`). The flow of each client crosses the ztunnel of its own node. The restore waits for the rollout of the DaemonSet |
+| `cannot-serve` | SIGHUP to each client before the fault, thus the keep-alive connections spread over the replicas again in each run (`SPREAD_LEAD`, 3 s). Then the flag file of the first replica for the hold. While the flag is there, the route of the replica fails with 503, and the flag check of the declaration fails. The heal and the restore remove the flag |
 | `certificate-expiry` | No act on a connection: the calls of each client go on for the hold. At the fault and at the heal, istioctl reads the leaf certificates of the ztunnel pod of each node (`ztunnelCertificates`). A line `certificates` keeps their validity, the evidence of their lifetime |
 
 A cgroup path that is not the scope of one container is refused. A write to the root of the node throttles or freezes each pod of the node.
@@ -207,6 +213,7 @@ The plan's `event:clock-jump` has no trigger. A step of CLOCK_REALTIME in the VM
 - **A reset from a leaving pod** leaves the conntrack entry of its flow in CLOSE for 10 s, with the NAT to that pod. From 0.1.6, a window can give that life (`afterReset`). Then a request outside the window can fail with the outcome of the rule (`reusesReset`). The condition is a reset of the same client on the same port, up to that life before the request. The catalog gives it to a deletion without a hold, in the modes with the NAT in conntrack.
 - **A fault of a shared dependency** trips the readiness latch of each replica (`failSpan`). Then the kubelet needs its threshold of probes, and the rules need one propagation.
 - **A restart through a coupling** adds the liveness probes of the kubelet and the preStop sleep to the latch.
+- **A replica that cannot serve** (`learnWindow`) fails each request on a connection to it from the fault. A turn is the time between two requests on one connection. With the lame duck, the latch closes each connection, and the rules select the replica until its removal. Thus the window ends from the fault, in the latch to the removal and one turn. Without the lame duck, no connection opens again, thus the removal changes nothing. The window ends from the heal, in the recovery of the readiness and one turn.
 
 `verdict.ts` judges each run, and each cell on its counted runs. Each time lands in its band ± 100 ms: the poll of the rules and the delay of the watcher. A median in a band gets the test of the harness: its order-statistic interval in the band, made 5 % wider.
 
@@ -254,6 +261,8 @@ From 0.8.0, a prediction `ends` can be the end of the tunnel of each client (`of
 - The first request of each client after the heal must be `ok`, on a new tunnel. The main value is the middle of the ends, in `s`.
 - From 0.8.1, `ends` has the optional band `answer`. A change to CLOSE_WAIT in that band ± δ also agrees: the peer answers the close of the client, and its FIN comes first. Before the band, the peer closed first.
 
+A new member of the union `TriggerKind`, `cannot-serve`, breaks a reader that switches on it. That is a breaking change before 1.0.0, thus the next version is 0.11.0. The record format stays `/2`, with the optional fields `connections` of a workload and `lameDuck` of a declaration. The line `client` of the reconnect is a new line of a run.
+
 A new member of the union `TriggerKind`, `certificate-expiry`, and a new member of the union `Step`, `certificates`, break a reader that switches on them. `KindLab` also has a new method, `record`. Each is a breaking change before 1.0.0, thus the version is 0.9.0. The record format stays `/2`: the lines `certificates` are new lines of a run.
 
 In 0.8.1, the middlebox uses the match recent, not the sysctls of conntrack, and `ends` has the optional field `answer`. `SYSCTL` and `NETNS_DEFAULTS` keep only `tcp_retries2`. These fields came in 0.8.0, which was not published. The fix and the optional field are a patch, thus the version is 0.8.1.

package/build/index.d.ts

@@ -1,6 +1,7 @@
-import { $ as updatesOf, $t as Declaration, A as TriggerV1, An as scenario, At as rulesOf, B as RunCheck, Bt as ztunnelPodsOf, C as KindCellResult, Cn as NodeImage, Ct as outcomesOf, D as PredictionV1, Dn as kindConfig, Dt as requestsOf, E as LoadSample, En as goDuration, Et as recordsOf, F as KIND_NOISE, Ft as slicesOf, G as judgeKind, Gt as PORT, H as checkRun, Ht as CLIENT, I as KindDecision, It as socketsOf, J as propagationOf, Jt as bandOf, K as observationAfter, Kt as REDIS, L as KindVerdict, Lt as stoppingOf, M as upgrade, Mt as signalsOf, N as upgradeCell, Nt as sliceDropOf, O as RunCheckV1, On as kubeProxyPatch, Ot as restartOf, P as Deleted, Pt as sliceEmptyOf, Q as unitOf, Qt as Band, R as LAST_SLACK, Rt as transitionOf, S as KindCellRecordV1, Sn as NODE_IMAGES, St as kernelCounts, T as KindVerdictV1, Tn as cidrRange, Tt as podsOf, U as deletionsOf, Ut as HEALTH_PORT, V as RunFacts, Vt as ztunnelReplaced, W as factsOf, Wt as LOG_LABEL, X as referenceOf, Xt as secondsOf, Y as propagationSource, Yt as seconds, Z as reusesReset, Zt as showBand, _ as ClusterManifest, _n as Workload, _t as dropOf, a as STAGES, an as MeshMode, at as RecordLine, b as FORMAT_V1, bn as KindScenario, bt as intakeOf, c as readRecord, cn as RedisDeclaration, ct as SliceLine, d as calibrationOf, dn as ResetReuse, dt as TunnelLine, en as KindCell, et as Endpoint, f as cellsTable, fn as Stack, ft as ZtunnelPodLine, g as BandV1, gn as WindowPrediction, gt as deletionOf, h as timeSpread, hn as TriggerKind, ht as byNode, i as quantileOf, in as MeshFlavor, it as PropagationSource, j as judge, jn as scenarioProblems, jt as signalOf, k as RunFactsV1, kn as nodeNames, kt as restartsAt, l as Judged, ln as Reference, lt as SocketLine, m as predictionText, mn as Trigger, mt as byClients, n as medianInterval, nn as KubeletProbe, nt as Line, o as kindCellRecordSchema, on as PodSettings, ot as Request, p as medianText, pn as Template, pt as backendsAt, q as observationEnd, qt as SERVER, r as middle, rn as Mesh, rt as PodLine, s as kindCellRecordV1Schema, sn as Prediction, st as RulesLine, t as RELATIVE, tn as KindRun, tt as HBONE_PORT, u as TimeSpread, un as Replica, ut as TCP_STATE, v as EnvironmentManifest, vn as KUBE_PROXY_MODES, vt as emptyOf, w as KindCellV1, wn as cidrOverlap, wt as podsAt, x as KindCellRecord, xn as KubeProxyMode, xt as isTunnelState, y as FORMAT, yn as KindConfig, yt as failuresOf, z as Propagated, zt as tunnelsOf } from "./judge-BD4UVQfy.js";
+import { $ as updatesOf, $t as Declaration, A as TriggerV1, An as scenario, At as rulesOf, B as RunCheck, Bt as ztunnelPodsOf, C as KindCellResult, Cn as NodeImage, Ct as outcomesOf, D as PredictionV1, Dn as kindConfig, Dt as requestsOf, E as LoadSample, En as goDuration, Et as recordsOf, F as KIND_NOISE, Ft as slicesOf, G as judgeKind, Gt as PORT, H as checkRun, Ht as CLIENT, I as KindDecision, It as socketsOf, J as propagationOf, Jt as bandOf, K as observationAfter, Kt as REDIS, L as KindVerdict, Lt as stoppingOf, M as upgrade, Mt as signalsOf, N as upgradeCell, Nt as sliceDropOf, O as RunCheckV1, On as kubeProxyPatch, Ot as restartOf, P as Deleted, Pt as sliceEmptyOf, Q as unitOf, Qt as Band, R as LAST_SLACK, Rt as transitionOf, S as KindCellRecordV1, Sn as NODE_IMAGES, St as kernelCounts, T as KindVerdictV1, Tn as cidrRange, Tt as podsOf, U as deletionsOf, Ut as HEALTH_PORT, V as RunFacts, Vt as ztunnelReplaced, W as factsOf, Wt as LOG_LABEL, X as referenceOf, Xt as secondsOf, Y as propagationSource, Yt as seconds, Z as reusesReset, Zt as showBand, _ as ClusterManifest, _n as Workload, _t as dropOf, a as STAGES, an as MeshMode, at as RecordLine, b as FORMAT_V1, bn as KindScenario, bt as intakeOf, c as readRecord, cn as RedisDeclaration, ct as SliceLine, d as calibrationOf, dn as ResetReuse, dt as TunnelLine, en as KindCell, et as Endpoint, f as cellsTable, fn as Stack, ft as ZtunnelPodLine, g as BandV1, gn as WindowPrediction, gt as deletionOf, h as timeSpread, hn as TriggerKind, ht as byNode, i as quantileOf, in as MeshFlavor, it as PropagationSource, j as judge, jn as scenarioProblems, jt as signalOf, k as RunFactsV1, kn as nodeNames, kt as restartsAt, l as Judged, ln as Reference, lt as SocketLine, m as predictionText, mn as Trigger, mt as byClients, n as medianInterval, nn as KubeletProbe, nt as Line, o as kindCellRecordSchema, on as PodSettings, ot as Request, p as medianText, pn as Template, pt as backendsAt, q as observationEnd, qt as SERVER, r as middle, rn as Mesh, rt as PodLine, s as kindCellRecordV1Schema, sn as Prediction, st as RulesLine, t as RELATIVE, tn as KindRun, tt as HBONE_PORT, u as TimeSpread, un as Replica, ut as TCP_STATE, v as EnvironmentManifest, vn as KUBE_PROXY_MODES, vt as emptyOf, w as KindCellV1, wn as cidrOverlap, wt as podsAt, x as KindCellRecord, xn as KubeProxyMode, xt as isTunnelState, y as FORMAT, yn as KindConfig, yt as failuresOf, z as Propagated, zt as tunnelsOf } from "./judge-4bzoeGIb.js";
 import { Quantity } from "@mark1russell7/quantity";
 import { CheckFn, Finding, HealthServer, Poll, Rule, Scenario, ServiceBuilder, ServiceDescription, Stage, StopWay, Timing, TransportImpl, Verdict } from "@mark1russell7/service";
+import { BuildImage } from "@mark1russell7/harness";
 import { RedisClient } from "@mark1russell7/service/redis";
 //#region src/backends.d.ts
 /** A service rule names the Service whose backends the agent reads: its namespace, its name, and its ClusterIP and port for IPVS. */
@@ -223,6 +224,11 @@ export declare const QUIET_LEAD: Quantity;
 export declare const CLOSE_LEAD: Quantity;
 /** This time is how long each client calls on its new channel at the fault of a gap with `close`, before it closes it. */
 export declare const CHANNEL_USE: Quantity;
+/**
+ * This time is between the SIGHUP of `cannot-serve` and its fault. Each client opens new connections
+ * at the signal, and the requests on its old connections end in this time (the timeout and 100 ms).
+ */
+export declare const SPREAD_LEAD: Quantity;
 /**
  * This function gives the command of the injector that sets one sysctl of its network namespace.
  * Only a sysctl of the namespace (`net.`) and a whole number reach it.
@@ -266,9 +272,16 @@ export declare const deletes: (cell: KindCell) => boolean;
  */
 export declare const stageLabImage: () => string;
 /**
- * This function builds the lab image one time for each process, and gives its tag. Docker keeps its
- * layers between builds. A build that fails leaves the cache. Thus a later call builds again after a
- * fault of Docker or of the network.
+ * This function gives the source of the lab image from a staged context, as a `build` source of
+ * the harness. The resolver of the harness plans it and builds it, as it does each other image.
+ * Thus each `FROM` has a digest, and each manifest of packages has a lockfile.
+ */
+export declare const labSource: (context: string) => BuildImage;
+/**
+ * This function gives the tag of the lab image, one time for each process. It stages the context,
+ * and the resolver of the harness builds it with a tag from its content (`imageOf`). The stage
+ * builds and packs packages, thus a tag that passed stays. A stage or a build that fails leaves
+ * the cache. Thus a later call stages and builds again after a fault of Docker or of the network.
  */
 export declare const labImage: () => Promise<string>;
 //#endregion
@@ -631,6 +644,32 @@ export declare const deletionWindow: (declaration: Declaration, preStop: Quantit
 export declare const pausedWindow: (declaration: Declaration, preStop: Quantity, lead: Quantity, hold: Quantity, propagation: Propagation) => DeletionWindow & {
   readonly resume: Quantity;
 };
+/**
+ * A learn window is what `cannot-serve` predicts: when the callers stop getting errors from a
+ * replica that cannot serve. The window starts at the fault, and its end comes from `end.from`.
+ */
+interface LearnWindow {
+  /** This time is the interval between two requests on one connection of a client: `every` times the connections. */
+  readonly turn: Quantity;
+  /** This band is the trip of the readiness latch of the service after the fault. */
+  readonly latch: Band;
+  /** This band is the first failed request after the fault: within one turn. */
+  readonly start: Band;
+  readonly end: {
+    readonly from: Reference;
+    readonly band: Band;
+  };
+}
+/**
+ * This function gives the learn window of a declaration, its readiness probe, a propagation and a
+ * workload of keep-alive connections. A caller selects a replica only when it opens a connection:
+ * - The lame duck (`lameDuck` true or no value): after the latch, each answer has `Connection: close`,
+ *   thus each connection of the replica gets one more failure and opens again. The rules select the
+ *   replica until the removal, thus the last failure comes in one turn after it.
+ * - No lame duck: no connection opens again, thus the rules do not matter. The failures end only
+ *   when the readiness of the service recovers after the heal, and one turn after it.
+ */
+export declare const learnWindow: (declaration: Declaration, readiness: KubeletProbe, propagation: Propagation, workload: Workload) => LearnWindow;
 //#endregion
 //#region src/rules.d.ts
 /**
@@ -770,5 +809,5 @@ export declare const toolPath: (tool: Tool, key?: string, cache?: string) => str
  */
 export declare const ensureTool: (tool: Tool, key?: string, cache?: string) => Promise<string>;
 //#endregion
-export { type Band, type BandV1, CLIENT, type Cluster, type ClusterHosts, type ClusterManifest, type Credentials, type Declaration, type Deleted, type DeletionWindow, type EdsSpec, Endpoint, type EnvironmentManifest, FORMAT, FORMAT_V1, type FaultPlan, HBONE_PORT, HEALTH_PORT, type Judged, KIND_NOISE, KUBE_PROXY_MODES, type KindCell, type KindCellRecord, type KindCellRecordV1, type KindCellResult, type KindCellV1, type KindConfig, type KindDecision, type KindLab, type KindRun, type KindScenario, type KindVerdict, type KindVerdictV1, type KubeProxyMode, type KubeletProbe, LAST_SLACK, LOG_LABEL, type LdsSpec, type LeafCertificate, Line, type ListenerCounts, type LoadSample, type Mesh, type MeshFlavor, type MeshImages, type MeshMode, NODE_IMAGES, type NodeImage, PORT, type Parts, PodLine, type PodSettings, type Prediction, type PredictionV1, type Propagated, type Propagation, PropagationSource, REDIS, RELATIVE, RecordLine, type RedisDeclaration, type Reference, type Replica, Request, type ResetReuse, RulesLine, type RunCheck, type RunCheckV1, type RunFacts, type RunFactsV1, type RunOptions, SERVER, STAGES, type ServiceRule, SliceLine, SocketLine, type Stack, type Stamp, type Step, TCP_STATE, type Target, type Template, type TimeSpread, type Tool, type Trigger, type TriggerKind, type TriggerV1, TunnelLine, type Verdicts, type WindowPrediction, type Workload, ZtunnelPodLine, backendsAt, bandOf, byClients, byNode, calibrationOf, cellsTable, checkRun, cidrOverlap, cidrRange, deletionOf, deletionsOf, dropOf, emptyOf, factsOf, failuresOf, goDuration, intakeOf, isTunnelState, judge, judgeKind, kernelCounts, kindCellRecordSchema, kindCellRecordV1Schema, kindConfig, kubeProxyPatch, medianInterval, medianText, middle, nodeNames, observationAfter, observationEnd, outcomesOf, podsAt, podsOf, predictionText, propagationOf, propagationSource, quantileOf, readRecord, recordsOf, referenceOf, requestsOf, restartOf, restartsAt, reusesReset, rulesOf, scenario, scenarioProblems, seconds, secondsOf, showBand, signalOf, signalsOf, sliceDropOf, sliceEmptyOf, slicesOf, socketsOf, stoppingOf, timeSpread, transitionOf, tunnelsOf, unitOf, updatesOf, upgrade, upgradeCell, ztunnelPodsOf, ztunnelReplaced };
+export { type Band, type BandV1, CLIENT, type Cluster, type ClusterHosts, type ClusterManifest, type Credentials, type Declaration, type Deleted, type DeletionWindow, type EdsSpec, Endpoint, type EnvironmentManifest, FORMAT, FORMAT_V1, type FaultPlan, HBONE_PORT, HEALTH_PORT, type Judged, KIND_NOISE, KUBE_PROXY_MODES, type KindCell, type KindCellRecord, type KindCellRecordV1, type KindCellResult, type KindCellV1, type KindConfig, type KindDecision, type KindLab, type KindRun, type KindScenario, type KindVerdict, type KindVerdictV1, type KubeProxyMode, type KubeletProbe, LAST_SLACK, LOG_LABEL, type LdsSpec, type LeafCertificate, type LearnWindow, Line, type ListenerCounts, type LoadSample, type Mesh, type MeshFlavor, type MeshImages, type MeshMode, NODE_IMAGES, type NodeImage, PORT, type Parts, PodLine, type PodSettings, type Prediction, type PredictionV1, type Propagated, type Propagation, PropagationSource, REDIS, RELATIVE, RecordLine, type RedisDeclaration, type Reference, type Replica, Request, type ResetReuse, RulesLine, type RunCheck, type RunCheckV1, type RunFacts, type RunFactsV1, type RunOptions, SERVER, STAGES, type ServiceRule, SliceLine, SocketLine, type Stack, type Stamp, type Step, TCP_STATE, type Target, type Template, type TimeSpread, type Tool, type Trigger, type TriggerKind, type TriggerV1, TunnelLine, type Verdicts, type WindowPrediction, type Workload, ZtunnelPodLine, backendsAt, bandOf, byClients, byNode, calibrationOf, cellsTable, checkRun, cidrOverlap, cidrRange, deletionOf, deletionsOf, dropOf, emptyOf, factsOf, failuresOf, goDuration, intakeOf, isTunnelState, judge, judgeKind, kernelCounts, kindCellRecordSchema, kindCellRecordV1Schema, kindConfig, kubeProxyPatch, medianInterval, medianText, middle, nodeNames, observationAfter, observationEnd, outcomesOf, podsAt, podsOf, predictionText, propagationOf, propagationSource, quantileOf, readRecord, recordsOf, referenceOf, requestsOf, restartOf, restartsAt, reusesReset, rulesOf, scenario, scenarioProblems, seconds, secondsOf, showBand, signalOf, signalsOf, sliceDropOf, sliceEmptyOf, slicesOf, socketsOf, stoppingOf, timeSpread, transitionOf, tunnelsOf, unitOf, updatesOf, upgrade, upgradeCell, ztunnelPodsOf, ztunnelReplaced };
 //# sourceMappingURL=index.d.ts.map

package/build/index.js

@@ -1,13 +1,13 @@
-import { $ as signalOf, A as medianInterval, B as emptyOf, C as propagationOf, D as unitOf, E as reusesReset, F as backendsAt, G as outcomesOf, H as intakeOf, I as byClients, J as recordsOf, K as podsAt, L as byNode, M as quantileOf, N as HBONE_PORT, O as updatesOf, P as TCP_STATE, Q as rulesOf, R as deletionOf, S as observationEnd, T as referenceOf, U as isTunnelState, V as failuresOf, W as kernelCounts, X as restartOf, Y as requestsOf, Z as restartsAt, _ as checkRun, _t as seconds, a as calibrationOf, at as stoppingOf, b as judgeKind, c as predictionText, ct as ztunnelPodsOf, d as FORMAT_V1, dt as HEALTH_PORT, et as signalsOf, f as judge, ft as LOG_LABEL, g as LAST_SLACK, gt as bandOf, h as KIND_NOISE, ht as SERVER, i as readRecord, it as socketsOf, j as middle, k as RELATIVE, l as timeSpread, lt as ztunnelReplaced, m as upgradeCell, mt as REDIS, n as kindCellRecordSchema, nt as sliceEmptyOf, o as cellsTable, ot as transitionOf, p as upgrade, pt as PORT, q as podsOf, r as kindCellRecordV1Schema, rt as slicesOf, s as medianText, st as tunnelsOf, t as STAGES, tt as sliceDropOf, u as FORMAT, ut as CLIENT, v as deletionsOf, vt as secondsOf, w as propagationSource, x as observationAfter, y as factsOf, yt as showBand, z as dropOf } from "./judge-BbLWVTYj.js";
+import { $ as signalOf, A as medianInterval, B as emptyOf, C as propagationOf, D as unitOf, E as reusesReset, F as backendsAt, G as outcomesOf, H as intakeOf, I as byClients, J as recordsOf, K as podsAt, L as byNode, M as quantileOf, N as HBONE_PORT, O as updatesOf, P as TCP_STATE, Q as rulesOf, R as deletionOf, S as observationEnd, T as referenceOf, U as isTunnelState, V as failuresOf, W as kernelCounts, X as restartOf, Y as requestsOf, Z as restartsAt, _ as checkRun, _t as seconds, a as calibrationOf, at as stoppingOf, b as judgeKind, c as predictionText, ct as ztunnelPodsOf, d as FORMAT_V1, dt as HEALTH_PORT, et as signalsOf, f as judge, ft as LOG_LABEL, g as LAST_SLACK, gt as bandOf, h as KIND_NOISE, ht as SERVER, i as readRecord, it as socketsOf, j as middle, k as RELATIVE, l as timeSpread, lt as ztunnelReplaced, m as upgradeCell, mt as REDIS, n as kindCellRecordSchema, nt as sliceEmptyOf, o as cellsTable, ot as transitionOf, p as upgrade, pt as PORT, q as podsOf, r as kindCellRecordV1Schema, rt as slicesOf, s as medianText, st as tunnelsOf, t as STAGES, tt as sliceDropOf, u as FORMAT, ut as CLIENT, v as deletionsOf, vt as secondsOf, w as propagationSource, x as observationAfter, y as factsOf, yt as showBand, z as dropOf } from "./judge-DdeJ1qVd.js";
 import { valueIn } from "@mark1russell7/quantity";
 import { execFile, execFileSync } from "node:child_process";
-import { chmodSync, copyFileSync, existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, realpathSync, renameSync, rmSync, writeFileSync } from "node:fs";
+import { chmodSync, copyFileSync, existsSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, renameSync, rmSync, writeFileSync } from "node:fs";
 import { dirname, join } from "node:path";
 import { promisify } from "node:util";
 import { createHash } from "node:crypto";
 import { fileURLToPath } from "node:url";
 import { gunzipSync } from "node:zlib";
-import { failSpan, lint, lintStop, service } from "@mark1russell7/service";
+import { failSpan, lint, lintStop, recoverSpan, service } from "@mark1russell7/service";
 import { createRequire } from "node:module";
 import { tmpdir } from "node:os";
 import { harnessImage, imageOf } from "@mark1russell7/harness";
@@ -691,6 +691,14 @@ const CHANNEL_USE = {
 	value: 5,
 	unit: "s"
 };
+/**
+* This time is between the SIGHUP of `cannot-serve` and its fault. Each client opens new connections
+* at the signal, and the requests on its old connections end in this time (the timeout and 100 ms).
+*/
+const SPREAD_LEAD = {
+	value: 3,
+	unit: "s"
+};
 const IPV4 = /^(25[0-5]|2[0-4]\d|1?\d?\d)(\.(25[0-5]|2[0-4]\d|1?\d?\d)){3}$/;
 /**
 * This function gives the command of the injector that sets one sysctl of its network namespace.
@@ -1248,6 +1256,32 @@ const planOf = (cell) => {
 			heal: [{ do: "certificates" }],
 			restore: []
 		};
+		case "cannot-serve": return {
+			prepare: [{
+				do: "signal",
+				target: "clients",
+				signal: "SIGHUP"
+			}, {
+				do: "wait",
+				time: SPREAD_LEAD
+			}],
+			inject: [{
+				do: "flag",
+				target: "server",
+				on: true
+			}],
+			hold: needs(trigger.hold, "a hold, the time that the replica cannot serve"),
+			heal: [{
+				do: "flag",
+				target: "server",
+				on: false
+			}],
+			restore: [{
+				do: "flag",
+				target: "server",
+				on: false
+			}]
+		};
 	}
 };
 /** This function tells if a trigger stops a pod of the service with its grace: a deletion, an eviction, a scale-in or a rollout. */
@@ -1269,6 +1303,8 @@ const WORKSPACE_PACKAGES = [
 	"algebra",
 	"service"
 ];
+/** This folder of a staged context holds the tarball of each package of the workspace. */
+const PACKED = "workspace";
 /** These are the files of the image, by their name in the context and their source in the package. */
 const FILES = [
 	["Dockerfile", "image/Dockerfile"],
@@ -1288,17 +1324,6 @@ const FILES = [
 	["client-lines.ts", "src/client-lines.ts"]
 ];
 const shell = process.platform === "win32";
-/** This function gives each file of a staged context, with its path in the context, in a stable sequence. */
-const filesOf = (root, at = "") => readdirSync(join(root, at), { withFileTypes: true }).flatMap((entry) => entry.isDirectory() ? filesOf(root, join(at, entry.name)) : [join(at, entry.name)]).sort();
-/** This function gives the tag of a staged context from its content: the name, then 12 digits of a sha256. */
-const contentTag = (name, root) => {
-	const hash = createHash("sha256");
-	for (const file of filesOf(root)) {
-		hash.update(file.replaceAll("\\", "/"));
-		hash.update(readFileSync(join(root, file)));
-	}
-	return `${name}:${hash.digest("hex").slice(0, 12)}`;
-};
 /** This function tells if the package is in the workspace of its repository. From npm, it is not. */
 const inWorkspace = () => existsSync(join(REPO, "pnpm-workspace.yaml")) && WORKSPACE_PACKAGES.every((name) => existsSync(join(REPO, "packages", name, "package.json")));
 /**
@@ -1328,7 +1353,7 @@ const installedFolders = () => {
 */
 const stageLabImage = () => {
 	const context = mkdtempSync(join(tmpdir(), "systems-kind-lab-"));
-	const packed = join(context, "workspace");
+	const packed = join(context, PACKED);
 	mkdirSync(packed);
 	if (inWorkspace()) {
 		const filters = WORKSPACE_PACKAGES.flatMap((name) => ["--filter", `@mark1russell7/${name}`]);
@@ -1359,21 +1384,30 @@ const stageLabImage = () => {
 	for (const [at, from] of FILES) copyFileSync(join(PACKAGE, from), join(context, at));
 	return context;
 };
+/**
+* This function gives the source of the lab image from a staged context, as a `build` source of
+* the harness. The resolver of the harness plans it and builds it, as it does each other image.
+* Thus each `FROM` has a digest, and each manifest of packages has a lockfile.
+*/
+const labSource = (context) => ({
+	kind: "build",
+	name: "systems-kind-lab",
+	context,
+	files: [...FILES.map(([at]) => at), PACKED]
+});
 let built = null;
 /**
-* This function builds the lab image one time for each process, and gives its tag. Docker keeps its
-* layers between builds. A build that fails leaves the cache. Thus a later call builds again after a
-* fault of Docker or of the network.
+* This function gives the tag of the lab image, one time for each process. It stages the context,
+* and the resolver of the harness builds it with a tag from its content (`imageOf`). The stage
+* builds and packs packages, thus a tag that passed stays. A stage or a build that fails leaves
+* the cache. Thus a later call stages and builds again after a fault of Docker or of the network.
 */
 const labImage = () => {
 	if (built !== null) return built;
 	const building = (async () => {
 		const context = stageLabImage();
 		try {
-			const tag = contentTag("systems-kind-lab", context);
-			const { GenericContainer } = await import("testcontainers");
-			await GenericContainer.fromDockerfile(context).build(tag, { deleteOnExit: false });
-			return tag;
+			return await imageOf(labSource(context));
 		} finally {
 			rmSync(context, {
 				recursive: true,
@@ -2181,14 +2215,18 @@ const POD_ENV = {
 	name: "POD_NAME",
 	valueFrom: { fieldRef: { fieldPath: "metadata.name" } }
 };
-/** This function gives the environment of the app: its config as JSON, and the name of its pod. */
+/**
+* This function gives the environment of the app: its config as JSON, and the name of its pod. With
+* `failsRoute`, the route fails while the flag file is there (`cannot-serve`).
+*/
 const appEnv = (cell) => [{
 	name: "CONFIG",
 	value: JSON.stringify({
 		name: "lab",
 		port: PORT,
 		healthPort: HEALTH_PORT,
-		declaration: cell.declaration
+		declaration: cell.declaration,
+		...cell.trigger.kind === "cannot-serve" ? { failsRoute: true } : {}
 	})
 }, POD_ENV];
 /** This file is the flag of the trigger `liveness-restart`, in the root of the server process (`run.ts`, the step `flag`). */
@@ -2418,6 +2456,10 @@ const clientObjects = (cell, namespace, image, target) => [{
 							name: "TIMEOUT_MS",
 							value: String(valueIn(cell.workload.timeout, "ms"))
 						},
+						...cell.workload.connections === void 0 ? [] : [{
+							name: "CONNECTIONS",
+							value: String(cell.workload.connections)
+						}],
 						{
 							name: "POD_NAME",
 							valueFrom: { fieldRef: { fieldPath: "metadata.name" } }
@@ -2538,6 +2580,42 @@ const pausedWindow = (declaration, preStop, lead, hold, propagation) => {
 		resume: seconds(resume)
 	};
 };
+/**
+* This function gives the learn window of a declaration, its readiness probe, a propagation and a
+* workload of keep-alive connections. A caller selects a replica only when it opens a connection:
+* - The lame duck (`lameDuck` true or no value): after the latch, each answer has `Connection: close`,
+*   thus each connection of the replica gets one more failure and opens again. The rules select the
+*   replica until the removal, thus the last failure comes in one turn after it.
+* - No lame duck: no connection opens again, thus the rules do not matter. The failures end only
+*   when the readiness of the service recovers after the heal, and one turn after it.
+*/
+const learnWindow = (declaration, readiness, propagation, workload) => {
+	const turn = us(s$1(workload.every) * (workload.connections ?? 1));
+	const latch = latchBand(declaration.readiness, declaration.poll, "refuses");
+	const start = band(0, turn);
+	if (declaration.lameDuck === false) {
+		const recover = recoverSpan(declaration.readiness, declaration.poll);
+		return {
+			turn: seconds(turn),
+			latch,
+			start,
+			end: {
+				from: "heal",
+				band: band(0, s$1(recover.max) + turn)
+			}
+		};
+	}
+	const removal = secondsOf(removalBand(latch, readiness, propagation));
+	return {
+		turn: seconds(turn),
+		latch,
+		start,
+		end: {
+			from: "fault",
+			band: band(secondsOf(latch).lo, removal.hi + turn)
+		}
+	};
+};
 //#endregion
 //#region src/rules.ts
 /**
@@ -2710,7 +2788,8 @@ const stopOf = (cell) => {
 		case "authorization-policy":
 		case "idle":
 		case "ztunnel-restart":
-		case "certificate-expiry": return null;
+		case "certificate-expiry":
+		case "cannot-serve": return null;
 	}
 };
 /**
@@ -3685,7 +3764,8 @@ const estimate = (cell, environments = 2) => {
 		"authorization-policy": hold + 20,
 		idle: hold + 5,
 		"ztunnel-restart": 40,
-		"certificate-expiry": hold + 20
+		"certificate-expiry": hold + 20,
+		"cannot-serve": hold + 20
 	};
 	const after = observationAfter(cell);
 	const observed = after === null ? 0 : hold + valueIn(after, "s");
@@ -3723,6 +3803,6 @@ const shardOf = (cells, index, count) => {
 	return out;
 };
 //#endregion
-export { ACCESS_LOG_FORMAT, CHANNEL_USE, CLIENT, CLIENT_ACCOUNT, CLOSE_LEAD, CONTROLLER, DOCUMENTED, DRAIN_RULES, EDS_EVERY, FLAG_CHECK, FORMAT, FORMAT_V1, HBONE_PORT, HEALTH_PORT, ISTIOCTL, ISTIO_IMAGES, ISTIO_VERSION, KEPT_KERNEL, KIND, KIND_NOISE, KUBECTL, KUBE_PROXY_MODES, LAST_SLACK, LDS_EVERY, LDS_FILTER, LOG_LABEL, MESH_IMAGES, MESH_WORKLOADS, MIDDLEBOX, NETNS_DEFAULTS, NODE_IMAGES, O04, O09, O10, O16, POLICY, PORT, QUIET_LEAD, REDIS, REDIS_IMAGE, RELATIVE, ROUTE, RULES_EVERY, SERVER, STAGES, SYSCTL, TCP_STATE, WATCHDOG, ZTUNNEL_PODS, backendsAt, backendsOf, bandOf, byClients, byNode, cacheFolder, calibrationOf, capOf, cellsTable, certificateArgs, checkRun, cidrOverlap, cidrRange, clientObjects, clusterFilter, compose, createCluster, deletes, deletionOf, deletionWindow, deletionsOf, describeCell, dropOf, emptyOf, ensureTool, estimatedTime, factsOf, failuresOf, followsZtunnel, goDuration, hostsOf, installMesh, intakeAfter, intakeOf, isTunnelState, istioOperator, judge, judgeKind, keptKernel, kernelCounts, kindCellRecordSchema, kindCellRecordV1Schema, kindConfig, kubeProxyPatch, kubeletBand, labImage, latchBand, leafCertificates, lintCell, listenerCountsOf, localPin, medianInterval, medianText, meshKey, middle, middleboxCommands, namespaceObject, namespaceOf, nodeNames, observationAfter, observationEnd, openKindLab, outboundCluster, outcomesOf, pausedWindow, planOf, platformKey, podsAt, podsOf, policyObject, pollsListeners, predictionText, probeOf, propagationOf, propagationSource, quantileOf, readRecord, recordsOf, redisObjects, referenceOf, removalBand, requestsOf, restartBand, restartOf, restartsAt, reusesReset, rulesOf, runKindCells, scenario, scenarioOf, scenarioProblems, seconds, secondsOf, serverCluster, serverObjects, setZtunnelEnv, sha256, shardOf, showBand, signalOf, signalsOf, sliceDropOf, sliceEmptyOf, slicesOf, socketsOf, stackOf, stageLabImage, stallRemovalBand, stopOf, stoppingOf, sysctlCommand, tarEntry, timeSpread, toolPath, transitionOf, tunnelsOf, unitOf, updatesOf, upgrade, upgradeCell, verdictsOf, writeResult, ztunnelCertificates, ztunnelEnvArgs, ztunnelPodsOf, ztunnelReplaced };
+export { ACCESS_LOG_FORMAT, CHANNEL_USE, CLIENT, CLIENT_ACCOUNT, CLOSE_LEAD, CONTROLLER, DOCUMENTED, DRAIN_RULES, EDS_EVERY, FLAG_CHECK, FORMAT, FORMAT_V1, HBONE_PORT, HEALTH_PORT, ISTIOCTL, ISTIO_IMAGES, ISTIO_VERSION, KEPT_KERNEL, KIND, KIND_NOISE, KUBECTL, KUBE_PROXY_MODES, LAST_SLACK, LDS_EVERY, LDS_FILTER, LOG_LABEL, MESH_IMAGES, MESH_WORKLOADS, MIDDLEBOX, NETNS_DEFAULTS, NODE_IMAGES, O04, O09, O10, O16, POLICY, PORT, QUIET_LEAD, REDIS, REDIS_IMAGE, RELATIVE, ROUTE, RULES_EVERY, SERVER, SPREAD_LEAD, STAGES, SYSCTL, TCP_STATE, WATCHDOG, ZTUNNEL_PODS, backendsAt, backendsOf, bandOf, byClients, byNode, cacheFolder, calibrationOf, capOf, cellsTable, certificateArgs, checkRun, cidrOverlap, cidrRange, clientObjects, clusterFilter, compose, createCluster, deletes, deletionOf, deletionWindow, deletionsOf, describeCell, dropOf, emptyOf, ensureTool, estimatedTime, factsOf, failuresOf, followsZtunnel, goDuration, hostsOf, installMesh, intakeAfter, intakeOf, isTunnelState, istioOperator, judge, judgeKind, keptKernel, kernelCounts, kindCellRecordSchema, kindCellRecordV1Schema, kindConfig, kubeProxyPatch, kubeletBand, labImage, labSource, latchBand, leafCertificates, learnWindow, lintCell, listenerCountsOf, localPin, medianInterval, medianText, meshKey, middle, middleboxCommands, namespaceObject, namespaceOf, nodeNames, observationAfter, observationEnd, openKindLab, outboundCluster, outcomesOf, pausedWindow, planOf, platformKey, podsAt, podsOf, policyObject, pollsListeners, predictionText, probeOf, propagationOf, propagationSource, quantileOf, readRecord, recordsOf, redisObjects, referenceOf, removalBand, requestsOf, restartBand, restartOf, restartsAt, reusesReset, rulesOf, runKindCells, scenario, scenarioOf, scenarioProblems, seconds, secondsOf, serverCluster, serverObjects, setZtunnelEnv, sha256, shardOf, showBand, signalOf, signalsOf, sliceDropOf, sliceEmptyOf, slicesOf, socketsOf, stackOf, stageLabImage, stallRemovalBand, stopOf, stoppingOf, sysctlCommand, tarEntry, timeSpread, toolPath, transitionOf, tunnelsOf, unitOf, updatesOf, upgrade, upgradeCell, verdictsOf, writeResult, ztunnelCertificates, ztunnelEnvArgs, ztunnelPodsOf, ztunnelReplaced };
 
 //# sourceMappingURL=index.js.map