@mark1russell7/kind 0.10.1 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +7 -0
- package/build/index.d.ts +33 -2
- package/build/index.js +87 -7
- package/build/index.js.map +1 -1
- package/build/{judge-BD4UVQfy.d.ts → judge-4bzoeGIb.d.ts} +18 -3
- package/build/{judge-BbLWVTYj.js → judge-DdeJ1qVd.js} +7 -4
- package/build/{judge-BbLWVTYj.js.map → judge-DdeJ1qVd.js.map} +1 -1
- package/build/judge.d.ts +1 -1
- package/build/judge.js +1 -1
- package/image/app.mjs +7 -2
- package/image/client.mjs +32 -7
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -150,6 +150,9 @@ A new member of the union `Template` breaks a reader that switches on it. That i
|
|
|
150
150
|
|
|
151
151
|
The lab image (`image/`) holds the service of a cell on `@mark1russell7/service`, its clients, the agent and the watcher, on Node 24.21.0. Its base is pinned by digest, and `npm-lock.json` pins the packages from npm. `stageLabImage()` stages its context, and `labImage()` gives the tag of the image.
|
|
152
152
|
|
|
153
|
+
- **The keep-alive clients.** A workload with `connections` gives each HTTP client that count of keep-alive connections. Each request goes on the next connection in turn. kube-proxy selects a backend only for a new connection. SIGHUP gives the client new connections, and its old connections close after the timeout. With no `connections`, each request has a new connection, as before.
|
|
154
|
+
- **The lame duck.** The declaration has the optional field `lameDuck`. With true or no value, the edge sends `Connection: close` while the service is not ready (`httpTransport` as it is). With false, the app gives the transport `keepAlive: () => true`, thus each connection stays open and a caller learns only from the endpoints.
|
|
155
|
+
|
|
153
156
|
- **The resolver.** From 0.10.1, the lab image is a `build` source of the harness (`labSource()`). The resolver of the harness plans it and builds it, as it does each other image (`planOf`, `imageOf`). Thus its tag comes from its content, and each `FROM` has a digest. Each manifest of packages has a lockfile, and a failed build tries again after each wait. The build goes through the CLI of Docker, not through Testcontainers.
|
|
154
157
|
|
|
155
158
|
- **In the workspace** of this repository, the stage builds `@mark1russell7/quantity`, `@mark1russell7/algebra` and `@mark1russell7/service`, and packs each one as npm gets it.
|
|
@@ -186,6 +189,7 @@ The lab image (`image/`) holds the service of a cell on `@mark1russell7/service`
|
|
|
186
189
|
| `authorization-policy` | `kubectl apply` of a DENY AuthorizationPolicy on the server pods for the principal of an account (`account`, the account `default` of the clients when not given), then `kubectl delete` after the hold |
|
|
187
190
|
| `idle` | SIGUSR1 to the gRPC client of each client pod, and SIGUSR2 after the hold: a gap of the calls (the target `clients`). The restore sends SIGUSR2 again. With `middlebox`, a stateful middlebox in each replica from before the gap to the restore. With `close`, a new channel at the fault for 5 s: SIGHUP before the fault, then SIGUSR2, a wait and SIGHUP |
|
|
188
191
|
| `ztunnel-restart` | `kubectl delete pod` of the ztunnel pod of each node, with its grace (`ZTUNNEL_PODS`). The flow of each client crosses the ztunnel of its own node. The restore waits for the rollout of the DaemonSet |
|
|
192
|
+
| `cannot-serve` | SIGHUP to each client before the fault, thus the keep-alive connections spread over the replicas again in each run (`SPREAD_LEAD`, 3 s). Then the flag file of the first replica for the hold. While the flag is there, the route of the replica fails with 503, and the flag check of the declaration fails. The heal and the restore remove the flag |
|
|
189
193
|
| `certificate-expiry` | No act on a connection: the calls of each client go on for the hold. At the fault and at the heal, istioctl reads the leaf certificates of the ztunnel pod of each node (`ztunnelCertificates`). A line `certificates` keeps their validity, the evidence of their lifetime |
|
|
190
194
|
|
|
191
195
|
A cgroup path that is not the scope of one container is refused. A write to the root of the node throttles or freezes each pod of the node.
|
|
@@ -209,6 +213,7 @@ The plan's `event:clock-jump` has no trigger. A step of CLOCK_REALTIME in the VM
|
|
|
209
213
|
- **A reset from a leaving pod** leaves the conntrack entry of its flow in CLOSE for 10 s, with the NAT to that pod. From 0.1.6, a window can give that life (`afterReset`). Then a request outside the window can fail with the outcome of the rule (`reusesReset`). The condition is a reset of the same client on the same port, up to that life before the request. The catalog gives it to a deletion without a hold, in the modes with the NAT in conntrack.
|
|
210
214
|
- **A fault of a shared dependency** trips the readiness latch of each replica (`failSpan`). Then the kubelet needs its threshold of probes, and the rules need one propagation.
|
|
211
215
|
- **A restart through a coupling** adds the liveness probes of the kubelet and the preStop sleep to the latch.
|
|
216
|
+
- **A replica that cannot serve** (`learnWindow`) fails each request on a connection to it from the fault. A turn is the time between two requests on one connection. With the lame duck, the latch closes each connection, and the rules select the replica until its removal. Thus the window ends from the fault, in the latch to the removal and one turn. Without the lame duck, no connection opens again, thus the removal changes nothing. The window ends from the heal, in the recovery of the readiness and one turn.
|
|
212
217
|
|
|
213
218
|
`verdict.ts` judges each run, and each cell on its counted runs. Each time lands in its band ± 100 ms: the poll of the rules and the delay of the watcher. A median in a band gets the test of the harness: its order-statistic interval in the band, made 5 % wider.
|
|
214
219
|
|
|
@@ -256,6 +261,8 @@ From 0.8.0, a prediction `ends` can be the end of the tunnel of each client (`of
|
|
|
256
261
|
- The first request of each client after the heal must be `ok`, on a new tunnel. The main value is the middle of the ends, in `s`.
|
|
257
262
|
- From 0.8.1, `ends` has the optional band `answer`. A change to CLOSE_WAIT in that band ± δ also agrees: the peer answers the close of the client, and its FIN comes first. Before the band, the peer closed first.
|
|
258
263
|
|
|
264
|
+
A new member of the union `TriggerKind`, `cannot-serve`, breaks a reader that switches on it. That is a breaking change before 1.0.0, thus the next version is 0.11.0. The record format stays `/2`, with the optional fields `connections` of a workload and `lameDuck` of a declaration. The line `client` of the reconnect is a new line of a run.
|
|
265
|
+
|
|
259
266
|
A new member of the union `TriggerKind`, `certificate-expiry`, and a new member of the union `Step`, `certificates`, break a reader that switches on them. `KindLab` also has a new method, `record`. Each is a breaking change before 1.0.0, thus the version is 0.9.0. The record format stays `/2`: the lines `certificates` are new lines of a run.
|
|
260
267
|
|
|
261
268
|
In 0.8.1, the middlebox uses the match recent, not the sysctls of conntrack, and `ends` has the optional field `answer`. `SYSCTL` and `NETNS_DEFAULTS` keep only `tcp_retries2`. These fields came in 0.8.0, which was not published. The fix and the optional field are a patch, thus the version is 0.8.1.
|
package/build/index.d.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { $ as updatesOf, $t as Declaration, A as TriggerV1, An as scenario, At as rulesOf, B as RunCheck, Bt as ztunnelPodsOf, C as KindCellResult, Cn as NodeImage, Ct as outcomesOf, D as PredictionV1, Dn as kindConfig, Dt as requestsOf, E as LoadSample, En as goDuration, Et as recordsOf, F as KIND_NOISE, Ft as slicesOf, G as judgeKind, Gt as PORT, H as checkRun, Ht as CLIENT, I as KindDecision, It as socketsOf, J as propagationOf, Jt as bandOf, K as observationAfter, Kt as REDIS, L as KindVerdict, Lt as stoppingOf, M as upgrade, Mt as signalsOf, N as upgradeCell, Nt as sliceDropOf, O as RunCheckV1, On as kubeProxyPatch, Ot as restartOf, P as Deleted, Pt as sliceEmptyOf, Q as unitOf, Qt as Band, R as LAST_SLACK, Rt as transitionOf, S as KindCellRecordV1, Sn as NODE_IMAGES, St as kernelCounts, T as KindVerdictV1, Tn as cidrRange, Tt as podsOf, U as deletionsOf, Ut as HEALTH_PORT, V as RunFacts, Vt as ztunnelReplaced, W as factsOf, Wt as LOG_LABEL, X as referenceOf, Xt as secondsOf, Y as propagationSource, Yt as seconds, Z as reusesReset, Zt as showBand, _ as ClusterManifest, _n as Workload, _t as dropOf, a as STAGES, an as MeshMode, at as RecordLine, b as FORMAT_V1, bn as KindScenario, bt as intakeOf, c as readRecord, cn as RedisDeclaration, ct as SliceLine, d as calibrationOf, dn as ResetReuse, dt as TunnelLine, en as KindCell, et as Endpoint, f as cellsTable, fn as Stack, ft as ZtunnelPodLine, g as BandV1, gn as WindowPrediction, gt as deletionOf, h as timeSpread, hn as TriggerKind, ht as byNode, i as quantileOf, in as MeshFlavor, it as PropagationSource, j as judge, jn as scenarioProblems, jt as signalOf, k as RunFactsV1, kn as nodeNames, kt as restartsAt, l as Judged, ln as Reference, lt as SocketLine, m as predictionText, mn as Trigger, mt as byClients, n as medianInterval, nn as KubeletProbe, nt as Line, o as kindCellRecordSchema, on as PodSettings, ot as Request, p as medianText, pn as Template, pt as backendsAt, q as observationEnd, qt as SERVER, r as middle, rn as Mesh, rt as PodLine, s as kindCellRecordV1Schema, sn as Prediction, st as RulesLine, t as RELATIVE, tn as KindRun, tt as HBONE_PORT, u as TimeSpread, un as Replica, ut as TCP_STATE, v as EnvironmentManifest, vn as KUBE_PROXY_MODES, vt as emptyOf, w as KindCellV1, wn as cidrOverlap, wt as podsAt, x as KindCellRecord, xn as KubeProxyMode, xt as isTunnelState, y as FORMAT, yn as KindConfig, yt as failuresOf, z as Propagated, zt as tunnelsOf } from "./judge-
|
|
1
|
+
import { $ as updatesOf, $t as Declaration, A as TriggerV1, An as scenario, At as rulesOf, B as RunCheck, Bt as ztunnelPodsOf, C as KindCellResult, Cn as NodeImage, Ct as outcomesOf, D as PredictionV1, Dn as kindConfig, Dt as requestsOf, E as LoadSample, En as goDuration, Et as recordsOf, F as KIND_NOISE, Ft as slicesOf, G as judgeKind, Gt as PORT, H as checkRun, Ht as CLIENT, I as KindDecision, It as socketsOf, J as propagationOf, Jt as bandOf, K as observationAfter, Kt as REDIS, L as KindVerdict, Lt as stoppingOf, M as upgrade, Mt as signalsOf, N as upgradeCell, Nt as sliceDropOf, O as RunCheckV1, On as kubeProxyPatch, Ot as restartOf, P as Deleted, Pt as sliceEmptyOf, Q as unitOf, Qt as Band, R as LAST_SLACK, Rt as transitionOf, S as KindCellRecordV1, Sn as NODE_IMAGES, St as kernelCounts, T as KindVerdictV1, Tn as cidrRange, Tt as podsOf, U as deletionsOf, Ut as HEALTH_PORT, V as RunFacts, Vt as ztunnelReplaced, W as factsOf, Wt as LOG_LABEL, X as referenceOf, Xt as secondsOf, Y as propagationSource, Yt as seconds, Z as reusesReset, Zt as showBand, _ as ClusterManifest, _n as Workload, _t as dropOf, a as STAGES, an as MeshMode, at as RecordLine, b as FORMAT_V1, bn as KindScenario, bt as intakeOf, c as readRecord, cn as RedisDeclaration, ct as SliceLine, d as calibrationOf, dn as ResetReuse, dt as TunnelLine, en as KindCell, et as Endpoint, f as cellsTable, fn as Stack, ft as ZtunnelPodLine, g as BandV1, gn as WindowPrediction, gt as deletionOf, h as timeSpread, hn as TriggerKind, ht as byNode, i as quantileOf, in as MeshFlavor, it as PropagationSource, j as judge, jn as scenarioProblems, jt as signalOf, k as RunFactsV1, kn as nodeNames, kt as restartsAt, l as Judged, ln as Reference, lt as SocketLine, m as predictionText, mn as Trigger, mt as byClients, n as medianInterval, nn as KubeletProbe, nt as Line, o as kindCellRecordSchema, on as PodSettings, ot as Request, p as medianText, pn as Template, pt as backendsAt, q as observationEnd, qt as SERVER, r as middle, rn as Mesh, rt as PodLine, s as kindCellRecordV1Schema, sn as Prediction, st as RulesLine, t as RELATIVE, tn as KindRun, tt as HBONE_PORT, u as TimeSpread, un as Replica, ut as TCP_STATE, v as EnvironmentManifest, vn as KUBE_PROXY_MODES, vt as emptyOf, w as KindCellV1, wn as cidrOverlap, wt as podsAt, x as KindCellRecord, xn as KubeProxyMode, xt as isTunnelState, y as FORMAT, yn as KindConfig, yt as failuresOf, z as Propagated, zt as tunnelsOf } from "./judge-4bzoeGIb.js";
|
|
2
2
|
import { Quantity } from "@mark1russell7/quantity";
|
|
3
3
|
import { CheckFn, Finding, HealthServer, Poll, Rule, Scenario, ServiceBuilder, ServiceDescription, Stage, StopWay, Timing, TransportImpl, Verdict } from "@mark1russell7/service";
|
|
4
4
|
import { BuildImage } from "@mark1russell7/harness";
|
|
@@ -224,6 +224,11 @@ export declare const QUIET_LEAD: Quantity;
|
|
|
224
224
|
export declare const CLOSE_LEAD: Quantity;
|
|
225
225
|
/** This time is how long each client calls on its new channel at the fault of a gap with `close`, before it closes it. */
|
|
226
226
|
export declare const CHANNEL_USE: Quantity;
|
|
227
|
+
/**
|
|
228
|
+
* This time is between the SIGHUP of `cannot-serve` and its fault. Each client opens new connections
|
|
229
|
+
* at the signal, and the requests on its old connections end in this time (the timeout and 100 ms).
|
|
230
|
+
*/
|
|
231
|
+
export declare const SPREAD_LEAD: Quantity;
|
|
227
232
|
/**
|
|
228
233
|
* This function gives the command of the injector that sets one sysctl of its network namespace.
|
|
229
234
|
* Only a sysctl of the namespace (`net.`) and a whole number reach it.
|
|
@@ -639,6 +644,32 @@ export declare const deletionWindow: (declaration: Declaration, preStop: Quantit
|
|
|
639
644
|
export declare const pausedWindow: (declaration: Declaration, preStop: Quantity, lead: Quantity, hold: Quantity, propagation: Propagation) => DeletionWindow & {
|
|
640
645
|
readonly resume: Quantity;
|
|
641
646
|
};
|
|
647
|
+
/**
|
|
648
|
+
* A learn window is what `cannot-serve` predicts: when the callers stop getting errors from a
|
|
649
|
+
* replica that cannot serve. The window starts at the fault, and its end comes from `end.from`.
|
|
650
|
+
*/
|
|
651
|
+
interface LearnWindow {
|
|
652
|
+
/** This time is the interval between two requests on one connection of a client: `every` times the connections. */
|
|
653
|
+
readonly turn: Quantity;
|
|
654
|
+
/** This band is the trip of the readiness latch of the service after the fault. */
|
|
655
|
+
readonly latch: Band;
|
|
656
|
+
/** This band is the first failed request after the fault: within one turn. */
|
|
657
|
+
readonly start: Band;
|
|
658
|
+
readonly end: {
|
|
659
|
+
readonly from: Reference;
|
|
660
|
+
readonly band: Band;
|
|
661
|
+
};
|
|
662
|
+
}
|
|
663
|
+
/**
|
|
664
|
+
* This function gives the learn window of a declaration, its readiness probe, a propagation and a
|
|
665
|
+
* workload of keep-alive connections. A caller selects a replica only when it opens a connection:
|
|
666
|
+
* - The lame duck (`lameDuck` true or no value): after the latch, each answer has `Connection: close`,
|
|
667
|
+
* thus each connection of the replica gets one more failure and opens again. The rules select the
|
|
668
|
+
* replica until the removal, thus the last failure comes in one turn after it.
|
|
669
|
+
* - No lame duck: no connection opens again, thus the rules do not matter. The failures end only
|
|
670
|
+
* when the readiness of the service recovers after the heal, and one turn after it.
|
|
671
|
+
*/
|
|
672
|
+
export declare const learnWindow: (declaration: Declaration, readiness: KubeletProbe, propagation: Propagation, workload: Workload) => LearnWindow;
|
|
642
673
|
//#endregion
|
|
643
674
|
//#region src/rules.d.ts
|
|
644
675
|
/**
|
|
@@ -778,5 +809,5 @@ export declare const toolPath: (tool: Tool, key?: string, cache?: string) => str
|
|
|
778
809
|
*/
|
|
779
810
|
export declare const ensureTool: (tool: Tool, key?: string, cache?: string) => Promise<string>;
|
|
780
811
|
//#endregion
|
|
781
|
-
export { type Band, type BandV1, CLIENT, type Cluster, type ClusterHosts, type ClusterManifest, type Credentials, type Declaration, type Deleted, type DeletionWindow, type EdsSpec, Endpoint, type EnvironmentManifest, FORMAT, FORMAT_V1, type FaultPlan, HBONE_PORT, HEALTH_PORT, type Judged, KIND_NOISE, KUBE_PROXY_MODES, type KindCell, type KindCellRecord, type KindCellRecordV1, type KindCellResult, type KindCellV1, type KindConfig, type KindDecision, type KindLab, type KindRun, type KindScenario, type KindVerdict, type KindVerdictV1, type KubeProxyMode, type KubeletProbe, LAST_SLACK, LOG_LABEL, type LdsSpec, type LeafCertificate, Line, type ListenerCounts, type LoadSample, type Mesh, type MeshFlavor, type MeshImages, type MeshMode, NODE_IMAGES, type NodeImage, PORT, type Parts, PodLine, type PodSettings, type Prediction, type PredictionV1, type Propagated, type Propagation, PropagationSource, REDIS, RELATIVE, RecordLine, type RedisDeclaration, type Reference, type Replica, Request, type ResetReuse, RulesLine, type RunCheck, type RunCheckV1, type RunFacts, type RunFactsV1, type RunOptions, SERVER, STAGES, type ServiceRule, SliceLine, SocketLine, type Stack, type Stamp, type Step, TCP_STATE, type Target, type Template, type TimeSpread, type Tool, type Trigger, type TriggerKind, type TriggerV1, TunnelLine, type Verdicts, type WindowPrediction, type Workload, ZtunnelPodLine, backendsAt, bandOf, byClients, byNode, calibrationOf, cellsTable, checkRun, cidrOverlap, cidrRange, deletionOf, deletionsOf, dropOf, emptyOf, factsOf, failuresOf, goDuration, intakeOf, isTunnelState, judge, judgeKind, kernelCounts, kindCellRecordSchema, kindCellRecordV1Schema, kindConfig, kubeProxyPatch, medianInterval, medianText, middle, nodeNames, observationAfter, observationEnd, outcomesOf, podsAt, podsOf, predictionText, propagationOf, propagationSource, quantileOf, readRecord, recordsOf, referenceOf, requestsOf, restartOf, restartsAt, reusesReset, rulesOf, scenario, scenarioProblems, seconds, secondsOf, showBand, signalOf, signalsOf, sliceDropOf, sliceEmptyOf, slicesOf, socketsOf, stoppingOf, timeSpread, transitionOf, tunnelsOf, unitOf, updatesOf, upgrade, upgradeCell, ztunnelPodsOf, ztunnelReplaced };
|
|
812
|
+
export { type Band, type BandV1, CLIENT, type Cluster, type ClusterHosts, type ClusterManifest, type Credentials, type Declaration, type Deleted, type DeletionWindow, type EdsSpec, Endpoint, type EnvironmentManifest, FORMAT, FORMAT_V1, type FaultPlan, HBONE_PORT, HEALTH_PORT, type Judged, KIND_NOISE, KUBE_PROXY_MODES, type KindCell, type KindCellRecord, type KindCellRecordV1, type KindCellResult, type KindCellV1, type KindConfig, type KindDecision, type KindLab, type KindRun, type KindScenario, type KindVerdict, type KindVerdictV1, type KubeProxyMode, type KubeletProbe, LAST_SLACK, LOG_LABEL, type LdsSpec, type LeafCertificate, type LearnWindow, Line, type ListenerCounts, type LoadSample, type Mesh, type MeshFlavor, type MeshImages, type MeshMode, NODE_IMAGES, type NodeImage, PORT, type Parts, PodLine, type PodSettings, type Prediction, type PredictionV1, type Propagated, type Propagation, PropagationSource, REDIS, RELATIVE, RecordLine, type RedisDeclaration, type Reference, type Replica, Request, type ResetReuse, RulesLine, type RunCheck, type RunCheckV1, type RunFacts, type RunFactsV1, type RunOptions, SERVER, STAGES, type ServiceRule, SliceLine, SocketLine, type Stack, type Stamp, type Step, TCP_STATE, type Target, type Template, type TimeSpread, type Tool, type Trigger, type TriggerKind, type TriggerV1, TunnelLine, type Verdicts, type WindowPrediction, type Workload, ZtunnelPodLine, backendsAt, bandOf, byClients, byNode, calibrationOf, cellsTable, checkRun, cidrOverlap, cidrRange, deletionOf, deletionsOf, dropOf, emptyOf, factsOf, failuresOf, goDuration, intakeOf, isTunnelState, judge, judgeKind, kernelCounts, kindCellRecordSchema, kindCellRecordV1Schema, kindConfig, kubeProxyPatch, medianInterval, medianText, middle, nodeNames, observationAfter, observationEnd, outcomesOf, podsAt, podsOf, predictionText, propagationOf, propagationSource, quantileOf, readRecord, recordsOf, referenceOf, requestsOf, restartOf, restartsAt, reusesReset, rulesOf, scenario, scenarioProblems, seconds, secondsOf, showBand, signalOf, signalsOf, sliceDropOf, sliceEmptyOf, slicesOf, socketsOf, stoppingOf, timeSpread, transitionOf, tunnelsOf, unitOf, updatesOf, upgrade, upgradeCell, ztunnelPodsOf, ztunnelReplaced };
|
|
782
813
|
//# sourceMappingURL=index.d.ts.map
|
package/build/index.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { $ as signalOf, A as medianInterval, B as emptyOf, C as propagationOf, D as unitOf, E as reusesReset, F as backendsAt, G as outcomesOf, H as intakeOf, I as byClients, J as recordsOf, K as podsAt, L as byNode, M as quantileOf, N as HBONE_PORT, O as updatesOf, P as TCP_STATE, Q as rulesOf, R as deletionOf, S as observationEnd, T as referenceOf, U as isTunnelState, V as failuresOf, W as kernelCounts, X as restartOf, Y as requestsOf, Z as restartsAt, _ as checkRun, _t as seconds, a as calibrationOf, at as stoppingOf, b as judgeKind, c as predictionText, ct as ztunnelPodsOf, d as FORMAT_V1, dt as HEALTH_PORT, et as signalsOf, f as judge, ft as LOG_LABEL, g as LAST_SLACK, gt as bandOf, h as KIND_NOISE, ht as SERVER, i as readRecord, it as socketsOf, j as middle, k as RELATIVE, l as timeSpread, lt as ztunnelReplaced, m as upgradeCell, mt as REDIS, n as kindCellRecordSchema, nt as sliceEmptyOf, o as cellsTable, ot as transitionOf, p as upgrade, pt as PORT, q as podsOf, r as kindCellRecordV1Schema, rt as slicesOf, s as medianText, st as tunnelsOf, t as STAGES, tt as sliceDropOf, u as FORMAT, ut as CLIENT, v as deletionsOf, vt as secondsOf, w as propagationSource, x as observationAfter, y as factsOf, yt as showBand, z as dropOf } from "./judge-
|
|
1
|
+
import { $ as signalOf, A as medianInterval, B as emptyOf, C as propagationOf, D as unitOf, E as reusesReset, F as backendsAt, G as outcomesOf, H as intakeOf, I as byClients, J as recordsOf, K as podsAt, L as byNode, M as quantileOf, N as HBONE_PORT, O as updatesOf, P as TCP_STATE, Q as rulesOf, R as deletionOf, S as observationEnd, T as referenceOf, U as isTunnelState, V as failuresOf, W as kernelCounts, X as restartOf, Y as requestsOf, Z as restartsAt, _ as checkRun, _t as seconds, a as calibrationOf, at as stoppingOf, b as judgeKind, c as predictionText, ct as ztunnelPodsOf, d as FORMAT_V1, dt as HEALTH_PORT, et as signalsOf, f as judge, ft as LOG_LABEL, g as LAST_SLACK, gt as bandOf, h as KIND_NOISE, ht as SERVER, i as readRecord, it as socketsOf, j as middle, k as RELATIVE, l as timeSpread, lt as ztunnelReplaced, m as upgradeCell, mt as REDIS, n as kindCellRecordSchema, nt as sliceEmptyOf, o as cellsTable, ot as transitionOf, p as upgrade, pt as PORT, q as podsOf, r as kindCellRecordV1Schema, rt as slicesOf, s as medianText, st as tunnelsOf, t as STAGES, tt as sliceDropOf, u as FORMAT, ut as CLIENT, v as deletionsOf, vt as secondsOf, w as propagationSource, x as observationAfter, y as factsOf, yt as showBand, z as dropOf } from "./judge-DdeJ1qVd.js";
|
|
2
2
|
import { valueIn } from "@mark1russell7/quantity";
|
|
3
3
|
import { execFile, execFileSync } from "node:child_process";
|
|
4
4
|
import { chmodSync, copyFileSync, existsSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, renameSync, rmSync, writeFileSync } from "node:fs";
|
|
@@ -7,7 +7,7 @@ import { promisify } from "node:util";
|
|
|
7
7
|
import { createHash } from "node:crypto";
|
|
8
8
|
import { fileURLToPath } from "node:url";
|
|
9
9
|
import { gunzipSync } from "node:zlib";
|
|
10
|
-
import { failSpan, lint, lintStop, service } from "@mark1russell7/service";
|
|
10
|
+
import { failSpan, lint, lintStop, recoverSpan, service } from "@mark1russell7/service";
|
|
11
11
|
import { createRequire } from "node:module";
|
|
12
12
|
import { tmpdir } from "node:os";
|
|
13
13
|
import { harnessImage, imageOf } from "@mark1russell7/harness";
|
|
@@ -691,6 +691,14 @@ const CHANNEL_USE = {
|
|
|
691
691
|
value: 5,
|
|
692
692
|
unit: "s"
|
|
693
693
|
};
|
|
694
|
+
/**
|
|
695
|
+
* This time is between the SIGHUP of `cannot-serve` and its fault. Each client opens new connections
|
|
696
|
+
* at the signal, and the requests on its old connections end in this time (the timeout and 100 ms).
|
|
697
|
+
*/
|
|
698
|
+
const SPREAD_LEAD = {
|
|
699
|
+
value: 3,
|
|
700
|
+
unit: "s"
|
|
701
|
+
};
|
|
694
702
|
const IPV4 = /^(25[0-5]|2[0-4]\d|1?\d?\d)(\.(25[0-5]|2[0-4]\d|1?\d?\d)){3}$/;
|
|
695
703
|
/**
|
|
696
704
|
* This function gives the command of the injector that sets one sysctl of its network namespace.
|
|
@@ -1248,6 +1256,32 @@ const planOf = (cell) => {
|
|
|
1248
1256
|
heal: [{ do: "certificates" }],
|
|
1249
1257
|
restore: []
|
|
1250
1258
|
};
|
|
1259
|
+
case "cannot-serve": return {
|
|
1260
|
+
prepare: [{
|
|
1261
|
+
do: "signal",
|
|
1262
|
+
target: "clients",
|
|
1263
|
+
signal: "SIGHUP"
|
|
1264
|
+
}, {
|
|
1265
|
+
do: "wait",
|
|
1266
|
+
time: SPREAD_LEAD
|
|
1267
|
+
}],
|
|
1268
|
+
inject: [{
|
|
1269
|
+
do: "flag",
|
|
1270
|
+
target: "server",
|
|
1271
|
+
on: true
|
|
1272
|
+
}],
|
|
1273
|
+
hold: needs(trigger.hold, "a hold, the time that the replica cannot serve"),
|
|
1274
|
+
heal: [{
|
|
1275
|
+
do: "flag",
|
|
1276
|
+
target: "server",
|
|
1277
|
+
on: false
|
|
1278
|
+
}],
|
|
1279
|
+
restore: [{
|
|
1280
|
+
do: "flag",
|
|
1281
|
+
target: "server",
|
|
1282
|
+
on: false
|
|
1283
|
+
}]
|
|
1284
|
+
};
|
|
1251
1285
|
}
|
|
1252
1286
|
};
|
|
1253
1287
|
/** This function tells if a trigger stops a pod of the service with its grace: a deletion, an eviction, a scale-in or a rollout. */
|
|
@@ -2181,14 +2215,18 @@ const POD_ENV = {
|
|
|
2181
2215
|
name: "POD_NAME",
|
|
2182
2216
|
valueFrom: { fieldRef: { fieldPath: "metadata.name" } }
|
|
2183
2217
|
};
|
|
2184
|
-
/**
|
|
2218
|
+
/**
|
|
2219
|
+
* This function gives the environment of the app: its config as JSON, and the name of its pod. With
|
|
2220
|
+
* `failsRoute`, the route fails while the flag file is there (`cannot-serve`).
|
|
2221
|
+
*/
|
|
2185
2222
|
const appEnv = (cell) => [{
|
|
2186
2223
|
name: "CONFIG",
|
|
2187
2224
|
value: JSON.stringify({
|
|
2188
2225
|
name: "lab",
|
|
2189
2226
|
port: PORT,
|
|
2190
2227
|
healthPort: HEALTH_PORT,
|
|
2191
|
-
declaration: cell.declaration
|
|
2228
|
+
declaration: cell.declaration,
|
|
2229
|
+
...cell.trigger.kind === "cannot-serve" ? { failsRoute: true } : {}
|
|
2192
2230
|
})
|
|
2193
2231
|
}, POD_ENV];
|
|
2194
2232
|
/** This file is the flag of the trigger `liveness-restart`, in the root of the server process (`run.ts`, the step `flag`). */
|
|
@@ -2418,6 +2456,10 @@ const clientObjects = (cell, namespace, image, target) => [{
|
|
|
2418
2456
|
name: "TIMEOUT_MS",
|
|
2419
2457
|
value: String(valueIn(cell.workload.timeout, "ms"))
|
|
2420
2458
|
},
|
|
2459
|
+
...cell.workload.connections === void 0 ? [] : [{
|
|
2460
|
+
name: "CONNECTIONS",
|
|
2461
|
+
value: String(cell.workload.connections)
|
|
2462
|
+
}],
|
|
2421
2463
|
{
|
|
2422
2464
|
name: "POD_NAME",
|
|
2423
2465
|
valueFrom: { fieldRef: { fieldPath: "metadata.name" } }
|
|
@@ -2538,6 +2580,42 @@ const pausedWindow = (declaration, preStop, lead, hold, propagation) => {
|
|
|
2538
2580
|
resume: seconds(resume)
|
|
2539
2581
|
};
|
|
2540
2582
|
};
|
|
2583
|
+
/**
|
|
2584
|
+
* This function gives the learn window of a declaration, its readiness probe, a propagation and a
|
|
2585
|
+
* workload of keep-alive connections. A caller selects a replica only when it opens a connection:
|
|
2586
|
+
* - The lame duck (`lameDuck` true or no value): after the latch, each answer has `Connection: close`,
|
|
2587
|
+
* thus each connection of the replica gets one more failure and opens again. The rules select the
|
|
2588
|
+
* replica until the removal, thus the last failure comes in one turn after it.
|
|
2589
|
+
* - No lame duck: no connection opens again, thus the rules do not matter. The failures end only
|
|
2590
|
+
* when the readiness of the service recovers after the heal, and one turn after it.
|
|
2591
|
+
*/
|
|
2592
|
+
const learnWindow = (declaration, readiness, propagation, workload) => {
|
|
2593
|
+
const turn = us(s$1(workload.every) * (workload.connections ?? 1));
|
|
2594
|
+
const latch = latchBand(declaration.readiness, declaration.poll, "refuses");
|
|
2595
|
+
const start = band(0, turn);
|
|
2596
|
+
if (declaration.lameDuck === false) {
|
|
2597
|
+
const recover = recoverSpan(declaration.readiness, declaration.poll);
|
|
2598
|
+
return {
|
|
2599
|
+
turn: seconds(turn),
|
|
2600
|
+
latch,
|
|
2601
|
+
start,
|
|
2602
|
+
end: {
|
|
2603
|
+
from: "heal",
|
|
2604
|
+
band: band(0, s$1(recover.max) + turn)
|
|
2605
|
+
}
|
|
2606
|
+
};
|
|
2607
|
+
}
|
|
2608
|
+
const removal = secondsOf(removalBand(latch, readiness, propagation));
|
|
2609
|
+
return {
|
|
2610
|
+
turn: seconds(turn),
|
|
2611
|
+
latch,
|
|
2612
|
+
start,
|
|
2613
|
+
end: {
|
|
2614
|
+
from: "fault",
|
|
2615
|
+
band: band(secondsOf(latch).lo, removal.hi + turn)
|
|
2616
|
+
}
|
|
2617
|
+
};
|
|
2618
|
+
};
|
|
2541
2619
|
//#endregion
|
|
2542
2620
|
//#region src/rules.ts
|
|
2543
2621
|
/**
|
|
@@ -2710,7 +2788,8 @@ const stopOf = (cell) => {
|
|
|
2710
2788
|
case "authorization-policy":
|
|
2711
2789
|
case "idle":
|
|
2712
2790
|
case "ztunnel-restart":
|
|
2713
|
-
case "certificate-expiry":
|
|
2791
|
+
case "certificate-expiry":
|
|
2792
|
+
case "cannot-serve": return null;
|
|
2714
2793
|
}
|
|
2715
2794
|
};
|
|
2716
2795
|
/**
|
|
@@ -3685,7 +3764,8 @@ const estimate = (cell, environments = 2) => {
|
|
|
3685
3764
|
"authorization-policy": hold + 20,
|
|
3686
3765
|
idle: hold + 5,
|
|
3687
3766
|
"ztunnel-restart": 40,
|
|
3688
|
-
"certificate-expiry": hold + 20
|
|
3767
|
+
"certificate-expiry": hold + 20,
|
|
3768
|
+
"cannot-serve": hold + 20
|
|
3689
3769
|
};
|
|
3690
3770
|
const after = observationAfter(cell);
|
|
3691
3771
|
const observed = after === null ? 0 : hold + valueIn(after, "s");
|
|
@@ -3723,6 +3803,6 @@ const shardOf = (cells, index, count) => {
|
|
|
3723
3803
|
return out;
|
|
3724
3804
|
};
|
|
3725
3805
|
//#endregion
|
|
3726
|
-
export { ACCESS_LOG_FORMAT, CHANNEL_USE, CLIENT, CLIENT_ACCOUNT, CLOSE_LEAD, CONTROLLER, DOCUMENTED, DRAIN_RULES, EDS_EVERY, FLAG_CHECK, FORMAT, FORMAT_V1, HBONE_PORT, HEALTH_PORT, ISTIOCTL, ISTIO_IMAGES, ISTIO_VERSION, KEPT_KERNEL, KIND, KIND_NOISE, KUBECTL, KUBE_PROXY_MODES, LAST_SLACK, LDS_EVERY, LDS_FILTER, LOG_LABEL, MESH_IMAGES, MESH_WORKLOADS, MIDDLEBOX, NETNS_DEFAULTS, NODE_IMAGES, O04, O09, O10, O16, POLICY, PORT, QUIET_LEAD, REDIS, REDIS_IMAGE, RELATIVE, ROUTE, RULES_EVERY, SERVER, STAGES, SYSCTL, TCP_STATE, WATCHDOG, ZTUNNEL_PODS, backendsAt, backendsOf, bandOf, byClients, byNode, cacheFolder, calibrationOf, capOf, cellsTable, certificateArgs, checkRun, cidrOverlap, cidrRange, clientObjects, clusterFilter, compose, createCluster, deletes, deletionOf, deletionWindow, deletionsOf, describeCell, dropOf, emptyOf, ensureTool, estimatedTime, factsOf, failuresOf, followsZtunnel, goDuration, hostsOf, installMesh, intakeAfter, intakeOf, isTunnelState, istioOperator, judge, judgeKind, keptKernel, kernelCounts, kindCellRecordSchema, kindCellRecordV1Schema, kindConfig, kubeProxyPatch, kubeletBand, labImage, labSource, latchBand, leafCertificates, lintCell, listenerCountsOf, localPin, medianInterval, medianText, meshKey, middle, middleboxCommands, namespaceObject, namespaceOf, nodeNames, observationAfter, observationEnd, openKindLab, outboundCluster, outcomesOf, pausedWindow, planOf, platformKey, podsAt, podsOf, policyObject, pollsListeners, predictionText, probeOf, propagationOf, propagationSource, quantileOf, readRecord, recordsOf, redisObjects, referenceOf, removalBand, requestsOf, restartBand, restartOf, restartsAt, reusesReset, rulesOf, runKindCells, scenario, scenarioOf, scenarioProblems, seconds, secondsOf, serverCluster, serverObjects, setZtunnelEnv, sha256, shardOf, showBand, signalOf, signalsOf, sliceDropOf, sliceEmptyOf, slicesOf, socketsOf, stackOf, stageLabImage, stallRemovalBand, stopOf, stoppingOf, sysctlCommand, tarEntry, timeSpread, toolPath, transitionOf, tunnelsOf, unitOf, updatesOf, upgrade, upgradeCell, verdictsOf, writeResult, ztunnelCertificates, ztunnelEnvArgs, ztunnelPodsOf, ztunnelReplaced };
|
|
3806
|
+
export { ACCESS_LOG_FORMAT, CHANNEL_USE, CLIENT, CLIENT_ACCOUNT, CLOSE_LEAD, CONTROLLER, DOCUMENTED, DRAIN_RULES, EDS_EVERY, FLAG_CHECK, FORMAT, FORMAT_V1, HBONE_PORT, HEALTH_PORT, ISTIOCTL, ISTIO_IMAGES, ISTIO_VERSION, KEPT_KERNEL, KIND, KIND_NOISE, KUBECTL, KUBE_PROXY_MODES, LAST_SLACK, LDS_EVERY, LDS_FILTER, LOG_LABEL, MESH_IMAGES, MESH_WORKLOADS, MIDDLEBOX, NETNS_DEFAULTS, NODE_IMAGES, O04, O09, O10, O16, POLICY, PORT, QUIET_LEAD, REDIS, REDIS_IMAGE, RELATIVE, ROUTE, RULES_EVERY, SERVER, SPREAD_LEAD, STAGES, SYSCTL, TCP_STATE, WATCHDOG, ZTUNNEL_PODS, backendsAt, backendsOf, bandOf, byClients, byNode, cacheFolder, calibrationOf, capOf, cellsTable, certificateArgs, checkRun, cidrOverlap, cidrRange, clientObjects, clusterFilter, compose, createCluster, deletes, deletionOf, deletionWindow, deletionsOf, describeCell, dropOf, emptyOf, ensureTool, estimatedTime, factsOf, failuresOf, followsZtunnel, goDuration, hostsOf, installMesh, intakeAfter, intakeOf, isTunnelState, istioOperator, judge, judgeKind, keptKernel, kernelCounts, kindCellRecordSchema, kindCellRecordV1Schema, kindConfig, kubeProxyPatch, kubeletBand, labImage, labSource, latchBand, leafCertificates, learnWindow, lintCell, listenerCountsOf, localPin, medianInterval, medianText, meshKey, middle, middleboxCommands, namespaceObject, namespaceOf, nodeNames, observationAfter, observationEnd, openKindLab, outboundCluster, outcomesOf, pausedWindow, planOf, platformKey, podsAt, podsOf, policyObject, pollsListeners, predictionText, probeOf, propagationOf, propagationSource, quantileOf, readRecord, recordsOf, redisObjects, referenceOf, removalBand, requestsOf, restartBand, restartOf, restartsAt, reusesReset, rulesOf, runKindCells, scenario, scenarioOf, scenarioProblems, seconds, secondsOf, serverCluster, serverObjects, setZtunnelEnv, sha256, shardOf, showBand, signalOf, signalsOf, sliceDropOf, sliceEmptyOf, slicesOf, socketsOf, stackOf, stageLabImage, stallRemovalBand, stopOf, stoppingOf, sysctlCommand, tarEntry, timeSpread, toolPath, transitionOf, tunnelsOf, unitOf, updatesOf, upgrade, upgradeCell, verdictsOf, writeResult, ztunnelCertificates, ztunnelEnvArgs, ztunnelPodsOf, ztunnelReplaced };
|
|
3727
3807
|
|
|
3728
3808
|
//# sourceMappingURL=index.js.map
|