@mark1russell7/kind 0.11.0 → 0.11.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +7 -1
- package/build/index.d.ts +11 -5
- package/build/index.js +10 -3
- package/build/index.js.map +1 -1
- package/build/{judge-DdeJ1qVd.js → judge-Bb4ebE9V.js} +3 -2
- package/build/{judge-DdeJ1qVd.js.map → judge-Bb4ebE9V.js.map} +1 -1
- package/build/{judge-4bzoeGIb.d.ts → judge-C3mbgYxY.d.ts} +19 -8
- package/build/judge.d.ts +2 -2
- package/build/judge.js +1 -1
- package/image/app.mjs +4 -2
- package/image/client.mjs +122 -9
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -152,6 +152,10 @@ The lab image (`image/`) holds the service of a cell on `@mark1russell7/service`
|
|
|
152
152
|
|
|
153
153
|
- **The keep-alive clients.** A workload with `connections` gives each HTTP client that count of keep-alive connections. Each request goes on the next connection in turn. kube-proxy selects a backend only for a new connection. SIGHUP gives the client new connections, and its old connections close after the timeout. With no `connections`, each request has a new connection, as before.
|
|
154
154
|
- **The lame duck.** The declaration has the optional field `lameDuck`. With true or no value, the edge sends `Connection: close` while the service is not ready (`httpTransport` as it is). With false, the app gives the transport `keepAlive: () => true`, thus each connection stays open and a caller learns only from the endpoints.
|
|
155
|
+
- **HTTP/2.** The declaration has the optional field `protocol` (`EdgeProtocol`). With `h2c`, the app listens with `node:http2` through the same `httpTransport`, and the lame duck is a GOAWAY on each session. The manifests give each client `PROTOCOL=h2c`. The client then keeps `connections` sessions, and sends each request as a stream on the next session in turn.
|
|
156
|
+
- A GOAWAY, a close or an error of a session makes its slot open a new session at the next stream. Then kube-proxy selects a backend for that new connection.
|
|
157
|
+
- The client sends a stream that the server did not take again, once, on a new session (RFC 9113, section 8.7). Its line has `r: 1`.
|
|
158
|
+
- Each stream writes the same `req` line as a request of HTTP/1.1, with the local port of the socket of its session.
|
|
155
159
|
|
|
156
160
|
- **The resolver.** From 0.10.1, the lab image is a `build` source of the harness (`labSource()`). The resolver of the harness plans it and builds it, as it does each other image (`planOf`, `imageOf`). Thus its tag comes from its content, and each `FROM` has a digest. Each manifest of packages has a lockfile, and a failed build tries again after each wait. The build goes through the CLI of Docker, not through Testcontainers.
|
|
157
161
|
|
|
@@ -213,7 +217,7 @@ The plan's `event:clock-jump` has no trigger. A step of CLOCK_REALTIME in the VM
|
|
|
213
217
|
- **A reset from a leaving pod** leaves the conntrack entry of its flow in CLOSE for 10 s, with the NAT to that pod. From 0.1.6, a window can give that life (`afterReset`). Then a request outside the window can fail with the outcome of the rule (`reusesReset`). The condition is a reset of the same client on the same port, up to that life before the request. The catalog gives it to a deletion without a hold, in the modes with the NAT in conntrack.
|
|
214
218
|
- **A fault of a shared dependency** trips the readiness latch of each replica (`failSpan`). Then the kubelet needs its threshold of probes, and the rules need one propagation.
|
|
215
219
|
- **A restart through a coupling** adds the liveness probes of the kubelet and the preStop sleep to the latch.
|
|
216
|
-
- **A replica that cannot serve** (`learnWindow`) fails each request on a connection to it from the fault. A turn is the time between two requests on one connection. With the lame duck, the latch closes each connection, and the rules select the replica until its removal. Thus the window ends from the fault, in the latch to the removal and one turn. Without the lame duck, no connection opens again, thus the removal changes nothing. The window ends from the heal, in the recovery of the readiness and one turn.
|
|
220
|
+
- **A replica that cannot serve** (`learnWindow`) fails each request on a connection to it from the fault. A turn is the time between two requests on one connection. With the lame duck, the latch closes each connection (`Connection: close`, or GOAWAY on HTTP/2), and the rules select the replica until its removal. Thus the window ends from the fault, in the latch to the removal and one turn. Without the lame duck, no connection opens again, thus the removal changes nothing. The window ends from the heal, in the recovery of the readiness and one turn.
|
|
217
221
|
|
|
218
222
|
`verdict.ts` judges each run, and each cell on its counted runs. Each time lands in its band ± 100 ms: the poll of the rules and the delay of the watcher. A median in a band gets the test of the harness: its order-statistic interval in the band, made 5 % wider.
|
|
219
223
|
|
|
@@ -261,6 +265,8 @@ From 0.8.0, a prediction `ends` can be the end of the tunnel of each client (`of
|
|
|
261
265
|
- The first request of each client after the heal must be `ok`, on a new tunnel. The main value is the middle of the ends, in `s`.
|
|
262
266
|
- From 0.8.1, `ends` has the optional band `answer`. A change to CLOSE_WAIT in that band ± δ also agrees: the peer answers the close of the client, and its FIN comes first. Before the band, the peer closed first.
|
|
263
267
|
|
|
268
|
+
The optional field `protocol` of a declaration, the type `EdgeProtocol` and the mode h2c of the lab image are new features. They keep the old behavior. Before `1.0.0`, a feature is a patch, thus the next version is `0.11.1`. The record format stays `/2`.
|
|
269
|
+
|
|
264
270
|
A new member of the union `TriggerKind`, `cannot-serve`, breaks a reader that switches on it. That is a breaking change before 1.0.0, thus the next version is 0.11.0. The record format stays `/2`, with the optional fields `connections` of a workload and `lameDuck` of a declaration. The line `client` of the reconnect is a new line of a run.
|
|
265
271
|
|
|
266
272
|
A new member of the union `TriggerKind`, `certificate-expiry`, and a new member of the union `Step`, `certificates`, break a reader that switches on them. `KindLab` also has a new method, `record`. Each is a breaking change before 1.0.0, thus the version is 0.9.0. The record format stays `/2`: the lines `certificates` are new lines of a run.
|
package/build/index.d.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { $ as updatesOf, $t as Declaration, A as TriggerV1, An as
|
|
1
|
+
import { $ as updatesOf, $t as Declaration, A as TriggerV1, An as nodeNames, At as rulesOf, B as RunCheck, Bt as ztunnelPodsOf, C as KindCellResult, Cn as NODE_IMAGES, Ct as outcomesOf, D as PredictionV1, Dn as goDuration, Dt as requestsOf, E as LoadSample, En as cidrRange, Et as recordsOf, F as KIND_NOISE, Ft as slicesOf, G as judgeKind, Gt as PORT, H as checkRun, Ht as CLIENT, I as KindDecision, It as socketsOf, J as propagationOf, Jt as bandOf, K as observationAfter, Kt as REDIS, L as KindVerdict, Lt as stoppingOf, M as upgrade, Mn as scenarioProblems, Mt as signalsOf, N as upgradeCell, Nt as sliceDropOf, O as RunCheckV1, On as kindConfig, Ot as restartOf, P as Deleted, Pt as sliceEmptyOf, Q as unitOf, Qt as Band, R as LAST_SLACK, Rt as transitionOf, S as KindCellRecordV1, Sn as KubeProxyMode, St as kernelCounts, T as KindVerdictV1, Tn as cidrOverlap, Tt as podsOf, U as deletionsOf, Ut as HEALTH_PORT, V as RunFacts, Vt as ztunnelReplaced, W as factsOf, Wt as LOG_LABEL, X as referenceOf, Xt as secondsOf, Y as propagationSource, Yt as seconds, Z as reusesReset, Zt as showBand, _ as ClusterManifest, _n as WindowPrediction, _t as dropOf, a as STAGES, an as MeshFlavor, at as RecordLine, b as FORMAT_V1, bn as KindConfig, bt as intakeOf, c as readRecord, cn as Prediction, ct as SliceLine, d as calibrationOf, dn as Replica, dt as TunnelLine, en as EdgeProtocol, et as Endpoint, f as cellsTable, fn as ResetReuse, ft as ZtunnelPodLine, g as BandV1, gn as TriggerKind, gt as deletionOf, h as timeSpread, hn as Trigger, ht as byNode, i as quantileOf, in as Mesh, it as PropagationSource, j as judge, jn as scenario, jt as signalOf, k as RunFactsV1, kn as kubeProxyPatch, kt as restartsAt, l as Judged, ln as RedisDeclaration, lt as SocketLine, m as predictionText, mn as Template, mt as byClients, n as medianInterval, nn as KindRun, nt as Line, o as kindCellRecordSchema, on as MeshMode, ot as Request, p as medianText, pn as Stack, pt as backendsAt, q as observationEnd, qt as SERVER, r as middle, rn as KubeletProbe, rt as PodLine, s as kindCellRecordV1Schema, sn as PodSettings, st as RulesLine, t as RELATIVE, tn as KindCell, tt as HBONE_PORT, u as TimeSpread, un as Reference, ut as TCP_STATE, v as EnvironmentManifest, vn as Workload, vt as emptyOf, w as KindCellV1, wn as NodeImage, wt as podsAt, x as KindCellRecord, xn as KindScenario, xt as isTunnelState, y as FORMAT, yn as KUBE_PROXY_MODES, yt as failuresOf, z as Propagated, zt as tunnelsOf } from "./judge-C3mbgYxY.js";
|
|
2
2
|
import { Quantity } from "@mark1russell7/quantity";
|
|
3
3
|
import { CheckFn, Finding, HealthServer, Poll, Rule, Scenario, ServiceBuilder, ServiceDescription, Stage, StopWay, Timing, TransportImpl, Verdict } from "@mark1russell7/service";
|
|
4
4
|
import { BuildImage } from "@mark1russell7/harness";
|
|
@@ -569,7 +569,8 @@ export declare const serverObjects: (cell: KindCell, namespace: string, image: s
|
|
|
569
569
|
/**
|
|
570
570
|
* These are the objects of the clients: a DaemonSet with one client on each node, the control plane
|
|
571
571
|
* too. The client of the service is `client.mjs`, and the client of grpc-go is `client-grpc.mjs`.
|
|
572
|
-
* Each one is in the lab image, and each one writes the same lines.
|
|
572
|
+
* Each one is in the lab image, and each one writes the same lines. A service with the protocol
|
|
573
|
+
* `h2c` gives its clients `PROTOCOL`, thus they speak HTTP/2 to it.
|
|
573
574
|
*/
|
|
574
575
|
export declare const clientObjects: (cell: KindCell, namespace: string, image: string, target: string) => unknown[];
|
|
575
576
|
//#endregion
|
|
@@ -649,7 +650,10 @@ export declare const pausedWindow: (declaration: Declaration, preStop: Quantity,
|
|
|
649
650
|
* replica that cannot serve. The window starts at the fault, and its end comes from `end.from`.
|
|
650
651
|
*/
|
|
651
652
|
interface LearnWindow {
|
|
652
|
-
/**
|
|
653
|
+
/**
|
|
654
|
+
* This time is the interval between two requests on one connection of a client: `every` times the
|
|
655
|
+
* connections. A client of HTTP/2 sends its streams on fewer connections, thus its turn is shorter.
|
|
656
|
+
*/
|
|
653
657
|
readonly turn: Quantity;
|
|
654
658
|
/** This band is the trip of the readiness latch of the service after the fault. */
|
|
655
659
|
readonly latch: Band;
|
|
@@ -663,9 +667,11 @@ interface LearnWindow {
|
|
|
663
667
|
/**
|
|
664
668
|
* This function gives the learn window of a declaration, its readiness probe, a propagation and a
|
|
665
669
|
* workload of keep-alive connections. A caller selects a replica only when it opens a connection:
|
|
666
|
-
* - The lame duck (`lameDuck` true or no value): after the latch, each answer
|
|
670
|
+
* - The lame duck (`lameDuck` true or no value): after the latch, each answer closes its connection,
|
|
667
671
|
* thus each connection of the replica gets one more failure and opens again. The rules select the
|
|
668
672
|
* replica until the removal, thus the last failure comes in one turn after it.
|
|
673
|
+
* - The signal of the lame duck is `Connection: close` on HTTP/1.1, and GOAWAY on HTTP/2. The
|
|
674
|
+
* arithmetic is the same: the turn is `every` times the connections, which are sessions on HTTP/2.
|
|
669
675
|
* - No lame duck: no connection opens again, thus the rules do not matter. The failures end only
|
|
670
676
|
* when the readiness of the service recovers after the heal, and one turn after it.
|
|
671
677
|
*/
|
|
@@ -809,5 +815,5 @@ export declare const toolPath: (tool: Tool, key?: string, cache?: string) => str
|
|
|
809
815
|
*/
|
|
810
816
|
export declare const ensureTool: (tool: Tool, key?: string, cache?: string) => Promise<string>;
|
|
811
817
|
//#endregion
|
|
812
|
-
export { type Band, type BandV1, CLIENT, type Cluster, type ClusterHosts, type ClusterManifest, type Credentials, type Declaration, type Deleted, type DeletionWindow, type EdsSpec, Endpoint, type EnvironmentManifest, FORMAT, FORMAT_V1, type FaultPlan, HBONE_PORT, HEALTH_PORT, type Judged, KIND_NOISE, KUBE_PROXY_MODES, type KindCell, type KindCellRecord, type KindCellRecordV1, type KindCellResult, type KindCellV1, type KindConfig, type KindDecision, type KindLab, type KindRun, type KindScenario, type KindVerdict, type KindVerdictV1, type KubeProxyMode, type KubeletProbe, LAST_SLACK, LOG_LABEL, type LdsSpec, type LeafCertificate, type LearnWindow, Line, type ListenerCounts, type LoadSample, type Mesh, type MeshFlavor, type MeshImages, type MeshMode, NODE_IMAGES, type NodeImage, PORT, type Parts, PodLine, type PodSettings, type Prediction, type PredictionV1, type Propagated, type Propagation, PropagationSource, REDIS, RELATIVE, RecordLine, type RedisDeclaration, type Reference, type Replica, Request, type ResetReuse, RulesLine, type RunCheck, type RunCheckV1, type RunFacts, type RunFactsV1, type RunOptions, SERVER, STAGES, type ServiceRule, SliceLine, SocketLine, type Stack, type Stamp, type Step, TCP_STATE, type Target, type Template, type TimeSpread, type Tool, type Trigger, type TriggerKind, type TriggerV1, TunnelLine, type Verdicts, type WindowPrediction, type Workload, ZtunnelPodLine, backendsAt, bandOf, byClients, byNode, calibrationOf, cellsTable, checkRun, cidrOverlap, cidrRange, deletionOf, deletionsOf, dropOf, emptyOf, factsOf, failuresOf, goDuration, intakeOf, isTunnelState, judge, judgeKind, kernelCounts, kindCellRecordSchema, kindCellRecordV1Schema, kindConfig, kubeProxyPatch, medianInterval, medianText, middle, nodeNames, observationAfter, observationEnd, outcomesOf, podsAt, podsOf, predictionText, propagationOf, propagationSource, quantileOf, readRecord, recordsOf, referenceOf, requestsOf, restartOf, restartsAt, reusesReset, rulesOf, scenario, scenarioProblems, seconds, secondsOf, showBand, signalOf, signalsOf, sliceDropOf, sliceEmptyOf, slicesOf, socketsOf, stoppingOf, timeSpread, transitionOf, tunnelsOf, unitOf, updatesOf, upgrade, upgradeCell, ztunnelPodsOf, ztunnelReplaced };
|
|
818
|
+
export { type Band, type BandV1, CLIENT, type Cluster, type ClusterHosts, type ClusterManifest, type Credentials, type Declaration, type Deleted, type DeletionWindow, type EdgeProtocol, type EdsSpec, Endpoint, type EnvironmentManifest, FORMAT, FORMAT_V1, type FaultPlan, HBONE_PORT, HEALTH_PORT, type Judged, KIND_NOISE, KUBE_PROXY_MODES, type KindCell, type KindCellRecord, type KindCellRecordV1, type KindCellResult, type KindCellV1, type KindConfig, type KindDecision, type KindLab, type KindRun, type KindScenario, type KindVerdict, type KindVerdictV1, type KubeProxyMode, type KubeletProbe, LAST_SLACK, LOG_LABEL, type LdsSpec, type LeafCertificate, type LearnWindow, Line, type ListenerCounts, type LoadSample, type Mesh, type MeshFlavor, type MeshImages, type MeshMode, NODE_IMAGES, type NodeImage, PORT, type Parts, PodLine, type PodSettings, type Prediction, type PredictionV1, type Propagated, type Propagation, PropagationSource, REDIS, RELATIVE, RecordLine, type RedisDeclaration, type Reference, type Replica, Request, type ResetReuse, RulesLine, type RunCheck, type RunCheckV1, type RunFacts, type RunFactsV1, type RunOptions, SERVER, STAGES, type ServiceRule, SliceLine, SocketLine, type Stack, type Stamp, type Step, TCP_STATE, type Target, type Template, type TimeSpread, type Tool, type Trigger, type TriggerKind, type TriggerV1, TunnelLine, type Verdicts, type WindowPrediction, type Workload, ZtunnelPodLine, backendsAt, bandOf, byClients, byNode, calibrationOf, cellsTable, checkRun, cidrOverlap, cidrRange, deletionOf, deletionsOf, dropOf, emptyOf, factsOf, failuresOf, goDuration, intakeOf, isTunnelState, judge, judgeKind, kernelCounts, kindCellRecordSchema, kindCellRecordV1Schema, kindConfig, kubeProxyPatch, medianInterval, medianText, middle, nodeNames, observationAfter, observationEnd, outcomesOf, podsAt, podsOf, predictionText, propagationOf, propagationSource, quantileOf, readRecord, recordsOf, referenceOf, requestsOf, restartOf, restartsAt, reusesReset, rulesOf, scenario, scenarioProblems, seconds, secondsOf, showBand, signalOf, signalsOf, sliceDropOf, sliceEmptyOf, slicesOf, socketsOf, stoppingOf, timeSpread, transitionOf, tunnelsOf, unitOf, updatesOf, upgrade, upgradeCell, ztunnelPodsOf, ztunnelReplaced };
|
|
813
819
|
//# sourceMappingURL=index.d.ts.map
|
package/build/index.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { $ as signalOf, A as medianInterval, B as emptyOf, C as propagationOf, D as unitOf, E as reusesReset, F as backendsAt, G as outcomesOf, H as intakeOf, I as byClients, J as recordsOf, K as podsAt, L as byNode, M as quantileOf, N as HBONE_PORT, O as updatesOf, P as TCP_STATE, Q as rulesOf, R as deletionOf, S as observationEnd, T as referenceOf, U as isTunnelState, V as failuresOf, W as kernelCounts, X as restartOf, Y as requestsOf, Z as restartsAt, _ as checkRun, _t as seconds, a as calibrationOf, at as stoppingOf, b as judgeKind, c as predictionText, ct as ztunnelPodsOf, d as FORMAT_V1, dt as HEALTH_PORT, et as signalsOf, f as judge, ft as LOG_LABEL, g as LAST_SLACK, gt as bandOf, h as KIND_NOISE, ht as SERVER, i as readRecord, it as socketsOf, j as middle, k as RELATIVE, l as timeSpread, lt as ztunnelReplaced, m as upgradeCell, mt as REDIS, n as kindCellRecordSchema, nt as sliceEmptyOf, o as cellsTable, ot as transitionOf, p as upgrade, pt as PORT, q as podsOf, r as kindCellRecordV1Schema, rt as slicesOf, s as medianText, st as tunnelsOf, t as STAGES, tt as sliceDropOf, u as FORMAT, ut as CLIENT, v as deletionsOf, vt as secondsOf, w as propagationSource, x as observationAfter, y as factsOf, yt as showBand, z as dropOf } from "./judge-
|
|
1
|
+
import { $ as signalOf, A as medianInterval, B as emptyOf, C as propagationOf, D as unitOf, E as reusesReset, F as backendsAt, G as outcomesOf, H as intakeOf, I as byClients, J as recordsOf, K as podsAt, L as byNode, M as quantileOf, N as HBONE_PORT, O as updatesOf, P as TCP_STATE, Q as rulesOf, R as deletionOf, S as observationEnd, T as referenceOf, U as isTunnelState, V as failuresOf, W as kernelCounts, X as restartOf, Y as requestsOf, Z as restartsAt, _ as checkRun, _t as seconds, a as calibrationOf, at as stoppingOf, b as judgeKind, c as predictionText, ct as ztunnelPodsOf, d as FORMAT_V1, dt as HEALTH_PORT, et as signalsOf, f as judge, ft as LOG_LABEL, g as LAST_SLACK, gt as bandOf, h as KIND_NOISE, ht as SERVER, i as readRecord, it as socketsOf, j as middle, k as RELATIVE, l as timeSpread, lt as ztunnelReplaced, m as upgradeCell, mt as REDIS, n as kindCellRecordSchema, nt as sliceEmptyOf, o as cellsTable, ot as transitionOf, p as upgrade, pt as PORT, q as podsOf, r as kindCellRecordV1Schema, rt as slicesOf, s as medianText, st as tunnelsOf, t as STAGES, tt as sliceDropOf, u as FORMAT, ut as CLIENT, v as deletionsOf, vt as secondsOf, w as propagationSource, x as observationAfter, y as factsOf, yt as showBand, z as dropOf } from "./judge-Bb4ebE9V.js";
|
|
2
2
|
import { valueIn } from "@mark1russell7/quantity";
|
|
3
3
|
import { execFile, execFileSync } from "node:child_process";
|
|
4
4
|
import { chmodSync, copyFileSync, existsSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, renameSync, rmSync, writeFileSync } from "node:fs";
|
|
@@ -2414,7 +2414,8 @@ const serverObjects = (cell, namespace, image, version = 1) => {
|
|
|
2414
2414
|
/**
|
|
2415
2415
|
* These are the objects of the clients: a DaemonSet with one client on each node, the control plane
|
|
2416
2416
|
* too. The client of the service is `client.mjs`, and the client of grpc-go is `client-grpc.mjs`.
|
|
2417
|
-
* Each one is in the lab image, and each one writes the same lines.
|
|
2417
|
+
* Each one is in the lab image, and each one writes the same lines. A service with the protocol
|
|
2418
|
+
* `h2c` gives its clients `PROTOCOL`, thus they speak HTTP/2 to it.
|
|
2418
2419
|
*/
|
|
2419
2420
|
const clientObjects = (cell, namespace, image, target) => [{
|
|
2420
2421
|
apiVersion: "apps/v1",
|
|
@@ -2460,6 +2461,10 @@ const clientObjects = (cell, namespace, image, target) => [{
|
|
|
2460
2461
|
name: "CONNECTIONS",
|
|
2461
2462
|
value: String(cell.workload.connections)
|
|
2462
2463
|
}],
|
|
2464
|
+
...cell.declaration?.protocol === "h2c" ? [{
|
|
2465
|
+
name: "PROTOCOL",
|
|
2466
|
+
value: "h2c"
|
|
2467
|
+
}] : [],
|
|
2463
2468
|
{
|
|
2464
2469
|
name: "POD_NAME",
|
|
2465
2470
|
valueFrom: { fieldRef: { fieldPath: "metadata.name" } }
|
|
@@ -2583,9 +2588,11 @@ const pausedWindow = (declaration, preStop, lead, hold, propagation) => {
|
|
|
2583
2588
|
/**
|
|
2584
2589
|
* This function gives the learn window of a declaration, its readiness probe, a propagation and a
|
|
2585
2590
|
* workload of keep-alive connections. A caller selects a replica only when it opens a connection:
|
|
2586
|
-
* - The lame duck (`lameDuck` true or no value): after the latch, each answer
|
|
2591
|
+
* - The lame duck (`lameDuck` true or no value): after the latch, each answer closes its connection,
|
|
2587
2592
|
* thus each connection of the replica gets one more failure and opens again. The rules select the
|
|
2588
2593
|
* replica until the removal, thus the last failure comes in one turn after it.
|
|
2594
|
+
* - The signal of the lame duck is `Connection: close` on HTTP/1.1, and GOAWAY on HTTP/2. The
|
|
2595
|
+
* arithmetic is the same: the turn is `every` times the connections, which are sessions on HTTP/2.
|
|
2589
2596
|
* - No lame duck: no connection opens again, thus the rules do not matter. The failures end only
|
|
2590
2597
|
* when the readiness of the service recovers after the heal, and one turn after it.
|
|
2591
2598
|
*/
|