@mark1russell7/service 0.5.11 → 0.5.12

package/README.md

@@ -15,6 +15,9 @@ This release has the declaration, its export, the law, the health runtime, the l
 - **The law.** `evaluate` and `verdictOf` decide one rule of the timer-ordering law on one binding. A rule is data: linear inequalities over `symbol@hop`, with a margin. A verdict is `holds`, `violated`, `unbounded` (it holds only because a timer is absent) or `depends` (nothing is bound).
 - **The linter.** `lint(description, scenario, rules)` applies the service rules (`SERVICE_RULES`) to a declaration. The declaration binds the symbols of the service: the polls of its checks, the windows of its policies, its drain. The scenario binds the symbols of the platform and of each dependency. A `depends` finding names the symbols that have no bound.
 - **The scopes of a rule.** A rule about a client applies one time for each dependency. A rule about a consumer, for example O20, applies one time for each consumer and each worker.
+- **The loops of a progress check.** A check of the step `progress` can name the loops that mark it, with its stall time: `marks: { loops, stall }`. Each loop is a check or a worker of the service.
+- **The stall time.** `lint()` compares the stall time with the longest gap between two marks of the slowest loop (`PROGRESS_RULE`). A loop marks at the end of each turn, also a turn that fails. The gap of a check is `interval × (1 + jitter) + timeout`. The gap of a worker is its sleep or its longest backoff, the longer one, plus `unit_processing_time`. At 300 s ± 20 % and a timeout of 10 s, a poll gives 370 s.
+- **A name that is not a loop.** A name that is not a check or a worker gets a `depends` finding. Without `marks`, a check gets no finding of this rule.
 - **Keyed limiters.** `svc.limiter(name, spec)` declares a rate limit for each key, as data. `keyedLimiter` makes its runtime on a store ([The keyed limiter](#the-keyed-limiter)).
 - **The hold.** The cap of the `fail readiness` phase binds `readiness_hold@process`, and a service without that phase holds for zero. The hold keeps the listener open after the signal, as a preStop sleep does. Thus O04 can compare the sum of the two with the propagation of the endpoints.
 
@@ -510,13 +513,27 @@ const life = await app.run(); // the process exits after SIGTERM, with 0 or 1
 - **Redis Cluster.** The names of the keys put the limiter in braces, a hash tag. Thus the state and the index of a limiter are in one slot. The prefix has no braces.
 - **The test clock.** With `{ clock }`, the script uses the time of that clock in place of `TIME`. The tests use it to compare each decision of Redis with `decide`.
 
+## The PostgreSQL client
+
+`@mark1russell7/service/pg` is the PostgreSQL client of the service, over a pool of node-postgres 8. This entry is a prototype of S6. pg is an optional peer dependency. The client loads it only when it connects.
+
+- **The declaration.** `pgClient(name, spec)` declares `connect_bound`, `request_timeout`, `statement_timeout`, `pool_idle_expiry`, `keepalive_idle`, `max_connection_age`, `close_bound` and `retrans_giveup_time`. Each knob is necessary, and a bound that nobody gives is "infinite". `specProblems` refuses a knob that pg cannot use, for example a zero, which pg reads as no timer.
+- **The starting knobs.** `PG_STARTING_KNOBS` and `PG_STARTING_POLL` are a declaration from the cells of S1. Each value has its measured reason in its doc comment.
+- **Three outcomes.** `query(operation, text, values)` gives `ok`, `error` or `indeterminate`. An ERROR of the server is certain. A FATAL, a reset or a query timeout is not certain. The client then gives the connection back with an error, thus the pool ends the orphaned query.
+- **Idempotency for each operation.** SQL text does not tell if a statement changes a state. Thus the client sends again only an idempotent operation that its table names, one time at most, in the bound of the request.
+- **The join.** At each connect, the client records the process id of the backend and the socket (`joins`).
+- **No unheard error.** The client listens to the errors of the pool and of each client that it holds. In Node, an error with no listener ends the process.
+- **The checks.** `checks({ poll, steps })` gives `local-state`, `checkout`, `round-trip` (a `SELECT 1` through the pool) and `dial` (an SSLRequest on a new connection).
+- **The close.** At the shutdown, the client ends the pool, and it destroys each socket that stays at its `close_bound`.
+- **The manifestations and the machine.** `meets` lists the manifestations and their detectors. `PG_CLIENT_MACHINE` is the client as a `mark1russell7.machine/1` document, and the tests replay the client against it.
+
 ## Installation
 
 ```sh
 pnpm add @mark1russell7/service
 ```
 
-For the Redis client, the application also installs `ioredis@6`.
+For the Redis client, the application also installs `ioredis@6`. For the PostgreSQL client, it installs `pg@8`.
 
 The package uses [`@mark1russell7/quantity`](https://www.npmjs.com/package/@mark1russell7/quantity) for each amount. No entry of the service loads `zod` at run time, thus `zod` is an optional peer dependency. An application that validates amounts with the schemas of `quantity` installs `zod`.
 

package/build/{builder-Dn0cby6Q.d.ts → builder-DawJhq4p.d.ts}

@@ -1,6 +1,6 @@
 import { n as Clock } from "./clock-mVqhXD0b.js";
 import { c as LimiterSpec } from "./policy-G4QJ1ZZq.js";
-import { B as Gate, Ct as Operation, D as Outcome, E as CheckFn, F as CooperativeFloor, H as Phase, I as Coupling, J as Poll, L as Dependency, N as Check, R as ExitCodes, T as StartOptions, U as RepeatSignal, V as MessageTrigger, W as ServiceDescription, X as Timing, ct as ExitTrigger, d as TransportImpl, k as Trace, mt as ProcessTrigger, n as HealthServer, nt as OperationOf, o as PhaseWork, r as Lifecycle, s as RunOptions, t as DependencyHandle, x as Health } from "./lifecycle-CGEegMI0.js";
+import { B as Gate, D as Outcome, E as CheckFn, F as CooperativeFloor, G as ServiceDescription, H as Phase, I as Coupling, L as Dependency, N as Check, R as ExitCodes, T as StartOptions, V as MessageTrigger, W as RepeatSignal, Y as Poll, Z as Timing, d as TransportImpl, ht as ProcessTrigger, k as Trace, lt as ExitTrigger, n as HealthServer, o as PhaseWork, r as Lifecycle, rt as OperationOf, s as RunOptions, t as DependencyHandle, wt as Operation, x as Health } from "./lifecycle-fgYfxrHK.js";
 import { Extent, Quantity } from "@mark1russell7/quantity";
 //#region src/profiles.d.ts
 /**
@@ -208,4 +208,4 @@ interface ServiceBuilder {
 declare const service: (name: string) => ServiceBuilder;
 //#endregion
 export { WatchdogSpec as a, Profile as c, kubernetes as d, localDev as f, Watchdog as i, daemon as l, ServiceBuilder as n, watchdog as o, service as r, PodTiming as s, ControllerBuilder as t, ephemeral as u };
-//# sourceMappingURL=builder-Dn0cby6Q.d.ts.map
+//# sourceMappingURL=builder-DawJhq4p.d.ts.map

package/build/core-C2wnF4mg.d.ts

@@ -0,0 +1,183 @@
+import { n as Clock } from "./clock-mVqhXD0b.js";
+import { B as Gate, E as CheckFn, G as ServiceDescription, K as Step, L as Dependency, M as Recorder, N as Check, Y as Poll, t as DependencyHandle, wt as Operation } from "./lifecycle-fgYfxrHK.js";
+import { n as ServiceBuilder } from "./builder-DawJhq4p.js";
+import { Extent, Quantity } from "@mark1russell7/quantity";
+//#region src/client/meets.d.ts
+/** A situation is what the connection did when the failure started: a command in flight, or idle. */
+type Situation = "in flight" | "idle";
+/** A detector is a mechanism that finds a failure, and the most time that it takes. */
+interface Detector {
+  /** This name identifies the mechanism: a knob of the declaration, or the kernel. */
+  readonly by: string;
+  /** This time is the most that the detector takes, after the failure starts. "infinite" tells that it does not find the failure. */
+  readonly within: Extent;
+  readonly situation: Situation;
+  /** This text tells what the detector does, for example that it fails a command without a change of state. */
+  readonly effect: string;
+}
+/**
+ * A meeting is a manifestation that the client can meet, and its detectors (E9). The names of the
+ * manifestations are the names of the vocabulary of the harness and the corpus. `app-stall` is a
+ * server whose process is alive and does not answer, a proposal of the vocabulary.
+ */
+interface Meeting {
+  readonly manifestation: string;
+  /** This text tells the failure in words. */
+  readonly what: string;
+  readonly detectors: readonly Detector[];
+}
+/** These are the manifestations that each client gives a meeting for, in this sequence. */
+declare const MANIFESTATIONS: readonly ["rst", "fin", "blackhole", "freeze-host", "freeze-process", "app-stall"];
+/** A manifestation is one name of `MANIFESTATIONS`. */
+type Manifestation = (typeof MANIFESTATIONS)[number];
+/** This function gives the first detector of a manifestation in a situation, or null when no detector has an end (C3). */
+declare const firstDetector: (meeting: Meeting, situation: Situation) => Detector | null;
+/** This function gives each manifestation and situation that no detector of the client finds: the standing rule, as a query (C3, E9). */
+declare const unwatched: (meets: readonly Meeting[]) => Array<{
+  readonly manifestation: string;
+  readonly situation: Situation;
+}>;
+//#endregion
+//#region src/client/core.d.ts
+/**
+ * A sent result is the outcome of a call, in three values (C8):
+ * - `ok`: the server answered.
+ * - `error`: the call did not apply. The server answered with an error (`reply`), or the call did
+ *   not leave the client.
+ * - `indeterminate`: the call can have reached the server, and nothing tells if it applied.
+ */
+type Sent<T> = {
+  readonly outcome: "ok";
+  readonly value: T;
+  readonly attempts: number;
+} | {
+  readonly outcome: "error";
+  readonly error: string;
+  readonly reply: boolean;
+  readonly attempts: number;
+} | {
+  readonly outcome: "indeterminate";
+  readonly error: string;
+  readonly attempts: number;
+};
+/** This type is the outcome of one try of a call, before the core counts the tries. */
+type Attempt<T = unknown> = {
+  readonly outcome: "ok";
+  readonly value: T;
+} | {
+  readonly outcome: "error";
+  readonly error: string;
+  readonly reply: boolean;
+} | {
+  readonly outcome: "indeterminate";
+  readonly error: string;
+};
+/** Send options tell how the client sends one call. */
+interface SendOptions {
+  /** This signal abandons the call. A call that left the client is then indeterminate. */
+  readonly signal?: AbortSignal | undefined;
+  /** False stops the client: it does not send the call again. A check that must fail at once uses it. */
+  readonly reissue?: boolean | undefined;
+}
+/** A dial function opens a new connection to the server, and reads one reply of the protocol. The `dial` check uses it. */
+type DialFn = (host: string, port: number, signal: AbortSignal) => Promise<void>;
+/** The steps of a connection that each client gives a check for (B14). A client can add more steps. */
+type ClientStep = Extract<Step, "dial" | "round-trip" | "local-state">;
+/** Check options tell which checks the client gives the health runtime, and which policy each one controls. */
+interface CheckOptions<S extends Step = ClientStep> {
+  readonly poll: Poll;
+  /** For each step, the policy that its check controls, or null for a check that only reports. A step that is absent has no check. */
+  readonly steps: Partial<Record<S, Gate | null>>;
+}
+/**
+ * These are the knobs that each client has, by the names of the ordering law. Each driver adds its
+ * own knobs. The comments of each driver tell which option of the driver a knob sets.
+ */
+interface ClientKnobs {
+  /** This time bounds a new connection. */
+  readonly connect_bound: Quantity;
+  /** This time bounds each call on the connection of the application. */
+  readonly request_timeout: Quantity;
+  /** This time is the idle time of TCP keepalive. Node then sends 10 probes 1 s apart. */
+  readonly keepalive_idle: Extent;
+  /** This time is the age at which the client replaces a connection. */
+  readonly max_connection_age: Extent;
+  /** This time bounds the close of the client at the shutdown. */
+  readonly close_bound: Quantity;
+  /**
+   * This time is when the kernel gives up a segment that it retransmits. Node cannot set it, thus
+   * the declaration states it for the ordering law (O17), from the platform.
+   */
+  readonly retrans_giveup_time: Extent;
+}
+/** A client spec is the declaration of one client: where it connects, and its knobs. Each driver adds its own fields. */
+interface ClientSpec<K extends ClientKnobs = ClientKnobs> {
+  readonly host: string;
+  readonly port: number;
+  readonly password?: string | undefined;
+  /** This identifier is the component in the corpus, for example `systems.component.redis`, or null. */
+  readonly system: string | null;
+  readonly knobs: K;
+  /** These operations add to the table of the client, or change it. */
+  readonly operations?: Readonly<Record<string, Operation>> | undefined;
+}
+/** A factory makes the connection or the pool of a client from the options of its driver. */
+type Factory<O, C> = (options: O) => C | Promise<C>;
+/** These options make a client. Without them, a client uses its driver, the real clock, and the dial of its protocol. */
+interface ClientOptions<F> {
+  readonly create?: F | undefined;
+  readonly clock?: Clock | undefined;
+  /** This function receives each transition of the client, for the prober (D3). */
+  readonly record?: Recorder | undefined;
+  readonly dial?: DialFn | undefined;
+}
+/** A client socket is the part of the socket of a connection that the join of the prober and the bounded close read. */
+interface ClientSocket {
+  readonly localAddress?: string | undefined;
+  readonly localPort?: number | undefined;
+  readonly remoteAddress?: string | undefined;
+  readonly remotePort?: number | undefined;
+  /** This method closes the socket at once. A close of the driver only half-closes it, and a silent peer does not answer that. */
+  destroy?(): void;
+}
+/**
+ * A client join is one connection, at the time when it can serve (D1, D2). The prober joins its
+ * kernel records to the server's view of the connection through it. Each driver adds the server's id of the connection.
+ */
+interface ClientJoin {
+  readonly generation: number;
+  /** This time is when the connection became ready, in milliseconds on the monotonic clock. */
+  readonly at: number;
+  readonly localAddress: string | null;
+  readonly localPort: number | null;
+  readonly remoteAddress: string | null;
+  readonly remotePort: number | null;
+}
+/** A client is a dependency of a service: its declaration, its handle and its checks. Each driver adds the method that sends a call. */
+interface Client<S extends Step = ClientStep, J extends ClientJoin = ClientJoin> {
+  readonly name: string;
+  /** This declaration goes to `svc.dependency(name, declaration, handle)`. */
+  readonly declaration: Omit<Dependency, "name">;
+  readonly handle: DependencyHandle;
+  /** These are the manifestations that the client can meet, and the detectors of each (E9). */
+  readonly meets: readonly Meeting[];
+  /** This text is the state of the client, as the machine document of the client names it. */
+  readonly status: string;
+  /** These are the connections of the client, the latest last. */
+  readonly joins: readonly J[];
+  /** This method gives the checks of the client and their functions, for `svc.check(check, run)`. */
+  checks(options: CheckOptions<S>): ReadonlyArray<readonly [Check, CheckFn]>;
+  /** This method declares the client in a service: its dependency, its handle and its checks. */
+  attach(svc: ServiceBuilder, options: CheckOptions<S>): ServiceBuilder;
+  /** This method gives the problems of the client in a description: a close that does not fit in its phase, for example. */
+  problems(description: ServiceDescription): string[];
+}
+/**
+ * This poll is a poll of the checks to start from: each 5 s with a jitter of 20 %, and a timeout of
+ * 4 s (B6, O12). Its slowest poll is 4 s + 5 s × 1.2 = 10 s, the slowest poll of the cells of O13.
+ * It does not depend on a driver. The starting knobs of each driver fit it.
+ */
+declare const STARTING_POLL: Poll;
+//#endregion
+export { Manifestation as _, ClientKnobs as a, firstDetector as b, ClientSpec as c, Factory as d, STARTING_POLL as f, MANIFESTATIONS as g, Detector as h, ClientJoin as i, ClientStep as l, Sent as m, CheckOptions as n, ClientOptions as o, SendOptions as p, Client as r, ClientSocket as s, Attempt as t, DialFn as u, Meeting as v, unwatched as x, Situation as y };
+//# sourceMappingURL=core-C2wnF4mg.d.ts.map

package/build/http.d.ts

@@ -1,6 +1,6 @@
 import { n as Clock } from "./clock-mVqhXD0b.js";
 import { d as Refused } from "./policy-G4QJ1ZZq.js";
-import { d as TransportImpl, m as RequestEnd, n as HealthServer, r as Lifecycle, v as RequestStream, w as ProbeKind, x as Health, y as RequestWork } from "./lifecycle-CGEegMI0.js";
+import { d as TransportImpl, m as RequestEnd, n as HealthServer, r as Lifecycle, v as RequestStream, w as ProbeKind, x as Health, y as RequestWork } from "./lifecycle-fgYfxrHK.js";
 import { Extent, Quantity } from "@mark1russell7/quantity";
 import { ListenOptions } from "node:net";
 import { IncomingMessage, Server, ServerEventMap, ServerResponse } from "node:http";

package/build/index.d.ts

@@ -2,8 +2,8 @@ import { a as backoffDelay, i as backoffBand, n as DelayBand, o as backoffProble
 import { a as realClock, i as platformClock, n as Clock, r as PlatformClockOptions, t as Cancel } from "./clock-mVqhXD0b.js";
 import { Heartbeat, HeartbeatOptions, detectionTime, heartbeat } from "./heartbeat.js";
 import { S as retryAfterSeconds, _ as freeAt, a as LimitRule, b as limiterProblems, c as LimiterSpec, d as Refused, f as SlidingWindow, g as decide, h as TokenBucket, i as LimitPolicy, l as LimiterStoreSpec, m as TakeStep, n as Decision, o as Limiter, p as StoreFailure, r as KeyState, s as LimiterEviction, t as Allowed, u as RefusalCause, v as idleSpan, x as longestRetryAfter, y as limitRule } from "./policy-G4QJ1ZZq.js";
-import { $ as failSpan, A as LifecycleEvent, B as Gate, C as ProbeAnswer, Ct as Operation, D as Outcome, E as CheckFn, F as CooperativeFloor, G as Step, H as Phase, I as Coupling, J as Poll, K as Transport, L as Dependency, M as Recorder, N as Check, O as Result, P as Controller, Q as asWindow, R as ExitCodes, S as HealthVerdict, St as Idempotency, T as StartOptions, U as RepeatSignal, V as MessageTrigger, W as ServiceDescription, X as Timing, Y as Span, Z as asDebounce, _ as RequestOptions, _t as isExitSignal, a as PhaseContext, at as EXIT_SIGNALS, b as CheckVerdict, bt as nodeProcess, c as STAGES, ct as ExitTrigger, d as TransportImpl, dt as PROCESS_TRIGGERS, et as heartbeatDetection, f as cooperativeAbortAt, ft as ProcessHost, g as RequestFn, gt as isExitCode, h as RequestEvent, ht as embeddedProcess, i as LifecycleState, it as RouteRequest, j as MachineName, l as ShutdownOptions, lt as HostStream, m as RequestEnd, mt as ProcessTrigger, n as HealthServer, nt as OperationOf, o as PhaseWork, ot as EmbeddedProcess, p as stageRank, pt as ProcessPort, q as FailsBy, r as Lifecycle, rt as RoutePattern, s as RunOptions, st as ExitSignal, t as DependencyHandle, tt as recoverSpan, u as Stage, ut as MessageShape, v as RequestStream, vt as isMessageShape, w as ProbeKind, wt as mayReissue, x as Health, xt as processPortOf, y as RequestWork, yt as matchesMessage, z as FORMAT } from "./lifecycle-CGEegMI0.js";
-import { a as WatchdogSpec, c as Profile, d as kubernetes, f as localDev, i as Watchdog, l as daemon, n as ServiceBuilder, o as watchdog, r as service, s as PodTiming, t as ControllerBuilder, u as ephemeral } from "./builder-Dn0cby6Q.js";
+import { $ as asWindow, A as LifecycleEvent, B as Gate, C as ProbeAnswer, Ct as Idempotency, D as Outcome, E as CheckFn, F as CooperativeFloor, G as ServiceDescription, H as Phase, I as Coupling, J as FailsBy, K as Step, L as Dependency, M as Recorder, N as Check, O as Result, P as Controller, Q as asDebounce, R as ExitCodes, S as HealthVerdict, St as processPortOf, T as StartOptions, Tt as mayReissue, U as ProgressMarks, V as MessageTrigger, W as RepeatSignal, X as Span, Y as Poll, Z as Timing, _ as RequestOptions, _t as isExitCode, a as PhaseContext, at as RouteRequest, b as CheckVerdict, bt as matchesMessage, c as STAGES, ct as ExitSignal, d as TransportImpl, dt as MessageShape, et as failSpan, f as cooperativeAbortAt, ft as PROCESS_TRIGGERS, g as RequestFn, gt as embeddedProcess, h as RequestEvent, ht as ProcessTrigger, i as LifecycleState, it as RoutePattern, j as MachineName, l as ShutdownOptions, lt as ExitTrigger, m as RequestEnd, mt as ProcessPort, n as HealthServer, nt as recoverSpan, o as PhaseWork, ot as EXIT_SIGNALS, p as stageRank, pt as ProcessHost, q as Transport, r as Lifecycle, rt as OperationOf, s as RunOptions, st as EmbeddedProcess, t as DependencyHandle, tt as heartbeatDetection, u as Stage, ut as HostStream, v as RequestStream, vt as isExitSignal, w as ProbeKind, wt as Operation, x as Health, xt as nodeProcess, y as RequestWork, yt as isMessageShape, z as FORMAT } from "./lifecycle-fgYfxrHK.js";
+import { a as WatchdogSpec, c as Profile, d as kubernetes, f as localDev, i as Watchdog, l as daemon, n as ServiceBuilder, o as watchdog, r as service, s as PodTiming, t as ControllerBuilder, u as ephemeral } from "./builder-DawJhq4p.js";
 import { a as keyedLimiter, i as TakeRequest, n as LimiterStore, o as memoryStore, r as MemoryStore, t as KeyedLimiter } from "./store-D5HLgBnW.js";
 import { a as limiterStateOf, i as limiterMachine, n as SLIDING_WINDOW_MACHINE, r as TOKEN_BUCKET_MACHINE, t as LimiterState } from "./limiter-B0849Vhp.js";
 import { Extent, Quantity } from "@mark1russell7/quantity";
@@ -152,10 +152,13 @@ export declare const serviceBindings: (svc: ServiceDescription, scenario: Scenar
 };
 /** This identifier is the rule of the findings about a field that the declaration does not give. */
 export declare const DECLARATION_RULE = "declaration";
+/** This identifier is the rule of the findings about the stall time of a progress check (`Check.marks`). */
+export declare const PROGRESS_RULE = "progress";
 /**
  * This function applies each service rule to a declaration and a scenario. A rule applies one time
  * for the service, for each client, for each consumer and worker, or for each limiter. The function
  * also gives a finding for each necessary field that the description does not give (`DECLARATION_RULE`).
+ * Each progress check that names its loops also gets its findings (`PROGRESS_RULE`).
  */
 export declare const lint: (svc: ServiceDescription, scenario: Scenario, rules: readonly Rule[]) => Finding[];
 //#endregion
@@ -433,5 +436,5 @@ export declare const drainMachine: (description: ServiceDescription) => Machine;
 /** These are the static machines of the runtime, by the name that their events carry. */
 export declare const MACHINES: Readonly<Record<Exclude<MachineName, "drain" | "client">, Machine>>;
 //#endregion
-export { type Allowed, type Backoff, type Binding, type Bound, type Cancel, type Check, type CheckFn, type CheckVerdict, type Clock, type Controller, type ControllerBuilder, type CooperativeFloor, type Coupling, type Decision, type DelayBand, type Dependency, type DependencyHandle, EXIT_SIGNALS, type EmbeddedProcess, type Evaluated, type ExitCodes, type ExitSignal, type ExitTrigger, FORMAT, type FailsBy, type Finding, type Gate, type Health, type HealthServer, type HealthVerdict, type Heartbeat, type HeartbeatOptions, type HostStream, type Idempotency, type Inequality, type IterationContext, type Jitter, type JobAction, type JobContext, type JobFailure, type JobSettle, type JobSource, type KeyState, type KeyedLimiter, type Lifecycle, type LifecycleEvent, type LifecycleState, type LimitPolicy, type LimitRule, type Limiter, type LimiterEviction, type LimiterSpec, type LimiterState, type LimiterStore, type LimiterStoreSpec, type MachineName, type MemoryStore, type MessageShape, type MessageTrigger, type Operation, type OperationOf, type Outcome, PROCESS_TRIGGERS, type Phase, type PhaseContext, type PhaseWork, type PlatformClockOptions, type PodTiming, type Poll, type ProbeAnswer, type ProbeKind, type ProcessHost, type ProcessPort, type ProcessTrigger, type Profile, type QueueDrain, type QueueDrainOptions, type QueueKnobs, type Recorder, type RedeliveryPolicy, type RefusalCause, type Refused, type RepeatSignal, type RequestEnd, type RequestEvent, type RequestFn, type RequestOptions, type RequestStream, type RequestWork, type RequeueCause, type Result, type RoutePattern, type RouteRequest, type Rule, type RunOptions, SLIDING_WINDOW_MACHINE, STAGES, type Scenario, type ServiceBuilder, type ServiceDescription, type ShutdownOptions, type Side, type SlidingWindow, type Span, type Stage, type StartOptions, type Step, type StoreFailure, TOKEN_BUCKET_MACHINE, type TakeRequest, type TakeStep, type Term, type Timing, type TokenBucket, type Transport, type TransportImpl, type UnitRequest, type Verdict, type Watchdog, type WatchdogSpec, type WorkerKnobs, type WorkerLoop, type WorkerLoopOptions, asDebounce, asWindow, backoffBand, backoffDelay, backoffProblems, cooperativeAbortAt, daemon, decide, detectionTime, embeddedProcess, ephemeral, failSpan, freeAt, heartbeat, heartbeatDetection, idleSpan, isExitCode, isExitSignal, isMessageShape, keyedLimiter, kubernetes, limitRule, limiterMachine, limiterProblems, limiterStateOf, localDev, longestRetryAfter, matchesMessage, mayReissue, memoryStore, nodeProcess, platformClock, processPortOf, realClock, recoverSpan, replay, retryAfterSeconds, service, stageRank, watchdog };
+export { type Allowed, type Backoff, type Binding, type Bound, type Cancel, type Check, type CheckFn, type CheckVerdict, type Clock, type Controller, type ControllerBuilder, type CooperativeFloor, type Coupling, type Decision, type DelayBand, type Dependency, type DependencyHandle, EXIT_SIGNALS, type EmbeddedProcess, type Evaluated, type ExitCodes, type ExitSignal, type ExitTrigger, FORMAT, type FailsBy, type Finding, type Gate, type Health, type HealthServer, type HealthVerdict, type Heartbeat, type HeartbeatOptions, type HostStream, type Idempotency, type Inequality, type IterationContext, type Jitter, type JobAction, type JobContext, type JobFailure, type JobSettle, type JobSource, type KeyState, type KeyedLimiter, type Lifecycle, type LifecycleEvent, type LifecycleState, type LimitPolicy, type LimitRule, type Limiter, type LimiterEviction, type LimiterSpec, type LimiterState, type LimiterStore, type LimiterStoreSpec, type MachineName, type MemoryStore, type MessageShape, type MessageTrigger, type Operation, type OperationOf, type Outcome, PROCESS_TRIGGERS, type Phase, type PhaseContext, type PhaseWork, type PlatformClockOptions, type PodTiming, type Poll, type ProbeAnswer, type ProbeKind, type ProcessHost, type ProcessPort, type ProcessTrigger, type Profile, type ProgressMarks, type QueueDrain, type QueueDrainOptions, type QueueKnobs, type Recorder, type RedeliveryPolicy, type RefusalCause, type Refused, type RepeatSignal, type RequestEnd, type RequestEvent, type RequestFn, type RequestOptions, type RequestStream, type RequestWork, type RequeueCause, type Result, type RoutePattern, type RouteRequest, type Rule, type RunOptions, SLIDING_WINDOW_MACHINE, STAGES, type Scenario, type ServiceBuilder, type ServiceDescription, type ShutdownOptions, type Side, type SlidingWindow, type Span, type Stage, type StartOptions, type Step, type StoreFailure, TOKEN_BUCKET_MACHINE, type TakeRequest, type TakeStep, type Term, type Timing, type TokenBucket, type Transport, type TransportImpl, type UnitRequest, type Verdict, type Watchdog, type WatchdogSpec, type WorkerKnobs, type WorkerLoop, type WorkerLoopOptions, asDebounce, asWindow, backoffBand, backoffDelay, backoffProblems, cooperativeAbortAt, daemon, decide, detectionTime, embeddedProcess, ephemeral, failSpan, freeAt, heartbeat, heartbeatDetection, idleSpan, isExitCode, isExitSignal, isMessageShape, keyedLimiter, kubernetes, limitRule, limiterMachine, limiterProblems, limiterStateOf, localDev, longestRetryAfter, matchesMessage, mayReissue, memoryStore, nodeProcess, platformClock, processPortOf, realClock, recoverSpan, replay, retryAfterSeconds, service, stageRank, watchdog };
 //# sourceMappingURL=index.d.ts.map

package/build/index.js

@@ -1656,6 +1656,25 @@ const watchdog = (clock, spec, trace) => {
 const isTime$1 = (amount) => convertible(amount.unit, "s");
 const above = (amount) => amount.value === "infinite" || isTime$1(amount) && valueIn(amount, "s") > 0;
 const whole = (count) => Number.isInteger(count) && count >= 1;
+/** This function copies a check, with its poll and the loops that mark it. */
+const copyCheck = (check) => ({
+	...check,
+	poll: { ...check.poll },
+	...check.marks === void 0 ? {} : { marks: {
+		loops: [...check.marks.loops],
+		stall: { ...check.marks.stall }
+	} }
+});
+/** This function gives the problems of the loops that mark a check. `lint()` compares the stall time with the loops. */
+const marksProblems = (check) => {
+	const marks = check.marks;
+	if (marks === void 0) return [];
+	const problems = [];
+	if (check.step !== "progress") problems.push(`check ${check.name} names the loops that mark it, but its step is ${check.step}, not progress`);
+	if (marks.loops.length === 0) problems.push(`check ${check.name} names no loop that marks it`);
+	if (typeof marks.stall.value !== "number" || !above(marks.stall)) problems.push(`check ${check.name} has a stall time that is not a time above zero`);
+	return problems;
+};
 /** This function gives the problems of a timing: a zero interval or count fails decoding. */
 const timingProblems = (policy, timing) => {
 	if (timing.type === "debounce") {
@@ -1844,14 +1863,11 @@ const service = (name) => {
 		check(check, run) {
 			once(checks, check.name, "check");
 			if (started && run === void 0) throw new Error(`check ${check.name} joins after the start, thus it needs its function`);
-			checks.push({ ...check });
+			checks.push(copyCheck(check));
 			if (run !== void 0) runs.set(check.name, run);
 			if (started) {
 				partial = true;
-				live?.addCheck({
-					...check,
-					poll: { ...check.poll }
-				}, run);
+				live?.addCheck(copyCheck(check), run);
 			}
 			return builder;
 		},
@@ -1989,6 +2005,7 @@ const service = (name) => {
 				if (!isTime$1(check.poll.interval) || !isTime$1(check.poll.timeout)) problems.push(`check ${check.name} polls in a unit that is not a time`);
 				else if (!above(check.poll.interval) || !above(check.poll.timeout)) problems.push(`check ${check.name} polls with an interval or a timeout that is not above zero`);
 				if (!(check.poll.jitter >= 0 && check.poll.jitter < 1)) problems.push(`check ${check.name} has a jitter of ${check.poll.jitter}, not a fraction below 1`);
+				problems.push(...marksProblems(check));
 				const timing = check.gates === null ? null : policies[check.gates];
 				if (timing?.type === "debounce" && isTime$1(check.poll.interval) && above(timing.interval) && valueIn(check.poll.interval, "s") !== valueIn(timing.interval, "s")) problems.push(`check ${check.name} polls every ${valueIn(check.poll.interval, "s")} s, but the ${check.gates} debounce counts polls every ${valueIn(timing.interval, "s")} s`);
 			}
@@ -2031,10 +2048,7 @@ const service = (name) => {
 					operations: { ...entry.operations }
 				})),
 				controllers: controllersOut,
-				checks: checks.map((entry) => ({
-					...entry,
-					poll: { ...entry.poll }
-				})),
+				checks: checks.map(copyCheck),
 				...limiters.length === 0 ? {} : { limiters: limiters.map((entry) => ({
 					...entry,
 					policy: { ...entry.policy },
@@ -2385,14 +2399,144 @@ const declarationFindings = (svc) => svc.shutdown.onRepeatSignal === void 0 ? [{
 	evaluated: [],
 	unbound: ["shutdown.onRepeatSignal"]
 }] : [];
+/** This identifier is the rule of the findings about the stall time of a progress check (`Check.marks`). */
+const PROGRESS_RULE = "progress";
+/**
+* This rule compares the stall time of a progress check with the longest gap between two marks of
+* the slowest loop that it names. A shorter stall time trips the check while each loop turns on time.
+*/
+const PROGRESS_LAW = {
+	id: PROGRESS_RULE,
+	rule: "the stall time of a progress check is at least the longest gap between two marks of the slowest loop that it names",
+	window: "a loop that turns on time can trip the check",
+	expr: {
+		lhs: { terms: [{
+			knob: "stall",
+			coef: 1,
+			at: "check"
+		}] },
+		op: ">=",
+		rhs: { terms: [{
+			knob: "mark_gap",
+			coef: 1,
+			at: "loop"
+		}] },
+		margin: 0
+	}
+};
+/** These are the knobs of a worker that bound the gap between two of its marks. */
+const WORKER_GAP_KNOBS = [
+	"worker_sleep",
+	"unit_processing_time",
+	"retry_base",
+	"retry_cap"
+];
+/**
+* This function gives the longest gap between two marks of each loop with a name. The rule counts
+* a mark at the end of each turn of the loop, also a turn that fails.
+* - A check polls on a fixed delay from the start of each poll, and each poll ends at its timeout
+*   at the latest. Thus the gap is `interval × (1 + jitter) + timeout`.
+* - After an iteration, a lane of a worker waits its sleep, or its backoff after a failure. The
+*   longest backoff is `retry_cap` plus its jitter of up to `retry_base`. The next iteration then
+*   takes up to `unit_processing_time`. Thus the gap is the longer wait plus that bound.
+*
+* A sleep or a bound without a limit gives a gap without a limit. An iteration that does not end
+* at its abort can take more time, and the rule does not bound it.
+*/
+const gapsOf = (svc, loop) => {
+	const gaps = svc.checks.filter((check) => check.name === loop).map((check) => ({
+		loop,
+		seconds: secondsOf(check.poll.interval) * (1 + check.poll.jitter) + secondsOf(check.poll.timeout),
+		unbound: []
+	}));
+	for (const worker of svc.transports) {
+		if (worker.kind !== "worker" || worker.name !== loop) continue;
+		const knob = (name) => {
+			const amount = worker.knobs[name];
+			if (amount === void 0 || !isTime(amount.unit)) return void 0;
+			return amount.value === "infinite" ? Infinity : secondsOf(amount);
+		};
+		const unbound = WORKER_GAP_KNOBS.filter((name) => knob(name) === void 0).map((name) => `${loop}.${name}`);
+		if (unbound.length > 0) {
+			gaps.push({
+				loop,
+				seconds: void 0,
+				unbound
+			});
+			continue;
+		}
+		const [sleep, unit, base, cap] = WORKER_GAP_KNOBS.map((name) => knob(name));
+		const backoff = Number.isFinite(base) && Number.isFinite(cap) ? backoffBand({
+			base,
+			factor: 2,
+			cap,
+			jitter: {
+				kind: "add",
+				amount: base
+			}
+		}).high : Infinity;
+		gaps.push({
+			loop,
+			seconds: Math.max(sleep, backoff) + unit,
+			unbound: []
+		});
+	}
+	return gaps;
+};
+/**
+* This function gives the findings of each check that names the loops that mark it. A name that is
+* not a check or a worker of the service gets a finding of its own, with the verdict `depends`.
+* Without `marks`, a check gets no finding.
+*/
+const progressFindings = (svc) => {
+	const findings = [];
+	const finding = (check, fields) => ({
+		rule: PROGRESS_RULE,
+		scope: check.name,
+		statement: PROGRESS_LAW.rule,
+		window: PROGRESS_LAW.window,
+		...fields
+	});
+	for (const check of svc.checks) {
+		const marks = check.marks;
+		if (marks === void 0) continue;
+		const gaps = [];
+		for (const loop of marks.loops) {
+			const found = gapsOf(svc, loop);
+			if (found.length > 0) gaps.push(...found);
+			else findings.push(finding(check, {
+				verdict: "depends",
+				statement: "each loop that a progress check names is a check or a worker of the service",
+				window: `the check waits for the marks of ${loop}, and the service has no loop with that name`,
+				evaluated: [],
+				unbound: [loop]
+			}));
+		}
+		if (gaps.length === 0) continue;
+		const unbound = [...isTime(marks.stall.unit) ? [] : [`${check.name}.marks.stall`], ...gaps.flatMap((gap) => gap.unbound)];
+		const slowest = gaps.reduce((worst, gap) => (gap.seconds ?? -Infinity) > (worst.seconds ?? -Infinity) ? gap : worst);
+		const bind = {};
+		if (isTime(marks.stall.unit)) bind["stall@check"] = boundOf(marks.stall);
+		if (unbound.length === 0 && slowest.seconds !== void 0) bind["mark_gap@loop"] = Number.isFinite(slowest.seconds) ? slowest.seconds : "inf";
+		const evaluated = evaluate(PROGRESS_LAW, bind);
+		findings.push(finding(check, {
+			verdict: verdictOf(PROGRESS_LAW, bind, evaluated),
+			...unbound.length === 0 ? { statement: `${PROGRESS_LAW.rule} (${slowest.loop})` } : {},
+			evaluated,
+			unbound
+		}));
+	}
+	return findings;
+};
 /**
 * This function applies each service rule to a declaration and a scenario. A rule applies one time
 * for the service, for each client, for each consumer and worker, or for each limiter. The function
 * also gives a finding for each necessary field that the description does not give (`DECLARATION_RULE`).
+* Each progress check that names its loops also gets its findings (`PROGRESS_RULE`).
 */
 const lint = (svc, scenario, rules) => {
 	const service = serviceBindings(svc, scenario).bind;
-	const findings = [...declarationFindings(svc)];
+	const findings = [...declarationFindings(svc), ...progressFindings(svc)];
 	for (const rule of rules) {
 		if (!SERVICE_RULES.includes(rule.id)) continue;
 		const symbols = symbolsOf(rule);
@@ -3411,6 +3555,6 @@ const MACHINES = {
 	watchdog: WATCHDOG_MACHINE
 };
 //#endregion
-export { DECLARATION_RULE, EXIT_MACHINE, EXIT_SIGNALS, FORMAT, GATE_MACHINE, LATCH_MACHINE, LIMITER_RULES, MACHINES, POLLER_MACHINE, PROCESS_TRIGGERS, SERVICE_RULES, SLIDING_WINDOW_MACHINE, STAGES, TOKEN_BUCKET_MACHINE, WATCHDOG_MACHINE, WINDOW_MACHINE, asDebounce, asWindow, backoffBand, backoffDelay, backoffProblems, capKnob, cooperativeAbortAt, daemon, decide, detectionTime, drainMachine, embeddedProcess, ephemeral, evaluate, failSpan, freeAt, heartbeat, heartbeatDetection, idleSpan, inequalities, isExitCode, isExitSignal, isMessageShape, keyedLimiter, kubernetes, limitRule, limiterMachine, limiterProblems, limiterStateOf, lint, localDev, longestRetryAfter, matchesMessage, mayReissue, memoryStore, nodeProcess, platformClock, processPortOf, queueDrain, realClock, recoverSpan, redeliverOnce, replay, retryAfterSeconds, service, serviceBindings, stageRank, symbolsOf, verdictOf, watchdog, workerLoop };
+export { DECLARATION_RULE, EXIT_MACHINE, EXIT_SIGNALS, FORMAT, GATE_MACHINE, LATCH_MACHINE, LIMITER_RULES, MACHINES, POLLER_MACHINE, PROCESS_TRIGGERS, PROGRESS_RULE, SERVICE_RULES, SLIDING_WINDOW_MACHINE, STAGES, TOKEN_BUCKET_MACHINE, WATCHDOG_MACHINE, WINDOW_MACHINE, asDebounce, asWindow, backoffBand, backoffDelay, backoffProblems, capKnob, cooperativeAbortAt, daemon, decide, detectionTime, drainMachine, embeddedProcess, ephemeral, evaluate, failSpan, freeAt, heartbeat, heartbeatDetection, idleSpan, inequalities, isExitCode, isExitSignal, isMessageShape, keyedLimiter, kubernetes, limitRule, limiterMachine, limiterProblems, limiterStateOf, lint, localDev, longestRetryAfter, matchesMessage, mayReissue, memoryStore, nodeProcess, platformClock, processPortOf, queueDrain, realClock, recoverSpan, redeliverOnce, replay, retryAfterSeconds, service, serviceBindings, stageRank, symbolsOf, verdictOf, watchdog, workerLoop };
 
 //# sourceMappingURL=index.js.map