@zukhruf/single-flight 0.3.20 → 0.3.21

package/README.md

@@ -70,7 +70,7 @@ The lease that the work gets comes from `@zukhruf/lease`, and its token comes fr
 
 ## The directory
 
-The directory is the local folder where the processes that share flights meet. Each process that uses a `SingleFlight` keeps four files there: `flight.lock`, `flight.epoch`, `flight.lock.clean` and `flight.sock`. The election of [`@zukhruf/election`](../election/README.md) claims `flight.lock` with a file lock, and records the epoch of the last term in `flight.epoch`. After a clean shutdown of a coordinator, `flight.lock.clean` holds the epoch of its term until the next term starts. The journal of the claim is in memory, so a coordinator that stops leaves no `flight.lock-journal` file. All the processes that name the same directory are one group, and the callers of one key share a flight in that group only.
+The directory is the local folder where the processes that share flights meet. Each process that uses a `SingleFlight` keeps three files and a folder there: `flight.lock`, `flight.epoch`, `flight.sock` and `execution-locks/`. The election of [`@zukhruf/election`](../election/README.md) claims `flight.lock` with a file lock, and records the epoch of the last term in `flight.epoch`. The folder `execution-locks/` has one file for each key that ever led: the key lock of that key, named by the SHA-256 of the key. The process that runs the work of a key holds a file lock on that file (see [When a process stops](#when-a-process-stops)). The files stay after the flights, so do not remove them while a process uses the directory. The journal of the claim is in memory, so a coordinator that stops leaves no `flight.lock-journal` file. All the processes that name the same directory are one group, and the callers of one key share a flight in that group only.
 
 There is no default directory. One default would put each application on the host into one group, and unrelated applications that use the same key would join each other's flights. Give each group its own directory, for example a folder of your application's data.
 
@@ -87,15 +87,14 @@ All the processes of the directory take part in one election. The winner is the
 The work gets the lease of the flight:
 
 - `token`: a `FencingToken`. Its high 32 bits are the epoch of the coordinator's term, so a newer term always gives higher tokens.
-- `signal`: it aborts with `LeaseLostError` when the flight is no longer the leader's. The `subject` of the error is the key. This occurs when the leader misses the grace window of a new coordinator.
+- `signal`: it aborts with `LeaseLostError` when the flight is no longer the leader's, so its outcome reaches no joiner. The `subject` of the error is the key. This occurs when a new coordinator refuses the reassert of the leader, because it knows a newer flight of the key, or when this process cannot connect to a coordinator again.
 
-| Option        | What it does                                                                                                                                                                                                                                  |
-| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `directory`   | The local folder where the processes of one group meet. See [The directory](#the-directory).                                                                                                                                                  |
-| `codec`       | Turns the value of a flight into text for the joiners, and back. See [The codec](#the-codec).                                                                                                                                                 |
-| `graceWindow` | Milliseconds that a new coordinator starts no flight, so the leaders of the flights in progress can reassert them. It must be longer than a process takes to connect again. A coordinator after a clean shutdown has none. Defaults to `500`. |
+| Option      | What it does                                                                                  |
+| ----------- | --------------------------------------------------------------------------------------------- |
+| `directory` | The local folder where the processes of one group meet. See [The directory](#the-directory).  |
+| `codec`     | Turns the value of a flight into text for the joiners, and back. See [The codec](#the-codec). |
 
-`flights[Symbol.asyncDispose]()` waits until each landing of this process reached a coordinator, and then it disconnects. When this process is the coordinator, the other processes elect a new one.
+`flights[Symbol.asyncDispose]()` waits until each landing of this process reached a coordinator, and then it disconnects. When this process is the coordinator, the other processes elect a new one. The work of a flight that this process leads continues, and this process keeps the key lock of that key until the work ends.
 
 ## The codec
 
@@ -165,17 +164,26 @@ Give a signal to stop the wait of one caller: `flights.run(key, work, { signal:
 
 ## When a process stops
 
-- **The process of a leader stops.** The coordinator tells each joiner that the flight is interrupted, and each joiner rejects with `FlightInterruptedError` at once. No joiner runs the work again. The next call of the key leads a new flight.
-- **The coordinator disposes with nothing in progress.** This is a clean shutdown: no flight is in progress, and the grace window of the coordinator is over, or it had none. The next coordinator then has no grace window, and each call leads or joins at once. [ADR 0009](./docs/adr/0009-the-next-coordinator-skips-the-grace-window-after-a-clean-shutdown.md) tells why.
-- **The coordinator stops in another way.** For example, its process stops, or it disposes while a flight is in progress. The other processes elect a new coordinator. Its term starts with a grace window. In the grace window:
+A leader keeps its flight while its process lives. Before its work starts, the leader takes a kernel lock on its key, the key lock. It frees the key lock only when its work settled and its flight needs nothing more from it: the coordinator acknowledged its landing, its lease was lost, or its `SingleFlight` closed. The kernel keeps the key lock while the process lives, also while the process is frozen, and it frees the key lock when the process dies. Thus no other process starts the work of the key beside it. [ADR 0010](./docs/adr/0010-a-leader-holds-a-kernel-lock-on-its-key-until-its-outcome-is-handed-over.md) tells why.
+
+- **The process of a leader stops.** The coordinator tells each joiner that the flight is interrupted, and each joiner rejects with `FlightInterruptedError` at once. No joiner runs the work again. The kernel frees the key lock, and the next call of the key leads a new flight.
+- **A leader disposes its `SingleFlight` while its work runs.** The coordinator interrupts the flight, because a disposed leader cannot send its outcome. Each joiner rejects with `FlightInterruptedError`. The work continues, and the leader keeps the key lock until the work ends. A call of the key in another process waits until then, and then it leads a new flight.
+- **The coordinator stops or disposes.** The other processes elect a new coordinator, and it answers at once:
   - Each leader reasserts its flight, and sends its landing again when it has one.
   - Each joiner rejoins its flight by the flight's token, and it gets the outcome of that flight.
-  - A new call waits for the grace window to end. Then it leads or joins.
-- **A leader misses the grace window**, for example because its process was frozen. Its lease is lost: the signal of the lease aborts with `LeaseLostError`, and its call rejects with that error. When the work throws its own error after the loss, the call rejects with a new `LeaseLostError`, and the error of the work is its `cause`. Thus `LeaseLostError` always tells the leader that its work did not run alone. Its joiners get `FlightInterruptedError`. [ADR 0007](./docs/adr/0007-a-work-error-after-a-lost-lease-rejects-with-leaselosterror.md) tells why.
-- **A joiner's leader and the coordinator stop together.** The joiner gets `FlightInterruptedError` at the end of the grace window. It never runs the work again.
+  - A new call of a key that no process holds leads or joins at once.
+  - A new call or a rejoin of a key whose key lock another process holds, while the new coordinator knows no flight of the key, waits. When the leader reasserts its flight, the call or the rejoin joins it. When the key lock frees first, the call leads, and the rejoin gets `FlightInterruptedError`.
+- **A leader is frozen or busy, for example stopped by a debugger or in a long synchronous step.** It keeps its flight and its key lock. When the coordinator stops in that time, the calls of the key wait. When the leader continues, it reasserts its flight, and the calls that waited join it. When its process dies instead, the kernel frees the key lock, the joiners get `FlightInterruptedError`, and the next call leads.
+- **A leader loses its lease.** This occurs only when a new coordinator refuses its reassert, because it knows a newer flight of the key, or when this process cannot connect to a coordinator again. The signal of the lease aborts with `LeaseLostError`, and the call of the leader rejects with that error. When the work throws its own error after the loss, the call rejects with a new `LeaseLostError`, and the error of the work is its `cause`. Thus `LeaseLostError` tells the leader that its outcome reached no joiner. Its joiners get `FlightInterruptedError`. The leader keeps the key lock until its work ends, so the next flight of the key runs after it, not beside it. [ADR 0007](./docs/adr/0007-a-work-error-after-a-lost-lease-rejects-with-leaselosterror.md) tells why.
+- **A joiner's leader and the coordinator stop together.** The joiner gets `FlightInterruptedError` from the new coordinator. It never runs the work again.
 
 A call that was on its way to a coordinator that stopped joined nothing. It goes again to the next coordinator as a new call. [ADR 0005](./docs/adr/0005-a-joiner-rejoins-its-flight-by-its-token.md) tells why.
 
+### How long a call waits
+
+- A call of a key waits with no limit while a process that lives holds the key lock, for example a frozen leader, or a leader whose work does not end. To set a limit, give a `signal`. A call of another key does not wait for it.
+- Two flights of one key that follow each other wait about 10 ms between them. The leader frees the key lock only when it reads the acknowledgment of its landing, so the next call of the key waits for one more check of the key lock.
+
 ## Changes from 0.3.x
 
 Version 0.3.13 to 0.3.15 had a pull design. These parts changed:
@@ -197,9 +205,9 @@ Version 0.3.13 to 0.3.15 had a pull design. These parts changed:
 - [ADR 0006: The election and the connection are a copy of the mutex code](./docs/adr/0006-the-election-and-the-connection-are-a-copy-of-the-mutex-code.md)
 - [ADR 0007: A work error after a lost lease rejects with LeaseLostError](./docs/adr/0007-a-work-error-after-a-lost-lease-rejects-with-leaselosterror.md)
 - [ADR 0008: A connection closes only when its framing breaks](./docs/adr/0008-a-connection-closes-only-when-its-framing-breaks.md)
-- [ADR 0009: The next coordinator skips the grace window after a clean shutdown](./docs/adr/0009-the-next-coordinator-skips-the-grace-window-after-a-clean-shutdown.md)
+- [ADR 0010: A leader holds a kernel lock on its key until its outcome is handed over](./docs/adr/0010-a-leader-holds-a-kernel-lock-on-its-key-until-its-outcome-is-handed-over.md)
 - [Code copied from the mutex](./docs/copied-from-mutex.md): each copied file, what changed, and why; the rules that the single flight shares with the mutex; and the code that is in two places inside the single flight.
-- Superseded: [ADR 0001](./docs/adr/0001-a-joiner-follows-its-flight-record-without-acquiring-the-key.md) and [ADR 0002](./docs/adr/0002-a-flight-that-all-callers-left-is-abandoned.md).
+- Superseded: [ADR 0001](./docs/adr/0001-a-joiner-follows-its-flight-record-without-acquiring-the-key.md), [ADR 0002](./docs/adr/0002-a-flight-that-all-callers-left-is-abandoned.md) and [ADR 0009](./docs/adr/0009-the-next-coordinator-skips-the-grace-window-after-a-clean-shutdown.md).
 
 ## Development
 
@@ -220,7 +228,8 @@ src/
   codec.ts              the Codec interface
   errors.ts             FlightFailedError, FlightInterruptedError
   client/               the requests of one process: runs, leads, landings
-  coordinator/          the coordinator of one term: flights, sessions, its server
+  coordinator/          the coordinator of one term: flights, sessions, parked runs, its server
+  key-lock/             the key locks: the kernel lock on each key that a leader holds
   protocol/             the messages between a process and its coordinator
   connection/           the socket, the handshake and the supervisor (a copy); the election of the directory (@zukhruf/election)
   shared/               the record check (a copy); file writes and the network directory check are in @zukhruf/fs

package/dist/client/flight-client.d.ts

@@ -1,5 +1,6 @@
 import { FencingToken } from '@zukhruf/fencing';
 import type { ConnectionSupervisor } from '../connection/connection-supervisor.ts';
+import type { KeyLocks } from '../key-lock/key-locks.ts';
 import { type FlightRequest, type Outcome } from '../protocol/flight-protocol.ts';
 /** This process leads a flight: it runs the work and lands the outcome. */
 export interface Lead {
@@ -8,6 +9,8 @@ export interface Lead {
     readonly signal: AbortSignal;
     /** Ends the flight with `outcome` for every joiner. Kept and sent again until a coordinator acknowledges it. */
     land(outcome: Outcome): void;
+    /** The work settled. The key's lock waits for this, so call it on every path, also when the work never landed. */
+    ended(): void;
 }
 /** How the coordinator answered a run for good. */
 export type Answer = {
@@ -19,6 +22,11 @@ export type Answer = {
 } | {
     kind: 'interrupted';
 };
+export interface FlightClientOptions {
+    locks: KeyLocks;
+    /** Whether no coordinator of a newer term than the one that minted `token` has been elected. */
+    isCurrentTerm(token: FencingToken): Promise<boolean>;
+}
 export interface RunRequest {
     signal?: AbortSignal | undefined;
     onJoin?: (() => void) | undefined;
@@ -32,7 +40,7 @@ export interface RunRequest {
  */
 export declare class FlightClient {
     #private;
-    constructor(link: ConnectionSupervisor<FlightRequest>);
+    constructor(link: ConnectionSupervisor<FlightRequest>, { locks, isCurrentTerm }: FlightClientOptions);
     run(key: string, { signal, onJoin }: RunRequest): Promise<Answer>;
     /**
      * Disconnects for good, once every landing reached a coordinator: a landing

package/dist/client/flight-client.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"flight-client.d.ts","sourceRoot":"","sources":["../../src/client/flight-client.ts"],"names":[],"mappings":"AAGA,OAAO,EAAE,YAAY,EAAE,MAAM,kBAAkB,CAAC;AAGhD,OAAO,KAAK,EAAE,oBAAoB,EAAE,MAAM,wCAAwC,CAAC;AACnF,OAAO,EACL,KAAK,aAAa,EAElB,KAAK,OAAO,EAEb,MAAM,gCAAgC,CAAC;AAExC,2EAA2E;AAC3E,MAAM,WAAW,IAAI;IACnB,QAAQ,CAAC,KAAK,EAAE,YAAY,CAAC;IAC7B,iFAAiF;IACjF,QAAQ,CAAC,MAAM,EAAE,WAAW,CAAC;IAC7B,gHAAgH;IAChH,IAAI,CAAC,OAAO,EAAE,OAAO,GAAG,IAAI,CAAC;CAC9B;AAED,mDAAmD;AACnD,MAAM,MAAM,MAAM,GACd;IAAE,IAAI,EAAE,MAAM,CAAC;IAAC,IAAI,EAAE,IAAI,CAAA;CAAE,GAC5B;IAAE,IAAI,EAAE,QAAQ,CAAC;IAAC,OAAO,EAAE,OAAO,CAAA;CAAE,GACpC;IAAE,IAAI,EAAE,aAAa,CAAA;CAAE,CAAC;AAE5B,MAAM,WAAW,UAAU;IACzB,MAAM,CAAC,EAAE,WAAW,GAAG,SAAS,CAAC;IACjC,MAAM,CAAC,EAAE,CAAC,MAAM,IAAI,CAAC,GAAG,SAAS,CAAC;CACnC;AAuBD;;;;;;GAMG;AACH,qBAAa,YAAY;;gBAOX,IAAI,EAAE,oBAAoB,CAAC,aAAa,CAAC;IAW/C,GAAG,CAAC,GAAG,EAAE,MAAM,EAAE,EAAE,MAAM,EAAE,MAAM,EAAE,EAAE,UAAU,GAAG,OAAO,CAAC,MAAM,CAAC;IA2BvE;;;;OAIG;IACG,KAAK;CAqKZ"}
+{"version":3,"file":"flight-client.d.ts","sourceRoot":"","sources":["../../src/client/flight-client.ts"],"names":[],"mappings":"AAGA,OAAO,EAAE,YAAY,EAAE,MAAM,kBAAkB,CAAC;AAGhD,OAAO,KAAK,EAAE,oBAAoB,EAAE,MAAM,wCAAwC,CAAC;AAEnF,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,0BAA0B,CAAC;AACzD,OAAO,EACL,KAAK,aAAa,EAElB,KAAK,OAAO,EAEb,MAAM,gCAAgC,CAAC;AAExC,2EAA2E;AAC3E,MAAM,WAAW,IAAI;IACnB,QAAQ,CAAC,KAAK,EAAE,YAAY,CAAC;IAC7B,iFAAiF;IACjF,QAAQ,CAAC,MAAM,EAAE,WAAW,CAAC;IAC7B,gHAAgH;IAChH,IAAI,CAAC,OAAO,EAAE,OAAO,GAAG,IAAI,CAAC;IAC7B,kHAAkH;IAClH,KAAK,IAAI,IAAI,CAAC;CACf;AAED,mDAAmD;AACnD,MAAM,MAAM,MAAM,GACd;IAAE,IAAI,EAAE,MAAM,CAAC;IAAC,IAAI,EAAE,IAAI,CAAA;CAAE,GAC5B;IAAE,IAAI,EAAE,QAAQ,CAAC;IAAC,OAAO,EAAE,OAAO,CAAA;CAAE,GACpC;IAAE,IAAI,EAAE,aAAa,CAAA;CAAE,CAAC;AAE5B,MAAM,WAAW,mBAAmB;IAClC,KAAK,EAAE,QAAQ,CAAC;IAChB,gGAAgG;IAChG,aAAa,CAAC,KAAK,EAAE,YAAY,GAAG,OAAO,CAAC,OAAO,CAAC,CAAC;CACtD;AAED,MAAM,WAAW,UAAU;IACzB,MAAM,CAAC,EAAE,WAAW,GAAG,SAAS,CAAC;IACjC,MAAM,CAAC,EAAE,CAAC,MAAM,IAAI,CAAC,GAAG,SAAS,CAAC;CACnC;AAiCD;;;;;;GAMG;AACH,qBAAa,YAAY;;gBAUrB,IAAI,EAAE,oBAAoB,CAAC,aAAa,CAAC,EACzC,EAAE,KAAK,EAAE,aAAa,EAAE,EAAE,mBAAmB;IAczC,GAAG,CAAC,GAAG,EAAE,MAAM,EAAE,EAAE,MAAM,EAAE,MAAM,EAAE,EAAE,UAAU,GAAG,OAAO,CAAC,MAAM,CAAC;IA8BvE;;;;OAIG;IACG,KAAK;CA2OZ"}

package/dist/client/flight-client.js

@@ -2,6 +2,7 @@ import { randomUUID } from 'node:crypto';
 import { untilAborted } from '@zukhruf/async';
 import { FencingToken } from '@zukhruf/fencing';
 import { LeaseController } from '@zukhruf/lease';
+import { KeyHold } from "../key-lock/key-hold.js";
 import { isFlightResponse, } from "../protocol/flight-protocol.js";
 /**
  * Sends runs to the coordinator over a supervised connection. When the
@@ -12,12 +13,16 @@ import { isFlightResponse, } from "../protocol/flight-protocol.js";
  */
 export class FlightClient {
     #link;
+    #locks;
+    #isCurrentTerm;
     #pending = new Map();
     #held = new Map();
     /** Wakes `close` each time a flight this process leads is acknowledged or lost. */
     #released = new Set();
-    constructor(link) {
+    constructor(link, { locks, isCurrentTerm }) {
         this.#link = link;
+        this.#locks = locks;
+        this.#isCurrentTerm = isCurrentTerm;
         link.on('connected', () => this.#resume());
         // A message this client cannot read answers nothing it waits for.
         link.on('message', (message) => {
@@ -37,7 +42,7 @@ export class FlightClient {
         const pending = {
             key,
             onJoin,
-            delivery: 'queued',
+            delivery: { kind: 'queued' },
             flight: undefined,
             resolve,
             reject,
@@ -50,8 +55,10 @@ export class FlightClient {
             return await untilAborted(promise, signal);
         }
         catch (error) {
-            if (signal?.aborted)
+            if (signal?.aborted) {
                 this.#withdraw(id);
+                this.#abandon(id);
+            }
             throw error;
         }
     }
@@ -68,6 +75,9 @@ export class FlightClient {
         for (const id of [...this.#pending.keys()]) {
             this.#take(id)?.reject(new Error('This single flight is closed.'));
         }
+        // The coordinator interrupts the flights of a closed connection, so they need nothing more from this process.
+        for (const id of [...this.#held.keys()])
+            this.#release(id);
         await closing;
     }
     #dispatch(id, pending) {
@@ -76,22 +86,72 @@ export class FlightClient {
             ? { op: 'run', id, key }
             : { op: 'run', id, key, flight };
         if (this.#link.send(request))
-            pending.delivery = 'sent';
+            pending.delivery = { kind: 'sent' };
     }
     /**
      * The caller gave up. A run the coordinator may have heard is withdrawn;
      * one that already leads is not pending any more, so its flight runs on.
+     * A run that waits for its key's lock leads at the coordinator, so the
+     * cancel interrupts that flight: its work never started.
      */
     #withdraw(id) {
         const pending = this.#take(id);
-        if (pending?.delivery === 'sent')
+        if (pending && pending.delivery.kind !== 'queued') {
             this.#link.send({ op: 'cancel', id });
+        }
     }
-    #hold(id, key, token) {
+    /**
+     * The caller gave up although its run had led: the abort won the race with
+     * the answer, so nobody runs the work. The flight is interrupted, and the
+     * key's lock goes, instead of leading for the life of the process.
+     */
+    #abandon(id) {
+        const held = this.#held.get(id);
+        if (!held)
+            return;
+        this.#link.send({ op: 'cancel', id });
+        this.#release(id);
+        held.hold.workEnded();
+    }
+    /**
+     * Told to lead: the run leads only once it holds its key's lock, so no
+     * other process runs the key's work beside it. A grant of an older term is
+     * stale: the coordinator that minted it is gone, and its successor may have
+     * let another run lead. That run goes again as a plain run.
+     */
+    async #lock(id, pending, token, stop) {
+        try {
+            using locked = new DisposableStack();
+            locked.use(await this.#locks.acquire(pending.key, stop));
+            const current = await this.#isCurrentTerm(token);
+            if (stop.aborted)
+                return;
+            if (!current) {
+                locked.dispose();
+                pending.delivery = { kind: 'queued' };
+                this.#dispatch(id, pending);
+                return;
+            }
+            this.#take(id);
+            pending.resolve({
+                kind: 'lead',
+                lead: this.#hold(id, pending.key, token, new KeyHold(locked.move())),
+            });
+        }
+        catch (error) {
+            // An abort is the run's own end: it was answered, withdrawn or sent again.
+            if (stop.aborted)
+                return;
+            this.#take(id)?.reject(error);
+            this.#link.send({ op: 'cancel', id });
+        }
+    }
+    #hold(id, key, token, hold) {
         const held = {
             key,
             token,
             lease: new LeaseController(key),
+            hold,
             landing: undefined,
         };
         this.#held.set(id, held);
@@ -106,6 +166,7 @@ export class FlightClient {
                 this.#updateRef();
                 this.#link.send({ op: 'land', id, outcome });
             },
+            ended: () => hold.workEnded(),
         };
     }
     /** The flight of `id` is no longer this leader's, so its lease is told and nothing reasserts it. */
@@ -118,6 +179,7 @@ export class FlightClient {
     }
     /** The flight of `id` needs nothing more from this process. */
     #release(id) {
+        this.#held.get(id)?.hold.flightEnded();
         this.#held.delete(id);
         this.#updateRef();
         const waiting = [...this.#released];
@@ -129,14 +191,13 @@ export class FlightClient {
         switch (response.op) {
             case 'lead': {
                 // A run withdrawn meanwhile sent `cancel`, which interrupts the flight at the coordinator.
-                const pending = this.#take(response.id);
+                const pending = this.#pending.get(response.id);
                 if (!pending)
                     return;
-                const token = FencingToken.parse(response.token);
-                pending.resolve({
-                    kind: 'lead',
-                    lead: this.#hold(response.id, pending.key, token),
-                });
+                const stop = new AbortController();
+                pending.delivery = { kind: 'locking', stop };
+                // The first try of the lock is synchronous, here: only microtasks separate it from the work.
+                void this.#lock(response.id, pending, FencingToken.parse(response.token), stop.signal);
                 return;
             }
             case 'joined': {
@@ -195,14 +256,21 @@ export class FlightClient {
                 this.#link.send({ op: 'land', id, outcome: landing });
         }
         for (const [id, pending] of this.#pending) {
-            if (pending.delivery === 'queued')
+            if (pending.delivery.kind === 'queued')
                 this.#dispatch(id, pending);
         }
     }
-    /** Nothing sent on the lost connection will be answered, so every run goes again on the next one. */
+    /**
+     * Nothing sent on the lost connection will be answered, so every run goes
+     * again on the next one. A run that waits for its key's lock had not led
+     * yet: the next coordinator may not know its flight, so it goes again too.
+     */
     #interrupt() {
-        for (const pending of this.#pending.values())
-            pending.delivery = 'queued';
+        for (const pending of this.#pending.values()) {
+            if (pending.delivery.kind === 'locking')
+                pending.delivery.stop.abort();
+            pending.delivery = { kind: 'queued' };
+        }
     }
     /** No coordinator heard the reassertions, so the flights this process leads may be interrupted already. */
     #fail(error) {
@@ -216,6 +284,8 @@ export class FlightClient {
         const pending = this.#pending.get(id);
         if (!pending)
             return undefined;
+        if (pending.delivery.kind === 'locking')
+            pending.delivery.stop.abort();
         this.#pending.delete(id);
         this.#updateRef();
         return pending;

package/dist/client/flight-client.js.map

@@ -1 +1 @@
-{"version":3,"file":"flight-client.js","sourceRoot":"","sources":["../../src/client/flight-client.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAEzC,OAAO,EAAE,YAAY,EAAE,MAAM,gBAAgB,CAAC;AAC9C,OAAO,EAAE,YAAY,EAAE,MAAM,kBAAkB,CAAC;AAChD,OAAO,EAAE,eAAe,EAAE,MAAM,gBAAgB,CAAC;AAGjD,OAAO,EAIL,gBAAgB,GACjB,MAAM,gCAAgC,CAAC;AA2CxC;;;;;;GAMG;AACH,MAAM,OAAO,YAAY;IACd,KAAK,CAAsC;IAC3C,QAAQ,GAAG,IAAI,GAAG,EAAmB,CAAC;IACtC,KAAK,GAAG,IAAI,GAAG,EAAgB,CAAC;IACzC,mFAAmF;IAC1E,SAAS,GAAG,IAAI,GAAG,EAAc,CAAC;IAE3C,YAAY,IAAyC;QACnD,IAAI,CAAC,KAAK,GAAG,IAAI,CAAC;QAClB,IAAI,CAAC,EAAE,CAAC,WAAW,EAAE,GAAG,EAAE,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC,CAAC;QAC3C,kEAAkE;QAClE,IAAI,CAAC,EAAE,CAAC,SAAS,EAAE,CAAC,OAAO,EAAE,EAAE;YAC7B,IAAI,gBAAgB,CAAC,OAAO,CAAC;gBAAE,IAAI,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC;QACxD,CAAC,CAAC,CAAC;QACH,IAAI,CAAC,EAAE,CAAC,cAAc,EAAE,GAAG,EAAE,CAAC,IAAI,CAAC,UAAU,EAAE,CAAC,CAAC;QACjD,IAAI,CAAC,EAAE,CAAC,QAAQ,EAAE,CAAC,KAAK,EAAE,EAAE,CAAC,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC;IAClD,CAAC;IAED,KAAK,CAAC,GAAG,CAAC,GAAW,EAAE,EAAE,MAAM,EAAE,MAAM,EAAc;QACnD,MAAM,EAAE,cAAc,EAAE,CAAC;QACzB,IAAI,IAAI,CAAC,KAAK,CAAC,MAAM,KAAK,QAAQ,EAAE,CAAC;YACnC,MAAM,IAAI,KAAK,CAAC,+BAA+B,CAAC,CAAC;QACnD,CAAC;QACD,MAAM,EAAE,GAAG,UAAU,EAAE,CAAC;QACxB,MAAM,EAAE,OAAO,EAAE,OAAO,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC,aAAa,EAAU,CAAC;QACrE,MAAM,OAAO,GAAY;YACvB,GAAG;YACH,MAAM;YACN,QAAQ,EAAE,QAAQ;YAClB,MAAM,EAAE,SAAS;YACjB,OAAO;YACP,MAAM;SACP,CAAC;QACF,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,EAAE,OAAO,CAAC,CAAC;QAC/B,IAAI,CAAC,UAAU,EAAE,CAAC;QAClB,IAAI,CAAC,KAAK,CAAC,IAAI,EAAE,CAAC;QAClB,IAAI,CAAC,SAAS,CAAC,EAAE,EAAE,OAAO,CAAC,CAAC;QAC5B,IAAI,CAAC;YACH,OAAO,MAAM,YAAY,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC;QAC7C,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,MAAM,EAAE,OAAO;gBAAE,IAAI,CAAC,SAAS,CAAC,EAAE,CAAC,CAAC;YACxC,MAAM,KAAK,CAAC;QACd,CAAC;IACH,CAAC;IAED;;;;OAIG;IACH,KAAK,CAAC,KAAK;QACT,OAAO,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,EAAE,OAAO,EAAE,EAAE,EAAE,CAAC,OAAO,CAAC,EAAE,CAAC;YAC/D,MAAM,IAAI,OAAO,CAAO,CAAC,OAAO,EAAE,EAAE,CAAC,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC;QACpE,CAAC;QACD,MAAM,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;QACnC,KAAK,MAAM,EAAE,IAAI,CAAC,GAAG,IAAI,CAAC,QAAQ,CAAC,IAAI,EAAE,CAAC,EAAE,CAAC;YAC3C,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,EAAE,MAAM,CAAC,IAAI,KAAK,CAAC,+BAA+B,CAAC,CAAC,CAAC;QACrE,CAAC;QACD,MAAM,OAAO,CAAC;IAChB,CAAC;IAED,SAAS,CAAC,EAAU,EAAE,OAAgB;QACpC,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC;QAChC,MAAM,OAAO,GACX,MAAM,KAAK,SAAS;YAClB,CAAC,CAAC,EAAE,EAAE,EAAE,KAAK,EAAE,EAAE,EAAE,GAAG,EAAE;YACxB,CAAC,CAAC,EAAE,EAAE,EAAE,KAAK,EAAE,EAAE,EAAE,GAAG,EAAE,MAAM,EAAE,CAAC;QACrC,IAAI,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,CAAC;YAAE,OAAO,CAAC,QAAQ,GAAG,MAAM,CAAC;IAC1D,CAAC;IAED;;;OAGG;IACH,SAAS,CAAC,EAAU;QAClB,MAAM,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;QAC/B,IAAI,OAAO,EAAE,QAAQ,KAAK,MAAM;YAAE,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,EAAE,EAAE,QAAQ,EAAE,EAAE,EAAE,CAAC,CAAC;IAC1E,CAAC;IAED,KAAK,CAAC,EAAU,EAAE,GAAW,EAAE,KAAmB;QAChD,MAAM,IAAI,GAAS;YACjB,GAAG;YACH,KAAK;YACL,KAAK,EAAE,IAAI,eAAe,CAAC,GAAG,CAAC;YAC/B,OAAO,EAAE,SAAS;SACnB,CAAC;QACF,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE,EAAE,IAAI,CAAC,CAAC;QACzB,OAAO;YACL,KAAK;YACL,MAAM,EAAE,IAAI,CAAC,KAAK,CAAC,MAAM;YACzB,IAAI,EAAE,CAAC,OAAO,EAAE,EAAE;gBAChB,oDAAoD;gBACpD,IAAI,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE,CAAC,KAAK,IAAI,IAAI,IAAI,CAAC,OAAO;oBAAE,OAAO;gBACxD,IAAI,CAAC,OAAO,GAAG,OAAO,CAAC;gBACvB,IAAI,CAAC,UAAU,EAAE,CAAC;gBAClB,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,EAAE,EAAE,MAAM,EAAE,EAAE,EAAE,OAAO,EAAE,CAAC,CAAC;YAC/C,CAAC;SACF,CAAC;IACJ,CAAC;IAED,oGAAoG;IACpG,KAAK,CAAC,EAAU;QACd,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC;QAChC,IAAI,CAAC,IAAI;YAAE,OAAO;QAClB,IAAI,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC;QAClB,IAAI,CAAC,KAAK,CAAC,IAAI,EAAE,CAAC;IACpB,CAAC;IAED,+DAA+D;IAC/D,QAAQ,CAAC,EAAU;QACjB,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC;QACtB,IAAI,CAAC,UAAU,EAAE,CAAC;QAClB,MAAM,OAAO,GAAG,CAAC,GAAG,IAAI,CAAC,SAAS,CAAC,CAAC;QACpC,IAAI,CAAC,SAAS,CAAC,KAAK,EAAE,CAAC;QACvB,KAAK,MAAM,IAAI,IAAI,OAAO;YAAE,IAAI,EAAE,CAAC;IACrC,CAAC;IAED,QAAQ,CAAC,QAAwB;QAC/B,QAAQ,QAAQ,CAAC,EAAE,EAAE,CAAC;YACpB,KAAK,MAAM,CAAC,CAAC,CAAC;gBACZ,2FAA2F;gBAC3F,MAAM,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC;gBACxC,IAAI,CAAC,OAAO;oBAAE,OAAO;gBACrB,MAAM,KAAK,GAAG,YAAY,CAAC,KAAK,CAAC,QAAQ,CAAC,KAAK,CAAE,CAAC;gBAClD,OAAO,CAAC,OAAO,CAAC;oBACd,IAAI,EAAE,MAAM;oBACZ,IAAI,EAAE,IAAI,CAAC,KAAK,CAAC,QAAQ,CAAC,EAAE,EAAE,OAAO,CAAC,GAAG,EAAE,KAAK,CAAC;iBAClD,CAAC,CAAC;gBACH,OAAO;YACT,CAAC;YACD,KAAK,QAAQ,CAAC,CAAC,CAAC;gBACd,MAAM,OAAO,GAAG,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC;gBAC/C,IAAI,CAAC,OAAO;oBAAE,OAAO;gBACrB,MAAM,KAAK,GAAG,OAAO,CAAC,MAAM,KAAK,SAAS,CAAC;gBAC3C,OAAO,CAAC,MAAM,GAAG,QAAQ,CAAC,MAAM,CAAC;gBACjC,IAAI,KAAK;oBAAE,IAAI,CAAC,WAAW,CAAC,QAAQ,CAAC,EAAE,EAAE,OAAO,CAAC,CAAC;gBAClD,OAAO;YACT,CAAC;YACD,KAAK,QAAQ;gBACX,IAAI,CAAC,KAAK,CAAC,QAAQ,CAAC,EAAE,CAAC,EAAE,OAAO,CAAC;oBAC/B,IAAI,EAAE,QAAQ;oBACd,OAAO,EAAE,QAAQ,CAAC,OAAO;iBAC1B,CAAC,CAAC;gBACH,OAAO;YACT,KAAK,aAAa;gBAChB,IAAI,CAAC,KAAK,CAAC,QAAQ,CAAC,EAAE,CAAC,EAAE,OAAO,CAAC,EAAE,IAAI,EAAE,aAAa,EAAE,CAAC,CAAC;gBAC1D,OAAO;YACT,KAAK,KAAK,CAAC,CAAC,CAAC;gBACX,sGAAsG;gBACtG,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC;gBACzC,IAAI,CAAC,IAAI,EAAE,OAAO;oBAAE,OAAO;gBAC3B,IAAI,CAAC,QAAQ,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC;gBAC3B,IAAI,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC;gBACjB,OAAO;YACT,CAAC;YACD,KAAK,UAAU;gBACb,IAAI,CAAC,KAAK,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC;gBACxB,OAAO;YACT,KAAK,aAAa;gBAChB,IAAI,CAAC,KAAK,CAAC,QAAQ,CAAC,EAAE,CAAC,EAAE,MAAM,CAC7B,IAAI,KAAK,CAAC,sDAAsD,CAAC,CAClE,CAAC;gBACF,OAAO;QACX,CAAC;IACH,CAAC;IAED,oFAAoF;IACpF,WAAW,CAAC,EAAU,EAAE,OAAgB;QACtC,IAAI,CAAC;YACH,OAAO,CAAC,MAAM,EAAE,EAAE,CAAC;QACrB,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,SAAS,CAAC,EAAE,CAAC,CAAC;YACnB,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;QACxB,CAAC;IACH,CAAC;IAED,2GAA2G;IAC3G,OAAO;QACL,qGAAqG;QACrG,IAAI,CAAC,UAAU,EAAE,CAAC;QAClB,KAAK,MAAM,CAAC,EAAE,EAAE,EAAE,GAAG,EAAE,KAAK,EAAE,OAAO,EAAE,CAAC,IAAI,IAAI,CAAC,KAAK,EAAE,CAAC;YACvD,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,EAAE,EAAE,UAAU,EAAE,EAAE,EAAE,GAAG,EAAE,KAAK,EAAE,KAAK,CAAC,QAAQ,EAAE,EAAE,CAAC,CAAC;YACtE,IAAI,OAAO;gBAAE,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,EAAE,EAAE,MAAM,EAAE,EAAE,EAAE,OAAO,EAAE,OAAO,EAAE,CAAC,CAAC;QACrE,CAAC;QACD,KAAK,MAAM,CAAC,EAAE,EAAE,OAAO,CAAC,IAAI,IAAI,CAAC,QAAQ,EAAE,CAAC;YAC1C,IAAI,OAAO,CAAC,QAAQ,KAAK,QAAQ;gBAAE,IAAI,CAAC,SAAS,CAAC,EAAE,EAAE,OAAO,CAAC,CAAC;QACjE,CAAC;IACH,CAAC;IAED,qGAAqG;IACrG,UAAU;QACR,KAAK,MAAM,OAAO,IAAI,IAAI,CAAC,QAAQ,CAAC,MAAM,EAAE;YAAE,OAAO,CAAC,QAAQ,GAAG,QAAQ,CAAC;IAC5E,CAAC;IAED,2GAA2G;IAC3G,KAAK,CAAC,KAAc;QAClB,KAAK,MAAM,EAAE,IAAI,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,EAAE,CAAC;YAAE,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;QACxD,KAAK,MAAM,EAAE,IAAI,CAAC,GAAG,IAAI,CAAC,QAAQ,CAAC,IAAI,EAAE,CAAC;YAAE,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,EAAE,MAAM,CAAC,KAAK,CAAC,CAAC;IAC5E,CAAC;IAED,sFAAsF;IACtF,KAAK,CAAC,EAAU;QACd,MAAM,OAAO,GAAG,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC;QACtC,IAAI,CAAC,OAAO;YAAE,OAAO,SAAS,CAAC;QAC/B,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC;QACzB,IAAI,CAAC,UAAU,EAAE,CAAC;QAClB,OAAO,OAAO,CAAC;IACjB,CAAC;IAED,gGAAgG;IAChG,UAAU;QACR,MAAM,OAAO,GAAG,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,EAAE,OAAO,EAAE,EAAE,EAAE,CAAC,OAAO,CAAC,CAAC;QACxE,IAAI,IAAI,CAAC,QAAQ,CAAC,IAAI,GAAG,CAAC,IAAI,OAAO;YAAE,IAAI,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC;;YACnD,IAAI,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;IAC1B,CAAC;CACF"}
+{"version":3,"file":"flight-client.js","sourceRoot":"","sources":["../../src/client/flight-client.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAEzC,OAAO,EAAE,YAAY,EAAE,MAAM,gBAAgB,CAAC;AAC9C,OAAO,EAAE,YAAY,EAAE,MAAM,kBAAkB,CAAC;AAChD,OAAO,EAAE,eAAe,EAAE,MAAM,gBAAgB,CAAC;AAGjD,OAAO,EAAE,OAAO,EAAE,MAAM,yBAAyB,CAAC;AAElD,OAAO,EAIL,gBAAgB,GACjB,MAAM,gCAAgC,CAAC;AA6DxC;;;;;;GAMG;AACH,MAAM,OAAO,YAAY;IACd,KAAK,CAAsC;IAC3C,MAAM,CAAW;IACjB,cAAc,CAA4C;IAC1D,QAAQ,GAAG,IAAI,GAAG,EAAmB,CAAC;IACtC,KAAK,GAAG,IAAI,GAAG,EAAgB,CAAC;IACzC,mFAAmF;IAC1E,SAAS,GAAG,IAAI,GAAG,EAAc,CAAC;IAE3C,YACE,IAAyC,EACzC,EAAE,KAAK,EAAE,aAAa,EAAuB;QAE7C,IAAI,CAAC,KAAK,GAAG,IAAI,CAAC;QAClB,IAAI,CAAC,MAAM,GAAG,KAAK,CAAC;QACpB,IAAI,CAAC,cAAc,GAAG,aAAa,CAAC;QACpC,IAAI,CAAC,EAAE,CAAC,WAAW,EAAE,GAAG,EAAE,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC,CAAC;QAC3C,kEAAkE;QAClE,IAAI,CAAC,EAAE,CAAC,SAAS,EAAE,CAAC,OAAO,EAAE,EAAE;YAC7B,IAAI,gBAAgB,CAAC,OAAO,CAAC;gBAAE,IAAI,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC;QACxD,CAAC,CAAC,CAAC;QACH,IAAI,CAAC,EAAE,CAAC,cAAc,EAAE,GAAG,EAAE,CAAC,IAAI,CAAC,UAAU,EAAE,CAAC,CAAC;QACjD,IAAI,CAAC,EAAE,CAAC,QAAQ,EAAE,CAAC,KAAK,EAAE,EAAE,CAAC,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC;IAClD,CAAC;IAED,KAAK,CAAC,GAAG,CAAC,GAAW,EAAE,EAAE,MAAM,EAAE,MAAM,EAAc;QACnD,MAAM,EAAE,cAAc,EAAE,CAAC;QACzB,IAAI,IAAI,CAAC,KAAK,CAAC,MAAM,KAAK,QAAQ,EAAE,CAAC;YACnC,MAAM,IAAI,KAAK,CAAC,+BAA+B,CAAC,CAAC;QACnD,CAAC;QACD,MAAM,EAAE,GAAG,UAAU,EAAE,CAAC;QACxB,MAAM,EAAE,OAAO,EAAE,OAAO,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC,aAAa,EAAU,CAAC;QACrE,MAAM,OAAO,GAAY;YACvB,GAAG;YACH,MAAM;YACN,QAAQ,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE;YAC5B,MAAM,EAAE,SAAS;YACjB,OAAO;YACP,MAAM;SACP,CAAC;QACF,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,EAAE,OAAO,CAAC,CAAC;QAC/B,IAAI,CAAC,UAAU,EAAE,CAAC;QAClB,IAAI,CAAC,KAAK,CAAC,IAAI,EAAE,CAAC;QAClB,IAAI,CAAC,SAAS,CAAC,EAAE,EAAE,OAAO,CAAC,CAAC;QAC5B,IAAI,CAAC;YACH,OAAO,MAAM,YAAY,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC;QAC7C,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,MAAM,EAAE,OAAO,EAAE,CAAC;gBACpB,IAAI,CAAC,SAAS,CAAC,EAAE,CAAC,CAAC;gBACnB,IAAI,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC;YACpB,CAAC;YACD,MAAM,KAAK,CAAC;QACd,CAAC;IACH,CAAC;IAED;;;;OAIG;IACH,KAAK,CAAC,KAAK;QACT,OAAO,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,EAAE,OAAO,EAAE,EAAE,EAAE,CAAC,OAAO,CAAC,EAAE,CAAC;YAC/D,MAAM,IAAI,OAAO,CAAO,CAAC,OAAO,EAAE,EAAE,CAAC,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC;QACpE,CAAC;QACD,MAAM,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;QACnC,KAAK,MAAM,EAAE,IAAI,CAAC,GAAG,IAAI,CAAC,QAAQ,CAAC,IAAI,EAAE,CAAC,EAAE,CAAC;YAC3C,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,EAAE,MAAM,CAAC,IAAI,KAAK,CAAC,+BAA+B,CAAC,CAAC,CAAC;QACrE,CAAC;QACD,8GAA8G;QAC9G,KAAK,MAAM,EAAE,IAAI,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,EAAE,CAAC;YAAE,IAAI,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC;QAC3D,MAAM,OAAO,CAAC;IAChB,CAAC;IAED,SAAS,CAAC,EAAU,EAAE,OAAgB;QACpC,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC;QAChC,MAAM,OAAO,GACX,MAAM,KAAK,SAAS;YAClB,CAAC,CAAC,EAAE,EAAE,EAAE,KAAK,EAAE,EAAE,EAAE,GAAG,EAAE;YACxB,CAAC,CAAC,EAAE,EAAE,EAAE,KAAK,EAAE,EAAE,EAAE,GAAG,EAAE,MAAM,EAAE,CAAC;QACrC,IAAI,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,CAAC;YAAE,OAAO,CAAC,QAAQ,GAAG,EAAE,IAAI,EAAE,MAAM,EAAE,CAAC;IACpE,CAAC;IAED;;;;;OAKG;IACH,SAAS,CAAC,EAAU;QAClB,MAAM,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;QAC/B,IAAI,OAAO,IAAI,OAAO,CAAC,QAAQ,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;YAClD,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,EAAE,EAAE,QAAQ,EAAE,EAAE,EAAE,CAAC,CAAC;QACxC,CAAC;IACH,CAAC;IAED;;;;OAIG;IACH,QAAQ,CAAC,EAAU;QACjB,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC;QAChC,IAAI,CAAC,IAAI;YAAE,OAAO;QAClB,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,EAAE,EAAE,QAAQ,EAAE,EAAE,EAAE,CAAC,CAAC;QACtC,IAAI,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC;QAClB,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,CAAC;IACxB,CAAC;IAED;;;;;OAKG;IACH,KAAK,CAAC,KAAK,CACT,EAAU,EACV,OAAgB,EAChB,KAAmB,EACnB,IAAiB;QAEjB,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,IAAI,eAAe,EAAE,CAAC;YACrC,MAAM,CAAC,GAAG,CAAC,MAAM,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC,GAAG,EAAE,IAAI,CAAC,CAAC,CAAC;YACzD,MAAM,OAAO,GAAG,MAAM,IAAI,CAAC,cAAc,CAAC,KAAK,CAAC,CAAC;YACjD,IAAI,IAAI,CAAC,OAAO;gBAAE,OAAO;YACzB,IAAI,CAAC,OAAO,EAAE,CAAC;gBACb,MAAM,CAAC,OAAO,EAAE,CAAC;gBACjB,OAAO,CAAC,QAAQ,GAAG,EAAE,IAAI,EAAE,QAAQ,EAAE,CAAC;gBACtC,IAAI,CAAC,SAAS,CAAC,EAAE,EAAE,OAAO,CAAC,CAAC;gBAC5B,OAAO;YACT,CAAC;YACD,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;YACf,OAAO,CAAC,OAAO,CAAC;gBACd,IAAI,EAAE,MAAM;gBACZ,IAAI,EAAE,IAAI,CAAC,KAAK,CAAC,EAAE,EAAE,OAAO,CAAC,GAAG,EAAE,KAAK,EAAE,IAAI,OAAO,CAAC,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC;aACrE,CAAC,CAAC;QACL,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,2EAA2E;YAC3E,IAAI,IAAI,CAAC,OAAO;gBAAE,OAAO;YACzB,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,EAAE,MAAM,CAAC,KAAK,CAAC,CAAC;YAC9B,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,EAAE,EAAE,QAAQ,EAAE,EAAE,EAAE,CAAC,CAAC;QACxC,CAAC;IACH,CAAC;IAED,KAAK,CAAC,EAAU,EAAE,GAAW,EAAE,KAAmB,EAAE,IAAa;QAC/D,MAAM,IAAI,GAAS;YACjB,GAAG;YACH,KAAK;YACL,KAAK,EAAE,IAAI,eAAe,CAAC,GAAG,CAAC;YAC/B,IAAI;YACJ,OAAO,EAAE,SAAS;SACnB,CAAC;QACF,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE,EAAE,IAAI,CAAC,CAAC;QACzB,OAAO;YACL,KAAK;YACL,MAAM,EAAE,IAAI,CAAC,KAAK,CAAC,MAAM;YACzB,IAAI,EAAE,CAAC,OAAO,EAAE,EAAE;gBAChB,oDAAoD;gBACpD,IAAI,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE,CAAC,KAAK,IAAI,IAAI,IAAI,CAAC,OAAO;oBAAE,OAAO;gBACxD,IAAI,CAAC,OAAO,GAAG,OAAO,CAAC;gBACvB,IAAI,CAAC,UAAU,EAAE,CAAC;gBAClB,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,EAAE,EAAE,MAAM,EAAE,EAAE,EAAE,OAAO,EAAE,CAAC,CAAC;YAC/C,CAAC;YACD,KAAK,EAAE,GAAG,EAAE,CAAC,IAAI,CAAC,SAAS,EAAE;SAC9B,CAAC;IACJ,CAAC;IAED,oGAAoG;IACpG,KAAK,CAAC,EAAU;QACd,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC;QAChC,IAAI,CAAC,IAAI;YAAE,OAAO;QAClB,IAAI,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC;QAClB,IAAI,CAAC,KAAK,CAAC,IAAI,EAAE,CAAC;IACpB,CAAC;IAED,+DAA+D;IAC/D,QAAQ,CAAC,EAAU;QACjB,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,IAAI,CAAC,WAAW,EAAE,CAAC;QACvC,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC;QACtB,IAAI,CAAC,UAAU,EAAE,CAAC;QAClB,MAAM,OAAO,GAAG,CAAC,GAAG,IAAI,CAAC,SAAS,CAAC,CAAC;QACpC,IAAI,CAAC,SAAS,CAAC,KAAK,EAAE,CAAC;QACvB,KAAK,MAAM,IAAI,IAAI,OAAO;YAAE,IAAI,EAAE,CAAC;IACrC,CAAC;IAED,QAAQ,CAAC,QAAwB;QAC/B,QAAQ,QAAQ,CAAC,EAAE,EAAE,CAAC;YACpB,KAAK,MAAM,CAAC,CAAC,CAAC;gBACZ,2FAA2F;gBAC3F,MAAM,OAAO,GAAG,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC;gBAC/C,IAAI,CAAC,OAAO;oBAAE,OAAO;gBACrB,MAAM,IAAI,GAAG,IAAI,eAAe,EAAE,CAAC;gBACnC,OAAO,CAAC,QAAQ,GAAG,EAAE,IAAI,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC;gBAC7C,6FAA6F;gBAC7F,KAAK,IAAI,CAAC,KAAK,CACb,QAAQ,CAAC,EAAE,EACX,OAAO,EACP,YAAY,CAAC,KAAK,CAAC,QAAQ,CAAC,KAAK,CAAE,EACnC,IAAI,CAAC,MAAM,CACZ,CAAC;gBACF,OAAO;YACT,CAAC;YACD,KAAK,QAAQ,CAAC,CAAC,CAAC;gBACd,MAAM,OAAO,GAAG,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC;gBAC/C,IAAI,CAAC,OAAO;oBAAE,OAAO;gBACrB,MAAM,KAAK,GAAG,OAAO,CAAC,MAAM,KAAK,SAAS,CAAC;gBAC3C,OAAO,CAAC,MAAM,GAAG,QAAQ,CAAC,MAAM,CAAC;gBACjC,IAAI,KAAK;oBAAE,IAAI,CAAC,WAAW,CAAC,QAAQ,CAAC,EAAE,EAAE,OAAO,CAAC,CAAC;gBAClD,OAAO;YACT,CAAC;YACD,KAAK,QAAQ;gBACX,IAAI,CAAC,KAAK,CAAC,QAAQ,CAAC,EAAE,CAAC,EAAE,OAAO,CAAC;oBAC/B,IAAI,EAAE,QAAQ;oBACd,OAAO,EAAE,QAAQ,CAAC,OAAO;iBAC1B,CAAC,CAAC;gBACH,OAAO;YACT,KAAK,aAAa;gBAChB,IAAI,CAAC,KAAK,CAAC,QAAQ,CAAC,EAAE,CAAC,EAAE,OAAO,CAAC,EAAE,IAAI,EAAE,aAAa,EAAE,CAAC,CAAC;gBAC1D,OAAO;YACT,KAAK,KAAK,CAAC,CAAC,CAAC;gBACX,sGAAsG;gBACtG,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC;gBACzC,IAAI,CAAC,IAAI,EAAE,OAAO;oBAAE,OAAO;gBAC3B,IAAI,CAAC,QAAQ,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC;gBAC3B,IAAI,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC;gBACjB,OAAO;YACT,CAAC;YACD,KAAK,UAAU;gBACb,IAAI,CAAC,KAAK,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC;gBACxB,OAAO;YACT,KAAK,aAAa;gBAChB,IAAI,CAAC,KAAK,CAAC,QAAQ,CAAC,EAAE,CAAC,EAAE,MAAM,CAC7B,IAAI,KAAK,CAAC,sDAAsD,CAAC,CAClE,CAAC;gBACF,OAAO;QACX,CAAC;IACH,CAAC;IAED,oFAAoF;IACpF,WAAW,CAAC,EAAU,EAAE,OAAgB;QACtC,IAAI,CAAC;YACH,OAAO,CAAC,MAAM,EAAE,EAAE,CAAC;QACrB,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,SAAS,CAAC,EAAE,CAAC,CAAC;YACnB,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;QACxB,CAAC;IACH,CAAC;IAED,2GAA2G;IAC3G,OAAO;QACL,qGAAqG;QACrG,IAAI,CAAC,UAAU,EAAE,CAAC;QAClB,KAAK,MAAM,CAAC,EAAE,EAAE,EAAE,GAAG,EAAE,KAAK,EAAE,OAAO,EAAE,CAAC,IAAI,IAAI,CAAC,KAAK,EAAE,CAAC;YACvD,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,EAAE,EAAE,UAAU,EAAE,EAAE,EAAE,GAAG,EAAE,KAAK,EAAE,KAAK,CAAC,QAAQ,EAAE,EAAE,CAAC,CAAC;YACtE,IAAI,OAAO;gBAAE,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,EAAE,EAAE,MAAM,EAAE,EAAE,EAAE,OAAO,EAAE,OAAO,EAAE,CAAC,CAAC;QACrE,CAAC;QACD,KAAK,MAAM,CAAC,EAAE,EAAE,OAAO,CAAC,IAAI,IAAI,CAAC,QAAQ,EAAE,CAAC;YAC1C,IAAI,OAAO,CAAC,QAAQ,CAAC,IAAI,KAAK,QAAQ;gBAAE,IAAI,CAAC,SAAS,CAAC,EAAE,EAAE,OAAO,CAAC,CAAC;QACtE,CAAC;IACH,CAAC;IAED;;;;OAIG;IACH,UAAU;QACR,KAAK,MAAM,OAAO,IAAI,IAAI,CAAC,QAAQ,CAAC,MAAM,EAAE,EAAE,CAAC;YAC7C,IAAI,OAAO,CAAC,QAAQ,CAAC,IAAI,KAAK,SAAS;gBAAE,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAC,KAAK,EAAE,CAAC;YACvE,OAAO,CAAC,QAAQ,GAAG,EAAE,IAAI,EAAE,QAAQ,EAAE,CAAC;QACxC,CAAC;IACH,CAAC;IAED,2GAA2G;IAC3G,KAAK,CAAC,KAAc;QAClB,KAAK,MAAM,EAAE,IAAI,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,EAAE,CAAC;YAAE,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;QACxD,KAAK,MAAM,EAAE,IAAI,CAAC,GAAG,IAAI,CAAC,QAAQ,CAAC,IAAI,EAAE,CAAC;YAAE,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,EAAE,MAAM,CAAC,KAAK,CAAC,CAAC;IAC5E,CAAC;IAED,sFAAsF;IACtF,KAAK,CAAC,EAAU;QACd,MAAM,OAAO,GAAG,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC;QACtC,IAAI,CAAC,OAAO;YAAE,OAAO,SAAS,CAAC;QAC/B,IAAI,OAAO,CAAC,QAAQ,CAAC,IAAI,KAAK,SAAS;YAAE,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAC,KAAK,EAAE,CAAC;QACvE,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC;QACzB,IAAI,CAAC,UAAU,EAAE,CAAC;QAClB,OAAO,OAAO,CAAC;IACjB,CAAC;IAED,gGAAgG;IAChG,UAAU;QACR,MAAM,OAAO,GAAG,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,EAAE,OAAO,EAAE,EAAE,EAAE,CAAC,OAAO,CAAC,CAAC;QACxE,IAAI,IAAI,CAAC,QAAQ,CAAC,IAAI,GAAG,CAAC,IAAI,OAAO;YAAE,IAAI,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC;;YACnD,IAAI,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC;IAC1B,CAAC;CACF"}

package/dist/coordinator/flight-coordinator.d.ts

@@ -1,5 +1,6 @@
 import type { FencingToken, TokenSource } from '@zukhruf/fencing';
 import type { Connection } from '../connection/connection.ts';
+import type { KeyLocks } from '../key-lock/key-locks.ts';
 import type { FlightResponse, Outcome } from '../protocol/flight-protocol.ts';
 import { Flight } from './flight.ts';
 import type { Party } from './party.ts';
@@ -7,29 +8,40 @@ export interface FlightCoordinatorOptions {
     /** Mints the lease token of each flight this coordinator starts. */
     tokens: TokenSource;
     /**
-     * Milliseconds after start during which no flight starts, so leaders from a
-     * previous coordinator can reassert their flights first. Zero when no
-     * earlier coordinator can have left a flight in progress.
+     * The kernel locks on the keys. A process that holds a key's lock runs its
+     * work, whether this coordinator knows that flight or not.
      */
-    graceWindow: number;
+    locks: KeyLocks;
 }
 /**
  * Coordinates the flights of one term. A run leads a new flight of its key, or
  * joins the flight of that key in progress, in one step, so two runs never
  * both lead. The outcome is pushed to every joiner as soon as it lands.
+ *
+ * A key whose lock another process holds, while no flight of it is known
+ * here, runs the work of a leader that has not reasserted its flight yet, so
+ * its runs and rejoins wait for that flight or for the lock to free. A leader
+ * keeps its flight while its process lives.
  */
 export declare class FlightCoordinator {
     #private;
-    constructor({ tokens, graceWindow }: FlightCoordinatorOptions);
-    /**
-     * Whether the next coordinator would find nothing to recover: no flight is
-     * in progress, and the grace window is over or there was none.
-     */
-    get leavesNothingToRecover(): boolean;
+    constructor({ tokens, locks }: FlightCoordinatorOptions);
     serve(connection: Connection<FlightResponse>): void;
+    /** Stops watching the locks of the keys whose runs wait. */
+    close(): void;
+    /** Rejects only when no token can be minted. */
     run(party: Party, key: string): Promise<void>;
-    /** A joiner that lost its connection, back for the flight it joined: it never leads, so its work never runs twice. */
-    rejoin(party: Party, key: string, flight: string): Promise<void>;
+    /**
+     * A joiner that lost its connection, back for the flight it joined: it never
+     * leads, so its work never runs twice. A rejoin joins only its own flight;
+     * any other answer could be a second run of the work.
+     */
+    rejoin(party: Party, key: string, flight: string): void;
+    /**
+     * A leader claims its flight again after its connection was lost. The same
+     * flight, through a new connection, stays one flight with its joiners. For
+     * two flights of one key, the newer token wins, and the older flight is lost.
+     */
     reassert(party: Party, key: string, token: FencingToken): void;
     land(flight: Flight, outcome: Outcome): void;
     /** The leader withdrew or its connection is gone. */

package/dist/coordinator/flight-coordinator.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"flight-coordinator.d.ts","sourceRoot":"","sources":["../../src/coordinator/flight-coordinator.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,YAAY,EAAE,WAAW,EAAE,MAAM,kBAAkB,CAAC;AAElE,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,6BAA6B,CAAC;AAC9D,OAAO,KAAK,EAAE,cAAc,EAAE,OAAO,EAAE,MAAM,gCAAgC,CAAC;AAC9E,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AACrC,OAAO,KAAK,EAAE,KAAK,EAAE,MAAM,YAAY,CAAC;AAGxC,MAAM,WAAW,wBAAwB;IACvC,oEAAoE;IACpE,MAAM,EAAE,WAAW,CAAC;IACpB;;;;OAIG;IACH,WAAW,EAAE,MAAM,CAAC;CACrB;AAWD;;;;GAIG;AACH,qBAAa,iBAAiB;;gBAGhB,EAAE,MAAM,EAAE,WAAW,EAAE,EAAE,wBAAwB;IAgB7D;;;OAGG;IACH,IAAI,sBAAsB,IAAI,OAAO,CAEpC;IAED,KAAK,CAAC,UAAU,EAAE,UAAU,CAAC,cAAc,CAAC,GAAG,IAAI;IAInD,GAAG,CAAC,KAAK,EAAE,KAAK,EAAE,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAI7C,sHAAsH;IACtH,MAAM,CAAC,KAAK,EAAE,KAAK,EAAE,GAAG,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAIhE,QAAQ,CAAC,KAAK,EAAE,KAAK,EAAE,GAAG,EAAE,MAAM,EAAE,KAAK,EAAE,YAAY,GAAG,IAAI;IAI9D,IAAI,CAAC,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,OAAO,GAAG,IAAI;IAI5C,qDAAqD;IACrD,SAAS,CAAC,MAAM,EAAE,MAAM,GAAG,IAAI;CAGhC"}
+{"version":3,"file":"flight-coordinator.d.ts","sourceRoot":"","sources":["../../src/coordinator/flight-coordinator.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,YAAY,EAAE,WAAW,EAAE,MAAM,kBAAkB,CAAC;AAElE,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,6BAA6B,CAAC;AAC9D,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,0BAA0B,CAAC;AACzD,OAAO,KAAK,EAAE,cAAc,EAAE,OAAO,EAAE,MAAM,gCAAgC,CAAC;AAC9E,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAErC,OAAO,KAAK,EAAE,KAAK,EAAE,MAAM,YAAY,CAAC;AAGxC,MAAM,WAAW,wBAAwB;IACvC,oEAAoE;IACpE,MAAM,EAAE,WAAW,CAAC;IACpB;;;OAGG;IACH,KAAK,EAAE,QAAQ,CAAC;CACjB;AAQD;;;;;;;;;GASG;AACH,qBAAa,iBAAiB;;gBAShB,EAAE,MAAM,EAAE,KAAK,EAAE,EAAE,wBAAwB;IAMvD,KAAK,CAAC,UAAU,EAAE,UAAU,CAAC,cAAc,CAAC,GAAG,IAAI;IAInD,4DAA4D;IAC5D,KAAK,IAAI,IAAI;IAIb,gDAAgD;IAC1C,GAAG,CAAC,KAAK,EAAE,KAAK,EAAE,GAAG,EAAE,MAAM;IAKnC;;;;OAIG;IACH,MAAM,CAAC,KAAK,EAAE,KAAK,EAAE,GAAG,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,GAAG,IAAI;IAevD;;;;OAIG;IACH,QAAQ,CAAC,KAAK,EAAE,KAAK,EAAE,GAAG,EAAE,MAAM,EAAE,KAAK,EAAE,YAAY,GAAG,IAAI;IAe9D,IAAI,CAAC,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,OAAO;IAKrC,qDAAqD;IACrD,SAAS,CAAC,MAAM,EAAE,MAAM;CAwDzB"}