@byollm/relay 0.1.0-alpha.11 → 0.1.0-alpha.13

package/README.md

@@ -1,5 +1,5 @@
 > [!WARNING]
-> **Alpha (`0.1.0-alpha.11`) — under active development. Don't use this yet.**
+> **Alpha (`0.1.0-alpha.13`) — under active development. Don't use this yet.**
 >
 > This is a walking skeleton. It routes real jobs between real daemons and real
 > sites, and it is the fixture byollm_009 freezes against — but it keeps its
@@ -80,7 +80,7 @@ daemons pin at pairing, verified against the `sites` half of the projection.
 Nothing here trusts a `siteId` in a body or a query string.
 
 That is newer than the rest of this package. The site plane took the caller's
-word for who it was until `0.1.0-alpha.11`, which on a relay reachable from the
+word for who it was until `0.1.0-alpha.13`, which on a relay reachable from the
 internet is an open enqueue endpoint into consenting users' machines and an
 open read of who is online. It was blind the whole time — nothing could open a
 payload — and blind is not the same as safe.
@@ -88,6 +88,24 @@ payload — and blind is not the same as safe.
 If you are running this: the site plane is authenticated but this is still a
 single-tenant relay with in-memory state. One site, one replica.
 
+## Breaking in `0.1.0-alpha.13`: `RelayState` is async
+
+Every method on `RelayState` now returns a `Promise`, and `Relay.sweep()` and
+`debugPage()` with it. `RelayState.requeue` is private — it was only ever a
+step inside another operation.
+
+Nothing about the behaviour changed. The shape did, and it had to before
+routing state can live anywhere but this process: a store on a network cannot
+offer a synchronous read, and — more importantly — cannot offer a *read the
+caller follows with a write*. So the operations are now decisions plus their
+writes (`claim`, `takePayload`, `complete`, `releaseLeases`, `seal`) rather
+than scans the caller mutates.
+
+`claim` is the one that matters. It was atomic for exactly one reason — Node
+is single-threaded and the Maps are local — and `CLAIM_ATOMIC` is a MUST. See
+`packages/relay/test/two-replicas.test.ts`, where the resulting race is a
+failing assertion waiting for the fix.
+
 ## Running it
 
 ```ts

package/dist/index.d.ts

@@ -1,4 +1,4 @@
-import { PublicIdentity, JobStub, SealedEnvelope } from '@byollm/protocol';
+import { PublicIdentity, JobStub, SealedEnvelope, ClaimedStub } from '@byollm/protocol';
 import { z } from 'zod';
 
 /**
@@ -93,6 +93,74 @@ interface Presence {
     /** Set on revocation so the next request is refused rather than routed. */
     revoked: boolean;
 }
+/**
+ * What a routing store must do, expressed as operations — cloud_006 §3.2.
+ *
+ * Every method below is a **decision plus its write**, never a read the caller
+ * follows with a mutation. That is the whole point, and it is the difference
+ * between an interface a shared store can implement and one it cannot.
+ *
+ * `claim` is the specimen. It used to live in `DaemonPlane` as
+ * `jobs()` → filter → mutate, which is atomic for exactly one reason: Node is
+ * single-threaded and these Maps are local, so nothing runs between the read
+ * and the write. Neither survives a store on a network, and
+ * `packages/relay/test/two-replicas.test.ts` holds the resulting race as a
+ * failing assertion.
+ *
+ * So the rule for anything added here: **if a caller has to read, decide, and
+ * write back, the operation is in the wrong place.** Move the decision in.
+ *
+ * ## Why the projection does not come with it
+ *
+ * `claim` takes `owners: string[]` rather than a projection or a predicate.
+ * A closure cannot travel to Valkey, and the projection replicates for free
+ * from the control plane — so the caller collapses it with
+ * `Projection.ownersRunnableBy` and hands over data the store can match on.
+ * That keeps the store ignorant of consent, which is also what keeps it
+ * replaceable.
+ */
+interface ClaimInput {
+    readonly runnerId: string;
+    readonly owner: string;
+    readonly device: PublicIdentity;
+    /** The site this relay routes for. Multi-tenancy widens this to a set. */
+    readonly siteId: string;
+    /** Job kinds this device can actually run. */
+    readonly kinds: ReadonlySet<string>;
+    /** Whose work it may run — the projection, already collapsed to data. */
+    readonly owners: ReadonlySet<string>;
+    readonly max: number;
+    readonly leaseMs: number;
+}
+/**
+ * Where the store's sense of time comes from — cloud_006 §3.4.
+ *
+ * **The store owns its clock; callers do not pass one.** Every deadline the
+ * relay decides — a lease's expiry, the `awaiting-payload` window, what a
+ * sweep considers due — is now stamped by one source, and it is the same
+ * source that will later stamp them for every replica.
+ *
+ * It used to be a parameter. `claim` took `now`, `sweep` took `now`, and each
+ * plane called its own `now()` before calling in — which is fine in one
+ * process and is the recurring bug the moment there are two. A lease granted
+ * by a pod whose clock runs fast is short; the same lease swept by a pod whose
+ * clock runs slow outlives it. Nobody is wrong and the lease has no length.
+ *
+ * A Valkey-backed store returns `TIME` here, so the deadline and the sweep
+ * that enforces it are read from the same server. The injected clock stays for
+ * tests, which is what lets them move time instead of sleeping.
+ *
+ * **What deliberately does not use this**: request-signature freshness. That
+ * is checked against the *local* clock on purpose — it is a question about the
+ * caller's clock versus this process's, `MAX_CLOCK_SKEW_MS` already tolerates
+ * two minutes of disagreement, and a network round trip to timestamp every
+ * inbound request would be a cost with no property behind it.
+ */
+interface RelayStateOptions {
+    readonly now?: () => number | Promise<number>;
+}
+/** Why a lease-scoped operation was refused, in the caller's vocabulary. */
+type HolderRefusal = "not-found" | "not-holder" | "stale-lease" | "not-ready";
 /**
  * In-memory routing state.
  *
@@ -102,8 +170,11 @@ interface Presence {
  * returns to its site's queue — which is the behaviour a lapsed lease already
  * has to produce, so nothing new needs to be true for this to be safe.
  */
-declare class RelayState {
+declare class RelayState implements RoutingStore {
     #private;
+    constructor(options?: RelayStateOptions);
+    /** The one clock every deadline in this store is stamped from. */
+    now(): Promise<number>;
     /**
      * Take a stub for routing. The payload is not here and will not be.
      *
@@ -126,24 +197,92 @@ declare class RelayState {
         id: string;
         siteId: string;
         stub: JobStub;
-    }): RoutedJob;
-    job(jobId: string): RoutedJob | undefined;
-    jobs(): RoutedJob[];
+    }): Promise<RoutedJob>;
+    job(jobId: string): Promise<RoutedJob | undefined>;
+    jobs(): Promise<RoutedJob[]>;
     /** Jobs a site must seal for, right now. */
-    awaiting(siteId: string): RoutedJob[];
+    awaiting(siteId: string): Promise<RoutedJob[]>;
     /** Sealed results waiting to go home. */
-    finished(siteId: string): RoutedJob[];
-    seen(presence: Omit<Presence, "revoked">): Presence;
-    presence(runnerId: string): Presence | undefined;
-    everyone(): Presence[];
+    finished(siteId: string): Promise<RoutedJob[]>;
+    /**
+     * Claim work — one operation, because it has to be.
+     *
+     * Moved here wholesale from `DaemonPlane`, where it was a scan followed by
+     * per-job mutation. Nothing about the *decision* changed; what changed is
+     * that a store can now implement it, because the filter and the write are
+     * one call rather than a loop the caller drives.
+     *
+     * The order of the guards is worth preserving as-is when this becomes a Lua
+     * script: cheapest first, and `owners` last because it is the only one that
+     * needed the projection.
+     */
+    claim(input: ClaimInput): Promise<ClaimedStub[]>;
     /**
-     * Return a job to the queue, forgetting the claim.
+     * Hand over the sealed payload to the device that holds the lease.
      *
-     * The stub survives; nothing is lost. That is `LEASE_RECLAIMABLE` and it is
-     * why the awaiting-payload timeout is cheap to fire: the worst case is that
-     * a device did nothing for ten seconds and another one gets a turn.
+     * The read and the state transition are one operation for the same reason
+     * `claim` is: `running` must be set by whoever was told the envelope, or two
+     * replicas can both hand out the same work and both believe they were first.
      */
-    requeue(job: RoutedJob): void;
+    takePayload(input: {
+        jobId: string;
+        runnerId: string;
+        leaseId: string;
+    }): Promise<{
+        envelope: SealedEnvelope;
+    } | {
+        refused: HolderRefusal;
+    }>;
+    /**
+     * Record a finished job.
+     *
+     * `RESULT_IDEMPOTENT` lives here rather than in the caller: a replayed
+     * result must be a no-op decided by the same operation that would have
+     * written it, or two replicas can both decide they were the first.
+     */
+    complete(input: {
+        jobId: string;
+        runnerId: string;
+        envelope: SealedEnvelope;
+        disposition: "ok" | "error" | "canceled";
+    }): Promise<{
+        accepted: boolean;
+        state: RoutedState;
+    } | {
+        refused: HolderRefusal;
+    }>;
+    /** Give back leases this runner holds, naming each grant it means. */
+    releaseLeases(input: {
+        runnerId: string;
+        leases: readonly {
+            jobId: string;
+            leaseId: string;
+        }[];
+    }): Promise<string[]>;
+    /**
+     * Take a site's sealed payload for a claimed job.
+     *
+     * Refuses anything not `awaiting-payload`, which is what makes the timeout
+     * mean something: a late seal must not land on a claim that has moved.
+     */
+    seal(input: {
+        jobId: string;
+        siteId: string;
+        envelope: SealedEnvelope;
+    }): Promise<{
+        state: RoutedState;
+    } | {
+        refused: "not-found" | "too-late";
+        was?: RoutedState;
+    }>;
+    /** Which of these leases this runner no longer holds. */
+    lostLeases(runnerId: string, active: readonly {
+        jobId: string;
+        leaseId: string;
+    }[]): Promise<string[]>;
+    seen(presence: Omit<Presence, "revoked" | "lastSeenAt">): Promise<Presence>;
+    presence(runnerId: string): Promise<Presence | undefined>;
+    everyone(): Promise<Presence[]>;
     /**
      * Fire whatever the clock says is due, and report it.
      *
@@ -151,10 +290,125 @@ declare class RelayState {
      * timeout that fires invisibly is indistinguishable from a job that was
      * never claimed, and those want very different debugging.
      */
-    sweep(now: number): RoutedJob[];
+    sweep(): Promise<RoutedJob[]>;
 }
 
-declare function debugPage(state: RelayState, now: number): string;
+/**
+ * What the relay needs from a place to keep routing state — cloud_006 §3.2.
+ *
+ * `RelayState` implements this in memory and is the reference; a Valkey-backed
+ * store implements the same thing across replicas. The interface exists so the
+ * relay depends on the *contract* rather than on either, and so the properties
+ * below are stated once rather than rediscovered per implementation.
+ *
+ * ## Every method is a decision plus its write
+ *
+ * Not one is a read the caller follows with a mutation. That is the whole
+ * design, and it is not a style preference: `claim` used to be
+ * `jobs()` → filter → mutate in the plane, which is atomic for exactly one
+ * reason — Node is single-threaded and the Maps are local. Neither survives a
+ * store on a network, and `packages/relay/test/two-replicas.test.ts` holds the
+ * resulting race as a failing assertion.
+ *
+ * **The rule for anything added here:** if a caller has to read, decide, and
+ * write back, the operation is in the wrong place. Move the decision in.
+ *
+ * ## What an implementation must guarantee
+ *
+ * 1. **`claim` is atomic.** Two callers claiming concurrently must not both
+ *    receive the same job. `CLAIM_ATOMIC` is a protocol MUST.
+ * 2. **`enqueue` is idempotent by job id.** A known id returns what is already
+ *    routing rather than rebuilding it — byollm_009 §4.2's replay argument
+ *    rests on every write being idempotent per the instance it names.
+ * 3. **`complete` is idempotent.** A replayed result changes nothing, and the
+ *    decision is made by the same operation that would have written it.
+ * 4. **Lease-scoped operations name the grant.** `takePayload`, `complete` and
+ *    `releaseLeases` check the lease *id*, not just the runner — a runner
+ *    survives a claim-release-reclaim cycle and a grant does not.
+ * 5. **`now()` is the only clock.** Every deadline the store stamps and every
+ *    deadline it enforces come from here (§3.4). An implementation backed by a
+ *    server returns that server's time, so two replicas cannot disagree about
+ *    how long a lease is.
+ *
+ * ## What it must not do
+ *
+ * Hold a key, or learn about consent. `claim` takes `owners` as data because a
+ * predicate cannot travel to Valkey — and the effect is that the store cannot
+ * express an opinion about who may route, only about what it was told. That is
+ * what keeps `RELAY_BLIND` a property of the shape rather than of the code.
+ */
+interface RoutingStore {
+    /** The one clock every deadline in this store is stamped from. */
+    now(): Promise<number>;
+    /** Take a stub for routing. Idempotent by id. */
+    enqueue(input: {
+        id: string;
+        siteId: string;
+        stub: JobStub;
+    }): Promise<RoutedJob>;
+    job(jobId: string): Promise<RoutedJob | undefined>;
+    jobs(): Promise<RoutedJob[]>;
+    /** Jobs a site must seal for, right now. */
+    awaiting(siteId: string): Promise<RoutedJob[]>;
+    /** Sealed results waiting to go home. */
+    finished(siteId: string): Promise<RoutedJob[]>;
+    /** Grant work to a device — one operation, because it has to be. */
+    claim(input: ClaimInput): Promise<ClaimedStub[]>;
+    /** Hand the sealed payload to the device that holds the lease. */
+    takePayload(input: {
+        jobId: string;
+        runnerId: string;
+        leaseId: string;
+    }): Promise<{
+        envelope: SealedEnvelope;
+    } | {
+        refused: HolderRefusal;
+    }>;
+    /** Record a finished job. Idempotent. */
+    complete(input: {
+        jobId: string;
+        runnerId: string;
+        envelope: SealedEnvelope;
+        disposition: "ok" | "error" | "canceled";
+    }): Promise<{
+        accepted: boolean;
+        state: RoutedState;
+    } | {
+        refused: HolderRefusal;
+    }>;
+    /** Give back the grants this runner names. */
+    releaseLeases(input: {
+        runnerId: string;
+        leases: readonly {
+            jobId: string;
+            leaseId: string;
+        }[];
+    }): Promise<string[]>;
+    /** Take a site's sealed payload for a claimed job. */
+    seal(input: {
+        jobId: string;
+        siteId: string;
+        envelope: SealedEnvelope;
+    }): Promise<{
+        state: RoutedState;
+    } | {
+        refused: "not-found" | "too-late";
+        was?: RoutedState;
+    }>;
+    /** Which of these leases this runner no longer holds. */
+    lostLeases(runnerId: string, active: readonly {
+        jobId: string;
+        leaseId: string;
+    }[]): Promise<string[]>;
+    /** Record a device as present. The store stamps when. */
+    seen(presence: Omit<Presence, "revoked" | "lastSeenAt">): Promise<Presence>;
+    presence(runnerId: string): Promise<Presence | undefined>;
+    everyone(): Promise<Presence[]>;
+    /** Fire whatever the clock says is due, and report it. */
+    sweep(): Promise<RoutedJob[]>;
+}
+
+declare function debugPage(state: RoutingStore, now: number): Promise<string>;
 
 /**
  * What the relay is told about the world — cloud_004 §14.
@@ -335,6 +589,30 @@ declare class Projection {
     deviceByFingerprint(identityPublic: string): DeviceRecord | null;
     /** The consent binding this owner to this site, if it exists and stands. */
     consentFor(owner: string, siteId: string): ConsentRecord | null;
+    /**
+     * Every owner whose work this device's owner may run, as a list.
+     *
+     * The same question {@link mayRunFor} answers, asked in the direction a
+     * *store* can use. That difference is the crux of making `claim` atomic
+     * (cloud_006 §3.2).
+     *
+     * Today `claim` scans every job and calls `mayRunFor` per candidate, which
+     * works because the projection is a local object. A shared routing store
+     * cannot do that: the filter has to travel to the store, and a predicate
+     * does not travel — you cannot send a closure to Valkey. So the projection
+     * is collapsed to **data** here and handed over as a set the store can
+     * match on.
+     *
+     * That the collapse is possible at all is a property of the design worth
+     * noticing: `mayRunFor` is a finite lookup over consent and rosters, not a
+     * computation over the jobs. If it ever became job-dependent — "may run
+     * work of this size", say — an atomic claim would stop being expressible,
+     * and that is the moment to argue rather than to add a parameter.
+     *
+     * The owner is always included: a device runs its owner's work, and the
+     * relay checks that before it checks a roster.
+     */
+    ownersRunnableBy(deviceOwner: string): string[];
     /**
      * May this device's owner run work belonging to `jobOwner`?
      *
@@ -388,11 +666,27 @@ interface RelayOptions {
     readonly now?: () => number;
     /** Where the daemon plane is mounted. */
     readonly basePath?: string;
+    /**
+     * Where routing state lives — cloud_006.
+     *
+     * Defaults to an in-process {@link RelayState}, which is correct for one
+     * replica and is what this package ships. A hub running more than one
+     * replica supplies a shared implementation of {@link RoutingStore} instead;
+     * `packages/relay/test/two-replicas.test.ts` is why that is not optional.
+     *
+     * **The implementation is deliberately not in this package.** A Valkey
+     * client is a dependency every consumer would carry to get a feature only a
+     * multi-replica deployment uses, and the production hub is the closed piece
+     * (cloud_001). What ships here is the interface, the reference
+     * implementation, and the tests that say what an implementation must
+     * guarantee.
+     */
+    readonly store?: RoutingStore;
 }
 /** A running relay: one fetch handler, two planes, one debug page. */
 declare class Relay {
     #private;
-    readonly state: RelayState;
+    readonly state: RoutingStore;
     readonly projection: Projection;
     constructor(options: RelayOptions);
     /** Replace the projection — a control-plane push, or a fixture edit. */
@@ -406,11 +700,11 @@ declare class Relay {
      * site vanished should return to the queue without waiting for someone to
      * ask about it.
      */
-    sweep(): {
+    sweep(): Promise<{
         requeued: string[];
-    };
+    }>;
     /** The whole HTTP surface. */
     handle(request: Request): Promise<Response>;
 }
 
-export { AWAITING_PAYLOAD_MS, ConsentRecord, DeviceRecord, EMPTY_FIXTURE, type Presence, Projection, Relay, RelayFixture, RelayFixture as RelayFixtureSchema, type RelayOptions, RelayState, RevocationRecord, RosterRecord, type RoutedJob, type RoutedState, SiteRecord, debugPage };
+export { AWAITING_PAYLOAD_MS, type ClaimInput, ConsentRecord, DeviceRecord, EMPTY_FIXTURE, type HolderRefusal, type Presence, Projection, Relay, RelayFixture, RelayFixture as RelayFixtureSchema, type RelayOptions, RelayState, RevocationRecord, RosterRecord, type RoutedJob, type RoutedState, type RoutingStore, SiteRecord, debugPage };