@toa.io/extensions.cadence 1.0.0-alpha.286 → 1.0.0-alpha.287

package/CHANGELOG.md

@@ -0,0 +1,10 @@
+# Change Log
+
+All notable changes to this project will be documented in this file.
+See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
+
+# [1.0.0-alpha.287](https://github.com/toa-io/toa/compare/v1.0.0-alpha.286...v1.0.0-alpha.287) (2026-09-06)
+
+### Bug Fixes
+
+* **cadence:** the scan reads what is owed, and nothing that is done ([581610e](https://github.com/toa-io/toa/commit/581610e87ac54c42491c882d5749ff3307700d35))

package/components/cadence.metronome/migrations/0003-pending-index.yaml

@@ -0,0 +1,16 @@
+# The index holds what is still owed, and nothing else.
+#
+# Without the filter a scan walks every row the collection has ever held: the criteria bounds
+# `due` from above and not below, and `DELETED` is not in the index, so every tombstone is
+# fetched and discarded on the way to the few rows that are live. The cost of a pass grows with
+# the age of the deployment until it outlives its own interval.
+#
+# The name is the one 0002 gave it, deliberately: an index whose declaration changed is dropped
+# and made again.
+- index:
+    name: index_due
+    keys:
+      lane: asc
+      due: asc
+    partial:
+      DELETED: null

package/components/cadence.metronome/migrations/0004-retention.yaml

@@ -0,0 +1,15 @@
+# A settled row is kept, and not kept forever.
+#
+# `DELETED` is what `settle` writes and what the runtime holds as a date, so the row ages out
+# on the moment its call was disposed of. A row still owed has `DELETED: null`, and the TTL
+# monitor passes over a record whose field is not a date — so nothing waiting to be called can
+# be reaped by this, however long it has been waiting.
+#
+# Thirty days. The record answers what was asked for and when it was disposed of, which is a
+# question asked while an incident is recent; a year of it is a collection nobody reads and
+# everybody stores.
+- index:
+    name: index_deleted
+    keys:
+      DELETED: asc
+    ttl: 2592000

package/components/cadence.metronome/operations/settle.js

@@ -1,6 +1,10 @@
 /**
- * The rows whose calls have been made. They are tombstoned rather than removed: the record of
- * what was asked for outlives the call, and a query answers only what is still owed.
+ * The rows whose calls have been disposed of — made, or found past the bound their caller gave
+ * them. They are tombstoned rather than removed: the record of what was asked for outlives the
+ * call, and a query answers only what is still owed.
+ *
+ * `DELETED` is also what the retention index reaps by, so that the record outlives the call
+ * without outliving the deployment.
  */
 export function transition(_, objects) {
   const now = Date.now()

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@toa.io/extensions.cadence",
-  "version": "1.0.0-alpha.286",
+  "version": "1.0.0-alpha.287",
   "type": "module",
   "description": "Toa Schedule",
   "author": "temich <tema.gurtovoy@gmail.com>",
@@ -18,12 +18,12 @@
     "access": "public"
   },
   "dependencies": {
-    "@toa.io/core": "1.0.0-alpha.286",
-    "@toa.io/schemas": "1.0.0-alpha.286",
+    "@toa.io/core": "1.0.0-alpha.287",
+    "@toa.io/schemas": "1.0.0-alpha.287",
     "openspan": "1.0.0-alpha.286"
   },
   "scripts": {
     "transpile": "tsc"
   },
-  "gitHead": "81f3d0688f17668b902be336d1749cc715edcdb6"
+  "gitHead": "5c7944a70c16065ab6372072385cc1f02f5db77c"
 }

package/readme.md

@@ -142,12 +142,23 @@ it already past its bound and leaves it. State an `overdue` several times `discr
 `null`.
 
 **A cancellation is reliable while the call is further out than `discreteness`**, and is a race
-closer than that, where the call may already be on its way.
+closer than that, where the call may already be on its way. What bounds it is the period between
+passes: a cancelled call stops being owed, the next pass stops reading it, and a call taken in
+hand is given up again when that happens.
+
+**A call is due by the clock of whichever replica took the `delay`**, and made by the clock of
+whichever one dispatches it. The overlap covers the disagreement between two machines kept in
+ordinary time; nothing here measures it.
 
 So `discreteness` and `overdue` are chosen together against what the application needs. A call
 that must be made within seconds of its time, or cancellable within seconds of it, needs a
 `discreteness` of seconds — at the cost of looking the calls over that much more often.
 
+**A pass takes a bounded number of calls**, 200 to a replica, oldest first. Where more than that
+come due to one replica in one period, the rest are taken on a later pass and are late by however
+long that took; where it is logged, the rate is past what one period can carry. Replicas divide
+the lanes between them, so the ceiling is per replica rather than for the application.
+
 ### What to expect
 
 **A call may be made more than once**, so where a second one would cost something, the target
@@ -186,6 +197,6 @@ cadence:
 ```
 
 `discreteness` is how often the calls waiting to be made are looked over. It defaults to 60,
-which is what an application that states nothing gets. It is the floor under how closely a
-delayed call can be cancelled and how small an `overdue` is worth stating — see
+which is what an application that states nothing gets. It sets how small an `overdue` is worth
+stating, and it is the floor under how closely a delayed call can be cancelled — see
 [Discreteness](#discreteness) — and lowering it costs a pass that runs more often.

package/source/Dispatcher.test.ts

@@ -1,8 +1,9 @@
 import { it, beforeEach, afterEach, mock } from 'node:test'
 import assert from 'node:assert/strict'
+import { console } from 'openspan'
 
 import { Dispatcher } from './Dispatcher.js'
-import { LANES } from './const.js'
+import { BATCH, LANES } from './const.js'
 import type { Local } from './Local.js'
 import type { atomicity } from '@toa.io/core/types'
 
@@ -17,19 +18,23 @@ let target: Local & { invoke: ReturnType<typeof mock.fn<Invoke>> }
 let atom: atomicity.Atom & { slots: ReturnType<typeof mock.fn<Slots>> }
 let listeners: Listener[]
 let rows: Row[]
+let warn: ReturnType<typeof mock.method<typeof console, 'warn'>>
 
 interface Row {
   id: string
   lane: number
   due: number
+  expires: number
   endpoint: string
   request: object | null
 }
 
-const row = (id: string, due: number, lane = 0): Row => ({
+/** `expires` defaults to no bound at all, which is what `overdue: null` writes */
+const row = (id: string, due: number, lane = 0, expires = Number.MAX_SAFE_INTEGER): Row => ({
   id,
   lane,
   due,
+  expires,
   endpoint: 'tea.pots.transit',
   request: { input: { id } }
 })
@@ -69,6 +74,7 @@ beforeEach(() => {
 
   rows = []
   listeners = []
+  warn = mock.method(console, 'warn', () => undefined)
 
   metronome = connector({
     invoke: mock.fn<Invoke>(async (endpoint: string) =>
@@ -93,10 +99,11 @@ beforeEach(() => {
 
 afterEach(() => {
   mock.timers.reset()
+  warn.mock.restore()
   delete process.env.TOA_CADENCE_DISCRETENESS
 })
 
-it('should read what is due within the interval, in the lanes it owns', async () => {
+it('should read two intervals ahead, in the lanes it owns', async () => {
   const dispatcher = create()
 
   await dispatcher.connect()
@@ -104,14 +111,122 @@ it('should read what is due within the interval, in the lanes it owns', async ()
 
   const [first] = reads()
 
-  // the first scan is the one the subscription runs as it is added, at time zero
+  // the first scan is the one the subscription runs as it is added, at time zero. Two
+  // intervals, so that consecutive passes overlap; and no bound on `expires`, because an
+  // expired row is settled rather than left where nothing will ever read it again
   assert.strictEqual(
     first.criteria,
-    `lane=in=(${range(LANES).join(',')});due<${INTERVAL};expires>0`
+    `lane=in=(${range(LANES).join(',')});due<${2 * INTERVAL}`
   )
   assert.deepStrictEqual(first.sort, ['due:asc'])
 })
 
+it('should reach past the next pass, so a row on its horizon is armed before it is due', async () => {
+  rows = [row('a', INTERVAL + 10)]
+
+  const dispatcher = create()
+
+  await dispatcher.connect()
+
+  // gone from the store before the next pass, so nothing but the scan at time zero can have
+  // armed it — which is what reaching two intervals rather than one is for
+  rows = []
+
+  await advance(INTERVAL)
+
+  assert.strictEqual(target.invoke.mock.callCount(), 0, 'armed, not yet due')
+
+  await advance(10)
+
+  assert.strictEqual(target.invoke.mock.callCount(), 1, 'and called at its due time')
+})
+
+it('should settle an expired row without calling it', async () => {
+  rows = [row('a', -10_000, 0, -5_000)]
+
+  const dispatcher = create()
+
+  await dispatcher.connect()
+  await advance(0)
+
+  assert.strictEqual(target.invoke.mock.callCount(), 0, 'past the bound its caller gave it')
+
+  rows = []
+  await advance(INTERVAL)
+
+  assert.deepStrictEqual(
+    settled(),
+    ['a'],
+    'and settled, or nothing would ever read it again'
+  )
+})
+
+it('should settle what it has called even while it owns nothing', async () => {
+  rows = [row('a', 10)]
+
+  const dispatcher = create()
+
+  await dispatcher.connect()
+  await advance(10)
+
+  assert.strictEqual(target.invoke.mock.callCount(), 1)
+
+  // the lane is somebody else's now, and a row left live is one they call again
+  rows = []
+  atom.slots.mock.mockImplementation(() => null)
+
+  await advance(INTERVAL)
+
+  assert.deepStrictEqual(settled(), ['a'])
+})
+
+it('should say so when a scan fills its batch', async () => {
+  rows = Array.from({ length: BATCH }, (_, i) => row(String(i), 10_000))
+
+  const dispatcher = create()
+
+  await dispatcher.connect()
+  await advance(0)
+
+  const warned = warn.mock.calls.map((call) => call.arguments[0] as string)
+
+  assert.ok(
+    warned.some((message) => message.includes('filled its batch')),
+    'the horizon is truncated and the rest arrive late'
+  )
+})
+
+it('should say so when a pass is skipped because the one before it is still running', async () => {
+  let release: () => void = () => {}
+
+  metronome.invoke.mock.mockImplementation(async (endpoint: string) => {
+    if (endpoint !== 'enumerate') return null
+
+    await new Promise<void>((resolve) => {
+      release = resolve
+    })
+
+    return []
+  })
+
+  const dispatcher = create()
+
+  await dispatcher.connect()
+  await advance(0) // the scan at time zero, which does not come back
+
+  await advance(INTERVAL)
+
+  const warned = warn.mock.calls.map((call) => call.arguments[0] as string)
+
+  assert.ok(
+    warned.some((message) => message.includes('has not returned')),
+    'a dispatcher that stops dispatching says so'
+  )
+
+  release()
+  await advance(0)
+})
+
 it('should read nothing while it owns nothing', async () => {
   atom.slots.mock.mockImplementation(() => null)
 
@@ -308,3 +423,57 @@ it('should scan at once when what it owns changes', async () => {
 
   assert.strictEqual(reads().length, 1, 'without waiting out an interval')
 })
+
+it('should give up a row a scan stops reading, so a cancellation still lands', async () => {
+  rows = [row('a', INTERVAL + 50)]
+
+  const dispatcher = create()
+
+  await dispatcher.connect()
+  await advance(1)
+
+  // cancelled: the row is tombstoned, so the next scan does not read it
+  rows = []
+
+  await advance(INTERVAL) // the scan that finds it gone
+  await advance(INTERVAL) // and past when it would have been called
+
+  assert.strictEqual(target.invoke.mock.callCount(), 0, 'the timer was given up with the row')
+})
+
+it('should keep a row the scan still reads', async () => {
+  rows = [row('a', INTERVAL + 50)]
+
+  const dispatcher = create()
+
+  await dispatcher.connect()
+  await advance(INTERVAL)
+
+  assert.strictEqual(target.invoke.mock.callCount(), 0, 'not yet due')
+
+  await advance(50)
+
+  assert.strictEqual(target.invoke.mock.callCount(), 1)
+})
+
+it('should not give up a row when the batch was filled', async () => {
+  const held = row('held', INTERVAL + 50)
+
+  rows = [held, ...Array.from({ length: BATCH - 1 }, (_, i) => row('b' + i, 10_000))]
+
+  const dispatcher = create()
+
+  await dispatcher.connect()
+  await advance(1)
+
+  // a full batch is a read that was cut short, so absence from the next one is not a
+  // cancellation — these are rows it never reached, not rows nobody owes
+  rows = Array.from({ length: BATCH }, (_, i) => row('c' + i, 20_000))
+
+  await advance(INTERVAL)
+  await advance(50)
+
+  const called = target.invoke.mock.calls.map((call) => call.arguments[1].input.id)
+
+  assert.deepStrictEqual(called, ['held'], 'the row it armed is still called')
+})

package/source/Dispatcher.ts

@@ -8,9 +8,10 @@ import type { atomicity } from '@toa.io/core/types'
  * Makes the calls that were put off.
  *
  * Two tiers, which is what a store polled rarely and a call due to the second need between
- * them: the rows are the record, and one scan primes the next interval of them into timers.
- * A crash loses the timers and nothing else — the next scan reads what is overdue along with
- * what is upcoming, so a dispatcher that died mid-interval costs a delay and not a call.
+ * them: the rows are the record, and one scan primes the next two intervals of them into
+ * timers. A crash loses the timers and nothing else — the next scan reads what is overdue
+ * along with what is upcoming, so a dispatcher that died mid-interval costs a delay and not a
+ * call.
  *
  * Partitioned by lane exactly as the outbox is, so every replica scans and dispatches its own
  * share of every interval rather than taking turns at whole ones.
@@ -37,6 +38,9 @@ export class Dispatcher extends Connector {
   private scanning = false
   private closing = false
 
+  /** when the scan that is running started, so a tick that finds it can say how long */
+  private started = 0
+
   public constructor(
     metronome: Local,
     resolve: (locator: Locator) => Local,
@@ -80,9 +84,28 @@ export class Dispatcher extends Connector {
 
   /** One scan at a time: a slow one must not have another started underneath it. */
   private tick(): void {
-    if (this.scanning || this.closing) return
+    if (this.closing) return
+
+    /*
+     * Skipped, and said so. The guard itself is right — a second scan on top of one that has
+     * not returned only makes it worse — but a pass that outlives its interval takes every
+     * pass after it with it, and nothing is dispatched at all while that lasts. Silently, it
+     * is a dispatcher that stops dispatching in a deployment that ran for months.
+     */
+    if (this.scanning) {
+      const running = Date.now() - this.started
+      const details = { running, discreteness: this.discreteness }
+      const message = 'Delayed calls scan has not returned, so this pass is skipped'
+
+      // past a few intervals it is not a slow pass but a stuck one
+      if (running > STALLED * this.discreteness) console.error(message, details)
+      else console.warn(message, details)
+
+      return
+    }
 
     this.scanning = true
+    this.started = Date.now()
 
     void this.scan()
       .catch((error: unknown) => {
@@ -117,25 +140,71 @@ export class Dispatcher extends Connector {
   }
 
   private async scan(): Promise<void> {
+    // ahead of the claim, because settling needs none: the rows are ones this replica has
+    // already called, and one left live is one the next owner of its lane calls again
+    await this.settle()
+
     const lanes = this.atom.slots(LANES)
 
     if (lanes === null || lanes.length === 0) return
 
-    await this.settle()
-
     const now = Date.now()
-    const until = now + this.discreteness
+
+    /*
+     * Two intervals, so that consecutive passes overlap by one whole one. Reaching exactly as
+     * far as the next pass leaves a sliver beyond the horizon that the next pass reads already
+     * overdue — the width of the drift between the two, which a row with a tight `overdue`
+     * does not survive. The overlap absorbs that, a settle that ran long, a tick skipped
+     * outright, and the clock skew between the replica that wrote `due` and this one.
+     *
+     * Reaching further costs nothing a cancellation would pay for, because what is armed is
+     * given up again the moment a scan stops reading it — see `forget`.
+     */
+    const until = now + 2 * this.discreteness
 
     const rows = (await this.metronome.invoke('enumerate', {
       query: {
-        // a call nothing was running to make in time has expired, and is simply not read
-        criteria: `lane=in=(${lanes.join(',')});due<${until};expires>${now}`,
+        // expiry is not a criterion: a row nobody reads is a row nobody settles, and it would
+        // stay live for good. It is read, and settled without the call being made
+        criteria: `lane=in=(${lanes.join(',')});due<${until}`,
         sort: ['due:asc'],
         limit: BATCH
       }
     })) as Row[]
 
+    // the horizon is truncated to the last row of a full batch, and everything past it arrives
+    // on a later pass already overdue. Where this is logged, the batch is too small for the rate
+    const truncated = rows.length === BATCH
+
+    if (truncated)
+      console.warn('Delayed calls scan filled its batch', { limit: BATCH, lanes: lanes.length })
+
     for (const row of rows) this.arm(row)
+
+    if (!truncated) this.forget(new Set(rows.map((row) => row.id)))
+  }
+
+  /**
+   * Gives up what the store no longer answers with. A scan reads everything still owed in the
+   * lanes this replica holds, so a row it armed before and does not read now is one nothing
+   * owes any more — cancelled, most of the time — and a timer left for it makes the call
+   * regardless.
+   *
+   * This is what lets a scan reach further ahead than the next one without a cancellation
+   * becoming any less reliable: a row is held only for as long as the store keeps saying it is
+   * owed, so what bounds cancelling is the period between passes and not how far one reaches.
+   *
+   * Two things are not evidence of a cancellation. A batch filled to its limit is a read that
+   * was cut short, and says nothing about what it did not reach. And a call already on its way
+   * is settled by its own path rather than by what a scan reads.
+   */
+  private forget(read: Set<string>): void {
+    for (const [id, { timer }] of this.armed) {
+      if (read.has(id) || this.called.has(id)) continue
+
+      clearTimeout(timer)
+      this.armed.delete(id)
+    }
   }
 
   /**
@@ -145,11 +214,25 @@ export class Dispatcher extends Connector {
   private arm(row: Row): void {
     if (this.armed.has(row.id) || this.called.has(row.id)) return
 
+    const now = Date.now()
+
+    /*
+     * Past the bound its caller gave it, so the call is not made — and the row is settled all
+     * the same, silently, because an expired call is not an event anything sees. Left unread
+     * it would never be settled either: a row nothing will ever act on, sitting in the
+     * collection and in front of every scan, for good.
+     */
+    if (row.expires <= now) {
+      this.called.add(row.id)
+
+      return
+    }
+
     const timer = setTimeout(
       () => {
         void this.dispatch(row)
       },
-      Math.max(row.due - Date.now(), 0)
+      Math.max(row.due - now, 0)
     )
 
     timer.unref()
@@ -239,6 +322,13 @@ interface Row {
   id: string
   lane: number
   due: number
+
+  /** past this the call is not made; the row is settled without it */
+  expires: number
+
   endpoint: string
   request?: object
 }
+
+/** intervals a scan may run for before it is a stuck pass rather than a slow one */
+const STALLED = 5

package/source/const.ts

@@ -6,6 +6,12 @@ export const COMPONENT = 'metronome'
  * Constant, never configuration: a row carries its lane, so lowering this would leave rows in
  * lanes nobody reads any more. It is also the ceiling on replicas dispatching, and a power of
  * two so that the common replica counts divide evenly. The outbox says the same of its own.
+ *
+ * A scan asks for its lanes as a set and sorts on `due`, which is the index's second key, so
+ * the plan that serves it explodes the set into one scan per lane and merges them in order.
+ * MongoDB stops exploding past `internalQueryMaxScansToExplode`, 200 by default, and falls
+ * back to sorting the whole result in memory — so this staying well under that is not
+ * incidental.
  */
 export const LANES = 128
 

package/transpiled/Dispatcher.d.ts

@@ -5,9 +5,10 @@ import type { atomicity } from '@toa.io/core/types';
  * Makes the calls that were put off.
  *
  * Two tiers, which is what a store polled rarely and a call due to the second need between
- * them: the rows are the record, and one scan primes the next interval of them into timers.
- * A crash loses the timers and nothing else — the next scan reads what is overdue along with
- * what is upcoming, so a dispatcher that died mid-interval costs a delay and not a call.
+ * them: the rows are the record, and one scan primes the next two intervals of them into
+ * timers. A crash loses the timers and nothing else — the next scan reads what is overdue
+ * along with what is upcoming, so a dispatcher that died mid-interval costs a delay and not a
+ * call.
  *
  * Partitioned by lane exactly as the outbox is, so every replica scans and dispatches its own
  * share of every interval rather than taking turns at whole ones.
@@ -28,6 +29,8 @@ export declare class Dispatcher extends Connector {
     private off?;
     private scanning;
     private closing;
+    /** when the scan that is running started, so a tick that finds it can say how long */
+    private started;
     constructor(metronome: Local, resolve: (locator: Locator) => Local, atom: atomicity.Atom);
     protected open(): Promise<void>;
     protected close(): Promise<void>;
@@ -42,6 +45,21 @@ export declare class Dispatcher extends Connector {
     /** What is armed for a lane this replica no longer holds: whoever holds it now will arm it. */
     private disarm;
     private scan;
+    /**
+     * Gives up what the store no longer answers with. A scan reads everything still owed in the
+     * lanes this replica holds, so a row it armed before and does not read now is one nothing
+     * owes any more — cancelled, most of the time — and a timer left for it makes the call
+     * regardless.
+     *
+     * This is what lets a scan reach further ahead than the next one without a cancellation
+     * becoming any less reliable: a row is held only for as long as the store keeps saying it is
+     * owed, so what bounds cancelling is the period between passes and not how far one reaches.
+     *
+     * Two things are not evidence of a cancellation. A batch filled to its limit is a read that
+     * was cut short, and says nothing about what it did not reach. And a call already on its way
+     * is settled by its own path rather than by what a scan reads.
+     */
+    private forget;
     /**
      * A row already armed is left alone: a scan reads what it read before, because a row stays
      * until its call has been made.