@memberjunction/scheduling-engine 5.38.0 → 5.40.0

package/README.md

@@ -47,9 +47,10 @@ graph TD
 
 - **Cron Evaluation**: Parses cron expressions to determine which jobs are due for execution
 - **Plugin Architecture**: Each job type has a registered `BaseScheduledJob` driver class
-- **Distributed Locking**: Token-based locking for multi-server environments with stale lock detection
+- **Distributed Locking**: Token-based locking for multi-server environments with stale lock detection. Atomic compare-and-swap via dedicated stored procedures (`spAcquireScheduledJobLock`, `spReleaseScheduledJobLockIfTokenMatches`).
 - **Concurrency Modes**: `Concurrent`, `Queue`, or `Skip` for overlapping executions
-- **Polling Lifecycle**: Adaptive polling interval with `StartPolling()` / `StopPolling()`
+- **Bounded Concurrency**: `MaxConcurrentJobs` (default 5) caps simultaneous in-flight jobs across the engine; configurable via `scheduledJobs.maxConcurrentJobs`
+- **Polling Lifecycle**: Adaptive polling interval with `StartPolling()` / `StopPolling()`. As of v5.39 the poll loop dispatches jobs without awaiting completion — a single hung job no longer stalls the scheduler. See [Operational notes](#operational-notes-v539) below.
 - **Notifications**: Configurable notifications on success, failure, or both
 - **Statistics Tracking**: Automatic update of RunCount, SuccessCount, FailureCount, and timing metrics
 - **Built-in Drivers**: `AgentScheduledJobDriver` for AI agents, `ActionScheduledJobDriver` for MJ Actions
@@ -70,11 +71,12 @@ import { SchedulingEngine } from '@memberjunction/scheduling-engine';
 const engine = SchedulingEngine.Instance;
 await engine.Config(false, contextUser);
 
-// Start continuous polling
+// Start continuous polling (async since v5.39 — must be awaited)
 await engine.StartPolling(contextUser);
 
-// Stop polling (e.g., on server shutdown)
-engine.StopPolling();
+// Stop polling on server shutdown. Graceful drain waits for currently-dispatched
+// jobs to settle, bounded by maxWaitMs so a hung job can't hang shutdown.
+await engine.StopPolling({ waitForInflight: true, maxWaitMs: 30_000 });
 ```
 
 ### Manual Job Execution
@@ -136,10 +138,96 @@ Abstract base class for all job drivers. Provides:
 
 Utility for parsing and evaluating cron expressions using `cron-parser`.
 
+### `RunImmediatelyIfNeverRun` Flag
+
+Each `MJ: Scheduled Jobs` row now has a `RunImmediatelyIfNeverRun` boolean (v5.38). When `true` AND `LastRunAt IS NULL`, `SchedulingEngine.initializeNextRunTimes()` sets `NextRunAt = now()` instead of the next cron tick, so a freshly-seeded job runs on the next polling cycle rather than waiting up to a full cron interval (e.g., 24h for a daily job) for its first run.
+
+Use this for seed metadata that should run as soon as it's installed (data backfills, initial syncs like Entity Vector Sync, etc.). The flag is a no-op once the job has run at least once — subsequent restarts follow the cron schedule normally.
+
 ### NotificationManager
 
 Manages notification delivery based on job configuration (on success, failure, or both).
 
+## Operational notes (v5.39+)
+
+The v5.39 release decoupled the poll loop from job execution to fix a bug where a single hung `plugin.Execute()` would stall the entire scheduler until process restart (GH #2736). The design doc for the full rationale is at [`plans/scheduled-job-engine-decoupling.md`](../../../plans/scheduled-job-engine-decoupling.md). The summary below covers what operators and integrators need to know.
+
+### Polling architecture
+
+The poll loop calls `DispatchScheduledJobs(contextUser)`, which:
+
+1. **Sweeps** stale in-flight entries (jobs whose DB lease has expired) — untracks them so their concurrency slot is freed, and fire-and-forget abandons their orphaned `Status='Running'` run records. The sweep is decoupled from `isJobDue` and from the cap; it runs unconditionally at the top of every poll.
+2. **Dispatches** each due job under a cap (`MaxConcurrentJobs`, default 5), atomically acquiring its lock via `spAcquireScheduledJobLock`. Dispatched jobs run in the background; the poll loop returns immediately so the next poll fires on schedule.
+
+Jobs that legitimately fail (throw) still recover via the existing try/catch in `executeJobWithLock`. The lock release uses `spReleaseScheduledJobLockIfTokenMatches` — atomic and token-checked, so a stale holder's late settlement after lease expiry can't clobber a fresh holder's lock.
+
+### Cap and lease semantics
+
+**`MaxConcurrentJobs` is a SOFT cap.** Under concurrent poll bodies overlapping, the cap may be transiently exceeded by a small amount bounded by the overlap count (typically 1–2). This is acceptable because:
+
+- The cap exists to bound order-of-magnitude fan-out, not to enforce an exact integer count.
+- The atomic lock sproc guarantees no same-job double-dispatch even under overshoot.
+- Steady-state behavior holds at the configured cap.
+
+If you need a hard ceiling, set `MaxConcurrentJobs` slightly below your true limit.
+
+**Tuning guidance:**
+
+- **Raise the cap** when scheduled work backs up despite plenty of headroom — many short, lightweight jobs.
+- **Lower the cap** on memory / DB connection / rate-limit pressure, or when each job spawns heavy work (agent runs, LLM calls).
+- **Shorten the lease** (`LeaseTimeoutMs` / `LeaseTimeoutMinutes`) to reduce the worst-case starvation window (see below), but never below the maximum runtime of any legitimate job — premature reclaim would re-dispatch healthy long jobs.
+
+Configure via MJServer's `scheduledJobs` config block:
+
+```javascript
+// mj.config.cjs
+module.exports = {
+    scheduledJobs: {
+        enabled: true,
+        systemUserEmail: 'system@example.com',
+        maxConcurrentJobs: 10,           // default 5
+        defaultLockTimeout: 5 * 60_000,  // default 600000 (10 min)
+    },
+    // ...
+};
+```
+
+`MJServer/services/ScheduledJobsService` reads these on startup and applies them to the engine via `MaxConcurrentJobs` and `LeaseTimeoutMs` setters.
+
+### Residual starvation window
+
+This fix eliminates the unbounded scheduler stall but does NOT eliminate all starvation scenarios. If **`≥ MaxConcurrentJobs` jobs hang simultaneously** within a single lease window, other scheduled work is starved until at least one lease expires and the sweep reclaims its slot.
+
+With defaults (`MaxConcurrentJobs=5`, lease=10 min), the worst-case starvation is **~10 minutes** — dramatically improved from the original "stalled until process restart," but not zero. Mitigations:
+
+- Raise `MaxConcurrentJobs` so healthy jobs have headroom while zombies hold slots.
+- Shorten `LeaseTimeoutMs` to accelerate reclaim (subject to the constraint above).
+- Bound `plugin.Execute()` in your job plugins — the structural cure, tracked as a follow-up.
+
+### Leaked promise behavior
+
+When the sweep reclaims a hung job's lease:
+
+1. The hung `executeJobWithLock` Promise stays in memory. JavaScript doesn't support Promise cancellation; the executing async function still holds the promise alive via its suspended `await`.
+2. Each leaked promise holds: the plugin instance, execution context, run entity, and any closures the plugin captured (typically single-digit MB).
+3. Bounded by the count of distinct hanging jobs. Process restart on deploy clears them.
+4. **Operational signal**: each sweep emits a `[sweep] Untracking inflight job <name>` log line. A spike in these is a canary for a misbehaving plugin — wire to monitoring if hangs are a concern.
+5. **Orphaned run records are NOT leaked.** The sweep marks them `Status='Failed'` with an explanatory message via `abandonOrphanedRunRecords`. The DB litter pattern (phantom `Running` rows accumulating over months) is prevented going forward.
+
+`Promise.race([plugin.Execute(), timeout])` does NOT solve the leak — `Promise.race` returns when the timeout wins, but the underlying `plugin.Execute()` promise is still alive. Same heap leak, different surface. Worker-thread isolation would let us terminate a hung plugin's execution context, but that's a multi-week design effort tracked separately.
+
+### Job-list refresh
+
+The engine loads the job list ONCE at `StartPolling` time. Runtime changes to the `MJ: Scheduled Jobs` table (adds, deletions, cron edits) are NOT picked up automatically by the engine. Callers must explicitly invoke `await engine.OnJobChanged(contextUser)` after making such changes. This is pre-existing behavior — not changed by v5.39 — but worth calling out because the per-poll `Config(false)` no-op was moved out of the poll path, making the explicit `OnJobChanged` trigger the only refresh mechanism.
+
+### Graceful shutdown
+
+`StopPolling({ waitForInflight, maxWaitMs })` lets shutdown wait for dispatched jobs to settle:
+
+- `waitForInflight: true` — awaits `Promise.allSettled` on all tracked promises.
+- `maxWaitMs` — bounds the wait so a zombie can't block shutdown indefinitely (recommended in production; `ScheduledJobsService.Stop()` uses 30s by default).
+- Default (no opts) — preserves the prior fire-and-forget behavior; tasks may continue running in the background while the process exits.
+
 ## Dependencies
 
 | Package | Purpose |

package/dist/ScheduledJobEngine.d.ts

@@ -44,6 +44,49 @@ export declare class SchedulingEngine extends BaseSingleton<SchedulingEngine> {
     private highFrequencyWarnedJobIds;
     /** Threshold (ms) below which a job's run frequency triggers a console warning. */
     private static readonly HIGH_FREQUENCY_WARNING_THRESHOLD_MS;
+    /**
+     * Maximum concurrent scheduled jobs on this engine instance. Default 5.
+     * Configurable via MJServer's `scheduledJobs.maxConcurrentJobs` config.
+     *
+     * SOFT CAP: under overlapping poll bodies the cap may be transiently
+     * exceeded by a small amount bounded by overlap count. See README
+     * "Cap and lease semantics" for tuning guidance.
+     */
+    private _maxConcurrentJobs;
+    /**
+     * Lock lease duration. Default 10 minutes. Configurable via
+     * mj.config.cjs `scheduling.leaseTimeoutMinutes`.
+     *
+     * Public API uses minutes (operational readability); internal computations
+     * use _leaseTimeoutMs for precision and testability. Tests inject sub-second
+     * leases via `_setLeaseTimeoutMsForTest`.
+     *
+     * Constraint: must be > maximum expected runtime of any job. Setting too
+     * low causes healthy long jobs to be reclaimed and re-dispatched.
+     */
+    private _leaseTimeoutMs;
+    /**
+     * Promises for jobs currently dispatched but not yet settled. Keyed by job ID.
+     *
+     * Three purposes:
+     *   1. Bounded concurrency — DispatchScheduledJobs checks size vs MaxConcurrentJobs.
+     *   2. Sweep untracking — sweepStaleInflightJobs deletes by ID for jobs whose
+     *      lease has expired, freeing the cap slot even though the JS promise leaks
+     *      (see README "Leaked promise behavior").
+     *   3. Graceful shutdown — StopPolling can await all in-flight via .values().
+     *
+     * Self-cleans via identity-checked .finally() on each dispatched promise
+     * (no-op if a sweep + re-dispatch already replaced the entry).
+     *
+     * NOT used for double-dispatch prevention — that's the atomic lock sproc's job.
+     */
+    private inflightJobPromises;
+    /**
+     * When false, DispatchScheduledJobs becomes a no-op. Set false in StopPolling
+     * BEFORE snapshotting inflightJobPromises for shutdown drain, so no new
+     * entries sneak in during the shutdown window.
+     */
+    private acceptingDispatches;
     /** Gets all scheduled job types. */
     get ScheduledJobTypes(): MJScheduledJobTypeEntity[];
     /** Gets scheduled jobs (active only by default). */
@@ -52,6 +95,27 @@ export declare class SchedulingEngine extends BaseSingleton<SchedulingEngine> {
     get ScheduledJobRuns(): MJScheduledJobRunEntity[];
     /** Gets the current active polling interval in milliseconds. */
     get ActivePollingInterval(): number | null;
+    /**
+     * Maximum concurrent scheduled jobs on this engine instance. Default 5.
+     * Configurable via MJServer's `scheduledJobs.maxConcurrentJobs` config.
+     */
+    get MaxConcurrentJobs(): number;
+    set MaxConcurrentJobs(value: number);
+    /**
+     * Lock lease duration in milliseconds. Default 600000 (10 minutes).
+     * Production callers should use this setter — matches the ms unit of
+     * MJServer's `scheduledJobs.defaultLockTimeout` config.
+     */
+    get LeaseTimeoutMs(): number;
+    set LeaseTimeoutMs(value: number);
+    /**
+     * Convenience accessor — lease duration as integer minutes. Production
+     * code may use either this or `LeaseTimeoutMs`. The setter validates
+     * positive integer minutes (no fractional minutes via this path; use
+     * `LeaseTimeoutMs` for sub-minute precision, including tests).
+     */
+    get LeaseTimeoutMinutes(): number;
+    set LeaseTimeoutMinutes(value: number);
     /** Find a job type by name. */
     GetJobTypeByName(name: string): MJScheduledJobTypeEntity | undefined;
     /** Find a job type by driver class. */
@@ -68,16 +132,37 @@ export declare class SchedulingEngine extends BaseSingleton<SchedulingEngine> {
      */
     Config(forceRefresh?: boolean, contextUser?: UserInfo, provider?: IMetadataProvider, includeRuns?: boolean, includeAllJobs?: boolean): Promise<boolean>;
     /**
-     * Start continuous polling for scheduled jobs
-     * Uses adaptive interval based on ActivePollingInterval
+     * Start continuous polling for scheduled jobs.
+     *
+     * Async (changed in v5.39) because upfront work — Config, initial-NextRunAt
+     * seeding, stale-lock cleanup, permission probe — runs ONCE before the
+     * first poll fires. Subsequent polls assume that work is complete.
+     *
+     * The poll callback re-arms its timer FIRST, before any awaited work, so
+     * that any hang downstream (Config, DispatchScheduledJobs, etc.) cannot
+     * prevent the next poll from firing on schedule. This is the load-bearing
+     * invariant of the decoupling fix (see plans/scheduled-job-engine-decoupling.md).
      *
      * @param contextUser - User context for execution
      */
-    StartPolling(contextUser: UserInfo): void;
+    StartPolling(contextUser: UserInfo): Promise<void>;
     /**
-     * Stop continuous polling
+     * Stop continuous polling.
+     *
+     * Async (changed in v5.39). With opts.waitForInflight=true, awaits all
+     * currently-dispatched jobs to settle before returning. With opts.maxWaitMs,
+     * bounds that wait so a zombie can't make shutdown hang indefinitely.
+     *
+     * Order matters: sets acceptingDispatches=false FIRST so no new entries
+     * can be added to inflightJobPromises during the snapshot for allSettled.
+     *
+     * @param opts.waitForInflight - Await dispatched jobs before returning
+     * @param opts.maxWaitMs - Bound the wait (only meaningful with waitForInflight)
      */
-    StopPolling(): void;
+    StopPolling(opts?: {
+        waitForInflight?: boolean;
+        maxWaitMs?: number;
+    }): Promise<void>;
     /**
      * Check if polling is currently active
      */
@@ -125,6 +210,43 @@ export declare class SchedulingEngine extends BaseSingleton<SchedulingEngine> {
      * @returns The scheduled job run record
      */
     ExecuteScheduledJob(jobId: string, contextUser: UserInfo): Promise<MJScheduledJobRunEntity>;
+    /**
+     * Dispatch all currently-due scheduled jobs WITHOUT awaiting their completion.
+     *
+     * This is the polling-path entry point introduced in v5.39 as part of the
+     * scheduler decoupling fix (GH #2736). The poll loop calls this and re-arms
+     * its timer based on the synchronous-portion return; jobs run in the background.
+     *
+     * Two phases:
+     *
+     *   PHASE 1 — Stale-inflight sweep (decoupled from isJobDue AND from the cap):
+     *     Walks inflightJobPromises looking for jobs whose DB lease has expired.
+     *     Untracks each, frees its cap slot, and fire-and-forget marks any
+     *     orphaned `Status='Running'` run records as abandoned. Runs first so
+     *     it can free slots BEFORE the cap check throttles dispatch.
+     *
+     *   PHASE 2 — Cap-bounded dispatch loop:
+     *     For each due job, atomically acquire its lock via spAcquireScheduledJobLock.
+     *     Only jobs whose lock was acquired count against MaxConcurrentJobs.
+     *     Lock-failed jobs are reported via `lockedOut` counter.
+     *     If at-cap, remaining due jobs counted via `skippedAtCapacity` and
+     *     picked up by subsequent polls as slots free (no in-memory queueing).
+     *
+     * Same-instance double-dispatch is structurally prevented by the atomic
+     * lock sproc — its WHERE clause filters held-and-not-stale locks, so any
+     * second attempt against the same job ID returns Acquired=0.
+     *
+     * In-flight dispatched promises are tracked in `inflightJobPromises` so
+     * `StopPolling({ waitForInflight: true })` can perform graceful shutdown.
+     *
+     * @returns Counters for observability.
+     */
+    DispatchScheduledJobs(contextUser: UserInfo, evalTime?: Date): Promise<{
+        swept: number;
+        dispatched: number;
+        lockedOut: number;
+        skippedAtCapacity: number;
+    }>;
     /**
      * Determine if a job is currently due for execution
      *
@@ -137,12 +259,28 @@ export declare class SchedulingEngine extends BaseSingleton<SchedulingEngine> {
     /**
      * Execute a single scheduled job
      *
-     * @param job - The job to execute
-     * @param contextUser - User context
-     * @returns The created job run record
+     * Execute a single scheduled job WITH a pre-acquired lock token.
+     *
+     * Caller (DispatchScheduledJobs / ExecuteScheduledJob / ExecuteScheduledJobs)
+     * is responsible for acquiring the lock atomically via tryAcquireLock and
+     * passing the resulting token. This method owns the lock's lifecycle from
+     * this point forward: every exit path (success, failure, exception)
+     * releases the lock via releaseLockIfTokenMatches.
+     *
+     * If lockToken is null, the job is running in `ConcurrencyMode='Concurrent'`
+     * (no lock acquired) — finally simply skips the release.
+     *
+     * @param job - The job entity. READ-ONLY from this method's perspective —
+     *              do not mutate or call Save on it. The shared entity in
+     *              this.ScheduledJobs must not be touched here.
+     * @param lockToken - The token returned by tryAcquireLock, or null for
+     *                    Concurrent mode where no lock was acquired.
+     * @param contextUser - User context for execution.
+     * @returns The created run record (Completed or Failed), or null if a
+     *          synchronous setup error prevented run creation.
      * @private
      */
-    private executeJob;
+    private executeJobWithLock;
     /**
      * Create a new ScheduledJobRun record
      * @private
@@ -158,36 +296,109 @@ export declare class SchedulingEngine extends BaseSingleton<SchedulingEngine> {
      * @private
      */
     private sendNotificationsIfNeeded;
+    /**
+     * Build a dialect-appropriate call to one of the scheduling lock sprocs,
+     * preserving the MJ positional-parameter-array convention.
+     *
+     * SQL Server: `EXEC [schema].[name] @JobID=@p0, @Token=@p1, …` — the sproc's
+     *   own parameters are bound by name to the positional `@pN` placeholders the
+     *   provider substitutes by index.
+     * PostgreSQL: `SELECT * FROM schema."name"($1, $2, …)` — the lock sprocs are
+     *   ported to plpgsql functions (see the v5.40.x scheduling-engine PG
+     *   migration) and `pg` binds `$N` placeholders positionally.
+     *
+     * In BOTH cases the SAME positional value array is handed to ExecuteSQL;
+     * only the call wrapper and placeholder syntax differ. The EXEC-vs-SELECT
+     * wrapper lives in `provider.Dialect.ProcedureCallSyntax`, so it is never
+     * duplicated here — this helper only supplies the per-dialect placeholders.
+     *
+     * @param sprocParamNames the sproc's parameter names IN DECLARED ORDER; their
+     *   positions must line up with the value array passed to ExecuteSQL.
+     * @private
+     */
+    private buildLockSprocCall;
     /**
      * Try to acquire a lock for job execution
      * @private
      */
+    /**
+     * Atomically acquire a lock on a job via spAcquireScheduledJobLock.
+     * The sproc's WHERE clause handles both the free-lock and stale-lease cases
+     * in a single statement — no TOCTOU window between check and write.
+     *
+     * Operates only on lock columns; never mutates the shared entity in
+     * this.ScheduledJobs. Caller passes jobId (string), not the entity object.
+     *
+     * @returns { acquired: true, token } on success; { acquired: false } otherwise.
+     * @private
+     */
     private tryAcquireLock;
     /**
-     * Attempt to acquire a fresh lock on a job that is currently unlocked.
-     * Separated from tryAcquireLock to keep the stale-cleanup and acquisition
-     * paths distinct and easier to follow.
+     * Atomically release a lock IF AND ONLY IF the current DB token matches
+     * expectedToken. Prevents the lost-mutex hazard under lease-expiry races:
+     * if a stale holder's execution eventually settles after the lease was
+     * reclaimed by a fresh holder, this no-ops (token mismatch).
+     *
+     * Idempotent — safe to call on an already-released lock (returns false).
+     *
+     * @returns true if released, false if token mismatch / already released
      * @private
      */
-    private attemptLockAcquisition;
+    private releaseLockIfTokenMatches;
     /**
-     * Clear in-memory lock fields without saving — used when a save attempt
-     * fails and we need the in-memory object to reflect "unlocked" so the
-     * next poll cycle doesn't see a phantom lock.
+     * Pre-flight: verify EXECUTE permission on lock sprocs. Fails LOUDLY at boot
+     * if the engine's DB principal lacks grants — much better than a silent
+     * runtime failure the next time a job tries to dispatch.
+     *
+     * Wrapped in try/catch: probe failure (e.g., non-SQL-Server provider where
+     * `sys.fn_my_permissions` doesn't exist) must NOT crash boot. We log and
+     * continue; any actual permission issue will surface at first sproc call.
+     *
      * @private
      */
-    private clearInMemoryLockFields;
+    private probeLockSprocPermissions;
     /**
-     * Release a lock after job execution
+     * Sweep stale inflight jobs. Runs unconditionally at top of every poll.
+     *
+     * SINGLE BATCH QUERY (not N round-trips). Returns only jobs whose lease has
+     * expired OR whose lock has already been cleared. In steady-state (no zombies)
+     * the query matches zero rows and the sweep is essentially free.
+     *
+     * For each stale entry:
+     *   - Untrack the leaked promise from inflightJobPromises (frees cap slot).
+     *   - FIRE-AND-FORGET abandon any orphaned `Status='Running'` run records.
+     *     NOT awaited because cleanup must not delay dispatch under a fleet-wide
+     *     hang event where the sweep finds many zombies at once.
+     *
+     * Decoupled from:
+     *   - isJobDue — irrelevant; we care about lease state, not cron.
+     *   - MaxConcurrentJobs — the sweep IS what frees the cap when saturated by hangs.
+     *
+     * See plans/scheduled-job-engine-decoupling.md for the full rationale.
+     *
+     * @returns count of inflight entries swept
      * @private
      */
-    private releaseLock;
+    private sweepStaleInflightJobs;
     /**
-     * Clean up a stale lock. Returns true if the lock was successfully
-     * cleared in the database, false if the save failed.
+     * Mark any Running run records for the given job as Failed/abandoned.
+     *
+     * IMPORTANT: the `Status='Running'` filter is LOAD-BEARING — not just for
+     * finding zombies. It also protects against a sweep/release race:
+     *
+     *   - Job completes normally.
+     *   - executeJobWithLock's finally calls releaseLockIfTokenMatches (clears LockToken).
+     *   - BEFORE that completes, a poll's sweep query sees LockToken IS NULL
+     *     and classifies the just-completed job as a zombie.
+     *   - But its run record is already Status='Completed' (set inside the try block,
+     *     before the finally), so THIS FILTER excludes it from abandonment.
+     *
+     * Removing or relaxing this filter would corrupt completed run records.
+     * If "optimizing" this method, preserve the Status='Running' filter.
+     *
      * @private
      */
-    private cleanupStaleLock;
+    private abandonOrphanedRunRecords;
     /**
      * Create a queued job run for later execution
      * @private
@@ -199,17 +410,31 @@ export declare class SchedulingEngine extends BaseSingleton<SchedulingEngine> {
      */
     private getInstanceIdentifier;
     /**
-     * Generate a GUID for lock tokens
-     * @private
-     */
-    private generateGuid;
-    /**
-     * Initialize NextRunAt for jobs that don't have it set
+     * Initialize NextRunAt for jobs that don't have it set.
+     *
+     * If a job has `RunImmediatelyIfNeverRun = true` AND has never run
+     * (`LastRunAt IS NULL`), `NextRunAt` is set to `now()` so the job
+     * executes on the next polling cycle instead of waiting for the next
+     * cron tick. Useful for freshly-seeded jobs that should not wait up
+     * to a full cron interval (e.g. 24h for a daily job) for their first run.
+     *
      * @private
      */
     private initializeNextRunTimes;
     /**
-     * Clean up stale locks on startup
+     * Clean up stale locks on startup using atomic sprocs.
+     *
+     * For each job whose DB shows a stale lock (ExpectedCompletionAt < now OR
+     * ExpectedCompletionAt IS NULL while LockToken IS NOT NULL):
+     *   1. Atomically acquire the stale lock with a fresh token (sproc's WHERE
+     *      handles the stale-detection in a single statement).
+     *   2. Immediately release it with that same token.
+     *
+     * Net effect: stale lock cleared atomically with zero TOCTOU window. Uses
+     * the new sproc-backed pattern instead of load-compare-save on shared
+     * this.ScheduledJobs entities (see plans/scheduled-job-engine-decoupling.md
+     * for why the old pattern was unsafe once polling became concurrent).
+     *
      * @private
      */
     private cleanupStaleLocks;

package/dist/ScheduledJobEngine.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"ScheduledJobEngine.d.ts","sourceRoot":"","sources":["../src/ScheduledJobEngine.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAGH,OAAO,EACH,QAAQ,EAER,iBAAiB,EAIpB,MAAM,sBAAsB,CAAC;AAC9B,OAAO,EAAE,oBAAoB,EAAE,uBAAuB,EAAE,wBAAwB,EAAE,MAAM,+BAA+B,CAAC;AACxH,OAAO,EAAE,aAAa,EAAwB,MAAM,wBAAwB,CAAC;AAE7E,OAAO,EAAE,oBAAoB,EAAE,MAAM,wCAAwC,CAAC;AAK9E;;;;;;;;;;;;;;;;;;;;;GAqBG;AACH,qBAAa,gBAAiB,SAAQ,aAAa,CAAC,gBAAgB,CAAC;IACjE;;OAEG;IACH,WAAkB,QAAQ,IAAI,gBAAgB,CAE7C;IAED;;OAEG;IACH,SAAS,KAAK,IAAI,IAAI,oBAAoB,CAEzC;IAED,OAAO,CAAC,YAAY,CAAC,CAAiB;IACtC,OAAO,CAAC,SAAS,CAAkB;IACnC,OAAO,CAAC,cAAc,CAAkB;IAExC,4EAA4E;IAC5E,OAAO,CAAC,yBAAyB,CAA0B;IAE3D,mFAAmF;IACnF,OAAO,CAAC,MAAM,CAAC,QAAQ,CAAC,mCAAmC,CAAiB;IAM5E,oCAAoC;IACpC,IAAW,iBAAiB,IAAI,wBAAwB,EAAE,CAEzD;IAED,oDAAoD;IACpD,IAAW,aAAa,IAAI,oBAAoB,EAAE,CAEjD;IAED,sCAAsC;IACtC,IAAW,gBAAgB,IAAI,uBAAuB,EAAE,CAEvD;IAED,gEAAgE;IAChE,IAAW,qBAAqB,IAAI,MAAM,GAAG,IAAI,CAEhD;IAED,+BAA+B;IACxB,gBAAgB,CAAC,IAAI,EAAE,MAAM,GAAG,wBAAwB,GAAG,SAAS;IAI3E,uCAAuC;IAChC,uBAAuB,CAAC,WAAW,EAAE,MAAM,GAAG,wBAAwB,GAAG,SAAS;IAIzF,uCAAuC;IAChC,aAAa,CAAC,SAAS,EAAE,MAAM,GAAG,oBAAoB,EAAE;IAI/D,mCAAmC;IAC5B,aAAa,CAAC,KAAK,EAAE,MAAM,GAAG,uBAAuB,EAAE;IAI9D,wDAAwD;IACjD,qBAAqB,IAAI,IAAI;IAQpC;;;OAGG;IACU,MAAM,CACf,YAAY,CAAC,EAAE,OAAO,EACtB,WAAW,CAAC,EAAE,QAAQ,EACtB,QAAQ,CAAC,EAAE,iBAAiB,EAC5B,WAAW,GAAE,OAAe,EAC5B,cAAc,GAAE,OAAe,GAChC,OAAO,CAAC,OAAO,CAAC;IAQnB;;;;;OAKG;IACI,YAAY,CAAC,WAAW,EAAE,QAAQ,GAAG,IAAI;IA6DhD;;OAEG;IACI,WAAW,IAAI,IAAI;IAa1B;;OAEG;IACH,IAAW,SAAS,IAAI,OAAO,CAE9B;IAED;;;;;;;OAOG;IACU,YAAY,CAAC,WAAW,EAAE,QAAQ,GAAG,OAAO,CAAC,IAAI,CAAC;IA0B/D;;;;;;;OAOG;IACH,OAAO,CAAC,0BAA0B;IA4DlC;;;;OAIG;IACH,OAAO,CAAC,cAAc;IAkBtB;;;;;;;;;OASG;IACU,oBAAoB,CAC7B,WAAW,EAAE,QAAQ,EACrB,QAAQ,GAAE,IAAiB,GAC5B,OAAO,CAAC,uBAAuB,EAAE,CAAC;IAkCrC;;;;;;OAMG;IACU,mBAAmB,CAC5B,KAAK,EAAE,MAAM,EACb,WAAW,EAAE,QAAQ,GACtB,OAAO,CAAC,uBAAuB,CAAC;IAWnC;;;;;;;OAOG;IACH,OAAO,CAAC,QAAQ;IAkBhB;;;;;;;OAOG;YACW,UAAU;IAgGxB;;;OAGG;YACW,YAAY;IAoB1B;;;OAGG;YACW,mBAAmB;IAoBjC;;;OAGG;YACW,yBAAyB;IAmCvC;;;OAGG;YACW,cAAc;IAiC5B;;;;;OAKG;YACW,sBAAsB;IAsCpC;;;;;OAKG;IACH,OAAO,CAAC,uBAAuB;IAO/B;;;OAGG;YACW,WAAW;IAiBzB;;;;OAIG;YACW,gBAAgB;IAK9B;;;OAGG;YACW,kBAAkB;IAsBhC;;;OAGG;IACH,OAAO,CAAC,qBAAqB;IAI7B;;;OAGG;IACH,OAAO,CAAC,YAAY;IAQpB;;;OAGG;YACW,sBAAsB;IAcpC;;;OAGG;YACW,iBAAiB;IAkD/B,OAAO,CAAC,GAAG;IAQX,OAAO,CAAC,QAAQ;CAGnB"}
+{"version":3,"file":"ScheduledJobEngine.d.ts","sourceRoot":"","sources":["../src/ScheduledJobEngine.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAIH,OAAO,EACH,QAAQ,EAER,iBAAiB,EAOpB,MAAM,sBAAsB,CAAC;AAC9B,OAAO,EAAE,oBAAoB,EAAE,uBAAuB,EAAE,wBAAwB,EAAE,MAAM,+BAA+B,CAAC;AACxH,OAAO,EAAE,aAAa,EAAwB,MAAM,wBAAwB,CAAC;AAE7E,OAAO,EAAE,oBAAoB,EAAE,MAAM,wCAAwC,CAAC;AAK9E;;;;;;;;;;;;;;;;;;;;;GAqBG;AACH,qBAAa,gBAAiB,SAAQ,aAAa,CAAC,gBAAgB,CAAC;IACjE;;OAEG;IACH,WAAkB,QAAQ,IAAI,gBAAgB,CAE7C;IAED;;OAEG;IACH,SAAS,KAAK,IAAI,IAAI,oBAAoB,CAEzC;IAED,OAAO,CAAC,YAAY,CAAC,CAAiB;IACtC,OAAO,CAAC,SAAS,CAAkB;IACnC,OAAO,CAAC,cAAc,CAAkB;IAExC,4EAA4E;IAC5E,OAAO,CAAC,yBAAyB,CAA0B;IAE3D,mFAAmF;IACnF,OAAO,CAAC,MAAM,CAAC,QAAQ,CAAC,mCAAmC,CAAiB;IAM5E;;;;;;;OAOG;IACH,OAAO,CAAC,kBAAkB,CAAa;IAEvC;;;;;;;;;;OAUG;IACH,OAAO,CAAC,eAAe,CAA0B;IAEjD;;;;;;;;;;;;;;OAcG;IACH,OAAO,CAAC,mBAAmB,CAAmE;IAE9F;;;;OAIG;IACH,OAAO,CAAC,mBAAmB,CAAiB;IAM5C,oCAAoC;IACpC,IAAW,iBAAiB,IAAI,wBAAwB,EAAE,CAEzD;IAED,oDAAoD;IACpD,IAAW,aAAa,IAAI,oBAAoB,EAAE,CAEjD;IAED,sCAAsC;IACtC,IAAW,gBAAgB,IAAI,uBAAuB,EAAE,CAEvD;IAED,gEAAgE;IAChE,IAAW,qBAAqB,IAAI,MAAM,GAAG,IAAI,CAEhD;IAED;;;OAGG;IACH,IAAW,iBAAiB,IAAI,MAAM,CAErC;IACD,IAAW,iBAAiB,CAAC,KAAK,EAAE,MAAM,EAOzC;IAED;;;;OAIG;IACH,IAAW,cAAc,IAAI,MAAM,CAElC;IACD,IAAW,cAAc,CAAC,KAAK,EAAE,MAAM,EAOtC;IAED;;;;;OAKG;IACH,IAAW,mBAAmB,IAAI,MAAM,CAEvC;IACD,IAAW,mBAAmB,CAAC,KAAK,EAAE,MAAM,EAK3C;IAED,+BAA+B;IACxB,gBAAgB,CAAC,IAAI,EAAE,MAAM,GAAG,wBAAwB,GAAG,SAAS;IAI3E,uCAAuC;IAChC,uBAAuB,CAAC,WAAW,EAAE,MAAM,GAAG,wBAAwB,GAAG,SAAS;IAIzF,uCAAuC;IAChC,aAAa,CAAC,SAAS,EAAE,MAAM,GAAG,oBAAoB,EAAE;IAI/D,mCAAmC;IAC5B,aAAa,CAAC,KAAK,EAAE,MAAM,GAAG,uBAAuB,EAAE;IAI9D,wDAAwD;IACjD,qBAAqB,IAAI,IAAI;IAQpC;;;OAGG;IACU,MAAM,CACf,YAAY,CAAC,EAAE,OAAO,EACtB,WAAW,CAAC,EAAE,QAAQ,EACtB,QAAQ,CAAC,EAAE,iBAAiB,EAC5B,WAAW,GAAE,OAAe,EAC5B,cAAc,GAAE,OAAe,GAChC,OAAO,CAAC,OAAO,CAAC;IAQnB;;;;;;;;;;;;;OAaG;IACU,YAAY,CAAC,WAAW,EAAE,QAAQ,GAAG,OAAO,CAAC,IAAI,CAAC;IA8D/D;;;;;;;;;;;;OAYG;IACU,WAAW,CAAC,IAAI,CAAC,EAAE;QAAE,eAAe,CAAC,EAAE,OAAO,CAAC;QAAC,SAAS,CAAC,EAAE,MAAM,CAAA;KAAE,GAAG,OAAO,CAAC,IAAI,CAAC;IA4BjG;;OAEG;IACH,IAAW,SAAS,IAAI,OAAO,CAE9B;IAED;;;;;;;OAOG;IACU,YAAY,CAAC,WAAW,EAAE,QAAQ,GAAG,OAAO,CAAC,IAAI,CAAC;IA0B/D;;;;;;;OAOG;IACH,OAAO,CAAC,0BAA0B;IA4DlC;;;;OAIG;IACH,OAAO,CAAC,cAAc;IAkBtB;;;;;;;;;OASG;IACU,oBAAoB,CAC7B,WAAW,EAAE,QAAQ,EACrB,QAAQ,GAAE,IAAiB,GAC5B,OAAO,CAAC,uBAAuB,EAAE,CAAC;IA8CrC;;;;;;OAMG;IACU,mBAAmB,CAC5B,KAAK,EAAE,MAAM,EACb,WAAW,EAAE,QAAQ,GACtB,OAAO,CAAC,uBAAuB,CAAC;IAmBnC;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;OA8BG;IACU,qBAAqB,CAC9B,WAAW,EAAE,QAAQ,EACrB,QAAQ,GAAE,IAAiB,GAC5B,OAAO,CAAC;QAAE,KAAK,EAAE,MAAM,CAAC;QAAC,UAAU,EAAE,MAAM,CAAC;QAAC,SAAS,EAAE,MAAM,CAAC;QAAC,iBAAiB,EAAE,MAAM,CAAA;KAAE,CAAC;IA8D/F;;;;;;;OAOG;IACH,OAAO,CAAC,QAAQ;IAkBhB;;;;;;;;;;;;;;;;;;;;;;;OAuBG;YACW,kBAAkB;IAmGhC;;;OAGG;YACW,YAAY;IAoB1B;;;OAGG;YACW,mBAAmB;IAoEjC;;;OAGG;YACW,yBAAyB;IAmCvC;;;;;;;;;;;;;;;;;;;OAmBG;IACH,OAAO,CAAC,kBAAkB;IAW1B;;;OAGG;IACH;;;;;;;;;;OAUG;YACW,cAAc;IA4B5B;;;;;;;;;;OAUG;YACW,yBAAyB;IAsBvC;;;;;;;;;;OAUG;YACW,yBAAyB;IAyCvC;;;;;;;;;;;;;;;;;;;;;OAqBG;YACW,sBAAsB;IAyCpC;;;;;;;;;;;;;;;;;OAiBG;YACW,yBAAyB;IAoCvC;;;OAGG;YACW,kBAAkB;IAsBhC;;;OAGG;IACH,OAAO,CAAC,qBAAqB;IAI7B;;;;;;;;;;OAUG;YACW,sBAAsB;IAyBpC;;;;;;;;;;;;;;;OAeG;YACW,iBAAiB;IA+C/B,OAAO,CAAC,GAAG;IAQX,OAAO,CAAC,QAAQ;CAGnB"}