@n8n/scheduler 1.7.1 → 1.8.1

package/README.md

@@ -14,13 +14,13 @@ restarts, the timers are gone. If you run several copies of the server for high
 availability, every copy fires the same timer and the work runs many times.
 
 This package takes a different approach: **write the work down before running it.**
-Every upcoming run is recorded in the database first, then picked up and fired from
-there. Because the source of truth lives in storage and not in one process's memory:
+Every upcoming run is recorded in shared, durable storage first, then picked up and
+fired from there. Because the source of truth lives in storage and not in one process's memory:
 
 - a restart or crash loses nothing (the run is still recorded and gets picked up
   again),
 - any server in the cluster can pick up any run, and
-- coordination in the database drives each run toward firing on **exactly one**
+- coordination through the shared store drives each run toward firing on **exactly one**
   server, with safety mechanisms (claims, leases, recovery) to fall back on when a
   server crashes mid-run.
 
@@ -44,10 +44,9 @@ From then on it:
 - recovers runs that were stranded by a crashed server,
 - cleans up finished runs after a retention window.
 
-**How it is built.** The package is **pure logic**. It talks to no database and
-uses no dependency injection directly. Instead the host (in n8n, the `cli`
-package) hands it a small set of storage functions and configuration, and the
-scheduler composes its algorithms over them. This keeps every algorithm easy to
+**How it is built.** The package is **pure logic**. It owns no storage and uses no
+dependency injection directly. Instead the host hands it a small set of storage
+functions and configuration, and the scheduler composes its algorithms over them. This keeps every algorithm easy to
 test in isolation with a fake store. The work is split into a few focused
 submodules, described below.
 
@@ -58,8 +57,10 @@ path during rollout.
 ## The moving parts
 
 The scheduler runs four small background routines, each on its own repeating
-timer. Each one is a separate submodule with a single responsibility. Their code
-names are in parentheses; the rest of this document uses the plain names.
+timer, plus an optional fifth when owner reconciliation is configured (see
+[The reconciliation sweep](#the-reconciliation-sweep)). Each one is a separate
+submodule with a single responsibility. Their code names are in parentheses; the
+rest of this document uses the plain names.
 
 ```mermaid
 flowchart TD
@@ -76,6 +77,8 @@ flowchart TD
     T -.->|"finished runs,<br/>past their window"| C["Cleanup<br/><i>(retention)</i>"]
     C -.->|deletes| T
 
+    RC["Reconciliation<br/><i>(owner sweep, optional)</i>"] -.->|"quarantines rules<br/>whose owner is gone"| J
+
     classDef store fill:#2d3748,stroke:#4a5568,color:#fff;
     class J,T store;
 ```
@@ -125,9 +128,9 @@ Successful and failed runs can be kept for different lengths of time.
   is the next run? It handles cron expressions, intervals, one-offs, and the
   "every N periods" variant described below, including timezone and daylight-saving
   behaviour.
-- **Lifecycle** is the set of repeating loops that drive the four passes above,
-  each on its own interval with a little randomness ("jitter") so multiple servers
-  don't all fire their passes in lockstep.
+- **Lifecycle** is the set of repeating loops that drive the four passes above
+  (five with reconciliation), each on its own interval with a little randomness
+  ("jitter") so multiple servers don't all fire their passes in lockstep.
 
 ## Schedule kinds
 
@@ -216,18 +219,152 @@ Schedule and poll triggers both use `skip`: it matches what n8n did before the
 durable scheduler, where a missed run was never replayed. Running late is opt-in
 per job, which is where `coalesce` and `coalesce_owner` come in.
 
+## Owning scheduled jobs
+
+Every scheduled job belongs to something: a published workflow, an instance-level
+maintenance task, an agent. That something is its **owner**, and it is plain data on
+the job row — three fields, none of which this package interprets:
+
+| Field | Meaning |
+|---|---|
+| `ownerType` | what kind of thing owns the job (`workflow`, `system-task`, `agent`, …) |
+| `ownerId` | which one (a workflow id, a task name, an agent id) |
+| `ownerMemberId` | optionally, which part of it (the trigger node for a workflow, the scheduled task for an agent) |
+
+Because ownership is data rather than a schema relationship, any part of the product
+can own scheduled jobs without the scheduler learning what its owners are, and
+without a migration. A job with nothing else to own it is **self-owned**: its
+`ownerId` is its own name. There is no such thing as an ownerless job, so there is
+no second code path for one.
+
+### The two obligations of a new owner type
+
+Ownership is a contract, not just a label. A module that owns scheduled jobs takes on
+**both** of the following. Neither substitutes for the other, and shipping an owner
+type without them leaves schedules running for things that no longer exist.
+
+**1. Deprovision inside your own delete transaction.**
+
+There is no storage-level cascade behind these rows. When your owner goes away,
+*you* delete its jobs, in the same transaction as the write that removes the owner:
+
+```ts
+// Inside the transaction that stops the owner from existing.
+await myOwnerRepository.remove(trx, ownerId);
+await jobProvisioner.deprovisionOwnerInTransaction(trx, { ownerType: 'my-thing', ownerId });
+```
+
+Committing the two together is what makes the guarantee: a crash can never leave a job
+firing for an owner that is already gone. Deprovisioning is idempotent, so a retried
+teardown is safe, and a path that no longer knows which of an owner's parts
+provisioned what can delete the whole owner at once.
+
+Where the teardown genuinely cannot share one transaction, **deprovision first** and
+remove the owner second. A crash between the two then leaves an owner with no
+schedules, which the next attempt fixes, rather than schedules with no owner, which
+only the sweep below would ever find.
+
+n8n's own `workflow` owner type is the worked example of both shapes: a deactivation
+deprovisions inside the transaction that writes `active = false` (`WorkflowService`),
+while an unpublish deprovisions the workflow's jobs immediately before removing the
+`workflow_published_version` mapping that made it an owner
+(`WorkflowPublicationApplier`).
+
+**2. Register a liveness resolver.**
+
+The scheduler cannot tell whether one of your owners still exists, so you answer that
+question for it: register a resolver for your owner type that, given owner ids, says
+which of them are still there.
+
+```ts
+registry.register('my-thing', {
+  async findExisting(ownerIds) {
+    return await myOwners.findExistingIds(ownerIds); // a Set of the ids that remain
+  },
+});
+```
+
+Build the registry in one place, a single manifest the host owns: declare every
+owner module there, and let both readers (the provisioner and the scheduler's
+reconciliation pass) build theirs from it, so neither can see an owner the other
+does not.
+
+Two rules make the answer safe to act on:
+
+- **Absence means gone.** An id you leave out of the result is read as a positive
+  statement that the owner no longer exists. Never omit an id your lookup simply did
+  not cover.
+- **Throw when you cannot tell.** If the lookup fails (a data source is unreachable, a
+  dependency is not ready), throw. "Could not tell" is never treated as "gone", and
+  the sweep leaves every job of your owner type exactly as it was.
+
+Provisioning **refuses** an owner type with no registered resolver, so a new kind of
+owner cannot ship without a cleanup story. The refusal is built into
+`createJobProvisioner` itself: the registry is one of its required dependencies.
+
+### The reconciliation sweep
+
+The resolvers feed a periodic sweep, the safety net behind obligation 1. It exists for
+what synchronous deprovisioning cannot cover: a process killed between two writes, a
+teardown path written before it knew about scheduled jobs, an owner deleted straight
+from storage.
+
+It runs as a fifth lifecycle loop, composed only when the host passes `reconciliation`
+to `createScheduler`, so a deployment without it pays nothing. It follows the same
+jittered cadence as the other passes, every fifteen minutes by default with a five
+minute timeout, and pages through live jobs in bounded batches so one pass never scans
+the whole table.
+
+Because it acts on an answer from code it does not own, it never deletes on first
+sight:
+
+```mermaid
+stateDiagram-v2
+    [*] --> Running: rule provisioned
+    Running --> Quarantined: owner reported gone<br/>(clock stopped, queued runs withdrawn)
+    Quarantined --> Running: owner reported alive again<br/>(clock recomputed)
+    Quarantined --> [*]: still gone past the grace period<br/>(deleted)
+    Running --> Running: owner reported alive
+```
+
+- A job whose owner is reported gone has its **clock stopped and is stamped**, and its
+  already-queued runs are withdrawn, so it stops firing immediately. Its `enabled` flag is
+  left as it was: the sweep never turns a job on or off.
+- It is **deleted only once that stamp is older than the quarantine grace** (a day by
+  default), and only while its owner is still reported gone.
+- A stamped job whose owner turns out to exist after all is **revived** with a freshly
+  computed clock, so a resolver bug corrected inside the grace window destroys nothing. A
+  job that was disabled before the quarantine comes back disabled, with no clock.
+
+A resolver answers whether an owner exists, not what it currently wants, so a revival
+restores the job exactly as it was stored. A job the owner would no longer provision
+(a part of it dropped while its jobs were quarantined) comes back too, and stays
+runnable until the owner provisions or deprovisions again and removes it. Provisioning
+is what reconciles an owner's set of jobs; the sweep only decides whether they may run
+at all.
+
+Two more guardrails: an owner type whose resolver throws, or that nothing claimed, is
+left entirely alone; and jobs younger than a short settle period are skipped, so an
+owner written just after its jobs is never mistaken for one that never existed.
+
+The third layer, behind both of these, is the handler itself: a task whose subject has
+disappeared should no-op and report "not dispatched" rather than fail. n8n's trigger
+handlers do that for a node they can no longer resolve and for an occurrence already
+executed, but not yet for a workflow whose published version is gone: that one fails the
+occurrence, so it retries and dead-letters until the sweep retires the job.
+
 ## Durable and distributed
 
 Two properties describe the whole design, and everything else follows from them.
 
 **Durable** means nothing important lives only in memory. The rules, every upcoming
-run, and each run's current state all live in the database. A process can restart
+run, and each run's current state all live in the shared store. A process can restart
 or crash at any moment and the scheduler simply picks up where the stored state left
 off. Compare an in-memory scheduler, where a restart forgets every pending timer.
 
 **Distributed and leaderless** means there is no elected "scheduler node" that the
 others depend on. Every server runs the same passes over the same shared state, and
-they coordinate purely through the database. There is no leader to elect, fail over,
+they coordinate purely through the shared store. There is no leader to elect, fail over,
 or lose. Adding or removing a server changes throughput, not correctness: any server
 can plan any rule and run any due task.
 
@@ -240,9 +377,9 @@ over never repeating it.** In rare recovery situations that can mean a run execu
 more than once, so handlers should tolerate being called again for the same run.
 The design leans on a few ideas working together.
 
-- **Claim.** A run is only ever picked up through an atomic database claim: a
-  single statement that flips exactly one waiting row to "running" and stamps it
-  with the server that claimed it. Because the database resolves the race, only one
+- **Claim.** A run is only ever picked up through an atomic claim in the shared
+  store: a single statement that flips exactly one waiting row to "running" and stamps it
+  with the server that claimed it. Because the store resolves the race, only one
   server can win a given run, no matter how many try at once.
 - **Lease.** A claim comes with an expiry (a *lease*). While the lease is valid the
   run belongs to that server. If the server dies, the lease lapses and the recovery
@@ -328,8 +465,8 @@ stateDiagram-v2
 ## What this package is (and is not)
 
 It is a **library of scheduling logic**, not a running service on its own. It owns
-the algorithms and the schedule math; it does not own a database connection, a DI
-container, or a configuration source. A **host** provides those and drives it.
+the algorithms and the schedule math; it does not own storage, a DI container, or
+a configuration source. A **host** provides those and drives it.
 
 The public surface is small:
 
@@ -345,7 +482,7 @@ The public surface is small:
 - A few pure helpers are exported too: schedule validation, next-run computation,
   and the provisioning entry points.
 
-In n8n, that host is the `cli` package's durable-scheduler module, which wires up
+In n8n, that host is the server's durable-scheduler module, which wires up
 DI, configuration and instance identity, and routes the scheduler's events to the
 logger. While `N8N_SCHEDULER_ENABLED` is off, the previous in-memory engine stays
 in place; turning the flag off again is the rollback.
@@ -354,7 +491,7 @@ The boundary this section describes is enforced, not just documented.
 `src/__tests__/dependency-purity.test.ts` fails if the package ever gains a
 forbidden dependency, whether a new `package.json` entry outside the allowlist or
 an `import` of something effectful (`@n8n/db`, TypeORM, `n8n-core`, the DI
-container, the `cli` package, or a Node I/O built-in). Its opening doc comment is
+container, the host server package, or a Node I/O built-in). Its opening doc comment is
 the living explanation of the approach: why scheduling is split from execution and
 storage, why each excluded dependency is excluded, and how far the idea could be
 taken. Read it first if you want the reasoning behind the split above.
@@ -394,7 +531,7 @@ A few things that are not obvious from the code but save a lot of confusion.
   event sink at `warn` level, so they land in the host's logs:
   - a **clock-skew warning** at start-up. Due-ness and leases are judged on the
     clock the scheduler coordinates on (the host supplies it via `now`, e.g. the
-    database clock), but fire timers are armed on this instance's own clock. When
+    shared store's clock), but fire timers are armed on this instance's own clock. When
     `start` samples the two and they differ by more than the threshold (default
     `1s`, `CLOCK_SKEW_WARN_THRESHOLD_MS`), it warns: tasks may fire early or late,
     and the instance clock should be synchronised (e.g. via NTP). The sample runs
@@ -412,7 +549,7 @@ A few things that are not obvious from the code but save a lot of confusion.
   to stop already-queued runs, unless the rule-writing path withdraws them (which
   provisioning does).
 
-- **Each background pass runs on its own timer, with jitter.** The four passes are
+- **Each background pass runs on its own timer, with jitter.** The passes are
   independent and slightly randomised, so several servers do not all run their
   passes in lockstep, and one slow pass does not stall the others.
 
@@ -420,10 +557,10 @@ A few things that are not obvious from the code but save a lot of confusion.
 
 - **A dedicated host module.** Today the wiring that turns this pure library into a
   running scheduler (DI, configuration, instance identity, event routing) lives
-  inside the large `cli` package. Extracting it into its own module, for example
-  `@n8n/scheduler-cli` or `@n8n/scheduler-features`, would keep the integration
-  concerns together and out of `cli`, and mirror the clean split this package
-  already draws between algorithm and host.
+  inside n8n's large server package. Extracting it into its own module, for example
+  `@n8n/scheduler-features`, would keep the integration concerns together on their
+  own, and mirror the clean split this package already draws between algorithm and
+  host.
 - **Lease renewal for long handlers.** A claim's lease is fixed for now, so a
   handler that runs longer than its lease risks being recovered and re-run. A
   heartbeat that extends the lease while a handler is genuinely still working would
@@ -433,14 +570,14 @@ A few things that are not obvious from the code but save a lot of confusion.
 
 Pushed all the way, that host module becomes its own deployable worker. On a due
 task it does not run the workflow, it creates an execution and enqueues a job (Bull
-and Redis) for the regular workers, so the real work is decomposing `cli`. The
-result is far leaner than `cli`, which bundles every node's dependencies (1Password,
-S3, and more). The worker only needs:
+and Redis) for the regular workers, so the real work is decomposing the server
+package. The result is far leaner than that package, which bundles every node's
+dependencies (1Password, S3, and more). The worker only needs:
 
 | Needs | Package | Pulls in |
 | --- | --- | --- |
 | Core logic | `@n8n/scheduler` | luxon, cron-parser |
-| Task store + transactions | `@n8n/db` | `@n8n/typeorm` + a database driver |
+| Task store + transactions | `@n8n/db` | `@n8n/typeorm` + its driver |
 | Enqueue executions (a new, extracted publisher) | `@n8n/…-publisher` | Bull + ioredis, execution writes, trigger-context |
 | Logging + lifecycle | `@n8n/backend-common` | winston |
 | Wiring + types | `@n8n/config`, `@n8n/di`, `@n8n/decorators`, `n8n-workflow` | reflect-metadata |