@n8n/scheduler 0.3.1 → 1.1.0

package/README.md

@@ -1,65 +1,413 @@
 # @n8n/scheduler
 
-Durable, multi-main scheduling for n8n. Upcoming work is recorded in storage
-before it runs, then claimed, fired and recovered under leases and epoch
-fencing, so each occurrence executes once across the cluster and survives
-restarts and failovers. Occurrences are dispatched to handlers by `taskType`;
-the Schedule Trigger node is the first consumer. Everything sits behind
-`N8N_SCHEDULER_ENABLED` (default off) while the legacy engine stays in place.
-
-## Surface
-
-The package is pure logic, with no DI and no database. It exposes exactly two
-things (plus the types their signatures reach):
-
-- **`Scheduler`**: `registerTaskHandler` plus one accessor per pass
-  (`materialize`, `execute`, `reap`, `prune`, `stop`). The host drives each
-  pass on its own cadence.
-- **`createScheduler(deps)`**: composes the internal algorithms (materializer,
-  executor, reaper, retention) into a `Scheduler`.
-
-The host binds `deps`: a task store and a transaction runner (the `@n8n/db`
-repositories satisfy the store contracts structurally, so no adapters are
-needed), optional per-pass tuning, and an `onEvent` sink for described
-incidents. In n8n that host is the cli's `DurableScheduler`
-(`packages/cli/src/scheduling/`), which wires DI, config and instance identity,
-and routes events to the logger.
+A durable, cluster-safe scheduler. You give it rules like "run this every Monday
+at 09:00" or "run this once at midnight on New Year's Eve", and it makes sure the
+work actually runs, at the right time, and (as close as a distributed system can
+get) exactly once, even when n8n is running on several servers at once and even
+across restarts and crashes.
+
+## TL;DR
+
+**The principle.** Most schedulers keep their timers in memory: the process holds
+a `setTimeout` for each upcoming run. That is simple but fragile. If the process
+restarts, the timers are gone. If you run several copies of the server for high
+availability, every copy fires the same timer and the work runs many times.
+
+This package takes a different approach: **write the work down before running it.**
+Every upcoming run is recorded in the database first, then picked up and fired from
+there. Because the source of truth lives in storage and not in one process's memory:
+
+- a restart or crash loses nothing (the run is still recorded and gets picked up
+  again),
+- any server in the cluster can pick up any run, and
+- coordination in the database drives each run toward firing on **exactly one**
+  server, with safety mechanisms (claims, leases, recovery) to fall back on when a
+  server crashes mid-run.
+
+**Two things you work with.**
+
+- A **job** is a *rule*: it says *when* something should happen (the cron
+  expression, the interval, the one-off instant). Think of it as a recurring
+  calendar event.
+- A **task** is *one concrete run* produced from that rule, scheduled for one
+  specific instant. Think of it as a single occurrence of that calendar event on a
+  specific date. Tasks are the unit that actually gets claimed, run, retried and
+  cleaned up.
+
+**The functionality surface.** You register a *handler* (a function that does the
+actual work) for a given task type, hand the scheduler your rules, and start it.
+From then on it:
+
+- turns each rule into concrete upcoming runs, ahead of time,
+- claims each due run on one server and calls your handler at the right moment,
+- retries a run that fails, with a growing backoff between attempts,
+- recovers runs that were stranded by a crashed server,
+- cleans up finished runs after a retention window.
+
+**How it is built.** The package is **pure logic**. It talks to no database and
+uses no dependency injection directly. Instead the host (in n8n, the `cli`
+package) hands it a small set of storage functions and configuration, and the
+scheduler composes its algorithms over them. This keeps every algorithm easy to
+test in isolation with a fake store. The work is split into a few focused
+submodules, described below.
+
+Everything sits behind the `N8N_SCHEDULER_ENABLED` flag (off by default). Turning
+it off reverts n8n to its previous in-memory scheduling, which is the rollback
+path during rollout.
+
+## The moving parts
+
+The scheduler runs four small background routines, each on its own repeating
+timer. Each one is a separate submodule with a single responsibility. Their code
+names are in parentheses; the rest of this document uses the plain names.
+
+```mermaid
+flowchart TD
+    P["Provisioning<br/><i>define & update rules</i>"] --> J[("Jobs<br/><i>the rules: when to run</i>")]
+
+    J -->|"claims rules whose<br/>next run is due"| M["Planner<br/><i>(materializer)</i>"]
+    M -->|"records upcoming<br/>runs, ahead of time"| T[("Tasks<br/><i>concrete queued runs</i>")]
+
+    T -->|"claims due runs<br/>for this server"| E["Runner<br/><i>(executor)</i>"]
+    E -->|"fires at the right<br/>instant"| H["Your handler<br/><i>does the actual work</i>"]
+    E -.->|"on failure: retry<br/>later with backoff"| T
+
+    RE["Recovery<br/><i>(reaper)</i>"] -.->|"rescues runs stranded<br/>by a crashed server"| T
+    T -.->|"finished runs,<br/>past their window"| C["Cleanup<br/><i>(retention)</i>"]
+    C -.->|deletes| T
+
+    classDef store fill:#2d3748,stroke:#4a5568,color:#fff;
+    class J,T store;
+```
+
+### Planner (materializer)
+
+Turns rules into concrete runs. On each pass it claims the jobs whose next run is
+due, computes their upcoming occurrences a little way into the future, records
+those as tasks, and advances each job's clock to its next run. Recording runs
+*ahead of time* means a frequent schedule doesn't need one planning pass per fire.
+
+Failures are isolated: if one rule cannot be planned (for example a timezone this
+server's runtime doesn't know about), that single job is set aside and retried
+later, while every other job in the batch still gets planned. One broken rule
+never wedges scheduling for everyone.
+
+### Runner (executor)
+
+Runs the concrete tasks. On each pass it claims the tasks that are due for this
+server and arms a precise in-memory timer for each, so it fires at its exact
+instant. When the timer fires it calls your registered handler for that task type.
+If the handler succeeds the task is marked done; if it throws and attempts remain,
+the task is rescheduled a bit later (backoff); if it throws with no attempts left,
+the task fails for good (a "dead letter").
+
+### Recovery (reaper)
+
+Rescues stranded runs. When a server claims a task it takes a time-limited *lease*
+on it. If that server crashes or stalls, the lease eventually expires and the task
+is left stuck in the "running" state. The recovery pass finds those expired-lease
+tasks and puts them back in line for another attempt (or fails them for good if
+they are out of attempts). This is what makes a mid-run crash safe.
+
+### Cleanup (retention)
+
+Deletes finished tasks once they are older than a retention window, oldest first,
+in small bounded batches so it never holds long locks or monopolises a pass.
+Successful and failed runs can be kept for different lengths of time.
+
+### Supporting submodules
+
+- **Provisioning** keeps the stored set of rules in sync with what a scope
+  (for example a workflow) currently wants: it inserts new rules, rewrites changed
+  ones, and removes rules that are gone. Unchanged rules keep their identity, so
+  re-publishing a workflow never double-fires runs that were already queued.
+- **Recurrence** is the pure date math: given a rule and a starting instant, what
+  is the next run? It handles cron expressions, intervals, one-offs, and the
+  "every N periods" variant described below, including timezone and daylight-saving
+  behaviour.
+- **Lifecycle** is the set of repeating loops that drive the four passes above,
+  each on its own interval with a little randomness ("jitter") so multiple servers
+  don't all fire their passes in lockstep.
 
 ## Schedule kinds
 
-A scheduled job says *when* something should run. There are four kinds:
+A job says *when* to run. There are four kinds:
 
 | Kind | Meaning | Example |
 |---|---|---|
 | `cron` | Fires at fixed clock times, written as a cron expression. | `0 0 9 * * 1` = every Monday at 09:00 |
 | `interval` | Fires every N seconds of elapsed time. | every 3600s = once an hour |
 | `one_off` | Fires once, at a single instant, then never again. | fire at 2026-01-01 00:00 |
-| `recurring_cron` | A cron *anchor* thinned by an "every N periods" gate, for cadences plain cron cannot express. | every 3 weeks on Monday |
+| `recurring_cron` | A cron pattern thinned by an "every N periods" filter, for cadences plain cron cannot express. | every 5 hours; every 3 weeks on Monday |
+
+A `cron` schedule is evaluated as *wall-clock* time in a timezone, so it follows
+daylight-saving shifts (09:00 stays 09:00). An `interval` schedule counts *elapsed*
+time, so daylight saving never moves it.
 
 ### Why `recurring_cron`?
 
-Plain cron can say "every Monday", but not "every *third* Monday".
-`recurring_cron` builds that from two parts:
+A cron expression can only pick times *within* a repeating calendar (which minute,
+hour, day-of-week, and so on). It cannot count across those cycles. So it can say
+"every Monday", but not "every *third* Monday", and it can say "at minute 0 of some
+hours", but not "every 5 hours" starting from a fixed point (the day has 24 hours,
+which 5 does not divide evenly, so no fixed set of hour values repeats cleanly).
 
-- an **anchor**: a normal cron that lists candidate dates (often too many),
-- a **gate**: keeps a candidate only once at least `recurrenceSize` periods of
-  `recurrenceUnit` (`hours` / `days` / `weeks` / `months`) have passed since the
-  last kept fire.
+`recurring_cron` covers exactly those cases by combining two parts:
 
-"Every 3 weeks on Monday" = anchor "every Monday" + gate "1 per 3 weeks":
+- an **anchor**: a normal cron pattern that lists candidate times (often too many),
+- a **filter**: keeps a candidate only once at least N periods (hours, days, weeks
+  or months) have passed since the last kept run.
+
+The most common case is a plain "every N hours". "Every 5 hours" = anchor "every
+hour" + filter "1 per 5 hours":
+
+```
+Anchor (every hour):    H  H  H  H  H  H  H  H  H  H  H
+Filter (1 per 5 hours): ✓  ✗  ✗  ✗  ✗  ✓  ✗  ✗  ✗  ✗  ✓
+Fires:                  H              H              H
+```
+
+The same shape expresses calendar cadences too. "Every 3 weeks on Monday" = anchor
+"every Monday" + filter "1 per 3 weeks":
 
 ```
 Anchor (every Monday):  M   M   M   M   M   M   M
-Gate (1 per 3 weeks):   ✓   ✗   ✗   ✓   ✗   ✗   ✓
+Filter (1 per 3 weeks): ✓   ✗   ✗   ✓   ✗   ✗   ✓
 Fires:                  M           M           M
 ```
 
-The gate looks only at the previous fire, with no stored counter. So any
-instance in the cluster can compute the next run, and a restart never loses the
+The filter looks only at the *previous* run, with no stored counter. So any server
+in the cluster can compute the next run on its own, and a restart never loses the
 cadence.
 
-## Status
+## Durable and distributed
+
+Two properties describe the whole design, and everything else follows from them.
+
+**Durable** means nothing important lives only in memory. The rules, every upcoming
+run, and each run's current state all live in the database. A process can restart
+or crash at any moment and the scheduler simply picks up where the stored state left
+off. Compare an in-memory scheduler, where a restart forgets every pending timer.
+
+**Distributed and leaderless** means there is no elected "scheduler node" that the
+others depend on. Every server runs the same passes over the same shared state, and
+they coordinate purely through the database. There is no leader to elect, fail over,
+or lose. Adding or removing a server changes throughput, not correctness: any server
+can plan any rule and run any due task.
+
+## Aiming for exactly-once across a cluster
+
+In a distributed system, servers crash mid-run, clocks drift, and network calls
+time out. A scheduler cannot make those failures impossible, so this one is built to
+**target exactly-once and, when something does go wrong, to prefer not losing work
+over never repeating it.** In rare recovery situations that can mean a run executes
+more than once, so handlers should tolerate being called again for the same run.
+The design leans on a few ideas working together.
+
+- **Claim.** A run is only ever picked up through an atomic database claim: a
+  single statement that flips exactly one waiting row to "running" and stamps it
+  with the server that claimed it. Because the database resolves the race, only one
+  server can win a given run, no matter how many try at once.
+- **Lease.** A claim comes with an expiry (a *lease*). While the lease is valid the
+  run belongs to that server. If the server dies, the lease lapses and the recovery
+  pass can safely take the run back. Without leases a crashed server would strand
+  its runs forever.
+- **Fencing.** Each claim carries a version number (an *epoch*) that increases every
+  time a run is claimed. Every final write ("mark succeeded", "mark failed") is
+  guarded by that number. So if a slow server comes back from the dead after its
+  run was already recovered and re-run elsewhere, its late write matches nothing and
+  is harmlessly ignored. It cannot overwrite the newer result.
+
+The claim, the lease and the recovery pass are the safety net: a crash at any point
+is recoverable rather than silently lost. A few supporting choices reinforce it:
+
+- **Identity and idempotency.** Each concrete run is unique on its
+  `(job, instant)` pair, so the same instant can never be queued twice, even if two
+  planning passes race.
+- **Recomputable recurrence.** The next run is derived from the last one, never from
+  a counter held in memory, so any server can compute it and a restart never drifts.
+- **Shared backoff.** A retry after a handler failure and a retry after a crash use
+  the same backoff curve, so recovery behaves consistently however a run failed.
+
+## Lifecycle of a job (a rule)
+
+A job carries a "clock" (its next-run time). The planner keeps advancing that clock
+as it queues runs.
+
+```mermaid
+stateDiagram-v2
+    [*] --> Active: rule provisioned
+    Active --> Active: planner queues upcoming runs, advances the clock
+    Active --> Deferred: schedule can't be planned<br/>(unknown timezone, corrupt row)
+    Deferred --> Active: retried on a later pass
+    Active --> Active: rule edited<br/>(queued runs withdrawn, clock reset)
+    Active --> Exhausted: last run queued (a one-off that fired)
+    Active --> [*]: rule removed
+    Exhausted --> [*]: rule removed
+```
+
+- **Active**: has a next-run time; the planner keeps materialising it.
+- **Deferred**: planning failed for now; the clock is pushed out and retried later,
+  so a transient cause (a fixed timezone, updated timezone data) heals on its own.
+- **Exhausted**: no future runs left (a one-off that has fired); the clock is empty.
+- Editing or removing a rule is handled by provisioning, which withdraws any runs
+  that were queued under the old definition.
+
+## Lifecycle of a task (a concrete run)
+
+A task is created by the planner and moves through claim, run, and a final outcome.
+
+```mermaid
+stateDiagram-v2
+    [*] --> Pending: planner queues the run
+
+    Pending --> Running: a server claims it (takes a lease)
+
+    Running --> Succeeded: handler finished
+    Running --> Pending: handler failed, attempts left<br/>(retry after backoff)
+    Running --> Failed: handler failed, no attempts left
+
+    Running --> Pending: server crashed, lease expired,<br/>attempts left (recovery)
+    Running --> Failed: server crashed, lease expired,<br/>no attempts left (recovery)
+
+    Pending --> Cancelled: its rule was edited or removed
+    Pending --> Missed: too stale to run (host policy)
+
+    Succeeded --> [*]: deleted by cleanup
+    Failed --> [*]: deleted by cleanup
+    Cancelled --> [*]: deleted by cleanup
+    Missed --> [*]: deleted by cleanup
+```
+
+- **Pending**: queued and waiting for its instant.
+- **Running**: claimed and owned by one server for the duration of its lease.
+- **Succeeded / Failed**: the two normal final outcomes. "Failed" means all attempts
+  were used up, whether from repeated handler errors or repeated crashes.
+- **Cancelled**: the run's rule changed or was removed before it fired, so the
+  queued run is withdrawn.
+- **Missed**: a run so old it is no longer worth firing; whether a stale run still
+  runs or is marked missed is the host's policy, not the scheduler's.
+- Finished runs (the four final states) are eventually deleted by the cleanup pass.
+
+## What this package is (and is not)
+
+It is a **library of scheduling logic**, not a running service on its own. It owns
+the algorithms and the schedule math; it does not own a database connection, a DI
+container, or a configuration source. A **host** provides those and drives it.
+
+The public surface is small:
+
+- **`createScheduler(deps)`** composes the submodules into a scheduler. `deps` is
+  the host's contribution: a task store and a transaction runner (the `@n8n/db`
+  repositories fit these shapes directly, so no adapter layer is needed), an
+  instance identity, optional per-pass tuning, and optional sinks for events,
+  tracing and metrics.
+- The resulting **`Scheduler`** exposes `registerTaskHandler` (bind a handler to a
+  task type), `start` (begin the repeating passes) and `stop` (drain them
+  gracefully). Individual passes can also be triggered one at a time, which is used
+  mainly for tests and manual driving.
+- A few pure helpers are exported too: schedule validation, next-run computation,
+  and the provisioning entry points.
+
+In n8n, that host is the `cli` package's durable-scheduler module, which wires up
+DI, configuration and instance identity, and routes the scheduler's events to the
+logger. While `N8N_SCHEDULER_ENABLED` is off, the previous in-memory engine stays
+in place; turning the flag off again is the rollback.
+
+The boundary this section describes is enforced, not just documented.
+`src/__tests__/dependency-purity.test.ts` fails if the package ever gains a
+forbidden dependency, whether a new `package.json` entry outside the allowlist or
+an `import` of something effectful (`@n8n/db`, TypeORM, `n8n-core`, the DI
+container, the `cli` package, or a Node I/O built-in). Its opening doc comment is
+the living explanation of the approach: why scheduling is split from execution and
+storage, why each excluded dependency is excluded, and how far the idea could be
+taken. Read it first if you want the reasoning behind the split above.
+
+## Good to know
+
+A few things that are not obvious from the code but save a lot of confusion.
+
+- **"Processing a task" is *scheduling* it, not doing the work.** From the
+  scheduler's point of view, handling a due task means claiming it and arming a
+  precise timer so it fires at its exact instant. The actual work is whatever your
+  registered handler does; the scheduler only hands the task to it at the right
+  moment and records the outcome. So a fast scheduler pass does not mean the work is
+  fast, and a slow handler does not block the passes that queue and claim other
+  tasks.
+
+- **A plain cron expression cannot count across cycles.** It selects times within a
+  repeating calendar (minutes, hours, days-of-week), so cadences like "every third
+  Monday" or "every 5 hours" are simply not expressible in cron. That is the whole
+  reason `recurring_cron` exists: it wraps a cron anchor with an "every N periods"
+  filter to cover those cases. If a schedule fits plain cron, use `cron`; reach for
+  `recurring_cron` only when the cadence spans cycles.
+
+- **The package never logs, traces or measures on its own.** Instead it reports
+  through optional hooks the host supplies, so observability is entirely the host's
+  choice:
+  - an **event sink** (`onEvent`) receives already-handled milestones and incidents
+    as plain, described messages, which the host typically routes to its logger;
+  - a **tracer** wraps each pass and each handler run in a span;
+  - a **metrics** sink records counts and timings (dispatches, retries, dead
+    letters, and so on).
+
+  All three default to no-ops, and a throwing hook is swallowed so a broken logger
+  or exporter can never break the scheduling it was only meant to observe.
+
+- **Two warnings tell an operator their timing is off.** Both arrive through the
+  event sink at `warn` level, so they land in the host's logs:
+  - a **clock-skew warning** at start-up. Due-ness and leases are judged on the
+    clock the scheduler coordinates on (the host supplies it via `now`, e.g. the
+    database clock), but fire timers are armed on this instance's own clock. When
+    `start` samples the two and they differ by more than the threshold (default
+    `1s`, `CLOCK_SKEW_WARN_THRESHOLD_MS`), it warns: tasks may fire early or late,
+    and the instance clock should be synchronised (e.g. via NTP). The sample runs
+    detached, so it never delays the loops, and a slow read cannot trip it on its
+    own (the offset must also exceed half the round-trip). Skipped entirely when no
+    `now` is supplied.
+  - a **dispatch-lag warning** when a task fires at least
+    `DEFAULT_DISPATCH_LAG_WARN_THRESHOLD_SECONDS` (default `30s`) past its scheduled
+    time, once per late fire. This flags a genuinely late dispatch (a blocked event
+    loop, a skewed clock), not routine sub-second jitter.
+
+- **Runs are recorded ahead of time, within a window.** Because upcoming runs are
+  queued in advance, a frequent schedule does not need one planning pass per fire.
+  The flip side: disabling or editing a rule may take until the end of that window
+  to stop already-queued runs, unless the rule-writing path withdraws them (which
+  provisioning does).
+
+- **Each background pass runs on its own timer, with jitter.** The four passes are
+  independent and slightly randomised, so several servers do not all run their
+  passes in lockstep, and one slow pass does not stall the others.
+
+## Future ideas
+
+- **A dedicated host module.** Today the wiring that turns this pure library into a
+  running scheduler (DI, configuration, instance identity, event routing) lives
+  inside the large `cli` package. Extracting it into its own module, for example
+  `@n8n/scheduler-cli` or `@n8n/scheduler-features`, would keep the integration
+  concerns together and out of `cli`, and mirror the clean split this package
+  already draws between algorithm and host.
+- **Lease renewal for long handlers.** A claim's lease is fixed for now, so a
+  handler that runs longer than its lease risks being recovered and re-run. A
+  heartbeat that extends the lease while a handler is genuinely still working would
+  lift that constraint.
+
+### Exploring the idea: a standalone `@n8n/scheduler-worker`
+
+Pushed all the way, that host module becomes its own deployable worker. On a due
+task it does not run the workflow, it creates an execution and enqueues a job (Bull
+and Redis) for the regular workers, so the real work is decomposing `cli`. The
+result is far leaner than `cli`, which bundles every node's dependencies (1Password,
+S3, and more). The worker only needs:
+
+| Needs | Package | Pulls in |
+| --- | --- | --- |
+| Core logic | `@n8n/scheduler` | luxon, cron-parser |
+| Task store + transactions | `@n8n/db` | `@n8n/typeorm` + a database driver |
+| Enqueue executions (a new, extracted publisher) | `@n8n/…-publisher` | Bull + ioredis, execution writes, trigger-context |
+| Logging + lifecycle | `@n8n/backend-common` | winston |
+| Wiring + types | `@n8n/config`, `@n8n/di`, `@n8n/decorators`, `n8n-workflow` | reflect-metadata |
+| Instance identity (today) | `n8n-core` | the execution engine, unless extracted |
 
-Early foundation. The algorithms, schedule math and composition surface are in
-place; the driver that runs the passes on their cadences in the main process
-lands in later milestones.
+Full reasoning in the doc comment of `src/__tests__/dependency-purity.test.ts`.