@pattern-stack/codegen 0.9.1 → 0.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. package/README.md +5 -0
  2. package/consumer-skills/bridge/SKILL.md +265 -0
  3. package/consumer-skills/codegen/SKILL.md +115 -0
  4. package/consumer-skills/entities/SKILL.md +111 -0
  5. package/consumer-skills/entities/families-and-queries.md +82 -0
  6. package/consumer-skills/entities/yaml-reference.md +118 -0
  7. package/consumer-skills/events/SKILL.md +71 -0
  8. package/consumer-skills/events/authoring-events.md +164 -0
  9. package/consumer-skills/events/typed-bus-and-outbox.md +163 -0
  10. package/consumer-skills/jobs/SKILL.md +66 -0
  11. package/consumer-skills/jobs/handler-authoring.md +236 -0
  12. package/consumer-skills/jobs/pools-and-ordering.md +161 -0
  13. package/consumer-skills/subsystems/SKILL.md +105 -0
  14. package/consumer-skills/subsystems/wiring-and-order.md +120 -0
  15. package/consumer-skills/sync/SKILL.md +134 -0
  16. package/consumer-skills/sync/audit-and-detection.md +302 -0
  17. package/consumer-skills/sync/change-sources-and-sinks.md +442 -0
  18. package/dist/runtime/subsystems/bridge/bridge.module.js +3 -0
  19. package/dist/runtime/subsystems/bridge/bridge.module.js.map +1 -1
  20. package/dist/runtime/subsystems/bridge/index.js +3 -0
  21. package/dist/runtime/subsystems/bridge/index.js.map +1 -1
  22. package/dist/runtime/subsystems/index.js +3 -0
  23. package/dist/runtime/subsystems/index.js.map +1 -1
  24. package/dist/runtime/subsystems/jobs/index.js +3 -0
  25. package/dist/runtime/subsystems/jobs/index.js.map +1 -1
  26. package/dist/runtime/subsystems/jobs/job-run-keyset-cursor.js +3 -0
  27. package/dist/runtime/subsystems/jobs/job-run-keyset-cursor.js.map +1 -1
  28. package/dist/runtime/subsystems/jobs/job-run-service.drizzle-backend.js +3 -0
  29. package/dist/runtime/subsystems/jobs/job-run-service.drizzle-backend.js.map +1 -1
  30. package/dist/runtime/subsystems/jobs/job-run-service.memory-backend.js +3 -0
  31. package/dist/runtime/subsystems/jobs/job-run-service.memory-backend.js.map +1 -1
  32. package/dist/runtime/subsystems/jobs/job-run-service.protocol.d.ts +9 -1
  33. package/dist/runtime/subsystems/jobs/job-worker.module.js +3 -0
  34. package/dist/runtime/subsystems/jobs/job-worker.module.js.map +1 -1
  35. package/dist/runtime/subsystems/jobs/jobs-domain.module.js +3 -0
  36. package/dist/runtime/subsystems/jobs/jobs-domain.module.js.map +1 -1
  37. package/dist/src/cli/index.js +913 -405
  38. package/dist/src/cli/index.js.map +1 -1
  39. package/dist/src/index.js +26 -4
  40. package/dist/src/index.js.map +1 -1
  41. package/package.json +2 -1
  42. package/runtime/subsystems/jobs/job-run-keyset-cursor.ts +3 -0
  43. package/runtime/subsystems/jobs/job-run-service.protocol.ts +9 -1
@@ -0,0 +1,236 @@
1
+ <!-- managed by @pattern-stack/codegen — re-run `codegen skills install` to refresh. Edit the package source, not this vendored copy. -->
2
+
3
+ # Authoring a Job Handler
4
+
5
+ This is the surface your application code touches most: how to write a `@JobHandler` class, use `JobContext` correctly, and kick a job off from a use case. Read the `jobs` `SKILL.md` first for the mental model.
6
+
7
+ ## Minimal handler
8
+
9
+ ```ts
10
+ import {
11
+ JobHandler,
12
+ JobHandlerBase,
13
+ JobContext,
14
+ ParentClosePolicy,
15
+ } from '@shared/subsystems/jobs';
16
+ import type { ScopeEntityType } from '@shared/jobs/scope-entity-type';
17
+
18
+ interface OnboardingInput {
19
+ accountId: string;
20
+ }
21
+
22
+ interface OnboardingOutput {
23
+ emailCount: number;
24
+ }
25
+
26
+ @JobHandler<OnboardingInput>('onboarding', {
27
+ pool: 'batch',
28
+ scope: {
29
+ entity: 'account' satisfies ScopeEntityType,
30
+ from: (input) => input.accountId,
31
+ },
32
+ retry: { attempts: 3, backoff: 'exponential', baseMs: 1000 },
33
+ concurrency: {
34
+ key: (input) => `account:${input.accountId}`,
35
+ collisionMode: 'queue',
36
+ },
37
+ dedupe: {
38
+ key: (input) => `onboarding:${input.accountId}`,
39
+ windowMs: 24 * 60 * 60 * 1000,
40
+ },
41
+ timeoutMs: 60 * 60 * 1000,
42
+ replayFrom: 'last_checkpoint',
43
+ })
44
+ export class OnboardingHandler extends JobHandlerBase<OnboardingInput, OnboardingOutput> {
45
+ constructor(
46
+ private readonly emails: EmailService,
47
+ private readonly facts: FactService,
48
+ ) {
49
+ super();
50
+ }
51
+
52
+ async run(ctx: JobContext<OnboardingInput>): Promise<OnboardingOutput> {
53
+ const emails = await ctx.step('pull_emails', () =>
54
+ this.emails.pullForAccount(ctx.input.accountId),
55
+ );
56
+
57
+ await ctx.spawnChild(
58
+ 'process_facts',
59
+ { emailIds: emails.map((e) => e.id) },
60
+ { closePolicy: ParentClosePolicy.Terminate },
61
+ );
62
+
63
+ return { emailCount: emails.length };
64
+ }
65
+ }
66
+ ```
67
+
68
+ The handler class:
69
+ - **extends `JobHandlerBase<TInput, TOutput>`** and implements the single `run(ctx)` method.
70
+ - is decorated with `@JobHandler<TInput>('job_type', meta)`. The string is the unique job type — it is what you pass to `orchestrator.start(...)`.
71
+
72
+ ### Register the class as a provider
73
+
74
+ The decorator registers the class with the job registry so the worker knows the type exists. It does **not** register it with Nest's DI container — that is on you. Add the handler to the `providers` array of its owning module (e.g. an entity/feature module). At runtime the worker resolves it via the Nest module tree; a handler that is not a registered provider fails when its first run is claimed. Constructor injection works like any provider, as long as the providing module is imported where your handler lives.
75
+
76
+ ## Decorator metadata reference
77
+
78
+ Everything except the `type` positional argument is optional.
79
+
80
+ | Field | Shape | Notes |
81
+ |---|---|---|
82
+ | `pool` | `string` | Default `'batch'`. Must not be a reserved `events_*` pool — app boot throws `ReservedPoolViolationError`. `interactive` is allowed but must be explicit. See `pools-and-ordering.md`. |
83
+ | `scope` | `{ entity: ScopeEntityType; from: (input) => string }` | Ties each run to a domain entity id so you can later list/cancel "everything for this account". `entity` should be `'<name>' satisfies ScopeEntityType`. |
84
+ | `retry` | `{ attempts, backoff: 'fixed' \| 'exponential', baseMs, nonRetryableErrors? }` | Run-level retry across the whole handler. Independent of step-level retry. |
85
+ | `concurrency` | `{ key: (input) => string; collisionMode: 'queue' \| 'reject' \| 'replace' }` | Evaluated at enqueue. See collision modes below. |
86
+ | `dedupe` | `{ key: (input) => string; windowMs: number }` | Collapses duplicate enqueues inside the window — returns the existing run id, no new row. |
87
+ | `timeoutMs` | `number` | Hard wall-clock cap across all retries. Breach → `status='timed_out'`. |
88
+ | `replayFrom` | `'scratch' \| 'last_step' \| 'last_checkpoint'` | Default `'last_checkpoint'`. Only matters when a run is replayed. |
89
+ | `triggers` | `JobTrigger<TInput>[]` | Bind this job to domain events — covered by the `bridge` skill, not here. |
90
+
91
+ ### Concurrency collision modes
92
+
93
+ Set once on the decorator, not per call site. Two runs collide when their `key(input)` matches and the incumbent is still in a non-terminal state.
94
+
95
+ - `queue` (default) — the new run is accepted as `pending`; it is only claimed once the incumbent leaves its non-terminal state. Serializes by key.
96
+ - `reject` — the new `start(...)` throws `JobCollisionError` carrying the incumbent's run id.
97
+ - `replace` — the incumbent is cancelled (cascade), the new run starts. The "latest wins" pattern.
98
+
99
+ Concurrency is orthogonal to dedupe: dedupe short-circuits (no new row); concurrency queues (new row exists, claim is gated).
100
+
101
+ ### Replay modes
102
+
103
+ `replayFrom` is a memoization policy only — the same handler code runs in all cases. It controls what `job_step` rows survive when a run is replayed:
104
+
105
+ - `scratch` — clear all step rows, re-enter from empty. Your steps must be safe to re-run.
106
+ - `last_step` — clear only the failing step's row; completed steps stay memoized.
107
+ - `last_checkpoint` (default) — clear nothing; every completed step returns its cached output.
108
+
109
+ ## Using `JobContext`
110
+
111
+ ```ts
112
+ ctx.input // your TInput, typed via @JobHandler<TInput> — no cast needed
113
+ ctx.run // the JobRun row (id, rootRunId, parentRunId, scope, tags, attempts, ...)
114
+ ctx.step(id, fn, opts?) // durable, memoized step
115
+ ctx.spawnChild(type, input, opts?) // launch a child run, returns its JobRun
116
+ ctx.logger // a NestJS Logger scoped to this run — prefer it over console.log
117
+ ```
118
+
119
+ ### `ctx.step(stepId, fn, opts?)` — durable memoized step
120
+
121
+ Wrap anything slow, side-effectful, or externally costly. The first successful run persists `fn`'s output to `job_step`; on a retry or replay, the cached value is returned without calling `fn` again.
122
+
123
+ Rules:
124
+ - `stepId` must be **stable across replays** — hardcode (`'pull_emails'`) or derive deterministically (`` `recompute:${ctx.input.accountId}` ``). Never `Date.now()`, never random, never a mutable counter.
125
+ - `stepId` is unique within a run. Calling `ctx.step('x', …)` twice with the same id returns the cached value the second time.
126
+ - `fn`'s return value must be JSON-serializable — it is stored in a `jsonb` column.
127
+ - Design `fn` to be idempotent if you use `replayFrom: 'scratch'`.
128
+ - An error inside `fn` marks the step failed and rethrows. If the run has retries left, the whole run re-enters `pending`; on the next tick, completed sibling steps stay memoized and only this step retries.
129
+
130
+ Step-level retry (separate from run-level retry, wraps just this step):
131
+
132
+ ```ts
133
+ await ctx.step('fetch_profile', () => api.get(...), {
134
+ retry: { attempts: 2, backoff: 'fixed', baseMs: 500 },
135
+ timeoutMs: 30_000,
136
+ });
137
+ ```
138
+
139
+ ### `ctx.spawnChild(type, input, opts?)` — launch a child run
140
+
141
+ ```ts
142
+ const child = await ctx.spawnChild(
143
+ 'process_facts',
144
+ { emailIds },
145
+ {
146
+ closePolicy: ParentClosePolicy.Terminate, // default; children die if the parent does
147
+ runAt: new Date(Date.now() + 5_000), // optional delay
148
+ priority: 10,
149
+ tags: { triggeredBy: 'onboarding' },
150
+ },
151
+ );
152
+ ```
153
+
154
+ - `parent_run_id` and `root_run_id` are wired automatically.
155
+ - `closePolicy` is recorded on the child at spawn time; later changes to the parent do not retroactively rewrite it.
156
+ - `ParentClosePolicy.Terminate` (default) — running children are cancelled when the parent reaches any terminal state.
157
+ - `ParentClosePolicy.Cancel` — same effect, but the parent's `finished_at` is held until children finish transitioning.
158
+ - `ParentClosePolicy.Abandon` — children are left running (fire-and-forget).
159
+ - Do not wrap `spawnChild` in a `ctx.step` — the child is its own memoization root. If you need cross-invocation idempotency on a child, put `dedupe` on the child job type.
160
+ - There is **no built-in "await child completion"** primitive. If you need sequencing, split the parent's logic: the parent spawns the child (with its own scope) and completes; a follow-up handler watches for child completion via `listForScope`.
161
+
162
+ ## Kicking a job off from a use case
163
+
164
+ Inject the orchestrator token and call `start`:
165
+
166
+ ```ts
167
+ import { Inject, Injectable } from '@nestjs/common';
168
+ import { JOB_ORCHESTRATOR, type IJobOrchestrator } from '@shared/subsystems/jobs';
169
+
170
+ @Injectable()
171
+ export class StartOnboardingUseCase {
172
+ constructor(
173
+ @Inject(JOB_ORCHESTRATOR) private readonly jobs: IJobOrchestrator,
174
+ ) {}
175
+
176
+ async execute(accountId: string) {
177
+ return this.jobs.start(
178
+ 'onboarding',
179
+ { accountId },
180
+ {
181
+ scope: { entityType: 'account', entityId: accountId },
182
+ triggerSource: 'manual',
183
+ tags: { source: 'admin-ui' },
184
+ },
185
+ );
186
+ }
187
+ }
188
+ ```
189
+
190
+ - `start` returns the inserted `JobRun` immediately; the worker picks it up on its next poll.
191
+ - If `dedupe` matches an in-window prior run, `start` returns the existing run id — free idempotency for the caller.
192
+ - With `collisionMode: 'reject'`, `start` throws `JobCollisionError` synchronously — catch it in the use case.
193
+ - Pass `triggerSource: 'manual' | 'schedule' | 'event' | 'parent'` so the run's `trigger_source` is accurate for observability.
194
+
195
+ ## Managing runs by entity — `IJobRunService`
196
+
197
+ `scope` on the decorator makes "everything for this account" queryable:
198
+
199
+ ```ts
200
+ import { JOB_RUN_SERVICE, type IJobRunService } from '@shared/subsystems/jobs';
201
+
202
+ @Inject(JOB_RUN_SERVICE) private readonly runs: IJobRunService;
203
+
204
+ await this.runs.listForScope('account', accountId, { status: 'running' });
205
+ await this.runs.cancelForScope('account', accountId); // cascades per close policy
206
+ await this.runs.rescheduleForScope('account', accountId, tomorrow);
207
+ ```
208
+
209
+ The entity type is a plain string at the database layer; type safety comes from `satisfies ScopeEntityType` at the call site. `ScopeEntityType` is a generated union of every entity that declared `scopeable: true` in its YAML — re-run codegen after adding a new scopeable entity to refresh `@shared/jobs/scope-entity-type`.
210
+
211
+ ## Common shapes
212
+
213
+ - **Fan-out** — parent `ctx.step`s a batch, then `ctx.spawnChild` per item with `ParentClosePolicy.Terminate`. Cancel the parent → children die.
214
+ - **Sequenced pipeline** — chain `ctx.step` calls in one handler (simplest), or split into dedicated handler types with `ParentClosePolicy.Abandon` so each hop is independently queryable and retryable.
215
+ - **Latest-wins loop** — `concurrency.collisionMode: 'replace'`, keyed by the entity id. A newer trigger cancels the incumbent.
216
+ - **Webhook ingestion** — `dedupe.key: (input) => input.externalId` collapses replays within the window, no app-level idempotency table needed.
217
+ - **Long batch with safe resume** — `replayFrom: 'last_checkpoint'` (the default) + one `ctx.step` per work unit → a crash only repeats unfinished work.
218
+
219
+ ## Testing a handler
220
+
221
+ `@JobHandler` classes work inside `Test.createTestingModule` with the memory backend, which is behavior-parity with the production backend for claim order, collision modes, step memoization, cascade cancel, dedupe, and replay:
222
+
223
+ ```ts
224
+ const moduleRef = await Test.createTestingModule({
225
+ imports: [JobWorkerModule.forRoot({ mode: 'embedded', backend: 'memory' })],
226
+ providers: [OnboardingHandler, /* ...deps */],
227
+ }).compile();
228
+ ```
229
+
230
+ Then `orchestrator.start('onboarding', input)`, advance a few ticks, and assert on the stored runs and steps.
231
+
232
+ ## When NOT to use a handler
233
+
234
+ - **Synchronous request/response** — if the caller awaits the result, write a use case, not a job.
235
+ - **Reacting to every domain event** — those flow through the events subsystem and (for durable async fanout) the bridge, not a direct handler. See the `events` and `bridge` skills.
236
+ - **Per-request rate limiting** — there is no built-in request rate-limit primitive in the core contract.
@@ -0,0 +1,161 @@
1
+ <!-- managed by @pattern-stack/codegen — re-run `codegen skills install` to refresh. Edit the package source, not this vendored copy. -->
2
+
3
+ # Pools, Ordering, and Configuration
4
+
5
+ How pools work, how to configure them in `codegen.config.yaml`, how to get the ordering guarantee you actually need, and how to wire the job worker into your app. Read this when you are choosing a pool for a handler, adding a custom pool, or deciding between embedded and standalone workers.
6
+
7
+ ## What a pool is
8
+
9
+ A pool is a logical lane. Each pool maps to:
10
+ - A distinct queue identifier (written into `job_run.pool`).
11
+ - One worker instance per active pool per process.
12
+ - A `concurrency` cap — the max number of in-flight runs that pool processes at once.
13
+
14
+ **Pools are absolute lanes — there is no cross-pool preemption.** A high-priority `batch` run does not jump ahead of `interactive`. If you need isolation between two classes of work, give them separate pools. This is deliberate: priority-based scheduling within one queue can starve low-priority work under sustained load; separate lanes cannot.
15
+
16
+ ## The default pools
17
+
18
+ Your installed `jobs:` config ships five pools:
19
+
20
+ | Pool | concurrency | reserved | Use for |
21
+ |---|---|---|---|
22
+ | `events_inbound` | 20 | yes | (framework) external → us event traffic |
23
+ | `events_change` | 30 | yes | (framework) internal change-event traffic |
24
+ | `events_outbound` | 10 | yes | (framework) us → external event traffic |
25
+ | `interactive` | 20 | no | User-waiting work: exports, renders, ad-hoc one-offs |
26
+ | `batch` | 5 | no | Background work: onboarding, ingest, long jobs. **Default for your handlers.** |
27
+
28
+ You may override `concurrency` (and `description`) on the non-reserved pools, and you may add your own.
29
+
30
+ ### Reserved pools are off-limits to your handlers
31
+
32
+ The three `events_*` pools exist only to carry event/bridge traffic, one lane per event direction. A `@JobHandler({ pool: 'events_change' })` (or any reserved pool) throws `ReservedPoolViolationError` at app boot — the error names the offending class. You cannot flip `reserved` off in config, and you cannot mark your own pools `reserved`.
33
+
34
+ To run a job *when an event fires*, declare `@JobHandler.triggers` and let the bridge enqueue it into your chosen (non-reserved) pool. See the `bridge` skill.
35
+
36
+ ## The `jobs:` config block
37
+
38
+ `codegen subsystem install jobs` injects this into `codegen.config.yaml`:
39
+
40
+ ```yaml
41
+ jobs:
42
+ backend: drizzle # 'drizzle' (default, Postgres) | 'memory' (tests) | 'bullmq' (opt-in)
43
+
44
+ extensions:
45
+ drizzle:
46
+ poll_interval_ms: 1000
47
+ # listen_notify: true # opt-in Postgres LISTEN/NOTIFY for sub-second wakeups
48
+
49
+ multi_tenant: false # true → service layer requires a tenantId
50
+
51
+ worker_mode: embedded # embedded | standalone (operational hint; see below)
52
+
53
+ pools:
54
+ events_inbound: { queue: jobs-events-inbound, concurrency: 20, reserved: true }
55
+ events_change: { queue: jobs-events-change, concurrency: 30, reserved: true }
56
+ events_outbound: { queue: jobs-events-outbound, concurrency: 10, reserved: true }
57
+ interactive: { queue: jobs-interactive, concurrency: 20 }
58
+ batch: { queue: jobs-batch, concurrency: 5 }
59
+ ```
60
+
61
+ Field notes:
62
+
63
+ | Key | What it controls |
64
+ |---|---|
65
+ | `backend` | Which orchestrator implementation runs. `drizzle` (Postgres) is the portable default. `memory` is for tests. `bullmq` is opt-in (see below). |
66
+ | `extensions.<backend>.*` | Backend-specific knobs. Each backend reads only its own key; unknown keys are ignored, not errors. |
67
+ | `multi_tenant` | When `true`, service methods require a `tenantId` (explicit `null` allowed for cross-tenant work). The `tenant_id` column exists regardless, so flipping this later needs no migration. |
68
+ | `worker_mode` | Informational hint only — both worker entrypoints are always scaffolded. See "Worker topology". |
69
+ | `pools.<name>.queue` | The queue identifier written into `job_run.pool`. Must be unique. |
70
+ | `pools.<name>.concurrency` | Per-process max in-flight for that pool. Running more processes multiplies it. |
71
+
72
+ ## Adding a custom pool
73
+
74
+ Pure config change — no code edits:
75
+
76
+ ```yaml
77
+ jobs:
78
+ pools:
79
+ # framework defaults are merged automatically; you only add yours
80
+ agents:
81
+ queue: jobs-agents
82
+ concurrency: 3
83
+ description: "Long-running LLM / agent work"
84
+ ```
85
+
86
+ Then any `@JobHandler({ pool: 'agents', … })` targets it. The worker discovers the pool at boot and starts its claim loop.
87
+
88
+ ## Ordering: parallelism vs. order
89
+
90
+ By default a pool runs at its configured concurrency, so two runs in the same pool can execute concurrently. There is **no implicit ordering guarantee**. If you genuinely need ordered execution, choose the narrowest knob that satisfies the requirement:
91
+
92
+ 1. **Per-entity ordering (preferred).** Set `concurrency` on the `@JobHandler` with a key derived from the entity and `collisionMode: 'queue'`:
93
+
94
+ ```ts
95
+ @JobHandler<ProvisionInput>('provision_workspace', {
96
+ concurrency: {
97
+ key: (input) => `account:${input.accountId}`,
98
+ collisionMode: 'queue',
99
+ },
100
+ })
101
+ ```
102
+
103
+ This serializes runs sharing the same key while keeping unrelated keys parallel. Keeps throughput high.
104
+
105
+ 2. **Whole-pool serialization (blunt).** Set `concurrency: 1` on the pool in config. Serializes *every* run in that pool end to end. Use only when every run in the pool genuinely needs strict order — it caps throughput hard.
106
+
107
+ If you think you need strict ordering *across different entities*, reconsider — that is usually a sign the work should tolerate independent timelines.
108
+
109
+ ## Worker topology — embedded vs. standalone
110
+
111
+ Both entrypoints are always scaffolded. The choice is operational; switching needs no regeneration.
112
+
113
+ **Embedded** — your `AppModule` imports `JobWorkerModule.forRoot({ mode: 'embedded' })`. The API process and the workers share the same process. Simplest; good default for dev and small deployments.
114
+
115
+ **Standalone** — run the scaffolded `worker.ts` as its own process. `main.ts` does not import `JobWorkerModule`; the worker boots a bare Nest application context (no HTTP listener) with the database module plus the jobs modules. Lets you scale workers independently of the API.
116
+
117
+ ## Wiring into your app
118
+
119
+ Two modules, both `global: true`:
120
+
121
+ ```ts
122
+ import { JobWorkerModule } from '@shared/subsystems/jobs';
123
+
124
+ @Module({
125
+ imports: [
126
+ DatabaseModule,
127
+ // Brings the orchestrator/services AND runs the worker claim loops:
128
+ JobWorkerModule.forRoot({ mode: 'embedded', backend: 'drizzle' }),
129
+ // ...
130
+ ],
131
+ })
132
+ export class AppModule {}
133
+ ```
134
+
135
+ `JobWorkerModule.forRoot({ mode, backend?, pools?, multiTenant?, shutdownTimeoutMs? })` imports `JobsDomainModule` internally and starts a worker per active pool. The protocol tokens (`JOB_ORCHESTRATOR`, `JOB_RUN_SERVICE`, `JOB_STEP_SERVICE`) become available project-wide.
136
+
137
+ - Pass `pools: ['batch', 'agents']` to restrict which pools *this* process services — useful for heterogeneous standalone deploys. Pools omitted from the list are not claimed by this process.
138
+ - A process that only needs to *start* jobs (not run them) can import `JobsDomainModule.forRoot({ backend })` alone — services available, no worker loop.
139
+
140
+ Tests swap the backend:
141
+
142
+ ```ts
143
+ JobWorkerModule.forRoot({ mode: 'embedded', backend: 'memory' })
144
+ ```
145
+
146
+ ## Backend choice
147
+
148
+ The portability contract is the same across backends: code written against `IJobOrchestrator` + `IJobRunService` + `IJobStepService` works on any backend. Backend-specific features live under `extensions.<backend>`.
149
+
150
+ - **`drizzle`** (default) — Postgres only, no extra infra. The worker polls `job_run`. Extensions: `poll_interval_ms`, opt-in `listen_notify`.
151
+ - **`memory`** — in-process, for tests. Behavior-parity with Drizzle for the scenarios that matter.
152
+ - **`bullmq`** — opt-in via `backend: bullmq`. Postgres `job_run` stays the domain source of truth; BullMQ replaces only the claim/dispatch half. Extensions include a Bull Board admin UI mount and `FlowProducer` access. Choose this only after you have a measured reason to.
153
+
154
+ ## Multi-tenancy
155
+
156
+ `jobs.multi_tenant: true` is a single opt-in:
157
+ - Backend methods accept a `tenantId`; a missing one throws when the flag is on (explicit `null` allowed for cross-tenant background work).
158
+ - Claim, `listForScope`, `cancel`, etc. filter by `tenantId`.
159
+ - The `tenant_id` column is always present, so flipping the flag never needs a migration.
160
+
161
+ Also pass `multiTenant: true` to `JobWorkerModule.forRoot(...)` so the runtime enforces it, and keep the config flag and the module option in agreement.
@@ -0,0 +1,105 @@
1
+ ---
2
+ name: subsystems
3
+ description: >-
4
+ Load when installing or wiring an infrastructure subsystem in a project that
5
+ uses @pattern-stack/codegen — events, jobs, cache, storage, sync, bridge,
6
+ observability, auth, or the OpenAPI config. Covers `codegen subsystem
7
+ install`, the `forRoot` registration ORDER in app.module.ts, which subsystems
8
+ depend on which, and multi-tenancy opt-in. Get the order wrong and the bridge
9
+ sits idle or observability sees nothing — this skill is the source of truth
10
+ for ordering until the CLI enforces it.
11
+ allowed-tools: Read, Write, Edit, Glob, Grep, Bash
12
+ user-invocable: false
13
+ ---
14
+
15
+ <!-- managed by @pattern-stack/codegen — re-run `codegen skills install` to refresh. Edit the package source, not this vendored copy. -->
16
+
17
+ # Infrastructure subsystems
18
+
19
+ Subsystems are the generated infrastructure your use cases call: an event bus,
20
+ a job queue, a cache, file storage, an external-sync engine, the event-to-job
21
+ bridge, a read-only observability facade, and OAuth auth. Each follows one
22
+ pattern — **Protocol (port) → Backend (adapter) → Factory (`DynamicModule.
23
+ forRoot`)** — and each is `global: true`, so you register it once in
24
+ `app.module.ts` and inject its token anywhere.
25
+
26
+ ## Mental model
27
+
28
+ - **Install vendors runtime + injects config.** `codegen subsystem install
29
+ <name>` copies the subsystem's runtime into `<subsystems-root>/<name>/`
30
+ (default `src/shared/subsystems/<name>/`) and adds its block to
31
+ `codegen.config.yaml`. You then add one `forRoot(...)` line to `app.module.ts`.
32
+ - **Backends are swappable; tests use memory.** Most subsystems ship a Drizzle
33
+ (Postgres) production backend and a memory backend for tests. Swap via the
34
+ `forRoot({ backend })` arg — app code is unchanged.
35
+ - **Order matters.** Some subsystems consume others. The bridge consumes events
36
+ + jobs; observability composes events/jobs/bridge/sync read ports via
37
+ optional DI. Registering them in the wrong order means a silently idle bridge
38
+ or an observability facade that reports nothing. See `wiring-and-order.md`.
39
+
40
+ ## The subsystems
41
+
42
+ | Subsystem | Token / module | Install | Depends on |
43
+ |---|---|---|---|
44
+ | events | `EventsModule` | `subsystem install events` | — |
45
+ | jobs | `JobsDomainModule` + `JobWorkerModule` | `subsystem install jobs` | — |
46
+ | cache | `CacheModule` | `subsystem install cache` | jobs (optional, for cleanup) |
47
+ | storage | `StorageModule` | `subsystem install storage` | — |
48
+ | sync | `SyncModule` | `subsystem install sync` | — |
49
+ | bridge | `BridgeModule` | `subsystem install bridge` | **events + jobs** |
50
+ | observability | `ObservabilityModule` | `subsystem install observability` | composes events/jobs/bridge/sync (optional) |
51
+ | auth | `AuthModule` | `subsystem install auth` | — |
52
+ | auth-integrations | `IntegrationsAuthModule` | `subsystem install auth-integrations` | **auth** |
53
+ | openapi | (config only) | `subsystem install openapi-config` | registry vendored at init |
54
+
55
+ ## Registration order (authoritative)
56
+
57
+ In `app.module.ts`, import in this order (omit what you haven't installed):
58
+
59
+ 1. `DatabaseModule` — provides `DRIZZLE`; must be first.
60
+ 2. `OpenApiModule` — the registry singleton (vendored at init).
61
+ 3. `EventsModule.forRoot(...)`
62
+ 4. `JobsDomainModule.forRoot(...)` **and** `JobWorkerModule.forRoot(...)`
63
+ 5. `CacheModule` / `StorageModule` / `SyncModule.forRoot(...)`
64
+ 6. `BridgeModule.forRoot(...)` — **after** events + jobs.
65
+ 7. `ObservabilityModule.forRoot(...)` — **last** of the subsystems (composes the
66
+ ones above via optional DI).
67
+ 8. `...GENERATED_MODULES` — your entity modules.
68
+
69
+ For auth: register `AuthModule.forRoot(...)` before the `IntegrationsAuthModule`
70
+ that depends on it. Full per-subsystem `forRoot` signatures, the bridge reserved
71
+ pools, and multi-tenancy are in `wiring-and-order.md`.
72
+
73
+ ## Non-obvious rules
74
+
75
+ - **Jobs is two modules, not one.** `JobsDomainModule.forRoot({ backend })`
76
+ wires the orchestrator/run-services; `JobWorkerModule.forRoot({ mode, backend,
77
+ pools })` runs the worker loop. Pool *definitions* (concurrency, reserved
78
+ lanes) live in `codegen.config.yaml` under `jobs.pools`; `JobWorkerModule`'s
79
+ `pools:` is the list of *active* pool names this process drains.
80
+ - **The bridge will sit idle unless its reserved pools are polled.** The worker
81
+ must drain `events_inbound` / `events_change` / `events_outbound` — spread
82
+ `...BRIDGE_RESERVED_POOLS` into `JobWorkerModule`'s `pools`, or use `allPools:
83
+ true`. `BridgeModule` fails fast at boot if they aren't polled. See the
84
+ `bridge` skill.
85
+ - **Observability composes optionally.** It reads whatever sibling subsystems
86
+ are present; missing ones are simply absent from its output. That's why it
87
+ must be registered after them.
88
+ - **Multi-tenancy is a config flip + a `forRoot` flag + a migration** — never a
89
+ runtime-only toggle. See `wiring-and-order.md`.
90
+ - **`--backend memory`** is for tests; the scaffolded default is `drizzle`
91
+ (`local` for storage).
92
+
93
+ ## Do not
94
+
95
+ - **Do not register `BridgeModule` before `EventsModule` + the jobs modules** —
96
+ it consumes their tokens.
97
+ - **Do not register `ObservabilityModule` before the subsystems it reports on.**
98
+ - **Do not route your own jobs into the reserved `events_*` pools** — those are
99
+ the bridge's; module init rejects it. Declare your own pool.
100
+ - **Do not hand-edit vendored subsystem files** under `<subsystems-root>/<name>/`
101
+ — `codegen update` overwrites them. Compose/subclass instead.
102
+ - **Do not expect `codegen update` to refresh subsystem *schemas*.** It re-syncs
103
+ runtime source, not the tenancy-gated Drizzle schema files. If a schema shape
104
+ changed across versions, re-run `subsystem install <name> --force
105
+ --force-config`.
@@ -0,0 +1,120 @@
1
+ <!-- managed by @pattern-stack/codegen — re-run `codegen skills install` to refresh. Edit the package source, not this vendored copy. -->
2
+
3
+ # Subsystem wiring & registration order
4
+
5
+ The exact `forRoot` signatures and a complete `app.module.ts` example. All
6
+ modules are `global: true` — register once here; inject the token anywhere.
7
+
8
+ ## Complete `app.module.ts`
9
+
10
+ ```ts
11
+ import { Global, Module } from '@nestjs/common';
12
+ import { DatabaseModule } from './shared/database/database.module';
13
+ import { GENERATED_MODULES } from './generated/modules';
14
+ import { OPENAPI_REGISTRY, OpenApiRegistry } from './shared/openapi';
15
+
16
+ import { EventsModule } from '@shared/subsystems/events';
17
+ import { JobsDomainModule, JobWorkerModule } from '@shared/subsystems/jobs';
18
+ import { CacheModule } from '@shared/subsystems/cache';
19
+ import { StorageModule } from '@shared/subsystems/storage';
20
+ import { SyncModule } from '@shared/subsystems/sync';
21
+ import { BridgeModule, BRIDGE_RESERVED_POOLS } from '@shared/subsystems/bridge';
22
+ import { ObservabilityModule } from '@shared/subsystems/observability';
23
+
24
+ @Global()
25
+ @Module({
26
+ providers: [{ provide: OPENAPI_REGISTRY, useValue: new OpenApiRegistry() }],
27
+ exports: [OPENAPI_REGISTRY],
28
+ })
29
+ class OpenApiModule {}
30
+
31
+ @Module({
32
+ imports: [
33
+ // 1. database first — provides DRIZZLE
34
+ DatabaseModule,
35
+ // 2. openapi registry singleton
36
+ OpenApiModule,
37
+ // 3. events
38
+ EventsModule.forRoot({ backend: 'drizzle' }),
39
+ // 4. jobs — domain layer + worker loop
40
+ JobsDomainModule.forRoot({ backend: 'drizzle' }),
41
+ JobWorkerModule.forRoot({
42
+ mode: 'embedded',
43
+ backend: 'drizzle',
44
+ // include the bridge's reserved pools so wrappers actually drain:
45
+ pools: ['interactive', 'batch', ...BRIDGE_RESERVED_POOLS],
46
+ }),
47
+ // 5. cache / storage / sync
48
+ CacheModule.forRoot({ backend: 'drizzle' }),
49
+ StorageModule.forRoot({ backend: 'local' }),
50
+ SyncModule.forRoot({ backend: 'drizzle' }),
51
+ // 6. bridge — AFTER events + jobs
52
+ BridgeModule.forRoot({ backend: 'drizzle', multiTenant: false }),
53
+ // 7. observability — LAST (composes the siblings above)
54
+ ObservabilityModule.forRoot({ reporters: { bridgeMetrics: true } }),
55
+ // 8. your generated entity modules
56
+ ...GENERATED_MODULES,
57
+ ],
58
+ })
59
+ export class AppModule {}
60
+ ```
61
+
62
+ ## Per-subsystem `forRoot`
63
+
64
+ | Module | Signature | Notes |
65
+ |---|---|---|
66
+ | `EventsModule` | `forRoot({ backend, multiTenant?, pools? })` | `backend: 'drizzle' \| 'memory'`. `pools` restricts this process's drain loop to specific event lanes. |
67
+ | `JobsDomainModule` | `forRoot({ backend, multiTenant?, extensions? })` | `backend: 'drizzle' \| 'memory' \| 'bullmq'`. Domain layer (orchestrator, run/step services). `extensions.bullmq` / `extensions.drizzle` are the opt-in backend extras. |
68
+ | `JobWorkerModule` | `forRoot({ mode, backend?, pools?, allPools?, shutdownTimeoutMs? })` | `mode: 'embedded' \| 'standalone'`. `pools` = active pool names this process drains (defaults to all non-reserved). `allPools: true` drains every pool incl. reserved. |
69
+ | `CacheModule` | `forRoot({ backend })` | optionally registers a cleanup job when jobs is present. |
70
+ | `StorageModule` | `forRoot({ backend })` | `backend: 'local' \| 'memory'`. Implement S3/GCS by implementing the storage protocol. |
71
+ | `SyncModule` | `forRoot({ backend, multiTenant? })` | wires the cursor store / run recorder / differ ports — NOT the orchestrator (that's per-entity; see the `sync` skill). |
72
+ | `BridgeModule` | `forRoot({ backend, multiTenant? })` | must come after events + jobs; fails fast at boot if reserved pools aren't polled. |
73
+ | `ObservabilityModule` | `forRoot({ reporters? })` | read-only facade; `reporters.bridgeMetrics: true` opts into the 60s bridge sampler. Register last. |
74
+ | `AuthModule` | `forRoot({ encryptionKey, oauthStateStore })` | `global: true`; provides `ENCRYPTION_KEY` + `OAUTH_STATE_STORE`. Register before `IntegrationsAuthModule`. |
75
+
76
+ ## Pool configuration (jobs)
77
+
78
+ Pool *definitions* live in `codegen.config.yaml`, not in `forRoot`:
79
+
80
+ ```yaml
81
+ jobs:
82
+ backend: drizzle
83
+ pools:
84
+ - { name: interactive, concurrency: 8 }
85
+ - { name: batch, concurrency: 2 }
86
+ # the bridge's reserved lanes (events_inbound/_change/_outbound) are
87
+ # provided by BRIDGE_RESERVED_POOLS — see the bridge skill
88
+ ```
89
+
90
+ `JobWorkerModule.forRoot({ pools })` then names which of those a given worker
91
+ process drains — scale horizontally by running one worker per pool subset.
92
+
93
+ ## Multi-tenancy opt-in (events / jobs / sync / bridge)
94
+
95
+ Three coordinated steps — never a runtime-only toggle:
96
+
97
+ 1. Flip the config flag, e.g. `events.multi_tenant: true` in
98
+ `codegen.config.yaml`.
99
+ 2. Re-run the install to re-emit the tenancy-aware schema:
100
+ `codegen subsystem install <name> --force --force-config`.
101
+ 3. Pass `multiTenant: true` to that subsystem's `forRoot(...)`, and cut an Atlas
102
+ migration for the new `tenant_id` column(s).
103
+
104
+ With `multiTenant: true`, the enforcement sites throw a `MissingTenantIdError`
105
+ when a tenant id is required but absent (explicit `null` is allowed for
106
+ tenant-less / cross-tenant work).
107
+
108
+ ## Why ordering matters (the dependency graph)
109
+
110
+ - **bridge → events + jobs.** The bridge claims `domain_events` rows and starts
111
+ wrapper job runs in the reserved pools. Without events + jobs registered (and
112
+ the reserved pools polled) it has nothing to consume and nowhere to write.
113
+ - **observability → events/jobs/bridge/sync (optional).** It composes the read
114
+ ports of whatever siblings exist, via optional DI. Register it last so those
115
+ ports are already bound; missing siblings are simply omitted from its output.
116
+ - **auth-integrations → auth.** The integration adapters need the encryption key
117
+ + token provided by `AuthModule`.
118
+
119
+ Until the CLI enforces this graph, treat this file as the source of truth and
120
+ keep `app.module.ts` in the order above.