@pattern-stack/codegen 0.9.2 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +5 -0
- package/consumer-skills/bridge/SKILL.md +265 -0
- package/consumer-skills/codegen/SKILL.md +115 -0
- package/consumer-skills/entities/SKILL.md +111 -0
- package/consumer-skills/entities/families-and-queries.md +82 -0
- package/consumer-skills/entities/yaml-reference.md +118 -0
- package/consumer-skills/events/SKILL.md +71 -0
- package/consumer-skills/events/authoring-events.md +164 -0
- package/consumer-skills/events/typed-bus-and-outbox.md +163 -0
- package/consumer-skills/jobs/SKILL.md +66 -0
- package/consumer-skills/jobs/handler-authoring.md +236 -0
- package/consumer-skills/jobs/pools-and-ordering.md +161 -0
- package/consumer-skills/subsystems/SKILL.md +105 -0
- package/consumer-skills/subsystems/wiring-and-order.md +120 -0
- package/consumer-skills/sync/SKILL.md +134 -0
- package/consumer-skills/sync/audit-and-detection.md +302 -0
- package/consumer-skills/sync/change-sources-and-sinks.md +442 -0
- package/dist/src/cli/index.js +913 -405
- package/dist/src/cli/index.js.map +1 -1
- package/dist/src/index.js +26 -4
- package/dist/src/index.js.map +1 -1
- package/package.json +2 -1
|
@@ -0,0 +1,118 @@
|
|
|
1
|
+
<!-- managed by @pattern-stack/codegen — re-run `codegen skills install` to refresh. Edit the package source, not this vendored copy. -->
|
|
2
|
+
|
|
3
|
+
# Entity YAML reference
|
|
4
|
+
|
|
5
|
+
Every block of an `entities/<name>.yaml` file and what it generates.
|
|
6
|
+
|
|
7
|
+
## `entity:` — identity
|
|
8
|
+
|
|
9
|
+
```yaml
|
|
10
|
+
entity:
|
|
11
|
+
name: contact # REQUIRED. singular snake_case. Drives class names + table.
|
|
12
|
+
plural: contacts # optional; inferred (pluralized) if omitted
|
|
13
|
+
table: contacts # optional; defaults to the plural
|
|
14
|
+
pattern: Synced # the family — Base | Synced | Activity | Metadata | Knowledge
|
|
15
|
+
# (or an app-defined pattern). See families-and-queries.md.
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
## `fields:` — columns
|
|
19
|
+
|
|
20
|
+
Each key is a `snake_case` column; the generated TS property is `camelCase`.
|
|
21
|
+
|
|
22
|
+
```yaml
|
|
23
|
+
fields:
|
|
24
|
+
email:
|
|
25
|
+
type: string # string | integer | decimal | boolean | uuid | date | datetime | json | enum
|
|
26
|
+
required: true # NOT NULL + required in Create DTO
|
|
27
|
+
max_length: 255 # string length constraint (DB + Zod)
|
|
28
|
+
index: true # single-column index
|
|
29
|
+
status:
|
|
30
|
+
type: enum
|
|
31
|
+
choices: [active, inactive, archived] # enum members
|
|
32
|
+
metadata:
|
|
33
|
+
type: json # jsonb column
|
|
34
|
+
score:
|
|
35
|
+
type: decimal
|
|
36
|
+
nullable: true
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
Type notes:
|
|
40
|
+
- `uuid` — typically the PK and foreign keys.
|
|
41
|
+
- `enum` — requires `choices:`; generates a TS union + DB check/enum.
|
|
42
|
+
- `json` — `jsonb`; typed as `Record<string, unknown>` unless you narrow it in
|
|
43
|
+
your own code.
|
|
44
|
+
- `datetime` vs `date` — timestamp vs date-only column.
|
|
45
|
+
|
|
46
|
+
## `behaviors:` — cross-cutting columns + logic
|
|
47
|
+
|
|
48
|
+
```yaml
|
|
49
|
+
behaviors:
|
|
50
|
+
- timestamps # createdAt, updatedAt (auto-managed)
|
|
51
|
+
- soft_delete # deletedAt; queries auto-filter deleted rows; GET :id 404s on deleted
|
|
52
|
+
- user_tracking # createdBy, updatedBy
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
Behaviors compose — list any subset.
|
|
56
|
+
|
|
57
|
+
## `relationships:` — foreign keys + typed accessors
|
|
58
|
+
|
|
59
|
+
```yaml
|
|
60
|
+
relationships:
|
|
61
|
+
account:
|
|
62
|
+
type: belongs_to # belongs_to | has_many | has_one
|
|
63
|
+
target: account # the referenced entity (must have its own YAML)
|
|
64
|
+
foreign_key: account_id
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
- `belongs_to` adds the FK column on this entity.
|
|
68
|
+
- `has_many` / `has_one` are the inverse side (no column here; drives the typed
|
|
69
|
+
relation accessor + Drizzle `relations()`).
|
|
70
|
+
- Cross-entity targets must resolve at generation time — regenerate the set with
|
|
71
|
+
`codegen entity new --all`.
|
|
72
|
+
|
|
73
|
+
## `generate:` — output toggles
|
|
74
|
+
|
|
75
|
+
```yaml
|
|
76
|
+
generate:
|
|
77
|
+
writes: true # default true — emit POST/PATCH/DELETE + create/update/delete use cases
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Set `writes: false` for a read-only resource (only GET routes + read use cases).
|
|
81
|
+
|
|
82
|
+
## EAV (custom fields)
|
|
83
|
+
|
|
84
|
+
Two independent flags:
|
|
85
|
+
|
|
86
|
+
```yaml
|
|
87
|
+
# On a normal entity (e.g. opportunity): opt into custom fields
|
|
88
|
+
eav: true
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
When `eav: true`:
|
|
92
|
+
- Service gains **paired reads**: `findById` (typed entity) and
|
|
93
|
+
`findByIdWithFields` (entity + merged `fields` bag); same for `list`.
|
|
94
|
+
- `Create*` / `Update*` use cases accept `{ ...core, fields?: Record<string,
|
|
95
|
+
unknown> }` and run a transactional dual-write.
|
|
96
|
+
- Controller adds `GET /:id/with-fields`, `GET /with-fields`, and accepts the
|
|
97
|
+
`fields` bag on POST/PATCH.
|
|
98
|
+
|
|
99
|
+
```yaml
|
|
100
|
+
# On the value-table entity itself (e.g. field_value): mark it AS the EAV store
|
|
101
|
+
eav_value_table: true
|
|
102
|
+
eav_definition_table: field_definition # where keys → definition ids resolve
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
When `eav_value_table: true`:
|
|
106
|
+
- Repository gets `upsertCurrentValues(rows, tx)` (composite conflict target).
|
|
107
|
+
- Service gets `upsertFieldsTransactional(...)` and `findMergedByEntity(...)`
|
|
108
|
+
with internal definition-id resolution.
|
|
109
|
+
- The module auto-imports the definition-table module so DI resolves without
|
|
110
|
+
consumer wiring.
|
|
111
|
+
|
|
112
|
+
The EAV dual-write is coordinated by the **use case** inside `db.transaction` —
|
|
113
|
+
services stay single-domain (see the layer rules in the `entities` L0 skill).
|
|
114
|
+
|
|
115
|
+
## `queries:` — declarative finders
|
|
116
|
+
|
|
117
|
+
See `families-and-queries.md` for the full `queries:` block (column finders,
|
|
118
|
+
unique, ordered, filtered+paginated search).
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: events
|
|
3
|
+
description: Load when authoring a domain event, publishing one from a use case, or subscribing to one in a project that ran `codegen subsystem install events`. Triggers include `events/*.yaml` files, the generated `TypedEventBus` facade, injecting `TYPED_EVENT_BUS` / `EVENT_BUS`, `publish(...)` inside a Drizzle transaction (the outbox), `subscribe(...)`, the `domain_events` table, and event directions / pools.
|
|
4
|
+
allowed-tools: Read, Write, Edit, Glob, Grep, Bash
|
|
5
|
+
user-invocable: false
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
<!-- managed by @pattern-stack/codegen — re-run `codegen skills install` to refresh. Edit the package source, not this vendored copy. -->
|
|
9
|
+
|
|
10
|
+
# Events
|
|
11
|
+
|
|
12
|
+
The events subsystem is the transactional event backbone vendored into your app by `codegen subsystem install events`. You declare each event type as a YAML file; codegen generates a typed `TypedEventBus` facade, a discriminated union of every event, Zod payload schemas, and a runtime registry. You publish events inside the same database transaction as your domain write (the outbox pattern); a background loop drains them and delivers to subscribers.
|
|
13
|
+
|
|
14
|
+
The vendored code lives under `<paths.subsystems>/events/` (default `src/shared/subsystems/events/`), imported as `@shared/subsystems/events`. The generated files live under `<paths.subsystems>/events/generated/` and are reproduced from `events/*.yaml` on every `codegen` run — do not hand-edit them.
|
|
15
|
+
|
|
16
|
+
## Mental model
|
|
17
|
+
|
|
18
|
+
**An event is an immutable fact** — "contact was created", "Stripe webhook arrived", "opportunity moved to `won`". It has no lifecycle beyond being delivered. **A job is the stateful work** that reacts to a fact. If you are tempted to put `status`, `attempts`, `retry_count`, or `scope` on an event, stop — you want a job (see the `jobs` skill). The event is the trigger; the job is the work.
|
|
19
|
+
|
|
20
|
+
### The outbox
|
|
21
|
+
|
|
22
|
+
Publishing an event is an `INSERT` into the `domain_events` table, performed **inside the same transaction as your domain write**. Either both commit or neither does — no phantom events, no domain drift if the process crashes between commit and publish. A separate polling loop drains pending rows and dispatches them to subscribers. This is strictly stronger than `await commit(); await publish()`.
|
|
23
|
+
|
|
24
|
+
The single most important rule: **always pass the transaction (`tx`) when publishing from a use case that also writes domain state.** Dropping it silently detaches the event from the transaction.
|
|
25
|
+
|
|
26
|
+
### Directions and pools
|
|
27
|
+
|
|
28
|
+
Every domain event declares a **direction**, which determines the lane it drains through:
|
|
29
|
+
|
|
30
|
+
| direction | carries | example |
|
|
31
|
+
|---|---|---|
|
|
32
|
+
| `inbound` | external → us (webhooks, pub/sub, inbound email) | `stripe_payment_received` |
|
|
33
|
+
| `change` | internal domain mutations (drive projections/reactions) | `contact_created` |
|
|
34
|
+
| `outbound` | us → external (webhooks fired, sync pushes) | `webhook_outbound_contact_sync` |
|
|
35
|
+
|
|
36
|
+
Direction is a **routing** concern, not a payload concern — two events with identical payloads can have different directions. Each direction drains through its own reserved lane so a slow outbound handler can't stall internal change-event propagation.
|
|
37
|
+
|
|
38
|
+
There is also an `audit` **tier** (orthogonal to direction) for observational facts that should live in the outbox but must never spawn work — see `authoring-events.md`.
|
|
39
|
+
|
|
40
|
+
### Typed facade vs. raw port
|
|
41
|
+
|
|
42
|
+
- `IEventBus` (token `EVENT_BUS`) is the narrow underlying port: `publish`, `publishMany`, `subscribe`. It knows nothing about your specific event types.
|
|
43
|
+
- `TypedEventBus` (token `TYPED_EVENT_BUS`) is the generated, injectable wrapper. Its `publish<T>()` enforces the typed payload for `T` and stamps `metadata.pool` / `metadata.direction` / `metadata.version` from the registry. **Use `TypedEventBus` in new code.** The raw port stays available for forwarders that publish types not in the registry.
|
|
44
|
+
|
|
45
|
+
## Routing table
|
|
46
|
+
|
|
47
|
+
| For this task | Read |
|
|
48
|
+
|---|---|
|
|
49
|
+
| Declaring an `events/*.yaml`, payload types, directions, `tier: audit`, entity `emits:` | `authoring-events.md` |
|
|
50
|
+
| Publishing inside a transaction, the outbox, idempotency, subscribing, wiring `EventsModule` | `typed-bus-and-outbox.md` |
|
|
51
|
+
|
|
52
|
+
To run a durable background job *when an event fires*, that is the Event-to-Job Bridge — see the `bridge` skill. To do the work directly (not via an event), see the `jobs` skill.
|
|
53
|
+
|
|
54
|
+
## Non-obvious rules
|
|
55
|
+
|
|
56
|
+
- **Always pass `tx` to `publish` inside a use-case transaction.** It is the entire outbox guarantee. There is no type-level enforcement of this yet — it is a discipline.
|
|
57
|
+
- **Events have no lifecycle; jobs do.** The `status` column on `domain_events` (`pending | processed | failed`) is delivery state for the drain loop, not a domain state machine.
|
|
58
|
+
- **Use `TypedEventBus.publish<'type'>(...)` once the type is generated.** Misspelled event names and wrong payload shapes become compile errors.
|
|
59
|
+
- **Subscribers must be fast.** A subscriber that makes an HTTP call blocks the drain batch (it dispatches ~50 rows serially). Heavy reactions belong in a job — subscribe, then enqueue.
|
|
60
|
+
- **The event `id` is the idempotency key.** For replays/backfills, derive the id deterministically from the source event (e.g. Stripe's `evt_...`) so a re-insert is a no-op rather than a duplicate.
|
|
61
|
+
- **Regenerate after editing YAML.** Re-run `codegen` (e.g. `codegen entity new --all`) after touching any `events/*.yaml` to refresh the generated facade, union, schemas, and registry.
|
|
62
|
+
|
|
63
|
+
## Do not
|
|
64
|
+
|
|
65
|
+
- Do not put job-style fields on an event (`status: running`, `attempts`, `scope`, `parent_id`). Those belong on a job.
|
|
66
|
+
- Do not drop the `tx` argument in `publish(...)` inside a transaction.
|
|
67
|
+
- Do not collapse the three directions into one pool — lane isolation is the point.
|
|
68
|
+
- Do not couple two services with a direct method call when the second merely reacts to a state change in the first. Publish a `change` event from the first, subscribe (or bridge a job) from the second.
|
|
69
|
+
- Do not do heavy I/O directly in a subscriber. Enqueue a job instead.
|
|
70
|
+
- Do not hand-edit anything under `<paths.subsystems>/events/generated/`. It is regenerated from `events/*.yaml`.
|
|
71
|
+
- Do not route events through user pools (`batch`, `interactive`) — events only ever drain through the reserved `events_*` lanes.
|
|
@@ -0,0 +1,164 @@
|
|
|
1
|
+
<!-- managed by @pattern-stack/codegen — re-run `codegen skills install` to refresh. Edit the package source, not this vendored copy. -->
|
|
2
|
+
|
|
3
|
+
# Authoring Events
|
|
4
|
+
|
|
5
|
+
How to declare an event in YAML, what each field means, what gets generated, and when to reach for the `audit` tier. Read the `events` `SKILL.md` first for the outbox/direction mental model. For publishing and subscribing, see `typed-bus-and-outbox.md`.
|
|
6
|
+
|
|
7
|
+
## One YAML file per event
|
|
8
|
+
|
|
9
|
+
Events live in `events/` at your repo root, sibling to `entities/`. One file per type; the filename matches the `type` field (snake_case).
|
|
10
|
+
|
|
11
|
+
```yaml
|
|
12
|
+
# events/contact_created.yaml
|
|
13
|
+
type: contact_created
|
|
14
|
+
direction: change
|
|
15
|
+
aggregate: contact # must match an entity name
|
|
16
|
+
version: 1
|
|
17
|
+
payload:
|
|
18
|
+
contact_id: { type: uuid }
|
|
19
|
+
account_id: { type: uuid, nullable: true }
|
|
20
|
+
created_by: { type: uuid }
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
```yaml
|
|
24
|
+
# events/stripe_payment_received.yaml
|
|
25
|
+
type: stripe_payment_received
|
|
26
|
+
direction: inbound
|
|
27
|
+
source: stripe # inbound: logical origin
|
|
28
|
+
version: 1
|
|
29
|
+
description: Stripe charge.succeeded webhook, post-signature-verification.
|
|
30
|
+
payload:
|
|
31
|
+
event_id: { type: string, description: "Stripe event id (evt_...)" }
|
|
32
|
+
customer_id: { type: string }
|
|
33
|
+
amount_cents: { type: number }
|
|
34
|
+
currency: { type: string }
|
|
35
|
+
received_at: { type: date }
|
|
36
|
+
retry:
|
|
37
|
+
attempts: 5
|
|
38
|
+
backoff: exponential
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
```yaml
|
|
42
|
+
# events/webhook_outbound_contact_sync.yaml
|
|
43
|
+
type: webhook_outbound_contact_sync
|
|
44
|
+
direction: outbound
|
|
45
|
+
destination: crm # outbound: logical target
|
|
46
|
+
aggregate: contact
|
|
47
|
+
payload:
|
|
48
|
+
contact_id: { type: uuid }
|
|
49
|
+
operation: { type: string } # "create" | "update" | "delete"
|
|
50
|
+
occurred_at: { type: date }
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
After editing any `events/*.yaml`, re-run codegen (e.g. `codegen entity new --all`) to regenerate the typed artifacts.
|
|
54
|
+
|
|
55
|
+
## Field reference
|
|
56
|
+
|
|
57
|
+
| Field | Required | Notes |
|
|
58
|
+
|---|---|---|
|
|
59
|
+
| `type` | yes | Unique snake_case business key; matches the filename. Becomes the discriminator and the `publish<T>()` key. |
|
|
60
|
+
| `direction` | for `tier: domain` | `inbound \| change \| outbound`. Drives the default pool. **Omit for `tier: audit`.** |
|
|
61
|
+
| `tier` | no | `domain` (default) or `audit`. Audit events are outbox-only and never spawn jobs — see below. |
|
|
62
|
+
| `aggregate` | for `change` | The owning entity name. Required for `change` events; optional elsewhere. Lets you query/replay events for a given entity id. |
|
|
63
|
+
| `source` | inbound only | Logical origin ("stripe", "email"). |
|
|
64
|
+
| `destination` | outbound only | Logical target ("crm", "slack"). |
|
|
65
|
+
| `payload` | yes | Map of snake_case keys → field specs. Keys become camelCase TS props. |
|
|
66
|
+
| `pool` | no | Override the direction's default pool. Only valid within the same category (a `change` event can't opt into `events_inbound`). User pools are never valid. Rarely needed — if you reach for it, revisit the direction. **Must be omitted for `tier: audit`.** |
|
|
67
|
+
| `retry` | no | `{ attempts, backoff }`; hints surfaced to the bus. |
|
|
68
|
+
| `version` | no | Integer, defaults to 1. Stamped on every publish; multi-version coexistence is not exercised yet. |
|
|
69
|
+
|
|
70
|
+
### Payload field types
|
|
71
|
+
|
|
72
|
+
`uuid | string | number | boolean | date | json | array`.
|
|
73
|
+
|
|
74
|
+
- `nullable: true` makes the TS prop `T | null`.
|
|
75
|
+
- `array` requires an `items:` scalar type (`items: uuid | string | number | boolean | date`) and emits `T[]` + `z.array(T)`.
|
|
76
|
+
- `json` means an arbitrary JSON object (`Record<string, unknown>`). **Do not use `json` for array-shaped data** — Zod will reject it at the publish boundary.
|
|
77
|
+
|
|
78
|
+
### Default pool from direction (domain tier)
|
|
79
|
+
|
|
80
|
+
| direction | default pool |
|
|
81
|
+
|---|---|
|
|
82
|
+
| `inbound` | `events_inbound` |
|
|
83
|
+
| `change` | `events_change` |
|
|
84
|
+
| `outbound` | `events_outbound` |
|
|
85
|
+
|
|
86
|
+
## Tier: domain vs. audit
|
|
87
|
+
|
|
88
|
+
`tier` classifies the *kind* of fact and controls whether a job can be bound to the event.
|
|
89
|
+
|
|
90
|
+
| tier | direction | pool | can a job trigger on it? |
|
|
91
|
+
|---|---|---|---|
|
|
92
|
+
| `domain` (default) | required | from direction | yes |
|
|
93
|
+
| `audit` | omitted | null | no — codegen rejects it |
|
|
94
|
+
|
|
95
|
+
Use `audit` for high-volume **observational** events that should exist in the outbox (so you can query/replay them) but must never spawn downstream work. The motivating case: a polling CRM sync emitting one "I scanned this row" event per record would otherwise queue thousands of inert bridge jobs per run.
|
|
96
|
+
|
|
97
|
+
```yaml
|
|
98
|
+
# events/crm_sync_completed.yaml
|
|
99
|
+
type: crm_sync_completed
|
|
100
|
+
tier: audit # outbox-only; not bridge-eligible
|
|
101
|
+
aggregate: integration # still scoped for query-by-id
|
|
102
|
+
version: 1
|
|
103
|
+
description: A CRM sync run finished. Observational — no domain state changed.
|
|
104
|
+
payload:
|
|
105
|
+
integration_id: { type: uuid }
|
|
106
|
+
provider: { type: string }
|
|
107
|
+
counts: { type: json }
|
|
108
|
+
duration_ms: { type: number }
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
For an `audit` event the generated registry entry has `pool: null` and `direction: null`; both columns land NULL. In-process subscribers (`subscribe(...)`) may still listen to audit events — only the bridge refuses to bind jobs to them.
|
|
112
|
+
|
|
113
|
+
**The discipline:** if a well-behaved consumer would write another row or enqueue work, it is a `domain` event; if it would only bump a counter or update a dashboard, it is `audit`.
|
|
114
|
+
|
|
115
|
+
Codegen hard-errors on misuse:
|
|
116
|
+
- `tier: audit` with a `pool` → error naming the event.
|
|
117
|
+
- `tier: audit` with a `direction` → error naming the event.
|
|
118
|
+
- A job whose `@JobHandler.triggers` points at an audit event → `AuditEventTriggerError`. Use a domain event or remove the trigger.
|
|
119
|
+
|
|
120
|
+
## Entity `events:` block is sugar for change events
|
|
121
|
+
|
|
122
|
+
An entity's YAML may keep an `events:` block. At parse time each entry desugars into an equivalent `events/<name>.yaml` with `direction: change` and `aggregate: <entity>`. Both paths produce the same registry entry — inline blocks are convenience, not a second source of truth.
|
|
123
|
+
|
|
124
|
+
## Entity `emits:` for typed auto-emission
|
|
125
|
+
|
|
126
|
+
An entity can declare which change events its generated use cases emit:
|
|
127
|
+
|
|
128
|
+
```yaml
|
|
129
|
+
# entities/contact.yaml
|
|
130
|
+
emits:
|
|
131
|
+
- contact_created
|
|
132
|
+
- contact_updated
|
|
133
|
+
- contact_deleted
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
Each entry must resolve to an `events/<type>.yaml` with `direction: change` and `aggregate: contact`, or codegen hard-errors. With `emits:` present, generated use cases call the typed facade inside the domain transaction:
|
|
137
|
+
|
|
138
|
+
```ts
|
|
139
|
+
async execute(input: CreateContactInput): Promise<Contact> {
|
|
140
|
+
return this.db.transaction(async (tx) => {
|
|
141
|
+
const contact = await this.contacts.create(input, tx);
|
|
142
|
+
await this.events.publish('contact_created', contact.id, {
|
|
143
|
+
contactId: contact.id,
|
|
144
|
+
accountId: contact.accountId,
|
|
145
|
+
createdBy: input.actorId,
|
|
146
|
+
}, { tx });
|
|
147
|
+
return contact;
|
|
148
|
+
});
|
|
149
|
+
}
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
## What gets generated
|
|
153
|
+
|
|
154
|
+
Five files under `<paths.subsystems>/events/generated/`, all reproduced from your YAML — never hand-edited:
|
|
155
|
+
|
|
156
|
+
| File | Contents |
|
|
157
|
+
|---|---|
|
|
158
|
+
| `types.ts` | Per-event interfaces, the `AppDomainEvent` discriminated union, and helpers `EventTypeName`, `EventOfType<T>`, `PayloadOfType<T>`. |
|
|
159
|
+
| `schemas.ts` | A Zod schema per payload, plus a keyed map for boundary validation. |
|
|
160
|
+
| `registry.ts` | `eventRegistry` — the runtime metadata map (direction, pool, aggregate, version, retry) read by the facade, the bridge, and startup validation. |
|
|
161
|
+
| `bus.ts` | The `TypedEventBus` facade with typed `publish<T>()` / `subscribe<T>()`. |
|
|
162
|
+
| `index.ts` | Re-export surface. |
|
|
163
|
+
|
|
164
|
+
The typed payload flows end to end: declaring `payload.contact_id: { type: uuid }` makes `PayloadOfType<'contact_created'>` require `contactId: string`, and `publish('contact_created', id, { ... })` typechecks the object against it.
|
|
@@ -0,0 +1,163 @@
|
|
|
1
|
+
<!-- managed by @pattern-stack/codegen — re-run `codegen skills install` to refresh. Edit the package source, not this vendored copy. -->
|
|
2
|
+
|
|
3
|
+
# Publishing, the Outbox, and Subscribing
|
|
4
|
+
|
|
5
|
+
How to publish events transactionally, how the outbox drain works, how to subscribe, idempotency, and how to wire `EventsModule` into your app. Read the `events` `SKILL.md` first for the mental model and `authoring-events.md` for the YAML shape.
|
|
6
|
+
|
|
7
|
+
## Publishing inside a transaction
|
|
8
|
+
|
|
9
|
+
Every use case that mutates domain state and emits an event **must pass the transaction** so the event row is part of the same atomic write. Inject the typed facade:
|
|
10
|
+
|
|
11
|
+
```ts
|
|
12
|
+
import { Inject, Injectable } from '@nestjs/common';
|
|
13
|
+
import { TypedEventBus, TYPED_EVENT_BUS } from '@shared/subsystems/events';
|
|
14
|
+
|
|
15
|
+
@Injectable()
|
|
16
|
+
export class CreateContactUseCase {
|
|
17
|
+
constructor(
|
|
18
|
+
@Inject(TYPED_EVENT_BUS) private readonly events: TypedEventBus,
|
|
19
|
+
private readonly db: DrizzleClient,
|
|
20
|
+
private readonly contacts: ContactRepository,
|
|
21
|
+
) {}
|
|
22
|
+
|
|
23
|
+
async execute(input: CreateContactInput): Promise<Contact> {
|
|
24
|
+
return this.db.transaction(async (tx) => {
|
|
25
|
+
const contact = await this.contacts.create(input, tx);
|
|
26
|
+
await this.events.publish(
|
|
27
|
+
'contact_created',
|
|
28
|
+
contact.id,
|
|
29
|
+
{
|
|
30
|
+
contactId: contact.id,
|
|
31
|
+
accountId: contact.accountId,
|
|
32
|
+
createdBy: input.actorId,
|
|
33
|
+
},
|
|
34
|
+
{ tx }, // ← the outbox guarantee
|
|
35
|
+
);
|
|
36
|
+
return contact;
|
|
37
|
+
});
|
|
38
|
+
}
|
|
39
|
+
}
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
`TypedEventBus.publish(type, aggregateId, payload, opts?)`:
|
|
43
|
+
- Validates the payload against the generated Zod schema at the boundary (skippable with `CODEGEN_EVENT_VALIDATE=off`).
|
|
44
|
+
- Stamps `id`, `occurredAt`, and `metadata.pool` / `metadata.direction` / `metadata.version` from the registry.
|
|
45
|
+
- Inserts one `domain_events` row using `opts.tx` if provided, else the top-level connection.
|
|
46
|
+
|
|
47
|
+
**Dropping `tx` silently detaches the event.** The backend uses `tx ?? db`, so without `tx` the insert runs outside your domain transaction — the domain write can commit while the event insert fails independently. There is no compile-time guard; treat passing `tx` as non-negotiable.
|
|
48
|
+
|
|
49
|
+
### Raw `EVENT_BUS` for untyped publishes
|
|
50
|
+
|
|
51
|
+
The raw port stays available for forwarders that publish types not in the registry (e.g. proxying an external source):
|
|
52
|
+
|
|
53
|
+
```ts
|
|
54
|
+
import { EVENT_BUS, type IEventBus } from '@shared/subsystems/events';
|
|
55
|
+
|
|
56
|
+
@Inject(EVENT_BUS) private readonly bus: IEventBus;
|
|
57
|
+
|
|
58
|
+
await this.bus.publish(
|
|
59
|
+
{ id, type, aggregateId, aggregateType, payload, occurredAt },
|
|
60
|
+
tx,
|
|
61
|
+
);
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
New code should prefer `TypedEventBus`.
|
|
65
|
+
|
|
66
|
+
## The outbox table and drain loop
|
|
67
|
+
|
|
68
|
+
`domain_events` is the outbox. Key columns: `id` (UUID, the idempotency key), `type`, `aggregate_id`, `aggregate_type`, `payload` (jsonb), `occurred_at`, `processed_at`, `status` (`pending | processed | failed`), `error`, `metadata`, and first-class `pool` / `direction` columns (populated from `metadata` at insert so the drain can filter by lane without unpacking JSON).
|
|
69
|
+
|
|
70
|
+
The drain loop (Drizzle backend):
|
|
71
|
+
1. Claims a batch (default 50) of `pending` rows with `FOR UPDATE SKIP LOCKED`, optionally filtered by pool, ordered by `occurred_at ASC`. `SKIP LOCKED` lets multiple worker processes drain concurrently without double-dispatching.
|
|
72
|
+
2. For each row, invokes every registered handler for `event.type`.
|
|
73
|
+
3. On success: `status='processed'`, `processed_at=now()` — in the same transaction that locked the row, so nothing can strand half-claimed.
|
|
74
|
+
4. On failure: retries up to 3 attempts; if still failing, `status='failed'` with the error. No automatic retry after that — a permanently broken handler does not burn a worker forever.
|
|
75
|
+
|
|
76
|
+
Default poll interval is 1s, so there is a ~1s latency floor. There is no stale-event sweeper — it is unnecessary because the lock lifetime equals the dispatch transaction.
|
|
77
|
+
|
|
78
|
+
## Subscribing
|
|
79
|
+
|
|
80
|
+
```ts
|
|
81
|
+
import { TypedEventBus, TYPED_EVENT_BUS } from '@shared/subsystems/events';
|
|
82
|
+
|
|
83
|
+
@Inject(TYPED_EVENT_BUS) private readonly events: TypedEventBus;
|
|
84
|
+
|
|
85
|
+
this.events.subscribe('contact_created', async (event) => {
|
|
86
|
+
// event is narrowed to ContactCreatedEvent — event.payload.contactId is typed
|
|
87
|
+
await this.readModel.upsert(event.payload.contactId);
|
|
88
|
+
});
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
`subscribe<T>()` returns an unsubscribe function. Subscriptions are **per-process** in the Drizzle backend — a subscriber registered in process A does not receive events drained in process B. In a multi-process deployment, each process runs its own subscribers.
|
|
92
|
+
|
|
93
|
+
**Keep subscribers fast.** The drain dispatches a batch serially; a subscriber doing HTTP or other slow I/O stalls the whole batch. For anything slow, durable, or that needs retry, do not work inline — enqueue a job. For durable async fanout from an event to a job, use the `bridge` skill; for fire-and-forget cheap reactions (metrics, cache busts), an in-process subscriber is fine.
|
|
94
|
+
|
|
95
|
+
## Idempotency
|
|
96
|
+
|
|
97
|
+
The event `id` is the dedup token. Handlers can redeliver (a process can crash between dispatch and the `processed` update), so:
|
|
98
|
+
|
|
99
|
+
- **Make handlers idempotent** — safe to call twice with the same `event.id`. A common pattern is a small "seen ids" table the handler checks first.
|
|
100
|
+
- **For inbound webhooks, reuse the source system's id.** If Stripe sends `evt_123` twice, deriving the outbox `id` from `evt_123` means the second insert hits `ON CONFLICT (id) DO NOTHING` instead of creating a duplicate.
|
|
101
|
+
- **Never regenerate `id` on replay/backfill** — compute it deterministically from the source event.
|
|
102
|
+
|
|
103
|
+
## Ordering
|
|
104
|
+
|
|
105
|
+
- Same aggregate, single worker: FIFO (`ORDER BY occurred_at ASC`). With multiple workers, same-aggregate ordering holds only probabilistically.
|
|
106
|
+
- Across aggregates: not strictly ordered — treat it as roughly wall-clock.
|
|
107
|
+
|
|
108
|
+
If a handler depends on strict cross-aggregate ordering, reshape it to tolerate independent timelines.
|
|
109
|
+
|
|
110
|
+
## Wiring `EventsModule`
|
|
111
|
+
|
|
112
|
+
Register once in `AppModule`; it is `global: true`, so entity modules need not import it individually:
|
|
113
|
+
|
|
114
|
+
```ts
|
|
115
|
+
import { EventsModule } from '@shared/subsystems/events';
|
|
116
|
+
|
|
117
|
+
@Module({
|
|
118
|
+
imports: [
|
|
119
|
+
DatabaseModule,
|
|
120
|
+
EventsModule.forRoot({ backend: 'drizzle' }),
|
|
121
|
+
// ...
|
|
122
|
+
],
|
|
123
|
+
})
|
|
124
|
+
export class AppModule {}
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
`forRoot` options:
|
|
128
|
+
- `backend: 'drizzle' | 'memory' | 'redis'` — match `events.backend` in your config. Tests override to `'memory'`.
|
|
129
|
+
- `multiTenant: true` — opt-in multi-tenancy (see below).
|
|
130
|
+
- `pools: ['events_change']` — restrict this process's drain loop to specific lanes. A common split is one process per direction so a slow outbound handler cannot stall change-event propagation. Undefined drains all lanes.
|
|
131
|
+
|
|
132
|
+
### Backend notes
|
|
133
|
+
|
|
134
|
+
- **`drizzle`** (default) — Postgres outbox; transactional, crash-safe; ~1s polling floor.
|
|
135
|
+
- **`memory`** — synchronous, in-process, for tests. `publish` dispatches immediately (no `tx` semantics); exposes `publishedEvents[]`, `publishedEventsForPool()`, `publishedEventsForDirection()`, and `clear()` for assertions. Because dispatch is synchronous, tests need no timers.
|
|
136
|
+
- **`redis`** — present but not a scaffold default. **It does not participate in the Drizzle transaction** — passing `tx` is a silent no-op, so you lose the outbox guarantee. Only reach for it after measuring an actual throughput bottleneck (the Drizzle outbox tops out around ~1000 events/s).
|
|
137
|
+
|
|
138
|
+
### Multi-tenancy
|
|
139
|
+
|
|
140
|
+
Set `events.multi_tenant: true` in config, re-run `codegen subsystem install events --force` to re-emit the schema with a `tenant_id` column (then cut a migration), and pass `multiTenant: true` to `EventsModule.forRoot(...)`:
|
|
141
|
+
|
|
142
|
+
```ts
|
|
143
|
+
EventsModule.forRoot({ backend: 'drizzle', multiTenant: true });
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
When on, `publish` throws `MissingTenantIdError` (naming the event type) if `opts.metadata.tenantId` is absent. Explicit `null` is permitted for tenant-less background events. Keep the config flag and the module option in agreement.
|
|
147
|
+
|
|
148
|
+
## Testing with events
|
|
149
|
+
|
|
150
|
+
Swap the backend to memory and assert on what was published:
|
|
151
|
+
|
|
152
|
+
```ts
|
|
153
|
+
Test.createTestingModule({
|
|
154
|
+
imports: [EventsModule.forRoot({ backend: 'memory' })],
|
|
155
|
+
});
|
|
156
|
+
|
|
157
|
+
// in a test:
|
|
158
|
+
expect(bus.publishedEvents).toContainEqual(
|
|
159
|
+
expect.objectContaining({ type: 'contact_created' }),
|
|
160
|
+
);
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
Call `bus.clear()` in `beforeEach`. For Drizzle integration tests, `DrizzleEventBus.drainOnce()` runs exactly one drain cycle so you do not have to sleep past the poll interval.
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: jobs
|
|
3
|
+
description: Load when authoring a `@JobHandler` class, kicking off a background job from a use case, or configuring job pools in a project that ran `codegen subsystem install jobs`. Triggers include `@JobHandler`, `JobContext`, `ctx.step` / `ctx.spawnChild`, injecting `JOB_ORCHESTRATOR` / `JOB_RUN_SERVICE`, registering `JobsDomainModule` / `JobWorkerModule`, and editing the `jobs:` block in `codegen.config.yaml`.
|
|
4
|
+
allowed-tools: Read, Write, Edit, Glob, Grep, Bash
|
|
5
|
+
user-invocable: false
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
<!-- managed by @pattern-stack/codegen — re-run `codegen skills install` to refresh. Edit the package source, not this vendored copy. -->
|
|
9
|
+
|
|
10
|
+
# Jobs
|
|
11
|
+
|
|
12
|
+
The jobs subsystem is the durable background-work engine vendored into your app by `codegen subsystem install jobs`. It gives you a way to run work asynchronously, retry it, scope it to a domain entity, cancel it as a tree, and resume it after a crash without redoing finished steps. You author jobs as plain TypeScript classes decorated with `@JobHandler`; the runtime handles claiming, retry, memoization, and lifecycle.
|
|
13
|
+
|
|
14
|
+
The vendored code lives under `<paths.subsystems>/jobs/` (default `src/shared/subsystems/jobs/`) and is imported as `@shared/subsystems/jobs`. Do not hand-edit it — it is managed by the package.
|
|
15
|
+
|
|
16
|
+
## Mental model
|
|
17
|
+
|
|
18
|
+
A job is **stateful work**. Each execution is a row in the `job_run` table — a durable state machine you can query, cancel, and replay. This is the sharp line versus events: an *event* is an immutable fact ("contact was created"); a *job* is the work that reacts to it. If you catch yourself wanting `status`, `attempts`, or a retry policy on something, you want a job. See the sibling `events` skill for that distinction.
|
|
19
|
+
|
|
20
|
+
Three tables back the system, one concept each:
|
|
21
|
+
|
|
22
|
+
| Table | Meaning |
|
|
23
|
+
|---|---|
|
|
24
|
+
| `job` | One row per registered handler type. Materialized from your `@JobHandler` metadata at app boot. |
|
|
25
|
+
| `job_run` | One row per execution. The durable state machine: queryable by scope, cancellable as a tree. |
|
|
26
|
+
| `job_step` | One row per checkpoint inside a run. Powers memoization (skip already-done work on retry). |
|
|
27
|
+
|
|
28
|
+
There is **one** claim mechanism: a worker polls `job_run` directly with `SELECT ... FOR UPDATE SKIP LOCKED`. There is no separate queue table and no separate "enqueue" port. You start work with `IJobOrchestrator.start(...)`; the worker discovers the row on its next poll.
|
|
29
|
+
|
|
30
|
+
Run hierarchy: a child run carries `parent_run_id` and `root_run_id`. Cancelling a run cascades to its tree via `root_run_id` according to each child's close policy.
|
|
31
|
+
|
|
32
|
+
### State machine
|
|
33
|
+
|
|
34
|
+
```
|
|
35
|
+
pending → running → { completed | failed | timed_out | canceled }
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
"Scheduled for later" is not a separate state — it is `status='pending'` with `run_at` in the future; the claim query simply skips it until `run_at` passes.
|
|
39
|
+
|
|
40
|
+
## Routing table
|
|
41
|
+
|
|
42
|
+
| For this task | Read |
|
|
43
|
+
|---|---|
|
|
44
|
+
| Writing a `@JobHandler`, using `JobContext`, `ctx.step`, spawning children, scope, plugging into a use case | `handler-authoring.md` |
|
|
45
|
+
| Choosing/configuring pools, concurrency, ordering guarantees, the `jobs:` config block, embedded vs. standalone workers | `pools-and-ordering.md` |
|
|
46
|
+
|
|
47
|
+
For running a job *in response to a domain event*, that is the Event-to-Job Bridge — see the `bridge` skill. You declare triggers on the same `@JobHandler` decorator; this skill covers everything else about the handler.
|
|
48
|
+
|
|
49
|
+
## Non-obvious rules
|
|
50
|
+
|
|
51
|
+
- **Jobs are TypeScript classes, not YAML.** There is no jobs-as-YAML codegen. You write a `@JobHandler` class; the package ships the orchestration around it.
|
|
52
|
+
- **A `@JobHandler` class must also be a registered NestJS provider.** The decorator registers the class with the job registry for orchestration; it does NOT register it in Nest's DI container. Add it to the `providers` of its owning module, or the worker throws an unresolvable-provider error when it tries to run it.
|
|
53
|
+
- **`step_id` must be stable across replays.** Hardcode it (`'pull_emails'`) or derive it deterministically from input. Never `Date.now()`, never random. Memoization is keyed on `(job_run_id, step_id)`; an unstable id defeats it.
|
|
54
|
+
- **The default pool is `batch`.** `interactive` exists but must be opted into explicitly.
|
|
55
|
+
- **The `events_inbound` / `events_change` / `events_outbound` pools are reserved.** A `@JobHandler({ pool: 'events_*' })` throws `ReservedPoolViolationError` at app boot. Those lanes belong to the event/bridge machinery. To run a job when an event fires, use `@JobHandler.triggers` (the `bridge` skill), not a reserved pool.
|
|
56
|
+
- **Concurrency and dedupe are different things.** Dedupe collapses duplicate enqueues (returns the existing run id, no new row). Concurrency lets the new row exist but gates when it is claimed. Both are set once on the decorator, not per call site.
|
|
57
|
+
- **`ctx.waitFor` / `ctx.signal` / `ctx.sleep` do not exist.** If you need a delay, use `ctx.spawnChild(type, input, { runAt })`. If you need to pause for external input, split the work across parent + child runs.
|
|
58
|
+
|
|
59
|
+
## Do not
|
|
60
|
+
|
|
61
|
+
- Do not look for an `IJobQueue` or a `job_queue` table — they do not exist. Start work with `IJobOrchestrator.start(type, input, opts)`.
|
|
62
|
+
- Do not target a reserved `events_*` pool from a `@JobHandler`. It fails at boot.
|
|
63
|
+
- Do not use `Date.now()` or randomness for a `step_id`.
|
|
64
|
+
- Do not wrap `ctx.spawnChild` inside a `ctx.step` — a child run is its own memoization root.
|
|
65
|
+
- Do not hand-edit anything under `<paths.subsystems>/jobs/`. It is vendored from the package.
|
|
66
|
+
- Do not reach for a job when the caller is waiting synchronously on the result — that is a use case, not a job. Jobs are durable because they are asynchronous.
|