@pattern-stack/codegen 0.12.2 → 0.13.0

package/CHANGELOG.md

@@ -4,6 +4,72 @@ All notable changes to this project will be documented in this file.
 
 ## [Unreleased]
 
+## [0.13.0] — 2026-05-31
+
+Track D round-2/3 — the integration codegen now emits the **full** integration
+layer. Where 0.12.x stopped at the read side (provider module, adapter scaffold,
+registry, typed views), 0.13.0 adds the **module assembly** (the write/run side —
+RFC-0002) and reshapes the read body into the **`IncrementalRead` primitive**
+(RFC-0003). After this release the author fills only the irreducible vendor seam:
+the `enumerate` / `hydrate` / `toCanonical` read methods and any non-generic sink
+write logic.
+
+Core and the four surface packages (`codegen-{mail,calendar,transcript,crm}`)
+**release together** — the surfaces carry the BREAKING port change below and
+require the matching core (peer dep `^0.13.0`).
+
+### BREAKING
+
+- **Surface ports declare `changeSources`, not `sources`.** The four surface
+  ports (`MailPort` / `CalendarPort` / `TranscriptPort` / `CrmPort`) now require
+  `readonly changeSources: Record<string, IChangeSource<unknown>>` — the
+  per-entity change sources the adapter *contributes*, keyed by entity name —
+  instead of the old `readonly sources: IEntityChangeSourceRegistry`. The folded,
+  entity-keyed registry (`<SURFACE>_ENTITY_SOURCES`) is now the **surface
+  module's** concern: the surface aggregator folds every provider's
+  `changeSources` into it, and entity-agnostic consumers read it at runtime. This
+  drops a vestigial registry injection from the adapter (it was read by nothing
+  and formed a latent DI cycle — RFC-0002 §3 E0), making the adapter
+  standalone-importable. The four surface packages bump to **0.2.0**; their peer
+  dep on `@pattern-stack/codegen` moves to **^0.13.0**. Conformance helpers
+  (`assert<Surface>Adapter`) check `changeSources` membership accordingly.
+
+### Added
+
+- **Integration module assembly emission (RFC-0002).** Per `(surface, provider,
+  entity-with-surface)`, codegen now emits the assembly that turns a registry of
+  change sources into a runnable integration per entity:
+  - `<surface>/modules/<provider>/<entity>-integration.module.ts` — `@generated`
+    per-entity feature module binding `INTEGRATION_CHANGE_SOURCE`
+    (= `adapter.changeSources['<entity>']`, Option A) + `INTEGRATION_SINK`, a
+    local `ExecuteIntegrationUseCase`, and a uniquely-tokened handle
+    (`<ENTITY>_INTEGRATION_USE_CASE__<PROVIDER>`) a trigger can grab.
+  - `<surface>/sinks/<entity>.sink.ts` — emit-once **default sink** scaffold
+    (`// <CODEGEN-SCAFFOLD-V1>`) over the entity's generated `Integrated`
+    repository (`pattern: Integrated` only — hard-errors otherwise); author fills
+    any non-generic `canonical ↔ local` mapping.
+  - `<surface>/<surface>-integration.module.ts` — `@generated` aggregator over
+    the per-entity modules.
+  - `<surface>/<surface>-integration.tokens.ts` — `@generated` use-case tokens.
+- **`IncrementalRead` read primitive (RFC-0003).** A universal read capability
+  (`IncrementalRead<T, F>` / `RandomRead<T>` / `IncrementalReadBase` +
+  `SourcedRecord` / `Ref` / `ReadMode` / `ReadRequest` / `mapConcurrent`) in
+  `runtime/subsystems/integration/`, exported from
+  `@pattern-stack/codegen/subsystems`. The base decomposes the read into
+  `enumerate(mode, filter) → AsyncIterable<Ref>` (cheap delta/backfill walk) and
+  `hydrate(ids) → Map<id, raw>` (batched fetch-by-id), and owns the orchestration
+  (drain, **filter-before-hydrate**, bounded-concurrency hydrate, per-ref cursor
+  emission, `listChanges` adaptation). Cursor divisibility is kind-keyed
+  (`CURSOR_DIVISIBILITY` / `isDivisibleCursor`); atomic strategies (`historyId` /
+  `syncToken`) withhold the per-ref cursor until a safe boundary so an
+  interrupted backfill never persists an unresumable token.
+- **Read-side scaffold reshape.** For interaction surfaces (mail / calendar /
+  transcript), `generateAdapterScaffold` now emits each `changeSources` entry as
+  an emit-once `IncrementalReadBase<Canonical<Entity>, ResolvedFilter[]>`
+  subclass — the buffer-all/serial/run-final-cursor regression becomes
+  structurally unwritable. CRM keeps its author-filled `changeSources` seam
+  (field-reader model, no single canonical `T`).
+
 ## [0.12.2] — 2026-05-31
 
 Track D consumer-CLI fix. The 0.12.0/0.12.1 generator was correct in the

package/README.md

@@ -138,6 +138,50 @@ modules/{plural}/
   use-cases/                       FindById, List, declarative queries
 ```
 
+## Integration Codegen (provider/adapter + assembly + read primitive)
+
+When an entity carries a `surface:` tag and `definitions/providers/*.yaml` exist,
+re-running `codegen entity new` emits the **full** integration layer for each
+`(surface, provider, entity)` — not just the read side. The author fills only the
+irreducible vendor seam: the `enumerate` / `hydrate` / `toCanonical` read methods
+and any non-generic sink write logic.
+
+**Read side** (provider/adapter — RFC-0001):
+```
+integrations/providers/<provider>/      Auth strategy + client (provider module)
+integrations/<surface>/adapters/<provider>/  Adapter scaffold: changeSources container
+integrations/<surface>/<surface>-adapters.module.ts  Aggregator → <SURFACE>_ENTITY_SOURCES registry
+integrations/<surface>/types.generated.ts            Typed views
+```
+
+The adapter *contributes* `changeSources` (per-entity, keyed by entity name); the
+aggregator folds every provider's contributions into the entity-keyed
+`<SURFACE>_ENTITY_SOURCES` registry consumers read at runtime.
+
+**Read primitive** (RFC-0003): for interaction surfaces (mail / calendar /
+transcript) each `changeSources` entry is emitted as an emit-once
+`IncrementalReadBase<Canonical<Entity>, ResolvedFilter[]>` subclass — the
+enumerate/hydrate read capability (`@pattern-stack/codegen/subsystems`). The base
+owns streaming, **filter-before-hydrate**, bounded-concurrency hydration, and
+per-ref cursor emission, so the buffer-all/serial/run-final-cursor regression is
+structurally unwritable; the author fills only `enumerate` / `hydrate` /
+`toCanonical`.
+
+**Write/run side** (assembly — RFC-0002):
+```
+integrations/<surface>/modules/<provider>/<entity>-integration.module.ts  @generated per-entity assembly
+integrations/<surface>/sinks/<entity>.sink.ts            emit-once default sink scaffold
+integrations/<surface>/<surface>-integration.module.ts   @generated aggregator
+integrations/<surface>/<surface>-integration.tokens.ts   @generated use-case tokens
+```
+
+Each per-entity module binds `INTEGRATION_CHANGE_SOURCE`
+(= `adapter.changeSources['<entity>']`) + `INTEGRATION_SINK`, provides a local
+`ExecuteIntegrationUseCase`, and exports a uniquely-tokened handle
+(`<ENTITY>_INTEGRATION_USE_CASE__<PROVIDER>`) a trigger grabs to run a sync. The
+default sink scaffolds over the entity's generated `Integrated` repository
+(`pattern: Integrated` only); the author overrides any non-generic write logic.
+
 ## Entity Families
 
 Families provide pre-built base classes with domain-specific query patterns:

package/dist/runtime/subsystems/index.d.ts

@@ -19,12 +19,16 @@ export { IObservability } from './observability/observability.protocol.js';
 export { OBSERVABILITY, OBSERVABILITY_MODULE_OPTIONS } from './observability/observability.tokens.js';
 export { ObservabilityModule, ObservabilityModuleOptions } from './observability/observability.module.js';
 export { ObservabilityError } from './observability/observability-errors.js';
-export { IChangeSource } from './integration/integration-change-source.protocol.js';
+export { IChangeSource, IntegrationSubscriptionView } from './integration/integration-change-source.protocol.js';
 export { CursorSnapshot } from './integration/integration-cursor-store.protocol.js';
+export { IIntegrationSink } from './integration/integration-sink.protocol.js';
 export { IntegrationRunSummary } from './integration/integration-run-recorder.protocol.js';
 export { IEntityChangeSourceRegistry, UnknownEntityError } from './integration/entity-change-source-registry.protocol.js';
 export { MemoryEntityChangeSourceRegistry } from './integration/entity-change-source-registry.memory.js';
-export { ENTITY_CHANGE_SOURCE_REGISTRY } from './integration/integration.tokens.js';
+export { CURSOR_DIVISIBILITY, ResolvedFilter, isDivisibleCursor } from './integration/detection-config.schema.js';
+export { IncrementalRead, IncrementalReadBase, RandomRead, ReadMode, ReadRequest, Ref, SourcedRecord, mapConcurrent } from './integration/incremental-read.js';
+export { ENTITY_CHANGE_SOURCE_REGISTRY, INTEGRATION_CHANGE_SOURCE, INTEGRATION_SINK } from './integration/integration.tokens.js';
+export { ExecuteIntegrationUseCase } from './integration/execute-integration.use-case.js';
 export { AuthCredentials, AuthResolveOptions, IAuthStrategy } from './auth/protocols/auth-strategy.js';
 export { IEncryptionKey } from './auth/protocols/encryption-key.js';
 export { IOAuthStateStore, OAuthStateError, OAuthStateRecord } from './auth/protocols/oauth-state-store.js';

package/dist/runtime/subsystems/index.js

@@ -4360,12 +4360,33 @@ var EventIdCursorSchema = z3.object({
   kind: z3.literal("eventId"),
   field: z3.string().min(1)
 });
+var HistoryIdCursorSchema = z3.object({
+  kind: z3.literal("historyId"),
+  field: z3.string().min(1)
+});
+var SyncTokenCursorSchema = z3.object({
+  kind: z3.literal("syncToken"),
+  field: z3.string().min(1)
+});
 var CursorStrategySchema = z3.discriminatedUnion("kind", [
   SystemModstampCursorSchema,
   ReplayIdCursorSchema,
   TimestampCursorSchema,
-  EventIdCursorSchema
+  EventIdCursorSchema,
+  HistoryIdCursorSchema,
+  SyncTokenCursorSchema
 ]);
+var CURSOR_DIVISIBILITY = {
+  systemModstamp: true,
+  timestamp: true,
+  replayId: true,
+  eventId: false,
+  historyId: false,
+  syncToken: false
+};
+function isDivisibleCursor(kind) {
+  return CURSOR_DIVISIBILITY[kind];
+}
 var PollDetectionSchema = z3.object({
   cursor: CursorStrategySchema,
   provenance: z3.enum(["poll", "cdc"]).optional()
@@ -4390,6 +4411,148 @@ var DetectionConfigSchema = z3.discriminatedUnion("mode", [
   WebhookModeSchema
 ]);
 
+// runtime/subsystems/integration/incremental-read.ts
+async function mapConcurrent(ids, fn, limit) {
+  const out = /* @__PURE__ */ new Map();
+  if (ids.length === 0) return out;
+  const width = Math.max(1, Math.min(limit, ids.length));
+  let next = 0;
+  const worker = async () => {
+    while (next < ids.length) {
+      const idx = next++;
+      const id = ids[idx];
+      out.set(id, await fn(id));
+    }
+  };
+  await Promise.all(Array.from({ length: width }, worker));
+  return out;
+}
+var IncrementalReadBase = class {
+  /**
+   * Whether the vendor takes the request predicate server-side. Declared, not
+   * enforced here — surfaced into the emission manifest (R3) so the falsifier
+   * suite (R4) can record which adapters filter post-hydrate. `false` is the
+   * honest floor (e.g. Gmail without `q=`), handled via `matchesRecord`.
+   */
+  filterPushdown = false;
+  /** Max concurrent in-flight calls for a `mapConcurrent`-built `hydrate`. */
+  hydrateConcurrency = 10;
+  /** `Change<T>.source` provenance stamped by `listChanges`. */
+  changeSource = "poll";
+  /**
+   * Whether this source's cursor strategy is divisible (RFC-0003 §3). When
+   * `true` (default — sortable watermarks like `systemModstamp`/`timestamp`/
+   * `replayId`), `listChanges` emits each record's per-ref cursor, so the
+   * orchestrator may checkpoint mid-walk and a crash resumes from the last
+   * delivered ref.
+   *
+   * When `false` (atomic opaque tokens — Gmail `historyId`, Calendar
+   * `syncToken`), `listChanges` WITHHOLDS per-ref cursors and emits the
+   * end-of-walk token only on the final record, so the orchestrator's
+   * persist-last-yielded lifecycle can never persist an unresumable mid-walk
+   * token. The cost is blast-radius: an interrupted atomic run resumes
+   * all-or-nothing from the prior persisted token. For atomic *backfills* that
+   * radius is the whole enumerate walk — bound it with `ReadRequest.pageSize`
+   * (smaller pages ⇒ shorter walks per run). Per-page atomic checkpointing is a
+   * future refinement; R2 gates at end-of-walk.
+   *
+   * Codegen (R3) sets this from the strategy kind via `isDivisibleCursor`.
+   */
+  cursorDivisible = true;
+  // ---- Optional filter hooks — exactly one is live per `filterPushdown` ----
+  /** Pre-hydrate predicate over the cheap ref (preferred — avoids hydration). */
+  matchesRef(_ref, _filter) {
+    return true;
+  }
+  /** Post-hydrate predicate over the canonical record (the no-pushdown floor). */
+  matchesRecord(_record, _filter) {
+    return true;
+  }
+  /**
+   * Resolve the filter for a subscription when adapting to `listChanges`
+   * (which has no filter argument). Defaults to none; codegen wiring (R3)
+   * overrides this to thread `DetectionConfig.filters`.
+   */
+  filterFor(_subscription) {
+    return void 0;
+  }
+  // ---- PROVIDED by the base ----
+  /**
+   * Stream canonical records for a request. Filter is applied BEFORE hydrate
+   * (structural: a kept ref is hydrated, a rejected one never is), so an
+   * adapter cannot hydrate-then-discard. A hydrate miss (deleted mid-run) is
+   * skipped, never fabricated.
+   */
+  async *read(req) {
+    for await (const refPage of this.enumerate(req.mode, req.filter, req.pageSize)) {
+      const kept = refPage.filter((ref) => this.matchesRef(ref, req.filter));
+      if (kept.length === 0) continue;
+      const raws = await this.hydrate(kept.map((ref) => ref.externalId));
+      for (const ref of kept) {
+        const raw = raws.get(ref.externalId);
+        if (raw === void 0 || raw === null) continue;
+        const record = this.toCanonical(raw);
+        if (record !== null && this.matchesRecord(record, req.filter)) {
+          yield { externalId: ref.externalId, record, raw, cursor: ref.cursor };
+        }
+      }
+    }
+  }
+  /**
+   * `RandomRead<T>` — single-record read, provided for free as
+   * `toCanonical ∘ hydrate([id])`. Reuses the adapter's batched fetch + miss
+   * tolerance; returns `null` for a missing or undecodable record.
+   */
+  async get(id) {
+    const raws = await this.hydrate([id]);
+    const raw = raws.get(id);
+    if (raw === void 0 || raw === null) return null;
+    return this.toCanonical(raw);
+  }
+  /**
+   * `IChangeSource<T>` adaptation. Maps the orchestrator's by-value cursor to a
+   * `ReadMode` (`null` → `full` backfill, else `delta`), streams `read()`, and
+   * stamps each `SourcedRecord` into a `Change<T>`. All records surface as
+   * `'updated'`; the orchestrator's diff stage classifies create-vs-update and
+   * deletes arrive as tombstone refs (`toCanonical` may flag them).
+   *
+   * Cursor emission honors `cursorDivisible` (RFC-0003 §3). Divisible: each
+   * record carries its own per-ref cursor. Atomic: per-ref cursors are withheld
+   * (`undefined`, which the orchestrator skips persisting) and the end-of-walk
+   * token rides only on the final record — so a mid-walk crash never persists
+   * an unresumable token. If an atomic run yields no surviving records, no
+   * cursor is persisted and the next run re-reads the same (empty) delta — a
+   * bounded inefficiency, never data loss.
+   */
+  async *listChanges(subscription, cursor) {
+    const mode = cursor === null || cursor === void 0 ? { kind: "full" } : { kind: "delta", cursor };
+    const filter = this.filterFor(subscription);
+    const stream = this.read({ mode, filter });
+    if (this.cursorDivisible) {
+      for await (const sourced of stream) {
+        yield this.toChange(sourced, sourced.cursor);
+      }
+      return;
+    }
+    let prev = null;
+    for await (const sourced of stream) {
+      if (prev !== null) yield this.toChange(prev, void 0);
+      prev = sourced;
+    }
+    if (prev !== null) yield this.toChange(prev, prev.cursor);
+  }
+  /** Stamp a `SourcedRecord` into a `Change<T>` with an explicit emitted cursor. */
+  toChange(sourced, cursor) {
+    return {
+      externalId: sourced.externalId,
+      operation: "updated",
+      record: sourced.record,
+      cursor,
+      source: this.changeSource
+    };
+  }
+};
+
 // runtime/subsystems/integration/integration-errors.ts
 var MissingTenantIdError3 = class extends Error {
   name = "MissingTenantIdError";
@@ -5741,6 +5904,7 @@ export {
   AuthController,
   AuthModule,
   CACHE,
+  CURSOR_DIVISIBILITY,
   CacheModule,
   ConnectionBrokenError,
   DrizzleCacheService,
@@ -5751,6 +5915,10 @@ export {
   EVENT_BUS,
   EnvEncryptionKey,
   EventsModule,
+  ExecuteIntegrationUseCase,
+  INTEGRATION_CHANGE_SOURCE,
+  INTEGRATION_SINK,
+  IncrementalReadBase,
   LocalStorageBackend,
   MemoryCacheService,
   MemoryEntityChangeSourceRegistry,
@@ -5771,6 +5939,7 @@ export {
   UnknownEntityError,
   authOAuthState,
   collisionModeEnum,
+  isDivisibleCursor,
   isSessionExpiredError,
   jobRunStatusEnum,
   jobRuns,
@@ -5778,6 +5947,7 @@ export {
   jobStepStatusEnum,
   jobSteps,
   jobs,
+  mapConcurrent,
   parentClosePolicyEnum,
   replayFromEnum,
   triggerSourceEnum,