graphddb 0.4.3 → 0.5.1

package/docs/cloudformation.md

@@ -0,0 +1,294 @@
+# GraphDDB — CloudFormation generation
+
+`graphddb generate cloudformation` emits an AWS CloudFormation template
+describing the **DynamoDB table(s)** a GraphDDB single-table design maps onto.
+GraphDDB is single-table by design: every entity that shares a `table` name
+contributes to the SAME `AWS::DynamoDB::Table`, and their Global Secondary
+Indexes are **unioned** by index name.
+
+The command consumes the same serializable manifest the Python bridge uses (it
+loads your entry model, builds the manifest from the shared metadata registry,
+and emits the template). Output is **deterministic** — the same model set always
+produces byte-identical output — and defaults to YAML (`--format json` for JSON).
+
+> This document describes the shipped `generate cloudformation` command. It is
+> the reference for the command's options, the resources it generates, the
+> derivation rules that map a model set onto a table, and its scope boundary.
+
+### Related specifications
+
+- [`spec.md`](./spec.md) — the core API: entities, structured keys / GSIs, the
+  physical key attributes (`PK` / `SK` / `<index>PK` / `<index>SK`).
+- [`design-patterns.md`](./design-patterns.md) — maintained access paths and the
+  `updateMode: 'stream'` signal that drives `--stream auto`.
+- [`python-bridge.md`](./python-bridge.md) — the other consumer of the same
+  manifest / SSoT.
+
+## Overview
+
+```bash
+graphddb generate cloudformation --entry models.ts --out table.yaml
+```
+
+- **`--entry`** loads the TypeScript (or pre-compiled `.js`/`.mjs`) module that
+  registers your entity models as a side effect of import. TypeScript modules are
+  transpiled on demand via `tsx` (an optional peer dependency — install it with
+  `npm i -D tsx`).
+- Without `--out`, the template is written verbatim to **stdout** (pipe-friendly).
+  With `--out`, it is written to that file and a JSON result
+  (`{"status":"ok","outDir":…,"files":[…]}`) is printed to stdout.
+
+## Options
+
+Every option below matches the shipped CLI contract (`cli-contract.yaml`) and the
+handler validation exactly.
+
+| Option | Alias | Type | Default | Notes / validation |
+|--------|-------|------|---------|--------------------|
+| `--entry` | `-e` | file | — (required) | Entry module that registers the entity models. Must exist / be readable; missing → exit `2`. |
+| `--out` | `-o` | file | stdout | Output file. Parent directories are created. Omitted → template to stdout. |
+| `--format` | `-f` | `yaml` \| `json` | `yaml` | Serialization format. Any other value → exit `2`. |
+| `--gsi-projection` | | `ALL` \| `KEYS_ONLY` | `ALL` | GSI `ProjectionType`. `ALL` projects every attribute; `KEYS_ONLY` projects only the key attributes. Other value → exit `2`. |
+| `--stream` | | `auto` \| `on` \| `off` | `auto` | DynamoDB Streams (`StreamSpecification` `NEW_AND_OLD_IMAGES`). `auto` emits a stream only when the model set uses stream-based maintenance (`updateMode: 'stream'`); `on` always emits; `off` never. When enabled, a `StreamArn` Output is added. Other value → exit `2`. |
+| `--billing-mode` | | `PAY_PER_REQUEST` \| `PROVISIONED` | `PAY_PER_REQUEST` | `PAY_PER_REQUEST` is on-demand (no provisioned throughput). `PROVISIONED` emits `BillingMode: PROVISIONED` with `ProvisionedThroughput` on the table **and every GSI**, and unlocks Auto Scaling. Other value → exit `2`. |
+| `--rcu` | | integer ≥ 1 | `5` | Provisioned `ReadCapacityUnits` on the table and every GSI. **Only valid with `--billing-mode PROVISIONED`** — supplying it with `PAY_PER_REQUEST` is a loud reject (exit `2`). Non-integer / `< 1` → exit `2`. |
+| `--wcu` | | integer ≥ 1 | `5` | Provisioned `WriteCapacityUnits`. Same rules as `--rcu`. |
+| `--autoscale` | | boolean flag | off | Enable Application Auto Scaling (a `ScalableTarget` + `ScalingPolicy` pair for the table read/write and each GSI read/write). **Only valid with `--billing-mode PROVISIONED`** — with `PAY_PER_REQUEST` it is a loud reject (exit `2`). |
+| `--autoscale-min` | | integer ≥ 1 | `5` | Auto Scaling `MinCapacity` for every scalable target. Must be `<= --autoscale-max` (else exit `2`). Meaningful only with `--autoscale`. |
+| `--autoscale-max` | | integer ≥ 1 | `100` | Auto Scaling `MaxCapacity` for every scalable target. Must be `>= --autoscale-min`. |
+| `--autoscale-target` | | integer 1–100 | `70` | Target utilization % (`TargetValue` of the target-tracking policy). `> 100` → exit `2`. |
+| `--pitr` | | boolean flag | off | Point-in-time recovery (`PointInTimeRecoverySpecification` `PointInTimeRecoveryEnabled: true`). Omitted entirely when off (never emitted as `false`). |
+| `--table-class` | | `STANDARD` \| `STANDARD_INFREQUENT_ACCESS` | omitted | Emits `TableClass: <value>`. Omitted by default (AWS defaults to `STANDARD`). Other value → exit `2`. |
+| `--sse` | | boolean flag | off | Server-side encryption with the AWS-owned/managed key (`SSESpecification` `SSEEnabled: true` — no KMS key ARN / SSEType). Omitted entirely when off. |
+
+### Exit codes
+
+- `0` — generation succeeded.
+- `1` — unexpected error (e.g. the entry module threw). JSON error on stderr.
+- `2` — invalid arguments (missing / unreadable `--entry`, an invalid enum value,
+  a bad integer, or a loud reject below). JSON error on stderr
+  (`{"code":"INVALID_ARGS","message":…}`).
+
+## Generated resources
+
+For each **physical table** the model set maps onto, the template emits:
+
+- **`AWS::DynamoDB::Table`** with:
+  - `TableName` — a `!Ref` to a `TableName` parameter whose default is the
+    physical table name. (Multi-table designs get one parameter per table, keyed
+    by logical id.)
+  - `BillingMode` — `PAY_PER_REQUEST` (default) or `PROVISIONED`.
+  - `AttributeDefinitions` — **key attributes only**, every one `AttributeType: S`
+    (see derivation rules).
+  - `KeySchema` — `PK` (HASH) plus `SK` (RANGE) when the base key has a sort key.
+  - `GlobalSecondaryIndexes` — one per unioned GSI (omitted when there are none).
+  - `ProvisionedThroughput` — on the table **and every GSI**, only under
+    `PROVISIONED`.
+  - `StreamSpecification` — `NEW_AND_OLD_IMAGES`, only when streams are enabled.
+  - `TimeToLiveSpecification` — only when a `@ttl` field is declared (see TTL).
+  - `PointInTimeRecoverySpecification` / `TableClass` / `SSESpecification` — only
+    when the corresponding `--pitr` / `--table-class` / `--sse` option is given.
+- **`AWS::ApplicationAutoScaling::ScalableTarget` + `::ScalingPolicy`** — separate
+  top-level resources (not table properties), only under `--autoscale`.
+- **Parameters** — a `TableName` (String, default = physical name).
+- **Outputs** — the table name (`Ref`) and ARN (`Fn::GetAtt … Arn`), plus a
+  `StreamArn` output when streams are enabled.
+
+All resources, attribute definitions, and GSIs are emitted in a stable (sorted)
+order, so the output is byte-deterministic.
+
+## Derivation rules
+
+How a GraphDDB model set is mapped onto the physical DynamoDB shape:
+
+1. **One physical table per logical `table` name.** Entities that share a `table`
+   are single-table siblings and contribute to the same `AWS::DynamoDB::Table`.
+   The CFn logical resource id is `<AlphanumericLogicalName>Table` (e.g.
+   `UserPermissions` → `UserPermissionsTable`).
+2. **Physical key attribute names match the runtime exactly.** The base partition
+   key is always `PK`; the base sort key (when present) is `SK`. Each GSI uses
+   `<indexName>PK` and `<indexName>SK` (e.g. `GSI1PK` / `GSI1SK`).
+3. **`AttributeDefinitions` are key attributes only, all `AttributeType: S`.** It
+   contains `PK`, `SK` (when the table has a sort key), and each GSI's `<index>PK`
+   plus `<index>SK` (only for GSIs that have a sort key). **Non-key attributes are
+   never added** — in particular the TTL attribute is deliberately kept out
+   (CloudFormation rejects a non-key attribute there; `TimeToLiveSpecification`
+   references it legitimately).
+4. **The base `KeySchema` is `PK` (HASH) + `SK` (RANGE).** `SK` is included when
+   **any** entity on the table declares a sort key; a PK-only design omits `SK`
+   and its attribute definition.
+5. **GSIs are unioned by index name across all entities on the table.** The same
+   `IndexName` declared by several entities collapses to one GSI. Each GSI's
+   `KeySchema` is `<index>PK` (HASH) plus `<index>SK` (RANGE) — HASH-only when the
+   GSI has no sort key. `Projection.ProjectionType` is `ALL` (default) or
+   `KEYS_ONLY`. (Full GSI-union consistency — same sort-key presence across
+   entities, ≤ 20 GSIs — is enforced statically at model registration by the
+   `cfn-schema-consistency` linter rule; the emitter keeps a backstop that throws
+   on a surviving inconsistency.)
+6. **Streams (`--stream auto`) are derived from maintenance metadata.** `auto`
+   emits a stream when any registered entity uses stream-based maintenance
+   (`updateMode: 'stream'` — on a relation `write`, a versioned relation preset,
+   an `@aggregate` `write`, or a view's `@maintainedFrom`). This signal comes from
+   the metadata registry, not the manifest. The single view type is
+   `NEW_AND_OLD_IMAGES` (stream-based maintenance needs both images).
+7. **TTL is derived from a `@ttl` field.** When an entity on the table declares a
+   `@ttl` field, its physical attribute name is carried in the manifest and
+   rendered as `TimeToLiveSpecification { AttributeName, Enabled: true }`. DynamoDB
+   allows exactly one TTL attribute per table; the manifest build enforces this,
+   and the emitter backstops it by throwing if two entities on a table disagree.
+8. **Provisioned throughput + Auto Scaling are pure deploy-time settings** (from
+   CLI options, not derived from the model). Under `PROVISIONED`, the table and
+   every GSI carry `ProvisionedThroughput` (`--rcu` / `--wcu`, default 5/5).
+   Under `--autoscale`, each of the table read/write and every GSI read/write gets
+   a `ScalableTarget` (`ServiceNamespace: dynamodb`, the matching
+   `ScalableDimension`, an `Fn::Sub` `ResourceId` tying to the table's `Ref`,
+   `MinCapacity`/`MaxCapacity`, and the DynamoDB service-linked role ARN) plus a
+   `TargetTrackingScaling` `ScalingPolicy` (predefined
+   `DynamoDBRead/WriteCapacityUtilization` metric, `TargetValue`). So **N GSIs →
+   `2×(1+N)` scalable targets + `2×(1+N)` scaling policies.** GSI targets
+   additionally `DependsOn` the table.
+9. **PITR / TableClass / SSE are pure deploy-time settings**, each defaulting to
+   OFF/omitted — the emitter omits the whole property rather than emitting a
+   `false`/default value.
+
+## Scope boundary
+
+This command generates **only the GraphDDB-managed DynamoDB resources** — the
+table(s), their GSIs, and the DynamoDB-adjacent settings above (Streams, TTL,
+PITR, TableClass, SSE, provisioned throughput, and the Application Auto Scaling
+resources for that table). It does **not** generate IAM roles/policies for your
+application, Lambda functions, API Gateway, stream consumers, DynamoDB Streams
+event-source mappings, or any other application infrastructure. The one IAM
+reference it emits is the AWS **service-linked** role ARN that Application Auto
+Scaling uses (built with `Fn::Sub` so `${AWS::AccountId}` resolves at deploy
+time) — it does not create an IAM role. Compose the generated template into your
+own stack (nested stack / module) alongside those app resources.
+
+## Loud-reject / validation behaviors
+
+The CLI validates arguments before generation and **rejects loudly** (exit `2`,
+JSON `INVALID_ARGS` on stderr) rather than silently ignoring a nonsensical
+combination. The pure emitter keeps the same guards as an authoritative backstop
+(it throws). The loud rejects are:
+
+- **`--rcu` / `--wcu` with `PAY_PER_REQUEST`** — on-demand tables have no
+  provisioned capacity, so accepting the value would mislead you.
+- **`--autoscale` with `PAY_PER_REQUEST`** — Application Auto Scaling on an
+  on-demand table is invalid in AWS.
+- **`--autoscale-min` > `--autoscale-max`** — an invalid scalable-target range.
+- **`--autoscale-target` > 100** — it is a utilization percentage (1–100).
+- **A non-integer or `< 1` value for `--rcu` / `--wcu` / `--autoscale-*`.**
+- **An invalid enum value** for `--format` / `--gsi-projection` / `--stream` /
+  `--billing-mode` / `--table-class`.
+- **A missing `--entry`** (or one that cannot be imported).
+
+Inconsistent single-table declarations (a GSI with conflicting sort-key presence,
+or two `@ttl` attributes on one table) are caught statically by the
+`cfn-schema-consistency` linter at model registration; the emitter throws as a
+backstop if one survives to generation time.
+
+## Worked example
+
+The [`user-permissions`](../examples/user-permissions/) example is a single
+`UserPermissions` table whose entities (users, groups, memberships, permissions)
+share one physical table, with a `GSI1` **unioned** across `UserModel` (email
+lookup) and `GroupMembershipModel` (the inverted membership index). It exercises
+the base `PK`/`SK` key and a GSI union — it does **not** use streams, TTL, PITR,
+provisioned throughput, or auto scaling (those are opt-in flags / a `@ttl`
+decorator this model does not trigger).
+
+Generate it (the committed sample
+[`examples/user-permissions/cloudformation.yaml`](../examples/user-permissions/cloudformation.yaml)
+is produced by exactly this command, and a drift-guard test keeps it in sync):
+
+```bash
+graphddb generate cloudformation \
+  --entry examples/user-permissions/models.ts \
+  --out examples/user-permissions/cloudformation.yaml
+```
+
+The generated template (default YAML):
+
+```yaml
+AWSTemplateFormatVersion: "2010-09-09"
+Description: DynamoDB table(s) for a GraphDDB single-table design (generated by graphddb generate cloudformation).
+Parameters:
+  TableName:
+    Type: String
+    Default: UserPermissions
+    Description: Name of the UserPermissions DynamoDB table.
+Resources:
+  UserPermissionsTable:
+    Type: "AWS::DynamoDB::Table"
+    Properties:
+      TableName:
+        Ref: TableName
+      BillingMode: PAY_PER_REQUEST
+      AttributeDefinitions:
+        - AttributeName: PK
+          AttributeType: S
+        - AttributeName: SK
+          AttributeType: S
+        - AttributeName: GSI1PK
+          AttributeType: S
+        - AttributeName: GSI1SK
+          AttributeType: S
+      KeySchema:
+        - AttributeName: PK
+          KeyType: HASH
+        - AttributeName: SK
+          KeyType: RANGE
+      GlobalSecondaryIndexes:
+        - IndexName: GSI1
+          KeySchema:
+            - AttributeName: GSI1PK
+              KeyType: HASH
+            - AttributeName: GSI1SK
+              KeyType: RANGE
+          Projection:
+            ProjectionType: ALL
+Outputs:
+  TableName:
+    Description: Name of the UserPermissions DynamoDB table.
+    Value:
+      Ref: UserPermissionsTable
+  TableArn:
+    Description: ARN of the UserPermissions DynamoDB table.
+    Value:
+      "Fn::GetAtt":
+        - UserPermissionsTable
+        - Arn
+```
+
+### More option combinations
+
+```bash
+# JSON output.
+graphddb generate cloudformation --entry models.ts --format json
+
+# Key-only GSI projection.
+graphddb generate cloudformation --entry models.ts --gsi-projection KEYS_ONLY --out table.yaml
+
+# Force a DynamoDB stream (NEW_AND_OLD_IMAGES) on.
+graphddb generate cloudformation --entry models.ts --stream on --out table.yaml
+
+# Deploy-time hardening: PITR + SSE + Infrequent-Access table class.
+graphddb generate cloudformation --entry models.ts --pitr --sse --table-class STANDARD_INFREQUENT_ACCESS --out table.yaml
+
+# Provisioned billing with fixed throughput.
+graphddb generate cloudformation --entry models.ts --billing-mode PROVISIONED --rcu 10 --wcu 5 --out table.yaml
+
+# Provisioned billing + Application Auto Scaling.
+graphddb generate cloudformation --entry models.ts \
+  --billing-mode PROVISIONED --autoscale --autoscale-min 5 --autoscale-max 100 --autoscale-target 70 --out table.yaml
+```
+
+## Validating the output
+
+Real `aws cloudformation validate-template` requires AWS credentials and the
+`cloudformation:ValidateTemplate` IAM permission. Where those are unavailable
+(this project's CI is IAM-blocked, for example), an equivalent **structural**
+check is enough for a syntax gate: generate the template in both formats, parse
+each (YAML + JSON), assert the two agree, and assert the table structure. The
+repository wires this as `npm run validate:cfn` (see `scripts/validate-cfn.mjs`),
+which the CI runs as its "Validate generated CloudFormation (structural)" step.

package/docs/design-patterns.md

@@ -47,7 +47,7 @@ graph, the stream drain, and the rebuild planner consume every pattern uniformly
 | 7 | [Aggregate Counter](#7-aggregate-counter) | `@aggregate(..., { pattern: 'counter', value: count() })` | ✅ Phase 1 (`count()`) / stream (`max()`) |
 | 8 | [Versioned / Latest Pointer](#8-versioned--latest-pointer) | `@hasOne(... versionedLatest)` + `@hasMany(... versionedHistory)` | ✅ Phase 3 |
 | 9 | [Reverse Lookup](#9-reverse-lookup) | GSI (both directions) or `dualEdge` two-row sync | ✅ Phase 1 (GSI) / Phase 3 (`dualEdge`) |
-| 10 | [External Projection](#10-external-projection) | typed-consumer-IF contract (redesign — see #153) | 🚧 redesign |
+| 10 | [External Projection](#10-external-projection) | typed-consumer-IF contract (`@cdcProjected` + `fromChange` / `subscribe`) | ✅ |
 
 ---
 
@@ -494,23 +494,28 @@ OpenSearch, S3, Neptune — for analytics or search, decoupled from the table. T
 projection is asynchronous and must tolerate at-least-once delivery (a redelivered
 event must not double-write).
 
-**(b) How it is being redesigned (🚧 #153).** The previous `defineProjection` builder +
+**(b) How to declare it in graphddb.** The previous `defineProjection` builder +
 `ProjectionSinkDrain` / `InMemoryProjectionSink` runtime baked the sink-delivery
 semantics (upsert, idempotency, dedup) into the library — a boundary overreach: the
 actual sink write happens inside the consumer's CDC infrastructure, outside graphddb.
-That runtime was **removed in 0.4.0** (issue #152). The redesign (issue #153) reframes
-external projection as a **typed-consumer-IF contract**: from a declared source +
-projection fields + idempotency-key extraction, graphddb emits a typed
-`(changeImage) => TypedProjectionRecord` mapper for the consumer to call inside its own
-sink delivery — graphddb owns the *parse → typed record* contract, not the delivery.
-External projection is neither a Model nor a relation, so it does not ride a decorator.
+That runtime was **removed in 0.4.0** (issue #152). External projection is now a
+**typed-consumer-IF contract** (issue #153): mark the source model with
+`@cdcProjected()`, then parse a CDC change event into typed instances — `Model.fromChange(event)`
+returns `[oldRecord, newRecord]`, and `DDBModel.subscribe(handlers)` (the GraphQL-style
+sibling of `query` / `mutate`) returns a `ChangeHandler` the consumer mounts on its own
+stream. graphddb owns the *parse → typed record* contract; subscription, sink delivery,
+and idempotency stay with the consumer. See [`cdc-projection.md`](./cdc-projection.md)
+for the full design.
 
 **(c) Read & write behavior.** Out of scope for this library (the data is read from the
-sink; delivery + idempotency live in the consumer's CDC code). See #153 for the
-typed-consumer-IF contract design.
-
-**(d) Constraints & phase status.** 🚧 Redesign (#153). The sink runtime is gone in
-0.4.0; the typed-consumer-IF replacement lands separately.
+sink; delivery + idempotency live in the consumer's CDC code). graphddb only parses the
+change event into a typed record; see [`cdc-projection.md`](./cdc-projection.md) for the
+typed-consumer-IF contract.
+
+**(d) Constraints & phase status.** ✅ Shipped (issue #153). The sink runtime was removed
+in 0.4.0; the typed-consumer-IF replacement (`@cdcProjected` + `fromChange` + `subscribe`
++ the generated `CdcModelRegistry`) is implemented and tested (unit + emulator-driven
+integration).
 
 ---
 

package/docs/spec.md

@@ -546,9 +546,10 @@ Shared vocabulary (confirmed across the epic):
   view model** — `materializedView` / `sparseView` — are declared with `@model({ kind })`
   + `@maintainedFrom` (§7.1 below). The **versioned** presets ride the `@hasOne` /
   `@hasMany` `pattern` discriminated union (`versionedLatest` / `versionedHistory`).
-  External projection is being redesigned as a typed-consumer-IF contract (#153) and
-  has no decorator. Unknown values on a relation are accepted by the open-ended union
-  but not lowered.
+  External projection is a typed-consumer-IF contract (#153); it is not a relation
+  `pattern` — the source model is marked with `@cdcProjected()` instead (see
+  [`cdc-projection.md`](./cdc-projection.md)). Unknown values on a relation are
+  accepted by the open-ended union but not lowered.
 - **`write.maintainedOn`** — cross-entity triggers `"<Entity>.<event>"`. The
   Entity segment is the **logical** entity name (`PostModel → 'Post'`; or the
   `@model({ prefix })` value sans `#`), and the event is one of
@@ -634,7 +635,11 @@ The latest/history key binding is derived from each target model's own `key` fie
 composed snapshot×2 IR. (A self-co-located form is a deferred follow-up.)
 
 External projection's old `defineProjection` / `ProjectionSinkDrain` runtime was
-**removed in 0.4.0** and is being redesigned as a typed-consumer-IF contract (#153).
+**removed in 0.4.0** and is now a typed-consumer-IF contract (#153):
+the source model is marked `@cdcProjected()`, `Model.fromChange(event)` parses a CDC
+change event into `[oldRecord, newRecord]`, and `DDBModel.subscribe(handlers)` returns a
+`ChangeHandler` the consumer mounts on its own stream (subscription / sink delivery /
+idempotency stay with the consumer). See [`cdc-projection.md`](./cdc-projection.md).
 
 ### 7.2 Stream maintenance — outbox → CDC drain (`updateMode: 'stream'`, #130)
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "graphddb",
-  "version": "0.4.3",
+  "version": "0.5.1",
   "description": "Graph data modeling on DynamoDB with adjacency list pattern",
   "type": "module",
   "main": "dist/index.js",
@@ -37,6 +37,7 @@
     "test:py": "python3 -m pytest python/tests -m \"not integration\"",
     "test:py:integration": "python3 -m pytest python/tests -m integration",
     "test:conformance": "tsx conformance/run.ts",
+    "validate:cfn": "node scripts/validate-cfn.mjs",
     "gen:cli": "cli-contracts generate",
     "gen:cli:check": "cli-contracts generate --dry-run",
     "clean": "rm -rf dist",
@@ -70,7 +71,8 @@
     "tsup": "^8.0.0",
     "tsx": "^4.22.4",
     "typescript": "^5.5.0",
-    "vitest": "^3.0.0"
+    "vitest": "^3.0.0",
+    "yaml": "^2.9.0"
   },
   "engines": {
     "node": ">=22"