graphddb 0.1.0

package/README.md

@@ -0,0 +1,553 @@
+# GraphDDB
+
+**Type-safe Graph Query Runtime for DynamoDB**
+
+GraphDDB maps DynamoDB's native access patterns to a GraphQL-like query model:
+PK, GSI, Projection, Relation, Cursor, and operation limits all remain visible.
+
+GraphDDB is not an ORM.
+It is a runtime for defining, validating, and executing DynamoDB access patterns with TypeScript types.
+
+## Why GraphDDB?
+
+DynamoDB is fast and scalable, but easy to design incorrectly. Teams often treat it like an
+RDBMS — one table per entity, joins in application code, ad-hoc access patterns hidden behind
+ORM abstractions — and end up with unstable performance, unnecessary GSIs, and unclear service
+boundaries.
+
+GraphDDB takes the opposite approach: it represents DynamoDB access patterns **directly in
+code**, keeping PK/GSI, projections, relations, cursors, and operation limits visible. Access
+patterns and relations become explicit, query plans become inspectable, and well-designed
+access patterns naturally align with service boundaries. GraphDDB does not teach DynamoDB
+design — it makes correct design the easiest way to write code.
+
+It does this through a natural correspondence between GraphQL's query model and DynamoDB's
+access patterns:
+
+```text
+GraphQL Query Model        DynamoDB
+arguments           ---->  PK / GSI lookup
+selection           ---->  ProjectionExpression
+relation            ---->  Query / Get / BatchGet
+connection          ---->  Cursor pagination
+complexity          ---->  Operation limits
+```
+
+A partition key is a graph entry point, a relation an explicit traversal path, a projection a
+selection set. GraphDDB does not invent a query language — it executes well-designed access
+patterns directly and makes ambiguity visible through plans, limits, and relation resolution.
+
+## User Permissions Example
+
+The [user-permissions](./examples/user-permissions/) example models users, groups, memberships, and permissions in a single DynamoDB table.
+This is where GraphDDB's design becomes easiest to see: the table is physical, while the query surface is graph-shaped.
+
+```text
+UserPermissions table
+
+Entity            PK                  SK                         GSI1PK              GSI1SK
+Group             GROUP#<groupId>     META
+User              USER#<userId>       PROFILE                    EMAIL#<email>       PROFILE
+GroupMembership   GROUP#<groupId>     USER#<userId>              USER#<userId>       GROUP#<groupId>
+Permission        GROUP#<groupId>     PERM#<resource>#<action>
+```
+
+`GroupMembership` uses the adjacency list pattern:
+
+- `PK/SK` reads members of a group
+- `GSI1PK/GSI1SK` reads groups for a user
+
+The same physical table supports both directions without hiding the keys.
+
+```ts
+const engGroup = await Group.query(
+  { groupId: 'eng' },
+  {
+    groupId: true,
+    name: true,
+    members: {
+      select: { userId: true, role: true },
+      limit: 20,
+    },
+    permissions: {
+      select: { resource: true, action: true },
+      limit: 20,
+    },
+  },
+);
+
+// Find a user by email (GSI), list their groups, and for each group fetch the
+// group and its permissions. The N memberships -> parent Group resolution
+// collapses into a single BatchGetItem (no N+1).
+const alice = await User.query(
+  { email: 'alice@example.com' },
+  {
+    userId: true,
+    name: true,
+    groups: {
+      select: {
+        groupId: true,
+        role: true,
+        group: {                                // membership -> Group (belongsTo, BatchGet)
+          select: {
+            name: true,
+            permissions: {                      // group -> permissions (hasMany)
+              select: { resource: true, action: true },
+              filter: { resource: { attributeExists: true } },
+            },
+          },
+        },
+      },
+      limit: 20,
+    },
+  },
+  { maxDepth: 3 },
+);
+```
+
+The first query expands into a small, inspectable DynamoDB plan:
+
+```text
+1. Query  PK = GROUP#eng, SK = META             -> Group
+2. Query  PK = GROUP#eng, SK begins_with USER#  -> members      ┐ dispatched
+3. Query  PK = GROUP#eng, SK begins_with PERM#  -> permissions  ┘ in parallel
+```
+
+Independent relation sub-queries (here `members` and `permissions`) are dispatched concurrently; the `belongsTo` resolution in the second query batches N lookups into one `BatchGetItem`. See the [example](./examples/user-permissions/) for a timeline that proves the parallel dispatch.
+
+## Usage
+
+The snippets below are compressed from the [user-permissions](./examples/user-permissions/) example.
+
+### Entity Definition
+
+A TypeScript class is the single source of truth. Keys and GSIs are built from `k` tagged-template
+**segments**, and the key builder's input type becomes the query parameter type.
+
+```ts
+const TABLE = 'UserPermissions';
+
+@model({ table: TABLE, prefix: 'USER' })
+class UserModel extends DDBModel {
+  static readonly keys = key<{ userId: string }>((c) => ({
+    pk: k`USER#${c.userId}`,
+    sk: k`PROFILE`,
+  }));
+
+  static readonly emailIndex = gsi<{ email: string }>(
+    'GSI1',
+    (c) => ({ pk: k`EMAIL#${c.email}`, sk: k`PROFILE` }),
+    { unique: true },
+  );
+
+  @string userId!: string;
+  @string name!: string;
+  @string email!: string;
+  @string status!: string;
+
+  @hasMany(() => GroupMembershipModel, { userId: 'userId' }, {
+    limit: { default: 20, max: 100 },
+  })
+  groups!: GroupMembershipModel[];
+}
+
+@model({ table: TABLE, prefix: 'GROUP' })
+class GroupMembershipModel extends DDBModel {
+  static readonly keys = key<{ groupId: string; userId: string }>((c) => ({
+    pk: k`GROUP#${c.groupId}`,
+    sk: k`USER#${c.userId}`,
+  }));
+
+  static readonly userGroupsIndex = gsi<{ userId: string; groupId: string }>(
+    'GSI1',
+    (c) => ({ pk: k`USER#${c.userId}`, sk: k`GROUP#${c.groupId}` }),
+  );
+
+  @string groupId!: string;
+  @string userId!: string;
+  @string role!: string;
+}
+
+export const User = UserModel.asModel();
+export const GroupMembership = GroupMembershipModel.asModel();
+```
+
+A multi-segment sort key is an array of segments (e.g. `sk: [k\`PERM#${c.resource}\`, k\`${c.action}\`]`);
+a partial key compiles to `begins_with` at the matching segment boundary. See the
+[specification](./docs/spec.md) for the full key model.
+
+### GraphQL-like Queries
+
+The first argument describes how to find data.
+The second argument describes what to return.
+The runtime resolves which index to use from the provided field names.
+
+```ts
+// Fetch a user by email and traverse to group memberships.
+const alice = await User.query(
+  { email: 'alice@example.com' },
+  {
+    userId: true,
+    name: true,
+    groups: {
+      select: { groupId: true, role: true },
+      limit: 20,
+    },
+  },
+);
+
+// Unknown key field -> compile-time error
+User.query({ foo: 'bar' }, { name: true });
+```
+
+The return type is inferred from `select`:
+
+```ts
+// alice: {
+//   name: string;
+//   groups: {
+//     items: { groupId: string; role: string }[];
+//     cursor: string | null;
+//   };
+// }
+```
+
+### List Queries
+
+```ts
+const result = await GroupMembership.list(
+  { groupId: 'eng' },
+  {
+    select: { userId: true, role: true },
+    limit: 20,
+    after: cursor,
+  },
+);
+// result.items: GroupMembership[]
+// result.cursor: string | null
+```
+
+### Filtering (server-side `filter`)
+
+`filter` is a declarative, type-safe condition compiled to a DynamoDB **`FilterExpression`** and
+evaluated **server-side**. Modeled on AWS AppSync's `ModelFilterInput`: a bare value is an
+equality shorthand, an operator object expresses comparisons, and `and` / `or` / `not` build
+logical groups. It is typed against the **full entity**, so it can reference attributes that are
+**not** in `select`.
+
+```ts
+const result = await Order.list(
+  { userId: 'u001' },
+  {
+    select: { orderId: true },        // amount / status need not be projected
+    filter: {
+      status: 'confirmed',            // #status = :v        (equality shorthand)
+      amount: { gt: 100 },            // #amount > :v
+      title:  { beginsWith: 'A' },    // begins_with(#title, :v)
+      shippedAt: { attributeExists: true },
+      or: [{ status: 'pending' }, { amount: { lt: 10 } }],
+    },
+  },
+);
+```
+
+Relations take the same `filter` via `Model.relation(...)` of the target model. For logic a
+server-side `FilterExpression` cannot express, call `result.items.filter(...)` on the returned
+projection — it keeps each selected field's declared scalar type, so `o.amount > 100` needs no
+cast (there is no built-in post-load predicate).
+
+Available operators (per field type): `eq`, `ne`, `ge`, `le`, `gt`, `lt`,
+`between`, `in`, `beginsWith`, `contains`, `notContains`, `attributeExists`,
+`attributeType`, `size`. Operators are **constrained by the field's declared
+type** — e.g. `beginsWith` / `contains` on a numeric field, or `gt: 'x'` on a
+number, are compile errors.
+
+For rare conditions the operator objects cannot express, the `cond` raw escape
+hatch accepts an expression fragment. Column names must use refactor-safe
+`Model.col.<field>` references (never bare strings); values are parameterized:
+
+```ts
+filter: cond`${Order.col.amount} > ${100} AND attribute_exists(${Order.col.status})`
+```
+
+All values are parameterized (`ExpressionAttributeValues`) and all column names
+are aliased (`ExpressionAttributeNames`) — there is no literal interpolation, so
+the compiled expression is injection-free, and it is attached independently of
+the `KeyConditionExpression` / `ProjectionExpression`.
+
+> **RCU note:** A `FilterExpression` does **not** reduce read capacity —
+> DynamoDB reads matching keys first, then filters. `limit` is applied
+> **before** the filter, so a single page may return **fewer** than `limit`
+> items (and, if the whole page is filtered out, an empty page with a non-null
+> cursor). Design keys for efficient narrowing; use `filter` for correctness.
+
+### Writes
+
+```ts
+await User.put({
+  userId: 'alice',
+  name: 'Alice',
+  email: 'alice@example.com',
+  status: 'active',
+});
+
+await GroupMembership.put({
+  groupId: 'eng',
+  userId: 'alice',
+  role: 'admin',
+});
+
+await User.update(alice, { status: 'disabled' });
+```
+
+### Inspect Execution Plans
+
+`explain()` shows the DynamoDB operations before they are executed.
+`select` is converted into a `ProjectionExpression`, so only requested attributes are read.
+
+```ts
+const plan = GroupMembership.explain(
+  { groupId: 'eng' },
+  { select: { userId: true, role: true }, limit: 10 },
+);
+// {
+//   operations: [{
+//     type: "Query",
+//     tableName: "UserPermissions",
+//     keyCondition: { PK: "GROUP#eng" },
+//     rangeCondition: {
+//       operator: "begins_with",
+//       key: "SK",
+//       value: "USER#"
+//     },
+//     limit: 10
+//   }]
+// }
+```
+
+For partial GSI key matches, the SK condition is separated as a `rangeCondition`:
+
+```ts
+const gsiPlan = GroupMembership.explain(
+  { userId: 'alice' },
+  { select: { groupId: true, role: true }, limit: 10 },
+);
+// {
+//   operations: [{
+//     type: "Query",
+//     keyCondition: { GSI1PK: "USER#alice" },
+//     indexName: "GSI1",
+//     rangeCondition: {
+//       operator: "begins_with",
+//       key: "GSI1SK",
+//       value: "GROUP#"
+//     },
+//     ...
+//   }]
+// }
+```
+
+## Core Concepts
+
+### Entity = Access Pattern
+
+An Entity is not an ORM model.
+It is a bounded access surface for DynamoDB.
+
+An Entity defines every supported way to access data within a service boundary:
+
+- physical keys
+- GSIs
+- relations
+- projections
+- traversal rules
+
+Physically, an Entity maps to a DynamoDB item.
+Architecturally, it defines the allowed access surface for that part of the service.
+
+### Key / GSI = Structured Segments
+
+`key()` and `gsi()` build keys from `k` tagged-template **segments**. The builder's input type
+defines the fields required to resolve the key, and also becomes the query parameter type.
+
+```ts
+// One definition serves two roles:
+// 1. Write path: generate PK/SK when saving an Entity
+// 2. Query path: define the parameter type for User.query({ email: ... })
+static readonly emailIndex = gsi<{ email: string }>(
+  'GSI1',
+  (c) => ({ pk: k`EMAIL#${c.email}`, sk: k`PROFILE` }),
+  { unique: true },
+);
+```
+
+Because keys are structured segments rather than opaque strings, a partial key — one supplying
+only a leading prefix of the sort-key segments — compiles to `begins_with` at that segment
+boundary. This is how relations and prefix queries resolve.
+
+### Relation = Bound Query
+
+A Relation is a query bound to values from the parent Entity.
+The same resolution logic applies: the runtime matches field names against Key / GSI definitions.
+
+```ts
+@hasMany(() => GroupMembershipModel, { userId: 'userId' }, {
+  limit: { default: 20, max: 100 },
+})
+groups!: GroupMembershipModel[];
+```
+
+### Field Decorators
+
+Type-safe semantic decorators define DynamoDB attribute types.
+Compile-time checks based on `ClassFieldDecoratorContext` detect mismatches between declared TypeScript types and DynamoDB types.
+
+```ts
+@string userId!: string;     // DynamoDB S
+@number amount!: number;     // DynamoDB N
+@datetime createdAt!: Date;  // stored as an ISO 8601 string
+@boolean isActive!: boolean; // DynamoDB BOOL
+```
+
+### Select = Projection + Type Inference
+
+The `select` object controls both DynamoDB `ProjectionExpression` generation and TypeScript return type inference.
+Only requested fields are returned, and only those fields appear in the result type.
+
+### Planner = Inspectable Execution Plan
+
+Before DynamoDB is accessed, the runtime generates an execution plan.
+Plans can be used for tests, debugging, and RCU estimation.
+
+## Architecture
+
+```text
+Layer 1: Schema Definition
+    Entity / Field / Key / GSI / Relation / Embedded
+    -> EntityMetadata Registry
+
+Layer 2: Query Builder
+    Type-safe DSL: key resolution, select validation
+    -> ExecutionPlan
+
+Layer 3: Query Planner
+    Relation traversal, Get vs Query vs BatchGet decisions
+    Projection generation, GSI selection
+    -> DynamoDBOperation[]
+
+Layer 4: Executor
+    DynamoDB DocumentClient wrapper
+    -> raw Items
+
+Layer 5: Hydrator
+    raw Items -> Partial Entity instances
+    type conversion, Embedded reconstruction, metadata attachment
+```
+
+## Design Rules
+
+| Rule | Description |
+|------|-------------|
+| Service Boundary First | One table represents one service boundary, not one entity. |
+| Access Pattern First | Model DynamoDB access patterns before modeling tables or ORM-style entities. |
+| Natural Mapping | Map Query / Get / BatchGet / Projection / cursor to a GraphQL-like query shape. |
+| Inspectable Plans | DynamoDB operations and execution plans are inspectable before execution. |
+| Single-item query | `query` is only for PK or unique GSI lookups. Use `list` for keys that can return multiple items. |
+| Explicit limits | `hasMany` and `list` require default/max limits. |
+| Bounded depth | Relation traversal defaults to depth=1. Deeper traversal must be explicitly allowed with `maxDepth`. |
+| Parallel relations | Independent relation sub-queries, BatchGet chunks, and nested resolution are dispatched concurrently (bounded). DynamoDB is HTTP-based, so there is no connection to serialize on. |
+| No Scan | Full table scans are forbidden. The linter detects and blocks them. |
+| Partial select | Query results are partial types. Only selected fields appear in the result type. |
+| Structured keys | Keys are built from `k` segment templates; partial keys compile to `begins_with` at a segment boundary. |
+| Typed consistentRead | `consistentRead` is only available for PK queries. GSI queries are rejected at compile time. |
+| Server-side filter | `filter` is a declarative DynamoDB `FilterExpression` (AppSync `ModelFilterInput`-compatible), typed against the full entity and able to reference unprojected attributes. It does not reduce RCU, and `limit` is applied before it. |
+| Post-load filtering | No built-in post-load predicate; use `result.items.filter(...)` on the typed projection. Efficient narrowing belongs in key design. |
+
+## Architectural Boundaries
+
+Relation traversal only follows access paths defined on Entities.
+Undefined relations, missing GSIs, lists without limits, and traversal that exceeds allowed depth are checked before execution.
+
+This makes cross-boundary queries and DynamoDB-unfriendly access patterns easier to see in code.
+The same architectural constraints apply regardless of whether code is written by humans or generated by AI.
+
+## Linter
+
+Static analysis detects problems in Entity definitions before runtime:
+
+- Scan usage
+- missing limits on `hasMany` / `list`
+- ambiguous GSI resolution
+- query/list boundary violations for unique GSIs
+
+## Beyond the Core
+
+The query / relation / write API above is the core layer. GraphDDB also ships:
+
+- **[CQRS contracts](./docs/cqrs-contract.md)** — public Query/Command contract models with an
+  N+1-safe cardinality matrix, cross-contract composition, and context-boundary enforcement.
+- **[Mutation → command derivation](./docs/mutation-command-derivation.md)** — an internal
+  write-plan composition DSL behind the public Command IF. Declare a model's write semantics
+  (`entityWrites`: referential integrity, uniqueness, edge effects, derived counters, outbox
+  events, idempotency) once; a `mutation` composes one or more write fragments that the compiler
+  merges into a single atomic `TransactWriteItems`. All write paths route through one shared
+  `commitTransaction` orchestration.
+- **[Opt-in class hydration](./docs/class-hydration.md)** — pass an `options.hydrate` factory to
+  load a read result onto a host-language domain object instead of the default Typed Plain Object.
+  Host-only and never serialized into the bridge SSoT. Phase 1 (`query` top-level) is implemented;
+  `list` / per-relation hydration are planned future phases.
+- **[Multi-language (Python bridge)](./docs/python-bridge.md)** — TypeScript is the single source
+  of truth; generate a Python client + runtime from the same definitions, kept in lockstep by a
+  TS↔Python conformance suite.
+- **[In-memory testing](./docs/testing.md)** — `graphddb/testing` runs model-mapping, query-plan,
+  relation-traversal, and CDC tests **in-process, without Docker**.
+- **[CDC emulator](./docs/cdc-emulator.md)** — DynamoDB-Streams-equivalent change events for
+  driving and testing incremental aggregation locally.
+
+## Documentation
+
+| Document | Description |
+|----------|-------------|
+| [Specification](./docs/spec.md) | Core API: entities, structured keys/GSIs, query/filter, relations, batch/transactions, design rules, runtime behavior. |
+| [CQRS contract layer](./docs/cqrs-contract.md) | Public Query/Command contracts, cardinality matrix, N+1 safety, cross-contract composition, context boundaries. |
+| [Mutation → command derivation](./docs/mutation-command-derivation.md) | The internal write-plan composition DSL behind the Command IF: model write-semantics (`entityWrites`), fragment composition, and atomic `TransactWriteItems` derivation. |
+| [Class hydration](./docs/class-hydration.md) | Opt-in `options.hydrate` factory that loads a read result onto a host-language object (Phase 1: `query` top-level; Phases 2–3 future). |
+| [Multi-language / Python bridge](./docs/python-bridge.md) | TS-as-SSoT code generation and the Python runtime; TS↔Python conformance. |
+| [In-memory test adapter](./docs/testing.md) | Docker-free, in-process testing via `graphddb/testing` and `MemoryInspector`. |
+| [CDC emulator](./docs/cdc-emulator.md) | Change-event emulator (dev/test) for incremental aggregation patterns. |
+| [Benchmark](./benchmark/RESULTS.md) | Library comparison vs. hand-written AWS SDK, ElectroDB, DynamoDB Toolbox, and OneTable. |
+
+## Examples
+
+| Example | Description |
+|---------|-------------|
+| [user-permissions](./examples/user-permissions/) | Manage users, groups, and permissions with Single Table Design. Includes adjacency lists, inverted indexes, relation traversal, and `explain`. |
+| [aggregate-tree-pattern](./examples/aggregate-tree-pattern/) | Incremental tree aggregation (`siteScoreAverage`) driven by the CDC emulator: dirty propagation, throttled sweep, and recompute. |
+
+## Features
+
+- Entity metadata / decorators
+- DDBModel base class / connection management
+- Type system (SelectableOf, QueryResult, QueryKeyOf)
+- CRUD (put, update, delete, conditional writes)
+- Query / List (planner, executor, hydrator, cursor pagination)
+- Type-safe filtering: declarative server-side `filter` (FilterExpression / AppSync `ModelFilterInput`-compatible), plus `Model.col` column refs and the `cond` raw escape hatch
+- Structured segment keys (`k` tag): partial keys compile to `begins_with` at a segment boundary
+- Explainable execution plans
+- Relations (hasMany, belongsTo, hasOne, depth limits, traversal)
+- BatchGet optimization
+- Parallel relation execution (independent sub-queries, BatchGet chunks, and nested resolution are dispatched concurrently with bounded concurrency)
+- Linter (no-scan, require-limit, query-boundary, GSI ambiguity, expanded design rules)
+- ConsistentRead type constraints
+- Transaction DSL
+- Batch operations
+- CQRS Query/Command contract layer (public IF, N+1-safe composition, context boundaries) — see [docs](./docs/cqrs-contract.md)
+- Mutation → command derivation: internal write-plan composition DSL (`entityWrites` + `mutation`) behind the Command IF, multi-fragment atomic composition into one `TransactWriteItems` — see [docs](./docs/mutation-command-derivation.md)
+- Opt-in class hydration (`options.hydrate`): load a read result onto a host object; host-only, never serialized. Phase 1 (`query` top-level) implemented — see [docs](./docs/class-hydration.md)
+- Multi-language: TS-as-SSoT Python code generator + runtime, with TS↔Python conformance — see [docs](./docs/python-bridge.md)
+- In-memory test adapter (`graphddb/testing`): Docker-free unit testing — see [docs](./docs/testing.md)
+- CDC emulator: DynamoDB-Streams-equivalent change events for dev/test — see [docs](./docs/cdc-emulator.md)
+
+## License
+
+MIT