@terreno/mcp 57.1.0 → 57.3.0

package/src/docs/versioned/57.1.0/implementationPlans/pluggable-database-sqlite-research.md

@@ -0,0 +1,150 @@
+# Research: Pluggable database layer and SQLite support
+
+**Status:** Complete — decisions made, IP written: [pluggable-database-sqlite.md](pluggable-database-sqlite.md)
+**Created:** 2026-08-20
+
+## Scope statement
+
+Evaluate how to make the persistence layer of `@terreno/api` (and the packages that build on it) pluggable so that SQLite can be used as an alternative to MongoDB/Mongoose, with special attention to:
+
+- Making the current Mongoose handler one implementation behind a database-agnostic seam.
+- Options for **schema definition** that minimize disruption to existing Terreno codebases (which all author `new mongoose.Schema(...)` today).
+- Which framework capabilities can realistically be ported to SQLite in a first release versus which stay Mongo-only behind capability flags.
+
+## PRD summary
+
+From the request:
+
+- **Problem:** Terreno is MongoDB-only. Every consumer must run a Mongo replica set (even for small apps and local dev), and the positioning docs explicitly rule out relational databases. This limits adoption (SQLite/libSQL is the default for many small deployments, edge platforms, and quick starts).
+- **Goal:** Support SQLite in `@terreno/api` and any other package that needs it; make the Mongoose handler pluggable rather than hard-wired.
+- **Explicit ask:** Present a few options for how to handle schemas with the minimal amount of disruption to existing codebases.
+
+## Codebase findings
+
+### Coupling depth (where Mongoose lives today)
+
+There is **no storage abstraction seam** in the framework today — no repository, `DatabaseAdapter`, or dialect-neutral query builder. Mongoose is the schema source of truth and the runtime data API.
+
+| Layer | Depth | Key touchpoints |
+|---|---|---|
+| CRUD (`modelRouter` + executors) | Very high | `api/src/api.ts`, `api/src/sync/executors.ts`, `api/src/docLoader.ts` — `find/findById/create/doc.save()/deleteOne/countDocuments/populate`, hydrated `Document`s, soft-delete via query middleware, native `model.collection.findOne` probe |
+| Schema plugins | Very high | `api/src/plugins.ts` — `createdUpdatedPlugin`, `isDeletedPlugin`, `findExactlyOne`/`findOneOrNone`, `upsertPlugin`, custom `DateOnly` `SchemaType`; plus `configurationPlugin`, `syncPlugin`, `errorsPlugin`, `githubUserPlugin` |
+| OpenAPI generation | High | `api/src/populate.ts` `getOpenApiSpecForModel` via `mongoose-to-swagger`; reads `schema.paths`, `options.ref`, virtuals, child schemas, `instance === "Mixed"`; feeds `openApi.ts`, AJV validation (`openApiValidator.ts`), and admin metadata |
+| Auth | High | `api/src/auth.ts` `UserModel extends Model<User>` requiring passport-local-mongoose statics; `betterAuthSetup.ts` hard-wires `mongodbAdapter` off the global `mongoose.connection` |
+| Realtime / sync | Extreme (Mongo-only) | `api/src/realtime/changeStreamWatcher.ts` (`db.watch()`, resume tokens, replica set required), `api/src/sync/models.ts` (`$inc` seq counters, optional `ClientSession`), `feature-flags` `.watch()` |
+| Query language | Public API | modelRouter passes Mongo filter objects through (`$and`/`$or`, `_gte`/`_lte` → `$gte`/`$lte`, `OwnerQueryFilter` returns Mongo filters); `realtime/queryMatcher.ts` re-implements Mongo matching in memory |
+
+`mongoose` is a **peerDependency** of `@terreno/api` (`^8 || ^9`); the catalog pins `9.7.4`. Connection lifecycle is the consumer's job (`example-backend/src/utils/database.ts`); the framework assumes the global `mongoose.connection` for change streams and Better Auth.
+
+### Schema definition surface (what "schemas" means downstream)
+
+Mongoose schemas are consumed by four downstream systems, all of which would need a story under any schema-handling option:
+
+1. **OpenAPI**: `mongoose-to-swagger` reads `required`, `enum`, `default`, `readOnly`, `description`; populate/ref metadata comes from raw `schema.path(key).options.ref`.
+2. **Admin** (`admin-backend/src/adminApp.ts`): builds `/admin/config` field metadata from `getOpenApiSpecForModel` + raw `schema.paths` for refs and allowlists.
+3. **syncdb codegen** ([syncdb-codegen.md](syncdb-codegen.md), approved): generates TypeScript **from the OpenAPI spec**, not from schemas directly — so anything that produces equivalent OpenAPI keeps codegen working unchanged.
+4. **MCP** (`mcp-server/src/tools.ts` `terreno_generate_model`): scaffolds Mongoose schema source code.
+
+### Package-by-package Mongoose exposure
+
+| Package | Exposure |
+|---|---|
+| `api` | Structural (see above) |
+| `admin-backend` | Deep — `Model<any>` config, `schema.paths`, ObjectId search, modelRouter CRUD |
+| `comms` | Deep — own Mongoose models (`PushToken`, `CommsMessage`) + modelRouter + direct queries |
+| `ai` | Deep — own Mongoose models (`AIRequest`, `GptHistory`) + modelRouter + aggregation |
+| `feature-flags` | Deep + Mongo-only `.watch()` |
+| `example-backend` | Canonical consumer (connect lifecycle, models, change streams) |
+| `syncdb` (client) | **None** — transport/protocol only, but the server side of the protocol is Mongo-only today |
+| `rtk`, `ui`, `admin-frontend`, `admin-spa` | None |
+| `mcp-server` | Codegen templates emit Mongoose; local tools connect via mongoose |
+| `test` | mongodb-memory-server harness (`TERRENO_TEST_MONGODB_URI`, replica set option, transaction patches) |
+
+### Useful precedents already in-repo
+
+- **String `_id`**: synced models already use String `_id` (hex string default) — precedent for non-ObjectId primary keys.
+- **In-memory Mongo query matcher** (`realtime/queryMatcher.ts`): the framework already treats "a Mongo filter object" as an evaluable spec independent of the database.
+- **Executor seam** (`executeCreate/Update/Delete`): HTTP and sync share one write path — the natural place to insert an adapter.
+- **`TerrenoPlugin` / comms adapter pattern**: the repo already ships pluggable-provider IPs ([comms-abstraction.md](comms-abstraction.md)) — an established shape for "core interface + adapter packages".
+- **zod** (`^4.4.3`) is already a peer of `@terreno/api` and used for action bodies via `zodOpenApi.ts`.
+
+## External findings
+
+- **Better Auth** supports SQLite first-class: built-in Kysely adapter accepts `better-sqlite3`, `node:sqlite` (Node ≥ 22.5), and `bun:sqlite` instances, plus a Drizzle adapter with `provider: "sqlite"`. The Mongo-only hard-wiring in `betterAuthSetup.ts` is a Terreno choice, not a Better Auth limitation.
+- **SQLite as a document store** is a well-trodden pattern: one table per collection with a JSON column (`TEXT` + `json_valid` check), queried via the JSON1 functions (`json_extract`), with **generated columns + indexes** for hot query fields. This preserves Mongo-style nested/Mixed/array data without a relational schema migration story.
+- **Mongo-filter → SQL translation** for the subset modelRouter actually emits (equality, `$in`, `$ne`, `$gte`/`$lte`, `$and`/`$or`, `$exists`) is small and testable; the existing `queryMatcher.ts` documents the exact operator surface needed.
+- **Change streams have no SQLite equivalent.** SQLite alternatives are `data_version` pragma polling, update hooks (better-sqlite3 exposes none; libSQL/Turso has limited CDC), or an app-level outbox table. Realtime/sync on SQLite therefore needs a different backend (likely outbox-table + in-process emit) or stays Mongo-only initially.
+- **Driver options**: `bun:sqlite` (zero-dep under Bun, not Node), `better-sqlite3` (Node, native build), `node:sqlite` (release candidate, Node ≥ 22.5), `libsql` (Turso-compatible, adds remote/replica options). Drizzle supports all of them if an ORM layer is wanted.
+
+## Candidate options
+
+### Decision 1 — Schema definition (the user's headline question)
+
+**Option A — Mongoose schema stays the authoring format; adapters interpret it.**
+Consumers keep writing `new mongoose.Schema({...})` exactly as today. A `DatabaseAdapter` receives the compiled model; the SQLite adapter reads `schema.paths` (types, refs, required, enum, defaults, descriptions) to derive its storage and to feed OpenAPI/admin unchanged. Mongoose remains a dependency even for SQLite users, but only as a schema DSL + document hydrator (it can define schemas without a connection).
+
+- Disruption: **zero** for existing codebases; plugins, `strict: "throw"`, virtuals, and the model-type pattern all keep working.
+- Risk: Mongoose document semantics (hooks, `doc.save()`, query middleware) must be emulated or bypassed on SQLite; hidden Mongo-isms can leak. Mongoose stays a hard dependency of the "database-agnostic" framework, which is philosophically odd but pragmatic.
+
+**Option B — New Terreno schema DSL (framework-owned AST).**
+`defineModel({fields: {...}})` produces a framework-native schema AST that compiles to a Mongoose schema on Mongo and to SQLite DDL/JSON layout on SQLite, and directly emits OpenAPI (dropping `mongoose-to-swagger`). Existing codebases migrate via codemod; a compatibility shim could wrap existing Mongoose schemas during transition.
+
+- Disruption: **high** (every model file in every consumer changes), but one-time and mostly mechanical.
+- Payoff: the framework owns its schema story end-to-end (OpenAPI, admin, MCP codegen, future Postgres); no per-adapter schema interpretation.
+
+**Option C — zod as the schema source.**
+Models defined as zod objects (already in the tree, already OpenAPI-bridged via `zodOpenApi.ts`) plus a metadata layer for refs/indexes/defaults. Adapters derive Mongoose schemas or SQLite DDL from the zod definition.
+
+- Disruption: **high** (same migration burden as B) and zod lacks natural homes for DB concerns (indexes, refs, hooks), so a wrapper DSL emerges anyway — effectively Option B with zod as the type layer.
+
+**Option D — Adopt a multi-dialect ORM (Drizzle/Prisma) as the schema layer.**
+- Disruption: **highest**; abandons the Mongoose ecosystem (plugins, populate, discriminators) and Mongo support in Drizzle is immature. Likely reject, but listed for completeness.
+
+A staged path is possible: **A now, B later** — ship the adapter seam with Mongoose-as-DSL (A), and treat a framework-owned schema AST (B) as the eventual next-major direction once the adapter surface is proven.
+
+### Decision 2 — SQLite storage model (if A is chosen)
+
+- **A1 — Document store**: one table per model: `_id TEXT PRIMARY KEY, doc JSON`, generated columns + indexes for `queryFields` and sort keys. Arrays, nested docs, and Mixed work unchanged; Mongo-filter translation targets `json_extract`. Closest semantics to today.
+- **A2 — Relational mapping**: schema paths → real columns; arrays/Mixed still fall back to JSON columns. More natural SQL, but introduces a migration story (ALTER TABLE on schema change) and diverges from Mongo semantics.
+
+### Decision 3 — Adapter seam shape
+
+Insert a `DatabaseAdapter` (working name) behind the executor/docLoader surface: `create / findById / list(query, sort, skip, limit) / count / save(diff) / delete / populate?`, plus a `translateQuery` boundary where the Mongo-filter subset is either executed natively (Mongo) or compiled to SQL (SQLite). Capabilities beyond CRUD are declared, not assumed:
+
+```
+interface DatabaseAdapter {
+  capabilities: {changeStreams: boolean; transactions: boolean; textSearch: boolean; ...};
+}
+```
+
+Mongo-only features (change-stream realtime, syncdb server protocol, feature-flag watchers, Atlas `$search`) check capabilities and either degrade (polling/outbox) or refuse with a clear error.
+
+### Decision 4 — Auth on SQLite
+
+- Better Auth: swap the hard-wired `mongodbAdapter` for adapter-provided database handles (Kysely/Drizzle SQLite are first-class upstream).
+- Legacy JWT/passport path: `passport-local-mongoose` is Mongo-only; either implement a portable credential store on the adapter surface, or make Better Auth the required auth provider on SQLite.
+
+### Decision 5 — Packaging
+
+- `@terreno/api` keeps the interface + Mongoose adapter (default, zero breaking change); SQLite ships as `@terreno/db-sqlite` (mirrors the comms adapter-package pattern), or everything lives in `api` behind optional peer deps.
+
+## Open questions (blocking)
+
+Recorded in the planning thread; the IP will not be written until these are answered:
+
+1. Schema authoring: A (Mongoose-as-DSL, zero disruption), B (new DSL, high one-time disruption), C (zod), or staged A→B?
+2. SQLite storage model: A1 document-style vs A2 relational?
+3. v1 capability line: which of realtime/syncdb-server/feature-flag-watch/Atlas-search are Mongo-only at launch?
+4. Auth on SQLite: Better Auth required, or portable credential store for the JWT path too?
+5. Query language: keep the Mongo filter subset as the public/adapter contract, or introduce a neutral filter DSL?
+6. Package scope: `api` + example-backend only for v1, or also `admin-backend`/`comms`/`ai`/`feature-flags`?
+7. SQLite driver: `bun:sqlite` vs `better-sqlite3` vs `node:sqlite` vs `libsql` (and does Node support matter for v1)?
+8. Packaging: separate `@terreno/db-sqlite` package vs in-`api` optional dependency?
+
+## References
+
+- `api/src/api.ts`, `api/src/sync/executors.ts`, `api/src/docLoader.ts`, `api/src/plugins.ts`, `api/src/populate.ts`, `api/src/openApi.ts`, `api/src/auth.ts`, `api/src/betterAuthSetup.ts`, `api/src/realtime/changeStreamWatcher.ts`, `api/src/realtime/queryMatcher.ts`, `api/src/sync/models.ts`
+- `admin-backend/src/adminApp.ts`, `mcp-server/src/tools.ts`, `test/src/mongo/mongoServer.ts`
+- [syncdb-codegen.md](syncdb-codegen.md), [comms-abstraction.md](comms-abstraction.md), [positioning-django-rails-universal.md](positioning-django-rails-universal.md), [deploy-to-gcp.md](deploy-to-gcp.md) (GC2 future-Postgres note)
+- Better Auth SQLite adapters: https://better-auth.com/docs/adapters/sqlite ; Drizzle adapter: https://better-auth.com/docs/adapters/drizzle
+- Prior art: [PR #145](https://github.com/FlourishHealth/terreno/pull/145) (closed) — an earlier SQL-adapter plan proposing Bun.SQL multi-dialect support and JOIN-based population; superseded by this IP's adapter seam + batched population design

package/src/docs/versioned/57.1.0/implementationPlans/pluggable-database-sqlite.md

@@ -0,0 +1,247 @@
+# Pluggable database layer and SQLite adapter
+
+**Status:** Approved — decisions 2026-08-20
+**Branch:** `cursor/sqlite-pluggable-db-ip-4d52`
+**Owner:** —
+**Created:** 2026-08-20
+**Research:** [pluggable-database-sqlite-research.md](pluggable-database-sqlite-research.md)
+**Roadmap issue:** *(pending `roadmap-item` handoff)*
+
+## Goal
+
+Make the persistence layer of `@terreno/api` pluggable: extract a `DatabaseAdapter` seam under `modelRouter`, make the current Mongoose/MongoDB handler the default adapter behind it (zero behavior change), and ship a SQLite adapter (`@terreno/db-sqlite`) that supports the full CRUD + permissions + population + OpenAPI + admin surface for apps that do not need Mongo-only capabilities.
+
+Existing codebases keep authoring models exactly as today — `new mongoose.Schema({...})` with the standard plugins — on both databases.
+
+## Non-Goals
+
+- Realtime change-stream events, the syncdb server protocol, feature-flag watchers, or Atlas `$search` on SQLite (Mongo-only in v1, gated by capability flags).
+- The legacy passport-local-mongoose JWT auth path on SQLite (Better Auth is the auth provider on SQLite).
+- A Postgres adapter (the seam must not preclude one, but none ships here).
+- Data migration tooling between Mongo and SQLite.
+- Certifying `admin-backend`, `comms`, `ai`, or `feature-flags` on SQLite (they are schema-driven and expected to mostly work; certification is follow-up work).
+- A new schema DSL. Mongoose remains the schema authoring format and source of truth for OpenAPI/admin metadata.
+
+## Decisions
+
+| # | Question | Decision |
+|---|----------|----------|
+| D1 | Schema authoring | **Option A** — Mongoose schemas stay the authoring format; adapters interpret the compiled model (`schema.paths`). Mongoose remains a dependency on SQLite as schema DSL + document layer. |
+| D2 | SQLite storage model | **Option A2** — relational column mapping. Schema paths become real columns; arrays/subdocuments/Mixed/Map fall back to JSON columns. |
+| D3 | v1 capability line | CRUD, permissions, population, OpenAPI, validation, and admin metadata work on SQLite. Change streams, syncdb server, feature-flag watch, and Atlas search stay Mongo-only behind adapter capability flags. |
+| D4 | Auth on SQLite | Better Auth, via its built-in SQLite support (Kysely adapter accepts `bun:sqlite` / `better-sqlite3` / `node:sqlite` handles). Legacy passport path requires the Mongo adapter. |
+| D5 | Query language | The existing Mongo-filter subset that modelRouter emits is the adapter contract; the SQLite adapter compiles it to SQL. |
+| D6 | Package scope v1 | `@terreno/api` + `@terreno/db-sqlite` + `example-backend` (SQLite mode) + `@terreno/test` harness support. |
+| D7 | Driver | `bun:sqlite` is the default; the adapter takes an injected driver handle so `better-sqlite3` / `node:sqlite` / `libsql` work too. |
+| D8 | Packaging | New workspace package `@terreno/db-sqlite` (mirrors the comms adapter-package pattern). The interface + Mongoose adapter live in `@terreno/api`. |
+| D9 | Population | First-class requirement. `populatePaths` (single refs, arrays of refs, field selection) must behave identically on both adapters, including populated create/update/read responses and soft-delete filtering of referenced docs. |
+
+## Architecture
+
+### The adapter seam
+
+New module `api/src/db/adapter.ts` defining the contract; everything in `api.ts`, `sync/executors.ts`, `docLoader.ts`, and the permissions doc-loading path is refactored to call it instead of Mongoose model APIs directly.
+
+```typescript
+export interface DatabaseAdapter {
+  readonly capabilities: AdapterCapabilities;
+  collection<T>(model: ModelLike<T>): AdapterCollection<T>;
+  // Schema lifecycle: no-op on Mongo; DDL sync on SQLite.
+  ensureSchema(models: ModelLike<unknown>[]): Promise<void>;
+  // Auth integration: what Better Auth's `database` option receives.
+  getBetterAuthDatabase(): unknown;
+}
+
+export interface AdapterCapabilities {
+  changeStreams: boolean;   // realtime + syncdb server + feature-flag watch
+  transactions: boolean;    // sync seq claiming sessions
+  textSearch: boolean;      // Atlas $search / $autocomplete
+}
+
+export interface AdapterCollection<T> {
+  create(data: Partial<T>): Promise<HydratedDoc<T>>;
+  findById(id: string, options?: {populate?: PopulatePath[]}): Promise<HydratedDoc<T> | null>;
+  list(options: {
+    query: MongoFilter;      // the validated modelRouter filter subset
+    sort?: string | Record<string, SortDirection>;
+    skip?: number;
+    limit?: number;
+    populate?: PopulatePath[];
+  }): Promise<HydratedDoc<T>[]>;
+  count(query: MongoFilter): Promise<number>;
+  save(doc: HydratedDoc<T>): Promise<HydratedDoc<T>>;      // runs validation, plugin semantics, version check
+  delete(doc: HydratedDoc<T>, options: {soft: boolean}): Promise<void>;
+  populate(docs: HydratedDoc<T>[], paths: PopulatePath[]): Promise<void>;
+}
+```
+
+Adapter selection happens at app construction; the default preserves today's behavior with no consumer changes:
+
+```typescript
+// Mongo (default — implicit when `database` is omitted)
+new TerrenoApp({userModel: User});
+
+// SQLite
+import {sqliteAdapter} from "@terreno/db-sqlite";
+new TerrenoApp({userModel: User, database: sqliteAdapter({filename: "app.db"})});
+```
+
+### MongooseAdapter (in `@terreno/api`)
+
+Thin delegation to the exact calls the framework makes today (`find/findById/create/doc.save()/deleteOne/countDocuments/.populate()`), including the native `model.collection.findOne` soft-delete probe in `docLoader.ts`. Capabilities: all `true`. `getBetterAuthDatabase()` returns the `mongodbAdapter` client as `betterAuthSetup.ts` builds it today. All existing api tests must pass unchanged against this adapter — the Phase 1 gate.
+
+### SqliteAdapter (`@terreno/db-sqlite`)
+
+**Documents.** Rows are hydrated into real Mongoose documents with `model.hydrate(row)` — no Mongo connection required — so virtuals, methods, `toJSON({virtuals: true})`, and consumer typing all work unchanged. Writes go through `adapter.save()`, which runs `doc.validate()`, applies the framework plugin semantics natively, and persists changed paths.
+
+**Plugin semantics (behavior spec, implemented natively).**
+
+| Plugin behavior | SQLite implementation |
+|---|---|
+| `createdUpdatedPlugin` | Set `created` on insert, `updated` on every save |
+| `isDeletedPlugin` | `deleted` column; every read appends `AND deleted IS NOT 1`; `docLoader` probe reads without the filter |
+| Optimistic concurrency / `__v` | `UPDATE ... WHERE _id = ? AND __v = ?`; zero rows → the same conflict error path as Mongoose `VersionError` |
+| `findExactlyOne` / `findOneOrNone` / `upsertPlugin` statics | Work as-is — they run through `model.find`, which the framework only calls via the adapter surface |
+| Consumer `schema.pre/post("save")` document middleware | Invoked through Mongoose's hook registry around persist (best-effort; verified by conformance tests; documented limitation if a hook depends on Mongo-only internals) |
+| Consumer custom **query** middleware | Not translated to SQL beyond the built-in soft-delete/archive filters — documented limitation |
+
+**Schema → DDL mapping (D2, A2).**
+
+| Mongoose path | SQLite column |
+|---|---|
+| `_id` | `TEXT PRIMARY KEY` — default `new mongoose.Types.ObjectId().toHexString()` (matches the synced-model String-`_id` precedent) |
+| String | `TEXT` |
+| Number | `NUMERIC` |
+| Boolean | `INTEGER` (0/1) |
+| Date | `TEXT` — ISO 8601 UTC (lexicographic order = chronological; Luxon-friendly) |
+| ObjectId / ref | `TEXT` (no `FOREIGN KEY` constraint — matches Mongo's non-enforcing refs) |
+| enum | `TEXT` + `CHECK (col IN (...))` |
+| Array / subdocument / Mixed / Map | `TEXT` JSON (`CHECK (json_valid(col))`) |
+| `DateOnly` custom type | `TEXT` (`YYYY-MM-DD`) |
+| `__v` | `INTEGER NOT NULL DEFAULT 0` |
+| `required: true` | `NOT NULL` |
+| `default` | Applied in the JS layer (supports function defaults), not DDL |
+| `index` / `unique` / `sparse` | `CREATE [UNIQUE] INDEX`; sparse → partial index `WHERE col IS NOT NULL` |
+
+**DDL sync.** `ensureSchema(models)` runs at startup: creates missing tables/columns/indexes when `autoMigrate: "additive"` (default in dev); destructive changes (type change, drop, constraint change) are never auto-applied — the adapter throws a clear `APIError`-style startup error and writes the proposed SQL to a migration file for manual review. Connection defaults: WAL mode, `busy_timeout`, `foreign_keys=OFF`.
+
+**Query translation.** The adapter compiles exactly the filter surface modelRouter emits: top-level equality on `queryFields`, `$in/$nin/$ne/$exists/$gt/$gte/$lt/$lte`, `$and`/`$or`, and the `_gte`/`_lte` date-range expansion. Column-backed paths compile to plain SQL predicates; nested/JSON paths use `json_extract`; membership in array columns uses `EXISTS (SELECT 1 FROM json_each(col) ...)`. Any operator outside the contract → 400 `APIError` (same as an invalid query today). Sort/skip/limit/count map directly. `realtime/queryMatcher.ts` documents the operator semantics and seeds the conformance fixtures.
+
+### Population (D9)
+
+Population is a first-class, conformance-tested part of the adapter contract — not an afterthought:
+
+- **Contract:** `populate(docs, paths)` with `PopulatePath = {path, fields?}`, plus a `populate` option on `findById`/`list`. The executors' post-create/post-update re-read with population goes through the same path, so populated response shapes are identical across adapters.
+- **Ref discovery:** from `schema.path(key).options.ref` and array-element refs — the same metadata `getOpenApiSpecForModel` reads, so the populated schemas in `/openapi.json` stay correct on SQLite with no OpenAPI changes.
+- **SQLite strategy:** batched two-phase loads, mirroring Mongoose's own populate. Per path: collect distinct FK values across the page, run one `SELECT _id, <fields> FROM <ref_table> WHERE _id IN (...)`, attach results (single ref → object, array of refs → array). No N+1, no JOIN row-explosion, and `fields` projection happens in SQL.
+- **Soft-delete parity:** when the referenced model has `isDeletedPlugin`, the batched lookup appends the soft-delete filter — a soft-deleted target populates to `null`/absent exactly as it does through Mongoose query middleware today.
+- **Missing refs:** dangling FK values populate to `null` (Mongoose parity).
+- **v1 scope:** single-level populate with field selection — the full surface `populatePaths` supports today. Nested (multi-hop) populate is future work on both adapters.
+- **Conformance suite:** single ref, array of refs, field selection, dangling ref, soft-deleted target, populate-on-create/update responses — run against both adapters.
+
+### Capability gating (D3)
+
+`TerrenoApp` checks `adapter.capabilities` at registration time:
+
+- Realtime/change-stream registration, syncdb route registration, and feature-flag watchers require `changeStreams: true`; on SQLite they fail fast at startup with an actionable error naming the capability (not a runtime surprise).
+- Atlas `$search`/`$autocomplete` stubs require `textSearch: true`.
+- Sync seq sessions require `transactions: true`.
+
+### Auth (D4)
+
+`betterAuthSetup.ts` stops reaching for the global `mongoose.connection` and instead calls `adapter.getBetterAuthDatabase()`. The Mongo adapter returns the `mongodbAdapter` client (today's behavior); the SQLite adapter returns its driver handle, which Better Auth's built-in Kysely adapter accepts directly for `bun:sqlite`, `better-sqlite3`, and `node:sqlite`. Better Auth schema tables are created via its migration tooling, invoked from `ensureSchema`. `setupAuth`/passport (`auth.ts`) is gated to the Mongo adapter with a clear startup error otherwise.
+
+## Models
+
+No new Mongoose models. `@terreno/db-sqlite` maintains one internal metadata table (`_terreno_schema`) recording the applied DDL per model for additive-migration diffing.
+
+## APIs
+
+No route surface changes. `modelRouter` options, generated endpoints, pagination envelope (`{data, limit, more, page, total}`), array-field endpoints, permissions, `queryFilter`, and OpenAPI output are identical on both adapters. New public API:
+
+- `TerrenoAppOptions.database?: DatabaseAdapter` (default: Mongoose adapter).
+- `@terreno/db-sqlite` exports `sqliteAdapter(options: {filename: string; driver?: SqliteDriverHandle; autoMigrate?: "additive" | "off"})`.
+- `@terreno/api` exports the `DatabaseAdapter` / `AdapterCollection` / `AdapterCapabilities` types and `mongooseAdapter()`.
+
+## Notifications
+
+None.
+
+## UI
+
+None (backend-only).
+
+## Phases
+
+### Phase 1 — Adapter seam + Mongoose adapter (no behavior change)
+
+Define the adapter interfaces; implement `MongooseAdapter`; refactor `api.ts`, `sync/executors.ts`, `docLoader.ts`, and permissions doc loading to consume the adapter; wire `TerrenoAppOptions.database`; add capability checks to realtime/sync/feature-flag/search registration (all pass on Mongo). Gate: entire existing api test suite green with zero test edits.
+
+### Phase 2 — `@terreno/db-sqlite` core
+
+Package scaffold; DDL mapping + `ensureSchema` additive migration; CRUD; query translation; sort/pagination/count; plugin semantics (timestamps, soft delete, `__v` conflict); **population** (batched loads, soft-delete parity, field selection); enum checks and indexes; driver injection with `bun:sqlite` default. Adapter conformance suite in `@terreno/api` run against both adapters.
+
+### Phase 3 — Auth + capability hardening
+
+`getBetterAuthDatabase()` seam; Better Auth on SQLite end-to-end (signup/login/session → `req.user`); passport path gated to Mongo; startup errors for missing capabilities; `@terreno/test` gains `TERRENO_TEST_DB=sqlite` (temp-file DB per suite, no mongod download).
+
+### Phase 4 — Example, docs, and integration
+
+`example-backend` `DB_ADAPTER=sqlite` mode (skips change-stream features via capability flags); verify `/openapi.json` and admin `/admin/config` output parity; docs (how-to: run Terreno on SQLite; explanation: adapter capabilities; update positioning doc's "MongoDB-only" claims); MCP `terreno_generate_model`/bootstrap templates mention adapter choice; changelog + upgrade note (none required for existing apps).
+
+## Feature Flags & Migrations
+
+- No feature flag: adapter choice is an explicit constructor option; omitting it preserves today's behavior exactly.
+- No data migrations for existing apps.
+- SQLite DDL migrations: additive-auto by default in dev; destructive changes always manual (generated SQL file + startup error). Documented in the how-to.
+
+## Activity Log & User Updates
+
+n/a.
+
+## Not Included / Future Work
+
+- Postgres adapter (the seam is designed dialect-neutral; JSON handling and DDL mapping are the SQLite-specific parts).
+- Realtime/outbox-based change feed for SQLite (would unblock syncdb + feature-flag watch off-Mongo).
+- Nested (multi-hop) populate on both adapters.
+- Certifying `admin-backend`, `comms`, `ai`, `feature-flags` on SQLite.
+- Mongo↔SQLite data migration tooling.
+- passport-local credential store on SQLite.
+
+## Files to Create / Modify
+
+**Create**
+
+- `api/src/db/adapter.ts` — interfaces + capability types
+- `api/src/db/mongooseAdapter.ts` — default adapter
+- `api/src/db/conformance/` — shared adapter conformance suite (CRUD, query, populate, plugins)
+- `db-sqlite/` — new workspace package: `src/sqliteAdapter.ts`, `src/ddl.ts`, `src/queryCompiler.ts`, `src/populate.ts`, `src/drivers.ts`, tests
+- `docs/how-to/run-on-sqlite.md`, `docs/explanation/database-adapters.md`
+
+**Modify**
+
+- `api/src/api.ts`, `api/src/sync/executors.ts`, `api/src/docLoader.ts`, `api/src/permissions.ts` — consume adapter
+- `api/src/terrenoApp.ts` — `database` option, capability gating
+- `api/src/betterAuthSetup.ts` — `getBetterAuthDatabase()` seam
+- `api/src/auth.ts` — Mongo-gate the passport path
+- `api/src/index.ts` — export adapter types + `mongooseAdapter`
+- `test/src/` — `TERRENO_TEST_DB=sqlite` harness mode
+- `example-backend/src/server.ts` + `utils/database.ts` — `DB_ADAPTER=sqlite` mode
+- `mcp-server/src/tools.ts` / bootstrap templates — adapter-aware scaffolds
+- Root `package.json` catalog + workspaces, CI workflows for the new package
+
+## Task List
+
+[docs/tasks/pluggable-database-sqlite.md](../tasks/pluggable-database-sqlite.md)
+
+## Acceptance Criteria
+
+- [ ] Existing api test suite passes unchanged with the Mongoose adapter as default (Phase 1 gate).
+- [ ] A modelRouter app (models, permissions, queryFilter, pagination, sorting, array endpoints) runs on SQLite with byte-identical response envelopes.
+- [ ] `populatePaths` behaves identically on both adapters: single ref, array of refs, field selection, dangling ref → `null`, soft-deleted ref filtered, populated create/update responses — verified by the conformance suite.
+- [ ] `/openapi.json` for the same models is identical on Mongo and SQLite.
+- [ ] Better Auth signup/login/session works on SQLite; passport path fails fast with a clear error.
+- [ ] Realtime, syncdb, and feature-flag registration on SQLite fail fast at startup with capability-named errors.
+- [ ] `ensureSchema` creates tables/columns/indexes additively; destructive changes error with generated SQL.
+- [ ] `example-backend` boots and serves CRUD in `DB_ADAPTER=sqlite` mode with no mongod running.
+- [ ] `bun run lint`, `bun run compile`, and api + db-sqlite tests green in CI.

package/src/docs/versioned/57.1.0/implementationPlans/positioning-django-rails-universal.md

@@ -0,0 +1,147 @@
+# Implementation Plan: Positioning — "Django/Rails for TypeScript, with Universal Apps"
+
+**Status:** Draft — blocking questions open
+**Roadmap issue:** https://github.com/FlourishHealth/terreno/issues/1008
+**Priority:** High
+**Effort:** Small batch
+**Owner:** unassigned
+**Created:** 2026-07-27
+**Program:** [OSS launch](oss-launch-program.md)
+**Depends on:** [`rtk-to-syncdb-migration-docs`](rtk-to-syncdb-migration-docs.md) (the architecture diagram and frontend story must be correct before it is repeated everywhere)
+**RTK deprecation flag:** **Partial** — tasks that touch architecture diagrams, package lists, and the integration flow are `[RTK]` marked and must run after #869.
+
+## Goal
+
+Say the same thing about Terreno everywhere, and say it well. Today the README makes a strong, well-written case ("batteries-included framework… Django for Python web development") but the docs landing page says "shared packages for full-stack applications with React Native and Express/Mongoose", `AGENTS.md` says "a monorepo containing shared packages", and npm package descriptions vary. A visitor's impression depends on which door they came through.
+
+The agreed framing, from the program doc:
+
+> **Terreno is Django/Rails for TypeScript — with universal app support.**
+
+Three pillars in priority order: **batteries included**, **universal by default**, **AI-native**.
+
+## Non-Goals
+
+- Adding governance files (that is [`oss-governance-baseline`](oss-governance-baseline.md)).
+- Writing new tutorials or reference pages (those are separate IPs).
+- Visual redesign of the docs site or a logo/brand refresh.
+- Renaming packages or the project.
+
+## Blocking questions
+
+**Recorded 2026-07-29** (defaults accepted).
+
+| # | Decision |
+|---|----------|
+| PO1 | Tagline **A** for README/docs H1; short variant **C** for social/repo description |
+| PO2 | Lead with **Django/Rails** |
+| PO3 | **Universal app** (not cross-platform) |
+| PO4 | AI-native pillar **co-equal** with batteries-included |
+| PO5 | README roadmap: **two-line summary + link `ROADMAP.md`** |
+| PO6 | **Trim README** — move maintainer content to CONTRIBUTING/docs |
+| PO7 | AI pillar names **both** MCP tools and `/terreno-*` pipeline, always paired |
+
+## Architecture
+
+### Canonical copy blocks
+
+Define the wording once, in `docs/explanation/positioning.md`, and have every other surface reuse it verbatim:
+
+| Block | Length | Used by |
+|-------|--------|---------|
+| `tagline` | one line | README H1 subtitle, docs site `tagline`, GitHub repo description, npm `description` prefix |
+| `elevator` | 2–3 sentences | README intro, docs landing intro, package READMEs |
+| `pillars` | three bullets | README, docs landing, `AGENTS.md` |
+| `aiPillar` | 2–3 sentences | Expansion of the third pillar wherever it gets more than a bullet. Must name **both** layers per PO7: the MCP tool layer and the `/terreno-*` process layer |
+| `pitch` | ~150 words | Blog post intro, conference abstracts, social |
+
+### Surfaces to align
+
+| Surface | Current | Action |
+|---------|---------|--------|
+| `README.md` | Strong but long and mixed-audience | Restructure: tagline → elevator → pillars → quickstart → packages → links. Move maintainer content out |
+| `docs/README.md` | "shared packages… React Native and Express/Mongoose" | Replace with tagline + elevator + pillars; add the complete package table |
+| `website/docusaurus.config.ts` | `tagline` field | Set to the canonical tagline; check `title` and og metadata |
+| `website/src/pages/index.tsx` (if present) | unknown | Landing hero uses tagline + pillars + one code sample |
+| `AGENTS.md` / `CLAUDE.md` | "monorepo containing shared packages" | Lead with the positioning so agents describe Terreno correctly to users |
+| `CLAUDE-consumer.md` | consumer-facing template | Lead with the positioning |
+| Every published `package.json` `description` | inconsistent | Standardize: `<what it does> — part of Terreno, the batteries-included TypeScript framework for universal apps.` |
+| Package READMEs | some are stubs pointing at `.claude/` | Handled by [`docs-reference-coverage`](docs-reference-coverage.md); this IP supplies the shared intro block |
+| GitHub repo description + topics | unknown | Set description to the short variant; add topics (`typescript`, `react-native`, `expo`, `express`, `mongoose`, `framework`, `mcp`, `ai`, `universal-apps`) |
+
+### The comparison table
+
+The Django/Rails claim invites the obvious question. `docs/explanation/positioning.md` should answer it head-on with an honest table — what maps, what does not:
+
+| Django / Rails concept | Terreno equivalent | Honest caveat |
+|------------------------|--------------------|----------------|
+| Models + ORM | Mongoose schemas + `@terreno/api` plugins | Document store, not relational; no migrations framework |
+| `ModelViewSet` / scaffolds | `modelRouter` | REST only; no GraphQL |
+| Django admin | `@terreno/admin-backend` + `admin-frontend` / `admin-spa` | Younger; see the admin parity IP for the gap list |
+| Auth + permissions | Better Auth + `Permissions` / RBAC | RBAC shipped (`rbac-permissions.md`, Complete) |
+| Templates / views | `@terreno/ui` components, one codebase for iOS/Android/web | SSR is not shipped yet — see [`web-ssr-and-admin-spa`](web-ssr-and-admin-spa.md) |
+| `manage.py` / generators | MCP server tools (the *tool* layer) + the `/terreno-*` SDLC pipeline (the *process* layer) | Agent-driven rather than CLI-driven. Django gives you `manage.py startapp`; Terreno gives you a reviewed path from a request to a mergeable PR |
+| Celery / ActiveJob | Not shipped | On the roadmap |
+| Migrations | No migrations framework | Schema evolution is convention + the `mongoose-schema-safety` skill |
+
+Being straight about the gaps is what makes the analogy credible rather than marketing.
+
+## Models / APIs / Notifications
+
+None.
+
+## UI
+
+Docs site landing page and README only. No application UI.
+
+## Phases
+
+1. **Canonical copy** — write `docs/explanation/positioning.md` with the five copy blocks and the comparison table.
+2. **README restructure** — apply the new structure, move maintainer content out.
+3. **Docs site** — landing page, config tagline, docs index.
+4. **Agent surfaces** — `AGENTS.md`, `CLAUDE.md`, `CLAUDE-consumer.md`, and the MCP overview resource.
+5. **Metadata** — `package.json` descriptions, GitHub repo description and topics.
+
+## Feature Flags & Migrations
+
+None. Moving README sections to `CONTRIBUTING.md` and `docs/` requires checking for inbound links from the docs site and agent rules.
+
+## Not Included / Future Work
+
+- Logo, brand palette, or docs site visual redesign.
+- Landing-page marketing site separate from the docs site.
+- Case studies or testimonials.
+- Renaming `@terreno/*` packages.
+
+## Files to Create / Modify
+
+**Create**
+
+- `docs/explanation/positioning.md`
+
+**Modify**
+
+- `README.md`
+- `docs/README.md`
+- `website/docusaurus.config.ts`, `website/src/pages/index.tsx` (if it exists)
+- `AGENTS.md`, `CLAUDE.md`, `CLAUDE-consumer.md`, `.rulesync/rules/00-root.md`
+- `CONTRIBUTING.md` (receives the maintainer content moved out of the README)
+- `mcp-server/src/docs/resources/overview.md`
+- Every published `package.json` (`description` field)
+
+## Task List
+
+See [`docs/tasks/positioning-django-rails-universal.md`](../tasks/positioning-django-rails-universal.md).
+
+## Acceptance Criteria
+
+- [ ] `docs/explanation/positioning.md` defines the five copy blocks and the honest comparison table.
+- [ ] The canonical tagline appears verbatim in `README.md`, `docs/README.md`, `website/docusaurus.config.ts`, and the GitHub repo description.
+- [ ] The phrase "Django/Rails for TypeScript" appears within the first two sentences of both `README.md` and `docs/README.md`.
+- [ ] "universal app" is the term used for the frontend in all five surfaces; no surface says "cross-platform".
+- [ ] The three pillars appear in the same order with consistent wording in `README.md`, `docs/README.md`, and `AGENTS.md`.
+- [ ] `README.md` contains no Flourish-internal instructions, GCP project names, or the feature-flag migration walkthrough; that content lives in `CONTRIBUTING.md`, `infra/flourish/`, or `docs/how-to/`.
+- [ ] Every published package's `description` follows the standard pattern and mentions Terreno.
+- [ ] The comparison table lists at least three honest gaps and links the IPs tracking them.
+- [ ] No surface claims an unshipped feature as available.
+- [ ] `bun run website:build` and `bun run rules:check` pass.