@lunora/cli 1.0.0-alpha.47 → 1.0.0-alpha.470

package/skills/lunora-setup-hyperdrive/SKILL.md

@@ -1,90 +1,65 @@
 ---
 name: lunora-setup-hyperdrive
-description: Connects an existing Postgres/MySQL database to a Lunora app from an action via Cloudflare Hyperdrive. Use for `@lunora/hyperdrive`, `ctx.sql`, `createHyperdrive` + `fromPostgresJs`/`fromNodePg`/`fromMysql2`, the `HYPERDRIVE` binding, `wrangler hyperdrive create`, and the action-only / non-reactive guardrails (live queries don't track external writes).
+description: Connects an existing, externally owned Postgres/MySQL database to a Lunora app through Cloudflare Hyperdrive. Lunora does not own its schema. Covers `@lunora/hyperdrive`, the action-only, non-reactive `ctx.sql` client (`createHyperdrive` + `fromPostgresJs`/`fromNodePg`/`fromMysql2`, wired with the app builder's `.hyperdrive()`), the `HYPERDRIVE` binding and `wrangler hyperdrive create`, and making external rows live through a projection or a `.source()` ingest table. Use when the user wants to "query our existing Postgres", "read the legacy DB from an action", "connect Neon/RDS/Supabase", or sync external rows into Lunora, or sees `ctx.sql` / `hyperdrive_outside_action` errors. For storing Lunora's own `.global()` tables in Postgres/MySQL with reactive live queries, use `lunora-setup-hyperdrive-global` instead.
 ---
 
 # Lunora Setup Hyperdrive
 
-Wire an **existing** Postgres/MySQL database into a Lunora app using
-`@lunora/hyperdrive`, which surfaces a [Cloudflare
-Hyperdrive](https://developers.cloudflare.com/hyperdrive/) binding's connection
-string and a driver-agnostic `ctx.sql` client — usable **only inside an
-`action`**.
-
-> **Integrate, don't replace.** Hyperdrive talks to a database Lunora has no
-> visibility into. Queries through `ctx.sql` are **non-deterministic** (forbidden
-> in `query`/`mutation`, enforced by the `hyperdrive_outside_action` advisor
-> lint) and external writes are **invisible to Lunora live queries** —
-> subscriptions will NOT re-run when external rows change. Use it to read/write a
-> legacy DB from an action; if you want that data reactive, write a projection
-> into a `defineSchema` DO/D1 table.
-
-## When to Use
-
-- Reading/writing an existing Postgres or MySQL database from a Lunora action.
-- Incrementally migrating off a legacy DB while standing up Lunora alongside it.
-- Backfilling/syncing external rows into reactive Lunora tables (projection).
-
-## When Not to Use
-
-- You want the external data to be **reactive** out of the box — it can't be;
-  Hyperdrive writes don't hit the change-feed. Project into a `defineSchema`
-  table instead.
-- You're reaching for `ctx.sql` inside a `query` or `mutation` — forbidden by
-  design (the advisor lint flags it). Move the access into an `action`.
-- The app has no Lunora backend yet — use `lunora-quickstart` first.
-- You'd be replacing `defineSchema` entirely — then you lose realtime, OCC,
-  optimistic updates, the offline queue, and the advisors. Use Lunora's own data
-  layer for app state.
-
-## Workflow
-
-1. Install `@lunora/hyperdrive` and one SQL driver.
-2. Create the Hyperdrive config (`wrangler hyperdrive create`) and add the
-   `HYPERDRIVE` binding to `wrangler.jsonc`.
-3. Regenerate types with `lunora codegen` so `ctx.sql` appears on `ActionCtx`.
-4. Read/write the external DB from an action via `ctx.sql`.
-5. (Optional) Project external rows into a `defineSchema` table to make them
-   reactive.
-
-## Step 1: Install the package + a driver
+`@lunora/hyperdrive` exposes a Cloudflare Hyperdrive binding as a
+driver-agnostic `ctx.sql` client that is available only inside an `action`. It integrates
+a database Lunora doesn't own. Don't use it to replace `defineSchema`, because you'd lose
+realtime, OCC, optimistic updates, the offline queue and the advisors.
+
+> **`ctx.sql` is not reactive.** Lunora cannot see this database, so subscriptions
+> do not re-run when its rows change, including rows you write through `ctx.sql`.
+> Queries are non-deterministic, so `ctx.sql` exists only on `ActionCtx`.
+> The `hyperdrive_outside_action` advisor lint flags any use in a
+> `query`/`mutation`. To make the data live, see Step 5. If you instead want Lunora-owned tables
+> stored in Postgres/MySQL and fully reactive, use `lunora-setup-hyperdrive-global`.
+
+## Step 1: Install the package + one driver
 
 ```bash
 pnpm add @lunora/hyperdrive
-# choose ONE driver (optional peer deps — none is bundled):
-pnpm add postgres        # postgres.js  → fromPostgresJs ($1, $2 placeholders)
-# or
-pnpm add pg              # node-postgres → fromNodePg     ($1, $2 placeholders)
-# or
-pnpm add mysql2          # mysql2        → fromMysql2      (?  placeholders)
+pnpm add postgres   # → fromPostgresJs   ($1, $2 placeholders)
+# or: pnpm add pg      → fromNodePg       ($1, $2)
+# or: pnpm add mysql2  → fromMysql2       (?)
 ```
 
+The drivers are optional peer dependencies, so none ships with the package.
+The package never rewrites SQL, so use your driver's native placeholders.
+
 ## Step 2: Create the binding
 
+Nothing to write by hand: `lunora dev` and `lunora deploy` add a `HYPERDRIVE` binding
+to `wrangler.jsonc` with a placeholder `id`.
+
+- **Dev.** `ctx.sql` runs on an embedded Postgres (PGlite), data in
+  `.wrangler/state/lunora-pglite`. Install its two optional packages:
+
 ```bash
-wrangler hyperdrive create my-db --connection-string="postgres://user:pass@host:5432/db" # gitleaks:allow -- placeholder, not a real secret
+pnpm add -D @electric-sql/pglite @electric-sql/pglite-socket
 ```
 
-This prints an `id`. Add the binding to `wrangler.jsonc`, using
-`localConnectionString` so `lunora dev` connects to your DB directly (no edge
-proxy locally):
-
-```jsonc
-{
-    "hyperdrive": [
-        {
-            "binding": "HYPERDRIVE",
-            "id": "<the id from the command above>",
-            "localConnectionString": "postgres://user:pass@localhost:5432/db", // gitleaks:allow -- placeholder, not a real secret
-        },
-    ],
-}
+To use your own database, set `localConnectionString` on the binding or
+`CLOUDFLARE_HYPERDRIVE_LOCAL_CONNECTION_STRING_<BINDING>`; neither is overridden.
+
+- **Deploy.** Give `lunora deploy` the origin and a token that may edit Hyperdrive:
+
+```bash
+LUNORA_HYPERDRIVE_URL="postgres://user:pass@host:5432/db" lunora deploy # gitleaks:allow -- placeholder, not a real secret
 ```
 
-Lunora validates this binding (errors if `binding` is missing; warns if `id` is
-empty/placeholder — it can't connect), but it does **not** auto-write the `id`:
-that's a remote resource only `wrangler hyperdrive create` can mint. Importing
-`@lunora/hyperdrive` surfaces a hint reminding you to add it.
+(`CLOUDFLARE_API_TOKEN` with Hyperdrive: Edit, and `CLOUDFLARE_ACCOUNT_ID`; the variable is
+`LUNORA_<BINDING>_URL`.) Lunora creates the config, or reuses one of the same name, and writes
+its `id` into `wrangler.jsonc`. The connection string is never written or logged.
+
+To manage the config yourself, run `wrangler hyperdrive create my-db --connection-string=...`
+and put the `id` in the binding. Validation fails without a `binding`.
+
+Hyperdrive caches read results (up to 60 s plus 15 s stale by default) and a write does not
+invalidate them. For reads that must see your own writes, add a second config created with
+`--caching-disabled` and bind it under another name.
 
 ## Step 3: Regenerate types
 
@@ -92,63 +67,131 @@ that's a remote resource only `wrangler hyperdrive create` can mint. Importing
 lunora codegen
 ```
 
-When codegen sees `ctx.sql` used, it adds `sql: SqlClient` to **`ActionCtx`
-only** — never `QueryCtx`/`MutationCtx` — with a JSDoc restating the
-determinism/realtime caveat.
+When `ctx.sql` is used, codegen adds `readonly sql: SqlClient` to `ActionCtx` only. It also
+emits a `.hyperdrive()` method on the generated app builder.
 
-## Step 4: Use `ctx.sql` from an action
+## Step 4: Wire the client on the app builder
 
-```ts
+`ctx.sql` is `readonly`, so assigning it in a handler is a `TS2540` error.
+Codegen can't build the client because it depends on your driver, so supply a
+thunk instead. It runs once per shard construction, not per request:
+
+```ts title="src/server.ts"
+import type { HyperdriveLike } from "@lunora/hyperdrive";
 import { createHyperdrive, fromPostgresJs } from "@lunora/hyperdrive";
-import postgres from "postgres";
 
-import { action, v } from "@lunora/server";
+import { defineApp } from "../lunora/_generated/app.js";
 
-export const listLegacyOrders = action.input({ orgId: v.string() }).action(async ({ ctx, args: { orgId } }) => {
-    const { connectionString } = createHyperdrive(ctx.env.HYPERDRIVE);
-    ctx.sql = fromPostgresJs(postgres(connectionString));
+const app = defineApp<Env>()
+    .shard((env) => env.SHARD)
+    .hyperdrive((env) =>
+        fromPostgresJs(async () => {
+            const { default: postgres } = await import("postgres");
+
+            return postgres(createHyperdrive(env.HYPERDRIVE as HyperdriveLike).connectionString, { fetch_types: false, max: 5 });
+        }),
+    )
+    .build();
+
+export const ShardDO = app.ShardDO;
+```
+
+The adapters take a loader, not a driver. It runs on the first query, so the
+driver stays out of the Worker's startup graph; never `import postgres from
+"postgres"` statically in the entry. For `pg`, the loader may `await
+client.connect()` before returning.
+
+Then query from any action:
 
-    return ctx.sql.query<{ id: string; total: number }>("select id, total from orders where org = $1", [orgId]);
+```ts
+import { LunoraError } from "lunorash/server";
+
+import { action, v } from "#lunora/_generated/server.js";
+
+export const listLegacyOrders = action.input({ orgId: v.string() }).action(async ({ ctx, args: { orgId } }) => {
+    if (!ctx.auth.userId) {
+        throw new LunoraError("UNAUTHORIZED", "not signed in");
+    }
+
+    // `orgId` comes from the client: the join scopes it to orgs the caller belongs to.
+    return ctx.sql.query<{ id: string; total: number }>(
+        "select o.id, o.total from orders o join org_members m on m.org_id = o.org where o.org = $1 and m.user_id = $2",
+        [orgId, ctx.auth.userId],
+    );
 });
 ```
 
-The package never rewrites SQL — use your driver's native placeholders (`$1` for
-Postgres, `?` for MySQL).
+**Verify:** call the action from the studio's function runner and check that it returns rows.
+
+## Step 5 (optional): Make external data reactive
 
-## Step 5 (optional): Make it reactive via a projection
+Pick one approach:
 
-External writes don't hit the change-feed, so a subscription won't re-fire on a
-Postgres change. To make external data reactive, write a projection into a
-`defineSchema` table from the same action — that write _is_ tracked:
+**A. Projection from an action.** This works when your app is the writer. Write the row into a
+`defineSchema` table through a mutation. That write is tracked:
 
 ```ts
+import { LunoraError } from "lunorash/server";
+
+import { internal } from "#lunora/_generated/internal.js";
+
 const [row] = await ctx.sql.query<{ id: string; total: number }>("select id, total from orders where id = $1", [id]);
 
-// This write re-runs live queries reading `orders`:
-await ctx.runMutation("orders:upsert", { id: row.id, total: row.total });
+if (!row) {
+    throw new LunoraError("NOT_FOUND", `order ${id} not found`);
+}
+
+// Re-runs live queries over `orders`. ctx.run* takes a generated reference, not a "file:fn" string.
+await ctx.runMutation(internal.orders.upsert, { id: row.id, total: row.total });
 ```
 
+**B. A `.source()` table.** This works when other systems also write the database. Lunora polls
+the external query on the shard's alarm, diffs the result, materializes it into the
+DO's SQLite and notifies subscribers, so you don't write an action or a cron:
+
+```ts
+// lunora/schema.ts
+messages: defineTable({ body: v.string(), channelId: v.string() })
+    .shardBy("channelId")
+    .source({
+        binding: "HYPERDRIVE",
+        query: 'select id, body, channel_id as "channelId" from messages where channel_id = $1',
+        tenantBy: (shardKey) => [shardKey], // required under .shardBy(): the tenant boundary
+    }),
+```
+
+```ts
+// worker entry: one client per binding name
+.sourceClient((env, binding) => fromPostgresJs(async () => {
+    const { default: postgres } = await import("postgres");
+
+    return postgres((env[binding] as { connectionString: string }).connectionString, { fetch_types: false, max: 5 });
+}))
+```
+
+How `.source()` behaves:
+
+- Freshness depends on the poll interval: every alarm tick by default,
+  `refresh: { everyMs }` to throttle it, or `"manual"`.
+- The table is read-only from Lunora: rows come only from the ingest loop.
+- The default `mode: "full-pull"` re-reads the whole slice on every tick, which
+  works up to about 10k rows. For larger slices, use `mode: "incremental"` with a
+  `cursor`, plus either `reconcileEveryMs` or `softDeleteColumn` so deletes are
+  still detected.
+- A source table can't also be `.global()`.
+
 ## Common Pitfalls
 
-1. **`ctx.sql` in a query/mutation.** It's action-only by design; the
-   `hyperdrive_outside_action` advisor lint flags it. Move the SQL into an
-   action.
-2. **Expecting external writes to be reactive.** They aren't — Lunora can't see
-   them. Project into a `defineSchema` table if you need reactivity.
-3. **Placeholder / empty `id`.** A Hyperdrive binding with no real `id` can't
-   connect (Lunora warns). Run `wrangler hyperdrive create` and paste the id.
-4. **Bundling a driver as a hard dependency.** Drivers are optional peer deps —
-   install the one you use; don't assume one ships with the package.
-5. **Wrong placeholder syntax.** `$1, $2` for Postgres drivers, `?` for mysql2.
-   The package does not rewrite SQL.
-
-## Checklist
-
-- [ ] `@lunora/hyperdrive` + one driver (`postgres`/`pg`/`mysql2`) installed.
-- [ ] `wrangler hyperdrive create` run; `HYPERDRIVE` binding added to
-      `wrangler.jsonc` with a real `id` and a `localConnectionString` for dev.
-- [ ] `lunora codegen` run so `ctx.sql` is typed on `ActionCtx`.
-- [ ] All `ctx.sql` access lives in actions (advisor clean — no
-      `hyperdrive_outside_action`).
-- [ ] If reactivity is needed, external rows are projected into a `defineSchema`
-      table.
+1. **`ctx.sql` in a query/mutation.** It's action-only. Move the SQL into an action.
+2. **Expecting `ctx.sql` writes to update subscriptions.** They don't. Use Step 5.
+3. **Placeholder `id` at deploy.** The deploy stops until `LUNORA_<BINDING>_URL` is set
+   (Lunora then creates the config) or the real `id` is pasted in. An empty `id` is not
+   provisioned: wrangler rejects it, so remove the entry and let Lunora add it again.
+4. **Wrong placeholders.** Use `$1` for `postgres`/`pg` and `?` for `mysql2`.
+5. **Array or JSON params on postgres.js with `fetch_types: false`.** A JS array bound to
+   `= ANY($1)` fails as a malformed array literal: bind `JSON.stringify(list)` to
+   `IN (SELECT jsonb_array_elements_text($1::text::jsonb))`. Cast JSON params `$1::text::jsonb`,
+   since a bare `$1::jsonb` gets your string JSON-encoded twice.
+6. **Unscoped `.source()` under `.shardBy()`.** Without `tenantBy`, every tenant's
+   rows are copied into each shard. `defineSchema` throws, and the
+   `external_source_unscoped` lint fails the build.