opencastle 0.33.9 → 0.34.0

package/src/orchestrator/plugins/convex/references/migrations.md

@@ -0,0 +1,259 @@
+# Convex Schema & Data Migrations
+
+Safe migration of Convex schemas and data when making breaking changes.
+
+## Core Constraint
+
+Convex will not let you deploy a schema that does not match the data at rest:
+- Cannot add a required field if existing documents don't have it
+- Cannot change a field's type if existing documents have the old type
+- Cannot remove a field from the schema if existing documents still have it
+
+## Safe Changes (No Migration Needed)
+
+Adding an optional field or a new table requires no migration:
+
+```typescript
+// Adding optional field — safe
+users: defineTable({
+  name: v.string(),
+  bio: v.optional(v.string()),  // new optional field
+})
+
+// Adding new table — safe
+posts: defineTable({
+  userId: v.id("users"),
+  title: v.string(),
+}).index("by_user", ["userId"])
+```
+
+## Widen-Migrate-Narrow Workflow
+
+Every breaking migration follows this multi-deploy pattern:
+
+**Deploy 1 — Widen:**
+1. Update schema to allow both old and new formats (e.g., add optional new field)
+2. Update code to handle both formats when reading
+3. Update code to write the new format for new documents
+4. Deploy
+
+**Between deploys — Migrate data:**
+5. Run migration to backfill existing documents
+6. Verify all documents are migrated
+
+**Deploy 2 — Narrow:**
+7. Update schema to require the new format only
+8. Remove code that handles the old format
+9. Deploy
+
+## Common Patterns
+
+### Adding a Required Field
+
+```typescript
+// Deploy 1: make optional
+users: defineTable({
+  name: v.string(),
+  role: v.optional(v.union(v.literal("user"), v.literal("admin"))),
+})
+
+// Migration: backfill
+export const addDefaultRole = migrations.define({
+  table: "users",
+  migrateOne: async (ctx, user) => {
+    if (user.role === undefined) {
+      await ctx.db.patch(user._id, { role: "user" });
+    }
+  },
+});
+
+// Deploy 2: make required
+users: defineTable({
+  name: v.string(),
+  role: v.union(v.literal("user"), v.literal("admin")),
+})
+```
+
+### Deleting a Field
+
+```typescript
+// Deploy 1: Make optional
+// isPro: v.boolean()  →  isPro: v.optional(v.boolean())
+
+// Migration: clear the field
+export const removeIsPro = migrations.define({
+  table: "teams",
+  migrateOne: async (ctx, team) => {
+    if (team.isPro !== undefined) {
+      await ctx.db.patch(team._id, { isPro: undefined });
+    }
+  },
+});
+
+// Deploy 2: Remove isPro from schema entirely
+```
+
+### Changing a Field Type
+
+Create a new field rather than modifying the existing one:
+
+```typescript
+// Deploy 1: Add new field, keep old field optional
+// isPro: v.boolean()  →  isPro: v.optional(v.boolean()), plan: v.optional(...)
+
+// Migration: convert old to new
+export const convertToEnum = migrations.define({
+  table: "teams",
+  migrateOne: async (ctx, team) => {
+    if (team.plan === undefined) {
+      await ctx.db.patch(team._id, {
+        plan: team.isPro ? "pro" : "basic",
+        isPro: undefined,
+      });
+    }
+  },
+});
+
+// Deploy 2: Remove isPro, make plan required
+```
+
+### Splitting Nested Data Into a Separate Table
+
+```typescript
+export const extractPreferences = migrations.define({
+  table: "users",
+  migrateOne: async (ctx, user) => {
+    if (user.preferences === undefined) return;
+    const existing = await ctx.db
+      .query("userPreferences")
+      .withIndex("by_user", (q) => q.eq("userId", user._id))
+      .first();
+    if (!existing) {
+      await ctx.db.insert("userPreferences", {
+        userId: user._id,
+        ...user.preferences,
+      });
+    }
+    await ctx.db.patch(user._id, { preferences: undefined });
+  },
+});
+```
+
+## @convex-dev/migrations Component
+
+For any non-trivial migration, use this component — it handles batching, cursor-based pagination, state tracking, resume from failure, dry runs, and progress monitoring.
+
+See `references/migrations-component.md` for installation, setup, and full API.
+
+```bash
+npm install @convex-dev/migrations
+```
+
+```typescript
+// convex/convex.config.ts
+import migrations from "@convex-dev/migrations/convex.config.js";
+const app = defineApp();
+app.use(migrations);
+
+// convex/migrations.ts
+import { Migrations } from "@convex-dev/migrations";
+export const migrations = new Migrations<DataModel>(components.migrations);
+export const run = migrations.runner();
+```
+
+Run from CLI:
+
+```bash
+npx convex run migrations:run '{"fn": "migrations:addDefaultRole"}'
+# Dry run first:
+npx convex run migrations:runIt '{"dryRun": true}'
+# Check status:
+npx convex run --component migrations lib:getStatus --watch
+```
+
+### Small Table Shortcut
+
+For tables with only a few thousand documents, use a single `internalMutation` without the component:
+
+```typescript
+export const backfillSmallTable = internalMutation({
+  handler: async (ctx) => {
+    const docs = await ctx.db.query("smallConfig").collect();
+    for (const doc of docs) {
+      if (doc.newField === undefined) {
+        await ctx.db.patch(doc._id, { newField: "default" });
+      }
+    }
+  },
+});
+```
+
+## Zero-Downtime Strategies
+
+### Dual Write (Preferred)
+
+Write to both old and new structures. Read from old until migration is complete. Safe to roll back at any point.
+
+```typescript
+// Good: writing both formats during migration
+export const createTeam = mutation({
+  handler: async (ctx, args) => {
+    const plan = args.isPro ? "pro" : "basic";
+    await ctx.db.insert("teams", {
+      name: args.name,
+      isPro: args.isPro,  // old format
+      plan,               // new format
+    });
+  },
+});
+```
+
+### Dual Read
+
+Read both formats (preferring new), write only the new format:
+
+```typescript
+function getTeamPlan(team: Doc<"teams">): "basic" | "pro" {
+  if (team.plan !== undefined) return team.plan;
+  return team.isPro ? "pro" : "basic";
+}
+```
+
+## Verification
+
+Query to check remaining unmigrated documents:
+
+```typescript
+export const verifyMigration = query({
+  handler: async (ctx) => {
+    const remaining = await ctx.db
+      .query("users")
+      .filter((q) => q.eq(q.field("role"), undefined))
+      .take(10);
+    return { complete: remaining.length === 0 };
+  },
+});
+```
+
+## Common Pitfalls
+
+1. **Making a field required before migrating data** — deploy rejects because documents lack the field
+2. **Using `.collect()` on large tables** — hits transaction limits; use the migrations component
+3. **Not writing the new format before migrating** — creates missed documents during migration window
+4. **Skipping the dry run** — use `dryRun: true` to validate before touching production data
+5. **Deleting fields prematurely** — prefer `v.optional` with a deprecation comment
+
+## Migration Checklist
+
+- [ ] Identified the breaking change and planned the multi-deploy workflow
+- [ ] Widened schema to allow both old and new formats
+- [ ] Updated code to handle both formats when reading
+- [ ] Updated code to write the new format for new documents
+- [ ] Deployed widened schema
+- [ ] Defined migration using `@convex-dev/migrations`
+- [ ] Tested with `dryRun: true`
+- [ ] Ran migration and monitored status
+- [ ] Verified all documents are migrated
+- [ ] Narrowed schema to require new format only
+- [ ] Cleaned up code handling old format
+- [ ] Deployed final schema

package/src/orchestrator/plugins/convex/references/occ-conflicts.md

@@ -0,0 +1,126 @@
+# OCC Conflict Resolution
+
+Use these rules when insights, logs, or dashboard health show OCC (Optimistic Concurrency Control) conflicts, mutation retries, or write contention on hot tables.
+
+## Core Principle
+
+Convex uses optimistic concurrency control. When two transactions read or write overlapping data, one succeeds and the other retries automatically. High contention means wasted work and increased latency.
+
+## Symptoms
+
+- OCC conflict errors in deployment logs or health page
+- Mutations retrying multiple times before succeeding
+- User-visible latency spikes on write-heavy pages
+- `npx convex insights --details` showing high conflict rates
+
+## Common Causes
+
+### Hot documents
+
+Multiple mutations writing to the same document concurrently. Classic examples: a global counter, a shared settings row, or a "last updated" timestamp on a parent record.
+
+### Broad read sets causing false conflicts
+
+A query that scans a large table range creates a broad read set. If any write touches that range, the query's transaction conflicts even if the specific document the query cared about was not modified.
+
+### Fan-out from triggers or cascading writes
+
+A single user action triggers multiple mutations that all touch related documents. Each mutation competes with the others.
+
+Database triggers (e.g. from `convex-helpers`) run inside the same transaction as the mutation that caused them. If a trigger does heavy work, reads extra tables, or writes to many documents, it extends the transaction's read/write set and increases the window for conflicts. Keep trigger logic minimal, or move expensive derived work to a scheduled function.
+
+### Write-then-read chains
+
+A mutation writes a document, then a reactive query re-reads it, then another mutation writes it again. Under load, these chains stack up.
+
+## Fix Order
+
+### 1. Reduce read set size
+
+Narrower reads mean fewer false conflicts.
+
+```ts
+// Bad: broad scan creates a wide conflict surface
+const allTasks = await ctx.db.query("tasks").collect();
+const mine = allTasks.filter((t) => t.ownerId === userId);
+```
+
+```ts
+// Good: indexed query touches only relevant documents
+const mine = await ctx.db
+  .query("tasks")
+  .withIndex("by_owner", (q) => q.eq("ownerId", userId))
+  .collect();
+```
+
+### 2. Split hot documents
+
+When many writers target the same document, split the contention point.
+
+```ts
+// Bad: every vote increments the same counter document
+const counter = await ctx.db.get(pollCounterId);
+await ctx.db.patch(pollCounterId, { count: counter!.count + 1 });
+```
+
+```ts
+// Good: shard the counter across multiple documents, aggregate on read
+const shardIndex = Math.floor(Math.random() * SHARD_COUNT);
+const shardId = shardIds[shardIndex];
+const shard = await ctx.db.get(shardId);
+await ctx.db.patch(shardId, { count: shard!.count + 1 });
+```
+
+Aggregate the shards in a query or scheduled job when you need the total.
+
+### 3. Skip no-op writes
+
+Writes that do not change data still participate in conflict detection and trigger invalidation.
+
+```ts
+// Bad: patches even when nothing changed
+await ctx.db.patch(doc._id, { status: args.status });
+```
+
+```ts
+// Good: only write when the value actually differs
+if (doc.status !== args.status) {
+  await ctx.db.patch(doc._id, { status: args.status });
+}
+```
+
+### 4. Move non-critical work to scheduled functions
+
+If a mutation does primary work plus secondary bookkeeping (analytics, notifications, cache warming), the bookkeeping extends the transaction's lifetime and read/write set.
+
+```ts
+// Bad: analytics update in the same transaction as the user action
+await ctx.db.patch(userId, { lastActiveAt: Date.now() });
+await ctx.db.insert("analytics", { event: "action", userId, ts: Date.now() });
+```
+
+```ts
+// Good: schedule the bookkeeping so the primary transaction is smaller
+await ctx.db.patch(userId, { lastActiveAt: Date.now() });
+await ctx.scheduler.runAfter(0, internal.analytics.recordEvent, {
+  event: "action",
+  userId,
+});
+```
+
+### 5. Combine competing writes
+
+If two mutations must update the same document atomically, consider whether they can be combined into a single mutation call from the client, reducing round trips and conflict windows.
+
+Do not introduce artificial locks or queues unless the above steps have been tried first.
+
+## Related: Invalidation Scope
+
+Splitting hot documents also reduces subscription invalidation, not just OCC contention. If a document is written frequently and read by many queries, those queries re-run on every write even when the fields they care about have not changed. See `subscription-cost.md` section 4 ("Isolate frequently-updated fields") for that pattern.
+
+## Verification
+
+1. OCC conflict rate has dropped in insights or dashboard
+2. Mutation latency is lower and more consistent
+3. No data correctness regressions from splitting or scheduling changes
+4. Sibling writers to the same hot documents were fixed consistently

package/src/orchestrator/plugins/convex/references/performance-audit.md

@@ -0,0 +1,80 @@
+# Convex Performance Audit
+
+Diagnose and fix performance problems in Convex applications, one problem class at a time.
+
+## First Step: Gather Signals
+
+1. Run `npx convex insights --details` (use `--prod`, `--preview-name`, or `--deployment-name` as needed)
+2. If CLI is too old: `npx -y convex@latest insights --details`
+3. If runtime signals unavailable, audit from code — but keep guardrails: don't recommend structural work without a measured signal
+
+## Signal Routing
+
+After gathering signals, identify the problem class and read the matching reference file:
+
+| Signal | Reference |
+|--------|-----------|
+| High bytes or documents read, JS filtering, unnecessary joins | `references/hot-path-rules.md` |
+| OCC conflict errors, write contention, mutation retries | `references/occ-conflicts.md` |
+| High subscription count, slow UI updates, excessive re-renders | `references/subscription-cost.md` |
+| Function timeouts, transaction size errors, large payloads | `references/function-budget.md` |
+| General "it's slow" with no specific signal | Start with `references/hot-path-rules.md` |
+
+Multiple problem classes can overlap. Read the most relevant reference first.
+
+## Workflow
+
+### 1. Scope the problem
+
+Pick one concrete user flow. Write down:
+- Entrypoint functions
+- Client callsites (`useQuery`, `usePaginatedQuery`, `useMutation`)
+- Tables read and tables written
+- Whether the path is high-read, high-write, or both
+
+### 2. Trace the full read and write set
+
+For each function:
+1. Trace every `ctx.db.get()` and `ctx.db.query()`
+2. Trace every `ctx.db.patch()`, `ctx.db.replace()`, `ctx.db.insert()`
+3. Note foreign-key lookups, JS-side filtering, and full-document reads
+4. Identify sibling functions touching the same tables
+5. Identify reactive stats/aggregates on the same page
+
+### 3. Apply fixes from the relevant reference
+
+Read the reference file matching your problem class. Each reference includes specific patterns and a recommended fix order.
+
+Do not stop at the single function named by an insight. Trace sibling readers and writers touching the same tables.
+
+### 4. Fix sibling functions together
+
+When one function has a bug, audit sibling functions for the same pattern. If one list query switches to a digest table, inspect the other list queries for that table. If one mutation needs no-op write protection, inspect all other writers to the same table.
+
+### 5. Escalate invasive fixes
+
+If the fix is invasive, cross-cutting, or migration-heavy:
+- Introducing digest or summary tables across multiple flows
+- Splitting documents to isolate frequently-updated fields
+- Reworking pagination strategy across several screens
+- Switching to a new index that needs migration-safe rollout
+
+Stop and present options before editing. Consult `references/migrations.md` when correctness depends on handling old and new states during rollout.
+
+### 6. Verify
+
+Confirm:
+1. Results are the same as before — no dropped records
+2. Eliminated reads/writes are no longer in the hot path
+3. Fallback behavior works when denormalized/indexed fields are missing
+4. No unnecessary invalidation when data is unchanged
+5. Every relevant sibling reader and writer was inspected
+
+## Reference Files
+
+- `references/hot-path-rules.md` — Read amplification, invalidation, denormalization, indexes, digest tables
+- `references/occ-conflicts.md` — Write contention, OCC resolution, hot document splitting
+- `references/subscription-cost.md` — Reactive query cost, subscription granularity, point-in-time reads
+- `references/function-budget.md` — Execution limits, transaction size, large documents, payload size
+
+Also see [Convex Best Practices](https://docs.convex.dev/understanding/best-practices/).

package/src/orchestrator/plugins/convex/references/quickstart.md

@@ -0,0 +1,176 @@
+# Convex Quickstart
+
+Set up a working Convex project as fast as possible.
+
+## Scaffolding Templates
+
+Use `npm create convex@latest` for new projects. Pass project name and template to avoid interactive prompts:
+
+```bash
+npm create convex@latest my-app -- -t react-vite-shadcn
+cd my-app
+npm install
+```
+
+| Template | Stack |
+|----------|-------|
+| `react-vite-shadcn` | React + Vite + Tailwind + shadcn/ui |
+| `nextjs-shadcn` | Next.js App Router + Tailwind + shadcn/ui |
+| `react-vite-clerk-shadcn` | React + Vite + Clerk auth + shadcn/ui |
+| `nextjs-clerk` | Next.js + Clerk auth |
+| `nextjs-convexauth-shadcn` | Next.js + Convex Auth + shadcn/ui |
+| `bare` | Convex backend only, no frontend |
+
+Default: `react-vite-shadcn` for simple apps, `nextjs-shadcn` for apps needing SSR or API routes.
+
+You can use any GitHub repo as a template:
+
+```bash
+npm create convex@latest my-app -- -t owner/repo
+npm create convex@latest my-app -- -t owner/repo#branch
+```
+
+## Adding Convex to an Existing App
+
+```bash
+npm install convex
+```
+
+Then ask the user to run `npx convex dev` in their terminal.
+
+## ConvexProvider Wiring
+
+Create the `ConvexReactClient` at module scope, not inside a component:
+
+```tsx
+// Bad: re-creates the client on every render
+function App() {
+  const convex = new ConvexReactClient(import.meta.env.VITE_CONVEX_URL as string);
+  return <ConvexProvider client={convex}>...</ConvexProvider>;
+}
+
+// Good: created once at module scope
+const convex = new ConvexReactClient(import.meta.env.VITE_CONVEX_URL as string);
+function App() {
+  return <ConvexProvider client={convex}>...</ConvexProvider>;
+}
+```
+
+#### React (Vite)
+
+```tsx
+// src/main.tsx
+import { ConvexProvider, ConvexReactClient } from "convex/react";
+
+const convex = new ConvexReactClient(import.meta.env.VITE_CONVEX_URL as string);
+
+createRoot(document.getElementById("root")!).render(
+  <StrictMode>
+    <ConvexProvider client={convex}>
+      <App />
+    </ConvexProvider>
+  </StrictMode>,
+);
+```
+
+#### Next.js (App Router)
+
+```tsx
+// app/ConvexClientProvider.tsx
+"use client";
+import { ConvexProvider, ConvexReactClient } from "convex/react";
+
+const convex = new ConvexReactClient(process.env.NEXT_PUBLIC_CONVEX_URL!);
+
+export function ConvexClientProvider({ children }: { children: ReactNode }) {
+  return <ConvexProvider client={convex}>{children}</ConvexProvider>;
+}
+```
+
+## Environment Variables
+
+| Framework | Variable |
+|-----------|----------|
+| Vite | `VITE_CONVEX_URL` |
+| Next.js | `NEXT_PUBLIC_CONVEX_URL` |
+| Remix | `CONVEX_URL` |
+| React Native | `EXPO_PUBLIC_CONVEX_URL` |
+
+`npx convex dev` writes the correct variable to `.env.local` automatically.
+
+## Agent Mode (Cloud/Headless)
+
+Set `CONVEX_AGENT_MODE=anonymous` in `.env.local` to run a local anonymous backend without interactive browser login:
+
+```bash
+CONVEX_AGENT_MODE=anonymous npx convex dev
+```
+
+## The Dev Loop
+
+`npx convex dev` is a long-running watcher — it's interactive on first run (browser-based OAuth). **Ask the user to run this themselves.** Once running it:
+- Creates a Convex project and dev deployment
+- Writes the deployment URL to `.env.local`
+- Creates `convex/_generated/` with types
+- Watches for changes and syncs continuously
+
+Deploy to production separately:
+
+```bash
+npx convex deploy
+```
+
+## First Function: Verification Round-Trip
+
+`convex/schema.ts`:
+
+```ts
+import { defineSchema, defineTable } from "convex/server";
+import { v } from "convex/values";
+
+export default defineSchema({
+  tasks: defineTable({
+    text: v.string(),
+    completed: v.boolean(),
+  }),
+});
+```
+
+`convex/tasks.ts`:
+
+```ts
+import { query, mutation } from "./_generated/server";
+import { v } from "convex/values";
+
+export const list = query({
+  args: {},
+  handler: async (ctx) => {
+    return await ctx.db.query("tasks").collect();
+  },
+});
+
+export const create = mutation({
+  args: { text: v.string() },
+  returns: v.null(),
+  handler: async (ctx, args) => {
+    await ctx.db.insert("tasks", { text: args.text, completed: false });
+    return null;
+  },
+});
+```
+
+## Other Frameworks
+
+- [Vue](https://docs.convex.dev/quickstart/vue)
+- [Svelte](https://docs.convex.dev/quickstart/svelte)
+- [React Native](https://docs.convex.dev/quickstart/react-native)
+- [TanStack Start](https://docs.convex.dev/quickstart/tanstack-start)
+- [Remix](https://docs.convex.dev/quickstart/remix)
+- [Node.js](https://docs.convex.dev/quickstart/nodejs)
+
+## Next Steps
+
+- Add authentication → `references/auth-setup.md`
+- Schema migrations → `references/migrations.md`
+- Performance optimization → `references/performance-audit.md`
+- Component creation → `references/components.md`