@agent-native/core 0.39.1 → 0.40.0

package/dist/templates/default/.agents/skills/real-time-collab/SKILL.md

@@ -1,183 +1,158 @@
 ---
 name: real-time-collab
 description: >-
-  How to enable multi-user collaborative editing with Yjs CRDT, TipTap
-  Collaboration extension, live cursors, and agent-driven edits.
+  Multi-user collaborative editing with Yjs CRDT and live cursors. Use when
+  adding real-time collaborative editing to a template, debugging sync issues,
+  or understanding how the agent and humans edit documents simultaneously.
+metadata:
+  internal: true
 ---
 
 # Real-Time Collaboration
 
-The framework provides a Yjs-based collaborative editing system in `@agent-native/core/collab`. Multiple users can edit the same document simultaneously with live cursor positions, and the AI agent can make surgical edits that appear in real-time.
+## Rule
 
-## Architecture
+Collaborative editing uses Yjs CRDT via TipTap. The agent and human users are equal participants — both edit the same Y.Doc and changes merge cleanly without conflicts.
 
-```
-User A (TipTap + Collaboration ext)  ←→  Y.XmlFragment  ←→  Server (_collab_docs table)
-User B (TipTap + Collaboration ext)  ←→  Y.XmlFragment  ←→       ↑
-Agent (edit-document action)         ←→  search-replace endpoint ─┘
-```
+## How It Works
 
-- **Yjs Y.Doc** stores the document as a `Y.XmlFragment` (ProseMirror node tree)
+- **`Y.Doc`** stores the document as a `Y.XmlFragment` (ProseMirror node tree)
 - **TipTap's Collaboration extension** binds the editor to the Y.XmlFragment via `ySyncPlugin`
 - **CollaborationCaret extension** renders remote users' cursors with names and colors
 - **Polling** (every 2s) syncs Y.Doc updates and awareness state between clients and server
-- **SQL `_collab_docs` table** persists Yjs state (base64-encoded binary)
+- **SQL `_collab_docs` table** persists Yjs state as base64-encoded binary (works across SQLite/Postgres)
 
-## Enabling Collaboration in a Template
+## Agent + Human Editing
 
-### 1. Install dependencies
+1. **Human edits** → TipTap → ySyncPlugin → Y.XmlFragment → `POST /_agent-native/collab/:docId/update`
+2. **Agent edits** → action edits canonical SQL content + bumps `updatedAt` → change-sync refetch → the open editor reconciles the new content into the live Y.Doc (see below) → poll update → all clients
 
-```bash
-pnpm add @tiptap/extension-collaboration @tiptap/extension-collaboration-caret @tiptap/y-tiptap --filter your-template
+Both produce Yjs operations that merge cleanly. Agent edits appear without destroying cursor position, selection, or undo history.
+
+This is how content (documents) and slides now work. The agent does **not** push edits into Yjs in-process, and it does **not** call any `findCollabOrigin()` / localhost probe — that approach silently no-op'd on serverless (the action runs in a different process), so agent edits didn't show up live until the user navigated away and back. Nor does it search-and-replace inside existing Y.XmlText nodes, which could never create new block structure (lists, headings, tables). The peer-editor model below replaces both.
+
+## Agent Edits As A Real-Time Peer Editor
+
+The agent edits documents the same way a human collaborator does: its change lands in the shared Y.Doc, propagates to every connected client, and persists. It gets there without any in-process Yjs push from the action.
+
+**SQL is the durable source of truth for document body content.** The agent action edits the canonical content (e.g. `documents.content`) and bumps `updatedAt`. That's the whole server side — no localhost calls, no Yjs mutation from the action.
+
+**The open editor reconciles authoritative external content into the live Y.Doc.** The action's `updatedAt` bump flows through the change-sync system (see `real-time-sync`), which refetches the record. The editor applies the new content through its real markdown/HTML pipeline via `setContent`, so new block structure (lists, headings, tables) renders correctly and merges with concurrent human edits through the Yjs CRDT diff. The result: the agent's edit propagates to every connected client and persists, exactly like a human collaborator's edit.
+
+### The `updatedAt` gate
+
+The editor only adopts content that is genuinely **newer** than what it already reflects. An older-or-equal `updatedAt` is a lagging poll or a stale snapshot and is **ignored**.
+
+```ts
+// Pseudocode in the editor's reconcile effect
+if (loaded.updatedAt > lastAppliedUpdatedAt.current) {
+  applyAuthoritativeContent(loaded.content); // adopt
+  lastAppliedUpdatedAt.current = loaded.updatedAt;
+}
+// else: lagging poll / stale snapshot → ignore
 ```
 
-### 2. Add Vite optimizeDeps
+**Why:** without the gate, a slightly-behind poll response re-applies old content right after the agent's edit, so the edit "reverts on the next poll" / "doesn't show until refresh" — the whack-a-mole we kept hitting. A **fresh mount or doc-switch has no baseline**, so it always adopts whatever content it loaded — which is why a manual refresh is always correct.
+
+### Lead-client election
+
+Exactly ONE connected client applies an authoritative snapshot into the shared Y.Doc; the rest receive it through normal Yjs sync. The lead is the present client with the lowest Yjs `clientID`, decided by the core helper:
 
-In `vite.config.ts`:
 ```ts
-export default defineConfig({
-  plugins: [reactRouter()],
-  optimizeDeps: {
-    include: [
-      "yjs",
-      "y-protocols/awareness",
-      "@tiptap/extension-collaboration",
-      "@tiptap/extension-collaboration-caret",
-      "@tiptap/y-tiptap",
-    ],
-  },
-});
+import { isReconcileLeadClient } from "@agent-native/core/client";
+
+if (
+  loaded.updatedAt > lastAppliedUpdatedAt.current &&
+  isReconcileLeadClient(provider.awareness, ydoc.clientID)
+) {
+  applyAuthoritativeContent(loaded.content);
+}
 ```
 
-This prevents Vite from re-bundling TipTap in incompatible ways during dev.
+**Why:** if every open editor independently diffed the same snapshot into the CRDT, each would insert the changed region at the same position, duplicating it N times (concurrent inserts → duplicated text). Electing one lead avoids that. The agent's awareness id (`AGENT_CLIENT_ID`, max int) can never win, and a client editing alone is always the lead. The election is deterministic across clients with no coordination round-trip.
 
-### 3. Add the collab server plugin
+### v1 limitation
+
+A full-content reconcile is **last-writer-wins for the rare case** where a human has unsaved edits in the exact region the agent simultaneously rewrites — the agent's snapshot can clobber that in-flight human edit. Inline and structural edits in **different** regions merge fine through the CRDT; only same-region simultaneous rewrites are at risk.
+
+## Enabling Collaboration
+
+### 1. Install packages
+
+```bash
+pnpm add @tiptap/extension-collaboration @tiptap/extension-collaboration-caret @tiptap/y-tiptap
+```
+
+### 2. Add collab server plugin
 
-Create `server/plugins/collab.ts`:
 ```ts
-import { createCollabPlugin } from "@agent-native/core/server";
+// server/plugins/collab.ts
+import { createCollabPlugin } from "@agent-native/core/collab";
+
 export default createCollabPlugin({
-  table: "your_table",
+  table: "documents",
   contentColumn: "content",
   idColumn: "id",
-  autoSeed: false, // Client-side seeding on first load
 });
 ```
 
-This mounts routes under `/_agent-native/collab/`:
-- `GET /:docId/state` — fetch Y.Doc state
-- `POST /:docId/update` — apply client update
-- `POST /:docId/text` — apply full text (diff-based)
-- `POST /:docId/search-replace` — surgical text find/replace in Y.XmlFragment
-- `POST /:docId/awareness` — sync cursor/presence state
-
-### 4. Use the `useCollaborativeDoc` hook
+### 3. Use the client hook
 
 ```ts
-import { useCollaborativeDoc, generateTabId } from "@agent-native/core/client";
+import { useCollaborativeDoc } from "@agent-native/core/client";
 
-const TAB_ID = generateTabId();
-
-const { ydoc, awareness, isLoading, activeUsers } = useCollaborativeDoc({
-  docId: documentId,
-  requestSource: TAB_ID,
-  user: { name: "Steve", email: "steve@example.com", color: "#60a5fa" },
-});
+const { ydoc, provider } = useCollaborativeDoc(documentId);
 ```
 
-The hook:
-- Creates a stable `Y.Doc` per docId (never changes identity)
-- Fetches server state and applies it
-- Sends local updates to server
-- Polls for remote updates (every 2s)
-- Tracks active users via awareness
-
-### 5. Add Collaboration extension to TipTap
+### 4. Add TipTap extensions
 
 ```ts
-import Collaboration from "@tiptap/extension-collaboration";
-import CollaborationCaret from "@tiptap/extension-collaboration-caret";
-import { Awareness } from "y-protocols/awareness";
-
-// Create awareness locally (must use same y-protocols as the caret extension)
-const awareness = new Awareness(ydoc);
-awareness.setLocalStateField("user", { name, color });
+import { Collaboration } from "@tiptap/extension-collaboration";
+import { CollaborationCaret } from "@tiptap/extension-collaboration-caret";
 
 const editor = useEditor({
   extensions: [
-    StarterKit.configure({ history: false }), // Disable history — Yjs handles undo
     Collaboration.configure({ document: ydoc }),
     CollaborationCaret.configure({
-      provider: { awareness },
-      user: { name, color },
+      provider,
+      user: { name: session.email, color: "#6366f1" },
     }),
-    // ... other extensions
   ],
-  content: initialContent, // Seeds Y.XmlFragment on first load
 });
 ```
 
-**Important:** Disable `history` in StarterKit when using Collaboration — Yjs handles undo/redo.
-
-### 6. Seed the Y.XmlFragment
-
-The Collaboration extension does NOT auto-seed from the `content` prop. You must seed manually:
+### 5. Add to vite.config.ts optimizeDeps
 
 ```ts
-useEffect(() => {
-  if (!editor || !ydoc || !content) return;
-  const fragment = ydoc.getXmlFragment("default");
-  if (fragment.length === 0) {
-    editor.commands.setContent(parseContent(content));
-  }
-}, [editor, ydoc, content]);
-```
-
-**Critical:** Guard against saving empty content back to SQL when the editor is in collab mode but hasn't been seeded yet:
-
-```ts
-onUpdate: ({ editor }) => {
-  const md = editor.storage.markdown.getMarkdown();
-  if (!md.trim() && ydoc) return; // Don't save empty during seeding
-  onChange(md);
+optimizeDeps: {
+  include: [
+    "@tiptap/extension-collaboration",
+    "@tiptap/extension-collaboration-caret",
+    "@tiptap/y-tiptap",
+  ],
 }
 ```
 
-## Agent Edits via `edit-document`
-
-The `edit-document` action uses search-and-replace:
-```bash
-pnpm action edit-document --id <docId> --find "old text" --replace "new text"
-```
-
-When collab state exists, the action calls the server's `search-replace` endpoint which:
-1. Walks the Y.XmlFragment tree
-2. Finds the text in Y.XmlText nodes
-3. Applies minimal delete/insert operations
-4. Emits a Yjs update via the poll system
-5. Client receives the update → ySyncPlugin applies a targeted ProseMirror transaction → cursor preserved
+## Collab Routes (auto-mounted)
 
-**Important:** Actions run in a separate process, so they must use the HTTP endpoint (not the collab module directly) to emit updates to the server's poll system.
-
-## Key Modules
-
-| Module | Path | Purpose |
-|--------|------|---------|
-| `@agent-native/core/collab` | `packages/core/src/collab/` | Server-side Yjs management |
-| `useCollaborativeDoc` | `packages/core/src/collab/client.ts` | Client hook |
-| `createCollabPlugin` | `packages/core/src/server/collab-plugin.ts` | Route mounting |
-| `searchAndReplace` | `packages/core/src/collab/ydoc-manager.ts` | Y.XmlFragment text mutation |
+| Route | Purpose |
+| ----- | ------- |
+| `GET /_agent-native/collab/:docId/state` | Fetch full Y.Doc state |
+| `POST /_agent-native/collab/:docId/update` | Apply client Yjs update |
+| `POST /_agent-native/collab/:docId/text` | Apply full text (diff-based) |
+| `POST /_agent-native/collab/:docId/search-replace` | Surgical find/replace in Y.XmlFragment |
+| `POST /_agent-native/collab/:docId/awareness` | Sync cursor/presence state |
+| `GET /_agent-native/collab/:docId/users` | List active users |
 
 ## Common Pitfalls
 
-1. **TipTap version mismatch** — All `@tiptap/*` packages must be the same version. The Collaboration extension requires `editor.utils` which was added in v3.22.2.
-
-2. **Empty editor on first load** — The Collaboration extension uses Y.XmlFragment as the source of truth. If the fragment is empty, the editor shows empty. Seed manually (see above).
-
-3. **Data loss from empty saves** — The `onUpdate` handler fires when the editor initializes with an empty Y.XmlFragment. If this empty content is saved to SQL, it overwrites the real content. Always guard against saving empty content in collab mode.
-
-4. **Stale content on document switch** — Use `key={documentId}` on the editor component to force a full remount when switching documents. This ensures the Y.Doc, seeding, and editor state are all fresh.
+- **Don't pass `content` as a TipTap prop** when Collaboration is enabled — Yjs owns the content. Set initial content via the Y.Doc instead.
+- **Don't call `editor.setContent()` ad hoc for agent edits.** The only sanctioned `setContent` is the editor's reconcile path described above — gated by `updatedAt` and guarded by `isReconcileLeadClient`. Calling it from elsewhere (e.g. on every poll, or from every client) re-applies stale content or duplicates the changed region across the CRDT.
+- **Add packages to `optimizeDeps`** — Vite won't pre-bundle Yjs packages correctly otherwise, causing runtime errors in dev.
+- **One `Y.Doc` per document** — Don't create multiple Y.Doc instances for the same document ID. Use the `useCollaborativeDoc` hook which caches by ID.
 
-5. **Separate process for actions** — Actions run via `pnpm action` in a new Node.js process. The in-memory EventEmitter in the action process doesn't reach the dev server's poll system. Use HTTP endpoints for collab operations from actions.
+## Related Skills
 
-6. **Vite dep optimization** — Adding Yjs-related packages to a template changes Vite's dependency bundling, which can break TipTap's React integration. Always add them to `optimizeDeps.include`.
+- `real-time-sync` — The change-sync system that delivers the `updatedAt` bump driving editor reconciliation; also `useReconciledState` for non-collaborative "copy a server value into local edit state" surfaces
+- `storing-data` — The `_collab_docs` table where Yjs state is persisted; SQL holds the canonical document body that the editor reconciles from
+- `self-modifying-code` — Agent edits to collaborative documents edit canonical SQL content, not raw Yjs

package/dist/templates/default/.agents/skills/real-time-sync/SKILL.md

@@ -4,6 +4,8 @@ description: >-
   How to keep the UI in sync with agent changes via SSE plus polling fallback.
   Use when wiring query invalidation for new data models, debugging UI not
   updating, or understanding jitter prevention.
+metadata:
+  internal: true
 ---
 
 # Real-Time Sync
@@ -20,7 +22,7 @@ The agent modifies data in SQL, but the UI runs in the browser. SSE bridges same
 
 1. **Server** increments a version counter on every database write. In-process events stream through the authenticated `/_agent-native/events` endpoint.
 
-2. **Client** listens for SSE/poll events and updates per-source change counters:
+2. **Client** listens for sync events and updates per-source change counters:
 
    ```ts
    import { useDbSync } from "@agent-native/core";
@@ -47,9 +49,9 @@ The agent modifies data in SQL, but the UI runs in the browser. SSE bridges same
 
    For list/sidebar queries, use the same pattern — pass the counter into the queryKey of every list query you want to keep fresh.
 
-3. **Fallback** polling calls `/_agent-native/poll?since=N`. It runs every 2 seconds until SSE is connected, then relaxes to 15 seconds. If SSE is disabled or unavailable, polling continues at the normal cadence.
+4. **Fallback** polling calls `/_agent-native/poll?since=N`. It runs every 2 seconds until SSE is connected, then relaxes to 15 seconds. If SSE is disabled or unavailable, polling continues at the normal cadence.
 
-4. When the agent writes to the database, the version increments, SSE/polling detects it, and React Query refetches the affected queries.
+5. When the agent writes to the database, the version increments, SSE/polling detects it, and React Query refetches the affected queries.
 
 ## Don't
 
@@ -132,9 +134,9 @@ The `use-navigation-state.ts` hook sends the same `TAB_ID` in the `X-Request-Sou
 
 Without jitter prevention, a cycle occurs: the UI writes state, sync detects the change, the UI refetches and re-renders, potentially overwriting what the user is actively editing. With `ignoreSource`, the UI only reacts to changes from other sources (agent scripts, other browser tabs, other users).
 
-## Action Routes and Polling
+## Action Routes and Live Sync
 
-Action routes (`/_agent-native/actions/:name`) work with the same sync system. When a POST/PUT/DELETE action writes to the database, the version counter increments and `useDbSync` picks up the change. Frontend mutations via `useActionMutation` automatically invalidate `["action"]` query keys on success, triggering refetches of `useActionQuery` hooks.
+Actions work with the same sync system. When a mutating action writes to the database, the version counter increments and `useDbSync` picks up the change. Frontend mutations via `useActionMutation` automatically invalidate `["action"]` query keys on success, triggering refetches of `useActionQuery` hooks. Client components should call actions through those hooks, not with raw action-route fetches.
 
 For custom apps, the best out-of-the-box path is:
 
@@ -146,14 +148,13 @@ This avoids duplicate `/api/*` JSON CRUD routes and makes agent-created records
 
 ### Auto-emit on mutating actions
 
-The framework emits a poll event with `source: "action"` whenever any non-read-only action runs to completion — whether called via HTTP (`/_agent-native/actions/:name`) or as an agent tool call. Read-only actions (`http: { method: "GET" }` or explicit `readOnly: true`) are skipped.
+The framework emits a change event with `source: "action"` whenever any non-read-only action runs to completion — whether called via HTTP (`/_agent-native/actions/:name`) or as an agent tool call. Read-only actions (`http: { method: "GET" }` or explicit `readOnly: true`) are skipped.
 
 This means UIs don't need the agent to remember to call `refresh-screen` after every mutation. A listener like this (used in the `macros` template) will refresh after any mutating agent call:
 
 ```ts
 useDbSync({
   queryClient,
-  queryKeys: [],
   ignoreSource: TAB_ID,
   onEvent: (data) => {
     if (data.requestSource === TAB_ID) return;
@@ -165,9 +166,41 @@ useDbSync({
 
 `refresh-screen` remains available for unusual cases — e.g. the agent mutated data via a path the framework can't see (external system the app mirrors), or the agent wants to pass a `scope` hint for narrower invalidation.
 
+## Keeping Stateful Components In Sync
+
+The `useChangeVersion` / `useActionQuery` pattern above keeps the **query layer** fresh. But components that copy a server value into local React state still go stale on agent edits — refetching the query updates the prop, yet the local copy never re-adopts it. This is a recurring bug.
+
+**Never do this** for a value the agent can mutate:
+
+```ts
+// BUG: `title` is captured once and never re-reads the prop.
+const [title, setTitle] = useState(props.title);
+```
+
+When the agent renames the record, the query refetches, `props.title` updates, but the input still shows the stale value until the component remounts.
+
+**Derived-state surfaces (form fields, inline editors, popovers): use `useReconciledState`.** It re-adopts the authoritative external value when it changes, except while the user is actively editing that field — so agent mutations show up live without clobbering in-progress typing:
+
+```ts
+import { useReconciledState } from "@agent-native/core/client";
+
+// `active` = true while the user is editing this field (focused / dirty).
+const [title, setTitle] = useReconciledState(props.title, { active: isEditing });
+```
+
+**Collaborative rich-text editors are different** — they don't copy a value into `useState`. They reconcile authoritative SQL content into a shared Y.Doc under an `updatedAt` gate with lead-client election. See `real-time-collab` → "Agent edits as a real-time peer editor". Don't reach for `useReconciledState` for a Yjs-backed editor.
+
+| Surface | Keep it fresh with |
+| ------- | ------------------ |
+| React Query reads | `useChangeVersion` / `useActionQuery` (above) |
+| Local edit state copied from a server value (inputs, popovers, inline editors) | `useReconciledState(externalValue, { active })` |
+| Collaborative rich-text editor (Yjs) | `updatedAt`-gated reconcile + `isReconcileLeadClient` — see `real-time-collab` |
+
 ## Related Skills
 
-- **storing-data** — Application-state and settings are the data stores that sync via polling
+- **storing-data** — Application-state and settings are data stores that sync through change events
 - **context-awareness** — Navigation state writes use jitter prevention to avoid overwriting active edits
-- **actions** — Action routes auto-expose actions as HTTP endpoints; database writes trigger poll events
-- **self-modifying-code** — Agent code edits trigger poll events; rapid edits can cause event storms
+- **actions** — Mutating actions trigger change events
+- **client-methods** — Route details belong in helpers/hooks, not components
+- **self-modifying-code** — Agent code edits trigger change events; rapid edits can cause event storms
+- **real-time-collab** — Collaborative editors reconcile agent edits into a shared Y.Doc, driven by the same change-sync `updatedAt` bump