@agent-native/core 0.45.1 → 0.46.0

package/dist/templates/workspace-core/.agents/skills/real-time-collab/SKILL.md

@@ -1,9 +1,10 @@
 ---
 name: real-time-collab
 description: >-
-  Multi-user collaborative editing with Yjs CRDT and live cursors. Use when
-  adding real-time collaborative editing to a template, debugging sync issues,
-  or understanding how the agent and humans edit documents simultaneously.
+  Multi-user collaborative editing with Yjs CRDT, SSE fast-path transport, and
+  granular server-side merge. Use when adding real-time collaborative editing to
+  a template, debugging sync issues, or understanding how the agent and humans
+  edit documents simultaneously.
 metadata:
   internal: true
 ---
@@ -12,39 +13,57 @@ metadata:
 
 ## Rule
 
-Collaborative editing uses Yjs CRDT via TipTap. The agent and human users are equal participants — both edit the same Y.Doc and changes merge cleanly without conflicts.
+Collaborative editing uses Yjs CRDT via TipTap. The agent and human users are
+equal participants — both edit the same Y.Doc and changes merge cleanly without
+conflicts. Always set `resourceType` on `createCollabPlugin`.
 
 ## How It Works
 
 - **`Y.Doc`** stores the document as a `Y.XmlFragment` (ProseMirror node tree)
-- **TipTap's Collaboration extension** binds the editor to the Y.XmlFragment via `ySyncPlugin`
-- **CollaborationCaret extension** renders remote users' cursors with names and colors
-- **Polling** (every 2s) syncs Y.Doc updates and awareness state between clients and server
-- **SQL `_collab_docs` table** persists Yjs state as base64-encoded binary (works across SQLite/Postgres)
+- **TipTap's Collaboration extension** binds the editor to the Y.XmlFragment
+  via `ySyncPlugin`
+- **CollaborationCaret extension** renders remote users' cursors with names and
+  colors
+- **SSE fast-path** — `/_agent-native/poll-events` `EventSource` delivers collab
+  events push-style; while SSE is healthy the collab poll interval relaxes to
+  ~12 s
+- **Polling fallback** — `/_agent-native/poll` is polled every 2 s when SSE is
+  unavailable; this is the universal serverless fallback
+- **Update batching** — local Yjs updates are debounced ~80 ms and coalesced
+  with `Y.mergeUpdates` before sending; flushed immediately on
+  `visibilitychange` / `pagehide`
+- **SQL `_collab_docs` table** persists Yjs state as base64 (SQLite/Postgres
+  compatible). Tombstone compaction fires automatically when the stored blob
+  exceeds 4× the fresh encoded size.
 
 ## Agent + Human Editing
 
 1. **Human edits** → TipTap → ySyncPlugin → Y.XmlFragment → `POST /_agent-native/collab/:docId/update`
 2. **Agent edits** → action edits canonical SQL content + bumps `updatedAt` → change-sync refetch → the open editor reconciles the new content into the live Y.Doc (see below) → poll update → all clients
 
-Both produce Yjs operations that merge cleanly. Agent edits appear without destroying cursor position, selection, or undo history.
+Both produce Yjs operations that merge cleanly. Agent edits appear without
+destroying cursor position, selection, or undo history.
 
-This is how content (documents) and slides now work. The agent does **not** push edits into Yjs in-process, and it does **not** call any `findCollabOrigin()` / localhost probe — that approach silently no-op'd on serverless (the action runs in a different process), so agent edits didn't show up live until the user navigated away and back. Nor does it search-and-replace inside existing Y.XmlText nodes, which could never create new block structure (lists, headings, tables). The peer-editor model below replaces both.
+The agent does **not** push edits into Yjs in-process and does **not** call any
+localhost probe — those approaches silently no-op on serverless (the action runs
+in a different process). The peer-editor model below replaced them.
 
 ## Agent Edits As A Real-Time Peer Editor
 
-The agent edits documents the same way a human collaborator does: its change lands in the shared Y.Doc, propagates to every connected client, and persists. It gets there without any in-process Yjs push from the action.
+**SQL is the durable source of truth for document body content.** The agent
+action edits the canonical content column and bumps `updatedAt`. No localhost
+calls, no in-process Yjs mutation.
 
-**SQL is the durable source of truth for document body content.** The agent action edits the canonical content (e.g. `documents.content`) and bumps `updatedAt`. That's the whole server side — no localhost calls, no Yjs mutation from the action.
-
-**The open editor reconciles authoritative external content into the live Y.Doc.** The action's `updatedAt` bump flows through the change-sync system (see `real-time-sync`), which refetches the record. The editor applies the new content through its real markdown/HTML pipeline via `setContent`, so new block structure (lists, headings, tables) renders correctly and merges with concurrent human edits through the Yjs CRDT diff. The result: the agent's edit propagates to every connected client and persists, exactly like a human collaborator's edit.
+**The open editor reconciles authoritative external content into the live
+Y.Doc.** The `updatedAt` bump flows through change-sync, which refetches the
+record. The lead client applies the new content via `setContent`, producing Yjs
+operations that merge with concurrent human edits. Every connected client
+receives the result through normal Yjs sync.
 
 ### The `updatedAt` gate
 
-The editor only adopts content that is genuinely **newer** than what it already reflects. An older-or-equal `updatedAt` is a lagging poll or a stale snapshot and is **ignored**.
-
 ```ts
-// Pseudocode in the editor's reconcile effect
+// In the editor's reconcile effect
 if (loaded.updatedAt > lastAppliedUpdatedAt.current) {
   applyAuthoritativeContent(loaded.content); // adopt
   lastAppliedUpdatedAt.current = loaded.updatedAt;
@@ -52,56 +71,107 @@ if (loaded.updatedAt > lastAppliedUpdatedAt.current) {
 // else: lagging poll / stale snapshot → ignore
 ```
 
-**Why:** without the gate, a slightly-behind poll response re-applies old content right after the agent's edit, so the edit "reverts on the next poll" / "doesn't show until refresh" — the whack-a-mole we kept hitting. A **fresh mount or doc-switch has no baseline**, so it always adopts whatever content it loaded — which is why a manual refresh is always correct.
+Without the gate, a slightly-behind poll response re-applies old content and
+the edit "reverts on next poll". A fresh mount always adopts whatever content
+it loaded.
 
 ### Lead-client election
 
-Exactly ONE connected client applies an authoritative snapshot into the shared Y.Doc; the rest receive it through normal Yjs sync. The lead is the present client with the lowest Yjs `clientID`, decided by the core helper:
+Exactly ONE connected client applies the authoritative snapshot; the rest
+receive it through Yjs sync:
 
 ```ts
 import { isReconcileLeadClient } from "@agent-native/core/client";
 
 if (
   loaded.updatedAt > lastAppliedUpdatedAt.current &&
-  isReconcileLeadClient(provider.awareness, ydoc.clientID)
+  isReconcileLeadClient(awareness, ydoc.clientID)
 ) {
   applyAuthoritativeContent(loaded.content);
 }
 ```
 
-**Why:** if every open editor independently diffed the same snapshot into the CRDT, each would insert the changed region at the same position, duplicating it N times (concurrent inserts → duplicated text). Electing one lead avoids that. The agent's awareness id (`AGENT_CLIENT_ID`, max int) can never win, and a client editing alone is always the lead. The election is deterministic across clients with no coordination round-trip.
+The agent's awareness entry (`AGENT_CLIENT_ID`, max int) can never be the
+lead. A sole client is always the lead. The election is deterministic with no
+coordination round-trip.
 
 ### v1 limitation
 
-A full-content reconcile is **last-writer-wins for the rare case** where a human has unsaved edits in the exact region the agent simultaneously rewrites — the agent's snapshot can clobber that in-flight human edit. Inline and structural edits in **different** regions merge fine through the CRDT; only same-region simultaneous rewrites are at risk.
+Full-content reconcile is **last-writer-wins** for the rare case where a human
+has unsaved edits in the exact region the agent simultaneously rewrites. Edits
+in **different** regions merge fine through the CRDT.
+
+## Security
+
+### Always set `resourceType`
+
+```ts
+// server/plugins/collab.ts
+import { createCollabPlugin } from "@agent-native/core/server";
+
+export default createCollabPlugin({
+  table: "documents",
+  contentColumn: "content",
+  idColumn: "id",
+  resourceType: "document", // required
+});
+```
+
+Without `resourceType`, the server logs a one-time warning and collab push
+events are delivered to **all authenticated users** without document-level
+scoping. Set it to the resource type name registered via
+`registerShareableResource`.
+
+Non-owner sharees who have explicit access fall back to state-vector catch-up
+(safe, slightly higher latency). Awareness routes require the same viewer
+access as read routes.
+
+### Payload limits
+
+Write routes reject payloads exceeding `maxPayloadBytes` (default 2 MB) with
+HTTP 413. Override:
+
+```ts
+createCollabPlugin({ resourceType: "document", maxPayloadBytes: 512 * 1024 });
+```
 
 ## Enabling Collaboration
 
 ### 1. Install packages
 
 ```bash
-pnpm add @tiptap/extension-collaboration @tiptap/extension-collaboration-caret @tiptap/y-tiptap
+pnpm add @tiptap/extension-collaboration @tiptap/extension-collaboration-caret @tiptap/y-tiptap @tiptap/core
 ```
 
-### 2. Add collab server plugin
+### 2. Add collab server plugin (with `resourceType`)
 
 ```ts
 // server/plugins/collab.ts
-import { createCollabPlugin } from "@agent-native/core/collab";
+import { createCollabPlugin } from "@agent-native/core/server";
 
 export default createCollabPlugin({
   table: "documents",
   contentColumn: "content",
   idColumn: "id",
+  resourceType: "document",
 });
 ```
 
 ### 3. Use the client hook
 
 ```ts
-import { useCollaborativeDoc } from "@agent-native/core/client";
-
-const { ydoc, provider } = useCollaborativeDoc(documentId);
+import { useCollaborativeDoc, emailToColor, emailToName } from "@agent-native/core/client";
+
+const { ydoc, awareness, activeUsers, agentActive, agentPresent } =
+  useCollaborativeDoc({
+    docId: documentId,
+    requestSource: TAB_ID,
+    user: {
+      name: emailToName(session.email),
+      email: session.email,
+      color: emailToColor(session.email),
+    },
+  });
 ```
 
 ### 4. Add TipTap extensions
@@ -112,12 +182,14 @@ import { CollaborationCaret } from "@tiptap/extension-collaboration-caret";
 
 const editor = useEditor({
   extensions: [
+    StarterKit.configure({ history: false }), // Yjs handles undo
     Collaboration.configure({ document: ydoc }),
     CollaborationCaret.configure({
-      provider,
+      provider: { awareness },
       user: { name: session.email, color: "#6366f1" },
     }),
   ],
+  // Do NOT pass content — Yjs owns it
 });
 ```
 
@@ -126,6 +198,9 @@ const editor = useEditor({
 ```ts
 optimizeDeps: {
   include: [
+    "yjs",
+    "y-protocols/awareness",
+    "@tiptap/core",
     "@tiptap/extension-collaboration",
     "@tiptap/extension-collaboration-caret",
     "@tiptap/y-tiptap",
@@ -137,22 +212,95 @@ optimizeDeps: {
 
 | Route | Purpose |
 | ----- | ------- |
-| `GET /_agent-native/collab/:docId/state` | Fetch full Y.Doc state |
+| `GET /_agent-native/collab/:docId/state` | Fetch full Y.Doc state (accepts `?stateVector=` for diff) |
 | `POST /_agent-native/collab/:docId/update` | Apply client Yjs update |
 | `POST /_agent-native/collab/:docId/text` | Apply full text (diff-based) |
 | `POST /_agent-native/collab/:docId/search-replace` | Surgical find/replace in Y.XmlFragment |
+| `POST /_agent-native/collab/:docId/json` | Apply full JSON diff to Y.Map/Y.Array |
+| `GET /_agent-native/collab/:docId/json` | Read current JSON state |
+| `POST /_agent-native/collab/:docId/patch` | Surgical JSON patch ops |
 | `POST /_agent-native/collab/:docId/awareness` | Sync cursor/presence state |
 | `GET /_agent-native/collab/:docId/users` | List active users |
 
+## Granular Server-Side Merge Pattern
+
+For structured documents (slides, forms, design files) where body collab would
+cause LWW conflicts at the container level, use **granular server-side merge**:
+define an action with targeted per-item operations.
+
+**When to use granular merge vs body collab:**
+
+| Scenario | Recommended approach |
+| -------- | -------------------- |
+| Free-form rich text, cursor-level CRDT matters | Body collab (Y.XmlFragment + TipTap) |
+| Structured items (slides, fields) where different users edit different items | Granular server-side merge (action with patch ops) |
+
+Example operation shape for slides:
+
+```ts
+type PatchDeckOp =
+  | { type: "patch"; slideId: string; fields: Partial<SlideFields> }
+  | { type: "add"; position: number; slide: SlideData }
+  | { type: "delete"; slideId: string }
+  | { type: "reorder"; slideId: string; newIndex: number };
+```
+
+Concurrent edits to different slides both succeed at the action level; there
+is no whole-deck LWW. Forms use the same shape with field-level ops.
+
+## Collaborative Undo Scoping (Y.UndoManager)
+
+Scope undo/redo to the local user's own edits so peer and agent changes are
+never accidentally reversed:
+
+```ts
+import * as Y from "yjs";
+
+const LOCAL_EDIT_ORIGIN = "local";
+
+const undoManager = new Y.UndoManager(ydoc.getText("content"), {
+  trackedOrigins: new Set([LOCAL_EDIT_ORIGIN]),
+  captureTimeout: 800, // coalesces rapid slider drags into one undo step
+});
+
+// Mark local edits with the tracked origin
+ydoc.transact(() => {
+  // apply local change
+}, LOCAL_EDIT_ORIGIN);
+
+undoManager.undo(); // only reverses LOCAL_EDIT_ORIGIN transactions
+undoManager.redo();
+```
+
+Rules:
+- Pass a `Set` to `trackedOrigins` — not an array.
+- Remote (`"remote"`) and agent (`"agent"`) origins are never captured.
+- Recreate and destroy the manager when the active document changes.
+
 ## Common Pitfalls
 
-- **Don't pass `content` as a TipTap prop** when Collaboration is enabled — Yjs owns the content. Set initial content via the Y.Doc instead.
-- **Don't call `editor.setContent()` ad hoc for agent edits.** The only sanctioned `setContent` is the editor's reconcile path described above — gated by `updatedAt` and guarded by `isReconcileLeadClient`. Calling it from elsewhere (e.g. on every poll, or from every client) re-applies stale content or duplicates the changed region across the CRDT.
-- **Add packages to `optimizeDeps`** — Vite won't pre-bundle Yjs packages correctly otherwise, causing runtime errors in dev.
-- **One `Y.Doc` per document** — Don't create multiple Y.Doc instances for the same document ID. Use the `useCollaborativeDoc` hook which caches by ID.
+- **Missing `resourceType`** — The server logs a warning on startup and
+  delivers collab events to all authenticated users without access scoping.
+  Always set `resourceType`.
+- **Don't pass `content` as a TipTap prop** when Collaboration is enabled —
+  Yjs owns the content. Seed via `editor.commands.setContent()` only when the
+  Y.XmlFragment is empty.
+- **Don't call `editor.setContent()` ad hoc for agent edits** — the only
+  sanctioned `setContent` is gated by `updatedAt` and guarded by
+  `isReconcileLeadClient`. Calling it from elsewhere duplicates content across
+  the CRDT or re-applies stale snapshots.
+- **Add packages to `optimizeDeps`** — Vite won't pre-bundle Yjs correctly
+  otherwise, causing runtime errors in dev.
+- **One `Y.Doc` per document** — Don't create multiple Y.Doc instances for the
+  same document ID. `useCollaborativeDoc` caches by ID.
+- **Destroy Y.UndoManager on doc change** — Stale managers hold Y.Doc
+  references and grow unboundedly. Recreate on `docId` change.
 
 ## Related Skills
 
-- `real-time-sync` — The change-sync system that delivers the `updatedAt` bump driving editor reconciliation; also `useReconciledState` for non-collaborative "copy a server value into local edit state" surfaces
-- `storing-data` — The `_collab_docs` table where Yjs state is persisted; SQL holds the canonical document body that the editor reconciles from
-- `self-modifying-code` — Agent edits to collaborative documents edit canonical SQL content, not raw Yjs
+- `real-time-sync` — The change-sync system that delivers the `updatedAt` bump
+  driving editor reconciliation; also `useReconciledState` for non-Yjs surfaces
+- `storing-data` — The `_collab_docs` table and SQL canonical content
+- `security` — `registerShareableResource`, `resolveAccess`, `assertAccess`
+- `self-modifying-code` — Agent edits to collaborative documents edit canonical
+  SQL content, not raw Yjs

package/dist/templates/workspace-core/.agents/skills/real-time-sync/SKILL.md

@@ -49,7 +49,7 @@ The agent modifies data in SQL, but the UI runs in the browser. SSE bridges same
 
    For list/sidebar queries, use the same pattern — pass the counter into the queryKey of every list query you want to keep fresh.
 
-4. **Fallback** polling calls `/_agent-native/poll?since=N`. It runs every 2 seconds until SSE is connected, then relaxes to 15 seconds. If SSE is disabled or unavailable, polling continues at the normal cadence.
+4. **Fallback** polling calls `/_agent-native/poll?since=N`. It runs every 2 seconds until SSE is connected, then relaxes to 15 seconds (`SSE_FALLBACK_INTERVAL_MS`). If SSE is disabled or unavailable (e.g., edge/serverless deployments), polling continues at the 2 s cadence. Polling is the universal serverless fallback — it detects DB timestamp changes even when the write happened in a different process or invocation.
 
 5. When the agent writes to the database, the version increments, SSE/polling detects it, and React Query refetches the affected queries.
 
@@ -196,6 +196,16 @@ const [title, setTitle] = useReconciledState(props.title, { active: isEditing })
 | Local edit state copied from a server value (inputs, popovers, inline editors) | `useReconciledState(externalValue, { active })` |
 | Collaborative rich-text editor (Yjs) | `updatedAt`-gated reconcile + `isReconcileLeadClient` — see `real-time-collab` |
 
+## Granular server-side merge for non-body fields
+
+For structured documents (slide decks, form builders, design files) where the
+Yjs body collab would cause LWW conflicts at the container level, pair the
+change-sync `updatedAt` bump with a **granular server-side merge action** that
+accepts targeted per-item operations (add/patch/delete/reorder). Concurrent
+edits to different items both survive at the action level; the `collab` source
+version bump then propagates the merged state to all open clients. See
+`real-time-collab` for the pattern and examples.
+
 ## Related Skills
 
 - **storing-data** — Application-state and settings are data stores that sync through change events
@@ -203,4 +213,4 @@ const [title, setTitle] = useReconciledState(props.title, { active: isEditing })
 - **actions** — Mutating actions trigger change events
 - **client-methods** — Route details belong in helpers/hooks, not components
 - **self-modifying-code** — Agent code edits trigger change events; rapid edits can cause event storms
-- **real-time-collab** — Collaborative editors reconcile agent edits into a shared Y.Doc, driven by the same change-sync `updatedAt` bump
+- **real-time-collab** — Collaborative editors reconcile agent edits into a shared Y.Doc, driven by the same change-sync `updatedAt` bump; also the granular server-side merge pattern for structured data