@aion0/forge 0.6.1 → 0.8.0

package/docs/Implementation-Plan-Browser-Agent.md

@@ -0,0 +1,487 @@
+# Browser Agent — Implementation Plan
+
+> **Status**: Ready to implement.
+> **Date**: 2026-05-16
+> **Owner**: aion0
+> **Related**:
+> - `docs/RFC-Browser-Connectors.md` — detailed connector design (this is the abridged action plan)
+> - `../temper` — memory service (existing)
+> - Forge main repo (this) — orchestration platform (existing)
+
+## TL;DR
+
+Build a **Chrome extension** that does two things:
+
+1. **Chat UI** — daily-driver enterprise AI assistant with Temper-backed memory
+2. **Site connectors** — installable per-company adapters that re-use the user's browser sessions to read/write internal systems (Mantis, Teams, JIRA, internal CRM, …)
+
+The extension is **independent of Forge** — composable, mutually enhancing. Forge stays as today (`localhost:8403` web UI) and is invoked when long tasks / code work are needed; the extension handles day-to-day chat + system queries.
+
+## Architecture (Final)
+
+```
+                         ┌──────────────────────────────────┐
+                         │  Browser (user's daily Chrome)   │
+                         │ ┌──────────────────────────────┐ │
+                         │ │ Forge Browser Extension      │ │
+                         │ │                              │ │
+                         │ │  ┌────────────────────────┐  │ │
+                         │ │  │ Chat side-panel UI     │  │ │
+                         │ │  │ (React)                │  │ │
+                         │ │  └────────────────────────┘  │ │
+                         │ │                              │ │
+                         │ │  ┌────────────────────────┐  │ │
+                         │ │  │ Agent loop             │  │ │
+                         │ │  │ (LLM + tool calling)   │  │ │
+                         │ │  └────────────────────────┘  │ │
+                         │ │                              │ │
+                         │ │  ┌────────────────────────┐  │ │
+                         │ │  │ Connector marketplace  │  │ │
+                         │ │  │ (install/uninstall)    │  │ │
+                         │ │  │ • Mantis               │  │ │
+                         │ │  │ • Teams                │  │ │
+                         │ │  │ • <yours>              │  │ │
+                         │ │  └────────────────────────┘  │ │
+                         │ └─┬──────────┬─────────┬───────┘ │
+                         └───┼──────────┼─────────┼─────────┘
+                             │          │         │
+                  HTTP REST  │          │ HTTP    │ HTTP (optional)
+                             ▼          ▼         ▼
+                    ┌─────────────┐ ┌──────┐ ┌──────────────┐
+                    │   Temper    │ │ LLM  │ │   Forge      │
+                    │  (memory)   │ │ API  │ │ (when need   │
+                    │  existing   │ │      │ │  pipelines/  │
+                    │             │ │      │ │  long tasks) │
+                    └─────────────┘ └──────┘ └──────────────┘
+```
+
+**Three products with clear, asymmetric roles** (not three equal peers — each plays a distinct part):
+
+| Product | Status | Role | Required when |
+|---|---|---|---|
+| **Browser Extension** | 🆕 to build | **Agent entry point** — main user-facing surface. Connector hub for company internal + external SaaS systems. | Always (the agent's "face") |
+| **Forge** | ✅ existing | **Agent backend + dev platform** — pipelines, crafts, workspaces, code work, long-running tasks | When you need orchestration, code work, scheduled flows (not always for pure chat) |
+| **Temper** | ✅ existing | **Agent memory** — supplements Forge's memory when agent chat is involved | Only when you have an agent that needs cross-session memory. Forge-only (no chat agent) usage doesn't need it. |
+
+### Reading the trinity
+
+- **Extension** is the most user-visible. If a non-technical user only sees one thing, it's the extension.
+- **Forge** sits behind the extension as the backend / dev platform. Power users open Forge UI directly when they need pipeline editing, multi-agent workspaces, or code work.
+- **Temper** is purely a memory backplane. Optional. **Forge without an agent doesn't need Temper** — you can run pipelines / crafts / Forge UI fine without any memory service. Temper kicks in once a conversational agent (e.g. the extension's chat) needs to remember things across sessions.
+
+### Dependency direction
+
+```
+Browser Extension ───depends on───▶  Forge (when long task)
+                  ───depends on───▶  Temper (for memory)
+                  ───depends on───▶  LLM API (always)
+
+Forge             ──optionally uses──▶  Temper (only when an agent runs)
+
+Temper            stands alone, dependent on nothing else
+```
+
+**Nobody is a peer of "depends on me"** — Temper is the deepest, Forge is middle, Extension is the apex / user-facing layer.
+
+## Component Responsibilities
+
+### Browser Extension (new)
+
+| Responsibility | How |
+|---|---|
+| Chat UI | React in side panel (`chrome.sidePanel`) |
+| Agent loop | LLM tool-use loop (lightweight — direct API calls, no pi-coding-agent) |
+| Memory | HTTP client → Temper REST API |
+| Site access | Per-domain content scripts (connectors) + `fetch` with credentials |
+| Connector management | Marketplace UI in extension popup (install/remove/update) |
+| Forge bridge (optional) | `fetch('http://localhost:8403/api/…')` when user wants pipeline / craft / long task |
+
+**Does NOT do:**
+- Code editing / file ops (Forge job)
+- tmux / terminal (Forge job)
+- Long-running cron / scheduled tasks (Forge job)
+- Cross-machine sync (Temper job)
+
+### Temper (existing, no changes needed)
+
+- HTTP REST memory API
+- KV memory blocks (preferences, profile, current state)
+- Graph episodes (third-party facts, recall over time)
+- Multi-tenant, multi-user
+
+Extension is a Temper client; nothing new to build on Temper side for v1.
+
+### Forge (existing, minor additions later)
+
+- Web UI for pipelines / crafts / workspace (as today)
+- Backend services (terminal, workspace, telegram, MCP)
+- Extension can call Forge's HTTP API when it needs heavy orchestration
+- **No required changes for extension v1** — composable, the extension works without Forge
+
+Later (post-v1):
+- Forge "Browser Bridge" service for extension-to-Forge MCP bridging (per RFC)
+- Forge UI showing connected extensions / paired devices
+
+## Connector Model
+
+Connectors are **installable per-company**:
+- Each company has their own internal systems
+- A connector is a JS module: `id`, `matches: [urlPattern]`, `tools: {...}`, optional `healthCheck`
+- Installable from a public registry (`forge-browser-connectors` GitHub repo, mirrors `forge-crafts`)
+- Or sideload-able from a company's private registry (URL configurable in extension settings)
+
+**v1 ships with empty marketplace.** Users add their own (or pick from public ones once registry exists).
+
+## Tech Stack
+
+| Layer | Choice | Why |
+|---|---|---|
+| Extension manifest | MV3 | Chrome's mandatory modern format |
+| Background | Service Worker | Required for MV3; OK because we don't need long-lived processes (the agent loop is request/response) |
+| UI framework | React 18 + TypeScript + Vite | Match Forge stack; Vite for fast SPA build |
+| State | Zustand (or jotai) | Lightweight; React Context for connector registry |
+| LLM SDK | `@anthropic-ai/sdk` + optional OpenAI | User config; Anthropic primary |
+| Memory client | Hand-rolled REST (or generated OpenAPI client) → Temper | Temper has stable API |
+| Connector SDK | Local TS package | Defines `defineBrowserConnector({...})` |
+| Bundler | Vite + `@crxjs/vite-plugin` | Standard MV3 toolchain |
+| Package layout | Monorepo? Or single repo? | Decision: single repo `forge-browser-extension`, with `packages/sdk` workspace for the connector SDK to be published to npm |
+
+## Phase Plan (4 weeks → MVP)
+
+### Week 1: Skeleton
+
+- New repo: `forge-browser-extension`
+- MV3 manifest, side panel, background SW, content script
+- Chat UI shell (input → static placeholder response)
+- Settings page: Temper URL, LLM API key
+- Build pipeline (Vite + crxjs) producing `dist/` loadable as unpacked extension
+- **Deliverable**: extension loads in Chrome, side panel opens, you can type something + see your own message echoed back
+
+### Week 2: Agent + Memory
+
+- Wire LLM API (Anthropic streaming)
+- Implement minimal tool-use loop (5-10 hand-coded tools, no connectors yet)
+- Temper memory:
+  - on session boot: load relevant memory blocks
+  - after every turn: extract & write back any "fact about user" → memory blocks
+  - episodes: write per-session to Temper graph
+- **Deliverable**: chat with persistent memory; close Chrome, reopen, agent remembers what you told it last time
+
+### Week 3: Connector framework + first connector
+
+- `@forge/browser-connector-sdk` (in `packages/sdk/`):
+  - `defineBrowserConnector({...})` factory
+  - `ctx.fetch / $ / $$ / waitFor / storage / log / AuthExpiredError`
+- Connector loader in extension:
+  - Discover connectors in `chrome.storage` (installed list)
+  - On tab match, inject content script
+  - Register tools globally
+- **Mantis connector** (first one, since you specifically mentioned it):
+  - Tier 1: `/api/rest/issues` with cookies
+  - Tier 2: scrape `view_all_bug_page.php`
+  - Tools: `mantis.list_my_bugs`, `mantis.view_bug`, `mantis.add_comment`
+- **Deliverable**: extension chat can say "list my open Mantis bugs" → returns real data
+
+### Week 4: Marketplace + second connector
+
+- Connector marketplace UI in extension popup
+- `forge-browser-connectors` registry (GitHub repo, schema mirrors `forge-crafts`)
+- Install / uninstall / update flows
+- **Second connector** (pick one: Teams or your highest-value internal system)
+- Connector health checks + auth-expired notification
+- **Deliverable**: end-to-end real use; ready to use day-to-day on at least 2 systems
+
+### Beyond MVP
+
+- Forge bridge (Phase 0-1 of RFC) — extension can call Forge MCP for heavy tasks
+- More connectors
+- Vision-tier fallback for awkward sites
+- Possible: smith engine folded in as alternative agent loop
+
+## Code Locations
+
+| Code | Repo | Path |
+|---|---|---|
+| Extension app | NEW `forge-browser-extension` | `src/` |
+| Connector SDK | NEW `forge-browser-extension` | `packages/sdk/` |
+| Connector registry index | NEW `forge-browser-connectors` (mirror `forge-crafts`) | `registry.json` + `<connector>/` |
+| Forge integration (later) | Existing Forge repo | `lib/browser-bridge-standalone.ts` (Phase 0+ per RFC) |
+| Temper changes | None | N/A (use existing API) |
+
+## Decisions Resolved
+
+| Question | Resolution |
+|---|---|
+| Bundle backend into extension? | No — backend stays in Forge node process; extension is UI + connectors only |
+| Run browser in Docker? | No — re-use user's session is the whole point |
+| Connectors per-company or generic? | Per-company, installable via marketplace, with public + private registries |
+| Forge required for extension to work? | No — composable. Extension works standalone for chat + connectors |
+| Smith engine folded into extension? | Deferred — start with simpler hand-rolled agent loop, fold smith later if needed |
+| MV3 service worker concerns? | Acceptable — chat is request/response, no long-lived state needed. Agent loop is per-message |
+| Chat UI location | Side panel (Chrome 117+). Popup for marketplace + settings |
+
+## Open Questions
+
+| Question | Need to answer by |
+|---|---|
+| Which LLM as default? Anthropic only, or also support OpenAI / local? | Week 1 (config) |
+| Memory granularity: every turn → memory? Or only on important facts? | Week 2 |
+| Connector code signing / supply-chain security | Week 3 (when registry goes public) |
+| Public connector registry URL — host on github.com/aiwatching/forge-browser-connectors? | Week 3 |
+| Auth expiry UX — toast? Inline chat message? Auto-resume? | Week 3 |
+| Chrome Web Store publication strategy | Week 4 (or unsigned sideload for first internal users) |
+
+## Tomorrow's First Concrete Step
+
+```bash
+mkdir -p ~/IdeaProjects/forge-browser-extension
+cd ~/IdeaProjects/forge-browser-extension
+git init
+gh repo create aiwatching/forge-browser-extension --private --source . --remote origin
+
+# Scaffold:
+pnpm init
+pnpm add -D vite @crxjs/vite-plugin @types/chrome typescript react @types/react @types/react-dom
+pnpm add @anthropic-ai/sdk zustand
+
+# Files to create on day 1:
+# manifest.json (MV3)
+# vite.config.ts (crxjs preset)
+# src/background.ts (SW skeleton)
+# src/sidepanel/index.html + sidepanel.tsx (chat UI shell)
+# src/popup/index.html + popup.tsx (settings + marketplace placeholder)
+# src/content/index.ts (per-tab content script bootstrap)
+```
+
+Aim by end of day 1: load unpacked in Chrome, click extension icon → side panel opens with React app saying "Hello".
+
+## Success Criteria (4 weeks)
+
+A user can:
+1. Install the extension once
+2. Configure Temper URL + LLM API key once
+3. Open the side panel from any tab
+4. Type: "list my open Mantis bugs"
+5. See real bug data from their Mantis without re-authenticating
+6. Continue: "summarise the top 3 by severity" — agent uses LLM
+7. Close Chrome, come back tomorrow — agent remembers user preferences
+8. Install a second connector from the marketplace with one click
+
+If all 8 work end-to-end, MVP is done.
+
+## Out of Scope (defer)
+
+- Mobile / Safari / Firefox
+- Centralized headless browser farm
+- Forge UI inside extension (Forge stays at `localhost:8403`)
+- Connector code signing infrastructure
+- Multi-user / SSO at extension level (Temper handles user separation)
+- Computer Use vision-tier fallback
+- Smith engine integration
+
+## ADR — 2026-05-16: Connectors are Forge plugins, not extension code
+
+After initial RFC, decided to **move connector implementations out of the
+extension and into Forge as plugins** (the existing `lib/builtin-plugins/*`
+system: `docker`, `jenkins`, `slack`, `llm-vision`, …).
+
+### Why plugins not crafts
+- Crafts are per-project + UI-heavy. Connectors are system-level, stateless,
+  pure capabilities.
+- Plugins already have manifest/marketplace machinery in Forge.
+- Plugins are already the "system capability" abstraction agents call.
+
+### Why not keep them in extension
+- Extension stays thin (chat UI + memory client + thin RPC to Forge).
+- Connector marketplace = Forge plugin marketplace (don't double-build).
+- Adding a connector = adding a YAML to Forge. No JS compiled into extension.
+
+### Plugin execution modes (new addition to Forge plugin schema)
+```yaml
+mode: server-side    # default; Forge backend calls external API
+mode: browser-side   # Forge dispatches to extension which fetches via tab cookies
+mode: hybrid         # per-tool: some server, some browser
+```
+
+### Data flow (Mantis, browser-side mode)
+```
+Extension chat → POST /api/plugins/mantis/invoke
+              ← Forge sees mode=browser-side → WS dispatch to extension
+              ← Extension runs content-script fetch in mantis tab w/ cookies
+              ← Mantis JSON → back up the chain → LLM → user
+```
+
+### Dependency revision
+With this model, the extension's connector capabilities **require Forge**.
+Pure-chat mode (no connectors) still works without Forge. The earlier
+"all composable, all standalone" framing was too theoretical — for any
+real enterprise use, the three-product stack is needed together.
+
+### Forge changes required (small but real)
+1. Plugin manifest accepts `mode: browser-side | server-side | hybrid`
+2. Plugin manifest accepts `category: connector | pipeline-node | mcp-server | tool | other` — marketplace + extension filter on this
+3. Plugin manifest accepts a `connectors: [...]` array (default 1:1 plugin↔connector; 1:N as escape hatch for same-vendor suites with shared auth — see RFC "Packaging in Forge")
+4. New plugin executor: when `mode=browser-side`, dispatch to extension via WS
+5. WebSocket back-channel: extension ↔ Forge persistent connection
+6. Extension capability registry: Forge tracks "which extension has which tabs matching which plugins"
+7. Plugin settings UI: show extension connectivity state + paired connectors
+
+### Defaults shipped with Forge
+- `gmail` (server-side, OAuth) — built-in
+- `teams` (browser-side, MSAL token extraction) — built-in
+- `jira` (hybrid — cloud OAuth, self-hosted cookie) — built-in
+- `github` (server-side, PAT) — built-in
+- `gitlab` (server-side, PAT) — built-in
+- User-installed: `mantis` and others from plugin marketplace
+
+### What this changes in the 4-week plan
+- **Week 3 deliverable changes**: instead of "connector SDK + first connector in extension", it becomes "first browser-side plugin in Forge + extension dispatch mechanism".
+- Extension code becomes thinner — no `defineBrowserConnector` SDK in extension itself; that lives as a Forge plugin author guide.
+- `forge-browser-connectors` registry → **not needed**; reuse Forge plugin marketplace.
+
+## ADR — 2026-05-16: Data architecture (what goes into Temper)
+
+Key rule: **opportunistic memory, not bulk mirroring.** Temper is the
+agent's *memory*, not a data lake. Mantis / Teams / Gmail stay as the
+sources of truth; we don't sync everything wholesale.
+
+### Data categories
+
+| Category | Example | Storage | Why |
+|---|---|---|---|
+| Conversation | "user asked X, agent replied Y" | Temper **episodes** (graph) | Core memory asset |
+| First-person facts | "user prefers concise answers", "user's team lead is Alice" | Temper **memory blocks** (KV) | Graphiti not great at these |
+| Agent-processed content | bug #142 full text, meeting notes | Temper **wiki** (new) | RAG / long-form retrieval |
+| Tool-call events | "called mantis.list_my_bugs, got 12 results" | Temper episode | Agent's own experience log |
+| Transient search results | current Mantis result page, Teams query | Extension `chrome.storage` (TTL min) | Source is authoritative, refetch cheap |
+| Files / code | git diff, file tree | NOT Temper — Forge workspace owns | Different concern |
+| Credentials / cookies | tokens, passwords | NEVER STORED | Security red line |
+
+### Three-tier data flow
+
+```
+Source system (Mantis / Teams / Gmail / GitHub)  ← single source of truth
+    │ fetch on demand only
+    ▼
+Extension chrome.storage / IndexedDB (TTL minutes)  ← per-session cache
+    │ promote to persistent memory when:
+    │   - agent actually used the data (auto)
+    │   - user said "remember this" (explicit)
+    │   - agent flagged as significant (auto, heuristic)
+    ▼
+Temper (cross-session, cross-device, cross-agent)
+    ├── episodes (graph) — chat + tool calls + agent events
+    ├── memory blocks (KV) — first-person facts
+    ├── wiki — agent-processed long-form content
+    └── postgres (Temper internal) — user/auth/meta
+```
+
+### Write strategy (Week 2 implementation)
+
+Every chat turn end, async write a batch to Temper:
+
+1. **Episode for the conversation turn** — always (cheap, builds context)
+2. **Episode per tool call** — always (summary, no raw secrets)
+3. **Memory blocks** for first-person facts extracted by LLM from turn
+4. **Wiki entry** when:
+   - User says "记住 / remember this / save this"
+   - Agent processes a specific document/ticket end-to-end (auto)
+   - Agent generates a long-form artifact (status report, summary)
+
+Sensitive fields (tokens, cookies, password-shaped strings) sanitized
+before any Temper write.
+
+### Read strategy (every turn start)
+
+1. Episode semantic search by user message → top N (~10)
+2. Load all memory blocks for current user
+3. Wiki keyword/semantic search → top 3
+4. Build LLM system context from above
+
+### What to NEVER do
+
+- ❌ Background cron mirroring Mantis / Teams full content into Temper
+- ❌ Storing raw tool-call JSON unfiltered (noise + secrets risk)
+- ❌ Using Temper as BI / aggregation store — it's per-user agent memory
+- ❌ Storing files / code / large binaries — that's Forge workspace's job
+
+### Wiki's specific role (new Temper primitive)
+
+Wiki sits between KV (short, structured) and episodes (medium, eventful):
+
+| Primitive | Length | Update freq | Retrieval | Example |
+|---|---|---|---|---|
+| KV blocks | sentence | low | by key | "team lead = Alice" |
+| Episodes | paragraph | every turn | semantic recall | "5/16 fetched bug #142, user satisfied" |
+| **Wiki** | document | when meaningful | full-text + semantic | "bug #142 full content + agent's deep analysis" |
+
+Use wiki for:
+- Specific ticket content the agent worked on (cached with backlink to source)
+- "Save this Confluence page" requests
+- Agent-generated reports / status updates / decision records
+- Team-level SOPs and shared docs
+
+### Integration timeline
+
+| Week | What |
+|---|---|
+| Week 2 | Episodes (chat + tool calls) + KV blocks (extracted facts) write/read on every turn |
+| Week 3 | Wiki writes when agent processes connector content end-to-end |
+| Week 4+ | Smart promotion (episode → wiki by access frequency / agent tags) |
+| Later | Forge pipelines also write episodes — chat and task share one memory plane |
+
+### One-line principle
+
+**Not "everything the connectors fetch goes to Temper" but "everything
+the agent actually experiences goes to Temper".** Fetching is data
+movement (don't auto-store); experiencing is agent memory (auto-store).
+
+## ADR — 2026-05-16: DOM extraction is the default execution path
+
+Surfaced after testing Mantis: REST API returned 403 because MantisBT's
+`/api/rest/*` requires an `Authorization` token, not a cookie. Same
+applies to GitLab `/api/v4/*`, JIRA Server REST, Teams Graph, etc.
+
+**Decision**: connector tools run via **DOM extraction through the
+extension's Chrome MCP capabilities**, not REST API calls.
+
+### Why
+
+If we route through REST, every connector requires the user to mint and
+manage a per-system API token. That's exactly the friction the browser
+extension model exists to remove. The whole product story is "use the
+browser session you already have, no setup". DOM-first preserves that.
+
+### Implementation rules
+
+- Tool descriptions describe **observable behavior** ("list bugs assigned
+  to me"), never URL paths.
+- Extension's connector runtime navigates tabs and parses HTML — does NOT
+  call `/api/rest/*` or `/api/v4/*`.
+- `ctx.fetch` is reserved for same-origin XHR endpoints the rendered page
+  itself uses (cookie-authenticated). Not for documented REST APIs.
+- Token-based fallback (`api_token` setting) is **opt-in per connector**,
+  only when DOM extraction is genuinely unworkable (canvas SPAs etc.).
+  Off by default.
+
+### Cost we accept
+
+- Selector maintenance when sites redesign — pin stable attributes
+  (`data-testid`, role-based queries, schema.org microdata).
+- Slower (200ms-2s vs 50-500ms for REST) — acceptable for interactive use.
+- Some pages need scrolling / interactions before all data is in DOM —
+  connector handlers wait for selectors and trigger interactions as needed.
+
+### What this changes
+
+- RFC's three-tier model rewritten: DOM is Tier 1, vision is Tier 2,
+  optional user-provided REST token is Tier 3 (escape hatch).
+- Connector examples updated (no more "Tier 1 REST → Tier 2 DOM" — flipped).
+- Mantis + GitLab manifests reflect this in their descriptions.
+
+## Changelog
+
+- **2026-05-16**: Initial plan
+- **2026-05-16 (later)**: ADR — connectors live in Forge as plugins, not in extension. Major simplification.
+- **2026-05-16 (later 2)**: ADR — data architecture: opportunistic memory not bulk mirroring; wiki for long-form agent-processed content.
+- **2026-05-16 (later 3)**: ADR — DOM extraction is the default execution path, not REST. Token-based fallback is opt-in per connector.