@aion0/forge 0.6.1 → 0.8.0

package/docs/Connector-DeclarativeExtract-Spec.md

@@ -0,0 +1,364 @@
+# Connector — Declarative Extraction Spec
+
+> **Goal**: Adding a new browser-side connector should be a Forge-only change.
+> No `pnpm build && release` of the browser extension required.
+>
+> **Today**: Forge ships the manifest (schema), Extension ships a hand-written
+> handler (`mantis.ts`, eventually `gitlab.ts`, …). Adding a new connector
+> means coordinated changes in two repos.
+>
+> **Target**: Manifest carries the extraction script. Extension is a generic
+> runner that knows nothing about specific connectors.
+
+---
+
+## 1. Architecture before / after
+
+### Today
+```
+Forge                            Extension
+─────                            ─────────
+mantis.yaml      ──────────────▶ list of tool schemas
+(schema only)                    │
+                                 │ LLM calls mantis.list_my_bugs
+                                 ▼
+                                 mantis.ts (hand-written, ~300 LOC)
+                                  ├─ findMantisTab()         ◀── Mantis-specific
+                                  ├─ ensureOnListPage()      ◀── Mantis URLs hardcoded
+                                  ├─ extractBugList()        ◀── #bug_list selector hardcoded
+                                  └─ Forge column quirks  ◀── per-instance hardcoded
+                                 │
+                                 ▼
+                                 chrome.scripting.executeScript → result
+```
+**Adding GitLab**: write `gitlab.ts` in extension, ship new extension version.
+
+### Target
+```
+Forge                                    Extension
+─────                                    ─────────
+mantis.yaml                              src/lib/connectors/runner.ts
+  tools:                                 (≤ 100 LOC, generic)
+    list_my_bugs:
+      page: { url, on_target, ... }     │ LLM calls mantis.list_my_bugs
+      script: |  ──────────────────────▶│
+        const list = document.qs(...)   │
+        return { bugs, ... }            ▼
+                                        runConnectorTool(plugin, tool, args)
+                                         ├─ acquireTab(host_match, url)
+                                         ├─ waitForLoad
+                                         ├─ detectLoginRedirect
+                                         └─ executeScript(script, args) → result
+```
+**Adding GitLab**: drop `gitlab.yaml` in Forge with `page` + `script` per tool. Done.
+
+---
+
+## 2. Schema change to plugin YAML
+
+A connector plugin's `tools` map gets two new keys per tool:
+`page` (where to extract from) and `script` (the in-page function body).
+
+```yaml
+id: mantis
+name: MantisBT
+icon: "🐞"
+version: "0.2.0"     # bump on schema change
+category: connector
+mode: browser-side
+
+settings:
+  base_url:
+    type: string
+    label: Mantis base URL
+    required: true
+
+# plugin-level: how to find / open an authenticated tab on this host
+host_match: "{base_url}/*"      # Chrome match pattern, used to find existing tabs
+login_redirect: "/login_page.php"   # substring; if URL contains this after nav, abort
+
+tools:
+  list_my_bugs:
+    description: List bugs assigned to me.
+    parameters:
+      status:
+        type: select
+        options: ["open", "closed", "all"]
+        default: "open"
+      limit:
+        type: number
+        default: 50
+
+    page:
+      url: "{base_url}/view_all_bug_page.php"
+      on_target: "/view_all_bug_page.php"   # substring match; if current URL has this, don't navigate
+
+    # Function body. Receives `args` (the LLM's parsed parameters).
+    # Returns a JSON-serializable value. Top-level Web APIs only.
+    # NO closures over Forge / extension globals — this runs in the user's tab.
+    script: |
+      const { status, limit } = args;
+      const list = document.querySelector('#bug_list');
+      if (!list) return { bugs: [], total: 0, _error: '#bug_list not found' };
+      // ... ports the body of mantis.ts/extractBugList() ...
+      return { bugs, total, _page: location.href };
+```
+
+### Field reference
+
+| Field | Level | Required | Notes |
+|---|---|---|---|
+| `host_match` | plugin | yes (browser-side) | Used by extension to find/open tabs. Chrome match pattern with `{settings.*}` substitution. |
+| `login_redirect` | plugin | optional | Substring. If the tab URL contains this after navigation, treat as "user not logged in". |
+| `page.url` | tool | yes | Template URL to navigate to. `{base_url}` etc. expanded from settings. |
+| `page.on_target` | tool | optional | If current tab URL contains this substring, skip the navigation step (current page is fine). |
+| `script` | tool | yes | Function body. Implicit param: `args`. Must return a JSON-serializable value. |
+
+### Multi-step tools (defer)
+
+`add_comment`-style tools that POST need form interaction (fill input + click submit + wait for redirect). For v1 we only spec read-only extraction. Destructive tools stay marked `destructive: true` but **`script` is allowed to call `fetch()` from within the tab context** (same-origin, cookies auto-attached) for cases where the rendered HTML form is too brittle. That's a per-connector decision the script author makes.
+
+---
+
+## 3. HTTP API change — `GET /api/connectors`
+
+Today's response (per `app/api/connectors/route.ts`):
+```json
+{
+  "connectors": [
+    {
+      "plugin_id": "mantis",
+      "name": "MantisBT",
+      "mode": "browser-side",
+      "installed": true,
+      "entries": [
+        {
+          "id": "mantis",
+          "host_permissions": ["..."],
+          "tools": {
+            "list_my_bugs": {
+              "description": "...",
+              "parameters": {...},
+              "returns": "..."
+            }
+          },
+          "settings": {...}
+        }
+      ]
+    }
+  ]
+}
+```
+
+After this change, **`tools[name]` must additionally include `page` and `script`** (and the plugin level needs `host_match` + `login_redirect`):
+
+```json
+{
+  "connectors": [
+    {
+      "plugin_id": "mantis",
+      "name": "MantisBT",
+      "mode": "browser-side",
+      "installed": true,
+      "host_match": "https://mantis.acme.com/*",
+      "login_redirect": "/login_page.php",
+      "entries": [
+        {
+          "id": "mantis",
+          "tools": {
+            "list_my_bugs": {
+              "description": "...",
+              "parameters": {...},
+              "page": {
+                "url": "https://mantis.acme.com/view_all_bug_page.php",
+                "on_target": "/view_all_bug_page.php"
+              },
+              "script": "const { status, limit } = args; ... return { bugs, total };"
+            }
+          },
+          "settings": {...}
+        }
+      ]
+    }
+  ]
+}
+```
+
+Note `{base_url}` / `{settings.base_url}` is **expanded server-side** using the user's saved settings before the response goes out — extension shouldn't need to know template syntax. (If the user hasn't saved settings yet, omit the tool from the listing or return `null` for `page.url`.)
+
+---
+
+## 4. Template substitution rules
+
+Anywhere `page.url`, `host_match`, `login_redirect` appear, expand `{<key>}` tokens:
+
+- `{base_url}` → `settings.base_url` (with trailing slashes stripped)
+- `{settings.<name>}` → `settings.<name>` (verbose form)
+- Unknown tokens → leave as literal `{foo}` (don't error — gives author a way to ship a static URL with curly braces)
+
+Do this in Forge before returning the manifest. Avoids re-implementing on the extension side.
+
+---
+
+## 5. Script execution contract (what the extension will do)
+
+The extension will:
+1. Take `tool.script` as a string
+2. Find or open a tab matching `host_match`, navigating to `page.url` unless `page.on_target` says current URL is fine
+3. Detect `login_redirect` substring in tab URL → return `{ error: 'login required', loginRequired: true }`
+4. Run via `chrome.scripting.executeScript`:
+   ```js
+   // executes IN THE USER'S TAB (Mantis page context)
+   func: function(scriptText, args) {
+     try {
+       const fn = new Function('args', scriptText);
+       return { ok: true, value: fn(args) };
+     } catch (err) {
+       return { ok: false, error: err.message };
+     }
+   }
+   ```
+5. Return `result.value` to the LLM as the tool result
+
+**Script authors should assume**:
+- `document`, `fetch`, `URL`, etc. are available
+- Same-origin cookies are attached to `fetch()` calls
+- No access to `chrome.*` APIs (running in page context)
+- No closures over Forge/extension code — script is a self-contained function body
+- Return value must be JSON-serializable (no DOM nodes, no functions)
+- Errors should be caught and returned as `{ _error: '...' }` shape; thrown errors are also caught by the runner
+
+---
+
+## 6. Security considerations
+
+This change moves trust: extension now executes arbitrary JS shipped by Forge. Constraints:
+
+| Risk | Mitigation |
+|---|---|
+| Compromised Forge serves malicious script | Forge is already a trusted authenticated peer (`X-Forge-Token`). User explicitly chose to connect. Same trust level as `npm install`. |
+| Script reads from / writes to sites it shouldn't | Run only in the tab matching `host_match`; never inject into arbitrary tabs. |
+| `new Function` blocked by page CSP | Most internal corp tools have permissive CSPs. Strict-CSP sites (github.com, banks) will fail; document as a known limitation. For those, write a server-side connector instead (REST with token). |
+| Script exfiltrates page content | Already inherent to the "scrape user's page" design — script could always do this. Extension UI should log connector tool calls so user can audit. |
+
+**Recommendation for v1**: log every script run to extension console with `{plugin_id, tool, tab_url, script_length}`. Add a "Connector audit log" panel in Settings later.
+
+---
+
+## 7. Concrete: full migration of `mantis.yaml`
+
+The current `mantis.ts` in extension has these page-script blocks:
+
+- `extractBugList(args)` — for `list_my_bugs`, runs on `/view_all_bug_page.php`
+- `extractBugDetail(args)` — for `get_bug`, runs on `/bug_view_page.php?bug_id=N`
+
+These two functions get **copy-pasted into the YAML's `script` fields** (minus type annotations — script is plain JS). Their function bodies are already self-contained, so the port is mechanical:
+
+```yaml
+id: mantis
+name: MantisBT
+version: "0.2.0"
+category: connector
+mode: browser-side
+
+settings:
+  base_url: { type: string, required: true, label: "Mantis base URL" }
+  default_project: { type: string, label: "Default project" }
+
+host_match: "{base_url}/*"
+login_redirect: "/login_page.php"
+
+tools:
+  list_my_bugs:
+    description: |
+      List bugs assigned to the current user.
+    parameters:
+      status:
+        type: select
+        options: ["open", "closed", "all"]
+        default: "open"
+      limit:
+        type: number
+        default: 50
+    page:
+      url: "{base_url}/view_all_bug_page.php"
+      on_target: "/view_all_bug_page.php"
+    script: |
+      // Body of current extractBugList in src/lib/connectors/mantis.ts
+      // (paste lines 69–148 of mantis.ts here, drop the function signature)
+      const list = document.querySelector('#bug_list');
+      if (!list) return { bugs: [], total: 0, _error: '#bug_list not found' };
+      // ... etc
+      return { bugs: filtered.slice(0, args.limit), total: filtered.length };
+
+  get_bug:
+    description: |
+      Get full details of a single bug.
+    parameters:
+      bug_id:
+        type: number
+        required: true
+    page:
+      # Note: bug_id is in args, so we use it in the URL via template
+      # extension expands `{args.bug_id}` at run-time (this is the one case
+      # where args, not settings, drives the URL — special-case in spec).
+      url: "{base_url}/bug_view_page.php?bug_id={args.bug_id}"
+      on_target: "/bug_view_page.php"
+    script: |
+      // Body of current extractBugDetail
+      ...
+```
+
+> ⚠ **`{args.*}` substitution in URLs is the one extra template feature** —
+> needed for tools that target a specific record by ID. Forge needs to do
+> two passes: settings-time substitution at API response time, then args-time
+> substitution inside the extension when actually executing the tool. (Or
+> Forge can leave `{args.*}` literals unexpanded; the extension finishes
+> them.) **Recommendation**: leave `{args.*}` for the extension to expand
+> from the LLM's input; Forge only expands `{settings.*}` / `{base_url}`.
+
+The `mantis.probe.js` file stays as a dev tool; it's not shipped to extension at runtime.
+
+---
+
+## 8. Migration order
+
+To avoid breaking the extension during the change:
+
+1. **Forge: add the schema fields, no behavior change**
+   - Extend manifest parser to accept `page` + `script` + `host_match` + `login_redirect`. Tools missing them are still valid (old shape).
+   - Update `/api/connectors` to emit the new fields when present, omit when absent.
+   - Mantis manifest gains `page` + `script` for both tools. Bump version `0.1.1` → `0.2.0`.
+   - Verify with curl: `/api/connectors/mantis` returns the new shape, fields populated.
+
+2. **Extension: write the generic runner alongside the existing per-connector handler**
+   - New file `src/lib/connectors/runner.ts` implementing `runConnectorTool(connector, toolName, input)`.
+   - Update `build-tools.ts` so a tool is "executable" if its manifest has `script` (not "if `hasHandler(plugin_id)`").
+   - Routing: if tool has `script` → runner; else (fallback) → old `dispatchConnector(plugin_id, …)` for backward compat.
+
+3. **Cut over**
+   - Once the runner works for `mantis.list_my_bugs` end-to-end, delete `src/lib/connectors/mantis.ts` and the `mantis` entry in `registry.ts`.
+   - Remove the fallback branch in routing.
+
+4. **Add second connector to prove the loop**
+   - Pick the simplest GitLab page (e.g. "my issues") and write the YAML. Confirm extension picks it up with zero extension-side changes.
+
+---
+
+## 9. Acceptance criteria
+
+A connector is "shippable via Forge-only changes" iff:
+
+- ✅ Adding a new `<name>.yaml` in `lib/builtin-plugins/` makes the tool visible in extension on next Connectors-tab refresh.
+- ✅ Saving connector settings (`base_url`) makes the tool callable by the LLM.
+- ✅ LLM calls succeed end-to-end without rebuilding/reloading the extension binary.
+- ✅ Existing `mantis` connector continues to work throughout the migration (no flag day).
+- ✅ Bumping a connector's `version` and changing only the YAML's `script` body propagates to all extension users on next refresh — no extension release needed.
+
+---
+
+## 10. Open questions for Forge side
+
+- [ ] Where does the YAML's `script` body actually live in a multi-line YAML? Confirm Forge's YAML parser preserves it verbatim (it should — block scalar `|` is the right syntax).
+- [ ] For users on `installed=false`, omit `page.url` / `script` from the response, or include them so the marketplace UI can preview? (Recommendation: include, but with `{settings.base_url}` left literal, so consumer can preview.)
+- [ ] Probe scripts (`mantis.probe.js`) — keep as devtool only, or also serve via `/api/connectors/<id>/probe` so the extension can re-run probe on a "selectors broken" report? (Recommendation: defer — solve when selectors actually break.)