@aion0/forge 0.6.3 → 0.8.1

package/docs/Connector-DeclarativeExtract-Handoff.md

@@ -0,0 +1,471 @@
+# Forge change request — declarative connector extraction
+
+> **Read this first**: full design rationale lives in
+> [`Connector-DeclarativeExtract-Spec.md`](./Connector-DeclarativeExtract-Spec.md).
+> This file is the **change list** for the Forge code + a ready-to-commit
+> `mantis.yaml` so you can ship without further design discussion.
+
+## TL;DR
+
+Today: adding a connector means coordinated changes in Forge (manifest) **and** the browser extension (handler code).
+Goal: adding a connector should be a **Forge-only YAML change**. Extension is now a generic runner; it has no Mantis/GitLab-specific code.
+
+Status:
+- Extension side is done. `src/lib/connectors/runner.ts` is a generic executor; `mantis.ts` / `registry.ts` / `types.ts` are deleted. The extension can run ANY connector whose manifest contains the new fields.
+- **Forge side is what this doc is about.** Until Forge ships the new fields, the LLM can't see any connector tools (the extension's gate filters out tools without `script`).
+
+## What Forge needs to change
+
+### 1. Plugin manifest parser — accept four new fields
+
+Allowed YAML at the plugin level:
+
+| Field | Required | Notes |
+|---|---|---|
+| `host_match` | yes (for browser-side mode) | Chrome match pattern. `{settings.*}` template variables allowed. Used by extension to find or open an authenticated tab. |
+| `login_redirect` | optional | Substring. If the tab URL contains this after navigation, the runner returns `loginRequired: true`. |
+
+Allowed at the **tool** level (under `tools.<name>`):
+
+| Field | Required | Notes |
+|---|---|---|
+| `page.url` | yes (executable tool) | URL the runner navigates to before running the script. `{settings.*}` AND `{args.*}` template variables allowed. |
+| `page.on_target` | optional | Substring. If current tab URL already contains this, skip the navigation step. |
+| `script` | yes (executable tool) | Function body. Receives `args` (LLM's parsed parameters), returns JSON-serializable. Runs in the user's tab. |
+
+Tools missing `page` or `script` are still valid manifests — they just won't be callable by the LLM. (Useful for tools that you want to surface in the Connectors UI but haven't yet implemented.)
+
+### 2. `GET /api/connectors[?id=<id>]` response — include the new fields
+
+The response body must include the new fields on every connector and every tool. Settings-template substitution happens server-side; args-template substitution happens client-side.
+
+```jsonc
+// GET /api/connectors  →
+{
+  "connectors": [
+    {
+      "plugin_id": "mantis",
+      "name": "MantisBT",
+      "icon": "🐞",
+      "version": "0.2.0",
+      "mode": "browser-side",
+      "installed": true,
+
+      // NEW — expanded from settings.base_url on the server side
+      "host_match": "https://mantis.acme.com/*",
+      "login_redirect": "/login_page.php",
+
+      "entries": [
+        {
+          "id": "mantis",
+          "tools": {
+            "list_my_bugs": {
+              "description": "...",
+              "parameters": {...},
+              // NEW — also expanded server-side
+              "page": {
+                "url": "https://mantis.acme.com/view_all_bug_page.php",
+                "on_target": "/view_all_bug_page.php"
+              },
+              // NEW — verbatim string (no template substitution; runs as JS)
+              "script": "const list = document.querySelector('#bug_list');\n..."
+            }
+          },
+          "settings": {...}
+        }
+      ]
+    }
+  ]
+}
+```
+
+### 3. Template substitution
+
+Forge expands at API response time:
+- `{base_url}` → `settings.base_url`
+- `{settings.<name>}` → `settings.<name>`
+- Strip trailing slashes from `base_url` before substituting (so `{base_url}/foo` doesn't double-slash)
+
+Forge does **not** expand `{args.*}`. The extension does that at tool-call time. (It also URL-encodes `{args.*}` substitutions inside URLs to be safe.) Forge can ship `{args.bug_id}` literally inside `page.url`.
+
+If a tool's `page.url` contains an unexpanded `{settings.foo}` because the user hasn't saved that setting yet, return the tool with `page.url` unchanged (literal `{settings.foo}`). The extension will surface a clear error when called.
+
+### 4. `installed=false` connectors
+
+For uninstalled connectors, the user hasn't saved `base_url` etc. Return `page.url` / `host_match` with `{settings.*}` left literal so the Connectors marketplace UI can still preview the manifest.
+
+### 5. Plugin id, version, mode
+
+Bump version on every manifest content change. The extension's Connectors tab uses `version` to flag stale cached manifests on refresh. Mode stays `browser-side` for these.
+
+## Acceptance criteria
+
+✅ Adding `lib/builtin-plugins/foo.yaml` with `host_match` + `tools.bar.page` + `tools.bar.script` makes `foo.bar` callable by the LLM after extension refresh, with no extension code changes.
+
+✅ Bumping a `script` body in a YAML and restarting Forge propagates to all extension users on next Connectors tab refresh — no extension release.
+
+✅ Old manifests (no `page` / `script` / `host_match`) parse without error; tools just don't show up as callable.
+
+✅ `mantis.list_my_bugs` works end-to-end:
+   1. User asks "list my Mantis bugs" in chat
+   2. LLM calls `mantis.list_my_bugs`
+   3. Extension runner: finds existing Mantis tab OR opens background tab on `host_match`
+   4. Runner navigates to `page.url`
+   5. Runner executes the `script` body in MAIN world via `chrome.scripting.executeScript`
+   6. LLM gets `{bugs: [...]}` back, summarizes for the user
+
+## Full `mantis.yaml` — ready to commit
+
+Drop this in `lib/builtin-plugins/mantis.yaml`. It contains the same five tools as before, with the extension's old `extractBugList` / `extractBugDetail` / `extractProjects` / `addBugnoteInPage` bodies inlined as `script`. Tested against Mantis 1.x classic (Forge self-hosted theme).
+
+```yaml
+id: mantis
+name: MantisBT
+icon: "🐞"
+version: "0.2.0"
+author: forge
+category: connector
+mode: browser-side
+description: |
+  Read + comment on MantisBT bugs from the user's logged-in browser session.
+  Reuses the existing PHPSESSID cookie — no API token required.
+
+  All tools run as DOM extraction scripts inside the user's Mantis tab,
+  shipped by Forge in this manifest. The extension is generic.
+
+settings:
+  base_url:
+    type: string
+    label: Mantis base URL
+    description: "e.g. https://mantis.mycompany.com (no trailing slash)"
+    required: true
+  default_project:
+    type: string
+    label: Default project name
+    description: "Optional — used when no project specified"
+
+host_match: "{base_url}/*"
+login_redirect: "/login_page.php"
+
+tools:
+  list_my_bugs:
+    description: |
+      List bugs assigned to the current user. Returns id, summary, status,
+      severity, priority, project, last_updated.
+    parameters:
+      status:
+        type: select
+        options: ["open", "feedback", "acknowledged", "confirmed", "assigned", "resolved", "closed", "all"]
+        default: "open"
+      limit:
+        type: number
+        default: 50
+    page:
+      url: "{base_url}/view_all_bug_page.php"
+      on_target: "/view_all_bug_page.php"
+    script: |
+      const list = document.querySelector('#bug_list');
+      if (!list) {
+        return { bugs: [], total: 0, _error: '#bug_list not found on ' + location.pathname };
+      }
+      const allRows = Array.from(list.querySelectorAll('tr'));
+      const headerRow = allRows.find(r => r.className === 'row-category');
+      const dataRows = allRows.filter(r => r.querySelector('input[name*="bug"]'));
+      const headerIdx = {};
+      if (headerRow) {
+        Array.from(headerRow.querySelectorAll('td, th')).forEach((c, i) => {
+          const name = (c.textContent || '').trim().toLowerCase();
+          if (name) headerIdx[name] = i;
+        });
+      }
+      const col = (cells, header) => {
+        const idx = headerIdx[header.toLowerCase()];
+        if (idx === undefined) return '';
+        return ((cells[idx] && cells[idx].innerText) || '').trim();
+      };
+      const STATUS_RE = /^(new|feedback|acknowledged|confirmed|assigned|open|resolved|closed|reopened)$/i;
+      const FILTERS = {
+        open:   /^(new|feedback|acknowledged|confirmed|assigned|open|reopened)$/i,
+        closed: /^(resolved|closed)$/i,
+        all:    /./,
+      };
+      const all = dataRows.map(r => {
+        const cells = Array.from(r.querySelectorAll('td'));
+        const checkbox = r.querySelector('input[type="checkbox"][name*="bug"]');
+        const link = r.querySelector('a[href*="bug_view_page.php?bug_id="]');
+        const id = checkbox && checkbox.value
+          ? Number(checkbox.value)
+          : Number((link && link.href.match(/bug_id=(\d+)/) || [])[1] || 0);
+        if (!id) return null;
+        // Forge quirk: "Status" column shows handler; real status is in "Resolution"
+        let status = col(cells, 'Status');
+        if (!STATUS_RE.test(status)) {
+          const resolution = col(cells, 'Resolution');
+          if (STATUS_RE.test(resolution)) status = resolution;
+        }
+        return {
+          id,
+          summary: (link && link.getAttribute('title') && link.getAttribute('title').trim()) || col(cells, 'Summary'),
+          status,
+          priority: col(cells, 'Priority'),
+          severity: col(cells, 'Severity') || col(cells, 'S'),
+          keywords: col(cells, 'Keyword') || col(cells, 'Category'),
+          reporter: col(cells, 'Reporter'),
+          assignee: (col(cells, 'Status') || col(cells, 'Handler') || col(cells, 'Assigned To')).replace(/^\(|\)$/g, '').trim(),
+          source: col(cells, 'Source'),
+          product_version: col(cells, 'Reported Version') || col(cells, 'Product Version'),
+          fix_schedule: col(cells, 'Fix Schedule'),
+          last_updated: col(cells, 'Updated') || col(cells, 'Last Updated'),
+          created_at: col(cells, 'Date Submitted') || col(cells, 'Date'),
+          url: (link && link.href) || '',
+        };
+      }).filter(b => b !== null);
+      const key = (args.status || 'all').toLowerCase();
+      const filter = FILTERS[key] || FILTERS.all;
+      const filtered = all.filter(b => filter.test(b.status));
+      return {
+        bugs: filtered.slice(0, args.limit || 50),
+        total: filtered.length,
+        rawCount: all.length,
+        _page: location.href,
+      };
+
+  get_bug:
+    description: |
+      Get full details of a single bug — description, all notes, history.
+    parameters:
+      bug_id:
+        type: number
+        required: true
+    page:
+      url: "{base_url}/bug_view_page.php?bug_id={args.bug_id}"
+      on_target: "/bug_view_page.php"
+    script: |
+      const detailsTitle = Array.from(document.querySelectorAll('.form-title'))
+        .find(e => e.textContent && e.textContent.trim().startsWith('Viewing Bug'));
+      const detailsTable = detailsTitle ? detailsTitle.closest('table') : null;
+      if (!detailsTable) return { _error: 'bug detail table not found on ' + location.pathname };
+      const rows = Array.from(detailsTable.querySelectorAll('tr'));
+      const fields = {};
+      if (rows[1] && rows[2]) {
+        const labelCells = Array.from(rows[1].querySelectorAll('td'));
+        const valueCells = Array.from(rows[2].querySelectorAll('td'));
+        labelCells.forEach((l, i) => {
+          const label = l.innerText.trim();
+          const val = valueCells[i] ? valueCells[i].innerText.trim() : '';
+          if (label && !fields[label]) fields[label] = val;
+        });
+      }
+      Array.from(document.querySelectorAll('td.category')).forEach(label => {
+        const labelText = (label.textContent || '').trim();
+        if (labelText.includes('\n') && labelText.length > 100) return;
+        if (labelText.startsWith('Select File')) return;
+        if (labelText === 'Bugnote') return;
+        const valueEl = label.nextElementSibling;
+        if (!valueEl || valueEl.tagName !== 'TD' || valueEl.classList.contains('category')) return;
+        const val = valueEl.innerText.trim();
+        if (!fields[labelText]) fields[labelText] = val;
+      });
+      const notesTitle = Array.from(document.querySelectorAll('.form-title'))
+        .find(e => e.textContent && e.textContent.trim().startsWith('Bug Notes'));
+      const notesTable = notesTitle && notesTitle.closest('table');
+      const notes = Array.from((notesTable && notesTable.querySelectorAll('tr.bugnote')) || []).map((r, i) => {
+        const cells = r.querySelectorAll('td');
+        const header = (cells[0] && cells[0].innerText.trim()) || '';
+        const body   = (cells[1] && cells[1].innerText.trim()) || '';
+        const m = header.match(/^(.*?)\n(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})/);
+        return {
+          idx: i,
+          author: (m && m[1]) ? m[1].trim() : (header.split('\n')[0] || '').trim(),
+          date:   (m && m[2]) || '',
+          body,
+        };
+      });
+      const historyTitle = Array.from(document.querySelectorAll('.form-title'))
+        .find(e => e.textContent && e.textContent.trim().startsWith('Bug History'));
+      const historyTable = historyTitle && historyTitle.closest('table');
+      const history = Array.from((historyTable && historyTable.querySelectorAll('tr')) || [])
+        .filter(r => r.querySelectorAll('td').length === 4)
+        .slice(1)
+        .map(r => {
+          const c = Array.from(r.querySelectorAll('td')).map(x => x.innerText.trim());
+          return { date: c[0], user: c[1], field: c[2], change: c[3] };
+        });
+      const id = Number(fields.ID || args.bug_id);
+      return {
+        id,
+        summary: fields.Summary || '',
+        description: fields.Description || '',
+        additional_information: fields['Additional Information'] || '',
+        status: fields.Status || '',
+        resolution: fields.Resolution || '',
+        priority: fields.Priority || '',
+        severity: fields.Severity || '',
+        reproducibility: fields.Reproducibility || '',
+        category: fields.Category || '',
+        keywords: fields.Keyword || '',
+        reporter: fields.Reporter || '',
+        assignee: fields['Assigned To'] || '',
+        qa_assignee: fields['QA Assignee'] || '',
+        view_status: fields['View Status'] || '',
+        eta: fields.ETA || '',
+        date_submitted: fields['Date Submitted'] || '',
+        last_update: fields['Last Update'] || '',
+        reported_version: fields['Reported Version'] || '',
+        fix_schedule: fields['Fix Schedule'] || '',
+        source: fields.Source || '',
+        tag: fields.Tag || '',
+        notes,
+        history,
+        url: location.href,
+      };
+
+  search_bugs:
+    description: |
+      Search bugs by text query. Mantis 1.x classic does this via
+      view_all_set.php which sets a temporary filter and redirects to
+      view_all_bug_page.php with results.
+    parameters:
+      query:
+        type: string
+        required: true
+      status:
+        type: string
+        description: "Optional Mantis status keyword: new|feedback|acknowledged|confirmed|assigned|resolved|closed"
+      limit:
+        type: number
+        default: 25
+    page:
+      # No on_target — we always want to re-set the filter on each call.
+      # {args.query} URL-encoded by the extension at call time.
+      url: "{base_url}/view_all_set.php?type=1&temporary=y&search={args.query}"
+    script: |
+      // After view_all_set.php's filter is set, Mantis redirects to
+      // view_all_bug_page.php. The script body is identical to list_my_bugs
+      // but with `status` forced to 'all' so the in-page filter governs.
+      const list = document.querySelector('#bug_list');
+      if (!list) return { bugs: [], total: 0, _error: '#bug_list not found' };
+      const allRows = Array.from(list.querySelectorAll('tr'));
+      const headerRow = allRows.find(r => r.className === 'row-category');
+      const dataRows = allRows.filter(r => r.querySelector('input[name*="bug"]'));
+      const headerIdx = {};
+      if (headerRow) {
+        Array.from(headerRow.querySelectorAll('td, th')).forEach((c, i) => {
+          const name = (c.textContent || '').trim().toLowerCase();
+          if (name) headerIdx[name] = i;
+        });
+      }
+      const col = (cells, header) => {
+        const idx = headerIdx[header.toLowerCase()];
+        if (idx === undefined) return '';
+        return ((cells[idx] && cells[idx].innerText) || '').trim();
+      };
+      const STATUS_RE = /^(new|feedback|acknowledged|confirmed|assigned|open|resolved|closed|reopened)$/i;
+      const all = dataRows.map(r => {
+        const cells = Array.from(r.querySelectorAll('td'));
+        const checkbox = r.querySelector('input[type="checkbox"][name*="bug"]');
+        const link = r.querySelector('a[href*="bug_view_page.php?bug_id="]');
+        const id = checkbox && checkbox.value
+          ? Number(checkbox.value)
+          : Number((link && link.href.match(/bug_id=(\d+)/) || [])[1] || 0);
+        if (!id) return null;
+        let status = col(cells, 'Status');
+        if (!STATUS_RE.test(status)) {
+          const resolution = col(cells, 'Resolution');
+          if (STATUS_RE.test(resolution)) status = resolution;
+        }
+        return {
+          id,
+          summary: (link && link.getAttribute('title') && link.getAttribute('title').trim()) || col(cells, 'Summary'),
+          status,
+          severity: col(cells, 'Severity') || col(cells, 'S'),
+          project: col(cells, 'Project') || col(cells, 'Category'),
+          url: (link && link.href) || '',
+        };
+      }).filter(b => b !== null);
+      const limit = args.limit || 25;
+      return {
+        query: args.query,
+        bugs: all.slice(0, limit),
+        total: all.length,
+      };
+
+  list_projects:
+    description: List projects visible to the current user (from the project_id dropdown on the list page).
+    page:
+      url: "{base_url}/view_all_bug_page.php"
+      on_target: "/view_all_bug_page.php"
+    script: |
+      const sel = document.querySelector('select[name="project_id"]');
+      if (!sel) return { _error: 'project_id select not found on page' };
+      const projects = Array.from(sel.options)
+        .map(o => ({
+          id: Number(o.value),
+          name: (o.text || '').trim(),
+          depth: ((o.text || '').match(/^[\s ]+/) || ['']).length,
+        }))
+        .filter(p => p.id > 0);
+      return { projects, total: projects.length };
+
+  add_comment:
+    description: |
+      Add a note (comment) to a bug. Posts to bugnote_add.php with the
+      session cookie. Destructive — extension prompts the user first.
+    destructive: true
+    parameters:
+      bug_id:
+        type: number
+        required: true
+      text:
+        type: string
+        required: true
+    page:
+      url: "{base_url}/bug_view_page.php?bug_id={args.bug_id}"
+      on_target: "/bug_view_page.php"
+    script: |
+      const body = new URLSearchParams({
+        bug_id: String(args.bug_id),
+        bugnote_text: args.text,
+      });
+      try {
+        const r = await fetch('bugnote_add.php', {
+          method: 'POST',
+          headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
+          body: body.toString(),
+          credentials: 'include',
+          redirect: 'follow',
+        });
+        return {
+          ok: r.ok,
+          status: r.status,
+          finalUrl: r.url,
+          success: r.ok && r.url.indexOf('bug_id=' + args.bug_id) >= 0,
+        };
+      } catch (e) {
+        return { ok: false, error: e.message };
+      }
+```
+
+## Notes on the YAML
+
+- **Block scalar `|`** preserves newlines verbatim. The `script` content is plain JS, no TypeScript syntax, no imports — runs as the body of `new Function('args', script)`.
+- **No closures**: scripts can't reference variables outside `args` and the page's global Web APIs (`document`, `fetch`, etc.).
+- **`async/await` allowed**: the runner uses `new Function('args', script)` which returns a value; if that value is a Promise, the runner awaits it. (See `runner.ts` — the wrapper handles both sync and async return values.) The `add_comment` script uses `await fetch(...)` — works fine.
+- **`{args.foo}` substitution and URL encoding**: the extension encodes args values before substituting into URLs. So `{args.query}` with `q="hello world"` becomes `search=hello%20world`, not `search=hello world`.
+- **Forge-specific column quirks** (Status vs Resolution fallback) are baked into the `list_my_bugs` script. If you have a non-Forge Mantis instance, you may need a separate `mantis-vanilla.yaml`.
+
+## Open questions / nice-to-haves (not blockers)
+
+- **Probe script delivery** — `mantis.probe.js` stays as dev-time tool. If we want auto-recovery from selector drift, expose `GET /api/connectors/<id>/probe` later.
+- **Server-side `script` linting** — Forge could `new Function(scriptStr)` at load time to catch parse errors before the manifest ever ships to the extension. Optional.
+- **Schema versioning** — when we change the manifest schema again in the future, bump a top-level `schema_version` and let the extension reject incompatible older shapes.
+
+## What happens when Forge ships this
+
+1. User refreshes the Connectors tab in the extension.
+2. Extension fetches `/api/connectors`, sees `script` + `page` populated on each Mantis tool.
+3. `buildConnectorTools()` un-filters those tools (they now satisfy the `tool.script` gate).
+4. On the next chat turn, the LLM's tool list includes `mantis.list_my_bugs`, `mantis.get_bug`, `mantis.search_bugs`, `mantis.list_projects`, `mantis.add_comment`.
+5. User asks "list my Mantis bugs" → LLM calls `mantis.list_my_bugs` → runner opens background Mantis tab (or reuses existing) → executes the YAML's `script` body in MAIN world → returns JSON to LLM → LLM summarizes.
+
+No extension code changes required at any point in this flow.