@aion0/forge 0.8.1 → 0.8.3

package/lib/help-docs/17-connectors.md

@@ -1,16 +1,72 @@
 # Connectors
 
-**Connectors** are Forge plugins of `category: connector` that expose tool
-schemas + **the extraction scripts that implement them** to the **Forge
-browser extension**. The extension is a generic runner: it doesn't know
-about Mantis or GitLab specifically — it loads the manifest, finds/opens
-the right tab, and executes whatever script the manifest ships.
-
-> **Architectural principle**: adding a new connector should be a
-> Forge-only change. No browser extension release required.
->
-> See `docs/Connector-DeclarativeExtract-Spec.md` for the full spec of
-> the manifest schema and runtime contract.
+**Connectors** expose external services (Mantis, GitLab, Teams, …) as
+tools that the Forge chat agent and the browser extension can call.
+Each connector is a declarative YAML manifest — schema + the extraction
+scripts that implement each tool — pulled from the remote
+[`forge-connectors`](https://github.com/aiwatching/forge-connectors)
+registry.
+
+> **Architecture**: connectors are an independent subsystem (`lib/connectors/`).
+> They are NOT plugins. Pipeline-node plugins (`lib/plugins/`) and connectors
+> share no schema, no registry, no config file.
+
+There are **no built-in connectors**. A fresh install starts with an empty
+connector list; the user picks what they want from the marketplace.
+
+## Marketplace flow
+
+1. On boot, Forge fetches `registry.json` from `connectorsRepoUrl`
+   (default `https://raw.githubusercontent.com/aiwatching/forge-connectors/main`)
+   and caches it at `<dataDir>/connectors/registry-cache.json`.
+2. The Settings → Connectors panel reads the cache and lists every
+   available connector with its install state:
+   - **Available** — in registry, not installed
+   - **Installed v0.5.0** — manifest on disk, config row present
+   - **Update 0.5 → 0.6** — registry has newer version
+3. Clicking **Install** fetches `<id>/manifest.yaml` from the registry,
+   writes it to `<dataDir>/connectors/<id>/manifest.yaml`, and adds a
+   row to `<dataDir>/connector-configs.json`.
+4. The user fills in settings (host URL, PAT, etc.) via the existing
+   settings UI in the extension or `/api/connectors/<id>/settings`.
+5. Chat tools become available the instant the manifest lands on disk —
+   no Forge restart.
+
+Sync re-runs every hour and on manual **Refresh** in the Settings panel.
+Network failures are silent — the cache from the last successful sync
+keeps the marketplace usable offline.
+
+## Repository layout
+
+The `forge-connectors` repo (forkable via `connectorsRepoUrl`) is:
+
+```
+registry.json
+<id>/
+  manifest.yaml      ← what Forge installs
+  README.md          ← optional, surfaced in the marketplace
+```
+
+`registry.json` lists every connector with id/name/version/icon/description
++ `min_forge_version`. Forge only consults the registry for the install
+flow; once installed, `<dataDir>/connectors/<id>/manifest.yaml` is the
+source of truth for that user.
+
+## Local data layout
+
+```
+<dataDir>/
+├── connectors/
+│   ├── registry-cache.json       last fetch from forge-connectors
+│   ├── mantis/manifest.yaml      installed manifest copy
+│   ├── gitlab/manifest.yaml
+│   └── …
+└── connector-configs.json        { "<id>": { config, installed_version, enabled } }
+```
+
+Secret fields (`type: secret`) in `config` are encrypted at rest with
+AES-256-GCM via the existing key at `<dataDir>/.encrypt-key`. The on-wire
+GET response masks them with bullets so plaintext never leaves the server.
 
 ## Execution model: DOM extraction, not REST
 
@@ -19,72 +75,56 @@ user's tabs and parse rendered HTML via `chrome.scripting.executeScript`.
 This is the whole point of routing through a browser extension — reuse
 the user's already-authenticated UI session, with zero token management.
 
-Why this matters in practice:
-
 | System | REST API needs | Browser DOM needs |
 |---|---|---|
-| MantisBT | Per-user API token (`Authorization` header) | Already-logged-in PHPSESSID cookie ✅ |
-| GitLab | Personal Access Token | Session cookie on `/-/issues/` etc. ✅ |
+| MantisBT | Per-user API token | Already-logged-in PHPSESSID cookie ✅ |
+| GitLab UI | Personal Access Token | Session cookie on `/-/issues/` etc. ✅ |
 | JIRA Server | PAT or basic auth | Session cookie ✅ |
 | Teams web | MSAL bearer token | Session cookie ✅ |
 
-If a connector hit the REST API path, every user would need to mint and
-manage a token per system. That defeats the extension's value
-proposition. **Token-based fallback is opt-in per connector, not the
-default code path.**
-
 This means:
-- **No PAT / API tokens to manage** — connector reuses the user's logged-in browser session.
-- **Data never flows through Forge** — Forge only ships the manifest + extraction script. The LLM in the extension calls the tool, the extension scrapes in the user's tab, the LLM sees the result.
-- Forge's role: **discovery + settings sync + script delivery**.
-- Trade-off: site redesigns break scripts. Bump the manifest `version` and update the `script` block — users get the fix on next refresh, no extension release.
 
-## Architecture
+- **No PAT / API tokens to manage** for browser connectors — they reuse
+  the user's logged-in session.
+- **Data never flows through Forge** — Forge only ships the manifest +
+  extraction script. The LLM in the chat backend calls the tool, the
+  extension scrapes in the user's tab, the LLM sees the result.
+- Trade-off: site redesigns break scripts. Bump the manifest `version`
+  in the registry and users pull the fix on next sync — no Forge or
+  extension release needed.
 
-```
-Forge                              Extension                    User's tabs
-─────                              ─────────                    ───────────
-mantis.yaml                                                     ┌──────────────┐
-  tools:                           Generic runner               │ Mantis tab   │
-    list_my_bugs:        ─────────▶  ① fetch manifest           │ (logged in)  │
-      page: { url, ... }              ② render templates        │              │
-      script: |                       ③ acquire tab matching    │ #bug_list    │
-        const list = ...                 host_match, navigate   │  ◀── runs    │
-        return { bugs, ... }            to page.url                 the script
-                                       ④ executeScript(script)      from the
-                                          in the tab                manifest
-                                       ⑤ return JSON to LLM     │              │
-                                                                 └──────────────┘
-```
-
-The extension has **no per-connector code**. Adding GitLab means dropping
-a `gitlab.yaml` next to `mantis.yaml` — extension auto-discovers it.
-
-## Plugin manifest format
-
-A connector plugin is a YAML in `lib/builtin-plugins/<id>.yaml` (or
-`~/.forge/plugins/<id>/plugin.yaml`):
+## Manifest format
 
 ```yaml
 id: mantis
 name: MantisBT
 icon: "🐞"
-version: "0.2.0"
-category: connector              # ← marks this as a connector
-mode: browser-side               # server-side | browser-side | hybrid
+version: "0.5.0"
+author: forge
+description: |
+  Read + comment on MantisBT bugs from the user's logged-in browser session.
 
-# Per-user settings (rendered as a form in the extension)
+min_forge_version: "0.8.0"
+
+# Default extension execution context.
+#   main     — permissive-CSP sites (Mantis, GitLab self-hosted)
+#   isolated — strict-CSP sites that block eval (Teams, github.com)
+runner: main
+
+# Per-user settings (rendered as a form in the extension).
 settings:
   base_url:
     type: string
     label: Mantis base URL
     required: true
+  token:
+    type: secret             # encrypted at rest (AES-256-GCM)
+    label: API token (optional fallback)
 
-# Plugin-level: where the extension finds / opens an authenticated tab.
-# Chrome match pattern; {settings.*} expanded at API response time.
+# Chrome match pattern; {settings.*} expanded server-side.
 host_match: "{base_url}/*"
 
-# Substring detected after navigation → tells the runner "user not logged in".
+# Substring detected after navigation → "user not logged in".
 login_redirect: "/login_page.php"
 
 tools:
@@ -94,22 +134,19 @@ tools:
       status: { type: select, options: ["open", "closed", "all"], default: "open" }
       limit:  { type: number, default: 50 }
 
-    # Page to navigate to (or stay on, if on_target matches current URL).
+    # Default protocol is 'browser' — runs in the user's tab.
     page:
       url: "{base_url}/view_all_bug_page.php"
       on_target: "/view_all_bug_page.php"
-
-    # Function BODY. Implicit `args` parameter. Returns JSON-serializable.
-    # Runs IN THE USER'S TAB (page context). No closures over Forge / extension.
     script: |
       const { status, limit } = args;
       const list = document.querySelector('#bug_list');
       if (!list) return { bugs: [], total: 0, _error: '#bug_list not found' };
-      // ... extract rows, filter, return
+      // …extract rows, filter, return
       return { bugs, total };
 
   add_comment:
-    destructive: true            # ← extension prompts user before running
+    destructive: true        # extension prompts before running
     description: "Add a note to a bug."
     parameters:
       bug_id: { type: number, required: true }
@@ -118,78 +155,104 @@ tools:
       url: "{base_url}/bug_view_page.php?bug_id={args.bug_id}"
     script: |
       // Form-submit or fetch() with same-origin cookies
-      ...
+      …
 ```
 
 **Template variables**:
 - `{base_url}` / `{settings.<name>}` → expanded server-side from saved settings
-- `{args.<name>}` → expanded by the extension at run time from the LLM's tool input
+- `{args.<name>}` → expanded by the runtime from the LLM's tool input
 
-**Script contract**:
+**`script` contract**:
 - Receives `args` (the LLM's parameters)
 - Returns a JSON-serializable value (no DOM nodes, no functions)
 - Has access to `document`, `fetch`, `URL`, etc. (page context)
-- Can call same-origin `fetch()` — cookies auto-attached
-- Must be self-contained — no closures over the manifest's surroundings
-- Errors thrown are caught by the runner and returned as tool errors
+- Same-origin `fetch()` — cookies auto-attached
+- Self-contained — no closures over the manifest's surroundings
+- Uncaught throws become tool errors
 
-See `docs/Connector-DeclarativeExtract-Spec.md` for the complete contract.
+## Server-side protocols (`http`, `shell`)
 
-### 1 plugin = 1 connector (default)
+When a service has a clean REST API and you don't need a browser tab at
+all, declare `protocol: http` on the tool. Forge issues the request
+server-side and returns the body to the LLM. Same for `protocol: shell`
+— Forge spawns a process with an explicit arg array (no `shell:true`,
+so templated values cannot inject metacharacters).
 
-The above shape is the **default 1:1 case**. For same-vendor suites with
-shared auth (Atlassian, Google Workspace, M365), use the `connectors[]`
-escape hatch — one plugin declares multiple connector entries that share
-the user's OAuth/SSO:
+```yaml
+tools:
+  get_repo:
+    protocol: http
+    parameters:
+      repo: { type: string, required: true }
+    request:
+      method: GET
+      url: 'https://api.github.com/repos/{args.repo}'
+      headers:
+        Accept: 'application/vnd.github+json'
+        Authorization: 'Bearer {settings.token}'
+
+  git_log:
+    protocol: shell
+    parameters:
+      repo: { type: string, required: true }
+      n: { type: number }
+    command: ['git', '-C', '{args.repo}', 'log', '-n', '{args.n}', '--oneline']
+    timeout_ms: 5000
+```
+
+Response body / stdout is truncated to ~8 KB for the LLM context;
+default timeout 30 s, max 5 min. `protocol: browser` (the default) runs
+in the extension; `http` and `shell` run entirely on Forge (chat-standalone,
+port 8408).
+
+**Safety:** `protocol: shell` lets a manifest invoke any binary on PATH.
+Always read the script before installing.
+
+## 1:1 vs 1:N suites
+
+The default is **one connector = one tool surface** (Mantis, GitLab,
+Teams, …). For same-vendor suites with shared auth (Atlassian, Google
+Workspace, M365), a manifest can declare multiple `connectors[]`
+entries that share user settings:
 
 ```yaml
 id: atlassian-suite
-category: connector
-mode: hybrid
 connectors:
   - id: jira
     host_match: "*://*.atlassian.net/*"
-    tools: { ... }
+    tools: { … }
   - id: confluence
     host_match: "*://*.atlassian.net/wiki/*"
-    tools: { ... }
+    tools: { … }
 ```
 
-Most plugins should stay 1:1. Use 1:N only for genuine shared-auth suites.
+Most manifests should stay 1:1. Use 1:N only for genuine shared-auth suites.
 
 ## HTTP API
 
-All endpoints require `X-Forge-Token` header (obtain via `POST /api/auth/verify`).
+All endpoints require `X-Forge-Token` (obtain via `POST /api/auth/verify`).
 
 ### `GET /api/connectors`
-List connector plugins (installed + available). For installed connectors,
-`page.url` / `host_match` etc. have `{base_url}` / `{settings.*}` expanded
-from the user's saved settings; `{args.*}` is left literal.
+List installed connectors. `page.url` / `host_match` etc. are expanded
+from the user's settings; `{args.*}` stays literal for the runtime.
 
 ```json
 {
   "connectors": [
     {
-      "plugin_id": "mantis",
+      "plugin_id": "mantis",         /* legacy wire-name; equals id */
       "name": "MantisBT",
       "icon": "🐞",
-      "version": "0.2.0",
-      "mode": "browser-side",
+      "version": "0.5.0",
       "installed": true,
       "host_match": "https://mantis.acme.com/*",
       "login_redirect": "/login_page.php",
+      "runner": "main",
       "entries": [
         {
           "id": "mantis",
-          "tools": {
-            "list_my_bugs": {
-              "description": "...",
-              "parameters": {...},
-              "page": { "url": "https://mantis.acme.com/view_all_bug_page.php", "on_target": "/view_all_bug_page.php" },
-              "script": "const { status, limit } = args; ..."
-            }
-          },
-          "settings": { "base_url": {...} }
+          "tools": { … },
+          "settings": { "base_url": { … } }
         }
       ]
     }
@@ -197,126 +260,115 @@ from the user's saved settings; `{args.*}` is left literal.
 }
 ```
 
-Query params:
-- `installed=true` → only plugins the user has configured
-- `id=<plugin_id>` → single connector detail (alternative to a path param)
-
-### `GET /api/connectors?id=<id>` / `GET /api/connectors/<id>`
-Single connector detail. Same shape as a list entry, wrapped in `{ connector: ... }`.
+### `GET /api/connectors?id=<id>`
+Single connector detail, wrapped in `{ connector: … }`.
 
 ### `GET /api/connectors/<id>/settings`
-Read the user's saved settings for a connector.
+Read the user's saved settings. Secret fields come back as bullets.
 
 ```json
 {
-  "settings": { "base_url": "https://mantis.acme.com", "default_project": "Web" },
-  "schema": { "base_url": {...}, "default_project": {...} },
+  "settings": { "base_url": "https://mantis.acme.com", "token": "••••••••" },
+  "schema":   { "base_url": { … },                       "token": { "type": "secret", … } },
   "installed": true
 }
 ```
 
-`schema` is the merged settings schema across all entries (1:N). `settings`
-includes any defaults declared in the schema, overlaid with the user's
-saved values.
-
 ### `POST /api/connectors/<id>/settings`
-Save settings. Body: `{ settings: { base_url: "..." } }` (or the object directly).
+Save settings. If the client sends `••••••••` for a secret, the stored
+value is preserved (so editing one field doesn't wipe the PAT).
 
-First save = creates the install record. Subsequent saves update it.
+### `GET /api/connectors/marketplace`
+List registry entries merged with installed state.
 
 ```json
-{ "ok": true, "settings": { "base_url": "..." } }
+{
+  "fetched_at": "2026-05-19T16:31:02.000Z",
+  "base_url":   "https://raw.githubusercontent.com/aiwatching/forge-connectors/main",
+  "entries": [
+    { "id": "mantis", "version": "0.5.0", "installed_version": "0.5.0", "update_available": false, … },
+    { "id": "gitlab", "version": "1.1.0", "installed_version": "1.0.0", "update_available": true,  … },
+    { "id": "teams",  "version": "0.8.1",                                /* available */            … }
+  ]
+}
 ```
 
-## Built-in connectors
+### `POST /api/connectors/marketplace`
+Marketplace actions.
 
-| ID | Description |
+| `action` | Effect |
 |---|---|
-| `mantis` | MantisBT — list/get/search bugs, add comments. Self-hosted. |
-| `gitlab-browser` | GitLab — issues, MRs, pipelines via browser session. gitlab.com + self-hosted. |
-
-The CLI-backed `glab` GitLab integration in `gitlab-issue-fix-and-review`
-is **separate** — that runs server-side as a pipeline. The browser
-connector is for the extension's interactive use.
-
-## Writing a new connector
-
-1. Drop a YAML in `~/.forge/plugins/<id>/plugin.yaml` with `category: connector`.
-2. Define `settings` (e.g. `base_url`) the user must provide.
-3. Define `host_match` and `login_redirect` at the plugin level.
-4. For each tool: schema (`parameters`), `page` block, `script` body.
-5. **Use DOM extraction**, not REST API calls. `script` runs in the tab,
-   has same-origin cookies via `fetch()`, can read the DOM directly.
-6. Use `mantis.probe.js`-style helper scripts at dev time to discover
-   stable selectors; paste the resulting selectors into your `script`.
-7. Restart Forge — extension picks it up via `GET /api/connectors` on
-   next refresh. **No extension code changes needed.**
-
-### When REST is unavoidable
-
-Some sites are pure SPAs that virtualize lists, render via canvas, or
-require many interactions to expose data. In those rare cases, add an
-optional `api_token` setting (`type: secret`) and call `fetch()` with
-the token from within `script`. Default code path stays DOM; token is
-the fallback the user explicitly accepts.
-
-## Server-side protocols (http, shell)
-
-When a site has a clean REST API and you don't need a browser tab at
-all, declare `protocol: http` on the tool. Forge issues the request
-server-side and returns the body to the LLM. Same for `protocol:
-shell` — Forge spawns a process with an explicit arg array (no
-`shell:true`, so templated values cannot inject metacharacters).
-
-```yaml
-# lib/builtin-plugins/github-api.yaml — see file for full example
-tools:
-  get_repo:
-    protocol: http
-    parameters:
-      repo: { type: string, required: true }
-    request:
-      method: GET
-      url: 'https://api.github.com/repos/{args.repo}'
-      headers:
-        Accept: 'application/vnd.github+json'
-        Authorization: 'Bearer {settings.token}'
-
-  git_log:
-    protocol: shell
-    parameters:
-      repo: { type: string, required: true }
-      n: { type: number }
-    command: ['git', '-C', '{args.repo}', 'log', '-n', '{args.n}', '--oneline']
-    timeout_ms: 5000
+| `sync` | Re-fetch `registry.json` (and refresh installed manifests with `refreshInstalled: true`) |
+| `install` | Fetch the manifest for `id` and write it to `<dataDir>/connectors/<id>/` |
+| `update` | Alias for `install` — pulls the latest manifest |
+| `uninstall` | Drop the manifest. `keepConfig: true` (default) preserves the user's settings + PAT |
+| `status` | Probe a single `id`: returns `{ installed, installed_version, enabled }` |
+
+## Authoring a new connector
+
+Three ways to land a custom connector:
+
+1. **Local-only (fastest)** — write a `manifest.yaml`, upload it via
+   the "+ Upload" button in Settings → Connectors (or drag-drop). For
+   multi-file connectors, zip it with `manifest.yaml` at the root.
+   See `21-build-connector.md` for the manifest schema and an
+   interview-style template.
+2. **Forge Help AI** — ask Forge chat "build me a connector for X"; the
+   AI walks through requirements, writes the manifest to
+   `<dataDir>/connectors/<id>/manifest.yaml`, and registers it via
+   `POST /api/connectors/install-local`.
+3. **Share via forge-connectors** — fork
+   [`forge-connectors`](https://github.com/aiwatching/forge-connectors),
+   add `<id>/manifest.yaml`, add an entry to `registry.json`, open a PR.
+   Maintainers review the script bodies before merge.
+
+Iterating on a private registry: point `settings.connectorsRepoUrl` at
+a private fork.
+
+### POST /api/connectors/install-local
+
+Used by both the Upload button and the Help AI:
+
+```http
+POST /api/connectors/install-local
+Content-Type: application/json
+X-Forge-Token: …
+
+{ "yaml": "id: my-jira\nname: MyJira\nversion: \"0.1.0\"\n..." }
 ```
 
-Templates `{base_url}`, `{settings.*}`, `{args.*}` all expand at
-dispatch time. For `http`, body can be a string or a JSON object; for
-`shell`, every arg is templated independently so an arg with spaces or
-quotes stays a single literal arg.
+or `multipart/form-data` with a `file` field accepting `.yaml`,
+`.yml`, or `.zip`. Zip must have `manifest.yaml` at the root.
 
-Response body / stdout is truncated to ~8 KB for the LLM context;
-default timeout 30 s, max 5 min. `protocol: browser` (the default)
-keeps using the extension runner; `http` and `shell` run entirely on
-Forge (chat-standalone, port 8408).
+## Migration from pre-v0.9
 
-**Safety:** `protocol: shell` lets a YAML pick any binary on PATH —
-review at install time. There is no auto allow-list in v1.
+Pre-v0.9 Forge stored connectors as built-in plugins under
+`lib/builtin-plugins/<id>.yaml` with `category: connector`. On upgrade:
 
-### Iterating selectors
+1. `migrateConnectorConfigs()` runs once per boot. For each known id
+   (mantis, gitlab, teams, pmdb, github-api), if there's a row in
+   `plugin-configs.json`, the encrypted config blob is moved to
+   `connector-configs.json` and the old row is removed.
+2. The next `syncRegistry({ refreshInstalled: true })` pulls the
+   matching manifest from `forge-connectors` and writes it to
+   `<dataDir>/connectors/<id>/manifest.yaml`.
+3. Chat tools come back online ~1 s after migration completes. PAT
+   and base URL are preserved across the upgrade.
 
-If a connector breaks after a site redesign, **only the YAML's `script`
-body needs to change**. Bump `version`, edit, restart Forge — extension
-users pick up the fix on their next refresh.
+If the user is offline at upgrade time, the marketplace will be empty
+until the network returns; config rows remain, so the next sync
+restores everything without re-entering credentials.
 
 ## Troubleshooting
 
 | Symptom | Cause + fix |
 |---|---|
-| Extension shows "no connectors" | Token expired — call `POST /api/auth/verify` to refresh |
-| Tool not appearing for LLM | Plugin loaded but `category` missing, OR `script` missing for the tool |
-| Settings POST returns 500 | Plugin id not found (typo?) or `plugin-configs.json` not writable |
-| Tool returns "login required" | `login_redirect` substring matched the tab URL after nav — user needs to log in to that site |
-| Script throws but error opaque | Wrap script body in try/catch and `return { _error: e.message }`; the runner catches uncaught throws but caller-visible message is cleaner |
-| Strict-CSP site (github.com etc.) refuses `new Function` | Page CSP blocks dynamic eval. For those sites, write a server-side connector (REST + token) instead of browser-side. Document the limitation per connector. |
+| Marketplace is empty | Initial sync failed (offline / bad URL). Click **Refresh** in Settings → Connectors. |
+| `Sync error: … self-signed certificate in certificate chain` | Corporate TLS-interception proxy (Zscaler, Palo Alto, etc.). Node doesn't trust the proxy's root CA by default. Export `NODE_EXTRA_CA_CERTS=/path/to/corporate-ca.pem` before starting Forge, or run `forge server restart` with that env var set. macOS users can extract the bundle from Keychain Access → System Roots. |
+| `Sync error: … ENOTFOUND` | DNS can't resolve `raw.githubusercontent.com` — VPN / firewall blocking. Point `connectorsRepoUrl` at a private mirror, or sync manifests manually into `<dataDir>/connectors/<id>/manifest.yaml`. |
+| Installed connector missing from chat | Manifest deleted but config row remains. Re-install from the marketplace. |
+| "Update available" badge stuck | Network failure on the refresh. Re-sync via **Refresh**. |
+| Tool not appearing for the LLM | Connector installed but disabled. Toggle it on in the Settings panel. |
+| Tool returns "login required" | `login_redirect` substring matched the tab URL after nav — user needs to log in to that site. |
+| Strict-CSP site refuses `new Function` | Page CSP blocks dynamic eval. Set `runner: isolated` in the manifest, or rewrite the tool as `protocol: http`. |
+| `POST /api/connectors/<id>/settings` returns 404 | Manifest not on disk (`<dataDir>/connectors/<id>/manifest.yaml`). Install the connector first. |