@aion0/forge 0.6.3 → 0.8.0

package/lib/help-docs/17-connectors.md

@@ -0,0 +1,322 @@
+# Connectors
+
+**Connectors** are Forge plugins of `category: connector` that expose tool
+schemas + **the extraction scripts that implement them** to the **Forge
+browser extension**. The extension is a generic runner: it doesn't know
+about Mantis or GitLab specifically — it loads the manifest, finds/opens
+the right tab, and executes whatever script the manifest ships.
+
+> **Architectural principle**: adding a new connector should be a
+> Forge-only change. No browser extension release required.
+>
+> See `docs/Connector-DeclarativeExtract-Spec.md` for the full spec of
+> the manifest schema and runtime contract.
+
+## Execution model: DOM extraction, not REST
+
+**Browser-side connectors do NOT call REST APIs.** They navigate the
+user's tabs and parse rendered HTML via `chrome.scripting.executeScript`.
+This is the whole point of routing through a browser extension — reuse
+the user's already-authenticated UI session, with zero token management.
+
+Why this matters in practice:
+
+| System | REST API needs | Browser DOM needs |
+|---|---|---|
+| MantisBT | Per-user API token (`Authorization` header) | Already-logged-in PHPSESSID cookie ✅ |
+| GitLab | Personal Access Token | Session cookie on `/-/issues/` etc. ✅ |
+| JIRA Server | PAT or basic auth | Session cookie ✅ |
+| Teams web | MSAL bearer token | Session cookie ✅ |
+
+If a connector hit the REST API path, every user would need to mint and
+manage a token per system. That defeats the extension's value
+proposition. **Token-based fallback is opt-in per connector, not the
+default code path.**
+
+This means:
+- **No PAT / API tokens to manage** — connector reuses the user's logged-in browser session.
+- **Data never flows through Forge** — Forge only ships the manifest + extraction script. The LLM in the extension calls the tool, the extension scrapes in the user's tab, the LLM sees the result.
+- Forge's role: **discovery + settings sync + script delivery**.
+- Trade-off: site redesigns break scripts. Bump the manifest `version` and update the `script` block — users get the fix on next refresh, no extension release.
+
+## Architecture
+
+```
+Forge                              Extension                    User's tabs
+─────                              ─────────                    ───────────
+mantis.yaml                                                     ┌──────────────┐
+  tools:                           Generic runner               │ Mantis tab   │
+    list_my_bugs:        ─────────▶  ① fetch manifest           │ (logged in)  │
+      page: { url, ... }              ② render templates        │              │
+      script: |                       ③ acquire tab matching    │ #bug_list    │
+        const list = ...                 host_match, navigate   │  ◀── runs    │
+        return { bugs, ... }            to page.url                 the script
+                                       ④ executeScript(script)      from the
+                                          in the tab                manifest
+                                       ⑤ return JSON to LLM     │              │
+                                                                 └──────────────┘
+```
+
+The extension has **no per-connector code**. Adding GitLab means dropping
+a `gitlab.yaml` next to `mantis.yaml` — extension auto-discovers it.
+
+## Plugin manifest format
+
+A connector plugin is a YAML in `lib/builtin-plugins/<id>.yaml` (or
+`~/.forge/plugins/<id>/plugin.yaml`):
+
+```yaml
+id: mantis
+name: MantisBT
+icon: "🐞"
+version: "0.2.0"
+category: connector              # ← marks this as a connector
+mode: browser-side               # server-side | browser-side | hybrid
+
+# Per-user settings (rendered as a form in the extension)
+settings:
+  base_url:
+    type: string
+    label: Mantis base URL
+    required: true
+
+# Plugin-level: where the extension finds / opens an authenticated tab.
+# Chrome match pattern; {settings.*} expanded at API response time.
+host_match: "{base_url}/*"
+
+# Substring detected after navigation → tells the runner "user not logged in".
+login_redirect: "/login_page.php"
+
+tools:
+  list_my_bugs:
+    description: "List bugs assigned to me."
+    parameters:
+      status: { type: select, options: ["open", "closed", "all"], default: "open" }
+      limit:  { type: number, default: 50 }
+
+    # Page to navigate to (or stay on, if on_target matches current URL).
+    page:
+      url: "{base_url}/view_all_bug_page.php"
+      on_target: "/view_all_bug_page.php"
+
+    # Function BODY. Implicit `args` parameter. Returns JSON-serializable.
+    # Runs IN THE USER'S TAB (page context). No closures over Forge / extension.
+    script: |
+      const { status, limit } = args;
+      const list = document.querySelector('#bug_list');
+      if (!list) return { bugs: [], total: 0, _error: '#bug_list not found' };
+      // ... extract rows, filter, return
+      return { bugs, total };
+
+  add_comment:
+    destructive: true            # ← extension prompts user before running
+    description: "Add a note to a bug."
+    parameters:
+      bug_id: { type: number, required: true }
+      text:   { type: string, required: true }
+    page:
+      url: "{base_url}/bug_view_page.php?bug_id={args.bug_id}"
+    script: |
+      // Form-submit or fetch() with same-origin cookies
+      ...
+```
+
+**Template variables**:
+- `{base_url}` / `{settings.<name>}` → expanded server-side from saved settings
+- `{args.<name>}` → expanded by the extension at run time from the LLM's tool input
+
+**Script contract**:
+- Receives `args` (the LLM's parameters)
+- Returns a JSON-serializable value (no DOM nodes, no functions)
+- Has access to `document`, `fetch`, `URL`, etc. (page context)
+- Can call same-origin `fetch()` — cookies auto-attached
+- Must be self-contained — no closures over the manifest's surroundings
+- Errors thrown are caught by the runner and returned as tool errors
+
+See `docs/Connector-DeclarativeExtract-Spec.md` for the complete contract.
+
+### 1 plugin = 1 connector (default)
+
+The above shape is the **default 1:1 case**. For same-vendor suites with
+shared auth (Atlassian, Google Workspace, M365), use the `connectors[]`
+escape hatch — one plugin declares multiple connector entries that share
+the user's OAuth/SSO:
+
+```yaml
+id: atlassian-suite
+category: connector
+mode: hybrid
+connectors:
+  - id: jira
+    host_match: "*://*.atlassian.net/*"
+    tools: { ... }
+  - id: confluence
+    host_match: "*://*.atlassian.net/wiki/*"
+    tools: { ... }
+```
+
+Most plugins should stay 1:1. Use 1:N only for genuine shared-auth suites.
+
+## HTTP API
+
+All endpoints require `X-Forge-Token` header (obtain via `POST /api/auth/verify`).
+
+### `GET /api/connectors`
+List connector plugins (installed + available). For installed connectors,
+`page.url` / `host_match` etc. have `{base_url}` / `{settings.*}` expanded
+from the user's saved settings; `{args.*}` is left literal.
+
+```json
+{
+  "connectors": [
+    {
+      "plugin_id": "mantis",
+      "name": "MantisBT",
+      "icon": "🐞",
+      "version": "0.2.0",
+      "mode": "browser-side",
+      "installed": true,
+      "host_match": "https://mantis.acme.com/*",
+      "login_redirect": "/login_page.php",
+      "entries": [
+        {
+          "id": "mantis",
+          "tools": {
+            "list_my_bugs": {
+              "description": "...",
+              "parameters": {...},
+              "page": { "url": "https://mantis.acme.com/view_all_bug_page.php", "on_target": "/view_all_bug_page.php" },
+              "script": "const { status, limit } = args; ..."
+            }
+          },
+          "settings": { "base_url": {...} }
+        }
+      ]
+    }
+  ]
+}
+```
+
+Query params:
+- `installed=true` → only plugins the user has configured
+- `id=<plugin_id>` → single connector detail (alternative to a path param)
+
+### `GET /api/connectors?id=<id>` / `GET /api/connectors/<id>`
+Single connector detail. Same shape as a list entry, wrapped in `{ connector: ... }`.
+
+### `GET /api/connectors/<id>/settings`
+Read the user's saved settings for a connector.
+
+```json
+{
+  "settings": { "base_url": "https://mantis.acme.com", "default_project": "Web" },
+  "schema": { "base_url": {...}, "default_project": {...} },
+  "installed": true
+}
+```
+
+`schema` is the merged settings schema across all entries (1:N). `settings`
+includes any defaults declared in the schema, overlaid with the user's
+saved values.
+
+### `POST /api/connectors/<id>/settings`
+Save settings. Body: `{ settings: { base_url: "..." } }` (or the object directly).
+
+First save = creates the install record. Subsequent saves update it.
+
+```json
+{ "ok": true, "settings": { "base_url": "..." } }
+```
+
+## Built-in connectors
+
+| ID | Description |
+|---|---|
+| `mantis` | MantisBT — list/get/search bugs, add comments. Self-hosted. |
+| `gitlab-browser` | GitLab — issues, MRs, pipelines via browser session. gitlab.com + self-hosted. |
+
+The CLI-backed `glab` GitLab integration in `gitlab-issue-fix-and-review`
+is **separate** — that runs server-side as a pipeline. The browser
+connector is for the extension's interactive use.
+
+## Writing a new connector
+
+1. Drop a YAML in `~/.forge/plugins/<id>/plugin.yaml` with `category: connector`.
+2. Define `settings` (e.g. `base_url`) the user must provide.
+3. Define `host_match` and `login_redirect` at the plugin level.
+4. For each tool: schema (`parameters`), `page` block, `script` body.
+5. **Use DOM extraction**, not REST API calls. `script` runs in the tab,
+   has same-origin cookies via `fetch()`, can read the DOM directly.
+6. Use `mantis.probe.js`-style helper scripts at dev time to discover
+   stable selectors; paste the resulting selectors into your `script`.
+7. Restart Forge — extension picks it up via `GET /api/connectors` on
+   next refresh. **No extension code changes needed.**
+
+### When REST is unavoidable
+
+Some sites are pure SPAs that virtualize lists, render via canvas, or
+require many interactions to expose data. In those rare cases, add an
+optional `api_token` setting (`type: secret`) and call `fetch()` with
+the token from within `script`. Default code path stays DOM; token is
+the fallback the user explicitly accepts.
+
+## Server-side protocols (http, shell)
+
+When a site has a clean REST API and you don't need a browser tab at
+all, declare `protocol: http` on the tool. Forge issues the request
+server-side and returns the body to the LLM. Same for `protocol:
+shell` — Forge spawns a process with an explicit arg array (no
+`shell:true`, so templated values cannot inject metacharacters).
+
+```yaml
+# lib/builtin-plugins/github-api.yaml — see file for full example
+tools:
+  get_repo:
+    protocol: http
+    parameters:
+      repo: { type: string, required: true }
+    request:
+      method: GET
+      url: 'https://api.github.com/repos/{args.repo}'
+      headers:
+        Accept: 'application/vnd.github+json'
+        Authorization: 'Bearer {settings.token}'
+
+  git_log:
+    protocol: shell
+    parameters:
+      repo: { type: string, required: true }
+      n: { type: number }
+    command: ['git', '-C', '{args.repo}', 'log', '-n', '{args.n}', '--oneline']
+    timeout_ms: 5000
+```
+
+Templates `{base_url}`, `{settings.*}`, `{args.*}` all expand at
+dispatch time. For `http`, body can be a string or a JSON object; for
+`shell`, every arg is templated independently so an arg with spaces or
+quotes stays a single literal arg.
+
+Response body / stdout is truncated to ~8 KB for the LLM context;
+default timeout 30 s, max 5 min. `protocol: browser` (the default)
+keeps using the extension runner; `http` and `shell` run entirely on
+Forge (chat-standalone, port 8408).
+
+**Safety:** `protocol: shell` lets a YAML pick any binary on PATH —
+review at install time. There is no auto allow-list in v1.
+
+### Iterating selectors
+
+If a connector breaks after a site redesign, **only the YAML's `script`
+body needs to change**. Bump `version`, edit, restart Forge — extension
+users pick up the fix on their next refresh.
+
+## Troubleshooting
+
+| Symptom | Cause + fix |
+|---|---|
+| Extension shows "no connectors" | Token expired — call `POST /api/auth/verify` to refresh |
+| Tool not appearing for LLM | Plugin loaded but `category` missing, OR `script` missing for the tool |
+| Settings POST returns 500 | Plugin id not found (typo?) or `plugin-configs.json` not writable |
+| Tool returns "login required" | `login_redirect` substring matched the tab URL after nav — user needs to log in to that site |
+| Script throws but error opaque | Wrap script body in try/catch and `return { _error: e.message }`; the runner catches uncaught throws but caller-visible message is cleaner |
+| Strict-CSP site (github.com etc.) refuses `new Function` | Page CSP blocks dynamic eval. For those sites, write a server-side connector (REST + token) instead of browser-side. Document the limitation per connector. |

package/lib/help-docs/18-chrome-mcp.md

@@ -0,0 +1,134 @@
+# Chrome MCP Setup
+
+Connect Forge's Claude Code sessions to a running Chrome instance via
+**chrome-devtools-mcp**, so the agent can navigate pages, query the
+DOM, and execute scripts against the user's real browser session.
+
+This is the development-time path for authoring browser connectors —
+the agent writes a connector handler, runs it against a real Mantis /
+GitLab / etc. page, sees the actual DOM output, and iterates.
+
+## How it fits
+
+```
+Forge Claude Code session
+        │
+        │  MCP (stdio)
+        ▼
+chrome-devtools-mcp  ──CDP──▶  Chrome (user's real instance)
+                                  │
+                                  ▼
+                               Mantis / GitLab / any logged-in tab
+```
+
+## One-time setup
+
+### 1. Start Chrome with remote debugging enabled
+
+Quit all Chrome windows first (so the new instance can take the debug
+port).
+
+**macOS:**
+```bash
+/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \
+  --remote-debugging-port=9222 \
+  --user-data-dir="$HOME/Library/Application Support/Google/Chrome"
+```
+
+The `--user-data-dir` argument keeps your normal profile (and all your
+logins) — drop it only if you want a fresh profile.
+
+**Linux:**
+```bash
+google-chrome --remote-debugging-port=9222 --user-data-dir="$HOME/.config/google-chrome"
+```
+
+**Windows:**
+```powershell
+"C:\Program Files\Google\Chrome\Application\chrome.exe" `
+  --remote-debugging-port=9222 `
+  --user-data-dir="$env:LOCALAPPDATA\Google\Chrome\User Data"
+```
+
+Verify it worked: open `http://localhost:9222/json/version` in the new
+Chrome — you should see a JSON page with `Browser`, `Protocol-Version`, etc.
+
+### 2. Add the MCP server to Forge settings
+
+Settings → MCP Servers → Add. Fill in:
+
+| Field | Value |
+|---|---|
+| Name | `chrome` |
+| Type | `stdio` |
+| Command | `npx` |
+| Args | `["-y", "chrome-devtools-mcp"]` |
+| Env | (leave empty — defaults to localhost:9222) |
+
+Save. Forge will now merge this into every project's `.mcp.json` next
+time a session is created or refreshed.
+
+Equivalent `settings.yaml` snippet:
+
+```yaml
+mcpServers:
+  chrome:
+    type: stdio
+    command: npx
+    args:
+      - "-y"
+      - "chrome-devtools-mcp"
+```
+
+### 3. Open a Forge project / workspace
+
+When Claude Code launches in the project, it'll auto-discover the new
+MCP server (the `.mcp.json` Forge writes now has both `forge` and
+`chrome` entries). The agent will see browser tools like
+`chrome_navigate`, `chrome_query_selector`, `chrome_evaluate`, etc.
+
+Verify in a session: ask the agent "what MCP tools do you have?" — it
+should list chrome_* tools alongside Forge tools.
+
+## Using it for connector development
+
+Workflow inside a Forge workspace:
+
+1. Ask the agent: *"Develop a Mantis list_my_bugs connector. Target
+   page is https://mantis.forge.com/view_all_bug_page.php."*
+2. Agent navigates the Chrome tab there via `chrome_navigate`.
+3. Agent queries the DOM via `chrome_query_selector` /
+   `chrome_evaluate` to understand the page structure.
+4. Agent writes the handler code (`lib/builtin-plugins/mantis.handler.js`
+   or wherever your code-publishing scheme lands).
+5. Agent runs the candidate code in the page via `chrome_evaluate`,
+   sees real output, iterates.
+6. When done, agent commits the connector via normal git flow / publishes
+   per the connector marketplace flow.
+
+No copy-paste loop. No probe scripts. Real DOM at every step.
+
+## Cleanup
+
+To stop using Chrome MCP:
+
+1. Remove the `chrome` entry from Settings → MCP Servers.
+2. Forge regenerates `.mcp.json` on the next session creation.
+3. Optionally close the debug-enabled Chrome and restart normally.
+
+## Troubleshooting
+
+| Symptom | Cause + fix |
+|---|---|
+| Agent says it has no chrome_* tools | Forge hasn't written the new `.mcp.json` yet — open the project tab in Forge to trigger `ensureMcpConfig`. Or just delete the project's `.mcp.json` and reopen. |
+| `chrome_navigate` errors with "no debug target" | Chrome isn't running with `--remote-debugging-port=9222`. Verify `curl localhost:9222/json/version`. |
+| Multiple Chrome processes | Quit all Chrome windows first, then start with the debug flag. macOS app launches and bare-binary launches don't share the port. |
+| Login redirect on every nav | You're using `--user-data-dir=/tmp` or a fresh profile — point it at your real one to reuse cookies. |
+| Enterprise / SSO restrictions | Some corporate MDM disables CDP. There's no workaround in those environments — fall back to the in-extension probe loop. |
+
+## Security note
+
+Connecting to Chrome via CDP means anything able to reach
+`localhost:9222` can drive your browser. Only enable on machines where
+you trust everything local. Don't expose the debug port over the
+network.

package/lib/help-docs/19-jobs.md

@@ -0,0 +1,140 @@
+# Jobs
+
+A **Job** runs a connector tool on a schedule, dedups items, and dispatches
+each new item to a Pipeline or a Chat session.
+
+Typical examples:
+- Every 30 min, search Mantis for open bugs in version 25.4 → for each new
+  bug, trigger the `bug-triage` pipeline bound to project `my-app`.
+- Every 5 min, list Teams messages in the current chat → for each new
+  message, POST to a chat session that drafts a reply.
+
+Job is a separate primitive from Task and Pipeline. The CLI keeps
+`forge task` (single agent invocation) and `forge pipeline` (DAG of tasks)
+unchanged. `forge jobs` is new.
+
+## Anatomy
+
+| Field | Purpose |
+|---|---|
+| `source_connector`, `source_tool` | Which connector + tool to call each tick (e.g. `mantis.search_bugs`) |
+| `source_input` | JSON passed verbatim to the tool |
+| `items_path` | Dotted path into the tool's response to find the item array (`bugs`, empty = whole response if already an array) |
+| `dedup_field` | Field on each item used as the dedup key (e.g. `id`) |
+| `schedule_interval_minutes` | How often the scheduler ticks this job |
+| `dispatch_type` | `pipeline` or `chat` |
+| `dispatch_params` | Shape depends on dispatch_type — see below |
+
+### `dispatch_type: pipeline`
+
+```json
+{
+  "workflow_name": "bug-triage",
+  "project_path": "/Users/me/code/my-app",
+  "project_name": "my-app",
+  "input_template": {
+    "bug_id": "{{item.id}}",
+    "summary": "{{item.summary}}"
+  }
+}
+```
+
+Per new item, Forge renders `input_template` and calls the existing
+`triggerPipeline`. The pipeline shows up in the regular Pipelines view.
+
+### `dispatch_type: chat`
+
+```json
+{
+  "agent_profile": "my-litellm",
+  "session_title_template": "Bug {{item.id}}: {{item.summary}}",
+  "message_template": "Triage bug {{item.id}}.\nSummary: {{item.summary}}",
+  "reuse_session": false
+}
+```
+
+Per new item, Forge creates (or reuses, if `reuse_session: true`) a chat
+session and POSTs a message rendered from `message_template`. The chat
+agent gets the message and decides what tools to call.
+
+## Template syntax
+
+Only `{{item.<dotted.path>}}` is supported in v1. Missing paths render to
+empty string; objects render via `JSON.stringify`. `{{item}}` alone dumps
+the full item.
+
+## Backfill guard
+
+New jobs default to `mark_existing_as_seen: true` — on first tick, every
+item the connector returns is added to `job_seen` but **no dispatch
+happens**. This stops a fresh job from firing on historical data. Set to
+`false` at create time to dispatch immediately on first tick.
+
+## CLI
+
+```
+forge jobs                              # list
+forge jobs show <id>                    # full JSON detail
+forge jobs runs <id>                    # recent ticks (summary line per run)
+forge jobs dispatches <id> <run_id>     # per-item dispatches for one run (target ids)
+forge jobs run <id>                     # fire now (manual trigger)
+forge jobs enable <id>
+forge jobs disable <id>
+forge jobs reset <id>                   # wipe dedup state — next tick re-processes everything
+forge jobs rm <id>
+```
+
+## Tracking results
+
+| Want to know | Where |
+|---|---|
+| Job list, schedule, enabled, last/next run | Extension → Jobs tab |
+| Last 10 ticks summary (seen/new/dispatched, error) | Expand a job in Jobs tab |
+| Which items got dispatched in a single tick | Expand a run row — shows item_key + preview + target id + open-link |
+| What the resulting pipeline run did | Click "open →" on a pipeline dispatch — deep-links to Forge web Pipelines view |
+| What the resulting chat session said | Click "open →" on a chat dispatch — switches the side panel to chat tab and loads that session |
+| Run / dispatch detail from terminal | `forge jobs runs <id>` then `forge jobs dispatches <id> <run_id>` |
+| Raw rows | sqlite: `select * from job_runs / job_dispatches` |
+| Server-side log lines (every tick / error / dispatch) | Forge web → Logs view, search `[jobs]` (per-job: `[jobs] <job_id>`). The Jobs view has "View logs" buttons that pre-fill this filter. CLI: `tail -f ~/.forge/data/forge.log \| grep '\[jobs\]'`. |
+| Why a run shows 0 dispatches | Read the italic note under the run row — the scheduler writes a `notes` field explaining backfill / non-JSON response / items_path mismatch. |
+
+Create + edit happen via the Forge extension Jobs tab, or by `curl`ing
+`POST /api/jobs`. There is no YAML-on-disk format for Jobs (definitions
+live in sqlite).
+
+## Lifecycle
+
+1. Scheduler ticks every 60s. For each enabled job where `next_run_at`
+   has elapsed:
+2. Skip if a previous tick is still running (idempotent).
+3. Advance `next_run_at = now + schedule_interval_minutes`.
+4. Spawn a background tick:
+   - Call the connector via the chat tool-dispatcher (handles
+     `http` / `shell` / `browser` protocols uniformly).
+   - Parse the response — find items via `items_path`.
+   - For each item, compute `dedup_key = item[dedup_field]`,
+     `INSERT OR IGNORE INTO job_seen`. If new: dispatch.
+   - Update the `job_runs` row with counts + status.
+
+## Browser-protocol caveat
+
+If `source_connector` is a `protocol: browser` connector (Mantis, GitLab,
+Teams, PMDB), the job tick can only fire **when the extension bridge is
+connected** — the tool needs a live tab and `chrome.scripting`. A tick
+with no extension fails with `connector ... failed: No extension
+connected to the bridge`. The run is marked errored; the next tick
+retries.
+
+For 24/7 background polling, prefer connectors with `protocol: http`
+(e.g. `github-api`) — those run server-side and don't need a browser.
+
+## Tables
+
+```
+jobs                — definition (one row per job)
+job_runs            — one row per tick (success / failure)
+job_seen            — dedup keys per job (PRIMARY KEY (job_id, dedup_key))
+job_dispatches      — one row per per-item dispatch attempt (link to pipeline run / chat session)
+```
+
+All cascaded on `ON DELETE`.