@debugg-ai/debugg-ai-mcp 2.4.0 → 2.5.0

package/README.md

@@ -34,7 +34,7 @@ docker run -i --rm --init -e DEBUGGAI_API_KEY=your_api_key quinnosha/debugg-ai-m
 
 ## Tools
 
-The server exposes **11** tools grouped into Browser (2), Search (3), Projects (3), and Environments (3). The headline tool is `check_app_in_browser`; the rest manage projects, environments + their credentials, and execution history through a uniform `search_*` + CRUD pattern.
+The server exposes **12** tools grouped into Browser (3), Search (3), Projects (3), and Environments (3). The headline tools are `check_app_in_browser` (full AI agent) and `probe_page` (lightweight no-LLM page probe); the rest manage projects, environments + their credentials, and execution history through a uniform `search_*` + CRUD pattern.
 
 ### Browser
 
@@ -55,9 +55,45 @@ Runs an AI browser agent against your app. The agent navigates, interacts, and r
 
 One focused check per call. The agent has a ~25-step internal budget; split broader suites across multiple calls.
 
+Every successful run returns a `browserSession` block alongside the screenshot — presigned S3 URLs for the captured **HAR** (full network trace) and **console log** (every JS console message). Use them to detect refetch loops, hydration errors, and other runtime issues that pass type-checks and unit tests:
+
+```json
+"browserSession": {
+  "harUrl": "https://...session_18139.har?X-Amz-...",
+  "consoleLogUrl": "https://...session_18139_console.json?X-Amz-...",
+  "recordingUrl": "https://...session_18139_recording.webm?X-Amz-...",
+  "harStatus": "downloaded",
+  "consoleLogStatus": "downloaded",
+  "harRedactionStatus": "redacted",
+  "consoleLogRedactionStatus": "redacted"
+}
+```
+
+URLs are short-lived presigned S3 — refetch the parent execution via `search_executions` to renew. `harStatus` / `consoleLogStatus` disambiguate `'downloaded'` (URL fetchable), `'not_available'` (page emitted nothing), `'failed'` (capture broke). On a fresh run the URLs are commonly `null` because capture uploads async after the agent finishes — poll `search_executions` with the returned `executionId` until status reaches `'downloaded'`. Authorization / Cookie / `token`/`secret`/`api_key` headers are scrubbed server-side before the artifacts are persisted.
+
 #### `trigger_crawl`
 
-Fires a server-side browser-agent crawl to populate the project's knowledge graph. Localhost URLs tunnel automatically. Returns `{executionId, status, targetUrl, durationMs, outcome?, crawlSummary?, knowledgeGraph?}` with `knowledgeGraph.imported === true` on successful ingestion.
+Fires a server-side browser-agent crawl to populate the project's knowledge graph. Localhost URLs tunnel automatically. Returns `{executionId, status, targetUrl, durationMs, outcome?, crawlSummary?, knowledgeGraph?, browserSession?}` with `knowledgeGraph.imported === true` on successful ingestion. The `browserSession` block (HAR + console-log URLs, same shape as above) is also present on completed crawls.
+
+#### `probe_page`
+
+**Lightweight no-LLM batch page probe.** Pass 1-20 URLs; each navigates, waits for load, and returns rendered state — screenshot + page metadata + structured console errors + network summary. No agent loop, no LLM cost, no scenario assertions. Use it for "did I just break /settings?", multi-route smoke after a refactor, CI per-PR sweeps, and quick is-it-up checks where `check_app_in_browser`'s 60-150s agent loop is overkill.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `targets` | array **required** | 1-20 entries: `[{url, waitForSelector?, waitForLoadState?, timeoutMs?}]` |
+| `targets[].url` | string **required** | Public URL or localhost (auto-tunneled) |
+| `targets[].waitForLoadState` | enum | `'load'` (default) / `'domcontentloaded'` / `'networkidle'` |
+| `targets[].waitForSelector` | string | Optional CSS selector to wait for after navigation |
+| `targets[].timeoutMs` | number | Per-URL timeout, 1000-30000 (default 10000) |
+| `includeHtml` | boolean | Return raw HTML in each result (default false) |
+| `captureScreenshots` | boolean | Return one PNG per target (default true) |
+
+The whole batch shares a single backend execution + browser session + tunnel — 5 URLs in one call is dramatically faster than 5 parallel single-URL calls. Per-URL `error` field preserves batch resilience: a single failed target doesn't fail the others.
+
+**`networkSummary` aggregation key is `origin + pathname`** — refetch loops (`?n=0..4` repeatedly hitting the same endpoint) collapse into a single entry with the count, so `/api/poll` showing up with `count: 47` is the actionable "infinite refetch loop" signal users originally asked for.
+
+Performance budget: <10s for 1 URL, <25s for 20. Localhost dead-port returns `LocalServerUnreachable` in <2s without burning a workflow execution.
 
 ### Search (dual-mode: uuid detail OR filtered list)
 
@@ -128,10 +164,42 @@ Response-shape changes: the bare `count` field on list responses is gone — use
 
 ## Configuration
 
+| Env var | Required | Purpose |
+|---|---|---|
+| `DEBUGGAI_API_KEY` | yes | Backend API key. Aliases: `DEBUGGAI_API_TOKEN`, `DEBUGGAI_JWT_TOKEN`. |
+| `DEBUGGAI_API_URL` | no | Backend base URL. Defaults to `https://api.debugg.ai`. |
+| `DEBUGGAI_TOKEN_TYPE` | no | `token` (default) or `bearer`. |
+| `LOG_LEVEL` | no | `error` / `warn` / `info` (default) / `debug`. |
+| `POSTHOG_API_KEY` | no | Override the embedded telemetry project key (e.g. private fork). |
+| `DEBUGGAI_TELEMETRY_DISABLED` | no | Set to `1` / `true` / `yes` / `on` to disable telemetry entirely. |
+
 ```bash
 DEBUGGAI_API_KEY=your_api_key
 ```
 
+## Telemetry
+
+The MCP server ships with telemetry enabled by default — an embedded write-only PostHog project key (`phc_*`) so the team can observe cache hit rates, poll cadence, tunnel reliability, and other operational metrics across the install base. Captured events:
+
+| Event | When |
+|---|---|
+| `tool.executed` / `tool.failed` | Per tool call |
+| `workflow.executed` | Per browser-agent execution (carries `pollCount`, `durationMs`, `finalIntervalMs`) |
+| `tunnel.provisioned` / `tunnel.provision_retry` / `tunnel.stopped` | Per tunnel lifecycle event |
+| `template.lookup` / `project.lookup` | Cache hit/miss with `durationMs` on cold-call |
+
+Privacy posture:
+- The distinct ID is `SHA-256(api_key).slice(0, 16)` — never the raw key, no PII.
+- `phc_*` keys are write-only by PostHog convention; safe to embed in source.
+- Set `DEBUGGAI_TELEMETRY_DISABLED=1` to opt out entirely (resolves to a no-op provider; no events leave the process).
+
+The active mode is logged at boot:
+```
+Telemetry enabled (PostHog, DebuggAI default project). Set DEBUGGAI_TELEMETRY_DISABLED=1 to opt out.
+Telemetry enabled (PostHog, custom POSTHOG_API_KEY)
+Telemetry disabled (DEBUGGAI_TELEMETRY_DISABLED is set)
+```
+
 ## Local Development
 
 ```bash

package/dist/handlers/index.js

@@ -1,5 +1,6 @@
 export * from './testPageChangesHandler.js';
 export * from './triggerCrawlHandler.js';
+export * from './probePageHandler.js';
 export * from './searchProjectsHandler.js';
 export * from './searchEnvironmentsHandler.js';
 export * from './searchExecutionsHandler.js';

package/dist/handlers/probePageHandler.js

@@ -0,0 +1,275 @@
+/**
+ * probePageHandler — lightweight no-LLM batch page probe.
+ *
+ * Mirrors triggerCrawlHandler's 4-step pattern (find template → execute →
+ * poll → format response) but: (a) takes a list of targets and produces a
+ * list of results, (b) does no agent steps (zero LLM in critical path),
+ * (c) MCP-side aggregates per-target HAR slices into NetworkSummary[].
+ *
+ * The backend "Page Probe" workflow template runs:
+ *   browser.setup → loop[targets](page.navigate → page.capture) → done
+ *
+ * Each page.capture node emits per-iteration outputData with consoleSlice
+ * + harSlice windowed to that URL's load span — that's what makes per-URL
+ * networkSummary attribution accurate.
+ */
+import { config } from '../config/index.js';
+import { Logger } from '../utils/logger.js';
+import { handleExternalServiceError } from '../utils/errors.js';
+import { imageContentBlock } from '../utils/imageUtils.js';
+import { DebuggAIServerClient } from '../services/index.js';
+import { TunnelProvisionError } from '../services/tunnels.js';
+import { tunnelManager } from '../services/ngrok/tunnelManager.js';
+import { probeLocalPort, probeTunnelHealth } from '../utils/localReachability.js';
+import { extractLocalhostPort } from '../utils/urlParser.js';
+import { buildContext, findExistingTunnel, ensureTunnel, sanitizeResponseUrls, touchTunnelById, } from '../utils/tunnelContext.js';
+import { getCachedTemplateUuid, invalidateTemplateCache } from '../utils/handlerCaches.js';
+import { summarizeHar, summarizeConsole } from '../utils/harSummarizer.js';
+const logger = new Logger({ module: 'probePageHandler' });
+const TEMPLATE_KEYWORD = 'page probe';
+export async function probePageHandler(input, context, rawProgressCallback) {
+    const startTime = Date.now();
+    logger.toolStart('probe_page', input);
+    // Bead 0bq: progress circuit-breaker — see testPageChangesHandler for rationale.
+    let progressDisabled = false;
+    const progressCallback = rawProgressCallback
+        ? async (update) => {
+            if (progressDisabled)
+                return;
+            try {
+                await rawProgressCallback(update);
+            }
+            catch (err) {
+                progressDisabled = true;
+                logger.warn('Progress emission failed; disabling further emissions for this request', {
+                    error: err instanceof Error ? err.message : String(err),
+                });
+            }
+        }
+        : undefined;
+    const client = new DebuggAIServerClient(config.api.key);
+    await client.init();
+    const abortController = new AbortController();
+    const onStdinClose = () => {
+        abortController.abort();
+        progressDisabled = true;
+    };
+    process.stdin.once('close', onStdinClose);
+    // Per-target tunnel contexts. Index aligns with input.targets[].
+    const targetContexts = [];
+    // Tunnel keys we provisioned this call (for cleanup if creation fails after key acquired).
+    const acquiredKeyIds = [];
+    // Progress budget: 1 pre-flight + 1 template + 1 execute + N per-target captures + 1 done
+    const TOTAL_STEPS = 3 + input.targets.length + 1;
+    let progressStep = 0;
+    try {
+        if (progressCallback) {
+            await progressCallback({ progress: ++progressStep, total: TOTAL_STEPS, message: `Pre-flight + tunnel setup (${input.targets.length} target${input.targets.length === 1 ? '' : 's'})...` });
+        }
+        // ── Per-target pre-flight + tunnel resolution ──────────────────────────
+        for (const target of input.targets) {
+            const ctx = buildContext(target.url);
+            if (ctx.isLocalhost) {
+                // Pre-flight TCP probe: fail fast if dev server isn't listening.
+                const port = extractLocalhostPort(ctx.originalUrl);
+                if (typeof port === 'number') {
+                    const probe = await probeLocalPort(port);
+                    if (!probe.reachable) {
+                        const payload = {
+                            error: 'LocalServerUnreachable',
+                            message: `No server listening on 127.0.0.1:${port}. Start your dev server on that port before running probe_page. Probe result: ${probe.code} (${probe.detail ?? 'no detail'}).`,
+                            detail: {
+                                port,
+                                probeCode: probe.code,
+                                probeDetail: probe.detail,
+                                elapsedMs: probe.elapsedMs,
+                            },
+                        };
+                        logger.warn(`Pre-flight port probe failed for ${ctx.originalUrl}: ${probe.code} in ${probe.elapsedMs}ms`);
+                        return { content: [{ type: 'text', text: JSON.stringify(payload, null, 2) }], isError: true };
+                    }
+                }
+                // Reuse existing tunnel for this port if any; otherwise provision.
+                const reused = findExistingTunnel(ctx);
+                if (reused) {
+                    targetContexts.push(reused);
+                }
+                else {
+                    let tunnel;
+                    try {
+                        tunnel = await client.tunnels.provisionWithRetry();
+                    }
+                    catch (provisionError) {
+                        const msg = provisionError instanceof Error ? provisionError.message : String(provisionError);
+                        const diag = provisionError instanceof TunnelProvisionError ? ` ${provisionError.diagnosticSuffix()}` : '';
+                        throw new Error(`Failed to provision tunnel for ${ctx.originalUrl}. ` +
+                            `(Detail: ${msg})${diag}`);
+                    }
+                    acquiredKeyIds.push(tunnel.keyId);
+                    let tunneled;
+                    try {
+                        tunneled = await ensureTunnel(ctx, tunnel.tunnelKey, tunnel.tunnelId, tunnel.keyId, () => client.revokeNgrokKey(tunnel.keyId));
+                    }
+                    catch (tunnelError) {
+                        const msg = tunnelError instanceof Error ? tunnelError.message : String(tunnelError);
+                        throw new Error(`Tunnel creation failed for ${ctx.originalUrl}. (Detail: ${msg})`);
+                    }
+                    // Tunnel health probe: catch the IPv4/IPv6 bind / dead-server case
+                    // before committing to a full backend execution.
+                    if (tunneled.targetUrl) {
+                        const health = await probeTunnelHealth(tunneled.targetUrl);
+                        if (!health.healthy) {
+                            const payload = {
+                                error: 'TunnelTrafficBlocked',
+                                message: `Tunnel established but traffic isn't reaching the dev server. ${health.detail ?? ''}`,
+                                detail: {
+                                    code: health.code,
+                                    status: health.status,
+                                    ngrokErrorCode: health.ngrokErrorCode,
+                                    elapsedMs: health.elapsedMs,
+                                },
+                            };
+                            if (tunneled.tunnelId) {
+                                tunnelManager.stopTunnel(tunneled.tunnelId).catch((err) => logger.warn(`Failed to stop broken tunnel ${tunneled.tunnelId}: ${err}`));
+                            }
+                            return { content: [{ type: 'text', text: JSON.stringify(payload, null, 2) }], isError: true };
+                        }
+                    }
+                    targetContexts.push(tunneled);
+                }
+            }
+            else {
+                // Public URL — no tunnel needed.
+                targetContexts.push(ctx);
+            }
+        }
+        // ── Locate workflow template ───────────────────────────────────────────
+        if (progressCallback) {
+            await progressCallback({ progress: ++progressStep, total: TOTAL_STEPS, message: 'Locating page-probe workflow template...' });
+        }
+        const templateUuid = await getCachedTemplateUuid(TEMPLATE_KEYWORD, async (name) => {
+            return client.workflows.findTemplateByName(name);
+        });
+        if (!templateUuid) {
+            throw new Error(`Page Probe Workflow Template not found. ` +
+                `Ensure the backend has a template matching "${TEMPLATE_KEYWORD}" seeded and accessible.`);
+        }
+        // ── Build contextData (camelCase; axiosTransport snake_cases on the wire) ──
+        const contextData = {
+            targets: input.targets.map((t, i) => ({
+                url: targetContexts[i].targetUrl ?? t.url,
+                waitForSelector: t.waitForSelector,
+                waitForLoadState: t.waitForLoadState,
+                timeoutMs: t.timeoutMs,
+            })),
+            includeHtml: input.includeHtml,
+            captureScreenshots: input.captureScreenshots,
+        };
+        // ── Execute ────────────────────────────────────────────────────────────
+        if (progressCallback) {
+            await progressCallback({ progress: ++progressStep, total: TOTAL_STEPS, message: 'Queuing workflow execution...' });
+        }
+        const executeResponse = await client.workflows.executeWorkflow(templateUuid, contextData);
+        const executionUuid = executeResponse.executionUuid;
+        logger.info(`Probe execution queued: ${executionUuid}`);
+        // ── Poll ───────────────────────────────────────────────────────────────
+        let lastCompleted = -1;
+        const finalExecution = await client.workflows.pollExecution(executionUuid, async (exec) => {
+            // Keep all active tunnels alive during polling.
+            for (const tc of targetContexts) {
+                if (tc.tunnelId)
+                    touchTunnelById(tc.tunnelId);
+            }
+            if (!progressCallback)
+                return;
+            const completedNodes = (exec.nodeExecutions ?? []).filter(n => n.nodeType === 'page.capture' && n.status === 'success').length;
+            if (completedNodes !== lastCompleted) {
+                lastCompleted = completedNodes;
+                await progressCallback({
+                    progress: Math.min(progressStep + completedNodes, TOTAL_STEPS - 1),
+                    total: TOTAL_STEPS,
+                    message: `Probed ${completedNodes}/${input.targets.length} target${input.targets.length === 1 ? '' : 's'}...`,
+                });
+            }
+        }, abortController.signal);
+        // ── Format response ────────────────────────────────────────────────────
+        const duration = Date.now() - startTime;
+        const captureNodes = (finalExecution.nodeExecutions ?? [])
+            .filter(n => n.nodeType === 'page.capture')
+            .sort((a, b) => a.executionOrder - b.executionOrder);
+        const results = [];
+        const screenshotBlocks = [];
+        for (let i = 0; i < input.targets.length; i++) {
+            const target = input.targets[i];
+            const node = captureNodes[i];
+            const data = node?.outputData ?? {};
+            const result = {
+                url: target.url, // ORIGINAL caller URL — not the tunneled rewrite
+                finalUrl: typeof data.finalUrl === 'string' ? data.finalUrl : (typeof data.url === 'string' ? data.url : target.url),
+                statusCode: typeof data.statusCode === 'number' ? data.statusCode : 0,
+                title: typeof data.title === 'string' ? data.title : null,
+                loadTimeMs: typeof data.loadTimeMs === 'number' ? data.loadTimeMs : 0,
+                consoleErrors: summarizeConsole(Array.isArray(data.consoleSlice) ? data.consoleSlice : []),
+                networkSummary: summarizeHar(Array.isArray(data.harSlice) ? data.harSlice : []),
+            };
+            if (input.includeHtml && typeof data.html === 'string') {
+                result.html = data.html;
+            }
+            if (typeof data.error === 'string' && data.error) {
+                result.error = data.error;
+            }
+            results.push(result);
+            if (input.captureScreenshots && typeof data.screenshotB64 === 'string' && data.screenshotB64) {
+                screenshotBlocks.push(imageContentBlock(data.screenshotB64, 'image/png'));
+            }
+        }
+        const responsePayload = {
+            executionId: executionUuid,
+            durationMs: typeof finalExecution.durationMs === 'number' ? finalExecution.durationMs : duration,
+            results,
+        };
+        if (finalExecution.browserSession) {
+            responsePayload.browserSession = finalExecution.browserSession;
+        }
+        // Sanitize ngrok URLs from the entire payload — agent-authored strings in
+        // node outputData (titles, HTML, console messages from the page itself)
+        // can occasionally contain the tunnel URL; rewrite to the original
+        // localhost origin per tunnel context. For multi-localhost batches we
+        // run sanitize once per localhost target since each may have its own
+        // tunnel↔origin mapping.
+        let sanitizedPayload = responsePayload;
+        for (const tc of targetContexts) {
+            if (tc.isLocalhost) {
+                sanitizedPayload = sanitizeResponseUrls(sanitizedPayload, tc);
+            }
+        }
+        logger.toolComplete('probe_page', duration);
+        return {
+            content: [
+                { type: 'text', text: JSON.stringify(sanitizedPayload, null, 2) },
+                ...screenshotBlocks,
+            ],
+        };
+    }
+    catch (error) {
+        const duration = Date.now() - startTime;
+        logger.toolError('probe_page', error, duration);
+        if (error instanceof Error && (error.message.includes('not found') || error.message.includes('401'))) {
+            invalidateTemplateCache();
+        }
+        throw handleExternalServiceError(error, 'DebuggAI', 'probe_page execution');
+    }
+    finally {
+        process.stdin.removeListener('close', onStdinClose);
+        // Tunnels intentionally NOT torn down — reuse pattern (bead vwd) +
+        // 55-min idle auto-shutoff. Revoke only orphaned keys (we acquired the
+        // key but tunnel creation failed before ensureTunnel completed).
+        for (let i = 0; i < acquiredKeyIds.length; i++) {
+            const keyId = acquiredKeyIds[i];
+            const tc = targetContexts[i];
+            if (tc && !tc.tunnelId && keyId) {
+                client.revokeNgrokKey(keyId).catch(err => logger.warn(`Failed to revoke unused ngrok key ${keyId}: ${err}`));
+            }
+        }
+    }
+}

package/dist/handlers/searchEnvironmentsHandler.js

@@ -61,6 +61,11 @@ export async function searchEnvironmentsHandler(input, _context) {
         const client = new DebuggAIServerClient(config.api.key);
         await client.init();
         // ── Resolve projectUuid ──
+        // Bead gb4n: when projectUuid is provided directly (caller skips git
+        // auto-resolution), `name` and `repoName` are unknown. OMIT those fields
+        // rather than emitting nulls — null fields surprised callers and
+        // muddied the contract. If a caller needs them, they fetch via
+        // search_projects.
         let projectUuid = input.projectUuid;
         let project = null;
         if (!projectUuid) {
@@ -73,10 +78,15 @@ export async function searchEnvironmentsHandler(input, _context) {
                 return noProjectResolved(pagination, `No DebuggAI project found for repo "${repoName}". Pass projectUuid explicitly.`);
             }
             projectUuid = resolved.uuid;
-            project = { uuid: resolved.uuid, name: resolved.name, repoName: resolved.repo?.name ?? repoName };
+            project = { uuid: resolved.uuid };
+            if (resolved.name)
+                project.name = resolved.name;
+            const rn = resolved.repo?.name ?? repoName;
+            if (rn)
+                project.repoName = rn;
         }
         else {
-            project = { uuid: projectUuid, name: null, repoName: null };
+            project = { uuid: projectUuid };
         }
         // ── uuid mode ──
         if (input.uuid) {