webctx 0.2.1 → 0.2.3

package/AGENTS.md

@@ -4,7 +4,7 @@ Guidance for coding agents working in `webctx`.
 
 ## Purpose
 
-This repo contains the pure Go port of the `webctx` CLI.
+This repo contains the Go `webctx` CLI and its hosted Origo MCP transport, sharing one retrieval engine.
 
 It is no longer a generic starter template. Treat the current CLI behavior as the source of truth unless the user explicitly asks to change it.
 
@@ -14,6 +14,9 @@ It is no longer a generic starter template. Treat the current CLI behavior as th
 - `internal/app/app.go`: CLI parsing and top-level command routing.
 - `internal/app/tools.go`: search provider clients, ranking, formatting, and HTTP helpers.
 - `internal/app/scrape.go`: GitHub raw-content optimization, `.md` fetch path, Firecrawl queue, and env loading.
+- `pkg/retrieval/`: public adapter to `internal/app` for Vercel's Go packaging restrictions.
+- `pkg/origo/`: authenticated Streamable HTTP MCP with exactly `read_link` and `map_site`.
+- `api/mcp.go`: Go Function entrypoint for independent Origo deployment.
 - `internal/app/app_test.go`: unit tests for CLI behavior and core helpers.
 - `bin/webctx.js`: npm shim that invokes the packaged native binary.
 - `scripts/postinstall.js`: downloads release binary on install, falls back to `go build`.
@@ -51,7 +54,7 @@ Preserve these commands unless the user explicitly asks to change them:
 Behavioral expectations:
 
 - `search` combines Brave, Tavily, and Exa results, then re-ranks them with duplicate-aware scoring.
-- `read-link` keeps the current GitHub raw-content fast path, `.md` fast path, and Firecrawl fallback settings.
+- `read-link` keeps the current GitHub raw-content and `.md` fast paths. Firecrawl uses its built-in 30-minute cache, automatic and enhanced proxies, then browser extraction as a final fallback.
 - `map-site` keeps the current Firecrawl map request settings.
 - The CLI should remain agent-friendly and emit plain markdown/text output.
 
@@ -71,7 +74,7 @@ Behavioral expectations:
 
 Release pipeline triggers on `v*` tags and expects:
 
-- `NPM_TOKEN` GitHub secret present.
+- The `NPM_TOKEN` GitHub Actions secret must contain a valid npm access token with publish rights for `webctx`; publishing uses `NODE_AUTH_TOKEN` (not OIDC trusted publishing).
 - npm package name in `package.json` is publishable under your account/org.
 - repository URL matches the release origin used by `scripts/postinstall.js`.
 
@@ -81,7 +84,9 @@ Release binaries should embed the tagged version into `internal/buildinfo.Versio
 
 - Prefer additive changes and keep the CLI output stable.
 - Do not silently change Firecrawl request settings unless the user explicitly wants behavioral changes.
-- Do not reintroduce MCP/server code unless requested; this repo is intentionally CLI-only.
+- Do not add search to Origo's MCP tool surface; only `read_link` and `map_site` are registered.
+- Keep `origo-api` deployment separate from the existing WebCTX docs Vercel project and tag-driven CLI releases.
+- Never introduce Redis, custom KV, or another caching service for Origo: the user explicitly chose Firecrawl's own 30-minute cache.
 
 ## Changelog Guidelines
 

package/CONTRIBUTORS.md

@@ -2,6 +2,8 @@
 
 Maintainer notes for `webctx`.
 
+Origo is the separately deployed MCP transport in this same repository. The CLI, MCP, and ZueDocs share source code but have independent distribution pipelines.
+
 ## Prerequisites
 
 - Go `1.26+`
@@ -78,39 +80,38 @@ make release-tag VERSION=x.y.z
 - GitHub release publish
 - npm publish
 
-## Required GitHub secret
-
-- `NPM_TOKEN`: npm automation token with publish rights for your package.
-
-Set via GitHub CLI:
-
-```bash
-gh secret set NPM_TOKEN --repo amxv/webctx
-```
-
-## npm token setup
+## Automatic npm access-token publishing
 
-Create token at npm:
+This project publishes npm releases using the repository's GitHub Actions secret `NPM_TOKEN`. Set it to a valid npm access token with publish access to the `webctx` package and appropriate npm two-factor-authentication settings for automated publishing. GitHub Actions supplies it as `NODE_AUTH_TOKEN` during the `release.yml` workflow's npm step.
 
-- Profile -> Access Tokens -> Create New Token
-- Use an automation/granular token scoped to required package/org
+If a token expires, rotate it in npm and update `NPM_TOKEN` using `gh secret set NPM_TOKEN --repo amxv/webctx`. Never commit tokens or print them in job logs. Trusted publishing is not used.
 
-Validate auth locally:
+If a release's GitHub binaries were successfully published while npm authentication failed, dispatch `npm-publish-retry.yml` with the existing version. Do not create or move the release tag just to retry npm publishing:
 
 ```bash
-npm whoami
+gh workflow run npm-publish-retry.yml --repo amxv/webctx --ref main -f version=0.2.2
 ```
 
 ## Project layout
 
 - `cmd/webctx/main.go`: CLI entrypoint
 - `internal/app/`: CLI parsing, search, ranking, scrape, env loading, and Firecrawl queue logic
+- `pkg/retrieval/`: small public adapter to the shared read-link / site-map engine
+- `pkg/origo/`: authenticated, stateless MCP server with exactly two tools
+- `api/mcp.go`: Vercel Go Function entrypoint for Origo
+- `vercel.mjs`: project-specific Vercel settings for docs and Origo
 - `internal/buildinfo/`: build-time version plumbing for `--version`
 - `bin/webctx.js`: npm shim that invokes the packaged native binary
 - `scripts/postinstall.js`: downloads the release binary on install and falls back to local `go build`
 - `.github/workflows/release.yml`: tag-driven release pipeline
 - `AGENTS.md`: guidance for coding agents
 
+## Origo deployment
+
+The `origo-api` Vercel project is linked to this GitHub repo and uses `ORIGO_DEPLOYMENT=1` to select its Go Function configuration. It needs `ORIGO_API_KEY` and `FIRECRAWL_API_KEY` production environment variables; optionally add `GH_TOKEN` for native authenticated GitHub reads. The docs project remains `webctx-docs` and never receives the Origo connection key.
+
+Both projects deploy automatically from `main`. `scripts/should-build.mjs` gates docs and API builds independently, while `v*` GitHub tags still drive the CLI release workflow.
+
 ## Notes on package naming
 
 `webctx` is already configured. If you ever rename or move the package, update all of the following together:

package/README.md

@@ -10,7 +10,22 @@ webctx read-link <url>
 webctx map-site <url>
 ```
 
-The output is plain text or markdown, so it is easy to hand to ChatGPT, Codex, Claude Code, a shell script, or another tool.
+The output is plain text or markdown, so it is easy to hand to ChatGPT, Codex, Claude Code, a shell script, or another tool. Search now retrieves relevant source excerpts automatically instead of returning only URLs. For a specific URL, add an optional question to explore related documentation and surface exact code and API details.
+
+```bash
+webctx search "Go MCP SDK read-only tool annotations"
+webctx read-link https://docs.example.com/api --question "How do authentication and pagination work?"
+```
+
+## Origo: hosted MCP access
+
+**Origo** is the MCP transport over the same WebCTX retrieval engine. Connect an MCP client to `https://api.origo.ashray.xyz/mcp?key=<your-private-key>` to access exactly two tools: `read_link` and `map_site`. Origo does not expose `search`; the CLI continues to provide all three commands. `read_link` accepts an optional `question` to retrieve focused source context while keeping ordinary URL-only reads intact.
+
+Both surfaces share native GitHub/Markdown readers and Firecrawl fallback behavior. Firecrawl's built-in cache is limited to **30 minutes**. Difficult pages automatically escalate through Firecrawl's proxy fallback and, if necessary, a temporary Browser Sandbox session. There is no Redis or additional cache database.
+
+Source-aware reading also prefers raw JSON, YAML, and OpenAPI files when the URL identifies one; question-focused reads can follow up to two relevant pages on the same host, including links found in `llms.txt`, and inspect embedded structured JSON. Firecrawl Alexandria tool matching and search discovery are automatic when supported, but **no paid Alexandria provider tools are executed**. The original source URLs remain explicit so an agent can verify facts.
+
+Origo deploys independently through the `origo-api` Vercel project on pushes to `main`. CLI releases and npm publishing are unchanged. See the [Origo MCP guide](https://webctx.ashray.xyz/docs/origo) for architecture, setup, and security notes.
 
 ## Install
 
@@ -63,7 +78,7 @@ Recognized native GitHub auth, private/not-found, and rate-limit failures stay a
 
 ## Search
 
-Normal search asks Brave, Tavily, and Exa, removes duplicate URLs, and returns one useful list.
+Normal search asks Brave, Tavily, and Exa, removes duplicate URLs, automatically reads three promising sources, follows up to two relevant links, and returns excerpts plus the ranked list. Alexandria tool discovery is included when available, without paid execution.
 
 ```bash
 webctx search "next.js server components"

package/go.mod

@@ -1,3 +1,16 @@
 module github.com/amxv/webctx
 
 go 1.26
+
+require github.com/modelcontextprotocol/go-sdk v1.8.0
+
+require (
+	github.com/google/jsonschema-go v0.4.3 // indirect
+	github.com/segmentio/asm v1.1.3 // indirect
+	github.com/segmentio/encoding v0.5.4 // indirect
+	github.com/yosida95/uritemplate/v3 v3.0.2 // indirect
+	golang.org/x/oauth2 v0.35.0 // indirect
+	golang.org/x/sync v0.20.0 // indirect
+	golang.org/x/sys v0.41.0 // indirect
+	golang.org/x/time v0.15.0 // indirect
+)

package/go.sum

@@ -0,0 +1,24 @@
+github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY=
+github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
+github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
+github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
+github.com/google/jsonschema-go v0.4.3 h1:/DBOLZTfDow7pe2GmaJNhltueGTtDKICi8V8p+DQPd0=
+github.com/google/jsonschema-go v0.4.3/go.mod h1:r5quNTdLOYEz95Ru18zA0ydNbBuYoo9tgaYcxEYhJVE=
+github.com/modelcontextprotocol/go-sdk v1.8.0 h1:KIvahhYqwtbeniWVPs3TcXEA7b8jEtwfBpOTAI+Urx4=
+github.com/modelcontextprotocol/go-sdk v1.8.0/go.mod h1:dL7u98E/zjJTGzEq+j30jQ8K2k1mb6LeAH4inEcSGts=
+github.com/segmentio/asm v1.1.3 h1:WM03sfUOENvvKexOLp+pCqgb/WDjsi7EK8gIsICtzhc=
+github.com/segmentio/asm v1.1.3/go.mod h1:Ld3L4ZXGNcSLRg4JBsZ3//1+f/TjYl0Mzen/DQy1EJg=
+github.com/segmentio/encoding v0.5.4 h1:OW1VRern8Nw6ITAtwSZ7Idrl3MXCFwXHPgqESYfvNt0=
+github.com/segmentio/encoding v0.5.4/go.mod h1:HS1ZKa3kSN32ZHVZ7ZLPLXWvOVIiZtyJnO1gPH1sKt0=
+github.com/yosida95/uritemplate/v3 v3.0.2 h1:Ed3Oyj9yrmi9087+NczuL5BwkIc4wvTb5zIM+UJPGz4=
+github.com/yosida95/uritemplate/v3 v3.0.2/go.mod h1:ILOh0sOhIJR3+L/8afwt/kE++YT040gmv5BQTMR2HP4=
+golang.org/x/oauth2 v0.35.0 h1:Mv2mzuHuZuY2+bkyWXIHMfhNdJAdwW3FuWeCPYN5GVQ=
+golang.org/x/oauth2 v0.35.0/go.mod h1:lzm5WQJQwKZ3nwavOZ3IS5Aulzxi68dUSgRHujetwEA=
+golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
+golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
+golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
+golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
+golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
+golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
+golang.org/x/tools v0.42.0 h1:uNgphsn75Tdz5Ji2q36v/nsFSfR/9BRFvqhGBaJGd5k=
+golang.org/x/tools v0.42.0/go.mod h1:Ma6lCIwGZvHK6XtgbswSoWroEkhugApmsXyrUmBhfr0=

package/internal/app/alexandria.go

@@ -0,0 +1,102 @@
+package app
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"os"
+	"strings"
+	"time"
+)
+
+// formatAlexandriaTools describes discovered capabilities, not their data.
+// Discovery is free but actual provider execution is not, and is never
+// silently authorized by reading a link or searching the web.
+func formatAlexandriaTools(raw any) string {
+	items, ok := raw.([]any)
+	if !ok || len(items) == 0 {
+		return ""
+	}
+	parts := []string{"**Available structured-data capabilities (not executed):**"}
+	count := 0
+	for _, item := range items {
+		tool, ok := item.(map[string]any)
+		if !ok {
+			continue
+		}
+		provider := safeToolField(tool["provider"], 80)
+		capability := safeToolField(tool["capability"], 160)
+		if provider == "" || capability == "" {
+			continue
+		}
+		description := safeToolField(tool["description"], 180)
+		if description == "" {
+			description = safeToolField(tool["name"], 180)
+		}
+		line := fmt.Sprintf("- `%s/%s`", provider, capability)
+		if description != "" {
+			line += ": " + description
+		}
+		parts = append(parts, line)
+		count++
+		if count >= 3 {
+			break
+		}
+	}
+	if count == 0 {
+		return ""
+	}
+	return strings.Join(parts, "\n")
+}
+
+func safeToolField(value any, limit int) string {
+	s, ok := value.(string)
+	if !ok {
+		return ""
+	}
+	s = strings.Join(strings.Fields(s), " ")
+	s = strings.ReplaceAll(s, "`", "'")
+	if len(s) > limit {
+		s = s[:limit] + "…"
+	}
+	return s
+}
+
+func withSourceTools(markdown, toolSummary string) string {
+	if toolSummary == "" {
+		return markdown
+	}
+	return markdown + "\n\n---\n\n" + toolSummary + "\n\n*Matched through Firecrawl Alexandria. Discovery does not verify, license, or execute these providers.*"
+}
+
+func discoverAlexandria(query string) string {
+	apiKey := strings.TrimSpace(os.Getenv("FIRECRAWL_API_KEY"))
+	if apiKey == "" {
+		return ""
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 12*time.Second)
+	defer cancel()
+	response, err := doJSONRequest(ctx, http.MethodPost, "https://api.firecrawl.dev/v2/search", map[string]string{
+		"Authorization": "Bearer " + apiKey,
+		"Content-Type":  "application/json",
+	}, map[string]any{
+		"query":      query,
+		"sources":    []string{"alexandria"},
+		"limit":      3,
+		"toolDetail": "compact",
+	})
+	if err != nil {
+		return "" // unavailable Alexandria is never a search failure
+	}
+	var result struct {
+		Success bool `json:"success"`
+		Data    struct {
+			Tools []any `json:"tools"`
+		} `json:"data"`
+	}
+	if json.Unmarshal(response, &result) != nil || !result.Success {
+		return ""
+	}
+	return formatAlexandriaTools(result.Data.Tools)
+}

package/internal/app/app.go

@@ -51,10 +51,10 @@ func Run(args []string, stdout, stderr io.Writer) int {
 	case "read-link":
 		if strings.TrimSpace(input) == "" {
 			_, _ = fmt.Fprintln(stderr, "Error: read-link requires a URL")
-			_, _ = fmt.Fprintln(stdout, "Usage: webctx read-link <url>")
+			_, _ = fmt.Fprintln(stdout, "Usage: webctx read-link <url> [--question 'what to find']")
 			return 1
 		}
-		text, err := ReadLink(input)
+		text, err := ReadLinkFocused(input, flags["question"])
 		if err != nil {
 			_, _ = fmt.Fprintln(stderr, err.Error())
 			return 1
@@ -86,7 +86,7 @@ func usageText() string {
 
 Usage:
   webctx search <query> [--exclude domain1,domain2] [--keyword phrase]
-  webctx read-link <url>
+  webctx read-link <url> [--question 'what to find']
   webctx map-site <url>
 
 Examples:
@@ -94,6 +94,7 @@ Examples:
   webctx search "react hooks" --exclude youtube.com,vimeo.com
   webctx search "drizzle orm" --keyword "migration guide"
   webctx read-link https://docs.example.com/guide
+  webctx read-link https://docs.example.com/api --question "How do auth and pagination work?"
   webctx map-site https://example.com`, version)
 }
 

package/internal/app/firecrawl_fallback.go

@@ -0,0 +1,176 @@
+package app
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/url"
+	"strconv"
+	"strings"
+	"time"
+)
+
+type scrapedPage struct {
+	Title       string
+	Markdown    string
+	ToolSummary string
+}
+
+func usableScrapedMarkdown(markdown string) bool {
+	text := strings.ToLower(strings.TrimSpace(markdown))
+	if len(text) < 10 {
+		return false
+	}
+	for _, marker := range []string{
+		"just a moment...", "checking your browser", "verify you are human",
+		"enable javascript and cookies", "access denied | cloudflare",
+		"attention required! | cloudflare", "no content extracted",
+	} {
+		if strings.Contains(text, marker) && len(text) < 2000 {
+			return false
+		}
+	}
+	return true
+}
+
+func firecrawlScrape(rawURL, apiKey, proxy string) (scrapedPage, error) {
+	requestBody := map[string]any{
+		"url":                rawURL,
+		"formats":            []string{"markdown"},
+		"domainTools":        true, // Alexandria discovery, not paid execution.
+		"onlyMainContent":    true,
+		"blockAds":           true,
+		"removeBase64Images": true,
+		"maxAge":             30 * 60 * 1000, // Firecrawl's built-in cache, 30 minutes.
+		"excludeTags":        []string{"script", "style", "meta", "noscript", "svg", "img", "nav", "footer", "header", "aside", ".advertisement", "#ad"},
+	}
+	if proxy != "" {
+		requestBody["proxy"] = proxy
+		// A blocked cached result must not mask an actual enhanced retry.
+		requestBody["maxAge"] = 0
+	}
+	if strings.HasSuffix(strings.ToLower(rawURL), ".pdf") {
+		requestBody["parsers"] = []string{"pdf"}
+	}
+	data, err := getFirecrawlQueue().enqueue(rawURL, func() (map[string]any, error) {
+		ctx, cancel := context.WithTimeout(context.Background(), 32*time.Second)
+		defer cancel()
+		body, err := doJSONRequest(ctx, http.MethodPost, "https://api.firecrawl.dev/v2/scrape", map[string]string{
+			"Authorization": "Bearer " + apiKey,
+			"Content-Type":  "application/json",
+		}, requestBody)
+		// Alexandria is a separately enabled Firecrawl capability. A 403 from
+		// domain matching must never break ordinary page extraction for users that
+		// haven't enabled Alexandria or have zero-data-retention configured.
+		if err != nil && strings.Contains(err.Error(), "403") {
+			requestBody["domainTools"] = false
+			body, err = doJSONRequest(ctx, http.MethodPost, "https://api.firecrawl.dev/v2/scrape", map[string]string{
+				"Authorization": "Bearer " + apiKey,
+				"Content-Type":  "application/json",
+			}, requestBody)
+		}
+		if err != nil {
+			return nil, err
+		}
+		var parsed map[string]any
+		if err := json.Unmarshal(body, &parsed); err != nil {
+			return nil, fmt.Errorf("decode Firecrawl scrape: %w", err)
+		}
+		if success, _ := parsed["success"].(bool); !success {
+			return nil, fmt.Errorf("Firecrawl scrape failed: %v", parsed["error"])
+		}
+		return parsed, nil
+	})
+	if err != nil {
+		return scrapedPage{}, err
+	}
+	pageData, _ := data["data"].(map[string]any)
+	metadata, _ := pageData["metadata"].(map[string]any)
+	page := scrapedPage{
+		Title:       stringValue(metadata["title"]),
+		Markdown:    stringValue(pageData["markdown"]),
+		ToolSummary: formatAlexandriaTools(pageData["tools"]),
+	}
+	if status, ok := metadata["statusCode"].(float64); ok && status >= 400 {
+		return page, fmt.Errorf("source responded with HTTP %d", int(status))
+	}
+	return page, nil
+}
+
+func firecrawlJSON(ctx context.Context, method, endpoint, apiKey string, data map[string]any) (map[string]any, error) {
+	body, err := doJSONRequest(ctx, method, endpoint, map[string]string{
+		"Authorization": "Bearer " + apiKey,
+		"Content-Type":  "application/json",
+	}, data)
+	if err != nil {
+		return nil, err
+	}
+	var parsed map[string]any
+	if err := json.Unmarshal(body, &parsed); err != nil {
+		return nil, err
+	}
+	if success, ok := parsed["success"].(bool); ok && !success {
+		return nil, fmt.Errorf("Firecrawl browser: %v", parsed["error"])
+	}
+	return parsed, nil
+}
+
+// quoteShellURL escapes a validated URL for the Firecrawl sandbox's bash mode.
+// This invokes only the installed agent-browser CLI, never model-written code.
+func quoteShellURL(rawURL string) string {
+	return "'" + strings.ReplaceAll(rawURL, "'", "'\\''") + "'"
+}
+
+func firecrawlBrowserFallback(rawURL, apiKey string) (string, string, error) {
+	ctx, cancel := context.WithTimeout(context.Background(), 28*time.Second)
+	defer cancel()
+	// Failed scrape responses do not reliably include a scrape-bound browser ID.
+	// Open a disposable session with the standalone browser API instead.
+	session, err := firecrawlJSON(ctx, http.MethodPost, "https://api.firecrawl.dev/v2/interact", apiKey, map[string]any{
+		"ttl": 60, "activityTtl": 30,
+	})
+	if err != nil {
+		return "", "", err
+	}
+	sessionID := stringValue(session["id"])
+	if sessionID == "" {
+		return "", "", fmt.Errorf("Firecrawl browser returned no session ID")
+	}
+	endpoint := "https://api.firecrawl.dev/v2/interact/" + url.PathEscape(sessionID)
+	defer stopFirecrawlBrowser(endpoint, apiKey)
+	result, err := firecrawlJSON(ctx, http.MethodPost, endpoint+"/execute", apiKey, map[string]any{
+		// The sandbox's agent-browser does not expose a "scrape" subcommand
+		// even though Firecrawl's examples use it. DOM innerText is supported.
+		// Suppress navigation's banner so stdout contains only the evaluated text.
+		"code":     "agent-browser open " + quoteShellURL(rawURL) + " >/dev/null && agent-browser eval 'document.body.innerText'",
+		"language": "bash",
+		"timeout":  20,
+	})
+	if err != nil {
+		return "", "", err
+	}
+	if code, ok := result["exitCode"].(float64); ok && code != 0 {
+		return "", "", fmt.Errorf("browser command exited with %d", int(code))
+	}
+	text := stringValue(result["stdout"])
+	if text == "" {
+		text = stringValue(result["output"])
+	}
+	if text == "" {
+		text = stringValue(result["result"])
+	}
+	// agent-browser eval serializes JS strings as JSON with surrounding quotes.
+	// Decode those escapes rather than handing the model a JSON string literal.
+	if parsed, err := strconv.Unquote(strings.TrimSpace(text)); err == nil {
+		text = parsed
+	}
+	return "Browser extraction", text, nil
+}
+
+func stopFirecrawlBrowser(endpoint, apiKey string) {
+	ctx, cancel := context.WithTimeout(context.Background(), 4*time.Second)
+	defer cancel()
+	// Best-effort cleanup regardless of whether extraction succeeded.
+	_, _ = firecrawlJSON(ctx, http.MethodDelete, endpoint, apiKey, nil)
+}

package/internal/app/firecrawl_fallback_test.go

@@ -0,0 +1,83 @@
+package app
+
+import (
+	"io"
+	"net/http"
+	"strings"
+	"testing"
+)
+
+func TestSourceURLSafety(t *testing.T) {
+	for _, raw := range []string{"file:///etc/passwd", "http://localhost:9999", "http://127.0.0.1:80", "http://[::1]", "http://user:pass@example.com", "http://192.168.0.1", "http://169.254.169.254"} {
+		if validateSourceURL(raw) == nil {
+			t.Fatalf("allowed unsafe URL %q", raw)
+		}
+	}
+	if err := validateSourceURL("https://example.com/guide"); err != nil {
+		t.Fatalf("rejected public URL: %v", err)
+	}
+}
+
+func TestScrapeEscalatesToEnhancedOnBlockedBody(t *testing.T) {
+	originalClient := http.DefaultClient
+	t.Cleanup(func() { http.DefaultClient = originalClient })
+	var proxies []string
+	http.DefaultClient = &http.Client{Transport: roundTripFunc(func(req *http.Request) (*http.Response, error) {
+		if req.Method == http.MethodPost && req.URL.String() == "https://api.firecrawl.dev/v2/scrape" {
+			body, _ := io.ReadAll(req.Body)
+			if strings.Contains(string(body), `"proxy":"enhanced"`) {
+				proxies = append(proxies, "enhanced")
+				if !strings.Contains(string(body), `"maxAge":0`) {
+					t.Errorf("enhanced retry must bypass blocked cached results: %s", body)
+				}
+				return testHTTPResponse(req, http.StatusOK, `{"success":true,"data":{"metadata":{"title":"Article"},"markdown":"# Article\n\nLonger useful body from enhanced proxy."}}`, nil), nil
+			}
+			proxies = append(proxies, "auto")
+			if !strings.Contains(string(body), `"maxAge":1800000`) {
+				t.Errorf("Firecrawl cache age not 30 minutes: %s", body)
+			}
+			return testHTTPResponse(req, http.StatusOK, `{"success":true,"data":{"markdown":"Just a moment... checking your browser"}}`, nil), nil
+		}
+		return testHTTPResponse(req, http.StatusNotFound, "", nil), nil
+	})}
+	t.Setenv("FIRECRAWL_API_KEY", "test")
+	_, markdown, err := scrapeLinkWithFirecrawl("https://example.com/article")
+	if err != nil || !strings.Contains(markdown, "Longer useful body") {
+		t.Fatalf("expected enhanced proxy content, got %q: %v", markdown, err)
+	}
+	if strings.Join(proxies, ",") != "auto,enhanced" {
+		t.Fatalf("unexpected proxy stages %v", proxies)
+	}
+}
+
+func TestScrapeEscalatesToBrowserWhenScrapeFails(t *testing.T) {
+	originalClient := http.DefaultClient
+	t.Cleanup(func() { http.DefaultClient = originalClient })
+	var calls []string
+	http.DefaultClient = &http.Client{Transport: roundTripFunc(func(req *http.Request) (*http.Response, error) {
+		calls = append(calls, req.Method+" "+req.URL.Path)
+		switch req.Method + " " + req.URL.Path {
+		case "POST /v2/scrape":
+			return testHTTPResponse(req, http.StatusOK, `{"success":false,"error":"blocked"}`, nil), nil
+		case "POST /v2/interact":
+			return testHTTPResponse(req, http.StatusOK, `{"success":true,"id":"abc123"}`, nil), nil
+		case "POST /v2/interact/abc123/execute":
+			body, _ := io.ReadAll(req.Body)
+			if !strings.Contains(string(body), "agent-browser eval") || strings.Contains(string(body), "agent-browser scrape") {
+				t.Errorf("missing browser extraction command: %s", body)
+			}
+			return testHTTPResponse(req, http.StatusOK, `{"success":true,"stdout":"\"Real page content from browser.\\nNew paragraph.\"","exitCode":0}`, nil), nil
+		case "DELETE /v2/interact/abc123":
+			return testHTTPResponse(req, http.StatusOK, `{}`, nil), nil
+		}
+		return testHTTPResponse(req, http.StatusNotFound, "", nil), nil
+	})}
+	t.Setenv("FIRECRAWL_API_KEY", "test")
+	_, markdown, err := scrapeLinkWithFirecrawl("https://example.com/dynamic")
+	if err != nil || !strings.Contains(markdown, "Real page content") || !strings.Contains(markdown, "browser.\nNew paragraph.") {
+		t.Fatalf("expected browser fallback content, got %q: %v", markdown, err)
+	}
+	if strings.Join(calls, ",") != "POST /v2/scrape,POST /v2/interact,POST /v2/interact/abc123/execute,DELETE /v2/interact/abc123" {
+		t.Fatalf("unexpected browser lifecycle: %v", calls)
+	}
+}

package/internal/app/github_test.go

@@ -759,11 +759,10 @@ func TestReadLinkUnsupportedGitHubFallsThroughToFirecrawlWithExistingSettings(t
 		t.Fatalf("unsupported GitHub route was swallowed: %q", out)
 	}
 	for key, want := range map[string]any{
-		"onlyMainContent":     true,
-		"skipTlsVerification": true,
-		"blockAds":            true,
-		"removeBase64Images":  true,
-		"maxAge":              float64(600000),
+		"onlyMainContent":    true,
+		"blockAds":           true,
+		"removeBase64Images": true,
+		"maxAge":             float64(1800000),
 	} {
 		if got := firecrawlPayload[key]; got != want {
 			t.Errorf("Firecrawl setting %s changed: got %#v want %#v", key, got, want)