webctx 0.2.1 → 0.2.2

package/AGENTS.md

@@ -4,7 +4,7 @@ Guidance for coding agents working in `webctx`.
 
 ## Purpose
 
-This repo contains the pure Go port of the `webctx` CLI.
+This repo contains the Go `webctx` CLI and its hosted Origo MCP transport, sharing one retrieval engine.
 
 It is no longer a generic starter template. Treat the current CLI behavior as the source of truth unless the user explicitly asks to change it.
 
@@ -14,6 +14,9 @@ It is no longer a generic starter template. Treat the current CLI behavior as th
 - `internal/app/app.go`: CLI parsing and top-level command routing.
 - `internal/app/tools.go`: search provider clients, ranking, formatting, and HTTP helpers.
 - `internal/app/scrape.go`: GitHub raw-content optimization, `.md` fetch path, Firecrawl queue, and env loading.
+- `pkg/retrieval/`: public adapter to `internal/app` for Vercel's Go packaging restrictions.
+- `pkg/origo/`: authenticated Streamable HTTP MCP with exactly `read_link` and `map_site`.
+- `api/mcp.go`: Go Function entrypoint for independent Origo deployment.
 - `internal/app/app_test.go`: unit tests for CLI behavior and core helpers.
 - `bin/webctx.js`: npm shim that invokes the packaged native binary.
 - `scripts/postinstall.js`: downloads release binary on install, falls back to `go build`.
@@ -51,7 +54,7 @@ Preserve these commands unless the user explicitly asks to change them:
 Behavioral expectations:
 
 - `search` combines Brave, Tavily, and Exa results, then re-ranks them with duplicate-aware scoring.
-- `read-link` keeps the current GitHub raw-content fast path, `.md` fast path, and Firecrawl fallback settings.
+- `read-link` keeps the current GitHub raw-content and `.md` fast paths. Firecrawl uses its built-in 30-minute cache, automatic and enhanced proxies, then browser extraction as a final fallback.
 - `map-site` keeps the current Firecrawl map request settings.
 - The CLI should remain agent-friendly and emit plain markdown/text output.
 
@@ -81,7 +84,9 @@ Release binaries should embed the tagged version into `internal/buildinfo.Versio
 
 - Prefer additive changes and keep the CLI output stable.
 - Do not silently change Firecrawl request settings unless the user explicitly wants behavioral changes.
-- Do not reintroduce MCP/server code unless requested; this repo is intentionally CLI-only.
+- Do not add search to Origo's MCP tool surface; only `read_link` and `map_site` are registered.
+- Keep `origo-api` deployment separate from the existing WebCTX docs Vercel project and tag-driven CLI releases.
+- Never introduce Redis, custom KV, or another caching service for Origo: the user explicitly chose Firecrawl's own 30-minute cache.
 
 ## Changelog Guidelines
 

package/CONTRIBUTORS.md

@@ -2,6 +2,8 @@
 
 Maintainer notes for `webctx`.
 
+Origo is the separately deployed MCP transport in this same repository. The CLI, MCP, and ZueDocs share source code but have independent distribution pipelines.
+
 ## Prerequisites
 
 - Go `1.26+`
@@ -105,12 +107,22 @@ npm whoami
 
 - `cmd/webctx/main.go`: CLI entrypoint
 - `internal/app/`: CLI parsing, search, ranking, scrape, env loading, and Firecrawl queue logic
+- `pkg/retrieval/`: small public adapter to the shared read-link / site-map engine
+- `pkg/origo/`: authenticated, stateless MCP server with exactly two tools
+- `api/mcp.go`: Vercel Go Function entrypoint for Origo
+- `vercel.mjs`: project-specific Vercel settings for docs and Origo
 - `internal/buildinfo/`: build-time version plumbing for `--version`
 - `bin/webctx.js`: npm shim that invokes the packaged native binary
 - `scripts/postinstall.js`: downloads the release binary on install and falls back to local `go build`
 - `.github/workflows/release.yml`: tag-driven release pipeline
 - `AGENTS.md`: guidance for coding agents
 
+## Origo deployment
+
+The `origo-api` Vercel project is linked to this GitHub repo and uses `ORIGO_DEPLOYMENT=1` to select its Go Function configuration. It needs `ORIGO_API_KEY` and `FIRECRAWL_API_KEY` production environment variables; optionally add `GH_TOKEN` for native authenticated GitHub reads. The docs project remains `webctx-docs` and never receives the Origo connection key.
+
+Both projects deploy automatically from `main`. `scripts/should-build.mjs` gates docs and API builds independently, while `v*` GitHub tags still drive the CLI release workflow.
+
 ## Notes on package naming
 
 `webctx` is already configured. If you ever rename or move the package, update all of the following together:

package/README.md

@@ -12,6 +12,14 @@ webctx map-site <url>
 
 The output is plain text or markdown, so it is easy to hand to ChatGPT, Codex, Claude Code, a shell script, or another tool.
 
+## Origo: hosted MCP access
+
+**Origo** is the MCP transport over the same WebCTX retrieval engine. Connect an MCP client to `https://api.origo.ashray.xyz/mcp?key=<your-private-key>` to access exactly two tools: `read_link` and `map_site`. Origo does not expose `search`; the CLI continues to provide all three commands.
+
+Both surfaces share native GitHub/Markdown readers and Firecrawl fallback behavior. Firecrawl's built-in cache is limited to **30 minutes**. Difficult pages automatically escalate through Firecrawl's proxy fallback and, if necessary, a temporary Browser Sandbox session. There is no Redis or additional cache database.
+
+Origo deploys independently through the `origo-api` Vercel project on pushes to `main`. CLI releases and npm publishing are unchanged. See the [Origo MCP guide](https://webctx.ashray.xyz/docs/origo) for architecture, setup, and security notes.
+
 ## Install
 
 ```bash

package/go.mod

@@ -1,3 +1,16 @@
 module github.com/amxv/webctx
 
 go 1.26
+
+require github.com/modelcontextprotocol/go-sdk v1.8.0
+
+require (
+	github.com/google/jsonschema-go v0.4.3 // indirect
+	github.com/segmentio/asm v1.1.3 // indirect
+	github.com/segmentio/encoding v0.5.4 // indirect
+	github.com/yosida95/uritemplate/v3 v3.0.2 // indirect
+	golang.org/x/oauth2 v0.35.0 // indirect
+	golang.org/x/sync v0.20.0 // indirect
+	golang.org/x/sys v0.41.0 // indirect
+	golang.org/x/time v0.15.0 // indirect
+)

package/go.sum

@@ -0,0 +1,24 @@
+github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY=
+github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
+github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
+github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
+github.com/google/jsonschema-go v0.4.3 h1:/DBOLZTfDow7pe2GmaJNhltueGTtDKICi8V8p+DQPd0=
+github.com/google/jsonschema-go v0.4.3/go.mod h1:r5quNTdLOYEz95Ru18zA0ydNbBuYoo9tgaYcxEYhJVE=
+github.com/modelcontextprotocol/go-sdk v1.8.0 h1:KIvahhYqwtbeniWVPs3TcXEA7b8jEtwfBpOTAI+Urx4=
+github.com/modelcontextprotocol/go-sdk v1.8.0/go.mod h1:dL7u98E/zjJTGzEq+j30jQ8K2k1mb6LeAH4inEcSGts=
+github.com/segmentio/asm v1.1.3 h1:WM03sfUOENvvKexOLp+pCqgb/WDjsi7EK8gIsICtzhc=
+github.com/segmentio/asm v1.1.3/go.mod h1:Ld3L4ZXGNcSLRg4JBsZ3//1+f/TjYl0Mzen/DQy1EJg=
+github.com/segmentio/encoding v0.5.4 h1:OW1VRern8Nw6ITAtwSZ7Idrl3MXCFwXHPgqESYfvNt0=
+github.com/segmentio/encoding v0.5.4/go.mod h1:HS1ZKa3kSN32ZHVZ7ZLPLXWvOVIiZtyJnO1gPH1sKt0=
+github.com/yosida95/uritemplate/v3 v3.0.2 h1:Ed3Oyj9yrmi9087+NczuL5BwkIc4wvTb5zIM+UJPGz4=
+github.com/yosida95/uritemplate/v3 v3.0.2/go.mod h1:ILOh0sOhIJR3+L/8afwt/kE++YT040gmv5BQTMR2HP4=
+golang.org/x/oauth2 v0.35.0 h1:Mv2mzuHuZuY2+bkyWXIHMfhNdJAdwW3FuWeCPYN5GVQ=
+golang.org/x/oauth2 v0.35.0/go.mod h1:lzm5WQJQwKZ3nwavOZ3IS5Aulzxi68dUSgRHujetwEA=
+golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
+golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
+golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
+golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
+golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
+golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
+golang.org/x/tools v0.42.0 h1:uNgphsn75Tdz5Ji2q36v/nsFSfR/9BRFvqhGBaJGd5k=
+golang.org/x/tools v0.42.0/go.mod h1:Ma6lCIwGZvHK6XtgbswSoWroEkhugApmsXyrUmBhfr0=

package/internal/app/firecrawl_fallback.go

@@ -0,0 +1,163 @@
+package app
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/url"
+	"strconv"
+	"strings"
+	"time"
+)
+
+type scrapedPage struct {
+	Title    string
+	Markdown string
+}
+
+func usableScrapedMarkdown(markdown string) bool {
+	text := strings.ToLower(strings.TrimSpace(markdown))
+	if len(text) < 10 {
+		return false
+	}
+	for _, marker := range []string{
+		"just a moment...", "checking your browser", "verify you are human",
+		"enable javascript and cookies", "access denied | cloudflare",
+		"attention required! | cloudflare", "no content extracted",
+	} {
+		if strings.Contains(text, marker) && len(text) < 2000 {
+			return false
+		}
+	}
+	return true
+}
+
+func firecrawlScrape(rawURL, apiKey, proxy string) (scrapedPage, error) {
+	requestBody := map[string]any{
+		"url":                rawURL,
+		"formats":            []string{"markdown"},
+		"onlyMainContent":    true,
+		"blockAds":           true,
+		"removeBase64Images": true,
+		"maxAge":             30 * 60 * 1000, // Firecrawl's built-in cache, 30 minutes.
+		"excludeTags":        []string{"script", "style", "meta", "noscript", "svg", "img", "nav", "footer", "header", "aside", ".advertisement", "#ad"},
+	}
+	if proxy != "" {
+		requestBody["proxy"] = proxy
+		// A blocked cached result must not mask an actual enhanced retry.
+		requestBody["maxAge"] = 0
+	}
+	if strings.HasSuffix(strings.ToLower(rawURL), ".pdf") {
+		requestBody["parsers"] = []string{"pdf"}
+	}
+	data, err := getFirecrawlQueue().enqueue(rawURL, func() (map[string]any, error) {
+		ctx, cancel := context.WithTimeout(context.Background(), 32*time.Second)
+		defer cancel()
+		body, err := doJSONRequest(ctx, http.MethodPost, "https://api.firecrawl.dev/v2/scrape", map[string]string{
+			"Authorization": "Bearer " + apiKey,
+			"Content-Type":  "application/json",
+		}, requestBody)
+		if err != nil {
+			return nil, err
+		}
+		var parsed map[string]any
+		if err := json.Unmarshal(body, &parsed); err != nil {
+			return nil, fmt.Errorf("decode Firecrawl scrape: %w", err)
+		}
+		if success, _ := parsed["success"].(bool); !success {
+			return nil, fmt.Errorf("Firecrawl scrape failed: %v", parsed["error"])
+		}
+		return parsed, nil
+	})
+	if err != nil {
+		return scrapedPage{}, err
+	}
+	pageData, _ := data["data"].(map[string]any)
+	metadata, _ := pageData["metadata"].(map[string]any)
+	page := scrapedPage{
+		Title:    stringValue(metadata["title"]),
+		Markdown: stringValue(pageData["markdown"]),
+	}
+	if status, ok := metadata["statusCode"].(float64); ok && status >= 400 {
+		return page, fmt.Errorf("source responded with HTTP %d", int(status))
+	}
+	return page, nil
+}
+
+func firecrawlJSON(ctx context.Context, method, endpoint, apiKey string, data map[string]any) (map[string]any, error) {
+	body, err := doJSONRequest(ctx, method, endpoint, map[string]string{
+		"Authorization": "Bearer " + apiKey,
+		"Content-Type":  "application/json",
+	}, data)
+	if err != nil {
+		return nil, err
+	}
+	var parsed map[string]any
+	if err := json.Unmarshal(body, &parsed); err != nil {
+		return nil, err
+	}
+	if success, ok := parsed["success"].(bool); ok && !success {
+		return nil, fmt.Errorf("Firecrawl browser: %v", parsed["error"])
+	}
+	return parsed, nil
+}
+
+// quoteShellURL escapes a validated URL for the Firecrawl sandbox's bash mode.
+// This invokes only the installed agent-browser CLI, never model-written code.
+func quoteShellURL(rawURL string) string {
+	return "'" + strings.ReplaceAll(rawURL, "'", "'\\''") + "'"
+}
+
+func firecrawlBrowserFallback(rawURL, apiKey string) (string, string, error) {
+	ctx, cancel := context.WithTimeout(context.Background(), 28*time.Second)
+	defer cancel()
+	// Failed scrape responses do not reliably include a scrape-bound browser ID.
+	// Open a disposable session with the standalone browser API instead.
+	session, err := firecrawlJSON(ctx, http.MethodPost, "https://api.firecrawl.dev/v2/interact", apiKey, map[string]any{
+		"ttl": 60, "activityTtl": 30,
+	})
+	if err != nil {
+		return "", "", err
+	}
+	sessionID := stringValue(session["id"])
+	if sessionID == "" {
+		return "", "", fmt.Errorf("Firecrawl browser returned no session ID")
+	}
+	endpoint := "https://api.firecrawl.dev/v2/interact/" + url.PathEscape(sessionID)
+	defer stopFirecrawlBrowser(endpoint, apiKey)
+	result, err := firecrawlJSON(ctx, http.MethodPost, endpoint+"/execute", apiKey, map[string]any{
+		// The sandbox's agent-browser does not expose a "scrape" subcommand
+		// even though Firecrawl's examples use it. DOM innerText is supported.
+		// Suppress navigation's banner so stdout contains only the evaluated text.
+		"code":     "agent-browser open " + quoteShellURL(rawURL) + " >/dev/null && agent-browser eval 'document.body.innerText'",
+		"language": "bash",
+		"timeout":  20,
+	})
+	if err != nil {
+		return "", "", err
+	}
+	if code, ok := result["exitCode"].(float64); ok && code != 0 {
+		return "", "", fmt.Errorf("browser command exited with %d", int(code))
+	}
+	text := stringValue(result["stdout"])
+	if text == "" {
+		text = stringValue(result["output"])
+	}
+	if text == "" {
+		text = stringValue(result["result"])
+	}
+	// agent-browser eval serializes JS strings as JSON with surrounding quotes.
+	// Decode those escapes rather than handing the model a JSON string literal.
+	if parsed, err := strconv.Unquote(strings.TrimSpace(text)); err == nil {
+		text = parsed
+	}
+	return "Browser extraction", text, nil
+}
+
+func stopFirecrawlBrowser(endpoint, apiKey string) {
+	ctx, cancel := context.WithTimeout(context.Background(), 4*time.Second)
+	defer cancel()
+	// Best-effort cleanup regardless of whether extraction succeeded.
+	_, _ = firecrawlJSON(ctx, http.MethodDelete, endpoint, apiKey, nil)
+}

package/internal/app/firecrawl_fallback_test.go

@@ -0,0 +1,83 @@
+package app
+
+import (
+	"io"
+	"net/http"
+	"strings"
+	"testing"
+)
+
+func TestSourceURLSafety(t *testing.T) {
+	for _, raw := range []string{"file:///etc/passwd", "http://localhost:9999", "http://127.0.0.1:80", "http://[::1]", "http://user:pass@example.com", "http://192.168.0.1", "http://169.254.169.254"} {
+		if validateSourceURL(raw) == nil {
+			t.Fatalf("allowed unsafe URL %q", raw)
+		}
+	}
+	if err := validateSourceURL("https://example.com/guide"); err != nil {
+		t.Fatalf("rejected public URL: %v", err)
+	}
+}
+
+func TestScrapeEscalatesToEnhancedOnBlockedBody(t *testing.T) {
+	originalClient := http.DefaultClient
+	t.Cleanup(func() { http.DefaultClient = originalClient })
+	var proxies []string
+	http.DefaultClient = &http.Client{Transport: roundTripFunc(func(req *http.Request) (*http.Response, error) {
+		if req.Method == http.MethodPost && req.URL.String() == "https://api.firecrawl.dev/v2/scrape" {
+			body, _ := io.ReadAll(req.Body)
+			if strings.Contains(string(body), `"proxy":"enhanced"`) {
+				proxies = append(proxies, "enhanced")
+				if !strings.Contains(string(body), `"maxAge":0`) {
+					t.Errorf("enhanced retry must bypass blocked cached results: %s", body)
+				}
+				return testHTTPResponse(req, http.StatusOK, `{"success":true,"data":{"metadata":{"title":"Article"},"markdown":"# Article\n\nLonger useful body from enhanced proxy."}}`, nil), nil
+			}
+			proxies = append(proxies, "auto")
+			if !strings.Contains(string(body), `"maxAge":1800000`) {
+				t.Errorf("Firecrawl cache age not 30 minutes: %s", body)
+			}
+			return testHTTPResponse(req, http.StatusOK, `{"success":true,"data":{"markdown":"Just a moment... checking your browser"}}`, nil), nil
+		}
+		return testHTTPResponse(req, http.StatusNotFound, "", nil), nil
+	})}
+	t.Setenv("FIRECRAWL_API_KEY", "test")
+	_, markdown, err := scrapeLinkWithFirecrawl("https://example.com/article")
+	if err != nil || !strings.Contains(markdown, "Longer useful body") {
+		t.Fatalf("expected enhanced proxy content, got %q: %v", markdown, err)
+	}
+	if strings.Join(proxies, ",") != "auto,enhanced" {
+		t.Fatalf("unexpected proxy stages %v", proxies)
+	}
+}
+
+func TestScrapeEscalatesToBrowserWhenScrapeFails(t *testing.T) {
+	originalClient := http.DefaultClient
+	t.Cleanup(func() { http.DefaultClient = originalClient })
+	var calls []string
+	http.DefaultClient = &http.Client{Transport: roundTripFunc(func(req *http.Request) (*http.Response, error) {
+		calls = append(calls, req.Method+" "+req.URL.Path)
+		switch req.Method + " " + req.URL.Path {
+		case "POST /v2/scrape":
+			return testHTTPResponse(req, http.StatusOK, `{"success":false,"error":"blocked"}`, nil), nil
+		case "POST /v2/interact":
+			return testHTTPResponse(req, http.StatusOK, `{"success":true,"id":"abc123"}`, nil), nil
+		case "POST /v2/interact/abc123/execute":
+			body, _ := io.ReadAll(req.Body)
+			if !strings.Contains(string(body), "agent-browser eval") || strings.Contains(string(body), "agent-browser scrape") {
+				t.Errorf("missing browser extraction command: %s", body)
+			}
+			return testHTTPResponse(req, http.StatusOK, `{"success":true,"stdout":"\"Real page content from browser.\\nNew paragraph.\"","exitCode":0}`, nil), nil
+		case "DELETE /v2/interact/abc123":
+			return testHTTPResponse(req, http.StatusOK, `{}`, nil), nil
+		}
+		return testHTTPResponse(req, http.StatusNotFound, "", nil), nil
+	})}
+	t.Setenv("FIRECRAWL_API_KEY", "test")
+	_, markdown, err := scrapeLinkWithFirecrawl("https://example.com/dynamic")
+	if err != nil || !strings.Contains(markdown, "Real page content") || !strings.Contains(markdown, "browser.\nNew paragraph.") {
+		t.Fatalf("expected browser fallback content, got %q: %v", markdown, err)
+	}
+	if strings.Join(calls, ",") != "POST /v2/scrape,POST /v2/interact,POST /v2/interact/abc123/execute,DELETE /v2/interact/abc123" {
+		t.Fatalf("unexpected browser lifecycle: %v", calls)
+	}
+}

package/internal/app/github_test.go

@@ -759,11 +759,10 @@ func TestReadLinkUnsupportedGitHubFallsThroughToFirecrawlWithExistingSettings(t
 		t.Fatalf("unsupported GitHub route was swallowed: %q", out)
 	}
 	for key, want := range map[string]any{
-		"onlyMainContent":     true,
-		"skipTlsVerification": true,
-		"blockAds":            true,
-		"removeBase64Images":  true,
-		"maxAge":              float64(600000),
+		"onlyMainContent":    true,
+		"blockAds":           true,
+		"removeBase64Images": true,
+		"maxAge":             float64(1800000),
 	} {
 		if got := firecrawlPayload[key]; got != want {
 			t.Errorf("Firecrawl setting %s changed: got %#v want %#v", key, got, want)

package/internal/app/source_url.go

@@ -0,0 +1,25 @@
+package app
+
+import (
+	"fmt"
+	"net"
+	"net/url"
+	"strings"
+)
+
+// Reject non-public URL destinations before calling native readers or
+// hosted scraping providers. Credentials embedded in URLs are never accepted.
+func validateSourceURL(raw string) error {
+	u, err := url.Parse(raw)
+	if err != nil || u == nil || (u.Scheme != "http" && u.Scheme != "https") || u.Hostname() == "" || u.User != nil {
+		return fmt.Errorf("readable URL must be an absolute HTTP(S) URL without credentials")
+	}
+	host := strings.ToLower(strings.TrimSuffix(u.Hostname(), "."))
+	if host == "localhost" || strings.HasSuffix(host, ".localhost") || strings.HasSuffix(host, ".local") || strings.HasSuffix(host, ".internal") {
+		return fmt.Errorf("private/internal host is not a readable public URL")
+	}
+	if ip := net.ParseIP(host); ip != nil && (!ip.IsGlobalUnicast() || ip.IsPrivate() || ip.IsLoopback()) {
+		return fmt.Errorf("private or reserved IP address is not a readable public URL")
+	}
+	return nil
+}

package/internal/app/tools.go

@@ -131,6 +131,9 @@ func Search(params SearchParams) (string, error) {
 }
 
 func ReadLink(rawURL string) (string, error) {
+	if err := validateSourceURL(rawURL); err != nil {
+		return "", err
+	}
 	native := readGitHubNative(rawURL)
 	switch native.Outcome {
 	case GitHubNativeSuccess:
@@ -193,55 +196,38 @@ func scrapeLinkWithFirecrawl(rawURL string) (string, string, error) {
 	if apiKey == "" {
 		return "", "", missingCredentialsError("Error reading web page", []string{"FIRECRAWL_API_KEY"}, "for non-.md URLs")
 	}
-
-	requestBody := map[string]any{
-		"url":                 rawURL,
-		"formats":             []string{"markdown"},
-		"onlyMainContent":     true,
-		"skipTlsVerification": true,
-		"blockAds":            true,
-		"removeBase64Images":  true,
-		"maxAge":              600000,
-		"excludeTags":         []string{"script", "style", "meta", "noscript", "svg", "img", "nav", "footer", "header", "aside", ".advertisement", "#ad"},
-	}
-	if strings.HasSuffix(strings.ToLower(rawURL), ".pdf") {
-		requestBody["parsers"] = []string{"pdf"}
-	}
-
-	data, err := getFirecrawlQueue().enqueue(rawURL, func() (map[string]any, error) {
-		ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
-		defer cancel()
-		body, err := doJSONRequest(ctx, http.MethodPost, "https://api.firecrawl.dev/v2/scrape", map[string]string{
-			"Authorization": "Bearer " + apiKey,
-			"Content-Type":  "application/json",
-		}, requestBody)
-		if err != nil {
-			return nil, err
-		}
-		var parsed map[string]any
-		if err := json.Unmarshal(body, &parsed); err != nil {
-			return nil, err
+	// Firecrawl's default auto proxy already tries basic then enhanced when the
+	// transport fails. A second explicit enhanced attempt is only useful when a
+	// scrape responds successfully but returns a challenge/empty document.
+	first, err := firecrawlScrape(rawURL, apiKey, "")
+	if err == nil && usableScrapedMarkdown(first.Markdown) {
+		return first.Title, first.Markdown, nil
+	}
+	lastError := err
+	if err == nil {
+		lastError = fmt.Errorf("Firecrawl returned empty or blocked page content")
+		second, enhancedErr := firecrawlScrape(rawURL, apiKey, "enhanced")
+		if enhancedErr == nil && usableScrapedMarkdown(second.Markdown) {
+			return second.Title, second.Markdown, nil
 		}
-		if success, _ := parsed["success"].(bool); !success {
-			return nil, fmt.Errorf("Scraping failed for %s: %v", rawURL, parsed["error"])
+		if enhancedErr != nil {
+			lastError = enhancedErr
 		}
-		return parsed, nil
-	})
-	if err != nil {
-		return "", "", fmt.Errorf("Error reading web page: %v", err)
 	}
-
-	dataMap, _ := data["data"].(map[string]any)
-	metadata, _ := dataMap["metadata"].(map[string]any)
-	title, _ := metadata["title"].(string)
-	markdown, _ := dataMap["markdown"].(string)
-	if markdown == "" {
-		markdown = "No content extracted"
+	title, markdown, browserErr := firecrawlBrowserFallback(rawURL, apiKey)
+	if browserErr == nil && usableScrapedMarkdown(markdown) {
+		return title, markdown, nil
+	}
+	if browserErr == nil {
+		browserErr = fmt.Errorf("browser produced no usable content")
 	}
-	return title, markdown, nil
+	return "", "", fmt.Errorf("Error reading web page: scrape: %v; browser fallback: %w", lastError, browserErr)
 }
 
 func MapSite(rawURL string) (string, error) {
+	if err := validateSourceURL(rawURL); err != nil {
+		return "", err
+	}
 	apiKey := strings.TrimSpace(os.Getenv("FIRECRAWL_API_KEY"))
 	if apiKey == "" {
 		return "", missingCredentialsError("Error mapping website", []string{"FIRECRAWL_API_KEY"}, "")

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "webctx",
-  "version": "0.2.1",
+  "version": "0.2.2",
   "description": "Pure Go web search and browsing CLI using Brave, Tavily, Exa, and Firecrawl",
   "license": "MIT",
   "author": "amxv",
@@ -24,6 +24,8 @@
     "cmd",
     "internal",
     "go.mod",
+    "go.sum",
+    "pkg",
     "README.md",
     "docs",
     "LICENSE",

package/pkg/origo/server.go

@@ -0,0 +1,79 @@
+// Package origo provides the independently deployed MCP transport for the
+// shared WebCTX retrieval engine. Search is intentionally not exposed.
+package origo
+
+import (
+	"context"
+	"crypto/sha256"
+	"crypto/subtle"
+	"net/http"
+	"os"
+	"strings"
+
+	"github.com/amxv/webctx/pkg/retrieval"
+	"github.com/modelcontextprotocol/go-sdk/mcp"
+)
+
+type linkInput struct {
+	URL string `json:"url" jsonschema:"Absolute HTTP(S) URL to read as source-grounded Markdown"`
+}
+
+func makeServer() *mcp.Server {
+	s := mcp.NewServer(&mcp.Implementation{Name: "origo", Version: "0.1.0"}, nil)
+	mcp.AddTool(s, &mcp.Tool{
+		Name:        "read_link",
+		Title:       "Read link",
+		Description: "Read a specific web page as clean Markdown, preferring native GitHub or site Markdown and using Firecrawl proxies/browser fallback when needed. Does not search the web.",
+	}, func(_ context.Context, _ *mcp.CallToolRequest, input linkInput) (*mcp.CallToolResult, any, error) {
+		result, err := retrieval.ReadLink(input.URL)
+		if err != nil {
+			return nil, nil, err
+		}
+		return &mcp.CallToolResult{Content: []mcp.Content{&mcp.TextContent{Text: result}}}, nil, nil
+	})
+	mcp.AddTool(s, &mcp.Tool{
+		Name:        "map_site",
+		Title:       "Map site",
+		Description: "Discover URLs on a site using Firecrawl's sitemap-based map API. Does not perform web search.",
+	}, func(_ context.Context, _ *mcp.CallToolRequest, input linkInput) (*mcp.CallToolResult, any, error) {
+		result, err := retrieval.MapSite(input.URL)
+		if err != nil {
+			return nil, nil, err
+		}
+		return &mcp.CallToolResult{Content: []mcp.Content{&mcp.TextContent{Text: result}}}, nil, nil
+	})
+	return s
+}
+
+var transport = mcp.NewStreamableHTTPHandler(func(*http.Request) *mcp.Server {
+	return makeServer()
+}, &mcp.StreamableHTTPOptions{Stateless: true, JSONResponse: true})
+
+func Handler() http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Cache-Control", "no-store")
+		w.Header().Set("Referrer-Policy", "no-referrer")
+		w.Header().Set("X-Content-Type-Options", "nosniff")
+		w.Header().Set("X-Robots-Tag", "noindex")
+		secret := strings.TrimSpace(os.Getenv("ORIGO_API_KEY"))
+		if secret == "" {
+			http.Error(w, "MCP endpoint not configured", http.StatusServiceUnavailable)
+			return
+		}
+		keys := r.URL.Query()["key"]
+		if len(keys) != 1 || !validKey(secret, keys[0]) {
+			http.Error(w, "Unauthorized", http.StatusUnauthorized)
+			return
+		}
+		transport.ServeHTTP(w, r)
+	})
+}
+
+func validKey(expected, supplied string) bool {
+	if supplied == "" {
+		return false
+	}
+	want := sha256.Sum256([]byte(expected))
+	got := sha256.Sum256([]byte(supplied))
+	return subtle.ConstantTimeCompare(want[:], got[:]) == 1
+}

package/pkg/origo/server_test.go

@@ -0,0 +1,63 @@
+package origo
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+)
+
+func request(t *testing.T, key, body string) *httptest.ResponseRecorder {
+	t.Helper()
+	path := "/mcp"
+	if key != "" {
+		path += "?key=" + key
+	}
+	req := httptest.NewRequest(http.MethodPost, path, strings.NewReader(body))
+	req.Header.Set("Content-Type", "application/json")
+	req.Header.Set("Accept", "application/json, text/event-stream")
+	req.Header.Set("Mcp-Protocol-Version", "2025-11-25")
+	response := httptest.NewRecorder()
+	Handler().ServeHTTP(response, req)
+	return response
+}
+
+func TestServerRequiresConfiguredKey(t *testing.T) {
+	t.Setenv("ORIGO_API_KEY", "")
+	response := request(t, "somekey", `{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}`)
+	if response.Code != http.StatusServiceUnavailable {
+		t.Fatalf("without a configured key: expected 503, got %d", response.Code)
+	}
+}
+
+func TestServerAuthentication(t *testing.T) {
+	t.Setenv("ORIGO_API_KEY", "secret")
+	for _, key := range []string{"", "wrong"} {
+		response := request(t, key, `{}`)
+		if response.Code != http.StatusUnauthorized {
+			t.Fatalf("key %q: expected 401, got %d", key, response.Code)
+		}
+		if response.Header().Get("Cache-Control") != "no-store" {
+			t.Fatalf("no-store header missing")
+		}
+	}
+	response := request(t, "secret", `{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"test","version":"1"}}}`)
+	if response.Code != http.StatusOK || !strings.Contains(response.Body.String(), `"name":"origo"`) {
+		t.Fatalf("initialize failed: %d %s", response.Code, response.Body.String())
+	}
+}
+
+func TestExactlyTwoToolsAndNoSearch(t *testing.T) {
+	t.Setenv("ORIGO_API_KEY", "secret")
+	response := request(t, "secret", `{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}`)
+	if response.Code != http.StatusOK {
+		t.Fatalf("tools/list failed: %d %s", response.Code, response.Body.String())
+	}
+	text := response.Body.String()
+	if !strings.Contains(text, `"name":"read_link"`) || !strings.Contains(text, `"name":"map_site"`) || strings.Contains(text, `"name":"search"`) {
+		t.Fatalf("unexpected tools: %s", text)
+	}
+	if count := strings.Count(text, `"name":`); count != 2 {
+		t.Fatalf("expected exactly 2 tools, got %d: %s", count, text)
+	}
+}

package/pkg/retrieval/retrieval.go

@@ -0,0 +1,11 @@
+// Package retrieval exposes WebCTX's shared retrieval engine to non-CLI
+// transports. The CLI remains backward-compatible while Origo MCP can call
+// these functions without importing Go internal packages from Vercel's
+// generated handler package.
+package retrieval
+
+import "github.com/amxv/webctx/internal/app"
+
+func ReadLink(url string) (string, error) { return app.ReadLink(url) }
+
+func MapSite(url string) (string, error) { return app.MapSite(url) }