webctx 0.2.2 → 0.2.4

package/AGENTS.md

@@ -74,7 +74,7 @@ Behavioral expectations:
 
 Release pipeline triggers on `v*` tags and expects:
 
-- `NPM_TOKEN` GitHub secret present.
+- The `NPM_TOKEN` GitHub Actions secret must contain a valid npm access token with publish rights for `webctx`; publishing uses `NODE_AUTH_TOKEN` (not OIDC trusted publishing).
 - npm package name in `package.json` is publishable under your account/org.
 - repository URL matches the release origin used by `scripts/postinstall.js`.
 

package/CONTRIBUTORS.md

@@ -80,27 +80,16 @@ make release-tag VERSION=x.y.z
 - GitHub release publish
 - npm publish
 
-## Required GitHub secret
+## Automatic npm access-token publishing
 
-- `NPM_TOKEN`: npm automation token with publish rights for your package.
+This project publishes npm releases using the repository's GitHub Actions secret `NPM_TOKEN`. Set it to a valid npm access token with publish access to the `webctx` package and appropriate npm two-factor-authentication settings for automated publishing. GitHub Actions supplies it as `NODE_AUTH_TOKEN` during the `release.yml` workflow's npm step.
 
-Set via GitHub CLI:
+If a token expires, rotate it in npm and update `NPM_TOKEN` using `gh secret set NPM_TOKEN --repo amxv/webctx`. Never commit tokens or print them in job logs. Trusted publishing is not used.
 
-```bash
-gh secret set NPM_TOKEN --repo amxv/webctx
-```
-
-## npm token setup
-
-Create token at npm:
-
-- Profile -> Access Tokens -> Create New Token
-- Use an automation/granular token scoped to required package/org
-
-Validate auth locally:
+If a release's GitHub binaries were successfully published while npm authentication failed, dispatch `npm-publish-retry.yml` with the existing version. Do not create or move the release tag just to retry npm publishing:
 
 ```bash
-npm whoami
+gh workflow run npm-publish-retry.yml --repo amxv/webctx --ref main -f version=0.2.2
 ```
 
 ## Project layout

package/README.md

@@ -10,14 +10,21 @@ webctx read-link <url>
 webctx map-site <url>
 ```
 
-The output is plain text or markdown, so it is easy to hand to ChatGPT, Codex, Claude Code, a shell script, or another tool.
+The output is plain text or markdown, so it is easy to hand to ChatGPT, Codex, Claude Code, a shell script, or another tool. Search now retrieves relevant source excerpts automatically instead of returning only URLs. For a specific URL, add an optional question to explore related documentation and surface exact code and API details.
+
+```bash
+webctx search "Go MCP SDK read-only tool annotations"
+webctx read-link https://docs.example.com/api --question "How do authentication and pagination work?"
+```
 
 ## Origo: hosted MCP access
 
-**Origo** is the MCP transport over the same WebCTX retrieval engine. Connect an MCP client to `https://api.origo.ashray.xyz/mcp?key=<your-private-key>` to access exactly two tools: `read_link` and `map_site`. Origo does not expose `search`; the CLI continues to provide all three commands.
+**Origo** is the MCP transport over the same WebCTX retrieval engine. Connect an MCP client to `https://api.origo.ashray.xyz/mcp?key=<your-private-key>` to access exactly two tools: `read_link` and `map_site`. Origo does not expose `search`; the CLI continues to provide all three commands. `read_link` accepts an optional `question` to retrieve focused source context while keeping ordinary URL-only reads intact.
 
 Both surfaces share native GitHub/Markdown readers and Firecrawl fallback behavior. Firecrawl's built-in cache is limited to **30 minutes**. Difficult pages automatically escalate through Firecrawl's proxy fallback and, if necessary, a temporary Browser Sandbox session. There is no Redis or additional cache database.
 
+Source-aware reading also prefers raw JSON, YAML, and OpenAPI files when the URL identifies one; question-focused reads can follow up to two relevant pages on the same host, including links found in `llms.txt`, and inspect embedded structured JSON. Firecrawl Alexandria tool matching and search discovery are automatic when supported, but **no paid Alexandria provider tools are executed**. The original source URLs remain explicit so an agent can verify facts.
+
 Origo deploys independently through the `origo-api` Vercel project on pushes to `main`. CLI releases and npm publishing are unchanged. See the [Origo MCP guide](https://webctx.ashray.xyz/docs/origo) for architecture, setup, and security notes.
 
 ## Install
@@ -71,7 +78,7 @@ Recognized native GitHub auth, private/not-found, and rate-limit failures stay a
 
 ## Search
 
-Normal search asks Brave, Tavily, and Exa, removes duplicate URLs, and returns one useful list.
+Normal search asks Brave, Tavily, and Exa, removes duplicate URLs, automatically reads three promising sources, follows up to two relevant links, and returns excerpts plus the ranked list. Alexandria tool discovery is included when available, without paid execution.
 
 ```bash
 webctx search "next.js server components"

package/internal/app/alexandria.go

@@ -0,0 +1,102 @@
+package app
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"os"
+	"strings"
+	"time"
+)
+
+// formatAlexandriaTools describes discovered capabilities, not their data.
+// Discovery is free but actual provider execution is not, and is never
+// silently authorized by reading a link or searching the web.
+func formatAlexandriaTools(raw any) string {
+	items, ok := raw.([]any)
+	if !ok || len(items) == 0 {
+		return ""
+	}
+	parts := []string{"**Available structured-data capabilities (not executed):**"}
+	count := 0
+	for _, item := range items {
+		tool, ok := item.(map[string]any)
+		if !ok {
+			continue
+		}
+		provider := safeToolField(tool["provider"], 80)
+		capability := safeToolField(tool["capability"], 160)
+		if provider == "" || capability == "" {
+			continue
+		}
+		description := safeToolField(tool["description"], 180)
+		if description == "" {
+			description = safeToolField(tool["name"], 180)
+		}
+		line := fmt.Sprintf("- `%s/%s`", provider, capability)
+		if description != "" {
+			line += ": " + description
+		}
+		parts = append(parts, line)
+		count++
+		if count >= 3 {
+			break
+		}
+	}
+	if count == 0 {
+		return ""
+	}
+	return strings.Join(parts, "\n")
+}
+
+func safeToolField(value any, limit int) string {
+	s, ok := value.(string)
+	if !ok {
+		return ""
+	}
+	s = strings.Join(strings.Fields(s), " ")
+	s = strings.ReplaceAll(s, "`", "'")
+	if len(s) > limit {
+		s = s[:limit] + "…"
+	}
+	return s
+}
+
+func withSourceTools(markdown, toolSummary string) string {
+	if toolSummary == "" {
+		return markdown
+	}
+	return markdown + "\n\n---\n\n" + toolSummary + "\n\n*Matched through Firecrawl Alexandria. Discovery does not verify, license, or execute these providers.*"
+}
+
+func discoverAlexandria(query string) string {
+	apiKey := strings.TrimSpace(os.Getenv("FIRECRAWL_API_KEY"))
+	if apiKey == "" {
+		return ""
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), 12*time.Second)
+	defer cancel()
+	response, err := doJSONRequest(ctx, http.MethodPost, "https://api.firecrawl.dev/v2/search", map[string]string{
+		"Authorization": "Bearer " + apiKey,
+		"Content-Type":  "application/json",
+	}, map[string]any{
+		"query":      query,
+		"sources":    []string{"alexandria"},
+		"limit":      3,
+		"toolDetail": "compact",
+	})
+	if err != nil {
+		return "" // unavailable Alexandria is never a search failure
+	}
+	var result struct {
+		Success bool `json:"success"`
+		Data    struct {
+			Tools []any `json:"tools"`
+		} `json:"data"`
+	}
+	if json.Unmarshal(response, &result) != nil || !result.Success {
+		return ""
+	}
+	return formatAlexandriaTools(result.Data.Tools)
+}

package/internal/app/app.go

@@ -51,10 +51,10 @@ func Run(args []string, stdout, stderr io.Writer) int {
 	case "read-link":
 		if strings.TrimSpace(input) == "" {
 			_, _ = fmt.Fprintln(stderr, "Error: read-link requires a URL")
-			_, _ = fmt.Fprintln(stdout, "Usage: webctx read-link <url>")
+			_, _ = fmt.Fprintln(stdout, "Usage: webctx read-link <url> [--question 'what to find']")
 			return 1
 		}
-		text, err := ReadLink(input)
+		text, err := ReadLinkFocused(input, flags["question"])
 		if err != nil {
 			_, _ = fmt.Fprintln(stderr, err.Error())
 			return 1
@@ -86,7 +86,7 @@ func usageText() string {
 
 Usage:
   webctx search <query> [--exclude domain1,domain2] [--keyword phrase]
-  webctx read-link <url>
+  webctx read-link <url> [--question 'what to find']
   webctx map-site <url>
 
 Examples:
@@ -94,6 +94,7 @@ Examples:
   webctx search "react hooks" --exclude youtube.com,vimeo.com
   webctx search "drizzle orm" --keyword "migration guide"
   webctx read-link https://docs.example.com/guide
+  webctx read-link https://docs.example.com/api --question "How do auth and pagination work?"
   webctx map-site https://example.com`, version)
 }
 

package/internal/app/firecrawl_fallback.go

@@ -12,31 +12,20 @@ import (
 )
 
 type scrapedPage struct {
-	Title    string
-	Markdown string
+	Title       string
+	Markdown    string
+	ToolSummary string
 }
 
 func usableScrapedMarkdown(markdown string) bool {
-	text := strings.ToLower(strings.TrimSpace(markdown))
-	if len(text) < 10 {
-		return false
-	}
-	for _, marker := range []string{
-		"just a moment...", "checking your browser", "verify you are human",
-		"enable javascript and cookies", "access denied | cloudflare",
-		"attention required! | cloudflare", "no content extracted",
-	} {
-		if strings.Contains(text, marker) && len(text) < 2000 {
-			return false
-		}
-	}
-	return true
+	return usableSourceContent(markdown)
 }
 
 func firecrawlScrape(rawURL, apiKey, proxy string) (scrapedPage, error) {
 	requestBody := map[string]any{
 		"url":                rawURL,
 		"formats":            []string{"markdown"},
+		"domainTools":        true, // Alexandria discovery, not paid execution.
 		"onlyMainContent":    true,
 		"blockAds":           true,
 		"removeBase64Images": true,
@@ -58,6 +47,16 @@ func firecrawlScrape(rawURL, apiKey, proxy string) (scrapedPage, error) {
 			"Authorization": "Bearer " + apiKey,
 			"Content-Type":  "application/json",
 		}, requestBody)
+		// Alexandria is a separately enabled Firecrawl capability. A 403 from
+		// domain matching must never break ordinary page extraction for users that
+		// haven't enabled Alexandria or have zero-data-retention configured.
+		if err != nil && strings.Contains(err.Error(), "403") {
+			requestBody["domainTools"] = false
+			body, err = doJSONRequest(ctx, http.MethodPost, "https://api.firecrawl.dev/v2/scrape", map[string]string{
+				"Authorization": "Bearer " + apiKey,
+				"Content-Type":  "application/json",
+			}, requestBody)
+		}
 		if err != nil {
 			return nil, err
 		}
@@ -76,8 +75,9 @@ func firecrawlScrape(rawURL, apiKey, proxy string) (scrapedPage, error) {
 	pageData, _ := data["data"].(map[string]any)
 	metadata, _ := pageData["metadata"].(map[string]any)
 	page := scrapedPage{
-		Title:    stringValue(metadata["title"]),
-		Markdown: stringValue(pageData["markdown"]),
+		Title:       stringValue(metadata["title"]),
+		Markdown:    stringValue(pageData["markdown"]),
+		ToolSummary: formatAlexandriaTools(pageData["tools"]),
 	}
 	if status, ok := metadata["statusCode"].(float64); ok && status >= 400 {
 		return page, fmt.Errorf("source responded with HTTP %d", int(status))
@@ -152,6 +152,9 @@ func firecrawlBrowserFallback(rawURL, apiKey string) (string, string, error) {
 	if parsed, err := strconv.Unquote(strings.TrimSpace(text)); err == nil {
 		text = parsed
 	}
+	if reason := challengeReason(text); reason != "" {
+		return "", "", fmt.Errorf("browser fallback returned %s rather than documentation", reason)
+	}
 	return "Browser extraction", text, nil
 }
 

package/internal/app/firecrawl_fallback_test.go

@@ -1,6 +1,7 @@
 package app
 
 import (
+	"encoding/json"
 	"io"
 	"net/http"
 	"strings"
@@ -18,6 +19,43 @@ func TestSourceURLSafety(t *testing.T) {
 	}
 }
 
+func TestLongBrowserCAPTCHAFallbackFailsClearly(t *testing.T) {
+	original := http.DefaultClient
+	t.Cleanup(func() { http.DefaultClient = original })
+	challenge := "hCaptcha Please verify you are human. " +
+		strings.Repeat("French Deutsch Español English 日本語 中文 ", 150)
+	http.DefaultClient = &http.Client{Transport: roundTripFunc(func(req *http.Request) (*http.Response, error) {
+		switch req.Method + " " + req.URL.Path {
+		case "POST /v2/scrape":
+			content, _ := json.Marshal(map[string]any{
+				"success": true,
+				"data":    map[string]any{"markdown": challenge},
+			})
+			return testHTTPResponse(req, http.StatusOK, string(content), nil), nil
+		case "POST /v2/interact":
+			return testHTTPResponse(req, http.StatusOK, `{"success":true,"id":"test-captcha"}`, nil), nil
+		case "POST /v2/interact/test-captcha/execute":
+			body, _ := json.Marshal(map[string]any{
+				"success": true, "stdout": challenge, "exitCode": 0,
+			})
+			return testHTTPResponse(req, http.StatusOK, string(body), nil), nil
+		case "DELETE /v2/interact/test-captcha":
+			return testHTTPResponse(req, http.StatusOK, `{"success":true}`, nil), nil
+		default:
+			t.Fatalf("unexpected request: %s %s", req.Method, req.URL)
+			return nil, nil
+		}
+	})}
+	t.Setenv("FIRECRAWL_API_KEY", "unit-test")
+	_, returned, err := scrapeLinkWithFirecrawl("https://docs.example.com/challenged")
+	if err == nil || returned != "" || !strings.Contains(err.Error(), "challenge") {
+		t.Fatalf("long browser challenge leaked into result or lacked clear failure: %q %v", returned, err)
+	}
+	if strings.Contains(err.Error(), "French Deutsch") {
+		t.Fatalf("challenge payload should not be logged verbatim in errors: %s", err)
+	}
+}
+
 func TestScrapeEscalatesToEnhancedOnBlockedBody(t *testing.T) {
 	originalClient := http.DefaultClient
 	t.Cleanup(func() { http.DefaultClient = originalClient })

package/internal/app/native_markdown.go

@@ -0,0 +1,76 @@
+package app
+
+import (
+	"context"
+	"net/url"
+	"path"
+	"strings"
+	"time"
+)
+
+const maxNativeMarkdownBytes = 1024 * 1024
+
+// Documentation sites frequently expose canonical HTML and an authored
+// Markdown representation. Use bounded GET probes instead of HEAD:
+// Stripe sends Content-Length: 0 on a successful Markdown HEAD response,
+// and Cloudflare uses /index.md for directory URLs.
+func markdownCandidateURLs(rawURL string) []string {
+	u, err := url.Parse(rawURL)
+	if err != nil {
+		return nil
+	}
+	p := u.Path
+	if strings.HasSuffix(strings.ToLower(p), ".md") {
+		return []string{rawURL}
+	}
+	var candidates []string
+	if strings.HasSuffix(p, "/") || p == "" {
+		candidates = append(candidates, strings.TrimSuffix(p, "/")+"/index.md")
+	} else if path.Ext(p) == "" {
+		candidates = append(candidates, p+".md", p+"/index.md")
+	} else {
+		return nil
+	}
+	seen := map[string]bool{}
+	var out []string
+	for _, candidate := range candidates {
+		next := *u
+		next.Path = candidate
+		next.RawPath = ""
+		next.Fragment = ""
+		next.RawFragment = ""
+		value := next.String()
+		if !seen[value] && validateSourceURL(value) == nil {
+			out = append(out, value)
+			seen[value] = true
+		}
+	}
+	return out
+}
+
+func usableNativeMarkdown(content string) bool {
+	text := strings.TrimSpace(content)
+	if len(text) < 18 || strings.IndexByte(text, 0) >= 0 || challengeReason(text) != "" {
+		return false
+	}
+	prefix := strings.ToLower(text[:min(len(text), 700)])
+	if strings.Contains(prefix, "<!doctype html") || strings.Contains(prefix, "<html") ||
+		strings.Contains(prefix, "<head") || strings.Contains(prefix, "<body") {
+		return false
+	}
+	return true
+}
+
+func readNativeMarkdown(rawURL string) (*MarkdownResult, error) {
+	for _, candidate := range markdownCandidateURLs(rawURL) {
+		ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+		body, err := fetchPublicText(ctx, candidate, maxNativeMarkdownBytes)
+		cancel()
+		if err != nil || !usableNativeMarkdown(body) {
+			continue
+		}
+		title := firstHeadingOrFallback(body, path.Base(rawURL))
+		return &MarkdownResult{URL: rawURL, SourceURL: candidate, Title: title, Markdown: body}, nil
+	}
+	return nil, nil
+}

package/internal/app/native_markdown_test.go

@@ -0,0 +1,78 @@
+package app
+
+import (
+	"net/http"
+	"strings"
+	"testing"
+)
+
+func TestCanonicalStripeGuidePrefersMarkdownWithZeroHeadLength(t *testing.T) {
+	previous := http.DefaultClient
+	t.Cleanup(func() { http.DefaultClient = previous })
+	calls := []string{}
+	http.DefaultClient = &http.Client{Transport: roundTripFunc(func(req *http.Request) (*http.Response, error) {
+		calls = append(calls, req.Method+" "+req.URL.Path)
+		if req.Method == http.MethodHead {
+			t.Fatal("native Markdown detection must not require HEAD metadata")
+		}
+		if req.Method == http.MethodGet && req.URL.Path == "/payments/checkout.md" {
+			r := testHTTPResponse(req, http.StatusOK, "# Build a payments page\n\nStripe-hosted checkout is supported.", map[string]string{"Content-Type": "text/markdown"})
+			r.ContentLength = 0 // Publishers can return zero or unknown length metadata.
+			return r, nil
+		}
+		t.Fatalf("unexpected extra request: %s %s", req.Method, req.URL)
+		return nil, nil
+	})}
+	out, err := ReadLink("https://docs.stripe.com/payments/checkout")
+	if err != nil || !strings.Contains(out, "**Markdown source:** https://docs.stripe.com/payments/checkout.md") {
+		t.Fatalf("native source not selected: %v %s", err, out)
+	}
+	if len(calls) != 1 {
+		t.Fatalf("unexpected calls: %+v", calls)
+	}
+}
+
+func TestCanonicalCloudflareDirectoryReadsIndexMarkdown(t *testing.T) {
+	previous := http.DefaultClient
+	t.Cleanup(func() { http.DefaultClient = previous })
+	http.DefaultClient = &http.Client{Transport: roundTripFunc(func(req *http.Request) (*http.Response, error) {
+		if req.Method != http.MethodGet || req.URL.Path != "/durable-objects/index.md" {
+			t.Fatalf("unexpected URL: %s %s", req.Method, req.URL)
+		}
+		return testHTTPResponse(req, http.StatusOK,
+			"# Cloudflare Durable Objects\n\n[Get started](https://developers.cloudflare.com/durable-objects/get-started/)",
+			map[string]string{"Content-Type": "text/markdown"}), nil
+	})}
+	out, err := ReadLink("https://developers.cloudflare.com/durable-objects/")
+	if err != nil || !strings.Contains(out, "**Markdown source:** https://developers.cloudflare.com/durable-objects/index.md") {
+		t.Fatalf("directory-index native Markdown failed: %v %s", err, out)
+	}
+}
+
+func TestCAPTCHADisguisedAsMarkdownIsRejected(t *testing.T) {
+	previous := http.DefaultClient
+	t.Cleanup(func() { http.DefaultClient = previous })
+	var scrapeCalled bool
+	http.DefaultClient = &http.Client{Transport: roundTripFunc(func(req *http.Request) (*http.Response, error) {
+		if req.URL.Path == "/guide.md" {
+			return testHTTPResponse(req, http.StatusOK, "# hCaptcha\nPlease verify you are human. "+
+				strings.Repeat("English French Deutsch Español Chinese 日本語 ", 200), map[string]string{"Content-Type": "text/markdown"}), nil
+		}
+		if req.URL.Path == "/guide/index.md" {
+			return testHTTPResponse(req, http.StatusNotFound, "", nil), nil
+		}
+		if req.Method == http.MethodPost && req.URL.Path == "/v2/scrape" {
+			scrapeCalled = true
+			return testHTTPResponse(req, http.StatusOK,
+				`{"success":true,"data":{"markdown":"# Legitimate guide\n\nFull instructions for a public API."}}`,
+				nil), nil
+		}
+		t.Fatalf("unexpected request: %s %s", req.Method, req.URL)
+		return nil, nil
+	})}
+	t.Setenv("FIRECRAWL_API_KEY", "test")
+	out, err := ReadLink("https://docs.example.com/guide")
+	if err != nil || !scrapeCalled || strings.Contains(out, "Deutsch Español") || !strings.Contains(out, "Full instructions") {
+		t.Fatalf("CAPTCHA disguised as Markdown leaked or fallback failed: %v %s", err, out)
+	}
+}