webctx 0.2.3 → 0.2.5

package/CONTRIBUTORS.md

@@ -97,7 +97,7 @@ gh workflow run npm-publish-retry.yml --repo amxv/webctx --ref main -f version=0
 - `cmd/webctx/main.go`: CLI entrypoint
 - `internal/app/`: CLI parsing, search, ranking, scrape, env loading, and Firecrawl queue logic
 - `pkg/retrieval/`: small public adapter to the shared read-link / site-map engine
-- `pkg/origo/`: authenticated, stateless MCP server with exactly two tools
+- `pkg/origo/`: authenticated, stateless MCP server with five tools (including structured-data discovery, inspection and execution)
 - `api/mcp.go`: Vercel Go Function entrypoint for Origo
 - `vercel.mjs`: project-specific Vercel settings for docs and Origo
 - `internal/buildinfo/`: build-time version plumbing for `--version`

package/README.md

@@ -2,12 +2,15 @@
 
 Web context for agents, from the terminal.
 
-`webctx` gives you three small commands:
+`webctx` provides six commands: three for web documents and three for structured data.
 
 ```bash
 webctx search "agent web research"
 webctx read-link <url>
 webctx map-site <url>
+webctx research "podcast conversations about AI agents"
+webctx inspect particle/podcasts/episodes/search
+webctx execute particle/podcasts/episodes/search --inputs '{"semantic_search":"AI agents","limit":2}' --max-credits 20
 ```
 
 The output is plain text or markdown, so it is easy to hand to ChatGPT, Codex, Claude Code, a shell script, or another tool. Search now retrieves relevant source excerpts automatically instead of returning only URLs. For a specific URL, add an optional question to explore related documentation and surface exact code and API details.
@@ -19,11 +22,25 @@ webctx read-link https://docs.example.com/api --question "How do authentication
 
 ## Origo: hosted MCP access
 
-**Origo** is the MCP transport over the same WebCTX retrieval engine. Connect an MCP client to `https://api.origo.ashray.xyz/mcp?key=<your-private-key>` to access exactly two tools: `read_link` and `map_site`. Origo does not expose `search`; the CLI continues to provide all three commands. `read_link` accepts an optional `question` to retrieve focused source context while keeping ordinary URL-only reads intact.
+**Origo** is the MCP transport over the same WebCTX retrieval engine. Connect an MCP client to `https://api.origo.ashray.xyz/mcp?key=<your-private-key>` to access five tools: `read_link`, `map_site`, `research`, `inspect`, and `execute`. Origo deliberately omits federated web `search` but exposes the complete progressive data-research workflow. `read_link` accepts an optional `question` and keeps URL-only behavior unchanged.
 
 Both surfaces share native GitHub/Markdown readers and Firecrawl fallback behavior. Firecrawl's built-in cache is limited to **30 minutes**. Difficult pages automatically escalate through Firecrawl's proxy fallback and, if necessary, a temporary Browser Sandbox session. There is no Redis or additional cache database.
 
-Source-aware reading also prefers raw JSON, YAML, and OpenAPI files when the URL identifies one; question-focused reads can follow up to two relevant pages on the same host, including links found in `llms.txt`, and inspect embedded structured JSON. Firecrawl Alexandria tool matching and search discovery are automatic when supported, but **no paid Alexandria provider tools are executed**. The original source URLs remain explicit so an agent can verify facts.
+Source-aware reading also prefers raw JSON, YAML, and OpenAPI files when the URL identifies one; question-focused reads can follow relevant references, including links found in `llms.txt`, and inspect embedded structured JSON. The new `research`, `inspect`, and `execute` commands give agents direct control over Firecrawl Alexandria's catalogue and paid data operations without learning provider-specific transport syntax. The original source URLs remain explicit so an agent can verify facts.
+
+## Structured data: discover → inspect → execute
+
+`research` discovers datasets and operations by meaning, website, source, category, or group. `inspect` retrieves exact parameter types, required fields, response shapes, pricing, and copy-ready `execute` arguments. `execute` validates the inputs and credit estimate before calling one or more data operations, and preserves their source-native JSON and pagination.
+
+```bash
+webctx research "podcast conversations about AI agents" --limit 5
+webctx research --view sources --limit 5
+webctx research --mode catalogue --view operations --sources particle --limit 10
+webctx inspect particle/podcasts/episodes/search
+webctx execute particle/podcasts/episodes/search --inputs '{"semantic_search":"AI agents","limit":2}' --max-credits 20
+```
+
+Browse [the structured-data guide](https://webctx.ashray.xyz/docs/data-sources) for filters, paging, batches, full upstream responses, validation, and credit handling. Discovery/inspection is free; `execute` uses paid Alexandria credits. The per-request default preflight cap is 100 credits (configurable via `WEBCTX_ALEXANDRIA_MAX_CREDITS`). Third-party provider terms require separate human acceptance; Origo never accepts them automatically.
 
 Origo deploys independently through the `origo-api` Vercel project on pushes to `main`. CLI releases and npm publishing are unchanged. See the [Origo MCP guide](https://webctx.ashray.xyz/docs/origo) for architecture, setup, and security notes.
 
@@ -78,7 +95,7 @@ Recognized native GitHub auth, private/not-found, and rate-limit failures stay a
 
 ## Search
 
-Normal search asks Brave, Tavily, and Exa, removes duplicate URLs, automatically reads three promising sources, follows up to two relevant links, and returns excerpts plus the ranked list. Alexandria tool discovery is included when available, without paid execution.
+Normal search asks Brave, Tavily, and Exa, removes duplicate URLs, automatically reads promising sources, follows relevant links, and returns excerpts plus the ranked list. Alexandria discovery is included when available; use `execute` when you need provider-native data.
 
 ```bash
 webctx search "next.js server components"
@@ -115,7 +132,7 @@ GH_TOKEN=...
 ```
 
 - `BRAVE_API_KEY`, `TAVILY_API_KEY`, and `EXA_API_KEY` power normal `search`.
-- `FIRECRAWL_API_KEY` powers `map-site` and page-crawl fallbacks.
+- `FIRECRAWL_API_KEY` powers `map-site`, page-crawl fallbacks, and Alexandria discovery/inspection/paid execution.
 - `GH_TOKEN` or `GITHUB_TOKEN` is optional. It increases GitHub capacity and unlocks reads such as blame, Discussions, some Actions job logs, private resources the token can access, and richer PR review state.
 
 Environment variables, `.env.local`, and macOS Keychain are supported. `GH_TOKEN` takes precedence over `GITHUB_TOKEN`.
@@ -129,6 +146,7 @@ Start with the guides in `src/content/docs`:
 - **Use webctx with agents** — practical research, repo, PR, and CI workflows
 - **Search the web** — federated search and filtering
 - **Map a site** — discover pages before reading them
+- **Research structured data** — browse data sources, inspect exact contracts, and execute paid queries with bounded credits
 - **Credentials** — keys and optional GitHub auth
 - **How URL reading works** — native/direct fast paths before Firecrawl
 - **How search ranking works** — URL normalization, position scoring, and provider agreement

package/internal/app/app.go

@@ -1,8 +1,11 @@
 package app
 
 import (
+	"encoding/json"
 	"fmt"
 	"io"
+	"os"
+	"strconv"
 	"strings"
 
 	"github.com/amxv/webctx/internal/buildinfo"
@@ -74,6 +77,90 @@ func Run(args []string, stdout, stderr io.Writer) int {
 		}
 		_, _ = fmt.Fprintln(stdout, text)
 		return 0
+	case "research":
+		research := ResearchDataInput{
+			Query:      strings.Join(positional, " "),
+			Mode:       flags["mode"],
+			URLs:       splitCSV(flags["urls"]),
+			Sources:    splitCSV(flags["sources"]),
+			Categories: splitCSV(flags["categories"]),
+			Groups:     splitCSV(flags["groups"]),
+			Operations: splitCSV(flags["operations"]),
+			View:       flags["view"],
+			Include:    splitCSV(flags["include"]),
+			Raw:        flags["raw"] == "true",
+		}
+		if q := flags["query"]; q != "" {
+			research.Query = q
+		}
+		if flag := flags["limit"]; flag != "" {
+			n, e := strconv.Atoi(flag)
+			if e != nil {
+				return cliDataError(stderr, dataError("invalid_limit", "--limit requires an integer."))
+			}
+			research.Limit = n
+		}
+		if flag := flags["offset"]; flag != "" {
+			n, e := strconv.Atoi(flag)
+			if e != nil {
+				return cliDataError(stderr, dataError("invalid_offset", "--offset requires an integer."))
+			}
+			research.Offset = n
+		}
+		result, e := ResearchData(research)
+		if e != nil {
+			return cliDataError(stderr, e)
+		}
+		writeDataJSON(stdout, result)
+		return 0
+	case "inspect":
+		if strings.TrimSpace(input) == "" {
+			return cliDataError(stderr, dataError("missing_id", "Usage: webctx inspect <operation-or-source-id> [--raw]"))
+		}
+		result, e := InspectData(InspectDataInput{ID: input, Include: splitCSV(flags["include"]), Raw: flags["raw"] == "true"})
+		if e != nil {
+			return cliDataError(stderr, e)
+		}
+		writeDataJSON(stdout, result)
+		return 0
+	case "execute":
+		request := ExecuteDataInput{Raw: flags["raw"] == "true", RequestID: flags["request-id"]}
+		if flag := flags["max-credits"]; flag != "" {
+			n, e := strconv.Atoi(flag)
+			if e != nil {
+				return cliDataError(stderr, dataError("invalid_budget", "--max-credits requires an integer."))
+			}
+			request.MaxCredits = n
+		}
+		if jsonCalls := flags["calls"]; jsonCalls != "" {
+			data, e := cliJSONString(jsonCalls)
+			if e != nil {
+				return cliDataError(stderr, e)
+			}
+			if e := json.Unmarshal([]byte(data), &request.Calls); e != nil {
+				return cliDataError(stderr, dataError("invalid_calls", "--calls must be a JSON array of {id,inputs} objects."))
+			}
+		} else if strings.TrimSpace(input) != "" {
+			var inputs map[string]any
+			if source := flags["inputs"]; source != "" {
+				data, e := cliJSONString(source)
+				if e != nil {
+					return cliDataError(stderr, e)
+				}
+				if e := json.Unmarshal([]byte(data), &inputs); e != nil {
+					return cliDataError(stderr, dataError("invalid_inputs", "--inputs must be a JSON object."))
+				}
+			}
+			request.Calls = []DataCall{{ID: input, Inputs: inputs}}
+		} else {
+			return cliDataError(stderr, dataError("missing_calls", "Usage: webctx execute <operation-id> --inputs JSON [--max-credits N] or --calls JSON."))
+		}
+		result, e := ExecuteData(request)
+		if e != nil {
+			return cliDataError(stderr, e)
+		}
+		writeDataJSON(stdout, result)
+		return 0
 	default:
 		_, _ = fmt.Fprintln(stderr, "Unknown tool:", tool)
 		_, _ = fmt.Fprintln(stdout, usageText())
@@ -88,11 +175,19 @@ Usage:
   webctx search <query> [--exclude domain1,domain2] [--keyword phrase]
   webctx read-link <url> [--question 'what to find']
   webctx map-site <url>
+  webctx research [question] [--view sources|groups|operations] [--sources IDs] [--include inputs,output,examples] [--limit N] [--offset N]
+  webctx inspect <operation-or-source-id> [--raw]
+  webctx execute <operation-id> --inputs '{"field":"value"}' [--max-credits N] [--request-id ID]
+  webctx execute --calls '[{"id":"source/operation","inputs":{}}]' [--max-credits N]
 
 Examples:
   webctx search "next.js server components"
   webctx search "react hooks" --exclude youtube.com,vimeo.com
   webctx search "drizzle orm" --keyword "migration guide"
+  webctx research "podcast conversations about AI agents"
+  webctx research --view sources --limit 5
+  webctx inspect particle/podcasts/episodes/search
+  webctx execute particle/podcasts/episodes/search --inputs '{"semantic_search":"AI agents","limit":2}' --max-credits 20
   webctx read-link https://docs.example.com/guide
   webctx read-link https://docs.example.com/api --question "How do auth and pagination work?"
   webctx map-site https://example.com`, version)
@@ -102,6 +197,16 @@ func parseArgs(args []string) (map[string]string, []string) {
 	flags := map[string]string{}
 	positional := make([]string, 0, len(args))
 	for i := 0; i < len(args); i++ {
+		if args[i] == "--raw" {
+			flags["raw"] = "true"
+			continue
+		}
+		if strings.HasPrefix(args[i], "--") {
+			if k, v, ok := strings.Cut(strings.TrimPrefix(args[i], "--"), "="); ok {
+				flags[k] = v
+				continue
+			}
+		}
 		if strings.HasPrefix(args[i], "--") && i+1 < len(args) {
 			flags[strings.TrimPrefix(args[i], "--")] = args[i+1]
 			i++
@@ -112,6 +217,38 @@ func parseArgs(args []string) (map[string]string, []string) {
 	return flags, positional
 }
 
+func cliJSONString(source string) (string, error) {
+	if !strings.HasPrefix(source, "@") {
+		return source, nil
+	}
+	path := strings.TrimPrefix(source, "@")
+	if path == "" {
+		return "", dataError("invalid_file", "The @file argument requires a path.")
+	}
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return "", dataError("invalid_file", "Unable to read JSON input file: "+err.Error())
+	}
+	if len(b) > 1<<20 {
+		return "", dataError("invalid_file", "JSON input file exceeds 1 MiB.")
+	}
+	return string(b), nil
+}
+
+func writeDataJSON(out io.Writer, value any) {
+	bytes, err := json.MarshalIndent(value, "", "  ")
+	if err != nil {
+		_, _ = fmt.Fprintln(out, "{}")
+		return
+	}
+	_, _ = fmt.Fprintln(out, string(bytes))
+}
+
+func cliDataError(stderr io.Writer, err error) int {
+	writeDataJSON(stderr, dataRawOrError(err))
+	return 1
+}
+
 func splitCSV(v string) []string {
 	if strings.TrimSpace(v) == "" {
 		return nil

package/internal/app/data_client.go

@@ -0,0 +1,220 @@
+package app
+
+import (
+	"context"
+	"crypto/rand"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"net/url"
+	"os"
+	"regexp"
+	"strconv"
+	"strings"
+	"time"
+)
+
+const dataAPIBase = "https://api.firecrawl.dev/v2"
+const defaultDataCreditBudget = 100
+const maxDataResponseBytes int64 = 12 << 20
+
+type DataError struct {
+	Code           string         `json:"code"`
+	Message        string         `json:"message"`
+	RequiresAction map[string]any `json:"requires_action,omitempty"`
+	Details        any            `json:"details,omitempty"`
+}
+
+func (e *DataError) Error() string { return e.Code + ": " + e.Message }
+func dataError(code, message string) error {
+	return &DataError{Code: code, Message: message}
+}
+
+func dataAPIKey() (string, error) {
+	key := strings.TrimSpace(os.Getenv("FIRECRAWL_API_KEY"))
+	if key == "" {
+		return "", dataError("missing_credentials", "Set FIRECRAWL_API_KEY to use data source discovery and queries.")
+	}
+	return key, nil
+}
+
+func dataCreditCap() int {
+	value, err := strconv.Atoi(strings.TrimSpace(os.Getenv("WEBCTX_ALEXANDRIA_MAX_CREDITS")))
+	if err != nil || value < 1 || value > 100000 {
+		return defaultDataCreditBudget
+	}
+	return value
+}
+
+func paidDataEnabled() bool {
+	switch strings.ToLower(strings.TrimSpace(os.Getenv("WEBCTX_ALEXANDRIA_PAID"))) {
+	case "0", "false", "no", "off":
+		return false
+	default:
+		return true
+	}
+}
+
+var safeRequestID = regexp.MustCompile("^[A-Za-z0-9_-]{8,128}$")
+
+func newRequestID() (string, error) {
+	var bytes [16]byte
+	if _, err := rand.Read(bytes[:]); err != nil {
+		return "", err
+	}
+	return "webctx-" + hex.EncodeToString(bytes[:]), nil
+}
+
+func doDataAPI(ctx context.Context, endpoint string, payload any, requestID string) (map[string]any, error) {
+	key, err := dataAPIKey()
+	if err != nil {
+		return nil, err
+	}
+	body, err := json.Marshal(payload)
+	if err != nil {
+		return nil, dataError("invalid_request", "Could not encode request as JSON: "+err.Error())
+	}
+	req, err := http.NewRequestWithContext(ctx, http.MethodPost, dataAPIBase+endpoint, strings.NewReader(string(body)))
+	if err != nil {
+		return nil, err
+	}
+	req.Header.Set("Authorization", "Bearer "+key)
+	req.Header.Set("Content-Type", "application/json")
+	if requestID != "" {
+		req.Header.Set("x-request-id", requestID)
+	}
+	res, err := http.DefaultClient.Do(req)
+	if err != nil {
+		return nil, &DataError{Code: "network_error", Message: "The provider request did not complete. For a paid call, retry only with the same request_id to prevent duplicate charges."}
+	}
+	defer res.Body.Close()
+	raw, err := io.ReadAll(io.LimitReader(res.Body, maxDataResponseBytes+1))
+	if err != nil {
+		return nil, dataError("read_error", "Could not read provider response.")
+	}
+	if int64(len(raw)) > maxDataResponseBytes {
+		return nil, dataError("response_too_large", "Provider response exceeded 12 MiB; use a smaller limit or pagination.")
+	}
+	var decoded map[string]any
+	if json.Unmarshal(raw, &decoded) != nil {
+		return nil, dataError("invalid_upstream_response", fmt.Sprintf("Provider returned non-JSON HTTP %d.", res.StatusCode))
+	}
+	if res.StatusCode < 200 || res.StatusCode >= 300 || decoded["success"] == false {
+		code := stringValue(decoded["code"])
+		if code == "" {
+			code = fmt.Sprintf("http_%d", res.StatusCode)
+		}
+		message := stringValue(decoded["error"])
+		if message == "" {
+			message = stringValue(decoded["message"])
+		}
+		if message == "" {
+			message = "The provider rejected the request."
+		}
+		if len(message) > 800 {
+			message = message[:800]
+		}
+		out := &DataError{Code: code, Message: message}
+		if action, ok := decoded["requiresAction"].(map[string]any); ok {
+			if target := stringValue(action["url"]); target != "" {
+				if u, e := url.Parse(target); e == nil && (u.Scheme == "https" || u.Scheme == "http") && u.Host != "" {
+					out.RequiresAction = map[string]any{"url": target}
+				}
+			}
+		}
+		return nil, out
+	}
+	return decoded, nil
+}
+
+func nativeDataCall(ctx context.Context, provider, capability string, options map[string]any, requestID string) (map[string]any, error) {
+	request := map[string]any{"alexandria": map[string]any{
+		"provider": provider, "capability": capability, "options": options,
+	}}
+	return doDataAPI(ctx, "/scrape", request, requestID)
+}
+
+func extractNativeData(response map[string]any) (map[string]any, error) {
+	data, ok := response["data"].(map[string]any)
+	if !ok {
+		return nil, dataError("invalid_upstream_response", "The response has no data object.")
+	}
+	items, ok := data["alexandria"].([]any)
+	if !ok || len(items) == 0 {
+		return nil, dataError("invalid_upstream_response", "The response contains no Alexandria results.")
+	}
+	item, ok := items[0].(map[string]any)
+	if !ok {
+		return nil, dataError("invalid_upstream_response", "The source returned an invalid result.")
+	}
+	return item, nil
+}
+
+func freeDataBrowse(ctx context.Context, options map[string]any) (map[string]any, map[string]any, error) {
+	response, err := nativeDataCall(ctx, "firecrawl", "find-tools", options, "")
+	if err != nil {
+		return nil, nil, err
+	}
+	item, err := extractNativeData(response)
+	if err != nil {
+		return nil, nil, err
+	}
+	details, ok := item["data"].(map[string]any)
+	if !ok {
+		return nil, nil, dataError("invalid_upstream_response", "The catalogue returned no browsable entries.")
+	}
+	return details, response, nil
+}
+
+func defaultDataContext() (context.Context, context.CancelFunc) {
+	return context.WithTimeout(context.Background(), 45*time.Second)
+}
+
+func dataRawOrError(err error) map[string]any {
+	var typed *DataError
+	if errors.As(err, &typed) {
+		return map[string]any{"error": typed}
+	}
+	return map[string]any{"error": map[string]any{"code": "internal_error", "message": err.Error()}}
+}
+
+// DataErrorResult preserves actionable failures in both CLI and MCP JSON.
+func DataErrorResult(err error) map[string]any { return dataRawOrError(err) }
+
+func listField(value any) []any {
+	switch v := value.(type) {
+	case []any:
+		return v
+	case []string:
+		out := make([]any, 0, len(v))
+		for _, s := range v {
+			out = append(out, s)
+		}
+		return out
+	default:
+		return nil
+	}
+}
+
+func stringField(value any) string      { s, _ := value.(string); return s }
+func mapField(value any) map[string]any { m, _ := value.(map[string]any); return m }
+func numberField(value any) (float64, bool) {
+	switch n := value.(type) {
+	case float64:
+		return n, true
+	case float32:
+		return float64(n), true
+	case int:
+		return float64(n), true
+	case int64:
+		return float64(n), true
+	case json.Number:
+		f, e := n.Float64()
+		return f, e == nil
+	default:
+		return 0, false
+	}
+}