webctx 0.2.8 → 0.2.10

package/AGENTS.md

@@ -47,13 +47,16 @@ Direct commands:
 Preserve these commands unless the user explicitly asks to change them:
 
 - `webctx search <query> [--exclude domain1,domain2] [--keyword phrase]`
+- `webctx status` and `webctx providers enable|disable <brave|tavily|exa>`
+- `webctx keys list|add|remove` for macOS Keychain-managed additional search keys; `BRAVE_API_KEYS`, `TAVILY_API_KEYS`, and `EXA_API_KEYS` also work via environment variables
 - `webctx read-link <url>`
 - `webctx map-site <url>`
 - `webctx --version`
 
 Behavioral expectations:
 
-- `search` combines Brave, Tavily, and Exa results, then re-ranks them with duplicate-aware scoring.
+- `search` combines enabled Brave, Tavily, and Exa results, then re-ranks them with duplicate-aware scoring; an unavailable Exa can fall back to quoted-phrase Brave/Tavily queries for `--keyword` (not a strict text filter).
+- Search key rotation tries plural credentials first, legacy singular credentials afterward; failover applies only to credential/quota/rate/server/transport failures, never bad requests. Keep Firecrawl paid operations outside this rotation.
 - `read-link` keeps the current GitHub raw-content and `.md` fast paths. Firecrawl uses its built-in 30-minute cache, automatic and enhanced proxies, then browser extraction as a final fallback.
 - `map-site` keeps the current Firecrawl map request settings.
 - The CLI should remain agent-friendly and emit plain markdown/text output.
@@ -100,3 +103,11 @@ When cutting a release, update `src/content/docs/changelog.md` before tagging.
 - Do not include docs-site-only changes such as site styling, Zuedocs/package bumps, deploy plumbing, footer/layout changes, or documentation navigation changes.
 - Rewrite commit subjects into clear release notes instead of pasting raw commit messages.
 - If a release contains only tagging/release metadata, write: `Maintenance release. No direct code behavior changes beyond release preparation.`
+
+### Docs deployment
+
+The Docs GitHub Actions workflow deploys production only for the watched site paths; Vercel Git auto-deployments and `ignoreCommand` stay disabled. Keep the project-scoped `VERCEL_TOKEN` Actions secret and ID variables configured. Check Astro locally; builds run in CI.
+
+### Origo deployment
+
+The separate `Origo API` GitHub Actions workflow deploys the Go function only for changes to its import graph and Vercel configuration. Use `VERCEL_ORIGO_TOKEN` and `VERCEL_ORIGO_PROJECT_ID` exclusively for Origo; never substitute the WebCTX Docs token or a static `--prebuilt` site bundle. Keep the `/mcp` rewrite and Origo production environment variables intact.

package/CONTRIBUTORS.md

@@ -108,9 +108,9 @@ gh workflow run npm-publish-retry.yml --repo amxv/webctx --ref main -f version=0
 
 ## Origo deployment
 
-The `origo-api` Vercel project is linked to this GitHub repo and uses `ORIGO_DEPLOYMENT=1` to select its Go Function configuration. It needs `ORIGO_API_KEY` and `FIRECRAWL_API_KEY` production environment variables; optionally add `GH_TOKEN` for native authenticated GitHub reads. The docs project remains `webctx-docs` and never receives the Origo connection key.
+The `origo-api` Vercel project uses `ORIGO_DEPLOYMENT=1` to select its Go Function configuration. It needs `ORIGO_API_KEY` and `FIRECRAWL_API_KEY` production environment variables; optionally add `GH_TOKEN` for native authenticated GitHub reads. The docs project remains `webctx-docs` and never receives the Origo connection key.
 
-Both projects deploy automatically from `main`. `scripts/should-build.mjs` gates docs and API builds independently, while `v*` GitHub tags still drive the CLI release workflow.
+The independent `Docs` and `Origo API` GitHub Actions workflows filter their own build inputs before deploying to Vercel. Origo uses the `VERCEL_ORIGO_TOKEN` Actions secret and `VERCEL_ORIGO_PROJECT_ID` variable, separate from the docs credentials. Git-triggered Vercel deployments are disabled; `v*` tags still drive CLI releases.
 
 ## Notes on package naming
 

package/README.md

@@ -95,7 +95,7 @@ Recognized native GitHub auth, private/not-found, and rate-limit failures stay a
 
 ## Search
 
-Normal search asks Brave, Tavily, and Exa, removes duplicate URLs, automatically reads promising sources, follows relevant links, and returns excerpts plus the ranked list. Alexandria discovery is included when available; use `execute` when you need provider-native data.
+Normal search asks the enabled Brave, Tavily, and Exa providers, removes duplicate URLs, automatically reads promising sources, follows relevant links, and returns excerpts plus the ranked list. One working search provider is enough; Alexandria discovery is included when available. Use `execute` when you need provider-native data.
 
 ```bash
 webctx search "next.js server components"
@@ -103,6 +103,31 @@ webctx search "react hooks" --exclude youtube.com,medium.com
 webctx search "drizzle orm" --keyword "migration guide"
 ```
 
+### Enable or disable search providers
+
+```bash
+webctx status                 # enabled/disabled providers and locally configured keys
+webctx providers disable exa  # stop all Exa search requests, even with a key present
+webctx providers enable exa   # turn Exa back on later
+webctx providers disable brave
+webctx providers enable brave
+```
+
+Provider settings persist across runs, independently of environment variables and Keychain credentials. The settings file lives in the OS user config directory (`webctx/config.json`); set `WEBCTX_CONFIG_FILE` to override its location. Disabled providers are never queried. `status` only reports key presence, **not** live credit balances or API health.
+
+Multiple keys per provider are supported. Add extra keys on macOS without placing secrets in command arguments or plaintext config files:
+
+```bash
+webctx keys list                         # count configured and managed keys (never prints them)
+pbpaste | webctx keys add exa --stdin    # securely store a copied new key in macOS Keychain
+webctx keys remove exa 1                # remove the newest Keychain-managed Exa key
+webctx providers enable exa
+```
+
+Or supply `BRAVE_API_KEYS`, `TAVILY_API_KEYS`, or `EXA_API_KEYS` as comma-separated keys (or JSON arrays) in the environment or `.env.local`, alongside the corresponding singular `*_API_KEY`. Keys in the plural list are attempted first, then the legacy single key. On a key-specific authentication, quota/credit, rate-limit, or server error, the next key is tried automatically; invalid requests are not retried. Manage only keys/accounts you are authorized to use and honor each provider's usage restrictions. The Firecrawl/Alexandria integration is separate and **is not included in automatic key rotation**, especially for paid operations.
+
+`--keyword` prefers Exa's strict include-text filter when available. If Exa is disabled, unconfigured, or fails, Brave/Tavily search for the quoted phrase instead; the output explicitly notes that this fallback cannot guarantee the phrase appears in the page text. Firecrawl is a separate reading/site-map/Alexandria integration, not one of these three search providers.
+
 ## Map a site
 
 Use `map-site` when you want to discover the useful pages before reading them.
@@ -131,7 +156,7 @@ FIRECRAWL_API_KEY=...
 GH_TOKEN=...
 ```
 
-- `BRAVE_API_KEY`, `TAVILY_API_KEY`, and `EXA_API_KEY` power normal `search`.
+- `BRAVE_API_KEY`, `TAVILY_API_KEY`, and `EXA_API_KEY` power normal `search`. The plural `*_API_KEYS` names supply additional fallback credentials. Configure any one or more, and optionally disable providers you no longer want to query.
 - `FIRECRAWL_API_KEY` powers `map-site`, page-crawl fallbacks, and Alexandria discovery/inspection/paid execution.
 - `GH_TOKEN` or `GITHUB_TOKEN` is optional. It increases GitHub capacity and unlocks reads such as blame, Discussions, some Actions job logs, private resources the token can access, and richer PR review state.
 

package/internal/app/app.go

@@ -16,8 +16,6 @@ const commandName = "webctx"
 var version = buildinfo.CurrentVersion()
 
 func Run(args []string, stdout, stderr io.Writer) int {
-	loadEnvLocal()
-
 	if len(args) == 0 || isHelpArg(args[0]) {
 		_, _ = fmt.Fprintln(stdout, usageText())
 		return 0
@@ -29,6 +27,71 @@ func Run(args []string, stdout, stderr io.Writer) int {
 	}
 
 	tool := args[0]
+	if tool == "keys" {
+		if len(args) == 2 && args[1] == "list" {
+			loadEnvLocal()
+			if err := printProviderStatus(stdout); err != nil {
+				_, _ = fmt.Fprintln(stderr, err)
+				return 1
+			}
+			if err := printManagedSearchKeySlots(stdout); err != nil {
+				_, _ = fmt.Fprintln(stderr, err)
+				return 1
+			}
+			return 0
+		}
+		if len(args) == 4 && args[1] == "add" && args[3] == "--stdin" {
+			count, err := addManagedSearchKey(args[2], os.Stdin)
+			if err != nil {
+				_, _ = fmt.Fprintln(stderr, err)
+				return 1
+			}
+			_, _ = fmt.Fprintf(stdout, "Saved an additional %s API key in macOS Keychain (%d managed key(s)).\n", strings.ToLower(args[2]), count)
+			return 0
+		}
+		if len(args) == 4 && args[1] == "remove" {
+			count, err := removeManagedSearchKey(args[2], args[3])
+			if err != nil {
+				_, _ = fmt.Fprintln(stderr, err)
+				return 1
+			}
+			_, _ = fmt.Fprintf(stdout, "Removed managed %s API key (%d managed key(s) remain).\n", strings.ToLower(args[2]), count)
+			return 0
+		}
+		_, _ = fmt.Fprintln(stderr, "Usage: webctx keys list | keys add <brave|tavily|exa> --stdin | keys remove <provider> <managed-key-index>")
+		return 1
+	}
+	isProviderTool := tool == "status" || tool == "providers" || tool == "provider"
+	isProviderStatus := (tool == "status" && len(args) == 1) ||
+		((tool == "providers" || tool == "provider") && (len(args) == 1 || (len(args) == 2 && args[1] == "status")))
+	if !isProviderTool || isProviderStatus {
+		loadEnvLocal()
+	}
+	if isProviderTool {
+		if isProviderStatus {
+			if err := printProviderStatus(stdout); err != nil {
+				_, _ = fmt.Fprintln(stderr, err)
+				return 1
+			}
+			return 0
+		}
+		if tool == "status" || len(args) != 3 || (args[1] != "enable" && args[1] != "disable") {
+			_, _ = fmt.Fprintln(stderr, "Usage: webctx providers [status|enable <brave|tavily|exa>|disable <brave|tavily|exa>]")
+			return 1
+		}
+		enabled := args[1] == "enable"
+		if _, err := setSearchProviderEnabled(args[2], enabled); err != nil {
+			_, _ = fmt.Fprintln(stderr, err)
+			return 1
+		}
+		state := "disabled"
+		if enabled {
+			state = "enabled"
+		}
+		_, _ = fmt.Fprintf(stdout, "%s %s. Run 'webctx status' to inspect all providers.\n", strings.ToLower(strings.TrimSpace(args[2])), state)
+		return 0
+	}
+
 	flags, positional := parseArgs(args[1:])
 	input := ""
 	if len(positional) > 0 {
@@ -197,6 +260,11 @@ func usageText() string {
 
 Usage:
   webctx search <query> [--exclude domain1,domain2] [--keyword phrase]
+  webctx status
+  webctx providers [status|enable <brave|tavily|exa>|disable <brave|tavily|exa>]
+  webctx keys list
+  webctx keys add <brave|tavily|exa> --stdin
+  webctx keys remove <brave|tavily|exa> <managed-key-index>
   webctx read-link <url> [--question 'what to find']
   webctx map-site <url> [--query topic] [--limit 30] [--offset 0] [--path-prefix /api] [--language en|all]
   webctx research [question] [--view sources|groups|operations] [--sources IDs] [--include inputs,output,examples] [--limit N] [--offset N]
@@ -208,6 +276,9 @@ Examples:
   webctx search "next.js server components"
   webctx search "react hooks" --exclude youtube.com,vimeo.com
   webctx search "drizzle orm" --keyword "migration guide"
+  webctx providers disable exa
+  webctx providers enable exa
+  webctx keys add exa --stdin < /path/to/private/key.txt
   webctx research "podcast conversations about AI agents"
   webctx research --view sources --limit 5
   webctx inspect particle/podcasts/episodes/search

package/internal/app/app_test.go

@@ -84,9 +84,13 @@ func TestScoreAndRankResultsPreservesInsertionOrderForTies(t *testing.T) {
 }
 
 func TestSearchMissingCredentialsErrorIsHelpful(t *testing.T) {
+	t.Setenv("WEBCTX_CONFIG_FILE", filepath.Join(t.TempDir(), "config.json"))
 	t.Setenv("BRAVE_API_KEY", "")
 	t.Setenv("TAVILY_API_KEY", "")
 	t.Setenv("EXA_API_KEY", "")
+	t.Setenv("BRAVE_API_KEYS", "")
+	t.Setenv("TAVILY_API_KEYS", "")
+	t.Setenv("EXA_API_KEYS", "")
 
 	text, err := Search(SearchParams{Query: "openai api"})
 	if err == nil {
@@ -179,6 +183,9 @@ func TestLoadKeychainEnvLoadsMissingOnly(t *testing.T) {
 	t.Setenv("BRAVE_API_KEY", "already-set")
 	t.Setenv("TAVILY_API_KEY", "")
 	t.Setenv("EXA_API_KEY", "")
+	t.Setenv("BRAVE_API_KEYS", "")
+	t.Setenv("TAVILY_API_KEYS", "")
+	t.Setenv("EXA_API_KEYS", "")
 	t.Setenv("FIRECRAWL_API_KEY", "")
 	t.Setenv("GH_TOKEN", "")
 	t.Setenv("GITHUB_TOKEN", "")

package/internal/app/providers.go

@@ -0,0 +1,165 @@
+package app
+
+import (
+	"encoding/json"
+	"fmt"
+	"io"
+	"os"
+	"path/filepath"
+	"slices"
+	"strings"
+)
+
+// These are the configurable search providers. Firecrawl powers independent
+// read-link, map-site, and Alexandria operations, not federated web search.
+var searchProviderKeys = []struct {
+	name string
+	key  string
+}{
+	{"brave", "BRAVE_API_KEY"},
+	{"tavily", "TAVILY_API_KEY"},
+	{"exa", "EXA_API_KEY"},
+}
+
+type providerConfig struct {
+	DisabledProviders []string `json:"disabled_providers,omitempty"`
+}
+
+func providerConfigPath() (string, error) {
+	if custom := strings.TrimSpace(os.Getenv("WEBCTX_CONFIG_FILE")); custom != "" {
+		return custom, nil
+	}
+	dir, err := os.UserConfigDir()
+	if err != nil {
+		return "", fmt.Errorf("locate WebCTX config directory: %w", err)
+	}
+	return filepath.Join(dir, "webctx", "config.json"), nil
+}
+
+func readProviderConfig(path string) (providerConfig, error) {
+	contents, err := os.ReadFile(path)
+	if os.IsNotExist(err) {
+		return providerConfig{}, nil // all providers default to enabled
+	}
+	if err != nil {
+		return providerConfig{}, fmt.Errorf("read provider settings %s: %w", path, err)
+	}
+	var config providerConfig
+	if err := json.Unmarshal(contents, &config); err != nil {
+		return providerConfig{}, fmt.Errorf("invalid provider settings %s: %w", path, err)
+	}
+	return config, nil
+}
+
+func currentProviderConfig() (providerConfig, error) {
+	path, err := providerConfigPath()
+	if err != nil {
+		return providerConfig{}, err
+	}
+	return readProviderConfig(path)
+}
+
+func (c providerConfig) enabled(name string) bool {
+	return !slices.Contains(c.DisabledProviders, name)
+}
+
+func validSearchProvider(name string) bool {
+	for _, p := range searchProviderKeys {
+		if p.name == name {
+			return true
+		}
+	}
+	return false
+}
+
+func setSearchProviderEnabled(name string, enabled bool) (string, error) {
+	name = strings.ToLower(strings.TrimSpace(name))
+	if !validSearchProvider(name) {
+		return "", fmt.Errorf("unknown search provider %q (choose brave, tavily, or exa)", name)
+	}
+	path, err := providerConfigPath()
+	if err != nil {
+		return "", err
+	}
+	config, err := readProviderConfig(path)
+	if err != nil {
+		return "", err
+	}
+	if config.enabled(name) == enabled {
+		return path, nil
+	}
+	if enabled {
+		config.DisabledProviders = slices.DeleteFunc(config.DisabledProviders, func(p string) bool { return p == name })
+	} else {
+		config.DisabledProviders = append(config.DisabledProviders, name)
+	}
+	slices.Sort(config.DisabledProviders)
+	if err := writeProviderConfig(path, config); err != nil {
+		return "", err
+	}
+	return path, nil
+}
+
+func writeProviderConfig(path string, config providerConfig) error {
+	if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
+		return fmt.Errorf("create provider config directory: %w", err)
+	}
+	data, err := json.MarshalIndent(config, "", "  ")
+	if err != nil {
+		return err
+	}
+	file, err := os.CreateTemp(filepath.Dir(path), ".webctx-config-*")
+	if err != nil {
+		return fmt.Errorf("write provider settings: %w", err)
+	}
+	defer os.Remove(file.Name())
+	if err := file.Chmod(0o600); err != nil {
+		_ = file.Close()
+		return err
+	}
+	if _, err := file.Write(append(data, '\n')); err != nil {
+		_ = file.Close()
+		return err
+	}
+	if err := file.Close(); err != nil {
+		return err
+	}
+	if err := os.Rename(file.Name(), path); err != nil {
+		return fmt.Errorf("save provider settings: %w", err)
+	}
+	return nil
+}
+
+func printProviderStatus(out io.Writer) error {
+	path, err := providerConfigPath()
+	if err != nil {
+		return err
+	}
+	config, err := readProviderConfig(path)
+	if err != nil {
+		return err
+	}
+	_, _ = fmt.Fprintf(out, "Search providers (settings: %s)\n", path)
+	for _, p := range searchProviderKeys {
+		state, key := "enabled", "key missing"
+		if !config.enabled(p.name) {
+			state = "disabled"
+		}
+		count := len(searchKeysForProvider(p.name))
+		if count > 0 {
+			key = "key configured"
+			if count > 1 {
+				key = fmt.Sprintf("key configured (%d keys)", count)
+			}
+		}
+		_, _ = fmt.Fprintf(out, "  %-7s %-9s %s\n", p.name, state, key)
+	}
+	_, _ = fmt.Fprintln(out, "\nFirecrawl (page reading, site maps, and Alexandria; separate from search):")
+	if strings.TrimSpace(os.Getenv("FIRECRAWL_API_KEY")) != "" {
+		_, _ = fmt.Fprintln(out, "  key configured")
+	} else {
+		_, _ = fmt.Fprintln(out, "  key missing")
+	}
+	_, _ = fmt.Fprintln(out, "\nKey presence is local configuration only; quotas and API availability are not checked.")
+	return nil
+}