webctx 0.2.8 → 0.2.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +10 -1
- package/CONTRIBUTORS.md +2 -2
- package/README.md +16 -2
- package/internal/app/app.go +35 -2
- package/internal/app/app_test.go +1 -0
- package/internal/app/providers.go +161 -0
- package/internal/app/providers_test.go +302 -0
- package/internal/app/tools.go +87 -34
- package/package.json +1 -1
package/AGENTS.md
CHANGED
|
@@ -47,13 +47,14 @@ Direct commands:
|
|
|
47
47
|
Preserve these commands unless the user explicitly asks to change them:
|
|
48
48
|
|
|
49
49
|
- `webctx search <query> [--exclude domain1,domain2] [--keyword phrase]`
|
|
50
|
+
- `webctx status` and `webctx providers enable|disable <brave|tavily|exa>`
|
|
50
51
|
- `webctx read-link <url>`
|
|
51
52
|
- `webctx map-site <url>`
|
|
52
53
|
- `webctx --version`
|
|
53
54
|
|
|
54
55
|
Behavioral expectations:
|
|
55
56
|
|
|
56
|
-
- `search` combines Brave, Tavily, and Exa results, then re-ranks them with duplicate-aware scoring.
|
|
57
|
+
- `search` combines enabled Brave, Tavily, and Exa results, then re-ranks them with duplicate-aware scoring; an unavailable Exa can fall back to quoted-phrase Brave/Tavily queries for `--keyword` (not a strict text filter).
|
|
57
58
|
- `read-link` keeps the current GitHub raw-content and `.md` fast paths. Firecrawl uses its built-in 30-minute cache, automatic and enhanced proxies, then browser extraction as a final fallback.
|
|
58
59
|
- `map-site` keeps the current Firecrawl map request settings.
|
|
59
60
|
- The CLI should remain agent-friendly and emit plain markdown/text output.
|
|
@@ -100,3 +101,11 @@ When cutting a release, update `src/content/docs/changelog.md` before tagging.
|
|
|
100
101
|
- Do not include docs-site-only changes such as site styling, Zuedocs/package bumps, deploy plumbing, footer/layout changes, or documentation navigation changes.
|
|
101
102
|
- Rewrite commit subjects into clear release notes instead of pasting raw commit messages.
|
|
102
103
|
- If a release contains only tagging/release metadata, write: `Maintenance release. No direct code behavior changes beyond release preparation.`
|
|
104
|
+
|
|
105
|
+
### Docs deployment
|
|
106
|
+
|
|
107
|
+
The Docs GitHub Actions workflow deploys production only for the watched site paths; Vercel Git auto-deployments and `ignoreCommand` stay disabled. Keep the project-scoped `VERCEL_TOKEN` Actions secret and ID variables configured. Check Astro locally; builds run in CI.
|
|
108
|
+
|
|
109
|
+
### Origo deployment
|
|
110
|
+
|
|
111
|
+
The separate `Origo API` GitHub Actions workflow deploys the Go function only for changes to its import graph and Vercel configuration. Use `VERCEL_ORIGO_TOKEN` and `VERCEL_ORIGO_PROJECT_ID` exclusively for Origo; never substitute the WebCTX Docs token or a static `--prebuilt` site bundle. Keep the `/mcp` rewrite and Origo production environment variables intact.
|
package/CONTRIBUTORS.md
CHANGED
|
@@ -108,9 +108,9 @@ gh workflow run npm-publish-retry.yml --repo amxv/webctx --ref main -f version=0
|
|
|
108
108
|
|
|
109
109
|
## Origo deployment
|
|
110
110
|
|
|
111
|
-
The `origo-api` Vercel project
|
|
111
|
+
The `origo-api` Vercel project uses `ORIGO_DEPLOYMENT=1` to select its Go Function configuration. It needs `ORIGO_API_KEY` and `FIRECRAWL_API_KEY` production environment variables; optionally add `GH_TOKEN` for native authenticated GitHub reads. The docs project remains `webctx-docs` and never receives the Origo connection key.
|
|
112
112
|
|
|
113
|
-
|
|
113
|
+
The independent `Docs` and `Origo API` GitHub Actions workflows filter their own build inputs before deploying to Vercel. Origo uses the `VERCEL_ORIGO_TOKEN` Actions secret and `VERCEL_ORIGO_PROJECT_ID` variable, separate from the docs credentials. Git-triggered Vercel deployments are disabled; `v*` tags still drive CLI releases.
|
|
114
114
|
|
|
115
115
|
## Notes on package naming
|
|
116
116
|
|
package/README.md
CHANGED
|
@@ -95,7 +95,7 @@ Recognized native GitHub auth, private/not-found, and rate-limit failures stay a
|
|
|
95
95
|
|
|
96
96
|
## Search
|
|
97
97
|
|
|
98
|
-
Normal search asks Brave, Tavily, and Exa, removes duplicate URLs, automatically reads promising sources, follows relevant links, and returns excerpts plus the ranked list. Alexandria discovery is included when available
|
|
98
|
+
Normal search asks the enabled Brave, Tavily, and Exa providers, removes duplicate URLs, automatically reads promising sources, follows relevant links, and returns excerpts plus the ranked list. One working search provider is enough; Alexandria discovery is included when available. Use `execute` when you need provider-native data.
|
|
99
99
|
|
|
100
100
|
```bash
|
|
101
101
|
webctx search "next.js server components"
|
|
@@ -103,6 +103,20 @@ webctx search "react hooks" --exclude youtube.com,medium.com
|
|
|
103
103
|
webctx search "drizzle orm" --keyword "migration guide"
|
|
104
104
|
```
|
|
105
105
|
|
|
106
|
+
### Enable or disable search providers
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
webctx status # enabled/disabled providers and locally configured keys
|
|
110
|
+
webctx providers disable exa # stop all Exa search requests, even with a key present
|
|
111
|
+
webctx providers enable exa # turn Exa back on later
|
|
112
|
+
webctx providers disable brave
|
|
113
|
+
webctx providers enable brave
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
Provider settings persist across runs, independently of environment variables and Keychain credentials. The settings file lives in the OS user config directory (`webctx/config.json`); set `WEBCTX_CONFIG_FILE` to override its location. Disabled providers are never queried. `status` only reports key presence, **not** live credit balances or API health.
|
|
117
|
+
|
|
118
|
+
`--keyword` prefers Exa's strict include-text filter when available. If Exa is disabled, unconfigured, or fails, Brave/Tavily search for the quoted phrase instead; the output explicitly notes that this fallback cannot guarantee the phrase appears in the page text. Firecrawl is a separate reading/site-map/Alexandria integration, not one of these three search providers.
|
|
119
|
+
|
|
106
120
|
## Map a site
|
|
107
121
|
|
|
108
122
|
Use `map-site` when you want to discover the useful pages before reading them.
|
|
@@ -131,7 +145,7 @@ FIRECRAWL_API_KEY=...
|
|
|
131
145
|
GH_TOKEN=...
|
|
132
146
|
```
|
|
133
147
|
|
|
134
|
-
- `BRAVE_API_KEY`, `TAVILY_API_KEY`, and `EXA_API_KEY` power normal `search`.
|
|
148
|
+
- `BRAVE_API_KEY`, `TAVILY_API_KEY`, and `EXA_API_KEY` power normal `search`. Configure any one or more, and optionally disable providers you no longer want to query.
|
|
135
149
|
- `FIRECRAWL_API_KEY` powers `map-site`, page-crawl fallbacks, and Alexandria discovery/inspection/paid execution.
|
|
136
150
|
- `GH_TOKEN` or `GITHUB_TOKEN` is optional. It increases GitHub capacity and unlocks reads such as blame, Discussions, some Actions job logs, private resources the token can access, and richer PR review state.
|
|
137
151
|
|
package/internal/app/app.go
CHANGED
|
@@ -16,8 +16,6 @@ const commandName = "webctx"
|
|
|
16
16
|
var version = buildinfo.CurrentVersion()
|
|
17
17
|
|
|
18
18
|
func Run(args []string, stdout, stderr io.Writer) int {
|
|
19
|
-
loadEnvLocal()
|
|
20
|
-
|
|
21
19
|
if len(args) == 0 || isHelpArg(args[0]) {
|
|
22
20
|
_, _ = fmt.Fprintln(stdout, usageText())
|
|
23
21
|
return 0
|
|
@@ -29,6 +27,37 @@ func Run(args []string, stdout, stderr io.Writer) int {
|
|
|
29
27
|
}
|
|
30
28
|
|
|
31
29
|
tool := args[0]
|
|
30
|
+
isProviderTool := tool == "status" || tool == "providers" || tool == "provider"
|
|
31
|
+
isProviderStatus := (tool == "status" && len(args) == 1) ||
|
|
32
|
+
((tool == "providers" || tool == "provider") && (len(args) == 1 || (len(args) == 2 && args[1] == "status")))
|
|
33
|
+
if !isProviderTool || isProviderStatus {
|
|
34
|
+
loadEnvLocal()
|
|
35
|
+
}
|
|
36
|
+
if isProviderTool {
|
|
37
|
+
if isProviderStatus {
|
|
38
|
+
if err := printProviderStatus(stdout); err != nil {
|
|
39
|
+
_, _ = fmt.Fprintln(stderr, err)
|
|
40
|
+
return 1
|
|
41
|
+
}
|
|
42
|
+
return 0
|
|
43
|
+
}
|
|
44
|
+
if tool == "status" || len(args) != 3 || (args[1] != "enable" && args[1] != "disable") {
|
|
45
|
+
_, _ = fmt.Fprintln(stderr, "Usage: webctx providers [status|enable <brave|tavily|exa>|disable <brave|tavily|exa>]")
|
|
46
|
+
return 1
|
|
47
|
+
}
|
|
48
|
+
enabled := args[1] == "enable"
|
|
49
|
+
if _, err := setSearchProviderEnabled(args[2], enabled); err != nil {
|
|
50
|
+
_, _ = fmt.Fprintln(stderr, err)
|
|
51
|
+
return 1
|
|
52
|
+
}
|
|
53
|
+
state := "disabled"
|
|
54
|
+
if enabled {
|
|
55
|
+
state = "enabled"
|
|
56
|
+
}
|
|
57
|
+
_, _ = fmt.Fprintf(stdout, "%s %s. Run 'webctx status' to inspect all providers.\n", strings.ToLower(strings.TrimSpace(args[2])), state)
|
|
58
|
+
return 0
|
|
59
|
+
}
|
|
60
|
+
|
|
32
61
|
flags, positional := parseArgs(args[1:])
|
|
33
62
|
input := ""
|
|
34
63
|
if len(positional) > 0 {
|
|
@@ -197,6 +226,8 @@ func usageText() string {
|
|
|
197
226
|
|
|
198
227
|
Usage:
|
|
199
228
|
webctx search <query> [--exclude domain1,domain2] [--keyword phrase]
|
|
229
|
+
webctx status
|
|
230
|
+
webctx providers [status|enable <brave|tavily|exa>|disable <brave|tavily|exa>]
|
|
200
231
|
webctx read-link <url> [--question 'what to find']
|
|
201
232
|
webctx map-site <url> [--query topic] [--limit 30] [--offset 0] [--path-prefix /api] [--language en|all]
|
|
202
233
|
webctx research [question] [--view sources|groups|operations] [--sources IDs] [--include inputs,output,examples] [--limit N] [--offset N]
|
|
@@ -208,6 +239,8 @@ Examples:
|
|
|
208
239
|
webctx search "next.js server components"
|
|
209
240
|
webctx search "react hooks" --exclude youtube.com,vimeo.com
|
|
210
241
|
webctx search "drizzle orm" --keyword "migration guide"
|
|
242
|
+
webctx providers disable exa
|
|
243
|
+
webctx providers enable exa
|
|
211
244
|
webctx research "podcast conversations about AI agents"
|
|
212
245
|
webctx research --view sources --limit 5
|
|
213
246
|
webctx inspect particle/podcasts/episodes/search
|
package/internal/app/app_test.go
CHANGED
|
@@ -84,6 +84,7 @@ func TestScoreAndRankResultsPreservesInsertionOrderForTies(t *testing.T) {
|
|
|
84
84
|
}
|
|
85
85
|
|
|
86
86
|
func TestSearchMissingCredentialsErrorIsHelpful(t *testing.T) {
|
|
87
|
+
t.Setenv("WEBCTX_CONFIG_FILE", filepath.Join(t.TempDir(), "config.json"))
|
|
87
88
|
t.Setenv("BRAVE_API_KEY", "")
|
|
88
89
|
t.Setenv("TAVILY_API_KEY", "")
|
|
89
90
|
t.Setenv("EXA_API_KEY", "")
|
|
@@ -0,0 +1,161 @@
|
|
|
1
|
+
package app
|
|
2
|
+
|
|
3
|
+
import (
|
|
4
|
+
"encoding/json"
|
|
5
|
+
"fmt"
|
|
6
|
+
"io"
|
|
7
|
+
"os"
|
|
8
|
+
"path/filepath"
|
|
9
|
+
"slices"
|
|
10
|
+
"strings"
|
|
11
|
+
)
|
|
12
|
+
|
|
13
|
+
// These are the configurable search providers. Firecrawl powers independent
|
|
14
|
+
// read-link, map-site, and Alexandria operations, not federated web search.
|
|
15
|
+
var searchProviderKeys = []struct {
|
|
16
|
+
name string
|
|
17
|
+
key string
|
|
18
|
+
}{
|
|
19
|
+
{"brave", "BRAVE_API_KEY"},
|
|
20
|
+
{"tavily", "TAVILY_API_KEY"},
|
|
21
|
+
{"exa", "EXA_API_KEY"},
|
|
22
|
+
}
|
|
23
|
+
|
|
24
|
+
type providerConfig struct {
|
|
25
|
+
DisabledProviders []string `json:"disabled_providers,omitempty"`
|
|
26
|
+
}
|
|
27
|
+
|
|
28
|
+
func providerConfigPath() (string, error) {
|
|
29
|
+
if custom := strings.TrimSpace(os.Getenv("WEBCTX_CONFIG_FILE")); custom != "" {
|
|
30
|
+
return custom, nil
|
|
31
|
+
}
|
|
32
|
+
dir, err := os.UserConfigDir()
|
|
33
|
+
if err != nil {
|
|
34
|
+
return "", fmt.Errorf("locate WebCTX config directory: %w", err)
|
|
35
|
+
}
|
|
36
|
+
return filepath.Join(dir, "webctx", "config.json"), nil
|
|
37
|
+
}
|
|
38
|
+
|
|
39
|
+
func readProviderConfig(path string) (providerConfig, error) {
|
|
40
|
+
contents, err := os.ReadFile(path)
|
|
41
|
+
if os.IsNotExist(err) {
|
|
42
|
+
return providerConfig{}, nil // all providers default to enabled
|
|
43
|
+
}
|
|
44
|
+
if err != nil {
|
|
45
|
+
return providerConfig{}, fmt.Errorf("read provider settings %s: %w", path, err)
|
|
46
|
+
}
|
|
47
|
+
var config providerConfig
|
|
48
|
+
if err := json.Unmarshal(contents, &config); err != nil {
|
|
49
|
+
return providerConfig{}, fmt.Errorf("invalid provider settings %s: %w", path, err)
|
|
50
|
+
}
|
|
51
|
+
return config, nil
|
|
52
|
+
}
|
|
53
|
+
|
|
54
|
+
func currentProviderConfig() (providerConfig, error) {
|
|
55
|
+
path, err := providerConfigPath()
|
|
56
|
+
if err != nil {
|
|
57
|
+
return providerConfig{}, err
|
|
58
|
+
}
|
|
59
|
+
return readProviderConfig(path)
|
|
60
|
+
}
|
|
61
|
+
|
|
62
|
+
func (c providerConfig) enabled(name string) bool {
|
|
63
|
+
return !slices.Contains(c.DisabledProviders, name)
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
func validSearchProvider(name string) bool {
|
|
67
|
+
for _, p := range searchProviderKeys {
|
|
68
|
+
if p.name == name {
|
|
69
|
+
return true
|
|
70
|
+
}
|
|
71
|
+
}
|
|
72
|
+
return false
|
|
73
|
+
}
|
|
74
|
+
|
|
75
|
+
func setSearchProviderEnabled(name string, enabled bool) (string, error) {
|
|
76
|
+
name = strings.ToLower(strings.TrimSpace(name))
|
|
77
|
+
if !validSearchProvider(name) {
|
|
78
|
+
return "", fmt.Errorf("unknown search provider %q (choose brave, tavily, or exa)", name)
|
|
79
|
+
}
|
|
80
|
+
path, err := providerConfigPath()
|
|
81
|
+
if err != nil {
|
|
82
|
+
return "", err
|
|
83
|
+
}
|
|
84
|
+
config, err := readProviderConfig(path)
|
|
85
|
+
if err != nil {
|
|
86
|
+
return "", err
|
|
87
|
+
}
|
|
88
|
+
if config.enabled(name) == enabled {
|
|
89
|
+
return path, nil
|
|
90
|
+
}
|
|
91
|
+
if enabled {
|
|
92
|
+
config.DisabledProviders = slices.DeleteFunc(config.DisabledProviders, func(p string) bool { return p == name })
|
|
93
|
+
} else {
|
|
94
|
+
config.DisabledProviders = append(config.DisabledProviders, name)
|
|
95
|
+
}
|
|
96
|
+
slices.Sort(config.DisabledProviders)
|
|
97
|
+
if err := writeProviderConfig(path, config); err != nil {
|
|
98
|
+
return "", err
|
|
99
|
+
}
|
|
100
|
+
return path, nil
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
func writeProviderConfig(path string, config providerConfig) error {
|
|
104
|
+
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
|
|
105
|
+
return fmt.Errorf("create provider config directory: %w", err)
|
|
106
|
+
}
|
|
107
|
+
data, err := json.MarshalIndent(config, "", " ")
|
|
108
|
+
if err != nil {
|
|
109
|
+
return err
|
|
110
|
+
}
|
|
111
|
+
file, err := os.CreateTemp(filepath.Dir(path), ".webctx-config-*")
|
|
112
|
+
if err != nil {
|
|
113
|
+
return fmt.Errorf("write provider settings: %w", err)
|
|
114
|
+
}
|
|
115
|
+
defer os.Remove(file.Name())
|
|
116
|
+
if err := file.Chmod(0o600); err != nil {
|
|
117
|
+
_ = file.Close()
|
|
118
|
+
return err
|
|
119
|
+
}
|
|
120
|
+
if _, err := file.Write(append(data, '\n')); err != nil {
|
|
121
|
+
_ = file.Close()
|
|
122
|
+
return err
|
|
123
|
+
}
|
|
124
|
+
if err := file.Close(); err != nil {
|
|
125
|
+
return err
|
|
126
|
+
}
|
|
127
|
+
if err := os.Rename(file.Name(), path); err != nil {
|
|
128
|
+
return fmt.Errorf("save provider settings: %w", err)
|
|
129
|
+
}
|
|
130
|
+
return nil
|
|
131
|
+
}
|
|
132
|
+
|
|
133
|
+
func printProviderStatus(out io.Writer) error {
|
|
134
|
+
path, err := providerConfigPath()
|
|
135
|
+
if err != nil {
|
|
136
|
+
return err
|
|
137
|
+
}
|
|
138
|
+
config, err := readProviderConfig(path)
|
|
139
|
+
if err != nil {
|
|
140
|
+
return err
|
|
141
|
+
}
|
|
142
|
+
_, _ = fmt.Fprintf(out, "Search providers (settings: %s)\n", path)
|
|
143
|
+
for _, p := range searchProviderKeys {
|
|
144
|
+
state, key := "enabled", "key missing"
|
|
145
|
+
if !config.enabled(p.name) {
|
|
146
|
+
state = "disabled"
|
|
147
|
+
}
|
|
148
|
+
if strings.TrimSpace(os.Getenv(p.key)) != "" {
|
|
149
|
+
key = "key configured"
|
|
150
|
+
}
|
|
151
|
+
_, _ = fmt.Fprintf(out, " %-7s %-9s %s\n", p.name, state, key)
|
|
152
|
+
}
|
|
153
|
+
_, _ = fmt.Fprintln(out, "\nFirecrawl (page reading, site maps, and Alexandria; separate from search):")
|
|
154
|
+
if strings.TrimSpace(os.Getenv("FIRECRAWL_API_KEY")) != "" {
|
|
155
|
+
_, _ = fmt.Fprintln(out, " key configured")
|
|
156
|
+
} else {
|
|
157
|
+
_, _ = fmt.Fprintln(out, " key missing")
|
|
158
|
+
}
|
|
159
|
+
_, _ = fmt.Fprintln(out, "\nKey presence is local configuration only; quotas and API availability are not checked.")
|
|
160
|
+
return nil
|
|
161
|
+
}
|
|
@@ -0,0 +1,302 @@
|
|
|
1
|
+
package app
|
|
2
|
+
|
|
3
|
+
import (
|
|
4
|
+
"bytes"
|
|
5
|
+
"compress/gzip"
|
|
6
|
+
"context"
|
|
7
|
+
"encoding/json"
|
|
8
|
+
"io"
|
|
9
|
+
"net/http"
|
|
10
|
+
"net/http/httptest"
|
|
11
|
+
"os"
|
|
12
|
+
"path/filepath"
|
|
13
|
+
"strings"
|
|
14
|
+
"sync"
|
|
15
|
+
"testing"
|
|
16
|
+
)
|
|
17
|
+
|
|
18
|
+
func TestProviderCommandsPersistAndReport(t *testing.T) {
|
|
19
|
+
path := filepath.Join(t.TempDir(), "webctx", "config.json")
|
|
20
|
+
t.Setenv("WEBCTX_CONFIG_FILE", path)
|
|
21
|
+
t.Setenv("BRAVE_API_KEY", "test-brave-secret")
|
|
22
|
+
t.Setenv("TAVILY_API_KEY", "")
|
|
23
|
+
t.Setenv("EXA_API_KEY", "test-exa-secret")
|
|
24
|
+
t.Setenv("FIRECRAWL_API_KEY", "")
|
|
25
|
+
|
|
26
|
+
run := func(args ...string) (int, string, string) {
|
|
27
|
+
t.Helper()
|
|
28
|
+
var out, errOut bytes.Buffer
|
|
29
|
+
code := Run(args, &out, &errOut)
|
|
30
|
+
return code, out.String(), errOut.String()
|
|
31
|
+
}
|
|
32
|
+
if code, text, errText := run("status"); code != 0 || !strings.Contains(text, "exa enabled") || !strings.Contains(text, "brave enabled") || errText != "" {
|
|
33
|
+
t.Fatalf("default status: code=%d text=%q error=%q", code, text, errText)
|
|
34
|
+
}
|
|
35
|
+
if _, err := os.Stat(path); !os.IsNotExist(err) {
|
|
36
|
+
t.Fatalf("status should not create config file, got error %v", err)
|
|
37
|
+
}
|
|
38
|
+
if code, text, errText := run("providers", "disable", "ExA"); code != 0 || !strings.Contains(text, "exa disabled") || errText != "" {
|
|
39
|
+
t.Fatalf("disable: code=%d text=%q error=%q", code, text, errText)
|
|
40
|
+
}
|
|
41
|
+
if code, text, _ := run("providers"); code != 0 || !strings.Contains(text, "exa disabled") || !strings.Contains(text, "key configured") {
|
|
42
|
+
t.Fatalf("persisted status: code=%d text=%q", code, text)
|
|
43
|
+
} else if strings.Contains(text, "test-exa-secret") || strings.Contains(text, "test-brave-secret") {
|
|
44
|
+
t.Fatalf("status exposed a credential: %q", text)
|
|
45
|
+
}
|
|
46
|
+
content, err := os.ReadFile(path)
|
|
47
|
+
if err != nil {
|
|
48
|
+
t.Fatal(err)
|
|
49
|
+
}
|
|
50
|
+
var config providerConfig
|
|
51
|
+
if err := json.Unmarshal(content, &config); err != nil || !strings.EqualFold(strings.Join(config.DisabledProviders, ","), "exa") {
|
|
52
|
+
t.Fatalf("settings not persisted: %s; error=%v", content, err)
|
|
53
|
+
}
|
|
54
|
+
if info, err := os.Stat(path); err != nil || info.Mode().Perm() != 0o600 {
|
|
55
|
+
t.Fatalf("config permissions: info=%v error=%v", info, err)
|
|
56
|
+
}
|
|
57
|
+
if code, text, errText := run("providers", "enable", "exa"); code != 0 || !strings.Contains(text, "exa enabled") || errText != "" {
|
|
58
|
+
t.Fatalf("enable: code=%d text=%q error=%q", code, text, errText)
|
|
59
|
+
}
|
|
60
|
+
if code, text, _ := run("status"); code != 0 || !strings.Contains(text, "exa enabled") {
|
|
61
|
+
t.Fatalf("restored status: code=%d text=%q", code, text)
|
|
62
|
+
}
|
|
63
|
+
}
|
|
64
|
+
|
|
65
|
+
func TestProviderCommandsRejectInvalidInputAndMalformedConfig(t *testing.T) {
|
|
66
|
+
path := filepath.Join(t.TempDir(), "config.json")
|
|
67
|
+
t.Setenv("WEBCTX_CONFIG_FILE", path)
|
|
68
|
+
for _, args := range [][]string{{"providers", "disable", "firecrawl"}, {"providers", "disable", "unknown"}, {"providers", "disable"}, {"status", "enable", "exa"}} {
|
|
69
|
+
var out, errOut bytes.Buffer
|
|
70
|
+
if Run(args, &out, &errOut) == 0 || errOut.Len() == 0 {
|
|
71
|
+
t.Fatalf("invalid command %q succeeded: %q", args, out.String())
|
|
72
|
+
}
|
|
73
|
+
}
|
|
74
|
+
if _, err := os.Stat(path); !os.IsNotExist(err) {
|
|
75
|
+
t.Fatalf("invalid command wrote config: %v", err)
|
|
76
|
+
}
|
|
77
|
+
if err := os.WriteFile(path, []byte("{invalid"), 0o600); err != nil {
|
|
78
|
+
t.Fatal(err)
|
|
79
|
+
}
|
|
80
|
+
var out, errOut bytes.Buffer
|
|
81
|
+
if Run([]string{"status"}, &out, &errOut) == 0 || !strings.Contains(errOut.String(), "invalid provider settings") {
|
|
82
|
+
t.Fatalf("malformed config not reported: %q", errOut.String())
|
|
83
|
+
}
|
|
84
|
+
if Run([]string{"providers", "disable", "exa"}, &out, &errOut) == 0 {
|
|
85
|
+
t.Fatal("malformed config should not be overwritten")
|
|
86
|
+
}
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
type providerRequests struct {
|
|
90
|
+
mu sync.Mutex
|
|
91
|
+
queries map[string][]string
|
|
92
|
+
failedExa bool
|
|
93
|
+
emptyExa bool
|
|
94
|
+
}
|
|
95
|
+
|
|
96
|
+
func mockSearchProviders(t *testing.T, exaFailure bool) *providerRequests {
|
|
97
|
+
t.Helper()
|
|
98
|
+
prev := http.DefaultClient.Transport
|
|
99
|
+
stats := &providerRequests{queries: make(map[string][]string), failedExa: exaFailure}
|
|
100
|
+
http.DefaultClient.Transport = roundTripFunc(func(req *http.Request) (*http.Response, error) {
|
|
101
|
+
provider, query := "", ""
|
|
102
|
+
switch req.URL.Host {
|
|
103
|
+
case "api.search.brave.com":
|
|
104
|
+
provider, query = "brave", req.URL.Query().Get("q")
|
|
105
|
+
case "api.tavily.com", "api.exa.ai":
|
|
106
|
+
if req.URL.Host == "api.tavily.com" {
|
|
107
|
+
provider = "tavily"
|
|
108
|
+
} else {
|
|
109
|
+
provider = "exa"
|
|
110
|
+
}
|
|
111
|
+
data, _ := io.ReadAll(req.Body)
|
|
112
|
+
var payload map[string]any
|
|
113
|
+
_ = json.Unmarshal(data, &payload)
|
|
114
|
+
query, _ = payload["query"].(string)
|
|
115
|
+
case "provider-test.example":
|
|
116
|
+
return testHTTPResponse(req, http.StatusOK, "# Example documentation\n\nThis markdown page contains enough content for a test.", map[string]string{"Content-Type": "text/markdown"}), nil
|
|
117
|
+
default:
|
|
118
|
+
return testHTTPResponse(req, http.StatusNotFound, "", nil), nil
|
|
119
|
+
}
|
|
120
|
+
stats.mu.Lock()
|
|
121
|
+
stats.queries[provider] = append(stats.queries[provider], query)
|
|
122
|
+
stats.mu.Unlock()
|
|
123
|
+
switch provider {
|
|
124
|
+
case "brave":
|
|
125
|
+
return testHTTPResponse(req, http.StatusOK, `{"web":{"results":[{"url":"https://provider-test.example/guide.md","title":"Guide","description":"Documentation"}]}}`, nil), nil
|
|
126
|
+
case "tavily":
|
|
127
|
+
return testHTTPResponse(req, http.StatusOK, `{"results":[]}`, nil), nil
|
|
128
|
+
case "exa":
|
|
129
|
+
if stats.failedExa {
|
|
130
|
+
return testHTTPResponse(req, http.StatusPaymentRequired, `{"error":"NO_MORE_CREDITS"}`, nil), nil
|
|
131
|
+
}
|
|
132
|
+
if stats.emptyExa {
|
|
133
|
+
return testHTTPResponse(req, http.StatusOK, `{"results":[]}`, nil), nil
|
|
134
|
+
}
|
|
135
|
+
return testHTTPResponse(req, http.StatusOK, `{"results":[{"url":"https://provider-test.example/guide.md","title":"Guide","text":"Documentation"}]}`, nil), nil
|
|
136
|
+
}
|
|
137
|
+
return testHTTPResponse(req, http.StatusNotFound, "", nil), nil
|
|
138
|
+
})
|
|
139
|
+
t.Cleanup(func() { http.DefaultClient.Transport = prev })
|
|
140
|
+
return stats
|
|
141
|
+
}
|
|
142
|
+
|
|
143
|
+
func (p *providerRequests) received(provider string) []string {
|
|
144
|
+
p.mu.Lock()
|
|
145
|
+
defer p.mu.Unlock()
|
|
146
|
+
return append([]string(nil), p.queries[provider]...)
|
|
147
|
+
}
|
|
148
|
+
|
|
149
|
+
func searchTestEnv(t *testing.T) {
|
|
150
|
+
t.Helper()
|
|
151
|
+
t.Setenv("WEBCTX_CONFIG_FILE", filepath.Join(t.TempDir(), "config.json"))
|
|
152
|
+
t.Setenv("BRAVE_API_KEY", "test-brave")
|
|
153
|
+
t.Setenv("TAVILY_API_KEY", "test-tavily")
|
|
154
|
+
t.Setenv("EXA_API_KEY", "test-exa")
|
|
155
|
+
t.Setenv("FIRECRAWL_API_KEY", "")
|
|
156
|
+
}
|
|
157
|
+
|
|
158
|
+
func TestSearchDoesNotCallDisabledExa(t *testing.T) {
|
|
159
|
+
searchTestEnv(t)
|
|
160
|
+
stats := mockSearchProviders(t, false)
|
|
161
|
+
if _, err := setSearchProviderEnabled("exa", false); err != nil {
|
|
162
|
+
t.Fatal(err)
|
|
163
|
+
}
|
|
164
|
+
result, err := Search(SearchParams{Query: "runtime docs"})
|
|
165
|
+
if err != nil || !strings.Contains(result, "Total Results: 1") {
|
|
166
|
+
t.Fatalf("search without Exa failed: %v; %q", err, result)
|
|
167
|
+
}
|
|
168
|
+
if got := stats.received("exa"); len(got) != 0 {
|
|
169
|
+
t.Fatalf("Exa was contacted despite being disabled: %v", got)
|
|
170
|
+
}
|
|
171
|
+
if got := stats.received("brave"); len(got) != 1 || got[0] != "runtime docs" {
|
|
172
|
+
t.Fatalf("Brave was not used: %v", got)
|
|
173
|
+
}
|
|
174
|
+
if got := stats.received("tavily"); len(got) != 1 {
|
|
175
|
+
t.Fatalf("Tavily was not used: %v", got)
|
|
176
|
+
}
|
|
177
|
+
}
|
|
178
|
+
|
|
179
|
+
func TestSearchSupportsOnlyBraveEnabled(t *testing.T) {
|
|
180
|
+
searchTestEnv(t)
|
|
181
|
+
stats := mockSearchProviders(t, false)
|
|
182
|
+
for _, name := range []string{"exa", "tavily"} {
|
|
183
|
+
if _, err := setSearchProviderEnabled(name, false); err != nil {
|
|
184
|
+
t.Fatal(err)
|
|
185
|
+
}
|
|
186
|
+
}
|
|
187
|
+
result, err := Search(SearchParams{Query: "runtime docs"})
|
|
188
|
+
if err != nil || !strings.Contains(result, "Total Results: 1") {
|
|
189
|
+
t.Fatalf("single-provider search failed: %v; %q", err, result)
|
|
190
|
+
}
|
|
191
|
+
if len(stats.received("exa")) != 0 || len(stats.received("tavily")) != 0 || len(stats.received("brave")) != 1 {
|
|
192
|
+
t.Fatalf("unexpected requests: %#v", stats.queries)
|
|
193
|
+
}
|
|
194
|
+
}
|
|
195
|
+
|
|
196
|
+
func TestKeywordFallbackSkipsDisabledExa(t *testing.T) {
|
|
197
|
+
searchTestEnv(t)
|
|
198
|
+
stats := mockSearchProviders(t, false)
|
|
199
|
+
if _, err := setSearchProviderEnabled("exa", false); err != nil {
|
|
200
|
+
t.Fatal(err)
|
|
201
|
+
}
|
|
202
|
+
result, err := Search(SearchParams{Query: "drizzle orm", IncludeKeyword: "migration guide"})
|
|
203
|
+
if err != nil || !strings.Contains(result, "Keyword fallback") {
|
|
204
|
+
t.Fatalf("keyword fallback failed: %v; %q", err, result)
|
|
205
|
+
}
|
|
206
|
+
if len(stats.received("exa")) != 0 {
|
|
207
|
+
t.Fatal("disabled Exa received keyword request")
|
|
208
|
+
}
|
|
209
|
+
for _, provider := range []string{"brave", "tavily"} {
|
|
210
|
+
got := stats.received(provider)
|
|
211
|
+
if len(got) != 1 || !strings.Contains(got[0], `"migration guide"`) {
|
|
212
|
+
t.Fatalf("%s did not receive a quoted phrase: %v", provider, got)
|
|
213
|
+
}
|
|
214
|
+
}
|
|
215
|
+
}
|
|
216
|
+
|
|
217
|
+
func TestKeywordFallbackOnExaCreditExhaustion(t *testing.T) {
|
|
218
|
+
searchTestEnv(t)
|
|
219
|
+
stats := mockSearchProviders(t, true)
|
|
220
|
+
result, err := Search(SearchParams{Query: "drizzle orm", IncludeKeyword: "migration guide"})
|
|
221
|
+
if err != nil || !strings.Contains(result, "Keyword fallback") || !strings.Contains(result, "Total Results: 1") {
|
|
222
|
+
t.Fatalf("keyword fallback after Exa error failed: %v; %q", err, result)
|
|
223
|
+
}
|
|
224
|
+
if len(stats.received("exa")) != 1 || len(stats.received("brave")) != 1 {
|
|
225
|
+
t.Fatalf("unexpected provider request sequence: %#v", stats.queries)
|
|
226
|
+
}
|
|
227
|
+
}
|
|
228
|
+
|
|
229
|
+
func TestKeywordExaSuccessDoesNotContactOtherProviders(t *testing.T) {
|
|
230
|
+
searchTestEnv(t)
|
|
231
|
+
stats := mockSearchProviders(t, false)
|
|
232
|
+
result, err := Search(SearchParams{Query: "drizzle orm", IncludeKeyword: "migration guide"})
|
|
233
|
+
if err != nil || strings.Contains(result, "Keyword fallback") || !strings.Contains(result, "Total Results: 1") {
|
|
234
|
+
t.Fatalf("unexpected keyword results: %v; %q", err, result)
|
|
235
|
+
}
|
|
236
|
+
if len(stats.received("exa")) != 1 || len(stats.received("brave")) != 0 || len(stats.received("tavily")) != 0 {
|
|
237
|
+
t.Fatalf("unexpected provider requests: %#v", stats.queries)
|
|
238
|
+
}
|
|
239
|
+
}
|
|
240
|
+
|
|
241
|
+
func TestKeywordExaNoMatchesDoesNotBroadenSearch(t *testing.T) {
|
|
242
|
+
searchTestEnv(t)
|
|
243
|
+
stats := mockSearchProviders(t, false)
|
|
244
|
+
stats.emptyExa = true
|
|
245
|
+
_, err := Search(SearchParams{Query: "drizzle orm", IncludeKeyword: "migration guide"})
|
|
246
|
+
if err == nil || !strings.Contains(err.Error(), "failed to return results") {
|
|
247
|
+
t.Fatalf("unexpected result for zero Exa matches: %v", err)
|
|
248
|
+
}
|
|
249
|
+
if len(stats.received("exa")) != 1 || len(stats.received("brave")) != 0 || len(stats.received("tavily")) != 0 {
|
|
250
|
+
t.Fatalf("strict keyword search was broadened: %#v", stats.queries)
|
|
251
|
+
}
|
|
252
|
+
}
|
|
253
|
+
|
|
254
|
+
func TestSearchWithAllProvidersDisabledFailsFast(t *testing.T) {
|
|
255
|
+
searchTestEnv(t)
|
|
256
|
+
stats := mockSearchProviders(t, false)
|
|
257
|
+
for _, name := range []string{"brave", "tavily", "exa"} {
|
|
258
|
+
if _, err := setSearchProviderEnabled(name, false); err != nil {
|
|
259
|
+
t.Fatal(err)
|
|
260
|
+
}
|
|
261
|
+
}
|
|
262
|
+
_, err := Search(SearchParams{Query: "anything"})
|
|
263
|
+
if err == nil || !strings.Contains(err.Error(), "all search providers are disabled") {
|
|
264
|
+
t.Fatalf("unexpected error when all disabled: %v", err)
|
|
265
|
+
}
|
|
266
|
+
if len(stats.received("exa"))+len(stats.received("tavily"))+len(stats.received("brave")) != 0 {
|
|
267
|
+
t.Fatal("search contacted provider with all disabled")
|
|
268
|
+
}
|
|
269
|
+
}
|
|
270
|
+
|
|
271
|
+
func TestBraveSearchTransparentlyDecodesGzip(t *testing.T) {
|
|
272
|
+
t.Setenv("BRAVE_API_KEY", "test-brave")
|
|
273
|
+
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
274
|
+
if r.Header.Get("Accept-Encoding") != "gzip" {
|
|
275
|
+
t.Errorf("expected Go HTTP transport to request gzip, got %q", r.Header.Get("Accept-Encoding"))
|
|
276
|
+
}
|
|
277
|
+
w.Header().Set("Content-Encoding", "gzip")
|
|
278
|
+
w.Header().Set("Content-Type", "application/json")
|
|
279
|
+
writer := gzip.NewWriter(w)
|
|
280
|
+
_, _ = writer.Write([]byte(`{"web":{"results":[{"url":"https://example.com/guide","title":"Guide","description":"Documentation"}]}}`))
|
|
281
|
+
_ = writer.Close()
|
|
282
|
+
}))
|
|
283
|
+
defer server.Close()
|
|
284
|
+
|
|
285
|
+
originalTransport := http.DefaultClient.Transport
|
|
286
|
+
http.DefaultClient.Transport = roundTripFunc(func(req *http.Request) (*http.Response, error) {
|
|
287
|
+
if req.Header.Get("Accept-Encoding") != "" {
|
|
288
|
+
t.Errorf("Brave search forced Accept-Encoding instead of letting Go decode it: %q", req.Header.Get("Accept-Encoding"))
|
|
289
|
+
}
|
|
290
|
+
copy := req.Clone(req.Context())
|
|
291
|
+
copy.URL.Scheme = "http"
|
|
292
|
+
copy.URL.Host = strings.TrimPrefix(server.URL, "http://")
|
|
293
|
+
copy.Host = copy.URL.Host
|
|
294
|
+
return http.DefaultTransport.RoundTrip(copy)
|
|
295
|
+
})
|
|
296
|
+
t.Cleanup(func() { http.DefaultClient.Transport = originalTransport })
|
|
297
|
+
|
|
298
|
+
result, err := searchWithBrave(context.Background(), "Go docs")
|
|
299
|
+
if err != nil || len(result.Docs) != 1 || result.Docs[0].Title != "Guide" {
|
|
300
|
+
t.Fatalf("Brave gzip response not decoded: %v; %#v", err, result)
|
|
301
|
+
}
|
|
302
|
+
}
|
package/internal/app/tools.go
CHANGED
|
@@ -10,6 +10,7 @@ import (
|
|
|
10
10
|
"net/url"
|
|
11
11
|
"os"
|
|
12
12
|
"sort"
|
|
13
|
+
"strconv"
|
|
13
14
|
"strings"
|
|
14
15
|
"sync"
|
|
15
16
|
"time"
|
|
@@ -37,6 +38,11 @@ type providerDocs struct {
|
|
|
37
38
|
Err error
|
|
38
39
|
}
|
|
39
40
|
|
|
41
|
+
type namedSearch struct {
|
|
42
|
+
name string
|
|
43
|
+
fn func(context.Context) (SearchResult, error)
|
|
44
|
+
}
|
|
45
|
+
|
|
40
46
|
type scoredURL struct {
|
|
41
47
|
Result SearchDoc
|
|
42
48
|
InsertionOrder int
|
|
@@ -47,6 +53,13 @@ type scoredURL struct {
|
|
|
47
53
|
}
|
|
48
54
|
|
|
49
55
|
func Search(params SearchParams) (string, error) {
|
|
56
|
+
config, err := currentProviderConfig()
|
|
57
|
+
if err != nil {
|
|
58
|
+
return "", err
|
|
59
|
+
}
|
|
60
|
+
if !config.enabled("brave") && !config.enabled("tavily") && !config.enabled("exa") {
|
|
61
|
+
return "", errors.New("Error searching the web: all search providers are disabled; run 'webctx providers enable brave', 'webctx providers enable tavily', or 'webctx providers enable exa'")
|
|
62
|
+
}
|
|
50
63
|
defaultExcludedDomains := []string{"youtube.com", "vimeo.com", "dailymotion.com", "twitch.tv", "tiktok.com", "instagram.com", "facebook.com"}
|
|
51
64
|
allExcludedDomains := append(append([]string{}, defaultExcludedDomains...), params.ExcludeDomains...)
|
|
52
65
|
truncatedKeyword := truncateWords(params.IncludeKeyword, 5)
|
|
@@ -56,46 +69,44 @@ func Search(params SearchParams) (string, error) {
|
|
|
56
69
|
alexandriaDone := make(chan string, 1)
|
|
57
70
|
go func() { alexandriaDone <- discoverAlexandria(params.Query) }()
|
|
58
71
|
|
|
59
|
-
type namedSearch struct {
|
|
60
|
-
name string
|
|
61
|
-
fn func(context.Context) (SearchResult, error)
|
|
62
|
-
}
|
|
63
|
-
|
|
64
72
|
searches := []namedSearch{}
|
|
73
|
+
keywordFallback := false
|
|
65
74
|
if strings.TrimSpace(truncatedKeyword) != "" {
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
75
|
+
if config.enabled("exa") && strings.TrimSpace(os.Getenv("EXA_API_KEY")) != "" {
|
|
76
|
+
searches = append(searches, namedSearch{name: "Exa", fn: func(ctx context.Context) (SearchResult, error) {
|
|
77
|
+
return searchWithExa(ctx, params.Query, nil, truncatedKeyword)
|
|
78
|
+
}})
|
|
79
|
+
}
|
|
69
80
|
} else {
|
|
70
|
-
searches = append(searches,
|
|
71
|
-
|
|
72
|
-
namedSearch{name: "
|
|
73
|
-
|
|
74
|
-
|
|
81
|
+
searches = append(searches, alternativeSearches(config, params.Query)...)
|
|
82
|
+
if config.enabled("exa") {
|
|
83
|
+
searches = append(searches, namedSearch{name: "Exa", fn: func(ctx context.Context) (SearchResult, error) {
|
|
84
|
+
return searchWithExa(ctx, params.Query, nil, "")
|
|
85
|
+
}})
|
|
86
|
+
}
|
|
75
87
|
}
|
|
76
88
|
|
|
77
|
-
results :=
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
results
|
|
91
|
-
}
|
|
89
|
+
results := runSearches(searches)
|
|
90
|
+
if truncatedKeyword != "" && (len(searches) == 0 || (len(results) == 1 && results[0].Err != nil)) {
|
|
91
|
+
// Exa's includeText filter has no equivalent in Brave or Tavily. When Exa
|
|
92
|
+
// is disabled, unconfigured, or fails, search for the quoted phrase instead
|
|
93
|
+
// of making keyword searches unusable. Disclose the weaker semantics.
|
|
94
|
+
fallbackQuery := params.Query + " " + strconv.Quote(truncatedKeyword)
|
|
95
|
+
fallbackSearches := alternativeSearches(config, fallbackQuery)
|
|
96
|
+
if len(fallbackSearches) > 0 {
|
|
97
|
+
results = append(results, runSearches(fallbackSearches)...)
|
|
98
|
+
keywordFallback = true
|
|
99
|
+
} else if len(searches) == 0 && config.enabled("exa") {
|
|
100
|
+
// Preserve the helpful missing-key error if Exa is the only enabled
|
|
101
|
+
// provider and it has no key.
|
|
102
|
+
results = append(results, providerDocs{Provider: "Exa", Err: errors.New("missing EXA_API_KEY")})
|
|
103
|
+
}
|
|
92
104
|
}
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
total := 0
|
|
96
|
-
for _, r := range results {
|
|
97
|
-
total += len(r.Docs)
|
|
105
|
+
if len(results) == 0 {
|
|
106
|
+
return "", errors.New("Error searching the web: no enabled search providers are available for this query; use 'webctx status' to inspect settings")
|
|
98
107
|
}
|
|
108
|
+
|
|
109
|
+
total := countSearchDocs(results)
|
|
99
110
|
if total == 0 {
|
|
100
111
|
missingKeys := uniqueMissingCredentialKeys(results)
|
|
101
112
|
if len(missingKeys) > 0 {
|
|
@@ -123,6 +134,9 @@ func Search(params SearchParams) (string, error) {
|
|
|
123
134
|
}
|
|
124
135
|
|
|
125
136
|
parts := []string{fmt.Sprintf("Total Results: %d\n", len(ranked))}
|
|
137
|
+
if keywordFallback {
|
|
138
|
+
parts = append(parts, "> **Keyword fallback:** Exa was disabled or unavailable. Brave/Tavily searched for the quoted phrase; this is not a strict page-text inclusion filter.", "")
|
|
139
|
+
}
|
|
126
140
|
// The ranked index remains available, but agents also get useful source
|
|
127
141
|
// context without having to issue several follow-up read-link calls.
|
|
128
142
|
if context := enrichSearch(params.Query, ranked); context != "" {
|
|
@@ -144,6 +158,46 @@ func Search(params SearchParams) (string, error) {
|
|
|
144
158
|
return strings.Join(parts, "\n"), nil
|
|
145
159
|
}
|
|
146
160
|
|
|
161
|
+
func alternativeSearches(config providerConfig, query string) []namedSearch {
|
|
162
|
+
searches := []namedSearch{}
|
|
163
|
+
if config.enabled("brave") {
|
|
164
|
+
searches = append(searches, namedSearch{name: "Brave", fn: func(ctx context.Context) (SearchResult, error) { return searchWithBrave(ctx, query) }})
|
|
165
|
+
}
|
|
166
|
+
if config.enabled("tavily") {
|
|
167
|
+
searches = append(searches, namedSearch{name: "Tavily", fn: func(ctx context.Context) (SearchResult, error) { return searchWithTavily(ctx, query, nil) }})
|
|
168
|
+
}
|
|
169
|
+
return searches
|
|
170
|
+
}
|
|
171
|
+
|
|
172
|
+
func countSearchDocs(results []providerDocs) int {
|
|
173
|
+
count := 0
|
|
174
|
+
for _, r := range results {
|
|
175
|
+
count += len(r.Docs)
|
|
176
|
+
}
|
|
177
|
+
return count
|
|
178
|
+
}
|
|
179
|
+
|
|
180
|
+
func runSearches(searches []namedSearch) []providerDocs {
|
|
181
|
+
results := make([]providerDocs, len(searches))
|
|
182
|
+
var wg sync.WaitGroup
|
|
183
|
+
for i, s := range searches {
|
|
184
|
+
wg.Add(1)
|
|
185
|
+
go func(i int, s namedSearch) {
|
|
186
|
+
defer wg.Done()
|
|
187
|
+
ctx, cancel := context.WithTimeout(context.Background(), 40*time.Second)
|
|
188
|
+
defer cancel()
|
|
189
|
+
res, err := s.fn(ctx)
|
|
190
|
+
if err != nil {
|
|
191
|
+
results[i] = providerDocs{Provider: s.name, Docs: []SearchDoc{}, Err: err}
|
|
192
|
+
return
|
|
193
|
+
}
|
|
194
|
+
results[i] = providerDocs{Provider: s.name, Docs: res.Docs}
|
|
195
|
+
}(i, s)
|
|
196
|
+
}
|
|
197
|
+
wg.Wait()
|
|
198
|
+
return results
|
|
199
|
+
}
|
|
200
|
+
|
|
147
201
|
func ReadLink(rawURL string) (string, error) {
|
|
148
202
|
if err := validateSourceURL(rawURL); err != nil {
|
|
149
203
|
return "", err
|
|
@@ -338,7 +392,6 @@ func searchWithBrave(ctx context.Context, query string) (SearchResult, error) {
|
|
|
338
392
|
params.Set("limit", "20")
|
|
339
393
|
body, err := doRawRequest(ctx, http.MethodGet, "https://api.search.brave.com/res/v1/web/search?"+params.Encode(), map[string]string{
|
|
340
394
|
"Accept": "application/json",
|
|
341
|
-
"Accept-Encoding": "gzip",
|
|
342
395
|
"x-subscription-token": apiKey,
|
|
343
396
|
}, nil)
|
|
344
397
|
if err != nil {
|