@mini-z/dsh-search-providers 0.1.2 → 0.1.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +150 -100
  2. package/package.json +7 -3
package/README.md CHANGED
@@ -1,113 +1,101 @@
1
- # dsh-search-providers
1
+ # @mini-z/dsh-search-providers
2
2
 
3
- A modular [DeepSeek Harness (DSH)](https://github.com/deepseek-ai/dsh) plugin that takes over `web_search` and `read_page` with an extensible provider registry and automatic failover.
3
+ [![npm version](https://img.shields.io/npm/v/@mini-z/dsh-search-providers.svg)](https://www.npmjs.com/package/@mini-z/dsh-search-providers)
4
+ [![license](https://img.shields.io/npm/l/@mini-z/dsh-search-providers.svg)](./package.json)
5
+
6
+ A modular [DeepSeek Harness (DSH)](https://github.com/deepseek-ai/dsh) plugin that provides `web_search` and `read_page` through multiple search services with capability-aware fallback and cooldown handling.
7
+
8
+ **Codex is the default `web_search` provider.** If its static credential is unavailable, or a request fails, the plugin proceeds through the configured fallback chain.
4
9
 
5
10
  ## Features
6
11
 
7
- - **Takes over native `web_search`**: registers as the DSH web seam search provider.
8
- - **Adds/replaces `read_page`**: fetches clean page content through the active provider.
9
- - **Multi-provider failover + cooldown**: automatic fallback; providers that fail are temporarily skipped to avoid repeated timeouts.
10
- - **Modular providers**: add more by implementing `SearchProvider`.
11
- - **Standard DSH configuration**: secrets via `~/.dsh/.env`, cordis config via `~/.dsh/profiles/<profile>/cordis.patch.yml`.
12
- - **Demo mode**: test wiring without any API key.
12
+ - Replaces the native `web_search` provider through the DSH web seam.
13
+ - Registers `read_page` for clean, structured page extraction.
14
+ - Uses Codex-first search routing by default.
15
+ - Filters fallback candidates by capability, so search-only providers are never used for page fetching.
16
+ - Temporarily cools failing provider capabilities instead of repeatedly calling them.
17
+ - Disables only confirmed authentication failures until DSH restarts.
18
+ - Propagates caller cancellation without cooling providers or continuing fallback.
19
+ - Supports custom active-provider and fallback order configuration.
20
+ - Includes a key-free demo provider for wiring tests.
21
+
22
+ ## Provider support and default routing
23
+
24
+ | Provider | `web_search` | `read_page` | Credential |
25
+ |----------|:------------:|:-----------:|------------|
26
+ | Codex | ✅ | ❌ | Static credential JSON referenced by `CODEX_CREDENTIAL_FILE` |
27
+ | TinyFish | ✅ | ✅ | `TINYFISH_API_KEY` |
28
+ | Tavily | ✅ | ❌ | `TAVILY_API_KEY` |
29
+ | Exa | ✅ | ✅ | `EXA_API_KEY` |
30
+ | Firecrawl | ✅ | ✅ | `FIRECRAWL_API_KEY` |
31
+ | Demo | ✅ | ✅ | None |
32
+
33
+ Default `web_search` order:
34
+
35
+ 1. Codex
36
+ 2. TinyFish
37
+ 3. Tavily
38
+ 4. Exa
39
+ 5. Firecrawl
40
+
41
+ Default `read_page` order:
13
42
 
14
- ## Supported providers
43
+ 1. TinyFish
44
+ 2. Exa
45
+ 3. Firecrawl
15
46
 
16
- | Provider | Search | Fetch | Requires API key |
17
- |----------|--------|-------|------------------|
18
- | TinyFish | ✅ | ✅ | Yes |
19
- | Tavily | ✅ | ❌ | Yes |
20
- | Exa | ✅ | ✅ | Yes |
21
- | Firecrawl| ✅ | ✅ | Yes |
22
- | Codex | ✅ | ❌ | Static credential file in `~/.dsh` |
23
- | Demo | ✅ | ✅ | No |
47
+ Codex and Tavily are skipped automatically for `read_page` because they do not implement page fetching. Providers without usable credentials are skipped without being marked as failed.
24
48
 
25
49
  ## Installation
26
50
 
27
- ### Option 1: Install from npm (when published)
51
+ ### Install from npm
28
52
 
29
53
  ```bash
30
54
  npx -y @deepseek-ai/dsh plugin --profile web add @mini-z/dsh-search-providers
31
55
  ```
32
56
 
33
- ### Option 2: Local / migrate from another machine
57
+ Replace `web` with another DSH profile name when needed. Restart the corresponding DSH process after installation or configuration changes.
34
58
 
35
- 1. Copy the project folder (or the migration zip) to the target machine.
36
- 2. Install dependencies:
59
+ ### Install from a local checkout or tarball
37
60
 
38
- ```bash
39
- cd dsh-search-providers
40
- bun install --frozen-lockfile
41
- ```
42
-
43
- 3. Configure API keys in `~/.dsh/.env` (see [Configuration](#configuration)).
44
-
45
- 4. Link the plugin into your DSH profile:
46
-
47
- ```bash
48
- cd ~/.dsh/profiles/<your-profile>
49
- npx -y @deepseek-ai/dsh plugin add /path/to/dsh-search-providers
50
- ```
51
-
52
- > The built plugin artifacts are already included in `dsh/`, so you do not need to run `bun run build` unless you modify the source.
61
+ ```bash
62
+ npx -y @deepseek-ai/dsh plugin --profile web add /path/to/dsh-search-providers
63
+ ```
53
64
 
54
- ### Option 3: Development install
65
+ The npm package and release tarballs include the built `dsh/` artifacts. A local source checkout only needs rebuilding after source changes:
55
66
 
56
67
  ```bash
57
- git clone <repo-url>
58
- cd dsh-search-providers
59
- bun install
68
+ bun install --frozen-lockfile
60
69
  bun run build
61
70
  ```
62
71
 
63
72
  ## Configuration
64
73
 
65
- This plugin follows DSH conventions: secrets go in `~/.dsh/.env`, and cordis-level config goes in the profile patch file.
66
-
67
- ### Environment variables
74
+ DSH loads secrets from `~/.dsh/.env`. Configure at least one provider:
68
75
 
69
- Put one or more of these in `~/.dsh/.env`:
76
+ ```dotenv
77
+ # Default web_search provider
78
+ CODEX_CREDENTIAL_FILE=/absolute/path/to/codex-credential.json
70
79
 
71
- ```bash
80
+ # Optional fallback providers
72
81
  TINYFISH_API_KEY=...
73
82
  TAVILY_API_KEY=...
74
83
  EXA_API_KEY=...
75
84
  FIRECRAWL_API_KEY=...
76
- CODEX_CREDENTIAL_FILE=C:/Users/you/.dsh/some-codex-auth.json
77
85
  ```
78
86
 
79
- DSH loads this file automatically when the profile starts. Providers are tried in this default priority order:
87
+ API-key providers also accept these optional environment variables, using the corresponding prefix (`TINYFISH`, `TAVILY`, `EXA`, or `FIRECRAWL`):
80
88
 
81
- 1. Codex
82
- 2. TinyFish
83
- 3. Tavily
84
- 4. Exa
85
- 5. Firecrawl
86
-
87
- API-key providers are attempted only when their key is configured. Codex is attempted only when `CODEX_CREDENTIAL_FILE` points to a usable static credential file. Missing or malformed configuration is silently skipped. Confirmed authentication failures disable that provider until DSH restarts; ordinary HTTP 403 responses do not. Rate limits use a 5-minute cooldown, network/timeouts 30 seconds, and other failures 10 seconds. Transient cooldowns are isolated between search and page fetching, and caller cancellation never cools a provider. Successful fallback responses identify cooling and disabled providers explicitly.
88
-
89
- ### Codex credential file
90
-
91
- Codex is fully independent of login or model plugins. Set `CODEX_CREDENTIAL_FILE` to any specific JSON credential file. When the variable is absent, the Codex provider is disabled. The minimal JSON shape is:
92
-
93
- ```json
94
- {
95
- "access_token": "eyJ...",
96
- "account_id": "account-id"
97
- }
89
+ ```dotenv
90
+ EXA_BASE_URL=https://api.exa.ai
91
+ EXA_TIMEOUT_MS=55000
98
92
  ```
99
93
 
100
- The reader explicitly supports top-level fields plus `credential` and `tokens` wrappers. Access-token fields may be named `access_token` or `access`; account fields may be named `account_id` or `accountId`. When the account field is absent, the standard `chatgpt_account_id` JWT claim is used. Extra fields—including `refresh_token`, `expired`, `expires_at`, and `expires`—are ignored.
94
+ Codex supports `CODEX_TIMEOUT_MS`. The default provider request timeout is 55 seconds. Environment variables take precedence over values in the Cordis configuration.
101
95
 
102
- This plugin only reads the file. It never starts a login, refreshes a token, checks local timestamps, or writes credential data. A missing or malformed credential makes Codex unavailable and is skipped. If Codex rejects the credential with HTTP 401/403, Codex is disabled until restart and the normal fallback chain continues. Restrict the file's filesystem permissions to the owning user.
96
+ ### Change the active provider or fallback chain
103
97
 
104
- Codex does not implement `read_page`, and its standalone endpoint ignores this plugin's recency, domain, location, language, and maximum-result filters.
105
-
106
- ### Override the active provider or fallback order
107
-
108
- The active `provider` is always tried first. `fallbackChain` lists the providers to try afterward, is de-duplicated in order, and may be an empty array to disable fallback.
109
-
110
- Edit `~/.dsh/profiles/<profile>/cordis.patch.yml`:
98
+ Add an id-targeted override to `~/.dsh/profiles/<profile>/cordis.patch.yml`:
111
99
 
112
100
  ```yaml
113
101
  - id: search-providers
@@ -119,70 +107,132 @@ Edit `~/.dsh/profiles/<profile>/cordis.patch.yml`:
119
107
  timeoutMs: 55000
120
108
  ```
121
109
 
122
- ## Usage
110
+ Routing rules:
123
111
 
124
- Once the plugin is loaded by DSH, the model can use the native `web_search` and `read_page` tools. No extra configuration is needed.
112
+ - `provider` is always attempted first when it is available and supports the requested capability.
113
+ - `fallbackChain` contains the providers attempted afterward.
114
+ - Duplicate provider ids are removed while preserving order.
115
+ - `fallbackChain: []` disables fallback and uses only the active provider.
116
+ - Providers that do not support the requested capability are omitted from attempts and diagnostics.
125
117
 
126
- ### web_search
118
+ For TinyFish, Tavily, Exa, and Firecrawl, provider entries support `apiKey`, `baseUrl`, `timeoutMs`, and provider-specific `options`. Prefer `~/.dsh/.env` for secrets rather than storing API keys in profile YAML. TinyFish has one additional page-fetch endpoint option:
127
119
 
128
- Ask the model to search the web:
120
+ ```yaml
121
+ - id: search-providers
122
+ config:
123
+ providers:
124
+ tinyfish:
125
+ options:
126
+ fetchBaseUrl: https://api.fetch.tinyfish.ai
127
+ ```
128
+
129
+ Codex only uses `timeoutMs` from provider configuration. Its endpoint is fixed, and it intentionally ignores `apiKey`, `baseUrl`, and `options`; credentials come exclusively from `CODEX_CREDENTIAL_FILE`.
130
+
131
+ ## Fallback, cooldown, and cancellation behavior
132
+
133
+ The plugin rechecks provider state before every attempt instead of relying on a stale snapshot. A provider whose cooldown expires while other providers are being tried may participate again in the same request.
134
+
135
+ | Failure class | Effect |
136
+ |---------------|--------|
137
+ | Confirmed authentication failure | Provider disabled globally until DSH restarts |
138
+ | HTTP 429/432, quota, or rate limit | Current capability cooled for 5 minutes |
139
+ | Network error or timeout | Current capability cooled for 30 seconds |
140
+ | Other request failure | Current capability cooled for 10 seconds |
141
+
142
+ Transient state is capability-scoped: a search failure does not cool `read_page`, and a fetch failure does not cool `web_search`. Authentication disablement is provider-wide.
143
+
144
+ The first eligible provider uses its configured request timeout. Each subsequent fallback attempt has an additional 20-second outer limit, which also stops providers that ignore `AbortSignal`. The registered `read_page` tool has a 200-second aggregate deadline. These request deadlines are separate from the cooldown durations in the table above.
145
+
146
+ Caller cancellation is different from failure: it propagates immediately, preserves the original abort reason, does not continue fallback, and does not alter provider state.
147
+
148
+ Successful search output identifies the provider that actually answered and reports fallback, cooling, and disabled-provider state. Final errors list the providers genuinely attempted and the current live state.
149
+
150
+ ## Codex credential file
151
+
152
+ Codex uses the standalone ChatGPT Codex search endpoint and is independent of model/login plugins. It only reads an existing static credential file; it never starts a login, refreshes a token, checks local expiry timestamps, or writes credential data.
153
+
154
+ Minimal supported JSON:
155
+
156
+ ```json
157
+ {
158
+ "access_token": "eyJ...",
159
+ "account_id": "account-id"
160
+ }
161
+ ```
162
+
163
+ The parser also accepts `credential` and `tokens` wrappers, `access` as an alternative token field, and `accountId` as an alternative account field. If the account id is omitted, the plugin reads the standard `chatgpt_account_id` claim from the access token. Missing or malformed credentials make Codex unavailable. Codex HTTP 401/403 responses are treated as confirmed credential failures and trigger the normal fallback chain.
164
+
165
+ Restrict the credential file to the owning user. Codex does not implement `read_page`, and its standalone endpoint does not apply this plugin's recency, domain, location, language, or maximum-result filters.
166
+
167
+ ## Usage
168
+
169
+ After loading the plugin and configuring at least one provider, ask the model to search:
129
170
 
130
171
  ```text
131
172
  Search the web for "latest Rust memory safety features".
132
173
  ```
133
174
 
134
- The plugin will use the active provider and automatically fall back to the next available provider if the first one fails. Every successful result starts with `[Search provider: ...]`, naming the provider that actually returned it.
175
+ Successful results start with the provider that actually answered:
135
176
 
136
- ### read_page
177
+ ```text
178
+ [Search provider: TinyFish]
179
+ ```
137
180
 
138
- Give the model a URL to read:
181
+ Read a specific page with:
139
182
 
140
183
  ```text
141
184
  Read this page: https://example.com/article
142
185
  ```
143
186
 
144
- The plugin will fetch the page content through the active provider (or the first available fetch-capable provider in the fallback chain).
187
+ `read_page` returns structured evidence containing a summary, extracted content, outgoing links, uncertainty, and operational warnings.
145
188
 
146
- > Note: Tavily and Codex support search only, so they are skipped for `read_page`.
189
+ ## Demo mode
147
190
 
148
- ## Demo mode (test without API key)
191
+ Run DSH with the key-free demo provider on POSIX shells:
149
192
 
150
193
  ```bash
151
194
  SEARCH_PROVIDERS_DEMO=1 npx -y @deepseek-ai/dsh --profile web --no-open --port 9876
152
195
  ```
153
196
 
154
- Then ask the model to run `web_search` or `read_page`; it will receive canned demo results.
197
+ PowerShell:
198
+
199
+ ```powershell
200
+ $env:SEARCH_PROVIDERS_DEMO = '1'
201
+ npx -y @deepseek-ai/dsh --profile web --no-open --port 9876
202
+ ```
203
+
204
+ Alternatively set `provider: demo` in the plugin configuration. Demo mode returns canned search and page-fetch results for integration testing.
155
205
 
156
206
  ## Development
157
207
 
158
208
  ```bash
159
- bun install
209
+ bun install --frozen-lockfile
160
210
  bun run typecheck
161
211
  bun run test
162
212
  bun run build
163
213
  ```
164
214
 
165
- ## Architecture
215
+ Main components:
166
216
 
167
- - `src/providers/types.ts` — `SearchProvider` interface.
168
- - `src/providers/registry.ts` — provider registry and active-provider selection.
169
- - `src/providers/cooldown.ts` — failure tracking and cooldown logic.
170
- - `src/providers/tinyfish.ts`, `tavily.ts`, `exa.ts`, `firecrawl.ts`, `codex.ts` — provider implementations.
171
- - `src/providers/demo.ts` — canned demo provider for testing.
172
- - `src/tools/web-search.ts` — registers the plugin as `ctx.web.registerSearchProvider` with failover.
173
- - `src/tools/read-page.ts` — registers the `read_page` tool with failover.
174
- - `src/index.ts` — DSH plugin entry (`apply`).
217
+ - `src/providers/types.ts` — provider interfaces and result types.
218
+ - `src/providers/registry.ts` — provider registration and active-provider selection.
219
+ - `src/providers/cooldown.ts` — authentication disablement and capability cooldown state.
220
+ - `src/providers/*.ts` — provider implementations.
221
+ - `src/tools/web-search.ts` — DSH web seam adapter and search fallback.
222
+ - `src/tools/read-page.ts` — `read_page` registration and fetch fallback.
223
+ - `src/index.ts` — plugin entry point and default provider order.
175
224
 
176
- ## Adding a provider
177
-
178
- Implement the `SearchProvider` interface, register it in `src/index.ts`, and add its config resolution.
225
+ To add a provider, implement `SearchProvider`, register it in `src/index.ts`, and add its configuration resolution.
179
226
 
180
227
  ## Publishing
181
228
 
182
229
  1. Bump the version in `package.json`.
183
- 2. `npm login`
184
- 3. `bun publish --access public`
230
+ 2. Run the development checks and inspect `bun pm pack --dry-run`.
231
+ 3. Log in with `npm login`.
232
+ 4. Publish the public scoped package with `npm publish --access public`.
233
+
234
+ Published npm versions are immutable; every release requires a new version number.
185
235
 
186
236
  ## License
187
237
 
188
- MIT. The Codex interoperability implementation was developed with reference to [dsh-codex-connect](https://github.com/franksong2702/dsh-codex-connect) (Apache-2.0); no runtime dependency on that package is required.
238
+ MIT. The Codex interoperability implementation was developed with reference to [dsh-codex-connect](https://github.com/franksong2702/dsh-codex-connect) (Apache-2.0); no runtime dependency on that package is required.
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "@mini-z/dsh-search-providers",
3
- "version": "0.1.2",
4
- "description": "Modular DSH web search plugin with TinyFish as the default provider. Takes over web_search and read_page with an extensible provider registry.",
3
+ "version": "0.1.3",
4
+ "description": "Modular DSH search plugin with Codex-first web search, page fetching, and capability-aware automatic failover.",
5
5
  "type": "module",
6
6
  "scripts": {
7
7
  "dev": "vite build --watch",
@@ -20,14 +20,18 @@
20
20
  "keywords": [
21
21
  "search",
22
22
  "web-fetch",
23
+ "codex",
23
24
  "tinyfish",
24
25
  "dsh-plugin",
25
26
  "deepseek-harness",
26
27
  "web-search",
27
28
  "modular-search"
28
29
  ],
29
- "author": "",
30
+ "author": "mini-z",
30
31
  "license": "MIT",
32
+ "publishConfig": {
33
+ "access": "public"
34
+ },
31
35
  "engines": {
32
36
  "node": ">=22.13"
33
37
  },