deepseek-web-retrieval 0.2.0 → 0.4.0

package/CHANGELOG.md

@@ -2,6 +2,208 @@
 
 All notable changes to this project are documented here.
 
+## 0.4.0
+
+- Ground the search defaults in surveyed industry factory defaults
+  ([research](docs/research/search-api-defaults.md)): every surveyed AI-native search API
+  returns ten results by default and none filters results client-side by score or
+  authority, so `maxResults` moves 5 → 10 (matching the API `count` default) and the
+  `minRerank`/`minAuthority` thresholds move to 0 with `allowUnknownAuthority` flipping
+  to `true` — the pass-through default returns what DeepSeek ranked, and the thresholds
+  remain available as opt-ins. `DEEPSEEK_SEARCH_ALLOW_UNKNOWN_AUTHORITY` becomes a
+  false-only switch (`false`/`0`/`no`/`off` disables) since the default is now on.
+  Unchanged and now evidence-backed: `maxRetries=0` (search-API SDK cohort: tavily-python
+  and Linkup ship no retries; the 2-retry defaults belong to chat-completion SDKs), the
+  30 s timeout, and the 1,200/16,000-character truncations
+- Drop the "Token- and cost-efficient" bullet from the README's Features section —
+  the default parameters are no longer endorsed as a selling point, and their behavior
+  is stated neutrally in General behavior
+
+## 0.3.0
+
+- Close the lifecycle gaps in the keyless e2e coverage with a `reinit` scenario: tier
+  downgrades revert orphaned artifacts through the real CLI (the Codex kill switch
+  restored inside the still-claimed config with the registration kept and the plan's
+  reverts section visible; the Pi preferred → optional downgrade deletes the orphaned,
+  installer-created rules file whole), a second Codex scope claiming the same config
+  file is refused with uninstall guidance, `list` reports the recorded installs, and
+  `XDG_CONFIG_HOME` routes the global config home with uninstall finding the record
+  there. The Codex project scope and OpenCode global scope path mappings are exercised
+  for the first time
+- Refuse drift between plan and apply (ADR-0001): apply now re-verifies that the value
+  at every set-op path still equals what the plan previewed, and refuses with re-run
+  guidance when the config changed since the plan was rendered — the preview, the secret
+  guard, and the install record describe one snapshot taken once at plan time, and an
+  interim user edit can never be silently overwritten. A file that stopped parsing in
+  that window refuses the same way
+- Deepen the modules the installer and adapters are built on, behavior-preserving
+  except where noted: revert and the cross-scope collision invariant each get exactly
+  one owner (a whole-file orphan revert now deletes a created file it leaves empty,
+  matching uninstall's semantics; partial orphan reverts never delete a still-claimed
+  file); the three config formats — comment-preserving JSONC, comment-lossy TOML, and
+  the host-marked rules block — sit behind one codec registry, with a plan's file edits
+  as a discriminated union instead of optional-field smuggling; each host module owns
+  its per-scope paths, tool ids, and shared plan header, the CLI roster derives from
+  the planner registry, and `docs/hosts.jsonc` records which hosts have installer
+  modules with test guards holding both in sync; the DeepSeek HTTP transport moves
+  behind its own seam with an injectable fetch, the per-call timeout becomes part of
+  the request context (the shorter of it and the configured timeout wins, for search
+  as well as fetch), and the OpenCode adapter stops rebuilding resolved options; the
+  core returns a host-neutral search result, owns the capability-fact description
+  fragments both adapters compose from, and resolves OpenCode's registration knobs
+  inside the OpenCode adapter where they are consumed; the `DEEPSEEK_*` environment
+  variable names have a single source in the core constants
+- Make the installer CLI edge testable and explicit: command output is injectable,
+  input validation returns a discriminated result instead of smuggling exit codes as
+  numbers, and the config home is an explicit input with the XDG comparison resolved
+  once at the entry (behavior unchanged)
+- Track the project's domain documentation: `CONTEXT.md` (the glossary hosts, adapters,
+  tiers, plans, records, reverts, codecs are written in) and `docs/adr/` now live in
+  the repository, alongside the `docs/agents/` agent-skills configuration (GitHub
+  Issues as the tracker, default triage labels, single-context domain docs)
+- Enforce the forced tier's exclusivity promise and complete the detect phase: other
+  registered MCP servers are now classified web-capable from their name plus
+  command/args/url (a heuristic the plan notes state openly). Under a forced tier, Pi
+  disables the web-capable ones via their per-server `enabled` key with the prior value
+  snapshotted for uninstall, while Codex — which has no per-server kill switch — refuses
+  the tier with guidance instead of promising non-exclusive routing (OpenCode's forced is
+  replacement, which no third-party server overrides). The target rules file is scanned
+  for steering toward other web tools (the installer's own marked block excluded) and the
+  conflict is reported, not resolved; every forced-tier plan names the README's "Privacy
+  and cost" section; the both-off error and the stay-registered/replacement notes become
+  shared strings instead of per-planner copies
+- Refuse cross-scope installs that would claim one config file: Codex's project-scope
+  fallback writes the user-level `~/.codex/config.toml`, so a project install beside a
+  global one left both scopes' records claiming `mcp_servers.deepseek` in that file, and
+  whichever uninstalled first reverted the other's registration. The plan now refuses a
+  second scope whose edits overlap the other scope's recorded files (pointing at the
+  uninstall command to run instead), and uninstall runs the same check defensively for
+  records that predate the guard. Pi and OpenCode scope pairs map to disjoint files and
+  install side by side as before
+- Split installer config editing into `configFile` (JSONC text editing) and
+  `configValue` (the parsed-object model), unifying the duplicated path walkers,
+  append-array dedupe, created-parents pruning, and the isPlainObject guard along the
+  way (no behavior change); add a guard test that every version literal in installer
+  prose appears in `docs/hosts.jsonc`, so a version bump without the re-verification
+  pass fails tests instead of shipping stale facts in user-facing plan notes
+- Close the remaining high- and medium-value e2e coverage gaps with four new Docker
+  scenarios: `installer` (keyless — the installer CLI's own product: init/apply/uninstall
+  for all three hosts, verified through the real agents' config readers, including the
+  Codex kill-switch write and the OpenCode engine replacement), `pkg` (keyless — the
+  `npm pack` tarball installed with npm in the container, its bin executed directly,
+  then registered as the Pi server for a real handshake), `capability-off` (keyless —
+  `DEEPSEEK_ENABLE_FETCH=false` leaves exactly the search tool, both flags false refuses
+  to start, and a registration without `exposure` reports `codemode`), and `preferred`
+  (billed — installer product at Pi project scope with preferred tiers, then
+  natural-language prompts that name no tool, so only the AGENTS.md rules block can steer
+  the model). The harness now accepts multiple scenario names, packs the npm tarball
+  before running, gives each scenario an isolated HOME and project dir (cleaned up
+  afterwards) while sharing one npm download cache per run, and suggests the keyless
+  subset when `DEEPSEEK_API_KEY` is missing
+- Fix the installer's secret guard refusing its own output: a re-init over an existing
+  entry whose `DEEPSEEK_API_KEY` is the `${DEEPSEEK_API_KEY}` indirection form was
+  rejected as a literal credential. Pure `${VAR}` expansions now pass the guard (found
+  by the new installer e2e scenario running after pi-billed in a shared HOME); literal
+  keys are still refused
+- Fix the `codex` e2e scenario, which had never actually completed a turn: the slim
+  Node base image ships no system CA bundle, so Codex's static-musl TLS stack (rustls,
+  unlike Node's bundled store) failed every HTTPS request as an opaque
+  "Reconnecting... waiting for network" loop — the Dockerfile now installs
+  `ca-certificates`. The scenario config also follows DeepSeek's official Codex
+  integration: a vendored `test/e2e/codex-models.json` catalog is written to
+  `~/.codex/models.json` and referenced via `model_catalog_json` (without it Codex
+  cannot resolve DeepSeek model slugs), with `wire_api = "responses"` and the key
+  still forwarded through `env_key`/`env_vars` instead of the official script's
+  bearer-token-in-file. The scenario now exercises the `web_search = "disabled"`
+  kill switch for real. All four scenarios (pi, pi-billed, opencode, codex) pass in
+  Docker with real billed requests
+- Add the billed `pi-billed` e2e scenario, bringing Pi to the same invocation-level
+  coverage as the opencode and codex scenarios: headless `pi --print` prompts whose
+  LLM is Pi's native DeepSeek provider (`deepseek/deepseek-flash`), so one key pays
+  for both legs and the model really calls `deepseek_fetch` / `deepseek_web_search`
+  through Pi's own MCP client. The keyless `pi` handshake scenario stays
+- Drop Continue and Gemini from the host matrix — they will not be supported.
+  Both generated tables, the per-host reference, the protocol research, the
+  installer examples, and the host-count invariants shrink accordingly (the
+  MCP-capable host count is now seven)
+- Fix the critical code-review findings on the installer: a tier-change re-init now
+  reverts the previous init's artifacts explicitly (shown in the plan, undone on
+  apply) instead of orphaning them; the plan renders old values for every set op and
+  existing counts for appends; TOML comment-loss detection covers trailing comments
+  (string-aware); rules-block markers are per host so pi and codex can share one
+  AGENTS.md; install records are written 0600 with `.deepseek-web-retrieval/`
+  gitignored; replacing an entry that contains a literal credential-looking key is
+  refused instead of snapshotted; the CLI resolves `--cwd`, falls back to
+  `os.homedir()`, rejects unknown flags/positionals with targeted messages, and
+  apply refuses to clobber non-array/non-object config values before writing
+  anything; the e2e runner rejects stale flags (including `--local`) and forwards
+  empty env values as empty instead of passing host values through
+- Add the installer CLI (`deepseek-web-retrieval init` / `uninstall` / `list`) for
+  OpenCode, Pi, and Codex: plan-then-apply with per-capability tiers (optional /
+  preferred / forced / off) and a first-class project/global scope, JSON edits that
+  preserve comments (jsonc-parser), TOML support (smol-toml), snapshot-based surgical
+  rollback that never clobbers user edits, and detect notes for competitors, other
+  registered MCP servers, and Codex's `[tools] web_search` no-op trap
+- Add `DEEPSEEK_ENABLE_SEARCH` / `DEEPSEEK_ENABLE_FETCH` (and the OpenCode
+  `searchEnabled` / `fetchEnabled` options) for per-capability control on every host;
+  the MCP server refuses to start when both tools are disabled
+- Promote Codex to supported: MCP-server adapter, the `docs/agents/codex.md` guide,
+  tier table with the top-level `web_search = "disabled"` kill switch, and
+  `env_vars`-based key forwarding that respects Codex's stdio env allowlist
+- Make the e2e suites Docker-only (remove `--local`) and add the billed `codex`
+  scenario: headless `codex exec` in an isolated HOME with DeepSeek as the Codex LLM
+  over the Responses wire (Codex 0.163.0 dropped `wire_api = "chat"`; DeepSeek serves
+  the Responses endpoint), one key paying for both legs
+- Replace the hand-written JSONC stripper with jsonc-parser (now also
+  `src/core/jsonc.ts`); drop `scripts/jsonc.ts`
+
+- Fix documentation review findings: resolve the Pi `structuredContent` contradiction
+  against the v1.3.0 source (content-first, structuredContent as a no-content
+  fallback), add the missing `extension` mechanism to the taxonomy and re-path
+  OpenCode to its native mechanisms, qualify the Forced tier in the Tiers table
+  itself, de-restate version locks and the baseline in CONTRIBUTING/integration,
+  replace "planned" with "shipped" for the MCP server, tighten installer rollback
+  and forced-unavailable/session-scoped semantics, and dedupe the provenance and
+  quirk summaries
+- State the installer principles — never overwrite user configuration without the
+  user's knowledge, and post-install behavior and impact scope must be controllable
+  and knowable — and specify the detect/merge/verify flow, tier semantics against
+  stock configuration, snapshot-based rollback, and a first-class project/global
+  scope choice (honor or report, never silently downgrade) in
+  `docs/design/installer-design.md`; qualify the control matrix's exclusion flavor as
+  relative to built-in competitors only
+- Categorize `docs/`: per-agent guides in `docs/agents/`, research notes in
+  `docs/research/`, and design notes in `docs/design/`; the integration hub and
+  `docs/hosts.jsonc` stay at the root
+- Add `docs/research/control-matrix.md`: control-tier research (optional / preferred /
+  forced, per host and capability) with a consolidated per-host reference — built-in web
+  tools and their kill switches, MCP registration, and rules files. The control matrix is
+  generated from the new `control` field in `docs/hosts.jsonc` (`bun run docs:matrix`;
+  drift fails tests)
+- Correct the Continue host note (the CLI has no built-in `search_web`; it is
+  IDE-extension-only) and state the built-in web tools in the Codex, Cursor, Devin, and
+  Copilot notes; extend the host-evaluation checklist with the rules and built-in
+  kill-switch questions
+- Move install-time tier planning to `docs/design/installer-design.md`, and fold the
+  `docs/research/mcp-compatibility.md` per-host setup notes and quirk catalog into the
+  control-matrix per-host reference; `integration.md` host notes now link there instead
+  of restating details
+- Fix the hosts.jsonc drift guard: the matrix renderer no longer rewrites
+  docs/integration.md when imported by tests, and CI and `bun run check` now fail
+  on drift (`docs:matrix --check`)
+- Serve both MCP tools end to end at every pinned client revision, not only
+  `listTools`
+- Build `dist/cli.js` on demand for the stdio smoke test instead of silently
+  skipping it on a fresh checkout
+- Share the fetch pipeline (parse, host policy, fetch, format) between the
+  OpenCode and MCP adapters through `fetchDocument` in the core
+- Stop restating host version locks in narrative docs; reference
+  `docs/hosts.jsonc` instead
+- Pin the e2e Docker base image (`node:22.23.3-slim`) and correct the 0.2.0
+  changelog wording about where e2e versions are pinned
+- Type the `DEEPSEEK_*` environment mapping so option keys and variable names
+  cannot drift apart
+
 ## 0.2.0
 
 - Make `docs/hosts.jsonc` the single source of truth for host facts: the
@@ -11,7 +213,8 @@ All notable changes to this project are documented here.
 - Add a local-only end-to-end script (`bun run test:e2e`): a keyless real
   `pi mcp list` handshake against the built server, and a headless `opencode run`
   scenario that makes real billed requests; never run in CI. Agents run inside
-  Docker with versions pinned by `test/e2e/Dockerfile` (`--local` opts out)
+  Docker with versions injected from `docs/hosts.jsonc` as Docker build args
+  (`--local` opts out)
 - Add an MCP server adapter (`src/mcp/`) exposing `deepseek_web_search` and `deepseek_fetch`
   over stdio, serving MCP clients from revision 2024-11-05 through 2026-07-28
 - Support `DEEPSEEK_*` environment variable configuration in the MCP server, mapped onto the

package/README.md

@@ -9,46 +9,74 @@ coding agents are in progress.
 
 This is an unofficial community project. It is not affiliated with OpenCode or DeepSeek.
 
+## Features
+
+- **One core, every host** — a native OpenCode v2 plugin plus a stdio MCP
+  server for everything else; identical retrieval behavior on each
+- **Tiered routing** — optional, preferred, forced, or off, per capability;
+  forced refuses when exclusivity is impossible
+- **Plan-then-apply installer** — every write previewed as old → new before
+  touching disk; arrays merge instead of replacing; uninstall reverses
+  exactly what was applied
+- **Trap-aware detection** — catches what the agents' own docs don't warn
+  about: Codex's silent `[tools] web_search` no-op, Pi's codemode hiding MCP
+  tools; conflicting rules and web-capable servers surface in the plan
+- **Git-safe installs** — the key never enters config files
+  (`${DEEPSEEK_API_KEY}` indirection or `env_vars`), so a project-scope
+  install commits clean
+- **Fails loudly, never falls back** — no silent provider switches;
+  localhost and private URLs blocked before they leave the machine
+
 ## Usage
 
 Shipped adapters:
 
-- [OpenCode v2 setup and usage](docs/opencode-v2.md) — native plugin API
-- [Pi setup and usage](docs/pi.md) — MCP server over stdio
-
-The MCP server works with any host that supports MCP tools over stdio; Pi is the first
-host with a shipped guide. For how other coding agents integrate—native websearch
-providers, native fetch tools, MCP, or wrappers—see the
-[integration model](docs/integration.md) with its version-locked host support matrix, and
-the [MCP compatibility research](docs/mcp-compatibility.md). A host is listed as supported
+- [OpenCode v2 setup and usage](docs/agents/opencode-v2.md) — native plugin API
+- [Pi setup and usage](docs/agents/pi.md) — MCP server over stdio
+- [Codex setup and usage](docs/agents/codex.md) — MCP server over stdio
+
+The MCP server works with any host that supports MCP tools over stdio. The
+`deepseek-web-retrieval init` CLI installs it with plan-then-apply semantics — search and
+fetch tiers chosen per capability, project or global scope, snapshots for rollback. For
+how coding agents integrate—native websearch providers, native fetch tools, MCP, or
+wrappers—see the
+[integration model](docs/integration.md) with its version-locked host support matrix, the
+[control-level research](docs/research/control-matrix.md), and the
+[MCP compatibility research](docs/research/mcp-compatibility.md). A host is listed as supported
 only after an adapter and guide exist.
 
 For shared search, fetch, retry, and privacy behavior, see below.
 
 ## General behavior
 
+What this adapter layer adds on top of the
+[DeepSeek Search and Fetch APIs](https://api-docs.deepseek.com/guides/search) — the APIs'
+own capabilities (Markdown-only output, no JavaScript rendering, no private networks or
+binary files, per-request billing) are documented there.
+
 ### Search
 
-- Sends `need_content: false` and returns DeepSeek's `summary`, falling back to `snippet`.
-- Filters by `rerank_score` and `authority`, deduplicates URLs, and returns at most five
-  results by default.
-- The final query, including `blockedDomains`, `includeDomains`, and `language` operators,
-  must be 128 characters or fewer. Longer queries are rejected locally.
+- Requests summaries only (`need_content: false`, `summary` falling back to `snippet`);
+  full page content never enters the results.
+- Returns at most ten results by default (matching the API's `count` default and the
+  industry norm for AI-native search APIs), deduplicating URLs. Optional
+  `rerank_score` and `authority` thresholds are off by default; when enabled they
+  drop results below the configured score or tier.
+- Builds `blockedDomains`, `includeDomains`, and `language` into query operators and
+  rejects the final query locally when it exceeds the API's 128-character limit.
 - An empty result set is normal and is not an error.
 
 ### Fetch
 
-- DeepSeek Fetch returns Markdown only. `format: "text"` and `format: "html"` are rejected.
-- Output is capped at 16,000 characters by default, including the title. A
+- Output is capped at 16,000 characters by default, including the title; a
   `[content truncated]` marker is added when the limit leaves room for it.
-- If the page already starts with the same title as a top-level heading, no duplicate title
-  is added.
-- Responses larger than 5 MiB are rejected before parsing.
-- URLs must be absolute `http://` or `https://` URLs with a host, no embedded credentials,
-  and at most 8,192 characters.
-- The core supports allowed/blocked domain lists and private-host blocking. DeepSeek does
-  not render JavaScript and cannot reach private networks; PDFs, images, and other binary
-  files are unsupported.
+- A page that already opens with its title as a top-level heading does not get a
+  duplicate title.
+- Responses larger than 5 MiB are rejected before parsing; URLs must be absolute
+  `http://` or `https://`, credential-free, and at most 8,192 characters.
+- `format: "text"` and `format: "html"` are rejected rather than misrepresented.
+- Allowed/blocked domain lists and private-host blocking are enforced on the client,
+  before a request leaves the machine.
 
 ### Timeouts and retries
 
@@ -64,10 +92,9 @@ provider or fetch implementation.
 
 ### Privacy and cost
 
-Queries and URLs are sent to `https://api.deepseek.com`. The API key is sent only as a
-Bearer token to DeepSeek. Search and Fetch are billed per successful request; check current
-DeepSeek pricing and rate limits before enabling a team deployment. A 429 means you should
-wait, reduce concurrency, or increase retries.
+Queries and URLs are sent to `https://api.deepseek.com`; the API key is sent only as a
+Bearer token to DeepSeek and to nothing else. Retrieval is billed per successful request —
+see the official docs for pricing and rate limits before enabling a team deployment.
 
 ## Development
 
@@ -81,7 +108,7 @@ The test suite mocks the DeepSeek HTTP API and the OpenCode plugin host, and dri
 MCP server with SDK clients pinned to each supported protocol revision. Real API calls
 require a funded DeepSeek account. Shared retrieval logic lives in `src/core/`; adapters
 live in `src/<agent>/` (the cross-host MCP server lives in `src/mcp/`). Adapter docs live
-in `docs/<agent>.md`.
+in `docs/agents/<agent>.md`.
 
 ## Prior art