dsh-search-tools 0.0.0-stage → 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,149 @@
+# Changelog
+
+## 0.1.0 — 2026-10-22
+
+### Added
+
+- Adapter-first core: `ctx.searchTools` registry, stable search/fetch dispatchers,
+  manifest-driven settings metadata.
+- SearXNG search adapter skeleton: config validation, JSON/HTML parsing, auth
+  lookup, failover, pagination, cache, probe.
+- Readability HTTP fetch adapter skeleton: DSH safe fetch, lazily imported
+  linkedom/Readability, fail-open extraction.
+- `web_research` tool and conversation card.
+- Settings section registration (`settings.section`) and authenticated host
+  routes for status, adapters, probe, search preview, fetch preview, cache clear.
+- YAML-backed config through DSH settings; credentials-backed secrets.
+- Project skeleton: TypeScript, tsdown host/client builds, Vitest, CI/release
+  workflows, bilingual README, MIT license.
+- Component tests for the settings page and `web_research` card
+  (`tests/settings-page.spec.tsx`, `tests/research-card.spec.tsx`), with jsdom
+  as a dev-only test environment.
+- Code-based adapter issue messages, localized for `zh` / `en`; the settings
+  page falls back to the English message when no code is present.
+- Generic manifest validation for external adapters, so a new adapter's required
+  fields and numeric bounds are checked before saving without importing its code
+  into the browser.
+- The research card only turns `http(s)` source URLs into links and drops
+  malformed presentation-meta entries instead of rendering empty rows.
+- `web_research` pipeline integration test: real HTML extraction, per-page
+  failure isolation, text clipping, total-budget accounting, disabled mode, and
+  presentation-meta projection.
+- The research card reports successful fetches separately when some pages fail
+  (`1/2 fetched`) instead of counting attempts as successes.
+- Search and Fetch tabs now have inline test panels. Results are structured:
+  elapsed time, safe `http(s)` source links, snippets, and fetch metadata/content,
+  instead of the previous single text blob. Tests refuse to run while the draft
+  has validation errors.
+- Advanced diagnostics can copy a credential-free summary and clear the in-memory
+  error ring (`POST /api/dsh-search-tools/diagnostics/clear`).
+- Search and Fetch validation is section-scoped: a broken search draft no longer
+  blocks a fetch test, and fetch errors no longer attach to search fields that
+  happen to share a key.
+- Tab semantics (`role=tablist/tab/tabpanel`, arrow/Home/End navigation),
+  `aria-invalid` on invalid fields, an aria-live save/diagnostic status, and
+  `Cmd/Ctrl+S` save.
+- The active-takeover instance rule is driven by the selected manifest's
+  `instances` field rather than a hard-coded SearXNG check. External adapters
+  without instances no longer produce a false save error.
+- `/status` lists credential refs declared by every registered adapter manifest,
+  not just SearXNG's two refs.
+- The settings footer is no longer sticky, so it cannot cover a long test result
+  in the 800px settings pane. Overview cards use `auto-fit` columns to avoid
+  label overlap at that width.
+- Activation-hint and takeover-warning cards use token fallbacks, so they render
+  with a real border/background outside the settings theme scope.
+- Adapter picker shows the selected manifest's description/capabilities and keeps
+  a visible “not installed” option for orphaned config.
+- Adapter picker is now a radio-card group with per-adapter
+  configured/needs-setup/needs-attention status, a “not installed” orphan card,
+  and a Refresh adapters action.
+- Research tab previews the model-facing `web_research` description and parameter
+  bounds from the current draft. The contract lives in `src/research/prompt.ts`
+  and is shared with the registered tool, so the preview cannot drift from the
+  tool description.
+- Instance rows stack the URL/label fields and their actions in the 800px
+  settings pane, and show a visible “Enabled” label next to the switch. This
+  fixes a 4px-wide URL input caused by the action row consuming the grid.
+- SearXNG's `unresponsive_engines` are surfaced as `Search notes` in the settings
+  preview, so an empty result caused by suspended engines is distinguishable from
+  a genuinely empty search.
+
+### Changed
+
+- `pnpm release` now uses a narrow publint policy that ignores only the known
+  `FILE_INVALID_FORMAT` verdict for `lib/client.js` (a DSH browser CJS factory)
+  and reports every other diagnostic. The same check runs in CI.
+- Package metadata now includes search keywords, `publishConfig.access: public`,
+  and publishes `CHANGELOG.md` / `SECURITY.md` in the tarball.
+
+### Verified
+
+- M0 on DSH `0.2.1-rc.2`: Settings → Search Tools renders; Save writes
+  `cordis.patch.yml`; authenticated routes return 401 without the browser
+  cookie and 200 with it; probe/search-preview work against a mock SearXNG;
+  search/fetch takeover and restore work; secret write stays out of YAML.
+  Details: `docs/m0-verification.md`.
+- A scripted model drove `web_research` through the real DSH agent loop and Web
+  GUI: the tool executed against a mock SearXNG, fetched two public pages, wrote
+  a `tool/result` with presentation meta, and rendered the card in the
+  conversation (`2 sources`, `2 fetched`, source links, fetched details).
+- 65 tests across 13 files, including settings-page, activation-hint,
+  research-card, takeover-warning, diagnostics text, publint policy, and
+  `web_research` pipeline tests (`pnpm test`).
+- A live Web GUI pass on the new form UX: inline search test rendered two
+  structured sources, the footer no longer covered the result, the zero-instance
+  empty state and one-column Overview layout rendered correctly, `Copy
+  diagnostics`/`diagnostics/clear` worked, and the activation hint rendered as a
+  bordered card after disabling/re-enabling the bundle. Dark theme and a 700px
+  narrow viewport also rendered correctly; tabs wrap instead of clipping.
+- A live Web GUI pass on the product-polish surface: the picker rendered as a
+  one-card radiogroup with `Configured`, `Refresh adapters` reported success, the
+  research prompt rendered at 530×272 with current bounds, and at 700px the prompt
+  reflowed to 382px without horizontal overflow. The same pass caught and fixed a
+  4px-wide instance URL input caused by the action row consuming the instance
+  grid; URL/label now render full width with actions below.
+- A real authenticated SearXNG instance was probed and searched through the
+  settings page: JSON API and Basic auth worked, 10 real sources rendered, and
+  credentials stayed out of YAML (`Source: env`). With the instance's default
+  engines suspended, the preview showed `Search notes`; selecting `bing` returned
+  real results. Details: `docs/m0-verification.md`.
+- A real OpenAI-compatible model (`agnes-3.0-flash`) called `web_research`,
+  built-in `web_search`, and built-in `web_fetch` through the live Web GUI. The
+  research turn returned 8 real SearXNG sources and fetched 2 pages (1,792 and
+  868 chars); `web_search` used the taken-over provider; `web_fetch` returned
+  HTTP 200 for example.com. The provider key stayed in the process environment,
+  with no secret in YAML, the repository, or the three test-session logs (the
+  chat transcript that supplied it is separate and should be treated as exposed).
+  Details: `docs/m0-verification.md`.
+- Final v0.1.0 gate: `pnpm release` passed with 65 tests / 13 files, host/client/
+  type builds, a pack dry run, and no publint issues after the documented
+  client-bundle exemption. `npm publish --dry-run` accepted the 99-file, 158.4 kB
+  tarball, including `CHANGELOG.md` / `SECURITY.md`; publication itself remains
+  pending npm authentication and a reachable repository remote.
+
+### Fixed during M0
+
+- Resolve credentials per request instead of capturing them at plugin apply.
+- Read the `{ ok, value }` envelope from `remote.credentials.describe`.
+- Render write-only secret fields from `manifest.secretRefs`.
+- Require a passing search test before search takeover and restore the previous
+  provider on disable.
+- Allow an empty instance list when takeover is off.
+- Localize page copy and adapter manifests for `zh` / `en`.
+- Rebuild the LRU when cache bounds change; wire `undici.ProxyAgent`.
+- Validate a missing `userAgent` against the adapter default.
+- Use linkedom instead of jsdom for Readability: jsdom's `tr46` dependency
+  calls `createRequire(...).resolve.paths`, which DSH's profile-resolution hook
+  breaks; linkedom is smaller and works inside the Host.
+- Add the `plugins.detail.section` takeover warning with a direct
+  “disable takeover” action.
+- Add per-instance probe buttons and an empty-state guide.
+- Add localized adapter manifests so external adapters can ship `en` / `zh`
+  labels without changing the page.
+- Drop malformed `web_research` presentation-meta entries instead of rendering
+  empty list items or duplicate React keys.
+- Give research number fields real label/id associations, so they are reachable
+  by accessible name as well as by mouse.
+- Recognize a scoped package name (`@scope/dsh-search-tools`) in the Plugins-page
+  takeover warning instead of requiring the unscoped name verbatim.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 ThzxxArt
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,114 @@
-# Temporary Holding Version
+# dsh-search-tools
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Adapter-first web search and readable page fetching for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness).
+
+> **Status: v0.1.0 release candidate (2026-10-22).** M0 is verified end to end:
+> the host and client bundles build, Settings → Search Tools renders in the real
+> Web GUI, Save writes YAML, authenticated routes work, search/fetch takeover and
+> restore work, and credentials stay out of YAML. The settings page has inline
+> search/fetch tests, diagnostics copy/clear, keyboard save, a card-based adapter
+> picker, and a live preview of the model-facing `web_research` contract. A real
+> OpenAI-compatible model called `web_research`, built-in `web_search`, and
+> built-in `web_fetch`; search used a real SearXNG instance and the research card
+> rendered. Build, test, pack, and publint checks pass. Publication is the
+> remaining external step; see `docs/release-v0.1.0.md`.
+
+## What it does
+
+- Exposes **stable** DSH providers `dsh-search-tools` (search) and
+  `dsh-search-tools-fetch` (fetch). The provider ids never change when a backend
+  is added or swapped.
+- Ships **SearXNG** as the first search adapter and **Readability HTTP** as the
+  first fetch adapter.
+- Adds a `web_research` tool that searches, fetches the top N pages, and returns
+  readable content in one call.
+- Registers **Settings → Search Tools** in the Web GUI. Configuration is edited
+  there, never by hand.
+- Search and Fetch tabs include inline test panels with structured sources and
+  fetch previews; Advanced can copy or clear diagnostics.
+- Stores configuration in the DSH profile's YAML patch
+  (`$DSH_HOME/profiles/<profile>/cordis.patch.yml`) through the native settings
+  service. Secrets go to DSH credentials, not YAML.
+
+## Adding a backend
+
+An adapter package depends on `ctx.searchTools` and registers a manifest plus a
+runtime:
+
+```ts
+export const inject = ['searchTools']
+
+export function apply(ctx: Context): void {
+  ctx.effect(() => ctx.searchTools.registerSearchAdapter(exaAdapter), 'exa adapter')
+}
+```
+
+The settings page reads adapter manifests from the host route and renders their
+fields generically. No page code changes when a new adapter is registered.
+
+## Development
+
+```sh
+pnpm install --registry=https://registry.npmjs.org
+pnpm typecheck
+pnpm test
+pnpm build
+pnpm pack:check
+```
+
+`pnpm build` emits:
+
+```text
+lib/index.js        Host half
+lib/client.js       Browser lazy-CJS factory
+lib/types/**        Declarations
+```
+
+## Install in DSH
+
+Primary path is the Web UI:
+
+1. Open **Plugins**.
+2. **Add plugin** → enter `dsh-search-tools`.
+3. Enable it.
+4. Open **Settings → Search Tools** and configure a SearXNG instance.
+
+Before the npm release, install a packed tarball instead:
+
+```sh
+pnpm pack
+dsh plugin --profile <profile> add ./dsh-search-tools-0.1.0.tgz
+```
+
+For local development, an absolute-path overlay can load `src/index.ts`; use the
+DSH checkout's own docs for the exact `--patch` command.
+
+## Project layout
+
+```text
+src/core/                 adapter types, registry, stable dispatchers
+src/search/adapters/      search adapters (searxng first)
+src/fetch/adapters/       fetch adapters (readable-http first)
+src/research/             web_research pipeline, prompt, render, tool
+src/host/                 authenticated settings routes
+src/client/               Settings → Search Tools + tool card
+src/shared/               config, validation, diff, URL helpers
+tests/                    host units + settings page / research card components
+```
+
+Readability uses `linkedom`, not `jsdom`: jsdom's `tr46` dependency calls
+`createRequire(...).resolve.paths`, which DSH's profile-resolution hook does not
+implement. linkedom is smaller and works inside the Host.
+
+## Security
+
+- Settings routes use DSH's `connection.requestRejection` fence.
+- Fetch reuses DSH's SSRF-safe `HttpFetchProvider`.
+- Tokens are read from `ctx.credentials`/environment per request and never enter
+  YAML, logs, or error text.
+- SearXNG requests use `redirect: 'error'` so credentials cannot follow a
+  cross-origin redirect.
+
+## License
+
+MIT. See [LICENSE](LICENSE) and [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).

package/README.zh.md

@@ -0,0 +1,92 @@
+# dsh-search-tools
+
+为 [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) 提供 adapter-first 的网页搜索与正文抓取。
+
+> **状态：v0.1.0 发布候选（2026-10-22）。** M0 已端到端验证：Host 与客户端 bundle 可构建，设置里的「搜索工具」在真实 Web GUI 中出现，保存写入 YAML，认证路由可用，搜索/抓取接管与恢复可用，密钥不落 YAML。设置页提供搜索/抓取内联测试、诊断复制/清空、键盘保存、卡片式 adapter picker，以及模型看到的 `web_research` 契约预览。真实 OpenAI 兼容模型已分别调用 `web_research`、内置 `web_search` 和内置 `web_fetch`，搜索走真实 SearXNG，研究卡片正常渲染。构建、测试、打包和 publint 检查均通过。发布是剩余的外部步骤；见 `docs/release-v0.1.0.md`。
+
+## 它做什么
+
+- 注册**稳定**的 DSH provider：`dsh-search-tools`（搜索）和 `dsh-search-tools-fetch`（抓取）。新增或切换后端时 provider id 不变。
+- 第一个搜索 adapter 是 **SearXNG**，第一个抓取 adapter 是 **Readability HTTP**。
+- 新增 `web_research` 工具：一次调用完成“搜索 + 抓前 N 条正文”。
+- 在 Web GUI 注册 **设置 → 搜索工具**，所有配置在页面完成，不需要手改文件。
+- 搜索/抓取页签内置测试面板，返回结构化来源和抓取预览；高级页可复制/清空诊断信息。
+- 配置通过 DSH 原生 settings 服务写入 profile 的 YAML patch（`$DSH_HOME/profiles/<profile>/cordis.patch.yml`）。密钥走 DSH credentials，不写入 YAML。
+
+## 新增后端
+
+Adapter 包依赖 `ctx.searchTools`，注册 manifest 和 runtime：
+
+```ts
+export const inject = ['searchTools']
+
+export function apply(ctx: Context): void {
+  ctx.effect(() => ctx.searchTools.registerSearchAdapter(exaAdapter), 'exa adapter')
+}
+```
+
+设置页从 Host 路由读取 manifest，自动渲染字段；新增 adapter 不需要改页面代码。
+
+## 开发
+
+```sh
+pnpm install --registry=https://registry.npmjs.org
+pnpm typecheck
+pnpm test
+pnpm build
+pnpm pack:check
+```
+
+`pnpm build` 产出：
+
+```text
+lib/index.js        Host 半侧
+lib/client.js       浏览器 lazy-CJS factory
+lib/types/**        类型声明
+```
+
+## 在 DSH 里安装
+
+主路径是 Web UI：
+
+1. 打开 **插件**。
+2. **添加插件** → 输入 `dsh-search-tools`。
+3. 启用。
+4. 打开 **设置 → 搜索工具**，配置 SearXNG 实例。
+
+npm 发布前，先用打包产物安装：
+
+```sh
+pnpm pack
+dsh plugin --profile <profile> add ./dsh-search-tools-0.1.0.tgz
+```
+
+本地开发可以用绝对路径 overlay 加载 `src/index.ts`，具体 `--patch` 命令以 DSH checkout 文档为准。
+
+## 目录
+
+```text
+src/core/                 adapter 类型、注册表、稳定 dispatcher
+src/search/adapters/      搜索 adapter（第一个是 searxng）
+src/fetch/adapters/       抓取 adapter（第一个是 readable-http）
+src/research/             web_research pipeline、prompt、render、tool
+src/host/                 受认证的设置路由
+src/client/               设置 → 搜索工具 + 工具卡片
+src/shared/               config、校验、diff、URL 辅助
+tests/                    Host 单测 + 设置页/研究卡片组件测试
+```
+
+Readability 使用 `linkedom`，不是 `jsdom`：jsdom 的 `tr46` 依赖会调用
+`createRequire(...).resolve.paths`，而 DSH 的 profile resolution hook 没有实现它。
+linkedom 更小，也能在 Host 内正常工作。
+
+## 安全
+
+- 设置路由使用 DSH 的 `connection.requestRejection` 栅栏。
+- 抓取复用 DSH 的 SSRF 安全 `HttpFetchProvider`。
+- token 每次请求从 `ctx.credentials`/环境变量读取，不进 YAML、日志或错误文本。
+- SearXNG 请求使用 `redirect: 'error'`，凭据不会跟随跨源重定向。
+
+## 许可证
+
+MIT。见 [LICENSE](LICENSE) 与 [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md)。

package/SECURITY.md

@@ -0,0 +1,20 @@
+# Security Policy
+
+## Supported versions
+
+v0.1.x is the current preview line. DSH itself is preview software; treat this
+plugin as preview too.
+
+## Reporting
+
+Please open a private security advisory on GitHub rather than a public issue.
+
+## Design notes
+
+- Configuration is written by DSH's settings/config-editor into the profile
+  `cordis.patch.yml` (mode 0600) and never contains bearer/basic secrets.
+- Secrets are referenced as `SEARXNG_TOKEN` / `SEARXNG_BASIC_AUTH` and resolved
+  per request from DSH credentials or the process environment.
+- The settings routes require DSH's browser authentication and Host/Origin fence.
+- Fetching reuses `@deepseek-ai/dsh-web-fetch-http`, including its SSRF policy.
+- Diagnostics redact URL query strings and never include credentials.

package/THIRD_PARTY_NOTICES.md

@@ -0,0 +1,17 @@
+# Third-party notices
+
+This package is MIT-licensed. It depends on:
+
+- `@mozilla/readability` — Apache-2.0
+- `linkedom` — MIT
+- `cheerio` — MIT
+- `lru-cache` — ISC
+- `normalize-url` — MIT
+- `turndown` — MIT
+- `@joplin/turndown-plugin-gfm` — MIT
+- `undici` — MIT
+
+DSH packages (`@deepseek-ai/*`) are peer dependencies supplied by the host
+installation; their licenses are those of DeepSeek Harness.
+
+See each package's own LICENSE for the authoritative text.

package/cordis.patch.yml

@@ -0,0 +1,16 @@
+# dsh-search-tools bundle layer
+#
+# The row uses the bare package name so the client module system attaches the
+# browser half. The stable provider ids are owned by the Host half; adapters
+# are selected inside this row's config.
+- insert:
+    - id: dsh-search-tools
+      name: dsh-search-tools
+      config:
+        search:
+          adapter: searxng
+        fetch:
+          adapter: readable-http
+        research:
+          enabled: true
+          defaultFetchTop: 3

package/icon.svg

@@ -0,0 +1,6 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64" role="img" aria-label="Search Tools">
+  <rect width="64" height="64" rx="14" fill="#1f6feb"/>
+  <circle cx="27" cy="27" r="12" fill="none" stroke="#fff" stroke-width="4"/>
+  <path d="M36 36l11 11" fill="none" stroke="#fff" stroke-width="4" stroke-linecap="round"/>
+  <path d="M20 24h14M20 29h9" fill="none" stroke="#fff" stroke-width="2.5" stroke-linecap="round"/>
+</svg>