@fffangchenghao/dsh-web-search 0.17.0 → 0.19.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +53 -0
- package/README.md +7 -7
- package/README.zh-CN.md +7 -7
- package/client.js +1 -1
- package/docs/rrf-ranking.md +37 -0
- package/lib/engines.js +10 -2
- package/lib/rank.js +15 -2
- package/lib/relevance.d.ts +57 -0
- package/lib/relevance.js +77 -0
- package/lib/transport.d.ts +0 -17
- package/lib/transport.js +31 -6
- package/lib/types.d.ts +17 -0
- package/lib/variants.d.ts +28 -0
- package/lib/variants.js +74 -0
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,58 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.19.0
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
- **Core-word retry for whole-phrase engines.** `cn.bing` and 360 Search match a
|
|
7
|
+
long Chinese query as one phrase and fall back to dictionary pages for its
|
|
8
|
+
first character when the phrase is unknown. `src/variants.ts` derives at most
|
|
9
|
+
one alternate query — the technical segments of a mixed query
|
|
10
|
+
(`如何配置 postgres 逻辑复制` → `postgres 逻辑复制`), or the first content
|
|
11
|
+
segment of a pure CJK query (`东数西算 八个国家枢纽节点 定位` → `东数西算`) —
|
|
12
|
+
and the transport asks it once, merging the rows after the original list.
|
|
13
|
+
Measured on the policy query, `cn.bing` went from zero usable rows to the NDRC
|
|
14
|
+
article, the gov.cn implementation opinion, and the ncsti overview; the fused
|
|
15
|
+
list is now eight relevant rows.
|
|
16
|
+
- **A five-query long-Chinese diagnostic corpus.** `bench/queries-long.json`
|
|
17
|
+
covers a policy question, a mixed PostgreSQL question, and Python / Kubernetes
|
|
18
|
+
/ React release-doc questions; fixtures live in `bench/fixtures-long/`.
|
|
19
|
+
`scripts/bench.mjs` grew `--queries` and `--fixtures` to record and evaluate it
|
|
20
|
+
without touching the 20-query baseline. Result: P@8 `0.4500`, R@8 `1.0000`,
|
|
21
|
+
MRR@8 `0.7667`, zero hits 0.
|
|
22
|
+
|
|
23
|
+
### Changed
|
|
24
|
+
- Two extra test files (`variants`, `variant-fetch`) and 5 tests; 498 cases
|
|
25
|
+
total (497 pass, one environment skip).
|
|
26
|
+
|
|
27
|
+
## 0.18.0
|
|
28
|
+
|
|
29
|
+
### Fixed
|
|
30
|
+
- **Chinese queries no longer drag dictionary pages into the answer.** A long or
|
|
31
|
+
mixed query (`东数西算 八个国家枢纽节点 定位`, `如何配置 postgres 逻辑复制`)
|
|
32
|
+
made `cn.bing` answer with rows about a single character — `东(汉语文字)`,
|
|
33
|
+
`如何(汉语词语)`, a company named 李在东 — and those rows collected votes in
|
|
34
|
+
the fusion. A CJK relevance gate now splits the query into character bigrams
|
|
35
|
+
and keeps a row only when its title, snippet or URL accumulates two units of
|
|
36
|
+
overlap; an ASCII word counts double, so a PostgreSQL page survives a mixed
|
|
37
|
+
query on `postgres` alone. An engine whose entire list is filtered is reported
|
|
38
|
+
as `<id>: no relevant results`, which lets the fallback tier run instead of
|
|
39
|
+
returning the junk.
|
|
40
|
+
- **Chinese official domains earn the authority boost.** `gov.cn`, `edu.cn`,
|
|
41
|
+
`news.cn`, `people.com.cn`, `xinhuanet.com`, `ncsti.gov.cn` and `cas.cn` join
|
|
42
|
+
the default authority list, so a ministry page outranks an SEO repost or a
|
|
43
|
+
Baidu Baike entry at the same fused position.
|
|
44
|
+
|
|
45
|
+
### Changed
|
|
46
|
+
- `RankOptions.query` carries the query into fusion; `scripts/bench.mjs` passes
|
|
47
|
+
the fixture's query, so offline replay applies the same gate production uses.
|
|
48
|
+
- The 20-query benchmark is byte-for-byte unchanged (`0.4375 / 0.7917 / 0.8071`):
|
|
49
|
+
the recorded fixtures do not contain the dictionary-page failure mode.
|
|
50
|
+
|
|
51
|
+
### Tests
|
|
52
|
+
- 493 cases (492 pass, one environment skip), including the new
|
|
53
|
+
`test/relevance.test.ts` coverage for dropped dictionary/company rows, URL-slug
|
|
54
|
+
matches, English passthrough and the mixed-query ASCII weight.
|
|
55
|
+
|
|
3
56
|
## 0.17.0
|
|
4
57
|
|
|
5
58
|
### Fixed
|
package/README.md
CHANGED
|
@@ -9,7 +9,7 @@
|
|
|
9
9
|
|
|
10
10
|
<p align="center">
|
|
11
11
|
<a href="https://www.npmjs.com/package/@fffangchenghao/dsh-web-search"><img src="https://img.shields.io/npm/v/@fffangchenghao/dsh-web-search.svg" alt="npm version"></a>
|
|
12
|
-
<img src="https://img.shields.io/badge/tests-
|
|
12
|
+
<img src="https://img.shields.io/badge/tests-497%20passing-brightgreen" alt="tests">
|
|
13
13
|
<img src="https://img.shields.io/badge/engines-8%2C%20zero%20API%20key-brightgreen" alt="engines">
|
|
14
14
|
<img src="https://img.shields.io/badge/node-%3E%3D22.18-blue" alt="node">
|
|
15
15
|
<img src="https://img.shields.io/badge/dsh-%3E%3D0.2.1--rc.2%20%3C0.3.0-blueviolet" alt="dsh">
|
|
@@ -21,7 +21,7 @@
|
|
|
21
21
|
- **No API key, eight engines.** AnySearch, CN Bing, DuckDuckGo HTML, www Bing, DuckDuckGo Lite and 360 Search are scraped directly; Exa joins the primary tier through its keyless MCP endpoint, and Firecrawl's search API is the measured fallback. The plugin works out of the box.
|
|
22
22
|
- **The ranking is measured, not asserted.** Cross-engine reciprocal-rank fusion with authority and freshness boosts, a 20-query offline benchmark, and an 81-configuration parameter sweep — the only search plugin here that publishes both its metric and its confidence intervals. See [How the ranking works](docs/rrf-ranking.md).
|
|
23
23
|
- **Fetching admits failure.** Binary documents and empty 200 responses are rejected by name (`SEARCH_UNSUPPORTED`, `SEARCH_EMPTY`) instead of being returned as noise; the direct → tunnel → render → extract ladder shares one 25 s budget and stops when there is no time left. The budget is per request — there is a regression test for exactly that.
|
|
24
|
-
- **
|
|
24
|
+
- **498 tests across 34 files**, a committed `lib/`, declared `engines.dsh` / `engines.node`, and a [SECURITY.md](SECURITY.md) that lists every process, host and file the plugin touches.
|
|
25
25
|
- **Bilingual documentation.** This README and [中文说明](README.zh-CN.md) are maintained side by side.
|
|
26
26
|
|
|
27
27
|

|
|
@@ -37,7 +37,7 @@ The web-search corner of the DSH plugin ecosystem is crowded, and most rivals ar
|
|
|
37
37
|
| Key-free engines | 8 | ~30 | 7 channels (+X) | 1 official API |
|
|
38
38
|
| Dedicated vertical sources | 34 (8 on by default) | 41 | — | — |
|
|
39
39
|
| Model-facing tools | 6 | ~10 | 3 | 5 |
|
|
40
|
-
| Test suite |
|
|
40
|
+
| Test suite | 34 files / 498 cases | none | 29 files | 6 files |
|
|
41
41
|
| Ranking benchmark | 20 queries + 81-config sweep | — | — | — |
|
|
42
42
|
| Store status | source contract satisfied, awaiting review | unlisted | no entry | blocked (review-required) |
|
|
43
43
|
|
|
@@ -240,9 +240,9 @@ Each line is the plain version first, the technical note second. The technical n
|
|
|
240
240
|
Domains such as `kubernetes.io`, `developer.mozilla.org` and `rfc-editor.org` get a boost, as does anything published within 30 days.
|
|
241
241
|
*Technical: `score = votes / (rrfConstant + position)`, then `× (1 + authorityBoost)` and a freshness factor; defaults `rrfConstant: 60`, `authorityBoost: 0.25`, `freshnessBoost: 0.2`, `freshnessDays: 30`. Publication dates are now parsed from each engine's results, so the freshness term scores a real date instead of sitting idle.*
|
|
242
242
|
|
|
243
|
-
5. **Chinese and English are handled separately.**
|
|
244
|
-
English queries lead with global sources, Chinese queries with Chinese sources, so "how to …" is not answered with a dictionary page for the word `how`.
|
|
245
|
-
*Technical: `lead = cn` when the query contains CJK characters, `lead = global` for plain ASCII; the other region only appends when the leading one comes up short.*
|
|
243
|
+
5. **Chinese and English are handled separately, and Chinese rows must talk about the question.**
|
|
244
|
+
English queries lead with global sources, Chinese queries with Chinese sources, so "how to …" is not answered with a dictionary page for the word `how`. A scraped Chinese engine can still fall back to single-character dictionary and company pages for a long query; those rows are dropped before fusion unless they share at least two words with the question, and `.gov.cn` / `.edu.cn` official pages get the authority boost.
|
|
245
|
+
*Technical: `lead = cn` when the query contains CJK characters, `lead = global` for plain ASCII; the other region only appends when the leading one comes up short. `src/relevance.ts` splits a CJK query into character bigrams (an ASCII word counts double), requires two weight units of overlap with title, snippet or URL, and returns English queries untouched. A whole-phrase engine that misses a long query is asked once more with its core words (`src/variants.ts`): `东数西算 八个国家枢纽节点 定位` also asks `东数西算`, `如何配置 postgres 逻辑复制` also asks `postgres 逻辑复制`. Measured on the first query, `cn.bing` answered with ten dictionary/company rows before this; it now contributes the NDRC and gov.cn articles, and the fused list has eight relevant rows.*
|
|
246
246
|
|
|
247
247
|
6. **You choose speed or completeness.**
|
|
248
248
|
`auto` waits up to 6 s for English and 3.5 s for Chinese; `fast` takes whoever answers first; `all` waits for everyone, up to 10 s.
|
|
@@ -306,7 +306,7 @@ Each line is the plain version first, the technical note second. The technical n
|
|
|
306
306
|
|
|
307
307
|
21. **Search quality has a repeatable benchmark.**
|
|
308
308
|
Ranking changes are no longer judged by feel: a fixed set of 20 queries produces the same scores every time.
|
|
309
|
-
*Technical: `bench/` holds 20 queries (10 Chinese, 10 English), five in each of docs / release / package / news, each annotated with 1–3 expected domains. `node scripts/bench.mjs` replays recorded fixtures offline, never touches the network, and prints precision@8, recall@8, MRR@8 plus each engine's contribution; `--record` re-records live (1.5 s between queries) to refresh the fixtures; `--json` gives machine-readable output for version comparisons, and past results live in `bench/results/`. Baseline for 0.15.0 (tier-aware replay, so the recorded fallback engines only count when production would run them): P@8 `0.4375`, R@8 `0.7917`, MRR@8 `0.8071`; Chinese `0.4000 / 0.8167 / 0.7976`, English `0.4750 / 0.7667 / 0.8167`. The controlled experiments and known gaps are in the appendix.*
|
|
309
|
+
*Technical: `bench/` holds 20 queries (10 Chinese, 10 English), five in each of docs / release / package / news, each annotated with 1–3 expected domains. `node scripts/bench.mjs` replays recorded fixtures offline, never touches the network, and prints precision@8, recall@8, MRR@8 plus each engine's contribution; `--record` re-records live (1.5 s between queries) to refresh the fixtures; `--json` gives machine-readable output for version comparisons, and past results live in `bench/results/`. Baseline for 0.15.0 (tier-aware replay, so the recorded fallback engines only count when production would run them): P@8 `0.4375`, R@8 `0.7917`, MRR@8 `0.8071`; Chinese `0.4000 / 0.8167 / 0.7976`, English `0.4750 / 0.7667 / 0.8167`. A separate five-query long-Chinese diagnostic (`bench/queries-long.json`, recorded with `--queries ... --fixtures ...`) scores `0.4500 / 1.0000 / 0.7667`, with every query hitting an expected domain; it is kept separate so the 20-query baseline stays comparable. The controlled experiments and known gaps are in the appendix.*
|
|
310
310
|
|
|
311
311
|
22. **Research, in one call.**
|
|
312
312
|
For "what do the official docs and two other sources actually say about X", `research` runs up to three searches, opens the leading pages through the same fetch ladder, and returns one evidence pack in which every excerpt sits under its URL. A page that will not load is reported by URL and reason instead of aborting the pack.
|
package/README.zh-CN.md
CHANGED
|
@@ -9,7 +9,7 @@
|
|
|
9
9
|
|
|
10
10
|
<p align="center">
|
|
11
11
|
<a href="https://www.npmjs.com/package/@fffangchenghao/dsh-web-search"><img src="https://img.shields.io/npm/v/@fffangchenghao/dsh-web-search.svg" alt="npm 版本"></a>
|
|
12
|
-
<img src="https://img.shields.io/badge/tests-
|
|
12
|
+
<img src="https://img.shields.io/badge/tests-497%20passing-brightgreen" alt="测试">
|
|
13
13
|
<img src="https://img.shields.io/badge/engines-8%2C%20zero%20API%20key-brightgreen" alt="引擎">
|
|
14
14
|
<img src="https://img.shields.io/badge/node-%3E%3D22.18-blue" alt="node">
|
|
15
15
|
<img src="https://img.shields.io/badge/dsh-%3E%3D0.2.1--rc.2%20%3C0.3.0-blueviolet" alt="dsh">
|
|
@@ -21,7 +21,7 @@
|
|
|
21
21
|
- **不要 API key,八个引擎。** AnySearch、必应中国、DuckDuckGo HTML、必应国际、DuckDuckGo Lite、360 搜索直接抓取;Exa 走它的免密钥 MCP 通道加入主引擎,Firecrawl 的搜索 API 作为实测后的兜底。装好就能用。
|
|
22
22
|
- **排序是量出来的,不是写出来的。** 跨引擎 RRF 融合,叠加权威度与时效加权;20 条离线基准 + 81 组参数扫描,是这类插件里唯一同时公开指标和置信区间的。方法见 [排序是怎么做的](docs/rrf-ranking.md)。
|
|
23
23
|
- **抓取会承认失败。** 二进制文档和「200 但正文为空」会被明确拒绝(`SEARCH_UNSUPPORTED`、`SEARCH_EMPTY`),而不是当噪音返回;direct → tunnel → render → extract 四级阶梯共享 25 秒预算,预算不足就不再起新的一级。预算是「每次请求」而不是「插件启动时」,这点有回归测试守着。
|
|
24
|
-
- **
|
|
24
|
+
- **34 个测试文件、498 个用例**,`lib/` 产物入库,声明了 `engines.dsh` / `engines.node`,[SECURITY.md](SECURITY.md) 逐项列出插件会碰的进程、主机和文件。
|
|
25
25
|
- **双语文档。** 本文件和 [English README](README.md) 同步维护。
|
|
26
26
|
|
|
27
27
|

|
|
@@ -37,7 +37,7 @@ DSH 插件生态里的「联网搜索」这一块已经很挤,大多数竞品
|
|
|
37
37
|
| 免 key 引擎 | 8 | ~30 | 7 条通道(含 X) | 1 个官方 API |
|
|
38
38
|
| 专门数据源 | 34(默认开 8 个)| 41 | — | — |
|
|
39
39
|
| 面向模型的工具 | 6 | ~10 | 3 | 5 |
|
|
40
|
-
| 测试 |
|
|
40
|
+
| 测试 | 34 文件 / 498 用例 | 无 | 29 文件 | 6 文件 |
|
|
41
41
|
| 排序基准 | 20 查询 + 81 组扫描 | — | — | — |
|
|
42
42
|
| 商店状态 | 源码契约已满足,待复核 | unlisted | 无条目 | blocked(review-required) |
|
|
43
43
|
|
|
@@ -240,9 +240,9 @@ Harness 里有个叫 `web` 的**服务**,它自己不会搜索,只负责「
|
|
|
240
240
|
`kubernetes.io`、`developer.mozilla.org`、`rfc-editor.org` 这类域名会被加权;30 天内发布的内容也会被加权。
|
|
241
241
|
*技术补充:`score = votes / (rrfConstant + position)`,再乘 `1 + authorityBoost` 和新鲜度系数;默认 `rrfConstant: 60`、`authorityBoost: 0.25`、`freshnessBoost: 0.2`、`freshnessDays: 30`。发布日期由各引擎的结果解析而来,所以新鲜度这一项现在按真实日期计算,不再空转。*
|
|
242
242
|
|
|
243
|
-
5.
|
|
244
|
-
英文问题先看国外来源,中文问题先看中文来源,避免「how to …」被某个中文引擎按单词 `how`
|
|
245
|
-
*技术补充:查询含 CJK 字符时 `lead = cn`,纯 ASCII 时 `lead = global
|
|
243
|
+
5. **中英文分开处理,而且中文结果必须真的在讲这个问题。**
|
|
244
|
+
英文问题先看国外来源,中文问题先看中文来源,避免「how to …」被某个中文引擎按单词 `how` 匹配成词典页。中文长查询仍可能被引擎退化成单字词典页、公司页;这些行在融合前会被丢掉,除非标题/摘要/URL 与问题至少重合两个词;`.gov.cn` / `.edu.cn` 官方页面另有权威度加权。
|
|
245
|
+
*技术补充:查询含 CJK 字符时 `lead = cn`,纯 ASCII 时 `lead = global`;非主导区只在主导区凑不够条数时才追加。`src/relevance.ts` 把中文查询切成二字 bigram(ASCII 单词权重翻倍),要求与标题、摘要或 URL 至少两个权重单位的重合;英文查询原样放行。对整串匹配型引擎还会用核心词补问一次(`src/variants.ts`):「东数西算 八个国家枢纽节点 定位」会补问「东数西算」,「如何配置 postgres 逻辑复制」会补问「postgres 逻辑复制」。实测第一条查询:`cn.bing` 原来返回十条词典/公司页,现在能贡献发改委和 gov.cn 文章,融合结果八条全部相关。*
|
|
246
246
|
|
|
247
247
|
6. **快慢可以选。**
|
|
248
248
|
默认 `auto`:英文最多等 6 秒、中文最多等 3.5 秒;`fast` 谁先回就用谁;`all` 等所有引擎,最多 10 秒。
|
|
@@ -306,7 +306,7 @@ Harness 里有个叫 `web` 的**服务**,它自己不会搜索,只负责「
|
|
|
306
306
|
|
|
307
307
|
21. **搜索质量有了可重复的基准。**
|
|
308
308
|
改排序不再靠感觉:仓库里有一组固定的 20 条查询,每次改完都能跑出同一套分数。
|
|
309
|
-
*技术补充:`bench/` 下 20 条查询(中文 10、英文 10),四类各 5 条(文档 / 版本发布 / 包与库 / 新闻),每条标注 1–3 个期望域名。`node scripts/bench.mjs` 离线读夹具重放,不联网,输出 precision@8、recall@8、MRR@8 与每个引擎的贡献;`--record` 联网重新录制(每条限速 1.5 秒),用于更新夹具;`--json` 输出机器可读结果,便于版本间对比,历史结果存在 `bench/results/`。当前基线(0.15.0,按 tier 回放):P@8 `0.4375`、R@8 `0.7917`、MRR@8 `0.8071`;中文 `0.4000 / 0.8167 / 0.7976`,英文 `0.4750 / 0.7667 / 0.8167
|
|
309
|
+
*技术补充:`bench/` 下 20 条查询(中文 10、英文 10),四类各 5 条(文档 / 版本发布 / 包与库 / 新闻),每条标注 1–3 个期望域名。`node scripts/bench.mjs` 离线读夹具重放,不联网,输出 precision@8、recall@8、MRR@8 与每个引擎的贡献;`--record` 联网重新录制(每条限速 1.5 秒),用于更新夹具;`--json` 输出机器可读结果,便于版本间对比,历史结果存在 `bench/results/`。当前基线(0.15.0,按 tier 回放):P@8 `0.4375`、R@8 `0.7917`、MRR@8 `0.8071`;中文 `0.4000 / 0.8167 / 0.7976`,英文 `0.4750 / 0.7667 / 0.8167`。另有五条中文长查询的独立诊断集(`bench/queries-long.json`,用 `--queries ... --fixtures ...` 录制):`0.4500 / 1.0000 / 0.7667`,五条都命中期望域名;它与 20 条基线分开保存,避免改变基线的可比性。对照实验和已知短板见附录。*
|
|
310
310
|
|
|
311
311
|
22. **一次调用完成小调研。**
|
|
312
312
|
问「官方文档和另外两个来源到底怎么说」时,`research` 先跑最多三条查询,再用与 `web_fetch` 相同的抓取阶梯打开靠前的页面,最后返回一份证据包:每段摘录下面就是它的 URL。打不开的页面按 URL 和原因列出,不会让整份证据包失败。
|
package/client.js
CHANGED
|
@@ -47,7 +47,7 @@
|
|
|
47
47
|
var ENTRY = 'web-search-bing';
|
|
48
48
|
var PACKAGE_NAME = 'dsh-web-search-bing';
|
|
49
49
|
/** Keep in step with package.json; compared with the npm registry. */
|
|
50
|
-
var VERSION = '0.
|
|
50
|
+
var VERSION = '0.19.0';
|
|
51
51
|
/** Snapshot used while no configuration service is reachable. */
|
|
52
52
|
var EMPTY_STATE = { status: 'unavailable', value: undefined, base: undefined, user: undefined, revision: undefined, writable: false };
|
|
53
53
|
|
package/docs/rrf-ranking.md
CHANGED
|
@@ -131,6 +131,43 @@ Exa therefore leads; Firecrawl is parsed and tested but ships as the fallback,
|
|
|
131
131
|
available through `hostIds: ['firecrawl']` and used whenever every primary
|
|
132
132
|
returns nothing.
|
|
133
133
|
|
|
134
|
+
|
|
135
|
+
## CJK relevance gate
|
|
136
|
+
|
|
137
|
+
Fusion assumes each engine's list is about the query. A scraped Chinese engine
|
|
138
|
+
sometimes is not: `cn.bing` answered `东数西算 八个国家枢纽节点 定位` with ten
|
|
139
|
+
rows for the character `东` (dictionary entries, a company named 李在东), and
|
|
140
|
+
those rows would otherwise collect votes like any other engine's. Before fusion,
|
|
141
|
+
`src/relevance.ts` builds overlapping two-character tokens from the query and
|
|
142
|
+
keeps a row only when its title, snippet or URL accumulates two units of
|
|
143
|
+
overlap. An ASCII word is worth two units, so a PostgreSQL page in a mixed
|
|
144
|
+
query (`如何配置 postgres 逻辑复制`) survives on `postgres` alone, while
|
|
145
|
+
`如何(汉语词语)_百度百科` does not. English queries are returned unchanged.
|
|
146
|
+
|
|
147
|
+
The gate lives in both the engine tier (so an engine whose whole list is junk is
|
|
148
|
+
reported as `no relevant results` and the fallback tier can run) and the ranker
|
|
149
|
+
(so offline replay applies the same rule). On the 20-query bench it changes no
|
|
150
|
+
metric: the recorded fixtures do not contain that failure mode.
|
|
151
|
+
|
|
152
|
+
|
|
153
|
+
## Long-query diagnostic
|
|
154
|
+
|
|
155
|
+
The 20-query corpus has two- or three-token queries and does not exercise the
|
|
156
|
+
whole-phrase failure mode. `bench/queries-long.json` adds five long Chinese
|
|
157
|
+
queries (policy, PostgreSQL, Python, Kubernetes, React) and is recorded into
|
|
158
|
+
`bench/fixtures-long/` separately:
|
|
159
|
+
|
|
160
|
+
```
|
|
161
|
+
node scripts/bench.mjs --record --queries bench/queries-long.json --fixtures bench/fixtures-long
|
|
162
|
+
node scripts/bench.mjs --queries bench/queries-long.json --fixtures bench/fixtures-long
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
Result on 0.19.0: P@8 `0.4500`, R@8 `1.0000`, MRR@8 `0.7667`, zero-hit 0, every
|
|
166
|
+
query hitting at least one expected domain. `cn.bing` contributes 25 of the 40
|
|
167
|
+
fused rows after the variant retry, against zero usable rows before it for the
|
|
168
|
+
policy query. The corpus is deliberately separate so the 20-query baseline in
|
|
169
|
+
`bench/queries.json` stays comparable.
|
|
170
|
+
|
|
134
171
|
## Known limits
|
|
135
172
|
|
|
136
173
|
- **Relative Chinese time words.** A query like "上周发布的那版" carries its time
|
package/lib/engines.js
CHANGED
|
@@ -3,6 +3,8 @@ import { filterByTimeRange, preferWindow } from "./freshness.js";
|
|
|
3
3
|
import { resolveProxy } from "./proxy.js";
|
|
4
4
|
import { parseAnySearch, parseBing, parseDdgHtml, parseDdgLite, parseExaMcp, parseFirecrawl, parseSo } from "./parsers.js";
|
|
5
5
|
import { mergeRegions } from "./rank.js";
|
|
6
|
+
import { filterRelevant } from "./relevance.js";
|
|
7
|
+
import { cjkQueryVariants } from "./variants.js";
|
|
6
8
|
import { cacheKey } from "./cache.js";
|
|
7
9
|
import { searchHost } from "./transport.js";
|
|
8
10
|
import { keyFailureKind, keyPoolFor } from "./keys.js";
|
|
@@ -111,6 +113,7 @@ export const HOSTS = [
|
|
|
111
113
|
// No `filters` parameter: see the measurement note before `HOSTS`.
|
|
112
114
|
build: (query) => `https://cn.bing.com/search?q=${encodeURIComponent(query)}`,
|
|
113
115
|
parse: parseBing,
|
|
116
|
+
variants: cjkQueryVariants,
|
|
114
117
|
},
|
|
115
118
|
{
|
|
116
119
|
id: 'so',
|
|
@@ -120,6 +123,7 @@ export const HOSTS = [
|
|
|
120
123
|
region: 'cn',
|
|
121
124
|
build: (query) => `https://www.so.com/s?q=${encodeURIComponent(query)}`,
|
|
122
125
|
parse: parseSo,
|
|
126
|
+
variants: cjkQueryVariants,
|
|
123
127
|
},
|
|
124
128
|
{
|
|
125
129
|
id: 'ddg-lite',
|
|
@@ -243,6 +247,8 @@ async function runTier(hosts, query, signal, timeoutMs, failures, proxy, limits,
|
|
|
243
247
|
const relayAbort = () => controller.abort();
|
|
244
248
|
signal?.addEventListener('abort', relayAbort, { once: true });
|
|
245
249
|
const outcomes = new Array(hosts.length).fill(undefined);
|
|
250
|
+
/** Rows each engine returned before the CJK relevance gate, for the failure text. */
|
|
251
|
+
const rawCounts = new Array(hosts.length).fill(0);
|
|
246
252
|
const answered = () => outcomes.filter((outcome) => outcome instanceof SearchError || outcome !== undefined);
|
|
247
253
|
const available = () => outcomes.reduce((total, outcome) => total + (outcome instanceof SearchError || outcome === undefined ? 0 : outcome.sources.length), 0);
|
|
248
254
|
const contributors = () => outcomes.filter((outcome, index) => counters(hosts[index]) && !(outcome instanceof SearchError) && outcome !== undefined && outcome.sources.length > 0).length;
|
|
@@ -268,7 +274,8 @@ async function runTier(hosts, query, signal, timeoutMs, failures, proxy, limits,
|
|
|
268
274
|
};
|
|
269
275
|
const running = hosts.map((host, index) => searchHostWithKeys(host, query, controller.signal, timeoutMs, proxy, limits.timeRange, pool, anysearchDomain).then((outcome) => {
|
|
270
276
|
if (outcome.ok) {
|
|
271
|
-
|
|
277
|
+
rawCounts[index] = outcome.sources.length;
|
|
278
|
+
outcomes[index] = { host, sources: filterRelevant(outcome.sources, query) };
|
|
272
279
|
health?.noteSuccess(host.id);
|
|
273
280
|
}
|
|
274
281
|
else {
|
|
@@ -307,7 +314,7 @@ async function runTier(hosts, query, signal, timeoutMs, failures, proxy, limits,
|
|
|
307
314
|
failures.push(outcome.message);
|
|
308
315
|
}
|
|
309
316
|
else if (outcome.sources.length === 0) {
|
|
310
|
-
failures.push(`${host.id}: no results`);
|
|
317
|
+
failures.push(`${host.id}: ${(rawCounts[index] ?? 0) > 0 ? 'no relevant results' : 'no results'}`);
|
|
311
318
|
}
|
|
312
319
|
else {
|
|
313
320
|
answers.push(outcome);
|
|
@@ -400,6 +407,7 @@ export async function search(query, signal, options) {
|
|
|
400
407
|
// documented shape of `ResolvedConfig.rank`.
|
|
401
408
|
const rank = {
|
|
402
409
|
...(options.rank ?? {}),
|
|
410
|
+
query,
|
|
403
411
|
...(timeRange === 'off' ? {} : { timeRange }),
|
|
404
412
|
};
|
|
405
413
|
/**
|
package/lib/rank.js
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { publishedTimestamp } from "./freshness.js";
|
|
2
|
+
import { filterRelevant } from "./relevance.js";
|
|
2
3
|
/**
|
|
3
4
|
* Reciprocal-rank-fusion constant over the interleaved slot order; a larger
|
|
4
5
|
* value flattens the curve so authority and freshness reorder more freely.
|
|
@@ -27,6 +28,17 @@ export const DEFAULT_AUTHORITY_DOMAINS = [
|
|
|
27
28
|
'dev.mysql.com',
|
|
28
29
|
'redis.io',
|
|
29
30
|
'nginx.org',
|
|
31
|
+
// Chinese official domains: policy and standards pages from a ministry or a
|
|
32
|
+
// university outrank the SEO reposts and dictionary pages that a scraped
|
|
33
|
+
// Chinese engine returns at the same rank. The suffix match in
|
|
34
|
+
// `isAuthoritative` makes `gov.cn` cover every ministry subdomain.
|
|
35
|
+
'gov.cn',
|
|
36
|
+
'edu.cn',
|
|
37
|
+
'news.cn',
|
|
38
|
+
'people.com.cn',
|
|
39
|
+
'xinhuanet.com',
|
|
40
|
+
'ncsti.gov.cn',
|
|
41
|
+
'cas.cn',
|
|
30
42
|
];
|
|
31
43
|
/** Multiplier added for an authoritative domain by default. */
|
|
32
44
|
export const DEFAULT_AUTHORITY_BOOST = 0.25;
|
|
@@ -331,8 +343,9 @@ function fuseEntries(entries, limit, options) {
|
|
|
331
343
|
for (const entry of entries) {
|
|
332
344
|
const seen = new Set();
|
|
333
345
|
const rows = [];
|
|
334
|
-
|
|
335
|
-
|
|
346
|
+
const sources = options.query === undefined ? entry.sources : filterRelevant(entry.sources, options.query);
|
|
347
|
+
for (let index = 0; index < sources.length; index += 1) {
|
|
348
|
+
const source = sources[index];
|
|
336
349
|
if (source === undefined)
|
|
337
350
|
continue;
|
|
338
351
|
if (typeof source.url !== 'string' || source.url.trim() === '')
|
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Lexical relevance for CJK queries.
|
|
3
|
+
*
|
|
4
|
+
* A Chinese query sent to a scraped engine can come back as a dictionary entry
|
|
5
|
+
* for one character of it: `东数西算 八个国家枢纽节点 定位` against `cn.bing`
|
|
6
|
+
* returned `东(汉语文字)`, `东的意思…`, and a company named 李在东, none of
|
|
7
|
+
* which share a word with the question. The engines' own rank order cannot be
|
|
8
|
+
* trusted for those rows, and letting them into the fusion is worse than
|
|
9
|
+
* dropping them: one engine's ten junk rows can outweigh the three good rows
|
|
10
|
+
* another engine found.
|
|
11
|
+
*
|
|
12
|
+
* The check is deliberately lexical and conservative: a CJK query is split into
|
|
13
|
+
* overlapping two-character tokens (its bigrams), and a row must share at least
|
|
14
|
+
* two of them with its title, snippet or URL. Two matches is enough for a page
|
|
15
|
+
* that mentions the entity and the aspect, and it is exactly what a
|
|
16
|
+
* single-character dictionary page cannot produce. English queries are left
|
|
17
|
+
* alone; their engines did not show this failure mode, and an ASCII word
|
|
18
|
+
* overlap would start rejecting legitimate paraphrases.
|
|
19
|
+
*
|
|
20
|
+
* @module @fangchenghao/dsh-web-search/relevance
|
|
21
|
+
*/
|
|
22
|
+
import type { Source } from './types.ts';
|
|
23
|
+
/** One overlap token and how much evidence a match is worth. */
|
|
24
|
+
export interface QueryToken {
|
|
25
|
+
text: string;
|
|
26
|
+
/** ASCII words are specific (`postgres`), so a match is worth two bigrams. */
|
|
27
|
+
weight: number;
|
|
28
|
+
}
|
|
29
|
+
/** A CJK query prepared for overlap checks. */
|
|
30
|
+
export interface QueryProfile {
|
|
31
|
+
/** True when the query contains at least one CJK character. */
|
|
32
|
+
cjk: boolean;
|
|
33
|
+
/** Distinct two-character tokens (CJK bigrams) plus ASCII words. */
|
|
34
|
+
tokens: QueryToken[];
|
|
35
|
+
/** How many weight units a row must accumulate. */
|
|
36
|
+
required: number;
|
|
37
|
+
}
|
|
38
|
+
/**
|
|
39
|
+
* Build the overlap profile for a query.
|
|
40
|
+
* @param query - the user's question.
|
|
41
|
+
* @returns the profile; `cjk` is false for an English-only query.
|
|
42
|
+
*/
|
|
43
|
+
export declare function queryProfile(query: string): QueryProfile;
|
|
44
|
+
/**
|
|
45
|
+
* Whether one row talks about the query at all.
|
|
46
|
+
* @param source - engine row with title, snippet and URL.
|
|
47
|
+
* @param profile - prepared query tokens.
|
|
48
|
+
* @returns true when at least `profile.required` distinct tokens appear.
|
|
49
|
+
*/
|
|
50
|
+
export declare function isRelevant(source: Source, profile: QueryProfile): boolean;
|
|
51
|
+
/**
|
|
52
|
+
* Drop rows that do not share enough words with a CJK query.
|
|
53
|
+
* @param sources - one engine's rows.
|
|
54
|
+
* @param query - the user's question.
|
|
55
|
+
* @returns the rows that survive; English queries are returned unchanged.
|
|
56
|
+
*/
|
|
57
|
+
export declare function filterRelevant(sources: readonly Source[], query: string): Source[];
|
package/lib/relevance.js
ADDED
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
/** Runs of CJK characters (Han, kana, Hangul). */
|
|
2
|
+
const CJK_RUN = /[\u3040-\u30ff\u3400-\u4dbf\u4e00-\u9fff\uac00-\ud7af\uf900-\ufaff]+/gu;
|
|
3
|
+
/** Common CJK function-word bigrams that match almost any page. */
|
|
4
|
+
const CJK_STOPWORDS = new Set([
|
|
5
|
+
'的是', '了吗', '什么', '怎么', '如何', '哪些', '多少', '一下', '请问',
|
|
6
|
+
'帮我', '告诉', '介绍', '以及', '或者', '还是', '就是', '可以', '需要',
|
|
7
|
+
]);
|
|
8
|
+
/**
|
|
9
|
+
* Build the overlap profile for a query.
|
|
10
|
+
* @param query - the user's question.
|
|
11
|
+
* @returns the profile; `cjk` is false for an English-only query.
|
|
12
|
+
*/
|
|
13
|
+
export function queryProfile(query) {
|
|
14
|
+
const text = query.toLowerCase();
|
|
15
|
+
const tokens = new Map();
|
|
16
|
+
let cjk = false;
|
|
17
|
+
for (const run of text.match(CJK_RUN) ?? []) {
|
|
18
|
+
cjk = true;
|
|
19
|
+
if (run.length === 1) {
|
|
20
|
+
// A one-character query has no bigram; keep the character itself so the
|
|
21
|
+
// check still has something to match.
|
|
22
|
+
tokens.set(run, { text: run, weight: 1 });
|
|
23
|
+
continue;
|
|
24
|
+
}
|
|
25
|
+
for (let index = 0; index < run.length - 1; index += 1) {
|
|
26
|
+
const bigram = run.slice(index, index + 2);
|
|
27
|
+
if (!CJK_STOPWORDS.has(bigram))
|
|
28
|
+
tokens.set(bigram, { text: bigram, weight: 1 });
|
|
29
|
+
}
|
|
30
|
+
}
|
|
31
|
+
// A specific ASCII word is stronger evidence than a two-character slice of a
|
|
32
|
+
// Chinese phrase: a PostgreSQL page can match `postgres` without containing
|
|
33
|
+
// any of the Chinese words around it, and dropping it would be worse than
|
|
34
|
+
// keeping a dictionary page.
|
|
35
|
+
for (const word of text.match(/[a-z][a-z0-9.+#_-]{2,}/gu) ?? [])
|
|
36
|
+
tokens.set(word, { text: word, weight: 2 });
|
|
37
|
+
const required = !cjk || tokens.size <= 1 ? 1 : Math.min(3, Math.max(2, Math.ceil(tokens.size * 0.2)));
|
|
38
|
+
return { cjk, tokens: [...tokens.values()], required };
|
|
39
|
+
}
|
|
40
|
+
/**
|
|
41
|
+
* Whether one row talks about the query at all.
|
|
42
|
+
* @param source - engine row with title, snippet and URL.
|
|
43
|
+
* @param profile - prepared query tokens.
|
|
44
|
+
* @returns true when at least `profile.required` distinct tokens appear.
|
|
45
|
+
*/
|
|
46
|
+
export function isRelevant(source, profile) {
|
|
47
|
+
if (!profile.cjk)
|
|
48
|
+
return true;
|
|
49
|
+
let haystack = `${source.title ?? ''} ${source.snippet ?? ''}`.toLowerCase();
|
|
50
|
+
try {
|
|
51
|
+
haystack += ` ${decodeURIComponent(source.url)}`.toLowerCase();
|
|
52
|
+
}
|
|
53
|
+
catch {
|
|
54
|
+
haystack += ` ${source.url}`.toLowerCase();
|
|
55
|
+
}
|
|
56
|
+
let matched = 0;
|
|
57
|
+
for (const token of profile.tokens) {
|
|
58
|
+
if (haystack.includes(token.text)) {
|
|
59
|
+
matched += token.weight;
|
|
60
|
+
if (matched >= profile.required)
|
|
61
|
+
return true;
|
|
62
|
+
}
|
|
63
|
+
}
|
|
64
|
+
return profile.required === 0;
|
|
65
|
+
}
|
|
66
|
+
/**
|
|
67
|
+
* Drop rows that do not share enough words with a CJK query.
|
|
68
|
+
* @param sources - one engine's rows.
|
|
69
|
+
* @param query - the user's question.
|
|
70
|
+
* @returns the rows that survive; English queries are returned unchanged.
|
|
71
|
+
*/
|
|
72
|
+
export function filterRelevant(sources, query) {
|
|
73
|
+
const profile = queryProfile(query);
|
|
74
|
+
if (!profile.cjk)
|
|
75
|
+
return [...sources];
|
|
76
|
+
return sources.filter((source) => isRelevant(source, profile));
|
|
77
|
+
}
|
package/lib/transport.d.ts
CHANGED
|
@@ -18,21 +18,4 @@ export type AttemptOutcome = {
|
|
|
18
18
|
ok: false;
|
|
19
19
|
error: SearchError;
|
|
20
20
|
};
|
|
21
|
-
/**
|
|
22
|
-
* Ask one engine, going direct first unless it is marked `viaProxy`.
|
|
23
|
-
*
|
|
24
|
-
* A domestic engine goes direct first, because that is both faster and the path
|
|
25
|
-
* its site expects; only if that fails is it retried through the tunnel, so a
|
|
26
|
-
* network that blocks it directly still works. An engine marked `viaProxy` is
|
|
27
|
-
* tunnelled from the start.
|
|
28
|
-
*
|
|
29
|
-
* @param host - the engine.
|
|
30
|
-
* @param query - the user query.
|
|
31
|
-
* @param signal - cancellation.
|
|
32
|
-
* @param timeoutMs - per-attempt deadline.
|
|
33
|
-
* @param proxy - tunnel URL, or `undefined` when none is available.
|
|
34
|
-
* @param timeRange - active publication window, forwarded to engines that can
|
|
35
|
-
* push it into their query (currently DuckDuckGo's `df`).
|
|
36
|
-
* @returns the outcome of the route that got furthest; cancellation is rethrown.
|
|
37
|
-
*/
|
|
38
21
|
export declare function searchHost(host: HostSpec, query: string, signal: AbortSignal | undefined, timeoutMs?: number, proxy?: string, timeRange?: TimeRange): Promise<AttemptOutcome>;
|
package/lib/transport.js
CHANGED
|
@@ -147,8 +147,16 @@ async function attempt(host, url, signal, timeoutMs, proxy, query) {
|
|
|
147
147
|
* push it into their query (currently DuckDuckGo's `df`).
|
|
148
148
|
* @returns the outcome of the route that got furthest; cancellation is rethrown.
|
|
149
149
|
*/
|
|
150
|
-
|
|
150
|
+
/** Ask one engine one query, with the direct-then-tunnel retry. */
|
|
151
|
+
async function attemptWithRoute(host, query, signal, timeoutMs, route, proxy, timeRange) {
|
|
151
152
|
const url = host.build(query, timeRange);
|
|
153
|
+
const first = await attempt(host, url, signal, timeoutMs, route, query);
|
|
154
|
+
if (first.ok || route !== undefined || proxy === undefined || signal?.aborted === true)
|
|
155
|
+
return first;
|
|
156
|
+
const retry = await attempt(host, url, signal, timeoutMs, proxy, query);
|
|
157
|
+
return retry.ok ? retry : first;
|
|
158
|
+
}
|
|
159
|
+
export async function searchHost(host, query, signal, timeoutMs = 15000, proxy, timeRange) {
|
|
152
160
|
const route = host.viaProxy === true ? proxy : undefined;
|
|
153
161
|
// A gate holds the permit for the whole engine attempt, including the
|
|
154
162
|
// direct-then-tunnel retry, so a burst of concurrent searches cannot stack
|
|
@@ -157,11 +165,28 @@ export async function searchHost(host, query, signal, timeoutMs = 15000, proxy,
|
|
|
157
165
|
? undefined
|
|
158
166
|
: await gateFor(host.id, host.maxConcurrent).acquire(signal);
|
|
159
167
|
try {
|
|
160
|
-
const
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
168
|
+
const result = await attemptWithRoute(host, query, signal, timeoutMs, route, proxy, timeRange);
|
|
169
|
+
const variants = host.variants?.(query) ?? [];
|
|
170
|
+
if (variants.length === 0 || signal?.aborted === true)
|
|
171
|
+
return result;
|
|
172
|
+
const merged = result.ok ? [...result.sources] : [];
|
|
173
|
+
const seen = new Set(merged.map((source) => source.url));
|
|
174
|
+
for (const variant of variants) {
|
|
175
|
+
// `signal?.aborted === true` above narrowed the false branch to
|
|
176
|
+
// `aborted: false`, so test truthiness here instead of the literal.
|
|
177
|
+
if (signal?.aborted)
|
|
178
|
+
break;
|
|
179
|
+
const extra = await attemptWithRoute(host, variant, signal, timeoutMs, route, proxy, timeRange);
|
|
180
|
+
if (!extra.ok)
|
|
181
|
+
continue;
|
|
182
|
+
for (const source of extra.sources) {
|
|
183
|
+
if (seen.has(source.url))
|
|
184
|
+
continue;
|
|
185
|
+
seen.add(source.url);
|
|
186
|
+
merged.push(source);
|
|
187
|
+
}
|
|
188
|
+
}
|
|
189
|
+
return merged.length > 0 ? { ok: true, sources: merged } : result;
|
|
165
190
|
}
|
|
166
191
|
finally {
|
|
167
192
|
release?.();
|
package/lib/types.d.ts
CHANGED
|
@@ -54,6 +54,15 @@ export interface HostSpec {
|
|
|
54
54
|
skipEcho?: boolean;
|
|
55
55
|
/** Confirm the payload really belongs to the query (scraped pages only). */
|
|
56
56
|
echoesQuery?: (payload: string, query: string) => boolean;
|
|
57
|
+
/**
|
|
58
|
+
* Alternate spellings to ask when the engine matches whole phrases.
|
|
59
|
+
*
|
|
60
|
+
* `cn.bing` answers a long Chinese query with a dictionary page for its first
|
|
61
|
+
* character; asking once more with the core words is the only way to get the
|
|
62
|
+
* rows it has. The transport merges the variant's rows after the original's,
|
|
63
|
+
* so ranking still sees the engine's own order first.
|
|
64
|
+
*/
|
|
65
|
+
variants?: (query: string) => string[];
|
|
57
66
|
/** Absolute request URL for a query. `timeRange` may be pushed down when the engine supports it. */
|
|
58
67
|
build: (query: string, timeRange?: TimeRange) => string;
|
|
59
68
|
/** Request body for a JSON POST. */
|
|
@@ -125,6 +134,14 @@ export interface RankOptions {
|
|
|
125
134
|
timeRange?: TimeRange;
|
|
126
135
|
/** Per-engine weight in the pre-fusion interleave. */
|
|
127
136
|
weights?: number[];
|
|
137
|
+
/**
|
|
138
|
+
* The query the engine rows answer.
|
|
139
|
+
*
|
|
140
|
+
* When present, CJK rows that share too few words with it are discarded
|
|
141
|
+
* before fusion (see `relevance.ts`). Bench replay passes the fixture's query
|
|
142
|
+
* so an offline run measures the same gate production applies.
|
|
143
|
+
*/
|
|
144
|
+
query?: string;
|
|
128
145
|
/**
|
|
129
146
|
* Evaluation instant in milliseconds for the freshness boost, defaulting to
|
|
130
147
|
* `Date.now()`. Benchmarks pin this to the fixture's recording time so a
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Query variants for engines that match a whole phrase.
|
|
3
|
+
*
|
|
4
|
+
* `cn.bing` treats a long Chinese query as one phrase: when the phrase is not
|
|
5
|
+
* in its index it falls back to a dictionary page for the first character.
|
|
6
|
+
* `东数西算 枢纽节点` answers correctly, `东数西算 八个国家枢纽节点 定位`
|
|
7
|
+
* answers with ten entries for `东`, and `postgres 逻辑复制` answers with the
|
|
8
|
+
* PostgreSQL manual while `如何配置 postgres 逻辑复制` answers with the
|
|
9
|
+
* dictionary entry for `如何`. Re-asking once with the core words turns those
|
|
10
|
+
* failures back into relevant rows, at the cost of one extra request that the
|
|
11
|
+
* tier's soft deadline bounds.
|
|
12
|
+
*
|
|
13
|
+
* The rule is deliberately small and explainable:
|
|
14
|
+
*
|
|
15
|
+
* - a mixed query (CJK plus ASCII) drops question/action/intent segments and
|
|
16
|
+
* keeps the technical ones (`如何配置 postgres 逻辑复制` → `postgres 逻辑复制`);
|
|
17
|
+
* - a pure CJK query keeps its first content segment, which is where the entity
|
|
18
|
+
* usually sits (`东数西算 八个国家枢纽节点 定位` → `东数西算`);
|
|
19
|
+
* - everything else gets no variant, so an ordinary two-word query pays nothing.
|
|
20
|
+
*
|
|
21
|
+
* @module @fangchenghao/dsh-web-search/variants
|
|
22
|
+
*/
|
|
23
|
+
/**
|
|
24
|
+
* Build at most one alternate query for a whole-phrase engine.
|
|
25
|
+
* @param query - the user's original query.
|
|
26
|
+
* @returns a single variant, or an empty array when none is likely useful.
|
|
27
|
+
*/
|
|
28
|
+
export declare function cjkQueryVariants(query: string): string[];
|
package/lib/variants.js
ADDED
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Query variants for engines that match a whole phrase.
|
|
3
|
+
*
|
|
4
|
+
* `cn.bing` treats a long Chinese query as one phrase: when the phrase is not
|
|
5
|
+
* in its index it falls back to a dictionary page for the first character.
|
|
6
|
+
* `东数西算 枢纽节点` answers correctly, `东数西算 八个国家枢纽节点 定位`
|
|
7
|
+
* answers with ten entries for `东`, and `postgres 逻辑复制` answers with the
|
|
8
|
+
* PostgreSQL manual while `如何配置 postgres 逻辑复制` answers with the
|
|
9
|
+
* dictionary entry for `如何`. Re-asking once with the core words turns those
|
|
10
|
+
* failures back into relevant rows, at the cost of one extra request that the
|
|
11
|
+
* tier's soft deadline bounds.
|
|
12
|
+
*
|
|
13
|
+
* The rule is deliberately small and explainable:
|
|
14
|
+
*
|
|
15
|
+
* - a mixed query (CJK plus ASCII) drops question/action/intent segments and
|
|
16
|
+
* keeps the technical ones (`如何配置 postgres 逻辑复制` → `postgres 逻辑复制`);
|
|
17
|
+
* - a pure CJK query keeps its first content segment, which is where the entity
|
|
18
|
+
* usually sits (`东数西算 八个国家枢纽节点 定位` → `东数西算`);
|
|
19
|
+
* - everything else gets no variant, so an ordinary two-word query pays nothing.
|
|
20
|
+
*
|
|
21
|
+
* @module @fangchenghao/dsh-web-search/variants
|
|
22
|
+
*/
|
|
23
|
+
/** Segments that ask rather than name. */
|
|
24
|
+
const QUESTION_PREFIX = /^(如何|怎么|怎样|为什么|为何|哪些|什么|多少|请问|帮我|帮忙|查一下|介绍一下|介绍|解释一下|解释|说明一下|说明|有没有|是否|想知道|求)/u;
|
|
25
|
+
/** Whole segments that add intent rather than retrieval signal. */
|
|
26
|
+
const INTENT_SEGMENTS = new Set([
|
|
27
|
+
'定位', '介绍', '含义', '意思', '意义', '区别', '对比', '比较', '方法',
|
|
28
|
+
'教程', '步骤', '指南', '攻略', '用法', '示例', '例子', '作用', '原理',
|
|
29
|
+
'配置', '安装', '使用', '设置', '部署', '怎么用', '如何用', '有哪些',
|
|
30
|
+
]);
|
|
31
|
+
/** Characters that make a segment CJK. */
|
|
32
|
+
const CJK = /[\u3400-\u9fff\uf900-\ufaff]/u;
|
|
33
|
+
/** Split a query into the labels a user separated with spaces. */
|
|
34
|
+
function segmentsOf(query) {
|
|
35
|
+
return query.trim().split(/\s+/u).filter((segment) => segment.length > 0);
|
|
36
|
+
}
|
|
37
|
+
/** Strip a leading question word and report whether anything usable is left. */
|
|
38
|
+
function contentOf(segment) {
|
|
39
|
+
const stripped = segment.replace(QUESTION_PREFIX, '').trim();
|
|
40
|
+
if (stripped.length < 2)
|
|
41
|
+
return undefined;
|
|
42
|
+
if (INTENT_SEGMENTS.has(stripped))
|
|
43
|
+
return undefined;
|
|
44
|
+
return stripped;
|
|
45
|
+
}
|
|
46
|
+
/** Whether a whole segment only carries intent. */
|
|
47
|
+
function isIntent(segment) {
|
|
48
|
+
return INTENT_SEGMENTS.has(segment) || QUESTION_PREFIX.test(segment);
|
|
49
|
+
}
|
|
50
|
+
/**
|
|
51
|
+
* Build at most one alternate query for a whole-phrase engine.
|
|
52
|
+
* @param query - the user's original query.
|
|
53
|
+
* @returns a single variant, or an empty array when none is likely useful.
|
|
54
|
+
*/
|
|
55
|
+
export function cjkQueryVariants(query) {
|
|
56
|
+
if (!CJK.test(query))
|
|
57
|
+
return [];
|
|
58
|
+
const segments = segmentsOf(query);
|
|
59
|
+
if (segments.length < 2)
|
|
60
|
+
return [];
|
|
61
|
+
const ascii = segments.some((segment) => /^[a-z0-9][a-z0-9.+#_-]*$/iu.test(segment));
|
|
62
|
+
const variant = ascii
|
|
63
|
+
? segments
|
|
64
|
+
.filter((segment) => !isIntent(segment))
|
|
65
|
+
.map((segment) => contentOf(segment) ?? '')
|
|
66
|
+
.filter((segment) => segment.length > 0)
|
|
67
|
+
.join(' ')
|
|
68
|
+
: (contentOf(segments[0] ?? '') ?? '');
|
|
69
|
+
if (variant.length === 0)
|
|
70
|
+
return [];
|
|
71
|
+
if (variant.toLowerCase() === query.trim().replace(/\s+/gu, ' ').toLowerCase())
|
|
72
|
+
return [];
|
|
73
|
+
return [variant];
|
|
74
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@fffangchenghao/dsh-web-search",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.19.0",
|
|
4
4
|
"description": "DSH bundle: a key-free web search provider. Leads with the anonymous AnySearch API and falls back to public result pages (DuckDuckGo, Bing, 360), merged by reciprocal-rank fusion with a bilingual region policy, an engine cooldown and a TTL cache.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "lib/index.js",
|