bookmark-research 0.5.0-beta.4 → 0.5.0-beta.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/plugins/marketplace.json +1 -1
- package/.codex-plugin/plugin.json +1 -1
- package/README.md +19 -1
- package/README.zh.md +19 -1
- package/docs/distribution.md +3 -3
- package/docs/installation.en.md +2 -2
- package/docs/installation.md +2 -2
- package/docs/release-notes-0.5.0-beta.4.md +1 -1
- package/docs/release-notes-0.5.0-beta.5.md +31 -0
- package/docs/wiki-quality.md +3 -1
- package/docs/wiki-quality.zh.md +3 -1
- package/package.json +1 -1
- package/skills/bookmark-research/SKILL.md +18 -5
- package/skills/bookmark-research/references/wiki-and-evaluation.md +3 -1
- package/skills/bookmark-research/references/zh/skill-guide.md +17 -4
- package/skills/bookmark-research/references/zh/wiki-and-evaluation.md +3 -1
- package/src/archive.py +24 -3
- package/src/mcp_server.py +2 -2
- package/src/wiki.py +30 -14
package/README.md
CHANGED
|
@@ -11,6 +11,8 @@ Works in **Codex, Claude Code, Pi and DSH (DeepSeek Harness)**.
|
|
|
11
11
|
> [!NOTE]
|
|
12
12
|
> **0.5.0 is a beta.** Report problems on [GitHub Issues](https://github.com/Browser-bookmark-hub/Bookmark-Research/issues).
|
|
13
13
|
|
|
14
|
+
**Install releases from [npm](https://www.npmjs.com/package/bookmark-research).** GitHub Releases retain version notes and npm links; downloadable installation/test packages have been withdrawn. GitHub's automatic **Source code** archives are source checkouts, not installation packages.
|
|
15
|
+
|
|
14
16
|
## Requirements
|
|
15
17
|
|
|
16
18
|
- Python 3.9+ with SQLite FTS5 (check with `bookmark-research doctor`)
|
|
@@ -20,7 +22,15 @@ Works in **Codex, Claude Code, Pi and DSH (DeepSeek Harness)**.
|
|
|
20
22
|
|
|
21
23
|
## Install
|
|
22
24
|
|
|
23
|
-
**
|
|
25
|
+
**Current beta (`0.5.0-beta.5`)**, interactive on macOS, Linux and Windows:
|
|
26
|
+
|
|
27
|
+
```sh
|
|
28
|
+
npx bookmark-research@0.5.0-beta.5 install --source npm:bookmark-research@0.5.0-beta.5
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
This pins both the installer and the content. The `@latest` examples below follow the default channel, which remains `0.5.0-beta.3`; `@beta` currently points to `0.5.0-beta.5`. Changing only the npx selector does not change the installer's default content source.
|
|
32
|
+
|
|
33
|
+
**Default-channel interactive installation:**
|
|
24
34
|
|
|
25
35
|
```sh
|
|
26
36
|
npx bookmark-research@latest install
|
|
@@ -102,6 +112,14 @@ Nothing is written into your bookmark files or the plugin. The data directory is
|
|
|
102
112
|
| `raw-library.sqlite3` | Rebuildable full-text index for `search_archive` |
|
|
103
113
|
| `${XDG_CONFIG_HOME:-~/.config}/bookmark-research/` | `settings.json` and `credentials.json` (private, owner-only) |
|
|
104
114
|
|
|
115
|
+
## Agent loops and delegation
|
|
116
|
+
|
|
117
|
+
The host model plans the research, searches or reads pages, evaluates the evidence and follows unresolved questions. The Skill guides this loop; MCP stores inputs, evidence, budgets and progress. Saving a research task does not start a background agent.
|
|
118
|
+
|
|
119
|
+
For a grouped report, the plugin supplies a reader → fresh verifier → coverage check → follow-up → report workflow. Codex uses its native subagent tools and the bundled delegation guide; Claude Code and DSH use available host workflow hooks, while Pi requires a loaded subagent/workflow extension. The host owns concurrency, cancellation and agent lifetime. All children must receive the relevant instructions and access the same research store.
|
|
120
|
+
|
|
121
|
+
The workflow adapters have simulated contract tests, but a successful install does not prove that every host can execute the complete multi-agent workflow. See [host workflows](skills/bookmark-research/references/host-workflows.md) for requirements and [the validation matrix](docs/validation-0.4.0.md#host-and-provider-boundaries) for observed limits.
|
|
122
|
+
|
|
105
123
|
## What's inside
|
|
106
124
|
|
|
107
125
|
A bundle of one Skill, one local MCP server, and the remote MCPs and services that server connects to. The client registers only the local MCP.
|
package/README.zh.md
CHANGED
|
@@ -11,6 +11,8 @@
|
|
|
11
11
|
> [!NOTE]
|
|
12
12
|
> **0.5.0 是测试版。** 遇到问题请到 [GitHub Issues](https://github.com/Browser-bookmark-hub/Bookmark-Research/issues) 反馈。
|
|
13
13
|
|
|
14
|
+
**发布版统一从 [npm](https://www.npmjs.com/package/bookmark-research) 安装。** GitHub Releases 保留版本说明和 npm 链接,安装包/测试包附件已撤下。GitHub 自动生成的 **Source code** 压缩包是源码快照,不是安装包。
|
|
15
|
+
|
|
14
16
|
## 环境要求
|
|
15
17
|
|
|
16
18
|
- Python 3.9+,含 SQLite FTS5(可用 `bookmark-research doctor` 检查)
|
|
@@ -20,7 +22,15 @@
|
|
|
20
22
|
|
|
21
23
|
## 安装
|
|
22
24
|
|
|
23
|
-
|
|
25
|
+
**当前测试版(`0.5.0-beta.5`)**,支持 macOS、Linux、Windows 交互安装:
|
|
26
|
+
|
|
27
|
+
```sh
|
|
28
|
+
npx bookmark-research@0.5.0-beta.5 install --source npm:bookmark-research@0.5.0-beta.5
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
这条命令同时固定安装器和插件内容。下面的 `@latest` 示例使用默认渠道,目前仍为 `0.5.0-beta.3`;`@beta` 当前指向 `0.5.0-beta.5`。只改 npx 的版本选择,不会改变安装器默认下载的内容来源。
|
|
32
|
+
|
|
33
|
+
**默认渠道的交互安装:**
|
|
24
34
|
|
|
25
35
|
```sh
|
|
26
36
|
npx bookmark-research@latest install
|
|
@@ -102,6 +112,14 @@ setup 还会问研究报告和证据放在哪里:统一目录(默认 `<数
|
|
|
102
112
|
| `raw-library.sqlite3` | `search_archive` 使用的全文索引,可重建 |
|
|
103
113
|
| `${XDG_CONFIG_HOME:-~/.config}/bookmark-research/` | `settings.json` 和 `credentials.json`(仅当前用户可读) |
|
|
104
114
|
|
|
115
|
+
## Agent 循环与子代理
|
|
116
|
+
|
|
117
|
+
宿主模型负责规划、搜索/阅读、评估证据,再追查未解决的问题。Skill 指导这个循环,MCP 保存输入范围、证据、预算和进度;保存研究任务不会启动后台代理。
|
|
118
|
+
|
|
119
|
+
对于分组报告,插件提供“阅读代理 → 新的验证代理 → 覆盖检查 → 补查 → 报告”的流程。Codex 使用原生子代理和附带的委派指南;Claude Code、DSH 使用已加载的宿主工作流接口;Pi 需要加载相应的子代理/工作流扩展。并发、取消和代理生命周期由宿主管理。每个子代理都需要拿到相关指令,并访问同一个研究存储。
|
|
120
|
+
|
|
121
|
+
工作流适配已有模拟接口测试,但安装成功不代表所有宿主都已跑通完整的多代理任务。所需条件见[宿主工作流](skills/bookmark-research/references/zh/host-workflows.md),实际验证范围见[验证记录](docs/validation-0.4.0.md#host-and-provider-boundaries)。
|
|
122
|
+
|
|
105
123
|
## 插件包含什么
|
|
106
124
|
|
|
107
125
|
由 1 个 Skill、1 个本地 MCP 服务,以及这个服务内部连接的远程 MCP 和网页服务组成。宿主里只需要登记这 1 个本地 MCP。
|
package/docs/distribution.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# 发布与安装约定
|
|
2
2
|
|
|
3
|
-
正式发布只使用 npm 的 `bookmark-research` 包。它包含共享 Python 运行时、Skill 和各宿主适配文件;不维护四份独立源码或四个发布版本。GitHub 用于源码、问题跟踪、版本说明及安装清单。
|
|
3
|
+
正式发布只使用 npm 的 `bookmark-research` 包。它包含共享 Python 运行时、Skill 和各宿主适配文件;不维护四份独立源码或四个发布版本。GitHub 用于源码、问题跟踪、版本说明及安装清单。GitHub Release 可选,仅保留版本说明和 npm 链接,不再附安装包、测试包或配套校验文件;历史附件已备份撤下。
|
|
4
4
|
|
|
5
5
|
| 入口 | 插件内容来源 | 登记和缓存 |
|
|
6
6
|
| --- | --- | --- |
|
|
@@ -15,11 +15,11 @@
|
|
|
15
15
|
|
|
16
16
|
默认安装器记录 `npm:bookmark-research`(`latest`),更新时重新下载,不依赖已清理的 npx 缓存。显式 `--source npm:bookmark-research@VERSION` 会保留版本;要测试同一个固定发布版,在四个宿主使用相同参数。Codex 原生 marketplace 使用清单固定版本,跟随清单更新。需要新版安装器时运行 `npx bookmark-research@latest update`,或先更新全局 npm 安装。
|
|
17
17
|
|
|
18
|
-
源码开发使用 `python3 scripts/install.py install --source PATH`,也可显式选择 Git 仓库和 `--ref`。旧 Git/本地安装保留来源,不静默转成 npm;切换时先用宿主命令处理旧登记。源码导出、ZIP 和测试包脚本保留为开发工具,CI
|
|
18
|
+
源码开发使用 `python3 scripts/install.py install --source PATH`,也可显式选择 Git 仓库和 `--ref`。旧 Git/本地安装保留来源,不静默转成 npm;切换时先用宿主命令处理旧登记。源码导出、ZIP 和测试包脚本保留为开发工具,CI 的验证产物仍可包含 npm `.tgz` 与 SHA-256 校验文件,但不作为 GitHub Release 下载入口。GitHub 自动生成的 Source code 压缩包是源码快照,不能单独关闭,不作为安装包。
|
|
19
19
|
|
|
20
20
|
发布顺序:修改源码版本并完成 CI,发布已经验证的 npm 包,确认 registry 可取得该版本,再更新 GitHub marketplace 的固定版本。未发布源码不能通过 npm 入口安装;需要先用源码入口测试。
|
|
21
21
|
|
|
22
|
-
测试版使用 npm 的 `beta`
|
|
22
|
+
测试版使用 npm 的 `beta` 标签;若创建 GitHub 版本说明,则标记为预发布。不会自动移动 npm 的 `latest` 标签。安装测试版时同时固定安装器与内容来源,例如 `npx bookmark-research@0.5.0-beta.5 install --source npm:bookmark-research@0.5.0-beta.5`。仅将 npx 改为 `@beta` 不会改变默认内容来源 `latest`;已固定的旧版安装也不会自动换成测试版。
|
|
23
23
|
|
|
24
24
|
## 官方资料与项目实例
|
|
25
25
|
|
package/docs/installation.en.md
CHANGED
|
@@ -79,7 +79,7 @@ Optional service failures appear under `setup.needs_attention` and do not block
|
|
|
79
79
|
|
|
80
80
|
## Source development and historical ZIPs
|
|
81
81
|
|
|
82
|
-
GitHub [Releases](https://github.com/Browser-bookmark-hub/Bookmark-Research/releases) retain
|
|
82
|
+
GitHub [Releases](https://github.com/Browser-bookmark-hub/Bookmark-Research/releases) retain version notes and npm links; installation and test attachments have been withdrawn. Release packages come from npm. ZIP/export helpers remain available for local development and tests; preserve required plugin manifests when extracting a historical package.
|
|
83
83
|
|
|
84
84
|
From a checkout or ZIP containing the installer:
|
|
85
85
|
|
|
@@ -110,7 +110,7 @@ The Python installer accepts the same host/setup flags. `verify --installed-path
|
|
|
110
110
|
|
|
111
111
|
### Native Codex installation of the published release
|
|
112
112
|
|
|
113
|
-
The repository marketplace points to the
|
|
113
|
+
The repository marketplace points to the published npm version recorded in its `source.version` field. Codex downloads that package without lifecycle scripts, then owns registration and caching. Adding the repository installs the published release, including its released Skill; it does not install unpublished source changes from `main`.
|
|
114
114
|
|
|
115
115
|
```sh
|
|
116
116
|
codex plugin marketplace add Browser-bookmark-hub/Bookmark-Research --ref main
|
package/docs/installation.md
CHANGED
|
@@ -96,7 +96,7 @@ Agent 可使用 `install --host pi --non-interactive --preferences FILE --skip-c
|
|
|
96
96
|
|
|
97
97
|
## 源码开发与历史 ZIP
|
|
98
98
|
|
|
99
|
-
GitHub [Releases](https://github.com/Browser-bookmark-hub/Bookmark-Research/releases)
|
|
99
|
+
GitHub [Releases](https://github.com/Browser-bookmark-hub/Bookmark-Research/releases) 保留版本说明和 npm 链接,安装包与测试包附件已撤下。发布包统一从 npm 获取;仓库的 ZIP/导出脚本仍可用于本地验证。
|
|
100
100
|
|
|
101
101
|
```sh
|
|
102
102
|
git clone https://github.com/Browser-bookmark-hub/Bookmark-Research.git
|
|
@@ -150,7 +150,7 @@ python3 "/absolute/path/Bookmark Research/scripts/install.py" install
|
|
|
150
150
|
|
|
151
151
|
旧版 Git marketplace 由 Codex 保留原登记和 ref。安装器先执行原生 `marketplace upgrade`,从宿主实际获取的 checkout 导出干净目录,再通过本次命令的 `-c` 参数将目录交给原生 `plugin add`;参数不会改写 Git 登记,也不需要猜测原生列表未返回的 ref。这条回退路径适用于仍指向仓库根目录的历史 catalog。当前发行 catalog 指向 npm 包,原生更新保留该发布来源。Git checkout 和 GitHub 自动生成的源码压缩包可以含开发文件,它们不是插件缓存。
|
|
152
152
|
|
|
153
|
-
当前仓库的 marketplace
|
|
153
|
+
当前仓库的 marketplace 固定指向其 `source.version` 字段记录的已发布 npm 版本。Codex 下载该包,不执行生命周期脚本,再自行登记和缓存。直接添加 GitHub 仓库会安装发布版及其已发布 Skill,不会安装 `main` 尚未发布的源码改动:
|
|
154
154
|
|
|
155
155
|
```sh
|
|
156
156
|
codex plugin marketplace add Browser-bookmark-hub/Bookmark-Research --ref main
|
|
@@ -36,4 +36,4 @@ The implementation passed the existing 495-test local suite (one opt-in test ski
|
|
|
36
36
|
|
|
37
37
|
Web reading was also exercised: one URL failed through all configured fallback readers; another returned a partial excerpt that could be retrieved from the local archive. This release does not guarantee webpage availability, complete extraction, interpretation of unstated user intent or large-canvas performance. End-to-end Skill triggering across every host and every MCP operation is not established by these tests.
|
|
38
38
|
|
|
39
|
-
|
|
39
|
+
Install the tested release from [npm](https://www.npmjs.com/package/bookmark-research/v/0.5.0-beta.4). GitHub keeps these version notes and installation catalogs; uploaded installation/test packages and their checksum attachments have been withdrawn from all Releases. GitHub's automatically generated source archives are source checkouts, not installation packages.
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# Bookmark Research 0.5.0 beta 5
|
|
2
|
+
|
|
3
|
+
**Beta / 测试版.** Uses the npm `beta` channel. The npm `latest` tag remains unchanged.
|
|
4
|
+
|
|
5
|
+
## Install this beta
|
|
6
|
+
|
|
7
|
+
Pin both the installer and the plugin content:
|
|
8
|
+
|
|
9
|
+
```sh
|
|
10
|
+
npx bookmark-research@0.5.0-beta.5 install --source npm:bookmark-research@0.5.0-beta.5
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
Choose Codex, Claude Code, Pi or DSH in the installer. Existing installations retain their recorded source and version selection. The GitHub Codex marketplace advances to this npm version after the registry confirms publication.
|
|
14
|
+
|
|
15
|
+
## Changes
|
|
16
|
+
|
|
17
|
+
- **Retrieve saved knowledge when needed.** The Skill uses Wiki when earlier research or personal knowledge is needed and missing from the conversation. Supplied text and sufficient existing evidence can be used directly; current public facts can go straight to current sources.
|
|
18
|
+
- **Reuse suitable host tools and evidence.** API, SDK and repository questions prefer available documentation or file tools. Attributable source excerpts can answer narrow questions; additional fetching follows evidence gaps or freshness requirements. Host-native evidence can be retained through the existing import tool.
|
|
19
|
+
- **Preserve fetched documentation.** Fenced code examples containing `URL:` fields or apparent page headers no longer cause a valid multi-page response to lose its extracted text. Ambiguous, unclosed examples retain the raw response.
|
|
20
|
+
- **Keep Wiki references usable after relocation.** New records use evidence paths relative to their research task. Reads resolve verified references at current locations, including legacy records, without rewriting history. Claims, quotes and artifact hashes remain checked.
|
|
21
|
+
- **Show review state in Wiki search.** Results include revision dates and validation. Changed bookmark input produces `needs_review`; invalid or retracted evidence remains excluded.
|
|
22
|
+
|
|
23
|
+
## Validation and limits
|
|
24
|
+
|
|
25
|
+
The functional diff passed [all 10 GitHub CI jobs](https://github.com/Browser-bookmark-hub/Bookmark-Research/actions/runs/37750192790), covering Python 3.9/3.12 on Linux, macOS and Windows, native Codex installation/update, npm installation and package contents. The local suite discovered 499 tests: 496 passed and three conditional integration tests were skipped. Platform-specific and unavailable-host skips are recorded separately in CI.
|
|
26
|
+
|
|
27
|
+
A replay of saved responses restored two extracted pages. A copy of real Wiki and research data recovered two searchable pages and verified 49 citations; original files and immutable revisions remained unchanged. These are data and implementation checks, not a new-session model behavior evaluation or a guarantee that upstream websites are currently available.
|
|
28
|
+
|
|
29
|
+
Move or sync Wiki indexes and revisions together with the cited research state, inventory and evidence. Preserve their relative layout for Markdown links and configure paths on the destination. Retrieval remains literal text matching. Evidence archival and Wiki synthesis remain separate, following the existing `suggest`, `auto` and `off` settings.
|
|
30
|
+
|
|
31
|
+
Install from [npm](https://www.npmjs.com/package/bookmark-research/v/0.5.0-beta.5). GitHub supplies source, version notes and installation catalogs; npm remains the shared release package for all four hosts.
|
package/docs/wiki-quality.md
CHANGED
|
@@ -50,12 +50,14 @@ Cross-links have the form `{"page_id": "existing-page", "relation": "The host's
|
|
|
50
50
|
| `get(page_id, revision=None)` | Authored page, evidence snapshot, update record and current provenance validation |
|
|
51
51
|
| `list(offset=0, limit=20)` | Paginated current page metadata |
|
|
52
52
|
| `lint(page_id=None)` | Source review, active-claim, citation, hash, link and history issues |
|
|
53
|
-
| `search(query, offset=0, limit=10)` | Matching
|
|
53
|
+
| `search(query, offset=0, limit=10)` | Matching Wiki prose, revision dates and validation; stale pages excluded, `needs_review` warnings retained |
|
|
54
54
|
|
|
55
55
|
Lint detects later rejection, withdrawal, source/claim changes and altered or missing artifacts. Historical text is preserved and identified as stale. Lint explicitly returns `semantic_support: "not_scored"`.
|
|
56
56
|
|
|
57
57
|
Search checks titles, section headings and authored section text. It uses case-folded literal terms, including Chinese substrings, and simple occurrence ordering. Saving a provider extract alone does not make that extract searchable as organized Wiki knowledge. This is not embedding retrieval or semantic reranking; neither is selected without actual question-answering evaluation.
|
|
58
58
|
|
|
59
|
+
New records use research-task-relative evidence paths. `get` resolves verified references at current research locations, including legacy records containing absolute paths, without rewriting immutable revisions. Identity, claim, quote and artifact hashes remain mandatory. Move the Wiki and cited research state/inventory/evidence together; preserve their relative layout for Markdown links and configure local research paths at the destination. Wiki Markdown alone is not a complete portable knowledge base.
|
|
60
|
+
|
|
59
61
|
## Quality evaluation input
|
|
60
62
|
|
|
61
63
|
`evaluation.evaluate(suite, runs, judgments=None)` is a pure computation over supplied data. The complete runnable input is [research-quality.json](https://github.com/Browser-bookmark-hub/Bookmark-Research/blob/main/tests/fixtures/research-quality.json), also included at `tests/fixtures/research-quality.json` in the source checkout and verification pack.
|
package/docs/wiki-quality.zh.md
CHANGED
|
@@ -50,12 +50,14 @@ wiki.write("reusable-sessions", revised_page, "解释新增限制", expected_rev
|
|
|
50
50
|
| `get(page_id, revision=None)` | 正文、证据快照、更新记录与当前 provenance 核验 |
|
|
51
51
|
| `list(offset=0, limit=20)` | 当前页元数据的分页 |
|
|
52
52
|
| `lint(page_id=None)` | 来源审阅、有效 claim、引文、哈希、链接和历史问题 |
|
|
53
|
-
| `search(query, offset=0, limit=10)` |
|
|
53
|
+
| `search(query, offset=0, limit=10)` | 匹配 Wiki 正文、修订日期及核验状态;排除已失效页,保留 `needs_review` 警告 |
|
|
54
54
|
|
|
55
55
|
Lint 检测后续来源拒绝、结论撤回、来源/claim 变化和缺失/改动的产物。历史正文保留并标记过时,明确返回 `semantic_support:"not_scored"`。
|
|
56
56
|
|
|
57
57
|
检索匹配标题、章节标题和整理后的正文,采用忽略大小写的字面词匹配(包含中文子串)及简单出现次数排序。仅保存 provider 摘录不会使它成为可检索的 Wiki 知识。没有 embedding 检索或语义重排,也不会未经实际问答评测就选择它们。
|
|
58
58
|
|
|
59
|
+
新记录使用相对于研究任务的证据路径。`get` 将已验证引用解析到当前研究位置,兼容旧绝对路径记录,不改写不可变修订;身份、结论、引文与产物哈希仍须一致。迁移时同时保留 Wiki 和引用的研究 state/inventory/证据,保持相对布局以维持 Markdown 链接,并配置目标机器的本地研究路径。仅有 Wiki Markdown 不构成完整可迁移的知识库。
|
|
60
|
+
|
|
59
61
|
## 质量评测输入
|
|
60
62
|
|
|
61
63
|
`evaluation.evaluate(suite, runs, judgments=None)` 只计算传入数据。完整可运行输入为 [research-quality.json](https://github.com/Browser-bookmark-hub/Bookmark-Research/blob/main/tests/fixtures/research-quality.json),源码和验证包也包含 `tests/fixtures/research-quality.json`。
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "bookmark-research",
|
|
3
|
-
"version": "0.5.0-beta.
|
|
3
|
+
"version": "0.5.0-beta.5",
|
|
4
4
|
"description": "Check bookmark URLs or Canvas packages, compare sources with host-led research, and retain verifiable reports.",
|
|
5
5
|
"license": "GPL-3.0-only",
|
|
6
6
|
"repository": {"type": "git", "url": "git+https://github.com/Browser-bookmark-hub/Bookmark-Research.git"},
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: bookmark-research
|
|
3
|
-
description: "Analyze Bookmark Canvas (.canvas, section JSON, directories or ZIPs) as a personal database:
|
|
3
|
+
description: "Analyze Bookmark Canvas (.canvas, section JSON, directories or ZIPs) as a personal database: explain bookmarks and spatial context, research linked pages or web topics, and reuse saved research or reviewed Wiki knowledge. Preserves original packages."
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Bookmark Research
|
|
@@ -14,12 +14,15 @@ Use the user's links, package paths or registered sources. Keep original package
|
|
|
14
14
|
| User request | Start here | Done when |
|
|
15
15
|
| --- | --- | --- |
|
|
16
16
|
| Find/count bookmarks, inspect a card, explain its neighbors or group | Local Canvas workflow below; no web readiness check | The requested items or relationships are explained with IDs, scope and data provenance. |
|
|
17
|
-
|
|
|
18
|
-
|
|
|
17
|
+
| Recall earlier findings or personal knowledge missing from this conversation | Saved knowledge workflow below; no web readiness check | Relevant saved conclusions and their evidence are identified, or the retrieval gap is explicit. |
|
|
18
|
+
| Read a known URL or check a few facts | `quick`: readiness → an appropriate existing reader / `fetch_web` → assess returned text | Relevant page evidence answers the question; disclose a failed or incomplete read. |
|
|
19
|
+
| Compare options or investigate an open question | `agentic`: readiness → suitable documentation/search tools → assess → follow gaps | Evidence supports the requested comparison, or remaining gaps are explicit. |
|
|
19
20
|
| Investigate a whole list/package online, retain progress, or deliver a verifiable research report | `deep`: read [deep research](references/deep-research.md), then `research_start` | Evidence, original-input coverage and report checks pass, or an incomplete report retains the gaps. |
|
|
20
21
|
|
|
21
22
|
Honor explicit depth, then saved `research.depth`; the table guides `auto`. Scope and depth are separate: “explain this card” does not request web research, and “count the whole package” remains local. Context around a card does not authorize researching every neighboring bookmark. A capabilities question does not start an investigation. The current host model conducts research; professional APIs and subagents are optional.
|
|
22
23
|
|
|
24
|
+
Choose tools for information the question lacks. Reuse evidence already in the conversation when its scope and date still fit. Rewriting supplied text or continuing a supported explanation needs no retrieval. Current public facts can go directly to current sources; comparing an earlier decision with today's facts may need both saved and web evidence.
|
|
25
|
+
|
|
23
26
|
Read `get_settings` when saved preferences are unknown. Answer in the requested language, then saved `research.response_language`; `auto` follows the task language. Pass that language to saved briefs and delegates. Read references in one language only.
|
|
24
27
|
|
|
25
28
|
## Bind tools once
|
|
@@ -28,6 +31,14 @@ Tool names below are operation names. Resolve them to this plugin's actual, full
|
|
|
28
31
|
|
|
29
32
|
If MCP is unavailable or lacks an operation, use the bundled Python CLI. Resolve `<plugin-root>` as `../..` from this Skill's directory and pass its absolute path: `python3 <plugin-root>/src/cli.py --help`. Read the relevant [CLI commands](references/cli.md) only for that fallback. Pi may use this path without an MCP extension.
|
|
30
33
|
|
|
34
|
+
## Reuse saved knowledge when needed
|
|
35
|
+
|
|
36
|
+
Use Wiki when the question needs earlier research or personal knowledge absent from the current context. Topic overlap alone does not require a lookup. The host makes this choice during the task; there is no mandatory Wiki-first step or separate background agent.
|
|
37
|
+
|
|
38
|
+
For a known page ID, use `wiki_get` directly. Otherwise extract distinctive terms, such as `RAG MCP` or a Chinese topic phrase, for `wiki_search`: it matches literal terms, not a full natural-language question. Read relevant hits with `wiki_get`, checking the cited scope, source dates, bookmark references and `validation`. `needs_review` flags changed bookmark input; assess that change before relying on the old conclusion. A miss can be a vocabulary mismatch; reformulate when saved evidence is still needed.
|
|
39
|
+
|
|
40
|
+
Use `search_archive` for saved original text and `research_status` to locate a previous task. These read persistent local data, independently of host conversation memory. Repeat retrieval only for missing context, a changed scope/input or a freshness requirement. Read [Wiki and evaluation](references/wiki-and-evaluation.md) for provenance, revisions or a requested write.
|
|
41
|
+
|
|
31
42
|
## Read Canvas as a personal database
|
|
32
43
|
|
|
33
44
|
For Canvas input, read [package semantics](references/package-semantics.md) once. `.canvas` is the relationship map (nodes, geometry, groups, text and edges); section JSONs hold descriptions and bookmark trees. SQLite is a derived, queryable index of these files, not a replacement for their meaning. Use the existing MCP/CLI or read JSON locally; no graph service or extra dependency is needed.
|
|
@@ -44,9 +55,11 @@ Example: “What is around card X?” → overview → resolve X → scoped cont
|
|
|
44
55
|
|
|
45
56
|
Before a new web investigation, call `research_readiness` with the actual host, observed tools and any explicitly restricted providers. Reuse checks within that investigation; configured keys, catalog reachability, retrieval and host OAuth are different statuses. For a connection/auth failure, inspect the affected route. Use CLI `setup` and [settings guidance](references/settings-and-archive.md) when remediation is needed; never request keys in chat.
|
|
46
57
|
|
|
47
|
-
For known URLs,
|
|
58
|
+
For API/SDK or repository questions, prefer an appropriate documentation or file-reading tool already available in the host. For known URLs, use an existing reader or `fetch_web({urls:[...]})`; for discovery use `search_web({targets:[{target:"question",query:"public search terms"}],limit_per_target:5})`. A narrow lookup usually needs one suitable route; broaden for evidence gaps or requested comparisons. Use existing per-call provider options or saved defaults, honoring the user's provider selections. Optional service failures do not block other available routes.
|
|
59
|
+
|
|
60
|
+
Assess the material actually returned. Titles and search summaries guide discovery; source excerpts or code with an attributable URL may already answer a narrow question. Check identity, relevant passages, date/version and support; an excerpt does not establish a full-page read. Read `fetch_web.pages[].text` or its archived body when needed, and fetch more only for missing context, insufficient evidence or freshness. Stop when the requested answer is supported or explain the remaining limit. Cite the actual source URL.
|
|
48
61
|
|
|
49
|
-
|
|
62
|
+
`fetch_web` archives its responses according to saved settings. Host-native results are not intercepted: when evidence retention is needed, use `research_import_evidence` with the actual text and provenance in the relevant research task. Already saved text can be found with `search_archive`. Archiving and Wiki synthesis are separate; saving evidence does not require compiling every answer into a Wiki page.
|
|
50
63
|
|
|
51
64
|
Read [research methods](references/research-methods.md) for systematic comparisons or conflicts, [provider access](references/research-workflow.md) for retrieval diagnosis, and [GitHub boundaries](references/github-and-sync.md) for repository/private-access or synchronization tasks.
|
|
52
65
|
|
|
@@ -35,10 +35,12 @@ Use the task's output language for titles, section headings, authored prose and
|
|
|
35
35
|
|
|
36
36
|
Write target entity/topic pages before adding `links:[{page_id,relation}]`. For updates, read the current revision through `wiki_get` and use it as `expected_revision`. On conflict, reread and merge rather than overwriting another update. Historical revisions and their referenced target versions remain unchanged.
|
|
37
37
|
|
|
38
|
-
`wiki_get` returns page content, claim/source mappings, bookmark instances and inventory IDs. `wiki_lint` checks hashes, retractions/rejections, changed
|
|
38
|
+
`wiki_get` returns page content, claim/source mappings, bookmark instances and inventory IDs. `wiki_search` includes revision dates and the same `validation` status: `needs_review` signals changed bookmark input and remains discoverable with the warning; invalid/retracted evidence is excluded. `wiki_lint` checks hashes, retractions/rejections, changed input, links and revisions. `semantic_support:"not_scored"` means it is not a semantic judge; no status proves a webpage is still current.
|
|
39
39
|
|
|
40
40
|
`wiki_list` and `wiki_search` are paginated. Search matches literal terms in authored titles, headings and text, including Chinese. Saving page text does not automatically write a Wiki. There are no embeddings, vector retrieval, reranking or monitoring. Default storage is the data directory's `wiki/`, overridable with `settings.wiki.directory`; keep it outside the original package.
|
|
41
41
|
|
|
42
|
+
New revision records store evidence paths relative to their research task. `wiki_get` resolves verified evidence at current task locations; legacy absolute-path records remain readable without rewriting history. Move/sync the Wiki index and revisions together with the cited research state, inventory and evidence, keeping their relative layout for Markdown links. Configure the destination's local data/research paths; copying only Wiki Markdown does not preserve a verifiable knowledge base. Git synchronization belongs to the host and stays outside the canvas-managed directory.
|
|
43
|
+
|
|
42
44
|
Each write regenerates `index.md` (current pages by kind: link, title, one-line summary, revision, date, claim count) and appends `log.md` (`## [date] write | title (rN)` with page ID, change note and research IDs). Read or grep them to orient; `index.json` and revision files remain authoritative, and `wiki_lint` warns `catalog_stale`/`catalog_missing` until the next write regenerates `index.md`.
|
|
43
45
|
|
|
44
46
|
After research, `research_finish.wiki_follow_up` applies `settings.wiki.after_research`: `suggest` (default) → show `candidates` and eligible claims and ask before writing; `auto` → update a matching page (read it for `expected_revision`) or create one using only `eligible_claim_ids`, then report the changes; `off` → nothing unless requested.
|
|
@@ -9,12 +9,15 @@
|
|
|
9
9
|
| 用户请求 | 从哪里开始 | 完成条件 |
|
|
10
10
|
| --- | --- | --- |
|
|
11
11
|
| 找/统计书签,查看卡片、邻居或分组 | 下方本地画布流程;无需联网检查 | 用 ID、范围和数据出处说明目标内容或关系 |
|
|
12
|
-
|
|
|
13
|
-
|
|
|
12
|
+
| 回顾当前对话缺少的旧研究或个人知识 | 下方已有知识流程;无需联网检查 | 找到相关结论和证据,或明确检索缺口 |
|
|
13
|
+
| 读取已知 URL,核实少量事实 | `quick`:检查可用性 → 合适的现有读取工具/`fetch_web` → 判断返回内容 | 正文证据回答了问题;失败或不完整时说明 |
|
|
14
|
+
| 比较选项,调查开放问题 | `agentic`:检查可用性 → 合适的文档/搜索工具 → 判断 → 补缺口 | 证据支持所需比较,或明确剩余缺口 |
|
|
14
15
|
| 联网研究整份列表/数据包,保留进度,交付可核查报告 | `deep`:先读[深度研究](deep-research.md),再 `research_start` | 证据、原输入覆盖和报告检查通过,或保留缺口并交付未完成报告 |
|
|
15
16
|
|
|
16
17
|
优先遵循本次明确深度,其次是保存的 `research.depth`;表格用于 `auto`。范围与深度分开:“解释这张卡片”不要求联网,“统计整个包”仍是本地任务。卡片周围的上下文不代表用户要求研究所有邻居中的书签。询问能力不自动启动调查。研究由当前宿主模型执行,专业 API 与子代理均为可选能力。
|
|
17
18
|
|
|
19
|
+
按问题缺少的信息选择工具。对话中已有证据的范围和日期仍适用时直接复用;改写用户给出的文字或继续已有依据的解释无需检索。当前公开事实可直接查当前来源;对比旧决定和最新情况可能同时需要本地与网络证据。
|
|
20
|
+
|
|
18
21
|
不知道已保存偏好时读取 `get_settings`。答复语言依次采用本次要求、`research.response_language`;`auto` 跟随任务语言。保存的研究 brief 和委派任务也使用该语言。参考文档只读一种语言。
|
|
19
22
|
|
|
20
23
|
## 先对应实际工具
|
|
@@ -23,6 +26,14 @@
|
|
|
23
26
|
|
|
24
27
|
MCP 不可用或缺少操作时使用内置 Python CLI。从本 Skill 的目录向上两级得到 `<plugin-root>`,使用绝对路径执行 `python3 <plugin-root>/src/cli.py --help`,只查阅所需的 [CLI 命令](cli.md)。Pi 无 MCP 扩展时也可使用这条路径。
|
|
25
28
|
|
|
29
|
+
## 需要时复用已有知识
|
|
30
|
+
|
|
31
|
+
问题依赖旧研究或个人知识,且当前上下文缺少这些依据时使用 Wiki。仅仅主题相同不要求查询。宿主在处理任务时做出判断,没有每轮必走的 Wiki 前置步骤,也无需另一个后台 agent。
|
|
32
|
+
|
|
33
|
+
已知 page ID 时直接 `wiki_get`,否则提取 `RAG MCP` 或中文主题短语等有区分度的词调用 `wiki_search`;它采用字面词匹配,不适合直接传整句自然语言问题。对相关命中用 `wiki_get` 读取,检查引用范围、来源日期、书签引用与 `validation`。`needs_review` 表示书签输入发生变化,使用旧结论前判断变化的影响。未命中也可能是用词不同;确实需要旧证据时再调整关键词。
|
|
34
|
+
|
|
35
|
+
保存的原文用 `search_archive`,旧研究任务用 `research_status` 定位。它们读取持久的本地数据,不依赖宿主的对话记忆。只有上下文缺失、范围/输入变化或时效要求才重新检索。出处、修订或用户要求写入时查阅 [Wiki 与评估](wiki-and-evaluation.md)。
|
|
36
|
+
|
|
26
37
|
## 把画布作为个人数据库读取
|
|
27
38
|
|
|
28
39
|
处理画布时先读一次[数据包语义](package-semantics.md)。`.canvas` 是关系图:节点、几何、分组、文本和边;栏目 JSON 保存说明和书签树。SQLite 是这些文件的派生查询索引,不代替原始语义。使用现有 MCP/CLI 或本地 JSON 读取即可,无需额外图服务或依赖。
|
|
@@ -39,9 +50,11 @@ MCP 不可用或缺少操作时使用内置 Python CLI。从本 Skill 的目录
|
|
|
39
50
|
|
|
40
51
|
每个新的联网调查先调用 `research_readiness`,提供实际宿主、已观察到的工具以及明确限定的服务。同一调查复用检查;密钥已配置、目录可达、检索成功与宿主 OAuth 是不同状态。连接/认证失败时检查受影响路径;需要修复时使用 CLI `setup` 和[配置指引](settings-and-archive.md),不在聊天里索取密钥。
|
|
41
52
|
|
|
42
|
-
|
|
53
|
+
API/SDK 或仓库问题优先使用宿主已有且合适的文档/文件读取工具。已知 URL 用现有读取器或 `fetch_web({urls:[...]})`,发现来源用 `search_web({targets:[{target:"question",query:"public search terms"}],limit_per_target:5})`。范围明确的小查询通常只需一条合适路径,有证据缺口或用户要求比较时再扩展。通过现有单次 provider 参数或保存的默认值选择服务,遵守用户的服务选择。可选服务故障不阻塞其他可用路径。
|
|
54
|
+
|
|
55
|
+
判断实际返回的材料:标题和搜索摘要用于发现;带有可归属 URL 的原文摘录或代码,可能已经回答具体问题。核对身份、相关段落、日期/版本与支持程度,摘录不能证明读完整页。需要时读取 `fetch_web.pages[].text` 或归档正文,仅为缺失上下文、证据不足或时效要求补抓。证据已支持请求即可回答,否则说明剩余限制。引用实际来源 URL。
|
|
43
56
|
|
|
44
|
-
|
|
57
|
+
`fetch_web` 按保存设置归档它收到的响应。宿主原生工具的结果不会被自动截获;需要保留证据时,在相应研究任务中用 `research_import_evidence` 导入实际文字和 provenance。已存原文可用 `search_archive` 查找。归档与 Wiki 整理分别处理,保存证据不要求把每次回答都编成 Wiki 页面。
|
|
45
58
|
|
|
46
59
|
系统比较或冲突核对查阅[研究方法](research-methods.md),检索排障查阅[服务访问](research-workflow.md),仓库/私有访问或同步任务查阅 [GitHub 边界](github-and-sync.md)。
|
|
47
60
|
|
|
@@ -35,10 +35,12 @@ kind 为 topic 或 entity。每节必须有有效 claim;每页至多 24 个不
|
|
|
35
35
|
|
|
36
36
|
先写被引用的实体/主题页,再用 `links:[{page_id,relation}]` 建立关系。更新先 `wiki_get` 读当前 revision,将它作为 `expected_revision` 写入;冲突时重新读取合并,不覆盖他人更新。历史修订和当时引用目标版本保持不变。
|
|
37
37
|
|
|
38
|
-
`wiki_get` 返回正文、结论与来源映射、书签实例和 inventory ID。`wiki_lint`
|
|
38
|
+
`wiki_get` 返回正文、结论与来源映射、书签实例和 inventory ID。`wiki_search` 返回修订日期及相同的 `validation` 状态:书签输入变化产生 `needs_review`,仍可检索并带警告;证据失效或被撤回的页面被排除。`wiki_lint` 检查哈希、撤回/拒绝、输入变化、链接和版本;`semantic_support:"not_scored"` 表示它不是语义裁判,任何状态都不证明网页目前仍然有效。
|
|
39
39
|
|
|
40
40
|
`wiki_list` 和 `wiki_search` 分页返回;搜索对象是整理后的标题、章节和正文,采用字面词匹配,可查询中文。保存一份网页正文不会自动写 Wiki;当前不提供 embedding、向量检索、重排或监控。Wiki 默认数据目录 `wiki/`,可通过 `settings.wiki.directory` 改路径,不能放回原包。
|
|
41
41
|
|
|
42
|
+
新修订记录使用相对于研究任务的证据路径;`wiki_get` 将已验证的证据解析到当前任务位置,旧绝对路径记录也可读取,不改写历史。迁移/同步时同时保留 Wiki 索引、修订及其引用的研究 state、inventory 和证据,并保留相对布局以维持 Markdown 链接。在目标机器配置本地数据/研究路径;仅复制 Wiki Markdown 不足以保留可验证的知识库。Git 同步交给宿主,存放位置避开画布管理目录。
|
|
43
|
+
|
|
42
44
|
每次写入会重新生成 `index.md`(按 kind 列出当前页面:链接、标题、一行摘要、版本、日期、结论数),并向 `log.md` 追加 `## [日期] write | 标题 (rN)` 及 page ID、修改说明和研究 ID。可阅读或 grep 它们了解全貌;`index.json` 与版本文件仍是准确数据,`index.md` 过期或缺失时 `wiki_lint` 给出 `catalog_stale`/`catalog_missing` 警告,下次写入即重新生成。
|
|
43
45
|
|
|
44
46
|
研究结束后,`research_finish.wiki_follow_up` 按 `settings.wiki.after_research` 处理:`suggest`(默认)→ 展示 `candidates` 与可用结论,征得同意再写;`auto` → 更新匹配页面(先读取以取得 `expected_revision`)或新建页面,只引用 `eligible_claim_ids`,再说明改动;`off` → 用户未要求则不处理。
|
package/src/archive.py
CHANGED
|
@@ -151,6 +151,24 @@ class SourceArchive:
|
|
|
151
151
|
rows.extend(cls._text_rows(text))
|
|
152
152
|
return rows
|
|
153
153
|
|
|
154
|
+
@staticmethod
|
|
155
|
+
def _boundary_text(text):
|
|
156
|
+
"""Mask fenced examples without shifting offsets into the original text."""
|
|
157
|
+
lines, fence = [], None
|
|
158
|
+
for line in text.splitlines(keepends=True):
|
|
159
|
+
marker = re.match(r" {0,3}(`{3,}|~{3,})([^\r\n]*)", line)
|
|
160
|
+
masked = fence is not None
|
|
161
|
+
if fence is not None:
|
|
162
|
+
if (marker and marker[1][0] == fence[0] and len(marker[1]) >= fence[1]
|
|
163
|
+
and not marker[2].strip()):
|
|
164
|
+
fence = None
|
|
165
|
+
elif marker and (marker[1][0] == "~" or "`" not in marker[2]):
|
|
166
|
+
fence = (marker[1][0], len(marker[1]))
|
|
167
|
+
masked = True
|
|
168
|
+
lines.append(re.sub(r"[^\r\n]", " ", line) if masked else line)
|
|
169
|
+
# A truncated example could contain what looks like the next page header.
|
|
170
|
+
return None if fence is not None else "".join(lines)
|
|
171
|
+
|
|
154
172
|
@classmethod
|
|
155
173
|
def _text_rows(cls, text):
|
|
156
174
|
rows = []
|
|
@@ -168,11 +186,14 @@ class SourceArchive:
|
|
|
168
186
|
# cannot be silently attributed to the preceding requested URL.
|
|
169
187
|
# Titles may retain indented continuation lines from the source HTML.
|
|
170
188
|
# Records must start the block or follow the provider's blank separator.
|
|
189
|
+
boundary_text = cls._boundary_text(text)
|
|
190
|
+
if boundary_text is None:
|
|
191
|
+
return rows
|
|
171
192
|
header = r"(?:\A|\n\n)(?:# |Title: )([^\n]+(?:\n[ \t][^\n]*)*)\nURL: (https?://[^\s]+)\n"
|
|
172
|
-
matches = list(re.finditer(header,
|
|
193
|
+
matches = list(re.finditer(header, boundary_text))
|
|
173
194
|
# A missed or malformed URL field could hide the next page boundary.
|
|
174
195
|
# Keep the whole block raw instead of assigning that page to its neighbor.
|
|
175
|
-
fields = [match.start() for match in re.finditer(r"(?m)^[ \t]*URL:",
|
|
196
|
+
fields = [match.start() for match in re.finditer(r"(?m)^[ \t]*URL:", boundary_text)]
|
|
176
197
|
matched_fields = [match.start(2) - len("URL: ") for match in matches]
|
|
177
198
|
if fields != matched_fields:
|
|
178
199
|
return rows
|
|
@@ -185,7 +206,7 @@ class SourceArchive:
|
|
|
185
206
|
for index, match in enumerate(matches):
|
|
186
207
|
end = matches[index + 1].start() if index + 1 < len(matches) else len(text)
|
|
187
208
|
body = text[match.end():end]
|
|
188
|
-
row = {"url": match.group(2), "title": match.
|
|
209
|
+
row = {"url": match.group(2), "title": text[match.start(1):match.end(1)]}
|
|
189
210
|
metadata = re.match(r"(?:(?:Published|Author): [^\n]*\n)*\n", body)
|
|
190
211
|
if metadata:
|
|
191
212
|
for line in metadata.group().splitlines():
|
package/src/mcp_server.py
CHANGED
|
@@ -296,9 +296,9 @@ TOOL_DESCRIPTIONS.update({
|
|
|
296
296
|
"research_service_attach": "Attach an independently known provider run ID without starting remote work. Can recover a missing create response or reference a run started through an existing host integration. Status remains unverified until observed.",
|
|
297
297
|
"research_service_import": "Import a saved completed provider report into a research session as unreviewed external-report evidence. Original URL coverage is unchanged; read and assess cited originals separately.",
|
|
298
298
|
"wiki_write": "Publish an authored topic/entity page outside source packages, with reviewed active claims, source provenance, relationships and an immutable revision. Supply the current expected_revision when updating. Semantic support requires the named human/model review.",
|
|
299
|
-
"wiki_get": "Read a Wiki page or an immutable earlier revision with source/claim links
|
|
299
|
+
"wiki_get": "Read a Wiki page or an immutable earlier revision with source/claim links resolved at current research locations. validation.status=needs_review signals changed bookmark input after the cited research; this does not automatically revise the page or refetch webpages.",
|
|
300
300
|
"wiki_list": "List paginated Wiki pages and current revision metadata without network access.",
|
|
301
|
-
"wiki_search": "
|
|
301
|
+
"wiki_search": "Find saved knowledge when the task needs prior research absent from the conversation. Return matching Wiki text, revision dates and validation; needs_review flags changed bookmark input. Invalidated or retracted evidence is excluded. Literal full-text retrieval, not vector search.",
|
|
302
302
|
"wiki_lint": "Check Wiki evidence hashes, active claims, source review states and cross-page links. Reports semantic review declarations separately from mechanical integrity checks.",
|
|
303
303
|
"evaluate_research": "Evaluate explicit benchmark outputs and reviewer labels against a supplied rubric. Computes coverage, labeled answer precision/recall/F1, semantic citation-label rates, report scores and measured cost/latency/variance. Missing judgments remain unknown; this tool runs no models or providers.",
|
|
304
304
|
})
|
package/src/wiki.py
CHANGED
|
@@ -24,6 +24,7 @@ class WikiStore:
|
|
|
24
24
|
"""Publish immutable revisions, then atomically advance the current index."""
|
|
25
25
|
|
|
26
26
|
MAX_RECORD_BYTES = 900000
|
|
27
|
+
ARTIFACT_FIELDS = ("body_file", "manifest_file", "response_file")
|
|
27
28
|
REVIEW_RUBRIC = (
|
|
28
29
|
"Check that each section is supported by its cited claims, preserves scope and uncertainty, "
|
|
29
30
|
"identifies the correct entities, and does not conceal contrary evidence. "
|
|
@@ -146,8 +147,7 @@ class WikiStore:
|
|
|
146
147
|
raise ValueError("Wiki requires an accepted source review: " + source["id"])
|
|
147
148
|
if citation["source_sha256"] != source["sha256"] or citation["quote"] not in self.research._body(path, source):
|
|
148
149
|
raise ValueError("Wiki citation no longer matches saved evidence")
|
|
149
|
-
artifacts = {key: self.research._artifact(path, source[key])
|
|
150
|
-
for key in ("body_file", "manifest_file", "response_file")}
|
|
150
|
+
artifacts = {key: self.research._artifact(path, source[key]) for key in self.ARTIFACT_FIELDS}
|
|
151
151
|
artifact_hashes = {key: hashlib.sha256(value.read_bytes()).hexdigest() for key, value in artifacts.items()}
|
|
152
152
|
citations.append({
|
|
153
153
|
"source_id": source["id"], "inventory_ids": source.get("inventory_ids", []),
|
|
@@ -156,28 +156,37 @@ class WikiStore:
|
|
|
156
156
|
"provider": source["provider"], "retrieved_at": source["retrieved_at"],
|
|
157
157
|
"review": copy.deepcopy(source["review"]),
|
|
158
158
|
"content_kind": source["content_kind"], "provenance": copy.deepcopy(source.get("provenance")),
|
|
159
|
-
"artifact_sha256": artifact_hashes,
|
|
159
|
+
"artifact_sha256": artifact_hashes,
|
|
160
|
+
**{key: value.relative_to(path).as_posix() for key, value in artifacts.items()}
|
|
160
161
|
})
|
|
161
162
|
if not citations:
|
|
162
163
|
raise ValueError("Wiki claims require saved citations")
|
|
163
164
|
return {"research_id": research_id, "claim_id": claim_id, "statement": claim["statement"],
|
|
164
165
|
"confidence": claim["confidence"], "inference": claim["inference"], "citations": citations}
|
|
165
166
|
|
|
166
|
-
def _issues(self, record, index, cache=None):
|
|
167
|
+
def _issues(self, record, index, cache=None, resolve_paths=False):
|
|
167
168
|
cache = {} if cache is None else cache
|
|
168
169
|
issues = []
|
|
169
170
|
for saved in record["evidence"]:
|
|
170
171
|
try:
|
|
171
172
|
current = self._evidence(saved["research_id"], saved["claim_id"], cache)
|
|
172
|
-
#
|
|
173
|
-
#
|
|
173
|
+
# Locations and review explanations may change. Stable IDs,
|
|
174
|
+
# accepted verdicts, quotes, content and artifact hashes must hold.
|
|
174
175
|
def substantive(value):
|
|
175
176
|
value = copy.deepcopy(value)
|
|
176
177
|
for citation in value["citations"]:
|
|
177
|
-
|
|
178
|
+
for key in ("review", *self.ARTIFACT_FIELDS):
|
|
179
|
+
citation.pop(key, None)
|
|
178
180
|
return value
|
|
179
181
|
if substantive(current) != substantive(saved):
|
|
180
182
|
raise ValueError("Claim or source mapping changed after Wiki publication")
|
|
183
|
+
if resolve_paths:
|
|
184
|
+
# Resolve only verified evidence for this returned view;
|
|
185
|
+
# immutable records (including legacy absolute paths) stay intact.
|
|
186
|
+
path = cache[saved["research_id"]][0]
|
|
187
|
+
for previous, now in zip(saved["citations"], current["citations"]):
|
|
188
|
+
for key in self.ARTIFACT_FIELDS:
|
|
189
|
+
previous[key] = str(self.research._artifact(path, now[key]))
|
|
181
190
|
except (OSError, ValueError, KeyError, TypeError) as error:
|
|
182
191
|
issues.append({"severity": "error", "code": "invalid_evidence", "research_id": saved["research_id"],
|
|
183
192
|
"claim_id": saved["claim_id"], "message": str(error)[:1000]})
|
|
@@ -322,6 +331,12 @@ class WikiStore:
|
|
|
322
331
|
def _artifacts(self, pointer):
|
|
323
332
|
return {key: str(self._path(pointer[key])) for key in ("record_file", "markdown_file")}
|
|
324
333
|
|
|
334
|
+
@staticmethod
|
|
335
|
+
def _validation(issues):
|
|
336
|
+
return {"status": "stale" if any(item["severity"] == "error" for item in issues) else
|
|
337
|
+
"needs_review" if any(item["code"] == "source_input_changed" for item in issues) else "current",
|
|
338
|
+
"issues": issues, "semantic_review": "caller_declared; not evaluated by lint"}
|
|
339
|
+
|
|
325
340
|
def get(self, page_id, revision=None):
|
|
326
341
|
page_id = self._slug(page_id)
|
|
327
342
|
if revision is not None:
|
|
@@ -333,11 +348,8 @@ class WikiStore:
|
|
|
333
348
|
if record["page_id"] != page_id:
|
|
334
349
|
raise ValueError("Wiki revision belongs to a different page")
|
|
335
350
|
if revision is None or record["revision"] == revision:
|
|
336
|
-
issues = self._issues(record, index)
|
|
337
|
-
return {**record, "artifacts": self._artifacts(pointer), "validation":
|
|
338
|
-
"status": "stale" if any(item["severity"] == "error" for item in issues) else
|
|
339
|
-
"needs_review" if any(item["code"] == "source_input_changed" for item in issues) else "current",
|
|
340
|
-
"issues": issues, "semantic_review": "caller_declared; not evaluated by lint"}}
|
|
351
|
+
issues = self._issues(record, index, resolve_paths=True)
|
|
352
|
+
return {**record, "artifacts": self._artifacts(pointer), "validation": self._validation(issues)}
|
|
341
353
|
if record["previous"] is not None and record["previous"]["revision"] >= record["revision"]:
|
|
342
354
|
raise ValueError("Invalid Wiki revision history")
|
|
343
355
|
pointer = record["previous"]
|
|
@@ -385,7 +397,8 @@ class WikiStore:
|
|
|
385
397
|
for page_id, pointer in index["pages"].items():
|
|
386
398
|
try:
|
|
387
399
|
record = self._revision(pointer)
|
|
388
|
-
|
|
400
|
+
validation = self._validation(self._issues(record, index, cache))
|
|
401
|
+
if validation["status"] == "stale":
|
|
389
402
|
excluded.append(page_id)
|
|
390
403
|
continue
|
|
391
404
|
page = record["page"]
|
|
@@ -398,6 +411,7 @@ class WikiStore:
|
|
|
398
411
|
begin = max(0, min(positions, default=0) - 60)
|
|
399
412
|
matches.append((rank, {"page_id": page_id, "revision": pointer["revision"], "title": page["title"],
|
|
400
413
|
"kind": page["kind"], "snippet": body[begin:begin + 500],
|
|
414
|
+
"updated_at": pointer["updated_at"], "validation": validation,
|
|
401
415
|
"artifacts": self._artifacts(pointer)}))
|
|
402
416
|
except (OSError, ValueError, KeyError, TypeError):
|
|
403
417
|
excluded.append(page_id)
|
|
@@ -426,16 +440,18 @@ class WikiStore:
|
|
|
426
440
|
lines.append("")
|
|
427
441
|
lines.extend(["## Evidence", ""])
|
|
428
442
|
for i, evidence in enumerate(record["evidence"], 1):
|
|
443
|
+
root = self.research._path(evidence["research_id"])
|
|
429
444
|
lines.extend(["### Evidence %s" % i, "", "%s / %s: %s" %
|
|
430
445
|
(evidence["research_id"], evidence["claim_id"], evidence["statement"]), ""])
|
|
431
446
|
for citation in evidence["citations"]:
|
|
447
|
+
artifacts = {key: link(self.research._artifact(root, citation[key])) for key in self.ARTIFACT_FIELDS}
|
|
432
448
|
lines.extend(["Source %s · %s · Retrieved %s" % (citation["source_id"], citation["url"], citation["retrieved_at"]),
|
|
433
449
|
"", "Inventory IDs: " + (", ".join(citation["inventory_ids"]) or "Not associated with a frozen inventory"), ""])
|
|
434
450
|
if (citation["provenance"] or {}).get("kind") == "external_report":
|
|
435
451
|
lines.extend(["Secondary evidence: an external report. Pages cited by that report have not been read by importing it.", ""])
|
|
436
452
|
lines.extend("> " + line for line in citation["quote"].splitlines())
|
|
437
453
|
lines.extend(["", "[Saved text](%s) · [Manifest](%s) · [Response](%s) · SHA-256: `%s`" %
|
|
438
|
-
(
|
|
454
|
+
(artifacts["body_file"], artifacts["manifest_file"], artifacts["response_file"],
|
|
439
455
|
citation["source_sha256"]), ""])
|
|
440
456
|
lines.extend(["## Update record", "", record["change_note"], ""])
|
|
441
457
|
if record["previous"]:
|