dsh-kb 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (49) hide show
  1. dsh_kb-0.1.0/CHANGELOG.md +115 -0
  2. dsh_kb-0.1.0/CONTRIBUTING.md +81 -0
  3. dsh_kb-0.1.0/LICENSE +21 -0
  4. dsh_kb-0.1.0/MANIFEST.in +13 -0
  5. dsh_kb-0.1.0/PKG-INFO +375 -0
  6. dsh_kb-0.1.0/README.md +348 -0
  7. dsh_kb-0.1.0/README.zh.md +343 -0
  8. dsh_kb-0.1.0/SECURITY.md +62 -0
  9. dsh_kb-0.1.0/docs/agent-integration.md +178 -0
  10. dsh_kb-0.1.0/docs/deployment.md +287 -0
  11. dsh_kb-0.1.0/docs/design.md +297 -0
  12. dsh_kb-0.1.0/docs/release.md +113 -0
  13. dsh_kb-0.1.0/examples/config.json +15 -0
  14. dsh_kb-0.1.0/examples/demo.sh +79 -0
  15. dsh_kb-0.1.0/pyproject.toml +48 -0
  16. dsh_kb-0.1.0/setup.cfg +4 -0
  17. dsh_kb-0.1.0/src/dsh_kb/__init__.py +16 -0
  18. dsh_kb-0.1.0/src/dsh_kb/__main__.py +6 -0
  19. dsh_kb-0.1.0/src/dsh_kb/cli.py +1513 -0
  20. dsh_kb-0.1.0/src/dsh_kb/config.py +399 -0
  21. dsh_kb-0.1.0/src/dsh_kb/index.py +710 -0
  22. dsh_kb-0.1.0/src/dsh_kb/mcp_server.py +325 -0
  23. dsh_kb-0.1.0/src/dsh_kb/mirror.py +468 -0
  24. dsh_kb-0.1.0/src/dsh_kb/mounts.py +189 -0
  25. dsh_kb-0.1.0/src/dsh_kb/py.typed +0 -0
  26. dsh_kb-0.1.0/src/dsh_kb/ragcheck.py +768 -0
  27. dsh_kb-0.1.0/src/dsh_kb/service.py +525 -0
  28. dsh_kb-0.1.0/src/dsh_kb/skills.py +168 -0
  29. dsh_kb-0.1.0/src/dsh_kb/sources.py +582 -0
  30. dsh_kb-0.1.0/src/dsh_kb/supersede.py +371 -0
  31. dsh_kb-0.1.0/src/dsh_kb/text.py +267 -0
  32. dsh_kb-0.1.0/src/dsh_kb/webdav.py +228 -0
  33. dsh_kb-0.1.0/src/dsh_kb.egg-info/PKG-INFO +375 -0
  34. dsh_kb-0.1.0/src/dsh_kb.egg-info/SOURCES.txt +47 -0
  35. dsh_kb-0.1.0/src/dsh_kb.egg-info/dependency_links.txt +1 -0
  36. dsh_kb-0.1.0/src/dsh_kb.egg-info/entry_points.txt +2 -0
  37. dsh_kb-0.1.0/src/dsh_kb.egg-info/top_level.txt +1 -0
  38. dsh_kb-0.1.0/tests/__init__.py +1 -0
  39. dsh_kb-0.1.0/tests/test_cli.py +894 -0
  40. dsh_kb-0.1.0/tests/test_e2e.py +785 -0
  41. dsh_kb-0.1.0/tests/test_index_cas.py +574 -0
  42. dsh_kb-0.1.0/tests/test_index_zone.py +326 -0
  43. dsh_kb-0.1.0/tests/test_kbask_zone.py +122 -0
  44. dsh_kb-0.1.0/tests/test_mcp.py +290 -0
  45. dsh_kb-0.1.0/tests/test_ragcheck.py +1042 -0
  46. dsh_kb-0.1.0/tests/test_skills.py +121 -0
  47. dsh_kb-0.1.0/tests/test_sources.py +539 -0
  48. dsh_kb-0.1.0/tests/test_supersede.py +350 -0
  49. dsh_kb-0.1.0/tests/test_units.py +331 -0
@@ -0,0 +1,115 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project are documented here.
4
+ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
5
+ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
6
+
7
+ ## [0.1.0] — 2026-10-25
8
+
9
+ First public version. Written against a real deployment: a Markdown bundle on
10
+ a network share, mirrored to a Linux host and searched by an AI agent.
11
+
12
+ ### Added
13
+
14
+ - **Bundle layout** with privacy zones (`z0-public`, `z1-restricted`,
15
+ `z2-sensitive`), a layout area (`_meta`, `_incoming`) and a root index.
16
+ - **Local mirror** with a manifest-driven incremental sync: a file whose size
17
+ and mtime are unchanged is never transferred. Opt-in `--prune`.
18
+ - **Local full-text index** on SQLite FTS5: CJK bigram expansion plus the
19
+ original text, a contentless FTS table with a separate `meta` table, and a
20
+ `fts5vocab` table used for selectivity.
21
+ - **Two-stage search**: strict `AND` for precision, relaxing to a selective
22
+ `OR` only when the strict pass is empty.
23
+ - **Zone policy compiled into code**, enforced at three points: mirroring,
24
+ indexing and querying. Sensitive zones are not even copied to disk.
25
+ - **Read sources**: a filesystem view (local directory, Samba/CIFS or NFS
26
+ mount, Windows drive or UNC path) and a dependency-free generic WebDAV client
27
+ (optional Basic auth and mutual TLS, with connect-host/SNI separation for
28
+ appliances whose certificate CN differs from their address).
29
+ - **Unmounted SMB** (`source=smb`) through `smbclient`, for hosts where mounting
30
+ needs root or a mount has gone stale. Parses the real `recurse ON; ls` output
31
+ (verified against a live server, including names with spaces and the trailing
32
+ `blocks of size …` line).
33
+ - **Network-mount awareness**: `/proc/mounts` inspection picks the real
34
+ filesystem rather than its `autofs` trigger, reports `rw`/`ro`, and turns
35
+ `ESTALE` / `EACCES` / `ENOENT` into actionable sentences. `sync --full` covers
36
+ the one-second timestamp granularity of network shares.
37
+ - **CLI** (`dsh-kb status|doctor|sync|index|refresh|search|read|push|mcp`) with
38
+ `--json` output, offline operation for everything except `sync`/`push`/
39
+ `doctor`, best-effort auto-refresh before a search, and a three-level
40
+ `doctor` report (`ok` / `~~` warning / `!!` failure).
41
+ - **MCP server** (`dsh-kb mcp`): newline-delimited JSON-RPC 2.0 on stdio,
42
+ exposing `search` / `read` / `status`. Hand-rolled on the standard library to
43
+ keep the zero-dependency promise; interoperability is asserted against the
44
+ official Python SDK in the test-suite rather than assumed. Tool schemas
45
+ measure ~1543 characters (~385 tokens) resident, with a test guarding that
46
+ budget.
47
+ - **Use-case layer** (`dsh_kb.service`) shared by the CLI and the MCP server, so
48
+ `--json` output and tool results cannot drift apart.
49
+ - **86→122 tests**: unit, end-to-end over a fixture bundle, transport tests with
50
+ a stub `smbclient`, CLI smoke tests, MCP protocol/process tests, and an
51
+ SDK interop test; no network needed.
52
+
53
+ ### Fixed
54
+
55
+ - **`--home` did not relocate `config.json`.** The file was loaded from the
56
+ environment-derived home *before* overrides were applied, so
57
+ `dsh-kb --home X` silently ran on the built-in defaults — even though the
58
+ CLI's own help calls `--home` the "state directory (mirror, index, manifest,
59
+ logs, config)". Measured on a real deployment: three `never` path exclusions
60
+ and every zone's `inject` flag were ignored, and the exclusions are a privacy
61
+ guard. `home` is now resolved before the file is read; covered by regression
62
+ tests that fail without the fix.
63
+
64
+ - **`WebdavSource.write` did not create parent collections.** A bare `PUT` into
65
+ a missing collection is just `HTTP 500 errno 2 No such file or directory`, and
66
+ because `MKCOL` does not recurse, every level must be created in turn. Found
67
+ the honest way: a 217-file import into a freshly laid-out bundle failed 217
68
+ times. Collections already handled are remembered per instance, so a bulk
69
+ import does not repeat them.
70
+
71
+ - **Token and resource thrift** (the base is meant to be a supplement consulted on
72
+ demand, not standing context): tool schemas trimmed 1543 → 1289 characters
73
+ (~385 → ~322 tokens), the `instructions` line rewritten from "search before
74
+ reading files" (which invited a lookup every turn) to an explicit on-demand
75
+ stance, all zones set to `inject: false`, and an empty refresh **no longer
76
+ rebuilds the index** — the scheduler's four daily runs and the pre-search
77
+ auto-refresh are now a directory walk when nothing changed.
78
+ - **Unknown `config.json` keys are ignored and reported** instead of crashing the
79
+ CLI with a `TypeError`; `doctor` lists them as a warning. Found the honest way:
80
+ a stray note key took the whole command down.
81
+
82
+ ### Known limitations
83
+
84
+ - Search is lexical only. A paraphrase with no shared wording returns nothing;
85
+ a semantic fallback is planned but not implemented.
86
+ - Relaxation to a broader `OR` match is deliberately *not* applied to a single
87
+ latin word or identifier — `UNIQ-…` is matched exactly or not at all.
88
+ - No write path for the model: writes go through `push` into `_incoming/` and
89
+ are promoted to a zone by a human or a scheduled job.
90
+ - The WebDAV client speaks the subset of RFC 4918 this project needs
91
+ (`PROPFIND`, `GET`, `PUT`, `MKCOL`, `DELETE`); no locking or `COPY`/`MOVE`.
92
+ - `source=smb` costs one process and one round trip per file; it is a fallback,
93
+ not a replacement for a mount.
94
+ - The MCP server implements the three methods an agent needs; prompts,
95
+ resources, sampling and subscriptions are not advertised.
96
+
97
+ ### Added before the first public release
98
+
99
+ - **`skills` subcommand and MCP tool** for a curated skill library kept in a
100
+ `z3-skills` asset zone. Such a zone has to be registered in `config.json`
101
+ (`index: false`) *before* anything is moved into it: `index_allows()` lets
102
+ **unregistered top-level directories through**, so moving files alone does not
103
+ keep them out of the index — and 2.3k `SKILL.md` files would otherwise drown
104
+ knowledge-page retrieval. Both entry points forward to the library's own
105
+ `bin/find.py`, keeping tokenisation and scoring in a single implementation, and
106
+ the library's location is resolved from configuration (`DSH_KB_SKILLS_LIB` →
107
+ `<source root>/z3-skills/skills-library` → `~/kb/z3-skills/skills-library`) —
108
+ never hardcoded.
109
+ - **`--prune` on the scheduled refresh**: the mirror is a derived cache, so files
110
+ deleted upstream must not linger locally and get re-indexed.
111
+
112
+ ### Changed before the first public release
113
+
114
+ - MCP resident cost 1289 → 1674 chars (~353 → ~450 tokens) with the fourth tool;
115
+ the schema budget in tests moved 1500 → 1800 and the docs' cost table follows.
@@ -0,0 +1,81 @@
1
+ # Contributing
2
+
3
+ Thanks for looking. This project is small on purpose, and a few of its rules are
4
+ load-bearing — please read the short list below before sending a patch.
5
+
6
+ ## Running the tests
7
+
8
+ No dependencies, no network, no fixtures to download:
9
+
10
+ ```bash
11
+ python -m unittest discover -s tests -t . -v
12
+ ```
13
+
14
+ There is nothing to install. If you find yourself needing `pip install` to run
15
+ the suite, that is a bug worth reporting.
16
+
17
+ ## House rules
18
+
19
+ **1. Zero runtime dependencies.** This is the headline feature, not an
20
+ accident. A pull request that adds a runtime dependency needs a very good
21
+ reason and a discussion first. Standard library only.
22
+
23
+ **2. The tests stay offline.** Every test must pass on a machine with no
24
+ network and no shares mounted. Transport tests use fakes and stubs (see
25
+ `tests/test_sources.py`, which drives the SMB parser through a stub
26
+ `smbclient`). If a feature cannot be tested offline, design it so it can.
27
+
28
+ **3. The MCP server is hand-rolled.** `src/dsh_kb/mcp_server.py` speaks
29
+ newline-delimited JSON-RPC on the standard library, and
30
+ `tests/test_mcp.py` proves it against the *official* MCP SDK. Do not replace it
31
+ with the SDK: dependency drift would become our users' problem.
32
+
33
+ **4. Docstrings explain *why*.** The code says what it does. Comments earn their
34
+ place by recording a reason, a constraint, or a scar. Where a bug was found the
35
+ hard way, the comment says how — several do, and they are the most useful lines
36
+ in the repository.
37
+
38
+ **5. Privacy invariants must not regress.** These are enforced by tests:
39
+
40
+ * the `z2-sensitive` zone is **never mirrored to disk**, not merely skipped at
41
+ query time;
42
+ * SQLite never lives on a network share (its locking is unsound there);
43
+ * writes by an agent are proposals into `_incoming/`, never edits to a zone;
44
+ * the `never` list is applied at the mirror *and* the index layer.
45
+
46
+ ## Adding a transport
47
+
48
+ Transports implement the small interface in `src/dsh_kb/sources.py`:
49
+
50
+ ```python
51
+ def walk(self) -> list[Entry]: ... # the whole tree, one call
52
+ def read(self, rel: str) -> bytes: ...
53
+ def write(self, rel: str, data: bytes) -> None # optional
54
+ def describe(self) -> dict: ... # for `doctor` and `status`
55
+ def ensure_available(self) -> None: ... # raise SourceUnavailable with a sentence a human can act on
56
+ ```
57
+
58
+ Two things matter more than the mechanics:
59
+
60
+ * **errors must read as sentences.** `ESTALE` becomes "the share looks stale:
61
+ remount it". A raw traceback is a bug.
62
+ * **writes must create their own parents.** `MKCOL`/`mkdir` do not recurse;
63
+ a bare `PUT` into a missing collection fails with an opaque 500.
64
+
65
+ ## Commits
66
+
67
+ English, imperative, a short subject and a body that explains the *why*:
68
+
69
+ ```
70
+ fix: create missing parent collections on WebDAV writes
71
+
72
+ A bare PUT into a missing collection is HTTP 500 errno 2, and MKCOL does not
73
+ recurse. A 217-file import failed 217 times before this existed.
74
+ ```
75
+
76
+ ## Reporting a problem
77
+
78
+ Open an issue with the command you ran, what you expected, and what happened —
79
+ `dsh-kb doctor` covers a lot of ground and its output is usually the fastest
80
+ path to a diagnosis. For anything security-related, see [SECURITY.md](SECURITY.md)
81
+ instead of opening a public issue.
dsh_kb-0.1.0/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 dsh-kb contributors
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,13 @@
1
+ # The source distribution should be enough to read the project and run its
2
+ # suite: the docs are part of the deliverable, and the tests are deliberately
3
+ # dependency-free so a reader can verify the claims without installing anything.
4
+ include README.md
5
+ include README.zh.md
6
+ include CHANGELOG.md
7
+ include CONTRIBUTING.md
8
+ include SECURITY.md
9
+ include LICENSE
10
+ include examples/config.json
11
+ include examples/demo.sh
12
+ recursive-include docs *.md
13
+ recursive-include tests *.py
dsh_kb-0.1.0/PKG-INFO ADDED
@@ -0,0 +1,375 @@
1
+ Metadata-Version: 2.4
2
+ Name: dsh-kb
3
+ Version: 0.1.0
4
+ Summary: A personal Markdown knowledge base with a local mirror and index, built to be queried cheaply by AI agents.
5
+ Author: dsh-kb contributors
6
+ License-Expression: MIT
7
+ Project-URL: Homepage, https://github.com/LUgABeEEE/dsh-kb
8
+ Project-URL: Documentation, https://github.com/LUgABeEEE/dsh-kb#readme
9
+ Project-URL: Source, https://github.com/LUgABeEEE/dsh-kb
10
+ Project-URL: Issues, https://github.com/LUgABeEEE/dsh-kb/issues
11
+ Project-URL: Changelog, https://github.com/LUgABeEEE/dsh-kb/blob/main/CHANGELOG.md
12
+ Keywords: knowledge-base,markdown,full-text-search,sqlite-fts5,ai-agents,local-first,webdav
13
+ Classifier: Development Status :: 3 - Alpha
14
+ Classifier: Intended Audience :: Developers
15
+ Classifier: Operating System :: OS Independent
16
+ Classifier: Programming Language :: Python :: 3
17
+ Classifier: Programming Language :: Python :: 3.10
18
+ Classifier: Programming Language :: Python :: 3.11
19
+ Classifier: Programming Language :: Python :: 3.12
20
+ Classifier: Programming Language :: Python :: 3.13
21
+ Classifier: Topic :: Text Processing :: Indexing
22
+ Classifier: Topic :: Utilities
23
+ Requires-Python: >=3.10
24
+ Description-Content-Type: text/markdown
25
+ License-File: LICENSE
26
+ Dynamic: license-file
27
+
28
+ # dsh-kb
29
+
30
+ **A personal knowledge base for AI agents: the source of truth lives on shared
31
+ storage, each machine keeps a mirror and an index, retrieval is fast and works
32
+ offline.**
33
+
34
+ English | [中文](README.zh.md)
35
+
36
+ ---
37
+
38
+ ## Who it is for, what it costs, and how to try it
39
+
40
+ **For** one person (or a small team) whose knowledge already lives in a Markdown
41
+ folder — on a network share, a NAS, or just a directory — and whose agent keeps
42
+ re-reading it. Not a hosted product, not a multi-tenant service, not a
43
+ note-taking UI: one machine, one bundle, one owner.
44
+
45
+ **What it costs**: Python ≥ 3.10 and **no dependencies** — no daemon, no server,
46
+ no database service, no root. One process per command. State (mirror + index)
47
+ lives under `~/.dsh-kb` and runs about 1.9× the size of the bundle; the index is
48
+ local, so a share that is offline or slow costs you freshness, not availability.
49
+
50
+ **Try it in two minutes** — no share and no config file needed:
51
+
52
+ ```bash
53
+ git clone https://github.com/LUgABeEEE/dsh-kb && cd dsh-kb
54
+ bash examples/demo.sh # builds a throwaway bundle, indexes it, searches it
55
+ ```
56
+
57
+ ---
58
+
59
+ ## The problem
60
+
61
+ Personal knowledge — notes, lessons learned, decisions, project records —
62
+ usually ends up scattered across session transcripts, a memory plugin, a pile
63
+ of Markdown reports and a network share. So every agent session does the same
64
+ thing: it **re-reads several sources**, **stuffs raw text into the context
65
+ window** (one measured deployment re-sent ~230 000 tokens of context per
66
+ request), and when it still cannot find the answer it says so.
67
+
68
+ dsh-kb collapses that into one path:
69
+
70
+ ```
71
+ one Markdown directory on shared storage ← single source of truth
72
+ │ incremental pull
73
+ ▼
74
+ local mirror + local FTS5 index ← milliseconds, works offline
75
+ │ two narrow tools
76
+ ▼
77
+ the agent asks for "path + line + 240-char snippet"
78
+ ```
79
+
80
+ **The size of a result is independent of the size of the corpus** — 200 pages or
81
+ 10 000 pages, a search returns roughly 2.4 KB. That property is the whole point:
82
+ growing the knowledge base does not make each lookup more expensive.
83
+
84
+ ---
85
+
86
+ ## Core ideas
87
+
88
+ | # | Idea | Why |
89
+ |---|---|---|
90
+ | 1 | **One source of truth, a derived mirror** | The mirror is never an authority, so **merge conflicts cannot happen**; refreshing is a pure function of `(remote listing, manifest)` — interruptible and cheap to run often |
91
+ | 2 | **The index is local and never on the share** | SQLite's locking is not reliable on network filesystems; the index is a disposable derivative that can always be rebuilt |
92
+ | 3 | **Privacy zones compiled into code** | A sensitive zone is **not even copied to disk**, is not indexed, and is not returned by a default query — three layers, no conventions required |
93
+ | 4 | **Proposal-based writes** | An agent only writes to `_incoming/`; promotion is done by a human or a scheduled job — this prevents "a dozen contradictory near-duplicates about one topic" |
94
+ | 5 | **Precision first, recall second** | A query starts as a strict `AND`; only when that is empty does it relax to `OR`, and then **generic terms are dropped first** — otherwise a single shared word such as `UNIQ` drags in unrelated documents |
95
+
96
+ Rationale and measurements: [docs/design.md](docs/design.md).
97
+
98
+ ---
99
+
100
+ ## Install
101
+
102
+ Standard library only, no third-party dependencies (Python ≥ 3.10):
103
+
104
+ ```bash
105
+ pip install -e . # or simply PYTHONPATH=src python3 -m dsh_kb
106
+ dsh-kb --version
107
+ ```
108
+
109
+ ---
110
+
111
+ ## Quick start
112
+
113
+ ### 1. Create a bundle (a directory on shared storage)
114
+
115
+ ```
116
+ <share>/kb/
117
+ index.md # optional: progressive-disclosure front door
118
+ log.md # optional: change log
119
+ z0-public/ # public zone
120
+ z1-restricted/ # restricted zone
121
+ z2-sensitive/ # sensitive zone (not mirrored, not indexed)
122
+ _incoming/ # proposals (the only place an agent writes)
123
+ _meta/ # conventions, registry, receipts
124
+ ```
125
+
126
+ ### 2. Configure
127
+
128
+ Configuration lives in `~/.dsh-kb/config.json` — deliberately **outside any
129
+ checkout**, so a public clone never carries your paths or private data:
130
+
131
+ ```json
132
+ {
133
+ "source": "auto",
134
+ "mount": "/mnt/nas/kb",
135
+ "local_root": "/srv/kb-share",
136
+ "webdav_root": "https://nas.example.com:5000/webdav/kb",
137
+ "stale_minutes": 15
138
+ }
139
+ ```
140
+
141
+ `source=auto` picks the first usable option in the order **mount → local
142
+ directory → Samba (smbclient) → WebDAV**. Walking a local filesystem tree takes
143
+ milliseconds; WebDAV costs one PROPFIND per directory (~55 ms on a consumer
144
+ NAS), so prefer a mount when you have one. Paths may be written POSIX-style, as
145
+ a Windows drive, or as a UNC path (`Z:\kb`, `\\nas\share\kb`, `/mnt/nas/kb`) —
146
+ all are normalised.
147
+
148
+ > **Samba/CIFS mounts need extra care** (we hit all of these in production): a
149
+ > mount can be `rw` and still read-only for the account; a dropped session makes
150
+ > the whole tree raise `ESTALE`; and timestamps have **one-second** granularity.
151
+ > All three are handled explicitly — see
152
+ > [docs/deployment.md](docs/deployment.md#samba--cifs-mounts-linux-and-windows).
153
+
154
+ ### 3. Pull, index, search
155
+
156
+ ```bash
157
+ dsh-kb doctor # check source reachability and mirror hygiene
158
+ dsh-kb refresh # sync + index — what a scheduler should call
159
+ dsh-kb search "mount stale" -n 5
160
+ dsh-kb search "UNIQ-000123" --json # machine-readable
161
+ dsh-kb read z0-public/mount.md --start 1 --lines 40
162
+ dsh-kb push # upload local _incoming/ proposals
163
+ ```
164
+
165
+ **Only `sync`, `push` and `doctor` write to the share** (and `doctor` writes just
166
+ two probe files — see the CLI table). `diff` also *reads* the source listing (to
167
+ report what changed remotely), but it never writes to the share, the mirror or
168
+ the index — it is safe to call from a timer. The share can be offline, a
169
+ certificate can expire, a laptop can be away — search still works. This is a
170
+ hard requirement: otherwise "offline capable" is a lie.
171
+
172
+ ---
173
+
174
+ ## Privacy model
175
+
176
+ | Zone | Mirrored locally | Indexed | Returned by default |
177
+ |---|---|---|---|
178
+ | `z0-public` | ✅ | ✅ | ✅ |
179
+ | `z1-restricted` | ✅ | ✅ | ✅ (`inject: false`, for injectors to respect) |
180
+ | `z2-sensitive` | ❌ **never copied** | ❌ | ❌ (requires explicit `--zone z2`) |
181
+ | `z3-skills` | ✅ | ❌ **never indexed** (once registered) | ❌ (own entry point: `dsh-kb skills`; 2.3k `SKILL.md` would drown retrieval) |
182
+ | `_meta/`, `_incoming/` | ✅ | ✅ | ✅ (layout, not private content) |
183
+ | paths matching `never` | ❌ | ❌ | ❌ |
184
+
185
+ `never` defaults to `credentials`, `.credentials`, `secrets`, `id_rsa`, `密钥`
186
+ ("keys"), `密钥库` and is extendable. All three enforcement points are covered
187
+ by tests.
188
+
189
+ > **`z3-skills` is not built in.** Register it in `config.json` with
190
+ > `"index": false` *before* moving files into it — an unregistered top-level
191
+ > directory is mirrored **and** indexed. See
192
+ > [`examples/config.json`](examples/config.json).
193
+
194
+ ---
195
+
196
+ ## CLI
197
+
198
+ | Command | Writes? | Purpose |
199
+ |---|---|---|
200
+ | `status` | — | configuration, source, mirror and index state, freshness |
201
+ | `doctor` | ✔ probes | per-check diagnostics, including "does the mirror contain anything it should not" (writes two probe files, see below) |
202
+ | `diff [--incoming-days DAYS] [--json]` | — (reads the source) | read-only health report: remote changes, local mirror gaps, index drift, backlog, conflicts, escapes |
203
+ | `rag-check [--prune] [--force]` | only with `--prune` | vector store ↔ index consistency, page **and chunk** level; report-only (dry-run) by default |
204
+ | `sync [--prune] [--full] [--dry-run]` | ✔ | incremental pull; `--prune` is opt-in; `--full` ignores the manifest |
205
+ | `index` | ✔ (local) | rebuild the index |
206
+ | `refresh [--prune] [--full]` | ✔ | sync + index |
207
+ | `search <q> [-n] [--zone] [--snippet] [--refresh] [--no-auto-refresh] [--json]` | — | ranked search |
208
+ | `read <path> [--start] [--lines] [--json]` | — | line window |
209
+ | `skills search\|show\|stats` | — | query the skill library in the `z3-skills` asset zone |
210
+ | `push [files…] [--force] [--expect SHA256] [--log [N]] [--json]` | ✔ | upload proposals |
211
+ | `mcp` | — | run the MCP server on stdio (for agent hosts) |
212
+
213
+ Shared flags are accepted before or after the subcommand.
214
+
215
+ **Not everything labelled "diagnostic" is read-only.** `doctor` is the one
216
+ exception: it creates and deletes a probe file in `home`, and — when the source
217
+ exposes it — writes and deletes a probe under `_incoming/` on the share, to
218
+ answer "can `push` work at all?". The probes are removed again; nothing else is
219
+ written. Don't wire `doctor` into anything that must be strictly read-only.
220
+
221
+ **Exit codes.** `diff` and `rag-check` share one contract: **0** clean, **10**
222
+ something needs a human, **11** local state is unreadable / the store is broken.
223
+ `rag-check --force` without `--prune` is a *usage* error and exits **1** — it
224
+ would override nothing, so it refuses rather than pretending it forced
225
+ something.
226
+
227
+ ### What the two reports actually compare
228
+
229
+ `dsh-kb diff` compares **two pairs, in two directions**, and splits both by file
230
+ kind (`markdown` / `other`) so image or JSON churn can't drown the `.md` diff:
231
+
232
+ * *remote listing vs manifest* → `content`: what changed on the share. **Report
233
+ only, never an alarm** — the next `sync` converges.
234
+ * *manifest vs local mirror* → `mirror_gaps`: what is wrong with **this
235
+ machine's copy**. Covers every file, not just Markdown. A local file that the
236
+ snapshot recorded and the disk lost is `mirror-gap` (exit 10) — it used to be
237
+ invisible, because remote and manifest were both unchanged while `sync` would
238
+ re-fetch it. Extra local files are counted but not alarmed: `sync --prune`
239
+ cannot delete them, so a human has to decide.
240
+
241
+ `dsh-kb rag-check` compares the vector store against the index at **page**
242
+ (`path`) and **chunk** (`path` + `startline`) granularity. The chunk view is
243
+ what catches "one page, five chunks, only two embedded" — page-level counters
244
+ stay at 0 for that shape. On an older store whose tables have no `startline`
245
+ column the comparison degrades to page-level and says so in `note`.
246
+
247
+ ---
248
+
249
+ ## Configuration reference
250
+
251
+ | Key | Env | Default | Meaning |
252
+ |---|---|---|---|
253
+ | `source` | `DSH_KB_SOURCE` | `auto` | `auto` / `local` / `mount` / `smb` / `webdav` |
254
+ | `mount` | `DSH_KB_MOUNT` | — | mount point (Samba/CIFS, NFS, or a plain directory) |
255
+ | `local_root` | `DSH_KB_LOCAL_ROOT` | — | plain directory |
256
+ | `smb_share` | `DSH_KB_SMB_SHARE` | — | unmounted Samba share, e.g. `//host/share` (needs `smbclient`) |
257
+ | `webdav_root` | `DSH_KB_WEBDAV_ROOT` | — | WebDAV collection URL |
258
+ | `stale_minutes` | `DSH_KB_STALE_MINUTES` | `15` | auto-refresh threshold |
259
+ | `mtime_tolerance` | `DSH_KB_MTIME_TOLERANCE` | `1.0` | mtime delta still counted as "unchanged" |
260
+ | `snippet_chars` | `DSH_KB_SNIPPET_CHARS` | `240` | snippet width |
261
+ | `default_limit` | `DSH_KB_DEFAULT_LIMIT` | `8` | hits per query |
262
+ | `zones` / `never` | — | see above | policy |
263
+
264
+ WebDAV secrets are read **only** from the environment or a credentials file,
265
+ never from `config.json`:
266
+
267
+ ```bash
268
+ export DSH_KB_WEBDAV_USER=me
269
+ export DSH_KB_WEBDAV_PASSWORD=... # or DSH_KB_WEBDAV_CREDENTIALS=<key=value file>
270
+ export DSH_KB_WEBDAV_CERT=... DSH_KB_WEBDAV_KEY=... DSH_KB_WEBDAV_CA=... # mutual TLS
271
+ export DSH_KB_WEBDAV_CONNECT=192.0.2.10 # connect to an IP, verify TLS against the CN
272
+ ```
273
+
274
+ ---
275
+
276
+ ## Search behaviour
277
+
278
+ - **CJK**: indexing keeps the original text *and* appends adjacent character
279
+ bigrams, so `挂载` matches `重新挂载` (a 2-character query, which a plain
280
+ `unicode61` tokenizer cannot do).
281
+ - **Latin and identifiers**: the original text is indexed verbatim, so
282
+ `z2-sensitive`, `page-000123.md` and `fts5` all match.
283
+ - **Two passes**: strict `AND` first, then a relaxed `OR` restricted to
284
+ *selective* tokens. Results carry `relaxed: true` and `dropped: [...]` so the
285
+ caller knows precision was traded for recall.
286
+ - **Snippet location** prefers the whole query as a literal, then word-boundary
287
+ token hits — so searching `mount -a` reports that line, not the line holding
288
+ `/proc/mounts`.
289
+
290
+ ---
291
+
292
+ ## Measured performance
293
+
294
+ Single Linux host, bundled SQLite with FTS5, synthetic corpus (~3.4 KB per page,
295
+ mixed Chinese and English):
296
+
297
+ | Pages | Corpus | Index build | Index size | Blow-up | Engine (worst case) | Exact identifier | CLI end-to-end |
298
+ |---|---|---|---|---|---|---|---|
299
+ | 1 000 | 3.3 MB | 0.38 s | 7.7 MB | 2.3× | — | — | 46 ms |
300
+ | 5 000 | 16.7 MB | 1.6 s | 33.1 MB | 2.0× | — | — | 59 ms |
301
+ | 10 000 | 33.4 MB | 3.2 s | 62.2 MB | **1.9×** | 29.6 ms | **0.2 ms** | **72 ms** |
302
+
303
+ - End-to-end includes ~30 ms of Python interpreter start-up.
304
+ - The spread is large: an exact identifier takes 0.2 ms, a generic keyword that
305
+ matches nearly everything takes 29.6 ms.
306
+ - A search returns about **2.4 KB**, independent of corpus size.
307
+
308
+ The index stays near 1.9× because the FTS5 table stores **no content**
309
+ (`content=''`) and the text lives once in a separate `meta` table, joined back
310
+ by rowid. Inlining the text into FTS5 measured **8.1×**.
311
+
312
+ ---
313
+
314
+ ## Agent integration
315
+
316
+ **Positioning: a supplement, not standing context.** Nothing is auto-injected and nothing is searched per turn — it is consulted once, when past decisions, environment facts or history are actually needed.
317
+
318
+ Two shapes, one implementation (`dsh_kb.service`), so their output cannot drift:
319
+
320
+ | Shape | When | Cost |
321
+ |---|---|---|
322
+ | **CLI + `--json`** | any agent that can run a command | one subprocess and one tool round-trip (~30 ms) |
323
+ | **MCP server (stdio)** | the host speaks MCP (recommended) | **~420 tokens** standing (1559-char `tools/list` + 125-char instructions, measured), one less layer per call |
324
+
325
+ ```bash
326
+ dsh-kb mcp # JSON-RPC on stdin/stdout, exposing search / read / status / skills
327
+ ```
328
+
329
+ Wiring it into DSH is one `insert` (other hosts:
330
+ [docs/agent-integration.md](docs/agent-integration.md)):
331
+
332
+ ```yaml
333
+ - id: mcp-kb
334
+ name: '@deepseek-ai/dsh-mcp-client'
335
+ config:
336
+ serverName: kb
337
+ transport: stdio
338
+ command: python3
339
+ args: [-m, dsh_kb, mcp, --home, /srv/dsh-kb, --source, auto, --mount, /mnt/nas/kb]
340
+ env: { PYTHONPATH: /opt/dsh-kb/src }
341
+ ```
342
+
343
+ > **Why the configuration goes in `args`, not `env`**: some hosts scrub the
344
+ > environment of the MCP child process (credential-shaped names, stale
345
+ > prefixes). Command-line arguments are never scrubbed. Measured: with every
346
+ > `DSH_KB_*` variable removed and only `args` set, it still works against a real
347
+ > share.
348
+
349
+ Design choices — no `mode`/`rerank` knobs, no write tools, failures returned as
350
+ tool results — are in [docs/design.md](docs/design.md#agent-integration).
351
+
352
+ ---
353
+
354
+ ## Status
355
+
356
+ Alpha (0.1.0) with 308 tests (unit, end-to-end, transport, CLI smoke and MCP — all offline):
357
+
358
+ ```bash
359
+ python3 -m unittest discover -s tests -t . -v
360
+ ```
361
+
362
+ With the official MCP SDK installed, one extra **interop test** runs (a real
363
+ client against this hand-rolled server); otherwise it skips.
364
+
365
+ ## Roadmap
366
+
367
+ - [x] MCP server (`mcp__kb__search` / `read` / `status` / `skills`)
368
+ - [ ] Semantic fallback, used only when keyword search returns nothing (**no vector service**)
369
+ - [ ] Scheduled refresh (systemd timer / cron examples)
370
+ - [ ] Chinese embedding evaluation to pick a default model
371
+ - [ ] Optional read-only HTTP viewer
372
+
373
+ ## License
374
+
375
+ MIT — see [LICENSE](LICENSE).