dsh-kb 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- dsh_kb-0.1.0/CHANGELOG.md +115 -0
- dsh_kb-0.1.0/CONTRIBUTING.md +81 -0
- dsh_kb-0.1.0/LICENSE +21 -0
- dsh_kb-0.1.0/MANIFEST.in +13 -0
- dsh_kb-0.1.0/PKG-INFO +375 -0
- dsh_kb-0.1.0/README.md +348 -0
- dsh_kb-0.1.0/README.zh.md +343 -0
- dsh_kb-0.1.0/SECURITY.md +62 -0
- dsh_kb-0.1.0/docs/agent-integration.md +178 -0
- dsh_kb-0.1.0/docs/deployment.md +287 -0
- dsh_kb-0.1.0/docs/design.md +297 -0
- dsh_kb-0.1.0/docs/release.md +113 -0
- dsh_kb-0.1.0/examples/config.json +15 -0
- dsh_kb-0.1.0/examples/demo.sh +79 -0
- dsh_kb-0.1.0/pyproject.toml +48 -0
- dsh_kb-0.1.0/setup.cfg +4 -0
- dsh_kb-0.1.0/src/dsh_kb/__init__.py +16 -0
- dsh_kb-0.1.0/src/dsh_kb/__main__.py +6 -0
- dsh_kb-0.1.0/src/dsh_kb/cli.py +1513 -0
- dsh_kb-0.1.0/src/dsh_kb/config.py +399 -0
- dsh_kb-0.1.0/src/dsh_kb/index.py +710 -0
- dsh_kb-0.1.0/src/dsh_kb/mcp_server.py +325 -0
- dsh_kb-0.1.0/src/dsh_kb/mirror.py +468 -0
- dsh_kb-0.1.0/src/dsh_kb/mounts.py +189 -0
- dsh_kb-0.1.0/src/dsh_kb/py.typed +0 -0
- dsh_kb-0.1.0/src/dsh_kb/ragcheck.py +768 -0
- dsh_kb-0.1.0/src/dsh_kb/service.py +525 -0
- dsh_kb-0.1.0/src/dsh_kb/skills.py +168 -0
- dsh_kb-0.1.0/src/dsh_kb/sources.py +582 -0
- dsh_kb-0.1.0/src/dsh_kb/supersede.py +371 -0
- dsh_kb-0.1.0/src/dsh_kb/text.py +267 -0
- dsh_kb-0.1.0/src/dsh_kb/webdav.py +228 -0
- dsh_kb-0.1.0/src/dsh_kb.egg-info/PKG-INFO +375 -0
- dsh_kb-0.1.0/src/dsh_kb.egg-info/SOURCES.txt +47 -0
- dsh_kb-0.1.0/src/dsh_kb.egg-info/dependency_links.txt +1 -0
- dsh_kb-0.1.0/src/dsh_kb.egg-info/entry_points.txt +2 -0
- dsh_kb-0.1.0/src/dsh_kb.egg-info/top_level.txt +1 -0
- dsh_kb-0.1.0/tests/__init__.py +1 -0
- dsh_kb-0.1.0/tests/test_cli.py +894 -0
- dsh_kb-0.1.0/tests/test_e2e.py +785 -0
- dsh_kb-0.1.0/tests/test_index_cas.py +574 -0
- dsh_kb-0.1.0/tests/test_index_zone.py +326 -0
- dsh_kb-0.1.0/tests/test_kbask_zone.py +122 -0
- dsh_kb-0.1.0/tests/test_mcp.py +290 -0
- dsh_kb-0.1.0/tests/test_ragcheck.py +1042 -0
- dsh_kb-0.1.0/tests/test_skills.py +121 -0
- dsh_kb-0.1.0/tests/test_sources.py +539 -0
- dsh_kb-0.1.0/tests/test_supersede.py +350 -0
- dsh_kb-0.1.0/tests/test_units.py +331 -0
|
@@ -0,0 +1,115 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project are documented here.
|
|
4
|
+
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
|
|
5
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
|
+
|
|
7
|
+
## [0.1.0] — 2026-10-25
|
|
8
|
+
|
|
9
|
+
First public version. Written against a real deployment: a Markdown bundle on
|
|
10
|
+
a network share, mirrored to a Linux host and searched by an AI agent.
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- **Bundle layout** with privacy zones (`z0-public`, `z1-restricted`,
|
|
15
|
+
`z2-sensitive`), a layout area (`_meta`, `_incoming`) and a root index.
|
|
16
|
+
- **Local mirror** with a manifest-driven incremental sync: a file whose size
|
|
17
|
+
and mtime are unchanged is never transferred. Opt-in `--prune`.
|
|
18
|
+
- **Local full-text index** on SQLite FTS5: CJK bigram expansion plus the
|
|
19
|
+
original text, a contentless FTS table with a separate `meta` table, and a
|
|
20
|
+
`fts5vocab` table used for selectivity.
|
|
21
|
+
- **Two-stage search**: strict `AND` for precision, relaxing to a selective
|
|
22
|
+
`OR` only when the strict pass is empty.
|
|
23
|
+
- **Zone policy compiled into code**, enforced at three points: mirroring,
|
|
24
|
+
indexing and querying. Sensitive zones are not even copied to disk.
|
|
25
|
+
- **Read sources**: a filesystem view (local directory, Samba/CIFS or NFS
|
|
26
|
+
mount, Windows drive or UNC path) and a dependency-free generic WebDAV client
|
|
27
|
+
(optional Basic auth and mutual TLS, with connect-host/SNI separation for
|
|
28
|
+
appliances whose certificate CN differs from their address).
|
|
29
|
+
- **Unmounted SMB** (`source=smb`) through `smbclient`, for hosts where mounting
|
|
30
|
+
needs root or a mount has gone stale. Parses the real `recurse ON; ls` output
|
|
31
|
+
(verified against a live server, including names with spaces and the trailing
|
|
32
|
+
`blocks of size …` line).
|
|
33
|
+
- **Network-mount awareness**: `/proc/mounts` inspection picks the real
|
|
34
|
+
filesystem rather than its `autofs` trigger, reports `rw`/`ro`, and turns
|
|
35
|
+
`ESTALE` / `EACCES` / `ENOENT` into actionable sentences. `sync --full` covers
|
|
36
|
+
the one-second timestamp granularity of network shares.
|
|
37
|
+
- **CLI** (`dsh-kb status|doctor|sync|index|refresh|search|read|push|mcp`) with
|
|
38
|
+
`--json` output, offline operation for everything except `sync`/`push`/
|
|
39
|
+
`doctor`, best-effort auto-refresh before a search, and a three-level
|
|
40
|
+
`doctor` report (`ok` / `~~` warning / `!!` failure).
|
|
41
|
+
- **MCP server** (`dsh-kb mcp`): newline-delimited JSON-RPC 2.0 on stdio,
|
|
42
|
+
exposing `search` / `read` / `status`. Hand-rolled on the standard library to
|
|
43
|
+
keep the zero-dependency promise; interoperability is asserted against the
|
|
44
|
+
official Python SDK in the test-suite rather than assumed. Tool schemas
|
|
45
|
+
measure ~1543 characters (~385 tokens) resident, with a test guarding that
|
|
46
|
+
budget.
|
|
47
|
+
- **Use-case layer** (`dsh_kb.service`) shared by the CLI and the MCP server, so
|
|
48
|
+
`--json` output and tool results cannot drift apart.
|
|
49
|
+
- **86→122 tests**: unit, end-to-end over a fixture bundle, transport tests with
|
|
50
|
+
a stub `smbclient`, CLI smoke tests, MCP protocol/process tests, and an
|
|
51
|
+
SDK interop test; no network needed.
|
|
52
|
+
|
|
53
|
+
### Fixed
|
|
54
|
+
|
|
55
|
+
- **`--home` did not relocate `config.json`.** The file was loaded from the
|
|
56
|
+
environment-derived home *before* overrides were applied, so
|
|
57
|
+
`dsh-kb --home X` silently ran on the built-in defaults — even though the
|
|
58
|
+
CLI's own help calls `--home` the "state directory (mirror, index, manifest,
|
|
59
|
+
logs, config)". Measured on a real deployment: three `never` path exclusions
|
|
60
|
+
and every zone's `inject` flag were ignored, and the exclusions are a privacy
|
|
61
|
+
guard. `home` is now resolved before the file is read; covered by regression
|
|
62
|
+
tests that fail without the fix.
|
|
63
|
+
|
|
64
|
+
- **`WebdavSource.write` did not create parent collections.** A bare `PUT` into
|
|
65
|
+
a missing collection is just `HTTP 500 errno 2 No such file or directory`, and
|
|
66
|
+
because `MKCOL` does not recurse, every level must be created in turn. Found
|
|
67
|
+
the honest way: a 217-file import into a freshly laid-out bundle failed 217
|
|
68
|
+
times. Collections already handled are remembered per instance, so a bulk
|
|
69
|
+
import does not repeat them.
|
|
70
|
+
|
|
71
|
+
- **Token and resource thrift** (the base is meant to be a supplement consulted on
|
|
72
|
+
demand, not standing context): tool schemas trimmed 1543 → 1289 characters
|
|
73
|
+
(~385 → ~322 tokens), the `instructions` line rewritten from "search before
|
|
74
|
+
reading files" (which invited a lookup every turn) to an explicit on-demand
|
|
75
|
+
stance, all zones set to `inject: false`, and an empty refresh **no longer
|
|
76
|
+
rebuilds the index** — the scheduler's four daily runs and the pre-search
|
|
77
|
+
auto-refresh are now a directory walk when nothing changed.
|
|
78
|
+
- **Unknown `config.json` keys are ignored and reported** instead of crashing the
|
|
79
|
+
CLI with a `TypeError`; `doctor` lists them as a warning. Found the honest way:
|
|
80
|
+
a stray note key took the whole command down.
|
|
81
|
+
|
|
82
|
+
### Known limitations
|
|
83
|
+
|
|
84
|
+
- Search is lexical only. A paraphrase with no shared wording returns nothing;
|
|
85
|
+
a semantic fallback is planned but not implemented.
|
|
86
|
+
- Relaxation to a broader `OR` match is deliberately *not* applied to a single
|
|
87
|
+
latin word or identifier — `UNIQ-…` is matched exactly or not at all.
|
|
88
|
+
- No write path for the model: writes go through `push` into `_incoming/` and
|
|
89
|
+
are promoted to a zone by a human or a scheduled job.
|
|
90
|
+
- The WebDAV client speaks the subset of RFC 4918 this project needs
|
|
91
|
+
(`PROPFIND`, `GET`, `PUT`, `MKCOL`, `DELETE`); no locking or `COPY`/`MOVE`.
|
|
92
|
+
- `source=smb` costs one process and one round trip per file; it is a fallback,
|
|
93
|
+
not a replacement for a mount.
|
|
94
|
+
- The MCP server implements the three methods an agent needs; prompts,
|
|
95
|
+
resources, sampling and subscriptions are not advertised.
|
|
96
|
+
|
|
97
|
+
### Added before the first public release
|
|
98
|
+
|
|
99
|
+
- **`skills` subcommand and MCP tool** for a curated skill library kept in a
|
|
100
|
+
`z3-skills` asset zone. Such a zone has to be registered in `config.json`
|
|
101
|
+
(`index: false`) *before* anything is moved into it: `index_allows()` lets
|
|
102
|
+
**unregistered top-level directories through**, so moving files alone does not
|
|
103
|
+
keep them out of the index — and 2.3k `SKILL.md` files would otherwise drown
|
|
104
|
+
knowledge-page retrieval. Both entry points forward to the library's own
|
|
105
|
+
`bin/find.py`, keeping tokenisation and scoring in a single implementation, and
|
|
106
|
+
the library's location is resolved from configuration (`DSH_KB_SKILLS_LIB` →
|
|
107
|
+
`<source root>/z3-skills/skills-library` → `~/kb/z3-skills/skills-library`) —
|
|
108
|
+
never hardcoded.
|
|
109
|
+
- **`--prune` on the scheduled refresh**: the mirror is a derived cache, so files
|
|
110
|
+
deleted upstream must not linger locally and get re-indexed.
|
|
111
|
+
|
|
112
|
+
### Changed before the first public release
|
|
113
|
+
|
|
114
|
+
- MCP resident cost 1289 → 1674 chars (~353 → ~450 tokens) with the fourth tool;
|
|
115
|
+
the schema budget in tests moved 1500 → 1800 and the docs' cost table follows.
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
# Contributing
|
|
2
|
+
|
|
3
|
+
Thanks for looking. This project is small on purpose, and a few of its rules are
|
|
4
|
+
load-bearing — please read the short list below before sending a patch.
|
|
5
|
+
|
|
6
|
+
## Running the tests
|
|
7
|
+
|
|
8
|
+
No dependencies, no network, no fixtures to download:
|
|
9
|
+
|
|
10
|
+
```bash
|
|
11
|
+
python -m unittest discover -s tests -t . -v
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
There is nothing to install. If you find yourself needing `pip install` to run
|
|
15
|
+
the suite, that is a bug worth reporting.
|
|
16
|
+
|
|
17
|
+
## House rules
|
|
18
|
+
|
|
19
|
+
**1. Zero runtime dependencies.** This is the headline feature, not an
|
|
20
|
+
accident. A pull request that adds a runtime dependency needs a very good
|
|
21
|
+
reason and a discussion first. Standard library only.
|
|
22
|
+
|
|
23
|
+
**2. The tests stay offline.** Every test must pass on a machine with no
|
|
24
|
+
network and no shares mounted. Transport tests use fakes and stubs (see
|
|
25
|
+
`tests/test_sources.py`, which drives the SMB parser through a stub
|
|
26
|
+
`smbclient`). If a feature cannot be tested offline, design it so it can.
|
|
27
|
+
|
|
28
|
+
**3. The MCP server is hand-rolled.** `src/dsh_kb/mcp_server.py` speaks
|
|
29
|
+
newline-delimited JSON-RPC on the standard library, and
|
|
30
|
+
`tests/test_mcp.py` proves it against the *official* MCP SDK. Do not replace it
|
|
31
|
+
with the SDK: dependency drift would become our users' problem.
|
|
32
|
+
|
|
33
|
+
**4. Docstrings explain *why*.** The code says what it does. Comments earn their
|
|
34
|
+
place by recording a reason, a constraint, or a scar. Where a bug was found the
|
|
35
|
+
hard way, the comment says how — several do, and they are the most useful lines
|
|
36
|
+
in the repository.
|
|
37
|
+
|
|
38
|
+
**5. Privacy invariants must not regress.** These are enforced by tests:
|
|
39
|
+
|
|
40
|
+
* the `z2-sensitive` zone is **never mirrored to disk**, not merely skipped at
|
|
41
|
+
query time;
|
|
42
|
+
* SQLite never lives on a network share (its locking is unsound there);
|
|
43
|
+
* writes by an agent are proposals into `_incoming/`, never edits to a zone;
|
|
44
|
+
* the `never` list is applied at the mirror *and* the index layer.
|
|
45
|
+
|
|
46
|
+
## Adding a transport
|
|
47
|
+
|
|
48
|
+
Transports implement the small interface in `src/dsh_kb/sources.py`:
|
|
49
|
+
|
|
50
|
+
```python
|
|
51
|
+
def walk(self) -> list[Entry]: ... # the whole tree, one call
|
|
52
|
+
def read(self, rel: str) -> bytes: ...
|
|
53
|
+
def write(self, rel: str, data: bytes) -> None # optional
|
|
54
|
+
def describe(self) -> dict: ... # for `doctor` and `status`
|
|
55
|
+
def ensure_available(self) -> None: ... # raise SourceUnavailable with a sentence a human can act on
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
Two things matter more than the mechanics:
|
|
59
|
+
|
|
60
|
+
* **errors must read as sentences.** `ESTALE` becomes "the share looks stale:
|
|
61
|
+
remount it". A raw traceback is a bug.
|
|
62
|
+
* **writes must create their own parents.** `MKCOL`/`mkdir` do not recurse;
|
|
63
|
+
a bare `PUT` into a missing collection fails with an opaque 500.
|
|
64
|
+
|
|
65
|
+
## Commits
|
|
66
|
+
|
|
67
|
+
English, imperative, a short subject and a body that explains the *why*:
|
|
68
|
+
|
|
69
|
+
```
|
|
70
|
+
fix: create missing parent collections on WebDAV writes
|
|
71
|
+
|
|
72
|
+
A bare PUT into a missing collection is HTTP 500 errno 2, and MKCOL does not
|
|
73
|
+
recurse. A 217-file import failed 217 times before this existed.
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
## Reporting a problem
|
|
77
|
+
|
|
78
|
+
Open an issue with the command you ran, what you expected, and what happened —
|
|
79
|
+
`dsh-kb doctor` covers a lot of ground and its output is usually the fastest
|
|
80
|
+
path to a diagnosis. For anything security-related, see [SECURITY.md](SECURITY.md)
|
|
81
|
+
instead of opening a public issue.
|
dsh_kb-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 dsh-kb contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
dsh_kb-0.1.0/MANIFEST.in
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
# The source distribution should be enough to read the project and run its
|
|
2
|
+
# suite: the docs are part of the deliverable, and the tests are deliberately
|
|
3
|
+
# dependency-free so a reader can verify the claims without installing anything.
|
|
4
|
+
include README.md
|
|
5
|
+
include README.zh.md
|
|
6
|
+
include CHANGELOG.md
|
|
7
|
+
include CONTRIBUTING.md
|
|
8
|
+
include SECURITY.md
|
|
9
|
+
include LICENSE
|
|
10
|
+
include examples/config.json
|
|
11
|
+
include examples/demo.sh
|
|
12
|
+
recursive-include docs *.md
|
|
13
|
+
recursive-include tests *.py
|
dsh_kb-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,375 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: dsh-kb
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: A personal Markdown knowledge base with a local mirror and index, built to be queried cheaply by AI agents.
|
|
5
|
+
Author: dsh-kb contributors
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/LUgABeEEE/dsh-kb
|
|
8
|
+
Project-URL: Documentation, https://github.com/LUgABeEEE/dsh-kb#readme
|
|
9
|
+
Project-URL: Source, https://github.com/LUgABeEEE/dsh-kb
|
|
10
|
+
Project-URL: Issues, https://github.com/LUgABeEEE/dsh-kb/issues
|
|
11
|
+
Project-URL: Changelog, https://github.com/LUgABeEEE/dsh-kb/blob/main/CHANGELOG.md
|
|
12
|
+
Keywords: knowledge-base,markdown,full-text-search,sqlite-fts5,ai-agents,local-first,webdav
|
|
13
|
+
Classifier: Development Status :: 3 - Alpha
|
|
14
|
+
Classifier: Intended Audience :: Developers
|
|
15
|
+
Classifier: Operating System :: OS Independent
|
|
16
|
+
Classifier: Programming Language :: Python :: 3
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
21
|
+
Classifier: Topic :: Text Processing :: Indexing
|
|
22
|
+
Classifier: Topic :: Utilities
|
|
23
|
+
Requires-Python: >=3.10
|
|
24
|
+
Description-Content-Type: text/markdown
|
|
25
|
+
License-File: LICENSE
|
|
26
|
+
Dynamic: license-file
|
|
27
|
+
|
|
28
|
+
# dsh-kb
|
|
29
|
+
|
|
30
|
+
**A personal knowledge base for AI agents: the source of truth lives on shared
|
|
31
|
+
storage, each machine keeps a mirror and an index, retrieval is fast and works
|
|
32
|
+
offline.**
|
|
33
|
+
|
|
34
|
+
English | [中文](README.zh.md)
|
|
35
|
+
|
|
36
|
+
---
|
|
37
|
+
|
|
38
|
+
## Who it is for, what it costs, and how to try it
|
|
39
|
+
|
|
40
|
+
**For** one person (or a small team) whose knowledge already lives in a Markdown
|
|
41
|
+
folder — on a network share, a NAS, or just a directory — and whose agent keeps
|
|
42
|
+
re-reading it. Not a hosted product, not a multi-tenant service, not a
|
|
43
|
+
note-taking UI: one machine, one bundle, one owner.
|
|
44
|
+
|
|
45
|
+
**What it costs**: Python ≥ 3.10 and **no dependencies** — no daemon, no server,
|
|
46
|
+
no database service, no root. One process per command. State (mirror + index)
|
|
47
|
+
lives under `~/.dsh-kb` and runs about 1.9× the size of the bundle; the index is
|
|
48
|
+
local, so a share that is offline or slow costs you freshness, not availability.
|
|
49
|
+
|
|
50
|
+
**Try it in two minutes** — no share and no config file needed:
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
git clone https://github.com/LUgABeEEE/dsh-kb && cd dsh-kb
|
|
54
|
+
bash examples/demo.sh # builds a throwaway bundle, indexes it, searches it
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
---
|
|
58
|
+
|
|
59
|
+
## The problem
|
|
60
|
+
|
|
61
|
+
Personal knowledge — notes, lessons learned, decisions, project records —
|
|
62
|
+
usually ends up scattered across session transcripts, a memory plugin, a pile
|
|
63
|
+
of Markdown reports and a network share. So every agent session does the same
|
|
64
|
+
thing: it **re-reads several sources**, **stuffs raw text into the context
|
|
65
|
+
window** (one measured deployment re-sent ~230 000 tokens of context per
|
|
66
|
+
request), and when it still cannot find the answer it says so.
|
|
67
|
+
|
|
68
|
+
dsh-kb collapses that into one path:
|
|
69
|
+
|
|
70
|
+
```
|
|
71
|
+
one Markdown directory on shared storage ← single source of truth
|
|
72
|
+
│ incremental pull
|
|
73
|
+
▼
|
|
74
|
+
local mirror + local FTS5 index ← milliseconds, works offline
|
|
75
|
+
│ two narrow tools
|
|
76
|
+
▼
|
|
77
|
+
the agent asks for "path + line + 240-char snippet"
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
**The size of a result is independent of the size of the corpus** — 200 pages or
|
|
81
|
+
10 000 pages, a search returns roughly 2.4 KB. That property is the whole point:
|
|
82
|
+
growing the knowledge base does not make each lookup more expensive.
|
|
83
|
+
|
|
84
|
+
---
|
|
85
|
+
|
|
86
|
+
## Core ideas
|
|
87
|
+
|
|
88
|
+
| # | Idea | Why |
|
|
89
|
+
|---|---|---|
|
|
90
|
+
| 1 | **One source of truth, a derived mirror** | The mirror is never an authority, so **merge conflicts cannot happen**; refreshing is a pure function of `(remote listing, manifest)` — interruptible and cheap to run often |
|
|
91
|
+
| 2 | **The index is local and never on the share** | SQLite's locking is not reliable on network filesystems; the index is a disposable derivative that can always be rebuilt |
|
|
92
|
+
| 3 | **Privacy zones compiled into code** | A sensitive zone is **not even copied to disk**, is not indexed, and is not returned by a default query — three layers, no conventions required |
|
|
93
|
+
| 4 | **Proposal-based writes** | An agent only writes to `_incoming/`; promotion is done by a human or a scheduled job — this prevents "a dozen contradictory near-duplicates about one topic" |
|
|
94
|
+
| 5 | **Precision first, recall second** | A query starts as a strict `AND`; only when that is empty does it relax to `OR`, and then **generic terms are dropped first** — otherwise a single shared word such as `UNIQ` drags in unrelated documents |
|
|
95
|
+
|
|
96
|
+
Rationale and measurements: [docs/design.md](docs/design.md).
|
|
97
|
+
|
|
98
|
+
---
|
|
99
|
+
|
|
100
|
+
## Install
|
|
101
|
+
|
|
102
|
+
Standard library only, no third-party dependencies (Python ≥ 3.10):
|
|
103
|
+
|
|
104
|
+
```bash
|
|
105
|
+
pip install -e . # or simply PYTHONPATH=src python3 -m dsh_kb
|
|
106
|
+
dsh-kb --version
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
---
|
|
110
|
+
|
|
111
|
+
## Quick start
|
|
112
|
+
|
|
113
|
+
### 1. Create a bundle (a directory on shared storage)
|
|
114
|
+
|
|
115
|
+
```
|
|
116
|
+
<share>/kb/
|
|
117
|
+
index.md # optional: progressive-disclosure front door
|
|
118
|
+
log.md # optional: change log
|
|
119
|
+
z0-public/ # public zone
|
|
120
|
+
z1-restricted/ # restricted zone
|
|
121
|
+
z2-sensitive/ # sensitive zone (not mirrored, not indexed)
|
|
122
|
+
_incoming/ # proposals (the only place an agent writes)
|
|
123
|
+
_meta/ # conventions, registry, receipts
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
### 2. Configure
|
|
127
|
+
|
|
128
|
+
Configuration lives in `~/.dsh-kb/config.json` — deliberately **outside any
|
|
129
|
+
checkout**, so a public clone never carries your paths or private data:
|
|
130
|
+
|
|
131
|
+
```json
|
|
132
|
+
{
|
|
133
|
+
"source": "auto",
|
|
134
|
+
"mount": "/mnt/nas/kb",
|
|
135
|
+
"local_root": "/srv/kb-share",
|
|
136
|
+
"webdav_root": "https://nas.example.com:5000/webdav/kb",
|
|
137
|
+
"stale_minutes": 15
|
|
138
|
+
}
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
`source=auto` picks the first usable option in the order **mount → local
|
|
142
|
+
directory → Samba (smbclient) → WebDAV**. Walking a local filesystem tree takes
|
|
143
|
+
milliseconds; WebDAV costs one PROPFIND per directory (~55 ms on a consumer
|
|
144
|
+
NAS), so prefer a mount when you have one. Paths may be written POSIX-style, as
|
|
145
|
+
a Windows drive, or as a UNC path (`Z:\kb`, `\\nas\share\kb`, `/mnt/nas/kb`) —
|
|
146
|
+
all are normalised.
|
|
147
|
+
|
|
148
|
+
> **Samba/CIFS mounts need extra care** (we hit all of these in production): a
|
|
149
|
+
> mount can be `rw` and still read-only for the account; a dropped session makes
|
|
150
|
+
> the whole tree raise `ESTALE`; and timestamps have **one-second** granularity.
|
|
151
|
+
> All three are handled explicitly — see
|
|
152
|
+
> [docs/deployment.md](docs/deployment.md#samba--cifs-mounts-linux-and-windows).
|
|
153
|
+
|
|
154
|
+
### 3. Pull, index, search
|
|
155
|
+
|
|
156
|
+
```bash
|
|
157
|
+
dsh-kb doctor # check source reachability and mirror hygiene
|
|
158
|
+
dsh-kb refresh # sync + index — what a scheduler should call
|
|
159
|
+
dsh-kb search "mount stale" -n 5
|
|
160
|
+
dsh-kb search "UNIQ-000123" --json # machine-readable
|
|
161
|
+
dsh-kb read z0-public/mount.md --start 1 --lines 40
|
|
162
|
+
dsh-kb push # upload local _incoming/ proposals
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
**Only `sync`, `push` and `doctor` write to the share** (and `doctor` writes just
|
|
166
|
+
two probe files — see the CLI table). `diff` also *reads* the source listing (to
|
|
167
|
+
report what changed remotely), but it never writes to the share, the mirror or
|
|
168
|
+
the index — it is safe to call from a timer. The share can be offline, a
|
|
169
|
+
certificate can expire, a laptop can be away — search still works. This is a
|
|
170
|
+
hard requirement: otherwise "offline capable" is a lie.
|
|
171
|
+
|
|
172
|
+
---
|
|
173
|
+
|
|
174
|
+
## Privacy model
|
|
175
|
+
|
|
176
|
+
| Zone | Mirrored locally | Indexed | Returned by default |
|
|
177
|
+
|---|---|---|---|
|
|
178
|
+
| `z0-public` | ✅ | ✅ | ✅ |
|
|
179
|
+
| `z1-restricted` | ✅ | ✅ | ✅ (`inject: false`, for injectors to respect) |
|
|
180
|
+
| `z2-sensitive` | ❌ **never copied** | ❌ | ❌ (requires explicit `--zone z2`) |
|
|
181
|
+
| `z3-skills` | ✅ | ❌ **never indexed** (once registered) | ❌ (own entry point: `dsh-kb skills`; 2.3k `SKILL.md` would drown retrieval) |
|
|
182
|
+
| `_meta/`, `_incoming/` | ✅ | ✅ | ✅ (layout, not private content) |
|
|
183
|
+
| paths matching `never` | ❌ | ❌ | ❌ |
|
|
184
|
+
|
|
185
|
+
`never` defaults to `credentials`, `.credentials`, `secrets`, `id_rsa`, `密钥`
|
|
186
|
+
("keys"), `密钥库` and is extendable. All three enforcement points are covered
|
|
187
|
+
by tests.
|
|
188
|
+
|
|
189
|
+
> **`z3-skills` is not built in.** Register it in `config.json` with
|
|
190
|
+
> `"index": false` *before* moving files into it — an unregistered top-level
|
|
191
|
+
> directory is mirrored **and** indexed. See
|
|
192
|
+
> [`examples/config.json`](examples/config.json).
|
|
193
|
+
|
|
194
|
+
---
|
|
195
|
+
|
|
196
|
+
## CLI
|
|
197
|
+
|
|
198
|
+
| Command | Writes? | Purpose |
|
|
199
|
+
|---|---|---|
|
|
200
|
+
| `status` | — | configuration, source, mirror and index state, freshness |
|
|
201
|
+
| `doctor` | ✔ probes | per-check diagnostics, including "does the mirror contain anything it should not" (writes two probe files, see below) |
|
|
202
|
+
| `diff [--incoming-days DAYS] [--json]` | — (reads the source) | read-only health report: remote changes, local mirror gaps, index drift, backlog, conflicts, escapes |
|
|
203
|
+
| `rag-check [--prune] [--force]` | only with `--prune` | vector store ↔ index consistency, page **and chunk** level; report-only (dry-run) by default |
|
|
204
|
+
| `sync [--prune] [--full] [--dry-run]` | ✔ | incremental pull; `--prune` is opt-in; `--full` ignores the manifest |
|
|
205
|
+
| `index` | ✔ (local) | rebuild the index |
|
|
206
|
+
| `refresh [--prune] [--full]` | ✔ | sync + index |
|
|
207
|
+
| `search <q> [-n] [--zone] [--snippet] [--refresh] [--no-auto-refresh] [--json]` | — | ranked search |
|
|
208
|
+
| `read <path> [--start] [--lines] [--json]` | — | line window |
|
|
209
|
+
| `skills search\|show\|stats` | — | query the skill library in the `z3-skills` asset zone |
|
|
210
|
+
| `push [files…] [--force] [--expect SHA256] [--log [N]] [--json]` | ✔ | upload proposals |
|
|
211
|
+
| `mcp` | — | run the MCP server on stdio (for agent hosts) |
|
|
212
|
+
|
|
213
|
+
Shared flags are accepted before or after the subcommand.
|
|
214
|
+
|
|
215
|
+
**Not everything labelled "diagnostic" is read-only.** `doctor` is the one
|
|
216
|
+
exception: it creates and deletes a probe file in `home`, and — when the source
|
|
217
|
+
exposes it — writes and deletes a probe under `_incoming/` on the share, to
|
|
218
|
+
answer "can `push` work at all?". The probes are removed again; nothing else is
|
|
219
|
+
written. Don't wire `doctor` into anything that must be strictly read-only.
|
|
220
|
+
|
|
221
|
+
**Exit codes.** `diff` and `rag-check` share one contract: **0** clean, **10**
|
|
222
|
+
something needs a human, **11** local state is unreadable / the store is broken.
|
|
223
|
+
`rag-check --force` without `--prune` is a *usage* error and exits **1** — it
|
|
224
|
+
would override nothing, so it refuses rather than pretending it forced
|
|
225
|
+
something.
|
|
226
|
+
|
|
227
|
+
### What the two reports actually compare
|
|
228
|
+
|
|
229
|
+
`dsh-kb diff` compares **two pairs, in two directions**, and splits both by file
|
|
230
|
+
kind (`markdown` / `other`) so image or JSON churn can't drown the `.md` diff:
|
|
231
|
+
|
|
232
|
+
* *remote listing vs manifest* → `content`: what changed on the share. **Report
|
|
233
|
+
only, never an alarm** — the next `sync` converges.
|
|
234
|
+
* *manifest vs local mirror* → `mirror_gaps`: what is wrong with **this
|
|
235
|
+
machine's copy**. Covers every file, not just Markdown. A local file that the
|
|
236
|
+
snapshot recorded and the disk lost is `mirror-gap` (exit 10) — it used to be
|
|
237
|
+
invisible, because remote and manifest were both unchanged while `sync` would
|
|
238
|
+
re-fetch it. Extra local files are counted but not alarmed: `sync --prune`
|
|
239
|
+
cannot delete them, so a human has to decide.
|
|
240
|
+
|
|
241
|
+
`dsh-kb rag-check` compares the vector store against the index at **page**
|
|
242
|
+
(`path`) and **chunk** (`path` + `startline`) granularity. The chunk view is
|
|
243
|
+
what catches "one page, five chunks, only two embedded" — page-level counters
|
|
244
|
+
stay at 0 for that shape. On an older store whose tables have no `startline`
|
|
245
|
+
column the comparison degrades to page-level and says so in `note`.
|
|
246
|
+
|
|
247
|
+
---
|
|
248
|
+
|
|
249
|
+
## Configuration reference
|
|
250
|
+
|
|
251
|
+
| Key | Env | Default | Meaning |
|
|
252
|
+
|---|---|---|---|
|
|
253
|
+
| `source` | `DSH_KB_SOURCE` | `auto` | `auto` / `local` / `mount` / `smb` / `webdav` |
|
|
254
|
+
| `mount` | `DSH_KB_MOUNT` | — | mount point (Samba/CIFS, NFS, or a plain directory) |
|
|
255
|
+
| `local_root` | `DSH_KB_LOCAL_ROOT` | — | plain directory |
|
|
256
|
+
| `smb_share` | `DSH_KB_SMB_SHARE` | — | unmounted Samba share, e.g. `//host/share` (needs `smbclient`) |
|
|
257
|
+
| `webdav_root` | `DSH_KB_WEBDAV_ROOT` | — | WebDAV collection URL |
|
|
258
|
+
| `stale_minutes` | `DSH_KB_STALE_MINUTES` | `15` | auto-refresh threshold |
|
|
259
|
+
| `mtime_tolerance` | `DSH_KB_MTIME_TOLERANCE` | `1.0` | mtime delta still counted as "unchanged" |
|
|
260
|
+
| `snippet_chars` | `DSH_KB_SNIPPET_CHARS` | `240` | snippet width |
|
|
261
|
+
| `default_limit` | `DSH_KB_DEFAULT_LIMIT` | `8` | hits per query |
|
|
262
|
+
| `zones` / `never` | — | see above | policy |
|
|
263
|
+
|
|
264
|
+
WebDAV secrets are read **only** from the environment or a credentials file,
|
|
265
|
+
never from `config.json`:
|
|
266
|
+
|
|
267
|
+
```bash
|
|
268
|
+
export DSH_KB_WEBDAV_USER=me
|
|
269
|
+
export DSH_KB_WEBDAV_PASSWORD=... # or DSH_KB_WEBDAV_CREDENTIALS=<key=value file>
|
|
270
|
+
export DSH_KB_WEBDAV_CERT=... DSH_KB_WEBDAV_KEY=... DSH_KB_WEBDAV_CA=... # mutual TLS
|
|
271
|
+
export DSH_KB_WEBDAV_CONNECT=192.0.2.10 # connect to an IP, verify TLS against the CN
|
|
272
|
+
```
|
|
273
|
+
|
|
274
|
+
---
|
|
275
|
+
|
|
276
|
+
## Search behaviour
|
|
277
|
+
|
|
278
|
+
- **CJK**: indexing keeps the original text *and* appends adjacent character
|
|
279
|
+
bigrams, so `挂载` matches `重新挂载` (a 2-character query, which a plain
|
|
280
|
+
`unicode61` tokenizer cannot do).
|
|
281
|
+
- **Latin and identifiers**: the original text is indexed verbatim, so
|
|
282
|
+
`z2-sensitive`, `page-000123.md` and `fts5` all match.
|
|
283
|
+
- **Two passes**: strict `AND` first, then a relaxed `OR` restricted to
|
|
284
|
+
*selective* tokens. Results carry `relaxed: true` and `dropped: [...]` so the
|
|
285
|
+
caller knows precision was traded for recall.
|
|
286
|
+
- **Snippet location** prefers the whole query as a literal, then word-boundary
|
|
287
|
+
token hits — so searching `mount -a` reports that line, not the line holding
|
|
288
|
+
`/proc/mounts`.
|
|
289
|
+
|
|
290
|
+
---
|
|
291
|
+
|
|
292
|
+
## Measured performance
|
|
293
|
+
|
|
294
|
+
Single Linux host, bundled SQLite with FTS5, synthetic corpus (~3.4 KB per page,
|
|
295
|
+
mixed Chinese and English):
|
|
296
|
+
|
|
297
|
+
| Pages | Corpus | Index build | Index size | Blow-up | Engine (worst case) | Exact identifier | CLI end-to-end |
|
|
298
|
+
|---|---|---|---|---|---|---|---|
|
|
299
|
+
| 1 000 | 3.3 MB | 0.38 s | 7.7 MB | 2.3× | — | — | 46 ms |
|
|
300
|
+
| 5 000 | 16.7 MB | 1.6 s | 33.1 MB | 2.0× | — | — | 59 ms |
|
|
301
|
+
| 10 000 | 33.4 MB | 3.2 s | 62.2 MB | **1.9×** | 29.6 ms | **0.2 ms** | **72 ms** |
|
|
302
|
+
|
|
303
|
+
- End-to-end includes ~30 ms of Python interpreter start-up.
|
|
304
|
+
- The spread is large: an exact identifier takes 0.2 ms, a generic keyword that
|
|
305
|
+
matches nearly everything takes 29.6 ms.
|
|
306
|
+
- A search returns about **2.4 KB**, independent of corpus size.
|
|
307
|
+
|
|
308
|
+
The index stays near 1.9× because the FTS5 table stores **no content**
|
|
309
|
+
(`content=''`) and the text lives once in a separate `meta` table, joined back
|
|
310
|
+
by rowid. Inlining the text into FTS5 measured **8.1×**.
|
|
311
|
+
|
|
312
|
+
---
|
|
313
|
+
|
|
314
|
+
## Agent integration
|
|
315
|
+
|
|
316
|
+
**Positioning: a supplement, not standing context.** Nothing is auto-injected and nothing is searched per turn — it is consulted once, when past decisions, environment facts or history are actually needed.
|
|
317
|
+
|
|
318
|
+
Two shapes, one implementation (`dsh_kb.service`), so their output cannot drift:
|
|
319
|
+
|
|
320
|
+
| Shape | When | Cost |
|
|
321
|
+
|---|---|---|
|
|
322
|
+
| **CLI + `--json`** | any agent that can run a command | one subprocess and one tool round-trip (~30 ms) |
|
|
323
|
+
| **MCP server (stdio)** | the host speaks MCP (recommended) | **~420 tokens** standing (1559-char `tools/list` + 125-char instructions, measured), one less layer per call |
|
|
324
|
+
|
|
325
|
+
```bash
|
|
326
|
+
dsh-kb mcp # JSON-RPC on stdin/stdout, exposing search / read / status / skills
|
|
327
|
+
```
|
|
328
|
+
|
|
329
|
+
Wiring it into DSH is one `insert` (other hosts:
|
|
330
|
+
[docs/agent-integration.md](docs/agent-integration.md)):
|
|
331
|
+
|
|
332
|
+
```yaml
|
|
333
|
+
- id: mcp-kb
|
|
334
|
+
name: '@deepseek-ai/dsh-mcp-client'
|
|
335
|
+
config:
|
|
336
|
+
serverName: kb
|
|
337
|
+
transport: stdio
|
|
338
|
+
command: python3
|
|
339
|
+
args: [-m, dsh_kb, mcp, --home, /srv/dsh-kb, --source, auto, --mount, /mnt/nas/kb]
|
|
340
|
+
env: { PYTHONPATH: /opt/dsh-kb/src }
|
|
341
|
+
```
|
|
342
|
+
|
|
343
|
+
> **Why the configuration goes in `args`, not `env`**: some hosts scrub the
|
|
344
|
+
> environment of the MCP child process (credential-shaped names, stale
|
|
345
|
+
> prefixes). Command-line arguments are never scrubbed. Measured: with every
|
|
346
|
+
> `DSH_KB_*` variable removed and only `args` set, it still works against a real
|
|
347
|
+
> share.
|
|
348
|
+
|
|
349
|
+
Design choices — no `mode`/`rerank` knobs, no write tools, failures returned as
|
|
350
|
+
tool results — are in [docs/design.md](docs/design.md#agent-integration).
|
|
351
|
+
|
|
352
|
+
---
|
|
353
|
+
|
|
354
|
+
## Status
|
|
355
|
+
|
|
356
|
+
Alpha (0.1.0) with 308 tests (unit, end-to-end, transport, CLI smoke and MCP — all offline):
|
|
357
|
+
|
|
358
|
+
```bash
|
|
359
|
+
python3 -m unittest discover -s tests -t . -v
|
|
360
|
+
```
|
|
361
|
+
|
|
362
|
+
With the official MCP SDK installed, one extra **interop test** runs (a real
|
|
363
|
+
client against this hand-rolled server); otherwise it skips.
|
|
364
|
+
|
|
365
|
+
## Roadmap
|
|
366
|
+
|
|
367
|
+
- [x] MCP server (`mcp__kb__search` / `read` / `status` / `skills`)
|
|
368
|
+
- [ ] Semantic fallback, used only when keyword search returns nothing (**no vector service**)
|
|
369
|
+
- [ ] Scheduled refresh (systemd timer / cron examples)
|
|
370
|
+
- [ ] Chinese embedding evaluation to pick a default model
|
|
371
|
+
- [ ] Optional read-only HTTP viewer
|
|
372
|
+
|
|
373
|
+
## License
|
|
374
|
+
|
|
375
|
+
MIT — see [LICENSE](LICENSE).
|