dsh-knit 0.19.0 → 0.20.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.en.md CHANGED
@@ -1,33 +1,67 @@
1
1
  # Knit
2
2
 
3
- > Your agent writes 20 documents a day. You can't find the one you just saw.
4
- > Knit puts them next to the conversation — whatever you're talking about, the relevant doc is on top.
3
+ > **Task-aware workspace context retrieval and lifecycle tracking for AI coding agents.**
5
4
  >
6
- > **Your agent has the same problem.** It gets the same ranking as a tool — the ranked documents plus the passage that matched in each.
5
+ > **面向 AI Coding Agent 的任务感知工作区上下文检索与生命周期追踪。**
7
6
 
8
- ![Screenshot from a real machine: the Knit sidebar left on the Code tab, ranking the workspace source files by relevance, with a code preview expanded in place](https://raw.githubusercontent.com/PolinniZhong/dsh-knit/main/docs/screenshot.png)
7
+ Knit is the layer between **the task you are on** and **your whole workspace**: it finds the most
8
+ relevant documents, source files and media, organises them into **Primary / Supporting / Related**
9
+ context (the Context Pack) for both the panel and the agent, and keeps tracking **whether they were
10
+ read, and whether they changed after being read**.
11
+
12
+ **Find → Organise → Track**, all of it plain string work on your own machine — no model calls, no
13
+ network, no latency, nothing leaves the machine.
14
+
15
+ ```text
16
+ Current task
17
+ │
18
+ Workspace retrieval
19
+ ┌───────────────┼───────────────┐
20
+ ▼ ▼ ▼
21
+ Primary Supporting Related
22
+ └───────────────┼───────────────┘
23
+ ▼
24
+ Context Pack ──► the `knit_docs` tool (for the agent)
25
+ ▼
26
+ Agent reads
27
+ ▼
28
+ Read Evidence · Context Epoch
29
+ ▼
30
+ Workspace changes
31
+ ▼
32
+ Re-read · Lifecycle
33
+ ```
34
+
35
+ > **It is not a smarter recent-files list, and it is not a code browser.**
36
+ > A recent-files list records what you happened to open and what is newest; Knit answers
37
+ > **which things in this project are worth looking at first for the task at hand — and what
38
+ > happened to them after they were read.**
39
+
40
+ ![Screenshot from a real machine: the Knit sidebar left on the Code tab, each row showing its matched passage, with a code preview expanded in place and stopped on that passage](https://raw.githubusercontent.com/PolinniZhong/dsh-knit/main/docs/screenshot.png)
9
41
 
10
42
  *Screenshot from a real machine, not a mock-up: the same workspace, this one left on the **Code tab** —
11
43
  since v0.19 source files are first-class citizens, just like Markdown.*
12
- *The "why" line under each row says how it matched: a direct hit on the task text, a path hit, or a
13
- workspace hit.*
14
- *Click a row to expand the preview **in place**: here `knit/src/host/classification.js`, with line
15
- numbers; the preview header holds only the path and four actions (copy / full screen / open locally / close).*
44
+ *The sub-row under each row says how it matched (a filename hit, a path hit, …) and **which passage**
45
+ of that file matched this query (`命中段 24–38`).*
46
+ *Click a row to expand the preview **in place**, stopped right on that passage: here
47
+ `knit/src/host/passage.js`, line numbers plus syntax highlighting, with the matched lines under a grey
48
+ band; the preview header holds only the path and four actions (copy / full screen / open locally / close).*
16
49
  *The header holds only the path, one count and two toggles; whether the list runs by
17
50
  relevance or by newest-first is decided by that pair of tabs on the left.*
18
51
  *Since v0.14 the list is always one column: the rank number sits in its own column on the far left
19
52
  (vertically centred on the title's first line), then Primary dot + title + relative time (time on the
20
53
  right), then the summary. **The path lives in the preview header only** — the directory collapses to a `…/` placeholder (hover the path for the full relative path); rows no longer repeat it.*
21
54
 
22
- ![Order follows the conversation: send a message and the sidebar re-ranks the document list by it](https://raw.githubusercontent.com/PolinniZhong/dsh-knit/main/docs/demo-reorder.gif)
55
+ ![Order follows the conversation: send a message and the sidebar re-ranks the list by it, each row showing its matched passage](https://raw.githubusercontent.com/PolinniZhong/dsh-knit/main/docs/demo-reorder.gif)
23
56
 
24
- *Order follows the conversation — same workspace, in a freshly opened session: before you have said
25
- anything, the list runs newest-first; ask "how is BM25 weighted for the ranking?" and the next poll
26
- swaps the top of the list, with the tier headings and the **continuous rank numbers across all
27
- three tiers** appearing alongside a one-line "why" under each row (what matched, and how).*
28
- *In this particular frame no document landed in the primary tier, so the groups start at "supporting".*
29
- *⚠️ The panel polls **every 5 seconds**, so the re-rank lands **0–5 s** after the message rather than
30
- instantly, and the GIF runs faster than real time.*
57
+ *Order follows the conversation — same workspace, in a separate session: the conversation had been
58
+ about a release checklist, so the top of the list was that family of documents; ask
59
+ "how is BM25 weighted for the ranking?" and the list swaps over to the BM25 family, keeping the tier
60
+ headings and the **continuous rank numbers across all tiers**, with a one-line "why" under each row
61
+ (what matched, how, and which passage).*
62
+ *⚠️ The panel polls **every 5 seconds**, so the re-rank is not instantaneous: in this take the top rows
63
+ only swapped once that turn had moved along (the tens of seconds in between were cut out).
64
+ The GIF runs faster than real time.*
31
65
 
32
66
  Scans Markdown / images / video across the project folder · Order follows the conversation · No model calls, no network
33
67
 
@@ -37,104 +71,129 @@ dsh plugin --profile web add dsh-knit
37
71
 
38
72
  ---
39
73
 
40
- ## "Isn't this just a recent-files list?"
74
+ ## Why it exists
41
75
 
42
- **Fair. Two differences:**
76
+ > An agent produces 20 documents a day, and you cannot find the one from a minute ago.
43
77
 
44
- 1. **Scope** — a recent-files list only remembers files you *opened*. Knit scans the
45
- **entire project folder**. Restart DSH, start a new session, come back tomorrow: it's all still there.
46
- 2. **Order** — that list sorts by time. Knit sorts by **what you're currently talking about**.
78
+ That is the problem it started from: **output got faster than anyone can browse directories.**
79
+ DSH's built-in "recent files" only records what you opened; once a turn ends, the earlier ones sink
80
+ out of sight.
47
81
 
48
- Three sentences on what it is:
82
+ Knit asks a different question: **not "what is newest", but "what is relevant to what I am doing".**
83
+ It scans the **whole workspace** (not just what this turn produced) and ranks by what you are
84
+ talking about — restart DSH, open a new session, come back the next day, and it is still there.
49
85
 
50
- 1. Scans **every Markdown file in the project folder**, not just the ones from this turn —
51
- they survive restarts, new sessions, and days.
52
- 2. The order **follows the conversation**: talk about architecture and the architecture doc
53
- floats up; talk about competitors and the competitor analysis floats up.
54
- 3. **No model calls, no network egress**: plain local string matching — zero latency,
55
- zero cost, your documents never leave the machine.
86
+ Then the problem grew one step further: when you actually change a feature, the first thing worth
87
+ reading is usually not a document but **a few lines of source**. That is what v0.19 added. And after
88
+ that, "ranked first" stopped being enough — **of the things it ranked, did the agent actually read
89
+ them? Did they change afterwards?** v0.15–v0.18 filled in that layer.
90
+
91
+ Three words: **Find** (scan the workspace, rank by the current task) → **Organise** (three tiers,
92
+ each with its reason) → **Track** (read or not, changed or not).
56
93
 
57
94
  ---
58
95
 
59
- ## How "relevant" is computed
96
+ ## It is not a recent-files list, and not a code browser
60
97
 
61
- No magic, just string operations. Three steps:
98
+ Both comparisons catch one **surface** of Knit and miss the question it answers.
62
99
 
63
- **1. Read the conversation.** The host takes the last 6 user/assistant messages, and only
64
- counts real user messages (`agent.inject()` synthetic context would drag the topic off course).
65
- Newer messages weigh more: 3 / 2 / 1 / 1 …
100
+ | | Recent-files list | Code browser | **Knit** |
101
+ |---|---|---|---|
102
+ | Scope | What you happened to open | A directory tree | **The whole workspace**: documents / code / images / video |
103
+ | Order | Modified time | You scroll it yourself | **Relevance to the current task** (BM25 + IDF, computed locally) |
104
+ | Output | One flat list | The one file you opened | **Primary / Supporting / Related**, each saying why it is there |
105
+ | For whom | You | You | You **and** the agent — the same Context Pack, through `knit_docs` |
106
+ | Afterwards | Nothing | Nothing | **Was it read? Did it change after being read?** (Read Evidence + lifecycle) |
66
107
 
67
- **2. Extract keywords.**
108
+ The three differences that matter:
68
109
 
69
- - **ASCII words** — high value, one occurrence is enough (`chokidar`, `mtime`)
70
- - **Chinese 2/3-grams** — either occurring twice, or appearing in the newest message
71
- - **Drop fragments that straddle a word boundary.** Chinese has no word boundaries, so
72
- n-grams glue the last character of one word to the first of the next (「图片和」, 「个插」,
73
- 「的排」). They share one trait — **the first or last character is a pure function word** —
74
- and are dropped. Leave them in and they fill every candidate slot, pushing the real words
75
- (「图片」, 「排序」) out of the query entirely
76
- - stop-word filtering plus greedy de-overlap (picking「相关性排序」drops「相关性」and「排序」)
110
+ 1. **Scope**: a recent-files list only records the files you **opened**; Knit scans **the whole project
111
+ folder** — restart DSH, open a new session, come back days later, and it is still there.
112
+ 2. **Order**: it sorts by time; Knit sorts by **what you are talking about** — discuss architecture and
113
+ the architecture docs rise; discuss a loader bug and `plugin-loader.ts` rises.
114
+ 3. **It does not stop at finding**: after finding comes **organising** (three tiers, each with a
115
+ verifiable reason) and **tracking** (which were really read, and whether they changed).
77
116
 
78
- **3. Score the documents — BM25.**
117
+ **No model calls, no network**: all plain string work, no latency, no cost, nothing leaves the machine.
79
118
 
80
- ```
81
- IDF per term first: rarer in the corpus means more valuable
82
- ln(1 + (N - df + 0.5) / (df + 0.5))
83
- then a weighted sum over fields: title ×4 + summary ×2 + first 2500 chars of body ×1
84
- each field saturated and length-normalised (k1 = 1.2, b = 0.3 / 0.5 / 0.75)
85
- plus a 10% recency nudge (relevance still dominates)
119
+ ---
120
+
121
+ ## Install
122
+
123
+ ```sh
124
+ dsh plugin --profile web add dsh-knit
86
125
  ```
87
126
 
88
- **Why BM25 and not "hits × weight"** (the original approach, since replaced):
127
+ Then **restart DSH** and hard-refresh the browser (`Cmd + Shift + R`).
89
128
 
90
- - with no IDF, a term that appears **everywhere** (the project name) is worth as much as a
91
- rare one — so the frequent term discriminates nothing and dilutes the rare ones
92
- - with no length normalisation, **a long document wins by piling up hits**
93
- - capping hits at 6 was a hand-drawn knee; `k1` / `b` exist precisely for this
129
+ **How to open it:**
94
130
 
95
- Measured on `test/eval/fixture.mjs` (21 cases, both engines on the same corpus):
131
+ - The **Knit icon button in the session header** (right next to the sidebar toggle) — one click
132
+ - Or the right sidebar's tab bar「+」→「Knit recent docs」
96
133
 
97
- | | top-1 | MRR |
98
- |---|---|---|
99
- | old (weighted hits) | 76.2% | 0.830 |
100
- | **BM25** | **95.2%** | **0.976** |
134
+ **Optional**: if `dsh-better-sidebar` is installed, the panel also registers
135
+ as one of its tabs. Each host is an independent optional dependency; missing one doesn't affect the other.
101
136
 
102
- That eval runs inside `npm test`, and the baseline is **recomputed each run** from the old
103
- engine frozen in `test/eval/legacy.mjs` — so "the new engine must be clearly better" is
104
- verified automatically rather than asserted against a hard-coded number.
137
+ **Upgrading**: same command as installing, and the same restart + hard-refresh afterwards.
105
138
 
106
- **Against counting keywords yourself** (`knit/tools/scale-benchmark.mjs`, N = 20/60/180/540):
107
- the corpus is built with a real trap — 12 short, focused topic documents, plus a pile of long
108
- distractors that mention every topic five times without explaining any of them (which is what a
109
- real project's CHANGELOG looks like). Half the topic documents have descriptive filenames, half
110
- are opaque.
139
+ ---
111
140
 
112
- | Approach | Descriptive filenames | Opaque filenames | MRR vs. scale |
113
- |---|---|---|---|
114
- | **Knit (BM25)** | **100%** | **100%** | **1.000 (flat)** |
115
- | `grep -c` keyword counting | 17% | **0%** | 0.313 → **0.089** |
116
- | Filenames only | 100% | **0%** | 0.602 |
141
+ ## Features
117
142
 
118
- Three things: **the ranking beats counting keywords yourself** (so having the agent recompute it
119
- is irrational); **filename matching only works when names describe content**, and Knit is the
120
- only approach that scores 100% either way; and **self-counting degrades as the corpus grows**.
143
+ | | |
144
+ |---|---|
145
+ | **Code Context** (v0.19): source files join the **same** retrieval pipeline as documents — classify, BM25, Context Pack, lifecycle. A dedicated classification layer decides what a file *is* and what it may do, so extension knowledge lives in exactly one place | ✅ |
146
+ | **Sorted by relevance to the current conversation** (BM25 + IDF, fully local, no model) | ✅ |
147
+ | **The `knit_docs` tool for the agent**: the model can look up this project's most relevant documents itself | ✅ |
148
+ | **Document lifecycle** (v0.17): each recommended document is shown as `Unread / Read / Updated after read / Re-read after update`, alongside the most recent read, how many reads fell outside the pack (expandable to which ones), and the latest context change | ✅ |
149
+ | One-click toggle between relevance / modification time (preference kept in localStorage) | ✅ |
150
+ | Scans **documents and code** in the session workspace (recursive, depth ≤ 6, skips `node_modules` / `.git` / `dist` / `build` / `out` / `coverage`) | ✅ |
151
+ | Each row shows H1 title (or filename) + relative time + first-paragraph summary | ✅ |
152
+ | The document list is **always one column** (v0.14 — the multi-column layout was deleted outright, not switched off); the **number sits in its own column on the far left** and is **vertically centred with the title line**, everything else stacks to its right starting with **one line: Primary dot + title + relative time** (the time is pushed to the right edge), then the summary. **The path is no longer shown in the list** — it duplicates the clickable **path** in the preview header, which is the one that stays | ✅ |
153
+ | Click to preview inline, click again to collapse | ✅ |
154
+ | Relative-path images actually render (`./img/a.png`, `../assets/b.png`) | ✅ |
155
+ | One-click switch between **Docs / Code / Media / All** (v0.19 adds Code; remembered, defaults to Docs); the selected tab is an **underline tab**, deliberately different from the list row's grey fill | ✅ |
156
+ | Images & video: square thumbnail grid — cells have a **fixed 104px baseline independent of the item count**, columns come from CSS `repeat(auto-fill, minmax(104px, 1fr))`, and **as many rows as there are items are laid out** (no max height, the grid never scrolls itself — scrolling belongs to the list); videos auto-grab the first frame with a play glyph and duration badge (no deps, no transcoding) | ✅ |
157
+ | **"Web preview" for `.html`** (v0.19): the preview header hands the file to DSH's own document-preview tab (`openResource`) instead of showing source; the inline preview still shows source, so the two do not duplicate. Knit itself sends no request | ✅ |
158
+ | Click an image / video to preview **inline**: large image, playable & seekable video streamed over HTTP Range (no full download) | ✅ |
159
+ | The **All** view splits into **three stacked sections**: docs (max 4, with "View all →" when truncated), **code** (max 4), then images & video (**never truncated**, count only) | ✅ |
160
+ | Preview pane is height-draggable (20%–80%, remembered), fullscreen-able, `Esc` to exit | ✅ |
161
+ | **"Open locally"** opens the document in your default app; the **path in the preview header is clickable** and now opens the system file manager with the file **selected** (`action:'reveal'`), and its tooltip carries the full relative path | ✅ |
162
+ | Double-click opens in a new tab (the official document preview, with its PDF renderer and renderer switching) | ✅ |
163
+ | Filter box over title / summary / path | ✅ |
164
+ | **Click the workspace path** to open the project folder in your file manager | ✅ |
165
+ | **Hover the entry button to peek**: a read-only floating list of the 5 most recent docs; click to open the right sidebar (doesn't push the layout; the popover is just header + list — no extra divider or hint line) | ✅ |
166
+ | Keyboard: `↑` `↓` move-and-preview, `Enter` toggle, `Esc` collapse (`←` `→` span rows in the **media grid only** — the document list is always one column, so they have no spatial meaning there) | ✅ |
167
+ | Auto-refresh every 5s plus a manual button; docs changed in the last 2 min get 🆕 | ✅ |
168
+ | Bilingual (zh/en), follows the DSH language live — no plugin reload needed | ✅ |
169
+ | Zero model calls, zero network egress | ✅ |
170
+ | **A `knit_docs` tool for the agent** — read-only, so the model can find this project's relevant docs itself | ✅ |
121
171
 
122
- **About the "sorted by「xxx」" line**: it shows the **span of your own text** that the matched
123
- terms cover, not the raw candidate tokens. Chinese has no word boundaries, so candidates always
124
- include fragments that straddle two words (「项目文档」 yields `项目文` / `目文档`), and showing
125
- those renders as gibberish — merging their spans and slicing the original text recovers `项目文档`.
126
- The label is **your own wording**, so its casing is preserved (type `BM25`, see `BM25`).
172
+ > Relevance is deliberately **not visualised** (no percentages, no bars) — the ranking itself is
173
+ > the answer; position is relevance.
127
174
 
128
- All of it is string arithmetic — **no embeddings, no model calls**.
175
+ ### The other two views
129
176
 
130
- **And the honest boundaries**:
177
+ ![Screenshot of the Media tab: a square thumbnail grid, six columns at this panel width](https://raw.githubusercontent.com/PolinniZhong/dsh-knit/main/docs/screenshot-media.png)
131
178
 
132
- - **With only one or two messages** there are too few keywords, so it falls back to sorting by
133
- modification time and says so in the panel — it does not pretend to rank
134
- - **With only three to five documents IDF barely does anything**: `df` only takes a few values,
135
- so its dynamic range collapses. The more documents, the better this ranking gets
136
- - **It can only rank documents that share vocabulary with the conversation**: if no term matches,
137
- every document scores the same and the order degrades to time
179
+ *Media: square thumbnails, with the **column count following the panel width continuously** —
180
+ six columns at this width, cells about 110px. These are the workspace's real image and SVG files
181
+ (18 of them in this take); this particular workspace holds several screenshots, a demo GIF and one
182
+ single-colour SVG icon (the solid black square is that icon, not a failed load). Videos get their first
183
+ frame as a poster with a play glyph and a duration badge — this workspace simply has no video, so none
184
+ shows here.*
185
+
186
+ ![Screenshot of the All tab: the documents section, the code section, and the media grid below](https://raw.githubusercontent.com/PolinniZhong/dsh-knit/main/docs/screenshot-all.png)
187
+
188
+ *All: the documents section shows at most 4, the **code section** shows at most 4 (each gets its own
189
+ "View all →" when truncated — here "4 / 21" and "4 / 8"), and the media grid (11 rows) comes last.*
190
+
191
+ ⚠️ All three sections share the **same 40-item window** the host returns — a category that ranks past
192
+ 40th by relevance drops out of the All view entirely. This screenshot uses the **Newest** sort: in this
193
+ workspace, under the default Relevance sort, media rank outside that window every time (at `limit=40`
194
+ those 40 rows are documents + code files). The **Media** and **Code** tabs are unaffected — they
195
+ request their own kind. A known defect, with reproduction and the fix recorded in
196
+ [`docs/README.md`](docs/README.md).*
138
197
 
139
198
  ---
140
199
 
@@ -219,6 +278,48 @@ There is **no** "the AI thinks this matters" and **no** percentages, stars or co
219
278
 
220
279
  ---
221
280
 
281
+ ## Code as context (v0.19)
282
+
283
+ Until v0.18 Knit understood Markdown, images and video. v0.19's product concept becomes
284
+ **documents / code / media** — not "a few more extensions", but letting Knit put the source files an
285
+ AI-coding task actually needs into the same Context Pack, on the same main line:
286
+
287
+ ```text
288
+ Current task -> Workspace Retrieval -> Document + Code Candidates
289
+ -> Primary / Supporting / Related -> Context Pack -> Agent Read
290
+ -> Read Evidence -> File Changed -> Lifecycle -> Context Epoch / Delta -> Usage Lens
291
+ ```
292
+
293
+ - **One classification layer, not scattered `if (ext === '.js')`** — `src/host/classification.js`
294
+ turns a filename into `{ kind, language, previewable, searchable, contextual, generated,
295
+ sourceMap }`. The scanner, retrieval, preview and UI all just read that verdict.
296
+ - **Supported in v0.19**: `.js .mjs .cjs .ts .tsx .jsx .py .json .html .htm .css .scss .yaml .yml
297
+ .sh .bash .zsh`. **Not yet**: `.go .rs .java .kt .c .cpp .h .hpp .cs .php .rb .swift .sql`.
298
+ - **Code gets its own field weights** — filename x4, path x2, body (first 8 KB) x1, instead of the
299
+ document weights (title x4 / summary x2 / body x1). A filename like `context-manager.ts` carries
300
+ the task signal.
301
+ - **Generated files are not context**: `.map`, `*.min.js`, `*.min.css`, `*.bundle.js`,
302
+ `*.generated.js`, `*.gen.js`, `*.lock` classify as `generated` — previewable, but never in
303
+ retrieval, the Context Pack or the Usage Lens. When `app.js` and `app.js.map` both exist, the
304
+ `app.js` preview offers one low-key **"Source map"** entry; the `.map` never shows up as a row.
305
+ **Context eligibility is not file-system visibility** — the built-in file tree still opens them.
306
+ - **Bounded and capped**: at most `MAX_CODE = 300` code candidates and a head-limited body read, so
307
+ a large repo cannot flood the Markdown corpus. Corpus pressure is the thing to watch in this
308
+ version, not recall.
309
+ - **The type tabs become four**: `Docs / Code / Media / All`. The Code tab is **still driven by the
310
+ current task** — it is not a file tree of the project.
311
+ - **Syntax highlighting is DSH's own (v0.20)** — the client reuses
312
+ `useCodeHighlighter` from `@deepseek-ai/dsh-client-ui-primitives` and takes every colour from the
313
+ official `--shiki-*` tokens, so light and dark match DSH instead of inventing a palette. Only the
314
+ matched stretch is coloured (60 lines either side, window capped at 800) and the rest stays plain
315
+ text — a file with no match gets no colour at all (an un-capped 4k-line file
316
+ costs ~2 s); if the host does not export the hook (older DSH) or the line counts disagree, it falls
317
+ back to plain text silently.
318
+ - **No second engine**: no second retrieval path, no second lifecycle, no second store, no new
319
+ dependencies, no AST, no LSP, no embedding, no model call, no network, no IDE.
320
+
321
+ ---
322
+
222
323
  ## Also for the agent: the `knit_docs` tool
223
324
 
224
325
  The same ranking that you see in the panel is also exposed to the model.
@@ -228,7 +329,8 @@ project are most relevant to what we're discussing?"* and get back **the same Co
228
329
  panel shows — the `primary` / `supporting` / `related` tiers, each item carrying its path, title,
229
330
  summary, a **structured reason** (`direct` / `titleMatch` / `summaryMatch` / `bodyMatch` /
230
331
  `linkTarget` / `linkSource` / `related`), its **project role** (`impl` / `test` / `config` /
231
- `design` / `doc`) and the matching snippet — then open one of them with its own `read` tool.
332
+ `design` / `doc`) and the matching snippet **with its line range** (`match: lines 147–163 — …`) —
333
+ then open one of them with its own `read` tool.
232
334
 
233
335
  **Why it helps**: to cite a document that already exists, an agent can only guess paths or glob
234
336
  and `read` them one by one. Knit has **already computed that ranking** every refresh; this tool
@@ -244,7 +346,12 @@ Three details:
244
346
  - **No relevance score in the result.** It is a *relative* score (something is always 100%, and it
245
347
  may be a different document on the next refresh); shown to a model it reads as absolute confidence.
246
348
  **The order is the relevance** — the same rule the panel follows.
247
- - **No document bodies.** The agent has its own `read` tool; Knit *finds*, it does not *carry*.
349
+ - **The whole file is searched** (since v0.20). Every file is split into structural passages and each
350
+ passage is scored, so "the relevant part is past the first 2,500 characters" no longer means
351
+ "unfindable"; a file over 256 KB is indexed only up to that size, and media has no text — both
352
+ boundaries are stated in the output rather than hidden.
353
+ - **No document bodies.** Only the matched passage (one per file at most, ≤200 characters) — the agent
354
+ has its own `read` tool; Knit *finds*, it does not *carry*.
248
355
  - **No workspace means an error, never a fallback.** The HTTP route falls back to the process cwd
249
356
  when a session can't be resolved (for older clients that send no session id); the tool has no such
250
357
  baggage — falling back would scan an unrelated project and return *its* documents.
@@ -256,103 +363,6 @@ Three details:
256
363
 
257
364
  ---
258
365
 
259
- ## Install
260
-
261
- ```sh
262
- dsh plugin --profile web add dsh-knit
263
- ```
264
-
265
- Then **restart DSH** and hard-refresh the browser (`Cmd + Shift + R`).
266
-
267
- **How to open it:**
268
-
269
- - The **Knit icon button in the session header** (right next to the sidebar toggle) — one click
270
- - Or the right sidebar's tab bar「+」→「Knit recent docs」
271
-
272
- **Optional**: if `dsh-better-sidebar` is installed, the panel also registers
273
- as one of its tabs. Each host is an independent optional dependency; missing one doesn't affect the other.
274
-
275
- **Upgrading**: same command as installing, and the same restart + hard-refresh afterwards.
276
-
277
- ---
278
-
279
- ## Feedback
280
-
281
- **Right now this project is not about adding features — it's about finding out whether
282
- "rank project docs by the current conversation" is something anyone actually uses.**
283
- So the most valuable thing you can send is not "could you add X" but **how you work around
284
- this today** — including "I installed it and never opened it", which is more useful than a
285
- feature request.
286
-
287
- - 💬 [Tell us when you actually open it](https://github.com/PolinniZhong/dsh-knit/issues/new?template=feature.yml) — two fields, thirty seconds
288
- - 🐞 [Won't install / panel won't open / ranking looks wrong](https://github.com/PolinniZhong/dsh-knit/issues/new?template=bug.yml)
289
- - 📖 Check [known limitations](#known-limitations) first — short conversations degrade to time order and Chinese uses n-gram approximation; those are deliberate trade-offs, not bugs
290
-
291
- > A single issue is treated as a real signal. So far this plugin has **no trace of a single
292
- > real user** — the npm download count is automated version enumeration, not people
293
- > (only 14% of it is `latest`, and a human install only ever pulls `latest`). One human reply
294
- > changes what gets built next.
295
-
296
- ---
297
-
298
- ## Features
299
-
300
- | | |
301
- |---|---|
302
- | **Code Context** (v0.19): source files join the **same** retrieval pipeline as documents — classify, BM25, Context Pack, lifecycle. A dedicated classification layer decides what a file *is* and what it may do, so extension knowledge lives in exactly one place | ✅ |
303
- | **Sorted by relevance to the current conversation** (BM25 + IDF, fully local, no model) | ✅ |
304
- | **The `knit_docs` tool for the agent**: the model can look up this project's most relevant documents itself | ✅ |
305
- | **Document lifecycle** (v0.17): each recommended document is shown as `Unread / Read / Updated after read / Re-read after update`, alongside the most recent read, how many reads fell outside the pack (expandable to which ones), and the latest context change | ✅ |
306
- | One-click toggle between relevance / modification time (preference kept in localStorage) | ✅ |
307
- | Scans **documents and code** in the session workspace (recursive, depth ≤ 6, skips `node_modules` / `.git` / `dist` / `build` / `out` / `coverage`) | ✅ |
308
- | Each row shows H1 title (or filename) + relative time + first-paragraph summary | ✅ |
309
- | The document list is **always one column** (v0.14 — the multi-column layout was deleted outright, not switched off); the **number sits in its own column on the far left** and is **vertically centred with the title line**, everything else stacks to its right starting with **one line: Primary dot + title + relative time** (the time is pushed to the right edge), then the summary. **The path is no longer shown in the list** — it duplicates the clickable **path** in the preview header, which is the one that stays | ✅ |
310
- | Click to preview inline, click again to collapse | ✅ |
311
- | Relative-path images actually render (`./img/a.png`, `../assets/b.png`) | ✅ |
312
- | One-click switch between **Docs / Code / Media / All** (v0.19 adds Code; remembered, defaults to Docs); the selected tab is an **underline tab**, deliberately different from the list row's grey fill | ✅ |
313
- | Images & video: square thumbnail grid — cells have a **fixed 104px baseline independent of the item count**, columns come from CSS `repeat(auto-fill, minmax(104px, 1fr))`, and **as many rows as there are items are laid out** (no max height, the grid never scrolls itself — scrolling belongs to the list); videos auto-grab the first frame with a play glyph and duration badge (no deps, no transcoding) | ✅ |
314
- | **"Web preview" for `.html`** (v0.19): the preview header hands the file to DSH's own document-preview tab (`openResource`) instead of showing source; the inline preview still shows source, so the two do not duplicate. Knit itself sends no request | ✅ |
315
- | Click an image / video to preview **inline**: large image, playable & seekable video streamed over HTTP Range (no full download) | ✅ |
316
- | The **All** view splits into **three stacked sections**: docs (max 4, with "View all →" when truncated), **code** (max 4), then images & video (**never truncated**, count only) | ✅ |
317
- | Preview pane is height-draggable (20%–80%, remembered), fullscreen-able, `Esc` to exit | ✅ |
318
- | **"Open locally"** opens the document in your default app; the **path in the preview header is clickable** and now opens the system file manager with the file **selected** (`action:'reveal'`), and its tooltip carries the full relative path | ✅ |
319
- | Double-click opens in a new tab (the official document preview, with its PDF renderer and renderer switching) | ✅ |
320
- | Filter box over title / summary / path | ✅ |
321
- | **Click the workspace path** to open the project folder in your file manager | ✅ |
322
- | **Hover the entry button to peek**: a read-only floating list of the 5 most recent docs; click to open the right sidebar (doesn't push the layout; the popover is just header + list — no extra divider or hint line) | ✅ |
323
- | Keyboard: `↑` `↓` move-and-preview, `Enter` toggle, `Esc` collapse (`←` `→` span rows in the **media grid only** — the document list is always one column, so they have no spatial meaning there) | ✅ |
324
- | Auto-refresh every 5s plus a manual button; docs changed in the last 2 min get 🆕 | ✅ |
325
- | Bilingual (zh/en), follows the DSH language live — no plugin reload needed | ✅ |
326
- | Zero model calls, zero network egress | ✅ |
327
- | **A `knit_docs` tool for the agent** — read-only, so the model can find this project's relevant docs itself | ✅ |
328
-
329
- > Relevance is deliberately **not visualised** (no percentages, no bars) — the ranking itself is
330
- > the answer; position is relevance.
331
-
332
- ### The other two views
333
-
334
- ![Screenshot of the Media tab: a square thumbnail grid, six columns at this panel width](https://raw.githubusercontent.com/PolinniZhong/dsh-knit/main/docs/screenshot-media.png)
335
-
336
- *Media: square thumbnails, with the **column count following the panel width continuously** —
337
- six columns at this width, cells about 110px. These are the workspace's real image and SVG files; this
338
- particular workspace happens to hold several screenshots, one demo GIF and one single-colour SVG icon
339
- (the solid black square is that icon, not a failed load). Videos get their first frame as a poster with
340
- a play glyph and a duration badge — this workspace simply has no video, so none shows here.*
341
-
342
- ![Screenshot of the All tab: the documents section, the code section, and the media grid below](https://raw.githubusercontent.com/PolinniZhong/dsh-knit/main/docs/screenshot-all.png)
343
-
344
- *All: the documents section shows at most 4, the **code section** shows at most 4 (each gets its own
345
- "View all →" when truncated — here "4 / 19" and "4 / 17"), and the media grid comes last.*
346
-
347
- ⚠️ All three sections share the **same 40-item window** the host returns — a category that ranks past
348
- 40th by relevance drops out of the All view entirely. This screenshot uses the **Newest** sort: in this
349
- workspace, under the default Relevance sort, media rank outside that window every time (at `limit=40`
350
- those 40 rows are 34 documents + 6 code files). The **Media** and **Code** tabs are unaffected — they
351
- request their own kind. A known defect, with reproduction and the fix recorded in
352
- [`docs/README.md`](docs/README.md).*
353
-
354
- ---
355
-
356
366
  ## Did it actually get used? (v0.16 → v0.17)
357
367
 
358
368
  Knit orders "what to read first right now" — but **whether that order was right could only be felt,
@@ -515,38 +525,89 @@ The `knit_docs` tool output **did not grow**: it still only finds context.
515
525
 
516
526
  ---
517
527
 
518
- ### v0.19: code as context (Code Context)
528
+ ## How retrieval works (implementation, not the pitch)
519
529
 
520
- Until v0.18 Knit understood Markdown, images and video. v0.19's product concept becomes
521
- **documents / code / media** — not "a few more extensions", but letting Knit put the source files an
522
- AI-coding task actually needs into the same Context Pack, on the same main line:
530
+ > This section is **evidence**, not the sales pitch: it is what backs the claim "ranked by the
531
+ > current task". The product question is answered on the first screen; this answers "why is it
532
+ > any good at ranking".
523
533
 
524
- ```text
525
- Current task -> Workspace Retrieval -> Document + Code Candidates
526
- -> Primary / Supporting / Related -> Context Pack -> Agent Read
527
- -> Read Evidence -> File Changed -> Lifecycle -> Context Epoch / Delta -> Usage Lens
534
+ No magic, just string operations. Three steps:
535
+
536
+ **1. Read the conversation.** The host takes the last 6 user/assistant messages, and only
537
+ counts real user messages (`agent.inject()` synthetic context would drag the topic off course).
538
+ Newer messages weigh more: 3 / 2 / 1 / 1 …
539
+
540
+ **2. Extract keywords.**
541
+
542
+ - **ASCII words** — high value, one occurrence is enough (`chokidar`, `mtime`)
543
+ - **Chinese 2/3-grams** — either occurring twice, or appearing in the newest message
544
+ - **Drop fragments that straddle a word boundary.** Chinese has no word boundaries, so
545
+ n-grams glue the last character of one word to the first of the next (「图片和」, 「个插」,
546
+ 「的排」). They share one trait — **the first or last character is a pure function word** —
547
+ and are dropped. Leave them in and they fill every candidate slot, pushing the real words
548
+ (「图片」, 「排序」) out of the query entirely
549
+ - stop-word filtering plus greedy de-overlap (picking「相关性排序」drops「相关性」and「排序」)
550
+
551
+ **3. Score the documents — BM25.**
552
+
553
+ ```
554
+ IDF per term first: rarer in the corpus means more valuable
555
+ ln(1 + (N - df + 0.5) / (df + 0.5))
556
+ then a weighted sum over fields: title ×4 + summary ×2 + the whole body ×1
557
+ each field saturated and length-normalised (k1 = 1.2, b = 0.3 / 0.5 / 0.75)
558
+ plus a 10% recency nudge (relevance still dominates)
528
559
  ```
529
560
 
530
- - **One classification layer, not scattered `if (ext === '.js')`** — `src/host/classification.js`
531
- turns a filename into `{ kind, language, previewable, searchable, contextual, generated,
532
- sourceMap }`. The scanner, retrieval, preview and UI all just read that verdict.
533
- - **Supported in v0.19**: `.js .mjs .cjs .ts .tsx .jsx .py .json .html .htm .css .scss .yaml .yml
534
- .sh .bash .zsh`. **Not yet**: `.go .rs .java .kt .c .cpp .h .hpp .cs .php .rb .swift .sql`.
535
- - **Code gets its own field weights** — filename x4, path x2, body (first 8 KB) x1, instead of the
536
- document weights (title x4 / summary x2 / body x1). A filename like `context-manager.ts` carries
537
- the task signal.
538
- - **Generated files are not context**: `.map`, `*.min.js`, `*.min.css`, `*.bundle.js`,
539
- `*.generated.js`, `*.gen.js`, `*.lock` classify as `generated` — previewable, but never in
540
- retrieval, the Context Pack or the Usage Lens. When `app.js` and `app.js.map` both exist, the
541
- `app.js` preview offers one low-key **"Source map"** entry; the `.map` never shows up as a row.
542
- **Context eligibility is not file-system visibility** — the built-in file tree still opens them.
543
- - **Bounded and capped**: at most `MAX_CODE = 300` code candidates and a head-limited body read, so
544
- a large repo cannot flood the Markdown corpus. Corpus pressure is the thing to watch in this
545
- version, not recall.
546
- - **The type tabs become four**: `Docs / Code / Media / All`. The Code tab is **still driven by the
547
- current task** — it is not a file tree of the project.
548
- - **No second engine**: no second retrieval path, no second lifecycle, no second store, no new
549
- dependencies, no AST, no LSP, no embedding, no model call, no network, no IDE.
561
+ **Why BM25 and not "hits × weight"** (the original approach, since replaced):
562
+
563
+ - with no IDF, a term that appears **everywhere** (the project name) is worth as much as a
564
+ rare one — so the frequent term discriminates nothing and dilutes the rare ones
565
+ - with no length normalisation, **a long document wins by piling up hits**
566
+ - capping hits at 6 was a hand-drawn knee; `k1` / `b` exist precisely for this
567
+
568
+ Measured on `test/eval/fixture.mjs` (21 cases, both engines on the same corpus):
569
+
570
+ | | top-1 | MRR |
571
+ |---|---|---|
572
+ | old (weighted hits) | 76.2% | 0.830 |
573
+ | **BM25** | **95.2%** | **0.976** |
574
+
575
+ That eval runs inside `npm test`, and the baseline is **recomputed each run** from the old
576
+ engine frozen in `test/eval/legacy.mjs` — so "the new engine must be clearly better" is
577
+ verified automatically rather than asserted against a hard-coded number.
578
+
579
+ **Against counting keywords yourself** (`knit/tools/scale-benchmark.mjs`, N = 20/60/180/540):
580
+ the corpus is built with a real trap — 12 short, focused topic documents, plus a pile of long
581
+ distractors that mention every topic five times without explaining any of them (which is what a
582
+ real project's CHANGELOG looks like). Half the topic documents have descriptive filenames, half
583
+ are opaque.
584
+
585
+ | Approach | Descriptive filenames | Opaque filenames | MRR vs. scale |
586
+ |---|---|---|---|
587
+ | **Knit (BM25)** | **100%** | **100%** | **1.000 (flat)** |
588
+ | `grep -c` keyword counting | 17% | **0%** | 0.313 → **0.089** |
589
+ | Filenames only | 100% | **0%** | 0.602 |
590
+
591
+ Three things: **the ranking beats counting keywords yourself** (so having the agent recompute it
592
+ is irrational); **filename matching only works when names describe content**, and Knit is the
593
+ only approach that scores 100% either way; and **self-counting degrades as the corpus grows**.
594
+
595
+ **About the "sorted by「xxx」" line**: it shows the **span of your own text** that the matched
596
+ terms cover, not the raw candidate tokens. Chinese has no word boundaries, so candidates always
597
+ include fragments that straddle two words (「项目文档」 yields `项目文` / `目文档`), and showing
598
+ those renders as gibberish — merging their spans and slicing the original text recovers `项目文档`.
599
+ The label is **your own wording**, so its casing is preserved (type `BM25`, see `BM25`).
600
+
601
+ All of it is string arithmetic — **no embeddings, no model calls**.
602
+
603
+ **And the honest boundaries**:
604
+
605
+ - **With only one or two messages** there are too few keywords, so it falls back to sorting by
606
+ modification time and says so in the panel — it does not pretend to rank
607
+ - **With only three to five documents IDF barely does anything**: `df` only takes a few values,
608
+ so its dynamic range collapses. The more documents, the better this ranking gets
609
+ - **It can only rank documents that share vocabulary with the conversation**: if no term matches,
610
+ every document scores the same and the order degrades to time
550
611
 
551
612
  ---
552
613
 
@@ -604,13 +665,32 @@ The relevance figure only affects ordering — it is **never displayed and never
604
665
 
605
666
  ---
606
667
 
668
+ ## Feedback
669
+
670
+ **Right now this project is not about adding features — it's about finding out whether
671
+ "rank project docs by the current conversation" is something anyone actually uses.**
672
+ So the most valuable thing you can send is not "could you add X" but **how you work around
673
+ this today** — including "I installed it and never opened it", which is more useful than a
674
+ feature request.
675
+
676
+ - 💬 [Tell us when you actually open it](https://github.com/PolinniZhong/dsh-knit/issues/new?template=feature.yml) — two fields, thirty seconds
677
+ - 🐞 [Won't install / panel won't open / ranking looks wrong](https://github.com/PolinniZhong/dsh-knit/issues/new?template=bug.yml)
678
+ - 📖 Check [known limitations](#known-limitations) first — short conversations degrade to time order and Chinese uses n-gram approximation; those are deliberate trade-offs, not bugs
679
+
680
+ > A single issue is treated as a real signal. So far this plugin has **no trace of a single
681
+ > real user** — the npm download count is automated version enumeration, not people
682
+ > (only 14% of it is `latest`, and a human install only ever pulls `latest`). One human reply
683
+ > changes what gets built next.
684
+
685
+ ---
686
+
607
687
  ## Development
608
688
 
609
689
  ```sh
610
690
  git clone https://github.com/PolinniZhong/dsh-knit.git
611
691
  cd dsh-knit
612
692
 
613
- npm test # 588 tests, zero dependencies, no npm install needed
693
+ npm test # 641 tests, zero dependencies, no npm install needed
614
694
  ```
615
695
 
616
696
  **How changes take effect**: the host half (`src/host/`) **requires a DSH restart** (no hot reload);
@@ -631,11 +711,12 @@ knit/
631
711
  │ ├── host/relevance.js # the relevance engine: BM25 + keyword extraction
632
712
  │ ├── host/links.js # reference parsing (pure; never touches ranking)
633
713
  │ ├── host/classification.js # v0.19 file classification: kind / language / searchable / contextual
714
+ │ ├── host/passage.js # passage layer: chunking / passage scoring / line mapping (pure, zero I/O)
634
715
  │ ├── host/context.js # context assembly: the three-tier Context Pack (pure, zero I/O)
635
716
  │ ├── host/feedback.js # usage feedback: read-time attribution / Context Epoch / delta (pure)
636
717
  │ ├── host/tool.js # the agent tool knit_docs (hand-written ToolDefinition)
637
718
  │ └── client/client.js # dual-host registration + panel UI
638
- └── test/ # 588 tests
719
+ └── test/ # 641 tests
639
720
  ├── eval/ # retrieval quality: corpus + cases + frozen v0.5.2 baseline
640
721
  └── context/ # context tiering: 24-doc corpus + 12 real-task cases
641
722
  ```