dsh-knit 0.18.0 → 0.19.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.en.md CHANGED
@@ -1,17 +1,51 @@
1
1
  # Knit
2
2
 
3
- > Your agent writes 20 documents a day. You can't find the one you just saw.
4
- > Knit puts them next to the conversation — whatever you're talking about, the relevant doc is on top.
3
+ > **Task-aware workspace context retrieval and lifecycle tracking for AI coding agents.**
5
4
  >
6
- > **Your agent has the same problem.** It gets the same ranking as a tool — the ranked documents plus the passage that matched in each.
5
+ > **面向 AI Coding Agent 的任务感知工作区上下文检索与生命周期追踪。**
6
+
7
+ Knit is the layer between **the task you are on** and **your whole workspace**: it finds the most
8
+ relevant documents, source files and media, organises them into **Primary / Supporting / Related**
9
+ context (the Context Pack) for both the panel and the agent, and keeps tracking **whether they were
10
+ read, and whether they changed after being read**.
11
+
12
+ **Find → Organise → Track**, all of it plain string work on your own machine — no model calls, no
13
+ network, no latency, nothing leaves the machine.
14
+
15
+ ```text
16
+ Current task
17
+ │
18
+ Workspace retrieval
19
+ ┌───────────────┼───────────────┐
20
+ ▼ ▼ ▼
21
+ Primary Supporting Related
22
+ └───────────────┼───────────────┘
23
+ ▼
24
+ Context Pack ──► the `knit_docs` tool (for the agent)
25
+ ▼
26
+ Agent reads
27
+ ▼
28
+ Read Evidence · Context Epoch
29
+ ▼
30
+ Workspace changes
31
+ ▼
32
+ Re-read · Lifecycle
33
+ ```
34
+
35
+ > **It is not a smarter recent-files list, and it is not a code browser.**
36
+ > A recent-files list records what you happened to open and what is newest; Knit answers
37
+ > **which things in this project are worth looking at first for the task at hand — and what
38
+ > happened to them after they were read.**
7
39
 
8
- ![Screenshot from a real machine: the Knit sidebar ranking the workspace documents by relevance, with a Markdown preview expanded in place and the references bar below the preview header](https://raw.githubusercontent.com/PolinniZhong/dsh-knit/main/docs/screenshot.png)
40
+ ![Screenshot from a real machine: the Knit sidebar left on the Code tab, ranking the workspace source files by relevance, with a code preview expanded in place](https://raw.githubusercontent.com/PolinniZhong/dsh-knit/main/docs/screenshot.png)
9
41
 
10
- *Screenshot from a real machine, not a mock-up: a workspace with 53 documents — the list, an inline
11
- preview, and the references bar.*
12
- *The references bar is new in v0.12: expand it to see which documents reference this one, and click
13
- one to jump straight there.*
14
- *The header holds only the path, the document count and two toggles; whether the list runs by
42
+ *Screenshot from a real machine, not a mock-up: the same workspace, this one left on the **Code tab** —
43
+ since v0.19 source files are first-class citizens, just like Markdown.*
44
+ *The "why" line under each row says how it matched: a direct hit on the task text, a path hit, or a
45
+ workspace hit.*
46
+ *Click a row to expand the preview **in place**: here `knit/src/host/classification.js`, with line
47
+ numbers; the preview header holds only the path and four actions (copy / full screen / open locally / close).*
48
+ *The header holds only the path, one count and two toggles; whether the list runs by
15
49
  relevance or by newest-first is decided by that pair of tabs on the left.*
16
50
  *Since v0.14 the list is always one column: the rank number sits in its own column on the far left
17
51
  (vertically centred on the title's first line), then Primary dot + title + relative time (time on the
@@ -35,104 +69,128 @@ dsh plugin --profile web add dsh-knit
35
69
 
36
70
  ---
37
71
 
38
- ## "Isn't this just a recent-files list?"
72
+ ## Why it exists
73
+
74
+ > An agent produces 20 documents a day, and you cannot find the one from a minute ago.
39
75
 
40
- **Fair. Two differences:**
76
+ That is the problem it started from: **output got faster than anyone can browse directories.**
77
+ DSH's built-in "recent files" only records what you opened; once a turn ends, the earlier ones sink
78
+ out of sight.
41
79
 
42
- 1. **Scope** — a recent-files list only remembers files you *opened*. Knit scans the
43
- **entire project folder**. Restart DSH, start a new session, come back tomorrow: it's all still there.
44
- 2. **Order** — that list sorts by time. Knit sorts by **what you're currently talking about**.
80
+ Knit asks a different question: **not "what is newest", but "what is relevant to what I am doing".**
81
+ It scans the **whole workspace** (not just what this turn produced) and ranks by what you are
82
+ talking about — restart DSH, open a new session, come back the next day, and it is still there.
45
83
 
46
- Three sentences on what it is:
84
+ Then the problem grew one step further: when you actually change a feature, the first thing worth
85
+ reading is usually not a document but **a few lines of source**. That is what v0.19 added. And after
86
+ that, "ranked first" stopped being enough — **of the things it ranked, did the agent actually read
87
+ them? Did they change afterwards?** v0.15–v0.18 filled in that layer.
47
88
 
48
- 1. Scans **every Markdown file in the project folder**, not just the ones from this turn —
49
- they survive restarts, new sessions, and days.
50
- 2. The order **follows the conversation**: talk about architecture and the architecture doc
51
- floats up; talk about competitors and the competitor analysis floats up.
52
- 3. **No model calls, no network egress**: plain local string matching — zero latency,
53
- zero cost, your documents never leave the machine.
89
+ Three words: **Find** (scan the workspace, rank by the current task) → **Organise** (three tiers,
90
+ each with its reason) → **Track** (read or not, changed or not).
54
91
 
55
92
  ---
56
93
 
57
- ## How "relevant" is computed
94
+ ## It is not a recent-files list, and not a code browser
58
95
 
59
- No magic, just string operations. Three steps:
96
+ Both comparisons catch one **surface** of Knit and miss the question it answers.
60
97
 
61
- **1. Read the conversation.** The host takes the last 6 user/assistant messages, and only
62
- counts real user messages (`agent.inject()` synthetic context would drag the topic off course).
63
- Newer messages weigh more: 3 / 2 / 1 / 1 …
98
+ | | Recent-files list | Code browser | **Knit** |
99
+ |---|---|---|---|
100
+ | Scope | What you happened to open | A directory tree | **The whole workspace**: documents / code / images / video |
101
+ | Order | Modified time | You scroll it yourself | **Relevance to the current task** (BM25 + IDF, computed locally) |
102
+ | Output | One flat list | The one file you opened | **Primary / Supporting / Related**, each saying why it is there |
103
+ | For whom | You | You | You **and** the agent — the same Context Pack, through `knit_docs` |
104
+ | Afterwards | Nothing | Nothing | **Was it read? Did it change after being read?** (Read Evidence + lifecycle) |
64
105
 
65
- **2. Extract keywords.**
106
+ The three differences that matter:
66
107
 
67
- - **ASCII words** — high value, one occurrence is enough (`chokidar`, `mtime`)
68
- - **Chinese 2/3-grams** — either occurring twice, or appearing in the newest message
69
- - **Drop fragments that straddle a word boundary.** Chinese has no word boundaries, so
70
- n-grams glue the last character of one word to the first of the next (「图片和」, 「个插」,
71
- 「的排」). They share one trait — **the first or last character is a pure function word** —
72
- and are dropped. Leave them in and they fill every candidate slot, pushing the real words
73
- (「图片」, 「排序」) out of the query entirely
74
- - stop-word filtering plus greedy de-overlap (picking「相关性排序」drops「相关性」and「排序」)
108
+ 1. **Scope**: a recent-files list only records the files you **opened**; Knit scans **the whole project
109
+ folder** — restart DSH, open a new session, come back days later, and it is still there.
110
+ 2. **Order**: it sorts by time; Knit sorts by **what you are talking about** — discuss architecture and
111
+ the architecture docs rise; discuss a loader bug and `plugin-loader.ts` rises.
112
+ 3. **It does not stop at finding**: after finding comes **organising** (three tiers, each with a
113
+ verifiable reason) and **tracking** (which were really read, and whether they changed).
75
114
 
76
- **3. Score the documents — BM25.**
115
+ **No model calls, no network**: all plain string work, no latency, no cost, nothing leaves the machine.
77
116
 
78
- ```
79
- IDF per term first: rarer in the corpus means more valuable
80
- ln(1 + (N - df + 0.5) / (df + 0.5))
81
- then a weighted sum over fields: title ×4 + summary ×2 + first 2500 chars of body ×1
82
- each field saturated and length-normalised (k1 = 1.2, b = 0.3 / 0.5 / 0.75)
83
- plus a 10% recency nudge (relevance still dominates)
117
+ ---
118
+
119
+ ## Install
120
+
121
+ ```sh
122
+ dsh plugin --profile web add dsh-knit
84
123
  ```
85
124
 
86
- **Why BM25 and not "hits × weight"** (the original approach, since replaced):
125
+ Then **restart DSH** and hard-refresh the browser (`Cmd + Shift + R`).
87
126
 
88
- - with no IDF, a term that appears **everywhere** (the project name) is worth as much as a
89
- rare one — so the frequent term discriminates nothing and dilutes the rare ones
90
- - with no length normalisation, **a long document wins by piling up hits**
91
- - capping hits at 6 was a hand-drawn knee; `k1` / `b` exist precisely for this
127
+ **How to open it:**
92
128
 
93
- Measured on `test/eval/fixture.mjs` (21 cases, both engines on the same corpus):
129
+ - The **Knit icon button in the session header** (right next to the sidebar toggle) — one click
130
+ - Or the right sidebar's tab bar「+」→「Knit recent docs」
94
131
 
95
- | | top-1 | MRR |
96
- |---|---|---|
97
- | old (weighted hits) | 76.2% | 0.830 |
98
- | **BM25** | **95.2%** | **0.976** |
132
+ **Optional**: if `dsh-better-sidebar` is installed, the panel also registers
133
+ as one of its tabs. Each host is an independent optional dependency; missing one doesn't affect the other.
99
134
 
100
- That eval runs inside `npm test`, and the baseline is **recomputed each run** from the old
101
- engine frozen in `test/eval/legacy.mjs` — so "the new engine must be clearly better" is
102
- verified automatically rather than asserted against a hard-coded number.
135
+ **Upgrading**: same command as installing, and the same restart + hard-refresh afterwards.
103
136
 
104
- **Against counting keywords yourself** (`knit/tools/scale-benchmark.mjs`, N = 20/60/180/540):
105
- the corpus is built with a real trap — 12 short, focused topic documents, plus a pile of long
106
- distractors that mention every topic five times without explaining any of them (which is what a
107
- real project's CHANGELOG looks like). Half the topic documents have descriptive filenames, half
108
- are opaque.
137
+ ---
109
138
 
110
- | Approach | Descriptive filenames | Opaque filenames | MRR vs. scale |
111
- |---|---|---|---|
112
- | **Knit (BM25)** | **100%** | **100%** | **1.000 (flat)** |
113
- | `grep -c` keyword counting | 17% | **0%** | 0.313 → **0.089** |
114
- | Filenames only | 100% | **0%** | 0.602 |
139
+ ## Features
115
140
 
116
- Three things: **the ranking beats counting keywords yourself** (so having the agent recompute it
117
- is irrational); **filename matching only works when names describe content**, and Knit is the
118
- only approach that scores 100% either way; and **self-counting degrades as the corpus grows**.
141
+ | | |
142
+ |---|---|
143
+ | **Code Context** (v0.19): source files join the **same** retrieval pipeline as documents — classify, BM25, Context Pack, lifecycle. A dedicated classification layer decides what a file *is* and what it may do, so extension knowledge lives in exactly one place | ✅ |
144
+ | **Sorted by relevance to the current conversation** (BM25 + IDF, fully local, no model) | ✅ |
145
+ | **The `knit_docs` tool for the agent**: the model can look up this project's most relevant documents itself | ✅ |
146
+ | **Document lifecycle** (v0.17): each recommended document is shown as `Unread / Read / Updated after read / Re-read after update`, alongside the most recent read, how many reads fell outside the pack (expandable to which ones), and the latest context change | ✅ |
147
+ | One-click toggle between relevance / modification time (preference kept in localStorage) | ✅ |
148
+ | Scans **documents and code** in the session workspace (recursive, depth ≤ 6, skips `node_modules` / `.git` / `dist` / `build` / `out` / `coverage`) | ✅ |
149
+ | Each row shows H1 title (or filename) + relative time + first-paragraph summary | ✅ |
150
+ | The document list is **always one column** (v0.14 — the multi-column layout was deleted outright, not switched off); the **number sits in its own column on the far left** and is **vertically centred with the title line**, everything else stacks to its right starting with **one line: Primary dot + title + relative time** (the time is pushed to the right edge), then the summary. **The path is no longer shown in the list** — it duplicates the clickable **path** in the preview header, which is the one that stays | ✅ |
151
+ | Click to preview inline, click again to collapse | ✅ |
152
+ | Relative-path images actually render (`./img/a.png`, `../assets/b.png`) | ✅ |
153
+ | One-click switch between **Docs / Code / Media / All** (v0.19 adds Code; remembered, defaults to Docs); the selected tab is an **underline tab**, deliberately different from the list row's grey fill | ✅ |
154
+ | Images & video: square thumbnail grid — cells have a **fixed 104px baseline independent of the item count**, columns come from CSS `repeat(auto-fill, minmax(104px, 1fr))`, and **as many rows as there are items are laid out** (no max height, the grid never scrolls itself — scrolling belongs to the list); videos auto-grab the first frame with a play glyph and duration badge (no deps, no transcoding) | ✅ |
155
+ | **"Web preview" for `.html`** (v0.19): the preview header hands the file to DSH's own document-preview tab (`openResource`) instead of showing source; the inline preview still shows source, so the two do not duplicate. Knit itself sends no request | ✅ |
156
+ | Click an image / video to preview **inline**: large image, playable & seekable video streamed over HTTP Range (no full download) | ✅ |
157
+ | The **All** view splits into **three stacked sections**: docs (max 4, with "View all →" when truncated), **code** (max 4), then images & video (**never truncated**, count only) | ✅ |
158
+ | Preview pane is height-draggable (20%–80%, remembered), fullscreen-able, `Esc` to exit | ✅ |
159
+ | **"Open locally"** opens the document in your default app; the **path in the preview header is clickable** and now opens the system file manager with the file **selected** (`action:'reveal'`), and its tooltip carries the full relative path | ✅ |
160
+ | Double-click opens in a new tab (the official document preview, with its PDF renderer and renderer switching) | ✅ |
161
+ | Filter box over title / summary / path | ✅ |
162
+ | **Click the workspace path** to open the project folder in your file manager | ✅ |
163
+ | **Hover the entry button to peek**: a read-only floating list of the 5 most recent docs; click to open the right sidebar (doesn't push the layout; the popover is just header + list — no extra divider or hint line) | ✅ |
164
+ | Keyboard: `↑` `↓` move-and-preview, `Enter` toggle, `Esc` collapse (`←` `→` span rows in the **media grid only** — the document list is always one column, so they have no spatial meaning there) | ✅ |
165
+ | Auto-refresh every 5s plus a manual button; docs changed in the last 2 min get 🆕 | ✅ |
166
+ | Bilingual (zh/en), follows the DSH language live — no plugin reload needed | ✅ |
167
+ | Zero model calls, zero network egress | ✅ |
168
+ | **A `knit_docs` tool for the agent** — read-only, so the model can find this project's relevant docs itself | ✅ |
119
169
 
120
- **About the "sorted by「xxx」" line**: it shows the **span of your own text** that the matched
121
- terms cover, not the raw candidate tokens. Chinese has no word boundaries, so candidates always
122
- include fragments that straddle two words (「项目文档」 yields `项目文` / `目文档`), and showing
123
- those renders as gibberish — merging their spans and slicing the original text recovers `项目文档`.
124
- The label is **your own wording**, so its casing is preserved (type `BM25`, see `BM25`).
170
+ > Relevance is deliberately **not visualised** (no percentages, no bars) — the ranking itself is
171
+ > the answer; position is relevance.
125
172
 
126
- All of it is string arithmetic — **no embeddings, no model calls**.
173
+ ### The other two views
127
174
 
128
- **And the honest boundaries**:
175
+ ![Screenshot of the Media tab: a square thumbnail grid, six columns at this panel width](https://raw.githubusercontent.com/PolinniZhong/dsh-knit/main/docs/screenshot-media.png)
129
176
 
130
- - **With only one or two messages** there are too few keywords, so it falls back to sorting by
131
- modification time and says so in the panel — it does not pretend to rank
132
- - **With only three to five documents IDF barely does anything**: `df` only takes a few values,
133
- so its dynamic range collapses. The more documents, the better this ranking gets
134
- - **It can only rank documents that share vocabulary with the conversation**: if no term matches,
135
- every document scores the same and the order degrades to time
177
+ *Media: square thumbnails, with the **column count following the panel width continuously** —
178
+ six columns at this width, cells about 110px. These are the workspace's real image and SVG files; this
179
+ particular workspace happens to hold several screenshots, one demo GIF and one single-colour SVG icon
180
+ (the solid black square is that icon, not a failed load). Videos get their first frame as a poster with
181
+ a play glyph and a duration badge — this workspace simply has no video, so none shows here.*
182
+
183
+ ![Screenshot of the All tab: the documents section, the code section, and the media grid below](https://raw.githubusercontent.com/PolinniZhong/dsh-knit/main/docs/screenshot-all.png)
184
+
185
+ *All: the documents section shows at most 4, the **code section** shows at most 4 (each gets its own
186
+ "View all →" when truncated — here "4 / 19" and "4 / 17"), and the media grid comes last.*
187
+
188
+ ⚠️ All three sections share the **same 40-item window** the host returns — a category that ranks past
189
+ 40th by relevance drops out of the All view entirely. This screenshot uses the **Newest** sort: in this
190
+ workspace, under the default Relevance sort, media rank outside that window every time (at `limit=40`
191
+ those 40 rows are 34 documents + 6 code files). The **Media** and **Code** tabs are unaffected — they
192
+ request their own kind. A known defect, with reproduction and the fix recorded in
193
+ [`docs/README.md`](docs/README.md).*
136
194
 
137
195
  ---
138
196
 
@@ -217,6 +275,41 @@ There is **no** "the AI thinks this matters" and **no** percentages, stars or co
217
275
 
218
276
  ---
219
277
 
278
+ ## Code as context (v0.19)
279
+
280
+ Until v0.18 Knit understood Markdown, images and video. v0.19's product concept becomes
281
+ **documents / code / media** — not "a few more extensions", but letting Knit put the source files an
282
+ AI-coding task actually needs into the same Context Pack, on the same main line:
283
+
284
+ ```text
285
+ Current task -> Workspace Retrieval -> Document + Code Candidates
286
+ -> Primary / Supporting / Related -> Context Pack -> Agent Read
287
+ -> Read Evidence -> File Changed -> Lifecycle -> Context Epoch / Delta -> Usage Lens
288
+ ```
289
+
290
+ - **One classification layer, not scattered `if (ext === '.js')`** — `src/host/classification.js`
291
+ turns a filename into `{ kind, language, previewable, searchable, contextual, generated,
292
+ sourceMap }`. The scanner, retrieval, preview and UI all just read that verdict.
293
+ - **Supported in v0.19**: `.js .mjs .cjs .ts .tsx .jsx .py .json .html .htm .css .scss .yaml .yml
294
+ .sh .bash .zsh`. **Not yet**: `.go .rs .java .kt .c .cpp .h .hpp .cs .php .rb .swift .sql`.
295
+ - **Code gets its own field weights** — filename x4, path x2, body (first 8 KB) x1, instead of the
296
+ document weights (title x4 / summary x2 / body x1). A filename like `context-manager.ts` carries
297
+ the task signal.
298
+ - **Generated files are not context**: `.map`, `*.min.js`, `*.min.css`, `*.bundle.js`,
299
+ `*.generated.js`, `*.gen.js`, `*.lock` classify as `generated` — previewable, but never in
300
+ retrieval, the Context Pack or the Usage Lens. When `app.js` and `app.js.map` both exist, the
301
+ `app.js` preview offers one low-key **"Source map"** entry; the `.map` never shows up as a row.
302
+ **Context eligibility is not file-system visibility** — the built-in file tree still opens them.
303
+ - **Bounded and capped**: at most `MAX_CODE = 300` code candidates and a head-limited body read, so
304
+ a large repo cannot flood the Markdown corpus. Corpus pressure is the thing to watch in this
305
+ version, not recall.
306
+ - **The type tabs become four**: `Docs / Code / Media / All`. The Code tab is **still driven by the
307
+ current task** — it is not a file tree of the project.
308
+ - **No second engine**: no second retrieval path, no second lifecycle, no second store, no new
309
+ dependencies, no AST, no LSP, no embedding, no model call, no network, no IDE.
310
+
311
+ ---
312
+
220
313
  ## Also for the agent: the `knit_docs` tool
221
314
 
222
315
  The same ranking that you see in the panel is also exposed to the model.
@@ -254,99 +347,6 @@ Three details:
254
347
 
255
348
  ---
256
349
 
257
- ## Install
258
-
259
- ```sh
260
- dsh plugin --profile web add dsh-knit
261
- ```
262
-
263
- Then **restart DSH** and hard-refresh the browser (`Cmd + Shift + R`).
264
-
265
- **How to open it:**
266
-
267
- - The **Knit icon button in the session header** (right next to the sidebar toggle) — one click
268
- - Or the right sidebar's tab bar「+」→「Knit recent docs」
269
-
270
- **Optional**: if `dsh-better-sidebar` is installed, the panel also registers
271
- as one of its tabs. Each host is an independent optional dependency; missing one doesn't affect the other.
272
-
273
- **Upgrading**: same command as installing, and the same restart + hard-refresh afterwards.
274
-
275
- ---
276
-
277
- ## Feedback
278
-
279
- **Right now this project is not about adding features — it's about finding out whether
280
- "rank project docs by the current conversation" is something anyone actually uses.**
281
- So the most valuable thing you can send is not "could you add X" but **how you work around
282
- this today** — including "I installed it and never opened it", which is more useful than a
283
- feature request.
284
-
285
- - 💬 [Tell us when you actually open it](https://github.com/PolinniZhong/dsh-knit/issues/new?template=feature.yml) — two fields, thirty seconds
286
- - 🐞 [Won't install / panel won't open / ranking looks wrong](https://github.com/PolinniZhong/dsh-knit/issues/new?template=bug.yml)
287
- - 📖 Check [known limitations](#known-limitations) first — short conversations degrade to time order and Chinese uses n-gram approximation; those are deliberate trade-offs, not bugs
288
-
289
- > A single issue is treated as a real signal. So far this plugin has **no trace of a single
290
- > real user** — the npm download count is automated version enumeration, not people
291
- > (only 14% of it is `latest`, and a human install only ever pulls `latest`). One human reply
292
- > changes what gets built next.
293
-
294
- ---
295
-
296
- ## Features
297
-
298
- | | |
299
- |---|---|
300
- | **Sorted by relevance to the current conversation** (BM25 + IDF, fully local, no model) | ✅ |
301
- | **The `knit_docs` tool for the agent**: the model can look up this project's most relevant documents itself | ✅ |
302
- | **Document lifecycle** (v0.17): each recommended document is shown as `Unread / Read / Updated after read / Re-read after update`, alongside the most recent read, how many reads fell outside the pack (expandable to which ones), and the latest context change | ✅ |
303
- | One-click toggle between relevance / modification time (preference kept in localStorage) | ✅ |
304
- | Scans `.md` in the session workspace (recursive, depth ≤ 6, skips `node_modules` / `.git` / `dist`) | ✅ |
305
- | Each row shows H1 title (or filename) + relative time + first-paragraph summary | ✅ |
306
- | The document list is **always one column** (v0.14 — the multi-column layout was deleted outright, not switched off); the **number sits in its own column on the far left** and is **vertically centred with the title line**, everything else stacks to its right starting with **one line: Primary dot + title + relative time** (the time is pushed to the right edge), then the summary. **The path is no longer shown in the list** — it duplicates the clickable **path** in the preview header, which is the one that stays | ✅ |
307
- | Click to preview inline, click again to collapse | ✅ |
308
- | Relative-path images actually render (`./img/a.png`, `../assets/b.png`) | ✅ |
309
- | One-click switch between **Docs / Media / All** (remembered; defaults to Docs, unchanged); the selected tab is a **neutral grey fill**, with no coloured outline (the tab label became "Media" on 2026-10-06, was "Images & video") | ✅ |
310
- | Images & video: square thumbnail grid — **at least 3 columns, more only as the pane gets wider**; cells start at 64px and the baseline is **8 per screen**; past 8 nothing is hidden, the whole grid scales down proportionally; videos auto-grab the first frame with a play glyph and duration badge (no deps, no transcoding) | ✅ |
311
- | Click an image / video to preview **inline**: large image, playable & seekable video streamed over HTTP Range (no full download) | ✅ |
312
- | The **All** view splits into **two stacked sections**: docs (max 4, with "View all →" when truncated) and images & video (**never truncated**, count only) | ✅ |
313
- | Preview pane is height-draggable (20%–80%, remembered), fullscreen-able, `Esc` to exit | ✅ |
314
- | **"Open locally"**: opens the previewed document in your default app (the path in the preview header is clickable too, and its tooltip carries the full relative path) | ✅ |
315
- | Double-click opens in a new tab (the official document preview, with its PDF renderer and renderer switching) | ✅ |
316
- | Filter box over title / summary / path | ✅ |
317
- | **Click the workspace path** to open the project folder in your file manager | ✅ |
318
- | **Hover the entry button to peek**: a read-only floating list of the 5 most recent docs; click to open the right sidebar (doesn't push the layout; the popover is just header + list — no extra divider or hint line) | ✅ |
319
- | Keyboard: `↑` `↓` move-and-preview, `Enter` toggle, `Esc` collapse (`←` `→` span rows in the **media grid only** — the document list is always one column, so they have no spatial meaning there) | ✅ |
320
- | Auto-refresh every 5s plus a manual button; docs changed in the last 2 min get 🆕 | ✅ |
321
- | Bilingual (zh/en), follows the DSH language live — no plugin reload needed | ✅ |
322
- | Zero model calls, zero network egress | ✅ |
323
- | **A `knit_docs` tool for the agent** — read-only, so the model can find this project's relevant docs itself | ✅ |
324
-
325
- > Relevance is deliberately **not visualised** (no percentages, no bars) — the ranking itself is
326
- > the answer; position is relevance.
327
-
328
- ### The other two views
329
-
330
- ![Screenshot of the Media tab: a square thumbnail grid, six columns at this panel width](https://raw.githubusercontent.com/PolinniZhong/dsh-knit/main/docs/screenshot-media.png)
331
-
332
- *Media: square thumbnails, with the **column count following the panel width continuously** —
333
- six columns at this width, cells about 112px. These are the workspace's real image and SVG files; this
334
- particular workspace happens to hold several screenshots, one demo GIF and one single-colour SVG icon
335
- (the solid black square is that icon, not a failed load). Videos get their first frame as a poster with
336
- a play glyph and a duration badge — this workspace simply has no video, so none shows here.*
337
-
338
- ![Screenshot of the All tab: the documents section showing the first 4 with a "View all →" link, and the media grid below](https://raw.githubusercontent.com/PolinniZhong/dsh-knit/main/docs/screenshot-all.png)
339
-
340
- *All: the documents section shows at most 4, with a "View all →" link when truncated (here "4 / 39"),
341
- and the media grid below it.*
342
-
343
- ⚠️ Both sections share the **same 40-item window** the host returns — when media rank past 40th by
344
- relevance, the All view keeps only the few inside that window (this screen keeps just one; the Media
345
- tab is unaffected, because it requests media on its own). A known defect, with reproduction and the
346
- fix recorded in [`docs/README.md`](docs/README.md).*
347
-
348
- ---
349
-
350
350
  ## Did it actually get used? (v0.16 → v0.17)
351
351
 
352
352
  Knit orders "what to read first right now" — but **whether that order was right could only be felt,
@@ -509,10 +509,100 @@ The `knit_docs` tool output **did not grow**: it still only finds context.
509
509
 
510
510
  ---
511
511
 
512
+ ## How retrieval works (implementation, not the pitch)
513
+
514
+ > This section is **evidence**, not the sales pitch: it is what backs the claim "ranked by the
515
+ > current task". The product question is answered on the first screen; this answers "why is it
516
+ > any good at ranking".
517
+
518
+ No magic, just string operations. Three steps:
519
+
520
+ **1. Read the conversation.** The host takes the last 6 user/assistant messages, and only
521
+ counts real user messages (`agent.inject()` synthetic context would drag the topic off course).
522
+ Newer messages weigh more: 3 / 2 / 1 / 1 …
523
+
524
+ **2. Extract keywords.**
525
+
526
+ - **ASCII words** — high value, one occurrence is enough (`chokidar`, `mtime`)
527
+ - **Chinese 2/3-grams** — either occurring twice, or appearing in the newest message
528
+ - **Drop fragments that straddle a word boundary.** Chinese has no word boundaries, so
529
+ n-grams glue the last character of one word to the first of the next (「图片和」, 「个插」,
530
+ 「的排」). They share one trait — **the first or last character is a pure function word** —
531
+ and are dropped. Leave them in and they fill every candidate slot, pushing the real words
532
+ (「图片」, 「排序」) out of the query entirely
533
+ - stop-word filtering plus greedy de-overlap (picking「相关性排序」drops「相关性」and「排序」)
534
+
535
+ **3. Score the documents — BM25.**
536
+
537
+ ```
538
+ IDF per term first: rarer in the corpus means more valuable
539
+ ln(1 + (N - df + 0.5) / (df + 0.5))
540
+ then a weighted sum over fields: title ×4 + summary ×2 + first 2500 chars of body ×1
541
+ each field saturated and length-normalised (k1 = 1.2, b = 0.3 / 0.5 / 0.75)
542
+ plus a 10% recency nudge (relevance still dominates)
543
+ ```
544
+
545
+ **Why BM25 and not "hits × weight"** (the original approach, since replaced):
546
+
547
+ - with no IDF, a term that appears **everywhere** (the project name) is worth as much as a
548
+ rare one — so the frequent term discriminates nothing and dilutes the rare ones
549
+ - with no length normalisation, **a long document wins by piling up hits**
550
+ - capping hits at 6 was a hand-drawn knee; `k1` / `b` exist precisely for this
551
+
552
+ Measured on `test/eval/fixture.mjs` (21 cases, both engines on the same corpus):
553
+
554
+ | | top-1 | MRR |
555
+ |---|---|---|
556
+ | old (weighted hits) | 76.2% | 0.830 |
557
+ | **BM25** | **95.2%** | **0.976** |
558
+
559
+ That eval runs inside `npm test`, and the baseline is **recomputed each run** from the old
560
+ engine frozen in `test/eval/legacy.mjs` — so "the new engine must be clearly better" is
561
+ verified automatically rather than asserted against a hard-coded number.
562
+
563
+ **Against counting keywords yourself** (`knit/tools/scale-benchmark.mjs`, N = 20/60/180/540):
564
+ the corpus is built with a real trap — 12 short, focused topic documents, plus a pile of long
565
+ distractors that mention every topic five times without explaining any of them (which is what a
566
+ real project's CHANGELOG looks like). Half the topic documents have descriptive filenames, half
567
+ are opaque.
568
+
569
+ | Approach | Descriptive filenames | Opaque filenames | MRR vs. scale |
570
+ |---|---|---|---|
571
+ | **Knit (BM25)** | **100%** | **100%** | **1.000 (flat)** |
572
+ | `grep -c` keyword counting | 17% | **0%** | 0.313 → **0.089** |
573
+ | Filenames only | 100% | **0%** | 0.602 |
574
+
575
+ Three things: **the ranking beats counting keywords yourself** (so having the agent recompute it
576
+ is irrational); **filename matching only works when names describe content**, and Knit is the
577
+ only approach that scores 100% either way; and **self-counting degrades as the corpus grows**.
578
+
579
+ **About the "sorted by「xxx」" line**: it shows the **span of your own text** that the matched
580
+ terms cover, not the raw candidate tokens. Chinese has no word boundaries, so candidates always
581
+ include fragments that straddle two words (「项目文档」 yields `项目文` / `目文档`), and showing
582
+ those renders as gibberish — merging their spans and slicing the original text recovers `项目文档`.
583
+ The label is **your own wording**, so its casing is preserved (type `BM25`, see `BM25`).
584
+
585
+ All of it is string arithmetic — **no embeddings, no model calls**.
586
+
587
+ **And the honest boundaries**:
588
+
589
+ - **With only one or two messages** there are too few keywords, so it falls back to sorting by
590
+ modification time and says so in the panel — it does not pretend to rank
591
+ - **With only three to five documents IDF barely does anything**: `df` only takes a few values,
592
+ so its dynamic range collapses. The more documents, the better this ranking gets
593
+ - **It can only rank documents that share vocabulary with the conversation**: if no term matches,
594
+ every document scores the same and the order degrades to time
595
+
596
+ ---
597
+
512
598
  ## What it reads, and what it doesn't
513
599
 
514
- - Scans `.md`, images and video **inside the current session workspace** (anything resolving
515
- outside is rejected); for media it reads metadata only, never the pixels
600
+ - Scans Markdown, **supported code files** (see the v0.19 list above) and media **inside the current
601
+ session workspace** (anything resolving outside is rejected); for media it reads metadata only,
602
+ never the pixels, and for code it reads a **bounded head** of the body, never the whole file
603
+ - **Generated / noisy files are excluded from context** (`.map`, `*.min.*`, `*.bundle.*`,
604
+ `*.generated.*`, `*.lock`): they classify as `generated`, so they never enter retrieval, the
605
+ Context Pack or the Usage Lens — but `/knit/api/doc` still serves them when you open one on purpose
516
606
  - Reads only **the current session's** conversation events (used for ranking)
517
607
  - The `knit_docs` tool is **read-only**: it writes no files and persists no index
518
608
  - v0.15 usage accounting is **off by default**: when enabled it reads only the **current session's
@@ -521,9 +611,10 @@ The `knit_docs` tool output **did not grow**: it still only finds context.
521
611
  the plugin's own same-origin routes
522
612
  - **No install-time scripts** (no `install` / `postinstall`)
523
613
  - **Zero dependencies** — nothing to build, no build-authorisation prompt
524
- - The file-reading route only allows an **image/video-extension allowlist** (images ≤ 12MB,
614
+ - The media route only allows an **image/video-extension allowlist** (images ≤ 12MB,
525
615
  video ≤ 256MB), serves video over HTTP Range, and responds with
526
- `nosniff` plus `default-src 'none'; sandbox`
616
+ `nosniff` plus `default-src 'none'; sandbox`; the **text route serves documents, code and
617
+ generated artifacts**, and refuses media and unknown extensions
527
618
 
528
619
  The relevance figure only affects ordering — it is **never displayed and never sent anywhere**.
529
620
 
@@ -558,13 +649,32 @@ The relevance figure only affects ordering — it is **never displayed and never
558
649
 
559
650
  ---
560
651
 
652
+ ## Feedback
653
+
654
+ **Right now this project is not about adding features — it's about finding out whether
655
+ "rank project docs by the current conversation" is something anyone actually uses.**
656
+ So the most valuable thing you can send is not "could you add X" but **how you work around
657
+ this today** — including "I installed it and never opened it", which is more useful than a
658
+ feature request.
659
+
660
+ - 💬 [Tell us when you actually open it](https://github.com/PolinniZhong/dsh-knit/issues/new?template=feature.yml) — two fields, thirty seconds
661
+ - 🐞 [Won't install / panel won't open / ranking looks wrong](https://github.com/PolinniZhong/dsh-knit/issues/new?template=bug.yml)
662
+ - 📖 Check [known limitations](#known-limitations) first — short conversations degrade to time order and Chinese uses n-gram approximation; those are deliberate trade-offs, not bugs
663
+
664
+ > A single issue is treated as a real signal. So far this plugin has **no trace of a single
665
+ > real user** — the npm download count is automated version enumeration, not people
666
+ > (only 14% of it is `latest`, and a human install only ever pulls `latest`). One human reply
667
+ > changes what gets built next.
668
+
669
+ ---
670
+
561
671
  ## Development
562
672
 
563
673
  ```sh
564
674
  git clone https://github.com/PolinniZhong/dsh-knit.git
565
675
  cd dsh-knit
566
676
 
567
- npm test # 532 tests, zero dependencies, no npm install needed
677
+ npm test # 588 tests, zero dependencies, no npm install needed
568
678
  ```
569
679
 
570
680
  **How changes take effect**: the host half (`src/host/`) **requires a DSH restart** (no hot reload);
@@ -584,11 +694,12 @@ knit/
584
694
  │ ├── host/index.js # /knit/api/recent · /doc · /raw · /links · /context
585
695
  │ ├── host/relevance.js # the relevance engine: BM25 + keyword extraction
586
696
  │ ├── host/links.js # reference parsing (pure; never touches ranking)
697
+ │ ├── host/classification.js # v0.19 file classification: kind / language / searchable / contextual
587
698
  │ ├── host/context.js # context assembly: the three-tier Context Pack (pure, zero I/O)
588
699
  │ ├── host/feedback.js # usage feedback: read-time attribution / Context Epoch / delta (pure)
589
700
  │ ├── host/tool.js # the agent tool knit_docs (hand-written ToolDefinition)
590
701
  │ └── client/client.js # dual-host registration + panel UI
591
- └── test/ # 532 tests
702
+ └── test/ # 588 tests
592
703
  ├── eval/ # retrieval quality: corpus + cases + frozen v0.5.2 baseline
593
704
  └── context/ # context tiering: 24-doc corpus + 12 real-task cases
594
705
  ```