dsh-locator 0.7.5 → 0.7.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +17 -0
- package/README-es.md +328 -106
- package/README-hi.md +328 -104
- package/README-pt.md +329 -103
- package/README.en.md +247 -123
- package/README.md +223 -84
- package/lib/content.d.ts +14 -0
- package/lib/content.d.ts.map +1 -1
- package/lib/content.js +62 -1
- package/lib/content.js.map +1 -1
- package/package.json +1 -1
- package/src/content.ts +57 -1
package/README.en.md
CHANGED
|
@@ -2,9 +2,9 @@
|
|
|
2
2
|
|
|
3
3
|
# 🔎 dsh-locator
|
|
4
4
|
|
|
5
|
-
**
|
|
5
|
+
**Search names when you remember the name, contents when you remember the phrase — then open the file or jump back to the conversation.**
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
Whole-disk file search for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) with its **own** index: no Everything install, no `es.exe`, one set of tools and one panel across Windows, macOS and Linux.
|
|
8
8
|
|
|
9
9
|
[](https://www.npmjs.com/package/dsh-locator)
|
|
10
10
|
[](https://www.npmjs.com/package/dsh-locator)
|
|
@@ -12,7 +12,8 @@ A self-built cross-platform index (Windows USN/MFT · macOS FSEvents · Linux in
|
|
|
12
12
|
[](https://github.com/PerryLink/dsh-locator/releases)
|
|
13
13
|
[](LICENSE)
|
|
14
14
|
[](https://github.com/topics/dsh-plugin)
|
|
15
|
-
[](https://github.com/awesome-dsh-plugin/awesome-dsh-plugin)
|
|
16
|
+
[](#version-and-compatibility)
|
|
16
17
|
[](https://gitee.com/perrylink/dsh-locator)
|
|
17
18
|
[](https://gitcode.com/PerryLink/dsh-locator)
|
|
18
19
|
|
|
@@ -24,19 +25,62 @@ dsh plugin --profile web add dsh-locator
|
|
|
24
25
|
|
|
25
26
|
</div>
|
|
26
27
|
|
|
27
|
-
> **Official repository:** [GitHub `PerryLink/dsh-locator`](https://github.com/PerryLink/dsh-locator)
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
28
|
+
> **Official repository:** [GitHub `PerryLink/dsh-locator`](https://github.com/PerryLink/dsh-locator); Gitee and GitCode are automatically synced mirrors, GitHub is the source of truth. Part of the [DSH plugin family](https://github.com/PerryLink) (Apache-2.0) and listed on [awesome-dsh-plugin](https://github.com/awesome-dsh-plugin/awesome-dsh-plugin). A ⭐ helps the next person find it — nothing is gated behind it.
|
|
29
|
+
|
|
30
|
+
## Contents
|
|
31
|
+
|
|
32
|
+
- [At a glance](#at-a-glance)
|
|
33
|
+
- [How it differs from Everything / Spotlight / grep](#how-it-differs-from-everything--spotlight--grep)
|
|
34
|
+
- [Install](#install)
|
|
35
|
+
- [First run](#first-run)
|
|
36
|
+
- [Two searches](#two-searches)
|
|
37
|
+
- [Web panel](#web-panel)
|
|
38
|
+
- [Tools](#tools)
|
|
39
|
+
- [Query language](#query-language)
|
|
40
|
+
- [Configuration](#configuration)
|
|
41
|
+
- [Performance and resources](#performance-and-resources)
|
|
42
|
+
- [Platform support](#platform-support)
|
|
43
|
+
- [Troubleshooting](#troubleshooting)
|
|
44
|
+
- [Build from source](#build-from-source)
|
|
45
|
+
- [Design](#design)
|
|
46
|
+
- [Limitations](#limitations)
|
|
47
|
+
- [Memory](#memory)
|
|
48
|
+
- [Security and privacy](#security-and-privacy)
|
|
49
|
+
- [Version and compatibility](#version-and-compatibility)
|
|
50
|
+
- [Roadmap](#roadmap)
|
|
51
|
+
- [License](#license)
|
|
52
|
+
|
|
53
|
+
## At a glance
|
|
54
|
+
|
|
55
|
+
File lookup has two shapes:
|
|
56
|
+
|
|
57
|
+
- **You remember the name but not the location.** `glob`/`grep` walk the workspace on every call — fine when you know the directory, painful when you remember half a filename.
|
|
58
|
+
- **You remember a sentence, a decision or a snippet, not the file or conversation.** A name index cannot help, and DSH session logs are first-class data too.
|
|
59
|
+
|
|
60
|
+
`dsh-locator` puts both in one plugin:
|
|
61
|
+
|
|
62
|
+
| Capability | How you use it |
|
|
63
|
+
|---|---|
|
|
64
|
+
| File name / path search | `locator_search`, or the panel's default mode |
|
|
65
|
+
| File body search | `locator_find`, or the panel's **Search content** mode |
|
|
66
|
+
| DSH session search | Same tools; hits carry `sessionId` / `messageId`, so the panel can jump back to the conversation |
|
|
67
|
+
| Open / reveal | Tools return paths; the panel double-click reveals, `Ctrl+O` opens, `Ctrl+Enter` inserts a quoted `@"path"` into the composer |
|
|
68
|
+
| External consumption | `locator_export` writes a read-only SQLite mirror |
|
|
32
69
|
|
|
33
|
-
|
|
70
|
+
The index is local: no network calls, no telemetry, no uploads, and the plugin never modifies, moves or deletes indexed files.
|
|
34
71
|
|
|
35
|
-
|
|
72
|
+
## How it differs from Everything / Spotlight / grep
|
|
36
73
|
|
|
37
|
-
|
|
74
|
+
| | dsh-locator | Everything | Spotlight | `glob` / `grep` |
|
|
75
|
+
|---|---|---|---|---|
|
|
76
|
+
| Platforms | Windows / macOS / Linux / WSL | Windows | macOS | Cross-platform |
|
|
77
|
+
| Index | Persistent, self-built | Persistent (NTFS USN) | System metadata/content index | Re-walk per call |
|
|
78
|
+
| Body search | Text files + DSH session messages | Names only by default | Yes, varies by system | Yes, unindexed |
|
|
79
|
+
| Session jump | Hits carry `sessionId` / `messageId` | No | No | No |
|
|
80
|
+
| Agent tools | Six tools, one query language | Needs a CLI/service | Needs glue scripts | Native, no persistent index |
|
|
81
|
+
| Windows speed | Close to Everything (USN optional) | Fastest and lightest | N/A | Slower than an index |
|
|
38
82
|
|
|
39
|
-
|
|
83
|
+
In short: the **idea** behind Everything (persistent index + structured query) is implemented independently and cross-platform, with DSH sessions added on top. It is not a wrapper around Everything.
|
|
40
84
|
|
|
41
85
|
## Install
|
|
42
86
|
|
|
@@ -44,27 +88,32 @@ Everything inspired the *ideas*: a persistent index of file names, a query langu
|
|
|
44
88
|
# npm (recommended)
|
|
45
89
|
dsh plugin --profile web add dsh-locator
|
|
46
90
|
|
|
91
|
+
# pin a version
|
|
92
|
+
dsh plugin --profile web add dsh-locator@0.7.6
|
|
93
|
+
|
|
47
94
|
# GitHub main
|
|
48
95
|
dsh plugin --profile web add github:PerryLink/dsh-locator
|
|
49
96
|
|
|
50
97
|
# from a checkout
|
|
51
98
|
pnpm install
|
|
52
|
-
pnpm build
|
|
53
|
-
pnpm build:engine
|
|
99
|
+
pnpm build
|
|
100
|
+
pnpm build:engine # Go engine for the host platform
|
|
54
101
|
dsh plugin --profile web add /absolute/path/to/dsh-locator
|
|
55
102
|
|
|
56
|
-
# from a packed tarball (
|
|
103
|
+
# from a packed tarball (npm pack output; substitute the real version)
|
|
57
104
|
npm pack
|
|
58
|
-
dsh plugin --profile web add /absolute/path/to/dsh-locator-0.
|
|
105
|
+
dsh plugin --profile web add /absolute/path/to/dsh-locator-0.7.6.tgz
|
|
59
106
|
```
|
|
60
107
|
|
|
61
|
-
The
|
|
62
|
-
|
|
63
|
-
After installing into a Web profile
|
|
108
|
+
- The package ships prebuilt engines for `linux-x64`, `linux-arm64`, `win32-x64`, `win32-arm64`, `darwin-x64` and `darwin-arm64`; no local Go is needed.
|
|
109
|
+
- If your npm mirror lags behind, add `--registry=https://registry.npmjs.org` to the install command.
|
|
110
|
+
- After installing into a **running** Web profile, **reload the page** (`F5`): host tools may mount immediately, but the browser panel is loaded from the boot graph at page load. Headless/TUI profiles get the tools only.
|
|
111
|
+
- Upgrade: `dsh plugin --profile web add dsh-locator@latest`, then `F5`. A content-schema change rebuilds the database once.
|
|
112
|
+
- Uninstall: `dsh plugin --profile web remove dsh-locator`; to keep the config but disable it, set `enabled: false`.
|
|
64
113
|
|
|
65
114
|
## First run
|
|
66
115
|
|
|
67
|
-
On first mount the engine indexes the configured roots (default: your home directory) and writes
|
|
116
|
+
On first mount the engine indexes the configured roots (default: your home directory) and writes the snapshot and content database to the platform cache directory:
|
|
68
117
|
|
|
69
118
|
| Platform | Default state directory |
|
|
70
119
|
|---|---|
|
|
@@ -72,38 +121,73 @@ On first mount the engine indexes the configured roots (default: your home direc
|
|
|
72
121
|
| macOS | `~/Library/Application Support/dsh-locator` |
|
|
73
122
|
| Linux | `$XDG_CACHE_HOME/dsh-locator` or `~/.cache/dsh-locator` |
|
|
74
123
|
|
|
75
|
-
The first scan of a large home directory reads the disk; later starts load the snapshot
|
|
124
|
+
The first scan of a large home directory reads the disk; later starts load the snapshot. Requests are served **during** the first scan: search returns the entries indexed so far with `indexing: true`, and the panel footer says the index is still building.
|
|
125
|
+
|
|
126
|
+
Defaults skip `node_modules`, `.git`, `.hg`, `.svn`, `.cache`, `__pycache__`, `.venv`, `venv` and `.next` by name; `excludes` changes the list.
|
|
127
|
+
|
|
128
|
+
Things to try:
|
|
129
|
+
|
|
130
|
+
```text
|
|
131
|
+
Find the PDFs I changed last week
|
|
132
|
+
Where was "contentRefreshMs" discussed?
|
|
133
|
+
Which conversation settled this design?
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
## Two searches
|
|
137
|
+
|
|
138
|
+
### By name: `locator_search`
|
|
139
|
+
|
|
140
|
+
Everything-style syntax returning path, size, modification time, extension and kind. Suited to “the file name contains report”, “PDFs changed this week”, “videos over 50 MB”.
|
|
141
|
+
|
|
142
|
+
Sorting, pagination and `limit`/`offset` happen in the host engine; common queries over 1.51 M entries return in tens of milliseconds.
|
|
143
|
+
|
|
144
|
+
### By content: `locator_find`
|
|
145
|
+
|
|
146
|
+
Searches the **bodies** of text files and DSH session messages and returns snippets plus their source:
|
|
147
|
+
|
|
148
|
+
- file hits carry a path and can be opened or revealed;
|
|
149
|
+
- session hits carry `sessionId` and `messageId`, so the panel can open the conversation and outline the matched message;
|
|
150
|
+
- Chinese and English substrings both work (trigram tokenizer, no word segmentation needed).
|
|
151
|
+
|
|
152
|
+
How the content index behaves:
|
|
153
|
+
|
|
154
|
+
- **Lazy by default.** Nothing is built at plugin start; the first content search (`locator_find` or the panel's content mode) starts the build in the background, and `contentLazy: false` restores the eager build. The first query reports `building: true` instead of an empty result with no hint.
|
|
155
|
+
- **Restart reuses the database.** An existing database is refreshed incrementally; a full build (session logs first, then file bodies) runs only when the database is missing, has a stale schema, was left unfinished by an interrupted build, or cannot be opened.
|
|
156
|
+
- **Incremental refresh** every `contentRefreshMs` (5 minutes by default) re-reads only files and sessions whose mtime changed; `0` disables it.
|
|
157
|
+
- **Archived and active sessions.** Archived sessions are skipped unless `contentIncludeArchived: true`; a session log that is still being written is indexed after it has been quiet for `contentActiveSessionDelayMs` (2 minutes), so the current log is not decompressed on every refresh. Deleted sessions leave the index when their files disappear.
|
|
158
|
+
- **Budget and truncation.** Defaults: 1 MB per text file, 200,000 files, 1 GB of body bytes, the 100 most recent sessions. Once the byte budget is reached, files later in scan order are skipped; `locator_find` reports `content budget reached` and the HTTP state carries `truncated: true`. Raise `contentMaxTotalBytes` or narrow `contentExcludes`.
|
|
159
|
+
- **Dependency trees are not read.** `node_modules`, `.cargo`, `.bun`, `.pnpm`, `site-packages`, `toolchain`, … are skipped by `contentExcludes` by default — otherwise library code would exhaust the budget before your own documents; the name index still covers them.
|
|
76
160
|
|
|
77
|
-
|
|
161
|
+
The build runs inside the DSH host process, but file reads and zstd decoding are asynchronous and both build loops yield about every 8 ms: on a 28.6 MB corpus the longest event-loop stall dropped from 968 ms to about 20 ms, so session loading and switching stay responsive. Content **queries** run in a worker thread with a read-only connection — common words take roughly 100 ms, rare terms a few milliseconds, and the host is never blocked.
|
|
162
|
+
|
|
163
|
+
## Web panel
|
|
78
164
|
|
|
79
165
|
|
|
80
166
|
|
|
81
|
-
Press `Ctrl+Shift+F` anywhere in the Web GUI, or click the magnifier at the sidebar foot.
|
|
167
|
+
Press `Ctrl+Shift+F` anywhere in the Web GUI, or click the magnifier at the sidebar foot.
|
|
168
|
+
|
|
169
|
+
- Empty query lists the most recently modified entries; typing runs the same Everything-style syntax after a 110 ms debounce.
|
|
170
|
+
- Click a column header to sort by name, size, modification time or path.
|
|
171
|
+
- Toggle **files only** / **directories only** to filter the kind.
|
|
172
|
+
- Tick **Search content** to switch from names to file bodies and session messages; while the index builds the footer shows the state and the panel re-polls.
|
|
173
|
+
- `↑`/`↓` select, `Enter` copies the path, `Ctrl+O` opens, `Ctrl+Enter` inserts a quoted `@"path"` into the composer, `Esc` closes.
|
|
174
|
+
- Double-click a file hit to reveal it in Explorer/Finder/`xdg-open`; double-click a session hit to open the conversation at the matched message.
|
|
175
|
+
- Panel text follows `navigator.language` (zh/en/es/pt/hi).
|
|
82
176
|
|
|
83
|
-
|
|
84
|
-
* Click a column header to sort by name, size, modification time or path; toggle ascending/descending.
|
|
85
|
-
* `只看文件` / `只看目录` filter the kind; the footer reports matches, the page cap and query latency.
|
|
86
|
-
* Tick **Search content** (`搜内容`) to switch from names to file bodies and session messages; hits show a one-line snippet, and a session hit carries its conversation.
|
|
87
|
-
* `↑`/`↓` select, `Enter` copies the selected path, `Ctrl+O` opens it, `Ctrl+Enter` inserts a quoted `@"path"` into the composer, **double-click reveals a file** in Explorer/Finder/`xdg-open`, and **double-click opens a session hit** at the matched message, outlined for a moment, `Esc` closes.
|
|
88
|
-
* The panel follows `navigator.language` (zh/en/es/pt/hi).
|
|
89
|
-
* The panel is optional: profiles without a Web server never mount it, and the three tools keep working.
|
|
177
|
+
The panel talks to same-origin host routes — `/dsh-locator/search`, `/dsh-locator/content`, `/dsh-locator/content/rebuild`, `/dsh-locator/status`, `/dsh-locator/reveal`, `/dsh-locator/open` — so it needs no second server and no extra port. Profiles without a Web server never mount it.
|
|
90
178
|
|
|
91
179
|
## Tools
|
|
92
180
|
|
|
93
181
|
| Tool | What it does |
|
|
94
182
|
|---|---|
|
|
95
|
-
| `locator_search` | Query
|
|
96
|
-
| `locator_find` | Search file bodies and
|
|
97
|
-
| `locator_status` | Roots,
|
|
98
|
-
| `locator_rebuild` | Replace the index with a fresh background scan
|
|
183
|
+
| `locator_search` | Query names and paths, with `ext:`, `size:`, `dm:`, `path:` filters plus sorting and pagination. |
|
|
184
|
+
| `locator_find` | Search file bodies and session messages; hits carry snippets and `sessionId`/`messageId`, plus the content-index state. |
|
|
185
|
+
| `locator_status` | Roots, name-index counts, watcher state, state directory and engine path. (Content progress is reported by `locator_find` and the panel, not here.) |
|
|
186
|
+
| `locator_rebuild` | Replace the name index with a fresh background scan; searches keep using the old index until the new one is ready. |
|
|
99
187
|
| `locator_reindex` | Refresh or rebuild the content index: incremental by default, `full: true` drops and rebuilds it. |
|
|
100
|
-
| `locator_export` | Write a read-only SQLite mirror of the index (`entries` table) for external SQL tools.
|
|
101
|
-
|
|
102
|
-
### Content search
|
|
188
|
+
| `locator_export` | Write a read-only SQLite mirror of the name index (`entries` table) for external SQL tools. |
|
|
103
189
|
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
### Query language
|
|
190
|
+
## Query language
|
|
107
191
|
|
|
108
192
|
Bare words are ANDed substrings of the file name. Everything-style filters combine with them:
|
|
109
193
|
|
|
@@ -125,56 +209,92 @@ Bare words are ANDed substrings of the file name. Everything-style filters combi
|
|
|
125
209
|
|
|
126
210
|
## Configuration
|
|
127
211
|
|
|
128
|
-
Every field is a Schemastery config field;
|
|
212
|
+
Every field is a Schemastery config field; override the `dsh-locator` entry in the profile patch, for example:
|
|
213
|
+
|
|
214
|
+
```yaml
|
|
215
|
+
- id: dsh-locator
|
|
216
|
+
name: dsh-locator
|
|
217
|
+
config:
|
|
218
|
+
roots: ['/home/me/Documents', '/home/me/notes']
|
|
219
|
+
contentMaxTotalBytes: 2000000000
|
|
220
|
+
contentExcludes: ['node_modules', '.git', '.cargo', 'site-packages']
|
|
221
|
+
```
|
|
129
222
|
|
|
130
223
|
| Key | Default | Meaning |
|
|
131
224
|
|---|---|---|
|
|
132
|
-
| `enabled` | `true` |
|
|
133
|
-
| `roots` |
|
|
134
|
-
| `excludes` | dev/cache
|
|
225
|
+
| `enabled` | `true` | Master switch; `false` mounts no tools and starts no engine. |
|
|
226
|
+
| `roots` | home directory | Directories to index; several may be given. |
|
|
227
|
+
| `excludes` | dev/cache names | Base-name globs skipped by the name index. |
|
|
135
228
|
| `watch` | `true` | Apply native filesystem notifications. |
|
|
136
229
|
| `refreshIntervalMs` | `1800000` | Rebuild cadence when no native watcher is active (kqueue fallback or a watcher failure); `0` disables. |
|
|
137
230
|
| `scanWorkers` | `0` | Directory reader concurrency during scans; `0` uses the CPU count. |
|
|
231
|
+
| `windowsUsn` | `false` | Try the NTFS USN/MFT enumeration before the walk (Windows, administrator; falls back automatically). |
|
|
138
232
|
| `memoryLimitMB` | `0` | Soft Go heap limit for the engine in MiB; `0` leaves it to `GOMEMLIMIT`/`GOGC`. |
|
|
139
|
-
| `contentIndex` | `true` | Build the file-body and session-message index at all. |
|
|
140
|
-
| `contentLazy` | `true` | Build the content index on the first content search instead of at plugin start
|
|
233
|
+
| `contentIndex` | `true` | Build the file-body and session-message index at all; `false` leaves name search with near-zero content overhead. |
|
|
234
|
+
| `contentLazy` | `true` | Build the content index on the first content search instead of at plugin start. |
|
|
141
235
|
| `contentMaxFileBytes` | `1000000` | Bytes read from one text file. |
|
|
142
236
|
| `contentMaxTotalBytes` | `1000000000` | Total body bytes kept in the content database; `0` means unlimited. |
|
|
143
237
|
| `contentMaxFiles` | `200000` | Text files indexed per build; `0` means unlimited. |
|
|
144
238
|
| `contentMaxSessions` | `100` | Most recent session logs indexed; `0` means all. |
|
|
145
239
|
| `contentRefreshMs` | `300000` | Incremental content refresh cadence; `0` disables it. |
|
|
146
|
-
| `contentIncludeArchived` | `false` | Index archived sessions as well
|
|
240
|
+
| `contentIncludeArchived` | `false` | Index archived sessions as well. |
|
|
147
241
|
| `contentActiveSessionDelayMs` | `120000` | Only index a session log after it has been quiet this long. |
|
|
148
242
|
| `contentSessionDirs` | `[]` | Extra session directories; empty means `$DSH_HOME/sessions`. |
|
|
149
243
|
| `contentExcludes` | dependency/toolchain names | Path segments or substrings the content index skips; the name index still covers them. |
|
|
150
|
-
| `
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
244
|
+
| `stateDir` | `''` | Snapshot and content database directory; empty means the platform cache directory. Use a separate value per DSH process. |
|
|
245
|
+
|
|
246
|
+
`binaryPath`, `maxResults`, `startupTimeoutMs` and `requestTimeoutMs` are also available for custom engines, page caps and deadlines; the defaults are fine for most setups.
|
|
247
|
+
|
|
248
|
+
## Performance and resources
|
|
249
|
+
|
|
250
|
+
Measured on a WSL2 development machine (Linux, warm page cache) and a real home directory; your machine will differ with disk, file count, cache state and GC.
|
|
251
|
+
|
|
252
|
+
**Name index**
|
|
253
|
+
|
|
254
|
+
| Metric | Result |
|
|
255
|
+
|---|---|
|
|
256
|
+
| Cold scan | ~1.51 M entries in 11–12 s |
|
|
257
|
+
| Snapshot size | ~171 MB for 1.51 M entries |
|
|
258
|
+
| Hot query (`locator_search`) | 6–25 ms for common queries; `path:` prefix scans ~70 ms |
|
|
259
|
+
| Search while a rebuild runs | The old index keeps serving until the new one is ready |
|
|
260
|
+
| Peak scan memory | Hundreds of MB; the engine is a child process and honours `memoryLimitMB` / `GOMEMLIMIT` |
|
|
261
|
+
|
|
262
|
+
**Content index** (real home, 1 GB budget)
|
|
263
|
+
|
|
264
|
+
| Metric | Result (0.7.x, real home) |
|
|
265
|
+
|---|---|
|
|
266
|
+
| Coverage | ~95.8 k text files / 95 sessions / 1,050 messages |
|
|
267
|
+
| Full rebuild | ~3.5 min in the background; host p95 14 ms / max 115 ms during it |
|
|
268
|
+
| Incremental refresh (no change) | ~12 s (dominated by the engine dump of 1.51 M entries) |
|
|
269
|
+
| Idle host latency | p50 2 ms / p95 3 ms / max 6 ms |
|
|
270
|
+
| Database size | ~3.1 GB (about 3× the body budget) |
|
|
271
|
+
| Content query | ~100 ms for common words, a few ms for rare terms (worker thread, host unaffected) |
|
|
156
272
|
|
|
157
|
-
|
|
273
|
+
**Cheaper configurations**
|
|
274
|
+
|
|
275
|
+
- Name search only: `contentIndex: false` drops content work to zero.
|
|
276
|
+
- Keep content but use less disk: lower `contentMaxTotalBytes` (e.g. 256000000) or `contentMaxFileBytes`, or raise `contentRefreshMs`.
|
|
277
|
+
- Cover more files: raise `contentMaxTotalBytes` / `contentMaxFiles` and check that `contentExcludes` does not exclude real document trees.
|
|
278
|
+
- The content database is a cache: deleting `stateDir` rebuilds it and never touches the source files.
|
|
158
279
|
|
|
159
280
|
## Platform support
|
|
160
281
|
|
|
161
282
|
| Platform | Scan + snapshot | Live updates | Notes |
|
|
162
283
|
|---|---|---|---|
|
|
163
284
|
| Linux | yes | inotify, recursive | Watch descriptors are per directory; very large trees can hit `fs.inotify.max_user_watches`. |
|
|
164
|
-
| Windows | yes | `ReadDirectoryChangesW`, recursive | No admin rights and no Everything needed. Optional USN/MFT enumeration when running elevated. Changes made by WSL to `/mnt/<drive>` may not raise notifications;
|
|
165
|
-
| macOS | yes | FSEvents in the shipped cgo binaries, recursive and uncapped | kqueue is the `CGO_ENABLED=0` fallback (up to 4,096 directories)
|
|
166
|
-
|
|
167
|
-
The engine is pure Go except the macOS FSEvents backend, which uses cgo and is built on a macOS runner. Linux, Windows and the kqueue fallback build with `CGO_ENABLED=0`; USN is optional and needs admin rights, everything else does not.
|
|
285
|
+
| Windows | yes | `ReadDirectoryChangesW`, recursive | No admin rights and no Everything needed. Optional USN/MFT enumeration when running elevated. Changes made by WSL to `/mnt/<drive>` may not raise notifications; rebuild those roots. |
|
|
286
|
+
| macOS | yes | FSEvents in the shipped cgo binaries, recursive and uncapped | kqueue is the `CGO_ENABLED=0` fallback (up to 4,096 directories); content changes rely on the periodic rebuild. |
|
|
168
287
|
|
|
169
288
|
### Windows and the USN journal
|
|
170
289
|
|
|
171
290
|
The engine carries a USN decoder, an opt-in fast path and a probe. The path is **off by default** (`windowsUsn: false`) because the volume ioctl needs administrator rights; it has been validated end to end on a GitHub Actions Windows runner (administrator, real NTFS volume):
|
|
291
|
+
|
|
172
292
|
- 1,345,546 MFT records read, 1,345,539 paths resolved
|
|
173
293
|
- 164,209 entries under `C:\Windows` (122,915 files, 41,294 dirs)
|
|
174
294
|
- `scan --usn --root C:\Windows` indexed 164,216 entries in 7.8 s; a query returned in 9.6 ms
|
|
175
295
|
- the change journal was read (8,233 records on one run) and an inactive journal on `D:` fell back to the walk with a warning
|
|
176
296
|
|
|
177
|
-
|
|
297
|
+
Probe it yourself:
|
|
178
298
|
|
|
179
299
|
```powershell
|
|
180
300
|
# Elevated PowerShell (or: Start-Process -Verb RunAs <path> ...)
|
|
@@ -183,41 +303,42 @@ You can still run the same probe yourself:
|
|
|
183
303
|
|
|
184
304
|
It prints the journal id/USNs, enumerates the MFT (records per second, files/dirs below the root, sample paths) and decodes the last few change-journal records. A non-elevated run fails with `Access is denied` — the parser and path reconstruction are covered by unit tests on every platform.
|
|
185
305
|
|
|
186
|
-
With `windowsUsn: true` (or `indexd scan --usn`) the engine enumerates the MFT instead of walking directories, then builds paths from the FRN parent map. A successful USN scan also writes `usn-state.json` (journal id, next USN, roots) and keeps the FRN in each record; on the next `serve --usn` the engine replays the journal from that cursor before answering, so files created while it was stopped appear without a full scan. A wrapped, deleted or mismatched journal falls back to the portable watcher. It keeps the volume handle and polls the change journal every two seconds, applying creates, deletes and renames through `usn.ApplyChanges` (the same FRN map, updated in place). USN records carry **no size or modification time**: results are correct for names and paths, `locator_status` marks such an index with `USN`, and the daemon stats the returned page so the rows
|
|
306
|
+
With `windowsUsn: true` (or `indexd scan --usn`) the engine enumerates the MFT instead of walking directories, then builds paths from the FRN parent map. A successful USN scan also writes `usn-state.json` (journal id, next USN, roots) and keeps the FRN in each record; on the next `serve --usn` the engine replays the journal from that cursor before answering, so files created while it was stopped appear without a full scan. A wrapped, deleted or mismatched journal falls back to the portable watcher. It keeps the volume handle and polls the change journal every two seconds, applying creates, deletes and renames through `usn.ApplyChanges` (the same FRN map, updated in place). USN records carry **no size or modification time**: results are correct for names and paths, `locator_status` marks such an index with `USN`, and the daemon stats the returned page so the rows still show real sizes and dates; `size:`/`dm:` filters over the whole index stay approximate until a metadata pass runs. Any failure (no elevation, unsupported volume, too many records) falls back to the portable walk with a warning; it never disables search.
|
|
187
307
|
|
|
188
308
|
Everything reads the NTFS Master File Table through the USN journal, which is why its first index is nearly instant. `dsh-locator` keeps a portable directory walk as the default (no service, no UAC, any filesystem) and uses USN/MFT as an opt-in fast path where the process holds `GENERIC_READ` (admin), with journal replay across restarts. The portable path stays the fallback.
|
|
189
309
|
|
|
190
|
-
##
|
|
310
|
+
## Troubleshooting
|
|
191
311
|
|
|
192
|
-
|
|
312
|
+
**The panel does not appear after installing.**
|
|
313
|
+
Reload the page (`F5`); confirm you installed into the `web` profile and that the panel is loaded from the boot graph. Headless/TUI profiles never mount a panel.
|
|
193
314
|
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
| Cold scan (uncached I/O) | 50.3 s |
|
|
197
|
-
| Warm scan (page cache, 8 readers) | 4.6–6.5 s (~190–265k entries/s) |
|
|
198
|
-
| Snapshot size | 133 MB |
|
|
199
|
-
| Peak scan RSS | ~490 MB |
|
|
200
|
-
| First search during the initial scan | 2 ms (partial, `indexing: true`) |
|
|
201
|
-
| `locator_search "report"` (981 matches) | 34 ms |
|
|
202
|
-
| `locator_search "package.json"` (1,287 matches) | 25 ms |
|
|
203
|
-
| Query process RSS (1.2 M entries) | ~197 MB |
|
|
204
|
-
| Small tree (39k entries) scan | 0.47 s |
|
|
315
|
+
**A file created a moment ago is not found.**
|
|
316
|
+
Notifications may not have arrived yet (common for WSL writes into `/mnt/c`, network shares and some container overlays). Wait a moment or run `locator_rebuild` for that root. USN mode replays the journal on restart.
|
|
205
317
|
|
|
206
|
-
|
|
318
|
+
**Content search returns nothing.**
|
|
319
|
+
Check `building` in the `locator_find` result: the first search triggers the lazy build and results may be partial until it finishes (the panel re-polls automatically). Also confirm the file extension is a text type, the file is under `contentMaxFileBytes`, it is not excluded by `contentExcludes`, and the session is not archived (archived sessions are skipped by default).
|
|
207
320
|
|
|
208
|
-
|
|
321
|
+
**The content database is huge / disk grows fast.**
|
|
322
|
+
See [Performance and resources](#performance-and-resources): lower the budget or set `contentIndex: false`. Deleting `stateDir` triggers a rebuild.
|
|
209
323
|
|
|
210
|
-
|
|
324
|
+
**An `inotify` warning appears.**
|
|
325
|
+
Too many directories hit the kernel watch limit; the index stays usable and changes rely on periodic rebuilds. Raise `fs.inotify.max_user_watches` or reduce `roots`.
|
|
211
326
|
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
| 20k files / 40 MB of bodies | 15.1 s | 71 MB | 0.17 s | 5 / 17 ms |
|
|
215
|
-
| 200k files / 331 MB of bodies | 31.5 s | 711 MB | 1.3 s | 5 / 156 ms |
|
|
216
|
-
| Real home, 177k documents at the 2 GB cap | 7.4 min | 5.99 GB | 2.9 s | — |
|
|
327
|
+
**Behaviour changed after an upgrade.**
|
|
328
|
+
Run `dsh plugin --profile web add dsh-locator@latest`, then `F5`. A content-schema change rebuilds the database once. See [CHANGELOG](CHANGELOG.md).
|
|
217
329
|
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
330
|
+
## Build from source
|
|
331
|
+
|
|
332
|
+
Requirements: Node `^22.19 || >=24.9`, pnpm, Go 1.22+ for the engine.
|
|
333
|
+
|
|
334
|
+
```sh
|
|
335
|
+
pnpm install
|
|
336
|
+
pnpm build
|
|
337
|
+
pnpm build:engine:all # or build:engine for the host target
|
|
338
|
+
pnpm check # typecheck + tests + packaging/install verification
|
|
339
|
+
```
|
|
340
|
+
|
|
341
|
+
The engine is a separate Go module under `engine/`; run its tests with `cd engine && go test ./...`. The macOS FSEvents backend needs cgo and is built on macOS CI. Node tests run on Linux, Windows and macOS in CI.
|
|
221
342
|
|
|
222
343
|
## Design
|
|
223
344
|
|
|
@@ -226,9 +347,12 @@ Measured on the same machine, warm page cache:
|
|
|
226
347
|
```
|
|
227
348
|
DeepSeek Harness
|
|
228
349
|
└─ dsh-locator (TypeScript plugin)
|
|
229
|
-
├─ locator_search / locator_status / locator_rebuild
|
|
350
|
+
├─ locator_search / locator_find / locator_status / locator_rebuild /
|
|
351
|
+
│ locator_reindex / locator_export
|
|
230
352
|
├─ client.js: Ctrl+Shift+F overlay (shell.overlay slot)
|
|
231
|
-
├─ POST /dsh-locator/{search,status} (same-origin
|
|
353
|
+
├─ POST /dsh-locator/{search,content,status,reveal,open} (same-origin)
|
|
354
|
+
├─ content store: SQLite FTS5 (trigram) in the host process,
|
|
355
|
+
│ queries in a worker thread
|
|
232
356
|
└─ child process: indexd serve (Go, NDJSON over stdio)
|
|
233
357
|
├─ scanner: recursive walk, excludes, progress
|
|
234
358
|
├─ index: names arena + fixed records + open-addressing path table
|
|
@@ -237,56 +361,56 @@ DeepSeek Harness
|
|
|
237
361
|
└─ snapshot: single file, mmap-friendly, atomic rename
|
|
238
362
|
```
|
|
239
363
|
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
364
|
+
- **Parallel scanner.** Directory readers drain a bounded work queue and flush one batch per directory (`AddBatch`), so a parent record exists before its children are enqueued. The queue never blocks: a full queue falls back to scanning that subtree inline.
|
|
365
|
+
- **Memory layout.** Records store a name offset/length into one byte arena plus a parent id; full paths are reconstructed only for the result page and for `path:` filters. That avoids one string header per file.
|
|
366
|
+
- **Path table.** An open-addressing table of `id+1` slots keeps lookup/update near 12 bytes per entry instead of the ~50 bytes a `map[string]int32` costs.
|
|
367
|
+
- **Query engine.** Bare terms and filters are compiled once, matched in parallel chunks, then reduced with a bounded max-heap: the top `offset+limit` rows are selected without sorting a million matches.
|
|
368
|
+
- **Content index.** Host-side SQLite FTS5 with the trigram tokenizer (UTF-8 substrings, CJK); the file list comes from the engine's own scan, session logs are decompressed with Node's zstd.
|
|
369
|
+
- **Content consistency.** A per-process build lock serializes writers sharing a `stateDir`, a completion flag forces a rebuild after an interrupted build, and a stale open handle is detected via device+inode so another process's rebuild cannot leave the host reading an unlinked database.
|
|
370
|
+
- **Persistence.** The snapshot is written to a temporary file and renamed; a generation counter lets the daemon autosave only when the index changed. Tombstones are compacted when deletions exceed a threshold.
|
|
371
|
+
- **Protocol.** One JSON request/response per line on stdio, with unsolicited `scan`, `stats`, `warning`, `error` and `rebuilt` events. No ports, no auth surface, no network listeners.
|
|
372
|
+
- **Read-only.** The engine opens files for metadata only; it never writes to indexed paths.
|
|
247
373
|
|
|
248
|
-
##
|
|
374
|
+
## Limitations
|
|
249
375
|
|
|
250
|
-
|
|
376
|
+
- Content search reads text files (by extension) and DSH session logs only; PDFs, Office documents and images are not text-extracted.
|
|
377
|
+
- The content budget is consumed in scan order, not by file importance; files later in that order are not searchable once the cap is reached (`truncated` is true).
|
|
378
|
+
- The index covers configured roots; it is not a filesystem driver.
|
|
379
|
+
- Linux inotify watches one descriptor per directory; a home directory with hundreds of thousands of directories can exceed the kernel limit. The plugin reports the warning and the snapshot stays usable; rebuilds catch up.
|
|
380
|
+
- WSL writes into `/mnt/c` may not produce Windows notifications (the Windows watcher cannot see the 9p writer's change stream). Rebuild those roots explicitly.
|
|
381
|
+
- Network shares are scanned only if they are reachable as ordinary directories; no SMB-specific change feed.
|
|
382
|
+
- Session content comes from local logs; messages not yet flushed to disk are not searchable.
|
|
383
|
+
- Two DSH processes sharing one `stateDir` should be avoided: the build lock serializes writers but cannot share one open database handle.
|
|
384
|
+
- macOS uses FSEvents (recursive) in the shipped binaries; a `CGO_ENABLED=0` build falls back to kqueue, capped at 4,096 directories, with content changes relying on the periodic rebuild.
|
|
385
|
+
- Windows USN/MFT is optional and only used when the process can read the volume and the journal is active; otherwise the portable walk is used.
|
|
251
386
|
|
|
252
|
-
|
|
253
|
-
pnpm install
|
|
254
|
-
pnpm typecheck
|
|
255
|
-
pnpm test
|
|
256
|
-
pnpm build
|
|
257
|
-
pnpm build:engine:all # or build:engine for the host target
|
|
258
|
-
```
|
|
387
|
+
## Memory
|
|
259
388
|
|
|
260
|
-
The engine is a
|
|
389
|
+
The engine is a single Go process and honours the standard `GOMEMLIMIT` and `GOGC` environment variables (or `memoryLimitMB` in the config). A 1.2–1.5 M-entry index peaks around several hundred MB of RSS with the default GC; set `GOMEMLIMIT=600MiB` (soft limit, the runtime trades CPU for memory) or `GOGC=50` when the host is tight on RAM. Neither changes the snapshot format. In the live setup used for the numbers above, the DSH host process (including the content database page cache) sat around 700 MB RSS with the content index built, and idle host latency stayed at p50 2 ms.
|
|
261
390
|
|
|
262
|
-
##
|
|
391
|
+
## Security and privacy
|
|
392
|
+
|
|
393
|
+
- The index is local: no network calls, no telemetry, no uploads. The plugin never modifies, moves or deletes indexed files.
|
|
394
|
+
- The content database and build lock are created `0600` and the state directory `0700`, so other local users cannot read indexed documents or session text.
|
|
395
|
+
- The snapshot and content database contain file names, paths and bodies; treat the state directory as private data. Deleting it clears the cache.
|
|
396
|
+
- Give concurrent DSH processes separate `stateDir` values.
|
|
263
397
|
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
* macOS uses FSEvents (recursive) in the shipped binaries; a `CGO_ENABLED=0` build falls back to kqueue, capped at 4,096 directories. Content-only changes rely on the periodic rebuild either way.
|
|
270
|
-
* Windows USN/MFT is optional and only used when the process can read the volume and the journal is active; otherwise the portable walk is used.
|
|
398
|
+
## Version and compatibility
|
|
399
|
+
|
|
400
|
+
- Node `^22.19.0 || >=24.9.0`; DSH `>=0.2.1-rc.2 <0.3.0` (see `engines` in `package.json`).
|
|
401
|
+
- Releases are published with GitHub Actions provenance, so the npm package's origin can be verified; Gitee / GitCode are synced by a mirror workflow.
|
|
402
|
+
- Changelog: [CHANGELOG.md](CHANGELOG.md); questions and reports: [Issues](https://github.com/PerryLink/dsh-locator/issues).
|
|
271
403
|
|
|
272
404
|
## Roadmap
|
|
273
405
|
|
|
274
|
-
|
|
406
|
+
Not a commitment, just the current direction:
|
|
407
|
+
|
|
408
|
+
1. Verify the Electron desktop runtime on a real Windows install.
|
|
275
409
|
2. Richer previews and result grouping in the panel.
|
|
276
410
|
3. Windows on ARM / Linux ARM runtime coverage.
|
|
277
|
-
4.
|
|
411
|
+
4. A lowercase-name arena and trigram prefilter for very large indexes.
|
|
278
412
|
5. Incremental snapshot compaction and per-root journals.
|
|
279
413
|
|
|
280
|
-
## Memory
|
|
281
|
-
|
|
282
|
-
The engine is a single Go process and honours the standard `GOMEMLIMIT` and `GOGC` environment variables. A 1.24 M-entry index peaks around 700–800 MB RSS with the default GC; set `GOMEMLIMIT=600MiB` (soft limit, the runtime trades CPU for memory) or `GOGC=50` when the host is tight on RAM. Neither changes the snapshot format.
|
|
283
|
-
|
|
284
|
-
## Security and privacy
|
|
285
|
-
|
|
286
|
-
The content database and build lock are created `0600` and the state directory `0700`, so other local users cannot read indexed documents or session text.
|
|
287
|
-
|
|
288
|
-
The index is local: no network calls, no telemetry, no uploads. The snapshot contains file names and paths, so treat the state directory as private data and exclude it from backups if that matters to you. The plugin never modifies, moves or deletes indexed files.
|
|
289
|
-
|
|
290
414
|
## License
|
|
291
415
|
|
|
292
416
|
Apache-2.0.
|