dsh-locator 0.0.0-stage → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +26 -0
- package/LICENSE +201 -0
- package/README-zh.md +222 -0
- package/README.md +221 -2
- package/SECURITY.md +28 -0
- package/bin/darwin-arm64/indexd +0 -0
- package/bin/darwin-x64/indexd +0 -0
- package/bin/linux-arm64/indexd +0 -0
- package/bin/linux-x64/indexd +0 -0
- package/bin/win32-arm64/indexd.exe +0 -0
- package/bin/win32-x64/indexd.exe +0 -0
- package/client.js +431 -0
- package/cordis.patch.yml +52 -0
- package/lib/config.d.ts +42 -0
- package/lib/config.d.ts.map +1 -0
- package/lib/config.js +31 -0
- package/lib/config.js.map +1 -0
- package/lib/http.d.ts +18 -0
- package/lib/http.d.ts.map +1 -0
- package/lib/http.js +151 -0
- package/lib/http.js.map +1 -0
- package/lib/index.d.ts +20 -0
- package/lib/index.d.ts.map +1 -0
- package/lib/index.js +48 -0
- package/lib/index.js.map +1 -0
- package/lib/paths.d.ts +20 -0
- package/lib/paths.d.ts.map +1 -0
- package/lib/paths.js +56 -0
- package/lib/paths.js.map +1 -0
- package/lib/runtime.d.ts +115 -0
- package/lib/runtime.d.ts.map +1 -0
- package/lib/runtime.js +322 -0
- package/lib/runtime.js.map +1 -0
- package/lib/tools.d.ts +12 -0
- package/lib/tools.d.ts.map +1 -0
- package/lib/tools.js +293 -0
- package/lib/tools.js.map +1 -0
- package/package.json +91 -4
- package/src/config.ts +67 -0
- package/src/export.ts +140 -0
- package/src/http.ts +187 -0
- package/src/index.ts +58 -0
- package/src/paths.ts +61 -0
- package/src/runtime.ts +409 -0
- package/src/tools.ts +326 -0
package/README.md
CHANGED
|
@@ -1,3 +1,222 @@
|
|
|
1
|
-
#
|
|
1
|
+
# dsh-locator
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
[](https://www.npmjs.com/package/dsh-locator)
|
|
4
|
+
[](https://github.com/PerryLink/dsh-locator/actions/workflows/ci.yml)
|
|
5
|
+
|
|
6
|
+
[中文](README-zh.md) | English
|
|
7
|
+
|
|
8
|
+
A whole-disk, millisecond file search for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) that builds its **own** index. No Everything installation, no `es.exe`, no Everything service.
|
|
9
|
+
|
|
10
|
+
`dsh-locator` scans the directories you choose once, keeps the result in a compact snapshot, applies filesystem changes as they happen, and answers name/path queries in tens of milliseconds across a million entries. The agent gets `locator_search`, `locator_status` and `locator_rebuild`; people get a global **search panel** in the Web GUI (`Ctrl+Shift+F` or the magnifier at the sidebar foot) with sortable columns, filters, keyboard navigation and one-key path copy.
|
|
11
|
+
|
|
12
|
+
Everything inspired the *ideas*: a persistent index of file names, a query language with `ext:` / `size:` / `dm:` filters, and an updater that consumes the filesystem's own change stream. The implementation is independent and cross-platform.
|
|
13
|
+
|
|
14
|
+
## Why another file search
|
|
15
|
+
|
|
16
|
+
`glob` and `grep` walk the workspace on every call — fine when you know the directory, slow when you remember only part of a name. Everything is excellent on Windows but is Windows-only, and wrapping it leaves macOS, Linux, WSL and remote hosts uncovered. `dsh-locator` owns the index, so the same tools work everywhere Node and Go run.
|
|
17
|
+
|
|
18
|
+
## Install
|
|
19
|
+
|
|
20
|
+
```sh
|
|
21
|
+
# from npm
|
|
22
|
+
dsh plugin --profile web add dsh-locator
|
|
23
|
+
|
|
24
|
+
# from a checkout
|
|
25
|
+
pnpm install
|
|
26
|
+
pnpm build # TypeScript plugin
|
|
27
|
+
pnpm build:engine # Go engine for the host platform
|
|
28
|
+
dsh plugin --profile web add /absolute/path/to/dsh-locator
|
|
29
|
+
|
|
30
|
+
# from a packed tarball (verified install path)
|
|
31
|
+
npm pack
|
|
32
|
+
dsh plugin --profile web add /absolute/path/to/dsh-locator-0.1.0.tgz
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
The npm package ships prebuilt engines for `linux-x64`, `linux-arm64`, `win32-x64`, `win32-arm64`, `darwin-x64` and `darwin-arm64`.
|
|
36
|
+
|
|
37
|
+
After installing into a Web profile that is already running, **reload the page** (`F5`): the host tools may mount immediately, but the browser panel (`client.js`) is loaded from the boot graph at page load. Headless/TUI profiles never mount the panel.
|
|
38
|
+
|
|
39
|
+
## First run
|
|
40
|
+
|
|
41
|
+
On first mount the engine indexes the configured roots (default: your home directory) and writes a snapshot to the platform cache directory:
|
|
42
|
+
|
|
43
|
+
| Platform | Default state directory |
|
|
44
|
+
|---|---|
|
|
45
|
+
| Windows | `%LOCALAPPDATA%\dsh-locator` |
|
|
46
|
+
| macOS | `~/Library/Application Support/dsh-locator` |
|
|
47
|
+
| Linux | `$XDG_CACHE_HOME/dsh-locator` or `~/.cache/dsh-locator` |
|
|
48
|
+
|
|
49
|
+
The first scan of a large home directory reads the disk; later starts load the snapshot instead. Requests are served **during** that first scan: search returns the entries indexed so far with `indexing: true`, and the panel footer says the index is still building. Nothing blocks on the cold scan. A default exclude list skips `node_modules`, `.git`, `.svn`, `.hg`, `.cache`, `__pycache__`, `.venv`, `venv` and `.next`.
|
|
50
|
+
|
|
51
|
+
## Web search panel
|
|
52
|
+
|
|
53
|
+
Press `Ctrl+Shift+F` anywhere in the Web GUI, or click the magnifier at the sidebar foot. The overlay queries the same index through two same-origin host routes (`POST /dsh-locator/search`, `POST /dsh-locator/status`), so it needs no second server and no extra port.
|
|
54
|
+
|
|
55
|
+
* Empty query lists the most recently modified entries; typing runs the same Everything-style syntax as the tool after a 110 ms debounce.
|
|
56
|
+
* Click a column header to sort by name, size, modification time or path; toggle ascending/descending.
|
|
57
|
+
* `只看文件` / `只看目录` filter the kind; the footer reports matches, the page cap and query latency.
|
|
58
|
+
* `↑`/`↓` select, `Enter` copies the selected path, `Esc` closes.
|
|
59
|
+
* The panel is optional: profiles without a Web server never mount it, and the three tools keep working.
|
|
60
|
+
|
|
61
|
+
## Tools
|
|
62
|
+
|
|
63
|
+
| Tool | What it does |
|
|
64
|
+
|---|---|
|
|
65
|
+
| `locator_search` | Query the index. Returns path, size, modification time, extension and kind, plus total/truncated so a capped page is never presented as complete. |
|
|
66
|
+
| `locator_status` | Roots, entry counts, watcher state, state directory and engine path. Use it to explain coverage or diagnose a stale result. |
|
|
67
|
+
| `locator_rebuild` | Replace the index with a fresh background scan. Searches keep using the old index until the new one is ready. |
|
|
68
|
+
| `locator_export` | Write a read-only SQLite mirror of the index (`entries` table) for external SQL tools. Streams page by page; the binary snapshot stays primary. |
|
|
69
|
+
|
|
70
|
+
### Query language
|
|
71
|
+
|
|
72
|
+
Bare words are ANDed substrings of the file name. Everything-style filters combine with them:
|
|
73
|
+
|
|
74
|
+
| Example | Meaning |
|
|
75
|
+
|---|---|
|
|
76
|
+
| `report` | name contains `report` |
|
|
77
|
+
| `report ext:pdf` | …and the extension is `pdf` |
|
|
78
|
+
| `ext:ts,js` / `ext:!tmp` | include or exclude extensions |
|
|
79
|
+
| `*.sln` / `?ata` | `*` and `?` wildcards |
|
|
80
|
+
| `"annual report"` | phrase with a space |
|
|
81
|
+
| `!archive` | negate a term |
|
|
82
|
+
| `path:src` | the full path contains `src` |
|
|
83
|
+
| `size:>10mb` / `size:1kb..2mb` | size comparisons |
|
|
84
|
+
| `dm:today` / `dm:thisweek` / `dm:2026-01-01` | modification time |
|
|
85
|
+
| `folder:` / `file:` | directories or files only |
|
|
86
|
+
| `case:1` / `wholeword:1` | case-sensitive / word boundaries |
|
|
87
|
+
|
|
88
|
+
`locator_search` also takes `sort` (`name`, `path`, `size`, `mtime`, `ext`), `desc`, `offset`, `limit`, `regex`, `caseSensitive` and `wholeWord` parameters.
|
|
89
|
+
|
|
90
|
+
## Configuration
|
|
91
|
+
|
|
92
|
+
Every field is a Schemastery config field; edit the `dsh-locator` entry in the profile patch (or the bundled `cordis.patch.yml` defaults).
|
|
93
|
+
|
|
94
|
+
| Key | Default | Meaning |
|
|
95
|
+
|---|---|---|
|
|
96
|
+
| `enabled` | `true` | Mount no tools and start no engine when `false`. |
|
|
97
|
+
| `roots` | `[]` | Directories to index; empty means the home directory. |
|
|
98
|
+
| `excludes` | dev/cache directories | Base-name globs skipped by scanner and watcher. |
|
|
99
|
+
| `watch` | `true` | Apply native filesystem notifications. |
|
|
100
|
+
| `refreshIntervalMs` | `1800000` | Rebuild cadence when no native watcher exists (macOS today); `0` disables. |
|
|
101
|
+
| `scanWorkers` | `0` | Directory reader concurrency during scans; `0` uses the CPU count. |
|
|
102
|
+
| `windowsUsn` | `false` | Try the NTFS USN/MFT enumeration before the walk (Windows, administrator; falls back automatically). |
|
|
103
|
+
| `stateDir` | `''` | Snapshot directory; empty means the platform cache directory. |
|
|
104
|
+
| `binaryPath` | `''` | Override the bundled engine (development). |
|
|
105
|
+
| `maxResults` | `100` | Hard cap for one search page. |
|
|
106
|
+
| `startupTimeoutMs` | `120000` | How long a cold build may take before the plugin reports failure. |
|
|
107
|
+
| `requestTimeoutMs` | `15000` | Per-request deadline. |
|
|
108
|
+
|
|
109
|
+
Narrow `roots` if indexing the whole home directory is too much I/O. A small, focused set (`~/Documents`, `~/Downloads`, one code tree) is usually the best trade-off.
|
|
110
|
+
|
|
111
|
+
## Platform support
|
|
112
|
+
|
|
113
|
+
| Platform | Scan + snapshot | Live updates | Notes |
|
|
114
|
+
|---|---|---|---|
|
|
115
|
+
| Linux | yes | inotify, recursive | Watch descriptors are per directory; very large trees can hit `fs.inotify.max_user_watches`. |
|
|
116
|
+
| Windows | yes | `ReadDirectoryChangesW`, recursive | No admin rights and no Everything needed. Optional USN/MFT enumeration when running elevated. Changes made by WSL to `/mnt/<drive>` may not raise notifications; use `locator_rebuild` for those. |
|
|
117
|
+
| macOS | yes | kqueue vnode watches for up to 4,096 directories, plus periodic rebuild | Directory-level events cover creates, deletes and renames. File content changes do not raise vnode events, so size/mtime refresh comes from `refreshIntervalMs` or `locator_rebuild`. |
|
|
118
|
+
|
|
119
|
+
The engine is pure Go with no cgo, cross-compiled from one machine. It never needs admin privileges.
|
|
120
|
+
|
|
121
|
+
### Windows and the USN journal
|
|
122
|
+
|
|
123
|
+
The engine carries a USN decoder, an opt-in fast path and a probe. The path is **off by default** (`windowsUsn: false`) because the volume ioctl needs administrator rights; it has been validated end to end on a GitHub Actions Windows runner (administrator, real NTFS volume):
|
|
124
|
+
- 1,345,546 MFT records read, 1,345,539 paths resolved
|
|
125
|
+
- 164,209 entries under `C:\Windows` (122,915 files, 41,294 dirs)
|
|
126
|
+
- `scan --usn --root C:\Windows` indexed 164,216 entries in 7.8 s; a query returned in 9.6 ms
|
|
127
|
+
- the change journal was read (8,233 records on one run) and an inactive journal on `D:` fell back to the walk with a warning
|
|
128
|
+
|
|
129
|
+
You can still run the same probe yourself:
|
|
130
|
+
|
|
131
|
+
```powershell
|
|
132
|
+
# Elevated PowerShell (or: Start-Process -Verb RunAs <path> ...)
|
|
133
|
+
& ".\bin\win32-x64\indexd.exe" usn-probe --volume C: --root "C:\Windows" --limit 5 --journal 5
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
It prints the journal id/USNs, enumerates the MFT (records per second, files/dirs below the root, sample paths) and decodes the last few change-journal records. A non-elevated run fails with `Access is denied` — the parser and path reconstruction are covered by unit tests on every platform.
|
|
137
|
+
|
|
138
|
+
With `windowsUsn: true` (or `indexd scan --usn`) the engine enumerates the MFT instead of walking directories, then builds paths from the FRN parent map. It keeps the volume handle and polls the change journal every two seconds, applying creates, deletes and renames through `usn.ApplyChanges` (the same FRN map, updated in place). USN records carry **no size or modification time**: results are correct for names and paths, `locator_status` marks such an index with `USN`, and the daemon stats the returned page so the rows the panel/model sees still show real sizes and dates. `size:`/`dm:` filters over the whole index stay approximate until a metadata pass runs. Any failure (no elevation, unsupported volume, too many records) falls back to the portable walk with a warning; it never disables search. If journal polling fails mid-session, the watcher reports it and stops; the portable `ReadDirectoryChangesW` watcher and `locator_rebuild` are still there. The journal applier is covered by synthetic-record tests and the ioctl path by the Windows CI job.
|
|
139
|
+
|
|
140
|
+
Everything reads the NTFS Master File Table through the USN journal, which is why its first index is nearly instant. `dsh-locator` v0.1 deliberately starts with a portable directory walk plus change notifications: it works without a service, without UAC and on every filesystem. USN/MFT enumeration is on the roadmap as an opt-in fast path for volumes where the process holds `GENERIC_READ` (admin) — the portable path stays the fallback.
|
|
141
|
+
|
|
142
|
+
## Performance
|
|
143
|
+
|
|
144
|
+
Measured on a developer WSL2 machine with a Linux home directory (1,234,757 entries: 1,128,288 files, 106,469 directories), excluding `node_modules`, `.git`, `.cache` and `.venv`:
|
|
145
|
+
|
|
146
|
+
| Metric | Result |
|
|
147
|
+
|---|---|
|
|
148
|
+
| Cold scan (uncached I/O) | 50.3 s |
|
|
149
|
+
| Warm scan (page cache, 8 readers) | 4.6–6.5 s (~190–265k entries/s) |
|
|
150
|
+
| Snapshot size | 133 MB |
|
|
151
|
+
| Peak scan RSS | ~490 MB |
|
|
152
|
+
| First search during the initial scan | 2 ms (partial, `indexing: true`) |
|
|
153
|
+
| `locator_search "report"` (981 matches) | 34 ms |
|
|
154
|
+
| `locator_search "package.json"` (1,287 matches) | 25 ms |
|
|
155
|
+
| Query process RSS (1.2 M entries) | ~197 MB |
|
|
156
|
+
| Small tree (39k entries) scan | 0.47 s |
|
|
157
|
+
|
|
158
|
+
Search is parallel over the record array; the slower cold scan is disk-bound. Numbers vary with the filesystem, cache state and CPU.
|
|
159
|
+
|
|
160
|
+
## Design
|
|
161
|
+
|
|
162
|
+
```
|
|
163
|
+
DeepSeek Harness
|
|
164
|
+
└─ dsh-locator (TypeScript plugin)
|
|
165
|
+
├─ locator_search / locator_status / locator_rebuild
|
|
166
|
+
├─ client.js: Ctrl+Shift+F overlay (shell.overlay slot)
|
|
167
|
+
├─ POST /dsh-locator/{search,status} (same-origin, loopback-hosted)
|
|
168
|
+
└─ child process: indexd serve (Go, NDJSON over stdio)
|
|
169
|
+
├─ scanner: recursive walk, excludes, progress
|
|
170
|
+
├─ index: names arena + fixed records + open-addressing path table
|
|
171
|
+
├─ query: parser, matcher, top-K selection, sort, pagination
|
|
172
|
+
├─ watcher: inotify (Linux) / ReadDirectoryChangesW (Windows)
|
|
173
|
+
└─ snapshot: single file, mmap-friendly, atomic rename
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
* **Parallel scanner.** Directory readers drain a bounded work queue and flush one batch per directory (`AddBatch`), so a parent record exists before its children are enqueued. The queue never blocks: a full queue falls back to scanning that subtree inline.
|
|
177
|
+
* **Memory layout.** Records store a name offset/length into one byte arena plus a parent id; full paths are reconstructed only for the result page and for `path:` filters. That avoids one string header per file.
|
|
178
|
+
* **Path table.** An open-addressing table of `id+1` slots keeps lookup/update near 12 bytes per entry instead of the ~50 bytes a `map[string]int32` costs.
|
|
179
|
+
* **Query engine.** Bare terms and filters are compiled once, matched in parallel chunks, then reduced with a bounded max-heap: the top `offset+limit` rows are selected without sorting a million matches.
|
|
180
|
+
* **Persistence.** The snapshot is written to a temporary file and renamed; a generation counter lets the daemon autosave only when the index changed. Tombstones are compacted when deletions exceed a threshold.
|
|
181
|
+
* **Protocol.** One JSON request/response per line on stdio, with unsolicited `scan`, `stats`, `warning`, `error` and `rebuilt` events. No ports, no auth surface, no network listeners.
|
|
182
|
+
* **Read-only.** The engine opens files for metadata only; it never writes to indexed paths.
|
|
183
|
+
|
|
184
|
+
## Build from source
|
|
185
|
+
|
|
186
|
+
Requirements: Node `^22.19 || >=24.9`, pnpm, Go 1.22+ for the engine.
|
|
187
|
+
|
|
188
|
+
```sh
|
|
189
|
+
pnpm install
|
|
190
|
+
pnpm typecheck
|
|
191
|
+
pnpm test
|
|
192
|
+
pnpm build
|
|
193
|
+
pnpm build:engine:all # or build:engine for the host target
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
The engine is a separate Go module under `engine/`; run its tests with `cd engine && go test ./...`.
|
|
197
|
+
|
|
198
|
+
## Limitations (v0.1)
|
|
199
|
+
|
|
200
|
+
* File names, paths, sizes and modification times only — **no content indexing**. Use `grep` for contents.
|
|
201
|
+
* The index covers configured roots; it is not a filesystem driver.
|
|
202
|
+
* Linux inotify watches one descriptor per directory; a home directory with hundreds of thousands of directories can exceed the kernel limit. The plugin reports the warning and the snapshot stays usable; rebuilds catch up.
|
|
203
|
+
* WSL writes into `/mnt/c` may not produce Windows notifications (the Windows watcher cannot see the 9p writer's change stream). Rebuild those roots explicitly.
|
|
204
|
+
* Network shares are scanned only if they are reachable as ordinary directories; no SMB-specific change feed.
|
|
205
|
+
* macOS watches are directory-level (kqueue, one descriptor per directory, capped at 4,096). Creates/deletes/renames land immediately; file content changes rely on the periodic rebuild. FSEvents remains the roadmap item for an uncapped, file-level feed.
|
|
206
|
+
* No USN/MFT fast enumeration yet (see above).
|
|
207
|
+
|
|
208
|
+
## Roadmap
|
|
209
|
+
|
|
210
|
+
1. macOS FSEvents watcher.
|
|
211
|
+
2. Windows USN/MFT fast enumeration when the process can read the volume; journal-based catch-up after downtime.
|
|
212
|
+
3. Dedicated Web GUI panel (table, sortable columns, filters, preview, insert-as-`@file`).
|
|
213
|
+
4. Optional lowercase-name arena and trigram prefilter for large indexes.
|
|
214
|
+
5. Incremental snapshot compaction and per-root journals.
|
|
215
|
+
|
|
216
|
+
## Security and privacy
|
|
217
|
+
|
|
218
|
+
The index is local: no network calls, no telemetry, no uploads. The snapshot contains file names and paths, so treat the state directory as private data and exclude it from backups if that matters to you. The plugin never modifies, moves or deletes indexed files.
|
|
219
|
+
|
|
220
|
+
## License
|
|
221
|
+
|
|
222
|
+
Apache-2.0.
|
package/SECURITY.md
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
# Security
|
|
2
|
+
|
|
3
|
+
## Data and network surface
|
|
4
|
+
|
|
5
|
+
`dsh-locator` is local-only. The engine:
|
|
6
|
+
|
|
7
|
+
* never opens a network listener, never makes a network request, and has no telemetry;
|
|
8
|
+
* reads directory entries and file metadata (name, size, timestamps) under the configured roots;
|
|
9
|
+
* never writes, moves, renames or deletes an indexed file;
|
|
10
|
+
* stores its snapshot under `stateDir` (default: the platform cache directory).
|
|
11
|
+
|
|
12
|
+
The snapshot contains file names and paths. Treat the state directory as private data. Deleting it only costs one rebuild.
|
|
13
|
+
|
|
14
|
+
## Process model
|
|
15
|
+
|
|
16
|
+
The engine is a separate Go process started by the plugin with stdio pipes. It does not need admin rights, a service, a driver, or Everything. It exits when the plugin unloads (or after a 1.5 s grace period).
|
|
17
|
+
|
|
18
|
+
## Binaries
|
|
19
|
+
|
|
20
|
+
The npm package ships prebuilt engines for six platform/architecture pairs. They are built from `engine/` by `scripts/build-engine.mjs` with `-trimpath` and no cgo. Rebuild them yourself if you prefer not to trust the bundled artifacts:
|
|
21
|
+
|
|
22
|
+
```sh
|
|
23
|
+
pnpm build:engine:all
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
## Reporting
|
|
27
|
+
|
|
28
|
+
Open an issue at <https://github.com/PerryLink/dsh-locator/issues> without attaching snapshots, file lists, or anything else that reveals directory contents. For a sensitive report, say so in the issue and ask for a private channel first.
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|