dsh-locator 0.0.0-stage → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,3 +1,222 @@
1
- # Temporary Holding Version
1
+ # dsh-locator
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ [![npm](https://img.shields.io/npm/v/dsh-locator)](https://www.npmjs.com/package/dsh-locator)
4
+ [![ci](https://github.com/PerryLink/dsh-locator/actions/workflows/ci.yml/badge.svg)](https://github.com/PerryLink/dsh-locator/actions/workflows/ci.yml)
5
+
6
+ [中文](README-zh.md) | English
7
+
8
+ A whole-disk, millisecond file search for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) that builds its **own** index. No Everything installation, no `es.exe`, no Everything service.
9
+
10
+ `dsh-locator` scans the directories you choose once, keeps the result in a compact snapshot, applies filesystem changes as they happen, and answers name/path queries in tens of milliseconds across a million entries. The agent gets `locator_search`, `locator_status` and `locator_rebuild`; people get a global **search panel** in the Web GUI (`Ctrl+Shift+F` or the magnifier at the sidebar foot) with sortable columns, filters, keyboard navigation and one-key path copy.
11
+
12
+ Everything inspired the *ideas*: a persistent index of file names, a query language with `ext:` / `size:` / `dm:` filters, and an updater that consumes the filesystem's own change stream. The implementation is independent and cross-platform.
13
+
14
+ ## Why another file search
15
+
16
+ `glob` and `grep` walk the workspace on every call — fine when you know the directory, slow when you remember only part of a name. Everything is excellent on Windows but is Windows-only, and wrapping it leaves macOS, Linux, WSL and remote hosts uncovered. `dsh-locator` owns the index, so the same tools work everywhere Node and Go run.
17
+
18
+ ## Install
19
+
20
+ ```sh
21
+ # from npm
22
+ dsh plugin --profile web add dsh-locator
23
+
24
+ # from a checkout
25
+ pnpm install
26
+ pnpm build # TypeScript plugin
27
+ pnpm build:engine # Go engine for the host platform
28
+ dsh plugin --profile web add /absolute/path/to/dsh-locator
29
+
30
+ # from a packed tarball (verified install path)
31
+ npm pack
32
+ dsh plugin --profile web add /absolute/path/to/dsh-locator-0.1.0.tgz
33
+ ```
34
+
35
+ The npm package ships prebuilt engines for `linux-x64`, `linux-arm64`, `win32-x64`, `win32-arm64`, `darwin-x64` and `darwin-arm64`.
36
+
37
+ After installing into a Web profile that is already running, **reload the page** (`F5`): the host tools may mount immediately, but the browser panel (`client.js`) is loaded from the boot graph at page load. Headless/TUI profiles never mount the panel.
38
+
39
+ ## First run
40
+
41
+ On first mount the engine indexes the configured roots (default: your home directory) and writes a snapshot to the platform cache directory:
42
+
43
+ | Platform | Default state directory |
44
+ |---|---|
45
+ | Windows | `%LOCALAPPDATA%\dsh-locator` |
46
+ | macOS | `~/Library/Application Support/dsh-locator` |
47
+ | Linux | `$XDG_CACHE_HOME/dsh-locator` or `~/.cache/dsh-locator` |
48
+
49
+ The first scan of a large home directory reads the disk; later starts load the snapshot instead. Requests are served **during** that first scan: search returns the entries indexed so far with `indexing: true`, and the panel footer says the index is still building. Nothing blocks on the cold scan. A default exclude list skips `node_modules`, `.git`, `.svn`, `.hg`, `.cache`, `__pycache__`, `.venv`, `venv` and `.next`.
50
+
51
+ ## Web search panel
52
+
53
+ Press `Ctrl+Shift+F` anywhere in the Web GUI, or click the magnifier at the sidebar foot. The overlay queries the same index through two same-origin host routes (`POST /dsh-locator/search`, `POST /dsh-locator/status`), so it needs no second server and no extra port.
54
+
55
+ * Empty query lists the most recently modified entries; typing runs the same Everything-style syntax as the tool after a 110 ms debounce.
56
+ * Click a column header to sort by name, size, modification time or path; toggle ascending/descending.
57
+ * `只看文件` / `只看目录` filter the kind; the footer reports matches, the page cap and query latency.
58
+ * `↑`/`↓` select, `Enter` copies the selected path, `Esc` closes.
59
+ * The panel is optional: profiles without a Web server never mount it, and the three tools keep working.
60
+
61
+ ## Tools
62
+
63
+ | Tool | What it does |
64
+ |---|---|
65
+ | `locator_search` | Query the index. Returns path, size, modification time, extension and kind, plus total/truncated so a capped page is never presented as complete. |
66
+ | `locator_status` | Roots, entry counts, watcher state, state directory and engine path. Use it to explain coverage or diagnose a stale result. |
67
+ | `locator_rebuild` | Replace the index with a fresh background scan. Searches keep using the old index until the new one is ready. |
68
+ | `locator_export` | Write a read-only SQLite mirror of the index (`entries` table) for external SQL tools. Streams page by page; the binary snapshot stays primary. |
69
+
70
+ ### Query language
71
+
72
+ Bare words are ANDed substrings of the file name. Everything-style filters combine with them:
73
+
74
+ | Example | Meaning |
75
+ |---|---|
76
+ | `report` | name contains `report` |
77
+ | `report ext:pdf` | …and the extension is `pdf` |
78
+ | `ext:ts,js` / `ext:!tmp` | include or exclude extensions |
79
+ | `*.sln` / `?ata` | `*` and `?` wildcards |
80
+ | `"annual report"` | phrase with a space |
81
+ | `!archive` | negate a term |
82
+ | `path:src` | the full path contains `src` |
83
+ | `size:>10mb` / `size:1kb..2mb` | size comparisons |
84
+ | `dm:today` / `dm:thisweek` / `dm:2026-01-01` | modification time |
85
+ | `folder:` / `file:` | directories or files only |
86
+ | `case:1` / `wholeword:1` | case-sensitive / word boundaries |
87
+
88
+ `locator_search` also takes `sort` (`name`, `path`, `size`, `mtime`, `ext`), `desc`, `offset`, `limit`, `regex`, `caseSensitive` and `wholeWord` parameters.
89
+
90
+ ## Configuration
91
+
92
+ Every field is a Schemastery config field; edit the `dsh-locator` entry in the profile patch (or the bundled `cordis.patch.yml` defaults).
93
+
94
+ | Key | Default | Meaning |
95
+ |---|---|---|
96
+ | `enabled` | `true` | Mount no tools and start no engine when `false`. |
97
+ | `roots` | `[]` | Directories to index; empty means the home directory. |
98
+ | `excludes` | dev/cache directories | Base-name globs skipped by scanner and watcher. |
99
+ | `watch` | `true` | Apply native filesystem notifications. |
100
+ | `refreshIntervalMs` | `1800000` | Rebuild cadence when no native watcher exists (macOS today); `0` disables. |
101
+ | `scanWorkers` | `0` | Directory reader concurrency during scans; `0` uses the CPU count. |
102
+ | `windowsUsn` | `false` | Try the NTFS USN/MFT enumeration before the walk (Windows, administrator; falls back automatically). |
103
+ | `stateDir` | `''` | Snapshot directory; empty means the platform cache directory. |
104
+ | `binaryPath` | `''` | Override the bundled engine (development). |
105
+ | `maxResults` | `100` | Hard cap for one search page. |
106
+ | `startupTimeoutMs` | `120000` | How long a cold build may take before the plugin reports failure. |
107
+ | `requestTimeoutMs` | `15000` | Per-request deadline. |
108
+
109
+ Narrow `roots` if indexing the whole home directory is too much I/O. A small, focused set (`~/Documents`, `~/Downloads`, one code tree) is usually the best trade-off.
110
+
111
+ ## Platform support
112
+
113
+ | Platform | Scan + snapshot | Live updates | Notes |
114
+ |---|---|---|---|
115
+ | Linux | yes | inotify, recursive | Watch descriptors are per directory; very large trees can hit `fs.inotify.max_user_watches`. |
116
+ | Windows | yes | `ReadDirectoryChangesW`, recursive | No admin rights and no Everything needed. Optional USN/MFT enumeration when running elevated. Changes made by WSL to `/mnt/<drive>` may not raise notifications; use `locator_rebuild` for those. |
117
+ | macOS | yes | kqueue vnode watches for up to 4,096 directories, plus periodic rebuild | Directory-level events cover creates, deletes and renames. File content changes do not raise vnode events, so size/mtime refresh comes from `refreshIntervalMs` or `locator_rebuild`. |
118
+
119
+ The engine is pure Go with no cgo, cross-compiled from one machine. It never needs admin privileges.
120
+
121
+ ### Windows and the USN journal
122
+
123
+ The engine carries a USN decoder, an opt-in fast path and a probe. The path is **off by default** (`windowsUsn: false`) because the volume ioctl needs administrator rights; it has been validated end to end on a GitHub Actions Windows runner (administrator, real NTFS volume):
124
+ - 1,345,546 MFT records read, 1,345,539 paths resolved
125
+ - 164,209 entries under `C:\Windows` (122,915 files, 41,294 dirs)
126
+ - `scan --usn --root C:\Windows` indexed 164,216 entries in 7.8 s; a query returned in 9.6 ms
127
+ - the change journal was read (8,233 records on one run) and an inactive journal on `D:` fell back to the walk with a warning
128
+
129
+ You can still run the same probe yourself:
130
+
131
+ ```powershell
132
+ # Elevated PowerShell (or: Start-Process -Verb RunAs <path> ...)
133
+ & ".\bin\win32-x64\indexd.exe" usn-probe --volume C: --root "C:\Windows" --limit 5 --journal 5
134
+ ```
135
+
136
+ It prints the journal id/USNs, enumerates the MFT (records per second, files/dirs below the root, sample paths) and decodes the last few change-journal records. A non-elevated run fails with `Access is denied` — the parser and path reconstruction are covered by unit tests on every platform.
137
+
138
+ With `windowsUsn: true` (or `indexd scan --usn`) the engine enumerates the MFT instead of walking directories, then builds paths from the FRN parent map. It keeps the volume handle and polls the change journal every two seconds, applying creates, deletes and renames through `usn.ApplyChanges` (the same FRN map, updated in place). USN records carry **no size or modification time**: results are correct for names and paths, `locator_status` marks such an index with `USN`, and the daemon stats the returned page so the rows the panel/model sees still show real sizes and dates. `size:`/`dm:` filters over the whole index stay approximate until a metadata pass runs. Any failure (no elevation, unsupported volume, too many records) falls back to the portable walk with a warning; it never disables search. If journal polling fails mid-session, the watcher reports it and stops; the portable `ReadDirectoryChangesW` watcher and `locator_rebuild` are still there. The journal applier is covered by synthetic-record tests and the ioctl path by the Windows CI job.
139
+
140
+ Everything reads the NTFS Master File Table through the USN journal, which is why its first index is nearly instant. `dsh-locator` v0.1 deliberately starts with a portable directory walk plus change notifications: it works without a service, without UAC and on every filesystem. USN/MFT enumeration is on the roadmap as an opt-in fast path for volumes where the process holds `GENERIC_READ` (admin) — the portable path stays the fallback.
141
+
142
+ ## Performance
143
+
144
+ Measured on a developer WSL2 machine with a Linux home directory (1,234,757 entries: 1,128,288 files, 106,469 directories), excluding `node_modules`, `.git`, `.cache` and `.venv`:
145
+
146
+ | Metric | Result |
147
+ |---|---|
148
+ | Cold scan (uncached I/O) | 50.3 s |
149
+ | Warm scan (page cache, 8 readers) | 4.6–6.5 s (~190–265k entries/s) |
150
+ | Snapshot size | 133 MB |
151
+ | Peak scan RSS | ~490 MB |
152
+ | First search during the initial scan | 2 ms (partial, `indexing: true`) |
153
+ | `locator_search "report"` (981 matches) | 34 ms |
154
+ | `locator_search "package.json"` (1,287 matches) | 25 ms |
155
+ | Query process RSS (1.2 M entries) | ~197 MB |
156
+ | Small tree (39k entries) scan | 0.47 s |
157
+
158
+ Search is parallel over the record array; the slower cold scan is disk-bound. Numbers vary with the filesystem, cache state and CPU.
159
+
160
+ ## Design
161
+
162
+ ```
163
+ DeepSeek Harness
164
+ └─ dsh-locator (TypeScript plugin)
165
+ ├─ locator_search / locator_status / locator_rebuild
166
+ ├─ client.js: Ctrl+Shift+F overlay (shell.overlay slot)
167
+ ├─ POST /dsh-locator/{search,status} (same-origin, loopback-hosted)
168
+ └─ child process: indexd serve (Go, NDJSON over stdio)
169
+ ├─ scanner: recursive walk, excludes, progress
170
+ ├─ index: names arena + fixed records + open-addressing path table
171
+ ├─ query: parser, matcher, top-K selection, sort, pagination
172
+ ├─ watcher: inotify (Linux) / ReadDirectoryChangesW (Windows)
173
+ └─ snapshot: single file, mmap-friendly, atomic rename
174
+ ```
175
+
176
+ * **Parallel scanner.** Directory readers drain a bounded work queue and flush one batch per directory (`AddBatch`), so a parent record exists before its children are enqueued. The queue never blocks: a full queue falls back to scanning that subtree inline.
177
+ * **Memory layout.** Records store a name offset/length into one byte arena plus a parent id; full paths are reconstructed only for the result page and for `path:` filters. That avoids one string header per file.
178
+ * **Path table.** An open-addressing table of `id+1` slots keeps lookup/update near 12 bytes per entry instead of the ~50 bytes a `map[string]int32` costs.
179
+ * **Query engine.** Bare terms and filters are compiled once, matched in parallel chunks, then reduced with a bounded max-heap: the top `offset+limit` rows are selected without sorting a million matches.
180
+ * **Persistence.** The snapshot is written to a temporary file and renamed; a generation counter lets the daemon autosave only when the index changed. Tombstones are compacted when deletions exceed a threshold.
181
+ * **Protocol.** One JSON request/response per line on stdio, with unsolicited `scan`, `stats`, `warning`, `error` and `rebuilt` events. No ports, no auth surface, no network listeners.
182
+ * **Read-only.** The engine opens files for metadata only; it never writes to indexed paths.
183
+
184
+ ## Build from source
185
+
186
+ Requirements: Node `^22.19 || >=24.9`, pnpm, Go 1.22+ for the engine.
187
+
188
+ ```sh
189
+ pnpm install
190
+ pnpm typecheck
191
+ pnpm test
192
+ pnpm build
193
+ pnpm build:engine:all # or build:engine for the host target
194
+ ```
195
+
196
+ The engine is a separate Go module under `engine/`; run its tests with `cd engine && go test ./...`.
197
+
198
+ ## Limitations (v0.1)
199
+
200
+ * File names, paths, sizes and modification times only — **no content indexing**. Use `grep` for contents.
201
+ * The index covers configured roots; it is not a filesystem driver.
202
+ * Linux inotify watches one descriptor per directory; a home directory with hundreds of thousands of directories can exceed the kernel limit. The plugin reports the warning and the snapshot stays usable; rebuilds catch up.
203
+ * WSL writes into `/mnt/c` may not produce Windows notifications (the Windows watcher cannot see the 9p writer's change stream). Rebuild those roots explicitly.
204
+ * Network shares are scanned only if they are reachable as ordinary directories; no SMB-specific change feed.
205
+ * macOS watches are directory-level (kqueue, one descriptor per directory, capped at 4,096). Creates/deletes/renames land immediately; file content changes rely on the periodic rebuild. FSEvents remains the roadmap item for an uncapped, file-level feed.
206
+ * No USN/MFT fast enumeration yet (see above).
207
+
208
+ ## Roadmap
209
+
210
+ 1. macOS FSEvents watcher.
211
+ 2. Windows USN/MFT fast enumeration when the process can read the volume; journal-based catch-up after downtime.
212
+ 3. Dedicated Web GUI panel (table, sortable columns, filters, preview, insert-as-`@file`).
213
+ 4. Optional lowercase-name arena and trigram prefilter for large indexes.
214
+ 5. Incremental snapshot compaction and per-root journals.
215
+
216
+ ## Security and privacy
217
+
218
+ The index is local: no network calls, no telemetry, no uploads. The snapshot contains file names and paths, so treat the state directory as private data and exclude it from backups if that matters to you. The plugin never modifies, moves or deletes indexed files.
219
+
220
+ ## License
221
+
222
+ Apache-2.0.
package/SECURITY.md ADDED
@@ -0,0 +1,28 @@
1
+ # Security
2
+
3
+ ## Data and network surface
4
+
5
+ `dsh-locator` is local-only. The engine:
6
+
7
+ * never opens a network listener, never makes a network request, and has no telemetry;
8
+ * reads directory entries and file metadata (name, size, timestamps) under the configured roots;
9
+ * never writes, moves, renames or deletes an indexed file;
10
+ * stores its snapshot under `stateDir` (default: the platform cache directory).
11
+
12
+ The snapshot contains file names and paths. Treat the state directory as private data. Deleting it only costs one rebuild.
13
+
14
+ ## Process model
15
+
16
+ The engine is a separate Go process started by the plugin with stdio pipes. It does not need admin rights, a service, a driver, or Everything. It exits when the plugin unloads (or after a 1.5 s grace period).
17
+
18
+ ## Binaries
19
+
20
+ The npm package ships prebuilt engines for six platform/architecture pairs. They are built from `engine/` by `scripts/build-engine.mjs` with `-trimpath` and no cgo. Rebuild them yourself if you prefer not to trust the bundled artifacts:
21
+
22
+ ```sh
23
+ pnpm build:engine:all
24
+ ```
25
+
26
+ ## Reporting
27
+
28
+ Open an issue at <https://github.com/PerryLink/dsh-locator/issues> without attaching snapshots, file lists, or anything else that reveals directory contents. For a sensitive report, say so in the issue and ask for a private channel first.
Binary file
Binary file
Binary file
Binary file
Binary file
Binary file