@karmaniverous/jeeves-watcher 0.19.1 → 0.21.0-0

package/LICENSE

@@ -0,0 +1,28 @@
+BSD 3-Clause License
+
+Copyright (c) 2025, Jason Williscroft
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

package/README.md

@@ -4,18 +4,20 @@ Filesystem watcher that keeps a [Qdrant](https://qdrant.tech/) vector store in s
 
 ## Requirements
 
-- **Node.js** ≥ 22
+- **Node.js** ≥ 24.22.2
 
 ## Features
 
-- **Filesystem watching** — monitors directories for file changes via [chokidar](https://github.com/paulmillr/chokidar)
-- **Multi-format extraction** — PDF, HTML, DOCX, Markdown, plain text, and more
-- **Configurable chunking** — token-based text splitting with overlap control
+- **Filesystem watching** — monitors directories for file changes via [@parcel/watcher](https://github.com/parcel-bundler/watcher) (one native subscription per watch root; [chokidar](https://github.com/paulmillr/chokidar) on request or when polling)
+- **Multi-format extraction** — PDF, HTML, DOCX, Markdown, JSON and plain text (other extensions are read as plain text)
+- **Configurable chunking** — character-based text splitting with overlap control
 - **Embedding providers** — Gemini (default) or mock (for testing); extensible via provider registry
 - **Qdrant sync** — automatic upsert/delete keeps the vector store current
 - **Rules engine** — glob-based inference rules for metadata enrichment
-- **REST API** — Fastify server for search, status, config, and management
-- **CLI** — `jeeves-watcher init`, `validate`, `start`, and more
+- **REST API** — Fastify server for search, status, config, and management, with optional bearer API key (`api.apiKey`)
+- **Source documents** — index records owned by another application, retrieved by HTTP callback on notification (`sources` config, `POST /documents`); see the [Source Documents guide](https://docs.karmanivero.us/jeeves-watcher/documents/Service_Guides.Source_Documents.html)
+- **SDK** — [`@karmaniverous/jeeves-watcher-sdk`](https://www.npmjs.com/package/@karmaniverous/jeeves-watcher-sdk), a typed TypeScript client for the application-facing API
+- **CLI** — `jeeves-watcher init`, `config validate`, `start`, and more
 
 ## Version Control (VCS)
 
@@ -51,9 +53,11 @@ Add a `vcs` block to your config:
 | Property | Default | Description |
 | --- | --- | --- |
 | `vcs.enabled` | `false` | Enable git-backed version control globally |
+| `vcs.author.name` / `vcs.author.email` | `"jeeves-watcher"` / `"watcher@localhost"` | Git commit author identity |
 | `vcs.commitThrottleMs` | `30000` | Throttle interval (ms) for batching file changes into commits. Min: 1000 |
 | `vcs.maxBatchSize` | `1000` | Maximum files per commit batch. Flushes immediately when exceeded. Min: 1 |
-| `vcs.branch` | `"master"` | Git branch name for VCS operations. Squash retention and startup recovery target this branch |
+| `vcs.maxFileBytes` | `100000000` | Size cap for versioned files (bytes, or `k`/`m`/`g`). A new file over it is not staged; a tracked file that grows past it is untracked in its own commit. `0` disables |
+| `vcs.branch` | `"master"` | Git branch name for VCS operations. Squash retention targets this branch, and startup warns when HEAD is not on it |
 | `vcs.commitMessage.enabled` | `true` | Enable AI-generated commit messages |
 | `vcs.commitMessage.provider` | `"anthropic"` | AI provider (currently only `"anthropic"` supported) |
 | `vcs.commitMessage.model` | `"claude-haiku-4-0"` | AI model for commit message generation |
@@ -62,6 +66,9 @@ Add a `vcs` block to your config:
 | `vcs.retention.maxVersions` | `100` | Maximum commits retained per root (older ones squashed) |
 | `vcs.retention.squashCron` | `"0 0 * * *"` | Cron schedule for squash retention (5-field, checked every 60s) |
 | `vcs.defaultAccessToken` | — | Fallback git access token for all roots. Supports env var substitution |
+| `vcs.staleLockThresholdMs` | `60000` | Age (ms) after which a git `index.lock` is considered stale and removed. Min: 5000 |
+| `vcs.maxConsecutiveFailures` | `5` | Commit circuit breaker: trips after this many consecutive commit failures. Min: 1 |
+| `vcs.circuitBreakerCooldownMs` | `300000` | Cooldown (ms) before a tripped breaker allows one recovery attempt; pending files are kept. Min: 1000 |
 
 ### Per-Root Overrides via `watch.paths`
 
@@ -71,15 +78,15 @@ Watch path entries can be objects with per-root VCS overrides:
 {
   "watch": {
     "paths": [
-      "J:/domains/notes/**/*.md",
+      "/data/notes/**/*.md",
       {
-        "path": "J:/domains/jira/**/*.json",
+        "path": "/data/tickets/**/*.json",
         "vcs": {
           "enabled": false
         }
       },
       {
-        "path": "J:/domains/email/**/*.json",
+        "path": "/data/email/**/*.json",
         "vcs": {
           "remote": "https://github.com/org/email-archive.git",
           "accessToken": "${GITHUB_TOKEN}",
@@ -99,7 +106,7 @@ Per-root properties (`remote`, `accessToken`) are only available in per-root ove
 ### Deployment Notes
 
 - **Git dependency:** `git` must be on the system PATH. The watcher initializes a git repository in each VCS-enabled watch root automatically.
-- **stateDir:** The `.jeeves-metadata/` directory (configurable via `stateDir`) is always gitignored. VCS state (git repos) lives inside the watch roots themselves, not in stateDir.
+- **stateDir:** With VCS enabled, `stateDir` (default `.jeeves-metadata`) must not overlap any watch root (equal, parent or child): startup fails otherwise. The `.gitignore` the watcher maintains in each VCS root always lists `.git/`, `node_modules/`, `.jeeves-watcher/` and `.jeeves-metadata/`. VCS state (git repos) lives inside the watch roots themselves, not in stateDir.
 - **Filesystem constraints:** Each VCS-enabled watch path entry must resolve to a single directory root (not a glob with wildcards in directory segments).
 
 ### Remote Push
@@ -115,7 +122,7 @@ When `remote` is set on a per-root VCS config, the watcher pushes to the remote
   "watch": {
     "paths": [
       {
-        "path": "J:/domains/notes",
+        "path": "/data/notes",
         "vcs": {
           "remote": "https://github.com/org/notes-archive.git"
         }
@@ -134,18 +141,19 @@ To prevent unbounded history growth, the squash retention system periodically co
 - **`maxAgeDays`** and **`maxVersions`** define retention boundaries. The tighter constraint wins — if `maxAgeDays: 30` would keep 200 commits but `maxVersions: 100` caps at 100, only 100 are retained.
 - **`squashCron`** controls when squash runs (e.g., `"0 0 * * *"` = daily at midnight). Checked every 60 seconds.
 - After squash, old commits are replaced with a single "historical baseline" commit. If a remote is configured, squash triggers a force push.
+- Squash rewrites history with git plumbing only: it never touches the working tree or the index, and moves the branch with a compare-and-swap, so a concurrent commit makes it stop without changing anything.
 
 ### API Endpoints
 
-| Endpoint               | Method | Description                       |
-| ---------------------- | ------ | --------------------------------- |
-| `/vcs/status`          | GET    | VCS state for all roots           |
-| `/vcs/history`         | GET    | Commit history for a path/glob    |
-| `/vcs/show`            | GET    | File content at a specific commit |
-| `/vcs/diff`            | GET    | Diff between commits              |
-| `/vcs/revert`          | POST   | Restore files from a past commit  |
-| `/vcs/exclude`         | POST   | Manage gitignore exclusions       |
-| `/vcs/check-exclusion` | GET    | Check gitignore status of a path  |
+| Endpoint | Method | Description |
+| --- | --- | --- |
+| `/vcs/status` | GET | VCS state for all roots |
+| `/vcs/history` | GET | Commit history for a path/glob |
+| `/vcs/show` | GET | File content at a specific commit |
+| `/vcs/diff` | GET | Diff between commits |
+| `/vcs/revert` | POST | Restore files from a past commit |
+| `/vcs/exclude` | POST | VCS-only exclusion (`.git/info/exclude`), untracking matches |
+| `/vcs/check-exclusion` | GET | Whether git ignores a path: rule, file, line |
 
 ## JsonMap Built-in Helpers
 
@@ -165,11 +173,11 @@ The following helpers are available in every JsonMap `lib` context:
 
 ### `fetchSiblings`
 
-Retrieves extracted text from sibling files in the same directory, useful for contextual embedding (e.g., injecting surrounding email thread messages).
+Retrieves text from sibling files (same directory, same extension) around the current file, useful for contextual embedding (e.g., injecting surrounding email thread messages). JSON siblings use the JSON text derivation; other files are read raw.
 
 **Options:** `{ before?: number (default 3), after?: number (default 1), sort?: "name" | "mtime" (default "name") }`
 
-Returns `string[]` of extracted text. Files that fail extraction are silently skipped.
+Returns `string[]` of sibling text. Files that cannot be read are silently skipped. For a source document there is no file to read around, so `fetchSiblings` returns `[]`.
 
 ## Install
 
@@ -180,14 +188,14 @@ npm install @karmaniverous/jeeves-watcher
 ## Quick Start
 
 ```bash
-# Generate a config file
-npx jeeves-watcher init --output ./jeeves-watcher.config.json
+# Generate ./config/config.json (--output is a directory)
+npx jeeves-watcher init --output ./config
 
 # Validate it
-npx jeeves-watcher validate --config ./jeeves-watcher.config.json
+npx jeeves-watcher config validate --config ./config/config.json
 
 # Start the watcher
-npx jeeves-watcher start --config ./jeeves-watcher.config.json
+npx jeeves-watcher start --config ./config/config.json
 ```
 
 ## Documentation
@@ -196,6 +204,8 @@ Full docs, guides, and API reference:
 
 **[docs.karmanivero.us/jeeves-watcher](https://docs.karmanivero.us/jeeves-watcher)**
 
+Upgrading an existing deployment? See [Upgrading](guides/upgrading.md).
+
 ## License
 
 BSD-3-Clause