@karmaniverous/jeeves-watcher 0.5.0-1 → 0.5.1

package/README.md

@@ -1,300 +1,42 @@
 # @karmaniverous/jeeves-watcher
 
-Filesystem watcher that keeps a Qdrant vector store in sync with document changes.
+Filesystem watcher that keeps a [Qdrant](https://qdrant.tech/) vector store in sync with document changes. Extract text from files, chunk it, generate embeddings, and query your documents with semantic search.
 
-## Overview
+## Features
 
-`jeeves-watcher` monitors a configured set of directories for file changes, extracts text content, generates embeddings, and maintains a synchronized Qdrant vector store for semantic search. It automatically:
+- **Filesystem watching** — monitors directories for file changes via [chokidar](https://github.com/paulmillr/chokidar)
+- **Multi-format extraction** — PDF, HTML, DOCX, Markdown, plain text, and more
+- **Configurable chunking** — token-based text splitting with overlap control
+- **Embedding providers** — Gemini, OpenAI, or mock (for testing)
+- **Qdrant sync** — automatic upsert/delete keeps the vector store current
+- **Rules engine** — glob-based inference rules for metadata enrichment
+- **REST API** — Fastify server for search, status, config, and management
+- **CLI** — `jeeves-watcher init`, `validate`, `start`, and more
 
-- **Watches** directories for file additions, modifications, and deletions
-- **Extracts** text from various formats (Markdown, PDF, DOCX, HTML, JSON, plain text)
-- **Chunks** large documents for optimal embedding
-- **Embeds** content using configurable providers (Google Gemini, mock for testing)
-- **Syncs** to Qdrant for fast semantic search
-- **Enriches** metadata via rules and API endpoints
-
-### Architecture
-
-![System Architecture](assets/system-architecture.png)
-
-For detailed architecture documentation, see [guides/architecture.md](guides/architecture.md).
-
-## Quick Start
-
-### Installation
-
-```bash
-npm install -g @karmaniverous/jeeves-watcher
-```
-
-### Initialize Configuration
-
-Create a new configuration file in your project:
-
-```bash
-jeeves-watcher init
-```
-
-This generates a `jeeves-watcher.config.json` file with sensible defaults.
-
-### Configure
-
-Edit `jeeves-watcher.config.json` to specify:
-
-- **Watch paths**: Directories to monitor
-- **Embedding provider**: Google Gemini or mock (for testing)
-- **Qdrant connection**: URL and collection name
-- **Inference rules**: Automatic metadata enrichment based on file patterns
-
-Example minimal configuration:
-
-```json
-{
-  "watch": {
-    "paths": ["./docs"],
-    "ignored": ["**/node_modules/**", "**/.git/**"]
-  },
-  "embedding": {
-    "provider": "gemini",
-    "model": "gemini-embedding-001",
-    "apiKey": "${GOOGLE_API_KEY}"
-  },
-  "vectorStore": {
-    "url": "http://localhost:6333",
-    "collectionName": "my_docs"
-  }
-}
-```
-
-### Start Watching
-
-```bash
-jeeves-watcher start
-```
-
-The watcher will:
-1. Index all existing files in watched directories
-2. Monitor for changes
-3. Update Qdrant automatically
-
-## CLI Commands
-
-| Command | Description |
-|---------|-------------|
-| `jeeves-watcher start` | Start the filesystem watcher (foreground) |
-| `jeeves-watcher init` | Initialize a new configuration file |
-| `jeeves-watcher status` | Show watcher status |
-| `jeeves-watcher reindex` | Reindex all watched files |
-| `jeeves-watcher rebuild-metadata` | Rebuild metadata files from Qdrant payloads |
-| `jeeves-watcher search <query>` | Search the vector store |
-| `jeeves-watcher enrich <path>` | Enrich document metadata with key-value pairs |
-| `jeeves-watcher validate` | Validate the configuration |
-| `jeeves-watcher service` | Manage the watcher as a system service |
-| `jeeves-watcher config-reindex` | Reindex after configuration changes (rules only or full) |
-
-## Configuration
-
-### Environment Variable Substitution
-
-Config strings support `${VAR_NAME}` syntax for environment variable injection:
-
-```json
-{
-  "embedding": {
-    "apiKey": "${GOOGLE_API_KEY}"
-  }
-}
-```
-
-If `GOOGLE_API_KEY` is set in the environment, the value is substituted at config load time. **Unresolvable expressions are left untouched** — this allows `${...}` template syntax used in inference rule property schemas (e.g. `${frontmatter.title}`, `${file.path}`) to pass through for later resolution by the rules engine.
-
-### Watch Paths
-
-```json
-{
-  "watch": {
-    "paths": ["./docs", "./notes"],
-    "ignored": ["**/node_modules/**", "**/*.tmp"]
-  }
-}
-```
-
-- **`paths`**: Array of glob patterns or directories to watch
-- **`ignored`**: Array of patterns to exclude
-- **`respectGitignore`**: (default: `true`) Skip processing files ignored by `.gitignore` in git repositories. Nested `.gitignore` files are respected within their subtree.
-
-### Embedding Provider
-
-#### Google Gemini
-
-```json
-{
-  "embedding": {
-    "provider": "gemini",
-    "model": "gemini-embedding-001",
-    "apiKey": "${GOOGLE_API_KEY}"
-  }
-}
-```
-
-### Vector Store
-
-```json
-{
-  "vectorStore": {
-    "url": "http://localhost:6333",
-    "collectionName": "my_collection"
-  }
-}
-```
-
-### Inference Rules
-
-Automatically enrich metadata based on file patterns using declarative JSON Schemas:
-
-```json
-{
-  "schemas": {
-    "base": {
-      "type": "object",
-      "properties": {
-        "domain": {
-          "type": "string",
-          "description": "Content domain"
-        }
-      }
-    }
-  },
-  "inferenceRules": [
-    {
-      "name": "meeting-classifier",
-      "description": "Classify files under meetings directory",
-      "match": {
-        "properties": {
-          "file": {
-            "type": "object",
-            "properties": {
-              "path": { "type": "string", "glob": "**/meetings/**" }
-            }
-          }
-        }
-      },
-      "schema": [
-        "base",
-        {
-          "properties": {
-            "domain": { "set": "meetings" },
-            "category": { "type": "string", "set": "notes" }
-          }
-        }
-      ]
-    }
-  ]
-}
-```
-
-**New in v0.5.0:** Inference rules now use `schema` arrays that reference global named schemas. Type coercion automatically converts string interpolation results to declared types (integer, number, boolean, array, object). See [Inference Rules Guide](guides/inference-rules.md) for details.
-
-### Chunking
-
-Chunking settings are configured under `embedding`:
-
-```json
-{
-  "embedding": {
-    "chunkSize": 1000,
-    "chunkOverlap": 200
-  }
-}
-```
-
-### Metadata Storage
-
-```json
-{
-  "metadataDir": ".jeeves-watcher"
-}
-```
-
-Metadata is stored as JSON files alongside watched documents.
-
-## API Endpoints
-
-The watcher provides a REST API (default port: 3456):
-
-| Endpoint | Method | Description |
-|----------|--------|-------------|
-| `/status` | GET | Health check, uptime, and collection stats |
-| `/search` | POST | Semantic search (`{ query: string, limit?: number, filter?: object }`) |
-| `/metadata` | POST | Update document metadata with schema validation (`{ path: string, metadata: object }`) |
-| `/reindex` | POST | Reindex all watched files |
-| `/rebuild-metadata` | POST | Rebuild metadata files from Qdrant |
-| `/config-reindex` | POST | Reindex after config changes (`{ scope?: "rules" \| "full" }`) |
-| `/config/schema` | GET | JSON Schema of merged virtual document (v0.5.0+) |
-| `/config/query` | POST | JSONPath query over config (`{ path: string, resolve?: string[] }`) (v0.5.0+) |
-| `/config/match` | POST | Test paths against inference rules (`{ paths: string[] }`) (v0.5.0+) |
-| `/issues` | GET | Current embedding failures and processing errors (v0.5.0+) |
-
-### Example: Search
+## Install
 
 ```bash
-curl -X POST http://localhost:3456/search \
-  -H "Content-Type: application/json" \
-  -d '{"query": "machine learning algorithms", "limit": 5}'
+npm install @karmaniverous/jeeves-watcher
 ```
 
-### Example: Search With Filter
+## Quick Start
 
 ```bash
-curl -X POST http://localhost:3456/search \
-  -H "Content-Type: application/json" \
-  -d '{
-    "query": "error handling",
-    "limit": 10,
-    "filter": {
-      "must": [{ "key": "domain", "match": { "value": "backend" } }]
-    }
-  }'
-```
+# Generate a config file
+npx jeeves-watcher init --output ./jeeves-watcher.config.json
 
-### Example: Update Metadata
+# Validate it
+npx jeeves-watcher validate --config ./jeeves-watcher.config.json
 
-```bash
-curl -X POST http://localhost:3456/metadata \
-  -H "Content-Type: application/json" \
-  -d '{
-    "path": "/path/to/document.md",
-    "metadata": {
-      "priority": "high",
-      "category": "research"
-    }
-  }'
+# Start the watcher
+npx jeeves-watcher start --config ./jeeves-watcher.config.json
 ```
 
-## OpenClaw Plugin
-
-This repo ships an OpenClaw plugin that exposes the jeeves-watcher API as native agent tools:
-
-- `watcher_status` (GET `/status`)
-- `watcher_search` (POST `/search`)
-- `watcher_enrich` (POST `/metadata`)
-
-Build output:
-
-- Plugin entry: `dist/plugin/index.js`
-- Plugin manifest: `dist/plugin/openclaw.plugin.json`
-- Skill: `dist/plugin/skill/SKILL.md`
-
-Plugin configuration supports `apiUrl` (defaults to `http://127.0.0.1:3458`).
+## Documentation
 
-## Supported File Formats
+Full docs, guides, and API reference:
 
-- **Markdown** (`.md`, `.markdown`) — with YAML frontmatter support
-- **PDF** (`.pdf`) — text extraction
-- **DOCX** (`.docx`) — Microsoft Word documents
-- **HTML** (`.html`, `.htm`) — content extraction (scripts/styles removed)
-- **JSON** (`.json`) — with smart text field detection
-- **Plain Text** (`.txt`, `.text`)
+**[docs.karmanivero.us/jeeves-watcher](https://docs.karmanivero.us/jeeves-watcher)**
 
 ## License
 

package/package.json

@@ -1,17 +1,54 @@
 {
+  "name": "@karmaniverous/jeeves-watcher",
+  "version": "0.5.1",
   "author": "Jason Williscroft",
+  "description": "Filesystem watcher that keeps a Qdrant vector store in sync with document changes",
+  "license": "BSD-3-Clause",
+  "type": "module",
+  "module": "dist/index.js",
+  "types": "dist/index.d.ts",
   "bin": {
     "jeeves-watcher": "./dist/cli/jeeves-watcher/index.js"
   },
-  "auto-changelog": {
-    "output": "CHANGELOG.md",
-    "unreleased": true,
-    "commitLimit": false,
-    "hideCredit": true
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "default": "./dist/index.js"
+    }
+  },
+  "files": [
+    "dist",
+    "config.schema.json"
+  ],
+  "publishConfig": {
+    "access": "public"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/karmaniverous/jeeves-watcher.git",
+    "directory": "packages/service"
   },
   "bugs": {
     "url": "https://github.com/karmaniverous/jeeves-watcher/issues"
   },
+  "homepage": "https://github.com/karmaniverous/jeeves-watcher#readme",
+  "keywords": [
+    "filesystem",
+    "watcher",
+    "qdrant",
+    "vector-store",
+    "embeddings",
+    "semantic-search",
+    "document-indexing",
+    "rag",
+    "langchain",
+    "gemini",
+    "typescript",
+    "cli"
+  ],
+  "engines": {
+    "node": ">=20"
+  },
   "dependencies": {
     "@commander-js/extra-typings": "^14.0.0",
     "@karmaniverous/jsonmap": "^2.1.0",
@@ -23,7 +60,6 @@
     "ajv-formats": "*",
     "cheerio": "^1.2.0",
     "chokidar": "^5.0.0",
-    "commander": "^14.0.3",
     "cosmiconfig": "*",
     "dayjs": "^1.11.19",
     "fastify": "*",
@@ -47,15 +83,12 @@
     "uuid": "*",
     "zod": "^4.3.6"
   },
-  "description": "Filesystem watcher that keeps a Qdrant vector store in sync with document changes",
   "devDependencies": {
     "@dotenvx/dotenvx": "^1.52.0",
-    "@eslint/js": "^9.39.2",
     "@rollup/plugin-alias": "^6.0.0",
     "@rollup/plugin-commonjs": "^29.0.0",
     "@rollup/plugin-json": "^6.1.0",
     "@rollup/plugin-node-resolve": "^16.0.3",
-    "@rollup/plugin-terser": "^0.4.4",
     "@rollup/plugin-typescript": "^12.3.0",
     "@types/fs-extra": "^11.0.4",
     "@types/js-yaml": "*",
@@ -63,82 +96,43 @@
     "@types/picomatch": "*",
     "@types/uuid": "*",
     "@vitest/coverage-v8": "^4.0.18",
-    "@vitest/eslint-plugin": "^1.6.9",
     "auto-changelog": "^2.5.0",
     "cross-env": "^10.1.0",
-    "eslint": "^9.39.2",
-    "eslint-config-prettier": "^10.1.8",
-    "eslint-plugin-prettier": "^5.5.5",
-    "eslint-plugin-simple-import-sort": "^12.1.1",
-    "eslint-plugin-tsdoc": "^0.5.0",
     "fs-extra": "^11.3.3",
     "happy-dom": "^20.7.0",
     "knip": "^5.85.0",
-    "lefthook": "^2.1.1",
-    "prettier": "^3.8.1",
     "release-it": "^19.2.4",
-    "rimraf": "^6.1.3",
     "rollup": "^4.59.0",
     "rollup-plugin-dts": "^6.3.0",
     "tslib": "^2.8.1",
-    "tsx": "^4.21.0",
     "typedoc": "^0.28.17",
     "typedoc-plugin-mdn-links": "^5.1.1",
     "typedoc-plugin-replace-text": "^4.2.0",
-    "typescript": "^5.9.3",
-    "typescript-eslint": "^8.56.0",
     "vitest": "^4.0.18"
   },
-  "engines": {
-    "node": ">=20"
-  },
-  "exports": {
-    ".": {
-      "import": {
-        "types": "./dist/index.d.ts",
-        "default": "./dist/mjs/index.js"
-      },
-      "require": {
-        "types": "./dist/index.d.ts",
-        "default": "./dist/cjs/index.js"
-      }
-    }
-  },
-  "files": [
-    "dist",
-    "config.schema.json"
-  ],
-  "homepage": "https://github.com/karmaniverous/jeeves-watcher#readme",
-  "keywords": [
-    "filesystem",
-    "watcher",
-    "qdrant",
-    "vector-store",
-    "embeddings",
-    "semantic-search",
-    "document-indexing",
-    "rag",
-    "langchain",
-    "gemini",
-    "typescript",
-    "cli"
-  ],
-  "license": "BSD-3-Clause",
-  "main": "dist/cjs/index.js",
-  "module": "dist/mjs/index.js",
-  "openclaw": {
-    "extensions": [
-      "./dist/plugin/index.js"
-    ]
+  "scripts": {
+    "generate:schema": "tsx src/config/generate-schema.ts",
+    "build": "npm run generate:schema && rimraf dist && cross-env NO_COLOR=1 rollup --config rollup.config.ts --configPlugin @rollup/plugin-typescript",
+    "changelog": "auto-changelog",
+    "diagrams": "cd diagrams && plantuml -tpng -o ../assets -r .",
+    "knip": "knip",
+    "lint": "eslint .",
+    "lint:fix": "eslint --fix .",
+    "release": "dotenvx run -f .env.local -- release-it",
+    "release:pre": "dotenvx run -f .env.local -- release-it --no-git.requireBranch --github.prerelease --preRelease",
+    "test": "vitest run",
+    "typecheck": "tsc"
   },
-  "name": "@karmaniverous/jeeves-watcher",
-  "publishConfig": {
-    "access": "public"
+  "auto-changelog": {
+    "output": "CHANGELOG.md",
+    "unreleased": true,
+    "commitLimit": false,
+    "hideCredit": true
   },
   "release-it": {
     "git": {
       "changelog": "npx auto-changelog --unreleased-only --stdout --template https://raw.githubusercontent.com/release-it/release-it/main/templates/changelog-compact.hbs",
-      "commitMessage": "chore: release v${version}",
+      "commitMessage": "chore: release @karmaniverous/jeeves-watcher v${version}",
       "requireBranch": "main"
     },
     "github": {
@@ -153,39 +147,16 @@
       ],
       "before:npm:release": [
         "npx auto-changelog -p",
-        "npm run docs",
         "git add -A"
       ],
       "after:release": [
-        "git switch -c release/${version}",
-        "git push -u origin release/${version}",
+        "git switch -c release/service/${version}",
+        "git push -u origin release/service/${version}",
         "git switch ${branchName}"
       ]
     },
     "npm": {
       "publish": true
     }
-  },
-  "repository": {
-    "type": "git",
-    "url": "git+https://github.com/karmaniverous/jeeves-watcher.git"
-  },
-  "scripts": {
-    "generate:schema": "tsx src/config/generate-schema.ts",
-    "build:skills": "node scripts/build-skills.js",
-    "build": "npm run generate:schema && rimraf dist && cross-env NO_COLOR=1 rollup --config rollup.config.ts --configPlugin @rollup/plugin-typescript && npm run build:skills && node -e \"const fs=require('fs-extra');fs.copySync('plugin/openclaw.plugin.json','dist/plugin/openclaw.plugin.json');\"",
-    "changelog": "auto-changelog",
-    "diagrams": "cd diagrams && plantuml -tpng -o ../assets -r .",
-    "docs": "typedoc",
-    "knip": "knip",
-    "lint": "eslint .",
-    "lint:fix": "eslint --fix .",
-    "release": "dotenvx run -f .env.local -- release-it",
-    "release:pre": "dotenvx run -f .env.local -- release-it --no-git.requireBranch --github.prerelease --preRelease",
-    "test": "vitest run",
-    "typecheck": "tsc"
-  },
-  "type": "module",
-  "types": "dist/index.d.ts",
-  "version": "0.5.0-1"
+  }
 }

package/LICENSE

@@ -1,28 +0,0 @@
-BSD 3-Clause License
-
-Copyright (c) 2025, Jason Williscroft
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-1. Redistributions of source code must retain the above copyright notice, this
-   list of conditions and the following disclaimer.
-
-2. Redistributions in binary form must reproduce the above copyright notice,
-   this list of conditions and the following disclaimer in the documentation
-   and/or other materials provided with the distribution.
-
-3. Neither the name of the copyright holder nor the names of its
-   contributors may be used to endorse or promote products derived from
-   this software without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
-AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
-FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
-DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
-SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
-CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
-OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.