@karmaniverous/jeeves-watcher 0.5.0 → 0.6.0

package/README.md

@@ -1,300 +1,42 @@
 # @karmaniverous/jeeves-watcher
 
-Filesystem watcher that keeps a Qdrant vector store in sync with document changes.
+Filesystem watcher that keeps a [Qdrant](https://qdrant.tech/) vector store in sync with document changes. Extract text from files, chunk it, generate embeddings, and query your documents with semantic search.
 
-## Overview
+## Features
 
-`jeeves-watcher` monitors a configured set of directories for file changes, extracts text content, generates embeddings, and maintains a synchronized Qdrant vector store for semantic search. It automatically:
+- **Filesystem watching** — monitors directories for file changes via [chokidar](https://github.com/paulmillr/chokidar)
+- **Multi-format extraction** — PDF, HTML, DOCX, Markdown, plain text, and more
+- **Configurable chunking** — token-based text splitting with overlap control
+- **Embedding providers** — Gemini, OpenAI, or mock (for testing)
+- **Qdrant sync** — automatic upsert/delete keeps the vector store current
+- **Rules engine** — glob-based inference rules for metadata enrichment
+- **REST API** — Fastify server for search, status, config, and management
+- **CLI** — `jeeves-watcher init`, `validate`, `start`, and more
 
-- **Watches** directories for file additions, modifications, and deletions
-- **Extracts** text from various formats (Markdown, PDF, DOCX, HTML, JSON, plain text)
-- **Chunks** large documents for optimal embedding
-- **Embeds** content using configurable providers (Google Gemini, mock for testing)
-- **Syncs** to Qdrant for fast semantic search
-- **Enriches** metadata via rules and API endpoints
-
-### Architecture
-
-![System Architecture](assets/system-architecture.png)
-
-For detailed architecture documentation, see [guides/architecture.md](guides/architecture.md).
-
-## Quick Start
-
-### Installation
-
-```bash
-npm install -g @karmaniverous/jeeves-watcher
-```
-
-### Initialize Configuration
-
-Create a new configuration file in your project:
-
-```bash
-jeeves-watcher init
-```
-
-This generates a `jeeves-watcher.config.json` file with sensible defaults.
-
-### Configure
-
-Edit `jeeves-watcher.config.json` to specify:
-
-- **Watch paths**: Directories to monitor
-- **Embedding provider**: Google Gemini or mock (for testing)
-- **Qdrant connection**: URL and collection name
-- **Inference rules**: Automatic metadata enrichment based on file patterns
-
-Example minimal configuration:
-
-```json
-{
-  "watch": {
-    "paths": ["./docs"],
-    "ignored": ["**/node_modules/**", "**/.git/**"]
-  },
-  "embedding": {
-    "provider": "gemini",
-    "model": "gemini-embedding-001",
-    "apiKey": "${GOOGLE_API_KEY}"
-  },
-  "vectorStore": {
-    "url": "http://localhost:6333",
-    "collectionName": "my_docs"
-  }
-}
-```
-
-### Start Watching
-
-```bash
-jeeves-watcher start
-```
-
-The watcher will:
-1. Index all existing files in watched directories
-2. Monitor for changes
-3. Update Qdrant automatically
-
-## CLI Commands
-
-| Command | Description |
-|---------|-------------|
-| `jeeves-watcher start` | Start the filesystem watcher (foreground) |
-| `jeeves-watcher init` | Initialize a new configuration file |
-| `jeeves-watcher status` | Show watcher status |
-| `jeeves-watcher reindex` | Reindex all watched files |
-| `jeeves-watcher rebuild-metadata` | Rebuild metadata files from Qdrant payloads |
-| `jeeves-watcher search <query>` | Search the vector store |
-| `jeeves-watcher enrich <path>` | Enrich document metadata with key-value pairs |
-| `jeeves-watcher validate` | Validate the configuration |
-| `jeeves-watcher service` | Manage the watcher as a system service |
-| `jeeves-watcher config-reindex` | Reindex after configuration changes (rules only or full) |
-
-## Configuration
-
-### Environment Variable Substitution
-
-Config strings support `${VAR_NAME}` syntax for environment variable injection:
-
-```json
-{
-  "embedding": {
-    "apiKey": "${GOOGLE_API_KEY}"
-  }
-}
-```
-
-If `GOOGLE_API_KEY` is set in the environment, the value is substituted at config load time. **Unresolvable expressions are left untouched** — this allows `${...}` template syntax used in inference rule property schemas (e.g. `${frontmatter.title}`, `${file.path}`) to pass through for later resolution by the rules engine.
-
-### Watch Paths
-
-```json
-{
-  "watch": {
-    "paths": ["./docs", "./notes"],
-    "ignored": ["**/node_modules/**", "**/*.tmp"]
-  }
-}
-```
-
-- **`paths`**: Array of glob patterns or directories to watch
-- **`ignored`**: Array of patterns to exclude
-- **`respectGitignore`**: (default: `true`) Skip processing files ignored by `.gitignore` in git repositories. Nested `.gitignore` files are respected within their subtree.
-
-### Embedding Provider
-
-#### Google Gemini
-
-```json
-{
-  "embedding": {
-    "provider": "gemini",
-    "model": "gemini-embedding-001",
-    "apiKey": "${GOOGLE_API_KEY}"
-  }
-}
-```
-
-### Vector Store
-
-```json
-{
-  "vectorStore": {
-    "url": "http://localhost:6333",
-    "collectionName": "my_collection"
-  }
-}
-```
-
-### Inference Rules
-
-Automatically enrich metadata based on file patterns using declarative JSON Schemas:
-
-```json
-{
-  "schemas": {
-    "base": {
-      "type": "object",
-      "properties": {
-        "domain": {
-          "type": "string",
-          "description": "Content domain"
-        }
-      }
-    }
-  },
-  "inferenceRules": [
-    {
-      "name": "meeting-classifier",
-      "description": "Classify files under meetings directory",
-      "match": {
-        "properties": {
-          "file": {
-            "type": "object",
-            "properties": {
-              "path": { "type": "string", "glob": "**/meetings/**" }
-            }
-          }
-        }
-      },
-      "schema": [
-        "base",
-        {
-          "properties": {
-            "domain": { "set": "meetings" },
-            "category": { "type": "string", "set": "notes" }
-          }
-        }
-      ]
-    }
-  ]
-}
-```
-
-**New in v0.5.0:** Inference rules now use `schema` arrays that reference global named schemas. Type coercion automatically converts string interpolation results to declared types (integer, number, boolean, array, object). See [Inference Rules Guide](guides/inference-rules.md) for details.
-
-### Chunking
-
-Chunking settings are configured under `embedding`:
-
-```json
-{
-  "embedding": {
-    "chunkSize": 1000,
-    "chunkOverlap": 200
-  }
-}
-```
-
-### Metadata Storage
-
-```json
-{
-  "metadataDir": ".jeeves-watcher"
-}
-```
-
-Metadata is stored as JSON files alongside watched documents.
-
-## API Endpoints
-
-The watcher provides a REST API (default port: 3456):
-
-| Endpoint | Method | Description |
-|----------|--------|-------------|
-| `/status` | GET | Health check, uptime, and collection stats |
-| `/search` | POST | Semantic search (`{ query: string, limit?: number, filter?: object }`) |
-| `/metadata` | POST | Update document metadata with schema validation (`{ path: string, metadata: object }`) |
-| `/reindex` | POST | Reindex all watched files |
-| `/rebuild-metadata` | POST | Rebuild metadata files from Qdrant |
-| `/config-reindex` | POST | Reindex after config changes (`{ scope?: "rules" \| "full" }`) |
-| `/config/schema` | GET | JSON Schema of merged virtual document (v0.5.0+) |
-| `/config/query` | POST | JSONPath query over config (`{ path: string, resolve?: string[] }`) (v0.5.0+) |
-| `/config/match` | POST | Test paths against inference rules (`{ paths: string[] }`) (v0.5.0+) |
-| `/issues` | GET | Current embedding failures and processing errors (v0.5.0+) |
-
-### Example: Search
+## Install
 
 ```bash
-curl -X POST http://localhost:3456/search \
-  -H "Content-Type: application/json" \
-  -d '{"query": "machine learning algorithms", "limit": 5}'
+npm install @karmaniverous/jeeves-watcher
 ```
 
-### Example: Search With Filter
+## Quick Start
 
 ```bash
-curl -X POST http://localhost:3456/search \
-  -H "Content-Type: application/json" \
-  -d '{
-    "query": "error handling",
-    "limit": 10,
-    "filter": {
-      "must": [{ "key": "domain", "match": { "value": "backend" } }]
-    }
-  }'
-```
+# Generate a config file
+npx jeeves-watcher init --output ./jeeves-watcher.config.json
 
-### Example: Update Metadata
+# Validate it
+npx jeeves-watcher validate --config ./jeeves-watcher.config.json
 
-```bash
-curl -X POST http://localhost:3456/metadata \
-  -H "Content-Type: application/json" \
-  -d '{
-    "path": "/path/to/document.md",
-    "metadata": {
-      "priority": "high",
-      "category": "research"
-    }
-  }'
+# Start the watcher
+npx jeeves-watcher start --config ./jeeves-watcher.config.json
 ```
 
-## OpenClaw Plugin
-
-This repo ships an OpenClaw plugin that exposes the jeeves-watcher API as native agent tools:
-
-- `watcher_status` (GET `/status`)
-- `watcher_search` (POST `/search`)
-- `watcher_enrich` (POST `/metadata`)
-
-Build output:
-
-- Plugin entry: `dist/plugin/index.js`
-- Plugin manifest: `dist/plugin/openclaw.plugin.json`
-- Skill: `dist/plugin/skill/SKILL.md`
-
-Plugin configuration supports `apiUrl` (defaults to `http://127.0.0.1:3458`).
+## Documentation
 
-## Supported File Formats
+Full docs, guides, and API reference:
 
-- **Markdown** (`.md`, `.markdown`) — with YAML frontmatter support
-- **PDF** (`.pdf`) — text extraction
-- **DOCX** (`.docx`) — Microsoft Word documents
-- **HTML** (`.html`, `.htm`) — content extraction (scripts/styles removed)
-- **JSON** (`.json`) — with smart text field detection
-- **Plain Text** (`.txt`, `.text`)
+**[docs.karmanivero.us/jeeves-watcher](https://docs.karmanivero.us/jeeves-watcher)**
 
 ## License
 

package/config.schema.json

@@ -138,7 +138,7 @@
       ]
     },
     "inferenceRules": {
-      "description": "Rules for inferring metadata from file attributes.",
+      "description": "Rules for inferring metadata from file attributes. Each entry may be an inline rule object or a file path to a JSON rule file (resolved relative to config directory).",
       "allOf": [
         {
           "$ref": "#/definitions/__schema49"
@@ -194,7 +194,7 @@
       ]
     },
     "search": {
-      "description": "Search configuration including score thresholds.",
+      "description": "Search configuration including score thresholds and hybrid search.",
       "allOf": [
         {
           "$ref": "#/definitions/__schema63"
@@ -579,57 +579,64 @@
     "__schema49": {
       "type": "array",
       "items": {
-        "type": "object",
-        "properties": {
-          "name": {
-            "type": "string",
-            "minLength": 1,
-            "description": "Unique name identifying this inference rule."
-          },
-          "description": {
-            "type": "string",
-            "minLength": 1,
-            "description": "Human-readable description of what this rule does."
-          },
-          "match": {
+        "anyOf": [
+          {
             "type": "object",
-            "propertyNames": {
-              "$ref": "#/definitions/__schema50"
-            },
-            "additionalProperties": {
-              "$ref": "#/definitions/__schema51"
-            },
-            "description": "JSON Schema object to match against file attributes."
-          },
-          "schema": {
-            "description": "Array of schema references (named schema refs or inline objects) merged left-to-right.",
-            "allOf": [
-              {
-                "$ref": "#/definitions/__schema52"
-              }
-            ]
-          },
-          "map": {
-            "description": "JsonMap transformation (inline definition, named map reference, or .json file path).",
-            "allOf": [
-              {
-                "$ref": "#/definitions/__schema53"
+            "properties": {
+              "name": {
+                "type": "string",
+                "minLength": 1,
+                "description": "Unique name identifying this inference rule."
+              },
+              "description": {
+                "type": "string",
+                "minLength": 1,
+                "description": "Human-readable description of what this rule does."
+              },
+              "match": {
+                "type": "object",
+                "propertyNames": {
+                  "$ref": "#/definitions/__schema50"
+                },
+                "additionalProperties": {
+                  "$ref": "#/definitions/__schema51"
+                },
+                "description": "JSON Schema object to match against file attributes."
+              },
+              "schema": {
+                "description": "Array of schema references (named schema refs or inline objects) merged left-to-right.",
+                "allOf": [
+                  {
+                    "$ref": "#/definitions/__schema52"
+                  }
+                ]
+              },
+              "map": {
+                "description": "JsonMap transformation (inline definition, named map reference, or .json file path).",
+                "allOf": [
+                  {
+                    "$ref": "#/definitions/__schema53"
+                  }
+                ]
+              },
+              "template": {
+                "description": "Handlebars content template (inline string, named ref, or .hbs/.handlebars file path).",
+                "allOf": [
+                  {
+                    "$ref": "#/definitions/__schema56"
+                  }
+                ]
               }
+            },
+            "required": [
+              "name",
+              "description",
+              "match"
             ]
           },
-          "template": {
-            "description": "Handlebars content template (inline string, named ref, or .hbs/.handlebars file path).",
-            "allOf": [
-              {
-                "$ref": "#/definitions/__schema56"
-              }
-            ]
+          {
+            "type": "string"
           }
-        },
-        "required": [
-          "name",
-          "description",
-          "match"
         ]
       }
     },
@@ -896,6 +903,21 @@
             "relevant",
             "noise"
           ]
+        },
+        "hybrid": {
+          "type": "object",
+          "properties": {
+            "enabled": {
+              "default": false,
+              "type": "boolean"
+            },
+            "textWeight": {
+              "default": 0.3,
+              "type": "number",
+              "minimum": 0,
+              "maximum": 1
+            }
+          }
         }
       }
     },