cursor-recursive-rag 0.2.0-alpha.3 → 0.2.0

package/README.md

@@ -5,23 +5,39 @@
 [![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
 [![Node](https://img.shields.io/badge/node-%3E%3D20.0.0-brightgreen.svg)](https://nodejs.org)
 
-Recursive RAG MCP server for Cursor IDE with interactive setup wizard and web dashboard. Build a knowledge base from your documentation and codebase, enabling multi-hop retrieval and iterative query refinement.
+Recursive RAG MCP server for Cursor IDE with interactive setup wizard, web dashboard, and AI-powered rules optimizer. Build a knowledge base from your documentation and codebase, enabling multi-hop retrieval, iterative query refinement, and intelligent rule management.
 
-> **Status:** Alpha - Core features working, API may change
+> **Status:** Beta - Core features stable, actively maintained
 
 ## Features
 
+### Core RAG
 - **Recursive Query**: Multi-hop retrieval with query decomposition and iterative refinement
-- **Configurable Vector Stores**: Redis Stack (Docker), Redis 8.x Native (Homebrew), Qdrant (local/cloud), ChromaDB, Cloudflare Vectorize
+- **Configurable Vector Stores**: Redis Stack, Redis 8.x Native, Qdrant, ChromaDB, Cloudflare Vectorize, Memory
 - **Configurable Embeddings**: Local (Xenova/transformers.js), OpenAI, Ollama
 - **Web Crawling**: Firecrawl integration for documentation ingestion
-- **Rotating Proxy Support**: Optional PacketStream/SmartProxy integration for URL fetching
-- **Web Dashboard**: Real-time monitoring, search, activity logging, and configuration UI (Tailwind CSS)
-- **Activity Logging**: Persistent activity log across MCP tools and dashboard (stored in `~/.cursor-rag/activity.json`)
-- **Interactive Setup**: Guided configuration wizard
+- **Rotating Proxy Support**: PacketStream/Decodo integration for URL fetching
+
+### Dashboard
+- **Web Dashboard**: Real-time monitoring, search, activity logging, and configuration
+- **Tools Panel**: 12+ built-in tools with parameter forms and execution history
+- **Rules Optimizer**: Analyze, detect duplicates, and optimize Cursor rules
+- **Settings**: Configure vector stores, embeddings, proxy, and LLM providers
+- **Modal/Toast System**: Modern in-app notifications (no browser alerts)
+
+### Rules Optimizer (NEW)
+- **Analyze Rules**: Detect duplicates, conflicts, and outdated patterns
+- **LLM-Powered Merging**: Intelligently merge duplicate rules preserving all content
+- **Natural Language Rules**: Define custom rules in plain English
+- **Folder Browser**: Server-side navigation to select rules folders
+- **Pattern Matching**: Version checks, deprecation patterns (works without LLM)
+- **Automatic Backups**: Creates backups before applying changes
+
+### Integrations
 - **MCP Integration**: Automatic registration with Cursor IDE
-- **MCP Gateway Integration**: Connect to [MCP Gateway](https://github.com/abdullah1854/MCPGateway) for 87+ aggregated tools with token optimization
-- **OpenSkills Integration**: Auto-discover and ingest skills from [OpenSkills](https://github.com/numman-ali/openskills) for semantic search
+- **MCP Gateway**: Connect to 87+ aggregated tools with token optimization
+- **OpenSkills**: Auto-discover and ingest skills for semantic search
+- **LLM Providers**: OpenAI, Anthropic, DeepSeek, Groq, Ollama, OpenRouter
 
 ## Quick Start
 
@@ -32,20 +48,14 @@ npm install -g cursor-recursive-rag
 # Run interactive setup
 cursor-rag setup
 
-# Ingest documentation (with Firecrawl)
+# Ingest documentation
 cursor-rag ingest https://nextjs.org/docs --crawl --max-pages 200
 
-# Ingest a local directory
-cursor-rag ingest ./docs
-
-# Check status
-cursor-rag status
-
-# Test search from CLI
-cursor-rag search "how to authenticate users"
-
 # Start the web dashboard
 cursor-rag dashboard
+
+# Analyze Cursor rules
+cursor-rag rules analyze ~/.cursor/rules
 ```
 
 ## Installation
@@ -63,135 +73,106 @@ git clone https://github.com/garethdaine/cursor-recursive-rag.git
 cd cursor-recursive-rag
 npm install
 npm run build
-npm link  # Makes cursor-rag available globally
+npm link
 ```
 
-## Setup Wizard
-
-Run `cursor-rag setup` to configure the system. The wizard will:
-
-1. **Select Vector Store**:
-   - **Redis Stack** (default): Docker-based with RediSearch module, `docker run -p 6379:6379 redis/redis-stack-server`
-   - **Redis 8.x Native**: Homebrew/native Redis 8.x with VADD/VSIM commands, `brew install redis`
-   - **Qdrant**: Persistent, local Docker or cloud, `docker run -p 6333:6333 qdrant/qdrant`
-   - **Memory**: In-process, file-based persistence (testing/development)
-   - **ChromaDB**: Requires separate server, `docker run -p 8000:8000 chromadb/chroma`
-   - **Cloudflare Vectorize**: Serverless (requires Cloudflare account)
-
-2. **Select Embedding Model**:
-   - **Xenova (local)**: Free, private, ~384 dimensions
-   - **Ollama**: Local, configurable models
-   - **OpenAI**: High quality, requires API key
-
-3. **API Keys**:
-   - OpenAI API key (if using OpenAI embeddings)
-   - Firecrawl API key (for web crawling)
-   - Qdrant URL/API key (if using Qdrant)
-   - Ollama URL/model (if using Ollama)
-
-4. **Auto-register** the MCP server with Cursor IDE
-
 ## CLI Commands
 
-### `cursor-rag setup`
-
-Interactive setup wizard to configure vector store, embeddings, proxy, and API keys.
+### Core Commands
 
 ```bash
-cursor-rag setup
-cursor-rag setup --vector-store chroma --embeddings xenova
+cursor-rag setup              # Interactive configuration wizard
+cursor-rag status             # Show configuration and statistics
+cursor-rag dashboard          # Start web dashboard (default: http://localhost:3333)
 ```
 
-### `cursor-rag ingest`
-
-Add documents to the knowledge base.
+### Ingestion
 
 ```bash
-# Crawl a website (requires Firecrawl API key)
-cursor-rag ingest https://docs.example.com --crawl --max-pages 100 --max-depth 3
-
-# Ingest a single URL
-cursor-rag ingest https://example.com/page
-
-# Ingest a local file
-cursor-rag ingest ./document.md
-
-# Ingest a directory (recursive)
-cursor-rag ingest ./docs
+cursor-rag ingest https://docs.example.com --crawl --max-pages 100
+cursor-rag ingest ./docs                    # Local directory
+cursor-rag ingest ./document.md             # Single file
 ```
 
-### `cursor-rag search`
-
-Test search from the command line.
+### Search
 
 ```bash
-cursor-rag search "how to set up authentication"
+cursor-rag search "how to authenticate users"
 cursor-rag search "database queries" --top-k 10
 ```
 
-### `cursor-rag status`
-
-Show current configuration and knowledge base statistics.
+### Chat History
 
 ```bash
-cursor-rag status
+cursor-rag chat list          # List Cursor conversations
+cursor-rag chat ingest        # Ingest chat history into RAG
+cursor-rag chat watch         # Watch for new conversations
+cursor-rag chat stats         # Show ingestion statistics
 ```
 
-### `cursor-rag dashboard`
-
-Start the web dashboard for monitoring and configuration.
+### Rules Optimizer
 
 ```bash
-cursor-rag dashboard
-cursor-rag dashboard --port 8080
+cursor-rag rules list <folder>       # List all rules
+cursor-rag rules analyze <folder>    # Analyze without changes
+cursor-rag rules duplicates <folder> # Show duplicates only
+cursor-rag rules conflicts <folder>  # Show conflicts only
+cursor-rag rules outdated <folder>   # Show outdated rules
+cursor-rag rules optimize <folder>   # Full optimization (dry-run)
+cursor-rag rules merge <folder>      # LLM-powered merge
+cursor-rag rules rewrite <folder>    # LLM-powered rewrite
 ```
 
-The dashboard provides:
-- **Overview**: Total chunks, vector store stats, MCP Gateway status, OpenSkills count
-- **Search**: Test queries against your knowledge base
-- **MCP Gateway**: Browse and search 87+ aggregated tools from connected backends
-- **OpenSkills**: Browse and search installed skills
-- **Activity Log**: Persistent activity log showing all searches, ingestions, and crawls (shared with MCP tools)
-- **Settings**: Configure vector store (Redis Stack, Redis 8.x, Qdrant, ChromaDB, etc.), embeddings, and proxy settings
+### Maintenance
 
-## Usage in Cursor
-
-### Option 1: Via `@Docs` (Recommended)
+```bash
+cursor-rag maintenance run <job>  # Run maintenance job
+cursor-rag maintenance start      # Start scheduler
+cursor-rag maintenance stats      # Show statistics
+cursor-rag maintenance cleanup    # Clean stale data
+```
 
-The dashboard serves your knowledge base as browsable documentation that Cursor can index:
+## Web Dashboard
 
-1. Start the dashboard: `cursor-rag dashboard`
-2. In Cursor, type `@Docs` and select **Add new doc**
-3. Enter: `http://localhost:3333/docs`
-4. Now use `@Docs cursor-recursive-rag` in your prompts!
+Start with `cursor-rag dashboard` (default: http://localhost:3333)
 
-### Option 2: Via MCP Tools (Natural Language)
+### Tabs
 
-The MCP tools are available to the AI agent in Cursor Chat. Ask questions naturally and the AI will use the appropriate tools:
+| Tab | Features |
+|-----|----------|
+| **Overview** | Stats, connection status, quick actions |
+| **Search** | Query knowledge base with results display |
+| **MCP Gateway** | Browse 87+ tools from connected backends |
+| **OpenSkills** | Browse and search installed skills |
+| **Tools** | Execute built-in tools with forms |
+| **Activity** | Persistent log of all operations |
+| **Settings** | Configure all system options |
 
-```
-Search my knowledge base for authentication patterns
+### Rules Optimizer (Tools Tab)
 
-What sources are indexed in my RAG knowledge base?
+The Rules Optimizer panel provides one-click analysis and optimization:
 
-Crawl and ingest https://docs.example.com into my knowledge base with a max of 50 pages
-```
+1. **Select Folder**: Browse or enter path to rules folder
+2. **Choose Mode**: Dry Run (preview) or Apply Changes
+3. **Run Optimizer**: Analyzes duplicates, conflicts, outdated rules
+4. **Review Results**: See all issues with severity indicators
 
-> **Note:** MCP tools don't use `@` syntax - that's reserved for Cursor's built-in features (`@Codebase`, `@Docs`, `@Files`). MCP tools are called automatically by the AI when relevant to your request.
+**Works with or without LLM**:
+- Without LLM: Pattern matching detects issues, reports for manual review
+- With LLM: Automatically merges duplicates preserving all content
 
-### Available MCP Tools
+### Settings Tab
 
-| Tool | Description |
-|------|-------------|
-| `recursive_query` | Multi-hop retrieval with query decomposition |
-| `search_knowledge` | Direct vector similarity search |
-| `ingest_document` | Add a single document (URL, file, or text) |
-| `crawl_and_ingest` | Crawl website with Firecrawl and index |
-| `list_sources` | List indexed document sources |
+Configure:
+- **Vector Store**: Redis Stack, Redis 8.x, Qdrant, ChromaDB, Memory, Vectorize
+- **Embeddings**: Xenova (local), OpenAI, Ollama
+- **Proxy**: PacketStream, Decodo with credentials
+- **Rules Analyzer**: Thresholds, patterns, LLM provider
+- **LLM Provider**: OpenAI, Anthropic, DeepSeek, Groq, Ollama, OpenRouter
 
 ## Configuration
 
-Configuration is stored in `~/.cursor-rag/config.json`:
+Configuration stored in `~/.cursor-rag/config.json`:
 
 ```json
 {
@@ -199,26 +180,12 @@ Configuration is stored in `~/.cursor-rag/config.json`:
   "embeddings": "xenova",
   "apiKeys": {
     "firecrawl": "fc-...",
-    "redis": {
-      "url": "redis://localhost:6379"
-    }
-  },
-  "proxy": {
-    "enabled": false,
-    "driver": "none"
+    "redis": { "url": "redis://localhost:6379" }
   },
-  "dashboard": {
-    "enabled": true,
-    "port": 3333
-  },
-  "mcpGateway": {
-    "enabled": true,
-    "url": "http://localhost:3010"
-  },
-  "openSkills": {
-    "enabled": true,
-    "autoIngestSkills": true
-  }
+  "proxy": { "enabled": false },
+  "dashboard": { "enabled": true, "port": 3333 },
+  "mcpGateway": { "enabled": true, "url": "http://localhost:3010" },
+  "openSkills": { "enabled": true, "autoIngestSkills": true }
 }
 ```
 
@@ -226,83 +193,70 @@ Configuration is stored in `~/.cursor-rag/config.json`:
 
 | Type | Description | Setup |
 |------|-------------|-------|
-| `redis-stack` | Redis Stack with RediSearch module (Docker) | `docker run -d -p 6379:6379 redis/redis-stack-server` |
-| `redis` | Redis 8.x native vectors (Homebrew) | `brew install redis && brew services start redis` |
+| `redis-stack` | Redis + RediSearch (Docker) | `docker run -d -p 6379:6379 redis/redis-stack-server` |
+| `redis` | Redis 8.x native vectors | `brew install redis` |
 | `qdrant` | Qdrant vector database | `docker run -d -p 6333:6333 qdrant/qdrant` |
 | `chroma` | ChromaDB | `docker run -d -p 8000:8000 chromadb/chroma` |
 | `memory` | In-memory with file persistence | No setup required |
 | `vectorize` | Cloudflare Vectorize | Requires Cloudflare account |
 
-### Activity Logging
-
-All operations (searches, ingestions, crawls) are logged to `~/.cursor-rag/activity.json`. This log is shared between the MCP server and dashboard, so activities from Cursor IDE chats will appear in the dashboard's Activity tab.
-
-### Proxy Configuration
-
-The optional rotating proxy is used for direct URL fetching (not needed when using Firecrawl, which handles proxying internally). Supported providers:
+### Rules Analyzer Config
 
-- **PacketStream**: Residential proxies with country targeting
-- **SmartProxy**: Datacenter and residential options
-
-### MCP Gateway Integration
-
-Connect to [MCP Gateway](https://github.com/abdullah1854/MCPGateway) to access 87+ aggregated tools with token optimization:
+Stored in `~/.cursor-rag/rules-config.json`:
 
 ```json
 {
-  "mcpGateway": {
-    "enabled": true,
-    "url": "http://localhost:3010",
-    "apiKey": "optional-api-key"
-  }
+  "analysis": {
+    "duplicateThreshold": 0.7,
+    "maxAgeDays": 365,
+    "detectConflicts": true,
+    "detectOutdated": true,
+    "useLLM": false
+  },
+  "llm": {
+    "provider": "openai",
+    "model": "gpt-4o-mini"
+  },
+  "versionChecks": [],
+  "deprecationPatterns": [],
+  "naturalRules": []
 }
 ```
 
-**MCP Tools exposed:**
-- `gateway_search_tools` - Search available tools across all backends
-- `gateway_call_tool` - Call any gateway tool with result filtering
-- `gateway_execute_skill` - Execute gateway skills
-- `gateway_health` - Check gateway status
+## MCP Tools
 
-### OpenSkills Integration
+Available when using Cursor IDE:
 
-Connect to [OpenSkills](https://github.com/numman-ali/openskills) for universal skills loading:
+| Tool | Description |
+|------|-------------|
+| `recursive_query` | Multi-hop retrieval with query decomposition |
+| `search_knowledge` | Direct vector similarity search |
+| `ingest_document` | Add document (URL, file, text) |
+| `crawl_and_ingest` | Crawl website and index |
+| `list_sources` | List indexed sources |
+| `chat_ingest` | Ingest Cursor chat history |
+| `chat_list` | List conversations |
+| `memory_stats` | Memory system statistics |
+| `gateway_*` | MCP Gateway tools |
+| `openskills_*` | OpenSkills tools |
 
-```json
-{
-  "openSkills": {
-    "enabled": true,
-    "autoIngestSkills": true
-  }
-}
-```
+## Usage in Cursor
 
-**MCP Tools exposed:**
-- `list_openskills` - List all installed skills
-- `read_openskill` - Read a specific skill's content
-- `ingest_openskills` - Ingest all skills into RAG knowledge base
-- `search_openskills` - Semantic search across ingested skills
+### Via @Docs (Recommended)
 
-**Skill Discovery Paths (priority order):**
-1. `./.agent/skills/` (project universal)
-2. `~/.agent/skills/` (global universal)
-3. `./.claude/skills/` (project Claude)
-4. `~/.claude/skills/` (global Claude)
+1. Start dashboard: `cursor-rag dashboard`
+2. In Cursor: `@Docs` → **Add new doc**
+3. Enter: `http://localhost:3333/docs`
+4. Use: `@Docs cursor-recursive-rag` in prompts
 
-The MCP server is registered in `~/.cursor/mcp.json`:
+### Via MCP Tools
+
+Ask naturally and the AI will use appropriate tools:
 
-```json
-{
-  "mcpServers": {
-    "recursive-rag": {
-      "command": "node",
-      "args": ["cursor-recursive-rag/dist/server/index.js"],
-      "env": {
-        "CURSOR_RAG_CONFIG": "/Users/you/.cursor-rag/config.json"
-      }
-    }
-  }
-}
+```
+Search my knowledge base for authentication patterns
+Crawl and ingest https://docs.example.com with max 50 pages
+What sources are indexed in my RAG?
 ```
 
 ## Architecture
@@ -310,71 +264,64 @@ The MCP server is registered in `~/.cursor/mcp.json`:
 ```
 cursor-recursive-rag/
 ├── src/
-│   ├── cli/               # CLI commands (setup, ingest, search, status)
-│   ├── server/            # MCP server and tool handlers
+│   ├── cli/               # CLI commands
+│   ├── server/            # MCP server and tools
+│   ├── dashboard/         # Web dashboard
 │   ├── adapters/
-│   │   ├── vector/        # Vector store adapters (Chroma, Qdrant, Vectorize)
-│   │   └── embeddings/    # Embedding adapters (Xenova, OpenAI, Ollama)
-│   ├── services/          # Chunker, query decomposer, config
-│   └── types/             # TypeScript type definitions
+│   │   ├── vector/        # Vector store adapters
+│   │   ├── embeddings/    # Embedding adapters
+│   │   └── llm/           # LLM provider adapters
+│   ├── services/          # Core services
+│   ├── config/            # Configuration schemas
+│   └── types/             # TypeScript definitions
 ├── bin/                   # CLI entry point
-├── dist/                  # Compiled JavaScript
-└── package.json
+└── dist/                  # Compiled JavaScript
 ```
 
 ## Requirements
 
 - Node.js >= 20.0.0
 - Cursor IDE (for MCP integration)
-- One of the following vector stores:
-  - Docker (for Redis Stack, Qdrant, ChromaDB)
-  - Redis 8.x (via Homebrew: `brew install redis`)
-  - Memory adapter (no dependencies, for testing)
-- Optional: Ollama (for local embeddings)
+- Vector store (Docker or Redis 8.x recommended)
+- Optional: Ollama (local embeddings), LLM API key (rules optimization)
 
 ## API Keys
 
 | Service | Purpose | Get Key |
 |---------|---------|---------|
 | Firecrawl | Web crawling | https://www.firecrawl.dev |
-| OpenAI | Embeddings | https://platform.openai.com |
+| OpenAI | Embeddings/LLM | https://platform.openai.com |
+| Anthropic | LLM | https://console.anthropic.com |
 | Qdrant Cloud | Vector store | https://cloud.qdrant.io |
 
 ## Troubleshooting
 
-### "Config file not found"
-
-Run `cursor-rag setup` to create the configuration.
-
-### MCP server not showing in Cursor
+### Config file not found
+Run `cursor-rag setup` to create configuration.
 
-1. Check `~/.cursor/mcp.json` has the `recursive-rag` entry
+### MCP server not in Cursor
+1. Check `~/.cursor/mcp.json` has `recursive-rag` entry
 2. Restart Cursor IDE
-3. Check the server path is correct
+3. Verify server path is correct
 
-### Firecrawl errors
+### Rules optimizer shows 0 rules
+Ensure the path is absolute (e.g., `/Users/you/.cursor/rules` not `~/.cursor/rules`)
 
-Ensure your Firecrawl API key is valid and starts with `fc-`.
-
-### ChromaDB permission errors
-
-The local ChromaDB stores data in `~/.cursor-rag/chroma-data`. Ensure this directory is writable.
+### LLM not configured error
+Either disable "Use LLM for Analysis" in Settings, or configure an LLM provider.
 
 ## Development
 
 ```bash
-# Install dependencies
-npm install
-
-# Build
-npm run build
+npm install          # Install dependencies
+npm run build        # Build TypeScript
+npm run dev          # Watch mode
+npm link             # Link for testing
+```
 
-# Development mode (watch)
-npm run dev
+## Contributing
 
-# Link for testing
-npm link
-```
+See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
 
 ## License
 

package/dist/adapters/llm/anthropic.d.ts

@@ -0,0 +1,27 @@
+/**
+ * Anthropic LLM Provider
+ *
+ * Supports: Claude 3.5 Sonnet, Claude 3.5 Haiku, Claude 3 Opus
+ */
+import { BaseLLMProvider } from './base.js';
+import type { LLMMessage, LLMCompletionOptions, LLMResponse, ModelInfo, ModelCapabilities, AnthropicProviderConfig } from '../../types/llmProvider.js';
+export declare class AnthropicProvider extends BaseLLMProvider {
+    readonly type: "anthropic";
+    readonly name = "Anthropic";
+    private apiKey;
+    private baseUrl;
+    private model;
+    private anthropicVersion;
+    constructor(config: AnthropicProviderConfig);
+    isAvailable(): Promise<boolean>;
+    listModels(): Promise<ModelInfo[]>;
+    getModelCapabilities(model: string): ModelCapabilities | null;
+    chat(messages: LLMMessage[], options?: LLMCompletionOptions): Promise<LLMResponse>;
+    private getHeaders;
+    private formatMessages;
+    private handleStream;
+    private parseErrorResponse;
+    private handleError;
+    private formatModelName;
+}
+//# sourceMappingURL=anthropic.d.ts.map

package/dist/adapters/llm/anthropic.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"anthropic.d.ts","sourceRoot":"","sources":["../../../src/adapters/llm/anthropic.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,OAAO,EAAE,eAAe,EAAE,MAAM,WAAW,CAAC;AAC5C,OAAO,KAAK,EACV,UAAU,EACV,oBAAoB,EACpB,WAAW,EACX,SAAS,EACT,iBAAiB,EACjB,uBAAuB,EACxB,MAAM,4BAA4B,CAAC;AAyEpC,qBAAa,iBAAkB,SAAQ,eAAe;IACpD,QAAQ,CAAC,IAAI,EAAG,WAAW,CAAU;IACrC,QAAQ,CAAC,IAAI,eAAe;IAE5B,OAAO,CAAC,MAAM,CAAS;IACvB,OAAO,CAAC,OAAO,CAAS;IACxB,OAAO,CAAC,KAAK,CAAS;IACtB,OAAO,CAAC,gBAAgB,CAAS;gBAErB,MAAM,EAAE,uBAAuB;IAQrC,WAAW,IAAI,OAAO,CAAC,OAAO,CAAC;IAuB/B,UAAU,IAAI,OAAO,CAAC,SAAS,EAAE,CAAC;IASxC,oBAAoB,CAAC,KAAK,EAAE,MAAM,GAAG,iBAAiB,GAAG,IAAI;IAIvD,IAAI,CACR,QAAQ,EAAE,UAAU,EAAE,EACtB,OAAO,CAAC,EAAE,oBAAoB,GAC7B,OAAO,CAAC,WAAW,CAAC;IAwFvB,OAAO,CAAC,UAAU;IAQlB,OAAO,CAAC,cAAc;YAmBR,YAAY;YA6GZ,kBAAkB;IAgDhC,OAAO,CAAC,WAAW;IAoBnB,OAAO,CAAC,eAAe;CAQxB"}