security-detections-mcp 2.1.0 → 3.0.0

package/README.md

@@ -2,8 +2,106 @@
 
 An MCP (Model Context Protocol) server that lets LLMs query a unified database of **Sigma**, **Splunk ESCU**, **Elastic**, and **KQL** security detection rules.
 
+> **New here? Start with the [Setup Guide](./SETUP.md)** -- covers macOS, Windows (WSL & native), and Linux step by step.
+
+## What's New in 3.0 - Autonomous Detection Platform
+
+Version 3.0 transforms this MCP into a **fully autonomous detection engineering platform**. Feed it threat intelligence, and it automatically:
+
+1. **Extracts TTPs** from threat reports, CISA alerts, or manual input
+2. **Analyzes coverage gaps** against your existing detections
+3. **Generates detections** in your SIEM's native format (SPL, KQL, EQL, or Sigma)
+4. **Runs Atomic Red Team tests** against your lab environment
+5. **Validates detections fire** by querying your SIEM
+6. **Exports attack data** for reproducibility
+7. **Stages DRAFT PRs** to your detection repo (never auto-merges)
+
+> **Multi-SIEM**: Set `SIEM_PLATFORM` to `splunk`, `sentinel`, `elastic`, or `sigma` in your `.env`. The pipeline was built on Splunk + Attack Range but adapts to any SIEM. See the **[E2E Testing Guide](./docs/E2E-TESTING-GUIDE.md)** for complete setup instructions per platform.
+
+### Architecture: LangGraph + Cursor Subagents
+
+The 3.0 architecture uses two complementary systems:
+
+| Component | Purpose | Location |
+|-----------|---------|----------|
+| **LangGraph Pipeline** | Core autonomous workflow - portable, testable, CI/CD ready | `agents/` |
+| **Cursor Subagents** | Interactive IDE agents for manual tasks | `.cursor/agents/` |
+
+### Quick Start - Autonomous Mode
+
+**Prerequisites**: Node.js 20+, an Anthropic API key. Full details in the [Setup Guide](./SETUP.md).
+
+```bash
+# Install the agents package
+cd agents && npm install --registry https://registry.npmjs.org/
+
+# Configure
+cp .env.example .env
+# Edit .env: set SIEM_PLATFORM, ANTHROPIC_API_KEY, SECURITY_CONTENT_PATH
+
+# Test with dry run first (uses mock data, no LLM calls)
+DRY_RUN=true npm run orchestrate -- --type technique --input "T1566.004 Spearphishing Voice"
+
+# Run with real LLM (creates actual detections)
+npm run orchestrate -- --type technique --input "T1566.004 Spearphishing Voice"
+
+# Or analyze a CISA alert
+npm run orchestrate -- --type cisa_alert --url https://www.cisa.gov/news-events/alerts/...
+
+# Or feed it a threat report
+npm run orchestrate -- --type threat_report --file ./report.md
+
+# Note: Use T1566.004 for testing - it has no existing coverage so will create a detection
+# T1003.001 has 100+ existing detections, so the pipeline will correctly skip it (no gap)
+```
+
+### Pipeline Stages
+
+```
+┌─────────────┐    ┌──────────────────┐    ┌────────────────────┐
+│ CTI Analyst │───>│ Coverage Analyzer│───>│ Detection Engineer │
+└─────────────┘    └──────────────────┘    └────────────────────┘
+                                                     │
+                                                     ▼
+┌───────────┐    ┌──────────────────┐    ┌──────────────────────┐
+│ PR Stager │<───│   Data Dumper    │<───│  Splunk Validator    │
+└───────────┘    └──────────────────┘    └──────────────────────┘
+                                                     ▲
+                                                     │
+                                          ┌──────────────────┐
+                                          │ Atomic Executor  │
+                                          └──────────────────┘
+```
+
+### MCP Integration
+
+The autonomous pipeline integrates with existing MCPs:
+- **security-detections** - Coverage analysis and gap identification
+- **splunk-mcp** - Detection validation (`run_detection`, `export_dump`)
+- **mitre-attack** - Technique lookups
+
+### Human-in-the-Loop
+
+**CRITICAL**: The system NEVER auto-commits or auto-merges. All PRs are created as **DRAFT** requiring human review:
+
+```
+[PR Stager] ✓ security_content DRAFT PR created: https://github.com/splunk/security_content/pull/123
+[PR Stager] ✓ attack_data DRAFT PR created: https://github.com/splunk/attack_data/pull/456
+```
+
+See the [Autonomous Platform Documentation](./docs/AUTONOMOUS.md) for full details, and the [E2E Testing Guide](./docs/E2E-TESTING-GUIDE.md) for per-SIEM setup (Splunk, Sentinel, Elastic, Sigma).
+
 [![Install MCP Server](https://cursor.com/deeplink/mcp-install-dark.svg)](https://cursor.com/en/install-mcp?name=security-detections&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsInNlY3VyaXR5LWRldGVjdGlvbnMtbWNwIl0sImVudiI6eyJTSUdNQV9QQVRIUyI6Ii9wYXRoL3RvL3NpZ21hL3J1bGVzLC9wYXRoL3RvL3NpZ21hL3J1bGVzLXRocmVhdC1odW50aW5nIiwiU1BMVU5LX1BBVEhTIjoiL3BhdGgvdG8vc2VjdXJpdHlfY29udGVudC9kZXRlY3Rpb25zIiwiU1RPUllfUEFUSFMiOiIvcGF0aC90by9zZWN1cml0eV9jb250ZW50L3N0b3JpZXMiLCJFTEFTVElDX1BBVEhTIjoiL3BhdGgvdG8vZGV0ZWN0aW9uLXJ1bGVzL3J1bGVzIiwiS1FMX1BBVEhTIjoiL3BhdGgvdG8va3FsLXJ1bGVzIn19)
 
+> **Detailed setup**: See the **[Setup Guide](./SETUP.md)** for step-by-step install on macOS, Windows (WSL & native), and Linux with troubleshooting for common issues.
+
+## 🐛 Version 2.1.1 (Bug Fix)
+
+- **Fixed Windows EBUSY crash** - SQLite database recreation now handles Windows file locking with retry logic. Previously, Windows users would get `EBUSY: resource busy or locked` on startup.
+- **SQLite journal cleanup** - WAL, SHM, and journal companion files are now cleaned up during database recreation.
+- **Windows CI** - Added Windows to the CI matrix. Build, tests, and full Sigma indexing pipeline now run on both Linux and Windows.
+- **Cross-platform test suite** - New `tests/cross-platform-test.js` validates database lifecycle on all platforms. New `tests/ci-integration-test.js` downloads and indexes 3,200+ Sigma rules to validate the full pipeline.
+
 ## 🚀 Version 2.1 Features
 
 **Security Detections MCP v2.1** introduces powerful new capabilities for detection engineering intelligence, analytical memory, autonomous analysis, and advanced MCP protocol features:

package/dist/db/connection.d.ts

@@ -29,6 +29,7 @@ export declare function clearDb(): void;
 /**
  * Force recreation of the database.
  * Useful when schema changes require a fresh start.
+ * Handles Windows file locking (EBUSY) with retry logic.
  */
 export declare function recreateDb(): void;
 /**

package/dist/db/connection.js

@@ -4,7 +4,7 @@
  * Manages SQLite connection singleton, path management, and database lifecycle.
  */
 import Database from 'better-sqlite3';
-import { homedir } from 'os';
+import { homedir, platform } from 'os';
 import { join } from 'path';
 import { mkdirSync, existsSync, unlinkSync } from 'fs';
 import { createSchema } from './schema.js';
@@ -143,18 +143,51 @@ export function clearDb() {
     const database = initDb();
     database.exec('DELETE FROM detections');
 }
+/**
+ * Safely delete a file with retry logic for Windows.
+ * Windows can throw EBUSY/EPERM if the file handle isn't fully released yet.
+ */
+function safeUnlink(filePath, maxRetries = 5, delayMs = 100) {
+    for (let attempt = 0; attempt < maxRetries; attempt++) {
+        try {
+            if (existsSync(filePath)) {
+                unlinkSync(filePath);
+            }
+            return;
+        }
+        catch (err) {
+            const code = err.code;
+            // EBUSY = file is busy, EPERM = operation not permitted, EACCES = access denied
+            if ((code === 'EBUSY' || code === 'EPERM' || code === 'EACCES') && attempt < maxRetries - 1) {
+                // Spin-wait since we need synchronous behavior
+                const end = Date.now() + delayMs * (attempt + 1);
+                while (Date.now() < end) { /* wait */ }
+                continue;
+            }
+            throw err;
+        }
+    }
+}
 /**
  * Force recreation of the database.
  * Useful when schema changes require a fresh start.
+ * Handles Windows file locking (EBUSY) with retry logic.
  */
 export function recreateDb() {
     if (db) {
         db.close();
         db = null;
     }
-    if (existsSync(DB_PATH)) {
-        unlinkSync(DB_PATH);
+    // Small delay on Windows to let file handles fully release
+    if (platform() === 'win32') {
+        const end = Date.now() + 100;
+        while (Date.now() < end) { /* wait */ }
     }
+    // Delete main db and SQLite journal/WAL files
+    safeUnlink(DB_PATH);
+    safeUnlink(DB_PATH + '-wal');
+    safeUnlink(DB_PATH + '-shm');
+    safeUnlink(DB_PATH + '-journal');
 }
 /**
  * Check if the database file exists.

package/dist/db.js

@@ -1,5 +1,5 @@
 import Database from 'better-sqlite3';
-import { homedir } from 'os';
+import { homedir, platform } from 'os';
 import { join } from 'path';
 import { mkdirSync, existsSync, unlinkSync } from 'fs';
 const CACHE_DIR = join(homedir(), '.cache', 'security-detections-mcp');
@@ -158,15 +158,43 @@ export function clearDb() {
     const database = initDb();
     database.exec('DELETE FROM detections');
 }
+// Safely delete a file with retry logic for Windows (EBUSY/EPERM)
+function safeUnlink(filePath, maxRetries = 5, delayMs = 100) {
+    for (let attempt = 0; attempt < maxRetries; attempt++) {
+        try {
+            if (existsSync(filePath)) {
+                unlinkSync(filePath);
+            }
+            return;
+        }
+        catch (err) {
+            const code = err.code;
+            if ((code === 'EBUSY' || code === 'EPERM' || code === 'EACCES') && attempt < maxRetries - 1) {
+                const end = Date.now() + delayMs * (attempt + 1);
+                while (Date.now() < end) { /* wait */ }
+                continue;
+            }
+            throw err;
+        }
+    }
+}
 // Force recreation of the database (needed when schema changes)
+// Handles Windows file locking with retry logic
 export function recreateDb() {
     if (db) {
         db.close();
         db = null;
     }
-    if (existsSync(DB_PATH)) {
-        unlinkSync(DB_PATH);
+    // Small delay on Windows to let file handles fully release
+    if (platform() === 'win32') {
+        const end = Date.now() + 100;
+        while (Date.now() < end) { /* wait */ }
     }
+    // Delete main db and SQLite journal/WAL files
+    safeUnlink(DB_PATH);
+    safeUnlink(DB_PATH + '-wal');
+    safeUnlink(DB_PATH + '-shm');
+    safeUnlink(DB_PATH + '-journal');
 }
 export function insertDetection(detection) {
     const database = initDb();

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "security-detections-mcp",
-  "version": "2.1.0",
+  "version": "3.0.0",
   "description": "Advanced MCP server for security detections with Detection Engineering Intelligence, Knowledge Graph (Tribal Knowledge), Elicitation, and Resource Subscriptions",
   "sigmaSpecVersion": "2.0.0",
   "type": "module",
@@ -20,7 +20,9 @@
     "build": "tsc",
     "start": "node dist/index.js",
     "dev": "tsc --watch",
-    "test": "npm run build && node tests/integration-test.js && node tests/engineering-tools-test.js",
+    "test": "npm run build && node tests/cross-platform-test.js && node tests/integration-test.js && node tests/engineering-tools-test.js",
+    "test:platform": "npm run build && node tests/cross-platform-test.js",
+    "test:ci": "npm run build && node tests/ci-integration-test.js",
     "test:integration": "npm run build && node tests/integration-test.js",
     "test:engineering": "npm run build && node tests/engineering-tools-test.js",
     "lint": "tsc --noEmit --strict",