@rbalchii/anchor-engine 4.7.0

package/docs/DEPLOYMENT.md

@@ -0,0 +1,448 @@
+# Anchor Engine - Deployment Guide
+
+**Version:** 4.8.0 | **Last Updated:** March 18, 2026
+
+---
+
+## Quick Deploy
+
+### Local Development
+```bash
+git clone https://github.com/RSBalchII/anchor-engine-node.git
+cd anchor-engine-node
+pnpm install
+pnpm build
+pnpm start
+```
+
+### Docker (Recommended for Production)
+```bash
+docker-compose up -d
+docker-compose logs -f
+```
+
+---
+
+## Deployment Options
+
+### Option 1: Local Machine
+
+**Best for:** Personal use, development, testing
+
+**Requirements:**
+- Node.js v18+ (v20+ recommended)
+- PNPM package manager
+- 1GB RAM minimum (4GB+ recommended)
+- 10GB free storage
+
+**Steps:**
+1. Clone repository
+2. Install dependencies: `pnpm install`
+3. Build: `pnpm build`
+4. Start: `pnpm start`
+5. Open: http://localhost:3160
+
+**Configuration:** Edit `user_settings.json`
+
+---
+
+### Option 2: Docker
+
+**Best for:** Production, isolated environments
+
+**Requirements:**
+- Docker 20.10+
+- Docker Compose 2.0+
+- 2GB RAM allocated to container
+
+**Steps:**
+```bash
+# Build and run
+docker-compose up -d
+
+# View logs
+docker-compose logs -f
+
+# Stop
+docker-compose down
+```
+
+**Volumes:**
+- `./inbox` → Auto-ingested files
+- `./external-inbox` → External sources
+- `./mirrored_brain` → Source of truth
+- `./backups` → Phoenix Protocol backups
+- `anchor-data` → PGlite database (persistent)
+
+**Environment Variables:**
+```yaml
+environment:
+  - PROJECT_ROOT=/app
+  - CONTEXT_DIR=/app/engine/context_data
+  - NOTEBOOK_DIR=/app/notebook
+```
+
+---
+
+### Option 3: Cloud VPS
+
+**Best for:** Team access, 24/7 availability
+
+**Recommended Providers:**
+- DigitalOcean Droplet ($12/mo - 2GB RAM)
+- Linode Nanode ($5/mo - 1GB RAM)
+- AWS EC2 t3.small ($15/mo - 2GB RAM)
+
+**Steps:**
+1. Provision Ubuntu 22.04 server
+2. Install Node.js:
+   ```bash
+   curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
+   sudo apt-get install -y nodejs
+   ```
+3. Install PNPM:
+   ```bash
+   sudo npm install -g pnpm
+   ```
+4. Clone and setup:
+   ```bash
+   git clone https://github.com/RSBalchII/anchor-engine-node.git
+   cd anchor-engine-node
+   pnpm install
+   pnpm build
+   ```
+5. Run as systemd service:
+   ```bash
+   sudo nano /etc/systemd/system/anchor-engine.service
+   ```
+   
+   ```ini
+   [Unit]
+   Description=Anchor Engine
+   After=network.target
+   
+   [Service]
+   Type=simple
+   User=ubuntu
+   WorkingDirectory=/home/ubuntu/anchor-engine-node
+   ExecStart=/usr/bin/pnpm start
+   Restart=always
+   
+   [Install]
+   WantedBy=multi-user.target
+   ```
+   
+   ```bash
+   sudo systemctl enable anchor-engine
+   sudo systemctl start anchor-engine
+   sudo systemctl status anchor-engine
+   ```
+
+6. Configure firewall:
+   ```bash
+   sudo ufw allow 3160/tcp
+   sudo ufw enable
+   ```
+
+---
+
+### Option 4: Kubernetes
+
+**Best for:** Enterprise, auto-scaling
+
+**Requirements:**
+- Kubernetes 1.25+
+- PersistentVolume for data
+
+**Deployment YAML:**
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: anchor-engine
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: anchor-engine
+  template:
+    metadata:
+      labels:
+        app: anchor-engine
+    spec:
+      containers:
+      - name: anchor-engine
+        image: anchor-engine:latest
+        ports:
+        - containerPort: 3160
+        volumeMounts:
+        - name: data
+          mountPath: /app/mirrored_brain
+        - name: backups
+          mountPath: /app/backups
+        resources:
+          requests:
+            memory: "1Gi"
+            cpu: "500m"
+          limits:
+            memory: "2Gi"
+            cpu: "1000m"
+      volumes:
+      - name: data
+        persistentVolumeClaim:
+          claimName: anchor-data-pvc
+      - name: backups
+        persistentVolumeClaim:
+          claimName: anchor-backups-pvc
+```
+
+---
+
+## Configuration
+
+### user_settings.json
+
+**Location:** `~/.config/anchor/user_settings.json` or project root
+
+**Key Settings:**
+```json
+{
+  "server": {
+    "host": "0.0.0.0",
+    "port": 3160,
+    "api_key": "your-secret-key"
+  },
+  "database": {
+    "wipe_on_startup": false
+  },
+  "watcher": {
+    "debounce_ms": 2000,
+    "exclude_patterns": ["**/*.log", "**/node_modules/**"]
+  },
+  "search": {
+    "max_chars_default": 524288,
+    "strategy": "hybrid"
+  },
+  "adaptive_concurrency": {
+    "environment": "auto",
+    "sequential_threshold_mb": 2048,
+    "parallel_threshold_mb": 8192
+  }
+}
+```
+
+### Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `ANCHOR_URL` | `http://localhost:3160` | API endpoint |
+| `ANCHOR_API_KEY` | (none) | API authentication |
+| `PROJECT_ROOT` | (auto-detect) | Project root path |
+| `CONTEXT_DIR` | (auto-detect) | Context data directory |
+
+---
+
+## Security
+
+### Firewall Rules
+
+**Minimum Required:**
+```bash
+# Allow Anchor Engine port
+sudo ufw allow 3160/tcp
+
+# If using MCP remotely (not recommended)
+sudo ufw allow 3161/tcp
+```
+
+### API Key Authentication
+
+1. Generate key:
+   ```bash
+   openssl rand -hex 32
+   ```
+
+2. Add to `user_settings.json`:
+   ```json
+   {
+     "server": {
+       "api_key": "your-generated-key"
+     }
+   }
+   ```
+
+3. Use in requests:
+   ```bash
+   curl -H "Authorization: Bearer your-key" \
+     http://localhost:3160/v1/stats
+   ```
+
+### HTTPS/TLS (Production)
+
+**With Nginx Reverse Proxy:**
+```nginx
+server {
+    listen 443 ssl;
+    server_name anchor.yourdomain.com;
+    
+    ssl_certificate /etc/letsencrypt/live/anchor.yourdomain.com/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/anchor.yourdomain.com/privkey.pem;
+    
+    location / {
+        proxy_pass http://localhost:3160;
+        proxy_http_version 1.1;
+        proxy_set_header Upgrade $http_upgrade;
+        proxy_set_header Connection 'upgrade';
+        proxy_set_header Host $host;
+        proxy_cache_bypass $http_upgrade;
+    }
+}
+```
+
+---
+
+## Monitoring
+
+### Health Checks
+
+**Endpoint:** `GET /health`
+
+**Response:**
+```json
+{
+  "status": "ok",
+  "uptime": "2d 4h 15m",
+  "memory_mb": 512
+}
+```
+
+**Uptime Monitoring:**
+```bash
+# Cron job every 5 minutes
+*/5 * * * * curl -f http://localhost:3160/health || echo "Anchor Engine DOWN" | mail -s "Alert" admin@example.com
+```
+
+### Logs
+
+**Location:** `engine/logs/server.log`
+
+**Log Rotation:**
+```bash
+sudo nano /etc/logrotate.d/anchor-engine
+```
+
+```
+/var/log/anchor-engine/*.log {
+    daily
+    rotate 7
+    compress
+    delaycompress
+    notifempty
+    create 0640 ubuntu ubuntu
+}
+```
+
+---
+
+## Backup & Restore
+
+### Manual Backup
+```bash
+# Create backup
+curl -X POST http://localhost:3160/v1/backup
+
+# Backup location: ./backups/backup-YYYY-MM-DDTHH-MM-SS/
+```
+
+### Automated Backup
+```bash
+# Cron job daily at 2am
+0 2 * * * curl -X POST http://localhost:3160/v1/backup
+```
+
+### Restore
+```bash
+curl -X POST http://localhost:3160/v1/backup/restore \
+  -H "Content-Type: application/json" \
+  -d '{"filename": "backup-2026-03-18T02-00-00"}'
+```
+
+---
+
+## Troubleshooting
+
+### Port Already in Use
+```bash
+# Find process using port 3160
+lsof -i :3160
+
+# Kill process
+kill -9 <PID>
+```
+
+### Out of Memory
+```json
+{
+  "adaptive_concurrency": {
+    "environment": "low_memory"
+  }
+}
+```
+
+### Database Corruption
+```bash
+# Stop engine
+pnpm stop
+
+# Delete database (data preserved in mirrored_brain/)
+rm -rf engine/context_data
+
+# Restart (auto-rebuilds)
+pnpm start
+```
+
+### Permission Errors
+```bash
+# Fix ownership
+sudo chown -R $USER:$USER ~/.config/anchor
+sudo chown -R $USER:$USER ~/.local/share/anchor
+```
+
+---
+
+## Performance Tuning
+
+### Low Memory (<2GB RAM)
+```json
+{
+  "adaptive_concurrency": {
+    "environment": "low_memory",
+    "sequential_threshold_mb": 512
+  },
+  "search": {
+    "max_chars_default": 262144
+  }
+}
+```
+
+### High Performance (>8GB RAM)
+```json
+{
+  "adaptive_concurrency": {
+    "environment": "high_memory",
+    "parallel_threshold_mb": 4096,
+    "max_concurrency": 10
+  },
+  "search": {
+    "max_chars_default": 1048576
+  }
+}
+```
+
+---
+
+## Support
+
+- **Documentation:** [`docs/`](./)
+- **Issues:** https://github.com/RSBalchII/anchor-engine-node/issues
+- **Discussions:** https://github.com/RSBalchII/anchor-engine-node/discussions

package/docs/INDEX.md

@@ -0,0 +1,226 @@
+# Anchor Engine - Documentation Index
+
+**Version:** 4.8.0 | **Updated:** March 18, 2026 | **Status:** ✅ Production Ready
+
+---
+
+## 📚 Quick Navigation
+
+### Getting Started
+- **[README.md](../README.md)** - Quick start, installation, usage guide
+- **[CHANGELOG.md](../CHANGELOG.md)** - Version history (latest: v4.8.0)
+- **[CONTRIBUTING.md](../CONTRIBUTING.md)** - How to contribute
+
+### Core Documentation
+- **[docs/API.md](API.md)** - Complete API reference
+- **[docs/DEPLOYMENT.md](DEPLOYMENT.md)** - Deployment guide (local, Docker, VPS, K8s)
+- **[docs/TROUBLESHOOTING.md](TROUBLESHOOTING.md)** - Troubleshooting guide
+- **[docs/whitepaper.md](whitepaper.md)** - STAR Algorithm whitepaper
+- **[specs/spec.md](../specs/spec.md)** - System specification with architecture diagrams
+- **[specs/current-standards/](../specs/current-standards/)** - Active architecture standards (001-010)
+
+### Development
+- **[engine/src/README.md](../engine/src/README.md)** - Source code overview
+- **[tests/README.md](../tests/README.md)** - Testing guide
+- **[benchmarks/README.md](../benchmarks/README.md)** - Benchmarking framework
+- **[mcp-server/README.md](../mcp-server/README.md)** - MCP server integration
+
+---
+
+## 🎯 Documentation by Use Case
+
+### "I want to install and run Anchor Engine"
+→ Start with **[README.md](../README.md)** - Quick Start section
+
+### "I need API documentation"
+→ Read **[docs/API.md](API.md)** - Complete API reference with examples
+
+### "I want to deploy to production"
+→ Follow **[docs/DEPLOYMENT.md](DEPLOYMENT.md)** - All deployment options
+
+### "Something's not working"
+→ Check **[docs/TROUBLESHOOTING.md](TROUBLESHOOTING.md)** - Common issues and fixes
+
+### "I need to understand the system architecture"
+→ Study **[specs/spec.md](../specs/spec.md)** - System specification with diagrams
+
+### "I'm researching the theory behind Anchor Engine"
+→ Read **[docs/whitepaper.md](whitepaper.md)** - Academic paper
+
+### "I want to understand the search algorithm"
+→ See **[specs/current-standards/004-streaming-search.md](../specs/current-standards/004-streaming-search.md)**
+
+### "How does distillation work?"
+→ Read **[specs/current-standards/010-radial-distillation-v2.md](../specs/current-standards/010-radial-distillation-v2.md)**
+
+### "I want to contribute code"
+→ Start with **[engine/src/README.md](../engine/src/README.md)** - Source overview
+
+### "I need to write tests"
+→ Follow **[tests/README.md](../tests/README.md)** - Testing guide
+
+### "I want to connect via MCP"
+→ See **[mcp-server/README.md](../mcp-server/README.md)** - MCP integration
+
+---
+
+## 📊 Key Metrics (v4.8.0)
+
+| Metric | Value | Status |
+|--------|-------|--------|
+| **Context Retrieval** | 618k chars | ✅ +18% vs whitepaper |
+| **Memory Peak** | 510MB | ✅ -70% vs whitepaper |
+| **Search Latency** | <200ms (p95) | ✅ Optimized |
+| **Ingestion Throughput** | 1,200 mol/sec | ✅ Verified |
+| **Deduplication Rate** | 40-50% | ✅ With SimHash |
+| **Session Index** | NEW in v4.8.0 | ✅ Two-tier retrieval |
+| **MCP Write Ops** | NEW in v4.8.0 | ✅ Opt-in ingest |
+
+---
+
+## 🗂️ Document Structure
+
+```
+anchor-engine-node/
+├── README.md                      # Start here - Quick start & usage
+├── CHANGELOG.md                   # What's new (v4.8.0)
+├── CONTRIBUTING.md                # How to contribute
+│
+├── docs/
+│   ├── API.md                     # ✅ NEW - Complete API reference
+│   ├── DEPLOYMENT.md              # ✅ NEW - Deployment guide
+│   ├── TROUBLESHOOTING.md         # ✅ NEW - Troubleshooting
+│   ├── whitepaper.md              # STAR algorithm paper
+│   ├── code-patterns.md           # Code patterns
+│   ├── INDEX.md                   # This file - navigation hub
+│   └── arxiv/                     # arXiv submission docs
+│
+├── specs/
+│   ├── spec.md                    # System spec with diagrams
+│   ├── API-ROUTE-MAP.md           # Detailed API spec
+│   ├── plan.md                    # Roadmap
+│   ├── tasks.md                   # Current tasks
+│   ├── current-standards/         # ✅ Active standards (001-010)
+│   ├── archive-standards/         # Historical standards (059-200+)
+│   └── archive-legacy/            # Legacy docs
+│
+├── engine/
+│   └── src/
+│       └── README.md              # ✅ NEW - Source code overview
+│
+├── tests/
+│   └── README.md                  # ✅ NEW - Testing guide
+│
+├── benchmarks/
+│   └── README.md                  # Benchmarking framework
+│
+└── mcp-server/
+    └── README.md                  # MCP server integration
+```
+
+---
+
+## 🎓 Learning Path
+
+### Beginner (Just Getting Started)
+1. **[README.md](../README.md)** - Installation & first query
+2. **[docs/API.md](API.md)** - API basics
+3. **[CHANGELOG.md](../CHANGELOG.md)** - Recent features
+
+### Intermediate (Understanding the System)
+1. **[specs/spec.md](../specs/spec.md)** - Full system specification
+2. **[specs/current-standards/004-streaming-search.md](../specs/current-standards/004-streaming-search.md)** - Search algorithm
+3. **[specs/current-standards/010-radial-distillation-v2.md](../specs/current-standards/010-radial-distillation-v2.md)** - Distillation v2.0
+4. **[engine/src/README.md](../engine/src/README.md)** - Source overview
+
+### Advanced (Deep Dive)
+1. **[docs/whitepaper.md](whitepaper.md)** - Theoretical foundation
+2. **[tests/README.md](../tests/README.md)** - Test suite
+3. **[specs/current-standards/](../specs/current-standards/)** - All active standards
+4. **[docs/DEPLOYMENT.md](DEPLOYMENT.md)** - Production deployment
+
+### Contributor (Writing Code)
+1. **[CONTRIBUTING.md](../CONTRIBUTING.md)** - Contribution guidelines
+2. **[engine/src/README.md](../engine/src/README.md)** - Source structure
+3. **[tests/README.md](../tests/README.md)** - Testing requirements
+4. **[specs/current-standards/](../specs/current-standards/)** - Architecture standards
+
+---
+
+## 🔬 Recent Updates (v4.8.0 - Mar 18, 2026)
+
+### New Features
+- **Session Index** - Two-tier memory retrieval (index → targeted fetch)
+- **MCP Write Operations** - `anchor_ingest_text` and `anchor_ingest_file` tools
+- **Paste & Ingest UI** - Quick text ingestion via Web UI
+- **Philosophy Documentation** - Core principles embedded throughout docs
+
+### New Documentation
+- **docs/API.md** - Complete API reference
+- **docs/DEPLOYMENT.md** - Deployment guide (all platforms)
+- **docs/TROUBLESHOOTING.md** - Troubleshooting by category
+- **engine/src/README.md** - Source code overview
+- **tests/README.md** - Testing guide
+
+### Documentation Consolidation
+- Merged architecture diagrams into `specs/spec.md`
+- Archived 7 redundant historical documents
+- Updated all links to point to current docs
+
+---
+
+## 📁 Documentation Categories
+
+### User-Facing
+| Document | Audience | Purpose |
+|----------|----------|---------|
+| README.md | All users | Quick start & usage |
+| docs/API.md | Developers | API reference |
+| docs/DEPLOYMENT.md | DevOps | Deployment guide |
+| docs/TROUBLESHOOTING.md | All users | Fix common issues |
+| mcp-server/README.md | MCP users | MCP integration |
+
+### Developer-Facing
+| Document | Audience | Purpose |
+|----------|----------|---------|
+| engine/src/README.md | Contributors | Source overview |
+| tests/README.md | Contributors | Testing guide |
+| benchmarks/README.md | Contributors | Performance testing |
+| specs/current-standards/ | Contributors | Architecture standards |
+
+### Academic/Research
+| Document | Audience | Purpose |
+|----------|----------|---------|
+| docs/whitepaper.md | Researchers | STAR algorithm paper |
+| specs/spec.md | Architects | System specification |
+| docs/arxiv/ | Researchers | arXiv submission |
+
+---
+
+## 🔗 External Resources
+
+- **GitHub Repository:** https://github.com/RSBalchII/anchor-engine-node
+- **License:** AGPL-3.0
+- **NPM Packages:** @rbalchii/* (native modules)
+- **DOI:** https://doi.org/10.5281/zenodo.18841399
+
+---
+
+## 📞 Support & Contribution
+
+### Getting Help
+- Check **[docs/TROUBLESHOOTING.md](TROUBLESHOOTING.md)** for common issues
+- Review **[specs/spec.md](../specs/spec.md)** for system understanding
+- Read **[CHANGELOG.md](../CHANGELOG.md)** for recent fixes
+
+### Contributing
+1. Read **[CONTRIBUTING.md](../CONTRIBUTING.md)** for guidelines
+2. Review **[engine/src/README.md](../engine/src/README.md)** for source structure
+3. Follow **[specs/current-standards/](../specs/current-standards/)** for architecture
+4. Run **[tests/README.md](../tests/README.md)** test suite before submitting
+
+---
+
+**Last Updated:** March 18, 2026  
+**Version:** 4.8.0  
+**Status:** ✅ Production Ready