@rbalchii/anchor-engine 4.7.0

package/docs/archive/adoption-preparation.md

@@ -0,0 +1,179 @@
+# ECE_Core: Preparing for Broader Adoption
+
+## Introduction
+
+This document outlines the strategy and preparations needed to facilitate broader adoption of ECE_Core as the standard for universal AI memory infrastructure. As we transition from a specialized tool to a foundational infrastructure component, we must ensure the system is accessible, reliable, and valuable to diverse user groups.
+
+## Adoption Strategy
+
+### 1. Lowering the Barrier to Entry
+
+#### Simplified Installation
+- Create one-click installer packages for major platforms
+- Implement automatic dependency resolution
+- Provide clear, step-by-step setup guides with visual aids
+- Develop Docker containers for easy deployment
+
+#### Documentation Enhancement
+- Create beginner-friendly tutorials
+- Develop use-case specific guides (research, development, personal knowledge management)
+- Provide API documentation with code examples
+- Create video walkthroughs for common tasks
+
+#### Community Resources
+- Establish a comprehensive FAQ section
+- Create a community forum for peer support
+- Develop a curated list of third-party integrations
+- Provide templates for common use cases
+
+### 2. Demonstrating Clear Value Propositions
+
+#### For Individual Users
+- Showcase privacy and data sovereignty benefits
+- Demonstrate performance on consumer hardware
+- Highlight cost savings compared to cloud solutions
+- Provide examples of personal knowledge management workflows
+
+#### For Developers
+- Offer extensive API documentation
+- Provide SDKs for popular programming languages
+- Create sample applications and integrations
+- Establish clear contribution guidelines
+
+#### For Enterprises
+- Detail security and compliance benefits
+- Provide deployment guides for enterprise environments
+- Offer performance benchmarks and scalability information
+- Create whitepapers on ROI and cost analysis
+
+### 3. Building Ecosystem Partnerships
+
+#### Tool Integration
+- Partner with popular note-taking applications
+- Integrate with development environments (IDEs)
+- Connect with research and academic tools
+- Support content management systems
+
+#### Platform Distribution
+- Submit to package managers (npm, brew, chocolatey, etc.)
+- Partner with cloud providers for easy deployment
+- Integrate with container registries
+- Collaborate with hardware manufacturers
+
+## Technical Preparation
+
+### 1. Performance Optimization
+- Optimize for resource-constrained environments
+- Implement efficient memory management
+- Ensure consistent performance across platforms
+- Provide performance monitoring tools
+
+### 2. Reliability & Stability
+- Implement comprehensive error handling
+- Create robust backup and recovery systems
+- Ensure graceful degradation when components fail
+- Provide detailed logging and diagnostics
+
+### 3. Security Hardening
+- Implement secure default configurations
+- Provide encryption for sensitive data
+- Create security audit procedures
+- Establish responsible disclosure processes
+
+### 4. Scalability Considerations
+- Optimize for large-scale deployments
+- Implement efficient data indexing
+- Provide clustering and distributed processing options
+- Ensure performance consistency as data grows
+
+## Community Building
+
+### 1. Educational Content
+- Host webinars and workshops
+- Create certification programs
+- Develop case studies and success stories
+- Produce regular technical blog posts
+
+### 2. Support Infrastructure
+- Establish multiple support channels
+- Create comprehensive troubleshooting guides
+- Implement user feedback systems
+- Provide professional services options
+
+### 3. Governance Model
+- Establish clear project governance
+- Create transparent decision-making processes
+- Implement community contribution guidelines
+- Foster inclusive participation
+
+## Market Positioning
+
+### 1. Competitive Differentiation
+- Emphasize unique "Browser Paradigm" approach
+- Highlight performance advantages
+- Showcase privacy and sovereignty benefits
+- Demonstrate cost-effectiveness
+
+### 2. Use Case Expansion
+- Identify new application domains
+- Develop domain-specific features
+- Create industry partnerships
+- Expand integration ecosystem
+
+### 3. Thought Leadership
+- Publish research papers and technical articles
+- Participate in conferences and events
+- Contribute to open standards development
+- Influence industry direction
+
+## Success Metrics
+
+### Adoption Metrics
+- Number of active installations
+- Community size and engagement
+- Third-party integrations
+- User-generated content and extensions
+
+### Performance Metrics
+- System reliability and uptime
+- User satisfaction scores
+- Performance benchmarks
+- Resource utilization efficiency
+
+### Business Metrics
+- Cost savings for users
+- Time-to-value for new users
+- Retention rates
+- Community contribution activity
+
+## Roadmap for Broader Adoption
+
+### Phase 1: Foundation (Months 1-3)
+- Complete documentation overhaul
+- Release user-friendly installer
+- Establish community forums
+- Create initial tutorial series
+
+### Phase 2: Outreach (Months 4-6)
+- Launch marketing campaign
+- Attend industry events
+- Partner with influencers and educators
+- Release first enterprise features
+
+### Phase 3: Ecosystem (Months 7-12)
+- Establish integration partnerships
+- Launch certification program
+- Create marketplace for extensions
+- Develop professional services offering
+
+## Conclusion
+
+Preparing ECE_Core for broader adoption requires a multi-faceted approach that addresses technical, educational, and community-building aspects. By focusing on lowering barriers to entry, demonstrating clear value propositions, and building a supportive ecosystem, we can establish ECE as the standard for universal AI memory infrastructure.
+
+The "Browser Paradigm" approach positions ECE uniquely in the market, offering universal compatibility, privacy preservation, and economic efficiency. With proper preparation and execution, ECE can become the foundational infrastructure tool that enables the next generation of AI applications while preserving user sovereignty and promoting innovation.
+
+Success will be measured not just in adoption numbers, but in the degree to which ECE enables users to maintain control over their intelligence infrastructure while accessing sophisticated AI memory capabilities. The goal is to make powerful AI memory systems universally accessible while preserving the values of privacy, sovereignty, and economic freedom.
+
+---
+
+*This document serves as a roadmap for the ECE_Core team and community to follow as we work toward broader adoption of the Browser Paradigm for AI memory systems.*

package/docs/archive/agent-harness-integration.md

@@ -0,0 +1,227 @@
+# Agent Harness Integration Patterns
+
+## Overview
+
+The ECE_Core system is designed to be **agent harness agnostic**, allowing multiple agent frameworks to integrate with the core data atomization service. This document outlines the integration patterns for connecting agent harnesses to the ECE system.
+
+## Core Philosophy: Data Atomization Service
+
+ECE/Anchor is fundamentally a **data atomization service** that:
+
+- Packages diverse data types into semantically meaningful units
+- Enables semantic utilization of multiple data types
+- Provides API access for querying and parsing data
+- Outputs data in standardized formats (JSON, CSV, tables)
+- Serves as a foundational layer for various agent systems
+
+## Integration Patterns
+
+### 1. Stateless Context Retrieval Pattern
+
+The recommended approach for agent harness integration is the stateless context retrieval pattern:
+
+```mermaid
+sequenceDiagram
+    participant AH as Agent Harness
+    participant ECE as ECE/Anchor Service
+    participant DB as PGlite Database
+    
+    AH->>ECE: Query request with context needs
+    ECE->>DB: Execute semantic search
+    DB-->>ECE: Retrieved context atoms
+    ECE-->>AH: Return context + metadata
+    AH->>AH: Combine context with agent logic
+    AH-->>User: Final response
+```
+
+**Benefits:**
+- No session memory dependencies
+- Consistent context retrieval
+- Scalable across multiple agent instances
+- Predictable performance characteristics
+
+### 2. API-Based Integration Pattern
+
+Agent harnesses should use the standardized API endpoints:
+
+#### Context Retrieval
+```
+POST /v1/memory/search
+{
+  "query": "natural language query",
+  "buckets": ["optional", "bucket", "filters"],
+  "tags": ["optional", "tag", "filters"],
+  "max_chars": 20000
+}
+```
+
+#### Molecule Search (Sentence-like chunks)
+```
+POST /v1/memory/molecule-search
+{
+  "query": "query split into sentence-like chunks",
+  "max_chars": 2400
+}
+```
+
+#### Data Management
+```
+GET /v1/buckets - Get available data buckets
+GET /v1/tags - Get available tags
+POST /v1/ingest - Ingest new content
+```
+
+### 3. Token Budget Management Pattern
+
+Agent harnesses should implement token budget management:
+
+```typescript
+interface ContextRequest {
+  query: string;
+  tokenBudget?: number; // Maximum tokens to retrieve
+  provenance?: 'internal' | 'external' | 'all'; // Data source filtering
+  buckets?: string[]; // Optional bucket filters
+  tags?: string[]; // Optional tag filters
+}
+```
+
+### 4. Fallback and Error Handling Pattern
+
+Implement robust fallback mechanisms:
+
+```typescript
+async function getContextWithFallback(query: string) {
+  try {
+    // Primary search
+    const results = await eceApi.search({
+      query,
+      max_chars: 20000
+    });
+    return results;
+  } catch (primaryError) {
+    console.warn('Primary search failed, trying fallback:', primaryError);
+    
+    // Fallback search with relaxed parameters
+    try {
+      const fallbackResults = await eceApi.search({
+        query,
+        max_chars: 10000, // Reduced budget
+        buckets: [], // Remove bucket filters
+        tags: [] // Remove tag filters
+      });
+      return fallbackResults;
+    } catch (fallbackError) {
+      console.error('Both primary and fallback searches failed:', fallbackError);
+      return { context: '', results: [] }; // Safe fallback
+    }
+  }
+}
+```
+
+## OpenCLAW Integration Example
+
+As the primary target harness, OpenCLAW follows these integration patterns:
+
+### 1. Context Injection
+- OpenCLAW sends user queries to `/v1/memory/search`
+- Retrieved context is injected into the LLM prompt
+- Response combines ECE context with OpenCLAW's reasoning
+
+### 2. State Management
+- OpenCLAW maintains its own session state
+- ECE provides context without session memory
+- Clean separation of concerns
+
+### 3. Performance Optimization
+- OpenCLAW implements caching for repeated queries
+- Token budgets are managed per interaction
+- Asynchronous context retrieval to avoid blocking
+
+## Generic Agent Harness Template
+
+For developing new agent harnesses, use this template:
+
+```typescript
+class GenericAgentHarness {
+  private eceEndpoint: string;
+  private tokenBudget: number;
+  
+  constructor(eceEndpoint: string, tokenBudget: number = 20000) {
+    this.eceEndpoint = eceEndpoint;
+    this.tokenBudget = tokenBudget;
+  }
+  
+  async processQuery(userQuery: string) {
+    // 1. Retrieve context from ECE
+    const context = await this.retrieveContext(userQuery);
+    
+    // 2. Combine context with agent logic
+    const fullPrompt = this.constructPrompt(userQuery, context);
+    
+    // 3. Process with agent's own logic
+    const response = await this.processWithAgent(fullPrompt);
+    
+    return response;
+  }
+  
+  private async retrieveContext(query: string) {
+    const response = await fetch(`${this.eceEndpoint}/v1/memory/search`, {
+      method: 'POST',
+      headers: { 'Content-Type': 'application/json' },
+      body: JSON.stringify({
+        query,
+        max_chars: this.tokenBudget
+      })
+    });
+    
+    return response.json();
+  }
+  
+  private constructPrompt(userQuery: string, context: any) {
+    return `Context:\n${context.context}\n\nQuestion: ${userQuery}`;
+  }
+  
+  private async processWithAgent(prompt: string) {
+    // Agent-specific processing logic
+    // This is where the agent's unique logic applies
+  }
+}
+```
+
+## Best Practices
+
+### 1. Error Resilience
+- Always implement fallback strategies
+- Handle network errors gracefully
+- Provide meaningful error messages to users
+
+### 2. Performance Optimization
+- Implement caching for repeated queries
+- Use appropriate token budgets
+- Consider asynchronous retrieval to avoid blocking
+
+### 3. Security Considerations
+- Validate all inputs before sending to ECE
+- Implement rate limiting if needed
+- Secure communication channels
+
+### 4. Monitoring and Logging
+- Log context retrieval performance
+- Monitor success rates of queries
+- Track token usage patterns
+
+## Future Extensions
+
+### Plugin Architecture
+Future versions may include a plugin system for dynamic agent harness loading:
+
+```typescript
+interface AgentPlugin {
+  name: string;
+  initialize(): Promise<void>;
+  processQuery(query: string, context: any): Promise<any>;
+  cleanup(): Promise<void>;
+}
+```
+
+This pattern ensures that ECE/Anchor remains a robust, flexible foundation for various agent systems while maintaining clean separation of concerns between the data atomization service and agent-specific logic.

package/docs/archive/api-reference.md

@@ -0,0 +1,106 @@
+# Anchor API Reference
+
+## Overview
+
+The Anchor system provides a comprehensive API for interacting with the context engine. All endpoints follow the UniversalRAG standard and are accessible under the `/v1/` prefix.
+
+## Authentication
+
+Currently, the API does not require authentication and is intended for local use only. All endpoints are accessible without API keys or tokens.
+
+## Core Endpoints
+
+### Memory Search
+- **POST** `/v1/memory/search` - Perform search using Tag-Walker protocol with 70/30 keyword/associative split
+- **POST** `/v1/memory/molecule-search` - Split query into sentence-like chunks and search each separately
+- **GET** `/v1/buckets` - Get all available data buckets
+- **GET** `/v1/tags` - Get all available tags (optionally filtered by buckets)
+
+### Content Management
+- **POST** `/v1/ingest` - Ingest new content into the system
+- **POST** `/v1/atoms/:id/quarantine` - Quarantine a specific atom by ID
+- **POST** `/v1/atoms/:id/restore` - Restore a quarantined atom by ID
+- **PUT** `/v1/atoms/:id/content` - Update the content of an atom
+
+### System Management
+- **GET** `/health` - Get system health status
+- **POST** `/v1/backup` - Create a backup of the database
+- **GET** `/v1/backups` - List available backups
+- **POST** `/v1/backup/restore` - Restore from a specific backup
+- **POST** `/v1/dream` - Trigger the dreamer service for background processing
+
+### Agent Integration
+- **POST** `/v1/chat/completions` - Chat completions endpoint with context injection
+- **GET** `/v1/models` - List available models
+- **POST** `/v1/inference/load` - Load a specific model
+
+### Research & Tools
+- **POST** `/v1/research/scrape` - Scrape and process web content
+- **GET** `/v1/research/web-search` - Perform web search
+- **POST** `/v1/terminal/exec` - Execute terminal commands (simulated)
+- **POST** `/v1/debug/sql` - Execute raw SQL queries (admin only)
+
+### Graph & Visualization
+- **POST** `/v1/graph/data` - Get graph data for visualization
+- **GET** `/v1/scribe/state` - Get scribe state
+- **DELETE** `/v1/scribe/state` - Clear scribe state
+
+### System Configuration
+- **GET** `/v1/system/config` - Get system configuration
+
+## Request/Response Format
+
+### Search Request
+```json
+{
+  "query": "search query text",
+  "buckets": ["bucket1", "bucket2"],
+  "tags": ["tag1", "tag2"],
+  "max_chars": 20000,
+  "provenance": "all"
+}
+```
+
+### Search Response
+```json
+{
+  "status": "success",
+  "context": "retrieved context string",
+  "results": [
+    {
+      "id": "atom_id",
+      "content": "content text",
+      "source": "source path",
+      "timestamp": 1234567890,
+      "buckets": ["bucket1"],
+      "tags": ["tag1"],
+      "epochs": "epoch_info",
+      "provenance": "internal",
+      "score": 0.95
+    }
+  ],
+  "metadata": {
+    "engram_hits": 0,
+    "tag_walker_latency": 0,
+    "provenance_boost_active": true
+  }
+}
+```
+
+## Error Handling
+
+All API endpoints return appropriate HTTP status codes:
+- `200` - Success
+- `400` - Bad request (invalid parameters)
+- `404` - Resource not found
+- `500` - Internal server error
+- `503` - Service temporarily unavailable (during initialization)
+
+Error responses include a JSON object with an `error` field describing the issue.
+
+## Performance Considerations
+
+- Search endpoints accept a `max_chars` parameter to limit response size
+- Token budgets can be adjusted based on use case requirements
+- The system implements caching for frequently accessed content
+- Background processing occurs asynchronously to avoid blocking requests

package/docs/archive/api_flows_diagram.md

@@ -0,0 +1,118 @@
+# API Flow Diagrams for ECE_Core
+
+## Search Flow
+
+```mermaid
+sequenceDiagram
+    participant UI as User Interface
+    participant API as API Service
+    participant SS as Search Service
+    participant DB as PGlite
+    participant TS as Tag-Walker Service
+
+    UI->>API: POST /v1/memory/search {query: "revenue optimization"}
+    API->>SS: executeSearch(query, buckets, maxChars)
+    SS->>TS: processQuery(query)
+    TS->>DB: FTS query on content
+    DB-->>TS: Initial results
+    TS->>TS: Apply Tag-Walker protocol
+    TS->>DB: Graph traversal for related atoms
+    DB-->>TS: Related results
+    TS-->>SS: Curated results
+    SS-->>API: Formatted response
+    API-->>UI: Search results with context
+```
+
+## Ingestion Flow
+
+```mermaid
+sequenceDiagram
+    participant FS as File System
+    participant WD as Watchdog
+    participant IS as Ingestion Service
+    participant AS as Atomizer Service
+    participant DB as PGlite
+    participant NM as Native Modules
+
+    FS->>WD: File change detected
+    WD->>IS: Process file content
+    IS->>AS: atomizeContent(content)
+    AS->>NM: Native atomization (C++)
+    NM-->>AS: Processed molecules
+    AS-->>IS: Molecules with semantic tags
+    IS->>IS: Generate fingerprints (SimHash)
+    IS->>DB: Insert atoms/molecules with deduplication
+    DB-->>IS: Confirmation
+    IS-->>WD: Ingestion complete
+```
+
+## Chat Flow with Context Injection
+
+```mermaid
+sequenceDiagram
+    participant UI as Chat Interface
+    participant API as API Service
+    participant AR as Agent Runtime
+    participant SS as Search Service
+    participant DB as PGlite
+
+    UI->>API: POST /v1/chat/completions {messages: [...]}
+    API->>AR: Initialize agent with contextual messages
+    AR->>SS: Search for context related to query
+    SS->>DB: Execute Tag-Walker search
+    DB-->>SS: Relevant context atoms
+    SS-->>AR: Retrieved context
+    AR->>AR: Combine context with user query
+    AR->>API: Stream response tokens
+    API-->>UI: SSE stream of response
+```
+
+## Backup/Restore Flow
+
+```mermaid
+sequenceDiagram
+    participant UI as Admin Interface
+    participant API as API Service
+    participant BS as Backup Service
+    participant DB as PGlite
+    participant FS as File System
+
+    UI->>API: POST /v1/backup
+    API->>BS: createBackup()
+    BS->>DB: Export database content
+    DB-->>BS: Database dump
+    BS->>FS: Write to backups/ directory
+    FS-->>BS: Backup file created
+    BS-->>API: Backup confirmation
+    API-->>UI: Backup completed
+
+    UI->>API: POST /v1/backup/restore {filename: backup.sql}
+    API->>BS: restoreBackup(filename)
+    BS->>FS: Read backup file
+    FS-->>BS: Backup content
+    BS->>DB: Import database content
+    DB-->>BS: Restore confirmation
+    BS-->>API: Restore completed
+    API-->>UI: System restored
+```
+
+## Health Check Flow
+
+```mermaid
+sequenceDiagram
+    participant UI as Monitoring System
+    participant API as API Service
+    participant HS as Health Service
+    participant DB as PGlite
+    participant NM as Native Modules
+
+    UI->>API: GET /health
+    API->>HS: checkSystemHealth()
+    HS->>HS: Check service initialization state
+    HS->>DB: Test connectivity
+    DB-->>HS: Connection OK
+    HS->>NM: Test native module availability
+    NM-->>HS: Module status
+    HS-->>API: Health report
+    API-->>UI: System status (healthy/degraded starting)
+```