@rbalchii/anchor-engine 4.7.0

package/docs/archive/sequence-diagrams.md

@@ -0,0 +1,128 @@
+# Sequence Diagrams for ECE_Core
+
+## 1. Agent Harness Integration Sequence
+
+```mermaid
+sequenceDiagram
+    participant AH as Agent Harness (e.g., OpenCLAW)
+    participant API as REST API (Express)
+    participant SR as Search Service
+    participant DB as PGlite Database
+    participant NS as Native Services (C++)
+    
+    AH->>API: Query request (e.g., "What is the latest state?")
+    API->>SR: Process search request
+    SR->>DB: Execute Tag-Walker search
+    DB->>NS: Native module acceleration (if needed)
+    NS-->>DB: Processed results
+    DB-->>SR: Retrieved context atoms
+    SR-->>API: Formatted context results
+    API-->>AH: Return context + agent logic response
+```
+
+## 2. Data Ingestion Sequence
+
+```mermaid
+sequenceDiagram
+    participant FC as File Content
+    participant WD as Watchdog Service
+    participant IS as Ingestion Service
+    participant NS as Native Services (C++)
+    participant DB as PGlite Database
+    
+    FC->>WD: New file detected in context/
+    WD->>IS: Ingest content request
+    IS->>NS: Sanitize content (Key Assassin)
+    NS-->>IS: Cleaned content
+    IS->>NS: Split content (Atomizer)
+    NS-->>IS: Semantic molecules
+    IS->>NS: Generate fingerprint (SimHash)
+    NS-->>IS: Fingerprint hash
+    IS->>DB: Store compound/molecules/atoms
+    DB-->>IS: Storage confirmation
+    IS-->>WD: Ingestion complete
+```
+
+## 3. Search Workflow Sequence
+
+```mermaid
+sequenceDiagram
+    participant U as User/Agent
+    participant API as REST API
+    participant SS as Search Service
+    participant TP as Tag-Walker Protocol
+    participant DB as PGlite Database
+    participant NLP as NLP Processor
+    
+    U->>API: Search query ("revenue optimization summary")
+    API->>SS: Process search request
+    SS->>NLP: Parse natural language
+    NLP-->>SS: Refined query terms
+    SS->>TP: Execute Tag-Walker search
+    TP->>DB: Phase 1 - Strict search (GIN index)
+    DB-->>TP: Initial results or empty
+    alt No results found
+        TP->>DB: Phase 2 - Fuzzy fallback (OR logic)
+        DB-->>TP: Fallback results
+    end
+    TP->>DB: Phase 3 - Semantic walk (related atoms)
+    DB-->>TP: Related context atoms
+    TP-->>SS: Final result set
+    SS-->>API: Formatted search results
+    API-->>U: Return context window
+```
+
+## 4. Stateless Context Retrieval Sequence
+
+```mermaid
+sequenceDiagram
+    participant AH as Agent Harness
+    participant API as REST API
+    participant CS as Context Service
+    participant DB as PGlite Database
+    participant IS as Inference Service
+    
+    AH->>API: Query with context requirement
+    API->>CS: Retrieve relevant context
+    CS->>DB: Execute semantic search
+    DB-->>CS: Retrieved context atoms
+    CS-->>API: Formatted context
+    API->>IS: Combine context + original query
+    IS-->>API: Processed response
+    API-->>AH: Return final response
+```
+
+## 5. Native Module Integration Sequence
+
+```mermaid
+sequenceDiagram
+    participant JS as JavaScript Layer
+    participant NAPI as N-API Boundary
+    participant CPP as C++ Native Module
+    participant RES as Result
+    
+    JS->>NAPI: Call native function (e.g., atomize)
+    NAPI->>CPP: Pass parameters via N-API
+    CPP->>CPP: Execute performance-critical operation
+    CPP-->>NAPI: Return result via N-API
+    NAPI-->>JS: Processed result
+    JS-->>RES: Final output to caller
+```
+
+## 6. Server Startup Sequence (Standard 088 Compliant)
+
+```mermaid
+sequenceDiagram
+    participant INIT as Initialization
+    participant SRV as HTTP Server
+    participant DB as Database
+    participant SVC as Services
+    
+    INIT->>SRV: Start HTTP server
+    SRV-->>INIT: Server listening on port
+    INIT->>DB: Initialize database (background)
+    DB-->>INIT: Database ready notification
+    INIT->>SVC: Initialize services
+    SVC-->>INIT: Services ready
+    Note over SRV,DB,SVC: Server available during DB initialization
+```

package/docs/archive/system_components_diagram.md

@@ -0,0 +1,296 @@
+# System Components Diagram for ECE_Core
+
+## High-Level Architecture
+
+```mermaid
+graph TB
+    subgraph "User Interface Layer"
+        UI[Web Interface<br/>React/Vanilla]
+        EO[Electron Overlay<br/>Always-on-Top]
+        CLI[Command Line Interface<br/>Node.js Scripts]
+    end
+
+    subgraph "API Gateway Layer"
+        AG[Express.js Server<br/>REST API & Health Checks]
+        AP[API Routes<br/>/v1/memory/*, /health, /monitoring]
+        MW[Middlewares<br/>CORS, Body Parsing, Auth]
+    end
+
+    subgraph "Core Engine Layer"
+        CE[Node.js Shell<br/>Orchestration & Networking]
+
+        subgraph "Native Modules (C++)"
+            AM[Atomizer<br/>Content Splitting]
+            KA[Key Assassin<br/>Content Sanitization]
+            FP[Fingerprint<br/>SimHash Generation]
+            DS[Distance<br/>Similarity Calculation]
+        end
+
+        subgraph "Database Layer"
+            DB[(PGlite<br/>PostgreSQL-Compatible)]
+            TA[Atoms Table<br/>Individual Knowledge Units]
+            TT[Tags Table<br/>Semantic Relationships]
+            TE[Edges Table<br/>Connections Between Atoms]
+            TS[Sources Table<br/>Document Origins]
+        end
+
+        subgraph "Core Services"
+            IS[Ingestion Service<br/>Content Processing Pipeline]
+            SS[Search Service<br/>Tag-Walker Protocol]
+            WS[Watchdog Service<br/>File System Monitoring]
+            DS[Dreamer Service<br/>Background Processing]
+            BS[Backup Service<br/>Snapshot Management]
+        end
+    end
+
+    subgraph "Utilities & Infrastructure"
+        PM[Path Manager<br/>Cross-Platform Paths]
+        NMM[Native Module Manager<br/>Graceful Degradation]
+        RM[Resource Manager<br/>Memory Optimization]
+        LM[Logger Manager<br/>Structured Logging]
+    end
+
+    UI --> AG
+    EO --> AG
+    CLI --> AG
+    AG --> AP
+    AP --> CE
+    CE --> AM
+    CE --> KA
+    CE --> FP
+    CE --> DS
+    CE --> DB
+    CE --> IS
+    CE --> SS
+    CE --> WS
+    CE --> DS
+    CE --> BS
+    DB --> TA
+    DB --> TT
+    DB --> TE
+    DB --> TS
+    CE --> PM
+    CE --> NMM
+    CE --> RM
+    CE --> LM
+```
+
+## Service Interaction Flow
+
+```mermaid
+sequenceDiagram
+    participant UI as User Interface
+    participant API as API Service
+    participant IS as Ingestion Service
+    participant SS as Search Service
+    participant WS as Watchdog Service
+    participant DB as PGlite Database
+    participant NS as Native Services
+
+    UI->>API: Ingest content request
+    API->>IS: Process ingestion
+    IS->>NS: Native atomization
+    NS-->>IS: Processed atoms
+    IS->>DB: Store atoms with deduplication
+    DB-->>IS: Storage confirmation
+    IS-->>API: Ingestion result
+    API-->>UI: Confirmation
+
+    Note over WS, DB: File system monitoring runs continuously
+    WS->>WS: Detect file changes
+    WS->>IS: Trigger ingestion for new files
+    IS->>NS: Process new content
+    NS-->>IS: Processed atoms
+    IS->>DB: Store new atoms
+    DB-->>IS: Confirmation
+
+    UI->>API: Search query
+    API->>SS: Execute search
+    SS->>DB: Query database
+    DB-->>SS: Retrieved results
+    SS-->>API: Formatted results
+    API-->>UI: Search results
+```
+
+## Data Flow Architecture
+
+```mermaid
+graph LR
+    subgraph "Input Sources"
+        FS[File System<br/>context/ directory]
+        API_IN[API Ingestion<br/>POST /v1/ingest]
+        WB[Web Scraping<br/>Research Plugin]
+    end
+
+    subgraph "Processing Pipeline"
+        RF[Refiner<br/>Content Sanitization]
+        AT[Atomizer<br/>Content Splitting]
+        FN[Fingerprint<br/>SimHash Generation]
+        TG[Tagger<br/>Semantic Classification]
+        ST[Storage<br/>Database Insertion]
+    end
+
+    subgraph "Storage Layer"
+        DB[(PGlite Database)]
+        TB[Atoms Table]
+        TC[Tags Table]
+        TD[Edges Table]
+        TE[Sources Table]
+    end
+
+    subgraph "Retrieval Pipeline"
+        SQ[Search Query<br/>Natural Language]
+        TW[Tag-Walker<br/>Graph Traversal]
+        CT[Context Assembly<br/>Result Aggregation]
+        RP[Response Prep<br/>Output Formatting]
+    end
+
+    subgraph "Output Destinations"
+        UI[User Interface<br/>Search Results]
+        AH[Agent Harness<br/>Context Injection]
+        EX[Export Formats<br/>JSON, CSV, Tables]
+    end
+
+    A --> D
+    B --> D
+    C --> D
+    D --> E
+    E --> F
+    F --> G
+    G --> H
+    H --> I
+    I --> J
+    J --> K
+    K --> L
+    L --> M
+    L --> N
+    L --> O
+    M --> P
+    N --> P
+    O --> P
+```
+
+## Native Module Integration
+
+```mermaid
+graph TD
+    subgraph "Node.js Layer"
+        JS[JavaScript Application<br/>Express, Services, etc.]
+        NAPI[N-API Boundary<br/>Stable Interface]
+        NM[Native Module Manager<br/>Loading & Fallbacks]
+    end
+
+    subgraph "C++ Native Layer"
+        AM[Atomizer Module<br/>Content Splitting<br/>std::string_view]
+        KM[Key Assassin Module<br/>Content Sanitization<br/>Zero-Copy Processing]
+        FM[Fingerprint Module<br/>SimHash Generation<br/>Performance Critical]
+        DM[Distance Module<br/>Similarity Calculation<br/>SIMD Optimized]
+    end
+
+    subgraph "Fallback Layer"
+        JSF[JavaScript Fallbacks<br/>Pure JS Implementations]
+    end
+
+    A --> B
+    B --> C
+    C --> D
+    C --> E
+    C --> F
+    C --> G
+    D --> H
+    E --> H
+    F --> H
+    G --> H
+    H --> I
+    H --> J
+```
+
+## Agent Harness Integration Architecture
+
+```mermaid
+graph TB
+    subgraph "Agent Harness (e.g., OpenCLAW)"
+        AH[Agent Query<br/>Natural Language]
+        AL[Agent Logic<br/>Reasoning & Planning]
+        AR[Agent Response<br/>Processed Output]
+    end
+
+    subgraph "ECE/Anchor Data Atomization Service"
+        AA[Query Processing<br/>NLP & Intent Detection]
+        AB[Context Retrieval<br/>Tag-Walker Search]
+        AC[Semantic Search<br/>Graph Traversal]
+        AD[Data Formatting<br/>JSON/CSV/Tables]
+    end
+
+    subgraph "Database Layer"
+        AE[(PGlite Storage)]
+        AF[Atoms Table<br/>Knowledge Units]
+        AG[Tags Table<br/>Relationships]
+        AH[Edges Table<br/>Connections]
+    end
+
+    subgraph "Native Acceleration"
+        AI[Native Modules<br/>Performance Boost]
+        AJ[C++ Processing<br/>SIMD Operations]
+    end
+
+    A --> D
+    B --> E
+    C --> F
+    D --> G
+    E --> G
+    F --> G
+    G --> H
+    G --> I
+    G --> J
+    H --> K
+    I --> K
+    J --> K
+    K --> L
+    K --> M
+    L --> N
+    M --> N
+```
+
+## Monitoring & Health Architecture
+
+```mermaid
+graph LR
+    subgraph "Health Monitoring"
+        HA[Health API<br/>GET /health]
+        HB[Component Status<br/>Database, Services, etc.]
+        HC[Performance Metrics<br/>Response Times, Memory]
+        HD[Resource Utilization<br/>CPU, Disk, Network]
+    end
+
+    subgraph "Monitoring Endpoints"
+        ME[/monitoring/metrics<br/>Prometheus Format]
+        ML[/monitoring/logs<br/>Recent Log Entries]
+        MC[/monitoring/components<br/>Detailed Status]
+        MR[/monitoring/resources<br/>System Resources]
+    end
+
+    subgraph "Alerting & Notifications"
+        AA[Anomaly Detection<br/>Performance Thresholds]
+        AB[Error Tracking<br/>Exception Monitoring]
+        AC[Resource Alerts<br/>Memory, Disk Space]
+        AD[Service Degradation<br/>Component Failures]
+    end
+
+    A --> E
+    B --> F
+    C --> G
+    D --> H
+    E --> I
+    F --> I
+    G --> I
+    H --> I
+    I --> J
+    I --> K
+    I --> L
+    I --> M
+    J --> N
+    K --> N
+    L --> N
+    M --> N
+```

package/docs/archive/test-framework-integration.md

@@ -0,0 +1,109 @@
+# ECE_Core Test Framework Integration Guide
+
+## Overview
+
+This document explains how the new centralized test framework integrates with the existing test suite in the ECE_Core project. The new framework enhances the existing tests with better organization, performance monitoring, and standardized execution patterns.
+
+## Current Test Structure
+
+The existing test suite is located in `engine/tests/` and contains various test files for different components:
+
+- `suite.js` - Main test suite that validates core API functionality
+- `test_*.ts/js` - Individual test files for specific components
+- `context_experiments.js` - Verification scripts for context experiments
+- `native_*` - Tests for native module functionality
+- `atomic_*` - Tests for atomic architecture components
+
+## Integration Approach
+
+Rather than replacing the existing tests, the new framework provides:
+
+1. **Enhanced Execution**: The new framework can run alongside existing tests
+2. **Performance Monitoring**: Adds performance tracking to existing tests
+3. **Centralized Reporting**: Consolidates results from all test types
+4. **Standardized Patterns**: Provides templates for new tests while preserving existing functionality
+
+## Migration Strategy
+
+### Phase 1: Coexistence
+- Keep existing tests running as before
+- Add new tests using the centralized framework
+- Gradually migrate critical tests to new framework
+
+### Phase 2: Enhancement
+- Wrap existing tests with performance monitoring
+- Add diagnostic capabilities to existing tests
+- Implement common setup/teardown patterns
+
+### Phase 3: Consolidation
+- Move non-critical tests to new framework
+- Maintain legacy tests that work well as-is
+- Archive obsolete test patterns
+
+## Running Tests
+
+### With Existing System
+```bash
+npm test
+# or
+node tests/suite.js
+```
+
+### With New Framework
+```bash
+# Run all tests through new framework
+npm run test:enhanced
+
+# Run specific test categories
+npm run test:performance
+npm run test:diagnostic
+
+# Run with detailed reporting
+npm run test:report
+```
+
+## Architecture
+
+The new test framework adds these capabilities:
+
+- **Performance Monitoring**: Tracks execution time and resource usage
+- **Diagnostic Tests**: Quick validation for issue reproduction
+- **Dataset-Specific Tests**: Configurable tests for different data sizes
+- **Centralized Configuration**: Unified settings for all test types
+- **Enhanced Reporting**: Detailed metrics and visualization
+
+## Best Practices
+
+1. **Preserve Existing Tests**: Don't remove working tests, enhance them
+2. **Gradual Migration**: Move tests incrementally to minimize disruption
+3. **Performance Tracking**: Add performance monitoring to critical paths
+4. **Diagnostic Capability**: Ensure each component has quick diagnostic tests
+5. **Documentation**: Keep test documentation synchronized
+
+## File Organization
+
+```
+engine/
+├── tests/                    # Existing tests (preserve)
+│   ├── suite.js             # Main test suite
+│   ├── context_experiments.js
+│   ├── native_tests/
+│   ├── atomic_tests/
+│   └── ...
+├── test-framework/          # New framework
+│   ├── core.ts              # Core framework
+│   ├── config.ts            # Configuration management
+│   ├── diagnostic-tests.ts  # Diagnostic tests
+│   ├── performance-tests.ts # Performance tests
+│   └── dataset-runner.ts    # Dataset-specific tests
+└── docs/
+    └── testing-framework.md # Documentation
+```
+
+## Next Steps
+
+1. Implement the Glass Panel UI integration for test visualization
+2. Create React components for test dashboard
+3. Add real-time test monitoring capabilities
+4. Implement test result persistence
+5. Create test automation pipelines