yakmesh 2.8.2 → 3.0.0

package/docs/WHITEPAPER.md

@@ -0,0 +1,623 @@
+# YAKMESH™ Whitepaper
+
+## A Hardened Yielding Atomic Kernel for Post-Quantum Mesh Orchestration
+
+**Author:** yakmesh.dev  
+**Date:** January 2026  
+**USPTO Serial No:** 99594620  
+**Version:** 1.5.0  
+
+---
+
+## Table of Contents
+
+1. [Executive Summary](#1-executive-summary)
+2. [Theoretical Framework: The Yielding Atomic Kernel](#2-theoretical-framework-the-yielding-atomic-kernel)
+3. [Technical Architecture](#3-technical-architecture)
+   - 3.1 [Physical Layer: Hardware Alignment](#31-physical-layer-hardware-alignment)
+   - 3.2 [Temporal Matrix Encoding (TME)](#32-temporal-matrix-encoding-tme)
+   - 3.3 [Security Hardening Modules](#33-security-hardening-modules)
+   - 3.4 [Yakmesh Annex: Encrypted P2P Channels](#34-yakmesh-annex-encrypted-p2p-channels)
+4. [Comparison: YAKMESH vs. Walrus (Red Stuff)](#4-comparison-yakmesh-vs-walrus-red-stuff)
+5. [Security & Cryptographic Hardening](#5-security--cryptographic-hardening)
+6. [Real-World Use Cases](#6-real-world-use-cases)
+7. [Roadmap & Conclusion](#7-roadmap--conclusion)
+
+---
+
+## 1. Executive Summary
+
+### The Problem
+
+Current decentralized networks struggle with the **"Inconsistency Gap"**—the latency between nodes in asynchronous environments. Existing protocols (like Walrus/Red Stuff) solve this for **storage** but fail in **real-time transmission**.
+
+The gap manifests in three critical ways:
+
+1. **Latency Amplification:** Retransmission-based recovery adds round-trip delays
+2. **Harvest Now, Decrypt Later (HNDL):** Quantum computers will break today's encryption
+3. **Byzantine Vulnerability:** Malicious nodes can poison routing tables and timing
+
+### The Solution
+
+YAKMESH introduces the **Yielding Atomic Kernel (YAK)**, a networking layer that utilizes:
+
+- **Hardware-based atomic timing** (PCIe/PTP) for nanosecond synchronization
+- **Post-Quantum Cryptography** (ML-DSA-65, FIPS 204) for quantum resistance
+- **Temporal Matrix Encoding (TME)** for zero-retransmit packet recovery
+
+The result: a "Synchronous Mesh" in an asynchronous world.
+
+---
+
+## 2. Theoretical Framework: The Yielding Atomic Kernel
+
+### 2.1 The "Yielding" Concept
+
+The kernel is designed to **gracefully degrade** under high interference without losing synchronization. When a node experiences:
+
+- High packet loss → TME reconstructs from temporal slices
+- Clock drift → Atomic sync re-calibrates from mesh consensus
+- Byzantine attack → NAVR and reputation systems isolate the threat
+
+The kernel "yields" resources to maintain core timing integrity, never sacrificing the mesh timebase for throughput.
+
+### 2.2 Atomic Precision
+
+YAKMESH utilizes nanosecond-level hardware timestamps to eliminate "drift" in mesh routing tables:
+
+| Timing Method | Precision | YAKMESH Suitability |
+|---------------|-----------|---------------------|
+| NTP | ~10ms | ❌ Too coarse |
+| SNTP | ~1ms | ❌ Still insufficient |
+| PTP (IEEE 1588) | ~100ns | ✅ Baseline |
+| PCIe Atomic Clock | ~1ns | ✅ Optimal |
+
+This precision enables **Temporal Matrix Encoding**—impossible with traditional timing.
+
+### 2.3 Security Layer
+
+Integration of NIST-standardized Post-Quantum Cryptography:
+
+- **ML-DSA-65 (FIPS 204):** Digital signatures resistant to Shor's algorithm
+- **Cryptographic Time Binding:** Each packet's hash includes its timestamp
+- **Temporal Chain Integrity:** Packets form an immutable chain via `prevTemporalHash`
+
+---
+
+## 3. Technical Architecture
+
+### 3.1 Physical Layer: Hardware Alignment
+
+#### The Role of PCIe Atomic Clock Interfaces
+
+Traditional network timing (NTP/SNTP) suffers from:
+
+- **Jitter:** Variable network latency corrupts timestamps
+- **Drift:** Crystal oscillators diverge over hours
+- **Spoofing:** Software timestamps can be forged
+
+YAKMESH's PCIe-based timing interface establishes a **Universal Mesh Timebase**:
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    YAKMESH NODE                             │
+├─────────────────────────────────────────────────────────────┤
+│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐  │
+│  │ Application  │    │   YAK Core   │    │  PCIe Clock  │  │
+│  │    Layer     │◄──►│   (Kernel)   │◄──►│  Interface   │  │
+│  └──────────────┘    └──────────────┘    └──────────────┘  │
+│         │                   │                   │          │
+│         ▼                   ▼                   ▼          │
+│  ┌──────────────────────────────────────────────────────┐  │
+│  │              Atomic Mesh Timebase (±1ns)             │  │
+│  └──────────────────────────────────────────────────────┘  │
+└─────────────────────────────────────────────────────────────┘
+```
+
+#### Comparative Analysis: Why NTP is Insufficient
+
+| Requirement | NTP | YAKMESH Atomic |
+|-------------|-----|----------------|
+| Sub-ms routing decisions | ❌ | ✅ |
+| Temporal slice alignment | ❌ | ✅ |
+| Tamper-resistant timestamps | ❌ | ✅ |
+| Cross-continent sync | ⚠️ ~50ms | ✅ ~100ns |
+
+---
+
+### 3.2 Temporal Matrix Encoding (TME)
+
+> *"Time IS the redundancy dimension."*
+
+#### The Space vs. Time Paradigm
+
+Traditional erasure coding (Walrus/Red Stuff) treats a network like a **Hard Drive**—data is spread across spatial nodes.
+
+YAKMESH TME treats the network like a **Synthesizer**—data is spread across temporal intervals.
+
+| Approach | Walrus (Spatial) | YAKMESH (Temporal) |
+|----------|------------------|-------------------|
+| **Encoding Dimension** | Space (N nodes) | Time (T intervals) |
+| **Recovery Method** | Quorum from K nodes | Reconstruction from adjacent slices |
+| **Sync Requirement** | Asynchronous | Atomically synchronized |
+| **Optimized For** | Static blobs | Dynamic packet flow |
+| **Latency on Loss** | Full retransmit | Zero (parity recovery) |
+
+#### How TME Works
+
+**Step 1: Temporal Slicing**
+
+A message is divided into slices, each bound to a specific timestamp:
+
+```
+Original Message: "Hello YAKMESH World!"
+
+Slice 0 @ T=0ns:    "Hello" + timestamp + prevHash → temporalHash₀
+Slice 1 @ T=50ms:   " YAKM" + timestamp + hash₀    → temporalHash₁  
+Slice 2 @ T=100ms:  "ESH W" + timestamp + hash₁    → temporalHash₂
+Slice 3 @ T=150ms:  "orld!" + timestamp + hash₂    → temporalHash₃
+```
+
+**Step 2: Cryptographic Time Binding**
+
+Each slice's `temporalHash` is computed from:
+
+```javascript
+temporalHash = SHA256(
+  data +           // Payload bytes
+  timestamp +      // Nanosecond precision BigInt
+  sequenceNumber + // Position in stream
+  streamId +       // Unique stream identifier
+  prevTemporalHash + // Chain to previous slice
+  meshPosition     // [x, y, z] topology coordinates
+)
+```
+
+This creates an **immutable temporal chain**. If any slice is tampered with, the chain breaks.
+
+**Step 3: Multi-Path Transmission**
+
+Slices are sent across different mesh paths:
+
+```
+Sender ──Path A──► Slice 0 ──────────► Receiver
+       ──Path B──► Slice 1 ──────────► Receiver
+       ──Path C──► Slice 2 ──(LOST)──► Receiver
+       ──Path A──► Slice 3 ──────────► Receiver
+```
+
+**Step 4: Temporal Reconstruction**
+
+If Slice 2 is lost, the receiver:
+
+1. Detects the gap via sequence numbers
+2. Verifies Slice 3's `prevTemporalHash` points to the expected Slice 2 hash
+3. Requests **Timing Proofs** from mesh neighbors who saw Slice 2
+4. Achieves consensus: "Slice 2 existed with hash X at time T"
+5. Reconstructs or requests the specific slice from a neighbor
+
+**No full retransmit. No round-trip delay.**
+
+#### The Atomic Sync Secret Sauce
+
+TME is **only possible** because of atomic synchronization:
+
+- **Traditional PTP/NTP:** Has too much jitter. The receiver can't distinguish "late packet" from "wrong temporal slice."
+- **YAKMESH Atomic Sync:** All nodes share Universal Mesh Time. A packet arriving at T+5ns is unambiguously Slice N, not Slice N+1.
+
+You are essentially treating the entire mesh as one giant, distributed CPU clock.
+
+#### Implementation: Core Classes
+
+```javascript
+import { 
+  TemporalSlice,        // Atomic unit with time binding
+  TemporalStream,       // Message → slices with chaining
+  TemporalReconstructor, // Recovery via timing proofs
+  TemporalMeshEncoder   // High-level encode/decode API
+} from 'yakmesh/mesh/temporal-encoder.js';
+
+// Encode a message
+const encoder = new TemporalMeshEncoder({ meshPosition: [1, 2, 3] });
+const { streamId, slices, metadata } = encoder.encode('Hello TME!');
+
+// Receive and decode
+const decoder = new TemporalMeshEncoder();
+decoder.initReceive({ streamId, ...metadata });
+for (const slice of slices) {
+  decoder.receiveSlice(slice);
+}
+const { success, data } = decoder.decode(streamId);
+```
+
+---
+
+### 3.3 Security Hardening Modules
+
+YAKMESH includes defense-in-depth security layers:
+
+#### 3.3.1 NAVR (Network Assimilation Validation Routine)
+
+> *"You're NAVR getting in without solving this puzzle."*
+
+A computational challenge for new node registration:
+
+- **Not blockchain PoW:** One-time puzzle, not per-transaction mining
+- **Configurable difficulty:** Scales with network threat level
+- **Purpose:** Prevents Sybil attacks by making identity creation costly
+
+```javascript
+import { NAVR } from 'yakmesh/mesh/sybil-defense.js';
+
+const navr = new NAVR({ difficulty: 16 });
+const challenge = navr.createChallenge('new-node-id');
+const solution = navr.solve(challenge);  // CPU-bound work
+const valid = navr.verify(challenge, solution);
+```
+
+#### 3.3.2 Reputation Tracker
+
+Trust scoring for mesh nodes (0.0 to 1.0):
+
+| Behavior | Trust Impact |
+|----------|--------------|
+| Valid packets forwarded | +0.01 |
+| Timing proof provided | +0.02 |
+| Invalid signature | -0.1 |
+| Timing manipulation | -0.3 |
+| Sybil attempt detected | -1.0 (ban) |
+
+#### 3.3.3 Subnet Diversity
+
+Eclipse attack prevention:
+
+- Maximum 3 connections per /24 subnet
+- Ensures no single ISP can dominate a node's view
+- Automatic connection balancing
+
+#### 3.3.4 Replay Defense
+
+Multi-layer protection against message replay:
+
+| Layer | Mechanism | Window |
+|-------|-----------|--------|
+| Nonce Registry | 32-byte cryptographic nonces | 1 hour expiry |
+| Timestamp Validator | 10-minute freshness window | ±10 minutes |
+| Sequence Tracker | Per-sender message ordering | 64-item window |
+
+#### 3.3.5 Message Validator
+
+Amplification attack prevention:
+
+| Message Type | Size Limit |
+|--------------|------------|
+| Gossip | 64 KB |
+| Handshake | 8 KB |
+| Listing | 128 KB |
+| Maximum | 1 MB |
+
+Additional protections:
+
+- Nesting depth limit: 10 levels
+- Prototype pollution protection (`__proto__` blocked)
+
+---
+
+### 3.4 Yakmesh Annex: Encrypted P2P Channels
+
+> *"Changes pass through math alone."*
+
+#### The Need for Sovereign Data Channels
+
+While the gossip protocol and TME handle public mesh communication, many use cases require **private, authenticated messaging** between specific peers:
+
+- Beacon acknowledgments that shouldn't reveal recipient identity
+- Application-specific payloads requiring confidentiality
+- Authentication tokens for CDN/site access
+- Direct peer-to-peer secure messaging
+
+#### ANNEX: Autonomous Network Negotiated Encrypted eXchange
+
+ANNEX establishes encrypted point-to-point channels using post-quantum cryptography:
+
+| Component | Algorithm | Purpose |
+|-----------|-----------|---------|
+| **Key Exchange** | ML-KEM-768 (Kyber) | NIST FIPS 203 post-quantum KEM |
+| **Encryption** | AES-256-GCM | Authenticated symmetric encryption |
+| **Authentication** | ML-DSA-65 | Signature verification on all messages |
+| **Replay Defense** | Sequence + AAD | Sequence numbers bound to session |
+
+#### How ANNEX Works
+
+**Step 1: Channel Opening (Key Exchange)**
+
+```text
+Initiator                                  Responder
+    │                                          │
+    │  KEY_EXCHANGE(sessionId, kemPublicKey)   │
+    │────────────────────────────────────────►│
+    │                                          │
+    │  KEY_RESPONSE(kemPublicKey, kemCiphertext)│
+    │◄────────────────────────────────────────│
+    │                                          │
+    ▼  Both derive shared secret via ML-KEM    ▼
+    [═══════ Encrypted Channel Established ═══════]
+```
+
+**Step 2: Message Encryption**
+
+Each message is encrypted with AES-256-GCM:
+
+```javascript
+// Additional Authenticated Data (AAD) includes session + sequence
+const aad = `${sessionId}:${sequenceNumber}`;
+
+// Encrypt with random nonce
+const { ciphertext, authTag, nonce } = encrypt(payload, key, aad);
+
+// Envelope is signed with ML-DSA-65
+envelope.signature = identity.sign(envelope);
+```
+
+**Step 3: Perfect Forward Secrecy**
+
+ANNEX automatically re-keys sessions:
+
+| Trigger | Action |
+|---------|--------|
+| 5 minutes elapsed | Generate new ephemeral KEM keys |
+| 10,000 messages sent | Force re-key |
+| Session expiry (1 hour) | Close and re-establish |
+
+This ensures that compromise of a session key doesn't expose past or future communications.
+
+#### Security Properties
+
+1. **Post-Quantum Confidentiality:** ML-KEM-768 is resistant to Shor's algorithm
+2. **Authentication:** Every envelope signed with ML-DSA-65
+3. **Replay Protection:** Sequence numbers + AAD verification
+4. **Forward Secrecy:** Automatic ephemeral key rotation
+5. **Integrity:** AES-GCM authenticated encryption
+
+#### API Usage
+
+```javascript
+import { Annex } from 'yakmesh/mesh/annex.js';
+
+// Initialize with node identity and mesh connection
+const annex = new Annex({ identity, mesh });
+
+// Open encrypted channel to peer
+await annex.openChannel(peerNodeId);
+
+// Send encrypted message
+await annex.send(peerNodeId, {
+  type: 'beacon_ack',
+  beaconId: 'emergency-123',
+  ack: true,
+});
+
+// Receive messages
+annex.onMessage(({ from, payload, sessionId }) => {
+  console.log(`Decrypted from ${from}:`, payload);
+});
+
+// Close channel
+await annex.closeChannel(peerNodeId);
+```
+
+#### Integration with Yakmesh Protocol Stack
+
+ANNEX slots into the protocol stack as the private messaging layer:
+
+```text
+┌─────────────────────────────────────────────────────────────┐
+│                    YAKMESH PROTOCOL STACK                   │
+├─────────────────────────────────────────────────────────────┤
+│  HTTP API          │ Public content delivery (CDN layer)    │
+├─────────────────────────────────────────────────────────────┤
+│  ANNEX             │ ML-KEM768 + AES-256-GCM encrypted P2P  │
+├─────────────────────────────────────────────────────────────┤
+│  Gossip            │ ML-DSA-65 signed message propagation   │
+├─────────────────────────────────────────────────────────────┤
+│  Beacon            │ Flood-based priority broadcasts        │
+├─────────────────────────────────────────────────────────────┤
+│  Nakpak            │ Post-quantum onion routing (NAKPAK)    │
+├─────────────────────────────────────────────────────────────┤
+│  Sherpa            │ Peer discovery DHT (SHERPA)            │
+├─────────────────────────────────────────────────────────────┤
+│  Mesh Core         │ WebSocket + Code Proof Protocol        │
+└─────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 4. Comparison: YAKMESH vs. Walrus (Red Stuff)
+
+### 4.1 Storage vs. Flow
+
+| Aspect | Walrus | YAKMESH |
+|--------|--------|---------|
+| **Purpose** | Static blob storage | Dynamic packet transmission |
+| **Data Lifetime** | Epochs (days/weeks) | Milliseconds |
+| **Node Requirements** | Storage capacity | Atomic timing hardware |
+| **Recovery Unit** | Data shard | Temporal slice |
+
+### 4.2 Encoding Philosophy
+
+**Walrus (2D Erasure Coding):**
+```
+     Node1  Node2  Node3
+Row1  [A]    [B]    [P_row1]
+Row2  [C]    [D]    [P_row2]
+     [P_c1] [P_c2]
+
+Recovery: Download from surviving nodes
+```
+
+**YAKMESH (Temporal Matrix Encoding):**
+```
+T=0ns:   [Slice0] → Path A
+T=50ms:  [Slice1] → Path B
+T=100ms: [Slice2] → Path C
+T=150ms: [Slice3] → Path A
+
+Recovery: Reconstruct from adjacent temporal slices + timing proofs
+```
+
+### 4.3 Synergy: The Transit Layer
+
+YAKMESH is the optimal **Transit Layer** for protocols like Walrus:
+
+```
+┌─────────────────────────────────────────────────┐
+│              APPLICATION LAYER                  │
+├─────────────────────────────────────────────────┤
+│  Walrus Storage  │  DeFi  │  IoT  │  Messaging  │
+├─────────────────────────────────────────────────┤
+│           YAKMESH TRANSIT LAYER                 │
+│    (TME + Atomic Sync + Post-Quantum Crypto)    │
+├─────────────────────────────────────────────────┤
+│              PHYSICAL NETWORK                   │
+└─────────────────────────────────────────────────┘
+```
+
+---
+
+## 5. Security & Cryptographic Hardening
+
+### 5.1 Quantum Resistance
+
+YAKMESH implements NIST-standardized post-quantum algorithms:
+
+| Algorithm | NIST Standard | Purpose | YAKMESH Usage |
+|-----------|---------------|---------|---------------|
+| ML-DSA-65 | FIPS 204 | Digital Signatures | Node identity, packet signing |
+| ML-KEM-768 | FIPS 203 | Key Encapsulation | Future: Session key exchange |
+
+**Why ML-DSA-65?**
+- Security Level 3 (128-bit post-quantum security)
+- Smaller signatures than Dilithium-5
+- Optimized for high-throughput signing
+
+### 5.2 Byzantine Fault Tolerance
+
+The Atomic Kernel identifies and "Yields" (isolates) malicious nodes:
+
+1. **Timing Manipulation Detection:** Nodes with >5ms clock drift are flagged
+2. **Reputation Degradation:** Bad behavior reduces trust score
+3. **Automatic Isolation:** Trust < 0.2 triggers connection termination
+4. **Subnet Rebalancing:** Connections redistributed to diverse subnets
+
+### 5.3 Quantum-Temporal Security
+
+By encoding cryptographic material across temporal slices:
+
+- An adversary must capture **100% of the temporal stream**
+- With **perfect timing precision** (sub-nanosecond)
+- Across **multiple mesh paths simultaneously**
+
+This makes HNDL attacks computationally infeasible even for quantum adversaries.
+
+---
+
+## 6. Real-World Use Cases
+
+### 6.1 Edge Intelligence
+
+Low-latency machine-to-machine communication for autonomous systems:
+
+- Self-driving vehicle mesh networks
+- Drone swarm coordination
+- Industrial robotics synchronization
+
+**TME Benefit:** Zero-retransmit recovery critical for real-time control loops.
+
+### 6.2 Critical Infrastructure
+
+Private, quantum-secure mesh networks for:
+
+- Power grid monitoring
+- Water treatment facilities
+- Transportation systems
+
+**Security Benefit:** ML-DSA-65 protects against future quantum attacks on long-lived infrastructure.
+
+### 6.3 Financial Systems
+
+High-frequency trading and settlement networks:
+
+- Atomic timestamp on every transaction
+- Verifiable ordering via temporal chain
+- Post-quantum protection for regulatory compliance
+
+### 6.4 Global Discovery Hub
+
+The `yakmesh.dev` portal serves as:
+
+- Node discovery and registration
+- Mesh topology visualization
+- Orchestration API for distributed deployments
+
+---
+
+## 7. Roadmap & Conclusion
+
+### 7.1 Development Phases
+
+| Phase | Timeline | Deliverables |
+|-------|----------|--------------|
+| **Phase 1** | Q1 2026 ✅ | Core kernel, TME, security modules, npm package |
+| **Phase 2** | Q2 2026 | Hardware reference designs (PCIe timing cards) |
+| **Phase 3** | Q3 2026 | Production mesh deployment, monitoring dashboard |
+| **Phase 4** | Q4 2026 | ML-KEM integration, full post-quantum handshake |
+
+### 7.2 Current Implementation Status
+
+| Component | Status | npm Package |
+|-----------|--------|-------------|
+| Network Identity Unification | ✅ Complete | yakmesh@1.5.0 |
+| Yakmesh Annex (Encrypted P2P) | ✅ Complete | yakmesh@1.4.0 |
+| Content Delivery API | ✅ Complete | yakmesh@1.3.2 |
+| TME (Temporal Matrix Encoding) | ✅ Complete | yakmesh@1.2.0 |
+| NAVR (Sybil Defense) | ✅ Complete | yakmesh@1.1.0 |
+| Replay Defense | ✅ Complete | yakmesh@1.1.0 |
+| Message Validator | ✅ Complete | yakmesh@1.1.0 |
+| Rate Limiter | ✅ Complete | yakmesh@1.0.3 |
+| ML-DSA-65 Signatures | ✅ Complete | yakmesh@1.0.0 |
+
+### 7.3 Conclusion
+
+The move toward a **"Sturdy & Secure"** internet begins at the kernel level.
+
+YAKMESH provides:
+
+1. **Temporal Resilience:** TME eliminates retransmission latency
+2. **Quantum Security:** ML-DSA-65 protects against HNDL attacks
+3. **Byzantine Tolerance:** Multi-layer defense against malicious nodes
+4. **Atomic Precision:** Hardware-based timing enables new paradigms
+
+The Yielding Atomic Kernel is not just an incremental improvement—it is a fundamental rethinking of how mesh networks achieve resilience.
+
+---
+
+## References
+
+- NIST FIPS 204: Module-Lattice-Based Digital Signature Standard (ML-DSA)
+- NIST FIPS 203: Module-Lattice-Based Key-Encapsulation Mechanism Standard (ML-KEM)
+- IEEE 1588-2019: Precision Time Protocol (PTP)
+- Walrus Whitepaper: 2D Erasure Coding for Decentralized Storage
+
+---
+
+## Legal
+
+**YAKMESH™** is a trademark. USPTO Serial No. 99594620.
+
+**TME (Temporal Matrix Encoding)** is a proprietary technology of the YAKMESH project.
+
+This whitepaper is provided for informational purposes. Implementation details may change.
+
+---
+
+*Powered by TME — The world's first temporal-erasure protocol for atomically-synced mesh networks.*