vayrm 0.1.0__tar.gz

vayrm-0.1.0/PKG-INFO

@@ -0,0 +1,166 @@
+Metadata-Version: 2.4
+Name: vayrm
+Version: 0.1.0
+Summary: Adaptive Compute Runtime
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: psutil>=5.9.0
+Requires-Dist: cryptography>=41.0.0
+Requires-Dist: PyYAML>=6.0.1
+Requires-Dist: aiohttp>=3.9.0
+Requires-Dist: qrcode>=7.4
+Requires-Dist: pillow>=10.0.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: rich>=13.0.0
+Provides-Extra: macos
+Provides-Extra: windows
+Provides-Extra: linux
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Requires-Dist: ruff>=0.0.260; extra == "dev"
+
+# Adaptive Compute Runtime (ACR)
+
+ACR is an **OS-independent**, **Network-provider-independent**, and **Scheduler-independent** Python library for intelligent, decentralized, and adaptive workload scheduling across heterogeneous machines.
+
+The core principle of ACR is to completely decouple the adaptive scheduling algorithms from static network constraints.
+
+---
+
+## 1. Architecture
+
+ACR is composed of a generic hierarchy that remains portable across macOS, Windows, and Linux.
+
+*   **Root**: The orchestrator role. The Root aggregates telemetry (NodeKnowledge), maintains the network route registry, and uses Schedulers to dispatch ExecutionPlans.
+*   **Node**: The worker role. The Node discovers its local capabilities, continually monitors its telemetry, and securely connects to the Root to advertise its service endpoint.
+*   **Endpoint Model**: ACR strictly distinguishes between the **Observed** IP/port (what the Root sees natively from the socket connection) and the **Advertised** IP/port (what the Node declares as its reachable listener). ACR always **Validates** the advertised endpoint over TCP before updating the internal Route table.
+*   **Authentication & Handshake**: Nodes register using an application-level handshake payload over a persistent TCP session that transmits the cluster ID, session ID, and endpoint topology.
+*   **Telemetry**: Node-native psutil monitors dynamically measure CPU and RAM degradation over time.
+*   **Scheduler**: Schedulers (RoundRobin, LeastLoaded, ResourceAware, HeterogeneityAware, Adaptive) perform workload allocation using real-time NodeSnapshot data without hardcoded hardware assumptions.
+
+---
+
+## 2. Platform Support & Validation Status
+
+**Root and Node are portable roles, not hardware-locked classes.** ACR guarantees identical algorithmic behavior across operating systems. However, validation evidence is tracked strictly:
+
+*   **macOS (Root / Node):**
+    *   Architecture/Implementation: **PASS**
+    *   Automated Deterministic Testing: **PASS**
+    *   Physical TCP Runtime Execution: **PASS**
+*   **Windows / Linux (Root / Node):**
+    *   Architecture/Implementation: **PASS**
+    *   Automated Deterministic Testing: **PASS**
+    *   Physical TCP Runtime Execution: **NOT RUN / DEFERRED** (pending available physical peers)
+
+---
+
+## 3. Running ACR
+
+ACR listener ports are **dynamically assigned by the OS** by default. You do not need to choose or remember a port number.
+
+### Running as Root
+
+```bash
+python3 -m acr.entrypoint --role root
+```
+
+The Root selects an available port at runtime, binds its listener, tests TCP reachability to itself, and publishes its actual endpoint over the discovery locator.
+
+### Running as Node
+
+The Node dynamically selects its own service port, searches for the Root via discovery, probes it via TCP, and connects automatically:
+
+```bash
+# macOS / Linux:
+python3 -m acr.entrypoint --role node
+
+# Windows:
+python -m acr.entrypoint --role node
+```
+
+### The Automatic Flow
+
+1. Root starts, binds to port `0` (dynamic OS assignment).
+2. Root retrieves its Tailscale identity and discovers its assigned port.
+3. Root tests if it can reach itself via TCP.
+4. Root publishes a discovery record on its locator.
+5. Node starts, asks Tailscale for peers.
+6. Node scans peers for the Root locator.
+7. Node extracts Root's dynamic endpoint and runs a TCP reachability probe.
+8. If TCP passes, Node handshakes with Root and advertises its own dynamic endpoint.
+9. Root validates Node's advertised endpoint via TCP probe.
+10. Route is established using the validated endpoints.
+
+### Failure Diagnostics
+
+The runtime actively distinguishes failure modes:
+- **Discovery Failure**: `[ACR NODE] discovery=FAIL`. Node cannot find Root. Ensure Root is running and Tailscale peers are online.
+- **TCP Failure / Firewall Failure**: `[ACR TCP CHECK] result=FAIL`. Discovery worked but connection failed. Windows or macOS Firewall may be blocking the dynamic port. Read the actual port from the log and allow it.
+- **Handshake Failure**: `[ACR NODE] handshake=FAIL`. TCP connected, but protocol failed (e.g. cluster ID mismatch).
+- **Authentication/Session Failure**: Protocol mismatched or node identity blocked.
+- **Endpoint Validation Failure**: Root connected to Node, but Root's return TCP probe to Node failed. Route will remain `is_routable = False`. Node must allow inbound on its actual port.
+
+---
+
+## 4. The Endpoint Model
+
+In complex mesh deployments (like Tailscale, NATs, or Docker), the network socket alone is insufficient for tracking reliable reverse routes.
+
+ACR utilizes a strict triplet of endpoint values to ensure node health:
+1.  **Observed**: `<incoming-ip>:<ephemeral-port>` - The raw socket origin the Root sees.
+2.  **Advertised**: `<mesh-ip>:<service-port>` - The Node tells the Root to reach it at this specific binding.
+3.  **Validated**: `<mesh-ip>:<service-port>` - The Root actively performs a TCP ping against the Advertised endpoint before approving it as a registered route.
+
+If a Node's advertised endpoint fails validation, the Root **must not** silently assume the route is valid, nor can it permanently trust the ephemeral observed port.
+
+---
+
+## 5. Physical Runtime Validation
+
+Current physical execution validation (Phase 3-9) focuses purely on **macOS Root ↔ macOS Node** over standard TCP.
+
+*Historical Context:* During Phase 6 exploratory testing, ACR used Tailscale purely as the **external networking environment**. Tailscale is NOT an ACR dependency. ACR core knows nothing about Tailscale binaries or hardcoded IPs. End-to-end Tailscale benchmarking across distributed networks is currently DEFERRED pending peer availability.
+
+---
+
+## 6. Windows Troubleshooting
+
+If you are running the Node on Windows and encountering connectivity issues, use this layered diagnostic sequence. Do NOT assume ACR is broken if ICMP fails.
+
+1. **Tailscale Status**: Run `tailscale status` and verify the peer is online.
+2. **ICMP Ping**: Run `ping <peer-ip>`. **Note**: Windows Firewall often blocks inbound ICMP by default. *An ICMP failure does not mean the application is unreachable.*
+3. **Check Node startup log**: ACR prints the dynamically assigned port:
+   ```
+   [ACR NODE]
+     actual_port=<RUNTIME_PORT>
+     advertised_endpoint=<WINDOWS_TAILSCALE_IP>:<RUNTIME_PORT>
+   ```
+4. **TCP Port Check**: Use the actual runtime port from the log:
+   ```powershell
+   Test-NetConnection -ComputerName <ROOT_IP> -Port <ROOT_ACTUAL_PORT>
+   ```
+5. **Listener Check**: Verify ACR bound using:
+   ```
+   netstat -an | findstr <RUNTIME_PORT>
+   ```
+   The listener must appear on `0.0.0.0:<RUNTIME_PORT>` or `<LOCAL_IP>:<RUNTIME_PORT>`. The *advertised* address must be the Tailscale IP, not `0.0.0.0`.
+6. **Windows Firewall (inbound)**: If TCP is blocked but the listener is up, add a narrowly scoped rule for the **actual runtime port**:
+   ```powershell
+   # Replace <RUNTIME_PORT> with the actual port printed in the Node startup log.
+   New-NetFirewallRule -DisplayName "ACR Node <RUNTIME_PORT>" `
+       -Direction Inbound -LocalPort <RUNTIME_PORT> -Protocol TCP -Action Allow
+   ```
+   Remove this rule after the experiment.
+7. **ACR Handshake**: If TCP connects but the handshake fails, check that the `--cluster-id` matches on both sides.
+
+---
+
+## 7. Automated Tests
+
+To ensure the library remains stable across platform adaptations, ensure all unit and integration tests pass:
+```bash
+pytest tests/
+```

vayrm-0.1.0/README.md

@@ -0,0 +1,143 @@
+# Adaptive Compute Runtime (ACR)
+
+ACR is an **OS-independent**, **Network-provider-independent**, and **Scheduler-independent** Python library for intelligent, decentralized, and adaptive workload scheduling across heterogeneous machines.
+
+The core principle of ACR is to completely decouple the adaptive scheduling algorithms from static network constraints.
+
+---
+
+## 1. Architecture
+
+ACR is composed of a generic hierarchy that remains portable across macOS, Windows, and Linux.
+
+*   **Root**: The orchestrator role. The Root aggregates telemetry (NodeKnowledge), maintains the network route registry, and uses Schedulers to dispatch ExecutionPlans.
+*   **Node**: The worker role. The Node discovers its local capabilities, continually monitors its telemetry, and securely connects to the Root to advertise its service endpoint.
+*   **Endpoint Model**: ACR strictly distinguishes between the **Observed** IP/port (what the Root sees natively from the socket connection) and the **Advertised** IP/port (what the Node declares as its reachable listener). ACR always **Validates** the advertised endpoint over TCP before updating the internal Route table.
+*   **Authentication & Handshake**: Nodes register using an application-level handshake payload over a persistent TCP session that transmits the cluster ID, session ID, and endpoint topology.
+*   **Telemetry**: Node-native psutil monitors dynamically measure CPU and RAM degradation over time.
+*   **Scheduler**: Schedulers (RoundRobin, LeastLoaded, ResourceAware, HeterogeneityAware, Adaptive) perform workload allocation using real-time NodeSnapshot data without hardcoded hardware assumptions.
+
+---
+
+## 2. Platform Support & Validation Status
+
+**Root and Node are portable roles, not hardware-locked classes.** ACR guarantees identical algorithmic behavior across operating systems. However, validation evidence is tracked strictly:
+
+*   **macOS (Root / Node):**
+    *   Architecture/Implementation: **PASS**
+    *   Automated Deterministic Testing: **PASS**
+    *   Physical TCP Runtime Execution: **PASS**
+*   **Windows / Linux (Root / Node):**
+    *   Architecture/Implementation: **PASS**
+    *   Automated Deterministic Testing: **PASS**
+    *   Physical TCP Runtime Execution: **NOT RUN / DEFERRED** (pending available physical peers)
+
+---
+
+## 3. Running ACR
+
+ACR listener ports are **dynamically assigned by the OS** by default. You do not need to choose or remember a port number.
+
+### Running as Root
+
+```bash
+python3 -m acr.entrypoint --role root
+```
+
+The Root selects an available port at runtime, binds its listener, tests TCP reachability to itself, and publishes its actual endpoint over the discovery locator.
+
+### Running as Node
+
+The Node dynamically selects its own service port, searches for the Root via discovery, probes it via TCP, and connects automatically:
+
+```bash
+# macOS / Linux:
+python3 -m acr.entrypoint --role node
+
+# Windows:
+python -m acr.entrypoint --role node
+```
+
+### The Automatic Flow
+
+1. Root starts, binds to port `0` (dynamic OS assignment).
+2. Root retrieves its Tailscale identity and discovers its assigned port.
+3. Root tests if it can reach itself via TCP.
+4. Root publishes a discovery record on its locator.
+5. Node starts, asks Tailscale for peers.
+6. Node scans peers for the Root locator.
+7. Node extracts Root's dynamic endpoint and runs a TCP reachability probe.
+8. If TCP passes, Node handshakes with Root and advertises its own dynamic endpoint.
+9. Root validates Node's advertised endpoint via TCP probe.
+10. Route is established using the validated endpoints.
+
+### Failure Diagnostics
+
+The runtime actively distinguishes failure modes:
+- **Discovery Failure**: `[ACR NODE] discovery=FAIL`. Node cannot find Root. Ensure Root is running and Tailscale peers are online.
+- **TCP Failure / Firewall Failure**: `[ACR TCP CHECK] result=FAIL`. Discovery worked but connection failed. Windows or macOS Firewall may be blocking the dynamic port. Read the actual port from the log and allow it.
+- **Handshake Failure**: `[ACR NODE] handshake=FAIL`. TCP connected, but protocol failed (e.g. cluster ID mismatch).
+- **Authentication/Session Failure**: Protocol mismatched or node identity blocked.
+- **Endpoint Validation Failure**: Root connected to Node, but Root's return TCP probe to Node failed. Route will remain `is_routable = False`. Node must allow inbound on its actual port.
+
+---
+
+## 4. The Endpoint Model
+
+In complex mesh deployments (like Tailscale, NATs, or Docker), the network socket alone is insufficient for tracking reliable reverse routes.
+
+ACR utilizes a strict triplet of endpoint values to ensure node health:
+1.  **Observed**: `<incoming-ip>:<ephemeral-port>` - The raw socket origin the Root sees.
+2.  **Advertised**: `<mesh-ip>:<service-port>` - The Node tells the Root to reach it at this specific binding.
+3.  **Validated**: `<mesh-ip>:<service-port>` - The Root actively performs a TCP ping against the Advertised endpoint before approving it as a registered route.
+
+If a Node's advertised endpoint fails validation, the Root **must not** silently assume the route is valid, nor can it permanently trust the ephemeral observed port.
+
+---
+
+## 5. Physical Runtime Validation
+
+Current physical execution validation (Phase 3-9) focuses purely on **macOS Root ↔ macOS Node** over standard TCP.
+
+*Historical Context:* During Phase 6 exploratory testing, ACR used Tailscale purely as the **external networking environment**. Tailscale is NOT an ACR dependency. ACR core knows nothing about Tailscale binaries or hardcoded IPs. End-to-end Tailscale benchmarking across distributed networks is currently DEFERRED pending peer availability.
+
+---
+
+## 6. Windows Troubleshooting
+
+If you are running the Node on Windows and encountering connectivity issues, use this layered diagnostic sequence. Do NOT assume ACR is broken if ICMP fails.
+
+1. **Tailscale Status**: Run `tailscale status` and verify the peer is online.
+2. **ICMP Ping**: Run `ping <peer-ip>`. **Note**: Windows Firewall often blocks inbound ICMP by default. *An ICMP failure does not mean the application is unreachable.*
+3. **Check Node startup log**: ACR prints the dynamically assigned port:
+   ```
+   [ACR NODE]
+     actual_port=<RUNTIME_PORT>
+     advertised_endpoint=<WINDOWS_TAILSCALE_IP>:<RUNTIME_PORT>
+   ```
+4. **TCP Port Check**: Use the actual runtime port from the log:
+   ```powershell
+   Test-NetConnection -ComputerName <ROOT_IP> -Port <ROOT_ACTUAL_PORT>
+   ```
+5. **Listener Check**: Verify ACR bound using:
+   ```
+   netstat -an | findstr <RUNTIME_PORT>
+   ```
+   The listener must appear on `0.0.0.0:<RUNTIME_PORT>` or `<LOCAL_IP>:<RUNTIME_PORT>`. The *advertised* address must be the Tailscale IP, not `0.0.0.0`.
+6. **Windows Firewall (inbound)**: If TCP is blocked but the listener is up, add a narrowly scoped rule for the **actual runtime port**:
+   ```powershell
+   # Replace <RUNTIME_PORT> with the actual port printed in the Node startup log.
+   New-NetFirewallRule -DisplayName "ACR Node <RUNTIME_PORT>" `
+       -Direction Inbound -LocalPort <RUNTIME_PORT> -Protocol TCP -Action Allow
+   ```
+   Remove this rule after the experiment.
+7. **ACR Handshake**: If TCP connects but the handshake fails, check that the `--cluster-id` matches on both sides.
+
+---
+
+## 7. Automated Tests
+
+To ensure the library remains stable across platform adaptations, ensure all unit and integration tests pass:
+```bash
+pytest tests/
+```

vayrm-0.1.0/acr/__init__.py

@@ -0,0 +1,29 @@
+"""
+Adaptive Computing Resource (ACR)
+=================================
+
+ACR is a platform-agnostic, distributed computing runtime for heterogeneous clusters.
+It provides node intelligence, cluster knowledge exchange, adaptive scheduling,
+and closed-loop execution feedback.
+"""
+
+from acr.facade import ACRNode, ACRRoot
+from acr.workloads import Workload
+from acr.schedulers.adaptive import AdaptiveScheduler
+from acr.schedulers.decision import SchedulingDecision
+from acr.execution.manager import ExecutionManager
+from acr.execution.models import ExecutionAttempt
+from acr.communication.protocol import ExecutionObservationMessage as ExecutionObservation
+from acr.resources.snapshot import NodeSnapshot
+
+__all__ = [
+    "ACRNode",
+    "ACRRoot",
+    "Workload",
+    "AdaptiveScheduler",
+    "SchedulingDecision",
+    "ExecutionManager",
+    "ExecutionAttempt",
+    "ExecutionObservation",
+    "NodeSnapshot",
+]

vayrm-0.1.0/acr/api/root_api.py

@@ -0,0 +1,87 @@
+from dataclasses import dataclass
+from typing import Dict, List, Optional
+from acr.node.knowledge import NodeKnowledge
+
+@dataclass
+class WorkerRecord:
+    worker_key: str
+    endpoint: Optional[Dict]
+    paladin_node_id: str
+    session_id: Optional[str]
+    connection: str
+    connection_status: str
+    telemetry: Optional[dict]
+    knowledge: Optional[NodeKnowledge]
+
+class RootDashboardAPI:
+    """
+    API for retrieving WorkerRecords for the dashboard.
+    """
+    def __init__(self, root_brain, control_plane):
+        self._brain = root_brain
+        self._cp = control_plane
+
+    def get_workers(self) -> List[WorkerRecord]:
+        workers = []
+        all_k = self._brain.all_node_knowledge()
+        tracker = getattr(self._cp, "telemetry_tracker", None)
+        
+        # We also want to include nodes that the tracker knows about but have no knowledge yet
+        worker_keys = set(all_k.keys())
+        if tracker:
+            worker_keys.update(rec.node_id for rec in tracker.all_records())
+            
+        for worker_key in worker_keys:
+            # worker_key is usually "{paladin_node_id}_{session_id}"
+            if "_" in worker_key:
+                paladin_node_id, session_id = worker_key.split("_", 1)
+            else:
+                paladin_node_id, session_id = worker_key, None
+                
+            k = all_k.get(worker_key)
+            ep = None
+            conn_status = "UNKNOWN"
+            tel = None
+            
+            if tracker:
+                rec = tracker.get_record(worker_key)
+                if rec:
+                    tel = rec.__dict__
+                    if rec.current_mesh_ip:
+                        ep = {
+                            "mesh_ip": rec.current_mesh_ip,
+                            "service_port": rec.current_service_port
+                        }
+                    conn_status = rec.connection_status
+                    
+            if not ep and hasattr(self._cp, "get_node_endpoint"):
+                # fallback to paladin directly if tracker missed it
+                ep_data = self._cp.get_node_endpoint(paladin_node_id)
+                if ep_data:
+                    ep = {
+                        "mesh_ip": ep_data.get("mesh_ip"),
+                        "service_port": ep_data.get("service_port")
+                    }
+                    if conn_status == "UNKNOWN":
+                        conn_status = ep_data.get("connection_status", "UNKNOWN")
+            
+            workers.append(WorkerRecord(
+                worker_key=worker_key,
+                endpoint=ep,
+                paladin_node_id=paladin_node_id,
+                session_id=session_id,
+                connection="PALADIN",
+                connection_status=conn_status,
+                telemetry=tel,
+                knowledge=k
+            ))
+        
+        # Sort so order is stable
+        workers.sort(key=lambda w: w.worker_key)
+        return workers
+
+    def get_worker(self, worker_key: str) -> Optional[WorkerRecord]:
+        for w in self.get_workers():
+            if w.worker_key == worker_key:
+                return w
+        return None

vayrm-0.1.0/acr/api/server.py

@@ -0,0 +1,70 @@
+"""
+acr/api/server.py
+FastAPI server exposing real-time state of the NodeBrain and WorkloadBrain.
+"""
+from fastapi import FastAPI, HTTPException
+import uuid
+import threading
+
+from acr.runtime.node_brain import NodeBrain
+from acr.runtime.workload_brain import WorkloadBrain
+from acr.runtime.monitoring import RuntimeMonitor
+from acr.workloads.pytorch.adapter import PyTorchWorkloadAdapter
+from acr.workloads.pytorch.checkpoint import PyTorchCheckpointer
+from acr.workloads.pytorch.runtime import run_training
+
+app = FastAPI(title="ACR Live Runtime API", version="1.0.0")
+
+# Global instances for the runtime
+run_id = str(uuid.uuid4())
+node_brain = NodeBrain(node_id="macos_node_1")
+workload_brain = WorkloadBrain(workload_id="pytorch_resnet_1")
+monitor = RuntimeMonitor(node_brain, workload_brain, run_id=run_id, interval_s=1.0)
+adapter = PyTorchWorkloadAdapter(workload_brain)
+checkpointer = PyTorchCheckpointer(f"checkpoints/{run_id}")
+
+training_thread = None
+
+@app.on_event("startup")
+def startup_event():
+    print("Starting Continuous Node Monitor...")
+    monitor.start()
+
+@app.on_event("shutdown")
+def shutdown_event():
+    print("Stopping runtime...")
+    monitor.stop()
+    adapter.stop()
+    if training_thread:
+        training_thread.join(timeout=2.0)
+
+@app.get("/api/v1/node")
+def get_node_state():
+    """Returns the current state and trends of the physical node."""
+    return node_brain.get_state_summary()
+
+@app.get("/api/v1/workload")
+def get_workload_state():
+    """Returns the current state and metrics of the running workload."""
+    return workload_brain.get_state_summary()
+
+@app.post("/api/v1/workload/start")
+def start_workload(device: str = "cpu", batch_size: int = 64):
+    """Start the real PyTorch workload in a background thread."""
+    global training_thread
+    if training_thread and training_thread.is_alive():
+        raise HTTPException(status_code=400, detail="Workload is already running")
+        
+    training_thread = threading.Thread(
+        target=run_training,
+        args=(adapter, checkpointer, device, batch_size),
+        daemon=True
+    )
+    training_thread.start()
+    return {"status": "started", "device": device, "batch_size": batch_size}
+
+@app.post("/api/v1/workload/stop")
+def stop_workload():
+    """Stop the running PyTorch workload."""
+    adapter.running = False
+    return {"status": "stopping"}

vayrm-0.1.0/acr/cache/__init__.py

@@ -0,0 +1,20 @@
+"""
+acr/cache/__init__.py
+
+Local state cache management for ACR nodes.
+"""
+from acr.cache.policy import (
+    CacheEntry, EvictionPolicy, AdmissionPolicy,
+    LRUEvictionPolicy, LFUEvictionPolicy, SizeAwareEvictionPolicy,
+    AlwaysAdmitPolicy, SizeThresholdPolicy,
+)
+from acr.cache.manager import (
+    CacheTier, CacheEventType, CacheEvent, CacheStats, CacheManager,
+)
+
+__all__ = [
+    "CacheEntry", "EvictionPolicy", "AdmissionPolicy",
+    "LRUEvictionPolicy", "LFUEvictionPolicy", "SizeAwareEvictionPolicy",
+    "AlwaysAdmitPolicy", "SizeThresholdPolicy",
+    "CacheTier", "CacheEventType", "CacheEvent", "CacheStats", "CacheManager",
+]