clustermesh 0.9.0__tar.gz

clustermesh-0.9.0/.gitignore

@@ -0,0 +1,19 @@
+*.pyc
+__pycache__/
+.venv/
+*.egg-info/
+dist/
+build/
+.coverage
+htmlcov/
+.pytest_cache/
+.mypy_cache/
+.DS_Store
+node_modules/
+frontend/dist/
+*.local
+.env
+.env.*
+!.env.example
+*.db
+*.tsbuildinfo

clustermesh-0.9.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 ClusterMesh Team
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

clustermesh-0.9.0/PKG-INFO

@@ -0,0 +1,200 @@
+Metadata-Version: 2.4
+Name: clustermesh
+Version: 0.9.0
+Summary: Enterprise compute fabric — pip install on any machine to join a compute cluster
+Project-URL: Homepage, https://github.com/neetishsingh/ClusterMesh
+Project-URL: Documentation, https://github.com/neetishsingh/ClusterMesh/blob/main/docs/join-mesh.md
+Project-URL: Repository, https://github.com/neetishsingh/ClusterMesh
+Author: ClusterMesh Team
+License-Expression: MIT
+License-File: LICENSE
+Keywords: cluster,compute,distributed,grpc,mesh,scheduler
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: System Administrators
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: System :: Clustering
+Classifier: Topic :: System :: Distributed Computing
+Requires-Python: >=3.11
+Requires-Dist: fastapi>=0.110
+Requires-Dist: grpcio>=1.60
+Requires-Dist: protobuf>=4.25
+Requires-Dist: psutil>=5.9
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: uvicorn[standard]>=0.27
+Requires-Dist: websockets>=12.0
+Requires-Dist: zeroconf>=0.131
+Provides-Extra: dev
+Requires-Dist: grpcio-tools>=1.60; extra == 'dev'
+Requires-Dist: httpx>=0.27; extra == 'dev'
+Requires-Dist: hypothesis>=6.100; extra == 'dev'
+Requires-Dist: pytest-cov>=5.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Provides-Extra: discovery
+Requires-Dist: zeroconf>=0.131; extra == 'discovery'
+Provides-Extra: phase6
+Requires-Dist: psycopg[binary]>=3.1; extra == 'phase6'
+Requires-Dist: redis>=5.0; extra == 'phase6'
+Requires-Dist: zeroconf>=0.131; extra == 'phase6'
+Provides-Extra: postgres
+Requires-Dist: psycopg[binary]>=3.1; extra == 'postgres'
+Provides-Extra: redis
+Requires-Dist: redis>=5.0; extra == 'redis'
+Description-Content-Type: text/markdown
+
+# ClusterMesh (ComputeMesh)
+
+**An operating system for enterprise compute** — turn every laptop, desktop, VM, and GPU workstation into a single elastic, fault-tolerant compute cloud.
+
+> Full vision: [Sparkpool](./Sparkpool) · Architecture: [docs/architecture.md](./docs/architecture.md) · Roadmap: [docs/roadmap.md](./docs/roadmap.md)
+
+## The Problem
+
+Organizations sit on thousands of idle cores:
+
+| Resource | Typical utilization |
+|----------|---------------------|
+| CPU | 10–20% |
+| RAM | 30–50% |
+| GPU | 5–10% |
+
+Databricks, Kubernetes, Spark, and Ray all require **dedicated** compute. Nobody fully solves:
+
+> *"Use all idle enterprise hardware automatically and safely."*
+
+ClusterMesh does.
+
+## What We're Building
+
+```
+                    Control Plane
+                           │
+         ┌─────────────────┼─────────────────┐
+         │                 │                 │
+ Metadata Service    Scheduler Service    Auth Service
+         │                 │                 │
+         └─────────────────┼─────────────────┘
+                           │
+                     Driver Cluster (Raft HA)
+                           │
+      ┌────────────────────┼────────────────────┐
+      │                    │                    │
+   Agent-1             Agent-2              Agent-3
+  Laptop               Desktop                 VM
+```
+
+**Killer features:** idle compute harvesting · GPU sharing · live discovery · fault-tolerant scheduling · work stealing · preemption handling · checkpoint recovery · multi-office clustering
+
+## Join a worker (any Python machine)
+
+```bash
+pip install clustermesh
+clustermesh join DRIVER_IP:50050 --open    # local worker UI on :50052
+```
+
+See [docs/join-mesh.md](./docs/join-mesh.md) for full details.
+
+## Quick Start (development)
+
+```bash
+# Install in development mode
+python -m venv .venv
+source .venv/bin/activate
+pip install -e ".[dev]"
+
+# Run tests
+pytest
+
+# Run a simulated 50-node cluster demo
+python -m mesh.sim.demo
+
+# Phase 5: platform with React dashboard (build UI first)
+cd frontend && npm install && npm run build && cd ..
+mesh-platform --port 8080 --db clustermesh.db   # driver + API + UI
+# Phase 6 options
+mesh-platform --port 8080 --mdns --site bangalore          # advertise via mDNS
+mesh-platform --store-url postgres://user:pass@localhost/clustermesh
+mesh-platform --api-key your-secret-key                    # require auth on API
+mesh-agent --discover                                      # auto-find driver on LAN
+
+# Phase 7: multi-site mesh VPN
+mesh-platform --mesh-config config/sites.example.yaml --site bangalore
+mesh-relay --listen 0.0.0.0:6000 --target 127.0.0.1:50050   # standalone relay
+mesh-soak --hours 24 --nodes 50                              # accelerated 24h chaos test
+mesh-bench --nodes 1000                                       # placement SLA benchmark
+./scripts/dogfood.sh                                           # local dogfood run
+```
+
+## Project Structure
+
+```
+ClusterMesh/
+├── docs/                  # Architecture, testing strategy, roadmap
+├── mesh/                  # Core Python package
+│   ├── models/            # Node, Task, Job, Resource types
+│   ├── health/            # Heartbeat FSM, node health tracking
+│   ├── scheduler/         # Scoring, placement, pool routing
+│   ├── execution/         # TaskExecutor, TaskContext
+│   ├── recovery/          # Checkpointing, work stealing, replication
+│   ├── driver/            # JobManager, DriverCluster, gRPC server
+│   ├── agent/             # Daemon, monitor, preemption, library
+│   ├── proto/             # gRPC protobuf definitions
+│   ├── tasks/             # Task registry + built-ins
+│   ├── sdk/               # @task decorator, submit() API
+│   └── sim/               # SimAgent, SimCluster, chaos injection
+├── tests/                 # Unit + integration tests
+├── frontend/              # React dashboard (Vite + Tailwind)
+└── Sparkpool              # Original product vision document
+```
+
+## Current Status (Phase 8) ✅
+
+| Component | Status |
+|-----------|--------|
+| Phases 0–7 (full platform + mesh VPN) | ✅ Done |
+| Distributed memory fabric | ✅ Done |
+| 1000-node placement SLA (`mesh-bench`) | ✅ Done |
+| Memory dashboard + dogfood script | ✅ Done |
+
+## Developer SDK
+
+```python
+from mesh import task, submit, TaskContext
+
+@task(cpu=4, ram="8GB", checkpoint=True, total_work=1_000_000)
+def process_records(ctx: TaskContext):
+    for i in range(int(ctx.progress), 1_000_000):
+        ctx.set_progress(i + 1, records=i + 1)
+    return "done"
+
+# Sync submit — blocks until complete
+result = submit(process_records)
+
+# Async submit — returns JobHandle
+job = submit(process_records, async_=True)
+result = job.wait(timeout=3600)
+```
+
+See [docs/api-spec.md](./docs/api-spec.md) for the full SDK specification.
+
+## Documentation
+
+| Document | Description |
+|----------|-------------|
+| [Architecture](./docs/architecture.md) | System design, components, data flows |
+| [Fault Tolerance](./docs/fault-tolerance.md) | All 10 recovery mechanisms in detail |
+| [Testing Strategy](./docs/testing-strategy.md) | Test pyramid, scenarios, SLAs |
+| [Roadmap](./docs/roadmap.md) | Phased build plan with milestones |
+| [API Spec](./docs/api-spec.md) | Developer SDK and internal APIs |
+| [Join mesh](./docs/join-mesh.md) | `pip install clustermesh` and worker CLI |
+| [Publish to PyPI](./docs/publish-pypi.md) | Build, token setup, and upload guide |
+
+## License
+
+MIT — see [LICENSE](./LICENSE).

clustermesh-0.9.0/README.md

@@ -0,0 +1,150 @@
+# ClusterMesh (ComputeMesh)
+
+**An operating system for enterprise compute** — turn every laptop, desktop, VM, and GPU workstation into a single elastic, fault-tolerant compute cloud.
+
+> Full vision: [Sparkpool](./Sparkpool) · Architecture: [docs/architecture.md](./docs/architecture.md) · Roadmap: [docs/roadmap.md](./docs/roadmap.md)
+
+## The Problem
+
+Organizations sit on thousands of idle cores:
+
+| Resource | Typical utilization |
+|----------|---------------------|
+| CPU | 10–20% |
+| RAM | 30–50% |
+| GPU | 5–10% |
+
+Databricks, Kubernetes, Spark, and Ray all require **dedicated** compute. Nobody fully solves:
+
+> *"Use all idle enterprise hardware automatically and safely."*
+
+ClusterMesh does.
+
+## What We're Building
+
+```
+                    Control Plane
+                           │
+         ┌─────────────────┼─────────────────┐
+         │                 │                 │
+ Metadata Service    Scheduler Service    Auth Service
+         │                 │                 │
+         └─────────────────┼─────────────────┘
+                           │
+                     Driver Cluster (Raft HA)
+                           │
+      ┌────────────────────┼────────────────────┐
+      │                    │                    │
+   Agent-1             Agent-2              Agent-3
+  Laptop               Desktop                 VM
+```
+
+**Killer features:** idle compute harvesting · GPU sharing · live discovery · fault-tolerant scheduling · work stealing · preemption handling · checkpoint recovery · multi-office clustering
+
+## Join a worker (any Python machine)
+
+```bash
+pip install clustermesh
+clustermesh join DRIVER_IP:50050 --open    # local worker UI on :50052
+```
+
+See [docs/join-mesh.md](./docs/join-mesh.md) for full details.
+
+## Quick Start (development)
+
+```bash
+# Install in development mode
+python -m venv .venv
+source .venv/bin/activate
+pip install -e ".[dev]"
+
+# Run tests
+pytest
+
+# Run a simulated 50-node cluster demo
+python -m mesh.sim.demo
+
+# Phase 5: platform with React dashboard (build UI first)
+cd frontend && npm install && npm run build && cd ..
+mesh-platform --port 8080 --db clustermesh.db   # driver + API + UI
+# Phase 6 options
+mesh-platform --port 8080 --mdns --site bangalore          # advertise via mDNS
+mesh-platform --store-url postgres://user:pass@localhost/clustermesh
+mesh-platform --api-key your-secret-key                    # require auth on API
+mesh-agent --discover                                      # auto-find driver on LAN
+
+# Phase 7: multi-site mesh VPN
+mesh-platform --mesh-config config/sites.example.yaml --site bangalore
+mesh-relay --listen 0.0.0.0:6000 --target 127.0.0.1:50050   # standalone relay
+mesh-soak --hours 24 --nodes 50                              # accelerated 24h chaos test
+mesh-bench --nodes 1000                                       # placement SLA benchmark
+./scripts/dogfood.sh                                           # local dogfood run
+```
+
+## Project Structure
+
+```
+ClusterMesh/
+├── docs/                  # Architecture, testing strategy, roadmap
+├── mesh/                  # Core Python package
+│   ├── models/            # Node, Task, Job, Resource types
+│   ├── health/            # Heartbeat FSM, node health tracking
+│   ├── scheduler/         # Scoring, placement, pool routing
+│   ├── execution/         # TaskExecutor, TaskContext
+│   ├── recovery/          # Checkpointing, work stealing, replication
+│   ├── driver/            # JobManager, DriverCluster, gRPC server
+│   ├── agent/             # Daemon, monitor, preemption, library
+│   ├── proto/             # gRPC protobuf definitions
+│   ├── tasks/             # Task registry + built-ins
+│   ├── sdk/               # @task decorator, submit() API
+│   └── sim/               # SimAgent, SimCluster, chaos injection
+├── tests/                 # Unit + integration tests
+├── frontend/              # React dashboard (Vite + Tailwind)
+└── Sparkpool              # Original product vision document
+```
+
+## Current Status (Phase 8) ✅
+
+| Component | Status |
+|-----------|--------|
+| Phases 0–7 (full platform + mesh VPN) | ✅ Done |
+| Distributed memory fabric | ✅ Done |
+| 1000-node placement SLA (`mesh-bench`) | ✅ Done |
+| Memory dashboard + dogfood script | ✅ Done |
+
+## Developer SDK
+
+```python
+from mesh import task, submit, TaskContext
+
+@task(cpu=4, ram="8GB", checkpoint=True, total_work=1_000_000)
+def process_records(ctx: TaskContext):
+    for i in range(int(ctx.progress), 1_000_000):
+        ctx.set_progress(i + 1, records=i + 1)
+    return "done"
+
+# Sync submit — blocks until complete
+result = submit(process_records)
+
+# Async submit — returns JobHandle
+job = submit(process_records, async_=True)
+result = job.wait(timeout=3600)
+```
+
+See [docs/api-spec.md](./docs/api-spec.md) for the full SDK specification.
+
+## Documentation
+
+| Document | Description |
+|----------|-------------|
+| [Architecture](./docs/architecture.md) | System design, components, data flows |
+| [Fault Tolerance](./docs/fault-tolerance.md) | All 10 recovery mechanisms in detail |
+| [Testing Strategy](./docs/testing-strategy.md) | Test pyramid, scenarios, SLAs |
+| [Roadmap](./docs/roadmap.md) | Phased build plan with milestones |
+| [API Spec](./docs/api-spec.md) | Developer SDK and internal APIs |
+| [Join mesh](./docs/join-mesh.md) | `pip install clustermesh` and worker CLI |
+| [Publish to PyPI](./docs/publish-pypi.md) | Build, token setup, and upload guide |
+
+## License
+
+MIT — see [LICENSE](./LICENSE).