macfleet 2.1.1__tar.gz → 2.2.0rc1__tar.gz

{macfleet-2.1.1 → macfleet-2.2.0rc1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: macfleet
-Version: 2.1.1
+Version: 2.2.0rc1
 Summary: Pool Apple Silicon Macs for distributed compute and ML training
 Author: MacFleet Contributors
 License: MIT
@@ -27,6 +27,9 @@ Requires-Dist: click>=8.1.0
 Requires-Dist: numpy>=1.24.0
 Requires-Dist: msgpack>=1.0.0
 Requires-Dist: cloudpickle>=3.0.0
+Requires-Dist: cryptography>=42.0.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: qrcode>=7.0
 Provides-Extra: torch
 Requires-Dist: torch>=2.1.0; extra == "torch"
 Provides-Extra: mlx
@@ -43,12 +46,15 @@ Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
 Requires-Dist: ruff>=0.3.0; extra == "dev"
 Requires-Dist: mypy>=1.8.0; extra == "dev"
 Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: mkdocs-material>=9.5.0; extra == "docs"
+Requires-Dist: mkdocs>=1.5.0; extra == "docs"
 
 # MacFleet
 
-**Pool Apple Silicon Macs into a distributed ML training cluster.**
-
-Turn spare MacBooks, Mac Minis, and Mac Studios into one big GPU. MacFleet connects them over Thunderbolt, Ethernet, or WiFi and splits training across all of them automatically.
+**Distributed ML training on Apple Silicon.** Pool your Macs into a
+cluster in 5 seconds, run PyTorch or MLX across them, keep zero cloud
+spend.
 
 ```
   macfleet join              macfleet join            macfleet join
@@ -63,78 +69,122 @@ Turn spare MacBooks, Mac Minis, and Mac Studios into one big GPU. MacFleet conne
                         Ring AllReduce (gradient sync)
 ```
 
+## Why MacFleet
+
+Apple Silicon is everywhere. Every researcher, student, and founder
+has a serious ML machine on their desk. What's missing is a way to
+team them up.
+
+- **PyTorch on MPS has no distributed story.** NCCL is CUDA-only.
+  Gloo is broken on MPS. Single-GPU-on-MPS only.
+- **MLX is native** but most researchers' code is still PyTorch.
+- **Cloud is expensive** and the iteration loop is slow.
+
+MacFleet fills that gap. Any two Macs on the same WiFi can pool their
+GPUs. Security is baked in (HMAC + TLS). Adaptive compression keeps
+WiFi viable for gradient sync. The framework-agnostic core lets you
+pick your engine (`torch` or `mlx`) per call.
+
 ## Install
 
 ```bash
-pip install macfleet            # core
-pip install macfleet[torch]     # + PyTorch
-pip install macfleet[mlx]       # + Apple MLX
-pip install macfleet[all]       # everything
+pip install macfleet                    # core
+pip install "macfleet[torch]"           # + PyTorch
+pip install "macfleet[mlx]"             # + Apple MLX
+pip install "macfleet[all]"             # everything
 ```
 
-## Quick Start
+## The 5-minute path
 
-**1. Join the pool** (run on each Mac):
+**1. Scaffold a starter script:**
 
 ```bash
-macfleet join
+macfleet quickstart
+# Wrote my_macfleet_demo.py
 ```
 
-No config files, no IP addresses. Macs find each other automatically via mDNS/Bonjour.
+**2. Run it:**
 
-**2. Train:**
+```bash
+python my_macfleet_demo.py
+# Pool world size: 1
+# Training done: {'loss': 0.31, 'epochs': 10, 'time_sec': 1.4}
+```
 
-```python
-import macfleet
-import torch.nn as nn
+**3. Pair a second Mac:**
 
-model = nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10))
+On Mac #1:
 
-with macfleet.Pool() as pool:
-    result = pool.train(model=model, dataset=(X_train, y_train), epochs=10)
+```bash
+macfleet join --bootstrap
+# prints a QR code + pairing URL, also copies URL to pasteboard
 ```
 
+On Mac #2 (same Apple ID → Handoff pasteboard sync):
+
+```bash
+macfleet pair && macfleet join
+```
+
+Or: scan the QR from Mac #1 with your iPhone camera. Tap the link.
+Done.
+
+**4. Set `enable_pool_distributed=True` in your script** — training now
+spans both Macs.
+
 ## Features
 
 - **Dual engine** — PyTorch (MPS) and Apple MLX, same pool infrastructure
 - **Zero config** — mDNS discovery, no coordinator setup, no config files
-- **Adaptive compression** — auto-selects TopK + FP16 based on link speed (1x–200x reduction)
-- **Heterogeneous scheduling** — faster Macs get bigger batches, adjusts for thermal throttling
-- **Secure by default** — auto-generated fleet tokens, HMAC mutual auth, mandatory TLS, gradient validation
-- **Framework-agnostic core** — communication layer uses only numpy, never imports torch or mlx
+- **Safe task dispatch** — `@macfleet.task` registry + msgpack args
+  (no cloudpickle on the wire)
+- **Adaptive compression** — auto-selects TopK + FP16 based on link
+  speed (1x–200x reduction)
+- **Heterogeneous scheduling** — faster Macs get bigger batches,
+  adjusts for thermal throttling
+- **Secure by default** — auto-generated fleet tokens, HMAC mutual
+  auth, mandatory TLS, per-IP rate limiting
+- **Framework-agnostic core** — communication layer uses only numpy,
+  never imports torch or mlx
 
 ## Security
 
-Security is enabled by default. The first `macfleet join` auto-generates a fleet token and saves it to `~/.macfleet/fleet-token`:
+Security is on by default. The first `macfleet join` auto-generates a
+fleet token at `~/.macfleet/token` (mode 0600). See the
+[security reference](docs/reference/security.md) for the full threat
+model.
 
-```bash
-macfleet join                    # auto-generates token, prints it
-macfleet join --token <token>    # join with a specific token (copy from first node)
-macfleet join --fleet-id lab     # isolate by fleet name
-macfleet join --open             # disable security (not recommended)
-```
+Short version:
 
-What's protected:
-- **Fleet isolation** — nodes with different tokens are invisible to each other on the network
-- **Mutual authentication** — HMAC-SHA256 challenge-response on every connection
-- **Encryption** — TLS enabled automatically (mandatory with auth)
-- **Authenticated heartbeat** — HMAC-signed liveness probes, replay-resistant
-- **Gradient validation** — rejects NaN, Inf, and extreme magnitudes (anti-poisoning)
+- **Fleet isolation** — nodes with different tokens can't see each
+  other on the network (mDNS service type is scoped by fleet hash)
+- **Mutual authentication** — HMAC-SHA256 challenge-response on every
+  connection, plus signed hardware profile exchange (v2.2)
+- **Encryption** — TLS mandatory whenever auth is enabled
+- **Rate limiting** — 5 failed auth attempts per IP → 5-minute ban,
+  exponential backoff in between (heartbeat read timeout tightened to
+  1s to stop slowloris)
+- **No cloudpickle over the wire** — `@macfleet.task` routes
+  registered callables by name, not by pickled closures
 
 ## CLI
 
 ```
-macfleet join       Join the pool (auto-discovers peers)
-macfleet status     Show pool members and network info
-macfleet info       Show local hardware profile
-macfleet train      Run training (demo or custom script)
-macfleet bench      Benchmark compute, network, or allreduce
-macfleet diagnose   System health check
+macfleet join         Join the pool (auto-discovers peers)
+macfleet pair         Read a pairing URL from pasteboard / stdin
+macfleet status       Show pool members and network info
+macfleet info         Show local hardware profile
+macfleet train        Run training (demo or custom script)
+macfleet bench        Benchmark compute, network, or allreduce
+macfleet doctor       System health check
+macfleet quickstart   Write a starter training script
 ```
 
-## How It Works
+## How it works
 
-MacFleet uses **data parallelism**: every Mac holds a full copy of the model, trains on a weighted portion of the data, and averages gradients via Ring AllReduce after each step.
+MacFleet uses **data parallelism**: every Mac holds a full copy of the
+model, trains on a weighted portion of the data, and averages
+gradients via Ring AllReduce after each step.
 
 | Network       | Compression     | 100 MB gradients become |
 |---------------|-----------------|-------------------------|
@@ -144,9 +194,22 @@ MacFleet uses **data parallelism**: every Mac holds a full copy of the model, tr
 
 ## Requirements
 
-- macOS with Apple Silicon (M1/M2/M3/M4)
+- macOS 14+ with Apple Silicon (M1/M2/M3/M4)
 - Python 3.11+
-- PyTorch 2.1+ or MLX 0.5+
+- PyTorch 2.1+ or MLX 0.5+ (optional, pick your engine)
+
+## Documentation
+
+Full docs: run `mkdocs serve` after `pip install "macfleet[docs]"`, or
+read the Markdown source in `docs/`:
+
+- [Quickstart](docs/getting-started/quickstart.md)
+- [Pairing flows](docs/getting-started/pairing.md)
+- [Pool.train API](docs/guides/train.md)
+- [@macfleet.task](docs/guides/tasks.md)
+- [Dashboard](docs/guides/dashboard.md)
+- [Security](docs/reference/security.md)
+- [Wire protocol](docs/reference/protocol.md)
 
 ## Development
 
@@ -154,7 +217,7 @@ MacFleet uses **data parallelism**: every Mac holds a full copy of the model, tr
 git clone https://github.com/vikranthreddimasu/MacFleet.git
 cd MacFleet
 pip install -e ".[dev,all]"
-make test       # 373 tests
+make test       # 425+ tests
 make lint       # ruff + mypy
 ```
 

macfleet-2.2.0rc1/README.md

@@ -0,0 +1,174 @@
+# MacFleet
+
+**Distributed ML training on Apple Silicon.** Pool your Macs into a
+cluster in 5 seconds, run PyTorch or MLX across them, keep zero cloud
+spend.
+
+```
+  macfleet join              macfleet join            macfleet join
+ ┌──────────────┐          ┌──────────────┐          ┌──────────────┐
+ │  MacBook Pro │◄────────►│  MacBook Air │◄────────►│  Mac Studio  │
+ │  M4 Pro      │  WiFi /  │  M4          │  WiFi /  │  M4 Ultra    │
+ │  16 GPU cores│  ETH /   │  10 GPU cores│  ETH /   │  60 GPU cores│
+ │  48 GB RAM   │  TB4     │  16 GB RAM   │  TB4     │  192 GB RAM  │
+ └──────────────┘          └──────────────┘          └──────────────┘
+         ▲                          ▲                          ▲
+         └──────────────────────────┴──────────────────────────┘
+                        Ring AllReduce (gradient sync)
+```
+
+## Why MacFleet
+
+Apple Silicon is everywhere. Every researcher, student, and founder
+has a serious ML machine on their desk. What's missing is a way to
+team them up.
+
+- **PyTorch on MPS has no distributed story.** NCCL is CUDA-only.
+  Gloo is broken on MPS. Single-GPU-on-MPS only.
+- **MLX is native** but most researchers' code is still PyTorch.
+- **Cloud is expensive** and the iteration loop is slow.
+
+MacFleet fills that gap. Any two Macs on the same WiFi can pool their
+GPUs. Security is baked in (HMAC + TLS). Adaptive compression keeps
+WiFi viable for gradient sync. The framework-agnostic core lets you
+pick your engine (`torch` or `mlx`) per call.
+
+## Install
+
+```bash
+pip install macfleet                    # core
+pip install "macfleet[torch]"           # + PyTorch
+pip install "macfleet[mlx]"             # + Apple MLX
+pip install "macfleet[all]"             # everything
+```
+
+## The 5-minute path
+
+**1. Scaffold a starter script:**
+
+```bash
+macfleet quickstart
+# Wrote my_macfleet_demo.py
+```
+
+**2. Run it:**
+
+```bash
+python my_macfleet_demo.py
+# Pool world size: 1
+# Training done: {'loss': 0.31, 'epochs': 10, 'time_sec': 1.4}
+```
+
+**3. Pair a second Mac:**
+
+On Mac #1:
+
+```bash
+macfleet join --bootstrap
+# prints a QR code + pairing URL, also copies URL to pasteboard
+```
+
+On Mac #2 (same Apple ID → Handoff pasteboard sync):
+
+```bash
+macfleet pair && macfleet join
+```
+
+Or: scan the QR from Mac #1 with your iPhone camera. Tap the link.
+Done.
+
+**4. Set `enable_pool_distributed=True` in your script** — training now
+spans both Macs.
+
+## Features
+
+- **Dual engine** — PyTorch (MPS) and Apple MLX, same pool infrastructure
+- **Zero config** — mDNS discovery, no coordinator setup, no config files
+- **Safe task dispatch** — `@macfleet.task` registry + msgpack args
+  (no cloudpickle on the wire)
+- **Adaptive compression** — auto-selects TopK + FP16 based on link
+  speed (1x–200x reduction)
+- **Heterogeneous scheduling** — faster Macs get bigger batches,
+  adjusts for thermal throttling
+- **Secure by default** — auto-generated fleet tokens, HMAC mutual
+  auth, mandatory TLS, per-IP rate limiting
+- **Framework-agnostic core** — communication layer uses only numpy,
+  never imports torch or mlx
+
+## Security
+
+Security is on by default. The first `macfleet join` auto-generates a
+fleet token at `~/.macfleet/token` (mode 0600). See the
+[security reference](docs/reference/security.md) for the full threat
+model.
+
+Short version:
+
+- **Fleet isolation** — nodes with different tokens can't see each
+  other on the network (mDNS service type is scoped by fleet hash)
+- **Mutual authentication** — HMAC-SHA256 challenge-response on every
+  connection, plus signed hardware profile exchange (v2.2)
+- **Encryption** — TLS mandatory whenever auth is enabled
+- **Rate limiting** — 5 failed auth attempts per IP → 5-minute ban,
+  exponential backoff in between (heartbeat read timeout tightened to
+  1s to stop slowloris)
+- **No cloudpickle over the wire** — `@macfleet.task` routes
+  registered callables by name, not by pickled closures
+
+## CLI
+
+```
+macfleet join         Join the pool (auto-discovers peers)
+macfleet pair         Read a pairing URL from pasteboard / stdin
+macfleet status       Show pool members and network info
+macfleet info         Show local hardware profile
+macfleet train        Run training (demo or custom script)
+macfleet bench        Benchmark compute, network, or allreduce
+macfleet doctor       System health check
+macfleet quickstart   Write a starter training script
+```
+
+## How it works
+
+MacFleet uses **data parallelism**: every Mac holds a full copy of the
+model, trains on a weighted portion of the data, and averages
+gradients via Ring AllReduce after each step.
+
+| Network       | Compression     | 100 MB gradients become |
+|---------------|-----------------|-------------------------|
+| Thunderbolt 4 | None            | 100 MB                  |
+| Ethernet      | TopK 10% + FP16 | ~5 MB                   |
+| WiFi          | TopK 1% + FP16  | ~500 KB                 |
+
+## Requirements
+
+- macOS 14+ with Apple Silicon (M1/M2/M3/M4)
+- Python 3.11+
+- PyTorch 2.1+ or MLX 0.5+ (optional, pick your engine)
+
+## Documentation
+
+Full docs: run `mkdocs serve` after `pip install "macfleet[docs]"`, or
+read the Markdown source in `docs/`:
+
+- [Quickstart](docs/getting-started/quickstart.md)
+- [Pairing flows](docs/getting-started/pairing.md)
+- [Pool.train API](docs/guides/train.md)
+- [@macfleet.task](docs/guides/tasks.md)
+- [Dashboard](docs/guides/dashboard.md)
+- [Security](docs/reference/security.md)
+- [Wire protocol](docs/reference/protocol.md)
+
+## Development
+
+```bash
+git clone https://github.com/vikranthreddimasu/MacFleet.git
+cd MacFleet
+pip install -e ".[dev,all]"
+make test       # 425+ tests
+make lint       # ruff + mypy
+```
+
+## License
+
+MIT

{macfleet-2.1.1 → macfleet-2.2.0rc1}/macfleet/__init__.py

@@ -7,7 +7,7 @@ Zero-config discovery. Framework-agnostic engines. Adaptive networking.
 
 import logging
 
-__version__ = "2.1.1"
+__version__ = "2.2.0rc1"
 
 logging.getLogger(__name__).addHandler(logging.NullHandler())
 
@@ -38,6 +38,10 @@ def __getattr__(name: str):
     if name == "RemoteTaskError":
         from macfleet.compute.models import RemoteTaskError
         return RemoteTaskError
+    # v2.2 PR 7: @macfleet.task decorator
+    if name == "task":
+        from macfleet.compute.registry import task
+        return task
     raise AttributeError(f"module 'macfleet' has no attribute {name!r}")
 
 
@@ -51,4 +55,5 @@ __all__ = [
     "MLXEngine",
     "TaskFuture",
     "RemoteTaskError",
+    "task",
 ]