macfleet 2.0.0__tar.gz → 2.1.1__tar.gz

macfleet-2.1.1/PKG-INFO

@@ -0,0 +1,163 @@
+Metadata-Version: 2.2
+Name: macfleet
+Version: 2.1.1
+Summary: Pool Apple Silicon Macs for distributed compute and ML training
+Author: MacFleet Contributors
+License: MIT
+Project-URL: Homepage, https://github.com/vikranthreddimasu/MacFleet
+Project-URL: Documentation, https://github.com/vikranthreddimasu/MacFleet#readme
+Project-URL: Repository, https://github.com/vikranthreddimasu/MacFleet
+Project-URL: Issues, https://github.com/vikranthreddimasu/MacFleet/issues
+Keywords: distributed,machine-learning,apple-silicon,mps,mlx,pytorch,training,gpu-pooling,data-parallel
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: MacOS
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: zeroconf>=0.131.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: click>=8.1.0
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: msgpack>=1.0.0
+Requires-Dist: cloudpickle>=3.0.0
+Provides-Extra: torch
+Requires-Dist: torch>=2.1.0; extra == "torch"
+Provides-Extra: mlx
+Requires-Dist: mlx>=0.5.0; extra == "mlx"
+Provides-Extra: yaml
+Requires-Dist: pyyaml>=6.0; extra == "yaml"
+Provides-Extra: all
+Requires-Dist: torch>=2.1.0; extra == "all"
+Requires-Dist: mlx>=0.5.0; extra == "all"
+Requires-Dist: pyyaml>=6.0; extra == "all"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
+Requires-Dist: ruff>=0.3.0; extra == "dev"
+Requires-Dist: mypy>=1.8.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
+
+# MacFleet
+
+**Pool Apple Silicon Macs into a distributed ML training cluster.**
+
+Turn spare MacBooks, Mac Minis, and Mac Studios into one big GPU. MacFleet connects them over Thunderbolt, Ethernet, or WiFi and splits training across all of them automatically.
+
+```
+  macfleet join              macfleet join            macfleet join
+ ┌──────────────┐          ┌──────────────┐          ┌──────────────┐
+ │  MacBook Pro │◄────────►│  MacBook Air │◄────────►│  Mac Studio  │
+ │  M4 Pro      │  WiFi /  │  M4          │  WiFi /  │  M4 Ultra    │
+ │  16 GPU cores│  ETH /   │  10 GPU cores│  ETH /   │  60 GPU cores│
+ │  48 GB RAM   │  TB4     │  16 GB RAM   │  TB4     │  192 GB RAM  │
+ └──────────────┘          └──────────────┘          └──────────────┘
+         ▲                          ▲                          ▲
+         └──────────────────────────┴──────────────────────────┘
+                        Ring AllReduce (gradient sync)
+```
+
+## Install
+
+```bash
+pip install macfleet            # core
+pip install macfleet[torch]     # + PyTorch
+pip install macfleet[mlx]       # + Apple MLX
+pip install macfleet[all]       # everything
+```
+
+## Quick Start
+
+**1. Join the pool** (run on each Mac):
+
+```bash
+macfleet join
+```
+
+No config files, no IP addresses. Macs find each other automatically via mDNS/Bonjour.
+
+**2. Train:**
+
+```python
+import macfleet
+import torch.nn as nn
+
+model = nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10))
+
+with macfleet.Pool() as pool:
+    result = pool.train(model=model, dataset=(X_train, y_train), epochs=10)
+```
+
+## Features
+
+- **Dual engine** — PyTorch (MPS) and Apple MLX, same pool infrastructure
+- **Zero config** — mDNS discovery, no coordinator setup, no config files
+- **Adaptive compression** — auto-selects TopK + FP16 based on link speed (1x–200x reduction)
+- **Heterogeneous scheduling** — faster Macs get bigger batches, adjusts for thermal throttling
+- **Secure by default** — auto-generated fleet tokens, HMAC mutual auth, mandatory TLS, gradient validation
+- **Framework-agnostic core** — communication layer uses only numpy, never imports torch or mlx
+
+## Security
+
+Security is enabled by default. The first `macfleet join` auto-generates a fleet token and saves it to `~/.macfleet/fleet-token`:
+
+```bash
+macfleet join                    # auto-generates token, prints it
+macfleet join --token <token>    # join with a specific token (copy from first node)
+macfleet join --fleet-id lab     # isolate by fleet name
+macfleet join --open             # disable security (not recommended)
+```
+
+What's protected:
+- **Fleet isolation** — nodes with different tokens are invisible to each other on the network
+- **Mutual authentication** — HMAC-SHA256 challenge-response on every connection
+- **Encryption** — TLS enabled automatically (mandatory with auth)
+- **Authenticated heartbeat** — HMAC-signed liveness probes, replay-resistant
+- **Gradient validation** — rejects NaN, Inf, and extreme magnitudes (anti-poisoning)
+
+## CLI
+
+```
+macfleet join       Join the pool (auto-discovers peers)
+macfleet status     Show pool members and network info
+macfleet info       Show local hardware profile
+macfleet train      Run training (demo or custom script)
+macfleet bench      Benchmark compute, network, or allreduce
+macfleet diagnose   System health check
+```
+
+## How It Works
+
+MacFleet uses **data parallelism**: every Mac holds a full copy of the model, trains on a weighted portion of the data, and averages gradients via Ring AllReduce after each step.
+
+| Network       | Compression     | 100 MB gradients become |
+|---------------|-----------------|-------------------------|
+| Thunderbolt 4 | None            | 100 MB                  |
+| Ethernet      | TopK 10% + FP16 | ~5 MB                   |
+| WiFi          | TopK 1% + FP16  | ~500 KB                 |
+
+## Requirements
+
+- macOS with Apple Silicon (M1/M2/M3/M4)
+- Python 3.11+
+- PyTorch 2.1+ or MLX 0.5+
+
+## Development
+
+```bash
+git clone https://github.com/vikranthreddimasu/MacFleet.git
+cd MacFleet
+pip install -e ".[dev,all]"
+make test       # 373 tests
+make lint       # ruff + mypy
+```
+
+## License
+
+MIT

macfleet-2.1.1/README.md

@@ -0,0 +1,117 @@
+# MacFleet
+
+**Pool Apple Silicon Macs into a distributed ML training cluster.**
+
+Turn spare MacBooks, Mac Minis, and Mac Studios into one big GPU. MacFleet connects them over Thunderbolt, Ethernet, or WiFi and splits training across all of them automatically.
+
+```
+  macfleet join              macfleet join            macfleet join
+ ┌──────────────┐          ┌──────────────┐          ┌──────────────┐
+ │  MacBook Pro │◄────────►│  MacBook Air │◄────────►│  Mac Studio  │
+ │  M4 Pro      │  WiFi /  │  M4          │  WiFi /  │  M4 Ultra    │
+ │  16 GPU cores│  ETH /   │  10 GPU cores│  ETH /   │  60 GPU cores│
+ │  48 GB RAM   │  TB4     │  16 GB RAM   │  TB4     │  192 GB RAM  │
+ └──────────────┘          └──────────────┘          └──────────────┘
+         ▲                          ▲                          ▲
+         └──────────────────────────┴──────────────────────────┘
+                        Ring AllReduce (gradient sync)
+```
+
+## Install
+
+```bash
+pip install macfleet            # core
+pip install macfleet[torch]     # + PyTorch
+pip install macfleet[mlx]       # + Apple MLX
+pip install macfleet[all]       # everything
+```
+
+## Quick Start
+
+**1. Join the pool** (run on each Mac):
+
+```bash
+macfleet join
+```
+
+No config files, no IP addresses. Macs find each other automatically via mDNS/Bonjour.
+
+**2. Train:**
+
+```python
+import macfleet
+import torch.nn as nn
+
+model = nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10))
+
+with macfleet.Pool() as pool:
+    result = pool.train(model=model, dataset=(X_train, y_train), epochs=10)
+```
+
+## Features
+
+- **Dual engine** — PyTorch (MPS) and Apple MLX, same pool infrastructure
+- **Zero config** — mDNS discovery, no coordinator setup, no config files
+- **Adaptive compression** — auto-selects TopK + FP16 based on link speed (1x–200x reduction)
+- **Heterogeneous scheduling** — faster Macs get bigger batches, adjusts for thermal throttling
+- **Secure by default** — auto-generated fleet tokens, HMAC mutual auth, mandatory TLS, gradient validation
+- **Framework-agnostic core** — communication layer uses only numpy, never imports torch or mlx
+
+## Security
+
+Security is enabled by default. The first `macfleet join` auto-generates a fleet token and saves it to `~/.macfleet/fleet-token`:
+
+```bash
+macfleet join                    # auto-generates token, prints it
+macfleet join --token <token>    # join with a specific token (copy from first node)
+macfleet join --fleet-id lab     # isolate by fleet name
+macfleet join --open             # disable security (not recommended)
+```
+
+What's protected:
+- **Fleet isolation** — nodes with different tokens are invisible to each other on the network
+- **Mutual authentication** — HMAC-SHA256 challenge-response on every connection
+- **Encryption** — TLS enabled automatically (mandatory with auth)
+- **Authenticated heartbeat** — HMAC-signed liveness probes, replay-resistant
+- **Gradient validation** — rejects NaN, Inf, and extreme magnitudes (anti-poisoning)
+
+## CLI
+
+```
+macfleet join       Join the pool (auto-discovers peers)
+macfleet status     Show pool members and network info
+macfleet info       Show local hardware profile
+macfleet train      Run training (demo or custom script)
+macfleet bench      Benchmark compute, network, or allreduce
+macfleet diagnose   System health check
+```
+
+## How It Works
+
+MacFleet uses **data parallelism**: every Mac holds a full copy of the model, trains on a weighted portion of the data, and averages gradients via Ring AllReduce after each step.
+
+| Network       | Compression     | 100 MB gradients become |
+|---------------|-----------------|-------------------------|
+| Thunderbolt 4 | None            | 100 MB                  |
+| Ethernet      | TopK 10% + FP16 | ~5 MB                   |
+| WiFi          | TopK 1% + FP16  | ~500 KB                 |
+
+## Requirements
+
+- macOS with Apple Silicon (M1/M2/M3/M4)
+- Python 3.11+
+- PyTorch 2.1+ or MLX 0.5+
+
+## Development
+
+```bash
+git clone https://github.com/vikranthreddimasu/MacFleet.git
+cd MacFleet
+pip install -e ".[dev,all]"
+make test       # 373 tests
+make lint       # ruff + mypy
+```
+
+## License
+
+MIT

{macfleet-2.0.0 → macfleet-2.1.1}/macfleet/__init__.py

@@ -7,7 +7,7 @@ Zero-config discovery. Framework-agnostic engines. Adaptive networking.
 
 import logging
 
-__version__ = "2.0.0"
+__version__ = "2.1.1"
 
 logging.getLogger(__name__).addHandler(logging.NullHandler())
 
@@ -32,6 +32,12 @@ def __getattr__(name: str):
     if name == "MLXEngine":
         from macfleet.engines.mlx_engine import MLXEngine
         return MLXEngine
+    if name == "TaskFuture":
+        from macfleet.compute.models import TaskFuture
+        return TaskFuture
+    if name == "RemoteTaskError":
+        from macfleet.compute.models import RemoteTaskError
+        return RemoteTaskError
     raise AttributeError(f"module 'macfleet' has no attribute {name!r}")
 
 
@@ -43,4 +49,6 @@ __all__ = [
     "DataParallel",
     "TorchEngine",
     "MLXEngine",
+    "TaskFuture",
+    "RemoteTaskError",
 ]

{macfleet-2.0.0 → macfleet-2.1.1}/macfleet/cli/main.py

@@ -37,12 +37,42 @@ def cli():
 @cli.command()
 @click.option("--name", default=None, help="Custom node name")
 @click.option("--port", default=50051, help="Communication port")
-@click.option("--token", default=None, help="Pool authentication token")
-def join(name: str | None, port: int, token: str | None):
-    """Join the compute pool. Auto-discovers peers on the network."""
+@click.option("--token", default=None, envvar="MACFLEET_TOKEN", help="Pool token (or set MACFLEET_TOKEN env var)")
+@click.option("--fleet-id", default=None, help="Fleet identifier (isolates pool on network)")
+@click.option("--tls", "use_tls", is_flag=True, default=False, help="Enable TLS encryption")
+@click.option("--open", "open_fleet", is_flag=True, default=False, help="Disable security (open fleet, no authentication)")
+@click.option("--peer", "peers", multiple=True, help="Peer address (IP:PORT). Use when mDNS is blocked. Repeatable.")
+def join(name: str | None, port: int, token: str | None, fleet_id: str | None, use_tls: bool, open_fleet: bool, peers: tuple):
+    """Join the compute pool. Auto-discovers peers on the network.
+
+    Security is enabled by default. A fleet token is auto-generated on first
+    run and saved to ~/.macfleet/fleet-token. Copy this token to other Macs
+    to let them join your fleet.
+
+    Use --open to disable security (not recommended).
+
+    \b
+    If mDNS discovery doesn't work (e.g. enterprise WiFi), use --peer:
+        Mac A: macfleet join
+        Mac B: macfleet join --token <token> --peer <Mac-A-IP>:50051
+    """
     from macfleet.pool.agent import PoolAgent
+    from macfleet.security.auth import resolve_token_with_file, TOKEN_FILE
+
+    if open_fleet:
+        if token:
+            console.print("[red]Error: --open and --token are mutually exclusive.[/red]")
+            sys.exit(1)
+        resolved_token = None
+    else:
+        resolved_token = resolve_token_with_file(token, auto_generate=True)
+        if token is None:
+            # Token was auto-generated or loaded from file — show it
+            console.print(f"\n[bold green]Fleet token:[/bold green] {resolved_token}")
+            console.print(f"[dim]Saved to {TOKEN_FILE}[/dim]")
+            console.print("[dim]Copy this token to other Macs: macfleet join --token <token>[/dim]\n")
 
-    agent = PoolAgent(name=name, port=port, token=token)
+    agent = PoolAgent(name=name, port=port, token=resolved_token, fleet_id=fleet_id, tls=use_tls, peers=list(peers))
 
     async def run():
         await agent.start()
@@ -98,13 +128,27 @@ def info():
 
 
 @cli.command()
-def status():
+@click.option("--token", default=None, envvar="MACFLEET_TOKEN", help="Pool token (scopes discovery to fleet)")
+@click.option("--fleet-id", default=None, help="Fleet identifier")
+@click.option("--open", "open_fleet", is_flag=True, default=False, help="Scan open fleet (ignore saved token)")
+def status(token: str | None, fleet_id: str | None, open_fleet: bool):
     """Show pool status (discovers peers for 3 seconds)."""
     from macfleet.pool.discovery import ServiceRegistry
+    from macfleet.security.auth import SecurityConfig, resolve_token_with_file
+
+    if open_fleet:
+        resolved = None
+    else:
+        resolved = resolve_token_with_file(token)
 
-    console.print("[bold]Scanning for pool members...[/bold]")
+    sec = SecurityConfig(token=resolved, fleet_id=fleet_id) if resolved else None
+    if sec and sec.is_secure:
+        fleet_label = fleet_id or "default"
+        console.print(f"[bold]Scanning fleet '{fleet_label}' for members...[/bold]")
+    else:
+        console.print("[bold]Scanning for pool members...[/bold]")
 
-    registry = ServiceRegistry()
+    registry = ServiceRegistry(security=sec)
     try:
         peers = registry.find_peers(timeout=3.0)
     finally:
@@ -326,6 +370,54 @@ def _train_from_script(
         sys.exit(1)
 
 
+@cli.command(name="run")
+@click.argument("script")
+@click.option("--fn", "fn_name", default="main", help="Function to execute (default: main)")
+@click.option("--token", default=None, envvar="MACFLEET_TOKEN", help="Pool token")
+@click.option("--open", "open_fleet", is_flag=True, default=False, help="Disable security")
+def run_command(script: str, fn_name: str, token: str | None, open_fleet: bool):
+    """Run a Python script on the pool.
+
+    The script must define the named function (default: main).
+    The function is executed across the pool's compute resources.
+
+    \b
+    Examples:
+        macfleet run process.py
+        macfleet run analysis.py --fn analyze
+    """
+    import importlib.util
+    import os
+
+    if not os.path.isfile(script):
+        console.print(f"[red]Error: Script not found: {script}[/red]")
+        sys.exit(1)
+
+    # Load user script
+    spec = importlib.util.spec_from_file_location("user_script", script)
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+
+    fn = getattr(module, fn_name, None)
+    if fn is None or not callable(fn):
+        console.print(f"[red]Error: Function '{fn_name}' not found in {script}[/red]")
+        console.print(f"[dim]The script must define a callable named '{fn_name}'.[/dim]")
+        sys.exit(1)
+
+    console.print(f"[bold blue]MacFleet Run[/bold blue] — {script}:{fn_name}()")
+
+    from macfleet.sdk.pool import Pool
+
+    with Pool(token=token, open=open_fleet) as pool:
+        t0 = time.time()
+        result = pool.run(fn)
+        elapsed = time.time() - t0
+
+    console.print(f"\n[green]Completed in {elapsed:.2f}s[/green]")
+    if result is not None:
+        console.print(f"Result: {result}")
+
+
 @cli.command()
 @click.option("--type", "bench_type", type=click.Choice(["network", "compute", "allreduce"]), default="network")
 @click.option("--size-mb", default=10, help="Payload size in MB for network tests")

{macfleet-2.0.0 → macfleet-2.1.1}/macfleet/comm/collectives.py

@@ -174,7 +174,7 @@ class CollectiveGroup:
         # Flatten for chunking
         original_shape = array.shape
         original_dtype = array.dtype
-        flat = array.astype(np.float64).flatten()  # promote for accumulation
+        flat = array.flatten()
         numel = len(flat)
 
         # Pad to be evenly divisible

macfleet-2.1.1/macfleet/comm/protocol.py

@@ -0,0 +1,135 @@
+"""Binary wire protocol for tensor transport.
+
+Extended from MacFleet v1's 16-byte header to 24 bytes with:
+- Stream multiplexing (stream_id)
+- CRC32 checksums (critical for WiFi reliability)
+- Chunking flags for large tensors
+- Sequence numbers for ordering
+
+Header (24 bytes):
+  stream_id:    uint32  (multiplexing: control=0, tensor=1..N)
+  msg_type:     uint16  (CONTROL=1, TENSOR=2, HEARTBEAT=3, GRADIENT=4, COMPRESSED=5)
+  flags:        uint16  (bit 0: compressed, bit 1: chunked, bit 2: last_chunk)
+  payload_size: uint32  (bytes)
+  sequence:     uint32  (ordering within stream)
+  checksum:     uint32  (CRC32 of payload)
+  reserved:     uint32  (future use)
+"""
+
+import struct
+import zlib
+from dataclasses import dataclass
+from enum import IntEnum, IntFlag
+
+
+class MessageType(IntEnum):
+    """Message types for the wire protocol."""
+    CONTROL = 0x01
+    TENSOR = 0x02
+    HEARTBEAT = 0x03
+    GRADIENT = 0x04
+    COMPRESSED_GRADIENT = 0x05
+    BARRIER = 0x06
+    STATE = 0x07
+    TASK = 0x08
+    RESULT = 0x09
+
+
+class MessageFlags(IntFlag):
+    """Bit flags for message metadata."""
+    NONE = 0x00
+    COMPRESSED = 0x01
+    CHUNKED = 0x02
+    LAST_CHUNK = 0x04
+
+
+# 24-byte header: stream_id(I) msg_type(H) flags(H) payload_size(I) sequence(I) checksum(I) reserved(I)
+HEADER_FORMAT = "!IHHIIII"
+HEADER_SIZE = struct.calcsize(HEADER_FORMAT)  # 24 bytes
+
+# SECURITY: Maximum payload size to prevent OOM from malicious headers.
+# 256 MB is larger than any realistic gradient tensor (100M float32 = 400 MB,
+# but compressed gradients are much smaller). Set conservatively high.
+MAX_PAYLOAD_SIZE = 256 * 1024 * 1024  # 256 MB
+
+
+@dataclass
+class WireMessage:
+    """A message on the wire."""
+    stream_id: int
+    msg_type: MessageType
+    flags: MessageFlags
+    sequence: int
+    payload: bytes
+    checksum: int = 0
+
+    def pack(self) -> bytes:
+        """Serialize to bytes (header + payload)."""
+        checksum = zlib.crc32(self.payload) & 0xFFFFFFFF
+        header = struct.pack(
+            HEADER_FORMAT,
+            self.stream_id,
+            self.msg_type,
+            self.flags,
+            len(self.payload),
+            self.sequence,
+            checksum,
+            0,  # reserved
+        )
+        return header + self.payload
+
+    @classmethod
+    def unpack(cls, data: bytes) -> "WireMessage":
+        """Deserialize from bytes."""
+        header = data[:HEADER_SIZE]
+        stream_id, msg_type, flags, payload_size, sequence, checksum, _ = struct.unpack(
+            HEADER_FORMAT, header
+        )
+        payload = data[HEADER_SIZE : HEADER_SIZE + payload_size]
+
+        # Verify checksum
+        actual_checksum = zlib.crc32(payload) & 0xFFFFFFFF
+        if actual_checksum != checksum:
+            raise ValueError(
+                f"CRC32 mismatch: expected {checksum:#x}, got {actual_checksum:#x}"
+            )
+
+        return cls(
+            stream_id=stream_id,
+            msg_type=MessageType(msg_type),
+            flags=MessageFlags(flags),
+            sequence=sequence,
+            payload=payload,
+            checksum=checksum,
+        )
+
+    @classmethod
+    async def read_from_stream(cls, reader) -> "WireMessage":
+        """Read a single message from an asyncio StreamReader."""
+        header_data = await reader.readexactly(HEADER_SIZE)
+        stream_id, msg_type, flags, payload_size, sequence, checksum, _ = struct.unpack(
+            HEADER_FORMAT, header_data
+        )
+        if payload_size > MAX_PAYLOAD_SIZE:
+            raise ValueError(
+                f"Payload size {payload_size} exceeds maximum {MAX_PAYLOAD_SIZE} "
+                f"— possible OOM attack or corrupt header"
+            )
+        payload = await reader.readexactly(payload_size)
+
+        actual_checksum = zlib.crc32(payload) & 0xFFFFFFFF
+        if actual_checksum != checksum:
+            raise ValueError(
+                f"CRC32 mismatch: expected {checksum:#x}, got {actual_checksum:#x}"
+            )
+
+        return cls(
+            stream_id=stream_id,
+            msg_type=MessageType(msg_type),
+            flags=MessageFlags(flags),
+            sequence=sequence,
+            payload=payload,
+            checksum=checksum,
+        )
+
+