distributed-llm 0.4.1__tar.gz

distributed_llm-0.4.1/LICENSE

@@ -0,0 +1,97 @@
+Apache License
+Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+"License" shall mean the terms and conditions for use, reproduction,
+and distribution as defined by Sections 1 through 9 of this document.
+
+"Licensor" shall mean the copyright owner or entity authorized by
+the copyright owner that is granting the License.
+
+"Legal Entity" shall mean the union of the acting entity and all
+other entities that control, are controlled by, or are under common
+control with that entity.
+
+"You" (or "Your") shall mean an individual or Legal Entity
+exercising permissions granted by this License.
+
+"Source" form shall mean the preferred form for making modifications,
+including but not limited to software source code, documentation
+source, and configuration files.
+
+"Object" form shall mean any form resulting from mechanical
+transformation or translation of a Source form, including but
+not limited to compiled object code, generated documentation,
+and conversions to other media types.
+
+"Work" shall mean the work of authorship, whether in Source or
+Object form, made available under the License.
+
+"Derivative Works" shall mean any work, whether in Source or Object
+form, that is based on (or derived from) the Work.
+
+"Contribution" shall mean any work of authorship, including
+the original version of the Work and any modifications or additions
+to that Work or Derivative Works thereof, that is intentionally
+submitted to the Licensor.
+
+"Contributor" shall mean Licensor and any individual or Legal Entity
+on behalf of whom a Contribution has been received by Licensor.
+
+2. Grant of Copyright License. Subject to the terms and conditions of
+this License, each Contributor hereby grants to You a perpetual,
+worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+copyright license to reproduce, prepare Derivative Works of,
+publicly display, publicly perform, sublicense, and distribute the
+Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of
+this License, each Contributor hereby grants to You a perpetual,
+worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+(except as stated in this section) patent license to make, have made,
+use, offer to sell, sell, import, and otherwise transfer the Work.
+
+4. Redistribution. You may reproduce and distribute copies of the
+Work or Derivative Works thereof in any medium, with or without
+modifications, provided that You meet the following conditions:
+
+(a) You must give any other recipients of the Work or
+Derivative Works a copy of this License; and
+
+(b) You must cause any modified files to carry prominent notices
+stating that You changed the files; and
+
+(c) You must retain, in the Source form of any Derivative Works
+that You distribute, all copyright, patent, trademark, and
+attribution notices from the Source form of the Work; and
+
+(d) You may add Your own copyright statement to Your modifications and
+may provide additional or different license terms and conditions
+for use, reproduction, or distribution of Your modifications.
+
+5. Submission of Contributions. Unless You explicitly state otherwise,
+any Contribution intentionally submitted for inclusion in the Work
+by You to the Licensor shall be under the terms and conditions of
+this License.
+
+6. Trademarks. This License does not grant permission to use the trade
+names, trademarks, service marks, or product names of the Licensor.
+
+7. Disclaimer of Warranty. Unless required by applicable law or
+agreed to in writing, Licensor provides the Work on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND.
+
+8. Limitation of Liability. In no event shall any Contributor be liable
+to You for damages arising from use of the Work or inability to use
+the Work.
+
+9. Accepting Warranty or Additional Liability. While redistributing
+the Work or Derivative Works thereof, You may offer to accept support,
+warranty, indemnity, or other liability obligations consistent with
+this License.
+
+END OF TERMS AND CONDITIONS

distributed_llm-0.4.1/PKG-INFO

@@ -0,0 +1,315 @@
+Metadata-Version: 2.4
+Name: distributed-llm
+Version: 0.4.1
+Summary: Distributed LLM inference — pool GPUs across multiple devices to run models no single machine can handle
+Author: Distributed LLM Contributors
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/distributed-llm/distributed-llm
+Project-URL: Documentation, https://github.com/distributed-llm/distributed-llm#readme
+Project-URL: Repository, https://github.com/distributed-llm/distributed-llm
+Project-URL: Issues, https://github.com/distributed-llm/distributed-llm/issues
+Keywords: llm,inference,distributed,pipeline-parallelism,gpu-sharing,peer-to-peer,open-source
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: System :: Distributed Computing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: fastapi>=0.109.0
+Requires-Dist: uvicorn>=0.27.0
+Requires-Dist: pydantic>=2.5.0
+Requires-Dist: pydantic-settings>=2.0.0
+Requires-Dist: httpx>=0.27
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: loguru>=0.7.0
+Requires-Dist: pyjwt>=2.0.0
+Requires-Dist: numpy>=1.26.0
+Requires-Dist: psutil>=5.9.0
+Provides-Extra: observability
+Requires-Dist: prometheus-client>=0.20; extra == "observability"
+Requires-Dist: opentelemetry-api>=1.21.0; extra == "observability"
+Requires-Dist: opentelemetry-sdk>=1.21.0; extra == "observability"
+Requires-Dist: opentelemetry-exporter-prometheus>=0.42b0; extra == "observability"
+Provides-Extra: self-hosted
+Requires-Dist: torch>=2.5.0; extra == "self-hosted"
+Requires-Dist: transformers>=4.48.0; extra == "self-hosted"
+Requires-Dist: accelerate>=1.0.0; extra == "self-hosted"
+Requires-Dist: safetensors>=0.4.0; extra == "self-hosted"
+Requires-Dist: huggingface-hub>=0.20.0; extra == "self-hosted"
+Requires-Dist: sentencepiece>=0.1.99; extra == "self-hosted"
+Requires-Dist: nvidia-ml-py>=12.0; extra == "self-hosted"
+Provides-Extra: ray
+Requires-Dist: ray[default]>=2.30.0; extra == "ray"
+Provides-Extra: backends
+Requires-Dist: vllm<0.9.0,>=0.7.0; extra == "backends"
+Requires-Dist: sglang<0.5.0,>=0.4.0; extra == "backends"
+Requires-Dist: llama-cpp-python>=0.3.0; extra == "backends"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
+Requires-Dist: hypothesis>=6.90; extra == "dev"
+Requires-Dist: pytest-benchmark>=4.0; extra == "dev"
+Requires-Dist: black>=24.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Requires-Dist: mypy>=1.8; extra == "dev"
+Requires-Dist: opentelemetry-test-utils>=0.42b0; extra == "dev"
+Requires-Dist: bandit[toml]>=1.7; extra == "dev"
+Requires-Dist: detect-secrets>=1.5; extra == "dev"
+Provides-Extra: sdk
+Requires-Dist: httpx>=0.27.0; extra == "sdk"
+Provides-Extra: cli
+Requires-Dist: typer[all]>=0.9.0; extra == "cli"
+Requires-Dist: rich>=13.7.0; extra == "cli"
+Requires-Dist: prompt_toolkit>=3.0.0; extra == "cli"
+Provides-Extra: ui
+Requires-Dist: jinja2>=3.1.0; extra == "ui"
+Requires-Dist: python-multipart>=0.0.6; extra == "ui"
+Requires-Dist: qrcode[pil]>=7.4; extra == "ui"
+Provides-Extra: examples
+Requires-Dist: langchain-openai>=0.1.0; extra == "examples"
+Requires-Dist: langchain-core>=0.1.0; extra == "examples"
+Requires-Dist: llama-index-llms-openai>=0.1.0; extra == "examples"
+Requires-Dist: llama-index-core>=0.10.0; extra == "examples"
+Requires-Dist: crewai>=0.30.0; extra == "examples"
+Provides-Extra: security
+Requires-Dist: cryptography>=41.0; extra == "security"
+Requires-Dist: pynacl>=1.5.0; extra == "security"
+Provides-Extra: moderation
+Requires-Dist: transformers>=4.36.0; extra == "moderation"
+Requires-Dist: onnxruntime>=1.16.0; extra == "moderation"
+Provides-Extra: mlflow
+Requires-Dist: mlflow>=2.0.0; extra == "mlflow"
+Provides-Extra: monitoring
+Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.21.0; extra == "monitoring"
+Requires-Dist: opentelemetry-proto>=1.21.0; extra == "monitoring"
+Requires-Dist: nvidia-ml-py>=12.0; extra == "monitoring"
+Requires-Dist: psutil>=5.9.0; extra == "monitoring"
+Provides-Extra: model-extras
+Requires-Dist: bitsandbytes>=0.41.0; extra == "model-extras"
+Requires-Dist: peft>=0.14.0; extra == "model-extras"
+Provides-Extra: wan
+Requires-Dist: aioquic>=1.0.0; extra == "wan"
+Provides-Extra: wandb
+Requires-Dist: wandb>=0.18; extra == "wandb"
+Requires-Dist: nvidia-ml-py>=12.0; extra == "wandb"
+Provides-Extra: spark
+Requires-Dist: pyspark>=3.4.0; extra == "spark"
+Provides-Extra: spark-mlflow
+Requires-Dist: distributed-llm[mlflow,spark]; extra == "spark-mlflow"
+Provides-Extra: openai-agents
+Requires-Dist: openai-agents>=0.1.0; extra == "openai-agents"
+Requires-Dist: openai>=1.0.0; extra == "openai-agents"
+Provides-Extra: agno
+Requires-Dist: agno>=1.0.0; extra == "agno"
+Provides-Extra: autogpt
+Requires-Dist: autogpt>=0.4.0; extra == "autogpt"
+Provides-Extra: vector-db-pinecone
+Requires-Dist: pinecone>=3.0.0; extra == "vector-db-pinecone"
+Provides-Extra: vector-db-qdrant
+Requires-Dist: qdrant-client>=1.9.0; extra == "vector-db-qdrant"
+Provides-Extra: vector-db-weaviate
+Requires-Dist: weaviate-client>=4.0.0; extra == "vector-db-weaviate"
+Provides-Extra: vector-db-milvus
+Requires-Dist: pymilvus>=2.3.0; extra == "vector-db-milvus"
+Provides-Extra: vector-db-all
+Requires-Dist: distributed-llm[vector-db-milvus,vector-db-pinecone,vector-db-qdrant,vector-db-weaviate]; extra == "vector-db-all"
+Provides-Extra: spot
+Requires-Dist: httpx>=0.27; extra == "spot"
+Provides-Extra: all
+Requires-Dist: distributed-llm[agno,autogpt,backends,cli,examples,mlflow,observability,openai-agents,ray,sdk,security,self-hosted,spark,spot,ui,wan]; extra == "all"
+Requires-Dist: distributed-llm[model-extras]; extra == "all"
+Provides-Extra: all-dev
+Requires-Dist: distributed-llm[all,dev]; extra == "all-dev"
+Requires-Dist: locust>=2.20; extra == "all-dev"
+Requires-Dist: safety>=3.0; extra == "all-dev"
+Dynamic: license-file
+
+# DistLLM — Distributed Inference Across All Your Devices
+
+**Pool GPUs from every device you own to run models no single machine can handle.**
+
+[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
+[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
+
+You have a gaming PC with an RTX 4090. Your laptop has an RTX 4060. Your friend has a desktop with an RTX 3080. None of you can run Llama 3.1 70B alone. **Together, you can.**
+
+DistLLM splits large language models across all your devices using pipeline parallelism. Each device runs a fraction of the model layers. Automatic discovery. Auto-partitioning. Works over LAN, WiFi, or internet.
+
+```
+Your Laptop (RTX 4060) ────┐
+                            │
+Your Gaming PC (RTX 4090) ──┼──► DistLLM Cluster ──► Run 70B models
+                            │
+Friend's PC (RTX 3080) ────┘
+```
+
+## Why DistLLM?
+
+| Problem | Solution |
+|---------|----------|
+| One GPU can't run today's best models | Split across all your devices |
+| Cloud inference costs thousands/month | Use the GPUs you already own |
+| Data privacy concerns with cloud APIs | Your data stays on your devices |
+| Slow single-device inference | Pipeline parallelism = faster generation |
+| Setting up distributed systems is hard | One command to start, one to join |
+
+## Quick Start
+
+```bash
+pip install distllm
+
+# On your main machine — start a cluster
+distllm cluster start --model meta-llama/Llama-3.2-7B
+
+# On every other machine — join the cluster
+distllm cluster join
+```
+
+## How It Works
+
+DistLLM uses **pipeline parallelism** — the model is split across devices by layers:
+
+```
+Device 1 (Laptop): Layers 0-5 ──→ Device 2 (Desktop): Layers 6-11 ──→ Device 3 (Friend's PC): Layers 12-17
+
+Each device runs ~6 layers → fits in 6-8GB VRAM
+Combined pool → runs models up to 70B parameters
+```
+
+Key capabilities:
+- **Auto-discovery**: devices find each other on the same network automatically
+- **Auto-partitioning**: automatically assigns layers based on each device's GPU
+- **Node recovery**: if a device disconnects, remaining nodes take over
+- **Straggler detection**: slow nodes are detected and worked around
+- **WAN optimization**: token accumulation for low-latency cross-internet inference
+- **Privacy-first**: keep sensitive layers on your own devices
+
+## Installation
+
+```bash
+# Core package
+pip install distllm
+
+# With vLLM backend (recommended for NVIDIA GPUs)
+pip install "distllm[vllm]"
+
+# With llama.cpp backend (CPU, AMD, Apple Silicon)
+pip install "distllm[llamacpp]"
+
+# Development
+pip install -e ".[dev]"
+```
+
+## Key Features
+
+- **Pipeline parallelism** — split any HuggingFace model across N devices
+- **Auto-discovery** — mDNS/zeroconf device finding on LAN
+- **6 backends** — vLLM, llama.cpp, TensorRT-LLM, ExLlamaV2, ONNX, PyTorch
+- **Auth plugin** — JWT authentication + RBAC role-based access control
+- **Health watchdog** — Continuous node health monitoring with auto circuit-breaking
+- **Semantic caching** — Deduplicate repeated prompts with embedding similarity
+- **Token streaming** — `generate_stream()` for real-time token-by-token responses
+- **Config validation** — Cross-field validation catches invalid combinations at load time
+- **Circuit breaker** — Graduated backpressure for load shedding
+- **Auto-partitioning** — hardware-aware DP solver for optimal layer assignment
+- **Node recovery** — checkpoint-based recovery when nodes disconnect
+- **Straggler detection** — statistical outlier detection for slow nodes
+- **Dynamic rebalancing** — redistribute layers when nodes join/leave
+- **P2P KV cache gossip** — CRDT-based cache sharing between nodes
+- **Wide-area support** — token accumulation for internet-scale inference
+- **Quantization** — 4-bit/8-bit to fit larger models on consumer GPUs
+- **OpenAI-compatible API** — use any OpenAI client to send requests
+- **Full observability** — Prometheus metrics, OTel tracing, structured logging
+- **Interactive chat** — `distllm chat` for CLI-based interaction
+- **Auth plugin** — JWT authentication with RBAC role-based access control
+- **Health watchdog** — continuous node health monitoring with automatic failover
+- **Semantic cache** — caching plugin with deduplication for repeated prompts
+- **Token streaming** — `generate_stream()` SDK method for token-by-token responses
+- **Health endpoints** — `/healthz` (liveness) and `/readyz` (readiness) for Kubernetes probes
+
+## CLI Commands
+
+```bash
+# Start a coordinator node
+distllm-coordinator --model meta-llama/Llama-3.2-1B --local --chat
+
+# Start distributed coordinator
+distllm-coordinator --model meta-llama/Llama-3.2-7B \
+  --nodes laptop:50051:0:5 desktop:50052:6:11 friend:50053:12:17
+
+# Start a worker node
+distllm-node --node-id laptop --model meta-llama/Llama-3.2-7B \
+  --start-layer 0 --end-layer 5 --total-layers 18 \
+  --coordinator-host 192.168.1.100 --coordinator-port 50050
+
+# Start the REST API server
+distllm-api --model meta-llama/Llama-3.2-1B --local
+
+# System diagnostics (check Python, CUDA, GPU, network, ports)
+distllm doctor
+
+# Quantize a model for smaller footprint
+distllm tune quantize --model meta-llama/Llama-3.2-7B --bits 4
+
+# Batch inference over a dataset
+distllm tune batch --input prompts.jsonl --output results.jsonl
+
+# Warm the semantic cache from prior prompts
+distllm tune cache --preload cache_seed.jsonl
+```
+
+## Documentation
+
+- [Architecture](docs/architecture.md) — Pipeline parallelism, node topology, KV cache management
+- [Deployment](DEPLOYMENT.md) — Local, Docker, and multi-machine deployment
+- [API Reference](docs/api.md) — OpenAI-compatible API docs
+
+## Project Structure
+
+```
+src/distllm/
+├── core/              # Coordinator, request pipeline, batch scheduler
+├── dist/              # Distributed inference engine
+│   ├── pipeline.py    # PipelineOrchestrator — multi-node execution
+│   ├── worker.py      # WorkerNode — per-device model subset
+│   ├── recovery.py    # NodeRecoveryManager — failure handling
+│   ├── straggler.py   # StragglerDetector — slow node detection
+│   ├── rebalancer.py  # Dynamic pipeline rebalancing
+│   ├── wide_area.py   # WAN-optimized inference
+│   ├── parallel.py    # Hybrid parallelism auto-selector
+│   ├── p2p/           # P2P gossip protocol & discovery
+│   └── partition/     # Hardware-aware auto-partitioner
+├── api/               # OpenAI-compatible REST API
+├── models/            # Model partitioning & loading
+├── cli/               # CLI tool
+├── sdk/               # Python client
+├── observability/     # Metrics, tracing, logging
+└── dashboard/         # Web dashboard
+```
+
+## Supported Model Architectures
+
+GPT-2, GPT-Neo, Llama 2/3, Mistral, Mixtral, Qwen2.5, Phi, DeepSeek, StableLM, Pythia, Baichuan, ChatGLM, InternLM, and more via HuggingFace AutoModel.
+
+## Roadmap
+
+- **Current**: Pipeline parallelism across LAN devices, manual node configuration
+- **Q3 2026**: Auto-discovery (mDNS), auto-partitioning, node recovery, GUI dashboard
+- **Q4 2026**: NAT traversal (cross-internet), P2P model distribution, GPU reputation system
+- **2027**: Federated clusters, speculative parallelism, privacy-preserving split, incentive system
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md).
+
+## License
+
+Apache 2.0. See [LICENSE](LICENSE).

distributed_llm-0.4.1/README.md

@@ -0,0 +1,181 @@
+# DistLLM — Distributed Inference Across All Your Devices
+
+**Pool GPUs from every device you own to run models no single machine can handle.**
+
+[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
+[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
+
+You have a gaming PC with an RTX 4090. Your laptop has an RTX 4060. Your friend has a desktop with an RTX 3080. None of you can run Llama 3.1 70B alone. **Together, you can.**
+
+DistLLM splits large language models across all your devices using pipeline parallelism. Each device runs a fraction of the model layers. Automatic discovery. Auto-partitioning. Works over LAN, WiFi, or internet.
+
+```
+Your Laptop (RTX 4060) ────┐
+                            │
+Your Gaming PC (RTX 4090) ──┼──► DistLLM Cluster ──► Run 70B models
+                            │
+Friend's PC (RTX 3080) ────┘
+```
+
+## Why DistLLM?
+
+| Problem | Solution |
+|---------|----------|
+| One GPU can't run today's best models | Split across all your devices |
+| Cloud inference costs thousands/month | Use the GPUs you already own |
+| Data privacy concerns with cloud APIs | Your data stays on your devices |
+| Slow single-device inference | Pipeline parallelism = faster generation |
+| Setting up distributed systems is hard | One command to start, one to join |
+
+## Quick Start
+
+```bash
+pip install distllm
+
+# On your main machine — start a cluster
+distllm cluster start --model meta-llama/Llama-3.2-7B
+
+# On every other machine — join the cluster
+distllm cluster join
+```
+
+## How It Works
+
+DistLLM uses **pipeline parallelism** — the model is split across devices by layers:
+
+```
+Device 1 (Laptop): Layers 0-5 ──→ Device 2 (Desktop): Layers 6-11 ──→ Device 3 (Friend's PC): Layers 12-17
+
+Each device runs ~6 layers → fits in 6-8GB VRAM
+Combined pool → runs models up to 70B parameters
+```
+
+Key capabilities:
+- **Auto-discovery**: devices find each other on the same network automatically
+- **Auto-partitioning**: automatically assigns layers based on each device's GPU
+- **Node recovery**: if a device disconnects, remaining nodes take over
+- **Straggler detection**: slow nodes are detected and worked around
+- **WAN optimization**: token accumulation for low-latency cross-internet inference
+- **Privacy-first**: keep sensitive layers on your own devices
+
+## Installation
+
+```bash
+# Core package
+pip install distllm
+
+# With vLLM backend (recommended for NVIDIA GPUs)
+pip install "distllm[vllm]"
+
+# With llama.cpp backend (CPU, AMD, Apple Silicon)
+pip install "distllm[llamacpp]"
+
+# Development
+pip install -e ".[dev]"
+```
+
+## Key Features
+
+- **Pipeline parallelism** — split any HuggingFace model across N devices
+- **Auto-discovery** — mDNS/zeroconf device finding on LAN
+- **6 backends** — vLLM, llama.cpp, TensorRT-LLM, ExLlamaV2, ONNX, PyTorch
+- **Auth plugin** — JWT authentication + RBAC role-based access control
+- **Health watchdog** — Continuous node health monitoring with auto circuit-breaking
+- **Semantic caching** — Deduplicate repeated prompts with embedding similarity
+- **Token streaming** — `generate_stream()` for real-time token-by-token responses
+- **Config validation** — Cross-field validation catches invalid combinations at load time
+- **Circuit breaker** — Graduated backpressure for load shedding
+- **Auto-partitioning** — hardware-aware DP solver for optimal layer assignment
+- **Node recovery** — checkpoint-based recovery when nodes disconnect
+- **Straggler detection** — statistical outlier detection for slow nodes
+- **Dynamic rebalancing** — redistribute layers when nodes join/leave
+- **P2P KV cache gossip** — CRDT-based cache sharing between nodes
+- **Wide-area support** — token accumulation for internet-scale inference
+- **Quantization** — 4-bit/8-bit to fit larger models on consumer GPUs
+- **OpenAI-compatible API** — use any OpenAI client to send requests
+- **Full observability** — Prometheus metrics, OTel tracing, structured logging
+- **Interactive chat** — `distllm chat` for CLI-based interaction
+- **Auth plugin** — JWT authentication with RBAC role-based access control
+- **Health watchdog** — continuous node health monitoring with automatic failover
+- **Semantic cache** — caching plugin with deduplication for repeated prompts
+- **Token streaming** — `generate_stream()` SDK method for token-by-token responses
+- **Health endpoints** — `/healthz` (liveness) and `/readyz` (readiness) for Kubernetes probes
+
+## CLI Commands
+
+```bash
+# Start a coordinator node
+distllm-coordinator --model meta-llama/Llama-3.2-1B --local --chat
+
+# Start distributed coordinator
+distllm-coordinator --model meta-llama/Llama-3.2-7B \
+  --nodes laptop:50051:0:5 desktop:50052:6:11 friend:50053:12:17
+
+# Start a worker node
+distllm-node --node-id laptop --model meta-llama/Llama-3.2-7B \
+  --start-layer 0 --end-layer 5 --total-layers 18 \
+  --coordinator-host 192.168.1.100 --coordinator-port 50050
+
+# Start the REST API server
+distllm-api --model meta-llama/Llama-3.2-1B --local
+
+# System diagnostics (check Python, CUDA, GPU, network, ports)
+distllm doctor
+
+# Quantize a model for smaller footprint
+distllm tune quantize --model meta-llama/Llama-3.2-7B --bits 4
+
+# Batch inference over a dataset
+distllm tune batch --input prompts.jsonl --output results.jsonl
+
+# Warm the semantic cache from prior prompts
+distllm tune cache --preload cache_seed.jsonl
+```
+
+## Documentation
+
+- [Architecture](docs/architecture.md) — Pipeline parallelism, node topology, KV cache management
+- [Deployment](DEPLOYMENT.md) — Local, Docker, and multi-machine deployment
+- [API Reference](docs/api.md) — OpenAI-compatible API docs
+
+## Project Structure
+
+```
+src/distllm/
+├── core/              # Coordinator, request pipeline, batch scheduler
+├── dist/              # Distributed inference engine
+│   ├── pipeline.py    # PipelineOrchestrator — multi-node execution
+│   ├── worker.py      # WorkerNode — per-device model subset
+│   ├── recovery.py    # NodeRecoveryManager — failure handling
+│   ├── straggler.py   # StragglerDetector — slow node detection
+│   ├── rebalancer.py  # Dynamic pipeline rebalancing
+│   ├── wide_area.py   # WAN-optimized inference
+│   ├── parallel.py    # Hybrid parallelism auto-selector
+│   ├── p2p/           # P2P gossip protocol & discovery
+│   └── partition/     # Hardware-aware auto-partitioner
+├── api/               # OpenAI-compatible REST API
+├── models/            # Model partitioning & loading
+├── cli/               # CLI tool
+├── sdk/               # Python client
+├── observability/     # Metrics, tracing, logging
+└── dashboard/         # Web dashboard
+```
+
+## Supported Model Architectures
+
+GPT-2, GPT-Neo, Llama 2/3, Mistral, Mixtral, Qwen2.5, Phi, DeepSeek, StableLM, Pythia, Baichuan, ChatGLM, InternLM, and more via HuggingFace AutoModel.
+
+## Roadmap
+
+- **Current**: Pipeline parallelism across LAN devices, manual node configuration
+- **Q3 2026**: Auto-discovery (mDNS), auto-partitioning, node recovery, GUI dashboard
+- **Q4 2026**: NAT traversal (cross-internet), P2P model distribution, GPU reputation system
+- **2027**: Federated clusters, speculative parallelism, privacy-preserving split, incentive system
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md).
+
+## License
+
+Apache 2.0. See [LICENSE](LICENSE).