morphml 1.0.0__py3-none-any.whl

morphml/distributed/proto/README.md

@@ -0,0 +1,224 @@
+# Protocol Buffer Stubs for gRPC Communication
+
+This directory contains Protocol Buffer definitions and generated Python stubs for distributed communication between Master and Worker nodes.
+
+## Files
+
+- **`worker.proto`** - Protocol Buffer definition (source)
+- **`worker_pb2.py`** - Generated message classes (auto-generated)
+- **`worker_pb2_grpc.py`** - Generated gRPC service stubs (auto-generated)
+- **`__init__.py`** - Package initialization with exports
+
+## Services Defined
+
+### MasterService (called by workers)
+- `RegisterWorker()` - Register a new worker with the master
+- `Heartbeat()` - Send periodic heartbeat to indicate worker is alive
+- `SubmitResult()` - Submit evaluation results back to master
+- `RequestTask()` - Request tasks from master (pull model)
+
+### WorkerService (called by master)
+- `Evaluate()` - Evaluate an architecture (push model)
+- `GetStatus()` - Get current worker status
+- `Shutdown()` - Gracefully shutdown worker
+- `CancelTask()` - Cancel a running task
+
+## Message Types
+
+### Registration
+- `RegisterRequest` / `RegisterResponse`
+
+### Heartbeat
+- `HeartbeatRequest` / `HeartbeatResponse`
+- `WorkerMetrics` - CPU, memory, GPU usage stats
+
+### Task Evaluation
+- `EvaluateRequest` / `EvaluateResponse`
+- `EvaluationConfig` - Training configuration
+- `TaskRequest` / `TaskResponse`
+
+### Results
+- `ResultRequest` / `ResultResponse`
+
+### Control
+- `StatusRequest` / `StatusResponse`
+- `ShutdownRequest` / `ShutdownResponse`
+- `CancelRequest` / `CancelResponse`
+
+## Regenerating Stubs
+
+If you modify `worker.proto`, regenerate the Python stubs:
+
+### Option 1: Using the provided script (Recommended)
+
+```bash
+python scripts/compile_protos.py
+```
+
+### Option 2: Manual compilation
+
+```bash
+python -m grpc_tools.protoc \
+  -I morphml/distributed/proto \
+  --python_out=morphml/distributed/proto \
+  --grpc_python_out=morphml/distributed/proto \
+  --pyi_out=morphml/distributed/proto \
+  morphml/distributed/proto/worker.proto
+```
+
+**Note:** Requires `grpcio-tools` to be installed:
+```bash
+pip install grpcio-tools
+# or with poetry
+poetry install --extras distributed
+```
+
+## Usage
+
+### Importing in Python
+
+```python
+from morphml.distributed.proto import (
+    worker_pb2,
+    worker_pb2_grpc,
+    RegisterRequest,
+    EvaluateRequest,
+    # ... other message types
+)
+```
+
+### Master Node Example
+
+```python
+import grpc
+from morphml.distributed.proto import worker_pb2_grpc, RegisterResponse
+
+class MyMasterServicer(worker_pb2_grpc.MasterServiceServicer):
+    def RegisterWorker(self, request, context):
+        print(f"Worker {request.worker_id} registered from {request.host}:{request.port}")
+        return RegisterResponse(
+            success=True,
+            message="Registration successful",
+            master_id="master-001"
+        )
+
+# Create server
+server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
+worker_pb2_grpc.add_MasterServiceServicer_to_server(MyMasterServicer(), server)
+server.add_insecure_port('[::]:50051')
+server.start()
+```
+
+### Worker Node Example
+
+```python
+import grpc
+from morphml.distributed.proto import worker_pb2, worker_pb2_grpc
+
+# Connect to master
+channel = grpc.insecure_channel('localhost:50051')
+stub = worker_pb2_grpc.MasterServiceStub(channel)
+
+# Register with master
+request = worker_pb2.RegisterRequest(
+    worker_id='worker-1',
+    host='localhost',
+    port=50052,
+    num_gpus=2,
+    gpu_ids=[0, 1]
+)
+response = stub.RegisterWorker(request)
+print(f"Registration: {response.success} - {response.message}")
+```
+
+## Dependencies
+
+Required packages:
+- `grpcio>=1.54.0` - gRPC runtime
+- `protobuf>=4.23.0` - Protocol Buffer runtime
+- `grpcio-tools>=1.54.0` - For compilation (development only)
+
+Install with:
+```bash
+pip install grpcio protobuf grpcio-tools
+```
+
+Or using poetry:
+```bash
+poetry install --extras distributed
+```
+
+## Communication Flow
+
+```
+Worker                          Master
+  |                               |
+  |-- RegisterWorker() ---------->|
+  |<-- RegisterResponse ----------|
+  |                               |
+  |-- Heartbeat() (every 10s) -->|
+  |<-- HeartbeatResponse ---------|
+  |                               |
+  |<-- Evaluate(task) ------------|  (push model)
+  |-- SubmitResult(metrics) ----->|
+  |                               |
+  |-- RequestTask() ------------->|  (pull model)
+  |<-- TaskResponse(tasks) -------|
+  |                               |
+```
+
+## Protocol Details
+
+- **Serialization:** Protocol Buffers v3 (binary)
+- **Transport:** gRPC over HTTP/2
+- **Default Ports:**
+  - Master: 50051
+  - Worker: 50052+
+- **Timeout:** Configurable per-request
+- **Compression:** Optional (gzip)
+
+## Troubleshooting
+
+### Import Errors
+
+If you see:
+```
+ImportError: cannot import name 'worker_pb2' from 'morphml.distributed.proto'
+```
+
+**Solution:** Generate the protobuf files:
+```bash
+python scripts/compile_protos.py
+```
+
+### gRPC Connection Issues
+
+If workers can't connect to master:
+1. Check firewall settings
+2. Verify master is running and listening
+3. Check network connectivity: `telnet master-host 50051`
+4. Review master logs for errors
+
+### Version Incompatibility
+
+Ensure compatible versions:
+```bash
+pip list | grep grpc
+pip list | grep protobuf
+```
+
+Recommended versions:
+- grpcio: ≥1.54.0, <2.0.0
+- protobuf: ≥4.23.0, <5.0.0
+
+## Learn More
+
+- [gRPC Python Documentation](https://grpc.io/docs/languages/python/)
+- [Protocol Buffers Guide](https://developers.google.com/protocol-buffers)
+- [MorphML Distributed Module](/morphml/distributed/)
+
+---
+
+**Authors:** Vedanth ([@vedanthq](https://github.com/vedanthq)) & Eshan Roy ([@eshanized](https://github.com/eshanized))  
+**Organization:** TONMOY INFRASTRUCTURE & VISION  
+**Auto-generated:** Do not edit `*_pb2.py` files manually

morphml/distributed/proto/__init__.py

@@ -0,0 +1,74 @@
+"""Generated Protocol Buffer stubs for distributed communication.
+
+Auto-generated files. To regenerate, run:
+    python scripts/compile_protos.py
+
+Author: Eshan Roy <eshanized@proton.me>
+Organization: TONMOY INFRASTRUCTURE & VISION
+"""
+
+try:
+    from morphml.distributed.proto.worker_pb2 import (
+        CancelRequest,
+        CancelResponse,
+        EvaluateRequest,
+        EvaluateResponse,
+        EvaluationConfig,
+        HeartbeatRequest,
+        HeartbeatResponse,
+        RegisterRequest,
+        RegisterResponse,
+        ResultRequest,
+        ResultResponse,
+        ShutdownRequest,
+        ShutdownResponse,
+        StatusRequest,
+        StatusResponse,
+        TaskRequest,
+        TaskResponse,
+        WorkerMetrics,
+    )
+    from morphml.distributed.proto.worker_pb2_grpc import (
+        MasterService,
+        MasterServiceServicer,
+        MasterServiceStub,
+        WorkerService,
+        WorkerServiceServicer,
+        WorkerServiceStub,
+        add_MasterServiceServicer_to_server,
+        add_WorkerServiceServicer_to_server,
+    )
+
+    __all__ = [
+        # Messages
+        "RegisterRequest",
+        "RegisterResponse",
+        "HeartbeatRequest",
+        "HeartbeatResponse",
+        "WorkerMetrics",
+        "EvaluateRequest",
+        "EvaluateResponse",
+        "EvaluationConfig",
+        "TaskRequest",
+        "TaskResponse",
+        "ResultRequest",
+        "ResultResponse",
+        "StatusRequest",
+        "StatusResponse",
+        "ShutdownRequest",
+        "ShutdownResponse",
+        "CancelRequest",
+        "CancelResponse",
+        # Services
+        "MasterServiceStub",
+        "MasterServiceServicer",
+        "MasterService",
+        "add_MasterServiceServicer_to_server",
+        "WorkerServiceStub",
+        "WorkerServiceServicer",
+        "WorkerService",
+        "add_WorkerServiceServicer_to_server",
+    ]
+except ImportError:
+    # Protobuf files not generated or grpc not installed
+    pass

morphml/distributed/proto/worker.proto

@@ -0,0 +1,170 @@
+syntax = "proto3";
+
+package morphml.distributed;
+
+// Master service - called by workers
+service MasterService {
+    // Register a new worker with the master
+    rpc RegisterWorker(RegisterRequest) returns (RegisterResponse);
+    
+    // Send periodic heartbeat
+    rpc Heartbeat(HeartbeatRequest) returns (HeartbeatResponse);
+    
+    // Submit evaluation result
+    rpc SubmitResult(ResultRequest) returns (ResultResponse);
+    
+    // Request a task (pull model)
+    rpc RequestTask(TaskRequest) returns (TaskResponse);
+}
+
+// Worker service - called by master
+service WorkerService {
+    // Evaluate an architecture (push model)
+    rpc Evaluate(EvaluateRequest) returns (EvaluateResponse);
+    
+    // Get worker status
+    rpc GetStatus(StatusRequest) returns (StatusResponse);
+    
+    // Shutdown worker gracefully
+    rpc Shutdown(ShutdownRequest) returns (ShutdownResponse);
+    
+    // Cancel running task
+    rpc CancelTask(CancelRequest) returns (CancelResponse);
+}
+
+// ====================
+// Registration
+// ====================
+
+message RegisterRequest {
+    string worker_id = 1;
+    string host = 2;
+    int32 port = 3;
+    int32 num_gpus = 4;
+    repeated int32 gpu_ids = 5;
+    map<string, string> metadata = 6;
+}
+
+message RegisterResponse {
+    bool success = 1;
+    string message = 2;
+    string master_id = 3;
+}
+
+// ====================
+// Heartbeat
+// ====================
+
+message HeartbeatRequest {
+    string worker_id = 1;
+    string status = 2;  // 'idle', 'busy', 'error'
+    string current_task_id = 3;
+    WorkerMetrics metrics = 4;
+}
+
+message HeartbeatResponse {
+    bool acknowledged = 1;
+    bool should_continue = 2;  // false = shutdown
+}
+
+message WorkerMetrics {
+    double cpu_usage = 1;
+    double memory_usage = 2;
+    double gpu_usage = 3;
+    double gpu_memory = 4;
+    int32 tasks_completed = 5;
+    int32 tasks_failed = 6;
+}
+
+// ====================
+// Task Evaluation
+// ====================
+
+message EvaluateRequest {
+    string task_id = 1;
+    string architecture = 2;  // JSON serialized ModelGraph
+    EvaluationConfig config = 3;
+}
+
+message EvaluateResponse {
+    string task_id = 1;
+    bool success = 2;
+    map<string, double> metrics = 3;
+    string error = 4;
+    double duration = 5;
+}
+
+message EvaluationConfig {
+    int32 num_epochs = 1;
+    int32 batch_size = 2;
+    double learning_rate = 3;
+    string dataset = 4;
+    int32 gpu_id = 5;
+    map<string, string> extra = 6;
+}
+
+// ====================
+// Task Request (Pull Model)
+// ====================
+
+message TaskRequest {
+    string worker_id = 1;
+    int32 max_tasks = 2;  // Number of tasks to request
+}
+
+message TaskResponse {
+    bool has_task = 1;
+    repeated EvaluateRequest tasks = 2;
+}
+
+// ====================
+// Result Submission
+// ====================
+
+message ResultRequest {
+    string task_id = 1;
+    string worker_id = 2;
+    bool success = 3;
+    map<string, double> metrics = 4;
+    string error = 5;
+    double duration = 6;
+    string architecture = 7;  // Optional: return modified architecture
+}
+
+message ResultResponse {
+    bool acknowledged = 1;
+    string message = 2;
+}
+
+// ====================
+// Status & Control
+// ====================
+
+message StatusRequest {
+    string worker_id = 1;
+}
+
+message StatusResponse {
+    string status = 1;  // 'idle', 'busy', 'error', 'dead'
+    string current_task_id = 2;
+    WorkerMetrics metrics = 3;
+    int64 uptime_seconds = 4;
+}
+
+message ShutdownRequest {
+    string worker_id = 1;
+    bool graceful = 2;  // Wait for current task to finish
+}
+
+message ShutdownResponse {
+    bool acknowledged = 1;
+}
+
+message CancelRequest {
+    string task_id = 1;
+}
+
+message CancelResponse {
+    bool success = 1;
+    string message = 2;
+}

morphml/distributed/proto/worker_pb2.py

@@ -0,0 +1,79 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler.  DO NOT EDIT!
+# source: worker.proto
+"""Generated protocol buffer code."""
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import descriptor_pool as _descriptor_pool
+from google.protobuf import symbol_database as _symbol_database
+from google.protobuf.internal import builder as _builder
+
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(
+    b'\n\x0cworker.proto\x12\x15morphml.distributed"\x87\x01\n\x0fRegisterRequest\x12\x11\n\tworker_id\x18\x01 \x01(\t\x12\x0c\n\x04host\x18\x02 \x01(\t\x12\x0c\n\x04port\x18\x03 \x01(\x05\x12\x10\n\x08num_gpus\x18\x04 \x01(\x05\x12\x0f\n\x07gpu_ids\x18\x05 \x03(\x05\x12"\n\x08metadata\x18\x06 \x03(\x0b\x32\x10.RegisterRequest.MetadataEntry\x1a/\n\rMetadataEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01"I\n\x10RegisterResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t\x12\x13\n\x0bmaster_id\x18\x03 \x01(\t"q\n\x10HeartbeatRequest\x12\x11\n\tworker_id\x18\x01 \x01(\t\x12\x0e\n\x06status\x18\x02 \x01(\t\x12\x17\n\x0fcurrent_task_id\x18\x03 \x01(\t\x12!\n\x07metrics\x18\x04 \x01(\x0b\x32\x10.WorkerMetrics">\n\x11HeartbeatResponse\x12\x14\n\x0cacknowledged\x18\x01 \x01(\x08\x12\x13\n\x0bshould_continue\x18\x02 \x01(\x08"\xae\x01\n\rWorkerMetrics\x12\x11\n\tcpu_usage\x18\x01 \x01(\x01\x12\x14\n\x0cmemory_usage\x18\x02 \x01(\x01\x12\x11\n\tgpu_usage\x18\x03 \x01(\x01\x12\x12\n\ngpu_memory\x18\x04 \x01(\x01\x12\x17\n\x0ftasks_completed\x18\x05 \x01(\x05\x12\x14\n\x0ctasks_failed\x18\x06 \x01(\x05"^\n\x0fEvaluateRequest\x12\x0f\n\x07task_id\x18\x01 \x01(\t\x12\x14\n\x0carchitecture\x18\x02 \x01(\t\x12$\n\x06\x63onfig\x18\x03 \x01(\x0b\x32\x14.EvaluationConfig"\x8b\x01\n\x10\x45valuateResponse\x12\x0f\n\x07task_id\x18\x01 \x01(\x08\x12\x0f\n\x07success\x18\x02 \x01(\x08\x12+\n\x07metrics\x18\x03 \x03(\x0b\x32\x1a.EvaluateResponse.MetricsEntry\x12\r\n\x05\x65rror\x18\x04 \x01(\t\x12\x10\n\x08\x64uration\x18\x05 \x01(\x01\x1a\x31\n\x0cMetricsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\x01:\x02\x38\x01"\xae\x01\n\x10\x45valuationConfig\x12\x12\n\nnum_epochs\x18\x01 \x01(\x05\x12\x12\n\nbatch_size\x18\x02 \x01(\x05\x12\x15\n\rlearning_rate\x18\x03 \x01(\x01\x12\x0f\n\x07\x64\x61taset\x18\x04 \x01(\t\x12\x0e\n\x06gpu_id\x18\x05 \x01(\x05\x12*\n\x05\x65xtra\x18\x06 \x03(\x0b\x32\x1b.EvaluationConfig.ExtraEntry\x1a,\n\nExtraEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01"5\n\x0bTaskRequest\x12\x11\n\tworker_id\x18\x01 \x01(\t\x12\x13\n\x0bmax_tasks\x18\x02 \x01(\x05"G\n\x0cTaskResponse\x12\x10\n\x08has_task\x18\x01 \x01(\x08\x12%\n\x05tasks\x18\x02 \x03(\x0b\x32\x16.EvaluateRequest"\xb1\x01\n\rResultRequest\x12\x0f\n\x07task_id\x18\x01 \x01(\t\x12\x11\n\tworker_id\x18\x02 \x01(\t\x12\x0f\n\x07success\x18\x03 \x01(\x08\x12*\n\x07metrics\x18\x04 \x03(\x0b\x32\x19.ResultRequest.MetricsEntry\x12\r\n\x05\x65rror\x18\x05 \x01(\t\x12\x10\n\x08\x64uration\x18\x06 \x01(\x01\x12\x14\n\x0carchitecture\x18\x07 \x01(\t\x1a/\n\x0cMetricsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\x01:\x02\x38\x01"9\n\x0eResultResponse\x12\x14\n\x0cacknowledged\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t"$\n\rStatusRequest\x12\x11\n\tworker_id\x18\x01 \x01(\t"l\n\x0eStatusResponse\x12\x0e\n\x06status\x18\x01 \x01(\t\x12\x17\n\x0fcurrent_task_id\x18\x02 \x01(\t\x12!\n\x07metrics\x18\x03 \x01(\x0b\x32\x10.WorkerMetrics\x12\x0e\n\x06uptime\x18\x04 \x01(\x03"?\n\x0fShutdownRequest\x12\x11\n\tworker_id\x18\x01 \x01(\t\x12\x10\n\x08graceful\x18\x02 \x01(\x08")\n\x10ShutdownResponse\x12\x14\n\x0cacknowledged\x18\x01 \x01(\x08"!\n\rCancelRequest\x12\x0f\n\x07task_id\x18\x01 \x01(\t"3\n\x0e\x43\x61ncelResponse\x12\x0f\n\x07success\x18\x01 \x01(\x08\x12\x0f\n\x07message\x18\x02 \x01(\t2\xf1\x01\n\rMasterService\x12H\n\x0eRegisterWorker\x12\x1a.RegisterRequest\x1a\x1a.RegisterResponse\x12\x38\n\tHeartbeat\x12\x1a.HeartbeatRequest\x1a\x1a.HeartbeatResponse\x12;\n\x0cSubmitResult\x12\x17.ResultRequest\x1a\x18.ResultResponse\x12\x1f\n\x0bRequestTask\x12\x15.TaskRequest\x1a\x16.TaskResponse2\xf1\x01\n\rWorkerService\x12;\n\x08\x45valuate\x12\x1a.EvaluateRequest\x1a\x1b.EvaluateResponse\x12\x35\n\tGetStatus\x12\x17.StatusRequest\x1a\x18.StatusResponse\x12\x38\n\x08Shutdown\x12\x19.ShutdownRequest\x1a\x1a.ShutdownResponse\x12\x32\n\nCancelTask\x12\x17.CancelRequest\x1a\x18.CancelResponseb\x06proto3'
+)
+
+_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, globals())
+_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, "worker_pb2", globals())
+if _descriptor._USE_C_DESCRIPTORS is False:
+    DESCRIPTOR._options = None
+    _REGISTERREQUEST_METADATAENTRY._options = None
+    _REGISTERREQUEST_METADATAENTRY._serialized_options = b"8\001"
+    _EVALUATERESPONSE_METRICSENTRY._options = None
+    _EVALUATERESPONSE_METRICSENTRY._serialized_options = b"8\001"
+    _EVALUATIONCONFIG_EXTRAENTRY._options = None
+    _EVALUATIONCONFIG_EXTRAENTRY._serialized_options = b"8\001"
+    _RESULTREQUEST_METRICSENTRY._options = None
+    _RESULTREQUEST_METRICSENTRY._serialized_options = b"8\001"
+    _REGISTERREQUEST._serialized_start = 40
+    _REGISTERREQUEST._serialized_end = 175
+    _REGISTERREQUEST_METADATAENTRY._serialized_start = 128
+    _REGISTERREQUEST_METADATAENTRY._serialized_end = 175
+    _REGISTERRESPONSE._serialized_start = 177
+    _REGISTERRESPONSE._serialized_end = 250
+    _HEARTBEATREQUEST._serialized_start = 252
+    _HEARTBEATREQUEST._serialized_end = 365
+    _HEARTBEATRESPONSE._serialized_start = 367
+    _HEARTBEATRESPONSE._serialized_end = 429
+    _WORKERMETRICS._serialized_start = 432
+    _WORKERMETRICS._serialized_end = 606
+    _EVALUATEREQUEST._serialized_start = 608
+    _EVALUATEREQUEST._serialized_end = 702
+    _EVALUATERESPONSE._serialized_start = 705
+    _EVALUATERESPONSE._serialized_end = 844
+    _EVALUATERESPONSE_METRICSENTRY._serialized_start = 795
+    _EVALUATERESPONSE_METRICSENTRY._serialized_end = 844
+    _EVALUATIONCONFIG._serialized_start = 847
+    _EVALUATIONCONFIG._serialized_end = 1021
+    _EVALUATIONCONFIG_EXTRAENTRY._serialized_start = 977
+    _EVALUATIONCONFIG_EXTRAENTRY._serialized_end = 1021
+    _TASKREQUEST._serialized_start = 1023
+    _TASKREQUEST._serialized_end = 1076
+    _TASKRESPONSE._serialized_start = 1078
+    _TASKRESPONSE._serialized_end = 1149
+    _RESULTREQUEST._serialized_start = 1152
+    _RESULTREQUEST._serialized_end = 1329
+    _RESULTREQUEST_METRICSENTRY._serialized_start = 795
+    _RESULTREQUEST_METRICSENTRY._serialized_end = 844
+    _RESULTRESPONSE._serialized_start = 1331
+    _RESULTRESPONSE._serialized_end = 1388
+    _STATUSREQUEST._serialized_start = 1390
+    _STATUSREQUEST._serialized_end = 1426
+    _STATUSRESPONSE._serialized_start = 1428
+    _STATUSRESPONSE._serialized_end = 1536
+    _SHUTDOWNREQUEST._serialized_start = 1538
+    _SHUTDOWNREQUEST._serialized_end = 1601
+    _SHUTDOWNRESPONSE._serialized_start = 1603
+    _SHUTDOWNRESPONSE._serialized_end = 1644
+    _CANCELREQUEST._serialized_start = 1646
+    _CANCELREQUEST._serialized_end = 1679
+    _CANCELRESPONSE._serialized_start = 1681
+    _CANCELRESPONSE._serialized_end = 1732
+    _MASTERSERVICE._serialized_start = 1735
+    _MASTERSERVICE._serialized_end = 1976
+    _WORKERSERVICE._serialized_start = 1979
+    _WORKERSERVICE._serialized_end = 2220
+# @@protoc_insertion_point(module_scope)