turbolane-cli 2.0.0__tar.gz

turbolane_cli-2.0.0/PKG-INFO

@@ -0,0 +1,206 @@
+Metadata-Version: 2.4
+Name: turbolane-cli
+Version: 2.0.0
+Summary: RL-optimized parallel TCP file transfer CLI (TurboLane)
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: System :: Networking
+Classifier: Programming Language :: Python :: 3.10
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: requires-python
+Dynamic: summary
+
+# TurboLane — Phase 2: CLI File Transfer Server
+
+RL-optimized parallel TCP file transfer system. Phase 2 builds a production-grade
+CLI application on top of the Phase 1 TurboLane engine.
+
+---
+
+## Folder Structure
+
+```
+.
+├── turbolane/                  # Phase 1 — RL engine (UNCHANGED)
+│   ├── __init__.py
+│   ├── engine.py               # TurboLaneEngine — only public import
+│   ├── policies/
+│   │   └── federated.py        # FederatedPolicy (DCI / Q-learning)
+│   └── rl/
+│       ├── agent.py            # RLAgent (Q-table, Bellman updates)
+│       └── storage.py          # QTableStorage (atomic JSON persistence)
+│
+├── turbolane_server/           # Phase 2 — CLI application layer
+│   ├── __init__.py
+│   ├── protocol.py             # Binary wire protocol (struct + CRC32)
+│   ├── metrics.py              # In-app RTT / throughput / loss metrics
+│   ├── adapter.py              # TurboLaneAdapter — engine ↔ server bridge
+│   ├── transfer.py             # TransferSession + StreamWorker (sender side)
+│   ├── server.py               # TurboLaneServer + FileAssembler (receiver)
+│   ├── sender.py               # TurboLaneSender — orchestrator
+│   └── cli.py                  # argparse CLI: start / send / status
+│
+├── models/
+│   └── dci/                    # Q-table persistence directory
+├── setup.py
+└── README.md
+```
+
+---
+
+## Architecture
+
+```
+SENDER SIDE                          RECEIVER SIDE
+───────────────────────────────      ────────────────────────────
+  turbolane-server send               turbolane-server start
+        │                                     │
+  TurboLaneSender                    TurboLaneServer
+        │                                     │
+  ┌─────────────────┐                ┌────────────────────┐
+  │ TurboLaneAdapter│                │   accept loop      │
+  │  (5s RL loop)   │                │   (one thread/conn)│
+  │                 │                └────────────────────┘
+  │ TurboLaneEngine │                         │
+  │ (embedded DCI)  │                ┌────────────────────┐
+  └────────┬────────┘                │  StreamHandler     │
+           │ adjust_streams(n)       │  HELLO/CHUNK/PING  │
+           ▼                         └────────────────────┘
+  TransferSession                              │
+  ┌─────────────────────────────┐    ┌────────────────────┐
+  │  ChunkQueue (thread-safe)   │    │  FileAssembler     │
+  │  StreamWorker × N           │    │  (sparse write,    │
+  │  (one thread per stream)    │    │   out-of-order OK) │
+  └─────────────────────────────┘    └────────────────────┘
+           │ N×TCP connections
+           └──────────────────────────────────────────────┘
+
+MetricsCollector (shared)
+  ├── per-stream StreamMetrics
+  ├── RTT: in-app PING/PONG round-trip timing
+  ├── Throughput: bytes_sent / elapsed per snapshot
+  └── Loss: chunk retransmit rate proxy
+```
+
+### Key design rules
+- **TurboLane engine is completely decoupled** — only `adapter.py` imports from `turbolane.*`
+- **No networking in the engine** — sockets live only in `transfer.py` and `server.py`
+- **Single transfer lock** — server rejects new connections with `BUSY` during active transfer
+- **RTT without root** — measured via application-layer PING/PONG timing per stream
+
+---
+
+## Wire Protocol
+
+Binary struct header (34 bytes, big-endian) + payload:
+
+| Field        | Bytes | Description                          |
+|--------------|-------|--------------------------------------|
+| magic        | 4     | `0x544C414E` ("TLAN")                |
+| msg_type     | 1     | MessageType enum                     |
+| stream_id    | 1     | Parallel stream index (0-255)        |
+| chunk_idx    | 4     | Chunk index within file              |
+| total_chunks | 4     | Total chunks in transfer             |
+| seq          | 4     | Sequence number                      |
+| file_offset  | 8     | Byte offset in source file           |
+| data_len     | 4     | Payload length (0 for control)       |
+| checksum     | 4     | CRC32 of payload                     |
+| **payload**  | N     | Raw file bytes / JSON metadata       |
+
+Message types: `HELLO`, `HELLO_ACK`, `CHUNK`, `CHUNK_ACK`, `PING`, `PONG`,
+`TRANSFER_DONE`, `COMPLETE`, `ERROR`, `BUSY`, `STATUS_REQ`, `STATUS_RESP`
+
+---
+
+## Installation
+
+```bash
+# From the project root (where setup.py lives)
+pip install -e .
+```
+
+---
+
+## Usage
+
+### 1. Start the receiver server
+
+```bash
+turbolane-server start --port 9000 --output-dir ./received
+```
+
+Options:
+```
+--host HOST         Bind interface (default: 0.0.0.0)
+--port PORT         TCP port (default: 9000)
+--output-dir DIR    Where to save received files (default: ./received)
+--verbose / -v      Debug logging
+```
+
+### 2. Send a file
+
+```bash
+turbolane-server send /data/large_dataset.tar \
+    --host 192.168.1.50 --port 9000 \
+    --streams 6 \
+    --min-streams 1 --max-streams 32 \
+    --model-dir models/dci \
+    --interval 5.0
+```
+
+Options:
+```
+FILE                File to send (required positional)
+--host HOST         Receiver hostname/IP (required)
+--port PORT         Receiver port (default: 9000)
+--streams N         Initial parallel TCP streams (default: 4)
+--min-streams N     Minimum streams TurboLane may use (default: 1)
+--max-streams N     Maximum streams TurboLane may use (default: 32)
+--model-dir DIR     Q-table persistence directory (default: models/dci)
+--interval SECS     RL decision interval in seconds (default: 5.0)
+--timeout SECS      Max wait for completion (default: unlimited)
+--verbose / -v      Debug logging
+```
+
+### 3. Query server status
+
+```bash
+turbolane-server status --host 192.168.1.50 --port 9000
+```
+
+---
+
+## How TurboLane integrates (5-second loop)
+
+```
+Every 5 seconds (TurboLaneAdapter._tick):
+  1. MetricsCollector.snapshot()
+       → throughput_mbps (sum of per-stream byte rates)
+       → rtt_ms          (mean of PING/PONG RTT samples)
+       → loss_pct        (chunk retransmit rate proxy)
+
+  2. engine.learn(throughput, rtt, loss)
+       → Q-table Bellman update from previous decision's outcome
+
+  3. engine.decide(throughput, rtt, loss)
+       → Q-learning ε-greedy action → new stream count
+
+  4. session.adjust_streams(new_count)
+       → spawn / stop StreamWorker threads to match recommendation
+```
+
+---
+
+## Future upgrades (designed-in hooks)
+
+| Capability           | Where to add              |
+|----------------------|---------------------------|
+| PPO algorithm        | `turbolane/rl/` only      |
+| Multi-session        | `server.py` busy logic    |
+| Shared policy learning | `adapter.py` FederatedPolicy |
+| Resume / checkpointing | `ChunkQueue` + `FileAssembler` |
+| TLS encryption       | `StreamWorker` + `StreamHandler` |

turbolane_cli-2.0.0/README.md

@@ -0,0 +1,190 @@
+# TurboLane — Phase 2: CLI File Transfer Server
+
+RL-optimized parallel TCP file transfer system. Phase 2 builds a production-grade
+CLI application on top of the Phase 1 TurboLane engine.
+
+---
+
+## Folder Structure
+
+```
+.
+├── turbolane/                  # Phase 1 — RL engine (UNCHANGED)
+│   ├── __init__.py
+│   ├── engine.py               # TurboLaneEngine — only public import
+│   ├── policies/
+│   │   └── federated.py        # FederatedPolicy (DCI / Q-learning)
+│   └── rl/
+│       ├── agent.py            # RLAgent (Q-table, Bellman updates)
+│       └── storage.py          # QTableStorage (atomic JSON persistence)
+│
+├── turbolane_server/           # Phase 2 — CLI application layer
+│   ├── __init__.py
+│   ├── protocol.py             # Binary wire protocol (struct + CRC32)
+│   ├── metrics.py              # In-app RTT / throughput / loss metrics
+│   ├── adapter.py              # TurboLaneAdapter — engine ↔ server bridge
+│   ├── transfer.py             # TransferSession + StreamWorker (sender side)
+│   ├── server.py               # TurboLaneServer + FileAssembler (receiver)
+│   ├── sender.py               # TurboLaneSender — orchestrator
+│   └── cli.py                  # argparse CLI: start / send / status
+│
+├── models/
+│   └── dci/                    # Q-table persistence directory
+├── setup.py
+└── README.md
+```
+
+---
+
+## Architecture
+
+```
+SENDER SIDE                          RECEIVER SIDE
+───────────────────────────────      ────────────────────────────
+  turbolane-server send               turbolane-server start
+        │                                     │
+  TurboLaneSender                    TurboLaneServer
+        │                                     │
+  ┌─────────────────┐                ┌────────────────────┐
+  │ TurboLaneAdapter│                │   accept loop      │
+  │  (5s RL loop)   │                │   (one thread/conn)│
+  │                 │                └────────────────────┘
+  │ TurboLaneEngine │                         │
+  │ (embedded DCI)  │                ┌────────────────────┐
+  └────────┬────────┘                │  StreamHandler     │
+           │ adjust_streams(n)       │  HELLO/CHUNK/PING  │
+           ▼                         └────────────────────┘
+  TransferSession                              │
+  ┌─────────────────────────────┐    ┌────────────────────┐
+  │  ChunkQueue (thread-safe)   │    │  FileAssembler     │
+  │  StreamWorker × N           │    │  (sparse write,    │
+  │  (one thread per stream)    │    │   out-of-order OK) │
+  └─────────────────────────────┘    └────────────────────┘
+           │ N×TCP connections
+           └──────────────────────────────────────────────┘
+
+MetricsCollector (shared)
+  ├── per-stream StreamMetrics
+  ├── RTT: in-app PING/PONG round-trip timing
+  ├── Throughput: bytes_sent / elapsed per snapshot
+  └── Loss: chunk retransmit rate proxy
+```
+
+### Key design rules
+- **TurboLane engine is completely decoupled** — only `adapter.py` imports from `turbolane.*`
+- **No networking in the engine** — sockets live only in `transfer.py` and `server.py`
+- **Single transfer lock** — server rejects new connections with `BUSY` during active transfer
+- **RTT without root** — measured via application-layer PING/PONG timing per stream
+
+---
+
+## Wire Protocol
+
+Binary struct header (34 bytes, big-endian) + payload:
+
+| Field        | Bytes | Description                          |
+|--------------|-------|--------------------------------------|
+| magic        | 4     | `0x544C414E` ("TLAN")                |
+| msg_type     | 1     | MessageType enum                     |
+| stream_id    | 1     | Parallel stream index (0-255)        |
+| chunk_idx    | 4     | Chunk index within file              |
+| total_chunks | 4     | Total chunks in transfer             |
+| seq          | 4     | Sequence number                      |
+| file_offset  | 8     | Byte offset in source file           |
+| data_len     | 4     | Payload length (0 for control)       |
+| checksum     | 4     | CRC32 of payload                     |
+| **payload**  | N     | Raw file bytes / JSON metadata       |
+
+Message types: `HELLO`, `HELLO_ACK`, `CHUNK`, `CHUNK_ACK`, `PING`, `PONG`,
+`TRANSFER_DONE`, `COMPLETE`, `ERROR`, `BUSY`, `STATUS_REQ`, `STATUS_RESP`
+
+---
+
+## Installation
+
+```bash
+# From the project root (where setup.py lives)
+pip install -e .
+```
+
+---
+
+## Usage
+
+### 1. Start the receiver server
+
+```bash
+turbolane-server start --port 9000 --output-dir ./received
+```
+
+Options:
+```
+--host HOST         Bind interface (default: 0.0.0.0)
+--port PORT         TCP port (default: 9000)
+--output-dir DIR    Where to save received files (default: ./received)
+--verbose / -v      Debug logging
+```
+
+### 2. Send a file
+
+```bash
+turbolane-server send /data/large_dataset.tar \
+    --host 192.168.1.50 --port 9000 \
+    --streams 6 \
+    --min-streams 1 --max-streams 32 \
+    --model-dir models/dci \
+    --interval 5.0
+```
+
+Options:
+```
+FILE                File to send (required positional)
+--host HOST         Receiver hostname/IP (required)
+--port PORT         Receiver port (default: 9000)
+--streams N         Initial parallel TCP streams (default: 4)
+--min-streams N     Minimum streams TurboLane may use (default: 1)
+--max-streams N     Maximum streams TurboLane may use (default: 32)
+--model-dir DIR     Q-table persistence directory (default: models/dci)
+--interval SECS     RL decision interval in seconds (default: 5.0)
+--timeout SECS      Max wait for completion (default: unlimited)
+--verbose / -v      Debug logging
+```
+
+### 3. Query server status
+
+```bash
+turbolane-server status --host 192.168.1.50 --port 9000
+```
+
+---
+
+## How TurboLane integrates (5-second loop)
+
+```
+Every 5 seconds (TurboLaneAdapter._tick):
+  1. MetricsCollector.snapshot()
+       → throughput_mbps (sum of per-stream byte rates)
+       → rtt_ms          (mean of PING/PONG RTT samples)
+       → loss_pct        (chunk retransmit rate proxy)
+
+  2. engine.learn(throughput, rtt, loss)
+       → Q-table Bellman update from previous decision's outcome
+
+  3. engine.decide(throughput, rtt, loss)
+       → Q-learning ε-greedy action → new stream count
+
+  4. session.adjust_streams(new_count)
+       → spawn / stop StreamWorker threads to match recommendation
+```
+
+---
+
+## Future upgrades (designed-in hooks)
+
+| Capability           | Where to add              |
+|----------------------|---------------------------|
+| PPO algorithm        | `turbolane/rl/` only      |
+| Multi-session        | `server.py` busy logic    |
+| Shared policy learning | `adapter.py` FederatedPolicy |
+| Resume / checkpointing | `ChunkQueue` + `FileAssembler` |
+| TLS encryption       | `StreamWorker` + `StreamHandler` |

turbolane_cli-2.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

turbolane_cli-2.0.0/setup.py

@@ -0,0 +1,27 @@
+from pathlib import Path
+from setuptools import setup, find_packages
+
+this_dir = Path(__file__).parent
+readme = (this_dir / "README.md").read_text(encoding="utf-8")
+
+setup(
+    name="turbolane-cli",
+    version="2.0.0",
+    description="RL-optimized parallel TCP file transfer CLI (TurboLane)",
+    long_description=readme,
+    long_description_content_type="text/markdown",
+    packages=find_packages(exclude=["tests*"]),
+    python_requires=">=3.10",
+    install_requires=[],
+    entry_points={
+        "console_scripts": [
+            "turbolane-server=turbolane_server.cli:main",
+        ],
+    },
+    classifiers=[
+        "Development Status :: 3 - Alpha",
+        "Intended Audience :: Science/Research",
+        "Topic :: System :: Networking",
+        "Programming Language :: Python :: 3.10",
+    ],
+)

turbolane_cli-2.0.0/turbolane/__init__.py

@@ -0,0 +1,10 @@
+"""
+TurboLane — RL-based network optimization engine.
+
+Public API:
+    from turbolane import TurboLaneEngine
+"""
+from turbolane.engine import TurboLaneEngine
+
+__version__ = "1.0.0"
+__all__ = ["TurboLaneEngine"]

turbolane_cli-2.0.0/turbolane/engine.py

@@ -0,0 +1,218 @@
+"""
+turbolane/engine.py
+
+TurboLaneEngine — the single public entry point for the TurboLane SDK.
+
+This is the ONLY class that application code (or the adapter) should import.
+All policy selection, agent wiring, and mode routing happens here.
+
+Supported modes:
+    'dci'    → FederatedPolicy (data center / private network) ← current use
+    'client' → EdgePolicy      (public internet / edge)         ← future
+
+Supported algorithms:
+    'qlearning' → RLAgent      ← current use
+    'ppo'       → PPOAgent     ← future (drop-in via FederatedPolicy)
+
+Usage (DCI + Q-learning):
+    from turbolane.engine import TurboLaneEngine
+
+    engine = TurboLaneEngine(mode='dci', algorithm='qlearning')
+    streams = engine.decide(throughput_mbps, rtt_ms, loss_pct)
+    engine.learn(throughput_mbps, rtt_ms, loss_pct)
+    engine.save()
+"""
+
+import logging
+
+logger = logging.getLogger(__name__)
+
+# Supported modes and their policy classes
+_MODE_POLICY_MAP = {
+    "dci":    "turbolane.policies.federated.FederatedPolicy",
+    "client": "turbolane.policies.edge.EdgePolicy",        # future
+}
+
+_VALID_ALGORITHMS = {"qlearning", "ppo"}
+
+
+class TurboLaneEngine:
+    """
+    Unified TurboLane control-plane engine.
+
+    Public interface (identical regardless of mode or algorithm):
+        decide(throughput_mbps, rtt_ms, loss_pct)  → int
+        learn(throughput_mbps, rtt_ms, loss_pct)
+        save()                                      → bool
+        get_stats()                                 → dict
+        reset()
+
+    Convenience properties:
+        .current_connections                        → int
+        .mode                                       → str
+        .algorithm                                  → str
+    """
+
+    def __init__(
+        self,
+        mode: str = "dci",
+        algorithm: str = "qlearning",
+        **policy_kwargs,
+    ):
+        """
+        Initialize TurboLane engine.
+
+        Args:
+            mode:          'dci' or 'client'
+            algorithm:     'qlearning' or 'ppo'
+            **policy_kwargs: Passed directly to the policy constructor.
+                           See FederatedPolicy.__init__ for valid keys.
+
+        Example:
+            TurboLaneEngine(
+                mode='dci',
+                algorithm='qlearning',
+                model_dir='models/dci',
+                min_connections=1,
+                max_connections=32,
+                default_connections=4,
+                monitoring_interval=5.0,
+            )
+        """
+        mode = mode.lower()
+        algorithm = algorithm.lower().replace("-", "").replace("_", "")
+
+        if mode not in _MODE_POLICY_MAP:
+            raise ValueError(
+                f"Unknown mode '{mode}'. Valid modes: {list(_MODE_POLICY_MAP)}"
+            )
+        if algorithm not in _VALID_ALGORITHMS:
+            raise ValueError(
+                f"Unknown algorithm '{algorithm}'. Valid: {list(_VALID_ALGORITHMS)}"
+            )
+
+        self.mode = mode
+        self.algorithm = algorithm
+
+        # Instantiate the appropriate policy
+        self._policy = self._build_policy(mode, algorithm, policy_kwargs)
+
+        logger.info(
+            "TurboLaneEngine ready: mode=%s algorithm=%s",
+            self.mode, self.algorithm,
+        )
+
+    # -----------------------------------------------------------------------
+    # Core interface — these are the ONLY methods the adapter calls
+    # -----------------------------------------------------------------------
+
+    def decide(
+        self,
+        throughput_mbps: float,
+        rtt_ms: float,
+        loss_pct: float,
+    ) -> int:
+        """
+        Make a stream count recommendation based on current network metrics.
+
+        Args:
+            throughput_mbps: Observed throughput in Mbps
+            rtt_ms:          Observed round-trip time in milliseconds
+            loss_pct:        Observed packet loss in percent (0–100)
+
+        Returns:
+            Recommended number of parallel TCP streams (int)
+        """
+        return self._policy.decide(throughput_mbps, rtt_ms, loss_pct)
+
+    def learn(
+        self,
+        throughput_mbps: float,
+        rtt_ms: float,
+        loss_pct: float,
+    ) -> None:
+        """
+        Update the policy from the outcome of the previous decision.
+
+        Call this once per monitoring cycle, AFTER decide(), with
+        the metrics observed after the previous action took effect.
+
+        Args:
+            throughput_mbps: Current throughput in Mbps
+            rtt_ms:          Current RTT in milliseconds
+            loss_pct:        Current packet loss in percent (0–100)
+        """
+        self._policy.learn(throughput_mbps, rtt_ms, loss_pct)
+
+    def save(self) -> bool:
+        """
+        Persist the policy to disk.
+
+        Returns:
+            True on success, False on failure.
+        """
+        return self._policy.save()
+
+    def get_stats(self) -> dict:
+        """Return a stats dict for logging, monitoring, and CLI display."""
+        stats = self._policy.get_stats()
+        stats["engine_mode"] = self.mode
+        stats["engine_algorithm"] = self.algorithm
+        return stats
+
+    def sync_current_connections(self, observed_connections: int) -> int:
+        """
+        Synchronize the policy with the real applied stream count observed by
+        the transfer layer.
+        """
+        sync_fn = getattr(self._policy, "sync_current_connections", None)
+        if sync_fn is None:
+            return self.current_connections
+        return sync_fn(observed_connections)
+
+    def reset(self) -> None:
+        """Clear the policy's learned state. Does not delete files on disk."""
+        self._policy.reset()
+
+    # -----------------------------------------------------------------------
+    # Convenience properties
+    # -----------------------------------------------------------------------
+
+    @property
+    def current_connections(self) -> int:
+        """Current recommended stream count."""
+        return self._policy.current_connections
+
+    # -----------------------------------------------------------------------
+    # Internal factory
+    # -----------------------------------------------------------------------
+
+    def _build_policy(self, mode: str, algorithm: str, kwargs: dict):
+        """
+        Instantiate the correct policy for the given mode.
+
+        DCI mode: always uses FederatedPolicy, passes algorithm as a hint
+        for future PPO support. Currently FederatedPolicy only supports
+        Q-learning; PPO support will be added to FederatedPolicy directly.
+        """
+        if mode == "dci":
+            from turbolane.policies.federated import FederatedPolicy
+            return FederatedPolicy(**kwargs)
+
+        elif mode == "client":
+            # Future: EdgePolicy
+            raise NotImplementedError(
+                "Client/Edge mode is not yet implemented. "
+                "Use mode='dci' for data center transfers."
+            )
+
+        else:
+            raise ValueError(f"Unknown mode: {mode}")
+
+    def __repr__(self) -> str:
+        return (
+            f"TurboLaneEngine("
+            f"mode={self.mode!r}, "
+            f"algorithm={self.algorithm!r}, "
+            f"connections={self.current_connections})"
+        )

turbolane_cli-2.0.0/turbolane/policies/__init__.py

@@ -0,0 +1,4 @@
+"""turbolane/policies — environment-specific policy wrappers."""
+from turbolane.policies.federated import FederatedPolicy
+
+__all__ = ["FederatedPolicy"]