teyyare 0.2.8 → 0.2.9

package/README.md

@@ -26,7 +26,7 @@ Teyyare is the first real integration workload for both projects, proving that a
              replication / transport
                        │
                        ▼
-                      TCP
+                      TCP (Multi-Channel)
                        │
                        ▼
                  Remote Teyyare
@@ -37,7 +37,7 @@ Teyyare is the first real integration workload for both projects, proving that a
                      flush
                        │
                        ▼
-                     Disk
+                     Disk (writev + pre-allocation)
 ```
 
 ---
@@ -52,9 +52,9 @@ Teyyare is the first real integration workload for both projects, proving that a
 
 ---
 
-## Data Flow & Zero-Copy Architecture
+## Architecture & Systems Optimizations
 
-### Sender Hot Path
+### 1. Sender Hot Path
 ```text
 FileHandle.read()
       │
@@ -76,29 +76,46 @@ MutableGeneration
 FrozenGeneration [metaBuf, payloadArena]
       │
       ▼
-Raptiye.submitBatch() (scatter/gather sendv)
+Raptiye.submitBatch() (In-Flight Sliding Window up to 128 MB)
       │
       ▼
-TCP (corked writev)
+Multi-Channel TCP Pool (Zero-Allocation Round-Robin sendv)
 ```
 
-### Receiver Hot Path
+### 2. Receiver Hot Path
 ```text
-TCP
+Multi-Channel TCP
  │
  ▼
-Raptiye (frame decode + CRC32 verification)
+Zero-Copy Pointer Framer (O(1) cursor, no shift(), scratch header reuse)
+ │
+ ▼
+Raptiye (Frame decode + verification)
  │
  ▼
 FrozenGeneration (O(1) zero-copy view via FrozenArenaStore.fromBuffers)
  │
  ▼
-Teyyare chunk consumer
+Chunk Coalescing Engine (Merges contiguous chunk runs)
+ │
+ ▼
+FileHandle.writev(buffers, startOffset) (Direct native scatter-gather writes)
  │
  ▼
-FileHandle.write(payload, 0, length, offset) (Direct random-access write)
+Debounced Asynchronous State Persistence (.teyyare-state)
 ```
 
+### 3. Key Systems Optimizations Implemented
+
+- **Disk Pre-Allocation (`FileHandle.truncate(targetSize)`)**: Zero-fragmentation disk pre-allocation upon receiving `FILE_BEGIN`, eliminating filesystem dynamic block allocation latency during high-speed writes.
+- **Native Scatter-Gather Writes (`writev`)**: Receiver coalesces contiguous chunks arriving in the same generation batch into a single system call via `writev`, reducing write syscalls by ~87.5%.
+- **Debounced Asynchronous State Saving**: `.teyyare-state` persistence is debounced to disk (every 250ms or 32 chunks) and drained cleanly before final file rename, completely eliminating blocking JSON serialization from the hot-path.
+- **In-Flight ACK Pipelining (Sliding Window)**: Supports up to 32 concurrent batches and 128 MB in-flight capacity. Senders stream generations continuously across 4–8 parallel TCP channels without stalling on single-batch ACKs.
+- **Pointer-Based Zero-Copy TCP Framer**: Eliminated $O(N)$ array `shift()` operations and repeated `Buffer.allocUnsafe` calls on the TCP stream reader using an $O(1)$ cursor and amortized array compaction.
+- **Zero-Allocation Channel Selection**: Eliminated `pool.filter` garbage collector churn in `TCPTransport.prototype.sendv` and `send` using an in-place circular lookup.
+- **Dynamic WAN Tuning**: `PULL_REQUEST` carries client-specified `--chunk-size` (e.g. 2 MB) and `--batch-size` (e.g. 16 MB) to saturate high-latency, high-bandwidth WAN connections.
+- **Multi-Channel Out-of-Order Packet Resilience**: Resilient state reconciliation for `TRANSFER_BEGIN`, `FILE_BEGIN`, `FILE_END`, and `TRANSFER_END` packets arriving out-of-order across parallel TCP sockets.
+
 ---
 
 ## CLI Usage
@@ -109,72 +126,95 @@ FileHandle.write(payload, 0, length, offset) (Direct random-access write)
 teyyare serve --port 7421 --dir /destination/folder
 ```
 
-### Send File or Directory
+### Send File or Directory (Push)
 ```bash
-# Single file transfer
-teyyare send ./large-file.bin 127.0.0.1:7421:/tmp/large-file.bin
+# Single file transfer with 4 parallel channels
+teyyare send ./large-file.bin 127.0.0.1:7421:/tmp/large-file.bin -c 4
 
-# Directory transfer with nested files
-teyyare send ./dist server.local:7421:/var/www/app
+# Directory transfer with nested hierarchy
+teyyare send ./dist server.local:7421:/var/www/app -c 4
 ```
 
-### Progress Display
-```text
-Teyyare
+### Pull File or Directory (Download)
+```bash
+# Single file download from remote server with 8 channels and 2MB WAN chunks
+teyyare pull 38.242.216.102:7421:/root/cdn/app.zip ./ -c 8 --chunk-size 2M --batch-size 16M
+
+# Directory pull
+teyyare pull server:7421:/backup/db.tar.zst ./ -c 4
+```
+
+### CLI Options
+
+| Flag | Description | Default |
+|---|---|---|
+| `-c, --channels <n>` | Number of parallel TCP socket channels | `4` (remote), `1` (local) |
+| `--chunk-size <size>` | Chunk size, e.g. `512K`, `1M`, `2M`, `4M` | `2M` (remote), `1M` (local) |
+| `--batch-size <size>` | Generation batch size, e.g. `8M`, `16M` | `8x chunk size` |
+| `--port <port>` | Server port number | `7421` |
+| `--dir <path>` | Root directory for `serve` command | Current directory (`./`) |
+
+---
 
-backup.tar.zst
+## Progress Display
 
-6.42 / 10.00 GB
-64.2%
+```text
+Teyyare [Download]
+
+app.zip
 
-throughput       312 MB/s
-ETA              00:11
+333.54 MB / 333.54 MB
+100.0%
 
-chunks           6574 / 10240
-generation       822
-inflight gen     4
-inflight bytes   31.8 MB
+network speed    324.5 MB/s
+disk speed       1240.2 MB/s
+channels         8
+ETA              --:--
 
-muttafa queue    2
-raptiye RTT      18 ms
-TCP pressure     0
+chunks           167 / 167
+generation       21
+verified CRC32   167
+resume hits      0
 ```
 
 ---
 
-## Benchmarks & Ablation Analysis
+## Test Suite & Softscope Profiling
 
-Run the benchmark suite with:
+### 1. Unit & Integration Tests
+Run all 24 automated tests covering protocol encoding, generation ownership, backpressure, chaos recovery, and multi-channel transfers:
 ```bash
-npm run bench
+npm test
 ```
 
-### 1. Layer Ablation (50 MB transfer)
-| Layer | Description | Throughput (MB/s) | Duration |
-|---|---|---|---|
-| **A** | raw fs -> TCP -> fs | **581.4 MB/s** | 86ms |
-| **B** | fs -> Muttafa -> TCP -> fs | **306.7 MB/s** | 163ms |
-| **C** | fs -> Raptiye -> fs | **85.5 MB/s** | 585ms |
-| **D** | Teyyare (Muttafa + Raptiye + fs) | **38.0 MB/s** | 1320ms |
+### 2. End-to-End (E2E) Test Suite
+Runs 4 comprehensive end-to-end scenarios (Single Upload, Directory Upload, Single Pull, 50% Resumed Transfer):
+```bash
+npm run e2e
+```
 
-### 2. Chunk & Generation Size Matrix (16 MB transfer)
-| Chunk Size | Generation Size | Throughput (MB/s) | Duration (ms) | Peak RSS (MB) |
-|---|---|---|---|---|
-| **64 KiB** | 4 MiB | 39.7 MB/s | 403ms | 660.8 MB |
-| **256 KiB** | 4 MiB | 34.9 MB/s | 458ms | 679.6 MB |
-| **1 MiB** | 4 MiB | **40.1 MB/s** | 399ms | 757.7 MB |
-| **1 MiB** | 8 MiB | **40.0 MB/s** | 400ms | 718.2 MB |
-| **4 MiB** | 16 MiB | 38.6 MB/s | 414ms | 745.3 MB |
+### 3. Softscope Profiling
+Profile CPU bottlenecks, hot call paths, and memory allocations using [`softscope`](https://github.com/litepacks/softscope):
+```bash
+# Markdown summary to stdout
+npm run e2e:md
+
+# Full interactive terminal dashboard (TUI)
+npm run e2e:profile
+
+# Live metrics during execution
+npm run e2e:live
+```
 
 ---
 
-## Test Suite
+## Benchmark & Throughput Results
 
-Run the full test suite with:
-```bash
-npm test
-```
+### End-to-End Suite Summary (`test/e2e/e2e.js`)
 
-Includes:
-- **Unit Tests**: Packed binary chunk metadata protocol, resume state bitmap, generation lifecycle & ownership assertions.
-- **Integration Tests**: 10 MB single file transfer, directory transfer with small-file packing, resume after 50% interruption with bounded retransmission, slow receiver disk write backpressure with bounded RSS plateau, and deterministic chaos recovery.
+| Scenario | Transfer Size | Duration | Throughput | Verification |
+|---|---|---|---|---|
+| **Single File Upload** (4 channels) | 32.0 MB | 121ms | **264.5 MB/s** | SHA-256 Match |
+| **Directory Upload** (Nested tree) | 16.0 MB | 104ms | **153.9 MB/s** | All files verified |
+| **Single File Pull** (4 channels) | 32.0 MB | 89ms | **359.6 MB/s** | SHA-256 Match |
+| **Resumed Transfer** (50% skipped) | 8.0 MB | 49ms | **163.3 MB/s** | 0 retransmitted bytes |

package/bin/teyyare.js

@@ -61,13 +61,13 @@ async function main() {
   const args = process.argv.slice(2);
   const command = args[0];
 
-  if (!command || command === '--help' || command === '-h') {
+  if (!command || args.includes('--help') || args.includes('-h')) {
     printUsage();
     process.exit(0);
   }
 
   if (command === '--version' || command === '-v' || command === 'version') {
-    console.log('0.2.7');
+    console.log('0.2.8');
     process.exit(0);
   }
 
@@ -203,7 +203,7 @@ Peak RSS:         ${(snap.peakRSS / (1024 * 1024)).toFixed(2)} MB
 `);
     process.exit(0);
   } else if (command === 'status') {
-    console.log('\x1b[1m\x1b[36mTeyyare\x1b[0m v0.2.7 status: ready');
+    console.log('\x1b[1m\x1b[36mTeyyare\x1b[0m v0.2.8 status: ready');
   } else {
     console.error(`Unknown command: ${command}`);
     printUsage();

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "teyyare",
-  "version": "0.2.8",
+  "version": "0.2.9",
   "description": "High-performance resumable file transfer engine and CLI built on muttafa and raptiye",
   "main": "src/index.js",
   "type": "module",

package/src/receiver/pipeline.js

@@ -421,7 +421,7 @@ export class ReceiverPipeline {
     }
 
     // 2. Sort by offset to form maximal contiguous runs
-    toWrite.sort((a, b) => Number(a.meta.offset - b.meta.offset));
+    toWrite.sort((a, b) => (a.meta.offset < b.meta.offset ? -1 : (a.meta.offset > b.meta.offset ? 1 : 0)));
 
     let currentRun = [toWrite[0]];
     let currentStartOffset = Number(toWrite[0].meta.offset);

package/src/sender/pipeline.js

@@ -20,6 +20,25 @@ import {
 import { extractGenerationBuffers } from '../core/generation.js';
 import { FileResumeState } from '../resume/state.js';
 
+/**
+ * Reusable bump allocator that returns its ArrayBuffer to a pool upon reset/release.
+ */
+class PooledBumpAllocator extends BumpAllocator {
+  constructor(capacity, pool = null) {
+    super(capacity);
+    this._pool = pool;
+  }
+
+  reset() {
+    super.reset();
+    if (this._pool && this._pool.length < 16 && !this._pool.includes(this)) {
+      this._pool.push(this);
+    }
+  }
+}
+
+const defaultAllocatorPool = [];
+
 class RaptiyeSink extends Sink {
   /**
    * @param {import('raptiye').ReplicationPipeline} replication
@@ -109,6 +128,8 @@ export class SenderPipeline {
     this.metrics = metrics;
     this.progress = progress;
     this.knownResumeState = knownResumeState;
+    this._allocatorPool = options.allocatorPool || defaultAllocatorPool;
+    this._readScratch = null;
 
     this.transferId = crypto.randomUUID();
 
@@ -119,10 +140,22 @@ export class SenderPipeline {
     this.sink = new RaptiyeSink(this.replication, this.metrics, this.progress);
     this.engine = new MutationEngine({
       sink: this.sink,
-      createStore: () => new ArenaStore({
-        chunkSize: Math.max(64, this.maxEntries),
-        allocator: new BumpAllocator(arenaCapacity)
-      }),
+      createStore: () => {
+        let allocator = null;
+        if (this._allocatorPool && this._allocatorPool.length > 0) {
+          const idx = this._allocatorPool.findIndex((a) => a.capacity >= arenaCapacity);
+          if (idx !== -1) {
+            allocator = this._allocatorPool.splice(idx, 1)[0];
+          }
+        }
+        if (!allocator) {
+          allocator = new PooledBumpAllocator(arenaCapacity, this._allocatorPool);
+        }
+        return new ArenaStore({
+          chunkSize: Math.max(64, this.maxEntries),
+          allocator
+        });
+      },
       maxEntries: this.maxEntries,
       maxBytes: this.maxBytes,
       maxImmutableBuffers: this.maxImmutableBuffers,
@@ -148,6 +181,7 @@ export class SenderPipeline {
 
   async abort() {
     this.aborted = true;
+    this._readScratch = null;
     for (const waiter of this._abortWaiters) {
       waiter();
     }
@@ -264,8 +298,14 @@ export class SenderPipeline {
               continue;
             }
 
-            // Direct read into fresh chunk buffer
-            const chunkBuf = new Uint8Array(chunkLen);
+            // Direct read into reusable chunk buffer (Muttafa copies synchronously into its arena on append)
+            if (!this._readScratch || this._readScratch.byteLength < chunkLen) {
+              this._readScratch = new Uint8Array(Math.max(this.chunkSize, chunkLen));
+            }
+            const chunkBuf = (chunkLen === this._readScratch.byteLength)
+              ? this._readScratch
+              : this._readScratch.subarray(0, chunkLen);
+
             const readStart = Date.now();
             await handle.read(chunkBuf, 0, chunkLen, Number(offset));
             if (this.metrics) {
@@ -340,5 +380,6 @@ export class SenderPipeline {
       });
       this.progress.stop();
     }
+    this._readScratch = null;
   }
 }