teyyare 0.2.7 → 0.2.9

package/README.md

@@ -26,7 +26,7 @@ Teyyare is the first real integration workload for both projects, proving that a
              replication / transport
                        │
                        ▼
-                      TCP
+                      TCP (Multi-Channel)
                        │
                        ▼
                  Remote Teyyare
@@ -37,7 +37,7 @@ Teyyare is the first real integration workload for both projects, proving that a
                      flush
                        │
                        ▼
-                     Disk
+                     Disk (writev + pre-allocation)
 ```
 
 ---
@@ -52,9 +52,9 @@ Teyyare is the first real integration workload for both projects, proving that a
 
 ---
 
-## Data Flow & Zero-Copy Architecture
+## Architecture & Systems Optimizations
 
-### Sender Hot Path
+### 1. Sender Hot Path
 ```text
 FileHandle.read()
       │
@@ -76,29 +76,46 @@ MutableGeneration
 FrozenGeneration [metaBuf, payloadArena]
       │
       ▼
-Raptiye.submitBatch() (scatter/gather sendv)
+Raptiye.submitBatch() (In-Flight Sliding Window up to 128 MB)
       │
       ▼
-TCP (corked writev)
+Multi-Channel TCP Pool (Zero-Allocation Round-Robin sendv)
 ```
 
-### Receiver Hot Path
+### 2. Receiver Hot Path
 ```text
-TCP
+Multi-Channel TCP
  │
  ▼
-Raptiye (frame decode + CRC32 verification)
+Zero-Copy Pointer Framer (O(1) cursor, no shift(), scratch header reuse)
+ │
+ ▼
+Raptiye (Frame decode + verification)
  │
  ▼
 FrozenGeneration (O(1) zero-copy view via FrozenArenaStore.fromBuffers)
  │
  ▼
-Teyyare chunk consumer
+Chunk Coalescing Engine (Merges contiguous chunk runs)
+ │
+ ▼
+FileHandle.writev(buffers, startOffset) (Direct native scatter-gather writes)
  │
  ▼
-FileHandle.write(payload, 0, length, offset) (Direct random-access write)
+Debounced Asynchronous State Persistence (.teyyare-state)
 ```
 
+### 3. Key Systems Optimizations Implemented
+
+- **Disk Pre-Allocation (`FileHandle.truncate(targetSize)`)**: Zero-fragmentation disk pre-allocation upon receiving `FILE_BEGIN`, eliminating filesystem dynamic block allocation latency during high-speed writes.
+- **Native Scatter-Gather Writes (`writev`)**: Receiver coalesces contiguous chunks arriving in the same generation batch into a single system call via `writev`, reducing write syscalls by ~87.5%.
+- **Debounced Asynchronous State Saving**: `.teyyare-state` persistence is debounced to disk (every 250ms or 32 chunks) and drained cleanly before final file rename, completely eliminating blocking JSON serialization from the hot-path.
+- **In-Flight ACK Pipelining (Sliding Window)**: Supports up to 32 concurrent batches and 128 MB in-flight capacity. Senders stream generations continuously across 4–8 parallel TCP channels without stalling on single-batch ACKs.
+- **Pointer-Based Zero-Copy TCP Framer**: Eliminated $O(N)$ array `shift()` operations and repeated `Buffer.allocUnsafe` calls on the TCP stream reader using an $O(1)$ cursor and amortized array compaction.
+- **Zero-Allocation Channel Selection**: Eliminated `pool.filter` garbage collector churn in `TCPTransport.prototype.sendv` and `send` using an in-place circular lookup.
+- **Dynamic WAN Tuning**: `PULL_REQUEST` carries client-specified `--chunk-size` (e.g. 2 MB) and `--batch-size` (e.g. 16 MB) to saturate high-latency, high-bandwidth WAN connections.
+- **Multi-Channel Out-of-Order Packet Resilience**: Resilient state reconciliation for `TRANSFER_BEGIN`, `FILE_BEGIN`, `FILE_END`, and `TRANSFER_END` packets arriving out-of-order across parallel TCP sockets.
+
 ---
 
 ## CLI Usage
@@ -109,72 +126,95 @@ FileHandle.write(payload, 0, length, offset) (Direct random-access write)
 teyyare serve --port 7421 --dir /destination/folder
 ```
 
-### Send File or Directory
+### Send File or Directory (Push)
 ```bash
-# Single file transfer
-teyyare send ./large-file.bin 127.0.0.1:7421:/tmp/large-file.bin
+# Single file transfer with 4 parallel channels
+teyyare send ./large-file.bin 127.0.0.1:7421:/tmp/large-file.bin -c 4
 
-# Directory transfer with nested files
-teyyare send ./dist server.local:7421:/var/www/app
+# Directory transfer with nested hierarchy
+teyyare send ./dist server.local:7421:/var/www/app -c 4
 ```
 
-### Progress Display
-```text
-Teyyare
+### Pull File or Directory (Download)
+```bash
+# Single file download from remote server with 8 channels and 2MB WAN chunks
+teyyare pull 38.242.216.102:7421:/root/cdn/app.zip ./ -c 8 --chunk-size 2M --batch-size 16M
+
+# Directory pull
+teyyare pull server:7421:/backup/db.tar.zst ./ -c 4
+```
+
+### CLI Options
+
+| Flag | Description | Default |
+|---|---|---|
+| `-c, --channels <n>` | Number of parallel TCP socket channels | `4` (remote), `1` (local) |
+| `--chunk-size <size>` | Chunk size, e.g. `512K`, `1M`, `2M`, `4M` | `2M` (remote), `1M` (local) |
+| `--batch-size <size>` | Generation batch size, e.g. `8M`, `16M` | `8x chunk size` |
+| `--port <port>` | Server port number | `7421` |
+| `--dir <path>` | Root directory for `serve` command | Current directory (`./`) |
+
+---
 
-backup.tar.zst
+## Progress Display
 
-6.42 / 10.00 GB
-64.2%
+```text
+Teyyare [Download]
+
+app.zip
 
-throughput       312 MB/s
-ETA              00:11
+333.54 MB / 333.54 MB
+100.0%
 
-chunks           6574 / 10240
-generation       822
-inflight gen     4
-inflight bytes   31.8 MB
+network speed    324.5 MB/s
+disk speed       1240.2 MB/s
+channels         8
+ETA              --:--
 
-muttafa queue    2
-raptiye RTT      18 ms
-TCP pressure     0
+chunks           167 / 167
+generation       21
+verified CRC32   167
+resume hits      0
 ```
 
 ---
 
-## Benchmarks & Ablation Analysis
+## Test Suite & Softscope Profiling
 
-Run the benchmark suite with:
+### 1. Unit & Integration Tests
+Run all 24 automated tests covering protocol encoding, generation ownership, backpressure, chaos recovery, and multi-channel transfers:
 ```bash
-npm run bench
+npm test
 ```
 
-### 1. Layer Ablation (50 MB transfer)
-| Layer | Description | Throughput (MB/s) | Duration |
-|---|---|---|---|
-| **A** | raw fs -> TCP -> fs | **581.4 MB/s** | 86ms |
-| **B** | fs -> Muttafa -> TCP -> fs | **306.7 MB/s** | 163ms |
-| **C** | fs -> Raptiye -> fs | **85.5 MB/s** | 585ms |
-| **D** | Teyyare (Muttafa + Raptiye + fs) | **38.0 MB/s** | 1320ms |
+### 2. End-to-End (E2E) Test Suite
+Runs 4 comprehensive end-to-end scenarios (Single Upload, Directory Upload, Single Pull, 50% Resumed Transfer):
+```bash
+npm run e2e
+```
 
-### 2. Chunk & Generation Size Matrix (16 MB transfer)
-| Chunk Size | Generation Size | Throughput (MB/s) | Duration (ms) | Peak RSS (MB) |
-|---|---|---|---|---|
-| **64 KiB** | 4 MiB | 39.7 MB/s | 403ms | 660.8 MB |
-| **256 KiB** | 4 MiB | 34.9 MB/s | 458ms | 679.6 MB |
-| **1 MiB** | 4 MiB | **40.1 MB/s** | 399ms | 757.7 MB |
-| **1 MiB** | 8 MiB | **40.0 MB/s** | 400ms | 718.2 MB |
-| **4 MiB** | 16 MiB | 38.6 MB/s | 414ms | 745.3 MB |
+### 3. Softscope Profiling
+Profile CPU bottlenecks, hot call paths, and memory allocations using [`softscope`](https://github.com/litepacks/softscope):
+```bash
+# Markdown summary to stdout
+npm run e2e:md
+
+# Full interactive terminal dashboard (TUI)
+npm run e2e:profile
+
+# Live metrics during execution
+npm run e2e:live
+```
 
 ---
 
-## Test Suite
+## Benchmark & Throughput Results
 
-Run the full test suite with:
-```bash
-npm test
-```
+### End-to-End Suite Summary (`test/e2e/e2e.js`)
 
-Includes:
-- **Unit Tests**: Packed binary chunk metadata protocol, resume state bitmap, generation lifecycle & ownership assertions.
-- **Integration Tests**: 10 MB single file transfer, directory transfer with small-file packing, resume after 50% interruption with bounded retransmission, slow receiver disk write backpressure with bounded RSS plateau, and deterministic chaos recovery.
+| Scenario | Transfer Size | Duration | Throughput | Verification |
+|---|---|---|---|---|
+| **Single File Upload** (4 channels) | 32.0 MB | 121ms | **264.5 MB/s** | SHA-256 Match |
+| **Directory Upload** (Nested tree) | 16.0 MB | 104ms | **153.9 MB/s** | All files verified |
+| **Single File Pull** (4 channels) | 32.0 MB | 89ms | **359.6 MB/s** | SHA-256 Match |
+| **Resumed Transfer** (50% skipped) | 8.0 MB | 49ms | **163.3 MB/s** | 0 retransmitted bytes |

package/bin/teyyare.js

@@ -19,6 +19,8 @@ function printUsage() {
 
 \x1b[1mOPTIONS:\x1b[0m
   -c, --channels <n>    Parallel TCP connections (default: 4 for remote, 1 for local)
+  --chunk-size <size>   Chunk size, e.g. 512K, 1M, 2M, 4M (default: 2M remote, 1M local)
+  --batch-size <size>   Generation batch size, e.g. 8M, 16M (default: 8x chunk size)
   --port <port>         Port number (default: 7421)
   --dir <path>          Serve root directory (default: current dir)
 
@@ -30,6 +32,15 @@ function printUsage() {
 `);
 }
 
+function parseByteSize(str) {
+  if (!str) return undefined;
+  const s = String(str).trim().toLowerCase();
+  if (s.endsWith('k') || s.endsWith('kb')) return Math.round(parseFloat(s) * 1024);
+  if (s.endsWith('m') || s.endsWith('mb')) return Math.round(parseFloat(s) * 1024 * 1024);
+  if (s.endsWith('g') || s.endsWith('gb')) return Math.round(parseFloat(s) * 1024 * 1024 * 1024);
+  return Number(str);
+}
+
 function parseDestination(target) {
   // Matches host:port:path or host:path or local path
   if (!target.includes(':')) {
@@ -50,13 +61,13 @@ async function main() {
   const args = process.argv.slice(2);
   const command = args[0];
 
-  if (!command || command === '--help' || command === '-h') {
+  if (!command || args.includes('--help') || args.includes('-h')) {
     printUsage();
     process.exit(0);
   }
 
   if (command === '--version' || command === '-v' || command === 'version') {
-    console.log('0.2.7');
+    console.log('0.2.8');
     process.exit(0);
   }
 
@@ -88,10 +99,16 @@ async function main() {
     await new Promise(() => {});
   } else if (command === 'send') {
     let channels = undefined;
+    let chunkSize = undefined;
+    let batchSize = undefined;
     const positional = [];
     for (let i = 1; i < args.length; i++) {
       if ((args[i] === '-c' || args[i] === '--channels') && args[i + 1]) {
         channels = Number(args[++i]);
+      } else if ((args[i] === '--chunk-size' || args[i] === '--chunk') && args[i + 1]) {
+        chunkSize = parseByteSize(args[++i]);
+      } else if ((args[i] === '--batch-size' || args[i] === '--gen-size') && args[i + 1]) {
+        batchSize = parseByteSize(args[++i]);
       } else {
         positional.push(args[i]);
       }
@@ -106,12 +123,17 @@ async function main() {
       process.exit(1);
     }
 
+    const pipelineOptions = {};
+    if (chunkSize) pipelineOptions.chunkSize = chunkSize;
+    if (batchSize) pipelineOptions.maxBytes = batchSize;
+
     const { host, port } = parseDestination(target);
     const metrics = await send({
       sourcePath: source,
       host,
       port,
       channels,
+      pipelineOptions,
       showProgress: true
     });
 
@@ -129,10 +151,16 @@ Peak RSS:         ${(snap.peakRSS / (1024 * 1024)).toFixed(2)} MB
     process.exit(0);
   } else if (command === 'pull') {
     let channels = undefined;
+    let chunkSize = undefined;
+    let batchSize = undefined;
     const positional = [];
     for (let i = 1; i < args.length; i++) {
       if ((args[i] === '-c' || args[i] === '--channels') && args[i + 1]) {
         channels = Number(args[++i]);
+      } else if ((args[i] === '--chunk-size' || args[i] === '--chunk') && args[i + 1]) {
+        chunkSize = parseByteSize(args[++i]);
+      } else if ((args[i] === '--batch-size' || args[i] === '--gen-size') && args[i + 1]) {
+        batchSize = parseByteSize(args[++i]);
       } else {
         positional.push(args[i]);
       }
@@ -147,6 +175,10 @@ Peak RSS:         ${(snap.peakRSS / (1024 * 1024)).toFixed(2)} MB
       process.exit(1);
     }
 
+    const pipelineOptions = {};
+    if (chunkSize) pipelineOptions.chunkSize = chunkSize;
+    if (batchSize) pipelineOptions.maxBytes = batchSize;
+
     const { host, port, destPath: remotePath } = parseDestination(source);
     const metrics = await pull({
       sourceRemotePath: remotePath,
@@ -154,6 +186,7 @@ Peak RSS:         ${(snap.peakRSS / (1024 * 1024)).toFixed(2)} MB
       port,
       channels,
       destinationDir: target,
+      pipelineOptions,
       showProgress: true
     });
 
@@ -170,7 +203,7 @@ Peak RSS:         ${(snap.peakRSS / (1024 * 1024)).toFixed(2)} MB
 `);
     process.exit(0);
   } else if (command === 'status') {
-    console.log('\x1b[1m\x1b[36mTeyyare\x1b[0m v0.2.7 status: ready');
+    console.log('\x1b[1m\x1b[36mTeyyare\x1b[0m v0.2.8 status: ready');
   } else {
     console.error(`Unknown command: ${command}`);
     printUsage();

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "teyyare",
-  "version": "0.2.7",
+  "version": "0.2.9",
   "description": "High-performance resumable file transfer engine and CLI built on muttafa and raptiye",
   "main": "src/index.js",
   "type": "module",
@@ -9,7 +9,11 @@
   },
   "scripts": {
     "test": "node --test test/**/*.test.js",
-    "bench": "node bench/run-all.js"
+    "bench": "node bench/run-all.js",
+    "e2e": "node test/e2e/e2e.js",
+    "e2e:profile": "softscope test/e2e/e2e.js",
+    "e2e:live": "softscope --live test/e2e/e2e.js",
+    "e2e:md": "softscope --md test/e2e/e2e.js"
   },
   "keywords": [
     "file-transfer",

package/src/core/protocol.js

@@ -76,10 +76,10 @@ export function unpackChunkMeta(buf, offset = 0) {
  * 14..45: sha256 hex (32 bytes raw or 64 hex chars)
  * 46..:   path utf8
  */
-export function packFileBegin({ fileId, fileSize, path, sha256 = '' }) {
+export function packFileBegin({ fileId, fileSize, path, sha256 = '', chunkSize = 1048576 }) {
   const pathBytes = encoder.encode(path);
   const shaBytes = encoder.encode(sha256.padEnd(64, '0').slice(0, 64));
-  const total = 14 + 64 + pathBytes.byteLength;
+  const total = 14 + 64 + pathBytes.byteLength + 4;
   const buf = new Uint8Array(total);
   const view = new DataView(buf.buffer, buf.byteOffset, total);
 
@@ -88,6 +88,7 @@ export function packFileBegin({ fileId, fileSize, path, sha256 = '' }) {
   view.setUint16(12, pathBytes.byteLength, false);
   buf.set(shaBytes, 14);
   buf.set(pathBytes, 14 + 64);
+  view.setUint32(14 + 64 + pathBytes.byteLength, chunkSize >>> 0, false);
   return buf;
 }
 
@@ -98,7 +99,12 @@ export function unpackFileBegin(buf) {
   const pathLen = view.getUint16(12, false);
   const sha256 = decoder.decode(buf.subarray(14, 14 + 64)).replace(/0+$/, '');
   const path = decoder.decode(buf.subarray(14 + 64, 14 + 64 + pathLen));
-  return { fileId, fileSize, path, sha256 };
+  let chunkSize = 1048576;
+  if (buf.byteLength >= 14 + 64 + pathLen + 4) {
+    const cs = view.getUint32(14 + 64 + pathLen, false);
+    if (cs > 0) chunkSize = cs;
+  }
+  return { fileId, fileSize, path, sha256, chunkSize };
 }
 
 /**
@@ -167,10 +173,18 @@ export function unpackTransferEnd(buf) {
 /**
  * Pack PULL_REQUEST metadata with optional verifiedChunks bitmap/list for resume.
  */
-export function packPullRequest({ remotePath, verifiedChunks = [] }) {
+export function packPullRequest({
+  remotePath,
+  verifiedChunks = [],
+  chunkSize = 0,
+  maxEntries = 0,
+  maxBytes = 0
+}) {
   const pathBytes = encoder.encode(remotePath);
   const count = verifiedChunks.length;
-  const buf = new Uint8Array(2 + pathBytes.byteLength + 4 + count * 4);
+  const hasTuning = (chunkSize > 0 || maxEntries > 0 || maxBytes > 0);
+  const tuningLen = hasTuning ? 10 : 0;
+  const buf = new Uint8Array(2 + pathBytes.byteLength + 4 + count * 4 + tuningLen);
   const view = new DataView(buf.buffer, buf.byteOffset, buf.byteLength);
   view.setUint16(0, pathBytes.byteLength, false);
   buf.set(pathBytes, 2);
@@ -182,6 +196,13 @@ export function packPullRequest({ remotePath, verifiedChunks = [] }) {
     view.setUint32(off, verifiedChunks[i], false);
     off += 4;
   }
+
+  if (hasTuning) {
+    view.setUint32(off, chunkSize >>> 0, false);
+    view.setUint16(off + 4, maxEntries & 0xffff, false);
+    view.setUint32(off + 6, maxBytes >>> 0, false);
+  }
+
   return buf;
 }
 
@@ -201,7 +222,19 @@ export function unpackPullRequest(buf) {
     }
   }
 
-  return { remotePath, verifiedChunks };
+  let chunkSize = undefined;
+  let maxEntries = undefined;
+  let maxBytes = undefined;
+  if (buf.byteLength >= off + 10) {
+    const cs = view.getUint32(off, false);
+    const me = view.getUint16(off + 4, false);
+    const mb = view.getUint32(off + 6, false);
+    if (cs > 0) chunkSize = cs;
+    if (me > 0) maxEntries = me;
+    if (mb > 0) maxBytes = mb;
+  }
+
+  return { remotePath, verifiedChunks, chunkSize, maxEntries, maxBytes };
 }
 
 /**