omnius 1.0.267 → 1.0.269

package/docs/operations/delay-analysis.md

@@ -0,0 +1,230 @@
+# Unnecessary Causes of Delays Between Agent Actions
+
+Analysis of `packages/orchestrator/src/` (95 files) — identified delay sources ranked by impact.
+
+---
+
+## HIGH IMPACT
+
+### 1. `streaming-executor.ts:171-192` — `waitAll()` busy-poll loop
+
+```typescript
+async waitAll(): Promise<void> {
+  while (true) {
+    const pending = Array.from(this.tools.values()).filter(
+      e => e.state === "queued" || e.state === "executing"
+    );
+    if (pending.length === 0) break;
+    const executing = pending.filter(e => e.promise);
+    if (executing.length > 0) {
+      await Promise.allSettled(executing.map(e => e.promise!));
+      this.processQueue();
+    } else {
+      this.processQueue();
+      await new Promise(r => setTimeout(r, 1));  // ← 1ms spin loop
+    }
+  }
+}
+```
+
+**Problem:** When tools are queued but none have started (no `promise`), the loop spins with 1ms `setTimeout` between `processQueue()` calls. Each iteration costs at least 1ms of wall-clock delay. With many queued-but-blocked tools, this adds up.
+
+**Fix:** Replace with Promise-based notification (event emitter or `AbortController`-based wait) so the loop yields until a tool completes or a new tool is enqueued.
+
+---
+
+### 2. `ollama-pool.ts:850-869` — Instance readiness probe loop
+
+```typescript
+await new Promise((r) => setTimeout(r, 500));  // ← 500ms between probes
+```
+
+**Problem:** When spawning a new Ollama instance, the code polls `/api/version` every **500ms** with a 2-second abort timeout. This adds **500-2000ms** of delay per new model instance. If the pool is empty and a model isn't loaded, every first request pays this cost.
+
+**Fix:** Use exponential backoff (100ms → 200ms → 400ms) or a readiness event emitter instead of fixed polling.
+
+---
+
+### 3. `cascadeBackend.ts:105-156` — Sequential fallback chain
+
+```typescript
+const result = await backend.chatCompletion(request);
+// On failure → try newBackend.chatCompletion(request);
+// On failure → try primaryBackend.chatCompletion({ timeoutMs: 10_000 });
+```
+
+**Problem:** The cascade backend tries multiple backends in sequence. Each failure triggers a retry on the next backend. If the first backend is slow to fail (not crash), the full timeout (10s+) is paid before the fallback activates.
+
+**Fix:** Start all backends simultaneously with `Promise.race()` and cancel losers on first success.
+
+---
+
+## MEDIUM IMPACT
+
+### 4. `ollama-pool.ts:1190` — Unbounded slot waiter queue
+
+```typescript
+await new Promise<void>((resolve) => this.slotWaiters.push(resolve));
+```
+
+**Problem:** When all GPU slots are occupied, new requests queue as promises in `slotWaiters`. They only resolve when a slot frees up — but there's no timeout or progress feedback. Under heavy load, this can stall indefinitely.
+
+**Fix:** Add a configurable timeout (e.g., 30s) and reject with a clear "no GPU available" error.
+
+---
+
+### 5. `ollama-pool.ts:1336` — GPU detection on every acquire
+
+```typescript
+const rawGpus = await this.gpuDetector();
+```
+
+**Problem:** GPU detection runs on every placement decision. If `gpuDetector()` calls `nvidia-smi` or similar system commands, this adds **10-100ms** per request.
+
+**Fix:** Cache GPU detection results with a TTL (e.g., 30s). GPU topology rarely changes at runtime.
+
+---
+
+### 6. `tool-batching.ts:247-253` — Forced serial execution of non-concurrent tools
+
+```typescript
+for (const call of batch.calls) {
+  results.push(await executeFn(call));
+}
+```
+
+**Problem:** Write/shell tools are forced to run serially. If the agent produces 3+ write tools in one turn, they queue sequentially. No parallelism for independent writes.
+
+**Fix:** Add a write-conflict analyzer (e.g., file-level locking) to allow parallel writes to different files.
+
+---
+
+### 7. `steeringIntake.ts:97` — 15s steering timeout
+
+```typescript
+timeoutMs = 15_000,
+```
+
+**Problem:** Steering intake has a 15-second timeout. If the model is slow or the prompt is large, this adds a full 15s delay before the steering response arrives.
+
+**Fix:** Reduce to 5-8s for steering (which is advisory, not critical) or make it non-blocking with a best-effort result.
+
+---
+
+### 8. `verifierRunner.ts:161` — 60s test timeout
+
+```typescript
+timeout: 60_000,
+```
+
+**Problem:** Test execution has a 60-second timeout. If tests hang or are slow, this blocks the entire agent loop.
+
+**Fix:** Add a per-test timeout and a total timeout with early termination on repeated failures.
+
+---
+
+## LOW-MEDIUM IMPACT
+
+### 9. `ollama-pool.ts:1236,1257` — VRAM estimation per model
+
+```typescript
+const vramNeededMB = await this.estimateModelVramMB(model);
+```
+
+**Problem:** VRAM estimation runs before every spawn/placement. If it involves model metadata lookups or API calls, this adds latency.
+
+**Fix:** Cache VRAM estimates keyed by model name with a TTL.
+
+---
+
+### 10. `streaming-executor.ts:21-26` — `stableValueKey` deep serialization
+
+```typescript
+function stableValueKey(value: unknown): string {
+  if (value === null || typeof value !== "object") return JSON.stringify(value);
+  if (Array.isArray(value)) return `[${value.map(stableValueKey).join(",")}]`;
+  const record = value as Record<string, unknown>;
+  return `{${Object.keys(record).sort().map((key) => ...).join(",")}}`;
+}
+```
+
+**Problem:** Deep serialization of tool arguments for deduplication. For large args (e.g., file contents), this adds O(n) serialization cost per tool call.
+
+**Fix:** Add a size cap (e.g., 10KB) — if the value exceeds it, use a hash instead of deep serialization.
+
+---
+
+### 11. `streaming-executor.ts:263-291` — Duplicate detection overhead
+
+```typescript
+private findPriorEquivalent(entry: StreamingToolEntry): StreamingToolEntry | null
+private cloneDuplicateResult(entry: StreamingToolEntry): boolean
+private mirrorPriorEquivalent(entry: StreamingToolEntry): boolean
+```
+
+**Problem:** Every tool call is checked against all prior calls for equivalence. This is O(n²) in the number of tools per turn.
+
+**Fix:** Use a hash-based index (tool name + arg hash) for O(1) lookup instead of linear scan.
+
+---
+
+### 12. `ollama-pool.ts:1697` — Stale process cleanup timer
+
+```typescript
+const handle = setTimeout(async () => {
+  const { cleanupStaleOllamaProcesses } = await import("./ollama-pool-cleanup.js");
+  const report = await cleanupStaleOllamaProcesses({ ... });
+}, ...);
+```
+
+**Problem:** Cleanup runs as a deferred `setTimeout` — adds latency before stale processes are actually cleaned up.
+
+**Fix:** Run cleanup eagerly when a slot is freed, not on a timer.
+
+---
+
+### 13. `ollama-pool.ts:298,304` — `execSync` with 3s timeout
+
+```typescript
+{ encoding: "utf8", timeout: 3_000 },
+```
+
+**Problem:** Synchronous child processes block the event loop. If the command takes the full 3s, the entire process stalls.
+
+**Fix:** Use `execFile` with `Promise` wrapper or `spawn` with timeout signal.
+
+---
+
+### 14. `preflightSnapshot.ts:284,311` — Multiple 1.5s probe timeouts
+
+```typescript
+timeout: 1500
+```
+
+**Problem:** Preflight checks run multiple probes, each with 1.5s timeout. If multiple probes fail, this adds 3-6s of startup delay.
+
+**Fix:** Run probes in parallel with `Promise.allSettled()` and use the first success.
+
+---
+
+### 15. `tool-batching.ts:212-240` — Concurrency limit on reads
+
+```typescript
+export async function withConcurrencyLimit<T>(...) {
+  // Uses a limiter to cap parallel reads
+}
+```
+
+**Problem:** Concurrent-safe tools (reads) are limited by a concurrency cap. If the limit is too low, parallel reads are serialized unnecessarily.
+
+**Fix:** Make the concurrency limit configurable and increase the default (e.g., from 2 to 8).
+
+---
+
+## Summary: Top 3 Fixes for Immediate Impact
+
+| Priority | File | Change | Expected Savings |
+|----------|------|--------|------------------|
+| 1 | `streaming-executor.ts` | Replace 1ms spin loop with Promise notification | Eliminates busy-poll latency entirely |
+| 2 | `ollama-pool.ts` | Cache GPU detection with 30s TTL | Eliminates 10-100ms per request |
+| 3 | `cascadeBackend.ts` | Parallelize backend fallbacks with `Promise.race()` | Eliminates cascading timeout accumulation |

package/npm-shrinkwrap.json

@@ -1,12 +1,12 @@
 {
   "name": "omnius",
-  "version": "1.0.267",
+  "version": "1.0.269",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "omnius",
-      "version": "1.0.267",
+      "version": "1.0.269",
       "bundleDependencies": [
         "image-to-ascii"
       ],

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "omnius",
-  "version": "1.0.267",
+  "version": "1.0.269",
   "description": "AI coding agent powered by open-source models (Ollama/vLLM) — interactive TUI with agentic tool-calling loop",
   "type": "module",
   "main": "./dist/index.js",