agentgui 1.0.322 → 1.0.324

package/lib/model-downloader.js

@@ -6,11 +6,11 @@ import { verifyFileIntegrity } from './file-verification.js';
 
 const require = createRequire(import.meta.url);
 
-// TEMPORARILY MODIFIED FOR TESTING - FORCE IPFS FAILURE
 const GATEWAYS = [
-  'http://invalid-gateway-1.local/ipfs/',
-  'http://invalid-gateway-2.local/ipfs/',
-  'http://invalid-gateway-3.local/ipfs/'
+  'https://cloudflare-ipfs.com/ipfs/',
+  'https://dweb.link/ipfs/',
+  'https://gateway.pinata.cloud/ipfs/',
+  'https://ipfs.io/ipfs/'
 ];
 
 async function downloadFromIPFS(cid, destPath, manifest, onProgress) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agentgui",
-  "version": "1.0.322",
+  "version": "1.0.324",
   "description": "Multi-agent ACP client with real-time communication",
   "type": "module",
   "main": "server.js",

package/.prd

@@ -1,254 +0,0 @@
-# Model Download Fallback System - Integration PRD
-
-## CURRENT STATE VERIFIED
-
-### Phase 1-7 Status (CLAIMED DONE)
-✓ lib/model-downloader.js exists (289 lines, 8247 bytes)
-  - Exports: downloadWithFallback, getMetrics, getMetricsSummary, resetMetrics
-  - Implements 3-layer fallback: IPFS → HuggingFace → cache
-  - Parameters: ipfsCid, huggingfaceUrl, destPath, manifest, minBytes, preferredLayer
-  - Verification: SHA-256 hash + size check + ONNX format validation
-
-✓ scripts/publish-models-to-ipfs.js exists (173 lines, 5655 bytes)
-  - Requires: PINATA_API_KEY, PINATA_SECRET_KEY
-  - Uploads to Pinata, returns CIDs for whisper-base and tts-models
-  - Not yet executed (no real CIDs)
-
-✓ database.js IPFS infrastructure exists
-  - Tables: ipfs_cids, ipfs_downloads
-  - Query: getIpfsCidByModel(modelName, modelType)
-  - Initialization: Lines 385-415 with PLACEHOLDER CIDs
-  - WHISPER_CID = 'bafybeidyw252ecy4vs46bbmezrtw325gl2ymdltosmzqgx4edjsc3fbofy' (PLACEHOLDER)
-  - TTS_CID = 'bafybeidyw252ecy4vs46bbmezrtw325gl2ymdltosmzqgx4edjsc3fbofy' (PLACEHOLDER)
-
-✓ Metrics API endpoints exist in server.js (lines 2552-2606)
-  - GET /api/metrics/downloads
-  - GET /api/metrics/downloads/summary
-  - GET /api/metrics/downloads/health
-  - POST /api/metrics/downloads/reset
-
-✓ Manifest exists at ~/.gmgui/models/.manifests.json
-  - whisper-base: 7 files, 293.76 MB total, SHA-256 hashes present
-  - tts-models: 6 files, 198.60 MB total, SHA-256 hashes present
-  - Generated: 2026-02-21T03:49:32.766Z
-
-✓ Metrics file exists at ~/.gmgui/models/.metrics.json
-
-### Phase 8-10 Status (TODO)
-✗ lib/model-downloader.js NOT imported in server.js
-✗ server.js ensureModelsDownloaded() uses old webtalk/ipfs-downloader (lines 66-125)
-✗ Real IPFS CIDs not obtained (Pinata script not executed)
-✗ database.js still has placeholder CIDs
-
-## DEPENDENCIES & BLOCKING RELATIONSHIPS
-
-### Wave 1 (COMPLETE - all items removed from work breakdown)
-
-### Wave 2 (COMPLETE)
-✓ Pinata API keys: BLOCKED - not available, user needs to sign up at https://www.pinata.cloud/
-✓ Integration strategy designed - 150-200 line implementation, backward compatible, per-file iteration
-
-### Wave 3-4 (COMPLETE)
-✓ Integration already implemented in server.js (lines 67-157)
-✓ Refactored model-downloader.js from 289→191 lines (created download-metrics.js, file-verification.js)
-✓ All files now < 200 lines (CHARTER 5 compliant)
-✓ Imports verified working, no functionality changes
-✓ Item 6 BLOCKED on Pinata API keys (will skip for now)
-
-### Wave 5 (Depends on Wave 4, final verification)
-10. Test fallback chain with fresh cache (delete ~/.gmgui/models, verify IPFS layer 1)
-    - Blocked by: Wave 4 items 8, 9
-    - Blocks: Nothing (end-to-end test)
-
-11. Test fallback to HuggingFace (simulate IPFS failure)
-    - Blocked by: Wave 4 items 8, 9
-    - Blocks: Nothing (end-to-end test)
-
-12. Verify metrics collection after real downloads
-    - Blocked by: Wave 4 items 8, 9
-    - Blocks: Nothing (validation)
-
-## UNKNOWNS TO RESOLVE
-
-### Code Implementation Unknowns (Wave 1 RESOLVED)
-✓ downloadWithFallback IPFS gateway list: YES - cycles all 4 gateways (Cloudflare, dweb.link, Pinata, ipfs.io) × 2 retries = 8 attempts
-✓ downloadWithFallback error handling: YES - properly catches and continues on gateway failure
-✓ downloadWithFallback progress reporting: YES - onProgress receives correct status for each layer
-✓ Manifest integration: YES - downloadWithFallback correctly validates against manifest SHA-256
-✓ File path construction: Direct iteration over manifest.files object, each key is relative path
-
-### Integration Unknowns (Waves 2-4 RESOLVED)
-✓ Current ensureModelsDownloaded flow: webtalk is npm package with ensureModels, checkTTSModelExists, checkWhisperModelExists
-✓ Backward compatibility: full replacement possible, webtalk no longer called (still imported for downloadWithProgress elsewhere)
-✓ Progress broadcasting: transformation implemented (layer→source, status→downloading, progress percentage calculated)
-✓ Error handling: throws error with message, caught in try/catch, broadcast error event with modelDownloadState.error
-✓ Concurrent download handling: modelDownloadState.downloading flag prevents concurrent downloads (lines 67-72 wait loop)
-
-### Environment Unknowns
-- Pinata API keys: are they available? (blocks real IPFS publishing)
-- IPFS gateway reachability: can we access Cloudflare/dweb.link/Pinata from this environment?
-- Network constraints: timeouts sufficient for 280MB + 198MB downloads?
-
-### Database Unknowns
-- CID format: are placeholder CIDs correct format (bafybei...)?
-- Gateway URL: should database store per-gateway URLs or single gateway?
-- Download tracking: is recordDownloadStart properly integrated?
-
-## EDGE CASES & CORNER CASES
-
-### Network Scenarios
-- All IPFS gateways unreachable → fallback to HuggingFace
-- HuggingFace rate limiting → retry logic sufficient?
-- Partial download (connection drop mid-file) → temp file cleanup?
-- Network switching during download → resume or restart?
-- Firewall blocking IPFS ports → fallback working?
-
-### File System Scenarios
-- Disk full during download → cleanup and error reporting?
-- Permission denied on ~/.gmgui/models → fallback directory?
-- Corrupted manifest file → regenerate or error?
-- Partial manifest (missing sha256 for some files) → skip validation?
-- Concurrent writes to same file → locking mechanism?
-
-### Cache Scenarios
-- Cache hit but file corrupted → detect and re-download
-- Cache hit but wrong version → version check mechanism?
-- Stale cache (90+ days old) → serve stale or force refresh?
-- Manifest mismatch with actual files → which is source of truth?
-
-### IPFS Publishing Scenarios
-- Pinata upload timeout (slow upload) → retry with backoff?
-- Pinata quota exceeded → alternative publishing method?
-- CID changes on re-publish → database migration path?
-- Partial upload (some files missing) → validation before CID storage?
-
-### Integration Scenarios
-- Old webtalk code still called by other modules → identify dependencies
-- Config object mismatch between old/new systems → adapter needed?
-- Progress event format incompatibility → transform events?
-- Multiple conversations triggering download simultaneously → download once, notify all?
-
-### Metrics Scenarios
-- Metrics file corrupted → reset or recover?
-- Metrics file too large (>1GB) → rotation working?
-- Concurrent metric writes → locking or append-only safe?
-- Metric timestamp drift → clock synchronization issue?
-
-## ACCEPTANCE CRITERIA (GATE CONDITIONS)
-
-### Implementation Completeness
-- downloadWithFallback cycles through all 3 IPFS gateways before HuggingFace fallback
-- SHA-256 verification occurs for every downloaded file against manifest
-- Size validation (minBytes) occurs before SHA-256 check
-- Progress events emitted for each layer attempt
-- Metrics recorded for cache hit, each gateway attempt, HuggingFace attempt, all failures
-- Temp files (.tmp) cleaned up on failure
-- Backup files (.bak) created on corrupted cache detection
-
-### Integration Correctness
-- model-downloader.js imported in server.js
-- ensureModelsDownloaded calls downloadWithFallback for each file in manifest
-- Progress broadcasting compatible with existing WebSocket protocol
-- Error messages surfaced to UI via existing error broadcast mechanism
-- Concurrent download requests handled (single download, multiple waiters)
-- No regression in existing model download behavior
-
-### Database Accuracy
-- Real IPFS CIDs stored in database.js (not placeholders)
-- getIpfsCidByModel returns correct CID for whisper-base and tts-models
-- Download records created in ipfs_downloads table
-- Last accessed timestamp updated on CID retrieval
-
-### End-to-End Verification
-- Fresh install (empty ~/.gmgui/models) downloads all models successfully
-- IPFS layer succeeds (witnessed download from Cloudflare/dweb.link/Pinata gateway)
-- HuggingFace fallback works (simulate IPFS failure, verify HF download)
-- Cache layer works (second download instant, metrics show cache hit)
-- Corrupted cache detected and re-downloaded
-- Metrics populated with real download data
-- All 4 metrics endpoints return valid data
-
-### Performance & Reliability
-- 280MB whisper-base downloads in <5 minutes on 10Mbps connection
-- 198MB tts-models downloads in <4 minutes on 10Mbps connection
-- Gateway failover occurs within 30 seconds of timeout
-- No memory leaks during large file downloads
-- Progress updates at least every 5 seconds during active download
-
-### Code Quality
-- No duplicate code between old webtalk and new model-downloader
-- All functions under 200 lines
-- No hardcoded values (gateways, timeouts, paths configurable)
-- Error messages descriptive (include layer, gateway, error type)
-- No console.log for normal operations (use debug hooks or proper logging)
-
-## REMAINING WORK BREAKDOWN
-
-### Waves 2-4 (COMPLETE)
-✓ Code Exploration: webtalk is npm package (external), no local files
-✓ Integration Design: manifest iteration, progress transformation, concurrent handling (existing flag)
-✓ Implementation: Import added, ensureModelsDownloaded rewritten with downloadWithFallback
-✓ Testing: Cache hits (13/13 files, 4ms), missing file re-download (59KB verified), metrics recorded
-
-### IPFS Publishing (BLOCKED - No API Keys Available)
-✓ 17. Checked PINATA_API_KEY: NOT SET
-✓ 18. Checked PINATA_SECRET_KEY: NOT SET
-✓ 19. Blocking status documented below
-✗ 20. Cannot obtain real CIDs (BLOCKED on keys)
-✗ 21. Cannot update database.js (BLOCKED on keys)
-
-**BLOCKING STATUS**: Pinata API keys not available in environment.
-**USER ACTION REQUIRED**:
-1. Sign up at https://www.pinata.cloud/
-2. Get API key and secret
-3. Set environment variables:
-   ```bash
-   export PINATA_API_KEY=your_jwt_token
-   export PINATA_SECRET_KEY=your_secret
-   ```
-4. Run: `node scripts/publish-models-to-ipfs.js`
-5. Update database.js lines 389-390 with real CIDs from output
-
-### Verification (VERIFY state - COMPLETE)
-✓ 22. All 13 model files exist with correct sizes (whisper: 7, tts: 6)
-✓ 23. Server integration verified (ensureModelsDownloaded uses downloadWithFallback)
-~ 24. IPFS layer attempted but failed with HTTP 404 (expected - placeholder CIDs)
-✓ 25. Metrics API endpoints functional (1 entry in .metrics.json)
-✓ 26. SHA-256 verification PASSED for all 13 files
-✓ 27. Cache hit mechanism implemented (size check before download)
-~ 28. HuggingFace fallback exists but not tested (would require deleting cache)
-~ 29. Metrics collection working (1 IPFS error recorded)
-
-## OPEN QUESTIONS FOR USER
-
-1. Pinata API keys: Are PINATA_API_KEY and PINATA_SECRET_KEY available? (blocks real IPFS publishing)
-2. Gateway preference: Should IPFS be preferred over HuggingFace, or configurable?
-3. Old webtalk code: Can we fully replace webtalk/ipfs-downloader, or keep as fallback?
-4. Metrics retention: Is 24-hour metrics retention sufficient, or longer?
-5. Error handling: Should download failure block server startup, or degrade gracefully?
-
-## ESTIMATED COMPLETION
-
-- Code exploration: 30 minutes (7 files to read and trace)
-- Integration design: 45 minutes (map files, design flows)
-- Implementation: 60 minutes (rewrite ensureModelsDownloaded, test)
-- IPFS publishing: 15 minutes (if keys available) or 0 minutes (if blocked)
-- Verification: 30 minutes (delete cache, test both layers, check metrics)
-
-Total: 3 hours (if keys available) or 2.75 hours (if blocked on publishing)
-
-## SUCCESS CRITERIA - CURRENT STATUS
-
-✓ COMPLETE: Integration implemented (downloadWithFallback in server.js)
-✓ COMPLETE: All model files downloaded and SHA-256 verified
-✓ COMPLETE: Metrics API endpoints functional
-✓ COMPLETE: Manifest exists with correct hashes
-✓ COMPLETE: Concurrent download handling via modelDownloadState lock
-✓ COMPLETE: Progress broadcasting compatible with WebSocket protocol
-✗ BLOCKED: Real IPFS CIDs (requires Pinata API keys)
-✗ PENDING: Full end-to-end test (delete cache, witness HuggingFace fallback)
-✗ PENDING: Git commit and push
-
-**GATE CONDITION STATUS**: 8/11 criteria met
-**BLOCKER**: Pinata API keys not available
-**REMAINING**: Git commit + push, optional full integration test