@appkit/llamacpp-cli 1.11.0 → 1.12.0

package/MONITORING-ACCURACY-FIX.md

@@ -1,199 +0,0 @@
-# Historical Monitoring Accuracy Fix
-
-**STATUS:** This document describes the initial fix attempt for memory calculations. However, the root issue was that historical monitoring was showing **system-wide metrics** instead of **per-process metrics**. See `PER-PROCESS-METRICS.md` for the correct implementation.
-
-## Issue Summary
-
-Comparison between our historical monitoring and macmon revealed discrepancies in memory usage calculations.
-
-## Issues Identified
-
-### 1. Memory Total Calculation (CRITICAL)
-
-**Problem:** Total memory was calculated by summing all vm_stat page counts, which doesn't equal the actual installed RAM.
-
-**Evidence:**
-- Historical monitor showed: ~60% memory usage
-- macmon showed: 26.86 / 32.0 GB = ~84% memory usage
-- The denominator (32.0 GB installed RAM) was being calculated incorrectly
-
-**Root Cause:**
-```typescript
-// OLD CODE (INCORRECT)
-const totalPages = pagesActive + pagesWired + pagesCompressed +
-                   pagesFree + pagesInactive + pagesSpeculative;
-const memoryTotal = totalPages * pageSize;
-```
-
-This approach has fundamental flaws:
-- vm_stat doesn't report all memory categories (kernel reserved, etc.)
-- Page counts don't sum to actual installed RAM
-- Results in artificially inflated "total" value
-- Makes memory usage appear lower than reality
-
-**Fix:**
-```typescript
-// NEW CODE (CORRECT)
-// Get total installed RAM from sysctl (accurate)
-const memoryTotal = await execCommand('sysctl -n hw.memsize 2>/dev/null');
-```
-
-Use `sysctl hw.memsize` to get actual installed RAM size in bytes. This matches what Activity Monitor and macmon report.
-
-### 2. Memory Used Calculation (VERIFIED CORRECT)
-
-**Current approach:**
-```typescript
-// Used = Active + Wired + Compressed
-const usedPages = pagesActive + pagesWired + pagesCompressed;
-const memoryUsed = usedPages * pageSize;
-```
-
-This formula is **correct** and matches what Activity Monitor and macmon report as "used memory".
-
-- **Active:** Recently used memory
-- **Wired:** Kernel memory that can't be paged out
-- **Compressed:** Compressed pages in RAM
-
-We removed the calculation of unused page types (free, inactive, speculative) since they're not needed.
-
-### 3. CPU Calculation (VERIFIED CORRECT)
-
-**Formula:**
-```typescript
-cpuUsage = ((pcpuUsage * pCoreCount) + (ecpuUsage * eCoreCount)) / totalCores * 100
-```
-
-This weighted average is mathematically correct:
-- macmon reports per-core-type averages (P-CPU: 25%, E-CPU: 36%)
-- Formula computes overall system average: `(25% × 6 + 36% × 4) / 10 = 29.4%`
-- Historical average of 33% is reasonable given fluctuations over time
-
-### 4. GPU Calculation (VERIFIED CORRECT)
-
-**Observation:**
-- Historical: Avg: 1.8%, Max: 4.0%, Min: 0.6%
-- macmon snapshot: GPU 4%
-
-This is **expected behavior**:
-- GPU is mostly idle (0-2%) between inference requests
-- Spikes to 4% during active token generation
-- Average of 1.8% correctly reflects mostly-idle state
-- Max of 4.0% matches macmon's instantaneous reading
-
-## Changes Made
-
-### `src/lib/system-collector.ts`
-
-**1. Removed total memory calculation from vm_stat parsing:**
-```typescript
-// Now only returns memoryUsed
-private parseVmStatOutput(output: string): { memoryUsed: number }
-```
-
-**2. Added method to get actual installed RAM:**
-```typescript
-private async getTotalMemory(): Promise<number> {
-  const output = await execCommand('sysctl -n hw.memsize 2>/dev/null');
-  return parseInt(output.trim(), 10) || 0;
-}
-```
-
-**3. Combined both sources in new method:**
-```typescript
-private async getMemoryMetrics(): Promise<{
-  memoryUsed: number;
-  memoryTotal: number;
-}> {
-  // Get used memory from vm_stat (active + wired + compressed)
-  const vmStatOutput = await execCommand('vm_stat 2>/dev/null');
-  const { memoryUsed } = this.parseVmStatOutput(vmStatOutput);
-
-  // Get total installed RAM from sysctl (accurate)
-  const memoryTotal = await this.getTotalMemory();
-
-  return { memoryUsed, memoryTotal };
-}
-```
-
-**4. Updated collector to use new method:**
-```typescript
-// Always get memory from vm_stat + sysctl (accurate total from sysctl)
-const memoryMetrics = await this.getMemoryMetrics();
-```
-
-## Verification
-
-After these changes, memory usage should now accurately match macmon and Activity Monitor:
-
-**Before:**
-- Total: Calculated from page sum (~40 GB equivalent)
-- Used: 26.86 GB
-- **Percentage: ~60% (WRONG)**
-
-**After:**
-- Total: 32.0 GB (from `sysctl hw.memsize`)
-- Used: 26.86 GB (from vm_stat)
-- **Percentage: ~84% (CORRECT)**
-
-## Testing Recommendations
-
-1. **Compare with macmon:**
-   ```bash
-   # Terminal 1: Run macmon
-   macmon
-
-   # Terminal 2: Monitor server
-   npm run dev -- server monitor <server-id>
-   ```
-
-   Memory percentages should now match within 1-2%.
-
-2. **Compare with Activity Monitor:**
-   - Open Activity Monitor → Memory tab
-   - Check "Memory Used" value
-   - Should match historical monitor's memory calculation
-
-3. **Verify historical data:**
-   ```bash
-   # View historical metrics (press H in monitor)
-   npm run dev -- server monitor <server-id>
-   # Press 'H' to toggle historical view
-   ```
-
-   Memory usage should now show realistic values (~80-90% on actively used system).
-
-4. **Check edge cases:**
-   - Fresh boot (low memory usage ~30-40%)
-   - Under load (high memory usage ~85-95%)
-   - Multiple servers running (memory should increase proportionally)
-
-## Impact on Historical Data
-
-**Note:** Existing historical data was collected with the old (incorrect) calculation.
-
-**Options:**
-
-1. **Keep old data as-is** (recommended for now)
-   - Historical charts will show old incorrect baseline
-   - New data will be accurate going forward
-   - Natural transition over 24 hours as old data ages out
-
-2. **Clear history and start fresh:**
-   ```bash
-   rm ~/.llamacpp/history/*.json
-   ```
-   - Immediate accuracy
-   - Lose historical context
-
-## Related Files
-
-- `src/lib/system-collector.ts` - System metrics collection (MODIFIED)
-- `src/lib/history-manager.ts` - History persistence (unchanged)
-- `src/tui/HistoricalMonitorApp.ts` - Historical UI (unchanged)
-
-## References
-
-- macOS `vm_stat` documentation: Reports memory in pages (16KB on Apple Silicon)
-- macOS `sysctl` documentation: `hw.memsize` reports installed RAM in bytes
-- Activity Monitor algorithm: Uses active + wired + compressed for "Memory Used"

package/PER-PROCESS-METRICS.md

@@ -1,190 +0,0 @@
-# Per-Process Metrics Implementation
-
-## Overview
-
-Historical monitoring now shows **per-process metrics** for the specific llama-server being monitored, rather than system-wide metrics. This provides accurate resource usage for each model.
-
-## What Changed
-
-### Before (System-Wide)
-- **GPU Usage:** All processes combined
-- **CPU Usage:** All processes combined
-- **Memory Usage:** All processes combined (% of total RAM)
-
-### After (Per-Process)
-- **GPU Usage:** System-wide (unchanged - can't isolate per-process on macOS)
-- **CPU Usage:** Just the llama-server process (from `ps`)
-- **Memory Usage:** Just the llama-server process in GB (from `top`)
-
-## Implementation Details
-
-### 1. Process Metrics Collection
-
-**Added CPU collection (`src/utils/process-utils.ts`):**
-```typescript
-// Batch collection for efficiency
-export async function getBatchProcessCpu(pids: number[]): Promise<Map<number, number | null>>
-
-// Single process collection
-export async function getProcessCpu(pid: number): Promise<number | null>
-```
-
-**Features:**
-- Uses `ps -p <pid> -o %cpu` to get per-process CPU percentage
-- 3-second cache to prevent excessive process spawning
-- Batch collection for multi-server monitoring
-- Returns percentage (0-100+, can exceed 100% on multi-core)
-
-### 2. Type Updates
-
-**ServerMetrics interface (`src/types/monitor-types.ts`):**
-```typescript
-export interface ServerMetrics {
-  // ... existing fields
-  processMemory?: number;     // Already existed
-  processCpuUsage?: number;   // NEW: Per-process CPU %
-}
-```
-
-**HistorySnapshot interface (`src/types/history-types.ts`):**
-```typescript
-export interface HistorySnapshot {
-  server: {
-    // ... existing fields
-    processMemory?: number;      // Already existed
-    processCpuUsage?: number;    // NEW: Per-process CPU %
-  };
-  system?: {
-    // ... system-wide metrics (kept for live monitoring)
-  };
-}
-```
-
-### 3. Metrics Collection
-
-**MetricsAggregator (`src/lib/metrics-aggregator.ts`):**
-- Added `processCpuUsage` parameter to `collectServerMetrics()`
-- Collects CPU in parallel with other metrics
-- Supports batch collection for multi-server scenarios
-
-**HistoryManager (`src/lib/history-manager.ts`):**
-- Saves `processCpuUsage` in snapshots
-- Maintains backward compatibility (optional field)
-
-### 4. Historical Monitor UI
-
-**HistoricalMonitorApp (`src/tui/HistoricalMonitorApp.ts`):**
-
-**Chart Changes:**
-
-**GPU Usage:**
-- **Unchanged:** Still system-wide
-- **Reason:** macOS doesn't provide per-process GPU metrics easily
-- **Label:** "GPU Usage (%)"
-
-**CPU Usage:**
-- **Before:** `snapshot.system.cpuUsage` (system-wide)
-- **After:** `snapshot.server.processCpuUsage` (per-process)
-- **Label:** "Process CPU Usage (%)"
-- **Range:** Not forced to 0-100% (can show >100% for multi-threaded workloads)
-
-**Memory Usage:**
-- **Before:** `(system.memoryUsed / system.memoryTotal) * 100` (system-wide %)
-- **After:** `processMemory / (1024 * 1024 * 1024)` (per-process GB)
-- **Label:** "Process Memory Usage (GB)"
-- **Format:** Shows 2 decimal places (e.g., "3.45 GB")
-- **Statistics:** Avg, Max, Min in GB
-
-**Multi-Server Comparison:**
-- Table also updated to show per-process CPU and memory
-- Memory column now shows GB instead of %
-
-## Benefits
-
-1. **Accurate Attribution:** See exactly what each model is using
-2. **Multi-Server Clarity:** Compare resource usage across different models
-3. **Debugging:** Identify which specific model is consuming resources
-4. **Capacity Planning:** Understand per-model requirements
-
-## Example Output
-
-**Before (System-Wide):**
-```
-CPU Usage (%)
-  Avg: 33.0% (±17.4)  Max: 86.6%  Min: 12.0%
-
-Memory Usage (%)
-  Avg: 31.0% (±0.6)  Max: 31.9%  Min: 29.9%
-```
-
-**After (Per-Process):**
-```
-Process CPU Usage (%)
-  Avg: 45.2% (±12.3)  Max: 120.5%  Min: 8.1%
-
-Process Memory Usage (GB)
-  Avg: 3.45 GB (±0.12)  Max: 3.67 GB  Min: 3.21 GB
-```
-
-## Edge Cases Handled
-
-1. **Missing Data:** Fields are optional, gracefully handles old snapshots
-2. **Process Not Running:** Returns null, charts skip those data points
-3. **Multi-Core:** CPU can exceed 100% (expected behavior)
-4. **Cache Expiry:** 3-second TTL prevents stale data
-5. **Batch Collection:** Efficient when monitoring multiple servers
-
-## Testing Recommendations
-
-1. **Single Server:**
-   ```bash
-   npm run dev -- server monitor <server-id>
-   # Press 'H' to view historical data
-   ```
-   - Verify CPU shows reasonable per-process values (not system-wide)
-   - Verify memory shows model size in GB (not total RAM %)
-
-2. **Multi-Server:**
-   ```bash
-   npm run dev -- server monitor
-   # Press 'H' to view comparison table
-   ```
-   - Verify each server shows different CPU/memory values
-   - Verify table shows GB for memory column
-
-3. **Compare with Activity Monitor:**
-   - Open Activity Monitor
-   - Filter for `llama-server` process
-   - CPU % should match within 5-10%
-   - Memory should match within 0.1 GB
-
-4. **Compare with `ps`:**
-   ```bash
-   ps -p <pid> -o %cpu,rss
-   ```
-   - CPU % should match
-   - RSS (memory) should match when converted to GB
-
-## Backward Compatibility
-
-- Old history files still work (missing fields treated as undefined)
-- System-wide metrics still collected for live monitoring
-- Live monitoring TUI unchanged (still shows system-wide for context)
-- Only historical view changed to per-process
-
-## Related Files
-
-- `src/utils/process-utils.ts` - Added CPU collection functions
-- `src/types/monitor-types.ts` - Added processCpuUsage field
-- `src/types/history-types.ts` - Added processCpuUsage to snapshots
-- `src/lib/metrics-aggregator.ts` - Collects CPU metrics
-- `src/lib/history-manager.ts` - Saves CPU metrics
-- `src/tui/HistoricalMonitorApp.ts` - Displays per-process charts
-
-## Future Improvements
-
-1. **Per-Process GPU:** Investigate Metal API for GPU attribution
-2. **Network I/O:** Track per-process network usage
-3. **Disk I/O:** Track per-process disk reads/writes
-4. **Thread Count:** Show number of threads used by process
-5. **Context Switches:** Show voluntary/involuntary context switches