pi-harness-runtime 1.1.166 → 1.1.167

package/harness/reports/PROJECT_STATE.md

@@ -0,0 +1,342 @@
+# Pi Harness Runtime - Project State Report
+
+**Generated:** 2026-07-06  
+**Last Updated:** 2026-07-06
+
+---
+
+## Project Overview
+
+Pi Harness Runtime is a local-first, provider-agnostic AI coding harness for pi.dev. It provides:
+
+- Quota tracking and management
+- Multi-provider support (MiniMax, OpenRouter, etc.)
+- Job state machine with checkpoint/resume
+- Task graph with dependency management
+- Auto-compaction and continuation on output token limits
+- Mobile notifications via Telegram/Ntfy/Webhook
+
+---
+
+## Directory Structure
+
+```
+pi-harness-runtime/
++-- index.ts                    # Main CLI extension entry point
++-- cli.ts                      # CLI commands
++-- mirror.ts                   # Provider mirror store
++-- renderer.ts                 # Status rendering
++-- tracker.ts                  # Usage tracking
++-- sync-form.ts                # Manual sync form
++-- windows.ts                  # Windows aggregation
++-- harness/                    # Core harness modules
+|   +-- agent-handoff.ts       # Agent handoff protocol
+|   +-- auto-compact.ts         # RFC-0019: Auto Compact
+|   +-- blackboard.ts           # Shared blackboard
+|   +-- context-window-manager.ts
+|   +-- job-state-machine.ts    # Job state machine
+|   +-- loop-runtime.ts         # RFC-0001: Main loop
+|   +-- master-planner.ts       # Task planning
+|   +-- notification-events.ts   # RFC-0022: Notification events
+|   +-- output-limit-handler.ts # RFC-0020: Output limit
+|   +-- partial-recovery.ts     # RFC-0021: Partial recovery
+|   +-- repair-engine.ts        # Repair/retry logic
+|   +-- task-graph.ts           # Task dependency graph
+|   +-- e2e/                    # E2E test engine
+|   +-- project-detector/       # Project type detection
++-- packages/
+|   +-- auth/                   # Authentication
+|   +-- checkpoint/             # Checkpoint manager
+|   +-- notification/           # RFC-0022: Notification center
+|   |   +-- notification-center.ts
+|   |   +-- types.ts
+|   |   +-- base-adapter.ts
+|   |   +-- adapters/
+|   |       +-- telegram-adapter.ts
+|   |       +-- ntfy-adapter.ts
+|   |       +-- email-adapter.ts
+|   |       +-- webhook-adapter.ts
+|   +-- provider-router/         # Provider routing
+|   +-- providers/              # Provider adapters
+|   +-- quota-manager/         # Quota tracking
+|   +-- runtime/               # Runtime components (NEW)
+|   +-- scheduler/             # Job scheduling
+|   +-- shared-context/        # Shared context
+|   +-- tui/                   # Terminal UI
+|   +-- types/                 # Type definitions
+|   +-- worktree/              # Git worktree management
++-- RFC/                       # RFC documents (0028 total)
++-- IMPLEMENTATION/             # Implementation tracking
+|   +-- RFC-XXXX/              # Per-RFC implementation tracking
+|       +-- AI_BRIEF.md
+|       +-- TASKS.md
+|       +-- ACCEPTANCE_CRITERIA.md
++-- schemas/                   # JSON schemas
++-- profiles/                  # Provider profiles
++-- test/                     # Tests
+```
+
+---
+
+## RFC Status
+
+### Completed Implementations
+
+| RFC | Name | Status | File(s) |
+| ----- | ------ | -------- | --------- |
+| 0001 | Loop Runtime | ✅ Implemented | `harness/loop-runtime.ts` |
+| 0002 | Provider Adapter | ✅ Implemented | `packages/providers/adapters.ts` |
+| 0003 | Quota Manager | ✅ Implemented | `packages/quota-manager/` |
+| 0004 | Playwright Adapter | ✅ Implemented | `packages/providers/` |
+| 0005 | Git Worktree | ✅ Implemented | `packages/worktree/` |
+| 0006 | Checkpoint Manager | ✅ Implemented | `packages/checkpoint/` |
+| 0007 | Scheduler | ✅ Implemented | `packages/scheduler/` |
+| 0008 | Provider Router | ✅ Implemented | `packages/provider-router/` |
+| 0009 | Shared Context | ✅ Implemented | `packages/shared-context/` |
+| 0010 | Context Window Manager | ✅ Implemented | `harness/context-window-manager.ts` |
+| 0011 | Shared Blackboard | ✅ Implemented | `harness/blackboard.ts` |
+| 0012 | Agent Handoff Protocol | ✅ Implemented | `harness/agent-handoff.ts` |
+| 0013 | E2E Test Engine | ✅ Implemented | `harness/e2e/` |
+| 0014 | Project Detector | ✅ Implemented | `harness/project-detector/` |
+| 0015 | Job State Machine | ✅ Implemented | `harness/job-state-machine.ts` |
+| 0016 | Task Graph | ✅ Implemented | `harness/task-graph.ts` |
+| 0017 | Master Planner | ✅ Implemented | `harness/master-planner.ts` |
+| 0018 | Repair Engine | ✅ Implemented | `harness/repair-engine.ts` |
+| 0019 | Auto Compact and Continue | ✅ Implemented | `harness/auto-compact.ts` |
+| 0020 | Output Token Limit Handler | ✅ Implemented | `harness/output-limit-handler.ts` |
+| 0021 | Partial Response Recovery | ✅ Implemented | `harness/partial-recovery.ts` |
+| 0022 | Notification Center | ✅ Implemented | `packages/notification/` |
+| 0023 | Local Browser Agent | 🔲 Stub Only | - |
+
+### Stubs (Need Implementation)
+
+| RFC | Name | Status |
+| ----- | ------ | -------- |
+| 0024 | Local Runtime Agent | ✅ Implemented - `packages/runtime/local-runtime-agent.ts` |
+| 0025 | Command Executor | ✅ Implemented - `packages/runtime/command-executor.ts` |
+| 0026 | Workspace Manager | ✅ Implemented - `packages/runtime/workspace-manager.ts` |
+| 0027 | Runtime API | ✅ Implemented - `packages/runtime/runtime-api.ts` |
+| 0028 | Policy Engine | ✅ Implemented - `packages/runtime/policy-engine.ts` |
+
+---
+
+## Key Implementation Patterns
+
+### 1. RFC Document Structure
+
+Each RFC in `/RFC/` follows this format:
+
+```markdown
+# RFC-XXXX - Title
+
+## Purpose
+## Motivation  
+## Goals
+## Artifact Layout (file structure)
+## Runtime Flow (state diagram)
+## Acceptance Criteria
+```
+
+### 2. Implementation Tracking
+
+Each RFC in `/IMPLEMENTATION/RFC-XXXX/` contains:
+
+- `AI_BRIEF.md` - Brief description of what to implement
+- `TASKS.md` - Task list (Read RFC, Implement, Test)
+- `ACCEPTANCE_CRITERIA.md` - Success criteria
+
+### 3. Class Naming Convention
+
+- **Core Engine:** `[Name]Engine` or `[Name]Manager`
+  - Example: `AutoCompactEngine`, `JobStateMachine`, `TaskGraphManager`
+- **Runtime:** `[Name]Runtime`
+  - Example: `LoopRuntime`, `LocalRuntimeAgent`
+- **Adapter:** `[Name]Adapter` extends `BaseChannelAdapter`
+  - Example: `TelegramAdapter`, `NtfyAdapter`
+- **Handler:** `[Name]Handler`
+  - Example: `OutputLimitHandler`, `PartialRecovery`
+
+### 4. TypeScript Conventions
+
+- Uses ESM (`import/export`)
+- JSDoc comments for documentation
+- Interfaces for public APIs, types for unions
+- Error handling with typed errors
+- Async/await pattern for I/O
+
+### 5. File Paths
+
+- Harness modules: `harness/[module-name].ts`
+- Package modules: `packages/[name]/[module-name].ts`
+- Adapters: `packages/notification/adapters/[name]-adapter.ts`
+
+---
+
+## Configuration
+
+### Environment Variables
+
+```
+# Provider
+MINIMAX_API_KEY=
+OPENROUTER_API_KEY=
+
+# Notification
+TELEGRAM_BOT_TOKEN=
+TELEGRAM_CHAT_ID=
+NTFY_TOPIC=
+NTFY_SERVER=https://ntfy.sh
+NTFY_TOKEN=
+NOTIFICATION_WEBHOOK_URL=
+NOTIFICATION_WEBHOOK_METHOD=POST
+
+# Paths
+PI_USAGE_DIR=~/.pi/usage-status
+PI_HARNESS_DIR=~/.pi/harness
+```
+
+### Runtime Storage
+
+- **Usage tracking:** `~/.pi/usage-status/`
+- **Harness state:** `~/.pi/harness/[jobId]/`
+  - `checkpoint.json` - Job checkpoint
+  - `context/` - Compaction artifacts
+  - `partial/` - Partial responses
+  - `events.jsonl` - Event log
+
+---
+
+## Event Types (RFC-0022)
+
+```typescript
+type NotificationEvent =
+  | "JobStarted"
+  | "TaskCompleted"
+  | "TaskFailed"
+  | "QuotaPaused"
+  | "ResumeScheduled"
+  | "ContextCompacted"
+  | "OutputLimitContinued"
+  | "E2EFailed"
+  | "HumanReviewNeeded"
+  | "ReadyForClient"
+  | "JobCancelled"
+  | "Error";
+```
+
+---
+
+## State Machine States
+
+```
+pending -> planning -> running <-> repairing <-> testing <-> reviewing
+    |         |          |              |            |
+    v         v          v              v            v
+ cancelled  paused   waiting_human   paused_quota  ready_for_client
+```
+
+---
+
+## ✅ IMPLEMENTED: RFCs 0024-0028
+
+### RFC-0024: Local Runtime Agent ✅
+
+**File:** `packages/runtime/local-runtime-agent.ts`
+
+- HTTP server for quota checking
+- Coordinates with Local Browser Agent (RFC-0023)
+- Security: redact sensitive data, localhost binding
+
+### RFC-0025: Command Executor ✅
+
+**File:** `packages/runtime/command-executor.ts`
+
+- Safe shell command execution
+- Timeout, output capture
+- Security policies with allowlist/denylist
+
+### RFC-0026: Workspace Manager ✅
+
+**File:** `packages/runtime/workspace-manager.ts`
+
+- Workspace directory management
+- Worktree lifecycle
+- Automatic cleanup policies
+
+### RFC-0027: Runtime API ✅
+
+**File:** `packages/runtime/runtime-api.ts`
+
+- HTTP/RPC API for runtime control
+- Job management endpoints
+- WebSocket support for real-time events
+
+### RFC-0028: Policy Engine ✅
+
+**File:** `packages/runtime/policy-engine.ts`
+
+- Command execution policies
+- Network access policies
+- Rate limiting & audit logging
+
+---
+
+## Provider Usage Data
+
+See: `harness/reports/PROVIDER_USAGE_RESEARCH.md`
+
+| Provider | Method | Status |
+| ---------- | -------- | -------- |
+| MiniMax | Browser scraping | 🔄 Testing |
+| OpenAI (GPT) | TUI parsing | ✅ Implemented |
+| GLM (Zhipu) | TUI parsing | ✅ Implemented |
+
+### Unified Approach
+
+All providers use TUI message parsing (except MiniMax which needs browser).
+No API keys needed - hooks into pi's error/status messages.
+
+---
+
+## Dependencies
+
+```json
+{
+  "dependencies": {
+    "node:fs": "built-in",
+    "node:path": "built-in",
+    "node:os": "built-in",
+    "node:http": "built-in",
+    "node:events": "built-in",
+    "ws": "WebSocket support"
+  }
+}
+```
+
+---
+
+## Testing
+
+Test files located in `/test/`:
+
+- Unit tests for core modules
+- Integration tests for state machine
+- E2E test engine in `harness/e2e/`
+
+---
+
+## Next Steps
+
+1. ~~Complete RFC-0024~~ - Local Runtime Agent ✅
+2. ~~Implement RFC-0025~~ - Command Executor ✅
+3. ~~Implement RFC-0026~~ - Workspace Manager ✅
+4. ~~Implement RFC-0027~~ - Runtime API ✅
+5. ~~Implement RFC-0028~~ - Policy Engine ✅
+6. ~~Update index.ts~~ - Package exports ✅
+
+### New Priority Items
+
+1. **Implement OpenAI billing API** usage fetcher
+2. **Research GLM** usage API
+3. Debug MiniMax browser scraping
+4. Add acceptance tests for RFC-0024-0028

package/harness/reports/PROVIDER_USAGE_RESEARCH.md

@@ -0,0 +1,115 @@
+# Provider Usage API Research
+
+**Date:** 2026-07-06  
+**Updated:** Unified TUI Integration
+
+---
+
+## Summary: Unified TUI Message Parsing
+
+All providers (except MiniMax) report quota exhaustion via TUI in pi.dev. No API keys needed!
+
+| Provider | Method | File | Status |
+| ---------- | -------- | ------ | -------- |
+| OpenAI (GPT) | TUI message | `tui-usage-monitor.ts` | ✅ |
+| GLM (Zhipu) | TUI message | `tui-usage-monitor.ts` | ✅ |
+| Anthropic (Claude) | TUI message | `tui-usage-monitor.ts` | ✅ |
+| OpenRouter | TUI message | `tui-usage-monitor.ts` | ✅ |
+| MiniMax | Browser scraping | `minimax-browser-auth.ts` | 🔄 |
+
+---
+
+## Architecture
+
+### Unified TUI Usage Monitor
+
+**File:** `packages/quota-manager/tui-usage-monitor.ts`
+
+The `TUIUsageMonitor` handles all TUI-based providers:
+
+- Detects provider from message content
+- Parses reset time and limit type
+- Records to QuotaManager automatically
+
+### Integration
+
+```typescript
+import { QuotaManager, TUIUsageMonitor } from "@pi-harness/quota-manager";
+
+const quotaManager = new QuotaManager();
+
+const monitor = new TUIUsageMonitor({
+  quotaManager,
+  debug: true,
+});
+
+// Hook into pi
+pi.on("error", (event) => monitor.processMessage(event.message));
+pi.on("message", (event) => monitor.processMessage(event.message));
+
+// Check availability
+if (!quotaManager.isAvailable("openai")) {
+  const waitTime = quotaManager.getWaitTime("openai");
+  console.log(`Waiting ${waitTime}ms for OpenAI`);
+}
+```
+
+### Detected Patterns
+
+| Provider | Patterns |
+| ---------- | ---------- |
+| OpenAI | `OpenAI`, `GPT`, `insufficient_quota`, `429` |
+| GLM | `GLM`, `Zhipu`, `tokens exhausted` |
+| Anthropic | `Claude`, `Anthropic`, `rate_limit_error` |
+| OpenRouter | `OpenRouter`, `quota exhausted` |
+
+---
+
+## MiniMax (Separate)
+
+**File:** `packages/auth/src/minimax-browser-auth.ts`
+
+MiniMax requires browser scraping because it doesn't expose usage via TUI.
+
+**Solution:** Local Browser Agent (RFC-0023 + RFC-0024)
+
+---
+
+## Files Created/Updated
+
+```
+packages/
++-- quota-manager/
+|   +-- quota-manager.ts       # Core quota tracking
+|   +-- tui-usage-monitor.ts  # ✅ NEW - Unified TUI parser
+|   +-- index.ts               # ✅ NEW - Exports
+|   +-- examples/
+|       +-- tui-usage-integration.ts  # ✅ NEW - Example
++-- providers/
+|   +-- openai-usage.ts       # Legacy (can be removed)
+|   +-- glm-usage.ts          # Legacy (can be removed)
+packages/
++-- types/src/
+    +-- runtime-types.ts       # Updated QuotaSignal.source
+```
+
+---
+
+## TODO
+
+### ✅ Completed
+
+1. [x] TUIUsageMonitor - unified parser
+2. [x] QuotaManager integration
+3. [x] Multi-provider support
+
+### 🔄 In Progress
+
+4. [ ] Test with actual pi TUI messages
+2. [ ] Add to pi extension
+
+### 📋 MiniMax
+
+- Browser scraping exists but not 100%
+- Needs Local Browser Agent (RFC-0023)
+- Not blocking TUI providers

package/harness/reports/SESSION_STATUS.md

@@ -0,0 +1,107 @@
+# Session Status Report
+
+**Date:** 2026-07-06  
+**Session:** Implementation Day
+
+---
+
+## ✅ Completed Implementations (RFC-0024 to RFC-0028)
+
+### Files Created
+
+| File | RFC | Description |
+| ------ | ----- | ------------- |
+| `packages/runtime/local-runtime-agent.ts` | 0024 | HTTP server for local browser agent coordination |
+| `packages/runtime/command-executor.ts` | 0025 | Safe shell command execution |
+| `packages/runtime/workspace-manager.ts` | 0026 | Workspace lifecycle management |
+| `packages/runtime/runtime-api.ts` | 0027 | HTTP/RPC API for job control |
+| `packages/runtime/policy-engine.ts` | 0028 | Policy enforcement with rate limiting |
+| `packages/runtime/index.ts` | - | Package exports |
+| `packages/providers/openai-usage.ts` | - | OpenAI billing API fetcher |
+| `packages/providers/glm-usage.ts` | - | GLM TUI message parser |
+| `packages/providers/index.ts` | - | Providers package exports |
+| `harness/reports/PROJECT_STATE.md` | - | Project documentation |
+| `harness/reports/SESSION_STATUS.md` | - | Session todo list |
+| `harness/reports/PROVIDER_USAGE_RESEARCH.md` | - | Provider research |
+
+### Dependencies Added
+
+- `@types/ws` - WebSocket support
+
+---
+
+## 🔬 In Progress: Provider Usage Data
+
+### MiniMax (Testing in Progress)
+
+- **Status:** Browser scraping via Playwright
+- **URL:** `https://platform.minimax.io/console/usage`
+- **File:** `packages/auth/src/minimax-browser-auth.ts`
+- **Issue:** Not 100% success rate yet
+- **Need:** Local Browser Agent to handle human login
+
+### GPT (OpenAI) - ✅ IMPLEMENTED
+
+- **File:** `packages/providers/openai-usage.ts`
+- **Method:** TUI message parsing (no API key needed)
+- **Patterns:** Context window, quota exhausted, rate limit, reset time
+
+### GLM (Zhipu AI) - ✅ IMPLEMENTED
+
+- **File:** `packages/providers/glm-usage.ts`
+- **Method:** TUI message parsing (no API key needed)
+- **Patterns:** Context window, tokens exhausted, rate limit, reset time
+
+---
+
+## 📋 TODO List
+
+### High Priority
+
+1. [x] Explore GPT/OpenAI usage API - ✅ Implemented
+2. [x] Explore GLM/Zhipu AI usage API - ✅ Implemented (TUI parsing)
+3. [x] Integrate usage data into QuotaManager - ✅ Done
+4. [ ] Add to pi extension
+5. [ ] Complete Local Browser Agent (RFC-0023 stub)
+
+### Medium Priority
+
+1. [ ] Add acceptance tests for RFC-0024-0028
+2. [ ] Integrate new runtime components into main index.ts
+3. [ ] Test CommandExecutor with actual commands
+
+### Low Priority
+
+1. [ ] Write RFC stubs for any new components needed
+2. [ ] Performance testing
+
+---
+
+## 🔍 Research Notes
+
+### MiniMax
+
+- Requires browser login (authenticated session)
+- Cannot be scraped via server-side fetch
+- Solution: Local Browser Agent (RFC-0023 + RFC-0024)
+
+### GPT (OpenAI)
+
+- API key based authentication ✅
+- Usage endpoint: `https://api.openai.com/v1/billing/usage` ✅
+- Implementation: `packages/providers/openai-usage.ts` ✅
+
+### GLM (Zhipu AI)
+
+- Reports quota via TUI in pi.dev ✅
+- Implementation: `packages/providers/glm-usage.ts` ✅
+- Hooks into error/status messages, extracts reset time
+
+---
+
+## Next Actions
+
+1. [x] ~~Check existing adapters~~ - Done
+2. [x] ~~Test OpenAI usage API~~ - Implemented, need to test
+3. [ ] Update QuotaManager to integrate all providers
+4. [ ] Test integration with pi extension