@dp-pcs/ogp 0.3.2 → 0.4.2

package/docs/case-studies/CRASH_RESOLUTION_20260407.md

@@ -0,0 +1,190 @@
+# OpenClaw Crash Resolution
+**Date:** April 7, 2026  
+**Status:** ✅ RESOLVED with mitigations
+
+> **Note:** This document is a sanitized version of internal debugging notes. System-specific details have been generalized. Original created during OGP development to document OpenClaw regression debugging.
+
+---
+
+## Quick Summary
+
+Your OpenClaw crashes were caused by **TWO KNOWN BUGS in version 2026.4.5** - NOT by your OGP work or dual-assistant setup. Multiple GitHub issues filed by other users in the last 1-2 days confirm this.
+
+**Fixes implemented:**
+1. ✅ Wrapper script with 8GB heap limit
+2. ✅ All cron jobs disabled
+3. ✅ BrainLift plugin disabled
+4. ✅ Gateway auto-restart enabled
+
+**Current Status:** Gateway running stable with mitigations. Exec lifecycle bug may still cause occasional crashes but will auto-restart.
+
+---
+
+## The Bugs
+
+### Bug #1: Exec Lifecycle Crash
+**GitHub Issues:** [#62137](https://github.com/openclaw/openclaw/issues/62137), [#61592](https://github.com/openclaw/openclaw/issues/61592), [#61812](https://github.com/openclaw/openclaw/issues/61812)
+
+**Error:** `Unhandled promise rejection: Error: Agent listener invoked outside active run`
+
+**Cause:** Regression in 2026.4.5 where background exec process stdout crashes gateway after agent run completes
+
+**Platforms Affected:** Linux, Windows, macOS (all platforms)
+
+**Your Impact:** Crashed every 10-60 minutes during normal operations
+
+**Mitigation:** Wrapper script enables auto-restart; upstream fix pending
+
+### Bug #2: Browser Automation OOM
+**Error:** `FATAL ERROR: JavaScript heap out of memory`
+
+**Cause:** Default 4GB V8 heap limit too small for heavy browser automation
+
+**Your Impact:** Crashed after 2+ hours of browser activity
+
+**Fix:** Increased heap to 8GB via `--max-old-space-size=8192` flag
+
+### Bug #3: Cron Job API Key Failures
+**Error:** `401 Incorrect API key provided` (env var expansion failing)
+
+**Cause:** Environment variable evaluation failing when cron jobs execute
+
+**Your Impact:** Cron job running every 5 minutes triggering cascading failures
+
+**Fix:** Disabled all cron jobs and BrainLift plugin
+
+---
+
+## What We Did
+
+### 1. Created Gateway Wrapper Script
+
+**File:** `$HOME/.openclaw/bin/gateway-wrapper.sh`
+
+**What it does:**
+```bash
+#!/bin/bash
+# - Sets all environment variables
+# - Launches gateway with 8GB heap limit
+# - Enables auto-restart via LaunchAgent
+exec <node-path>/bin/node --max-old-space-size=8192 \
+  <openclaw-path>/dist/index.js gateway --port <port>
+```
+
+### 2. Updated LaunchAgent
+
+**File:** `$HOME/Library/LaunchAgents/ai.openclaw.gateway.plist`
+
+**Change:** Now calls wrapper script instead of node directly
+
+### 3. Disabled Cron Jobs
+
+**Files:**
+- `$HOME/.openclaw/openclaw.json` - BrainLift disabled
+- `$HOME/.openclaw/cron/jobs.json` - All cron jobs disabled
+
+---
+
+## OGP Cleared of Suspicion
+
+**Verdict:** Your OGP work is NOT causing the crashes.
+
+**Evidence:**
+- Same bugs reported by users not using OGP
+- GitHub issues filed 1-2 days ago across all platforms
+- Crashes occur with zero OGP activity
+- Known regressions in OpenClaw 2026.4.5
+
+**Your dual-assistant setup (OpenClaw + Hermes) may have exposed the bugs faster due to higher load, but didn't create them.**
+
+---
+
+## Current Gateway Status
+
+- **PID:** [Running]
+- **Port:** ✅ listening
+- **Heap Limit:** 8GB (doubled from 4GB)
+- **Wrapper:** ✅ Active
+- **Cron Jobs:** ✅ Disabled
+- **BrainLift:** ✅ Disabled
+- **LaunchAgent:** ✅ Auto-restart enabled
+
+**Uptime:** Started and currently stable
+
+---
+
+## Expected Behavior
+
+**Fixed:**
+- ✅ No more cron-triggered crashes
+- ✅ No more browser OOM crashes (unless you exceed 8GB heap)
+- ✅ Auto-restart on any crash
+
+**Still Possible:**
+- ⚠️ Exec lifecycle bug may still crash gateway occasionally
+- When this happens, LaunchAgent will auto-restart within seconds
+
+---
+
+## Monitoring
+
+**Check gateway status:**
+```bash
+launchctl list | grep openclaw
+lsof -i :<port>
+ps aux | grep openclaw-gateway
+```
+
+**Watch for crashes:**
+```bash
+tail -f ~/.openclaw/logs/gateway.err.log | grep -E "unhandled|FATAL"
+```
+
+**Success metrics:**
+- Uptime > 24 hours without manual intervention
+- No API key errors in logs
+- Auto-restart working if crashes occur
+
+---
+
+## Rollback (if needed)
+
+```bash
+# Restore original LaunchAgent
+cp $HOME/Library/LaunchAgents/ai.openclaw.gateway.plist.backup-* \
+   $HOME/Library/LaunchAgents/ai.openclaw.gateway.plist
+
+# Restore cron jobs
+cp $HOME/.openclaw/cron/jobs.json.backup-* \
+   $HOME/.openclaw/cron/jobs.json
+
+# Reload
+launchctl unload $HOME/Library/LaunchAgents/ai.openclaw.gateway.plist
+launchctl load $HOME/Library/LaunchAgents/ai.openclaw.gateway.plist
+```
+
+Or downgrade to OpenClaw 2026.4.2:
+```bash
+npm install -g openclaw@2026.4.2
+```
+
+---
+
+## Documentation
+
+**Full Details:** See `OpenClaw_Stability_Fix_Summary.md`  
+**Original Analysis:** See `crash_observations.md`  
+**Status Report:** See `OpenClaw_Hermes_Status_Report_20260407.md`
+
+**GitHub Issues to Watch:**
+- https://github.com/openclaw/openclaw/issues/62137
+- https://github.com/openclaw/openclaw/issues/61592
+- https://github.com/openclaw/openclaw/issues/61812
+- https://github.com/openclaw/openclaw/issues/61733
+
+---
+
+**Resolution Date:** April 7, 2026  
+**Gateway Status:** ✅ Running with mitigations  
+**Next Check:** Monitor for 24 hours  
+**Sanitized for Publication:** April 8, 2026

package/docs/case-studies/OpenClaw_Hermes_Status_Report_20260407.md

@@ -0,0 +1,142 @@
+# OpenClaw & Hermes — Status Report
+**Date:** April 7, 2026
+
+> **Note:** This document is a sanitized version of internal status reporting. System-specific paths, PIDs, and operational details have been generalized. Created during OGP development to compare gateway stability.
+
+---
+
+## Executive Summary
+
+Both local AI gateways (OpenClaw, Hermes) were evaluated. OpenClaw crashed multiple times in 24 hours from two distinct bugs. Hermes has been stable. A fact-check of an agent-drafted comparison article revealed it was substantially wrong. Config changes were made to switch OpenClaw's primary model to GPT-5.4 and adjust provider configurations.
+
+---
+
+## OpenClaw Gateway — Crash Analysis
+
+### Crash #1: OOM (April 6, evening)
+
+**Root cause:** The BrainLift plugin kicked off its nightly run for all 5 agents simultaneously. The agents hit Anthropic's rate limit (429) on Claude Sonnet 4.6. The embedded agent runner retried aggressively with no backoff ceiling and no memory cleanup between attempts. Heap grew to 4.08 GB and Node.js SIGABRT'd.
+
+**Contributing factors:**
+- All 5 agents scheduled at the same time
+- No exponential backoff on 429 retries
+- Default V8 heap limit (4 GB) with no `--max-old-space-size` override
+- Auth error mixed in (API key issue)
+
+**Evidence:** Logs showed repeated 429s, followed by `FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory`
+
+### Crash #2: Unhandled Promise Rejection (April 7, afternoon)
+
+**Root cause:** Bug in `pi-agent-core` — the exec tool's stdout handler fired a callback after the agent run had already ended. The gateway's global unhandled rejection handler treated this as fatal.
+
+**Stack trace origin:** `Agent.processEvents` in `pi-agent-core/src/agent.ts:533` — "Agent listener invoked outside active run"
+
+**Trigger:** Agent was editing files via the exec tool when the run completed but the exec process continued emitting stdout.
+
+### Crash #3: Same as #2 (April 7, evening)
+
+Identical stack trace. Same exec lifecycle bug. Reproducible.
+
+---
+
+## OpenClaw — Config Issues Found
+
+### 1. LaunchAgent Environment Variables Don't Work
+
+The LaunchAgent plist uses `$(security find-generic-password ...)` shell expansion syntax for API keys. **This doesn't work in launchd plists** — plist values are literal strings, not shell-evaluated. Keychain-derived env vars are empty when launched via launchd.
+
+**Impact:** Gateway starts without API keys → auth failures → retry loops → OOM.
+
+**Fix needed:** Either use a wrapper shell script in the plist that resolves keys before exec'ing the gateway, or store keys directly in the plist (less secure).
+
+### 2. Kimi Provider Configuration Issue
+
+The Kimi direct provider was referencing an API key that wasn't properly configured. The gateway's secret resolver treated this as a hard failure.
+
+**Current workaround:** Disabled the kimi plugin, removed kimi auth profile. The Fireworks-routed Kimi K2.5 still works via FIREWORKS_API_KEY.
+
+### 3. Skills Loading Issues
+
+On every startup, many skills log `"Skipping skill path that resolves outside its configured root."` These are likely symlinks or relative path references. A significant portion of the skill set is silently not loading.
+
+**Impact:** Agent capabilities reduced without any user-visible error.
+
+### 4. BrainLift Double-Fires
+
+The BrainLift plugin logged two `"starting nightly run"` entries within seconds of each other — running the full 5-agent sweep twice. This doubles API usage and compounds the rate limit problem.
+
+---
+
+## Changes Made This Session
+
+| Change | File | Detail |
+|--------|------|--------|
+| Primary model → GPT-5.4 | `openclaw.json` | Was `anthropic/claude-sonnet-4-6` |
+| Fallback chain updated | `openclaw.json` | Multiple fallback providers configured |
+| `openai/gpt-5.4` added to models | `openclaw.json` | New model entry with Responses API |
+| Kimi plugin disabled | `openclaw.json` | `plugins.entries.kimi.enabled: false` |
+| Kimi auth profile removed | `openclaw.json` | Removed kimi auth profile |
+| Gateway started with 8GB heap | Manual launch | `--max-old-space-size=8192` |
+| Logs truncated | `logs/` | Log rotation applied |
+
+---
+
+## Hermes Gateway — Status
+
+Hermes has been stable throughout. Running Python 3.11, `hermes gateway run --replace`. Port responding (403 on unauthenticated requests, expected). Low resource usage.
+
+OGP bridge process also running.
+
+No crashes, no issues.
+
+---
+
+## Article Fact-Check: "Hermes vs OpenClaw"
+
+An agent-drafted article was **substantially wrong**. Its central thesis — "OpenClaw is desktop-first, Hermes is cloud-native" — is fabricated. Both are local daemons running on the same machine.
+
+**Key errors corrected:**
+- Hermes is NOT cloud-hosted (it's a local Python process)
+- Hermes storage is NOT cloud-backed (it's local SQLite + markdown)
+- Hermes skills are NOT synced cloud storage (local filesystem)
+- Hermes does NOT have built-in public endpoints (needs tunnels like OpenClaw)
+- "Turn off your phone and federation continues" is false (machine off = Hermes off)
+
+**Corrected article delivered** with fact-checked claims against live configs and running processes.
+
+---
+
+## Recommended Actions
+
+### Immediate (Stability)
+
+1. **Fix the exec lifecycle crash** — File issue against `pi-agent-core`. The unhandled rejection in `Agent.processEvents` when exec stdout fires after run completion is a repeatable crasher. Until fixed, the gateway will keep dying.
+
+2. **Fix LaunchAgent env vars** — Replace the `$(...)` plist values with a wrapper script:
+   ```bash
+   #!/bin/bash
+   export ANTHROPIC_API_KEY=$(security find-generic-password ...)
+   # ... other keys ...
+   exec <node-path> --max-old-space-size=8192 \
+     <openclaw-path> gateway --port <port>
+   ```
+   Point the plist's ProgramArguments at this script instead of node directly.
+
+3. **Add `--max-old-space-size=8192`** to the LaunchAgent permanently (via the wrapper script above).
+
+### Short-term (Reliability)
+
+4. **Stagger BrainLift agent runs** — Don't fire all agents at the same cron tick. Space them apart to avoid rate limit contention.
+
+5. **Investigate the skipped skills issue** — Check for broken symlinks or path traversal in skills directory. These represent a significant portion of the skill set not loading.
+
+### Medium-term (Resilience)
+
+6. **Request backoff/retry ceiling in embedded agent runner** — The 429 retry loop with no backoff is the #1 contributor to OOM crashes. Needs exponential backoff + max retry count + memory cleanup between attempts.
+
+7. **Add process supervision** — Current state: launchd throttles after crash, manual nohup doesn't survive reboot. Consider a wrapper that catches SIGABRT and restarts with a cooldown.
+
+---
+
+**Document Created:** April 7, 2026  
+**Sanitized for Publication:** April 8, 2026

package/docs/case-studies/OpenClaw_Stability_Fix_Summary.md

@@ -0,0 +1,209 @@
+# OpenClaw Stability Fix Summary
+**Date:** April 7, 2026  
+**Status:** RESOLVED - Mitigations Implemented
+
+> **Note:** This document is a sanitized version of internal debugging notes. File paths, process IDs, and system-specific details have been generalized. The original was created during OGP development to debug an unrelated OpenClaw regression.
+
+---
+
+## Problem Summary
+
+OpenClaw gateway (v2026.4.5) was crashing every 10-60 minutes with two distinct failure modes:
+
+1. **Exec Lifecycle Bug** - "Agent listener invoked outside active run" error
+2. **Browser Automation OOM** - V8 heap exhaustion from heavy browser use
+
+---
+
+## Root Cause Analysis
+
+### Bug #1: Exec Lifecycle Crash (CRITICAL)
+
+**Status:** **KNOWN BUG in OpenClaw 2026.4.5** - Regression from 2026.4.2
+
+**Error:** `Unhandled promise rejection: Error: Agent listener invoked outside active run`
+
+**GitHub Issues:**
+- [#62137](https://github.com/openclaw/openclaw/issues/62137) - Exec/PTY unhandled promise rejection
+- [#61592](https://github.com/openclaw/openclaw/issues/61592) - Background exec process crashes
+- [#61812](https://github.com/openclaw/openclaw/issues/61812) - Regression in 2026.4.5
+- [#61733](https://github.com/openclaw/openclaw/issues/61733) - Windows crashes with same error
+
+**Technical Details:**
+When a background exec process emits stdout after the agent run has completed, the gateway crashes instead of safely ignoring or buffering the output. The `pi-agent-core` library's `Agent.processEvents` method throws when called outside an active run context.
+
+**Trigger Scenarios:**
+- File operations
+- Long-running exec processes
+- Bash tools calling `openclaw message send`
+- Cron jobs spawning exec sessions
+
+**Impact:** Gateway crashes every 10-60 minutes during normal operation
+
+### Bug #2: Browser Automation OOM
+
+**Error:** `FATAL ERROR: v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath Allocation failed - JavaScript heap out of memory`
+
+**Root Cause:** Heavy browser automation creates large serialized objects that overflow the default V8 heap limit (4GB)
+
+**Impact:** Gateway crashes after extended browser automation sessions (2-4 hours)
+
+### Bug #3: Cron Job API Key Failures (FIXED)
+
+**Status:** RESOLVED by disabling cron jobs
+
+**Error:** `401 Incorrect API key provided` (environment variable expansion failing)
+
+**Root Cause:** Environment variable evaluation failing in LaunchAgent context when cron jobs execute, triggering cascading model fallback failures and eventual OOM
+
+**Fix:** Disabled all cron jobs + BrainLift plugin
+
+---
+
+## Solutions Implemented
+
+### ✅ Solution #1: Wrapper Script with 8GB Heap Limit
+
+**File:** `$HOME/.openclaw/bin/gateway-wrapper.sh`
+
+**What it does:**
+- Sets all required environment variables explicitly
+- Launches gateway with `--max-old-space-size=8192` (8GB heap limit)
+- Provides logging for debugging
+
+**LaunchAgent Integration:**
+Updated LaunchAgent plist to use wrapper instead of calling node directly
+
+**Benefits:**
+- Doubles heap limit to prevent browser OOM crashes
+- Ensures env vars are always set correctly
+- Survives OpenClaw updates (wrapper script is outside node_modules)
+
+### ✅ Solution #2: Disabled All Cron Jobs
+
+**Files Modified:**
+- `$HOME/.openclaw/openclaw.json` - BrainLift plugin disabled
+- `$HOME/.openclaw/cron/jobs.json` - All cron jobs disabled
+
+**Impact:**
+- Eliminates cron-triggered API key evaluation failures
+- Prevents BrainLift OOM crashes from simultaneous agent runs
+- Stops scheduled jobs that were triggering crashes
+
+### ✅ Solution #3: API Keys in Config File
+
+**Status:** Already fixed via config modification
+
+**What happened:** OpenClaw config's `env` section was modified to include API keys directly instead of shell command expansion
+
+**Effect:** Environment variables now always available, preventing auth cascades
+
+---
+
+## Remaining Issues
+
+### ⚠️ Exec Lifecycle Bug - NOT FIXED, MITIGATED
+
+**Status:** Waiting for OpenClaw developers to fix in pi-agent-core
+
+**Mitigation:** Gateway will still crash when exec lifecycle bug triggers, but LaunchAgent will auto-restart it
+
+**Upstream Fix Options:**
+1. Wait for OpenClaw team to release patch
+2. Roll back to 2026.4.2 (workaround mentioned in GitHub issues)
+3. Avoid file operations that trigger long-running exec processes
+
+**Recommended Action:** Monitor for OpenClaw 2026.4.6 or later that fixes these issues
+
+---
+
+## OGP Correlation
+
+**Conclusion:** OGP work is **NOT** the cause of crashes
+
+**Evidence:**
+- Both bugs are known OpenClaw 2026.4.5 regressions affecting all users
+- Crashes occur with zero OGP activity
+- GitHub issues filed by users not using OGP
+- Dual-assistant setup (OpenClaw + Hermes) may have exposed bugs faster due to higher load, but didn't create them
+
+---
+
+## Current Status
+
+**Gateway:** ✅ Running  
+**Heap Limit:** ✅ 8GB (doubled from default 4GB)  
+**Cron Jobs:** ✅ Disabled  
+**BrainLift:** ✅ Disabled  
+**Wrapper Script:** ✅ Active via LaunchAgent
+
+**Expected Stability:**
+- ✅ No more cron-triggered crashes
+- ✅ No more browser OOM crashes (unless >8GB heap usage)
+- ⚠️ Exec lifecycle bug may still cause occasional crashes (auto-restart enabled)
+
+---
+
+## Testing & Monitoring
+
+**To verify stability:**
+
+```bash
+# Check gateway status
+launchctl list | grep openclaw
+lsof -i :<gateway-port>
+
+# Monitor for crashes
+tail -f ~/.openclaw/logs/gateway.err.log | grep -E "unhandled|crash|FATAL"
+
+# Check uptime
+ps aux | grep openclaw-gateway
+```
+
+**Success Metrics:**
+- Gateway uptime > 24 hours without manual restart
+- No API key evaluation errors in logs
+- No OOM crashes during browser automation
+
+---
+
+## Rollback Instructions
+
+If issues persist, to rollback:
+
+```bash
+# Restore original LaunchAgent
+cp $HOME/Library/LaunchAgents/ai.openclaw.gateway.plist.backup-* \
+   $HOME/Library/LaunchAgents/ai.openclaw.gateway.plist
+
+# Restore cron jobs
+cp $HOME/.openclaw/cron/jobs.json.backup-* \
+   $HOME/.openclaw/cron/jobs.json
+
+# Re-enable BrainLift in openclaw.json
+# (manually change "enabled": false to true)
+
+# Reload LaunchAgent
+launchctl unload $HOME/Library/LaunchAgents/ai.openclaw.gateway.plist
+launchctl load $HOME/Library/LaunchAgents/ai.openclaw.gateway.plist
+```
+
+Or consider rolling back OpenClaw to 2026.4.2:
+```bash
+npm install -g openclaw@2026.4.2
+# Note: May require removing plugins.entries.memory-core.config.dreaming from config
+```
+
+---
+
+## Next Steps
+
+1. ✅ Monitor gateway stability for 24-48 hours
+2. ⏸️ Wait for OpenClaw 2026.4.6+ release with exec lifecycle fix
+3. 🔍 Investigate skipped skills issue (low priority)
+
+---
+
+**Document Created:** April 7, 2026  
+**Last Updated:** April 7, 2026  
+**Sanitized for Publication:** April 8, 2026

package/docs/case-studies/README.md

@@ -0,0 +1,40 @@
+# OGP Development Case Studies
+
+This directory contains sanitized debugging notes from real-world OGP development and deployment challenges. These documents capture the messy reality of building federated AI systems — including the false starts, red herrings, and lessons learned.
+
+> **⚠️ Note:** These files are sanitized versions of internal debugging notes. System-specific details (file paths, PIDs, API key fragments, port numbers) have been removed or generalized to protect operational security while preserving the technical narrative.
+
+---
+
+## Contents
+
+| File | Description |
+|------|-------------|
+| `OpenClaw_Stability_Fix_Summary.md` | Comprehensive analysis of OpenClaw 2026.4.5 regression bugs encountered during OGP development. Includes root cause analysis, mitigations, and wrapper script implementation. |
+| `CRASH_RESOLUTION_20260407.md` | Quick reference guide for the same stability issues — condensed version for immediate action. |
+| `crash_observations.md` | Raw timeline and observations from the debugging session. Shows the iterative process of elimination that ultimately cleared OGP of suspicion. |
+| `OpenClaw_Hermes_Status_Report_20260407.md` | Comparative analysis of OpenClaw vs. Hermes gateway stability during federation testing. Includes fact-check of an AI-drafted article that was substantially wrong. |
+
+---
+
+## Context
+
+These documents were created on April 7, 2026, during intensive OGP federation testing. The initial hypothesis was that OGP's dual-assistant setup (OpenClaw + Hermes) was causing gateway instability. **The reality:** OpenClaw 2026.4.5 had known regression bugs affecting all users.
+
+**Key Lesson:** When debugging complex systems, correlation is not causation. The OGP work exposed OpenClaw bugs faster due to higher load, but didn't create them.
+
+---
+
+## Related Article
+
+The debugging narrative behind these files is documented in:
+
+**"[Case Study] When Your AI Tools Keep Crashing: A Meta-Debugging Loop with OpenClaw and Claude"**
+
+This Substack article tells the story of using Claude (via Dispatch) to diagnose OpenClaw crashes while OpenClaw was down, then using OpenClaw/Claude Code to fix OGP bugs, then back to Claude when OpenClaw crashed again — a meta-loop that became the only way forward.
+
+---
+
+**Why These Are Here:**
+
+The article promised these files would be "available in dp-pcs/ogp." Rather than leave them as unverified claims, we're publishing the sanitized source material. Real debugging is messy. Real systems fail in unexpected ways. Federation requires resilience not just in protocol design, but in the development process itself.