@elvatis_com/openclaw-self-healing-elvatis 0.2.7 → 0.2.9

package/.ai/handoff/CONVENTIONS.md

@@ -0,0 +1,67 @@
+# CONVENTIONS.md — openclaw-self-healing-elvatis
+
+## Language & Runtime
+- TypeScript strict mode, ESM (`"type": "module"`)
+- Node 16 module resolution (`"moduleResolution": "Node16"`)
+- Target: ES2022
+
+## Package
+- Scope: `@elvatis_com/openclaw-self-healing-elvatis`
+- Plugin ID: `openclaw-self-healing-elvatis`
+- Test runner: vitest
+
+## File Layout
+```
+.ai/handoff/         ← AAHP protocol files (this folder)
+src/                 ← (future) extracted helpers
+index.ts             ← plugin entry point + all business logic
+test/                ← vitest test suites
+openclaw.plugin.json ← manifest
+package.json / tsconfig.json / tsconfig.check.json
+scripts/             ← one-off utilities (roadmap issue creation, etc.)
+```
+
+## Code Style
+- Named exports for all pure helpers (testable without plugin API)
+- `saveState()` must be called BEFORE any destructive system call (e.g. `openclaw gateway restart`)
+  → Reason: systemd-managed gateway restarts kill the process mid-execution; state written after the call is never persisted
+- No secrets in logs — redact with `[REDACTED]`
+- `runCmd` always uses `bash -lc` for login shell compatibility
+- All `autoFix.*` flags default to the safest value (`false` unless stated otherwise)
+- `whatsappRestartEnabled` is the one exception — defaults to `true` (most useful default)
+
+## Release Checklist (mandatory for every publish)
+
+### Before release
+1. `npm run typecheck` — must pass
+2. `npm test` — all tests must pass
+3. Bump version in `package.json` AND `openclaw.plugin.json`
+
+### Publish (all three platforms, no exceptions)
+4. `git tag vX.Y.Z && git push origin main && git push origin vX.Y.Z`
+5. `gh release create vX.Y.Z --title "..." --notes "..."`
+6. `npm publish --access public`
+7. ClawHub (use rsync workaround — `.clawhubignore` is NOT respected by `clawhub publish`):
+   ```bash
+   TMP=$(mktemp -d)
+   rsync -a --exclude='node_modules' --exclude='.git' --exclude='dist' \
+     --exclude='package-lock.json' ./ "$TMP/"
+   clawhub publish "$TMP" --slug openclaw-self-healing-elvatis \
+     --name "OpenClaw Self Healing" --version X.Y.Z --changelog "..." --no-input
+   rm -rf "$TMP"
+   ```
+
+### After release
+8. Update ALL docs in this repo: STATUS.md, DASHBOARD.md, LOG.md, NEXT_ACTIONS.md, README.md, SKILL.md
+9. Update MEMORY.md on server if architecture decisions changed
+
+## Documentation Rule (MANDATORY)
+**Every release MUST update the following files before committing:**
+- `.ai/handoff/STATUS.md` — current version, state, open risks
+- `.ai/handoff/DASHBOARD.md` — task table
+- `.ai/handoff/LOG.md` — append entry for this session
+- `.ai/handoff/NEXT_ACTIONS.md` — move done tasks, add new ones
+- `README.md` — version number + any changed behavior
+- `SKILL.md` — if commands or config changed
+
+Skipping documentation = incomplete release. No exceptions.

package/.ai/handoff/DASHBOARD.md

@@ -1,13 +1,38 @@
-# DASHBOARD (AAHP)
-
-| Task | Ready? | Owner | Notes |
-|---|---|---|---|
-| Create GitHub issues | ✅ | Human | Run `gh auth login` then `bash scripts/create-roadmap-issues.sh` |
-| Unit test suite | ✅ | Akido | Highest priority - vitest + tsconfig |
-| TS build pipeline | ✅ | Akido | Add tsconfig.json, build script |
-| Plugin health monitoring | Blocked | Akido | Waiting for `openclaw plugins list --json` |
-| Health status endpoint | ✅ | Akido | Medium priority |
-| Observability events | ✅ | Akido | Medium priority |
-| Dry-run mode | ✅ | Akido | Medium priority |
-| Active recovery probing | ✅ | Akido | Low priority |
-| Config hot-reload | ✅ | Akido | Low priority |
+# DASHBOARD.md — openclaw-self-healing-elvatis
+
+_Last updated: 2026-03-07_
+
+## 🏗️ Plugin Status
+
+| Component | Version | Build | Tests | Status |
+|-----------|---------|-------|-------|--------|
+| openclaw-self-healing-elvatis | 0.2.8 | ✅ | 181/181 ✅ | ✅ Stable |
+
+## 🚀 Release State
+
+| Platform | Version | Status |
+|----------|---------|--------|
+| GitHub | v0.2.8 | ✅ Tagged + Release |
+| npm | 0.2.8 | ✅ Published |
+| ClawHub | 0.2.8 | ✅ Published |
+
+## 📋 Open Tasks
+
+| ID | Task | Priority | Status |
+|----|------|----------|--------|
+| T-014 | Export heal metrics to `~/.aahp/metrics.jsonl` | 🟡 MEDIUM | Ready |
+
+## ✅ Completed Tasks
+
+| Task | Title | Version |
+|------|-------|---------|
+| T-015 | Fix infinite gateway restart loop (lastRestartAt before restart) | 0.2.8 |
+| T-005 | Plugin health monitoring via JSON API + auto-disable | 0.2.7 |
+| T-013 | Status snapshot file on each monitor tick | 0.2.6 |
+| T-012 | Startup config validation (fail-fast) | 0.2.6 |
+| T-011 | Integration tests for monitor tick flows | 0.2.6 |
+| T-004 | TypeScript build pipeline + typecheck | 0.2.5 |
+| T-010 | Observability events + status endpoint | 0.2.4 |
+| T-009 | Dry-run mode | 0.2.3 |
+| T-008 | Active recovery probing | 0.2.3 |
+| T-007 | Config hot-reload | 0.2.3 |

package/.ai/handoff/LOG.md

@@ -1,4 +1,62 @@
-# LOG (AAHP)
+# LOG.md — openclaw-self-healing-elvatis
 
-- 2026-02-27: Defined v0.3 roadmap with 8 items (3 high, 3 medium, 2 low). Created ROADMAP.md and scripts/create-roadmap-issues.sh. GitHub issue creation deferred - gh CLI not authenticated.
-- 2026-02-24: Initialized AAHP handoff structure.
+## 2026-03-07 — Session (Akido / claude-sonnet-4-6)
+
+**Bug: Infinite gateway restart loop (fixed in v0.2.8)**
+
+Root cause: `openclaw gateway restart` kills the process via systemd. `lastRestartAt = nowSec()`,
+`disconnectStreak = 0`, and `saveState()` were placed AFTER `runCmd("openclaw gateway restart")`.
+These lines never executed when the process was killed. On next boot, `lastRestartAt = 0`,
+`since = huge`, bypassing the 5-minute rate-limit guard. WhatsApp takes 60–120s to reconnect
+after any restart → self-healing saw 2 disconnect ticks → hit threshold → restarted again.
+Infinite loop.
+
+Fix: moved `lastRestartAt = nowSec()`, `disconnectStreak = 0`, and `saveState()` to BEFORE
+the `runCmd("openclaw gateway restart")` call.
+
+**Release pipeline:**
+- v0.2.8 committed, tagged, pushed to GitHub
+- GitHub release created: https://github.com/elvatis/openclaw-self-healing-elvatis/releases/tag/v0.2.8
+- npm published: `@elvatis_com/openclaw-self-healing-elvatis@0.2.8`
+- ClawHub published: `openclaw-self-healing-elvatis@0.2.8` (rsync workaround — .clawhubignore not respected by clawhub publish)
+- Added `.clawhubignore` and documented rsync workaround in CONVENTIONS.md
+
+All 181 tests passed before release.
+
+---
+
+## 2026-03-02 — Session (Akido)
+
+- T-013: Status snapshot file written on each monitor tick
+- T-012: Startup config validation with fail-fast
+- T-011: Integration tests for monitor service tick flows
+- T-005: Plugin health monitoring implemented via `openclaw plugins list --json`
+- Atomic state writes (tmp + rename)
+
+---
+
+## 2026-03-01 — Session (Akido)
+
+- T-004: TypeScript build pipeline + typecheck
+
+---
+
+## 2026-02-28 — Session (Akido)
+
+- T-010: Self-heal status endpoint / observability events
+- T-009: Dry-run mode
+- T-008: Active recovery probing
+- T-007: Config hot-reload
+
+---
+
+## 2026-02-27 — Session (Akido)
+
+- Defined v0.3 roadmap with 8 items. Created ROADMAP.md + scripts/create-roadmap-issues.sh.
+- GitHub issue creation deferred (gh CLI not authenticated at the time).
+
+---
+
+## 2026-02-24 — Session (Akido)
+
+- Initialized AAHP handoff structure.

package/.ai/handoff/NEXT_ACTIONS.md

@@ -1,23 +1,25 @@
-# NEXT_ACTIONS (AAHP)
+# NEXT_ACTIONS.md — openclaw-self-healing-elvatis
+
+_Last updated: 2026-03-07_
 
 ## Status Summary
 
-| Status  | Count |
-|---------|-------|
-| Done    | 13    |
-| Ready   | 1     |
-| Blocked | 1     |
+| Status | Count |
+|--------|-------|
+| Done   | 15    |
+| Ready  | 1     |
+| Blocked | 0   |
 
 ---
 
-## Ready - Work These Next
+## Ready — Work These Next
 
-### T-014: [medium] - Export heal metrics to ~/.aahp/metrics.jsonl (issue #12)
-- **Goal:** Append one JSONL line per heal event to `~/.aahp/metrics.jsonl` for analysis and alerting.
-- **Context:** Heal events are currently only visible in logs and via api.emit(). There is no persistent record for analysis.
+### T-014: [medium] — Export heal metrics to `~/.aahp/metrics.jsonl`
+- **Goal:** Append one JSONL line per heal event for analysis and alerting.
+- **Context:** Heal events are only visible in logs and via `api.emit()`. No persistent record for analysis.
 - **What to do:**
   - Export `appendMetric(line, metricsFile)` helper
-  - Write entries for: model-cooldown, session-patched, whatsapp-restart, cron-disabled, model-recovered
+  - Write entries for: `model-cooldown`, `session-patched`, `whatsapp-restart`, `cron-disabled`, `model-recovered`
   - Default metrics file: `~/.aahp/metrics.jsonl` (configurable via `metricsFile`)
   - Skip or mark dry-run events
   - Create parent directory if missing
@@ -27,28 +29,17 @@
 
 ---
 
-## Blocked
-
-### T-005: [high] - Implement structured plugin health monitoring and auto-disable (issue #3)
-- **Blocked by:** Waiting for `openclaw plugins list --json` API from openclaw core
-- **Goal:** Monitor plugin health and auto-disable failing plugins using structured JSON output.
-- **Context:** Current code has a stub that parses plain text output from `openclaw plugins list`. No robust parsing is possible without the `--json` flag.
-- **What to do (when unblocked):**
-  - Parse `openclaw plugins list --json` output for plugin status
-  - Auto-disable plugins with `status=error` (respecting `pluginDisableCooldownSec`)
-  - Create GitHub issues for disabled plugins
-- **Files:** `index.ts`, `test/index.test.ts`
-- **Definition of done:** Failing plugins are detected via JSON API and auto-disabled; tests cover detection and disable logic.
-- **GitHub Issue:** #3
-
----
-
 ## Recently Completed
 
-| Task  | Title | Date |
-|-------|-------|------|
-| T-013 | Write status snapshot file on each monitor tick | 2026-03-02 |
-| T-012 | Add startup configuration validation with fail-fast behavior | 2026-03-02 |
-| T-011 | Add integration tests for monitor service tick flows | 2026-03-02 |
-| T-004 | Add TypeScript build pipeline and type-checking | 2026-03-01 |
-| T-010 | Expose self-heal status for external monitoring | 2026-02-28 |
+| Task | Title | Date |
+|------|-------|------|
+| T-015 | Fix infinite gateway restart loop | 2026-03-07 |
+| T-005 | Plugin health monitoring via JSON API | 2026-03-07 |
+| T-013 | Status snapshot file on each monitor tick | 2026-03-02 |
+| T-012 | Startup config validation (fail-fast) | 2026-03-02 |
+| T-011 | Integration tests for monitor tick flows | 2026-03-02 |
+| T-004 | TypeScript build pipeline + typecheck | 2026-03-01 |
+| T-010 | Observability events + status endpoint | 2026-02-28 |
+| T-009 | Dry-run mode | 2026-02-28 |
+| T-008 | Active recovery probing | 2026-02-28 |
+| T-007 | Config hot-reload | 2026-02-28 |

package/.ai/handoff/STATUS.md

@@ -1,20 +1,37 @@
-# STATUS (AAHP)
-
-(Verified/Assumed/Unknown)
-
-## Current State
-- Status: (Verified) Scaffold complete, v0.3 roadmap defined
-- Version: 0.2.4
-- CI: (Unknown) No build/test pipeline yet - top priority
-
-## Architecture Notes
-- (Verified) Single-file plugin (`index.ts`) using openclaw plugin API
-- (Verified) Three healing domains: model failover, WhatsApp reconnect, cron failure
-- (Verified) Plugin disable feature is a stub (blocked on structured API)
-- (Verified) State persisted to JSON file, no external dependencies
-
-## Risks / Open Questions
-- (Verified) Zero test coverage - any change risks silent regressions
-- (Verified) No TypeScript type-checking in build pipeline
-- (Unknown) When will `openclaw plugins list --json` be available?
-- (Verified) `gh` CLI not authenticated - GitHub issue creation deferred
+# STATUS.md — openclaw-self-healing-elvatis
+
+_Last updated: 2026-03-07 by Akido (claude-sonnet-4-6)_
+
+## Current Version: 0.2.8 — STABLE
+
+## What is done
+
+- ✅ Repo: `https://github.com/elvatis/openclaw-self-healing-elvatis`
+- ✅ npm: `@elvatis_com/openclaw-self-healing-elvatis@0.2.8`
+- ✅ ClawHub: `openclaw-self-healing-elvatis@0.2.8`
+- ✅ Model failover: rate-limit + auth-scope cooldown, configurable fallback order
+- ✅ WhatsApp reconnect: disconnect streak detection + gateway restart
+- ✅ Cron failure: consecutive fail threshold → disable + GitHub issue
+- ✅ Plugin health monitoring: `openclaw plugins list --json` → auto-disable on crash/error
+- ✅ Config guardrails: never modify/restart if `openclaw.json` is invalid JSON
+- ✅ Startup config validation (fail-fast)
+- ✅ Status snapshot: written to file on every tick for external monitoring
+- ✅ Active recovery probing: polls limited models to detect early recovery
+- ✅ Dry-run mode: full behavior simulation without side effects
+- ✅ Config hot-reload: reads `api.pluginConfig` on each tick
+- ✅ Atomic state writes: tmp file + rename to avoid partial writes
+- ✅ 181 tests across 2 suites (unit + integration)
+- ✅ TypeScript build + typecheck pipeline
+- ✅ `.clawhubignore` + rsync-based ClawHub publish documented in CONVENTIONS.md
+
+## Critical Bug Fixed (v0.2.8)
+
+**Infinite gateway restart loop** — `lastRestartAt` was being saved AFTER
+`openclaw gateway restart`, which kills the process via systemd. On the next
+boot, `lastRestartAt = 0`, bypassing the `whatsappMinRestartIntervalSec`
+rate-limit guard. Fixed by saving state BEFORE the restart call.
+
+## Open Risks
+
+- `T-014`: Heal metrics export to JSONL not yet implemented (low priority, no blocker)
+- ClawHub publish via `clawhub publish <dir>` ignores `.clawhubignore` — use rsync workaround (see CONVENTIONS.md)

package/.clawhubignore

@@ -0,0 +1,8 @@
+node_modules
+.git
+dist
+*.tgz
+npm-debug.log*
+.vscode
+.idea
+package-lock.json

package/README.md

@@ -1,27 +1,34 @@
 # openclaw-self-healing-elvatis
 
+**Current version: `0.2.8`**
+
 OpenClaw plugin that improves resilience by automatically fixing reversible failures.
 
-## What it can heal (v0.2)
+## What it heals
+
+- **Model outage** — Detect rate limit / quota / auth-scope failures, put model into cooldown, patch pinned session to a safe fallback
+- **WhatsApp disconnect** — If WhatsApp appears disconnected repeatedly: restart the gateway (streak threshold + minimum restart interval guard)
+- **Cron failures** — If a cron job fails repeatedly: disable it + create a GitHub issue
+- **Plugin crashes** — If a plugin reports `status=error` or `status=crash`: auto-disable + GitHub issue
 
-Implemented now:
+## Changelog
 
-- Model outage healing
-  - Detect rate limit / quota / auth-scope failures
-  - Put the affected model into cooldown
-  - Patch pinned session model overrides to a safe fallback (prevents endless `API rate limit reached` loops)
+### v0.2.8 — 2026-03-07
+**Fix: Infinite gateway restart loop**
+`lastRestartAt` and `disconnectStreak` are now saved to disk **before** calling
+`openclaw gateway restart`. Previously they were saved after, but systemd kills
+the process during restart — state was never persisted, the rate-limit guard was
+bypassed on every boot, causing an infinite restart loop when used alongside
+any plugin that triggers a config-driven gateway restart (e.g. `openclaw-cli-bridge-elvatis`).
 
-- WhatsApp disconnect healing
-  - If WhatsApp appears disconnected repeatedly: restart the gateway
-  - Guardrails: streak threshold + minimum restart interval
+### v0.2.7 — 2026-03-07
+Fix `runCommandWithTimeout` call signature + field name.
 
-- Cron failure healing (optional)
-  - If a cron job fails repeatedly: disable it
-  - Create a GitHub issue with last error context (rate limited)
+### v0.2.6 — 2026-03-02
+Status snapshot file, startup config validation, integration tests.
 
-Not implemented yet (next):
-- Plugin install error rollback (disable plugin) based on structured plugin status
-  - Waiting for `openclaw plugins list --json` or an equivalent stable API
+### v0.2.5 and earlier
+Model failover, WhatsApp reconnect, cron failure, dry-run mode, active recovery probing, config hot-reload.
 
 ## Install
 

package/index.ts

@@ -618,7 +618,11 @@ export default function register(api: any) {
         if (config.whatsappRestartEnabled) {
           const st = await runCmd(api, "openclaw channels status --json", 15000);
           if (st.ok) {
-            const parsed = safeJsonParse<any>(st.stdout);
+            // openclaw CLI prints plugin startup lines (e.g. "[plugins] [self-heal] enabled...")
+            // to stdout before the JSON payload. Extract the first JSON object to avoid
+            // safeJsonParse returning undefined and falsely treating WA as disconnected.
+            const jsonMatch = st.stdout.match(/\{[\s\S]*\}/);
+            const parsed = jsonMatch ? safeJsonParse<any>(jsonMatch[0]) : undefined;
             const wa = parsed?.channels?.whatsapp;
             const connected = wa?.status === "connected" || wa?.connected === true;
 
@@ -651,12 +655,18 @@ export default function register(api: any) {
                   if (!v.ok) {
                     api.logger?.error?.(`[self-heal] NOT restarting gateway: openclaw.json invalid: ${v.error}`);
                   } else {
+                    // CRITICAL: persist lastRestartAt + reset streak BEFORE calling gateway restart.
+                    // openclaw gateway restart kills this process via systemd. Any state updates
+                    // placed AFTER runCmd will never execute, leaving lastRestartAt=0 and
+                    // bypassing the whatsappMinRestartIntervalSec rate-limit guard on the next boot.
+                    // This was the root cause of the infinite restart loop.
+                    state.whatsapp!.lastRestartAt = nowSec();
+                    state.whatsapp!.disconnectStreak = 0;
+                    saveState(config.stateFile, state);
                     backupConfig("pre-gateway-restart");
                     await runCmd(api, "openclaw gateway restart", 60000);
                     // If we are still alive after restart, attempt cleanup.
                     await cleanupPendingBackups("post-gateway-restart");
-                    state.whatsapp!.lastRestartAt = nowSec();
-                    state.whatsapp!.disconnectStreak = 0;
                   }
                 }
                 api.emit?.("self-heal:whatsapp-restart", {
@@ -672,7 +682,8 @@ export default function register(api: any) {
         if (config.disableFailingCrons) {
           const res = await runCmd(api, "openclaw cron list --json", 15000);
           if (res.ok) {
-            const parsed = safeJsonParse<any>(res.stdout);
+            const cronJsonMatch = res.stdout.match(/\{[\s\S]*\}/);
+            const parsed = cronJsonMatch ? safeJsonParse<any>(cronJsonMatch[0]) : undefined;
             const jobs: any[] = parsed?.jobs ?? [];
             for (const job of jobs) {
               const id = job.id;
@@ -761,7 +772,8 @@ export default function register(api: any) {
           const res = await runCmd(api, "openclaw plugins list --json", 15000);
           if (res.ok) {
             type PluginEntry = { id: string; name?: string; enabled?: boolean; status?: string; version?: string; error?: string };
-            const parsed = safeJsonParse<{ plugins: PluginEntry[] }>(res.stdout);
+            const pluginJsonMatch = res.stdout.match(/\{[\s\S]*\}/);
+            const parsed = pluginJsonMatch ? safeJsonParse<{ plugins: PluginEntry[] }>(pluginJsonMatch[0]) : undefined;
             const plugins: PluginEntry[] = parsed?.plugins ?? [];
             const selfId = "openclaw-self-healing-elvatis";
 

package/openclaw.plugin.json

@@ -1,7 +1,7 @@
 {
   "id": "openclaw-self-healing-elvatis",
   "name": "OpenClaw Self Healing",
-  "version": "0.2.7",
+  "version": "0.2.9",
   "description": "Self-healing health checks + guardrails + auto-fix for reversible failures (rate limits, auth errors, stuck session pins).",
   "configSchema": {
     "$schema": "http://json-schema.org/draft-07/schema#",

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@elvatis_com/openclaw-self-healing-elvatis",
-  "version": "0.2.7",
+  "version": "0.2.9",
   "type": "module",
   "openclaw": {
     "extensions": [